17Category-level / unseen objectsOpen worldNeural 3D representationsEmbodied perception

Extensions & Frontier Problems

Frontiers are defined by
changing problem boundaries

As assumptions about CAD, identity, rigidity, static scenes, or passive observations are relaxed, output spaces, observability, and evaluation protocols must also change. Learn to identify which assumptions a new capability removes and where it shifts the cost.

Open the capability-boundary lab Freeze the task contract first

10-second summary

Remember eight rules

01
Write the task tuple first: prior, observation, output, generalization, adaptation, budget.
02
CAD-free ≠ prior-free: reference images/video, category prototypes, and test-time reconstruction all provide object information.
03
9D commonly means pose + size: not nine rotational degrees of freedom; monocular metric scale requires additional evidence.
04
Open vocabulary also needs absent-target tests: text-conditioned localization on positive samples alone does not establish a complete open world.
05
Foundation models are components, not guarantees: geometric consistency, metric scale, calibration, real-time performance, and safety each need evidence.
06
Nonrigid states require more than one SE(3): parts, joints, deformation, and time enter the state definition.
07
More modalities bring information and calibration requirements: event, tactile, F/T, and vision must align in space and time.
08
The weakest evidence limits the strongest claim: splits, priors, budgets, failure denominators, and task outcomes are all essential.

01 · Problem Regime Before Method Name

Every frontier claim starts with six questions: what is given, observed, output, generalized to, adapted when, and spent in resources?

Numbers are not directly comparable across problem boundaries. Priors range from exact CAD to references, categories, text, or none; outputs expand from SE(3) to size, shape, joints, trajectories, beliefs, and actions.

Problem family	Allowed priors	Output state	Capability actually tested
Instance-level · Known objects	Exact CAD / texture / symmetry; object identity is known or detectable	Metric SE(3) for one rigid body, sometimes with score / covariance	BOP model-based tasks, industrial localization, and tracking mapped objects
Unseen instances · Model-based	Test objects were unseen during training, but onboarding may read CAD and render / build indices	Metric SE(3) for a novel instance	Tests cross-object generalization while retaining CAD
Unseen instances · Model-free	No CAD; reference RGB/RGB-D images or video and bounded onboarding are usually provided	SE(3) relative to a reference-object frame, sometimes with scale / reconstruction	CAD-free setups can still use object priors and incur onboarding costs
Category-level 6D / 9D	Known categories, training instances, canonical axes / shape priors; new instances of those categories at test time	Rotation + translation; 9D commonly adds 3D size	Defining canonical frames and handling instance variation are part of the problem
Open vocabulary / open world	Text or reference queries; categories are not closed and the target may be absent	Detection/segmentation + relative/absolute pose + unknown/abstain	Good positive-target localization does not establish unknown-object rejection
Joint pose–shape estimation	Category, generative/reconstruction priors, or multiview onboarding	pose + metric shape / implicit field / uncertainty	Disclose gauge, scale, shape–pose coupling, and computational costs
Articulated / deformable / dynamic	Part graphs, joint types, canonical templates, deformation models, or temporal priors	root/part poses, joint states, deformation field, trajectory	One SE(3) no longer describes the full state
Embodied / task-conditioned	Robot state, action space, task equivalence, safety, and cost	Belief / action / task-equivalent poses rather than only one matrix	Perception quality depends jointly on information gain and task outcomes

A reproducible frontier task is a typed tuple

\mathcal T=\big(\mathcal P_{\mathrm{object}},\ \mathcal O_{\mathrm{sensor}},\ \mathcal Y_{\mathrm{output}},\ \mathcal G_{\mathrm{split}},\ \mathcal A_{\mathrm{adapt}},\ \mathcal B_{\mathrm{resource}},\ \mathcal M_{\mathrm{metric}}\big)

P is the prior available before inference / during onboarding; O is current observations; Y is output state; G is the instance/category boundary; A is permitted adaptation; B is time/compute/acquisition budget; M is the matching, symmetry, failure, and statistics protocol. Changing any field may define a different task.

01

novel / unseen object

The test physical instance did not appear in object-specific training

Does not specify whether CAD, references, category training, or test-time optimization are available

02

zero-shot

Usually means no gradient training or object-specific fine-tuning for that test object

May still use CAD rendering, template libraries, reference video, pretrained features, or test-time optimization

03

CAD-free / model-free

Inference or benchmark protocols provide no exact CAD mesh

May provide reference images/video, scale, depth, object masks, or onboarding time

04

category-level

The test instance is unseen, but its category and canonical convention are known

Neither category-agnostic nor open-set

05

open-vocabulary

Targets can be specified by open text / cross-modal queries

Does not automatically establish 6D absolute pose, metric scale, unknown rejection, or arbitrary-category generalization

06

foundation model / feature

Reuses general representations or generative capabilities from large-scale pretraining

Does not automatically establish coordinate consistency, metric estimates, calibrated uncertainty, real-time performance, or robot safety

07

mesh-free / neural representation

Localization uses fields, point/feature maps, or splats instead of an explicit triangle mesh

The representation still requires acquisition, coordinate gauge, scale, storage, initialization, and optimization

Account for hidden priors.Pretraining corpora, detectors/masks, CAD renders, reference capture, known size, gravity/table planes, calibrated motion, initial poses, and test-time reconstruction all narrow the solution space. Their use can be valid, but must appear in the contract and baseline.

Interactive Lab

Turn terminology into explicit task boundaries and verify why monocular images cannot determine metric scale

The first panel generates formal task labels. The second shows exactly unchanged pixels when size and depth scale together. The third checks evidence gaps against a frozen headline. Open directly with ?lab=ambiguity or ?lab=evidence.

Task presets

Choose a capability claim, then edit each contract field

typed task

Object priorAvailable before inference / during onboarding ObservationCurrent measurements Output stateWhat the system promises to recover GeneralizationTest / training object relationship Adaptation budgetPermitted optimization / onboarding for test objects

Formal task label

—

Current contract Retained assumptions Relaxed assumptions A method name cannot replace these five fields

Relaxed

Relative to a classical instance-level pipeline

—Relaxing one assumption often adds new priors or adaptation costs

Retained / paid

Remaining dependencies

—Disclose training categories, reference capture, calibration, depth, initialization, and compute

Observability limit

What current inputs cannot determine

—A larger network cannot create measurements of unobservable quantities

Required evidence

—

——

Keep terminology distinct 'Novel object' describes instance relationships; 'CAD-free / model-free' describes asset priors; 'zero-shot' describes updating; 'open-vocabulary' describes query/category boundaries. They can be combined, but are not synonyms.

02 · Category-level Pose, Size & Canonical Space

Category-level pose starts by defining how different instances share an object frame

For exact CAD, the object frame binds to one asset. For new instances, canonical axes, origin, scale, and symmetry are modeling choices. Unstable definitions make even accurate networks fit inconsistent labels.

Canonical point to metric camera point

\mathbf p_C=\mathbf R\,\operatorname{diag}(\mathbf s)\,\mathbf p_N+\mathbf t,\qquad \mathbf p_N\in\mathcal C_{\mathrm{canonical}}

p_N may be a point in NOCS / normalized shape space. s denotes metric dimensions or per-axis scales; for uniform scale only, use sI. The common 9D output counts rotation 3 + translation 3 + size 3.

Category-level correspondence fitting

\min_{\mathbf R\in\mathrm{SO}(3),\,\mathbf t,\,\mathbf s}\sum_{i\in\mathcal I}w_i\,\rho\!\left(\left\|\mathbf p_{C,i}-\mathbf R\operatorname{diag}(\mathbf s)\mathbf p_{N,i}-\mathbf t\right\|_2^2\right)

Reliable absolute-size recovery needs metric depth or another metric constraint. Record outliers in canonical correspondences, completed points, and observed points separately.

01

Canonical frame

Shared origin, axes, front direction, and scale conventions across category instances

A cup handle or camera-lens direction may define yaw; an unmarked cylinder retains continuous ambiguity

02

NOCS / normalized coordinates

Maps instance surfaces into a unit canonical space to establish observed-point / category-coordinate correspondences

Incorrect training-CAD canonicalization directly becomes pose-label bias

03

Shape prior / prototype

Completes unseen regions with means/prototypes, deformation bases, implicit decoders, or retrieved shapes

Separate completion confidence from observed support; plausible completion is not a measurement

04

Pose + size fitting

RGB-D aligns camera points to canonical points with scale to recover R, t, and dimensions

Specify uniform scale, anisotropic dimensions, bbox conventions, and object origin

05

Intra-class symmetry

Instances may be more symmetric than category labels suggest; semantic axes can also differ across instances

Use instance-aware symmetry / equivalence rather than applying one discrete group to the entire category

06

Evaluation split

Group by physical instance, room/table, capture session, and background

Leakage through adjacent frames, reconstructed meshes, or reference videos of the same instance disguises memorization as category generalization

Single-view projective scale family

\pi\!\left(\mathbf K\,\lambda(\mathbf R\mathbf X+\mathbf t)\right)=\pi\!\left(\mathbf K(\mathbf R\mathbf X+\mathbf t)\right),\qquad \lambda>0

Scaling object shape/size and camera translation together by λ leaves monocular pixels unchanged. Networks may infer common sizes from training distributions, but that remains a prior; validate metric scale separately under cross-category / domain shift.

6DR + tRequires a defined object frame and metric scale

9DR + t + sizeCommon bookkeeping rather than a single mathematical group

Pose + shapeR + t + z / fieldSpecify shape gauge, units, and observed/completed regions

Task quotient[T] / actionsDistinguishes only modes that change downstream outcomes

Prerequisite review: Symmetry, Ambiguity & Uncertainty, Pose Evaluation & Error Analysis.

03 · Novel Object, CAD-free & Open Vocabulary

An unseen training object defines the generalization target, not the information available at inference

Novel-object systems may read test CAD, model objects from reference video, or use one anchor. Before comparing, disclose onboarding, references, scale, masks, initialization, caches, and test-time optimization.

01

CAD onboarding

Read mesh, units, origin, texture, and symmetry; render templates offline or build a 3D-feature index

Include onboarding time, GPU count, rendered views, cache size, and failed objects in the protocol

02

Reference onboarding

Recover object appearance, relative camera trajectories, scale, or neural representations from short RGB/RGB-D videos

Reference coverage, background, handheld occlusion, camera calibration, and object-frame definitions set the upper bound

03

Single-anchor matching

Establish cross-view correspondences from one or a few reference views, then solve relative pose / scale

Large out-of-plane changes, unseen backsides, and repeated textures are structural blind spots

04

Test-time reconstruction

Optimize shape/fields/poses on test objects, using multiview consistency in CAD-free scenes

Counts toward adaptation costs; include initialization, iterations, peak VRAM, termination, and failure rates

05

Open-vocabulary query

Text/image queries first locate semantic targets, then establish cross-view or 2D–3D geometry

Detection, identity association, pose, and scale are separate responsibilities

06

True no-update inference

Uses shared networks/features directly without creating new parameters or caches for the test object

Still depends on pretraining data and task definitions; audit whether test objects/categories appeared in the pretraining corpus

Reference-conditioned relative pose

{}^{C_q}\mathbf T_O={}^{C_q}\mathbf T_{C_r}\,{}^{C_r}\mathbf T_O,\qquad {}^{C_q}\mathbf T_{C_r}={} ^{C_q}\mathbf T_O\left({}^{C_r}\mathbf T_O\right)^{-1}

Two views of one rigid body can provide a relative camera/object relationship. Whether this becomes a public dataset's absolute pose depends on the reference O-frame definition, metric scale, gravity, and canonical axes.

Open-set decision includes non-existence

\hat y=\begin{cases}(\hat m,\hat{\mathbf T},\hat u),&p(\mathrm{target\ present}\mid\mathcal O,q)\ge\tau\\ \varnothing\ \text{or abstain},&\text{otherwise}\end{cases}

q is a text/image query. Evaluation must include absent targets, near-semantic distractors, unknown categories, repeated instances, and incorrect queries; scoring cannot begin only after an oracle confirms target presence.

Model-based unseenUnseen instance + test CAD

Focuses on shared features, rendering/retrieval, and bounded CAD onboarding.

Model-free unseenUnseen instance + reference capture

Focuses on mesh-free object onboarding, frame/scale establishment, and cross-view matching.

Category-levelKnown category + shared canonical prior

Focuses on within-category shape variation and canonical pose/size.

Open-vocabularyOpen query + open-set decision

Focuses on joint semantic localization, instance association, geometry, and rejection.

04 · Foundation Features, Reconstruction & Generative Priors

Large models change priors and initialization; geometric constraints, coordinate gauge, and failure modes remain

General segmentation/matching features reduce object-specific training, feed-forward 3D models quickly provide geometry and camera relationships, and generative models represent multiple hypotheses. All must enter verifiable solvers and protocols.

01

Foundation 2D / 3D features

Transferable features for detection, segmentation, dense correspondence, retrieval, and similarity

Validate cross-view stability, geometric accuracy, physical-instance discrimination, and domain shift separately

02

Feed-forward 3D reconstruction

Predicts point maps, camera relationships, or scene geometry directly from image pairs/video, then estimates target poses

Outputs often retain gauge / scale freedom; object masks, static-scene assumptions, and camera-motion conditions must be stated

03

Generative pose distribution

Uses diffusion/flow/energy to generate multiple keypoint, rotation, or pose hypotheses

Multiple generated samples do not establish a calibrated posterior; validate mode coverage and probability quality

04

NeRF / implicit field

Continuous density, color, SDF, or feature fields support analysis-by-synthesis and mesh-free tracking

Optimization basins, view-dependent appearance, shape–radiance ambiguity, and onboarding costs

05

3D Gaussian Splatting

Explicit Gaussian positions, covariances, opacity, and view-dependent colors enable fast differentiable rendering

High rendering FPS does not establish real-time pose solving; scene gauge, dynamic updates, and object separation remain bottlenecks

06

Hybrid geometry + learned prior

Classical PnP/Umeyama/ICP/BA enforces physical constraints; learned modules estimate correspondences, weights, completion, or initialization

Retain raw correspondences, solver conditioning, and independent residuals so end-to-end scores do not hide degeneracy

Learned correspondence, geometric solve

\{(\mathbf x_i,\mathbf X_i,w_i)\}=f_\theta(\mathcal O,\mathcal P),\qquad (\hat{\mathbf R},\hat{\mathbf t})=\operatorname{RobustSolve}\!\left(\{(\mathbf x_i,\mathbf X_i,w_i)\}\right)

Foundation features can generate correspondences and weights, but PnP/3D-alignment rank, inliers, cheirality, symmetry, and held-out residuals still need explicit checks.

A generated set is not yet a calibrated posterior

\{\mathbf T^{(k)}\}_{k=1}^{K}\sim q_\theta(\mathbf T\mid\mathcal O),\qquad \text{coverage}_\alpha=\Pr\!\left(\mathbf T^\star\in\mathcal C_\alpha(\mathcal O)\right)

Validate mode recall, set precision, coverage, reliability, and task risk. Attractive, diverse samples do not establish that qθ's probability mass matches actual frequencies.

A foundation checkpoint is not a measurement.

Pretraining expands empirical priors and may improve detection, masks, correspondence, completion, or initialization. Data and protocols must still establish metric-scale sources, camera/object alignment, unknown-target rejection, and calibration.

05 · NeRF, 3D Gaussian Splatting & Neural Object Maps

Neural representations can replace explicit meshes; frames, scale, initialization, and observation coverage remain essential

Analysis-by-synthesis treats pose as an inverse differentiable-rendering problem. Joint representation and camera/object-pose optimization approaches neural bundle adjustment. Success depends on gauge, basins, occlusion, and appearance modeling.

NeRF volume rendering along a camera ray

\hat{\mathbf C}(\mathbf r)=\int_{t_n}^{t_f}T(t)\,\sigma(\mathbf r(t))\,\mathbf c(\mathbf r(t),\mathbf d)\,dt,\qquad T(t)=\exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf r(s))ds\right)

Field coordinates define the map gauge. If training cameras are known only up to scale, the field and subsequent poses also remain up to scale. View-dependent color can additionally absorb geometric errors.

Pose by differentiable rendering inversion

\hat{\boldsymbol\xi}=\arg\min_{\boldsymbol\xi}\sum_{u\in\Omega}w_u\,\rho\!\left(\mathbf I(u)-\mathcal R_\Phi\!\left(u;\exp(\boldsymbol\xi^\wedge)\mathbf T_0\right)\right)

T0, ray sampling, masks/occlusion, lighting mismatch, and optimizer basins all affect results. Low photometric residuals may reflect flexible appearance rather than correct geometry.

3D Gaussian alpha compositing

\hat{\mathbf C}(u)=\sum_{i\in\operatorname{sort}(u)}\mathbf c_i\,\alpha_i(u)\prod_{j<i}\big(1-\alpha_j(u)\big)

Each Gaussian has position, covariance, opacity, and appearance. Fast rasterization helps pose optimization, but sorting, visibility, object/scene separation, dynamic updates, and metric gauge remain system issues.

Joint representation–pose optimization has gauge freedom

(\Phi^\star,\{\mathbf T_k^\star\})=\arg\min_{\Phi,\{\mathbf T_k\}}\mathcal L_{\mathrm{render}},\qquad (\mathbf S\Phi,\{\mathbf T_k\mathbf S^{-1}\})\ \text{may render equivalently}

Fix a world/object frame, scale, or camera reference. Otherwise coordinate values from different runs are incomparable, and object poses cannot directly enter robot-base coordinates.

Capture

Reference coverage and camera trajectories

Views, distances, lighting, backgrounds, occlusion, and motion blur determine the representation's observed domain.

Onboarding

Optimization time / GPU / memory

Include cold starts, failed reconstructions, caches, and representation size in the resource envelope.

Localization

Initialization, basin, and per-frame latency

Separate rendering FPS from end-to-end pose FPS; record multiple starts and tracking resets.

Geometry

Gauge, scale, and shape fidelity

Validate independently with depth, meshes, landmarks, or robot touch, alongside novel-view PSNR.

System review: Multiview Geometry & Visual SLAM, Point Cloud Registration, Engineering Implementation & Debugging.

06 · Beyond One Rigid SE(3)

When objects open, bend, slip, or become hand-occluded, expand the state before choosing an estimator

Articulated objects need kinematic graphs; deformable objects need canonical material coordinates and deformation; dynamic worlds also need time, association, and births/deaths. Fitting one global rigid pose merely moves structural errors into residuals.

World model	Minimum structure	Typical state
Rigid object	One body frame and SE(3)	T_CO(t); an equivalence class [T] for symmetric objects
Category instance	canonical frame + metric dimensions / shape code	R, t, s or R, t, z_shape
Articulated object	root frame + part frames + joint graph / limits	T_Croot, {q_j}, {T_Cpart}
Deformable object	canonical material coordinates + deformation field / mesh state	φ_t(x), latent z_t, or dense correspondences
Dynamic scene	Multiple object trajectories, births/deaths, occlusion, camera motion, and time	{T_COi(t)} + association + covariance
Hand–object system	Hand kinematics, object poses, contacts, slip, and affordances	q_hand(t), T_HO(t), contact state / wrench
Task-equivalent state	Distinguishes only degrees of freedom that change grasping, insertion, or human-interaction outcomes	Quotient / action sets, without forcing a visually unidentifiable unique frame

Articulated part transform through a kinematic chain

{}^{C}\mathbf T_{P_k}={}^{C}\mathbf T_{\mathrm{root}}\prod_{j\in\operatorname{path}(k)}\exp\!\left(\hat{\boldsymbol\xi}_j q_j\right)\,{}^{P_{j^-}}\mathbf T^{0}_{P_j}

Also requires joint types/axes/origins/limits, part topology, and a zero configuration. Independent SE(3) estimates per part lose physical constraints and may produce interpenetration.

Canonical-to-deformed correspondence

\mathbf x_t=\boldsymbol\phi_t(\mathbf x_0)=\mathbf x_0+\mathbf d_\psi(\mathbf x_0,\mathbf z_t),\qquad \mathbf x_0\in\mathcal C_{\mathrm{material}}

Deformation fields, canonical poses, and camera poses can absorb each other's errors. Material correspondence, invertibility, local rigidity/elasticity, and topology changes need object-specific constraints.

Dynamic object trajectory on SE(3)

\mathbf T_{k+1}=\mathbf T_k\exp\!\left((\Delta t\,\mathbf v_k+\mathbf w_k)^\wedge\right),\qquad p(\mathbf T_{k+1}\mid\mathbf T_k,\mathbf a_k)

Timestamps, rolling shutter, object/camera motion, and robot actions enter the transition model. Mean per-frame tracking scores cannot replace loss/recovery, identity-switch, and latency evaluation.

Hand–object relative pose and slip

{}^{H}\mathbf T_O(t)=\left({}^{W}\mathbf T_H(t)\right)^{-1}{}^{W}\mathbf T_O(t),\qquad \Delta\mathbf T_{\mathrm{slip}}={}^H\mathbf T_O(t)^{-1}{}^H\mathbf T_O(t+\Delta t)

Joint hand-articulation, tactile-contact, and object-pose estimation can maintain beliefs under visual occlusion. Contact changes observations and may physically push or slide the object.

07 · Hard Materials, Egocentric & New Modalities

New modalities change observation equations, timing, and visibility, beyond adding another tensor

Transparent/specular materials challenge conventional appearance and depth; events capture fast changes; touch provides local geometry after contact; egocentric sensing couples hand, camera, and object motion. Define each modality's frames and clocks before fusion.

01

Transparent / specular RGB-D

Texture reflections, refraction, invalid ToF/structured-light depth, and multipath

Validity/confidence, multiple views, polarization/thermal/active sensing; depth holes are not background

02

Event camera

Asynchronous brightness changes, microsecond temporal resolution, and high dynamic range; no absolute-brightness frames

High-speed tracking and blur-free motion cues; requires event–frame/depth/IMU clocks and extrinsics

03

Tactile sensing

Local geometry, normals, shear, slip, and forces after contact

Can reduce hand–object belief uncertainty during visual occlusion, but observes only contacted regions, and contact itself changes state

04

Egocentric multi-view

Head-mounted cameras, hands, targets, and environments move rapidly together under strong occlusion

Joint hand–object–camera tracking, gaze/attention, and long-term identity; human self-occlusion and rolling shutter are prominent

05

Robot proprioception / F/T

Joints, TCP, commands, contact wrenches, and controller state

Places object poses in base/task frames and validates contact events; depends on hand-eye/TCP/time calibration

06

Audio / language / scene context

Material, events, target semantics, and task priors

Can narrow hypotheses or select actions, but usually does not independently measure metric pose

Event generation model

e_k=(u_k,t_k,p_k),\qquad L(u_k,t_k)-L(u_k,t_k^-)\approx p_k C,\quad p_k\in\{-1,+1\}

Events record log-intensity changes rather than absolute brightness. C, timestamps, refractory behavior/noise, and motion determine event likelihood; fusion with depth/frames/robot state must use acquisition time.

Multimodal belief update with explicit calibration

p(\mathbf x_t\mid\mathcal Z_{1:t})\propto p(\mathbf z_t^{\mathrm{rgb}},\mathbf z_t^{\mathrm{depth}},\mathbf z_t^{\mathrm{event}},\mathbf z_t^{\mathrm{tactile}}\mid\mathbf x_t,\boldsymbol\theta_{\mathrm{cal}})\,p(\mathbf x_t\mid\mathcal Z_{1:t-1})

θcal includes intrinsics, extrinsics, time offsets, depth/tactile scale, and robot kinematics. Even when calibration is treated as fixed, propagate its errors through the uncertainty budget.

Transparent / specular

Correct masks, but depth holes / ghosts

Evaluate detection, valid depth, shape completion, and pose separately; completed geometry is not sensor truth.

Event

No events at rest, frame blur at high speed

Slice by speed, illumination, contrast, and time offset; report latency for reconstruction and pose stages separately.

Tactile

Local contact matches several global surfaces

Retain multimodal global hypotheses; actively change contact points and combine hand kinematics with nonpenetration.

Egocentric

Hands and objects occlude together while the camera moves

Estimate camera/hand/object states jointly; separately report identity switches, out-of-view events, and re-entry recovery.

08 · Active, Embodied & Task-aware Pose

Systems can change future observations when current ones are insufficient, but actions cost time, risk, and disturbance

Active perception expands a single estimate into a belief–action–observation loop. The best next view provides the most task-relevant information about remaining pose modes while remaining safely executable.

01

Next-best-view

Selects camera/robot viewpoints to maximize expected visibility, entropy reduction, or mode separation

Reachability, collision, time, and motion blur all impose costs

02

Active illumination / sensor choice

Changes observation conditions through exposure, lights, polarization, depth, or event sensing

Gains come from new measurements rather than rerunning inference on the same frame

03

Manipulation for perception

Pushes, rotates, lifts, or touches objects to expose surfaces / contact constraints

Actions change object state; account for belief transitions and safety boundaries

04

Temporal filtering / tracking

Propagates multimodal beliefs with motion models and past observations

Incorrect associations and model drift accumulate; redetection and reset belong to the protocol

05

Task-aware stopping

Stops observing when all remaining pose modes are action-equivalent or risk is below threshold

A unique canonical pose irrelevant to the task need not be recovered for a visual benchmark

06

Safe abstention / recovery

Outputs no-pose / request-view / human-check when observations are insufficient, targets are unknown, or inputs exceed training support

Coverage–risk curves and abstention denominators are more informative than averages over successful cases alone

Bayesian belief update after an action

b_{t+1}(\mathbf x')\propto p(\mathbf z_{t+1}\mid\mathbf x',\mathbf a_t)\int p(\mathbf x'\mid\mathbf x,\mathbf a_t)b_t(\mathbf x)\,d\mathbf x

a may move the camera, grasp/push the object, switch illumination, or request human help. Actions can change object state as well as camera pose.

Task- and cost-aware next action

\mathbf a^\star=\arg\max_{\mathbf a\in\mathcal A_{\mathrm{safe}}}\underbrace{\mathbb E\!\left[H(b_t)-H(b_{t+1})\right]}_{\text{information gain}}-\lambda_c C(\mathbf a)-\lambda_r\,\operatorname{Risk}(\mathbf a)

Entropy is one possible objective; task entropy is more appropriate when multiple visual poses imply equivalent grasps. Also compare actual wall time, path length, collision margins, and observation failures.

Stop when residual uncertainty is task-safe

\operatorname{Stop}\iff \Pr_{\,\mathbf T\sim b_t}\!\left(g(\mathbf T,\mathbf a_{\mathrm{task}})\in\mathcal S_{\mathrm{safe}}\right)\ge 1-\delta\quad\text{or no safe informative action remains}

g maps poses to clearance, contact, insertion, or overlay error. If constraints cannot be satisfied, abstain/recover instead of treating a posterior mean as a certain pose.

Robot pipeline: Robot Coordinate Transforms & System Calibration, Robotic Grasping & Manipulation.

09 · Claim–Assumption–Evidence Protocol

The weakest protocol component limits a frontier claim's credibility

Freeze the headline capability first, then provide independent evidence for every relaxed assumption. Results must address mean accuracy, failure locations, awareness of uncertainty, resource costs, and whether tasks become safer.

Evidence layer	Must freeze / report	Misleading conclusion prevented
Task contract	Freeze priors, observations, outputs, instance/category relationships, adaptation, and budgets	Avoid comparing different tasks under the same 'zero-shot' label
Split & leakage audit	Deduplicate physical instances / scenes / sessions / reference assets / pretraining corpora	Establish that generalization is not adjacent-frame, shared-mesh, or network memorization
Metric & equivalence protocol	Units, scale sources, canonical axes, proper symmetry, matching, and thresholds	Defines what 'pose correct' means in the headline
Complete denominator	Include no-detection, abstention, timeout, onboarding failure, unknown targets, and tracking loss	Prevents accuracy reporting restricted to successful solver cases
Resource envelope	Onboarding and per-frame wall time, GPU/CPU, VRAM/RAM, caches, energy, and reference capture	Turns hidden computational priors into comparable costs
Uncertainty & calibration	Pose sets/distributions, coverage, reliability, risk–coverage, and OOD/unknown behavior	Multiple hypotheses do not inherently carry probability semantics
Robust slices	visibility, symmetry, size, distance, motion, material, light, background, category/instance	Means do not represent tail failures or deployment boundaries
Downstream / safety	Grasping, insertion, tracking, AR overlays, collision, damage, and recovery outcomes	Establishes that geometric improvements translate to task value
Fair baseline & oracle	Identical masks/detectors, data, budgets, refinement, and postprocessing; pose/shape/mask oracles	Locates the layer responsible for gains rather than protocol advantages
Reproducible artifact	Code, models, data manifests, random seeds, raw predictions, failed samples, and versions	Supports independent recomputation, error slicing, and future protocol migration

Complete-system success keeps every failure in the denominator

\operatorname{SuccessRate}=\frac{N_{\mathrm{task\ success}}}{N_{\mathrm{all\ eligible\ trials}}},\quad N_{\mathrm{all}}=N_{\mathrm{success}}+N_{\mathrm{wrong}}+N_{\mathrm{miss}}+N_{\mathrm{abstain}}+N_{\mathrm{timeout}}+N_{\mathrm{onboard\ fail}}

Report conditional pose accuracy alongside end-to-end coverage, without substituting it for coverage. Open-world evaluation also needs absent-target and false-positive trials.

Selective risk exposes the abstention trade-off

\operatorname{Coverage}(\tau)=\Pr(s\ge\tau),\qquad \operatorname{Risk}(\tau)=\mathbb E[\ell(\hat{\mathbf T},\mathbf T^\star)\mid s\ge\tau]

Plot complete risk–coverage curves and slice by unknown objects, material, visibility, and motion. Uncalibrated scores support ranking, not direct interpretation as success probabilities.

Claim card

A one-sentence claim must be falsifiable

Complete boundaries for inputs, priors, instance relationships, outputs, adaptation, budgets, real scenes, and metrics.

01No test-object, category, scene, or reference/pretraining leakage
02Recomputable canonical frames, scale, symmetry, and metric sources
03No detection / abstention / timeout / onboarding failure included in denominators
04Real sensors separated from sim-to-real results, with material/dynamic/occlusion slices
05Budgets covering capture, cold onboarding, caches, per-frame processing, and resets
06Coverage / calibration for uncertainty, beyond top-K visualizations
07Baselines sharing detectors/masks/data/compute/refinement and postprocessing
08At least one independent geometric measurement or downstream task outcome with no shared model origin
09Explicit unknown / absent / distractor and failure-recovery tests
10Traceable raw predictions, manifests, failed samples, and reproduction environments

10 · Open Questions & Research Navigation

Good research makes new capabilities definable, observable, and calibratable in real systems

These ten questions connect 3D vision, SLAM, robot grasping, embodied intelligence, and multimodal perception. They guide both research and critical reading of frontier papers.

Q01

Truly open object onboarding

How can stable object frames be established with few references, no CAD, unknown scale, and complex backgrounds, while providing evidence of failure?

Q02

Definability of canonicalization

Who defines category-semantic axes? For objects without a front/back, continuously varying shapes, or instance-specific symmetry, should outputs be task-equivalence classes?

Q03

Disentangling pose, shape, and appearance

How can camera motion, rigid pose, nonrigid deformation, material reflection, and generative hallucination be distinguished?

Q04

Calibrated multimodal uncertainty

How can symmetry, occlusion, and unknown-object modes be covered, and coverage translated into robot chance constraints?

Q05

Long-term, dynamic worlds

How do identity, part states, world maps, and covariance remain consistent through grasping, opening/closing, occlusion, and reappearance?

Q06

Net benefit of active perception

Does information gain offset viewpoint motion, collision, latency, energy, and scene disturbance? When should observation stop?

Q07

Cross-modal spatiotemporal calibration

How can event, RGB-D, tactile, F/T, and robot-state sensors be calibrated online, with quantified clock/extrinsic drift?

Q08

Real materials and extreme conditions

Can transparent, specular, dark, tiny, flexible, fast-moving, and heavily occluded objects be handled reliably with real sensors?

Q09

Geometric integrity of foundation models

Can pretrained features / reconstruction / generation produce coordinate-consistent, metric, reproducible outputs with the ability to abstain?

Q10

From benchmarks to safe manipulation

How can public evaluation cover task costs, tail risk, abstention, recovery, and hardware incidents alongside mean scores?