Paper Reading Notes (Deep): Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects Via 2D Point Trackers

1. Basic information

• Abbreviation: Point2Pose
• Authors: Tzu-Yuan Lin, Ho Jae Lee, Kevin Doherty, Yonghyeon Lee, Sangbae Kim
• Affiliations: MIT Biomimetic Robotics Lab / MIT; Kevin Doherty lists Boston Dynamics, with work independently conducted in personal time
• Year / venue: ECCV 2026; full-text evidence: arXiv:2604.10415 v1
• Paper / PDF: arXiv / PDF
• Video: YouTube
• Code: promised for future release; the paper page currently links only paper / video, so this site does not label it open-source
• Reading mode: Deep Dive
• Evidence scope: values / figures follow the 2026-04-12 preprint; code and camera-ready versions may change later.

2. Positioning in one sentence

Long-range tracks replace framewise matching, continuously lifting RGB-D into object-specific maps. Sequential RANSAC proposes poses, TSDF supplies independent dense selection / refinement, and GTSAM optimizes keyframe consistency. Multiple CAD-free rigid objects are tracked / reconstructed, with initial-frame-relative recovery after complete occlusion.

2.1 Guide to the paper’s figures and tables

Figure / table	Evidence	Interpretation
Figure 2	Flow / update frequencies	Tracker: persistent associations; TSDF: reconstruction and scoring surface
Figure 4	Texture failures	Low texture raises uncertainty, jitter, and ADD error
Figure 5	Complete occlusion	Distinguish invisible-period output from correct recovery
Table 3	Multiple objects	Serial per-object BundleSDF still fails; advantages center on new benchmarks
Tables 4–5	System / sampling	Hypotheses matter most; confidence plus coverage beats either alone

3. Problem setting and task boundaries

3.1 Coordinate frames

Assume fixed camera $C$; for object $i$ define rigid frame $O_i$, initializing:

$$T^{C}_{O_i,0}=I_{4\times 4}.$$

Estimate $T^{C}_{O_i,t}$ thereafter: motion from the first frame rather than CAD / semantic / NOCS canonical orientation.

Thus:

• Recover rotation / translation since initialization;
• Do not independently recover pose relative to semantic canonical models;
• ADD / ADD-S transform through initial-frame / GT-mesh alignment; meshes are not operational inputs.

3.2 Inputs and outputs

Item	Setting
Stream	Causal video from one calibrated RGB-D camera
Initialization	User points per object, SAM2 masks
Foundation models	TAP / CoTracker-style long-range tracker, SuperPoint, SAM2
Priors	No CAD, category, or trained target instances
Outputs	Per-object relative trajectories, maps, TSDF / meshes
Objects	Multiple rigid instances with stable appearance / geometry
Camera	Fixed to world; no joint moving-camera / object decomposition

3.3 Causality and full occlusion

• Causal: online video processing without future-frame batch optimization; keyframes update graphs / TSDF.
• Recovery: invisible objects provide no new 3D observations. Reappearing query identities restore old-map correspondences beyond last-pose local refinement.
• Not invisible-motion prediction: evidence concerns recovery after reappearance, not true trajectories during occlusion.

3.4 Fairness

BundleSDF is single-object, so each target is run serially. Exclusive memory / compute helps that baseline without upgrading relocalization. Capability is compared, without equal-total-latency / memory curves.

4. Research context

4.1 model-based tracking

Known-CAD FoundationPose can refine locally and use detectors / relocalizers, but CAD, rendering, and instance libraries constrain open-world onboarding.

4.2 model-free tracking + reconstruction

• BundleTrack: feature matches, BA, and surfel / mesh maps; interrupted adjacent chains drift.
• BundleSDF: neural SDF reconstruction / relocalization; strong fidelity, mainly single-object and history-initialized after full occlusion.
• Point2Pose: delegate persistence to 2D trackers, retaining SVD, TSDF, and graphs for multi-object recovery rather than maximum neural reconstruction fidelity.

4.3 Why point tracks support poses

With depth, persistent identities associate map point $p_n$ with observation $\tilde p_n$ without descriptor retrieval. Occlusion sets visibility $\delta_t=0$ without deleting identity.

This moves the bottleneck to point tracking: weak / repeated texture and blur corrupt 3D correspondences.

5. Research motivation

Adjacent tracking assumes a nearby prior pose, broken by disappearance / crossings. RANSAC alone is insufficient when many wrong static body tracks outvote few correct handle tracks.

Three complementary levels:

1. Long-range identity: retain pre- / post-occlusion matches;
2. Multiple hypotheses: preserve correct minority consensuses;
3. Dense TSDF validation: judge candidates with surfaces rather than the same sparse tracks.

This is the central systems insight.

6. Method details

6.1 2D point tracker

For query $q=(t_q,u_q)$, the tracker outputs:

• Pixel $u_t\in\mathbb R^2$;
• Visibility $\delta_t\in\{0,1\}$;
• Uncertainty $\sigma_t\in[0,1]$.

Separate object sets $Q_i$ aggregate into one tracker pass, with cost mainly following point count rather than repeating the image backbone per object.

TAP / CoTracker interchangeability makes this a framework whose performance depends on tracker versions. Exact checkpoints / causal mode need confirmation when code is released.

6.2 Trackability and coverage

SuperPoint masked candidates $(p_k,s_k)$ are greedily selected by:

$$J_k=\lambda s_k+(1-\lambda)\min\left(\frac{d_k}{r_{ideal}},1\right)-\beta\rho(d_k),$$

$s_k$ is confidence; $d_k$ the nearest selected / mapped distance; $r_{ideal}$ controls ideal spacing; $\rho$ penalizes crowding below $r_{min}$.

These address:

• Uniform sampling in untrackable textureless regions;
• High-score clustering with degenerate 3D registration;
• Joint trackability and spread.

6.3 pending keypoint verification

Back-project new points with current pose / depth, but keep them pending. Each verified observation adds +1; promote only after a window threshold.

Immediate insertion of bad-pose / depth-edge points can self-reinforce map pollution and later registration errors.

6.4 frame-to-map SVD

Depth lifts visible tracks into camera observations $\tilde p_n$ with map counterparts $p_n$. Solve:

$$T^C_{O_{i,t}}=\arg\min_{T\in SE(3)}\sum_{n=1}^{N}\|\tilde p_n-Tp_n\|^2,$$

SVD gives an efficient closed form when most matches are correct; outlier structure is the real difficulty.

6.5 sequential multi-hypothesis registration

Ordinary RANSAC retains only the largest consensus. Repeat:

1. RANSAC / SVD for current consensus;
2. Save pose;
3. Remove consensus;
4. Continue on remaining matches.

Correct handle tracks can form a second hypothesis even when wrong symmetric-body tracks dominate, resembling greedy motion segmentation.

6.6 TSDF selection and refinement

Transform masked dense depth into each candidate object frame and sum absolute TSDF values. The lowest score best fits the reconstructed surface.

Starting at $T_0$, optimize increment $D\in SE(3)$:

$$D^*=\arg\min_D\sum_{p\in\mathcal P_{cur}}\rho_H\left(\Phi(T_i^{-1}D^{-1}p)^2\right),$$

Numerical TSDF gradients and LM separate sparse proposals from dense verification, preventing one bad track set from both proposing and deciding poses.

6.7 online factor graph

Equivalent fixed-object / moving-virtual-camera coordinates use inverse poses $X_m=T^O_C(m)$ for keyframes and $P$ for landmarks:

$$\mathcal L(X,P)=\mathcal L_{prior}+\mathcal L_{odom}+\mathcal L_{obs}.$$

• Prior: fix $X_0=I$ to remove gauges;
• Odometry: relative keyframe alignment with Huber robustness;
• Observation: bearing / range jointly constrain angular and radial landmark error.

New keyframes trigger GTSAM / LM pose / landmark optimization, improving map consistency rather than predicting occluded dynamics.

6.8 TSDF reconstruction

Optimized poses fuse segmented RGB-D into object voxels. Truncation $\tau$ limits projective signed-distance updates; Marching Cubes extracts meshes and disconnected-noise components are filtered.

Classical TSDF is online, stable, and supplies usable surface gradients, trading detail / hole filling against neural BundleSDF.

6.9 Modules

Module	Input	Output	Role	Failure
SAM2 / masks	Clicks and RGB	Object masks	Bound points / depth	Leakage, merging, background
Long-range tracker	RGB and query sets	$u_t,\delta_t,\sigma_t$	Identity across occlusion	Low / repeated texture, blur
Sampling / pending	SuperPoint, masks, maps	Verified keypoints	Safe map expansion	Depth edges / wrong pose
Multiple-hypothesis SVD	Observations and map	Pose candidates	Preserve minority motion	All-wrong tracks / collinearity
TSDF score / refine	Depth, candidates, TSDF	Selected refined pose	Dense verification	Incomplete TSDF / depth
Factor graph	Keyframes, poses, landmarks	Optimized poses / points	Global consistency	Bad factors / nonrigidity
TSDF fusion	Masked RGB-D and pose	Mesh / SDF	Reconstruction and scoring	Reflection, transparency, masks

7. Experimental analysis

7.1 Datasets

Dataset	Sequences / objects	GT	Purpose
HO3D v3	13 sequences, 4 objects	Automatic multiframe registration	Heavy hand-occlusion comparisons
YCBInEOAT	9 sequences, 5 objects	Manual framewise 6D pose	Distant dual-arm manipulation / small targets
Synthetic YCBMultiTrack	7 YCB objects; 1 single + 3 two-object scenarios	Exact Isaac Lab poses / depth	Controlled motion / mutual occlusion
Real YCBMultiTrack	5 objects; 5 single + 4 two + 2 three-object scenarios	OptiTrack, AprilTag extrinsics, synchronization	Full occlusion, reentry, crossings

D435i captures real sequences with 4–5 mocap markers per object. More reliable than pseudo-labels, but markers may add trackable texture; that effect is unquantified.

7.2 Metrics

• ADD AUC: exact pose, preserving rotational / symmetric-axis error;
• ADD-S AUC: nearest-point equivalence may remain high with wrong orientation axes;
• Chamfer: reconstructed versus GT meshes, beyond tracking alone.

AP12 / bleach v1 show why both ADD and ADD-S matter: symmetric-axis errors can look plausible under ADD-S.

7.3 Ordinary single-object benchmarks

Dataset	Method	ADD-S AUC	ADD AUC	Chamfer	Assessment
HO3D	BundleSDF	93.34	92.62	0.58 m	Better exact pose / reconstruction
HO3D	Point2Pose	94.63	80.79	1.02 m	Slightly better symmetry tolerance, weaker exact pose / mesh
YCBInEOAT	BundleSDF	94.51	89.07	—	Tomato fails, excluded from mean
YCBInEOAT	Point2Pose	92.67	85.11	—	Slightly lower, all groups output

Competitive is appropriate; universal SOTA contradicts the tables. Value lies in recovering where full-occlusion / multi-object baselines fail, beyond highest continuously visible single-object means.

7.4 Multi-object results

• Synthetic mean: ADD-S +11.93 points, ADD +14.89 points.
• Real gains: ADD-S +27.32 and ADD +28.85 points, conservative since baseline failures are excluded.
• Not every object: synthetic pitcher ADD-S 92.60 < 94.53; airplane 88.46 / 55.07 < 96.02 / 92.21. Fewer catastrophic failures drive the overall gain.

7.5 Full-occlusion recovery

Recovery does not reveal invisible trajectories; it requires reliable tracks after reappearance and may fail on wholly different visible surfaces.

7.6 Texture failures

Correspondences remain essential: stronger persistent memory replaces framewise matching, while insufficient texture remains fundamental.

7.7 Ablation studies

Ablation	ADD-S AUC	ADD AUC	ADD drop	Claim tested
Full	95.07	82.76	—	Complete system
No multiple hypotheses	89.81	65.52	-17.24	One dominant consensus is insufficient
No SDF refine	94.40	77.37	-5.39	Dense local alignment helps
No graph optimization	94.56	78.80	-3.96	Global keyframe / landmark refinement helps
SuperPoint only	93.09	77.61	-5.15	Trackability alone lacks geometric coverage
Uniform	91.47	72.31	-10.45	Uniform textureless points are unreliable

Ablations support complementarity, without isolating TSDF selection from refinement or uncertainty weighting.

7.8 Evidence assessment

Contribution	Evidence	Sufficiency	Site assessment
Full-occlusion recovery	Real YCBMultiTrack, Figure 5, failures	Fairly strong	Quantitative / qualitative agreement, needing more objects / duration strata
Multiple objects	2 / 3-object scenarios, aggregated queries	Broad	Functional, without FPS / memory curves
Multiple hypotheses	Table 4	Strong	ADD -17.24 points
Sampling	Table 5	Strong	Beats two natural baselines; uncertainty sampling remains open
Reconstruction	HO3D Chamfer	No superiority	Online, lower fidelity than BundleSDF
Competitive standard tests	HO3D, YCBInEOAT	Broad	Similar ADD-S, weaker ADD / Chamfer

8. Critical analysis

8.1 Strengths

1. Occlusion as association: direct design with replaceable trackers.
2. Structured outliers: preserve correct small consensuses rather than merely adding RANSAC.
3. Sparse / dense roles: tracks propose; independent TSDF judges.
4. Functional graphs / TSDF: global consistency versus surface scoring / reconstruction.
5. Useful new data: dynamic multi-object occlusion with mocap fills a gap.

8.2 Weaknesses

1. Fixed camera: narrower than egocentric / camera-motion motivation.
2. Depth / mask dependence: errors contaminate both maps and TSDF.
3. Weaker exact pose: HO3D ADD / Chamfer trail BundleSDF.
4. Unquantified scaling: per-object points, landmarks, volumes grow; memory is only discussed qualitatively.
5. Uncertainty outside optimization: $\sigma_t$ is not systematically mapped to weights / factor covariances in main equations.
6. Identity errors: similar crossings / long absences may swap track identities.

8.3 Assumptions

• Rigid targets without articulation / deformation;
• Fixed camera;
• Reliable initial clicks / masks;
• Separate masks / stable identities;
• Aligned RGB and depth;
• Enough initial geometry for maps / TSDF;
• Some old queried surfaces reappear;
• Markers do not substantially help tracking.

8.4 Practical use

Useful for fixed-camera tabletop interaction / assembly prototypes. Robot deployment also needs:

• Camera compensation / SLAM;
• Automatic prompts / detection;
• Joint point / instance identities;
• Depth / mask quality gates;
• Budgets for targets, tracks, and voxels;
• Occluded-motion priors / recovery confidence.

9. Extensions

9.1 Authors’ limitations

• Low / repeated texture;
• Background contamination from masks;
• Point / memory growth across objects;
• Lower TSDF fidelity, motivating online representations such as EfficientNeRF.

9.2 Site proposals

1. Uncertainty weighting: $\sigma_t$ informs SVD / RANSAC, TSDF scores, and factor covariances beyond visualization.
2. Track lifecycle: manage visibility, uncertainty, coverage, and identities within memory budgets.
3. Moving cameras: jointly estimate $T^W_C$ and $T^W_O$ with background motion / SLAM factors.
4. Reappearance checks: object embeddings / mask shapes verify recovered identities.
5. Hybrid TSDF: local neural residuals improve fidelity while retaining online stability.
6. Active recovery: move cameras / objects to expose known surfaces when confidence drops.

9.3 Additional tests

• Occlusion duration, out-of-view ratio, and recovery curves;
• FPS, GPU memory, tracks at 1→2→3→5→10 objects;
• GT / SAM2 masks plus erosion, dilation, and merging;
• RealSense depth dropout / quantization / reflective objects;
• moving camera + moving objects;
• Similar-instance crossings / ID switches;
• Voxel resolution versus pose, Chamfer, and runtime Pareto curves.

10. Implications for my research

• Abstraction: tracks are sparse recoverable coordinate memory, lifted by depth into 3D constraints.
• Modules: trackability / coverage, pending verification, consensus removal, TSDF scoring, keyframe–landmark graphs.
• Baselines: BundleTrack, BundleSDF, FoundationPose tracking, plus detector-based framewise relocalization to isolate persistence benefits.
• Directions: moving-camera multiple objects, uncertainty-aware point-to-pose, same-category identities, active recovery.
• Evaluation: report ADD and ADD-S together to reveal symmetric-axis errors.

11. Open questions

Conceptual questions

• Is fixed camera essential or convenient? Can known camera motion transform observations for reuse?
• How do initial object frames align to GT meshes per benchmark, and is extra GT introduced?

Mathematical / implementation questions

• How many sequential hypotheses, which removal thresholds, and when to stop?
• How are TSDF scores normalized by masks / valid depth?
• Does code use $\sigma_t$ for filtering / weights beyond main-text formulas?

Experimental questions

• What are sequence lengths, occlusion durations, and reentry distributions?
• How much does excluding failed baseline groups inflate means versus scoring failures as 0?
• Do mocap markers add high-contrast visual features?

Reproduction questions

• When will promised code, parameters, and data be complete?
• Which causal TAP / CoTracker checkpoints, query updates, and resolutions?
• Are per-object graphs / volumes parallel or serial, and what is full real-time performance?