Deep Reading Notes: TwinTrack

1. Key assessment

• Paper: arXiv 2505.22882
• Project: TwinTrack
• Core problem: unknown objects abruptly change velocity during collision, bouncing, or in-hand manipulation while becoming occluded by the environment or fingers; visual trackers lose observations precisely when state updates matter most.
• Core answer: reconstruct object geometry from visual history, calibrate collision geometry and physical parameters using real contact trajectories, then fuse one-step contact simulation with visual matches as a second pose cue according to confidence.
• My assessment: the contribution is the asynchronous loop connecting asset reconstruction, system identification, real-time visual tracking, and contact simulation. Results support one-step / short-horizon prediction, not long rollouts without vision; identified parameters should not be read as precise physical measurements.

2. Problem setting and assumptions

Input is a monocular RGB-D sequence $(I_k,D_k)$, plus proprioception when a robot is present. The test object is unknown and has no CAD; SAM2 needs a first-frame prompt. Output is per-frame 6D pose $q_k$ in camera coordinates.

Implicit deployment conditions include:

• One rigid object; walls, floors, or robot geometry can be represented in the simulator;
• Sufficiently accurate first-frame visual state, since wrong poses contaminate reconstruction, collision geometry, and later dynamics;
• Contact excitation rich enough for learning; otherwise friction, mass, and inertia are unobservable;
• Available depth, camera intrinsics, and robot–camera calibration;
• A slow backend that can run asynchronously, while the online frontend performs only one simulation step.

“Unknown object” thus means no object CAD, rather than no environment model, segmenter, or physics engine.

3. Overall architecture: a loop across two time scales

Sim2Real outputs poses each frame and maintains a sparse keyframe buffer and trajectory history. Real2Sim reads them asynchronously to update:

1. Keyframes with consistent poses;
2. Object visual geometry;
3. A collision-query SDF;
4. Physical parameters including mass, inertia, and friction.

The updated model returns to the online frontend. Initially, the physics model is unreliable, so vision dominates; accumulated contact data gradually allows physics to take over during occlusion. Expensive GS / SDF / parameter identification stays outside the per-frame path.

4. Real2Sim: from visual resemblance to correct contact

4.1 Joint visual-geometry and keyframe-pose optimization

Object appearance uses Gaussian Splats $G_{gs}$. For keyframe set $\mathcal K$, the authors jointly optimize GS and each keyframe pose:

$$\min_{\{q_i\},G_{gs}}\sum_{i\in\mathcal K} \lambda_{\mathrm{pho}}E_i^{\mathrm{pho}}+ (1-\lambda_{\mathrm{pho}})E_i^{\mathrm{geo}},$$

The photometric loss compares rendered and observed RGB, while the geometric loss compares rendered and observed depth. Optimized keyframe poses anchor online visual matching; ray-casting samples of GS-rendered depth train

$$\phi=\mathrm{SDF}_{vision}(p).$$

This division of representations matters: GS suits multiview appearance / depth optimization, while a neural SDF suits collision-distance queries. One representation is not forced to handle both rendering and physics.

4.2 Contact-geometry residual

Visual reconstruction may have missing or swollen regions under occlusion or poor depth; these may matter little for rendering but change collision timing. The authors define

$$\mathrm{SDF}_{collision}(p) =\mathrm{SDF}_{vision}(p)-\delta(p).$$

The implementation restricts $\delta(p)$ to a constant $\delta$, uniformly expanding or shrinking the object. Expressiveness is limited, but sparse intermittent contact cannot reliably fit a spatial neural residual; a scalar is easier to identify and corrects common global thickness bias.

Customized MJX samples points on floors, walls, and finger surfaces and queries the neural SDF for collision detection. JAX parallelizes candidate physical parameters and trajectory simulations on the GPU.

4.3 Physical-parameter identification

Parameters to estimate are

$$\theta=\{m_{\mathrm{obj}},\operatorname{diag}(I_x,I_y,I_z),\mu,\delta\}.$$

The objective is one-step prediction error over Sim2Real trajectory history:

$$L(\theta)=\sum_{(q_k,v_k,q_{k+1})\in D_{\mathrm{tracking}}} \|\operatorname{sim}_\theta(q_k,v_k)-q_{k+1}\|_W^2.$$

Contact dynamics are nonsmooth, so local MJX gradients do not provide a stable global-search direction across contact modes. A CEM variant instead samples parameters in parallel each round and updates the sampling mean using softmax loss weights rather than only averaging elites, resembling MPPI.

The objective is predictive equivalence, rather than textbook recovery of true system parameters. Learned masses in Table II still differ substantially from real masses: a free-fall Spam can is estimated at $15\pm5$ g versus a true 28 g. Possible reasons include:

• Some trajectories insufficiently excite friction / inertia;
• MJX point / rigid contact differs from real contact, causing model bias;
• Mass and friction may compensate for one another;
• LEAP simulation and hardware control parameters differ.

A parameter combination that fits one-step motion may still help tracking, but does not establish accurate material-property measurement.

5. Sim2Real: adaptive visual–physics fusion

5.1 Visual branch

A first-frame prompt initializes SAM2. Between the current frame and the nearest optimized keyframe:

1. SuperPoint extracts object-region features, lifted to 3D through depth;
2. LightGlue matches them;
3. RANSAC rejects incorrect correspondences and estimates relative pose;
4. A view enters the buffer when its rotation distance from existing keyframes exceeds a threshold.

Matching a keyframe rather than always the preceding frame reduces accumulated drift; Real2Sim also optimizes these keyframe poses retrospectively.

5.2 Fusion objective

The final pose solves

$$\min_{q_k}\;w_kL_{\mathrm{vision}}(q_k) +(1-w_k)L_{\mathrm{dynamics}}(q_k)$$

The visual term measures distances between matched 3D points in the current frame and keyframe; the physical term is

$$L_{\mathrm{dynamics}}(q_k)= \|q_k-\operatorname{sim}(q_{k-1},v_{k-1})\|_2^2.$$

A sigmoid-like function of reliable matches after RANSAC determines $w_k$:

• Many features suggest less occlusion / blur, giving vision more weight;
• Few features allow one-step contact prediction to fill missing observations;
• Neither is a hard switch, avoiding sudden jumps at confidence boundaries.

This better suits collisions than a constant-velocity prior: collisions create velocity discontinuities, while the simulator explicitly computes impulses and contact constraints.

6. Real2Sim experiments: does geometry correction help?

Table I defines one-step prediction error as $100\times$ position error plus quaternion error. Across 8 object–scenario combinations:

Condition	snack	tissue	bottle	spam	duck	mustard	LEAP spam	LEAP duck
Random parameters	18.78	9.57	9.82	7.24	5.13	10.18	4.18	23.03
Learned properties, no $\delta$	0.29	0.47	0.87	1.16	0.38	0.56	0.18	0.38
Learned properties and $\delta$	0.20	0.22	0.70	0.43	0.21	0.15	0.13	0.28

The uniform geometry residual further lowers errors for every combination, showing that millimeter-scale visual-geometry bias can amplify contact-prediction error. Figure 7’s appealing multistep rollout does not establish long-term predictability: the authors explicitly note contact dynamics’ sensitivity to initial states and surface detail. The method suits one / few steps with continual corrections from new visual observations.

7. Main tracking results and evaluation caveats

Method	Mean ADD ↓	Mean ADD-S ↓	Meaning
visual baseline	3.01 ± 1.86 cm	1.15 ± 0.40 cm	FoundationPose with the same keyframes
w/o dynamics	7.21 ± 5.76	2.98 ± 3.78	Vision + constant-velocity prior
random physics	>50	>50	Unaligned physics catastrophically misleads tracking
learned physics, w/o $\delta$	1.56 ± 0.91	0.74 ± 0.38	Physical properties help, but collision geometry remains biased
Full TwinTrack	1.23 ± 0.65	0.59 ± 0.31	Physical properties + geometry residual

Three points must accompany these results:

1. “w/o dynamics” is worse than the pure FoundationPose baseline, showing that a constant-velocity model unsuitable for collisions can pull estimates away.
2. Random physics frequently diverges: adaptive fusion cannot replace simulator alignment.
3. Ground truth is pseudo-ground truth generated by FoundationPose using high-quality textured meshes obtained offline with Scaniverse, rather than external motion capture; results may share biases with related visual estimators.

8. Real-time performance

Per-frame breakdown on RTX 4090 / Ryzen 5955WX:

Module	Time
SAM2	12 ms
SuperPoint	8 ms
keyframe selection + LightGlue	15 ms
One-step neural-SDF contact prediction	5 ms
adaptive fusion	5 ms
Other	3 ms

The total is about 48 ms, exceeding 20 Hz. This covers the online frontend after a usable Real2Sim model exists; asynchronous GS reconstruction and CEM identification are not amortized into per-frame latency, and wall-clock time from first observation to a reliable physics model is not reported.

9. Limitations and reproduction checklist

1. Cascading initialization dependence: wrong initial poses bias GS, SDF, contacts, and physical properties together.
2. Drift during prolonged loss of vision: repeatedly rolling out the one-step model without vision accumulates model bias.
3. Overly global geometry residual: scalar $\delta$ cannot correct a local bump or indentation confined to one surface.
4. One rigid object: current support excludes multiple colliding objects and deformable objects.
5. Observability: mass, friction, and inertia require excitation involving sliding, collisions, and rotation; slow planar motion is insufficient.
6. Evaluation ground truth: add motion capture or independent marker-based GT during reproduction to avoid shared errors from FoundationPose pseudo-ground truth.
7. Uncertainty propagation: $w_k$ currently depends mainly on match count; a fuller version should estimate visual covariance, a physical-parameter posterior, and contact-mode uncertainty together.
8. Open-source status: the project provides a paper, videos, and demonstrations, but no complete code repository was found when this site compiled the notes, so it is not marked open source.

10. Transferable insights

TwinTrack suggests an engineering principle for 6D pose systems: treat a physics model as a sensor calibrated against real trajectories and entering fusion with confidence, rather than as a permanently correct hard constraint. Vision corrects states and initializes geometry; contact supplies motion evidence when vision is weakest. The useful mechanism is their feedback loop around errors and time scales, beyond adding a physics loss to network training.