Deep Reading Notes: DynamicPose

1. Key assessment

DynamicPose addresses three initialization failures of FoundationPose-style tracking under fast motion, rather than designing another temporal network that directly regresses pose from video:

• Camera motion shifts the ROI projected from the previous pose away from the target;
• Even after camera compensation, large object translation still causes ROI drift;
• A correct ROI is insufficient if large inter-frame rotation exceeds the local refiner’s convergence basin.

The authors address these with VIO, a depth-informed 2D tracker, and VIO-guided Kalman multiple hypotheses, respectively, then reuse the pretrained FoundationPose refiner to update and select candidates. The main value lies in connecting existing robot RGB-D / IMU / VIO with a foundation pose model in a feedback loop, rather than the novelty of an individual module.

2. Task boundaries and priors

This is CAD model-based, instance-level, unseen-object tracking. It assumes:

• The target’s accurate CAD model and identity;
• A continuous, calibrated, time-synchronized RGB-D stream;
• An onboard IMU and inter-frame camera motion from visual-inertial odometry;
• Camera intrinsics;
• A first-frame object mask / initial pose.

“Unseen / retraining-free” only means that FoundationPose’s pretrained refiner and scorer need not be retrained for each new CAD model. It does not imply category-level estimation, a model-free setting, or automatic object discovery in arbitrary video. The problem definition explicitly requires an initial pose; the implementation can obtain it with a SAM2 mask, median depth, and FoundationPose global candidate search.

3. Why the previous pose fails under fast motion

FoundationPose tracking typically uses the previous pose as the current coarse pose, projects the CAD model to obtain an ROI, and predicts $\Delta t,\Delta R$ with a local refiner. This implicitly requires:

1. The current target to remain near the previous projection;
2. The current rotation to stay within the refiner’s local correction range.

Sudden camera or object motion first breaks the ROI assumption, leaving insufficient target information in the refiner’s crop. Even with an adequate ROI, large rotation can create excessive mismatch between the rendered object and observation. An incorrect pose then produces a worse ROI in the next frame, causing cascading drift. DynamicPose’s feedback loop aims to interrupt this chain.

4. First-frame initialization and the FoundationPose refiner

Given a SAM2 object mask in the first frame:

1. Initialize translation with median depth inside the mask and crop RGB-D;
2. Uniformly sample $N_s$ viewpoints on an icosahedron centered on the CAD model;
3. Sample $N_i$ in-plane rotations per viewpoint to form $N_sN_i$ poses;
4. Iteratively refine them with FoundationPose;
5. Use the scorer to select the candidate that best matches the current image.

Tracking reuses only the refiner: the CAD rendering at the coarse pose and the current RGB-D crop pass through a shared encoder, residual convolutions, and a transformer to produce translation and rotation updates. DynamicPose’s central task is to provide translation, ROI, and rotation candidates sufficiently close to ground truth.

5. Translation and ROI compensation

5.1 Camera-motion compensation

Let the previous object pose relative to the previous camera be $T^{c(t-1)}_{o(t-1)}$, and let VIO provide the transform from the previous camera to the current camera, $T^{c(t)}_{c(t-1)}$. Temporarily assuming no object translation, transform the previous object translation into the current camera frame to remove the ROI shift caused purely by camera motion.

This handles a change in the camera coordinate frame, rather than predicting independent object motion. Significant object translation therefore leaves a residual after VIO compensation alone.

5.2 Depth-informed 2D tracking

The 2D tracker obtains the current RGB bounding box from its previous location. Take its center $(u,v)$ and the median depth $z$ of the bounding box / target region, then use

$$\mathbf t_o^{c(t)} = \mathbf K^{-1}[u,v,1]^\top z$$

to back-project a current 3D translation hypothesis. This combines the large-displacement tolerance of 2D appearance tracking with metric depth. The paper treats the specific tracker as replaceable; the public repository, as recorded in the source notes, integrated MixFormerV2 and additional frontends such as Grounding DINO.

This is not a precise 3D object-motion estimate: the bounding-box center need not correspond to the object-frame origin, and median depth may lie on the front surface, an occluder, or an incorrect region. Its role is to bring coarse translation back into the refiner’s convergence basin.

6. World-frame anchoring and Kalman rotation prediction

Estimating object angular velocity directly in a moving camera frame mixes in camera rotation. The authors use VIO’s world–camera rotation to anchor object rotation in world coordinates:

$$\mathbf R_o^w(t)=\mathbf R_c^w(t)\mathbf R_o^{c(t)}(t).$$

Rotation is then represented by Tait–Bryan angles

$$\boldsymbol\theta=[\theta_{\text{roll}},\theta_{\text{pitch}},\theta_{\text{yaw}}]$$

and predicted using constant angular velocity:

$$\boldsymbol\omega(t) = \frac{\boldsymbol\theta(t)-\boldsymbol\theta(t-1)}{\Delta t}, \qquad \boldsymbol\theta(t+1) = \boldsymbol\theta(t)+\boldsymbol\omega(t)\Delta t$$

The Kalman process noise is set to

$$\mathbf Q=\|\dot{\boldsymbol\omega}(t)\|\mathbf I,$$

so greater angular acceleration yields higher predicted covariance. The refiner’s final rotation serves as the observation for the filter update; measurement covariance is fixed at 0.05 in the experiments.

This model is lightweight, but Tait–Bryan angles are not a globally nonsingular state representation of $SO(3)$, and independent Gaussian axes ignore coupling between axes. It is suitable for generating coarse candidates and should not be interpreted as rigorous manifold filtering.

7. Variance-aware multiple hypotheses

For each axis $i\in\{\text{roll,pitch,yaw}\}$, if the predicted standard deviation is below $\sigma_{\mathrm{th}}$, sample only the mean $\mu_i$; otherwise sample

$$\mu_i-\sigma_i,\quad \mu_i,\quad \mu_i+\sigma_i.$$

This yields:

• All three axes stable: 1 candidate;
• One unstable axis: 3;
• Two unstable axes: 9;
• All three axes unstable: up to 27.

This costs less than fixed dense rotation sampling and is less vulnerable to sudden direction changes than using only the Kalman mean. However, candidates extend only one standard deviation from the mean; the threshold, covariance calibration, and angle periodicity can affect coverage. The paper does not systematically report candidate count–accuracy–latency curves.

8. Hierarchical refinement and selection criterion

At level $i$, the FoundationPose refiner outputs the following for each of $K_i$ candidates:

$$\mathbf t_{i+1}=\mathbf t_i+\Delta\mathbf t_i, \qquad \mathbf R_{i+1}=\Delta\mathbf R_i\otimes\mathbf R_i.$$

The authors use the $L_2$ norm $\|\Delta R_i\|_2$ of the refiner’s rotation update as a proximity proxy: a candidate close to the observation should theoretically require only a small update. Each level retains the top-$K_{i+1}$ candidates with the smallest updates until $K_N=1$.

No ground truth is used here; “smaller update is better” is a heuristic. Selection can fail if the refiner saturates at an incorrect candidate and predicts a small update, or makes small updates for near-symmetric poses. The main experiments support this strategy on the tested motion distribution, but do not separately compare a learned scorer, rendering residual, or different numbers of levels.

The final pose has two feedback paths:

• Project the CAD model to obtain an RGB region and update the 2D tracker’s target template;
• Transform it into world coordinates as a Kalman measurement to update the next rotation prediction.

9. Simulation protocol and metrics

Existing benchmarks lack synchronized camera poses / IMU, so the authors do not directly compare on the original YCB-Video sequences. Instead, they place 13 YCB-Video objects in Isaac Sim.

• Main test: run each method with the same prescribed object and camera trajectories;
• Relative linear velocity $>1.5$ m/s;
• Relative angular velocity $>3.0$ rad/s;
• baselines: FoundationPose, SRT3D;
• Use ADD-S for symmetric objects and additionally report ADD for other objects;
• Integrate accuracy curves up to a 0.1 m threshold; tables report AUC.

ADD averages the distance between corresponding CAD points transformed by ground-truth and predicted poses. ADD-S finds the nearest predicted point for each ground-truth point to reduce symmetric correspondence ambiguity. These AUC values cannot be directly compared with results using 0.1d, BOP AR, or the real YCB-V protocol.

10. Main results

Method	Mean ADD-S AUC	Mean ADD AUC
FoundationPose	17.1	16.0
SRT3D	19.7	6.7
DynamicPose	88.9	75.8

DynamicPose achieves ADD-S AUC of 86.5–92.4 and ADD of 69.0–84.2 across all 13 objects. Compared with FoundationPose, mean gains are ADD-S +71.8 and ADD +59.8; compared with SRT3D, gains are +69.2 and +69.1 absolute points, respectively.

These gaps mainly show that “previous pose + one local update” can fail through cascading errors under the authors’ fast-motion conditions, while additional localization and multiple hypotheses preserve tracking. They do not independently establish higher static accuracy, because the experiment deliberately targets a failure regime of the baselines.

11. Qualitative comparison, ablations, and speed

The ablation uses harder relative motion: linear velocity $>3$ m/s and angular velocity $>8$ rad/s.

• W/o translation compensation: remove the depth-informed 2D tracker;
• W/o rotation estimation: remove VIO-guided Kalman candidate generation;
• Ours: the full system.

Reading the 0.1 m endpoints in Figure 7 gives approximate ADD accuracy of 0.72 for the full system, 0.23 without rotation estimation, and 0.04 without translation compensation. The paper provides no exact numerical table for these curves, so these are visual approximations only. Translation compensation matters most on this trajectory, which does not establish the same ranking in every dynamic scene.

Mean speed is approximately 10 Hz on an Intel i7-13700F and RTX 3060. The paper gives no timing breakdown for VIO, the 2D tracker, candidate count, or rendering / refinement, and no worst-frame latency. “Real-time” here means approximately 100 ms/frame on the authors’ platform, rather than high-frequency control.

12. Real-world evidence

In the real experiment, the target is rigidly attached to a moving robot arm’s end effector and follows combined translation / rotation trajectories. An experimenter manually moves the RGB-D camera to introduce camera motion. Figure 1 shows pose overlays and third-person views.

This demonstrates operation with real sensors, VIO, and robot communication, but the evidence remains qualitative:

• No pose ground truth;
• No ADD, drift, success rate, or count of tracking losses;
• No quantitative baseline comparison on the same trajectories;
• No measured motion-speed distribution;
• The target always remains within the camera’s field of view.

The strongest accuracy evidence therefore comes from simulation; the real experiment primarily supports system feasibility.

13. Failure modes and limitations

The authors explicitly identify three limitations:

1. Rapid camera / object motion causes RGB motion blur, removing texture and degrading refinement accuracy;
2. Median depth within the bounding box / mask can differ from the true object origin, systematically biasing the translation hypothesis;
3. Depth measurement becomes inaccurate beyond 3 m, substantially degrading translation compensation.

Additional considerations:

• VIO drift or time-synchronization errors affect both the ROI and world-frame rotation;
• The feedback loop provides no guaranteed global redetection after prolonged full occlusion or departure from the field of view;
• Tait–Bryan angles with independent Gaussian axes are an imperfect rotation-statistics model for large angles and singular regions;
• $\|\Delta R\|_2$ is not calibrated pose confidence;
• Up to 27 candidates increase refiner computation under extreme uncertainty;
• Accurate CAD, reliable depth, and a first-frame mask / pose remain strong dependencies;
• Coverage of real robot conditions by simulation trajectories, materials, motion blur, and VIO noise is not quantified.

14. Code and reproduction

At the time recorded in the source notes, the official GitHub repository exposed part of the code and stated an initial release date of 2025-09-30. Visible dependencies included:

• The FoundationPose refiner / scorer and their weights;
• MixFormerV2 2D tracker;
• Grounding DINO;
• SAM2-related components;
• VINS-Mono VIO;
• ROS / catkin, RGB-D–IMU sensor calibration, and demo bags.

The source notes record GPL-3.0 licensing. The main reproduction challenges concern integration rather than training:

• RGB, depth, and IMU timing synchronization and extrinsics;
• VINS-Mono frame conventions and object / camera transforms;
• Bounding-box centers and valid-pixel selection for median depth;
• $\sigma_{\mathrm{th}}$, initial Kalman covariance, and process / measurement noise;
• Candidate pruning schedule $K_i$ and refiner iteration counts;
• FoundationPose weights, CAD scale, and rendering coordinates;
• Tracking-loss detection and reinitialization strategies.

A partial code release does not establish one-command reproduction of all paper experiments. In particular, the Isaac Sim trajectories and complete evaluation scripts need checking against the relevant repository version.

15. Overall assessment

DynamicPose is a systems-oriented study with clear failure analysis. It uses VIO to separate camera motion, 2D tracking plus depth to recover translation, and covariance-aware candidates to enlarge the rotation basin, then places an established refiner in a feedback loop. The main simulation table over 13 objects and the aggressive-motion ablations strongly support this design.

Two judgments should be retained together: the method substantially improves tracking robustness under the authors’ fast-motion conditions, while that conclusion depends on simulation, accurate CAD, RGB-D–IMU, and reliable VIO; real-world accuracy has not been quantified. For fast tracking of known objects on mobile robots, it is a useful example of geometry / state estimation first bringing inputs back into a foundation refiner’s convergence basin.