Deep Reading Notes: RGBTrack

1. Key assessment

RGBTrack’s premise is that known physical CAD size lets projected image size constrain camera-object depth. This initializes metric translation, while later local FoundationPose updates can discard unreliable depth and use RGB alone.

XMem, Kalman filtering, and a state machine prevent continued refinement from incorrect previous poses after occlusion / fast motion. This is a low-training-cost system adaptation rather than a new backbone.

2. Conditions for depth-free operation

RGB-only mode still requires:

• Known object identity;
• Accurate CAD geometry and metric scale;
• First-frame mask;
• Intrinsics;
• A search interval $[z_{\min},z_{\max}]$ containing the true depth;
• Pretrained FoundationPose RefineNet / ScoreNet;
• A subsequent 2D mask tracker.

It is neither model-free nor arbitrary monocular scale recovery. No new-CAD retraining comes from FoundationPose’s object generalization; metric translation is constrained by known physical CAD dimensions.

3. Initial RGB registration

FoundationPose originally uses median depth within the mask to obtain

$$z_c=\operatorname{median}(\text{depth}[\text{mask}]),$$

then recovers translation from mask center $(u_c,v_c)$ and intrinsics:

$$x_c=\frac{z(u_c-c_x)}{f_x}, \qquad y_c=\frac{z(v_c-c_y)}{f_y}.$$

RGBTrack searches unknown $z$ within a preset interval. Each iteration:

1. $z_r=(z_{\min}+z_{\max})/2$;
2. Recover $x_r,y_r$ from the mask center;
3. Sample rotation hypotheses at that translation;
4. Update poses with FoundationPose RefineNet;
5. Select the best with ScoreNet;
6. Render its CAD mask;
7. Compare rendered and observed areas.

A larger rendered mask suggests the CAD hypothesis is too near, so set $z_{\min}\leftarrow z_r$; otherwise set $z_{\max}\leftarrow z_r$. Stop when consecutive refiner depth changes are below $10^{-2}$ or interval width falls below $10^{-2}$.

This is not strictly one-dimensional geometric bisection: each iteration’s neural refiner changes rotation and translation, so area need not remain monotonic in the midpoint. Occlusion, truncation, mask errors, and incorrect CAD geometry violate an area-difference-from-depth-only assumption.

4. Why not reuse previous rendered depth?

Rendering CAD depth from the previous pose for the current FoundationPose input seems natural, but experiments show accumulated drift: pose errors produce incorrect depth that biases later refinement.

Instead, the last three point-cloud channels are zeroed. RefineNet separately encodes RGB and depth-derived point clouds before weighted fusion; zeroing leaves RGB features and previous poses to drive relative updates.

This simple change reveals substantial local correction capacity in FoundationPose’s RGB branch, with costs:

• No independent metric depth correction per frame;
• Translation drift depends on RGB, mask geometry, and recovery;
• Large inter-frame motion can exceed the refiner’s basin.

5. Tracking-loss state machine

5.1 With depth

Either condition triggers loss:

• Tracker mask pixel count below threshold;
• Euclidean separation between mask-depth coarse 3D center and FoundationPose pose center above $\theta$.

Kalman predicts translation and rotation; instead of full 252-hypothesis registration, sample 20 rotations near the prediction and refine / score them. Resume tracking when the pose agrees with predicted / coarse estimates.

5.2 Without depth

Two image-space heuristics replace 3D center comparison:

$$\sqrt{(u-u_p)^2+(v-v_p)^2}>\theta_1,$$

The projected pose center deviates from the 2D tracker center; and

$$|\operatorname{Area}(M_{\text{obs}}) -\operatorname{Area}(M_{\text{render}})|>\theta_2.$$

After loss, Kalman roughly predicts $z$ and rotation and samples 20 nearby candidates. Sensitivity to $\theta,\theta_1,\theta_2$, mask-size thresholds, Kalman noise, and switching hysteresis is not fully analyzed, creating reproduction / transfer risks.

6. CAD scale recovery

With incorrect CAD scale, projected area cannot identify size and distance simultaneously. RGBTrack adds true first-frame depth and binary render-and-compare search for a scale factor.

• Metric first-frame depth recovers metric CAD scale, followed by RGB-only tracking;
• ZoeDepth / Metric3D alone align within their predicted relative / offset scales, without guaranteed metric accuracy;
• No scale signal leaves monocular RGB and unknown CAD scale unidentifiable from mask area alone.

Unknown-scale CAD with RGB-only metric tracking is therefore not fully solved physically; metric scale information is concentrated into initial depth.

7. Metrics and evaluation

The authors use the BOP 2019 error families:

• VSD: visible surface discrepancy;
• MSSD: maximum symmetry-aware surface distance;
• MSPD: maximum symmetry-aware projection distance;
• Recall averaged over thresholds for each, then averaged across errors, yields AR / table Rec.

Per-sequence mean translation errors in m and rotation errors in degrees include frames after complete loss, reflecting recovery but without separate initialization / steady-tracking errors.

8. Main YCBInEOAT results

Representative sequences:

Sequence	Method	Rec ↑	T.Err m ↓	R.Err ° ↓
mustard0	FoundationPose RGB-D	0.998	0.0031	3.20
mustard0	RGBTrack RGB	0.875	0.020	3.64
mustard_easy	FoundationPose RGB-D	0.979	0.0038	2.54
mustard_easy	RGBTrack RGB	0.883	0.016	3.45
cracker_box_reorient	FoundationPose RGB-D	0.984	0.0038	2.73
cracker_box_reorient	RGBTrack RGB	0.861	0.022	3.34

With good depth, RGBTrack translation clearly trails RGB-D FoundationPose while rotation is closer. Both severely fail on sugar_box_yalehand0: FoundationPose R.Err 176.85° versus RGBTrack 73.97°. RGB-only recovery cannot eliminate strong hand occlusion / symmetry ambiguity.

Last Depth translation errors are mostly 0.49–0.82 m and rotation errors 8–150°, supporting accumulated-depth drift. FoundPose recall stays below 0.01, but framewise initialization is not a matched temporal tracking system; coarse templates cannot simply replace tracking.

Qualitative bad / distant-depth examples favor RGBTrack, but no depth-noise / range-binned benchmark quantifies its sensor-failure advantage.

9. Recovery and speed

ClearPose set 8 scene 2, over all opaque objects and frames:

Method	Recall ↑	T.Err m ↓	R.Err ° ↓
FoundationPose	0.0442	0.598	130.36
RGBTrack + depth recovery	0.616	0.0129	67.795
RGBTrack RGB	0.4682	0.0620	25.806

RGB-only has the lowest rotation error, while depth-assisted recovery improves recall / translation; no variant wins all metrics. Recovery is quantified in only one ClearPose scene, with scarce public sequences combining CAD, full pose GT, and occlusion. This demonstrates recoverability rather than average cross-object success.

RTX 3090, 640×480, default refinement iterations:

• RGBTrack + XMem / Kalman / recovery: about 22 FPS;
• Original FoundationPose: about 45 FPS.

The roughly 2× speed cost comes from 2D tracking, status checks, and recovery candidates. Initial binary-search registration latency and worst-case recovery-frame latency are not reported.

10. Scale recovery results

Artificially enlarging each YCBInEOAT CAD by 3× yields zero Rec on all nine sequences without recovery. With first-frame depth + scale search:

• cracker_box_reorient: Rec 0.894, T.Err 0.0179 m, R.Err 3.353°;
• bleach_hard_00_03: 0.853 / 0.0169 m / 4.18°;
• mustard0: 0.726 / 0.0389 m / 3.60°.

Hand-occluded first frames still give sugar_box_yalehand0 Rec 0, cracker_box_yalehand0 only 0.003, and tomato_soup_can_yalehand0 0.299. Occluded area is mistaken for smaller / more distant objects during scale and depth search.

11. Open source and reproduction

The official repository is accessible and contains:

• run_demo_without_depth.py;
• run_demo_unkown_scale.py;
• run_demo_clearpose.py;
• XMem and SAM2 submodules / wrappers;
• FoundationPose-derived code and custom demo data.

README still marks parts of SAM2 / TensorRT as ToDo. Dependencies include FoundationPose, nvdiffrast, XMem, possible ZoeDepth / Metric3D, and several weight sets. Check:

• $z_{\min},z_{\max}$ and units;
• Sampling of 252 initial rotations;
• Bisection stopping versus post-RefineNet depth updates;
• $\theta,\theta_1,\theta_2$ and mask-area thresholds;
• Kalman states, noise, and 20-candidate distributions;
• Initial XMem masks and memory after occlusion;
• CAD coordinate, metric scale, BOP symmetry annotation;
• Whether 22 FPS includes mask inference and recovery frames.

12. Limitations

• Accurate-scale CAD is required for RGB-only metric poses;
• Unknown-scale recovery still needs first-frame metric depth;
• Mask area confounds depth, scale, occlusion, truncation, and segmentation;
• Binary search needs a depth range containing the truth;
• Zero-depth tracking lacks per-frame metric corrections;
• Recovery depends on the 2D tracker; wrong-object masks mislead the state machine;
• Recovery is quantified on only one ClearPose scene;
• Speed is about half FoundationPose’s;
• Overall accuracy still trails RGB-D FoundationPose with good depth.

13. Overall assessment

RGBTrack pragmatically reuses FoundationPose’s RGB branch with zero point-cloud input, initializes depth from metric CAD projected area, and adds recovery after occlusion. Public code, 22 FPS, and YCBInEOAT / ClearPose results make it an informative engineering reference.

Its scope is a depth-free, recovery-enabled FoundationPose tracker constrained by known-scale CAD and masks, not arbitrary RGB-only metric 6D pose. Unknown CAD scale retains first-frame depth dependence; unstable masks / trackers jointly undermine scale estimation and recovery.