Paper Reading Notes (Deep): OmniPose6D

1. Key assessment

Two related contributions should be evaluated separately:

1. Data / task: short-term model-free monocular relative-object pose in dynamic scenes, with 40K synthetic sequences.
2. Method: object-motion synthetic fine-tuning, uncertainty classification, and track selection atop CoTracker, followed by standard SfM / COLMAP.

It turns point tracking into selecting stable, geometrically informative object tracks for multi-view geometry. Outputs remain first-frame-relative and up to scale, evaluated after GT-based 7-DoF alignment; this differs from CAD-input metric-translation trackers.

2. What 6D pose means here

Given monocular window $\{I_t\}_{t=1}^{8}$ and a first-frame target mask, estimate per-frame rigid motion relative to the first. Besides object rigidity, implicit assumptions include:

• Mask propagation remains on the same object.
• Surface texture supports tracking.
• Sufficient window parallax / rotation initializes SfM.
• Relative object motion can be treated as virtual-camera motion around a static object.

Monocular SfM cannot recover absolute scale. Object motion, camera motion, or both appear as relative motion; the method does not separate object motion from camera ego-motion.

3. Dataset

3.1 Scale and objects

• 40,000 sequences;
• 100 frames each, approximately 4,000,000 total.
• 4,000+ scanned OmniObject3D meshes after Blender cleanup / decimation.
• 203 categories, including 21 validation categories.
• One foreground object per sequence.

Scanned geometry / texture is more realistic than manual ShapeNet CAD, but blank backgrounds bypass scene realism through crops / masks rather than full interior synthesis.

3.2 Four motion types

Each has 10,000 sequences:

1. Random Walk: fixed camera, random 3D object motion.
2. Circling Camera: varying heights / views around objects.
3. Noisy Trajectory: perturbed circular trajectories.
4. Real Dataset Trajectories: short HO3D training hand-object motions transferred to different meshes.

The fourth narrows synthetic-to-interaction motion gaps, using HO3D training trajectories. Evaluation uses the official held-out set, without directly using test poses, but represents domain-targeted synthetic pretraining rather than unrelated-domain generalization.

3.3 Labels and augmentation

BlenderProc outputs RGB, depth, poses, normals, and instance masks. GT depth / poses project first-frame grid points into other views; depth consistency / cycle-projection errors determine visibility.

Random perturbed black circles at mask boundaries mimic hand / object occlusion, with radius 0.1–0.5 of mask size. Color jitter, Gaussian / motion blur, shade, flips, and crop shifts / scales add noise; occlusions remain simpler than real continuous hands.

4. From masks to tracks

Track Anything propagates supplied / manual initial masks and crops to 512×512. Tracks / uncertainty can reject some boundary errors without perfect segmentation, but sustained wrong-object propagation lacks global geometric recovery.

Training:

• $N=8$ frames per sample.
• Adjacent samples are at most $K=4$ original frames apart.
• Initial mask grid size = 3.
• Randomly select 500 tracks for memory.
• Network resolution 384×512.
• CoTracker CNN downsamples 8× to 128D features.
• Alternating temporal / group attention refines 4 times.

Inference:

• 8-frame sliding windows.
• Grid size 3, up to 1,500 random tracks.
• Simple optical-motion thresholds select sufficiently moving keyframes.
• Distance-change threshold $t_{\mathrm{dis}}=2.0$ and scale-change threshold $t_{\mathrm{scale}}=1.3$.

Near-static frames violate SfM geometry and are filtered. The benchmark therefore favors windows with observable motion.

5. Uncertainty-aware refinement

The network jointly outputs:

• Per-point per-frame 2D positions $\hat P$.
• Visibility $v$.
• uncertainty level.

Pixel error $e_i$ is prediction-to-GT $L_2$ distance, divided into five bins:

$$[1,3,5,10,\infty]\ \text{pixels}.$$

Balanced cross-entropy trains uncertainty softmax. Total loss:

$$\mathcal L = \mathcal L_{\text{keypoint}} +5\mathcal L_{\text{visibility}} +5\mathcal L_{\text{uncertainty}}.$$

Keypoint losses weight four refinement updates with discount $\gamma=0.8$. Training uses AdamW, LR $5\times10^{-4}$, 1-cycle, 100 epochs, 5,000 samples per epoch, batch 32, and 16 V100 GPUs.

Inference rejects invisible points, ranks the most likely uncertainty class, and retains reliable fraction $R=0.95$. Removing only the worst 5% avoids starving SfM of correspondences. Appendix Figure 7 also favors ratios >0.8.

6. From tracks to poses

Restore selected keypoints to image coordinates and organize all view pairs with valid covisible points. Default COLMAP mapping performs:

• pairwise geometric estimation;
• camera / object-relative pose initialization;
• multi-view bundle adjustment.

Treating objects as local static worlds makes inverse virtual-camera motion represent relative object motion. The first frame is reference; poses remain defined only up to similarity.

Separating learned tracking from classical SfM is interpretable but inherits initialization, coplanarity, low-parallax, outlier, and window-connection constraints.

7. Evaluation: the main interpretation pitfall

KITTI-odometry-style metrics include:

• Translation error $t_{\mathrm{err}}$ (%).
• Rotation error $r_{\mathrm{err}}$ (deg/m).
• Absolute Trajectory Error (ATE);
• Relative Pose Error (RPE translation / rotation).

Each table value is average precision below a specified error threshold, not mean error; higher is better. Monocular predictions first undergo Umeyama 7-DoF similarity alignment:

• 3D rotation;
• 3D translation;
• 1D global scale.

Metric thresholds such as ATE@0.03 m and RPE@0.015 m therefore follow GT-assisted frame / scale alignment. They measure trajectory shape and relative consistency, not independently recovered real translation scale.

GT rows need not equal 1: GT-visible correspondences enter the same SfM and remain limited by occlusion, coplanarity, and solving. They are pipeline-specific reference upper bounds.

8. Main results

8.1 OmniPose6D validation

Five random sequences per each of 21 classes give 105 originals, split into 8-frame windows.

Method	ATE AP@0.03m	RPE trans AP@0.015m	RPE rot AP@2°
LoFTR	0.63	0.63	0.52
DROID-SLAM	0.50	0.57	0.36
TAPIR	0.62	0.64	0.32
CoTracker	0.64	0.69	0.52
Ours	0.74	0.76	0.71

Versus CoTracker + SfM, gains are +0.10, +0.07, and +0.19. Rotation improves most, matching the emphasis on out-of-plane motion.

8.2 HO3D v3

HO3D uses 4 objects and 13 videos longer than 1,000 frames. RGB-only input is downsampled 3×; initial annotations drive Track Anything.

Method	ATE AP@0.03m	RPE trans AP@0.015m	RPE rot AP@2°
LoFTR	0.46	0.47	0.43
DROID-SLAM	0.26	0.28	0.26
TAPIR	0.47	0.47	0.39
CoTracker	0.51	0.52	0.51
Ours	0.64	0.62	0.61

CoTracker gains are +0.13, +0.10, and +0.10. Synthetic data transfers to real interaction, with a remaining gap: RPE rot@2° falls from 0.71 to 0.61.

9. Ablations: data beyond uncertainty alone

HO3D with CoTracker weights:

Training source	ATE@0.03	RPE trans@0.015	RPE rot@2°
Original pretraining	0.52	0.54	0.50
OmniPose6D scratch	0.60	0.54	0.53
Pretraining + OmniPose6D fine-tuning	0.65	0.60	0.60

Synthetic dynamic-pose data is a major contributor complementary to TAP-Vid-Kubric pretraining.

uncertainty strategy:

Strategy	ATE@0.03	RPE trans@0.015	RPE rot@2°
No uncertainty	0.65	0.60	0.60
Remove by level	0.62	0.62	0.59
Retain 95% by ranking	0.64	0.62	0.61

Ranking slightly helps some RPE / rotation measures but lowers ATE@0.03 by 0.01 versus no uncertainty. Gains cannot all be attributed to uncertainty; fine-tuning, crops, refinement, and selection act together.

10. Additional real datasets

The appendix evaluates:

• YCBInEOAT: 5 YCB objects, 9 approximately 1,000-frame dual-arm robot videos.
• DexYCB S3: 20 YCB objects, subject 0, 120 videos × 72 frames.

Near-linear YCBInEOAT object motion with little rotation challenges SfM. Ours does not lead every metric: RPE rot@2° 0.24 trails DROID-SLAM 0.34. On DexYCB, Ours generally leads listed predictions, e.g. RPE rot@2° 0.34 versus CoTracker 0.21.

Cross-dataset operation does not make every dynamic sequence observable: too little translation or near-linear motion can hinder monocular SfM.

11. Failure cases

The authors identify:

1. Textureless regions lack stable tracks.
2. Initially coplanar points degenerate SfM constraints.
3. Low texture combined with large out-of-plane rotation.
4. Small translation / parallax prevents relative-pose initialization.

DexYCB adds strong hand occlusion and small distant objects. The pipeline is offline and point selection heuristic.

12. Speed and deployment

An 8-frame 512×512 crop window takes about 10 seconds on RTX 3090, including refinement and COLMAP / BA. Amortized 1.25 s/frame cannot directly serve real-time robot control.

Cross-window consistency, accumulated drift, recovery, long-video memory, and latency remain unresolved. Short-term tracking is explicitly a foundation for future long-term methods.

13. Availability and reproducibility

At this site’s verification time:

• arXiv and author publication pages were accessible.
• No verifiable author-provided official code repository found.
• No OmniPose6D download / license page found.
• Papers with Code listed no implementation.

Metadata therefore marks it unavailable as open source. Even if released later, reproduction needs:

• Licenses and cleanup lists for 4,000+ meshes.
• Four motion sampling ranges and HO3D extraction.
• Occlusion circles, crop jitter, visibility thresholds.
• Track Anything version / propagation parameters.
• CoTracker initialization, 500 / 1,500 track sampling;
• COLMAP defaults and failed-window handling.
• Exact 7-DoF alignment and conditional AP.

14. Overall assessment

OmniPose6D clearly defines short-window CAD-free RGB-only relative-object motion, with supporting synthetic data and shared baselines. Generic trackers alone are insufficient; object-rotation data and reliable-track selection improve synthetic and real HO3D performance.

Deployable metric tracking remains distant: up-to-scale outputs, GT-assisted alignment, 10 seconds per 8 frames, SfM texture / parallax dependence, and unavailable code / data links. It is best positioned as a data and short-term relative-pose benchmark.