Deep Reading Notes: DynOPETs

1. Why dynamic-object / dynamic-camera data are needed

Most 6D object pose datasets assume at least one controllable condition:

• A fixed camera, with a turntable or robot moving the object;
• A stationary object, with the camera moving around it;
• Relatively independent frames, without long-term tracking-consistency requirements;
• Dynamic video covering only one of category-level pose, unseen-object estimation, or tracking.

Real AR, mobile robotics, and human–robot interaction often involve a person moving an unfamiliar object while the observer also moves. Blur, hand occlusion, fast rotation, weak texture, and camera motion then occur together, challenging both single-frame pose and temporal tracking. DynOPETs mainly contributes long videos supporting the following tasks, rather than a new network:

1. category-level object pose estimation (COPE);
2. unseen-object pose estimation (UOPE);
3. dynamic object pose tracking (OPT).

2. Dataset scale and composition

The paper releases 175 distinct object sequences, each with approximately 500 frames at 15 Hz, totaling approximately 210K RGB-D frames and over 530K annotations. Annotations include instance masks, camera intrinsics / extrinsics, camera poses, object poses, and scanned CAD models.

COPE-119

For category-level 6D pose / size, the subset covers standard NOCS categories such as bottle, bowl, camera, can, laptop, and mug, with within-category shape and scale variation.

UOPE-56

The remaining 56 household objects evaluate novel-object estimators not previously trained on the target data, given a query object model / scan.

Tracking

All 175 sequences support object pose tracking. Challenges include both object motion and a moving camera; evaluation concerns camera-to-object trajectories expressed in the camera frame.

Acquisition uses an iPad Pro + Structure Sensor Pro. External OptiTrack mocap provides camera trajectories, aligned with RGB-D through hand–eye calibration. Objects carry no markers, avoiding texture occlusion or changes to everyday appearance. Scanning, scale recovery, and canonicalization align models to object-local coordinates (OLC).

3. Why annotation needs three geometric sources

3.1 Absolute pose: FoundationPose + EKF/RTS

FoundationPose estimates each frame’s pose from RGB-D, an object mask, and a CAD model. Absolute poses do not accumulate drift but are affected by occlusion, symmetry, and mask errors; individual frames can abruptly jump to an incorrect mode.

The authors use a constant-velocity state over the global sequence:

$$x=[p,q,v,\omega],$$

Here $p,q$ denote position and quaternion, and $v,\omega$ linear and angular velocity. A forward EKF reduces noise, followed by backward Rauch–Tung–Striebel smoothing. This reduces jitter but cannot automatically recover the correct symmetry branch when observations remain wrong.

3.2 Relative pose: TAP point trajectories + 3D registration

Query points are selected in a reference frame and followed through video with a Track-Any-Point model. Per-frame depth back-projects their 2D correspondences into 3D; point-set registration estimates inter-frame $SE(3)$ motion.

Relative estimation avoids recognizing canonical object pose anew in every frame and is smoother for approximately symmetric or weakly textured objects. However, drifting tracks or missing depth accumulate error, and long sequences lack a global anchor.

3.3 Pose graph optimization

Each frame’s object pose becomes a graph node. Absolute estimates form unary / prior-like edges; relative motion forms adjacent or cross-frame edges. Information matrices express their reliability. All nodes are optimized in GTSAM:

• Absolute edges suppress long-term drift;
• Relative edges preserve local motion continuity;
• Visibly incorrect absolute poses require manual inspection and edge deletion.

This manual step matters. The data result from algorithmic proposals + smoothing + manual curation + graph optimization, rather than one-click automatic ground truth.

4. Annotation validation: scope of the evidence

The authors attach mocap markers to 10 additional objects and collect validation sequences separate from the main data. OptiTrack then measures both object and camera reference trajectories, allowing comparison of four annotation routes:

• Rel-Pose;
• Raw Abs-Pose;
• Refined Abs-Pose;
• PGO Pose.

Key averages are:

Annotation route	ATE max	ATE mean	ATE median
Rel-Pose	327.37 mm	144.62 mm	139.76 mm
Raw Abs-Pose	16.65 mm	4.98 mm	4.53 mm
Refined Abs-Pose	15.38 mm	4.93 mm	4.54 mm
PGO Pose	15.38 mm	4.94 mm	4.54 mm

The results indicate:

• Substantial long-term drift from relative-only estimation;
• FoundationPose absolute estimates are already within a centimeter;
• Smoothing / PGO only slightly improve average error; their main value may be removing discrete jumps and improving trajectory consistency;
• The 10 validation sequences support millimeter-scale average errors, but are far fewer than the 175 main sequences, and marker-equipped validation objects may have different motion / occlusion distributions.

The main paper summarizes rotation error below 1° and translation below 1 cm. These are best treated as results of an independent small-sample audit, rather than guarantees for every frame.

5. Category-level pose benchmark

COPE evaluates both 3D IoU and pose thresholds. Methods share SAM2 segmentation where possible to reduce detector differences.

For the original models:

• SecondPose: IoU25 98.55, 5°5 cm 64.00, 10°5 cm 83.29;
• AGPose: IoU50 92.94, IoU75 66.34, 5°2 cm 61.61, 10°2 cm 80.02.

Different methods favor scale / IoU or strict pose thresholds; a single bold column cannot establish overall superiority.

The authors also fine-tune on 84 training sequences for only 2 epochs and evaluate on 35 test sequences:

Method	5°2 cm	5°5 cm	10°5 cm
SecondPose original	61.88	67.05	84.08
SecondPose fine-tuned	76.33	82.91	97.19
AGPose original	67.11	67.50	81.92
AGPose fine-tuned	75.10	75.74	96.46

This shows that DynOPETs training annotations provide useful supervision. Fine-tuning gains do not independently prove absolute annotation correctness, however, because domain matching can also improve results. The preceding marker-based mocap validation remains the stronger check.

6. Unseen-object pose benchmark

UOPE-56 uses BOP VSD, MSSD, MSPD, and their mean AR, and reports FPS:

Method	Input	AR	FPS
FoundPose coarse	RGB	60.46	0.50
MegaPose	RGB	72.23	2.25
GigaPose	RGB	69.79	3.37
MegaPose	RGB-D	80.49	2.61
SAM6D	RGB-D	79.75	3.75
FoundationPose	RGB-D	92.41	0.98

FoundationPose has a clear accuracy advantage but does not lead in speed. The paper also states that FoundPose’s core refinement code was unavailable, so only coarse results could be reported. This makes the comparison partly asymmetric between full methods and reproducible portions.

7. Dynamic tracking benchmark

The authors compare MaskFusion, DROID-SLAM, BundleTrack, BundleSDF, and FoundationPose. For DROID-SLAM, SAM2 masks set non-object region weights to zero, then camera motion is converted into object-relative motion.

Trajectory error

Method	ATE ↓	RPE Rot. ↓	RPE Trans. ↓
MaskFusion	0.320 m	0.060 rad	0.033 m
DROID-SLAM	0.173 m	0.063 rad	0.040 m
BundleTrack	0.068 m	0.037 rad	0.021 m
BundleSDF	0.094 m	0.044 rad	0.026 m
FoundationPose	0.037 m	0.018 rad	0.010 m

Pose threshold / AUC

FoundationPose leads in ADD / ADD-S AUC at 89.80 / 96.40. However, its ADD-0.1d of 92.42 is slightly below BundleTrack’s 94.30, and its ADD-S-0.1d of 99.29 is slightly below BundleSDF’s 99.98. The paper attributes this to FoundationPose tracking failures in a few sequences. AUC and strict threshold recall differ in sensitivity to the failure tail, so both should be considered.

8. Risks when using the data

8.1 Pseudo-annotations and inherited algorithmic bias

FoundationPose initializes absolute poses in the main dataset. Even with relative edges and PGO, annotations may inherit its biases regarding texture, symmetry, or depth quality. Evaluation of the same or a closely related model can therefore exhibit annotation-model affinity.

8.2 Manual edge rejection matters

Clearly incorrect absolute edges are deleted manually. Reproducing data generation should record rejection rules, reviewer agreement, and retention rates; otherwise the cost of the “automatic annotation pipeline” is underestimated.

8.3 Neighboring-frame leakage

Each video contains approximately 500 highly correlated frames. Train / test splitting must operate by sequence or object, rather than randomly by frame, to prevent leakage of poses, backgrounds, and hand appearance.

8.4 175 objects do not imply 175 dynamic conditions

The paper largely provides one main sequence per object. Object coverage is broad, but repeated capture of the same instance across backgrounds, operators, speeds, and lighting is limited. It supports studying object diversity but cannot independently isolate acquisition-condition variance.

8.5 Scale of mocap validation

Independent marker ground truth covers only 10 additional sequences. Benchmark users should ideally consider quality signals such as per-frame annotation confidence, deleted absolute edges, and depth support, rather than assuming identical accuracy in every frame.

9. Differences from existing datasets

DynOPETs is distinctive because three tasks share one dynamic domain, beyond simply providing more frames:

• COPE methods can test within-category shape variation and dynamic occlusion;
• UOPE methods can test novel-object generalization given CAD models;
• Trackers can test long-term temporal behavior, fast motion, and moving cameras.

This enables comparisons between strong single-frame recognition and temporal stability, as well as stagewise diagnosis of detection / segmentation / pose initialization / tracking.

10. Overall assessment

DynOPETs is a valuable 2026 dynamic object pose benchmark, especially as a complement to static / short-term data such as YCB-V and REAL275. Recommended use in the source notes is to:

1. Specify the task subset and split;
2. Report both ATE/RPE and ADD(S) thresholds;
3. Stratify by symmetry, hand occlusion, and rapid motion;
4. Treat annotations as pose-graph-optimized pseudo ground truth audited with small-scale mocap;
5. For supervised training, apply annotation-confidence filtering and cross-annotator validation.

It addresses the shortage of dynamic-domain data while highlighting a lasting research question: when large-scale 6D pose data rely on strong models for annotation, how can a benchmark quantify and avoid inheriting those models’ systematic biases?