Paper Reading Notes (Deep): Effects of Markers in Training Datasets on the Accuracy of 6D Pose Estimation

1. What Do Training Images Teach a Pose Network to Use?

Many real 6D pose datasets rely on complex annotation:

• Manually align CAD to RGB-D clouds, then refine with ICP;
• Use robot kinematics, multiple cameras, or optical motion capture for poses;
• Attach active / passive markers for automatic accurate trajectories.

The third option is fast and accurate but creates a conflict: markers appear in training RGB and usually disappear at deployment. Rigidly attached markers have pose-correlated 2D positions, visible counts, and projected shapes. Networks may use them as extra keypoints, easier to learn than object textures.

The controlled experiments ask:

1. How much do fixed markers hurt markerless testing?
2. Which is preferable: accurate marker-based labels with image cleanup, or naturally markerless images with coarser labels?

The contribution fits WACV as an empirical analysis of data production, shortcut learning, and GT fidelity rather than a larger model.

2. Causal Variables and Five Training Conditions

BlenderProc2 generates paired images with identical objects, backgrounds, distractors, lighting, physically simulated resting poses, and camera poses. Only marker presence / treatment changes. This isolates causes more effectively than comparing separate real datasets.

Case	Training condition	Question
I	w/o marker	Ideal markerless upper bound
II	w marker	Direct effect of fixed-marker shortcuts
III	inpainting	Recover the upper bound after removal? Do artifacts create a new domain gap?
IV	marker augmentation	Do random 1–5 markers per image induce marker invariance?
V	aug. + marker	Half fixed, half random: real fixed capture + synthetic augmentation
VI / VII	Train with I / II respectively, test with markers	Test actual marker use versus occlusion alone
VIII	Synthetic baseline without real fine-tuning	Measure benefits of 200 real-image fine-tuning

Only training data change; no new pose loss is introduced. GDR-Net retains original parameters and augmentation, with ZebraPose checking model dependence.

3. Dataset Construction

3.1 Controlled Teabox Dataset

• One box-shaped tea package;
• Synthetic training: 20,000 images, about 800 unique scenes;
• Each scene uses random backgrounds, distractors, and physics-generated resting poses, rendered from 25 random camera positions;
• Synthetic test: 1,000 images, 500 unique scenes, two views each;
• Real training: 200 images each with and without markers;
• Real markerless test: 59 images across 3 scenes.

Teabox makes full data-condition and label-noise sweeps tractable, but its simple shape, strong texture, and single identity may not represent textureless or complex symmetric objects.

3.2 YCB-V Subset

Five YCB-V objects are selected:

• 002 master chef can;
• 004 sugar box;
• 008 pudding box;
• 025 mug;
• 036 wood block (symmetric).

The large synthetic set has about 80,000 images; synthetic testing uses 1,000 images with about 300 appearances per object. Real BOP / YCB-V testing contains 975 object instances. GDR-Net shares one network across five objects; ZebraPose trains separately per object on about 10,000 images / object.

This exposes cross-object trends and synthetic-to-real failures, but coverage is small and later averages exclude the two worst failures.

4. Marker Removal: Geometry-Guided Two-Stage Inpainting

The authors use LaMa’s pretrained large-mask inpainting model (Suvorov et al.), without object / dataset adaptation. Mask provenance is central:

1. Marker 3D positions are known;
2. Project markers into the current camera image;
3. Check visibility using scene depth;
4. Obtain accurate marker masks;
5. Apply nearest-neighbor binary dilation about 4 times;
6. After the first inpainting pass, repeat with masks dilated about 8 times.

This uses stronger information than generic automatic segmentation because the capture system already knows marker poses. Capture for motion-capture annotation should therefore preserve marker geometry / masks alongside RGB.

Examples cover synthetic YCB-V, real teabox, and textureless tools. Some artifacts and elongated shadows remain without manual cleanup, testing whether an automatic pipeline suffices rather than whether perfect retouching is lossless.

5. Marker Augmentation: Randomizing Correlation

Fixed markers remain at the same object-surface locations, strongly correlating projection and pose. Each synthetic image instead samples 1–5 markers on the surface, resampling overlap or unsuitable geometry.

The aim is to disrupt stable $p(\text{marker appearance}\mid R,t)$ so one marker position cannot predict pose. Case IV randomizes every image; V randomizes half and retains fixed markers in half, modeling real capture supplemented by synthetic randomization.

This avoids hallucinating hidden textures, but synthetic marker materials, shadows, contact, and specularities may be unrealistic, and random markers still occlude surfaces.

6. Constructing Label Error

To simulate markerless annotation, the BOP toolkit forms colored RGB / depth clouds; a person places CAD near the object and runs ICP. The authors annotate 37 multi-object frames and compare them with exact synthetic poses:

Error interval	0–2	2–4	4–6	6+
Translation-error frame counts (mm)	79	20	9	2
Rotation-error frame counts (°)	76	12	14	10

The authors describe translation errors with a normal distribution of mean $\mu=0.1$ and “variance” $\sigma=2.2$. Sparse rotation samples fit common distributions poorly. The subsequent sweep nevertheless approximates translation / rotation perturbations as independent zero-mean Gaussian noise:

$$\sigma_t\in\{0,2,4,8\}\ \text{mm},\qquad \sigma_R\in\{0,2.5,5\}^{\circ}.$$

Images remain fixed; only training-pose labels are perturbed. Test GT stays exact.

This quantifies random-label-noise sensitivity, not all real error sources. Marker-to-object registration can create object-level systematic bias, while depth scale, extrinsics, and ICP can produce view-correlated errors.

7. Networks, Training, and Metrics

7.1 GDR-Net

• RGB-only 6D pose estimator;
• teabox: batch size 24, 10 epochs;
• YCB-V: batch size 24, 80 epochs;
• All data conditions use the same seed to reduce initialization differences;
• GDR-Net’s original parameters and augmentation are retained.

A shared seed improves control but no multi-seed mean / variance is reported, leaving uncertainty about 0.5–1-point differences versus training variation.

7.2 Metrics

• ADD: mean distance between corresponding model points;
• ADD-S: mean predicted-point distance to nearest GT surface points, suitable for symmetry;
• ADD-(S): ADD-S for symmetric objects, ADD otherwise;
• ADD-(S) success: error below 10% of object diameter;
• AUC: ADD-S / ADD-(S) curve area up to a 10 cm threshold.

Table 2 reports AUC ADD-S, AUC ADD-(S), and ADD-(S). AUC ADD-S varies by only about 1 point across conditions; ADD-(S) is more sensitive and drives the shortcut discussion.

8. Teabox Results: Separating Shortcuts, Occlusion, and Domain Mismatch

8.1 Five Training Conditions on Markerless Tests

Case	AUC ADD-S	AUC ADD-(S)	ADD-(S)
I w/o marker	97.1	90.0	84.8
II w marker	96.2	87.8	76.6
III inpainting	97.0	90.0	84.8
IV marker aug.	97.0	89.8	84.2
V aug. + marker	96.7	88.8	80.0

The findings are clear:

• Fixed-marker training and markerless testing create a domain mismatch;
• Inpainting fully restores the table’s ADD-(S) baseline;
• Fully random marker augmentation nearly restores it;
• Fixed + random mixtures retain some shortcut.

The paper has an internal numeric inconsistency: Table 2 gives an ADD-(S) gap of $84.8-76.6=8.2$ points, Section 4.1 says 5.6 percentage points, and the conclusion says 8. This site uses the table’s 8.2 points rather than repeating 5.6.

The conclusion’s roughly 3-point augmentation gain over unedited markers is closer to mixed Case V, $80.0-76.6=3.4$. Fully random Case IV improves by 7.6 points. State the augmentation protocol explicitly.

8.2 Why Surface Occlusion Alone Is Insufficient

Table 3 switches to tests with markers:

Training	AUC ADD-S	AUC ADD-(S)	ADD-(S)
VI w/o marker	96.4	88.3	77.7
VII w marker	97.5	91.0	87.1

If markers only occluded surfaces, marker-trained models would not clearly win on marker tests. The network uses markers as pose-discriminative features, helpful when present in both domains. Deployment mismatch is the real issue.

8.3 Real Teabox Fine-Tuning

Fine-tune a markerless synthetic model on 200 real images for 6 epochs, then evaluate on 59 real markerless images:

Real fine-tuning data	AUC ADD-S	AUC ADD-(S)	ADD-(S)
I markerless real	93.6	79.0	45.8
III inpainted real	93.2	79.1	47.5
VIII no real finetuning	88.8	71.9	20.3

The two fine-tuning strategies are close and both exceed no fine-tuning by over 25 points. This supports automatic cleanup of real marker images before pose training, but 59 test images, one object, and 3 scenes cannot resolve small differences across broad domains.

9. YCB-V: Similar Averages and Object-Level Failures

9.1 Synthetic test

Five-object GDR-Net averages:

• markerless I: AUC ADD-S 79.9, ADD-S 78.0;
• inpainting III: AUC ADD-S 81.3, ADD-S 77.5.

Objects improve or decline without a consistent advantage: evidence supports no systematic harm from inpainting, not universal pose improvement.

9.2 Real test

Object	I AUC ADD-S	III AUC ADD-S	I ADD-S	III ADD-S
002 master chef can	92.1	94.3	79.7	86.7
004 sugar box	96.8	96.1	98.9	95.7
008 pudding box	89.1	89.1	76.0	78.7
025 mug	65.3	52.1	18.7	4.7
036 wood block	17.7	4.3	0.0	0.0

Only the first three objects enter the reported average: AUC ADD-S 92.6 → 93.2 and ADD-S 84.9 → 87.0. Mug and wood block are excluded for synthetic-to-real failures.

This averaging must be explicit, rather than simply claiming a 2.1-point YCB-V improvement. Including all five would change the conclusion substantially. The authors openly show failures and distinguish their shared network from GDR-Net’s per-object / alternative setup in Table 7, but post-hoc exclusion still makes the average selective.

9.3 ZebraPose Cross-Check

To test dependence on GDR-Net, ZebraPose trains per object:

Test domain	Markerless I	Inpainting III
Synthetic AUC ADD-S	92.9	92.3
Real AUC ADD-S	60.9	61.8

The strategies remain close and swap direction between synthetic / real data. This strengthens no-obvious-harm evidence but not stable 0.6–0.9-point gains, especially without multi-seed variance.

10. Label Noise: Accurate GT Can Offset Marker-Removal Costs

Four observations:

1. With $\sigma_t=2$ mm and $\sigma_R=0$, curves barely change;
2. AUC / ADD-(S) consistently deteriorates beyond $\sigma_t\ge 4$ mm;
3. Adding 2.5° rotation noise alone can reduce recall by about 1.5–7 points;
4. Case II’s black line provides a trade-off: sufficiently coarse manual markerless labels may be worse than accurate labels with marker shortcuts. The preferred approach here is accurate tracking followed by inpainting.

Curves are not strictly monotonic: some $\sigma_t=2$ combinations improve slightly, and $\sigma_t=8,\sigma_R=2.5°$ may beat pure 8 mm noise. The authors note this. With one seed, training randomness or noise regularization is more plausible than a general benefit from rotation noise.

11. What Is Actually Established?

Claim	Evidence strength	Interpretation
Fixed markers create pose shortcuts	Strong	Paired synthetic controls plus marker-test sanity check
Two-stage inpainting restores teabox baseline	Strong	Exact Table 2 match and consistent real fine-tuning
Inpainting is broadly harmless on YCB-V	Moderate	GDR-Net / ZebraPose averages are close, but few objects, two severe failures, and no multiple seeds
Marker augmentation works	Moderate	Fully random IV nearly recovers; practically feasible mixed V only partly recovers
Millimeter / degree label errors matter	Moderate	Clear controlled sweep, but idealized noise
Marker-based capture always beats markerless capture	Unproven	Conditional on better labels, known masks, suitable repair, and stable calibration

12. Limitations and Potential Biases

12.1 Narrow Object / Scene Coverage

The causal study uses teabox; YCB-V adds only five objects. Texture, relative marker size, mounting surfaces, and symmetry can all change shortcut strength.

12.2 Nearly Oracle Masks

Known 3D marker positions, camera poses, and depth visibility give highly accurate masks. A capture system preserving only object pose would make cleanup harder.

12.3 Residual Marker Shadows

Elongated shadows can remain, although uncommon under the chosen lighting. Networks could infer marker locations from shadows; limited impact here does not establish lighting-independent robustness.

12.4 Selective Real-Test Averaging

Table 6 excludes mug / wood block after observing failures. Publishing complete per-object rows is valuable; this site must retain failed rows and explain the average.

12.5 One Seed and No Uncertainty Intervals

A shared seed controls comparisons but is not repeated training. The 8.2-point teabox effect is robust; 0.6–2.1-point YCB-V differences may reflect variation.

12.6 Label Noise Is Not Full Calibration Error

Independent zero-mean Gaussian noise mainly models random manual errors. Object–marker registration, camera extrinsics, or clock errors may impose a consistent SE(3) bias across frames, harder to average away.

13. Reproduction Essentials

1. Use 6DPoseDataGenTools to record paired scenes under the same seed;
2. Ensure markerless / marker / inpainting / augmentation conditions change only the intended variables;
3. Preserve marker 3D poses and masks, not just final object GT;
4. Distinguish AUC ADD-S, AUC ADD-(S), and 10%d ADD-(S);
5. Reproduce the Table 2 8.2-point gap, not the contradictory 5.6 in the text;
6. Report real YCB-V averages for both 3 retained objects and all 5;
7. Run at least 3–5 seeds and provide confidence intervals for small differences;
8. Test systematic SE(3) calibration bias, depth-scale bias, and correlated temporal drift;
9. Use activation / occlusion attribution to check attention to inpainting boundaries or marker shadows.

14. Dataset-Building Recommendations

• Preserve reversible capture information: marker IDs, 3D geometry, per-frame poses, visibility masks, object-to-marker transforms, calibration covariance.
• Release raw and cleaned RGB: allow later inpainting choices and artifact audits.
• Add marker-swap tests: place markers inconsistently with GT to quantify shortcut reliance.
• Evaluate all four train / test marker combinations: report more than marker training → markerless testing.
• Provide GT uncertainty: record object–marker registration and synchronization errors rather than treating motion capture as exact.
• Look beyond averages: stratify by material, symmetry, marker-area fraction, occlusion, and pose bins.

15. Overall assessment

The contribution is a data-quality principle rather than a new state-of-the-art estimator:

Any device used to obtain labels can also become a visual shortcut for predicting them. Removing its appearance and preserving label accuracy must be designed together.

Paired synthetic controls, real fine-tuning, cross-object YCB-V tests, a second pose network, and label-noise sweeps provide a broad evidence chain. Fixed markers cause domain mismatch; two-stage inpainting is nearly lossless here; label accuracy matters.

Generalization remains limited by five objects, one seed, oracle masks, Gaussian noise, and selective averages. New datasets should integrate marker removal, calibration uncertainty, and shortcut auditing into their protocol rather than treat cleanup as final retouching.