Paper Reading Notes (Deep): Omni6D

1. Key assessment

Omni6D advances category-level pose from within-category generalization over a few classes toward large vocabularies, long tails, and cross-category generalization:

1. 166 categories and 4,688 instances expose both within- and cross-category shape variation.
2. Free-fall placement breaks the upright bias common in NOCS / CAMERA.
3. Symmetry moves from category conventions to per-instance three-axis annotations.
4. Omni6Dout explicitly tests unseen training categories.
5. Category-count scaling directly tests vocabulary expansion.

The central conclusion is that translation / size generalize more readily, while rotation is sensitive to category expansion. IoU50 or 5cm alone can overstate system capability.

2. What exactly is real?

The table labels Omni6D 3D models Real-Scanned, which can be misreported as 800K real images. The precise breakdown is:

• Real scans: 4,688 textured OmniObject3D meshes.
• Synthetic scenes: 9 Replica rooms.
• Synthetic placement: BlenderProc 2.5.0 free fall of 6–8 objects per scene, with random object scale 0.8–1.2.
• Synthetic cameras: 10 views per scene, radius 8–9 m, elevation 30°–90°.
• Synthetic outputs: 640×480 RGB, depth, mask, NOCS, pose, size; intrinsics $[577.5,577.5,319.5,239.5]$.

Scanned shape / texture is more realistic than primitive CAD, but rendering, Replica lighting, sensor noise, and compositing domain gaps remain.

3. Scale, splits, and annotations

There are 166 categories, 4,688 instances, and over 0.8M captures. A 7:2:1 instance split defines Omni6Dtrain / val / test, making test instances unseen but categories seen.

Omni6Dout adds:

• 52 object models.
• 17 categories entirely unseen during training.
• 4,762 images.

These splits separate two generalization types:

• test: unseen instance within seen categories;
• out: unseen instances from unseen categories.

Omni6D-Real adds 30 real scenes, 39 categories, 73 instances, and about 1K Azure Kinect images. SAM assists masks; ICP registration provides object poses. It is a small domain-transfer sanity check rather than a replacement for large real benchmarks.

4. Instance-level three-axis symmetry

For predicted rotation $\hat R$, GT $R$, and equivalent-rotation set $\mathcal S$, symmetry-aware error is:

$$e_R=\min_{S\in\mathcal S} d\!\left(\hat R,RS\right),$$

Here $d$ is the SO(3) geodesic angle. Sym-0 / 1 / 2 / 3 denote no equivalent rotation, arbitrary axial angles, 90° multiples, and 180° multiples. Combining three axes needs at most $4^3$ evaluations.

This improves on category-wide axial symmetry: bottle, lamp, or toy instances may differ through handles, logos, or geometry. Approximation boundaries remain:

• Continuous symmetry is approximated by discrete evaluation.
• Shape invariance need not reflect texture.
• Multiaxis discrete combinations may duplicate rotations.
• Canonical frames still follow human conventions.

5. Why the distribution is harder

The analysis finds:

• Centroids resemble CAMERA / REAL distributions, but occlusion and cloud spatial variation are stronger.
• Wider relative object-size distributions.
• Free fall makes orientation relative to up more uniform, reducing upright priors.
• Agglomerative clustering of VAE shape priors groups geometrically similar but semantically different categories.

Performance correlates more with instance counts in shape clusters than semantic-class sample counts. Models may learn shared geometric modes rather than labels alone; category-balanced sampling without shape diversity may not improve generalization.

6. Seen- and unseen-category main results

6.1 Seen categories

Method	IoU50	IoU75	5°2cm	10°5cm	2cm	5cm
SPD	44.56	20.37	7.55	19.23	37.49	70.09
DualPoseNet	58.84	25.49	8.28	19.05	73.82	96.37
HS-Pose	62.65	23.02	4.26	11.61	80.93	97.78

HS-Pose leads in translation / IoU but struggles on strict joint thresholds. Its 5cm is 97.78 while 5°2cm is only 4.26: center / size can be right without useful orientation.

6.2 Unseen categories

Method	IoU50	IoU75	5°2cm	10°5cm	2cm	5cm
SPD	7.56	0.95	0.18	1.65	8.88	40.59
DualPoseNet	36.85	12.06	3.24	8.51	78.00	98.60
HS-Pose	36.75	8.92	1.54	5.16	79.95	98.27

Explicit translation / size heads in DualPoseNet and HS-Pose preserve high 2cm / 5cm despite rotation collapse. Unseen-category generalization needs angular thresholds alongside 5cm.

7. Category-scale experiments

Training vocabularies grow through 3, 6, 12, 24, and 48 categories, comparing scratch training with preceding-stage fine-tuning. Fine-tuning usually helps, but performance still declines; HS-Pose 5°2cm falls from 62.52 at 3 categories to 14.42 at 48.

This indicates:

• More classes are not simply more independent samples.
• Capacity, shared shape priors, and conditioned heads interfere.
• Curriculum / fine-tuning helps optimization without resolving representation conflicts.
• Structures successful on six NOCS classes should not automatically be extrapolated to large vocabularies.

8. Synthetic-to-real results

In the 70-person survey, Omni6D scores $2.69\pm0.39$, above CAMERA’s $1.55\pm0.08$ but below REAL’s $3.53\pm0.28$ and WILD6D’s $4.38\pm0.23$. Scanned assets plus indoor PBR look more realistic, without reaching photorealism.

DualPoseNet on REAL275:

Training set	IoU50	IoU75	5°2cm	10°5cm
Omni6D	78.76	32.69	6.55	21.38
REAL	84.51	43.43	8.76	25.39
REAL + Omni6D	85.28	58.59	14.10	38.97

Omni6D supports useful pretraining / augmentation: joint-training IoU75 is 15.16 points above REAL-only. Omni6D-only still trails on strict pose, leaving synthetic-to-real transfer unresolved.

9. Evaluation boundaries and limitations

• GT masks exclude large-vocabulary detection, classification, and mask errors.
• Real-scanned meshes are not real images; renderer / sensor gaps persist.
• About 1K real images are limited, with mask / registration annotation errors.
• 166 categories still follow OmniObject3D assets, not open-world long tails.
• Category labels and shape clusters differ; class-conditioned architectures may be unsuitable inductive biases.
• Symmetry-minimized errors may treat texture-distinguishable orientations as equivalent.
• Mainly single frames; video scanning and temporal modeling remain future work.
• The appendix extends Omni6D-xl to 13,161 instances / 339 categories; main experiments and conclusions use base Omni6D.

10. Usage and reproduction checklist

• State clearly that assets are real scans and observations synthetic renders.
• Keep the 7:2:1 instance split; never share meshes across train / test.
• Report seen and unseen categories separately.
• Report IoU, rotation-only, translation-only, and joint thresholds.
• Separate GT-mask backends from actual detector / segmenter systems.
• Minimize over per-instance three-axis symmetry sets.
• Analyze head / common / tail classes and shape-cluster frequencies.
• Fix BlenderProc, Replica rooms, cameras, scales, and free-fall settings.
• Stratify real depth noise, transparency / reflection, and heavy occlusion.
• Track capacity, sampling, and catastrophic interference as classes grow.

11. Overall assessment

Omni6D changes the question from accuracy on six tabletop categories to orientation under 166 categories, long-tailed shapes, and unseen categories. Results show clearly that strong translation / size scores do not solve 6D pose.

Use it with its boundaries intact: mainly synthetic RGB-D from real-scanned assets, GT-mask tables, and a small real set. Within these limits, it is a valuable stress test for large-vocabulary pose, instance symmetry, and degradation with category scale.