Paper Reading Notes (Deep): MonoDiff9D: Monocular Category-Level 9D Object Pose Estimation via Diffusion Model

1. Task definition

Given one RGB image, camera intrinsics $K$, and an object mask, predict:

$$y=(R,t,s),$$

Here $R\in SO(3)$, $t\in\mathbb R^3$, and $s\in\mathbb R^3$ is metric size. Category-level means an unseen target instance belongs to a training category. Beyond 6D pose alone, 9D additionally predicts size along three axes.

2. Constructing geometric conditions from RGB

For a masked pixel $(u,v)$ and estimated depth $d(u,v)$:

$$X=d(u,v)K^{-1}[u,v,1]^\top .$$

This converts monocular depth to a cloud for PointNet to encode 3D structure. With relative-scale depth alone, translation and object size drift together, making metric training supervision and the depth model’s scale generalization crucial.

The RGB branch uses ResNet-18 for texture / contours, while the global point branch encodes estimated geometry. Cross-attention does not recover true depth: it learns pose-denoising conditions from the available noisy geometry.

3. Diffusion pose generation

The forward process gradually adds noise to the ground-truth pose vector:

$$q(y_t\mid y_0)= \mathcal N(\sqrt{\bar\alpha_t}y_0,(1-\bar\alpha_t)I).$$

The Transformer denoiser receives $y_t$, a timestep embedding, and visual conditions to predict noise or clean pose. Inference begins at $y_T\sim\mathcal N(0,I)$ and uses DDIM to progressively obtain $\hat y_0$.

Potential diffusion benefits:

• Represent multiple modes caused by symmetry / occlusion.
• Iterative denoising may correct coupling among components more easily than single-pass regression.
• Sample and rank multiple candidates.

If the final output is only one sample without reliable ranking, however, the system-level benefit of multimodal modeling still needs experimental validation.

4. Coupling rotation, translation, and size

The 9D components have substantially different numerical ranges. An effective denoiser must handle:

• Valid rotation representations.
• Scale coupling between translation depth and size.
• Different canonical coordinates across categories.
• Multiple symmetry-equivalent rotations.

Diffusion in Euclidean vectors does not automatically keep every step on $SO(3)$; valid rotation representation or decoding is needed. Reading and reproduction should inspect normalization, rotation projection, and symmetry-aware losses in code, beyond the fact that diffusion is used.

5. Experimental results

Dataset	IoU50	IoU75	10 cm	10°	10° 10 cm
CAMERA25	35.2	6.7	33.6	80.1	28.2
REAL275	31.5	6.3	41.0	56.3	25.7

Three observations:

1. Rotation-only 10° is much higher than joint 10° 10 cm, identifying translation as a major bottleneck.
2. IoU75 is only around 6–7, leaving strict scale / position alignment difficult.
3. Synthetic CAMERA25 rotation exceeds REAL275, reflecting sim-to-real and monocular-depth generalization issues.

Wild6D tests cross-domain generalization without training on that dataset. This supports some cross-dataset transfer, rather than transfer to arbitrary novel categories.

6. Speed and deployment

The paper reports about 26.3 FPS on an RTX 3090. Interpret this alongside:

• Whether instance masks are included.
• Whether DINOv2 depth estimation is included.
• DDIM step count.
• Objects per image.
• Whether multiple hypotheses are sampled.

Reducing diffusion steps improves speed but may sacrifice denoising quality, making this the most direct engineering adjustment.

7. Limitations and improvements

• Depth quality is an explicit bottleneck; object boundaries, transparent surfaces, and reflections produce erroneous clouds.
• Masks and intrinsics remain required; this is not an end-to-end open-scene system.
• Category canonical frames still need definition, with complex cross-category symmetry rules.
• Monocular metric scale cannot be recovered reliably from arbitrary images without data priors.
• Extensions include multi-hypothesis ranking, uncertainty calibration, and test-time geometric consistency.

8. Deep-reading conclusion

MonoDiff9D goes beyond replacing a regression head with diffusion: it first converts RGB into noisy 3D conditions, then learns a conditional distribution over 9D pose. Competitive rotation but low strict IoU and joint translation scores point to monocular metric geometry as the system’s limiting factor.