Deep Reading Notes: Unsupervised Category-Level 3D Pose from Object-Centric Videos

1. Key assessment

The paper addresses missing canonical frames, beyond training a regressor without pose labels. Each video in a category has its own SfM coordinates; without aligning them, a model cannot know which direction means bicycle forward across videos.

The authors divide the problem into:

1. Instance neural-mesh alignment: combine geometry, DINOv2 appearance, and 3D cycle consistency to align coarse category reconstructions in a canonical frame;
2. Category neural-mesh learning: learn viewpoint-invariant surface embeddings in that frame;
3. Single-image pose fitting: initialize with template matching at test time, then optimize rotation through render-and-compare.

The main boundary is that unsupervised does not remove geometric tools. Training still needs SfM, full-circuit videos, coarse meshes, and differentiable rendering, while avoiding manual poses, category CAD, and depth sensors.

2. Data flow and supervision sources

Training material:

• Multiple object-centric RGB videos within one category;
• Relative camera poses and coarse clouds per video from off-the-shelf SfM;
• Cloud cleaning and watertight-mesh reconstruction;
• Image patch features from frozen DINOv2.

Not used:

• Manual pose labels;
• Category or instance CAD;
• RGB-D camera;
• PASCAL3D+ / ObjectNet3D pose labels for training.

CO3D GT alignment follows the ZSP evaluation protocol and only measures Acc30 / Acc15. Rule-based filtering of failed SfM and poor viewpoint coverage remains necessary, so data selection is an important engineering prior.

3. Neural-mesh representation

A neural mesh is written as:

$$S=\{V,F\}, \quad V=\{v_i\in\mathbb R^3\}, \quad F=\{\{f_i^k\in\mathbb R^D\}_{k=1}^{|F_i|}\}.$$

Each vertex may store several features because DINOv2 appearance varies by viewpoint. In the first training stage, mesh vertices project into each frame’s DINOv2 map and aggregate visible patch features at the vertex, retaining view-dependent appearance better than a simple average.

The second-stage category neural mesh learns viewpoint-invariant vertex features: the same canonical surface location should map to the same vertex prototype from any view, enabling stable single-image 2D–3D correspondences.

4. Self-supervised cross-instance alignment

Given source neural mesh $S=(V,F)$ and reference $\bar S=(\bar V,\bar F)$, estimate similarity transform $T$ containing rotation, translation, and scale. The basic objective combines:

$$\min_T D_{\text{geo}}(V,\bar V) +D_{\text{app}}(S,\bar S).$$

The geometry term uses bidirectional Chamfer-like nearest-neighbor distances; the appearance term finds cross-mesh matches through DINOv2 feature nearest neighbors. Geometry alone suffers from topology variation and symmetry, while appearance alone suffers from patch ambiguity and reconstruction noise.

4.1 3D cyclic distance

If source vertex $v_i$ has feature nearest neighbor $\bar v_j$, and matching back from $\bar v_j$ returns source vertex $v_{i'}$:

$$d_{\text{cycle}}(v_i,f_i)=\|v_i-v_{i'}\|_2.$$

Returning to the starting point indicates more reliable correspondence. A softmax temperature $\tau$ converts cycle distances to weights so stable matches dominate Umeyama fitting, rather than forcing erroneous topologies into one-to-one matches.

The final distance is:

$$\mathcal L(S,\bar S) =(1-\alpha)D_{\text{geo}}^\* +\alpha D_{\text{app}}^\*.$$

With $\alpha=0.2,\tau=100$, geometry has about four times the appearance weight, while more correspondences contribute. RANSAC randomly samples four source vertices and their feature correspondences, then fits a similarity transform with Umeyama.

5. From aligned videos to single-image pose

Aligned videos share one canonical frame. The model learns vMF compatibility between image features $f_i$ and template-vertex features $f_r$, with background prototype $\beta$. Training classifies both foreground vertices and background.

Implementation:

• DINOv2 ViT-S/14, 21M parameters;
• A 448×448 input yields 32×32×384 features;
• Freeze DINOv2 and add three ResNet blocks in stage two;
• Final dense features are 64×64×128;
• Adam, 10 epochs;
• Less than one hour per category on RTX 2080.

Inference enumerates predefined camera poses, chooses initialization by render-and-compare joint likelihood, then updates pose iteratively through differentiable rendering. Foreground pixels may be explained by background features to reduce clutter interference.

6. CO3D alignment results

Method	Acc30	Acc15
TEASER++	3.8	1.1
ZSP	49.4	28.4
UCD+	69.8	54.6
This method	77.0	61.6

Five reference videos are sampled per category; reported means and standard deviations vary across references. Acc30 standard deviation is 11.9, so the canonical frame still depends substantially on reference choice. Laptop and motorcycle reaching 100% does not mean all 20 categories saturate; the six displayed categories are only a subset.

7. In-the-wild single-image results

Seven PASCAL3D+ categories:

Method	Supervision	Acc30
StarMap	pose labels	82.49
VoGE	pose labels	90.93
ZSP	reference depth / CAD / pose	46.0
This method	No manual pose supervision	69.2

Bicycle scores 58.4, slightly below ZSP’s 61.7, while bus reaches 79.3 versus 21.4 and car 98.2 versus 61.6. Means hide category differences: stable-topology, well-represented car / couch categories perform better, while variable categories such as chair are harder.

Twenty ObjectNet3D categories:

Dataset	ZSP Acc30	This method Acc30	This method Acc15
PASCAL3D+	46.0	69.2	41.3
ObjectNet3D	42.2	52.4	25.5

Inference takes 0.22 s / sample versus ZSP’s 10.92 s. Learned category templates replace multireference image matching, but 0.22 s still includes initialization and differentiable pose optimization rather than one regression pass.

8. Why not direct rotation regression?

The paper compares direct 6D rotation-representation regression with the same backbone:

Data	Direct regression Acc30 / Acc15	Neural mesh Acc30 / Acc15
PASCAL3D+	66.0 / 25.8	69.2 / 41.3
ObjectNet3D	44.5 / 16.4	52.4 / 25.5

Larger Acc15 than Acc30 gains suggest explicit surface correspondences and render-and-compare particularly help fine angles. Direct regression can collapse symmetry and within-category variation into average directions.

9. Data filtering and reproducibility

CO3D contains about 1.5M frames, 50 categories, and 19k scenes, but not every original video is suitable:

• Too little viewpoint change causes SfM failure;
• Cameras do not circle the object center;
• Views are too close to capture whole-object geometry.

The target is 50 videos per category, but remote has only 17, mouse 15, TV 16, and toilet 7. Uneven category data is directly linked to lower ObjectNet3D scores. Filtering, mesh cleaning, reference videos, and object centering may matter more for reproduction than network hyperparameters.

10. Limitations

• Rigid category templates struggle with large topology changes such as chairs or deformable objects.
• Near-360° videos and usable SfM are required; transparent, reflective, and weak-texture objects are hard to reconstruct.
• Random references define canonical orientation, allowing semantic-direction drift across a category.
• Testing assumes object-centered crops, without evaluating detection / segmentation.
• Still below fully supervised methods, especially on strict Acc15.
• The main paper provides no official code link, increasing reproduction cost relative to open-source methods.

11. Overall assessment

The useful principle is align training-data frames before learning pose. Many unsupervised-pose approaches design equivariance losses without addressing incompatible canonical frames across videos; this work explicitly bridges them using SfM geometry, DINOv2 appearance, and cycle consistency.

Acquisition and reconstruction costs remain, but ordinary RGB videos replace expensive manual pose annotations and CAD. This is practical for new categories with available full-circuit videos but difficult pose labeling. Single-image training, large topology variations, or unreliable SfM still call for weaker supervision or generative 3D priors.