RelPose++: Recovering 6D Poses from Sparse-view Observations
RelPose++ updates pairwise rotation energy distributions with a multiview Transformer and regresses translation in look-at-centered coordinates, recovering a consistent set of camera 6D poses from sparse object views.
roughly center-facing object viewscamera intrinsics for downstream reconstructionglobal similarity alignment for evaluation
Reading Notes
Overview / Deep Dive
Available src/paper-notes/relpose-plus-plus-3dv2024/coarse.md
Paper overview notes
Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.
RelPose++: Coarse Reading
One-sentence conclusion
RelPose++ recovers rotations and translations for a camera set from 2–8 object-centered RGB images. A multiview Transformer lets additional views resolve pairwise symmetry ambiguities; look-at-centered coordinates decouple translation prediction from an arbitrary first-frame rotation.
Method
ResNet image features jointly enter a Transformer. An energy-based pairwise relative rotation distribution replaces single-value regression, followed by global camera consistency optimization. Translation is regressed in coordinates centered near the optical axes’ intersection.
Multiview context updates image features before rotation distribution and translation estimation. Source: original Figure 2; copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
Results
CO3D seen-category 8-view camera-center accuracy@0.2 is 85.5, versus PoseDiffusion 73.7; unseen-category accuracy is 67.5, versus 55.1. Zero-shot Objectron 8-view rotation@15° / center@0.2 is 77.0/77.6. Full multiview joint rotation accuracy reaches 74.9 at 8 views, while the pair-only N=2 variant reaches 71.2.
COLMAP often fails with wide baselines and low overlap; RelPose++ supplies coarse camera initialization for reconstruction. Source: original Figure 5; copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
Boundaries
It estimates cameras around objects, not absolute object poses in a scene. Similarity-aligned evaluation leaves absolute scale and world origin undetermined. The authors acknowledge lower accuracy than successfully converged classical SfM, positioning it as sparse wide-baseline initialization.
Available src/paper-notes/relpose-plus-plus-3dv2024/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
RelPose++: Deep Reading
1. Why pairwise information is insufficient
Symmetric objects can admit multiple relative rotations from two images: views of a handleless mug side cannot locate the hidden handle. A third view revealing it resolves that ambiguity. Original RelPose models pairwise uncertainty but each distribution sees only its pair; RelPose++ first aggregates all inputs with a Transformer.
2. Rotation: distributions instead of single values
Image features and candidate relative rotations feed an energy model to obtain P(Ri→j∣I1,…,IN). Inference finds global rotations maximizing total pairwise energy. Multimodal hypotheses survive until global consistency selects a solution, unlike direct regression.
Energy-based pairwise distributions and global optimization recover rotations; a separate head predicts translations in the new frame. Source: original Figure 2; copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
3. Translation: coordinate design matters
With the first camera as world origin, other translations vary with its rotation error, coupling two tasks. Assuming roughly center-facing images, the paper centers the world near the optical axes’ intersection and normalizes global scale. Translation then describes relative positions around the object with weaker coupling to one frame’s rotation errors.
Seen-category 8-view center accuracy is 85.5 in look-at coordinates versus first-frame-centered 77.7; unseen is 67.5 versus 65.3. Translation-specific accuracy is 86.2 versus constant baseline 56.5 for seen, and 73.6 versus 48.2 for unseen categories.
4. Experiments
CO3D v2 uses 41 training categories and 10 held-out categories, about 22,375 sequences / 2.21M images. Each set samples 2–8 views. Rotation uses pairwise angular error<15°; centers first undergo optimal 7DoF similarity alignment, then count errors below 20% of scene scale. This measures internal camera-set structure, not absolute metric localization.
COLMAP is accurate when successful, but sparse wide-baseline images often lack enough matches; learned methods are more reliable but produce coarse poses. Source: original Figure 5; copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
As inputs grow from 2 to 8 views, the N=2 rotation variant stays around 69.8→71.2 while Full rises from 69.8 to 74.9. This supports ambiguity resolution from additional context rather than model capacity alone.
The upper crop shows Table 3 camera-center / translation regression versus a constant baseline; below are only the Table 4 header and partial method rows, not the look-at / first-camera frame ablation. Source: original Tables 3–4; copyright belongs to the authors and 3DV 2024 publisher; reproduced here for academic analysis.
Zero-shot Objectron 8-view rotation / center is 77.0/77.6 versus PoseDiffusion 70.0/67.2, supporting cross-dataset transfer of CO3D object priors. Outputs also initialize NeRS reconstruction.
5. Limitations
Roughly center-facing views are essential; ordinary scene trajectories may not satisfy intersecting optical-axis assumptions.
Energy-based rotation sampling and global optimization are costly and still use pairwise factors.
Similarity alignment removes scale, origin, and global rotation errors; it does not establish absolute metric poses.
Successful classical SfM is usually more accurate; RelPose++ improves initialization / convergence probability.
Crops, backgrounds, and category priors may limit non-object-centered images in the wild.
6. My assessment
Two reusable ideas are multiview context for pairwise uncertainty and a learnable, weakly coupled coordinate system before translation regression. This is not BOP pose estimation, but a useful precursor to sparse-view CAD-free object modeling.
