Object Pose Estimation · 2024 · ICRA · CCF B

RGB-Based Category-Level Object Pose Estimation via Decoupled Metric Scale Recovery

DMSR decouples correspondence learning from metric-scale recovery for RGB-only category-level pose. Monocular depth / normals provide local geometry, while a separate scale branch prevents scale errors from contaminating PnP, jointly recovering pose and size for unseen category instances.

Paper Code

Research area

Object Pose Estimation

Task level

Category-level

Object setting

Unseen objects

Input modalities

single RGBmonocular geometry priors

Requirements

camera intrinsicsobject mask / cropcategory labelcategory shape prior during training

Full paper metadata Authors, affiliations, research area, datasets, metrics, method tags, and reading status Expand

Authors: Jiaxin Wei, Xibin Song, Weizhe Liu, Laurent Kneip, Hongdong Li, Pan Ji
Affiliations: ShanghaiTech University / Tencent / Australian National University
Conference / journal: Conference · ICRA · CCF B
Research area: Object Pose Estimation
Code availability: Code available
Reading status: Overview:Available Deep Dive:Available
Task setting: Category-level · Unseen objects
Datasets: CAMERA25 REAL275 NOCS
Evaluation metrics: 3D IoU n° mcm
Method tags: Category-level Pose Estimation 2D–3D Correspondences Keypoints + PnP Dense Correspondences & Coordinate Fields Synthetic Data & Foundation Models Absolute Pose Estimation
Requirements: camera intrinsicsobject mask / cropcategory labelcategory shape prior during training

Reading Notes

Overview / Deep Dive

Available src/paper-notes/dmsr-icra2024/coarse.md

Paper overview notes

Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.

Overview Notes: RGB-Based Category-Level Object Pose Estimation Via Decoupled Metric Scale Recovery

1. Basic information

Abbreviation: DMSR
Authors / affiliations: Jiaxin Wei, Xibin Song, Weizhe Liu, Laurent Kneip, Hongdong Li, Pan Ji; ShanghaiTech, Tencent, ANU
Publication: ICRA 2024
Paper / code: arXiv / GitHub
Reading mode: Overview

2. Positioning in one sentence

DMSR separates scale-free NOCS correspondences from metric size for single-RGB category-level pose. Pretrained monocular depth / normals serve only as local geometry features; an independent branch regresses scale from category priors, followed by RANSAC-PnP for 6D pose.

DMSR Figure 1: decoupled 2D–3D correspondence and metric-scale pipeline — Figure 1 from the paper: the upper branch learns NOCS correspondences between image points and a deformed category prior; the lower branch independently regresses metric scale. They meet only before final PnP. Source: arXiv. Image copyright belongs to the authors and publisher; reproduced here for paper commentary.

3. Problem setting

Task: category-level 6D pose and metric size for unseen instances.
Input: one RGB image, target crop / mask, category mean shape, and camera intrinsics.
Auxiliary information: Omnidata-predicted relative depth and surface normals.
Output: $R,t$ and 3D scale.
Key distinction: no real depth at test time, and predicted depth is not directly treated as metric depth.

4. Core motivation

RGB-only methods often predict absolute depth and reuse an RGB-D 3D–3D alignment pipeline. Metric-depth bias then corrupts scale, translation, and rotation together. DMSR explicitly decouples rigid-pose correspondence from metric scale because their learning difficulties differ.

5. Method overview

Extract and fuse separate RGB, relative-depth, and normal features.
Adapt a category-shape prior to the instance through a deformation field.
A correspondence matrix associates 2D image points with deformed NOCS points.
An independent MLP predicts scale offsets from category statistics.
Restore metric scale to NOCS points and recover pose through RANSAC-PnP.

6. Experimental overview

Datasets: CAMERA25 and REAL275.
Metrics: 3D IoU and joint $n^\circ$ / $m$ cm criteria.
Results: clear gains on rotation-sensitive metrics over contemporary RGB-only methods; real-data scale prediction remains affected by synthetic-to-real category-size distribution shifts.
Ablations: depth + normal 2.5D sketches improve over RGB alone or a single geometry channel; independent scale offsets improve over entangled scale / correspondence.

7. Strengths and limitations

Strengths

Clearly explains how monocular scale ambiguity propagates.
Final pose remains constrained by correspondence + PnP.
No test-time real depth, with public code.

Limitations / risks

Still requires masks, category labels, mean shapes, and training NOCS / pose supervision.
Bias in category mean-scale statistics directly affects translation / size.
Monocular depth / normals are not metric measurements; domain shifts can still damage correspondence.

8. Research takeaways

Intermediate geometry with different reliability should not all enter one similarity-transform solver. Separating rotation-friendly correspondence from metric scale reduces error coupling.

Available src/paper-notes/dmsr-icra2024/deep.md

In-depth paper notes

Support reproduction, comparison, related-work writing, and identifying opportunities for further research.

Deep Reading Notes: RGB-Based Category-Level Object Pose Estimation Via Decoupled Metric Scale Recovery

1. Basic information

Abbreviation: DMSR
Authors / affiliations: Jiaxin Wei (ShanghaiTech), Xibin Song, Weizhe Liu, Pan Ji (Tencent), Laurent Kneip (ShanghaiTech), Hongdong Li (Tencent / ANU)
Year / venue: 2024 / ICRA
Paper: arXiv / PDF
Code: goldoak/DMSR
Reading mode: Deep Dive

2. Positioning in one sentence

DMSR uses pretrained relative depth / normals as correspondence features rather than demanding reliable absolute monocular depth. It learns 2D–3D correspondences in normalized category space and independently recovers metric scale from category priors. Perspective PnP then solves $R,t$ , structurally limiting propagation of scale errors into rotation.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Key conclusion
Figure 1	RGB / 2.5D features, prior deformation, correspondences, scale branch, PnP	Pose and scale meet only before the solver
Figure 2	CAMERA25 / REAL275 qualitative results	Boxes / axes broadly align under real clutter, with remaining scale bias
Tables I–III	Main results, input-modality and scale-strategy ablations	2.5D geometry and decoupled scale contribute independently

DMSR Figure 1: complete method framework — Figure 1 from the paper: RGB and Omnidata 2.5D sketches supply instance features; prior deformation produces NOCS 3D points. A correspondence matrix maps image points to NOCS, while metric-scale recovery predicts size separately. Source: paper PDF. Image copyright belongs to the authors and publisher; reproduced here for paper commentary.

3. Problem setting and task boundaries

Attribute	Setting
Task	Category-level 6D pose + metric size
Test objects	Unseen instances within known categories
Observation	One RGB crop; no real test-time depth
Geometric assistance	Pretrained Omnidata relative depth and surface normals
Object prior	Normalized mean shape / point set per category
Training supervision	Masks, categories, NOCS correspondences, pose / size
solver	perspective-n-point + RANSAC

DMSR is not prior-free: instance CAD is unnecessary, but category mean shapes are required. Unlike RGB-D category-level methods, it does not directly obtain reliable metric point clouds.

4. Research context

NOCS / SPD / SGPA typically align observed depth clouds to normalized category priors through Umeyama / Procrustes for joint pose and scale.
RGB-only methods such as OLD-Net first recover object-level absolute depth and return to an RGB-D pipeline, propagating metric-depth errors into rigid transforms.
DMSR treats relative depth / normals as correspondence cues rather than metric coordinates.

5. Method details

5.1 2.5D Geometric Features

For crop $I$ , preprocessing gives

\hat D=f_d(I),\qquad \hat N=f_n(I),

where $\hat D$ is relative depth and $\hat N$ surface normals. Omnidata pretrained models are used without retraining metric-depth supervision inside DMSR.

RGB, depth, and normals pass through separate CNNs / projections to produce semantic and geometric features. Relative-depth values are not directly back-projected: even with incorrect global scale, useful local ordering and surface directions can still support correspondence.

5.2 Category-Prior Deformation and Correspondence

Feature interactions on category prior $P_r$ predict deformation field $\Delta P_r$ :

\tilde P_r=P_r+\Delta P_r.

Between image points and $\tilde P_r$ , the network predicts soft correspondence matrix $C$ , yielding normalized object coordinates for each 2D point. These 3D coordinates reside in NOCS without metric scale.

5.3 Decoupled Metric Scale Recovery

An independent MLP predicts offset $\Delta s$ from category mean size using instance / category features. With mean scale $s_r$ :

\hat s=s_r+\Delta s.

Correspondence learning does not depend on accurate $\hat s$ . Only before PnP are NOCS points multiplied by $\hat s$ to recover metric object coordinates.

5.4 Perspective PnP

Given image points $P_i$ , metric 3D points $\hat s\,\tilde P_i$ , and intrinsics $K$ :

\hat R,\hat t =\operatorname{PnP}\!\left(P_i,\hat s\,\tilde P_i,K\right).

RANSAC suppresses correspondence outliers. Compared with 3D–3D Procrustes, PnP fits RGB perspective observations without treating pseudo-depth as true scene coordinates.

6. Training and Engineering

Correspondence and scale offsets train jointly with supervised losses but explicitly separate branches.
The external pretrained 2.5D estimators’ domain robustness affects DMSR.
Queries still need masks / crops; open-world detection is not evaluated.
Computation includes Omnidata, PSPNet / ResNet18 features, prior interactions, and RANSAC-PnP.
Public code supports reproduction as an RGB-only category-level baseline.

7. Experimental analysis

7.1 Data and Metrics

CAMERA25: synthetic NOCS test set with 25k images.
REAL275: real NOCS test set with 2,750 images across six everyday categories.
Metrics: 3D IoU25 / 50 / 75 and $5^\circ2$ cm, $5^\circ5$ cm, $10^\circ2$ cm, $10^\circ5$ cm.

These metrics jointly depend on rotation, translation, and size. AP curves across error thresholds additionally show rotation / translation behavior more continuously.

DMSR Tables I–III: main results, modality and scale ablations — Tables I–III from the paper: Table I compares RGB category-level pose; Table II tests RGB / predicted-depth / normal combinations; Table III compares mean-scale and scale-offset recovery strategies. Source: paper PDF. Image copyright belongs to the authors and publisher; reproduced here for paper commentary.

7.2 Main Conclusions

Stronger rotation-sensitive CAMERA25 / REAL275 criteria than Synthesis, MSOS, and OLD-Net.
The paper summarizes approximately 50% lower REAL275 rotation error than MSOS with only approximately 5.1% of its parameter count.
Quantitative gains are more consistent on synthetic CAMERA25; REAL275 metric size is affected by category-instance distribution shifts.

7.3 Interpreting Ablations

Ablation	Claim tested	My assessment
RGB only → + depth / + normal	2.5D geometry improves correspondence	Supported, but external estimator cost should be reported
Depth only vs. depth + normal	Normals add local surface orientation	Particularly plausible for rotation
Mean scale → learned scale offset	Category statistics cannot cover all instance sizes	Supported
Joint scale vs. decoupled branch	Scale errors should not contaminate pose features	The central structural evidence

DMSR Figure 2: CAMERA25 and REAL275 qualitative results — Figure 2 from the paper: CAMERA25 above and REAL275 below; predicted boxes are red and ground truth green. Local translation / scale deviations remain in real scenes. Source: paper PDF. Image copyright belongs to the authors and publisher; reproduced here for paper commentary.

8. Critical analysis

Strengths

A structural treatment of monocular category-level scale ambiguity, beyond changing the backbone.
Restrained use of relative geometry: helps matching without presenting it as metric depth.
PnP explicitly restores the camera model to the pipeline.

Weaknesses and Implicit Assumptions

RGB-only still runs two pretrained 2.5D estimators, rather than relying on a single network.
Category priors, masks, and NOCS supervision are strong assumptions.
Mean-scale residuals assume similar training / test category-size distributions; REAL275 already reveals shifts.
Validation mainly covers six NOCS categories, without demonstrating long-tail extension.
Symmetry handling in training / evaluation does not provide separate multi-solution modeling.

9. Improvement Directions

Output pose / size distributions through metric-depth uncertainty rather than one scale offset.
Replace Omnidata with foundation monocular geometry and evaluate cross-domain calibration.
Propagate mask uncertainty into correspondence weights and PnP covariance.
Retrieve open-category shape priors to reduce fixed-mean-shape dependence.
Evaluate real-robot grasping / placement success beyond NOCS thresholds.

10. Research Implications

The reusable principle is to assign unreliable geometry only the roles it can support. Relative depth provides ordering and local shape, while metric scale needs independent statistics or measurements. Forcing both into pseudo-point clouds can increase rotation error.

11. Open questions

How do scale-offset errors quantitatively propagate into PnP translation?
Can DMSR preserve correspondence without category mean shapes?
How do Omnidata failures appear on transparent, reflective, or textureless objects?
How does DMSR compare with FAP-Net under identical detectors, masks, and training data?

Related papers

2026 · CVPR

3PT

3PT unifies detection, segmentation, and 6-DoF pose with two Transformers for 2D / 3D object-centric understanding. Continuous refinement of object representations and rendered-feature alignment support zero-shot cross-object perception on BOP-H3, BOP-Industrial, and BOP-Classic.

2026 · AAAI

6DAttack

Studies backdoor attacks on PnP-based and end-to-end 6-DoF pose pipelines. Structurally 3D triggers and modified pose labels are injected into training data so that models behave normally on clean inputs but produce controlled incorrect poses when triggered.

2026 · ICRA

Active Tactile Exploration

A violation-implicit loss incorporates physical-constraint violations into joint estimation of freely moving unknown convex rigid-body trajectories and cuboid / 20-vertex polytope shapes. Observed / Fisher information at a local MLE estimates EIG to select the next touch. In simulation, the EIG strategy yields stable errors below 4 mm after 5 cuboid actions; real Trifinger results at step 5 substantially outperform random selection. Convex geometry, local optima, multiple initializations, and tactile / proprioceptive errors limit the method.