Object Pose Estimation · 2024 · CVPR · CCF A

Confronting Ambiguity in 6D Object Pose Estimation via Score-Based Diffusion on SE(3)

The method learns p(R,t|I) directly, defining score diffusion on SE(3) rather than independently diffusing rotation and translation. A surrogate Stein score avoids expensive Jacobians. On T-LESS, the SE(3) model achieves MSPD / R@5 of 93.16 / 86.94. Sampling runs at 20 FPS with 100 steps and 250 FPS with 5 steps.

Paper

Research area

Object Pose Estimation

Task level

Instance-level

Object setting

Seen objects

Input modalities

single RGB object cropvisible-object segmentation mask in the T-LESS protocol

Requirements

ground-truth pose labels for trainingground-truth bounding boxes and visible masks in the T-LESS evaluationknown test-object identities represented during trainingiterative reverse diffusionno CAD model or symmetry annotations

Full paper metadata Authors, affiliations, research area, datasets, metrics, method tags, and reading status Expand

Authors: Tsu-Ching Hsiao, Hao-Wei Chen, Hsuan-Kung Yang, Chun-Yi Lee
Affiliations: Elsa Lab, National Tsing Hua University
Conference / journal: Conference · CVPR · CCF A
Research area: Object Pose Estimation
Code availability: Unavailable / not yet recorded
Reading status: Overview:Available Deep Dive:Available
Task setting: Instance-level · Seen objects
Datasets: SYMSOL SYMSOL-T T-LESS
Evaluation metrics: Rotation Error Translation Error MSPD MSSD VSD
Method tags: Instance-level Pose Estimation Absolute Pose Estimation rgb-only score-based-diffusion se3-lie-group pose-distribution multi-hypothesis symmetry-ambiguity occlusion-ambiguity surrogate-stein-score cad-free
Requirements: ground-truth pose labels for trainingground-truth bounding boxes and visible masks in the T-LESS evaluationknown test-object identities represented during trainingiterative reverse diffusionno CAD model or symmetry annotations

Reading Notes

Overview / Deep Dive

Available src/paper-notes/se3-diffusion-pose-cvpr2024/coarse.md

Paper overview notes

Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.

Overview Notes: SE(3) Score Diffusion

1. Basic information

Title: Confronting Ambiguity in 6D Object Pose Estimation via Score-Based Diffusion on SE(3)
Authors: Tsu-Ching Hsiao, Hao-Wei Chen, Hsuan-Kung Yang, Chun-Yi Lee
Affiliation: Elsa Lab, National Tsing Hua University
Year / venue: 2024 / CVPR
Paper: CVF publication page · published PDF
Reading mode: Overview

2. Positioning in one sentence

Rather than forcing single-image pose into one $R,t$ , this work learns a conditional score on the rigid-motion group $SE(3)$ and samples plausible poses through reverse diffusion. A surrogate Stein score addresses the expensive true score and deviating denoising direction caused by the $SE(3)$ Jacobian.

3. Guide to figures and tables

Conditional SE3 score-diffusion network and denoising paths — Original Figure 2 method diagram: RGB is encoded once into condition $c$; pose and noise time receive positional embeddings, and an MLP score network iteratively updates noisy poses in the Lie algebra. At right, more substeps bring curved-space denoising trajectories closer to the true inverse transform. Source: published CVPR 2024 paper. Image copyright belongs to the authors and IEEE/CVF; reproduced here for paper analysis.

SE3 diffusion results on SYMSOL, SYMSOL-T, and T-LESS — Original Tables 2–4: mean angular spread is 0.37° on SYMSOL; joint SE(3) substantially improves icosahedron rotation multimodality on SYMSOL-T; T-LESS MSPD / R@5 are 93.16 / 86.94. Source: published CVPR 2024 paper. Table copyright belongs to the authors and IEEE/CVF; reproduced here for paper analysis.

4. Problem setting

Input: one RGB object crop; the T-LESS protocol also supplies ground-truth bounding boxes and visible-object masks.
Output: pose samples from conditional distribution $p(X\mid I)$ , with $X=(R,T)\in SE(3)$ .
Supervision: RGB and GT poses, without 3D CAD or symmetry annotations.
Data: SYMSOL, the paper’s new SYMSOL-T, and 30 real T-LESS objects.
Goal: cover ambiguity from symmetry, occlusion, self-occlusion, and perspective, rather than average it into one pose.

5. Method overview

Perturb GT poses with a Gaussian perturbation kernel on the Lie group.
Extract image condition $c$ once through ResNet, avoiding image-feature recomputation at every step.
Feed noisy poses, noise levels, and image conditions to an MLP score network in the corresponding Lie algebra.
Gradually generate pose samples through reverse Geodesic Random Walk.
For $SO(3)$ and $\mathbb R^3SO(3)$ , the Stein score simplifies to $-z/\sigma^2$ . The same Jacobian symmetry does not hold on $SE(3)$ ; the authors use this quantity as a surrogate target and smaller substeps to approximate the inverse transform.

6. Experiments at a glance

Five-shape mean SYMSOL spread: ResNet34 0.42°, ResNet50 0.37°; Normalizing Flows 0.70°.
SYMSOL-T: $SE(3)$ rotation spreads over five shapes are 0.59, 0.58, 0.64, 0.54, 0.41°; $\mathbb R^3SO(3)$ scores 29.35° on the icosahedron.
T-LESS SE(3): MSPD 93.16, MSSD 60.17, VSD 56.88, R@2 47.21, R@5 86.94, R@10 94.78.
GDRNPP has better translation: T@2 90.31 versus 71.72, exposing weaker diffusion depth estimates without 3D geometry.
SE(3), 100 / 10 / 5 steps: 20 / 161 / 250 FPS; MSPD remains 92.40 at 5 steps.

7. Quick assessment

Highlight: expresses ambiguity through a pose posterior, beyond symmetry-aware single-valued losses.
Strongest evidence: T-LESS visualizations shift between discrete and continuous modes with visible parts, exceeding static symmetry annotations’ expressive power.
Limitations: real experiments use GT boxes / masks; T-LESS objects are known during training; translation trails geometry-based GDRNPP substantially.
Recommended for: Lie-group diffusion, pose uncertainty, symmetry, multiple 6DoF hypotheses, and score-based generative modeling.

Available src/paper-notes/se3-diffusion-pose-cvpr2024/deep.md

In-depth paper notes

Support reproduction, comparison, related-work writing, and identifying opportunities for further research.

Deep Reading Notes: Confronting Ambiguity via Score Diffusion on SE(3)

1. Key assessment

Single-RGB 6DoF pose is not always a function $I\mapsto(R,t)$ . Symmetric objects, occlusion, and self-occlusion can associate one image with several equivalent or nearly equivalent poses. Single-valued regression on this one-to-many problem tends to:

Average across modes;
Depend on manually specified symmetry annotations;
Ignore ambiguity induced dynamically by current visibility.

The paper directly learns:

X\sim p(X\mid I),\qquad X=(R,T)\in SE(3).

Beyond rotation distributions on $SO(3)$ alone, the method attempts joint rotation and translation. T-LESS shows strong rotation posteriors and visibility-conditioned symmetry, but translation trails CAD-based GDRNPP substantially. This is primarily research into ambiguity-aware pose posteriors, with a complete deployment pipeline still pending.

2. Why distinguish $SE(3)$ from $\mathbb R^3SO(3)$ ?

Both parameterizations contain $R,t$ , but their group operations differ.

Independent product:

(R_2,T_2)(R_1,T_1) =(R_2R_1,T_2+T_1).

Rigid-motion group:

(R_2,T_2)(R_1,T_1) =(R_2R_1,T_2+R_2T_1).

In $SE(3)$ , rotation acts on translation, matching rigid composition; $\mathbb R^3SO(3)$ diffuses them independently. Perspective couples rotation with image-space translation / depth, motivating the authors’ joint $SE(3)$ distribution.

SYMSOL-T specifically tests this: random translations are added to SYMSOL’s five symmetric shapes so viewpoint ambiguity involves $R,t$ jointly.

3. Lie-group score matching

On group $G$ , perturb GT pose $X$ with Gaussian Lie-algebra noise $z$ :

\tilde X=X\operatorname{Exp}(z), \qquad z\sim\mathcal N(0,\Sigma).

The network learns the perturbation distribution’s Stein score:

s_\theta(\tilde X,\sigma) \approx\nabla_{\tilde X}\log p_\sigma(\tilde X\mid X).

Reverse sampling uses Geodesic Random Walk:

\tilde X_{i+1} =\tilde X_i\operatorname{Exp} \left( \epsilon_i s_\theta(\tilde X_i,\sigma_i) +\sqrt{2\epsilon_i}z_i \right).

Exp / Log keep every update on the group, rather than adding to a 4×4 transform as an ordinary Euclidean vector.

4. Surrogate Stein score

On $SO(3)$ , the left / right Jacobians satisfy:

J_l(z)=J_r^\top(z),\qquad J_l^{-1}(z)=J_r^{-\top}(z),

Together with $J_l(z)z=z$ , this simplifies the score to:

\nabla_{\tilde X}\log p_\sigma(\tilde X\mid X) =-\frac{1}{\sigma^2}z.

$\mathbb R^3SO(3)$ supports this simplification independently for each component. However, the $SE(3)$ Jacobian has a rotation–translation coupling block $Q(\rho,\phi)$ and lacks the same symmetry. The true score vector need not follow the geodesic inverse direction from $\tilde X$ to $X$ .

Instead of an expensive Jacobian / autodiff score, the authors use:

\tilde s_X(\tilde X,\sigma) =-\frac{1}{\sigma^2}z

as a surrogate target. Large steps introduce curved-space approximation errors; subdividing them into smaller substeps progressively approaches the correct inverse transform. This engineering approximation does not make the true SE(3) score identical to the Euclidean Gaussian score.

Conditional score network and sub-step denoising on a Lie group — Original Figure 2 graphical excerpt: MLP blocks fuse image and time conditions; the SE(2) example at right shows surrogate directions approaching the true inverse with more integration substeps. Source: published CVPR 2024 paper. Image copyright belongs to the authors and IEEE/CVF; reproduced here for paper analysis.

5. Conditional network

The network separates:

Conditioning: ResNet encodes RGB once into $c$ ;
Denoising: pose / noise-level positional embeddings and multiple MLP blocks.
output: Lie algebra vector score.

Avoiding repeated image encoding supports 20–250 FPS. Ordinary scale-bias conditioning proved insufficient for periodic $SO(3)$ score fields, motivating Fourier-like conditioning:

f_i(x,c)= \sum_j W_{ij} \left[ A_j(c)\cos(\pi x_j) +B_j(c)\sin(\pi x_j) \right].

Periodic functions retain periodic derivatives, an inductive bias better suited to rotation manifolds.

6. SYMSOL and SYMSOL-T

SE3 score diffusion quantitative benchmark tables — Original Tables 2–4. SYMSOL evaluates rotation density; SYMSOL-T reports rotation spread and translation distance; T-LESS reports symmetry-aware BOP recalls and threshold accuracies. Source: published CVPR 2024 paper. Table copyright belongs to the authors and IEEE/CVF; reproduced here for paper analysis.

SYMSOL’s five shapes are tetrahedron, cube, icosahedron, cone, and cylinder: the first 3 have discrete symmetries and the last 2 continuous symmetries:

Method	Mean rotation spread
Implicit-PDF	3.96°
HyperPosePDF	1.94°
Normalizing Flows	0.70°
Proposed ResNet34	0.42°
Proposed ResNet50	0.37°

The icosahedron is particularly informative on SYMSOL-T:

$\mathbb R^3SO(3)$ : rotation 29.35°;
$SE(3)$ : rotation 0.64°.

Independent diffusion misses perspective-induced coupling, while the joint group model recovers multimodal rotations. Some independent-model translation errors are lower because it avoids interference between $R,t$ . The joint model therefore favors consistent rigid distributions rather than improving every individual metric.

7. Multimodal visualizations

SE3 diffusion samples on discrete and continuous symmetric SYMSOL-T shapes — Original Figure 3: 1,000 samples per plot. Tetrahedron, cube, and icosahedron form 12, 24, and 60 discrete modes; cone and cylinder follow one or two continuous symmetry orbits. Source: published CVPR 2024 paper. Image copyright belongs to the authors and IEEE/CVF; reproduced here for paper analysis.

This extends beyond Gaussian uncertainty: discrete symmetries form separated modes, while continuous symmetries form manifold bands. For pose systems, this better reflects the topology than a single covariance.

8. T-LESS: visibility-induced dynamic ambiguity

T-LESS visibility-conditioned pose distributions — Original Figure 4 graphical excerpt: discrete / continuous symmetries appear above. Below, object 4 changes from unimodal to bimodal or continuous posteriors depending on whether its top feature or bottom holes are visible. Source: published CVPR 2024 paper. Image copyright belongs to the authors and IEEE/CVF; reproduced here for paper analysis.

Object 4 has a manually annotated continuous symmetry, but visibility determines its actual posterior:

Visible top feature: unambiguous, one mode;
Occluded top with two visible bottom holes: two discrete modes;
Occluded holes too: approximately continuous symmetry.

Fixed object-level symmetry annotations cannot express these image-conditioned changes; the generative posterior is the paper’s strongest conceptual contribution.

Quantitative results:

Method	MSPD	MSSD	VSD	R@2	R@5	R@10	T@2
GDRNPP	90.17	75.06	67.60	21.60	71.18	90.56	90.31
$\mathbb R^3SO(3)$ diffusion	85.73	52.03	48.41	27.98	72.42	89.26	60.37
$SE(3)$ diffusion	93.16	60.17	56.88	47.21	86.94	94.78	71.72

SE(3) improves rotation markedly, while translation / surface metrics trail GDRNPP. The authors attribute GDRNPP’s better depth to 3D model geometry; this method uses no CAD.

9. Speed and step trade-offs

T-LESS, ResNet34, 224×224, RTX 2080 Ti:

steps	FPS	MSPD	MSSD	VSD
100	20	93.16	60.17	56.88
50	38	93.00	59.96	56.64
10	161	92.79	60.35	57.08
5	250	92.40	59.30	56.15

10 steps barely reduce accuracy, supporting substantial step reduction with the surrogate score and conditioning network. FPS per sample differs from FPS for a complete image posterior: drawing 1,000 samples to approximate the distribution still increases total cost linearly.

10. Experimental protocol boundaries

T-LESS uses:

30 objects known during training;
GT bounding boxes;
visible-part segmentation masks;
A single model across all 30 objects;
Strong image augmentation and Dynamic Zoom-In.

These tables exclude detection, segmentation, and novel-object generalization errors. The claim of avoiding 3D models / symmetry annotations holds, with more ideal frontend conditions than deployment.

11. Limitations and reproduction checks

Weak T-LESS translation: missing CAD / depth geometry remains a central bottleneck.
GT poses are required for training; this is supervised learning.
Calibration, NLL, and coverage are not evaluated; main results primarily measure sample accuracy.
Real experiments use GT boxes / masks; deployment needs a frontend and tests of posterior stability under segmentation noise.
Sample counts, final single-pose aggregation, and random seeds affect reported accuracy.
The surrogate score is approximate; empirically effective step reduction has no guarantee across all group distributions.
The main text provides no official code link; reproduce Lie-group conventions, Exp / Log, left / right multiplication, and unit scales carefully.

12. Overall assessment

The paper elevates symmetry from a static object property to image-conditioned posterior topology. T-LESS object 4’s visibility-dependent modes are more persuasive than a single AR value.

Rotation and ambiguity modeling are strong; translation, real frontends, and posterior calibration remain incomplete. As a methodological baseline for ambiguity-aware pose estimation, it suggests retaining multimodal samples for visibility checks, collision tests, or task costs before selecting one pose for reliable grasping.

Related papers

2026 · CVPR

3PT

3PT unifies detection, segmentation, and 6-DoF pose with two Transformers for 2D / 3D object-centric understanding. Continuous refinement of object representations and rendered-feature alignment support zero-shot cross-object perception on BOP-H3, BOP-Industrial, and BOP-Classic.

2026 · AAAI

6DAttack

Studies backdoor attacks on PnP-based and end-to-end 6-DoF pose pipelines. Structurally 3D triggers and modified pose labels are injected into training data so that models behave normally on clean inputs but produce controlled incorrect poses when triggered.

2026 · ICRA

Active Tactile Exploration

A violation-implicit loss incorporates physical-constraint violations into joint estimation of freely moving unknown convex rigid-body trajectories and cuboid / 20-vertex polytope shapes. Observed / Fisher information at a local MLE estimates EIG to select the next touch. In simulation, the EIG strategy yields stable errors below 4 mm after 5 cuboid actions; real Trifinger results at step 5 substantially outperform random selection. Convex geometry, local optima, multiple initializations, and tactile / proprioceptive errors limit the method.