3D Representations / Reconstruction · 2025 · ICRA · CCF B
OmniShape: Zero-Shot Multi-Hypothesis Shape and Pose Estimation in the Real World
OmniShape uses two conditional diffusion models to sample partial NORF point clouds and triplanar shape completions, respectively. Depth registration then places multiple pose–shape hypotheses into the real scene.
Two conditional diffusion models first sample camera-view partial NORF clouds, then complete triplanar neural fields. Each sample is a possible shape–pose hypothesis, placed into real scenes through optional depth / normal registration.
2.1 Guide to the paper’s figures and tables
Original Figure 1: an occluded image admits multiple plausible complete shapes. OmniShape samples alternatives rather than forcing one deterministic answer. Source: authors' public PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for academic analysis.
3. Method pipeline
Predict visible-surface partial NORF clouds from one RGB image and object crop.
First-stage diffusion samples local geometry and pose uncertainty.
Second-stage diffusion generates triplanar fields conditioned on partial observations.
Extract complete meshes / clouds.
Register metric geometry using predicted normals or optional depth.
Repeat for multiple hypotheses and select through observation-inlier rules or similar criteria.
NORF represents points in a camera / observation-related normalized frame, coupling pose and shape rather than treating them as isolated tasks.
4. Results at a glance
Real data with RGB inlier selection:
Dataset
Chamfer ↓
F1 ↑
TYO-L
8.81 ± 4.04
0.54 ± 0.11
NOCS
10.67 ± 1.22
0.42 ± 0.04
HOPE
10.36 ± 7.64
0.54 ± 0.07
Best-of-10 is better, e.g. TYO-L 5.76 / 0.65, but this is an oracle upper bound rather than reliably automatic selection.
5. Coarse-reading conclusion
OmniShape acknowledges ambiguous single-image completion and samples shape / pose jointly. It suits novel-object modeling without target CAD; unique, real-time, millimeter-level pose needs stronger ranking and metric registration.
Available src/paper-notes/omnishape-icra2025/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
Paper Reading Notes (Deep): OmniShape: Zero-Shot Multi-Hypothesis Shape and Pose Estimation in the Real World
1. Task boundaries
Beyond known-CAD pose, one RGB image jointly predicts:
Visible surfaces.
Complete occluded 3D shape.
Camera-relative shape pose.
Multiple possible hypotheses.
This is more underconstrained than model-based pose: unseen backs allow different complete shapes with similar projections.
Original Figure 2: partial NORF generation precedes triplanar completion; optional depth / normal registration aligns candidates to real camera coordinates. Source: authors' public PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for academic analysis.
2. Why two stages?
Direct RGB-to-complete-shape generation mixes:
Visible-surface locations and camera viewpoint.
Completion of unseen geometry.
Partial NORF followed by complete fields separates observation explanation from shape priors. Both stages support repeated sampling: each partial hypothesis can yield multiple completions.
3. NORF and pose
NORF can be understood as an observation-related normalized object reference frame for novel objects. Predicted points carry geometry and canonical relations for pose recovery. Compared with conventional NOCS, OmniShape emphasizes:
Independence from a few predefined categories.
Multiple hypotheses.
Joint generative shape completion.
Final metric pose still needs intrinsics and depth / normal registration constraints. A generated canonical shape does not resolve RGB-only scale / translation automatically.
4. Multiple hypotheses and selection
For N candidates {Hi}, the paper reports:
Single / inlier-selected: choices available to actual systems.
Best-of-N: oracle upper bounds with retrospective GT knowledge.
Their gap measures ranking difficulty rather than whether good hypotheses can be generated.
Original Figure 3 and Table II: diverse Ocrtoc3D / Pix3D completions above, and first / inlier / best-of-10 real-scene metrics for TYO-L, NOCS, and HOPE centrally. The bottom partially shows candidate-count / Chamfer curves and real-scene visualizations. Source: authors' public PDF. Image copyright belongs to the paper's authors and publisher; reproduced here for academic analysis.
Ocrtoc3D best-of-25 gives about 0.286 F-score / 0.230 Chamfer; Pix3D about 0.257 / 0.272. Similar sampling configurations give close results, suggesting diminishing returns from more samples.
5. Real-scene results
Method / selection
TYO-L CD / F1
NOCS CD / F1
HOPE CD / F1
ZeroShape
13.35 / 0.31
14.57 / 0.30
9.94 / 0.49
OmniShape RGB inlier
8.81 / 0.54
10.67 / 0.42
10.36 / 0.54
OmniShape best-of-10
5.76 / 0.65
7.73 / 0.51
7.04 / 0.64
On HOPE, inlier Chamfer slightly trails ZeroShape but F1 is higher, so one aggregate cannot fully describe local alignment and full-shape coverage.
Best-of-10 consistently beats automatic inlier selection. Good candidates exist, but selectors underuse them; visible-surface fit can favor incorrect complete shapes.
6. Speed and applications
The authors report about 0.9 s per hypothesis on A6000. Multiple hypotheses add roughly linear cost, favoring:
Offline asset creation.
Scene modeling before manipulation.
Candidate generation for a few objects.
Rather than inherently supporting high-frequency closed-loop tracking.
7. Limitations
Limited fine geometry, thin structures, and topological consistency.
Discontinuous or noisy generated surfaces.
OOD normal / depth noise affects registration.
Best-of-N accuracy is unavailable directly at deployment.
Probability calibration and task-specific ranking remain unresolved.
8. Deep-reading conclusion
OmniShape makes uncertainty explicit: unseen backs call for multiple plausible shape–pose solutions, rather than deterministic average shapes. Evidence shifts the bottleneck from generation to selection: selecting good candidates without GT determines real performance.
