Paper Reading Notes (Deep): GigaPose

1. Key assessment

GigaPose’s design is simple: do not discretize every rotational degree of freedom with templates.

Sampling azimuth, elevation, and in-plane rotation rapidly expands conventional template banks. GigaPose renders only 162 out-of-plane views:

• Retrieval determines azimuth / elevation;
• One local 2D–2D correspondence supplies scale, in-plane rotation, and 2D translation;
• Scale + intrinsics recover depth / 3D translation;
• RANSAC selects among one-correspondence candidates.

Precomputed dense features reduce MegaPose’s query–template scoring from 1.68 s to 48 ms / detection. Seven-dataset coarse mean AR rises from 20.8 to 26.8, retaining better initialization with matched refiners.

One correspondence does not guarantee a stable final pose alone: each match generates a candidate, selected by multicandidate RANSAC. The title refers to information sufficient to define one complete candidate.

2. Inputs, Outputs, and System Boundaries

Input:

• RGB image;
• CNOS object masks / detections;
• Intrinsics $K$;
• New-object CAD at test time;
• Templates / features precomputed at onboarding.

Outputs:

• Coarse $R,t$;
• Optional top-1 / top-5 hypotheses;
• Optional MegaPose or GenFlow refinement.

Training excludes test objects. Approximately 2M BlenderProc renderings of Google Scanned Objects / ShapeNet train shared weights, without updates for BOP test objects.

Novel means unseen instances, while CAD identities remain known and CNOS supplies discovery / segmentation. This is not open-vocabulary, model-free, or category-level pose.

3. Decomposing Degrees of Freedom

Full 6DoF pose is reparameterized as:

• out-of-plane rotation: azimuth, elevation;
• In-plane rotation $\alpha$;
• Image-plane translation $t_x,t_y$;
• Metric scale $s$, converted to depth through template geometry / intrinsics.

After nearest-template retrieval, out-of-plane rotation is known. For template patch $i$ and query patch $i'$:

• Their positions yield image translation;
• Both $F_{\mathrm{ist}}$ features enter scale and angle MLPs;
• Form affine transform $M_{t\rightarrow q}$;
• Compose with template pose into a full 6D candidate.

Where standard 3D–2D PnP needs multiple correspondences, known template viewpoint and learned scale / angle priors let one 2D–2D match define a candidate.

4. Why Separate Feature Sets?

Retrieval features $F_{\mathrm{ae}}$ should:

• Ignore scale, in-plane rotation, and 2D translation;
• Retain out-of-plane sensitivity;
• Match across synthetic templates / real queries.

In contrast, $F_{\mathrm{ist}}$ must retain:

• in-plane rotation;
• scale.

Fully invariant features would discard the required scale / angle information. Separate representations handle retrieval invariance and remaining-DoF regression.

5. Local Contrastive Learning for $F_{\mathrm{ae}}$

Training pairs render the same CAD with:

• Identical out-of-plane rotation;
• Different scale, in-plane angle, and translation.

Known render geometry establishes GT 2D–2D matches. Positives have:

• The same object;
• The same surface patch;
• Allowed scale / in-plane / translation changes.

Negatives include:

• Incorrect positions on the same object;
• Different out-of-plane rotations;
• Other objects;
• Other batch templates.

Local contrastive learning brings matching patches together and separates others. Training additionally uses:

• color augmentation;
• random crop;
• random in-plane rotation.

These reduce synthetic-to-real gaps and teach $F_{\mathrm{ae}}$ invariance. The backbone is DINOv2 ViT, not frozen off-the-shelf DINOv2: Table 4 gives mean AR 19.0 without fine-tuning versus 27.9 full, a gain of 8.9.

6. Template Count and Onboarding

162 viewpoints come from a regular icosphere, subdividing Blender triangles into four and covering camera direction / out-of-plane rotation.

For each new object:

1. Render 162 templates;
2. Extract and store $F_{\mathrm{ae}}$ dense features;
3. Build a nearest-neighbor index.

Onboarding times:

• GigaPose: 11.5 s / object;
• MegaPose: 0.82 s / object.

More computation moves to onboarding for repeated-inference speed. The 11.5 s cost amortizes best when the same object is used repeatedly.

The 576-template ablation gives:

• 162 templates mean AR: 27.9;
• 576 templates: 28.5;
• Only 0.6 improvement.

This supports one-correspondence recovery reducing dense template requirements for remaining degrees of freedom.

7. Main BOP Results

Across seven datasets:

• 132 objects;
• 19,048 test instances;
• Mixed clutter, occlusion, texturelessness, and industrial objects;
• Standard BOP AR averages threshold recalls for VSD, MSSD, MSPD.

Without refinement:

Method	LM-O	T-LESS	TUD-L	IC-BIN	ITODD	HB	YCB-V	Mean	time
MegaPose + CNOS	22.9	17.7	25.8	15.2	10.8	25.1	28.1	20.8	15.5 s
GigaPose + CNOS	29.6	26.4	30.0	22.3	17.5	34.1	27.8	26.8	0.4 s

GigaPose trails only YCB-V by 0.3 and improves every other dataset. T-LESS, IC-BIN, ITODD gains exceed 5 points, supporting textureless / industrial robustness.

With different refinements:

Initialization + refiner	Mean AR	time
MegaPose init + MegaPose 1 hyp	50.9	17.0 s
GigaPose init + MegaPose 1 hyp	54.7	2.3 s
MegaPose init + MegaPose 5 hyp	54.7	21.9 s
GigaPose init + MegaPose 5 hyp	57.8	7.7 s
MegaPose init + GenFlow 5 hyp	57.0	20.8 s
GigaPose init + GenFlow 5 hyp	60.5	10.6 s

Better coarse pose also helps the same refiner avoid local minima.

8. Qualitative Results, Predicted Models, and Speed

Per detection on V100:

Stage	MegaPose	GigaPose
Onboarding / object	0.82 s	11.5 s
Coarse pose / detection	1.68 s	48 ms
MegaPose refinement	33 ms	33 ms

The paper reports approximately 35× speedup, consistent with 1.68 / 0.048 = 35×, while elsewhere saying more than 38×, possibly from unrounded timings. This site consistently uses approximately 35×.

Table 1’s 0.4 s is mean per image including CNOS and every detection, unlike 48 ms / detection.

Wonder3D predicted model, LM-O:

3D source	Method	Coarse	Refined
GT CAD	MegaPose	16.3	25.6
GT CAD	GigaPose	18.3	27.9
single-image model	MegaPose	15.4	25.2
single-image model	GigaPose	17.6	27.2

Wonder3D tests use carefully selected reference images because reconstruction is input-sensitive. They show potential, not general removal of CAD dependencies.

9. Segmentation Robustness and Ablations

On T-LESS / YCB-V, AP varies less with mask IoU than MegaPose, which mainly scores well at high IoU. Reasons include:

• Dense-patch retrieval avoids compressing the whole crop into one score;
• RANSAC ignores some boundary / background outliers;
• Precomputed template features avoid joint forwards for every noisy crop / template pair.

LM-O improves less because:

• Objects are small in images;
• heavy occlusion;
• Resized masks leave few usable patches;
• Some CAD has low fidelity.

Table 4:

Setting	LM-O	T-LESS	YCB-V	Mean
Without fine-tuning $F_{\mathrm{ae}}$	20.1	19.3	17.7	19.0
36-bin in-plane templates, 5832 total	23.3	21.1	22.1	22.1
3D–2D ePnP, 4 correspondences	28.0	25.3	26.3	26.5
2D–2D Kabsch, 2 correspondences	30.0	25.6	26.0	27.2
one-correspondence full	29.6	26.4	27.8	27.9
full + 576 templates	30.1	27.1	28.4	28.5

Key conclusions:

• Local-feature fine-tuning is among the largest gains;
• Sampling in-plane rotation in templates creates 5832 templates yet trails learned one-correspondence recovery by 5.8 mean;
• Standard 3D–2D ePnP does not beat task-specific 2D–2D parameterization;
• One correspondence gives the best mean, while two are slightly better on LM-O.

10. Geometry and Learning in One Correspondence

The system is not purely analytic:

• Discrete templates supply out-of-plane rotation;
• MLPs learn scale / in-plane rotation;
• Patch coordinates provide 2D translation;
• CAD geometry and intrinsics provide depth / 3D translation;
• RANSAC uses multicandidate consensus.

Features, template cameras, and learned MLPs must work consistently. Changing intrinsics, render conventions, or CAD scale requires checking affine-to-3D conversion.

11. Related Methods

MegaPose:

• Coarse scoring runs per query–template pair;
• More templates and slower runtime;
• Strong refinement, expensive initialization.

GigaPose:

• Precomputed template features;
• nearest-neighbor sublinear retrieval;
• 162 templates;
• Faster coarse estimates feeding the same MegaPose refiner.

ZS6D:

• Also uses DINOv2 and correspondences;
• Favors 3D–2D correspondences + PnP;
• GigaPose experiments favor hybrid learned 2D scale / angle under this protocol.

FoundationPose:

• Unifies CAD / model-free pose and tracking;
• Heavier render-and-compare refinement;
• GigaPose mainly improves CAD-based novel-object initialization and can front other systems.

12. Limitations

• Test-time CAD remains the main setting;
• Predicted models require selected reference images;
• CNOS masks / detections are mandatory frontend inputs;
• Small objects, heavy occlusion, or poor CAD can break retrieval and matching together;
• 162-template quantization limits out-of-plane initialization;
• Onboarding at 11.5 s is slower than MegaPose;
• Coarse mean AR 26.8 cannot replace precise refinement;
• Historical V100 timings do not directly compare with current hardware;
• Training requires 2M synthetic images and BOP/MegaPose data;
• One-correspondence MLPs may suffer camera / rendering shifts;
• BOP metrics primarily handle symmetry; the network outputs no pose distribution;
• Multicandidate RANSAC leaves worst-case runtime dependent on correspondence count.

13. Reproduction Checklist

• Use the same BOP synthetic / MegaPose training sets;
• Match CAD units, origin, and camera conventions to BOP;
• Render exactly 162 icosphere views;
• Compute template features only once at onboarding;
• Keep $F_{\mathrm{ae}}$ and $F_{\mathrm{ist}}$ distinct;
• Derive positive / negative patches from actual render geometry;
• Include color, crop, and in-plane augmentation;
• Implement nearest-neighbor template retrieval;
• Use separate scale / angle MLPs;
• Verify 2D translation / scale to 3D translation and intrinsics;
• Generate multicorrespondence candidates before RANSAC;
• Match CNOS detections against MegaPose;
• Report coarse, single-hypothesis, and five-hypothesis results separately;
• Separate onboarding / object, coarse / detection, and whole-image times;
• Report mask-IoU, small-object, and CAD-fidelity failure curves.

14. Overall assessment

GigaPose combines engineering practicality with clear geometry. It redesigns pose parameterization rather than simply enlarging backbones: 162 templates cover out-of-plane rotation, while local correspondence recovers remaining freedom. Table 4 strongly supports this through dense in-plane, ePnP, two-correspondence, and template-count ablations.

CAD, masks, and refinement remain necessary; coarse 26.8 AR shows speed is not final precision. Yet matched CNOS / refiners consistently receive faster, better initialization. For repeated inference on CAD objects unseen during training, decomposing template retrieval and local correspondences by degrees of freedom reduces coarse estimation from seconds to tens of milliseconds and improves attainable refinement accuracy.