Deep Reading Notes: dGeDi

1. Key assessment

dGeDi starts from a specific bottleneck: FreeZe’s GeDi geometric encoder is effective for zero-shot pose but builds a local reference frame for each query point and encodes neighborhoods individually, costing seconds per image. Rather than redesigning the pose estimator, the authors freeze GeDi as a teacher and distill its local descriptors into a PointTransformerV3 student that processes the full cloud in parallel.

The real challenge is that teacher descriptors are not reliable everywhere. On MegaPose-GSO, only approximately 21.4% of target points match in teacher-feature space; mean BOP ratio-of-nearest points is approximately 3%. Equal-weight MSE lets poorly discriminative teacher points dominate gradients. The paper combines geometric-correspondence target supervision with a low-error-prioritizing loss.

2. Where dGeDi Replaces GeDi in FreeZe

FreeZe takes RGB-D image $I$ and object CAD $O$:

1. SAM-6D segments the target crop.
2. Depth + intrinsics lift the crop into partial target cloud $P^T$.
3. CAD-surface sampling produces query cloud $P^Q$.
4. Geometric and visual encoders extract and concatenate features.
5. Feature nearest neighbors establish $P^Q\leftrightarrow P^T$ correspondences.
6. RANSAC / SVD estimates a rigid transform, followed by ICP refinement.
7. SAR performs symmetry-aware refinement.

dGeDi replaces only the GeDi geometric encoder in step 4. Visual foundation features, SAM-6D, RANSAC, ICP, and SAR remain; the 173× figure is not a 173-fold acceleration of the full FreeZe pipeline.

3. Object-oriented distillation

3.1 Teacher and Student

• Teacher $\Phi_\Theta$: GeDi, a pointwise local encoder with explicit local reference frames for rotation invariance.
• Student $\Psi_\Omega$: PTV3 with voxel / serialized point processing and decoder channels reduced from 64 to 32.
• Output: a 32D descriptor per point.
• The teacher is frozen; the student trains with AdamW and a cosine scheduler.

Input clouds are randomly sampled to 4,000 points. Targets first undergo statistical outlier removal, then scale normalization by query-object diameter. At 5K evaluation points, dGeDi peak memory is 717 MB.

3.2 Learning via correspondences

Directly computing $F^Q=\Phi(P^Q)$ and $F^T=\Phi(P^T)$ has two problems:

• $P^Q$ comes from clean CAD, while $P^T$ comes from partial noisy depth, creating a teacher-feature domain gap.
• Approximately 850 MegaPose query objects correspond to approximately 16M target observations; precomputing / caching all target teacher features would take approximately 14 TB.

Training samples already have ground-truth $T_{Q\to T}$. Transform query points into the target frame, then find nearest query points by Euclidean distance:

$$\Gamma(p)=\arg\min_{q\in P^Q}\lVert T_{Q\to T}q-p\rVert .$$

Then use $\tilde F^T=\Gamma(F^Q)$ to supervise $G^T=\Psi(P^T)$. The target student learns clean descriptors from corresponding CAD-surface points instead of unreliable teacher outputs on noisy targets. Caching only query teacher features reduces estimated storage from 14 TB→34 GB, approximately 411×.

This assumes aligned ground-truth poses, CAD, depth, and masks. Background or occluder target points would still be forced to take nearest-CAD labels. Training therefore uses ground-truth masks, with a domain gap when SAM-6D masks are deployed.

3.3 Loss Prioritizing Low Errors

Finding that most teacher descriptors lack distinctiveness, the authors propose:

$$\mathcal{L}(F,G)= \begin{cases} \alpha\lVert F-G\rVert^2,& \lVert F-G\rVert<\delta\\ \beta\lVert F-G\rVert+\gamma,& \text{otherwise} \end{cases}$$

Here $\alpha=8,\beta=1/5,\delta=0.3,\gamma=8\delta^2-\delta/5$. A steep quadratic low-error region encourages precise fitting of reliable features the student can learn; the high-error region approaches a plateau, limiting weights on ambiguous / noisy teacher points.

A tested focal-like variant $\tilde{\mathcal L}=\lVert F-G\rVert^\epsilon\mathcal L$, with $\epsilon=15$, instead emphasizes large errors and performs substantially worse. This supports the authors’ view that not all teacher knowledge deserves copying.

4. Data, Metrics, and Runtime Environment

• Training: MegaPose-ShapeNetCore, 1M synthetic images, 8×NVIDIA A100.
• Evaluation: LM-O, T-LESS, TUD-L, IC-BIN, and YCB-V on one NVIDIA A40.
• Segmentation: both ground-truth masks and SAM-6D zero-shot masks.
• RANSAC: 100K iterations; ICP: 1K iterations;
• Descriptors: 32D; GeDi neighborhood radius is 30% of query diameter.

Metrics:

• RON: proportion of query points whose feature nearest neighbor falls within a true geometric-distance threshold of 3% of object diameter.
• FMR: successful-matching rate when RON exceeds 5%.
• BOP AR: pose accuracy after registration.
• Runtime: per-image geometric-feature extraction or full-pipeline time; check the relevant table column.

RON / FMR directly evaluate descriptors. AR also depends on RANSAC, ICP, masks, and symmetry refinement, so improved features need not improve AR on every dataset.

5. Direct Teacher–Student Comparison

Mean results:

Mask	Encoder	RON	FMR	BOP AR	Feature time
GT	GeDi	3.0	24.5	62.9	3.284 s
GT	dGeDi	7.1	36.3	66.8	0.019 s
SAM-6D	GeDi	2.9	18.5	55.1	3.284 s
SAM-6D	dGeDi	5.6	27.9	56.8	0.019 s

The table’s 0.019 s versus 3.284 s corresponds to 173×. The main text also describes over 50 FPS, or approximately 260 point clouds/s with approximately 5 target objects per image. Target counts vary across datasets: dGeDi takes 3 ms on TUD-L and 34 ms on IC-BIN.

Points to note:

• T-LESS ground-truth-mask AR improves 41.6→64.5, a +22.9 gain. The authors attribute this to sharp industrial geometry benefiting from smooth, discriminative student features.
• TUD-L / YCB-V ground-truth-mask AR instead falls from 96.8 / 59.1 to 93.2 / 58.8.
• Student superiority therefore holds for average feature metrics, not pose AR on every dataset.

6. Ablations and the Complete System

6.1 Distillation Ablations

The query-supervision-only + Eq.1 baseline without augmentation averages AR 60.8.

• Add target-correspondence supervision + Eq.1: 65.5, a +4.7 gain.
• MSE / Huber: 62.1 / 62.4;
• High-error-emphasizing Eq.2: 57.0, below baseline by 3.8.
• Target supervision + Eq.1 + rotation / jitter / point dropout: final 66.8, a +6.0 gain.

Evidence supports three design choices: partial-target supervision, suppression of high-error teacher features, and geometry / missing-point augmentation.

6.2 End-to-End FreeZe

Setting	FreeZe / dGeDi	Avg AR	Avg runtime
SAM-6D mask + SAR	FreeZe	71.5	13.7 s
SAM-6D mask + SAR	dGeDi	70.4	8.5 s
GT mask, no SAR	FreeZe	79.6	8.7 s
GT mask, no SAR	dGeDi	79.6	6.0 s

Under deployment conditions, dGeDi cuts end-to-end time by 5.2 s but loses 1.1 AR. With ground-truth masks, AR stays equal and runtime falls by 2.7 s. Table values of FoundationPose 74.4 AR / 35.9 s use A100, whereas SAM-6D 71.0 / 4.5 s uses RTX3090. Different GPUs and implementations prevent a strictly hardware-matched runtime ranking.

7. Qualitative Results

The authors explain that GeDi’s local reference frames can fail when too little structure is visible, whereas the feed-forward student produces smoother descriptors. All illustrated dGeDi RON values exceed GeDi, with more plausible poses. These are author-selected successes without accompanying failures. Unique poses recovered from tiny asymmetries also need checking against benchmark symmetry equivalence.

8. Limitations and Reproduction Risks

The authors explicitly state that dGeDi optimizes only geometric-feature extraction; registration remains the system bottleneck.

Additional observations from this site:

• Training requires ground-truth poses and masks. Zero-shot refers to unseen test objects, not unsupervised training.
• SAM-6D mask noise lowers full-system AR, showing sensitivity to segmentation domain shifts.
• The 14 TB / 34 GB storage estimates depend on the authors’ data scale and feature layout; verify them with actual formats.
• Training on 8×A100 is not lightweight; the deployment encoder is.
• 100K RANSAC + 1K ICP iterations occupy substantial end-to-end time.
• Descriptors are validated only in the RGB-D FreeZe pipeline, without tests on pure point-cloud registration, events, or RGB-only input.
• Parameter count, training duration, energy use, and low-end edge-GPU latency are not reported.

9. Reproduction Checklist

• Fix PTV3 voxel / serialization settings, the 32D decoder, and point count.
• Reproduce query-diameter normalization and target statistical outlier removal.
• Check the direction of $T_{Q\to T}$, nearest-neighbor mapping $\Gamma$, and target visibility.
• Record cached teacher-query feature dtype, compression, and the 34 GB estimate.
• Implement the piecewise loss’s continuity constant $\gamma$ and $\delta=0.3$.
• Match rotation, jitter, and point-dropout magnitudes.
• Time the geometric encoder, visual encoder, matching, RANSAC, ICP, and SAR separately.
• Report both ground-truth and predicted masks, avoiding concealment of front-end domain gaps in mean AR.

10. Overall assessment

dGeDi clearly defines a local-module problem and provides detailed engineering measurements. It precisely addresses slow GeDi inference and unreliable teacher descriptors without claiming to redesign the full zero-shot pose pipeline. The 173× gain belongs to geometric-feature extraction; full FreeZe changes from 13.7→8.5 s with a slight AR reduction. With those scales distinguished, dGeDi remains a valuable example of efficient 3D-descriptor distillation.