Paper Reading Notes (Deep): FreeZe

1. Key assessment

FreeZe’s central idea is a 3D point descriptor that jointly encodes geometry and appearance, beyond simply using DINOv2 for pose:

• GeDi handles textureless surfaces, occlusion, and local shape;
• DINOv2 distinguishes geometrically similar surfaces with different textures;
• Both are lifted to / defined on 3D points, and the final solution still uses 3D–3D correspondences and rigid registration.

This is more interpretable than directly regressing SE(3) from foundation features and requires no pose-specific training. Ablations show 56.5 AR for geometry alone and at best 39.6 for vision alone, compared with 64.7 after fusion. Vision thus supplies discriminative evidence on top of the geometric backbone rather than replacing it.

“Training-free” also needs precision: FreeZe is not trained on BOP / MegaPose synthetic data, but GeDi and DINOv2 are frozen pretrained models, and front-end CNOS / SAM6D have their own training and object onboarding. It is free of task-specific pose training, rather than free of learned components.

2. Inputs and System Boundaries

Input:

• RGB-D image $I$ and intrinsics;
• A query object 3D model $Q$;
• Instance-mask candidates from CNOS or SAM6D.

The output is model-to-camera $T=(R,t)$. Sample $N=5K$ points from the query surface and $M=1K$ points from the target masked cloud.

GeDi can still operate when CAD provides only geometry, while the visual query branch needs renderable texture. Ordinary BOP objects are mostly textured meshes reconstructed from RGB-D; industrial CAD in T-LESS / ITODD relies mainly on geometry.

3. Symmetric Query / Target Feature Construction

3.1 Query object

1. Sample $P^Q$ from the CAD surface;
2. Frozen GeDi extracts 32D local descriptors at radii 0.3 / 0.4 of object diameter, concatenated into 64D;
3. Render RGB-D from multiple views;
4. DINOv2 ViT-G outputs patch features from RGB crops, converted to pixel features by bilinear interpolation;
5. Rendered depth back-projects pixel features to 3D;
6. Average features from multiple views for the same query point.

3.2 Target observation

Mask + depth + intrinsics directly back-project into $P^T$. GeDi processes the point cloud; DINOv2 processes the RGB crop. One-to-one RGB and depth pixel correspondences directly bind foreground pixel features to target points.

3.3 Fusion

L2-normalize geometry and vision separately, then concatenate:

$$f=\left[\frac{g}{\|g\|_2}\,\middle|\,\frac{v}{\|v\|_2}\right].$$

Each feature is 64D, yielding a 128D fused descriptor. Independent normalization prevents the raw scale of 1,536D DINOv2 features, or one modality’s norm, from dominating. PCA reduces vision to 64D and balances its dimensionality with geometry.

4. 3D Correspondences, RANSAC, and Refinement

RANSAC samples query-point triplets, finds target nearest neighbors in fused feature space, rejects inconsistent matches, and estimates $P^QR+t\approx P^T$. Estimate separately for multiple mask candidates and select the registration with the most inliers.

ICP then refines point-level geometry. Every stage allows inspection of correspondences / inliers / residuals, but speed and robustness depend on RANSAC iterations, overlap, mask outliers, and the ICP convergence basin.

The conclusion also identifies simple RANSAC as a direction for improvement. However, runtime, RANSAC iterations, and standardized hardware profiling are absent, leaving the engineering cost unclear.

5. Symmetry-Aware Refinement

SAR has two steps:

1. Uniformly sample rotations in SO(3), identifying geometry-only symmetries through transforms with small $CD(R_sP^Q,P^Q)$;
2. Compose each $R_s$ with the ICP pose and render it. Extract DINOv2 patch features, average cosine similarity against the input crop’s patch features, and select the highest-scoring candidate.

This specifically addresses objects whose geometry is equivalent under a transform but whose texture is not. If both physical shape and texture are symmetric, candidates remain indistinguishable. If the coarse-pose error is unrelated to symmetry, SAR also cannot replace general refinement.

6. Main BOP Results

The metric is BOP $AR=(AR_{\mathrm{VSD}}+AR_{\mathrm{MSSD}}+AR_{\mathrm{MSPD}})/3$.

Prior	Method	Refinement	Mean AR
CNOS	FoundPose RGB	None	37.3
CNOS	SAM6D RGB-D	None	55.4
CNOS	FreeZe RGB-D	None	64.0
CNOS	SAM6D RGB-D	Yes	63.2
CNOS	FreeZe RGB-D	ICP + SAR	69.3
SAM6D	SAM6D RGB-D	None	60.2
SAM6D	FreeZe RGB-D	None	65.7
SAM6D	SAM6D RGB-D	Yes	68.2
SAM6D	FreeZe RGB-D	Yes	70.9

FreeZe coarse at 64.0 already exceeds CNOS + refined SAM6D at 63.2, although their pipelines are not identical. TUD-L is especially strong: CNOS + FreeZe coarse reaches 86.1 and refined reaches 93.6. The authors attribute this to strong geometric features under difficult lighting.

The mask prior itself has a clear effect: FreeZe coarse changes from 64.0→65.7 and refined from 69.3→70.9. Comparing only pose backends while ignoring localization quality is insufficient.

7. Ablations: Geometry as the Backbone, Vision as Added Evidence

LM-O geometry-only:

• FPFH 20.7;
• FCGF 20.8;
• GeDi radius 0.3: 55.3;
• MS-GeDi (0.3, 0.4): 56.5.

vision-only:

• raw RGB 15.1;
• CLIP ViT-L 30.5;
• DINOv2 ViT-L 36.8;
• DINOv2 ViT-G 39.6.

Fused MS-GeDi + ViT-G reaches 64.7. Geometry exceeds vision alone by 16.9 points, and fusion adds 8.2 beyond geometry, supporting the interpretation of a geometric backbone with visual disambiguation.

Box / mask priors both yield 64.7 at the coarse stage; after ICP, box reaches 68.6 and mask 69.0. Masks are cleaner, but missing object parts can also harm correspondences; ICP is more sensitive to background outliers in boxes.

refinement:

• none: 64.0;
• +ICP: 68.8;
• +SAR: 69.3.

SAR adds only 0.5 on average, with gains of +2.6 on IC-BIN, +0.6 on HB, and +0.4 on YCB-V, concentrated in data with geometric symmetry and discriminative textures.

8. Failure Modes and Limitations

Figure 5 shows four types of failure:

• Under occlusion, only the nearest corner aligns, leaving the overall rotation roughly perpendicular to the correct orientation;
• Simple box geometry and weak texture cause confusion between lying on the side and standing upright;
• The mug body aligns but the small handle does not;
• The drill handle is completely occluded, leaving only the visible body to align.

System limitations:

• Depth is required; sensor noise / transparent or reflective objects directly contaminate 3D;
• GeDi pointwise local features and ViT-G are heavy; the authors also identify difficulties with edge deployment;
• Runtime, GPU memory, and parameter count are not reported;
• RANSAC / ICP remain bottlenecks and depend on overlap;
• RGB-only operation is unavailable; predicting geometric features from RGB is proposed as future work;
• Differences in query texture / rendered lighting from real images affect visual features;
• CNOS / SAM6D candidate calibration is unstable, requiring multiple masks and additional computation;
• SAR estimates symmetries by discrete sampling and a Chamfer threshold, without sufficient analysis of accuracy or candidate count.

9. Reproduction Checklist

• query 5K points, target 1K points;
• GeDi radii = 0.3d / 0.4d, each 32D;
• DINOv2 ViT-G, 224×224, 1,536D→PCA 64D;
• Aggregate query multiview visual features by 3D point;
• L2-normalize the two modalities separately before concatenation;
• Preserve both box / mask priors and multiple candidates;
• Fix feature nearest neighbors, triplet sampling, inlier pruning, and RANSAC parameters;
• Evaluate coarse, ICP, and SAR separately;
• Compare baselines using the same CNOS / SAM6D masks;
• Add stage-wise runtime and memory for feature extraction, matching, RANSAC, ICP, and SAR;
• Stratify by missing depth, visible ratio, small-part area, and symmetry type.

10. Overall assessment

FreeZe is a clearly structured RGB-D zero-shot pose method: it trains no new pose network, instead placing established pretrained geometric / visual features on 3D points and solving SE(3) through classical registration. Across seven BOP datasets, its 64.0 coarse, 69.3 CNOS-refined, and 70.9 SAM6D-refined scores show that strong features plus explicit geometry could outperform large-scale pose-specific training in 2024.

It is no lightweight universal solution: depth, CAD, masks, GeDi, ViT-G, RANSAC, and ICP are all required, with no reported runtime. With these dependencies understood, FreeZe remains a representative example of how foundation models and 3D geometry complement each other; subsequent dGeDi work specifically improves its slow geometric encoder.