Paper Reading Notes (Deep): GeoReF

1. Key assessment

GeoReF starts from a CATRE failure: category-level refinement lacks the exact target CAD available to instance-level ICP. An observed narrow-mouth cup may differ from a wide-mouth mean prior. Separate PointNet encoding followed by late concatenation struggles to align shapes and can mistake prior mismatch for pose error.

The solution has three levels:

1. HS layers extract multiscale geometry;
2. Learnable Affine Transformations (LAT) adapt point and feature coordinates;
3. Cross-Cloud Transformation (CCT) explicitly applies prior-generated transforms to observed features.

REAL275 IoU75 improves from CATRE 43.6 to 51.8. More directly supporting shape adaptation are stable results under random prior replacement and outperforming full-data CATRE with only 2% of CAMERA25 training images.

2. Task Definition

Given:

$$(R_0,t_0,s_0),\quad O\in\mathbb R^{N_O\times3},\quad P\in\mathbb R^{N_P\times3},$$

The network predicts:

$$(\Delta R,\Delta t,\Delta s) =\phi(R_0,t_0,s_0,O,P).$$

Updated estimates can reenter the network iteratively. Each observed / prior cloud has 512 sampled points. This is category-level 9DoF: 3 rotation, 3 translation, and 3 metric-size dimensions, beyond 6DoF refinement alone.

It does not:

• Detect objects from RGB;
• Segment object clouds from depth;
• Produce initial poses;
• Learn category priors.

Upstream category estimators and preprocessing supply those inputs.

3. Point-cloud focalization

GeoReF follows CATRE focalization:

$$\hat O=\{\hat o_i\mid \hat o_i=o_i-t_0\},$$ $$\hat P=\{\hat p_i\mid \hat p_i=\operatorname{diag}(s_0)R_0p_i\}.$$

Subtract initial translation from observations and transform priors by initial rotation / anisotropic size into comparable coordinates. Remaining differences combine:

• Initial pose / size errors;
• Actual within-category shape variation;
• Missing depth, occlusion, and noise.

Refinement must distinguish the first two: correct pose errors without turning shape variation into incorrect pose corrections.

4. HS Layers and Learnable Affine Transformations

Simply replacing PointNet with stronger geometry encoders hurts: HS-Encoder IoU75 is 30.1 and 3DGCN 28.4 versus CATRE PointNet 43.6. Strong single-cloud geometry operators do not automatically align different shapes.

Matrix Net produces:

• $M\in\mathbb R^{3\times3}$: adapts input points;
• $M_r\in\mathbb R^{64\times64}$: adapts rotation features;
• $M_{ts}\in\mathbb R^{64\times64}$: adapts translation / size features.

These are learned affine alignments, not explicit rigid transforms. Rotation needs directional structure, while translation / size rely more on scale and centers, motivating separate transformations.

The ablations are clear:

Configuration	IoU75	5°2cm	5°5cm
Full GeoReF	51.8	54.4	60.3
No input-point LAT	39.3	46.6	53.0
No feature LAT	48.8	47.4	53.0
No separate rotation LAT	50.6	50.4	56.2

Input LAT matters most, highlighting coordinate compatibility before graph convolution.

5. Cross-Cloud Transformation

CATRE concatenates cloud features late. GeoReF applies prior-branch matrices to observations:

$$f_r^O=M_r^P f_r^O,\qquad f_{ts}^O=M_{ts}^P f_{ts}^O.$$

This is prior-conditioned feature-space alignment, not pointwise nearest-neighbor correspondence. Global aggregation combines both clouds before separate pose-error heads.

Global concatenation gives IoU75 / 5°2cm 48.4 / 47.8, versus CCT 51.8 / 54.4. Letting the other cloud change the feature basis works better than feeding adjacent vectors to an MLP.

The rotation head predicts 3D vectors $r_x,r_y$, with third axis:

$$r_z=r_x\times r_y.$$

Normalization / orthogonalization are still needed: cross product alone does not ensure $R\in SO(3)$. Translation / size heads output $\Delta t,\Delta s$.

6. REAL275 Component Ablations

The key progression in the full table:

Design	IoU75	5°2cm	10°2cm
CATRE	43.6	45.8	61.4
+ Prior in translation / size branch	45.8	48.0	63.8
+ HS-layer + LAT	51.0	52.4	69.4
+ CCT (full)	51.8	54.4	71.9

Priors help translation / size; LAT provides the largest gain; CCT completes cross-cloud fusion. Naive HS-Encoder replacement fails, so LAT enables geometry encoding for refinement rather than serving as an optional peripheral module.

7. Robustness across Initializations

Benefits are not limited to SPD:

• Self-DPDN: 5°2cm 44.3 → GeoReF 53.9; CATRE instead gives 44.0.
• GPV-Pose: 32.0 → GeoReF 47.4; CATRE 39.7.
• HS-Pose: 46.5 → GeoReF 51.7; CATRE 48.7.

CATRE can reduce 10°2cm / 10°5cm from strong initial estimates; GeoReF improves all three initializations, suggesting less confusion of instance shape with pose residuals.

8. Generalization with Less Data

CAMERA25 has 275k training images. The paper samples only:

Method	Training images	IoU75	5°2cm	5°5cm
CATRE	275k	76.1	75.4	80.3
CATRE	5k	63.2	66.4	72.3
GeoReF	5k	77.5	75.4	81.1
CATRE	10k	66.5	69.7	75.5
GeoReF	10k	79.2	77.9	84.0

GeoReF with 5k beats CATRE with 275k on IoU75, supporting lower sample requirements from structure. It does not establish cross-category generalization: train / test still share six categories, varying instance / image counts.

9. Main Results and Correct Interpretation

Method	IoU75	5°2cm	5°5cm	10°2cm	10°5cm
SPD	27.0	19.1	21.2	43.5	54.0
SPD + CATRE	43.6	45.8	54.4	61.4	73.1
SPD + GeoReF	51.8	54.4	60.3	71.9	79.4

The authors distinguish estimators and refiners. GeoReF sees initial pose, a prior, and depth points over a longer pipeline. The table shows improvements to existing estimators, not fewer inputs to one model.

10. Training and Reproduction

• GPU: RTX 4090;
• batch 12, 150 epochs;
• 512 points per observed / prior cloud;
• Ranger optimizer, base learning rate $10^{-4}$;
• Cosine annealing starts at 72% of total epochs;
• Augmentation: point dropout, Gaussian noise, pose perturbation;
• Losses and base augmentations follow CATRE;
• REAL275 ablations use SPD initial poses.

Conventions to verify:

• A small CATRE IoU implementation issue is corrected under later protocols;
• Compare refiners with the same initial estimator and iteration count;
• Random priors come from CAMERA25 training, not unknown categories;
• REAL275 has six categories and 36 objects, not representative open-category coverage.

11. Limitations

• RGB-D and reliable masks are required; not RGB-only.
• Category shape priors are required; articulated / nonrigid objects are unsupported.
• Initial pose must lie within the convergence basin; badly focalized clouds may lack usable correspondences.
• General affine feature transforms need not be physically interpretable or rigidly consistent.
• Total detector / initializer / refiner latency is not a central reported metric.
• Training / evaluation cover only six NOCS categories.

12. Overall assessment

GeoReF clarifies the central conflict: observation–prior mismatch combines pose error and shape variation; a stronger single-cloud encoder plus late concatenation cannot automatically separate them. LAT and CCT offer simple, well-supported cross-shape feature alignment.

It is a research baseline behind RGB-D category-level estimators, not a complete system. Deployment must evaluate segmentation, initialization, depth, iterations, and total latency together, beyond refined leaderboard scores.