Paper Reading Notes (Deep): MIRETR

1. Key assessment

Multi-instance registration changes more than output count. Global scene attention lets complete neighbors and background dominate sparse occluded-instance points; subsequent multi-model fitting must cluster huge numbers of correspondence hypotheses.

MIRETR separates instances during correspondence learning. Each scene superpoint predicts an anchor-relative mask for same-instance context; the mask later expands coarse matches into candidates. Features, matching, and pose grouping share one signal.

The global-attention ROBI variant reaches only 10.96 MF versus 39.80 for the full model, providing strong causal evidence.

2. Problem Definition

Model cloud:

$$P=\{p_i\in\mathbb R^3\}_{i=1}^{N},$$

Scene cloud:

$$Q=Q_0\cup Q_1\cup\cdots\cup Q_K,$$

Here $Q_0$ is background, $Q_1,\ldots,Q_K$ are $K$ instances of the model, and $K$ is unknown. Solve each instance:

$$T_k=(R_k,t_k) =\arg\min_{R,t} \sum_{(p_i,q_i)\in C_k} \|Rp_i+t-q_i\|_2^2.$$

Two challenges:

• Correspondences should cover many instances, not only the most complete;
• They must be assigned correctly, avoiding transforms that mix objects.

Open-set means geometric models can come from unseen objects / categories, not model-free open-vocabulary detection.

3. Instance-Aware Context

KPFCNN downsamples into $\hat P,\hat Q$ and assigns local patches by point-to-node partition. The Instance-aware Geometric Transformer repeats:

1. Geometric encoding: encode anchor-neighborhood distances / angles, using the previous mask to exclude other instances;
2. Cross-attention: exchange model / scene superpoint information for geometric consistency;
3. Instance masking: predict the anchor’s neighborhood mask from updated features.

Initial masks are imperfect, but local neighborhoods provide weak priors. Better features improve masks and purify subsequent context: a learned EM-like iteration.

Three supervision terms:

$$\mathcal L =\mathcal L_{\mathrm{circle}} +\mathcal L_{\mathrm{nll}} +\mathcal L_{\mathrm{mask}},$$

They constrain superpoint features, dense matching, and instance masks, respectively.

4. Superpoint Matches to Instance Poses

Traditional pipelines give all correspondences to T-Linkage, RansaCov, ECC, or PointCLM. For every superpoint correspondence, MIRETR:

1. Takes the model patch;
2. Collects neighboring scene patches within the predicted instance mask;
3. Applies optimal transport to local correspondences;
4. Forms candidate-specific dense matches;
5. Solves candidate $T_i$ with weighted SVD.

Inlier ratios score candidates. ADD-based pose similarity merges duplicates:

$$s_{ij}=1-\frac{\operatorname{ADD}(T_i,T_j)}{r},$$

$r$ is model diameter. Merged candidates are reestimated and refined using surviving inliers. Approximate instance purity avoids global multi-model fitting.

5. Scan2CAD

The Scan2CAD subset contains 2,184 model–scene pairs, split 70% / 10% / 20%. Key comparisons:

Method	IR	MR	MP	MF
GeoTransformer + PointCLM	44.02	82.81	81.90	82.35
GeoTransformer + ECC	44.02	94.63	74.83	83.57
MIRETR + PointCLM	56.59	91.85	91.08	91.46
MIRETR + ECC	56.59	96.52	89.03	92.62
MIRETR full	—	95.70	91.21	93.40

ECC has high recall but low precision, suggesting more false positives. Full MIRETR raises precision to 91.21 with little recall loss.

6. ROBI: Challenging Clutter

ROBI has 7 reflective metal parts, 63 bin-picking scenes, and 4,880 pairs, split 6:1:2 by scene. Results:

Method	IR	MR	MP	MF
GeoTransformer + ECC	39.64	18.55	30.99	23.20
MIRETR + PointCLM	45.14	18.68	40.11	25.48
MIRETR + ECC	45.14	24.65	34.85	28.91
MIRETR full	—	38.51	41.19	39.80

Full MIRETR improves MF over GeoTransformer + ECC by 16.6 points, but absolute MF is only 39.80. Missing depth, reflections, and severe occlusion remain difficult.

Mask mIoU is 69.26 versus 38.13 without masking. Mean successful instances are 13.7 versus MIRETR + PointCLM 6.2, emphasizing recovered hard / low-overlap instances rather than precision on easy ones alone.

7. Unseen Categories and Ablations

Train on the first 30 ShapeNet categories, test on the remaining 25; scenes contain 4–16 randomly transformed instances. Full model:

$$\mathrm{IR/MR/MP/MF} =77.13/94.95/93.94/94.44.$$

GeoTransformer + ECC MF is 70.37, with full MIRETR higher by 24.07 points. Synthetic CAD clouds have easier noise / occlusion than ROBI; 94.44 does not predict real bin-picking accuracy, but supports geometry beyond fixed categories.

ROBI ablations:

Configuration	IR	MR	MP	MF
Full	45.14	38.51	41.19	39.80
No instance-aware point matching	40.52	18.15	27.54	21.88
No instance-masking block	36.78	17.07	45.37	24.80
Neither	38.81	14.13	27.44	18.65
global attention	25.98	7.72	18.86	10.96

Without masking, precision is highest but recall collapses, retaining few easy instances. Global attention is worst, supporting cross-instance context contamination as the core issue.

8. Engineering Significance

MIRETR suits known-part bin picking:

• Unknown instance counts;
• Multiple poses from one forward process;
• Better low-overlap / touching-instance handling than global matching;
• Reusable weighted SVD and registration tools;
• Geometry-based cross-category matching.

A complete robot system still needs:

• Model selection from a CAD library;
• Adequate depth quality;
• Symmetry handling;
• Collision / graspability checks per candidate;
• End-to-end timing and memory tests against cycle-time targets.

The main paper gives no runtime; efficiency over multi-model fitting is primarily structural and still needs deployment benchmarks.

9. Limitations and Reproduction Checks

• Scan2CAD replaces scans with aligned CAD, reducing original scan-noise difficulty.
• ShapeNet scenes copy transformed instances of one CAD, leaving domain gaps.
• ROBI scenes contain one part type; mixed-CAD multi-category clutter is untested.
• Correct model clouds are supplied, excluding retrieval failures.
• Masks are anchor-relative, not globally consistent segmentation; merging still needs thresholds.
• High symmetry may require symmetry-aware ADD candidate similarity.
• No main-text FPS, parameter counts, candidate-count sensitivity, or depth-noise calibration.

10. Overall assessment

Instance boundaries move from postprocessing into features, then coarse matches, dense correspondences, and pose candidates. Low-overlap curves, mask mIoU, and ablations support this logic beyond leaderboard scores.

This remains model-to-scene registration rather than general object discovery, with real ROBI MF only 39.80. Natural extensions include mixed CAD, symmetry-aware merging, and RGB / foundation features to compensate for reflective-part depth gaps.