LFM-3D: Learnable Feature Matching Across Wide Baselines Using 3D Signals
LFM-3D encodes NOCS or monocular depth as 3D positional signals for keypoints, injecting them into a graph matching network to improve wide-baseline correspondences and relative rotation estimation between object images.
instance segmentationcamera intrinsicsSuperPoint featuresNOCS model or monocular depth estimator
Reading Notes
Overview / Deep Dive
Available src/paper-notes/lfm-3d-3dv2024/coarse.md
Paper overview notes
Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.
LFM-3D: Coarse Reading
One-sentence conclusion
LFM-3D adds per-keypoint 3D signals to a SuperPoint/SuperGlue-style matcher: NOCS for supported categories and monocular depth for open categories. The goal is better local matching and relative rotation between object images with 90°–120° baselines and low overlap.
Method
RGB local descriptors remain unchanged. Separate positional MLPs encode NOCS / depth, which enter graph neural networks and optimal-transport matching alongside 2D keypoint positions. The 3D-signal networks are frozen during matcher training, so the study asks whether coarse estimated 3D cues help matching rather than jointly optimizing depth.
Each image produces local features and 3D positional embeddings that interact through a graph network. Outputs remain 2D–2D matches suitable for essential-matrix estimation. Source: original Figure 2. Copyright belongs to the original authors and 3DV 2024 publisher; reproduced here for academic analysis.
Results
On Objectron shoes, the NOCS variant achieves 40.2 Acc@15°, versus 31.6 for 2D-only fine-tuned SuperGlue and 34.9 for Dense NOCS-PnP. On five unseen categories, generic MDE achieves 19.1/28.9/34.5 (5°/10°/15°), exceeding generic fine-tuned SuperGlue at 17.7/26.9/32.4.
NOCS gives the largest gain on the known shoe category. Monocular depth gives smaller open-category gains without a category CAD library. Source: original Table 1. Copyright belongs to the original authors and 3DV 2024 publisher; reproduced here for academic analysis.
Scope
LFM-3D outputs relative camera rotation between image pairs, rather than BOP-style object-to-camera 6D pose. Rotation is the main evaluation target; translation is not a core metric. It is a frontend for sparse object reconstruction and should not be ranked directly against absolute object-pose methods such as FoundationPose.
Available src/paper-notes/lfm-3d-3dv2024/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
LFM-3D: Deep Reading
1. Why Wide Baselines Need 3D Signals
Generic local matchers rely on repeatedly visible textures and consistent local appearance. Object-centric images separated by 90°–120° share few surfaces; shoes, cameras, and similar objects also exhibit self-occlusion and near-repeated structures. Pure 2D descriptors infer identity from the current patch, while NOCS or depth supplies geometric context about an object side or distance from the camera.
LFM-3D does not require highly accurate 3D signals. The hypothesis is that noisy NOCS / depth can still reduce geometric ambiguity among match candidates, with the graph network learning when to trust them.
2. Three Settings
Known category + NOCS: train a category-specific NOCS network on synthetic Google Scanned Objects renders; suitable for shoes.
Known category + monocular depth: use frozen DPT to supply class-agnostic depth for categories such as cameras, where a shared NOCS is difficult.
Unseen categories + monocular depth: train a generic matcher on the full GSO catalog and test zero-shot on Objectron bike/book/cereal box/chair/laptop.
3D positional embeddings complement SuperPoint appearance features; the matcher still outputs discrete matches through optimal transport. Source: original Figure 2. Copyright belongs to the original authors and 3DV 2024 publisher; reproduced here for academic analysis.
3. Training and Geometry Backend
Training correspondences use known depth / cameras: back-project keypoints from one image and reproject them into the other; mutual nearest neighbors within 3 pixels provide ground truth. NOCS / depth predictors remain frozen during matcher training. At inference, correspondences and camera intrinsics estimate an essential matrix and recover relative rotation, reporting fractions with angular errors below 5°/10°/15°.
Translation is not fully evaluated. An essential matrix recovers translation direction with scale ambiguity, so the connection to 6D pose is frontend correspondence estimation for object-centric sparse-view alignment.
4. Experimental Breakdown
At 90°–120° baselines, 3D signals help bridge appearance changes. The figure also shows long-tail categories and NOCS failures. Source: original Figure 5. Copyright belongs to the original authors and 3DV 2024 publisher; reproduced here for academic analysis.
On shoes:
2D-only finetuned SuperGlue: 11.0/22.2/31.6
Dense NOCS-PnP: 4.2/18.9/34.9
LFM-3D NOCS: 14.3/29.3/40.2
This suggests that treating estimated NOCS as soft matcher context is better than assuming exact coordinates for PnP. At 15°, it exceeds Dense PnP by 5.3 points and the 2D matcher by 8.6 points.
On unseen categories, generic MDE LFM-3D reaches 19.1/28.9/34.5, versus 17.7/26.9/32.4 for generic fine-tuned SuperGlue, a gain of about 2 points. This supports class-agnostic 3D cues, though gains are much smaller than with category-specific NOCS.
Compare shoe, camera, and unseen-category settings separately; NOCS and MDE results do not form one unified ranking. Source: original Table 1. Copyright belongs to the original authors and 3DV 2024 publisher; reproduced here for academic analysis.
5. Limitations and Reproduction Notes
The NOCS network sees mostly sneakers in GSO and fails on rarer shoe types, revealing substantial data bias in category-specific 3D priors.
Results depend on off-the-shelf instance segmentation; mask errors change the keypoint set.
The paper establishes improved relative rotation; full camera translation, scale, and downstream reconstruction quality require separate validation.
Training needs synthetic models and camera / depth data for correspondence supervision; it is not entirely free of 3D data.
6. My Assessment
The main lesson is that estimated 3D can serve as matching context instead of a direct answer. When NOCS / depth is imperfect, soft encoding that lets a matcher learn its use is often more robust than hard PnP. The idea extends naturally to modern depth foundation models, dense correspondence, and object-level SfM.
