Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception
NeMO encodes a few RGB templates into object-centric point memory with geometry and semantics. A shared decoder reuses this memory for detection, segmentation, surface prediction, and 6-DoF pose of objects unseen in training.
NeMO encodes a few RGB templates into reusable object memory with its own coordinate frame. A shared decoder queries that memory using RGB for novel-object detection, segmentation, surface prediction, and 6DoF pose, without target-object retraining.
Inputs: several target RGB templates and one query RGB; model-based inference can additionally use CAD renderings.
Outputs: amodal detection, modal / amodal segmentation, 2D–3D point maps, and camera-to-object pose.
Boundary: object priors remain, pre-encoded from template views into NeMO during onboarding.
4. Method overview
ViT extracts template patches, fused by a multiview encoder.
Sampled points, an MLP, and geometric mapping inject geometry into point memory.
Query features interact with NeMO through cross-/self-attention; a DPT decoder predicts dense maps and confidence.
Optional ICP aligns predicted surfaces with depth; decoder weights are shared across tasks and objects.
5. Experimental overview
BOP: HOPEv2, HANDAL, T-LESS, TUD-L, YCB-V.
Model-free 6DoF AP with NeMO detections: HOPEv2 / HANDAL 0.302 / 0.235; model-free detection AP 0.411 / 0.273.
Model-based, no-refinement 6D AR with CNOS detections: T-LESS / TUD-L / YCB-V 0.190 / 0.476 / 0.504.
The authors identify T-LESS texturelessness, symmetry, and same-instance clutter as major difficulties.
6. Strengths, limitations, and implications
Strengths
Replace repeated pairwise template matching with reusable object onboarding.
One network supports detection, segmentation, surfaces, and pose for rapid novel-object integration.
Limitations / risks
Training is costly: 16 A100s for approximately 10 days. Dense decoding still depends on crop / detection quality.
Template count, view coverage, and depth-refinement availability directly affect performance.
7. Deep-reading priorities
Examine how templates establish NeMO coordinates and why improved model-free detection does not fully translate into pose AP. Better boxes do not automatically mean more accurate geometric alignment.
Available src/paper-notes/finding-nemo-3dv2026/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
Deep Reading Notes: Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception
1. Basic information
Authors / affiliation: Sebastian Jung, Leonard Klüpfel, Rudolph Triebel, Maximilian Durner; German Aerospace Center (DLR).
A few RGB templates build a queryable Neural Memory Object with object coordinates. Onboarding absorbs repeated matching against every template, allowing one dense decoder to support unseen-object detection, segmentation, and 6DoF pose.
2.1 Guide to the paper’s figures and tables
Figure / table
Reading focus
Figure 2
How template features become point memory and how queries read it
Figures 4–5
Model-free detection / pose and surfaces; distinguish better boxes from better poses
Tables 1–6
Detection sources, model-free/model-based settings, BOP AP/AR, and template-point tradeoffs
Original Figure 2: ViT and a multiview encoder construct NeMO from templates; query images decode dense predictions from memory. Source: arXiv PDF. Image copyright belongs to the paper’s authors and publisher; shown here for paper analysis.Crop from the experimental page containing original Figure 4: model-free few-shot detection and pose overlays. Source: arXiv PDF. Image copyright belongs to the paper’s authors and publisher; shown here for paper analysis.
3. Problem setting and task boundaries
Task: model-free / model-based few-shot perception of training-unseen objects. Model-free uses no CAD; model-based uses CAD renders only at inference.
Inputs: object RGB templates, often 32 in default experiments, and query RGB. Object crops can be supplied; additional HOPEv2 pose ICP uses depth.
Not required: target-object fine-tuning or known template-camera intrinsics / extrinsics to establish NeMO coordinates.
Still required: template coverage, detection / crops, and depth refinement in some protocols. This is not prior-free open-world pose.
4. Method details
4.1 Pipeline
ViT extracts each template’s patches; the multiview encoder cross-attends across views.
Sample points from [−1,1]3 and map them through an MLP; a Geometric Mapping Block links points and image features into feature-enhanced 3D memory.
Query ViT features alternate cross-attention / self-attention with memory; DPT predicts point maps, confidence, modal/amodal masks, and other dense maps.
Surface / point correspondences produce pose, optionally refined with depth ICP where allowed.
4.2 Key modules
Module
Input
Output
Role
Multi-view encoder
Template patch features
Attended template features
Aggregate few-shot views instead of independent matching
Geometric mapping
Template features + sampled 3D points
NeMO point memory
Encode geometry and semantics, beyond image tokens
Rendered training data contain 11,077 Objaverse, GSO, and OmniObject3D objects; BOP test objects are excluded from training and target fine-tuning.
DINOv2 ViT, DPT regression, and AdamW train for 400k steps on 16×A100 for approximately 10 days. This is the main cost of learning general object memory.
Expensive template attention can be precomputed, but onboarding views, memory size, and scene coverage still determine practical cost.
5. Experimental analysis
Setting
Metric / result
My interpretation
Model-free detection
HOPEv2 / HANDAL AP = 0.411 / 0.273
Reported gains of 2.7 / 1.9 pp over previous best support amodal object memory for detection
Model-free 6DoF, NeMO detections
HOPEv2 / HANDAL AP = 0.302 / 0.235
Its own detections need not yield the best pose; detection AP differs from pose AP
Model-based 6D, no refinement
T-LESS / TUD-L / YCB-V AR = 0.190 / 0.476 / 0.504
Textureless / symmetric T-LESS is difficult; TUD-L and YCB-V are competitive
Capacity saturates; more sampled points alone do not solve the task
5.1 Authors’ claims and my assessment
Author claim
Evidence
My assessment
Novel-object onboarding without fine-tuning
Disjoint train / test objects and few-shot BOP
Well supported, but multiple object templates remain necessary
One representation across tasks
Detection, segmentation, surface, and pose tables
Supported, with shared-decoder tradeoffs rather than best results everywhere
Reliability in clutter
HOPEv2 / HANDAL quantities and visuals
Some support; T-LESS reveals symmetry/texturelessness failures
6. Critical analysis
Strengths: externalize object information into memory without scaling model weights per object; joint segmentation / point predictions can support pose; both BOP model-free and model-based protocols are covered.
Limitations: template coverage, symmetry, weak texture, and same-category clutter remain difficult. Few-shot inference does not imply cheap training; AP/AR depends on detectors and ICP availability, requiring matched protocols.
Research implications: learn confidence / visibility fields for template correspondences and use them in weighted PnP / refinement. Memory uncertainty could prevent amodal detection gains from being mistaken for pose confidence.
