Back to venue collections

ICRA 2024 · Paper Collection

Object Pose Estimation Papers

ICRA 2024 moves object pose beyond target-specific training and static single-frame assumptions: pretrained visual features support novel-object matching, NeRF / NGP becomes queryable 3D object models, and active vision, video, and events extend estimation into robot motion.

14

Papers meeting the inclusion criteria

3

NeRF / NGP approaches

3

Video / event tracking

2

Event-camera papers

Using the official ICRA 2024 proceedings, this page selects 14 papers directly estimating object 6D pose, category-level pose / size, object tracking, or task-related cross-pose. The conference took place in Yokohama, Japan, on May 13–17, 2024.

Collection in One Sentence

Fewer object priors, stronger 3D representations, more active observation

ZS6D, DMSR, and FAP-Net use pretrained descriptors, scale decoupling, and implicit 3D perception. Fewer training requirements and sensors retain geometry through correspondences, metric scale, and pose solvers.

IFFNeRF, NeRF + Feature Matching, and Fit-NGP turn neural fields into 3D pose interfaces. RGBManip, MOTPose, EDOPT, and event-model tracking examine whether robot motion and temporal evidence can continuously improve pose reliability.

Research Trends

Four themes to watch in 2024

Trends summarize problem settings and methods across 14 selected core papers, rather than official ICRA topic categories.

01 · Reusable Descriptors

Pretrained visual features support zero-shot geometry

ZS6D directly uses ViT descriptors for template-view retrieval and local correspondence without pose-specific fine-tuning, shifting generalization toward reusable visual representations.

02 · Monocular 3D

Monocular category-level pose addresses scale and missing 3D geometry

DMSR decouples metric scale from correspondence, while FAP-Net uses geometry supervision for coarse-to-fine 3D perception. Both explicitly recover structure needed for RGB-only pose estimation.

03 · Neural Fields for Pose

NeRF / NGP becomes a geometric query interface

IFFNeRF learns query-to-NeRF-ray correspondences; NeRF + Feature Matching lifts 2D matches into PnP inputs using rendered depth; Fit-NGP fits CAD models directly into density fields.

04 · Pose over Time

Active vision, video, and fast events extend single-frame estimation

RGBManip actively selects arm viewpoints, MOTPose fuses video history through cross-attention, and EDOPT and model-based event tracking maintain high-frequency updates under fast motion and blur.

Data Statistics

Research-area distribution

Each paper belongs to one group by primary contribution; card tags retain overlapping attributes.

Fewer priors and category-level generalization

4

Reduces target-specific annotations, rendering, and depth-sensor dependence through pretrained visual features, monocular geometry, scale decoupling, and self-supervised learning.

Neural fields and active perception

4

Turns NeRF / NGP into queryable 3D models for object pose and uses robot motion to actively supplement monocular observations.

Industry and challenging materials

2

Targets stacked bins, transparent objects, tiny metal parts, and sim-to-real transfer through complete perception–localization–manipulation systems.

Temporal, event-based, and relational pose

4

Extends single-frame estimation to video multi-object fusion, high-speed event tracking, and multimodal relative goal poses learned from a few demonstrations.

Interpreting the statistics: Generalization with fewer priors and neural-field / active-perception methods total 8 papers, more than half the core set. Two event-camera tracking papers also show attention extending toward real-time, continuous estimation during robot motion.

Recommended Reading

Recommended papers

No core paper carries an official ICRA 2024 Best Paper Finalist label. Recommendations reflect methodological representativeness, completeness, and robotic deployment value.

Representative zero-shot approach

ZS6D

An early demonstration that pretrained ViT descriptors already support geometric generalization for novel-object template matching and local correspondence.

Read the paper and deep notes →

Representative category-level approach

DMSR

Explicitly decouples pose correspondences from category-level metric scale so incorrect monocular scale does not directly contaminate rigid transformations.

Read the paper and deep notes →

Priority reading

FAP-Net

FAP-Net combines geometry-supervised coarse perception and pose/size-constrained refinement, reporting 42 FPS alongside a clear geometric design.

Read the paper and deep notes →

Representative neural-field approach

IFFNeRF

Samples NeRF surfaces as colored parameterized rays and matches them to queries through attention, avoiding iNeRF's dependence on nearby initialization and iterative photometric optimization.

Read the paper and deep notes →

Priority reading

NeRF + Feature Matching

Combines NeRF-rendered depth with existing image matchers for 2D–3D correspondence, replacing hundreds of photometric optimization steps with RANSAC-PnP.

Read the paper and deep notes →

Representative industrial approach

Fit-NGP

Fit-NGP uses neural fields as precise model-fitting spaces, reporting approximately millimeter-level localization for small reflective nuts, bolts, and similar parts.

Read the paper and deep notes →

Priority reading

RGBManip

One policy balances viewpoints that improve pose observability against completing manipulation quickly, making pose estimation part of the action process.

Read the paper and deep notes →

Representative industrial approach

Sim-to-Real Bin Picking Pose

Represents an important industrial approach in 2024: fully automating per-workpiece data generation and annotation for direct sim-to-real transfer.

Read the paper and deep notes →

Representative event-vision approach

Event Model-Based Tracking

Uses motion interpolation to predict brightness-increment images and align asynchronous events with 3D model projections, evaluating both perspective and fisheye cameras.

Read the paper and deep notes →

Representative temporal approach

MOTPose

MOTPose aggregates historical evidence through cross-attention at both object-embedding and pose-parameter levels, rather than merely smoothing single-frame outputs.

Read the paper and deep notes →

Representative relational-pose approach

Distributional Cross-Pose

Explicitly models multimodal goals, such as a cup fitting several hooks, avoiding invalid poses produced by averaging valid placement modes.

Read the paper and deep notes →

Representative event-vision approach

EDOPT

EDOPT separates changing event rates, online template generation, and pose-hypothesis updates to achieve event-only 6DoF tracking above 300 Hz.

Read the paper and deep notes →

Paper List

Paper list and summaries

Each card gives task boundaries, input modalities, and a short summary, linking to coarse and deep notes. Original-paper links remain as supporting sources.

4 papers

Fewer priors and category-level generalization

Reduces target-specific annotations, rendering, and depth-sensor dependence through pretrained visual features, monocular geometry, scale decoupling, and self-supervised learning.

ICRA 2024 Method paper

ZS6D: Zero-Shot 6D Object Pose Estimation Using Vision Transformers

Philipp Ausserlechner · David Haberger · Stefan Thalhammer · Jean-Baptiste Weibel · Markus Vincze

zero-shotnovel objectViT descriptortemplate matchingPnPBOP

Setting: Instance-level 6D pose for CAD objects unseen during training, without pose-specific fine-tuning

Input: RGB scene + CAD-rendered templates

Summary: ZS6D extracts descriptors from queries and CAD-rendered templates with a pretrained Vision Transformer, matches template viewpoints, establishes local 2D–3D correspondences, then solves RANSAC-PnP. Requires no task-specific fine-tuning on LM-O, YCB-V, or T-LESS.

ICRA 2024 Method paper

RGB-Based Category-Level Object Pose Estimation Via Decoupled Metric Scale Recovery

Jiaxin Wei · Xibin Song · Weizhe Liu · Laurent Kneip · Hongdong Li · Pan Ji

category-levelRGB-onlymetric scale2D–3D correspondenceRANSAC-PnP

Setting: 6D pose and metric size of unseen within-category instances from one RGB image

Input: monocular RGB

Summary: DMSR uses a pretrained monocular geometry estimator for local 3D cues and reliable 2D–3D correspondences. An independent branch recovers metric scale from category statistics, followed by RANSAC-PnP, directly addressing RGB-only category-level scale ambiguity.

ICRA 2024 Method paper

Implicit Coarse-To-Fine 3D Perception for Category-Level Object Pose Estimation from Monocular RGB Image

Jia Li · Li Jin · Xibin Song · Yeheng Chen · Nan Li · Xueying Qin

category-levelRGB-onlycoarse-to-finegeometry supervision42 FPS

Setting: Real-time category-level pose / size through implicit 3D perception from monocular RGB

Input: monocular RGB

Summary: Feature Auxiliary Perception Network learns coarse 3D features through geometry supervision, then refines them with pose and size constraints. On REAL275, reports gains of 14.7 percentage points at 10°10cm and 11.4 percentage points at IoU50, maintaining real-time inference.

ICRA 2024 Method paper

Joint-Loss Enhanced Self-Supervised Learning for Refinement-Coupled Object 6D Pose Estimation

Fengjun Mu · Shixiang Sun · Rui Huang · Chaobin Zou · Wenjiang Li · Huayi Zhan · Hong Cheng

self-supervisedrefinement-coupledjoint losslow annotationindustrial pose

Setting: Reduces dense real 6D pose annotations and jointly learns initialization and refinement

Input: RGB / RGB-D object observation

Summary: Joint losses couple self-supervised pose representations and refinement so initialization and subsequent correction are learned together. Primarily reduces industrial-object 6D annotation requirements, rather than eliminating all object priors at test time.

4 papers

Neural fields and active perception

Turns NeRF / NGP into queryable 3D models for object pose and uses robot motion to actively supplement monocular observations.

ICRA 2024 Method paper

IFFNeRF: Initialisation Free and Fast 6DoF Pose Estimation from a Single Image and a NeRF Model

Matteo Bortolon · Theodore Tsesmelis · Stuart James · Fabio Poiesi · Alessio Del Bue

NeRF / NGPinitialisation-freeray matchingattention34 FPS

Setting: Given an object NeRF, directly recovers camera–object 6DoF pose from an arbitrary query image

Input: single RGB + object NeRF

Summary: IFFNeRF samples surface points from NeRF through Metropolis-Hastings and projects rays, learns query-to-ray-bundle correspondences with attention, then solves pose by least squares. Reports 34 FPS and substantially lower angular and translation errors than iNeRF.

ICRA 2024 Method paper

Marrying NeRF with Feature Matching for One-Step Pose Estimation

Ronghan Chen · Yang Cong · Yu Ren

NeRF / NGPnovel objectLoFTRone-step PnP6 FPS

Setting: Rapid object-NeRF construction from a few multiview images, followed by real-time pose for new images

Input: single RGB + object NeRF

Summary: Renders RGB / depth from an initial viewpoint, matches query and rendering with LoFTR or similar methods, then lifts 2D–2D matches into 2D–3D correspondences using NeRF depth. 3D consistency mining and occlusion sampling filter false geometry. Reports approximately 90× speedup and 6 FPS.

ICRA 2024 Systems paper

Fit-NGP: Fitting Object Models to Neural Graphics Primitives

Marwan Taher · Ignacio Alzugaray · Andrew J. Davison

NeRF / NGProbot scanningknown CADsmall objectreflective surface

Setting: A robot scans a scene from scratch and localizes multiple known tiny objects in the reconstructed density field

Input: wrist RGB video + known 3D models

Summary: A wrist-mounted monocular camera actively scans the scene, efficient Neural Graphics Primitives reconstruct a density field, and known object models fit directly to it. Automatically detects and estimates multiple objects, reporting approximately 1 mm accuracy for small reflective industrial parts.

ICRA 2024 Systems paper

RGBManip: Monocular Image-Based Robotic Manipulation Through Active Object Pose Estimation

Boshi An · Yiran Geng · Kai Chen · Xiaoqi Li · Qi Dou · Hao Dong

active perceptionRGB-onlyeye-in-handreinforcement learningmanipulation

Setting: Actively collects multiview observations during arm motion and performs manipulation without depth sensors

Input: eye-in-hand monocular RGB

Summary: RGBManip actively acquires multiple views with a gripper-mounted monocular camera and estimates object 6D pose from images. A reinforcement-learning policy jointly optimizes viewpoints, pose accuracy, and manipulation time, evaluating image-only manipulation in simulation and real systems.

2 papers

Industry and challenging materials

Targets stacked bins, transparent objects, tiny metal parts, and sim-to-real transfer through complete perception–localization–manipulation systems.

ICRA 2024 Systems paper

Perception Through Cognitive Emulation: A Second Iteration of NaivPhys4RP for Learningless and Safe Recognition and 6D-Pose Estimation of (Transparent) Objects

Franklin Kenghagho Kenfack · Michael Neumann · Patrick Mania · Michael Beetz

transparent objectlearninglesswhite-box modelcognitive emulationcausal physics

Setting: Interpretable physical-world models identify transparent objects and estimate pose from weak sensory observations

Input: poor / partial sensor observations + symbolic physical knowledge

Summary: NaivPhys4RP uses cognitive simulation to construct possible worlds satisfying commonsense, causal, and physical constraints, rather than large training datasets. White-box reasoning explains observations and recovers transparent-object identity and 6D pose, emphasizing interpretable, safe perception beyond standard BOP ranking.

ICRA 2024 Systems paper

Sim-To-Real Object Pose Estimation for Random Bin Picking

Boyoung Kim · Junhong Min

bin-pickingsim-to-realsynthetic dataphysics simulatortextureless object

Setting: Simulation-only training for target-workpiece detection and 6D pose in real randomly arranged bins

Input: RGB-D / point cloud + CAD-based simulation

Summary: Physics simulation and target CAD automatically generate stacked-bin scenes, instance segmentations, and pose annotations. A localization system trains only on synthetic data and deploys to real random bin picking. Evaluation should consider real pose accuracy, domain gaps, and grasp success together.

4 papers

Temporal, event-based, and relational pose

Extends single-frame estimation to video multi-object fusion, high-speed event tracking, and multimodal relative goal poses learned from a few demonstrations.

ICRA 2024 Method paper

Direct 3D Model-Based Object Tracking with Event Camera by Motion Interpolation

Yufan Kang · Guillaume Caron · Ryoichi Ishikawa · Adrien Escande · Kevin Chappellet · Ryusuke Sagawa · Takeshi Oishi

event cameratrackingknown 3D modelmotion interpolationfisheye

Setting: Continuously aligns a known 3D object model directly to asynchronous events during fast camera motion

Input: event stream + object 3D model

Summary: Constructs a new event representation, predicts brightness increments through object-motion interpolation, and directly registers them to projected 3D models. Evaluates tracking duration and high-speed robustness, operating at camera speeds around 2 m/s.

ICRA 2024 Method paper

MOTPose: Multi-Object 6D Pose Estimation for Dynamic Video Sequences Using Attention-Based Temporal Fusion

Arul Selvam Periyasamy · Sven Behnke

trackingmulti-objectvideocross-attentionjoint detection + pose

Setting: Joint multi-object detection and 6D pose in dynamic, occluded bin videos

Input: RGB video sequence

Summary: Processes video sequences to detect all objects and estimate poses in one forward pass. Temporal fusion aggregates object embeddings and parameters across time, improving detection and pose on SynPick and YCB-Video.

ICRA 2024 Method paper

Learning Distributional Demonstration Spaces for Task-Specific Cross-Pose Estimation

Jenny Wang · Octavian Donca · David Held

relative posemultimodal placementfew-shot demonstrationcategory-levelno annotation

Setting: Learns multiple valid inter-object relative poses for one task from 10–20 demonstrations

Input: paired object point clouds + demonstrations

Summary: Extends TAX-Pose geometric-relation reasoning with a spatially grounded distributional demonstration latent space for multimodal relative placements. With only 10–20 demonstrations and no manual semantic labels, generates precise task-related cross-poses across different within-category objects.

ICRA 2024 Method paper

EDOPT: Event-Camera 6-DoF Dynamic Object Pose Tracking

Arren Glover · Luna Gava · Zhichao Li · Chiara Bartolozzi

event cameratrackingevent-onlymesh prior300+ Hz

Setting: High-frequency online 6DoF tracking of fast known rigid objects with an event camera alone

Input: asynchronous events + object mesh

Summary: EDOPT projects object meshes into event-compatible templates, maintains future pose hypotheses, and adapts tracking updates to varying event rates. The paper claims the first event-only 6DoF object-pose method, with accuracy approaching deep networks combining frames, depth, and events.

Scope & Sources

Counting rules and scope

Inclusion: official ICRA 2024 proceedings papers whose titles and abstracts directly estimate object 6D pose, category-level pose / size, current object-tracking states, or inter-object relative poses as transferable manipulation goals.

Excluded: grasp pose, human / hand pose, camera and robot localization, SLAM-first joint mapping, surgical-instrument and single-agricultural-object applications, and manipulation policies taking object pose as input. Distributional Cross-Pose is grouped as a relational-pose extension to distinguish it from standard BOP tasks.

Related works such as ViHOPE, ContourPose, SyMFM6D, and ROV6D are RA-L journal papers presented at ICRA and are excluded from these 14 proceedings papers. A separate journal track can avoid double-counting publication and conference presentation.

Primary sources: Official ICRA 2024 Program, Official proceedings contents, IEEE DOI metadata, arXiv, and public author project pages. Classifications, summaries, and trends are editorial work and do not replace the original papers.