Deep Reading Notes: CoordAR: One-Reference 6D Pose Estimation of Novel Objects via Autoregressive Coordinate Map Generation

1. Basic information

• Title: CoordAR: One-Reference 6D Pose Estimation of Novel Objects via Autoregressive Coordinate Map Generation
• Authors / affiliations: Dexin Zuo, Ang Li, Wei Wang, Wenxian Yu, Danping Zou; Shanghai Jiao Tong University, Midea Group
• Year / conference or journal: 2026 / AAAI-26 Technical Tracks 16
• Paper: https://ojs.aaai.org/index.php/AAAI/article/view/38424
• PDF: https://ojs.aaai.org/index.php/AAAI/article/download/38424/42386
• Project page: https://sjtu-visys-team.github.io/CoordAR
• Reading mode: Deep Dive

2. Positioning in one sentence

The paper reformulates one-reference pose estimation as conditional query-ROC-map generation. A discrete tokenizer compresses a continuous 3D coordinate field into tokens; a modality-decoupled Transformer predicts them autoregressively, and a geometric solver recovers relative pose.

2.1 Guide to the paper’s figures and tables

Figure / table	Content	Reading focus
Figure 1	Overview of reference / query inputs, ROC maps, token prediction, and final pose	Understand the split into coordinate-map generation + Umeyama pose recovery
Figure 2	Complete CoordAR network pipeline	See how RGB and ROC are encoded separately, then interact in fusion blocks
Tables 1–3	Main Real275, Toyota-Light, YCB-V, and LINEMOD results	Note input differences between one-reference, multi-view, and CAD-assisted methods
Tables 4–5	Architecture ablations and speed analysis	Examine the actual accuracy and speed effects of autoregressive decoding

3. Problem setting and task boundaries

• Target: rigid object instances unseen during training, emphasizing novel objects rather than category-level canonical pose across instances.
• Reference input: $I_R=(C_R,D_R)$, with RGB, depth, and an object mask; absolute reference object pose $T_{RO}$ is known.
• Query input: $I_Q=(C_Q,D_Q)$ from an unknown viewpoint, also requiring a query mask.
• Goal: estimate the query-to-reference relative transform $T_{RQ}$, then compute $T_{QO}=\hat{T}_{RQ}^{-1}T_{RO}$.
• CAD requirement: no complete CAD model is needed; reference depth is required to generate the ROC map.
• Multi-view requirement: only one reference is needed at inference, without first reconstructing a 3D model from multiple posed images.
• Masks / intrinsics: both are required. Experimental inputs are relatively ideal, making mask quality an important deployment variable.
• Output representation: the network outputs a query ROC map rather than quaternions, axis–angle, or translation directly.

4. Research context

• Traditional instance-level methods usually require object-specific training or known CAD. Category-level methods improve within-category generalization, which does not imply support for arbitrary new categories.
• Model-free approaches include reconstructing objects from multiple posed views before render-and-compare, and directly establishing correspondences from a few references.
• CoordAR directly revisits One2Any’s continuous ROC regression. It retains reference object coordinates but predicts coordinate maps through autoregressive generation of discrete tokens.
• Unlike conventional diffusion / image generation, the token sequence generates structured 3D coordinates for geometric alignment rather than appearance.

5. Research motivation

Problems the Authors Address

1. A convolutional decoder’s local receptive field may fail to establish global spatial consistency across patches.
2. Continuous-coordinate regression lacks an explicit probabilistic representation of ambiguity from symmetry, occlusion, and undefined canonical frames.
3. Directly concatenating RGB appearance and ROC coordinates mixes modalities with different statistical properties.

My Analysis

The key insight is that pose errors may already arise during correspondence-map generation, before the final Umeyama / PnP solver. The paper therefore retains the geometric solver and reformulates learning as conditional autoregressive modeling that can better represent global dependencies.

6. Method details

6.1 Overall pipeline

1. Back-project reference depth using the mask, then center and scale-normalize it to obtain reference ROC map $X_R$.
2. Align query depth to the reference object frame using the relative transform, yielding ground-truth query ROC map $X_Q$ for training supervision only.
3. Encode $C_R,C_Q$ with a shared RGB encoder and $X_R$ with a separate encoder.
4. Inject reference conditioning into pixel-aligned query features through multiple fusion blocks.
5. Compress ROC maps into patch-level token sequences with a VQ-VAE codebook; the decoder predicts query tokens autoregressively.
6. Detokenize to obtain $\hat{X}_Q$, then use query depth and inverse normalization to obtain 3D points in the reference frame.
7. Recover $\hat{T}_{RQ}$ with Umeyama and combine it with the absolute reference pose to obtain the absolute query pose.

6.2 Key Modules

Module	Input	Output	Role	Interface to the next module
ROC map construction	Depth, masks, intrinsics	Normalized 3D coordinate map	Converts depth into comparable object-frame geometry	Reference ROC enters a separate encoder
Modality-decoupled encoding	RGB, reference ROC	RGB / geometry features	Avoids imposing one encoding assumption on RGB and 3D coordinates	Same-modality features enter fusion blocks
Fusion blocks	Reference features, query features	Conditioning features	Injects reference conditions through self-attention + cross-attention	Supplies pixel-aligned conditions to the token decoder
VQ-VAE tokenizer	Continuous ROC map	Discrete token sequence	Compresses continuous coordinates into a finite codebook	Decoder predicts token distributions
Autoregressive decoder	Query features, generated tokens	Next-token probabilities	Models long-range token dependencies and uncertainty	Detokenization produces query ROC
Umeyama solver	Predicted / observed 3D correspondences	$SE(3)$ transform	Recovers a rigid transform by least squares	Composes with reference pose

6.3 Training and supervision

• Training is required. The ROC tokenizer first compresses continuous coordinate maps into a discrete codebook; the main model learns conditional token distributions.
• The autoregressive decoder objective is: $$\mathcal{L}_{AR}=-\sum_k \log p(S_k\mid S_{<k},C_F).$$
• During training, query depth constructs ground-truth ROC maps; inference uses generated ROC maps and query depth for recovery.
• Training data include the FoundationPose dataset and an OO3D-9D subset; testing covers Real275, Toyota-Light, LINEMOD, and YCB-V.
• The training objective and final metrics differ: token NLL constrains discrete coordinate-map generation, while final ADD(-S) also depends on masks, depth, normalization, and Umeyama stability.

6.4 Inference and engineering complexity

• Autoregressive token decoding dominates inference cost; the paper compares accuracy / speed at 64, 16, 4, and 1 steps.
• 16 steps achieve ADD(-S) 75.0 in approximately 0.25 s; 4 steps take approximately 0.13 s and 1 step approximately 0.10 s, with small accuracy drops.
• The decoder configuration in the paper’s table has approximately 0.37B parameters, excluding the tokenizer. Model / tokenizer memory and preprocessing costs need separate evaluation.
• Reproduction challenges include VQ-VAE codebook training, mask / depth preprocessing, dataset-specific reference protocols, and symmetric-object evaluation.

7. Experimental analysis

7.1 Datasets and Input Settings

• Real275: 42 real objects in 6 categories, evaluating novel objects.
• Toyota-Light: 21 rigid household objects under 5 lighting conditions.
• YCB-Video: occluded scenes, with emphasis on ADD / ADD-S AUC when both reference and query are occluded.
• LINEMOD: textureless objects, testing large viewpoint changes.
• Evaluation uses the first frame as reference or randomly samples 2K reference–query pairs. Compared methods differ in input modality and reference count; read the tables accordingly.

7.2 Main results

Setting	CoordAR result	Interpretation
Real275	AR 71.0, ADD(-S) 82.2	Exceeds One2Any’s AR 54.9 and ADD(-S) 41.0
Toyota-Light	AR 62.5, ADD(-S) 82.6	Strong generalization across lighting conditions
Occluded YCB-V	ADD AUC 78.5, ADD-S AUC 95.5	Symmetry-aware evaluation favors results on geometrically symmetric objects
LINEMOD	ADD(-S) recall 75.0	Strong results with one reference and large viewpoint changes

The tables broadly support the authors’ state-of-the-art claim, but multi-view methods and CAD-assisted FoundationPose use richer inputs; these are not all strictly matched-setting comparisons.

7.3 Ablation studies

Ablation	Reported result	My assessment
Convolutional vs. autoregressive ROC decoder	Full autoregressive ADD(-S) 73.6, convolutional decoder 70.9	Supports the benefit of global token dependencies
Remove training- and test-time autoregression	ADD(-S) falls to 60.7 / 68.0 in the corresponding settings	Links progressive generation with spatial coherence
Remove tokenization and directly regress real-valued ROC	ADD(-S) falls to 60.7	Supports discretization for ambiguity modeling
Role-specific vs. modality-decoupled encoding	61.6 vs. 73.6 ADD(-S)	Supports separate RGB and ROC encoding
Test-time steps	Equal accuracy at 64 and 16 steps, with slight drops at 4 / 1 steps	Offers a useful speed-control parameter

7.4 Robustness and Failure Cases

• Occlusion: good ADD-S performance on YCB-V when reference and query are both occluded.
• Symmetry: ADD and ADD-S differ in sensitivity to texture-defined symmetry; the paper highlights discrepancies caused by evaluation metrics.
• Large viewpoints: LINEMOD experiments show good recovery from top / side / back views.
• Dependency risks: large mask errors, depth holes, reflective / transparent objects, and extreme low light are not systematically evaluated, leaving important deployment unknowns.

8. Critical analysis

Strengths

• Clear reformulation: targets correspondence-generation quality as the main bottleneck rather than adding more final refinement.
• Discrete tokens represent multiple solutions and local ambiguity probabilistically, with speed controlled through decoding steps.
• ROC maps plus Umeyama retain geometric interpretability, making error sources easier to analyze than direct pose-parameter regression.

Weaknesses and Implicit Assumptions

• Requires reference depth and precise masks; one-reference estimation with fewer priors still depends on sensors.
• Reference pose is known, so practical systems must first acquire and calibrate the reference.
• Autoregressive generation may depend on token order, codebook quality, and error accumulation. Token count and codebook-training details are crucial for reproduction.
• Some baselines use CAD or more references. Research comparisons must clearly distinguish model-free, one-reference, and multi-view settings.

Practical Deployment Feasibility

This is a candidate module for RGB-D robot perception, but still needs a front end independent of ground-truth masks, missing-depth handling, a low-latency decoder, transparent / reflective-object stress tests, and long-term tracking with real acquired references.

9. Future Work and Improvement Directions

• Extend ROC-map generation to RGB-only input or monocular-depth uncertainty.
• Use confidence-aware token decoding under occlusion and limited visibility.
• Combine autoregressive maps with 3DGS / neural-rendering refinement for a coarse-to-fine pose pipeline.
• Study codebook transfer across categories, sensors, and datasets.
• Add object symmetry groups or uncertainty-aware pose metrics to reduce misinterpretation from texture-related ADD / ADD-S inconsistencies.

10. Implications for my research

• Directly reusable: the three-stage design dense correspondence → structured token → geometric solver.
• Essential baselines: One2Any, FoundationPose, and Any6D, with strictly aligned reference counts, CAD use, depth sources, and mask sources.
• Potential research direction: combine this site’s focus on unseen-object / category-level pose with tokenized 3D representations, studying uncertainty calibration under noisy depth and imperfect masks.

11. Open questions

• On what data distribution is the VQ-VAE tokenizer trained, and can low texture or new categories cause codebook collapse?
• Do training masks / depth resemble real test inputs, or are there strong oracle conditions?
• Is 16-step decoder latency acceptable at robot-control frequencies?
• For symmetric objects, is the predicted ROC map actually unique, or only equivalent under ADD-S?