Object Pose Estimation · 2025 · ICCV · CCF A

CapeLLM: Support-Free Category-Agnostic Pose Estimation with Multimodal Large Language Models

Query images and textual keypoint descriptions replace conventional support images and annotations. MLLM visual-language tokens, dynamic round training, and flexible coordinate decoding support category-agnostic keypoint pose estimation; this is not BOP-style 6D pose estimation.

Paper Code

Research area

Object Pose Estimation

Task level

Category-level

Object setting

Unseen objects

Input modalities

RGBtext description

Requirements

keypoint text descriptionpre-trained vision encoderpre-trained LLM

Full paper metadata Authors, affiliations, research area, datasets, metrics, method tags, and reading status Expand

Authors: Junho Kim, Hyungjin Chung, Byung-Hoon Kim
Affiliations: EverEx / Yonsei University
Conference / journal: Conference · ICCV · CCF A
Research area: Object Pose Estimation
Code availability: Code available
Reading status: Overview:Available Deep Dive:Available
Task setting: Category-level · Unseen objects
Datasets: MP-100
Evaluation metrics: PCK@0.2 PCK@0.1 1-shot accuracy 5-shot accuracy
Method tags: Category-level Pose Estimation Open Vocabulary / Language Prompts Keypoints + PnP Synthetic Data & Foundation Models Absolute Pose Estimation
Requirements: keypoint text descriptionpre-trained vision encoderpre-trained LLM

Reading Notes

Overview / Deep Dive

Available src/paper-notes/capellm-iccv2025/coarse.md

Paper overview notes

Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.

Paper Overview: CapeLLM: Support-Free Category-Agnostic Pose Estimation with Multimodal Large Language Models

1. Basic information

Year / venue: 2025 / ICCV; reading mode: Overview.
Task: Category-agnostic keypoint pose / correspondence, rather than BOP-style object 6D pose.
Paper: ICCV Open Access; code: GitHub.

2. Positioning in one sentence

CapeLLM replaces annotated support images with a query image and a textual description of the target keypoint. It expresses category-agnostic keypoint localization as coordinate-token generation by an MLLM, using dynamic-round training to improve spatial reasoning.

2.1 Guide to the paper’s figures and tables

Figure or table	What to examine
Figure 2	How DINOv2 visual tokens and text instructions enter the LLaMA decoder and produce coordinate strings.

CapeLLM Figure 2: multimodal language model architecture — Original paper, Figure 2: CapeLLM vision–language keypoint decoding. Source: ICCV Open Access. Copyright belongs to the paper's authors and publisher; reproduced here solely for paper discussion.

3. Method overview

DINOv2 extracts image tokens. Tokenized keypoint names, descriptions, and questions are concatenated with visual tokens for LLaMA3.1.
The decoder generates quantized coordinates as text. The training design includes instructions and dynamic rounds, with inference refinement for unseen keypoints.
Outputs are 2D keypoint poses. These could feed downstream geometric pose estimation, but the paper does not complete a CAD/PnP 6D pose pipeline.

4. Experimental overview

Dataset: MP-100, comparing 1-shot / 5-shot CAPE using keypoint-localization metrics such as PCK@0.2.
Authors’ claim: Outperforms earlier CAPE methods in 1-shot and even 5-shot settings without support images.
My analysis: Language lowers support-annotation costs, but description quality, ambiguity, fine-grained parts, and grounding under occlusion remain key risks.

5. Strengths, limitations, and implications

Strength: Separates target-keypoint semantics from examples in support images, exploring an open-vocabulary entry point for pose.
Risks: It does not estimate 6D rigid pose. Coordinate-string quantization and LLM hallucination limit precision.
Implication: Language can specify objects/keypoints within pose pipelines, beyond detection prompts alone.

Available src/paper-notes/capellm-iccv2025/deep.md

In-depth paper notes

Support reproduction, comparison, related-work writing, and identifying opportunities for further research.

Deep Reading Notes: CapeLLM: Support-Free Category-Agnostic Pose Estimation with Multimodal Large Language Models

1. Basic information

Authors / affiliations: Junho Kim, Hyungjin Chung, Byung-Hoon Kim; EverEx, Yonsei University.
Year / venue: 2025 / ICCV; reading mode: Deep reading.
Paper: ICCV Open Access; code: GitHub.

2. Positioning in one sentence

CapeLLM addresses category-agnostic keypoint pose. A query image and detailed language specify which point to locate, replacing support images with keypoint annotations. DINOv2 + LLaMA3.1 then generate discretized coordinates.

2.1 Guide to the paper’s figures and tables

CapeLLM Figure 2: visual token and language instruction architecture — Original paper, Figure 2: Visual tokens and textual instructions enter the LLM, which directly decodes a keypoint coordinate string. Source: ICCV Open Access. Copyright belongs to the paper's authors and publisher; reproduced here solely for paper discussion.

3. Task boundaries

Inputs: Query image, keypoint question/name, and description; no same-category support image.
Output: Target-keypoint 2D coordinates, evaluated on MP-100 with metrics such as PCK@0.2.
Relationship to 6D object pose: Could serve as an open-category keypoint/correspondence frontend, but does not output rigid-object $R,t$ or include CAD + PnP / symmetry handling.

4. Method details

Module	Input	Output	Purpose
Visual encoder	Query image	DINOv2 visual tokens	Cross-category visual representations
Text instruction	Keypoint description, question	Language tokens	Specify the semantic point to locate
LLaMA3.1 decoder	Visual + text tokens	Coordinate-token distribution	Spatial reasoning from multimodal context
Coordinate decoding	Token sequence	2D keypoint	Treat coordinates as language generation rather than a fixed Gaussian head
Dynamic-round training	Multiple rounds / target keypoints	More robust spatial reasoning	Reduce limitations of fixed support pairs or single-round training

The authors emphasize prompt design, training strategy, and flexible distribution decoding as factors in fine-grained CAPE with MLLMs.

5. Experimental analysis

MP-100 1-shot and 5-shot settings are the main comparisons; earlier CAPE methods typically require support sets.
Authors’ claim: Achieves new state-of-the-art PCK@0.2 without support images, maintaining advantages in 1-shot / 5-shot comparisons.
My analysis: Additional tests should cover description paraphrases, incorrect descriptions, visually similar keypoints, and occlusion. These directly stress-test replacing support examples with language.

6. Critical analysis and reproduction

Strength: Reduces support-annotation demands by changing input information, using language for object-part semantics rather than fixed category indices.
Limitations: Coordinate-token quantization bounds pixel precision. Language priors may substitute typical locations for actual image evidence. It is not a complete 6D pose solution.
Reproduction priorities: Keypoint descriptions, coordinate vocabulary, image-crop normalization, DINOv2/LLaMA token connectors, and dynamic-round sampling.
Next steps: Generate multiple keypoint / object hypotheses from language and combine them with 2D–3D matching + PnP for open-world 6D pose. Calibrate confidence under language ambiguity.

7. Research implications

Language can replace keypoint semantic specification in open scenes, but does not necessarily replace geometric solving. These should be composable modules.

8. Additional original-paper results

CapeLLM qualitative keypoint localization and main MP-100 results — Excerpt from the original paper's experimental page: Support-free keypoint-localization visualizations and main MP-100 comparisons against support-based methods. Source: ICCV 2025 paper PDF. Copyright belongs to the paper's authors and publisher; reproduced here solely for paper discussion.

The main table supports efficient semantic-keypoint localization across categories without support images. Qualitative examples show language distinguishing some visually similar parts. However, PCK allows normalized distance error and is not equivalent to precise 6D correspondence. Connecting to PnP requires several geometrically well-distributed keypoints per prediction and checks that quantization, occlusion, and incorrect descriptions do not introduce systematic bias.

Related papers

2026 · CVPR

3PT

3PT unifies detection, segmentation, and 6-DoF pose with two Transformers for 2D / 3D object-centric understanding. Continuous refinement of object representations and rendered-feature alignment support zero-shot cross-object perception on BOP-H3, BOP-Industrial, and BOP-Classic.

2026 · AAAI

6DAttack

Studies backdoor attacks on PnP-based and end-to-end 6-DoF pose pipelines. Structurally 3D triggers and modified pose labels are injected into training data so that models behave normally on clean inputs but produce controlled incorrect poses when triggered.

2026 · ICRA

Active Tactile Exploration

A violation-implicit loss incorporates physical-constraint violations into joint estimation of freely moving unknown convex rigid-body trajectories and cuboid / 20-vertex polytope shapes. Observed / Fisher information at a local MLE estimates EIG to select the next touch. In simulation, the EIG strategy yields stable errors below 4 mm after 5 cuboid actions; real Trifinger results at step 5 substantially outperform random selection. Convex geometry, local optima, multiple initializations, and tactile / proprioceptive errors limit the method.