Deep Reading Notes: Instance-Adaptive and Geometric-Aware Keypoint Learning for Category-Level 6D Object Pose Estimation

• Abbreviation: AG-Pose
• Paper: arXiv / PDF
• Code: GitHub
• Venue: CVPR 2024
• Authors: Xiao Lin, Wenfei Yang, Yuan Gao, Tianzhu Zhang
• Affiliations: University of Science and Technology of China; Jianghuai Advance Technology Center
• Datasets: CAMERA25, REAL275

AG-Pose is a category-level RGB-D method for 6D object pose estimation. It replaces dense correspondence, where every point predicts NOCS coordinates, with a small set of learned instance-adaptive keypoints and their NOCS correspondences. Local neighborhood geometry and global geometry between keypoints are explicitly encoded into keypoint features.

Method inputs and outputs

Item	Content (compiled for this site)
Inference inputs	RGB-D, camera intrinsics $K$, and instance masks / categories from an upstream detector. The core network receives $I_{obj}\in\mathbb R^{H\times W\times3}$ and a camera-frame point cloud $P_{obj}\in\mathbb R^{N\times3}$.
Final outputs	$R\in SO(3)$ and $t\in\mathbb R^3$ mapping the category-canonical object frame to the camera frame, plus three-axis object dimensions $s\in\mathbb R^3$. Test instances may be unseen during training, but their categories are known.
Training inputs	RGB-D target observations from the training categories, masks / categories, ground-truth pose and size, instance models, and NOCS alignment information. Instance models filter outliers for the object-aware Chamfer loss; they are not test inputs.
Training objectives	Pose and size regression, supervision of keypoint NOCS coordinates, and constraints that spread keypoints out and keep them on the object surface. Keypoint coordinates / features and NOCS correspondences are intermediate results.
Output details	The network constructs a rotation matrix from a continuous 6D rotation representation and recovers translation from a residual relative to the point-cloud mean. $t$ and $s$ use length units consistent with depth; NOCS coordinates are normalized.
Requirements	The paper uses a $192\times192$ RGB crop, 1024 observed points, and 96 learned keypoints. Regression heads produce the final pose / size; the keypoints are not passed directly to PnP.

Source: Paper, Section 3. In this category-level setting, “no shape prior” specifically means that no category shape template is supplied at inference. It does not mean that training avoids 3D model supervision.

1. Problem definition

Category-level 6D pose estimation targets rotation, translation, and size of unseen instances within a known category, rather than a particular CAD instance already observed during training. AG-Pose follows the NOCS setting: RGB-D observations, camera intrinsics, category labels, and instance masks yield a target point cloud and RGB crop, from which the object pose and scale in the camera frame are predicted.

The key background is that methods such as NOCS / SPD / SGPA / DPDN / Query6DoF largely rely on dense coordinates or dense correspondence. This works well for known categories, but with large within-category shape variation, local appearance or geometry often cannot uniquely determine a point’s NOCS correspondence. Boundary points, missing depth, and outliers further amplify errors.

2. Method overview

The AG-Pose pipeline has four stages:

1. Feature extractor: The inputs are an RGB crop Iobj and point cloud Pobj. PointNet++ extracts point features, and PSPNet with ResNet-18 extracts RGB features. These are concatenated into object features Fobj.
2. IAKD: Category-shared learnable queries Qcat and object features generate instance-adaptive queries Qins. Keypoint heatmaps then yield keypoint coordinates Pkpt and features Fkpt.
3. GAFA: Local KNN aggregation for each keypoint is followed by global keypoint aggregation, encoding local and global geometric relations into keypoint features.
4. Pose & size estimator: Keypoint NOCS coordinates are predicted, then Pkpt, Fkpt, the predicted NOCS coordinates, and aggregated keypoint features are concatenated to regress R, t, s.

3. IAKD: Why not FPS?

FPS is the most direct way to obtain sparse keypoints. It considers geometric coverage of the point cloud, but does not know which points are more discriminative for pose recovery or necessarily avoid depth noise and outliers. IAKD acts as a learnable keypoint detector: it learns category-shared queries and uses attention to incorporate the current object’s features, making each query an instance-adaptive detector.

IAKD outputs a heatmap H, then uses softmax weights over object points and features to obtain keypoint coordinates/features. Keypoints are thus differentiable weighted combinations of coordinates and features, rather than hard point selections, allowing end-to-end pose supervision to influence them.

The authors introduce two losses to constrain the keypoint distribution:

• Ldiv: Encourages keypoints to spread apart, preventing all keypoints from clustering in one local region.
• Locd: An object-aware Chamfer distance that pulls keypoints toward the object surface while filtering outliers using the instance model.

4. GAFA: Both local and global geometry matter

If IAKD keypoints carry only appearance / point features, they can still form incorrect correspondences in categories with substantial shape variation. GAFA therefore injects geometric context into keypoint features.

Local aggregation finds K neighboring object points for each keypoint and performs attention-like aggregation using relative positions and neighboring point features. It supports fine-grained geometric discrimination between parts of the same instance.

Global aggregation first average-pools all keypoint features into a global geometric summary, then fuses it with each keypoint’s features and relative position encoding. This addresses instances with different overall shapes but similar local parts. The ablations show lower 5° 2cm scores when the entire GAFA module, local aggregation, or global aggregation is removed.

5. Training objectives and implementation details

The total loss is Lall = lambda1 Locd + lambda2 Ldiv + lambda3 Lnocs + lambda4 Lpose. Lnocs supervises keypoint NOCS coordinates, while Lpose supervises rotation, translation, and size. The method uses a 6D rotation representation and predicts translation as a residual relative to the point-cloud mean.

Implementation settings include a crop resized to 192x192, N=1024 sampled points, Nkpt=96 keypoints, K=16 local neighbors, and main feature dimensions of 128/256. Training uses Adam, a triangular2 learning-rate schedule, batch size 24, and one RTX 3090Ti.

Although inference does not require the target instance’s CAD model, training uses category data, NOCS supervision, and constraints involving instance models. Inference still requires depth, camera intrinsics, an object mask, and a category label.

6. Experimental setup

CAMERA25 is a synthetic dataset covering 6 categories, commonly used to train and evaluate category-level NOCS pose estimation. REAL275 is a real RGB-D dataset whose validation set includes unseen instances in each category, making it a more important test of category-level pose generalization.

The metrics include:

• 3D IoU: Evaluates pose and size using 3D bounding-box IoU thresholds of 50 / 75.
• n° m cm: A prediction is correct only if rotation error is below n degrees and translation error is below m cm.

7. Main results

On REAL275, AG-Pose achieves IoU50 / IoU75 / 5°2cm / 5°5cm / 10°2cm / 10°5cm of 83.7 / 79.5 / 54.7 / 61.7 / 74.7 / 83.1. The paper highlights a 5°2cm improvement of 8.7 points over the shape-prior-based DPDN and 5.7 points over the prior-free Query6DoF.

On CAMERA25, AG-Pose achieves IoU50 / IoU75 / 5°2cm / 5°5cm / 10°2cm / 10°5cm of 93.8 / 91.3 / 77.8 / 82.8 / 85.5 / 91.6. It leads Query6DoF on IoU and the 10° metrics but scores slightly lower on the 5° metrics. Distinguish leadership on most metrics from leadership at every threshold when reading this table.

8. Ablation studies

IAKD versus FPS provides the most direct evidence: with the same number of keypoints, FPS reaches 46.2 on 5°2cm and IAKD reaches 54.7. This indicates that learnable, instance-adaptive keypoints suit pose recovery better than sampling for geometric coverage alone.

The keypoint-count experiment reaches 47.9 on 5°2cm with only Nkpt=16, showing that few keypoints can carry substantial geometric information. Nkpt=96 gives the best overall performance; Nkpt=128 does not consistently improve it further.

The loss ablation favors Ldiv + Locd. Replacing Locd with ordinary Chamfer distance lowers 5°2cm from 54.7 to 49.8. Keeping only Locd or removing these keypoint-distribution constraints causes substantial degradation. Spreading keypoints out and placing them on the object surface are therefore critical to the method.

9. Qualitative results

Figure 5 compares AG-Pose and DPDN using NOCS error maps and final pose boxes. DPDN produces more incorrect correspondences on instances with stronger shape variation, such as cameras and cups. AG-Pose has fewer erroneous regions and predicted boxes closer to ground truth. This figure alone cannot establish superiority, but together with the REAL275 results and Figure 4’s error distributions, it supports more reliable keypoint-level geometric correspondence.

10. Relationship to other approaches

Method	Approach	How AG-Pose differs
NOCS	Dense NOCS coordinates	AG-Pose retains NOCS supervision but establishes correspondences at sparse keypoints
SPD / SGPA	Shape-prior adaptation	AG-Pose emphasizes that it uses no category-specific shape prior
DPDN	Dense correspondence + depth	AG-Pose addresses poor generalization of dense correspondence to shape variation in unseen instances
Query6DoF	Sparse queries + dense feature updates	AG-Pose explicitly models geometry using sparse keypoints and establishes keypoint-level correspondences
PVNet	Instance-level keypoints + PnP	AG-Pose uses category-level RGB-D; its keypoints are not fixed CAD keypoints

11. Suitable use cases

AG-Pose is a strong baseline for category-level RGB-D pose estimation, especially for:

• Category-level 6D pose within the NOCS / REAL275 / CAMERA25 framework.
• Generalization to unseen instances, especially when objects in the same category vary substantially in shape.
• Method designs that move from dense correspondence to sparse geometric keypoints.
• Settings that avoid an explicit category shape prior but can use RGB-D and category supervision.

It should not be treated directly as:

• An RGB-only pose method.
• A mask-free / open-vocabulary perception frontend.
• A fully CAD-free / model-free training method.
• An instance-level BOP CAD pose method.

12. My assessment

Authors’ claim: Instance-adaptive keypoints and geometric-aware aggregation improve correspondence generalization to unseen instances in category-level 6D pose estimation.

My analysis: The evidence is fairly strong. Gains on strict REAL275 metrics are substantial, most CAMERA25 metrics improve, and the IAKD / GAFA / loss ablations align with the claims. The most useful lesson is to combine learning the keypoint distribution with aggregating geometric context, rather than simply downsampling dense correspondences into sparse points.

The limitations are also clear. AG-Pose still relies on RGB-D, instance masks, category labels, and NOCS-style training supervision. Locd also uses instance-model information to filter outliers. It therefore addresses correspondence generalization in category-level pose, rather than open-world perception or unsupervised category discovery.

13. Possible research directions

• Replace the fixed Mask R-CNN frontend with a stronger open-vocabulary segmenter and test how mask noise affects keypoint learning.
• Connect AG-Pose’s keypoint-level correspondences to a refinement or render-and-compare backend and examine improvements at strict accuracy thresholds.
• Explore an RGB-only version using monocular depth or learned geometry instead of measured depth.
• Estimate keypoint uncertainty so the pose regressor can downweight unreliable keypoints.
• Combine AG-Pose’s sparse keypoint representation with newer foundation features or 3D representations for more open unseen-object pose estimation.