CAP-Net: A Unified Network for 6D Pose and Size Estimation of Categorical Articulated Parts from a Single RGB-D Image
CAP-Net jointly predicts part classes, center offsets, and NPCS with a single-stage RGB-D network, recovering pose, size, and joint states of category-level articulated parts on RGBD-Art.
Task: Jointly estimate segmentation, 6D pose, and size of each part in a category-level articulated object from one RGB-D image.
Reading mode: Overview.
2. Positioning in one sentence
CAP-Net combines fine-grained RGB semantics and depth geometry in a single-stage network, directly predicting part classes, centroid offsets, and NPCS, then fitting point clouds to recover each part’s pose / size.
3. Problem and motivation
Multistage pipelines that segment first and predict coordinates per part can amplify small-part errors and frontend failures.
Traditional point-cloud methods often neglect dense RGB semantics; sim-to-real depth noise also hinders robot deployment.
4. Method overview
An RGB-D backbone predicts point-wise classes, center offsets, and NPCS.
Group points into part instances by category and center distance.
Align NPCS with corresponding point clouds to fit pose / size. RGBD-Art supports training with realistic sensor-noise simulation.
5. Experiments at a glance
The authors report segmentation, part pose, and size on RGBD-Art, together with real RealSense robot experiments.
Evaluation requires more than one 6D pose score: Examine small / large parts, seen / unseen categories, and sim-to-real conditions together.
RGBD-Art examples and statistics: 1,045 objects, approximately 63K RGB-D images, and approximately 408K part annotations, with sensor-style depth noise for sim-to-real transfer. Source: original CVPR 2025 paper. Copyright belongs to the paper's authors and publisher; reproduced here solely for paper discussion.
The paper contributes both a method and data. RGBD-Art contains 1,045 objects, approximately 63K images, and 408K part annotations, jointly supervising small-part categories, instances, NPCS, pose, and size. However, predefined part types underpin the dataset, so strong synthetic results do not automatically extend to arbitrary unknown joint topologies.
6. Quick assessment
Contribution: Consolidates multitask articulated-part perception into a practical RGB-D pipeline.
Boundaries: Requires object-level masks, a part taxonomy, and RGBD-Art category definitions. It does not provide open-world articulation understanding.
Available src/paper-notes/cap-net/deep.md
In-depth paper notes
Support reproduction, comparison, related-work writing, and identifying opportunities for further research.
Deep Reading Notes: CAP-Net: A Unified Network for 6D Pose and Size Estimation of Categorical Articulated Parts from a Single RGB-D Image
Key problem: Semantic segmentation, instance grouping, pose, and size of small parts in category-level articulated objects are coupled. Serial pipelines amplify errors.
Main answer: Use point-wise multihead predictions from one RGB-D network as a shared frontend, followed by clustering and NPCS fitting for each part’s pose / size.
2. Positioning in one sentence
CAP-Net shares representations first and fits individual part instances afterward, connecting category-level articulated-part pose to deployable RGB-D sensor inputs through a single-stage approach.
2.1 Guide to the paper’s figures and tables
Figure / table
What to look for
Figure 2
Whole-object versus part-based pose and the difficulty of part canonicalization.
Figure 4
How RGB, depth, and SAM2 / FeatUp features enter the unified network and its three outputs.
Tables 2–3
Compare part segmentation and pose separately for seen / unseen categories.
Figure 5
RGBD-Art and RealSense qualitative results, especially small-part and occlusion failures.
Original paper, method-diagram region: CAP-Net predicts part classes, centroid offsets, and NPCS from RGB-D semantic and geometric features, then fits pose / size. Source: CVF Open Access. Copyright belongs to the paper's authors and publisher; reproduced here solely for paper discussion.
3. Problem setting and boundaries
Inputs are one RGB-D image and an object-level mask. Outputs include categories, instances, 6D poses, and sizes for multiple articulated parts within that object.
This is category-level generalization within predefined part categories, rather than parsing unknown topologies or arbitrary joint models.
NPCS fitting depends on depth, point-cloud quality, and sufficient visible surface for every part.
4. Method details
Unified features: RGB supplies dense semantic cues; depth / point branches supply geometry. Pretrained visual features strengthen small-part recognition.
Three output heads: Each point predicts a class label, offset to its part centroid, and Normalized Part Coordinate Space (NPCS).
Instance-to-pose: Cluster same-class points by centroid distance, then align each part’s NPCS to its observed point cloud to recover rotation, translation, and size through a similarity transform.
Data design: RGBD-Art combines photorealistic synthetic RGB with realistic sensor-style depth noise for sim-to-real transfer.
5. Experimental analysis
Segmentation, pose, and size are reported separately, with seen / unseen object categories distinguished. These are necessary dimensions for assessing category-level and part-level performance.
RGBD-Art tests training transfer under sensor-depth noise, beyond adding more data. RealSense manipulation provides supplementary evidence rather than replacing quantitative benchmarks.
Focus on small parts when reading results: Large visible parts can dominate averages.
Original CAP-Net paper, main results and ablations: Part segmentation, pose / size, multitask training, and feature modules explain final accuracy at four levels. Source: original CVPR 2025 paper. Copyright belongs to the paper's authors and publisher; reproduced here solely for paper discussion.
RGBD-Art part-segmentation AP50 is 53.58 for seen categories and 19.38 for unseen categories. This sharp drop identifies category generalization as a major bottleneck. The complete pose / size system reports mean rotation error of 10.39°, relative translation error of 0.055, relative scale error of 0.026, and 3D IoU of 56.23. Accuracy at 5° / 10° is 33.91 / 58.44. Joint multitask training reduces rotation error from 15.30° to 10.39° and raises 3D IoU from 51.75 to 56.23 versus separate-task training, supporting shared semantic–geometric representations.
Ablations prevent attributing gains simply to a larger backbone. Base PNet has rotation error of 21.30° and 10° accuracy of 38.67. Adding SAM features gives 12.20° / 47.90; the complete combination reaches 10.39° / 58.44. FeatUp or DINOv2 alone does not consistently exceed the SAM version, showing that pretrained-feature usefulness depends on fusion with point geometry and multitask heads.
Original CAP-Net paper, qualitative and robot experiments: Examine manipulation success alongside part-axis alignment. CAP-Net succeeds in 28/30 trials across three part types, compared with GAPartNet's 5/30. Source: original CVPR 2025 paper. Copyright belongs to the paper's authors and publisher; reproduced here solely for paper discussion.
Real-robot experiments connect perception to executable tasks. GAPartNet succeeds in 1/10 hinge-handle, 2/10 drawer, and 2/10 hinge-lid trials, totaling 5/30. CAP-Net reaches 9/10, 10/10, and 9/10, totaling 28/30. This is practically meaningful but uses a small sample and does not separate mask, part-classification, pose, grasp-planning, or control failures. It establishes greater reliability of the complete demo pipeline on this test set, rather than resolution of every vision module.
6. Authors’ claims and my assessment
Authors’ claims
My assessment
A unified single-stage network outperforms multistage part pipelines
Simpler task interfaces; separate mask / clustering effects on pose still need reporting.
RGB semantics improve small parts
Plausible, but coupled to backbone, resolution, and RGBD-Art rendering quality; cross-ablations are needed.
7. Reproduction considerations and limitations
Check part canonical frames, scale conventions, depth units, and camera intrinsics before NPCS alignment.
Clustering radius and minimum point count substantially affect small-part instance recall.
Object-level masks are assumed available. If the frontend omits articulated parts, the unified backend cannot recover absent observations.
Reproduce instance AP, part pose / size, and robot success together, retaining stage labels for every failure. Otherwise, improved offline averages are difficult to distinguish from improved manipulation.
