Paper Reading Notes (Deep): GraspClutter6D

1. Positioning of the paper

GraspClutter6D is data infrastructure linking perception annotations, grasp candidates, and real robot validation. The authors identify several gaps in existing data:

• Limited object counts or material coverage;
• Sparse scenes, insufficient occlusion, and too few multiple instances of the same category;
• RGB-D pose data separated from grasp annotations;
• Many synthetic grasp labels unvalidated in real clutter;
• Fixed camera viewpoints and insufficient cross-sensor and viewpoint coverage.

The dataset therefore supports instance segmentation, 6D object pose estimation, 6-DoF grasp detection, and training for real grasping.

2. Objects, scenes, and capture hardware

2.1 200 objects

The objects comprise 100 household, 50 industrial, and 50 warehouse items, covering groceries, kitchen, hygiene, office, electronics, shoes, toys, tools, parts, 3D prints, APC objects, boxes, cylinders, cardboard, and bags.

Of these, 108 objects were scanned with Artec Leo and 92 came from existing resources such as YCB, HOPE, GraspNet, DexNet, and APC. Experts repaired the newly scanned models into watertight meshes; reflective / transparent objects were coated with matte spray to capture their geometry. The released meshes use a consistent scale.

2.2 Rigid four-camera platform

Mounted on the UR5 end effector:

• RealSense D415;
• RealSense D435;
• Azure Kinect;
• Zivid One+ M.

The first three represent common low-cost RGB-D sensors; Zivid provides high-quality point-cloud fusion and annotation. Intrinsics and extrinsics are calibrated using ChArUco, MC-Calib, PnP-RANSAC, and bundle adjustment. The authors also fit a depth correction:

• D415 mean absolute depth error: 10.2 → 5.0 mm;
• D435: 15.6 → 7.0 mm;
• Azure Kinect: 25.4 → 15.8 mm.

For each bin / shelf / table category, 5 furniture variants and 5 background variants give 75 environment configurations in total. Each scene randomly contains 5–20 objects, captured from 13 peripheral / central Zivid viewpoints; the cameras together produce 52K RGB-D images.

3. 6D pose annotation pipeline

3.1 Align once, propagate across views

The 13 Zivid views are first fused into a scene point cloud. Annotators align all object CAD models in this high-precision cloud rather than editing 52K images frame by frame. Camera extrinsics then propagate each object pose to the D415, D435, Azure Kinect, and Zivid images.

3.2 Coarse-to-fine alignment and quality heatmaps

The custom tool simultaneously displays:

• RGB;
• segmentation;
• Differences between captured depth and rendered CAD depth;
• The object model and scene point cloud.

Annotators first adjust the pose coarsely, then run ICP refinement. The tool color-codes pixels with $|\delta|\le5$ mm, 5–15 mm, and >15 mm residuals, targeting a mean residual below 5 mm per object. All poses undergo double review by independent experts.

The result is 735,545 annotated object instances with 6D poses and segmentation masks.

3.3 Interpreting annotation-quality statistics

The authors define

$$\delta=d_r-d_c,$$

where $d_r$ is the CAD depth rendered at the annotated pose and $d_c$ is captured depth. For opaque objects, Table II reports:

| Sensor | mean $|\delta|$ | median $|\delta|$ | std | |---|---:|---:|---:| | Zivid | 3.22 mm | 1.55 mm | 11.10 mm | | D415 | 5.71 mm | 3.77 mm | 14.67 mm | | D435 | 7.02 mm | 4.58 mm | 13.82 mm | | Azure Kinect | 13.85 mm | 6.83 mm | 32.59 mm |

These numbers do not directly equal pose error. The residual also contains:

• 6D pose misalignment;
• mesh reconstruction error;
• depth quantization / multipath / missing pixels;
• Sensor artifacts caused by object materials;
• Occlusion-boundary and segmentation errors.

Transparent / strongly reflective objects are excluded from the fair quality comparison because of depth artifacts, leaving a difficult subset that needs additional auditing.

4. Generating grasp annotations

Grasp labels have two levels:

1. Object level: sample parallel-jaw gripper poses on each watertight mesh and score them using a force-closure metric, decreasing the friction coefficient from 1.0 to 0.1;
2. Scene level: transform object grasps into the scene using ground-truth 6D object poses, then check collisions against the reconstructed scene point cloud.

The paper’s wording “14.4K grasps per point” for each model is potentially ambiguous; the final release contains approximately 9.3B collision-free 6-DoF grasp poses, averaging about 178K per image.

Three concepts must be distinguished:

• Force closure: static grasp feasibility under ideal rigid-body / friction assumptions;
• Collision-free: the candidate gripper pose has no obvious collision with the observed scene;
• Real success: the robot completes a grasp after following an actual approach trajectory.

The 9.3B figure covers only the first two concepts, not 9.3B physical executions, and does not cover gripper compliance, sensing errors, dynamic collisions, or the approach path.

5. Scene statistics and coverage

The paper reports:

• 1000 real scenes;
• 200 objects;
• Mean 14.1 object instances / image;
• Mean visibility 77.1%;
• 62.6% of instances have visibility $\le0.95$;
• Object–camera distance 0.31–1.37 m, mean 0.77 m;
• object diameter 4.5–47.3 cm;
• weight 6–2250 g;
• Mean viewpoint coverage 67.6%, versus 57.8% for GraspNet-1B.

Multiple cameras and 13 viewpoints generate multiple images of the same physical scene. Image counts in the statistics are large, but scene independence is still measured over 1000 scenes. Evaluation splits should group by scene rather than randomly splitting frames.

6. Real robot grasping results

The authors test unseen YCB objects with a UR5 + Robotiq gripper and RealSense D435. Settings include:

• packed 5 objects;
• pile 5 / 10 / 15 objects.

Each setting runs 20 rounds per method, with a limit on consecutive failures. Contact-GraspNet is compared using different training data:

Method / training data	Packed 5	Pile 5	Pile 10	Pile 15
AnyGrasp / GraspNet-1B++	62.7	66.4	60.7	59.6
Contact-GraspNet / ACRONYM	32.1	27.8	34.2	25.7
Contact-GraspNet / GraspNet-1B	77.5	62.5	58.8	54.9
Contact-GraspNet / GraspClutter6D	93.4	77.2	74.0	67.9

With the same backbone / inference algorithm, training data is the main variable, providing stronger evidence of data value than offline AP alone. However, this does not fully isolate causality: data volume, sampling density, label distribution, and scene style change together.

The paper observes that even an accurate grasp pose can fail when a simple straight-line approach trajectory collides with nearby objects. Future work needs gripper-aware / reactive motion in addition to better static grasp scores.

7. Benefits for downstream perception training

On YCB-Video, the additional data comprises GraspNet-1B 56K versus GraspClutter6D 14K images:

Mask2Former segmentation

• YCB-V only: AP 57.4;
• • GraspNet-1B: 61.9;
• • GraspClutter6D: 66.1.

FFB6D pose

Both columns in Table VI are labeled ADD-S, but the main text indicates the standard ADD(-S) / ADD-S combination. Values:

• YCB-V only: 63.48 / 81.46;
• • GraspNet-1B: 64.24 / 85.09;
• • GraspClutter6D: 68.80 / 85.76.

Larger gains with roughly one quarter of the additional images suggest that dense clutter and object diversity provide more information per image. A fairer comparison should also control effective object instances, total training steps, and category overlap.

8. 6D object pose benchmark

The authors compare on the standard 21 YCB-V objects:

• instance-level: FFB6D, GDR-Net;
• foundation / novel-object: MegaPose, FoundationPose.

The metrics are ADD(-S) / ADD-S AUC, with thresholds up to 10 cm, stratified into low, medium, and high occlusion by visibility.

Method	All	Low occ.	Medium occ.	High occ.
FFB6D	36.2 / 59.3	45.1 / 69.5	35.1 / 59.5	13.2 / 26.8
GDR-Net	44.5 / 59.4	52.6 / 69.0	48.2 / 64.0	27.7 / 39.3
MegaPose	69.6 / 78.6	77.9 / 86.7	71.5 / 80.5	42.3 / 51.7
FoundationPose	70.5 / 92.3	76.2 / 94.1	70.0 / 92.8	55.0 / 85.9

The large ADD-S versus ADD(-S) gap shows that symmetry handling strongly affects the overall conclusion. FoundationPose remains strong under heavy occlusion, but accuracy under strict correspondences drops substantially, making the data useful for visibility-aware confidence or tracking-recovery analysis.

9. 6-DoF grasp detection benchmark

All four methods are trained on 100 GraspNet-1B scenes and tested on both GraspNet-1B and GraspClutter6D. The metric is force-closure AP over friction coefficients 0.2–1.2. All methods drop substantially on GraspClutter6D:

• Contact-GraspNet: 19.07 → 10.73 AP;
• GraspNet-Baseline: 23.43 → 17.95;
• ScaleBalancedGrasp: 44.85 → 22.69;
• EconomicGrasp: 51.63 → 21.67.

GraspClutter6D is therefore not an easy nearby-domain test set already solved by GraspNet training; the results also highlight the effects of training domain and evaluation clutter.

10. Main limitations

1. Analytic grasp counts are easy to misread. The 9.3B labels are candidates, not independent real trials.
2. The pose-quality metric mixes error sources. Depth residuals cannot individually identify rotation, translation, mesh, or sensor error.
3. Transparent / reflective quality statistics are incomplete. These difficult materials appear in the data but are excluded from the opaque-only comparison.
4. Multiview correlation. 52K images are not 52K independent scenes; splits must be scene-wise.
5. Overlapping object sources. Relationships between public objects such as YCB and the downstream YCB-V benchmark must be explicit in unseen / transfer claims.
6. Simplified grasp dynamics. Force closure and point-cloud collision checking do not model approach planning, compliance, slip, or closed-loop visual / tactile feedback.
7. Controlled capture setup. A rigid UR5 rig and well-calibrated static clutter still differ from handheld-camera / moving-object scenes.

11. Overall assessment

The main contribution is an evidence chain from CAD-aligned poses to scene-level grasp candidates and then real robot success. For the 6D pose community, GraspClutter6D is particularly useful for:

• Pose robustness under heavy occlusion;
• Scene-level failures of RGB-D foundation pose estimators;
• Downstream training gains from pose / segmentation data;
• Linking pose uncertainty with grasp planning;
• Multisensor depth bias and annotation-quality research.

With strict control of scene splits, object overlap, and the opaque-only quality caveat, this dataset can become an important benchmark connecting pose estimation and manipulation.