Paper Reading Notes (Deep): HouseCat6D

1. Key assessment

HouseCat6D improves four weak points in category-level 6D pose data together:

• Observation distribution: free camera motion covers surrounding and lower-hemisphere views beyond the tabletop upper hemisphere;
• Materials: transparent / strongly reflective objects such as glass and cutlery become formal categories;
• Annotation quality: external infrared tracking, a measurement probe, hand–eye calibration, synchronization, and bundle adjustment control point-level error to 1.35–1.73 mm;
• Downstream tasks: the same scenes provide dense parallel-jaw grasp labels.

The benchmark also reveals two facts: VI-Net reaches only 56.4 overall 3D50, leaving realistic category pose far from saturation; even ground-truth NOCS coordinates suffer sensor-depth lifting errors on glass / tube. It exposes both representation and sensor-geometry errors.

2. Dataset size and splits

Item	Count / setting
Objects	194
Categories	10
Scenes	41
Total frames	About 23.5k
Object poses	About 160k
Grasp poses	About 10M
Pose split	34 train / 2 val / 5 test scenes
Frame split	20k train / 1.4k val / 3k test
Object split	124 train / 20 val / 50 test
Grasp split	14 train / 1 val / 1 test scenes

The 10 categories are bottle, box, can, cup, cutlery, glass, remote, shoe, teapot, and tube. Test instances are absent from training, making this a known-category, unseen-instance task.

Each frame can contain:

• RGB;
• active-stereo depth;
• RGBP at four polarization angles;
• camera trajectory;
• Object masks, 6D poses, and CAD meshes;
• Feasible / infeasible grasp poses for some sequences.

RGBD+P describes availability of ordinary RGB, depth, and polarization data; it does not imply a network fusing depth and polarization. Main pose baselines still primarily use RGB-D and do not establish substantial polarization gains for 6D pose.

3. Capture hardware and coordinate chain

Core coordinate frames include:

• Tracker system base $TB$;
• Infrared marker body $MB$;
• Camera center $CB$;
• Checkerboard base $BB$;
• object / measurement-tip coordinates.

An external four-camera infrared system supplies marker poses instead of placing a checkerboard in every image. At least two infrared cameras must see the marker body’s four spheres to recover its transform relative to the tracker base.

Hardware:

• Dedicated sensors for ordinary / polarization RGB and active-stereo depth;
• Hardware triggers synchronize cameras as closely as possible;
• Structured-light scans provide object models;
• Evaporating spray assists glass / reflective-surface scanning;
• A freely movable handheld rig expands workspace and viewpoint coverage.

4. From measurement probe to object pose

4.1 Pivot calibration

The physical probe tip differs from the marker-body origin. Holding the tip fixed while rotating the marker body, multiple observations minimize fixed-point position variance to recover the tip offset. The final calibration uses 18 poses and reports tip-position variance about 0.040 mm.

4.2 Object-pose annotation

Annotators collect 3D surface points with the probe and align them to the mesh:

1. Select surface correspondences for an initial transform;
2. Refine with ICP;
3. Cover several surfaces to avoid one-sided / coplanar degeneracy.

Simulated pivot and tracking errors give estimated mean RMSE about 0.32 mm / 0.43° for this step.

4.3 Hand–eye calibration

The authors collect camera and tracking-body trajectories at multiple positions, then use Horn alignment for a stable offset rather than relying on one checkerboard detection’s closed-form solution. Reported RMSE:

• translation: 0.27 mm;
• rotation: 0.42°.

4.4 Synchronization and trajectory postprocessing

Even with hardware triggers, infrared tracker / camera frequencies and timestamps differ. Trajectory ICP estimates their offset, with simulated error about 0.03 s, still enough to cause significant pose offsets during fast camera motion.

Postprocessing does not start SfM from scratch:

• Existing tracking trajectories initialize it;
• Humans select fixed frames with the best rendered-mask / RGB agreement;
• Fixed frames do not drift freely afterward;
• COLMAP feature matching, structure reconstruction, and bundle adjustment mainly correct motion-induced offsets.

This improves accuracy but adds manual fixed-frame selection. Reproducing capture requires annotation engineering as well as network code.

5. Interpreting annotation accuracy

Table 2 compares object-model point RMSE under combined system error, not a single sensor’s nominal accuracy:

Data / approach	Annotation method	Point RMSE
Ordinary RGB-D lifting	Depth map	≥17 mm
TOD	multi-view	3.4 mm
StereoOBJ	multi-view	2.3 mm
PhoCaL	robot	0.80 mm
HouseCat6D	IR tracker	1.35–1.73 mm

The 1.35 and 1.73 bounds reflect different system-error combinations. SfM postprocessing gains cannot be fully quantified, so this range is not a directly measured error for every pixel.

Accuracy is slightly below robot-arm ground truth, but the handheld rig enables:

• Larger workspaces;
• More varied backgrounds;
• More uniform camera poses;
• Checkerboard-free images.

6. Viewpoint coverage

Compared with NOCS, StereoOBJ-1M, PhoCaL, and Wild6D, HouseCat6D camera orientations provide:

• Coverage beyond the upper hemisphere;
• Multidirectional trajectories for every category;
• A wider object-to-camera distance range;
• Occlusion and nearby objects within real scenes.

Restricting VI-Net training to PhoCaL-like coverage gives:

Training coverage	3D25	3D50
HouseCat6D full	80.7	56.4
reduced VI-Net*	68.4	31.3

The 3D50 drop of 25.1 points makes pose coverage a quantifiable generalization variable beyond qualitative dataset diversity.

7. Category-level pose benchmark

Overall:

Method	3D25	3D50
NOCS	50.0	21.2
FS-Net	74.9	48.0
GPV-Pose	74.9	50.7
VI-Net	80.7	56.4

VI-Net 3D25 / 3D50 by category:

• bottle: 90.6 / 79.6;
• box: 44.8 / 12.7;
• can: 99.0 / 67.0;
• cup: 96.7 / 72.1;
• remote: 54.9 / 17.1;
• teapot: 52.6 / 47.3;
• cutlery: 89.2 / 76.4;
• glass: 99.1 / 93.7;
• tube: 94.9 / 36.0;
• shoe: 85.2 / 62.4.

High glass 3D IoU does not guarantee accurate rotation: IoU can tolerate approximate axial symmetry and size / center variation. Low box / remote scores combine intra-class appearance, thin structures, occlusion, and NOCS canonicalization effects.

8. Depth-error ablation

Ground-truth NOCS maps replace network predictions while retaining sensor-depth lifting:

• Overall 3D25 / 3D50 reaches 96.7 / 93.6;
• This attributes much ordinary-category error to 2D NOCS prediction;
• Glass reaches only 80.0 / 68.0, well below VI-Net 99.1 / 93.7;
• Strict tube performance also suffers from depth error.

Perfect canonical coordinates plus sensor depth can thus underperform learned 3D pipelines: transparent / reflective depth points may not lie on real surfaces, and perfect 2D coordinates cannot repair incorrect 3D measurements. Learned priors can instead bypass some missing data.

At 75% IoU, ground-truth NOCS + sensor depth averages only 22.6%, exposing lifting error more strongly than loose 3D50.

9. Grasp annotation and real experiments

Grasp generation:

1. Antipodal sampling on meshes;
2. Isaac Gym separates successful / failed candidates;
3. Object poses transform grasps into the tracker frame;
4. Reconstruct backgrounds and check collisions;
5. Project remaining grasps into each camera frame.

KGN predicts gripper keypoints from RGB-D and solves 6D grasp pose through PnP. Reported results include:

• Test-set grasp coverage and object success rates;
• Real grasp success on a Franka 7-DoF robot;
• 15 trials per object.

Real results (without / with keypoint refinement):

• box: 80.0 / 80.0;
• cup: 66.7 / 66.7;
• glass: 66.7 / 73.3;
• remote: 53.3 / 33.3;
• unknown: 53.3 / 60.0.

Refinement does not consistently help; remote degrades clearly. The small trial count cannot tightly estimate category success, but demonstrates a complete perception-to-manipulation use of the data.

10. Limitations and usage risks

• Only 10 categories, not an open-world household dataset;
• Main pose benchmarks do not systematically use polarization; its benefit remains for future work;
• Active-stereo depth still fails on transparent / reflective surfaces;
• Bundle adjustment requires manually chosen fixed frames;
• 1.35–1.73 mm is a system-error estimate, not independent per-frame GT error;
• Preserve both scene-level and instance-level splits; random frame splits leak information;
• Loose 3D IoU thresholds can hide rotation / symmetry issues;
• Real grasping has only 15 trials per object, leaving wide statistical intervals;
• Complex category imbalance and about 19 objects per category still limit coverage;
• Main tables do not fully separate detector / segmentation errors from pose error.

11. Reproduction and usage checklist

• Follow the official 34 / 2 / 5 scene split rather than splitting frames randomly;
• Keep train / val / test object instances disjoint;
• Specify RGB, RGB-D, RGB-P, or all modalities;
• Align intrinsics, units, and invalid masks for depth-to-cloud conversion;
• Use official object origins, mesh scales, and symmetries;
• Report overall and per-category 3D25 / 3D50, ideally adding rotation / translation metrics;
• Separate detection, segmentation, lifting, and pose-solver error;
• Report depth-valid ratios separately for glass / cutlery / tube;
• Compare full and reduced viewpoint coverage;
• Report GCR, OSR, and real GSR together in grasping;
• Record each real robot failure type alongside mean success;
• Establish an RGB-D baseline for polarization studies; data availability alone does not prove modality effectiveness.

12. Overall assessment

HouseCat6D is a strong 2024 category-pose dataset paper, connecting hardware, calibration, trajectories, materials, views, pose benchmarks, and grasping. VI-Net 80.7 / 56.4, reduced coverage 68.4 / 31.3, and category-dependent ground-truth NOCS + noisy-depth behavior identify interpretable sources of difficulty beyond dataset size.

It is not a final benchmark: categories remain limited, polarization underused, transparent depth fragile, and grasp statistics small. Its clear role is: a public, high-precision, wide-view, multimaterial stress test for realistic household category pose, with pose and grasping in one coordinate chain.