Object Pose Estimation · 2024 · ECCV · CCF B

PACE: A Large-Scale Dataset with Pose Annotations in Cluttered Environments

PACE provides 300 real cluttered videos, 55K frames, and 258K pose annotations spanning 238 objects, 43 categories, and rigid / articulated objects, plus 100K PACE-Sim frames. PPF reaches 52.2 BOP AR on known instances versus only 4.4–9.0 for learned methods. Controlled experiments separate performance collapses caused by sim-to-real transfer and scaling to the full object set.

Paper Project Code

Research area

Object Pose Estimation

Task level

Instance / Category

Object setting

Seen / Unseen

Input modalities

real RGB-D videosynthetic PBR RGB-Dscanned object CAD modelspart-level articulated annotations

Requirements

calibrated three-camera capture system for annotationobject scans and canonical alignmentsynthetic training dataGT detections for reported pose baselinesGT first pose for trackingGT masks / sizes for several tracking baselines

Full paper metadata Authors, affiliations, research area, datasets, metrics, method tags, and reading status Expand

Authors: Yang You, Kai Xiong, Zhening Yang, Zhengxiang Huang, Junwei Zhou, Ruoxi Shi, Zhou Fang, Adam W. Harley, Leonidas Guibas, Cewu Lu
Affiliations: Stanford University / Shanghai Jiao Tong University / Horizon Robotics / UC San Diego
Conference / journal: Conference · ECCV · CCF B
Research area: Object Pose Estimation
Code availability: Code available
Reading status: Overview:Available Deep Dive:Available
Task setting: Instance / Category · Seen / Unseen
Datasets: PACE PACE-Sim NOCS-REAL275
Evaluation metrics: BOP AR VSD MSSD MSPD 3D IoU ADD ADD-S Rotation Error Translation Error ADD / ADD-S AUC
Method tags: Instance-level Pose Estimation Category-level Pose Estimation dataset rgb-d-video pose-tracking articulated-object real-world-clutter multi-camera-annotation synthetic-to-real scalability Absolute Pose Estimation
Requirements: calibrated three-camera capture system for annotationobject scans and canonical alignmentsynthetic training dataGT detections for reported pose baselinesGT first pose for trackingGT masks / sizes for several tracking baselines

Reading Notes

Overview / Deep Dive

Available src/paper-notes/pace-eccv2024/coarse.md

Paper overview notes

Quickly assess the paper's scope, problem setting, core method, experimental findings, and whether to read it in depth.

Paper Reading Notes (Coarse): PACE

1. Basic information

Title: PACE: A Large-Scale Dataset with Pose Annotations in Cluttered Environments
Authors: Yang You, Kai Xiong, Zhening Yang, Zhengxiang Huang, Junwei Zhou, Ruoxi Shi, Zhou Fang, Adam W. Harley, Leonidas Guibas, Cewu Lu
Affiliations: Stanford University; Shanghai Jiao Tong University; Horizon Robotics; UC San Diego
Year / venue: 2024 / ECCV
Paper: Published ECVA PDF · ECCV page · Data and code
Reading mode: Overview

2. Positioning in one sentence

Three calibrated RealSense D415 cameras capture 300 cluttered real videos, yielding 55K frames, 258K pose annotations, 238 objects, and 43 categories. PACE covers instance / category estimation, model-based / model-free tracking, and rigid / articulated objects. Its main result is not a new SOTA, but evidence that methods scoring highly on BOP / NOCS fail sharply in large, cluttered real scenes.

3. Guide to figures and tables

PACE 3D scanning, three-camera acquisition, and pose annotation pipeline — Original Figure 3: scan objects, align canonical frames and annotate symmetry, then capture calibrated multiview RGB-D for poses and occlusion-aware masks. Source: published ECVA paper. Image copyright belongs to the authors and Springer Nature; reproduced here for paper analysis.

PACE instance-level and category-level pose baseline results — Original Tables 2–3: conventional RGB-D PPF reaches 52.2 BOP AR, far above CosyPose / SurfEmb / GDRNPP at 4.4 / 9.0 / 7.0. Category-level slash-separated values denote rigid / articulated objects; strict pose scores are generally low. Source: published ECVA paper. Table copyright belongs to the authors and Springer Nature; reproduced here for paper analysis.

4. Dataset composition

PACE-Real: 300 videos, 55K frames, 258K annotations, 238 objects, and 43 categories.
Environments: 10 background / lighting conditions, 1–7 objects per sequence, distances of 0.5–1.5 m, and 1280×720 RGB-D.
Object types: rigid objects and articulated scissors, cutters, clips, boxes, and similar multipart objects, with CAD, part hierarchy, canonical alignment, and rotational symmetry.
Dynamics / occlusion: static, moving, stacked, hand-interacting objects at varying visibility.
PACE-Sim: 100K photorealistic PBR frames, 2.4M annotations, and 931 objects, with separate instance / category synthetic training subsets.
Default split: real data is 20% validation / 80% test, plus synthetic category validation. A separate 80/10/10 real split supports failure analysis and must remain separate from the default leaderboard.

5. Annotation pipeline

Scan each object with Einscan Pro 2X in about 5–10 minutes; align bounding-box centers / directions and annotate symmetry.
Mount three RealSense D415 cameras on a frame. SuperPoint + SuperGlue, a trifocal tensor, and bundle adjustment estimate rotation; ArUco translation supplies metric scale missing from SfM.
Background-marker PnP / tracking annotates static objects, with manual drift correction every 40 frames; markers are then inpainted.
Initialize moving-object tracks with BundleTrack. Every 10 frames, inspect 3D cloud alignment and three-view 2D agreement, correcting manually.
Render CAD depth for occlusion-aware masks; subtract SAM hand masks during hand occlusion.
Reannotation on NOCS-REAL275 Scene 1 gives mean differences of 0.9° / 2.3 mm.

6. Experiments at a glance

Instance evaluation uses GT detections throughout: PPF 52.2 AR, SurfEmb 9.0, GDRNPP 7.0, CosyPose 4.4.
For rigid category pose, CPPF++ reaches 44.5 / 24.9 on IoU25 / joint 0:60°–0:15cm AP; articulated scores are only 3.6 / 5.9.
Real2Real single-object / category fitting reaches 58.6–88.2, but single-object / category Sim2Real drops 9.3–86.4 points. Scaling Real2Real to all objects / categories drops 57.2–78.5 points.
In model-based tracking, ICG ADD(-S) AUC is 38.1 for rigid objects and 10.1 for articulated objects.
Best model-free rigid 5°5cm accuracy is only CAPTRA’s 12.9%; for articulated objects, BundleTrack leads with 11.2%.

7. Quick assessment

Value: one dataset connects known-instance and unseen-instance category pose, tracking, articulation, clutter, and movement, exposing gaps between static benchmarks and real video systems.
Key finding: failures extend beyond synthetic-to-real transfer. Even within Real2Real, models severely underfit when expanded from one object / class to all.
Limitations: pose benchmarks assume GT detections; several tracking protocols use GT masks / sizes / initial poses. Treating each articulated part as a category is not joint-aware, and large / expensive tables, cameras, and laptops are absent.
Audience: researchers studying real clutter, pose tracking, articulation, annotation, synthetic-to-real transfer, and scalable pose learning.

Available src/paper-notes/pace-eccv2024/deep.md

In-depth paper notes

Support reproduction, comparison, related-work writing, and identifying opportunities for further research.

Paper Reading Notes (Deep): PACE

1. Key assessment

PACE challenges benchmark saturation. HS-Pose, DualPoseNet, and SAR-Net often approach 100% AP@15°5cm for REAL275 bottles / bowls / cans, yet transfer to corresponding PACE objects is mostly near zero. Four evaluation tracks dissect these failures:

known-instance pose estimation;
unseen-instance category-level pose estimation;
model-based pose tracking;
model-free category-level tracking.

Table 4 is more revealing: without changing synthetic / real domains, expanding from one object / category to all still loses 57–79 points. PACE exposes distinct domain and scale / diversity gaps.

2. Scale and task matrix

PACE-Real:

Item	Scale
RGB-D videos	300
Frames	55K
pose annotations	258K
Scanned objects	238
Categories	43
Environments	10
Objects per sequence	1–7

PACE-Sim supplies 100K PBR frames, 2.4M annotations, and 931 objects, with separate synthetic training subsets for instance / category pose.

Default real data has 20% validation and 80% test, with methods mainly trained on PACE-Sim. A separate 80/10/10 real split analyzes domain / scale. Mixing Table 4’s real training set into default testing causes leakage.

Both rigid and articulated objects appear. AKB48 articulated categories include scissors, cutters, clips, and boxes, decomposed into hierarchical parts with individual poses. This supports articulation-aware tracking beyond videos with only root poses.

3. Why three-camera annotation?

PACE three-camera annotation system — Original Figure 3: scans and canonical / symmetry annotation define model frames. Three fixed cameras provide depth scale, multiview consistency, and triple observations. Source: published ECVA paper. Image copyright belongs to the authors and Springer Nature; reproduced here for paper analysis.

The three RealSense D415 cameras provide more than extra pictures:

Multiview constraints reduce single-view translation–depth ambiguity;
3D alignment and three-view 2D projections are checked together;
Static-object marker PnP becomes more stable;
Each action yields triple observations.

SuperPoint / SuperGlue correspondences, a trifocal tensor, and bundle adjustment estimate extrinsics. SfM translation is only up to scale; metric ArUco translation $t'$ calibrates scale:

s=\frac{\hat t\cdot t'}{\hat t\cdot\hat t}.

Repeated-texture matching failures receive manual correspondences. Camera one defines the reference frame; Intel dynamic calibration estimates intrinsics.

4. Separate static / dynamic annotation paths

Static objects:

Background markers establish a world frame;
Marker tracking propagates pose;
Correct drift manually every 40 frames;
Inpaint markers using a previously captured marker-free background.

Dynamic objects:

BundleTrack initializes trajectories from RGB correspondences and a pose graph;
Manually inspect CAD / cloud alignment every 10 frames;
Check all three 2D projections;
Correct drift manually.

Moving-object annotation is most labor-intensive. BundleTrack is an initializer, not an automatic GT source; dense manual verification determines the final annotations.

Reannotation of NOCS-REAL275 Scene 1 with official GT gives mean 0.9° rotation / 2.3 mm translation differences. This supports local accuracy in one simpler scene, without fully quantifying uncertainty under severe PACE occlusion, fast motion, or articulation.

Masks come from CAD depth rendering, subtracting SAM hand masks during interaction. Small pose errors affect mask boundaries; hand undersegmentation / oversegmentation also changes visibility ratios.

5. Instance pose: why learned methods trail PPF

PACE instance-level and category-level pose results — Original Tables 2–3: all instance baselines use GT detections. Category baselines also use GT detections, except NOCS with its own Mask R-CNN. Source: published ECVA paper. Table copyright belongs to the authors and Springer Nature; reproduced here for paper analysis.

Metrics follow BOP:

AR=(AR_{\mathrm{VSD}}+AR_{\mathrm{MSSD}}+AR_{\mathrm{MSPD}})/3.

Method	Modality	VSD AR	MSSD AR	MSPD AR	Mean AR
PPF	Depth	53.4	48.1	55.2	52.2
CosyPose	RGB	1.4	0.3	11.5	4.4
SurfEmb	RGB	6.2	3.0	17.8	9.0
GDRNPP	RGB-D	3.6	2.1	15.4	7.0

PPF directly matches local point-pair geometry to CAD without PBR learning; the three learned methods train on synthetic PBR. Its 52.2 versus 4.4–9.0 is not simply conventional superiority; at least these factors differ:

PPF directly uses online depth and CAD geometry;
RGB methods are more sensitive to appearance / rendering gaps;
Shared networks suffer capacity / interference at large instance counts;
PBR inadequately models RealSense depth noise.

All receive GT detections, eliminating detector recall as a cause; real end-to-end operation would be harder.

6. Category pose: articulated objects remain largely unsolved

PACE AP averages thresholds from zero to a bound rather than counting hits at one threshold:

AP@0:20°: averaged every 1°;
AP@0:5cm: averaged every 0.25cm;
Joint AP@0:20°,0:5cm: angle and translation jointly satisfied;
3D IoU25 / IoU50 are also reported.

Table 3 separates rigid / articulated results. Rigid CPPF++ leads at IoU25 44.5, 0:20° 15.2, and joint 0:60°/0:15cm 24.9, but trails HS-Pose / SAR-Net on translation alone. For articulation:

CPPF++ IoU25 3.6, joint 0:60°/0:15cm 5.9;
ANCSH joint AP is only 0.2;
Most IoU50 and strict joint scores are zero.

To adapt rigid-category methods, movable parts become separate categories. This ignores kinematics and parent–child consistency: it exposes rigid-method limitations without fairly bounding full articulated formulations.

7. Separating domain and scaling gaps

PACE single-object / category, synthetic-to-real, and full-scale learning ablations — Original Table 4: high Real2Real single fitting contrasts with Sim2Real domain gaps and severe Real2Real-all degradation when sharing models across many objects / categories, even within-domain. Source: published ECVA paper. Table copyright belongs to the authors and Springer Nature; reproduced here for paper analysis.

Controlled fitting uses the can category and object #57:

Setting	Instance-level scope	Category-level scope
Real2Real Single	76.4–88.2 AR	58.6–86.7 mean AP
Sim2Real Single	67.1–78.1 AR	0.3–55.8 mean AP
Real2Real All	8.2–11.2 AR	0.3–13.7 mean AP

Depth-based HS-Pose / SGPA single-category Sim2Real drops from 81.2 / 86.7 to 1.1 / 0.3, showing particularly severe synthetic-depth / real-noise gaps.

Real2Real All still loses 57.2–78.5 points against Single; the authors observe plateaued training loss / underfitting. Rendering gaps alone cannot explain PACE. Larger models, object conditioning, retrieval / foundation features, balanced sampling, and modular geometry solvers are potential directions.

8. Tracking: correct initialization is not enough

Model-based:

Method	Modality	Rigid / articulated ADD(-S) AUC
RBOT	RGB	7.4 / 0.5
ICG	RGB-D	38.1 / 10.1

Model-free:

Method	Rigid / articulated 5°5cm	Rigid / articulated IoU25
BundleTrack	6.4 / 11.2	9.1 / 14.1
CAPTRA	12.9 / 4.4	45.8 / 20.6
6-PACK	9.2 / 3.9	23.1 / 16.7

All trackers receive first-frame GT poses; mask-dependent methods use GT masks, and evaluation uses GT box sizes. Even with these priors, best strict accuracy remains below 13%.

Rerr / Terr are evaluated only when IoU>25%, producing survivorship bias: BundleTrack’s low mean errors exclude frames after losing track. Interpret conditional errors alongside IoU25 and 5°5cm.

9. Limitations and usage boundaries

No default real training split; training mainly uses PACE-Sim;
Pose uses GT detections; tracking adds GT initial poses, masks / sizes;
Inpainted marker artifacts may still create shortcuts;
Dynamic poses are corrected every 10 frames, without error stratification by intervening frames or motion speed;
NOCS Scene 1’s 0.9° / 2.3mm verification excludes the hardest PACE conditions;
Heterogeneous articulated adaptations prevent direct joint-aware versus part-wise rankings;
Large / expensive tables, cameras, and laptops are absent;
Despite 238 objects / 43 categories, long tails and scene–object correlations may affect results.

10. Reproduction and new-method checklist

Strictly separate default and Table 4 80/10/10 analysis splits;
Report instance / category, rigid / articulated, and estimation / tracking separately;
Disclose oracle boxes, masks, initial poses, and sizes;
Retain BOP AR, joint angle–translation scores, IoU, and unconditional tracking success;
Stratify by visibility, motion speed, depth noise, object count, and part count;
Check single-object / category fitting before scaling to all;
Quantify synthetic RGB and depth gaps separately;
Constrain parent–child transforms and joint limits for articulation;
Log tracking failure time, recovery, and drift alongside successful-frame errors;
Add detection / segmentation recall in end-to-end systems.

11. Overall assessment

PACE’s main value is controlled evidence of two cliffs: synthetic→real domain transfer and single→all learning scale. PPF’s 52.2 AR advantage over three deep baselines in known-instance pose also shows the strength of explicit geometry amid large vocabularies, clutter, and depth noise.

This is not a complete end-to-end benchmark: many evaluations use oracle detections / masks / initial poses / sizes, and articulated adaptation is preliminary. Its explicit boundaries make PACE a useful stress test of what remains after clean-benchmark saturation when entering dynamic real scenes.

Related papers

2026 · CVPR

3PT

3PT unifies detection, segmentation, and 6-DoF pose with two Transformers for 2D / 3D object-centric understanding. Continuous refinement of object representations and rendered-feature alignment support zero-shot cross-object perception on BOP-H3, BOP-Industrial, and BOP-Classic.

2026 · AAAI

6DAttack

Studies backdoor attacks on PnP-based and end-to-end 6-DoF pose pipelines. Structurally 3D triggers and modified pose labels are injected into training data so that models behave normally on clean inputs but produce controlled incorrect poses when triggered.

2026 · ICRA

Active Tactile Exploration

A violation-implicit loss incorporates physical-constraint violations into joint estimation of freely moving unknown convex rigid-body trajectories and cuboid / 20-vertex polytope shapes. Observed / Fisher information at a local MLE estimates EIG to select the next touch. In simulation, the EIG strategy yields stable errors below 4 mm after 5 cuboid actions; real Trifinger results at step 5 substantially outperform random selection. Convex geometry, local optima, multiple initializations, and tactile / proprioceptive errors limit the method.