10 Real data 6D Pose Annotation Data Engineering Quality control

Dataset Design & Real-data Annotation

From a Physical Object to
Auditable Pose Ground Truth

A 6D pose dataset binds physical instances to CAD, frames to units, cameras to time, and labels to measurement chains, visibility, splits, versions, and errors. Ambiguity in any link grows into systematic bias as data scale, beyond collecting images and saving R and t.

Open the Annotation Lab Start with deliverables

Dataset Contract

Define the claim before choosing capture and annotation

Instance localization, category pose-and-size, novel objects, multiple views, tracking, and robotic grasping differ in test-time assets and outputs. Encode the task directly in the schema instead of forcing a protocol from whatever fields remain after capture.

Layer	Minimum fields	What it guarantees
Identity	obj_id, instance_id, category_id, track_id	Distinguishes repeated category instances, scene re-entry, and reappearance after occlusion
Geometry	mesh/CAD, object frame, metric unit, diameter, bbox, symmetry	Training, rendering, PnP, evaluation, and robot chains share asset semantics
Camera	RGB / Depth K,D, resolution, frame, timestamp, extrinsics	Reproducible per-frame projection and RGB-D alignment without hidden defaults
Pose	Default ᶜTₒ, measurement chain, covariance / quality, validity	Identifies direction, units, time, and provenance beyond a sequence of 12 numbers
Visibility	mask, mask_visib, bbox_obj, bbox_visib, counts, visib_fract	Separates full silhouettes, visible portions, truncation, and invalid depth
Provenance	raw / filtered / aligned / completed, calibration / asset / annotation version	Traces labels to their inputs and tool versions
Protocol	train/val/test manifest, target list, license, evaluation contract	Prevents leakage and makes results comparable

TEST-TIME ASSET

Exact CAD, reference video, category name, or none?

Defines instance / category / model-free / open-world settings.

TARGET PRESENCE

Are target IDs / counts / ROIs known?

Defines localization versus detection and the need for negative frames.

OUTPUT SPACE

6D, Sim(3), 9DoF, tracks, or grasps?

Determines size, velocity, identity, grasp, and uncertainty fields.

GENERALIZATION

New views, scenes, instances, or sensors?

Directly determines the atomic split-grouping key.

Dataset size has at least five units. Frames, independent trajectories, scenes, physical instances, and sensors/capture days are not interchangeable. One hundred thousand adjacent frames may cover only one lighting setup, background, and object trajectory.

Object Inventory & Assets

Fix the physical object–digital asset–object frame relationship first

Vertices, pose translation, diameter, symmetry transforms, and robot grasps depend on one frame/unit contract. Changing CAD origins, axes, or dimensions changes label semantics even if stored matrices remain identical.

Asset item	Required decision	Acceptance test
Physical instance ↔ digital asset	Bind each physical instance to a unique asset hash; record CAD, scan, or reconstruction origins	Match dimensions, quality, and distinguishing features to the real object; do not silently reuse approximate meshes within a category
Object frame	Fix origin, +x/+y/+z, natural upright direction, and handedness; save physical-to-model extrinsics	Render axes, silhouettes, and keypoints from independent views so reviewers can interpret every axis
Metric unit	Use m or mm consistently internally and convert explicitly on export; synchronize vertices, translation, and diameter	Measured caliper sizes, mesh bounds, projected sizes, and depth distances agree
Mesh topology	Remove isolated pieces, nonmanifold geometry, and duplicate faces; standardize normals, winding, textures, and watertightness requirements	Consistent masks/depth across renderers, outward normals, no NaNs or extreme coordinates
Symmetry	Label discrete/continuous equivalents and whether the task distinguishes texture orientation	Verify that equivalent rotations truly preserve indistinguishable geometry / appearance
Category canonical	Fix category NOCS / canonical frames, size vectors, and instance-deformation rules	Align semantic parts, up/front, and normalized scale across instances, beyond resizing bounding boxes into unit cubes

Physical tracker frame → published object frame

{}^{C}\mathbf T_O=\left({}^{W}\mathbf T_C\right)^{-1}{}^{W}\mathbf T_M\,{}^{M}\mathbf T_O

M may be an object marker, mocap rigid body, turntable fixture, or temporary reconstruction alignment frame. The static mount ^MTO connects physical tracking to the published object/model frame; calibrate, version, and verify it independently.

Symmetry affects more than evaluation. It affects manual alignment, PnP alternatives, temporal continuity, pose losses, visual QA, and grasp semantics. If texture breaks geometric symmetry, record geometry symmetry separately from task / appearance symmetry.

Capture Matrix

Turn the deployment distribution into inspectable capture factors

List factors before combinations, budgets, and coverage checks. Full Cartesian products are usually infeasible; stratification, orthogonal designs, active sampling, or difficulty buckets help. Record omitted regions explicitly instead of assuming random capture covers them naturally.

01 · Viewpoint

azimuth, elevation, roll, in-plane, distance, image size

Cover the task distribution beyond frontal tabletop views

02 · Object state

Stable placement, handheld, motion, articulation, grasp contact

Match deployed motion and occlusion patterns

03 · Scene

Backgrounds, tables, shelves, bins, indoor/outdoor scenes, spatial scale

Prevent fixed backgrounds from becoming object identity cues

04 · Clutter

Instance counts, repeated categories, similar distractors, contact, stacks

Cover detection, association, and occlusion difficulty

05 · Visibility

occlusion, truncation, self-occlusion, tiny object

Sample by strata before discovering that low-visibility examples are absent

06 · Illumination

Direction, intensity, color temperature, exposure, highlights, shadows, HDR

Cover RGB appearance and sensor operating conditions

07 · Material

Texture, transparency, mirrors, black surfaces, thin edges, nearby deformable objects

Expose structured RGB / depth failures

08 · Sensor

Devices, lenses, resolution, focal length, distance, synchronization, temperature

Define cross-device generalization; avoid treating device-specific bias as universal

SPECIFY

Task distribution

Deployment logs, robot workspace, materials, failure costs

→

STRATIFY

Coverage buckets

view × distance × visibility × scene × sensor

→

CAPTURE

session manifest

Atomic sessions, device states, assets, operators

→

AUDIT

Gaps and biases

Inspect independent groups and joint distributions alongside frame counts

Detection data need negatives and similar distractors. If every frame contains a target, categories use fixed backgrounds, or videos contain one object each, systems may learn presence priors and scene shortcuts instead of demonstrating discovery.

Calibration Before Annotation

The weakest static extrinsic or timing link limits ground-truth accuracy

Intrinsics, RGB–Depth, camera–mocap, marker–object, hand–eye, robot base, turntable axes, and synchronization may all enter a label. Each needs independent data, residuals, validation, temperature/focus state, and validity periods beyond one final YAML file.

INTRINSIC

K, D, and resolution

Cover field of view and distance; inspect position-dependent reprojection residuals on independent views.

CROSS-SENSOR

RGB–Depth / camera–tracker

Check distances and image edges; log target-from-source direction, units, and covariance.

HAND-EYE / FIXTURE

robot, end-effector, object mount

Cover rotation axes and workspace; test pose-dependent bias and mounting repeatability.

TIME

Timestamps, exposure, and latency

Static spatial correctness does not establish moving synchronization; estimate offsets/jitter with observable motion.

Mocap / marker chain with explicit mounts

{}^{C}\mathbf T_O=\left({}^{W}\mathbf T_{M_C}\,{}^{M_C}\mathbf T_C\right)^{-1}\left({}^{W}\mathbf T_{M_O}\,{}^{M_O}\mathbf T_O\right)

MC and MO are directly tracked camera/object rigid-body frames; MC→C and MO→O are static mount calibrations. If tracker poses differ in time, interpolate or compensate motion.

Calibration residuals do not fully characterize label uncertainty. Include tracker noise, robot repeatability, mount/object deformation, timing offsets, CAD-to-physical bias, and propagation Jacobians. At minimum, retain individual calibration residuals and independent end-to-end overlay errors.

Annotation Routes

No ground-truth route suits every object, scene, and budget

Turntables suit systematic single-object views; mocap and robots support dynamic capture; reconstruction + CAD alignment propagates one manual alignment through long RGB-D sequences; per-frame PnP repairs selected keyframes. Systems often cross-check two routes.

Route	Direct measurement	Core chain	Strengths	Risks
Fiducial / turntable	Marker / board pose, turntable angle, calibrated mount	camera ← board/marker ← object	Simple, precise, and suitable for systematic single-object view-sphere coverage	Visible markers/boards become shortcuts; mount extrinsics and fixture offsets introduce error
Optical mocap / tracker	Camera-rig and object-rigid-body poses in a mocap world	CTo=(WTc)⁻¹WTo, including tracker-to-optical / marker-to-object	High-rate multi-object capture with real motion/occlusion and robot synchronization	Occluded reflective markers, rigid-body swaps, time offsets, loose mounts
Robot kinematics	Joint encoders, base-to-end-effector, hand–eye calibration	CTo=(BTe · ETc)⁻¹BTo or the corresponding eye-to-hand chain	Repeatable automated capture with planned viewpoints and active-sensing trajectories	Backlash, compliance, and pose-dependent TCP / hand–eye / base calibration errors
Reconstruction + CAD alignment	RGB-D / multiview camera trajectories, scene mesh, one CAD-to-scan alignment	CTo=(STc)⁻¹STo	Propagates limited manual alignment to long sequences and dense pixel labels	Camera drift, dynamics, reconstruction deformation, and CAD misalignment propagate globally
Per-frame 2D–3D / PnP	Manual keypoints, edges, silhouettes, or pose gizmos	model point ↔ image pixel → candidate ᶜTₒ	No external hardware; repairs selected keyframes and unusual samples	Expensive labor, invisible keypoints, symmetric alternatives, temporal jitter
Hybrid refinement	Any coarse ground truth + RGB/depth render residuals + human review	measured prior → visibility-aware optimization → audit	Combines system measurements with image evidence to reveal mount and timing bias	Optimization basins, incorrect depth/masks, circular validation; the same model output is not independent ground truth

Robot eye-in-hand example

{}^{C}\mathbf T_O=\left({}^{B}\mathbf T_E(\mathbf q)\,{}^{E}\mathbf T_C\right)^{-1}{}^{B}\mathbf T_O

The chain changes when an object is end-effector-mounted, a camera is external, or both move. Draw the frame graph and compose source→target instead of memorizing one hand–eye formula. Joint timestamps must match camera exposure centers.

Reconstruction propagation

{}^{C_i}\mathbf T_O=\left({}^{S}\mathbf T_{C_i}\right)^{-1}{}^{S}\mathbf T_O

S is the scan / reconstruction frame. One manual ^STO alignment propagates per-frame labels, but inherits trajectory and CAD-alignment errors. Regenerate every label and mask after loop-closure trajectory updates.

Interactive Annotation Lab

Make hidden pose, visibility, and split assumptions measurable

The first view propagates rotation, translation, timing, and CAD-scale errors through 4×4 transforms. The second separately counts full silhouettes, valid depth, and visible pixels on a teaching raster. The third applies four group-split strategies to 240 deterministic records and reports scene, trajectory, and instance leakage.

camera / sensor object / model measured intermediate X–Y diagram; numbers use 4×4 homogeneous rigid transforms

Route contract

—

——

Pose label

estimated ᶜTₒ

—translation · yaw slice

Propagation error

against clean chain

——

Asset scale

pose matrix does not store CAD scale

——

Object-to-camera ground truth

^CT_O · 4 × 4

clean

A precise tracker guarantees only its own frame. Object ground truth also passes through tracker-to-camera, marker-to-object, hand–eye, and CAD-to-physical extrinsics. Missing calibration or loose mounts propagate systematic bias to every frame, regardless of dataset size.

Masks, Visibility & Occlusion

Render full geometry from pose, then use scene evidence for visibility

A full mask is the model's projected silhouette; a visible mask excludes scene occlusion under the visibility rules; a depth-valid mask describes sensor measurements. Store them separately and version boundary tolerances, antialiasing, and depth deltas.

BOP-style visible fraction

\mathrm{visib\_fract}=\frac{\mathrm{px\_count\_visib}}{\mathrm{px\_count\_all}}

BOP stores px_count_all, px_count_valid, and px_count_visib separately. Visibility is not divided by depth-valid pixels; otherwise transparency or depth holes turn sensor failure into artificially high visibility.

MASK / FULL

Full projected silhouette

Rendered from CAD, K/D, and pose, including in-image portions occluded by other objects.

MASK_VISIB

Actually visible in the scene

Compare object and scene depth, accounting for z-buffers, deltas, and invalid depth.

AMODAL / TRUNCATION

Out of frame differs from occlusion

Store full amodal extent and truncation when needed; in-image boxes cannot represent off-image geometry.

01

First render object IDs, depth, normals, and full masks consistently.

02

Retain only the nearest camera surface per pixel; keep instance GTIDs distinct.

03

On valid scene depth, compare |z_render−z_scene|≤δ in physical units.

04

Specify invalid-depth treatment consistently across splits.

05

Record bboxes, counts, visib_fract, and failure reasons for stratified evaluation.

BOP & Export Contract

Format conversion must preserve geometric semantics

BOP uses right-handed model/camera/world frames, OpenCV optical cameras, model-to-camera poses, and commonly millimeter models/translations. Internal meters/custom schemas are valid, but export must jointly convert mesh, t, diameter, bbox, symmetry, and depth_scale.

File	Key semantics
models/obj_XXXXXX.ply	Metric meshes, normals, textures; BOP commonly uses mm
models/models_info.json	Diameter, bbox, and symmetry transforms
scene_camera.json	Per-image cam_K, depth_scale, optional world-to-camera pose
scene_gt.json	Per-frame, per-instance obj_id, cam_R_m2c, cam_t_m2c
scene_gt_info.json	bbox_obj / bbox_visib, px_count_all / valid / visib, visib_fract
rgb/ · depth/	Color and uint16 depth share image_id; depth × depth_scale gives mm
mask/ · mask_visib/	IMID_GTID.png; GTID indexes instances in scene_gt
test_targets_*.json	Actual evaluation scene/image/object/instance targets, rather than every object in a folder

Minimal BOP pose record

scene_camera.json
"0": {
  "cam_K": [fx,0,cx, 0,fy,cy, 0,0,1],
  "depth_scale": 0.1
}

scene_gt.json
"0": [{
  "obj_id": 7,
  "cam_R_m2c": [r00,...,r22],
  "cam_t_m2c": [tx,ty,tz]
}]

# model vertices and cam_t_m2c: millimetres
# depth_mm = depth_uint16 * depth_scale

Do not assume one K across the datasetBOP allows per-image cam_K; save it per frame when crops, sensors, or zoom change.

Rerender after export instead of comparing JSON alone. Generate overlays, masks, and depth from exported mesh, K, depth_scale, and `cam_R_m2c / cam_t_m2c`. Compare pixels/points with internal outputs to expose transposes, direction, meter/millimeter, and instance-index errors.

Quality Assurance

Quality control spans algebra, images, depth, timing, and distributions

Ten reviewed overlays cannot establish reliable million-frame ground truth; one mean reprojection error can hide an object's mount bias or a timing fault. Use per-sample metrics, stratified control charts, anomaly queues, and independent spot checks.

01 · Transform algebra

All T·T⁻¹≈I and alternative chain paths close; RᵀR≈I, detR≈1, consistent units

02 · Projection overlay

Plot axes, CAD silhouettes, keypoints, and 3D boxes per frame; inspect image centers and edges

03 · Depth residual

Compare rendered object and measured scene depth only in valid, expected-visible regions

04 · Multi-view consistency

One object point projects onto matching tracks across views; adjacent poses show no unphysical jumps

05 · Independent measurement

Calipers, known boards, a second tracker, or robot contact verify absolute scale and local position

06 · Distribution audit

Inspect counts/errors by object, scene, distance, direction, visibility, material, device, and annotation route

07 · Human spot check

Stratified random checks plus all extreme cases; two-person review for high-risk or uncertain samples

08 · Round-trip export

Internal schema→BOP/COCO/custom→internal, preserving hashes, poses, K, and mask counts

Visibility-aware depth residual

r_z(\mathbf u)=z_{\rm scene}(\mathbf u)-z_{\rm render}(\mathbf u;\mathbf K,{}^{C}\mathbf T_O),\qquad \mathbf u\in\Omega_{\rm valid}\cap\Omega_{\rm expected\ visible}

Compare only valid, expected-visible scene-depth regions. Occluding foreground, transparent holes, and off-image pixels should not receive ordinary-surface weights. Report medians, quantiles, and spatial maps alongside means.

Do not generate ground truth with the model being evaluated. Predictions can propose labels, flag anomalies, or initialize tools. Confirm labels through independent measurements, human evidence, or an explicit protocol isolated from the training model, retaining correction history.

Split, Bias & Governance

Splits must isolate factors relevant to the evaluation claim

Seen-instance novel views may group sessions/trajectories; new scenes isolate backgrounds; category-level unseen instances isolate physical objects and all their assets/onboarding data; cross-sensor claims isolate devices/domains. Publish separately named manifests for distinct claims.

FRAME SPLIT

Usually unsafe

Adjacent frames are highly correlated; use only with proven independent sampling.

TRAJECTORY / DAY

New views and capture reproducibility

Group complete sessions atomically, including lighting/operators when needed.

SCENE / SENSOR

Environment or device generalization

Keep backgrounds, calibration, sensor fingerprints, and domains from crossing splits.

OBJECT INSTANCE

Category-level new instances

All frames, scans, CAD, and reference videos of a physical object belong to one split.

VERSION

Immutable releases

Hash raw manifests, assets, calibration, labels, splits, and tools separately; revisions create new versions.

LICENSE & PRIVACY

Real capture also has compliance boundaries

Object assets, trademarks, people, residential information, and third-party data need permissions, de-identification, and deletion policies.

CHANGELOG

Per-label provenance

Record old/new values, reasons, tools, human/automatic origins, and affected splits; avoid silent overwrites.

Release gate · 12 questions

Are tasks, test-time assets, and outputs fixed?
Is each physical instance bound to a mesh hash?
Are frames, units, and symmetry visually verified?
Are all K/D, extrinsics, and timestamps versioned?
Can raw measurements reproduce the pose chain?
Are full / visible / valid masks separate?
Are per-frame anomaly metrics and review states saved?
Is coverage audited across joint factors?
Do split groups isolate the evaluation claim?
Has export passed round-trip rendering?
Are licenses, privacy, and deletion policies clear?
Does the release include manifests, hashes, changelogs, and baselines?

Formats, Real Annotation Workflows, and 6D Pose DatasetsExpand BOP, LabelFusion, T-LESS, YCB-Video, NOCS, HouseCat6D, and AprilTag sources