13 Pose Errors BOP AR / AP Matching Statistical Diagnosis

Pose Evaluation & Error Analysis

Behind every score,
there is an entire protocol

Pose evaluation is more than subtracting a predicted matrix from ground truth. First choose the error space and equivalence relation, then set thresholds, match candidates, handle misses and duplicates, and finally aggregate across instances, objects, thresholds, and datasets. Changing any step can produce a different ranking from the same predictions.

Open the evaluation lab Explore the evaluation metrics library

01 · Evaluation Contract

Freeze the ten-layer evaluation contract before running the evaluator

A comparable number must specify at least the claim, units, equivalence, thresholds, matching, aggregation, and independent sampling unit. Simply saying 'we use ADD-S / AR' is far from enough.

01

Claim

State the claim first: pose of a known instance, end-to-end 6D detection, tracking stability, grasping/assembly success, or real-time system constraints.

The same model can have entirely different 'best metrics' for different claims.

02

Evaluation unit

A frame, object instance, scene, sequence, physical instance, dataset, or task episode.

This unit determines statistical independence, bootstrap clusters, and averaging weights.

03

Error function

Rotation / translation, surface distance, visible depth, image projection, box overlap, or task cost.

The error space determines what the metric captures and what it overlooks.

04

Equivalence

Object geometry / appearance symmetry, current-view ambiguity, visibility, and task equivalence.

These must come from frozen metadata / protocols and must not change after seeing the results.

05

Correctness

Continuous error, success at one threshold, threshold curves, prediction-set coverage, or calibrated probability.

Thresholds, units, diameter normalization, and strict < versus ≤ are all part of the protocol.

06

Matching

Class constraints between predictions and GT, one-to-one assignment, score order, top-N, and ignore / visibility rules.

Misses, duplicates, and incorrect ranking become TP / FP / FN only after matching.

07

Aggregation

instance micro, object macro, dataset macro, threshold average, AUC / AR / AP.

The headline score is a multilevel reduction, not a new pose distance.

08

Uncertainty

Multiple seeds, paired differences, scene/sequence cluster bootstrap, effect sizes, and intervals.

A CI covers only the specified sampling process; it does not automatically cover domain shift or systematic label errors.

09

Diagnostics

Slice by object, scale, range, visibility fraction, symmetry, material, sensor, scene, and failure stage.

Overall scores rank methods; slices identify what to fix.

10

Reproducibility

Dataset split/hash, evaluation mesh, symmetry, targets, toolkit commit, renderer, thresholds, and result CSV.

Without any one of these, a paper's reported numbers may be irreproducible.

Protocol ID · Make result tables machine-traceable

task: ModelBased-6DDet-Unseen
dataset: ycbv/test + targets_bop24@sha256:...
prediction: class + ᶜTₒ(mm) + confidence + runtime
assets: models_eval@sha256:... + models_info@sha256:...
errors: [MSSD, MSPD] @ bop_toolkit:<commit>
thresholds: <named config>          # never inferred from test
matching: score_ordered_one_to_one
aggregation: threshold → object → metric → dataset
independent_unit: scene             # for paired bootstrap
failure_rule: missing/timeout → FN
hardware: GPU / driver / precision / batch / warmup

How this page relates to the metrics library This page explains how to assemble metrics into a credible experiment and interpret conflicts. The site's evaluation metrics library maintains individual formula details, common aliases, and links back to papers. Every metric card here links to those individual entries to prevent definitions from drifting apart.

02 · Component Pose Errors

Rotation and translation are useful diagnostic coordinates, but do not form a complete task metric

Decomposing 6DoF quickly reveals problems in orientation, metric depth, scale, frames, or calibration. These components do not automatically account for object size, surface shape, or visibility.

Geodesic rotation error

e_R=\cos^{-1}\!\left(\operatorname{clip}\!\left(\frac{\operatorname{tr}\!\left((R^\star)^\top\hat R\right)-1}{2},-1,1\right)\right)

State whether angles are in radians or degrees. For symmetric objects, minimize over the proper symmetry group first. Clipping prevents floating-point errors from making acos return NaN.

Translation error and components

e_t=\left\|\hat t-t^\star\right\|_2,\qquad e_{xy}=\sqrt{\Delta t_x^2+\Delta t_y^2},\qquad e_z=\left|\Delta t_z\right|

Report lateral and depth components alongside total L2 to distinguish detection-center / PnP-bearing issues from scale / depth-estimation issues. All vectors must use the same frame and units.

Symmetry-aware component rotation

e_R^{\mathcal S}(\hat R,R^\star)=\min_{S\in\mathcal S_R}\cos^{-1}\!\left(\frac{\operatorname{tr}\!\left((R^\star S)^\top\hat R\right)-1}{2}\right)

Considering only the rotation subgroup is valid only when the symmetry axis passes through the object origin and the diagnosis concerns rotation alone. Full pose / surface metrics require a 4×4 S containing the offset translation.

SO(3)

Rotation Error

Trace / quaternion geodesics, angular thresholds, and symmetry-aware extensions

Open metric page → ℝ³

Translation Error

Units, frames, lateral/depth components, and fairness across scales

Open metric page → joint threshold

n° / m cm

Success event requiring both rotation and translation to fall below their thresholds

Open metric page →

03 · Surface & Projection Errors

Corresponding, nearest, worst, and projected points: four relationships yield four conclusions

ADD, ADD-S, MSSD, and MSPD all use model surfaces, but differ in correspondence, symmetry, mean versus maximum, and 3D versus 2D space.

ADD · average corresponding model-point distance

e_{\mathrm{ADD}}=\frac1{|\mathcal M|}\sum_{x\in\mathcal M}\left\|\hat R x+\hat t-\left(R^\star x+t^\star\right)\right\|_2

Each x has a fixed identity, making this suitable for asymmetric objects. The point-sampling measure, mesh origin, and units directly affect the result.

ADD-S · average closest-point distance

e_{\mathrm{ADD\text{-}S}}=\frac1{|\mathcal M|}\sum_{x_1\in\mathcal M}\min_{x_2\in\mathcal M}\left\|\hat R x_1+\hat t-\left(R^\star x_2+t^\star\right)\right\|_2

Nearest-point matching is not an explicit symmetry group. It accommodates repeated geometry, but can also understate large errors on imperfect symmetries or densely sampled regions.

Average 2D projection error

e_{\mathrm{proj}}=\frac1{|\mathcal M|}\sum_{x\in\mathcal M}\left\|\pi_K(\hat R x+\hat t)-\pi_K(R^\star x+t^\star)\right\|_2

Pixel errors depend on focal length, resolution, and camera model. A small projection error does not guarantee sufficiently accurate metric translation along the optical axis.

MSSD · explicit symmetry + worst 3D surface point

e_{\mathrm{MSSD}}=\min_{S\in\mathcal S_M}\max_{x\in\mathcal V_M}\left\|\hat P x-P^\star Sx\right\|_2

BOP uses predefined global symmetries and evaluation-model vertices. The maximum emphasizes worst local deviation, so its numerical magnitude cannot be compared directly with ADD's mean.

MSPD · explicit symmetry + worst projected point

e_{\mathrm{MSPD}}=\min_{S\in\mathcal S_M}\max_{x\in\mathcal V_M}\left\|\pi_K(\hat P x)-\pi_K(P^\star Sx)\right\|_2

MSPD measures perceptible projection displacement without using test depth. It explicitly handles global object symmetry, which differs from ambiguity in the currently visible view.

Metric	Space	Requires	Symmetry	Visibility	Most sensitive to	Blind spots / risks
Rotation geodesic	Shortest angle on SO(3)	R̂, R*	None by default; can minimize over 𝒮	None	Orientation error, decoupled from model size and translation	Translation, surface radius, image scale, and task contact
Translation L2	Metric 3D in camera / world coordinates	t̂, t* and their units/frame	Rotational symmetry leaves the center unchanged; take care with off-axis definitions	None	Center position, depth scale, and extrinsic calibration bias	Rotation, worst surface-point error, and difficulty relative to object size
ADD	Mean 3D displacement of corresponding model points	Fixed model points and two poses	Not handled	Full model	Rotation + translation; points farther from the origin contribute more	Occlusion; penalizes visually equivalent symmetry representatives
ADD-S	Mean nearest-point 3D distance	Model point set and nearest-neighbor implementation	Implicitly through nearest surface points	Full model	Overall surface proximity; accommodates axial / repeated geometry	Imperfectly repeated structures, texture orientation, and point-sampling bias
2D Projection	Mean pixel displacement of corresponding points	Model points, pose, K / camera model	None by default	Usually the projection of the full model	Mean image displacement of the final overlay	3D errors along the viewing ray; resolution / focal length changes the meaning of a pixel
VSD	Depth discrepancy of surfaces visible in the current image	Model, pose, K, test depth, renderer, δ / τ	Through indistinguishable visible surfaces	Explicit and central	Observable surface alignment under occlusion	Invisible back surfaces; depends on depth validity and the rendering contract
MSSD	Maximum 3D surface displacement after accounting for symmetry	eval vertices, pose, global symmetry set	Explicit minimization over 𝒮	Full model	Worst local 3D misalignment, safety / contact tolerances	Whether the current view distinguishes poses; sensitive to extreme vertices / sampling
MSPD	Maximum projection displacement after accounting for symmetry	eval vertices, pose, 𝒮, K	Explicit minimization over 𝒮	Does not use test depth	Worst perceptible pixel misalignment, RGB / AR	Metric 3D errors along the optical axis; depends on focal length / resolution
Task success / cost	Grasping, insertion, collision, trajectory, or control outcomes	Robot, environment, action policy, and repeated trials	Defined by action equivalence	Enters indirectly through the entire closed loop	The failure costs that matter in deployment	Hard to isolate perception; simulation / hardware and policy jointly affect outcomes

mean · paired

04 · Visibility, Thresholds & BOP

VSD examines currently visible surfaces; BOP AR aggregates three application perspectives across thresholds

VSD uses test depth and rendering to determine surface visibility; MSSD/MSPD use previously identified global symmetry. They are complementary metrics with different purposes.

VSD · visible surface discrepancy

e_{\mathrm{VSD}}=\operatorname*{avg}_{p\in\hat V\cup V^\star}\begin{cases}0,&p\in\hat V\cap V^\star\ \land\ |\hat D(p)-D^\star(p)|<\tau\\1,&\text{otherwise}\end{cases}

D is the rendered distance of the object under the predicted / GT pose. V is constructed from this distance, the test-image distance, and visibility tolerance. The official definition also specifies renderer behavior, missing depth, δ, and other details; use a frozen toolkit version.

Threshold form	Example	Interpretation and risks
Absolute metric	5 mm, 20 mm, 5 px, 5°	Directly matches an engineering tolerance, but is unfair across sizes, ranges, and resolutions
Diameter normalized	ADD / MSSD < αd, commonly α=0.1	More comparable across object sizes; small critical features and task tolerances can still differ
Image normalized	Normalize pixel thresholds by image width or focal length	More stable across cameras, but requires reinterpretation for actual human / AR tolerances
Multiple thresholds	Average recall / precision over a set of θ values	Reduces dependence on a single threshold; the threshold set and weights still determine the headline
AUC / integral	Integrate the recall–error-threshold curve	Fix the upper limit, normalization, and numerical integration; 0.1 m and 0.1d are not interchangeable
Task calibrated	Choose from insertion clearance, gripper contact, collision margin, or control stability	Most meaningful for applications; requires independent task validation, without tuning thresholds on the test set

Recall curve and normalized AUC

\operatorname{Recall}_e(\theta)=\frac1N\sum_{i=1}^{N}\mathbf1[e_i<\theta],\qquad \operatorname{AUC}(e_{\max})=\frac1{e_{\max}}\int_0^{e_{\max}}\operatorname{Recall}_e(\theta)\,d\theta

An AUC upper limit may be 0.1 m, 0.1d, or another value. Even identically named curves are incomparable when their limits / normalization differ.

BOP localization aggregation

AR_D=\frac{AR_{\mathrm{VSD}}+AR_{\mathrm{MSSD}}+AR_{\mathrm{MSPD}}}{3},\qquad AR_C=\frac1{|\mathcal D_{\mathrm{core}}|}\sum_{D\in\mathcal D_{\mathrm{core}}}AR_D

Each AR_e already averages recall over multiple correctness thresholds; VSD also averages over misalignment tolerances τ. Specify the year / task version for the core datasets and threshold configuration.

01

Pose error

Compute VSD / MSSD / MSPD for every estimate–GT pair.

→

02

Correctness

At each θ (and τ for VSD), test e<θ.

→

03

Recall

Compute the correctly localized fraction under the target and matching rules.

→

04

ARₑ

Average recall across threshold settings for that error.

→

05

AR_D / AR_C

Average equally across 3 errors, then across core datasets.

Version reminder BOP is an evolving benchmark. As of 2025, its official task page lists seen / unseen, model-based / model-free, single / multi-view localization and detection. Do not infer a new detection protocol from 'BOP AR' in an older paper, and always state the challenge year.

05 · Task, Matching & Candidate Semantics

Pose-only, localization, detection, tracking, and manipulation require different leaderboards

Task inputs determine what the system may know. Candidate sets, scores, top-K, and one-to-one matching determine how misses and duplicates enter the denominator.

6D pose of a provided crop / GT box

Input: Known ROI and class; sometimes even a GT mask
Output: One or more poses
Score: pose error / oracle top-K
Question: Tests only the pose head or the upper bound of correspondence estimation / refinement
Warning: Cannot be ranked directly against recall / AP from an end-to-end detector pipeline

BOP-style 6D localization

Input: Test image + a list of visible instance counts per class
Output: The prescribed number of pose estimates + confidence for each class
Score: Multithreshold VSD/MSSD/MSPD recall → AR
Question: Whether localization is correct given the known target list
Warning: 'AR' still requires the dataset, metric component, year, and targets file

6D detection

Input: Image; no visible object-instance list is provided
Output: A set of candidates with class, pose, and confidence
Score: score-ordered one-to-one matching → AP
Question: Find all objects with correct poses while suppressing false positives / duplicates
Warning: As of 2025, official BOP 6D detection uses MSSD/MSPD AP, not VSD AP

Tracking

Input: Video and an initial detection / continuous observations
Output: Trajectories with identity, timestamp, pose, and possibly uncertainty
Score: per-frame pose + drift, failure/recovery, ID switches, latency
Question: Temporal continuity, failure recovery, and usability for long-term control
Warning: Per-frame averages hide catastrophic loss; adjacent frames are not independent samples

Robot manipulation

Input: Perception + calibration + planning + execution
Output: grasp / insertion / placement / servo action
Score: Success rate, collisions, contact error, cycle time, and recovery
Question: Whether the complete system performs the task safely
Warning: Task success does not isolate causes in the pose head; also report pose oracles and stage ablations

Detection precision, recall and AP

\operatorname{Precision}(k)=\frac{TP(k)}{TP(k)+FP(k)},\qquad \operatorname{Recall}(k)=\frac{TP(k)}{N_{GT}},\qquad AP=\int_0^1p_{\mathrm{interp}}(r)\,dr

k denotes the first k predictions ranked by confidence. Follow the relevant benchmark code for exact interpolation, threshold grids, and per-object / per-dataset reductions.

01

Candidate generation

Save image / object / pose / score / time for every candidate; first apply the benchmark's allowed maximum predictions and target rules.

02

Eligibility

Class, image/view, GT ignore / visibility, metric threshold, and task-target conditions must be satisfied before matching.

03

One-to-one

Each GT can match at most one prediction, and each prediction can explain at most one GT. Duplicates become FP or are filtered by task rules.

04

Ordering

Detection usually sorts confidence from high to low. Incorrect high scores create FP at low recall, depressing the entire PR curve.

05

Assignment details

Greedy, Hungarian, target-count, and metric-specific matching can differ. Use the official implementation directly and freeze its version.

06

Ignored targets

Follow the protocol for low-visibility GT / predictions and those outside the targets file. 'Excluded from FN' does not necessarily mean the corresponding prediction counts as FP.

07

Multi-hypothesis

Treating top-K as K detections, one prediction set, or oracle best-of-K defines three protocols with entirely different results.

08

Ties / duplicates

Stable sorting of equal scores, numerical precision, NMS / symmetry deduplication can change boundary outcomes; deterministic rules are required.

Localization mental model

input: image + [(object_id, visible_count), ...]
for each object:
  keep task-allowed number of hypotheses
  compute error matrix to same-object GTs
  one-to-one match under each threshold
recall = matched targets / all targets
AR_e = average recall over threshold settings
headline = aggregate AR_e / datasets

Detection mental model

input: image only (no visible-instance list)
predictions = class + pose + confidence
sort / cap predictions by protocol
for each metric threshold and object:
  score-ordered one-to-one matching
  duplicate / wrong / threshold-fail -> FP
  unmatched GT -> FN
AP_e,o = area under precision–recall
headline = aggregate objects / metrics / datasets

06 · Interactive Lab

See metric conflicts, duplicate candidates, and averaging weights in action

The first experiment computes eight error types for the same transform. The second sends identical candidates through localization / detection. The third switches micro/macro aggregation, pose oracles, and scene-cluster bootstrap over 240 records.

ground truth prediction occluder VSD here is an explicitly labeled teaching raster; official results require a frozen BOP toolkit version

Rotation

raw / quotient-aware

——

Translation

camera-frame center

—Unaffected by rotational symmetry

ADD / ADD-S

mean 3D surface

——

Proj2D

mean corresponding pixels

—Depends on K and resolution; does not measure 3D accuracy

MSSD / MSPD

symmetry-aware maximum

—Worst 3D / 2D displacement over the full model

VSD-style

visible raster discrepancy

——

Each metric answers a different question. ADD-S's nearest points, MSSD/MSPD's explicit global symmetry, and VSD's currently visible surfaces answer different questions. One prediction can pass one metric and fail another. Select and report metrics together according to the application claim, rather than choosing the most favorable number.

07 · Aggregation & Statistical Evidence

The headline is a reduction tree that must trace back to per-sample records

Save every prediction, GT, error, and matching outcome before generating aggregate tables. Retain more than the final average.

Level	How it averages	Changed weights / risks
Per-instance micro	Pool and average all annotated instances	Objects / scenes with more samples dominate; appropriate for claims about actual frequencies
Per-object macro	Compute each object's score, then weight objects equally	Long-tail objects receive equal weight; objects with few samples have higher variance
Per-dataset macro	Compute each dataset score, then weight core datasets equally	Prevents large datasets from overwhelming small ones; changing the dataset definition changes the overall score
Across thresholds	Average recall / precision across correctness thresholds	These are not independent trials; disclose the threshold grid and range
Across metrics	For example, BOP localization averages AR_VSD / AR_MSSD / AR_MSPD	The three metrics target different applications; averaging can hide conflicts
Across seeds / runs	Report mean±SD / CI over complete training and evaluation runs, retaining every seed	Per-image variance cannot substitute for training-seed variance
Across time	Per frame, per sequence, per failure episode, or within a fixed time window	Aggregate tracking by sequence / episode and report failure duration

Micro vs object-macro success

\hat s_{\mathrm{micro}}=\frac{\sum_o\sum_{i=1}^{n_o}y_{oi}}{\sum_o n_o},\qquad \hat s_{\mathrm{macro}}=\frac1{|\mathcal O|}\sum_o\left(\frac1{n_o}\sum_{i=1}^{n_o}y_{oi}\right)

Micro weights each instance equally; macro weights each object equally. Both are valid, but support different deployment-distribution claims. Do not select only the more favorable result.

Paired cluster bootstrap

\Delta^{(b)}=S_A\!\left(\bigcup_{c\in C^{(b)}}\mathcal I_c\right)-S_B\!\left(\bigcup_{c\in C^{(b)}}\mathcal I_c\right),\qquad C^{(b)}\sim\operatorname{Resample}(C,|C|)

Resample scene / sequence clusters with replacement, always using the same bootstrap sample for A/B. The percentile CI describes uncertainty in the paired score difference under this sampling assumption.

Paired comparison

Obtain paired outcomes / errors for both methods on the same GT instances and compare per-unit differences

Removes shared variation in sample difficulty

If preprocessing, detector targets, or failed-sample sets differ, define missing-data rules before pairing

Cluster bootstrap

Resample scenes, sequences, physical instances, or capture sessions with replacement

Preserves within-cluster correlation rather than treating adjacent frames as independent

Percentile CIs are unstable with very few clusters; also report the original cluster counts

Multiple seeds

Repeat the complete training / sampling / inference process; report individual seeds and central tendency

Reveals instability from optimization and random data order

Repeating only the evaluator with a fixed checkpoint does not measure training-seed variance

Effect size

Report absolute percentage points, relative change, continuous error differences, and actual task margins

Shows whether an improvement exceeds engineering tolerances

Relative percentages exaggerate changes near a zero baseline; p-values alone do not establish practical significance

Distribution report

Report mean, median, P90/P95, CDF / recall curves, and catastrophic failure rate together

Reveals tails and the distribution near thresholds beyond the mean

Different sample weights or filtering rules can make percentiles incomparable

Subgroup uncertainty

Report sample counts and intervals for every visibility / scale / symmetry / object slice

Prevents an incidental 100% on a small slice from becoming a conclusion

Many slices invite selective reporting; freeze the slice schema in advance

A CI will not detect a faulty protocol for you An experiment with wrong units, test leakage, incorrect symmetry metadata, or evaluation restricted to matched detections can still produce a very narrow confidence interval. Statistical intervals quantify only modeled randomness. Schema, labels, domain shift, and protocol uncertainty require independent audits and sensitivity analysis.

08 · Error Slicing & Stage Oracles

Overall scores show who ranks higher; slices and oracles explain why

Define every slice in advance, report sample counts and intervals, and retain links to original images / depth / overlays / error heatmaps. Otherwise, slices invite post hoc selection.

Slice axis	Diagnostic target	Suggested reporting
Object / category / instance	Geometry, texture, symmetry, training frequency, and CAD quality	per-object table, macro score, worst-K objects
Visible fraction / truncation	Distinguishes occlusion handling in the front end, amodal reasoning, and VSD	Use visib_fract bins and treat truncation separately
Image footprint / diameter	Small-object pixel quantization, MSPD, and detector recall	Stratify by bbox area, projected diameter, and metric diameter
Range / depth	Z error, focal–translation coupling, and depth noise	By camera Z, sensor range, and translation components
Rotation / viewpoint	Training-pose coverage, in-plane / out-of-plane rotation, and canonical boundaries	SO(3) bins, view sphere, roll bins, symmetry quotient
Material / illumination	Transparent, specular, dark, weakly textured materials, and domain gaps	Asset labels + bins for sensor validity / exposure / lighting
Scene / clutter / count	Detector overload, inter-object occlusion, and duplicates	scene-level count, occluder type, background domain
Modality health	RGB blur, depth holes, synchronization / calibration, and missing modalities	raw sensor QA, confidence, dropout / fallback route
Failure stage	detector, mask, correspondence, solver, refiner, score, tracking	GT crop / GT mask / oracle PnP / oracle selection stage ablations
Task / robot state	Reachability, grasp family, contact tolerance, calibration, and motion	pose-only / calibrated chain / simulated / hardware success ladder

Observed E2E

Detector → Mask → Correspondence → Solver → Refiner → Score

Retain all deployed candidates, misses, duplicates, and timeouts.

↑

Oracle selection

Select the minimum error among actual candidates

Upper bound on current candidate coverage; the gap from actual score-based selection diagnoses ranking / calibration.

↑

Oracle solver / correspondence

GT correspondences or GT pose initialization

Diagnoses PnP / RANSAC / the refinement attraction basin.

↑

GT mask / crop

Remove detector / segmentation errors

Upper bound for the pose head; cannot serve as the end-to-end headline.

Per-instance error record · Make every plot recomputable

scene_id / image_id / gt_id / object_id / physical_instance
visibility / truncation / projected_size / camera_z / material_tags
prediction_id / rank / confidence / runtime / failure_stage
rotation_raw / rotation_sym / translation_xyz
ADD / ADD-S / Proj2D / VSD[tau] / MSSD / MSPD
matched_gt / TP-FP-FN per threshold / ignore_reason
asset_hash / symmetry_hash / evaluator_commit / render_backend
overlay_path / depth_diff_path / log_path

09 · Robotics, Tracking & Efficiency

Geometric scores are proxies; robots also need task success, safety, and a computation budget

Evaluate deployment along a ladder: pose-only → calibrated robot chain → planner / simulator → hardware trials. Retain inputs, failure stages, and pose oracles at each level to identify what to improve.

Grasping

Contact / collision sensitivity

MSSD's worst surface displacement is closer to local contact risk than mean ADD, but task trials must still account for grasp family, jaw width, collision, and force closure.

Insertion

Axis / key / tolerance

Separate insertion-axis, lateral, roll, and key-feature errors; an overall 5°5cm threshold cannot express submillimeter clearance.

Visual servo

Temporal continuity / latency

Report control-rate latency, jitter, tracking failure duration, recovery, and jumps between quotient representatives alongside per-frame mean pose errors.

Safety

Tail and abstention

P95 / worst-slice results, risk–coverage, OOD / sensor-health fallbacks, and hardware interlocks often matter more than mean AR.

Efficiency dimension	Required details
Latency definition	End-to-end wall-clock time per image, object, hypothesis, or batch throughput; whether warmup and preprocessing are included
Hardware	CPU/GPU/accelerator, memory, driver, precision, power mode, threads, and batch size
Onboarding / training	Per-object onboarding time for seen/unseen-object tasks, rendering/templates/cache, and maximum resource limits
Memory	Peak VRAM / RAM, model size, template / feature database, and persistent cache
Throughput vs tail	Alongside mean FPS, report median / P95 latency, queuing, synchronization, and worst-case candidate count
Energy / deployment	Power consumption, thermal throttling, edge device, network / server latency, and fallback resources
Accuracy–cost frontier	Sweep refinement iterations, top-K, resolution, and ensembles under the same protocol; report the Pareto frontier

Accuracy–cost Pareto report

\mathcal F=\left\{m\ \middle|\ \nexists m'\!:\ S(m')\ge S(m),\ C(m')\le C(m)\ \text{and at least one strict}\right\}

S may be AR/AP/task success; C may be P95 latency, energy, or memory. Comparing only FPS or only accuracy hides the actual trade-offs of refinement, top-K, and ensembles.

10 · Reproducibility & Release Gates

The smallest reproducible evaluation unit is a result file plus complete evaluator state

First reconcile expected targets, prediction records, and evaluated records. Then verify assets, cameras, symmetry, renderer, thresholds, and aggregation.

Failure modes	Symptoms	Detection	Fix
Wrong units / frame	All methods show huge translation errors on one axis; ADD is off by 1000×	Matrix projection, known synthetic poses, m/mm, and m2c/c2m round trips	Stop evaluation first; do not hide the error with a larger threshold
Wrong evaluation mesh / origin	Rotation error looks normal while all surface metrics are biased	Hash models_eval; check bbox, diameter, vertex count, and frame overlays	Freeze asset versions and archive them with the results
Inconsistent symmetry metadata	Equivalent representatives yield conflicting ADD/MSSD results and per-object anomalies	Render symmetry orbits; audit surface invariance, texture, and instances	Freeze geometry/appearance symmetry and toolkit discretization
Overly permissive ADD-S	Large errors on imperfectly repeated structures still yield nearby nearest points	Report ADD, MSSD, rendered overlays, and task outcomes together	Do not treat ADD-S as a general explicit symmetry metric
VSD depth / renderer mismatch	Anomalous scores on transparent/specular or invalid-depth samples	Record depth validity, renderer conventions, δ/τ, and visual heatmaps	Use the official renderer/toolkit; report sensor failures separately
Comparing 5px across cameras	High-resolution / long-focal-length datasets appear worse	Compare pixel error with image width, fx, and metric displacement	Fix the camera protocol or also report normalized / 3D metrics
Evaluating only matched detections	The pose head scores highly while the end-to-end system misses many objects	Report GT-crop / detector-crop results, the pose-oracle gap, and AP/AR together	Retain detector failures and identify their stage
The illusion that duplicates help	Oracle top-K scores are high, but actual scores select the wrong candidate and AP drops	One-to-one matching, top-1/top-K, NLL/calibration, and duplicates	Train ranking / deduplication; evaluate through the deployed interface
Micro hides the long tail	Common large objects improve and the overall score rises, while small objects fail	micro vs object-macro, per-object / scale bins	Preregister weights and disclose counts
False per-frame independence	Extremely narrow CIs and tiny p-values, yet failure on a different sequence	Account for correlation by scene/sequence/physical instance	cluster bootstrap / grouped split
Test-set tuning	Thresholds, symmetry tolerances, NMS, and temperature are optimized on test data	Track configuration provenance and access logs	Freeze validation/calibration splits; run only the final protocol on test data
Silently dropping failed samples	NaN, missing poses, timeouts, or missing files are excluded from the denominator	Reconcile counts/checksums of expected targets and evaluated records	Explicitly convert failures to FN / maximum error and record reasons according to the protocol
Incomplete AUC / AR names	Scores with the same 'AR' or 'AUC' name differ by tens of points	Check error, threshold grid, upper bound, dataset/year/aggregation	Include the full protocol ID in field names
Irreproducible speed	Reported FPS measures only network forward passes; deployment is several times slower	End-to-end profiler, warmup, synchronization, P95, and candidate count	Report stage breakdowns and hardware/precision together

Release gate · Before publishing any pose-benchmark number

Is the task input explicit: GT crop, localization target list, detection, tracking, or robot episode?
Do dataset/split/targets, evaluation mesh, camera, diameter, and symmetry all have hashes / versions?
Do all poses share the same `^C T_O`, frames, m/mm units, timestamps, and camera model?
Does the error function specify mean/max, correspondence, visibility, symmetry, and renderer?
Are thresholds, strict inequalities, AUC limits, VSD δ/τ, and continuous-symmetry discretization frozen?
Is matching one-to-one, with defined score ties, duplicates, ignore rules, top-N, and missing outputs?
Can every AR/AP/AUC layer trace from the headline to metrics, thresholds, objects, datasets, and instances?
Are counts reported alongside micro, macro, per-object, visibility, scale, distance, and failure-stage results?
Are method comparisons paired, with intervals based on independent scene/sequence/physical-instance units?
Do NaN, timeouts, crashes, missing poses, and missing files enter the denominator rather than being silently dropped?
Is speed measured with synchronized end-to-end wall-clock time, reporting hardware, batch, precision, warmup, P95, and memory?
Is test data fully excluded from threshold, NMS, symmetry tolerance, calibration, and model selection?
Are raw official-evaluator outputs, per-instance errors, result CSV, configuration, commits, and logs saved?
Have geometric scores been validated against actual task / hardware holdouts for their relationship to deployment success and safety?

Original sources for pose errors, BOP tasks, detection AP, and evaluation implementationsExpand for 6D evaluation, BOP 2018/2020/2024, official tasks/toolkit, PoseCNN, LINEMOD, and COCO

Behind every score, there is an entire protocol

Freeze the ten-layer evaluation contract before running the evaluator

Claim

Evaluation unit

Error function

Equivalence

Correctness

Matching

Aggregation

Uncertainty

Diagnostics

Reproducibility

Rotation and translation are useful diagnostic coordinates, but do not form a complete task metric

Rotation Error

Translation Error

n° / m cm

Corresponding, nearest, worst, and projected points: four relationships yield four conclusions

ADD

ADD-S

2D Projection

MSSD

MSPD

VSD examines currently visible surfaces; BOP AR aggregates three application perspectives across thresholds

Pose error

Correctness

Recall

ARₑ

AR_D / AR_C

Pose-only, localization, detection, tracking, and manipulation require different leaderboards

Candidate generation

Eligibility

One-to-one

Ordering

Assignment details

Ignored targets

Multi-hypothesis

Ties / duplicates

See metric conflicts, duplicate candidates, and averaging weights in action

raw / quotient-aware

camera-frame center

mean 3D surface

mean corresponding pixels

symmetry-aware maximum

visible raster discrepancy

—

predictions after task rule

one GT at most once

at full ranked list

—

—

A − B · scene bootstrap

—

pose-only − end-to-end

—

The headline is a reduction tree that must trace back to per-sample records

Obtain paired outcomes / errors for both methods on the same GT instances and compare per-unit differences

Resample scenes, sequences, physical instances, or capture sessions with replacement

Repeat the complete training / sampling / inference process; report individual seeds and central tendency

Report absolute percentage points, relative change, continuous error differences, and actual task margins

Report mean, median, P90/P95, CDF / recall curves, and catastrophic failure rate together

Report sample counts and intervals for every visibility / scale / symmetry / object slice

Overall scores show who ranks higher; slices and oracles explain why

Detector → Mask → Correspondence → Solver → Refiner → Score

Select the minimum error among actual candidates

GT correspondences or GT pose initialization

Remove detector / segmentation errors

Geometric scores are proxies; robots also need task success, safety, and a computation budget

Contact / collision sensitivity

Axis / key / tolerance

Temporal continuity / latency

Tail and abstention

The smallest reproducible evaluation unit is a result file plus complete evaluator state

Symmetry, Ambiguity & Uncertainty

Robot Coordinate Transforms & System Calibration

Behind every score,
there is an entire protocol