16Engineering contractsNumerical diagnosticsSystem debuggingDeployment gates

Engineering Implementation & Debugging

From incorrect results to
a reproducible fault at the responsible layer

The costliest 6D-system errors often arise when two individually plausible modules assign different meanings to the same array. This topic turns implicit assumptions into machine-checkable contracts and uses independent evidence to locate the first failing layer.

Open the fault-injection lab Freeze the artifact contract first

10-second summary

Remember seven rules

01
Check contracts before tuning algorithms: frames, directions, units, ordering, pixel domains, timestamps, and asset hashes.
02
Valid numbers can have incorrect semantics: det=1, q norm=1, and round-trip=0 miss inverse, wxyz, and degree/radian errors.
03
Image transforms must carry K and labels with them: crop, resize, pad, flip, distortion domains, and pixel conventions.
04
Low residuals do not establish correctness: degeneracy, multiple solutions, outliers, ICP local basins, and reflections can all fit well.
05
Static/dynamic, near/far, and center/edge A/B tests locate calibration, timing, K, or distortion issues faster than blind model changes.
06
Ground truth must be independent: overlays, labels, and closure generated by one pipeline can all be wrong together.
07
Failures are data: NaN, timeout, no-solution, stale, and abstain have explicit states, owners, and denominators.

01 · Artifact Contract

Every array must identify what it is, its frame and units, when it is valid, and which version produced it

Do not transmit physical semantics through variable names and team assumptions. Use the same verifiable manifest for files, messages, caches, model inputs/outputs, and robot interfaces.

Field group	Minimum declaration	Forbidden implicit assumptions
Identity	sample / scene / object / instance / camera / robot / task IDs	Strings and integers also need namespaces; array indices must not silently serve as object IDs
Transform	from/to frame, map direction, active/passive, column/row, handedness	`extrinsic`, `camera_pose`, and `T` are insufficient field names
Rotation	Matrix / quaternion / axis-angle / Euler; xyzw/wxyz; Euler sequence and intrinsic/extrinsic convention	Quaternion norm=1 or det=1 establishes numerical validity, not shared conventions
Translation	Unit, origin, reference point, scale policy, and valid range	Convert m/mm, CAD units, and normalized object scales once at interfaces
Camera	resolution, K/D/P, distortion model, raw/rectified domain, ROI/binning	Every image/tensor uses a camera model consistent with its own pixel domain
Image transform	Order, parameters, interpolation, and pixel conventions for crop/resize/pad/flip/rotate	Synchronize images, masks, keypoints, bboxes, flow, depth, normals, and pose labels
Depth / modality	encoding, unit/depth_scale, invalid/no-return, registered-to, timestamp/clock	Equal dimensions do not establish RGB-D alignment; never guess the semantics of 0, NaN, or ±Inf
Asset	CAD hash/version, unit, origin, axes, diameter, symmetry, mesh/texture/collision variants	Visualization, evaluation, and collision meshes and network templates may differ
Estimate	timestamp, source/model/config/checkpoint/preprocess hash, score/uncertainty/status	NaN, timeout, no-solution, and abstain are explicit states; do not disguise them as identity poses
Runtime	library/driver/device/precision, seed/determinism, latency, valid interval	A shared seed does not guarantee identical results across releases, CPU/GPU, or nondeterministic kernels

Typed SE(3) artifact, not an anonymous 4×4 array

{}^{A}\mathbf T_B(t)=\begin{bmatrix}{}^{A}\mathbf R_B(t)&{}^{A}\mathbf t_B(t)\\\mathbf0^{\mathsf T}&1\end{bmatrix},\qquad \mathbf p_A={}^{A}\mathbf T_B\,\mathbf p_B

Include from=B, to=A, column-vector mapping, right-handedness, translation units, timestamp/clock, source/version, and validity. Matrix values alone cannot recover this metadata.

Pose artifact manifest · Minimal executable example

pose:
  map: object -> camera_optical      # {}^C T_O
  vector: column · active · right_handed
  rotation: matrix3x3 · row_major_storage · projected_to_SO3=false
  translation_unit: metre
  timestamp: exposure_time · clock=camera_hw · valid_until=...
camera: image_domain=rectified · width/height · P_hash · preprocess_hash
asset: object_id · mesh_hash · unit=millimetre · origin_hash · symmetry_hash
producer: model/checkpoint/config/code hashes · dtype/device · status
quality: score_semantics · covariance_perturbation · failure_reason

02 · Evidence Ladder & Differential Diagnosis

Start with the cheapest, most certain checks; contain errors at the first failing layer

Change one factor at a time, save inputs and evidence, and propose at least two competing hypotheses for each symptom. Final optimizer logs cannot override earlier schema / geometry gates.

L0 · File / transport

Paths, keys, shapes, dtypes, endianness, finiteness, row strides, and message timestamps

File counts/checksums, schema validators, and the first non-finite index

L1 · Contract

frame, direction, unit, ordering, pixel/depth domain, asset lineage

manifest diff, sentinel transform, unit/range assertions

L2 · Algebra

SO(3)/SE(3), bottom row, inverse/compose, quaternion norm, reflection

RᵀR, det, T·T⁻¹, known point, Jacobian finite difference

L3 · Sensor geometry

K/D/P, crop/resize, RGB-D registration, cheirality, render domain

project↔back-project, raw↔rectified A/B, depth histogram

L4 · Observation

correspondence, visibility, mask, point support, symmetry mode, timestamp

matches overlay, per-point residual, static/dynamic and near/far slices

L5 · Solver

degeneracy, conditioning, initial basin, robust sampling, termination, multi-solution

spectrum/condition, top-K solutions, seed/init sweep, held-out residual

L6 · System

hand-eye/TCP, TF graph, latency, cache/version, thread/queue, robot state

fixed target closure, speed reversal, cold start, artifact atomic-load test

L7 · Task

Grasp/contact/clearance/insertion/safety outcomes and recovery

pose/grasp/motion oracle, hardware sentinel, failure-stage denominator

Static ↔ dynamic

Fails only in motion

Check timestamps, clocks, rolling shutter, latency, robot interpolation, and compliance before static K.

Near ↔ far

Error grows with distance

Check focal/scale/depth models first; fixed pixel offsets suggest principal-point / crop / pad issues.

Center ↔ edge

Radial patterns at image edges

Check distortion/rectification and pixel domains first; a fixed global offset suggests K′/frame errors.

Cold ↔ warm / old ↔ new

Depends on startup or version

Check caches, partial deployment, configuration shadowing, device/precision, and nondeterminism first.

Interactive Lab

Inject one fault and see which checks catch it and which good-looking metrics still pass

The first panel shows semantic errors in valid matrices. The second composes pixel affines, K, and depth back-projection. The third routes PnP, ICP, timing, NaN, reflection, and version issues through the first failing gate.

reference contract decoded artifact invalid / behind camera Low round-trip error establishes algebraic consistency, not correct frames, units, or semantics

Schema contract

metadata before math

——

SO(3) invariants

‖RᵀR−I‖F / det R

——

Geometry

positive depth / translation

——

Independent overlay

reference reprojection RMSE

—Independent 2D observations; labels are not generated from the pose under test

Algebraic closure

T · T⁻¹

—Wrong directions, units, and quaternions can still be perfectly invertible

Route

—

——

Validate contracts before numbers. det=1, orthogonality, and quaternion norm=1 cannot detect valid rotations with wrong semantics. Use known points, real images, fixed targets, robot touch, or independent sensors as semantic sentinels.

03 · Transform, Rotation & Numerical Hygiene

Invariants reject invalid values; sentinels reject incorrect semantics

Rotation orthogonalization restores a numerical manifold constraint; it cannot repair frames, units, or labels. Count every automatic correction and gate its magnitude.

SO(3) input gate

\|\mathbf R^{\mathsf T}\mathbf R-\mathbf I\|_F<\varepsilon_R,\qquad |\det\mathbf R-1|<\varepsilon_d,\qquad \operatorname{isfinite}(\mathbf R,\mathbf t)

Thresholds depend on dtype and chain length. det≈−1 indicates reflection; det≈1 still cannot determine active/passive conventions, frame direction, Euler order, or quaternion ordering.

Nearest proper rotation after SVD

\mathbf M=\mathbf U\boldsymbol\Sigma\mathbf V^{\mathsf T},\qquad \Pi_{\mathrm{SO}(3)}(\mathbf M)=\mathbf U\,\operatorname{diag}\!\left(1,1,\det(\mathbf U\mathbf V^{\mathsf T})\right)\mathbf V^{\mathsf T}

The last factor prevents reflection. Large corrections should block execution and trigger model/integration investigation, rather than silent projection and continuation.

Quaternion normalization and sign-invariant comparison

\bar{\mathbf q}=\mathbf q/\|\mathbf q\|_2,\qquad d_{\mathrm{SO}(3)}(\mathbf q,\hat{\mathbf q})=2\arccos\!\left(\operatorname{clip}(|\mathbf q^{\mathsf T}\hat{\mathbf q}|,0,1)\right)

Check component ordering and rotation conventions first; q and −q encode the same rotation. Normalization cannot fix wxyz/xyzw errors.

Issue	Typical signature	Minimal isolation experiment / fix
m ↔ mm / deg ↔ rad	Translation differs by exactly 10³; angles jump periodically with input while R remains orthogonal	Use SI internally; include units in field names; range histograms + a known baseline
T_AB ↔ T_BA	Axes look plausible, but the object lies behind the camera or the robot moves the wrong way	Verify mapping direction with a known non-origin point, alongside T·T⁻¹ tests
Left ↔ right multiplication / row ↔ column vectors	One transform works, but composing two edges fails; translation is rotated incorrectly	Freeze the column-vector contract; test composition with noncommuting transforms
active ↔ passive	Values resemble an inverse/transpose; Euler signs are reversed throughout	State whether vectors rotate or bases change, and provide a physical-axis sentinel
right ↔ left handed	Mirrored overlays, reversed normals/triangle winding, det=-1, or det=1 after implicit axis flips	Represent every handedness conversion with an explicit matrix; check winding, normals, and image axes
xyzw ↔ wxyz	R remains valid, and some special rotations may even be correct by coincidence	Serialize named fields or scalar_first; test identity and an asymmetric sentinel quaternion
q ↔ −q	Elementwise quaternion loss / interpolation jumps while the rotation matrix is unchanged	Compare with \|qᵀq̂\| or an SO(3) geodesic; sequences may be canonicalized while retaining double-cover semantics
Euler order	The same three angles produce entirely different orientations; discontinuities near gimbal lock	Record sequence, intrinsic/extrinsic convention, and radians in the API; use Lie / quaternion variables for internal optimization
rotation drift	RᵀR drifts from I after long compositions; transposing no longer gives an accurate inverse	Monitor orthogonality and project to SO(3) when needed, while tracing integration / precision causes
SVD reflection	Low Umeyama/Kabsch/Procrustes residuals but det=-1	R=U diag(1,1,det(UVᵀ)) Vᵀ; report whether correction was triggered
4×4 bottom row	The last row becomes zeros, NaN, or perspective terms after deserialization / concatenation	Assert `[0,0,0,1]`, finiteness, shape, and contiguous-memory semantics
dtype / tolerance	Errors grow with float32 chains, tiny objects, or large coordinates; default allclose settings misclassify results	Choose atol/rtol by units and task; test shape/dtype and retain float64 for critical calibration and solving

Prerequisite review: Coordinate systems and rigid transforms, Mathematics & Rigid Geometry, Rotation & Orientation Representations.

04 · Camera Preprocess, Distortion & RGB-D

The camera model must match the current pixel array, not a historical sensor resolution

Treat crop/resize/pad as pixel homogeneous transform A. Raw, undistorted, rectified, network-input, RGB, and depth images occupy different domains.

Image affine and updated intrinsics

\tilde{\mathbf p}'=\mathbf A\tilde{\mathbf p},\qquad \mathbf K'=\mathbf A\mathbf K,\qquad \mathbf A=\begin{bmatrix}s_x&0&p_x-s_xc_x\\0&s_y&p_y-s_yc_y\\0&0&1\end{bmatrix}

Here c_x,c_y are raw crop offsets, and p_x,p_y are post-resize padding. More complex rotations/flips continue left-multiplying A. If they produce reflection, define the 3D pose-label action separately.

Pinhole back-projection closure

\mathbf p_C=z\mathbf K^{-1}[u,v,1]^{\mathsf T}=\left[\frac{(u-c_x)z}{f_x},\frac{(v-c_y)z}{f_y},z\right]^{\mathsf T}

Pixels, K, and z must share the same camera/rectification/registration domain. z is optical-axis depth, which may differ from Euclidean range.

Radial/tangential distortion domain

x_d=x(1+k_1r^2+k_2r^4+k_3r^6)+2p_1xy+p_2(r^2+2x^2),\quad r^2=x^2+y^2

Do not compare raw distorted pixels directly with ideal K projections. Fisheye/rational and other models require their own formulas; empty D may also mean the image is already rectified.

Depth encoding to metric value

z_{\mathrm{SI}}=d_{\mathrm{raw}}\,s_d\,10^{-3}\quad\text{for a BOP-style }s_d\text{ that yields millimetres}

BOP multiplies each image's raw depth by depth_scale to obtain mm. ROS 16UC1 commonly uses mm and 32FC1 commonly uses meters, but the producer contract is authoritative; dtype alone is insufficient.

Issue	signature	Fix / evidence
Raw K on a resized tensor	Error grows with pixel distance from the principal point	`p′=Ap, K′=AK`; generate K and the preprocessing manifest from the same source
Wrong crop / resize order	A fixed, scale-dependent principal-point offset	Compose pixel-affine transforms in execution order instead of scattering manual cx/cy patches
Nonuniform resize	x is largely correct, but y has a systematic offset	Maintain sx/sy separately and update fx/fy/cx/cy
Missing pad / letterbox offsets	All points have a fixed padding offset, including near the center	Include padding translation in A; share code between training labels and inverse inference transforms
Horizontal flip	2D labels are flipped without synchronizing 3D pose / object symmetry	Specify image augmentation and its pose action; reflection is not an ordinary SO(3) rotation
Mixing raw / undistorted / rectified domains	Good at the center, with radial / tangential edge patterns	Record the distortion model and domain; use K/D or P/R as a consistent set
Pixel center / align_corners	Approximately 0.5 px bias after resize round trips, more evident at low resolution	Freeze sampling conventions and library/version; add impulse-grid regression checks
16UC1 / 32FC1 depth	Point clouds are enlarged 1000-fold or drastically shrunk	Encoding + scale determines SI values; BOP depth = raw·depth_scale mm
Invalid depth	0 is treated as the camera center; NaN silently contaminates SVD/ICP	Apply invalid masks before calculations; record valid fractions and the first bad index
Unregistered RGB-D	Persistent color/depth edge offsets, stronger on nearby objects	Reproject with depth K and RGB←depth extrinsics, or consume an explicitly registered product
Timestamp / rolling shutter	Correct at rest, biased in motion; errors vary along the row direction	Use exposure time, a readout model, and a shared clock; arrival time is insufficient
Channel / normalization	Network outputs degrade globally while geometric self-checks remain normal	Record RGB/BGR, range, mean/std, gamma/bit depth; save a checksum of the first-layer input

Prerequisite review: Camera Models & Imaging Geometry, RGB, RGB-D & Multimodal Pipelines.

05 · CAD, Labels & Dataset Integrity

Pose semantics bind to a specific asset; changing mesh origin changes the coordinate frame

Version assets, labels, augmentations, splits, and symmetry together. A convincing overlay may simply mean the render mesh and labels share the same error.

01

CAD unit / scale

A mesh stored in mm is rendered as meters; normalized category shapes are treated as metric

bbox/diameter/range gate + physical ruler sentinel

02

Origin / canonical axes

Silhouettes align, but translation/rotation differs from published labels by a fixed transform

Version asset transforms separately; do not alter pose labels to absorb CAD-origin changes

03

Mesh variants

Mixing render, evaluation, collision, or simplified meshes

Hash each purpose separately; specify the fixed model-to-variant transform

04

Symmetry metadata

Visually equivalent predictions are penalized, or task-critical keys are incorrectly merged

Proper symmetry group + task equivalence; version continuous/discrete sampling

05

Normals / winding

Abnormal point-to-plane ICP direction, rendering back-face culling, or reversed suction normals

Recompute / visualize normals; check orientation consistency and face winding

06

Texture / color space

Systematic color differences between synthetic templates and real inputs

Include ICC/gamma/linear-sRGB, tone mapping, and texture versions in the manifest

07

Label augmentation

Images are cropped/flipped without synchronizing masks/keypoints/poses/K

Apply the same typed transform to all annotations; test round-trip properties

08

Split / leakage

Unusually high validation scores collapse on new instances / scenes

Group by physical instance/scene/session, check duplicate hashes, and freeze split manifests

Asset-frame migration must preserve the physical pose

{}^{C}\mathbf T_{O'}={}^{C}\mathbf T_O\,{}^{O}\mathbf T_{O'},\qquad \mathbf p_C={}^{C}\mathbf T_O\mathbf p_O={}^{C}\mathbf T_{O'}\mathbf p_{O'}

Mesh recentering / scaling / axis migration is an explicit asset transform; updating vertices requires corresponding pose-annotation changes.

Avoid shared-source validation loops.GT poses, rendered masks, bboxes, visible masks, and quality checks generated with the same wrong CAD origin/K/transform can agree perfectly. Retain manual landmarks, real depth, independent trackers/touch, or a second implementation as external sentinels.

Engineering review: Real-world Annotation, Simulation and synthetic data, Symmetry, Ambiguity & Uncertainty.

06 · PnP, Registration & Optimizer Diagnosis

Solvers must establish solvability and trustworthy data before reporting an optimum

Save input support, inlier masks, candidate sets, initialization, condition spectra, termination reasons, and held-out evidence. Final R/t and one residual cannot reconstruct the investigation.

Reprojection objective with an explicit valid/inlier set

E_{\mathrm{PnP}}(\mathbf R,\mathbf t)=\frac{1}{|\mathcal I|}\sum_{i\in\mathcal I}\rho\!\left(\left\|\mathbf u_i-\pi(\mathbf K,\mathbf D,\mathbf R\mathbf X_i+\mathbf t)\right\|_2^2\right)

I, pixel domains, K/D, robust losses, and units all belong to the protocol. Low E does not rule out planar ambiguity, mirror solutions, wrong assets, or shared-source correspondences.

3D support planarity diagnostic

\mathbf C=\frac1N\sum_i(\mathbf X_i-\bar{\mathbf X})(\mathbf X_i-\bar{\mathbf X})^{\mathsf T},\quad \lambda_1\ge\lambda_2\ge\lambda_3,\quad \eta_{\mathrm{plane}}=\lambda_3/\lambda_1

Small η suggests near-planarity; small λ₂/λ₁ additionally suggests near-collinearity. Thresholds depend on noise, scale, and solver settings rather than universal constants.

Point-to-plane ICP objective

E_{\mathrm{ICP}}(\mathbf T)=\sum_{(i,j)\in\mathcal K}w_{ij}\left[\mathbf n_j^{\mathsf T}(\mathbf T\mathbf p_i-\mathbf q_j)\right]^2

K, normals, weighting/rejection, maximum correspondence distance, initial T, and overlap determine the basin. Small point-to-plane residuals can coexist with large tangential planar errors.

Issue	signature	Must retain / isolate
PnP correspondence	Wrong or duplicate object/image-point indices, or inconsistent coordinate domains	Draw point connections and residual vectors first; check undistortion, K, and object units
PnP planar / collinear	Multiple low-reprojection solutions with depth / orientation jumps	Inspect the centered 3D spectrum, retain `solvePnPGeneric` multiple solutions, and add nonplanar / depth / temporal evidence
PnP cheirality / mirror	Points behind the camera, or occasional selection of a planar mirror solution	Positive depth for all control points + silhouette/depth/temporal validation
RANSAC	Wrong threshold units / pixel domain; high confidence with incorrect inliers	Record sampler/seed/threshold/iterations, inlier masks, and post-refinement residuals
Conditioning / scaling	Rotation/translation scale differences make normal equations ill-conditioned	Center/scale data, report singular spectra / κ, and choose appropriate parameterizations and robust losses
ICP initial basin	Low RMSE at an incorrect local surface	Global/coarse initialization, overlap/fitness, reverse checks, multiple starts, and independent keypoints
ICP correspondence gate	Large thresholds admit background; small thresholds yield 0 correspondences	Sweep thresholds against resolution/noise/initial error; report correspondence counts
Point-to-plane normals	Fast planar convergence with unconstrained tangential motion; incorrect normals misdirect updates	normal orientation/scale, Hessian eigenvalues, point-to-point cross-check
Symmetry / multi-mode	The solver jumps among equivalent modes; averaging poses yields an invalid intermediate	Output sets/distributions and evaluate in quotient/task space; do not average rotations directly
Termination	'Converged' only because steps are small or iteration limits are reached, while residuals / independent errors remain large	Gate on finiteness, gradient/step, support, held-out evidence, physical closure, and task risk together

Algorithm review: 2D–3D Correspondences & PnP, 3D–3D Alignment & Point Cloud Registration, Pose Evaluation & Error Analysis.

07 · Time, Runtime & Reproducibility

Offline correctness is necessary; online systems must also prevent stale values, mixed versions, and disguised failures

Throughput is not latency, arrival time is not exposure time, a seed does not ensure full reproduction, and hot updates are not atomic deployments. Every estimate has an age and valid_until.

First-order motion error from timestamp mismatch

\|\delta\mathbf p\|\lesssim \|\mathbf v\|\,|\Delta t|+\|\boldsymbol\omega\|\,|\Delta t|\,\ell

Static A/B tests isolate temporal terms; ℓ is the observed/contact-point lever arm. Also consider rolling-shutter row timing, queue jitter, and controller actuation delay.

Absolute + relative numerical test tolerance

|a-b|\le \mathrm{atol}+\mathrm{rtol}|b|

Choose atol by units and physical tolerance near zero; rtol handles scale. Default allclose may be too permissive for small values, while broadcasting/equal_nan can hide shape/NaN bugs.

NaN / Inf

Failure appears inside the optimizer, with a stack trace far from the first bad value

Per-layer finite gates + sample/stage indices; use explicit status/null in JSON instead of nonstandard NaN

Shape broadcasting

Code runs without errors but incorrectly reuses batch/point axes

Typed dimensions and strict shape tests; do not let implicit broadcasting conceal errors

Device / precision

Normal on CPU, intermittent failures on GPU/FP16

Record device/dtype/autocast; compare critical SVD/solves and reductions with float64/FP32 A/B tests

Nondeterminism

Variation despite a fixed seed; inconsistencies across releases/platforms

Lock versions/hardware/algorithms; use deterministic mode and repeated replay, accepting performance costs

Stale queue

A new image is displayed with an old-frame pose; latency varies with load

frame_id + exposure timestamp + sequence; discard expired results and measure end-to-end age

Cache / partial deploy

A new checkpoint uses old CAD/K/preprocessing; cold starts and hot updates disagree

Atomic manifest loading, content hashes, cache namespaces, and cold-start CI

Serialization

Changes in matrix flattening order, integer truncation, JSON nulls / duplicate keys

Schema round-trip golden tests; do not depend on object-key ordering

Config shadowing

Unclear sources of CLI/env/YAML default overrides

Print resolved configuration + provenance at startup; reject unknown fields instead of silently ignoring them

Concurrency

Reused shared buffers, unsynchronized asynchronous GPU work, or out-of-order callbacks

Immutable messages / copy ownership, sequence assertions, synchronized timing, and race tests

Silent failure

Missing poses/timeouts are replaced by identity or the previous frame

Status enum + reason + valid_until; downstream consumers must explicitly handle abstain/stale/failure

Atomic deployment tuple

release_id = hash(
  code_commit, resolved_config, model_arch, checkpoint,
  preprocess, camera_intrinsics, distortion/registration,
  CAD/eval/collision assets, symmetry metadata,
  robot URDF/kinematics, hand-eye, TCP, controller,
  libraries/drivers, device/precision, schema version
)
estimate.status ∈ {ok, abstain, no_solution, timeout, stale, invalid_input, runtime_error}

System review: Robot Coordinate Transforms & System Calibration, Robotic Grasping & Manipulation.

08 · Debug Views Are Measurement Instruments

Each plot answers a hypothesis; label every overlay with frame, domain, timestamp, and scale

Visualization should support investigation as well as demonstration. Enable drill-down to samples, points, edges, stages, and versions. Share data with machine-readable residuals while independently implementing critical projections.

RGB overlay

Projected model axes / bbox / silhouette over the original image

Direction, K, distortion, crop, and coarse pose; cannot reveal all depth-unit / occluded-geometry problems

Depth residual

Rendered Z versus measured Z, compared only on visible, valid pixels

Depth scale, z translation, occlusion, and RGB-D registration; affected by sensor bias / missing depth

Point / match vectors

Every correspondence, inlier/outlier, and residual arrow

Indices, pixel domains, local bad regions, and RANSAC; mean RMSE hides structure

3D overlay

CAD / point clouds / axes / frusta / trajectories in one world frame

Scale, origin, handedness, and registration; label the viewer's up/forward conventions too

Frame graph

Direction, age, source, covariance, and loop closure for each edge

Wrong frames, stale transforms, duplicate broadcasters; a connected tree does not establish extrinsic accuracy

Error slices

near/far, center/edge, static/dynamic, view/load/temp, object/visibility

Narrow the responsible layer from signatures; global means are insufficient

Time plot

Exposure, arrival, inference, command, actuation, and queue age

Offset, jitter, stale results, and confusion between throughput and latency

Physical sentinel

Fixed calibration boards, known CAD poses, touch points, robot gauges, or independent trackers

Breaks shared-source data loops; must be independent of the artifact and solver under test

Depth residual on mutually visible valid pixels

r_Z(u)=Z_{\mathrm{sensor}}(u)-Z_{\mathrm{render}}(u;\mathbf T),\qquad u\in\Omega_{\mathrm{valid}}\cap\Omega_{\mathrm{visible}}

Show signed maps, histograms, and image-location slices together. Missing depth, occluded models, and neighboring surfaces are not pose residuals.

Fixed vector everywhere

K principal point / crop / TCP / fixed extrinsic

If offsets stay fixed as depth, view, and speed change, check constant edges and pixel offsets first.

Grows with radius

distortion / focal / scale

Small at the center, large at edges: plot residuals by radial direction.

Grows with depth

focal / translation scale / depth model

Separate near/far and pixel/mm domains to avoid projection-induced confusion.

Only with speed / direction

timestamp / rolling shutter / backlash / flex

Three A/B comparisons: static–dynamic, forward–reverse, and light–heavy payload.

09 · Test Pyramid, Logging & Regression

Test physical invariants, equivalence across representations, and failure paths alongside successful API returns

Golden truth is independent of the pipeline under test; reduce every production incident to a minimal regression sample. CI runs schema, numerical, static-build, runtime, and representative real-data checks together.

Unit

inverse/compose, projection/backprojection, quaternion/Euler, K affine, depth scale

Asymmetric numerical cases, nonzero translations, noncommuting transforms, and failures

Property

Random SE(3) round trips, SO(3), A·K pixel closure, and augmentation round trips

Fixed seeds with varied scales / boundaries; shrink counterexamples and add regressions

Golden artifact

A small manually confirmed set of images/depth/CAD/K/poses and expected overlays/metrics

File hashes + human-reviewed snapshots; ground truth is independent of the current pipeline

Metamorphic

Physical results should remain unchanged after unit conversions, frame changes, image resizing, or symmetry actions

Equivariance / invariance tests catch contract bugs better than a single expected number

Differential

Compare identical inputs across two libraries/solvers/precisions/CPU-GPU/old-new versions

Align conventions first; retain per-sample evidence of differences alongside means

Integration

Data loading→preprocessing→estimation→rendering→evaluation and camera→robot chains

Real files/message schemas, timestamps, cache cold starts, and failure states

Hardware-in-loop

Fixed targets, touch, slow motion, speed reversal, and payload/workspace sweeps

Begin safely at low speed; static/dynamic A/B isolates calibration/timing/control

Release / monitor

Manifest compatibility, P95 latency, drift heartbeats, risk/abstain behavior, and rollback

Derive gates from the deployment envelope; alerts include the owner and a minimal reproduction

Minimum debug record per estimate

sample/instance/task IDs · raw file/message checksums
image/depth/K/D/P/preprocess + asset/symmetry hashes
all frame edges with timestamp/clock/age/source
candidate poses, scores, covariance, inlier/correspondence masks
solver method/init/thresholds/iterations/termination/spectrum
projection/depth/closure/held-out/task residuals
runtime stage timestamps, device/dtype, resolved config/release_id
status + first_failure_stage + owner + recovery/action

10 · Incident Playbook & Release Gate

First identify the earliest contract violation and its layer before choosing a model

Before release, inject unit, inverse, quaternion, K, depth, reflection, degeneracy, NaN, timing, and cache faults. Confirm that the system blocks them and identifies the correct owner.

Release gate · Before poses enter evaluation or a robot

Does every pose declare from/to, mapping direction, row/column, active/passive convention, handedness, rotation representation/order, and units?
Are shape/dtype/finiteness, the 4×4 bottom row, RᵀR, det R, q norm, translation/depth ranges, and cheirality checked?
Do asymmetric known points / axes / physical sentinels validate semantics alongside inverse round trips?
Are current image/tensor resolution, crop/resize/pad/flip affine, raw/rectified domain, and K/D/P version-consistent?
Are depth encoding, depth_scale, invalid/no-return semantics, RGB↔depth registration, and timestamp/clock explicit?
Are units, origins, axes, diameters, normals, symmetry, and hashes frozen for CAD/eval/collision/render assets?
Does augmentation synchronize image/mask/keypoint/bbox/depth/normal/K/pose, with round-trip property tests?
Does PnP retain correspondences/inliers/multiple solutions, check planarity/conditioning/cheirality, and select solutions with silhouette/depth/temporal evidence?
Does ICP retain initial T, overlap/fitness, thresholds/correspondence counts, normals, conditioning, and reverse/independent checks?
Is optimizer convergence jointly gated by finiteness, support, conditioning, held-out/physical residuals, and task tolerance?
Are static/dynamic, near/far, center/edge, view/load/temperature, CPU/GPU, and cold/warm A/B tests used to identify signatures?
Do estimates carry exposure timestamps, clocks, and age/valid_until, with stale results discarded from queues?
Does each release atomically bind code/config/checkpoint/preprocess/K/assets/symmetry/robot calibration/driver/library/device/precision?
Are NaN/Inf, no-solution, timeout, stale, and abstain explicit states, without silent JSON/database replacement by identity / previous frames?
Do tests cover unit/property/golden/metamorphic/differential/integration/HIL, permanently retaining incident samples as regressions?
Can logs replay offline down to points/stages, including first failure, thresholds, versions, P50/P95 latency, and raw evidence?
Are domains/frames/units/timestamps labeled on overlays/depth/frame graphs, with critical sentinels independent of the tested pipeline?
Do task tolerances, risk, and the deployment envelope determine final gates, with abstention, rollback, recalibration, and human takeover?

Freeze

Freeze inputs and versions

Preserve the original data without further overwrites.

→

Reproduce

Minimal offline replay

Establish the deterministic envelope.

→

Bracket

Find the first failing layer

Use contracts / invariants / sentinels.

→

Isolate

Single-factor A/B

Assign the owner and contain patching to the cause.

→

Prevent

Add gates and regressions

Fix the root cause and verify rollback.

Original and official sources for coordinate / camera / solver / runtime contractsExpand for ROS REP/tf2/sensor_msgs, OpenCV, HZ, Umeyama, Higham, ICP, BOP, SciPy, NumPy, PyTorch, and RFC 8259