11 Synthetic Data Physical simulation PBR Sim-to-real Data Engineering

Simulation & Synthetic Data Generation

Beyond Putting CAD
into a Renderer

Synthetic data provide controlled scene-state sampling and precise, traceable multichannel labels from shared geometry. A reliable pipeline must establish the distribution, assets/physics, sensor behavior, and label correctness separately, beyond generating unlimited images.

Open the Synthetic Data Lab Start with the generation contract

01 · Generation Contract

Define the world to sample before rendering

A generator samples scene states from versioned random variables θ, then uses renderers and sensor models for observations and labels. Tasks, assets, cameras, channels, provenance, and validation form one contract.

01

Task support

Deployment objects, poses, views, occlusion, failure distributions

→

02

Scene state

Assets, physics, cameras, lights, trajectories, seeds

→

03

Clean renderer

RGB, Z, normals, IDs, XYZ from one z-buffer

→

04

Sensor layer

Exposure, distortion, noise, holes, quantization, timing

→

05

Audit & export

Distributions, labels, real holdouts, manifests, shards

Synthetic training distribution

\theta \sim p_{\mathrm{sim}}(\theta \mid \text{task}),\quad (x,y)=G(\theta),\qquad \min_\phi\ \mathbb E_{\theta\sim p_{\mathrm{sim}}}\left[\ell\!\left(f_\phi(x),y\right)\right]

θ includes assets, poses, cameras, scenes, visibility, motion, and sensor state beyond material colors. Broaden p_sim along task-relevant real distributions and remain robust to unknown nuisances, rather than making it arbitrarily wide.

Contract layer	Fields to fix	Verifiable deliverables
Task distribution	Object instances, scenes, poses, scale, distance, occlusion, truncation, motion, sensor state	Coverage report for synthetic and deployment distribution support
Asset contract	Mesh / collision mesh, object frame, units, materials, textures, physics, license, hash	Traceable assets with semantics matching real instances
Camera contract	optical frame, K / D, resolution, shutter, exposure, spectral response, timestamp	Every raster and pose traces to the same camera state
Scene state	Object/camera/light/background transforms, contacts, velocities, random seeds	Reconstruct any frame from state, beyond its rendered image
Clean labels	pose, clean Z / range, normal, ID, full / visible mask, object coordinate, flow	Derived from one z-buffer / scene snapshot
Sensor outputs	RGB response, distortion, blur, noise, quantization, invalid / confidence	Versioned observation layers that preserve clean ground truth
Provenance	code commit, engine/version, config, asset hash, seed, host, shard, checksum	Reproducible failures and resumable production
Validation	Synthetic unit tests, distribution audits, real holdouts, downstream ablations	Establish correct labels and a task-relevant generated distribution

Keep four quality axes separate from a single realism score Sampling fidelity Tests the task distribution; physical / rendering fidelity tests geometry and optics; sensor fidelity tests measurement formation; label integrity tests coordinates, units, visibility, and synchronization. Each needs distinct diagnostics and release gates.

02 · Digital Assets

Visual meshes, collision shapes, materials, and frames form one asset package

The real-data chapter fixed physical/digital semantics; synthesis must also make assets usable for optics, contact, and version governance. High-poly meshes may be poor collision models, and attractive textures may use incorrect color spaces.

Asset stage	Fixed decisions	Admission checks
Geometry import	Fix m/mm conversion, frames, up/front, origins, winding, normals, and UVs; separate visual and simplified collision meshes	Cross-check bounds, diameter, mass scale, face orientation, and projected axes between the renderer and BOP / OpenCV
Topology and contact	Remove isolated pieces, duplicate faces, self-intersections, and nonmanifold geometry; choose collision representations for thin shells, concavities, and support faces	No penetration or floating after drops; stable poses match real support surfaces rather than coarse convex-hull artifacts
Material semantics	Distinguish base color, roughness, metallic, normal, transmission, IOR, and texture color spaces	Mark numerical maps Non-Color; avoid physically impossible material combinations
Instance variation	Model manufacturing tolerances, wear, stickers, contamination, and texture symmetry from evidence instead of arbitrary recoloring	Keep training labels, evaluation symmetry, and real identity consistent
Physical properties	Record mass, center of mass, inertia, friction, restitution, damping, and joint limits	Drop times, rebound, rest poses, and stack stability match real physical scales
Copyright and versions	Retain source, author, license, modification/redistribution permissions, and content hashes	Generate per-asset release manifests; replacements must not silently change old data

Visual geometry

For imaging

Preserve silhouettes, holes, thin structures, normals, and UVs; LOD changes must preserve object frames and metric scale.

Collision geometry

For contact solving

Convex decomposition, SDFs, and triangle meshes alter stable poses. Collision approximations must be auditable assets rather than hidden engine options.

Semantic geometry

For tasks and labels

Symmetry, canonical axes, grasp affordances, articulation, and instance identity need explicit metadata beyond mesh filenames.

03 · Pose, View & Scene Sampling

Uniform parameters, uniform groups, and task distributions differ

Specify sampling space and measure. Uniform pitch clusters directions near poles; Haar-uniform SO(3) is unbiased over rotations but does not encode gravity, support, camera reachability, or real occlusion.

Shoemake · uniform unit quaternion

\begin{aligned}u_1,u_2,u_3&\overset{\mathrm{iid}}{\sim}\mathcal U(0,1),\\ q&=\left[\sqrt{1-u_1}\sin 2\pi u_2,\ \sqrt{1-u_1}\cos 2\pi u_2,\ \sqrt{u_1}\sin 2\pi u_3,\ \sqrt{u_1}\cos 2\pi u_3\right]\end{aligned}

Quaternion order here is [x,y,z,w]. q and −q encode one rotation; do not sample four independent uniform components and use them directly.

View sphere factorization

R = R_{\mathrm{look}}(\mathbf v,\mathbf u_{\mathrm{ref}})\,R_z(\gamma),\qquad \mathbf v\in\mathbb S^2,\ \gamma\in[-\pi,\pi)

View direction v specifies out-of-plane direction; γ specifies in-plane roll about the optical axis. Define look-at up/reference and degeneracies to establish zero roll consistently.

Sampler	Sampling target	Limits	Suitable settings
Independent uniform Euler angles	Uniform yaw/pitch/roll separately	Simple, but nonuniform on SO(3), with viewing-direction bias near spherical poles	Use only when the task genuinely follows an angle-box distribution
Haar-uniform SO(3)	Shoemake quaternions / normalized Gaussian quaternions	Unbiased over rotations, without gravity, reachability, or stable-support priors	General orientation coverage and ablation baseline
View sphere + roll	Equal-area directions + independent in-plane roll + distance	Separately controls out-of-plane and in-plane angles; define zero look-at roll	Template libraries, calibration boards, systematic view-sphere capture
Stable-pose prior	Enumerate support faces or cluster physical-drop outcomes	Matches tabletop tasks, but depends on collisions, friction, center of mass, and simulation steps	Tabletop detection and pregrasp perception
Physics scene sampling	Joint multi-object distribution, contacts, occlusions, camera visibility constraints	Closest to clutter; costly and vulnerable to penetration, unstable dynamics, and hidden filtering bias	Bins, stacks, robot manipulation
Empirical task prior	Hierarchical or conditional distributions fitted to real deployment logs	Most relevant, but logs may contain policy bias, missed detections, and missing long tails	Production and active data generation

Coverage

Plot histograms in the correct space

Use equal-area z–azimuth bins for directions, SO(3) distances/grids for rotations, and logarithmic or task-stratified distance/scale bins.

Joint distribution

Preserve correlations

Distance, pixel size, occlusion, blur, and depth validity are often correlated; independent randomization can create impossible combinations.

Quota

Cover long tails and record weights

Oversample rare poses or low visibility if needed, but retain training weights separately from real evaluation frequencies.

Split

Split by generator lineage

Near-duplicate frames sharing scene seeds, asset variants, textures, or trajectories must not cross train/test.

04 · Physics & Layout

Physical drops sample joint scenes, beyond decorating random rotations

Poses, occlusion, and contacts are correlated in bins, stacks, and tabletop scenes. Physics generates this structure only when scale, collisions, mass, solvers, and settling criteria are controlled together.

INIT

Nonoverlapping initial state

Drop regions, orientations, velocities, instance combinations

→

SIMULATE

Fixed dynamics contract

gravity, dt, substeps, contact, solver

→

SETTLE

Stop by state

‖v‖, ‖ω‖, contact duration, penetration

→

CONSTRAIN

Task-conditioned acceptance

Visibility, truncation, camera reachability, object count

Settling gate

\|\mathbf v_i\|<\varepsilon_v,\quad \|\boldsymbol\omega_i\|<\varepsilon_\omega\quad \forall i\quad \text{for }N_{\mathrm{stable}}\text{ steps},\qquad d_{\mathrm{penetration}}<\varepsilon_d

Define thresholds by units/object scale; log samples exceeding simulation limits as timeouts rather than settled states. Contact solving is usually not bit-exact across hardware, so replay uses tolerances and statistical fingerprints.

01

Scene scale

Gravity, dimensions, mass, collision margins, and solver tolerances must use consistent scales; treating cm meshes as m radically changes dynamics.

02

Initial placement

Start with broad-phase overlap tests; airborne positions, initial velocities, and orientations should match the feeder mechanism, rather than teleport from unreachable regions.

03

Time integration

Fix time steps/substeps, solver iterations, and determinism settings; do not assume bit-exact behavior across GPUs/backends.

04

Contact model

Static/dynamic friction, restitution, damping, contact offsets, and collision meshes jointly determine stable poses and occlusion.

05

Settling criterion

Check linear/angular velocity, sustained contact, and maximum penetration instead of a fixed frame count alone; define timeouts and rejection reasons.

06

Constraint filtering

Log rejection rates for camera clipping, visibility, truncation, collisions, and grasp reachability so rejection sampling does not silently alter the distribution.

07

Dynamic scenes

Save camera/object trajectories throughout exposure; rolling shutter, motion blur, and flow require more than midpoint poses.

08

Reproducibility

Alongside seeds, save engine versions, threads/GPUs, asset hashes, and solver configuration; compare replay states within tolerance.

05 · PBR, Lighting & Camera

Photographic appearance comes from a calibratable radiometric camera chain

PBR jointly models geometry, BRDF, illumination, transport, exposure, and color management. Randomize meaningful physical parameters and preserve the boundary between scene-linear and display outputs.

Rendering equation · surface point x

L_o(\mathbf x,\omega_o)=L_e(\mathbf x,\omega_o)+\int_{\Omega^+} f_r(\mathbf x,\omega_i,\omega_o)\,L_i(\mathbf x,\omega_i)\,(\mathbf n\!\cdot\!\omega_i)\,\mathrm d\omega_i

Material randomization changes BRDF f_r; HDRIs/lights change incoming radiance L_i; geometry/normals determine visibility and cosine terms. These effects exceed a background image and RGB multiplier.

Finite exposure

I(\mathbf u)=\frac{1}{\Delta t}\int_{t_0}^{t_0+\Delta t}\mathcal R\!\left(L\!\left(\mathbf u,t;\,T_C(t),T_O(t)\right)\right)\mathrm dt

With camera/object motion, pixels integrate trajectories over exposure. Rolling shutter varies t₀ by image row, making one midpoint ᶜTₒ a defined reference pose.

Variable family	Sampled factors	Required invariants	Typical failure
Geometry	mesh detail, normal / displacement, support, background geometry	object frame, metric size, collision/visual correspondence	Realistic materials cannot fix systematic silhouette/contact errors from mismatched CAD
Material / BRDF	base color, roughness, metallic, specular, transmission, IOR, texture	Material ranges and combinations have physical/product evidence	Unbounded colors break identity; mirrors/transparency need optical-path changes beyond color
Illumination	HDR environment, area/point light, intensity, CCT, size, shadow	Version scene-linear radiance, units/exposure, and color management	Brightness augmentation alone lacks shadows, interreflection, and saturation mechanisms
Camera response	exposure, white balance, tone mapping, vignetting, noise, compression	Separate clean linear renders from display-ready RGB	Training PNGs and real streams use different tone maps/ISPs while validation checks only visual color similarity
Temporal	shutter, trajectory, rolling readout, frame interval, light flicker	Pose, flow, blur, and timestamps share an exposure model	Blurred images with instantaneous poses; inconsistent dynamic occlusion and rolling-shutter geometry

Separate color and numerical data RGB can follow scene-linear → view transform/tone mapping → display encoding. Depth, normals, coordinates, IDs, and flow are Non-Color Data and should bypass white balance, gamma, and compressed color transforms. Preserve lossless float intermediates for auditing.

06 · Ground-truth Render Passes

Every channel needs frame, unit, visibility, and sampling definitions

Depth/mask output alone is insufficient. RGB, Z, normals, IDs, coordinates, and flow at one pixel must share scene state and visibility tests, with explicit antialiasing, transparency, motion, and background rules.

Pass	Precise semantics	Implementation notes
RGB / radiance	Display RGB or scene-linear HDR; specify color space, view transform, exposure, and alpha	Do not apply color management to normals, IDs, or depth
Z-depth / range	Optical-frame Z or ray length, with explicit units, clipping, and background-invalid semantics	Render clean floats, then derive sensor quantization and holes
Surface normal	Camera / world / object frames; direction, handedness, normalization	Store normal-map perturbations separately from geometric normals if needed
Instance / semantic ID	Stable integer pixel IDs with fixed background/ignore values	Color previews are not labels; antialiasing creates nonexistent mixed IDs
Full / visible mask	Separate amodal full silhouettes from z-buffer-visible regions	Define mask semantics for transparency, translucency, and motion blur by task
Object coordinate	Object-frame XYZ / NOCS at visible pixels, with fixed units and canonical frames	Use perspective-correct interpolation and bypass display color transforms
Optical flow / motion vector	Pixel displacement direction, time interval, occlusion/bounds validity, rolling-shutter conventions	Postprocessed vector blur differs from actual exposure integration
Pose / camera	Default ᶜTₒ, per-frame K/D, timestamps, velocities/trajectories; name inverses separately	Both camera and object motion affect relative pose

Pose and per-pixel object coordinate

{}^C\mathbf P(u,v) = {}^C R_O\,{}^O\mathbf P(u,v)+{}^C\mathbf t_O,\qquad z(u,v)=\left[{}^C\mathbf P(u,v)\right]_z

Object-coordinate passes give visible object-frame surface points; clean Z is the same point's third optical-camera component. Transforming XYZₒ by ᶜTₒ and projecting should recover its pixel and clean depth, providing a strong automatic check.

Pixel oracle

XYZ → pose → pixel

Sample visible pixels, transform coordinates with pose/K, and bound reprojection error by the raster convention.

Visibility oracle

IDs, Z, and masks share provenance

Visible masks equal the instance's z-buffer-winning pixels; render or compute full masks separately.

Frame oracle

Normals agree with pose

Rotate object normals with R into camera coordinates and compare normal-C, unit length, and orientation conventions.

Temporal oracle

Close the flow–trajectory loop

Advance surface points from t to t+Δt, project, and compare flow with explicit occlusion/out-of-bounds validity.

07 · Sensor Simulation

Real noise is often structured failure rather than independent Gaussian jitter

Noise varies with distance, incidence, brightness, material, and motion. Holes, flying pixels, multipath, saturation, and rolling shutter have spatial/temporal structure. Estimate mechanisms from raw real streams and implement separate, switchable sensor layers.

Modality	Formation mechanism to model	Calibration evidence	Common mistakes
RGB / ISP	shot/read noise, exposure, white balance, vignetting, bloom, demosaic, tone map, compression	raw / linear capture, flat-field, dark frames, color chart, MTF / blur target	Homoscedastic Gaussian noise in sRGB ignores saturation, photon statistics, and ISP order
Passive stereo	correspondence ambiguity, disparity quantization, left-right occlusion, baseline/calibration error	Real disparity residuals stratified by texture, edges, distance, and exposure	White noise on perfect Z does not reproduce holes and occlusion bands
Structured / active stereo	Projector patterns, IR response, ambient light, speckle, stereo occlusion, device interference	IR left/right streams, confidence, invalid patterns, multirange calibration boards	Random Z dropout without simulating the projector
ToF	phase noise, amplitude, multipath, mixed pixel, wrap-around, temperature	Amplitude/residuals across materials, incidence angles, distances, and multipath scenes	One σ(z) cannot capture flying edges, reflectance, and multipath bias
Rolling shutter	Rowwise exposure times, continuous trajectories, flow, occlusion changes	Known trajectories, flashes/turntables, or high-frame-rate reference cameras	One whole-frame warp with midpoint global-shutter labels
Event / LiDAR / tactile	Threshold/refractory events, scan timing/beam patterns, contact deformation/noise	Raw timestamp/packet/contact calibration alongside final rasters	Random filters on static images omit measurement formation

CLEAN

Geometric/radiometric truth

linear RGB, Z / range, normal, ID, pose, trajectory

+

OPTICS

Lens and shutter

distortion, PSF / DoF, vignetting, rolling shutter, blur

+

MEASUREMENT

Modality mechanism

stereo / ToF / projector / event / scan / contact process

+

READOUT

Readout and ISP

shot/read, quantization, demosaic, tone map, compression

Keep real test data out of sensor-model tuning Separate sensor-calibration subsets, model-selection validation, and untouched tests. Fitting sensor residuals and evaluating downstream networks on the same frames turns sim-to-real findings into hidden test tuning.

08 · Interactive Lab

From sampling coverage to multichannel truth and release gates

Compare naive Euler, Haar SO(3), view-sphere, and stable-pose priors; change PBR/sensor layers; then inject four sim-to-real gaps to see why label defects must block production first.

object viewing direction 12 × 6 equal-area bins Vertical axis: z = sin(elevation), not uniform latitude

Sampler semantics

—

——

Equal-area coverage

occupied bins

—72 equal-area direction bins; compare equal sample counts

Distribution entropy

normalized H

—1 indicates uniform bin occupancy, not deployment agreement

Directional bias

‖mean direction‖

——

Task interpretation

—

——

Uniform Euler angles are not uniform rotations. Independent uniform yaw/pitch/roll creates polar direction bias. For Haar-uniform SO(3), sample unit quaternions directly. Tabletop/bin tasks instead need explicit priors validated against deployment statistics.

09 · Domain Randomization & Sim-to-real

DR, photorealistic simulation, and real fine-tuning can be combined

Improve physical consistency where measurable, randomize unknown task-irrelevant nuisances, and use limited real data for calibration/selection. Judge final performance on real tasks untouched by tuning.

01 · Domain Randomization

Broaden nuisance distributions beyond the real domain while keeping task cues stable

Suitable use: Hard-to-model real appearance, long-tail coverage, robustness stress tests
Risks: Unmotivated or excessive randomness breaks correlations and teaches renderer artifacts
Validation: Per-variable ablations with real holdouts; compare narrow/wide and correlated sampling

02 · Photorealistic / calibrated simulation

Improve physical consistency of assets, transport, camera response, and sensing

Suitable use: Measurable scenes, industrial twins, render-and-compare, sensor-residual research
Risks: Costly assets can still be wrong; attractive images hide distribution, label, and ISP gaps
Validation: Stratified pixel/feature/geometric residuals and real task metrics, rather than subjective screenshots

03 · Hybrid

PBR with trusted geometry/cameras, then constrained randomization of uncertain nuisances

Suitable use: Most production 6D pose pipelines, balancing labels, coverage, and interpretability
Risks: Unclear layer/variable ownership hinders ablation; version combinations multiply
Validation: Fix a base scene and enable material / light / sensor / compositing layers individually

04 · Real fine-tune / adaptation

Use limited deployment ground truth to calibrate features, scores, normalization, or sensor statistics

Suitable use: A small labelable real set and relatively stable deployment domain
Risks: Leakage between calibration and test data; adapting only common cases harms the long tail
Validation: Untouched scene/instance-grouped holdouts and learning curves over multiple seeds

05 · Image translation / generative augmentation

Vary appearance or extend long tails while preserving geometric labels

Suitable use: Background, weather, texture, and lighting changes; hard negatives
Risks: Geometry, edges, IDs, or poses no longer match generated images; generator bias
Validation: Cycle/geometry checks, mask-edge audits, human spot checks, task ablations

Minimum sim-to-real experiment matrix

Real-onlySynthetic-onlyMixedPretrain → real FT − material DR− lighting DR− physics− sensor layer wrong sampling controlclean-label oracle3+ train seedsuntouched real holdout

Inspect gap diagnostics alongside task metrics Compare image statistics, feature distances, pose/visibility strata, and robot success separately. A lower proxy with worse task performance is not the dominant causal gap; avoid optimizing an easy-to-view domain score alone.

10 · Toolchains

Tools execute; data contracts make results comparable

Blender, BlenderProc, Isaac Sim, Unity, and custom renderers have different strengths. Tool names do not define frames, units, passes, sensors, versions, or validation.

Blender + Python

General modeling, Cycles/EEVEE, compositing, animation, scripting

Flexible, but requires explicit coordinates, passes, versions, and large-scale scheduling contracts

BlenderProc

Modular scene sampling, physics, rendering, BOP loaders/writers, reproducible configuration

Suited to 6D pose / CV synthesis; audit module parameters and asset sources

Isaac Sim Replicator

USD scenes, PhysX, robots/sensors, randomization, annotators

Suited to closed-loop robotics and digital twins; track versions, USD layers, GPU determinism

Unity / custom real-time engines

Real-time interaction, game-engine assets, custom shaders, large-scale frame generation

Verify physics, cameras, color, and label passes against research contracts

Offline ray tracer

High-quality transport, controlled materials/cameras, reference renders

Slow but useful for calibration/oracles; does not inherently provide robot/sensor timing

Dataset compositor

Real backgrounds + rendered foregrounds, copy-paste, mixed reality

Low cost; contact shadows, occlusion, color, depth of field, and edges expose domain gaps

11 · Scalable Production

Establish sample IDs, atomic shards, and idempotency before scaling GPUs

Scale turns rare faults into thousands of bad labels. Reconstruct samples from fixed inputs, publish shards atomically after validation, and preserve sample identities through retries.

01

Plan

Fix schemas, stratified quotas, configuration IDs, and budgets; run all unit/visual checks on small samples first

02

Sample

Seed → scene graph; log proposals and rejection reasons to expose sampling bias

03

Simulate

Physical settling / trajectories; save state summaries, contacts, timeouts, penetration, determinism fingerprints

04

Render

Separate clean and sensor passes using one scene snapshot / exposure trajectory per frame

05

Validate

Check matrices, projections, IDs, masks, depth, finite values, ranges, distributions, licenses, and round trips

06

Shard

Write temporary shards → fully validate → atomically rename; manifests, checksums, counts, idempotent retries

07

Export

Internal schema → BOP / COCO / WebDataset, with explicit units, frames, depth_scale, channel dtypes

08

Benchmark

Fixed real holdouts, real-only / synth-only / mixed baselines, multiple seeds, layerwise ablations

Per-sample manifest

sample_id: sha256(config_id | scene_index)
generator: {git_commit, engine, version}
random: {root_seed, sampler_streams}
assets: [{asset_id, geometry_hash, material_hash}]
scene: {transforms, velocities, contacts}
camera: {frame, K, D, shutter, timestamp}
passes: [{name, dtype, frame, unit, checksum}]
sensor_model: {name, version, parameters}
qa: {status, checks, reject_reason}

BOP export boundary

internal scene state
  ├─ metric mesh + object frame
  ├─ camera optical K / D
  ├─ ᶜTₒ at reference time
  ├─ clean + sensor depth
  └─ full / visible masks
          ↓ explicit conversion
BOP scene_camera / scene_gt / scene_gt_info
  ├─ cam_R_m2c / cam_t_m2c
  ├─ cam_K / depth_scale
  ├─ px_count_all / valid / visib
  └─ mask / mask_visib

Export involves more than copying fieldsConvert meshes, translations, diameters, bboxes, depth_scale, and symmetry together; round-trip to compare projections and mask counts.

12 · Quality Assurance

Verify analytical truth, inspect samples, then judge real tasks

Known generating states allow stricter automatic oracles than real annotation. Visual spot checks and real holdouts become interpretable only after mathematical/schema checks pass.

Failure modes	Symptoms	Detection	Fix
Uniform Euler bias	Too many polar directions and too few near the equator	Equal-area z–azimuth histograms, SO(3) bins, task-conditional coverage	Use Haar quaternions, or explicitly name the task prior
Hidden rejection bias	Visibility/collision filtering leaves only easy examples from random samples	Save proposal/acceptance counts and conditional rejection heatmaps	Constrained sampling or importance weighting; include rejection in the distribution model
Physics explosion	Penetration, explosive motion, floating, extreme angular velocity, irreproducibility	Penetration/contact/energy/settling-timeout assertions	Fix units, collisions, time steps, solvers, initial overlap
Pretty but wrong asset	Photorealistic images with wrong silhouettes, sizes, holes, or frames	Real silhouettes/depth/caliper measurements and frame overlays	Repair asset contracts before material randomization
ID / mask antialiasing	Mixed colors or unknown IDs at instance edges	unique ID set, mask counts, z-buffer consistency	Integer passes / Cryptomatte with task-defined discretization
Depth semantics mismatch	Mixing Z-depth, ray range, clipping values, or units	Analytical plane/sphere tests and back-projection/reprojection	Semantic field names; separate clean floats and sensor integers
Motion-label mismatch	Blurred/rolling-shutter images paired with one instantaneous pose	Check fast edges, flow, and exposure-trajectory reprojection	Save trajectories / row times; define the evaluation reference instant
Synthetic leakage	Shared asset/seed/scene variants across train/test, or tuning on real test data	Deduplication and split audits by generator lineage	Split by asset/scene families; reserve real holdouts for the final protocol
More data, worse transfer	More synthetic data lowers real performance	Learning curves / stagewise oracles with fixed training budgets and sample weights	Find the dominant gap, narrow unhelpful randomness, then adjust the mixing curriculum
Non-reproducible shards	Reruns change content/IDs and break configuration traceability	Manifest, content-hash, sample-ID idempotency checks	Derive sample IDs from fixed inputs; use atomic shards and versioned RNGs

Release Gate · Before Generating One Million Images

Are task support, split claims, and real holdouts fixed?
Does every asset pass frame, unit, size, collision, material, license, and hash checks?
Are SO(3)/view/stable-pose measures and joint distributions visualized?
Are penetration, settling, timeouts, rejection, and cross-version replay logged?
Do RGB, Z/range, normals, IDs, masks, XYZ, flow, and poses close pixelwise loops?
Are clean truth and sensor effects separately saved, switchable, and ablatable?
Do asset, scene, seed, and generator-lineage splits prevent leakage?
Are shards atomic, checksummed, retryable, and reconstructable from manifests?
Are real-only, synth-only, mixed, fine-tuned, and ablated models compared under one protocol?
Is the real holdout excluded from asset, randomization-range, sensor, and model selection?

Synthetic Data, Domain Randomization, Rendering/Physics, and Format SourcesExpand BlenderProc, Domain Randomization, Falling Things, Blender, Isaac Sim, PhysX, and BOP sources