Paper Reading Notes (Deep): Industrial Plenoptic Dataset

1. Key assessment

Conventional benchmarks use 10%-diameter thresholds at 50–100 cm. Assembly may need less than 3 mm maximum deviation at 99% recall over 150–200 cm. Ground truth with 2–5 mm error cannot reliably distinguish submillimeter algorithms.

The solution has two parts:

• Co-evaluation data with shared scenes, 13 multimodal cameras, 4 exposures, and 3 lighting conditions;
• Robot Consistency estimates prediction error across robot poses, reducing dependence on manual GT precision.

IPD is therefore a vision-system design platform beyond single-model training.

2. Why ADD is insufficient

ADD averages distances over mesh vertices:

$$\operatorname{ADD}(T_p,T_g) =\frac1{|V|} \sum_{v\in V} \|T_pv-T_gv\|_2.$$

Averages depend on sampling and can hide worst-point deviation, crucial for assembly tolerances. The paper uses:

$$\operatorname{MVD}(T_p,T_g) =\max_{v\in V} \|T_pv-T_gv\|_2.$$

MVD upper-bounds ADD. Experimental true positives still use 10 mm + 10°, while the goal is 3 mm / 99% recall; current tables do not establish that final industrial requirement.

3. Robot Consistency derivation

Let camera, robot base, gripper, and object frames be $C$, $R$, $G$, $O$. Hand–eye transform $T_{CR}$ and robot pose $T_{RG}^i$ are known; the network predicts $T_{CO}^{i,\mathrm{pred}}$.

Map each prediction into gripper coordinates:

$$\hat T_{GO}^{i} =T_{GR}^{i}T_{RC}T_{CO}^{i,\mathrm{pred}}.$$

Rigid attachment makes all $\hat T_{GO}^{i}$ estimates share one constant. Estimate its Fréchet mean:

$$T_{GO}^{\star} =\arg\min_{\mu\in SE(3)} \sum_{i=1}^{N} \rho^2(\mu,\hat T_{GO}^{i}),$$

Translation uses the sample mean and rotation quaternion averaging. Project $T_{GO}^{\star}$ back into each camera observation and compute MVD:

$$E_{\mathrm{rc}} =\frac1N\sum_i d\!\left( T_{CO}^{i,\mathrm{pred}}, T_{CR}T_{RG}^{i}T_{GO}^{\star} \right).$$

Estimating $T_{GO}^{\star}$ consumes degrees of freedom and creates finite-sample bias. Under independent, unbiased, and mild conditions, translation / rotation estimates are consistent at rate $N^{-1/2}$. Uniform spherical views can also cancel some camera-frame-correlated bias.

4. Synthetic validation

Synthetic data uses 20 physical-scene analogues, 10 instances per scene, 30 viewpoints, and 4 cameras at 5MP. The authors train 24 keypoint models with varied backbones / convergence and compare:

• Ground truth;
• Robot Consistency;
• Manual-style labels with 1 / 2 / 3 mm MVD jitter.

Mean absolute gap to true error is 0.13 mm for Robot Consistency versus 0.47 mm for 1 mm jittered labels. Label noise increasingly overestimates error as models improve, harming precise-model ranking.

This is synthetic validation, not real IPD ground-truth accuracy of 0.13 mm. Robot / hand–eye calibration, thermal drift, and deformation add real error sources.

5. Capture system

Cameras

• 3× Basler-LR, 1920×1200 RGB;
• 5× Basler-HR, 2592×1944 RGB;
• 4× FLIR-MonoP, 2448×2048 grayscale + polarization;
• 1× Photoneo, 2064×1544 RGB-D, about 500 μm depth accuracy at 2 m.

Camera baselines span 10 cm–1 m. Kinematically calibrated UR5e relative accuracy is below 100 μm. Each scene uses 30 robot configurations, pitch / roll up to 30°, yaw over 360°, and workspace about 1 m × 1 m × 0.5 m.

Lighting and exposure

• 1 / 30 / 80 / 200 ms;
• Roomlight about 1,000–2,000 lux;
• Spotlight spans about 100 lux dark regions to 100,000 lux bright regions;
• Daylight varies with windows / weather.

Photoneo supplies 12-bit HDR; RGB / polarization retain multiple exposures for matched-scene fusion research.

6. Co-evaluation results

Camera × lighting

Camera	Daylight MVD / R	Roomlight MVD / R	Spotlight MVD / R
Basler-LR	5.361 / .66	4.580 / .69	4.898 / .60
Basler-HR	3.078 / .83	2.640 / .83	3.061 / .77
FLIR-MonoP	4.808 / .70	4.875 / .69	5.743 / .62
Photoneo	6.348 / .60	5.676 / .63	6.646 / .61

Photoneo is not always best: spotlight red / near-infrared interferes with structured light, while dark reflective parts may return little laser signal. High-resolution multiview RGB is most stable overall.

HDR

spotlight + Basler-HR:

Method	Median MVD	Recall	Precision
No HDR	3.271	.72	.960
Debevec	3.300	.74	.951
Robertson	3.351	.74	.952
Mertens	3.061	.77	.963

HDR is not universally better: Debevec / Robertson MVD is slightly worse than no HDR; only Mertens improves MVD, recall, and precision together.

Polarization

For dark reflective objects:

• Pipe fitting: no refinement 9.92, gray 9.66, AOLP/DOLP 8.43 mm;
• Oblong float: 6.17, 5.91, 5.41 mm.

Angle / degree of linear polarization mainly improves dark-background edge contrast, not direct pose output.

7. Hardest parts

Typical failures:

• Thin structures: incomplete depth / edge sampling;
• Small parts: too few pixels at 150–200 cm;
• Dark reflective parts: poor RGB dynamic range and structured-light return;
• Spotlight: over- and underexposure on the same object;
• Weak geometry: symmetry or insufficient keypoint discrimination.

Industrial errors should be stratified by sensor × material × geometry × lighting, beyond algorithm averages.

8. Experimental pose pipeline

Shared main-experiment components:

1. Mask R-CNN detection;
2. HRNet-style 2D keypoint heatmap;
3. Multiview RGB reprojection minimization / PnP;
4. Structured-light point registration / ICP;
5. Optional grayscale or AOLP/DOLP edge refinement;
6. NVISII / BlenderProc-style synthetic training.

Table 2 is controlled system ablation, not a latest-backbone contest. Researchers can replace algorithms while preserving cameras, scenes, and lighting.

9. Limitations and usage

• Robot Consistency cannot detect every shared bias; consistently shifted predictions can still be inaccurate.
• Independent / unbiased or specific camera-frame-bias assumptions may fail for correlated network errors.
• High-quality robot kinematics and hand–eye calibration remain essential.
• Semi-automatic annotations still group instance correspondences; the protocol is not annotation-free.
• Moderate clutter does not cover heavily occluded multi-instance bins.
• 20 parts and one workstation limit material / robot / factory diversity.
• Current 10 mm / 10° true positives remain above the 3 mm / 99% recall goal.
• The costly rig should be subsetted for single-camera deployment rather than copied wholesale.

10. Overall assessment

IPD’s main contribution is controlled experimental design across camera, baseline, resolution, depth, polarization, exposure fusion, and algorithms in one physical scene. Robot Consistency offers a practical route when GT is insufficiently precise.

It does not establish industrial 6DoF at 3 mm / 99% recall. Spotlight, thin, small, and dark reflective failures motivate system co-design and conditional error analysis beyond another global mean.