Deep Reading Notes: Shape From Shading for Robotic Manipulation

1. Basic information

• Short name: Active Shape from Shading / Controlled-Illumination Workspace
• Authors: Arkadeep Narayan Chaudhury, Leonid Keselman, Christopher G. Atkeson
• Affiliation: The Robotics Institute, Carnegie Mellon University
• Venue / pages: WACV 2024, pp. 8389–8398
• Official page: CVF Open Access
• Paper PDF: CVF PDF
• Public manuscript: arXiv:2304.11824
• Reading mode: Deep Dive
• Collection scope: extended inclusion. Active workspace shape sensing is the main task; pose / tracking is one of four downstream application groups.

2. Positioning in one sentence

Six low-angle lights and one overhead light turn a tabletop into a large photometric-stereo sensor. Two calibrated cameras observe static objects; illumination inversion recovers normals, shadows reveal depth discontinuities, and integration optionally produces local surfaces. Applications include suction grasping, thin-object detection, known-object deformation, CAD-based render-and-compare, and model-free relative planar tracking.

Three boundaries for pose readers

1. Known objects: complete 3D model pose may be optimized, but quantitative results report image-edge pixel mismatch without GT rotation / translation errors.
2. Unknown objects: relative registration between observations of one object, evaluating only planar $\Delta X,\Delta Y,\Delta\theta$ rather than absolute pose.
3. Sensing assumptions: main experiments use monochromatic matte-white Lambertian surfaces, sprayed when necessary; unconstrained natural-RGB perception is outside scope.

2.1 Guide to the paper’s figures and tables

Figure / table	Module	Correct interpretation
Figure 1	Hardware / representation	2 cameras + 7 lights, seven images→normals→surface; these define input conditions
Table 2	Sensing accuracy	Normal errors 13.31°–29.19°, mean surface errors 0.50–1.74 mm; accurate average surfaces do not imply accurate local normals
Figure 5	Pose / tracking	CAD-based multimodal comparisons; without CAD, register two integrated point clouds
Tables 4–5	Pose evidence	Known objects use pixels, unknown objects planar mm / degrees; do not combine them into unified 6D accuracy

3. Problem setting and task boundaries

3.1 Beyond a single pose benchmark

The paper asks: can controlled illumination provide fine normals and depth discontinuities cheaply enough to support multiple tabletop manipulation tasks?

Experimental levels include:

1. Sensing quality: compare normals with commercial depth sensors;
2. Metric accuracy: measure normal / integrated-surface errors against printed known meshes;
3. Grasping: arbitrary 3D surfaces and thin flat objects;
4. deformation: PVC card buckling;
5. pose: known-object CAD alignment;
6. tracking: unknown-object relative point-cloud registration;
7. generic-material extension: PIE-Net intrinsic decomposition + DiLiGenT.

Calling it solely a 6D pose estimator misses the main contribution and overstates the evaluation protocol.

3.2 Hardware and capture

Component	Role	Conditions
Camera C1	Robot-mounted machine vision	Controlled pose / intrinsics
Camera C2	Fixed second view	Calibrated to C1 for triangulation / frontal views
Lights L1:6	Six surrounding low-angle sources	Approximately parallel, calibrated directions; emphasize shading / occlusion edges
Light L7	Overhead illumination	Ordinary appearance / auxiliary localization
Matte-white surface	Uniform albedo, approximately Lambertian	Rust-Oleum flat-white paint when needed
Static scene	Sequential multi-light capture	No substantial object / camera motion during capture

3.3 Outputs

• Per-pixel normals $n_k$;
• shadow / occlusion / depth edge pixels;
• Optional integrated depth / point clouds;
• Grasp locations and approach normals;
• deforming known mesh parameters;
• known-object model alignment;
• unknown-object relative planar translation / rotation.

3.4 Models and visibility

• Known-object pose / deformation require 3D models / initial meshes;
• Model-free tracking needs adequate shared visible surface across observations;
• Integrated unseen surfaces are explicitly not metrically correct;
• Deformation comparisons assume near-complete visibility;
• Skewed views reduce shared effective pixels, motivating mostly frontal C1 / C2 views.

4. Research context

4.1 From passive perception to active illumination

Ordinary RGB-D treats lighting as environment; this system controls it as an action. Active perception can control:

• environment: illumination;
• extrinsics: camera position / pose;
• Intrinsics such as gain / focal length.

The paper chiefly controls the first variable, alongside fixed / robot-mounted cameras. It changes imaging physics to strengthen geometric pixel signals before inference.

4.2 Relationship to photometric stereo

Classical photometric stereo infers normals from intensity changes under fixed cameras and varying light directions. Contributions adapt this to robotics:

• Workspace-scale calibration of linear / quadratic illumination models;
• Six low-angle lights exposing local geometry / depth discontinuities;
• Explicit shadow handling;
• Normals / edges connected to manipulation and pose;
• Near-field normal-quality comparisons with commercial depth sensors.

4.3 Differences from learned 6D pose

Dimension	This system	Typical RGB / RGB-D pose network
Data	Seven actively illuminated captures	Natural single images or video
Features	Calibrated photometric normals + edges	Learned RGB / depth features
Appearance	Mainly white-painted Lambertian surfaces	Attempts natural textures / materials
known pose	analysis-by-synthesis optimization	direct / keypoint / correspondence + solver
Unknown pose	Two-time local-surface registration	Novel-object reference matching / reconstruction
Evaluation	Pixels or planar $x/y/\theta$	ADD(-S), BOP AR, R/t errors

This is active geometric sensing plus classical alignment, with different assumptions from FoundationPose / MegaPose baselines.

5. Research motivation

5.1 RGB-D challenges for small, thin, weakly textured objects

At tabletop scale:

• Stereo needs texture / projected patterns;
• ToF / structured-light resolution and edge noise can obscure millimeter structures;
• Temporal / spatial smoothing weakens curvature and sharp edges;
• Commercial depth sensors may miss a 3.15 mm-thick disc.

5.2 Why low-angle illumination helps

• Fine surface discontinuities cast visible small shadows;
• Depth boundaries cast direction-dependent occlusion shadows;
• Low-frequency curvature changes shading gradients;
• Multiple directions reveal edges hidden under individual lights.

Active illumination improves geometric signal-to-noise beyond increasing RGB spatial resolution alone.

5.3 Why restrict materials to matte white?

Intensity mixes geometry, reflectance, and illumination. Unknown colors / specularity complicate seven-image inversion. White matte coatings approximately fix reflectance, leaving geometric inversion under calibrated lights.

This enables the results while limiting generalization.

6. Method I: normals and depth edges

6.1 Linear Lambertian model

For pixel $k$ under light $i$:

$$I_i^k=\frac{\rho}{\pi}\langle \mathbf n_k,\mathbf l_i\rangle, \qquad i\in\{1,\ldots,6\}.$$

where:

• $I_i^k$: measured pixel intensity;
• $\rho$: known / uniform albedo;
• $\mathbf n_k$: unknown surface normal;
• $\mathbf l_i$: calibrated light direction / intensity vector.

6.2 Quadratic illumination model

Second-order spherical harmonics capture ambient / imperfect illumination:

$$I_i^k=\mathbf n_k^T\mathbf M_i\mathbf n_k.$$

$\mathbf M_i$ is the per-light quadratic model. Inference uses per-channel linear / quadratic calibrations, sequentially inverting normals under L1:6 and handling shadows. Full details are in the supplement.

6.3 Shadows and depth discontinuities

Low-angle lights create dark–bright transitions near protrusions / edges. Direction-dependent brightness helps distinguish:

• Color texture edges, with different locations / responses across lights;
• Geometric occlusion edges, with stable shadow cues along light directions.

The thin disc cannot be separated from table texture under overhead light alone. Low-angle shadows reveal its edges; C1 / C2 minimum-enclosing circles triangulate its center.

6.4 Normal integration

Normal integration supplies view-specific depth / points, scaled with intrinsics. Errors accumulate and unseen parts lack metric geometry. Meshing these points for CAD-based pose later performs worse than directly registering observed point clouds.

7. Method II: known-object pose

7.1 Pipeline

1. Estimate measured normals $N_R$, depth edges, and silhouette $M_R$;
2. Render candidate-pose normals $N_S$, edges, and silhouette $M_S$;
3. Match local normal patches for initial poses;
4. Optimize at multiple scales;
5. Combine normal, edge, and silhouette mismatches;
6. Retain / converge to poses best explaining observations.

7.2 Why multimodal costs tolerate errors

About 50% of Happy Buddha pixels exceed 20° measured-normal error, with high costs around undercuts. Boundaries and other patches still constrain pose. Modalities complement each other:

• Normals constrain local orientation;
• Silhouettes constrain overall scale / image alignment;
• Depth edges constrain discontinuities;
• Patch matches reduce poor local-optimization initializations.

The authors report convergence within 5.5 px for ten objects, within this controlled workspace and image-overlay protocol.

7.3 Evaluation protocol

• C1 field of view: 227 mm × 129 mm;
• Resolution: about 7 pixels/mm;
• Random positions / orientations per object;
• Overlay simulated and measured data after alignment;
• Manually measure edge pixel mismatch;
• Repeat 5 times per object;
• Report means / standard deviations.

No mocap / calibrated-turntable GT $R,t$ is supplied. Table 4 cannot yield rotation degrees, translation millimeters, or ADD(-S).

8. Method III: model-free relative tracking

8.1 Redefining the problem

Without CAD or a canonical object frame, classical absolute pose from one observation is ill-defined. Instead, estimate pose change between two observations of the same object.

8.2 Pipeline

1. Capture multi-light images at two times;
2. Estimate normals / depth edges;
3. Integrate view-dependent points using intrinsics;
4. Extract salient point features;
5. Register point clouds rigidly;
6. Recover relative $\Delta X,\Delta Y,\Delta\theta$.

8.3 Why this is not arbitrary 6DoF tracking

Table 5 has three error columns:

• $\Delta X$;
• $\Delta Y$;
• $\Delta\theta$.

No quantitative $\Delta Z$, roll, or pitch results appear. The text summarizes about 2° planar rotation and 4 mm translation, so these notes call it relative planar tracking. Registration could extend to 6DoF, but evidence does not establish these errors across all six freedoms.

8.4 View-dependent geometry

Integrated unseen surfaces are explicitly not metrically correct, such as knife belt-clip sides. With sufficient overlap, relative depth and scale support registration; new views / large rotations undermine this condition.

9. Surface-sensing experiments

9.1 Commercial depth-sensor comparisons

Objects / conditions:

• Flat and textured surfaces;
• Frontal optical-axis and 45° views;
• D455, D405, D435 stereo;
• L515, Kinect II ToF;
• Ours photometric stereo.

For depth sensors:

• Stereo: 50 frames per pixel, discard the top / bottom 10% and average; capture takes 1.5–2 s;
• Project visible / infrared patterns on textureless surfaces as recommended;
• ToF: 3×3 median filter + temporal trimmed mean;
• Compute normals from spatial depth derivatives.

Table 1 normal-distribution dissimilarity (lower better):

Sensor	Flat 0°	Flat 45°	Texture 0°	Texture 45°
D455	0.12	0.12	0.27	0.12
D405	0.09	0.04	0.21	0.15
D435	0.09	0.23	0.22	0.22
L515	0.06	0.05	0.20	0.15
Kinect II	0.24	0.14	0.19	0.16
Ours	0.02	0.01	0.18	0.10

Comparison limits: active sensing uses a dedicated near-field workspace, material priors, and multiple captures; most commercial devices target room scale. Results support superior normals in this setting, rather than universal sensor superiority.

9.2 Printed-object metric accuracy

Object	Normal mean(std)	Pixels <20°	Surface mean(std)	Surface max
Pyramid	13.31° (12.89°)	93.06%	1.31 (0.93) mm	4.17 mm
Star	18.85° (14.04°)	86.73%	0.80 (0.60) mm	3.65 mm
Bent Cylinder	18.81° (15.08°)	86.19%	0.50 (0.36) mm	2.03 mm
Octagon I	16.23° (12.53°)	83.45%	0.87 (0.85) mm	4.50 mm
Octagon II	17.17° (12.09°)	82.52%	1.20 (0.90) mm	5.05 mm
Spot	17.38° (10.42°)	83.55%	0.54 (0.48) mm	2.82 mm
Bas-relief	18.24° (10.01°)	76.24%	0.87 (0.94) mm	4.26 mm
Bunny	20.07° (14.80°)	75.88%	1.60 (1.27) mm	8.12 mm
Textured Pyramid	19.19° (17.52°)	61.75%	1.74 (1.31) mm	7.30 mm
Happy Buddha	29.19° (20.52°)	54.29%	1.54 (1.22) mm	7.66 mm

Undercuts, self-shadows, and textures worsen local normals. Pose needs robust multipatch / multimodal fitting rather than treating every normal as precise.

10. Manipulation and deformation

10.1 Suction grasping

For ordinary 3D objects:

• Find locally flat regions large enough for the cup;
• Exclude occlusion edges;
• C1 / C2 triangulate world point;
• Orient the gripper using local mean normals;
• Approach and grasp along minimum-jerk trajectories.

For thin flat objects:

• Detect edges with low-angle shadows;
• Fit enclosing circles across two views;
• triangulate center;
• Approach along the table normal.

Experiments:

• 35 grasps on 3D objects;
• 10 grasps on flat textured objects: 5 disc, 5 irregular octagon;
• Four failures, all textured-pyramid cases;
• Causes: suction attachment or triangulation-induced collisions;
• Compared commercial depth sensors did not reliably detect the 3.15 mm-thick disc.

Totals imply 41/45 successes, but the authors do not present this as a standard success-rate benchmark or statistically compare grasping pipelines.

10.2 PVC card deformation

• card: 0.76 mm thick, 86×54 mm;
• Task: infer vertical-load buckling from surface-normal changes;
• Analysis by synthesis: deformed meshes explain observed normals;
• theoretical curve: slope 1.78, offset 0.114, $R^2=0.975$;
• Repeatability: 5 cards, varied C1 / C2 positions, 23 points; slope 1.9, offset 0.055, $R^2=0.966$.

Only vision is used, without force / torque inputs. Near-complete two-view visibility is required. Initial-mesh analysis by synthesis is more reliable than direct surface reconstruction because normal integration depends strongly on viewpoints.

11. Quantitative pose / tracking

11.1 Known-object edge alignment

Object	Bounding box	Pixel error mean(std)
Pyramid	100×100×55 mm	2.50 (1.70) px
Star	110×110×15 mm	3.62 (1.03) px
Bent Cylinder	116×48×13 mm	5.50 (2.12) px
Octagon I	100×100×29 mm	3.35 (0.88) px
Octagon II	100×100×29 mm	2.85 (1.83) px
Spot	52×15×53 mm	1.50 (0.23) px
Bas-relief	83×110×7 mm	0.80 (0.20) px
Bunny	93×94×43 mm	4.06 (1.19) px
Textured Pyramid	100×100×55 mm	4.02 (0.96) px
Happy Buddha	115×45×37 mm	3.10 (0.69) px

At roughly 7 px/mm, 0.8–5.5 px corresponds approximately to 0.11–0.79 mm of projected edge mismatch, not object-frame translation. Projection mixes depth, rotation, translation, intrinsics, and edge choices; dividing by 7 does not establish 6D translation accuracy.

11.2 Unknown-object planar tracking

Object	Bounding box	$\Delta X$	$\Delta Y$	$\Delta\theta$
Knife	172×30×20 mm	1.02 mm	1.66 mm	0.36°
Monkey	46×64×8 mm	1.67 mm	0.97 mm	0.22°
Circuit	13×21×33 mm	3.28 mm	2.40 mm	1.28°
IO shield	120×39×21 mm	1.48 mm	1.86 mm	0.25°

The text’s approximate 4 mm translation / 2° planar uncertainty is more conservative than table maxima. Four objects have six repetitions each, without standard deviations, confidence intervals, or breakdowns for large rotations, low overlap, depth changes, and occlusion.

12. Generic-object extension

The main system assumes matte-white Lambertian objects. The extension:

1. Uses pretrained PIE-Net for intrinsic-image decomposition;
2. Splits natural RGB into reflectance and shading;
3. Applies global linear photometric stereo to shading images;
4. Selects 30–40 directions from DiLiGenT’s 95 lighting conditions;
5. Uses input resolution 256².

Qualitative findings:

• bear and buddha work relatively well;
• Glossy, dark, self-occluding harvest fails markedly;
• Shadows / specularities are not explicitly handled;
• Metric quality remains far from the main robot setup, as the authors acknowledge.

This is a possible extension, without eliminating material priors.

13. Robustness and failures

13.1 Shadow ambiguity

Low-angle shadows reveal depth edges but can break local normal inversion. Complex edges / undercuts exceed 20° local errors; the authors suggest non-colocated lights and cameras as one cause.

13.2 Specularity / color texture

• Main experiments remove texture through white paint;
• Sensor-comparison advantages shrink on textured surfaces;
• The generic extension fails on glossy dark harvest.

Natural reflective, transparent, or colored objects therefore remain unresolved.

13.3 Occlusion / visibility

• Deformation needs near-complete two-view visibility;
• Unseen integrated surfaces lack metric correctness;
• Model-free registration depends on temporal surface overlap;
• Skewed views reduce shared pixels and worsen deformation estimates.

13.4 Motion

Seven lights are captured sequentially. Object / arm motion mixes illumination with motion-induced intensity changes, violating classical photometric stereo. Dynamic capture FPS and motion compensation are unreported.

13.5 Grasp failures

Four textured-pyramid failures arise from:

• Unstable suction attachment;
• Triangulation errors causing collisions.

Good normals / edges alone cannot guarantee successful end-effector mechanics and calibration throughout the pipeline.

13.6 Representation failure

Meshing unknown-object integrated points and feeding them to the known-object pipeline performs worse. Normal estimation, integration, and meshing accumulate errors, and view-dependent meshes fail metrically from new views. This is a valuable negative result.

14. How well do experiments support contributions?

Authors’ claim	Evidence	Support	Assessment
High-quality normals / edges from controlled light	Tables 1, 2; Figure 2	Substantial	Effective near-field controlled sensing; complex-object local errors remain high
Better than commercial depth sensors	Table 1	Partial	Best on this near-field normal metric, with differing device ranges / conditions
Supports general suction grasping	45 trials, 4 failures	Partial	Real robots, but few objects and no baseline / confidence intervals
Measures known-object deformation	Figure 4, Table 3, $R^2$	Substantial	Reproduces buckling with mesh / visibility assumptions
Estimates known-object poses	Figure 5a, Table 4	Partial	Small overlays, without GT $R/t$ or pose baselines
Tracks unknown objects	Figure 5b, Table 5	Partial	Four-object planar tests, without full 6DoF / large-view validation
Extends to generic materials	Figure 6	Weak	Qualitative proof of concept; glossy dark failures, 30–40 lights

15. Critical analysis

15.1 Strengths

1. Improve physics-level observability: illumination design strengthens evidence before learned priors.
2. Full-system evidence: calibration / sensing connect to real grasps, poses, and deformation.
3. Explicit negatives: complex normals, skewed views, meshing degradation, and glossy-harvest failures.
4. Complementary geometry: normals, silhouettes, and depth edges support robust pose fitting.
5. Value for small thin objects: a setting not optimized by many room-scale RGB-D devices.

15.2 Weaknesses

1. Strong material priors: white paint limits production / household use.
2. Limited dynamics: sequential lighting needs static scenes.
3. Hardware complexity: seven lights, two cameras, workspace calibration, and ambient control.
4. Nonstandard pose protocols: no known-object $R/t$ GT; only planar unknown-object motion.
5. Mismatched comparisons: commercial sensors operate over different design ranges.
6. Supplement-dependent implementation: main text emphasizes results over algorithmic details.
7. Limited objects / trials: no significance tests, cross-day drift, or long-term reliability.

15.3 Implicit assumptions

• Uniform Lambertian albedo;
• lighting direction / intensity stable;
• Ambient illumination absorbed by the model;
• Static objects / cameras within lighting sequences;
• Accurate light / camera calibration;
• Sufficient shared visible surfaces;
• Integration errors small enough for registration;
• Edge overlays reflect useful downstream pose quality.

15.4 Practical applicability

Best suited to:

• Fixed workstations;
• Small opaque objects;
• Manufacturing, inspection, or experiments allowing coatings / material control;
• Temporarily static objects;
• Fine normals, thin-part detection, or suction orientation.

Poor fit for direct deployment:

• Fast conveyors / moving objects;
• Transparent, mirror-like, or dark natural objects;
• Changing outdoor lighting;
• Confined spaces without light-array room;
• Single-exposure immediate full-6D pose;
• Applications forbidding object / workspace modification.

16. Reproduction guide

16.1 Hardware

1. Build a rigid table frame with six roughly evenly distributed low-angle lights and one overhead source.
2. Calibrate C1 / C2 intrinsics, extrinsics, and robot hand–eye transforms.
3. Calibrate linear vectors $l_i$ and quadratic matrices $M_i$ per light / color channel.
4. Measure ambient light and fix exposure, gain, and white balance.
5. Verify normal inversion with matte-white calibration objects.
6. Synchronize lights / cameras and verify static sequences.

16.2 Minimal sensing reproduction

1. Capture six L1:6 images sequentially, plus overhead / ambient imagery.
2. Detect shadowed / saturated pixels.
3. Initialize with linear inversion, refine quadratically.
4. Integrate normals with integrability constraints into depth.
5. Align to known planes / printed meshes; report normal angles and surface distances.

16.3 Minimal known-object pose reproduction

1. Render CAD normal, silhouette, and depth-edge pyramids.
2. Compute observed / rendered normal-patch descriptors.
3. Build correspondence candidates and pose initializations.
4. Jointly optimize normal / edge / silhouette costs across scales.
5. Add calibrated-turntable / mocap rotation and translation errors, ADD(-S), or BOP metrics alongside pixel overlays.

16.4 Minimal model-free tracking reproduction

1. Capture one object under two known perturbations;
2. Use matched integration / scale for both normal maps;
3. feature matching + robust rigid registration;
4. Sweep overlap, rotation magnitude, out-of-plane motion, and occlusion;
5. Report full $SE(3)$ errors against ICP / RGB-D registration baselines.

16.5 Missing engineering measurements

• Total seven-image acquisition time;
• normal / edge / integration / pose optimization latency;
• light switching rate;
• Calibration time and cross-day drift;
• ambient-light sensitivity;
• texture / albedo / specularity sweep;
• object motion tolerance;
• Power, workspace dimensions, and occlusion risks.

17. Improvements

17.1 Multiplexed / coded illumination

Hadamard- or frequency-coded lights could replace seven sequential captures to:

• Reduce exposures;
• Improve SNR;
• Tolerate some motion;
• Combine with event cameras / fast global shutters;
• Estimate shadow confidence jointly.

17.2 RGB-D fusion

The authors also suggest combining active illumination with commercial depth:

• Depth supplies metric low-frequency shape;
• Photometric normals supply fine local geometry;
• Edges repair depth discontinuities;
• Uncertainty-weighted fusion limits integration drift.

This may be more practical than replacing depth sensors entirely.

17.3 Learned reflection / shadow models

• Generate supervision with calibrated lights;
• Predict normals, albedo, specularity, and shadow masks;
• Preserve integrability / photometric consistency through physical losses;
• Estimate uncertainty on colored / non-Lambertian objects;
• Fall back to depth / touch in failed regions.

17.4 Pose-aware sensor planning

Choose the next light / viewpoint from current pose hypotheses to maximize information:

• Reveal asymmetric patches under symmetric ambiguities;
• Increase shared visible surfaces under occlusion;
• Reward illumination policies through PnP / alignment covariance;
• Reduce redundant fixed seven-light capture.

17.5 Standardized pose evaluation

Add:

• known-object GT $R/t$;
• ADD(-S), MSSD / MSPD / VSD;
• full-6D unknown relative pose;
• large-rotation / low-overlap curves;
• baseline: ICP, RGB-D registration, FoundationPose / classical model alignment;
• Separate end-to-end robot success from perception-only error.

18. Research takeaways

Directly reusable ideas

• Improve observability before networks: lighting, views, and sensor parameters belong in pose-system design.
• Local normals + global silhouettes: complementary fine / global refinement constraints.
• Avoid unconditional meshing of noisy observations: repeated representation changes compound errors; task-appropriate intermediates matter.
• Match metrics to claims: pixel overlays support alignment, without establishing metric 6D accuracy.

When to cite this in related work

• active perception / active illumination;
• photometric stereo for manipulation;
• suction grasp surface-normal estimation;
• analysis-by-synthesis known-object pose;
• model-free relative object tracking;
• small / thin object sensing.

Potential research entry point

Build uncertainty-aware RGB-D + coded-light pose refinement: a foundation model supplies masks / coarse poses, then 2–3 informative lighting directions support predicted normals, shadow confidence, and depth-edge corrections. Symmetry-aware render-and-compare outputs pose covariance, evaluated with standard BOP and robot-grasp protocols.

19. Open questions

Conceptual questions

• How does 3D uncertainty vary with depth / object size for equal edge errors?
• When are low-angle shadows useful edges versus normal-inversion outliers?
• Do normal-patch descriptors yield multimodal matches on symmetric geometry?

Formula / implementation questions

• Combining linear / quadratic inversion and sequential solvers;
• Shadow thresholds and per-channel calibration;
• Integration boundaries and metric scale recovery;
• Multimodal weights, pyramid levels, and optimizers;
• Model-free point features and robust registration.

Experimental questions

• Seven-exposure timing and tolerated object speed;
• known-object full $R/t$ ground truth;
• unknown-object out-of-plane translation / rotation;
• Ambient-light changes and long-term drift;
• Errors versus natural albedo, roughness, and metallicity without paint;
• Tracking failure thresholds as C1 / C2 overlap falls.

Reproduction questions

• Completeness of public code / hardware bills;
• Light models, distances, angles, and intensities;
• two-camera synchronization;
• Alignment between painted CAD objects and GT meshes;
• Matched distances, FOVs, and resolutions in sensor comparisons.

20. Final assessment

Authors’ claim

Workspace-scale controlled lighting yields fine normals and depth discontinuities cheaply, outperforming several commercial depth sensors for small tabletop objects and supporting suction grasps, deformation, known-object pose, and model-free tracking.

My analysis

Evidence supports controlled matte-Lambertian small-object sensing and manipulation utility: mean surface errors 0.50–1.74 mm, ten known-object edge errors 0.80–5.50 px, four unknown-object planar table errors at most 3.28 mm / 2.40 mm / 1.28°, plus 45 real grasps and PVC deformation.

Pose claims require limits: known-object tables lack GT $R/t$, and unknown-object tests cover only $x/y/\theta$, not generic 4 mm / 2° full-6D accuracy. White paint, static seven-light capture, calibration, and high visibility remain prerequisites. The transferable lesson is design illumination / observation when passive images cannot reveal fine geometry, then use physically interpretable normals / edges for pose and manipulation.