09 RGB / RGB-D Point clouds Multimodal fusion Robot perception

RGB, RGB-D & Multimodal Pipelines

Beyond an Extra
Depth Channel

RGB observes appearance and perspective; depth measures visible-surface geometry. They complement each other but have different optical centers, timing, and noise. A reliable multimodal system preserves each sensor's contract, chooses a fusion level, and defines verifiable fallback paths for misalignment, holes, and complete stream failure.

Open the Multimodal Lab Start with information limits

Information Boundary

List direct observations before discussing learned inference

Sensor measurements, task priors, and learned inference are distinct sources. RGB depth prediction uses training priors without adding an independent metric measurement. Depth provides surface points without automatically identifying instances, whole objects, or hidden shapes.

Input	Direct observations	metric scale	Common pose approaches	Main failures
RGB-only	Color, texture, silhouettes, semantics, occlusion order, and perspective bearings	One image has no inherent metric depth; known CAD / size supplies scale through PnP	2D keypoints / coordinates, templates, render-and-compare, direct pose	Low light, blur, textureless surfaces, domain shift, single-view ambiguity
Depth-only	Visible-surface z / range, metric points, local normals, and occlusion boundaries	Metric scale with correct depth scale, K, and sensor calibration	3D voting, PPF / feature, SVD / RANSAC, ICP / render depth	Transparent, mirrored, black/absorbing materials, flying pixels, holes, and no color identity
RGB-D	Registered appearance and visible-surface geometry	Valid depth supplies scale; RGB itself does not add metric scale	pixel–point fusion, 3D keypoint voting, RGB proposal + depth refine	Extrinsic/time misalignment, occlusion-edge ghosting, invalid depth treated as real
Point cloud only	Discrete 3D surfaces, density, normals, and neighborhood geometry	Depends on origin: RGB-D / LiDAR are usually metric; monocular predicted clouds may not be	Global registration, local ICP, point networks, geometric voting	Partial overlap, sparse/nonuniform density, normal noise, symmetry, weak identity
Stereo / multi-view	Cross-view correspondences, disparity, visibility changes, and temporal structure	Calibrated stereo baselines supply scale; pure monocular sequences retain Sim(3) gauge	E / PnP / BA, MVS, TSDF / surfel / neural fusion	Degenerate correspondences, pose drift, dynamic objects, and occlusion conflicts across frames

MEASUREMENT

Direct sensor output

RGB intensity, raw depth, disparity, events, force; carry frames, timestamps, units, and validity.

CALIBRATION

Convert values into geometry

K/D, depth scale, baseline, extrinsics, and time offsets connect raster values to rays and metric 3D.

INFERENCE

Model and task priors

Semantics, full shape, object identity, monocular depth, and grasp feasibility come from learning or assets, rather than raw measurements.

RGB-D input alone does not define the task contract. Specify depth source, range, scale, frame, resolution, alignment direction, synchronization, invalid codes, filtering, and completion provenance. Identical four-channel tensors can represent different physical data.

Depth Sensor Physics

Bad depth samples follow sensor-specific noise mechanisms

Passive/active stereo estimates disparity, structured light interprets patterns, ToF measures optical paths, and LiDAR samples sparse range. They share calibration/occlusion issues but respond differently to sunlight, materials, distance, motion, and interference; one filter cannot replace source-specific modeling.

Sensor	Measurement mechanism	Strengths	Typical noise / holes
Passive stereo	Left/right image disparity d	Works outdoors; known baseline yields metric depth	Depth error grows with Z²; weak/repeated texture and occlusions lack correspondences
Active stereo	IR patterns assist left/right matching	Easier disparity estimation on weak indoor texture	Strong sunlight can overwhelm patterns; active devices can interfere
Structured light	Deformation / correspondence of projected coded patterns	Dense geometry on nearby static objects	Limited by range, ambient light, material, and camera–projector occlusion
Time of Flight	Modulated-light phase / time of flight	Direct ranging, works with low texture, often high frame rates	Multipath, flying pixels, phase wrapping, ambient light, reflectance dependence
LiDAR / laser profile	Laser time-of-flight or triangulation samples	Stable range and scale over longer distances	Sparse sampling, scan timing, motion distortion, material, RGB extrinsics/synchronization
Predicted monocular depth	Relative or metric depth inferred from RGB by a network	No depth hardware; can provide priors and ranking cues	Not an independent sensor observation; scale, detail, and domain shift depend on training

Stereo depth and first-order uncertainty

Z=\frac{fB}{d},\qquad \sigma_Z\approx\left|\frac{\partial Z}{\partial d}\right|\sigma_d=\frac{Z^2}{fB}\sigma_d

f is in pixels, B and Z share length units, and d is rectified disparity. Error grows with Z²; distant depth becomes unstable as d approaches 0. Active stereo assists matching without changing triangulation geometry.

Depth Representation

Depth maps, range images, and point clouds can differ in definition as well as formula

Most pinhole RGB-D data store optical-z depth; some ToF, LiDAR, and renderers store Euclidean ray range. They agree at the principal point and differ off-axis. Read the data specification before back-projection, rather than inferring semantics from depth.png.

Z-depth deprojection · pinhole

z=d_{\rm raw}s_d,\qquad {}^{D}\mathbf P(u,v)=z\,\mathbf K_D^{-1}\!\begin{bmatrix}u\\v\\1\end{bmatrix}=\begin{bmatrix}(u-c_x)z/f_x\\(v-c_y)z/f_y\\z\end{bmatrix}

sd is depth_scale, with units such as m/raw-unit. For distorted inputs, obtain rays from the matching camera model; do not apply ideal K⁻¹ directly to distorted pixels.

Ray-range deprojection

{}^{D}\mathbf P(u,v)=r\,\frac{\mathbf K_D^{-1}[u,v,1]^\mathsf T}{\left\|\mathbf K_D^{-1}[u,v,1]^\mathsf T\right\|_2}

Range r is Euclidean distance from the camera center to the surface. Confusing it with Z-depth increasingly bends planar clouds away from the principal point; the inverse confusion also causes distortion.

DEPTH MAP

Regular raster

Compatible with image networks and projective lookup; adjacent pixels need not belong to one surface.

POINT CLOUD

metric 3D samples

Convenient for Kabsch / ICP / point networks; density varies with distance and viewpoint.

NORMAL

Local tangent-plane direction

Estimated from neighboring 3D differences or fitting; depth edges require discontinuity-aware neighborhoods.

TSDF / SURFEL

Multiframe surface state

Depends on camera pose, visibility, and fusion weights. Fusion reduces noise but can embed systematic bias in the map.

Depth normal from local 3D tangents

\mathbf n(u,v)\propto\left(\mathbf P_{u+1,v}-\mathbf P_{u-1,v}\right)\times\left(\mathbf P_{u,v+1}-\mathbf P_{u,v-1}\right)

Fit robustly in valid, depth-continuous neighborhoods and standardize normal orientation. Differencing across silhouettes creates false normals connecting foreground and background.

RGB–Depth Registration

Matching array coordinates represent the same surface only after calibrated registration

RGB and depth usually differ in optical centers, K, distortion, resolution, and exposure time. Back-project depth pixels, transform with RGB-from-Depth extrinsics, then project into the RGB raster. Resizing two images alone does not align geometry.

Depth pixel → RGB pixel

{}^{D}\mathbf P=\pi_D^{-1}(\mathbf u_D,z_D),\qquad {}^{R}\mathbf P={}^{R}\mathbf T_D\,{}^{D}\mathbf P,\qquad \mathbf u_R=\pi_R(\mathbf K_R,\mathbf D_R,{}^{R}\mathbf P)

R denotes the RGB camera and D the depth camera. Invert stored Depth-from-RGB transforms first. Every projection must also check ZR>0, bounds, distortion, z-buffer, and timestamps.

01

raw depth

decode scale, invalid, confidence

→

02

Depth ray

K_D / D_D Back-project

→

03

3D transform

^RT_D With consistent units

→

04

RGB raster

K_R / D_R Project

→

05

visibility

Z-Buffers, Holes, and Boundaries

MANY → ONE

Several depth points reach one RGB pixel

Keep the nearest positive-depth point in the RGB frame; averaging foreground and background creates nonexistent surfaces.

ONE → NONE

An RGB pixel has no depth sample

Resolution, parallax, and occlusion naturally create holes; validity should remain false.

TIME SHIFT

Correct spatial calibration can still produce ghosting

When the camera or object moves, t_RGB≠t_D projects old geometry onto a new silhouette.

Interactive Multimodal Lab

From information limits to coordinates and system fallback

The first view shows direct evidence and gaps by modality and challenging scene. The second computes Z-depth / ray-range back-projection, Depth-to-RGB extrinsics, and reprojection. The third uses RGB quality, depth validity, alignment, and synchronization to choose fusion, single-modality fallback, delayed fusion, or rejection.

Direct / strong observation Conditional / indirect Unavailable Structured teaching judgments, not dataset accuracy

Information contract

—

——

Metric scale

Is Scale Directly Observed?

——

Current challenge

—

——

Pose route

Preferred Geometric Interface

——

Modality names cannot replace data contracts. RGB-D requires both intrinsics, distortion, Depth-to-RGB extrinsics, timestamps, depth scale, Z-depth/range semantics, invalid values, confidence, and post-alignment occlusion rules. A fourth channel alone does not establish geometric ground truth.

Fusion Architecture

Fusion levels determine available information and error propagation

Early, middle, late, and cascade fusion depend on tasks and sensor quality. Define branch semantics, trace cross-modal correspondences, gate by quality, train behavior for stream failures, and calibrate final scores to task risk.

Level	Interface	Example	Strengths	Risks
Early fusion	Concatenate registered raster / voxel inputs	RGB + depth + validity channels	Simple and efficient; complementary cues enter shallow layers	Strong dependence on pixel registration, equal resolution, and complete modalities; convolutions may treat zero depth as a surface
Middle fusion	Projection, sampling, or cross-attention between separate 2D / 3D encoders	pixel feature ↔ point feature	Retains image/cloud inductive biases with multiscale local/global feature exchange	Complex mappings, memory use, visibility, density, and debugging
Late fusion	Pose hypotheses, likelihoods, scores, or covariances	Independent RGB-pose and depth-alignment verification	Clearer fault isolation and missing-modality handling; reuses mature single-modality systems	Less low-level correspondence and fine-detail complementarity; branch scores need calibration
Cascade	One modality proposes; another verifies / refines	RGB detection → depth ICP / render compare	Clear responsibilities, interpretability, and stagewise oracle tests	Downstream cannot recover upstream misses; refinement has a limited basin

Depth completion is inference, not recovery of raw measurements. Completion can support features and proposals, but preserve measured/completed provenance and validate independently. Pulling background depth onto a transparent foreground does not create a real front surface merely because it looks smooth.

Pose Estimation Routes

One RGB-D input supports at least four geometric pipelines

Beyond backbones, the representation determines the verifiable solver: pixel correspondences enter PnP, pointwise keypoints/coordinates enter 3D alignment, templates/rendering yield image/depth residuals, and multiframe data enter tracking, BA, or surface fusion.

2D → 3D

RGB correspondence + PnP

Color/texture localizes 2D keypoints or object coordinates; known CAD provides metric 3D. Depth can verify visible surfaces, initialize t_z , or support refinement.

pixel identity → PnP → render/depth check

3D → 3D

Depth point / keypoint alignment

Predict object centers, keypoints, or NOCS/metric coordinates in clouds, then recover pose through Kabsch / Umeyama / RANSAC / ICP.

point identity → rigid alignment

DENSE FUSION

Pointwise joint appearance–geometry features

Sample registered RGB features at matching 3D points to predict pointwise pose/keypoints/confidence; prevent boundary misalignment from contaminating features.

pixel ↔ point → vote / regress

RENDER & TRACK

RGB/depth residuals and multiframe fusion

From a coarse pose, render CAD color/silhouette/depth and refine with visibility-consistent observations. Video can also accumulate TSDF / surfels and optimize jointly.

hypothesis → visibility → robust residual

06PnPRGB 2D–3D correspondence 07Point Cloud RegistrationDepth 3D–3D alignment 08General pipelineproposal · verify · refine 05Multiview / SLAMtracking · BA · map fusion

Difficult Materials & Scenes

Difficult materials change measurement models, beyond increasing random noise

Transparent surfaces refract and transmit backgrounds; mirrors alter optical paths; black materials weaken returns; thin edges mix foreground/background. Recognize these structured failures, reduce weights, or switch modalities instead of uniformly interpolating all zero depth.

Scene	Observation	Reliable strategy
Transparent objects	RGB sees boundaries, refraction, and background; active depth may be missing, measure background, or suffer multipath	Preserve invalid masks; use RGB / multiple views / polarization / touch; do not fabricate front surfaces by interpolation
Mirrorlike metal	View-dependent highlights can destabilize both RGB correspondences and active depth	Multiple exposures/polarization, view planning, silhouettes/edges, touch, or validated CAD rendering
Black / light-absorbing materials	Underexposed RGB, weak active-IR returns, low ToF amplitude	Control exposure and lighting; read sensor confidence / amplitude; reject weak returns
Textureless white objects	Weak RGB and passive-stereo matching, but active depth may remain healthy	Use silhouettes, structured light / ToF, geometric keypoints, point features
Thin rods / edges	Pixel mixing, quantization, and stereo occlusion create flying points	Edge confidence, z-buffering, conservative erosion; prevent wrong depth from dominating ICP
Heavy occlusion / tight stacks	Little visible surface; depth sees only the front layer; instance boundaries touch	Amodal / CAD priors, multiple hypotheses, temporal/multiview cues, task-driven active sensing
Moving objects	Unsynchronized RGB/depth projects old geometry onto new silhouettes; rolling shutter adds rowwise timing differences	Hardware synchronization, short exposure, time compensation, dynamic models; log separate stream timestamps
Multiple active-IR devices	Projected patterns interfere; depth quality varies with device phase and viewpoint	Hardware synchronization/time multiplexing, randomized patterns, isolated-device tests, confidence gating

MEASURED

Valid measured depth

Use in metric geometry, weighted by sensor noise and task residuals.

→

LOW CONFIDENCE

Suspicious numerical values

Downweight and check neighborhood consistency; distinguish from invalid samples.

→

INVALID

No surface measurement

Preserve the mask; use RGB / shape priors if suitable, without constructing zero-meter points.

→

COMPLETED

Model completion

Separate provenance and uncertainty; use as a prior or evidence requiring verification.

Temporal & Additional Modalities

Time, events, IMU, and touch change observability and complementary failures

Multiple frames reveal occluded regions, reduce random depth noise, and track objects, while adding drift, dynamic conflicts, and identity lifecycles. Events improve fast temporal response, IMU constrains short-term motion, and touch provides local metric/mechanical evidence after contact. Each needs frames, synchronization, and noise models.

MULTI-VIEW RGB-D

Coverage and surface fusion

TSDF / surfel / neural maps integrate views; camera poses and dynamic masks determine correct placement.

EVENT CAMERA

Rapid brightness changes

Low latency and high dynamic range suit motion edges; static scenes generate no events, and events alone give no metric depth.

IMU / PROPRIOCEPTION

Short-term motion and robot state

Angular rates, specific force, joint states, and wheel speeds support tracking and scale, with bias, extrinsic, gravity, and timing models.

TACTILE / FORCE

Local contact shape and constraints

When hands occlude vision, touch can update in-hand pose, normals, and slip; contact frames, soft deformation, and robot-chain calibration are prerequisites.

Multimodality does not require concatenating every stream. Identify failures where a new modality supplies independent information before choosing sampling rates, states, and factors. Shared systematic bias can make extra channels confidently wrong.

Engineering & Validation

Start with one pixel's geometric round trip before testing fusion

Verify depth-pixel back-projection, the expected 3D point, extrinsic projection to the correct RGB pixel, and return to the original frame. Then add occlusion, noise, synchronization, filtering, completion, and networks. Final pose alone cannot isolate sensor, registration, and fusion errors.

01 · frame

Which optical frame owns depth pixels? After alignment, which camera ray does each depth value represent?

02 · units

How does raw integer × depth_scale produce m/mm? What do 0, NaN, and maximum values mean?

03 · intrinsics

Is K tied to resolution, crop/resize, and undistortion state? RGB and depth each need their own.

04 · extrinsics

Is the stored transform RGB-from-Depth or its inverse? What are its translation units, thermal drift, and recalibration time?

05 · timestamps

Hardware synchronization or software nearest-time matching? Are maximum offset, exposure center, and latency logged?

06 · validity

Are invalid, saturated, low-confidence, completed, and measured samples stored distinctly?

07 · visibility

How are forward-splat z-buffers, holes, occlusion edges, and out-of-bounds pixels handled?

08 · provenance

Can versions of raw depth, filtering, alignment, completion, and fused clouds be traced?

01
Check depth semantics with the principal point and a known plane
The principal point should back-project to [0,0,z]. For a bent known plane, first inspect Z/range, K, distortion, and scale.
02
Run an RGB–Depth round trip
Select textured corners with continuous depth: D pixel→3D→R pixel. Validate edge errors with an independent calibration target.
03
Test spatial error statically and timing error during motion
Static errors usually implicate K / D / extrinsics; motion-only ghosting usually implicates timestamps, rolling shutter, or caches.
04
Run RGB-only and depth-only oracles separately
Establish each branch's independent ceiling, failure regions, and correct top-K coverage before judging fusion gains.
05
Inject missing, stale, and misaligned modalities
Check fallback / rejection rather than greater confidence than healthy dual modalities; log decision reasons.
06
Evaluate by material, distance, and validity
Overall averages can hide catastrophic transparent, distant, edge-pixel, and low-depth-coverage subsets.

Recommended sample record

rgb:
  frame: rgb_optical
  stamp: 1730000000.016
  K_D_resolution: [640, 480]
depth:
  frame: depth_optical
  stamp: 1730000000.004
  encoding: uint16
  scale_m: 0.001
  semantic: z_depth
  invalid_value: 0
registration:
  T_rgb_depth: calibrated
  aligned: false
  max_time_offset_ms: 20
provenance:
  raw: true
  filtered: false
  completed: false

Official Sensor Documentation, Data Contracts, and RGB-D MethodsExpand primary sources on RGB–Depth registration, formats, dense fusion, and 6D pose