Object Pose 6D Estimation 中文
← Back to the knowledge system
04 Imaging geometry 3D Vision RGB-D Data Engineering

Camera Models & Imaging Geometry

Camera Models and
Imaging geometry

Projection first transforms 3D points into camera coordinates, then applies perspective division, distortion, and intrinsics. RGB-D also requires depth scale, back-projection, and sensor-to-sensor extrinsics.

Open the projection lab Start with the full chain

Projection Pipeline

Four spaces connect object points to pixels

Each stage has different units and validity conditions. Unpacking homogeneous projection helps diagnose direction, scale, distortion, and visibility beyond memorizing a 3×4 camera matrix.

01 · OBJECT / WORLD

Op or Wp

3D points in CAD, map, or robot coordinates

m / mm
→ R, t
02 · CAMERA

Cp = [X,Y,Z]ᵀ

Camera optical frame; visible pinhole rays require Z > 0

m / mm
→ ÷ Z
03 · NORMALIZED

(x, y) = (X/Z, Y/Z)

Normalized image plane with focal length 1

dimensionless
→ D(·), K
04 · PIXEL

p = [u,v]ᵀ

Column and row coordinates in the image array

px
Full projective chain · distortion omitted
s[uv1]=K[CROCtO][XOYOZO1],s=ZCs\begin{bmatrix}u\\v\\1\end{bmatrix}=\mathbf K\begin{bmatrix}{}^{C}\mathbf R_{O}&{}^{C}\mathbf t_{O}\end{bmatrix}\begin{bmatrix}X_O\\Y_O\\Z_O\\1\end{bmatrix},\qquad s=Z_C
Using ^A T_B notation, extrinsics ^C T_O map object coordinates into the camera. Here s is homogeneous scale, equal to camera-coordinate ZC for pinhole projection. Distortion prevents representing the complete mapping with one linear 3×4 matrix.

Camera Intrinsics

K maps camera rays onto the pixel grid

Pixel-unit focal lengths incorporate physical focal length and pixel pitch. The principal point is the optical-axis intersection with the image plane, not necessarily the image center. Skew is often zero in modern cameras, but its formula position still matters.

Intrinsic matrix
K=[fxscx0fycy001],u=fxx+sy+cx,v=fyy+cy\mathbf K=\begin{bmatrix}f_x&s&c_x\\0&f_y&c_y\\0&0&1\end{bmatrix},\qquad u=f_xx+sy+c_x,\quad v=f_yy+c_y
fx, fy, cx, cy, and s use pixel units. Many implementations assume skew s=0, as does this lab. Pixel u usually points right and v down.
fx, fy

Focal length / pixel scale

Larger values narrow the field of view and produce larger pixel offsets for the same normalized coordinates.

cx, cy

Principal point

The pixel corresponding to normalized-coordinate origin; cropping and padding change it.

s

skew

Describes coupling from nonorthogonal pixel axes; usually approximately 0 in modern digital cameras.

W, H

Image size

Store K with its resolution; the same lens has different numerical intrinsics at different output sizes.

Nominal field of view
FOVx=2tan⁡−1 ⁣(W2fx),FOVy=2tan⁡−1 ⁣(H2fy)\mathrm{FOV}_x=2\tan^{-1}\!\left(\frac{W}{2f_x}\right),\qquad \mathrm{FOV}_y=2\tan^{-1}\!\left(\frac{H}{2f_y}\right)
This formula assumes a centered principal point and ideal pinhole boundaries. For offset principal points, compute left/right and upper/lower half-fields separately. Distortion models and cropping also affect effective FOV.

Interactive Lab

Connect projection, intrinsics updates, and back-projection

Three experiments share a 640×480 camera. Move a 3D point to explore perspective and distortion, simulate crop/resize updates to K, then recover camera coordinates from pixels and raw depth while comparing Z-depth and range.

Interactive · Camera Geometry

Pinhole Camera and RGB-D Geometry Lab

Trace projection through camera, normalized, distorted, and pixel coordinates, then practice updating intrinsics after image transforms and back-projecting depth.
OpenCV optical frame +x right · +y down · +z forward pixel = (u, v)
CAMERA SPACE IMAGE / PIXEL SPACE Oᵢ = (0, 0) 640 × 480 px Pᶜ = (X, Y, Z) → (x, y) = (X/Z, Y/Z) → (xd, yd) → p = (u, v)
Target ray Projected outline Principal point Mapped to 640×480; SVG is a teaching diagram

Coordinate chain

Projection Results at Each Stage

VISIBLE
Camera coordinates
Normalized coordinates
Distorted coordinates
Pixel coordinates

Intrinsic matrix

Camera Matrix K

3 × 3
[
]

Field of view

Field of View from Focal Length

640 × 480
Horizontal FOV
Vertical FOV
Pixel aspect ratio

Image dimensions are fixed here; shifting the principal point does not change the nominal FOV calculated from sensor width and height.

Lens Distortion

Distortion acts on normalized rays

Standard OpenCV models apply perspective division, radial/tangential distortion on the normalized plane, then K. Undistortion solves the inverse problem, typically through iteration or lookup tables.

Radial–tangential distortion · common form
r2=x2+y2,L(r)=1+k1r2+k2r4+k3r6xd=xL(r)+2p1xy+p2(r2+2x2)yd=yL(r)+p1(r2+2y2)+2p2xy\begin{aligned}r^2&=x^2+y^2,\quad L(r)=1+k_1r^2+k_2r^4+k_3r^6\\x_d&=xL(r)+2p_1xy+p_2(r^2+2x^2)\\y_d&=yL(r)+p_1(r^2+2y^2)+2p_2xy\end{aligned}
k coefficients model radial distortion; p1/p2 model tangential distortion from lens–sensor misalignment. OpenCV also supports rational, thin-prism, and tilted-sensor extensions. Check coefficient order in the specific API.
ModelCore projectionTypical parametersApplicationsCautions
Pinholex=X/Z, y=Y/Zfx, fy, cx, cy, skewConventional lenses, undistorted images, and basic geometryCannot represent strong wide-angle distortion
Radial–TangentialPinhole + radial/tangential distortionk1…k6, p1, p2; optional thin-prism/tiltStandard OpenCV models and industrial-camera calibrationExcess parameters can overfit or compensate for each other
Fisheye / EquidistantBased on incident angle θ rather than simply X/Zfx, fy, cx, cy + fisheye coefficientsFisheye, ultrawide, surround-view, and head-mounted camerasStandard pinhole/radtan coefficients do not transfer directly
Omnidirectional / UnifiedFirst maps to a unit sphere or unified image surfaceIntrinsics + mirror/unified parametersCatadioptric, very-wide-field, and omnidirectional camerasProjection, back-projection, and Jacobians depend on the model
More complexity does not guarantee accuracy. With insufficient viewpoint coverage, distortion parameters can compensate for focal length and principal point, producing low training reprojection error but poor new-image performance. Use the simplest model that explains the residuals.

Image-space Transform

Image augmentation changes camera geometry

Resize, crop, padding, flips, and rotations change pixel coordinates. For a 2D affine or projective mapping A, transform pixels and camera matrices together to avoid systematic PnP, coordinate-map, and projection-supervision errors.

Update intrinsics after an image transform
p′=Ap,K′=AK,Acrop+resize=[sx0−sxx00sy−syy0001]\mathbf p'=\mathbf A\mathbf p,\qquad \mathbf K'=\mathbf A\mathbf K,\qquad \mathbf A_{\mathrm{crop+resize}}=\begin{bmatrix}s_x&0&-s_xx_0\\0&s_y&-s_yy_0\\0&0&1\end{bmatrix}
The crop origin is (x0,y0), and sx, sy scale the cropped region to the output. If pixels are then normalized to [−1,1], additionally left-multiply by the pixel-normalization matrix.
RESIZE

Scale focal lengths and principal points together

Nonuniform resizing permits sx≠sy, so fx and fy are updated separately.

CROP

Subtract the crop origin from the principal point

A pure crop does not change focal lengths. The principal point may legitimately lie outside the new image.

PADDING

Add the padding offset to the principal point

Padding on the left and top increases cx and cy, respectively.

FLIP / ROTATE

Update label semantics as well

A reflection can change handedness; K′=AK alone does not make it an ordinary 3D rotation.

Depth & RGB-D

A depth pixel becomes a 3D point only with K, a scale, and a camera frame

RGB-D data often appear aligned, but raw RGB and depth usually have different optical centers, resolutions, and distortion models. The order of back-projection, extrinsic transformation, and reprojection must be explicit.

Back-project Z-depth
Cp=ZK−1[uv1]=[(u−cx)Z/fx(v−cy)Z/fyZ]{}^{C}\mathbf p=Z\mathbf K^{-1}\begin{bmatrix}u\\v\\1\end{bmatrix}=\begin{bmatrix}(u-c_x)Z/f_x\\(v-c_y)Z/f_y\\Z\end{bmatrix}
This equation assumes pixels that match K, with distortion already handled, and input depth that represents the camera z coordinate. If the sensor outputs range d along the ray, multiply a unit ray by d instead.
Depth camera to RGB camera
Crp=CrTCd Cdp,pr=π ⁣(Kr,Dr,Crp){}^{C_r}\mathbf p={}^{C_r}\mathbf T_{C_d}\,{}^{C_d}\mathbf p,\qquad \mathbf p_r=\pi\!\left(\mathbf K_r,\mathbf D_r,{}^{C_r}\mathbf p\right)
Back-project with the depth camera's Kd, transform in 3D using the RGB–Depth extrinsics, then project with the RGB camera's Kr and distortion Dr. Multiple depth points may reach the same RGB pixel, requiring a z-buffer or an interpolation strategy.
Z-DEPTH

Coordinate along the camera z axis

Pixels on the same plane z=constant have equal depth; their actual distance from the optical center increases farther from the principal point.

RANGE

Euclidean distance from the optical center to the point

Substituting it directly into the Z-depth equation places points near the image edges incorrectly.

DISPARITY

Stereo disparity is approximately inversely proportional to depth

Z = fB/d; zero disparity, subpixel precision, and baseline errors strongly affect distant depth estimates.

DEPTH SCALE

Stored integers are not physical units

In BOP, raw depth multiplied by depth_scale gives millimeters; other devices commonly use m, mm, or device units.

INVALID VALUE

0, NaN, and saturation do not represent surfaces

Reflective, transparent, and black surfaces, as well as occlusion boundaries, can produce holes or flying pixels.

ALIGNMENT

Alignment changes sampling and occlusion relationships

Depth-to-RGB and RGB-to-depth alignment require 3D extrinsics and reprojection, rather than a simple resize.

Depth sourceGeometric mechanismCommon strengthsTypical issues
Stereo / multi-viewDisparity + triangulationPassive sensing; generally suitable for outdoor useWeak or repetitive texture; increasing error at long range
Structured lightProject a known pattern and triangulateFine detail at close range; works with weak textureSunlight interference, reflective/transparent surfaces, limited working range
ToFModulated-light time of flight or phaseDirect ranging; relatively high frame ratesMultipath, flying pixels, temperature drift, phase ambiguity
Monocular predictionLearn an image-to-depth or inverse-depth mappingNo depth hardware required; dense coverageUncertain scale, generalization, and boundary geometry

Camera Calibration

Calibration means validating an imaging model, beyond obtaining a K file

Camera calibration jointly estimates intrinsics, distortion, and the extrinsics of each calibration image. Reliable results depend on model selection, view coverage, corner quality, focal stability, and independent validation, rather than a single overall RMS value.

  1. 01

    Choose the appropriate model

    Conventional, fisheye, and omnidirectional models cannot be interchanged merely by adding k coefficients. Oversimplification leaves systematic residuals; excessive complexity can destabilize estimation.

  2. 02

    Capture informative views

    Calibration boards should cover the center, corners, distances, and tilts. Translation near the image center alone cannot sufficiently constrain distortion and focal length.

  3. 03

    Estimate intrinsics and per-frame extrinsics

    Minimize corner reprojection error while estimating K, distortion, and each calibration-board pose relative to the camera.

  4. 04

    Inspect residual structure

    Low mean error does not prove a correct model. Examine residuals by image, position, and ray direction, especially systematic edge errors.

  5. 05

    Validate with independent data

    Use held-out images, straight lines, known lengths, or multiview reprojection. Record resolution, focus, and zoom settings.

Reprojection objective
min⁡K,D,{Ri,ti}∑i∑j∥pij−π ⁣(K,D,RiPj+ti)∥22\min_{\mathbf K,\mathbf D,\{\mathbf R_i,\mathbf t_i\}}\sum_i\sum_j\left\|\mathbf p_{ij}-\pi\!\left(\mathbf K,\mathbf D,\mathbf R_i\mathbf P_j+\mathbf t_i\right)\right\|_2^2
Pixel error reflects corner detection noise, calibration board flatness, and model mismatch. A low reprojection error only shows that the parameters explain these observations; it does not automatically guarantee correct focal length, 3D scale, or extrapolation near image edges.

6D Pose Connection

How camera models enter pose estimation systems

PNP

Intrinsics define 2D viewing rays

The same 2D–3D correspondences define different spatial rays under different K matrices, yielding different R and t. Unhandled distortion also enters the reprojection residual directly.

TRAINING

Keep labels synchronized with resize/crop

After letterboxing, dynamic cropping, or multiscale training, update K and the pixel transforms used by 2D keypoints, bounding boxes, and coordinate maps.

RENDER

Match synthetic cameras to the real ray distribution

Convert the renderer's FOV, principal point, axes, and pixel-center convention to the training data definition; distortion is usually simulated separately after rendering.

REFINE

Silhouette and depth refinement depend on the same model

Render-and-compare, ICP, and depth residuals require RGB, depth, CAD, K, and extrinsics to use consistent resolutions, units, and frames.

Debug Protocol

If projection is wrong, check the coordinate chain one stage at a time

First verify that a camera-space point produces the expected pixel, then add extrinsics, distortion, image augmentation, and depth alignment. Enabling every module at once can let multiple errors cancel each other out.

  1. 01

    Optical-axis test

    [0,0,Z] should project to (cx,cy), regardless of Z. Failure usually indicates an incorrect K layout, homogeneous division, or pixel axes.

  2. 02

    Unit-offset test

    [Z/fx,0,Z] should lie about 1 px to the right of the principal point; this directly checks fx, column vectors, and the +x direction.

  3. 03

    Disable distortion

    Set D=0 to verify pinhole projection before adding distortion; avoid adjusting K and D simultaneously while guessing the cause.

  4. 04

    Log every coordinate stage

    Output Pc, normalized, distorted, and pixel coordinates together, checking the transition from m/mm to dimensionless values and then px.

  5. 05

    Check the image transform A

    Apply crop/resize/pad to the four corners and principal point, and check that K′=AK produces exactly the same new pixels.

  6. 06

    Depth-plane test

    Back-project a known plane z=constant and check that the point cloud is flat; curvature often indicates an incorrect distortion model, K, or depth definition.

  7. 07

    Finally overlay the CAD silhouette and coordinate axes

    Axis directions, silhouette offsets, and scale errors have different visual patterns, helping distinguish extrinsic, K, and unit errors.

Conventions, equations, and official sourcesExpand camera model and data format documentation

Next geometry topic · 05

Multiview Geometry & Visual SLAM

With camera rays and imaging models established, the next step uses cross-frame correspondences to form E / F / H constraints, recover camera motion, triangulate scenes, and organize local observations into a SLAM graph with loop closures and object states.
Explore the full topic