Deep Reading Notes: Direct 3D Model-Based Object Tracking with Event Camera by Motion Interpolation

1. Basic information

• Authors / affiliations: Yufan Kang et al.; CNRS-AIST JRL, the University of Tokyo, Inria, and others
• Year / venue: 2024 / ICRA
• DOI: 10.1109/ICRA57147.2024.10611576
• Project: Project & Dataset
• Reading mode: Deep Dive

2. Core problem

Event cameras provide no conventional frames, so direct tracking must convert a window of events into a representation comparable with 3D model rendering. Fast motion carries the same 3D edge across several pixels within a window; predicting only from the endpoint pose creates a systematic shape mismatch. BIAM includes this trajectory through motion interpolation.

3. Conditions and outputs

Item	Paper setting
Sensor	Prophesee Gen3.1, 640×480; perspective or fisheye
Object representation	Known dense CAD / scanned mesh with renderable brightness
Initialization	Manually supplied first-frame pose
State	Current 6DoF pose and camera velocity
Background model	Not required; pose-conditioned ROI removes most background events

This is a frame-to-frame local tracker. There is no global redetection when the target leaves the ROI, initialization is wrong, or prolonged full occlusion occurs.

4. Method details

4.1 Why absolute brightness increments?

An event $e_k=(u_k,t_k,p_k)$ triggers when log-brightness change crosses a threshold, with $p_k\in\{-1,+1\}$. Contour polarity depends on the unknown background as well as the object: the same gray object has reversed edge polarity against white versus black. The authors therefore accumulate the following within the ROI:

$$\delta I^\*(u)=\left|\sum_{t_k\in T,u_k\in ROI}p_kC\delta(u-u_k)\right|.$$

Discarding the polarity sign reduces mismatch from changing backgrounds. The ROI pads the previous-pose rendered bounding box; the event count is 0.15 times the ROI pixel count, balancing signal-to-noise ratio against accumulation blur.

4.2 Predicting events from the 3D model

For rendered brightness gradient $\nabla I(u)$, small-motion brightness increments are approximated by

$$\delta I(u)\approx |-\nabla I(u)L_u\dot p\,\Delta t|,$$

where $L_u$ is the pixel sensitivity / interaction matrix. A unified camera model covers both perspective and omnidirectional lenses.

4.3 Motion interpolation

Conventional predictions use only the window endpoint, while real events arise along the full projected trajectory. BIAM divides inter-pose SE(3) motion into segments, uses intermediate positions along $u_m\rightarrow u_{m+1}$ to evaluate $\delta I(u_i)$, and averages them into $\delta I^+$. The objective is normalized SSD between observation and prediction:

$$(\widehat{\Delta p},\widehat{\dot p}) =\arg\min_{\Delta p,\dot p} \left\| \frac{\delta I^+}{\|\delta I^+\|_2} - \frac{\delta I^\*}{\|\delta I^\*\|_2} \right\|^2.$$

This models the imaging process of event accumulation rather than merely smoothing trajectories over time.

5. Data and results

The custom dataset contains a large industrial coil, noodle box, controller, bleach cleanser, and cracker box: 14 perspective and 15 omnidirectional sequences lasting 2–20 s. A UR10 or handheld setup moves the camera at approximately 0.1–2.0 m/s. Robot kinematics provide coil GT; other objects are primarily qualitative demonstrations.

• BIAM maintains tracking on all fisheye sequences; maximum sequence RMS position / rotation errors are 36.8 mm / 3.87°.
• On fast trajectories 6–8, BIA position RMS is 227–502 mm, reduced by BIAM to 12.9–28.0 mm.
• BIA fails on fast trajectories 9–12; BIAM achieves 18.7 mm / 1.17° on trajectory 9 and 16.3 mm / 1.59° on trajectory 10.
• Over 70% of sequences fail without absolute increments, demonstrating that polarity ambiguity is a substantial issue.

5.1 Interpreting speed figures

The paper reports event-window update frequencies of approximately 351–750 Hz, but its computational-cost discussion gives approximately 0.63 s optimization per event frame for the large coil. The former appears to describe event-data / window formation, while the latter is wall-clock solver cost in that implementation. This site does not interpret 750 Hz as real-time throughput of the complete CPU tracker.

6. Limitations and research value

• Only the coil is quantitatively evaluated; five-object generalization is mainly visualized.
• Manual initialization, accurate 3D appearance, and camera calibration are required.
• Absolute increments improve robustness but discard some discriminative polarity information.
• The ROI depends on the old pose, so drift can reinforce itself by excluding correct events.
• No detector / relocalizer is provided; the setting differs from modern learned event-pose methods.

The research value is an explicit optimization chain linking event formation, camera projection, and 3D alignment. Possible extensions include uncertainty-aware ROIs, GPU differentiable rendering, global event-feature relocalization, and learned residuals compensating differences between model brightness and real materials.