Deep Reading Notes: 6-DoF Object Tracking with Event-based Optical Flow and Frames

1. Key assessment

Fast objects cause ordinary-camera blur and large inter-frame pose jumps. Event cameras offer frequent motion observations without waiting for full exposures, but lack stable appearance for global recognition of known-object 6D pose. The paper addresses this sensor mismatch.

The authors assign roles as follows:

• Events supply local velocity: per-event optical flow → 6D object velocity;
• RGB supplies global anchors: DOPE absolute 6D poses at 5 Hz;
• Depth connects 2D and 3D: the interaction matrix uses pixel depth to project 6D velocity into 2D flow;
• Two filters fuse over time: KF tracks velocity, quaternion UKF tracks pose.

The main novelty replaces the earlier 1-DoF event-generation observation with a more informative 2-DoF optical-flow residual inside classical state estimation, rather than directly predicting global pose from events.

2. System inputs and boundaries

Complete inputs are:

• RGB frames $\{I_i\}$;
• Synchronized depth images $\{D_i\}$;
• High-temporal-resolution events $\{E_q\}$;
• Calibrated intrinsics and event / RGB-D extrinsics;
• A known object and its DOPE estimator; visualizations also use an object mesh.

Foreground-event segmentation / object ROI generation is not fully specified as a separate module. Selecting target events directly affects velocity, especially with multiple objects, camera motion, or strong background edges. Controlled single-object trajectories do not cover these difficulties.

3. Per-event optical flow

Assume approximately constant motion over short intervals in a small image region. Three neighboring events

$$e_i=(x_i,t_i),\quad e_j=(x_j,t_j),\quad e_k=(x_k,t_k)$$

satisfying

$$\frac{x_j-x_i}{t_j-t_i} \approx \frac{x_k-x_j}{t_k-t_j+\xi},$$

provide flow for event $e_i$:

$$F(x_i)=\frac{x_k-x_i}{t_k-t_i}.$$

Each event can have several triplets, with wrong matches corrupting 6D velocity. Within small RoIs, select a representative from candidate set $M$, minimizing the summed nearest distance to each event’s candidate subset $M_i$:

$$F^\star = \arg\min_{F'\in M} \sum_{i=1}^{n} \min_{F_{ic}'\in M_i} \left\lVert F'-F_{ic}'\right\rVert.$$

This resembles robust medoid aggregation: select an existing vector consistent with the candidate sets rather than averaging conflicting flows. It is lightweight but assumes local constant velocity, sufficient event density, and correct RoIs.

4. Why 2-DoF observations suit object motion better

Earlier Hybrid Object Tracking with Events and Frames uses a 1-DoF brightness-change timing residual. Full-image camera motion can provide enough events, while local object motion with self-occlusion can lack observability during simultaneous multi-axis motion.

Measured flow $F(u,v)$ contains direction and magnitude. Ours (GE) is the earlier 1D generative-event version; Ours (OF) is generally more stable on fast sequences, supporting improved local object-motion observability from 2D measurements.

5. From 2D flow to 6D velocity

Define velocity state as

$$V_i=[v_{o_i},\omega_i],$$

where $v_o\in\mathbb{R}^3$ is linear velocity of an object-attached point instantaneously at the camera origin, and $\omega\in\mathbb{R}^3$ is angular velocity.

The motion model uses decaying velocity:

$$V_{i+1}=\alpha V_i+w,\qquad \alpha=0.5.$$

Events occur only during brightness changes; a stopped object provides no measurement corrections. Using $\alpha<1$ gradually brings predicted velocity to zero without events, rather than maintaining an incorrect constant velocity.

For pixel $(u,v)$, predicted flow is

$$\hat F(u,v)=J(u,v)V_i\Delta T,$$

where $J=[J_{v_o}\ J_\omega]$ is the image interaction matrix. Its translation block contains

$$\frac{f_x}{d},\quad \frac{f_y}{d},\quad \frac{u-c_x}{d},\quad \frac{v-c_y}{d}.$$

where $d$ is corresponding pixel depth. This is not depth-free: translation in 2D flow cannot directly provide metric 3D velocity without synchronized depth. The Discussion explicitly lists removal of RGB-D depth as future work.

6. Normal-flow constraints and measurement weighting

Events frequently arise at edges. With only one local edge, the aperture problem constrains normal flow while leaving tangential flow unobservable. Instead of directly using $F-\hat F$, project predicted flow onto the measured-flow direction before comparison:

$$r_k= \frac{F_k\cdot\hat F_k} {\lVert F_k\rVert\lVert\hat F_k\rVert} F_k-\hat F_k.$$

An incomplete normal-flow measurement can then constrain the prediction along its observable direction.

Residual weights depend on deviation from median $M$:

$$L_k= \max\left( \frac{1}{2b} \exp\left[-\frac{|r_k-M|}{b}\right], 10^{-6} \right).$$

Event flows far from the residual median contribute less to correction. Table III reports:

Version	Translation RMSE cm ↓	Rotation RMSE ° ↓
w/o normal-flow constraint	4.93	12.35
w/o measurement weighting	4.10	10.45
Full model	3.24	10.19

Normal-flow constraints matter more for both translation and rotation. Weighting clearly helps translation and slightly improves rotation. Grid size, event windows, Laplacian $b$, and noise-matrix sensitivities are not reported, but matter for reproduction.

7. KF and quaternion UKF

The first Kalman filter predicts with decaying velocity and corrects using accumulated event-flow residuals, outputting frequent $V_i$.

The second filter’s pose state is

$$x_i=[t_i,q_i]^T,$$

where $q_i$ is a unit quaternion. UKF integrates translation and rotation using $V_i$; an arriving DOPE pose $T_i$ corrects position $p(T_i)$ and quaternion $q(T_i)$. Without a current global pose, UKF skips correction and propagates using event velocities only.

DOPE is artificially limited to 5 Hz to simulate low-power / small GPUs. The paper mentions 30 Hz on high-end GPUs but does not report this system’s hardware, actual event throughput, filter frequency, or end-to-end energy at 5 Hz.

8. Data generation and metrics

Synthetic experiments use Unreal Engine and four YCB objects:

• 003 cracker box;
• 005 tomato soup can;
• 006 mustard bottle;
• 010 potted meat can.

Each has regular and faster random 6DoF trajectories. RGB is first rendered at 500 FPS and supplied to an event-camera model, then averaged over fixed windows into 60 FPS frames to introduce motion blur. The simulator supplies object-pose and 6D velocity GT.

Position uses Euclidean translation RMSE against GT; rotation uses se(3) rotation-vector RMSE in degrees. These are not BOP ADD(-S) / AR or symmetry-aware surface metrics.

Real data use calibrated RealSense D415 and event cameras, both 640×480. RealSense runs at 60 FPS; the event camera is asynchronous. Cracker box and mustard bottle move by hand as fast as possible. Without pose GT, evaluation checks CAD-render overlap visually.

9. Main results

Four-object mean results:

Method	Regular translation cm ↓	Regular rotation ° ↓	Faster translation cm ↓	Faster rotation ° ↓
se(3)-TrackNet	7.69	13.56	38.88	38.84
ROFT	1.97	9.15	22.27	36.60
Ours (OF)	2.53	8.82	3.94	11.56

Interpretation:

• Under regular motion, ROFT has lower mean translation error; this method slightly improves rotation;
• Under faster motion, both frame-based baselines degrade substantially, while this method changes only from 2.53 / 8.82 to 3.94 / 11.56;
• se(3)-TrackNet can be very accurate on individual regular sequences yet fail completely at high speed;
• ROFT shares 5 Hz DOPE with this method, helping isolate frame versus event optical-flow stability under blur.

10. Scope of real-world evidence

Real-sequence observations:

• se(3)-TrackNet fails during some fast translations / rotations and struggles to recover;
• ROFT can recover from periodic global poses but has clear overlay errors under blur;
• This method aligns CAD overlays more closely during translation;
• Rotation errors remain substantial, attributed to poor observability from sparse events.

Without GT, these support offline operation on real sensors and plausible visual trends, not centimeter / degree accuracy. Faster synthetic averages cannot be transferred directly to real robot scenes.

11. Real-time status and reproduction

The paper describes per-event flow as lightweight and scalable to real time and limits DOPE to 5 Hz, but experiments are explicitly offline. Real-time flow / filter operation is an expectation; live latency and stability remain future tests.

Reproduction requires checking at least:

• Event / RGB-D clock synchronization and spatial calibration;
• Depth registration to event pixels, holes, and noise handling;
• Object RoIs and foreground-event selection;
• Event-triplet search radius, timing tolerance $\xi$, and grid size;
• Events accumulated per KF correction;
• $\alpha=0.5$, $Q_v,Q_\omega,R_F$;
• Exact vector implementation of normal-flow residuals;
• Laplacian coefficient $b$;
• UKF $Q_t,Q_q,R_t,R_q$;
• DOPE 5 Hz scheduling and output delay;
• Domain gaps in simulated events, exposure averaging, and real contrast thresholds.

No official code / dataset repository was found when the notes were compiled, and not all parameters above are specified. Reproduction is harder than the classical filter architecture might suggest.

12. Limitations

• Synchronized RGB-D depth is required; the pipeline is not fully event-based;
• Low-rate absolute poses still require known-object DOPE;
• Synthetic quantities cover only four YCB objects under controlled scenes / motion;
• Real data cover two objects without GT;
• All experiments are offline, without end-to-end real-time measurements;
• Sparse events weaken rotation observability;
• Object-event segmentation / ROI generation lacks systematic analysis;
• Fast camera motion, multiple objects, occlusion, background events, and cross-sensor misalignment are not evaluated;
• No same-protocol comparison with learned event-flow / event-pose networks;
• Pose-vector RMSE does not handle object symmetry.

13. Overall assessment

The main lesson is that events need not perform full global recognition for fast tracking. Observable, robust 2D motion measurements plus low-rate frame-based poses can remove integration drift and expand the tracker’s speed range. Faster synthetic errors fall from ROFT’s 22.27 cm / 36.60° to 3.94 cm / 11.56°, providing clear evidence for this mechanism.

Fast event-assisted 6D tracking must be stated with three conditions: synchronized depth, 5 Hz known-object DOPE, and no quantitative real-world GT. This is a research prototype for asynchronous event / frame / depth fusion, rather than a comprehensively validated real-time event-pose system.