Deep Reading Notes: Event-based Motion & Appearance Fusion

1. Key assessment

• Paper: arXiv 2603.08264
• Core problem: event cameras avoid motion blur and have low latency but only report brightness changes. Fast motion can escape local template searches, while integrating optical-flow velocities alone accumulates drift.
• Core answer: decode the event stream into motion and speed-independent appearance cues, combine them in a propagation–correction architecture, and use UKF to reduce jitter from discrete template corrections.
• My assessment: a clear, interpretable event-only tracking baseline. Fast synthetic sequences and ablations provide the strongest evidence; absent real 6D GT and a complete online implementation, approximately 110 Hz is only a latency-budget estimate.

2. Publication and task boundaries

The preprint first page states acceptance for IEEE ICRA 2026 publication, but every footer identifies the ICRA 2026 neuromorphic field robotics workshop. This site records ICRA 2026 Workshop to avoid labeling it as a main-conference paper.

Given:

• A known object 3D mesh;
• A known initial 6D pose $P_0=\{t_0,R_0\}$;
• A calibrated event stream $E=\{x_i,y_i,t_i\}$;
• A static camera and usually stationary background.

The output is continuous pose in camera coordinates. The algorithm neither detects objects nor estimates an initial pose from scratch. Event information weakens when the object stops or moves together with the background; camera motion / dynamic backgrounds create additional events outside the experimental assumptions.

3. Dual-channel overview

The architecture addresses complementary failures:

• Propagation alone: unrestricted search-center motion, but small velocity biases integrate across frames;
• Local correction alone: avoids continually integrating the same error, but cannot recover once the true pose leaves its small perturbation neighborhood;
• Combined: propagation moves the local neighborhood with the object, and correction brings the propagated center back toward observations.

4. Motion branch

4.1 Spatiotemporal triplet optical flow

If a surface point successively triggers three events $e_i,e_j,e_k$ within a short interval, displacement and timing approximately follow constant velocity. Candidate flow is

$$F(x_i)=\frac{x_k-x_i}{t_k-t_i}.$$

A single triplet is vulnerable to background noise or incorrect association. The authors divide the image into small RoIs, collect candidate sets in each, and select flow through spatiotemporal registration that jointly explains multiple events:

$$F=\arg\min_{F\in M}\sum_{i=1}^{n}\min_{F_i^c\in M_i}|F-F_i^c|.$$

This is a robust representative vector from local event consistency, rather than dense learned optical flow.

4.2 From 2D flow to 6D velocity

Rigid velocity is written as

$$V_t=[v_t^o,\omega_t^o],$$

where $v_t^o$ is the linear velocity of an object-attached point instantaneously coincident with the camera origin, and $\omega_t^o$ is angular velocity. The Kalman motion model is

$$V_{t+1}=\rho V_t+w,\qquad \rho=0.5,$$

and the observation residual compares event flow $F(u,v)$ with the interaction-matrix projection of

$$\hat F(u,v)=J(u,v)V_t.$$

The image Jacobian requires point depth. Earlier work reads RGB-D depth; this method renders it from the known mesh and current pose, reducing online inputs to events only. This creates circular dependence: pose error corrupts rendered depth and recovered 6D velocity, making appearance correction essential.

4.3 Pose propagation

Integrate linear velocity into translation and angular velocity through quaternions to obtain $\hat P_{t+1}$. This predicts a search center under large motion but does not remove long-term bias.

5. Appearance branch

5.1 EROS

EROS is an image-sized state in $[0,1]$. An event at $(u,v)$ sets that pixel to 1 and multiplies neighboring old responses by decay factor $\lambda$. This converts asynchronous events into an edge-like, relatively speed-insensitive image representation:

• No fixed-exposure blur under fast motion;
• More consistent contour strength across slow / fast motion than direct accumulation;
• No absolute color or static texture, favoring rendered edges rather than RGB templates.

5.2 Local templates

Around the propagated pose, generate positive / negative perturbations along 6 degrees of freedom:

• Use the image Jacobian to set translation increments for approximately 1 pixel of image motion;
• Use $\pm0.5^\circ$ on each rotation axis;
• Include zero perturbation for 13 pose hypotheses.

Render each hypothesis, extract Sobel gradients, and widen edges using Difference of Gaussians for matching tolerance. The template with the highest EROS dot product provides $\Delta P_{t+1}$ to correct $\hat P_{t+1}$.

Image-plane steps make sense because the same 1 mm projects differently at different depths. However, 13 axial candidates do not search coupled rotation–translation changes and select only one discrete direction per update. Propagation errors beyond this neighborhood still cause failure.

5.3 UKF

UKF finally smooths corrected poses, handling nonlinear SE(3) dynamics and discrete template jumps. Table II(c) reports:

Setting	Translation RMSE	Rotation RMSE
Without UKF	2.43 cm	15.50°
With UKF	2.18 cm	14.57°

The improvement is modest; propagation plus correction delivers the main gains, rather than UKF being the central innovation.

6. Data and evaluation protocol

Synthetic sequences come from earlier work and include mustard bottle, potted meat can, cracker box, and tomato soup can:

• Events, RGB-D, and GT originally at 500 Hz;
• RGB frames averaged to 60 FPS to introduce exposure blur;
• Gaussian depth noise with standard deviation 25 mm;
• Regular and fast motion per object, totaling 8 sequences.

Real data synchronize a 1280×720 event camera with a RealSense D415. The D415 serves only RGB-D baselines, not this tracker. Objects move by hand, without real-sequence pose ground truth.

Metrics are position RMSE $e_p$ in cm, rotation RMSE $e_r$ in degrees, and temporal standard deviation $\sigma$.

7. Main synthetic results

Representative fast-motion results:

Object	FoundationPose	This method
mustard	1.49 cm / 12.69°	1.14 cm / 11.53°
potted	1.04 / 6.33	1.30 / 13.61
cracker	1.42 / 3.70	0.90 / 3.05
tomato	0.89 / 4.51	1.74 / 4.68

The supported conclusion is competitive or better performance on some fast sequences, rather than overall event-only superiority over RGB-D transformers. Regular motion has little frame blur, preserving richer texture / depth for FoundationPose. Event methods particularly lag in potted and tomato rotation.

Most sequences show clear gains over EDOPT, whose local templates can lose poses outside the perturbation region. Flow propagation here first relocates the search center.

8. Component ablations

Design	Translation RMSE	Rotation RMSE	Interpretation
Local correction only	14.26 cm	30.58°	True pose readily leaves the local template region
Velocity propagation only	9.28	20.04°	Integration drift
Both combined	2.18	14.57°	Effective complementarity
Velocity from adjacent pose differences	24.56	35.55°	Differentiates correction noise
Event-flow velocity	2.18	14.57°	More stable direct motion cues

Rotation RMSE remains 14.57°: maintaining a track differs from high accuracy. Translation gains are stronger than orientation gains.

9. Real experiments and latency

Real mustard / cracker results show mesh overlays only:

• FoundationPose on RGB frames;
• EDOPT and this method on short event accumulations;
• The authors observe retained alignment for this method and FoundationPose, with later EDOPT drift.

Different image types and absent GT prevent quantifying how closely performance matches.

Independent component timing:

• Optical flow + velocity KF: approximately 2 ms;
• Template correction: approximately 5 ms;
• Propagation + final UKF: below 2 ms.

These timings motivate an estimated online rate of approximately 110 Hz, but the Discussion explicitly states that no complete full-online pipeline exists yet. Report this as “estimated from component timings,” rather than measured end-to-end FPS.

10. Limitations and reproduction advice

1. Initial pose and mesh are required; future event-pose detection is suggested for initialization / relocalization.
2. Background motion, moving cameras, and independently moving objects add events that violate the static-background assumption.
3. A stopped object generates insufficient events, requiring state priors; stop–start behavior is not systematically evaluated.
4. The 13 templates cover only local axial discrete perturbations; larger errors, combined perturbations, and coarse-to-fine searches need testing.
5. Averaging 500 FPS frames for synthetic blur may not reproduce real rolling- / global-shutter exposure.
6. Real data need motion-capture 6D GT covering fast translation, out-of-plane rotation, low contrast, and varying illumination.
7. Separately report event rate, events per pose update, end-to-end latency, and hardware utilization.
8. No official repository was found when the source notes were compiled; reproduction requires integrating earlier event flow, EROS, rendering, and filters.

11. Methodological lesson

Event pose tracking benefits from different representations for different jobs: local spatiotemporal structure conveys motion, while a decaying memory image conveys contours. One representation serving both large-range prediction and precise alignment often trades capture range against drift. Propagation–correction assigns each cue its stronger role and provides clear interfaces for learned event reconstruction or novel-object descriptors.