Deep Reading Notes: Aleatoric Uncertainty from AI-based 6D Object Pose Predictors for Object-relative State Estimation

1. What problem does the paper address?

A typical 6D pose network outputs a single $\hat T_{CO}\in SE(3)$, whereas a robot state estimator also needs a measurement covariance $R$. Common engineering approaches compute one global constant from a validation set or rely on expert tuning. Yet the same model clearly has different reliability for a close, unoccluded, richly textured view and a distant, occluded, symmetric view. A fixed $R$ treats all these observations equally.

The paper pursues two objectives:

1. Predict a data-dependent aleatoric variance for every object and each 6D pose component in a single forward pass.
2. Show that these variances support more than plots: they can serve as dynamic measurement noise for object-relative visual-inertial state estimation and drive anchor selection and outlier rejection.

This setting is practical. It does not require converting the original pose predictor into a large sampling model or performing multiple forward passes for MC dropout, making it suitable for edge devices such as Jetson.

2. Network extension and rotation parameterization

The authors build on PoET. Scaled-YOLOv4 and a multiscale backbone first extract features, and a deformable transformer produces object queries carrying category and contextual information. The original model has translation and rotation heads. Two MLPs are added separately from these pose heads:

• Translation uncertainty head: Outputs $\hat\Sigma_t\in\mathbb R^{3\times3}$.
• Rotation uncertainty head: Outputs $\hat\Sigma_\vartheta\in\mathbb R^{3\times3}$.

Translation is handled directly in $\mathbb R^3$. Rotation-matrix elements should not be treated as independent Gaussian variables. The authors first orthogonalize the network’s continuous 6D rotation representation into $R\in SO(3)$ using Gram–Schmidt, then take the matrix logarithm:

$$\log(R)=\theta[v]_\times\in\mathfrak{so}(3),\qquad \vartheta=\theta v\in\mathbb R^3.$$

The error between ground-truth and predicted rotations can then be written as a local axis–angle vector. This is three-dimensional, continuous, and locally Euclidean, but remains a local representation: one Gaussian cannot fully cover rotations near $\pi$, symmetric objects, or multimodal posteriors.

3. What does the heteroscedastic loss learn?

Let $y$ denote a 3D translation or rotation vector. The pose head predicts a mean $\hat y$, and the uncertainty head predicts a diagonal covariance:

$$\hat\Sigma_y=\operatorname{diag}(\hat\sigma_{y,1}^2,\hat\sigma_{y,2}^2,\hat\sigma_{y,3}^2).$$

Assuming a Gaussian likelihood and taking its negative logarithm gives the loss for one object:

$$\mathcal L_y =\frac12\sum_{i=1}^{3} \left( \frac{e_i^2}{\hat\sigma_i^2} +\log\hat\sigma_i^2 \right), \quad e_i=y_i-\hat y_i.$$

In practice, predicting $s_i=\log\hat\sigma_i^2$ yields a more numerically stable form:

$$\mathcal L_y =\frac12\sum_i\left[\exp(-s_i)e_i^2+s_i\right].$$

The first term prevents variance from being too small when errors are large; the second prevents the network from inflating all variances without bound. Here, aleatoric means that, given the current image, the model tries to learn the typical residual size for similar observations in the training data.

The paper describes two training strategies:

• Calib: Freeze the original pose predictor and train only the uncertainty heads for approximately 10 epochs.
• E2E: Train means and variances together for approximately 50 epochs.

Their coverage is similar, supporting the feasibility of adding a confidence layer to an existing pose system. However, as $e\to0$, NLL drives $\sigma\to0$. The authors also observe steep gradients near zero error and error distributions that are not strictly zero-mean. This loss does not automatically guarantee calibration; independent reliability / coverage checks remain necessary.

4. From network coordinates to state-estimation covariance

The network separately predicts translation covariance and rotation tangent-space covariance in the camera frame. The authors combine them into:

$$\Sigma_{\text{pose}}= \begin{bmatrix} \Sigma_t&0\\ 0&\Sigma_\vartheta \end{bmatrix},$$

They then rotate / transform the covariance into the object-relative frame used by the MaRS EKF, according to the measurement direction and object–camera coordinate relationship. The IMU propagates the system state at 200 Hz, while RGB pose updates arrive at 15 Hz.

The paper compares three strategies:

1. Fixed: Set a fixed covariance from average validation-set errors.
2. AU: Directly use the predicted aleatoric covariance for every frame.
3. AU + Dynamic Anchor Switching: Use dynamic covariance and select the visible object with the lowest total predicted uncertainty as the current navigation anchor.

The YCB-V experiments also use AU + AOR: Reject observations when predicted uncertainty or the innovation relative to the estimated state indicates inconsistency.

A key engineering insight is that variance can drive discrete decisions as well as continuous filter weights: which object to trust now, and whether to use this frame. Such decisions suppress erroneous observations more effectively.

5. Calibration: Generally useful, with limits exposed by symmetric rotations

The paper checks each component using 95% prediction interval coverage probability (PICP). The ideal value is 0.95; substantially higher values indicate overly broad intervals, while substantially lower values indicate overconfidence.

For power-pole insulators, most PICP components fall within 0.93–1.00, indicating that intervals cover the errors but tend to be conservative. On synthetic YCB-V, most translation or rotation components for objects such as the cracker box, mustard bottle, and power drill behave well. Exceptions include:

• Mug: $\vartheta_1=0.703,\ \vartheta_3=0.228$.
• Scissors: $\vartheta_3=0.313$.
• Depth $z$ coverage of approximately 0.84–0.90 for several objects.

The paper attributes the mug / scissors failures to approximate symmetry and visibility. When a mug handle is hidden, several yaw angles are plausible; local scissor geometry can also create rotation ambiguity. A diagonal unimodal Gaussian compromises between such equivalent modes and cannot simultaneously provide an accurate mean and correct coverage.

On this YCB-V subset, PoET achieves mean AUC ADD-S of 78.1, mean translation error of 0.040 m, and mean rotation error of 36.06°. The latter two values show why high coverage does not imply accurate pose. Even an inaccurate predictor can attain high coverage by producing wide intervals; sharpness and task metrics must be reported alongside coverage.

The time series show a strong correlation between translation uncertainty along $z$ and camera–object distance, while rotation uncertainty grows for side views and occlusions. This matches the intuition that fewer informative image pixels imply greater observation noise. However, some error peaks lack a corresponding increase in uncertainty. The uncertainty is a statistical association, not an upper bound on individual prediction errors.

6. Interpreting the state-estimation results

Power-pole simulation (5 trajectories)

Strategy	Position RMSE	Orientation RMSE	Maximum position error
Fixed	0.140 m	2.41°	0.317 m
AU	0.141 m	2.44°	0.322 m
AU + dynamic anchor	0.126 m	2.18°	0.280 m

Real AIVIO flights (7 flights)

Strategy	Position RMSE	Orientation RMSE	Maximum position error
Fixed	0.172 m	2.21°	0.347 m
AU	0.179 m	2.21°	0.358 m
AU + dynamic anchor	0.146 m	2.01°	0.303 m

Synthetic YCB-V trajectories (5 trajectories)

Mean position / orientation RMSE is 0.425 m / 25.50° for Fixed and 0.311 m / 11.73° for AU + AOR. The improvement primarily comes from rejecting failed poses, rather than softly downweighting every failed observation.

These results do not support the strong claim that learned covariance is inherently better than fixed covariance. AU alone matches or slightly underperforms Fixed in the two power-pole experiments. A more accurate conclusion is that per-observation uncertainty provides a decision signal unavailable from fixed noise. It translates consistently into trajectory gains when the system uses that signal to switch anchors or reject anomalies.

7. Computation and deployment

On Jetson Orin AGX with TensorRT full precision:

• Original PoET: 106.73 ms.
• With uncertainty heads: 107.43 ms.

The approximately 0.70 ms / 0.6% overhead is small because the new modules are merely MLPs operating on object queries, without an ensemble or repeated sampling. This is a direct advantage over more complex Bayesian pose estimators.

8. Limitations and reproduction risks

1. Strong unimodal, diagonal assumptions. The six independent components ignore translation–rotation coupling. Symmetric objects require multimodal or symmetry-aware densities.
2. Only aleatoric uncertainty is modeled. The authors exclude epistemic uncertainty because synthetic data can be generated in large quantities. This does not establish the absence of model uncertainty in the real domain or under unfamiliar lighting or occlusion.
3. Coverage alone is insufficient. Overly wide intervals can yield high PICP. Interval width, NLL, ECE, or a proper scoring rule should also be reported.
4. State-estimation gains are confounded by strategy changes. The best rows also add anchor switching / AOR, so the entire improvement cannot be attributed to more accurate covariance values.
5. Class-specific heads. The current implementation targets known categories. Unknown objects or open-vocabulary detection require a generalizable uncertainty representation.
6. Local rotation modeling. A Lie-algebra Gaussian is natural for small errors but insufficient for large angles and topology-induced multimodality.

9. Implications for 6D pose research

The paper offers a useful evaluation framework: uncertainty methods should go beyond plotting distributions and answer questions at three levels:

• Statistical level: Are coverage / calibration correct?
• Estimation level: Does dynamic covariance improve filtering?
• Decision level: Can it change anchor selection, outlier rejection, or action planning?

The strongest evidence is at the third level. Replacing unimodal variance with a symmetry-aware mixture, Bingham / matrix-Fisher distribution, or pose hypotheses while retaining low single-pass overhead would be more valuable than simply enlarging the network.