Paper Reading Notes (Deep): Visuo-Tactile Object Pose Estimation for a Multi-Finger Robot Hand with Low-Resolution In-Hand Tactile Sensing

1. Core Problem

Vision often sees an object clearly before grasping, then loses visibility behind fingers. High-resolution tactile images estimate contact geometry, but cost, wiring, and durability may be unsuitable for multi-finger hands. Can one bit per pad still improve 6D pose?

The key transformation is geometric: one bit contains little information, but with known pad positions, hand kinematics, and an object mesh, contact at that location defines a concrete feasible region.

2. Factor-Graph Formulation

At each time step, only the current object pose $T_t$ is optimized. An explanatory shorthand for the three factor types is (not a cross-time smoothing objective):

$$\min_{T_t} \rho_v(e_v(T_t)) +\rho_c(e_c(T_t,q_t,z_t)) +\rho_n(e_n(T_t,q_t)),$$

Here:

• $e_v$: FoundationPose / RGB-D visual-observation residual;
• $e_c$: contact residual between activated pads and object surface;
• $e_n$: object / finger-link penetration residual;
• State transitions use a uniform model; the previous pose initializes optimization without adding a temporal motion-smoothing residual.
• $q_t$: joint angles; $z_t$: binary contacts.

The Welsh robust cost saturates for large residuals, limiting false contacts or visual jumps so they do not dominate optimization.

3. Complementary Contact and Non-Penetration Constraints

Contact says only that the surface passes near a pad, leaving many feasible poses for one contact. Contacts from fingers in different directions intersect these feasible sets and progressively narrow pose.

Non-penetration also uses non-contacting hand geometry: candidate objects cannot pass through finger links. This negative evidence excludes geometrically impossible poses that satisfy only a few contacts.

Symmetric objects may retain rotational freedom; binary contact cannot create observability along inherently unobservable directions.

4. Simulation Protocol

• 17 YCB objects;
• Dvary: visual occlusion / noise varies over time, approximating visual degradation during grasping;
• Docc: stable power grasps with more complete contact coverage;
• Compare vision alone, different tactile / geometry factors, and full fusion;
• Evaluate with symmetry-tolerant pose metrics such as ADD-S.

Dvary tests whether touch can compensate for visual failures; Docc more closely tests geometric constraints under high contact coverage. These conditions should be distinguished.

5. Real System

• Custom 15-DoF multi-finger hand;
• Four fingers with multiple low-resolution tactile pads;
• RGB-D at about 30 Hz, touch / proprioception at about 50 Hz;
• Factor-graph output at about 13.3 Hz.

The 13.3 Hz rate reflects effective complete-optimization updates, more useful for engineering than visual-network forward time alone. It may be low for fast in-hand manipulation but supports slower post-grasp closed-loop correction.

6. Uncertainty and Weights

The paper uses fixed measurement covariance. This is simple, but sensor quality varies between stages:

• Vision is reliable before occlusion and degrades rapidly as the hand closes;
• Binary pads may flicker as contact area and pressure vary;
• Kinematic errors vary with joint configuration.

Future systems could learn or estimate factor covariances online, allowing graph optimization to adapt visual / tactile weights.

7. Limitations

• Requires object meshes, initial visual poses, and accurate hand geometry.
• Pad-position / threshold calibration errors can be interpreted as object-pose errors.
• Rigid-object and rigid-contact assumptions do not fit substantial deformation.
• Fixed covariances poorly represent time-varying reliability.
• Low-resolution contact improves observability but cannot fully replace external vision.

8. Deep Reading Conclusion

Low tactile resolution need not imply weak geometric information: combined with known hand kinematics, object meshes, and non-penetration, multiple binary contacts provide strong constraints. This offers a more transparent, debuggable tracking approach for low-cost multi-finger hands than end-to-end tactile regression.