Paper Reading Notes (Deep): InvariantCloud

1. Key assessment

• Paper: arXiv 2605.25216
• Core problem: tactile contact patches are small, and markers shear / slip. Framewise LK / ICP mismatches accumulate drift, especially for yaw about the sensor normal.
• Answer: fix a dense, uniquely indexed marker-derived cloud in sensor coordinates. Contact changes active IDs and heights; direct ID matching solves roll / pitch, contact-contour PCA gives absolute yaw, and centroids give translation.
• My assessment: defining correspondence at initialization improves repeatability and weak-contour yaw. However, pose is local contact pose relative to the sensor; observability depends on contact geometry, continuity, and calibration.

2. Task boundaries

Not required:

• Object CAD;
• External RGB / depth cameras;
• Online motion capture;
• object-specific learning.

Required:

• A GelSight-like visual tactile sensor with stable intrinsic markers;
• Known-sphere press calibration;
• An undeformed global marker reference;
• Stable extractable contact contours and heights;
• Continuous contact during framewise tracking.

Outputs use the sensor frame, supporting local in-gripper tracking rather than direct robot-base / world pose. Manipulation also needs sensor–gripper kinematics.

3. Height maps and global reference clouds

3.1 Sphere-press calibration

A known-diameter sphere indents the elastomer at multiple locations. Per-pixel features include:

• RGB;
• Normalized pixel coordinates;
• Intrinsic-marker embeddings.

An MLP predicts local normal inclinations; DCT solves the Poisson equation to integrate gradients into height map $H$. Pixel $(x,y)$ maps using image center and pixel-to-mm scale $s$ to

$$P=((x-c_x)s,(c_y-y)s,h(x,y)s).$$

This determines Z translation and Kabsch-cloud accuracy. Objects need no training, but the sensor needs prior training / calibration: object-agnostic is not calibration-free.

3.2 Permanent IDs

Detect $N_{\mathrm{markers}}$ undeformed markers and sort rectangular rows / columns. Bilinear interpolation between markers creates a denser lattice; each grid location uses

$$g=r(n+1)+c+1$$

to obtain a globally unique ID. Experiments commonly use 19×25, or 475 points; cross-contact reconstruction uses 31×41.

The key is fixed identity rather than density alone. Re-covering the same sensor region restores the same ID set, so returning to the start does not inherit earlier matching errors.

4. Contact subset

Dual color-difference / height-depression thresholds followed by dilation / erosion produce contact mask $C_t$. Sampling it at integer lattice pixels selects active points, lifted into 3D using current heights.

The grid is sensor-fixed; object motion changes:

• Which IDs lie inside the contact silhouette;
• Their heights;
• Contact centroid and principal axis.

Markers may shear / slip on the elastomer. Unlike LK+ICP, local marker displacement is not the sole evidence for object yaw.

5. Decomposed 6D solving

5.1 Kabsch for roll / pitch

Common contact-cloud IDs give corresponding 3D sets $\{p_i\},\{q_i\}$. Center them and construct

$$H=\sum_i\tilde p_i\tilde q_i^\top=U\Sigma V^\top,$$

then apply standard Kabsch

$$R^\star=V\,\operatorname{diag}(1,1,\operatorname{sign}\det(VU^\top))\,U^\top$$

to solve the best rigid rotation.

This X / Y rotation estimate mainly uses contact height / normal changes for sensor-plane-relative roll / pitch. Yaw is separate because torsional elastomer slip breaks marker-based Z rotation registration.

5.2 PCA for yaw

For current contacting 2D global points

$$S_t=\{p_i\in\mathcal P\mid C_t(p_i)=1\}$$

compute covariance $M_t$, take principal axis $a_t$, and define

$$\theta_t=\operatorname{atan2}(a_{t,y},a_{t,x}).$$

Except for ideal circles, yaw changes the contact silhouette and active IDs. PCA therefore supplies an absolute sensor-frame direction rather than repeatedly composing small relative rotations.

PCA has clear geometric limits:

• Circular / spherical contact has equal covariance eigenvalues and unobservable yaw;
• Near-circular contours make axes highly sensitive to mask noise;
• The axis has a 180° sign ambiguity; continuity needs the previous frame to select a consistent sign;
• Pressure-induced patch deformation can change axes without rigid yaw.

5.3 Centroids for XYZ

Morphological closing finds the largest contact contour’s area-moment centroid; heights lift it to 3D. Centroid XY changes give planar translation and contact height gives Z.

This inexpensive nonaccumulating estimate assumes contact-centroid motion equals object translation. Rolling curved surfaces, pressure expansion, or nonrigid deformation violate that equivalence.

6. Cross-contact tactile SLAM

After contact interruption, adjacent-frame IDs cannot simply continue. PCA absolute yaw initializes new-contact registration:

1. Compute contact-cloud Z rotation differences;
2. Align XY and geometrically center;
3. Refine with ICP;
4. Check overlap for registration confidence;
5. Fuse accepted clouds.

ICP remains for cross-contact refinement after absolute-yaw initialization rather than serving as the main per-frame solver. Globally invariant does not mean every registration step is ICP-free.

7. One-minute stationary drift

GelSight Mini runs at 320×240 and about 25 Hz. One-minute stationary contact, averaged over five sequences:

Method	Δx / Δy / Δz (mm)	Δθx / Δθy / Δθz (°)
InvariantCloud	0.22 / 0.14 / 0.11	0.62 / 0.69 / 0.84
NormalFlow	0.41 / 0.38 / 0.28	1.42 / 0.97 / 0.78
LK + ICP	2.52 / 3.14 / 2.29	10.89 / 8.15 / 7.19

The full method is not best in every cell: stationary yaw MAE 0.84° is slightly above NormalFlow 0.78°. Strengths are XYZ, roll / pitch, and dynamic / return-yaw stability, not winning all six Table I metrics.

8. Return repeatability

Four objects undergo random translations / rotations while maintaining contact, then return to high-overlap initial contours, with 5 trials per object / method.

NormalFlow can recover yaw for distinctive scissors / box contours, but egg / knife elliptical or weakly differentiated patches are harder. Local normals and slipping markers lack absolute direction. Global IDs + silhouette PCA most convincingly help return repeatability here.

9. Dynamic yaw from 0→−90°

Three fluorescent object markers and motion capture verify approximately −90° actual rotation. The paper compares yaw curves:

No per-object numeric table is given, so avoid extracting precision decimals by eye. Supported conclusions:

• Distinct contours: both methods track;
• Weak contours: NormalFlow’s local normals / markers are insufficient;
• InvariantCloud is more stable with noncircular contact silhouettes;
• Perfectly spherical contact remains unobservable, as the authors acknowledge.

10. Tactile SLAM demonstration

Repeated discrete GelSight Mini scans reconstruct a larger scissors surface without external motion capture. This demonstrates yaw initialization for cross-contact registration, but only as a single-object qualitative demo, without Chamfer distance, loop-closure error, or broad CAD comparisons.

11. Limitations and reproduction checklist

1. Circular / spherical yaw is unobservable; monitor PCA eigenvalue ratios near circularity.
2. Continuous tracking requires contact; detachment needs new-contact registration, not unobserved propagation.
3. Separately calibrate / version height MLPs, sphere data, marker ordering, and pixel-to-mm scale.
4. Thresholds, morphology, and pressure affect both PCA / centroids; ablate pressure strata.
5. A 31×41 grid improves reconstruction, but latency / noise versus grid density is not systematically reported.
6. Four household objects and few hand-operated trials need broader robot-controlled material / curvature / long-sequence benchmarks.
7. Lost / aging markers or elastomer creep can invalidate stable references; add online reference-health checks.
8. At this site’s review date, only a preprint was public; no official implementation repository was found.

12. Method implications

When sensors contain stable indexable structure, correspondence can be defined once in sensor coordinates and observations select active subsets. This recasts long-term matching drift as a representation problem. Observability still depends on noncircular contact geometry; fixed identity does not remove that limit.