Deep Reading Notes: TacGraph

1. Key assessment

• Paper: arXiv 2512.23856
• Project: TacGraph
• Problem: local touch cannot uniquely resolve in-hand poses / external contacts, and vision may be occluded. Pose, contact location, and force must jointly satisfy rigid geometry and quasi-static mechanics.
• Answer: bilateral GelSight predicts surfaces, in-hand displacements, and wrenches; geometric consistency, non-penetration, contact kinematics, and force balance jointly solve rest pose, contact points, and forces.
• Assessment: contact becomes a constraint rather than a disturbance. Precision depends on known meshes, point contacts, elastic no-slip grasps, and good initialization, limiting arbitrary in-hand generalization.

2. Task and state variables

Known inputs:

• Object triangle mesh $\mathcal M_o$;
• Static-environment mesh $\mathcal M_e$;
• Gripper / camera calibration;
• Bilateral GelSight Mini images;
• Optional external RGB-D points.

At each time, recover pose $o_t$, contact $c_t$, and force $f_t$. Instead of independently optimizing every full pose, a network measures relative grasp deformation, introducing one time-invariant rest pose $o_r$:

$$o_t=g_t\,\delta_t\,o_r,$$

Here $g_t$ is gripper pose and $\delta_t$ tactile in-hand displacement. Removing external force should restore $o_r$. This reduces temporal freedoms while excluding permanent slip.

3. From tactile images to optimization observations

Three object-agnostic convolutional models are trained.

3.1 Tactile depth

Initial bilateral RGB predicts heights and tactile points $P^T$ in the gripper frame. Optional visual points $P^V$ merge into $P$. Known fixtures under random grasps provide rendered training labels.

3.2 In-hand displacement

Initial / current tactile pairs regress $\delta_t\in SE(3)$. Fixed fixtures and small gripper motions provide known relative-transform labels. Errors in tactile-deformation-to-rigid-displacement prediction propagate into every pose.

3.3 In-hand wrench

A third network predicts gripper wrench $w_t\in\mathbb R^6$ and contact boolean $b_t$ through a weighted-norm threshold. Environment-mounted ATI Gamma F/T supplies training GT, without participating in TacGraph inference.

4. Four factors

MAP becomes nonlinear least squares with Gaussian factor covariances, solved through GTSAM / iSAM2.

4.1 Geometric consistency $h_1$

Transform observed points to object coordinates and measure signed distances to $\mathcal M_o$. Initial tactile / visual points should approach the surface. This constrains rest pose; $\delta_t$ propagates current poses.

4.2 Non-penetration $h_2$

Sample object surfaces, transform to environment coordinates, evaluate $\operatorname{SDF}(\cdot\mid\mathcal M_e)$, and penalize negative interior distances:

$$h_2=\min(0,\operatorname{SDF}(o_t p\mid\mathcal M_e)).$$

This rejects locally plausible poses that penetrate tables / fixtures.

4.3 Contact kinematics $h_3$

Contacts lie on both surfaces, so $c_t$ has near-zero signed distance to both meshes. Infeasible simultaneous contact rejects candidate poses.

4.4 Force balance $h_4$

Contact Jacobian $J(o_t^{-1}c_t)$ maps force to grasp-frame wrench, matched to prediction $w_t$:

$$h_4=J(o_t^{-1}c_t)f_t-w_t.$$

Torque couples contact locations and forces; pose / geometry further disambiguate similar force readings. $h_3,h_4$ activate only when $b_t=1$.

5. Multiple initialized particles

Local nonlinear optimization starts with $K$ rest-pose particles from ICP / grasp-direction heuristics. Each maintains an independent graph; total factor cost $H$ selects the best. Most objects need only 1–2 initializations, but experiments uniformly use 10 for symmetry / local minima.

TacGraph uses multiple local MAP initializations and cost-based selection, without a strict global posterior. Plotted weights reflect $e^{-H}$ rather than establishing normalized, calibrated pose uncertainty.

6. Experiments and ground truth

Hardware: KUKA LBR iiwa Med R820, WSG-50, one GelSight Mini per finger, optional RealSense D435. Object / environment ATI sensors supply training / evaluation GT only.

Pose / contact tests use 3 fixed angled pokes. Object-specific fixtures provide pose GT with roughly 1 mm tolerance; ADD uses 1000 mesh points. Environment F/T infers contact GT by sampling surfaces and matching $c\times f$ to measured torque. Thus contact GT itself inverts a point-contact model rather than independently tracking contact precisely.

7. Pose, contacts, and forces

7.1 Vision + Tactile pose

TacGraph ADD spans 0.68–1.62 mm across six geometries, versus ICP 2.19–4.86 mm and CHSEL 12.28 mm on slanted rectangles. Vision already helps ICP; contact constraints still guide physically feasible solutions.

7.2 Tactile-only pose

Tactile-only cylinder / rectangle / quarter cylinder / slanted rectangle / wrench / screwdriver errors are 2.96 / 1.29 / 8.45 / 7.58 / 0.78 / 1.54 mm. Curvature and weak normal sensitivity make quarter cylinders / slanted rectangles substantially harder; precision varies by object.

7.3 Contact point

Overall contact-point errors:

Method	Vision + Tactile (mm)	Tactile-only (mm)
ICP	4.14 ± 2.92	17.53 ± 9.75
CHSEL	4.33 ± 5.62	24.41 ± 14.52
SCOPE v1	12.28 ± 13.28	16.20 ± 14.49
SCOPE v2	9.83 ± 11.59	10.40 ± 10.93
TacGraph	3.68 ± 3.72	3.72 ± 3.78

These better support joint estimation than pose means alone: removing vision lets baseline contacts drift with wrong poses, while $h_3+h_4$ anchor contacts to surfaces and wrenches.

7.4 Contact force

TacGraph gives 0.61 ± 0.36 N in both sensing settings; others about 0.61–0.68 N. Shared force models mean differences mainly follow contact-location / torque consistency, without establishing independent force-estimator superiority at 0.61 N.

8. Open-loop peg insertion

Per-object fixtures have 3 mm clearance. After exploratory contacts and rest-pose estimation, insertion proceeds open-loop; force above 5 N stops / releases. Each method has 10 attempts per object, 40 total.

Per-object successes: cylinder 7/10, rectangle 6/10, quarter cylinder 1/10, wrench 9/10, total 23/40. ICP gives 11/40, CHSEL 14/40, SCOPE v1 7/40, v2 3/40. Estimates affect task success, but execution remains open-loop after estimation, without coupled feedback control or information-seeking poke selection.

9. Failures and assumptions

1. Known geometry: accurate $\mathcal M_o,\mathcal M_e$ are essential; CAD errors contaminate all factors.
2. Elastic rest pose: forces must leave no permanent displacement; slip makes historical data inconsistent with one $o_r$.
3. Point contact without torque: face, edge, or simultaneous contacts break the $c_t,f_t$ representation.
4. Quasi-static: impacts / significant inertia are absent from force balance.
5. Local optimization: symmetry needs particle coverage, trading computation for robustness.
6. Tactile normal quality: noisy quarter-cylinder depths damage pose and insertion.
7. Manual interaction: experimental contacts are selected manually; action selection remains open.
8. Code at compilation: the official project provided videos / paper, without an implementation repository.

10. Reproduction checklist

• Specify sensor, gripper, object, environment, and robot-base frames.
• Reproduce all three networks’ fixtures, data scales, losses, and calibration.
• Record every factor covariance, which sets relative evidence / physics weights.
• Report particles, ICP seeds, and failed initializations per object.
• Separate learned wrench errors, joint contact localization, and final task success.
• Test mesh errors, slip, double contacts, face contacts, and impacts.
• Use contact uncertainty to correct closed-loop insertion beyond executing one final pose.

11. Methodological takeaways

TacGraph turns interpretable intermediates into physical constraints: points enable geometry consistency, displacement compresses state, and wrenches enforce force balance. This makes conflicts easier to diagnose than large fused-feature networks. Reconstruction, slip detection, and multiple-contact factors are needed to extend precision-fixture demonstrations toward open-world manipulation.