Deep Reading Notes: SurgiPose

1. Key assessment

SurgiPose bridges surgery video without synchronized robot state to action trajectories for imitation learning:

$$\text{monocular video} \rightarrow \text{mask / crop} \rightarrow \{T_{CE,t},\mathbf q_t\}_{t=1}^{T} \rightarrow \Delta \text{actions} \rightarrow \text{ACT policy}.$$

Known URDF and an offline-trained joint-conditioned Gaussian renderer recover framewise end-effector poses / joints through render-and-compare, then adjacent states become relative actions.

The important experiment uses the same 220 / 224 demonstrations with GT or estimated labels. Roughly 1 cm trajectory errors still permit learning, but 70% versus 100% and 60% versus 80% show real costs, not lossless replacement.

2. Inputs, outputs, and hidden priors

Direct inputs

• monocular endoscopic video frames;
• camera intrinsics;
• Target URDF / geometry;
• Pretrained articulated Gaussian renderer;
• SAM2 segmentation.

Output states

$$T_{CE,t}\in SE(3), \qquad \mathbf q_t=[q_{\mathrm{pitch}},q_{\mathrm{jaw1}},q_{\mathrm{jaw2}}]^\top.$$

Only three end-effector joints are modeled, rather than all PSM states. Absolute poses use camera coordinates.

Interpreting hardware independence

Test inference avoids robot encoders, stereo depth, and mocap, making existing monocular videos attractive. Priors remain:

• Accurate intrinsics;
• Tool URDF;
• Per-tool MuJoCo data and renderer training;
• Initial bilateral jaw visibility;
• Continuous tool visibility;
• Stable SAM2 masks.

This is model-based video-kinematics recovery without synchronized telemetry, with different assumptions from arbitrary-video model-free motion capture.

3. Producing 324 first-frame candidates

Initial guess means the first video frame’s tool pose:

1. Compute the mask center;
2. Center a parallel 3×3 image-plane grid;
3. Generate 36 $z$ rotations per point, spanning $[0,2\pi)$;
4. Obtain $9\times36=324$ hypotheses;
5. Refine each through differentiable rendering;
6. Select lowest pixel-averaged loss as $T_{CE}^{initial}$.

Pixel averaging avoids favoring / penalizing near-camera candidates merely through greater projected area.

Coarse sampling excludes:

• Image $x$ rotation;
• Image $y$ rotation;
• depth.

Refinement must recover these variables. Candidates stay manageable, but weak monocular depth outside convergence basins is not solved by 324 starts. Missing experiments include:

• Coarse-only versus refined errors;
• 324 versus single / random-start success;
• Grid scale / initial depth sources;
• Candidate-count accuracy / timing curves;
• First-frame iterations per start.

4. Joint-conditioned Gaussian tool models

MuJoCo renders large-needle-driver URDF configurations:

• 500 canonical poses with neutral joints;
• 10,000 random collision-free three-joint configurations;
• 12 random cameras per configuration, varying azimuth, elevation, and distance;
• Joint, extrinsic, intrinsic, depth / point-cloud, and image labels.

Literally, $500+10{,}000=10{,}500$ configurations at 12 views imply about 126,000 views, but final image counts, splits, and storage are not tabulated.

Three articulated-Gaussian training stages:

1. Learn canonical static geometry;
2. Add deformation for jaw / pitch configurations;
3. Optimize canonical geometry and deformation jointly.

Public 3DGS code trains for 20,000 steps on RTX 4090. Shortened URDF shafts focus capacity on small jaws.

This learns articulated radiance from synthetic appearance / depth instead of directly rasterizing CAD masks. Potential appearance benefits introduce:

• synthetic-to-real appearance gap;
• New-tool generation / training costs;
• Identifiability between deformation and real joints;
• Opacity / scale shortcuts that shrink projections to reduce loss.

5. Joint differentiable pose / angle optimization

Rendered $I_{\mathrm{ren}}$ and observed $I_{\mathrm{obs}}$ crops use:

$$\mathcal L_{\mathrm{combined}} = \alpha\left(1-\mathrm{SSIM}(I_{\mathrm{ren}},I_{\mathrm{obs}})\right) +(1-\alpha)\|I_{\mathrm{ren}}-I_{\mathrm{obs}}\|_2^2, \qquad \alpha=0.8.$$

SSIM constrains local structure / appearance; MSE retains pixel errors. SurgiPose uses articulated Gaussian appearance rather than only binary silhouettes, unlike the preceding calibration paper.

Optimization settings:

• Rotation learning rate $\alpha_R=0.3$;
• Translation learning rate $\beta=3\times10^{-4}$;
• Translation-gradient clamp $[-0.02,0.02]$;
• Joint learning rate $\gamma=10^{-3}$;
• learning-rate reduction factor 0.5;
• plateau patience 20 epochs;
• Stop when loss changes stay below $10^{-7}$ over 10 iterations;
• Clamp joints to $[\mathbf q_{\min},\mathbf q_{\max}]$.

The reported approximate rotation update is:

$$R_{\mathrm{updated}} = R_{\mathrm{current}}\left(I+\alpha_R\nabla R\right),$$

Then enforce a valid homogeneous transform. Projection to $SO(3)$ is unspecified: SVD, Gram–Schmidt, and quaternion normalization differ materially.

Joint update:

$$\mathbf q_{\mathrm{updated}} = \operatorname{clamp}\left( \mathbf q_{\mathrm{current}} -\gamma\nabla_{\mathbf q}\mathcal L, \mathbf q_{\min}, \mathbf q_{\max} \right).$$

Bounds enforce mechanical limits without resolving pose–articulation ambiguity in monocular projections.

6. Temporal tracking

After first-frame initialization / refinement, subsequent frames:

1. Reuse preceding refined poses;
2. Optimize poses and joints jointly;
3. Limit optimization to 10 steps;
4. Stop at plateaus.

This is framewise warm-start optimization, without explicit:

• velocity / acceleration motion model;
• Kalman / particle filter;
• optical flow;
• temporal consistency loss;
• re-detection / lost-state recovery;
• occlusion memory.

Continuous visibility / small interframe motion allow smooth following; occlusion, exiting views, and mask jumps propagate errors. The authors explicitly require continuous visibility and both initial jaws.

Ten iterations alone do not establish real-time operation. Extraction timing, FPS, sequence lengths, and memory are unreported; hardware / time concern model and policy training only.

7. Absolute states to learnable actions

The dVRK Si setup includes:

• PSM + large needle driver;
• A fixed ECM endoscope defining the global frame;
• A needle-driver wrist camera for policy-local imagery;
• Pink suture-pad background;
• White 2D tissue-lifting pad;
• Surgical needles for pickup.

Only endoscopic monocular video estimates kinematics. Wrist images do not enter pose estimation, but supply additional ACT training / execution feedback.

SRT hybrid-relative actions use:

• Endoscope-frame translation deltas;
• Current-end-effector-frame rotation deltas;
• Separate jaw commands.

Relative actions reduce encoder absolute-state effects. Policies are image-conditioned feedback systems rather than blind trajectory replays; both needle policies visually align using wrist cameras. Success therefore also depends on encoders, ACT chunks, and feedback, beyond label accuracy.

8. Data and evaluation

Main data:

• tissue lifting: 220 demonstrations;
• needle pickup: 224 demonstrations;
• Synchronized videos and GT kinematics;
• The same videos reestimated by SurgiPose.

Each task trains:

1. Images with GT relative actions;
2. Identical images with estimated relative actions.

Shared demonstrations isolate label replacement reasonably well. Splits, use of all 444 demonstrations, seeds, horizons, ACT settings, and augmentation remain unspecified.

Each policy / task has 10 trials: one sample changes success by 10 points. The 100% / 70% and 80% / 60% comparisons have substantial binomial uncertainty, without intervals or multiseed results.

9. Trajectory errors

Definitions:

$$\mathrm{ADE} = \frac1T\sum_{t=1}^{T} \|\hat{\mathbf p}_t-\mathbf p_t\|, \qquad \mathrm{FDE} = \|\hat{\mathbf p}_T-\mathbf p_T\|.$$

Results:

Metric	Tissue lifting	Needle pickup
ADE ↓	9.7 ± 2.8 mm	12.0 ± 2.8 mm
FDE ↓	15.3 ± 4.7 mm	14.4 ± 4.5 mm
Mean $(x,y,z)$	$[-4.64,0.25,6.64]$ mm	$[-6.84,4.20,5.61]$ mm
Std $(x,y,z)$	$[1.27,1.40,3.85]$ mm	$[2.38,1.48,5.84]$ mm

Two interpretations:

1. Needle pickup has greater average error, but tissue lifting slightly greater final error, suggesting different accumulation patterns.
2. Both tasks have positive, largest $z$ bias, consistent with monocular scale / perspective ambiguity.

Rendering may shrink projected tools to reduce discrepancy, placing them too far from cameras. This can be systematic loss / representation bias rather than zero-mean noise.

GT comes from robot forward kinematics, itself subject to cable-driven encoder errors. Hybrid-relative actions mitigate this. These notes distinguish recorded paper GT from external physical ground truth.

10. Imitation learning has a measurable cost

Policy supervision	Tissue lifting	Needle pickup
Ground-truth kinematics	10/10; 25.0 ± 3.9 s	8/10; 23.5 ± 5.0 s
Estimated kinematics	7/10; 33.0 ± 2.4 s	6/10; 22.8 ± 2.9 s

Tissue lifting

• Estimated labels reduce absolute success by 30 points;
• Successful trials average an extra 8.0 s;
• Pushing tissue away rather than lifting matches the depth bias;
• Failed trials are excluded from timing; 33.0 s is not an all-10-trial average.

Needle pickup

• GT policy success is itself only 80%, showing task / data limitations;
• Estimated labels yield 60%, down 20 points;
• Both policies visually align through wrist views rather than open-loop replay;
• Mostly successful training demonstrations lack recovery behavior for accumulated errors.

Comparable supports useful supervision / feasibility, without statistical equivalence. Ten trials without seed variation cannot establish stable gap estimates.

11. Cross-dataset evidence

SurgRIPE

SurgRIPE uses keydot markers for absolute tool poses, then inpaints markers; it lacks joint-angle GT. SurgiPose reports random-frame qualitative silhouettes only, despite available absolute GT, without rotation / translation errors.

Ex vivo cholecystectomy

Captured ex vivo cholecystectomy includes kinematics and ProGrasp forceps. Figure 6 shows:

• Blue estimates / red recorded trajectory projections;
• Three segmented-tool / rendered overlays.

ADE, FDE, and joint errors remain absent. New tools need geometry / URDF and articulated Gaussians; synthetic ProGrasp counts / parameter reuse are unspecified. This does not establish arbitrary-tool zero-shot transfer.

Evidence supports operation with a second tool / ex vivo domain, without quantitative cross-tool validation.

12. Gaps toward internet-scale surgical learning

Although motivated by online video, main data use synchronized dVRK Si captures:

• Fixed cameras;
• Known intrinsics;
• Known tool types;
• Known URDFs;
• Relatively controlled first poses;
• Synchronized video / kinematics, unused by the estimation branch.

Public videos additionally introduce:

• Compression, scaling, and unknown / changing intrinsics;
• Camera motion, zoom, and cropping;
• Multiple / replaced / partially visible tools;
• Smoke, blood, tissue occlusion, and highlights;
• Unknown robot models;
• Edits, shot changes, and discontinuous trajectories;
• Unknown metric scales;
• Left / right and symmetric-tool ambiguities.

Current results demonstrate controlled-video feasibility rather than completed internet-scale harvesting.

13. Reproduction status

Available dependencies

• dVRK URDF / model repository;
• SAM2;
• MuJoCo;
• Base 3D Gaussian Splatting;
• ACT, EfficientNet-B3, and related public methods.

Reported settings

• 500 canonical + 10,000 articulated configurations;
• 12 views per configuration;
• 3DGS training 20,000 iterations, RTX 4090;
• SSIM weight 0.8;
• rotation lr 0.3;
• Translation learning rate $3\times10^{-4}$;
• Translation-gradient clamp $[-0.02,0.02]$;
• Joint learning rate $10^{-3}$;
• LR reduction factor 0.5, patience 20;
• Loss-change threshold $10^{-7}$ over 10 steps;
• later-frame maximum 10 iterations;
• Policy training about 20 h on RTX 4090.

Missing

• Method repository / checkpoints;
• MuJoCo scenes, rendering ranges, and splits;
• exact articulated 3DGS / deformation implementation;
• 3×3 spacing, initial depth, and crop normalization;
• first-frame candidate refinement steps;
• SAM2 prompt / tracking strategy;
• $SO(3)$ projection implementation;
• pose extraction runtime;
• ACT config, train split, random seeds;
• 220 / 224 demonstrations and ex vivo data;
• per-frame rotation error / joint-angle error;
• Failure detection / reinitialization.

As of 2026-07-31, the author page linked the paper, but no public SurgiPose code, data, or checkpoints were found. Cited dVRK / 3DGS dependencies do not establish that the method itself is open source.

14. Limitations

• Per-tool URDF and specially trained Gaussian models;
• First-frame planar / optical-axis search without coarse depth or $x,y$ rotation;
• Unreported 324-candidate initialization time;
• Unreported FPS despite 10-step later frames;
• No explicit temporal loss, dynamics, occlusion recovery, or redetection;
• Continuous visibility required;
• Both jaws visible initially;
• SAM2 first-frame dependence;
• Systematic positive depth bias;
• Kinematic rather than externally measured GT;
• Translation trajectories only, without rotation / joint errors;
• No quantitative comparison with other monocular articulated trackers;
• Only 10 policy tests per task;
• Estimated supervision loses 20–30 success points;
• Qualitative SurgRIPE / ex vivo transfer only;
• No actual online-video dataset experiment;
• Unreleased method code / training data.

15. Overall assessment

SurgiPose asks whether recovered kinematics suffice for policy supervision beyond framewise pose metrics. Its 324-candidate initialization, articulated Gaussians, and temporal warm starts form a clear model-based pipeline. Roughly 1 cm ADE still permits tissue / needle policies, a useful positive result.

Estimated and GT kinematics are not equivalent: 20–30-point success gaps, depth-induced pushing failures, and qualitative-only transfer remain. On controlled known-tool, fixed-camera, continuously visible dVRK data, approximate RGB kinematics can supervise imitation. Internet-video learning still needs camera self-calibration, occlusion recovery, metric depth, cross-tool models, and large independent-GT evaluation.