15 Robotic Grasping Contact mechanics Motion planning Closed-loop manipulation

Robot Grasping & Manipulation

From object poses to
executable, verifiable actions

A 6D grasp pose is only the geometric center of a candidate action. Actual grasping also needs gripper embodiment, opening, contacts, pre-grasp, swept volumes, IK / paths, task semantics, uncertainty, post-grasp hand-object poses, and failure recovery.

Open the grasping lab Unify grasp frames first

01 · What Problem Are We Solving?

Grasp detection can bypass object pose; grasping with 6D pose is one available route

Specify what is known, what is output, which gripper is used, and the downstream task. Model-based, scene-grasp, affordance, suction, dexterous, and insertion methods cannot share a ranking based only on one grasp score.

Instance / model-based grasping

Known object ID, CAD, and ᶜTₒ / ᴮTₒ

Precompute or generate ᴼTɢ online, then transform it into world coordinates using the object pose

Can use full unseen geometry, mass/COM, semantic surfaces, and task constraints; strongly depends on model-to-physical alignment, scale, and the object frame

Model-free scene grasp detection

RGB-D / point cloud without requiring object identity or CAD

Directly outputs a 6-DoF grasp distribution + width / score in camera/world coordinates

Fast onboarding of new objects; weaker reasoning about visible surfaces, occlusion, target assignment, unseen collisions, and task semantics

Category / affordance grasping

Categories, parts, keypoints, language tasks, and partial geometry

Maps handles / rims / supports / functional regions to embodiment-specific grasps

Transfers across instances for tasks such as pouring or handing over; both affordance and geometric feasibility require validation

Suction / vacuum picking

Local surface, normals, curvature, material/sealing, and external wrench

Contact point + approach + cup/pressure; often fewer parameters than a full jaw-grasp pose

Effective in confined spaces and industrial sorting; porous / rough / soft surfaces, edge leaks, and center-of-mass moments are key limits

Dexterous / multi-finger

Hand pose, finger joints, contacts, friction, and task intent

High-dimensional hand configuration + contact sequence, rather than a single 6D palm pose

Highly capable manipulation; substantially harder IK/self-collision/contact dynamics, data collection, and control

Assembly / insertion

Task frames for the object, gripper, and hole/fixture, with low clearance

Choose grasps preserving assembly axes / keys and plan compliance / search / force control

Grasp success is only a prerequisite; hand-object slip, lateral/angular errors, and jamming/wedging determine the final outcome

Object pose and grasp pose are different random variables. Known-CAD approaches often estimate ^BT_O before querying an object-relative grasp library; model-free grasp detectors can directly output ^CT_G without recovering a unique object frame. Both must satisfy robot / gripper / task feasibility; identical output dimensionality does not make them interchangeable.

02 · Frames & Grasp Representations

Define planner grasp frame G, then convert it to the robot's actual controlled tool frame U

Grasp datasets, gripper drivers, URDF end-effector parents, palm centers, finger midpoints, and suction-cup centers often use different origins / axes. Bind every grasp asset to a gripper convention.

Object-relative grasp to controller tool target

{}^{B}\mathbf T_G={}^{B}\mathbf T_O\,{}^{O}\mathbf T_G,\qquad {}^{B}\mathbf T_U^\star={}^{B}\mathbf T_G\,{}^{G}\mathbf T_U

If the stored transform is ᵁTɢ, use its inverse in the second equation. Manually tuning ᴼTɢ to hide a missing G↔U/TCP edge contaminates every grasp library.

Pre-grasp waypoint under a declared approach axis

{}^{B}\mathbf T_P={}^{B}\mathbf T_G\,\operatorname{Trans}(-d_{\mathrm{pre}}\mathbf e_a)

This interaction declares +x_G as approach; other libraries may use ±x/±y/±z. Signs, frames, and desired/minimum distances are part of the action contract.

Representation	Parameters	Additional requirements	Uses and pitfalls
Parallel-jaw 6-DoF	grasp frame pose T, jaw width, approach / closing axes, insertion depth	finger/palm collision geometry, stroke, force, contact pad	Full spatial grasping and clutter planning; dataset axes and origins are not interchangeable
Planar rectangle	Image center (u,v), in-plane angle, pixel width, and sometimes depth	Camera K, depth, and camera→robot transform	Simple for tabletop top-down grasping; lacks general 6-DoF motion, out-of-plane structure, and occlusion geometry
Contact pair	p₁,p₂, normals n₁,n₂, friction / curvature	Recover gripper pose, opening, and approach family from contacts	Closest to antipodal mechanics; the same pair can admit several palm-roll / approach choices
Contact-rooted 6-DoF	observed surface contact + approach / roll + width	Scene point cloud and gripper conventions	Reduces prediction dimensionality and anchors visible geometry; still requires full gripper-collision checks and target assignment
Suction target	contact point, normal / approach, cup radius, seal / wrench score	Surface smoothness, material, vacuum system, and COM/load	Parallel-jaw force-closure scores cannot be reused directly for suction
Multi-finger hand	SE(3) palm pose + q_hand + contact set / synergy	hand kinematics, tendon/compliance, self/object collision	A 6D palm pose is only part of a high-dimensional grasp configuration
Object-relative library	ᴼTɢ, gripper type, width, contacts, quality, task tags	object asset/version, symmetry, mass/friction assumptions	Fast reuse with ᴮTₒ; invalidate the whole library when model scale, frame, or grasp conventions change

Executable grasp record · Pose is just one field

grasp_id / target_object / task_id / gripper_id + geometry_hash
frame_convention: origin / approach_axis / closing_axis / handedness
object_T_grasp OR world_T_grasp / planner_grasp_T_controller_tool
pregrasp_distance / approach_min_desired / jaw_open / jaw_close / force
contact_points / normals / friction + mass/COM assumptions / analytic_quality
collision_padding / allowed_touch(stage, link, object) / unknown-space rule
IK branch / joint margin / path / retreat / attached_object_transform
pose_belief_version / robust_success_event / score_calibration_id
reobserve / verify / lift / abort / recovery policy

03 · Contacts, Friction & Grasp Wrench Space

Antipodal grasps, force closure, and stability start with the contact model

Contact locations, normals, friction, contact patches, gripper forces, object mass / COM, and task wrenches jointly determine what a grasp can resist. Two geometrically symmetric points are insufficient.

Contact model	Allowed interactions	Parameterization	Engineering limits
Frictionless point	Only compressive forces along contact normals; no pulling	1 force DoF per point	A theoretical baseline; its conclusions do not directly replace soft-finger / friction models for real parallel jaws
Point contact with friction	Normal + tangential forces constrained by a Coulomb cone	The cone is often discretized into polyhedral primitive forces	Unknown μ, anisotropy, speed, and surface contamination change the cone; discretization resolution affects analytic scores
Soft finger	Allows a bounded torsional moment about the contact normal in addition to frictional forces	Suitable for finite contact patches and soft pads	Torsional limits depend on pressure distribution; torque is not unlimited
Suction contact	Seal, cup compliance, normal pull, tangential/friction forces, and moment bounds	Must resist gravity / acceleration wrenches without leaking	A locally planar appearance does not establish a material's sealability; edges, holes, curvature, and pneumatic delay matter
Compliant / enveloping	Contacts evolve during closing and deformation, potentially forming caging / power grasps	Requires hand/object dynamics or empirical models	Static point-contact GWS may be too conservative or too optimistic

Coulomb friction cone

f_n\ge 0,\qquad \|\mathbf f_t\|_2\le \mu f_n,\qquad \alpha=\arctan\mu

Contacts push but do not pull; larger μ widens the cone. Numerical planning often uses polyhedral approximations; record the number of primitives and force normalization.

Contact force to object wrench

\mathbf w_i=\begin{bmatrix}\mathbf f_i\\(\mathbf p_i-\mathbf c)\times\mathbf f_i+\boldsymbol\tau_i\end{bmatrix},\qquad \mathbf w=\mathbf G\mathbf f

The upper block is force; the lower block is torque about reference point c. Their units differ. GWS metrics require a characteristic length or task-wrench scaling.

Force-closure geometric test with discretized primitive wrenches

\mathbf 0\in\operatorname{int}\operatorname{conv}\{\mathbf w_{ij}\}\quad\text{and}\quad \operatorname{span}\{\mathbf w_{ij}\}=\mathbb R^6

Intuitively, admissible positive contact forces resist arbitrarily directed small external wrenches. Force closure does not establish that required forces are acceptable, contacts are reachable, or the object will remain undamaged.

Ferrari–Canny epsilon quality

\epsilon=\min_{\|\mathbf u\|_2=1}\ \max_{\mathbf w\in\mathcal W}\mathbf u^{\mathsf T}\mathbf w

ε is the radius of the largest origin-centered ball inside the GWS convex hull, measuring margin in the weakest wrench direction. It depends on force/torque scaling, friction-cone discretization, and force budgets; it is not inherently a calibrated success probability.

Antipodal pair

The line joining two contacts lies inside both friction cones

An efficient candidate condition for parallel jaws, but it does not check palm/finger collisions, opening, COM, external task wrenches, reach, or paths.

Form vs force closure

Geometric constraints differ from available contact forces

Form closure relies mainly on nonpenetration geometry; force closure uses constrained contact forces. A stable enveloping grasp cannot be reduced to 'two antipodal points.'

Gravity / task wrench

Resisting arbitrary 6D wrenches may be unnecessary

Some tasks need stability only in specified directions and accelerations. Task-wrench metrics can better match the application, but loads / trajectories must be disclosed.

Damage & compliance

More gripping force is not always better

Fragile, flexible, deformable, or liquid-filled containers also impose pressure, deformation, slip, and content-dynamics constraints.

Interactive Lab

From frame chains to hard-feasibility funnels and belief-space action gates

All three experiments follow the same principle: preserve physical semantics and hard constraints before considering scores. Map errors to specific contact / collision / task events to decide whether execution is justified.

Interactive · Grasp Frames, Feasibility & Action Risk

Object-to-grasp, candidate feasibility, and closed-loop risk lab

Transform object-relative grasps correctly into base / controller-TCP coordinates, pass the same candidates through contact, width, swept collision, IK / path, and task constraints, then convert pose / calibration uncertainty into action-failure probability.

pose candidate ≠ executable grasp hard constraints before soft ranking analytic quality ≠ calibrated success probability

approach axis +x_G closing axis +y_G controller TCP U Real systems also require z / roll / pitch; this canvas shows only a table-plane slice

Grasp in base

^BT_G = ^BT_O^OT_G

——

Controller target

^BT_U = ^BT_G^GT_U

—Add the fixed edge whenever planner-grasp and robot-tool frames differ

Pre-grasp

same orientation · −approach

——

Command

gripper mode / width

—Pose, opening, approach, closure, and retreat form one action contract

Frame audit

—

——

04 · Candidate Generation

Candidates come from CAD, visible point clouds, affordances, simulation, or task graphs

Candidate sets should cover geometric and execution diversity. Millions of near-duplicate poses do not establish coverage. Deduplication should account for SE(3), contact sets, object symmetry, gripper embodiment, and task equivalence.

01

CAD library / analytic sampling

Sample contacts, approach, roll, and depth on the object mesh; compute antipodal / GWS / collision properties

Known instances, stable object frames, and repeat production; can be recomputed offline

02

Retrieval / transfer

Retrieve similar grasps by object shape/category/parts, then register / optimize

Similar objects and functional grasps; account for scale and local-part differences

03

RGB-D / point-cloud detector

Predict a grasp-pose distribution directly from the visible scene

Unseen objects, fast onboarding, and clutter; linking to object identity / task targets needs additional mechanisms

04

Affordance / task-conditioned

Generate from functional regions, intended actions, placement frames, or language constraints

Task-specific grasps for handles, handovers, pouring, and assembly

05

Simulation / learned policy

Learn success / Q / action trajectories from physical simulation or real trials

Can capture complex contact and control; depends on domain coverage, reward / success labels, and a safety layer

06

Regrasp / TAMP

Treat grasps as discrete modes connecting transit, transfer, placement, handoff, and fixture states

When one grasp cannot satisfy both current reachability and the final task orientation

Symmetry representative and object-relative grasp

{}^{B}\mathbf T_O'={}^{B}\mathbf T_O\mathbf S,\qquad {}^{O'}\mathbf T_G=\mathbf S^{-1}{}^{O}\mathbf T_G\ \Longrightarrow\ {}^{B}\mathbf T_G'={}^{B}\mathbf T_G

When changing an equivalent object-pose representative, transform the object-relative grasp inversely to preserve the world action. Merge only task/action-equivalent modes; check collision and IK individually in the world / robot state.

Pose NMS

SE(3) proximity

Specify scales for translation, rotation, and gripper-specific axes.

→

Contact NMS

Same contact region

Avoid filling top-K with slight palm-roll variations.

→

Symmetry quotient

Object equivalence

Transform grasps consistently when changing pose representatives.

→

Action equivalence

Same robot command / outcome

Deduplicate using IK branches, task frames, and paths.

05 · Hard Feasibility Before Ranking

The final grasp pose is one instant in a trajectory

Record the first failed stage in a fixed order to identify whether candidate generation, collision, robot reach, or task constraints create the actual bottleneck.

Check layer	Question	Engineering requirement
Target & semantics	Whether the correct object / part is grasped, functional surfaces remain accessible, and placement/pouring/assembly remains possible	Freeze the target, task frame, allowed contacts, and forbidden regions before geometric scoring
Contact feasibility	Contacts lie on touchable surfaces; normals / friction cones / seals / jaw widths are plausible	Use an appropriate contact model; apply sensitivity analysis or conservative bounds for unknown friction/material
Final-pose collision	Whether fingers / palm intersect forbidden target regions, neighboring objects, table, or bin walls	Version collision meshes, padding, allowed-touch lists, and open/closed-hand configurations
Approach swept volume	Whether the open gripper collides anywhere from pre-grasp to grasp	Use continuous or sufficiently dense discrete swept-volume checks, covering more than the final frame
Closure motion	Whether closing fingers reach contacts without first pushing away / grasping the wrong object	Simulate jaw trajectories, contact order, compliance, and clutter interaction
IK & robot state	Whether grasp / pre-grasp / retreat poses have IK solutions satisfying joint and self-collision constraints	Save all IK branches, joint margins, manipulability, and the current state
Path & dynamics	Whether continuous trajectories from current q to pre-grasp, approach, lift/transport are feasible	Joint/velocity/acceleration/torque limits, singularities, cables, control interfaces, and deadlines
Attached-object motion	After grasping, check lift / transport / placement with the object attached as robot-link geometry	Use actual ᵁTₒ / hand-object estimates rather than an ideal object center
Uncertainty / robustness	Probability / margin of satisfying these constraints under pose, shape, calibration, friction, and actuation perturbations	Sample the belief, use linearized chance constraints or worst-case sets, and retain systematic bias
Recovery viability	Whether failures or low-confidence observations allow safe retreat, reobservation, pushing, regrasping, or human takeover	Include recoverability in candidate scores; irreversible collisions need more than a low score

Transit

current q → pre-grasp

Open hand, unattached target; whole-robot collision / dynamics.

→

Approach

pre-grasp → grasp

Controlled direction and speed; open-hand swept volume, visibility, and early contact.

→

Closure

finger / suction contact

Stage-specific allowed touch, contact order, and object motion.

→

Transfer

attach → lift / transport

Use actual ᵁTₒ and the attached mesh; check slip, environment, and joint margins.

→

Place

approach → release → retreat

Stable support, task pose, attachment removal, and safe retreat.

Manipulability is a margin, not an IK existence test

m(\mathbf q)=\sqrt{\det\!\left(\mathbf J(\mathbf q)\mathbf J(\mathbf q)^{\mathsf T}\right)}

Near-zero Yoshikawa-style manipulability suggests a local singularity, but depends on units/scaling and cannot replace joint-limit, collision, torque, or global-path feasibility checks.

06 · Robust, Task-Aware Ranking

Rank the same hard-feasible survivors by task utility and risk

Separate analytic proxies, learned logits, calibrated success probabilities, clearance, motion cost, and task value in the score schema. Even simple linear weighting requires disclosed normalization, units, and veto rules.

Analytic contact quality

force closure, Ferrari–Canny ε, GWS volume, min singular value

Specify contact/friction, force/torque scaling, and gravity/task wrenches; compare only hard-feasible candidates

Learned success

Grasp score or outcome probability from a network / policy

Calibrate to the target robot/object/clutter/task event; raw logits across datasets are not a shared probability scale

Clearance robustness

Minimum finger/palm/sweep distance to obstacles / unknown space and its uncertainty

Prefer large margins over nominally collision-free trajectories that barely clear obstacles

IK / motion margin

joint-limit, singularity/manipulability, path length, planning time, control smoothness

IK feasibility and execution difficulty are distinct

Task utility

Approach direction, center of mass, subsequent placement/insertion, visibility retention, and handoff / regrasp cost

A stable grasp that blocks a cup opening, key, or assembly surface is still a poor grasp

Information / observability

Whether pre-grasp / approach improves pose, retains target visibility, or provides discriminative contact

Jointly plan active perception and execution; prevent the gripper from fully blocking a wrist camera

Risk / recoverability

Expected success, CVaR / worst case, collision cost, drops/damage, and recoverability

Frequent minor losses and rare catastrophes cannot be reduced to one undifferentiated average score

Belief-aware action selection

g^\star=\arg\max_{g\in\mathcal G_{\mathrm{hard}}}\ \mathbb E_{\mathbf x\sim b}\!\left[U(g,\mathbf x)\right]-\lambda C(g)\quad\text{s.t.}\quad \Pr[c_k(g,\mathbf x)\ge0]\ge1-\alpha_k

x may contain object pose/shape, calibration, friction, mass, and actuation. Expected utility does not replace chance constraints for collision / damage. High-cost events may require worst-case or CVaR treatment.

Pose error at an object-relative contact / grasp point

\delta\mathbf p_G\approx\delta\mathbf t_O-\left[\mathbf R_{BO}\,{}^{O}\mathbf p_G\right]_\times\delta\boldsymbol\theta_O

Object-rotation error becomes position error through the contact lever arm; the sign depends on perturbation conventions. Compute grasp sensitivity at contact / task points, beyond the object origin.

Linearized clearance chance constraint

\sigma_c^2\approx\mathbf J_c\boldsymbol\Sigma\mathbf J_c^{\mathsf T},\qquad \bar c-z_{1-\alpha}\sigma_c\ge c_{\min}

Applicable only where the clearance function is locally smooth and the belief is approximately unimodal. For multimodal poses, contact switching, and nonconvex collisions, directly sample / optimize each mode.

Score calibration is tied to the outcome event. A 'grasp score of 0.9' may mean normalized analytic ε, a simulated lift label, a network logit, success rate on a single-object table, or confidence in visible contacts alone. Interpret it as a deployment probability only after calibration for the same gripper, domain, attempt/retry rules, and success definition.

07 · Closed-Loop Pick State Machine

Split 'move to pose' into observable stages that can be aborted and recovered

Each stage has entry conditions, success evidence, timeouts, abort motions, and recovery. Execution logs retain object beliefs, robot states, planned/actual paths, contact / pressure, and failure stages.

01

Observe & target

Target detection, pose belief / scene-grasp set, and sensor/calibration health

Wrong target, missing modes, stale timestamps → acquire / abstain

02

Generate & filter

Candidate contacts / ᴼTɢ / width, layered hard constraints, and deduplication

No survivors → change view, gripper, target pose, or rearrange first

03

Plan pre-grasp

IK branch, collision-free transit, open posture, visibility

Planning timeout / low clearance → change candidate, move the base, reduce speed

04

Reobserve

Re-estimate pose / grasp nearby; compare innovation against the predicted belief

Target motion / occlusion / drift → update the plan rather than following the stale trajectory

05

Approach

Visual servoing or controlled linear motion, monitoring distance, speed, torque, and tracking

Deviation / premature contact → stop / retreat / compliant local search

06

Close / seal

Command jaw force/position or establish vacuum; record contact / pressure

No contact, unilateral contact, or pressure leak → reopen / micro-adjust / new grasp

07

Verify

finger position/force, vacuum, tactile, object motion / in-hand pose

Unstable grasp → no high-speed lift; allow a small regrasp

08

Lift & attach

Update ᵁTₒ and attached collision geometry; leave the support surface slowly

Slip, double picks, or neighboring objects lifted → lower / release / recover

09

Transport / place

Monitor slip, collisions, and joint margins throughout, while satisfying the target task frame

Changed hand-object pose → re-estimate before placement / insertion

10

Release & retreat

Confirm stable support, remove attachment, and retreat in a safe direction

Sticking / unstable placement → regrasp or use fixture assistance

Hand–object pose after contact

{}^{U}\mathbf T_O(t)=\left({}^{B}\mathbf T_U(t)\right)^{-1}{}^{B}\mathbf T_O(t)

This should be approximately constant under an ideal rigid grasp. Slip, compliant contacts, or in-hand manipulation change it. Transport / placement / insertion should use updated ᵁTₒ rather than nominal pre-grasp values.

Vision

Global, noncontact, easily occluded

Close-range reobservation, target tracking, and scene updates; affected by finger occlusion, lighting, and timing/calibration.

Tactile

Local contact geometry / slip

Updates hand-object pose, centering, and slip after contact; cannot provide distant clearance before collision.

Force / torque

Contact events and task wrenches

Early contact, insertion, jamming, and excessive forces; requires bias, frame, filtering, and speed-dependent thresholds.

Proprioception / gripper

jaw position, motor current, vacuum

Detects empty grasps, foreign objects, seals, and approximate object width; one threshold often cannot distinguish a stable grasp from a jam.

08 · Hand–Object Pose, Regrasp & Precision Assembly

A liftable pose may leave later tasks unreachable, invisible, or impossible to insert

Assembly constrains grasps backward from the task frame: preserve critical surfaces, keys, and insertion axes. If the current grasp is incompatible, regrasp on a table / fixture / through handoff rather than forcing a singular configuration.

Task frame

Define admissible errors from the hole / fixture

Lateral, axial, tilt, and roll / key-yaw tolerances differ. An overall SE(3) norm hides the most sensitive directions.

Grasp compatibility

Check the final object-in-tool relationship

The grasp must leave insertion features accessible, and the robot must still reach the tool / object task frame after transport.

Regrasp graph

Stable placements and grasps are discrete modes

Edges represent collision-free transit / transfer; nodes require stability, IK feasibility, and task progress.

Compliance / RCC

Use force and deformation to correct local errors passively / actively

Stiffness center, chamfers, friction, speed, and force thresholds determine convergence, jamming, or wedging.

Search

Local spiral / raster / tilt strategies

Use only within expected error bounds, safe forces, and fixture geometry; search does not replace gross pose / frame correction.

In-hand estimation

Re-estimate ᵁTₒ after contact

Fuse vision, touch, and force/contact constraints; explicitly detect slip and grasp settling.

Task-frame error decomposition

\delta\boldsymbol\xi_{\mathrm{task}}=\operatorname{Log}\!\left[\left({}^{F}\mathbf T_O^\star\right)^{-1}{}^{F}\mathbf T_O\right]=[\delta x,\delta y,\delta z,\delta\phi,\delta\theta,\delta\psi]^{\mathsf T}

Insertion is usually more sensitive to lateral/tilt/key-yaw errors, while insertion-axis translation offers planning freedom. Derive thresholds from clearance, chamfers, compliance, and force tests.

09 · Evaluation & Failure Attribution

From analytic proxies to actual task success, retain denominators and failure stages at every level

Attempt success, first-attempt success, success after retries, object-removal rate, task completion, and picks-per-hour answer different questions. Retries improve eventual success but also increase cycle time, collision exposure, and selection bias.

Level	What to report	Interpretation limits
Analytic	force closure / ε / wrench resistance, collision / IK / path margin	Scalable and interpretable; depends on ideal contact/friction/geometry and does not establish actual success
Simulation	Dynamic lifts, perturbations, slip, clutter / contact sequences, and policy rollouts	Broad coverage; simulation gaps in contact, material, sensors, and controllers require real-world validation
Pose-conditioned hardware	Attempt success given a known target + input pose	Isolates grasping/planning/execution; does not measure detection recall or target selection
End-to-end picking	Correct-target picks / attempts, clearance rate, PPH, and damage-free rate	Includes perception, planning, execution, and recovery
Task completion	Placement pose, insertion depth/force, pouring/handover, and assembly acceptance rate	Closest to the application; stage oracles are needed to diagnose failures
Safety / tail	collision, drop, excess force, near miss, P95 cycle, manual intervention	Mean success cannot hide catastrophic events; report risk–coverage and stop/recovery behavior

Hardware outcome contract · Example

target event:
  correct target leaves support by >= 100 mm
  held for >= 3 s through declared acceleration profile
  no environment/robot collision, no neighbor object lifted
  no drop, damage, forbidden contact, excessive force
denominator:
  all commanded attempts, including planning timeout / no grasp / sensor failure
report:
  first-attempt + retry success, per-object/clutter/task bins
  failure stage, cycle P50/P95, intervention, safety events
  pose-oracle / grasp-oracle / motion-oracle ablations

End-to-end stage factorization is diagnostic, not an independence claim

P(S)=P(D)\,P(G\mid D)\,P(M\mid G,D)\,P(E\mid M,G,D)

D=correct perception/target, G=suitable grasp available, M=motion plan, E=successful execution. Conditional probabilities reveal stage ceilings; the actual variables are correlated, so multiplying separate benchmark numbers does not predict deployment performance.

10 · Failure Taxonomy & Release Gate

Localize grasp failures to frames, contacts, collision, motion, execution, or tasks

For each failure, save raw sensor data, pose/grasp sets, filter counts, planning scenes, trajectories, controller/tactile/force logs, and final states. Video alone is incomplete machine-readable evidence.

Failure modes	Symptoms	Detection	Fix
Mixing grasp frames and TCP	The planner appears aligned, but all fingertips have a fixed-vector offset	Visualize G/U axes, command a known pose in free space, and measure the planner center	Add and version the fixed ᴳTᵁ edge instead of compensating through the object-grasp library
Checking only final-pose collision	The endpoint is collision-free, but the open gripper hits a bin wall during approach	Play back swept volumes and continuous pre-grasp→grasp collision traces	Include the complete approach / closure / retreat
Relying only on grasp scores	The highest analytic / network score has no IK solution or is unreachable	Save per-layer rejection counts and scores before filtering	Hard feasibility first; rank only survivors
Overly broad allowed-touch configuration	Palm penetration into the target or pushing neighboring objects is still considered 'legal'	Record contact pairs and penetration by link / object	Specify allowed touch by link, stage, and target surface
Ignoring object attachment	The grasped object collides with the environment or the motion exceeds joint limits	Display the attached mesh and ᵁTₒ during lift/transport	Update the collision scene and hand-object pose after grasping
Using a single object-pose representative	Chooses the wrong grasp surface or misses feasible modes under symmetry / occlusion multimodality	Map each pose mode to a grasp set and deduplicate in action space	Propagate top-K / distributions; merge by task equivalence
Guiding grasps with mean pose error	Good ADD but wrong contact edges / assembly axes, or amplified errors at distant lever arms	contact/task Jacobian, clearance / capture sensitivity	Evaluate risk for the specific grasp and task frame
Optimistic friction / mass assumptions	Stable in simulation / analysis, but slips or tips during real lifts	Sweep μ / COM / mass; test tactile/force feedback and lift acceleration	Robust scores, conservative forces, and alternative contacts / grippers
Reobservation does not fix the error	Random pose accuracy improves each time, but grasping retains a fixed-direction bias	Separate random innovation from bias consistent across frames	Check hand-eye/TCP/model scale; reobservation does not replace calibration
Successful grasp, failed task	Grasping blocks the cup opening or leaves no feasible placement / insertion direction	Derive grasp constraints backward from the final task frame	task-oriented grasp / regrasp / fixture / handoff
High nominal closed-loop rate	The network runs at 50 Hz, but queue / robot delays make control oscillate	End-to-end exposure→command→actuation latency/P95 and phase	Synchronize clocks, discard stale frames, and gate speed / gains
An overly permissive success definition	Brief contact / momentary lifting counts as success, making papers incomparable	Audit lift height, hold time, correct object, and drops/collisions/damage	Freeze the outcome event, denominator, and attempt/retry rules

Release gate · Before autonomous grasping / manipulation

Is the task explicit: known-pose model-based, scene grasping, suction, dexterous manipulation, placement, or insertion?
Are G-frame origins / axes, gripper geometry, opening/force, and the fixed G↔controller-TCP edge versioned?
^BT_G, pre-grasp, approach signs, and object-symmetry representatives: are these derived from the frame chain and visualized?
Are assumptions for contact models, μ, mass/COM, force/torque scaling, and analytic quality disclosed?
Do grasp candidates include width, contacts, approach, closure, retreat, and task tags alongside the 4×4 pose?
Are candidates hard-filtered by target/contact/width/collision/IK/path/task before ranking survivors?
Do collision checks cover open-gripper approach, closure, target/neighbor contacts, and lift/transport/placement/retreat after attachment?
Is IK tied to the current robot state, all branches, joint/self-collision constraints, singularities, and a complete motion plan?
Are unknown / stale geometry, allowed-touch objects / links / stages, and safety padding explicit?
Are object-pose modes and symmetry mapped to grasp/action sets, deduplicated in the world frame, and validated per candidate?
Is pose, shape, calibration, friction, and actuation uncertainty propagated to clearance/contact/task events?
Are learned scores bound to gripper/domain/outcome/calibration IDs, and analytic scores kept distinct from success probabilities?
Is there pre-grasp reobservation, with timeouts, aborts, and safe recovery for approach / closure / verification / lift?
Are ᵁTₒ and attached collision geometry updated after grasping, with slip / wrong-object / double-pick detection?
Does placement / insertion validate grasp compatibility, regrasping, and compliance / force limits backward from the task frame?
Does hardware success freeze lift/hold/collision/drop/damage/retry definitions and include timeouts / no-plan cases in the denominator?
Are first-attempt, with-retry, per-object/clutter/task, cycle P95, safety, and human-intervention results reported together?
Has the system passed pose/grasp/motion oracles, perturbation, payload/friction, calibration-drift, and recovery-injection tests?

Original sources for grasp mechanics, 6-DoF grasp detection, robust planning, closed loops, and assemblyExpand for Nguyen, Ferrari–Canny, Murray–Li–Sastry, Dex-Net, GraspNet, ACRONYM, Contact-GraspNet, GG-CNN, belief planning, TAMP, MoveIt, and Whitney

From object poses to executable, verifiable actions

Grasp detection can bypass object pose; grasping with 6D pose is one available route

Known object ID, CAD, and ᶜTₒ / ᴮTₒ

RGB-D / point cloud without requiring object identity or CAD

Categories, parts, keypoints, language tasks, and partial geometry

Local surface, normals, curvature, material/sealing, and external wrench

Hand pose, finger joints, contacts, friction, and task intent

Task frames for the object, gripper, and hole/fixture, with low clearance

Define planner grasp frame G, then convert it to the robot's actual controlled tool frame U

Antipodal grasps, force closure, and stability start with the contact model

The line joining two contacts lies inside both friction cones

Geometric constraints differ from available contact forces

Resisting arbitrary 6D wrenches may be unnecessary

More gripping force is not always better

From frame chains to hard-feasibility funnels and belief-space action gates

BTG = BTOOTG

BTU = BTGGTU

same orientation · −approach

gripper mode / width

—

object / scene generator

antipodal proxy / opening

open gripper · approach

whole robot, not palm point

—

no swept collision

object enters jaw / fixture

friction + orientation

—

—

Candidates come from CAD, visible point clouds, affordances, simulation, or task graphs

CAD library / analytic sampling

Retrieval / transfer

RGB-D / point-cloud detector

Affordance / task-conditioned

Simulation / learned policy

Regrasp / TAMP

SE(3) proximity

Same contact region

Object equivalence

Same robot command / outcome

The final grasp pose is one instant in a trajectory

current q → pre-grasp

pre-grasp → grasp

finger / suction contact

attach → lift / transport

approach → release → retreat

Rank the same hard-feasible survivors by task utility and risk

force closure, Ferrari–Canny ε, GWS volume, min singular value

Grasp score or outcome probability from a network / policy

Minimum finger/palm/sweep distance to obstacles / unknown space and its uncertainty

joint-limit, singularity/manipulability, path length, planning time, control smoothness

Approach direction, center of mass, subsequent placement/insertion, visibility retention, and handoff / regrasp cost

Whether pre-grasp / approach improves pose, retains target visibility, or provides discriminative contact

Expected success, CVaR / worst case, collision cost, drops/damage, and recoverability

Split 'move to pose' into observable stages that can be aborted and recovered

Observe & target

Generate & filter

Plan pre-grasp

Reobserve

Approach

Close / seal

Verify

Lift & attach

Transport / place

Release & retreat

Global, noncontact, easily occluded

Local contact geometry / slip

Contact events and task wrenches

jaw position, motor current, vacuum

A liftable pose may leave later tasks unreachable, invisible, or impossible to insert

Define admissible errors from the hole / fixture

Check the final object-in-tool relationship

Stable placements and grasps are discrete modes

Use force and deformation to correct local errors passively / actively

Local spiral / raster / tilt strategies

Re-estimate ᵁTₒ after contact

From analytic proxies to actual task success, retain denominators and failure stages at every level

Localize grasp failures to frames, contacts, collision, motion, execution, or tasks

Robot Coordinate Transforms & System Calibration

From object poses to
executable, verifiable actions

^BT_G = ^BT_O^OT_G

^BT_U = ^BT_G^GT_U