Paper Reading Notes (Deep): JENGA

1. Key assessment

A correctly posed bottom brick can still be pinned by others, and attempting it can cause collisions or destabilize the stack. JENGA therefore adds selection to standard 6D pose:

$$G=F(\Pi,I,M,\alpha_s),$$

Here $\Pi$ is all detector / estimator hypotheses, $G\subseteq\Pi$ the preferred graspable subset, $I$ RGB, $M$ the known model, and $\alpha_s$ IMU gravity direction.

The contribution is two interpretable filters and a protocol checking graspability labels / pose together, not a new pose network. YOLOv8 and ZebraPose are replaceable.

2. Inputs, outputs, and system boundaries

Required:

• Known brick instance and metric CAD;
• YOLOv8 detector;
• Object-trained RGB-only ZebraPose;
• camera intrinsics;
• Camera–IMU rigid transform $T_{sc}$;
• Gravity measurements while stationary or with negligible added linear acceleration;
• Pose-conditioned rendering for amodal silhouettes;
• Top-$k$ and visibility thresholds.

Not required:

• RGB-D depth inputs;
• IMU angular velocity;
• Full world pose from SLAM / VIO;
• learned grasp network.

Although D435i and synthetic data provide depth, method equations do not use it. The site classifies this as RGB + IMU + CAD rather than RGB-D pose.

3. Generating pose hypotheses

YOLOv8 boxes each brick; ZebraPose processes each crop:

1. Encode CAD surface points with coarse-to-fine binary codes;
2. Predict pixel surface codes and segmentation;
3. Build 2D-pixel–3D-model correspondences;
4. PnP recovers $T^i_{co}=[R^i_{co},o^i_c]$.

All results form

$$\Pi=\{T^1_{co},\ldots,T^n_{co}\}.$$

JENGA neither changes ZebraPose nor refines poses. Missed detections, instance-merged masks, and wrong symmetric PnP solutions can only be filtered, not repaired.

4. Visual visibility filtering

For each predicted pose:

• Amodal mask $M_a^i$ renders full CAD at $T^i_{co}$, including currently occluded parts;
• Modal mask $M_m^i$ is ZebraPose’s visible segmentation.

Define

$$r_i=\frac{|M_m^i|}{|M_a^i|}.$$

Discard poses with $r_i<\epsilon_{\mathrm{vis}}$; experiments fix $\epsilon_{\mathrm{vis}}=0.80$.

One pose supplies ideal full and observed visible projections, requiring no extra graspability network. Risks include:

• Amodal area uses predicted pose, so pose errors change the denominator;
• Severe occlusion can lose modal pixels or merge neighboring bricks;
• The 80% threshold has synthetic validation but no sensitivity curve;
• High visibility does not guarantee finger-approach clearance.

The ratio is a pose-conditioned confidence heuristic, not independent physical graspability measurement.

5. Gravity-height filtering

Map object positions from camera to sensor coordinates:

$$o_s^i=T_{sc}^i o_c^i.$$

A stationary IMU measures

$$\alpha_s=R_{sg}g_{\mathrm{global}}.$$

Only relative gravity-axis ranking is needed, using

$$o_{g,z}^i=\alpha_s^\top o_s^i$$

to score candidates and retain highest top-$k$ set $G_h$. Missing acceleration normalization does not affect same-frame ranking, but robot-motion acceleration can contaminate $\alpha_s$. With a flange camera, inference during motion may not observe pure gravity; stationary sampling, low-pass filtering, and dynamic compensation are unspecified.

Typically height filtering precedes visibility:

$$G=F_v(F_h(\Pi,I,M,\alpha_s)).$$

Top-$k$ can prematurely discard lower, more visible objects. Filter order / joint scoring are not compared.

6. Synthetic dataset

A scanned brick forms 3×3 units and procedural stacks. Cameras sample a spherical shell around stack centers. Outputs:

• RGB;
• depth;
• modal / amodal masks;
• 6DoF pose;
• binary graspability label;
• BOP and COCO formats.

The 12K images split 10K / 1K / 1K. Missing details include per-image brick counts, topology counts / split isolation, camera radii, lighting parameters, and real-training / synthetic-testing domain-gap analysis.

7. Graspability truth is geometric heuristics

Transformed eight-corner model boxes and a center KD-tree find neighbors. Shift each box one own dimension along $\pm X,\pm Y,\pm Z$ and test box intersection. The text defines:

• At least four neighbor-free directions → graspable;
• Exactly three also require free $+Z$, at least one free left / right side, and one free front / back side.

These scalable labels omit real grasp factors:

• Axial / principal-direction boxes miss holes and local geometry;
• Friction, mass, and center of gravity;
• Finger thickness, approach direction, and collisions;
• Stack dynamics after extraction;
• Labels mean surrounding clearance / top access, not physics-validated grasp success.

Documentation also differs: the text includes the exactly-three rule, while Algorithm 1 shows only $m\ge4$; pseudocode IoU conditions are unclear. Reproduction should follow author code, which was not public at review time.

8. Joint evaluation protocol

Construct ADD-S / MSSD costs between predictions $G$ and GT graspable poses, then use Hungarian matching. Errors below $\theta$ are TP; wrong poses / unmatched predictions are FP.

$$\mathrm{Precision}= \frac{TP}{TP+FP}, \qquad AP_e= \frac{1}{|\Theta_e|} \sum_{\theta\in\Theta_e} \mathrm{Precision}(\theta).$$

Average $AP_{\mathrm{ADD-S}}$ and $AP_{\mathrm{MSSD}}$ into the paper’s mAP.

This differs from detection AP:

• No confidence-swept precision–recall curve;
• Precision is averaged over pose-error thresholds;
• Recall is not reported;
• Smaller top-$k$ means fewer outputs and naturally fewer FP;
• One easy brick can score well while other graspable bricks are omitted.

It measures trustworthiness of a few returned candidates, not complete graspable-object recovery or stack-clearing efficiency.

9. Main results and ablations

Output set	APADD-S ↑	APMSSD ↑	mAP ↑
all poses	0.19	0.16	0.17
top-1	0.84	0.74	0.79
top-2	0.74	0.66	0.70
top-3	0.68	0.62	0.65

Low all-pose precision shows why predictions need selection. However, top-1 and all-pose output counts differ greatly, so this is not a one-to-one pose-accuracy comparison. Evidence most clearly supports ranking one trustworthy brick first; later candidates are less reliable.

Strategy	mAP ↑
Random choice	0.17
Detector confidence	0.60
Inertial only	0.61
Vision only	0.73
Vision + inertial	0.79

Visibility supplies most gains, gravity a smaller complement. Confidence intervals / seeds and bins for regularity, occlusion, or camera tilt are absent.

10. Real deployment evidence

Hardware:

• Yaskawa HC20DTP six-axis collaborative arm;
• horizontal reach 1700 mm, vertical reach 3400 mm, payload 20 kg;
• custom two-pronged gripper;
• flange-mounted Intel RealSense D435i.

Project videos show real stacks / construction-site grasping, but the paper omits:

• Total trials and success rate;
• Selection, pose, and gripper-failure breakdowns;
• Inference / robot-action time;
• Post-grasp stack disturbance;
• Motion-time IMU filtering;
• sim-to-real pose accuracy.

Deployment is qualitative operability evidence, not real-world validation of synthetic mAP 0.79.

11. Availability and reproduction

As of 2026-07-31, the accessible video project retained github.com/YOUR REPO HERE and “Code, Dataset and more details coming soon…”; downloadable data / method code could not be verified.

Reproduction still needs:

• YOLOv8 / ZebraPose training settings and synthetic-real augmentation;
• camera–IMU calibration;
• gravity normalization / motion rejection;
• Top-$k$ and filter order;
• Amodal rasterization / clipping;
• Exact threshold sets $\Theta_e$;
• Consistency between Algorithm 1 and the textual exactly-three rule;
• Real sequences, robot commands, and grasp outcomes.

12. Limitations

• One brick category / fixed CAD does not establish general stacked-object performance;
• All 12K quantitative images are synthetic;
• Box-neighbor labels are not physical grasps;
• mAP naming can be confused with detection / BOP AP;
• Precision without recall, coverage, or throughput;
• Some top-1 gains come from fewer outputs;
• Visual ratios depend on correct pose / masks;
• Gravity ranking assumes no extra linear acceleration;
• No threshold, filter-order, or stack-difficulty sensitivity;
• Real robot video without success statistics;
• Code / data not released at review time.

13. Overall assessment

JENGA adds a practical decision layer: prefer high, visible poses in structured stacks rather than pass every detection equally to grasp planning. Modal / amodal ratios and gravity ranking are lightweight; ablations favor visibility over confidence for ranking.

Its precise role is pose-aware stacked-object candidate selection: one trustworthy synthetic candidate and a running real robot, not a complete physical grasp benchmark. Coverage, long-term clearing, and cross-object generalization need open data, physical labels, and larger real trials.