Shape assembly

A robot arm picks up a circle, square or triangle block, turns it to match the opening, inserts it into the matching hole of a fixed board and lets go. Each of the thirty task setups changes one thing: clearance, pose, clutter, appearance, viewpoint, latency or bumps. Every model gets the same 1,500 test tries, with the same starting positions.

Measured · test splitTask list v1.0.0, locked, sha256 c7ac7a273f63ManiSkill 3.0.1, SAPIEN 3.0.3Jetson AGX Orin

Task

The board has three 110 mm cells in a row: circle, square and triangle. Its plate is 12 mm thick, with the top 30 mm above the table. Each hole is the block outline grown by the clearance, 2.0 mm by default. Blocks are 40 mm tall with a grasp knob on top. The geometry is chosen so each block fits only its own hole, at any yaw.

A try succeeds when all four of these hold for 10 control steps in a row:

Matched. The target block is centred over its own hole, within the clearance plus 3 mm.
Deep. The block bottom is at least 25 mm below the plate top, so it is through the plate.
Released. No finger presses on the block and the gripper is open.
Settled. The block moves slower than 2 cm/s.

A block in the wrong hole, balanced on the rim or passed through solid geometry by a physics glitch is flagged and never counts as a success. Models see two colour cameras (third person and wrist), the joint and gripper positions, and a written instruction such as put the square block into the square hole.

30 setups in 7 task groups

Each setup changes one thing from the default scene, A01. Difficulty labels help you read the results and do not change the score.

A01 A02 A03 A04

Clearance

4 setups

B01 B02 B03 B04 B05

Pose and layout

5 setups

C01 C02 C03 C04 C05

Clutter

5 setups

D01 D02 D03 D04 D05

Appearance

5 setups

E01 E02 E03 E04

Viewpoint

4 setups

F01 F02 F03 F04 F05

Latency and disturbance

5 setups

G01 G02

Combined stress

2 setups

Task setups, v1. Source: shape-assembly-bench README.md, 'Configurations'.
Id	Name	Task group	Difficulty	Change from A01
A01	nominal	Clearance	easy	none: 2 mm clearance, block xy noise ±3 cm, yaw ±30°
A02	clearance loose	Clearance	easy	clearance 3.0 mm
A03	clearance tight	Clearance	medium	clearance 1.0 mm
A04	clearance very tight	Clearance	hard	clearance 0.5 mm
B01	wide block xy	Pose and layout	medium	block xy noise ±10 cm
B02	full block yaw	Pose and layout	medium	block yaw any (±180°)
B03	board offset	Pose and layout	medium	board xy offset ±5 cm
B04	board yaw	Pose and layout	medium	board yaw ±30°
B05	far reach	Pose and layout	hard	blocks near the reach limit, any yaw
C01	two blocks	Clutter	easy	2 blocks
C02	three blocks	Clutter	medium	3 blocks
C03	distractor objects	Clutter	medium	3 distractor objects
C04	blocks and distractors	Clutter	hard	3 blocks + 3 distractors
C05	packed blocks	Clutter	hard	3 blocks touching each other
D01	dim light	Appearance	easy	dim lighting
D02	colored light	Appearance	medium	bright coloured lighting
D03	random colors	Appearance	medium	random colours for blocks, board, table
D04	color swap	Appearance	medium	each block wears another shape's colour, 3 blocks
D05	colors and light	Appearance	hard	random colours + dim lighting
E01	camera shift small	Viewpoint	easy	third-person camera moved slightly
E02	camera shift large	Viewpoint	hard	third-person camera moved a lot
E03	partial occlusion	Viewpoint	medium	panel partly hides the board from the third-person camera
E04	shift and occlusion	Viewpoint	hard	small camera shift + occlusion
F01	latency 1	Latency and disturbance	easy	actions applied 1 step (50 ms) late
F02	latency 3	Latency and disturbance	hard	actions applied 3 steps (150 ms) late
F03	shove before grasp	Latency and disturbance	medium	target block kicked at step 15 (0.8-1.1 m/s)
F04	drop after lift	Latency and disturbance	hard	gripper forced open for 10 steps after the first 4 cm lift
F05	nudge near hole	Latency and disturbance	medium	board moved 1.5 cm and turned 10° as the block nears its hole
G01	stress geometry clutter	Combined stress	hard	1 mm clearance, any yaw, 3 blocks, 3 distractors, random colours
G02	stress perception dynamics	Combined stress	hard	1 mm clearance, small camera shift, occlusion, 1-step latency, dim, shove

Scoring

p_c = k_c / n_c config success rate r_g = mean of p_c over category g category rate S = mean of r_g over 7 categories PRIMARY SCORE M = mean of p_c over 30 configs secondary

Every task group counts the same, so the 2-setup stress group weighs as much as the 5-setup pose group. If any setup has no tries, the score S is shown as n/a and the run is marked incomplete.

Per setup: Wilson score interval, the standard 95% range for a success rate. With 50 tries and p = 0.5 it is about ±13 points.
Task groups, score and difficulty: bootstrap. We redraw the tries at random 5,000 times, keeping the count per setup, and take the middle 95% of the results (stratified percentile bootstrap).
Comparing two models: the same bootstrap, paired: both models are redrawn on the same setups and starting positions.
Splits: dev for trial runs, test for official numbers (locked task list only), confirm for re-checking anything found while exploring the results.

track

reference

scripted controller that reads the exact part positions from the simulator; shows the task can be done

track

sanity

moves at random or holds still, with no camera input; checks the task cannot be passed by chance

track

frozen

released model, not trained on this task; the only ranked track

track

adapted

trained on this task, so listed apart from the released models

Results: test split, Franka Panda

Task list v1.0.0, 50 tries per setup, run on a Jetson AGX Orin. Hover or tap a bar for exact numbers.

octo-small-1.5 and octo-base-1.5 inserted no blocks in 3,000 tries and dropped none. Neither model was trained on this robot, its cameras or this task, and we ran them as released with no extra training (zero-shot). We have not tested fine-tuned versions.

Leaderboard

Only released models that were not trained on this task are ranked. The rows below them are shown for comparison. Scores on the xArm6 and WidowX are in Scripted reference on three arms below.

Score = average success rate over the 7 task groups, with a 95% range. Shape assembly, Franka Panda, test split.
Model	Track	Score, % [95% range]	Tries	Status
Released VLA models, not trained on this task
octo-base-1.5	frozen	0.0 [0.0, 0.0]	1,500	Complete
octo-small-1.5	frozen	0.0 [0.0, 0.0]	1,500	Complete
openvla-7b	frozen	not tested yet	-	Set up, not tested on this task yet
openvla-oft (LIBERO)	frozen	not tested yet	-	Code written, not tested on this task yet
For comparison, not ranked
bc-state	adapted	92.1 [91.1, 92.9]	1,500	Complete. Trained on this task, so not ranked.
scripted	reference	99.2 [98.7, 99.7]	1,500	Complete. Knows the exact block positions. Shows the task can be done.
randomIncomplete: 10 of 50 tries per setup	sanity	0.0 [0.0, 0.0]	300	Incomplete: 10 of 50 tries per setup. Moves at random.
holdIncomplete: 10 of 50 tries per setup	sanity	0.0 [0.0, 0.0]	300	Incomplete: 10 of 50 tries per setup. Holds the arm still.

frozen: released model, not trained on this task; the only ranked track. adapted: trained on this task, so listed apart from the released models. reference: scripted controller that reads the exact part positions from the simulator; shows the task can be done. sanity: moves at random or holds still, with no camera input; checks the task cannot be passed by chance. OpenVLA-OFT (LIBERO) was fine-tuned on LIBERO, a public set of simulated tabletop tasks.

Success by task group

bc-state (adapted, trained on scripted demos) does worst when the pose and layout change (72.8 [68.8, 76.8]) and under latency or bumps (78.8 [77.2, 80.0]). It succeeds in 6/50 tries when the block starts far away (B05), and in 0/50 with 150 ms latency (F02).

scripted (reference)bc-state (adapted)octo-base-1.5 (frozen)octo-small-1.5 (frozen)

Clearance4 setups

Pose and layout5 setups

Clutter5 setups

Appearance5 setups

Viewpoint4 setups

Latency and disturbance5 setups

Combined stress2 setups

Success rate (%) by task group, with a 95% range (bootstrap).
Task group	scripted (reference)	bc-state (adapted)	octo-base-1.5 (frozen)	octo-small-1.5 (frozen)
Clearance	96.5 [94.0, 98.5]	99.5 [98.5, 100.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]
Pose and layout	99.6 [98.8, 100.0]	72.8 [68.8, 76.8]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]
Clutter	99.6 [98.8, 100.0]	98.8 [97.2, 100.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]
Appearance	100.0 [100.0, 100.0]	100.0 [100.0, 100.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]
Viewpoint	100.0 [100.0, 100.0]	99.5 [98.5, 100.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]
Latency and disturbance	100.0 [100.0, 100.0]	78.8 [77.2, 80.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]
Combined stress	99.0 [97.0, 100.0]	95.0 [90.0, 99.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]

Success by difficulty

6 easy, 13 medium and 11 hard setups.

scripted (reference)bc-state (adapted)octo-base-1.5 (frozen)octo-small-1.5 (frozen)

Easy6 setups

Medium13 setups

Hard11 setups

Success rate (%) by difficulty, with a 95% range (bootstrap).
Difficulty	scripted (reference)	bc-state (adapted)	octo-base-1.5 (frozen)	octo-small-1.5 (frozen)
easy	100.0 [100.0, 100.0]	99.0 [97.7, 100.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]
medium	100.0 [100.0, 100.0]	95.7 [94.3, 97.1]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]
hard	98.2 [97.1, 99.1]	81.8 [80.5, 83.1]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]

Every setup

Success rate (%) per setup, 50 tries per cell. Hover a cell for successes out of tries and the 95% range.

Success rate per setup and run, test split
Config	scripted	bc-state	octo-base-1.5	octo-small-1.5	scripted · xArm6	scripted · WidowX
Clearance
A01 nominal	100	100	0	0	100	100
A02 clearance loose	100	100	0	0	100	100
A03 clearance tight	100	100	0	0	100	100
A04 clearance very tight	86	98	0	0	76	36
Pose and layout
B01 wide block xy	100	84	0	0	100	100
B02 full block yaw	100	96	0	0	100	100
B03 board offset	100	88	0	0	100	100
B04 board yaw	100	84	0	0	100	100
B05 far reach	98	12	0	0	94	0
Clutter
C01 two blocks	100	96	0	0	100	100
C02 three blocks	100	98	0	0	100	100
C03 distractor objects	100	100	0	0	100	100
C04 blocks and distractors	98	100	0	0	100	100
C05 packed blocks	100	100	0	0	50	100
Appearance
D01 dim light	100	100	0	0	100	100
D02 colored light	100	100	0	0	100	100
D03 random colors	100	100	0	0	100	100
D04 color swap	100	100	0	0	100	100
D05 colors and light	100	100	0	0	100	100
Viewpoint
E01 camera shift small	100	98	0	0	100	100
E02 camera shift large	100	100	0	0	100	100
E03 partial occlusion	100	100	0	0	100	100
E04 shift and occlusion	100	100	0	0	100	100
Latency and disturbance
F01 latency 1	100	100	0	0	100	100
F02 latency 3	100	0	0	0	12	100
F03 shove before grasp	100	96	0	0	100	100
F04 drop after lift	100	100	0	0	100	92
F05 nudge near hole	100	98	0	0	100	100
Combined stress
G01 stress geometry clutter	100	96	0	0	100	98
G02 stress perception dynamics	98	94	0	0	100	94

Secondary metrics

Shown next to the score and left out of it. Recovery is the share of tries that still succeeded after a disturbance, such as the block being knocked or the gripper forced open.

Secondary metrics, test split.
Run	Recovery after disturbance [95% range]	Dropped blocks	Completion time, s (successes)	Inference, ms per query
scripted (reference)	99.5 [97.2, 99.9] (199/200)	0	7.57	0.2
bc-state (adapted)	97.0 [93.6, 98.6] (194/200)	51	8.06	1.4
octo-base-1.5 (frozen)	0.0 [0.0, 3.7] (0/100)	0	n/a	398.6
octo-small-1.5 (frozen)	0.0 [0.0, 3.7] (0/100)	0	n/a	327.8
scripted · xArm6 + Robotiq	100.0 [98.1, 100.0] (200/200)	0	8.03	0.3
scripted · WidowX 250s	96.5 [93.0, 98.3] (193/200)	0	11.72	0.4

Source: shape-assembly-bench/results/reports/jetson_v1_test (REPORT.md, summary.json, per_config.csv). Scripted A04 succeeded 50/50 on an x86 PC and 43/50 on the Jetson. The physics gives slightly different results on different hardware, so we report only Jetson numbers.

Scripted reference on three arms

These runs check the task list on other arms. They are not on the leaderboard.

Scripted reference by category

At 0.5 mm clearance (A04): Panda 43/50, xArm6 38/50, WidowX 18/50. The WidowX cannot reach the far start position in B05 (0/50), and the xArm6 succeeds in 6/50 at 150 ms latency (F02).

Franka PandaxArm6 + RobotiqWidowX 250s

Clearance4 setups

Pose and layout5 setups

Clutter5 setups

Appearance5 setups

Viewpoint4 setups

Latency and disturbance5 setups

Combined stress2 setups

Scripted reference, success rate (%) by task group, with a 95% range, test split.
Task group	Franka Panda	xArm6 + Robotiq	WidowX 250s
Clearance	96.5 [94.0, 98.5]	94.0 [91.0, 97.0]	84.0 [81.0, 87.5]
Pose and layout	99.6 [98.8, 100.0]	98.8 [97.2, 100.0]	80.0 [80.0, 80.0]
Clutter	99.6 [98.8, 100.0]	90.0 [87.2, 92.8]	100.0 [100.0, 100.0]
Appearance	100.0 [100.0, 100.0]	100.0 [100.0, 100.0]	100.0 [100.0, 100.0]
Viewpoint	100.0 [100.0, 100.0]	100.0 [100.0, 100.0]	100.0 [100.0, 100.0]
Latency and disturbance	100.0 [100.0, 100.0]	82.4 [80.8, 84.4]	98.4 [96.8, 99.6]
Combined stress	99.0 [97.0, 100.0]	100.0 [100.0, 100.0]	96.0 [92.0, 99.0]

All three blocks in one try

A separate task that does not count toward the score. Up to 900 steps. All three blocks must be inserted at the same time.

Scripted reference, Franka Panda

This run covers only setups FB01 to FB03, a subset, so it has no overall score.

FB01nominal

FB021 mm clearance

FB03any block yaw

Full-board task, test split, with a 95% range (Wilson).
Setup	Change	Success	95% range	Blocks inserted (mean of 3)
FB01	nominal	48/50 (96.0%)	86.5 to 98.9	2.94
FB02	1 mm clearance	39/50 (78.0%)	64.8 to 87.2	2.56
FB03	any block yaw	38/50 (76.0%)	62.6 to 85.7	2.70

Models and status

Models and their status. Source: shape-assembly-bench README.md, 'Tracks' and 'Robot and model compatibility'; context/blockers.md.
Model	Track	Inputs	Status
scripted	reference	exact positions	Tested on the test split (Panda, xArm6, WidowX)
random, hold	sanity	none	Tested on the test split, 10 of 50 tries per setup (incomplete)
octo-small-1.5, octo-base-1.5	frozen	cameras	Tested on the test split, 1,500 tries each
bc-state	adapted	exact positions	Tested on the test split. A small neural network (MLP) trained on scripted runs from separate training starting positions
openvla-7b	frozen	cameras	Set up, not tested on this task yet
openvla-oft (LIBERO)	frozen	cameras	Code written, not tested on this task yet

Known limits. The blocks are straight prisms with square edges, so tilted insertion is not tested. We designed the board and blocks ourselves. Contact forces come from the simulator and have not been checked against real hardware. Octo runs without temporal ensembling, its option to average several predicted actions for each step.