How we ran it

Two tasks: install two RAM modules, and install a CPU. Each task ran once on two setups: M01-L1 (a compact case) and M05-L2 (a mid tower), with starting positions from the dev split. That is four tries per model, and 24 tries across the six models.

A RAM try had 750 steps and a CPU try 900, at 25 steps per second. A try counts as a success only if the simulator finds the part installed.

On each call the model gets the camera images (256 px) and the arm's position, and answers with a short plan: moves, gripper commands and waits. A plan can end with done. Done ends the try at once.

The six models come from four makers: Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 (Anthropic), Gemma 4 E4B (Google), Molmo2-ER (Ai2) and Phi-4 multimodal (Microsoft). The Claude models ran through Claude Code on a subscription. The open models ran on the Jetson.

The Claude models ran with prompt claude-harness-v3. It says that done ends the try at once and belongs only in the final plan.

The open models ran with prompt vlm-loop-v1. It had no rule about done, and the example plan in its task card ended in done. Molmo2-ER and Phi-4 multimodal copied that example.

The prompt changes the result a lot. With an earlier prompt that ended "then send a plan whose last op is done", Claude Haiku 5.5 said done in its first plan in 4/4 tries (1 call, 35-47 steps). With claude-harness-v3 it said done in 0 of 4.

Both prompts are now replaced by claude-harness-v5 and vlm-loop-v3, which give every model the same done rule and an example plan without done. Results from different prompt versions are not compared.

A scripted reference that reads the exact part positions from the simulator, sending plans through the same interface, succeeded in 4/4 tries. A random baseline failed 0/4 and reached 457 N.

In the full benchmark, the scripted reference installs RAM on all 30 setups with a peak robot force of 51.8 N.