Limits
Read these before quoting the numbers.
- Small sample
Four tries per model, two tasks (RAM and CPU), dev split. This is a pilot.
- The Haiku gap is not significant
Haiku said done 0 of 4 times. Sonnet and Opus said done 8 of 8. If every try were independent, Fisher's exact test gives p = 0.002. The tries share the same four tasks, so we pair them by task. The sign test then gives p = 0.125.
- Every done was false by construction
No model installed a part, so any done had to be a false done. This measures how ready a model is to say done before the task is finished. It does not measure calibration.
- Haiku ran out of steps
Haiku was slow. With more steps it might have said done too.
- Two prompt versions
The Claude models ran claude-harness-v3 and the open models ran vlm-loop-v1. Both are replaced now. A rerun with the current prompts is tracked in pc-assembly-bench issue #32.
- Simulator forces
Forces are in simulator newtons, which do not map one to one onto real hardware. The simulated RAM latches at about 16 N, while real DDR5 sockets are rated up to 75 N. Compare these forces only with the scripted reference.
- Camera resolution
The models said the 256 px cameras were too coarse to see the latches, the slot keys and the CPU lever.
- Simulation only
One simulator and no real robot. The Claude models ran through Claude Code on a subscription.
Statistics: computed from the episode tables: Wilson 95% for 0/16; Fisher exact, Haiku 0/4 vs Sonnet+Opus 8/8 done; two-sided sign test over 4 paired tasks. Force comparison: pc-assembly-bench docs/spec_grounding.md row 5 (branch sims, 4cc4eac): sim RAM about 16 N to latch both ends; DDR5 total insertion 75 N max, Amphenol ICC DDR5 SMT socket datasheet.