GeoAnalystBench · Task 34 · 2026-07-27
Estimate the accessibility of roads to rural areas in Japan. This record separates permission to use the task, fidelity to published benchmark gold, and what the agents actually achieved.
OPERATING PATH PROVED
1 · Rights scope
Repository owner attested academic research and dissemination use on 2026-07-27.
Input handling: private inputs; not uploaded to Arena. Only generated results, maps, code, and hashes are published.
2 · Benchmark gold
The bundled reference image reproduced pixel-for-pixel: yes.
The reference code maps an area fraction but does not compute the requested aggregate population percentage. Benchmark match and spatial validity therefore require separate verdicts.
3 · Agent performance
Two Kimi routes produced the benchmark-compatible numeric result and inspectable artifacts. Other routes produced alternate semantics, wrong selections, runtime errors, serialization failure, or no visible program.
Cloudflare Workers AI · local bubblewrap sandbox with network namespace isolated
Published tutorial
about 84 percent
Benchmark-compatible · EPSG:4087
736231.6 estimated accessible residents.
Spatial-validity audit · EPSG:6672
750557.2 estimated accessible residents. This remains an audit result, not accepted benchmark gold.
| Model | Attempts | Terminal state | Benchmark match | Spatial validity | Reported access | Failure detail |
|---|---|---|---|---|---|---|
| moonshotai/kimi-k2.6 | 2 | executed | MATCH | NOT VALIDATED | 84.3349% | — |
| zai-org/glm-4.7-flash | 1 | runtime_error | NO RESULT | NO RESULT | — | GeoSeries has no dissolve method |
| zai-org/glm-5.2 | 1 | executed | NO MATCH | NOT VALIDATED | 86.2101% | — |
| moonshotai/kimi-k2.7-code | 1 | executed | MATCH | NOT VALIDATED | 84.3349% | — |
| openai/gpt-oss-20b | 1 | executed_wrong_result | NO RESULT | NO RESULT | 78.9400% | — |
| google/gemma-4-26b-a4b-it | 2 | no_visible_program | NO RESULT | NO RESULT | — | output limit exhausted |
| qwen/qwen3-30b-a3b-fp8 | 2 | runtime_error | NO RESULT | NO RESULT | — | NumPy int64 was not JSON serializable |
| mistralai/mistral-small-3.1-24b-instruct | 1 | runtime_error | NO RESULT | NO RESULT | — | municipality layer has no ID field |
READBACK MATCHED
READBACK MATCHED
