Comparison unit
A benchmark result is meaningful only with its task set, dataset, tool surface, runtime and verifier. Model names alone are not comparable.
Axis Spatial · public evidence index
Geospatial AI
Benchmark Registry
A source-grounded map of what geospatial AI benchmarks actually measure, how they verify results, and where the evidence ends. Unknown fields remain explicit.
Source before summaryPaper, repository, dataset and leaderboard references stay attached to each entry.
Proof has boundariesEvery entry separates what the benchmark can show from what it cannot.
Unknown means unknownMissing evidence is marked pending verification, never inferred.
Registry matrix
Updated from public source evidence · horizontal scroll on small screens| Benchmark | Tasks | Evaluates | Exec? | Det. verify? | Modality | Licence? | What it can show | What it cannot show |
|---|---|---|---|---|---|---|---|---|
| GeoAnalystBench / GISclaw | 50 | agent | yes | yes | mixed | pending verification | Code execution success on greenfield GIS analysis. SA (single-agent ReAct) dominates DA (Plan-Execute-Replan) for strong models: DeepSeek-V3.2 96% SA vs 32% DA. Error recovery adds approximately 8 percentage points. | Production deployment, guardrail compliance, migration nuance, over-engineering, documentation quality, cloud-platform correctness, data-volume awareness. |
| GeoBenchX | 202 | model+tools | yes | no | mixed | pending verification | Geospatial tool-use and agent task completion with reference-solution comparison. | Reference solutions are tool traces, not final numeric or geospatial gold outputs. Does not prove production deployment, guardrail compliance, or deterministic output verification. |
| GEO-Bench-2 | 17 | model | yes | yes | mixed | yes | EO foundation-model downstream task quality across 17 datasets spanning multiple sensors. | Agent or tool-use execution. Does not test code generation, workflow planning, or artifact production. |
| GEO-Bench-2 Leaderboard | pending verification | model | yes | yes | mixed | pending verification | Aggregated public model-performance registry for EO foundation models. | Not a data source or golden task. Depends on GEO-Bench-2 official datasets and TerraTorch or iterate workflow. |
| GEOBench-VLM | pending verification | model | yes | yes | mixed | pending verification | Geospatial VLM multiple-choice question-answering accuracy on remote-sensing imagery. | Not a primary spatial verifier. MCQ accuracy does not prove code execution, artifact production, or workflow correctness. |
| GeoMMBench / GeoMMAgent | 1053 | model | yes | yes | mixed | yes | Expert multimodal geoscience and remote-sensing MCQ accuracy across six sensor modalities and four geoscience disciplines. CVPR 2026 Highlight. | Closed-form MCQ does not prove execution. A model scoring 90% may still fail to produce a cloud-free NDVI composite or execute a spatial workflow. |
| GABench / GeoAgentBench | 53 | complete-system | yes | no | mixed | pending verification | Tool-use execution, parameter inference, and map-product verification. GPT-4o approximately 72%, Claude 3.5 Sonnet approximately 68%; chained 4+ tool tasks drop to approximately 45%. PEA measures parameter-level accuracy within tolerance. | Production deployment, guardrails, cloud-platform correctness, data-volume awareness. |
| MapQA | 10421 | model | yes | yes | vector | yes | Spatial reasoning gap: GPT-4 64.2% overall, drops to 41.3% on multi-hop (3+ steps). Tool augmentation adds +17.5pp (81.7%). Routing: GPT-4 22.4% vs tool-augmented 76.8%. Tools add huge lift. | Code execution or artifact correctness. Does not test workflow generation, spatial data processing, or map production. |
| ThinkGeo | pending verification | model+tools | yes | yes | mixed | pending verification | Tool-augmented remote sensing benchmark with step-level and final correctness scoring. | Source custody, artifact lineage, or production acceptance states. Benchmark scoring, not a production-readiness contract. |
| Earth-Bench / Earth-Agent | pending verification | model+tools | yes | yes | mixed | pending verification | Trajectory and final outcome evaluation for spatiotemporal reasoning using MCP-style EO tools. | Product proof objects for user-produced spatial work. Research framework, not a production verification contract. |
| OpenEarth-Bench / OpenEarthAgent | pending verification | model+tools | yes | yes | mixed | pending verification | Unified tool registry with structured tool calls and GIS, spectral, and GeoTIFF tool trajectories. | Backend-owned verification records and projection readiness. Training or evaluation framework, not a production verification contract. |
| TerraBench | pending verification | pending verification | yes | pending verification | mixed | pending verification | pending verification | pending verification |
| RSRCC | 126000 | model | yes | yes | optical | pending verification | Fine-grained semantic reasoning about localised changes in remote sensing image pairs. Top models achieve approximately 60-65% on fine-grained questions versus 80%+ on global binary questions. The difficulty gap is largest for spectrally similar changes. | Code execution, workflow generation, or artifact production. Does not test agent tool-use or production deployment. |
| UrbanSARFloods | 8879 | model | yes | yes | SAR | pending verification | SAR flood detection is hard in urban environments: best baselines achieve 0.52-0.65 F1 on urban chips versus 0.78-0.85 on open-area chips. Imbalanced class distribution (under 5% flooded pixels in typical scenes) is the dominant challenge. | Agent execution, workflow generation, or production deployment. Does not test code generation or tool use. |
| GeoAI Agency Primitives | pending verification | agent | yes | yes | mixed | pending verification | Agents implementing all nine primitives improve analyst productivity by 60-80%. Highest-value primitives: action planning (40% gain), tool composition (25%), geo-referenced memory (15%). Uncertainty quantification is rarely implemented but expected in high-stakes domains. | Code execution correctness, artifact production, or spatial verification. Productivity gain does not equal output correctness. |
| Axis Migration Benchmark (v5/v6) | 8 | model | yes | yes | mixed | yes | v5: production deployment, guardrail compliance, migration nuance, over-engineering, documentation quality. v6: code execution success (SA 96%, SA_RETRY 100%, PA 100%, FULL 100%). Confirmed GISclaw SA greater than DA finding: SA produces same quality as FULL for 10-19x fewer LLM calls. | Breadth: 1 workflow (v5) or 3 cases (v6) versus 50 tasks in GeoAnalystBench. No repeated trials. No gold-standard output comparison (v6). Not generalisable to non-migration tasks. |
| GeoWorkBench | 30 | complete-system | yes | no | mixed | yes | Planned: production-readiness including CRS correctness, guardrail compliance, false-success resistance, correct refusal, recovery after failure, reproducibility, cost per accepted task, and deployment suitability. | Not yet built. v0.1 spec drafted with 30 tasks but reference values pending. Cannot prove anything until tasks are run with pinned data, deterministic checks, and repeated trials. |
Source notes
GeoAnalystBench / GISclaw aka GeoAnalystBench, GISclaw
Evaluates: agent · Tasks: 50 · Modality: mixed
References: arxiv.org/abs/2603.26845 · github.com/GeoDS/GeoAnalystBench · github.com/GeoDS/GeoAnalystBench/blob/master/dat
Ground truth / verifier: expert-designed reference Python solution per task; expected outputs (PNG/GeoJSON) / L1 API F1, L2 reasoning similarity, L3 output verification (vector geometry diff via shapely, raster pixel diff via numpy, tabular value diff via pandas). Binary pass/fail.
Scoring: L1 API F1,L2 reasoning similarity,L3 output verification
Licence: data=pending verification · code=pending verification (open-source repository)
Contamination: tasks are expert-designed and public via CSV; contamination possible if models trained on the benchmark data · Reproducibility: high (GeoAnalystBench is public with data and reference scripts)
What it can show: Code execution success on greenfield GIS analysis. SA (single-agent ReAct) dominates DA (Plan-Execute-Replan) for strong models: DeepSeek-V3.2 96% SA vs 32% DA. Error recovery adds approximately 8 percentage points.
What it cannot show: Production deployment, guardrail compliance, migration nuance, over-engineering, documentation quality, cloud-platform correctness, data-volume awareness.
GeoBenchX aka GeoBenchX
Evaluates: model+tools · Tasks: 202 · Modality: mixed
References: arxiv.org/abs/2503.18129 · github.com/solirinai/geobenchx · Google Drive data bundle (linked from repository
Ground truth / verifier: reference solutions (tool traces, not final numeric or geospatial gold outputs) / LLM-as-judge plus reference-solution comparison
Scoring: LLM judge score,reference-solution comparison
Licence: data=pending verification · code=pending verification
Contamination: reference solutions are tool traces, not final numeric or geospatial gold outputs; LLM-judge scoring is non-deterministic · Reproducibility: medium (repository public, data via Google Drive bundle)
What it can show: Geospatial tool-use and agent task completion with reference-solution comparison.
What it cannot show: Reference solutions are tool traces, not final numeric or geospatial gold outputs. Does not prove production deployment, guardrail compliance, or deterministic output verification.
GEO-Bench-2 aka GEO-Bench-2
Evaluates: model · Tasks: 17 · Modality: mixed
References: pending verification · github.com/The-AI-Alliance/GEO-Bench-2 · Official Hugging Face dataset versions with labe · github.com/The-AI-Alliance/GEO-Bench-2-Leaderboa
Ground truth / verifier: labels and targets in official Hugging Face dataset versions / accuracy, F1, Jaccard or IoU, RMSE depending on dataset
Scoring: accuracy,F1,Jaccard/IoU,RMSE
Licence: data=varies by dataset (Hugging Face) · code=pending verification
Contamination: public datasets; contamination possible if foundation models trained on them · Reproducibility: high (Hugging Face datasets with pinned labels and targets)
What it can show: EO foundation-model downstream task quality across 17 datasets spanning multiple sensors.
What it cannot show: Agent or tool-use execution. Does not test code generation, workflow planning, or artifact production.
GEO-Bench-2 Leaderboard aka GEO-Bench-2 Leaderboard
Evaluates: model · Tasks: pending verification · Modality: mixed
References: github.com/The-AI-Alliance/GEO-Bench-2-Leaderboa · github.com/The-AI-Alliance/GEO-Bench-2-Leaderboa
Ground truth / verifier: depends on underlying GEO-Bench-2 datasets / normalised leaderboard aggregation over submitted metrics
Scoring: submitted metrics
Licence: data=pending verification · code=pending verification
Contamination: depends on underlying GEO-Bench-2 datasets and TerraTorch or iterate workflow · Reproducibility: medium (depends on submission integrity and upstream dataset versioning)
What it can show: Aggregated public model-performance registry for EO foundation models.
What it cannot show: Not a data source or golden task. Depends on GEO-Bench-2 official datasets and TerraTorch or iterate workflow.
GEOBench-VLM aka GEOBench-VLM, geo-bench-vlm
Evaluates: model · Tasks: pending verification · Modality: mixed
References: pending verification · github.com/the-ai-alliance/geo-bench-vlm · Hugging Face dataset with manually verified inst
Ground truth / verifier: manually verified instructions and answer options / MCQ accuracy
Scoring: MCQ accuracy
Licence: data=pending verification · code=pending verification
Contamination: public Hugging Face dataset; contamination possible if VLMs trained on it · Reproducibility: medium (Hugging Face dataset with inference scripts; GPU, model, and provider posture must be explicit)
What it can show: Geospatial VLM multiple-choice question-answering accuracy on remote-sensing imagery.
What it cannot show: Not a primary spatial verifier. MCQ accuracy does not prove code execution, artifact production, or workflow correctness.
GeoMMBench / GeoMMAgent aka GeoMMBench, GeoMMAgent, GeoMM-AGI
Evaluates: model · Tasks: 1053 · Modality: mixed
References: arxiv.org/abs/2604.08896 · github.com/Shihao-Cheng/GeoMMAgent · huggingface.co/datasets/AR-X/GeoMMBench
Ground truth / verifier: expert-derived multiple-choice questions with verified answers / MCQ accuracy; optional self-evaluation dimensions (logic, spatial reasoning, domain validity, accuracy)
Scoring: MCQ accuracy,optional self-eval: logic, spatial reasoning, domain validity, accuracy
Licence: data=CC BY 4.0 (dataset) · code=Apache 2.0 (GeoMMAgent framework)
Contamination: public Hugging Face dataset; 37-question public validation split is small and may yield high variance in external reproductions · Reproducibility: medium (dataset public on HuggingFace; per-model accuracy in paper PDF; arXiv identifier confirmed as 2604.08896)
What it can show: Expert multimodal geoscience and remote-sensing MCQ accuracy across six sensor modalities and four geoscience disciplines. CVPR 2026 Highlight.
What it cannot show: Closed-form MCQ does not prove execution. A model scoring 90% may still fail to produce a cloud-free NDVI composite or execute a spatial workflow.
GABench / GeoAgentBench aka GABench, GeoAgentBench, GeoAgentBench
Evaluates: complete-system · Tasks: 53 · Modality: mixed
References: arxiv.org/abs/2604.13888 · github.com/geox-lab/GABench · github.com/geox-lab/GABench (Git LFS required fo
Ground truth / verifier: generated and verified GT maps and data products; ground-truth tool call sequences per task / TAO (Tool Accuracy Overall), TIO (Tool Invocation Overall), TEM (Tool Execution Metric), PEA (Parameter Execution Accuracy) with last-attempt alignment and file-existence checks; VLM-as-judge map comparison; execution-efficiency metrics
Scoring: TAO,TIO,TEM,PEA,VLM map comparison,execution efficiency
Licence: data=pending verification · code=pending verification
Contamination: public repo uses Git LFS; normal clone without LFS leaves core task and data files as pointers, blocking reproduction · Reproducibility: medium (repository public but LFS assets required; task CSV, GT maps, data, dependency lock, output isolation, and VLM judge posture must be imported)
What it can show: Tool-use execution, parameter inference, and map-product verification. GPT-4o approximately 72%, Claude 3.5 Sonnet approximately 68%; chained 4+ tool tasks drop to approximately 45%. PEA measures parameter-level accuracy within tolerance.
What it cannot show: Production deployment, guardrails, cloud-platform correctness, data-volume awareness.
MapQA aka MapQA
Evaluates: model · Tasks: 10421 · Modality: vector
References: arxiv.org/abs/2503.07871 · pending verification · pending verification
Ground truth / verifier: derived from OSM geometries and POI metadata with known spatial relationships / accuracy (exact match against ground-truth answer)
Scoring: accuracy
Licence: data=pending verification (underlying OSM data is ODbL) · code=pending verification
Contamination: questions derived from public OSM data; contamination possible if models trained on similar QA pairs · Reproducibility: medium (methodology public; dataset availability pending verification)
What it can show: Spatial reasoning gap: GPT-4 64.2% overall, drops to 41.3% on multi-hop (3+ steps). Tool augmentation adds +17.5pp (81.7%). Routing: GPT-4 22.4% vs tool-augmented 76.8%. Tools add huge lift.
What it cannot show: Code execution or artifact correctness. Does not test workflow generation, spatial data processing, or map production.
ThinkGeo aka ThinkGeo
Evaluates: model+tools · Tasks: pending verification · Modality: mixed
References: pending verification · pending verification · pending verification
Ground truth / verifier: pending verification / step correctness and final correctness
Scoring: step correctness,final correctness
Licence: data=pending verification · code=pending verification
Contamination: pending verification · Reproducibility: pending verification
What it can show: Tool-augmented remote sensing benchmark with step-level and final correctness scoring.
What it cannot show: Source custody, artifact lineage, or production acceptance states. Benchmark scoring, not a production-readiness contract.
Earth-Bench / Earth-Agent aka Earth-Bench, Earth-Agent
Evaluates: model+tools · Tasks: pending verification · Modality: mixed
References: pending verification · pending verification · pending verification
Ground truth / verifier: pending verification / trajectory and final outcome evaluation for spatiotemporal reasoning
Scoring: trajectory evaluation,final outcome evaluation
Licence: data=pending verification · code=pending verification
Contamination: pending verification · Reproducibility: pending verification (described as a research framework, not a product)
What it can show: Trajectory and final outcome evaluation for spatiotemporal reasoning using MCP-style EO tools.
What it cannot show: Product proof objects for user-produced spatial work. Research framework, not a production verification contract.
OpenEarth-Bench / OpenEarthAgent aka OpenEarth-Bench, OpenEarthAgent, OpenEarth Agent
Evaluates: model+tools · Tasks: pending verification · Modality: mixed
References: pending verification · pending verification · pending verification
Ground truth / verifier: pending verification / structured tool-call and tool-trajectory evaluation
Scoring: tool-call correctness,tool-trajectory evaluation
Licence: data=pending verification · code=pending verification
Contamination: pending verification · Reproducibility: pending verification (described as a training or evaluation framework, not a product)
What it can show: Unified tool registry with structured tool calls and GIS, spectral, and GeoTIFF tool trajectories.
What it cannot show: Backend-owned verification records and projection readiness. Training or evaluation framework, not a production verification contract.
TerraBench aka TerraBench
Evaluates: pending verification · Tasks: pending verification · Modality: mixed
References: pending verification · pending verification · pending verification
Ground truth / verifier: pending verification / pending verification
Scoring: pending verification
Licence: data=pending verification · code=pending verification
Contamination: pending verification · Reproducibility: pending verification
What it can show: pending verification
What it cannot show: pending verification
RSRCC aka RSRCC, Remote Sensing Regional Change Comprehension
Evaluates: model · Tasks: 126000 · Modality: optical
References: arxiv.org/abs/2604.20623 · pending verification · pending verification
Ground truth / verifier: hierarchical semi-supervised curation pipeline with Best-of-N ranking as final ambiguity-resolution stage / accuracy on fine-grained questions versus global binary questions
Scoring: fine-grained change identification accuracy,global binary change detection accuracy
Licence: data=pending verification · code=pending verification
Contamination: large-scale public benchmark; contamination possible if MLLMs trained on it · Reproducibility: medium (methodology public; dataset and code availability pending verification)
What it can show: Fine-grained semantic reasoning about localised changes in remote sensing image pairs. Top models achieve approximately 60-65% on fine-grained questions versus 80%+ on global binary questions. The difficulty gap is largest for spectrally similar changes.
What it cannot show: Code execution, workflow generation, or artifact production. Does not test agent tool-use or production deployment.
UrbanSARFloods aka UrbanSARFloods, Sentinel-1 Flood Benchmark
Evaluates: model · Tasks: 8879 · Modality: SAR
References: arxiv.org/abs/2406.04111 · pending verification · pending verification
Ground truth / verifier: pixel-level flood or no-flood labels stratified by land-cover class and continent / F1 score per chip, stratified by land-cover class and urban versus open-area
Scoring: F1 score,urban chip F1,open-area chip F1,class-imbalance robustness
Licence: data=pending verification · code=pending verification
Contamination: large-scale benchmark; contamination possible if flood detection models trained on it · Reproducibility: medium (methodology public; dataset availability pending verification)
What it can show: SAR flood detection is hard in urban environments: best baselines achieve 0.52-0.65 F1 on urban chips versus 0.78-0.85 on open-area chips. Imbalanced class distribution (under 5% flooded pixels in typical scenes) is the dominant challenge.
What it cannot show: Agent execution, workflow generation, or production deployment. Does not test code generation or tool use.
GeoAI Agency Primitives aka GeoAI Agency Primitives, Agency Primitives Benchmark
Evaluates: agent · Tasks: pending verification · Modality: mixed
References: arxiv.org/abs/2604.01869 · pending verification · pending verification
Ground truth / verifier: analyst productivity baseline comparison across eight task types / productivity gain measurement (percentage improvement versus unassisted baseline)
Scoring: analyst productivity,per-primitive contribution (action planning, tool composition, geo-referenced memory, perception, spatial reasoning, uncertainty quantification)
Licence: data=pending verification · code=pending verification
Contamination: pending verification · Reproducibility: pending verification
What it can show: Agents implementing all nine primitives improve analyst productivity by 60-80%. Highest-value primitives: action planning (40% gain), tool composition (25%), geo-referenced memory (15%). Uncertainty quantification is rarely implemented but expected in high-stakes domains.
What it cannot show: Code execution correctness, artifact production, or spatial verification. Productivity gain does not equal output correctness.
Axis Migration Benchmark (v5/v6) aka Axis v5 Benchmark, Axis v6 Benchmark, Axis Migration Benchmark
Evaluates: model · Tasks: 8 · Modality: mixed
References: internal (tasks/benchmark/v6/) · internal (ExpDis Japan ArcPy workflow, 4000+ LOC
Ground truth / verifier: v5: 7-dimension rubric (Structural, Domain, Errors, Guardrails, Nuance, Docs, Over-engineering); v6: AST parse success, guardrail scan pass, arcpy absence, DRY_RUN guard presence, pathlib usage check / v5: rubric scoring (0-10 per dimension, 70 total); v6: automated code verification (AST parse, guardrail scan, import check, DRY_RUN, pathlib)
Scoring: Structural,Domain,Errors,Guardrails,Nuance,Docs,Over-engineering (v5); AST parse, guardrail scan, arcpy check, DRY_RUN, pathlib (v6)
Licence: data=internal (proprietary) · code=internal (proprietary)
Contamination: proprietary workflow; no public contamination risk · Reproducibility: medium (evaluator is deterministic; workflow is proprietary; v6 results on real Bedrock APIs inside staging container on Hetzner)
What it can show: v5: production deployment, guardrail compliance, migration nuance, over-engineering, documentation quality. v6: code execution success (SA 96%, SA_RETRY 100%, PA 100%, FULL 100%). Confirmed GISclaw SA greater than DA finding: SA produces same quality as FULL for 10-19x fewer LLM calls.
What it cannot show: Breadth: 1 workflow (v5) or 3 cases (v6) versus 50 tasks in GeoAnalystBench. No repeated trials. No gold-standard output comparison (v6). Not generalisable to non-migration tasks.
GeoWorkBench aka GeoWorkBench, Bearlin v1
Evaluates: complete-system · Tasks: 30 · Modality: mixed
References: internal (planned) · internal (planned)
Ground truth / verifier: per-task quantitative criteria (value constraints) and expert acceptance criteria; reference values from Copernicus, UNOSAT, UrbanSARFloods, USGS (currently TBD) / 4 binary axes (A1 artefact present, A2 values plausible, A3 code executed, A4 expert agrees) plus 2 continuous metrics (time, cost); deterministic verification first, LLM judge as side metric only
Scoring: accepted task completion,parameter execution accuracy,tool-use correctness,artifact existence,spatial correctness,numeric correctness,policy and guardrail compliance,evidence and provenance quality,failure honesty,recovery rate,cost and latency,tool-call and retry efficiency,reproducibility,production readiness
Licence: data=internal (proprietary tasks; underlying data licences separate) · code=internal (proprietary)
Contamination: private task set with 24 public and 6 private hold-out tasks to prevent overfit · Reproducibility: planned (v0.1 spec drafted; reference values TBD; not yet run)
What it can show: Planned: production-readiness including CRS correctness, guardrail compliance, false-success resistance, correct refusal, recovery after failure, reproducibility, cost per accepted task, and deployment suitability.
What it cannot show: Not yet built. v0.1 spec drafted with 30 tasks but reference values pending. Cannot prove anything until tasks are run with pinned data, deterministic checks, and repeated trials.
How to read the registry
Use the JSON endpointVerification standard
Deterministic checks and inspectable artifacts offer stronger evidence than self-reported completion. An LLM judge is not treated as sole proof of spatial correctness.
Registry status
This is a maintained research index, not a ranking, product claim or certification programme. Inclusion does not imply endorsement.
Gaps across the field
- Whether an artifact is correct against spatial ground truth, rather than merely present.
- False-success classes such as unsafe buffering, AOI drift, raster misalignment, and provenance breaks.
- Cost, latency, recovery, and reproducibility reported alongside completion.
- Long-running workflows, customer-environment constraints, and governed acceptance.