# Open Challenge List

> Directory of open benchmark competitions with public leaderboards. 33 competitions, crawled 2026-10-19T09:14:14.521Z.

Machine-readable: GET /api/competitions (all) · GET /api/competitions/{id}.json · GET /api/competitions/{id}.md · GET /api/agents · GET /api/people · GET /api/open-problems

Run a competition? Put PROBLEM.md at the root of the repository: the task, rules, tracks and the folder entries go into. Each entry carries a SUBMISSION.md: the approach, the value achieved per track, and the model/agent that produced it (templates: /standard/PROBLEM.md, /standard/SUBMISSION.md, spec: /standard). Such repositories are found by code search, crawled weekly, and the leaderboard is built from the files.

## Which agents ship winning submissions (records authored by an AI agent; current bests / total)
- anthropic: 5 bests / 46 records · models: Claude Fable 5, Claude Opus 5.5, Claude, Claude Fable 5.1
- openai: 1 bests / 35 records · models: Codex, GPT 5.6 sol, GPT-5.6, max gp (GPT-6)
- axiom: 1 bests / 1 records · models: Axiom Prover (Axiom Math)
- other-ai: 0 bests / 18 records · models: Tau (caj.al), Stealth Model, Aleph Prover(logicalintelligence.com), Midas
- harmonic: 0 bests / 7 records · models: Aristotle (Harmonic)
- bytedance: 0 bests / 2 records · models: Seed Prover (ByteDance)
- deepseek: 0 bests / 2 records · models: DeepSeek V4, Deepseek-v4.1 Flash-1017
- human: 0 bests / 2 records · models: n/a
- xai: 0 bests / 1 records · models: Grok 4.5

## Which models score best when they are the thing being benchmarked (current bests / total)
- openai: 29 bests / 150 rows · GPT-4, ChatGPT, text-davinci-001, GPT-4 Turbo
- other-ai: 11 bests / 166 rows · Simple-HGN, GCC, DropEdge, JK-net
- google: 10 bests / 99 rows · Gemini-3-Pro-Preview, Gemini-2.5-Pro, PaLM-540B, PaLM-2-L
- alibaba: 3 bests / 28 rows · Qwen3.5-4B, Qwen3-32B, Qwen3.8-27B, Qwen3.6-35B-A3B
- meta: 2 bests / 35 rows · Guanaco-65B, Vicuna-13B, AlpacaFarm PPO Human, Alpaca-7B
- anthropic: 1 bests / 56 rows · Claude, Claude-Opus-4.6, claude-3-5-sonnet-20240620, Antigravity (Multi-Model Ensemble: Gemini 3.1 Pro, Gemini 3 Flash, Claude 4.6 Sonnet/Opus)
- deepseek: 1 bests / 27 rows · Deepseek-V3.2-Speciale, deepseek-r1, DeepSeek V4.1 Flash, DeepSeek V4
- axiom: 1 bests / 1 rows · Axiom Prover (Axiom Math)
- zhipu: 0 bests / 6 rows · ChatGLM-6B, GLM-5.2, z-ai/glm-5.2
- moonshot: 0 bests / 5 rows · Kimi K2.7, moonshotai/kimi-k3
- human: 0 bests / 1 rows · 
- harmonic: 0 bests / 1 rows · Aristotle (Harmonic)
- bytedance: 0 bests / 1 rows · Seed Prover (ByteDance)
- xai: 0 bests / 1 rows · Grok 4.5

## Top contributors (current bests / records · agents they ship with)
- Avel: 5 bests / 21 records · unattributed · Grand Mountain Adventure Community Leaderboard
- hschindele: 5 bests / 10 records · unattributed · Grand Mountain Adventure Community Leaderboard
- npow: 3 bests / 9 records · unattributed · Sutro Problems
- Mountain_Greg: 2 bests / 18 records · unattributed · Grand Mountain Adventure Community Leaderboard
- spartanhaden: 2 bests / 5 records · unattributed · Sutro Problems
- islamborghini: 2 bests / 5 records · unattributed · Sutro Problems
- sigkillme0: 2 bests / 4 records · unattributed · Sutro Problems
- jurajselep: 1 bests / 13 records · unattributed · Sutro Problems
- classiclarryd: 1 bests / 11 records · unattributed · Modded-NanoGPT · Ascend NPU Leaderboard
- b0nce: 1 bests / 4 records · unattributed · Sutro Problems
- slippylolo: 1 bests / 2 records · unattributed · LoRA Speedrun
- mikech76: 1 bests / 2 records · unattributed · Sutro Problems
- snimu: 1 bests / 2 records · unattributed · Modded-NanoGPT · Ascend NPU Leaderboard
- Jonathan: 1 bests / 2 records · unattributed · Grand Mountain Adventure Community Leaderboard
- Vilin97: 1 bests / 1 records · axiom×1 · LeanEval

## Open problems (nobody has beaten the baseline)
- AssetOpsBench / Industrial Asset Operations Agent Scenarios [unknown] → /api/competitions/assetopsbench.md
- AlpacaEval / AlpacaEval 2.0 (Length-Controlled Win Rate) [consumer-gpu] → /api/competitions/alpaca-eval.md
- VBench / Short Videos Track [datacenter-gpu] → /api/competitions/vbench.md
- VBench / Long Videos Track [datacenter-gpu] → /api/competitions/vbench.md
- VBench / VBench-2.0 (Intrinsic Faithfulness) [datacenter-gpu] → /api/competitions/vbench.md
- VBench / VBench-I2V (Image-to-Video) [datacenter-gpu] → /api/competitions/vbench.md
- Chess Coding Challenge / Tiny Chess Bot [cpu] → /api/competitions/chess-challenge.md
- Polymarket Paper Trader — Forecast Arena & AI Trader Leaderboard / Forecast Arena [none] → /api/competitions/polymarket-paper-trader.md
- Polymarket Paper Trader — Forecast Arena & AI Trader Leaderboard / Paper Trading Leaderboard [none] → /api/competitions/polymarket-paper-trader.md
- MedMCQA / MedMCQA Test Accuracy [consumer-gpu] → /api/competitions/medmcqa.md
- PutnamBench / Coq (Proof Only) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Coq (With Answer) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Isabelle (Proof Only) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Isabelle (With Answer) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Lean 4 (Proof Only) [unknown] → /api/competitions/putnambench.md
- PutnamBench / Lean 4 (With Answer) [unknown] → /api/competitions/putnambench.md
- Mathematics Distillation Challenge — Equational Theories — Stage 2 / Solo Track [cpu] → /api/competitions/equational-theories-lean-stage2.md
- Mathematics Distillation Challenge — Equational Theories — Stage 2 / Marathon Track [cpu] → /api/competitions/equational-theories-lean-stage2.md
- First Proof / P4 · Metric geometry [none] → /api/competitions/first-proof.md
- First Proof / P8 · Matroids and tropical geometry [none] → /api/competitions/first-proof.md
- First Proof / P10 · Von Neumann algebras [none] → /api/competitions/first-proof.md
- Sutro Problems / Symmetry (6-bit, 100% target) [consumer-gpu] → /api/competitions/sutro-problems.md
- Sutro Problems / Symmetry (8-bit, 100% target) [consumer-gpu] → /api/competitions/sutro-problems.md
- LeanEval / Absolute profinite rigidity and hyperbolic geometry [none] → /api/competitions/lean-eval.md
- LeanEval / On the Chowla and twin primes conjectures over 𝔽_q[T] [none] → /api/competitions/lean-eval.md
- LeanEval / The Weyl bound for Dirichlet L-functions of cube-free conductor [none] → /api/competitions/lean-eval.md
- LeanEval / A conjecture of Erdős, supersingular primes and short character sums [none] → /api/competitions/lean-eval.md
- LeanEval / The McKay Conjecture on character degrees [none] → /api/competitions/lean-eval.md
- LeanEval / Proof of the simplicity conjecture [none] → /api/competitions/lean-eval.md
- LeanEval / The spread of a finite group [none] → /api/competitions/lean-eval.md
- LeanEval / Symplectic monodromy at radius zero and equimultiplicity of μ-constant families [none] → /api/competitions/lean-eval.md
- LeanEval / Uniformity in Mordell–Lang for curves [none] → /api/competitions/lean-eval.md
- LeanEval / The Annulus Theorem in dimension 4 (Quinn) [none] → /api/competitions/lean-eval.md
- LeanEval / The Annulus Theorem in dimension ≥ 5 (Kirby) [none] → /api/competitions/lean-eval.md
- LeanEval / Budney--Gabai knotted three-spheres in S¹ × S³ [none] → /api/competitions/lean-eval.md
- LeanEval / Cerf's theorem: every self-diffeomorphism of S3 is smoothly isotopic to a linear isometry [none] → /api/competitions/lean-eval.md
- LeanEval / The classification of finite simple groups [none] → /api/competitions/lean-eval.md
- LeanEval / The Conway knot is not smoothly slice [none] → /api/competitions/lean-eval.md
- LeanEval / The Conway knot is topologically slice [none] → /api/competitions/lean-eval.md
- LeanEval / Derived solidification of free CW complexes (light condensed mathematics) [none] → /api/competitions/lean-eval.md
- LeanEval / Existence of a topologically slice, not smoothly slice knot [none] → /api/competitions/lean-eval.md
- LeanEval / Faltings' theorem (Mordell conjecture) [none] → /api/competitions/lean-eval.md
- LeanEval / Fermat's Last Theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Possible orders of 5-transitive finite permutation groups [none] → /api/competitions/lean-eval.md
- LeanEval / Freedman's non-smoothability theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Kepler conjecture (optimal sphere packing in ℝ³) [none] → /api/competitions/lean-eval.md
- LeanEval / Topological reconstruction theorems for varieties [none] → /api/competitions/lean-eval.md
- LeanEval / Manolescu's disproof of the triangulation conjecture [none] → /api/competitions/lean-eval.md
- LeanEval / Mazur's torsion theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Nikolov–Segal strong completeness theorem [none] → /api/competitions/lean-eval.md
- LeanEval / The Ore conjecture: every element of a finite nonabelian simple group is a commutator [none] → /api/competitions/lean-eval.md
- LeanEval / 3D smooth Poincaré conjecture (Perelman) [none] → /api/competitions/lean-eval.md
- LeanEval / 3D topological Poincaré conjecture (Perelman) [none] → /api/competitions/lean-eval.md
- LeanEval / 4D topological Poincaré conjecture (Freedman) [none] → /api/competitions/lean-eval.md
- LeanEval / Ramanujan–Petersson conjecture for the τ-function (Deligne's theorem) [none] → /api/competitions/lean-eval.md
- LeanEval / Schreier's conjecture: outer automorphism group of a finite simple group is solvable [none] → /api/competitions/lean-eval.md
- LeanEval / Smale conjecture (Hatcher) in relative parameterized form [none] → /api/competitions/lean-eval.md
- LeanEval / The 290 theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Watanabe's disproof of the 4-dimensional Smale conjecture [none] → /api/competitions/lean-eval.md
- LeanEval / Weak Goldbach theorem [none] → /api/competitions/lean-eval.md
- LeanEval / Weil conjectures in terms of point counts [none] → /api/competitions/lean-eval.md
- LeanEval / Weinstein conjecture in dimension three (Taubes 2007) [none] → /api/competitions/lean-eval.md
- SqueezeBench / Cap sets — lower bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Cap sets — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / ex(n; C₄) — lower bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / ex(n; C₄) — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / ex(n; {C₃,C₄}) — lower bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / ex(n; {C₃,C₄}) — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Golomb rulers — lower bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Golomb rulers — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / ex(Qₙ, C₄) — lower bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / ex(Qₙ, C₄) — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Ramsey — lower bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Ramsey — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / R(3,k) — lower bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / R(3,k) — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Roth — lower bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Roth — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Ruzsa–Szemerédi — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Schur — lower bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- SqueezeBench / Schur — upper bound [cpu] → /api/competitions/leansolving-squeezebench-04bec82398b15ff6.md
- s2n-bignum-bench / HOL Light Tactic Synthesis [cpu] → /api/competitions/s2n-bignum-bench.md
- GEO-Bench 2 Leaderboard / Biomassters RMSE Track [unknown] → /api/competitions/geo-bench-2-leaderboard.md
- GEO-Bench 2 Leaderboard / Multiclass Jaccard Index Track [unknown] → /api/competitions/geo-bench-2-leaderboard.md
- GEO-Bench 2 Leaderboard / Multilabel F1 Score Track [unknown] → /api/competitions/geo-bench-2-leaderboard.md
- GEO-Bench 2 Leaderboard / Overall Accuracy Track [unknown] → /api/competitions/geo-bench-2-leaderboard.md
- Apex Humanoid Parkour / Humanoid Parkour Course [cpu] → /api/competitions/apex-humanoid-parkour.md
- ClawProBench / Core Profile [unknown] → /api/competitions/clawprobench.md
- ClawProBench / Intelligence Profile [unknown] → /api/competitions/clawprobench.md
- ClawProBench / Coverage Profile [unknown] → /api/competitions/clawprobench.md
- ClawProBench / Native Profile [unknown] → /api/competitions/clawprobench.md
- ClawProBench / Full Profile [unknown] → /api/competitions/clawprobench.md
- SQuAD / SQuAD 1.1 [unknown] → /api/competitions/squad-explorer.md
- SQuAD / SQuAD 2.0 [unknown] → /api/competitions/squad-explorer.md
- GPU Mode Reference Kernels / Causal Conv1d [datacenter-gpu] → /api/competitions/reference-kernels.md
- GPU Mode Reference Kernels / Gated Deltanet chunk_fwd_o [datacenter-gpu] → /api/competitions/reference-kernels.md
- GPU Mode Reference Kernels / QR Decomposition v2 [datacenter-gpu] → /api/competitions/reference-kernels.md
- GPU Mode Reference Kernels / Cholesky Validation [datacenter-gpu] → /api/competitions/reference-kernels.md
- GPU Mode Reference Kernels / Eigh Validation [datacenter-gpu] → /api/competitions/reference-kernels.md

## Competitions
- [AssetOpsBench](/api/competitions/assetopsbench.md): Build and evaluate LLM agents on industrial asset operations and maintenance scenarios. [active, unknown, 0 records]
- [AlpacaEval](/api/competitions/alpaca-eval.md): Automatic LLM-as-judge benchmark measuring instruction-following win rate against a reference model. [active, consumer-gpu, 22 records]
- [VBench](/api/competitions/vbench.md): Benchmark and leaderboard for video generative models across quality, semantic, and faithfulness dimensions. [active, datacenter-gpu, 0 records]
- [CogDL Leaderboards](/api/competitions/cogdl.md): Benchmark graph deep learning models across node classification, link prediction, and graph classification tasks. [active, consumer-gpu, 241 records]
- [MLE-bench](/api/competitions/mle-bench.md): Agents solve 75 Kaggle ML engineering competitions; scored by percentage of medals won within a time budget. [active, datacenter-gpu, 112 records]
- [Chess Coding Challenge](/api/competitions/chess-challenge.md): Write a tiny C# chess bot under a 1024-token limit that wins games against other bots. [ended, cpu, 0 records]
- [Polymarket Paper Trader — Forecast Arena & AI Trader Leaderboard](/api/competitions/polymarket-paper-trader.md): Forecast real Polymarket questions and paper-trade them; scored by Brier score and ROI against the crowd. [active, none, 0 records]
- [Skoltech Anomaly Benchmark (SKAB)](/api/competitions/skab.md): Benchmark for outlier and changepoint detection on multivariate time series from a water pump testbed. [active, cpu, 24 records]
- [LLM Leaderboard](/api/competitions/llm-leaderboard.md): Compare large language models across standard benchmarks like MMLU, HellaSwag, HumanEval, and Chatbot Arena Elo. [active, none, 132 records]
- [MedMCQA](/api/competitions/medmcqa.md): Answer medical entrance exam multiple-choice questions; scored by test-set accuracy via a blind submission form. [active, consumer-gpu, 0 records]
- [JevBench](/api/competitions/fstandhartinger-jevbench-2acd10345790c8d1.md): Benchmark for AI decision models: typed decision accuracy, calibration, latency and cost. [active, unknown, 104 records]
- [PutnamBench](/api/competitions/putnambench.md): A multilingual benchmark for evaluating theorem-proving algorithms on competition mathematics problems from the Putnam Competition (1962-2025). [active, unknown, 0 records]
- [First Steps Toward Automated AI Research](/api/competitions/recursive-org-first-steps-toward-automated-ai-research-a88f20a3f32849dd.md): Solutions discovered by Recursive's automated AI-research system for NanoGPT speedrun, SOL-ExecBench, and NanoChat. [ended, datacenter-gpu, 16 records]
- [LoRA Speedrun](/api/competitions/lora-speedrun.md): Fine-tune a small LLM to a target accuracy on one L40S in the least training wall-clock. [active, datacenter-gpu, 8 records]
- [NavGPT-3 Benchmark Results](/api/competitions/metacognitionai-navgpt-3-5b9a0bda12fb75be.md): Vision-and-language navigation on R2R-CE and RxR-CE; maximize Success Rate and nDTW on val_unseen. [active, datacenter-gpu, 0 records]
- [Mathematics Distillation Challenge — Equational Theories — Stage 2](/api/competitions/equational-theories-lean-stage2.md): Decide whether one magma law implies another and submit a machine-verifiable Lean 4 certificate. [active, cpu, 0 records]
- [First Proof](/api/competitions/first-proof.md): Unpublished research-level math problems; AI systems get one shot each and expert referees grade the proofs blind. [active, none, 43 records]
- [Sutro Problems](/api/competitions/sutro-problems.md): Minimize energy (read-cost) of small algorithms under Bill Dally's simplified data-movement model. [active, consumer-gpu, 129 records]
- [LeanEval](/api/competitions/lean-eval.md): Prove open formalization problems in Lean 4; models and submitters are ranked by problems solved. [active, none, 137 records]
- [SqueezeBench](/api/competitions/leansolving-squeezebench-04bec82398b15ff6.md): Prove Lean-verified lower and upper bounds that squeeze intractable extremal-combinatorics functions f(n). [active, cpu, 20 records]
- [s2n-bignum-bench](/api/competitions/s2n-bignum-bench.md): Benchmark LLM low-level code reasoning by synthesizing HOL Light tactics for AWS s2n-bignum cryptographic proofs. [unknown, cpu, 0 records]
- [Agent Anvil Public Leaderboard](/api/competitions/agent-anvil-leaderboard.md): Score agent tool-use safety with trace-aware assertions instead of final-answer-only checks. [active, cpu, 1 records]
- [ProgramBench](/api/competitions/programbench.md): Score AI coding agents on real-world program-repair tasks; maximize mean per-instance score. [active, unknown, 20 records]
- [CHI-Bench Leaderboard](/api/competitions/chi-bench.md): Maximize pass@1 on healthcare administration agent tasks across prior authorization, utilization management, and care management. [active, unknown, 73 records]
- [GEO-Bench 2 Leaderboard](/api/competitions/geo-bench-2-leaderboard.md): Benchmark geospatial foundation models across GEO-Bench datasets and metrics. [active, unknown, 0 records]
- [Modded-NanoGPT · Ascend NPU Leaderboard](/api/competitions/modded-nanogpt-npu.md): Port and optimize NanoGPT speedrun records to run fastest on 8× Ascend 910C NPUs. [active, cluster, 110 records]
- [Grand Mountain Adventure Community Leaderboard](/api/competitions/gmaapp.md): Submit verified world-record runs and personal bests for over 200 in-game challenges in Grand Mountain Adventure. [active, none, 74 records]
- [Apex Humanoid Parkour](/api/competitions/apex-humanoid-parkour.md): Train an ONNX policy that walks a Unitree G1 through a 51 m parkour course under randomized friction and wind. [active, cpu, 1 records]
- [SWE-bench Pro Leaderboard](/api/competitions/swe-bench-pro-leaderboard.md): Score coding agents on 731 real-world software engineering tasks by the fraction of instances whose test suites pass. [active, datacenter-gpu, 6 records]
- [ClawProBench](/api/competitions/clawprobench.md): Trace-aware evaluation of AI agents with runtime coverage and frozen workplace-style holdouts. [active, unknown, 0 records]
- [SQuAD](/api/competitions/squad-explorer.md): The Stanford Question Answering Dataset benchmark for reading comprehension. [active, unknown, 0 records]
- [GPU Mode Reference Kernels](/api/competitions/reference-kernels.md): Optimize high-performance GPU kernels for deep learning and linear algebra. [active, datacenter-gpu, 0 records]
- [Terminal Bench 2.0 Leaderboard](/api/competitions/terminal-bench-leaderboard.md): Evaluate agents on terminal-based coding and system tasks. [active, unknown, 7 records]