Scores from different versions, task subsets, harnesses and scoring rules are not directly comparable. How results are reviewed
Documented results
5 selected tests| View ReportCustom evaluation | Devstral-24B | MMLU57 subjects · first item per subject1 concurrent request | 80.7 %Accuracy · 46 / 57 · Exact-choice parser | — | — | 2 × R9700Tensor parallel · 32 GB VRAM/GPUConfiguration
|
|---|---|---|---|---|---|---|
| View ReportCustom evaluation | Qwen3.8-27B | MMLU57 subjects · first item per subject1 concurrent request | 84.21 %Accuracy · 48 / 57 · Exact-choice parser | — | — | 2 × R9700Tensor parallel · 32 GB VRAM/GPUConfiguration
|
| View ReportCustom evaluation | gemma-4-26B-A4B-it | MMLU57 subjects · first item per subject1 concurrent request | 77.19 %Accuracy · 44 / 57 · Exact-choice parser | — | — | 2 × R9700Tensor parallel · 32 GB VRAM/GPUConfiguration
|
| View ReportCustom evaluation | gemma-4-31B-it | MMLU57 subjects · first item per subject1 concurrent request | 91.23 %Accuracy · 52 / 57 · Exact-choice parser | — | — | 2 × R9700Tensor parallel · 32 GB VRAM/GPUConfiguration
|
| View ReportCustom evaluation | Qwen3-Coder-30B-A3B | MMLU57 subjects · first item per subject1 concurrent request | 85.96 %Accuracy · 49 / 57 · Exact-choice parser | — | — | 2 × R9700Tensor parallel · 32 GB VRAM/GPUConfiguration
|
Versions and harnesses
Versions: Not reported
Harnesses: scripts/eval/eval_and_chart.py
This is a selection across sources, not a ranking or a matched experiment. Reading benchmark reports
GPU benchmark pages: R9700