Scores from different versions, task subsets, harnesses and scoring rules are not directly comparable. How results are reviewed
Documented results
5 selected tests| View ReportCoding session | Qwen3.6-27BPartial evaluation | SWE-bench Verified MiniVerified Mini: 50 tasks (25 Django + 25 Sphinx) | 74 %LLM judge · 37 / 50 66 %Exit-zero rate · 33 / 50 · Container test exit | — | — | 2 × R9700Multi-GPU · split unknownConfiguration
|
|---|---|---|---|---|---|---|
| View ReportCoding session | Qwen3.6-27BPartial evaluation | SWE-bench Verified MiniVerified Mini: 50 tasks (25 Django + 25 Sphinx) | 70 %LLM judge · 35 / 50 58 %Exit-zero rate · 29 / 50 · Container test exit | — | — | 1 × R9700Single GPUConfiguration
|
| View ReportCoding session | Qwen3.6-27BPartial evaluation | SWE-bench Verified MiniVerified Mini: 50 tasks (25 Django + 25 Sphinx) | 72 %LLM judge · 36 / 50 62 %Exit-zero rate · 31 / 50 · Container test exit | — | — | 1 × R9700Single GPUConfiguration
|
| View ReportCoding session | Qwen3.6-35B-A3BPartial evaluation | SWE-bench Verified MiniVerified Mini: 50 tasks (25 Django + 25 Sphinx) | 66 %LLM judge · 33 / 50 56 %Exit-zero rate · 28 / 50 · Container test exit | — | — | 1 × R9700Single GPUConfiguration
|
| View ReportCoding session | gemma-4-26B-A4B-itPartial evaluation | SWE-bench Verified MiniVerified Mini: 50 tasks (25 Django + 25 Sphinx) | 50 %LLM judge · 25 / 50 52 %Exit-zero rate · 26 / 50 · Container test exit | — | — | 1 × R9700Single GPUConfiguration
|
Versions and harnesses
Versions: Not reported
Harnesses: pi-coding-agent / pi-bench
This is a selection across sources, not a ranking or a matched experiment. Reading benchmark reports
GPU benchmark pages: R9700