Historical max-accuracy v1 results

Canonical machine-readable summaries:

  • results/max-accuracy-v1/summary.csv
  • results/max-accuracy-v1/summary.json
  • results/max-accuracy-v1/combined-max-accuracy-20260504.md

Max-accuracy summary

Profile Exact Weighted Errors Elapsed Short/code/UI Common needle Extended >65k needle
unsloth-65k-f16 33/39 36/48 1 22.0s 30/33 3/3 0/3
unsloth-65k-q8 33/39 36/48 1 23.7s 30/33 3/3 0/3
bartowski-128k-f16 36/39 45/48 0 55.7s 30/33 3/3 3/3
bartowski-128k-q8 36/39 45/48 0 56.7s 30/33 3/3 3/3
unsloth-128k-f16 30/39 30/48 2 634.1s* 30/33 0/3 0/3
unsloth-128k-q8 36/39 45/48 0 57.6s 30/33 3/3 3/3

* Loaded, then timed out on both long-context prompts under local memory/throughput pressure. The elapsed value includes those timeouts and should not be treated as completed throughput.

Visuals

Exact score

Weighted score

Elapsed seconds

Case heatmap

Raw results

Raw API responses and per-case scoring records for the headline exact-scored suite are included under:

results/max-accuracy-v1/raw/

Follow-up: sidecar companion v1/v2 — 2026-08-08

After the original profile sweep, the reusable companion suite was revised and a harder v2 draft was added. The following direct-API check applied those suites to two newer thinking profiles. It is kept separate from the historical matrix because the suite wording and sampler configuration differ.

Model V1 run 1 V1 run 2 V1 run 3 V1 combined V2 run 1 V2 run 2 V2 run 3 V2 combined Overall Parse errors
BigBang v1 36B-A3B Q5_K_M 39/39 37/39 36/39 112/117 (95.7%) 38/43 43/43 34/43 115/129 (89.1%) 227/246 (92.3%) 3
Qwen3.6 27B Q6_K_XL 38/39 39/39 39/39 116/117 (99.1%) 43/43 42/43 43/43 128/129 (99.2%) 244/246 (99.2%) 0
Model V1 weighted V2 weighted Overall weighted
BigBang v1 36B-A3B Q5_K_M 139/144 (96.5%) 142.3/159.9 (89.0%) 281.3/303.9 (92.6%)
Qwen3.6 27B Q6_K_XL 143/144 (99.3%) 158.7/159.9 (99.2%) 301.7/303.9 (99.3%)

Both used thinking with preserve_thinking=true, an 8192-token output cap, temperature 0.6, and top-k 20. BigBang used top-p 0.95 and presence penalty 0.0; Qwen used top-p 0.8 and presence penalty 1.0.

The official scores preserve strict JSON parsing. BigBang’s three parse failures contained the intended values but added duplicated or stray formatting. Manually normalising those responses would produce approximately 239/246 (97.2%), but that is not an official suite score.

Machine-readable aggregates and the 12 validated run packages are under:

results/sidecar-companion-v1v2-3x-20260808/

Timing is not compared because two BigBang per-case records contain a wall-clock discontinuity. Scores are unaffected.