What this evaluates

This repository began as a comparison of local Qwen3.6 27B GGUF profiles used as a coding-agent sidecar, not as a broad benchmark leaderboard.

The target workflow:

  • Robert is the human operator and final decision-maker.
  • Codex owns local inspection, patching, deterministic validation, and integration.
  • Qwen is used as a local sidecar for challenge, validation, UI/design critique, and alternative reasoning.

The reusable sidecar-companion suites can also be applied to another local model as a bounded follow-up. Cross-model results remain secondary to the original Qwen profile study and are separated when suite wording, sampling, or runtime conditions differ.

Suites

sidecar-companion-v1 (canonical)

The fast 10-case contribution-facing regression suite. It covers exact coding judgment, directive fidelity, UI regression recognition, artifact triage, and two long-context retrieval envelopes.

sidecar-companion-v2 (draft)

The harder 10-case suite adds multi-file contradictions, partial-evidence refusal, buried hard directives, plausible over-building, blocker prioritisation, text-only UI review, tool-evidence conflicts, unstable external claims, governance carry-over, and authority boundaries.

Both use JSON-field scoring. Enumerated values remain exact. A case may declare an explanatory field in semantic_fields; that field passes only when a non-empty value is returned and is not treated as an exact prose match.

max-accuracy-v1

Exact-scored JSON cases with known answers:

  • arithmetic plus preserved constraints;
  • route-selection exactness;
  • Python patch review;
  • TypeScript async bug detection;
  • hard-directive compliance;
  • UI regression detection;
  • artifact parse-warning classification;
  • Python sorting behavior;
  • common long-context needle retrieval;
  • extended >65k context needle retrieval.

This suite is intentionally strict: semantic near-misses can lose points if an exact enum/list/value is wrong.

Historical qualitative sidecar pack

An earlier qualitative sidecar pack was used during local iteration, but it is not part of the public canonical suite. It was removed from the repo before broader release to keep the project focused on the reproducible exact-scored suite.

Sampling and API shape

The runner sends OpenAI-compatible chat completions with deterministic defaults:

{
  "temperature": 0,
  "top_p": 1,
  "top_k": 1,
  "max_tokens": 8192,
  "chat_template_kwargs": {"enable_thinking": false}
}

The CLI can override sampling, penalties, thinking, and thinking-budget fields for profile-specific measurements. Those overrides are recorded in run.json. Deterministic-default and profile-sampled results must be labelled separately. The server launch profile still matters for the model file, context length, KV cache, templates, and runtime implementation.

Limitations

  • Single local machine and single-run profile samples.
  • Synthetic but workflow-derived test cases.
  • Exact scoring penalizes values that are semantically close but not exact.
  • The suite is designed for a Codex + Qwen sidecar workflow, not for general reasoning leaderboard claims.
  • Server logs with local device identifiers are not published; raw API outputs and scoring records are included.

Community suites

The canonical contribution-facing suite is evals/sidecar-companion-v1. The harder evals/sidecar-companion-v2 suite remains draft. Both avoid subjective prose grading by defining scoreable JSON fields; any declared semantic field is limited to a non-empty-value check.

Historical test packs under tests/ are kept for provenance. New contributors should start with evals/sidecar-companion-v1 and the tools under tools/.