TuringCorp Models

Team Junior

ProfBench — 2026-09-10

ProfBench — 40 expert-level tasks, ten each in Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA, scored criterion by criterion with the official rubric.

Self-run with the official rubric protocol: per-criterion Yes/No with the official prompt verbatim, temperature 0 / top_p 1, no output cap. Team delivery is content + reason. 38 of 40 tasks scored: two tasks were excluded because no answer was produced for them during the run (disclosed rather than imputed). The DeepSeek V4 Flash baseline and the official o3 draft were read by the same judging pipeline, and all three columns are reported on the 38 tasks every model completed.

Score by domain

Team, the DeepSeek V4 Flash baseline and the official o3 draft were all read by the same judging pipeline (per-criterion Yes/No, official prompt, temperature 0, no output cap), and are reported on the 38 tasks that all three completed — so the three self-run columns are directly comparable.

All three self-run columns come from one judging pipeline on the same 38 tasks: Team 63.3 · DeepSeek V4 Flash (direct) 57.4 · official o3 draft 55.6.

Judge calibration

Our judging pipeline reproduces the official per-criterion labels on the o3 draft with 74.0% agreement (F1 0.761) and a mean score difference of +2.4 points — the numbers above sit on our pipeline's scale, calibrated against the official one.

Head-to-head arbitration by our decision kernel

36 : 1 — team draft : official o3 draft, 38 tasks, 1 unresolved.

On the same 38 tasks we also asked our own decision kernel to choose between the team draft and the official o3 draft. It was shown the question and the two drafts and nothing else — no rubric, no scores — and it selected the team draft 36 times.

• Position swap. On all 37 tasks where the kernel had made a pick, the comparison was re-run with the two drafts exchanged between option A and option B. 36 kept the same draft. The single change came on a task the kernel had already flagged as a near-tie (23% confidence); every pick it made at 80% confidence or above — 23 of the 37 — came back identical.
• No self-grading. The judgment comes from a panel of independent models rather than one model scoring its own output, which removes the self-preference bias of a single-judge setup.

This head-to-head run is supporting evidence, not the primary measurement: the per-criterion rubric columns above are the measurement, and the kernel reached its conclusion without ever seeing them.

Confidence reported with every judgment

The kernel reports a confidence value with every judgment, so a preference is never a bare claim. Median confidence on this benchmark is 85.9%, and it is not uniform across domains: highest on Physics PhD (95.3%) and Chemistry PhD (87.0%), lowest on Finance MBA (83.8%). It describes how far apart the two drafts are in the kernel's judgment: the lower it is, the closer the two are in quality — where either draft is a reasonable choice.

The 38-task run used a single fixed presentation order, with the team draft always offered as option A; the position-swap audit above, not this run, is the control for that.

Official reference (official judging pipeline)

Official evaluation of the official reference drafts, produced by the official judging pipeline. A different judge from the self-run columns above, so these rows are context only — not a head-to-head comparison.

Self-run with the official per-criterion rubric protocol (official prompt verbatim, temperature 0, top_p 1, no output cap). Our judging pipeline reproduces the official criterion labels on the o3 draft with 74.0% agreement and a mean difference of +2.4 points. Confidence describes how close the two drafts were in that judgment.

Decider Junior

JudgeBench — 2026-09-08

JudgeBench, 620 pairs (sources: MMLU-Pro / LiveBench Reasoning / Math / LiveCodeBench)

Self-run with the official judging protocol. Both columns use first successful verdict per pair; the six pairs whose first verdict failed are disclosed rather than imputed. Reference model measured on the same judged set.

Agreement between the two: 96.1%.

Confidence calibration

Confidence is emitted with every judgment and calibrated against outcomes on this benchmark (JudgeBench, 2026-09-08). It describes how far apart the two candidates are — and the accuracy column shows what that delivered here. Set your own threshold from the accuracy column; for high-stakes or irreversible decisions, apply your own review policy.

Every judgment ships with a calibrated confidence value. A high value means the comparison was decisive and the pick can be acted on directly; a low value means the two are evenly matched, where either choice is defensible and the decision belongs to criteria outside the answers.

ContextualJudgeBench — 2026-09-09

ContextualJudgeBench (Salesforce), full official set of 2,000 pairs across all 8 splits; official reference values from the paper's Table 2.

Self-run with the official vanilla pairwise protocol; consistent accuracy (both response orders judged correctly); random floor 25%; failures disclosed (12 orders rerun-excluded). Reference model measured on the same judged set.

Completed 1991 / 2000 official pairs.

Full official 8-split run completed (2026-09-09). Consistent accuracy 67.1% vs reference 65.4%: the advantage is strongest on faithfulness and refusal tasks, while the deliberately near-tie splits sit in the 46-60% range, reflecting the benchmark's difficulty design. Confidence tiers below are calibrated on this benchmark.

Confidence calibration

Confidence is emitted with every judgment and calibrated against outcomes on this benchmark (ContextualJudgeBench, 2026-09-09). It describes how far apart the two candidates are, and the accuracy column shows what that delivered here. This benchmark deliberately contains near-tie splits, so the accuracies run lower by design — the bands describe the comparison, not a promise about either answer.

Every judgment ships with a calibrated confidence value. A high value means the comparison was decisive and the pick can be acted on directly; a low value means the two are evenly matched, where either choice is defensible.

Self-run with the official protocol over the full 2,000 official pairs. Consistent accuracy requires the same correct pick in both presentation orders (random floor 25%). 12 orders (0.3%) were rerun-excluded after repeated platform failures; the reference model was measured on the same judged set. Previously published results measured on a subset are archived in the repository changelog.