Deep Pearl AI
Deep Pearl AI

Judge audit

Pilot complete · preregistered 2026-10-23

PartialCredit

Do LLM judges rank partially-correct code fixes the way hidden tests do?

Teams increasingly let one language model grade the code another one wrote, because asking a model is cheaper than running the tests. PartialCredit builds pairs of bug fixes that are both partly right — each passes some of a hidden test suite, neither passes all of it — shows them to a panel of LLM judges blind, and measures how often the judge prefers the fix that execution says is worse, and whether that rate climbs as the two fixes get closer in quality.

Repository not yet public
ONE TRIAL · SCHEMATIC 100 CELLS STAND FOR ONE TASK’S HIDDEN INPUTS PATCH A PATCH B 806 / 1000 491 / 1000 GAP 0.315 NESTED · B’S PASSES ⊂ A’S STRATUM ORDER · HI_FIRST · PATCH A SHOWN AS CANDIDATE A

What the judge sees

  • specification
  • buggy program
  • candidate A
  • candidate B

never tests, never pass rates, never model names

VERDICT: _

FAIR COIN · SIMULATED AT THE NULL, 0.50 · NOT A MEASURED RATE 01 SCORING · EVERY CANDIDATE RUNS ON EVERY HIDDEN INPUT

hi passes 806 lo passes 491 gap 0.315 stratum medium order hi_first relation nested null = 0.50 verdict _ outcome —

A is fixed at 806; the slider moves B
FIG. 1 — ONE TRIAL, SCHEMATIC Schematic of one trial. The verdict is a coin at the null (0.50), not a measured rate. A HumanEval+ task has ~972 hidden inputs; MBPP+ ~105.

Why this matters

Existing code-judge benchmarks label one patch correct and one incorrect by whether it passes an entire test suite. A patch passing 5 of 6 hidden tests and one passing 2 of 6 are both stamped “incorrect” and never compared. The binary label throws away exactly the comparison a judge is asked to make in practice.

Judges already rank patches and filter training data, so the partial case is the normal case. For the human-inserted bugs PartialCredit starts from, the buggy program’s median pass rate on the hidden inputs is 0.11 before any repair — half of the starting programs already pass more than a tenth of the suite.

PartialCredit measures the gap directly. Two partly right fixes for the same task, both scored by execution on a hidden suite, shown blind to a judge that must pick one. Picking the fix that passes strictly fewer hidden inputs is an inversion. Every pair is shown in both orders, so a judge with no signal scores exactly 0.50 — and the primary question is whether inversions climb as the two fixes get closer.

Method

How one trial runs

  1. A repair task

    A buggy program and a hidden suite of inputs it should satisfy. HumanEval+ tasks carry a median of 972 hidden inputs, MBPP+ a median of 105; the suites are EvalPlus’s, pre-existing and human-authored. HumanEval+ bugs are HumanEvalPack’s human-inserted bugs; MBPP+ bugs are freshly seeded by a model that is neither a generator nor a judge, and reported separately.

  2. Two partial patches, scored by execution

    Small generator models propose fixes at three temperatures. Every candidate runs on every hidden input, twice; pass rate is passes over inputs. Only candidates with a pass rate strictly between 0 and 1 are paired, equal pass rates are never paired, and pairs are stratified by the gap between the two rates.

  3. The judge sees the specification, the buggy program, and A and B

    Nothing else: never tests, never test output, never pass rates, never model names. Complete functions rather than diffs, one pair per call, temperature 0, one fixed prompt.

  4. VERDICT: A or B

    Forced choice with no tie option; the last VERDICT line in the reply wins. Refusals, hedges, truncations and unparseable replies are excluded from the estimate and reported as answer rate.

  5. An inversion is picking the lower patch

    If the judge selects the candidate that passes strictly fewer hidden inputs, the trial is an inversion. Whether that makes the judge wrong, or the suite weak, is what the manual audit decides.

  6. Both orders, null 0.50

    Every pair is shown twice, hi first and lo first. A judge with no signal, or one that always picks the same slot, scores exactly 0.50. That is the null for every hypothesis on this page.

Scope

What gets measured

H1a
Pooled inversion rate against 0.50, in the both-functional stratum, across the six frontier/strong judges. Interval from a task-cluster bootstrap; the Wilson interval is kept for comparison only.
H1b · primary
A negative coefficient on gap: a statsmodels logit with task-clustered standard errors, and an R glmer with crossed random effects. Supported only if the 95% CI excludes zero in both.
Position lock
The same slot chosen in both orders, reported per judge alongside position consistency. A lock marginalises to exactly 0.50 and is auditable.
Baselines
Prefer-longer and prefer-shorter heuristics against the same null of 0.50; the length difference between candidates enters the model as a covariate.
Design effect
Clusters are tasks. ICC by task and by pair, task-cluster bootstrap CIs with B = 5000. The realised near-tie MDE decides pooling: strata pool only if it is at or under 10 points.
Audit
About 150 disagreement pairs, both authors labelling blind as judge_error / weak_test / ambiguous, at least 50 pairs double-labelled, Cohen’s kappa reported.

Ground truth

Ground truth rules

  1. Hidden tests are pre-existing and human-authored (EvalPlus). No LLM writes a grading test.
  2. No judge authored any test. Generator/judge family overlap is minimised and reported.
  3. Generators never see the hidden suite. Judges never see tests or test output.
  4. Each candidate is executed twice; unstable pass sets are dropped and the rate reported.
  5. Pass rate is a proxy for correctness. The manual audit is what turns “judge disagreed with execution” into “judge was wrong”.

Validation, 2026-09-30. The HumanEval+ canonical solution passes 100% of hidden inputs on 163 of 164 tasks, 0 flaky; the MBPP+ canonical solution on 377 of 378, 0 flaky.

Exclusions. HumanEval/32 (its find_zero has many valid roots, so exact-output comparison is the wrong oracle), tasks with fewer than 20 hidden inputs, candidates whose pass sets are unstable across two executions (rule 4), and, by amendment 5, tasks whose expected-output payload exceeds 50 MB: HumanEval/15, HumanEval/130 and Mbpp/255.

Apparatus

The panel and the bank

Slugs exactly as configs/models.yaml, in configuration order. Judges are never ordered by any score, and no judge family writes patches.

Judge panel · 8 judges
  • openai/gpt-5 reasoning: low
  • anthropic/claude-sonnet-4.5
  • google/gemini-2.5-pro reasoning: low
  • deepseek/deepseek-chat-v3.1
  • cohere/command-a replaced z-ai/glm-4.6, amendment 2
  • moonshotai/kimi-k2-0905 max_tokens 10,000
  • google/gemma-3-12b-it small · contrast row
  • openai/gpt-4o-mini small · contrast row

The headline pools the six frontier/strong judges; the two small judges are contrast rows, reported alongside, never pooled in. Each pair × judge × both orders at temperature 0, max_tokens 6000 (Kimi 10,000); 10% of pairs × 3 repeats, 10% × 2 extra prompt variants. The preregistration counts the panel as 7 families.

Generators · small models (D7)
  • meta-llama/llama-3.1-8b-instruct
  • qwen/qwen-2.5-7b-instruct
  • meta-llama/llama-3.2-3b-instruct
  • microsoft/phi-4
  • qwen/qwen3-14b replaced qwen/qwen3-8b, amendment 8
  • mistralai/mistral-small-3.2-24b-instruct
  • meta-llama/llama-3.3-70b-instruct mid-size reserve, only if pair yield is short; unused so far

Decision D7: a 70B generator fixed the bugs perfectly almost every time and produced no usable pairs, so partial patches come from small instruct models at temperatures 0.2, 0.8 and 1.2. No generator family sits on the judge panel (decision D8). The pilot used the first four; the full run uses six.

Bug seeders · MBPP+
  • google/gemma-3-27b-itfirst pass
  • amazon/nova-micro-v1second pass, amendment 6

Neither is a generator nor a judge, so the model that writes a bug never also writes patches for it. 147 MBPP+ tasks enter the full run; 226 are dropped because even a weak model solves them every time. Chosen bugs have a median pass rate of 0.51. Seeding cost $0.17.

Datasets · frozen 2026-09-30
  • HumanEval+ v0.1.10164 tasks
  • MBPP+ v0.2.0378 tasks
  • HumanEvalPack python164 buggy programs

SHA-256 hashes in data/datasets/manifest.json. Hidden inputs per task: HumanEval+ median 972 (min 2, max 1000); MBPP+ median 105. Eligible for the full run: 160 HumanEval+ + 147 MBPP+ = 307 tasks.

Profiles
  • smoke5 tasks, 2 generators, 1 judge, K = 1
  • pilotall eligible HumanEval+ tasks, 4 generators, 8 judges, K = 1
  • fullHumanEval+ and MBPP+, 6 generators, 8 judges, K = 3

K is pairs per stratum per task. Sampling seed 20260922 everywhere. Every stage writes JSONL under data/ and resumes; every API call is cached.

Gap strata · |rate(A) − rate(B)|
  • near_tie(0, 0.05]
  • small(0.05, 0.15]
  • medium(0.15, 0.35]
  • large(0.35, 0.70]
  • huge(0.70, 1.0]
  • no-opplanted; reported separately, never in the headline

H1b lives in the first two strata. A planted no-op pairs the buggy program with a cosmetic edit against a partial candidate whose pass rate is strictly higher than the buggy program’s.

As of 2026-11-04

Status

Repository HEAD be79a73 of 2026-10-27; 25 commits since 2026-09-22; one tag, prereg-v1. No activity after that commit is shown; status checked 2026-11-04.

Done

  • Datasets frozen with SHA-256 hashes (2026-09-30).
  • Ground-truth validation on all tasks.
  • Smoke run end to end: 5 tasks, 17 pairs, 34 judgments, $0.38.
  • Generator probes (decision D7): small models only.
  • Preregistration frozen 2026-10-23, tag prereg-v1.
  • Pilot complete: 162 HumanEval+ tasks, 4 generators, 3,888 candidates, 348 pairs from 124 tasks (76% yield), 7,200 judgments, $45.10. Non-answer rate 1.1%.
  • Pilot analysis with task-cluster bootstrap (amendment 9).
  • Full run sized at K = 3 on both datasets.
  • MBPP+ bug seeding complete: 147 tasks enter (decision D10).
  • Nested-vs-crossed secondary analysis added (amendment 7).
  • Pre-full-run code review (amendment 8).
  • Preregistered glmer fit run on the pilot as a sizing-stage check (fit 2026-10-26, committed 2026-10-27).
  • Raw pilot artifacts committed for co-author review.

Not done

  • T1.8 — reviewer read of the judge prompt for leaks.
  • T2.4 / T2.6 — audit codebook (docs/AUDIT_CODEBOOK.md does not exist).
  • T3.4 — pilot audit (data/audit/ is empty).
  • T3.6 — related-work pass.
  • T4.1 — freeze configs/models.yaml (tag full-config).
  • T4.2 to T4.5 — the full run: generate, score, pairs; ~26k judgments, $200–300 expected, cap $1,000; analysis; audit of ~150 pairs. No full-run artifacts exist.
  • T4.6 to T4.8 — paper draft, datasheet, arXiv.
  • The plan’s full-run window (Oct 20 to Nov 2) has passed with the run not started.
Pilot pairs per stratum · K = 1
near_tie 68 · small 44 · medium 38 · large 47 · huge 27. Planted no-op 124, of which 77 valid after amendment 3.
Design effect · by task
ICC 0.21; design effect 6.7; effective N 383 of 2,650. Near-tie MDE at K = 1 about 11 points. 22.4 judgments per pair in the full profile.
Projected near-tie MDE · K = 3
About 6 to 8 points, bounds 4 to 11. A projection, not a result: the bounds are the result.

Amendments to the preregistration

  1. 2026-10-23GLM-4.6 reasoning settings and token caps; a new "truncated" abstention status.
  2. 2026-10-23z-ai/glm-4.6 replaced by cohere/command-a; panel stays at 8 judges.
  3. 2026-10-23Planted no-op stratum rule corrected (47 of 124 pilot no-op pairs affected; headline unaffected).
  4. 2026-10-23K = 3 for the full run (its sizing figures withdrawn by amendment 9; K = 3 stands).
  5. 2026-10-23Exclusion rule 5: tasks whose expected-output payload exceeds 50 MB.
  6. 2026-10-23Second MBPP+ seeding pass with amazon/nova-micro-v1.
  7. 2026-10-23Nested vs crossed secondary analysis; hi_contains_lo bug fixed.
  8. 2026-10-23Pre-full-run code review, eight fixes.
  9. 2026-10-26Clustering corrected from (judge, task) to task; sizing redone; task-cluster bootstrap CIs (B = 5000); strata pooled only if the realised near-tie MDE is at or under 10 points.

Stop condition. One pilot. One full run. If inconclusive, published as inconclusive. No third run.

Pilot inversion rates are sizing inputs, not results, and are not shown here.

Preregistered, verbatim

What we will not claim

Not that judges are broken. Not that any model is best. Not that disagreement with the suite is automatically judge error. Only the measured numbers and what the audit supports.

Known threats, stated now

Pass rate is a proxy (patch overfitting); the audit bounds it. HumanEvalPack bugs and fixes are public and likely in judge training data; MBPP+ bugs are freshly seeded and serve as the uncontaminated contrast, reported separately. Pairwise forced choice is justified by CodeJudgeBench’s tie finding; position consistency is reported so it is auditable. Small-function tasks limit external validity; stated, not hidden.

Source

The repository

Not yet public. Each file is named by its path from the repository root.

  • github.com/Yash2378/partial-credit-judge repository, MIT
  • docs/PREREGISTRATION.md frozen v1, 2026-10-23, tag prereg-v1
  • docs/DECISIONS.md decisions log
  • docs/proposal-v2.md original proposal, history only
  • configs/models.yaml panel, generators, profiles, strata
  • partialcredit/prompts.py the judge prompt
  • data/analysis/pilot/ pilot artifacts: report, sizing, glmer, no-op

Upstream: EvalPlus (Liu et al., NeurIPS 2023) and HumanEvalPack (Muennighoff et al., ICLR 2024).

Yash Darji, Oyindolapo Komolafe. MIT licensed.