Multivac
Paper under reviewFig. 1 · The peer matrix
Independent, blind evaluation of frontier language models. Fresh questions models haven't memorized, judged by a cross-family peer matrix — no vendor grades its own homework.
Open MultivacIndependent research lab
Intelligence, measured. Judgment, audited.
Deep Pearl AI studies two sides of the same question: how do we know an AI system actually works — and how do we know the judge grading it is right? Multivac measures frontier models with blind, cross-family peer evaluation. PartialCredit measures the judges themselves against code that actually runs.
NACRE SIM LAYER 00/44 Δr 0.00px T+0.0s
The work
Two instruments, one thesis
Frontier models are judged by benchmarks they can game, then graded by judges nobody has checked. We work both ends of that gap. Each instrument below is a working model of its research — open the drawer to see it run.
PartialCredit
Preregistered · pilot completeFig. 2 · The pass vector
Pairs of partially-correct code fixes, scored by hidden tests, shown to LLM judges blind. It measures how often a judge prefers the fix that execution says is worse — and whether that rate climbs as the two fixes get closer.
Open PartialCreditHow we work
First principles, honestly reported
No one grades their own homework
Evaluations are blind and judged across model families. A benchmark's design shapes its rankings, so we design against our own bias first.
Run the code, then audit the judge
PartialCredit scores every patch on a hidden, human-authored test suite that no judge ever sees. Pass rate is a proxy, so a blind manual audit decides whether a disagreement is a judge error, a weak test or ambiguous.
Preregister, then look
Hypotheses, exclusions and analyses are frozen before the data exists, and every change is a dated amendment. Inconclusive is published as inconclusive.