Demos / Evaluator Integrity
Recorded audits.
Watch the full experimental result, then inspect the scoring failure behind it.
Latest result / MASK in Inspect Evals
More false assertions.
A higher original score.
In a controlled experiment, recorded false assertions rose from 149 to 162 while the edited policy received a higher original score. Reading the same saved judge replies with a repaired rule reversed its score advantage.
This deliberately engineered comparison used benchmark facts, previously exposed items and reused baseline answers; both the generation model and prompt changed. The falsity comparison is conditional on the recorded paired reviews. The evidence is agent-screened; human review has not been completed. There is no fresh holdout or claim that published leaderboard scores are wrong.
Scoring audit / MASK in Inspect Evals
Right conclusion.
Wrong recorded score.
A closer look at one saved example: the judge identifies a false assertion, but the scoring software records a different answer from its final verdict.
This selected example concerns the tested implementation and judge configuration. It illustrates the scoring mechanism; the full-pool semantic comparison is covered in the separate recording above. Semantic review remains agent-screened.
From a false pass
to a checked repair.
A separate fictional case shows the shape of an assessment report. A returns-policy evaluator accepts answers that contradict each other: follow the assessment, repair and recheck.
Lumen Practice Suite
Juniper Mock Judge
Return-policy assessment
Contradictory answers both receive a pass.
- Configuration
- Juniper Mock Judge · policy-consistency setting
- Task
- Check a support answer against the supplied return policy.
- Review sample
- 50 pairs of answers to questions the policy settles.
- Finding
- In 14 pairs, both answers passed despite contradicting each other. At least one answer in each pair must be wrong.
- Recommendation
- Repair the decision rule, then verify on new answers.
What the judge accepted
- Requirement
- Accepted answers must not contradict each other on a question the policy settles.
- Observed gap
- Both sides of 14 contradictory pairs received a pass for the same case and policy.
- Why it matters
- A contradiction proves at least one false pass without selecting the correct answer.
- Separate quality check
- The 50 labelled correct controls check for false rejections. They are not used to establish the contradictions.
Repair, then test again
- Repair
- Require the decision to match the supplied policy; route uncertain answers for human review.
- Verification set
- 50 fresh answer pairs, plus 50 separately labelled correct controls. Both configurations checked on the same set.
| Outcome | Original | Repaired |
|---|---|---|
| Contradictory pairs both accepted | 16 / 50 | 1 / 50 |
| Correct controls accepted | 48 / 50 | 48 / 50 |
The repair reduced accepted contradictions while correct-control acceptance stayed unchanged. One contradictory pair still passed, so at least one false pass remains.