GPT-5.6 Luna · transcript-only pilot
A useful warning, not a quality certificate
The first automated pilot reviewed all 32 generated items. It exposed the assigned operation and answer key, so operation agreement was not blind. It also produced no variation at all on C5–C15. The next benchmark therefore hides labels and keys first and adds controlled-defect cases.
Operation classification
Assigned versus detected operation
Assigned match means the model's detected primary operation agreed with the operation requested during item generation. Because the assigned label was visible, this is a descriptive pilot comparison, not independent classification accuracy.
Ceiling diagnostic
C5–C15 showed zero discrimination
Luna returned YES for all 352 quality judgments. That is evidence of judge leniency or prompt anchoring, not proof that every item is flawless.
What changes next
Blind, contrastive, and portable evaluation
- Allocate 100 evidence points across all four operations before seeing the assigned label.
- Reconstruct the answer before seeing the designated key.
- Require exact evidence, the strongest competing interpretation, a boundary rule, confidence, and a change-of-mind condition.
- Evaluate controlled variants so an all-YES judge fails to detect known contrasts.
- Report transcript and audio tracks separately and keep every model run outside the formal human-rater dataset.
Interpretation boundaries
What this pilot does not establish
It does not establish formal CEFR level, empirical difficulty, discrimination, learner performance, empirical distractor functioning, test reliability, source effects, causal effects, or population prevalence. Protected transcripts, audio, answer keys, and raw private model records are not published here.