Corpus and selection
The rating workspace is fixed to three prepared C1 recording packs: S1 Botany trip in South Africa (c1-cam-s1-p2), S2 Journalists Jenny Langdon and Peter Sharples (c1-cam-s1-p3), and S3 Beach conservation project (c1-cam-s2-p2). Each recording has eight first-generation AI-created four-option MCQs: two each for Gist, Detail, Inference and Speaker Attitude/Intention. Each rater completes 24 question ratings plus three recording-wide reviews.
Anyone with the private study invitation can join. A linked browser resumes the displayed rater directly; another browser needs the same rater name and password. A legacy profile without a password can set one from its already linked browser. Unfinished work saves privately as raters work. The intended context is C1-targeted; this is not formal CEFR validation.
Complete Zalo HTML checklist
This trial uses all 32 checks from the latest Zalo operationalized quality-check manual. The attachment labels itself a design prototype, not a frozen formal questionnaire. That status stays visible in the rater workspace and results.
- 27 scored checks appear for each question.
- I1–I4 are scored once per recording after all eight questions are reviewed.
- H2 is an optional written prompt under H1 and has no separate score.
- F3 includes N/A when the stem does not imply a semantic answer type.
- Each check in the rater workspace includes the manual’s definition, inspection points, procedure, decision anchors, pass/fail and borderline examples, do-not-penalize note, and question probes.
Open the complete HTML manual, including its close-criterion comparison.
Independent review
Three raters submit independently. A brief evidence comment is required for NO or UNCLEAR. The first submission for each question and each recording-wide review locks. One browser/device is used per rater so the site can enforce three distinct enrolled identities; use separate devices or browser profiles.
Analysis
Results show criterion-level counts, operation-level counts, recording × operation counts, and rater progress. Agreement is calculated over all three possible rater pairs for matched items; recording-wide criteria use the three rater pairs for each selected recording. Raw agreement and unweighted Gwet’s AC1 are reported, with unweighted Cohen’s κ as a secondary comparison. These are descriptive agreement measures.
Claim boundaries
- No empirical difficulty, discrimination or empirical distractor-functioning claim is made without learner responses.
- No source effect, causal effect or learner-performance conclusion is estimated.
- Results describe the selected questions and recordings and do not establish population prevalence.
- Agreement describes consistency applying this prototype checklist; it is not test reliability or proof of checklist validity.
- Automated pilot judgments remain exploratory and are not pooled with human ratings.