Live rating progress
Complete Zalo checklist across three selected recordings
Each rater reviews eight questions per recording. The form displays all 32 checks and their decision details: 27 scored per question, I1–I4 scored once for each recording, and H2 kept as a written diagnostic prompt under H1. Anyone with the private invitation can join; the results use anonymous rater labels.
Instrument status: this is the latest Zalo HTML checklist prototype selected by the project team for this rating round. It is not a frozen or validated questionnaire.
Loading current results…
Progress
Question and recording-review tiles
Each tile tracks one submitted question (Q1–Q8) or the recording-wide I1–I4 review. Drafts stay private. “Not submitted” can mean untouched or saved as a draft.
Per-question checks
27 scored item-level criteria
Counts show YES, NO, UNCLEAR and N/A where a criterion permits it. H2 is a written prompt and does not receive an independent score.
Once per recording
Four set-level criteria
I1–I4 are submitted once after each rater reviews all eight questions in a recording.
Question operation
Patterns by listening operation
Context check
Recording × operation counts
These are descriptive counts from the selected recordings.
Method QA
Independent-rater agreement
Pairwise comparisons use all possible rater pairs across matched items or recordings. Raw agreement and Gwet’s AC1 are primary descriptive outputs; Cohen’s κ is secondary. These are not validity estimates.
Item-level checks
Recording-level checks
Interpretation boundaries
What these results can show
- Ratings describe this selected set of recordings and questions; they are not population prevalence estimates.
- The study does not establish formal CEFR validation, empirical difficulty, discrimination, or empirical distractor functioning.
- Agreement describes consistency in applying this prototype checklist, not test reliability or checklist validity.
- The automated pilot is exploratory and is not pooled with the human ratings.