NCPP Operationalized Quality Check Manual

A1: Is the stem and each option grammatical enough to understand without mentally repairing it? F4: Do the options fit the stem comparably, so grammar does not reveal the key?

Description examples

• A1 fails, F4 can pass: The stem has a verb-form error, but all four answer choices are parallel, grammatical “because…” reasons. The item has a language defect even though the options do not distinguish the key by form.
• F4 fails while A1 may pass: Every option is a grammatical phrase on its own, but only one naturally completes the stem. If the rater must judge grammar in context, the current A1 and F4 anchors may both fail; that makes F4 a likely diagnostic subcheck, not a cleanly independent score.

Actual question examples

• A1 = NO; F4 = YES: Stem: “Why did the speaker left early?” Options: “Because she felt unwell”; “Because the train was late”; “Because childcare fell through”; “Because the room was closing.” The stem is ungrammatical; all options have the same usable form.
• F4 = NO; A1 is debatable under the current wording: Stem: “The speaker was interested ___.” Options: “in the proposal”; “about the proposal”; “to the proposal”; “for the proposal.” The phrases are grammatical individually, but only “in the proposal” fits the stem naturally. If A1 is judged in stem context, it may also fail. This is a sign to define whether F4 is a separate score or an A1 diagnostic.

A2 asks whether the whole item has one stable interpretation. A4 zooms in on whether pronouns and reference words point to one person, event, or idea. A material A4 problem will usually also make A2 fail.

Description examples

• A2 fails, A4 passes: Pronouns all have clear referents, but the stem asks two different things at once, such as the planned action and the reason another person objected.
• A4 fails, A2 may pass: A pronoun could refer to either of two people, but both interpretations lead to the same answer. Decide whether A4 is meant to flag every unclear reference or only reference problems that change the task/answer.

Actual question examples

• A2 = NO; A4 = YES: Audio: “Linh proposed moving the meeting, and An objected because the room was booked.” Stem: “What change did Linh propose, and why did An object?” The names are clear, but this combines two answer tasks; a single-answer MCQ stem is unstable.
• A4 = NO; A2 could be YES if the answer is unchanged: Audio: “Mai and Lan both said the room was too cold.” Stem: “What did she say about the room?” “She” is unclear, but either referent yields the same response. If the rubric treats any unclear referent as failure, A4 is stricter than A2.

A3 is specifically about removable or unnecessarily difficult written wording. H3 covers any unrelated demand that dominates success, such as specialist knowledge or calculation. A3 is narrower; it can be a diagnostic subtype of H3.

Description examples

• A3 = NO; H3 can be YES: The stem is wordier than necessary, adding reading effort, but the learner can still answer by understanding the audio. The extra wording is a local writing defect, not the main skill required.
• H3 = NO; A3 = YES: The stem is short and clear, but answering requires specialist knowledge never provided in the recording. There is no avoidable wording to remove.

Actual question examples

• A3 = NO; H3 can be YES: Audio: “The meeting was postponed because the speaker was ill.” Stem: “For which principal reason, according to the speaker’s account, was the previously scheduled meeting ultimately postponed?” Shorten it to “Why was the meeting postponed?” without changing the listening task.
• H3 = NO; A3 = YES: Audio: “The sample was heated to 40 degrees.” Stem: “At what temperature was the sample heated?” If the options require converting Celsius to an unfamiliar scientific unit not taught or stated, the burden is outside listening even though the question is concise.

B1 checks whether the stem invents/contradicts a fact. B2 asks whether the supporting span can be pointed to. B3 asks whether the question is about content represented in the source. C1 asks whether enough information exists to answer. D3 asks whether the information is audible to the learner. These are related; B2/C1 and B2/D3 are especially likely to overlap.

Description examples

• B1 = YES; B2/C1 = NO: The stem accurately states that an event happened, but the source never gives the reason asked for. The premise is true; the answer is unsupported.
• B1 = NO; B3 = YES: The question is about the right project/topic, but it changes “delayed” into “canceled.” It is relevant and still contains a false premise.
• B2 = YES; D3 = NO: A transcript provides an exact bracketed note for the reviewer to locate, but the audio does not convey that information. Evidence is findable in the transcript and unavailable to the listener.
• B2 = NO; D3 = YES: A detail is clearly audible, but the rater cannot locate or identify the necessary span from the supplied materials. The sound contains evidence; traceability is poor.

Actual question examples

• B1 = YES; B2/C1 = NO: Audio: “Maria canceled the meeting.” No reason is given. Stem: “Why did Maria cancel the meeting?” The cancellation is real, but no answer evidence exists.
• B1 = NO; B3 = YES: Audio: “The city delayed the riverside path project after residents asked for more safety lighting.” Stem: “Why did the city cancel the riverside path project?” Same project, wrong event: B1 fails; B3 passes.
• B2 = YES; D3 = NO: Audio is neutral: “I could live with that.” Transcript adds “[sarcastically].” Stem: “What was the speaker’s attitude?” The editor note can be located, but it is not audible evidence.
• B3 = NO: Audio discusses a library renovation. Stem: “What was the speaker’s favorite childhood pet?” No part of the recording makes that a source-based question. B1 may also fail if the stem’s assumption is treated as an unsupported fact.

B2 records where evidence is. C1 asks whether there is enough evidence. C2 checks the AI-designated key. C3 checks whether the option set has exactly one defensible best answer. E3 checks each distractor after listening. F1 checks whether options mean different things. C2/C3/E3/F1 can all flag one competing answer, so separate ratings need a clear purpose.

Description examples

• C2 = NO; C3 = YES: The option set has one uniquely supported answer, but the AI marked a different option as the key. There is a wrong key, not multiple defensible answers.
• C2 = YES; C3 = NO: The designated key is supported, but another option is also supported. The key is not uniquely best.
• F1 = NO; C3 can be YES: Two wrong options duplicate each other, while a separate option is uniquely supported. F1 identifies the duplicate; C3 can still pass because there is one defensible key.
• E3 = NO; C3 = NO: A distractor is also supported by the source. E3 identifies that distractor; C3 summarizes the item-level consequence.

Actual question examples

• C2 = NO; C3 = YES: Audio: “The bus was cheaper, so I took it.” Stem: “Why did the speaker take the bus?” Options: A) It was cheaper. B) It was faster. C) It was more comfortable. D) It stopped closer. If the AI key is B, C2 fails; A is still the unique best answer, so C3 passes.
• C2 = YES; C3 = NO; E3 = NO: Audio: “I took the bus because it was cheaper, and because it stopped outside my office.” Stem: “Why did the speaker take the bus?” Options: A) It cost less. B) It stopped outside the office. C) It was faster. D) It was more comfortable. If A is the designated key, it is supported, but B is also correct.
• F1 = NO; C3 = YES: Audio: “The ferry left at 8:10.” Stem: “When did the ferry leave?” A) 8:10. B) 8:30. C) 9:00. D) 9:00. A is the unique supported key, but C and D duplicate one another.

D1 excludes answers available from general knowledge alone. D2 excludes clues built into the written stem/options and excludes background knowledge from that check. D3 requires the evidence to be audible. B3 asks whether the question is substantively related to the source.

Description examples

• D1 = NO; D2 = YES: A knowledgeable learner already knows the answer, but the options themselves do not give it away. World knowledge is the shortcut; written design is not.
• D1 = YES; D2 = NO: No one could know the source-specific reason from general knowledge, but one option is conspicuously longer or is the only one that fits the stem. Source-specificity does not prevent a written clue.
• D2 = YES; D3 = NO: The printed options do not reveal the key, but the answer depends on an editor’s note absent from the audio.
• B3 = YES; D1 = NO: The question is directly about something in the recording, yet a learner could answer it from common knowledge without listening.

Actual question examples

• D1 = NO; D2 = YES: Audio discusses France and names Paris. Stem: “What is the capital of France?” Options: Paris / Rome / Madrid / Berlin. The options do not reveal the key by form, but many learners know it without the recording. B3 may pass because the recording is about France; D1 fails.
• D1 = YES; D2 = NO: Stem: “Why did the shuttle launch get canceled?” Options: A) Because severe crosswinds made the launch unsafe during the morning window and the crew had no later slot. B) Cost. C) Staffing. D) Scheduling. World knowledge cannot establish the reason (D1 passes), but the uniquely long detailed option can cue the key (D2 fails).
• D2 = YES; D3 = NO: Audio is neutral: “That was helpful.” Transcript: “[sarcastically] That was helpful.” Attitude options are balanced. The text does not reveal which option is keyed, but the audio itself does not carry the editor’s claimed sarcasm.

D2 is an item-level test: hide the audio and ask whether the key can be guessed from the written item. E1–E4 inspect each distractor: plausible before full comprehension, source-connected, definitely wrong after comprehension, and comparable in specificity. An implausible distractor may cause both E1 and D2 to fail, but the scores answer different questions.

Description examples

• E1 = NO; D2 can be YES: One option is absurd, but three other choices remain plausible. The option set has a weak distractor; the key still cannot be chosen without listening.
• E1 = YES; D2 = NO: Every distractor is believable, but the key is much longer/more detailed, so the written form points to it.
• E1 = YES; E2 = NO: A distractor is a believable possible reason in general, but it is unrelated to this recording and not a likely misunderstanding of it.
• E2 = YES; E3 = NO: A distractor is drawn from the passage and sounds plausible, but the audio actually supports it too.
• E1–E3 = YES; E4 = NO: Distractors are believable, source-related, and wrong, but the key is much more specific than all of them.

Actual question examples

• E1 = NO; D2 can be YES: Audio: “Mina took the bus because it was cheaper than the train.” Stem: “Why did Mina take the bus?” A) It cost less. B) It was faster. C) It stopped near her home. D) It was made of cheese. D is implausible (E1 fails), while A/B/C leave the key uncertain without audio (D2 can pass).
• E1 = YES; D2 = NO; G2 = NO: Audio: “The committee rejected the proposal because the narrow walkway would block emergency access at crowded events.” Stem: “Why was the proposal rejected?” A) The narrow walkway would block emergency access at crowded events. B) It cost too much. C) It would delay opening. D) It required more staff. B–D are plausible reasons, but A is conspicuously longer and more detailed.
• E1 = YES; E2 = NO: Audio: “The shop closed early because the freezer failed.” Stem: “Why did the shop close early?” A) The freezer stopped working. B) A storm damaged the roof. C) The manager lost a key. D) A delivery was late. B–D are plausible causes generally, but unless they connect to the passage or a likely mishearing, E2 fails.
• E2 = YES; E3 = NO: Audio: “The shop closed early because the freezer failed and the manager had sent all the staff home.” Stem: “Why did the shop close early?” Options: A) The freezer failed. B) The manager sent staff home. C) A delivery arrived late. D) The roof leaked. B is source-connected and plausible, but it is also stated as a reason; it is not definitely wrong.
• E4 = NO; G2 = NO: Stem: “Why was the plan changed?” A) Because the access ramp was too steep for emergency use during crowded events. B) Cost. C) Timing. D) Staffing. The key’s specificity/length stands out; E4 and G2 may both flag it. Consider making G2 a specific subcheck of E4.

F1 checks whether options mean different things. F2 checks whether each option is self-contained, without referring to or combining with another. F3 checks whether options answer the same kind of question, when the stem implies a type. F4 checks whether their grammar fits the stem comparably. C3 is the final one-best-answer decision. A set can pass one property and fail another.

Description examples

• F1 = NO; F2 = YES: Two options are paraphrases, but each makes sense independently.
• F1 can be YES; F2 = NO: An option has a distinct meaning but refers to another option, such as “the opposite of A”; it cannot be judged independently.
• F3 = NO; F4 = YES: All choices fit grammatically, but a “Why?” question mixes reasons, dates, and people.
• F3 = YES; F4 = NO: All choices are reasons, but only one fits the stem’s grammar. A1 may also fail if that fit requires repairing an option.
• C3 = NO; F1 = YES: Two semantically distinct options are both supported by the recording. The options do not overlap, but the answer is not unique.
• F3 only applies when the stem implies a response type. If the stem asks for an open-ended interpretation with no single category, do not force this criterion into YES or NO; decide whether the form needs a separate Not Applicable response.

Actual question examples

• F1 = NO; F2 = YES: Stem: “Why did the speaker object?” A) The cost increased. B) It became more expensive. C) Safety rules changed. D) The deadline moved. A and B are distinct, independently readable options with overlapping meanings.
• F2 = NO; F1 can be YES: Stem: “Why did the speaker object?” A) The cost increased. B) The opposite of A. C) The deadline moved. D) Safety rules changed. B has a distinct logical meaning, but the learner must interpret A first; F2 fails.
• F3 = NO; F4 = YES: Stem: “Why did the speaker leave?” A) Because the room was closing. B) At 6 p.m. C) Her colleague. D) To catch a bus. The options fit as written, but they mix reason, time, person, and purpose.
• F3 = YES; F4 = NO: Stem: “The speaker objected ___.” A) to the extra fee. B) against the extra fee. C) at the extra fee. D) with the extra fee. All are intended as reasons/objects of objection, but only one construction fits naturally. A1 may also be implicated.
• C3 = NO; F1 = YES: Audio: “I left because the room was noisy, and because the meeting had run late.” Stem: “Why did the speaker leave?” A) The room was noisy. B) The meeting had run late. C) She needed to catch a bus. D) She had another appointment. A and B are different meanings, and both are supported, so uniqueness fails without option overlap.

G1 checks whether source-word matching lets the learner bypass the intended interpretation; exact wording can be legitimate for direct-detail questions. G2/G3 inspect written option cues such as key length or extreme wording. D2 checks whether the item can be guessed from the printed material alone. G4 checks one item revealing another; I1 checks repeated target propositions. E4 is close to G2 because both concern specificity/detail.

Description examples

• G1 = NO; D2 = YES: An item assigned to inference asks what an unusual phrase implies, and only the key repeats that phrase. A learner can match the audio phrase to the option without doing the intended interpretation; the printed item alone still does not reveal the key.
• D2 = NO; G1 = YES: The item reveals the key by option length, while no option unusually repeats the source wording.
• G4 = NO; I1 = YES: Two questions test the same proposition, but neither reveals the other’s answer. They are duplicates without leakage.
• I1 = NO; G4 = YES: Two questions test different facts, but one option states the answer to the other. There is leakage without duplication.
• E4/G2 = NO; G1/G3 = YES: One option is much more detailed, while its vocabulary does not copy the audio and no extreme wording is used.

Actual question examples

• G1 = NO; D2 = YES: Assigned inference: the audio uses “shelved for now,” and the key alone echoes that phrase. Matching it after hearing may bypass interpreting the proposal’s status, while the written options alone do not identify the key. If the assigned task were direct detail (“What phrase did the chair use?”), the same overlap could be valid.
• D2 = NO; G1 = YES: Stem: “Why was the launch canceled?” A) Because severe crosswinds made the launch unsafe during the morning window and no later slot was available. B) Cost. C) Staffing. D) Timing. The long key gives a written clue; assume the recording uses different wording, so G1 passes.
• G4 = NO; I1 = YES: Q2: “Where did the group meet?” Q5: “Which building did the group use?” Both ask the same proposition. If neither gives a clue to the other’s answer, I1 fails while G4 may pass.
• G4 = YES; I1 = NO: Q2 asks when the group met. Q5 asks why the speaker left; one Q5 option says “because the meeting began at 8,” directly giving Q2’s answer. The targets differ, but there is cross-item leakage.

H1 checks whether an item’s assigned operation label matches its primary answer target and the main route to that answer. Classify what the question is asking the learner to identify; do not count every small mental step as a separate operation. H2 is a prompt for deciding H1, not another score. I2 checks the observed operation counts across the set. With a complete set of exactly 2/2/2/2, all H1 matches necessarily make I2 pass. I2 can still pass when H1 fails on two offsetting items.

Operation-label checklist

1. Read the stem and identify the answer target: overall point, local fact, unstated conclusion, or speaker stance/feeling/purpose.
2. Check the source and key. What evidence actually gets the learner to the answer? Is it stated, or must the learner combine clues?
3. Choose the main target: gist = overall point/purpose; detail = a local fact stated in the recording; inference = an unstated conclusion, not mainly a stance/feeling/purpose; attitude/intention = the speaker’s view, feeling, or communicative purpose, whether stated directly or inferred.
4. Compare that classification with the assigned label. Mark H1 NO only for a clear mismatch; mark UNCLEAR if the item genuinely mixes targets or the source does not settle it.
5. Record a secondary process if useful, but use one primary label for the quota count. Do not penalize a good item just because its label is wrong.

Description examples

• H1 = YES: Assigned detail; the stem asks what time the doors open; the recording states “six.” The answer target is one local fact.
• H1 = NO, item still usable: Same clear detail question, but it was assigned inference. The question works; the label does not.
• Inference vs attitude/intention: If the stem asks how the speaker feels, classify the answer target as attitude even when the learner must infer that feeling. Use inference for an unstated conclusion that is not primarily a stance, feeling, or purpose.
• H1 = NO; I2 = YES: In a 2/2/2/2 set, one assigned inference item is actually detail and one assigned detail item is actually inference. Both H1 labels mismatch, but the observed totals still equal 2/2/2/2.
• H1 = NO; I2 = NO: One assigned inference item is actually detail and no other item offsets it. The observed set has 3 detail and 1 inference.
• No valid counterexample: In a complete, correctly counted 2/2/2/2 set, “all H1 pass but I2 fails” is impossible. If a question is missing, record a set-completeness failure separately.

Actual question examples

• Works for H1: Assigned operation: detail. Audio: “The doors open at six.” Stem: “What time do the doors open?” Options: A) 5:00. B) 6:00. C) 7:00. D) 8:00. Key: B. The requested label fits the explicit local fact.
• Fails H1 only: Keep that audio, stem, options, and key exactly the same, but assign “inference.” It remains a perfectly usable detail question. The operation label is the problem.
• Works for inference: Assigned operation: inference. Audio: “The last bus left before Jun finished work; she reached the stop after dark.” Stem: “Why did Jun get home late?” The learner must combine the timing clues; the answer is not stated in one sentence.
• Attitude even when inferred: Audio: “I expected the talk to drag, but I ended up taking notes the whole time.” Stem: “How did the listener feel about the talk?” The answer target is the listener’s attitude; inference may be part of the route.
• Offsetting H1 failures; I2 still passes: In the same requested 2/2/2/2 set, an assigned-inference item asks for an explicitly stated time, while an assigned-detail item asks why two events together caused a delay. Each label mismatches, but the observed counts remain 2/2/2/2.
• I2 fails as well: If only the assigned-inference time question is mislabeled as detail, the set has 3 detail and 1 inference. The item fails H1 and the set fails I2.
• H2 is not a second failure: “Could the learner answer from one explicit sentence?” is a diagnostic question. If it explains why an inference label mismatches, record the reason under H1 once.

B2 checks whether a reviewer can locate evidence for one question. I4 checks whether questions draw from an appropriate spread of the recording. I1 checks repeated targets; I3 checks answer-position patterns. These are different levels of analysis.

Description examples

• B2 = YES; I4 = NO: Every question has a traceable evidence span, but nearly all spans come from the first minute despite a full-recording coverage requirement.
• B2 = NO; I4 may be YES: Questions are distributed across the whole recording, but the evidence needed for one item cannot be located confidently.
• I3 = NO; G4 can pass: Correct answers repeatedly occupy option A, but no item gives away another item’s answer.
• G4 = NO; I3 can pass: Answer positions look varied, but one item reveals another item’s answer.

Actual question examples

• B2 = YES; I4 = NO: All eight items cite exact audio spans, but seven cite 00:20–00:40 even though the task specification requires coverage across the full recording.
• B2 = NO; I4 may be YES: Items cite 00:30, 02:10, 04:25, and 06:00, but Q4 asks “Why did the speaker change jobs?” and no reason can be found at or around its cited span.
• I3 = NO; G4 = YES: Keys are A, A, A, A, while each question tests a separate fact and none supplies another answer.
• G4 = NO; I3 = YES: Keys are B, D, A, C. Q4 asks what time the event began and includes “at 8 a.m.” as an option; Q1 asks for that time. Q4 leaks Q1’s answer despite the varied key positions.