Purpose of reviewer work
Reviewers inspect initial model proposals, informed Pass 2 decisions, adjudication outputs and review flags. Their task is not to rubber-stamp the system but to decide whether each sense and function label is defensible.
What one row contains
| Layer | Main question |
|---|---|
| Target occurrence | Which token, lemma and POS is being analysed? |
| Lexical sense | What does the target lemma mean in this sentence? |
| Sentence function | What is the whole sentence doing communicatively? |
| Pass 2 review | Did the informed reviewer accept, change or remain uncertain about Pass 1? |
| Interaction note | Did the proposed function help or distort the sense decision, or vice versa? |
Core principle
Sense and function are distinct decisions. Reviewers may use one as contextual evidence when evaluating the other, but should not assume that a particular function proves a particular sense or that a sense proves a function.
Review sequence
- Read the sentence and identify the target occurrence.
- Check the proposed lexical sense against the approved inventory.
- Check the whole-sentence communicative function against the taxonomy.
- Inspect Pass 1 rationales and Pass 2 accept/change/uncertain decisions.
- Check the interaction notes for anchoring or circular reasoning.
- Record separate expert decisions for sense and function.
Recommended decisions
| Decision | Use when |
|---|---|
accept | The proposed label is defensible and no clearly better alternative exists. |
change | Another approved sense or taxonomy function is clearly better. |
uncertain | Context is insufficient or two analyses remain plausible. |
reject | The row is malformed, unusable, not in the target language or lacks interpretable context. |
Priority review cases
- Pass 2 changes Pass 1.
- Pass 2 marks uncertainty or recommends human review.
- Either annotation has low confidence.
- Sense is OTHER or UNCLEAR.
- The interaction note suggests that one label may have improperly determined the other.
- The approved inventory or taxonomy appears incomplete.
Suggested human columns
expert_sense_decision
expert_sense_id
expert_sense_comment
expert_function_decision
expert_function_id
expert_function_comment
adjudication_requiredBlind validation
Blind-validation rows should be reviewed without revealing the informed Pass 2 answer until the reviewer has made a decision. Blind, informed and human outcomes can then be compared to assess reliability and anchoring.
Example
| Sentence | Target sense | Sentence function | Review concern |
|---|---|---|---|
| Apply the cream twice daily. | put a substance onto a surface | giving an instruction | The instruction supports the interpretation but does not itself prove the sense. |
| The nurse applied the cream. | put a substance onto a surface | reporting an event | The same sense occurs with a different function. |