Stage 6: Combined QA and Expert Review

Build the combined dataset

python scripts/06_make_final_dataset.py \
 --samples data/en_sampled_occurrences.csv \
 --sense_pass1 data/en_sense_pass1.csv \
 --sense_pass2 data/en_sense_pass2.csv \
 --sense_pass3 data/en_sense_pass3.csv \
 --function_pass1 data/en_function_pass1.csv \
 --function_pass2 data/en_function_pass2.csv \
 --function_pass3 data/en_function_pass3.csv \
 --output data/en_combined_review.csv

Human review

Reviewers may see the initial sense, initial function, informed review decisions and interaction notes in one workbook. They still make two explicit final decisions: one about the target lemma sense and one about the whole-sentence function. A correction to one does not automatically change the other.

Priority cases

• Pass 2 changes Pass 1.
• Pass 2 marks either annotation uncertain.
• Either annotation has low confidence.
• Sense Pass 2 selects OTHER or UNCLEAR.
• An interaction note reports tension between the proposed sense and function.
• The reviewer suspects a gap in the sense inventory or function taxonomy.

Automated QA

• Every output maps to one sampled row ID.
• Final sense IDs belong to the approved inventory.
• Final function IDs belong to the controlled taxonomy.
• Pass 2 review mode is recorded as informed_review or blind_validation.
• Blind and informed outputs are stored separately.
• Changes, uncertainty, interaction notes, OTHER and UNCLEAR remain visible.
• Model output is never represented as validated CEFR data.

Validation comparison

The pilot should compare informed Pass 2, blind Pass 2 and human decisions. This allows the project to test whether seeing Pass 1 improves final quality and whether it also creates anchoring.

Aggregation

After review, function distributions are calculated by:

language + target_lemma + approved_sense_id

This creates sense-specific functional environment profiles rather than mixing all meanings of a headword.

Cross-language alignment

Reviewed language-specific senses may later be linked to broader concept IDs. Alignment must allow one-to-many and constructional relationships rather than forcing identical inventories.