What happens in this stage?
Lemmas are filtered and sampled so that the project can test the workflow without tagging an unmanageable number of sentences. The current pilot design samples candidate lemmas that pass a minimum ARF threshold and then extracts sentence examples for those lemmas.
Current sampling settings
| Setting | Current pilot value | Reason |
|---|---|---|
| Languages | en, fr, es, de, cs |
Five-language feasibility pilot. |
| Content POS | NOUN, VERB, ADJ, ADV |
Focuses on lexical content items. |
| Minimum ARF per million | ≥ 50 |
Reduces unstable or very low-dispersion items. |
| Initial sample | 15 lemmas per language |
Small enough for testing but large enough to expose workflow issues. |
| Sentence extraction | all sentences for selected lemmas |
Allows functional distribution to be inspected by lemma. |
Example sampling row
| Column | Example value |
|---|---|
language | en |
lemma | send |
pos | VERB |
arf_per_million | 58.7 |
sample_status | selected |
sentence_count | 50 |
Example sampled sentence row
| Column | Example value |
|---|---|
row_id | en_send_0001 |
language | en |
lemma | send |
sentence | Could you send me the report by Friday? |
Output folders
| Output | Folder |
|---|---|
| Filtered lemma lists | Drive/data/interim/{lang}/filtered_lemmas/ |
| Sampled lemma files | Drive/data/interim/{lang}/samples/ |
| Sentence samples for tagging | Drive/data/interim/{lang}/samples/ |
Checks and risks
- Sampling should not be mistaken for final vocabulary selection.
- ARF helps with dispersion, but does not determine CEFR level.
- Random sampling should be recorded so that the run is reproducible.
- All sentence rows need stable
row_idvalues before LLM tagging.