What happens in this stage?
The corpus is processed using language-appropriate NLP tools. The output should allow the project to connect each sampled lemma to all the sentences in which it appears.
Core processing steps
| Step | Description | Example output |
|---|---|---|
| Sentence splitting | Corpus text is divided into individual sentences. | sentence_id = en_00018452 |
| Tokenisation | Each sentence is split into tokens. | Could | you | send | me | the | report | by | Friday | ? |
| POS tagging | Each token receives a part-of-speech tag. | send = VERB |
| Lemmatisation | Inflected forms are linked to dictionary-style lemmas. | sent, sends, sending → send |
| Lemma statistics | Raw frequency, frequency per million, dispersion and ARF are calculated. | arf_per_million = 58.7 |
Example token row
| Column | Example value |
|---|---|
language | en |
sentence_id | en_00018452 |
token | send |
lemma | send |
pos | VERB |
token_index | 3 |
Example lemma statistics row
| Column | Example value |
|---|---|
lemma | send |
pos | VERB |
raw_frequency | 1,284 |
frequency_per_million | 64.2 |
dispersion | 0.82 |
arf_per_million | 58.7 |
Output folders
| Output | Folder |
|---|---|
| Token files | Drive/data/interim/{lang}/tokens/ |
| Lemma-sentence index | Drive/data/interim/{lang}/lemma_sentence_index/ |
| Lemma statistics | Drive/data/interim/{lang}/lemma_stats/ |
| Filtered lemmas | Drive/data/interim/{lang}/filtered_lemmas/ |
Checks and risks
- POS tagging and lemmatisation quality may vary by language.
- Czech and other morphologically rich languages require particular care.
- Content-word filtering should normally use
NOUN,VERB,ADJandADV. - Frequency is supporting evidence only; it is not a direct CEFR-level assignment.