What happens in this stage?
Raw corpus files are collected, checked, documented and placed into the agreed Google Drive folder structure. This stage ensures that later processing steps know where each text came from, what language it belongs to, and what restrictions apply to its use.
Inputs
| Input | Description | Stored where? |
|---|---|---|
| Raw corpus files | Original text files or corpus exports for English, French, Spanish, German and Czech. | Drive/data/raw/{lang}/corpus_original/ |
| Metadata | Information about source, language, date, register, corpus origin and processing notes. | Drive/data/raw/{lang}/metadata/ |
| Licensing notes | Notes on what can and cannot be redistributed. | Drive/data/raw/{lang}/licensing/ |
Example
| Field | Example value |
|---|---|
language | en |
source_file | en_news_2024_part001.txt |
sentence | Could you send me the report by Friday? |
sentence_id | en_00018452 |
register | web_or_news |
Outputs
The output is not a final dataset. The main output is a clean, documented corpus folder ready for processing.
- Raw corpus files placed in the correct language folders.
- Metadata and licensing notes recorded.
- Source-level documentation sufficient for later audit.
Checks and risks
- Do not put large raw corpus files in GitHub.
- Do not put restricted or licensed data in a public repo.
- Check that each language uses the agreed folder naming convention.
- Keep metadata even when it feels boring; it becomes important later for validation.