Choosing your data
A model can only become what its data is. Decide what you want it to do, then pick text that shows exactly that, and enough of it.
7 minute readOne thing to try6 questions
Start from the job
Before collecting anything, finish this sentence: "I want a model that writes…"
- …names that sound like a particular language,
- …product titles in the style of my shop,
- …short poems in the style of my own notes,
- …answers to a fixed set of questions.
Every example in your data should look like the output you want. Anything else teaches it something else.
See the effect immediately
This model learns nothing but which letter follows which, and it already picks up the style of its data. Press Make names, then switch the data and do it again.
Even the simplest model copies the shape of what it is given. Bigger models copy it far more faithfully.
How much is enough?
For the tiny transformer in this course, trained on characters:
| Amount of text | What to expect |
|---|---|
| Under 2,000 characters | It memorises quickly. Fine only for a fixed set of answers |
| 5,000 to 50,000 characters | Learns the style, and starts to make new examples |
| 100,000 or more | Much steadier, slower to overfit |
Variety matters as much as size. Five thousand names from one family sound like one family.
Where data comes from
- Your own writing, notes or records. The best source: you know it, and you have the right to use it.
- Public, openly licensed text. Check the licence before using it.
- Not private messages, other people's personal details, or text you do not have permission to use.
What to remember
- Decide the job first. Every example should look like the output you want.
- More text delays memorising, and varied text gives varied output.
- Use data you have the right to use, and never other people's private information.
Check yourself
3 questions. Each one explains itself once you answer. Get 2 right to complete the lesson.
0 of 3 answered
Next lesson: Cleaning your text