Glossary
73 terms, each defined in a sentence or two and linked to the lesson where it is taught.
A
- Activation function
A simple curved function applied after each layer, such as ReLU or GELU. Without it, any stack of layers would collapse into a single matrix.
- Adam
An optimiser that combines momentum with a separate step size for every parameter, based on the recent size of its gradients. The default for training transformers.
- Attention
The operation that lets each token gather information from other tokens. Scores are dot products of queries with keys, softmax turns them into weights, and the output is a weighted mix of values.
- Attention head
One of several attention operations that run side by side in a layer, each with its own weights, so that different relationships between tokens can be followed at once.
- Augmentation
Making extra training examples by altering real ones in ways that keep the label true, such as shifting or flipping an image.
- Autoregressive
Generating a sequence one token at a time, feeding each new token back in as input for the next.
B
- Backpropagation
The algorithm that computes the gradient of the loss for every parameter by applying the chain rule backwards through the network, one operation at a time.
- Batch
A group of training examples processed together in one step. Using a small random batch gives a noisy but cheap estimate of the gradient.
- Bayes' rule
The rule for updating a belief on new evidence: . The belief afterwards is proportional to how well the evidence fits, times the belief before.
- Bias
A learned number added to a neuron's weighted sum. It shifts the point at which the neuron responds.
- Bigram model
A language model that predicts each token from only the one before it, using a table of counts.
- Byte-pair encoding
A way of building a tokeniser: start from single characters and repeatedly merge the most frequent neighbouring pair into one new token.
C
- Causal mask
The rule that stops a token from attending to tokens that come after it, so the model cannot see the answer it is being trained to predict.
- Chain rule
The rule for differentiating a function of a function: multiply the derivatives of the links, .
- Classifier
A model that assigns an input to one of a fixed set of classes, usually by producing a probability for each class and picking the largest.
- Clustering
Grouping unlabelled data so that similar items end up together. The k-means method alternates between assigning each point to its nearest centre and moving each centre to the mean of its points.
- Context length
The number of tokens a model can look back over when predicting the next one.
- Context window
All the text a model can see at once: instructions, the conversation so far and the reply in progress. Anything outside it cannot affect the output.
- Convolution
Sliding a small grid of weights, called a filter, across an image and taking a dot product at every position. The same weights are reused everywhere.
- Cosine similarity
The dot product of two vectors divided by both their lengths. It measures direction only, from (opposite) to (the same).
- Cross-entropy
The loss used for predicting categories or tokens: the average of , where is the probability the model gave to the correct answer.
D
- Derivative
The slope of a function at a point: how fast the output changes when the input is nudged.
- Determinant
The factor by which a matrix scales areas. A determinant of zero means the matrix flattens space and cannot be undone.
- Diffusion model
A model trained to predict the noise that was added to an image. It generates a picture by starting from pure noise and removing a little at a time.
- Discount
In reinforcement learning, the factor between 0 and 1 by which a reward is multiplied for every step of delay.
- Dot product
Multiply matching entries of two vectors and add the results. It is large when the vectors point the same way and zero when they are at right angles.
- Dropout
Switching off a random share of the neurons on each training step, so that the network cannot rely on any single one.
E
- Early stopping
Ending training when the loss on held-out data stops improving, and keeping the weights from the best step.
- Embedding
A learned vector that stands for a token. Tokens used in similar ways end up with nearby vectors.
- Entropy
The average surprise of a distribution, in bits. It is zero when one outcome is certain and largest when all are equally likely.
- Epoch
One pass through the whole training set.
F
- Fine-tuning
Continuing to train a pre-trained model on a smaller dataset chosen for a purpose, starting from the weights it already has.
- Function
A rule that turns each input into exactly one output. Every model is a function whose behaviour is set by its parameters.
G
- Gradient
The vector of partial derivatives of a function, written . It points in the direction of steepest increase.
- Gradient descent
Repeatedly stepping the parameters against the gradient of the loss: .
H
- Hallucination
A fluent, confident statement from a model that is not true. It follows from always choosing a likely next token, whether or not the facts are known.
- Hyperparameter
A setting chosen by the person training the model and not learned from data, such as the learning rate or the number of layers.
L
- Layer
One stage of a network: multiply the input by a weight matrix, add biases, and usually apply an activation function.
- Layer normalisation
Rescaling each token's vector to mean 0 and variance 1, followed by a learned scale and shift. It keeps values in a steady range in deep networks.
- Learning rate
The size of each gradient descent step, written . Too small is slow, and too large makes the loss diverge.
- Logarithm
The inverse of an exponential: is the power to which must be raised to give . It turns multiplication into addition.
- Logit
A raw score produced by the last layer of a network, before softmax turns the scores into probabilities.
- Loss
A single number measuring how wrong a model's predictions are. Training tries to make it small.
M
- Matrix
A rectangular grid of numbers. Multiplying a vector by a matrix transforms it into a new vector.
- Matrix factorisation
Writing a large table as the product of two much smaller ones. In a recommender, each person and each item gets a vector, and their dot product predicts a rating.
- Mean
The average: add up the values and divide by how many there are.
- Momentum
Keeping a running velocity of past gradient steps so that consistent directions build up speed and zig-zags cancel out.
N
- Nearest neighbours
Finding the stored vectors that are closest to a given one. Used to classify by example, to search and to recommend.
- Neuron
A unit that takes a dot product of its inputs with its weights, adds a bias, and applies an activation function.
O
- Overfitting
When a model memorises details of its training data and gets worse on new data. It shows up as training loss falling while held-out loss rises.
P
- Parameter
A number inside a model that training adjusts. Weights, biases and embeddings are all parameters.
- Perplexity
The exponential of the cross-entropy loss, . Roughly, the number of equally likely options the model is choosing between.
- Pooling
Shrinking a feature map by keeping one value, usually the largest, from each small block.
- Pre-training
The first and largest stage of training a language model: predicting the next token on a very large body of general text.
- Precision and recall
Precision is the share of a model's positive answers that were right. Recall is the share of the real positives that it found.
- Prompt
The text given to a model to continue. It changes what the model is conditioned on, not its weights.
Q
- Q-value
In reinforcement learning, the total discounted reward expected from taking a given action in a given state and acting well afterwards.
R
- Regularisation
Anything added to training that makes fitting the training data harder in order to do better on new data, such as weight decay, dropout or early stopping.
- Reinforcement learning
Learning which actions to take from rewards alone, with no example answers. A reward may arrive long after the action that earned it.
- Residual connection
Adding a sub-layer's output to its input instead of replacing it. This gives gradients a direct path through deep networks.
- Retrieval
Finding passages relevant to a question by comparing embedding vectors, and adding them to the context so that the model can read the answer instead of recalling it. Also called RAG.
S
- Saddle point
A point where the gradient is zero but which is a valley in one direction and a hilltop in another.
- Self-supervised learning
Training on labels made from the data itself, by hiding part of each example and predicting it. Next-token prediction is the main example.
- Softmax
The function that turns a list of scores into probabilities that sum to 1: .
- Supervised learning
Training on examples that come with the correct answer, called the label.
T
- Temperature
A number that divides the logits before softmax. Low values make the model predictable, and high values make it varied.
- Tensor
An array of numbers with any number of axes. A vector has one axis, a matrix has two and a colour image has three.
- Token
A piece of text, such as a character, a word or part of a word, that a model treats as one unit and represents by an id.
- Transformer
A network built from repeated blocks of attention and an MLP, with residual connections and layer normalisation. The architecture behind GPT and most current language models.
- Transpose
A matrix flipped over its diagonal so that rows become columns, written .
V
- Vector
An ordered list of numbers. It can be pictured as an arrow with a length and a direction.
W
- Weight
A learned number that multiplies an input. The weights of a layer form a matrix.
- Weight decay
A penalty on large weights that shrinks every weight slightly towards zero on each step.