Glossary

73 terms, each defined in a sentence or two and linked to the lesson where it is taught.

A

Activation function

A simple curved function applied after each layer, such as ReLU or GELU. Without it, any stack of layers would collapse into a single matrix.

Lesson: Layers and non-linearity

Adam

An optimiser that combines momentum with a separate step size for every parameter, based on the recent size of its gradients. The default for training transformers.

Lesson: Momentum and Adam

Attention

The operation that lets each token gather information from other tokens. Scores are dot products of queries with keys, softmax turns them into weights, and the output is a weighted mix of values.

Lesson: Attention

Attention head

One of several attention operations that run side by side in a layer, each with its own weights, so that different relationships between tokens can be followed at once.

Lesson: Heads, layers and what they learn

Augmentation

Making extra training examples by altering real ones in ways that keep the label true, such as shifting or flipping an image.

Lesson: Classifying from start to finish

Autoregressive

Generating a sequence one token at a time, feeding each new token back in as input for the next.

Lesson: From logits to language

B

Backpropagation

The algorithm that computes the gradient of the loss for every parameter by applying the chain rule backwards through the network, one operation at a time.

Lesson: Backpropagation by hand

Batch

A group of training examples processed together in one step. Using a small random batch gives a noisy but cheap estimate of the gradient.

Lesson: Momentum and Adam

Bayes' rule

The rule for updating a belief on new evidence: P(A∣B)=P(B∣A) P(A)/P(B)P(A \mid B) = P(B \mid A)\,P(A) / P(B). The belief afterwards is proportional to how well the evidence fits, times the belief before.

Lesson: Bayes' rule

Bias

A learned number added to a neuron's weighted sum. It shifts the point at which the neuron responds.

Lesson: Layers and non-linearity

Bigram model

A language model that predicts each token from only the one before it, using a table of counts.

Lesson: A language model from counting

Byte-pair encoding

A way of building a tokeniser: start from single characters and repeatedly merge the most frequent neighbouring pair into one new token.

Lesson: How tokenisers are built

C

Causal mask

The rule that stops a token from attending to tokens that come after it, so the model cannot see the answer it is being trained to predict.

Lesson: Attention

Chain rule

The rule for differentiating a function of a function: multiply the derivatives of the links, dydx=dydu⋅dudx\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}.

Lesson: The chain rule

Classifier

A model that assigns an input to one of a fixed set of classes, usually by producing a probability for each class and picking the largest.

Lesson: Classifying from start to finish

Clustering

Grouping unlabelled data so that similar items end up together. The k-means method alternates between assigning each point to its nearest centre and moving each centre to the mean of its points.

Lesson: Clustering

Context length

The number of tokens a model can look back over when predicting the next one.

Lesson: Tokens and positions

Context window

All the text a model can see at once: instructions, the conversation so far and the reply in progress. Anything outside it cannot affect the output.

Lesson: Context, prompts and examples

Convolution

Sliding a small grid of weights, called a filter, across an image and taking a dot product at every position. The same weights are reused everywhere.

Lesson: Convolution

Cosine similarity

The dot product of two vectors divided by both their lengths. It measures direction only, from −1-1 (opposite) to 11 (the same).

Lesson: The dot product

Cross-entropy

The loss used for predicting categories or tokens: the average of −ln⁡p-\ln p, where pp is the probability the model gave to the correct answer.

Lesson: Cross-entropy

D

Derivative

The slope of a function at a point: how fast the output changes when the input is nudged.

Lesson: Slope and the derivative

Determinant

The factor by which a matrix scales areas. A determinant of zero means the matrix flattens space and cannot be undone.

Lesson: Matrices move space

Diffusion model

A model trained to predict the noise that was added to an image. It generates a picture by starting from pure noise and removing a little at a time.

Lesson: How image generators work

Discount

In reinforcement learning, the factor γ\gamma between 0 and 1 by which a reward is multiplied for every step of delay.

Lesson: Reinforcement learning

Dot product

Multiply matching entries of two vectors and add the results. It is large when the vectors point the same way and zero when they are at right angles.

Lesson: The dot product

Dropout

Switching off a random share of the neurons on each training step, so that the network cannot rely on any single one.

Lesson: Regularisation

E

Early stopping

Ending training when the loss on held-out data stops improving, and keeping the weights from the best step.

Lesson: Train a tiny transformer

Embedding

A learned vector that stands for a token. Tokens used in similar ways end up with nearby vectors.

Lesson: Embeddings

Entropy

The average surprise of a distribution, −∑plog⁡2p-\sum p \log_2 p in bits. It is zero when one outcome is certain and largest when all are equally likely.

Lesson: Entropy and information

Epoch

One pass through the whole training set.

Lesson: Batches, epochs and schedules

F

Fine-tuning

Continuing to train a pre-trained model on a smaller dataset chosen for a purpose, starting from the weights it already has.

Lesson: Pre-training and fine-tuning

Function

A rule that turns each input into exactly one output. Every model is a function whose behaviour is set by its parameters.

Lesson: Functions

G

Gradient

The vector of partial derivatives of a function, written ∇f\nabla f. It points in the direction of steepest increase.

Lesson: Gradients

Gradient descent

Repeatedly stepping the parameters against the gradient of the loss: θ←θ−η∇L(θ)\theta \leftarrow \theta - \eta \nabla L(\theta).

Lesson: Gradient descent

H

Hallucination

A fluent, confident statement from a model that is not true. It follows from always choosing a likely next token, whether or not the facts are known.

Lesson: Why models make things up

Hyperparameter

A setting chosen by the person training the model and not learned from data, such as the learning rate or the number of layers.

Lesson: Training in practice

L

Layer

One stage of a network: multiply the input by a weight matrix, add biases, and usually apply an activation function.

Lesson: Layers and non-linearity

Layer normalisation

Rescaling each token's vector to mean 0 and variance 1, followed by a learned scale and shift. It keeps values in a steady range in deep networks.

Lesson: The transformer block

Learning rate

The size of each gradient descent step, written η\eta. Too small is slow, and too large makes the loss diverge.

Lesson: Gradient descent

Logarithm

The inverse of an exponential: log⁡bx\log_b x is the power to which bb must be raised to give xx. It turns multiplication into addition.

Lesson: Exponentials and logarithms

Logit

A raw score produced by the last layer of a network, before softmax turns the scores into probabilities.

Lesson: Softmax and temperature

Loss

A single number measuring how wrong a model's predictions are. Training tries to make it small.

Lesson: Loss, a number for how wrong

M

Matrix

A rectangular grid of numbers. Multiplying a vector by a matrix transforms it into a new vector.

Lesson: Matrices move space

Matrix factorisation

Writing a large table as the product of two much smaller ones. In a recommender, each person and each item gets a vector, and their dot product predicts a rating.

Lesson: How recommendations work

Mean

The average: add up the values and divide by how many there are.

Lesson: Sums, averages and the sigma sign

Momentum

Keeping a running velocity of past gradient steps so that consistent directions build up speed and zig-zags cancel out.

Lesson: Momentum and Adam

N

Nearest neighbours

Finding the stored vectors that are closest to a given one. Used to classify by example, to search and to recommend.

Lesson: Distance and nearest neighbours

Neuron

A unit that takes a dot product of its inputs with its weights, adds a bias, and applies an activation function.

Lesson: The neuron

O

Overfitting

When a model memorises details of its training data and gets worse on new data. It shows up as training loss falling while held-out loss rises.

Lesson: Training in practice

P

Parameter

A number inside a model that training adjusts. Weights, biases and embeddings are all parameters.

Lesson: Everything is a number

Perplexity

The exponential of the cross-entropy loss, eLe^{L}. Roughly, the number of equally likely options the model is choosing between.

Lesson: Cross-entropy

Pooling

Shrinking a feature map by keeping one value, usually the largest, from each small block.

Lesson: From filters to a network

Pre-training

The first and largest stage of training a language model: predicting the next token on a very large body of general text.

Lesson: Pre-training and fine-tuning

Precision and recall

Precision is the share of a model's positive answers that were right. Recall is the share of the real positives that it found.

Lesson: Measuring a classifier

Prompt

The text given to a model to continue. It changes what the model is conditioned on, not its weights.

Lesson: Context, prompts and examples

Q

Q-value

In reinforcement learning, the total discounted reward expected from taking a given action in a given state and acting well afterwards.

Lesson: Reinforcement learning

R

Regularisation

Anything added to training that makes fitting the training data harder in order to do better on new data, such as weight decay, dropout or early stopping.

Lesson: Regularisation

Reinforcement learning

Learning which actions to take from rewards alone, with no example answers. A reward may arrive long after the action that earned it.

Lesson: Reinforcement learning

Residual connection

Adding a sub-layer's output to its input instead of replacing it. This gives gradients a direct path through deep networks.

Lesson: The transformer block

Retrieval

Finding passages relevant to a question by comparing embedding vectors, and adding them to the context so that the model can read the answer instead of recalling it. Also called RAG.

Lesson: Retrieval

S

Saddle point

A point where the gradient is zero but which is a valley in one direction and a hilltop in another.

Lesson: Valleys, peaks and saddles

Self-supervised learning

Training on labels made from the data itself, by hiding part of each example and predicting it. Next-token prediction is the main example.

Lesson: Three kinds of learning

Softmax

The function that turns a list of scores into probabilities that sum to 1: pi=ezi/∑jezjp_i = e^{z_i} / \sum_j e^{z_j}.

Lesson: Softmax and temperature

Supervised learning

Training on examples that come with the correct answer, called the label.

Lesson: Three kinds of learning

T

Temperature

A number that divides the logits before softmax. Low values make the model predictable, and high values make it varied.

Lesson: Softmax and temperature

Tensor

An array of numbers with any number of axes. A vector has one axis, a matrix has two and a colour image has three.

Lesson: Tensors and shapes

Token

A piece of text, such as a character, a word or part of a word, that a model treats as one unit and represents by an id.

Lesson: Tokens and positions

Transformer

A network built from repeated blocks of attention and an MLP, with residual connections and layer normalisation. The architecture behind GPT and most current language models.

Lesson: The transformer block

Transpose

A matrix flipped over its diagonal so that rows become columns, written A⊤A^{\top}.

Lesson: Transpose, identity and inverse

V

Vector

An ordered list of numbers. It can be pictured as an arrow with a length and a direction.

Lesson: Vectors

W

Weight

A learned number that multiplies an input. The weights of a layer form a matrix.

Lesson: Layers and non-linearity

Weight decay

A penalty on large weights that shrinks every weight slightly towards zero on each step.

Lesson: Regularisation