The bigram sees one character of context. Attention is the mechanism that lets every position consult everything before it and decide, position by position, what matters — and it is the reason the architecture at the end of this course is called a transformer. This chapter runs it at the smallest honest size, two positions, with numbers chosen so every step lands on exact halves and quarters.

The working file is examples/Attention — run it with make -C examples/Attention run.

Three questions, asked in bulk

Every position derives three vectors from its embedding. The query: what am I looking for? The key: what do I announce about myself? The value: what do I contribute, if someone attends to me? The trick is that all positions ask all positions at once, as matrix products:

tape
    kt := Transpose(k);
    sc := q * kt;         { all queries against all keys }
    sm := sc + mask;      { no looking forward }
    p := RowSoftmax(sm);  { each row becomes weights that sum to one }
    o := p * vp;          { mix the values by those weights }
    s := Sum(o);
end;

Five recorded lines. Cell [i, j] of the score matrix says how relevant position j looks to position i — every pairing, one product.

        keys  k₁   k₂
   q₁        0.50  0.00      position 1 asks; position 1's key answers best
   q₂        0.75  0.75      position 2 asks; both keys answer equally

The mask, and the soft choice

A model that predicts the next character must not read it. The causal mask adds negative infinity to every future cell, so the softmax hands it a weight of exactly zero — the future contributes nothing, not almost nothing:

  weights row 1: 1.000000 0.000000     position 1 sees only itself
  weights row 2: 0.500000 0.500000     position 2 splits its tied scores evenly

Each row now sums to one, and this is the design’s quiet elegance: attention is soft. Position 2 did not choose a winner — its two scores tied, and it takes exactly half of each contribution. A hard choice would be a step function, and blame cannot flow through a cliff; the soft mix is differentiable everywhere, which is what keeps the whole mechanism learnable.

The output mixes the values by those weights:

  position 1: 2.000000 4.000000        its own value, passed through
  position 2: 5.000000 10.000000       the exact average of both values

Blame flows through it, exactly

Attention is not a fixed recipe bolted onto the network — the query, key, and value derivations are knobs, trained like everything since chapter 3. The example walks the blame for the summed output backward through the mix, the softmax, the mask, and the transpose, down to the keys:

blame reaching the keys
  row 1: 0.000000 -4.500000
  row 2: 0.000000 4.500000

Exact, like everything else on this page — and readable: raising key 2’s second channel would tilt position 2’s even split, moving weight between a value row worth little and one worth much, and the signs say which way. The tape walked a softmax and got the exact answer; the per-link rules of chapter 7 covered even that.

What to take forward

What you ran is one attention head, whole. In Gpt.mica’s training window these same five sentences appear nearly verbatim — preceded by the embedding lookup you know from chapter 9, followed by a residual add, a layer norm, and the Tanh feed-forward pair you know from chapters 5 and 6, with a scale factor on the scores to keep their variance width-independent. There is exactly one page left to turn: assembling those known parts into the whole machine, and reading it — the GPT.