Ten chapters built parts. This chapter reads the machine — gpt.mica, a complete, working GPT in one file: it reads a corpus, learns to predict the next character, saves what it learned, loads it back, and writes new text in the corpus’s style. Nothing in it will be new to you. That is the claim this page makes and keeps: you already know every line.

The working file is samples/gpt/gpt.mica — open it beside this page.

The shape, and the count

Four numbers fix the whole model, and the file does the arithmetic in its own comment:

ContextLen = 64;      { how many characters the model sees at once }
Channels   = 256;     { the width of every internal representation }
Hidden     = 1024;    { the feed-forward net's inner width, 4 x Channels as in GPT-2 }
VocabSize  = 128;     { one id per ASCII code point }

The embedding holds VocabSize × Channels values, the three attention projections 3 × Channels × Channels, the layer norm 2 × Channels, the feed-forward pair 2 × Channels × Hidden, and the output projection Channels × VocabSize:

32768 + 196608 + 512 + 524288 + 32768 = 786944 learned numbers

Every one of them is a knob in the sense of chapter 1 — storage the loop nudges — and the whole model is nine tensors, declared like any other Mica variables: the embedding table of chapter 9, the query, key, and value projections of chapter 10, the layer norm’s gain and bias, the feed-forward pair of chapters 5 and 6, and the output projection back to character scores. Their shapes are in their types, so a dimension mistake anywhere in what follows is a build error — never an exception three hours into training.

The window: eighteen sentences

The heart of the file is one tape window. Here it is, whole, with each line’s home chapter:

tape
    h  := oh × e;                 { ch 9: one-hot times table is a row lookup   }
    q  := h × wq;                 { ch 10: what every position is looking for   }
    k  := h × wk;                 { ch 10: what every position announces        }
    vp := h × wv;                 { ch 10: what every position contributes      }
    kt := Transpose(k);
    sc := q × kt;                 { ch 10: all queries against all keys         }
    ss := sc * Scale;             { keep score variance width-independent       }
    sm := ss + mask;              { ch 10: no looking forward                   }
    p  := RowSoftmax(sm);         { ch 10: weights that sum to one              }
    ao := p × vp;                 { ch 10: mix the values by the weights        }
    r₁ := ao + h;                 { residual: refine the embedding, keep it     }
    y  := LayerNorm(r₁, γ, β);    { re-center and re-scale each position        }
    u  := y × w₁;                 { ch 6: expand to the hidden width            }
    ac := TanhMap(u);             { ch 5: the bend that buys depth its power    }
    mo := ac × w₂;                { ch 6: project back down                     }
    r₂ := mo + y;                 { the second residual                         }
    lg := r₂ × wu;                { every position's next-character scores      }
    l  := CrossEntropy(lg, tgv);  { one number: the average surprise            }
end;

That is the transformer — the same block that, stacked and widened, is every GPT in production use. The file is deliberately one block deep: depth is repetition of exactly this, so one block read carefully teaches more than twelve read never. The few lines you did not build yourself are small and honest. The scale 1/√Channels — with 256 channels, exactly 1/16 — keeps score variance width-independent so the softmax starts responsive. The residuals add the block’s input back to its output: the block refines the representation rather than replacing it, which is what lets blame flow through many blocks without fading. LayerNorm re-centers each position, training’s stabilizer. And CrossEntropy is the honest loss for a model that answers with a probability distribution: the average surprise at the true next characters — the squared miss’s role, played by the right verb for probabilities. All four are ordinary Mica in the math unit; open math.mica and read the body of every verb you see.

The rest of the loop, already yours

After the window, the file is chapter 7 and chapter 8 at scale — one backward walk, then Adam on each of the nine masters:

Backward(l);

ge := Gradient(e);
...
AdamStep(address ei, ge, address me, address ve, step, α, β₁, β₂, ε);
...

AdamStep is chapter 8’s two-moment arithmetic as a bulk verb over whole tensors, at the same β₁ = 0.9 and β₂ = 0.999 you ran there. Nine calls, because nine tensors; the loop of chapter 3, because it never changed.

What remains is bookkeeping

The rest of the file is the honest engineering around the loop: the corpus read once and reduced to ids (each ASCII code point its own id — chapter 9’s crossing, at its plainest), a checkpoint that saves exactly the nine tensors behind a named magic number and refuses a wrong file by name, and generation — the trained model reading its own last ContextLen characters, scoring the next, and sampling from the softmax with PickByWeight, driven by one seeded generator. One seed, stated in the constants, is the run’s only randomness: change it and a different but equally reproducible run unfolds.

What to take forward

Read the file end to end once — it repays the hour. Then turn the last page of the course and run it: train your own GPT.