Everything trained so far learned from numbers that were already numbers. The GPT’s raw material is text, and this chapter is the border crossing: three small steps that turn characters into values the loop can reach — ending with the smallest language model that exists, trained to perfection in a hundred rounds.

The working file is examples/TextBecomesNumbers — run it with make -C examples/TextBecomesNumbers run.

Step one: characters become ids

A vocabulary is a fixed list of the characters the model may meet; a character’s id is simply its position in that list. The example’s vocabulary is "abcd" and its text is "abca":

  'a' is id 0
  'b' is id 1
  'c' is id 2
  'a' is id 0

That scan is the tokenizer. Production models use larger vocabularies whose entries are word fragments rather than single characters — and Gpt.mica, at the course’s end, simply uses each character’s code directly, a fixed 128-entry vocabulary. The crossing is always exactly this: text in, ids out.

Step two: ids become one-hot rows

Arithmetic cannot eat an id — id 2 is not “twice id 1”; the numbers are names, not quantities. The honest number spelling of a name is the one-hot row: all zeros, a single one at the id’s column.

  id 0  →  1 0 0 0
  id 1  →  0 1 0 0        one row per character of the text,
  id 2  →  0 0 1 0        as wide as the vocabulary
  id 0  →  1 0 0 0

And one-hot rows have a superpower: a one-hot row times a table is exactly a row lookup. Multiply the text’s one-hot matrix by any table with one row per vocabulary entry, and each character receives its own row of that table:

oh := OneHot(ids);
emb := oh * e;        { each character, replaced by its row of the table e }

Step three: the table is knobs

That table e is the embedding: each character’s row is its meaning, as a vector. And here is the chapter’s punchline — nothing distinguishes that table from every guess this course has trained. Its entries are knobs; the tape reaches them through the product; blame flows into exactly the rows the text touched. The meaning of a character is learned, not designed.

The example proves it with the smallest language model that exists — a bigram: one table scoring which character follows which. The lookup is the one-hot trick in its column form (a matrix times a one-hot vector selects a column), the loss is the miss against the observed next character, and the loop is chapter 3’s, untouched:

  round 1: loss 3.000000
  round 100: loss 0.000000

what the table learned, one column per current character
  after id 0: a 0.000000  b 1.000000  c 0.000000  d 0.000000
  after id 1: a 0.000000  b 0.000000  c 1.000000  d 0.000000
  after id 2: a 1.000000  b 0.000000  c 0.000000  d 0.000000
  after id 3: a 0.000000  b 0.000000  c 0.000000  d 0.000000

The text "abca" held three transitions — a→b, b→c, c→a — and the table learned each one to 1.000000 exactly. Read the last line closely: d never appeared in the text, and its column is exactly as it started. The record never read it, and blame cannot reach what was never touched — chapter 3’s rule, now visible as a property of what a model does and does not learn.

What to take forward

The bigram is a real language model with a real flaw: its prediction depends on one character of context, always. "abca" could be learned perfectly because one character was enough — real text is not so kind, and the next character routinely depends on words that ended long ago. What the machine needs is a way for every position to consult everything before it, and to decide for itself what matters. That mechanism has a name: attention.