Since chapter 3 the tape has answered every gradient, and you have been right to trust it. This chapter opens the lid exactly once: you will do the backward walk yourself, link by link, with one rule per link and nothing harder than multiplication — and check your ledger against the tape’s answers to the last digit. After this page, backpropagation is not a term of art; it is bookkeeping you have done.

The working file is examples/BlameFlow — run it with make -C examples/BlameFlow run.

A story of three links

The forward story: a weighted sum, a squaring, a squared miss — two knobs, one input, one target.

tape
    u := a * X + b;
    sq := u * u;
    miss := sq - Target;
    loss := miss * miss;
end;

With a = 0.5, b = −0.25, X = 1.5, Target = 0.5 — every value exact in binary:

forward: u = 0.500000, s = u² = 0.250000, loss = 0.062500

The one question, asked backward

The backward walk asks the same question at every link, ending at the knobs: if this value had been slightly larger, how much would the loss have moved? Call the answer the value’s blame. The walk starts at the loss itself — whose blame is 1, by definition — and each link passes blame to its inputs, scaled by the link’s own local slope:

   forward  ───▶   u = a·X + b   ───▶   s = u²   ───▶   loss = (s − t)²

   backward ◀───  ·X    ·1   ◀───────  ·2u  ◀──────────  ·2(s − t)
              a's    b's

The example’s ledger, each line one link:

the ledger, link by link
  miss's blame: 2 · miss            = -0.500000
  s's blame:    passed through      = -0.500000
  u's blame:    s's · 2u            = -0.500000
  a's blame:    u's · X             = -0.750000
  b's blame:    u's, unchanged      = -0.500000

Three rules covered everything. A squared miss passes 2·miss. A nonlinear link passes its local slope — here 2u for the squaring, and for the bent wire of chapter 5 it would be 1 − tanh², which is why the ledger needs the recorded forward values: the slope depends on where the story crossed the link. A weighted sum passes its blame to each input scaled by that input’s partner — a gets u’s blame times X, b gets it unchanged.

And the check:

the tape's answers
  Gradient(a) = -0.750000   (hand: -0.750000)
  Gradient(b) = -0.500000   (hand: -0.500000)

Identical — because it is the same arithmetic. The tape records each operation’s inputs on the way forward precisely so it can apply these same per-link rules on the way back, in reverse order, mechanically. That is the entire secret. The name backpropagation means: the blame propagates backward.

Why one walk answers everything

Notice the ledger computed u’s blame once, then spent it twice — on a and on b. In a network with a million knobs, every shared intermediate value is computed once and spent by all its consumers; the whole walk costs about as much as the forward story did. That is why training is affordable at all: the price of every gradient together is one backward pass, not one pass per knob. Chapter 2’s probing, at two loss measurements per knob, would cost a million times more.

What to take forward

You have now seen the entire mechanism of learning, with no remaining black box: forward story, per-link blame rules, one backward walk, subtract. What the loop has not yet learned is speed — two thousand rounds for four observations, as chapter 6 counted, will not survive contact with real data. Making the step itself smarter — momentum, and the optimizer the GPT actually trains with — is next: the art of the step.