Chapter 2 ended on an honest debt: the computed slope came from doing algebra on our particular loss by hand, and nobody hand-differentiates a rule with a thousand layers. This chapter pays the debt with machinery — the tape, Mica’s autograd. It is the single feature this whole course stands on, and its contract fits in one sentence: record the forward story once, then walk it backward and collect every slope in one pass.

The working file is examples/TheTape — run it with make -C examples/TheTape run.

The notebook

Imagine doing the loss computation while a meticulous clerk writes every step in a notebook: multiplied the guess by 1, subtracted 2, squared it, added it to the running total… Once the total is written, the clerk can read the notebook backward, and at each line answer: if this line’s result had been slightly different, how much would the final total have moved? By the time the clerk reaches the first line, that question has been answered for the guess itself — and that answer is exactly the gradient chapter 2 computed by algebra.

   forward: the recording                backward: the walk
   ───────────────────────▶              ◀───────────────────────
   m1 := a·1 − 2      ┐                  ┌ how much was a to blame?
   m2 := a·2 − 4      │  the tape        │ …through each recorded
   m3 := a·3 − 6      │  remembers       │  step, blame flows from
   loss := m1² + m2² + m3² ┘             └ the loss back to a

Three spellings carry the whole surface. A knob is declared tracked — a value whose operations are worth recording. The recording happens inside a tape … end window. And after Backward(loss) has walked the record, Gradient(a) answers the slope for any tracked leaf. (This course teaches what the machinery is for; the feature’s own rules — the recorded vocabulary, the window’s placement law, every wall the compiler holds — live on tracked values and the tape.)

tape
    m1 := a * 1.0 - 2.0;
    m2 := a * 2.0 - 4.0;
    m3 := a * 3.0 - 6.0;
    loss := m1 * m1 + m2 * m2 + m3 * m3;
end;

Backward(loss);
a := a - Rate * Gradient(a);

Look hard at the window: it is chapter 1’s loss, verbatim. Nothing in it mentions derivatives. The recording is the only addition — and that is the design’s point. You write the forward story; the machine owns the backward one.

The proof is the echo

Run it, and compare against chapter 2’s descent table:

the descent, slopes by tape
  round 1: a=1.750000, slope -56.000000, loss 56.000000
  round 2: a=1.968750, slope -7.000000, loss 0.875000
  round 3: a=1.996094, slope -0.875000, loss 0.013672
  round 4: a=1.999512, slope -0.109375, loss 0.000214
  round 5: a=1.999939, slope -0.013672, loss 0.000003
  round 6: a=1.999992, slope -0.001709, loss 0.000000

The guess column — 1.75, 1.96875, 1.996094 … — is chapter 2’s column digit for digit. The tape is not a new method and not an approximation: it is the same derivative, computed by reading a record instead of by doing algebra. (One reading note: each line’s loss is the value the window recorded before that round’s step — which is why round 1 pairs the slope −56 with the loss 56 that produced it.)

Two properties of the record are worth naming now, because later chapters lean on them. Fan-out sums: if a knob appears three times in the story, its blame arrives from all three paths, added — that is why a weight used by every prediction in a batch gets one honest total gradient. Unread means zero: a tracked value the window never touched answers a gradient of exactly zero, because the record cannot blame what it never saw.

What to take forward

The tape removes the last hand-made piece from the learning loop. Loss: you write it. Slope: recorded and walked. Update: one subtraction. From here on, making the machine smarter never means new calculus — it means richer forward stories: more knobs, better shapes for them, and rules worth recording. The next chapter takes the first of those steps: the guess stops being one number — many knobs at once.