The gradient says which way is downhill. How far to step is a separate question, and chapter 6 showed why it matters: two thousand rounds for four observations. This chapter is about the three answers that matter in practice — the plain step you have used since chapter 2, momentum, and Adam, the optimizer the GPT at the end of this course actually trains with.

The working file is examples/ArtOfTheStep — run it with make -C examples/ArtOfTheStep run.

The unfair valley

The example digs one loss surface: (p−2)² + 256·(q−1)² — a canyon whose wall is two hundred and fifty-six times steeper than its floor. Every real loss has this shape somewhere; the exaggeration just makes it printable.

        steep wall (q): one grain too bold and the step overshoots,
   ═╗   lands higher on the far side, overshoots harder — explosion
    ║
    ╚═══════════════════════▶  shallow floor (p): safe steps are
         tiny steps only        grains, progress is a crawl

The cruelty is the coupling: one rate serves both directions. The wall demands a tiny step or the descent explodes across it; the floor then inherits that tiny step and crawls. The wall prices the floor.

Three descents, printed

The plain step, capped by the wall at a rate of about 1/512:

the plain step
  round 1: loss 260.000000
  round 50: loss 2.725719
  round 200: loss 0.842453
  round 800: loss 0.007688

Eight hundred rounds and still not done — all of it spent crawling the floor, since the wall direction was finished almost immediately.

Momentum, at the same capped rate, gives each knob a velocity that accumulates blame. Where the slope keeps agreeing — a long floor — the stride grows toward a multiple of the plain one; where the slope alternates — across a wall — the contributions cancel. A rolling ball, not a cautious walker:

vp := Beta1 * vp + Gradient(p);
p := p - WallRate * vp;
momentum
  round 50: loss 1.872343
  round 200: loss 0.000000

Adam repeals the wall’s pricing altogether. Beside the momentum average m, each knob keeps a running average s of its blame’s square, and steps by m / √s: a knob whose blame is habitually large takes modest strides, one whose blame is habitually small takes confident ones. Each direction is normalized to its own scale — so the example runs Adam at a rate thirty-two times bolder than the wall permits the plain step, safely:

mp := Beta1 * mp + (1.0 - Beta1) * gp;
sp := Beta2 * sp + (1.0 - Beta2) * gp * gp;
p := p - AdamRate * mHat / (Sqrt(sHat) + Epsilon);
adam
  round 50: loss 0.213920
  round 200: loss 0.000000

Ahead of both at every milestone. The β₁ = 0.9 and β₂ = 0.999 here are the values almost every published model uses unchanged, and the ones gpt.mica trains with; the mHat/sHat corrections repair the running averages’ early rounds, when they still remember their zero start. The example spells all of it out on two scalar knobs so you can read every line — and the math unit’s AdamStep verb is this exact arithmetic over whole tensors at once, which is how the GPT will consume it.

What to take forward

One honest caution, then the summary. Adam is not magically better — on this convex canyon everything converges eventually, and there are problems where plain momentum generalizes better; what Adam buys is robustness to scale differences between knobs, which a 786944-parameter model has in abundance. The summary: direction comes from the backward walk, pace from the optimizer, and both are now in hand. What the machine still cannot do is read — the GPT’s raw material is text, and text is not numbers. Turning one into the other is next: tokens and embeddings.