The guessing machine had to measure the loss on both sides of its guess before it knew which way to move — two probes per knob, per round. This chapter replaces the probing with an answer read off the guess itself: the gradient, the slope of the loss under your feet. It is the single most important number in machine learning, and by the end of this chapter you will have computed it two independent ways and watched them agree to the last digit.

The working file is examples/GradientDescent — run it with make -C examples/GradientDescent run.

The fog-walker

Picture descending a hill in fog so thick you cannot see your own boots. You cannot survey the valley — but you can feel which way the ground tilts where you stand, and that is enough: face against the tilt, step, feel again. The tilt is the gradient; the walk is gradient descent.

 loss
  │  ●  a=0                        the loss over every possible guess
  │   ╲                            is a valley; the gradient at a point
  │    ╲         slope here        is the valley wall's slope there —
  │     ╲___     is negative:      negative means the floor lies to
  │         ╲──__ downhill is      the right, positive means it lies
  │              ╲──__ rightward   to the left, zero means you stand
  │                   ╲●____       on the floor itself
  └──────────────────────▔▔●▔───────▶ guess a
                        a=2 (loss 0)

The slope, measured

The fog-walker’s honest method first: nudge the guess a little both ways, measure the loss at each, and divide the difference by the width of the nudge:

function MeasuredSlope(guess : float64) : float64;
begin
    MeasuredSlope := (Loss(guess + Nudge) - Loss(guess - Nudge)) / (2.0 * Nudge);
end;

This needs nothing but the ability to measure the loss, which is why it works on any rule — and why it stays in the toolbox forever as the referee: whatever cleverer method you use, this one checks it.

The slope, computed

For our squared-miss loss, algebra answers directly: each observation contributes twice its miss times its x. No probing, no nudge width to choose:

function ComputedSlope(guess : float64) : float64;
var
    k : int64;
begin
    ComputedSlope := 0.0;

    for k := 0 to 2 do
        ComputedSlope := ComputedSlope + 2.0 * (guess * xs[k] - ys[k]) * xs[k];
end;

Now the two meet, at three different guesses:

two answers to one question
  at a=0.000000: measured -56.000000, computed -56.000000
  at a=1.500000: measured -14.000000, computed -14.000000
  at a=3.000000: measured 28.000000, computed 28.000000

Identical — and for this loss, exactly identical, not approximately: a squared-miss loss is a parabola in the guess, and the two-sided measurement of a parabola’s slope is exact regardless of the nudge width. The signs tell the story: negative at a = 0 and a = 1.5 (the floor lies to the right), positive at a = 3 (you have walked past it).

The descent

With direction and steepness in one number, the whole learning loop collapses into one line: subtract the gradient, scaled by a small learning rate that keeps each stride from leaping over the valley floor:

for round := 1 to 6 do
begin
    a := a - Rate * ComputedSlope(a);
    WriteLn("  round %lld: a=%lf, loss %lf", round, a, Loss(a));
end;
the descent
  round 1: a=1.750000, loss 0.875000
  round 2: a=1.968750, loss 0.013672
  round 3: a=1.996094, loss 0.000214
  round 4: a=1.999512, loss 0.000003
  round 5: a=1.999939, loss 0.000000
  round 6: a=1.999992, loss 0.000000

Watch the loss column: each round cuts the remaining distance to the floor by the same factor, and six rounds in, the printed loss is zero to six places. No sideways probes, no step-halving — the gradient carried both the direction and the confidence.

One honest note: the guess reads 1.999992, not 2. Gradient descent approaches; it does not land. How close is close enough, and how the learning rate trades speed against overshoot, are questions with real depth — the course returns to them in the optimization chapter.

What to take forward

The gradient turns learning from search into arithmetic — that is why it scales from this chapter’s one knob to the GPT’s 786944. But the computed slope came from doing algebra on our particular loss by hand, and nobody hand-differentiates a thousand-layer rule. The next chapter is the machinery that does it for you: a notebook that records every step of the forward computation and then runs it backward — the tape.