Layers

Chapter 5 proved a hidden layer can solve what no weighted sum can. This chapter asks the better question — what did the layer learn? — and answers it with a table you can print. It is the shortest chapter in the course, and the one whose idea carries furthest: everything from here to the GPT is this chapter’s move, repeated.

The working file is examples/Layers — run it with make -C examples/Layers run.

The hidden pair is a place

The bent-wire network computes, for every input, a pair of hidden values h = (h₁, h₂) before anything is answered. Read that pair as coordinates: the layer takes each input and places it somewhere in a new two-dimensional space of its own making. The final weighted sum then draws its one straight line — but in the layer’s space, not the input’s.

The example prints where the four XOR corners land, before and after training. Before, with the starting knobs, the placement is arbitrary:

before training
  0 xor 0: h = (-0.244919, 0.124353), answer -0.153548
  0 xor 1: h = (0.244919, -0.358357), answer 0.212049
  1 xor 0: h = (0.462117, 0.358357), answer 0.141469
  1 xor 1: h = (0.761594, -0.124353), answer 0.411885

After training — same network, two thousand rounds of the same descent:

after training
  0 xor 0: h = (-0.614142, -0.962144), answer -0.000000
  0 xor 1: h = (0.873396, -0.504305), answer 1.000000
  1 xor 0: h = (0.906102, -0.470178), answer 1.000000
  1 xor 1: h = (0.998410, 0.720481), answer -0.000000

the reader of that space: answer = 0.953502·h1 + -0.913793·h2 + -0.293615

Read the placement. The two want-1 corners — (0,1) and (1,0), which sat apart before — now land nearly on top of each other, at (0.87, -0.50) and (0.91, -0.47). The two want-0 corners were pushed away in different directions. In the input’s space, XOR’s ones sat on opposite diagonal corners; in the layer’s learned space, they are neighbours:

   input space: no line works        the layer's space: an easy line
   1  ●        ○                        h₂    ○ (1,1)
   │                                     │        ╲  the reader's line
   0  ○        ●                    ─────┼───────  ╲ ─────▶ h₁
      0        1                    ○ (0,0)   ●● both want-1
                                                  corners, together

That is the thesis of deep learning, whole: a layer does not memorize answers — it re-describes the input, moving the problem into a space where the answer is simple. The last line of the run shows the “simple” concretely: one weighted sum over the learned coordinates, 0.95·h₁ − 0.91·h₂ − 0.29, finishes the job a weighted sum over the raw inputs provably could not start.

Stacking is repeating the move

Nothing limits the move to one repetition. A second layer reads the first layer’s space and re-describes it; depth is re-description compounded. The word for a network of such layers — multilayer perceptron, MLP — will reappear inside the GPT by name, as will this exact shape: Tanh of a matrix product, read by another product. When the course reaches it, you will have printed its inner life already.

What to take forward

Width buys vocabulary — more hidden wires, more coordinates to place things by. Depth buys re-description — spaces built on spaces. Both are spent by the same currency, knobs, and trained by the same loop — whose backward half you have trusted since chapter 3 without once seeing inside it. The next chapter opens the lid, exactly once, and does the walk by hand: blame flows backward.