Every rule so far was a weighted sum, and chapter 4 made weighted sums powerful: any number of knobs, learned in one loop. This chapter shows you the wall they cannot climb no matter how their knobs are set — and then climbs it with one new idea, which is the idea the word neural has stood for all along.

The working file is examples/BentWire — run it with make -C examples/BentWire run.

The wall

The exclusive-or is four observations: answer 1 when exactly one input is on.

    x₂
    1   ●1        ○0          ● want 1   ○ want 0
    │
    0   ○0        ●1          no straight line puts both ● on one
    └────────────────── x₁    side and both ○ on the other — try it
        0         1

A weighted sum w₁·x₁ + w₂·x₂ + b draws one straight boundary and grades everything by which side it lands on. But XOR’s ones sit on one diagonal and its zeros on the other: any line that separates (0,1) from (0,0) fails at (1,1). This is not a training difficulty — it is an expressiveness fact, and the descent reports it with a number:

the line descends
  round 1: loss 2.000000
  round 10: loss 1.003077
  round 200: loss 1.000000
  the wall: the loss will never go below 1 — no line separates XOR

The loss falls, then flattens at exactly one and stays. The best a line can do on XOR is answer one half everywhere — wrong by a half on all four corners, four times a quarter, loss 1. The machine converged perfectly to the best rule of its shape; the shape was the problem.

The bend

Tanh takes a weighted sum and bends it smoothly toward limits of −1 and +1 — a wire with a soft kink instead of a straight rod. One hidden layer is two bent wires side by side; a final weighted sum combines them:

z := w1 * x + b1;        { two weighted sums at once — a matrix row each }
h := Tanh(z);            { the bend, applied to both }
o := Dot(w2, h) + c;     { a weighted sum of the bent pair }

Intuition for why this wins: one bent wire can learn “is at least one input on?”, the other “are both inputs on?” — and “exactly one” is the first minus the second, which the output sum can spell. Straight rods only add; bent wires give addition something worth adding.

One honest subtlety from the example’s own start: the initial knobs are fixed literals chosen with no symmetry. If both hidden wires start identical, their gradients stay identical forever, and two copies of one wire are just one wire — the descent can never split them. Real frameworks break this tie with random starts; this course breaks it in the literals, where you can see it, because the reproducibility promise does not allow “random”.

The same descent, over the bend

Nothing else changes — the window records the bent story, Backward walks it, the update subtracts:

the bend descends
  round 1: loss 1.551169
  round 10: loss 0.906573
  round 200: loss 0.125026
  round 2000: loss 0.000000

the four answers
  0 xor 0 = -0.000000   (want 0)
  0 xor 1 = 1.000000   (want 1)
  1 xor 0 = 1.000000   (want 1)
  1 xor 1 = -0.000000   (want 0)

Same loop, same tape, same rate — and the wall is gone: all four corners answered to six places. (The -0.000000 is honesty, not error: the residue is a vanishingly small negative number, and its sign survives the rounding to six zeros.)

What to take forward

Weighted sums plus bends is the complete recipe — a GPT contains nothing else at its core, just very many of each. And the tape earned its keep silently: Tanh joined the recorded story and its slope flowed backward through the bend without you writing any calculus. The next question is what those hidden bent wires actually learn — and it has a printable answer: layers.