The guessing machine

This is the first chapter of the AI course: a walk from “what is learning” to training your own GPT, every step of it in Mica you compile and run yourself. The course keeps one promise the whole way, and it is worth stating before anything else: every number you print in this course is the same on every machine, every run. Not close — identical, digit for digit, on either supported architecture, today and next year. Learning systems are famously hard to reproduce; this one is reproducible by construction, and you will see why as the course goes on. When a chapter shows output, your output is that output.

The working file is examples/GuessingMachine — run it with make -C examples/GuessingMachine run.

Learning is a search

Think of adjusting a shower tap in an unfamiliar hotel. You do not derive the correct angle from the plumbing diagrams: you guess, feel whether the water got closer to right or further away, and nudge. A few rounds of that and the temperature is right — without you ever knowing anything about the pipes.

That is the whole of machine learning, honestly stated. A machine that learns holds a guess, measures how wrong the guess is as a single number, and nudges the guess in whichever direction makes that number smaller:

        ┌──────────────┐
   ┌───▶│    guess     │
   │    └──────┬───────┘
   │           │  use it
   │    ┌──────▼───────┐
   │    │   measure    │   the loss: how wrong, as one number
   │    └──────┬───────┘
   │           │
   │    ┌──────▼───────┐
   └────│    nudge     │   whichever direction lowers the loss
        └──────────────┘

Everything in this course — the gradient, the tape, layers, attention, the GPT — is a refinement of one box in this loop. The loop itself never changes.

Three observations and one number

Suppose the world hands us three observations: when x was 1, y was 2; when x was 2, y was 4; when x was 3, y was 6. We suspect a rule of the shape y = a·x, and want the machine to find a.

The loss turns “how wrong is guess a” into one number: for each observation, take the miss a·x − y, square it (so misses in either direction count alike, and big misses count more), and add them up. A few values by hand:

The loss is a report card, not an explanation. It does not say what to do — it only scores. That is what makes it powerful: anything you can score, you can learn.

The loop, in Mica

The machine starts ignorant (a := 0.0) with a step of one whole unit, and each round it tries the step in both directions, keeps whatever lowers the loss, and halves the step when neither direction helps — too coarse a stride means you are standing at the bottom already, so look again with finer eyes:

for round := 1 to 8 do
begin
    if Loss(a + step) < Loss(a) then
        a := a + step
    else if Loss(a - step) < Loss(a) then
        a := a - step
    else
        step := step / 2.0;

    WriteLn("  round %lld: guess a=%lf, step %lf, loss %lf", round, a, step, Loss(a));
end;

Run it, and watch the machine learn:

the search
  round 1: guess a=1.000000, step 1.000000, loss 14.000000
  round 2: guess a=2.000000, step 1.000000, loss 0.000000
  round 3: guess a=2.000000, step 0.500000, loss 0.000000
  round 4: guess a=2.000000, step 0.250000, loss 0.000000
  ...

the learned rule: y = 2.000000 · x
prediction for x = 10: y = 20.000000

Two rounds to the answer, and then the halving steps confirm it: no nudge in any direction, at any stride, improves on a = 2 — the observations really were made by y = 2·x. And the last line is the point of the whole exercise: the machine answers for an x it never saw. A learned rule that only repeats its observations is a lookup table; one that extends beyond them is a model.

What to take forward

Hold on to three things. The loss is the interface between a problem and learning — nothing else about the problem is visible to the loop. The guess is just storage — here one number, later a matrix, in the final chapter 786944 of them, and the loop does not care. And the nudge is the expensive part: this chapter probed both sides of the guess to find downhill, which costs two full loss measurements per knob. With thousands of knobs, that is thousands of probes per round — and there is a better way. The next chapter measures which way is downhill without stepping anywhere: the gradient.