Between the tape and a full trainer sits a toolkit every training loop reaches for: activations, dropout, losses, update steps, a learning-rate schedule, and the seeded starts. Each is one call, each records where the tape needs it, and each prints the same digits every time it runs: the seeded starts, the masks, MseLoss and the update steps on every machine, and Gelu, Silu, CrossEntropy and CosineSchedule to the last digit of the C library whose tanh, exp, log and cos they call.

The example is examples/LearningToolkit in the tutorial repository.

Getting the file

make -C examples/LearningToolkit run
LearningToolkit — the pieces of a training loop

relu: s = 3.500000 g = 1.000000 0.000000 0.000000 1.000000
gelu at zero: g = 0.500000 0.500000
silu at zero: g = 0.500000 0.500000
mse: l = 2.500000 g = 1.000000 2.000000

mask (seed 42, rate one half): 2.000000 0.000000 0.000000 0.000000
dropout: s = 6.000000 g = 2.000000 0.000000 0.000000 0.000000

sgd: w = 0.950000 1.900000 v = 0.500000 1.000000
clip: norm was 5.000000, g = 1.500000 2.000000
cosine: 0.000000 0.500000 1.000000 0.000000

xavier: -0.269861 -1.183622 0.981659 0.203137
he:     -0.164746 -0.867995 -0.111014 -0.595560

The modern activations

s := Sum(Relu(x));      { negative cells drop; the derivative is the saved sign }
s := Sum(Gelu(z));      { the tanh spelling the GPT lineage trains with }
s := Sum(Silu(z2));     { the sigmoid-weighted input of the llama family }

Relu, Gelu, and Silu join Tanh in the recorded vocabulary, with matrix twins ReluMap, GeluMap, and SiluMap beside them exactly as TanhMap stands beside Tanh. Each records with its derivative rule: relu’s is the saved sign (an input at exactly zero answers zero on both sides), gelu’s and silu’s are their exact slopes — both pass one half at zero, and the printed gradients say so. RmsNorm conditions rows the way modern language models do: LayerNorm without the mean subtraction and the bias, with a learned gain and its own recorded rule.

Dropout is a pair of visible values

r := Seeded(42);
mk := DropoutMask(0.5, address r);

tape
    s := Sum(Dropout(m, mk));
end;

The mask is a value: drawn once from the seeded generator, each kept cell carrying 1/(1-rate) (the inverted scaling that keeps the expectation put), each dropped cell zero. You can print it — the example does. The recorded product saves the mask beside the activation, and the derivative multiplies by exactly it: the printed gradient IS the printed mask. Inference is the same program without those two lines — no evaluation mode, no hidden flag.

Same seed, same mask, same digits — on every machine at every tier, because the generator is the random unit’s bit-stable splitmix64.

The losses

CrossEntropy you know from the GPT. MseLoss stands beside it for regression: the mean over all cells of the squared difference, recorded, with the exact derivative 2·(a−b)/n flowing to both operands. Its views take a vector, a matrix, or a higher-rank tensor read flat.

The update family

SgdStep(address w, g, address v, 0.1, 0.5);         { v ← μ·v + g;  w ← w − α·v }
AdamWStep(address w, g, address m, address v,
          step, α, β₁, β₂, ε, decay);               { AdamStep + decoupled decay }
nrm := ClipGradNorm(address g, ceiling);            { cap the length, keep the direction }
α := CosineSchedule(step, warmup, total, peak);     { the warmup ramp + half cosine }

All four are plain code over plain values — they run outside the window, on gradients you extracted with Gradient. AdamWStep at zero decay is bit-equal to AdamStep; ClipGradNorm answers the norm it measured before any clipping, the number the field logs; and the schedule is a function you call once per step — the learning rate is a variable in your loop, not an object’s hidden state.

The seeded starts

r := Seeded(7);
w1 := XavierInit(address r);    { uniform in ±sqrt(6/(R+C)) — Glorot's bound }
w2 := HeInit(address r);        { uniform in ±sqrt(6/R) — Kaiming's, fan-in = rows }

Both draw from the caller’s own generator in row-major cell order, so a training run’s starting point replays exactly — the reproducibility promise extended to the very first step. The fan-in is the row count because this library multiplies activations from the left (h * w): a weight’s rows face the incoming features.

The elementwise pair, by name

* between matrices is the linear-algebra product, so elementwise multiplication and division have names instead: Hadamard/HadamardMap for the product and Divide/DivideMap for the quotient — and, like every matrix verb, the map twins admit a static higher-rank operand through the trailing-axis flattening. Recorded, the product’s derivative multiplies by the other side’s saved value, and the quotient carries the textbook rule. The llama family’s SwiGLU gate is exactly this pair of lines:

g := SiluMap(x * wg);              { the gate branch }
h := HadamardMap(g, x * wu);       { gated elementwise - the SwiGLU sentence }

And the quotient is what keeps a batch normalization’s inference scale one visible sentence — s := Divide(gain, Sqrt(rv + eps)) — the line the Image networks chapter builds on.

Where this leads

The batched trainer uses every piece on this page in one program, and checks the whole run against an independent oracle.