Sequence models read a batch as rows of tokens; image models read it as a stack of pictures. This chapter walks the layer family that difference calls for — convolution, pooling, batch normalization — and trains a small classifier end to end, every mechanism a line you can point at.

The example is examples/ConvNetTrainer in the tutorial repository.

Getting the file

make -C examples/ConvNetTrainer run
step 1 loss = 1.084716
step 2 loss = 0.905115
step 3 loss = 0.800130
wc = 0.081205 0.147125 0.271853 0.091787
gnbs = 0.996370 0.473855 0.197574 -0.305713
wl = 0.204526 -0.021231 -0.014861 -0.104598

Three training steps over two tiny images — a vertical bar and a horizontal bar — then a sample of the trained weights from every parameter family. Every digit has been checked against an independent oracle that replays the entire computation, and every digit repeats at every optimization tier. The cross-entropy and the cosine schedule call the C library’s exp, log and cos, so the last digits are the ones the machine’s C library gives.

The image tensor

An image batch is a rank-4 tensor in channels-first order — [batch, channels, height, width], the layout pytorch calls NCHW:

xdata : tensor[2, 1, 4, 4] of float64;    { two one-channel 4x4 images }

The rank is part of the type, so every shape below — what the convolution answers, what the pool tiles, what the reshape bridges — is proven where it is written.

Convolution

Conv2d(x, w, stride, padding) slides a stack of filters over every image. The weight stack’s shape says everything about it: [out, in, kh, kw] — how many filters, over how many input channels, at what window:

h1 := Conv2d(xc, wct, 1, 1);       { two 3x3 filters, stride 1, padding 1 }

Stride and padding are compile-time constants because the result’s shape depends on them — here [2, 2, 4, 4], the same height and width because one ring of padding exactly feeds a 3×3 window. There is no bias argument, and that is deliberate: a convolution feeding a batch normalization gets its shift there, which is how the field trains these stacks. On tracked values the convolution records, and its derivative is the transposed convolution against the saved operands — the textbook rule, run by the same walk that serves every other recorded operation.

The pooling pair

MaxPool(x, k) and AvgPool(x, k) tile each image plane with k-by-k windows at stride k, the trailing remainder dropped — the floor semantics every framework defaults to:

h4 := MaxPool(h3, 2);              { [2, 2, 4, 4] down to [2, 2, 2, 2] }

The max pool’s backward routes each window’s gradient whole to the window’s maximum — recomputed from the saved input as the first maximum in ascending order, so a tied window has one documented winner and no index mask is stored anywhere. The average pool spreads each window’s gradient uniformly over its cells.

BatchNorm, with nothing hidden

BatchNorm(x, gain, bias) standardizes each channel over its batch-height-width samples and applies the learned gain and shift — the training form, recorded with the textbook three-term rule:

h2 := BatchNorm(h1, gnt, bst);     { per-channel statistics, gain and bias learned }

Pytorch’s version hides two more things: running statistics updated as a side effect, and an eval-mode switch that changes what the layer computes. Mica keeps both visible. The running statistics are plain values your program declares and updates with ordinary arithmetic:

rm := ChannelMean(x);                        { the biased per-channel statistics }
rv := ChannelVariance(x);
rmean := 0.9 * rmean + 0.1 * rm;             { the running update, in the open }
rvar  := 0.9 * rvar  + 0.1 * rv;

And inference is not a mode — it is a different line. You compute the affine form once from the running statistics and apply it per channel:

s := Divide(gain, Sqrt(rvar + eps));         { the vector sentences themselves }
t := bias - Hadamard(rmean, s);
y := ChannelScale(x, s, t);                  { the inference form, plain code }

At the batch’s own statistics, ChannelScale answers the training forward’s digits exactly — the equivalence is pinned in the compiler’s test suite to every printed digit.

The whole classifier

The model is the stack a pytorch user writes first, and every step of its training loop is visible:

tape
    h1 := Conv2d(xc, wct, 1, 1);
    h2 := BatchNorm(h1, gnt, bst);
    h3 := ReluMap(h2);
    h4 := MaxPool(h3, 2);
    hf := Reshape(h4);             { [2, 2, 2, 2] read as matrix[2, 8] - same cells }
    lg := hf * wlt;                { the linear head }
    loss := CrossEntropy(lg, ti);
end;

Backward(loss);

gwc := Gradient(wct);
SgdStep(address wc, gwc, address vc, alpha, 0.5);

Reshape is the same bridge the batched transformer used — the same row-major cells read in the assignment target’s shape. The optimizer view is rank-general: the rank-4 filter tensor feeds SgdStep directly, read as its contiguous elements, exactly as the head’s matrix does. The loss falls 1.084716 → 0.905115 → 0.800130 over the three steps, and each number is the oracle’s to the last digit.

Where this leaves you

Convolution, pooling, and batch normalization complete the layer families a pytorch user expects: the sequence stack from Batched training and the image stack here share the same tape, the same visible walk, and the same everything-is-a-value discipline. What pytorch does with buffers and modes, Mica does with lines you can read.