A layer of a model is a handful of parameters that belong together. In Mica that is what it looks like: a record whose fields are tracked, and a model is an array of those records. The forward pass loops over the blocks and records through the places; the parameter walk pytorch hides behind model.parameters() is a loop you write, read, and step through.

The example is examples/TrackedRecords in the tutorial repository.

Getting the file

make -C mica-container/examples/TrackedRecords run
TrackedRecords — records hold the parameters, the walk is your loop

loss = 6.000000
w0 after the step = 0.900000 -0.111111 -0.100000 0.888889
w1 after the step = 1.916667 -0.100000 -0.083333 1.900000
loss after the step = 4.262593

sgd over the elements = 0.875000 1.750000 2.875000 3.750000

Identical on every machine, at every optimization tier, on both architectures.

A block is a record, a model is an array

type
    Block = record
        w : tracked matrix[2, 2] of float64;
    end;

var
    blocks : array[0..1] of Block;

A tracked field is a place: blocks[k].w declares, records, answers Gradient, and re-adopts exactly as a tracked variable does. The member’s inner type is a float scalar or a static-shaped float tensor, and a variable whose type carries tracked members lives where every tracked value lives: in the program’s own var section, or in a routine, which records when a window calls it.

The depth loop records through the places

A block’s forward pass is a routine of its own, which takes the block as a constant record:

function Forward(const b : Block, input : tracked vector[2] of float64) : tracked vector[2] of float64;
begin
    Forward := b.w * input;
end;
tape
    for k := 0 to 1 do
        h := Forward(blocks[k], h);
    l := Sum(h);
end;

Backward(l);

The loop is ordinary code inside the window. Each pass through the body hands the routine a different place, and the tape records what actually executed — depth is unrolled by having run, not by a graph compiler. The call enrolls the block’s tracked members on the tape before the routine reads them, so what Forward records joins the history blocks[k].w carries, and Gradient(blocks[k].w) after the walk answers for the model’s own place.

Enrolling reaches the place a second time, so a record holding tracked members is handed to a routine — or copied — from the place grammar the tape reads everywhere: a variable or a parameter, field steps on it, and at most one index run, each index a literal or a plain name. The tape holds onto the storage of the variable or the parameter itself, so a record behind a pointer is out of its reach, whether a pointer variable, a pointer field or a pointer element leads to it. An index that computes and a record behind a pointer are refused (error 5631):

a ‘Block’ holds tracked members, so it is copied or handed to a routine from a place the tape can hold onto: a variable or a parameter, or field steps on one with at most one index run between them, each index a literal or a plain name — its members are enrolled on the tape before it is read, so the place is reached twice, and an index that could compute would run twice, while a deeper shape or a pointer’s target would lose the members’ history — name the index in an ordinary variable first, keep the record in a variable or a parameter rather than behind a pointer, and reach the value through the admitted shape

A tracked member behind a pointer is refused the same way wherever the tape reads it (error 5524). Keep a model the tape trains in a record variable. A program body that keeps one in a heap cell copies the cell’s record into a record variable before the window, and stores the stepped record back after the step.

The walk is your loop

Each block keeps a plain twin record beside it, carrying the weights the optimizer updates and its two moment runs — visible state, no optimizer object:

type
    Plain = record
        w : matrix[2, 2] of float64;
        m : matrix[2, 2] of float64;
        v : matrix[2, 2] of float64;
    end;
for k := 0 to 1 do
begin
    gw := Gradient(blocks[k].w);
    AdamStep(address twins[k].w, gw, address twins[k].m, address twins[k].v,
             1, 0.125, 0.5, 0.5, 0.5);
    blocks[k].w := twins[k].w;
end;

Three sentences per block: read the gradient at the place, step the plain twin, re-adopt. address twins[k].w lends the twin’s field to the update step through the conformant borrow — field steps with one index step, the same place grammar the tape itself reads — and the update writes the caller’s own storage. The re-adoption blocks[k].w := twins[k].w moves the stepped value back into the tracked place; the next window enrolls it fresh.

A second window over the updated places closes the cycle: the loss falls from 6 to 4.262593, and both numbers are the same on every machine.

Why the twin, and not the place itself

analyzer error 5119: data type 'tracked matrix[2, 2] of float64' cannot be
used in memory operation 'address of' (requires addressable)

A tracked place is a pair — the value and the id of the node that produced it — and a borrowed view writing the value behind the pair’s back is exactly the hidden mutation the pair refuses everywhere. The twin record keeps the optimizer’s writes in plain storage, and the adoption is the one visible moment the model changes.

The borrow serves bare elements too

for k := 0 to 1 do
    SgdStep(address ws[k], gsd, address vs[k], 0.25, 0.0);

An element of a plain array — ws[k], no record anywhere — lends through the same borrow, so parallel-array models update in place with the same three sentences.

Where this leads

The batched trainer uses exactly this shape at full size: eight parameters per block, two blocks, an embedding above them, and one visible walk stepping everything under a cosine schedule.