The measurement protocol behind TR-2

The public edition of the protocol by which TR-2 measures Mica beside PyTorch. The protocol was written and merged on 2026-10-01, before the first measurement it governs, and every change after that is an amendment at its end, dated, with its reason. This edition gives its sections 1 and 3 to 10 and all eight amendments word for word, under their own numbers. It leaves out section 2, a review of the numbers given out before the protocol existed, which the protocol’s results replace, with the paragraph of section 10 on those numbers; section 11, the schedule of the release that carried the protocol out; and section 12, the questions its merge answered.

1. What it governs

Every statement that compares Mica with PyTorch, wherever it appears: step time, memory, footprint and loss, on the website, in TR-2, in the container’s results, in release articles, in applications to NVIDIA, Anthropic and Google, in talks and in videos. A number that no run of this protocol produced is not published. A number already given out is replaced by the protocol’s as soon as it exists (§10).

3. The rules

1. Fixed before measuring. The models, configurations, number of runs, statistics and success criteria are written here, and no result decides which configuration is the comparison.
2. PyTorch at its best. The comparison is against the fastest of PyTorch’s four modes as measured on that card, model and precision. nanoGPT’s model code is never changed; the changes to its train.py are listed in §4 and published as a patch, and the timing runs of its default mode use none of them.
3. One instrument for both sides. The outside clock of §5 times both. Each program’s own reading is recorded beside it and never replaces it.
4. Every number carries its conditions: the number of runs, the median and the range, the card, every software version, the date, the Mica commit, and the link to its raw logs.
5. A deficit is published as measured, in the same table and the same type as a lead.
6. Speed is published only for a configuration whose correctness check passed (§7).
7. The raw data is published: every log, kernel table, loss curve and script, so anyone can recompute every number from the container.
8. A measurement is not a projection. A statement is a protocol result or it is labelled a hypothesis, and a hypothesis is never published as a result.
9. Published numbers come from the released compiler. A run on a development build is preparation and says so wherever it appears, Michael’s page included.

4. What is measured

The changes to nanoGPT’s train.py, all published as one patch: the mode string handed to torch.compile, for the two modes beyond the default; a seed variable in place of the fixed 1337, for the five seeds of §7; and, for §7’s method B only, a switch for TF32 and a batch order read from a file. With none of them applied, train.py runs as nanoGPT ships it.

5. How time is measured

• The clock. One reader starts each program and stamps every line of its output with the monotonic clock as the line arrives. nanoGPT runs under python -u and the Mica program with +mica stdout=line -mica, so every line leaves the moment it is written; neither changes what is computed. Both sides print a line every 10 steps right after a loss read that waits for the card, so the time from the line of step 100 to the line of step 400 is 300 whole steps, with nothing in flight at either end. A run’s step time is that time divided by 300.
• Warm-up and steady state. Steps 0 to 99 are warm-up and never measured: Inductor’s and NVRTC’s compilations, autotuning, the capture of CUDA graphs, the memory pools growing and the clocks rising. Each run trains 400 steps, with evaluation off; nanoGPT keeps its single evaluation batch at step 0, inside the warm-up.
• Runs. Five runs per configuration, each in a fresh process, in five rounds: each round runs every configuration of the card once, in an order that rotates from round to round, so heat and drift touch every configuration alike. One discarded run of each configuration comes first and fills NVRTC’s and Inductor’s caches.
• What is reported. Per configuration, the median of its five step times and their range, fastest to slowest. The comparison, per card, model and precision, sets Mica’s median against the median of PyTorch’s fastest mode, with both ranges beside them.
• Conditions. Before every run the card must hold no other program, as run.sh checks today. During every run the card’s clocks, temperature and power are sampled every second, and its peak memory is read by nvidia-smi for both sides alike. The clocks are not locked: the cards run as their owners’ do, and the samples show what they did.
• Cross-checks. The Mica program’s own reading must agree with the outside clock within 1%, or the run is void and its cause is found before anything is published. nanoGPT’s own time field is recorded and not compared, for the reason in §2. The difference of two run lengths, our former cross-check, is retired.
• Profiled runs are separate. NVIDIA’s Nsight Systems slows launches, so a profiled run never yields a step time; it yields the kernel tables of §6.

6. What every series records besides time

• PyTorch’s kernels. One profiled run per mode, precision and model: the kernel table by name, total time and launches. Its attention is named from the kernels it ran: fmha_cutlassF and fmha_cutlassB are the memory-efficient attention, pytorch_flash:: is FlashAttention, a cudnn attention kernel is cuDNN’s, and softmax kernels between batched products are the unfused math path. Beside them, the backends PyTorch allows, read from torch.backends.cuda in the same environment.
• Graph breaks. One run per mode under TORCH_LOGS=graph_breaks,recompiles, its log published and its counts in the results.
• Mica’s kernels. The same table for the Mica program, and which of its modules loaded precompiled and which NVRTC compiled at the start; amendment 2 below states how a series records that until the runtime reports it.
• Families. Both tables are grouped into the same families by one rules file, benchmarks/NanoGpt/families.tsv in the container: a kernel-name pattern and its family, the first match wins, and an unmatched kernel is “other”. The file is fixed before the first series; the full tables are published, so anyone can group them differently. The families explain a step time and never stand in for one, and they are labelled as the profiled run’s kernel time.
• Versions. Everything run.sh records today, and Triton’s, cuDNN’s and Python’s versions beside them.

7. Correctness: the same model, trained alike

A speed means nothing unless both sides train the same model to the same loss. Two methods, both required for M1 and M2:

• A. Five seeds a side. Each side trains the model’s whole schedule five times, with the seeds 1337 to 1341, in both precisions, on the RTX 4090; the RTX 3090 Ti runs one seed a side as a check. M1 trains nanoGPT’s 5,000 steps, M2 its 2,000, both evaluated every 250 steps on 200 batches of each split. The two stacks draw different dropout masks and batches, since their random generators differ, so the curves are compared as families. A passes when, at every evaluation from step 1,000 on, the two sides’ median validation losses differ by no more than the larger of the two sides’ ranges at that step. All ten curves are published on one plot, and the final losses as median and range.
• B. The same start and the same data. Both sides start from one set of initial weights, made once by nanoGPT’s own initialization and saved, and both read one recorded batch order; nanoGPT reads them through its own resume path, and the Mica program through files. Dropout is 0 and the products are plain float32, without TF32, so only the order of summation differs between the stacks. B passes when the losses at step 0 agree within 1e-5 relative, and the loss of each of the first 200 steps within 1% relative. Both curves are published on one plot.

A configuration that fails A or B publishes no speed until the cause is found and fixed. Amendment 7 below names the L4 beside the RTX 3090 Ti and runs method B on every card.

8. Footprint, and NVRTC out of the run-time path

• Measured images. Each side gets the smallest container image that runs its training on a machine with NVIDIA’s driver, built from Dockerfiles published in benchmarks/NanoGpt/images/: Mica’s holds the binary and the NVIDIA libraries it opens, PyTorch’s holds Python and what pip install torch numpy brings. Each image’s size and file list are published. The driver’s libcuda comes from the host in both, and the results say so. A ratio of footprints is computed from the two measured sizes and from nothing else.
• NVRTC out of the run-time path (Q4). The compiler’s build already precompiles the floor’s kernels and the provider’s attention for five card generations, and a card loads them by their source’s hash (gpu_nvidia.c). A program’s own kernels still go through NVRTC when it starts. The link would precompile them too, with the same precompiler, and embed their images in the program, so a card of a listed generation loads every kernel ready and never opens libnvrtc. A card of another generation compiles through NVRTC, and +mica info -mica says so. What a run needs on a listed card is then the binary, cuBLAS and the driver, which §8’s image shows. The machine that builds a GPU program needs NVRTC, as the compiler’s own build already does. Amendment 3 below adds the release build itself to this step.

9. Wording, checked before anything is published

• How kernels are made: “Mica writes every kernel in its intermediate language; Dragon’s NVIDIA provider renders it as CUDA C, and NVIDIA’s NVRTC compiles it to the card’s machine code; matrix products run in cuBLAS.” PTX is named only for 7.5 and earlier.
• PyTorch is named with its version, mode, precision and attention kernels, every time.
• “Faster”, “level” and “slower” are used only as §10 defines them.
• Every M1 number travels with its model’s size: “a 10.7-million-parameter model with a 256-character context, where launches and small products weigh more than in larger models”. M2’s numbers stand beside M1’s, whatever they show.
• The domain is mica-dev.com, and every published number links its result directory.

10. Success criteria, fixed now

Per card, model and precision, against PyTorch’s fastest mode:

Amendment 1 below restates “level” and “slower”, because the table above reads a lead whose ranges overlap as slower.

7.7’s goal, Michael’s “beat or parity”, is met for a configuration whose verdict is faster or level. The release states the verdict of every configuration and rounds none up. A configuration without a passed correctness check has no verdict.

Amendments

Each amendment states its date, whether any series had run when it was written, and its reason.

Amendment 1, 2026-10-01, before any series: the verdicts

The table in §10 reads a Mica lead as slower whenever the two sides’ ranges overlap and Mica’s median stands more than 5% below PyTorch’s: such a lead is neither faster, because the ranges overlap, nor level, because the medians differ by more than 5%. That says the opposite of what was measured. The verdicts now read:

A lead whose ranges overlap therefore reads level and never faster, and “faster” keeps its meaning: no run of Mica’s was as slow as PyTorch’s fastest.

Amendment 2, 2026-10-01, before any series: which modules NVRTC compiled

§6 asks every series to record which of the Mica program’s modules loaded precompiled and which NVRTC compiled at the start. The runtime does not report that yet; step 5e adds the report to +mica info -mica, with NVRTC’s removal from the run-time path. Until then a series records which NVIDIA libraries the dynamic loader opened during each configuration’s first run, the discarded one: an opened libnvrtc shows that NVRTC compiled at least one module, and the same list shows what each side needs from NVIDIA at run time.

Amendment 3, 2026-10-01, before any series: where the precompiled kernels exist

§2 and §8 say that the floor’s kernels and the provider’s attention come precompiled for sm_80, sm_86, sm_89, sm_90 and sm_120. scripts/build-compiler.sh precompiles them only on a machine where NVIDIA’s NVRTC is installed; elsewhere it prints a note and embeds none. The CI machines that build the release packages have no NVRTC, as the ladder’s build log for !98 states, so a released compiler as it is packaged today carries no precompiled kernels, and every GPU program it builds compiles its kernels through NVRTC when it starts. The 4090’s development build carries them: in step 5b’s smoke run both nanoGPT programs, float32 and bfloat16, opened only the driver, cuBLAS and cuBLASLt, never NVRTC, and neither carries kernels of its own.

Step 5e therefore also covers the release build: the packaging machine gets NVRTC, packaging refuses to finish without the precompiled kernels instead of noting their absence, and mica --inspect states whether a compiler’s runtime carries them. The published series run on a release that carries them.

Amendment 4, 2026-10-02, before any published series: float16, half roundings, measured products, GPT-2 medium

Michael’s burndown of 2026-10-02 asks for float16 beside float32 and bfloat16, and for GPT-2’s medium shape beside its small one. The preparation series of that day also showed that PyTorch’s half precisions round where §4’s Mica build does not, and that PyTorch’s fastest mode chooses its products by timing them. No published series has run.

1. float16. §4’s precisions gain float16 as nanoGPT trains it with --dtype=float16: float32 weights, float16 products, and PyTorch’s GradScaler scaling the loss. The factor starts at 2^16, a step whose gradients are not all finite is skipped and halves it, and 2,000 steps taken in a row double it. The Mica program writes the same scaler out (gen-gptref.py --precision float16): it scales the loss on the card, reads every gradient back unscaled, and skips the step and halves the factor when the gradients’ norm is not finite. The Mica side reads that norm to the host every step to decide, a wait nanoGPT’s fused optimizer does not make; its cost is part of the measured step. §7’s method A covers float16 as it covers bfloat16.
2. PyTorch’s half roundings. In bfloat16 and float16 PyTorch lets cuBLAS add a product’s split sums in the half (allow_bf16_reduced_precision_reduction and allow_fp16_reduced_precision_reduction, both on by default), and FlashAttention-2 rounds the attention’s weights and score gradients into the half. The Mica side’s half builds therefore add fp_half_intermediates beside fp_tensor_float32, the opt-in flag that permits exactly those roundings. The flag changes digits, so every half series also measures the build without it, and both stand in the same table; the verdict compares the build with the flag, and §7’s check runs the build the verdict compares.
3. Products chosen on the card. PyTorch’s fastest mode times its products’ kernels and keeps the fastest. Every Mica build of a series takes --runtime-flags open and runs with +mica products=FILE -mica, which times each product’s cuBLASLt routines on the card and keeps the fastest in FILE, cuBLAS’s own choice unless another is at least 2% faster. The discarded first run of a configuration writes FILE and every timed run of the series reads it, so the timed runs print the same digits; FILE is published with the series, and §7’s check runs with the record its series uses.
4. GPT-2 medium. M2 gains GPT-2’s medium shape, M2b: 24 layers, 16 heads, 1,024 channels, about 350 million parameters, with M2’s data, batch, steps and schedule. If either side cannot hold a batch of 8 on a 24 GB card in float32, the batch halves for both, as §4 says for M2.

The runner takes the models by name (run.sh --model gpt2-small and --model gpt2-medium), and nanoGPT reads M2’s settings from benchmarks/NanoGpt/gpt2-shakespeare.py, a configuration file beside the runner, which leaves train.py as nanoGPT ships it.

Amendment 5, 2026-10-03, before any published series: a PyTorch mode that runs out of the card’s memory

The preparation table of the merged release ran M2b in float32 on the RTX 4090, and PyTorch’s max-autotune mode ran out of the card’s 24 GB while it tuned its kernels: the process held 22.8 GiB and asked for 1.5 GiB more. The same model with the same batch of 8 trains in PyTorch’s default mode on the same card, and Mica’s program trains it too. No published series has run.

1. A mode is not a side. §4’s rule, and amendment 4’s for M2b, halves the batch for both sides where a side cannot hold a batch of 8 on a 24 GB card in float32. A side holds the batch when any of its modes trains it, so one mode that runs out of memory halves nothing.
2. Recorded, not dropped. A PyTorch mode that runs out of the card’s memory is recorded in the series as out of memory, with its log, and has no time. The table names it in its row, and the comparison of its element type names it beside the mode it compares.
3. The verdict compares Mica with the fastest of the modes that ran. A Mica program that runs out of memory, or a run that fails in any other way, still ends the series, as before.

The runner records such a mode in its warm run and goes on with the series, by the container change that follows this amendment.

Amendment 6, 2026-10-03, before any published series: method B for GPT-2 medium judges the steps float32 holds

The preparation checks of 2026-10-03 ran method B for GPT-2 small and medium for the first time, on the RTX 3090 Ti. Their first runs failed at step 0, which found that Mica’s normalizations added 2^-16 under their square roots where nanoGPT adds 1e-5; compiler !149 makes them add 1e-5. With it both models agree with PyTorch at step 0 within 1e-7 and through step 25 within 5e-6, and GPT-2 small passes. GPT-2 medium’s later steps part by up to 1.42% by step 200, with the sign changing from step to step. No published series has run.

PyTorch parts from itself the same way. From the same shared start, nanoGPT trained once more in its default mode with every weight moved by at most one unit in its last place after every optimizer step, the size of the difference two stacks summing in different orders leave at each step. Its losses stood from its own run’s by up to:

Moved once, at the start alone, PyTorch stood 0.52% apart through step 100 and 0.97% by step 200. A model of 350 million parameters at GPT-2’s rates carries float32’s last bit to a percent within 200 steps, so after step 100 the 1% of §7 B judges chance for any two stacks, PyTorch and itself included.

1. GPT-2 medium’s window. For M2b, method B passes when the losses at step 0 agree within 1e-5 relative and the loss of each of the steps 1 to 100 within 1% relative. Its steps 101 to 200 stand on the same plot and judge nothing.
2. The earlier wording beside it. M2b’s record states the verdict under §7 B’s 200 steps beside the verdict under this window, as the amendment rule asks.
3. The evidence is published with the first published check: the two runs of PyTorch against itself, the change to train.py that made them, and their curves.
4. M1 and M2 keep their 200 steps: their stacks stood within 0.42% and 0.28% of each other there.

The runner judges M2b’s window, and keeps the earlier verdict beside it, by the container change that follows this amendment.

Amendment 7, 2026-10-04, before any published series: the L4’s own correctness check

§4 measures three cards, the third being the machine anyone can rent of Q3, Google Cloud’s g2-standard-8 with one NVIDIA L4. §7 names only the first two: method A trains five seeds a side on the RTX 4090, and the RTX 3090 Ti one seed a side as a check. A verdict belongs to a card, a model and a precision (§10), and rule 6 publishes a speed only after its correctness check passed, yet no check names the L4. Its series trains with the products its own first runs chose on the L4 (amendment 4.3), under the versions pinned when that series started (§4), so its digits are its own, and no check on another card has run them. Mica 7.7.0 is released, and no published series has run.

1. The L4 beside the RTX 3090 Ti. §7 A’s last clause now reads: the RTX 3090 Ti and the L4 each run one seed a side as a check. On both cards each side trains the seed 1337, nanoGPT’s own, in every precision of every model the series measures, with the build that card’s series compares and the products record it uses, as amendment 4 asks. One seed a side has no range, so the check gives no verdict: it is read, and published beside the RTX 4090’s five seeds a side, whose families give method A’s verdict.
2. Method B on every card. §7 B names no card. It runs on each of the three, the RTX 4090 included, and gives each card its own verdict by §7 B, GPT-2 medium’s by amendment 6’s window.
3. No card’s check stands for another’s. A configuration’s check has passed when method A passed on the RTX 4090 for its model and precision, method B passed on its own card for its model, and, on the RTX 3090 Ti and the L4, that card’s one seed a side is published. Only method A’s five seeds, which judge a model and a precision as families, stand for all three cards.
4. The runner is unchanged. correctness.sh runs the L4’s check as it runs the RTX 3090 Ti’s, with --seeds 1337 and --products naming the results directory of that card’s own series.

Amendment 8, 2026-10-04, while the published series ran: a run out of the card’s memory ends no series, on either side

⚖ Michael, 2026-10-04, after the published series had started: “If we have an OOM on the GPU, continue and mark this run as OOM.” Amendment 5 recorded a PyTorch mode that ran out of the card’s memory in its first run and went on with the series, but a Mica program out of memory, and a run of either side that ran out after its first, still ended the whole series, and with it every configuration not yet measured. When this was written, no run of the published series had run out of the card’s memory, and no result of it had been read.

1. Either side, any run. A configuration whose run runs out of the card’s memory, in its discarded first run or in any timed run, PyTorch’s or the Mica program’s, is recorded as out of memory with its log, has no time and runs no further, and the series goes on.
2. The verdict. Where a PyTorch mode ran out, the verdict compares the fastest of the modes that ran, as amendment 5 says. Where the Mica program ran out, its element type has no time and no verdict, and the table states it in its row: a deficit, published as measured (rule 5).
3. The other records. A profiled run or a graph-break run that runs out of the card’s memory is recorded as such, and the series names the configuration whose kernel table or break counts it lacks.
4. A failure of any other kind still ends the series.

The runner records these since container !51. On each card the series’ first model ran with amendment 5’s runner, under which such a run would have ended the series, and every later model with !51’s; each series’ record names its runner.