TR-2: Training transformers in Mica, measured beside PyTorch

Mica Technical Report 2 - first edition, October 2026, revised with each side’s footprint. States the compiler at release 7.7.0, commit ff523c023, measured by the protocol of 2026-10-01 and its eight amendments, in the series run from 2026-10-04 to 2026-10-05.

Abstract

A Mica program trains transformers on an NVIDIA card as native code: it needs no Python, no framework and no runtime compiler where it runs. This report measures it beside PyTorch 2.14.1, training the same models through nanoGPT: nanoGPT’s character-level Shakespeare model and GPT-2’s small and medium shapes, each in float32, bfloat16 and float16, on an RTX 4090 and an RTX 3090 Ti. A protocol written before the first measurement decides what is measured, how often, with which statistics and in which words, and it publishes a step time only where two correctness checks have shown that both sides train the same model.

Against the fastest of PyTorch’s four modes on each card, model and element type, the Mica program is faster in 6, level in 11 and slower in 1 of the 18 comparisons, level meaning a median less than 5% above PyTorch’s. GPT-2 medium in float32 ran PyTorch out of the card’s memory in its max-autotune mode on both cards, where the Mica program trained it with less memory than any of PyTorch’s modes. It opens cuBLAS, cuBLASLt and the driver from NVIDIA and nothing else. Every number links the raw data it comes from, and the public container repeats each one.

1. Introduction

A Mica program can train a transformer on an NVIDIA card. It holds its tensors on the card, records its forward pass on a tape and walks the tape backwards for its gradients, and its optimizer updates the weights where they live. Nothing of it is interpreted: the program is native code, built ahead of time by the Mica compiler, and it needs no Python, no framework and no runtime compiler where it runs.

This report measures such a program beside PyTorch, the framework nanoGPT is written in, on the trainings people run: nanoGPT’s character-level Shakespeare model, and GPT-2’s small and medium shapes, each in float32, bfloat16 and float16. Both sides train the same model on the same data for the same steps, on the same card, timed by one clock.

Three questions are asked, in this order:

1. Do both sides train the same model? A step time means nothing unless both sides reach the same loss by the same path. Two methods answer it (section 7).
2. How long does a training step take? Per card, model and element type, against the fastest of PyTorch’s four modes (section 6).
3. What does each side need at run time? The NVIDIA libraries each program opens while it trains (section 9).

Every number in this report comes from a protocol written and merged before the first measurement it governs. The protocol fixes the configurations, the number of runs, the statistics and the words a result may be called by, so no result could choose how it is reported. Deficits stand in the same tables, in the same type, as leads.

2. The words, one at a time

A step is one update of the model: a forward pass over a batch, the backward pass for its gradients, and the optimizer’s update of every weight.

The window. Each timed run trains 400 steps. The first 100 are warm-up and never measured: compilations, the tuning of kernels, the capture of graphs, memory pools growing, clocks rising. A run’s step time is the time from its report line of step 100 to its line of step 400, divided by 300.

The outside clock. One reader starts each program and stamps every line of its output as it arrives. Both programs print a line every ten steps, right after a read of the loss that waits for the card, so the window holds whole steps, with nothing in flight at either end. The same reader times both sides.

A configuration is one side’s program in one element type and, for PyTorch, one mode. Each configuration runs once, discarded, and then five times, once in each of five rounds; the order rotates from round to round, so heat and drift touch every configuration alike.

PyTorch’s modes. Eager runs every operation as its own launch. The default mode of torch.compile, as nanoGPT ships it, compiles the model into fused kernels. reduce-overhead adds CUDA graphs, which replay a recorded step in one launch, and max-autotune adds the timing of candidate kernels to choose the fastest. The comparison is against the fastest of the four, as measured on the card, model and element type.

The element types. float32 multiplies on the tensor cores in TF32 on both sides, nanoGPT’s own setting; the Mica program permits it with fp_tensor_float32. In bfloat16 and float16 the weights and the residual stream stay float32 and every matrix product reads the half type, as PyTorch’s autocast does; float16 scales the loss as PyTorch’s GradScaler does.

Half roundings. PyTorch lets cuBLAS add a product’s partial sums in the half type, and its FlashAttention rounds the attention’s weights into the half type. The Mica build compared in a half type permits exactly those roundings with the opt-in flag fp_half_intermediates; the build without it, every digit kept, stands beside it in every table.

The products record. PyTorch’s fastest mode chooses its kernels by timing them. Each Mica build does the same for its matrix products: its first run, discarded, times the cuBLASLt routines for every product shape on the card and keeps the fastest in a file, which every timed run of the series reads, so they all run the same routines.

The verdicts. Per card, model and element type: faster when Mica’s slowest run is faster than PyTorch’s fastest run; level when not faster and Mica’s median stands less than 5% above PyTorch’s; slower when Mica’s median stands 5% or more above PyTorch’s, published with the ratio of the medians. A configuration whose correctness check has not passed has no verdict.

3. The two programs

3.1 nanoGPT

nanoGPT is Andrej Karpathy’s compact implementation of GPT training in PyTorch. Its train.py runs unchanged in its eager and default modes. The two modes beyond the default take one published change, which hands the mode to torch.compile; the correctness checks take two more, one making the seed a setting and one reading a recorded batch order (section 7). GPT-2’s shapes read their settings from a configuration file beside the runner, so train.py stays as nanoGPT ships it.

3.2 The Mica program

The Mica side is one program, GptRef.mica, which a script writes from the same numbers nanoGPT’s configuration states, layer by layer in nanoGPT’s order: embeddings, the blocks with their layer normalizations, causal self-attention over one product holding the queries, keys and values side by side, the GELU MLP, the residual stream, the loss, AdamW with its warm-up and cosine decay, and the clipping of the gradients’ norm.

How its kernels are made: Mica writes every kernel in its intermediate language; Dragon’s NVIDIA provider renders it as CUDA C, and NVIDIA’s NVRTC compiles it to the card’s machine code; matrix products run in cuBLAS. The compiler’s own kernels and the attention ship precompiled for five card generations (sm_80, sm_86, sm_89, sm_90, sm_120), and a program’s own kernels are precompiled when it is linked, so on those generations the program loads every kernel ready and never opens NVRTC; it opens cuBLAS, cuBLASLt and the driver alone, and each series records that it did.

The published builds are --optimize O2,fp_tensor_float32, with fp_half_intermediates for the half types, and every check of O2 stays on; each build’s manifest is recorded beside its runs.

4. Method

The method is the measurement protocol, published with this report. Its rules, in short:

1. Fixed before measuring. Models, configurations, runs, statistics and success criteria were written down and merged before the first run they govern; a later change is an amendment, dated, with its reason, and never changes which configuration is the comparison.
2. PyTorch at its best: the fastest of its four modes, as measured.
3. One instrument for both sides: the outside clock. Each program’s own reading is recorded beside it; the Mica program’s must agree with the clock within 1%, or the run is void.
4. Every number carries its conditions: runs, median and range, card, versions, date, commit and raw logs.
5. A deficit is published as measured.
6. Speed is published only for a configuration whose correctness check passed.
7. The raw data is published: every log, kernel table, loss curve and script.
8. A measurement is not a projection.
9. Published numbers come from the released compiler.

4.1 The amendments

4.2 Correctness

Method A, families of runs. Each side trains the model’s whole schedule with five seeds, 1337 to 1341, in every element type, evaluated every 250 steps on 200 batches of each split. The two sides draw different dropout masks and batches, so no single pair of runs can match; the families must. The check passes when, at every evaluation from step 1,000 on, the two sides’ median validation losses differ by no more than the larger of the two sides’ ranges at that step. The RTX 4090 runs the five seeds; the RTX 3090 Ti runs one seed a side as a check of its own.

Method B, one shared start. Both sides start from one set of initial weights made by nanoGPT’s own initialization, read one recorded batch order, train without dropout and multiply in plain float32, so only the order of their sums differs. The check passes when the losses at step 0 agree within 1e-5 of the loss and those of each later step within 1%; for GPT-2 medium, through step 100, where float32 alone carries two runs of PyTorch itself a percent apart by step 200 (amendment 6). Each card runs it.

5. Setup

Two cards ran the series this report publishes, the RTX 4090 and the RTX 3090 Ti, in two workstations, each with the released compiler as its packages ship it. The protocol names a third card; it is not part of this report, and its series and checks stand in the container as they were measured. The table is read from each series’ machine record:

Every series ran Mica Optimizing Compiler Version 7.7.0 ff523c023, PyTorch 2.14.1+cu130 cuda 13.0 cudnn 92400, Triton 3.8.0, Python 3.14.4 and nanoGPT at 3adf61e, measured from 2026-10-04 to 2026-10-05.

The clocks are not locked: the cards run as their owners’ do, and every run samples the card’s clocks, temperature, power and memory once a second.

6. Step time

Every comparison at a glance: the verdict, the ratio of the medians and, under it, the mode of PyTorch 2.14.1 it was measured against, the fastest of its four on that card, model and type. Each model links its series, and the tables below name both sides’ times and PyTorch’s attention kernels.

nanoGPT’s Shakespeare model is a 10.7-million-parameter model with a 256-character context, where launches and small products weigh more than in larger models. GPT-2 small and medium stand beside it, with 123.59M and 353.55M parameters as nanoGPT counts them, over GPT-2’s context and vocabulary.

Each table below is one card. A row is one model in one element type: the Mica program’s median step time over its five timed runs, with the fastest and the slowest of them in brackets; PyTorch’s fastest mode on that card, model and type, with its median and range and the attention kernels it ran; the ratio of the two medians, below 1.000 where the Mica program is faster; and the verdict of section 2. Times are milliseconds per step.

RTX 4090

RTX 3090 Ti

The half types without the half roundings

Each half type's build without fp_half_intermediates, every digit kept, against the same PyTorch mode as above.

Slower, as measured and in the same tables: nanoGPT's 10.7-million-parameter model in bfloat16 on the RTX 4090, 1.052 against PyTorch 2.14.1's max-autotune with FlashAttention. Section 8 shows where the time goes.

Where PyTorch ran out of the card’s memory

A mode that runs out of the card’s memory has no time, and the comparison is against the fastest of the modes that ran (amendment 5). It happened on PyTorch’s side alone, and only to GPT-2 medium in float32. The table gives, for those runs, each configuration’s peak memory over its timed runs, or that it ran out of the card’s memory.

The Mica program trained every model in every element type on both cards, and in these rows with less memory than any of PyTorch’s modes.

7. Correctness

Each check ran after its series, with the series’ record of measured products and with nanoGPT in the mode the series compares for each element type (amendment 4). Method A’s families ran on the RTX 4090 and give the verdict for every card; method B and the single seed a side belong to the card they ran on (amendment 7). Each check links its results: every run’s output, both sides’ curves plotted together, and the record the verdict was read from.

Method A, families of runs

Method B, one shared start

Each card's own seed a side

One seed a side on the cards that did not run the families, read beside them and judged by none.

Why GPT-2’s final losses stand high. GPT-2’s two shapes learn the small Shakespeare corpus by heart within their schedule: their validation loss falls at first and then rises while their training loss falls toward zero, on both sides alike at every evaluation, so their final validation loss stands above its lowest. Method A compares the two sides’ curves at every evaluation, which overfitting leaves as comparable as learning does.

GPT-2 medium’s window. Method B judges GPT-2 medium through step 100 and states the 200 steps beside it. From one start, float32 carries the last bit of a model of 353.55M parameters to a percent within 200 steps, for any two stacks: nanoGPT trained from weights moved by one unit in their last place, once at the start or after every step, the size of the difference two stacks summing in different orders leave, parted from its own run about as far as the Mica program parts from it. Those two runs of PyTorch against itself, the change to train.py that made them and their curves are published with the checks (amendment 6).

8. Where the time goes

Each series profiles one run of every configuration with NVIDIA’s Nsight Systems and sorts the kernels of its steady window into families by one rules file, the same for both sides (families.tsv in the container). A family’s time is its kernels’ time per step; launches, gaps and the host are not in it. A profiled run’s kernel time explains a step time and never stands in for one.

nanoGPT’s 10.7-million-parameter model with a 256-character context, in bfloat16 on the RTX 4090, the comparison PyTorch wins by the widest margin:

RTX 4090, nanoGPT, bfloat16: the profiled runs' window, the Mica program against PyTorch 2.14.1 in max-autotune with FlashAttention, 20261004-211558-NVIDIA-GeForce-RTX-4090.

The products stand level. The difference sits in the kernels around them - the GELU, the layer normalizations with the dropout and the residual, and the attention - which PyTorch’s max-autotune partly fuses into its products, and in the time between kernels, which its CUDA graphs shorten; PyTorch spends time on layout kernels the Mica program never runs.

The largest model in the same type and on the same card, GPT-2 medium:

RTX 4090, GPT-2 medium, bfloat16: the profiled runs' window, the Mica program against PyTorch 2.14.1 in max-autotune with FlashAttention, 20261004-171016-NVIDIA-GeForce-RTX-4090.

Every series holds the tables of all its configurations, and the kernels by name.

9. What each side needs at run time

What each program opened from NVIDIA while it trained, read from the dynamic loader in the first run of every configuration, counted over the comparisons of section 6:

The Mica program loads every kernel precompiled for the card’s generation and opens NVIDIA’s runtime compiler on neither card: what it needs where it runs is the program, cuBLAS and the driver.

The protocol’s section 8 also measures each side’s smallest container image that trains, built from the Dockerfiles in the container’s benchmarks/NanoGpt/images/: Mica’s holds the program, the NVIDIA libraries it opens and the system libraries those need; PyTorch’s holds Python and what pip install torch numpy brings, with nanoGPT’s own files. Each image proves itself by training on a card, on a machine with NVIDIA’s container toolkit:

nanoGPT in bfloat16, PyTorch in its mode default, 5000 steps on each side; the driver's libcuda comes from the host in both: 20261005-113556-NVIDIA-L4-footprint.

Each image trained the model on the card to the loss its row shows. PyTorch’s also needs a C compiler beside Python, which Triton asks for when it compiles the kernels of a compiled mode.

10. Limits, stated

• Model size. nanoGPT’s model is small; GPT-2’s shapes are measured beside it.
• The corpus. GPT-2’s shapes train on nanoGPT’s Shakespeare text, far smaller than what they were made for, and overfit it within their schedule: this report measures how fast both sides train and whether they train alike, not a model worth using.
• The processor. Training on the processor against PyTorch’s CPU path is not measured here.
• The image network. Convolutional networks are not measured here.
• The clocks are not locked, as stated in section 5.

11. Related work

nanoGPT [1] is the compact GPT training this report measures against, unchanged in its model and in its training loop. GPT-2 [2] gives the two larger shapes. PyTorch 2 [3] compiles a model’s Python with TorchDynamo and generates its kernels with TorchInductor, largely in Triton [4]; its reduce-overhead mode replays CUDA graphs and its max-autotune mode times candidate kernels, which is why the comparison is against the fastest mode as measured rather than any one. Its attention runs FlashAttention [5, 6] in the half types and the memory-efficient attention of xFormers [7] in float32. Mixed-precision training with a scaled loss [8] is what PyTorch’s autocast and GradScaler do, and what the Mica program does in the half types. The Mica program’s matrix products run in NVIDIA’s cuBLAS and cuBLASLt [9]; PyTorch’s run in cuBLAS too in its first three modes, and under max-autotune largely in the Triton kernels its timing chose, many with the next operation fused in.

12. Conclusion

A Mica program, compiled ahead of time to native code, trains nanoGPT’s Shakespeare model and GPT-2’s small and medium shapes on two NVIDIA cards in float32, bfloat16 and float16, and trains the same models as PyTorch does: both correctness checks say so for every model on every card. Against the fastest of PyTorch’s four modes, measured on each card, model and type, it is faster in 6, level in 11 and slower in 1 of the 18 comparisons; what is slower is published as measured, with where its time goes.

It does so without what PyTorch brings to the machine it trains on: no interpreter, no framework, no runtime compiler, and a fraction of the NVIDIA libraries. Every number in this report was fixed in its method before it was measured, links the raw data it comes from, and can be measured again by anyone with an NVIDIA card or a rented one, from the public container.

How to repeat every number

Every number of this report is measured again from the public container, mica-dev/mica-container, on a machine with an NVIDIA card and the released compiler, with the versions each series records. One card’s series and checks of one model, as the published ones ran:

mica-provision nvidia gpu-profiling gpu-containers     # cuBLAS and NVRTC, the profiler, Docker for the footprint
cd benchmarks/NanoGpt
./run.sh both --model nanogpt --dtypes float32,bfloat16,float16 --modes eager,default,reduce-overhead,max-autotune \
    --rounds 5 --publication --optimize O2,fp_tensor_float32,fp_half_intermediates --products-record \
    --torch 2.14.1 --nanogpt 3adf61e                      # the series: results/<series>
./correctness.sh --publication --method b --model nanogpt --modes-of results/<series> \
    --torch 2.14.1 --nanogpt 3adf61e                      # method B: results/<check-b>
./correctness.sh --publication --method a --model nanogpt --dtypes float32,bfloat16,float16 \
    --seeds 1337,1338,1339,1340,1341 --products results/<series> --modes-of results/<series> \
    --optimize O2,fp_tensor_float32,fp_half_intermediates \
    --torch 2.14.1 --nanogpt 3adf61e                      # method A: results/<check-a>
./series.py take results/<series> results/<check-a> results/<check-b>

GPT-2’s shapes take --model gpt2-small or --model gpt2-medium. A card that does not run the families runs method A with --seeds 1337, and its series takes the families’ check beside its own. The footprint is ./footprint.sh --model nanogpt --dtype bfloat16 --mode <the series' bfloat16 mode> with the same versions. The container’s benchmarks/NanoGpt/README.md states every option, and its results/ holds every series and check of this report, each linked from its table, and the scripts that started each card’s series and checks, with their logs and what happened on the way.

References

1. A. Karpathy. nanoGPT. https://github.com/karpathy/nanoGPT, 2022.
2. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI, 2019.
3. J. Ansel et al. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. ASPLOS 2024.
4. P. Tillet, H. T. Kung, D. Cox. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. MAPL 2019.
5. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022.
6. T. Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR 2024.
7. B. Lefaudeux et al. xFormers: A modular and hackable Transformer modelling library. https://github.com/facebookresearch/xformers, 2022.
8. P. Micikevicius et al. Mixed Precision Training. ICLR 2018.
9. NVIDIA. cuBLAS. https://docs.nvidia.com/cuda/cublas/.