Runtime flags

A Mica program can be told a few things when it starts, on its own command line, ahead of its own arguments:

./server +mica carriers=2 -mica --port 8080

Everything between +mica and -mica belongs to the runtime. It is read before any unit initializes and before the program’s first statement, and none of its words reach the program: ArgCount and Arg see --port 8080 and nothing else.

What a program may be told is decided when it is built, not when it starts. A program tested with two carriers and shipped with that promise keeps two carriers wherever it is started. No environment variable takes part: every choice that changes a running program stands on the line that started it, where anyone can read it.

The flags

info answers after every other flag in the group was read, so +mica carriers=2 info -mica shows the pool the program would start with:

{"mica_runtime":1,"manifest":{"mica_manifest":1,...,"tasking":{"flavor":"multicore","carriers":4,"pinned":false,"processes":null},...},"configuration":{"carriers":{"effective":2,"source":"flag","oversubscribed":false},"processes":{"effective":16,"source":"processors","oversubscribed":false},"stdout":"default","products":null}}

manifest is the program’s manifest, the same document mica --inspect reads from the program file without starting it. configuration is what only a start can know:

A program that places code on a GPU answers one more member, device, about the card it would run on:

"device":{"provider":"nvidia","card":0,"name":"NVIDIA GeForce RTX 4090","generation":"sm_89","precompiled":true}

check adds one member, check, with the facts behind the count:

taskset -c 0,1 ./server +mica check -mica

{"mica_runtime":1,"manifest":{...},"configuration":{"carriers":{"effective":2,"source":"affinity","oversubscribed":false},"processes":{"effective":2,"source":"affinity","oversubscribed":false},"stdout":"default","products":null},"check":{"processors":{"online":16,"affinity":2,"quota":{"readable":true,"cgroup":null,"microseconds":null,"period":null,"carriers":null},"usable":{"carriers":2,"source":"affinity"}}}}

usable matters beside a count the build or a flag named: a program built with --tasking multicore,8 and started in a container limited to two processors still starts eight carriers, because the build’s count is a promise, and check shows that six of them can never run at once, with "oversubscribed":true.

How many carriers a pool starts

A program built with --tasking multicore and no count sizes its carrier pool when it starts, to the least of three facts the kernel states:

• the processors online on the machine;
• the processors its affinity mask allows, as taskset or a container’s --cpuset-cpus restricts it;
• the CPU quota of its cgroup, as a container’s --cpus or a Kubernetes CPU limit sets it, rounded up to whole processors. The tightest quota on the path from the program’s cgroup up to the cgroup root applies.

A program in a container limited to one and a half processors on a sixteen-processor machine starts two carriers, not sixteen. The quota is read from the cgroup hierarchy that holds the CPU controller: cgroup v2, or cgroup v1 on older hosts and hybrid layouts where the controller still sits there.

Any count from 1 to 1024 runs. On a sixteen-processor machine, a thousand tasks ran on 1024 carriers in 0.3 seconds with about 100 MB of memory, and gave the same results as on one carrier. Carriers beyond the processors spend processor time looking for work rather than doing it, so the processors the program may use are the healthy count, which is what a pool without a count starts with.

Which card a program uses

A program that places code on a GPU runs its device work on one card: the first, unless the start names another.

./train +mica check -mica           # lists the machine's cards
./train +mica gpu=1 -mica           # trains on the second

check lists every card with the number the flag names it by, as NVIDIA’s driver numbers them:

"cards":[{"card":0,"name":"NVIDIA GeForce RTX 4090","generation":"sm_89"},{"card":1,"name":"NVIDIA GeForce RTX 3090 Ti","generation":"sm_86"}]

The number is checked against the machine’s cards before the program’s first statement, so a number with no card behind it is refused at the start rather than at the program’s first device use, which may come late in a run. Which card a program uses is a fact of the machine it runs on, not of its build, so every policy that reads a group admits the flag, sealed included.

How many worker processes run at once

A task placed on a process runs in a worker process of its own. However many tasks a program places, at most so many workers run at once; a task placed beyond the count waits until a worker ends, and then runs. A thousand placements on a two-processor machine run two at a time rather than starting a thousand processes together.

The count is, in order:

• the processes=<n> flag, where the build’s policy admits it;
• the build’s --processes <n>;
• the processors the program may use when it starts, the same count a carrier pool without a count starts with.

A placement that waits holds no carrier: the program’s other tasks run on while it waits, and a cancelled scope ends the wait without starting a worker. A stream on a process takes no worker’s place, because its consumer paces it: a program may read a stream from a worker and place tasks on processes for what it reads, whatever the count.

Tasks placed on processes that wait for each other, such as a server and its client placed side by side, need a count at least as large as the tasks that wait together, which --processes states for every machine the program runs on.

Where the group stands

The group is read only at the very head of the arguments. A +mica anywhere later is an ordinary argument of the program, so a runtime never takes words that belong to the program. A program whose first argument must be the word +mica is started with an empty group in front of it:

./tool +mica -mica +mica      # the program receives one argument: +mica

The run mode passes the group through unchanged: mica --run tool.mica +mica info -mica.

The build’s policy

--runtime-flags states what the built program accepts:

A shipped release binary therefore answers questions about itself and changes only where its build said so. stdout=line belongs with the questions: it decides when the lines arrive, not what the program does, so a release service under a supervisor or in CI shows its log as it writes it. gpu=<n> does too: the card is the machine’s, so a release program trains on the card the one starting it chooses.

Products measured on the card

A matrix product on a GPU runs by a routine of NVIDIA’s cuBLAS, which picks one by its own ranking. That choice is right for most shapes on most cards, and leaves time on some. With products=<file>, the first product of each shape times the routines cuBLAS’s own cuBLASLt ranks for it, and then searches past them through every configuration of the algorithms cuBLASLt lists for the product’s types, on the card the program runs on, and keeps the fastest; every later product of the shape runs by it. Products of float32, bfloat16 and float16 are measured; a float64 product keeps cuBLAS’s own choice. cuBLAS’s own routine takes part in every measurement, and a measured routine replaces it only when it is clearly faster, so a record never makes a program slower. On an RTX 3090 Ti, the ranked routines took nanoGPT’s bfloat16 training step from 25.9 ms with cuBLAS’s own choice to 24.4 ms, and the search past them to 23.8 ms.

The search costs its time once, in the run that writes the record: about twenty seconds over nanoGPT’s dozen product shapes, about forty over GPT-2 small’s. Every later run reads the record and measures nothing.

The routine decides in which order a product’s sums are added, so it decides the last digits of the answer. That is why the choice is kept: the record holds one line for each shape, and a run that names the same file reads its choices instead of measuring again. The run that measured and every run that reads its record print the same digits:

./train +mica products=train.products -mica     # measures, and keeps the choices
./train +mica products=train.products -mica     # reads them: the same digits

A record’s first line names the card’s architecture and the library’s version it was measured with. A record made for another card or another version is begun again, and the program says so on standard error. Which routines may be chosen follows the build: a program built without fp_half_intermediates chooses only among routines that add a product’s partial sums in float32, as cuBLAS’s own choice does.

The flag changes which digits a build prints, so a build admits it only under the open policy, the debug default.

Pinned values

A value spelled as fixed in the build cannot be changed at start, whatever the policy:

mica --tasking multicore,2,fixed ...

,fixed stands behind a count and nowhere else.

Refusals

A flag the build does not allow ends the start before anything of the program has run: one line on standard error, and exit status 2.