A Mica string holds code points in the platform’s own encoding, utf-32 by default and utf-8 when the platform says so, and a program never sees which: s[i] is the i-th code point and Length counts code points under either. Outside the program the world speaks UTF-8, on disk, on the wire, across the C boundary. The utf unit is where a program crosses that line on purpose, and this page walks its two directions and its one cursor.

The working example is examples/Transcoding:

make -C examples/Transcoding run

Bytes, the one name for a run of bytes

The unit trades in Bytes, declared once in std as a plain array of uint8. Because it is an ordinary dynamic array, everything a dynamic array offers applies to an encoded run unchanged: Length, Append, Reserve, Capacity, spans, and the arrays unit.

imp Bytes : std;
imp EncodeUtf8, Utf8Length : utf;

wire := EncodeUtf8(text);               { the interchange bytes, whatever the platform encoding }
WriteLn("runes %lld, bytes %lld, measured %lld", Length(text), Length(wire), Utf8Length(text));

Length(text) counts code points and never bytes; Utf8Length is the byte count of the interchange form, the size a wire frame or a file header wants, without producing the bytes. Under the utf-8 platform the encode is a copy of the string’s own bytes; under utf-32 it is a transcode. A program cannot tell, which is the point.

Two roads back, and a domain for the reasons

Bytes come back as text on two roads. DecodeUtf8 is the strict one: it reads well-formed UTF-8 and fails with the reason when the bytes are not, in the words of the UnicodeError domain. DecodeUtf8Lossy reads whatever it is handed, the replacement character U+FFFD standing in for every ill-formed sequence, one per maximal ill-formed subpart, the policy the Unicode standard recommends. ValidUtf8 asks the question alone.

imp DecodeUtf8, DecodeUtf8Lossy, ValidUtf8 : utf;
imp UnicodeError, Malformed, Truncated, Overlong, Surrogate, OutOfRange : utf;

s := DecodeUtf8(wire) on fail e do
    case e of
        Malformed:  WriteLn("a byte that opens or continues no sequence");
        Truncated:  WriteLn("the bytes end inside a sequence");
        Overlong:   WriteLn("a code point spelled in more bytes than it needs");
        Surrogate:  WriteLn("a surrogate, which UTF-8 never carries");
        OutOfRange: WriteLn("a value past the last code point")
    end;

The case over the domain must name every reason, so a reason added in a later release is a compile error in every handler that forgot it, the error domains story applied to text. Which road a program takes is a statement about its data: a protocol frame, a document or a key is a contract, and a silent repair would hide the fault, so the strict road fails; a log line, a foreign file name or a peer’s bytes must be shown even when broken, so the lossy road repairs. Truncated is the one reason a reader assembling a stream from chunks waits on rather than rejects. The other four are damage.

Well-formedness is the Unicode standard’s own, checked by shape as the bytes are read: a lead byte fixes the width of its sequence, and for the four leads E0, ED, F0 and F4 it also narrows the second byte, which is where an overlong form, a surrogate and a value past U+10FFFF are refused without decoding them first. The same reading decides every reader in the runtime, files and sockets included, so damaged input reads as the same text whichever encoding the program was compiled for.

The cursor: a stream that arrives in chunks

A socket fills a window, a file drains in blocks, a pipe answers what it has. A chunk boundary falls wherever it falls, and a code point three bytes wide is cut in half by it as often as not. Decoding chunk by chunk through the one-shot verbs would turn every cut code point into two replacement characters. The decode cursor carries the cut across.

imp DecodeCursor, DecodeStart, Decoding, NextText, EndText : utf;

c := DecodeStart();
while Decoding(address c) do begin
    if at >= Length(b) then
        piece := EndText(address c) on fail leave
    else begin
        piece := NextText(Chunk(b, at, size), address c) on fail leave;
        at := at + size;
    end;

    Append(address out, piece);
end;

It is the same three-verb idiom the strings and regex units teach, over a stream instead of a text: DecodeStart opens the walk, Decoding guards the loop, NextText decodes one chunk continuing whatever the previous chunk ended inside, and EndText closes the stream, which is the one moment a sequence still pending can be found truncated. The strict verbs fail with the reason and set the cursor’s position to the stream offset of the offending sequence’s first byte, inside an earlier chunk when the sequence began there; NextTextLossy and EndTextLossy are the same walk with the replacement character instead of a failure. The example feeds one text through the cursor in chunks of every size from one to five bytes and gets the same seven runes back every time.

What the compiler already gives text

The unit adds nothing the language did not have a place for. The byte-level codec, one code point to its bytes and one byte run classified, is the runtime’s, reached through the standard library’s realization entries; everything above it is written in Mica over Bytes and the stringbuffer. A reading verb takes a Bytes by value, as every value in Mica is passed, and the compiler passes the caller’s array by address because the body only reads it: the one address such a body hands out lands in a parameter the runtime declares constant, the resting place the read-only inference admits, so no call copies its input. The type system reference states the constant, and the contract reference the const_pointer kind that carries the promise across the boundary.