Characters, names and literals

This chapter describes the smallest pieces of a Mica program: the characters a source is made of, the comments and white space between its tokens, the names it declares, the words that belong to the language, the operators, and the literals. Each construct names its rule in the grammar, says what it means, and quotes what the compiler answers when a source breaks the rule. How tokens form a program, a library and its imports is chapter 2.

Every example is a complete program that the compiler’s own test suite compiles and runs on AMD64 and AArch64, and the output shown beneath it is the output those runs are held to.

Reading a refusal

A source the compiler refuses is answered with a report that names the file, the code of each refusal, its position as line and column, and its words. The refusals below are quoted by code and words, as the compiler prints them:

Error Report for source 'Probe.mica'
Errors:
    10: scanner error 1039 [10,10]: '<>' is not an operator in Mica: '=' compares for equality and '#' for inequality
Failed to compile source 'Probe.mica': compilation aborted

A scanner error comes from reading the characters, a parser error from the grammar, and an analyzer error from what the program means. Lines are counted from one; a column counts the characters of its line from one, and a tab counts up to the next multiple of four.

The source text

A source is a file of UTF-8 text, read into tokens with white space and comments between them (source). Before the first token is read, three things are set aside:

• a byte-order mark as the first character of the file (byte-order-mark);
• a first line beginning with #!, described below;
• every carriage return, so a file whose lines end in a carriage return and a line feed reads exactly as one whose lines end in a line feed. That holds inside a string literal too: a string holds a carriage return only through the escape \r.

A line ends at the line feed U+000A (newline). Any Unicode white space character stands between two tokens: the space, the tab, the line feed, the no-break space and the others of its class (white-space). A tab inside a literal is a character of the literal.

The compiler refuses a source that is not honest text, and changes nothing of it silently:

The first five such findings are reported each with its offset; the rest are counted in one closing finding, 1038.

#!/usr/bin/env mica
{
    SourceText - the characters a Mica source is made of, and the text that stands between its tokens.

    A source is UTF-8 text, so a string may hold any character a Unicode editor writes. The line above is set aside
    before the source is read: the system finds the program that runs a script by it, and it still counts as line one.
    A block comment runs from its brace to the brace that matches it,
        { and a block comment may hold one of its own, }
    while a line comment runs from two slashes to the end of its line.
}
program SourceText;

imp
    WriteLn : std;

const
    Greeting = "Grüße, κόσμε, 世界";

begin
    // the program body is one statement, a compound one here
    WriteLn("%ls", Greeting); { a comment may stand between any two tokens }
end.

It prints:

Grüße, κόσμε, 世界

Comments

A comment is a block in braces or the rest of a line behind two slashes (comment), and it may stand between any two tokens.

A block comment runs from { to the } that matches it, and may hold block comments of its own (block-comment). The braces are counted wherever they stand inside it, within quotes as well: a comment that mentions "}" ends at that brace. A block comment the file never closes is refused with scanner error 1002, end of file reached inside comment, and the parser then reports the end of the file it met at each place that still waited for a token.

A line comment runs from // to the end of its line (line-comment).

The first line

A source whose first line begins with #! carries the line the system reads to find the program that runs it as a script (interpreter-line). The compiler sets the line aside, in a build, in the editor and in a run alike, and counts it: the line after it is line two, as an editor shows it. Anywhere else # is the inequality operator, so no program can begin with it. The command line runs such a file with mica --run, or as mica tool.mica.

Names

A name is a word that is not a reserved word (identifier). A word follows Unicode’s identifier rule: it begins with a letter of any script and goes on with letters, decimal digits, the combining marks a script writes its words with and the subscript digits ₀ to ₉, a single underscore allowed between two of them (word):

• A letter is a letter of any script: größe, θ, ℓ and 名前 are names.
• A combining mark goes on a name behind the letter it belongs to: नमस्ते holds a virama and a vowel sign, and สวัสดี its vowels and tone, each a character of the name.
• A digit may continue a name, a decimal digit of any script included, so x٣ is a name; a name never begins with one.
• An underscore stands between two letters or digits, so max_depth and O_RDONLY are names.
• The middle dot · is the dot product’s operator, so it is no character of a name, though Unicode’s rule admits it in one.
• Upper and lower case are different letters: Point and point are two names, and Begin is a name though begin is a reserved word.
• A name has no length limit, and every one of its characters counts.
• Two names are the same name when their characters are the same in Unicode’s Normalization Form C, the one spelling the compiler compares and keeps: é typed as one character and é typed as e followed by the combining accent U+0301 are one name. Two letters that only look alike, a Latin a and a Cyrillic а, stay two letters.
• A combining mark that no letter carries, at the start of a word, and an enclosing mark, which draws a circle or a keycap around its letter, are no part of a name; the refusal names the mark by its code point, and an accent by its Unicode name too.
• The zero width joiner U+200D and the zero width non-joiner U+200C draw nothing themselves; they ask for two characters joined or kept apart. A name holds one only where its script needs it to choose a written form: a joiner behind a virama, as Hindi writes the half form क्‍ष beside the conjunct क्ष, and a non-joiner behind a virama before a letter, or between two letters a script would otherwise join, as Persian keeps two letters of نامه‌ای apart, all of one script. Anywhere else a joiner changes nothing a reader sees, so the name is refused, and the refusal names the joiner by its code point and its Unicode name.

{
    Names - what a name may be made of, and the words that stay names outside their one position.

    A name begins with a letter of any script and goes on with letters, decimal digits of any script, the combining
    marks a script writes its words with and the subscript digits, a single underscore allowed between two of them, as
    the Hindi greeting holds its virama and its vowel sign. Upper and lower case are different letters, so 'Point' and
    'point' are two names. The words that carry a meaning only in one position of the grammar - 'next', 'select' and
    'task' among them - are ordinary names everywhere else.
}
program Names;

imp
    WriteLn : std;

type
    { A point of the plane, its type's name told from the variable's by the case of its first letter alone. }
    Point = record
        x : float64;
        y : float64;
    end;

var
    point : Point;
    größe : int64;
    θ₂ : float64;
    x٣ : int64;
    max_depth : int64;
    नमस्ते : int64;
    next, select, task : int64;

begin
    point.x := 1.5;
    point.y := 2.5;
    größe := 180;
    θ₂ := 0.25;
    x٣ := 3;
    max_depth := 16;
    नमस्ते := 7;
    next := 1;
    select := 2;
    task := 3;
    WriteLn("point %lf %lf", point.x, point.y);
    WriteLn("größe %lld, θ₂ %lf, x٣ %lld, max_depth %lld", größe, θ₂, x٣, max_depth);
    WriteLn("नमस्ते %lld", नमस्ते);
    WriteLn("next %lld, select %lld, task %lld", next, select, task);
end.

It prints:

point 1.500000 2.500000
größe 180, θ₂ 0.250000, x٣ 3, max_depth 16
नमस्ते 7
next 1, select 2, task 3

Reserved words

A reserved word belongs to the grammar everywhere and never names anything. The reserved words are the ones that open, close or join the structures of a program (its blocks, declarations and statements) and the operators written as words. Forty-seven words are reserved (reserved-word):

and · array · as · begin · bitrecord · bitset · borrowed · case · const · defer · dispose · do · downto · else · end · exp · for · function · gen · if · imp · in · is · leave · mod · nil · not · odd · of · or · owning · packed · pointer · procedure · program · record · repeat · reserve · shl · shr · then · to · type · until · var · while · xor

They are written in lower case. A reserved word written where a name is declared is refused once, with parser error 2065, and the compiler reads on as if the name had been written: 'begin' is a reserved word and cannot stand as a name here: it belongs to the grammar — rename what you are declaring if you meant a name, or check the keyword's syntax if you meant the construct itself.

Contextual words

A contextual word has its meaning only in the place where a rule of the grammar uses it, and is an ordinary name everywhere else: a field can be named value, a variable file, and a constant library. Forty-four words are contextual:

absolute · address · at · buffer · callback · concurrent · continue · emit · error · fail · fails · file · generator · gpu · heapless · int · into · intrinsic · library · lost · matrix · new · next · object · on · process · readonly · schedule · select · set · span · stream · synchronized · tape · task · tensor · timeout · tracked · uint · use · value · vector · when · writeclear

Where a contextual word and a name could stand in the same place, the token behind the word decides: value p reads what p points at, and value := 3 assigns a variable named value. In a type, set of, file of, vector of and tensor of always begin the language’s own types, so a generic type cannot be named set or file, and value ( and address ( always read what stands in the parentheses, so a routine cannot be named value or address. Each rule of the grammar that uses a contextual word says where it is special, and with is an ordinary name.

Words the language adds arrive as contextual words, so a program that compiles keeps compiling when a new word arrives.

Predeclared names

Beside its reserved words, the language declares names of its own, and a program cannot declare any of them again: not in its root, not in a routine and not in a block.

• Six names are declared in every unit: main, True, False, std, cstd and math. True and False are the two values of the type bool, and main is the name a program’s entry point carries. std and math name the language’s own units and cstd the C library’s surface, which chapter 2 imports from.
• Eighteen names are the built-in types: int8, int16, int32, int64, uint8, uint16, uint32, uint64, float16, bfloat16, float32, float64, bool, unicode, string, stringbuffer, stringpart and descriptor.

Each kind of declaration refuses these names with a code of its own:

The other names are refused in the same words: a variable called True is refused with variable identifier cannot use built-in reserved name: True. A type’s refusal names the meaning the type would hide: type std = int64; is refused with the type 'std' would hide the namespace of the standard library: a type cannot take a name the language predeclares, so give it a name of its own, and type True = int64; with the type 'True' would hide the logical constant of that name: a type cannot take a name the language predeclares, so give it a name of its own.

A field is named inside its record, so a record may hold a field called int64 or string, read as box.int64.

The routines of the standard library are not predeclared names. A program that does not import WriteLn may declare a variable called WriteLn. A program that imports it holds the name in its root, so a second declaration there is refused as every duplicate is, with variable identifier already declared: WriteLn (5019), while a routine may still declare its own WriteLn, which hides the imported one inside that routine. Every library follows this rule, the C library, POSIX, Linux and the Dragon backend among them: a program that imports UnitNew from dragon may still declare a type called ContextNew.

Context words

Thirty-seven words carry a meaning in one position of the grammar and are ordinary names everywhere else, so a program may keep a variable called next, select or error:

next and when are refused as the names of loops, so a statement that names its loop can never be read two ways: 'next' cannot name a loop: the word introduces a statement of the leave family, so a loop of this name could never be targeted (5229), and for when, 'when' cannot name a loop: 'leave … when' and 'next when' read the word as the start of their condition, so a loop of this name could never be targeted.

Operators and punctuation

The operators and punctuation are these spellings (operator):

:= · .. · + · - · * · / · = · # · < · <= · > · >= · ( · ) · [ · ] · , · : · ; · . · · · ⊙

At each place the longest spelling that can begin there is read, so := is one token and not a colon and an equals sign, and .. is one token and not two periods. The other operators of the language are reserved words: and, or, xor, not, mod, shl, shr, odd, in, is, as, address and value.

Four formula glyphs are spellings of an ASCII operator, the same token as their twin, so nothing in a program can tell the two apart:

= compares for equality and # for inequality. Two glyphs are operators of their own over vectors and matrices: ·, the dot product, whose ASCII spelling is the function Dot, and ⊙, the elementwise product, whose ASCII spelling is the elemental function Hadamard.

The spellings another language uses for an operator Mica writes otherwise are refused by name, and the compiler reads on as if Mica’s operator had been written (refused-operator):

Any other character that begins no token is an unknown character, which the parser refuses where it stands, naming it: unexpected set of tokens, found unknown character '@' (2037), for @ where an operator belongs.

{
    OperatorGlyphs - the formula glyphs beside the ASCII spellings they stand for.

    A comparison glyph is the same token as its ASCII twin: '≠' is '#', '≤' is '<=' and '≥' is '>=', and '×' is '*'. The
    dot product '·' and the elementwise product '⊙' are operators of their own over vectors and matrices. 'True' and
    'False' are the two values of 'bool', names the language declares in every unit.
}
program OperatorGlyphs;

imp
    WriteLn : std;

var
    a, b : int64;
    u, v, w : vector[3] of float64;
    smaller : bool;

begin
    a := 3;
    b := 4;

    if (a ≠ b) = (a # b) then
        WriteLn("≠ is #");

    if (a ≤ b) = (a <= b) then
        WriteLn("≤ is <=");

    if (a ≥ b) = (a >= b) then
        WriteLn("≥ is >=");

    WriteLn("× is *: %lld %lld", a × b, a * b);
    u[0] := 1.0;
    u[1] := 2.0;
    u[2] := 3.0;
    v := u;
    w := u ⊙ v;
    WriteLn("u · v is %lf, and (u ⊙ v)[2] is %lf", u · v, w[2]);
    smaller := a < b;

    if smaller = True then
        WriteLn("3 < 4 is True");
end.

It prints:

≠ is #
≤ is <=
≥ is >=
× is *: 12 12
u · v is 14.000000, and (u ⊙ v)[2] is 9.000000
3 < 4 is True

Integer literals

An integer literal is a whole number in decimal, in hexadecimal behind 0x or 0X, or in binary behind 0b or 0B (integer-literal). A single underscore may stand between two of its digits, and changes nothing of its value. A number is written with the digits 0 to 9: a decimal digit of another script inside a number is refused at that digit.

A literal’s value is the whole number its digits write, and it has no type of its own: it keeps its exact value until the place it is used gives it one, as the type system’s constants state. Where nothing gives it a type, as an argument of WriteLn, it takes the first of int32, int64 and uint64 that holds it. A literal larger than every machine integer, in any of the three bases, reaches a wide integer type, such as uint128 of the integers unit; anywhere else it is refused with analyzer error 5368, here met by 0x1_0000_0000_0000_0005 at a uint64: the integer literal 18446744073709551621 is larger than any machine integer and 'uint64' does not admit it: give it a wide integer target 'int[N]' or 'uint[N]' whose width holds the value.

A binary literal with ? among its digits is a pattern, not a number: ? marks a bit that may be either (masked-integer-literal). It stands only as the label of a case arm over a bitset or an unsigned integer, and matches every value that agrees with its written bits; elsewhere it is refused with 5278, a masked bit pattern '1?' is a case label, not a value: its don't-care digits only gain meaning against a case selector, so it is legal only as a case label over a bitset or unsigned integer selector.

{
    IntegerLiterals - a whole number written in decimal, in hexadecimal behind '0x' and in binary behind '0b'.

    A single underscore may stand between two digits of any base and changes nothing of the value. A binary pattern with
    '?' for a bit that may be either is no number at all: it labels a case arm, which matches every value that agrees
    with the pattern's written bits.
}
program IntegerLiterals;

imp
    WriteLn : std;

const
    Million = 1_000_000;
    Mask = 0xFF_FF;
    Flags = 0b1010_0101;
    Largest : uint64 = 18446744073709551615;

var
    opcode : uint8;

begin
    WriteLn("%d %d %d", Million, Mask, Flags);
    WriteLn("%llu", Largest);
    opcode := 0b1011_0110;

    case opcode of
        0b10??_0110: WriteLn("a load")
    else
        WriteLn("another opcode");
    end;
end.

It prints:

1000000 65535 165
18446744073709551615
a load

Real literals

A real literal is a number with a fraction, a power of ten, or both: decimal digits followed by a point and decimal digits, by an exponent, or by the two (floating-point-literal). The exponent is e or E, an optional sign and decimal digits (exponent). A point needs a digit on each side: 5. is the number 5 and a period, which the parser reads as the start of a field name, and .5 is a period and the number 5. A single underscore may stand between two digits of each part.

A real literal’s value is exact until its use rounds it once, to float64, float32, float16 or bfloat16, as the type system’s constants state. A literal past the range of every floating-point type, such as 1e400, is refused with analyzer error 5653, the real number 1e400 is past the range of 'float64', whose largest finite value is 1.7976931348623157e+308.

{
    RealLiterals - a number with a fraction, a power of ten, or both.

    A real literal needs a digit on each side of its point, and its exponent is written behind 'e' or 'E' with an
    optional sign. Its value is exact until the type it meets rounds it, once, so the sum of the two constants below is
    computed as the processor computes it and prints the rounding every double-precision program meets.
}
program RealLiterals;

imp
    WriteLn : std;

const
    Half = 0.5;
    Avogadro = 6.022_140_76e23;
    Nano = 1e-9;
    Planck = 6.626_070_15E-34;
    Tenth = 0.1;
    Fifth = 0.2;

begin
    WriteLn("%lf", Half);
    WriteLn("%le %le %le", Avogadro, Nano, Planck);
    WriteLn("%.17lg", Tenth + Fifth);
end.

It prints:

0.500000
6.022141e+23 1.000000e-09 6.626070e-34
0.30000000000000004

Character literals

A character literal is exactly one character between single quotes (unicode-literal), a value of the type unicode: one Unicode scalar value. A character a source cannot hold for itself is written as an escape (escape-sequence):

\x names a character from U+0000 to U+00FF, not a byte: "\xE9" is the text é, which a UTF-8 build stores as its two bytes. A braced or a long escape names a Unicode scalar value: at most U+10FFFF, and none of the surrogates U+D800 to U+DFFF.

A letter written with a combining accent is two characters, so 'é' typed as e and U+0301 is refused with 1006; the precomposed é is one.

String literals

A string literal is text between double quotes (string-literal), a value of the type string. It may hold any character but the double quote and the backslash, which it writes as escapes, and it takes every escape a character literal takes. It may run over several lines, and every line break inside it is part of its text. How a string is stored, as UTF-32 or as UTF-8, is a fact of the build rather than of the literal, as the type system states.

{
    CharacterLiterals - one character between single quotes, and text between double quotes.

    A backslash names a character that cannot stand for itself: a control character, a quote, the backslash, or any
    character by its number in hexadecimal. A string may run over several lines, and every line break inside it is part
    of its text.
}
program CharacterLiterals;

imp
    WriteLn, Ord : std;

const
    Letter = 'ß';
    Smile = '\u{263A}';
    Ampersand = '\x26';
    Grin = '\U0001F600';
    Apostrophe = '\'';
    Columns = "name\tsize";
    Quoted = "say \"hello\" to C:\\mica";
    Verse = "first line
second line";

begin
    WriteLn("%lc %lld", Letter, Ord(Letter));
    WriteLn("%lc %lld", Smile, Ord(Smile));
    WriteLn("%lc %lld", Ampersand, Ord(Ampersand));
    WriteLn("%lc %lld", Grin, Ord(Grin));
    WriteLn("%lc %lld", Apostrophe, Ord(Apostrophe));
    WriteLn("%ls", Columns);
    WriteLn("%ls", Quoted);
    WriteLn("%ls", Verse);
end.

It prints, a tab between the two words of its sixth line:

ß 223
☺ 9786
& 38
😀 128512
' 39
name	size
say "hello" to C:\mica
first line
second line

Regex literals

A regex literal is a regular expression written as 0r and its pattern in double quotes (regex-literal). The pattern is read as it is matched: a backslash in it is a backslash of the pattern, and \" is a quote. The compiler checks the pattern while it compiles the program and makes it part of the program. The regex unit’s Matches, Find and Count take the literal in place of a pattern compiled at run time; its other routines take a pattern that Compile made, and refuse a literal with analyzer error 5398. The patterns the compiler admits are the ones it matches in time linear in the text, as the grammar’s regex-pattern and the rules below it state; lookaround and backreferences are not among them.

A pattern that never closes is refused with 1028, and a pattern the rules do not admit with the code that names its fault: 1031 for an escape, 1032 for a class, 1033 for a parenthesis, 1034 for a quantifier with nothing to repeat, 1042 for a property, and 1043 for a lookaround, each naming the position in the pattern.

{
    RegexLiteral - a regular expression written as a literal, checked and compiled while the program compiles.

    A pattern behind '0r' is read as it is matched, so a backslash in it belongs to the pattern. The compiler refuses a
    pattern it could not match in time linear in the text, and makes every other one part of the program, so the regex
    unit's routines take the literal wherever they take a pattern compiled at run time.
}
program RegexLiteral;

imp
    WriteLn : std;
    Count, Matches : regex;

const
    Sentence = "Mica reads 3 words and 1 number";

begin
    WriteLn("%lld words", Count(0r"\p{L}+", Sentence));
    WriteLn("%lld numbers", Count(0r"[0-9]+", Sentence));

    if Matches(0r"^Mica", Sentence) then
        WriteLn("the sentence starts with Mica");
end.

It prints:

5 words
2 numbers
the sentence starts with Mica

Booleans and nil

True and False are the two values of bool, names the language declares in every unit rather than literals of their own, and a comparison answers one of them. nil is a reserved word: the pointer that points at nothing.

Where chapter 2 begins

The tokens of this chapter form units: a program, a library or an object, with its sections, its routines and its body, and the imports that bring names from other units into it. Chapter 2 describes them.