A Constant to mark "abstract" TokenTypes that are used purely as token categories. See: categories
Will lex(Tokenize) a string. Note that this can be called repeatedly on different strings as this method does not modify the state of the Lexer.
The string to lex
The initial Lexer Mode to start with, by default this will be the first mode in the lexer's definition. If the lexer has no explicit modes it will be the implicit single 'default_mode' mode.
Generated using TypeDoc
Structure composed of Tokens Types this lexer will identify.
In the simple case the structure is an array of TokenTypes. In the case of IMultiModeLexerDefinition the structure is an object with two properties:
A lexer with MultiModesDefinition is simply multiple Lexers where only one Lexer(mode) can be active at the same time. This is useful for lexing languages where there are different lexing rules depending on context.
The current lexing mode is selected via a "mode stack". The last (peek) value in the stack will be the current mode of the lexer. Defining entering and exiting lexer modes is done using the "push_mode" and "pop_mode" properties of the ITokenConfig config properties.
The Lexer will match the first pattern that matches, Therefor the order of Token Types is significant. For example when one pattern may match a prefix of another pattern.
Note that there are situations in which we may wish to order the longer pattern after the shorter one. For example: keywords vs Identifiers.