• Lexer





  • Parameters

    • lexerDefinition: TokenType[] | IMultiModeLexerDefinition

      Structure composed of Tokens Types this lexer will identify.

      In the simple case the structure is an array of TokenTypes. In the case of IMultiModeLexerDefinition the structure is an object with two properties:

      1. a "modes" property where each value is an array of TokenTypes.
      2. a "defaultMode" property specifying the initial lexer mode.

      for example:

      modes : {
      modeX : [Token1, Token2],
      modeY : [Token3, Token4]

      defaultMode : "modeY"

      A lexer with MultiModesDefinition is simply multiple Lexers where only one Lexer(mode) can be active at the same time. This is useful for lexing languages where there are different lexing rules depending on context.

      The current lexing mode is selected via a "mode stack". The last (peek) value in the stack will be the current mode of the lexer. Defining entering and exiting lexer modes is done using the "push_mode" and "pop_mode" properties of the ITokenConfig config properties.

      • The Lexer will match the first pattern that matches, Therefor the order of Token Types is significant. For example when one pattern may match a prefix of another pattern.

        Note that there are situations in which we may wish to order the longer pattern after the shorter one. For example: keywords vs Identifiers.

    • Optional config: ILexerConfig

    Returns Lexer


lexerDefinitionErrors: ILexerDefinitionError[]
NA: RegExp

A Constant to mark "abstract" TokenTypes that are used purely as token categories. See: categories

SKIPPED: string


  • Will lex(Tokenize) a string. Note that this can be called repeatedly on different strings as this method does not modify the state of the Lexer.


    • text: string

      The string to lex

    • Optional initialMode: string

      The initial Lexer Mode to start with, by default this will be the first mode in the lexer's definition. If the lexer has no explicit modes it will be the implicit single 'default_mode' mode.

    Returns ILexingResult

Generated using TypeDoc