Class Lexer

constructor

• new Lexer(lexerDefinition, config?): Lexer

• Parameters

  • lexerDefinition: TokenType[] | IMultiModeLexerDefinition

    Structure composed of Tokens Types this lexer will identify.

    In the simple case the structure is an array of TokenTypes. In the case of IMultiModeLexerDefinition the structure is an object with two properties:

    1. a "modes" property where each value is an array of TokenTypes.
    2. a "defaultMode" property specifying the initial lexer mode.

    for example:

      {
          modes : {
            modeX : [Token1, Token2],
            modeY : [Token3, Token4]
          },
    
          defaultMode : "modeY"
      }
    Copy

    A lexer with MultiModesDefinition is simply multiple Lexers where only one Lexer(mode) can be active at the same time. This is useful for lexing languages where there are different lexing rules depending on context.

    The current lexing mode is selected via a "mode stack". The last (peek) value in the stack will be the current mode of the lexer. Defining entering and exiting lexer modes is done using the "push_mode" and "pop_mode" properties of the ITokenConfig config properties.

    • The Lexer will match the first pattern that matches, Therefor the order of Token Types is significant. For example when one pattern may match a prefix of another pattern.

      Note that there are situations in which we may wish to order the longer pattern after the shorter one. For example: keywords vs Identifiers.

  • Optional config: ILexerConfig

  Returns Lexer

lexerDefinitionErrors

Static NA

Static SKIPPED

tokenize

• tokenize(text, initialMode?): ILexingResult

• Will lex(Tokenize) a string. Note that this can be called repeatedly on different strings as this method does not modify the state of the Lexer.

  Parameters

  • text: string

    The string to lex

  • Optional initialMode: string

    The initial Lexer Mode to start with, by default this will be the first mode in the lexer's definition. If the lexer has no explicit modes it will be the implicit single 'default_mode' mode.

  Returns ILexingResult