Options
All
  • Public
  • Public/Protected
  • All
Menu

Hierarchy

  • Lexer

Index

Constructors

constructor

  • Parameters

    • lexerDefinition: TokenType[] | IMultiModeLexerDefinition
      • Structure composed of Tokens Types this lexer will identify.

        In the simple case the structure is an array of TokenTypes. In the case of IMultiModeLexerDefinition the structure is an object with two properties:

        1. a "modes" property where each value is an array of TokenTypes.
        2. a "defaultMode" property specifying the initial lexer mode.

        for example:

          {
        modes : {
        modeX : [Token1, Token2],
        modeY : [Token3, Token4]
        },

        defaultMode : "modeY"
        }

        A lexer with MultiModesDefinition is simply multiple Lexers where only one Lexer(mode) can be active at the same time. This is useful for lexing languages where there are different lexing rules depending on context.

        The current lexing mode is selected via a "mode stack". The last (peek) value in the stack will be the current mode of the lexer. Defining entering and exiting lexer modes is done using the "push_mode" and "pop_mode" properties of the {@link createToken.config} parameter.

        • The Lexer will match the first pattern that matches, Therefor the order of Token Types is significant. For example when one pattern may match a prefix of another pattern.

          Note that there are situations in which we may wish to order the longer pattern after the shorter one. For example: keywords vs Identifiers.

    • Optional config: ILexerConfig

    Returns Lexer

Properties

lexerDefinitionErrors

lexerDefinitionErrors: ILexerDefinitionError[]

Static NA

NA: RegExp

A Constant to mark "abstract" TokenTypes that are used purely as token categories. See: {@link createToken.categories}

Static SKIPPED

SKIPPED: string

Methods

tokenize

  • Will lex(Tokenize) a string. Note that this can be called repeatedly on different strings as this method does not modify the state of the Lexer.

    Parameters

    • text: string

      The string to lex

    • Optional initialMode: string

    Returns ILexingResult

Generated using TypeDoc