Interface ILexerConfig

An optional flag indicating that lexer definition errors should not automatically cause an error to be raised. This can be useful when wishing to indicate lexer errors in another manner than simply throwing an error (for example in an online playground).

When true this flag will cause the Lexer to throw an Error When it is unable to perform all of its performance optimizations.

In addition error messages will be printed to the console with details how to resolve the optimizations issues.

Use this flag to guarantee higher lexer performance. The optimizations can boost the lexer's performance anywhere from 30% to 100%+ depending on the number of TokenTypes used.

A custom error message provider. Can be used to override the default error messages. For example:

• Translating the error messages to a different languages.
• Changing the formatting.

Characters or CharCodes that represent line terminators for this lexer. This always needs to be provided when using a custom lineTerminatorsPattern. In the future this duplication may be removed or reduced.

A regExp defining custom line terminators. This will be used to calculate the line and column information.

Note that the regExp should use the global flag, for example: /\n/g

The default is: /\n|\r\n?/g

But some grammars have a different definition, for example in ECMAScript: https://www.ecma-international.org/ecma-262/8.0/index.html#sec-line-terminators U+2028 and U+2029 are also treated as line terminators.

In that case we would use /\n|\r|\u2028|\u2029/g

Note that it is also possible to supply an optimized RegExp like implementation as only a subset of the RegExp APIs is needed, ILineTerminatorsTester for details.

keep in mind that for the default pattern: /\n|\r\n?/g an optimized implementation is already built-in. This means the optimization is only relevant for lexers overriding the default pattern.

"full" location information means all six combinations of /(end|start)(Line|Column|Offset)/ properties. "onlyStart" means that only startLine, startColumn and startOffset will be tracked "onlyOffset" means that only the startOffset will be tracked.

The less position tracking the faster the Lexer will be and the less memory used. However the difference is not large (~10% On V8), thus reduced location tracking options should only be used in edge cases where every last ounce of performance is needed.

Should the lexer halt on the first error, or continue attempting to tokenize by dropping characters until a match is found or the end of input is reached.

Setting recoveryEnabled to false is useful when you want to halt quickly on faulty inputs, particularly when dealing with large faulty inputs.

By default, recoveryEnabled is true

Can be used to disable lexer optimizations If there is a suspicion they are causing incorrect behavior. Note that this would have negative performance implications.

This flag will avoid running the Lexer validations during Lexer initialization.

This can substantially improve the Lexer's initialization (constructor) time.

See

ILexerConfig.traceInitPerf to measure the Lexer validations cost for your Lexer.

Note that the Lexer validations are extremely useful during development time, e.g: Detecting empty/invalid regExp Patterns. So they should not be skipped during development flows.

• For example: via a conditional that checks an env variable.

Enabling this flag will print performance tracing logs during lexer Initialization (constructor invocation), this is useful to narrow down the cause of the initialization performance problem.

You can also pass a numerical value which affects the verbosity of the traces, this number is the maximum nesting level of the traces, e.g: 0: Traces disabled === 'false' 1: Top Level traces only. 2: One level of nested inner traces. ...

Note that passing the boolean true is identical to passing the numerical value infinity