IToken | Chevrotain

* Things to note:

"do" {

   startColumn : 1, endColumn: 2,
   startOffset: x, endOffset: x +1} --> the range is inclusive to exclusive 1...2 (2 chars long).

"\n" {startLine : 1, endLine: 1} --> a lineTerminator as the last character does not effect the Token's line numbering.

"'hello\tworld\uBBBB'" {image: "'hello\tworld\uBBBB'"} --> a Token's image is the "literal" text

                                                       (unicode escaping is untouched).

Index

endColumn: number

Column of the last character of the Token.

endLine: number

Line of the last character of the Token.

endOffset: number

Offset of the last character of the Token.

image: string

The textual representation of the Token as it appeared in the text.

isInsertedInRecovery: boolean

this marks if a Token does not really exist and has been inserted "artificially" during parsing in rule error recovery.

startColumn: number

Column of the first character of the Token.

startLine: number

Line of the first character of the Token.

startOffset: number

Offset of the first character of the Token.

tokenClassName: number

A human readable name of the Token Class, This property will only be avilaible if the Lexer has run in @see {ILexerConfig} debug flag.

This property should not be used in productive flows as it will not always exist!

tokenType: number

An number index representing the type of the Token use to get the Token Type from a token "instance"