[This is preliminary documentation and is subject to change.]
The Token type exposes the following members.Constructors
Name | Description | |
---|---|---|
![]() | Token(ILexer, SourcePosition, SourcePosition, String) | Initializes a new instance of the Token class |
![]() | Token(ILexer, SourcePosition, SourcePosition, String, Object) | Initializes a new instance of the Token class |
Methods
Name | Description | |
---|---|---|
![]() | ![]() | Determines whether the specified object is equal to the current object. (Inherited from ![]() |
![]() | ![]() | Allows an object to try to free resources and perform other cleanup operations before it is reclaimed by garbage collection. (Inherited from ![]() |
![]() | ![]() | Serves as the default hash function. (Inherited from ![]() |
![]() | ![]() | Gets the (Inherited from ![]() ![]() |
![]() | ![]() | Creates a shallow copy of the current (Inherited from ![]() ![]() |
![]() | ToString | (Overrides ![]() |
Extension Methods
Fields
Name | Description | |
---|---|---|
![]() | EndPosition | |
![]() | Lexer | |
![]() | StartPosition | |
![]() | Text | |
![]() | Value |
Properties
Name | Description | |
---|---|---|
![]() | IsBOF |
Indicates whether this token indicates an BEGINNING-OF-FILE condition
|
![]() | IsComment |
Indicates whether this token represents a comment
|
![]() | IsDirective |
Indicates whether this token represents a directive
|
![]() | IsEOF |
Indicates whether this token indicates an END-OF-FILE condition
|
![]() | IsIdentifier |
Indicates whether this token represents an identifier
|
![]() | IsKeyword |
Indicates whether this token represents a language keyword
|
![]() | IsLiteral |
Indicates whether this token represents a literal
|
![]() | IsNonLanguage |
This property is needed due to the fact that language lexers may be used to analyze special supersets of regular
language grammars i.e. for pattern matches, template parser etc. Code compilers may elect to throw errors when this property is true.
Returns true for tokens that are not part of the valid language grammar, however they exist for other reasons,
for example - for pattern capture match analysis, or for template processing
|
![]() | IsNumericLiteral |
Indicates whether this token represents a literal, which is a numeric literal (i.e. int, double)
This flag is useful for pattern searches that look for particular constant values in numbers (i.e. year = 2000)
|
![]() | IsOperator |
Indicates whether this token represents an operator
|
![]() | IsPrimary |
Returns true for tokens that are primary part of the language, not control, metadata, directive, comment etc...
For example BOF,EOF markers are not primary part of the language, compiler directives, comments are not either.
This property is useful for pattern searches, when comments and other non-primary tokens need to be quickly skipped
|
![]() | IsSymbol |
Indicates whether this token represents a symbol
|
![]() | IsTextualLiteral |
Indicates whether this token represents a literal, which is a string or a character sequence.
This flag is useful for pattern searches that examine comments and strings for sub-patterns
|
![]() | Kind |
Provides language-agnostic classification for token type
|
![]() | Language |
Returns language that this token is a part of
|
![]() | OrdinalType |
Returns token type as a grammar-agnostic ordinal
|
See Also