Package org.antlr.v4.runtime
Class Lexer
- All Implemented Interfaces:
TokenSource
- Direct Known Subclasses:
LexerInterpreter
,XPathLexer
A lexer is recognizer that draws input symbols from a character stream.
lexer grammars result in a subclass of this object. A Lexer object
uses simplified match() and error recovery mechanisms in the interest
of speed.
-
Field Summary
FieldsModifier and TypeFieldDescriptionint
The channel number for the current tokenprotected TokenFactory
<?> How to create token objectsboolean
Once we see EOF on char stream, next token will be EOF.int
final IntegerStack
You can set the text for the current token to override what is in the input char buffer.The goal of all lexer rules/methods is to create a token object.protected Pair
<TokenSource, CharStream> int
What character index in the stream did the current token start at? Needed, for example, to get the text for current token.int
The character position of first character within the lineint
The line on which the first character of the token residesint
The token type for the current tokenstatic final int
static final int
static final int
static final int
static final int
static final int
static final int
Fields inherited from class org.antlr.v4.runtime.Recognizer
_interp, EOF
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionemit()
The standard method called to automatically emit a token at the outermost lexical rule.void
By default does not support multiple emits per nextToken invocation for efficiency reasons.emitEOF()
Return a list of all Token objects in input char stream.int
String[]
getCharErrorDisplay
(int c) int
What is the index of the current character of lookahead?int
Get the index into the current line for the current position in the input stream.getErrorDisplay
(int c) Get theCharStream
from which this token source is currently providing tokens.int
getLine()
Get the line number for the current position in the input stream.String[]
Gets the name of the underlying input source.getText()
Return the text matched so far for the current token or any text override.getToken()
Override if emitting multiple tokens.TokenFactory
<? extends Token> Gets theTokenFactory
this token source is currently using for creatingToken
objects from the input.String[]
Deprecated.int
getType()
void
mode
(int m) void
more()
Return a token from this source; i.e., match a token on the char stream.void
int
popMode()
void
pushMode
(int m) void
void
Lexers can normally match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out.void
reset()
void
setChannel
(int channel) void
setCharPositionInLine
(int charPositionInLine) void
setInputStream
(IntStream input) Set the char stream and reset the lexervoid
setLine
(int line) void
Set the complete text of this token; it wipes any previous changes to the text.void
void
setTokenFactory
(TokenFactory<?> factory) Set theTokenFactory
this token source should use for creatingToken
objects from the input.void
setType
(int ttype) void
skip()
Instruct the lexer to skip creating a token for current lexer rule and look for another token.Methods inherited from class org.antlr.v4.runtime.Recognizer
action, addErrorListener, getATN, getErrorHeader, getErrorListenerDispatch, getErrorListeners, getGrammarFileName, getInterpreter, getParseInfo, getRuleIndexMap, getRuleNames, getSerializedATN, getState, getTokenErrorDisplay, getTokenType, getTokenTypeMap, getVocabulary, precpred, removeErrorListener, removeErrorListeners, sempred, setInterpreter, setState
-
Field Details
-
DEFAULT_MODE
public static final int DEFAULT_MODE- See Also:
-
MORE
public static final int MORE- See Also:
-
SKIP
public static final int SKIP- See Also:
-
DEFAULT_TOKEN_CHANNEL
public static final int DEFAULT_TOKEN_CHANNEL- See Also:
-
HIDDEN
public static final int HIDDEN- See Also:
-
MIN_CHAR_VALUE
public static final int MIN_CHAR_VALUE- See Also:
-
MAX_CHAR_VALUE
public static final int MAX_CHAR_VALUE- See Also:
-
_input
-
_tokenFactorySourcePair
-
_factory
How to create token objects -
_token
The goal of all lexer rules/methods is to create a token object. This is an instance variable as multiple rules may collaborate to create a single token. nextToken will return this object after matching lexer rule(s). If you subclass to allow multiple token emissions, then set this to the last token to be matched or something nonnull so that the auto token emit mechanism will not emit another token. -
_tokenStartCharIndex
public int _tokenStartCharIndexWhat character index in the stream did the current token start at? Needed, for example, to get the text for current token. Set at the start of nextToken. -
_tokenStartLine
public int _tokenStartLineThe line on which the first character of the token resides -
_tokenStartCharPositionInLine
public int _tokenStartCharPositionInLineThe character position of first character within the line -
_hitEOF
public boolean _hitEOFOnce we see EOF on char stream, next token will be EOF. If you have DONE : EOF ; then you see DONE EOF. -
_channel
public int _channelThe channel number for the current token -
_type
public int _typeThe token type for the current token -
_modeStack
-
_mode
public int _mode -
_text
You can set the text for the current token to override what is in the input char buffer. Use setText() or can set this instance var.
-
-
Constructor Details
-
Lexer
public Lexer() -
Lexer
-
-
Method Details
-
reset
public void reset() -
nextToken
Return a token from this source; i.e., match a token on the char stream.- Specified by:
nextToken
in interfaceTokenSource
-
skip
public void skip()Instruct the lexer to skip creating a token for current lexer rule and look for another token. nextToken() knows to keep looking when a lexer rule finishes with token set to SKIP_TOKEN. Recall that if token==null at end of any token rule, it creates one for you and emits it. -
more
public void more() -
mode
public void mode(int m) -
pushMode
public void pushMode(int m) -
popMode
public int popMode() -
setTokenFactory
Description copied from interface:TokenSource
Set theTokenFactory
this token source should use for creatingToken
objects from the input.- Specified by:
setTokenFactory
in interfaceTokenSource
- Specified by:
setTokenFactory
in classRecognizer<Integer,
LexerATNSimulator> - Parameters:
factory
- TheTokenFactory
to use for creating tokens.
-
getTokenFactory
Description copied from interface:TokenSource
Gets theTokenFactory
this token source is currently using for creatingToken
objects from the input.- Specified by:
getTokenFactory
in interfaceTokenSource
- Specified by:
getTokenFactory
in classRecognizer<Integer,
LexerATNSimulator> - Returns:
- The
TokenFactory
currently used by this token source.
-
setInputStream
Set the char stream and reset the lexer- Specified by:
setInputStream
in classRecognizer<Integer,
LexerATNSimulator>
-
getSourceName
Description copied from interface:TokenSource
Gets the name of the underlying input source. This method returns a non-null, non-empty string. If such a name is not known, this method returnsIntStream.UNKNOWN_SOURCE_NAME
.- Specified by:
getSourceName
in interfaceTokenSource
-
getInputStream
Description copied from interface:TokenSource
Get theCharStream
from which this token source is currently providing tokens.- Specified by:
getInputStream
in interfaceTokenSource
- Specified by:
getInputStream
in classRecognizer<Integer,
LexerATNSimulator> - Returns:
- The
CharStream
associated with the current position in the input, ornull
if no input stream is available for the token source.
-
emit
By default does not support multiple emits per nextToken invocation for efficiency reasons. Subclass and override this method, nextToken, and getToken (to push tokens into a list and pull from that list rather than a single variable as this implementation does). -
emit
The standard method called to automatically emit a token at the outermost lexical rule. The token object should point into the char buffer start..stop. If there is a text override in 'text', use that to set the token's text. Override this method to emit custom Token objects or provide a new factory. -
emitEOF
-
getLine
public int getLine()Description copied from interface:TokenSource
Get the line number for the current position in the input stream. The first line in the input is line 1.- Specified by:
getLine
in interfaceTokenSource
- Returns:
- The line number for the current position in the input stream, or 0 if the current token source does not track line numbers.
-
getCharPositionInLine
public int getCharPositionInLine()Description copied from interface:TokenSource
Get the index into the current line for the current position in the input stream. The first character on a line has position 0.- Specified by:
getCharPositionInLine
in interfaceTokenSource
- Returns:
- The line number for the current position in the input stream, or -1 if the current token source does not track character positions.
-
setLine
public void setLine(int line) -
setCharPositionInLine
public void setCharPositionInLine(int charPositionInLine) -
getCharIndex
public int getCharIndex()What is the index of the current character of lookahead? -
getText
Return the text matched so far for the current token or any text override. -
setText
Set the complete text of this token; it wipes any previous changes to the text. -
getToken
Override if emitting multiple tokens. -
setToken
-
setType
public void setType(int ttype) -
getType
public int getType() -
setChannel
public void setChannel(int channel) -
getChannel
public int getChannel() -
getChannelNames
-
getModeNames
-
getTokenNames
Deprecated.Used to print out token names like ID during debugging and error reporting. The generated parsers implement a method that overrides this to point to their String[] tokenNames.- Specified by:
getTokenNames
in classRecognizer<Integer,
LexerATNSimulator>
-
getAllTokens
Return a list of all Token objects in input char stream. Forces load of all tokens. Does not include EOF token. -
recover
-
notifyListeners
-
getErrorDisplay
-
getErrorDisplay
-
getCharErrorDisplay
-
recover
Lexers can normally match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out. You can instead use the rule invocation stack to do sophisticated error recovery if you are in a fragment rule.
-