Package org.antlr.v4.runtime
Class CommonTokenFactory
java.lang.Object
org.antlr.v4.runtime.CommonTokenFactory
- All Implemented Interfaces:
TokenFactory<CommonToken>
This default implementation of
TokenFactory
creates
CommonToken
objects.-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected final boolean
Indicates whetherCommonToken.setText(java.lang.String)
should be called after constructing tokens to explicitly set the text.static final TokenFactory
<CommonToken> The defaultCommonTokenFactory
instance. -
Constructor Summary
ConstructorsConstructorDescriptionCommonTokenFactory
(boolean copyText) Constructs aCommonTokenFactory
with the specified value forcopyText
. -
Method Summary
Modifier and TypeMethodDescriptionGenerically usefulcreate
(Pair<TokenSource, CharStream> source, int type, String text, int channel, int start, int stop, int line, int charPositionInLine) This is the method used to create tokens in the lexer and in the error handling strategy.
-
Field Details
-
DEFAULT
The defaultCommonTokenFactory
instance.This token factory does not explicitly copy token text when constructing tokens.
-
copyText
protected final boolean copyTextIndicates whetherCommonToken.setText(java.lang.String)
should be called after constructing tokens to explicitly set the text. This is useful for cases where the input stream might not be able to provide arbitrary substrings of text from the input after the lexer creates a token (e.g. the implementation ofCharStream.getText(org.antlr.v4.runtime.misc.Interval)
inUnbufferedCharStream
throws anUnsupportedOperationException
). Explicitly setting the token text allowsToken.getText()
to be called at any time regardless of the input stream implementation.The default value is
false
to avoid the performance and memory overhead of copying text for every token unless explicitly requested.
-
-
Constructor Details
-
CommonTokenFactory
public CommonTokenFactory(boolean copyText) Constructs aCommonTokenFactory
with the specified value forcopyText
.When
copyText
isfalse
, theDEFAULT
instance should be used instead of constructing a new instance.- Parameters:
copyText
- The value forcopyText
.
-
CommonTokenFactory
public CommonTokenFactory()Constructs aCommonTokenFactory
withcopyText
set tofalse
.The
DEFAULT
instance should be used instead of calling this directly.
-
-
Method Details
-
create
public CommonToken create(Pair<TokenSource, CharStream> source, int type, String text, int channel, int start, int stop, int line, int charPositionInLine) Description copied from interface:TokenFactory
This is the method used to create tokens in the lexer and in the error handling strategy. If text!=null, than the start and stop positions are wiped to -1 in the text override is set in the CommonToken.- Specified by:
create
in interfaceTokenFactory<CommonToken>
-
create
Description copied from interface:TokenFactory
Generically useful- Specified by:
create
in interfaceTokenFactory<CommonToken>
-