Class UnbufferedTokenStream<T extends Token>
- All Implemented Interfaces:
IntStream
,TokenStream
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected int
Absolute token index.protected Token
This is theLT(-1)
token for the current position.protected Token
protected int
The number of tokens currently intokens
.protected int
protected int
0..n-1 index intotokens
of next token.protected Token[]
A moving window buffer of the data being scanned.protected TokenSource
Fields inherited from interface org.antlr.v4.runtime.IntStream
EOF, UNKNOWN_SOURCE_NAME
-
Constructor Summary
ConstructorsConstructorDescriptionUnbufferedTokenStream
(TokenSource tokenSource) UnbufferedTokenStream
(TokenSource tokenSource, int bufferSize) -
Method Summary
Modifier and TypeMethodDescriptionprotected void
void
consume()
Consumes the current symbol in the stream.protected int
fill
(int n) Addn
elements to the buffer.get
(int i) Gets theToken
at the specifiedindex
in the stream.protected final int
Gets the name of the underlying symbol source.getText()
Return the text of all tokens in the stream.Return the text of all tokens within the specifiedinterval
.getText
(RuleContext ctx) Return the text of all tokens in the source interval of the specified context.Return the text of all tokens in this stream betweenstart
andstop
(inclusive).Gets the underlyingTokenSource
which provides tokens for this stream.int
index()
Return the index into the stream of the input symbol referred to byLA(1)
.int
LA
(int i) Gets the value of the symbol at offseti
from the current position.LT
(int i) int
mark()
Return a marker that we can release later.void
release
(int marker) This method releases a marked range created by a call tomark()
.void
seek
(int index) Set the input cursor to the position indicated byindex
.int
size()
Returns the total number of symbols in the stream, including a single EOF symbol.protected void
sync
(int want) Make sure we have 'need' elements from current positionp
.
-
Field Details
-
tokenSource
-
tokens
A moving window buffer of the data being scanned. While there's a marker, we keep adding to buffer. Otherwise,consume()
resets so we start filling at index 0 again. -
n
protected int nThe number of tokens currently intokens
.This is not the buffer capacity, that's
tokens.length
. -
p
protected int p0..n-1 index intotokens
of next token.The
LT(1)
token istokens[p]
. Ifp == n
, we are out of buffered tokens. -
numMarkers
protected int numMarkers -
lastToken
This is theLT(-1)
token for the current position. -
lastTokenBufferStart
WhennumMarkers > 0
, this is theLT(-1)
token for the first token intokens
. Otherwise, this isnull
. -
currentTokenIndex
protected int currentTokenIndexAbsolute token index. It's the index of the token about to be read viaLT(1)
. Goes from 0 to the number of tokens in the entire stream, although the stream size is unknown before the end is reached.This value is used to set the token indexes if the stream provides tokens that implement
WritableToken
.
-
-
Constructor Details
-
UnbufferedTokenStream
-
UnbufferedTokenStream
-
-
Method Details
-
get
Description copied from interface:TokenStream
Gets theToken
at the specifiedindex
in the stream. When the preconditions of this method are met, the return value is non-null.The preconditions for this method are the same as the preconditions of
IntStream.seek(int)
. If the behavior ofseek(index)
is unspecified for the current state and givenindex
, then the behavior of this method is also unspecified.The symbol referred to by
index
differs fromseek()
only in the case of filtering streams whereindex
lies before the end of the stream. Unlikeseek()
, this method does not adjustindex
to point to a non-ignored symbol.- Specified by:
get
in interfaceTokenStream
-
LT
Description copied from interface:TokenStream
Get theToken
instance associated with the value returned byLA(k)
. This method has the same pre- and post-conditions asIntStream.LA(int)
. In addition, when the preconditions of this method are met, the return value is non-null and the value ofLT(k).getType()==LA(k)
.- Specified by:
LT
in interfaceTokenStream
- See Also:
-
LA
public int LA(int i) Description copied from interface:IntStream
Gets the value of the symbol at offseti
from the current position. Wheni==1
, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). Wheni==-1
, this method returns the value of the previously read symbol in the stream. It is not valid to call this method withi==0
, but the specific behavior is unspecified because this method is frequently called from performance-critical code.This method is guaranteed to succeed if any of the following are true:
i>0
i==-1
andindex()
returns a value greater than the value ofindex()
after the stream was constructed andLA(1)
was called in that order. Specifying the currentindex()
relative to the index after the stream was created allows for filtering implementations that do not return every symbol from the underlying source. Specifying the call toLA(1)
allows for lazily initialized streams.LA(i)
refers to a symbol consumed within a marked region that has not yet been released.
If
i
represents a position at or beyond the end of the stream, this method returnsIntStream.EOF
.The return value is unspecified if
i<0
and fewer than-i
calls toconsume()
have occurred from the beginning of the stream before calling this method. -
getTokenSource
Description copied from interface:TokenStream
Gets the underlyingTokenSource
which provides tokens for this stream.- Specified by:
getTokenSource
in interfaceTokenStream
-
getText
Description copied from interface:TokenStream
Return the text of all tokens in the stream. This method behaves like the following code, including potential exceptions from the calls toIntStream.size()
andTokenStream.getText(Interval)
, but may be optimized by the specific implementation.TokenStream stream = ...; String text = stream.getText(new Interval(0, stream.size()));
- Specified by:
getText
in interfaceTokenStream
- Returns:
- The text of all tokens in the stream.
-
getText
Description copied from interface:TokenStream
Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call toTokenStream.getText(Interval)
, but may be optimized by the specific implementation.If
ctx.getSourceInterval()
does not return a valid interval of tokens provided by this stream, the behavior is unspecified.TokenStream stream = ...; String text = stream.getText(ctx.getSourceInterval());
- Specified by:
getText
in interfaceTokenStream
- Parameters:
ctx
- The context providing the source interval of tokens to get text for.- Returns:
- The text of all tokens within the source interval of
ctx
.
-
getText
Description copied from interface:TokenStream
Return the text of all tokens in this stream betweenstart
andstop
(inclusive).If the specified
start
orstop
token was not provided by this stream, or if thestop
occurred before thestart
token, the behavior is unspecified.For streams which ensure that the
Token.getTokenIndex()
method is accurate for all of its provided tokens, this method behaves like the following code. Other streams may implement this method in other ways provided the behavior is consistent with this at a high level.TokenStream stream = ...; String text = ""; for (int i = start.getTokenIndex(); i <= stop.getTokenIndex(); i++) { text += stream.get(i).getText(); }
- Specified by:
getText
in interfaceTokenStream
- Parameters:
start
- The first token in the interval to get text for.stop
- The last token in the interval to get text for (inclusive).- Returns:
- The text of all tokens lying between the specified
start
andstop
tokens.
-
consume
public void consume()Description copied from interface:IntStream
Consumes the current symbol in the stream. This method has the following effects:- Forward movement: The value of
index()
before calling this method is less than the value ofindex()
after calling this method. - Ordered lookahead: The value of
LA(1)
before calling this method becomes the value ofLA(-1)
after calling this method.
index()
is incremented by exactly 1, as that would preclude the ability to implement filtering streams (e.g.CommonTokenStream
which distinguishes between "on-channel" and "off-channel" tokens). - Forward movement: The value of
-
sync
protected void sync(int want) Make sure we have 'need' elements from current positionp
. Last validp
index istokens.length-1
.p+need-1
is the tokens index 'need' elements ahead. If we need 1 element,(p+1-1)==p
must be less thantokens.length
. -
fill
protected int fill(int n) Addn
elements to the buffer. Returns the number of tokens actually added to the buffer. If the return value is less thann
, then EOF was reached beforen
tokens could be added. -
add
-
mark
public int mark()Return a marker that we can release later.The specific marker value used for this class allows for some level of protection against misuse where
seek()
is called on a mark orrelease()
is called in the wrong order. -
release
public void release(int marker) Description copied from interface:IntStream
This method releases a marked range created by a call tomark()
. Calls torelease()
must appear in the reverse order of the corresponding calls tomark()
. If a mark is released twice, or if marks are not released in reverse order of the corresponding calls tomark()
, the behavior is unspecified.For more information and an example, see
IntStream.mark()
. -
index
public int index()Description copied from interface:IntStream
Return the index into the stream of the input symbol referred to byLA(1)
.The behavior of this method is unspecified if no call to an
initializing method
has occurred after this stream was constructed. -
seek
public void seek(int index) Description copied from interface:IntStream
Set the input cursor to the position indicated byindex
. If the specified index lies past the end of the stream, the operation behaves as thoughindex
was the index of the EOF symbol. After this method returns without throwing an exception, then at least one of the following will be true.index()
will return the index of the first symbol appearing at or after the specifiedindex
. Specifically, implementations which filter their sources should automatically adjustindex
forward the minimum amount required for the operation to target a non-ignored symbol.LA(1)
returnsIntStream.EOF
index
lies within a marked region. For more information on marked regions, seeIntStream.mark()
. The behavior of this method is unspecified if no call to aninitializing method
has occurred after this stream was constructed. -
size
public int size()Description copied from interface:IntStream
Returns the total number of symbols in the stream, including a single EOF symbol. -
getSourceName
Description copied from interface:IntStream
Gets the name of the underlying symbol source. This method returns a non-null, non-empty string. If such a name is not known, this method returnsIntStream.UNKNOWN_SOURCE_NAME
.- Specified by:
getSourceName
in interfaceIntStream
-
getText
Description copied from interface:TokenStream
Return the text of all tokens within the specifiedinterval
. This method behaves like the following code (including potential exceptions for violating preconditions ofTokenStream.get(int)
, but may be optimized by the specific implementation.TokenStream stream = ...; String text = ""; for (int i = interval.a; i <= interval.b; i++) { text += stream.get(i).getText(); }
- Specified by:
getText
in interfaceTokenStream
- Parameters:
interval
- The interval of tokens within this stream to get text for.- Returns:
- The text of all tokens within the specified interval in this stream.
-
getBufferStartIndex
protected final int getBufferStartIndex()
-