Class BufferedTokenStream

java.lang.Object
org.antlr.v4.runtime.BufferedTokenStream
All Implemented Interfaces:
IntStream, TokenStream
Direct Known Subclasses:
CommonTokenStream

public class BufferedTokenStream extends Object implements TokenStream
This implementation of TokenStream loads tokens from a TokenSource on-demand, and places the tokens in a buffer to provide access to any previous token by index.

This token stream ignores the value of Token.getChannel(). If your parser requires the token stream filter tokens to only those on a particular channel, such as Token.DEFAULT_CHANNEL or Token.HIDDEN_CHANNEL, use a filtering token stream such a CommonTokenStream.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected boolean
    Indicates whether the Token.EOF token has been fetched from tokenSource and added to tokens.
    protected int
    The index into tokens of the current token (next token to consume()).
    protected List<Token>
    A collection of all tokens fetched from the token source.
    protected TokenSource
    The TokenSource from which tokens for this stream are fetched.

    Fields inherited from interface org.antlr.v4.runtime.IntStream

    EOF, UNKNOWN_SOURCE_NAME
  • Constructor Summary

    Constructors
    Constructor
    Description
     
  • Method Summary

    Modifier and Type
    Method
    Description
    protected int
    Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation.
    void
    Consumes the current symbol in the stream.
    protected int
    fetch(int n)
    Add n elements to buffer.
    void
    Get all tokens from lexer until EOF
    protected List<Token>
    filterForChannel(int from, int to, int channel)
     
    get(int i)
    Gets the Token at the specified index in the stream.
    get(int start, int stop)
    Get all tokens from start..stop inclusively
    getHiddenTokensToLeft(int tokenIndex)
    Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
    getHiddenTokensToLeft(int tokenIndex, int channel)
    Collect all tokens on specified channel to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
    getHiddenTokensToRight(int tokenIndex)
    Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF.
    getHiddenTokensToRight(int tokenIndex, int channel)
    Collect all tokens on specified channel to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF.
    Gets the name of the underlying symbol source.
    Get the text of all tokens in this buffer.
    getText(Interval interval)
    Return the text of all tokens within the specified interval.
    Return the text of all tokens in the source interval of the specified context.
    getText(Token start, Token stop)
    Return the text of all tokens in this stream between start and stop (inclusive).
     
    getTokens(int start, int stop)
     
    getTokens(int start, int stop, int ttype)
     
    getTokens(int start, int stop, Set<Integer> types)
    Given a start and stop index, return a List of all tokens in the token type BitSet.
    Gets the underlying TokenSource which provides tokens for this stream.
    int
    Return the index into the stream of the input symbol referred to by LA(1).
    int
    LA(int i)
    Gets the value of the symbol at offset i from the current position.
    protected final void
     
    protected Token
    LB(int k)
     
    LT(int k)
    Get the Token instance associated with the value returned by LA(k).
    int
    A mark provides a guarantee that seek() operations will be valid over a "marked range" extending from the index where mark() was called to the current index().
    protected int
    nextTokenOnChannel(int i, int channel)
    Given a starting index, return the index of the next token on channel.
    protected int
    previousTokenOnChannel(int i, int channel)
    Given a starting index, return the index of the previous token on channel.
    void
    release(int marker)
    This method releases a marked range created by a call to mark().
    void
    Deprecated.
    Use seek(0) instead.
    void
    seek(int index)
    Set the input cursor to the position indicated by index.
    void
    Reset this token stream by setting its token source.
    protected void
     
    int
    Returns the total number of symbols in the stream, including a single EOF symbol.
    protected boolean
    sync(int i)
    Make sure index i in tokens has a token.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

    • tokenSource

      protected TokenSource tokenSource
      The TokenSource from which tokens for this stream are fetched.
    • tokens

      protected List<Token> tokens
      A collection of all tokens fetched from the token source. The list is considered a complete view of the input once fetchedEOF is set to true.
    • p

      protected int p
      The index into tokens of the current token (next token to consume()). tokens[p] should be LT(1).

      This field is set to -1 when the stream is first constructed or when setTokenSource(org.antlr.v4.runtime.TokenSource) is called, indicating that the first token has not yet been fetched from the token source. For additional information, see the documentation of IntStream for a description of Initializing Methods.

    • fetchedEOF

      protected boolean fetchedEOF
      Indicates whether the Token.EOF token has been fetched from tokenSource and added to tokens. This field improves performance for the following cases:
      • consume(): The lookahead check in consume() to prevent consuming the EOF symbol is optimized by checking the values of fetchedEOF and p instead of calling LA(int).
      • fetch(int): The check to prevent adding multiple EOF symbols into tokens is trivial with this field.
    • Constructor Details

      • BufferedTokenStream

        public BufferedTokenStream(TokenSource tokenSource)
    • Method Details

      • getTokenSource

        public TokenSource getTokenSource()
        Description copied from interface: TokenStream
        Gets the underlying TokenSource which provides tokens for this stream.
        Specified by:
        getTokenSource in interface TokenStream
      • index

        public int index()
        Description copied from interface: IntStream
        Return the index into the stream of the input symbol referred to by LA(1).

        The behavior of this method is unspecified if no call to an initializing method has occurred after this stream was constructed.

        Specified by:
        index in interface IntStream
      • mark

        public int mark()
        Description copied from interface: IntStream
        A mark provides a guarantee that seek() operations will be valid over a "marked range" extending from the index where mark() was called to the current index(). This allows the use of streaming input sources by specifying the minimum buffering requirements to support arbitrary lookahead during prediction.

        The returned mark is an opaque handle (type int) which is passed to release() when the guarantees provided by the marked range are no longer necessary. When calls to mark()/release() are nested, the marks must be released in reverse order of which they were obtained. Since marked regions are used during performance-critical sections of prediction, the specific behavior of invalid usage is unspecified (i.e. a mark is not released, or a mark is released twice, or marks are not released in reverse order from which they were created).

        The behavior of this method is unspecified if no call to an initializing method has occurred after this stream was constructed.

        This method does not change the current position in the input stream.

        The following example shows the use of mark(), release(mark), index(), and seek(index) as part of an operation to safely work within a marked region, then restore the stream position to its original value and release the mark.

         IntStream stream = ...;
         int index = -1;
         int mark = stream.mark();
         try {
           index = stream.index();
           // perform work here...
         } finally {
           if (index != -1) {
             stream.seek(index);
           }
           stream.release(mark);
         }
         
        Specified by:
        mark in interface IntStream
        Returns:
        An opaque marker which should be passed to release() when the marked range is no longer required.
      • release

        public void release(int marker)
        Description copied from interface: IntStream
        This method releases a marked range created by a call to mark(). Calls to release() must appear in the reverse order of the corresponding calls to mark(). If a mark is released twice, or if marks are not released in reverse order of the corresponding calls to mark(), the behavior is unspecified.

        For more information and an example, see IntStream.mark().

        Specified by:
        release in interface IntStream
        Parameters:
        marker - A marker returned by a call to mark().
        See Also:
      • reset

        @Deprecated public void reset()
        Deprecated.
        Use seek(0) instead.
        This method resets the token stream back to the first token in the buffer. It is equivalent to calling seek(int)(0).
        See Also:
      • seek

        public void seek(int index)
        Description copied from interface: IntStream
        Set the input cursor to the position indicated by index. If the specified index lies past the end of the stream, the operation behaves as though index was the index of the EOF symbol. After this method returns without throwing an exception, then at least one of the following will be true.
        • index() will return the index of the first symbol appearing at or after the specified index. Specifically, implementations which filter their sources should automatically adjust index forward the minimum amount required for the operation to target a non-ignored symbol.
        • LA(1) returns IntStream.EOF
        This operation is guaranteed to not throw an exception if index lies within a marked region. For more information on marked regions, see IntStream.mark(). The behavior of this method is unspecified if no call to an initializing method has occurred after this stream was constructed.
        Specified by:
        seek in interface IntStream
        Parameters:
        index - The absolute index to seek to.
      • size

        public int size()
        Description copied from interface: IntStream
        Returns the total number of symbols in the stream, including a single EOF symbol.
        Specified by:
        size in interface IntStream
      • consume

        public void consume()
        Description copied from interface: IntStream
        Consumes the current symbol in the stream. This method has the following effects:
        • Forward movement: The value of index() before calling this method is less than the value of index() after calling this method.
        • Ordered lookahead: The value of LA(1) before calling this method becomes the value of LA(-1) after calling this method.
        Note that calling this method does not guarantee that index() is incremented by exactly 1, as that would preclude the ability to implement filtering streams (e.g. CommonTokenStream which distinguishes between "on-channel" and "off-channel" tokens).
        Specified by:
        consume in interface IntStream
      • sync

        protected boolean sync(int i)
        Make sure index i in tokens has a token.
        Returns:
        true if a token is located at index i, otherwise false.
        See Also:
      • fetch

        protected int fetch(int n)
        Add n elements to buffer.
        Returns:
        The actual number of elements added to the buffer.
      • get

        public Token get(int i)
        Description copied from interface: TokenStream
        Gets the Token at the specified index in the stream. When the preconditions of this method are met, the return value is non-null.

        The preconditions for this method are the same as the preconditions of IntStream.seek(int). If the behavior of seek(index) is unspecified for the current state and given index, then the behavior of this method is also unspecified.

        The symbol referred to by index differs from seek() only in the case of filtering streams where index lies before the end of the stream. Unlike seek(), this method does not adjust index to point to a non-ignored symbol.

        Specified by:
        get in interface TokenStream
      • get

        public List<Token> get(int start, int stop)
        Get all tokens from start..stop inclusively
      • LA

        public int LA(int i)
        Description copied from interface: IntStream
        Gets the value of the symbol at offset i from the current position. When i==1, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When i==-1, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with i==0, but the specific behavior is unspecified because this method is frequently called from performance-critical code.

        This method is guaranteed to succeed if any of the following are true:

        • i>0
        • i==-1 and index() returns a value greater than the value of index() after the stream was constructed and LA(1) was called in that order. Specifying the current index() relative to the index after the stream was created allows for filtering implementations that do not return every symbol from the underlying source. Specifying the call to LA(1) allows for lazily initialized streams.
        • LA(i) refers to a symbol consumed within a marked region that has not yet been released.

        If i represents a position at or beyond the end of the stream, this method returns IntStream.EOF.

        The return value is unspecified if i<0 and fewer than -i calls to consume() have occurred from the beginning of the stream before calling this method.

        Specified by:
        LA in interface IntStream
      • LB

        protected Token LB(int k)
      • LT

        public Token LT(int k)
        Description copied from interface: TokenStream
        Get the Token instance associated with the value returned by LA(k). This method has the same pre- and post-conditions as IntStream.LA(int). In addition, when the preconditions of this method are met, the return value is non-null and the value of LT(k).getType()==LA(k).
        Specified by:
        LT in interface TokenStream
        See Also:
      • adjustSeekIndex

        protected int adjustSeekIndex(int i)
        Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i. If an exception is thrown in this method, the current stream index should not be changed.

        For example, CommonTokenStream overrides this method to ensure that the seek target is always an on-channel token.

        Parameters:
        i - The target token index.
        Returns:
        The adjusted target token index.
      • lazyInit

        protected final void lazyInit()
      • setup

        protected void setup()
      • setTokenSource

        public void setTokenSource(TokenSource tokenSource)
        Reset this token stream by setting its token source.
      • getTokens

        public List<Token> getTokens()
      • getTokens

        public List<Token> getTokens(int start, int stop)
      • getTokens

        public List<Token> getTokens(int start, int stop, Set<Integer> types)
        Given a start and stop index, return a List of all tokens in the token type BitSet. Return null if no tokens were found. This method looks at both on and off channel tokens.
      • getTokens

        public List<Token> getTokens(int start, int stop, int ttype)
      • nextTokenOnChannel

        protected int nextTokenOnChannel(int i, int channel)
        Given a starting index, return the index of the next token on channel. Return i if tokens[i] is on channel. Return the index of the EOF token if there are no tokens on channel between i and EOF.
      • previousTokenOnChannel

        protected int previousTokenOnChannel(int i, int channel)
        Given a starting index, return the index of the previous token on channel. Return i if tokens[i] is on channel. Return -1 if there are no tokens on channel between i and 0.

        If i specifies an index at or after the EOF token, the EOF token index is returned. This is due to the fact that the EOF token is treated as though it were on every channel.

      • getHiddenTokensToRight

        public List<Token> getHiddenTokensToRight(int tokenIndex, int channel)
        Collect all tokens on specified channel to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF. If channel is -1, find any non default channel token.
      • getHiddenTokensToRight

        public List<Token> getHiddenTokensToRight(int tokenIndex)
        Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF.
      • getHiddenTokensToLeft

        public List<Token> getHiddenTokensToLeft(int tokenIndex, int channel)
        Collect all tokens on specified channel to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL. If channel is -1, find any non default channel token.
      • getHiddenTokensToLeft

        public List<Token> getHiddenTokensToLeft(int tokenIndex)
        Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
      • filterForChannel

        protected List<Token> filterForChannel(int from, int to, int channel)
      • getSourceName

        public String getSourceName()
        Description copied from interface: IntStream
        Gets the name of the underlying symbol source. This method returns a non-null, non-empty string. If such a name is not known, this method returns IntStream.UNKNOWN_SOURCE_NAME.
        Specified by:
        getSourceName in interface IntStream
      • getText

        public String getText()
        Get the text of all tokens in this buffer.
        Specified by:
        getText in interface TokenStream
        Returns:
        The text of all tokens in the stream.
      • getText

        public String getText(Interval interval)
        Description copied from interface: TokenStream
        Return the text of all tokens within the specified interval. This method behaves like the following code (including potential exceptions for violating preconditions of TokenStream.get(int), but may be optimized by the specific implementation.
         TokenStream stream = ...;
         String text = "";
         for (int i = interval.a; i <= interval.b; i++) {
           text += stream.get(i).getText();
         }
         
        Specified by:
        getText in interface TokenStream
        Parameters:
        interval - The interval of tokens within this stream to get text for.
        Returns:
        The text of all tokens within the specified interval in this stream.
      • getText

        public String getText(RuleContext ctx)
        Description copied from interface: TokenStream
        Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to TokenStream.getText(Interval), but may be optimized by the specific implementation.

        If ctx.getSourceInterval() does not return a valid interval of tokens provided by this stream, the behavior is unspecified.

         TokenStream stream = ...;
         String text = stream.getText(ctx.getSourceInterval());
         
        Specified by:
        getText in interface TokenStream
        Parameters:
        ctx - The context providing the source interval of tokens to get text for.
        Returns:
        The text of all tokens within the source interval of ctx.
      • getText

        public String getText(Token start, Token stop)
        Description copied from interface: TokenStream
        Return the text of all tokens in this stream between start and stop (inclusive).

        If the specified start or stop token was not provided by this stream, or if the stop occurred before the start token, the behavior is unspecified.

        For streams which ensure that the Token.getTokenIndex() method is accurate for all of its provided tokens, this method behaves like the following code. Other streams may implement this method in other ways provided the behavior is consistent with this at a high level.

         TokenStream stream = ...;
         String text = "";
         for (int i = start.getTokenIndex(); i <= stop.getTokenIndex(); i++) {
           text += stream.get(i).getText();
         }
         
        Specified by:
        getText in interface TokenStream
        Parameters:
        start - The first token in the interval to get text for.
        stop - The last token in the interval to get text for (inclusive).
        Returns:
        The text of all tokens lying between the specified start and stop tokens.
      • fill

        public void fill()
        Get all tokens from lexer until EOF