Class TokenIterator


  • public class TokenIterator
    extends java.lang.Object
    An iterator class that returns successive string tokens from some source. String tokens are, in general, separated by Pattern_White_Space in the source test. Furthermore, they may be delimited by either single or double quotes (opening and closing quotes must match). Escapes are processed using standard ICU unescaping.

    2015-sep-03 TODO: Only used in com.ibm.icu.dev.test.format, move there.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      private java.lang.StringBuffer buf  
      private boolean done  
      private int lastpos  
      private java.lang.String line  
      private int pos  
      private ResourceReader reader  
    • Constructor Summary

      Constructors 
      Constructor Description
      TokenIterator​(ResourceReader r)
      Construct an iterator over the tokens returned by the given ResourceReader, ignoring blank lines and comment lines (first non-blank character is '#').
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.lang.String describePosition()
      Return a string description of the position of the last line returned by readLine() or readLineSkippingComments().
      int getLineNumber()
      Return the one-based line number of the line of the last token returned by next().
      java.lang.String next()
      Return the next token from this iterator, or null if the last token has been returned.
      private int nextToken​(int position)
      Read the next token from 'this.line' and append it to 'this.buf'.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • line

        private java.lang.String line
      • buf

        private java.lang.StringBuffer buf
      • done

        private boolean done
      • pos

        private int pos
      • lastpos

        private int lastpos
    • Constructor Detail

      • TokenIterator

        public TokenIterator​(ResourceReader r)
        Construct an iterator over the tokens returned by the given ResourceReader, ignoring blank lines and comment lines (first non-blank character is '#'). Note that trailing comments on a line, beginning with the first unquoted '#', are recognized.
    • Method Detail

      • next

        public java.lang.String next()
                              throws java.io.IOException
        Return the next token from this iterator, or null if the last token has been returned.
        Throws:
        java.io.IOException
      • getLineNumber

        public int getLineNumber()
        Return the one-based line number of the line of the last token returned by next(). Should only be called after a call to next(); otherwise the return value is undefined.
      • describePosition

        public java.lang.String describePosition()
        Return a string description of the position of the last line returned by readLine() or readLineSkippingComments().
      • nextToken

        private int nextToken​(int position)
        Read the next token from 'this.line' and append it to 'this.buf'. Tokens are separated by Pattern_White_Space. Tokens may also be delimited by double or single quotes. The closing quote must match the opening quote. If a '#' is encountered, the rest of the line is ignored, unless it is backslash-escaped or within quotes.
        Parameters:
        position - the offset into the string
        Returns:
        offset to the next character to read from line, or if the end of the line is reached without scanning a valid token, -1