class Grammaphone::TokenStream
This is not a descendant of Enumerator. This is explicit and intentional, due to use as an almost tree-like object. This implementation makes it behave as a near-functional list structure, which is extremely useful for this parser.
Public Class Methods
Creates a new instance of TokenStream
, using the data from `tokens`. If `tokens` is a String, it's split using `split_method`, which takes a String and returns an Array. if `split_method` isn't provided, then `String#split` is called on `tokens`, using the space character as the separator.
If `tokens` is an Array of Strings, the Array is duplicated, and used directly.
If `tokens` is not a String or Array, then `to_a` is called on `tokens` and the result is used as the token stream.
# File lib/grammaphone/tokens.rb, line 21 def initialize(tokens, &split_method) case tokens when String if split_method.nil? @enum = tokens.split(" ") else @enum = split_method.call(tokens).to_a end when Array raise TokenStreamError unless tokens.all?{|t| t.kind_of?(String)} @enum = tokens.dup else raise TokenStreamError unless tokens.respond_to?(:to_a) @enum = tokens.to_a.dup # dup just in case end @pointer = 0 end
Public Instance Methods
Provided because there's a chance that it'll be useful. At the very least, it can't hurt, since any arrays produced are copies.
# File lib/grammaphone/tokens.rb, line 110 def each if block_given? @enum.each { |t| yield t } self else to_enum(:each) end end
Returns `true` if there are no tokens remaining in the stream and `false` otherwise. That is, any calls to `peek_token`, `peek`, `next_token`, or `next` are guaranteed to return `nil` if `empty?` returns `true`.
# File lib/grammaphone/tokens.rb, line 104 def empty? @pointer >= @enum.size end
This ensures that all instances refer to the exact same token stream, but not necessarily at the same point. This saves a great deal of memory, without risking stream data integrity.
# File lib/grammaphone/tokens.rb, line 42 def initialize_copy(orig) @enum = orig.instance_variable_get(:@enum) super end
Gets the next non-empty token, consuming all viewed tokens.
Follows the same relationship as `peek` and `peek_token`
# File lib/grammaphone/tokens.rb, line 50 def next token = next_token token = next_token while token&.empty? token end
Gets the next token, consuming it.
# File lib/grammaphone/tokens.rb, line 57 def next_token token = @enum[@pointer] raise NonstringTokenError unless token.nil? || token.kind_of?(String) @pointer += 1 token end
Peeks at the nth token from the current pointer, not counting empty tokens, not consuming any tokens.
if no count is given, deaults to the next immediate token.
Follows the same relationship as `next` and `next_token`
# File lib/grammaphone/tokens.rb, line 70 def peek(n = 0) offset = (0..n).inject(0) do |acc, p| peek_token(p)&.empty? ? acc + 1 : acc end peek_token(n + offset) end
Peeks at the nth token from the current pointer, not consuming it.
If no count is given, defaults to the next immediate token.
# File lib/grammaphone/tokens.rb, line 80 def peek_token(n = 0) raise ArgumentError.new("can't look back in the token stream") if n < 0 @enum[@pointer + n] end
Resets the pointer to the beginning of the token stream.
# File lib/grammaphone/tokens.rb, line 96 def reset @pointer = 0 self end
Consumes the next n tokens, returning `self`.
This has no meaningful effect if the stream is empty.
If no count is given, defaults to consuming a single token
# File lib/grammaphone/tokens.rb, line 90 def skip(n = 1) @pointer += n self end
Returns the remaining tokens as an Array.
# File lib/grammaphone/tokens.rb, line 120 def to_a @enum[@pointer..].dup end