class Grammaphone::TokenStream

This is not a descendant of Enumerator. This is explicit and intentional, due to use as an almost tree-like object. This implementation makes it behave as a near-functional list structure, which is extremely useful for this parser.

Public Class Methods

new(tokens, &split_method) click to toggle source

Creates a new instance of TokenStream, using the data from `tokens`. If `tokens` is a String, it's split using `split_method`, which takes a String and returns an Array. if `split_method` isn't provided, then `String#split` is called on `tokens`, using the space character as the separator.

If `tokens` is an Array of Strings, the Array is duplicated, and used directly.

If `tokens` is not a String or Array, then `to_a` is called on `tokens` and the result is used as the token stream.

# File lib/grammaphone/tokens.rb, line 21
def initialize(tokens, &split_method)
  case tokens
  when String
    if split_method.nil?
      @enum = tokens.split(" ")
    else
      @enum = split_method.call(tokens).to_a
    end
  when Array
    raise TokenStreamError unless tokens.all?{|t| t.kind_of?(String)}
    @enum = tokens.dup
  else
    raise TokenStreamError unless tokens.respond_to?(:to_a)
    @enum = tokens.to_a.dup # dup just in case
  end
  @pointer = 0
end

Public Instance Methods

each() { |t| ... } click to toggle source

Provided because there's a chance that it'll be useful. At the very least, it can't hurt, since any arrays produced are copies.

# File lib/grammaphone/tokens.rb, line 110
def each
  if block_given?
    @enum.each { |t| yield t }
    self
  else
    to_enum(:each)
  end
end
empty?() click to toggle source

Returns `true` if there are no tokens remaining in the stream and `false` otherwise. That is, any calls to `peek_token`, `peek`, `next_token`, or `next` are guaranteed to return `nil` if `empty?` returns `true`.

# File lib/grammaphone/tokens.rb, line 104
def empty?
  @pointer >= @enum.size
end
initialize_copy(orig) click to toggle source

This ensures that all instances refer to the exact same token stream, but not necessarily at the same point. This saves a great deal of memory, without risking stream data integrity.

Calls superclass method
# File lib/grammaphone/tokens.rb, line 42
def initialize_copy(orig)
  @enum = orig.instance_variable_get(:@enum)
  super
end
next() click to toggle source

Gets the next non-empty token, consuming all viewed tokens.

Follows the same relationship as `peek` and `peek_token`

# File lib/grammaphone/tokens.rb, line 50
def next
  token = next_token
  token = next_token while token&.empty?
  token
end
next_token() click to toggle source

Gets the next token, consuming it.

# File lib/grammaphone/tokens.rb, line 57
def next_token
  token = @enum[@pointer]
  raise NonstringTokenError unless token.nil? || token.kind_of?(String) 
  @pointer += 1
  token
end
peek(n = 0) click to toggle source

Peeks at the nth token from the current pointer, not counting empty tokens, not consuming any tokens.

if no count is given, deaults to the next immediate token.

Follows the same relationship as `next` and `next_token`

# File lib/grammaphone/tokens.rb, line 70
def peek(n = 0)
  offset = (0..n).inject(0) do |acc, p|
    peek_token(p)&.empty? ? acc + 1 : acc
  end
  peek_token(n + offset)
end
peek_token(n = 0) click to toggle source

Peeks at the nth token from the current pointer, not consuming it.

If no count is given, defaults to the next immediate token.

# File lib/grammaphone/tokens.rb, line 80
def peek_token(n = 0)
  raise ArgumentError.new("can't look back in the token stream") if n < 0
  @enum[@pointer + n]
end
reset() click to toggle source

Resets the pointer to the beginning of the token stream.

# File lib/grammaphone/tokens.rb, line 96
def reset
  @pointer = 0
  self
end
skip(n = 1) click to toggle source

Consumes the next n tokens, returning `self`.

This has no meaningful effect if the stream is empty.

If no count is given, defaults to consuming a single token

# File lib/grammaphone/tokens.rb, line 90
def skip(n = 1)
  @pointer += n
  self
end
to_a() click to toggle source

Returns the remaining tokens as an Array.

# File lib/grammaphone/tokens.rb, line 120
def to_a
  @enum[@pointer..].dup
end