class RMMSeg::Ferret::Tokenizer
The Tokenizer
tokenize text with RMMSeg::Algorithm
.
Public Class Methods
new(str)
click to toggle source
Create a new Tokenizer
to tokenize text
# File lib/rmmseg/ferret.rb, line 36 def initialize(str) self.text = str end
Public Instance Methods
next()
click to toggle source
Get next token
# File lib/rmmseg/ferret.rb, line 41 def next @algor.next_token end
text()
click to toggle source
Get the text being tokenized
# File lib/rmmseg/ferret.rb, line 46 def text @text end
text=(str)
click to toggle source
Set the text to be tokenized
# File lib/rmmseg/ferret.rb, line 51 def text=(str) @text = str @algor = RMMSeg::Config.algorithm_instance(@text, ::Ferret::Analysis::Token) end