class RMMSeg::Ferret::PunctuationFilter
PunctuationFilter
filter out the stand alone Chinese punctuation tokens.
Public Class Methods
new(stream)
click to toggle source
# File lib/rmmseg/ferret.rb, line 84 def initialize(stream) @stream = stream end
Public Instance Methods
next()
click to toggle source
Get next token, skip stand alone Chinese punctuations.
# File lib/rmmseg/ferret.rb, line 89 def next token = @stream.next dic = Dictionary.instance until token.nil? || !(dic.include? token.text) token = @stream.next end token end
text()
click to toggle source
# File lib/rmmseg/ferret.rb, line 100 def text @stream.text end
text=(str)
click to toggle source
# File lib/rmmseg/ferret.rb, line 104 def text=(str) @stream.text = str end