class RMMSeg::Ferret::Analyzer
Public Class Methods
new(&brk)
click to toggle source
Construct an Analyzer
. Optional block can be used to add more +TokenFilter+s. e.g.
analyzer = RMMSeg::Ferret::Analyzer.new { |tokenizer| Ferret::Analysis::LowerCaseFilter.new(tokenizer) }
# File lib/rmmseg/ferret.rb, line 19 def initialize(&brk) @brk = brk end
Public Instance Methods
token_stream(field, text)
click to toggle source
# File lib/rmmseg/ferret.rb, line 23 def token_stream(field, text) t = PunctuationFilter.new(Tokenizer.new(text)) if @brk @brk.call(t) else t end end