class LogStash::Codecs::Multiline
The multiline codec is for taking line-oriented text and merging them into a single event.
The original goal of this codec was to allow joining of multi-line messages from files into a single event. For example - joining java exception and stacktrace messages into a single event.
The config looks like this:
input { stdin { codec => multiline { pattern => "pattern, a regexp" negate => true or false what => "previous" or "next" } } }
The 'pattern' should match what you believe to be an indicator that the field is part of a multi-line event.
The 'what' must be “previous” or “next” and indicates the relation to the multi-line event.
The 'negate' can be “true” or “false” (defaults false). If true, a message not matching the pattern will constitute a match of the multiline filter and the what will be applied. (vice-versa is also true)
For example, java stack traces are multiline and usually have the message starting at the far-left, then each subsequent line indented. Do this:
input { stdin { codec => multiline { pattern => "^\s" what => "previous" } } }
This says that any line starting with whitespace belongs to the previous line.
Another example is to merge lines not starting with a date up to the previous line..
input { file { path => "/var/log/someapp.log" codec => multiline { # Grok pattern names are valid! :) pattern => "^%{TIMESTAMP_ISO8601} " negate => true what => previous } } }
This is the base class for logstash codecs.
Public Instance Methods
# File lib/logstash/codecs/multiline.rb, line 157 def buffer(text) @time = Time.now.utc if @buffer.empty? @buffer << text end
# File lib/logstash/codecs/multiline.rb, line 141 def decode(text, &block) text.force_encoding(@charset) if @charset != "UTF-8" # Convert to UTF-8 if not in that character set. text = text.encode("UTF-8", :invalid => :replace, :undef => :replace) end match = @grok.match(text) @logger.debug("Multiline", :pattern => @pattern, :text => text, :match => !match.nil?, :negate => @negate) # Add negate option match = (match and !@negate) || (!match and @negate) @handler.call(text, match, &block) end
# File lib/logstash/codecs/multiline.rb, line 173 def do_next(text, matched, &block) buffer(text) flush(&block) if !matched end
# File lib/logstash/codecs/multiline.rb, line 178 def do_previous(text, matched, &block) flush(&block) if !matched buffer(text) end
# File lib/logstash/codecs/multiline.rb, line 184 def encode(data) # Nothing to do. @on_event.call(data) end
# File lib/logstash/codecs/multiline.rb, line 162 def flush(&block) if @buffer.any? event = LogStash::Event.new("@timestamp" => @time, "message" => @buffer.join("\n")) # Tag multiline events event.tag @multiline_tag if @multiline_tag && @buffer.size > 1 yield event @buffer = [] end end
# File lib/logstash/codecs/multiline.rb, line 103 def register require "grok-pure" # rubygem 'jls-grok' # Detect if we are running from a jarfile, pick the right path. patterns_path = [] if __FILE__ =~ /file:\/.*\.jar!.*/ patterns_path += ["#{File.dirname(__FILE__)}/../../patterns/*"] else patterns_path += ["#{File.dirname(__FILE__)}/../../../patterns/*"] end @grok = Grok.new @patterns_dir = patterns_path.to_a + @patterns_dir @patterns_dir.each do |path| # Can't read relative paths from jars, try to normalize away '../' while path =~ /file:\/.*\.jar!.*\/\.\.\// # replace /foo/bar/../baz => /foo/baz path = path.gsub(/[^\/]+\/\.\.\//, "") end if File.directory?(path) path = File.join(path, "*") end Dir.glob(path).each do |file| @logger.info("Grok loading patterns from file", :path => file) @grok.add_patterns_from_file(file) end end @grok.compile(@pattern) @logger.debug("Registered multiline plugin", :type => @type, :config => @config) @buffer = [] @handler = method("do_#{@what}".to_sym) end