class LogStash::Codecs::Multiline

The multiline codec is for taking line-oriented text and merging them into a single event.

The original goal of this codec was to allow joining of multi-line messages from files into a single event. For example - joining java exception and stacktrace messages into a single event.

The config looks like this:

input {
  stdin {
    codec => multiline {
      pattern => "pattern, a regexp"
      negate => true or false
      what => "previous" or "next"
    }
  }
}

The 'pattern' should match what you believe to be an indicator that the field is part of a multi-line event.

The 'what' must be “previous” or “next” and indicates the relation to the multi-line event.

The 'negate' can be “true” or “false” (defaults false). If true, a message not matching the pattern will constitute a match of the multiline filter and the what will be applied. (vice-versa is also true)

For example, java stack traces are multiline and usually have the message starting at the far-left, then each subsequent line indented. Do this:

input {
  stdin {
    codec => multiline {
      pattern => "^\s"
      what => "previous"
    }
  }
}

This says that any line starting with whitespace belongs to the previous line.

Another example is to merge lines not starting with a date up to the previous line..

input {
  file {
    path => "/var/log/someapp.log"
    codec => multiline {
      # Grok pattern names are valid! :)
      pattern => "^%{TIMESTAMP_ISO8601} "
      negate => true
      what => previous
    }
  }
}

This is the base class for logstash codecs.

Public Instance Methods

buffer(text) click to toggle source
# File lib/logstash/codecs/multiline.rb, line 157
def buffer(text)
  @time = Time.now.utc if @buffer.empty?
  @buffer << text
end
decode(text, &block) click to toggle source
# File lib/logstash/codecs/multiline.rb, line 141
def decode(text, &block)
  text.force_encoding(@charset)
  if @charset != "UTF-8"
    # Convert to UTF-8 if not in that character set.
    text = text.encode("UTF-8", :invalid => :replace, :undef => :replace)
  end

  match = @grok.match(text)
  @logger.debug("Multiline", :pattern => @pattern, :text => text,
                :match => !match.nil?, :negate => @negate)

  # Add negate option
  match = (match and !@negate) || (!match and @negate)
  @handler.call(text, match, &block)
end
do_next(text, matched, &block) click to toggle source
# File lib/logstash/codecs/multiline.rb, line 173
def do_next(text, matched, &block)
  buffer(text)
  flush(&block) if !matched
end
do_previous(text, matched, &block) click to toggle source
# File lib/logstash/codecs/multiline.rb, line 178
def do_previous(text, matched, &block)
  flush(&block) if !matched
  buffer(text)
end
encode(data) click to toggle source
# File lib/logstash/codecs/multiline.rb, line 184
def encode(data)
  # Nothing to do.
  @on_event.call(data)
end
flush() { |event| ... } click to toggle source
# File lib/logstash/codecs/multiline.rb, line 162
def flush(&block)
  if @buffer.any?
    event = LogStash::Event.new("@timestamp" => @time, "message" => @buffer.join("\n"))
    # Tag multiline events
    event.tag @multiline_tag if @multiline_tag && @buffer.size > 1

    yield event
    @buffer = []
  end
end
register() click to toggle source
# File lib/logstash/codecs/multiline.rb, line 103
def register
  require "grok-pure" # rubygem 'jls-grok'
  # Detect if we are running from a jarfile, pick the right path.
  patterns_path = []
  if __FILE__ =~ /file:\/.*\.jar!.*/
    patterns_path += ["#{File.dirname(__FILE__)}/../../patterns/*"]
  else
    patterns_path += ["#{File.dirname(__FILE__)}/../../../patterns/*"]
  end

  @grok = Grok.new

  @patterns_dir = patterns_path.to_a + @patterns_dir
  @patterns_dir.each do |path|
    # Can't read relative paths from jars, try to normalize away '../'
    while path =~ /file:\/.*\.jar!.*\/\.\.\//
      # replace /foo/bar/../baz => /foo/baz
      path = path.gsub(/[^\/]+\/\.\.\//, "")
    end

    if File.directory?(path)
      path = File.join(path, "*")
    end

    Dir.glob(path).each do |file|
      @logger.info("Grok loading patterns from file", :path => file)
      @grok.add_patterns_from_file(file)
    end
  end

  @grok.compile(@pattern)
  @logger.debug("Registered multiline plugin", :type => @type, :config => @config)

  @buffer = []
  @handler = method("do_#{@what}".to_sym)
end