class FormatParser::ReadLimitsConfig
We need to apply various limits so that parsers do not over-read, do not cause too many HTTP requests to be dispatched and so on. These should be balanced with one another- for example, we cannot tell a parser that it is limited to reading 1024 bytes while at the same time limiting the size of the cache pages it may slurp in to less than that amount, since it can quickly become frustrating. ReadLimitsConfig
computes these limits for us, in a fairly balanced way, based on one setting.
Constants
- MAX_PAGE_FAULTS
Public Class Methods
# File lib/read_limits_config.rb, line 10 def initialize(total_bytes_available_per_parser) @max_read_bytes_per_parser = total_bytes_available_per_parser.to_i end
Public Instance Methods
How big should the cache page be. Each cache page read will incur one `#read` on the underlying IO object, remote or local
# File lib/read_limits_config.rb, line 28 def cache_page_size @max_read_bytes_per_parser / 4 end
Each parser can incur HTTP requests when performing `parse_http`. This constant sets the maximum number of pages each parser is allowed to hit that have not been fetched previously and are not stored in the cache. For example, with most formats the first cache page and the last cache page - tail and head of the file, respectively - will be available right after the first parser retreives some data. The second parser accessing the same data will reuse the in-memory cache.
# File lib/read_limits_config.rb, line 38 def max_pagefaults_per_parser MAX_PAGE_FAULTS end
Defines how many bytes each parser may request to read from the IO object given to it. Is used to artificially limit unbounded reads in parsers that may wander off and try to gulp in the file given to them indefinitely due to infinite loops or wrongly implemented skips - or when handling data that has been deliberately crafted in a way that can make a parser misbehave. This is less strict than one could think - for example, the MOOV parser used for Quicktime files will skip over the actual atom contents of the atoms, and will only read atom headers - which stays under this limit for quite some time.
# File lib/read_limits_config.rb, line 22 def max_read_bytes_per_parser @max_read_bytes_per_parser end
Defines how many `#read` calls each parser may perform on the IO object given to it. Is used to artificially limit unbounded reads in parsers that may wander off and try to gulp in the file given to them indefinitely due to infinite loops or wrongly implemented skips - or when handling data that has been deliberately crafted in a way that can make a parser misbehave.
# File lib/read_limits_config.rb, line 47 def max_reads_per_parser # Imagine we read per single byte @max_read_bytes_per_parser / 2 end
Defines how many `#seek` calls each parser may perform on the IO object given to it. Is used to artificially limit unbounded reads in parsers that may wander off and try to gulp in the file given to them indefinitely due to infinite loops or wrongly implemented skips - or when handling data that has been deliberately crafted in a way that can make a parser misbehave.
# File lib/read_limits_config.rb, line 57 def max_seeks_per_parser # Imagine we have to seek once per byte @max_read_bytes_per_parser / 2 end