class S3DataPacker::Configuration
Attributes
Number of items of the final batch size
Whether to keep or delete the finalized batch file. Set to false if you want to keep the output files in the workdir.
Whether to compress the final batch file or not. If set to true, the output file will be compressed with GZip, and the uncompressed file will be removed.
Standard logger to output information
Maximum number of items to maintain in queue to not overflow while workers process items.
Time in seconds to wait when the queue reached max_queue_size
to keep adding new items.
Desired pattern to construct output filenames
String prefix to include in output filenames for the batches
Splitter character to concat different values to generate a filename
String suffix to include in output filenames for the batches
How many threads to run for reading and processing items. This needs to be balanced out with the speed at which item keys are gathered to prevent emptying the queue too early.
Time in seconds for thread to wait when locked.
Time in seconds to let a thread sleep when there’s no pending items in queue.
Directory to keep working files. Make sure you have permissions on the path set. If the path does not exist, the packer will try to create it before using it.
Public Class Methods
# File lib/s3_data_packer/configuration.rb, line 57 def initialize @thread_count = 2 @thread_sleep_time = 1 @thread_lock_wait_time = 1 @max_queue_size = 10000 @max_queue_wait = 5 @batch_size = 100000 @workdir = 'tmp/s3_data_packer' @cleanup_batch = true @compress_batch = true @output_filename_prefix = nil @output_filename_suffix = 'batch' @output_filename_pattern = %i[timecode_int suffix] @output_filename_splitter = '_' end
Public Instance Methods
# File lib/s3_data_packer/configuration.rb, line 77 def cleanup_batch? cleanup_batch == true end
# File lib/s3_data_packer/configuration.rb, line 73 def compress_batch? compress_batch == true end
# File lib/s3_data_packer/configuration.rb, line 85 def default_s3_credentials return nil unless s3_credentials? Aws::Credentials.new(s3_api_key, s3_api_secret) end
# File lib/s3_data_packer/configuration.rb, line 91 def filename_generator_defaults { prefix: output_filename_prefix, suffix: output_filename_suffix, pattern: output_filename_pattern, splitter: output_filename_splitter } end
# File lib/s3_data_packer/configuration.rb, line 81 def s3_credentials? s3_api_key && s3_api_secret end