class JobIteration::EnumeratorBuilder
Public Class Methods
# File lib/job-iteration/enumerator_builder.rb, line 30 def initialize(job, wrapper: Wrapper) @job = job @wrapper = wrapper end
Public Instance Methods
Builds Enumerator from Active Record Relation and enumerates on batches, yielding Active Record Relations. See documentation for build_active_record_enumerator_on_batches
.
# File lib/job-iteration/enumerator_builder.rb, line 121 def build_active_record_enumerator_on_batch_relations(scope, cursor:, **args) JobIteration::ActiveRecordBatchEnumerator.new( scope, cursor: cursor, **args ).each end
Builds Enumerator from Active Record Relation and enumerates on batches of records. Each Enumerator tick moves the cursor batch_size
rows forward.
batch_size:
sets how many records will be fetched in one batch. Defaults to 100.
For the rest of arguments, see documentation for build_active_record_enumerator_on_records
# File lib/job-iteration/enumerator_builder.rb, line 110 def build_active_record_enumerator_on_batches(scope, cursor:, **args) enum = build_active_record_enumerator( scope, cursor: cursor, **args ).batches wrap(self, enum) end
Builds Enumerator from Active Record Relation. Each Enumerator tick moves the cursor one row forward.
columns:
argument is used to build the actual query for iteration. columns
: defaults to primary key:
1) SELECT * FROM users ORDER BY id LIMIT 100
When iteration is resumed, cursor:
and columns:
values will be used to continue from the point where iteration stopped:
2) SELECT * FROM users WHERE id > $CURSOR ORDER BY id LIMIT 100
columns:
can also take more than one column. In that case, cursor
will contain serialized values of all columns at the point where iteration stopped.
Consider this example with +columns: [:created_at, :id]+. Here's the query will use on the first iteration:
1) SELECT * FROM `products` ORDER BY created_at, id LIMIT 100
And the query on the next iteration:
2) SELECT * FROM `products` WHERE (created_at > '$LAST_CREATED_AT_CURSOR' OR (created_at = '$LAST_CREATED_AT_CURSOR' AND (id > '$LAST_ID_CURSOR'))) ORDER BY created_at, id LIMIT 100
As a result of this query pattern, if the values in these columns change for the records in scope during iteration, they may be skipped or yielded multiple times depending on the nature of the update and the cursor's value. If the value gets updated to a greater value than the cursor's value, it will get yielded again. Similarly, if the value gets updated to a lesser value than the curor's value, it will get skipped.
# File lib/job-iteration/enumerator_builder.rb, line 95 def build_active_record_enumerator_on_records(scope, cursor:, **args) enum = build_active_record_enumerator( scope, cursor: cursor, **args ).records wrap(self, enum) end
Builds Enumerator object from a given array, using cursor
as an offset.
# File lib/job-iteration/enumerator_builder.rb, line 49 def build_array_enumerator(enumerable, cursor:) unless enumerable.is_a?(Array) raise ArgumentError, "enumerable must be an Array" end if enumerable.any? { |i| defined?(ActiveRecord) && i.is_a?(ActiveRecord::Base) } raise ArgumentError, "array cannot contain ActiveRecord objects" end drop = if cursor.nil? 0 else cursor + 1 end wrap(self, enumerable.each_with_index.drop(drop).to_enum { enumerable.size }) end
Builds Enumerator objects that iterates once.
# File lib/job-iteration/enumerator_builder.rb, line 38 def build_once_enumerator(cursor:) wrap(self, build_times_enumerator(1, cursor: cursor)) end
# File lib/job-iteration/enumerator_builder.rb, line 129 def build_throttle_enumerator(enum, throttle_on:, backoff:) JobIteration::ThrottleEnumerator.new( enum, @job, throttle_on: throttle_on, backoff: backoff ).to_enum end
Builds Enumerator objects that iterates N times and yields number starting from zero.
# File lib/job-iteration/enumerator_builder.rb, line 43 def build_times_enumerator(number, cursor:) raise ArgumentError, "First argument must be an Integer" unless number.is_a?(Integer) wrap(self, build_array_enumerator(number.times.to_a, cursor: cursor)) end
Private Instance Methods
# File lib/job-iteration/enumerator_builder.rb, line 148 def build_active_record_enumerator(scope, cursor:, **args) unless scope.is_a?(ActiveRecord::Relation) raise ArgumentError, "scope must be an ActiveRecord::Relation" end JobIteration::ActiveRecordEnumerator.new( scope, cursor: cursor, **args ) end