module NdrDevSupport::IntegrationTesting::FlakeyTests
Grudging handling of flakey integration tests. Allows tests to be declared with ‘flakey_test`. Our CI reporter gathers information on flakey failures.
Public Instance Methods
Source
# File lib/ndr_dev_support/integration_testing/flakey_tests.rb, line 26 def flakes @flakes ||= [] end
Source
# File lib/ndr_dev_support/integration_testing/flakey_tests.rb, line 13 def flakey_test(description, attempts: 3, &block) test(description, &block).tap do |test_name| self.attempts_per_test = attempts_per_test.merge(test_name.to_s => attempts) end end
Source
# File lib/ndr_dev_support/integration_testing/flakey_tests.rb, line 30 def run attempts_remaining = attempts_per_test[name] return super unless attempts_remaining previous_failure = failures.last failed_attempts = [] result = nil loop do break if attempts_remaining < 1 result = super # No failure was added; we passed! break if failures.last == previous_failure # Ran out of attempts: break if (attempts_remaining -= 1) < 1 # Loop round and have another go: failed_attempts << failures.pop end # Attempts were only flakey if we eventually passed: flakes.concat(failed_attempts) if failures.last == previous_failure result end
Calls superclass method
Source
# File lib/ndr_dev_support/integration_testing/flakey_tests.rb, line 19 def test_repeatedly(description, times: 100, &block) (1..times).map do |n| test("#{description} - #{n}/#{times}", &block) end end