class Enigma::Download

Handle polling the url returned by the Export api until we’re able to download it. Will load the zipped contents of the file into memory, and can then either write it to disk (‘write`), write the unzipped version to disk (`write_csv`) or parse the unzipped contents using the CSV library as an array of hashes

Constants

DELAY

Attributes

download_contents[RW]
raw_download[RW]
response[RW]

Public Class Methods

new(res) click to toggle source
# File lib/enigma/download.rb, line 14
def initialize(res)
  self.response = res
end

Public Instance Methods

datapath() click to toggle source
# File lib/enigma/download.rb, line 25
def datapath
  response.datapath
end
do_download() click to toggle source
# File lib/enigma/download.rb, line 89
def do_download
  Enigma.logger.info "Trying to download #{download_url}"
  success = false
  until success
    req = Typhoeus::Request.new(download_url)
    req.on_complete do |response|
      if response.response_code == 404
        sleep DELAY
      else
        success = true
        return response.body
      end
    end
    req.run
  end
end
download_url() click to toggle source

This url is where the file will eventually be available. Returns a 404 until then

# File lib/enigma/download.rb, line 21
def download_url
  response.export_url
end
get() click to toggle source

Actually goes and fetches the download. Don’t return the raw_download because it’s a massive amount of data that will take over your terminal in IRB mode.

@return true on success

# File lib/enigma/download.rb, line 35
def get
  @raw_download ||= do_download
  true
end
parse(opts = {}) click to toggle source
# File lib/enigma/download.rb, line 84
def parse(opts = {})
  opts = { headers: true, header_converters: :symbol }
  CSV.parse(unzip, opts.merge(opts || {}))
end
unzip() click to toggle source

Handle either .zip or .gz responses since they’ve changed the format.

# File lib/enigma/download.rb, line 68
def unzip
  @download_contents ||=
    begin
      if download_url =~ /\.gz/
        unzip_gz
      else
        unzip_zip
      end
    end
end
unzip_gz() click to toggle source
# File lib/enigma/download.rb, line 59
def unzip_gz
  tmp = write_tmp
  Zlib::GzipReader.open(tmp.path) do |zipfile|
    zipfile.read
  end
end
unzip_zip() click to toggle source
# File lib/enigma/download.rb, line 52
def unzip_zip
  tmp = write_tmp
  Zip::File.open(tmp.path) do |zipfile|
    zipfile.first.get_input_stream.read
  end
end
write(io) click to toggle source
# File lib/enigma/download.rb, line 40
def write(io)
  get
  io.write raw_download
end
write_csv(io) click to toggle source
# File lib/enigma/download.rb, line 79
def write_csv(io)
  unzip
  io.write download_contents
end
write_tmp() click to toggle source
# File lib/enigma/download.rb, line 45
def write_tmp
  tmp = Tempfile.new(datapath)
  write(tmp)
  tmp.rewind
  tmp
end