class Enigma::Download
Handle polling the url returned by the Export
api until we’re able to download it. Will load the zipped contents of the file into memory, and can then either write it to disk (‘write`), write the unzipped version to disk (`write_csv`) or parse the unzipped contents using the CSV library as an array of hashes
Constants
- DELAY
Attributes
download_contents[RW]
raw_download[RW]
response[RW]
Public Class Methods
new(res)
click to toggle source
# File lib/enigma/download.rb, line 14 def initialize(res) self.response = res end
Public Instance Methods
datapath()
click to toggle source
# File lib/enigma/download.rb, line 25 def datapath response.datapath end
do_download()
click to toggle source
# File lib/enigma/download.rb, line 89 def do_download Enigma.logger.info "Trying to download #{download_url}" success = false until success req = Typhoeus::Request.new(download_url) req.on_complete do |response| if response.response_code == 404 sleep DELAY else success = true return response.body end end req.run end end
download_url()
click to toggle source
This url is where the file will eventually be available. Returns a 404 until then
# File lib/enigma/download.rb, line 21 def download_url response.export_url end
get()
click to toggle source
Actually goes and fetches the download. Don’t return the raw_download
because it’s a massive amount of data that will take over your terminal in IRB mode.
@return true on success
# File lib/enigma/download.rb, line 35 def get @raw_download ||= do_download true end
parse(opts = {})
click to toggle source
# File lib/enigma/download.rb, line 84 def parse(opts = {}) opts = { headers: true, header_converters: :symbol } CSV.parse(unzip, opts.merge(opts || {})) end
unzip()
click to toggle source
Handle either .zip or .gz responses since they’ve changed the format.
# File lib/enigma/download.rb, line 68 def unzip @download_contents ||= begin if download_url =~ /\.gz/ unzip_gz else unzip_zip end end end
unzip_gz()
click to toggle source
# File lib/enigma/download.rb, line 59 def unzip_gz tmp = write_tmp Zlib::GzipReader.open(tmp.path) do |zipfile| zipfile.read end end
unzip_zip()
click to toggle source
# File lib/enigma/download.rb, line 52 def unzip_zip tmp = write_tmp Zip::File.open(tmp.path) do |zipfile| zipfile.first.get_input_stream.read end end
write(io)
click to toggle source
# File lib/enigma/download.rb, line 40 def write(io) get io.write raw_download end
write_csv(io)
click to toggle source
# File lib/enigma/download.rb, line 79 def write_csv(io) unzip io.write download_contents end
write_tmp()
click to toggle source
# File lib/enigma/download.rb, line 45 def write_tmp tmp = Tempfile.new(datapath) write(tmp) tmp.rewind tmp end