bdpe_fetch_chunks {BigDataPE} | R Documentation |
Fetch data from the BigDataPE API in chunks
Description
This function retrieves data from the BigDataPE API iteratively in chunks.
It uses bdpe_fetch_data
as the base function and supports limits for the
total number of records to fetch and the size of each chunk.
Usage
bdpe_fetch_chunks(
base_name,
total_limit = Inf,
chunk_size = 500000L,
query = list(),
verbosity = 0L,
endpoint = "https://www.bigdata.pe.gov.br/api/buscar"
)
Arguments
base_name |
A string specifying the name of the dataset associated with the token. |
total_limit |
An integer specifying the maximum number of records to fetch. Default is Inf (all available data). |
chunk_size |
An integer specifying the number of records to fetch per chunk. Default is 50000 |
query |
A named list of additional query parameters to filter the API results. Default is an empty list. |
verbosity |
An integer specifying the verbosity level for the API requests.
Values are:
- |
endpoint |
A string specifying the API endpoint URL. Default is "https://www.bigdata.pe.gov.br/api/buscar". |
Value
A tibble containing all the data retrieved from the API.
Examples
## Not run:
# Store a token for the dataset
bdpe_store_token("dengue_dataset", "token")
# Fetch up to 500 records in chunks of 100
data <- bdpe_fetch_chunks("dengue_dataset", total_limit = 500, chunk_size = 100)
# Fetch all available data in chunks of 200
data <- bdpe_fetch_chunks("dengue_dataset", chunk_size = 200)
## End(Not run)