tsbacktest.tsissm.autospec {tsissm} | R Documentation |
Walk Forward Model Backtest
Description
Generates an expanding window walk forward backtest.
Usage
## S3 method for class 'tsissm.autospec'
tsbacktest(
object,
start = floor(length(object$y)/2),
end = length(object$y),
h = 1,
estimate_every = 1,
rolling = FALSE,
weights_scheme = c("AIC", "BIC", "U"),
weights = NULL,
seed = NULL,
solver = "nloptr",
control = NULL,
trace = FALSE,
...
)
## S3 method for class 'tsissm.spec'
tsbacktest(
object,
start = floor(length(object$target$y_orig)/2),
end = length(object$target$y_orig),
h = 1,
estimate_every = 1,
rolling = FALSE,
seed = NULL,
solver = "nloptr",
control = NULL,
trace = FALSE,
...
)
Arguments
object |
an object of class “tsissm.spec” “tsissm.autospec”. |
start |
numeric data index from which to start the backtest. |
end |
numeric data index on which to end the backtest. The backtest will end 1 period before that date in order to have at least 1 out of sample value to compare against. |
h |
forecast horizon. As the expanding window approaches the “end”, the horizon will automatically shrink to the number of available out of sample periods. |
estimate_every |
number of periods at which the model is re-estimated and new predictions are generated (defaults to 1). |
rolling |
this indicates whether forecasts are made only on the estimation date (FALSE) or whether to filter the data 1 period at a time and forecast from the filtered data (TRUE). |
weights_scheme |
the weighting scheme to use when using ensembling (see note). |
weights |
a vector of fixed user supplied weights of length |
seed |
an value specifying if and how the random number generator should be initialized (‘seeded’). Either NULL or an integer that will be used in a call to set.seed before simulating the response vectors. |
solver |
only “nlortr” currently supported. |
control |
optional control parameters. |
trace |
whether to show the progress bar. The user is expected to have set up appropriate handlers for this using the “progressr” package. |
... |
not used. |
Value
A list with the following data.tables:
prediction : the backtest table with forecasts and actuals
metrics: a summary performance table showing metrics by forecast horizon (MAPE, MSLRE, BIAS and MIS if alpha was not NULL).
Note
The function can use parallel functionality as long as the user has
set up a plan
using the future package. Model ensembling
is used when the input object is of class “tsissm.autospec” and
top_n is greeter than 1. The following weighting schemes are available:
- U:
User supplied fixed weights.
- AIC:
Models are weighted based on their Akaike Information Criterion (AIC), favoring models with lower AIC values. The weights are computed as:
w_i = \frac{\exp(-0.5 \times \Delta_i)}{\sum_{j} \exp(-0.5 \times \Delta_j)}
where
\Delta_i = AIC_i - \min(AIC)
- BIC:
Similar to AIC-based weighting but uses the Bayesian Information Criterion (BIC) instead, which penalizes model complexity more strongly.
The final ensemble prediction is computed as:
\hat{y}_{\text{ensemble}} = \sum_{i} w_i \hat{y}_i
where w_i
are the model weights and \hat{y}_i
are the individual model predictions.
AIC-based weighting is preferred when models have different complexities but are estimated on the same dataset,
while BIC-based weighting may be better suited for large sample sizes due to its stronger penalty on complexity.
See Also
[tsensemble()]