mape {tsaux} | R Documentation |
Forecast Performance Metrics
Description
Functions to calculate a number of performance metrics.
Usage
mape(actual, predicted)
bias(actual, predicted)
mslre(actual, predicted)
mase(actual, predicted, original_series = NULL, frequency = 1)
mis(actual, lower, upper, alpha)
wape(actual, predicted, weights)
wslre(actual, predicted, weights)
wse(actual, predicted, weights)
pinball(actual, distribution, alpha = 0.1)
crps(actual, distribution)
rmape(actual, predicted)
smape(actual, predicted)
msis(actual, lower, upper, original_series, frequency = 1, alpha)
Arguments
actual |
the actual values corresponding to the forecast period. |
predicted |
the predicted values corresponding to the forecast period. |
original_series |
the actual values corresponding to the training period. |
frequency |
the seasonal frequency of the series used in the model. |
lower |
the lower distributional forecast for the quantile corresponding to the coverage ratio alpha (i.e. alpha/2). |
upper |
the upper distributional forecast for the quantile corresponding to the coverage ratio alpha (i.e. 1 - alpha/2). |
alpha |
the distributional coverage. |
weights |
a vector of weights for generating weighted metrics. If the actual and predicted inputs are univariate, this should be equal to the length of the actual series and calculates a time-weighted average; otherwise, the weights should be of length equal to the number of series in a multivariate case, in which case a cross-sectional average is calculated. |
distribution |
the forecast distribution (returned in the distribution slot of the prediction object). This is used in the continuous ranked probability score (crps) of Gneiting et al. (2005), and calculated using the function from the 'scoringRules' package. |
Details
The following performance metrics are implemented:
- Mean Absolute Percentage Error (MAPE)
Measures the average percentage deviation of predictions from actual values.
MAPE = \frac{1}{n} \sum_{t=1}^{n} \left| \frac{A_t - P_t}{A_t} \right|
where
A_t
is the actual value andP_t
is the predicted value.- Rescaled Mean Absolute Percentage Error (RMAPE)
A transformation of MAPE using a Box-Cox transformation for scale invariance (Swanson et al.).
- Symmetric Mean Absolute Percentage Error (SMAPE)
An alternative to MAPE that symmetrizes the denominator.
SMAPE = \frac{2}{n} \sum_{t=1}^{n} \frac{|A_t - P_t|}{|A_t| + |P_t|}
- Mean Absolute Scaled Error (MASE)
Compares the absolute error to the mean absolute error of a naive seasonal forecast.
MASE = \frac{\frac{1}{n} \sum_{t=1}^{n} |P_t - A_t|}{\frac{1}{N-s} \sum_{t=s+1}^{N} |A_t - A_{t-s}|}
where
s
is the seasonal period.- Mean Squared Logarithmic Relative Error (MSLRE)
Measures squared log relative errors to penalize large deviations.
MSLRE = \frac{1}{n} \sum_{t=1}^{n} \left( \log(1 + A_t) - \log(1 + P_t) \right)^2
- Mean Interval Score (MIS)
Evaluates the accuracy of prediction intervals.
MIS = \frac{1}{n} \sum_{t=1}^{n} (U_t - L_t) + \frac{2}{\alpha} [(L_t - A_t) I(A_t < L_t) + (A_t - U_t) I(A_t > U_t)]
where
L_t
andU_t
are the lower and upper bounds of the interval.- Mean Scaled Interval Score (MSIS)
A scaled version of MIS, dividing by the mean absolute seasonal error.
MSIS = \frac{1}{h} \sum_{t=1}^{h} \frac{(U_t - L_t) + \frac{2}{\alpha} [(L_t - A_t) I(A_t < L_t) + (A_t - U_t) I(A_t > U_t)]}{\frac{1}{N-s} \sum_{t=s+1}^{N} |A_t - A_{t-s}|}
- Bias
Measures systematic overestimation or underestimation.
Bias = \frac{1}{n} \sum_{t=1}^{n} (P_t - A_t)
- Weighted Absolute Percentage Error (WAPE)
A weighted version of MAPE.
WAPE = \sum_{t=1}^{n} \mathbf{w} \frac{|P_t - A_t|}{A_t}
where
\mathbf{w}
is the weight vector.- Weighted Squared Logarithmic Relative Error (WSLRE)
A weighted version of squared log relative errors.
WSLRE = \sum_{t=1}^{n} \mathbf{w} (\log(P_t / A_t))^2
- Weighted Squared Error (WSE)
A weighted version of squared errors.
WSE = \sum_{t=1}^{n} \mathbf{w} \left( \frac{P_t}{A_t} \right)^2
- Pinball Loss
A scoring rule used for quantile forecasts.
\text{Pinball} = \frac{1}{n} \sum_{t=1}^{n} \left[ \tau (A_t - Q^\tau_t) I(A_t \geq Q^\tau_t) + (1 - \tau) (Q^\tau_t - A_t) I(A_t < Q^\tau_t) \right]
where
Q^\tau_t
is the predicted quantile at level
\tau
.
- Continuous Ranked Probability Score (CRPS)
A measure of probabilistic forecast accuracy.
CRPS = \frac{1}{n} \sum_{t=1}^{n} \int_{-\infty}^{\infty} (F_t(y) - I(y \geq A_t))^2 dy
where
F_t(y)
is the cumulative forecast distribution.
Value
A numeric value.
Note
The RMAPE is the rescaled measure for MAPE based on the paper by Swanson et al.
Author(s)
Alexios Galanos
References
Tofallis C (2015).
“A better measure of relative prediction accuracy for model selection and model estimation.”
Journal of the Operational Research Society, 66(8), 1352–1362.
Hyndman RJ, Koehler AB (2006).
“Another look at measures of forecast accuracy.”
International journal of forecasting, 22(4), 679–688.
Gneiting T, Raftery AE, Westveld III AH, Goldman T (2005).
“Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation.”
Monthly weather review, 133(5), 1098–1118.
Gneiting T, Raftery AE (2007).
“Strictly proper scoring rules, prediction, and estimation.”
Journal of the American statistical Association, 102(477), 359–378.
Swanson DA, Tayman J, Bryan TM (2011).
“MAPE-R: a rescaled measure of accuracy for cross-sectional subnational population forecasts.”
Journal of Population Research, 28, 225–243.