reltest2 {fitdistcp} | R Documentation |
Evaluation of Reliability for Certain Additional Models in the fitdistcp
Package
Description
This routine is mainly for reproducing certain results in Jewson et al. (2025), and not of general interest.
It uses simulations to evaluate the reliability of
the predictive quantiles produced by the
qgev_cp
, ggpd_cp
and qgev_p1_cp
routines in the fitdistcp
package.
For each model, results for 5 models are calculated.
This is to illustrate that the calibrating prior predictions dominate
the ml
, flat
, crhp_ml
and jp
predictions,
in terms of reliability.
Usage
reltest2(
model = "gev",
ntrials = 100,
nrepeats = 3,
nx = 50,
params = c(0, 1, 0),
alpha = seq(0.005, 0.995, 0.005),
plotflag = TRUE,
verbose = TRUE
)
Arguments
model |
which distribution to test. Possibles values are
" |
ntrials |
the number of trials to run. 5000 typically gives good results. |
nrepeats |
the number of entire repeats of the test to run, to check for convergence. 3 is a good choice. |
nx |
the length of the training data. |
params |
values for the parameters for the specified distribution |
alpha |
the alpha values at which to test |
plotflag |
logical to turn the plotting on and off |
verbose |
logical to turn loop counting on and off |
Details
The maximum likelihood quantiles (plotted in blue) do not give good reliability. They typically underestimate the tails (see panel (f)).
The cp
predictive quantiles generally give reasonably good reliability, especially for sample sizes of ~100.
The other predictions generally give poor reliability.
Value
A plot showing 9 different reliability checks, and a list containing various outputs, including the probabilities shown in the plot.
Author(s)
Stephen Jewson stephen.jewson@gmail.com
References
If you use this package, we would be grateful if you would cite the following reference, which gives the various calibrating priors, and tests them for reliability:
Jewson S., Sweeting T. and Jewson L. (2024): Reducing Reliability Bias in Assessments of Extreme Weather Risk using Calibrating Priors; ASCMO Advances in Statistical Climatology, Meteorology and Oceanography), https://ascmo.copernicus.org/articles/11/1/2025/.
See Also
An introduction to fitdistcp
, with more examples,
is given on this webpage.
The fitdistcp
package currently includes the following models (in alphabetical order):
Cauchy (
cauchy
),Cauchy with linear predictor on the mean (
cauchy_p1
),Exponential (
exp
),Exponential with log-linear predictor on the scale (
exp_p1
),Frechet with known location parameter (
frechet_k1
),Frechet with log-linear predictor on the scale and known location parameter (
frechet_p2k1
),Gamma (
gamma
),Generalized normal (
gnorm
),GEV (
gev
),GEV with linear predictor on the location (
gev_p1
),GEV with linear predictor on the location and log-linear prediction on the scale (
gev_p12
),GEV with linear predictor on the location, log-linear prediction on the scale, and linear predictor on the shape (
gev_p123
),GEV with linear predictor on the location and known shape (
gev_p1k3
),GEV with known shape (
gev_k3
),GPD with known location (
gpd_k1
),Gumbel (
gumbel
),Gumbel with linear predictor on the mean(
gumbel_p1
),Half-normal (
halfnorm
),Inverse gamma (
invgamma
),Inverse Gaussian (
invgauss
),t distribution with unknown location and scale and known DoF (
lst_k3
),t distribution with unknown location and scale, linear predictor on the location, and known DoF (
lst_p1k3
),Logistic (
logis
),Logistic with linear predictor on the location (
logis_p1
),Log-normal (
lnorm
),Log-normal with linear predictor on the location (
lnorm_p1
),Normal (
norm
),Normal with linear predictor on the mean (
norm_p1
),Pareto with known scale (
pareto_k2
),Pareto with log-linear predictor on the shape and known scale (
pareto_p1k2
),Uniform (
unif
),Weibull (
weibull
),Weibull with linear predictor on the scale (
weibull_p2
),
The level of predictive probability matching achieved
by the maximum likelihood and calibrating prior quantiles, for any model,
sample size and true parameter values, can be demonstrated using the
routine reltest
.
Model selection among models can be demonstrated using the routines
ms_flat_1tail
,
ms_flat_2tail
,
ms_predictors_1tail
,
and ms_predictors_2tail
,
Examples
set.seed(1)
# example 1
# -runs the default settings, which test reliability for the GEV distribution
reltest2(nrepeats=1)