loo_validation.bgmfit {bsitar}R Documentation

Perform leave-one-out (LOO) cross-validation

Description

The loo_validation() function is a wrapper around the brms::loo() function to perform approximate leave-one-out cross-validation based on the posterior likelihood. See brms::loo() for more details.

Usage

## S3 method for class 'bgmfit'
loo_validation(
  model,
  compare = TRUE,
  resp = NULL,
  dpar = NULL,
  pointwise = FALSE,
  moment_match = FALSE,
  reloo = FALSE,
  k_threshold = 0.7,
  save_psis = FALSE,
  moment_match_args = list(),
  reloo_args = list(),
  model_names = NULL,
  ndraws = NULL,
  draw_ids = NULL,
  cores = 1,
  deriv_model = NULL,
  verbose = FALSE,
  dummy_to_factor = NULL,
  expose_function = FALSE,
  usesavedfuns = NULL,
  clearenvfuns = NULL,
  envir = NULL,
  ...
)

loo_validation(model, ...)

Arguments

model

An object of class bgmfit.

compare

A logical flag indicating if the information criteria of the models should be compared using loo::loo_compare().

resp

Optional names of response variables. If specified, predictions are performed only for the specified response variables.

dpar

Optional name of a predicted distributional parameter. If specified, expected predictions of this parameters are returned.

pointwise

A flag indicating whether to compute the full log-likelihood matrix at once or separately for each observation. The latter approach is usually considerably slower but requires much less working memory. Accordingly, if one runs into memory issues, pointwise = TRUE is the way to go.

moment_match

A logical flag to indicate whether loo::loo_moment_match() should be applied to problematic observations. Defaults to FALSE. For most models, moment matching will only work if save_pars = save_pars(all = TRUE) was set when fitting the model with brms::brm(). See brms::loo_moment_match() for more details.

reloo

A logical flag indicating whether brms::reloo() should be applied to problematic observations. Defaults to FALSE.

k_threshold

The Pareto k threshold for which observations loo_moment_match or reloo is applied if argument moment_match or reloo is TRUE. Defaults to 0.7. See pareto_k_ids for more details.

save_psis

Should the "psis" object created internally be saved in the returned object? For more details see loo.

moment_match_args

An optional list of additional arguments passed to loo::loo_moment_match().

reloo_args

An optional list of additional arguments passed to brms::reloo().

model_names

If NULL (the default) will use model names derived from deparsing the call. Otherwise will use the passed values as model names.

ndraws

A positive integer indicating the number of posterior draws to use in estimation. If NULL (default), all draws are used.

draw_ids

An integer specifying the specific posterior draw(s) to use in estimation (default NULL).

cores

The number of cores to be used for parallel computations if future = TRUE. On non-Windows systems, this argument can be set globally via the mc.cores option. By default, NULL, the number of cores is automatically determined using future::availableCores(), and it will use the maximum number of cores available minus one (i.e., future::availableCores() - 1).

deriv_model

A logical value specifying whether to estimate the velocity curve from the derivative function or by differentiating the distance curve. Set deriv_model = TRUE for functions that require the velocity curve, such as growthparameters() and plot_curves(). Set it to NULL for functions that use the distance curve (i.e., fitted values), such as loo_validation() and plot_ppc().

verbose

A logical argument (default FALSE) to specify whether to print information collected during the setup of the object(s).

dummy_to_factor

A named list (default NULL) to convert dummy variables into a factor variable. The list must include the following elements:

  • factor.dummy: A character vector of dummy variables to be converted to factors.

  • factor.name: The name for the newly created factor variable (default is 'factor.var' if NULL).

  • factor.level: A vector specifying the factor levels. If NULL, levels are taken from factor.dummy. If factor.level is provided, its length must match factor.dummy.

expose_function

A logical argument (default FALSE) to indicate whether Stan functions should be exposed. If TRUE, any Stan functions exposed during the model fit using expose_function = TRUE in the bsitar() function are saved and can be used in post-processing. By default, expose_function = FALSE in post-processing functions, except in optimize_model() where it is set to NULL. If NULL, the setting is inherited from the original model fit. It must be set to TRUE when adding fit criteria or bayes_R2 during model optimization.

usesavedfuns

A logical value (default NULL) indicating whether to use already exposed and saved Stan functions. This is typically set automatically based on the expose_functions argument from the bsitar() call. Manual specification of usesavedfuns is rarely needed and is intended for internal testing, as improper use can lead to unreliable estimates.

clearenvfuns

A logical value indicating whether to clear the exposed Stan functions from the environment (TRUE) or not (FALSE). If NULL, clearenvfuns is set based on the value of usesavedfuns: TRUE if usesavedfuns = TRUE, or FALSE if usesavedfuns = FALSE.

envir

The environment used for function evaluation. The default is NULL, which sets the environment to parent.frame(). Since most post-processing functions rely on brms, it is recommended to set envir = globalenv() or envir = .GlobalEnv, especially for derivatives like velocity curves.

...

Additional arguments passed to the brms::loo() function. Please see brms::loo for details on various options available.

Details

The function supports model comparisons using loo::loo_compare() for comparing information criteria across models. For bgmfit objects, LOO is simply an alias for loo. Additionally, you can use brms::add_criterion() to store information criteria in the fitted model object for later use.

Value

If only one model object is provided, an object of class loo is returned. If multiple objects are provided, an object of class loolist is returned.

Author(s)

Satpal Sandhu satpal.sandhu@bristol.ac.uk

See Also

brms::loo()

Examples




# Fit Bayesian SITAR model 

# To avoid mode estimation which takes time, the Bayesian SITAR model fit to 
# the 'berkeley_exdata' has been saved as an example fit ('berkeley_exfit').
# See 'bsitar' function for details on 'berkeley_exdata' and 'berkeley_exfit'.

# Check and confirm whether model fit object 'berkeley_exfit' exists
berkeley_exfit <- getNsObject(berkeley_exfit)

model <- berkeley_exfit

# Perform leave-one-out cross-validation
loo_validation(model, cores = 1)



[Package bsitar version 0.3.2 Index]