Efficient leave-one-out cross-validation and WAIC for fitted Bayesian models

• CRAN

install.packages("loo")

• GitHub

if (!require(devtools)) install.packages("devtools")
devtools::install_github("stan-dev/loo", build_vignettes = TRUE)

Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps.

This package implements the fast and stable computations for LOO and WAIC from Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC (preprint, published). From existing posterior simulation draws, we compute LOO using Pareto smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing predictive errors between two models.

Aki Vehtari, Andrew Gelman, Jonah Gabry

Corresponding Python and Matlab/Octave code can be found at the avehtari/PSIS repository.