Toward a new framework to evaluate process-based model configurations and quantify data worth prior to calibration

Abstract

Model criticism, discrimination, and selection methods often rely on calibrated model outputs. Because calibration can be computationally expensive, model criticism can first be undertaken by assessing model outputs obtained from limited prior parameter ensembles. However, such prior-based methods are often heuristic and do not formalize the notion of balancing model consistency with data and model complexity (i.e., model adequacy). We present a new framework to discriminate among candidate models prior to calibration that formalizes prior-to-calibration model adequacy into a metric to implicitly balance prior model output data coverage with model complexity represented by prior output (co)variance. The prior model adequacy metric “Mahalanobis distance deviation” quantifies the deviation of (a) the set of squared Mahalanobis distances of data from a prior model output distribution from (b) the set of squared Mahalanobis distances of data from their own distribution. A new data worth metric “discernment value” is also presented which quantifies the value of data for screening less-adequate models prior to calibration. Discernment value is calculated from the change in variance of a weighted average of prior model outputs from all candidate models due to less-adequate model outputs receiving lower weight. The framework is demonstrated using a one-dimensional groundwater flow model with eight possible configurations. A synthetic data network is used to test the framework. Results show the framework identifies the candidate models most similar to the true model used to create the synthetic data. Discernment values show variation in the value of different data types and locations for screening less-adequate models.