Procedure: Uncertainty analysis for dynamic emulators¶
Description and Background¶
We describe an approximate method for quantifying uncertainty about dynamic simulator outputs given uncertainty about the simulator inputs. For a general discussion of uncertainty analysis, see its definition page (DefUncertaintyAnalysis) and the procedure page for carrying out uncertainty analysis using a GP emulator (ProcUAGP). This method is based on emulating the single step function, following the procedures described in the variant thread on dynamic emulation (ThreadVariantDynamic).
We suppose that there is a true, but uncertain sequence of forcing inputs \(A_1,\ldots,A_T\) and true values of the simulator parameters \(\phi\) and initial conditions \(W_0\), corresponding to the modelling situation of interest. We denote the true values of all these quantities by a vector \(X\). Uncertainty about \(X\) is described by a joint probability distribution \(\omega(\cdot)\). The corresponding, uncertain sequence of state variables that would be obtained by running the simulator at inputs \(X\) is denoted by \(W_1,\ldots,W_T\). The procedure we describe quantifies uncertainty about \(W_1,\ldots,W_T\) given uncertainty about \(X\).
Inputs¶
- An emulator for the single step function \(w_t=f(w_{t-1},a_t,\phi)\), formulated as a GP or t-process conditional on hyperparameters, training inputs \(D\) and training outputs \(f(D)\).
- A set \(\{\theta^{(1)},\ldots,\theta^{(s)}\}\) of emulator hyperparameter values.
- A joint distribution \(\omega(\cdot)\) for the forcing variables, initial conditions and simulator parameters
Outputs¶
- Approximate mean and variance for each of \(W_1,\ldots,W_T\)
Procedure¶
We describe a Monte Carlo procedure with \(N\) defined to be the number of Monte Carlo iterations. For notational convenience, we suppose that \(N\le s\). For discussion of the choice of \(N\), including the case \(N>s\), see the discussion page on Monte Carlo estimation (DiscMonteCarlo).
Generate a random value of \(X\) from its distribution \(\omega(\cdot)\). Denote this random value by \(X_i\)
Given \(X_i\), and one set of emulator hyperparameters \(\theta_i\), iterate the single step emulator using the approximation method described in the procedure page ProcApproximateIterateSingleStepEmulator to obtain \(\textrm{E}[W_t \|f(D),X_i,\theta^{(i)}]\) and \(\textrm{Var}[W_t |f(D),X_i,\theta^{(i)}]\) for all \(t\) of interest.
Repeat steps 2 and 3 \(N\) times and estimate \(\textrm{E}[W_t |f(D)]\) and \(\textrm{Var}[W_t|f(D)]\) by
\[\begin{split}\hat{\textrm{E}}[W_t |f(D)] &= \frac{1}{N}\sum_{i=1}^N \textrm{E}[W_t |f(D),X_i,\theta^{(i)}] \\ \widehat{\textrm{Var}}[W_t \|f(D)] &= \frac{1}{N}\sum_{i=1}^N \textrm{Var}[W_t |f(D),X_i,\theta^{(i)}] + \frac{1}{N-1}\sum_{i=1}^N \left\{\textrm{E}[W_t |f(D),X_i,\theta^{(i)}] - \hat{\textrm{E}}[W_t |f(D)] \right\}^2\end{split}\]
Additional Comments¶
Note that this procedure does not enable us to fully consider the two sources of uncertainty (uncertainty about inputs and uncertainty about the simulator) separately. (See the discussion page on uncertainty analysis (DiscUncertaintyAnalysis)). However, one term that is useful to consider is
This gives us the expected reduction in our variance of \(W_t\) obtained by learning the true inputs \(X\). If this term is small relative to \(\textrm{Var}[W_t |f(D)]\), it suggests that uncertainty about the simulator is large, and that more training runs of the simulator would be beneficial for reducing uncertainty about \(W_t\).