Example: Using Automatic Relevance Determination (ARD)

In this page we demonstrate the implementation of the ARD (see the procedure page ProcAutomaticRelevanceDetermination) method on a simple 1D synthetic example.

The simulator function is \(f(x_1,x_2) = sin(x_1/10) + 0 \times x_2\), i.e. a two variable function which ignores the second input altogether.

A 7 point design was used to train a emulator with a squared exponential function:

Table of Data

Variable	Value
\(x_1\)	0.1000	0.2333	0.3667	0.5000	0.6333	0.7667	0.9000
\(x_2\)	0.2433	0.3767	0.9100	0.6433	0.1100	0.5100	0.7767
\(f(x_1,x_2)\)	0.8415	0.7231	-0.5013	-0.9589	0.0501	0.9825	0.4121

Note that both input factors are operating on the same scale so no standardisation is needed in this case.

The inference is done by using a scaled conjugate gradient algorithm to maximise the log likelihood of the Gaussian Process emulator (that is in this case no priors are placed over the length scales).

To check the fit of the emulator we used a grid test set of 1000 points. The simulator values are plotted in black dots and the emulation mean prediction is the smooth coloured surface. We can clearly see from the plot that the both the simulator and emulator responses are insensitive to the value of \(x_2\).

To further validate the emulator and examine the output predictive variance, we plot below a profile of the simulator function at \(x_2=1\) and \(x_1\) a grid design of 1000 points. The simulator function is shown in green against the emulator prediction in black with the predictive variance in grey. The training data are also shown as crosses (although the \(x_2\) coordinate varies in the training data but clearly this has no effect on the output value from the simulator).

The length scales obtained through maximum likelihood are \(\delta_1 = 0.16\) and \(\delta_2 = 48.6\) which can be interpreted as the emulator using the first variable and ignoring the second.