Bandwidth Selection for Prediction in Regression ^†

Abstract

There exist many different methods to choose the bandwidth in kernel regression. If, however, the target is regression based prediction for samples or populations with potentially different distributions, then the existing methods can easily be suboptimal. This situation occurs for example in impact evaluation, data matching, or scenario simulations. We propose a bootstrap method to select a global bandwidth for nonparametric out-of-sample prediction. The asymptotic theory is developed, and simulation studies show the successful operation of our method. The method is used to predict nonparametrically the salary of Spanish women if they were paid along the same wage equation as men, given their own characteristics.

1. Introduction

While there exist a considerable literature on bandwidth selection for kernel based nonparametric density and regression estimation, the problem of nonparametric prediction has largely been ignored. To our knowledge, such selection method does not exist albeit the relevance and frequency of such prediction problems in practise. They include for example any situation for which you want to predict counterfactuals like in impact evaluation (also known as treatment effect estimation). Other examples are statistical matching or data matching (see [1,2,3], and references therein), the imputation of missings (see e.g., [4,5,6], and references therein), or the simulation of scenarios. Note that we are not thinking of extrapolation far outside of the support of the observed covariates, a problem that would go beyond the here described ones, see [7]. We do not refer to bandwidth selection in stationary time series. In this context, various bandwidth and other model selection methods have been developed, see e.g. the review of Antoniadis, [8] or [9].

In all these situations have the following three features in common: you can think of a regression model with Y being the left-hand, and X the observed right-hand variables. You have one sample, denoted as ’source’, in which both are given such that you can conduct a nonparametric regression. At the same time you have or simulate another sample or population, denoted as ’target’, for which the same (as for ’source’) potential response Y is not obtained. The basic assumption is that the dependence structure between, or in our case the conditional expectation of Y given X, $m\left(x\right):=E\left[Y\right|X=x]$ is the same in both populations. In data matching, and similarly when imputing missings, the Y were not sampled for the target sample; in scenarios the X of the target refer to an artificial, maybe future population, for which we just cannot observe any Y; in counterfactual exercises you typically have Y observed for the target sample, but under a different situation, called ’treatment’. Then you use the source sample to impute the potential Y of the target group for the situation ’without treatment’. The difference between the observed Y (under treatment) and the imputed (without treatment) gives the so-called ’treatment effect for the treated’.

Our proposal relies on the so-called smooth bootstrap approach, see [10]. That is, you aim to draw bootstrap samples from a nonparametric pre-estimate of the joint distribution of $(X,Y)$. For the original source sample, and for each bootstrap sample you estimate $m\left(x\right)$. These allow us to approximate the mean squared error of $\widehat{m}\left(x\right)$ for any x inside the support of X. Finally you average these over the $x_i$ observed in the target sample. We said ’you aim’ because it can be shown that there exists a closed analytical form for the resulting MASE estimate. This simplifies the procedure drastically making it quite attractive in practise. One may argue that the exactness of this MASE approximation hinges a lot upon the pre-estimate. Yet, for finding the optimal bandwidth (or model) it suffices that our MASE approximations take their minimum at the same bandwidth as the true but unknown MASE. Our simulation studies show that this is actually the case. This work is collected in [11].

2. The Bandwidth Selection Method

Suppose we are provided with a complete sample ${\left\{(x_i^0,y_i^0)\right\}}_{i=1}^{n^0}$ from the source population with $X^0\sim f^0$ and $m\left(x\right):=E_0\left[Y^0\right|X^0=x]$. For the target population we only are provided with observations ${\left\{x_i^1\right\}}_{i=1}^{n_1}$ from density $f^1$ which is potentially different from $f^0$. We are interested in predicting the expected ${\left\{y_i^1\right\}}_{i=1}^{n_1}$ assuming that $m\left(x\right)=E_1\left[Y^1\right|X^1=x]$, or to estimate $\mathbb{E}\left[Y^1\right]=\mathbb{E}\left[m\left(X^1\right)\right]$. Moreover, if some ’outcomes’ $y_i$ are observed for the target population, their conditional expectation is supposed to differ from $m(·)$; recall our example of outcome under treatment vs without, or see our application were $m\left(x\right)$ is the expected wage given x if you were a man.

For the prediction we have to estimate $m(·)$ by a Nadaraya-Watson estimator ${\widehat{m}}_h^{NW}$ with bandwidth h. Let us suppress for a moment the hyper-indices thinking for now always in the source sample with Y observed. The challenge is to find a bandwidth h which is MASE optimal for our predicting problem. The point-wise MSE, and afterwards the MASE are approximated by their bootstrap versions obtained as follows: Imagine $\left(X_1^{*},Y_1^{*}\right),\dots ,\left(X_n^{*},Y_n^{*}\right)$ are bootstrap samples drawn from the kernel density ${\widehat{f}}_g(x,y)=n^{-1}\sum _{i=1}^n{K}_g(x-X_i)K_g(y-Y_i)$ with bandwidth g. Then, for ${\tilde{m}}_h^{NW*}\left(x\right)f\left(x\right)={\widehat{f}}_h\left(x\right){\widehat{m}}_h^{NW}\left(x\right)$ we get

$$\begin{array}{ccc}\hfill {\tilde{m}}_h^{NW*}\left(x\right)-{\widehat{m}}_g^{NW}\left(x\right)& =& {\displaystyle \frac{1}{n{\widehat{f}}_g\left(x\right)}}\sum _{i=1}^n{K}_h(x-X_i^{*})(Y_i^{*}-{\widehat{m}}_g^{NW}\left(x\right)),\hfill \end{array}$$

(1)

where $X^{*}$ has bootstrap marginal density ${\widehat{f}}_g$, and $\mathbb{E}\left[\left.Y^{*}\right|X^{*}=x\right]={\widehat{m}}_g^{NW}\left(x\right)$. Clearly, this is the bootstrap analogue to

$$\begin{array}{ccc}\hfill {\tilde{m}}_h^{NW}\left(x\right)-m\left(x\right)& =& {\displaystyle \frac{1}{nf\left(x\right)}}\sum _{i=1}^n{K}_h(x-X_i)(Y_i-m\left(x\right)).\hfill \end{array}$$

(2)

In order to compute the MASE we need to carefully distinguish between source and target sample, and have therefore to use the hyper-indices again. For finding a globally optimal bandwidth h we would like to minimise

$$\begin{array}{ccc}\hfill MAS{E}_{{\tilde{m}}_h^{NW},X^1}\left(h\right)& =& {\displaystyle \frac{1}{n_1}}\sum _{j=1}^{n_1}\left[{\mathbb{E}}_0\left[{\left({\tilde{m}}_h^{NW}\left(X_j^1\right)-m\left(X_j^1\right)\right)}^2\right]\right],\hfill \end{array}$$

(3)

where ${\mathbb{E}}_0$ refers to the expectation in the source population. We have in the bootstrap world

$$\begin{array}{ccc}\hfill MAS{E}_{{\tilde{m}}_h^{NW},X^1}^{*}\left(h\right)& =& {\displaystyle \frac{1}{n_1}}\sum _{j=1}^{n_1}{\displaystyle \frac{1}{{\widehat{f}}_g^0{\left(X_j^1\right)}^2}}\left[\left(1-{\displaystyle \frac{1}{n_0}}\right)·{\left(\left[K_h*{\widehat{q}}_{X_j^1,g}^0\right]\left(X_j^1\right)\right)}^2\right.\hfill \\ \hfill & & \left.+{\displaystyle \frac{1}{n_0}}\left[{\left(K_h\right)}^2*{\widehat{p}}_{X_j^1,g}^0\right]\left(X_j^1\right)+{\displaystyle \frac{g^2{d}_K}{n_0^2}}\sum _{i=1}^{n_0}\left[{\left(K_h\right)}^2*K_g\right](X_j^1-X_i^0)\right].\hfill \end{array}$$

(4)

A bootstrap bandwidth selector for prediction is defined as

$$\begin{array}{ccc}\hfill h_{BOOT}^{NW}& =& h_{MAS{E}_{{\tilde{m}}_h^{NW},X^1}^{*}}=arg\underset{h>0}{min}MAS{E}_{{\tilde{m}}_h^{NW},X^1}^{*}\left(h\right).\hfill \end{array}$$

Note that the computation of $h_{BOOT}^{NW}$ does not require the use of Monte Carlo approximation nor the nonparametric estimation of the density $f^1$ of the target population.