8 Bootstrap Methods in Python
In this chapter, we intend to present the basic concepts of bootstrap in a very simple way. We will build some simple functions in a few particular cases.
8.1 Introduction
One of the main topics involving computationally intensive methods and, perhaps, the greatest responsible for the popularity of these methods is the bootstrap technique. The bootstrap method involves resampling with replacement from the original sample. Inspired by sampling theory, the bootstrap is used for hypothesis testing, parameter estimation by intervals, and estimation of standard errors.
The great advantage of using resampling methods, such as bootstrap, to perform inference arises when we do not know the probability distribution of the population and the normal model is not suitable for the data or residuals. The problem arises when we violate the conditions assumed for applying a test or obtaining a confidence interval or region.
In this chapter, we will study bootstrap methods for performing inference about parameters of interest.
8.2 Non-Parametric Bootstrap
Non-parametric bootstrap: the idea is to resample from the original sample. Let \(Y_1\), \(Y_2\), \(\ldots\), \(Y_n\) be a random sample of size \(n\) from a population with unknown distribution, and let the parameter of interest be \(\theta\). We can obtain a series of resamples with replacement of size \(n\) and estimate the parameter of interest by a function \(\hat{\theta}\) of the obtained samples.
This finite series of estimates represents a sample from the sampling distribution of the estimator, allowing us to make inference about \(\theta\). If we could obtain the sampling distribution of \(\hat{\theta}\) directly from sampling theory, we could make inference about \(\theta\) without needing bootstrap. For that, we would have to know the probability distribution of \(Y_j\), \(j\) \(=\) \(1\), \(2\), \(\ldots\), \(n\), and, from it, deduce the sampling distribution of the function of interest \(\hat{\theta}\).
This is not always possible, especially if we do not know the probability distribution of the random variable being sampled. For most probabilistic models, exact theories are not known or are analytically intractable. In that case, we use the empirical probability distribution, by which we assign to each of the sample observations the probability \(1/n\).
The idea of the bootstrap is to replace the unknown distribution of the population by the empirical distribution. For this reason, we call this method the non-parametric bootstrap. By obtaining a finite series of resamples of the same size as the original sample and building a sample from the distribution of the estimator, we are performing a mimicry process with respect to sampling theory.
This process is based on the fact that the sample obtained from the population contains all the available information of that underlying population. Hence, it comes to represent the “population” for the resampling process.
Efron (1979) was the researcher who organized the theories about this method and connected the non-parametric bootstrap with the statistical techniques for estimating errors, accepted since the 1930s, such as the jackknife and the delta method. Both the bootstrap technique and the permutation methods assume only that the random variables satisfy the principle of being exchangeable, which is a weaker assumption than being independent and identically distributed. By exchangeable we should understand that the probability distribution of any \(k\) consecutive observations (\(k\) \(=\) \(1\), \(2\), \(\ldots\), \(n\)) does not change when the order of the observations is changed by some permutation.
The general algorithm of the bootstrap resamples, to estimate an unknown parameter \(\theta\), whose probability distribution is \(f\) (of \(Y\)), considering also that the estimator \(\hat{\theta}\) is a function of the sample values, that is, \(\hat{\theta}\) \(=\) \(g(Y_1\), \(Y_2\), \(\ldots\), \(Y_n)\), is given by:
- assign probability mass \(1/n\) to each sample observation \(Y_j\), \(j\) \(=\) \(1\), \(2\), \(\ldots\), \(n\), creating the empirical probability distribution \(\hat{f}\);
- generate random samples with replacement from the empirical probability distribution \(\hat{f}\), called the bootstrap sample, denoted by \(\tilde{Y}_1\), \(\tilde{Y}_2\), \(\ldots\), \(\tilde{Y}_n\);
- compute an estimate of \(\hat{\theta}\) by \(\tilde{\theta}\), using the bootstrap sample in place of the original sample, as follows: \(\tilde{\theta}\) \(=\) \(g(\tilde{Y}_1\), \(\tilde{Y}_2\), \(\ldots\), \(\tilde{Y}_n)\);
- repeat steps \(2\) and \(3\) \(B\) times.
The number of resamples was denoted by \(B\), and the choice of adequate values according to the situation will be discussed later. The inference we intend to make depends on the unknown distribution of \(\hat{\theta}\) \(-\) \(\theta\). When we apply the previous algorithm, we obtain a Monte Carlo sample, from the bootstrap distribution, of the quantity \(\tilde{\theta}\) \(-\) \(\hat{\theta}\). If \(n\) is sufficiently large, by the ideas of the bootstrap method, extremely proven and accepted today, we expect the two distributions to be approximately identical.
8.3 Estimation
To estimate parameters of a population, the bootstrap technique is an excellent alternative, especially if the distribution is not normal or another distribution whose estimation processes are well-founded theoretically, such as, for example, the estimation of binomial proportions \(p\). If the bootstrap method is applied to these cases, it will not be a problem, since the results will be equivalent to those of classical theory.
However, if the assumptions of a classical estimation process are violated, in general, bootstrap methods will result in estimation processes with better properties. Initially, we must understand that bootstrap methods do not improve point estimates, but they provide appropriate mechanisms for estimating standard errors, confidence intervals, and the sampling distribution of the estimator, no matter how complex that estimator may be.
8.3.1 Standard Error
To estimate the standard error of an estimator \(\hat{\theta}\) of a parameter \(\theta\), we must obtain the bootstrap distribution of that estimator, as described in the algorithm presented earlier. Thus, if we consider a random sample \(Y_1\), \(Y_2\), \(\ldots\), \(Y_n\) of size \(n\) from the population of interest, which depends on a parameter \(\theta\) that we intend to estimate, then we apply the previous algorithm and obtain a bootstrap sample of size \(B\) from the distribution of that estimator.
If the sample is \(\tilde{\theta}_1\), \(\tilde{\theta}_2\), \(\tilde{\theta}_3\), \(\ldots\), \(\tilde{\theta}_B\), we can estimate the standard error of \(\hat{\theta}\) by \[ S_{\hat{\theta}} = \sqrt{\frac{1}{B-1} \sum_{j=1}^B \left(\tilde{\theta}_j - \bar{\tilde{\theta}} \right)^2} = \sqrt{\frac{1}{B-1}\left[\sum_{j=1}^B \tilde{\theta}_j^2 - \frac{\left(\sum_{j=1}^B \tilde{\theta}_j\right)^2}{B} \right]}, \tag{8.1}\] where \[ \bar{\tilde{\theta}}= \dfrac{\displaystyle \sum_{j=1}^B \tilde{\theta}_j}{B}. \]
We infer from expression (8.1) that the standard error of the estimator of \(\theta\) is the standard deviation of the bootstrap distribution of that estimator. If, however, the parameter \(\theta\) is the mean \(\mu\) of the sampled probability distribution, we can simplify the expression for the standard error estimator of the sample mean \(\bar{Y}\), since, in that case, we do not need the bootstrap sample of the estimator.
From statistical theory, we know that the standard error of the sample mean \(\bar{Y}\) is given by \(S_{\bar{Y}}\) \(=\) \((1/n)^{1/2}S\), where \(S\) is given by \[ S= \sqrt{\frac{1}{n-1} \sum_{j=1}^n (Y_j-\bar{Y})^2}=\sqrt{\frac{1}{n-1}\left[\sum_{j=1} Y_j^2 - \frac{\left( \sum_{j=1}^n Y_j\right)^2}{n}\right]}. \tag{8.2}\]
8.3.2 Bias Correction
We should explore in the bootstrap the possibility of bias correction for biased estimators. We know from classical theory that the bias \(\delta\) of an estimator \(\hat{\theta}\) is defined by \[ \delta_{\hat{\theta}} = E(\hat{\theta}) - \theta; \tag{8.3}\]
Since in the bootstrap distribution of an estimator \(\hat{\theta}\), the mean represents the expectation in (8.3), and the estimator \(\hat{\theta}\) in the original sample plays the same role as \(\theta\) in the population, we can estimate the bias by \[ \tilde{\delta}_{\hat{\theta}} = \bar{\tilde{\theta}} - \hat{\theta}. \tag{8.4}\]
It is common to think that \(\bar{\tilde{\theta}}\) would be the bias-corrected estimator, which is not true. We can obtain an adjusted estimator by \[ \begin{aligned} \hat{\theta}_{\textrm{adj.}}=& \hat{\theta} - \tilde{\delta}_{\hat{\theta}} =\hat{\theta} -\bar{\tilde{\theta}} + \hat{\theta} \nonumber\\ =& 2\hat{\theta} - \bar{\tilde{\theta}}; \end{aligned} \tag{8.5}\]
Next, we will address some of the different types of confidence intervals usually employed with the so-called non-parametric bootstrap method.
8.3.3 Standard Bootstrap Confidence Interval
In some situations, we can assume that the bootstrap distribution of the estimator \(\tilde{\theta}\) is normal. This occurs in special sampling situations: such as the specific case where \(\tilde{\theta}\) is the sample mean \(\bar{Y}\) in sampling from the normal distribution, and also in the case of large samples \(n\) as a consequence of the central limit theorem.
We know that \(\tilde{\theta}\) is a function of the sample elements, and as a consequence, for many of these functions, the normal distribution arises as a result of the central limit theorem. Unfortunately, experience has shown that this interval is, as a rule, very poor, in the sense that its properties are not preserved.
The main property required of a confidence interval is that it have coverage probability equal to the nominal confidence value \(1-\alpha\) adopted, which, in general, is not met in this case. The idea is to admit the effect of the central limit theorem: \[ \tilde{\theta} \dot{\sim} N\left(\hat{\theta},\, S_{\hat{\theta}}^2\right), \tag{8.6}\] where the symbol \(\dot{\sim}\) should be read as “is approximately distributed as”, \(\hat{\theta}\) is the value of the estimator in the original sample, and \(S_{\hat{\theta}}\) is the estimator of the standard error of \(\hat{\theta}\), obtained in the bootstrap distribution.
As a consequence of the result in (8.6), we can construct the standard bootstrap interval with confidence of approximately \(1-\alpha\) by \[ IC_{1-\alpha} (\theta): \left[\hat{\theta} - Z_{1-\alpha/2}S_{\hat{\theta}} ;\, \hat{\theta} - Z_{\alpha/2}S_{\hat{\theta}}\right], \tag{8.7}\] where \(Z_{\alpha/2}\) and \(Z_{1-\alpha/2}\) are the quantiles of the standard normal distribution. The confidence interval (8.7), as we have already commented, has a coverage probability, in general, below the nominal value of \(100(1-\alpha)\%\).
We can improve this interval by obtaining the confidence interval limits using a bias-corrected estimator. The result for this case is: \[ IC_{1-\alpha} (\theta): \left[\hat{\theta}_{\textrm{adj.}} - Z_{1-\alpha/2}S_{\hat{\theta}} ;\, \hat{\theta}_{\textrm{adj.}} - Z_{\alpha/2}S_{\hat{\theta}}\right], \tag{8.8}\] where \(\hat{\theta}_{\textrm{adj.}}\) is given in (8.5).
Next, we present an example for the gamma distribution. In the subsequent examples, we will use the same sample from the same distribution.
8.3.4 Confidence Interval Based on Bootstrap Percentiles
It is one of the most interesting, since its derivation is, at the very least, curious. Assuming normality of the bootstrap distribution underuses the potential of the method. However, we can think that there exists a monotone function \(\phi\) such that \(\phi(\tilde{\theta})\) has a normal distribution.
If we obtain the bootstrap distribution of this function, obtained by applying it to each resample with replacement of the original sample, we can obtain its standard deviation \(S_{\phi(\tilde{\theta})}\). Since this distribution, obtained by considering a finite number \(B\), is a sample from a normal distribution, the intervals obtained by \[ IC_{1-\alpha} (\phi(\theta)): \left[\phi(\hat{\theta}) - Z_{1-\alpha/2}S_{\phi(\tilde{\theta})} ;\, \phi(\hat{\theta}) - Z_{\alpha/2}S_{\phi(\tilde{\theta})}\right] \] or directly by sorting the sample of size \(B\) from the bootstrap distribution and taking its percentiles \(100(\alpha/2)\%\) and \(100(1-\alpha/2)\%\), should be equivalent as \(B\) \(\rightarrow\) \(\infty\).
According to Efron and Tibshirani (1993), the transformation \(\phi\) that leads to normality always exists. Thus, the confidence interval for \(\theta\), since \(\phi\) is a monotone function, is given by \[ IC_{1-\alpha} (\theta): \left[\phi^{-1}\left(\phi(\hat{\theta}) - Z_{1-\alpha/2}S_{\phi(\tilde{\theta})}\right) ;\, \phi^{-1}\left(\phi(\hat{\theta}) - Z_{\alpha/2}S_{\phi(\tilde{\theta})}\right)\right], \tag{8.9}\] where \(\phi^{-1}\) is the inverse function of \(\phi\).
We can verify from expression (8.9) that the confidence interval for the parameter on the original scale is obtained by the inverse transformation of the interval limits obtained on the transformed scale. We used in (8.9) the inverse transformation of the limits obtained from the standard normal percentiles and from the mean and variance of the bootstrap distribution to obtain the confidence interval limits on the original scale. However, we could have used a version that uses the inverse transformation of the percentiles \(100(\alpha/2)\%\) and \(100(1-\alpha/2)\%\) obtained directly from the bootstrap distribution on the transformed scale.
This can be done by virtue of what we have just commented, that is, that the bootstrap distribution obtained by the transformation, except for the fact that it is a finite sample, is normal, and the two intervals on the transformed scale are asymptotically equivalent.
Put this way, it seems easy to obtain an interval such as the one presented in (8.9). This is not true, since we would need to obtain the function \(\phi\), which would lead to a normal bootstrap distribution. Finding such a function for each real case is an infeasible task. On the other hand, we do not need to know such a function; it suffices to assume its existence.
Since the interval values obtained on the transformed scale, when mapped to the original scale, are equivalent to the percentiles \(100(\alpha/2)\%\) and \(100(1-\alpha/2)\%\) of the bootstrap distribution of \(\tilde{\theta}\), the construction of the confidence interval by this method is trivial. This result is extremely interesting and was preponderant for the success of bootstrap estimation methods. As Efron and Tibshirani (1993) state, we can understand the percentile method for obtaining the confidence interval as an automatic algorithm for incorporating such transformations that lead to normality. To make it clearer, we can say that the limits obtained from (8.9) correspond exactly to the interval based on percentiles of the bootstrap distribution on the original scale; therefore, we do not need to know the transformation that will lead the bootstrap distribution to normality, but only to assume its existence!
To apply the confidence interval based on bootstrap percentiles, we must apply the algorithm presented earlier. We obtain: \(\tilde{\theta}_1\), \(\tilde{\theta}_2\), \(\tilde{\theta}_3\), \(\ldots\), \(\tilde{\theta}_B\).
We must sort these values, obtaining the order statistics by: \(\tilde{\theta}_{(1)}\), \(\tilde{\theta}_{(2)}\), \(\tilde{\theta}_{(3)}\), \(\ldots\), \(\tilde{\theta}_{(B)}\). Then, we can obtain the confidence interval from this empirical distribution function, obtaining the percentiles \(100(\alpha/2)\%\) and \(100(1-\alpha/2)\%\). Thus, we construct the confidence interval based on bootstrap percentiles by \[ IC_{1-\alpha} (\theta): \left[\tilde{\theta}_{(k_1)} ;\, \tilde{\theta}_{(k_2)}\right], \tag{8.10}\] where \(k_1\) \(=\) \(\lfloor(B+1)(\alpha/2)\rfloor\) and \(k_2\) \(=\) \(\lfloor(B+1)(1-\alpha/2)\rfloor\), that is, they are the largest integers that are not greater than \((B+1)(\alpha/2)\) and \((B+1)(1-\alpha/2)\), respectively; \(\tilde{\theta}_{(k_1)}\) is the percentile \(100(\alpha/2)\%\) of the empirical bootstrap distribution function; and \(\tilde{\theta}_{(k_2)}\), the percentile \(100(1-\alpha/2)\%\) of the empirical bootstrap distribution function.
This interval has better properties than the previous one and works well in most simpler cases. However, there are better alternatives. We will see below some other types of intervals that are better than the first two, although in some situations the percentile method is superior to some of them.
A great advantage of the percentile interval is that it has the property of respecting the transformation, that is, the limits of this interval obey the domain of the parameter being estimated, which does not happen in some methods.
8.3.5 Basic Bootstrap Confidence Interval
A variation of the confidence interval based on bootstrap percentiles results from obtaining the distribution of \(\tilde{\theta}\) \(-\) \(\hat{\theta}\), which simply represents the bootstrap version of the sampling distribution of \(\hat{\theta}\) \(-\) \(\theta\). For large values of \(B\), \(\tilde{\theta}\) \(-\) \(\hat{\theta}\) converges in distribution to \(\hat{\theta}\) \(-\) \(\theta\).
If we consider that there exists a monotone function \(\phi(\tilde{\theta}\) \(-\) \(\hat{\theta})\) capable of leading to normality, then we would only need to obtain the percentiles \(100(\alpha/2)\%\) and \(100(1-\alpha/2)\%\) of the original distribution, since they map equivalent values on the transformed normal scale.
If we denote by \(\tilde{\delta}^{(\alpha/2)}\) and \(\tilde{\delta}^{(1-\alpha/2)}\) these percentiles on the original scale, then \[ P\left[\tilde{\delta}^{(\alpha/2)}\le \tilde{\theta} - \hat{\theta}\le\tilde{\delta}^{(1-\alpha/2)}\right]= 1-\alpha\\ P\left[\tilde{\theta}^{(\alpha/2)} - \hat{\theta} \le \tilde{\theta} - \hat{\theta} \le \tilde{\theta}^{(1-\alpha/2)}- \hat{\theta}\right]= 1-\alpha.\]
Since \[ \tilde{\theta}-\hat{\theta} \xrightarrow{d} \hat{\theta}-\theta, \] then, substituting into the previous expression, we have \[ P\left[\tilde{\theta}^{(\alpha/2)} - \hat{\theta} \le \hat{\theta}-\theta \le \tilde{\theta}^{(1-\alpha/2)}- \hat{\theta}\right]= 1-\alpha.\]
If we isolate \(\theta\) we have \[ P\left[2\hat{\theta} - \tilde{\theta}^{(1-\alpha/2)} \le \theta \le 2\hat{\theta} - \tilde{\theta}^{(\alpha/2)}\right]= 1-\alpha. \tag{8.11}\]
The limits of the probabilistic statement in (8.11) are the limits of a basic bootstrap \(100(1\) \(-\) \(\alpha)\%\) confidence interval, given by \[ IC_{1-\alpha} (\theta): \left[2\hat{\theta} - \tilde{\theta}^{(1-\alpha/2)} ;\, 2\hat{\theta} - \tilde{\theta}^{(\alpha/2)}\right]. \tag{8.12}\]
This interval can also be seen as the bootstrap \(t\) confidence interval, which will be the next one to be studied, when the standard error in each resample is considered equal to unity. The quantities \(\tilde{\theta}^{(1-\alpha/2)}\) and \(\tilde{\theta}^{(\alpha/2)}\) of interval (8.12) are, respectively, the percentiles \(\tilde{\theta}_{(k_1)}\) and \(\tilde{\theta}_{(k_2)}\) of interval (8.10) based on percentiles. We observe that the present method is merely a variation of the percentile method.
8.3.6 Bias-Corrected and Accelerated Bootstrap Confidence Interval
The bootstrap confidence intervals previously presented have deficiencies relative to coverage probabilities, at least in part of practical situations. Another method for obtaining bootstrap intervals has better qualities than the previous intervals, although it has greater difficulty of derivation: bias-corrected and accelerated bootstrap, \((BC_a)\).
This is the ideal method to be used. The \(BC_a\) method is obtained similarly to the percentile bootstrap method, but when determining the percentiles of the empirical bootstrap distribution \(\tilde{\theta}_{(1)}\), \(\tilde{\theta}_{(2)}\), \(\tilde{\theta}_{(3)}\), \(\ldots\), \(\tilde{\theta}_{(B)}\) of the estimator, two quantities \(\hat{a}\) and \(\hat{z}_0\) are also needed. The quantities \(\hat{a}\) and \(\hat{z}_0\) are defined as acceleration and bias correction.
The confidence interval based on bootstrap percentiles with bias-corrected and accelerated correction, \(BC_a\), is given by \[ IC_{1-\alpha} (\theta): \left[\tilde{\theta}_{(k_1)} ;\, \tilde{\theta}_{(k_2)}\right], \tag{8.13}\] where \(k_1\) \(=\) \(\lfloor(B+1)p_1\rfloor\) and \(k_2\) \(=\) \(\lfloor(B+1)p_2\rfloor\), that is, they are the largest integers that are not greater than \((B+1)p_1\) and \((B+1)p_2\), respectively; \(\tilde{\theta}_{(k_1)}\) is the percentile \(100(p_1)\%\) of the empirical bootstrap distribution function; and \(\tilde{\theta}_{(k_2)}\), the percentile \(100p_2\%\) of the empirical bootstrap distribution function.
For known values of \(\hat{a}\) and \(\hat{z}_0\), the values of \(p_1\) and \(p_2\), used to determine the orders \(k_1\) and \(k_2\) of the value of \(\tilde{\theta}\) in the empirical bootstrap distribution, as presented in (8.13), are determined by \[ p_1= \Phi\left(\hat{z}_0 + \dfrac{\hat{z}_0+z_{\alpha/2}}{1-\hat{a}\left[\hat{z}_0+z_{\alpha/2}\right]}\right) \textrm{ and } p_2= \Phi\left(\hat{z}_0 + \dfrac{\hat{z}_0+z_{1-\alpha/2}}{1-\hat{a}\left[\hat{z}_0+z_{1-\alpha/2}\right]}\right), \tag{8.14}\] where \(\Phi(x)\) is the standard normal distribution function evaluated at the value \(x\), and \(z_{\alpha/2}\) and \(z_{1-\alpha/2}\) are the quantiles \(100(\alpha/2)\%\) and \(100(1-\alpha/2)\%\) of that same distribution, respectively.
If both \(\hat{a}\) and \(\hat{z}_0\) are null, the values of \(p_1\) and \(p_2\) will be, respectively, \(\alpha/2\) and \(1-\alpha/2\), as can be easily deduced from (8.14). In that case, the intervals \(BC_a\), expression (8.13), and percentile bootstrap, expression (8.10), are equivalent. Non-null values of these quantities modify the limits of interval (8.13) and reduce some deficiencies of the percentile interval (8.10), as stated by Efron and Tibshirani (1993).
The value of \(\hat{z}_0\) is defined as the standard normal quantile whose cumulative probability is given by the proportion of estimates in the bootstrap distribution that is less than the estimate of \(\theta\) in the original sample. This value reflects the discrepancy that exists between the median of the distribution of \(\tilde{\theta}\) and the median of the distribution of \(\hat{\theta}\), expressed in standard normal units.
Thus, we define \(\hat{z}_0\) by \[ \hat{z}_0= \Phi^{-1}\left(\dfrac{\displaystyle \sum_{i=1}^B I\left(\tilde{\theta}_i <\hat{\theta}\right)}{B}\right), \tag{8.15}\] where \(\Phi^{-1}(p)\) is the inverse of the standard normal distribution function evaluated at \(p\) between \(0\) and \(1\), and \(I(\cdot)\) is the indicator function that returns \(1\) if the value of its argument is true and \(0\) if false.
The value \(\hat{z}_0\) will be zero only if the proportion of values of the bootstrap distribution that are less than \(\hat{\theta}\) equals \(50\%\). The value of \(\hat{a}\) can be computed using jackknife estimates of the parameter of interest.
Suppose the estimator \(\hat{\theta}\) is obtained by a function of the vector of sample observations \(\mathbf{Y}\) \(=\) \([Y_1\), \(Y_2\), \(\ldots\), \(Y_n]^\top\) by \(\hat{\theta}\) \(=\) \(g(\mathbf{Y})\). If we remove the \(i\)-th observation \(Y_i\), we have the vector \(\mathbf{Y}_{(i)}\) \(=\) \([Y_1\), \(Y_2\), \(\ldots\), \(Y_{i-1}\), \(Y_{i+1}\), \(\ldots\), \(Y_n]^\top\), which corresponds to the original vector without the \(i\)-th observation. The jackknife estimator is obtained by applying this same function to the resulting vector without the \(i\)-th observation, being given by \(\hat{\theta}_{(i)}\) \(=\) \(g(\mathbf{Y}_{(i)})\).
If we compute the mean of the \(n\) jackknife estimates, obtained by removing each of the \(n\) original observations, by \[ \bar{\hat{\theta}}_{(.)}= \dfrac{\displaystyle \sum_{i=1}^n \hat{\theta}_{(i)}}{n}, \] the acceleration \(\hat{a}\) can be obtained by \[ \hat{a} = \dfrac{\displaystyle \sum_{i=1}^n\left(\bar{\hat{\theta}}_{(.)} - \hat{\theta}_{(i)}\right)^3 }{6\left[\displaystyle \sum_{i=1}^n\left(\bar{\hat{\theta}}_{(.)} - \hat{\theta}_{(i)}\right)^2\right]^{3/2}}. \tag{8.16}\]
The quantity \(\hat{a}\) is called acceleration because it represents the rate of change of the standard error of \(\hat{\theta}\) relative to the change of the values of the true parameter \(\theta\) (EFRON; TIBSHIRANI, 1993). This confidence interval has two advantages: a) it yields estimates of the confidence interval limits within the parameter space, that is, it has the property of respecting the transformation; and b) it provides very precise results, in the sense that the coverage probabilities \(P(\theta<\tilde{\theta}_{(k_1)})\) \(=\) \(\alpha/2\) and \(P(\theta>\tilde{\theta}_{(k_2)})\) \(=\) \(\alpha/2\) are observed with second-order accuracy, whereas the percentile bootstrap method has only first-order accuracy.
8.3.7 Bootstrap Confidence Interval with Bias Correction
This method is a variation of the previous method, in which the acceleration \(\hat{a}\) is considered null. In that case, the confidence interval is obtained by the same expression (8.14), although the values of \(p_1\) and \(p_2\) are computed by \[ p_1= \Phi\left(2\hat{z}_0 + z_{\alpha/2}\right) \textrm{ and } p_2= \Phi\left(2\hat{z}_0 + z_{1-\alpha/2}\right), \tag{8.17}\] where \(\hat{z}_0\) is obtained in the same way as described earlier, through equation (8.15).
This method is particularly interesting in some situations where the values of \(\hat{a}\) are difficult to obtain. These cases are discussed in Efron and Tibshirani (1993). However, the \(BC_a\) method is, in general, more efficient and should be preferred.
8.3.8 Bootstrap in Python
Among some possibilities, we can use scipy.stats.bootstrap in Python to apply the bootstrap interval estimation methods. The bootstrap function from scipy is given by bootstrap(data, statistic, *, n_resamples=9999, batch=None, vectorized=None, paired=False, axis=0, confidence_level=0.95, alternative='two-sided', method='BCa', bootstrap_result=None, rng=None, random_state=None). Three methods are available: method{‘percentile’, ‘basic’, ‘BCa’}, with BCa being the default method. The following program shows how to obtain the confidence interval by this scipy procedure.
The data must be converted into a tuple before using the bootstrap method. See the example: