The Central Limit Theorem

Java Applet Simulation of the sample mean experiment

The Random Sample

As usual, we start with a basic experiment and a random variable X of interest. We denote the mean and variance of X by

We repeat the experiment n times to from a random sample of size n from the distribution of X:

(X₁, X₂, ..., X_n)

Recall that these are independent random variables, each with the distribution of X.

The Central Limit Theorem

1. In the simulation of the sample mean experiment, set the basic distribution as indicated below. Increase the sample size from 1 to 10 and note how the shape of the distribution of the sample mean changes.

1. Binomial with m = 1 and p = 0.3.
2. Binomial with m = 5 and p = 0.7.
3. Poisson with mean 1.
4. Poisson with mean 3.
5. Gamma with a = 1 and r = 0.3.
6. Gamma with a = 3 and r = 1.
7. Normal with mean 2 and standard deviation 1.

In Exercise 1, you should have been struck by the fact that the density of the sample mean becomes increasingly bell-shaped, as the sample size increases, regardless of the shape of the density from which we are sampling. Even more remarkably, this phenomenon is not just qualitative: one particular family of density functions (the normal family) describes the limiting distribution of the sample mean, regardless of the basic distribution we start with. This theorem, known as the central limit theorem, is one of the fundamental theorems of probability. Its importance is hard to overstate; it is the reason that many statistical procedures are possible.

We will now make the central limit theorem precise. We know from the law of large numbers that the distribution of the sample mean converges to a point-mass distribution as the sample size increases. Thus, to obtain a limiting distribution that is not degenerate, we need to consider, not the sample mean itself, but the standard score of the sample mean. Thus, let

2. Show that E(Z_n) = 0 and var(Z_n) = 1.

3. Show that Z_n is also the standard score of the sum of the sample variables

The central limit theorem states that the distribution of the standard score of the sample mean converges to the standard normal distribution as n increases to infinity.

Proof of the Central Limit Theorem

There are several equivalent mathematical formulations of the central limit theorem. One is

where F_n is the distribution function of Z_n and the integral on the right is the distribution function of the standard normal distribution.

Another is

where M_n is the moment generating function of Z_n and the expression on the right is the moment generating function of the standard normal distribution. The following exercises will sketch a proof of the convergence of the moment generating functions. Thus let

4. Show that

5. Use the result of Exercise 2 and properties of moment generating functions to show that

6. Use Taylor's theorem with remainder and properties of the moment generating function to show that

7. In the context of Exercise 4, show that

8. Use the results of Exercises 3 and 4 and a theorem from calculus to show that

Normal Approximations

The central limit theorem implies that if the sample size n is "large," then the distribution of the sample mean is approximately normal, with the same mean and standard deviation as the underlying basic distribution. This fact is of fundamental importance in statistics, because it means that we can approximate the probability of an event involving the sample mean, even if we know very little about the underlying distribution.

Of course, the term "large" is relative. Roughly, the more "abnormal" the basic distribution, the larger n must be for normal approximations to work well. The rule of thumb is that a sample size n of at least 30 will suffice.

9. In the simulation of the sample mean experiment, set the basic distribution as indicated below. Increase the sample size and note the point at which the distribution of the sample mean looks reasonably normal.

1. Binomial with m = 1 and p = 0.3.
2. Binomial with m = 5 and p = 0.7.
3. Poisson with mean 1.
4. Poisson with mean 3.
5. Gamma with k = 1 and lambda = 0.3.
6. Gamma with k = 3 and lambda = 1.
7. Normal with mean 2 and standard deviation 1.

A slight technical problem arises when the basic variable X is discrete. In this case, the sample mean is also discrete and hence we are approximating a discrete distribution with a continuous one.

10. Suppose that X takes integer values and that k is a possible value of X. Show that the following events are equivalent for any h in the interval [0, 1):

In Exercise 10, different values of h lead to different normal approximations, even though the events are equivalent. The smallest approximation would be 0 when h = 0, and the approximations increase as h increases. It is customary to split the difference by using h = 0.5 for the normal approximation. This is sometimes called the continuity correction. The continuity correction is extended to other events in the natural way, using the additivity of probability.

The Sample Mean