Complete Randomization Tests

Posted December 21st, 2008 by mrychlik

We have $n$ observational units, which we divide into 2 blocks (say, $A$ and $B$ ) of $k$ and $l$ experimental units respectively, in advance of an experiment. Thus, $k+l=n$ .

We conduct the experiment consisting in applying a different treatment to each block. We measure a non-categorical response $X$ , resulting in measurements $x_1,x_2,\ldots,x_n$ .

Let $\overline{x}_A$ and $\overline{x}_B$ be the group means of the two groups.
Let

$ \delta = \overline{x}_A-\overline{x}_B $

be the mean difference.

We would like to test whether treatment $A$ has significantly larger mean than treatment $B$ . If there is no difference (the null hypothesis, or $H_0$ ), the assignment of the observational units to the blocks has no effect, and $A$ and $B$ are only labels. Thus, any label assignment should produce the same result as the selected one.

There are

$ {n\choose k} = \frac{n!}{k! l!} $

ways to assign labels $A$ and $B$ to $n$ observational units, and under null hypothesis, all of them are equally probable. Thus, we may consider a pure thought experiment in which we consider all possible label assignments. For every such assignment, we may compute the difference of the means $\delta = \overline{x}_A-\overline{x}_B$ .

The proportion of label assignments for which the diference $\delta$ is larger than that of the original assignment is the significance level of the test.

It can be shown that the normal theory can be applied to analyze the resulting test from the point of view of significance level and power. Below we outline the corresponding argument.

We see that

$ \delta = \sum_{i=1}^n U_i X_i $

where

$ U_i\in\{1/k,-1/l\} $

with probability $1/2$ each and $X_i$ is the value of the response of an observational unit.

Since all random variables $X_i$ and $U_i$ are jointly independent, the conditional distribution of $(X_1,X_2,\ldots,X_n)$ conditioned on some particular set of values of variables $(U_1,U_2,\ldots,U_n)$ is exactly the same as the marginal distribution with all $U_i$ discarded.

Thus, the distribution of $\delta$ is that of a difference of two random variables $Z-W$ , where $Z$ is an average of $k$ independent copies of $X_1$ and $W$ is an average of $l$ independent copies of $X_1$ . Thus, if $k$ and $l$ are approximately $n/2$ and thus large, by the Central Limit Theorem both these variables are approximately normally distributed with the same mean and variance as $X_1$ . Hence, we may assume that the distributions are approximately $N(\mu,\sigma^2/k)$ and $N(\mu,\sigma^2/l)$ .