How many replications?

The significance level

We draw $n$ pairs of samples of size $r$ from a normal population $N(0, \sigma)$. We calculate the statistic, which is the normalized difference of the means. We compute type I error rate as the fraction of $n$ of those experiments for which the statistic exceeded $z_{\alpha/2}$ value.

The power

We draw $n$ pairs of samples of size $r$ from two normal population $N(0, \sigma)$ and $N(\delta,\sigma)$. We calculate the statistic, which is the normalized difference of the means. We compute type II error rate as the fraction of $n$ of those experiments for which the absolute value of the statistic fell below $z_{\alpha/2}$ value.

Significance level (alpha):

One less power (type II error rate, beta):

Standard deviation (sigma):

Difference of the means (delta):

Simulation length (n):

Estimated r and observed error rates

z.alpha.half:	1.959964
z.beta:	0.8416212
Number of replications (r):	16
Type I error rate:	0.054
Type II error rate:	0.196

## Note: 1-beta is the probability of detecting difference delta

## Let us find the Z-scores
z.alpha.half <- qnorm(alpha/2, mean=0, sd=1, lower.tail = FALSE);
z.beta       <- qnorm(beta, mean=0, sd=1, lower.tail = FALSE);

## Estimate the number of replications
r.min        <- 2 * (z.alpha.half + z.beta)^2 * (sigma / delta)^2;
r.min        <- ceiling(r.min);

type.I.error.count <- 0;
type.II.error.count <- 0;

## Select a random sample n times
for(i in 1:n) {

  ## Select sample with mean zero
  x.insignificant <- rnorm(r.min, mean = 0, sd = sigma)
  y.insignificant <- rnorm(r.min, mean = 0, sd = sigma)
  y.significant <- rnorm(r.min, mean = delta, sd = sigma)

  ## Difference of the means
  d.insignificant <- mean(x.insignificant) - mean(y.insignificant);

  ## Shifted difference of the means by the resulting mean
  d.significant <-  mean(x.insignificant) - mean(y.significant);

  ## Exact standard deviation of the difference of the means
  sd <- sigma * sqrt(2 / r.min);

  ## Evaluate the statistic
  statistic.insignificant <- d.insignificant / sd;
  statistic.significant   <- d.significant / sd;

  ## Check for type I and type II errors
  type.I.error    <- abs( statistic.insignificant ) >= z.alpha.half;
  type.II.error   <- abs( statistic.significant )   < z.alpha.half;

  if(type.I.error) {
    type.I.error.count = type.I.error.count + 1;
  }

  if(type.II.error) {
    type.II.error.count = type.II.error.count + 1;
  }
}

## Find the error rates
type.I.error.rate = as.double(type.I.error.count) / n;
type.II.error.rate = as.double(type.II.error.count) / n;

Copyright(©) Marek Rychlik, 2008.