ADVANTAGES OF PERMUTATION (RANDOMIZATION) TESTS IN CLINICAL AND EXPERIMENTAL PHARMACOLOGY AND PHYSIOLOGY

SUMMARY1. The statistical procedures that are used most commonly in clinical and experimental pharmacology and physiology are designed to test for differences between two means.2. The classical procedures for detecting such differences are those in which, under the population model of inference, the test statistic is referred to thet‐orF‐distributions. The validity of statistical inferences from these tests depends on a number of assumptions. Foremost among these is that the experimental groups have been constructed by taking random samples from defined populations. The statistical inferences then apply to the sampled populations.3. In biomedical research this sampling process is seldom followed. Instead, samples are usually acquired by non‐random selection, and are then divided by randomization into experimental groups. This being the case, it is theoretically invalid to use the classicalt‐or F‐tests to analyse the experimental results.4. The validity of inferences from the classical tests also depends on other assumptions, such as that the sampled populations are normal in form and of equal variance. It is difficult to be certain that these assumptions are fulfilled when group sizes are small, as they usually are in pharmacology and physiology. Breach of them, especially if the groups are unequal in size, can lead to serious statistical errors.5. Exact permutation tests are designed to make statistical inferences under the randomization model. These conclusions apply only to the results of experiments actually performed. By permuting the statistic of interest, such as the difference between arithmetic means, geometric means, medians, mid‐ranges or mean‐ranks of randomized groups of observations, the probability is calculated that the observed difference or a more extreme one could have occurred by chance. This inferential process is consistent with the way most biomedical experiments are designed and conducted.6. Exact permutation tests, or sampled permutation tests based on Monte Carlo random sampling of all possible permutations, can now be performed on personal computers. They are commended to biomedical investigators as being superior to the classical tests for analysing their experimental results when the central tendencies of two independent groups, or of two sets of measurements on the same group, are compared.7. When there is doubt that the assumptions for t‐tests are satisfied, investigators sometimes use non‐parametric rank‐order procedures such as the Wilcoxon‐Mann‐Whitney rank‐sum test for independent groups or the Wilcoxon signed rank‐sum test for paired observations. These procedures are permutation tests for differences between mean‐ranks and are invalid tests for differences between medians or means.8. When experiments are complex and based on randomization rather than random sampling, as in randomized block, Latin square, factorial or split‐unit designs, analysis of these by permutation tests is to be preferred on theoretical grounds, though the necessary computer software is not readily available.