Multiple Comparisons

Multiple Comparison Procedures

Planned and Unplanned Tests

A planned approach is when you don't care about the ANOVA results - you know in advance what comparisons you are going to make. Planned tests are Dunn, Dunnett, and POC. All the rest are post hoc.

Simple vs. Complex Comparisons

Simple comparisons are pair-wise. The best tests for simple comparisons are N-K and Tukey. N-K is more powerful (lower beta) which means it'll pick up borderline cases. Tukey is less powerful but gives you a more conservative alpha (lower Type I error rate) because it keeps the family alpha low.

Complex comparisons involve more than two means. For complex combinations:

Use Dunn if you want high power but you are willing to settle for J(J-1)/4 comparisons, which is only about half your cases.
Use POC if you want the highest power but you are willing only to look at orthogonal comparisons.
Use Scheffe if you want to look at all possible combinations but you are willing to sacrifice power (it is least powerful).

Statistics Used

Regardless of the statistic used, all these tests use N-J for the second degree of freedom. However, the first degree of freedom varies for each of these tests:

N-K and Tukey (and Duncan, which is a crummy test) both use the q statistic. That is the one without the factor of 2.
- Tukey uses J, not J-1.
- N-K starts with J, then goes down stepwise J-1, J-2,... until it hits a comparison with no significance, then stops.
- Both Tukey and N-K use the Tukey table.
All the rest use the F ratio from the ANOVA. That has the factor of 2.
- Dunn uses c instead of J. c is the actual number of comparisons and must equal J(J-1)/4 or less. Dunn has its own table.
- Dunnett uses J-1. Dunnett has its own table.
- Scheffe uses J-1. Scheffe uses sqrt[(J-1)*F ratio] so it has no table of its own.
- POC uses 1, not J. For POC you take the sqrt[F ratio] using 1 instead of J.
Multiple t uses the t-test statistic so all it needs is the N-J term.

Crummy Tests

Don't use multiple t or Duncan. They let alpha grow unacceptably large, which means lots of Type I error - too much chance to be believable.

Power

Highest power is POC, if you are willing to deal only with orthogonal comparisons.
Lowest power is Scheffe, but you can do all comparisons with it.
For simple comparisons, N-K is more powerful than Tukey, but it also has a higher Type I error rate.

Flexibility

Scheffe allows you to handle all possible simple and complex comparisons. What you trade off is power: it has the lowest power of all tests. POC is least flexible but has the highest power.

Limitations

POC only deals with orthogonal comparisons. They are linearly independent. You have to make all the Ho coefficients sum to zero.
Dunnett only deals with comparisons vs. the control group.
Dunn only can deal with about half your comparisons, i.e. J(J-1)/4 of them - any more and it begins to push the LCD too high.
Multiple t and Duncan have alpha values that are unacceptably high.

Back to Statistics course

Lorraine Sherry
lsherry@carbon.cudenver.edu
Updated October 10, 1996