Controlling Type I Experimental Error

Full disclosure: I’m currently taking a high-speed graduate class in Experimental Design from the Colorado State University statistics department. Right now, we’re working on methods for controlling Type I errors (“false alarms”) in cases where you might want to do multiple hypothesis tests.

For example, if you are testing sample means representing patient responses to three different disease treatments plus a fourth control treatment, you might want to test each individual disease treatment mean vs. the control mean to see whether patient responses under treatment are significantly different. This gives a total of at least 3 separate hypothesis tests. If you’ve designed each test so that your Type I error rate (the rate at which you erroneously decide that the treatments represent a genuine improvement when they really don’t) is capped at some value, usually 5%, then your overall Type I rate will be higher, maybe even as high as 15%. So some adjustment has to be made to your process to keep the overall Type I error rate controlled.

This writeup was done in an effort to collect and synthesize all the different methods for measuring and controlling Type I error into a digestible package. In short, I wrote this for my own understanding because, for some reason, I think much better about things when I write about them (because how can I know what I’m thinking until I see what I write?). I hope you find it useful.

Download Notes on multiple comparisons in statistical hypothesis testing

About Carolyn Johnston

I am the principal consultant at Johnston Consulting Services. I help small companies and nonprofits embrace new techniques, and win new business, in fields involving applied mathematics and geospatial technology.
This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s