ERROR ERROR IN THE WALL – PLEASE TAKE IT AWAY!

By: Aly Diana

As a disclaimer, I would like to clarify that while I am not an expert in statistics, I am keen on learning and expanding my knowledge. Recently, a professor asked me about the adjustment of multiple comparisons, and my initial reaction was, “Oh noooooo, I totally forgot!” Therefore, I decided to share a brief reminder with all of you in the hope that it might be helpful.

The adjustment of multiple comparisons is a crucial step in many fields of research, such as genetics, medicine, and social sciences. Whenever multiple tests are conducted on a dataset, the risk of making at least one Type I error increases, leading to false discoveries. As a result, the adjustment of multiple comparisons is necessary to control the familywise error rate (FWER).

The FWER refers to the probability of making at least one Type I error across all the tests conducted. In other words, it is the likelihood of rejecting a true null hypothesis in at least one of the tests. For instance, if a researcher performs ten independent tests with a significance level of 0.05, the FWER is not 0.05 but rather 0.401. This implies that there is a 40.1% chance of making at least one Type I error in the ten tests.

To control the FWER, one of the most widely used methods is the Bonferroni correction. This method adjusts the significance level for each test based on the number of tests conducted. For example, if a researcher performs ten independent tests with a significance level of 0.05, the Bonferroni correction would adjust the significance level to 0.005 (0.05/10) for each test. This adjustment ensures that the overall probability of making at least one Type I error across all the tests is less than the desired level of significance.

Another widely used method is the Holm-Bonferroni method, which is more powerful than the Bonferroni correction. This method adjusts the significance level for each test in a stepwise manner, based on the tests’ p-values. The tests are ranked from the most significant to the least significant, and the significance level is adjusted starting from the most significant test and proceeding to the least significant test. The Holm-Bonferroni method is more powerful than the Bonferroni cor-rection because it can detect smaller effects while still controlling the FWER.

There are other methods for controlling the FWER, such as the Sidak correction, the Benjamini-Hochberg (BH), and the Hochberg method. Each method has its strengths and weaknesses, and the choice of method depends on the specific research question and the data’s characteristics.

In conclusion, controlling the FWER is a critical concept in multiple-hypothesis testing. The risk of making at least one Type I error increases when conducting multiple tests on a dataset, which can lead to false discoveries. To control the FWER, researchers use various methods, including the Bonferroni correction, the Holm-Bonferroni method, and the Benjamini-Hochberg method. These methods adjust the significance level or p-value threshold to ensure that the overall probability of making at least one Type I error across all the tests is less than the desired level of significance. By controlling the FWER, researchers can ensure that their results are reliable and meaningful.

References:

Chen SY, Feng Z, Yi X. A general introduction to adjustment for multiple comparisons. J Thorac Dis. 2017 Jun;9(6):1725-1729. doi: 10.21037/jtd.2017.05.34. PMID: 28740688; PMCID: PMC5506159.
Lee S, Lee DK. What is the proper way to apply the multiple comparison test? Korean J Anesthesiol. 2018 Oct;71(5):353-360. doi: 10.4097/kja.d.18.00242. Epub 2018 Aug 28. Erratum in: Korean J Anesthesiol. 2020 Dec;73(6):572. PMID: 30157585; PMCID: PMC6193594.
Vickerstaff V, Omar RZ, Ambler G. Methods to adjust for multiple comparisons in the analysis and sample size calculation of randomised controlled trials with multiple primary outcomes. BMC Med Res Methodol. 2019 Jun 21;19(1):129. doi: 10.1186/s12874-019-0754-4. Erratum in: BMC Med Res Methodol. 2019 Jul 22;19(1):158. PMID: 31226934; PMCID: PMC6588937.