Is There Only One Analysis?

There has been much controversy about the role of p<0.05 - the traditional requirement that the attained significance level from an experiment yield this result in order for the outcome to be considered evidence to reject the null hypothesis. The p < 0.05 controversy is most acute in clinical research because the regulatory agencies (e.g. U.S. Food and Drug Administration, European Medicines Agency) have traditionally insisted that sponsors (pharmaceutical company, medical device company) submit a frequentist intent-to-treat (analyze all patients as randomized regardless of how they are actually treated post treatment start) efficacy analysis of confirmatory clinical trials (usually Phase III) with the requirement that Type I error for the primary efficacy endpoint treatment comparison be less than 0.05 for marketing approval. We can refer to this analysis as the regulatory analysis because regulatory agencies find this useful in their work. Most often for a given medical product, we will generalize with the term “experimental treatment” herein, the regulatory analysis is the only one that will ever be reported in a journal and this is why many physicians, biostatisticians and other sciences are mounting a pushback for the p < 0.05 requirement.