The Power of Power Analysis
How can power analysis help us in designing our method? Isn’t significance enough? Not really. Significance only tells us how unlikely a Type II error would be, given our method—we may still have a potential Type I error. As well, something can be significant, and not be important. Cohen (1992) has long argued that significance doesn’t really tell us all that much. It’s the statistical power—how well we reject the null hypothesis that really gives us confidence. So, how can we design for success? How can we get at the power of power analysis?
Discover How We Assist to Edit Your Dissertation Chapters
Aligning theoretical framework, gathering articles, synthesizing gaps, articulating a clear methodology and data plan, and writing about the theoretical and practical implications of your research are part of our comprehensive dissertation editing services.
- Bring dissertation editing expertise to chapters 1-5 in timely manner.
- Track all changes, then work with you to bring about scholarly writing.
- Ongoing support to address committee feedback, reducing revisions.
The Power of Sample Size: One way to increase power during power analysis, is to simply use a very large sample size, and—given even a very small difference—obtain significance. That’s because sample size has such a huge impact on our results. As sample size goes up, our test statistic gets better and our standard error goes down. The problem is, this doesn’t mean the difference tells us anything worthwhile. If our effect size is small, this difference may actually be meaningless. Besides—we may not be able to obtain a large sample (Nolan & Heinzen, 2011).
The Power of Alpha: Another way to increase power during power analysis, is to increase our alpha from .01 to .05, or from .05 to .10. The drawback is that now we are increasing our chances of rejecting the null when we shouldn’t (a Type I error). What we might gain in power, we might lose in confidence (Nolan & Heinzen, 2011).
The Power of Tails: A third way to benefit during power analysis, is by switching from a two-tailed test to a one-tailed test, because when we look both ways (in both tails) we split our power too. But that, too, is the disadvantage—unless we are certain, looking both ways may be required to avoid a Type I error (UCLA: Statistical Consulting Group, 2013).
The Power of Design: A fourth way to improve via power analysis, is the use of repeated-measures designs (as opposed to between-groups or quasi-experimental). Gathering more time points will also increase power—up to a point (UCLA: Statistical Consulting Group, 2013).
The Power of Procedure: Nonparametric tests may actually give you more power, especially if you are using ANOVA and are unsure of your assumptions—like independence, normality, and heterogeneity (UCLA: Statistical Consulting Group, 2013).
The Power of Effect Size: Lastly, why don’t we just manipulate the effect size, if it increases power? Well, it depends on your experimental method—you may not be able to. A better way to tell during your power analysis, is to consider the type of effect size index you will use, e.g. the standardized mean difference (Cohen’s d or Hedges’ g) or the point-biserial coefficient (rpb). Generally you use d or g for experimental measures and r for correlations, but McGrath and Meyer (2006) suggest you may wish to be more studious about their use. They note that d is advantageous for mean differences during experimental measures, is more intuitive, and is independent of base rates. But r has a more direct relationship with power than d does, offers more flexibility, and works better with general linear models (prediction). On the downside, r is sensitive to base rates. It really depends on what you are trying to achieve. Do you want the relative impact size of one variable on another, the real predictive value, or to evidence whether one is a risk factor for the other? (Cole, 2013; McGrath & Meyer, 2006).
Summary: How can we get at the power of power analysis? Sample size, alpha, number of tails, experimental design, parametric or nonparametric procedure, and effect size index should all be kept in mind.
References
Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. doi:10.1037/0033-2909.112.1.155
Cole, J. C. (2013). Power analysis: Intermediate course in the UCLA statistical consulting series on power. UCLA: Statistical Consulting Group. Retrieved August 30, 2013, from http://www.ats.ucla.edu/stat/seminars/power_analysis/Power_analysis_-_intermediate_course_for_UCLA_white.pdf
McGrath, R. E., & Meyer, G. J. (2006). When Effect Sizes Disagree: The Case of r and d. Psychological Methods, 11(4), 386-401. doi:10.1037/1082-989X.11.4.386
Nolan, S. A., & Heinzen, T. E. (2011). Essentials of statistics for the behavioral sciences. New York: Worth Publishers.
UCLA: Statistical Consulting Group. (2013). Introduction to Power Analysis. Retrieved August 30, 2013, from http://www.ats.ucla.edu/stat/seminars/Intro_power/default.htm