Bluffer's guide to power analysis (and weblinks)

 Posts: 25
 Joined: Sun Mar 25, 2007 6:43 pm
 Contact:
Bluffer's guide to power analysis (and weblinks)
If you are putting together a grant, research proposal, or course project, there are three pragmatic characteristics of sample sizes which should be satisfied.
a.) Feasible  Testing 400 people in 2 years is tough
b.) Sounds good  50 is a nice round number
c.) Is similar to what other people have done  If they got it published somewhere, anywhere, it's good enough for you.
A rough guide to samples (where you know the effect size) is here:
http://rgs.usu.edu/irb/wpcontent/uploa ... is_USU.pdf
Links to power calculators on the web:
http://powerandsamplesize.com/Calculators/
G*Power3  a very user friendly power calculator (honest!)
If you're going to be all scientific about it, you need to know a few things from the literature. Assuming it's the difference in mean scores between two groups you need to know:
* The mean for each group
* A pooled standard deviation (you can take the mean if they're similar, if they're not you should run multiple power analyses and present them all)
Now, one option is to turn to a single paper and go from there. If you're feeling more inspired you can draw up a spreadsheet with the means, S.D's, and sample sizes drawn from the literature. You then create a weighted mean and S.D. in excel. Here's a far too complex explanation on how to do that: http://en.wikipedia.org/wiki/Weighted_mean
Put simply, take an average like you normally would (sum of scores divided by number of measurements), and multiply them by the proportion of the total number of measurements which they represent.
So, if we have three studies:
N=10, mean = 23.5
N=80, mean = 26.4
N=10, mean = 12.5
Then we get the weighted mean by (0.1 x 23.5) + (0.8 x 26.4) + (0.1 x 12.5), which give us 24.72. The point of this is that we've assigned weights to the size of the studies. In small, fragmented literatures, this is essential. In larger, more mature literatures, less so. You might even be able to get hold of a metaanalysis or a review to figure out your sample size.
Anyway, what next? Well, my advice is go see a statistician, or find a copy of NQuery. If its the former, find a German. They're good. If the latter, use the wizards and provided you've collected the information above, you should muddle through. Those settings you're looking for by the way are power of 0.8 (80%) and alpha of 0.05 (i.e p=0.05).
You should write up your analysis along these lines:
"Based on the previous literature, we would expect group x to have a mean score of 5 and group y to have a mean score of 10. With a shared standard deviation of 2.5, 25 participants per group would be needed to detect a difference in means with 80% power and an alpha value set at 0.05"
If the S.D's look very different between the two populations, do a worst case/best case and present both.
If you're doing a million zillion comparisons, make the alpha 0.01. If the effect is massive but you don't want to look lazy testing 6 people, make the power higher (like 95%).
If you want the sample size to come out smaller, reduce the standard deviation, even by a small amount.
Also PLEASE remember. You'll be doing the power analysis on known effects. If your study is using any experimental measures that haven't been deployed in that population before, assume you'll need double the participants that you'll need for a known effect.
Note: If you have a suggestion about how to improve or add to this wiki please post it here. If you want to discuss this post please post a new thread in the forum. There is information about the structure, rules and copyright of the wiki here.
Content checked by qualified Clinical Psychologist on 21/01/2018
Last modified on 21/01/2018
a.) Feasible  Testing 400 people in 2 years is tough
b.) Sounds good  50 is a nice round number
c.) Is similar to what other people have done  If they got it published somewhere, anywhere, it's good enough for you.
A rough guide to samples (where you know the effect size) is here:
http://rgs.usu.edu/irb/wpcontent/uploa ... is_USU.pdf
Links to power calculators on the web:
http://powerandsamplesize.com/Calculators/
G*Power3  a very user friendly power calculator (honest!)
If you're going to be all scientific about it, you need to know a few things from the literature. Assuming it's the difference in mean scores between two groups you need to know:
* The mean for each group
* A pooled standard deviation (you can take the mean if they're similar, if they're not you should run multiple power analyses and present them all)
Now, one option is to turn to a single paper and go from there. If you're feeling more inspired you can draw up a spreadsheet with the means, S.D's, and sample sizes drawn from the literature. You then create a weighted mean and S.D. in excel. Here's a far too complex explanation on how to do that: http://en.wikipedia.org/wiki/Weighted_mean
Put simply, take an average like you normally would (sum of scores divided by number of measurements), and multiply them by the proportion of the total number of measurements which they represent.
So, if we have three studies:
N=10, mean = 23.5
N=80, mean = 26.4
N=10, mean = 12.5
Then we get the weighted mean by (0.1 x 23.5) + (0.8 x 26.4) + (0.1 x 12.5), which give us 24.72. The point of this is that we've assigned weights to the size of the studies. In small, fragmented literatures, this is essential. In larger, more mature literatures, less so. You might even be able to get hold of a metaanalysis or a review to figure out your sample size.
Anyway, what next? Well, my advice is go see a statistician, or find a copy of NQuery. If its the former, find a German. They're good. If the latter, use the wizards and provided you've collected the information above, you should muddle through. Those settings you're looking for by the way are power of 0.8 (80%) and alpha of 0.05 (i.e p=0.05).
You should write up your analysis along these lines:
"Based on the previous literature, we would expect group x to have a mean score of 5 and group y to have a mean score of 10. With a shared standard deviation of 2.5, 25 participants per group would be needed to detect a difference in means with 80% power and an alpha value set at 0.05"
If the S.D's look very different between the two populations, do a worst case/best case and present both.
If you're doing a million zillion comparisons, make the alpha 0.01. If the effect is massive but you don't want to look lazy testing 6 people, make the power higher (like 95%).
If you want the sample size to come out smaller, reduce the standard deviation, even by a small amount.
Also PLEASE remember. You'll be doing the power analysis on known effects. If your study is using any experimental measures that haven't been deployed in that population before, assume you'll need double the participants that you'll need for a known effect.
Note: If you have a suggestion about how to improve or add to this wiki please post it here. If you want to discuss this post please post a new thread in the forum. There is information about the structure, rules and copyright of the wiki here.
Content checked by qualified Clinical Psychologist on 21/01/2018
Last modified on 21/01/2018
Bluffer's guide to power analysis (and weblinks)
If you are putting together a grant, research proposal, or course project, there are three pragmatic characteristics of sample sizes which should be satisfied.
a.) Feasible  Testing 400 people in 2 years is tough
b.) Sounds good  50 is a nice round number
c.) Is similar to what other people have done  If they got it published somewhere, anywhere, it's good enough for you.
A rough guide to samples (where you know the effect size) is here:
https://usermanual.wiki/Pdf/AResearcher ... 089975.pdf
Links to power calculators on the web:
http://powerandsamplesize.com/Calculators/
G*Power3  a very user friendly power calculator (honest!)
If you're going to be all scientific about it, you need to know a few things from the literature. Assuming it's the difference in mean scores between two groups you need to know:
* The mean for each group
* A pooled standard deviation (you can take the mean if they're similar, if they're not you should run multiple power analyses and present them all)
Now, one option is to turn to a single paper and go from there. If you're feeling more inspired you can draw up a spreadsheet with the means, S.D's, and sample sizes drawn from the literature. You then create a weighted mean and S.D. in excel. Here's a far too complex explanation on how to do that: http://en.wikipedia.org/wiki/Weighted_mean
Put simply, take an average like you normally would (sum of scores divided by number of measurements), and multiply them by the proportion of the total number of measurements which they represent.
So, if we have three studies:
N=10, mean = 23.5
N=80, mean = 26.4
N=10, mean = 12.5
Then we get the weighted mean by (0.1 x 23.5) + (0.8 x 26.4) + (0.1 x 12.5), which give us 24.72. The point of this is that we've assigned weights to the size of the studies. In small, fragmented literatures, this is essential. In larger, more mature literatures, less so. You might even be able to get hold of a metaanalysis or a review to figure out your sample size.
Anyway, what next? Well, my advice is go see a statistician, or find a copy of NQuery. If its the former, find a German. They're good. If the latter, use the wizards and provided you've collected the information above, you should muddle through. Those settings you're looking for by the way are power of 0.8 (80%) and alpha of 0.05 (i.e p=0.05).
You should write up your analysis along these lines:
"Based on the previous literature, we would expect group x to have a mean score of 5 and group y to have a mean score of 10. With a shared standard deviation of 2.5, 25 participants per group would be needed to detect a difference in means with 80% power and an alpha value set at 0.05"
If the S.D's look very different between the two populations, do a worst case/best case and present both.
If you're doing a million zillion comparisons, make the alpha 0.01. If the effect is massive but you don't want to look lazy testing 6 people, make the power higher (like 95%).
If you want the sample size to come out smaller, reduce the standard deviation, even by a small amount.
Also PLEASE remember. You'll be doing the power analysis on known effects. If your study is using any experimental measures that haven't been deployed in that population before, assume you'll need double the participants that you'll need for a known effect.
Note: If you have a suggestion about how to improve or add to this wiki please post it here. If you want to discuss this post please post a new thread in the forum. There is information about the structure, rules and copyright of the wiki here.
Content checked on 14/09/2022
Last modified on 14/09/2022
a.) Feasible  Testing 400 people in 2 years is tough
b.) Sounds good  50 is a nice round number
c.) Is similar to what other people have done  If they got it published somewhere, anywhere, it's good enough for you.
A rough guide to samples (where you know the effect size) is here:
https://usermanual.wiki/Pdf/AResearcher ... 089975.pdf
Links to power calculators on the web:
http://powerandsamplesize.com/Calculators/
G*Power3  a very user friendly power calculator (honest!)
If you're going to be all scientific about it, you need to know a few things from the literature. Assuming it's the difference in mean scores between two groups you need to know:
* The mean for each group
* A pooled standard deviation (you can take the mean if they're similar, if they're not you should run multiple power analyses and present them all)
Now, one option is to turn to a single paper and go from there. If you're feeling more inspired you can draw up a spreadsheet with the means, S.D's, and sample sizes drawn from the literature. You then create a weighted mean and S.D. in excel. Here's a far too complex explanation on how to do that: http://en.wikipedia.org/wiki/Weighted_mean
Put simply, take an average like you normally would (sum of scores divided by number of measurements), and multiply them by the proportion of the total number of measurements which they represent.
So, if we have three studies:
N=10, mean = 23.5
N=80, mean = 26.4
N=10, mean = 12.5
Then we get the weighted mean by (0.1 x 23.5) + (0.8 x 26.4) + (0.1 x 12.5), which give us 24.72. The point of this is that we've assigned weights to the size of the studies. In small, fragmented literatures, this is essential. In larger, more mature literatures, less so. You might even be able to get hold of a metaanalysis or a review to figure out your sample size.
Anyway, what next? Well, my advice is go see a statistician, or find a copy of NQuery. If its the former, find a German. They're good. If the latter, use the wizards and provided you've collected the information above, you should muddle through. Those settings you're looking for by the way are power of 0.8 (80%) and alpha of 0.05 (i.e p=0.05).
You should write up your analysis along these lines:
"Based on the previous literature, we would expect group x to have a mean score of 5 and group y to have a mean score of 10. With a shared standard deviation of 2.5, 25 participants per group would be needed to detect a difference in means with 80% power and an alpha value set at 0.05"
If the S.D's look very different between the two populations, do a worst case/best case and present both.
If you're doing a million zillion comparisons, make the alpha 0.01. If the effect is massive but you don't want to look lazy testing 6 people, make the power higher (like 95%).
If you want the sample size to come out smaller, reduce the standard deviation, even by a small amount.
Also PLEASE remember. You'll be doing the power analysis on known effects. If your study is using any experimental measures that haven't been deployed in that population before, assume you'll need double the participants that you'll need for a known effect.
Note: If you have a suggestion about how to improve or add to this wiki please post it here. If you want to discuss this post please post a new thread in the forum. There is information about the structure, rules and copyright of the wiki here.
Content checked on 14/09/2022
Last modified on 14/09/2022