I'm pasting everything below, but also attaching a word document that is formatted for clarity.
Specify a hypothesis regarding a particular exposure and outcome and a binary effect modifier including specific measures of association (specify the magnitudes of that association you anticipate: I suggest making everything cross-sectional). Using the software of your choice, generate a population with 1000 people under a causal structure consistent with this hypothesis. Draw a simple random sample 100 individuals from this population and estimate the population average exposure-outcome association and the association stratified by your modifier of interest within this subset. Repeat this 10 times and write the parameter estimates and CI each time.
Hypothesis: Poverty during childhood increases the probability of being obese/overweight in young adulthood (Hernandez and Pressler, 2014), but exposure to poverty increases risk of obesity more for females than for males.
Hernandez DC, Pressler E Accumulation of childhood poverty on young adult overweight or obese status: race/ethnicity and gender disparities J Epidemiol Community Health 2014;68:478-484.
Exposure: Household income (in $10,000) (average from years 2-10)
Outcome: BMI at age 20
Modifier: sex
Modeled effects:
BMI 24.73 (STD 5.13)
Effect in females: -0.25 BMI/$10,000
Effect in males: -0.17 BMI/$10,000
Difference (females = 1): -0.07
/////generating random exposure variable household income and played around to make plausible//////
. drawnorm house_income_norm, n(1000) means(16) sds(4) clear
(obs 1,000)
. generate house_income = house_income_norm - 2.7
. replace house_income = house_income/2.3
(1,000 real changes made)
////generating random error term for outcome BMI////
. generate error_BMI = rnormal(0,1)
////generating random bilevel mediator variable female/////
. generate female = 1
. generate randomsex = runiform()
. sort randomsex
. replace female = 0 in 501/1000
(500 real changes made)
/////generating outcome BMI using regression equation////
. generate BMI = 24.73 + error_BMI*3 - house_income*(0.25+0.07*female)
(1000 missing values generated)
. generate income_female = house_income*female
///running regression models to confirm that the sample reasonably approximates my input parameters and IT DOES!!///
. regress BMI house_income income_female
Source | SS df MS Number of obs = 1,000
-------------+---------------------------------- F(2, 997) = 13.80
Model | 224.315473 2 112.157737 Prob > F = 0.0000
Residual | 8101.32545 997 8.12570256 R-squared = 0.0269
-------------+---------------------------------- Adj R-squared = 0.0250
Total | 8325.64092 999 8.3339749 Root MSE = 2.8506
-------------------------------------------------------------------------------
BMI | Coef. Std. Err. t P>|t| [95% Conf. Interval]
--------------+----------------------------------------------------------------
house_income | -.1805483 .0553692 -3.26 0.001 -.2892018 -.0718947
income_female | -.0918186 .0297546 -3.09 0.002 -.1502075 -.0334297
_cons | 24.92822 .3230297 77.17 0.000 24.29433 25.56212
-------------------------------------------------------------------------------
. regress BMI house_income female income_female
Source | SS df MS Number of obs = 1,000
-------------+---------------------------------- F(3, 996) = 9.20
Model | 224.478177 3 74.8260591 Prob > F = 0.0000
Residual | 8101.16275 996 8.13369754 R-squared = 0.0270
-------------+---------------------------------- Adj R-squared = 0.0240
Total | 8325.64092 999 8.3339749 Root MSE = 2.852
-------------------------------------------------------------------------------
BMI | Coef. Std. Err. t P>|t| [95% Conf. Interval]
--------------+----------------------------------------------------------------
house_income | -.1869639 .0715987 -2.61 0.009 -.3274656 -.0464621
female | -.0920911 .6511219 -0.14 0.888 -1.369819 1.185637
income_female | -.0772236 .107401 -0.72 0.472 -.2879818 .1335346
_cons | 24.96872 .4317816 57.83 0.000 24.12141 25.81603
-------------------------------------------------------------------------------
/////running bootstrap option in stata for repeated sample size estimation/////
. bootstrap, reps(1000) size(100): regress BMI house_income female income_female
(running regress on estimation sample)
Bootstrap replications (1000)
----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5
.................................................. 50
.................................................. 100
.................................................. 150
.................................................. 200
.................................................. 250
.................................................. 300
.................................................. 350
.................................................. 400
.................................................. 450
.................................................. 500
.................................................. 550
.................................................. 600
.................................................. 650
.................................................. 700
.................................................. 750
.................................................. 800
.................................................. 850
.................................................. 900
.................................................. 950
.................................................. 1000
Linear regression Number of obs = 1,000
Replications = 1,000
Wald chi2(3) = 2.59
Prob > chi2 = 0.4587
R-squared = 0.0270
Adj R-squared = 0.0240
Root MSE = 2.8520
-------------------------------------------------------------------------------
| Observed Bootstrap Normal-based
BMI | Coef. Std. Err. z P>|z| [95% Conf. Interval]
--------------+----------------------------------------------------------------
house_income | -.1869639 .2423548 -0.77 0.440 -.6619705 .2880428
female | -.0920911 2.223167 -0.04 0.967 -4.449418 4.265236
income_female | -.0772236 .3665592 -0.21 0.833 -.7956664 .6412192
_cons | 24.96872 1.453286 17.18 0.000 22.12033 27.81711
-------------------------------------------------------------------------------
Here it would appear that the effect estimates are quite accurate after 1000 runs, but the confidence in the estimate is low with nothing achieving the level of statistical significance.
Repeat the data set construction, setting the causal effect to the null. Again repeat this 10 times and write the parameter estimate and CI each time (if you figure out how to automate it, run it 1000 times and post the histogram of the parameter estimates and p-values).
. replace BMI = 24.73 + error_BMI*3
(1,000 real changes made)
. bootstrap, reps(1000) size(100): regress BMI house_income female income_female
(running regress on estimation sample)
Bootstrap replications (1000)
----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5
.................................................. 50
.................................................. 100
.................................................. 150
.................................................. 200
.................................................. 250
.................................................. 300
.................................................. 350
.................................................. 400
.................................................. 450
.................................................. 500
.................................................. 550
.................................................. 600
.................................................. 650
.................................................. 700
.................................................. 750
.................................................. 800
.................................................. 850
.................................................. 900
.................................................. 950
.................................................. 1000
Linear regression Number of obs = 1,000
Replications = 1,000
Wald chi2(3) = 0.02
Prob > chi2 = 0.9990
R-squared = 0.0003
Adj R-squared = -0.0027
Root MSE = 2.8520
-------------------------------------------------------------------------------
| Observed Bootstrap Normal-based
BMI | Coef. Std. Err. z P>|z| [95% Conf. Interval]
--------------+----------------------------------------------------------------
house_income | -.0169639 .2395083 -0.07 0.944 -.4863915 .4524638
female | -.0920912 2.195122 -0.04 0.967 -4.394451 4.210268
income_female | .0027764 .3578001 0.01 0.994 -.6984988 .7040517
_cons | 24.96872 1.435228 17.40 0.000 22.15572 27.78172
-------------------------------------------------------------------------------
Everything is pretty much null with p>0.9 for all effect estimates
Use your code above and also a canned software command to estimate statistical power to detect the difference in means under the settings below:
*n=100, μ0=.02, μ1=.12, SD=1, α=.05
. power twomeans 0.02 0.12, n(100)
Estimated power for a two-sample means test
t test assuming sd1 = sd2 = sd
Ho: m2 = m1 versus Ha: m2 != m1
Study parameters:
alpha = 0.0500
N = 100
N per group = 50
delta = 0.1000
m1 = 0.0200
m2 = 0.1200
sd = 1.0000
Estimated power:
power = 0.0785
*n=100,μ0=.02, μ1=.12, SD=2, α=.05
. power twomeans 0.02 0.12, sd(2) n(100)
Estimated power for a two-sample means test
t test assuming sd1 = sd2 = sd
Ho: m2 = m1 versus Ha: m2 != m1
Study parameters:
alpha = 0.0500
N = 100
N per group = 50
delta = 0.1000
m1 = 0.0200
m2 = 0.1200
sd = 2.0000
Estimated power:
power = 0.0570
*n=500, μ0=.3, μ1=.3, SD=1, α=.05
. power twomeans 0.3 0.3, n(500)
the control-group mean and the experimental-group mean are equal; this is not allowed
r(198);
I will say that
For each of the 3 settings above, what is the power to detect whether the ratio of the means=1?
Well, as everything above is set up for detecting a ratio that is not null, the power to detect a null ratio is zero. If on the other hand you are running a test for equivalence rather than a test for difference, then there are alternative tests to run, I however couldn’t figure out how to perform this calculation, as everything I tried gave errors. I tried also using an online calculator:http://powerandsamplesize.com/Calculators/Compare-2-Proportions/2-Sample-Equivalence
n=100, μ0=.02, μ1=.12, SD=1, α=.05
ð power = 0.763
*n=100,μ0=.02, μ1=.12, SD=2, α=.05