Week 1

Week 1

by Sarah Raifman -

1) Examples of threats to validity:

Internal validity: In a cross-sectional data project, I face the challenge of ambiguous temporal precedence. I cannot determine clearly whether my exposure (experience with a behavior) precedes my outcome (attitudes about the legalization of that behavior). Therefore, confounding by previous attitudes about legalization of that behavior is a challenge in causal inference.  

External validity: An RCT of different pain treatment for medication abortion in Nepal, Vietnam, and South Africa may not be generalizable to other country contexts.

Statistical conclusion validity:

·       Unreliability of measures is a common challenge faced in research on family planning because the measures are often about sexual behavior and therefore associated with stigma and shame or embarrassment. For example, overreporting of sexual encounters per month and underreporting of STI symptoms.

·       Often we collect primary data rather than use data available in large databases. Therefore lack of power can sometimes be a challenge to statistical conclusion validity, particularly if we fail to enroll the target number of participants to power the study due to financial, logistical, or other challenges in recruitment. This will cause effect size estimates to be less precise and lead to incorrect conclusion that there is no effect.  

Construct validity:

·       Due to the lack of data availability on abortion and family planning, data I work with are often self-reported in surveys by participants. Therefore, I face the challenge of mono-method bias, where all operationalizations use the same method (self-report) and that method is therefore part of the construct studied.

·       Another example of a threat to construct validity that pertains to a study I worked on is “treatment diffusion” – where participants may receive services from a condition to which they were not assigned, making construct descriptions of both conditions more difficult. In a 3-arm RCT to investigate the effects of pain medications (ibuprofen, tramadol, placebo) on medication abortion, some patients did not take pain medications they were given because of a cultural expectation that pain meds were not necessary while other participants who were not assigned pain meds sometimes sought them at pharmacies to deal with pain they felt during the procedure. Alternatively or additionally, participants receiving the placebo may have placebo effects (reactivity to the experimental situation), or participants who felt significant pain during the procedure may not report equivalently high pain reports for fear of being seen as weak or unable to handle the pain.

2) Sampling frame 

National Survey of Family Growth (NSFG) –

  • Independent, national probability sample of women and men 15-44 years of age.
  • In-person face-to-face interviews conducted by professional female interviewers using laptops.
  • Sampling frame based on goal of completing a minimum of 5000 interviews per year with oversampling of non-hispanic blacks, Hispanics, teens, and females 
  • In series of 5 stages, geographically defined sampling units of decreasing size are selected with probability proportionate to size
First-stage selection of MSAs, counties, and county groups (50 states + DC divided into 2149 PSUs – select national sample of 110 PSUs, divide in to 4 nationally representative samples, then choose one each year without replacement. Each year, 5500 men/women interviewed)

Second-stage: selection of neighborhoods defined by census blocks

Third-stage: selection of housing units (interviewers updated commercially-available lists of units or created lists from scratch; interviewers contacted selected units to determine if any members of household are eligible)

Fourth-stage: selection of persons within households – one eligible person per household

A second-phase sample was drawn during the field period to address nonresponse.


Re: Week 1

by Maria Glymour -

Sarah:

Great examples.  I'd like to hear more about the 2nd phase of sampling in the Family Growth study used to address non-response.  This is a super useful approach that is conceptually parallel to typical sampling methods.

Maria


Re: Week 1

by Sarah Raifman -

Maria, 

Regarding the second phase... from what I have read, at the end of week 10 of each quarter of each year of the sampling, they draw a probability subsample of remaining nonrespondent cases. The sample is stratified by interviewer, screener vs main interview status, and expected propensity to provide an interview. Then they offer a different incentive and greater interview effort is given for these subselected cases. From doing this they found that outcomes for active main cases sampled into the second phase sample were better than those for screener cases -- so the sample is disproportionately allocated to main cases. The revised incentive used in phase 2 raised the propensities of the remaining cases, bringing into the respondent pool people who would have remained nonrespondent without the second phase... 

They also estimated models predicting response to screener interview and main interview and then developed nonresponse adjustments (created by forming deciles of estimates propensities from each model and using the inverse of the response rate within each decile as an adjustment weight). They multiplied these nonresponse adjustments by the probability of selection weight to obtain the final weight... apparently this led to a reduction in the variation of response rates for important subgroups. Reducing the variation of subgroup response rates helps reduce nonresponse bias.