Please post your responses to the readings in this forum Reading responses are due on the Thursday evening before class (please aim for submitting by 9pm). For each reading there will be a new topic thread, your responses should be posted as a reply within the topic thread. Reading responses are required but you can skip one week. The week you skip, please announce that in the forum so I know not to expect it. Thank you.
Hi Maria,
Do you know when you are going to post Quiz 3 and the instructions for this week's assignment? With the end of the quarter and finals week approaching, I am trying to schedule my time accordingly.
Thanks!
Teresa
Hi Maria -- I'm wondering if you can post the answer keys for the in class quizzes. I see them for lecture 2 and lecture 3 quizzes, but not for the rest (unless I'm missing them somewhere?). Thank you!
Sarah
Paper Title: Effects of delayed compared with early umbilical cord clamping on maternal postpartum hemorrhage and cord blood gas sampling: a randomized trial
https://www.ncbi.nlm.nih.gov/pubmed/22913332
Brief Topic background:
Delayed cord clamping (DCC) is when you delay the clamping of the umbilical cord after birth typically by at least 60 seconds. The idea is that during this delay you are allowing blood to continue to flow from the placenta to the infant. DCC has been shown to decrease mortality, necrotizing enterocolitis, hypoxic ischemic encephalitis, and increase hematocrit, blood volume, and iron stores. There used to be concern that DCC increases the risk of maternal blood loss during delivery and how it affects umbilical blood gas values. For the sake of this assignment I will focus on just the maternal blood loss.
Brief Study background:
This study was a secondary analysis of an RCT (parallel-group study with 1:1 randomization) with the primary objective to investigate whether iron stores in term-born infants differ at four months of age as a result of DCC compared with early cord clamping (ECC). In this CURRENT study, they are evaluating the effects of DCC on post-partum hemorrhage and cord blood gas sampling.
Describe:
1) What was the exposure and outcome being evaluated?
Exposure:
Control Group = Early cord clamping (ECC) which is clamping at < 10 s
Treatment Group = DCC which was clamping at >=180 s
Outcomes:
Post partum hemorrhage and umbilical artery blood gas
2) What was the adherence to randomly assigned treatment (and how was it measured)?
Overall the adherence was ~85% in both groups. They simply timed how long until the cord was clamped so they could easily tell who did and didn’t have appropriate clamping times. Here is a snap shot from their flow diagram that nicely details who received the intended treatment and who didn’t.
3) What was the primary intent-to-treat effect estimate?
Maternal blood loss is difficult measure directly so its commonly reported categorically as either >500 or 1000 mL of blood loss. For comparison between the proportions they used Chi-square and they reported a difference of proportions of 1.2(-6.5 to 8.8).
4) Did they report an IV effect estimate?
no
5) Would an IV effect estimate have been of interest in this study?
I don’t think so given this was an RCT and they already performed an ITT analysis. I’m also not too concerned about there being any unmeasured confounders biasing their ITT results. If there was concern for that type of bias then perhaps I could imaging an IV analysis being of interest.
6) If so, do you think the IV estimate would be of more interest than the ITT estimate? Why/why not?
n/a. They did present the ITT estimate which I believe was appropriate. Had they presented an “As-treated” analysis it would have potentially messed up the randomization.
7) Can you calculate the IV effect estimate based on the information provided? If so, what is it? If not, why not?
No, I don’t think they provide any information about a possible IV. However, I can easily imagine one for a potential study. Because DCC has become the standard of care so I could imagine using policy change as an IV. For example, you could do a historical comparison study at an institution. At my hospital we began performing >60s of DCC in 2011 and then >180s in 2017. We could use that change in policy as an IV I think.
Thank you!Matthew
I read the VanderWeele mediation tutorial and I find the difference method very easy to understand. Where I am not as clear is with the product method and I am not appreciating the difference between these approaches vs the "natural direct and indirect effects". I have been looking at other papers in the area and all I have appreciated so far is that the latter is based in the counterfactual, but I still am not appreciating how this differs from the simple difference method.
if anyone has any helpful "dumbed" down references that you can share I would really appreciate hearing about them.
thank you!
I am skipping this week (because I'll be out of town).
Mendola, P. et al. Controlled direct effects of preeclampsia on neonatal health after accounting for mediation by preterm birth. Epidemiology 26, 17–26 (2015).
What is the primary discipline of the authors?
Epidemiology and OB-GYN
Draw a DAG representing the implicit or explicit causal model explored in this paper (you do not need to post your DAG, but we will try to discuss in class).
What is the exposure of interest?
Preeclampsia
What is the outcome of interest?
Neonatal complications studied included:
- perinatal mortality ≥23 weeks of gestation
- small for gestational age
- NICU admission
- respiratory distress syndrome
- transient tachypnea of the newborn
- anemia
- apnea
- asphyxia
- peri- or intraventricular hemorrhage
- cardiomyopathy
What is the hypothesized mediator of interest and how is it measured?
Preterm birth – estimates of gestational age were obtained from medical records
Describe the modeling approach and briefly report the estimated total, direct, and indirect effects (if these are reported).
Total effects of preeclampsia on neonatal outcomes were estimated using logistic regression with generalized estimating equations adjusted for exposure-outcome confounders.
To estimate controlled direct effects, the authors used marginal structural models with stabilized inverse probability weights. They used weighted logistic regression with GEE to estimate the parameters of the model. Two sets of weights were estimated – one for dichotomous preeclampsia status (with exposure-outcome and exposure-mediator confounders) and one for the categories of pre-term status (additionally included mediator-outcome confounders that could be caused by preeclampsia).
Total effects for the many outcomes ranged from 1.6-4.2. Direct effects ranged from 1.5-3.2, although for cardiomyopathy and anemia, the 95% CIs included 1.
If the direct effect is reported, would you describe this as a natural direct effect, a controlled direct effect, or something else?
Controlled direct effects setting gestational age to >=37 weeks
Do you think there is potential measurement error in the mediator and how would that affect the results?
It is possible there is measurement error if the MD estimated the gestational age incorrectly or it was recorded incorrectly. If such error was non-differential, I believe the indirect path through pre-eclampsia wouldn’t be fully blocked, thus the indirect effect would be underestimated. This would overestimate the direct effect. It is also possible such error could be differentially higher among those with preeclampsia, but I’m not sure which direction the error would go or how it would affect the estimation of the CDE.
Do you think there are unmeasured confounders of the mediator-outcome association and how would that affect the results of the mediation analysis?
The authors were concerned about lack of data on maternal infection (potential mediator-outcome confounder). They did an extensive sensitivity analyses and found it difficult to plausibly explain the magnitudes of the observed associations.
Do you have any critiques of the paper?
It would’ve been helpful to see absolute increases in risk rather than just ORs, especially as some outcomes, such as NICU admission, were quite prevalent.
Exposure: depression, anxiety, and stress in after HIV-seroconversion
Outcome: time to cognitive decline among adults with HIV
Lifecourse model: Social trajectories as well as accumulation of risk, whereby risks are primarily mediated by a succession of harmful social exposures. This aligns with the theory of syndemics, which is increasingly being applied to HIV research. Using this framework, I would hypothesize that levels of depression, anxiety, and/or stress in combination with new HIV infection may increase the likelihood of earlier cognitive decline through multiple biological and psychosocial pathways.
Regression model: I think that a model for time-to-decline incorporating interactions between co-occurring psychosocial and structural problems to represent accumulation of risk from stress and/or long term exposure to material adversity would be appropriate. Literature on syndemics suggests that interactions should be studied using models that suggest synergy, rather than additive associations with the outcome. A 'fully saturated linear regression model could be appropriate to capture all possible interactions between variables (Tsai,. & Venkataramani, 2016). Mediation analysis might also be helpful.
Y = α + β1X1 + β2X2 + β3X3 + β4X1X2 + β5X1X3 + β6X2X3 + β7X1X2X3
Data set: Multicenter AIDS Cohort Study (MACS) public data set. This public dataset has been released from CAMACS (Center for Analysis and Management of Multicenter AIDS Cohort Study) since 1994. The data comprise baseline and 6 month follow-up interview data, including medical history, behavior, SF36, physical examination data, frailty measurements, neuropsychology tests, and concurrent laboratory test results, and summary files of HIV status and medical events.
Concerns about interpretation:
One concern would be that people who have a history of depression/anxiety/stress would be more likely to seek medical care and therefore more likely be diagnosed with cognitive impairment at earlier stages (aka selection bias).
I would also be concerned about cohort effects due to the different types of ART medications that individuals may have been exposed to (with increasing advances in medication safety and tolerability over time).
Additionally, based on my understanding of the current clinical literature, it may take many years (even decades) for depression to manifest in and/or increase risk for dementias in the general population. Therefore, I would be hesitant to make definitive statements from data that did not follow participants for a substantial length of time.
Reference:
Tsai, A. C., & Venkataramani, A. S. (2016). Syndemics and health disparities: A methodological note. AIDS and Behavior, 20(2), 423-430. doi:10.1007/s10461-015-1260-2Setting: Resource-limited settings
Exposure: Repeated gastrointestinal (GI) infections throughout childhood
Outcome: Adult educational attainment
Hypothesis: Increased number of GI infections in early childhood (<2 years of age) is associated with lower overall educational attainment in adulthood.
Lifecourse model: Accumulation of risk. While early childhood could be thought of as a “critical period”, based on the conceptual model of childhood GI infections, the potential harm is thought to increase with additional infections and there is a self-perpetuating feedback loop that encourages repeat infections (see attached figure for conceptual model). The potential harm due to one GI infection in early childhood is also not thought to be irreversible, and thus contributing to future health or socioeconomic outcomes, which would be consistent with a “critical period” model. My interest is in early childhood exposures to GI infections, infections that children are particularly vulnerable to; therefore, an adult mobility/infection model would also not be appropriate.
Regression model: As described by Mishra, et al., I could estimate the direct an cumulative causal effect GI infection by fitting a linear regression model where:
Y=a + B lifetime GI infection score + B covariates
Where B = the change in the cumulative effect divided by the number of measurements, and the model is adjusted for confounders such as demographic (sex, village), socioeconomic factors (parents’ educational attainment, number of children <5 in the home, access to clean water, type of toilet in the home, parents’ occupations, etc.), and child comorbidities (prematurity, chronic GI disease, HIV, vaccination status, etc.). GI infection would need to be a binary exposure in each time interval. For example, if measurements are made every month, GI infection =0 corresponds to no GI infection in the last month, GI infection=1 corresponds to at least one GI infection in the last month.
The ideal dataset would be from a birth study cohort. Fortunately, one exists. The Fogarty Institute conducted a five-year multi-site birth cohort study to investigate associations between malnutrition and GI infections and their effects on children in resource-limited settings called the ‘Etiology, Risk Factors and Interactions of Enteric Infections and Malnutrition and the Consequences for Child Health and Development (MAL-ED)’ study. While this study collected detailed data over the first 5 years of life in these children, ongoing data collection throughout childhood and adulthood are possible and could potentially (eventually) answer my research question.
Concerns and limitations: Given the multisite nature of this study and that it was conduct in a resource-limited setting, measurement error between sites and missing data are potential issues. Unmeasured/residual confounding is also a concern; the relationship between early childhood GI infection and adult educational attainment is highly complex, not completely understood, and influenced by social and biological factors, meaning that other events not measured between time points may be contributing but not accounted for in this model. There could also be a survival bias; diarrheal disease is a major cause of mortality in children <5 years in resource-limited settings and in studying adult educational attainment, I will only be evaluating the outcome in subjects that survived, potentially those with less severe disease. The net result of these limitations is a potentially biased estimate.
Please post your reading responses to week 4 in response to this thread.
Please post your reading responses to week 3 in response to this thread.
Thank you.
In this paper they were assessing the Neonatal outcomes in preterm multiples receiving layed cord clamping. They were measuring to see if there was any difference in various key delivery room measures and clinical outcomes. They used a hierarchy model with clustering around the pregnancy for mothers. They used both a mixed effect (ME) model and a generalized estimating equations (GEE). Continuous variables were analyzed using ME with exchangeable covariance structures and restricted maximum likelihoods. Categorical variables were analyzed using generalized estimating equations.
File size is too large to fit for some reason.
Name of Paper: Neonatal outcomes in preterm multiples receiving delayed cord clamping,
Jegatheesan P, et al. Arch Dis Child Fetal Neonatal Ed 2019;0:F1–F7. doi:10.1136/archdischild-2018-316479
1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
Statistical conclusion validity: At my institution I found that people would frequently apply various statistical techniques without validating assumptions. For example we wanted to compare gestational age of newborn infants between year epochs. Originally they simply applied a one-way anova. However, they should have first checked for sample size, distributions, and variance to decide which test to use.
Internal validity: I’ve encountered an instrumentation threat in which we were comparing our hematocrit value today compared to those described in the past. However, method of used to calculate hematocrit and should likely be taken into account.
Construct validity: I’ve encountered a construct confounding threat during one of our studies where we wanted to evaluate how long mother’s breast feeding after NICU discharge. Mothers were either discharged with formula fortification or with breast milk with supplemental formula. Initially it appeared as though one was favored over the other but this difference may disappear after we adjust how much mothers were breastfeeding prior to discharge. Still an ongoing investigation so may more to tell =]
External validity: We constantly deal with this threat at our institution which is predominately a low-income community hospital that is a predominately Hispanic population. This often ends up as a limitation for all our studies because it may limit the generalizability especially if it’s a topic with known racial influences.
2) For any data set you frequently use, look up the sample design and describe it.
My team utilizes and prospectively maintained database of all NICU infants. Our studies utilize all infants that meet those particular inclusions so I guess you would call it a convenience sample.
getting bogged down trying to open the dat file on a PC. I first had to download a free trial of winzip to open the gz downloaded from IPUM
perhaps this troubleshooting is part of the exercise? if so, I will keep working on it.
If not, maybe you have suggestions as I am not yet able to see the data to work on the questions.
thanks
1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
My example involves a cross-sectional study of Ebola virus disease survivors in the Democratic Republic of the Congo (DRC). We aimed to understand the long-term clinical and psychosocial sequelae of Ebola virus disease (EVD) by comparing survivors against contacts. The study population was a cohort of survivors from an Ebola outbreak that occurred more than 20 years ago (1995).
Statistical conclusion validity: The small sample size has been a major statistical threat. For the psychosocial sequelae, we used continuous outcomes and this may have partially been why we were able to detect a difference between survivor and contact groups. In contrast with clinical sequelae, we used binary outcomes and found an association with some health outcomes such as uveitis assuming a p-value cutoff of 0.05. However, we were also multiple hypothesis testing so if we were to adjust the alpha, we would not consider any of the findings statistically significant. As a result, our conclusions must be tempered.
Internal validity: Working in a place like DRC created challenges that threaten internal validity. The country language is French and there are many local languages, so we had to rely on translation and training to ensure that there was high quality data collection. There were several interviewers collecting data and while there were attempts of quality control, it is possible that quality of data collection varied by interviewer.
Construct validity: Some of the outcomes such as depressive and anxiety symptoms as well as stigma were social measurements that had potential threats to their construct validity. Ebola stigma is an example because there are several psychometric domains to the social process and we didn't ask questions that captured certain domains of stigma such as internalized stigma.
External validity: Studying a cohort of Ebola survivors more than 20 years after an Ebola outbreak is very specific and doesn't lend itself well to be generalizable to Ebola survivors from other outbreaks. However, the study does give a sense of what life could be like for Ebola survivors of recent outbreaks in West Africa and DRC.
2) For any data set you frequently use, look up the sample design and describe it.
In addition to this DRC study of Ebola survivors, I study the natural history of asymptomatic or unrecognized, symptomatic EVD in Liberia. This dataset is one that I will be frequently using. It is an observational, longitudinal cohort of Ebola survivors and contacts. The study enrolled 1134 Ebola survivors and 2400 contacts. Among this cohort, I was able to identify contacts who may have an asymptomatic or unrecognized, symptomatic Ebola virus infection. All of these participants are being followed for a 5-year period and attend study visits every 6 months at which time they undergo a questionnaire, clinical exam, and blood draw. These data will be used in longitudinal analyses to test hypotheses about the transmission and sequelae of asymptomatic or unrecognized EVD.
Please post your week 2 reading response in reply to this post, so they are all in one thread. Thank you.
1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
Statistical conclusion validity:
Shadish, Cook and Campbell argue that a conclusion about covariation may be inaccurate if either variable is measured unreliably. In my data project examining the effect of angiotensin converting enzyme inhibitors (ACE-I) or angiotensin II receptor blocker (ARB) on further episodes of hospitalized acute kidney injury, there is measurement error of the exposure. Patients self-reported medications at the baseline study visit and this was not confirmed with electronic health record or prescription data. There is also likely measurement error of the outcome given that there are multiple different definitions of acute kidney injury. One definition is a 50% relative or 0.3 mg/dl absolute increase in creatinine from outpatient baseline to peak inpatient vs. a 50% relative change from minimum inpatient to maximum inpatient creatinine. The former definition is likely very sensitive and may be capturing cases of chronic kidney disease progression.
Internal validity:
One of the threats to internal validity is selection. The authors explain that at the start of an experiment, the average person receiving one experimental condition already differs from the average person receiving another condition, and that this difference might account for any result observed after the experiment ends. This is a concern in my data project where physicians are more likely to prescribe ACE-I/ARB to healthier patients. Therefore, the improved outcomes seen with these medications in AKI survivors may be due to healthy user bias.
Construct validity:
A threat to construct validity is inadequate explication of constructs where the construct is identified at too specific of a level. In my data project, the underlying question is how ACE-I or ARB medications after the risk of acute kidney injury. However, one often needs frequent (daily) creatinine measurements to identify acute kidney injury. Given that it would be very difficult to do this for outpatients, my data project is on the risk of hospitalized acute kidney injury and ACE-I or ARB use. Patients who are admitted to hospital typically have daily bloodwork including creatinine measurements. It is likely that participants in ASSESS-AKI are experiencing subclinical, undetected acute kidney injury in the community, but this data is not captured.
External validity:
Interaction of the causal relationship over treatment variations is a threat to external validity. In my data project, any use of ACE-I or ARB at the baseline study visit was classified as ACE-I/ARB use. However, there are many different brands of ACE-I and ARB medications with varying doses. Furthermore, the ASSESS-AKI study population may have been using less nephrotoxic medications such as diuretics or non-steroid anti-inflammatory drugs which are known to increase the risk of acute kidney injury. These results may therefore not translate to a different population, for instance those with heart failure where diuretic use is common. Combining diuretics with ACE-I or ARB in a heart failure population may lead to more acute kidney injury events, in comparison to my data project which showed decrease acute kidney injury with ACE-I or ARB use.
2) For any data set you frequently use, look up the sample design and describe it.
ASSESS-AKI is a parallel, matched, prospective cohort design of adult participants with and without acute kidney injury. Adult patients with acute kidney injury were identified during the index hospitalization and screened for initial eligibility. Hospitalized adult patients who did not appear to suffer an acute kidney injury episode (controls) were matched in a 1:1 acute kidney injury:non-acute kidney injury ratio, with each non-acute kidney injury subject individually matched to their corresponding acute kidney injury subject on center and presence of baseline chronic kidney (eGFR <60 ml/min/1.73 m2).
For my data project, I used the unmatched cohort and stratified the cohort by the presence or absence of acute kidney injury at the index hospitalization. There was no clustering or weighting of observations.
1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
Statistical conclusion validity: Work with African American men with prostate cancer under active surveillance. Recruiting an underrepresented group with at a very specific stage of their illness has resulted in low Ns for the study. While a power analysis can provide an ideal target N for sufficient power reaching that goal is another hurdle. In addition, the treatment under study is exercise, which is done at home by the participants. While there is a standardize treatment and follow up (calls, Fitbit tracking) to ensure compliance issues of unreliability of treatment implementation can still arise. As the treatment is not conducted with in a lab setting the participants may not be executing the treatment as recommended.
Internal validity: Selection of participants is a treat to internal validity within this study as the population is extremely difficult to recruit. Thus the ones that are recruited were able to navigate the barriers that it difficult for African American men to participate within research studies. While it is not possible to know all the factors that differ as we do not know what we do not have some factors such as being within the UCSF medical system or having a doctor that is familiar with the study are factors that can be controlled for.
Construct: In a study examining impact of stereotype threat of Black identity the both treatment and outcome were measured the same way for all participants, depending on survey responses. Creating a treat to construct validity due to monomethod bias. While measures were validated and checking not to be leading a difference was observed when the study was administered by an interviewer instead of a survey.
External validity: With the previous study we examined the effect in two extreme Black students in psychology and African studies. These were the target populations for the study but those recruited into the study differ from their peers as the incentive was extra credit so they might have wanted the points or needed them. In addition, Black students that do not sit in either of these extremes such as those in criminal justice which is not culturally affirming but has a high representation of Black students may not see the same effect. This introduces interaction of causal relationship with units threat to external validity similar we can also see threat of causal relationship with setting.
2) For any data set you frequently use, look up the sample design and describe it.
Currently working with TRUEnth data N=218, men with prostate cancer at any stage, age from multiple recruitment sites around the US. Men need to have access to internet to do surveys and phone for one time contact with coordinator for on boarding into survey platform. Surveys measure dietary, exercise habits plus other lifestyle factors in additional to prostate cancer diagnosis for baselines measures. After which tailored diet and exercise plans are made for a 12-week course in a cohort study.
1) Examples of threats to validity:
Internal validity: In a cross-sectional data project, I face the challenge of ambiguous temporal precedence. I cannot determine clearly whether my exposure (experience with a behavior) precedes my outcome (attitudes about the legalization of that behavior). Therefore, confounding by previous attitudes about legalization of that behavior is a challenge in causal inference.
External validity: An RCT of different pain treatment for medication abortion in Nepal, Vietnam, and South Africa may not be generalizable to other country contexts.
Statistical conclusion validity:
· Unreliability of measures is a common challenge faced in research on family planning because the measures are often about sexual behavior and therefore associated with stigma and shame or embarrassment. For example, overreporting of sexual encounters per month and underreporting of STI symptoms.
· Often we collect primary data rather than use data available in large databases. Therefore lack of power can sometimes be a challenge to statistical conclusion validity, particularly if we fail to enroll the target number of participants to power the study due to financial, logistical, or other challenges in recruitment. This will cause effect size estimates to be less precise and lead to incorrect conclusion that there is no effect.
Construct validity:· Due to the lack of data availability on abortion and family planning, data I work with are often self-reported in surveys by participants. Therefore, I face the challenge of mono-method bias, where all operationalizations use the same method (self-report) and that method is therefore part of the construct studied.
· Another example of a threat to construct validity that pertains to a study I worked on is “treatment diffusion” – where participants may receive services from a condition to which they were not assigned, making construct descriptions of both conditions more difficult. In a 3-arm RCT to investigate the effects of pain medications (ibuprofen, tramadol, placebo) on medication abortion, some patients did not take pain medications they were given because of a cultural expectation that pain meds were not necessary while other participants who were not assigned pain meds sometimes sought them at pharmacies to deal with pain they felt during the procedure. Alternatively or additionally, participants receiving the placebo may have placebo effects (reactivity to the experimental situation), or participants who felt significant pain during the procedure may not report equivalently high pain reports for fear of being seen as weak or unable to handle the pain.
2) Sampling frame
National Survey of Family Growth (NSFG) –
Second-stage: selection of neighborhoods defined by census blocks
Third-stage: selection of housing units (interviewers updated commercially-available lists of units or created lists from scratch; interviewers contacted selected units to determine if any members of household are eligible)
Fourth-stage: selection of persons within households – one eligible person per household
A second-phase sample was drawn during the field period to address nonresponse.
Laura Koth
HW1 Epi Methods III
1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
Statistical conclusion validity: My technician is currently performing data analysis on a set of data obtained using a method called flow cytometry. In this method, the possibility for measurement error is great due to processing and analyzing samples in batches on different days using an instrument that can vary in terms of laser intensity from day to day. Therefore, we have to constantly be paying attention to all of our controls we use with each batch of experiments ranging from various beads we run, to the same internal reference controls, and despite all these attempts, we still found that we are getting a range of data that is making it challenging to have confidence in finding a statistically significant difference between our disease and control patient samples. Another problem I face is that fact that the effect size of the targets I am measuring may be small in magnitude (even though that can translate into important clinical outcomes) and do this measurement error, in addition to the small sample size I have, makes it difficult if not impossible to confidently answer the question of interest.
Internal validity: One of the big problems I face in my studies is that the cause of the disease I study is completely unknown. Therefore, the fundamental premise of observing exposure A which led to disease B is not possible for my hypotheses. For example, interferon gamma is a cytokine which is associated and thought to cause the inflammation in sarcoidosis, but perhaps it is not causing the inflammation but is a result of whatever is causing the sarcoidosis.
Construct validity: I also collect surveys completed by patients that intend to capture symptom experiences. I think this type of research has to deal with this concept of is the survey capturing what we really are hoping to study, for example symptom experiences related to fatigue, depression, or shortness of breath? I think that is what inadequate explication of constructs might be getting at?
External validity: this is a big problem for my studies in my cohort because of the issues with selection bias, so that generalizing findings from my cohort may not hold to other people with sarcoidosis who live in different parts of the country or world.
2) For any data set you frequently use, look up the sample design and describe it. My data set consists of patient reported surveys and clinical data collected from a group of patients with pulmonary and systemic sarcoidosis. My cohort was developed through recruitment using online ads and ads in local clinics around the Bay Area. Thus, my data set suffers from selection bias because it required interested participants to reach out to us. We did not randomly call or send letters to the population in the Bay Area. Because I did not send surveys to representative populations as described in Korn et al. it is not clear to me after reading the Korn paper whether any of these concepts would apply to the type of data set that I have. I actually am not sure if there is a word to describe the “sample design” for the data obtained from participants in my cohort. It isn’t a case control study. I did not match cases and controls either. Perhaps it is a prevalent disease sample design??
1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
A. Statistical conclusion validity
· Low statistical power – the power calculation is based on the estimated difference in prevalence in an outcome between two groups however little thought is given to how sampling might affect the assumed prevalence. For example a study of elder mistreatment which draws from general population and assumes a prevalence of 30% doesn’t consider that study participation rates might be related to elder mistreatment (families where elders are being mistreated are less likely to participate in research studies) so the actual prevalence in the study is much lower and so the power is much lower.
· Error rate – this feels like it is quite common in studies using neuropsychological tests as these tests often have many sub-tests and often results are presented for individual tests and I have seen as many as 25 tests on a sample of 200 people.
· Unreliability of treatment implementation – a study trying to evaluate whether or not a text message program to support chronic disease management found no effect between the intervention and control arm however the research team also found that a very large portion of the participants had not received the intervention at all.
B. Internal validity
· Attrition – a study looking at the use of text messaging to improve iron tablet intake in pregnant women in India had an attrition rate of over 40%. I have also seen differential attrition rates between intervention and control groups which are important and can’t be corrected for.
· Regression – in my research experience this is a frequent problem with measures of blood pressure which is why in most research protocols the first reading is disregarded and the mean of at least another two readings is used. I would like to think more about how regression and variability (e.g. seasonal variation in performance) interact
· Maturation – in a study of traumatic brain injury not restricting to people who had completed their education i.e. including children who couldn’t have completed high school in the group of people who didn’t complete high school and trying to assess the effect of education on traumatic brain injury outcome (the problem is extended if age of injury isn’t accounted for since having a severe head injury as a child may preclude educational attainment, perhaps this would fall under ambiguous temporal precedence.)
C. Construct validity
· Inadequate explication – this feels very common in reading literature of cognitive testing; there seems to be lots of variation in how people define constructs like “executive function.” Sometimes researchers will use the same tests but decide they measure different constructs
· Construct confounding – in research using occupational cohorts and using sex and self-reported or otherwise obtained ethnicity but not accounting for differential exposure rates due to sex and class differences in the type of work being done. For example using a sample of veterans and looking at race differences in brain injury which likely hide factors like rank and military service type which may influence likelihood of exposure to circumstances where brain injury occurs. This may also happen with cohort effect for example shifting from conscription to voluntary service.
· Novelty and disruption – we embedded a research study in a large tent in a primary care clinic. When we did a process evaluation of the intervention (text messages to support treatment adherence) a major theme that emerged was how much everyone loved coming to the tent. (Would we consider this similar to the Hawthorne effect?)
D. External validity
· Causal relationship with outcomes – we designed a brief messaging intervention to improve treatment adherence among people with high blood pressure. We were not sure whether such an intervention would work in people with other or multiple chronic conditions.
· Causal relationship with setting – the same study looked at participants at a single clinic site. We were also not sure whether the intervention might work in other health system settings (we designed a subsequent study to test this.)
2) For any data set you frequently use, look up the sample design and describe it.
Health ABC – population-based sample of 3075 older adults (70 to 79 years of age at enrolment. Participants were identified either through a random sample of Medicare beneficiaries (white), or all age-eligible community residents in designated zip code areas surrounding Pittsburgh, Pennsylvania, and Memphis, Tennessee (black.) Inclusion criteria included no difficulty in walking one-quarter mile or climbing 10 stairs without resting. Exclusion criteria included difficulties with activities of daily living, obvious cognitive impairment, inability to communicate with the interviewer, intention of moving within 3 years, or participation in a trial involving a lifestyle intervention.
1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
· Statistical conclusion validity (1): When studying rare diseases (such as Parkinson’s disease and rarer ones like dystonia, a movement disorder) low power is a chronic problem that permeates many observational and some interventional studies. Currently, one of my studies entails evaluation of pregnancy and delivery complications among women with two inherited causes of dystonia. The outcome of obstetric complications was obtained through a cross-sectional survey and 129 women reported at least one pregnancy (total of 310 pregnancies). While this is a very large number for women with inherited dystonia, it represents a small number of pregnancies overall to study number of complications. The results for both genetic causes together show a greater instance of complications among those mutation carriers, but when studied separately (further decreasing the sample size), while on the same direction, the confidence intervals become very wide (and the results are no longer significant) for one of the genetic forms studied.
· Statistical conclusion validity (2): Violation of the assumption of independently distributed errors à In our neurophysiology lab, we perform neuronal recordings from deep brain electrodes implanted during surgeries for Parkinson’s disease (and other movement disorders). These recordings from neurons are recorded for both sides of the brain for the same person. Statistical analyses are performed then to link specific brain recordings to measurements of movements (like rating scales performed by investigators) or measurements of behaviors/symptoms (via questionnaires). Recordings from one hemi-brain are usually analyzed independently, without taking into consideration the likely correlations between the recordings and outcomes from both sides of the brain from the same individual.
· Internal validity: I suspect that regression to the mean may explain some treatment effects in some of our observational studies of deep brain stimulation surgery for movement disorders. The subjects are selected into treatment because they have extreme values -otherwise, they would not qualify for surgical treatment-. Usually, one measurement is taken before surgery and one or several are taken postoperatively at determined time periods (such as 3, 6 and 12 months). It is possible that the extreme value (rating the severity of motor symptoms) would have naturally lowered to some degree naturally without the treatment intervention; this would be particularly important to note when there is no comparison group, such as in observational predictive studies.
· Construct validity: In a paper published in Brain in 2019, the authors tested the hypothesis that a reduction of brain connectivity in a particular area was associated with a higher risk for impulsive-compulsive behavior in Parkinson’s disease. Eighty subjects underwent functional MRI to evaluate brain connectivity as well as a questionnaire (QUIP-RS) to quantify the severity of impulsivity/compulsivity in Parkinson’s disease. In addition to likely problems of temporality and reverse causation (what was first, the lack of connectivity or the behavior, or are both the consequence of something else), the authors defined presence/absence of impulsivity/compulsivity based on a cut-off point in the questionnaire. I believe this may be an example of inadequate explanation of construct, since this may have been a too crude of a measure of complex behaviors in this population.
· External validity: Interaction of the causal relationship with units (The effect found with certain kinds of units might not hold if other kinds of units were used)à We are currently a site for two similar large multicenter randomized controlled trials for two similar and novel therapies aimed to slow down the progression of Parkinson’s disease (a progressive neurodegenerative disorder). The treatments in question are monthly infusions of two different types of antibodies against misfolded alpha-synuclein (a component of the pathological hallmark of this disease). The trials are recruiting only very recently diagnosed patients (symptoms <3 years) not yet on any symptomatic therapy. One important research question if these trials show a difference in progression for the treatment arm is to whether the findings would be generalizable to more advanced Parkinson’s disease patients.
2) For any data set you frequently use, look up the sample design and describe it.
The Parkinson’s Progression Markers Initiative (PPMI) is an observational, multi-center study funded by the Michael J Fox Foundation that actively collects clinical and imaging data, and biologic samples from several cohorts, that is available for download free to scientists, with the goal of establishing markers of disease progression in Parkinson’s disease.
The dataset contains several cohorts: a) De Novo PD patients: subjects with a diagnosis of PD for <=2 years who are not taking PD medications; b) Control subjects: people without PD at least 30 years of age who do not have any first degree relatives with PD; c) Prodromal subjects: subjects with PD who have a diagnosis of hyposmia (lack of sense of smell, a known prodromal factor for PD) or REM sleep behavior disorder (also a known predictor of PD); d) Genetic cohort: subjects with and without PD who have a genetic mutation in LRRK2, GBA or SNCA genes.
I am uncertain about the specific sampling of these cohorts. I believe subjects with PD and their spouses (without relatives with PD) were approached at the recruiting clinical centers and recruited for the studies. Anyone can request genetic testing (and counseling) through the PPMI website and be recruited for the genetic cohort, links to this study are also available through 23&me (direct to consumer genetic testing company). Subjects with hyposmia and REM sleep behavior disorder were approached and recruited through advertisement and by approaching sleep disorder clinics.
1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
Statistical conclusion validity: I performed a study on malnourished children in Bangladesh with sepsis and measured outcomes before and after implementation of an evidence-based sepsis protocol. Surprisingly, we did not find an improvement in mortality after protocol implementation, and there was evidence that other clinical outcomes – length of stay, fluid overload – worsened after implementation. There were two potential threats to statistical conclusion validity that may have contributed to this result. First, the study had a limited sample size ~300 children, and was powered to detect a difference in mortality of 18 percentage points, which is quite high, while a clinically significant mortality difference may be as low as 5 percentage points. The second threat was the unreliability of treatment implementation. Based on proxy measures of protocol compliance, such as antibiotic administration within one-hour, overall protocol compliance was poor.
Internal validity: Again, with the above study, we observed an increased number of sepsis cases post-protocol implementation. Sepsis was defined by provider diagnosis and, while protocol compliance was poor, implementation of a sepsis protocol may have increased providers’ awareness of sepsis, thus resulting in more sepsis diagnoses post and a systematic difference in patient characteristics pre vs. post. Likewise, we had no way to assess for the number of sepsis cases misdiagnosed or missed by providers; it is possible that more misdiagnoses occurred pre-protocol implementation and that the baseline mortality rate was actually higher pre compared to post. Both of these are an example of selection resulting in a threat to internal validity.
Construct validity: I study pediatric sepsis in resource-limited settings. Recently, the definition for sepsis changed for adults, but not children, and the current definition of pediatric sepsis, based on systemic inflammatory response syndrome (SIRS) criteria, has been extensively criticized for being too sensitive and not specific. In my studies in East Africa, I still use the current definition to define pediatric sepsis, which means I am likely including children with mild illness that do not have a life-threatening infection. This is an example of an inadequate explication of constructs, which means we may be drawing incorrect inferences about the relationship between pediatric sepsis, a potentially life threatening infection, and mortality.
External validity: I conducted another sepsis study in the (only) national tertiary care hospital in Tanzania and found that delayed presentation to care was associated with mortality. Results from this study cannot be extrapolated to district hospitals in Tanzania, which have a different referral pattern, in general a less sick patient population, and decreased availability of resources. This is an example of a potential interaction of the causal relationship with the setting.
2) For any data set you frequently use, look up the sample design and describe it.
The dataset I most frequently use is a Pediatric Sepsis Database from Tanzania. It includes ~2,000 children aged 28 days to 14 years who presented to Muhimbili National Hospital in Dar es Salaam, Tanzania, from July 1, 2016-June 30, 2017 with sepsis, as defined by clinical systemic inflammatory response syndrome (SIRS) criteria. This was a prospective cohort study that captured baseline characteristics/demographics, interventions received, lab values, functional status, and outcomes. We had research assistants available 24 hours a day, 7 days a week to screen, consent and perform data collection. All pediatric patients presenting to the emergency department were screened for inclusion criteria and then approached consent if appropriate. >90% of eligible patients were enrolled.
