Week 1 Reading

Week 1 Reading

by Teresa Kortz -

1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).

Statistical conclusion validity: I performed a study on malnourished children in Bangladesh with sepsis and measured outcomes before and after implementation of an evidence-based sepsis protocol. Surprisingly, we did not find an improvement in mortality after protocol implementation, and there was evidence that other clinical outcomes – length of stay, fluid overload – worsened after implementation. There were two potential threats to statistical conclusion validity that may have contributed to this result. First, the study had a limited sample size ~300 children, and was powered to detect a difference in mortality of 18 percentage points, which is quite high, while a clinically significant mortality difference may be as low as 5 percentage points. The second threat was the unreliability of treatment implementation. Based on proxy measures of protocol compliance, such as antibiotic administration within one-hour, overall protocol compliance was poor. 

Internal validity: Again, with the above study, we observed an increased number of sepsis cases post-protocol implementation. Sepsis was defined by provider diagnosis and, while protocol compliance was poor, implementation of a sepsis protocol may have increased providers’ awareness of sepsis, thus resulting in more sepsis diagnoses post and a systematic difference in patient characteristics pre vs. post. Likewise, we had no way to assess for the number of sepsis cases misdiagnosed or missed by providers; it is possible that more misdiagnoses occurred pre-protocol implementation and that the baseline mortality rate was actually higher pre compared to post. Both of these are an example of selection resulting in a threat to internal validity.

Construct validity: I study pediatric sepsis in resource-limited settings. Recently, the definition for sepsis changed for adults, but not children, and the current definition of pediatric sepsis, based on systemic inflammatory response syndrome (SIRS) criteria, has been extensively criticized for being too sensitive and not specific. In my studies in East Africa, I still use the current definition to define pediatric sepsis, which means I am likely including children with mild illness that do not have a life-threatening infection. This is an example of an inadequate explication of constructs, which means we may be drawing incorrect inferences about the relationship between pediatric sepsis, a potentially life threatening infection, and mortality.

External validity: I conducted another sepsis study in the (only) national tertiary care hospital in Tanzania and found that delayed presentation to care was associated with mortality. Results from this study cannot be extrapolated to district hospitals in Tanzania, which have a different referral pattern, in general a less sick patient population, and decreased availability of resources. This is an example of a potential interaction of the causal relationship with the setting.

2) For any data set you frequently use, look up the sample design and describe it. 

The dataset I most frequently use is a Pediatric Sepsis Database from Tanzania. It includes ~2,000 children aged 28 days to 14 years who presented to Muhimbili National Hospital in Dar es Salaam, Tanzania, from July 1, 2016-June 30, 2017 with sepsis, as defined by clinical systemic inflammatory response syndrome (SIRS) criteria. This was a prospective cohort study that captured baseline characteristics/demographics, interventions received, lab values, functional status, and outcomes. We had research assistants available 24 hours a day, 7 days a week to screen, consent and perform data collection. All pediatric patients presenting to the emergency department were screened for inclusion criteria and then approached consent if appropriate. >90% of eligible patients were enrolled.


Re: Week 1 Reading

by Adrienne Epstein -

1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).

Statistical conclusion validity:  I was part of a study team that evaluated the effectiveness of an infant and young child feeding program. A major component of the program was counseling on breastfeeding and complementary feeding with caregivers of young children performed by service providers. The treatment was defined as nutritional counseling, with several self-reported behavioral among caregivers as outcomes (some examples include exclusive breastfeeding and early initiation of breastfeeding). Women were supposed to receive counseling a set amount of times over the course of their pregnancy and up until their child turned 2; however, in reality, women received different amounts of counseling due to behavior, moving, loss to follow-up, etc. We therefore may have underestimated the effect of counseling on these outcomes due to this unreliability of treatment implementation, as women who did not receive the full exposure may have diluted the true effect.

Internal validity: A paper was recently published evaluating the association between drought and HIV prevalence in Lesotho. The exposure was defined as whether the individual experienced drought in the past 2 years, defined using satellite precipitation data and the outcome was laboratory-confirmed HIV prevalence. The authors found that drought was associated with HIV prevalence in young rural women. However, this analysis was plagued with the issue that the outcome may have occurred before the exposure given the fact that HIV is a chronic disease that individuals live with for years. It is impossible, therefore, to state that drought increases probability of HIV infection; it may simply be that individuals with HIV have less social capital and are pushed to areas in drought. I now hope to conduct an analysis, expanding other sub-Saharan African countries, and measure a longer-term exposure of drought to improve the temporal issue (knowing that a major limitation is having only access to prevalent -- not incident -- HIV cases).

Construct validity: In the evaluation of the infant and young child feeding program described above, one construct we hoped to study was counseling quality, measured through direct service observation checklists. This entailed enumerators sitting in on counseling sessions and taking note of what occurred. This was most likely affected by the Hawthorne effect, or the phenomenon during which individuals who are aware of being observed change their behavior. We likely observed higher quality care than typical counseling sessions.

External validity: I have conducted analyses looking at the impacts of drought on several child health outcomes in Uganda. Although unmeasured, I assume there are a number of characteristics that interact with the exposure (such as coping mechanisms, resilience, and sources of income) that would impact the external validity of these findings. For example, they may not apply in other sub-Saharan African countries that have had more historic occurrences of drought, as individuals may have adapted more than the Ugandan sample.

2) For any data set you frequently use, look up the sample design and describe it. 

I am currently using data from the Demographic and Health Surveys (DHS), which are cross-sectional, nationally-representative household-based surveys funded by the United States Agency for International Development (USAID) and implemented by the private company ICF International. Typically, the surveys use a stratified two-stage cluster sampling design (although for a third of studies, the sampling is three-stage, with an additional urban/rural stratification), first selecting a random sample of enumeration areas (EAs), followed by a random sample of households within each EA. All women from the ages to 15 to 49 within selected households are invited to complete the Women’s Questionnaire and one male from each household is randomly selected to fill out the Men’s Questionnaire. The DHS provides sample weights to allow analyses to represent the country as a whole.


Re: Week 1 Reading

by Jean Digitale -

1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).

Statistical conclusion validity (unreliability of treatment implementation): A community-randomized intervention for HIV prevention had very poor uptake. It was being evaluated with repeated cross-sectional samples of individuals, but few people who were sampled had actually been exposed to the program. Thus, it was very difficult to determine whether the program was effective.

Internal validity (maturation): In adolescent research, we found that girls' scores on many knowledge scales (e.g. sexual/reproductive health knowledge) naturally improve over time as they age. Without a control group, it would be impossible to separate these gains from a treatment effect of an educational intervention.

Construct validity (treatment diffusion): When evaluating an educational program for adolescent girls, we were concerned that the control group might be friends/classmates with girls in the treatment group in neighboring compounds. We attempted to measure this by including questions on whether they had heard of the program and who their friends were to assess the degree of treatment diffusion and whether this threat to validity was present or not in our evaluation.

External validity (interactions of causal relationship with settings): When assessing the relationship between phototherapy and breastfeeding, one might find different treatment effects in different hospitals depending on hospital policy. Some hospitals, for example, allow the baby to stay in the same room as the mother to receive phototherapy, whereas others send babies to the NICU. Additionally, the amount of breastfeeding support may vary by hospital. In hospitals where babies who receive phototherapy stay in their mothers’ rooms with lots of breastfeeding support, one may find that phototherapy does not have a negative effect on breastfeeding. In hospitals where babies who receive phototherapy get admitted to the NICU with minimal breastfeeding support, a study may find that phototherapy does have a negative effect on breastfeeding.

2) For any data set you frequently use, look up the sample design and describe it. 

I am analyzing a dataset of three parallel cohort studies to investigate the effect of immunogenetic markers on malaria. All households in each study site (of three different malaria transmission intensities) were enumerated, and 100 households per site were randomly selected. All children in these households aged 6 months – 10 years old at baseline in Sept 2011 were invited to participate. The cohorts were dynamic and additional children who became eligible (age 6 mo – 10 years) were invited to participate. They were followed until they reached 11 years of age.

Re: Week 1 Reading

by Maria Glymour -

Adrienne,

Great examples.  For the drought-->HIV example, even though you cannot establish temporal order, what's the competing causal structure?  HIV--> drought (unless you think drought may be increasing HIV survival?)  ?  That doesn't really make sense, so it must be that you believe drought prone locations might increase risk of HIV for other reasons.  But then having temporal order won't necessarily help you.

Maria

Re: Week 1 Reading

by Andrea Pedroza Tobias -

1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).

Statistical conclusion validity

In an analysis using the Mexican National Health and Nutrition Survey, evaluating predictors of Metabolic Syndrome (MS) groups, according to lipid alterations in adults, we could not reject the null hypothesis, but we could have type 2 error, since our sample size was too small, and thus, we had low statistical power to conclude that the predictors are significant..

Internal validity

We performed a pragmatic clinical trial comparing metformin + lifestyle intervention vs. Lifestyle intervention, for diabetes prevention in Mexico City. The sociodemographic and anthropometric baseline characteristics were balanced between both groups. However, the follow-up rate was less than 50%, and we found that those with follow-up were on average older, with lower education and with the previous diagnosis of other risk factors such as hypertension and dyslipidemia, compared with those that were lost on the follow-up. Furthermore, we found that among those with follow-up, the metformin group had on average, higher waist circumference, BMI, lower systolic and diastolic blood pressure, higher prevalence of smoking and less education level, than the lifestyle intervention arm.  Even though the baseline characteristics at the recruitment were balanced, the differential loss of follow-up, affected the internal validity of the study. 

Construct validity. 

On the pragmatic study of diabetes prevention that I mentioned before, there could be compensatory equalization, since the providers on the clinics knew about the two arms of the study, and they might have different treatment to those that are not on the metformin group to “compensate” those that are not on the intervention group. The lifestyle intervention group consists on a counseling by a nurse with a certificate on diabetes education. However, in a visit to the clinics, we realized that  providers from the clinics that were on the lifestyle arm were more likely to refer the participants with a dietitian.  

External validity

We performed a cross-sectional study to evaluate risk factors for diabetes and diabetic retinopathy in 12,000 inhabitants of rural communities in Mexico.  The sampling was by invitation, and we found that among those with diabetes, the proportion of adults that reported to have previous diagnosis of the disease was 78% in this study, compared with 52% on rural communities according to the Mexican National Health and Nutrition Survey (ENSANUT). Thus, because of selection bias, the study cannot be extrapolated to all adults living in rural communities in Mexico. 

2) For any data set you frequently use, look up the sample design and describe it. 

The data sets that I most frequently use are the Mexican National Health and Nutrition Surveys. There is a probabilistic multistage stratified cluster sampling design. On each state, 1,440 households were selected, proportional to the rural/urban distribution. The primary sample units were the AGEBs, (similar to census tract in the US). The number of census tract were obtained based on the rural/urban distribution. On each census tract selected, there were randomly selected six blocks (second stage), and on each block, there were selected six households (third stage). On each household, it was selected, if possible, a <5 years old children, a 5-9 years old children, a teenager, and an adult. 

 


Re: Week 1 Reading

by Alice Guan -

1.      Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity and external validity)

Statistical conclusion validity: I recently taught undergraduate students in a program designed to promote educational equity among first generation, low income students. As participants in this program, all students are required to provide survey responses as a means of evaluating the effectiveness of the additional educational support that they receive. I can see two potential threats to statistical conclusion validity in the evaluation of this program, including: (1) unreliability of treatment implementation – even though all instructors in this program are provided with templatized assignments, there was tremendous flexibility in curriculum development, years of teaching experience, cultural humility training, etc. This substantial variability of classroom experience likely resulted in tremendous heterogeneity of “treatment”, which could potentially lead to decreased effect sizes in an evaluation of the program outcomes. Additional threats to statistical conclusion could be (2) extraneous variance in the experimental setting [apart from heterogeneity of implementation of curriculum in the classroom setting, students were differentially exposed in other support services that were available on campus] and (3) heterogeneity of the respondents [since this program was offered to a diverse group of “under-represented” students].

Internal validity: I was involved in a study that aimed to examine the effect of neighborhood tobacco retail density on intention to quit smoking. Ambiguous temporal precedence definitely threatened internal validity. Due to the cross-sectional design of the study, it remains unclear whether causation is uni- or bi-directional. Further, history could threaten internal validity because participants in this study were not isolated from outside events, so they were obviously exposed to other factors that effect their intention to quit smoking.

Construct validity: I was part of a study team that was investigating the effect of food insecurity of smoking behaviors. We used the six-item food security survey as the instrument to measure the construct of food insecurity. Construct validity was likely threated by confounding constructs with levels of constructs (as per Jones, Ngure, Pelto, Young, 2013, food security is an ever-evolving construct consisting of multiple, overlapping concepts that are often not enumerated in food security evaluations), and also reactivity to the experimental situation (since participants were asked about their food security status AND about their smoking behaviors in the same survey; also demonstrated by some participants’ verbalization of what they believed to be the primary research question of interest).

External validity: I’m part of a research team working on an ongoing trial examining the efficacy of a smoking cessation intervention – in this study, participants are extremely unique – they are all recruited as dyads comprised of a male daily smoker and a non-smoking household member, fluent in either Chinese or Vietnamese, and available to attend two in-person health education sessions. Additionally, since recruitment was conducted by lay health workers (or, in rare instances, by the partnered community based organizations), all participants had to be somewhat socially engaged in order to be recruited. As a result, the cause-effect relationship is unlikely to hold, for instance, to ALL Asian Americans or even to Asian Americans who smoke (interaction of causal relationship with units).

2.      For any data set you frequently use, look up the sample design and describe it.

I am currently using a dataset consisting of 680 individuals enrolled in a lifestyle intervention trial. There are two clusters to account for based on how participants were selected. (1) All participants were recruited by lay health workers [ideally and theoretically, they are all part of the lay health worker’s social network], and (2) Participants were recruited in dyads living in the same household.


Re: Week 1 Reading

by Eduardo Santiago-Rodriguez -

1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).

These examples are from different research projects I have been involved in.

Statistical conclusion validity- Low statistical power- This study evaluated the effect of a behavioral intervention in women who had a history of substance use (QOL was the outcome, measured pre and post intervention). There was a subgroup of participants who had either a diagnosis of schizophrenia or neurodevelopmental disorders (n=23) and their results were evaluated separately. Although the effect size observed for one of the QOL domains was similar than in the rest of the group, that result was not statistically significant (it was for the rest of the group). On post-hoc assessments we found power for the analysis in the subgroup was 0.4.

Internal validity- Ambiguous temporal precedence- This was a cross-sectional study aimed to determine predictors of anemia (at study enrollment) in a cohort of HIV-positive individuals. One of the variables evaluated was employment status and we found unemployed participants had higher odds of anemia than those employed. Because of the study design used and the nature of this variable (as opposed to one that does not change with time; for example: sex) we could not elucidate if in fact it predicted anemia. Two scenarios could explain the observed result: 1- unemployed people were less likely to receive an adequate medical care to prevent the progression of HIV and the presentation of one of its most common complications; 2- fatigue and weakness associated to anemia (which affect the quality of life and functioning of participants) impeded sustained employment.

Construct validity- Experimenter expectancies- This was a randomized controlled study evaluating the effect of two exercise interventions (low-intensity versus moderate-intensity) on physical functioning among breast cancer survivors. Physical therapists would be in contact with participants throughout the trial (many study visits), and during the first phase of the study investigators measured the PT’s fidelity to the interventions. I don’t know whether this threat occurred or not (left the job before the study started) but they used that strategy to reduce this potential bias.         

External validity- Interactions of the causal relationship with settings- The HIV cohort already mentioned is part of the North American AIDS Cohort Collaboration on Research and Design (NA-ACCORD), a group of more than 20 cohorts from US and Canada evaluating HIV care and outcomes. It is the only site (located in Puerto Rico) in which 100% of participants are Hispanics. I am not completely sure, but I think it’s the major contributor of Hispanics to the group (more than 1,000 individuals). I consider there is potential for this threat if what is observed in Hispanics in one of the analysis conducted with data from different sites, is generalized to all Hispanics living in the United States, where living conditions and access to treatment might be different to Puerto Rico.    

2) For any data set you frequently use, look up the sample design and describe it.

Recently, I have been using a dataset from the San Francisco version of the Health Information National Trends Survey (SF-HINTS). The sample of the HINTS is representative of the US population, but the SF-HINTS is not representative of San Francisco. Investigators of SF-HINTS used purposive sampling. They aimed to characterize a diverse population often excluded from research: non-speaking English individuals, and racial and ethnic minorities. Therefore, a community-based snowball sampling was employed with predefined proportions of the total sample corresponding to specific characteristics. The research team went to popular places in the city where the population of interest was accessible. Half of all interviews were conducted in English (of those, 50% were African Americans) and the other half were conducted in Spanish (25%) and Chinese (25%).


Re: Week 1 Reading

by Monica Ospina Romero -

1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).

Threat to statistical conclusion validity: I have encountered in my research project an example of unreliability of measures. I used the Health and Retirement Study to investigate the association between history of cancer and Alzheimer’s disease. The variable I am using as the exposure is a self-reported history of cancer diagnosis. One might think that most participants will report this medical condition accurately, but some participants might miss to report cancer. The reliability of this measurement might be associated with other characteristics of the participants such as educational level, race, or cognition. The problem of unreliability of measures complicates more when we consider that the measurement of the outcome variable (memory change) is also susceptible to error as well as some of the covariates included in the model which are self-reported medical conditions.

Threat to internal validity: Attrition, participants that continue in follow-up may have a different rate of memory decline than those who drop-out from the study. This issue is important in longitudinal studies with older adults. Testing is another threat to validity; participants’ memory is assessed at multiple time points and they might become familiar with the test.

Threat to construct validity: Inadequate explication of constructs vs. construct confounding: In 2018, the National Institute of Aging and Alzheimer’s Association published a new research framework for Alzheimer’s Disease diagnosis intended to use in observational and interventional studies. This new definition of AD is based on biomarkers (β amyloid deposition and pathologic tau) as opposed to the previous definition where the definitive diagnosis was at autopsy and in life AD was only classified as possible and probable.

It is possible that this new biologic construct solves a previous problem of construct confounding if we think that people with the clinical syndrome without biomarkers have a different condition (cognitive impairment of unknown etiology).

This new definition of AD could be an example of an inadequate explanation of construct since we don’t really understand the etiology of AD and the biomarkers used in this definition could represent minor part of the pathological mechanism of AD.

Threat to external validity: Interaction of the causal relationship with setting. I participated in a project looking at the causes of nonadherence to chemotherapy in children with cancer. We found that the lack of social support systems was the main cause for nonadherence to this treatment. However, our research setting was very specific, a public university hospital that cares for patients with subsidized health insurance. This potential causal relationship between the lack of social support systems in the families of children with cancer and nonadherence to chemotherapy might not be replicated in more affluent countries or event in other hospitals of the same city.

 2) For any data set you frequently use, look up the sample design and describe it. 

Participants of the Health and Retirement Study (HRS) were randomly sampled from the US population using a multistage area of probability sample design. Oversampling of Blacks and Hispanics on a rate 2:1 relative to whites because many of the factors that influence retirement decisions are thought to be quite different for blacks/Hispanics than for whites. There was oversampling of residents of Florida because of its high densities and numbers of older populations. HRS researchers decided to include participant’s spouses regardless of age.


Re: Week 1 Reading

by Maria Glymour -

Monica

Great examples.  Note that testing in this case is more likely to threaten construct validity (are we measuring what we think we're measuring?) than to introduce threats to internal validity.  When SCC refer to testing they are referring to an association between the testing instrument and the exposure status. 

For HRS: one reason they enrolled extra people in Florida was because the Florida state government gave them extra money! So now one of the most important studies on older people in the US is overrepresentative of Floridians!

Maria

Re: Week 1 Reading

by Matthew -

Thanks for the post monica, I enjoyed reading. Surprising that participants miss report cancer events given its seriousness. My team has also found similar findings with regard to non-adherence. We work with NICU discharged infants and have found the vast majority of poor followup and non-adherence is strongly related to social support systems and resources. Thanks again for the post =]

Re: Week 1 Reading

by Maria Glymour -

Eduardo,

Great examples.  The issue of Latinos being extremely heterogeneous comes up in Alzheimer's research too, where studies report opposite evidence on risk among Latinos vs non-Latinos. 

Is SF-HInTS a subsample of HINTS or its own, separate, study?

Maria


Re: Week 1 Reading

by Eduardo Santiago-Rodriguez -

SF-HINTS is a separate study, but most questions in the survey were taken from HINTS in any of its versions. For example, SF-HINTS has  information about types of CRC screening tests. These test-specific questions were dropped from HINTS in 2009 and now they only ask about any CRC screening test. 

Re: Week 1 Reading

by Maria Glymour -

Alice

Great examples.  Was there a target population for the intervention?

Maria

Re: Week 1 Reading

by Maria Glymour -

Teresa,

Terrific bias examples. 

Would you consider that you had nearly a census of the target population in the sepsis study?

Maria


Re: Week 1 Reading

by Teresa Kortz -

Hi Maria,

Unfortunately, no. Many children with sepsis are treated at district hospitals and not referred to the tertiary care hospital. Additionally, we actually don't have great estimates of the number of children that die at home or during transfer between health centers with sepsis.

Teresa

Re: Week 1 Reading

by Crystal Langlais -

1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).

Statistical Conclusion Validity: I worked with a research group to understand the relationship between pediatric hypotension and need for early transfusion (within four hours of injury) following blunt liver and/or spleen injury in pediatric patients. In a database of over 1000 patients, only 47 patients met inclusion criteria, with roughly halve (n=22) having documented hypotension. Patients with and without hypotension prior to transfusion were compared and no statistically significant difference was identified. Although this study was one of the largest at the time, and was disclosed as a secondary analysis (i.e., no power analysis performed), with such a small sample, it is likely we were underpowered to see a clinically relevant difference.

Internal Validity: In the above study, patients who underwent transfusion within 4 hours of injury were included in the analysis. Although most patients who succumbed to injuries within 4 hours of injury were actually transfused (and therefore included), some patients were not transfused (and therefore not included in the analysis). These patients who died prior to attempt at transfusion represent a clinically important subset of individuals. This selection can result in issues of internal validity.

Construct Validity: I am starting to work on various studies looking at the association between diet and prostate cancer progression. These studies use different approaches to capture the dietary exposure, for example: food frequency questionnaires, 24-hour recall, food diaries. Patients are apt to change their diet when questioned about it, which is a form of construct validity (reactive self-report changes). My current understanding is this is potentially problematic when asking patients to complete a food diary as they might be likely to modify their behavior when they know their diet is ‘observed’.  

External Validity: In my work with the CaPSURE database (described below), we are always cautious to acknowledge that our results might not be generalizable for two main reasons.  First, the cohort is dominantly white, and therefore results may not be generalizable to all men with prostate cancer (particularly important due to the known health disparities experienced by black men with prostate cancer). Second, these men are enrolled only if they seek out medical care at a participating center. Therefore, these men are likely to have some form of healthcare coverage and have undergone some form of screening that led them to see a urologist. Therefore, results may not be generalizable to men without healthcare coverage.

2) For any data set you frequently use, look up the sample design and describe it. 

Cancer of the Prostate Strategic Urologic Research Endeavor (CaPSURE). CaPSURE has enrolled over 15,000 men diagnosed with prostate cancer at 43 participating urologic practices (including academic, community-based, and veteran facilities) in 27 states since 1995. Men seek out the site/clinician through routine care utilization practices. If that care results in a diagnosis of prostate cancer, the patient is asked to participate in the study and are followed until death or withdrawal.


Re: Week 1 Reading

by Maria Glymour -

Crystal,

Terrific examples- the issue of mortality during the exposure period is subtle and can introduce huge bias.  

Was there a target population for capsure?

Maria


Re: Week 1 Reading

by Maria Glymour -

From Sepehr (moved from week 2 reading thread): 

Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate

1. Statistical conclusion validity

1a) Unreliability of measures may be a concern in my current research, which is assessing whether sleep duration causes TMJ (jaw joint) pain. This outcome is measured via interview response to a survey question that is similar to another already validated question— however, the question is not exactly the same. In short, the validated question asks for repeated TMJ pain in the previous weeks, while the survey question I am using asks for a at least a one time severe TMJ pain lasting a complete day instead. Since this difference in the question is not validated, it is very possible for it to introduce unreliability of outcome measurement, which would weaken the link between sleep duration and possible TMJ pain. Of note, the exposure, sleep duration, is reliably over-reported.

1b) Internal validity

There is wonderful ambiguous temporal precedence in this same research question, where it is very feasible (as historically believed) that those with jaw pain have different sleep duration, instead of sleep duration influencing TMJ integrity. However, some small-animal studies have shown histological changes in animal TMJ after induced sleep deprivation. Also, a recent prospective cohort found that those with high OSA likelihood are 2.29 (1.54-3.42) fold more likely to develop incident first-onset TMJ pain symptoms and TMD. Nonetheless these evidence are weak, and temporality remains ambiguous for such a question.

1c) Construct validity

This research may suffer from inadequate explication of constructs or use of a wrong construct. The outcome, Temporomandibular Disorder (TMD), should be instead referred to as self-report of TMD-Type pain, since this more accurately measures what the outcome that the survey has measured. There may also be construct confounding, where those who sleep less and have joint pain may suffer from a medical comorbidity that lowers their threshold for both sleep problems and chronic pain conditions. I attempted to control for these (e.g. autoimmune disorders, depression, etc) as best as the data allows, however, there may still be unknown medical comorbidities that confound the construct (e.g. hypertension?).

1d) External validity

Going further with the same research, there may be concern for interaction of the causal relationship with units being studied. Those included in the national survey I am using may not be representative of the United States general population, despite the population weighting recommended and done by the survey designers.

2.     For any data set you frequently use, look up the sample design and describe it. 

I am using National Health Interview Survey (NHIS), a multistage cross-sectional household interview survey conducted by US Department of Health and Human Services. Interviews are performed continuously through the year, with clusters in each state. Those aged 65 or older who are black, Hispanic, or Asian are intentionally oversampled (2:1). From each family surveyed a random adult and a random child (if available) are selected. National estimates are inflated using calculated weight factors to represent the target US population.


Re: Week 1 Reading

by Maria Glymour -

Sepehr:

These are great examples.  Can you be more precise about the NHIS sample design? What were the clusters?  Were there multiple stages of clustering? How did they implement the over-sampling (ie, at what stage did they have a sampling frame to use for over-sampling?)

Maria


Re: Week 1 Reading

by Maria Glymour -

From Zara (moved from Week 2 thread): 

Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).

Statistical conclusion validity: The validity of inferences about the correlation (covariation) between treatment and outcome.

1)     Low statistical power. I have studied data with small sample size and resulting low statistical power. This could lead to incorrectly concluding that the relationship between treatment and outcome is not significant.

2)     Outcomes in chronic diseases such as RA are heterogeneous. Increased variability on the outcome variable (such as disease activity in RA) within conditions (phenotypes) increases error variance, making detection of a relationship more difficult. One could study outcomes within strata of conditions but phenotypic categories are often complex to define or data may not be available.

Internal validity: The validity of inference about whether observed covariation between the treatment and outcome reflects a causal relationship from treatment to outcome as those variables were manipulated or measured.

3)     Ambiguous temporal precedence: I have studied cross-sectional data with lack of clarity about which variable occurred first. This can yield confusion about which variable is the cause and which is the effect.

Construct validity: The validity of inferences about the higher order constructs that represent sampling particulars

4)     Construct confounding: A patient-reported outcome commonly captured in RA is patients’ global assessment of disease activity. It is collected on a scale of 0-100 using a visual analogue scoring system. Patients’ perceptions of their RA disease activity may be affected by other constructs such as mental health. Operations usually involve more than one construct, and failure to describe all constructs may result in incomplete construct inferences.

5)     Reactivity to the experimental situation: In the above example, based on their satisfaction with care/treatment patients may score their global assessments of disease activity differently. Responses reflect not just treatments and measures but also participants perceptions of the experimental situation, and those perceptions are part if the treatment construct actually tested.

External validity: The validity of inferences about whether the cause-effect relationship holds over variation in persons, setting, treatment variables, and measurement variables.  

6)     Interaction of the causal relationship with outcomes. In rheumatology a discordance between patient-reported functional status and physician reported outcomes has been commonly reported in literature. Treatment may therefore be associated with improvements in physician reported outcomes but not patient-reported functional status. An effect found on one kind of outcome observation may not hold if other outcome observations were used. 

For any data set you frequently use, look up the sample design and describe it. 

I have not used any survey-based datasets previously. For the purpose of this exercise I have chosen the HRS dataset.

HRS uses a national area probability sample of U.S. households with supplemental oversamples of Blacks, Hispanics and residents of the state of Florida. Sampling weights are provided on all HRS data sets to compensate for the unequal probabilities of selection between the core and oversample domains.

The HRS sample is selected under a multi-stage area probability sample design. The sample includes four distinct selection stages:

1)     The primary stage of sampling involves probability proportionate to size (PPS) selection of U.S. Metropolitan Statistical Areas (MSAs) and non-MSA counties. National Sample PSUs were assigned to 84 explicit strata based on MSA/non-MSA status, PSU size and geographic location.

2)     Stage 1 is followed by a second stage sampling of area segments (SSUs) within sampled primary stage units (PSUs). A minimum of 72 housing units was required for core sample SSUs. If a block had no housing units or fewer than 72 housing units, it was linked with adjacent blocks to form SSUs of sufficient size.

3)     The third stage of sample selection is preceded by a complete listing (enumeration) of all housing units (HUs) that are physically located within the bounds of the selected SSU. The third sampling stage is a systematic selection of housing units from the HU listings for the sample SSUs.

4)     The fourth and final stage in the multi-stage design is the selection of the household financial unit (the HRS observational unit) within a sample HU.

The household analysis weight is a composite weight which has been formed from the product of five component factors: (1) the housing unit selection weight, (2) an adjustment factor for non-listed segments, (3) an adjustment factor for subsampled areas, (4) a household nonresponse adjustment factor, and (5) a household post-stratification factor.

The Person-level Analysis Weight is the product of the Household Analysis Weight, the Respondent Selection Weight and the Person-level Poststratification Weight. 

Poststratification adjustments are made at both the household and person level in order to control sample demographic distributions to known 1990 Census totals. Rationale: Post-stratification factors are small adjustments to analysis weights that are designed to bring weighted sample frequencies for important demographic and geographic subgroups in line with corresponding population totals that are available from a source that is external to the survey data collection process. The post-stratification procedure is expected to reduce the mean square error of sample estimates.


Re: Week 1 Reading

by Crystal Langlais -

I think the goal was all men living in the US. In reality, it is limited to those men with access to healthcare. 

Re: Week 1 Reading

by Teresa Kortz -

Hi Crystal,

As a pediatric researcher, we often run into issues with inadequate patient numbers. I wonder if a case-control study design using the same dataset may be a more efficient way to test for an association.

-Teresa

Re: Week 1 Reading

by Crystal Langlais -

Hi Teresa, 

Thanks for your response. The dataset we had only included pediatric patients with blunt liver and spleen injury. I assume you're suggesting the controls to be those without transfusion to increase total sample size.  This is an interesting thought. If i'm understanding correctly though, I think this would answer a slightly different question - although I know I didn't give enough information in my initial post for you to know that. We were trying to answer the question: do patients requiring early transfusion have documented hypertension prior to transfusion?  I think using non-transfused patients as controls and transfused patients as cases would answer the question: Are transfused patients at greater risk of hypertension compared to non-transfused patients?  The later question may still be worth documenting; however, its been a while since I worked in this area so I'm not current on the literature. 

-crystal

Re: Week 1 Reading

by Shelley DeVost -

Part I:

1.     Threat to statistical conclusion validity: Unreliability of measures.

Salivary osmolality is the concentration of proteins, electrolytes, and other solutes in saliva. It can be measured precisely in milliosmoles per kilogram of water (mOsm/kg). Despite the precision of the measurements, salivary osmolality is a highly variable measure and is very sensitive to external factors including recent food and fluid consumption, oral hygiene, and time of day. A study I previously worked on used salivary osmolality as a biomarker for chronic underhydration in older adults. In addition to other steps taken to improve the reliability of the osmolality measurements (such as a water rinse fifteen minutes before collecting saliva samples), the investigators decided to collect up to six repeated measurements (one morning and one afternoon measurement on each of three consecutive days) for each participant in the study.

2.     Threat to internal validity: Ambiguous temporal precedence.

In the study described above, salivary osmolality—the presumed outcome—was measured at the same time as the covariates of interest: mobility, number of comorbid conditions, and barriers to hydration. Though the temporal relationship between barriers to hydration and salivary osmolality may be safely assumed, it is quite plausible that an older adult’s degree of mobility and number of comorbid conditions could actually be consequences of chronic underhydration (measured by salivary osmolality) rather than causes.

Other threats to internal validity in this study include selection (a convenience sample from two senior community centers and one adult day care center such that the three facilities differed markedly from one another by race/ethnicity and by participants’ degree of independence and overall health status) and attrition (participants were expected to contribute six measurements of salivary osmolality over three consecutive days, but many individuals failed to return on the second or third days, and this loss to measurement may be correlated with degree of illness, mobility, and dehydration).

3.     Threat to construct validity: Inadequate explication of constructs.

In a study of the effect of hookup app use on STI incidence among gay, bisexual, and other men who have sex with men (MSM), I stratified the exposure of interest in the primary analysis—how participants met their sexual partners in the past three months—into three categories: in person, through a social networking or dating website online, and through a dating or hookup app. However, this stratification of the exposure oversimplifies and obscures many potentially relevant factors in the acquisition of an STI. For example, the construct of meeting sexual partners “in person” is far too general and probably conflates two or more distinct constructs. To address this potential criticism, I could have performed this analysis with unstratified values of the exposure data. Continuing the example above, the “in person” venues collected in the raw dataset included “bar/club,” “party/mixer,” “through a friend,” “school,” “work,” “gym,” “street,” and “bathhouse/sex club.” These eight venues clearly differ from one another in terms of a variety of factors that would influence the risk of STI acquisition.

4.     Threat to external validity: Interaction of the causal relationship with settings (and units).

In the study of whether meeting sexual partners through hookup apps increases the risk of STI acquisition among MSM, the data were obtained from a convenience sample of clients at a nonprofit, community-based, LGBT sexual health clinic. Any causal effect between hookup app use and STI incidence could plausibly differ across other settings and other client populations. Even if the data had been extracted from the primary care clinic at the same health center, very different effects could be observed between sexual partner meeting site and STI acquisition due to the differing demographic and risk profiles of the clients in each setting.

 

Part II:

The National Health and Nutrition Examination Survey (NHANES) uses a four-stage, probability-based sampling design. In the first stage of selection, individual counties (or groups of neighboring counties where county populations were too small) were selected from among all counties in the United States. The counties are considered to be the primary sampling units (PSUs). In the second stage, individual census blocks (or groups of neighboring census blocks) were selected from all census blocks in all the PSUs selected in stage one. These census blocks are called the “segments” of the PSUs. In the third stage, a subsample of all “dwelling units” in a given segment was compiled and screened for potential participants, sampled in the fourth and final stage of sampling. Each stage of the sampling design used probability sampling to construct a nationally representative sample in which particular subgroups are oversampled. This oversampling improves the reliability and precision of the data for those meaningful yet small populations.