1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
Statistical conclusion validity: In one of the studies I have been working on, we often struggle to enroll participants, and it is even harder to follow them for a year. Our population for this study is older adults with schizophrenia in San Francisco, many of whom have substance use disorders and are unhoused. Therefore, we have a low sample size (and low retention rate), resulting in low power. This makes it difficult to detect smaller effect sizes and we have very wide confidence intervals, confirming the suspicion of low statistical power and possible type 2 error.
Internal validity: For one of the studies I am working on now I am examining the exposures of depression, loneliness, and social isolation. I have been struggling with the ambiguous temporal precedence of these exposures. In my anecdotal and clinical observations, loneliness and/or social isolation and/or depression may occur concomitantly and are likely to act reciprocally. There is some scientific literature supporting the hypothesis that loneliness precedes depression, however the literature is not conclusive and other studies have found the opposite to be true.
Construct validity: Part of the theoretical framework I am using for my dissertation research is the theory of structural violence, which says that the social arrangements that cause ill-health/harm/injury to populations are structural i.e. embedded in the political and economic organization of our social world. Using this theory, it is critical for me to examine the heterogeneity of effects in different racial groups. We use the construct of participant-identified race as a variable, however what we really wish to measure is the construct of living-as-black (or living-as-latinx or living-as-white). In this case, when racialization is reduced/simplified to an attribute we call “race,” the interpretation becomes less clear and this can affect study conclusions, sometimes in profound ways. As Cook, Campbell, and Shadish say, “The naming of things is a key problem in all science, for names reflect category memberships that themselves have implications about relationships to other concepts, theories, and uses.” (2002, p.66). This may best described as inadequate explication of constructs. This also relates to Statistical conclusion validity--many of the studies in my area of research fail to examine, or even mention, the importance of racial group heterogeneity in the relationship between schizophrenia and brain health.
External validity: Context dependent mediation may occur in my study of loneliness/social isolation, depression, and cognitive outcomes in older adults with HIV. I am hypothesizing that depression mediates the effect of loneliness on cognitive performance. However, because the study only recruited people who had no active substance use, these findings may not be true in many of the “real-world” clinical settings in which people with HIV live and receive healthcare. This is an important threat to external validity in my area of nursing research.
2) For any data set you frequently use, look up the sample design and describe it.
I have been using a dataset from a study at the UCSF Memory and Aging Study. For this study, 170 people living with HIV aged 55 and older with confirmed HAND in the San Francisco Bay Area. (The first 120 participants were enrolled under the eligibility criteria of being 60 years or older, which was later changed to to 55 years and older.) Participant eligibility was determined during primary screening calls and secondary screening visits. Potential participants underwent a two-tiered screening process, with primary screen administered in person or by phone to assess key exclusions (e.g. unsuppressed plasma viral load, not on cART) and assure the presence of cognitive or behavioral symptoms. Participants who passed the primary screen then completed a secondary screen including one-hour neuropsychological testing at the NCRU. Data from the secondary screening visit were reviewed at consensus conference attended by a physician and neuropsychologist each trained in HAND who use clinical acumen to determine the participant’s cognitive diagnosis. Individuals with neuropsychological testing that was not deemed to be within normal variability were eligible for inclusion. Consensus conference diagnosis were guided by the 2007 Frascati criteria.
Terrific examples Sarah.