1) Provide an example of 4 threats to validity that you have encountered in your research, drawing one from each of the domains Cook and Campbell delineate (statistical conclusion validity, internal validity, construct validity, and external validity).
A. Statistical conclusion validity
· Low statistical power – the power calculation is based on the estimated difference in prevalence in an outcome between two groups however little thought is given to how sampling might affect the assumed prevalence. For example a study of elder mistreatment which draws from general population and assumes a prevalence of 30% doesn’t consider that study participation rates might be related to elder mistreatment (families where elders are being mistreated are less likely to participate in research studies) so the actual prevalence in the study is much lower and so the power is much lower.
· Error rate – this feels like it is quite common in studies using neuropsychological tests as these tests often have many sub-tests and often results are presented for individual tests and I have seen as many as 25 tests on a sample of 200 people.
· Unreliability of treatment implementation – a study trying to evaluate whether or not a text message program to support chronic disease management found no effect between the intervention and control arm however the research team also found that a very large portion of the participants had not received the intervention at all.
B. Internal validity
· Attrition – a study looking at the use of text messaging to improve iron tablet intake in pregnant women in India had an attrition rate of over 40%. I have also seen differential attrition rates between intervention and control groups which are important and can’t be corrected for.
· Regression – in my research experience this is a frequent problem with measures of blood pressure which is why in most research protocols the first reading is disregarded and the mean of at least another two readings is used. I would like to think more about how regression and variability (e.g. seasonal variation in performance) interact
· Maturation – in a study of traumatic brain injury not restricting to people who had completed their education i.e. including children who couldn’t have completed high school in the group of people who didn’t complete high school and trying to assess the effect of education on traumatic brain injury outcome (the problem is extended if age of injury isn’t accounted for since having a severe head injury as a child may preclude educational attainment, perhaps this would fall under ambiguous temporal precedence.)
C. Construct validity
· Inadequate explication – this feels very common in reading literature of cognitive testing; there seems to be lots of variation in how people define constructs like “executive function.” Sometimes researchers will use the same tests but decide they measure different constructs
· Construct confounding – in research using occupational cohorts and using sex and self-reported or otherwise obtained ethnicity but not accounting for differential exposure rates due to sex and class differences in the type of work being done. For example using a sample of veterans and looking at race differences in brain injury which likely hide factors like rank and military service type which may influence likelihood of exposure to circumstances where brain injury occurs. This may also happen with cohort effect for example shifting from conscription to voluntary service.
· Novelty and disruption – we embedded a research study in a large tent in a primary care clinic. When we did a process evaluation of the intervention (text messages to support treatment adherence) a major theme that emerged was how much everyone loved coming to the tent. (Would we consider this similar to the Hawthorne effect?)
D. External validity
· Causal relationship with outcomes – we designed a brief messaging intervention to improve treatment adherence among people with high blood pressure. We were not sure whether such an intervention would work in people with other or multiple chronic conditions.
· Causal relationship with setting – the same study looked at participants at a single clinic site. We were also not sure whether the intervention might work in other health system settings (we designed a subsequent study to test this.)
2) For any data set you frequently use, look up the sample design and describe it.
Health ABC – population-based sample of 3075 older adults (70 to 79 years of age at enrolment. Participants were identified either through a random sample of Medicare beneficiaries (white), or all age-eligible community residents in designated zip code areas surrounding Pittsburgh, Pennsylvania, and Memphis, Tennessee (black.) Inclusion criteria included no difficulty in walking one-quarter mile or climbing 10 stairs without resting. Exclusion criteria included difficulties with activities of daily living, obvious cognitive impairment, inability to communicate with the interviewer, intention of moving within 3 years, or participation in a trial involving a lifestyle intervention.