Discuss briefly; When is a quasi- or natural-experiment more appropriate than a randomized experiment? When is a quasi- or natural-experiment more informative than a conventional observational study? Give an example of a substantive question and a stakeholder (e.g., policymaker, patient, clinician) who would be more interested in an ITT effect estimate vs an IV effect estimate. Discuss how each (ITT and IV) correspond to effect estimates from conventional studies.
When is a quasi- or natural-experiment more appropriate than a randomized experiment?
A quasi- or natural-experiment would be more appropriate in any case where randomization is unethical. Using the compulsory education laws example, we could not randomize people to stop going to school to study the effects of education. We could, however, take advantage of a policy change that encouraged people to stay in school longer to compare people's outcomes before and after the change (among those for whom the policy altered the duration of schooling/compliers).
When is a quasi- or natural-experiment more informative than a conventional observational study?
Quasi- or natural experiments help us deal with the confounding we struggle with in conventional observational studies. Taking the compulsory schooling law example again, if we simply compared outcomes among people with more vs. less education it will be a terrible confounded study. There are numerous factors that affect people's length of education, that would also affect health outcomes. Comparing people who get X years of education to people who get X+1 years of education after the CSL reduces this problem, because the only cause (presumably) of additional education in the second group is the policy change.
Give an example of a substantive question and a stakeholder (e.g., policymaker, patient, clinician) who would be more interested in an ITT effect estimate vs an IV effect estimate.
Does reducing criminal penalties for drug possession affect drug-related ED visits? An ITT estimate would tell a state-level policymaker the average state-level effect of the policy change, so that might be their priority. However, an IV effect estimate might still be of interest to them. For example, differing exposures may result from variation in policy implementation across the state, and correspond with substantial differences in the outcome. Results of an IV estimate may therefore indicate that encouraging more consistent policy implementation across counties could be a worthwhile endeavor.
Discuss briefly; When is a quasi- or natural-experiment more appropriate than a randomized experiment?
I think quasi or natural experiments are more appropriate when it is unethical to do RCT or just not feasible to do an RCT. If we want to test how bicycle helmet laws affect brain injury due to bicycle riding, it would be great to randomize some cities to pass a law and others not to. However, given our governmental system it would be nearly impossible to pull this RCT off. Therefore, an observational study would be the best tool we have.
When is a quasi- or natural-experiment more informative than a conventional observational study?
Natural experiments have seemingly "random" assignment and thus helps us to control for confounding in an efficient way compared to observational studies. For example, if we changed the law on bicycle helmets, those riding bicycles before and after the law are more or less randomly assigned (unless they moved to a city specifically because of the law).
Give an example of a substantive question and a stakeholder (e.g., policymaker, patient, clinician) who would be more interested in an ITT effect estimate vs an IV effect estimate.
I think hospital administrators or policymakers are more interested in an ITT effect of an intervention for let's say hypertension. This is because they care less about individual patient outcomes rather than the intervention as a whole for all their patients in the hospital. ITT analysis focuses on the intervention and not whether or not the person actually went through the the intervention.
Discuss how each (ITT and IV) correspond to effect estimates from conventional studies.
My guess is that ITT attenuates the effect estimates from conventional studies. This is because we know that not all participants will actually go through with the treatment. IV probably have mixed effects on the effect estimates depending on the assumptions that are violated (i.e. the IV is associated with your exposure, no confounding between IV and outcome, and all pathways go from IV through exposure to get to outcome). Depending on which assumption is violated, I could see the effect estimates to be attenuated or spurious (esp if there is confounding between IV and outcome).
Thomas
If the IV assumptions are violated, ie direct effect of the IV on the outcome not via the exposure, what is the interpretation of the ITT estimate? Many of the things that would violate an IV assumption, e.g., common causes of the instrument and the outcome or pathways from the IV to the outcome not via the treatment of interest probably represent serious problems with your RCT and make the ITT not useful.
So, for example, if you randomly assign inpatients to a new care regimen (say, not waking patients up as much during the night) but nurses somehow subvert randomization because they really want certain patients to be able to sleep through the night, this does mean the IV assumptions are violated, but it also means the ITT effect estimate is bunk.
The ITT is *usually* attenuated compared to the effect of actually receiving the treatment that was randomized, although there are exceptions if people's potential response to treatment is correlated with whether they adhere to randomization or not.
We live in a world where we cannot randomize many treatments or exposures of interest, for ethical reasons. Therefore quasi- or natural experiment designs allow us to examine the effect of exposures on outcomes when we cannot do a RCT and there are many potential confounders that would be expensive or impossible to measure using traditional observational research designs. The quasi/natural experiment design assumes a situation where the exposure or treatment is as good as randomized, meaning that there are no measured or unmeasured common causes of exposure and outcome (exchangeability between treated and untreated).
I think ITT effect estimates are particularly helpful when looking at the effect of a policy on health outcomes in a municipal area. In this case, not all people will adhere to the policy, but if the policy improves overall health then it is considered effective. Policy effects generally aim to improve the average health of the population not effect of the policy only on those who adhere. An example of this would be soda tax on diabetes outcomes.
The IV estimate takes into account the people who actually adhere to the treatment. This would be more important in an RCT when you want to know if a specific drug prevents death. In this case you don’t want to know if the drug vs. placebo prevents death in those randomized to it (though if it does, it is likely to prevent death even more for those who take it), but for future clinical purposes you want to know if those who take the drug have better outcomes.
The ITT estimate would reflect an estimate of the ATE from a conventional study – that is the marginal effect of treatment in the population. Because [marginal] randomization holds, you are estimating the effect of treatment in the population. The IV estimate would reflect the effect of treatment on the treated, because you are correcting the ITT to estimate what the effect of randomization is on those who actually took the treatment. If there is a low level of non-compliance, the estimates will be similar but they will diverge quickly if compliance is low. *****However, if compliance is differential between groups, there are likely to be reasons why that may also affect the outcome (confounding). The IV estimator doesn’t seem to account for this; therefore would the IV estimates be biased (similar to an as-treated analysis in RCT)?
Natalie
This is not quite right. The ITT in general will not correspond with the ATE for treatment received or the ATE one typically tries to estimate from an observational study. Consider a trial such as:
Z--> X --> Y
where Z is random assignment. The ITT is the association between Z and Y. In an observational study you would just have X and Y, and you would estimate the association between X and Y. To the extent there is any non-adherence in the trial, you would expect it to diverge from the ATE for X.
The IV can be used to estimate the ATE, the LATE, or the ETT, depending on what assumptions you are willing to make (in addition to the structural assumptions of the IV). The LATE is most common, but the LATE only equals the ETT under special circumstances (like you couldn't get the treatment except via being randomized to it in the study, as in the MTO trial).
The situation you raise, where non compliance is differential between Z=1 and Z=0 is not a problem for the IV, and does not put you back into the bias of per-protocol land. A related situation that does cause problems - for both the IV and the ITT- is the situation I mentioned for Thomas: if one's potential treatment response is associated with adherence. But again, then you have a problem interpreting either the ITT or the IV.
Alyssa
Nice discussion, but what is the exposure of interest for the policy change? It's actually hard to really conceptualize the IV analysis because it seems hard to specify the endogenous variable. What do you think it is? Police contact? Sentence received?
Discuss briefly;
When is a quasi- or natural-experiment more appropriate than a randomized experiment?
Quasi- or natural- experimental studies are often more appropriate for evaluating the impact of large scale policies or interventions. These are cases where it is not possible to really do a randomized experiment since it is not possible to manipulate certain exposures such as policy implementation. Other barriers to RCT which would make a natural experiment more appropriate include cases where it is unethical to manuplate the exposure, cases in which there is extreme variability in implementation or very short/long timescales.
When is a quasi- or natural-experiment more informative than a conventional observational study?
Quasi- or natural-experiments are more informative than a conventional observational study if they allow for the avoidance of treatment/ self selection bias and thus ensure treatment status is not related to baseline characteristics of the respective cohorts. This is frequently the case when the natural-experiment is some chance event.
Give an example of a substantive question and a stakeholder (e.g., policymaker, patient, clinician) who would be more interested in an ITT effect estimate vs an IV effect estimate. Discuss how each (ITT and IV) correspond to effect estimates from conventional studies.
We can just take the typical example of testing the efficacy of a new treatment. The FDA would prefer an ITT effect estimate because they are most interested in preserving randomization though this might produce a conservative estimate of the treatment effect. Instrumental variables might be preferred by the manufacturer of the drug that is trying to get it approved since it may produce a more accurate estimate of the true biological effect of the drug. An instrumental variable would address for part of the deviation from protocol. How each corresponds to effect estimates from conventional studies relates to the extent of deviation from protocol or missing data/loss to follow up. However typically ITT is used to preserve the randomization but is generally more conservative than a conventional study, while IV (if an appropriate one is chosen) is closer to the true effect.
James
Nice discussion. If you can estimate an ITT effect because you have done an RCT of a device or a drug, you can nearly always estimate the IV because randomization is the ideal instrumental variable.
When is a quasi- or natural-experiment more appropriate than a randomized experiment?
A quasi- or natural experiment is more appropriate than a randomized experiment when it is either unethical or not possible to randomize individuals to treatment/exposure groups.
When is a quasi- or natural-experiment more informative than a conventional observational study?
A quasi or natural experiment is more informative than a conventional observational study when we have major concerns about confounding, both measured and unmeasured. It is also beneficial in the case of a natural experiment where there is already an "exposure/treatment" difference between two populations that we can take advantage of to assess our outcome of interest. One example of a natural experiment that is more informative would be the study comparing the rate of alcohol-related car accidents in two states, one where the legal BAC limit was recently lowered to 0.05 and another where the legal BAC limit stayed the same (at 0.08).
Give an example of a substantive question and a stakeholder (e.g., policymaker, patient, clinician) who would be more interested in an ITT effect estimate vs an IV effect estimate. Discuss how each (ITT and IV) correspond to effect estimates from conventional studies.
Question comes from study conducted by Xu et al.: Does primary tumor resection increase survival in Stage 4 colorectal cancer?
http://www-ncbi-nlm-nih-gov.ucsf.idm.oclc.org/pmc/articles/PMC4643284/
The study used primary tumor resection rates within a Health Service Area (HSA) as the instrument in the measurement of the effect of undergoing primary tumor resection on survival from stage IV colorectal cancer. A clinician would be interested in the ITT effect estimate in this scenario because it would best estimate the average effect of primary tumor resection on survival, which would help the clinician better understand quality of care. The IV effect estimate would help the clinician understand how increased rates of primary tumor resection in certain areas impact survival from stage IV colorectal cancer. These are not analogous effects, and knowing specifically about survival in individuals who undergo primary tumor resection would likely be more informative for a clinician.
Josh
Can you explain your example more? The instrument here is the regional average resection rate (among all diagnoses of stage IV colorectal cancer)? And the endogenous variable (treatment) is the actual receipt of a tumor resection? If so, I think the clinician would be more interested in the IV effect (which would estimate the effect of receipt of resection) than the ITT effect, which would be the difference in outcomes between regions w/ high and low resection rates.
Not sure i understand the example though.
Discuss briefly; When is a quasi- or natural-experiment more appropriate than a randomized experiment?
A quasi- or natural-experiment may be more appropriate under the following conditions: 1) it is unethical to randomize to treatment/intervention groups 2) outcome information has been collected on at least two groups before and after an intervention/treatment 3) data can be collected on the state of the outcome data prior to the intervention 3) the intervention has taken place but is only received by members of one the proposed groups and 4) after the delivery of an intervention/treatment, the data have been collected from the same members of the groups as were collected before the intervention/treatment was delivered (Issel, 2013).
When is a quasi- or natural-experiment more informative than a conventional observational study?
Quasi- or natural experiments can be more informative than conventional observational studies when an intervention/treatment uses an outcome that is unbounded and the intervention/treatment is delivered at a level that is not under direct control of an experimenter, such as a new policy or law. Bounded outcomes include knowledge, attitudes, behaviors and other physiological measurements that can be measured before and after an intervention has been delivered.
Give an example of a substantive question and a stakeholder (e.g., policymaker, patient, clinician) who would be more interested in an ITT effect estimate vs an IV effect estimate.
An example of the substantive question: 'What was the effect of military drafteligibilityfor the Vietnam war on mortality? '
The stakeholder: policy maker
Discuss how each (ITT and IV) correspond to effect estimates from conventional studies.
ITT analysis corresponds to effect estimates from conventional studies because it retains the integrity of randomization and provides an unbiased estimate of the effects.
IV analysis correspond to effect estimates from conventional studies because instrumental variables are associated with the treatment but not to the outcome (except via the association with the treatment). These instruments allow researchers to obtain unbiased effects of a treatment, or the 'effect of the treatment on the treated.'
Apologize for the delayed post...
When is a quasi- or natural-experiment more appropriate than a randomized experiment?
I think a “natural-experiment” is more appropriate when randomization is not feasible, or ethical. An example that had come up in a previous class had to do with the military and pain medications during childbirth. Thus, women were only allowed a particular intervention in one year versus the other based on military policy. This administrative policy allowed for the comparison of the impact of pain medication on delivery time.
Give an example of a substantive question and a stakeholder (e.g., policymaker, patient, clinician) who would be more interested in an ITT effect estimate vs an IV effect estimate.
Intention to treat analysis is used in the setting of randomization and addresses the question of effectiveness of treatment, and admits that noncompliance and deviations from instructions exist. For example, a physician prescribing a medication might be more interested in an ITT effect estimate.
An IV effect estimate might be more helpful to look at a problem on a larger scale. For example, researchers might use the passage of the ACA as a instrumental variable when looking at outcomes as you could never do a randomized trial of having or not having health insurance. This would be more helpful to policy makers.
When is a quasi- or natural-experiment more appropriate than a randomized experiment?
A quasi or natural experiment is more appropriate than a randomized experiment when randomization is not possible or desirable. For example, it would be impossible to randomize mothers to breastfeeding or not breast feeding their infants and see the effect on a particular outcome, say toddler weight. But perhaps there was a policy change in which greater support was given to mothers to breast feed through lactation consultants and no more free formula from WIC. Then perhaps this policy change could be used to study the relationship between breastfeeding and the outcome of interest.
When is a quasi- or natural-experiment more informative than a conventional observational study?
A quasi or natural experiment are often more informative than a conventional observational study in that they can avoid a lot of concerns about confounding. It is always very helpful when we can take advantage of two populations that are already different in their exposure.
Give an example of a substantive question and a stakeholder (e.g., policymaker, patient, clinician) who would be more interested in an ITT effect estimate vs an IV effect estimate.
I think that in general policy makers are more interested in the overall effect on a population rather than the effect on individuals. Whereas patients and clinicians tend to me more interested in the effect on the individual. All parties may be interested in the same question but having differing opinions of value. For example, suppose there was a question about a specific early intervention to improve symptoms of autism. A policy maker might be interested in the net effect on the population of patients with autism as the specific intervention is very expensive and may provide only small overall improvement. ITT analysis focuses on the intervention and not whether the individual patients actually got the intervention. This is more similar to a policy in which an intervention would always be offered but maybe not received like in outpatient autism therapy. On the other hand a parent or clinician might be more interested in the question of whether if the therapy is received the individual patient would have improvement in their autism symptoms. Thus an IV effect estimate in which all/most of the patients who were supposed to receive the therapy did receive the therapy would give better information as to whether the therapy would work in their particular child or patient.
Discuss how each (ITT and IV) correspond to effect estimates from conventional studies.
It is my understanding that in general ITT effect estimates tend to attenuate the effect estimates you might see in an RCT. Whereas an IV can have various directional impact on the effect estimate. In general IV’s are felt to be very close to the true effect estimate from an RCT. But just as in an RCT there can be biases introduced because of deviation from the protocol or missing data or other such issues you can have some of the same problems with your IV. Glymour et al in our reading today pointed out some ways of assessing the bias in your IV and making sure that your IV is appropriate and does not violate the basic required assumptions of IV.
When is a quasi- or natural-experiment more appropriate than a randomized experiment?
A quasi or natural experiment is more appropriate than a randomized experiment when randomization is not possible or desirable. For example, it would be impossible to randomize mothers to breastfeeding or not breast feeding their infants and see the effect on a particular outcome, say toddler weight. But perhaps there was a policy change in which greater support was given to mothers to breast feed through lactation consultants and no more free formula from WIC. Then perhaps this policy change could be used to study the relationship between breastfeeding and the outcome of interest.
When is a quasi- or natural-experiment more informative than a conventional observational study?
A quasi or natural experiment are often more informative than a conventional observational study in that they can avoid a lot of concerns about confounding. It is always very helpful when we can take advantage of two populations that are already different in their exposure.
Give an example of a substantive question and a stakeholder (e.g., policymaker, patient, clinician) who would be more interested in an ITT effect estimate vs an IV effect estimate.
I think that in general policy makers are more interested in the overall effect on a population rather than the effect on individuals. Whereas patients and clinicians tend to me more interested in the effect on the individual. All parties may be interested in the same question but having differing opinions of value. For example, suppose there was a question about a specific early intervention to improve symptoms of autism. A policy maker might be interested in the net effect on the population of patients with autism as the specific intervention is very expensive and may provide only small overall improvement. ITT analysis focuses on the intervention and not whether the individual patients actually got the intervention. This is more similar to a policy in which an intervention would always be offered but maybe not received like in outpatient autism therapy. On the other hand a parent or clinician might be more interested in the question of whether if the therapy is received the individual patient would have improvement in their autism symptoms. Thus an IV effect estimate in which all/most of the patients who were supposed to receive the therapy did receive the therapy would give better information as to whether the therapy would work in their particular child or patient.
Discuss how each (ITT and IV) correspond to effect estimates from conventional studies.
It is my understanding that in general ITT effect estimates tend to attenuate the effect estimates you might see in an RCT. Whereas an IV can have various directional impact on the effect estimate. In general IV’s are felt to be very close to the true effect estimate from an RCT. But just as in an RCT there can be biases introduced because of deviation from the protocol or missing data or other such issues you can have some of the same problems with your IV. Glymour et al in our reading today pointed out some ways of assessing the bias in your IV and making sure that your IV is appropriate and does not violate the basic required assumptions of IV.
When is a quasi- or natural-experiment more appropriate than a randomized
experiment?
All other things being equal, a randomized experiment is stronger than a quasi- or natural experience due to threats to internal validity. Namely, inabilitiy to control for confounding as a randomized study design does. However, there are situations where randomization is not feasible due to ethical, logistical, or temporal hurdles.
Examples include:
Temporal: Retrospective evaluation of an intervention ("Let's go back and take look at the effect of this policy") cannot take place with randomization.
Ethical challenges: If an intervention is known to be efficacious, or deleterious, selection of treatment/control group by researchers is questionable ethically.
Logistical challenges: Hard (from financial and legal standpoint, amongst others ) to put on a large RCT of health care plans nationally, but rather, use state by state differences to compare each state's plan.
When is a quasi- or natural-experiment more informative than a conventional
observational study?
Looking at an intervention's efficacy.
Give an example of a substantive question and a stakeholder (e.g.,
policymaker, patient, clinician) who would be more interested in an ITT
effect estimate vs an IV effect estimate.
Technology adoption, say, HHS is inexplicably considering buying FitBits for America. IV could get you the actual effect of the intervention, but IT would arrive at a more realistic result that could be shared with funders.
Discuss how each (ITT and IV) correspond to effect estimates from
conventional studies.
ITT would "dilutes" (glymour 2006 Natural Experiments...), but I think IV in bearing more similarity to RCT but could be higher/lower than the truth secondary to external validity.
Quasi-experimental studies are especially useful if you want to make causal attribution to interventions that are not easy to study in a traditional randomized experiment. For example, I know a student who wanted to study whether a feedback intervention to nurses on a maternity ward in Uganda can improve the use of partographs in labor monitoring. I thought that this was a good question to apply to a quasi-experimental approach (the student later gave up on the project, I suspect because he ended up getting negative feedback himself from his peers and his bosses about the proposal).
Patographs are like forms that are used to monitor the progress of labor among expecting mothers. They are widely recommended and have been shown to improve outcomes for both mothers and infants on obstetric wards. However, they are rarely filled at least in this setting. The student was interested in improving the use of partographs at this hospital, and wanted to use a simple feedback intervention, where he would frequently meet with the midwives on duty and talk to them briefly about the deliveries that they had just performed. The plan was that these conversations would be structured to include some form of feedback on the partograph, and, hopefully, this would improve the behavior of the midwives with regard to filling partographs, and eventually, overall use of partographs in this hospital.
A conventional observational study can possibly observe changes in such a system, but it would be difficult to attribute any changes in behavior to the feedback intervention. A quasi-experiment would also face challenges, but would be more able to make causal attribution. A traditional randomized trial is probably conceivable, but I don't see it working here. I think quasi-experiments are good for situations where it is easy to visualize the intervention but not its control (i.e., you probably have a treatment, but you are not going to easily get a placebo).
One possible way to do a quasi-experiment to address this issue might be to randomize different midwives or different wards (assuming they are many) to joining the feedback intervention at different times in a stepped-wedge design. If individual nurses are randomized, one could even put each nurse on and off the intervention at different times. For example, in week 1, you give feedback to nurse 1, and no feedback to the rest, in week 2, nurse 2 joins the intervention, etc., and perhaps by week 10, when all nurses have joined, you could begin taking them off the intervention, one by one. The same design could be used to study this question in a sample of many maternity units in this part of the country so as to address sample size issues.
Despite any problems that might be encountered, the quasi-experiment, in my opinion, is always going to be more informative than any form of traditional observational study in dealing with this system.
Policy makers, I think, are always interested in ITT estimates. In the above example, if results were going to be used by the hospital administration to decide whether the feedback program should be mandated on all maternity wards, I assume, the administrators would want to know the ITT estimate. I guess the problem with an IV estimate is that it tends to be a LATE; I would assume that patients and clinicians might be more interested in LATEs.