Reading Response Discussions

When running a regression model to do analysis, what should we be worried about?

We are usually worried about correctly specifying the model and positivity violations, especially when estimated the causal effect.  

Why use TMLE instead of regression?

Targeted maximum likelihood estimation targets the MLE estimate of the parameter of interest in a way that reduces bias.

Why use TMLE instead of inverse probability of treatment weighting?

When the model is correctly specified TMLE is more efficient. The TMLE estimate will also be more consistent even when the model is not correctly specified. IPTWs are more sensitive to positivity violations.

Are there instances when we wouldn't want to use TMLE?

When there are positivity violations and the model is misspecified, TMLE will not necessarily be more efficient than IPTWs.

What are marginal effects? What are conditional effects? Which do we usually report? Which are the most useful?

The marginal effect is the average causal effect in the population:

E[Y^a=0 =1] - E[Y^a=1 =1]

The conditional effect is the causal effect conditional on some set of measured covariates. This is typically what we estimate in regression analysis when we interpret the coefficient for the variable of interest “X increases the risk of Y holding all other variables constant”.

In public health and causal inference we are really interested in the marginal effects because they tell us about the causal effect in the whole population. Clinicians may be more interested in conditional effects because they give information about effects given a certain set of covariates, which can help inform decisions about treating individual patients.

Can you think of applications in your field in which TMLE might be helpful? I’m looking forward to learning more about TMLEs in class to better understand when they would be useful in my field.  

When running a regression model to do analysis, what should we be worried about?

I worry about many things for any regression model:

1. whether I selected the correct predictors
2. whether I adjusted for the correct covariates
3. whether I checked the model, like linearity, normality, constant variance, influential points and covariate overlap
4. whether I drew the correct directed acyclic graph (e.g., avoid adjusted for mediators or colliders)
5. whether I should include any interaction terms
6. whether I should account for clustering

Why use TMLE instead of regression?

Per Gruber and van der Laan’s 2009 article, TMLE “targets the MLE estimate of the parameter of interest in a way that reduces bias” (section 3).

Why use TMLE instead of inverse probability of treatment weighting?

Per Rudolph et al.’s 2014 article (page 741) – Since TMLE is doubly robust, TMLE will be consistent under misspecification of the treatment of subsample selection models if the outcome model is correctly specified. Whereas, IPW estimation relies exclusively on the IP weights to account for nonrandom subsample selection and nonrandom treatment assignment. This is why TMLE is advantageous over IPW under misspecification of the treatment of selection models.

Are there instances when we wouldn't want to use TMLE?

Per Rudolph et al.’s 2014 article (pages 744-745) – In situations in which the outcome regression model was misspecified to exclude treatment effect heterogeneity, TMLE isn’t more efficient than IPW estimation. Under practical positivity violations and model misspecification, TMLE isn’t expected to have smaller CI width than IPW estimation.

What are marginal effects? What are conditional effects? Which do we usually report? Which are the most useful?

Per Vittinghoff et al.’s 2012 “Regression Methods in Biostatistics” Textbook (page 285) –

1. Marginal effects – A model that holds averaged over all clusters, which is called population averaged. Coefficients are interpreted as the average change in the response (over the entire population) for a unit change in the predictor.
2. Conditional effects – A model specific to each cluster, which is called subject-specific. Coefficients are interpreted as the change in the response for each cluster in the population for a unit change in the predictor.

Marginal effects are usually reported. What is useful depends on the scenario. As a physician, I might be more interested in conditional effects when treating patients at an individual level. As a clinical researcher, I might be more interested in marginal effects when thinking about a substantive issue at a population level.

Can you think of applications in your field in which TMLE might be helpful?

Building off of Rudolph et al.’s 2014 article where the authors looked a large national surveys, there are similar national surveys in the cognitive field & in the substance use disorder field. Examples are:

1.  National Alzheimer’s Coordinating Center’s Uniform Data Set, which is an observational study. https://www.alz.washington.edu/WEB/data_descript.html  I think TMLE might be relevant there to look at some substantive question. The same issues of nonrandom treatment assignment, nonrandom subsample selection, and positive violations are there in this longitudinal dataset.
2. I am sure TMLE could be used in various national substance use datasets, such as the National Survey of Substance Abuse Treatment Services, the Treatment Episode Dataset, the National Survey on Drug Use and Health, and the Drug Abuse Warning Network. http://www.icpsr.umich.edu/icpsrweb/SAMHDA/browse

(1) When running a regression model to do analysis, what should we be worried about? 

We should be concerned that model assumptions are met and that the model is correctly specified.

(2) Why use TMLE instead of regression? 

TMLE would be preferred over regression modeling since it allows for the estimation of causal effects. Conversely, using regression modeling gives us conditional effects.

(3) Why use TMLE instead of inverse probability of treatment weighting? 

TMLE may prove to be more robust in instances where the model is misspecified. 

(4) Are there instances when we wouldn't want to use TMLE? 

In situations where the regression model was misspecified to exclude treatment heterogeneity (i.e. differing probabilities due to effect modifiers), TMLE is not more efficient IPTW estimation. In fact, the 95% CI from TMLE may be wider than those calculated from IPTW.

(5) What are marginal effects? What are conditional effects? Which do we usually report? Which are the most useful? 

Marginal effects – The estimates are averaged over the population (i.e. effects at the margins of the population)

Conditional effects – The estimates are conditional on a specific covariate patterns (i.e. effects are true for those who have particular values for these covariates) 

We typically report conditional effects from regression modeling in our analyses. However, reporting causal effects may be more useful. In particular, the average causal effect has more public health and clinical relevance.

(6) Can you think of applications in your field in which TMLE might be helpful?

It seems like TMLE is appropriate for models with large nuisance parameters. The method reduces bias and variance for the parameter of interest at the expense of nuisance parameters. This suggests it may be useful for longitudinal cohort studies in which there may be many covariates in the model. For example, if we have an outcome such as MI and want to look at one specific factor, TMLE may prove useful since there are typically many covariates in models that assess causal relationships for MI. 

1. When running a regression model to do analysis, what should we be worried about?
   • Regression to the mean
   • Internal and external validity
2. Why use TMLE instead of regression?
   • Minimizing global measure such as mean squared error (MSE)
   • Reducing bias
   • Increasing variance of the estimate
3. Why use TMLE instead of inverse probability of treatment weighting?
   • Being robust
4. Are there instances when we couldn’t want to use TMLE?
   • Misspecification of treatment
5. What are marginal effects? What are conditional effects? Which do we usually report? Which are the most useful?
   • Marginal effects
   • Being unbiased
6. Can you think of applications in your field in which TMLE might be helpful?
   • I hope we will learn that

1) When running a regression model to do analysis, what should we be worried about? 

Violating model assumptions including positivity violations and that groups conditional on covariates are not exchangeable and misspecifying the model. 

(2) Why use TMLE instead of regression? 

If you are interested in one specific parameter, TMLE produces a more efficient estimator by targeting the MLE estimate of the parameter and reducing bias.

(3) Why use TMLE instead of inverse probability of treatment weighting? 

Inverse probability of treatment weighting is more sensitive to positivity violations and in that situation produces weights that are highly variable, and in the face of model misspecification TMLE produces lower MSEs.

(4) Are there instances when we wouldn't want to use TMLE? 

When there are major positivity violation or model misspecifications, TMLE does not always perform better than IPTW.

(5) What are marginal effects? What are conditional effects? Which do we usually report? Which are the most useful? 

Marginal effects are population level estimates while conditional effects are estimates conditional on covariates. We usually report conditional effects in standard research papers. Marginal effects are more useful if we are truly interested in generalizing our inferences to populations outside of our sample, while conditional models can be useful for individual.

(6) Can you think of applications in your field in which TMLE might be helpful?

I'm interested in adaptive RCTs and it seems like TMLE is used in this situation but I don't think I fully understand how, so I hope to learn more about that tomorrow.

When running a regression model to do analysis, what should we be worried about?

• Conditional exchangeability--how to achieve it? That is, which covariates to adjust for.

• Positivity/ETA violations--i.e. model extrapolation

• Model misspecification

Why use TMLE instead of regression?

• If the parameter of interest is borderline identifiable and there is a theoretical ETA violation (i.e. theoretical positivity violation), regression would tend to give overly narrow variances, whereas TMLE is non-model based and will give much wider variances. That is, TMLE will correctly discern that the parameter is borderline identifiable. Also, TMLE is doubly robust meaning that estimators are consistent if either the outcome model or weights are correctly specified.

Why use TMLE instead of inverse probability of treatment weighting?

• TML estimators may be more efficient and robust that IPW estimators when the model is correctly specified and under many model misspecifications.

Are there instances when we couldn’t want to use TMLE?

• If there are positivity violations, CTMLE may be able to remedy the situation and produce unbiased estimates by excluding covariates that lead to these positivity violations.

• TMLE usually performs better that IPW under most model misspecifications. However, there are certain model misspecifications--excluding heterogeneity in treatment effect when it in fact exists--in which IPW will perform better. This is not a far-fetched scenario.

What are marginal effects? What are conditional effects? Which do we usually report? Which are the most useful?

• Marginal effects: the average causal effect of treatment in a sample or population of interest

• Conditional effects: the causal effect of treatment in individuals with a particular set of covariates

• Conditional effects are more frequently reported, but marginal effects may well be of greater interest, particularly if we want to know whether at a population level a treatment expected to be harmful or beneficial. Conditional effects may be useful for targeting specific subpopulations, however.

Can you think of applications in your field in which TMLE might be helpful?

• I use maximum likelihood techniques frequently in my field with MCMC techniques to estimate credible intervals. However, I can imagine instances where we would be interested in estimating certain parameters very well, and don’t care much about the others.

(1) When running a regression model to do analysis, what should we be worried about?

In observational data when estimating Population Average Treatment Effect (PATE), we should be worried about:

• Unmeasured or measured confounding present in the data causing lack of exchangeability.
• Violation of Positivity – does your data set have enough information contained in it to accurately represent those who are more likely or less likely to be treated? If not, then weighting may cause individual’s exposures to be weighted to extreme values and thus cause bias problems.
• Generalizability – A non-random sample may cause issues with external validity.
• Selection bias -- for example subsample effects may not be generalizable if the selection probabilities depended on the effect modifiers within your data
• Effect Heterogeneity—The effect of treatment may vary within your sample

(2) Why use TMLE instead of regression?

If you were interested in one particular parameter of the data distribution and considered the other parameters to be nuisance parameters, you might use TMLE instead of regression. For example you have a target parameter that you would rather have an estimate that has smaller bias and variance at the expense of increased bias and/or variance of the nuisance parameters.

(3) Why use TMLE instead of inverse probability of treatment weighting?

If your models are correctly specified TMLE outperforms IPW models in terms of variance and  MSE. If you have misspecification of the treatment or selection models, a TMLE model out performs IPW models.  This is because IPW estimation relies exclusively on the IPW weights to account for nonrandom sub sample selection and nonrandom treatment assignment. TMLE is “doubly robust” so they will be consistent under misspecification of the treatment or subsample selection models if the outcome model is correctly specified.

(4) Are there instances when we wouldn't want to use TMLE?

If you have a practical positivity violation, sometimes using TMLE will not work as well as a DRWLS (doubly robust weighted least squares) model, because the DRWLS model uses the combined treatment-selection weights for the treatment and selection conditions that were actually observed, whereas TMLE uses the combined treatment-selection weights for the observed and unobserved counterfactual treatment and selection conditions.  If individuals usually receive their most likely treatment and selection assignment, then using the counterfactuals can result in greater positivity violations and thus poorer performance of TMLE as compared to DRWLS.

(5) What are marginal effects? What are conditional effects? Which do we usually report? Which are the most useful?

Marginal effects (aka causal effects) are the effect the treatment has on your population if everyone in your population was treated compared to if everyone in your population was not treated.

Conditional effects are the effect of the treatment conditional on a baseline covariate’s value being held constant.

We usually report the marginal effects.

Marginal effects are useful for public health implications to understand what an intervention will have in the real world. Conditional effects are useful in clinical settings when you are interested in knowing what effect a treatment will have for a patient with a specific set of conditions (usually values of your baseline covariates).

(1) When running a regression model to do analysis, what should we be worried about?

We are typically worried about whether we have correctly specified the regression model to estimate the effect we are interested in, and misspecification can be due to a number of reasons that include: wrong assumptions about relationships between exposure and outcome (i.e. true relationship is linear but we assume a linear relationship), missing factors from the model (i.e. unmeasured confounding or missing interactions between factors), or any other factors that could account for unexplained variance. If we are also interested in a causal effect, we would also have to consider whether we have any assumption violations for the assumptions of exchangeability, positivity, or consistency.

(2) Why use TMLE instead of regression?

You do not need to make as many assumptions when using TMLE compared to regression, in which you assume that your model is the correct model to describe the effect. This is because the regression approach uses information from the data you observed to extrapolate to describe the data you did not observe (i.e. by using a model), and to correctly extrapolate, you would need to assume that your model is correctly specified. Any violations in your modeling assumptions, which includes no model misspecification, will result in a biased estimate of the effect with an estimated variance that is smaller than the true uncertainty in your data. In contrast, the TMLE uses a semi-parametric model that doesn’t require extrapolation, it tries to minimize the mean squared error for the parameter of interest and will give an unbiased estimate of that parameter which may also result in a larger estimated variance but this may more accurately describe the true uncertainty in your data.

(3) Why use TMLE instead of inverse probability of treatment weighting?

The TMLE is more efficient than the IPW estimators approach in that they minimize the estimated variance while giving unbiased estimates. TMLE will also be more consistent than the IPW even in the presence of model misspecification, in contrast IPW is very sensitive to model specification since the entire analysis relies on correct estimation of the weights. The IPW approach is also vulnerable to positivity violations and can result in biased or overly confidence variance estimates, while TMLE is theoretically more robust even when there is a positivity violation.

(4) Are there instances when we wouldn't want to use TMLE?

When comparing TMLE to IPW and DRWLS methods, Rudolph et al 2014 found that TMLE did not outperform IPW (in terms of estimating a smaller variance) when the model for the outcome was misspecified due to a missing interaction effect that resulted in treatment effect heterogeneity.

(5) What are marginal effects? What are conditional effects? Which do we usually report? Which are the most useful?

Marginal effects typically refer to a population-average effect that describes the association across an entire population; the comparison is between the average risk for the outcome in a population when everyone was exposed versus the average risk for the outcome in a population when no one was exposed. In contrast conditional effects typically refer to subject-specific effect that describes the effect for a particular person; the comparison is the risk for that person if they were exposed versus the risk for that person if they were unexposed. Most often we are reporting conditional effects (i.e. odds ratios, risk ratios, etc). In terms of usefulness, it could be argued that for public health or other population based scientists (i.e. research for policy makers), where you hope to make predictions about the entire population, the marginal effects are more useful since it will take into account the distribution of covariates within your population. However some might argue that if you want to make a prediction about an individual, a conditional effect might be more useful. 

(6) Can you think of applications in your field in which TMLE might be helpful?

In genetic epidemiology TMLE is a useful tool since often we are trying to estimate the effect of a single genetic variant while treating all other genetic markers in the genome as nuisance parameters, especially in the case of genome-wide association studies. This results in a large number of nuisance parameters we are not interested in, so TMLE could potentially give a tighter confidence interval for the genetic variant we are interested in, which would be extremely helpful since many genetic effects are quite small (and close to the null) and a precise estimate is required to correctly estimate whether there is a true effect (i.e. if confidence interval does not cross the null). TMLE is especially appealing because it blends both machine learning and causal inference approaches which gives interpretable estimates while also acknowledging limitations of the data that you have. Many groups at UC Berkeley and the Broad Institute are currently developing TMLE methods for genome-wide data.