Michael, and other faculty:
One of the students in section asked a great question (unanimously) about calibration. Specifically, about when would it NOT be appropriate to re-calibrate? (vs. when it is?)
Please share your insights. Thanks.
-Luis
It seems to me that your initial net benefit calculation should use the original (unrecalibrated) risk estimates. If you are comparing the performance of two different risk models (or weathermen or robots) in a dataset, it doesn't make sense to use that dataset to re-calibrate the risk models. That seems like cheating -- looking at the answer key before taking the test. Depending on your misclassification cost ratio C/B and the severity of miscalibration, one or both of the risk models may have negative net benefit or (which I didn't mention on Thursday) have net benefit less than the "Treat All" strategy.
However, once you have done your initial net benefit calculation, you may have to choose a strategy (i.e., risk model to use) going forward. Then, you should recalibrate the risk models and choose the one with the higher net benefit, which should also be the one with the better discrimination. For a given C/B, a recalibrated risk model should have a net benefit >= 0 and be at least as good as the "Treat All" strategy.
To summarize, I think you should do an un-recalibrated net benefit calculation first. Then, in planning future risk predictions, you should recalibrate.
I will look around to see what else I can find on this question.