Protocol Assignment Forum: Program Evaluation

Dear Steve,

This is great.  It’s really coming together. 

Q1.  As I’ve suggested with others, I’d consider making a table for this with columns being process/outcome measures and data source.  These kinds of tables make looking at your plan more accessible.  As Adithya wrote, these are the ways of separating out the indicators:

Impact indicators: Overall intervention/program goals (e.g., improved health, improved quality of life, and/or reduced healthcare costs).

Outcome indicators: Intervention/program objectives (e.g., behavior changes; program components you want to implement)

Process indicators: Whether activities you propose are done (e.g., if training is one of the activities, process indicators might include how many providers/patients attended the trainings and whether the training changed knowledge/attitudes/self-efficacy). 

So, in your case, impact indicators would be reduced cost, improved health; outcome indicators would be: referral to CDU; process measures would be number of times a provider views the clinical pathways in Apex, or proportion of time “superusers” are on shift; or # of people taking trainings. 

Q2:  I love that you are using these superusers to collect information on barriers.  That is a great idea.  But, I’d be careful about considering this an intermediate indicator.  You might think of intermediate as being between process and outcome.  So, if your process (e.g., activity) is # of people taking a training on CDU, you might have an intermediate outcome measure being proportion of people who know about the CDU.

Q2: You are thinking right in making a logical argument where a provider changes their behavior as a result of some activity you did—but you want to be even more specific and make it between “process indicator” and “outcome indicator.”  So, for example, if your goal (outcome) is to increase screening of the 5 measures, an intermediate measure might be some of your current “outcome measures” (e.g., frequency with which clinician accessed registry).  These intermediate measures give us a sense of how your intervention really impacts your outcome.

Q3: Great that you are looking at this with a mixed-method approach.  I’d try to flesh out the qualitative methods more here.  Describe how many doctors you will approach for how long an interview 90min.  Will it be MDs or RNs?   

Hi Steve,

I think you had the same confusion I had (and perhaps others) on outcome vs process indicators. I think the outcome you desire is just referral/use of the CDU right? The patient satisfaction and other metrics seem more like overall impact indicators. A process indicator could be related to provider training on the screening tools/indications for admission to the CDU.  A mixed method approach could also incorportate focus groups after a staff meeting or another way to efficiently collect data from physicians. I like the superuser idea too. Great work!

Emily

Hi Steve, This is coming along well. As others have provided input on the first question, I will limit my comments to the second part of the assignment. I think its great that you are considering the use of semi-structured interviews at several points in the project, are there opportunities to use more quantitative survey methods at these time points as well? For example, to help identify barriers you could administer a survey to most individuals in the department to ask about a broad range of barriers you anticipate based on what you have learned in your research and in the planning stages, you could then have one focus group with medical providers and another with support staff to investigate the 'top hit' barriers further and illicit input on potential plans to address those barriers. This allows the focus groups to be more informative and less likely to get pulled off track by an outlier.