Roya's assignment 6

Roya's assignment 6

by Roya -

Hi class. I'm in New York for a meeting (related to this project, specifically), but here is my assignment 6. As per our discussion last week (and thank you everyone so much for all of the help) I've completely restarted my protocol. Since it was a lot to redo everything from the start, it's on the rough side, but I'll refine it over the next few weeks. Also pasted below is assignment 6 so that you don't have to scroll all the way down to read it. Thank you!

 

PROGRAM EVALUATION

Thinking about the protocol you are developing, identify the process and outcome indicators associated with the intervention/program and briefly describe an approach to measuring each.

Outcome variables and how they will be measured

  • Adoption of tool (quantitative outcome) will be measured by calculating change in frequency of clinician accessing and completion of registry form over time (3, 6, and 12 months), which will be measured as follows:
    • Frequency with which clinician accesses the tool: # of times clinician accesses tool/# of patient encounters by clinician
    • Frequency of clinicians’ completion of registry form: # of times clinician completes registry form/# of patient encounters by clinician

The numerator in each equation will be accessed via the registry database, and the denominator in each equation will be accessed via the web-based health records at each site.

  • Barriers/facilitators to implementation of tool (qualitative outcome): Participants at the three practice sites will be given the opportunity to participate in two focus group interviews designed to understand barriers/facilitators to adoption of registry tools.


Define one or more “intermediate” outcome measures [reflecting changes in environment, organizational culture, systems of care, patient or public behavior, and/or clinician behaviors] that can inform you about the mechanism by which your intervention achieves its downstream effect on health and inform you about the acceptability of your intervention. 

Practices engaging in quality improvement processes as a result of feedback from data collected through the registry.


Identify a mixed methods study design and briefly describe the quantitative and qualitative data you will collect for program/intervention evaluation.

Study design: This will be a multi-site, mixed methods observational pilot study in a prospective cohort design.

Timing:Clinicians participating in registry use will be followed for 12 months. The study will begin when the registry has been integrated into the three home-centered practice sites involved in Stage 2 implementation. The study will end when the last participating clinician has been followed for 12 months.

Study participants: All participants will be clinicians (physicians, nurse practitioners, physician assistants) atsix home-centered primary care practices that have already been recruited to pilot the implementation of the quality measures and data registry. These sites reflect a diversity of practice types (rural/urban, academic/community-based, etc.) and are highly motivated to test the measures.

Outcome variables and how they will be measured : See above 

Re: Roya's assignment 6

by Christina Mangurian -

Dear Roya

Glad you have restarted your project to be more in-line with what you are doing.  No worries that it is “on the rough side,” that’s to be expected.  Hope you are having a great time in NY!

Q1: I think we need to step back a bit and figure out what is the real outcome indicator.  As Adithya wrote, these are the distinctions between the different indicators:

Impact indicators: Overall intervention/program goals (e.g., improved health, improved quality of life, and/or reduced healthcare costs).

Outcome indicators: Intervention/program objectives (e.g., behavior changes; program components you want to implement)

Process indicators: Whether activities you propose are done (e.g., if training is one of the activities, process indicators might include how many providers/patients attended the trainings and whether the training changed knowledge/attitudes/self-efficacy). 

Try using this outline to frame your outcome and process measures. 

Q2: You are thinking right in making a logical argument where a provider changes their behavior as a result of some activity you did—but you want to be even more specific and make it between “process indicator” and “outcome indicator.”  So, for example, if your goal (outcome) is to increase screening of the 5 measures, an intermediate measure might be some of your current “outcome measures” (e.g., frequency with which clinician accessed registry).  These intermediate measures give us a sense of how your intervention really impacts your outcome.

Q3: Once you have reworked Q1&2 your quantitative data collection will be clear.  Here, I’d describe more about your qualitative design—who are you going to talk to, how many, how are you going to talk to them (e.g., individual interviews vs focus groups)

Re: Roya's assignment 6

by Roya -

Thank you Christina, this is very helpful feedback. I incorporated your suggestions into my updated protocol so let me know if my changes make sense/are correct when I submit the next assignment. 

Re: Roya's assignment 6

by Elizabeth -

Hi Roya,

I looked at your revised protocol that you just posted, so I'll comment on that. Nice work doing a massive overhaul!

Stylistic point - I think it might look nice and be easy to think about each set of measures (Process/Outcome/Impact) if you make a table with the measure in one column and the data source in the next. This helped me really focus on - what is the process i'm looking at and how will i know if it is working - then put the right measure into that column.

For your process measures, in addition to provider satisfaction, I wonder whether you can include a time variable, like weekly time spent entering data into registry, # of program glitches/computer snafus encountered by individuals trying to enter data, # of hardward issues encountered by individuals trying to enter data, # of patients for whom data was entered in a 1 month period/eligible # of patients eligible to have data entered, lag time between patient encounter and data entry, and proportion of entered data that is valid at each site. These will help you understand whether there are real on the ground issues that are barriers. You will likely identify more of these domains with focus groups.

Good luck.

 

Re: Roya's assignment 6

by Roya -

Those are great ideas! Those are all data points (or similar to) that we are already collecting and keeping track of, so will incorporate into my protocol. Thanks!