Protocol Assignment Forum: Program Evaluation

1.     Considering the protocol you are developing: identify the process and outcome indicators associated with the program and briefly describe an approach to measuring each

Process measures

• Proportion of patients with documentation of annual metabolic screening visit by their psychiatrist (AVATAR)
• Proportion of psychiatrists who watched the on-line CRANIUM metformin education videos.
• Proportion of patients with abnormal metabolic screening who had an eReferral request (eReferral)
• Satisfaction with metabolic screening visit and eReferral (psychiatrist survey)
• Satisfaction with and acceptability of metabolic screening visit (consumers)

Outcome measures*

• Proportion of patients with annual glucose, lipid, and other metabolic indicators testing (AVATAR)
• Proportion of patients with a diagnosis of DM (but not on metformin at baseline) who were prescribed metformin or any other glucose lowering medication by their psychiatrist or PCP (in AVATAR)
• Proportion of patients with high HgA1c who were prescribed metformin or any other glucose lowering medication by their psychiatrist or PCP (AVATAR)
• Proportion of clients who visited a primary care physician at least once in the past year.

*For each of the outcome measures, we will stratify by whether or not their physician had watched the videos and/or visited the CRANIUM website.

2.     Define one or more “intermediate” outcome measures (reflecting changes in environment, organizational culture, systems of care, patient or public behavior, and/or clinician behaviors) that can inform you about the mechanism by which your intervention achieves its downstream effect on health inform you about the acceptability of your intervention

I think there are two “intermediate” outcome measures that might inform me about the mechanism by which CRANIUM achieves its downstream effects. 

• Proportion of psychiatrists who know the indications and side effects of metformin (on-line survey)
• Proportion of psychiatrists who visited the CRANIUM website (which has the metformin algorithm)
• Proportion of psychiatrists who made eReferrals

The CRANIUM Protocol to Improve Diabetes Screening and Preliminary Treatment in Community Mental Health Clinics

Translating Evidence

A.     What evidence are you proposing to translate into practice?  I am proposing improving annual metabolic screening of people taking second-generation antipsychotic medications, who are being served in community mental health clinics.  I may try to find other models in the HIV literature where they are also trying to improve metabolic screening

1. 1.       Justify that this evidence is “ready for translation.”  The evidence is ready for translation because there have been national guidelines recommended by the American Diabetes Association and the American Psychiatric Association (ADA 2004).  Multiple RCTs and observational studies have shown that most second-generation antipsychotic medications increase metabolic risk.  Expert opinion recommends this annual screening given this increased risk.
2. 2.       Identify a single, key behavior change target for your translational activity. The behavioral target is annual metabolic screening by psychiatrists in community mental health clinics.  Basically, I want to increase the number of patients who obtain guideline-recommended annual metabolic screening in community mental health clinics.  At minimum, this would involve annual BMI, blood pressure, and fasting lipid/glucose monitoring.  Notably, I also plan to link screening to treatment, so will be encouraging preliminary medication treatment (e.g., statins) for any abnormal screening tests.
3. 3.       Conduct a “gap analysis” of your target behavior.  Look to diverse sources for “best guess” estimates if specific measures are not available. 10 years after the ADA/APA guidelines, metabolic screening rates are still only about 30% (Morrato 2010; Mangurian in manuscript form).

B.  What is the quality (performance) gap?  Only 30% of people with severe mental illness receive metabolic screening by their physicians (Morrato 2010).  Therefore, there is a 70% gap in quality of care for this vulnerable population.  Ideally, I would like to eliminate this gap altogether.  Through my intervention, I aim to realistically reduce this gap for an additional 30% of the population, leaving a gap of 40%.

C.  What is the outcome gap? People with severe mental illness taking these medications die 25 years earlier than the general population (Colton 2006).  Adjusting for age, among Medicaid recipients in six states, the number of deaths among people with SMI ranged from 1.2-4.9 times higher than the general population (Colton 2006).  The proportion of deaths among due to heart disease were comparable between the general population and those with SMI (~30%) (Colton 2006)

D.  Is there evidence that changing performance will improve health (clinical outcomes)?  There is strong evidence that early detection and treatment of prediabetes, diabetes, hypertension, and dyslipidemia will reduce cardiovascular disease.  Unfortunately, there is no evidence to date about this prevention effort among people with severe mental illness.

References:

Colton CW, et al.  Congruencies in increased mortality rates, years of potential life lost, and causes of death among public mental health clients in eight states. Prev Chronic Dis. 2006;3(2):A42.

Morrato EH, et al.  Metabolic testing rates in 3 state Medicaid programs after FDA warnings and ADA/APA recommendations for second-generation antipsychotic drugs.  Arch Gen Psychiatry 2010; 67(1):17-24. 

American Diabetes Association, American Psychiatric Association, American Association of Clinical Endocrinologists, North American Association for the Study of Obesity. Consensus development conference on antipsychotic drugs and obesity and diabetes. Diabetes Care 2004; 27: 596– 601.

Mangurian C, et al.  Diabetes screening and prevalence among a cohort of Medicaid recipients with severe mental illness in California. In Manuscript form.

Community Engagement

1. 1.       Define the communit(ies) for your project and explain why each is a stakeholder for your study.

• Psychiatrists: These are the people who will be asked to change their behavior.
• Primary care providers: These are the people who will be referred to and potentially have their “turf” challenged.
• Consumers (patients) with SMI:  These are the people who will benefit from the improved primary care.
• Caregivers of people with SMI: These people are obviously invested and concerned in the well-being of their loved ones. 
• Clinical Diabetes Educators: This group can provide Medi-Cal billable services.  They could help navigate and link folks with DM to primary care providers.
• SF DPH administrators: These are people who will have to “OK” this proposal and this expansion of focus.
• Payers: These are the people who will pay for the services provided.

1. Describe your plan for approaching potential community partners to ask for their involvement.  Focus groups with consumers, psychiatrists, primary care providers, and administrators have happened.  I plan to conduct a stakeholder workgroup at the clinic comprised of the following people (all psychiatrists at the pilot clinic, a primary care provider, a patient consumer, a caregiver, and local leaders at the clinic).  I also need to establish a CAB, which will include a psychiatrist, primary care provider, SF DPH administrators, consumers, payers, and caregivers.  In approaching each of these partners individually, I plan to emphasize how they will benefit the project and how they will benefit specifically:

• Consumers (patients): The project will benefit by understanding the real barriers in care and opinions of the patients.  It will also be much more patient-centered by included their perspectives.  The patients will benefit from having a community clinic that addresses all of their needs (mental and physical) while they are quite ill.
• Caregivers:  The project will benefit by recognizing more of the role that the family provides in terms of this health.  The caregivers will benefit because the program will hopefully reduce cardiovascular risk in their loved ones.
• Psychiatrists: The project will benefit by being able to tailor the intervention to the needs of the psychiatrists (the main target of behavioral change in this study).  The psychiatrists will benefit by having a system of care that provides the supports they need to better care for their patients (e.g., reminders, primary care consultation, etc).
• Primary care providers:  This project will benefit from the content expertise of primary care providers and the increased capacity to build stronger collaborative relationships with this provider group.  The primary care providers will benefit by having more of their patients obtain metabolic screening and preliminary treatment—thereby being able to focus efforts on those with more serious illness.
• Clinical Diabetes Educators: They will benefit the project by sharing their experience linking people to DM care.  They will also be able to get reimbursed for their work.
• SF DPH administrators:  This project will benefit from this group’s ability to identify systemic hurdles that may need to be overcome (e.g., documentation of this medical service).  The administrators will benefit by having improved quality of care for this vulnerable population (streamlining), and improved patient satisfaction.
• Payers: Similarly, the project will benefit from this group’s ability to identify financial hurdles that may need to be overcome (e.g., billing ).  I will try to include potential cost-savings that may be realized by this streamlined approach, the improved quality of care for this vulnerable population, and the improved patient satisfaction.

1. Identify which stages of your project you'll incorporate community input, and describe what types of input you'll solicit.  I will need to incorporate community input throughout.  Specifically, I’ll plan on the following:

• Weekly stakeholder workgroup at pilot site.  I’d like input from the multidisciplinary workgroup (which will include patients) on various aspects of the proposal from design, implementation, and analysis.  They will give input throughout of regular hurdles we have to overcome.  When I ask them to participate, I will consider the main benefits that each of these individuals will provide to the discussion (e.g., knowledge of practical logistical hurdles from the patient and caregiver perspectives that providers may have overlooked).
• Quarterly CAB: I’d like input from the multidisciplinary workgroup on various aspects of the proposal from design, implementation, and analysis. 
• Monthly meetings with CBHS Medical Director, Irene Sung (this is already happening).  I’d like to incorporate input on the systemic barriers from these meetings. 
• Quarterly meeting with the CBSH Integration of Care group (during their regularly scheduled time).  I’d like to incorporate input on how this project fits into the overall plan to integrate care in CBHS.
• Quarterly meetings with DPH Primary Care Director (during their regularly scheduled time). I’d like to incorporate input on how this project fits into the overall plan to integrate care in CBHS.
• I’d also like to attend some of the integration of care meetings being held at DPH.  This is invitation only.

1. Name three ways you plan to share your results, beyond writing an academic article or presenting at an academic conference.  I plan to make my training tool publicly accessible.  I also plan to make the following presentations:

• Pilot clinic
• CBHS Medical Directors’ meeting
• DPH Integration of Care Meeting
• California Safety Net Hospital Institute annual meeting.
• California Association of Public Hospitals
• UCSF Center of Vulnerable Populations has the capacity to develop training programs and disseminate evidence-based interventions.  Specifically, Sarkar/Schillinger have Project PHoENIX (the Public Healthcare system Evidence Network and Innovation eXchange)
• San Francisco National Alliance of the Mentally Ill (community-level advocacy group)
• National Association of State Mental Health Directors annual meeting
• American Psychiatric Association’s Integration of Care Task Force annual meeting.

Mapping

1. Identify a patient or community group that contributes to or is involved in the principal behavior you are attempting to improve with your intervention.  Community psychiatrists are my target group.  I am attempting to improve diabetes screening and preliminary treatment for any abnormalities with metformin.
2. 2.       Using any of the individual explanatory theories in “Theory at a Glance”, develop an explanatory model for the target behavior (above) that you will be attempting to influence with your intervention.  The Theory of Behavior Change makes the most sense for this problem.  Although the health and behavior model specifies key concepts that are problematic for the psychiatrists (e.g., self-efficacy in knowing how to treat metabolic abnormalities), cues to action (reminders), and barriers (limited access to primary care)—I thought I could incorporate them into model below

1. 3.       Identify how one or more of your specific interventions will target one or more of these key factors contributing to the behavior of interest.

The social marketing campaign will be at the community clinic. 

I will likely have a CRANIUM poster (picture of Freud with a stethoscope).  Not sure I’m going to include pictures of patients.  Would be interested in figuring out how to “message” this topic.  The message would have to address:

• If you screen, the good thing that will happen are:
  • Patients will reduce their cardiovascular disease risk.  This improved health meets CBHS goals to care for the whole patient. 
  • This integration of care initiative could allow for more free flow of dialogue and help with management from primary care. 
  • This could improve patient satisfaction ratings. 
  • You will get rewarded—psychiatrists who have 70% of their patients with HgA1c will be in drawing for a free ipad.

• If you screen, the bad thing that might happen is that the patients could develop side effects from metformin (unlikely)

• If you don’t screen, the good thing that will happen is that you will have less work

• If you don’t screen, the bad thing that will happen are:

• Your patient will be at higher risk for early morbidity and mortality from cardiovascular disease.
• Your patient will be sicker and this will make it harder to manage their psychiatric illness.
• Patients and their families will be upset with you for not appropriately screening for side effects of their medications.
• P

1. 4.       Create a framework that draws upon a socio-ecological framework to orient your target behavior within a larger context.  ie, what are some of broader, external forces that influence the individual behavior of interest...see Figure 2 of “Theory at a Glance.”

Organizational Culture

Describe the organizational and/or delivery system environment in which your intervention will take place.

The pilot intervention will take place in one of the San Francisco County Department of Public Health’s Community Behavioral Health Service’s clinics, the UCSF Citywide Focus Clinic.  The cluster randomized trial will occur within all of the SF DPH CBHS clinics.

Based on Shortell’s 4 domains of organizational change, identify organizational barriers that could potentially impede successful implementation of your proposed intervention.

According to Shortell’s 4 domains of organizational change, the following are the organizational barriers that could potentially impede successful implementation of my proposed intervention:

1. Clinical Quality Performance: It was only recently recommended that CBHS medical directors monitor their staff’s performance of regularly monitoring of vital signs (height, weight, blood pressure) of clients annually (4/8/14).  There is no current formal QI monitoring of either metabolic screening or primary care physician.  As such, there is also no monitoring of the actual metabolic screening labs results (e.g., evidence of dyslipidemia or prediabetes/diabetes) or how well these abnormalities are being treated. This lack of monitoring at the systemic level could impede successful implementation of my proposed intervention.  However, I do believe that this type of quality monitoring is slowly becoming more commonplace in CBHS.
2. Patient Satisfaction:  It is complicated to obtain information about patient satisfaction from people with severe mental illness because the illnesses themselves can sometimes influence how they view “satisfaction” in a service. 
3. Organizational Learning:  There are several barriers to organizational learning that could influence my proposed intervention, including: a) Access to electronic laboratory results is currently unavailable to CBHS providers; b) Although there is usually little provider turnover in CBHS, there have been significant recent changes in leadership and staff given significant DPH transformation in the setting of health care reform; c) the CBHS wants to encourage adaptability, but I would say that most providers view being “adaptable” as getting overloaded with more work; and d) there is impaired communication across roles in DPH (Primary care and mental health), but this is improving with an Integration of Care group in CBHS.
4. Financial Performance: The fact that visits targeting management of metabolic disorders by psychiatrists might not be reimbursable is a barrier.  In addition, CBHS is chronically highly over-budget and threatens cutting services every budget season.

Using the same 4-domain model, describe how your intervention plan can take advantage of organizational strengths OR propose practical methods for addressing these barriers within your program.

1. Clinical Quality Performance: The CRANIUM plan to monitor metabolic screening performance at the provider and clinic-level will be an extension of the recent CBHS medical director’s mandate to monitor staff’s performance of vital signs (height, weight, blood pressure) (4/8/14).  CRANIUM also plans to provide some support (a registry) to help track this monitoring and make it easier for clinicians.
2. Patient Satisfaction: This study will specifically query patients on their satisfaction with the CRANIUM intervention.  In fact, if the CRANIUM intervention is preferred by patients, it could be extremely beneficial in this setting of health care reform.
3. Organizational Learning:  The CRANIUM intervention includes a Web-based clinical decision support tool that will facilitate easy entry of laboratory results and decisions on management.  CRANIUM also improves communication between roles in DPH (between psychiatry and primary care) via the PCP e-consult.
4. Financial Performance: CRANIUM will provide providers (and most importantly medical directors) with specific reassurance that performing these tasks (expanding their scope of practice) is billable and therefore will generate revenue for CBHS. 

A. Describe one tool that you will employ in your intervention strategy using the following domains....

1. Tool—type: Clinical Decision Support Tool.  This is CDS tool will be on a website that has four components: 1) An on-line, asynchronous video training on starting metformin for the community psychiatrist with “typical cases,”  2) Questions to test knowledge base before and after training; 3) Basic guidelines (algorithm) for management; and 4) E-mail for the primary care provider consultant. 
2. Target Population: Community Psychiatrists at Citywide Focus Clinic in San Francisco County
3. Target Behavior: Prescribing metformin for DM2
4. PRECEDE Category: Enabling
5. Platform: Website with “live” PMD (chat)

B. For a multi-tool intervention strategy, use the PER worksheet attached to describe how you will address each of the PRECEDE framework components (Predisposing, Enabling and Reinforcing). You may download this PER worksheet and/or copy/paste.

PER Worksheet 

• What level of government did you target to translate your research into policy and why did this make the most sense?

Initially, I was thinking of targeting the local (county) level of government—specifically the San Francisco County Department of Public Health’s Community Behavioral Health Service (DPH/CBHS).  This level made the most sense since I am testing out this intervention at this local level.  If CRANIUM is effective at one UCSF/SFGH pilot clinic, I plan/hope/wish to roll out as a cluster-randomized trial in the county.  Therefore, I need their buy-in from the get-go.  Also, within California, the Counties have much more control than the other State I’m more familiar with (NYS).  As such, CBHS seemed to make the most sense.  They also know me, an old fellow of mine is in a leadership role there, and it seemed likely to get my research translated into policy here locally as a first step.

• What level of government makes the most sense for you to translate your research into policy and why?

Again, I’m starting at the County level.  Next, I hope to move to the State level and then Nationally.  I believe starting locally where I had a good reputation with the policymakers made the most sense as a first-step.  I have contacts at the state level to facilitate translation here in CA, and also in NY and MO.  Finally, I have connections with leadership within large organizations (American Psychiatric Association and National Association of State Mental Health Program Directors) who might be able to help me translate my research into policy.

• What strategies did you use to reach policymakers? 

The policymaker that I have targeted in SF DPH/CBHS is primarily the Medical Director of CBHS.  I also am targeting the Director of CBHS and the Director of DPH.  The strategies that I have been using to reach these policymakers include:

1)     Find out what their (CBHS) goals are for advancing health in this area.  What do they see as their major challenges/goals and how does my work fit-in and/or contribute to these goals.

2)     Making my work locally relevant.  I have done surveys that were distributed all safety net primary care providers and psychiatrists throughout the DPH system (70% response rate).  I have also done focus groups of this same population, in addition to local administrators and patients.

3)     Sharing results of my work with them

4)     Meeting and producing in areas that are also useful to them to gain credibility.  Specifically regarding #3, I co-Founded and Director of the Public Psychiatry Fellowship which produced the fellow who is currently working at DPH.  They really like the fellowship since fellows produce both great QI projects that are formed from public-academic partnerships and have produced publications from their work.  I have also been serving as the Community Liaison for SFGH Psychiatry, so have helped improve communication between CBHS and SFGH.

• What steps are available to you to reach policymakers?

Practically speaking, I have good access right now.  I have monthly meetings with the CBHS medical director.  I also meet monthly with all medical directors of CBHS clinics.  I would like to start working closer with the new Integration of Care group through CBHS/DPH (I know the core 3 well and have a good relationship with them). 

Since I want to get CRANIUM implemented in Citywide focus this fall, I would like CBHS or Citywide policymakers to help me with:

1. Motivating this behavior (public endorsements and letters to providers)
2. Enabling this behavior (support costs for decision support tools in the EMR) or
3. Reinforcing this behavior (additional care management payments for the office visit).

Some additional steps I need to do are:

1. Do more education of the CBHS medical director on how this project will fit into the triple aim of health care reform by improving quality, reducing costs, and serving the population. 
2. Start having regular meetings with the Medical Director of Behavioral Health Homes since he will be the political champion here locally.
3. Develop a short policy statement
4. Transform my data from Medi-Cal into a form that is readily understandable to the policymakers.
5. Start writing about my work (or issuing a press release) for local media.
6. Engage more with NAMI to help activate the community based coalitions.
7. Confirm the scope-of practice for MDs in California from the Medical License Board to show that the scope of practice of physicians in CA includes prescribing metformin.
8. Get DPH/CBHS and SFGH credentialing for psychiatrists.  Again, this should be done to show that the scope of practice of psychiatrists in SFGH and the County
9. Talk to the APA Council on Advocacy and Government Relations to learn more about Medicaid reimbursements—specifically for “medical” visits by psychiatrists, giving medications to people with SMI, carve out, and whether the APA is advocating for FFS in the public sector or not.  Also ask about any relevant discussions about this in the Energy and Commerce and/or Ways and Means. 
10. Review CMS rules re: what you can bill for or not.  Public comments are in June.