- What evidence are you proposing to translate into practice?
Mental health modules can increase knowledge, skills and self-efficacy among primary care providers. My focus is on disseminating these modules to 11 government-owned, district-level hospitals in Nepal.
Justify that this evidence is “ready for translation.”
Continued Medical Education (CME) is a critical component of meeting gaps in provider’s knowledge and skills after completion of formal training. A needs-assessment study that I conducted in Nepal has shown that PCPs lack knowledge and skills for screening, diagnosing and treating patients with mental illness, largely because these topics are not covered in their professional schools and psychiatry is often an optional rotation. Preliminary data on effectiveness and feasibility of these modules are currently being generated from a district-level hospital in Nepal but they have been shown to increase knowledge, skills and self-efficacy. As a crucial part of a multi-component intervention, they can change provider behavior so that they can screen, diagnose and treat patients with mental illness. The next step is trying to disseminate these modules to 11 hospitals in Nepal.
- Identify a single, key behavior change target for your translational activity.
Primary Care Providers will complete all the modules in mental health at least once.
2. Conduct a “gap analysis” of your target behavior. Look to diverse sources for “best guess” estimates if specific measures are not available.
A. What is the quality (performance) gap?
CME credits are not required for maintenance of certification in Nepal. Very few organizations work in mental health so supplemental training is largely not available. When it is available, it requires providers to leave their clinical practice and travel to the capital city to obtain training. In the hospital I am currently involved with, only 1 out of 18 providers has received supplemental mental health training.
B. What is the outcome gap?
Patients with mental illness are not being diagnosed or appropriately treated. Most of such patients are receiving pain medications (because they present with pain and PCPs chronically give pain meds even when there is no improvement) or Vitamins (when PCPs suspect that mental illness is contributing to somatic symptoms and they believe vitamins will work as placebo and improve pt’s symptoms). The only patients who may be receiving appropriate care are those who have direct access to a psychiatrist. Most psychiatrists are concentrated in the capital and other major cities, which only covers about 20% of the country’s population.
The first step in changing provider’s clinical behavior is equipping them with the knowledge, skills and self-efficacy around caring addressing mental illness. For this study, change in these three variables is the outcome of interest.
3. Is there evidence that changing performance will improve health (clinical outcomes)?
This question (providing training leading to improving outcomes) has not been studied in mental health but similar studies have been able to link improved training with clinical outcomes (1). My current study has shown that participation in these modules increases provider skills, knowledge and self-efficacy about WHO protocols on diagnosis and management of mental illness.
Week#2
1. Define the community/communities for your project and explain why each is a stakeholder for your study.
One includes PCPs and the other includes the organization that is interested in helping scale up the intervention.
PCPs:
- At least 4 PCPs from the 11 district hospitals in Nepal. Ideally I would like to include 1 physician and 1 health-assistants (3 yr undergraduate medical training, similar to PA in the US).
- The PCPs will be the target of the modules and the study is attempting to engage them in conducting the change behavior (primary: taking the modules, secondary: screening, diagnosing and treating mental illness).
- The sites are in rural Nepal so a larger sample of representative PCPs is less likely to be feasible.
Leaders of NSI
- NSI provides training, monitoring and support to the 11 district hospitals. They measure quality of care of the hospitals.
A note about engaging patients: Most patients are not thinking about their illness as a mental disorder and given the stigma of mental illness, most of them refuse a referral to a psychiatrist. This is a problem in the US as well. One way to approach this problem would be to discuss mental illness with patients, try to destigmatize it and encourage them to discuss psychosocial problems with their providers and other types of health-seeking behavior. We decided to start by focusing on the PCPs because we were worried that if we sensitize patients/communities and they present to PCPs wanting to discuss this, the PCPs are woefully unprepared to handle this (based on the focus groups that I did with them). So made sense to train PCPs first so they know how to screen/diagnose and then work with patients/communities in the future.
2. Describe your plan for approaching potential community partners to ask for their involvement.
- Formal partnership with NSI has already been established. They approached our team because they believe mental health training and support is lacking in the district hospitals in Nepal and are interested in addressing this with computer-based modules that we have tested at a different hospital in Nepal.
- To connect with the PCPs, I plan on asking the leaders of NSI to give me names of those who might be interested in discussing the research study.
3. Identify which stages of your project you'll incorporate community input, and describe what types of input you'll solicit.
Research Design:
- Current plan is to disseminate the online modules. However, I plan to ask providers what they think about the idea and to see if they have other thoughts on the most effective way to disseminate the training modules. To learn about this, I would ask the 4 PCPs the following questions: How prepared do you and other PCPs feel in screening, diagnosing and treating mental illness? What would be the best way to provide additional training on mental health for all the providers in several hospitals in Nepal without making everyone travel to the capital city? What has been your experience with other clinical trainings in the past? Which ones did you like and why? Which you not that much and why?
Refining the training:
- To ensure that the training is accessible, we will seek feedback from the 4 PCPs but sending them sample modules to get feedback around ease of use, style, delivery, language and content.
Informing the dissemination/scale up process:
- I would ask the PCPs what they think would be the prohibitive and enabling factors around uptake of the training.
Taking the modules:
- Finally, depending on the strategy we use (diffusion of innovation, social marketing etc), we would ask PCPs to help with dissemination of the modules.
4. Name three ways you plan to share your results, beyond writing an academic article or presenting at an academic conference.
- Providers would receive individual-level feedback on their performance of pre- and post-tests in the modules immediately after they complete it.
- NSI will be given a report of the summary of the results.
- The 4 PCPs will also be given a report summarizing the results. All PCPs will have the option of giving us their email address if they opt to be informed about the results. This email list-serve will be stored separately from the research study data.
Wk#3 Organization culture
- Based on Shortell’s article “An empirical assessment of high-performing medical groups: results from a national study", describe the organization/system in which your intervention will take place using the specific predictor measures within Environment, Resource acquisition, Resource deployment and Quality-centered care (Table 1).
Environment:
- All 11 hospitals are owned by the Nepali Government and they are required to meet several requirements: provide free care to the poor, stock on Nepali Government’s essential medications package, have at least one physician on staff and generate several reports like diagnoses, procedures performed, number of births etc. The Nepali government is also responsible for oversight of professionals (e.g. providing licenses for clinicians).
- These hospitals are supported by Nick Simons Institute (NSI), which provides support and training for the hospitals. It is developing protocols and trainings in commonly encountered clinical conditions. NSI also ensures that an MD-GP (similar to a family medicine physician in the US) is on staff at the supported hospitals. NSI provides electricity backup, computers and internet service to the hospitals.
Resource Acquisition:
- Most government-supported hospitals have minimal capacity to acquire resources. Creating new positions in one hospital is usually not possible because decision-making for all 75 district hospitals is centralized. Similarly, adding a new program or a new department can take years, with lot of back-and-forth with the government bureaucrats in the capital city.
- NSI has the ability to acquire additional resources for the 11 hospitals that it supports. It has a track record of responding to the needs of the patients and clinicians e.g. it has trained certified anesthesia assistant and purchased anesthesia equipment allowing lot of life saving surgeries in rural Nepal. It has now established a yearly stipend for the supported hospital to purchase equipment.
Resource Deployment
- Lot of clinical and non-clinical positions in government hospitals are vacant due to inability to recruit and those who are listed as employees may still be absent (running a private practice on the side, spending majority of time in an urban center either for training or for unauthorized extended leaves).
- However, NSI has been lot more successful. NSI has been able to increase staff retention and decreased absenteeism (compared to government hospitals not supported by NSI) by supporting the rural healthcare staff e.g. it sponsors education of MD-GP’s children in private schools in the capital, helps spouses obtain employment in the hospital and pays close attention to job satisfaction among clinicians to retain them in the rural hospitals.
- There is variation among the 11 hospitals but all of them have at least one MD-GP and 3 more physicians and several non-physician providers. All of them have nurses, a pharmacist, an in-house pharmacy that stocks most of the essential medications.
Quality-centered culture
- The Nepali Government is largely focused on collecting aggregate numbers like total pts served, total surgeries completed, total medications dispensed etc rather than process-focused data.
- NSI does have various programs in quality improvements so clinicians so these hospitals do have some level of quality-centered culture. However, the traditional roles of clinicians and other staff in the district hospitals leave little room for protected time to collect process-level data so quality improvement initiatives still tend to be top-down and not necessarily sustained by internal staff.
2. Discuss which of these factors might serve as barriers to successful implementation of your intervention, and how you will try to design an intervention to overcome these barriers.
Environment:
Barriers: Although these 11 hospitals are supported by NSI, the Nepali Government ultimately has final say in all aspects of health services. In the past, political changes in the capital (e.g. a differential political party obtaining power in the Ministry of Health) have led to staff and policy changes in the rural hospitals. This might affect personnel/partners, which can stall dissemination efforts.
This can be overcome by seeking partner staff who have a track record of being in the same hospital for many years, working with mid-level providers, who are less likely to be affected by political changes in the capital (e.g. someone in the capital would be more interested in replacing the head of the hospital rather than replacing a mid-level provider). Additionally, working closely with NSI, which has an excellent track record of working with these hospitals, will help maintain stability during implementation.
Resource Acquisition:
If it turns out that the hospitals will need more computers to enable to the staff to take the CMEs, it will be tough to leverage the Nepali Government’s resources to make this happen. In this case, we would either use research budget to purchase handheld devices and would convince NSI of the utility of the modules so that they can use their funds to purchase computers. Additionally, if PCPs see the benefit of the modules, they might be convinced to use the equipment stipend to buy computers.
Resource Deployment:
Absenteeism and poor retention are major challenges in rural healthcare. Computer-based modules are therefore preferable because of low cost of training new personnel (traditional training usually happens in the capital requiring providers to close clinical services and travel to the city). Additional challenge from poor retention is that if we were to look at provider behavior in the future, it would be difficult to make a connection in a setting where the PCPs are already overburdened due to staffing shortages. The modules are therefore designed for mid-level providers and physicians because the former are usually more likely to remain in the hospitals for the long-term because they usually have family ties in rural Nepal and are also not in as high of a demand as physicians are in the capital city.
Quality-centered culture
Given that quality improvement initiatives are usually top-down, it will be very important to engage the PCPs early on to hear their views on rolling out the study. Several process-level data will likely be collected to assess the rollout to understand challenges and address them in the future (e.g. number of people who were able to conduct the study, challenges in accessing computers, power outages, internet outages, technical challenges in going through the module, performance on the pre and post-tests etc). Given that there are no staff members who have designated tasks to study quality, PCPs and other staff will hopefully find the modules helpful and become interested in helping to collect the data. As discussed earlier, 4 PCPs will be invited to review the modules and the plan for dissemination so that their suggestions and feedback can be incorporated in the study.
Wk #4 Mapping
- Identify an individual (e.g., patient or provider) or group (e.g., community group or organization) that contributes to or is involved in the principal behavior you are attempting to change. Specify the desired behavior change (who needs to change what, when, where and how)?
All PCPs at the 11 hospitals will be engaged in learning about WHO protocols for screening, diagnosis and basic treatment of mental illness.
- Using any of the individual explanatory theories in “Theory at a Glance”, develop an explanatory model for the target behavior (above) that you will be attempting to influence with your intervention. This can be an extension/based on previously published literature or your best guess of expected findings. Figures are always very useful... keep it simple. You can also use the logic model of the problem concept as presented in Bartholemew and Mullen (Five roles for using theory and evidence in the design and testing of behavior change interventions). Be sure to include components beyond the individual, such as suggested by the socio-ecological model.
|
Theory of Planned Behavior |
|
|
Behavioral intention
|
At this time, I would imagine very few PCPs intend to learn about the WHO protocols. This is because mental health is largely ignored in health professional schools and most PCPs may consequently not be aware that they ought to be screening and diagnosing mental illness. In focus groups I conducted, majority of PCPs estimated that prevalence of mental illness among their patients was less than 10%, with several estimating that it was about 1%. |
|
Attitude
|
The behavior of learning about the protocols will likely be seen in different light by different providers. Some have biases against mental illness and mental health providers and may view it negatively. Others may be neutral to slightly positive because PCPs are often expected to be knowledgeable about a wide variety of protocols. They would perceive this particular behavior as slightly important because it is indeed learning about WHO protocols but perhaps fairly low on their priority list. |
|
Subjective norm
|
Mental illness carries much stigma within the medical community so becoming a specialist in mental illness can be seen as something that is disapproved by most peers. There is perception that dealing with patients with mental illness can in fact make the provider mentally ill. Two specific circumstances are known to lead to desire to learn about mental health protocols: 1) Desire to seek residency training in the US, which requires knowledge of psychiatry to perform well on the USMLE exams and 2) Providers who have spent more than 1-2 years in rural Nepal, have not been able to effectively refer patients to be seen by a psychiatrist (because most of them are in the capital) and have realized that there is a subset of patients in their clinic who are just not getting better and return with the same set of complaints of aches and pains. |
|
Perceived behavioral control |
In the hospital where I have piloted this program, all clinicians are essentially required to attend the mental health training. However, this is not the case for the 11 hospitals where I would like to expand. Additionally, there are no requirements for maintenance of certification/gathering CME credits so the providers would likely believe that whether they engage in the behavior or not is entirely up to them. |
- Create your own version of Table 1 from Michie et al (Making psychological theory useful for implementing evidence based practice: a consensus approach) that reflects some of the different theoretical domains and interview questions, tailored to the behavior you want to change. Be sure to select several domains likely to be of interest based on the literature or your experience/best guess.
|
Knowledge |
- Do they know about the protocols? - Do they know the prevalence of mental illness among their patients? - Do they know which current clinical behaviors cause no improvement or cause harm? - Do they know the impact of untreated mental illness on other co-morbid conditions? |
|
Skills |
- Do they know how to access the protocols? - (If using computer-based modules on the protocols): Do they have the skills in using the computer to access the videos, take the pre-test etc? - (If already known or after learning the protocols): Do they have the interpersonal skills to conduct a screening/diagnostic interview? |
|
Professional role |
- Do they think following mental health guidelines is part of their role as physicians? - Do they think WHO protocols are relevant to their clinical/cultural context? |
|
Beliefs about capabilities |
- How comfortable are they with learning about protocols using computer modules? - How capable they feel about learning the protocols while also maintaining their busy clinical and other responsibilities? |
|
Beliefs about consequences |
- Do they think learning the protocols will have any impact on their knowledge? - Do they think the behavior will improve patient care? - What would happen if they didn’t engage in the behavior and never learned the protocols? |
|
Motivation and goals |
- Which rewards are or are not available if they learn about the protocols? - Will there be conflict between the protocols and what senior clinicians might direct them to do (e.g. with non-physician PCPs who are technically under the supervision of physicians)? |
|
Memory, attention and decision processes |
- How they usually learn about new protocols/guidelines? - How do they usually remember to spend time in self-directed learning? |
|
Environmental context and resources |
- What electronic or other modalities do they have to access the protocols? If using computers, is there equitable access among physician and non-physician PCPs? - Are there dedicated resources for such training (e.g. conference room with a projector where videos can be played rather than on personal computers) - Is there protected time during their work hours to engage in self-directed learning? |
|
Social influences (Norms) |
- Will respected/senior clinicians engage in the behavior, setting the norm for others? - Is learning about mental health the norm? - Is self-directed learning the norm? How do others do it? |
Wk #5 Implementation Frameworks
- Use the COM-B model to identify what needs to change in order for ONE of your selected target behaviors to occur. Note that TDF domains corresponding to each COM-B category are in parentheses, and that some of you may have already done this or aspects of this in Week 4.
|
COM-B Components |
What needs to happen for the target behavior to occur? |
Is there a need for change? |
|
Physical capability (Physical skills) |
Able to effectively navigate through the computer-based modules |
Majority of providers are comfortable because they use computers for education and social media. There might be few who are not comfortable and will need assistance. |
|
Psychological capability (Knowledge; Cognitive and interpersonal skills; Memory, attention and decision processes; Behavioral regulation) |
Know that the WHO protocols exist. Know the impact of untreated mental illness on functioning and also on outcomes for co-morbid conditions. |
Yes: providers who do not know these things are likely to not be interested in taking the modules |
|
Physical opportunity (Environmental context and resources) |
Have access to a reliable computer or other device able to run the modules |
Yes: not all providers may have such devices. Might have to purchase dedicated CME computers for each hospital |
|
Social opportunity (Social influences) |
Medical Director and other senior clinicians are using the modules and are encouraging others to do the same. |
Yes: need to get buy-in from senior clinicians in each hospital |
|
Reflective motivation (Professional/social role and identity; Beliefs about capabilities; Optimism; Beliefs about consequences; Intentions; Goals) |
PCPs believe that completing the modules will improve + bolster their identity as clinicians. They believe that learning the modules will actually improve patient outcomes and/or make their clinical work easier/more effective. |
Yes: Some clinicians still carry stigma against mental illness. |
|
Automatic motivation (Reinforcement; Emotion) |
PCPs enjoy going through the module |
No: feedback from the pilot group has been positive and that the modules are engaging. However, will need to continue collecting feedback and responding to it. |
|
Behavioral diagnosis of the relevant COM-B components: |
List the COM-B categories you want to target with your intervention Psychological capability Social opportunity Reflective motivation |
|
- Use the APEASE criteria to identify appropriate intervention functions based on the behavioral diagnosis (See Table 2.3 in Michie et al Chapter 2)
|
Candidate intervention functions
|
Is the intervention function needed based on the behavioral diagnosis? |
|
Education |
Y |
|
Persuasion |
Y |
|
Incentivisation |
Y |
|
Coercion |
Y |
|
Training |
Y |
|
Restriction |
N |
|
Environmental restructuring |
Depends on computer access |
|
Modelling |
Y |
|
Enablement |
Y |
|
Selected intervention functions: |
List the selected functions your intervention(s) will serve. Select based on APEASE criteria (affordability, practicability, effectiveness/cost-effectiveness, acceptability, side-effects/safety, equity).
|
3. For one of the intervention functions you selected, select a specific behavior change technique you will employ in your intervention strategy and specify how it will be delivered. See Table 3.3 in Chapter 3 of Behavioral Change Wheel (Michie et al) for list of most frequently used behavior change techniques for each intervention function.
a. BCW Intervention Function:
b. Behavior Change Technique:
c. Mode of delivery (i.e., intervention details):
Incentivisation:
Provide a certificate of completion that describes that WHO-approval guidelines were mastered by the learner. PCPs would be required to complete all the modules and will receive a confirmation page, which they can email back to receive a paper certificate. This can also have ripple effect if senior clinicians display their certificates then others may want to complete the modules, too.
Wk#6 Program Evaluation
- Thinking about the protocol you are developing, identify the process and outcome indicators associated with the intervention/program and briefly describe an approach to measuring each.
Process:
- Proportion of providers who started the modules vs all PCPs in the hospitals
- Proportion who completed them vs all PCPs in the hospitals
- Proportion who received a certificate of completion vs all PCPs who completed the modules.
- Proportion of providers who displayed the certificate vs all who received a certificate.
Outcome:
- Change in pooled average score from pre- to post-test in knowledge and self-efficacy around mental health.
2. Define one or more “intermediate” outcome measures [reflecting changes in environment, organizational culture, systems of care, patient or public behavior, and/or clinician behaviors] that can inform you about the mechanism by which your intervention achieves its downstream effect on health and inform you about the acceptability of your intervention.
- Providers’ beliefs about the utility of the modules
- Providers’ beliefs about the quality and level of engagement provided by the modules.
- Provider’s beliefs on how likely they are to change their behaviors after completing the modules
- Change in rates of screening for mental illness in the hospital
- Change in number of prescriptions for psychotropic medications.
3. Identify a mixed methods study design and briefly describe the quantitative and qualitative data you will collect for program/intervention evaluation.
Quantitative: measurement of the quantitative indicators listed in 1 and 2 above.
Qualitative: After 2 months of making the modules available, I will conduct three focus groups:
- Providers who never attempted the modules
- What did you think about the modules?
- What are your thoughts about mental illness?
- Providers who started but didn’t complete it
- What made you want to start?
- What made you want to stop?
- What would you like to see different?
- Providers who completed it.
- What made you start?
- What make you want to continue?
- Did certain thoughts (get ideas from prior focus groups) come to you mind and you felt like it was not worth it? If they didn’t, what would be your response to those thoughts (from prior FGDs)?
Wk#7 Analytical Issues
- Describe the study design you will employ in order to determine if your intervention has had an effect on the outcome variable of interest.
Stepped-wedge across all PCPs in the 11 hospitals.
2. Define the unit-of-analysis for your main outcome evaluation, the minimum meaningful effect size, and the sample size necessary to detect this effect size.
My understanding with clusters is that if you are looking at patient-level outcomes, then each cluster functions as one unit but since the outcome is change in individual scores measured by paired t-test, the unit-of-analysis in this case is the individual PCP.
To obtain an effect size of 0.5, at 80% power, beta error 0.2 and alpha error 0.05, total n needed will be at least 63.