Protocol Assignment Forum: Clustering, Time Series

Protocol Assignment 1

1. What evidence are you proposing to translate into practice?

I am proposing to develop a pediatric antibiotic stewardship program (ASP) to improve antibiotic prescribing at a tertiary care children’s hospital.

A. Justify that this evidence is “ready for translation.”

At least one third of all antibiotic use is inappropriate. Inappropriate antibiotic use is associated with adverse patient outcomes including multi-drug resistant infections, C. difficile infection, adverse drug reactions, and increased costs of care. The harms associated with inappropriate antibiotic use extend not only to the individual patient but to the entire inpatient clinical population that is placed at risk for infection with drug-resistant organisms.

Antibiotic stewardship refers to coordinated efforts to improve antibiotic prescribing, with core strategies consisting of prospective audit and feedback to prescribers, and restriction of high risk, high cost and/or broad spectrum agents. The ability of ASPs to reduce costs and improve patient care are well-established. In the pediatric setting, ASPs have demonstrated ability to decrease overall antibiotic utilization and antibiotic purchasing costs, reduce antibiotic-related prescribing errors and improve appropriateness of therapy (Hersh 2014, Di Pentima 2010, Metjian 2008, Agwu 2008, Sick 2013). In combination with other interventions, ASPs have demonstrated reduction in C. difficile infection, and adherence to ASP recommendations has been associated with decreased unintended readmissions (Feazel 2014, Newland 2014).

Implementation of ASPs is recommended by the Infectious Diseases Society of America, Society for Healthcare Epidemiology of America, and the Pediatric Infectious Diseases Society (IDSA/SHEA/PIDS 2012). The Centers for Disease Control, Centers for Medicare and Medicaid Services, and the Joint Commission all advocate for ASPs and are piloting mechanisms for public reporting of inpatient antibiotic utilization. According to the recently updated California Health and Safety Code, all California acute care hospitals will be required to have an ASP as of July 2015.

B. Identify a single, key behavior change target for your translational activity.

            The pediatric ASP will reduce the overall utilization of broad spectrum antibiotics, as measured in days of therapy (DOT) per 1000 patient-days by:

a) reducing inappropriate initiation

b) prompting de-escalation to narrower spectrum agents based on available microbiologic data

c) prompting discontinuation of therapy when evidence of infection is lacking

c) decreasing inappropriately long duration of therapy

It is likely that failure to appropriately discontinue or de-escalate therapy is the biggest overall driver of high utilization, thus the core strategy of the program will be prospective audit and feedback to providers.

2. Conduct a “gap analysis” of your target behavior.  Look to diverse sources for “best guess” estimates if specific measures are not available.

A.  What is the quality (performance) gap?

Approximately one third of pediatric inpatients in our hospital currently receive antibiotic therapy; currently the most commonly utilized agents are broad spectrum agents: vancomycin and piperacillin-tazobactam (Zosyn). Based on benchmarking data from the University HealthSystem Consortium, our current antibiotic utilization is above the median among comparator hospitals, and has not changed significantly over the past 5 years. A recent point-prevalence survey revealed that approximately half of the hospitalized pediatric patients who were receiving antibiotic therapy met all criteria for appropriate use. In the remaining patients, there were opportunities for intervention to improve use of antibiotics.   

 
B.  What is the outcome gap?

While overall rates of C. difficile infection for our hospital are within target, some units have rates that are above the target goal, despite implementation of appropriate infection control interventions. High rates of C. difficile infection correlate with high antibiotic-utilizing hospital units.

Although rates of monitored multi-drug resistant infections are relatively low in the Children’s Hospital currently, our institutional antimicrobial resistance monitoring shows an overall decline in susceptibility to the most commonly utilized agents over the past several years, and poor clinical outcomes have been associated with multi-drug resistant infections in particular cases of highly antibiotic-exposed patients.  

3. Is there evidence that changing performance will improve health (clinical outcomes)?

There is moderate quality evidence supporting the role of antibiotic stewardship in reducing C. difficile infection when combined with other interventions. The impact of antibiotic stewardship on other clinical outcomes (readmission, mortality, multi-drug resistant infections) is less clear.

Hersh A, De Lurgio S, Thurm C, et al. Antimicrobial stewardship programs in freestanding children’s hospitals. Pediatrics, 2014 Epub: DOI: 10.1542/peds.2014–2579.

Di Pentima M, Chan S. Impact of antimicrobial stewardship program on vancomycin use in a pediatric teaching hospital. Pediatr Infect J, 2010 29: 707–711.

Metjian T, Prasad P, Kogon A, et al. Evaluation of an antimicrobial stewardship program at a pediatric teaching hospital. Pediatr Infect J, 2008 27: 106–111.

Agwu A, Lee C, Jain S, et al. A World Wide Web-based antimicrobial stewardship program improves efficiency, communication, and user satisfaction and reduces cost in a tertiary care pediatric medical center. Clin Infect Dis, 2008 47: 747–753.

Sick A, Lehmann C, Tamma P, et al. Sustained savings from a longitudinal cost analysis of an internet-based preapproval antimicrobial stewardship program. Infect Control Hospital Epidemiol, 2013 34: 573–580.

Feazel L, Malhotra A, Perencevich E, et al. Effect of antibiotic stewardship programmes on Clostridium difficile incidence: a systematic review and meta-analysis. J Antimicrob Chemother, 2014 69: 1748–1754.

Newland J, Goldman J, Stach L, et al. Impact of a pediatric antimicrobial stewardship program on length of stay and readmission. Presented at: IDWeek; 2014 Oct 10; Philadelphia, PA.

Policy statement on antimicrobial stewardship by the Society for Healthcare Epidemiology of America (SHEA), the Infectious Diseases Society of America (IDSA), and the Pediatric Infectious Diseases Society. Infect Control Hosp Epidemiol 2012;33:322-327.

Protocol Assignment 2 – Community Engagement

1. Define the community/communities for your project and explain why each is a stakeholder for your study. 

Patients and their parents are stakeholders because they receive antibiotic therapy and its associated benefits and harms. Parents may be influence clinicians to prescribe antibiotics inappropriately by expressing a desire for antibiotic treatment. Patients and parents also frequently take antibiotics differently from the prescribing directions, sometimes saving old antibiotics from prior prescriptions. While this is less of a factor in the hospital, discharge from hospitalization is an important opportunity for education regarding appropriate antibiotic use.

Pediatric clinicians (residents, fellows, attending physicians, nurse practitioners) on inpatient services are stakeholders in antibiotic stewardship because they are the primary prescribers of antibiotics in the hospital. They are burdened by the consequences of inappropriate antibiotic use when their patients develop complications such as C. difficile infection. They benefit from simplified decision-making around antibiotic prescribing, optimization of antibiotic dosing, and selection of definitive therapy.

Pharmacists and pharmacy administrators are stakeholders because they are involved in verification and dispensing of antibiotic prescriptions, therapeutic drug monitoring, and assessment of antibiotic-related adverse drug events and drug-drug interactions. Antibiotic costs account for approximately 1/3 of the pharmacy budget, and one of the benefits of antibiotic stewardship is to reduce unnecessary expenditure for antibiotics.

Infection control practitioners are stakeholders because they are involved in coordinated efforts to reduce hospital-acquired infections and antibiotic overuse contributes to infections such as C. difficile infection and infections with other multi-drug resistant organisms.

Hospital administrators are stakeholders because establishment of a pediatric ASP will ensure that we are in compliance with California Health and Safety Code which requires ASPs for all acute care hospitals. The presence of an ASP is publically reported in the U.S. News and World Report Best Children’s Hospitals rankings, and is anticipated to become an increasingly important metric for organizations such as the CDC, the Joint Commission, and CMS.

2. Describe your plan for approaching potential community partners to ask for their involvement.

One of the first tasks for the ASP will be to develop an antibiotic stewardship policy and procedure describing the program structure, operating procedures and key metrics. The ASP team will reach out to members of each key stakeholder group to identify champions for stewardship within the different clinical and administrative groups. Parents and patients will be identified from membership of the Patient Advisory Council. With these individuals, we will form an ASP working group to provide feedback on the policy and procedures as well as identify the first targets and interventions for the ASP to demonstrate success. This working group will continue to be engaged in ASP strategy and will help to solicit buy-in for stewardship from providers within their clinical divisions.

3. Identify which stages of your project you'll incorporate community input, and describe what types of input you'll solicit.

Development of policy and procedures – review and feedback of early drafts through final policy.

Development of specific stewardship initiatives – this will involve input from champions of the affected clinical services, with opportunities for input from other providers from those services.

Development of educational initiatives for trainees, staff, patients and parents.

Monitoring impact of program on C. difficile infection rates – will involve collaboration with Infection Control.

4. Name three ways you plan to share your results, beyond writing an academic article or presenting at an academic conference.

The ASP will report to the Children’s Hospital Quality Improvement Committee and will disseminate a twice yearly report to all pediatric clinicians describing trends in antimicrobial utilization, highlighting important developments in local antimicrobial resistance and changes in the antimicrobial formulary, and describing specific recommendations and initiatives of the ASP.

Protocol Assignment 3

1. Describe the organizational and/or delivery system environment in which your intervention will take place.

The ASP will be implemented within a referral children’s hospital affiliated with a public healthcare and research university.

2. Based on Shortell’s 4 domains of organizational change, identify organizational barriers that could potentially impede successful implementation of your proposed intervention.

Quality Performance: The institution tracks multiple publically reported metrics, several of which are infection-related, and has a strong investment in improving these metrics. Currently, neither overall antimicrobial utilization nor appropriate antimicrobial utilization are publically reported metrics, so there has not previously been significant attention directed to monitoring antimicrobial utilization or reducing inappropriate utilization.

Patient Satisfaction: ASPs typically have minimal direct interaction with patients themselves, but rather are focused on helping the provider to improve antibiotic prescribing. Providers sometimes cite parents’ and patients’ desire to “do everything you can” as a motivator for overprescribing of antibiotics.

Organizational Learning: Currently, we do not have robust systems for monitoring antimicrobial utilization and reporting back to providers. Education regarding appropriate antibiotic prescribing is fragmented and directed at a relatively small portion of providers. Clinicians are not accustomed to audit and feedback of their medication prescribing, so implementation of a prospective audit and feedback program may be seen as threatening to their autonomy.

Financial Performance: Implementation of the program requires ongoing support for physician and pharmacist salaries, as well as IT resources and project management support. The impact of antimicrobial stewardship on reimbursement varies greatly based on patient insurance – with some insurance models (ACO, Medicaid), stewardship has a positive financial impact because it reduces the total resource expenditure per case. With some private insurers, increased resource use results in greater reimbursement, thus negatively incentivizing stewardship.

3. Using the same 4 domain model, describe how your intervention plan can take advantage of organizational strengths OR propose practical methods for addressing these barriers within your program.

Quality Performance: The existence of an antimicrobial stewardship program is publically reported via the U.S. News and World Report Best Children’s Hospitals Report, so there is strong institutional support to develop a program that fulfills the criteria specified by this venue. It is likely that antimicrobial utilization and ASP activities will ultimately be measured and publically reported on a larger scale via the CDC, CMS and Joint Commission. We can leverage these developments as motivating factors not only for implementing a program, but for establishing success in reducing antibiotic utilization.

Patient Satisfaction: Modifying prescriber behavior will likely translate indirectly to improved patient satisfaction by reducing unintended consequences of antimicrobial use, including adverse effects, super-infections, drug resistant infections, and other complications such as IV catheter-related problems. We can leverage providers’ motivation to improve patient satisfaction by conveying the potential benefits of improved prescribing for the patient experience, and by giving providers tools to help their patients understand the importance of using antibiotics appropriately.

Organizational Learning: As a teaching hospital, we have the opportunity to influence providers at an early stage of training when they are more receptive to modifying behavior. For established providers, we will try to overcome the issue of perceived loss of autonomy by engaging them early in the process of program development, soliciting their input in developing our policy and procedures, as well as identifying clinical target areas that are important to them. Along with implementation of audit and feedback, we will implement complementary initiatives to “give back” to providers so that the ASP is perceived as a valuable service – e.g. focused consultation on antibiotic selection, dose optimization, therapeutic drug monitoring.

Financial Performance: Antibiotics represent approximately 1/3 of the annual pharmacy budget. By reducing antibiotic utilization, the ASP will have a significant impact on drug acquisition costs. The challenge will be that once use of antibiotics is better optimized, ongoing reductions in cost will not be demonstrated from year to year. Thus, the program must continue to justify ongoing support on the basis of improved quality of care and avoidance of costly and dangerous unintended consequences (e.g. readmission, adverse events, prolonged length of stay due to superinfections). As more patients are transitioned to ACO models of care, the impact of stewardship on overall resource utilization (length of stay and drug costs) becomes more favorable from a financial standpoint.

Protocol Assignment 4

1. Identify an individual (e.g., patient or provider) or group (e.g., community group or organization) that contributes to or is involved in the principal behavior you are attempting to change. Specify the desired behavior change (who needs to change what, when, where and how)?

The primary target group is pediatric clinicians practicing at UCSF Benioff Children’s Hospital. The desired behavior change is to decrease inappropriate use of antibiotics in hospitalized children by reducing inappropriate new antibiotic starts, and discontinuing or de-escalating therapy by 72 hours of initiation.

1. Using any of the individual explanatory theories in “Theory at a Glance”, develop an explanatory model for the target behavior (above) that you will be attempting to influence with your intervention. 

1. Create your own version of Table 1 from Michie et al that reflects some of the different theoretical domains and interview questions, tailored to the behavior you want to change. Be sure to select several domains likely to be of interest based on the literature or your experience/best guess. 

Protocol Assignment 5

1. Use the COM-B model to identify what needs to change in order for ONE of your selected target behaviors to occur. Note that TDF domains corresponding to each COM-B category are in parentheses, and that some of you may have already done this or aspects of this in Week 4. 

Here I am focusing on the target behavior of de-escalating from broad spectrum to narrow spectrum antibiotic therapy.

1. Use the APEASE criteria to identify appropriate intervention functions based on the behavioral diagnosis (See Table 2.3 in Michie et al Chapter 2) 

  3. For one of the intervention functions you selected, select a specific behavior change technique you will employ in your intervention strategy and specify how it will be delivered. See Table 3.3 in Chapter 3 of Behavioral Change Wheel (Michie et al) for list of most frequently used behavior change techniques for each intervention function.

a. BCW Intervention Function: Education
b. Behavior Change Technique: Feedback on behavior
c. Mode of delivery (i.e., intervention details): The core strategy of our Antimicrobial Stewardship Program will be prospective audit and feedback. The ASP pharmacist or physician (both with specialized training in infectious diseases) will identify inpatients on broad spectrum antibiotic therapy and will assess the regimen for appropriateness using established criteria. If there is an opportunity to de-escalate to a narrower spectrum regimen, the pharmacist or physician will contact the prescribing provider to make a recommendation and provide patient-specific feedback and education regarding the rationale for selecting a narrower spectrum regimen.

Protocol Assignment 6 (see attached logic model for this assignment)

1. Thinking about the protocol you are developing, identify the process and outcome indicators associated with the intervention/program and briefly describe an approach to measuring each.

Process measures:

• Patients identified as appropriate for de-escalation or discontinuation of antibiotics
• Contact made to clinicians by ASP pharmacist with recommendation for de-escalation
• Rate of acceptance of interventions

Outcome measures:

• Utilization of antibiotics in hospitalized pediatric patients, measured by days of therapy per 1000 patient-days. This data can be obtained from claims data in the University HealthSystem Consortium (UHC) database, but we plan to develop and validate a system for tracking antibiotic utilization through Apex barcode medication administration data.

Impact measures:

• Rates of infection with multi-drug resistant organisms including C. difficile, monitored through Infection Control and Prevention service

Balancing measures: Trends in LOS, readmission, ICU admission and ICU days monitored through UHC

2. Define one or more “intermediate” outcome measures [reflecting changes in environment, organizational culture, systems of care, patient or public behavior, and/or clinician behaviors] that can inform you about the mechanism by which your intervention achieves its downstream effect on health and inform you about the acceptability of your intervention. 

Intermediate measures: Clinician knowledge and attitudes regarding antibiotic use, awareness and perception of antimicrobial stewardship program, point-prevalence audits of percent of patients on antibiotics receiving appropriate vs. inappropriate therapy. Recommendations made by pharmacist, rate of acceptance

3. Identify a mixed methods study design and briefly describe the quantitative and qualitative data you will collect for program/intervention evaluation.

Quantitative Component:

• Antibiotic utilization data will be tracked through UHC and analyzed using an interrupted time-series approach, with segmented regression, using the start of the stewardship program as a change point.
• Similar analyses will be conducted tracking rates of C. difficile infection.
• Clinical outcomes including length of stay, ICU days, and readmission rates will be monitored as balancing measures.
• Point prevalence audits will be conducted monthly for more detailed assessment of appropriateness of therapy, which cannot be evaluated based on utilization trends alone

Qualitative Component:

• The stewardship program structure and processes will be evaluated using a framework of stages for antimicrobial stewardship implementation developed by the California Department of Public Health (www.cdph.ca.gov/programs/hai/Pages/AntimicrobialStewardshipProgramInitiative.aspx)
• The Medical and Pharmacist Directors will meet with an ASP clinician advisory group and additional representatives from the major clinical services to elicit feedback about the program and identify opportunities for modification
• Clinicians will be surveyed annually regarding knowledge and attitudes about antibiotic use, and perception of the program

Protocol Assignment 7

1. Describe the study design you will employ in order to determine if your intervention has had an effect on the outcome variable of interest.

An interrupted time series analysis will be performed to evaluate changes in antibiotic utilization with implementation of the stewardship program. Antibiotic utilization metrics (days of therapy/1000 patient-days) can be calculated from administrative claims data in UHC. The pre-intervention level, pre-intervention trend, post-intervention level, and post-intervention trend will be evaluated with segmented linear regression models. The analysis will be performed based on total anti-bacterial utilization, but also for individual broad spectrum agents and groups of broad spectrum agents.

2. Define the unit-of-analysis for your main outcome evaluation, the minimum meaningful effect size, and the sample size necessary to detect this effect size.

The unit of analysis is the institution, as antibiotic utilization data is aggregated for a given month. Each data point represents a month of utilization, normalized to the inpatient census (total days of therapy with that antibiotic or group of antibiotics per 1000 patient-days). At least 3 years of pre-intervention data is available, and the initial outcomes evaluation will be performed based on 12 months of post-intervention data.

Interrupted time series analyses are made stronger by longer duration of pre-intervention and post-intervention data tracking (more data points); there are no easy sample size calculations for this analysis but generally at least 12 months of pre-intervention and 12 months of post-intervention data are ideal. For our institution we have at least 3 years of pre-intervention data and plan to do repeated evaluations in 6 month increments post-intervention. We anticipate observable effects with 12 months of post-intervention follow-up. As a rough estimate of needed sample size, using the sample size calculation for a paired t-test, assuming a reduction in our baseline anti-bacterial utilization from 800 DOT/1000 patient-days to 640 DOT/1000 patient-days (20% decrease), with a SD ~ 10% = 80 DOT/1000 patient-days (and alpha=0.05, beta=0.8), we would need 5 observations per group (5 months pre- and post-). We plan to have more observations because we would like to be able to accurately describe pre-existing trends and evaluate the effect of the intervention on both level and trend. There may also be a lag in intervention effect and we would like to be able to follow the effects of the program over a longer time period. More pre- and post-intervention data will also enable us to capture smaller reductions in antibiotic utilization which could demonstrate early success.