Protocol Assignment Forum: Program Evaluation

#7 – Program Evaluation – Post-Op Debrief

 1.      Considering the protocol you are developing: identify the process and outcome indicators associated with the program and briefly describe an approach to measuring each.

 2.      Define one or more “intermediate” outcome measures (reflecting changes in environment, organizational culture, systems of care, patient or public behavior, and/or clinician behaviors) that can inform you about the mechanism by which your intervention achieves its downstream effect on health or inform you about the acceptability of your intervention

Intermediate Outcome:

• Change in OR culture regarding team work and multi-disciplinary communication.  Assessed by the Safety Attitudes Questionnaire for OR (SAQ-OR) pre- and post-implementation.  The debrief is a large culture change; it is hypothesized that the change in culture to foster communication, and take a moment to recap a case is an intermediate step to improved patient outcomes.

#1 Protocol development

1. Define the communit(ies) for your project and explain why each is a stakeholder for your study.

The UC Office of the President (UCOP) is the primary stakeholder/grant funder. UCOP and the Center for Health Quality and Innovation partnered with the Office of Risk Management to identify the services within the UC medical schools that were the highest cost. Ortho or neurosurgical services were targeted for quality improvement interventions for this reason. Each of the UC medical centers’ CMOs recognize the need to demonstrate improvements in cost while maintaining or improving the quality of these high impact services.

Each UC medical center has staff and faculty at multiple levels that are the communities and stakeholders of this project. These include the site leads of the grant, administrative leadership such as the CMO; IT; the Chair of Neurosurgery (NSG); NSG faculty, residents and fellows; anesthesia attendings and residents; neurophysiologists; OR nurses; perioperative leadership and staff; and circulating nurses.

Each of these levels of staff have an interest in improving team communication, coordination, and improving efficiency of their work, while improving patient outcomes.

The five UC medical center sites have formed a collaborative to share information, successes and barriers in implementing this intervention.

2. Describe your plan for approaching potential community partners to ask for their involvement.

In order to implement the post-operative debrief, we are approaching NSG leadership for buy-in and support in disseminating the principles and mechanisms of the intervention. We are attending NSG faculty meetings to solicit feedback on the intervention and what components are necessary to keep NSG attendings engaged and involved. We are reaching out to other UCSF services that have implemented their own post-operative debrief in the past to gain insights on successes and barriers.

We have included nursing at these meetings to build trust and partnership because the major work load will fall on nursing staff. We have established small working groups with nursing QI to help develop the work flow and process improvement segments of the intervention. They in turn are communicating with pod leaders and other nursing stakeholders to guarantee all nursing feedback included. We will be conducting training sessions with nurses after the intervention is vetted by nursing leadership.

We engaged the NSG residents by proposing this initiative as their annual quality improvement project required for their completion of their program. The residents voted to take up the post-op debrief and help champion the effort. Because it is their QI requirement, they have additional incentive to guarantee success of the project.

We have presented at grand rounds, MnM, and other staff lectures to highlight the importance of the project and network with people with previous experience or vested interest in the intervention.

IT has been approached about modification to the OR templates for documentation of the post-op debrief. They are being kept up to date on the pilot’s progress, iterations, and refinement so that when the debrief is rolled out to all services, they will be ready for the build.

3. Identify which stages of your project you'll incorporate community input, and describe what types of input you'll solicit.

The development of the post-op debrief content requires input from all team members: NSG attendings and residents; anesthesia; neurophysiologists; and OR nurses. This is done within UCSF, and at each UC medical center. Feedback is shared with the collaborative and consensus reached on the minimum content of the debrief.

The data collection requires intensive input from nursing staff, as they are the primary data collectors for the project. Since this data collection must be incorporated into already busy work flows, it is imperative that there is additional value to the data beyond the grant purposes. Nursing leadership is using the opportunity to collect information on OR inefficiencies so that problems can be identified, triaged to appropriate units, communicated, and resolved.

4. Name three ways you plan to share your results, beyond writing an academic article or presenting at an academic conference.

Compliance with the debrief will be shared directly with NSG and OR leadership staff. Any problems identified and resolutions implemented will be communicated with the surgical team. Success of this pilot will be shared with the UCSF OR committee in order to plan staged roll out to other services within UCSF. Successful implementation of this post-op debrief can be shared with patient experience units as demonstration of improved teamwork and communication of the surgical team.

#2 Defining the Quality and Outcome Gaps

A. What evidence are you proposing to translate into practice?

Reduction in mortality, surgical errors, and post-op complications through implementation of a post-operative debrief on all neurosurgical patients.

 1. Justify that this evidence is “ready for translation.”

In 2003 and 2004, the Joint Commission established the Universal Protocol for  Preventing Wrong Site, Wrong Procedure and  Wrong Person Surgery™ as a part of a series of requirements of their National Patient Safety Goals.

The timeout prior to first incision is required for accreditation by the Joint Commission. This strong delivery system incentive has resulted in near 100% uptake of the pre-incision timeout, but the post-op debrief is not specifically required by JACHO, has not been standardized, is inconsistently implemented, and has not been fully embraced by the operating room culture.

In 2008, the World Health Organization (WHO) as a part of their Safe Surgeries Saves Lives global challenge revised their Surgical Safety Checklist that includes multi-disciplinary timeouts at three time-points: before the induction of anesthesia, before the first incision, and before the patient leaves the operating room.

The Center for Medicare and Medicaid Services (CMS) is implementing new quality reporting requirements that, beginning in 2014 will include implementation of 
a Surgical Safety Checklist and effect payment in 2015.

2. Identify a single, key behavior change target for your translational activity.

Targeting neurosurgical attendings and residents to initiate the debrief process and become champions of the effort, including creating space for all members of the surgical team to voice concerns and process improvements. The buy-in from surgeons is based upon closing the process improvement loop; generating a mechanism for concerns, malfunctions, and inefficiencies to be documented at debrief and triaged to appropriate channels for resolution.

 3. Conduct a “gap analysis” of your target behavior. Look to diverse sources for “best guess” estimates if specific measures are not available.

B. What is the quality (performance) gap?

In 2012 and 2013, UCSF Neurosurgical services launched a pilot post-operative debrief program that included a "secret Shopper" data collection method to monitor compliance with the debrief.  Although compliance was achieved up to 75% in the first few months, it waned slowly over the year, and was allowed to taper off naturally when funding was secured for a broad neurosurgical safety initiative that included a debrief (allowed to return to baseline before relaunch). Compliance with the debrief was less than 25% of neurosurgical cases at the end of 2013.

C. What is the outcome gap?
Neurosurgical cases often result in higher than institutional average for several key quality outcome measures such as SSIs. The UC Office of the President and the Office of Risk Management assessed the most costly services in the UC system and found these to be consistently Ortho- and Neurosurgery.  This was the impetus for selecting Neurosurgery for a targeted quality intervention.  Preventable errors such as retained objects, missing or mislabeled specimens, forgotten procedures, inadequate communication of post-operative instructions, operating room inefficiencies,
and delays in OR turnover that directly effect patient outcomes, revenue, and the patient experience.

D. Is there evidence that changing performance will improve health (clinical outcomes)?

There is strong evidence that Surgical Safety Checklists reduce complications and mortality, though specific evidence of improved clinical outcomes from just the post-operative debrief component are less demonstrated. There is strong evidence the debrief component results in improvement of the safety culture of the OR teams,
increased communication, and increased teamwork.

#3 Mapping the Post-op Debrief

1. Identify a patient or community group that contributes to or is involved in the principal behavior you are attempting to improve with your intervention.

Attending neurosurgeons.

2. Using any of the individual explanatory theories in “Theory at a Glance”, develop an explanatory model for the target behavior (above) that you will be attempting to influence with your intervention.  This can be an extension/based on expected findings (or previously published literature) from your answers to Homework #3.  Figures are always very useful... keep it simple.

Using the Theory of Planned Behavior (TPB):
Targeted behavior: Performing the post-op debrief before the attending leaves the OR.

Attitudes towards behavior: Although the fundamental quality and safety benefits behind the post-op debrief are readily 
acknowledged by nearly all surgeons, many feel that the never-events the debrief is meant to prevent are too rare. Within one institution, too many debriefs would have to be performed to collect enough data to show meaningful change in never events within a reasonable timeframe. Attendings do feel like there are more common efficiency concerns that could be highlighted by the debrief, and stand a better chance to improve than the never events.

Subjective norm: Attending surgeons are concerned about the beliefs of key people (e.g. the NSG Chair) and are motivated to comply with the behavior both out of the concern of belief of respected others, and the desire to do as well (or better) than others (their colleagues).

Perceived behavioral control: Attending surgeons are being asked to champion the debrief effort. It is agreed (by surgeons, and especially by perioperative nursing leadership) that they are in control of whether the debrief is performed or not. Attendings perceive the power to perform the debrief rests with them.

3. Identify how one or more of your specific interventions will target one or more of these key factors contributing to the behavior of interest.

In order to increase the buy-in and liklihood attendings will perform the behavior, the debrief is being framed as a mechanism to address OR inefficiencies rather than never events. A process improvement system is being created with the input of all community stakeholders to address identified problems and solutions will be communicated back to the surgical team. This intervention addresses the attending surgeons' attitude towards the behavior (performing the debrief) that a lot of time and effort is invested with little or no return (reduction in never-events).

Compliance with the post-operative debrief will be tracked, including percentage completion of items. These results will be de-identified and posted in comparison to other attending surgeon performance within UCSF, and compared to the other 4 UC medical center campuses. This addresses the subjective norm of attending physicians who are concerned with how well they perform compared to colleagues.

4. Create a framework that draws upon a socio-ecological framework to orient your target behavior within a larger context. ie, what are some of broader, external forces that influence the individual behavior of interest...see Figure 2 of “Theory at a Glance.”

Although performing the post-operative debrief is an individual behavior (on the part of the attending surgeon), it is nested within establishing a system for the debrief among the surgical team. The surgical teams are among a community of other surgical teams within the NSG service. These in turn are nested within a quality improvement committee established to address problems identified by the debrief and feed back information to surgical teams. These activities are supported by NSG and UCSF leadership. And these participate in a collaborative of the other four UC NSG and executive leadership.

#4 Organizational Culture

2. Describe the organizational and/or delivery system environment in which your intervention will take place.

The post-operative debrief takes place within the operating rooms (ORs).  The focus of this intervention is in the neurosurgical (NSG) services unit at UC San Francisco, and the four other UC medical campuses (UC Los Angeles, UC San Diego, UC Davis, and UC Irvine).  There is a collaborative of the five medical campuses composed of NSG chairs, Chief Medical Officers (CMO), co-investigators, site leads, and site coordinators in order to standardize common core principles, and share barriers and solutions.  But within each individual site is a unique delivery system environment with their own unique system change strategies.

At UCSF this intervention will be piloted first within NSG at the Parnassus location, with the intent of expanding to other service lines (and campuses) after iteration and refinement.  Although the intervention itself occurs within the OR, change in service delivery will need to take place within faculty, nursing pods, and nursing quality improvement. 

Attending surgeons will no longer be able to leave the OR at any time and will be required to remain until specific items of the debrief are completed.  This is a large culture change for surgeons who often leave for the next case before the current is finished.  Nurse pod managers must facilitate nursing participation and data collection of the debrief, beginning with systems changes in pre-op, through the case, and into post-op.  Nurse quality managers will be aggregating debrief data, creating systems for tracking and triaging problems, and acting as liaisons for process improvement identified by the debrief process.

3. Based on Shortell’s 4 domains of organizational change, identify organizational barriers that could potentially impede successful implementation of your proposed intervention.

Clinical Quality Performance

The debrief has the potential to add time to OR cases, or at least the perception of more time spent at the end of a case.  This is viewed as a negative by some attending surgeons and potential barrier to full adoption of the intervention. 

One of the main goals of the debrief is to increase surgical team communication.  This improved communication reduces the likelihood that serious errors will occur.  This is a culture change that may encounter significant barriers at implementation.  Traditionally disparate groups may be challenged to create a space where all team members are heard and recognized for their contribution to the case.  It has also been unanimously decided by all groups that the debrief is initiated by the attending surgeon and not regulated by the circulating nurse.  This top-down approach may be a barrier to team culture change.

There is disagreement about when the debrief should be initiated: at the beginning of closing or after the initial surgical counts have been completed.  Surgeon buy-in hinges on the ability to complete the case quickly and leave for the next case, not waiting for the initial counts; while opposing views believe the initial counts should be complete prior to closing so that if any missing items were identified, the patient would not have to be re-opened. 

Patient Satisfaction

Patients are minimally involved in the process of this intervention, though stand to benefit from increased OR efficiency, reduction in delays in cases, and ultimately improved quality and safety.  If implementation snags result in more delayed cases, this may decrease patient satisfaction.

Organizational learning

The principles of the intervention are known by clinical staff and have been reiterated over the course of the last year through grand rounds, staff meetings, and messaging from clinical leadership.  But the actual launch of the debrief and changing OR behavior from “no debrief,” to “debrief on every case” will require messaging to attending surgeons primarily via staff meetings, email, and survey acknowledgment.  There is a potential for surgeons to miss this messaging.  Nurse managers and staff will need to hold their own training meeting prior to launch to instruct on data collection and systems changes.  There is a potential for staff to be absent or miss this messaging.  And finally, after the initial implementation takes place, there is potential loss of organizational knowledge at the next academic year and a new wave of residents and faculty join.

Financial performance

The debrief is currently not a billable action.  It is one more thing that must be done without clear reimbursement.  The debrief is designed to prevent rare, but costly events.  But without a large number of cases tracked over a long time, it will be difficult to demonstrate the financial case for instituting the debrief.

4. Using the same 4 domain model, describe how your intervention plan can take advantage of organizational strengths OR propose practical methods for addressing these barriers within your program.

Clinical Quality Performance

The number of delayed first cases will be closely tracked pre- and post-launch of the debrief.  The hypothesis is that by addressing OR inefficiencies within the debrief and designing systems to address identified problems, delays in cases will reduce over time. 

Patient Satisfaction

HCAHPS scores will be monitored pre- and post-implementation of the debrief, along with delays in cases.  Successes measured will be communicated to Patient Experience teams for messaging to patient groups.

Organizational Learning

Compliance with the debrief will be monitored closely after launch and de-identified data on completion of the debrief will be shared with all NSG surgical teams.  This benchmarking against colleagues is hoped to promote competition and awareness.

Organizational learning in the coming years is being addressed on multiple levels.  One is that the residents have already picked the post-op debrief as the QI goal for next year, meaning there will be another whole year of passionate, incentivized resident behavior to ensure at least 80% compliance with the debrief.  This pilot will also be the testing grounds for implementing the debrief across UCSF services.  If the debrief is approved by the OR committee it will result in changes to the OR templates within APeX, further reinforcing standardized behavior on every case.

Financial Performance

Tying internal incentive programs to compliance with the post-op debrief is one mechanism for addressing otherwise absent immediate financial rewards.

Being able to demonstrate cost-savings from improved patient outcomes will also be vital to the long-term buy-in of multiple stakeholders.  This will be accomplished first through published literature, then pooling collaborative data (from all five medical campuses) and looking at long term historical cost trends before and after launch, then finally gathering enough data to see trends within institutions.  This will be enhanced further if data can be collected from all service lines before and after staged roll-out in a step-wedge approach.

More immediately, cost savings will be able to be demonstrated if delays in cases are reduced.  This metric is more likely to show improvement in cost in a shorter time period than long-term patient quality and safety outcomes.

Other mechanisms that may enhance financial performance are accreditation and Medicare reimbursement.  There are potential changes to Joint Commission accreditation to include mandated post-op debriefs, and Medicare is already including debriefs as potentially satisfying quality metrics that effect future reimbursement.

#5 Tool – Post-Op Debrief Audit Cards

A. Describe one tool that you will employ in your intervention strategy using the following domains....

1. Tool—type: Debrief Audit Cards
2. Target Population: Circulating Nurses
3. Target Behavior: Performing Debrief on all neurosurgical cases
4. PRECEDE Category: Reinforcing
5. Platform: Initially paper cards with all neurosurgical case charts until process refined, then once ready for all UCSF services, built as OR template in APeX.  Systems/efficiency concerns documented on same paper audit card initially, with goal of documenting in separate protected electronic system that automatically messages appropriate departments (e.g. rounding mobile software)

B. For a multi-tool intervention strategy, use the PER worksheet attached to describe how you will address each of the PRECEDE framework components 

PER Worksheet. 

PER Worksheet. 

#6 – Translating Research into Policy – Post Op Debrief

What level of government will you target to translate your research into policy and why will this make the most sense?

Institutional level medical leadership and parallel roles at the other four UC medical campuses.  In order to change surgeon behavior and build surgical team communication capacity, executive leadership buy-in was needed.  This intervention requires a significant culture change and needs the support of multi-disciplinary leadership in order to be successful.

What level of government makes the most sense for you to translate your research into policy and why?

As far as achieving local policy change, targeting local medical leadership made the most sense.  But if this intervention was to be expanded outside UC medical campuses there are a couple other levels of policy makers to target.

Targeting CMS and influencing reimbursement for in-patient safety and quality is an effective behavior modification strategy.  Currently there is a lot of resistance to the debrief because of the time, or at least perceived time it takes to perform on every case.  It would be highly effective to change reimbursement dependent on performing the debrief.

What strategies will you use to reach policymakers? 

CMS is going to be promulgating rules regarding in-patient safety quality that include surgical checklists which will impact reimbursement payments starting in 2015.  So far, the rules have not been specific to requiring a post-op debrief.  Instead, a debrief can count as satisfying an in-patient quality metric, but so can other quality actions. 

We can provide evidence and commentary to CMS during the open comment period for when reimbursement rules are finalized showing that a debrief specifically improves patient outcomes and safety, reduces cost, and access to surgical services.

Another strategy is targeting the Joint Commission to recommend the debrief become required for accreditation. 

Use UCOP forums, conferences and leadership for dissemination of pilot findings.

What steps are available to you to reach policymakers?

1. Publication of patient and institutional outcomes post-pilot.
2. Expansion of the debrief to all services within UCSF; standardization improved implementation process
3. Expansion across all services across all five medical campuses (plus UC Riverside down the road)
4. Leverage UCOP connection to policy makers to disseminate pilot findings
5. Provide evidence at open comment periods for CMS promulgation