In May 2007, the board of trustees for the Dartmouth-Hitchcock (D-H) healthcare system called for the replacement of its homegrown electronic health record (EHR) system, which was targeted at ambulatory care, with Epic, a readily available, commercial system. Epic provides comprehensive solutions for all aspects of healthcare, ranging from inpatient and outpatient support right on down to section-specific programs, like cardiology, emergency, and perioperative services. We acquired sixteen of their applications for initial implementation and planned to go live with fourteen of them on April 1, 2011. We called the new system eD-H.
Nearly three years later, in March 2010, after configuring the system, ratifying workflows, and installing a massive new infrastructure, we decided that an April 1, 2011, go-live was possible. However, we were challenged by the diversity of legacy systems operating in the information systems (IS) environment. All told, it was running twelve different homegrown incident management systems, segregated by team. In addition, there were two regional help desks using different service management systems. The team-based systems used a hodgepodge of products: FileMaker Pro, SharePoint, Excel, and Outlook. There was no way to roll incidents up into common problems, and there was no tracking of cross-team incidents from first call to resolution. Knowledge was shared between teams through individual emails, and escalating issues from the help desk to one of the application teams was like tossing a brick over the Berlin Wall. It was clear that the existing, segregated nature of the IS systems would not be able to support the new system. Thus, we were given approval to explore new enterprise service management systems.
The selection process included a comprehensive review of both hosted and on-premises solutions. We required a development environment that would allow us to customize a complex PC lifecycle management module, replacing the existing homegrown system. The new solution also needed to include fully integrated change, knowledge, incident, and problem management modules, and full CMDB implementation and self-service options. If that was not enough, it had to be affordable, too.
Because of the team-specific requirements from all our groups, as well as significant financial constraints, our selection criteria were stringent. We had to satisfy a number of diverse teams, each with unique requirements. Ultimately, we chose Cherwell Service Management, primarily because of its concurrent license solution for the full range of modules, but also because it offered a developmental environment that could support new versions (without rework), and it could be plugged into our standardized virtual server and Citrix environments.
We replaced the data center’s outage tracking system in October 2010. This required an enterprise implementation, so that all teams could enter server-based problems. Next, we developed a PC lifecycle management system to replace both regional help desk solutions, and we implemented incident, problem, and knowledge management, including a comprehensive CMDB platform, for both help desks by end of 2010.
How We Did It
Accomplishing our task required the efforts of two dedicated internal (“factory-trained”) application administrators for the implementation period, fourteen days of professional services from the vendor, a project manager (not dedicated), and a management sponsor acting as project director. One of our better ideas was having a team of two fully trained administrators working together from the beginning. Together, this group executed a four-phase implementation based on a legacy system replacement scheme:
- Technical services and data center;
- Application teams;
- Help desks and field service; and
- Clinical transformation and the new EHR system.
We branded the solution Dartmouth-Hitchcock Service Management (DHSM). We used it as the incident management solution for the first module of the new EHR, the Epic Willow Pharmacy solution, which itself was a pilot for the fourteen-application, single-day cut-over on April 2, 2011. DHSM was rock solid for the Willow implementation and performed flawlessly. Next, we began configuring the seventy-five categories and subcategories for the EHR system, as well as 250 reports for the larger go-live. We wanted a system that was flexible enough to categorize incidents and problems effectively, and capable enough to pinpoint their effects on our business, and, ultimately, on patient care.
A review of other institutions’ large Epic go-live experiences revealed a dramatic increase in incident volumes during the first couple of weeks following implementation. For maximum effectiveness, we needed to prepare for this outcome by beefing up our first-level contact capabilities, significantly augmenting our call center capabilities, and creating a self-service portal for direct entry. More than 600 clinical providers were identified as local support staff (LSS), and they were enlisted to help us with “at-elbow” support. LSS were granted the authority to enter incidents directly into the self-service portal. Training this defined population for incident entry was a more achievable goal than training the entire user base. Concurrently, we also selected fifty internal and external call center professionals to staff a 24×7 go-live call center for two weeks. The majority of these agents were internal staff, from areas not affected by the new EHR system, like administrative systems support, human resources, and finance.
With a battalion of internal foot soldiers—a combined group of over 200 vendor consultants, trainers, and analysts, many of whom spent the previous three years configuring the system—as well as an enlarged call center, the support foundation was set. However, we still needed to create a command structure that could manage all of these resources and keep them focused on the most-critical problems affecting our institution. Thus was born our unified command, with a technical command center of 100 people (twenty-five three-to-five person teams) at the heart of the operation. This command received incidents from the call center as well as the LSS, and distributed them appropriately to the various teams for prioritization and resolution. Technicians and analysts only accepted incidents submitted through DHSM.
An institutional command composed of senior leaders met twice a day to review DHSM reports, as well as to listen to the feedback from the technical commanders themselves and allocate resources to the most-critical problems. Divisional and departmental command centers were also created to address local incidents and given the ability to run reports directly from DHSM to review and reprioritize any such incidents.
DHSM itself was configured to unify these command centers by providing reports from the same data repository, capturing the information for each constituency in a defined structure. We accomplished this by mandating that all incidents had to be entered by call center or LSS staff. The technical command center would then prioritize and dispatch resources to provide incident resolution. Likewise, the institutional and local command centers were able to constantly monitor incident and problem dashboards so they could prioritize issues and allocate resources.
In the weeks and days leading up to the go-live, the LSS was trained in the direct entry of incidents via self-service, and DHSM was used to enter all testing and workflow abnormalities. In doing this, we experienced a steady increase of concurrent users. Our licensing needs were growing, but the Cherwell licensing model was able to accommodate our needs. Using self-service, the Report Viewer and Dashboard Viewer significantly lowered the demand for additional seats, because these components do not require additional licensing. We were also able to acquire temporary technician licenses to get us through the initial months of go-live, until our actual concurrent use stabilized. In doing so, our infrastructure team was able to steadily increase virtual server resources to provide quick responses for the 200 concurrent users that were to come.
To further accommodate the expected increase in call volume, we built an augmented call center in two computer training rooms adjacent to the existing help desk, giving us a total of twenty-four ACD phones for call center support (which was well beyond our resource plan to staff them). Our recruits became the new tier 1 call center. They were given a full eight hours of just-in-time training in advance. This would not have been possible if we had not brought together internal, highly skilled IT staff and internal administrative support staff who were already familiar with our telephone and computer systems. The existing help desk staff became the tier 2 team for all desktop issues. A carefully crafted and widely circulated call-triage document was created and updated twice a day. For example, for urgent calls, agents entered the incidents and warm-transferred the calls to tier 2 desktop or eD-H analysts.
Peak staffing for the first few days of go-live saw twenty-nine first-shift tier 1 agents and sixteen first-shift tier 2 agents logged into the ACD simultaneously (a far cry from the previous week’s five on tier 1 and five on tier 2). A minimum of four tier 2 agents provided over-the-shoulder support to the new call center staff on the first shift. We progressively reduced staffing levels for the second and third shifts, by about 30 percent each. Furthermore, as illustrated by the graphic below, self-service incidents entered by the LSS exceeded those entered by call center technicians for the first two weeks of go-live. This enabled us to step down call center staffing each day, sending people back to their home departments as volumes decreased.
By April 15, 2011, the new call center stood down and the staff returned to its original help desk stations. For the next four months, a slightly augmented help desk with increased hours provided tier 1 support for the eD-H, at which point we returned to our prelaunch hours, services, and staffing levels. We were also able to step down our DHSM server resources and concurrent licenses during this period.
What We Learned
The enterprise service management system served as an effective unifying agent for all of the organizational/sectional incident command centers. Successful communication with operational leaders required significant effort (and server overhead) for report design and generation, but by integrating the Cherwell Report Writer directly into the database (instead of the application server), we were able to accommodate the demand.
We learned that our siloed incident management systems would never have been tolerated in the new environment, as they would have required redundant entry, front-to-end incident tracking would not have been possible, and comprehensive reporting would not have been available. We also found that self-service was highly effective for reducing call volumes. However, it does need to be highly defined with required fields; many of our self-service tickets had incomplete information.
We made a significant investment in training staff for the comprehensive EHR system, but DHSM training was considered secondary, so the investment we made there was insufficient. Also, we failed to fully leverage the power of associating incidents to problems, which would have enabled us to resolve all of the associated incidents as each problem was resolved.
Implementing a major institutional application can improve the adoption of an enterprise service management system. However, even with that weight overhead, there will still be outliers that require leadership motivation. And, as with almost all applications, the implementation of an enterprise service management system will not immediately increase the productivity of all individuals; some staffing pockets have seen their workloads increase because they lacked required documentation. This outcome needs to be accommodated in future staffing models. Overall, however, we are extremely proud of what we accomplished. We hope our experience helps others with similar implementations.
Bill Weyrick is a thirty-year veteran of the Dartmouth-Hitchcock (D-H) healthcare system, where he is currently in charge of their regional service management strategy, including the help desk, field service, email, and software distribution. Bill’s recent projects include the replacement of a proprietary email system with Microsoft Exchange, whole-disk encryption of all PC and laptop drives, and a rollout strategy for application virtualization. He is certified in incident management systems by FEMA and New England Healthcare Incident Command Services, LLC.