Â鶹ÊÓƵ

People

Â鶹ÊÓƵResponse to CrowdStrike Issue: Information for Faculty and Staff

Jul. 19, 2024

This page was created to provide frequent updates for Â鶹ÊÓƵfaculty, staff, and students as we recover from the the CrowdStrike outage. If you are having issues with your computer, please attempt to reboot the laptop or desktop computer multiple times before contacting the ISD Help desk at (585) 275-3200.

Update: 5:00 p.m. Monday

As of this afternoon, all UR Medicine facilities are open as scheduled and providing safe patient care.

Our most important task today was identifying clinical computers that still need to be restored to resume normal operations.

  • IT responded promptly this morning to a few sites mentioned this morning as being out of operation.
  • Less than 5% of ambulatory clinical computers now remain in need of repair, after another day’s worth of focused IT support efforts, who will continue to prioritize restoring the remaining clinical computers and address non-clinical ones as quickly as possible.
  • Patient status boards in clinical areas are more difficult to restore and will be worked on later this week after PCs and laptops have been handled.
  • Use the new “” on the upper right of the to report a related issue. This link points to an that is more effective and efficient than sending email to the Help Desk. Nearly all workstations are operating normally and can be used by the individual or someone on their behalf to submit a request.

We’ve seen a tremendous response from IT, where nearly a third of the 600-person department has enlisted to train on making the fixes needed and gotten into clinical areas as quickly and thoroughly as possible. While they are prioritizing this work, clinics are asked to refrain from raising other IT-related issues with the on-site support staff. Those requests should go to the ISD Help Desk via phone call to 585-275-3200 or the .

Update: 10:30 a.m. Saturday

We wanted to provide an update on how our enterprise is recovering from the CrowdStrike outage.

Simply put, the recovery work across Â鶹ÊÓƵhas been astounding, particularly within our clinical enterprise.  Thanks to a true team effort between our IT and clinical teams, a precise and focused recovery plan, and innovative problem-solving, we were able to quickly overcome multiple obstacles. 

While the root cause of the outage was quickly uncovered early Friday morning, the prescribed fix was sobering:  it required a manual repair of dozens of servers and systems, as well as an estimated 20,000 desktops and laptop computers that support clinical and administrative functions. But within less than 24 hours, fewer than 10,000 computers remain to be brought online. Our IT teams will continue working through the day to reboot more computers at a rate of 300 to 500 per hour, with a goal of returning all clinical services to full efficiency by early next week.

Once again, the resiliency displayed by each and every member of our Â鶹ÊÓƵteam is simply amazing.  You all displayed tremendous teamwork and patience to work through yesterday’s challenges and offered creative solutions to ensure we were able to provide the safe and high quality care that our community relies on us to deliver each day. 

Below is a status update including reminders where faculty and staff can bring impacted laptops or desktops to be fixed. Additionally, hospital leaders are rounding this morning to check in with staff and to ensure we are aware of any outstanding issues and that proper prioritization for service continues. 

  • As of 9 a.m. Saturday, all UR Medicine facilities are open as scheduled and providing safe patient care. Some clinical service areas have returned to fully normal operations, but some computers remain down in others, which makes it difficult for staff to work at full efficiency.
  • Walk-in service clinics for impacted laptops and desktop units:
    • Today (Saturday, July 20):  Support available for all faculty and staff at Corporate Woods, Building 30 through 7 p.m.
    • Monday, July 22:  Support will be available to all faculty and staff from 7 a.m. to 7 p.m. at both Corporate Woods and Rochester Tech Park (RTP), 905 Elmgrove Road, Bldg. 5.

Update: 3:00 p.m. Friday

As we work through the impacts resulting from the global CrowdStrike software outage, we are now ready to address individual computer and laptop outages. A small team is available as outlined below at Corporate Woods, Building 30 on a first-come, first-served basis; the technical fix will take approximately 15 to 20 minutes for each device. Additionally, on Monday, we’ll open up a satellite location at RTP (Rochester Tech Park, 905 Elmgrove Road, Bldg. 5).

  • Today (Friday, July 19): If you are a clinical or administrator on call this weekend, please bring your laptop to Corporate Woods through 7 p.m.
  • Tomorrow (Saturday, July 20): Support will be available to all faculty and staff from 7 a.m. to 7 p.m. at Corporate Woods.
  • Monday, July 22: Support will be available to all faculty and staff from 7 a.m. to 7 p.m. at both Corporate Woods and RTP.

If you have a desktop that has been impacted (outside of clinical areas; these are being prioritized and serviced through a different process), please put a ticket into the Help Desk and it will be serviced as soon as possible.

Update from CEO David Linehan: 12:00 p.m. Friday

By now, we’re all aware of the massive global outage to technical systems inadvertently brought about by a CrowdStrike software update that downed machines running the Microsoft Windows operating system last night. Thankfully a fix has been issued, and our ISD team is and will continue to work around-the-clock to reinstate all impacted systems.
 
I’m happy to report that emergency and inpatient care at Strong Memorial continues without interruption. While our electronic medical record system is intact, multiple computers that support clinical care functions are still being restored. We are also working to ensure that elective procedures and outpatient appointments can take place as scheduled, as well as services provided at our Urgent Care, lab, imaging and ambulatory locations. 

Outside of the clinical enterprise, other system outages are impacting the work of our faculty and staff. Our leaders are working as quickly as possible to prioritize and reinstate systems, however it will take some time to fully restore operations. As soon as details are finalized, we will provide more information on what to expect over upcoming days using a variety of communications channels including email, our Intranet, public website and UR Medicine social media accounts. 
 
While we’re just hours into this issue, it’s not too soon to thank the team of IT and clinical leaders who immediately jumped into action in the early hours today to ensure the continuation of emergency and inpatient care. As we speak, hundreds of IT staff throughout the Medical Center are coming on-site to apply the manual fix needed to reinstate service. It’s teamwork like this that makes me proud to be part of the Â鶹ÊÓƵfamily. Thanks to each and every one of you for your patience and flexibility as we work together to navigate this challenge.