Resolved -
This Major Incident is at a close.
We will carry out any follow up work required over the coming days.
If you are having any issues on Monday morning please contact the BTG helpdesk for support.
Jul 20, 18:15 NZST
Monitoring -
****
Good afternoon BTG clients,
We have resolved most of the issues stemming from the Crowdstrike Event.
Our focus has been on addressing customer servers, and the majority of work on the affected devices has been completed.
For remote physical servers that remain offline, we will need to visit the site to address the issue as we are unable to connect remotely.
Regarding workstations and laptops, we will fully understand the extent of any remaining issues once customers return to work on Monday morning. Many devices appear to be resolving issues on their own with multiple reboots.
Devices that won't boot and are running Microsoft BitLocker will require additional effort to fix. Our staff will assist with these issues on Monday.
If any device continues to blue screen on Monday morning, please contact the BTG helpdesk for support. We recommend calling first, but if phone lines are busy, please email the helpdesk to log a ticket.
For contact details, please visit our website at www.btg.co.nz/contact or email us at helpdesk@btg.co.nz.
Thank you,
The BTG Support Team
Jul 20, 14:10 NZST
Update -
Most client devices are now operational.
This issue impacted Windows devices on our Merge platform and on client sites. We originally reported the issue affected Compute resources on the platform but it was not the platform itself. Affected systems for this incident have been updated to 'Windows Devices'.
Engineers continue working on the remainder of devices affected.
Jul 20, 10:11 NZST
Update -
BTG engineers continue to work through mission critical customer devices this morning.
BTG Service Desk is open for support based on our clients support agreement.
CrowdStrike have official posted a statement with the latest workaround steps to resolve this issue for anyone that would like to try it themselves.
Please follow the link below.
https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/
Jul 20, 08:28 NZST
Update -
BTG engineers continue to work through customer devices, many are now operational.
Our engineers will continue to work through the night to restore systems.
BTG Helpdesk will be open in the morning for anyone to call in.
Jul 19, 22:38 NZST
Update -
Our key medical services clients have had the fix applied and we're focusing on clients providing 24x7 services.
Our team are working through remaining clients' servers and we will then focus on end user devices that have been affected.
Jul 19, 20:10 NZST
Identified -
The issue has been identified and a fix is being implemented.
Jul 19, 19:08 NZST
Update -
*******
Some customers have asked how to resolve the issue themselves. We are positing the article from Crowdstrike on what you need to do. Please ***ONLY*** do this if you are confident with administering Microsoft Windows devices.
****************************
Tech Alert | Windows crashes related to Falcon Sensor | 2024-07-19
Cloud:
US-1EU-1US-2
Published Date: 19/07/2024
________________________________________
Summary:
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.
Details:
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
Current Action:
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:
Workaround Steps:
1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally.
********************************
Jul 19, 19:01 NZST
Update -
*********
BTG have tested the fix from Crowdstrike and starting to apply the fix to customers.
We are starting with customers that are "tagged" as essential services first, for example medical clinics etc.
We will then move onto clients that run 24x7 and are open in the weekends.
The fix is a manual process and at this stage and we have to touch each machine individually.
We are focusing on customers "Servers" first and then will look at workstations & laptops.
Thanks you form the BTG Team.
*********
Jul 19, 18:45 NZST
Update -
Crowdstrike have just published a manual fix for the issue.
BTG are currently testing the fix and will contact customers soon.
Thank you for your patience while we work through this event.
Jul 19, 18:35 NZST
Update -
At this stage we believe the issue has been caused by the Anti Virus product Crowdstrike. Crowdstrike have acknowledged there is a problem and are investigating. This appears to be affecting systems all over the world.
BTG engineers have found a work around and are testing it on our internal devices first before considering making changes to customers systems.
All BTG engineering staff that are available have been called in to help.
Account managers are contacting clients one by one.
Jul 19, 18:03 NZST
Update -
This is a worldwide issue affecting multiple systems. We are working with our vendors to find a cause and resolution.
This status page is also affect so updates may not be as frequent as usual.
Jul 19, 17:15 NZST
Update -
We are continuing to investigate this issue.
Jul 19, 17:09 NZST
Investigating -
We are currently investigating this issue.
Jul 19, 16:57 NZST
