keystone - Various Workstation Performance issues – Incident details

The incident has been resolved

The issues reported appear to have been largely resolved by the underlying vendors This incident will go into a 24-hour hold and then close automatically. Current KeyStone Guidance: If you are still having problems, shut down any impacted computers (not restart/reboot) and turn it back on after 10-15 seconds. This seems to kick-start the update process and allow things to clear out of memory fully. If after a full shutdown, you are still experiencing problems, please open a ticket with us (if not already) by emailing service@keystone.solutions. Simply mention that you are impacted by the global performance issue so we can be sure to confirm with you once we have and have pushed a fix.

It appears that the underlying trouble vendors have pushed updates that have corrected the issues for most of the people who reported problems. We now believe that Webroot manage AV pushed an update that was causing problems with an MS Office update from Saturday that required Webroot to push an emergency update this morning. We have pushed a polling update to all computers using Webroot to try to force the updated policy. This appears to be clearing the issue on most computers. Current KeyStone Guidance: If you are still having problems, shut down any impacted computers (not restart/reboot) and turn it back on after 10-15 seconds. This seems to kick-start the update process and allow things to clear out of memory fully. If after a full shutdown, you are still experiencing problems, please open a ticket with us (if not already) by emailing service@keystone.solutions. Simply mention that you are impacted by the global performance issue so we can be sure to confirm with you once we have and have pushed a fix. We are continuing to monitor the situation

We are receiving reports that some users' machines are self-resolving. We suspect the impacted vendor may have pushed an emergency update. Our current guidance: Shutdown any impacted computers (not restart/reboot). If after a full shutdown, you are still experiencing problems, please open a ticket with us (if not already) by emailing service@keystone.solutions. Simply mention that you are impacted by the global performance issue so we can be sure to confirm with you once we have and have pushed a fix.

We have potentially isolated the issue and are testing workarounds.

We are getting similar reports from other vendors as well as many of our partner companies that this is looking like a globally widespread problem. We will update every 2 hours until we have a resolution.