keystone - Notice history

Incident resolved The issue causing authentication failures on all affected deployments is fully resolved and all services are now fully functional. We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue. Time posted May 13, 17:42 EDT

From DUO SOC: Identified The issue has been identified and a fix is being implemented. Posted 1 minute ago. May 13, 2024 - 16:16 EDT

Customers, Our Service Provider has let us know the situation should be resolved. If you continue to encounter issues, please call us at 423.464.6424, or email us at service@keystone.solutions From service provider: "Service in both data centers has now been fully restored. If you manually failed your devices over to PHX you can now fail them back to the primary server."

Customers. Below is the latest update from our service provider: "There continue to be issues with calling services, including inbound and outbound calling, and registration. The PHL data center is impacted and the PHX data center is partially degraded. Since our last update, to allow for recovery of the PHL network, we purposefully prevented a portion of SIP traffic from entering the PHL Network. We’re now slowly reopening the network to prevent large bursts of traffic that could cause more issues. We’re still investigating the root cause of this issue and will provide more information as soon as possible."

Customers, Our provider has let us know that they are also now seeing issues with customers in the PHX Datacenter. We will continue to monitor the status.

The incident has been resolved

Our provider has let us know that they believe this issue to be resolved. Service Provider update: "Incident resolved The elevated number of calls experiencing audio issues in the PHL data center has returned to normal, therefore we’re closing this incident. A Root Cause Analysis will be provided in the coming days."

Our provider has been able to normalize quality within their Philadelphia DataCenter. If you have voice quality issues at this time, please let us know. Please see the latest update from our service provider: "After making further adjustments over the weekend, and specifically since yesterday afternoon, our monitoring shows more normalized audio quality in the PHL datacenter. However, we’re acutely aware of how important it is that we fully resolve this issue for you and your customers, so our engineers continue to closely monitor the network. If you are on the PHX data center, please stay there for now. If you are on the PHL data center, please continue to report any call quality issues your customers encounter. Your reporting has been, and will continue to be, an important part of this troubleshooting event."

Please see the latest update from our Service Provider: "While the platform metrics in the PHL data center stabilized after the load balancing performed on the platform last night, we continue to see QoS issues. The stabilization efforts relieved part of the quality-related issues like garbled audio due to Jitter and we’re now addressing audio drop-out or gaps due to packet delay variation. Engineering is continuing to address the situation and will work 24/7, throughout the weekend, until this is fully resolved."

Please see the latest update from our provider: "Last night, the new servers were successfully stood up and put into production. This, among other things, should result in a more appropriate load balance across all servers. Engineering is continuing to monitor and work the situation. If you are on the PHL servers, your customers should feel a noticeable improvement today. However, please report any Call Quality issues in PHL to our support team. We’ll send another update at 4pm ET today, or earlier if we have additional information to provide. " If you are continuing to have issues, please report them to Keystone.

If your business is being impacted - we may be able to help mitigate. Please call our help desk at 423.464.6424 and we will work with you. Latest update from our provider: "We are still experiencing QoS issues affecting some customers built in the PHL datacenter. Our Engineers continue to investigate the issue and have introduced changes to balance the network environment. Our goal is to restore call audio quality for all customers as soon as possible. In response to data provided by our partners we are focusing on load optimization improvements. We’ve identified several areas to increase performance and have started to implement changes that should have a direct impact on overall call quality. These changes will continue into the evening as we implement ten additional physical servers to further balance the load. We’ll continue to provide improvements until the issue is resolved. We apologize for the inconvenience this is causing. We’ll send another update at 9am tomorrow morning, or earlier if we have additional information to provide."