We've deployed an update that increases the throughput of the database that was the bottleneck in today's incident. We'll have more info about additional remediations and a postmortem for today's incident within the next few days.
Posted Nov 25, 2024 - 19:26 UTC
Update
Our metrics have stayed at normal levels since our remediating actions about 30 minutes ago. We're continuing to monitor the platform while we discuss longer-term solutions to make absolutely sure we've addressed the root cause here.
Posted Nov 25, 2024 - 18:08 UTC
Monitoring
We've made some changes to the affected database, and our metrics and error rates have returned to normal. We've also re-enabled delivery of all webhooks, and we're monitoring for any further issues.
Posted Nov 25, 2024 - 17:33 UTC
Update
We're addressing an issue with an internal database that's causing problems with existing meetings, as well as starting new ones. Your users are likely seeing some failures when trying to join meeting sessions, and users in ongoing sessions are seeing occasional meeting moves. We'll post more information as soon as soon as it's available.
Posted Nov 25, 2024 - 17:25 UTC
Identified
We've temporarily disabled the component that sends webhooks.
Posted Nov 25, 2024 - 17:10 UTC
Update
We're continuing to investigate the source of meeting disruptions. Customers may be experiencing 'meeting moves' where a call session moves from one server to another, causing a 2-3 second disruption to the call. You may also see delays in receiving meeting.started and meeting.ended webhooks.
Posted Nov 25, 2024 - 17:07 UTC
Investigating
We're investigating issues that may be causing problems with network connections between regions.
Posted Nov 25, 2024 - 16:48 UTC
This incident affected: API and Core Call Experience.