Back

Service Incident - February 28, 2023

12:19 CET: Following a scheduled release, we are seeing issues with chatbots responding as expected. We are actively investigating the issue. More updates to follow as soon as possible.
13:10 CET: The issue has been mitigated, and bots are now responding as expected. We are continuing to investigate the incident to bring about a permanent solution for the issue. More information to come once the issue has been resolved.

Root cause: Yesterday's release included certain changes in our logic that required a database migration script to perform changes in some of our database tables. For reasons that are still under investigation, write access to database was blocked by another process and the job executing the database migration scripts was unable to run. The job kept trying to create blocked database connections, making the access to the production database temporarily impossible. Both from the platform and also for our bots. The problem was properly detected by our automated tests and alerts and our development team identified and mitigated the issue within one hour.