Resolved

We have identified the problem which was related to a combination of a user performing a mass update of firewall rules, causing large database queues and too strict settings with regards to allowances of database lag.

We have throttled and denounced firewall updates to fix the root cause and we will also be addressing the acceptable database lag to help mitigate fluctuations in the primary database lag moving forwards.

We are sorry for any inconvenience caused.

Avatar for
Updated

Services have been restored however we are still considering the platform at risk until we can perform a full investigation of the outage.

Avatar for
Updated

Some things are back up now.

Avatar for
Updated

We have identified the issues and should have full services back up and running shortly.

Avatar for
Updated

The problem is with our backend and we are working as fast as we can to rectify the situation as soon as possible.

Avatar for
Ongoing

We are receiving messages about outages and we are currently investigating.

Avatar for
Began at: