Control Panel Performance Problems

We are now fully functional again.

We are pleased to inform that we now have a new primary database running. Our initial checks confirm that everything is working correctly.

This was class as a P0 (Critical Priority) incident and it did cause a major interruption. Our primary database server experienced a big performance impediment which ultimately resulted in resource exhaustion. Which we initially tired to resolve, upon identifying that the problem was hardware related we opted for an emergency replacement / upgrade.

We are pleased to report that our fail-safes worked so no data was lost and calls continued to flow.

Our emergency upgrade has more than doubled our previous server spec which maintained 4 years of solid up time.

We will now be closing this incident and complete minor related tasks during normal working hours.

Thank you for your patience during this incident and we apologise for the inconvenience caused.

We have completed the data transfer. We are now working on configuring the server. ETA 2:30 UTC.

We have experienced repeated performance problems restoring the primary database server.

We have deployed a new more powerful primary database server and are currently in the process of migrating the data.

We expect to be able to issue a further update at 1:00am UTC.

We are sorry to inform you that we are experiencing a major outage on our master transnational database.

Calls are still flowing, but user interface is down. We apologize for the inconvenience caused and appreciate our patience whilst we resolve this incident.

We are experiencing problems with the control panel. Currently investigating.