Some systems are experiencing issues

Past Incidents

19th January 2023

No incidents reported

18th January 2023

No incidents reported

17th January 2023

No incidents reported

16th January 2023

No incidents reported

15th January 2023

Maidenhead Loss of SQL Cluster

We are currently seeing a loss of SQL Cluster. We are working to resolve this and have also begun DR tasks to restore to a 2nd cluster if required.

Update 0630 - We are still working to restore the failed cluster and a restore of data from replication slave is also being performed. Next update at 0700

Update 0730 - We are still working to restore. The new cluster is up and data is being transfered to it. Next update at 0800

update 0800 - This has now been restored. We are monitoring the DB

Update 0830 - We have identified further corruption on 1 db in the cluster. We are working to resolve this. Next update at 0900

Update 0900 - Sadlt this remaining corruption is taking time to resolve. THis only effects 1 instance but does keep taking down the cluster. We are looking at remioving that 1 instance Next update at 0930

Update 0930 - The issue is still oingoing... We are now having to recreate the entire cluster .. However this time we are removing the corruption before this is restored. Next update at 1000

Update 1000 - Restore of new New cluster is underway. Next update at 1030

Update 1030 - Restore of new New Cluster from Replication Server is still oingoing this is sadly a slow process. Next Update is at 1130

Update 1130 - Restore of new New Cluster from Replication Server is still oingoing this is sadly a slow process. At 25% Done Next Update is at 1230

Update 1230 - Restore of new New Cluster from Replication Server is still oingoing this is sadly a slow process. At 35% Done Next Update is at 1330

Update 1330 - Restore of new New Cluster from Replication Server is still oingoing this is sadly a slow process. At 45% Done Next Update is at 1430 - We expect to be starting instances in the next hour as each finihses their restore

Update 1430 - Continuing Restore. At 65% Done Next Update is at 1530 - 50% of Effected Instances (Was around 5% of all Instances initially now only 2%) have been restored and are available . Instances will continue to be started as each finishes its restore

Update 1530 - Continuing Restore. At 75% Done Next Update is at 1730 - 70% of Effected Instances (Was around 5% of all Instances initially now only 2%) have been restored and are available . Instances will continue to be started as each finishes its restore

Update 1730 - Continuing Restore. At 85% Done Next Update is at 1830 - 85% of Effected Instances (Currently 1% of instances effected) have been restored and are available . Instances will continue to be started as each finishes its restore

Update 1830 - Continuing Restore. At 90% Done Next Update is at 1930 - 90% of Effected Instances (Currently 1% of instances effected) have been restored and are available . Instances will continue to be started as each finishes its restore

Update 1930 - Continuing Restore. At 95% Done Next Update is at 2030 - 95% of Effected Instances (Currently less than 1% of instances effected) have been restored and are available . Instances will continue to be started as each finishes its restore

2130 All Restored . All Instances Available.

14th January 2023

No incidents reported

13th January 2023

No incidents reported