13th February 2018

TMD Windows Cluster 01 Migration following OS Corruption - win01.tmd.cloud

UPDATE 2/14/18 7:40 AM CST:

Good news here! We are happy to inform you that our System Administrators successfully completed the post restore configuration of the server and all hosting related services are up and running properly. At this point we consider the situation handled and we do not expect further complications. The issue that has occurred is an extremely rare one and this is why it took that much to handle it.

As we have informed you earlier, all data was restored on a newly built server with the following IP address:

45.63.7.246

We have already taken all necessary steps on our end to make sure everything is working from the new IP address, including an update of the main name servers associated with the server, so no further actions are required on your end. All traffic has been successfully rerouted from the old server to the new one.

IMPORTANT: If your domain name's DNS servers are not ns1.win01.tmd.cloud and ns2.win01.tmd.cloud or if you are using CloudFlare or another type of DNS/CDN service or generally use 3rd party DNS services and only point A record to us, please make sure that you will update the records to the above provided IP address. In case you need any assistance with that, feel free to raise a new support ticket and our team will assist you further.


UPDATE 2/14/18 4:00 AM CST:

The File System Check has been completed and our System Administrators are working to fine tune the server configuration, so we can restore all hosting related services as soon as possible.

Please, stand by for updates.


UPDATE 2/14/18 2:00 AM CST: Filesystem check is almost finished. We prepare to do a warm up of the server services now and perform a final QA audit of the setup & backup restoration. We estimate approximately two hours until service is fully restored and operational.

Please, do not make any A record changes until explicitly noted here. We continue to be focused on restoring service back to 100% operational status and we would like to thank you for your continuous support and understanding during the past 24 hours. Please, stay tuned for updates.


UPDATE 2/13/18 9:30 PM CST: The restoration of the backup has now been completed and a file system check on the server has started, in order to ensure data integrity. We anticipate that in the upcoming two to four hours, we will be able to restore service back up and running.

Please, do not make any A record changes until explicitly noted here. We continue to be focused on restoring service back to 100% operational status and we would like to thank you for your continuous support and understanding during the past 24 hours. Please, stay tuned for updates.


UPDATE 2/13/18 4:30 PM CST: We have almost completed the restoration of the backup as per our previous update. Good news are that we will probably be able to detach the public IP address and move it to the newly set server with the accounts data exported from the backup.

Please, do not make any A record changes until explicitly noted here. We continue to be focused on restoring service back to 100% operational status and we would like to thank you for your continuous support and understanding during the past 24 hours. Please, stay tuned for updates.


Issue identified and actions are being taken 2/13/18 2:30PM CST: Earlier today, win01.tmd.cloud began experiencing performance issues which did not have any pattern or clear source. Our operations team has been investigating the situation with our data center engineers and managed to restore service successfully at first. Regretfully, shortly thereafter performance issues returned and after carefully observing the rescue console logs we have been able to retrieve, it looks like that some form of irreparable corruption at the OS level. At this point, we are unable to boot the OS.

Since all our efforts to revive the OS were not successful, we have decided to lose no further time and began restoring from our latest backup that was captured overnight. We anticipate that in the next few hours, the process of restoration will be completed. We are doing everything humanly possible to expedite it, however, we are also careful of moving data properly.

In order to make sure that we do not bring the corruption over by restoring also the OS from the backup, we had to set up a brand new server. We will restore only the accounts data on top of a newly built server with the following A record:

45.63.7.246

Your domain names with DNS servers under TMDHosting operation will undergo an A record change. No action is required on your end. We will carry this out for you, however, please be advised that there might be a certain period of DNS propagation. It will likely go within few hours.

IMPORTANT: If your domain name's DNS servers are not ns1.win01.tmd.cloud and ns2.win01.tmd.cloud or if you are using CloudFlare or another type of DNS/CDN service or generally use 3rd party DNS services and only point A record to us -- please keep an eye on our status page. As soon as we are ready with the restoration, we'll announce it here. We will make sure to advise you to proceed to update your A records accordingly.