7th February 2018

Cloud Sites Infrastructure - UK Connectivity issues - uk05.tmd.cloud

UPDATE 17:00 GMT First of all good news here! We have recorded zero DDoS activity during the past 48 hours and this is why we consider the issue as resolved and all services are fully restored. We will closely monitor the server performance in the next couple of days and in case there is any need for post-emergency actions they will be performed immediately.

Considering the above we would like to apologize for the temporary inconvenience this unpleasant event might have caused. Unfortunately, such attacks are absolutely unpredictable and in this case very hard to contain due to its severity.

Last, but not least - I would like to once again thank you for giving us the opportunity to make up after the action-packed week. We are working around the clock to ensure we deliver you better service and we have no other priority than bringing things back to normal.

If you experience any difficulties with your websites do not hesitate to get in contact with our technical support team which is 24/7 at your immediate disposal.


UPDATE 07:00 AM GMT We are happy to say that during the last night server monitoring there was no trace of the DDoS attack. The server performed absolutely flawless and we believe that the attack has been gone. However, we will officially announce it later today after 48 hours since the last DDoS trace is passed. Meanwhile, we have already initiated the last phase of the data analysis of the incoming traffic flow and will update this status if any further actions would be required. Please, stay tuned for updates.


UPDATE 17:00 GMT During the past 10 hours, all incoming traffic was carefully reviewed by our NOC and system administrators. We have noticed that throughout the day there was no trace of the DDoS attack which is a positive signal. We will continue to monitor the situation for 24 more hours and we will call the issue resolved if there is no trace of the DDoS until tomorrow. Meanwhile, if you experience any difficulties do not hesitate to contact our support team which is 24/7 on your immediate disposal. Stay tuned for more updates.


UPDATE 07:00 AM GMT Thank you for your patience up to now. Without it, our work would have been much harder. We are happy to say that after diligently monitoring the server performance it appears that the situation is under control for now. We have changed the status of the service back to Fully Operational and in case the issue re-appears we will act accordingly. Our plan, for now, is to continue monitoring the incoming traffic flow and do our best to completely isolate the potential DDoS source. Rest assured that our team will not rest until we bring everything back to normal as this is our top priority. Stay tuned for more updates on the matters.


UPDATE 01:00 AM GMT We are happy to announce that we have managed to execute our IP change plan and service is restored to nearly 100%. Our team will remain to monitor the server overnight and if everything looks good around 7 AM GMT, we will further review the incoming traffic flow and may consider the issue completely resolved. We absolutely do not want to mislead you and that's we are completely honest here - service is back up and running and we will stay around to monitor its stability, taking action where necessary. We would like to assure you that our team continues to work around the clock and immediately when it is safe to confirm the end of the attack, we will do it via our status page. Please, stay tuned for updates.


UPDATE 23:15 GMT We still don't consider the situation resolved completely, however, we can report that more than 75% of the service is fully restored by now, following our IP change move. Keep in mind that this action has incurred propagation, which may result in you seeing the cPanel default page placeholder. This will resolve as soon as the IP address we assign to your batch propagates across the Internet. Keep in mind that your browser will probably cache the cPanel default page and you may be misleading your website is not back online. Do clear your browser caches frequently while trying to load your page. Our team continues to be focused on bringing back online more and more websites hosted on this server and please rest assured that we are fully committed to completing this as soon as possible. Bringing things back to normal is our company technical staff's top priority until it is completely over. Please, rest assured that our primary focus remains to bring your website back online and we are doing everything humanly possible to do so. Please, stay tuned for updates.


UPDATE 19:15 GMT We are about to begin the IP addresses split procedure, as explained in our previous status update (see below). Keep in mind that this might incur slight propagation, which may result in you seeing the cPanel default page placeholder. This will resolve as soon as the IP address we assign to your batch propagates across the Internet. Keep in mind that your browser will probably cache the cPanel default page and you may be misleading your website is not back online. Do clear your browser caches frequently while trying to load your page. Our team continues to be focused on bringing back online more and more websites hosted on this server and please rest assured that we are fully committed to completing this as soon as possible. Bringing things back to normal is our company technical staff's top priority until it is completely over.


UPDATE 17:30 GMT Following a very rough 36 hours period, we have managed to contain the majority of the attack, however service is not fully consistent yet. As you are reading this update, our operations team and UK based NOC specialists are deploying additional hardware appliances. As the attack is clearly targeted amongst a tenant in our shared hosting environment, it is extremely hard to exactly pinpoint the root cause as all tenants share the same IP address. Within the next two hours, we will begin splitting the accounts on the server on different IP addresses, which will minimize the potential DDoS target, as the moment we finish the IP change and it propagates, the attack will be redirected to the IP address of the batch that the offenders reside. Once we manage to break down and close the circle, we will be closer to identifying the target and therefore isolating them immediately. Our team continues to be focused on bringing back more and more websites hosted on this server and please rest assured that we are fully committed to completing this as soon as possible. Bringing things back to normal is our company technical staff's top priority until it is completely over.


UPDATE 13:00 GMT We have noticed that the attack was re-initiated but our mitigation device is holding up pretty well for now. Note that due to the mitigation of all incoming data it is possible to experience a bit slower than the usual server speed. Our system administrators continue with the server audit which should be over shortly. Stay tuned for updates.


UPDATE 11:25 GMT We are happy to announce that 99% of the services hosted on uk05.tmd.cloud are now restored. Right now our system administrators are performing in-depth QA audit of the server, ensuring that its configuration is up to our standards. Stay tuned for our final update.


UPDATE 07:00 AM GMT 90% of the services hosted on uk05.tmd.cloud are now restored. We anticipate that in the next three to four hours tops, we will be able to complete the restoration of the remaining services, as well as conduct a final QA audit of the server, in order to ensure everything is properly done. Service itself will be restored to 100% shortly. We would like to thank you for your continuous support and understanding during the past 24 hours. Please, stay tuned for further updates.


UPDATE 06:00 AM GMT 80% of the services hosted on uk05.tmd.cloud are now restored. We anticipate that in the next few hours, we will be able to complete the restoration of the remaining services, as well as conduct a final QA audit of the server, in order to ensure everything is properly done. We continue to be focused on restoring service back to 100% operational status and we would like to thank you for your continuous support and understanding during the past 24 hours. Please, stay tuned for updates.


UPDATE We are happy to inform you that we are ready with the setup of the new infrastructure, which we have installed, in order to migrate your account and evade the DDoS attack that has been unfolding in the past 24 hours. We expect that we will begin restoring service partially, one account at a time and within the next 12 hours, service will be fully restored. Our team continues to be focused on bringing back more and more websites hosted on this server and please rest assured that we are fully committed into completing this as soon as possible. Bringing things back to normal is our company technical staff's top priority until it is completely over.

IMPORTANT In case that you are using third-party DNS service, please change the A record of your domain names hosted on uk05.tmd.cloud to 185.62.87.43 or reach out to a TMD representative for further assistance.


MAJOR DEVELOPMENT OF EVENTS We have tried various mitigation techniques, but unfortunately, neither of them worked. Given the magnitude of impact this DDoS attack is causing, we will begin evacuating accounts to a another UK location we maintain, where we have shipped special hardware firewall appliances already. Our operations team is now working on bringing up the new environment and within the next couple of hours (approx 00:30 GMT) we will begin moving data to the new server which will host your website. We expect to fully restore service in the next 12 to 14 hours. Please, rest assured that our primary focus is to bring your website back online and we are doing everything humanly possible to do so. Please, stay tuned for updates.


UPDATE To our greatest regret, even the hardware mitigation devices that we have installed in our London data center are unable to cope with the size of the still ongoing DDoS attack. Usually, attacks come and go quickly, but here it appears that we are dealing with high magnitude of DDoS traffic that is spiking higher than the average attack. Our data center technicians and NOC are working around the clock, in order to reverse the course of attack, however due to the high density of the packets, it is still unclear where the traffic originates from. We would like to assure you that our team continues to work around the clock until we reach a final solution and we really do our best to mitigate the attack. Please, stay tuned for updates.


UPDATE We regret to inform you that the DDoS attack has been initiated once again this time with even higher intensity leaving us no other choice but to block all traffic to the server again. Our data center technicians are analyzing the traffic and doing their best to filter the incoming traffic and isolate the malicious from the legit request. We are fully aware of the magnitude of this event and how it affects our customers hosted on the targeted server. We would like to assure you that we will be working around the clock until a final solution is being found. Please, stay tuned for updates.


UPDATE The attack was mitigated and the server performance is now fully restored. Please, check your websites and in case you experience any further difficulties do not hesitate to contact our Technical Support team for additional assistance. We will keep monitoring the server performance and will update the status if any further actions would be required.


UPDATE We have just detected that the DDoS attack is again active and our NOC specialists have taken all necessary actions in order to mitigate it. We would like to inform you again that your data is safe and sound and you do not have anything to worry about. Once we have more information on the matter, we will update this status page.


UPDATE The server is up and running and fully operational with all services running properly. Feel free to check your websites and contact us via our ticketing system in case that you experience any issues.


UPDATE The detected Distributed Denial of Service (DDoS) is still ongoing at this time and our NOC specialists are working around the clock to resolve the issue. We would like to assure you once again that all your data is safe - once the DDoS ends, we will your update you here immediately.


UPDATE The attack intensity is still very high and the server remains under a controlled environment. As soon the attack subsides we will fully re-activate the server. Meanwhile, our data center technicians will continue to work around the clock to mitigate the attack as fast as humanly possible. We will continue to update the situation


UPDATE The coordinated attack towards the primary IP address of the server becаme quite severe and started to affect the performance of the server. To ensure that there will be no data loss as a result of the attack we have temporarily blocked all access the server IP. Our Network Operation Center technicians are doing their best to mitigate the attack and restore the server functionality to its regular pace. Stay tuned for more updates.


Our Network Operations Center (NOC) has detected Distributed Denial of Service (DDoS) attack towards the primary IP address of the server. At the moment there is a mitigation device set which will filter the incoming traffic. Due to a large number of requests, the response from the server will be slower than usual. Note, that all of your data is safe and sound. Once the DDoS is over, we will update you here immediately.