On November 28, 2022, without our knowledge, GoDaddy (the company hosting our website) attempted to perform a migration of our website to a new server. Unfortunately, the migration failed part way through the process. As a result, our website was unavailable for a period of 7 days. During this time, GoDaddy did not permit us to take any actions. For example, they would not allow us to post a simple “under construction” notice or redirect our domain to a different server. Eventually, after countless hours of fighting with GoDaddy, they were able to restore access to our site.
Once our website was restored, we promptly began taking steps to migrate our website to a different hosting company. This included making daily off-site backups of our website. However, when we attempted to restore our site at the new hosting company, we discovered there were significant issues with our new off-site backups. We were actively in the process of troubleshooting issues with the backups when, on December 26, our production website went down again.
After several days of troubleshooting, our engineers were able to piece together what GoDaddy had done. GoDaddy had accidentally deleted our production website instead of the failed migration site. Further, we paid for GoDaddy’s premium backup service but discovered GoDaddy failed to point their backup service to the new server and our off-site backups were copies of the failed server. Ultimately, GoDaddy admitted that it was not possible to recover any of our website or data and they fully acknowledged that the loss was entirely their fault.
The good news…
Fortunately, we have slightly older off-site backups of our customer data, and a secondary automated synchronization between our public website and our internal Customer Relationship Management (CRM) software. This synchronization process replicates all of our customer information, including orders, subscriptions, license keys, and payment history to our the CRM. As a result, no customer data was lost, however, rebuilding the website will take some time.
As you would expect, there has been a lot of hand-wringing here at VoIPTools as we worked to troubleshoot GoDaddy’s unforgivable mistakes and complete incompetence. Our experience serves as a cautionary tale for anyone who assumes that a large, well-known hosting company like GoDaddy will take appropriate measures to safeguard their customers’ data and have a functional disaster recovery process in place.
