As promised in the last post, I’m back with more details about the server crash that took down this blog and several other web-projects of mine.

The hosting provider where I used to host my stuff had a catastrophic meltdown that resulted in my server being unavailable for a few days. I noticed that the server was not accessible on May 29th after which I tried fixing the problem in their managment portal. As most management portals are absolutely unusable, I was able to fix exactly nothing at all. Tried to use their diagnostics: unexpected error
Tried to restart my server: unexpected error
So this mess of a management solution could not even manage to execute the most basic troubleshooting step.

I was forced to contact their customer support team, which told me (on June 12th!) that they messed up something in their virtualization and that my server was not recoverable and I had to reinstall the OS. This time the management portal actually worked and let me install a fresh copy of debian and once again I was able to access my server using ssh. But when I went back to use their backup-feature (yeah also in their mess of a management portal) to restore all my data, all backups were gone. Pretty unfortunate if your backup solution is tied to the thing its backing up…

But, as they say, every cloud has a silver lining. I took this unfortunate experience as an opportunity to redo everything and do it right this time. After some thorough research (basically not selecting the cheapest thing I could find), I migrated to a new hosting provider. One of the most important lessons I learned from this ordeal was the significance of a dependable backup solution. To ensure my data’s safety, I decided to take matters into my own hands this time. I set up a simple backup solution that backs up all my data to my home-server so I have full control over my data. I’ve also switched from running multiple docker containers orchestrated using custom scripts to using a Kubernetes environment. This should also help managing the configuration of all the applications.

Setting up everything from scratch also tought me to better document my setup and decisions. All in all I learned a ton from this experience and I’m probably going to write a few more posts about some of these things:

  • Kubernetes Installation

  • Continuous Deployment to Kubernetes Cluster

  • Backup to home-server

  • Documenting setups and configurations

  • …