Hey *,

I'd like to apologize for the downtime and I'm not using "we" because it is in part my fault.

Our main web proxy crashed which made the website unavailable; proxy server is physically dead, still waiting on ISP to advise on what happened.
We do have 2 additional proxies in standby but the monitoring box was also dead (VPS freezed) so the script which should have switched to a backup did not execute.

All this combined with me being offline (for good reasons ... but not really an excuse) led to the downtime.


We have a complex setup here at MTV in order to protect your privacy and ours and we don't relay on public services except for Cloudflare but since I don't trust CF completely we also run proxies behind CF to protect the important servers which actually run the services. I am not disclosing the full details of our architecture for obvious reasons but I take this as my fault because in retrospective this could have been handled better.

To avoid a similar situation in the future I'll make a few changes to our monitoring system as soon as I find some free time.