2011-11-28

Apology and explanation for our downtime today

Were you trying to find emails with big attachments today, discovered Find Big Mail and then ... the site had a message saying to try again later?

We offer our sincere apologies to everyone affected. After much back and forth with our hosting provider, it turns out that their zealous operations staff had installed a new firewall to keep the server safe. Unfortunately, it also blocked our scans from connecting with Google.

We plan to take the following actions to mitigate a similar problem in the future:

  • better monitoring on the server to ensure that critical processes have access to ports, files and processes they need
  • review our site hosting arrangements (possibly switch providers, or have a backup site hosted by another provider)

The total downtime was around 4 hours. Everything is running normally now. Sorry for any inconvenience caused.