Resolved: January 10, 2018 - Downtime 04:02 to 04:48 UTC

UPDATE: Issue Resolved as of 8:48pm PDT (04:48 UTC).    Email that was sent during this period to our customers would likely have been deferred back to the senders, and their email servers should simply retry the email automatically.  No email should be lost.

EXPLANATION: Ironically, the process of installation of new servers to our database cluster, intended to increase our redundancy and reliability, damaged a network cable, causing the sql proxy server cluster to freak out and bounce back and forth between a working and non-working state so quickly that it stopped almost all network queries from coming through. I'm not sure who lost more years of their life in this one - the dba or the network person, but they're both pointing fingers (at themselves, not at each other) and vowing to come up with a way to keep this from ever happening again.


ORIGINAL NOTICE: We are experiencing an issue with our database server cluster and are exploring the problem to determine the quickest possible solution.




Have more questions? Submit a request


Article is closed for comments.