Originally published on Aug 20, 12:45
MailRoute experienced a network issue on a server cluster on Thursday, August 18 and Friday, August 19 that caused a small percentage of email (less than 2% of our total email volume) to be delayed. No email has been lost.
Our servers have multiple network interfaces - each has one to the outside world and one for our internal network. One server cluster in our lax datacenter developed a bad internal network interface which caused enough network errors to cause mail to fail to transfer within our network for proper processing. This caused email entering that cluster to be delayed and affected accurate monitoring of the problem. The interface was fixed at 14:50 on 19 August, and all mail that was queued up was delivered over the next 7 hours. We are re-examining our server monitoring processes to make additional checks over the public network interfaces in addition to our internal network checks.