Office 365 Outbound
There was a temporary issue with outbound email relay for some of our o365 customers. A small subset of our servers were affected as a result of Microsoft, for the very first time in a long time, adding a new bank of servers for outbound relay.
We autodetect this by pulling the list of records from their SPF records with a small script, and that script runs twice a day, which has always been fine- they haven't updated that list in several years, at least. And running it twice a day seemed like a good setting, since the TTL on that SPF record used to be over 12 hours long.
The script ran on all of our servers, but on 007.lax and 011.lax it didn't run until, apparently, shortly after they added those servers, leaving them out of date for up to 12 hours.
We've pushed the changes to those servers, and increased this script to run every 5 minutes, since they've set the TTL for that record down to 600 seconds now, to ensure we pick up changes very rapidly.