2018-11-13 Incident post-mortem report

At approximately 1:30 PM Mountain Time (3:30 PM Eastern) on November 13, 2018, we observed a significant latency spike in our external monitoring tools configured to access the load-balancing tier of our cloud-based APIs. These monitoring tools provide end-to-end testing and simulate a complete user experience with our application.

By design, our systems can easily process more than 25x the usual amount of traffic we receive. We do this because of our customer usage patterns, in which we may observe a 10-fold increase in traffic within a short period, usually a few minutes or even a few seconds. While technically wasteful to run this much excess capacity, our goal is to ensure that our APIs remain available and responsive even with massive amounts of requests flowing through them.

As we observed the latency spike, moving from around 100ms to over 1 second for many requests, we checked all of our other metrics to ensure it wasn't an errant metric. Sometimes, the external monitoring providers will report latency numbers from South Africa or Australia rather than from within North America. Most of our internal metrics showed that our systems were humming along at 10-15% CPU utilization. Our load balancing tier also showed healthy numbers for CPU utilization, memory usage, and total bytes flowing through the network.

We checked our own tools to see if we could confirm the latency spike. Regardless of the numbers reported by our internal systems, we observed that calling the system through our load balancers resulted in many requests returning far slower than acceptable.

Part of our strategy for increased latency is to invoke cloud APIs that provision new hardware and bring it into production service. We followed this process and immediately doubled our capacity—despite our internal metrics showing only nominal activity within the system.

As these new load balancers came online, external latency returned to normal, expected levels. However, even though these numbers appeared normal, we continued to receive a smaller yet steady stream of reports from customers stating that they were unable to connect to our servers.

While working with these customers, we discovered that a significant number of customers were unable to resolve our DNS records for our various APIs, e.g., us-street.api.smarty.com. Their respective systems would return the equivalent of NXDOMAIN. This was puzzling because we clearly had that domain defined. We finally found that many of these customers had upstream providers that were using either misconfigured or buggy DNS server implementations (of which there are a lot), and were unable to handle more than 20 or so entries within a single A record. In our case, we had close to 40 IPs listed within a given A record.

To combat this issue, at about 4:30 PM Mountain Time, we reduced the number of IPs listed in a single A record to a more reasonable level. As we did so, we saw an immediate spike in latency numbers, as reported by our external monitoring tools.

All of this meant we were stuck between a rock and a hard place. On the one hand, if we didn't have enough load balancers, customers would be affected by significantly higher than expected latency, while if we had too many servers listed, some customers would be negatively impacted as a result of using DNS servers that weren't designed to handle large numbers of IPs within a single DNS record.

At about 4:35 PM Mountain Time, we decided to split our DNS resolution into several parts so that only a portion of the IPs for a given record was returned at any given time. After making this change, the latency numbers dropped to expected levels, and the availability reports stopped; all internal and external metrics showed that things had stabilized.

At around 9am the following morning—without any configuration changes having been made during the intervening 12+ hours—we began receiving reports of reduced connectivity from some customers. After some diagnostics with these customers, we determined that each had specific regulatory requirements for their corporate firewalls. Because we had brought so much hardware online, it exposed the mismatch between regulatory compliance and cloud scaling. From this, we decided to bring the original set of IPs back into rotation. After doing so and closely watching all available metrics, all connectivity reports ceased, and all metrics and tools showed normal connectivity.

Now that all systems were behaving normally, we turned our attention to understanding exactly what was happening within our system at the time of the original incident. After investigating CPU utilization, memory, disk, application logs, and other available resources, we found that a particular piece of software designed to facilitate secure server-to-server communication between our systems was misbehaving. The software in question provides a "mesh network" so that our production systems can communicate with each other over the internet securely. 

As best we can determine, this errant behavior was not due to misconfiguration but was caused by a latent bug in the software itself. This bug would result in a "flapping" connection. This means that a TCP connection between our load balancing tier and our application servers would be established, and then traffic would begin to flow over that connection, only to be terminated unexpectedly. We had anticipated that this connection would be unavailable, and we had designed our software to use an alternative secure channel as a result. However, because the connection was flapping on and off, the system was designed to prefer the mesh network connection when it was available. This caused many requests to fail, while others succeeded.

In the months prior to the incident, we had been looking to remove this particular software from our stack to reduce the need for an additional dependency. Instead of using a peer-to-peer mesh network VPN, we had decided to begin having our various software components communicate directly via standard TLS-based channels.

While making a configuration change of this nature isn't trivial, it was already something we had been preparing for. Further, we had concerns about the potential for another incident during the fourth quarter of the year, which is often the busiest time for many of our customers and the worst possible time for additional incidents. We tested and deployed these configuration changes on Wednesday and monitored all available metrics closely throughout the deployment.

As of this writing, all elements of the offending software have been removed from our infrastructure.

Key changes

One critical takeaway from this is that we need to communicate what's happening to our customers much faster. This has been tricky because our goal is to fix the problem as quickly as possible, and those individuals who know the most about what's going on are the most engaged on the front lines of solving it. Even so, when customers are negatively affected, we need to report things quickly. Therefore, in the event of any future incident, we will post a status update immediately upon observing any errant behavior, and we will provide incident updates in a streaming fashion, with no less than 30 minutes between updates until the issue is fully resolved.

Another takeaway is aligning all of our metrics and tooling around the customer experience. Regardless of internal numbers, if requests are failing, customers are negatively impacted. We are looking at ways to better configure monitoring and alerting to ensure we can maintain a consistent view of the application's overall behavior.

One thing we spent a significant amount of time on is engineering around the principles of availability—especially availability in the face of failure. There is a lot more work to be done in this area. We are fully committed to ensuring a positive and uneventful customer experience, even in the event of software or hardware failures.

Notes

As a reminder to customers with regulatory or other compliance requirements for a stable, known set of IPs, we offer a dedicated solution as part of your subscription. It is called our Forward Proxy API (https://www.smarty.com/docs/apis/forward-proxy-api). As part of our guarantee for this API, we won’t add new IPs to production without at least 2 weeks (14 days) advance notice, as specified in this JSON contract: https://proxy.api.smarty.com/ip-ranges.json. 

Because of the wide range of traffic we receive, we often need to provision new hardware quickly in anticipation of increased customer demand (https://www.smarty.com/docs/apis/requirements#dns). As a result, new IPs come and go with our cloud APIs. Those customers using our Forward Proxy API will communicate through that system on a known set of IPs, and it’ll dynamically resolve our servers on your behalf and send your request to our application servers in a fully encrypted fashion.