Sunday, April 19, 2020 - 4:57 PM

I was getting ready to cook dinner for my family when an alarm went off on my phone ¹ — All of our production servers were reporting that they were unhealthy, and our average response time had crept above an acceptable threshold. I set dinner aside and headed to my home office to investigate.

These sorts of events are uncommon, but not super rare. Our servers are set to alert us before things get bad for our customers. Usually the cause is some customer running a bunch of queries to gather information about something on the Ethereum blockchain; perhaps a wallet ² is indexing the transaction of a new token they’re adding support for, or maybe a customer is seeing a surge in usage beyond what they’re used to. Less often, the cause is someone with malicious intent trying to hurt us or our customers. Usually the solution is to spin up new servers (which happens automatically) and the extra capacity is online within a few minutes even if our team does nothing. But ensuring a high level of service is critical to our business, so when these alarms go off we investigate just to be sure.

When we see these events happen, we have a process that we go through:

1. Check the servers for responsiveness. Just because they’re reporting unhealthy doesn’t mean they’re down, just that they’re not operating at full capacity.
2. If the autoscaler ³ hasn’t already done so, bump up the number of servers we’re running. If it turns out we don’t need them, we can always spin them back down, but we’d rather have extra capacity to make sure our users aren’t impacted while we investigate.
3. Depending on the level of impact to our service, we notify customers who may need to make adjustments to their services.
4. Pull down the slow query log ⁴. We deal with tens of thousands of requests a minute. A tiny fraction of those take more than a second to run. We filter out the slow queries we’ve come to expect, and look at what’s left.
5. Generally, the slow query log shows some new query or set of queries that’s causing a problem. We run that query against our testing environment ⁵ to see what it’s doing. Depending on the nature of the query, we evaluate whether or not the query is serving some legitimate function for a customer. If it looks valid, we look for ways we can better handle that query (caching, optimizations, etc.). If it looks malicious in nature, we write a filter to block that query from running.
6. We create a ticket to address the underlying problem more holistically — Blocking a query is generally a short term fix, and there’s some underlying issue that query was exploiting that requires a more robust solution. We schedule those tickets into workload based on our assessment of the threat.

In general, the process described above takes between 10 and 20 minutes. On the evening of April 19, the slow query log showed us hundreds of occurrences of a particular query.

  {
    "jsonrpc":"2.0",
    "method":"eth_call",
    "params":[{
      "from":"0x2D2af99CCeBc1Bf26849732f3a0ac2DAfa982421",
      "gas":"0x5f5e100",
      "gasPrice":"0x174876e800",
      "to":"0x01E11a017c18551863F244203f1aDCd50DA43c3a",
      "data":"0xcefe0e210000000000000000000000000000000000000000000000000000000005f5e100"
    },"latest"],
    "id":417
  }

Whoever was bogging down our systems was running the exact same query dozens of times per second. This is a tell-tale sign of a Denial of Service (DoS) attack. What’s more, the request was timing out.⁶ Whoever was running the query wasn’t getting a useful answer, so blocking it wasn’t going to cause them any new problems. We set up a filter to block this request and I went back to making dinner.

It wasn’t long before I was disrupted again. One of the problems with blocking specific queries is that it can be trivially easy for a determined attacker to circumvent. In this case, we had blocked calls dealing with a particular Ethereum smart contract ⁷, so all the attacker had to do was redeploy the contract at a different address to resume their attacks.

We ran through our process again, and determined that the smart contract 0x8eec0acff407e82531031a10c4673a08a057d844 was the culprit this time, which was deployed just minutes after we blocked the previous contract by the same address that had deployed the original contract. But the attacks this time only lasted a few minutes, and we neglected to block the offending address (which, of course, we would come to regret).

Continue the story on Monday