To identify queries impacted by our changes, we took the data recorded by our random query log and fed it through a tool called Versus, which runs the query against two different servers and compared the results. We compared our testing environment (which had the change) to our production environment (which did not), and identified the small handful of queries that behaved differently after the change.

This complicated things.

At OpenRelay we subscribe to the philosophy of immutable infrastructure. That is to say, we almost never update a server; we replace it with an updated server.

Our servers are behind a load balancer that only routes to healthy servers. During a deployment new servers start up, but are not considered healthy until they have all the latest data pulled down, and enough of it loaded into memory that they can answer queries quickly. Under normal circumstances this means that we can start new servers, they don’t get any queries until they are able to handle them, then we can shutdown the old servers and all requests get routed to the new ones.

When we’re under attack, this approach becomes a problem. We’re rather conservative about when we consider a server to be healthy. If response times start to creep up on a server or too many requests get backlogged, a server will mark itself as unhealthy to give it a chance to catch back up. Usually this ensures a high level of performance. When you experience a DoS attack during a deployment, it’s a problem.

The DoS attack caused all of our old servers’ response times to exceed their healthy threshold and mark themselves as unhealthy. When all of the servers are unhealthy, the loadbalancer routes requests to all of them. If we hadn’t been in the middle of a deployment, this would have meant that all of our servers shared the load. Response times would be high, but the error rate would remain low.

But since we were in the middle of a deployment, the existing servers getting marked as unhealthy meant that the new servers that had yet to come online started getting routed requests, and instead of just having high response times, these servers were returning errors. We manually removed the new servers from the loadbalancer until they were ready to serve requests.

We got pretty lucky that our attacker was just using a single Ethereum address for all of their attacks. Had they switched up what addresses they deployed contracts from, we would have been ten minutes into an attack before we had the contract blocked. They also could have executed the attack in the constructor function of a smart contract deployed with eth_call — the contract doesn’t actually get deployed to the network, but our node would still execute its code, which would be much harder to block.