This is part 5 of an 8 part series. If you haven't read from the beginning, we suggest starting with the intro.
Thursday, April 23
After several days of attacks, it was time for a much more thorough post-mortem than the one we had done Monday:
- We identified several additional steps we could take to protect ourselves from alternative Denial of Service attack vectors not employed by this weeks’ attacker.
- We signed up for Blocknative Notify to get alerts the moment our attacker’s next contract hit the mempool 1.
- We started logging the IP address of requests matching any of our attack signatures 2.
We also began to speculate on the motivations of an attacker. At this point, we weren’t 100% sure who the target of these attacks even was. Were they targeting us, or were they going after one of our customers? Was this a Russian or Chinese hacker going after blockchain infrastructure? Was someone trying to hurt one of our customers in an attempt to move token markets 3? Or was this just somebody who wants to watch the world burn? At the time, we were guessing we’d never know.
We also did some research to see whether any other node service providers had talked about imposing gas limits on requests. We found this Infura discussion thread from November 2019 where the community discussed problems with the introduction of eth_call gas limits on Infura, and an Infura employee apologizing that those limits had been added to mitigate DoS attacks. Meanwhile, the first Ethereum contract used in attacks against us was deployed to the network just a week before that discussion thread. Was it possible that the attacker coming after us was the same person who prompted Infura’s introduction of gas limits back in November?
We spent time on Thursday working on some additional mitigation efforts, anticipating other attacks we will eventually see, and cleaning up after the week. We made sure our testing environment was aligned with production, and that all of the testnets 4 we support matched our mainnet deployment.
All the while, we were expecting alarms to start going off. But at least on Thursday, they never did.
-
With the Ethereum blockchain, the mempool is a collection of transactions that have not yet been officially recognized on the blockchain. Transactions live in the mempool for anywhere from a few seconds to several hours before they become officially confirmed transactions, giving us the opportunity to block the next contract before the attacker can possibly use it. ↩
-
To protect our users’ privacy we record as little information about our users as necessary to provide our service, so we don’t normally log information about requests beyond a count of the requests by each customer. We have our slow query log to help ensure quality of service, but up until this time the slow query log didn’t record IP address or even which customer’s API key made a request. We decided to capture that information during the next attack, if there ever was a next attack. ↩
-
Some of our customers have done Initial Coin Offerings, or ICOs. They sold a token on the Ethereum blockchain to fund their initial development, and offer some kind of value back to token holders. We speculated that if someone shorted one of those tokens they might be hoping to drive down the market price by attacking the infrastructure supporting the token’s project. ↩
-
Blockchain testnets are low-stakes peer-to-peer networks that act like the official “mainnet” network, but are intended for testing purposes rather than high value transactions ↩
