This is part 7 of an 8 part series. If you haven't read from the beginning, we suggest starting with the intro.
Saturday, April 25, 2020 - 1:13 PM
Friday night passed without any new attacks. Our team finally got a good night sleep, confident that we had the issue in hand. I spent a relaxing morning with my wife and kids. Then, just after 1:00 in the afternoon, I got a stunning reply from Dr. Tang.
Dear Mr. Austin Roberts, greetings.
We are a group of international academic researchers and ethical hackers, with expertise in blockchain security.
Our tests intend to discover bug/design flaws of your services and to facilitate mitigation design. We try our best to minimize and avoid the damage to your service and other customers using your service. In our test on April 24, we sent a total of two (crafted) requests to your service. We apologize for the inconvenience caused here.
We notice some protective measures have been recently deployed to your service. Unfortunately, we believe they can be bypassed or further exploited (by others on the Internet).
Under your permission, we want to help you to discover and fix upcoming DoS vulnerabilities in your service. Our intention is to help our collaborating service(s) in terms of DoS security without losing the quality of their services (QoS), for them to gain competitive advantages in the market. We thus humbly ask for your interest in this collaboration. Of course, if not, we will stop interacting with your service. Either way, we’d like to hear from you. Thank you.
Attached please find the bug report by courtesy of my colleagues, for reproducing the problem and preliminary mitigation strategies. If you could confirm the bug (on your previous service), we would greatly appreciate it.
With this bug report attached.
It wasn’t Russian hackers. It wasn’t anyone trying to manipulate token markets. It wasn’t a bored college kid acting on his own. It wasn’t even a research lab’s computer that had been compromised by a third party. A professor from a major US University had just admitted that his research group of “ethical hackers” was responsible for a week of persistent attacks that took place at all hours of the day and night. I was floored.
We learned a lot from this letter:
First, it became clear from the bug report that they believed they were attacking a specific one of our customers, and did not even know they were attacking Rivet. It’s not uncommon for our customers to put Rivet behind their CDN 1 so their users have no indication they are getting data from us.
Second, Dr. Tang had a number of false or misleading statements in his letter, probably to protect himself or his students from repercussions within the University. To elaborate:
- Dr. Tang only admitted to the attacks on April 24th, as I had indicated in my email that was the only day we could link his IP address to the attacks.
- They did not send a total of two crafted requests on April 24th. We have server logs showing hundreds if not thousands of requests crafted to harm our service. Even the bug report he provided indicates that the crafted requests were sent repeatedly for a period of 60 seconds.
- It is not possible that the graph included in the bug report came from April 24th, as we blocked all of their attempts that night. Instead, we believe those measurements came from the attacks on April 21st or 22nd.
- In the bug report, Dr. Tang cites a smart contract deployed at the address 0x01E11a017c18551863F244203f1aDCd50DA43c3a. We had blocked that address on April 19th, and it was not used on the April 24th attacks.
- The smart contract source code provided in the bug report corresponds neither to the smart contract deployed at the specified address, nor the contract used on April 24th; instead it corresponds to the contract that crashed our application on April 21st.
When Dr. Tang indicated that they believed our mitigation efforts could be bypassed and exploited by others on the Internet, this is because we blocked their contracts by address. After we deployed our gas cap limits on April 22, we blocked all of their contracts before they had a chance to attack us with them, so they never had a chance to see their attacks land as intended.
Finally, the proposed mitigations in Dr. Tang’s bug report were essentially worthless. We had already implemented the gas cap. His static code analysis proposals fundamentally misunderstand properties of the Ethereum Virtual Machine, and even if they could be implemented would be far too restrictive for a general purpose Ethereum Gateway. His anomaly detection proposals, while interesting, miss the fact that we already have completely legitimate requests running against our service that would be blocked by his proposal; one of our top priorities while mitigating his lab’s attacks was to avoid interfering with our customers’ legitimate use cases, but his purely academic idea of a mitigation effort would cut off important queries run by our biggest customers.
Needless to say, we had no interest in working with Dr. Tang’s lab, and were deeply concerned by the idea that the University of Syracuse was sanctioning these activities.
Read the conclusion
A Content Distribution Network, or CDN, is a network of servers all over the world that accelerate the performance of a website. When you connect to this blog, for example, you are likely connecting to a server in your own city instead of our data center in Ohio; if the server in your city had this page it gave it to you right away, if it didn’t, it would have contacted our Ohio data center. When a customer puts Rivet behind their CDN, it may be that the CDN routes requests for
example.com/app/to their servers in London, while
example.com/eth/gets routed to Rivet servers in Ohio. From the customer’s perspective they’re talking to an example.com server geographically close to home. ↩