For the Embiggen Airdrop we loaded OpenRelay with 27 million orders ready for people to fill. Being able to prune an order book with 27 million orders was a big technical challenge, and today we explain how we did it.

One of the biggest responsibilities of a relayer is making sure their order book is current. This means making sure that orders that have expired, have been filled, or for which the order’s maker doesn’t have tokens to fill the order, aren’t showing up in the order book.

Before we set out to handle the airdrop, an automated process reviewed all of the orders on the OpenRelay order book every minute to detect any orders that had become unfillable. For each order, we made 4 Ethereum API calls to check if:

Under that model, scaling to 27 million orders would entail running 108 million RPC calls every minute. By comparison, publicly available information suggests Infura currently handles about 3.5 million RPC calls a minute.

We didn’t want our solution for reaching 27 Million Orders to treat the Embiggen airdrop as a special case. We could say “we control these addresses, we know those orders will always be fillable” and not bother checking them, but the point here is to make our system more robust for our users, not just for internal exercises.

The 0x.js Order Watcher

We knew we needed to move to an order watcher system. 0x.js ships with an order watcher tool. The basic idea is that you register orders with the Order Watcher, and it monitors the block chain for state changes that would make the order unfillable. We had several problems with the 0x.js order watcher.

First, the 0x.js Order Watcher is written in JavaScript, while nearly all of OpenRelay is written in Go. Our microservices architecture enables us to mix and match languages for different services where we must, but we’d rather have Go service. By itself this wouldn’t have kept us from using the 0x.js Order Watcher, but it didn’t make us excited.

Second, the 0x.js Order Watcher keeps details about all of the orders in memory. We estimated that holding all of our 27 million orders would require a minimum of 10 GB of RAM, assuming an efficient memory model (which JavaScript is not known for). 10 GB of RAM is doable, but we’re trying to keep our systems leaner than that.

Another consequence of keeping all of the order information in memory is that if the order watcher crashes or has to be restarted, it has to be reloaded with all of the orders, which is a time consuming process when you’re dealing with over 27 million orders.

Finally, the 0x.js order watcher only checks for state changes in new blocks. There’s no way to tell it “start checking for state changes 1,000 blocks back”. So if your order watcher goes down, not only do you have to load 27 million orders back into the order watcher, you have to check whether each order is fillable first (using the same 108 million queries mentioned earlier). Having to run 108 million queries when you have to restart a service is better than having to run 108 million queries once a minute, but it was far from what we considered acceptable performance.

The 0x.js order watcher still has a place on the client side. If you query OpenRelay for an order and need to keep track of it while you decide whether or not to fill it the 0x.js order watcher can be very helpful, but it wasn’t going to meet the needs of an order book at the scale we were aiming for.

The OpenRelay Order Watcher

The OpenRelay order watcher takes a moderately different approach. We broke the order watcher into several microservices (all written in Go), following our message passing architecture.

We have one service that polls the blockchain, watching for new blocks. When a new block is mined, it emits a message containing:

In the event of a chain re-org, it will emit all of the events from the new chain. If the block monitor gets restarted, it looks up which block it left off on and resumes from there.

We have three different consumers of the block monitors as it pertains to the order watcher subsystem. Each of these consumers pulls blocks from a persistent message queue, discovering blocks as they are emitted by the block monitor, without any RPC calls. The consumers check the block’s bloom filter to see if it has any events pertinent to that consumer. If the bloom filter indicates an event is present, only then will the consumer query the RPC server for pertinent events. Because these consumers pull blocks from persistent queues, they can be restarted without any complicated or time consuming resumption processes.

The Allowance Watcher

The allowance watcher consumes blocks from the block monitor, and checks to see if anyone changed the 0x Exchange Contract’s allowance to pull from their account. It iterates over any such events, and emits a Spend Record to the Spend Indexer downstream.

The Spend Watcher

Like the allowance watcher, the spend watcher consumes blocks from the block monitor and looks for any token transfer events at all. If it finds them, it queries the RPC server for the spender’s new balance of the token, and the allowance that spender has set for the 0x Exchange contract, if any. Given that information, it emits a spend record to the Spend Indexer downstream.

The Spend Indexer

The Spend Indexer watches a persistent queue for Spend Records, which originate from either the Allowance Watcher or the Spend Watcher. When it receives a Spend Record it runs a SQL query to update the status of any order where the maker corresponds to the spender in the spend record, and the remaining fillable amount of the maker token is less than the spender’s remaining balance.

Since the Spend Indexer gets spend records from every single token transfer on the Ethereum blockchain, most of those spend records will not update any records in the database. But the query has been tuned to efficiently find relevant records in our SQL database, so that there’s not a huge overhead. Given that the Ethereum blockchain processes 15 transactions per second on a busy day, and only a fraction of those are token transfers, the spend indexer should have no problem keeping up. We may have to revisit this approach once sharding is in place, but for now it’s a workable solution.

The Fill Monitor

Aside from users spending their tokens, orders can stop being fillable because they have been filled or cancelled. We have had a fill / cancel monitor since OpenRelay first launched. The original was written in JavaScript, but we rewrote it in Go to take advantage of our block monitoring system. The fill monitor watches blocks for fill and cancel events, then emits Fill Records onto a queue.

The Fill Indexer

The fill indexer didn’t actually require any changes to work with the Go-based fill monitor. The Go fill monitor emits the same Fill Records that the JavaScript fill monitor emitted. The fill indexer consumes those fill records, and updates any corresponding orders in the database as they are filled.

Other Considerations

Watching events isn’t guaranteed to catch everything that may make an order unfillable.

On one hand, you have some tokens that allow minting and destroying new tokens. Most tokens emit Transfer events when they mint or destroy tokens, but some (such as the original WETH contract) do not. Our current order watcher system may miss such events.

On the other hand, Embiggen demonstrates that it’s possible for balances to change without any transactions occurring at all. Since Embiggen only grows it will never make an order unfillable, but if someone created an enshrinkening token it might become unfillable without any events.

For right now we’re accepting this risk. We plan to identify specific tokens that have the possibility of balances decreasing without monitorable events, and we will take an optimized approach of checking whether each order is fillable on an iterative basis just for that subset of tokens.

Final Thoughts

As the name implies, OpenRelay is Open Source. If you want to see how this works in detail, come check us out on Github. We believe we’ve developed a pretty great order watcher, and would invite other teams to leverage it if it looks useful to you. If there’s any interest in exposing our order watcher as a service (making our spend and fill records available via a public API) let us know. We’re currently tracking every token spend and allowance change on the blockchain, so it wouldn’t be a huge stretch for us make them available for public consumption.

And lastly, if you haven’t claimed your Embiggen from the Embiggen Airdrop, head over to get.embiggen.me to claim some Embiggen. You’ll be helping us test OpenRelay, and thanks to the power of compounding interest, the sooner you claim your Embiggen the more you’ll have over time.