Polkadot runtime arbitrage - Preventing perverse incentives of block producers

This is not a proposal for funding but rather an idea I'd like to open up for economic and technical discussion. I'd like to know if the system I describe is possible under Polkadot technology, and get some technical guidance on the best way it could be implemented if it is. I'd also like to hear criticisms and reasons why this might not work at all.

The problem of DeFi

Arbitrage is an essential component of a healthy financial ecosystem. The action of arbitrageurs naturally moves prices to equilibrium levels, and contribute to liquidity between markets. Without arbitrage, markets would be less efficient.

On Blockchain many DeFi applications are built with arbitrage in mind, relying on the actions of arbitrage traders to control the balance and prices of various financial assets, such as DEX's and other forms of liquidity pools.

However, arbitrage on Blockchain can create perverse incentives, both from the user experience, and even from a consensus-level itself.

Arbitrage traders will naturally compete with each other, and on traditional Blockchains they do so by bidding against each other in a fee-market, driving up prices for transactions, and lowering the usefulness of the Blockchain for typical users.

On a consensus-level, the effects can be even more perverse. Arbitrage transactions are highly profitable, and constitute another form of "consensus-creator extractable value" (CCEV), by either miners, collators, validators, or other parties involved in the consensus mechanism.

Consensus relies on the people providing it to be incentivised to behave in a socially optimal manner for the network. The ability of parties who decide consensus to monitor and detect arbitrage profits, gives financial motive to engage in front-running, block orphaning, transaction denial, and other malicious behaviours. As the DeFi ecosystem grows, and billions of dollars are involved, the profit incentives from bad behaviour can begin to compete with those of good behaviour.

It's natural to feel that taking arbitrage off the table altogether is a good solution, for example randomising the order of exchange executions. However, any obstruction to efficient arbitrage will conversely have a negative effect on overall market liquidity and harm the speed of price discovery.

It's therefore important that the issue of CCEV is dealt with appropriately. A healthy DeFi ecosystem must nurture arbitrage itself, while simultaneously structuring incentives on the network to deter harmful behaviour.

Optimising an arbitrage incentive system

In this writeup, I explore a protocol-based system which nurtures healthy price discovery and liquidity, while preventing the misalignment of network incentives. The aim is to protect the network against being victimised by its own success in attracting mainstream financial applications.

Such a protocol is only currently possible on a Substrate-style chain, as the solution is built into the runtime system of the Blockchain itself, capturing CCEV and sharing it amongst protocol participants, removing the risk of user-experience damaging fee markets, and dis-incentivizing bad behaviour by consensus creators.

Conceptual overview of runtime arbitrage

(1) Right of first execution

DeFi applications on the Blockchain subscribe to a component of the runtime, and give the component "right to first execution".

Whenever a new transaction is executed by the DeFi application, the application checks the runtime components stack for a pending transaction, and executes it first if it is present.

The essence of “right of first execution” is to impose a voluntary transaction ordering on the block, whereas the runtime component can always execute a transaction ahead of participating applications.

Note: The system could leave the DeFi applications out of the equation altogether, with the runtime component being given absolute right of first execution each block. However, this may create perverse incentives against applications wishing to use the blockchain.

(2) Candidate transaction pool

Arbitrage traders submit "candidates transactions" to the runtime component, which are placed in an ordered list. They stake a nominal amount of funds to do this, but are only slashed if they behave badly by publishing junk transactions into the list.

If possible, candidate transactions would include a proof of the effects of execution, which can quickly be verified.

Each new block, the runtime component checks the effects of the transaction, to find the transaction which brings the most total arbitrage profit back to the component. If two transactions have equal profit the first-in is chosen as the winner.

This implementation delegates the mathematical computation of arbitrage to outside participants, removing the need for the runtime component to concern itself with discovering the most effective trades - which is not feasible from a computing resources perspective.

Note: Typical arbitrage calculations involve running a shortest-path algorithm (Bellman Ford, Flloyd’s, etc) to find profitable trips, followed by a profit maximization over discovered trips through binary search of simulated trade sizes. However, the computations involve a high degree of nuance and constant adjustment.

(3) Arbitrage execution

The candidate transaction is queued for execution by the first DeFi participant in the protocol. It is executed, and the profit is shared between the candidate provider and all of the participants in the protocol who give the right of first execution to the runtime component.

Candidate transactions can include exchanges and DeFi applications who aren't subscribed to the protocol, but these applications do not share in the rewards given to protocol participants.

(4) Incentives of participants

Liquidity providers are incentivised to provide liquidity to pools which participate in this system, due to the arbitrage profit distribution resulting in higher yields.

Arbitrage traders are incentivised to provide transaction candidates to the protocol, as they risk no fees if the trade fails, and can gather more profits by being provided with the right of first execution.

Regular DeFi users benefit from lower costs of DeFi application interaction, and more efficient price discovery.

The opportunity to profit from many forms of bad behaviour is removed from consensus providers, strengthening the overall network.

Protocol participants & components

(1) Compute engine (off-chain worker)

The compute engine performs the validation logic of emulating and selecting appropriate candidate transactions for the on-chain stack. This computation is done independently to runtime execution so as not to interfere with functionality of the chain. The compute engine receives a small proportion of rewards to incentivise good behaviour, and has a slashable stake to discourage bad behaviour.
More info: https://substrate.dev/docs/en/knowledgebase/learn-substrate/off-chain-workers

(2) Candidate stack

This is a stack of selected transactions pending arbitrage execution. Any participating DeFi application must execute a single transaction from the top of the stack before any of its own transactions, if the stack is not empty.

(3) DeFi applications

DeFi applications plug into the protocol by granting the right of first execution to it, and share in the resulting arbitrage profits.This right is secured by a small stake of funds, which can only be slashed if the DeFi application reneges on the right of first execution.

(4) Liquidity providers

Liquidity providers are incentivised to share their funds with participating DeFi applications, as the alternative is to allow others to profit from their funds.

(5) Arbitrage traders

Arbitrage traders assume no financial risk from failed trades, and are incentivised away from fee-market wars.

(6) Regular DeFi users (non-LP)

Regular users see more homogenous prices across financial products, and face lower costs of transacting.

(7) Supporting blockchains

Supporting blockchains benefit from better network incentives that encourage good behaviour, while fostering the healthy growth of DeFi technology.

Considerations & open questions

(1) Handling of the candidate queue

It’s important to prevent any mechanism by which a bidding system can erupt in the candidate queue, or by which the execution in the queue can be manipulated.

(2) Assurance of security to DeFi applications

The system should be trustless and its operations verifiable by DeFi applications.

(3) Integration simplicity

The system should require minimal effort for a DeFi application to integrate.

(4) Handling sandwich and other in-block arbitrage

Sandwich arbitrage is where a trader monitors the mempool for incoming DeFi trades which will sufficiently move the spot price of a liquidity pool. Once such a trade is discovered in the mempool the trader attempts to “sandwich” it, by pushing one of their own transactions ahead of the incoming trade in the block, and a second transaction behind it. The first transaction artificially raises the pool price with a buy order for the opposing asset, and the second transaction takes profit with a sell.

Other forms of in-block arbitrage include pushing arbitrage taking transactions into blocks where it is created, however this is very risky and unreliable as you need to price yourself lower than the arbitrage creating transaction while still being included in the same block. Sandwich arbitrage is not as risky because failure of the second transaction still leaves the trader with assets in-hand under market price.

(5) Preventing junk transactions

It's important that the runtime component is not bombarded with junk transactions. A requirement to submit candidates is therefore a staked amount of funds, which is slashed for submission of junk (transactions which yield insufficient profit).

(6) Efficient simulation by runtime component

It must be as simple as possible for the runtime component to validate the effects of queued transactions. If possible, validation would be delegated to candidate transaction providers, since in a normal arbitrage environment they are performing this work already by simulating opponent transactions in the mempool to search for racing.

(7) Proving right of first execution

Participating DeFi applications stake an amount of funds, which can be slashed if it is provided they did not provide the right of first execution to the protocol.

Future expansion of such a system

(1) Other forms of CCEV

The system does not need to be confined to arbitrage, and could include any other form of CCEV.

(2) Multiple candidates selected

If the runtime verification of the system can be appropriately scaled, it would be beneficial to have multiple winners if the results of transactions do not clash with each other.