The Arbitrum network experienced significant downtime on December 15 due to problems with its sequencer and feed. The network had been down for almost three hours. The major outage began at 10:29 a.m. ET amid a substantial increase in a type of network traffic called Inscriptions. Arbitrum’s layer-2 network had processed over 22.29 million transactions and had a total value locked of $2.3 billion. Despite the success of the network, the current design suffers from a significant chokepoint when posting transactions to L1, causing the sequener to stall. While advancements such as Arbitrum Nova and Proto-danksharding might alleviate these design issues, this is not the first time Arbitrum has experienced such issues – a bug in the sequencer also halted the network in June 2023.

Arbitrum Sequencer Outage | Background

Arbitrum is a Layer-2 (L2) solution which settles transactions off the Ethereum mainnet. L2s provide lower gas fees and reduce congestion on the primary blockchain (In this case, Ethereum, L1). The current incarnation of Arbitrum is called Nitro. Arbitrum Nitro processes transactions in two stages: sequencing, where transactions are ordered and committed to this sequence, and deterministic execution, where each transaction undergoes a state transition function. Nitro combines Ethereum emulation software with extensions for cross-chain functionalities and uses an optimistic rollup protocol based on interactive fraud proofs. The Sequencer is a key component in the Nitro architecture. Its primary role is to order incoming transactions honestly, typically following a first-come, first-served policy. This is a centralized component operated by Offchain Labs. The Sequencer publishes its transaction order both as a real-time feed and to Ethereum, in the calldata of an “Inbox” smart contract. This publication ensures the final and authoritative transaction ordering. Additionally, a Delayed Inbox mechanism exists for L1 Ethereum contracts to submit transactions and as a backup for direct submission in case of Sequencer failure or censorship.

Arbitrum Sequencer Outage | Root cause

In the two hours prior to the outage more than 90% of Arbitrum traffic consisted of Ethscriptions. Ethscriptions are digital artifacts on EVM chains created using Ethereum calldata. Unlike traditional NFTs managed by smart contracts, Ethscriptions make the blockchain data itself a unique NFT. They are inspired by Bitcoin inscriptions (Ordinals) but function differently. Creating an Ethscription involves selecting an image, converting it to data URI format, then to hexadecimal format, and finally embedding it into a 0 ETH transaction’s Hex data field. Each Ethscription must be unique; duplicate data submissions are ignored. Owners can use Ethscriptions IDs for proof or transfer of ownership. In practice the calldata or Ethscriptions look like the code below:

data: {"p":"fair-20","op":"mint","tick":"fair","amt":"1000"}

Calldata example of an Ethscription. This represents a token mint.

Since Ethscriptions are very cheap, one can do a lot of them for the same unit of cost. Indeed, a staggering 90% of transactions posted on-chain were Ethscriptions. Also, for a relatively low cost, the amount of transaction entropy that needed to be committed to L1 increased to 80MB/hr vs. the 3MB/hr that was typical before the traffic spike. We calculated this by looking at average on-chain transaction postings for the sequencer.

Now, look at the architecture diagram of Arbitrum below. Note that in order to commit transaction sequences to L1, the data poster needs to post the increased amount of data over a larger number of transactions. Prior to the outage, the number of transactions posted per hour was around 10 – 20x higher than the December mean.

However, the code responsible for posting these transactions has an in-built limitation that imposes limits to the rate at which L1 batches are posted. Prior to the outage, if there are 10 batches still in the L1 mempool, no more batches are sent to L1, stalling the sequencer. This limit was subsequently raised to 20 batches after the outage. This is probably not a good long-term solution however, as it increases the chances of batches needing to be reposted due to transaction nonce issues.

// Check that posting a new transaction won't exceed maximum pending
// transactions in mempool.
if cfg.MaxMempoolTransactions > 0 {
  unconfirmedNonce, err := p.client.NonceAt(ctx, p.Sender(), nil)
  if err != nil {
    return fmt.Errorf("getting nonce of a dataposter sender: %w", err)
  }
  if nextNonce >= cfg.MaxMempoolTransactions+unconfirmedNonce {
    return fmt.Errorf(
      "... transaction with nonce: %d will exceed max mempool size ...",
      nextNonce, cfg.MaxMempoolTransactions, unconfirmedNonce
    )
  }
}
return nil

Batch poster is responsible for posting the sequenced transaction sequence as Ethereum calldata.

Arbitrum Sequencer Outage | Recommendations

There are several indications that point towards the sequencer, and thus the network, not being tested enough in a realistic setting or in an adversarial environment. However, luckily the upcoming Proto-Danksharding upgrade to Ethereum should also help for reducing L1-induced congestion. Irrespective of this the Arbitrum engineers can consider the following recommendations:

• Whether the Arbitrum gas price of L2 calldata is set too low, compared to other kinds of operations. Gas is an anti-DoS mechanism, which is intimately tied to the L1 characteristics. If this increase in L2 calldata causes a proportionally large increase in batch size, then attackers can craft L2 transactions with large calldatas that result in batches that don’t compress well under Brotli compression, causing a DoS attack on the sequencer. Note that Arbitrum Nova should not suffer as much from this issue as the transaction data is not stored on L1, only a hash is.
• Whether there is a tight feedback loop between the size of the L1 batches currently in the mempool and L2 gas price. There is an indirect feedback loop, via the gas price on L1 and backlog sizes, but this may not be too tight. In addition, since the sequencer is centralized anyway, anti-DoS measures might be encoded directly into it to reject transactions. (Note: A more decentralized sequencer is being considered for the future, so this last measure wouldn’t work)
• Long-term, the engineers more research into making the rollups more efficient to decrease the sizes of batches committed to L1. This may include ZKP rollups at some point.
• Additionally, security audits to the sequencer should consider DoS situations, both through simulation/fuzzing and also by having auditors think of hostile situations through adversarial thinking based off their deep knowledge of the involved chains.

Finally, the Arbitrum team made a small change to the way transactions are soft-committed. In this change the feed backlog is populated irrespective of whether the sequencer coordinator is running, which carries its own risks but enables dApps running on Arbitrum to be more responsive during certain periods.

Disclaimer: The Arbitrum sequencer is solely operated by Offchain labs. Thus, most of the information regarding its operational issues (such as logs) are not publicly available so it’s hard to get a complete picture of the issue. Dedaub has not audited Arbitrum or Offchain labs software. Dedaub has however audited other (non-Arbitrum) software and projects running on Arbitrum such as GMX, Chainlink, Rysk & Stella.

Summary: The root cause of the thirdweb critical vulnerability is that independent libraries implementing ERC2771 & Multicall, such as OpenZeppelin Libraries, interact badly, when combined. This allows attackers to spoof the _msgSender() with all sorts of access control implications including loss of funds.

The issue is complex, but can be explained using a simple analogy. Imagine a bank that will let one of the bank officials carry out a transaction on your behalf, as long as the instruction is written on a piece of paper with your verified signature. This is a very common scenario, for instance with some preferred bank clients. So, you go to the bank official and hand him a signed piece of paper. Your instructions are “take this sealed box to the cashier, open it, and give him what’s inside”. The bank official happily executes your signed instructions, after checking your id against your signature. The sealed box contains another piece of paper reading …”do a withdrawal on behalf of Elon Musk”, signed with a fake signature. The cashier takes this piece of paper from the bank official, thinking that the signature was checked, when, really, the only signature that was checked was on the instructions to deliver and open the box. That’s it!

Now let’s look into the technical mechanics for how this vulnerability works, and how to protect your project from this issue.

The Critical Thirdweb Vulnerability | Background

First, we need to cover some preliminaries. In particular we need to first understand the implementation of the ERC2771 standard and the OpenZeppelin Multicall library. ERC2771 gives the ability to have a “virtual” msg.sender, i.e., caller of a public function of a smart contract.

ERC2771 defines a contract-level protocol for Recipient contracts to accept meta-transactions through trusted Forwarder contracts. No protocol changes are made. Recipient contracts are sent the effective msg.sender (referred to as _msgSender()) and msg.data (referred to as _msgData()) by appending additional calldata.

ETHEREUM ERC-2771

Therefore, this virtual msg.sender, called _msgSender() is set by a trusted external party, the forwarder. And how does the forwarder tell the contract what is the virtual msg.sender? It appends an extra parameter to all calls. This means that all functions of a contract that supports such virtual msg.senders need to take in an extra parameter which they interpret as the msg.sender. The other side of the vulnerability is Multicall. It is a way to have a single call that becomes many calls (to the same contract) in sequence. How does this happen? By making all the info of the “many calls” be parameters of the “outer” single call.

The Critical Thirdweb Vulnerability | Root cause?

The problem with these two libraries is that the forwarders (in ERC2771) were not designed to work with multicall. They add a single _msgSender() parameter to the outer call of a multicall. But remember: all functions now expect this parameter! Where can they get it from? The parameters of the *outer* multicall.

So, if an attacker uses multicall to call, say, 3 functions in sequence, the attacker can define all the parameters to these function calls, including the _msgSender()! This means that the attacker can make a call appear to be coming from anyone!

The Critical Thirdweb Vulnerability | Evaluating the impact

We have tried to reach out to most large projects that might have been affected (in collaboration with thirdweb and OpenZeppelin) over the last few days. However, if you are worried about this issue affecting your contract, we have flagged any contract affected on Watchdog and made this information available to the public. However, the extent to which your contract is affected depends on actual implementation of the contract. First, evaluate functions with access to _msgSender() (transitively). Do these functions contract check access control mechanisms using _msgSender()? For example, can someone withdraw or burn coins for the _msgSender()? In that case, the issue affects your contract, critically. In many of these contracts there may be onlyOwner or onlyRole modifiers that make use of _msgSender(). In addition, look for common transfer functions such as safeTransferFrom() or transfer(). The effect is also modulated by the value of the assets held by the contract, or if this contract represents an asset. Make sure to find out if your contract is a token in a Uniswap-like liquidity pool. It is possible that all the liquidity in this pool could be stolen due to this issue.

The Critical Thirdweb Vulnerability | Mitigation

The rest of the article outlines mitigation. Thirdweb has developed and deployed a mitigation tool that can possibly assist you. A large number of affected contracts were deployed by their product. However, oftentimes you’d need to take additional actions. Should you require assistance the team at Dedaub can at least point you to the right information. You may contact us here. In the rest of the article we list some some mitigation options we’ve observed over the last few days to be successful. Legal disclaimer: This should not be construed to be professional advice by our team.

PREFERRED MITIGATION: DISABLE TRUSTED FORWARDER

Some ERC2771 library implementations allow resetting a trusted forwarder. Doing so will prevent any gasless transaction from being executed through the forwarder, solving the issue (albeit at the cost of missing functionality). Unfortunately there are many instances where resetting the trusted forwarder is not possible, so the rest of the mitigation steps apply. Otherwise, your smart contract is probably safe from this issue.

ADVANCED MITIGATION METHODS

These mitigation methods may take time and expertise to successfully execute. If time is critical, you can consider decreasing the blast radius in the next section in case an attacker hacks the contract while you are in the process of planning a mitigation.

If your contract is Upgradeable, prepare an upgrade. Removing either multicall (and all functionality that delegatecalls to the same smart contract) prevents the attack. In addition, removing ERC-2771 functionality also prevents the attack. Other ways to prevent this attack involve adding a module that allows doctoring of the contract’s storage and removing the trusted forwarders in this way. This latter is difficult to execute correctly.

DECREASING THE BLAST RADIUS

Some steps can be taken in cases you do not manage to mitigate the issue in a timely fashion to limit the amount of stolen assets from your contract. This can be done in several ways:

1. Ask your users to remove approvals from your contract. You can additionally check which users have approved your smart contract to transfer funds by checking on app.dedaub.com, navigating to your smart contract, navigating to balances and then allowers. Note that publicly announcing removal of approvals can work both in favor and against you since malicious hackers could be tipped off.
2. Pausing your contract may help users from continuing to use it, but, depending on the implementation it might not prevent the attack.
3. Remove liquidity from Uniswap-like pools in case the token is held by a pool, otherwise the liquidity in this pool may be drained in some cases.

Conclusion

The thirdweb vulnerability is an unfortunate issue that came about due to the composability of libraries in a single smart contract, through inheritance mechanisms. Unfortunately, although libraries are supposed to be abstractions, when it comes to security abstractions can easily be broken and implementations can affect each other in unforseen ways. This was even the case despite the overwhelming majority of affected libraries were developed by the same organization. In their defence however, it is very hard to make libraries interoperable, and, furthermore even harder to make them upgradable. Our audit team at Dedaub regularly finds issues in smart contracts that employ “safe” 3rd party libraries. Our decompiler and contract analysis tools really help in such cases as they work on the actual deployed code of a smart contract. We regularly find issues related to upgradability, but other issues may be lurking.

We would like to commend the work of countless other security engineers who have helped reach out to affected projects!

Platypus Finance Hack: The platform was targeted by a flashloan attack, resulting in an approximate $2 million loss. This sophisticated attack utilized flashloans alongside tactics to alter slippage calculations in various swaps, thereby manipulating the price of the swapped assets to benefit the attacker.

Platypus Finance Hack | The Attack Summary

At 12th Oct 2023, 06:32 UTC, an attacker on Avalanche C-Chain (addresses: 0x0cd4fd0eecd2c5ad24de7f17ae35f9db6ac51ee7 & 0x464073F659591507d9255B833D163ef1Af5ccc2C), performed multiple on-chain transactions via smart contracts deployed within the same transaction itself. We shall concentrate on a single instance on this attack, where the attacker profits around $570k. The operations performed are as follows:

Flash loan on AAVE
Deposit 1050k WAVAX
Deposit 316k sAVAX
Swap 600k sAVAX to 659k WAVAX
Withdraw 728k WAVAX
Swap 1200k WAVAX to 1250 sAVAX
Withdraw 34k WAVAX
Swap 600k sAVAX to 840k WAVAX
Withdraw 316k sAVAX
Repay AAVE Flash Loan

Note that the deposits and swaps are performed on a relatively novel “Stable Swap” AMM (Platypus).

Judging by the events that took place and by looking at calculations performed throughout the attack, we are fairly sure that the root cause of this attack occurs due to a manipulated slippage calculation. Moreover, the mechanism employed in calculating the slippage (and thus the price at which a swap takes place) is flawed, in cases where liability balance and cash balances are manipulated differently. When such a condition arises, the slippage manipulation can be in the attacker’s favor in both directions of the swap, thus breaking the invariant that the slippage is symmetric.

In order to understand the intimate mechanics of the protocol, let’s first back up and look the key features of the Platypus AMM and lending protocol:

1. Unilateral Liquidity Provision: Platypus allows users to provide liquidity to just one side of a trading pair, rather than requiring liquidity for both tokens in a pair.
2. Account-based Design: Instead of using pools for each token pair, the protocol uses accounts to record assets and liabilities, allowing for a more flexible and capital-efficient system.
3. Coverage Ratio: Platypus uses a “coverage ratio” as an input parameter instead of simply focusing on liquidity. The coverage ratio is defined as the assets (A) divided by the liabilities (L) for a given token account. A higher ratio indicates lower default risk. This is a departure from Curve’s stableswap invariant and allows the token pool to grow based on organic demand and supply.
4. Open Liquidity Pool: The protocol is designed to be extensible, allowing new tokens to be added to existing pools easily. For example, starting with a base of USDT, USDC, and DAI, more tokens like TUSD and FRAX can be added later.
5. Price Oracle: Platypus uses external price oracles like Chainlink to track the exchange rate of each token in terms of USD. This is important for maintaining pegs and calculating exchange rates for swaps.
6. Solvency Risk: The protocol aims to keep the coverage ratio above a certain level to mitigate the risk of default. If a withdrawal request exceeds the assets available in a specific token account, that could trigger a default.

Platypus Finance Hack | DETAILED DESCRIPTION

Now that we got a glimpse of the features, in this section we will discuss how prices are calculated.

Platypus uses an Oracle to calculate the ideal prices between assets. When the assets are of the same variety (e.g., wrapped vs. staked versions), such a price Oracle is easily implemented. However, what the makes a big impact to the price in this exploit is the slippage calculation, which can benefit the attacker. The goal of the attacker was to amplify the slippage in their favor, by using a clever trick.

Normally, in this protocol, depositing and withdrawing increases or decreases, respectively, both assets (called cash) and liability in tandem. However, when a withdrawal takes place but there is not enough cash remaining to satisfy the withdrawal, the full liability amount is decreased, despite the asset amount partially decreasing. When this happens, the slippage amount is seemingly manipulated towards the attacker in both directions of a swap.

/**
     * @notice Yellow Paper Def. 2.4 (Asset Slippage)
     * @dev Calculates -Si or -Sj (slippage from and slippage to)
     * @param k K slippage parameter in WAD
     * @param n N slippage parameter
     * @param c1 C1 slippage parameter in WAD
     * @param xThreshold xThreshold slippage parameter in WAD
     * @param cash cash position of asset in WAD
     * @param cashChange cashChange of asset in WAD
     * @param addCash true if we are adding cash, false otherwise
     * @return The result of one-sided asset slippage
     */
    function _slippage(
        uint256 k,
        uint256 n,
        uint256 c1,
        uint256 xThreshold,
        uint256 cash,
        uint256 liability,
        uint256 cashChange,
        bool addCash
    ) internal pure returns (uint256) {
        uint256 covBefore = cash.wdiv(liability);
        uint256 covAfter;
        if (addCash) {
            covAfter = (cash + cashChange).wdiv(liability);
        } else {
            covAfter = (cash - cashChange).wdiv(liability);
        }

        // if cov stays unchanged, slippage is 0
        if (covBefore == covAfter) {
            return 0;
        }

        uint256 slippageBefore = _slippageFunc(k, n, c1, xThreshold, covBefore);
        uint256 slippageAfter = _slippageFunc(k, n, c1, xThreshold, covAfter);

        if (covBefore > covAfter) {
            return (slippageAfter - slippageBefore).wdiv(covBefore - covAfter);
        } else {
            return (slippageBefore - slippageAfter).wdiv(covAfter - covBefore);
        }
    }

    /**
     * @notice Yellow Paper Def. 2.5 (Swapping Slippage). Calculates 1 - (Si - Sj).
     * Uses the formula 1 + (-Si) - (-Sj), with the -Si, -Sj returned from _slippage
     * @dev Adjusted to prevent dealing with underflow of uint256
     * @param si -si slippage parameter in WAD
     * @param sj -sj slippage parameter
     * @return The result of swapping slippage (1 - Si->j)
     */
    function _swappingSlippage(uint256 si, uint256 sj) internal pure returns (uint256) {
        return WAD + si - sj;
    }

Decreasing Liability but Not Asset balance manipulates slippage

LESSONS LEARNED

The more complex a protocol’s financial algorithms are, the more difficult it is to protect from design deficiencies being exploited. Platypus emphasized the ability to maintain a very low slippage and one-sided deposits, which are very hard to implement in a decentralized manner. The Platypus Finance Hack was a difficult one to follow, but there are still ways to improve the security of protocols like these.

One way we could help with similar protocols is by increasing the security posture, in multiple ways. Note that this attacker was relatively smart, and bypassed the mempool when exploiting the protocol, which renders simple mempool scanning techniques ineffective. At the same time the attacks on different pools happened in different transaction. Some vaults could have been paused more quickly had a sophisticated monitoring solution like Watchdog been employed. Finally the protocol’s financial design & calculations are to blame here, employing the services of specialist security firms like ours to conduct financial design audits could have prevented this attack.

The attacker themselves also made a mistake in the attack, and the Platypus team rescued $575 (such funds were transferred to 0x068e297e8ff74115c9e1c4b5b83b700fda5afdeb).

We wish the Platypus team good luck in getting the protocol up and running again, and recovering from this serious incident.

Your project is at an advanced state of engineering and you have decided to hire an auditor to maximize security and legitimacy. Great decision! However, not all audit experiences are equal, so follow this guide to maximize your mileage.

During auditing, you are employing security consultants to go over your code. For auditors, studying your code and issuing an Web3 audit report is a complex balance. Auditors need to have some level of confidence to sign off on your project, yet they will not spend infinite time to gain this confidence. The time allotted has typically been scoped based on your code size and apparent complexity, using extensive past experience.

Unpleasant Truth #1: At the end of an Web3 audit, no professional auditor will ever claim 100% confidence. Sometimes the auditor will explicitly assess that they do not have as high confidence as they would like, but the time they can allocate is simply over!

To enhance the auditor’s confidence and facilitate their work:

1. Provide Succinct, But Sufficient, Documentation: Before auditors start their work, provide them with clear and comprehensive documentation that explains the intent and design of your project. This is not merely about code specifics, but should also provide an understanding of the project at a high level. Of course, when one understands both levels, it should match the specifics.
2. Use Consistent Naming and Comments: Try to use consistent variable and function names to explicitly document the intent of the code. Use comments to document complex parts of the code but also make sure these are consistent with the code. If you can write invariants or explain the intent of complex parts, this can go a long way.
3. Set Up a Communication Channel: Establish a synchronous communication channel between your team and the auditors to facilitate information sharing. If requested, walk the auditors through your code before they start the Web3 audit or when they prefer to (e.g., after they have read the documentation, or after one pass over the code). Be responsive during the audit period.
4. Ensure the Project is Fully Tested and Compiles: By the time an audit begins, make sure the project compiles without errors and is fully tested. This will allow the auditors to focus on the difficult parts of the code, rather than discovering that some functions are uncallable or do not do what they are expected to do under straightforward inputs. Deploy your code on a testnet and exercise it to its limits, focusing on unexpected, corner-case, and possibly adversarial behavior.
5. Understand the Limitations of an Audit: An Web3 audit is not a testing service or a magical way to find all bugs. Auditors will not know your specification (i.e., your mathematical formulas, your desired behavior) if you do not clearly communicate it. Auditing will likely miss violations of functional correctness when the definition of correct calculations is not given.
6. Focus on Adversarial Environmental Conditions: Auditors’ time is best spent thinking about adversarial environmental conditions and not simply uncommon inputs. The latter is a functional correctness issue, best uncovered via testing.

Unpleasant Truth #2: Auditing sometimes misses issues that the programmers themselves find. This is not surprising. The auditor is not the owner of your code. You are!

Auditors will think about all issues to the best of their ability, but, in the end, they are only “involved”, not “committed” to your project. Therefore, you will benefit the most from an audit if you have the right frame of mind. You are the owner. Your project will face an adversarial world. You have the final word, the final command. Thankfully, good people are here to help you. Help them help you. And magnify their help. If an issue is found, ask yourself: how did I miss it? How does it generalize? What other analogous issues may I have missed?

We are looking forward to working with you to best secure your project! Request an audit.

On July 2nd, 2023 06:47:20 PM UTC Poly Network suffered what was initially reported to be a notional $34b hack (the actual realized amounts were far less, due to most of the tokens being illiquid). The Poly team paused their smart contracts EthCrossChainManager on several chains, most notably on Metis, BSC and Ethereum. After our team reconstructed the attack, we concluded that the root cause was not a logical bug on the smart contract, but, most likely, stolen (or misused) private keys of 3 out of 4 of Poly network’s keepers (off-chain systems controlled by the team). In order to understand how the attack took place, we need to understand the architecture of Poly’s cross-chain managers.

Poly runs a network of cross-chain management contracts, allowing tokens to be “transferred” from an origin chain to a destination chain. These contracts accept proofs of token transfer changes on the origin chain, together with encoded arguments for a transaction that withdraws these tokens on the current chain.

function verifyHeaderAndExecuteTx(bytes memory proof, bytes memory rawHeader, bytes memory headerProof, bytes memory curRawHeader,bytes memory headerSig) whenNotPaused public returns (bool)
// proof = Poly chain tx merkle proof
// rawHeader = The header containing crossStateRoot to verify the above tx merkle proof
// headerSig = The converted signature variable for solidity derived from Poly chain's keepers

Main entry point allowing users to “unlock” tokens on the “destination” chain that were “locked” on the origin chain

In Poly, the operation to transfer tokens from the origin chain is referred to as “lock” and the function to retrieve the tokens is referred to “unlock”. Poly employs a system of so-called “consensus nodes” that are essentially EOAs that sign off on the “unlock” event on the destination chain, by including relevant entropy from the origin chain confirming the lock event. This entropy consists of a state root reflecting the locked tokens on the origin chain. Here’s the relevant code which checks that the “header” structure was correctly signed by the “consensus nodes”. The header contains the state root of a Merkle tree. Since the entire header is signed, so is the state root, and, by extension the entire state as witnessed by the Merkle tree.

function verifySig(bytes memory _rawHeader, bytes memory _sigList, address[] memory _keepers, uint _m) internal pure returns (bool){
        // (Dedaub comment)        
        //_rawHeader = 0x0000000000000000000000001e8bb7336ce3a75ea668e10854c6b6c9530dab7...
        //_sigList = // List of 3 signatures from 0x3dFcCB7b8A6972CDE3B695d3C0c032514B0f3825,0x4c46e1f946362547546677Bfa719598385ce56f2,0x51b7529137D34002c4ebd81A2244F0ee7e95B2C0
        //_keepers = ["0x3dFcCB7b8A6972CDE3B695d3C0c032514B0f3825","0x4c46e1f946362547546677Bfa719598385ce56f2","0xF81F676832F6dFEC4A5d0671BD27156425fCEF98","0x51b7529137D34002c4ebd81A2244F0ee7e95B2C0"]
        //_m = 3
        
        bytes32 hash = getHeaderHash(_rawHeader);

        uint sigCount = _sigList.length.div(POLYCHAIN_SIGNATURE_LEN);
        address[] memory signers = new address[](sigCount);

        // (Dedaub comment)
        //   signers = [
        //     0x4c46e1f946362547546677Bfa719598385ce56f2,
        //     0x3dFcCB7b8A6972CDE3B695d3C0c032514B0f3825,
        //     0x51b7529137D34002c4ebd81A2244F0ee7e95B2C0
        // ]
        bytes32 r;
        bytes32 s;
        uint8 v;
        for(uint j = 0; j  < sigCount; j++){
            r = Utils.bytesToBytes32(Utils.slice(_sigList, j*POLYCHAIN_SIGNATURE_LEN, 32));
            s =  Utils.bytesToBytes32(Utils.slice(_sigList, j*POLYCHAIN_SIGNATURE_LEN + 32, 32));
            v =  uint8(_sigList[j*POLYCHAIN_SIGNATURE_LEN + 64]) + 27;
            signers[j] =  ecrecover(sha256(abi.encodePacked(hash)), v, r, s);
            if (signers[j] == address(0)) return false;
        }
        return Utils.containMAddresses(_keepers, signers, _m);
    }

Function to verify signed header, which contains the very-important state root. Comments added by Dedaub.

Our team verified that the code above was correctly invoked and that the header was indeed signed by 3 of the centralized keepers, satisfying the (k-1) out of k keeper signature scheme. We also checked that the list of keepers was not modified prior to the attack. Indeed, over the span of 2 years, the list of keepers remains unchanged and consists of 4 EOAs. It is common for decentralized protocols to employ “keepers”, i.e., external systems controlled by the development team, that feed vital information into the smart contracts. This is sometimes necessary since smart contracts cannot operate autonomously, and need to be invoked externally. What’s less common, however, is to rely on 3 keepers for the end-to-end security in a high TVL cross-chain bridge.

Continuing with our investigation, assuming the attacker did not have control over 3 of the EOAs, the Merkle prover would have been the likely cause of a logical bug in the smart contracts. We therefore looked into this next.

/* @notice                  Verify Poly chain transaction whether exist or not
    *  @param _auditPath        Poly chain merkle proof
    *  @param _root             Poly chain root
    *  @return                  The verified value included in _auditPath
    */
    function merkleProve(bytes memory _auditPath, bytes32 _root) internal pure returns (bytes memory) {
        uint256 off = 0;
        bytes memory value;
        //_auditPath = 0xef20a106246297a2d44f97e78f3f402804011ce360c224ac33b87fe8b6d7b7e618c306000000000000002000000000000000000000000000000000000000000000000000000000000382fc20114c912bcc8ae04b5f5bd386a4bddd8770ae2c3111b7537327c3a369d07179d6142f7ac9436ba4b548f9582af91ca1ef02cd2f1f03020000000000000014250e76987d838a75310c34bf422ea9f1ac4cc90606756e6c6f636b4a14cd1faff6e578fa5cac469d2418c95671ba1a62fe14e0afadad1d93704761c8550f21a53de3468ba5990008f882cc883fe55c3d18000000000000000000000000000000000000000000
        (value, off)  = ZeroCopySource.NextVarBytes(_auditPath, off);

        bytes32 hash = Utils.hashLeaf(value);
        uint size = _auditPath.length.sub(off).div(33);
        bytes32 nodeHash;
        byte pos;
        for (uint i = 0; i < size; i++) {
            (pos, off) = ZeroCopySource.NextByte(_auditPath, off);
            (nodeHash, off) = ZeroCopySource.NextHash(_auditPath, off);
            if (pos == 0x00) {
                hash = Utils.hashChildren(nodeHash, hash);
            } else if (pos == 0x01) {
                hash = Utils.hashChildren(hash, nodeHash);
            } else {
                revert("merkleProve, NextByte for position info failed");
            }
        }
        require(hash == _root, "merkleProve, expect root is not equal actual root");
        return value;
    }

Poly Network Hack | Merkle prover of the Poly chain

The Merkle prover above takes as input a byte sequence (_auditPath) containing the leaf node followed by a path through the Merkle tree that proves the existence of the leaf node, given the state root (_root). Remember, this state root has already been signed by the keepers. In case you’re unfamiliar how Merkle trees work, the picture below depicts a Merkle tree, which is a cryptographically-secure data structure. The key of the algorithm rests on the fact that the root of the Merkle tree contains (transitively) entropy from all the leaf nodes in the tree. Therefore a proof (often called “witness”) can be easily constructed, and cheaply verified. We only need to trust the root of the tree, and if it’s trusted, so is anything else verified by a Merkle tree witness.

The kicker here is that in order to simplify the exploitation scenario, the attacker made full use of the flexibility afforded by the verifier’s implementation. It turns out that the verifier allows for zero-length witnesses. Essentially, the attacker passed in the leaf node, which is exactly 240 bytes in this case, and an empty path as a proof. As it turns out in this case, the hash of the leaf node needs to correspond to the state root (hash) in order for this proof to succeed. This further adds merit to the hypothesis that the Poly chain keepers were likely compromised and signed a state root that turned out to be artificially constructed. The only information within it contained an unlock command that sends tokens to the attacker.

It is unfortunate to note that Poly network had been previously attacked by a greyhat hacker almost two years ago.

Finally, it took Poly network 7 hours to react to today’s attack, and in the meantime the attacker had orchestrated several transactions on multiple chains to exploit this.

Despite the narrative above, there is no definitive proof so far that the keys were stolen. It could have been a rugpull, or it could have been compromised off-chain software running on 3 out of 4 of the keepers. The effect is the same, as far as we can observe. What appears to be unequivocal in the Poly network hack is the fact that a logical bug was not exploited in the smart contracts carrying out the token transfers and that the keepers signed a maliciously-crafted proof. If indeed the Poly network developers confirm the attack has to do with compromised signature keys, as is likely the case, this brings to question the suitability of centralized bridges controlling so much funds. The attack also suggests less-than-perfect monitoring by the Poly network team of the underlying bridge. Had the protocol been set up with a fast monitoring solution, such as Dedaub Watchdog, this would have significantly reduced the reaction time and possibly saved some funds.