Most Dapp developers have heard of and probably use the excellent Multicall contract to bundle their eth_calls and reduce latency for bulk ETL in their applications (we do too, we even have a python library for it: Manifold).

Unfortunately, we cannot use this same trick when getting storage slots, as we discovered when developing our storage explorer, forcing developers to issue an eth_getStorageAt for each slot they want to query. Luckily, Geth has a trick up its sleeve, the “State Override Set”, which, with a little ingenuity, we can leverage to get bulk storage extraction.

Bulk Storage Extraction | Geth Trickery

The “state-override set” parameter of Geth’s eth_call implementation is a powerful but not very well-known feature. (The feature is also present in other Geth-based nodes, which form the base infrastructure for most EVM chains!) This feature enables transaction simulation over a modified blockchain state without any need for a local fork or other machinery!

Using this, we can change the balance or nonce for any address, as well set the storage or the code for any contract. The latter modification is the important one here, as it allows us to replace the code at an address we want to query the storage for with our own contract that implements arbitrary storage lookups.

Here is the detailed structure of state-override set entries:

FIELD	TYPE	BYTES	OPTIONAL	DESCRIPTION
balance	Quantity	<32	Yes	Fake balance to set for the account before executing the call.
nonce	Quantity	<8	Yes	Fake nonce to set for the account before executing the call.
code	Binary	any	Yes	Fake EVM bytecode to inject into the account before executing the call.
state	Object	any	Yes	Fake key-value mapping to override all slots in the account storage before executing the call.
stateDiff	Object	any	Yes	Fake key-value mapping to override individual slots in the account storage before executing the call.

Bulk Storage Extraction | Contract Optimizoor

To better understand what’s going on we can take a look at the high level code (this was actually generated by our decompiler)

[00] PUSH0              # [0], initial loop counter is 0  
[01] JUMPDEST  
[02] DUP1               # [loop_counter, loop_counter]  
[03] CALLDATASIZE       # [size, loop_counter, loop_counter]
[04] EQ                 # [bool, loop_counter]  
[05] PUSH1 0x13         # [0x13, bool, loop_counter]  
[07] JUMPI              # [loop_counter]  
[08] DUP1               # [loop_counter, loop_counter]  
[09] CALLDATALOAD       # [, loop_counter]  
[0a] SLOAD              # [, loop_counter]  
[0b] DUP2               # [loop_counter, , loop_counter]  
[0c] MSTORE             # [loop_counter]  
[0d] PUSH1 0x20         # [0x20, loop_counter]  
[0f] ADD                # [loop_counter] we added 32 to it, to move 1 word  
[10] PUSH1 0x1          # [0x1, loop_counter]  
[12] JUMP               # [loop_counter]  
[13] JUMPDEST  
[14] CALLDATASIZE       # [size]  
[15] PUSH0              # [0, size]  
[16] RETURN             # []

The following handwritten smart contract has been optimized to maximize the number of storage slots we can read in a given transaction. Before diving into the results I’d like to take an aside to dive into this contract as it’s a good example of an optimized single-use contract, with some clever (or at least we think so) shortcuts.

function function_selector() public payable {

    v0 = v1 = 0;

    while (msg.data.length != v0) {
        MEM[v0] = STORAGE[msg.data[v0]];
        v0 += 32;
    }

    return MEM[0: msg.data.length];
}

Walking through the code we can see that we loop through the calldata, reading each word, looking up the corresponding storage location, and writing the result into memory.

The main optimizations are:

• removing the need for a dispatch function
• re-using the loop counter to track the memory position for writing results
• removing abi-encoding by assuming that the input is a contiguous array of words (32-byte elements) and using calldatalength to calculate the number of elements

If you think you can write a shorter or more optimized bytecode please submit a PR to storage-extractor and @ us on twitter.

Bulk Storage Extraction | Results

THEORETICAL RESULTS

To calculate the maximum number of storage slots we can extract we need three equations, the execution cost (calculated as the constant cost plus the cost per iteration), the memory expansion cost $$(3x+(x^2/512))$$ and the calldata cost.

We can break down the cost of the execution as follows:

• The start, the range check and the exit will always run at least once
• Each storage location will result in 1 range check and 1 lookup

Calculating the calldata cost is slightly more complex as its variably-priced: empty (zero-byte) calldata is priced at 4 gas per byte and non-zero calldata is priced at 16 gas. Therefore we need to calculate a placeholder for the average price of a word (32-bytes).

zero_byte_gas = 4
non_zero_byte_gas = 16

# We calculate this as the probability each bit of a byte is a 0
prob_rand_byte_is_zero = (0.5**8) # 0.00390625
prob_rand_byte_non_zero = 1 - prob_rand_byte_is_zero # 0.99609375

avg_cost_byte = (non_zero_byte_gas * prob_rand_byte_non_zero) + \
				(zero_byte_gas * prob_rand_byte_is_zero) # (16 * 0.99609375) + (04 * .00390625) = 15.953125

Therefore the average word costs: $$15.953125 * 32 * x$$

We can combine all of these equations and solve for the gas limit to get the maximum number of storage slots that can be read in one call.

Therefore given a 50 million gas limit (which is the default for Geth) we can read an average of 18514 slots.

This number will change based on the actual storage slots being accessed, with most users being able to access more. This is due to the fact that most storage variables are in the initial slots of the contract, with only mapping and dynamic arrays being pushed to random slots (or people using advanced storage layouts such as those used in Diamond proxies).

PRACTICAL RESULTS

To show the impact of this approach, we wrote a python script which queries a number of storage slots, first using normal RPC requests and batched RPC requests for the normal eth_getStorageAt, and then comparing to the optimized eth_call with state-override set. All the testing code can be found in the storage-extractor repo, along with the bytecode and results.

To isolate variable latency as a concern, we ran the tests on the same machine as our node, with latency being re-added by utilizing asyncio.sleep to have a controlled testing environment. To properly understand the results, lets look at the best-case scenario of 200 concurrent connections.

In order to properly represent the three methods we need to set the y-axis to be logarithmic since standard parallel `eth_getStorageAt`s are too slow. As you can see even with 200 connections standard RPC calls are 57 times slower than RPC batching and 103 times slower than `eth_call` with state-override.

We can take a closer look at the difference between batching and call overrides in the next graph. As you can see, call overrides are faster in all scenarios since they require fewer connections, this is most noticeable with the graph in the top left which highlights the impact of latency on the overall duration.

Conclusion

To wrap up this Dedaub blog post, I’d like to thank the Geth developers for all the hard work they’ve been doing, and the extra thought they put into their RPC to enable us to do funky stuff like this to maximize the performance of our applications.

If you have a cool use of the state-override set please tweet us, and, if you’d like to collaborate, you can submit a PR on the accompanying github repo (storage-extractor).

The Arbitrum network experienced significant downtime on December 15 due to problems with its sequencer and feed. The network had been down for almost three hours. The major outage began at 10:29 a.m. ET amid a substantial increase in a type of network traffic called Inscriptions. Arbitrum’s layer-2 network had processed over 22.29 million transactions and had a total value locked of $2.3 billion. Despite the success of the network, the current design suffers from a significant chokepoint when posting transactions to L1, causing the sequener to stall. While advancements such as Arbitrum Nova and Proto-danksharding might alleviate these design issues, this is not the first time Arbitrum has experienced such issues – a bug in the sequencer also halted the network in June 2023.

Arbitrum Sequencer Outage | Background

Arbitrum is a Layer-2 (L2) solution which settles transactions off the Ethereum mainnet. L2s provide lower gas fees and reduce congestion on the primary blockchain (In this case, Ethereum, L1). The current incarnation of Arbitrum is called Nitro. Arbitrum Nitro processes transactions in two stages: sequencing, where transactions are ordered and committed to this sequence, and deterministic execution, where each transaction undergoes a state transition function. Nitro combines Ethereum emulation software with extensions for cross-chain functionalities and uses an optimistic rollup protocol based on interactive fraud proofs. The Sequencer is a key component in the Nitro architecture. Its primary role is to order incoming transactions honestly, typically following a first-come, first-served policy. This is a centralized component operated by Offchain Labs. The Sequencer publishes its transaction order both as a real-time feed and to Ethereum, in the calldata of an “Inbox” smart contract. This publication ensures the final and authoritative transaction ordering. Additionally, a Delayed Inbox mechanism exists for L1 Ethereum contracts to submit transactions and as a backup for direct submission in case of Sequencer failure or censorship.

Arbitrum Sequencer Outage | Root cause

In the two hours prior to the outage more than 90% of Arbitrum traffic consisted of Ethscriptions. Ethscriptions are digital artifacts on EVM chains created using Ethereum calldata. Unlike traditional NFTs managed by smart contracts, Ethscriptions make the blockchain data itself a unique NFT. They are inspired by Bitcoin inscriptions (Ordinals) but function differently. Creating an Ethscription involves selecting an image, converting it to data URI format, then to hexadecimal format, and finally embedding it into a 0 ETH transaction’s Hex data field. Each Ethscription must be unique; duplicate data submissions are ignored. Owners can use Ethscriptions IDs for proof or transfer of ownership. In practice the calldata or Ethscriptions look like the code below:

data: {"p":"fair-20","op":"mint","tick":"fair","amt":"1000"}

Calldata example of an Ethscription. This represents a token mint.

Since Ethscriptions are very cheap, one can do a lot of them for the same unit of cost. Indeed, a staggering 90% of transactions posted on-chain were Ethscriptions. Also, for a relatively low cost, the amount of transaction entropy that needed to be committed to L1 increased to 80MB/hr vs. the 3MB/hr that was typical before the traffic spike. We calculated this by looking at average on-chain transaction postings for the sequencer.

Now, look at the architecture diagram of Arbitrum below. Note that in order to commit transaction sequences to L1, the data poster needs to post the increased amount of data over a larger number of transactions. Prior to the outage, the number of transactions posted per hour was around 10 – 20x higher than the December mean.

However, the code responsible for posting these transactions has an in-built limitation that imposes limits to the rate at which L1 batches are posted. Prior to the outage, if there are 10 batches still in the L1 mempool, no more batches are sent to L1, stalling the sequencer. This limit was subsequently raised to 20 batches after the outage. This is probably not a good long-term solution however, as it increases the chances of batches needing to be reposted due to transaction nonce issues.

// Check that posting a new transaction won't exceed maximum pending
// transactions in mempool.
if cfg.MaxMempoolTransactions > 0 {
  unconfirmedNonce, err := p.client.NonceAt(ctx, p.Sender(), nil)
  if err != nil {
    return fmt.Errorf("getting nonce of a dataposter sender: %w", err)
  }
  if nextNonce >= cfg.MaxMempoolTransactions+unconfirmedNonce {
    return fmt.Errorf(
      "... transaction with nonce: %d will exceed max mempool size ...",
      nextNonce, cfg.MaxMempoolTransactions, unconfirmedNonce
    )
  }
}
return nil

Batch poster is responsible for posting the sequenced transaction sequence as Ethereum calldata.

Arbitrum Sequencer Outage | Recommendations

There are several indications that point towards the sequencer, and thus the network, not being tested enough in a realistic setting or in an adversarial environment. However, luckily the upcoming Proto-Danksharding upgrade to Ethereum should also help for reducing L1-induced congestion. Irrespective of this the Arbitrum engineers can consider the following recommendations:

• Whether the Arbitrum gas price of L2 calldata is set too low, compared to other kinds of operations. Gas is an anti-DoS mechanism, which is intimately tied to the L1 characteristics. If this increase in L2 calldata causes a proportionally large increase in batch size, then attackers can craft L2 transactions with large calldatas that result in batches that don’t compress well under Brotli compression, causing a DoS attack on the sequencer. Note that Arbitrum Nova should not suffer as much from this issue as the transaction data is not stored on L1, only a hash is.
• Whether there is a tight feedback loop between the size of the L1 batches currently in the mempool and L2 gas price. There is an indirect feedback loop, via the gas price on L1 and backlog sizes, but this may not be too tight. In addition, since the sequencer is centralized anyway, anti-DoS measures might be encoded directly into it to reject transactions. (Note: A more decentralized sequencer is being considered for the future, so this last measure wouldn’t work)
• Long-term, the engineers more research into making the rollups more efficient to decrease the sizes of batches committed to L1. This may include ZKP rollups at some point.
• Additionally, security audits to the sequencer should consider DoS situations, both through simulation/fuzzing and also by having auditors think of hostile situations through adversarial thinking based off their deep knowledge of the involved chains.

Finally, the Arbitrum team made a small change to the way transactions are soft-committed. In this change the feed backlog is populated irrespective of whether the sequencer coordinator is running, which carries its own risks but enables dApps running on Arbitrum to be more responsive during certain periods.

Disclaimer: The Arbitrum sequencer is solely operated by Offchain labs. Thus, most of the information regarding its operational issues (such as logs) are not publicly available so it’s hard to get a complete picture of the issue. Dedaub has not audited Arbitrum or Offchain labs software. Dedaub has however audited other (non-Arbitrum) software and projects running on Arbitrum such as GMX, Chainlink, Rysk & Stella.

Hello everyone, this is Yannis Bollanos, Security Researcher at Dedaub. A few days ago, we published a tweet about the thestandard.io exploit that took place on November 6th, 2023, which you can find here: https://twitter.com/dedaub/status/1734598398055981471.

The positive response from the X audience indicates a strong interest in the topic. As a result, I have decided to expand it into a blog post that can be used as a reference in the future.

Thestandard.io exploit occurred on November 6th, 2023, and according to Crypto.news, approximately 280K EUROs were at risk. Fortunately, most of the funds have been recovered, so this is a hack story with a happy ending.

After the excitement and tension of the moment subside, it is important to reflect on what happened and how we can prevent similar attacks in the future. It’s a great opportunity to re-emphasize that protocols should use defensive checks/assertions at every point their code interacts with a decentralized exchange (DEX).

The @thestandard.io protocol issues coins to users who open over-collateralized positions, helping the protocol’s assets maintain a stable value by adjusting liquidity provision by actual market rates.

In the @thestandard.io attack scenario, a SmartVault contract oversees the management of each user’s position, taking responsibility for adequately verifying the position’s liquidity. Users can issue coins by calling `mint`:

The SmartVault allows the exchanging of deposited collateral tokens through Uniswap’s V3 router (0xe592427a0aece92de3edee1f18e0157c05861564 on Arbitrum). Here is where things get interesting:

SmartVaultV2 – Arbitrum – Source code (0x2E9f9Cc46679DBb5D94a1397Bd922cA5F6dA99Cd) a smart contract deployed on Arbitrum.

One may inspect the source code in the (Dedaub) Contract Library for SmartVaultV2 – 0x2E9f9Cc46679DBb5D94a1397Bd922cA5F6dA99Cd. Below is the screenshot.

The things to note are:

• With amountOutMinimum set to 0, the swap operation would succeed no matter the extent of the slippage incurred.
• There were no other safeguards in place to ensure a fair exchange for the value provided in the contract.

This enabled the owner of the vault contract to initiate a swap on a pool that might have been maliciously ’tilted,’ allowing for an exchange at an arbitrarily different price from the market price.

There are two ways to profit from this:

• (1) Utilize a flash loan and purposely sandwich the swap operation between a tilting and an un-tiling swap on the pool. This is a fairly typical attack pattern commonly used in exploits.

OR

• (2) Have the swap operation occur on a pool, the liquidity of which (as well as the execution price) is entirely controlled by the attacker. This can be done only on freshly created pools or in pools with near-0 liquidity.

The attacker chose option (2) since a Uniswap V3 pool for PAXG-WBTC didn’t exist then. Here’s how everything is put together to form the attack:

Attack Transaction:

1. The attacker creates the Uniswap v3 PAXG-WBTC pool
2. The attacker flash borrows 10 WBTC ( and a tiny extra amount to provide as initial liquidity)
3. The attacker provides 10 WBTC as collateral and mints as many EUROs as possible

One may inspect the relevant transaction in the (Dedaub) Contract Library for 0x51293c1155a1d33d8fc9389721362044c3a67e0ac732b3a6ec7661d47b03df9f – Arbitrum. Below is the screenshot.

The attacker provides liquidity to the PAXG/WBTC pool. WBTC and PAXG are at a 1:1 ratio within the tick range in which liquidity is minted. This is over-valuing PAXG by a lot.

The attacker swaps the deposited WBTC for PAXG, and the swap operation goes through the attacker-controlled pool. The vault is now under-collateralized, in terms of real value: the PAXG it obtained has much less value than the EUROs issued.

The attacker then burns all of his liquidity on Uniswap, and he notably receives ~9.9 WBTC. At this point, the attacker still holds the originally minted EUROs.

The attacker swaps 10k of his EUROs for USDCs. Some USDCs are then employed to obtain the few remaining WBTCs needed to repay the flash loan.

In the end, the attacker walks away with 280k EUROs and ~8.5k USDC.

Fortunately, the attacker has returned ~240k EUROs back to the protocol:

One may inspect the relevant transaction in the (Dedaub) Contract Library for 0xb08633c44d5f7c6fc10ad5685642c54e97900165bd1d64a1d003c99d1eec9a4b – Arbitrum. Below is the screenshot.

Thestandard.io Exploit | Key Learning

Smart Contract developers should not solely rely on assumptions about on-chain liquidity/asset prices. The code should consistently enforce these assumptions (within a reasonable deviation).

Introduction to Transaction Simulation Solutions

Transaction simulation tools improve developer and user experience when operating decentralized Web3 applications (Smart Contracts running on programmable blockchains).

These tools can lower the risk and guesswork during development, deployment, and subsequent operation of Web3 applications. And they’re particularly useful in hostile security environments such as public blockchains.

Transaction simulation tools allow developers and users to “dry-run” the execution of transactions on the blockchain without committing the state changes of this transaction to the ledger.

For example, an end user can deposit funds in a yield farming vault and understand what proportion of the vault the deposit would be entitled to.

Another example is the simulation of a decentralized autonomous organization (DAO) proposal to evaluate its integrity and functionality, ensuring it’s not malicious before implementation.

In this article, we will explore the user experience and security issues that users and developers face when interacting with Web3 applications and how transaction simulation tools can help mitigate them.

By the end of this article, you’ll better understand what transaction simulation tools do, how they work, and how they can improve both user and developer experience.

The Need for Transaction Simulation Solutions in Blockchain

Web3 applications, such as DeFi applications, enable novel financial primitives with many more possibilities for end users. However, the complexity and irreversibility of blockchain transactions have led to unexpected fund losses for many users, often due to poorly designed interfaces in these applications.

Loss of funds is not the only issue for Web3 applications. We often face reverted or out-of-gas transactions, wasting funds, which is especially detrimental to our experience when interacting with Web3 applications.

The impact of these challenges is not limited to regular end-users. Developers and Web3 teams face the complex task of ensuring their contracts perform as intended.

Interacting with a blockchain protocol in a complex manner, for instance, through a multisig account, is a highly daunting task. Typically, it can be accomplished by forking the blockchain, but this is time-consuming.

Real-world scenarios underscore how critical transaction simulation solutions are. For instance, in platforms Yearn Finance or Uniswap Labs, where complex financial transactions are constant, the necessity to simulate transactions is invaluable.

In these cases, simulations allow users to review the outcomes of Smart Contract transactions in a controlled environment, giving teams time to identify and address potential issues before running them on-chain.

Types of Transaction Simulation Solutions Available

The market offers a variety of transaction simulation solutions, each catering to different needs and preferences.

Browser Extensions are popular for their ease of use, integrating with web browsers to offer simulation capabilities alongside wallet interactions.

In-Wallet Simulations integrate with the wallet software, providing a seamless experience for users to simulate transactions within the wallet interface.

Standalone Tools are comprehensive software solutions. These offer advanced features and greater flexibility for complex simulations. Developers and organizations needing detailed analyses and custom simulation scenarios prefer standalone tools.

Advantages of Using Transaction Simulation Tools

ERROR PREVENTION

Error prevention is a crucial advantage of transaction simulation tools, as they enable developers to simulate transactions in a controlled environment.

This process helps identify and correct errors before executing them on the blockchain, significantly reducing the likelihood of costly mistakes such as failed transactions that consume resources without achieving their intended outcomes.

Consequently, these tools greatly enhance blockchain applications’ overall reliability and efficiency.

EDUCATIONAL VALUE

For newcomers to blockchain development, transaction simulation solutions are invaluable educational resources. They provide a hands-on, risk-free platform for understanding the intricacies of blockchain transactions.

They allow developers to experiment with different scenarios, gaining practical insights into the operation of Smart Contracts. This experiential learning accelerates any developer’s expertise in blockchain technology, empowering them to build more sophisticated and secure dApps.

Choosing the Right Transaction Simulation Solution

Selecting an appropriate transaction simulation solution is crucial for blockchain developers. These tools come in various forms, each suited to different needs and environments.

Factors to Consider:

• Network Support: Ensure the tool supports all relevant blockchain networks your project interacts with. For instance, if your Smart Contract runs on Ethereum and Polygon, the chosen transaction simulation solution must accommodate both.
• Ease of Integration: Assess how seamlessly the tool integrates into your existing development. A smooth integration minimizes disruptions and maintains development flow.
• User Experience: Assess the tool’s user interface and usability. A good simulator should offer clear insights into the transaction process, aiding decision-making and error identification.
• Type of Tool: Decide between browser extensions and wallet-based simulators. Browser extensions are generally more flexible and accessible to test across various wallets, whereas wallet-based solutions offer a more integrated experience.

EVALUATION CRITERIA:

• Reliability and Support: Investigate the tool’s performance history and the provider’s responsiveness to support queries and updates.
• Track Record: Consider the provider’s reputation within the blockchain community. Long-standing, positively reviewed tools often indicate reliability and efficacy.

RECOMMENDATIONS:

• Opt for solutions that prioritize security and accuracy in transaction simulation.
• Avoid tools that are overly complex or do not offer transparent processes, as these can hinder rather than help your development efforts.
• Stay informed about the latest developments in transaction simulation technologies to ensure your choice remains relevant and effective.

Selecting the right tool is crucial. It must meet technical requirements and adhere to the highest security and efficiency standards in the blockchain space.

Dedaub Watchdog Transaction Simulator

The Dedaub Watchdog Transaction Simulator allows users to simulate transactions when interacting with complex Smart Contracts before committing to the main chain.

It allows an understanding of all the various actions that would happen without the risk of losing funds. The Dedaub Watchdog transaction simulation provides three approaches, depending on specific use cases:

• Through the Dedaub Simulation API, developers can integrate simulation directly into their applications.
• Through the read/write/simulate feature on any Smart Contract page in Watchdog.
• By installing the Dedaub TX Simulator Snap in Metamask.

When used by an end-user, such as in the latter two approaches, the transaction simulation presents relevant information in convenient formats through the Watchdog UI.

One such format is the (i) trace format, which contains all intermediate Smart Contract functions called, new Smart Contracts created, and events fired.

The other format contains fund transfer, and (ii) includes the amount of funds transferred, both for the user and other participants in the transaction.

(Trace format above)

(funds transferred above)

When used by Web3 users, an important use case is checking the legitimacy and reliability of the accounts and Smart Contracts involved in the transaction. By simulating transactions, users can also gain insight into potential outcomes, allowing them to identify risks proactively.

The Dedaub Watchdog Transaction Simulator leverages the Dedaub Smart Contract database. The database offers detailed, real-time information on all deployed Smart Contracts on-chain, providing deep insights into the workings of Smart Contracts.

Conclusion

In conclusion, transaction simulation tools, particularly those exemplified by the Dedaub Watchdog Transaction Simulator, represent an advancement in Web3 application development and user interaction. They provide an extra layer of security and insight, allowing developers and end-users to identify and rectify potential issues in Smart Contract transactions promptly. These tools prevent costly errors and fund losses and serve as educational resources for those new to blockchain technology. With their ability to simulate complex financial transactions in a controlled environment, transaction simulation solutions enhance the efficiency, reliability, and overall user experience of interacting with Web3 applications.