Overview

HDP (Herodotus Data Processor) is a modular framework for validating on-chain data from multiple blockchain RPC sources, executing user-defined logic written in Cairo1, and producing an execution trace that can be used to generate a zero-knowledge proof. The proof attests to the correctness of both the on-chain data and the performed computation.

HDP is designed with a two-stage process:

1. Data Verification

  • RPC Data Fetch & Validation:
    HDP connects to blockchain RPC endpoints (e.g., Ethereum, StarkNet) to download raw data along with the necessary on-chain proofs.

  • Verifiers:
    Specialized circuits validate the data by checking inclusion proofs (for example, the Merkle Patricia Trie in Ethereum) and confirming that block headers are authentic and correctly linked in the chain.

  • Memorizers:
    Once verified, the data is stored in an internal dictionary (memorizers) to be readily available for subsequent processing.

2. Computation

  • User-defined Logic:
    Developers write modules in Cairo1, to specify the computation or checks to be performed on the data.

  • Bootloader:
    The bootloader loads the compiled Cairo1 bytecode. It retrieves the necessary data from the memorizers and executes the program.

  • Trace and Proof Generation:
    After execution, a trace is produced. This trace forms the basis for a zero-knowledge proof that attests to the correctness of the on-chain data and the user-defined computation.

This combined approach allows HDP to securely prove statements about historical on-chain events, voting power, balances, and other chain-dependent attributes in a trustless and verifiable manner.

HDP is built to support data from multiple blockchains within a single Cairo1 execution. This capability is especially useful for use cases such as bridging, where data from various chains is verified and processed within one unified pipeline.

hdp diagram

Installation and Setup

  1. Install the CLI:

    curl -fsSL https://raw.githubusercontent.com/HerodotusDev/hdp-cairo/main/install-cli.sh | bash

    To install a specific version:

    VERSION=vX.X.X curl -fsSL https://raw.githubusercontent.com/HerodotusDev/hdp-cairo/main/install-cli.sh | bash

  2. ENVs: HDP CLI requires and RPC endpoint for Ethereum and Starknet. Please expose env vars before running present in .env

Option 2: Building from Source

  • Dependencies:
    • Rust.
    • Python: For Cairo0 compiler.
  • Requirements:
    • Access to blockchain RPC endpoints (e.g., Ethereum, StarkNet).

Installation Steps

  1. Clone the Repository:

    git clone https://github.com/HerodotusDev/hdp-cairo.git
cd hdp-cairo

  2. Set Up the Environment:

    • Install the cairo0 toolchain.

      make

  3. Configuration:

    • Copy the example environment file:

      cp example.env .env

    • Edit the .env file to provide the correct RPC endpoints and configuration details.

Running an Example Module

Overview

This guide demonstrates how to run an example Cairo1 module that verifies StarkGate solvency by comparing token supplies on Ethereum (L1) and StarkNet (L2).

Example Module: StarkGate Solvency Check

The following code compares the total token supply on both chains:

#[starknet::contract]
mod example_starkgate {
    use hdp_cairo::HDP;
    use hdp_cairo::evm::{account::{AccountKey, AccountImpl}, ETHEREUM_TESTNET_CHAIN_ID};
    use hdp_cairo::starknet::{storage::{StorageKey, StorageImpl}, STARKNET_TESTNET_CHAIN_ID};

    #[storage]
    struct Storage {}

    #[external(v0)]
    pub fn main(ref self: ContractState, hdp: HDP) -> u128 {
        // Define the L1 Ethereum bridge account key.
        // More details: https://github.com/starknet-io/starknet-addresses/blob/master/bridged_tokens/sepolia.json#L2-L10
        let starkgate_evm_account_key = AccountKey {
            chain_id: ETHEREUM_TESTNET_CHAIN_ID,
            block_number: 7692344,
            address: 0x8453FC6Cd1bCfE8D4dFC069C400B433054d47bDc, // l1_bridge_address
        };

        // Define the L2 StarkNet token storage key (ERC20 total supply).
        let starkgate_starknet_storage_key = StorageKey {
            chain_id: STARKNET_TESTNET_CHAIN_ID,
            block_number: 517902,
            address: 0x049d36570d4e46f48e99674bd3fcc84644ddd6b96f7c741b1562b82f9e004dc7, // l2_token_address
            storage_slot: 0x0110e2f729c9c2b988559994a3daccd838cf52faf88e18101373e67dd061455a, // ERC20 totalSupply slot
        };

        // Retrieve the Ethereum balance for the L1 bridge account.
        let starkgate_balance_ethereum: u256 = hdp.evm.account_get_balance(starkgate_evm_account_key);

        // Ensure the balance is within 128 bits.
        assert!(starkgate_balance_ethereum.high == 0x0);

        // Retrieve the StarkNet token total supply.
        let starkgate_balance_starknet: u128 = hdp
            .starknet
            .storage_get_slot(starkgate_starknet_storage_key)
            .try_into()
            .unwrap();

        // Define an acceptable accuracy range (0.1% of the Ethereum balance).
        let starkgate_balance_ethereum_accuracy: u128 = starkgate_balance_ethereum.low / 1000;

        // Validate that the StarkNet balance is within the acceptable range of the Ethereum balance.
        assert!(
            starkgate_balance_ethereum.low + starkgate_balance_ethereum_accuracy > starkgate_balance_starknet,
        );
        assert!(
            starkgate_balance_ethereum.low - starkgate_balance_ethereum_accuracy < starkgate_balance_starknet,
        );

        println!("StarkGate balance {}", starkgate_balance_ethereum.low);

        // Return the Ethereum balance (low part).
        starkgate_balance_ethereum.low
    }
}

Running the Pipeline

You can run the pipeline either using the CLI tool or building from source.

Build the Cairo1 Modules

scarb build

Using CLI Tool

  1. Dry Run Process:
  • Purpose:
    Identify the required on-chain data and proofs.
  • Command: 
    hdp-cli dry-run -m target/dev/example_starkgate_module.compiled_contract_class.json --print_output
  1. Fetcher Process:
  • Purpose:
    Connect to blockchain RPC endpoints to fetch on-chain data and corresponding proofs, using the keys identified during the dry run.
  • Command: 
    hdp-cli fetch-proofs
  1. Sound Run Process:
  • Purpose:
    Execute the compiled Cairo1 bytecode with the verified data. During this process, the bootloader retrieves data, handles system calls, and runs user logic, generating an execution trace.
  • Command: 
    hdp-cli sound-run -m target/dev/example_starkgate_module.compiled_contract_class.json --print_output

Using Source Build

  1. Dry Run Process:
  • Purpose:
    Identify the required on-chain data and proofs.
  • Command: 
    cargo run --release --bin hdp-cli -- dry-run -m target/dev/example_starkgate_module.compiled_contract_class.json --print_output
  1. Fetcher Process:
  • Purpose:
    Connect to blockchain RPC endpoints to fetch on-chain data and corresponding proofs, using the keys identified during the dry run.
  • Command: 
    cargo run --release --bin hdp-cli --features progress_bars -- fetch-proofs
  1. Sound Run Process:
  • Purpose:
    Execute the compiled Cairo1 bytecode with the verified data. During this process, the bootloader retrieves data, handles system calls, and runs user logic, generating an execution trace.
  • Command: 
    cargo run --release --bin hdp-cli -- sound-run -m target/dev/example_starkgate_module.compiled_contract_class.json --print_output

Testing

  1. Build the Cairo1 Modules:

    scarb build

  2. Run Tests with Nextest:

    cargo nextest run

Note: Ensure that the environment variables from .env are set before running the tests.

Architecture

The HDP system is composed of several interlocking parts, each responsible for a specific piece of the data processing pipeline.

1. Verifiers

  • Function:
    Validate on-chain data by checking the correctness of inclusion proofs (e.g., Merkle Patricia Trie for Ethereum) and confirming the presence of block headers.

  • Chain-Specific Implementation:
    Different verifiers exist for each supported blockchain (e.g., Ethereum, StarkNet) and the system is designed to be extended to support additional chains.

2. Memorizers

  • Function:
    Act as internal dictionaries to store verified and decoded data for later retrieval by the execution engine.

  • Usage:
    The bootloader retrieves data from these memorizers to ensure that the user-defined logic is executed on validated inputs.

3. Decoders

  • Function:
    Convert raw RPC data into formats that are usable by the user logic. This includes processes like RLP decoding or field element conversion.

  • Output:
    The decoded results are stored in the memorizers, ensuring quick and secure access during the execution phase.

4. Bootloader

  • Function:
    Serves as the execution engine that loads the user-defined Cairo1 bytecode, runs the logic, and manages the data flow from the memorizers.

  • Workflow:

    • Initialization: Runs after the verifiers complete data validation and the memorizers are populated.
    • Execution: Runs the user’s logic using the verified data.
    • Syscall Handling: Invokes system calls (e.g., cryptographic hash functions provided by Cairo0) as required.
    • Final Checks: Validates the outputs of syscalls and ensures the integrity of the execution trace.

After execution, the produced trace can be used in a zero-knowledge proving pipeline. The resulting proof attests that the data used was valid and that the user-defined logic was executed correctly.

hdp diagram

Debugging Guide

Effective debugging is key to rapidly identifying and resolving issues while writing your custom Cairo module.

Printing Debug Messages

To help with debugging, you can use the println! macro to print debug messages to the console.

To disable the validation pass while developing and using that macro, you can add the following to your Scarb.toml:

[[target.starknet-contract]]
allowed-libfuncs-deny = true

#[starknet::contract]
mod contract {
    use hdp_cairo::{HDP};

    #[storage]
    struct Storage {}

    #[external(v0)]
    pub fn main(
        ref self: ContractState,
        hdp: HDP
    ) {
        println!("Hello, world!");
    }
}

Apply these practices to streamline your debugging workflow and quickly trace bugs.

Happy debugging!