Introduction

Welcome to The Dash Platform Book. This is a guide to the design philosophy, architectural patterns, and engineering conventions that shape the Dash Platform Rust codebase. It is not a user manual or an API reference -- it is the book you read before you open your editor, so that the thousands of files in the monorepo start to make sense.

Who This Book Is For

  • Contributors who want to add a new state transition, query endpoint, or Drive operation and need to know where things go and why.
  • Auditors reviewing the codebase for correctness, who need a map of the security-critical paths (validation pipelines, fee calculations, proof verification).
  • Architects evaluating the system design, particularly the versioning strategy, the ABCI integration, and the GroveDB storage model.

If you have written Rust before and understand traits, enums, and feature flags, you are ready.

The Design Philosophy

Four principles recur throughout the codebase. Understanding them up front will save you hours of "why is it done this way?" confusion.

1. Version Everything

Every method that touches consensus has a version number. The version is not embedded in the code path -- it lives in a central PlatformVersion struct that is threaded through the entire call stack:

#![allow(unused)]
fn main() {
// From packages/rs-platform-version/src/version/protocol_version.rs
#[derive(Clone, Debug)]
pub struct PlatformVersion {
    pub protocol_version: ProtocolVersion,
    pub dpp: DPPVersion,
    pub drive: DriveVersion,
    pub drive_abci: DriveAbciVersion,
    pub consensus: ConsensusVersions,
    pub fee_version: FeeVersion,
    pub system_data_contracts: SystemDataContractVersions,
    pub system_limits: SystemLimits,
}
}

When you see a method that dispatches on a version number, this is the canonical pattern:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/execution/engine/finalize_block_proposal/mod.rs
pub(crate) fn finalize_block_proposal(
    &self,
    request_finalize_block: FinalizeBlockCleanedRequest,
    block_execution_context: BlockExecutionContext,
    transaction: &Transaction,
    platform_version: &PlatformVersion,
) -> Result<block_execution_outcome::v0::BlockFinalizationOutcome, Error> {
    match platform_version
        .drive_abci
        .methods
        .engine
        .finalize_block_proposal
    {
        0 => self.finalize_block_proposal_v0(
            request_finalize_block,
            block_execution_context,
            transaction,
            platform_version,
        ),
        version => Err(Error::Execution(ExecutionError::UnknownVersionMismatch {
            method: "finalize_block_proposal".to_string(),
            known_versions: vec![0],
            received: version,
        })),
    }
}
}

This pattern allows the network to upgrade without hard forks. Every node on the network agrees on which version to run for each method at each block height, and the version table is the single source of truth. The Versioning section covers this in depth.

2. Explicit Costs

Platform is a metered system. Every operation -- inserting a document, updating a contract, creating an identity -- has a fee expressed in credits. Costs are not estimated after the fact; they are tracked as operations execute, through GroveDB's cost accounting layer. The FeeResult type flows alongside the data, and sum trees in GroveDB let the platform verify fee totals against the Merkle-proven state.

The fee version itself is part of PlatformVersion, which means the network can adjust fee schedules across protocol upgrades without any node disagreeing on how much an operation costs at a given block height.

3. Clear Transformation Stages

A state transition does not go from "bytes on the wire" to "committed to disk" in one step. It passes through well-defined stages:

  1. Decode -- raw bytes become a StateTransition enum.
  2. Structure validation -- syntactic checks (field lengths, required fields).
  3. State validation -- checks against current platform state (does this identity exist? does the nonce match?).
  4. Transform into action -- the validated transition becomes a StateTransitionAction, a representation of what to do rather than what was requested.
  5. Convert to operations -- the action becomes a list of DriveOperation values (GroveDB inserts, deletes, replacements).
  6. Apply -- the operations execute inside a GroveDB transaction.

Each stage is a separate module with its own versioned dispatch. This separation means you can audit validation independently from execution, and you can test each stage in isolation.

4. Trait-Based Polymorphism

The codebase avoids dynamic dispatch where performance matters and embraces it where extensibility matters. ABCI handlers are defined against traits like PlatformApplication, TransactionalApplication, and BlockExecutionApplication:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/abci/app/mod.rs
pub trait PlatformApplication<C = DefaultCoreRPC> {
    fn platform(&self) -> &Platform<C>;
}

pub trait TransactionalApplication<'a> {
    fn start_transaction(&self);
    fn transaction(&self) -> &RwLock<Option<Transaction<'a>>>;
    fn commit_transaction(&self, platform_version: &PlatformVersion)
        -> Result<(), Error>;
}

pub trait BlockExecutionApplication {
    fn block_execution_context(&self) -> &RwLock<Option<BlockExecutionContext>>;
    fn unsigned_withdrawal_txs_by_round(&self) -> &RwLock<UnsignedWithdrawalTxsByRound>;
}
}

This makes it possible to swap in mock implementations for testing while keeping the production path zero-cost.

How This Book Is Organized

The book follows the data as it flows through the system, from the outside in:

SectionWhat You Will Learn
ArchitectureThe monorepo layout, crate responsibilities, and the request pipeline from client to GroveDB.
VersioningHow PlatformVersion controls every consensus-critical code path, and how upgrades propagate.
State TransitionsThe lifecycle of a state transition: validation, transformation, operation generation, and application.
Error HandlingThe split between consensus errors (returned to users) and execution errors (node-level panics).
SerializationThe platform-serialization crate and its derive macros for versioned binary encoding.
Data ModelData contracts, documents, identities, and tokens as Rust types.
DriveGroveDB operations, batch processing, cost tracking, and finalize tasks.
TestingUnit test patterns, strategy tests for randomized multi-block scenarios, and test configuration.
SDKThe client-side dash-sdk crate: builder patterns, fetch traits, and proof verification.
WASMBinding patterns for the browser-facing wasm-dpp and wasm-sdk crates.

Each chapter follows the same arc: why the pattern exists (the problem it solves), what the pattern is (the types and modules involved), how it works (real code from the repository), and rules (the do's and don'ts that keep the codebase consistent).

Coding Conventions

A few conventions appear consistently across the codebase and are worth calling out early:

  • #![forbid(unsafe_code)] is set in both dpp and drive. The platform avoids unsafe Rust entirely in its core logic. Unsafe operations are confined to external dependencies (RocksDB, cryptographic libraries).
  • #![deny(missing_docs)] is enabled in drive, enforcing doc comments on every public item. DPP has this commented out but is moving toward it.
  • Feature-gated compilation is pervasive. A typical crate has 20-80 Cargo features controlling which modules, serialization formats, and integrations are compiled. This keeps binary sizes small and compile times manageable for downstream consumers that only need a subset of functionality.
  • Versioned module layout: When a function has multiple versions, they live in sibling directories named v0/, v1/, etc., with a parent mod.rs that dispatches based on PlatformVersion. This is the dominant structural pattern in Drive-ABCI.

A Note on Reading the Source

The codebase lives in a monorepo at packages/. The Rust crates are prefixed with rs- on disk but have shorter names in Cargo.toml:

Disk pathCrate name
packages/rs-dppdpp
packages/rs-drivedrive
packages/rs-drive-abcidrive-abci
packages/rs-sdkdash-sdk
packages/rs-platform-versionplatform-version
packages/rs-platform-valueplatform-value
packages/rs-platform-serializationplatform-serialization
packages/rs-drive-proof-verifierdrive-proof-verifier

The workspace currently targets Rust 1.98 and protocol version 14 (as of 4.2.0-dev). The workspace Cargo.toml lists 47 member crates, but the core platform logic lives in the first eight listed above.

Let's begin with the architecture.

Platform Comparison

How Dash Platform compares to other blockchain networks across architecture, features, and developer experience. Ratings from - (not supported) through + (basic), ++ (good), to +++ (best in class) reflect relative strength in each dimension.

Overview

BitcoinEthereumSolanaPolkadotNEARCosmos SDKAvalancheDash Platform
Primary purposePaymentsGeneral-purpose smart contractsHigh-throughput smart contractsMulti-chain shared securitySharded smart contractsApp-chain frameworkMulti-chain smart contractsDecentralized data storage and querying
ConsensusNakamoto (PoW)Gasper (PoS)Tower BFT (PoS)GRANDPA + BABE (PoS)Nightshade (PoS)CometBFT (PoS)Snowman (PoS)Tenderdash SBFT (masternode quorums, BLS threshold signatures)
Finality- Probabilistic (~60 min)+ ~13 min (2 epochs)+++ ~0.4s (optimistic)+ ~12-60s (2 rounds)++ ~1-2s+++ Instant (1 block)++ ~1-2s+++ Instant (1 block)
Throughput (simple tx)- ~7 tx/s+ ~15-30 tx/s+++ ~65,000 tx/s+++ Scales per parachain+++ ~100,000 tx/s (sharded)+++ Per-chain++ ~4,500 tx/s++ ~1,000 tx/s

Data and Querying

BitcoinEthereumSolanaPolkadotNEARCosmos SDKAvalancheDash Platform
Data model- UTXOs+ Account / key-value+ Account / key-value+ Account / key-value+ Account / key-value+ App-defined+ Account / key-value+++ Structured documents with secondary indexes
Decentralized querying- Keys only (UTXO lookup)+ Keys only (no native indexing)+ Keys only (via RPC, no proofs)+ Keys only (per parachain)+ Keys only (via RPC, no proofs)+ Keys only (app-specific)+ Keys only (via RPC, no proofs)+++ Rich queries with indexes, ordering, and ranges -- all with proofs
State proofs+ SPV (block headers)++ Merkle-Patricia proofs- No native proofs+ Merkle proofs (per parachain)++ Merkle-Patricia proofs+ IAVL proofs+ Merkle proofs+++ GroveDB Merkle proofs for every query
Light client trust+ Follows longest chain+ Needs sync committee- Trusts RPC provider+ Trusts relay chain- Trusts RPC provider+ Trusts IBC relayer- Trusts RPC provider+++ Cryptographic proof per response -- same security as a full node

The standout difference is light client verification. Most chains either offer no state proofs (Solana, Avalanche), require trusting intermediaries (Polkadot's relay chain, Cosmos IBC relayers, NEAR's RPC providers), or give proofs that are expensive to verify (Ethereum's sync committee). Dash Platform serves a cryptographic proof with every query response, and a single BLS threshold signature is all a client needs to verify it. A mobile wallet gets the same security guarantees as a full node.

Smart Contracts and Programmability

BitcoinEthereumSolanaPolkadotNEARCosmos SDKAvalancheDash Platform
Smart contracts- Limited Script opcodes+++ Solidity / Vyper on EVM+++ Rust / C on SVM++ Per-parachain, typically Wasm++ Rust / JS / AssemblyScript on Wasm VM+ App-specific (Go)++ Solidity on EVM, Rust on Wasm- Coming in v4.0
VM / execution- Script interpreter+++ EVM+++ SVM (eBPF)++ Wasm (per parachain)++ Wasm VM+ No VM (compiled Go)++ EVM + Wasm subnets- No VM (data contracts; VM planned for v4.0)
Developer languages- Script+++ Solidity, Vyper++ Rust, C++ Rust (Substrate)++ Rust, JS, AssemblyScript+ Go++ Solidity, Rust+ JSON Schema (data contracts), Rust/JS/Swift (SDKs)
Smart contract securityN/A+ Reentrancy, gas exploits++ No reentrancy, but complexity++ Sandboxed per parachain++ Wasm sandboxingN/A+ Inherits EVM risksN/A (data contracts are declarative)

Dash Platform takes a fundamentally different approach: instead of a VM that executes arbitrary code, developers define data contracts -- JSON Schema-based specifications that describe the structure and validation rules for their application data. The network stores, indexes, and enforces these schemas directly. This eliminates entire classes of smart contract vulnerabilities (reentrancy, unchecked external calls, gas manipulation). Smart contract support is planned for Platform v4.0 (targeted for mainnet in 2027).

Token Support

BitcoinEthereumSolanaPolkadotNEARCosmos SDKAvalancheDash Platform
Native token standard- BRC-20 via inscriptions+++ ERC-20 / ERC-721 / ERC-1155++ SPL Token+ Per-parachain++ NEP-141 / NEP-171+ Per-chain++ ERC-20 (C-Chain)+++ Protocol-native tokens with declarative rules
Token creation- Requires third-party indexer++ Deploy smart contract++ Deploy program+ Deploy parachain++ Deploy smart contract+ Build app-chain++ Deploy smart contract+++ Declare in data contract -- no code deployment
Freeze / pause- No+ Only if contract implements it++ Mint authority can freeze+ Per-parachain+ Only if contract implements it+ Per-chain+ Only if contract implements it+++ Protocol-level freeze, pause, and destroy
Minting authority- N/A+ Contract owner / governance+ Mint authority+ Per-parachain+ Contract owner+ Per-chain+ Contract owner+++ Individuals, groups with threshold signing, or pre-programmed schedules
Pre-programmed distributions- No+ Requires contract logic+ Requires program logic- No+ Requires contract logic+ Requires app logic+ Requires contract logic+++ Native: time-based, epoch-based perpetual distributions

Dash Platform tokens are first-class protocol objects rather than smart contract deployments. Token behavior (minting rules, supply caps, freeze authority, distribution schedules) is configured declaratively in data contracts and enforced by the protocol itself.

Project and Ecosystem

BitcoinEthereumSolanaPolkadotNEARCosmos SDKAvalancheDash Platform
LicenseMITVarious (GPL, Apache, MIT)Apache 2.0GPL 3.0Apache 2.0 / MITApache 2.0BSD 3-ClauseMIT
Open sourceYesYesYesYesYesYesYesYes
Core languageC++Go, RustRustRustRustGoGoRust
Client SDKs+ Multiple (community)+++ web3.js, ethers.js, viem++ @solana/web3.js+ Polkadot.js+ near-api-js+ CosmJS++ ethers.js (C-Chain)++ Rust, JavaScript, Swift (iOS), Android (coming)
Launched200920152020202020202019 (SDK)20202024 (v1.0 mainnet)
Ecosystem maturity+++ Largest, most established+++ Largest smart contract ecosystem++ Fast-growing DeFi ecosystem+ Growing parachain ecosystem+ Growing dApp ecosystem++ Many sovereign chains++ Growing subnet ecosystem+ Early stage, growing
Identity system- Addresses only+ ENS (contract-based)- No native identity- No native identity+ Named accounts- No native identity- No native identity+++ Protocol-native identities with hierarchical keys and DPNS usernames
Native token privacy+ UTXO model allows address rotation- Account model, all activity linked to one address- Account model, fully transparent- Per-parachain, generally transparent- Account model, fully transparent- Generally transparent- Account model, fully transparent+++ Shielded pool with Orchard/Halo2 ZK proofs

SDK Support

Dash Platform provides SDKs for multiple languages and environments so developers can build applications on whatever stack they prefer.

Available SDKs

SDKLanguageStatusPackageUse case
Rust SDKRustAvailable nowrs-sdkServer-side applications, full-node tooling, direct protocol access
JavaScript SDKJavaScript / TypeScriptAvailable nowjs-evo-sdkNode.js backends, scripts, CLI tools
iOS SDKSwiftComing in v3.1swift-sdkiOS and macOS applications
Android SDKKotlinComing in v3.2--Android applications

Supporting packages

PackagePurpose
rs-sdk-ffiC FFI layer over the Rust SDK; used by the Swift SDK, the Android SDK, and any language that can call C

Choosing an SDK

Building a server or CLI tool? Use the Rust SDK for maximum performance and direct access to all protocol features, or the JavaScript SDK if your stack is Node.js.

Building an iOS or macOS app? Use the Swift SDK (v3.1+), which wraps the Rust SDK through an FFI layer and provides native Swift types.

Building an Android app? The Android SDK (v3.2+) will wrap the same FFI layer with native Kotlin types.

Building for another language? The FFI layer (rs-sdk-ffi) exposes a C-compatible interface that can be called from Python, C#, or any language with C interop support.

What every SDK provides

All SDKs share the same underlying Rust implementation, so behavior is consistent across platforms:

  • Identity management -- create, top up, and manage identities with hierarchical key support
  • Data contract deployment -- define and publish JSON Schema-based data contracts
  • Document operations -- create, update, delete, and query documents with proof verification
  • Token operations -- query balances, supply, statuses, and pre-programmed distributions
  • Name registration -- register and resolve DPNS usernames
  • Proof verification -- every query response can be cryptographically verified against the platform state root

Getting Started

This guide covers prerequisites and local development setup for the Dash Platform monorepo.

Prerequisites

  • Node.js v20+

  • Docker v20.10+

  • Rust v1.92+, with the wasm32 target:

    rustup target add wasm32-unknown-unknown
    
  • protoc (Protocol Buffers compiler) v32.0+. If protoc is not on your PATH, set the PROTOC environment variable to the binary location.

  • wasm-bindgen-cli:

    cargo install wasm-bindgen-cli@0.2.103
    

    Important: the wasm-bindgen-cli version must match the wasm-bindgen version in Cargo.lock. Check with grep 'name = "wasm-bindgen"' Cargo.lock.

    Depending on your system, you may need additional packages before wasm-bindgen-cli will compile (e.g. clang, llvm, libssl-dev).

  • wasm-pack:

    curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
    
  • Build essentials (Debian / Ubuntu):

    apt install -y build-essential libssl-dev pkg-config clang cmake llvm
    

macOS-specific notes

The built-in Apple llvm toolchain does not work for WASM compilation. Install LLVM from Homebrew and put it on your PATH:

brew install llvm
echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

If you use Bash, replace ~/.zshrc with ~/.bash_profile. You can check your default shell with echo $SHELL.

Setup

# Enable corepack (ships with Node.js) to get the correct yarn version
corepack enable

# Install dependencies, configure, and build all packages
yarn setup

# Start the local development environment (runs a local Dash network in Docker)
yarn start

# Run the full test suite (requires a running local node)
yarn test

# Stop the local environment (frees system resources)
yarn stop

# Rebuild after making changes
yarn build

# If you need to restart services after a rebuild
yarn restart

# Complete reset of all local data and builds
yarn reset

Running package-level tests

You can run tests for a single package instead of the entire suite:

yarn workspace <package_name> test

For example:

yarn workspace @dashevo/dapi-client test

See the packages directory for the full list of available packages.

Rust development

# Run tests for a specific Rust crate
cargo test -p <crate_name>

# Run all Rust workspace tests
cargo test --workspace

# Check compilation without building
cargo check --workspace

# Run the clippy linter
cargo clippy --workspace

# Format all Rust code
cargo fmt --all

Monorepo Overview

Dash Platform ships as a single Git repository containing 47 Rust crates, a handful of JavaScript/TypeScript packages, and supporting tooling. This chapter maps the territory: what each crate owns, how they depend on one another, and where the boundaries are drawn.

The Workspace

The top-level Cargo.toml declares a workspace with resolver = "2". It pins a few critical external dependencies at the workspace level -- most notably dashcore (the Rust Dash Core library) and the GroveDB family of crates:

# From Cargo.toml (workspace root)
[workspace.dependencies]
dashcore = { git = "https://github.com/dashpay/rust-dashcore", rev = "53d699c..." }

The workspace version (4.2.0-dev at time of writing, Rust edition 2021, MSRV 1.98) is shared by all member crates through version.workspace = true.

The Core Dependency Chain

Four crates form the spine of the platform. Understanding their dependency order is the single most important thing for navigating the codebase:

    dpp (rs-dpp)
      |
      v
    drive (rs-drive)
      |
      v
    drive-abci (rs-drive-abci)
      |
      v
    dash-sdk (rs-sdk)

Each layer adds a concern:

dpp -- Dash Platform Protocol

Crate name: dpp Path: packages/rs-dpp

DPP defines the data model of the platform. This is where you will find:

  • Identity, IdentityPublicKey, and identity state transitions
  • DataContract and the JSON Schema validation logic
  • Document and document state transitions
  • StateTransition -- the enum that unifies all transition types
  • Prelude types used everywhere: Identifier, BlockHeight, IdentityNonce
  • Token, voting, and withdrawal types
#![allow(unused)]
fn main() {
// From packages/rs-dpp/src/lib.rs
pub mod data_contract;
pub mod document;
pub mod identifier;
pub mod identity;
pub mod state_transition;
pub mod tokens;
pub mod voting;
pub mod fee;
pub mod validation;
}

DPP is deliberately storage-agnostic. It knows nothing about GroveDB, ABCI, or gRPC. It depends on platform-version, platform-value, and platform-serialization for versioned encoding, but it never opens a database.

A key design choice: DPP uses feature flags extensively. The Cargo.toml lists over 80 features that control which serialization formats, validation paths, and system contracts are compiled in. The abci feature, for instance, pulls in the subset needed by Drive-ABCI without dragging in client-side concerns:

# From packages/rs-dpp/Cargo.toml
[features]
abci = [
  "state-transitions",
  "state-transition-validation",
  "validation",
  "random-public-keys",
  "identity-serialization",
  "vote-serialization",
  "platform-value-cbor",
  "core-types",
  "core-types-serialization",
  "core-types-serde-conversion",
  "core_rpc_client",
]

Rule: If you are adding a new type that both the server and the SDK need, put it in DPP. If it needs GroveDB, it belongs in Drive.

drive -- Dash Drive

Crate name: drive Path: packages/rs-drive

Drive is the storage engine. It wraps GroveDB (a Merkle-tree-based key-value store) and provides domain-specific operations: inserting documents, managing identity balances, tracking fee pools, building cryptographic proofs.

#![allow(unused)]
fn main() {
// From packages/rs-drive/src/lib.rs
#[cfg(any(feature = "server", feature = "verify"))]
pub mod drive;
#[cfg(any(feature = "server", feature = "verify"))]
pub mod query;
#[cfg(feature = "server")]
pub mod state_transition_action;
#[cfg(any(feature = "server", feature = "verify"))]
pub mod verify;
#[cfg(feature = "server")]
pub mod fees;
}

Notice the feature split: server includes everything needed to write to the database, while verify includes only what is needed to prove and verify queries. The SDK only enables verify, which means it can check GroveDB proofs without linking in the full storage engine.

Drive depends on DPP for type definitions and on the GroveDB family of crates for storage:

# From packages/rs-drive/Cargo.toml
dpp = { path = "../rs-dpp", features = ["state-transitions"], default-features = false }
grovedb = { git = "https://github.com/dashpay/grovedb", rev = "33dfd48...", optional = true }
grovedb-path = { git = "https://github.com/dashpay/grovedb", rev = "33dfd48..." }
grovedb-costs = { git = "https://github.com/dashpay/grovedb", rev = "33dfd48...", optional = true }

Rule: Never import drive with the server feature from a client-side crate. Use verify only.

drive-abci -- The ABCI Application

Crate name: drive-abci Path: packages/rs-drive-abci

This is the application server. It implements the Tenderdash ABCI interface, orchestrates block processing, validates state transitions, manages platform state, and serves gRPC queries. It is the binary that masternodes run.

Drive-ABCI depends on both DPP and Drive, plus the Tenderdash ABCI library and the dapi-grpc protobuf definitions:

# From packages/rs-drive-abci/Cargo.toml
drive = { path = "../rs-drive", default-features = false, features = ["server"] }
dpp = { path = "../rs-dpp", default-features = false, features = ["abci"] }
tenderdash-abci = { git = "https://github.com/dashpay/rs-tenderdash-abci", tag = "v1.5.0" }
dapi-grpc = { path = "../dapi-grpc", default-features = false, features = ["server", "platform"] }

The crate is organized into three major subsystems:

  1. abci/ -- Tenderdash handler functions (prepare_proposal, process_proposal, finalize_block, check_tx, etc.)
  2. execution/ -- Block processing engine, state transition validation, platform events (epoch changes, withdrawals, voting)
  3. query/ -- gRPC query service implementing the Platform API

Rule: Business logic goes in execution/. The abci/ handlers should be thin wrappers that delegate to the execution engine.

dash-sdk -- The Client SDK

Crate name: dash-sdk Path: packages/rs-sdk

The SDK is what application developers use. It provides high-level methods for fetching documents, creating identities, broadcasting state transitions, and verifying proofs. It depends on DPP for types, Drive (with verify only) for proof verification, and rs-dapi-client for network communication:

# From packages/rs-sdk/Cargo.toml
dpp = { path = "../rs-dpp", default-features = false, features = ["dash-sdk-features"] }
drive = { path = "../rs-drive", default-features = false, features = ["verify"] }
drive-proof-verifier = { path = "../rs-drive-proof-verifier", default-features = false }
rs-dapi-client = { path = "../rs-dapi-client", default-features = false }

Supporting Crates

Several smaller crates provide cross-cutting infrastructure:

platform-version

Path: packages/rs-platform-version

The versioning backbone. Defines PlatformVersion, ProtocolVersion, and the version tables for every consensus-critical method across DPP, Drive, and Drive-ABCI. Currently tracks 14 protocol versions (v1 through v14).

#![allow(unused)]
fn main() {
// From packages/rs-platform-version/src/version/mod.rs
pub type ProtocolVersion = u32;
pub const LATEST_VERSION: ProtocolVersion = PROTOCOL_VERSION_14;
pub const INITIAL_PROTOCOL_VERSION: ProtocolVersion = 1;
}

Every crate in the dependency chain depends on platform-version. It is the root of the version tree.

platform-serialization

Path: packages/rs-platform-serialization

A thin wrapper around bincode that adds platform-version-aware serialization. Paired with platform-serialization-derive for derive macros that generate versioned Encode/Decode implementations.

platform-value

Path: packages/rs-platform-value

A dynamically-typed value type (think serde_json::Value but with binary support). Used as the interchange format when converting between JSON, CBOR, and Rust types, especially in data contract and document processing.

drive-proof-verifier

Path: packages/rs-drive-proof-verifier

Client-side proof verification. Takes a GroveDB proof returned by a platform query and verifies it against a known root hash. Used by the SDK and the WASM bindings.

dapi-grpc

Path: packages/dapi-grpc

Protobuf definitions and generated Rust code for the Platform gRPC API. Both server-side (Drive-ABCI) and client-side (SDK, DAPI client) depend on this crate, using the server and client features respectively.

What Is GroveDB?

GroveDB is an external dependency -- a Merkle-tree-based authenticated data structure built on RocksDB. It is not part of this repository but is central to understanding Drive.

Key properties:

  • Authenticated: Every read can produce a cryptographic proof that the data (or its absence) is consistent with the root hash stored in the block header.
  • Hierarchical: Data is organized into nested trees (subtrees), addressed by paths. A document lives at a path like [contract_id, document_type, document_id].
  • Sum trees: Some subtrees track the sum of their leaf values, used for balance accounting and fee verification.
  • Transactional: All writes happen inside a transaction that can be committed or rolled back atomically.
  • Cost-tracking: Every operation returns a CostResult that records storage and processing costs.

GroveDB is pinned to a specific Git revision in the workspace Cargo.toml and referenced by five sub-crates: grovedb, grovedb-path, grovedb-costs, grovedb-storage, and grovedb-version.

The Full Crate Map

Here is a simplified view of every Rust workspace member, grouped by role:

RoleCrates
Protocol typesdpp, platform-value, platform-serialization, platform-serialization-derive, platform-versioning, platform-value-convertible
Storagedrive
Application serverdrive-abci
Client SDKdash-sdk, rs-dapi-client, dash-context-provider, rs-sdk-trusted-context-provider
Proof verificationdrive-proof-verifier
gRPC definitionsdapi-grpc
WASM bindingswasm-dpp, wasm-dpp2, wasm-sdk, wasm-drive-verify
iOS/FFIrs-sdk-ffi
System contractsdpns-contract, dashpay-contract, withdrawals-contract, masternode-reward-shares-contract, wallet-utils-contract, token-history-contract, keyword-search-contract, document-history-contract, app-connect-contract, moderation-charters-contract, data-contracts
Toolingdashmate (JS), strategy-tests, simple-signer, check-features, json-schema-compatibility-validator
Otherdash-platform-macros, rs-dash-event-bus, rs-platform-wallet, dash-platform-balance-checker, rs-dapi

Rules

Do:

  • Follow the dependency direction. DPP never imports Drive. Drive never imports Drive-ABCI.
  • Use feature flags to keep compilation lean. The SDK should never compile server-side code.
  • Put new domain types in DPP. Put new storage operations in Drive. Put new validation logic in Drive-ABCI.

Don't:

  • Add GroveDB as a dependency to DPP. If you need tree structure knowledge in a type definition, use a path abstraction.
  • Enable the server feature of Drive in client-facing crates. This pulls in RocksDB and doubles compile times.
  • Create new top-level crates without updating the workspace Cargo.toml and ensuring the dependency direction is maintained.

Component Pipeline

This chapter traces a request from the moment it leaves a client application to the moment its effects are committed to GroveDB. Understanding this pipeline is essential because every bug, every audit finding, and every performance issue lives somewhere along this path.

The Big Picture

Client App
  |
  | gRPC (protobuf)
  v
DAPI (rs-dapi) -----------> Drive-ABCI query service (reads)
  |
  | BroadcastStateTransition
  v
Tenderdash mempool
  |
  | ABCI (check_tx, prepare_proposal, process_proposal, finalize_block)
  v
Drive-ABCI (rs-drive-abci)
  |
  | DriveOperations
  v
Drive (rs-drive)
  |
  | GroveDB transaction
  v
GroveDB -> RocksDB

There are two fundamentally different paths through this pipeline:

  1. Reads (queries): Client sends a gRPC query, DAPI forwards it to Drive-ABCI's query service, which reads from GroveDB and returns data with a Merkle proof. No consensus involved.

  2. Writes (state transitions): Client broadcasts a state transition, it enters the Tenderdash mempool, goes through consensus, and is applied to GroveDB during block processing.

Let's trace the write path in detail -- it is where all the complexity lives.

DAPI: The Entry Point

DAPI (packages/rs-dapi) is the internet-facing gRPC server. It implements two roles:

  • Platform queries: Forwarded directly to Drive-ABCI's gRPC service (which runs as a separate listener within the same process).
  • State transition broadcast: Submitted to Tenderdash via its RPC interface.
// Simplified from packages/rs-dapi/src/services/platform_service/broadcast_state_transition.rs
Client --gRPC--> DAPI --Tenderdash RPC--> Tenderdash mempool

DAPI itself does minimal validation. It is a routing layer. The heavy lifting happens in Drive-ABCI.

The ABCI Handler Layer

When Tenderdash needs the application to do something -- check a transaction, build a block, validate a proposal, or finalize a block -- it calls an ABCI method. Drive-ABCI implements these in packages/rs-drive-abci/src/abci/handler/:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/abci/handler/mod.rs
mod check_tx;
mod echo;
mod extend_vote;
mod finalize_block;
mod info;
mod init_chain;
mod prepare_proposal;
mod process_proposal;
mod verify_vote_extension;
}

These handlers are implemented against trait bounds, not concrete types:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/abci/handler/finalize_block.rs
pub fn finalize_block<'a, A, C>(
    app: &A,
    request: proto::RequestFinalizeBlock,
) -> Result<proto::ResponseFinalizeBlock, Error>
where
    A: PlatformApplication<C> + TransactionalApplication<'a> + BlockExecutionApplication,
    C: CoreRPCLike,
{ ... }
}

The FullAbciApplication struct wires everything together. It holds a reference to Platform, a GroveDB transaction, the current block execution context, and the withdrawal transactions of every proposal accepted at the current height, which vote extensions are verified against:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/abci/app/full.rs
pub struct FullAbciApplication<'a, C> {
    pub platform: &'a Platform<C>,
    pub transaction: RwLock<Option<Transaction<'a>>>,
    pub block_execution_context: RwLock<Option<BlockExecutionContext>>,
    pub unsigned_withdrawal_txs_by_round: RwLock<UnsignedWithdrawalTxsByRound>,
}
}

It implements the tenderdash_abci::Application trait, delegating each method to the corresponding handler function.

Block Processing Lifecycle

Tenderdash uses a proposal-based consensus model. Here is the sequence of ABCI calls for a single block:

1. check_tx -- Mempool Gatekeeper

Before a state transition enters the mempool, Tenderdash calls check_tx. This is a lightweight validation that runs outside of consensus:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/abci/handler/check_tx.rs
pub fn check_tx<C>(
    platform: &Platform<C>,
    core_rpc: &C,
    request: proto::RequestCheckTx,
) -> Result<proto::ResponseCheckTx, Error>
where
    C: CoreRPCLike,
{
    let platform_state = platform.state.load();
    let platform_version = platform_state.current_platform_version()?;

    let validation_result = platform.check_tx(
        tx.as_slice(),
        r#type.try_into()?,
        &platform_ref,
        platform_version,
    );
    // ...
}
}

check_tx operates in two modes: mode 0 (new transaction) and mode 1 (re-check existing mempool transactions after a new block). It returns a fee estimate (gas_wanted), a priority for ordering, and a sender identifier for deduplication. Importantly, check_tx does not run inside a GroveDB transaction -- it reads committed state only.

2. prepare_proposal -- The Proposer Builds a Block

The block proposer calls prepare_proposal with a list of candidate transactions. The handler starts a GroveDB transaction and runs the full block proposal:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/abci/handler/prepare_proposal.rs
// Start a GroveDB transaction
app.start_transaction();

// Run the full proposal (validates + executes all state transitions)
let mut run_result = app.platform().run_block_proposal(
    block_proposal,
    true,         // known_from_us = true (we are the proposer)
    &platform_state,
    transaction,
    Some(&timer),
)?;
}

The response tells Tenderdash which transactions to keep, remove, or delay:

  • TxAction::Unmodified -- valid transition, include in block
  • TxAction::Removed -- unpaid error or internal error, strip from block
  • TxAction::Delayed -- exceeded max block size, try next block

3. process_proposal -- Validators Verify the Block

Non-proposing validators receive the block and call process_proposal. This runs the same run_block_proposal logic but with known_from_us = false:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/abci/handler/process_proposal.rs
let run_result = app.platform().run_block_proposal(
    (&request).try_into()?,
    false,        // known_from_us = false (we are validating someone else's proposal)
    &platform_state,
    transaction,
    None,
)?;
}

A key optimization: if the validator was also the proposer for this round (same height and round), the cached result from prepare_proposal is reused:

#![allow(unused)]
fn main() {
// From process_proposal.rs -- cache hit path
if let Some(proposal_info) = block_execution_context.proposer_results() {
    return Ok(proto::ResponseProcessProposal {
        status: proto::response_process_proposal::ProposalStatus::Accept.into(),
        app_hash: proposal_info.app_hash.clone(),
        tx_results: proposal_info.tx_results.clone(),
        // ...
    });
}
}

If the proposal contains failed or unpaid transitions, the validator rejects it.

4. finalize_block -- Commit

After consensus is reached, Tenderdash calls finalize_block. This is where the GroveDB transaction is committed to disk:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/abci/handler/finalize_block.rs
let block_finalization_outcome = app.platform().finalize_block_proposal(
    request_finalize_block,
    block_execution_context,
    transaction,
    platform_version,
)?;

// Commit the GroveDB transaction
let result = app.commit_transaction(platform_version);
}

After commit, the block height counter is updated and, if needed, a GroveDB checkpoint is created for crash recovery.

The finalize response also carries a proposer hint, propose_next_block_immediately (Tenderdash 1.8.0, ABCI 1.4.0). Drive sets it when the block leaves withdrawal work for the next block: untied withdrawal transactions waiting in the queue to be signed. Tenderdash then proposes round 0 of the next height without waiting for transactions or the empty-block interval, so a withdrawal is signed one block after it was pooled instead of one interval later. The hint is local to the node and never part of consensus: it is read from the same GroveDB transaction the block committed, but it does not change the state or the app hash. See has_pending_withdrawal_work under packages/rs-drive-abci/src/execution/platform_events/withdrawals/.

Inside run_block_proposal

The run_block_proposal method in packages/rs-drive-abci/src/execution/engine/run_block_proposal/v0/mod.rs is the heart of the block processing engine. It orchestrates everything that happens within a single block. Here is the sequence, in order:

1.  Verify protocol version matches expected version
2.  Validate block follows previous block (height, core height)
3.  Clear drive block cache
4.  Verify chain lock (if core chain lock update present)
5.  Update core info (masternode list, quorums)
6.  Update validator proposed app version
7.  Rebroadcast expired withdrawals
8.  Update broadcasted withdrawal statuses
9.  Dequeue and build unsigned withdrawal transactions
10. Run DAO platform events (vote tallying, contested documents)
11. Process raw state transitions  <-- the main work
12. Store address balance changes
13. Clean up expired address balance entries
14. Pool withdrawals into transaction queue
15. Clean up expired withdrawal amount locks
16. Process block fees and validate sum trees
17. Compute root hash (app_hash)
18. Determine validator set update

Steps 1-10 are "block-level housekeeping." Step 11 is where individual state transitions are decoded, validated, transformed into actions, and applied to GroveDB. Steps 12-18 finalize the block's effects.

State Transition Processing

State transition processing (step 11 above) follows its own pipeline within packages/rs-drive-abci/src/execution/platform_events/state_transition_processing/:

decode_raw_state_transitions
       |
       v
process_raw_state_transitions
       |
       v
   For each transition:
       |
       +-> validate (structure, state, signatures)
       +-> transform_into_action
       +-> validate_fees_of_event
       +-> execute_event (apply DriveOperations to GroveDB)

The validation itself is split into modules per transition type:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/mod.rs
pub mod batch;
pub mod identity_create;
pub mod identity_credit_transfer;
pub mod identity_credit_withdrawal;
pub mod identity_top_up;
pub mod identity_update;
pub mod data_contract_create;
pub mod data_contract_update;
pub mod masternode_vote;
// ... and more
}

Each module provides versioned validation methods dispatched through PlatformVersion, following the same pattern shown in the Introduction.

The Query Path

Queries bypass consensus entirely. Drive-ABCI implements the Platform gRPC service directly:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/query/service.rs
// Implements dapi_grpc::platform::v0::platform_server::Platform
// with methods like:
//   get_identity()
//   get_documents()
//   get_data_contract()
//   get_identity_balance()
//   ... (50+ query endpoints)
}

Each query reads from the committed GroveDB state (no transaction), generates a Merkle proof, and returns both the data and the proof. The client SDK uses drive-proof-verifier to independently verify that the returned data matches the proof against a known root hash.

Protocol Version Upgrades During Block Processing

One subtlety worth highlighting: protocol version changes happen at epoch boundaries. The run_block_proposal method checks whether the current block is the first block of a new epoch and whether the locked-in next protocol version differs from the current one:

#![allow(unused)]
fn main() {
// From packages/rs-drive-abci/src/execution/engine/run_block_proposal/mod.rs
let block_platform_version = if epoch_info.is_epoch_change_but_not_genesis()
    && platform_state.next_epoch_protocol_version()
        != platform_state.current_protocol_version_in_consensus()
{
    let next_protocol_version = platform_state.next_epoch_protocol_version();
    let next_platform_version = PlatformVersion::get(next_protocol_version)?;

    // Perform structural changes for the new protocol version
    self.perform_events_on_first_block_of_protocol_change(
        platform_state,
        &block_info,
        transaction,
        old_protocol_version,
        next_platform_version,
    )?;

    next_platform_version
} else {
    last_committed_platform_version
};
}

This ensures that all nodes switch protocol versions at exactly the same block, and that any structural migrations (new GroveDB trees, schema changes) are applied atomically as part of that block's transaction.

Rules

Do:

  • Keep ABCI handlers thin. They should parse the request, delegate to the execution engine, and format the response. No business logic.
  • Always pass platform_version through the call stack. Never hard-code a version number in execution logic.
  • Run check_tx validation as a strict subset of proposal validation. If check_tx accepts a transition, process_proposal should not reject it (modulo state changes between the two calls).

Don't:

  • Read uncommitted state in check_tx. It reads committed state only because it runs outside the block transaction.
  • Assume prepare_proposal and process_proposal see the same state. Another block may have been committed between the two calls if a round change occurs.
  • Add new block-level events without inserting them in the correct position in the run_block_proposal sequence. Order matters -- withdrawals must be processed before state transitions, fee accounting must happen after.
  • Panic in ABCI handlers except for truly unrecoverable situations (like app hash mismatches, which indicate data corruption).

Coding Conventions

The other chapters of this book explain how each subsystem works. This chapter is the prescriptive companion: the rules a change has to follow to keep the codebase maintainable, and the reason behind each one. Most of these rules were learned in code review rather than designed up front, so each entry says what goes wrong when the rule is skipped. Where a rule has mechanics, it links to the chapter that shows them.

Read this before your first non-trivial PR. Skim the checklists at the end before every PR.

The one constraint behind every rule

Dash Platform is a replicated state machine. Every masternode must produce a byte-identical state root for every block, including blocks from years ago that a fresh node replays from genesis. That means code that shipped inside a released protocol version must keep behaving exactly as it did on the day it shipped, forever, while the same binary also runs the newest behaviour for current blocks.

Almost everything below follows from that constraint: why behaviour is split into frozen generations, why numbers live in version tables, why a stray unwrap is a network outage and not a bug report, and why a check has to sit in the right validation tier. When a rule here seems pedantic, the question to ask is "could two nodes disagree because of this?"

Where code goes

The crates are layered, and dependencies only point downward: platform-version at the bottom, then dpp, then drive. Above drive the graph forks: drive-abci (the node) and dash-sdk (the client) both depend on drive, the SDK with the verify feature only, and dash-sdk does not depend on drive-abci at all (see the Monorepo Overview). Put a change in the lowest crate that has what the change needs, and no lower.

You are adding…It lives inNotes
A protocol type, its wire shape, or a pure-data invariantpackages/rs-dppKnows nothing about storage. Structural validation that needs only the data itself goes here.
Storage layout, GroveDB operations, index maintenance, query lowering, proof generationpackages/rs-drive (server feature)One method per directory with versioned dispatch.
Proof verification a client runspackages/rs-drive/src/verify/Must compile with --no-default-features --features verify.
A validation step that reads platform state, block execution, an ABCI handlerpackages/rs-drive-abci/src/execution/Handlers under abci/ stay thin and delegate.
A gRPC query handlerpackages/rs-drive-abci/src/query/<name>/ plus the proto in packages/dapi-grpcRequest version dispatch mirrors method version dispatch.
A version number, a limit, a fee ratepackages/rs-platform-versionNever a literal in a method body.
Client-side proof composition (FromProof, Tenderdash signature check)packages/rs-drive-proof-verifierCalls into drive::verify, never into GroveDB directly.
A client APIpackages/rs-sdkFollow the query checklist in packages/rs-sdk/README.md.
A JavaScript bindingpackages/wasm-dpp2, packages/wasm-sdkMirror the Rust shape; never validate. See packages/wasm-dpp2/CONVENTIONS.md.
Mobile orchestration (sync, identity registration, DashPay)packages/rs-platform-walletThe FFI crates and the Swift and Kotlin SDKs marshal; they do not decide.

Three boundaries are enforced by CI and worth knowing by name:

  • The verify-only cut. packages/wasm-drive-verify builds drive with default-features = false, features = ["verify"]. Anything under src/verify/** that reaches a server-gated helper breaks the JavaScript build on a Rust-only PR. A helper both the prover and the verifier need goes in drive::util::common, and the server-side code delegates to it.
  • The transport-free cut. drive-proof-verifier, dash-platform-queries, and the wasm32 dependency trees of dash-sdk and wasm-sdk must not pull in hyper, rustls, tower, or mio. Embedders with their own networking consume verification through these crates, so a dependency added for convenience in a shared crate can fail a job that has nothing to do with the change.
  • The wallet closure. The wallet CI fast path compiles only the wallet crates and their known dependents. Adding a dependency on a wallet crate from a new place fails the build until the closure list in .github/scripts/check-wallet-closure.py is updated deliberately.

Versioned behaviour

The mechanics of PlatformVersion, feature version tables, and the dispatcher shape are covered in the Versioning chapters. The rules here are about what to do with those mechanics.

Shipped generations are frozen unless the change cannot modify consensus

A behaviour change to a versioned method means a new vN module selected only by the tables of the unreleased protocol version. A shipped vN may be edited in place only when we are sure the edit cannot modify consensus at any protocol version that selects it: the new code is unreachable there by construction (the data it acts on cannot exist under those versions, such as a keyword every one of their meta-schemas refuses and their parser ignores, judged through a dpp method whose own gate is None there), or the edit is a pure refactor with identical output. "Probably inert" is not enough. If the argument takes more than a sentence, or rests on a runtime check inside the shipped module, add a generation instead. Inside a new generation the capability is a constant fact (Index::try_from_value_map(map, true)), not a runtime check. For new changes, the patterns that compare protocol_version on purpose are the protocol upgrade ladder, the is_allowed gate for new transition kinds, and chain-creation content (create_genesis_state v1 and the Drive helpers that build the initial state structure); the Versioned Dispatch chapter describes them. Older comparisons elsewhere (for example the pre-version-9 arithmetic in DocumentPropertyType) stay because history replays through them; they are not a pattern to copy.

Why: replay safety is structural when a shipped file stays byte-identical, and becomes a proof the reviewer has to check the moment it does not. An in-place edit is acceptable when that proof is short and written down; a dead version check inside v1 that misleads the next reader into thinking v1 can take that path is not.

How, new generation: copy the previous generation into the new module, make the change there, move the tests that exercise the new behaviour into the new module, and bump the method's number in the new protocol version's tables only. Duplication between generations is the accepted cost; it is cheaper than a drift-prone flag.

How, in place: make the edit, leave a comment at the edited lines naming why they are inert for every protocol version that selects the module, and give the pull request description an "In-place changes to shipped generations" section that lists each edited generation, the protocol versions that select it, and the reason consensus cannot change there. Reviewers read that section first.

Table versions follow protocol-version boundaries, not PRs

Version-table constants (DRIVE_ABCI_VALIDATION_VERSIONS_V10, CONTRACT_VERSIONS_V6, SYSTEM_LIMITS_V4, and so on) exist to mark the point where a released protocol version froze them. If the unreleased protocol version already introduced a new table version, a second feature landing in the same protocol version amends that table in place. Adding another table version would leave one referenced by no PLATFORM_V* at all.

Leaf implementation modules are different: each feature still gets its own new vN implementation module under the method directory. The rule is about the tables.

Shared helpers get a versioned method, not a capability flag

When new behaviour lives in a helper reached from several generations (a value walker, a property-reference resolver, a shared insert path), the fix is the standard versioned-method pattern: an OptionalFeatureVersion field in the tables (None for versions that predate the feature, Some(0) to dispatch to a _v0 implementation), and the helper takes &PlatformVersion. A admit_property_references: bool on a shared context struct is the wrong shape, because it moves the version decision away from the tables into whichever caller happened to set the flag.

Generation-owned grammar constants may live on a per-generation struct. Feature gates reachable from more than one generation may not.

Numbers live in the version tables

If a protocol version changes only a number (a cap, a floor, a count), the number goes into SystemLimits or the relevant *_constants table, and the versioned method reads it. Do not add a vN method module whose entire body is return NEW_CONSTANT.

How: add the field with a doc comment naming the method version that reads it. Backfill every shipped SYSTEM_LIMITS_V* (and the mock tables under version/mocks/) with the old value so shipped protocol versions are behaviour-preserving. Add the next SYSTEM_LIMITS_V{n+1} for the unreleased protocol version with the new value. Keep a real method version only where the logic differs: in packages/rs-dpp/src/withdrawal/daily_withdrawal_limit/, v0 computes a tiered percentage of total credits and v2 reads daily_withdrawal_limit_percent from SystemLimits; that is a logic change and earns its own version. Raising the percentage later would be a table edit, not a v3.

Why: reviewers look for limits in the tables. A constant hidden in a method module is a second place to check and a dead module per bump.

Fees change through named schedules

A fee change for a new protocol version is a new named schedule (FEE_VERSION2 in packages/rs-platform-version/src/version/fee/v2.rs, and the next one for the next protocol version) referenced from that PLATFORM_V*, with versioned fee-group constants where a group changed. Never inline a nested FeeVersion { .. } override inside the platform version constant. Preserve earlier schedules. If the unreleased protocol version already owns a new schedule, amend that schedule instead of creating another generation nobody references.

fee_version_number is a separate concept from the schedule generation: it keys persisted fee history and storage-refund behaviour, and it is stored in platform state. Inspect its consumers before changing it. FEE_VERSION1 and FEE_VERSION2 both carry fee_version_number: 1 because storage rates did not change between them, only a fee group did. See the Fee System Overview.

Consensus code reads the version from state

PlatformVersion::latest() is what the binary supports. The active version is what the network agreed on, obtained from platform_state.current_platform_version(). During an upgrade window they differ. latest() belongs in tests and in client code that is choosing which format to emit; it never belongs on a path that decides what a block does.

Serde-based conversions of DataContract read a process-wide current version set through PlatformVersionCurrentVersion::set_current. Tests that build platforms at different protocol versions race on it. Prefer the explicit to_value(platform_version) style API wherever a version is in hand.

Dispatcher shape

The Versioned Dispatch chapter shows the canonical match and its rules. Three details that come up in review:

  • The implementation is pub(super) fn name_v0 in v0/mod.rs, usually #[inline(always)]. Nothing outside the method directory calls it.
  • The catch-all arm is version => Err(...UnknownVersionMismatch { method, known_versions, received: version }), using the crate's own error (DriveError, ExecutionError, ProtocolError). Optional stages add a None => Err(...VersionNotActive { .. }) arm.
  • #[allow(clippy::too_many_arguments)] on a dispatcher and its implementations is accepted. Do not invent a one-off parameter struct only to silence the lint.

platform_version is the last parameter

The observed order in drive and drive-abci is: &self, domain inputs, block_info, mode flags (apply, stateful), the estimation map, transaction, the operations accumulator, platform_version. A new parameter goes up with the context items, never after platform_version. The same holds for fields on context structs such as parse contexts. The known exception is previous_fee_versions, which trails it on fee-returning methods.

Why: readers of a few hundred versioned signatures expect the closing parameter to be the version. Appending after it breaks that scan.

Versioned types

A protocol type that may evolve is an enum over per-version structs (DataContract { V0(DataContractV0), V1(DataContractV1) }) with accessor traits (DataContractV0Getters) implemented on the enum, so call sites never match on the variant. The serde tag for versioned enums is $formatVersion with variants renamed to "0", "1"; $version is reserved for the legacy protocol-version fields and must not be reused. New variants are appended.

The full pattern, including the derive stack and the round-trip test template, is in docs/json-value-conversion-canonical-pattern.md and the Serialization chapters.

Where a check belongs

A new validation rule has exactly one correct tier, determined by what it needs to read. The Validation Pipeline chapter walks the stages in order; this table is the placement guide.

TierLocationMay readError class
Pure-data invariantsrs-dpp (validate_basic_structure, document type and contract self-validation)The value itself and PlatformVersionBasicError
Basic structuredrive-abci <transition>/basic_structure/vN/Network and PlatformVersion onlyBasicError
Signature and noncesdrive-abci processor traitsThe signing identity, noncesSignatureError, unpaid rejection
Advanced structure without statedrive-abci <transition>/advanced_structure/vN/ (validate_advanced_structure)The transition, the fetched PartialIdentity, and PlatformVersion; no Drive readsConsensusError, paid
Advanced structure with statedrive-abci batch/advanced_structure/vN/ and the per-action validators under batch/action_validation/*/advanced_structure_vN/ (validate_advanced_structure_from_state)The action produced by transform_into_action, including the contracts it fetched, plus block, network, and identity context; no further Drive reads in the validatorConsensusError, paid
Statedrive-abci <transition>/state/vN/Drive through PlatformRef and the transactionStateError, paid
Executiondrive operationsGroveDBInternal Error only

Rules that fall out of the table:

  • Basic structure runs inside check_tx on every mempool entry. Keep it cheap and stateless.
  • Advanced structure without state uses the transition and the already fetched signing identity. Advanced structure with state uses data fetched by transform_into_action, such as the contract schema carried by a document action: the transformer performs the Drive reads, and the advanced validator consumes the resulting action. A transition opts into the second variant through has_advanced_structure_validation_with_state; today only Batch does.
  • Checks that require additional Drive queries, such as uniqueness checks, belong in state validation. Needing a fetched contract does not by itself make a structural check a state-validation check.
  • Preserve mempool coverage. Batch runs advanced structure with state during check_tx, while full state validation is skipped there (validates_full_state_on_check_tx defaults to false; masternode votes are the one transition that opts in, because a block refuses them unpaid, and they run advanced structure with state there too). Moving a contract-dependent structural check into state validation would remove that rejection from mempool admission.
  • Validation outcomes are ConsensusValidationResult, returned as Ok. A Result::Err from a validation function means the node is broken, not that the transition is invalid. The block loop converts it into an internal-error result instead of halting, and process_proposal rejects any block that contains one.
  • Validation lives once. wasm-dpp2, the SDKs, and the FFI layer never duplicate a length, range, or count check that rs-dpp enforces, even when the duplicate would give a friendlier early error. Two definitions of "valid" drift the day one of them changes.
  • Consensus errors carry numeric codes in fixed bands (packages/rs-dpp/src/errors/consensus/codes.rs): basic errors in 10000-19999, signature in 20000-29999, fee in 30000-39999, state in 40000-49999, with sub-bands per domain. A new error takes the next free code in its band and is appended to its enum. See Consensus Errors and Error Codes.

Errors, panics, and arithmetic

A panic in block execution halts the chain

packages/rs-drive-abci/src/main.rs installs a panic hook that cancels the node, and there is no catch_unwind anywhere in the crate. A panic reachable from process_proposal or finalize_block is deterministic, so every validator hits it on the same block and the network stops. This is the strongest rule in the codebase:

  • On any path reachable from block execution, no unwrap, no expect without a proof, no slice indexing without a bounds check, no unreachable!, no division by a value that could be zero.
  • expect is allowed when the message states the invariant established immediately above it, in the style the codebase already uses: .expect("check above enforces it exists"), .expect("we have already shown there is 1 byte"). If you cannot write that sentence, return an error.
  • A condition that "cannot happen" but is reachable returns DriveError::CorruptedCodeExecution (or CriticalCorruptedState when the right response is to stall) or ExecutionError::CorruptedCodeExecution. See Drive Errors.
  • Action transformers under state_transition_action/**/transformer.rs run before fee and funding checks, so they are the most exposed surface. Treat every conversion there as untrusted input.

The *_blocking accessors in rs-platform-wallet are a related hazard on the client side: tokio::sync::RwLock::blocking_read panics on a runtime worker, and the iOS profiles build with panic = "abort". When the current thread may be inside a runtime, use try_read and return a typed busy error.

Integer arithmetic wraps in release

The root Cargo.toml does not enable overflow-checks, so a release build wraps silently and a debug build panics. Neither is acceptable in credit, fee, or balance math. Use the checked forms and surface Overflow; FeeResult already exposes checked_add_assign for the common case.

Append-only structures are enforced by CI

Structures marked // @append_only (the drive-abci error enums, the data contract error enums, document property types, storage key requirements, CheckTxLevel) accept new variants at the end and nothing else. Structures marked // @immutable (Epoch, BlockInfo) accept no change at all. The Detect immutable structure changes step in .github/workflows/tests-rs-workspace.yml diffs the tagged block against the base branch and fails the PR on a deletion or, for @immutable, any change. The reason is serialization: these types are encoded into proofs, responses, and stored state with positional discriminants. Reordering or removing a variant changes what old bytes mean.

The same rule applies without a marker to anything with a DO NOT CHANGE ORDER banner, to StateTransitionType discriminants, and to SystemDataContract slots (note the reserved FeatureFlags = 2).

Consensus errors versus internal errors

A ConsensusError is something two nodes must agree on: it is deterministic, carries a code, and is serialized to the client. A ProtocolError, DriveError or ExecutionError means the build is wrong, data is corrupt, or an API was misused. Never return a consensus error for an internal failure or an internal error for a user mistake; the block loop treats the two classes differently.

Module layout and style

  • One method, one directory. method_name/mod.rs holds the public dispatcher and its doc comment; method_name/v0/mod.rs holds the implementation. The dispatcher's doc comment has # Parameters and # Returns sections. drive and drive-abci compile with #![deny(missing_docs)], so every public item needs a doc line.
  • Drive's naming triad. *_operations gathers Vec<LowLevelDriveOperation> without applying anything and takes the estimated_costs_only_with_layer_info map for dry runs; *_apply_and_add_to_operations appends into a caller-owned accumulator and applies; the bare public method owns the accumulator and returns a FeeResult. Keep new storage methods in this shape so cost estimation and real execution share one code path. See Drive Operations.
  • Tests sit with the implementation. A #[cfg(test)] mod tests block at the bottom of vN/mod.rs, or vN/tests/ when it outgrows the file. The dispatcher's mod.rs may carry end-to-end tests that need every version.
  • Imports at the top. No fully qualified crate::a::b::c::function(...) at a call site or in a signature. Add a use and call it bare, or keep at most one module qualifier when the bare name would be ambiguous. Inline paths are fine in doc comments and inside macros that cannot see imports.
  • No unsafe. dpp, drive, and drive-abci compile with #![forbid(unsafe_code)]. Unsafe stays in the FFI crates and in dependencies.
  • Test-code lints are real lints. CI runs cargo clippy --workspace --all-targets --all-features -- -D warnings, so clippy::type_complexity in a test helper or &vec![..] passed as a slice in a test fails the build. drive-abci allows a fixed list of test lints at the crate root; do not extend it for convenience.
  • State access is snapshot-based. Platform.state is an ArcSwap; read with load() and pass PlatformRef or PlatformStateRef down. The only locks are the ABCI application's transaction, block_execution_context and unsigned_withdrawal_txs_by_round, taken in that order. In async code, a guard's scope, not a drop() call, is what clears clippy::await_holding_lock; wrap the guard in a block that ends before the first .await.

Tests

  • The latest generation tests against PlatformVersion::latest(). Only frozen older generations pin an explicit version with PlatformVersion::get(n). When you introduce v(N+1), that is the moment to pin the now-frozen vN module's tests. Pinning the current generation to today's number only creates churn when the next version lands.
  • Test behaviour through the dispatcher, on both sides of the gate. "PV13 rejects this query shape, PV14 accepts it" is a real test. "Slot foo is 0 in PLATFORM_V13 and 1 in PLATFORM_V14" restates the table literal and cannot fail without the edit being deliberate; do not write it. The exception is a historical freeze test that guards a shipped on-chain table with a replay rationale, like historical_method_table_freeze in the drive document method versions.
  • Names start with should, unit tests never touch the network, and randomness is seeded (StdRng::seed_from_u64). Platforms come from TestPlatformBuilder; Drives come from setup_drive_with_initial_state_structure. See Unit Tests.
  • Multi-block behaviour is a strategy test. Anything involving epochs, upgrades, withdrawals over time, or proposer churn belongs in packages/rs-drive-abci/tests/strategy_tests/. See Strategy Tests.
  • Fixtures are shared. rs-dpp exposes its fixtures under the fixtures-and-mocks feature; rs-sdk records test vectors from a devnet and replays them offline. Add to those rather than hand-rolling payloads.
  • Struct-literal churn is caught by --all-targets. A new required field compiles clean under a lib-only cargo check and then fails in a test module in CI. After adding a field or changing a signature, run cargo check --workspace --all-targets.

Client layers

Proof verification is layered

drive::verify does the GroveDB work: each verifier is a method on the Drive*Query type, builds the path query with the same builder the prover uses, and returns (RootHash, T). drive-proof-verifier takes the full Proof message, calls into that method, and layers the Tenderdash signature check on top; the FromProof impl is the SDK's entry point. Never call GroveDb::verify_* from the proof-verifier crate and never build the path query at the SDK call site; both create a "these bytes must match the prover" invariant that nothing checks. Entry types live in drive and are re-exported so SDK consumers need no direct drive dependency.

The wasm layer mirrors, it does not reshape

rs-dpp serde defines the canonical wire shape and wasm-dpp2 mirrors it one to one, differing only in primitive encoding (Uint8Array versus base64, bigint versus number-or-string). Sum types are internally tagged with type, never wrapped in data. Field-like accessors are properties, verbs are methods. The rules and their reasons are in packages/wasm-dpp2/CONVENTIONS.md and the WASM chapters.

FFI and mobile SDKs hold no business logic

The FFI crates (rs-sdk-ffi, rs-platform-wallet-ffi, key-wallet-ffi) and the Swift and Kotlin SDKs do three things: persist Rust-emitted state, load it for views, and call a function that already exists in a Rust library. If the wrapper is deciding anything (an index, a derivation path, which key, how many), that decision moves into Rust, usually rs-platform-wallet. An FFI function is named for the user-facing verb (sendContactRequest, sendToContact), not for a cryptographic or wallet building block; DIP-14 and DIP-15 derivation primitives stay internal. The motivating case was a Swift view that iterated the gap limit, built the DIP-9 path, and pulled the mnemonic across the FFI twice to do what one preview_identity_registration_keys call now does.

Commits, pull requests, and what to run before pushing

  • Conventional Commits, with the repository's own scope list. Types and scopes are enforced by .github/workflows/pr.yml. The subject starts lowercase. A change that spans packages uses no scope rather than an invented one (deps is not a scope; rs-platform-wallet-ffi changes use platform-wallet).

  • ! means consensus-breaking. feat!: and fix!: are reserved for changes that would fork the network if rolled out unevenly: protocol rules, state-transition shapes, the fee model, anything nodes must agree on. Removing a public Swift symbol or an extern "C" function is ordinary feat: or refactor: work described in the PR body.

  • Design documents are working artifacts. A spec or plan written to drive a change stays in the working tree and is never staged. Stage with explicit paths. Facts worth keeping after merge go into a code comment, a test name, or the PR description.

  • Local gate before pushing. The exhaustive pass is CI's job; the local pass is scoped to what changed:

    cargo fmt --all
    cargo clippy -p <crate> --all-features --all-targets -- -D warnings
    cargo check --workspace --all-targets          # after any field or signature change
    cargo check -p drive --no-default-features --features verify   # when touching src/verify/**
    cargo test -p <crate> <filter>                 # the tests for the change, not the suite
    

    Do not block a push on the full drive-abci suite; it runs for a quarter of an hour and CI runs it anyway.

Checklists

Adding a new generation of a versioned method

  1. Copy vN/ to v(N+1)/, make the change there, leave vN/ byte-identical.
  2. Add the N+1 => arm to the dispatcher and extend known_versions.
  3. Bump the method's number in the unreleased protocol version's tables only. If that protocol version already has a new table constant, amend it.
  4. Move or write the behaviour tests in v(N+1)/ against PlatformVersion::latest(); pin vN/'s tests to PlatformVersion::get(n).
  5. Add a test that runs both versions through the dispatcher.

Editing a shipped generation in place

  1. Only when the edit cannot modify consensus at any protocol version that selects the module: unreachable by construction there, or output-identical.
  2. Comment the edited lines with why they are inert for those versions.
  3. Add a test that runs the module at the last shipped protocol version and shows the outcome unchanged.
  4. Add an "In-place changes to shipped generations" section to the pull request description: each edited generation, the versions that select it, and the reason consensus cannot change there.

Changing a limit or a fee

  1. A number: add or update the SystemLimits (or *_constants) field, backfill shipped tables with the old value, add the next table for the unreleased protocol version.
  2. A fee: add or amend the unreleased protocol version's named FEE_VERSION* schedule; do not touch fee_version_number unless storage rates changed and you have read its consumers.
  3. The method reads the table. No new vN unless the logic changed.

Adding a consensus error

  1. New file under errors/consensus/{basic,state,signature,fee}/<domain>/ with the standard derive stack, private fields, new(), getters, and the ordering banner.
  2. Append the variant to its sub-enum; add the From impl for ConsensusError.
  3. Assign the next free code in the band in codes.rs.
  4. If JavaScript branches on it, mirror the code in packages/wasm-dpp2/src/consensus_error.rs.

Adding a check to a state transition

  1. Pick the tier from the table above by what the check reads.
  2. Put it in <transition>/<tier>/v(N+1)/ if the transition has shipped, or in the existing vN if that generation is still unreleased.
  3. Return it through ConsensusValidationResult; never Err.
  4. Test the rejection through process_raw_state_transitions, and test that the previous protocol version still accepts the input.

Adding GroveDB structure

  1. Put the key constant and the *_path() / *_path_vec() pair in the area's paths.rs. A new root key goes where writes below it are rare; see The GroveDB Structure.
  2. Build it in one function that both create_initial_state_structure and the protocol upgrade (transition_to_version_N) call, in the same order, so a fresh chain and an upgraded one hold the same state.
  3. Describe it in the structure.rs beside that paths.rs, from the same constant, with since set to the protocol version that introduces it and the element flags it is written with.
  4. Regenerate the exported file and commit it: UPDATE_GROVEDB_STRUCTURE=1 cargo test -p drive --lib structure::tests.
  5. Make sure something writes to the new nodes under test: a fixture in structure/tests.rs, or a strategy test (every chain they run is checked against the description). UNVERIFIED is empty; keep it that way.
  6. The pull request gets a comment with a link to the structure viewer showing the new nodes. Check that it shows what you meant to add.

Adding a query end-to-end

  1. Proto message in packages/dapi-grpc/protos/platform/v0/platform.proto, registered in build.rs's versioned request and response lists.
  2. Drive query type and prover, then drive::verify method with its FeatureVersion slot, with a prover-verifier round trip test.
  3. drive-abci handler under query/<name>/{mod.rs,v0/} dispatching on the request version against drive_abci.query bounds.
  4. drive-proof-verifier FromProof wrapper, then the rs-sdk Query and Fetch/FetchMany impls per the checklist in packages/rs-sdk/README.md.
  5. Genesis test data and a recorded test vector so the SDK test runs offline.

Platform Version

The Problem: Deterministic Upgrades in a Distributed System

Dash Platform is a replicated state machine. Every masternode in the network processes the same transactions and must arrive at the exact same state. If even one node computes a fee differently, serializes a document with one extra byte, or validates a field that others skip, the chain forks.

Now imagine you need to ship a bug fix. In a normal application you deploy the new binary and move on. In a blockchain you have a harder constraint: not every node upgrades at the same moment. Some masternodes will still be running the old code when the new protocol version activates. The system needs a way to say "at protocol version N, use these exact behaviors" -- and it needs to be impossible for a developer to accidentally mix behaviors from different versions.

The answer is the PlatformVersion struct: a single, massive, immutable snapshot that pins every versioned behavior in the entire platform to a concrete value.

The PlatformVersion Struct

Open packages/rs-platform-version/src/version/protocol_version.rs and you will find the heart of the system:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug)]
pub struct PlatformVersion {
    pub protocol_version: ProtocolVersion,
    pub dpp: DPPVersion,
    pub drive: DriveVersion,
    pub drive_abci: DriveAbciVersion,
    pub consensus: ConsensusVersions,
    pub fee_version: FeeVersion,
    pub system_data_contracts: SystemDataContractVersions,
    pub system_limits: SystemLimits,
}
}

Where ProtocolVersion is simply:

#![allow(unused)]
fn main() {
pub type ProtocolVersion = u32;
}

Every field inside PlatformVersion is itself a version struct -- and those structs contain more version structs, all the way down to individual method version numbers. We will explore that nesting in the next chapter. For now, the key insight is that PlatformVersion is the root of a tree. Given a single protocol version number (like 7), you can resolve the exact version of every method, every fee parameter, every system limit, and every data contract schema across the entire platform.

Think of it like a lockfile in a package manager. Cargo.lock pins every transitive dependency to a specific version so that builds are reproducible. PlatformVersion does the same thing for runtime behavior: it pins every function version so that execution is deterministic.

The Version Array

Each protocol version gets its own constant, defined in a separate file. At the time of writing, the platform has fourteen versions:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/version/mod.rs

pub type ProtocolVersion = u32;

pub const LATEST_VERSION: ProtocolVersion = PROTOCOL_VERSION_14;
pub const INITIAL_PROTOCOL_VERSION: ProtocolVersion = 1;
pub const ALL_VERSIONS: RangeInclusive<ProtocolVersion> = 1..=LATEST_VERSION;
}

These fourteen snapshots are collected into a single static array in protocol_version.rs:

#![allow(unused)]
fn main() {
pub const PLATFORM_VERSIONS: &[PlatformVersion] = &[
    PLATFORM_V1,
    PLATFORM_V2,
    PLATFORM_V3,
    PLATFORM_V4,
    PLATFORM_V5,
    PLATFORM_V6,
    PLATFORM_V7,
    PLATFORM_V8,
    PLATFORM_V9,
    PLATFORM_V10,
    PLATFORM_V11,
    PLATFORM_V12,
    PLATFORM_V13,
    PLATFORM_V14,
];

pub const LATEST_PLATFORM_VERSION: &PlatformVersion = &PLATFORM_V14;
pub const DESIRED_PLATFORM_VERSION: &PlatformVersion = LATEST_PLATFORM_VERSION;
}

The array is indexed by protocol version number minus one (since versions are 1-indexed). PLATFORM_V1 sits at index 0, PLATFORM_V14 at index 13. This simple layout is what makes the get function so fast.

One file, one protocol version. v14.rs was created when the first consensus change after version 13 shipped needed somewhere to live, and every later change destined for version 14 amends that same file. The day version 14 is released the file freezes: from then on it is part of the chain's historical record, and the next consensus change creates v15.rs. There is never a v14.rs that means one thing on a node built last month and another on a node built today.

What a Version Snapshot Looks Like

Here is the very first version, PLATFORM_V1, slightly abbreviated:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/version/v1.rs

pub const PROTOCOL_VERSION_1: ProtocolVersion = 1;

pub const PLATFORM_V1: PlatformVersion = PlatformVersion {
    protocol_version: 1,
    drive: DRIVE_VERSION_V1,
    drive_abci: DriveAbciVersion {
        structs: DRIVE_ABCI_STRUCTURE_VERSIONS_V1,
        methods: DRIVE_ABCI_METHOD_VERSIONS_V1,
        validation_and_processing: DRIVE_ABCI_VALIDATION_VERSIONS_V1,
        withdrawal_constants: DRIVE_ABCI_WITHDRAWAL_CONSTANTS_V1,
        query: DRIVE_ABCI_QUERY_VERSIONS_V1,
        checkpoints: DRIVE_ABCI_CHECKPOINT_PARAMETERS_V1,
    },
    dpp: DPPVersion {
        costs: DPP_COSTS_VERSIONS_V1,
        validation: DPP_VALIDATION_VERSIONS_V1,
        state_transition_serialization_versions: STATE_TRANSITION_SERIALIZATION_VERSIONS_V1,
        state_transition_conversion_versions: STATE_TRANSITION_CONVERSION_VERSIONS_V1,
        state_transition_method_versions: STATE_TRANSITION_METHOD_VERSIONS_V1,
        state_transitions: STATE_TRANSITION_VERSIONS_V1,
        contract_versions: CONTRACT_VERSIONS_V1,
        document_versions: DOCUMENT_VERSIONS_V1,
        identity_versions: IDENTITY_VERSIONS_V1,
        voting_versions: VOTING_VERSION_V1,
        token_versions: TOKEN_VERSIONS_V1,
        asset_lock_versions: DPP_ASSET_LOCK_VERSIONS_V1,
        methods: DPP_METHOD_VERSIONS_V1,
        factory_versions: DPP_FACTORY_VERSIONS_V1,
    },
    system_data_contracts: SYSTEM_DATA_CONTRACT_VERSIONS_V1,
    fee_version: FEE_VERSION1,
    system_limits: SYSTEM_LIMITS_V1,
    consensus: ConsensusVersions {
        tenderdash_consensus_version: 0,
    },
};
}

Now compare with PLATFORM_V14, the latest at the time of writing. By convention, each sub-constant slot that was bumped carries a trailing // changed: comment saying what changed. The protocol_version field is the snapshot's identity and is never annotated. One bumped slot in this snapshot, validation (DPP_VALIDATION_VERSIONS_V4 to V5), is missing its comment, which is exactly the omission the convention exists to prevent:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/version/v14.rs

pub const PLATFORM_V14: PlatformVersion = PlatformVersion {
    protocol_version: PROTOCOL_VERSION_14,
    drive: DRIVE_VERSION_V9, // changed: drive document method versions v4 (v2 index walkers, detect_ranked_mode slot)
    drive_abci: DriveAbciVersion {
        structs: DRIVE_ABCI_STRUCTURE_VERSIONS_V1,
        methods: DRIVE_ABCI_METHOD_VERSIONS_V10, // changed: records the per-block total credits history
        validation_and_processing: DRIVE_ABCI_VALIDATION_VERSIONS_V10, // changed: contested-index cross-check + refersTo validation
        withdrawal_constants: DRIVE_ABCI_WITHDRAWAL_CONSTANTS_V3, // changed: prune bound for the total credits history
        query: DRIVE_ABCI_QUERY_VERSIONS_V2, // changed: ranked + boolean-HAVING routing gate
        checkpoints: DRIVE_ABCI_CHECKPOINT_PARAMETERS_V1,
    },
    dpp: DPPVersion {
        costs: DPP_COSTS_VERSIONS_V1,
        validation: DPP_VALIDATION_VERSIONS_V5,
        state_transition_serialization_versions: STATE_TRANSITION_SERIALIZATION_VERSIONS_V3, // changed: documentIndexOnlyDelete joins the wire
        state_transition_conversion_versions: STATE_TRANSITION_CONVERSION_VERSIONS_V2,
        state_transition_method_versions: STATE_TRANSITION_METHOD_VERSIONS_V1,
        state_transitions: STATE_TRANSITION_VERSIONS_V3,
        contract_versions: CONTRACT_VERSIONS_V6, // changed: v3 document meta-schema (ranked, refersTo, requiredSince, timeRange)
        document_versions: DOCUMENT_VERSIONS_V4, // changed: document serialization format 3
        identity_versions: IDENTITY_VERSIONS_V1,
        voting_versions: VOTING_VERSION_V2,
        token_versions: TOKEN_VERSIONS_V2,
        asset_lock_versions: DPP_ASSET_LOCK_VERSIONS_V1,
        methods: DPP_METHOD_VERSIONS_V3, // changed: daily_withdrawal_limit v2
        factory_versions: DPP_FACTORY_VERSIONS_V1,
    },
    system_data_contracts: SYSTEM_DATA_CONTRACT_VERSIONS_V3, // changed: DashPay v2 profile payment address fields
    fee_version: FEE_VERSION2,
    system_limits: SYSTEM_LIMITS_V4, // changed: relative daily withdrawal limit + time-range overlap cap
    consensus: ConsensusVersions {
        tenderdash_consensus_version: 1,
    },
};
}

Notice how only some subsystem versions change between V1 and V14. The ABCI structure versions and checkpoint parameters are still at V1 because nothing in them ever changed. The ABCI method versions, on the other hand, went from V1 to V10 -- ten revisions of the block processing logic -- and the query versions from V1 to V3.

This is the power of the snapshot model: each subsystem version evolves at its own pace. A new protocol version does not require bumping everything. You only change the sub-constants that actually differ.

Two annotation conventions make a version file reviewable:

  • The file's doc comment is the changelog. v14.rs opens with a numbered list of every consensus change the version hosts, each with a paragraph on what it does and why it needed a protocol version. A change landing in the unreleased version adds an item there.
  • Every bumped slot gets a // changed: comment saying what the new sub-constant does differently. A reviewer reading PLATFORM_V14 top to bottom sees every behaviour difference from PLATFORM_V13 without opening another file.

What Belongs in the Snapshot

The snapshot holds three kinds of values, and the third is the one people forget:

  1. Method versions. FeatureVersion numbers that select a v0, v1, ... implementation. The next two chapters are about these.
  2. Format bounds. FeatureVersionBounds for serialized structures: which structure versions a node accepts and which one it writes.
  3. Protocol parameters. Plain numbers the code reads at runtime: size caps, per-block limits, penalty amounts, fee rates, retention windows, expiry heights. SystemLimits, FeeVersion, DriveAbciWithdrawalConstants, DriveAbciValidationConstants and PenaltyAmounts are all tables of these.

The rule for the third kind: a constant that might change in a future protocol version is declared in the version tables, not in an implementation file. Code reads it through platform_version. A const MAX_SOMETHING: u16 = 50; at the top of a Drive module is a value that cannot change without editing shipped code, which is exactly what the versioning system exists to avoid. A field on SystemLimits can change at a protocol-version boundary with the old value preserved for replay.

Two consequences:

  • Changing a number never creates a new method version. You change the table, and the method that reads the table keeps its version. See Non-Method Version Fields for the recipe.
  • Constants that genuinely cannot change stay in code: key and hash lengths, encoding widths, tree key bytes, anything whose change would be a new storage format rather than a new parameter. The test is whether a future protocol version could plausibly want a different value. If it could, it belongs in the tables from the start; moving it later means touching shipped code.

The Get Dispatch

The most important function on PlatformVersion is get:

#![allow(unused)]
fn main() {
impl PlatformVersion {
    pub fn get<'a>(version: ProtocolVersion) -> Result<&'a Self, PlatformVersionError> {
        if version > 0 {
            PLATFORM_VERSIONS.get(version as usize - 1).ok_or_else(|| {
                PlatformVersionError::UnknownVersionError(
                    format!("no platform version {version}")
                )
            })
        } else {
            Err(PlatformVersionError::UnknownVersionError(
                format!("no platform version {version}")
            ))
        }
    }
}
}

This is a simple array lookup. Protocol version 1 maps to index 0, version 14 to index 13. If the version number is out of range, you get a clear error. No hash maps, no runtime registration, no dynamic dispatch -- just a static array of compile-time constants.

There are also convenience methods:

#![allow(unused)]
fn main() {
impl PlatformVersion {
    pub fn first<'a>() -> &'a Self {
        PLATFORM_VERSIONS.first()
            .expect("expected to have a platform version")
    }

    pub fn latest<'a>() -> &'a Self {
        PLATFORM_VERSIONS.last()
            .expect("expected to have a platform version")
    }

    pub fn desired<'a>() -> &'a Self {
        DESIRED_PLATFORM_VERSION
    }
}
}

first() is used in tests that need to verify behavior under the initial protocol. latest() is the default for new code. desired() returns the version that nodes want to upgrade to -- it equals latest() during normal operation but could theoretically differ during a staged rollout.

Version-Aware Traits

The rs-platform-version crate also defines traits that thread the platform version through standard Rust conversion patterns:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/lib.rs

pub trait TryFromPlatformVersioned<T>: Sized {
    type Error;

    fn try_from_platform_versioned(
        value: T,
        platform_version: &PlatformVersion,
    ) -> Result<Self, Self::Error>;
}

pub trait DefaultForPlatformVersion: Sized {
    type Error;

    fn default_for_platform_version(
        platform_version: &PlatformVersion,
    ) -> Result<Self, Self::Error>;
}
}

These are the versioned equivalents of TryFrom and Default. When you convert a data structure, you pass the platform version so the implementation can pick the right serialization format, the right field set, or the right validation rules. There is also FromPlatformVersioned for infallible conversions, and blanket IntoPlatformVersioned implementations that mirror the standard library pattern.

Mock Versions for Testing

The version system supports a mock-versions feature flag for tests:

#![allow(unused)]
fn main() {
#[cfg(feature = "mock-versions")]
pub static PLATFORM_TEST_VERSIONS: OnceLock<Vec<PlatformVersion>> = OnceLock::new();
}

When this feature is enabled, PlatformVersion::get checks for a special bit in the version number. If set, it routes to the test version array instead of the production one. This lets tests create synthetic platform versions with specific behaviors without polluting the production constants:

#![allow(unused)]
fn main() {
#[cfg(feature = "mock-versions")]
{
    if version >> TEST_PROTOCOL_VERSION_SHIFT_BYTES > 0 {
        let test_version = version - (1 << TEST_PROTOCOL_VERSION_SHIFT_BYTES);
        let versions = PLATFORM_TEST_VERSIONS
            .get_or_init(|| vec![TEST_PLATFORM_V2, TEST_PLATFORM_V3]);
        return versions.get(test_version as usize - 2).ok_or(/* ... */);
    }
}
}

This is a clever design: tests can exercise version upgrade logic (like "what happens when we transition from test version 2 to test version 3?") without needing to create real protocol versions.

Why Immutable Snapshots?

You might wonder: why not use a mutable configuration object? Why not a HashMap<&str, u16> that maps method names to versions?

Three reasons:

  1. Determinism. A const value is baked into the binary at compile time. There is no way to accidentally modify it at runtime. Every node running the same binary with the same protocol version will use the exact same values.

  2. Exhaustiveness. Because the version struct has named fields for every subsystem, adding a new versioned method forces you to set its version in every platform version constant. The compiler will refuse to compile if you forget one. A hash map cannot give you this guarantee.

  3. Performance. Looking up a version number is a struct field access -- zero overhead at runtime. The entire version tree lives in static memory. No allocations, no lookups, no indirection.

The cost is verbosity. Each new platform version file is large and repetitive. But this is a deliberate trade-off: the system favors correctness and auditability over conciseness. When you read PLATFORM_V14, you can see every single version number in one place. There is no mystery about what version 14 means.

Rules

Do:

  • Always pass &PlatformVersion (or &DriveVersion, etc.) to functions that have versioned behavior. Never hardcode a version number at a call site.
  • Use PlatformVersion::latest() for tests of the current generation. Pin an explicit version with PlatformVersion::get(n) only for tests of a frozen, older generation, and use PlatformVersion::first() when you need the initial protocol behavior.
  • Create vN.rs once, when the first consensus change after version N-1 ships needs a home: copy the previous version file and change only the constants that differ. Every later change destined for version N amends that file: add a numbered item to its doc comment and a // changed: comment on each slot it bumps.
  • Declare any constant that might change in a future protocol version in the version tables (SystemLimits, FeeVersion, the *Constants structs) and read it through platform_version. Keep only genuinely invariant values as const items in implementation files.

Do not:

  • Never mutate platform version data at runtime. The constants are const for a reason.
  • Never edit a released vN.rs, or any table constant it references. Once a protocol version has run on the network its snapshot is part of the chain's history; a change there makes a fresh node replay history differently from every node that was there at the time.
  • Never add a new field to PlatformVersion without also updating every PLATFORM_V* constant. The compiler will enforce this, but be aware that the fix is updating fourteen files, not one.
  • Never use PlatformVersion::latest() in consensus-critical code paths. Always use the version from the current platform state, obtained via platform_state.current_platform_version(). The "latest" version is what the binary supports; the active version is what the network has agreed upon -- and they may differ during an upgrade window.

Feature Versions

The Problem: Granularity

The previous chapter showed how PlatformVersion is an immutable snapshot of the entire platform's behavior at a given protocol version. But a snapshot is only useful if it can describe behavior at a fine enough granularity.

Consider the Drive storage layer. It has dozens of grove operations, hundreds of document methods, contract methods, identity methods, and more. When you fix a bug in update_contract, you need to bump that one method's version without affecting insert_contract or prove_contract. The system needs a way to assign a version number to individual methods and then compose those numbers into larger subsystem snapshots.

This is where FeatureVersion and the nested version structs come in.

The FeatureVersion Type

At the very bottom of the version tree is a single type, defined in the external versioned-feature-core crate:

#![allow(unused)]
fn main() {
// versioned-feature-core/src/lib.rs

pub type FeatureVersion = u16;
pub type OptionalFeatureVersion = Option<u16>;
}

That is it. A FeatureVersion is a u16 -- a number that says "use version N of this particular function." The value 0 means "use the v0 implementation," 1 means "use v1," and so on.

OptionalFeatureVersion is Option<u16>. It represents a feature that did not exist in earlier protocol versions. When the value is None, the feature is not active -- calling it returns a VersionNotActive error. When it is Some(0), the feature exists and should use its v0 implementation.

There is also a bounds type for serialization format versions:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Default)]
pub struct FeatureVersionBounds {
    pub min_version: FeatureVersion,
    pub max_version: FeatureVersion,
    pub default_current_version: FeatureVersion,
}
}

This is used when a field can accept a range of versions -- for example, a data contract serialization format where the system can read versions 0 through 2 but writes version 2 by default.

Version Structs: The Middle of the Tree

Between the top-level PlatformVersion and the leaf-level FeatureVersion numbers sit dozens of intermediate structs. These structs group related method versions together, forming a hierarchy that mirrors the codebase's module structure.

Let us trace a path from the top down.

Level 1: PlatformVersion

#![allow(unused)]
fn main() {
pub struct PlatformVersion {
    pub protocol_version: ProtocolVersion,
    pub dpp: DPPVersion,
    pub drive: DriveVersion,
    pub drive_abci: DriveAbciVersion,
    pub consensus: ConsensusVersions,
    pub fee_version: FeeVersion,
    pub system_data_contracts: SystemDataContractVersions,
    pub system_limits: SystemLimits,
}
}

Level 2: DriveVersion

The drive field contains DriveVersion, which groups all storage layer versions:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/version/drive_versions/mod.rs

#[derive(Clone, Debug, Default)]
pub struct DriveVersion {
    pub structure: DriveStructureVersion,
    pub methods: DriveMethodVersions,
    pub grove_methods: DriveGroveMethodVersions,
    pub grove_version: GroveVersion,
}
}

Level 3: DriveMethodVersions

The methods field expands into every category of Drive operation:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Default)]
pub struct DriveMethodVersions {
    pub initialization: DriveInitializationMethodVersions,
    pub credit_pools: DriveCreditPoolMethodVersions,
    pub protocol_upgrade: DriveProtocolUpgradeVersions,
    pub prefunded_specialized_balances: DrivePrefundedSpecializedMethodVersions,
    pub balances: DriveBalancesMethodVersions,
    pub document: DriveDocumentMethodVersions,
    pub vote: DriveVoteMethodVersions,
    pub contract: DriveContractMethodVersions,
    pub fees: DriveFeesMethodVersions,
    pub estimated_costs: DriveEstimatedCostsMethodVersions,
    pub asset_lock: DriveAssetLockMethodVersions,
    pub verify: DriveVerifyMethodVersions,
    pub identity: DriveIdentityMethodVersions,
    pub token: DriveTokenMethodVersions,
    pub platform_system: DrivePlatformSystemMethodVersions,
    pub operations: DriveOperationsMethodVersion,
    pub batch_operations: DriveBatchOperationsMethodVersion,
    pub fetch: DriveFetchMethodVersions,
    pub prove: DriveProveMethodVersions,
    pub state_transitions: DriveStateTransitionMethodVersions,
    pub platform_state: DrivePlatformStateMethodVersions,
    pub group: DriveGroupMethodVersions,
    pub address_funds: DriveAddressFundsMethodVersions,
    pub saved_block_transactions: DriveSavedBlockTransactionsMethodVersions,
}
}

Level 4: Individual Method Categories

Each category struct contains FeatureVersion fields for individual methods. For example, the contract method versions:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Default)]
pub struct DriveContractMethodVersions {
    pub prove: DriveContractProveMethodVersions,
    pub apply: DriveContractApplyMethodVersions,
    pub insert: DriveContractInsertMethodVersions,
    pub update: DriveContractUpdateMethodVersions,
    pub costs: DriveContractCostsMethodVersions,
    pub get: DriveContractGetMethodVersions,
}

#[derive(Clone, Debug, Default)]
pub struct DriveContractUpdateMethodVersions {
    pub update_contract: FeatureVersion,
    pub update_description: FeatureVersion,
    pub update_keywords: FeatureVersion,
}
}

So the full path to read "which version of update_contract should I use?" is:

#![allow(unused)]
fn main() {
platform_version.drive.methods.contract.update.update_contract
}

That is a five-level deep field access, and it resolves to a plain u16.

The Grove Methods Branch

Let us trace a different path. The grove_methods field on DriveVersion holds versions for low-level GroveDB operations:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Default)]
pub struct DriveGroveMethodVersions {
    pub basic: DriveGroveBasicMethodVersions,
    pub batch: DriveGroveBatchMethodVersions,
    pub apply: DriveGroveApplyMethodVersions,
    pub costs: DriveGroveCostMethodVersions,
}
}

The basic struct is where individual grove operations live:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Default)]
pub struct DriveGroveBasicMethodVersions {
    pub grove_insert: FeatureVersion,
    pub grove_insert_empty_tree: FeatureVersion,
    pub grove_insert_if_not_exists: FeatureVersion,
    pub grove_clear: FeatureVersion,
    pub grove_delete: FeatureVersion,
    pub grove_get_raw: FeatureVersion,
    pub grove_get_raw_optional: FeatureVersion,
    pub grove_get: FeatureVersion,
    pub grove_get_path_query: FeatureVersion,
    pub grove_get_proved_path_query: FeatureVersion,
    pub grove_get_sum_tree_total_value: FeatureVersion,
    pub grove_has_raw: FeatureVersion,
    // ... and many more
}
}

So the path for grove_get_raw is:

#![allow(unused)]
fn main() {
drive_version.grove_methods.basic.grove_get_raw
}

Notice something: grove operations take a &DriveVersion rather than &PlatformVersion. This is a minor optimization -- when you are deep in the Drive layer, you only need the drive-specific version numbers, not the entire platform snapshot. The caller extracts &platform_version.drive once and passes it down.

The DPP Branch

The Dash Platform Protocol has its own deep tree. DPPVersion contains fourteen sub-version structs:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Default)]
pub struct DPPVersion {
    pub costs: DPPCostsVersions,
    pub validation: DPPValidationVersions,
    pub state_transition_serialization_versions: DPPStateTransitionSerializationVersions,
    pub state_transition_conversion_versions: DPPStateTransitionConversionVersions,
    pub state_transition_method_versions: DPPStateTransitionMethodVersions,
    pub state_transitions: DPPStateTransitionVersions,
    pub contract_versions: DPPContractVersions,
    pub document_versions: DPPDocumentVersions,
    pub identity_versions: DPPIdentityVersions,
    pub voting_versions: DPPVotingVersions,
    pub token_versions: DPPTokenVersions,
    pub asset_lock_versions: DPPAssetLockVersions,
    pub methods: DPPMethodVersions,
    pub factory_versions: DPPFactoryVersions,
}
}

And those go deeper. For example, DPPContractVersions contains not just FeatureVersion values but also FeatureVersionBounds and further nesting:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Default)]
pub struct DPPContractVersions {
    pub max_serialized_size: u32,
    pub contract_serialization_version: FeatureVersionBounds,
    pub contract_structure_version: FeatureVersion,
    pub created_data_contract_structure: FeatureVersion,
    pub config: FeatureVersionBounds,
    pub methods: DataContractMethodVersions,
    pub document_type_versions: DocumentTypeVersions,
    pub token_versions: TokenVersions,
}
}

Notice max_serialized_size: u32. Not every field is a FeatureVersion. Some are configuration values -- limits, thresholds, constants -- that change between protocol versions. The version struct is flexible enough to hold both "which implementation to use" and "what parameters to use."

The Drive ABCI Branch

The DriveAbciVersion struct covers the application blockchain interface -- the layer that processes blocks, validates state transitions, and handles protocol upgrades:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Default)]
pub struct DriveAbciVersion {
    pub structs: DriveAbciStructureVersions,
    pub methods: DriveAbciMethodVersions,
    pub validation_and_processing: DriveAbciValidationVersions,
    pub withdrawal_constants: DriveAbciWithdrawalConstants,
    pub query: DriveAbciQueryVersions,
    pub checkpoints: DriveAbciCheckpointParameters,
}
}

The validation_and_processing field is where state transition validation versions live. This is where OptionalFeatureVersion becomes important:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, Default)]
pub struct DriveAbciStateTransitionValidationVersion {
    pub basic_structure: OptionalFeatureVersion,
    pub advanced_structure: OptionalFeatureVersion,
    pub identity_signatures: OptionalFeatureVersion,
    pub nonce: OptionalFeatureVersion,
    pub state: FeatureVersion,
    pub transform_into_action: FeatureVersion,
}
}

basic_structure is OptionalFeatureVersion -- in some protocol versions, basic structure validation may not exist for a particular state transition. The dispatch code handles this with a three-arm match:

#![allow(unused)]
fn main() {
match platform_version
    .drive_abci
    .validation_and_processing
    .state_transitions
    .identity_create_state_transition
    .basic_structure
{
    Some(0) => self.validate_basic_structure_v0(platform_version),
    Some(version) => Err(Error::Execution(
        ExecutionError::UnknownVersionMismatch { /* ... */ }
    )),
    None => Err(Error::Execution(
        ExecutionError::VersionNotActive { /* ... */ }
    )),
}
}

Compare with state and transform_into_action which are plain FeatureVersion -- those validations always exist, so there is no None arm.

Non-Method Version Fields

Some version structs contain values that are not method versions at all, but protocol parameters. SystemLimits is the main one:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/version/system_limits/mod.rs

#[derive(Clone, Debug, Default)]
pub struct SystemLimits {
    pub estimated_contract_max_serialized_size: u16,
    pub max_field_value_size: u32,
    /// `None` preserves the behavior of protocol versions that predate this limit.
    pub max_document_value_depth: Option<u16>,
    pub max_state_transition_size: u64,
    pub max_transitions_in_documents_batch: u16,
    pub withdrawal_transactions_per_block_limit: u16,
    pub retry_signing_expired_withdrawal_documents_per_block_limit: u16,
    pub max_withdrawal_amount: u64,
    /// `None` for the protocol versions that predate the relative rule.
    pub daily_withdrawal_limit_percent: Option<u8>,
    pub core_credit_pool_unlock_limit_percent: Option<u8>,
    pub core_credit_pool_unlock_limit_floor: Option<u64>,
    pub core_credit_pool_window_blocks: Option<u32>,
    pub regtest_core_credit_pool_window_blocks: Option<u32>,
    pub min_withdrawal_amount: u64,
    pub max_contract_group_size: u16,
    pub max_token_redemption_cycles: u32,
    pub max_shielded_transition_actions: u16,
    pub max_time_range_overlap_factor: Option<u64>,
}
}

There are four SYSTEM_LIMITS_V* constants, one for each protocol version at which a limit changed. The Option fields show the idiom for a parameter that did not exist before some version: None in the tables of the versions that predate the rule, Some(value) from the version that introduced it. It is the parameter-shaped twin of OptionalFeatureVersion.

The same shape recurs wherever a subsystem owns tunables:

#![allow(unused)]
fn main() {
// drive_abci_versions/drive_abci_withdrawal_constants/mod.rs
pub struct DriveAbciWithdrawalConstants {
    pub core_expiration_blocks: u32,
    pub cleanup_expired_locks_of_withdrawal_amounts_limit: u16,
    pub total_credits_history_prune_limit: u16,
    pub core_blocks_scanned_per_block_limit: u16,
}

// drive_abci_versions/drive_abci_validation_versions/mod.rs
pub struct PenaltyAmounts {
    pub identity_id_not_correct: u64,
    pub unique_key_already_present: u64,
    // ...
    pub shielded_proof_verification_failure: u64,
}

pub struct DriveAbciCoreChainLockMethodVersionsAndConstants {
    pub choose_quorum: FeatureVersion,
    pub verify_chain_lock: FeatureVersion,
    // ...
    pub recent_block_count_amount: u32,
}
}

The chain lock struct mixes method versions (choose_quorum: FeatureVersion) with protocol constants (recent_block_count_amount: u32). This is perfectly fine -- the version snapshot captures all protocol-specific values, whether they control dispatch or configure behavior. Fee rates follow the same idea one level up: FeeVersion is a table of numbers, and a fee change is a new named FEE_VERSION* schedule referenced from the platform version, never an inline override inside PLATFORM_V*.

Changing a number

Because parameters live in tables, changing one is a table edit and not a method version:

  1. If the field does not exist yet, add it to the struct with a doc comment naming the method version that reads it.
  2. Backfill every shipped table constant (and the mock tables under version/mocks/) with the old value, so released protocol versions keep their behaviour. The compiler forces this step.
  3. Add the next table constant for the unreleased protocol version with the new value, and point that version's PLATFORM_V* at it. If the unreleased version already introduced a new table constant, edit that one in place instead.
  4. Make the method read the field through platform_version. The method keeps its version number: shipped versions read their old value from their own table, so their behaviour is unchanged.

Do not create a vN method module whose entire body returns the new constant. That hides a protocol parameter in an implementation file, which is the situation the tables exist to prevent, and it leaves a dead module behind every time the number moves. A new method version is warranted only when the logic changes. daily_withdrawal_limit in rs-dpp is the reference case: v0 derives the limit from the current total credits, v2 reads daily_withdrawal_limit_percent from SystemLimits. Raising the percentage later is a SYSTEM_LIMITS_V5, not a v3.

How Subsystem Version Constants Compose

Each subsystem version constant (like DRIVE_VERSION_V1) is assembled from smaller constants:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/version/drive_versions/v1.rs

pub const DRIVE_VERSION_V1: DriveVersion = DriveVersion {
    structure: DRIVE_STRUCTURE_V1,
    methods: DriveMethodVersions {
        initialization: DriveInitializationMethodVersions {
            create_initial_state_structure: 0,
        },
        credit_pools: CREDIT_POOL_METHOD_VERSIONS_V1,
        protocol_upgrade: DriveProtocolUpgradeVersions {
            clear_version_information: 0,
            fetch_versions_with_counter: 0,
            // ...
        },
        balances: DriveBalancesMethodVersions {
            add_to_system_credits: 0,
            remove_from_system_credits: 0,
            calculate_total_credits_balance: 0,
            // ...
        },
        contract: DRIVE_CONTRACT_METHOD_VERSIONS_V1,
        // ...
    },
    grove_methods: DRIVE_GROVE_METHOD_VERSIONS_V1,
    grove_version: GROVE_V1,
};
}

Notice the mix of inline construction and named constants. Small structs like DriveProtocolUpgradeVersions are often written inline because all their fields are 0 in every version. Larger, frequently-changing structs like DRIVE_CONTRACT_METHOD_VERSIONS_V1 get their own named constant so they can be reused or overridden in later versions.

When DRIVE_VERSION_V9 (used in PLATFORM_V14) needs new document method versions, it references DRIVE_DOCUMENT_METHOD_VERSIONS_V4 where V8 referenced V3, and annotates the slot:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/version/drive_versions/v9.rs

pub const DRIVE_VERSION_V9: DriveVersion = DriveVersion {
    structure: DRIVE_STRUCTURE_V1,
    methods: DriveMethodVersions {
        // ...
        document: DRIVE_DOCUMENT_METHOD_VERSIONS_V4, // changed in v9: v2 index walkers + v1 update walker
        contract: DRIVE_CONTRACT_METHOD_VERSIONS_V3, // changed in v8: count-tree-aware contract-insertion cost estimation
        // ...
        verify: DRIVE_VERIFY_METHOD_VERSIONS_V2, // changed in v8: compacted address-balance proof envelope
        identity: DRIVE_IDENTITY_METHOD_VERSIONS_V2, // changed in v9: v1 withdrawal-by-transaction-index query builder
        // ...
    },
    grove_methods: DRIVE_GROVE_METHOD_VERSIONS_V1,
    // changed in v9: GROVE_V4 activates the indexed-tree batch cleanup
    grove_version: GROVE_V4,
};
}

The unchanged parts reference the same constants they always did. Only the parts that actually changed get new constants. grove_version is how GroveDB behaviour is versioned from Platform's side: a new DRIVE_VERSION_V* points at a new GROVE_V*, and every GroveDB call receives it.

Two ways to write a new table constant

Older table files are full copies of the previous version with the changed slots edited. Newer ones use struct update syntax, so that the file is the diff:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/version/drive_abci_versions/drive_abci_query_versions/v2.rs

/// Differs from v1 in two slots.
/// `document_query_helpers.compute_aggregate_mode_and_check_limit` is 2
/// rather than 0: the ranked and boolean-`HAVING` routing gate. ...
pub const DRIVE_ABCI_QUERY_VERSIONS_V2: DriveAbciQueryVersions = DriveAbciQueryVersions {
    document_query_helpers: DriveAbciDocumentQueryHelperVersions {
        compute_aggregate_mode_and_check_limit: 2,
    },
    data_contract_query_helpers: DriveAbciDataContractQueryHelperVersions {
        latest_versions_read: 1,
    },
    ..DRIVE_ABCI_QUERY_VERSIONS_V1
};
}

It nests, so one changed leaf deep in a table still reads as one line:

#![allow(unused)]
fn main() {
// packages/rs-platform-version/src/version/dpp_versions/dpp_validation_versions/v5.rs

pub const DPP_VALIDATION_VERSIONS_V5: DPPValidationVersions = DPPValidationVersions {
    document_type: DocumentTypeValidationVersions {
        validate_update: 1,
        ..DPP_VALIDATION_VERSIONS_V4.document_type
    },
    ..DPP_VALIDATION_VERSIONS_V4
};
}

Use the struct update form for new table constants. Either way, the doc comment on the constant names the slots that differ from the previous table and why, in the same spirit as the // changed: comments on PLATFORM_V*.

When to create a new table constant

Table constants mark the boundaries between released protocol versions. Create DRIVE_ABCI_QUERY_VERSIONS_V4 when the constant you would otherwise edit is referenced by a released PLATFORM_V*. If the unreleased protocol version already introduced a new constant for that table, a second feature landing in the same protocol version amends it in place. Otherwise the crate accumulates table versions that no protocol version references, and the numbering stops saying anything about what shipped when.

The File Layout

The version structs follow a consistent directory layout in packages/rs-platform-version/src/version/:

version/
  mod.rs                        # ProtocolVersion type alias, LATEST_VERSION, module declarations
  protocol_version.rs           # PlatformVersion struct, PLATFORM_VERSIONS array, get()
  consensus_versions.rs         # ConsensusVersions (Tenderdash consensus version)
  feature_initial_protocol_versions.rs  # first protocol version of whole new transition kinds
  v1.rs .. v14.rs               # PLATFORM_V* snapshot constants, one per protocol version
  drive_versions/
    mod.rs                      # DriveVersion, DriveMethodVersions, etc.
    v1.rs .. v9.rs              # DRIVE_VERSION_V* constants
    drive_grove_method_versions/
      mod.rs                    # DriveGroveMethodVersions struct
      v1.rs                     # DRIVE_GROVE_METHOD_VERSIONS_V1
    drive_contract_method_versions/
      mod.rs                    # DriveContractMethodVersions struct
      v1.rs .. v3.rs            # versioned constants
    drive_document_method_versions/
      mod.rs
      v1.rs .. v4.rs
    drive_verify_method_versions/
      mod.rs
      v1.rs, v2.rs
    ...
  drive_abci_versions/
    mod.rs                      # DriveAbciVersion struct
    drive_abci_method_versions/
      mod.rs                    # DriveAbciMethodVersions and sub-structs
      v1.rs .. v10.rs           # versioned constants
    drive_abci_validation_versions/
      mod.rs                    # DriveAbciValidationVersions, PenaltyAmounts, validation constants
      v1.rs .. v10.rs
    drive_abci_query_versions/
      mod.rs
      v0.rs .. v2.rs
    drive_abci_withdrawal_constants/
      mod.rs                    # DriveAbciWithdrawalConstants (parameters, not method versions)
      v1.rs .. v3.rs
    drive_abci_structure_versions/      v1.rs
    drive_abci_checkpoint_parameters/   v1.rs
  dpp_versions/
    mod.rs                      # DPPVersion struct
    dpp_contract_versions/
      mod.rs                    # DPPContractVersions struct
      v1.rs .. v6.rs
    dpp_validation_versions/
      mod.rs
      v1.rs .. v5.rs
    ...
  fee/
    mod.rs                      # FeeVersion struct, FEE_VERSIONS
    v1.rs, v2.rs                # FEE_VERSION* schedules
    storage/, signature/, processing/, ...   # per-group tables, each with its own v*.rs
  system_limits/
    mod.rs                      # SystemLimits struct
    v1.rs .. v4.rs
  system_data_contract_versions/
    mod.rs
    v1.rs .. v3.rs
  mocks/
    v2_test.rs, v3_test.rs      # TEST_PLATFORM_V* behind the mock-versions feature

The pattern is: mod.rs defines the struct, and v*.rs files define the concrete constants. The struct definition is the schema. The version files are the data. Note that each table's v* numbering is its own: DRIVE_VERSION_V9 is what PLATFORM_V14 uses, and SYSTEM_LIMITS_V4 likewise. The number says how many times that table has changed, not which protocol version it belongs to.

Rules

Do:

  • When adding a new method to Drive, DPP, or Drive ABCI, add a corresponding FeatureVersion field to the appropriate version struct. Then set its value in every v*.rs constant -- the compiler will force you.
  • Use OptionalFeatureVersion for features that are being introduced in a non-initial protocol version. Set them to None in earlier versions and Some(0) in the version that introduces the feature. Use an Option field the same way for a parameter that did not exist before some version.
  • Declare every protocol parameter that might change (limits, penalties, retention windows, fee rates) as a table field and read it through platform_version. Change a number by adding a table version, not a method version.
  • Write new table constants with struct update syntax (..PREVIOUS) and a doc comment naming the slots that differ and why.
  • Group related methods into their own sub-struct when the parent struct grows too large. Follow the existing naming pattern: Drive<Category>MethodVersions.

Do not:

  • Never use a raw u16 where you mean FeatureVersion. The type alias exists for readability and future-proofing -- if we ever need to change the underlying type, the alias is the single point of change.
  • Never put runtime-computed values into a version struct. Every field must be a compile-time constant. This is what makes the snapshot deterministic.
  • Never reuse a version constant with different semantics. If DRIVE_CONTRACT_METHOD_VERSIONS_V1 means something, creating a V2 that changes one field is correct. Silently modifying V1 is not -- it would change the behavior of every platform version that references it.
  • Never create a new table constant for a second change landing in the same unreleased protocol version. Table versions mark released boundaries; amend the unreleased version's own constant in place.
  • Never leave a const in an implementation file that a future protocol version might want to change. Move it to the tables the first time it is touched.

Versioned Dispatch

The Problem: Running the Right Code

The previous two chapters explained what gets versioned (every method in the platform) and how version numbers are stored (nested structs inside an immutable PlatformVersion snapshot). This chapter covers the most important part: how those version numbers actually select which code runs.

The core idea is simple. Every versioned function has a dispatch method that reads a FeatureVersion value and calls the corresponding implementation. But the way this dispatch is organized across files, the error handling conventions, and the step-by-step process of adding a new version -- these are the details that make the pattern work at scale.

The Canonical Match Pattern

Here is the most common pattern in the codebase. This is from packages/rs-drive/src/util/grove_operations/grove_get_raw/mod.rs:

#![allow(unused)]
fn main() {
impl Drive {
    pub fn grove_get_raw<B: AsRef<[u8]>>(
        &self,
        path: SubtreePath<'_, B>,
        key: &[u8],
        direct_query_type: DirectQueryType,
        transaction: TransactionArg,
        drive_operations: &mut Vec<LowLevelDriveOperation>,
        drive_version: &DriveVersion,
    ) -> Result<Option<Element>, Error> {
        match drive_version.grove_methods.basic.grove_get_raw {
            0 => self.grove_get_raw_v0(
                path,
                key,
                direct_query_type,
                transaction,
                drive_operations,
                drive_version,
            ),
            version => Err(Error::Drive(DriveError::UnknownVersionMismatch {
                method: "grove_get_raw".to_string(),
                known_versions: vec![0],
                received: version,
            })),
        }
    }
}
}

Let us break down what is happening:

  1. The public method (grove_get_raw) is the entry point. It takes all the business parameters plus a version reference (drive_version: &DriveVersion).

  2. The version lookup reads the specific FeatureVersion for this method: drive_version.grove_methods.basic.grove_get_raw. This resolves to a u16.

  3. The match dispatches to the right implementation. Version 0 calls grove_get_raw_v0. The catch-all arm (version =>) returns an error.

  4. The error (UnknownVersionMismatch) includes the method name, the list of known versions, and the version that was actually received. This makes debugging version mismatches trivial.

This pattern appears hundreds of times across the codebase. It is the fundamental building block of versioned execution.

Multiple Versions

When a method has been revised, the match grows. Here is update_contract from packages/rs-drive/src/drive/contract/update/update_contract/mod.rs:

#![allow(unused)]
fn main() {
impl Drive {
    pub fn update_contract(
        &self,
        contract: &DataContract,
        block_info: BlockInfo,
        apply: bool,
        transaction: TransactionArg,
        platform_version: &PlatformVersion,
        previous_fee_versions: Option<&CachedEpochIndexFeeVersions>,
    ) -> Result<FeeResult, Error> {
        match platform_version
            .drive
            .methods
            .contract
            .update
            .update_contract
        {
            0 => self.update_contract_v0(
                contract, block_info, apply,
                transaction, platform_version, previous_fee_versions,
            ),
            1 => self.update_contract_v1(
                contract, block_info, apply,
                transaction, platform_version, previous_fee_versions,
            ),
            version => Err(Error::Drive(DriveError::UnknownVersionMismatch {
                method: "update_contract".to_string(),
                known_versions: vec![0, 1],
                received: version,
            })),
        }
    }
}
}

The structure is identical. The only differences are: there are now two known versions (0 and 1), and the known_versions vector in the error arm lists both. When a node running protocol version 1 processes a block, the version number is 0 and update_contract_v0 runs. When the network upgrades and the version number becomes 1, update_contract_v1 runs instead.

Both v0 and v1 implementations coexist in the binary. Old code is never deleted (at least not until a version is permanently retired from the network). This is critical for replaying historical blocks -- a node syncing from genesis needs to execute v0 for early blocks and v1 for later ones.

OptionalFeatureVersion Dispatch

For features introduced after the initial protocol version, the dispatch handles a None case:

#![allow(unused)]
fn main() {
// From identity_create/mod.rs

match platform_version
    .drive_abci
    .validation_and_processing
    .state_transitions
    .identity_create_state_transition
    .basic_structure
{
    Some(0) => {
        self.validate_basic_structure_v0(platform_version)
    }
    Some(version) => Err(Error::Execution(
        ExecutionError::UnknownVersionMismatch {
            method: "identity create transition: validate_basic_structure"
                .to_string(),
            known_versions: vec![0],
            received: version,
        }
    )),
    None => Err(Error::Execution(
        ExecutionError::VersionNotActive {
            method: "identity create transition: validate_basic_structure"
                .to_string(),
            known_versions: vec![0],
        }
    )),
}
}

Three arms instead of two:

  • Some(0) -- the feature exists, use v0.
  • Some(version) -- the feature exists but the version is unrecognized.
  • None -- the feature does not exist in this protocol version.

The VersionNotActive error is different from UnknownVersionMismatch. It means "this feature is legitimately not available," not "something went wrong." This distinction matters for callers that need to handle graceful degradation.

The Directory Convention

Versioned methods follow a strict directory layout. Let us use grove_get_raw as the example:

packages/rs-drive/src/util/grove_operations/
  grove_get_raw/
    mod.rs          # dispatch method (the match statement)
    v0/
      mod.rs        # grove_get_raw_v0 implementation

The dispatch method lives in grove_get_raw/mod.rs. Each implementation version gets its own subdirectory: v0/mod.rs, v1/mod.rs, etc. The dispatch file declares the version modules:

#![allow(unused)]
fn main() {
// grove_get_raw/mod.rs
mod v0;
}

And each version module provides the actual implementation as a method on Drive:

#![allow(unused)]
fn main() {
// grove_get_raw/v0/mod.rs

impl Drive {
    pub(super) fn grove_get_raw_v0<B: AsRef<[u8]>>(
        &self,
        path: SubtreePath<'_, B>,
        key: &[u8],
        direct_query_type: DirectQueryType,
        transaction: TransactionArg,
        drive_operations: &mut Vec<LowLevelDriveOperation>,
        drive_version: &DriveVersion,
    ) -> Result<Option<Element>, Error> {
        // actual implementation
        match direct_query_type {
            DirectQueryType::StatelessDirectQuery { /* ... */ } => {
                // estimate costs
            }
            DirectQueryType::StatefulDirectQuery => {
                let CostContext { value, cost } =
                    self.grove.get_raw(path, key, transaction,
                                       &drive_version.grove_version);
                drive_operations.push(CalculatedCostOperation(cost));
                Ok(Some(value.map_err(Error::from)?))
            }
        }
    }
}
}

Notice the visibility: pub(super). The v0 function is only visible to its parent module (the dispatch file). External code calls the public dispatch method, never the versioned implementation directly.

The layout is the versioning contract made physical, and three rules follow from it:

  • One directory per generation. A behaviour change to a versioned method is a new v1/ (or v2/, ...) directory with its own mod.rs, plus a new match arm. An edit inside a shipped v0/ is allowed only when it cannot modify consensus at any protocol version that selects v0/, because the code it adds is unreachable there by construction or its output is identical; the edited lines say why, and the pull request description carries an "In-place changes to shipped generations" section (see the coding conventions). An if platform_version.protocol_version >= 14 inside v0/ is not that: it is a runtime check the reader has to trust, so it gets a generation.
  • Inside a generation, a capability is a constant fact, not a check. If v1 admits a new keyword, v1 admits it unconditionally (Index::try_from_value_map(map, true)). The decision of whether the keyword is allowed was made by the table that selected v1. Old generations cannot reach the new path at all, so there is nothing for them to check.
  • Start the new generation as a copy of the old one. Duplicate v0/ into v1/, rename the function, make the change, and move the tests that exercise the new behaviour across. Duplication between generations is the accepted cost; a shared helper with a flag is the thing it replaces.

When new behaviour lives in a helper reached from several generations (a value walker, a property-reference resolver), the helper itself becomes a versioned method: an OptionalFeatureVersion slot in the tables (None for versions that predate the feature, Some(0) to dispatch to _v0), and the helper takes &PlatformVersion. A bool on a shared context struct is the wrong shape, because it moves the version decision from the tables to whoever set the flag. Per-generation grammar constants may live on a generation-owned struct; feature gates reachable from more than one generation may not.

Tests live with the generation they test: a #[cfg(test)] mod tests at the bottom of vN/mod.rs, or vN/tests/ when it grows. The dispatcher's mod.rs may carry end-to-end tests that need every generation.

For state transitions in Drive ABCI, the same pattern applies but with trait implementations:

packages/rs-drive-abci/src/execution/validation/state_transition/
  state_transitions/
    identity_create/
      mod.rs                  # dispatch traits and match statements
      basic_structure/
        mod.rs                # just declares v0
        v0/
          mod.rs              # BasicStructureValidationV0 implementation
      advanced_structure/
        mod.rs
        v0/
          mod.rs
      state/
        mod.rs
        v0/
          mod.rs

The Error Types

There are two UnknownVersionMismatch error variants in the codebase -- one for Drive and one for Drive ABCI -- but they have the same shape:

#![allow(unused)]
fn main() {
// packages/rs-drive/src/error/drive.rs

#[derive(Debug, thiserror::Error)]
pub enum DriveError {
    #[error("drive unknown version on {method}, received: {received}")]
    UnknownVersionMismatch {
        method: String,
        known_versions: Vec<FeatureVersion>,
        received: FeatureVersion,
    },

    #[error("{method} not active for drive version")]
    VersionNotActive {
        method: String,
        known_versions: Vec<FeatureVersion>,
    },
    // ...
}
}
#![allow(unused)]
fn main() {
// packages/rs-drive-abci/src/error/execution.rs

#[derive(Debug, thiserror::Error)]
pub enum ExecutionError {
    #[error("platform unknown version on {method}, received: {received}")]
    UnknownVersionMismatch {
        method: String,
        known_versions: Vec<FeatureVersion>,
        received: FeatureVersion,
    },

    #[error("{method} not active for drive version")]
    VersionNotActive {
        method: String,
        known_versions: Vec<FeatureVersion>,
    },
    // ...
}
}

Both carry three pieces of information:

  • method: A human-readable name identifying which dispatch failed.
  • known_versions: The versions this binary knows how to handle.
  • received: The version number that was actually in the platform version.

This makes the error message self-diagnosing. If you see "drive unknown version on update_contract, received: 2, known versions: [0, 1]", you immediately know that the binary is too old to handle the active protocol version.

How to Add a New Version: Step by Step

Let us walk through the exact steps to add a v1 implementation of a method that currently only has v0. We will use a fictional example: my_grove_operation.

Step 1: Write the new implementation

Create the v1 module:

my_grove_operation/
  mod.rs          # existing dispatch
  v0/
    mod.rs        # existing v0
  v1/
    mod.rs        # NEW: v1 implementation
#![allow(unused)]
fn main() {
// my_grove_operation/v1/mod.rs

impl Drive {
    pub(super) fn my_grove_operation_v1(
        &self,
        // same signature as v0, or possibly different
    ) -> Result<SomeResult, Error> {
        // new implementation with bug fix or feature
    }
}
}

Step 2: Update the dispatch

In my_grove_operation/mod.rs, declare the new module and add the match arm:

#![allow(unused)]
fn main() {
mod v0;
mod v1;  // NEW

impl Drive {
    pub fn my_grove_operation(
        &self,
        // ...
        drive_version: &DriveVersion,
    ) -> Result<SomeResult, Error> {
        match drive_version.grove_methods.basic.my_grove_operation {
            0 => self.my_grove_operation_v0(/* ... */),
            1 => self.my_grove_operation_v1(/* ... */),  // NEW
            version => Err(Error::Drive(DriveError::UnknownVersionMismatch {
                method: "my_grove_operation".to_string(),
                known_versions: vec![0, 1],  // UPDATED
                received: version,
            })),
        }
    }
}
}

Step 3: Bump the slot in the unreleased protocol version's tables

The new arm is dead until a table selects it. Which table you edit depends on whether the unreleased protocol version already owns one.

At the time of writing the latest released version is 13 and version 14 is in development. PLATFORM_V14 already references DRIVE_VERSION_V9, which was created for version 14, so a version-14 change edits v9.rs in place:

#![allow(unused)]
fn main() {
// drive_versions/v9.rs  (existing file, amended)

pub const DRIVE_VERSION_V9: DriveVersion = DriveVersion {
    // ...
    grove_methods: DRIVE_GROVE_METHOD_VERSIONS_V2, // changed in v9: my_grove_operation v1
    // ...
};
}

DRIVE_GROVE_METHOD_VERSIONS_V1 is referenced by released versions, so it cannot be edited. Create the next one with struct update syntax:

#![allow(unused)]
fn main() {
// drive_grove_method_versions/v2.rs  (NEW file)

/// Differs from v1 in one slot: `basic.my_grove_operation` is 1 rather
/// than 0. v1 of the operation <what changed and why>.
pub const DRIVE_GROVE_METHOD_VERSIONS_V2: DriveGroveMethodVersions =
    DriveGroveMethodVersions {
        basic: DriveGroveBasicMethodVersions {
            my_grove_operation: 1,
            ..DRIVE_GROVE_METHOD_VERSIONS_V1.basic
        },
        ..DRIVE_GROVE_METHOD_VERSIONS_V1
    };
}

Had DRIVE_VERSION_V9 also been shared with a released version, the same logic would apply one level up: a new drive_versions/v10.rs pointing at DRIVE_GROVE_METHOD_VERSIONS_V2, and PLATFORM_V14 pointing at DRIVE_VERSION_V10. The rule at every level is the same: edit in place if the constant belongs only to the unreleased version; create the next constant if a released version references it.

Step 4: Annotate the platform version file

Add or extend the // changed: comment on the affected slot of the unreleased PLATFORM_V*, and add a numbered item to the file's doc comment describing the consensus change. That doc comment is the release changelog for the protocol version.

Step 5: If this is the first change after a release, create the version

Only the first consensus change after a release creates a new protocol version. If version 14 had already shipped, the change above would start version 15:

#![allow(unused)]
fn main() {
// version/v15.rs  (NEW file, copied from v14.rs)

pub const PROTOCOL_VERSION_15: ProtocolVersion = 15;

/// v15 hosts one consensus change so far:
///
/// 1. **my_grove_operation v1**: ...
pub const PLATFORM_V15: PlatformVersion = PlatformVersion {
    protocol_version: PROTOCOL_VERSION_15,
    drive: DRIVE_VERSION_V10, // changed: my_grove_operation v1
    // ... everything else unchanged from V14
};
}

Then register it:

#![allow(unused)]
fn main() {
// version/mod.rs
pub mod v15;
pub const LATEST_VERSION: ProtocolVersion = PROTOCOL_VERSION_15;

// version/protocol_version.rs
pub const PLATFORM_VERSIONS: &[PlatformVersion] = &[
    PLATFORM_V1,
    // ...
    PLATFORM_V14,
    PLATFORM_V15,  // NEW
];

pub const LATEST_PLATFORM_VERSION: &PlatformVersion = &PLATFORM_V15;
}

A test in system_limits/mod.rs asserts that PLATFORM_VERSIONS.len() equals LATEST_VERSION, so a version that is declared but not registered fails the test run rather than silently resolving to the previous one. Every subsequent change destined for version 15 amends v15.rs and the constants it introduced, as in step 3.

Step 6: Write tests

Test both generations through the dispatcher, and put each test with the generation it exercises:

#![allow(unused)]
fn main() {
// my_grove_operation/v1/mod.rs
#[cfg(test)]
mod tests {
    #[test]
    fn should_apply_new_behaviour() {
        let platform_version = PlatformVersion::latest();
        // ... drive.my_grove_operation(..., &platform_version.drive)
    }
}

// my_grove_operation/v0/mod.rs
#[cfg(test)]
mod tests {
    #[test]
    fn should_keep_old_behaviour() {
        // frozen generation: pin the last protocol version that selected it
        let platform_version = PlatformVersion::get(13).expect("known version");
        // ...
    }
}
}

The current generation tests against PlatformVersion::latest() so it keeps tracking the tip; the moment v1 is introduced is the moment v0's tests get pinned to an explicit version. Do not write a test that merely asserts the table slot's value (assert_eq!(PLATFORM_V14.drive.grove_methods.basic.my_grove_operation, 1)); it restates the literal and cannot fail without the edit being deliberate. A behaviour test that runs both versions through the dispatcher pins the gate meaningfully.

This is a lot of steps, but each one is mechanical and the compiler guides you through most of it. If you add a field to a version struct and forget to set it in one of the fourteen platform version constants, the build fails.

Passing Version References

A subtle but important convention is which version reference a function receives. There are three patterns:

&PlatformVersion -- used by high-level code that might need any part of the version tree. State transition processing, block execution, and similar entry points take this.

&DriveVersion -- used by mid-level Drive code that only needs drive- specific versions. The caller extracts &platform_version.drive once.

&GroveVersion -- used by the lowest-level GroveDB operations. Extracted from &drive_version.grove_version.

This layering avoids passing the entire PlatformVersion into the deepest functions. It also makes the dependency explicit: a function taking &DriveVersion cannot accidentally use a DPP version number.

The Version Flow in Block Processing

Here is how the version flows through a real execution path:

Block arrives from Tenderdash
    |
    v
PlatformState has the current protocol_version (e.g., 14)
    |
    v
PlatformVersion::get(14) -> &PLATFORM_V14
    |
    v
process_raw_state_transitions(&platform_version)
    |
    v
validate_state_for_identity_create_transition()
    reads: platform_version.drive_abci.validation_and_processing
           .state_transitions.identity_create_state_transition.state
    dispatches to: validate_state_v0()
    |
    v
drive.update_contract(&platform_version)
    reads: platform_version.drive.methods.contract.update.update_contract
    dispatches to: update_contract_v1()
    |
    v
drive.grove_get_raw(&platform_version.drive)
    reads: drive_version.grove_methods.basic.grove_get_raw
    dispatches to: grove_get_raw_v0()

The protocol version number enters at the top and the correct implementation is selected at every level. No function chooses its own version -- it is always determined by the version reference passed from above.

The First Block of a New Protocol Version

The version flow above assumes the protocol version is already known. The switch itself happens in run_block_proposal (packages/rs-drive-abci/src/execution/engine/run_block_proposal/mod.rs). On the first block of an epoch, if the protocol version the network locked in during the previous epoch differs from the one in consensus, the block runs under the new version:

#![allow(unused)]
fn main() {
// abbreviated
let block_platform_version = if epoch_info.is_epoch_change_but_not_genesis()
    && platform_state.next_epoch_protocol_version()
        != platform_state.current_protocol_version_in_consensus()
{
    let next_protocol_version = platform_state.next_epoch_protocol_version();

    // We should panic if this node is not supported a new protocol version
    let Ok(next_platform_version) = PlatformVersion::get(next_protocol_version) else {
        panic!("Failed to upgrade the network protocol version {next_protocol_version}. ...");
    };

    let old_protocol_version = block_platform_state.current_protocol_version_in_consensus();
    block_platform_state.set_current_protocol_version_in_consensus(next_protocol_version);

    // This is for events like adding stuff to the root tree, or making structural changes/fixes
    self.perform_events_on_first_block_of_protocol_change(
        platform_state, &block_info, transaction, old_protocol_version, next_platform_version,
    )?;

    next_platform_version
} else {
    last_committed_platform_version
};
}

Three things to take from this:

  • Epoch info is computed with the old version, before the switch. The new version applies to everything after it.
  • A binary that does not know the new version panics with an upgrade message. That is deliberate: a node that cannot run the agreed protocol must stop rather than produce a divergent state root.
  • perform_events_on_first_block_of_protocol_change is where state migrations live. Anything that needs to be done to the tree once for a new protocol version, before its first state transition runs, goes here: creating a new root-tree subtree, rewriting a system contract, back-filling a sum tree.

The migration hook is itself a versioned method (drive_abci.methods.protocol_upgrade.perform_events_on_first_block_of_protocol_change: None in the earliest method tables, Some(0) once the first migration existed, Some(1) in the two most recent tables). Its v0 is a ladder of guarded rungs, one per protocol version that needed a migration:

#![allow(unused)]
fn main() {
// packages/rs-drive-abci/src/execution/platform_events/protocol_upgrade/
//   perform_events_on_first_block_of_protocol_change/v0/mod.rs

if previous_protocol_version < 4 && platform_version.protocol_version >= 4 {
    self.transition_to_version_4(platform_state, block_info, transaction, platform_version)?;
}
if previous_protocol_version < 6 && platform_version.protocol_version >= 6 {
    self.transition_to_version_6(block_info, transaction, platform_version)?;
}
// ... 8, 9, 11, 12, 13 ...
if previous_protocol_version < 14 && platform_version.protocol_version >= 14 {
    self.transition_to_version_14(block_info, transaction, platform_version)?;
}
}

The guard shape matters. A node can cross more than one protocol version in a single switch (a network that skipped a version, or a devnet started at an old one), and the previous < N && new >= N form runs every rung it crossed, in order. A rung written as new == N would be skipped by such a node, and its state tree would be missing a subtree every other node has.

v1 exists because the migrations write system contracts through a path that bypasses the drive operation batch's cache invalidation; it runs the same ladder and then refreshes the cached contract definitions so that a validator with a warm cache and one with a cold cache serialize the same bytes. Read its doc comment before touching the hook: it is a worked example of a consensus-critical cache bug.

To add a migration for a new protocol version: add a rung at the bottom of the ladder in the current generation of the hook, guarded by that version, with a transition_to_version_N helper next to the others. A failure in a rung returns Err from block processing on every node, so a rung either succeeds deterministically or halts the network. The version 8 rung's or_else that logs and continues is the exception for a migration that does not touch the state structure, not a pattern to copy.

Whole new state transition kinds are gated separately, by the is_allowed stage of the validation pipeline reading the constants in feature_initial_protocol_versions.rs (ADDRESS_FUNDS_INITIAL_PROTOCOL_VERSION = 11, SHIELDED_POOL_INITIAL_PROTOCOL_VERSION = 12). A transition submitted before its initial version is rejected with a StateTransitionNotActiveError rather than an unknown-version dispatch error.

Genesis content is the other place a protocol_version comparison is the intended shape. create_genesis_state runs once per chain, under the protocol version the chain is born at. Mainnet and testnet were born at protocol version 1 and replay generation 0, which stays frozen. Generation 1 is selected only for chains born at protocol version 9 or later, and every such chain is a devnet, a local network or a test suite that is created again for each release, so no live node reproduces its genesis. When a protocol version adds a system contract or other genesis content, it goes into create_genesis_state_v1 behind if platform_version.protocol_version >= N (document history at 13, app connect and moderation charters at 14), and a chain that already exists gets the same content from its transition_to_version_N rung. A new genesis generation would add code for no replay benefit. Never edit generation 0.

The Drive helpers that build the initial state structure follow the same rule for the same reason: they run once, at chain creation, under the chain's initial protocol version, and a chain that already exists gets the same trees from its upgrade rung. add_initial_withdrawal_state_structure_operations adds the withdrawal sum trees behind >= 4, which replaying mainnet's genesis at protocol version 1 does not take. The withdrawal limit trees of protocol version 14 are not in that batch: genesis and transition_to_version_14 both add them one insert at a time through Drive::insert_withdrawal_limit_trees, so both build the withdrawals Merk in the same shape.

Rules

Do:

  • Always include the method name in the UnknownVersionMismatch error. Use the same string format you see in existing code: the plain method name for Drive methods ("grove_get_raw"), and a descriptive path for ABCI methods ("identity create transition: validate_basic_structure").
  • Keep the known_versions vector in the error arm up to date. When you add version 2, the vector should be vec![0, 1, 2].
  • Make versioned implementation methods pub(super) -- visible to the dispatch module but not to external code.
  • Put every new generation in its own vN/ directory, started as a copy of the previous one, with its tests inside it.
  • Bump the slot in the unreleased protocol version's tables only: amend a constant that only the unreleased version references, create the next constant when a released version references it.
  • Test the current generation against PlatformVersion::latest() and pin the previous generation's tests to the last protocol version that selected it.
  • Put one-time state changes a protocol version needs in a guarded rung of perform_events_on_first_block_of_protocol_change.

Do not:

  • Never call a versioned implementation directly (e.g., grove_get_raw_v0). Always go through the dispatch method. Direct calls bypass version control and break determinism.
  • Never add a version to the match without also adding the corresponding FeatureVersion field value in the version constants. The dispatch will never be reached if no platform version sets that number.
  • Never use _ => as the catch-all arm in a version dispatch. Always use version => so the variable is available for the error message. And never silently ignore unknown versions -- always return an error.
  • Never modify a shipped generation, not even by threading a parameter or a version check through it. If v0 takes five parameters and v1 needs six, that is fine -- v0 keeps its original signature and body forever.
  • Never gate new behaviour with a bool on a shared context struct. A helper reached from several generations gets an OptionalFeatureVersion slot and takes &PlatformVersion.
  • Never write a test that only asserts a table slot's value. Test the behaviour through the dispatcher on both sides of the gate.

The State Transition Lifecycle

Every change to Dash Platform -- creating an identity, registering a data contract, storing a document, casting a masternode vote -- follows the same fundamental pattern: a state transition. If you understand this one concept, you understand the heartbeat of the entire platform.

What Is a State Transition?

A state transition is the atomic unit of state change on Dash Platform. Think of the platform's state as a database. You cannot write to that database directly. Instead, you construct a state transition object that describes what you want to change, sign it with your private key, serialize it to bytes, and broadcast it to the network. Validators receive it, validate it through a multi-stage pipeline, and -- if everything checks out -- apply it to their copy of the state.

This is fundamentally different from a smart contract model. There is no arbitrary code execution. Every possible mutation is one of a fixed set of state transition types, each with its own validation rules hardcoded into the platform. The benefit is predictability: you can reason about fees, security, and correctness without worrying about Turing-complete execution.

The StateTransition Enum

At the Rust level, every state transition is a variant of a single enum. This is defined in packages/rs-dpp/src/state_transition/mod.rs:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Encode, Decode, PlatformSerialize,
         PlatformDeserialize, PlatformSignable, From, PartialEq)]
#[platform_serialize(unversioned)]
#[platform_serialize(limit = 100000)]
pub enum StateTransition {
    DataContractCreate(DataContractCreateTransition),
    DataContractUpdate(DataContractUpdateTransition),
    Batch(BatchTransition),
    IdentityCreate(IdentityCreateTransition),
    IdentityTopUp(IdentityTopUpTransition),
    IdentityCreditWithdrawal(IdentityCreditWithdrawalTransition),
    IdentityUpdate(IdentityUpdateTransition),
    IdentityCreditTransfer(IdentityCreditTransferTransition),
    MasternodeVote(MasternodeVoteTransition),
    IdentityCreditTransferToAddresses(IdentityCreditTransferToAddressesTransition),
    IdentityCreateFromAddresses(IdentityCreateFromAddressesTransition),
    IdentityTopUpFromAddresses(IdentityTopUpFromAddressesTransition),
    AddressFundsTransfer(AddressFundsTransferTransition),
    AddressFundingFromAssetLock(AddressFundingFromAssetLockTransition),
    AddressCreditWithdrawal(AddressCreditWithdrawalTransition),
    Shield(ShieldTransition),
    ShieldedTransfer(ShieldedTransferTransition),
    Unshield(UnshieldTransition),
    ShieldFromAssetLock(ShieldFromAssetLockTransition),
    ShieldedWithdrawal(ShieldedWithdrawalTransition),
    IdentityCreateFromShieldedPool(IdentityCreateFromShieldedPoolTransition),
    ShieldFromIdentity(ShieldFromIdentityTransition),
    IdentityTopUpFromShieldedPool(IdentityTopUpFromShieldedPoolTransition),
    IdentityKeyLimitsUpdate(IdentityKeyLimitsUpdateTransition),
}
}

Each variant wraps a dedicated struct. Notice the derive macros: Encode and Decode for bincode serialization, PlatformSerialize and PlatformDeserialize for the platform's own serialization layer, and PlatformSignable for generating the "signable bytes" that get signed.

These variants fall into natural groups:

Identity lifecycle:

  • IdentityCreate -- Register a new identity (funded by an asset lock on the Dash core chain)
  • IdentityTopUp -- Add credits to an existing identity (also asset-lock funded)
  • IdentityUpdate -- Add or disable public keys on an identity
  • IdentityCreditWithdrawal -- Withdraw credits back to the core chain
  • IdentityCreditTransfer -- Transfer credits between identities
  • IdentityKeyLimitsUpdate -- Raise the budget or extend the expiry of one of the identity's keys

Data contracts and documents:

  • DataContractCreate -- Register a new data contract (schema)
  • DataContractUpdate -- Update an existing data contract
  • Batch -- Create, replace, delete, or transfer documents; mint, burn, transfer, or freeze tokens

Governance:

  • MasternodeVote -- Cast a vote in a contested resource election

Address-based (newer):

  • IdentityCreateFromAddresses, IdentityTopUpFromAddresses, AddressFundsTransfer, AddressFundingFromAssetLock, AddressCreditWithdrawal -- Operations that use platform addresses instead of (or in addition to) identity-based authentication

Shielded pool:

  • Shield, ShieldedTransfer, Unshield, ShieldFromAssetLock, ShieldedWithdrawal, IdentityCreateFromShieldedPool, ShieldFromIdentity, IdentityTopUpFromShieldedPool -- Operations that move credits into, inside, and out of the shielded pool

Each variant has its own numeric discriminant, defined in packages/rs-dpp/src/state_transition/state_transition_types.rs:

#![allow(unused)]
fn main() {
#[repr(u8)]
pub enum StateTransitionType {
    DataContractCreate = 0,
    Batch = 1,
    IdentityCreate = 2,
    IdentityTopUp = 3,
    DataContractUpdate = 4,
    IdentityUpdate = 5,
    IdentityCreditWithdrawal = 6,
    IdentityCreditTransfer = 7,
    MasternodeVote = 8,
    IdentityCreditTransferToAddresses = 9,
    IdentityCreateFromAddresses = 10,
    IdentityTopUpFromAddresses = 11,
    AddressFundsTransfer = 12,
    AddressFundingFromAssetLock = 13,
    AddressCreditWithdrawal = 14,
    Shield = 15,
    ShieldedTransfer = 16,
    Unshield = 17,
    ShieldFromAssetLock = 18,
    ShieldedWithdrawal = 19,
    IdentityCreateFromShieldedPool = 20,
    ShieldFromIdentity = 21,
    IdentityTopUpFromShieldedPool = 22,
    IdentityKeyLimitsUpdate = 23,
}
}

This type tag is what allows the platform to deserialize a raw byte buffer into the correct variant. When bytes arrive over the wire, the first byte tells the deserializer which struct to decode into.

The Batch Transition: A Swiss Army Knife

The Batch variant deserves special attention because it is the most complex. A single BatchTransition can contain multiple sub-transitions, each operating on a different document or token. The sub-transitions include:

  • Document operations: Create, Replace, Delete, Transfer, UpdatePrice, Purchase
  • Token operations: Transfer, Mint, Burn, Freeze, Unfreeze, DestroyFrozenFunds, EmergencyAction, ConfigUpdate, Claim, DirectPurchase, SetPriceForDirectPurchase

This batching is important for atomicity: either all operations in the batch succeed or none of them do. It also means a single identity nonce covers the entire batch, preventing partial replay attacks.

Signatures and Authentication

State transitions carry cryptographic signatures that prove authorization. There are two fundamentally different authentication models:

Identity-signed transitions -- The majority of transition types. The signer is an identity that already exists on the platform. The transition carries a signature_public_key_id referencing a key in the identity's key set, plus a signature over the signable bytes.

Asset-lock transitions -- IdentityCreate and IdentityTopUp are special because the identity may not exist yet (or the transition does not require an identity signature). Instead, they carry an AssetLockProof that references a transaction on the Dash core chain. The signature proves ownership of the funds being locked.

Address-based transitions -- Newer transition types like IdentityCreateFromAddresses use platform address inputs with their own nonces and balances, rather than identity-based authentication.

The sign method on StateTransition handles this:

#![allow(unused)]
fn main() {
pub fn sign(
    &mut self,
    identity_public_key: &IdentityPublicKey,
    private_key: &[u8],
    bls: &impl BlsModule,
) -> Result<(), ProtocolError> { ... }
}

It first verifies that the key's purpose and security level are appropriate for this transition type, then signs the signable_bytes() with the private key. Supported key types are ECDSA_SECP256K1, ECDSA_HASH160, and BLS12_381.

The signable bytes are produced by the PlatformSignable derive macro. It serializes the transition with the signature field zeroed out, producing a deterministic byte sequence that both client and platform can independently compute.

Serialization

The platform uses bincode for wire serialization of state transitions, wrapped in the PlatformSerialize / PlatformDeserialize layer. This gives us:

  • Compact binary encoding (no field names, no JSON overhead)
  • Deterministic output (critical for signature verification)
  • Size limits (#[platform_serialize(limit = 100000)] -- 100KB max)
  • Version awareness through the #[platform_serialize(unversioned)] annotation

Deserialization includes version-range checks:

#![allow(unused)]
fn main() {
pub fn deserialize_from_bytes_in_version(
    bytes: &[u8],
    platform_version: &PlatformVersion,
) -> Result<Self, ProtocolError> {
    let state_transition = StateTransition::deserialize_from_bytes(bytes)?;
    let active_version_range = state_transition.active_version_range();
    if active_version_range.contains(&platform_version.protocol_version) {
        Ok(state_transition)
    } else {
        Err(ProtocolError::StateTransitionError(
            StateTransitionIsNotActiveError { ... },
        ))
    }
}
}

This ensures that a transition type introduced in protocol version 9 cannot be submitted to a node running protocol version 8.

The Full Journey

Here is the end-to-end lifecycle of a state transition, from a client's perspective all the way through to state application:

1. Client constructs the transition. Using the Rust SDK (rs-sdk), JavaScript SDK (js-dash-sdk), or any client library, the application builds a concrete transition struct -- for example, a DataContractCreateTransition containing the new contract's schema.

2. Client signs the transition. The client calls sign() or sign_external(), which computes signable bytes, signs them with the appropriate private key, and attaches the signature and key ID to the transition struct.

3. Client serializes and broadcasts. The signed StateTransition is serialized to bytes via serialize_to_bytes() and sent to the network through DAPI (the Decentralized API).

4. Platform receives the bytes in check_tx. Before a state transition enters the mempool, it goes through a lighter validation pass. The platform deserializes the bytes, checks the signature, verifies the identity has sufficient balance, and validates basic structure. This is a gatekeeper -- it rejects obviously invalid transitions cheaply.

5. Platform processes during process_proposal. When a block is proposed, each state transition goes through the full validation pipeline: is_allowed, signature verification, nonce validation, basic structure, balance checks, advanced structure, transform_into_action, and state validation. (We will cover this pipeline in detail in the next chapter.)

6. Platform applies the action. If validation succeeds, the resulting StateTransitionAction is converted into DriveOperations, which are converted into LowLevelDriveOperations, which are applied atomically to GroveDB. Fees are calculated and deducted. The state has changed.

7. Result is returned. The client receives confirmation (or rejection) through DAPI.

The Versioning Pattern

You will notice that almost every method on StateTransition dispatches through a version table. For example:

#![allow(unused)]
fn main() {
match platform_version
    .drive_abci
    .validation_and_processing
    .process_state_transition
{
    0 => v0::process_state_transition_v0(...),
    version => Err(Error::Execution(ExecutionError::UnknownVersionMismatch { ... })),
}
}

This is the protocol versioning pattern. Every behavior that could conceivably change between protocol versions is gated behind a version number looked up from PlatformVersion. This means a node running protocol version 9 and a node running protocol version 10 can both validate the same block correctly, each using its own version's logic. It is how the platform achieves hard-fork-free upgrades.

The call_method Macro

Since StateTransition is an enum with 15 variants, dispatching a method call to the inner type would require writing out a 15-arm match statement every time. The codebase solves this with a family of macros:

#![allow(unused)]
fn main() {
macro_rules! call_method {
    ($state_transition:expr, $method:ident) => {
        match $state_transition {
            StateTransition::DataContractCreate(st) => st.$method(),
            StateTransition::DataContractUpdate(st) => st.$method(),
            StateTransition::Batch(st) => st.$method(),
            // ... all 15 variants
        }
    };
}
}

There are several variants: call_method for universal methods, call_getter_method_identity_signed for methods that return Option (returning None for non-identity-signed transitions like IdentityCreate), and call_errorable_method_identity_signed for methods that return Result (returning an error for inapplicable variants).

Rules and Guidelines

Do:

  • Always deserialize with deserialize_from_bytes_in_version in production code, to enforce version-range checks.
  • Use signable_bytes() when computing what to sign or verify -- never serialize the full transition (which includes the signature itself).
  • Check active_version_range() before processing a transition to ensure it is valid for the current protocol version.

Do not:

  • Assume all transitions have signatures. IdentityCreateFromAddresses and AddressFundsTransfer return None from signature().
  • Assume all transitions have an owner_id. Address-based transitions do not.
  • Modify the StateTransitionType discriminant values -- they are part of the wire format and changing them would break all existing serialized data.
  • Add new variants without also updating every call_method macro and every match statement in the validation pipeline.

The Validation Pipeline

When a state transition arrives at a Dash Platform node, it does not get applied immediately. It must survive a gauntlet of validation stages, each designed to catch a different category of error. Understanding this pipeline is essential to understanding how the platform maintains security, prevents abuse, and ensures deterministic state across all validators.

Two Entry Points: check_tx and process_proposal

State transitions enter the pipeline through two different doors, depending on when they are being validated.

check_tx runs when a transition first arrives at a node (before entering the mempool) and again when rechecking mempool contents. It performs a lighter validation: signature verification, basic structure, and balance checks. The goal is to filter out obvious garbage cheaply, without doing expensive state lookups.

A MasternodeVote is the exception. A block refuses a failed vote without charging anyone, and the proposer drops it from the block without a trace, so check_tx runs the vote's advanced structure and state validation as well. A vote that a block would refuse, such as a Lock vote on a contest without locking, is refused when it is broadcast, with its error.

This is implemented in packages/rs-drive-abci/src/execution/validation/state_transition/check_tx_verification/mod.rs:

#![allow(unused)]
fn main() {
pub(in crate::execution) fn state_transition_to_execution_event_for_check_tx<'a, C: CoreRPCLike>(
    platform: &'a PlatformRef<C>,
    state_transition: StateTransition,
    check_tx_level: CheckTxLevel,
    platform_version: &PlatformVersion,
) -> Result<ConsensusValidationResult<Option<ExecutionEvent<'a>>>, Error> { ... }
}

process_proposal (also called the "Validator" path) runs during block execution. This is the full pipeline. It is implemented in packages/rs-drive-abci/src/execution/validation/state_transition/processor/v0/mod.rs:

#![allow(unused)]
fn main() {
pub(super) fn process_state_transition_v0<'a, C: CoreRPCLike>(
    platform: &'a PlatformRef<C>,
    block_info: &BlockInfo,
    state_transition: StateTransition,
    transaction: TransactionArg,
    platform_version: &PlatformVersion,
) -> Result<ConsensusValidationResult<ExecutionEvent<'a>>, Error> { ... }
}

Let us walk through the full pipeline, stage by stage.

Stage 1: Is Allowed

Some state transition types are only available starting from a certain protocol version. For example, address-based transitions like IdentityCreateFromAddresses require protocol version 11 or higher. The first check asks: is this transition type even permitted on the current network?

#![allow(unused)]
fn main() {
if state_transition.has_is_allowed_validation()? {
    let result = state_transition.validate_is_allowed(platform_version)?;
    if !result.is_valid() {
        return Ok(ConsensusValidationResult::new_with_errors(result.errors));
    }
}
}

The trait is defined in packages/rs-drive-abci/src/execution/validation/state_transition/processor/traits/is_allowed.rs:

#![allow(unused)]
fn main() {
pub(crate) trait StateTransitionIsAllowedValidationV0 {
    fn has_is_allowed_validation(&self) -> Result<bool, Error>;
    fn validate_is_allowed(
        &self,
        platform_version: &PlatformVersion,
    ) -> Result<ConsensusValidationResult<()>, Error>;
}
}

Transitions available since protocol version 1, like DataContractCreate, IdentityCreate and Batch, skip this check entirely.

Stage 2: Identity Signature Verification

For identity-signed transitions (everything except IdentityCreate, IdentityTopUp, and the address-based types), the platform fetches the signer's identity from state and verifies the signature against one of its registered public keys.

#![allow(unused)]
fn main() {
let mut maybe_identity = if state_transition.uses_identity_in_state() {
    let result = if state_transition.validates_signature_based_on_identity_info() {
        state_transition.validate_identity_signed_state_transition(
            platform.drive,
            transaction,
            &mut state_transition_execution_context,
            platform_version,
        )
    } else {
        state_transition.retrieve_identity_info(...)
    }?;
    if !result.is_valid() {
        return Ok(ConsensusValidationResult::new_with_errors(result.errors));
    }
    Some(result.into_data()?)
} else {
    None
};
}

If the signature does not match, the transition is rejected without charging a fee. This is critical: if someone forges a transition with your identity ID but a wrong signature, you should not pay for their garbage.

Stage 3: Address Witness Validation

For address-based transitions, instead of identity signatures, the platform validates witnesses (proofs of ownership) for the input addresses:

#![allow(unused)]
fn main() {
if state_transition.has_address_witness_validation(platform_version)? {
    let result = state_transition.validate_address_witnesses(
        &mut state_transition_execution_context,
        platform_version,
    )?;
    if !result.is_valid() {
        return Ok(ConsensusValidationResult::new_with_errors(result.errors));
    }
}
}

Stage 4: Address Balances and Nonces

For transitions that use platform addresses as inputs, the platform verifies that each input address has sufficient balance and that the provided nonces are correct (preventing replay):

#![allow(unused)]
fn main() {
let remaining_address_balances =
    if state_transition.has_addresses_balances_and_nonces_validation() {
        let result = state_transition.validate_address_balances_and_nonces(
            platform.drive,
            &mut state_transition_execution_context,
            transaction,
            platform_version,
        )?;
        if !result.is_valid() {
            return Ok(ConsensusValidationResult::new_with_errors(result.errors));
        }
        Some(result.into_data()?)
    } else {
        None
    };
}

Stage 5: Identity Nonce Validation

Nonces prevent replay attacks. Each identity-signed transition carries a nonce that must be strictly greater than the last used nonce for that identity (or identity-contract pair). The platform checks this against the stored nonce in state.

#![allow(unused)]
fn main() {
if state_transition.has_identity_nonce_validation(platform_version)? {
    let result = state_transition.validate_identity_nonces(
        &platform.into(),
        platform.state.last_block_info(),
        transaction,
        &mut state_transition_execution_context,
        platform_version,
    )?;
    if !result.is_valid() {
        return Ok(ConsensusValidationResult::new_with_errors(result.errors));
    }
}
}

Identity create and identity top up skip this check -- they do not have nonces because the identity may not exist yet.

Stage 6: Basic Structure Validation

This stage checks that the transition's data is well-formed without looking at platform state. For example: are all required fields present? Are field values within allowed ranges? Is the data contract schema valid JSON Schema?

#![allow(unused)]
fn main() {
if state_transition.has_basic_structure_validation(platform_version) {
    let consensus_result = state_transition.validate_basic_structure(
        platform.config.network,
        platform_version,
    )?;
    if !consensus_result.is_valid() {
        return Ok(ConsensusValidationResult::new_with_errors(
            consensus_result.errors,
        ));
    }
}
}

The trait lives in processor/traits/basic_structure.rs:

#![allow(unused)]
fn main() {
pub(crate) trait StateTransitionBasicStructureValidationV0 {
    fn validate_basic_structure(
        &self,
        network_type: Network,
        platform_version: &PlatformVersion,
    ) -> Result<SimpleConsensusValidationResult, Error>;

    fn has_basic_structure_validation(&self, _platform_version: &PlatformVersion) -> bool {
        true
    }
}
}

MasternodeVote skips basic structure validation entirely. DataContractCreate and DataContractUpdate conditionally enable it based on the platform version.

Stage 7: Balance Pre-Check

Before doing expensive state validation, the platform checks that the identity has enough credits to plausibly pay for this transition. For credit transfers and withdrawals, this includes checking that the transfer amount plus estimated fees does not exceed the balance.

#![allow(unused)]
fn main() {
if state_transition.has_identity_minimum_balance_pre_check_validation() {
    let result = state_transition.validate_identity_minimum_balance_pre_check(
        identity, platform_version,
    )?;
    if !result.is_valid() {
        return Ok(ConsensusValidationResult::new_with_errors(result.errors));
    }
}
}

A MasternodeVote is paid by its vote poll's prefunded specialized balance, not by the voter, so its pre-check is on that pot: a vote on a poll whose pot does not exist, or holds less than the single vote cost, is refused unpaid with PrefundedSpecializedBalanceNotFoundError or PrefundedSpecializedBalanceInsufficientError.

Stage 8: Advanced Structure Validation (without State)

Some transitions need structural validation that goes beyond basic checks but does not require reading from platform state. For example, IdentityUpdate verifies that added public keys have valid signatures. DataContractCreate validates the contract schema more deeply.

#![allow(unused)]
fn main() {
if state_transition.has_advanced_structure_validation_without_state() {
    let consensus_result = state_transition.validate_advanced_structure(
        identity, &mut state_transition_execution_context, platform_version,
    )?;
    if !consensus_result.is_valid() {
        // Note: this returns an action so the nonce gets bumped even on failure
        return consensus_result.map_result(|action| {
            ExecutionEvent::create_from_state_transition_action(...)
        });
    }
}
}

An important detail: if advanced structure validation fails, the identity's nonce is still bumped. This prevents an attacker from replaying a structurally invalid transition over and over without cost.

Stage 9: Transform Into Action + Advanced Structure (with State)

For certain transition types (Batch, IdentityCreate, MasternodeVote, AddressFundingFromAssetLock, IdentityCreateFromAddresses), the platform needs to read state before it can finish structure validation. For example, a document create transition needs to fetch the data contract to validate the document against its schema.

This stage first transforms the raw state transition into an action (reading state in the process), then validates the action's structure:

#![allow(unused)]
fn main() {
let action = if state_transition.has_advanced_structure_validation_with_state() {
    let state_transition_action_result = state_transition.transform_into_action(
        platform, block_info, &remaining_address_balances,
        ValidationMode::Validator, &mut state_transition_execution_context, transaction,
    )?;
    if !state_transition_action_result.is_valid_with_data() {
        return state_transition_action_result.map_result(|action| { ... });
    }
    let action = state_transition_action_result.into_data()?;

    let result = state_transition.validate_advanced_structure_from_state(
        block_info, platform.config.network, &action,
        maybe_identity.as_ref(), &mut state_transition_execution_context,
        platform_version,
    )?;
    if !result.is_valid() {
        return result.map_result(|action| { ... });
    }
    Some(action)
} else {
    None
};
}

We will cover transform_into_action in detail in the next chapter.

For IdentityCreateFromAddresses this stage also checks the proof of possession of each key the transition registers. The address witnesses cannot sign those signatures (both sign the same bytes), so the owners of the inputs never signed what this check judges. From protocol version 14 a failure is therefore refused without a fee, like a failed witness, instead of charging the inputs a penalty; check_tx runs this stage for the transition too, so such a transition does not reach a block.

Stage 10: State Validation

The final validation stage checks for state-level conflicts. Does a data contract with this ID already exist? Is there already a document with this unique index value? Has this asset lock already been spent?

#![allow(unused)]
fn main() {
let result = if state_transition.has_state_validation() {
    state_transition.validate_state(
        action, platform, ValidationMode::Validator, block_info,
        &mut state_transition_execution_context, transaction,
    )?
} else if let Some(action) = action {
    ConsensusValidationResult::new_with_data(action)
} else {
    state_transition.transform_into_action(...)? // For transitions that skipped earlier
};
}

Not all transitions need state validation. IdentityTopUp, IdentityCreditWithdrawal, AddressFundsTransfer, and several others skip it -- their validation is fully covered by the earlier stages.

ConsensusValidationResult: How Errors Accumulate

Throughout the pipeline, errors are communicated through ConsensusValidationResult, defined in packages/rs-dpp/src/validation/validation_result.rs:

#![allow(unused)]
fn main() {
pub type ConsensusValidationResult<TData> = ValidationResult<TData, ConsensusError>;
pub type SimpleConsensusValidationResult = ConsensusValidationResult<()>;

pub struct ValidationResult<TData: Clone, E: Debug> {
    pub errors: Vec<E>,
    pub data: Option<TData>,
}
}

Key properties of this type:

  • It can carry data alongside errors. When transform_into_action partially succeeds but has warnings, the action is in data and the warnings are in errors.
  • is_valid() checks if errors is empty. Any error means failure.
  • is_valid_with_data() checks both. Valid and has associated data.
  • Errors can be merged from multiple validation steps using add_errors() or merge().
  • Chainable via and_then_validation() for pipeline-style composition.

The SimpleConsensusValidationResult alias (where TData = ()) is used by validation stages that just check pass/fail without producing a transformed object.

ValidationMode

The pipeline behavior varies based on context:

#![allow(unused)]
fn main() {
pub enum ValidationMode {
    CheckTx,       // Mempool admission -- lighter checks
    RecheckTx,     // Periodic mempool revalidation
    Validator,     // Full block execution
    NoValidation,  // Testing/tooling only
}
}

For example, should_fully_validate_contract_on_transform_into_action() returns true only in Validator mode. During CheckTx, the platform skips expensive contract validation to keep mempool admission fast.

The Early-Return Pattern

You will notice a consistent pattern throughout the pipeline: each stage checks is_valid() and returns early if validation failed. This is intentional.

When the pipeline returns early due to a signature failure or invalid nonce, the transition produces no execution event -- the user is not charged. This protects users from being billed for transitions they did not actually submit (forged signatures) or that are replays (invalid nonces).

But when the pipeline returns early after the nonce has been validated -- for example, during advanced structure validation or state validation -- it returns a nonce-bump action so the identity still pays a small fee. This prevents a different attack: submitting thousands of structurally invalid transitions to waste validator resources for free.

Rules and Guidelines

Do:

  • Run the full pipeline in Validator mode during block execution. Skipping stages can lead to consensus divergence.
  • Return early on signature/nonce failures without charging the user.
  • Bump the nonce (and charge) when structure or state validation fails after the nonce check has passed.
  • Use ConsensusValidationResult everywhere -- do not use Result<(), Error> for validation outcomes, because a validation failure is not a system error.

Do not:

  • Add expensive state reads to basic structure validation. It runs during check_tx and must be fast.
  • Skip is_allowed validation -- it is the mechanism that enables protocol upgrades to gate new transition types.
  • Assume the pipeline order is arbitrary. Each stage depends on information established by previous stages (e.g., the identity fetched during signature validation is used in balance checks).
  • Confuse ConsensusError (a validation failure that the user caused) with Error (a system error like a database failure). The former accumulates in ValidationResult::errors; the latter propagates via Result::Err.

Transform Into Action

After a state transition passes the early validation stages -- signature verification, nonce checks, basic structure -- the platform needs to translate it from a raw "what the client sent" representation into a "what the platform should do" representation. This translation step is called transform into action, and it is where the platform reads state, resolves references, and prepares a validated, self-contained instruction for the Drive storage layer.

Why Actions Exist

Consider a DataContractCreateTransition. The client sends the contract in its serialized form. But before the platform can store it, it needs to:

  1. Deserialize the contract from its wire format
  2. Validate the contract schema (JSON Schema validation, index rules, etc.)
  3. Compute the contract ID from the owner ID and nonce
  4. Check version compatibility

The transition is what the client sends. The action is what the platform will execute. The action is a validated, resolved, ready-to-apply object. It has already been through deserialization, its references have been resolved against current state, and it carries exactly the information needed to generate Drive operations.

This separation is a deliberate design choice. The transition type lives in rs-dpp (the protocol library, shared between client and platform). The action type lives in rs-drive (the storage layer, platform-only). By keeping them separate:

  • Clients never need to depend on storage internals
  • The platform can evolve its internal representation without changing the wire format
  • Validation logic lives close to state access, not scattered across the protocol layer

The StateTransitionActionTransformer Trait

The transformation is defined by a trait in packages/rs-drive-abci/src/execution/validation/state_transition/transformer/mod.rs:

#![allow(unused)]
fn main() {
pub trait StateTransitionActionTransformer {
    fn transform_into_action<C: CoreRPCLike>(
        &self,
        platform: &PlatformRef<C>,
        block_info: &BlockInfo,
        remaining_address_input_balances: &Option<
            BTreeMap<PlatformAddress, (AddressNonce, Credits)>,
        >,
        validation_mode: ValidationMode,
        execution_context: &mut StateTransitionExecutionContext,
        tx: TransactionArg,
    ) -> Result<ConsensusValidationResult<StateTransitionAction>, Error>;
}
}

The key parameters:

  • platform -- Access to the full platform state, including Drive (the storage layer), the current platform state, and configuration.
  • block_info -- The current block height, time, epoch. Some validations are time-dependent.
  • remaining_address_input_balances -- For address-based transitions, the balances remaining after earlier deductions.
  • validation_mode -- Controls how thorough the validation is (lighter for CheckTx, full for Validator).
  • execution_context -- Accumulates information about what operations were performed during validation (used for fee estimation).
  • tx -- The GroveDB transaction handle for consistent state reads.

The return type is ConsensusValidationResult<StateTransitionAction> -- it can carry both the resulting action and any validation errors encountered during transformation.

The Top-Level Dispatch

The StateTransition enum implements StateTransitionActionTransformer by dispatching to each variant's own implementation:

#![allow(unused)]
fn main() {
impl StateTransitionActionTransformer for StateTransition {
    fn transform_into_action<C: CoreRPCLike>(
        &self,
        platform: &PlatformRef<C>,
        block_info: &BlockInfo,
        remaining_address_input_balances: &Option<BTreeMap<PlatformAddress, (AddressNonce, Credits)>>,
        validation_mode: ValidationMode,
        execution_context: &mut StateTransitionExecutionContext,
        tx: TransactionArg,
    ) -> Result<ConsensusValidationResult<StateTransitionAction>, Error> {
        match self {
            StateTransition::DataContractCreate(st) => st.transform_into_action(
                platform, block_info, remaining_address_input_balances,
                validation_mode, execution_context, tx,
            ),
            StateTransition::Batch(st) => st.transform_into_action(
                platform, block_info, remaining_address_input_balances,
                validation_mode, execution_context, tx,
            ),
            StateTransition::IdentityCreate(st) => {
                let signable_bytes = self.signable_bytes()?;
                st.transform_into_action_for_identity_create_transition(
                    platform, signable_bytes, validation_mode,
                    execution_context, tx,
                )
            },
            StateTransition::IdentityTopUp(st) => {
                let signable_bytes = self.signable_bytes()?;
                st.transform_into_action_for_identity_top_up_transition(
                    platform, signable_bytes, validation_mode,
                    execution_context, tx,
                )
            },
            // ... remaining variants
        }
    }
}
}

Notice that IdentityCreate and IdentityTopUp use specialized method names (transform_into_action_for_identity_create_transition) and pass signable_bytes explicitly. This is because these transitions need the signable bytes to verify the asset lock proof signature, and computing them at the StateTransition level (before the inner struct is extracted) ensures the correct bytes are used.

The StateTransitionAction Enum

The output of transformation is a StateTransitionAction, defined in packages/rs-drive/src/state_transition_action/mod.rs:

#![allow(unused)]
fn main() {
pub enum StateTransitionAction {
    DataContractCreateAction(DataContractCreateTransitionAction),
    DataContractUpdateAction(DataContractUpdateTransitionAction),
    BatchAction(BatchTransitionAction),
    IdentityCreateAction(IdentityCreateTransitionAction),
    IdentityTopUpAction(IdentityTopUpTransitionAction),
    IdentityCreditWithdrawalAction(IdentityCreditWithdrawalTransitionAction),
    IdentityUpdateAction(IdentityUpdateTransitionAction),
    IdentityCreditTransferAction(IdentityCreditTransferTransitionAction),
    MasternodeVoteAction(MasternodeVoteTransitionAction),
    // ... address-based actions ...
    BumpIdentityNonceAction(BumpIdentityNonceAction),
    BumpIdentityDataContractNonceAction(BumpIdentityDataContractNonceAction),
    PartiallyUseAssetLockAction(PartiallyUseAssetLockAction),
    BumpAddressInputNoncesAction(BumpAddressInputNoncesAction),
}
}

Notice the system actions at the bottom: BumpIdentityNonceAction, BumpIdentityDataContractNonceAction, PartiallyUseAssetLockAction, and BumpAddressInputNoncesAction. These are not user-requested actions. They are generated by the platform when a transition fails validation after the nonce check. The platform still needs to bump the nonce (so the same invalid transition cannot be replayed) and charge a fee. These system actions handle that housekeeping.

Versioned Dispatch Within transform_into_action

Each transition type's transform_into_action implementation uses the standard versioned dispatch pattern. Here is the data contract create example from packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/data_contract_create/mod.rs:

#![allow(unused)]
fn main() {
impl StateTransitionActionTransformer for DataContractCreateTransition {
    fn transform_into_action<C: CoreRPCLike>(
        &self,
        platform: &PlatformRef<C>,
        block_info: &BlockInfo,
        _remaining_address_input_balances: &Option<
            BTreeMap<PlatformAddress, (AddressNonce, Credits)>,
        >,
        validation_mode: ValidationMode,
        execution_context: &mut StateTransitionExecutionContext,
        _tx: TransactionArg,
    ) -> Result<ConsensusValidationResult<StateTransitionAction>, Error> {
        let platform_version = platform.state.current_platform_version()?;

        match platform_version
            .drive_abci
            .validation_and_processing
            .state_transitions
            .contract_create_state_transition
            .transform_into_action
        {
            0 => self.transform_into_action_v0::<C>(
                block_info, validation_mode,
                execution_context, platform_version,
            ),
            version => Err(Error::Execution(ExecutionError::UnknownVersionMismatch {
                method: "data contract create transition: transform_into_action".to_string(),
                known_versions: vec![0],
                received: version,
            })),
        }
    }
}
}

The version number (0 here) is looked up from the platform version struct, allowing the logic to change in future protocol upgrades without modifying the dispatch layer.

What Happens Inside: Reading State

The core work inside transform_into_action varies by transition type, but the common thread is resolving references against current state. Here are the key patterns for different transition types:

DataContractCreate: Deserializes the contract from its wire format, validates the schema, and wraps it in a DataContractCreateTransitionAction. The validation depth depends on ValidationMode -- CheckTx mode skips full schema validation for speed.

DataContractUpdate: Fetches the existing contract from Drive to compare against the proposed update. Validates that schema changes are backward-compatible (e.g., you can add fields but not remove required ones).

Batch (Documents): This is the most complex transformation. For each document sub-transition in the batch, the platform must:

  1. Fetch the data contract that the document belongs to
  2. Look up the document type within that contract
  3. For creates: validate the document against the type's schema
  4. For replaces/deletes: fetch the existing document to verify ownership and revision
  5. For token operations: validate token configuration and authorization

IdentityCreate: Validates the asset lock proof against the core chain (via RPC), verifies the signature on the transition using keys embedded in the transition itself, and constructs the identity that will be created.

MasternodeVote: Fetches the contested resource being voted on, verifies that the masternode is eligible to vote, and constructs the vote action.

The Batch Transition: A Deeper Look

The BatchTransition is worth examining more closely because it demonstrates how transform_into_action handles multiple sub-operations. A single batch can contain dozens of document and token transitions targeting different contracts and document types.

During transformation, the platform:

  1. Groups transitions by contract. This allows fetching each contract only once.
  2. Fetches all needed contracts. Each contract is loaded from Drive (with caching).
  3. Resolves each sub-transition. Document creates get validated against their type's schema. Replaces fetch the existing document. Token operations check authorization rules.
  4. Produces a BatchTransitionAction containing individual BatchedTransitionAction items -- each of which is either a DocumentAction or a TokenAction.

If any sub-transition fails validation, the entire batch fails, and the platform produces a BumpIdentityDataContractNonceAction instead (bumping the nonce to prevent replay, while charging the user).

When Transformation Fails

Transformation can fail in two ways:

System error (Result::Err): Something unexpected happened -- a database read failed, a version mismatch, corrupted state. These propagate up as Error and typically halt block processing.

Consensus error (ValidationResult with errors): The transition itself is invalid. The document does not match its schema, the contract does not exist, the asset lock is already spent. In this case, the result still carries data -- typically a nonce-bump action -- so the pipeline can charge the user and move on.

This distinction is reflected in the return type: Result<ConsensusValidationResult<StateTransitionAction>, Error>. The outer Result is for system errors. The inner ConsensusValidationResult is for user errors.

Rules and Guidelines

Do:

  • Always respect ValidationMode. Skip expensive work during CheckTx but never skip it during Validator mode.
  • Accumulate validation costs in execution_context -- every state read, every schema validation -- so that fee calculation is accurate.
  • Return a nonce-bump action when transformation fails for user-caused reasons. Never silently swallow the transition.
  • Fetch contracts through Drive's caching layer. A batch transition might reference the same contract dozens of times; fetching it from disk each time would be prohibitively expensive.

Do not:

  • Perform state writes during transform_into_action. This is a read-only phase. State is only modified when the resulting action is applied later.
  • Mix up the transition and action types. The transition is DataContractCreateTransition (from rs-dpp). The action is DataContractCreateTransitionAction (from rs-drive). They live in different crates for good reason.
  • Forget to handle the remaining_address_input_balances parameter for address-based transitions. It is None for identity-based transitions but must be Some for address-based ones -- failing to provide it is a corrupted code execution error.
  • Add new transition types without implementing both the transformer trait and adding the variant to the StateTransitionAction enum. These must stay in sync.

Drive Operations: From Action to Storage

By this point in the pipeline, a state transition has been validated and transformed into a StateTransitionAction. But the action is still an abstract description of what should change. The final step is translating it into concrete storage mutations that get applied atomically to GroveDB, the platform's Merkle tree database. This translation happens through a three-tier pipeline of progressively lower-level operation types.

The Three-Tier Pipeline

The pipeline looks like this:

StateTransitionAction
    |
    | DriveHighLevelOperationConverter::into_high_level_drive_operations()
    v
Vec<DriveOperation>
    |
    | DriveLowLevelOperationConverter::into_low_level_drive_operations()
    v
Vec<LowLevelDriveOperation>
    |
    | apply_batch_low_level_drive_operations()
    v
GroveDB (applied atomically)

Each tier exists for a reason:

  • StateTransitionAction speaks the language of the protocol: "create this identity," "store this document," "transfer these credits."
  • DriveOperation speaks the language of Drive's domain model: "insert a document into this contract's document type tree," "update an identity's balance."
  • LowLevelDriveOperation speaks the language of GroveDB: "insert this key-value pair at this path," "delete this element."

This layering allows each tier to be tested independently and evolved separately. A change to how documents are indexed in GroveDB only affects the DriveLowLevelOperationConverter implementation for DocumentOperationType -- it does not ripple up to the action layer.

Tier 1: DriveHighLevelOperationConverter

The first conversion step is defined in packages/rs-drive/src/state_transition_action/action_convert_to_operations/mod.rs:

#![allow(unused)]
fn main() {
pub trait DriveHighLevelOperationConverter {
    fn into_high_level_drive_operations<'a>(
        self,
        epoch: &Epoch,
        platform_version: &PlatformVersion,
    ) -> Result<Vec<DriveOperation<'a>>, Error>;
}
}

The StateTransitionAction enum implements this trait by dispatching to each variant:

#![allow(unused)]
fn main() {
impl DriveHighLevelOperationConverter for StateTransitionAction {
    fn into_high_level_drive_operations<'a>(
        self,
        epoch: &Epoch,
        platform_version: &PlatformVersion,
    ) -> Result<Vec<DriveOperation<'a>>, Error> {
        match self {
            StateTransitionAction::DataContractCreateAction(action) => {
                action.into_high_level_drive_operations(epoch, platform_version)
            }
            StateTransitionAction::DataContractUpdateAction(action) => {
                action.into_high_level_drive_operations(epoch, platform_version)
            }
            StateTransitionAction::BatchAction(action) => {
                action.into_high_level_drive_operations(epoch, platform_version)
            }
            StateTransitionAction::IdentityCreateAction(action) => {
                action.into_high_level_drive_operations(epoch, platform_version)
            }
            // ... all other variants
        }
    }
}
}

Each action type knows how to decompose itself into the appropriate DriveOperation variants. A single action often produces multiple drive operations. For example, IdentityCreateAction might produce:

  1. An IdentityOperation to insert the identity
  2. An IdentityOperation to set the initial balance
  3. Multiple IdentityOperations to add each public key
  4. A SystemOperation to update system credit tracking

For the BatchTransitionAction, there is an additional layer of delegation. The batch contains multiple BatchedTransitionAction items, each of which implements DriveHighLevelBatchOperationConverter:

#![allow(unused)]
fn main() {
pub trait DriveHighLevelBatchOperationConverter {
    fn into_high_level_batch_drive_operations<'a>(
        self,
        epoch: &Epoch,
        owner_id: Identifier,
        platform_version: &PlatformVersion,
    ) -> Result<Vec<DriveOperation<'a>>, Error>;
}
}

Notice the extra owner_id parameter -- batch operations need to know which identity owns the documents being created or modified.

Tier 2: The DriveOperation Enum

The DriveOperation enum represents domain-level storage operations. It is defined in packages/rs-drive/src/util/batch/drive_op_batch/mod.rs:

#![allow(unused)]
fn main() {
pub enum DriveOperation<'a> {
    DataContractOperation(DataContractOperationType<'a>),
    DocumentOperation(DocumentOperationType<'a>),
    TokenOperation(TokenOperationType),
    WithdrawalOperation(WithdrawalOperationType),
    IdentityOperation(IdentityOperationType),
    PrefundedSpecializedBalanceOperation(PrefundedSpecializedBalanceOperationType),
    SystemOperation(SystemOperationType),
    GroupOperation(GroupOperationType),
    AddressFundsOperation(AddressFundsOperationType),
    GroveDBOperation(QualifiedGroveDbOp),
    GroveDBOpBatch(GroveDbOpBatch),
}
}

Each variant wraps a type-specific operation enum. For example, DocumentOperationType includes operations like AddDocument, UpdateDocument, DeleteDocument -- each carrying the document data, contract reference, and storage flags needed for insertion.

The last two variants -- GroveDBOperation and GroveDBOpBatch -- are escape hatches for when higher-level abstractions are not needed. They wrap raw GroveDB operations directly.

The DriveOperation enum implements the DriveLowLevelOperationConverter trait to convert itself into the next tier:

#![allow(unused)]
fn main() {
pub trait DriveLowLevelOperationConverter {
    fn into_low_level_drive_operations(
        self,
        drive: &Drive,
        estimated_costs_only_with_layer_info: &mut Option<
            HashMap<KeyInfoPath, EstimatedLayerInformation>,
        >,
        block_info: &BlockInfo,
        transaction: TransactionArg,
        platform_version: &PlatformVersion,
    ) -> Result<Vec<LowLevelDriveOperation>, Error>;
}
}

Two important parameters here:

  • estimated_costs_only_with_layer_info: When this is Some, the converter does not actually read from or write to GroveDB. Instead, it estimates the cost of the operations using layer information. This is used for fee estimation before execution.
  • transaction: The GroveDB transaction handle. All reads and writes within a single block happen within one transaction, ensuring atomicity.

Tier 3: LowLevelDriveOperation

The lowest tier is defined in packages/rs-drive/src/fees/op.rs:

#![allow(unused)]
fn main() {
pub enum LowLevelDriveOperation {
    GroveOperation(QualifiedGroveDbOp),
    FunctionOperation(FunctionOp),
    CalculatedCostOperation(OperationCost),
    PreCalculatedFeeResult(FeeResult),
}
}

At this level, there are only four kinds of things:

  • GroveOperation: A concrete GroveDB operation -- insert, delete, or update a key-value pair at a specific path in the Merkle tree.
  • FunctionOperation: A CPU-bound operation with a pre-defined cost (like hashing or signature verification). These do not touch storage but still cost processing fees.
  • CalculatedCostOperation: A pre-computed cost that gets folded into the fee calculation.
  • PreCalculatedFeeResult: An already-computed fee result, used when the fee for an operation was determined earlier in the pipeline.

The GroveOperation variant is where the rubber meets the road. QualifiedGroveDbOp is GroveDB's own batch operation type -- it specifies a path (a vector of byte-string segments navigating the tree), a key, and an operation (insert element, delete, replace, etc.).

Applying the Batch

The entire sequence -- from DriveOperation collection to GroveDB application -- is orchestrated by Drive::apply_drive_operations, defined in packages/rs-drive/src/util/batch/drive_op_batch/drive_methods/apply_drive_operations/v0/mod.rs:

#![allow(unused)]
fn main() {
pub(crate) fn apply_drive_operations_v0(
    &self,
    operations: Vec<DriveOperation>,
    apply: bool,
    block_info: &BlockInfo,
    transaction: TransactionArg,
    platform_version: &PlatformVersion,
    previous_fee_versions: Option<&CachedEpochIndexFeeVersions>,
) -> Result<FeeResult, Error> {
    if operations.is_empty() {
        return Ok(FeeResult::default());
    }
    let mut low_level_operations = vec![];
    let mut estimated_costs_only_with_layer_info = if apply {
        None
    } else {
        Some(HashMap::new())
    };

    let mut finalize_tasks: Vec<DriveOperationFinalizeTask> = Vec::new();

    for drive_op in operations {
        if let Some(tasks) = drive_op.finalization_tasks(platform_version)? {
            finalize_tasks.extend(tasks);
        }
        low_level_operations.append(
            &mut drive_op.into_low_level_drive_operations(
                self, &mut estimated_costs_only_with_layer_info,
                block_info, transaction, platform_version,
            )?
        );
    }

    let mut cost_operations = vec![];
    self.apply_batch_low_level_drive_operations(
        estimated_costs_only_with_layer_info, transaction,
        low_level_operations, &mut cost_operations, &platform_version.drive,
    )?;

    for task in finalize_tasks {
        task.execute(self, platform_version);
    }

    Drive::calculate_fee(
        None, Some(cost_operations), &block_info.epoch,
        self.config.epochs_per_era, platform_version, previous_fee_versions,
    )
}
}

Let us break this down:

  1. Collect finalization tasks. Some operations need post-processing. For example, DataContractOperation may produce a RecordShieldedAnchor finalization task that runs after the batch is committed. Tasks are collected first, executed last.

  2. Convert to low-level operations. Each DriveOperation is expanded into one or more LowLevelDriveOperations. A single document insertion might produce dozens of GroveDB operations (one for each index, plus the document itself, plus metadata).

  3. Apply the batch atomically. apply_batch_low_level_drive_operations collects all GroveOperation items into a single GroveDB batch and applies them in one atomic write. This is critical -- if the node crashes mid-application, either all operations succeed or none do.

  4. Execute finalization tasks. Post-commit callbacks run (e.g., updating caches).

  5. Calculate fees. The cost of every operation (storage bytes written, bytes read, processing time) is tallied into a FeeResult that determines how many credits the user pays.

The apply Parameter

Notice the apply: bool parameter. When apply is false, the entire pipeline runs in estimation mode: operations are not actually written to GroveDB. Instead, the estimated_costs_only_with_layer_info map is populated with what would be written, and fees are estimated from that.

This is used during check_tx and fee estimation. The platform needs to know approximately how much a transition will cost before actually applying it, both for balance pre-checks and for returning fee estimates to clients.

The Fee Calculation

At the end of apply_drive_operations, fees are calculated from the accumulated cost operations:

#![allow(unused)]
fn main() {
Drive::calculate_fee(
    None,
    Some(cost_operations),
    &block_info.epoch,
    self.config.epochs_per_era,
    platform_version,
    previous_fee_versions,
)
}

The fee has two components:

  • Storage fee: Proportional to the bytes written to disk. Stored bytes have an ongoing cost because they consume space in the state tree indefinitely (until deleted). When bytes are later removed, a portion of the storage fee is refunded.
  • Processing fee: Proportional to the CPU work performed -- hashing, signature verification, tree traversal. This is ephemeral and not refundable.

The user's user_fee_increase (a percentage multiplier) applies to the processing fee, allowing users to bid higher for priority.

Finalization Tasks

Some DriveOperation variants carry finalization tasks -- callbacks that run after the batch is committed. These are defined in packages/rs-drive/src/util/batch/drive_op_batch/finalize_task.rs:

#![allow(unused)]
fn main() {
pub(crate) trait DriveOperationFinalizationTasks {
    fn finalization_tasks(
        &self,
        platform_version: &PlatformVersion,
    ) -> Result<Option<Vec<DriveOperationFinalizeTask>>, Error>;
}
}

The most common finalization task is RecordShieldedAnchor, used by shielded transaction operations to record Merkle tree anchors after the state changes are committed. Finalization tasks are intentionally limited -- they must be deterministic and must not fail, since they run after the batch is already committed.

Putting It All Together: A Document Create

Let us trace a document creation through the entire three-tier pipeline:

  1. Action: BatchTransitionAction containing a DocumentAction::CreateAction with the document data, its type, and the contract reference.

  2. DriveHighLevelOperationConverter: The document create action produces a DriveOperation::DocumentOperation(AddDocument { ... }) containing the owned document, contract info, document type info, and storage flags.

  3. DriveLowLevelOperationConverter: The AddDocument operation produces multiple LowLevelDriveOperation::GroveOperation items:

    • Insert the serialized document at its primary key path
    • Insert index entries for each indexed property
    • Update the document type's document count
    • Record storage flags for fee tracking
  4. GroveDB batch: All the GroveOperation items from all documents in the batch are collected into a single GroveDbOpBatch and applied atomically.

  5. Fee calculation: The total bytes written, bytes read, and processing operations are summed to produce the FeeResult.

Rules and Guidelines

Do:

  • Implement DriveHighLevelOperationConverter for new action types. This is the contract between the validation layer and the storage layer.
  • Keep into_low_level_drive_operations deterministic. Given the same inputs and state, it must always produce the same operations. Non-determinism causes consensus failures.
  • Use the estimation mode (apply = false) for fee pre-checks. Do not skip fee estimation -- users need accurate cost information before committing to a transition.
  • Test both the estimation path and the application path. They can diverge if the estimation layer info is stale or incomplete.

Do not:

  • Access GroveDB directly from action conversion code. Always go through Drive's methods, which handle versioning, caching, and error translation.
  • Produce side effects in into_high_level_drive_operations. This conversion must be pure -- it maps data, it does not read or write state.
  • Assume a 1:1 mapping between actions and GroveDB operations. A single document create can produce 10+ GroveDB operations (one per index). A batch with 50 documents can produce hundreds.
  • Forget finalization tasks when adding new operation types that require post-commit work. If your operation needs to update a cache or record an anchor, implement DriveOperationFinalizationTasks.
  • Mix up DriveOperation (high-level, domain-aware) with LowLevelDriveOperation (low-level, GroveDB-aware). The naming can be confusing, but the distinction is important: the former knows about documents and identities, the latter knows about tree paths and elements.

Fee System Overview

Every state transition on Dash Platform costs credits. Credits are the internal unit of account (1 Dash = 100,000,000,000 credits). The fee system ensures that validators are compensated for computation and storage, that spam is economically infeasible, and that the platform's state does not grow unboundedly without payment.

This section covers the three fee eras that the platform has gone through:

  1. Identity Credit Fees (protocol versions 1--9) — Fees paid from an identity's credit balance, funded by asset lock transactions on Core.
  2. Platform Address Fees (protocol versions 10--11) — Fees paid from platform address balances using a UTXO-like input/output model.
  3. Shielded Transaction Fees (protocol version 12) — Fees embedded in zero-knowledge proofs and cryptographically bound to the Orchard bundle.

Each era introduced new ExecutionEvent variants and fee validation logic, but the underlying cost accounting (storage fees, processing fees, epoch distribution) is shared across all three.

Credits and Denomination

Platform credits are the smallest unit of value:

UnitCredits
1 credit1
1 mDash100,000,000
1 Dash100,000,000,000

All fee constants in the codebase are denominated in credits.

Cost Components

The platform distinguishes two fundamental kinds of cost:

Storage Fees

Storage fees pay for bytes that persist in GroveDB indefinitely. The rate is set in FeeStorageVersion:

ParameterValueDescription
storage_disk_usage_credit_per_byte27,000Permanent disk storage cost
storage_processing_credit_per_byte400I/O cost to write the bytes
storage_load_credit_per_byte20I/O cost to read stored bytes
non_storage_load_credit_per_byte10I/O cost for ephemeral reads
storage_seek_cost2,000Cost of a single disk seek

Storage fees are refundable: when data is deleted, a portion of the original storage fee is returned to the identity that paid it (see Refunds below).

The documents of a type that declares a ttl (protocol version 14) are the exception: they carry no storage flags and refund nothing, their bytes are priced for the time they live, and their storage fees are paid out to the epochs they live in, through the lifetime storage fee pools, instead of over the perpetual distribution. See Document Time To Live.

Processing Fees

Processing fees pay for computation that does not leave a permanent trace in storage: signature verification, hashing, tree traversal, and so on. These are non-refundable — the computation has already been performed.

Processing costs are built up from individual operations:

processing_fee =
    seek_count × storage_seek_cost
  + added_bytes × storage_processing_credit_per_byte
  + replaced_bytes × storage_processing_credit_per_byte
  + loaded_bytes × storage_load_credit_per_byte
  + hash_node_calls × (blake3_base + blake3_per_block)

Signature verification adds a fixed cost per algorithm:

AlgorithmCost (credits)
ECDSA secp256k115,000
BLS12-381300,000
ECDSA hash16015,500
BIP13 script hash300,000
EdDSA ed25519 hash1603,500

Hashing costs scale with the number of blocks processed:

Hash FunctionBasePer Block
SHA-2561005,000
Blake3100300
SHA-256 + RIPEMD-1606,0005,000

Minimum Fees

Every state transition type has a minimum fee that must be met regardless of the actual computation cost. This prevents zero-cost spam. The minimums are defined in StateTransitionMinFees:

Identity-Based Transitions (protocol versions 1--9)

TransitionMinimum Fee (credits)
Credit Transfer100,000
Credit Transfer to Addresses500,000
Credit Withdrawal400,000,000
Identity Update100,000
Document Batch (per sub-transition)100,000
Contract Create100,000
Contract Update100,000
Masternode Vote100,000

Address-Based Transitions (protocol versions 10--11)

TransitionMinimum Fee (credits)
Address Funds Transfer (per input)500,000
Address Funds Transfer (per output)6,000,000
Address Credit Withdrawal400,000,000
Identity Create (base)2,000,000
Identity Create (per key)6,500,000
Identity Top-Up (base)500,000

Data Contract Registration Fees (protocol version 9+)

Protocol version 9 introduced significant registration fees for data contracts to prevent namespace squatting:

ComponentFeeDash Equivalent
Base contract registration10,000,000,0000.1 Dash
Document type registration2,000,000,0000.02 Dash
Non-unique index1,000,000,0000.01 Dash
Unique index1,000,000,0000.01 Dash
Contested index100,000,000,0001.0 Dash
Token registration10,000,000,0000.1 Dash
Token uses a perpetual distribution10,000,000,0000.1 Dash
Token uses a pre-programmed distribution10,000,000,0000.1 Dash
Token uses a once-per-identity distribution (protocol version 14+)10,000,000,0000.1 Dash
Search keyword10,000,000,0000.1 Dash

Before protocol version 9, all registration fees were zero.

Contest Funds

A document create that opens or joins a contest (a contested unique index) prefunds the masternode votes: the amount leaves the contender's balance for the contest's prefunded balance, each vote takes a fixed cost from it, and what is left when the contest ends is released as processing fees. The amounts are VoteResolutionFundFees in the fee version:

ComponentProtocol versions 1 to 13Protocol version 14
Contested document fund (DPNS and every other contest)0.2 Dash0.1 Dash
Moderation election fund (an electedCharter application)none exist0.5 Dash
One vote0.0001 Dash0.00002 Dash

required_vote_resolution_fund in rs-dpp picks between the two funds; the schedules before 14 carry the contested document amount in the moderation field, so the choice changes nothing there.

User Fee Increase

Every state transition carries a user_fee_increase field (a UserFeeIncrease value). This allows the sender to voluntarily pay more than the base fee to prioritize their transition. The multiplier works as follows:

  • 0 = 100% of base fee (no increase)
  • 1 = 101% of base fee
  • 10 = 110% of base fee
  • 100 = 200% of base fee

The increase applies only to the processing fee component, not to storage fees. This is because storage fees are a direct function of bytes stored and should not be inflated.

#![allow(unused)]
fn main() {
fn apply_user_fee_increase(&mut self, user_fee_increase: UserFeeIncrease) {
    let increase = self.processing_fee * user_fee_increase as u64 / 100;
    self.processing_fee = self.processing_fee.saturating_add(increase);
}
}

ExecutionEvent Variants

The ExecutionEvent enum (in rs-drive-abci) determines how fees are collected for each state transition. There are eight variants:

VariantFee SourceUsed By
PaidIdentity credit balance, or the contract owner's for a sponsored document batch (below)Most identity-based transitions
PaidFromAssetLockAsset lock transaction valueIdentityCreate, IdentityTopUp
PaidFromAssetLockWithoutIdentityAsset lock (fixed amount)PartiallyUseAssetLock
PaidFromAssetLockToPoolAsset lock value; fee routed to the fee poolsShieldFromAssetLock
PaidFromAddressInputsPlatform address balancesAll address-based transitions; Shield (metered + a ZK compute fee via additional_fixed_fee_cost)
PaidFixedCostFixed fee to poolMasternodeVote
PaidFromShieldedPoolShielded pool value_balanceShieldedTransfer, Unshield, ShieldedWithdrawal
PaidFromShieldedPoolToNewIdentityShielded pool (the fixed denomination); the metered write + ZK compute fee is moved from the new identity's balance into the fee poolsIdentityCreateFromShieldedPool

Each variant carries the operations to execute and enough context for the fee validation and execution pipeline to deduct the correct amount from the correct source.

Gas paid by the contract owner

From protocol version 14 a document action that is paid for with a token can have its gas (the storage and processing fee) paid by the contract owner. The document type's token cost offers it (gasFeesPaidBy: DocumentOwner, ContractOwner or PreferContractOwner) and the transition's $tokenPaymentInfo asks for it with the same enum; GasFeesPaidBy::resolve in rs-dpp names the payer. A document owner can always opt out, can always state a preference, and can insist (ContractOwner) only on a document type that commits to paying. Only a token-paid action can be sponsored, so every sponsored transition is backed by a token the contract owner chose to hand out.

The batch transformer (v2) resolves one payer for the whole batch, reads the contract owner's balance into the action (ResolvedGasSponsor, billed to the batch) and the execution event carries it in Paid.gas_sponsor. Fee validation v1 judges the fee against the sponsor's balance and the signer only has to fund removed_balance; when the sponsor's balance falls short a batch that insists is refused unpaid (GasSponsorInsufficientBalanceError, 40222) and a batch that prefers falls back to the signer's balance. Execution v1 then charges whoever fee validation admitted. A batch that fails validation is never sponsored: its signer pays for the work that ran, and a request the document type does not offer is a paid rejection (GasFeesPaidByNotAllowedError, 40129). Storage refunds still go to the document's owner, whoever paid the storage: a sponsored document refunds its owner when it is deleted or replaced by a smaller one, even when the sponsor pays for that transition too. Each token the sponsor hands out is therefore worth up to the storage fee of the largest document the type allows, so a document type that offers sponsorship should bound its documents' size (maxLength, maxItems) and price the action accordingly.

The signer's minimum balance pre-check runs before the contracts are loaded; its v1 asks a batch that requests sponsorship for its principal only (purchases, contest collateral) and leaves the gas to fee validation, so an identity without credits can act on tokens it was given. Such a signer could not pay for a failed batch, and a failed batch is never sponsored, so check tx validates the batch of a signer under the fee minimum against the state in full, on the first check and on every recheck, as it does a masternode vote: what nobody could be charged for is refused there, or leaves the mempool once the tokens it counted on are spent, instead of being executed for free by a proposer.

Optional token costs

A token cost may declare optional: true (v3 meta-schema, protocol version 14). A transition on such an action may leave $tokenPaymentInfo out: it then pays no token, its signer pays the gas in credits as on an action without a token cost, and no sponsorship applies. With $tokenPaymentInfo present the token is charged exactly as for a required cost, sponsorship included, and an insufficient token balance is a rejection rather than a fallback to credits: the client chooses between token and credits before signing. Together with contract-owner gas this is the "free usage" pattern: an app hands out tokens, a user posts for free while they last, and keeps posting on credits after.

Document action fees

From protocol version 14 a document type may charge a fixed fee in credits for an action on one of its documents, on top of the gas. The actionFees keyword (v3 document meta-schema) sits beside tokenCost and prices the same six actions:

"post": {
  "type": "object",
  "actionFees": {
    "pricing": "feeMultiplier",
    "create": { "moderators": 100000000, "owner": 10000000 }
  }
}

Creating a post here costs an extra 0.001 Dash for the contract's moderation team and 0.0001 Dash for its owner. Each fee has those two parts, either of which may be left out. The owner parts collect in the contract's owner pot and the moderators parts in its moderators pot, and a ContractFeeClaim state transition pays a pot out (see Contract Moderation). A moderators part needs a contract that declares moderation (DocumentActionFeesWithoutModerationError, 10902): the moderation team is who that pot is for.

Pricing. fixed charges the declared amounts as written. feeMultiplier, the default, scales them by the fee multiplier of the epoch the action executes in (declared * multiplier_permille / 1000, rounded down), so a fee follows the network's fees. The multiplier is the item every epoch tree records, read once per batch and billed to it (fetch_action_fee_multiplier_with_fee). In the first block of an epoch that item is not there yet, because state transitions execute before the end of the block, where the epoch is initialized; the multiplier the epoch is about to be initialized with, the fee schedule's, is used then. Nothing else reads the epoch multiplier today: the metered fees do not scale with it. A scaled amount is held at the maximum number of credits rather than overflowing: a fee nobody can pay refuses the action for an insufficient balance, a consensus error, where an overflow would have failed every transition on the action with an internal one.

The transition agrees to the fee. The contract is read when the action executes, not when the transition was signed, so a transition that said nothing would pay whatever the contract declares by then. Every transition on an action that charges a fee therefore carries an action fee agreement ($actionFeeAgreement, on version 2 of the document base transition, the default from protocol version 14):

"$actionFeeAgreement": {
  "$formatVersion": "0",
  "owner": 10000000,
  "moderators": 100000000,
  "feeMultiplier": { "knownPermille": 1000, "increaseTolerancePercent": 20 }
}
  • owner and moderators are the amounts the document type declares for the action, before any multiplier. They must match exactly, each pot on its own: a fee that was raised, lowered, or moved between the pots since the signer read the contract refuses the action (DocumentActionFeeAgreementMismatchError, 40133), and the signer reads the contract again.
  • feeMultiplier says how the fee is priced. It is named for a feeMultiplier fee and left out for a fixed one, and an agreement to the other pricing is the same mismatch. knownPermille is the fee multiplier the signer priced the fee with, and increaseTolerancePercent how far above it the multiplier of the executing epoch may be, in percent of the known one: 20 accepts up to 1.2 times. A transition signed just before an epoch boundary is then not refused for a small move; one the multiplier outran is (DocumentActionFeeMultiplierNotToleratedError, 40134). A multiplier that fell is always accepted. What is charged follows the epoch's multiplier, never the known one.
  • A transition without an agreement on an action that charges a fee is refused (DocumentActionFeeAgreementNotSetError, 40132), whoever pays: a sponsored action says what it agrees to as well, since a preferred sponsor can hand the fee back to the signer. An agreement on an action that charges nothing is ignored.

All three are paid refusals that bump the nonce and charge no fee. They are judged in the batch's advanced structure validation (BatchTransitionAction::validate_action_fee_agreements), off the action alone: the base action carries the declaration beside the agreement, and the batch action the multiplier its transformer read. The mempool runs the same check when a transition arrives and again on every recheck, so a transition whose agreement no longer holds leaves the mempool with the same error instead of failing in the block that would have refused it. A client builds the agreement from the contract it showed its user with DocumentActionFeeAgreement::for_document_type_action, never from a contract fetched behind their back at signing time.

A seated team's discount. On a document type an elected contract moderates, the moderators part of an agreement may name less than the declared amount: the share the contract's seated moderation charter takes (its proposal's moderatorsShare, a percentage; none declared is the full amount), applied to the declared amount and rounded down to the credit (moderation_charter::moderators_share_of). Everything else must still match: the owner part and the pricing. With a share of 60, the post above admits exactly 60000000 for the moderators:

"$actionFeeAgreement": {
  "$formatVersion": "0",
  "owner": 10000000,
  "moderators": 60000000,
  "feeMultiplier": { "knownPermille": 1000, "increaseTolerancePercent": 20 }
}

The action is then charged the agreed amount, which is what reaches the moderators pot (scaled by the multiplier for a feeMultiplier fee, like the declared amount). An agreement to the declared amount stays valid whatever the team charges and reads no charter; only one that names less has the batch transformer read the seated charter (the byTargetContract index of the moderation charters contract) and the proposal it runs on, billed to the batch. Any other amount below the declared one, including a discount on a contract with no seated charter yet, is refused like a mismatch, paid and without a fee (DocumentActionFeeModeratorsShareMismatchError, 40139). A lower amount anywhere else (a type the contract does not moderate, a contract that is not elected) is the plain mismatch (40133). The mempool judges it the same way on arrival and on every recheck, since the recheck transforms the batch anew.

The amounts do not change yet. A contract update may not add, change or remove the actionFees of an existing document type, nor switch their pricing (DocumentTypeUpdateError). A document type added by an update may declare its own, so a live contract gets fees through new document types only. The agreement is what makes lifting this safe later: an owner who changes a fee cannot make a transition signed against the old one pay the new one.

Who pays. Whoever pays the gas pays the action fee: the signer, or the contract owner when they sponsor the gas. A sponsor's balance has to cover the gas and the fees they would owe; one that insisted and falls short is the same unpaid refusal as before (40222), and one that only preferred hands the gas and the fees back to the signer. The gas is estimated for the payer: first with the sponsor paying, then, when the sponsor does not pay, with the signer paying. Fee validation settles the payer through one function (gas_sponsor_pays) and hands it to execution, which charges that payer, so they always name the same one. The contract owner never pays the owner part: it would travel through the owner pot back to them and only cost writes. A sponsor is always the contract owner, so a sponsored action pays into the moderators pot only, which a contract that sponsors gas should price in. The fee counts against the budget of a budgeted signing key when its identity pays it.

Only an action that executes is charged. A transition that fails, in the transformer or later in state validation, becomes a nonce bump, and a bump owes nothing. The fees are therefore read off the transitions when the execution event is built, after state validation had its say, not when the transformer ran.

The fee is not part of the FeeResult. It moves as balance operations in the batch's own operation list: one removal from the payer, one addition per pot. The transition may already write the payer's balance: a purchase moves its price, a contested document its voting fund, and a sale pays a contract owner who may be sponsoring the gas. A balance operation computes the new balance from the one committed before its batch and GroveDB keeps only the last write of a key, so apply_drive_operations (generation 1, protocol version 14) merges every write of one identity balance, one fee pot or one prefunded specialized balance in a batch into a single net operation. Token writes cannot be merged that way (a transfer writes two balances, a mint or a burn a balance and the supply), so a batch that writes one token balance or supply twice is refused; no state transition makes one. The fee pools and the proposers see exactly what they saw before. The pots sit under the PreFundedSpecializedBalances root sum tree, which the per-block total credits check already sums, so the credits stay accounted for while they wait to be claimed.

FeeResult

All fee calculations produce a FeeResult:

#![allow(unused)]
fn main() {
pub struct FeeResult {
    pub storage_fee: Credits,
    pub processing_fee: Credits,
    pub fee_refunds: FeeRefunds,
    pub removed_bytes_from_system: u32,
}
}
  • storage_fee — credits for new bytes written to persistent storage
  • processing_fee — credits for computation and I/O
  • fee_refunds — credits returned because previously stored data was deleted
  • removed_bytes_from_system — bytes removed that were stored by the system (not by any identity), so no refund is issued

The total base fee is storage_fee + processing_fee. The FeeResult is produced by Drive::apply_drive_operations(), which executes the GroveDB operations and measures the actual cost of each insert, delete, and query.

Refunds

When data is removed from GroveDB (a document is deleted, a key is removed), the system calculates a refund of the original storage fee. Refunds are tracked per identity per epoch:

#![allow(unused)]
fn main() {
pub struct FeeRefunds(pub CreditsPerEpochByIdentifier);
// BTreeMap<IdentifierBytes32, BTreeMap<EpochIndex, Credits>>
}

Refunds are not 1:1 with the original fee because storage fees are distributed across future epochs (see below). The refund amount depends on how many epochs have elapsed since the data was stored — the longer the data has been stored, the smaller the refund, because more of the distributed fees have already been paid out to proposers.

There is a dust limit: refunds below 32 bytes worth of storage credits are discarded to prevent micro-refund spam.

Epoch-Based Fee Distribution

Fees do not go directly to the block proposer. Instead, they accumulate in epoch-specific pools and are distributed to proposers at epoch boundaries.

How Epochs Work

  • An epoch is a fixed window of blocks
  • An era consists of 40 epochs
  • Storage fees are distributed across 50 eras (2,000 epochs, roughly 50 years) using a declining schedule

The distribution table allocates percentages per era:

EraPercentageCumulative
05.000%5.0%
14.800%9.8%
24.600%14.4%
.........
490.125%100.0%

Within each era, the percentage is divided equally among the era's epochs. For example, a 1,000,000-credit storage fee distributes 50,000 credits (5%) to era 0, split evenly across 40 epochs = 1,250 credits per epoch.

Fee Flow

Block execution
  └→ FeeResult (storage + processing)
       └→ End of block: add_distribute_block_fees_into_pools()
            ├→ Processing fees → current epoch pool
            └→ Storage fees → global distribution pool → spread across future epochs

Epoch change
  └→ add_distribute_fees_from_oldest_unpaid_epoch_pool_to_proposers()
       ├→ Calculate Core block rewards for the epoch
       ├→ Add Core rewards to system credits
       └→ Distribute (Platform fees + Core rewards) to proposers

Processing fees are paid to proposers at the end of the epoch in which they were collected. Storage fees are spread across 50 eras of future epochs, providing a long-term revenue stream for validators.

Fee Versioning

All fee parameters are versioned through FeeVersion, stored in PlatformVersion. This allows the protocol to adjust fee rates without a hard fork — a new protocol version simply references different fee constants.

The current fee version structure:

#![allow(unused)]
fn main() {
pub struct FeeVersion {
    pub fee_version_number: FeeVersionNumber,
    pub uses_version_fee_multiplier_permille: Option<u64>,
    pub storage: FeeStorageVersion,
    pub signature: FeeSignatureVersion,
    pub hashing: FeeHashingVersion,
    pub processing: FeeProcessingVersion,
    pub data_contract_validation: FeeDataContractValidationVersion,
    pub data_contract_registration: FeeDataContractRegistrationVersion,
    pub state_transition_min_fees: StateTransitionMinFees,
    pub vote_resolution_fund_fees: VoteResolutionFundFees,
}
}

Fee versions are stored in the FEE_VERSIONS array and looked up by number. The uses_version_fee_multiplier_permille field allows a global scaling factor (permille = divide by 1000; a value of 1000 means no change).

Key Source Files

FileContents
rs-platform-version/src/version/fee/All fee version definitions
rs-platform-version/src/version/fee/storage/v1.rsStorage fee rates
rs-platform-version/src/version/fee/signature/v1.rsSignature verification costs
rs-platform-version/src/version/fee/state_transition_min_fees/v1.rsMinimum fees per transition
rs-platform-version/src/version/fee/data_contract_registration/v2.rsContract registration fees
rs-platform-version/src/version/fee/data_contract_registration/v3.rsProtocol version 14 addition: once-per-identity distribution surcharge
rs-drive/src/fees/op.rsLowLevelDriveOperation and cost calculation
rs-dpp/src/fee/fee_result/mod.rsFeeResult, BalanceChangeForIdentity
rs-dpp/src/fee/epoch/distribution.rsEpoch distribution table and refund logic
rs-drive-abci/src/execution/types/execution_event/mod.rsExecutionEvent enum
rs-drive-abci/src/execution/platform_events/fee_pool_inwards_distribution/Block fee collection
rs-drive-abci/src/execution/platform_events/fee_pool_outwards_distribution/Proposer payout

Platform Address Fees

Protocol versions 10 and 11 introduced a new class of state transitions that operate on platform addresses rather than identities. Platform addresses are derived from public keys (similar to Bitcoin addresses) and hold a credit balance directly, without requiring an identity to be registered.

This chapter explains how fees work for address-based transitions, how they differ from the identity credit model, and how the fee strategy mechanism gives clients control over fee deduction.

Background: Why Platform Addresses?

Before protocol version 10, every action on the platform required an identity — a registered entity funded by an asset lock transaction. Creating an identity required a Core transaction, waiting for confirmations, and then submitting a state transition. This was a multi-step process that created friction for simple operations like "send credits to an address."

Platform addresses solve this by allowing credits to exist at addresses without a full identity. Users can fund an address (via an asset lock or a transfer from another address) and then spend from it directly using a signature from the address's key pair.

Address-Based State Transitions

Protocol versions 10 and 11 added the following transition types:

TransitionProtocol VersionDescription
IdentityCreateFromAddresses10Create an identity funded from platform address balances
IdentityTopUpFromAddresses10Add credits to an existing identity from address balances
AddressFundsTransfer11Transfer credits between platform addresses
AddressFundingFromAssetLock11Fund an address directly from an asset lock
AddressCreditWithdrawal11Withdraw credits from an address back to Core

All of these use the PaidFromAddressInputs execution event variant.

The Input/Output Model

Address-based transitions follow a UTXO-inspired model with inputs and outputs:

Inputs

Each input specifies a platform address, the expected nonce, and the amount to consume from that address's balance:

inputs: [
    { address: A, nonce: 5, amount: 100_000_000 },
    { address: B, nonce: 3, amount:  50_000_000 },
]

The platform validates each input by:

  1. Checking the address exists in state
  2. Verifying the nonce is exactly current_nonce + 1 (replay protection)
  3. Confirming the address has sufficient balance for the requested amount

After validation, the remaining balance of each input is tracked:

remaining = actual_balance - requested_amount

This remaining balance is what is available to pay fees from (after the requested amount is committed to outputs).

Outputs

Outputs specify destination addresses and the credits to send:

outputs: [
    { address: C, amount: 80_000_000 },
    { address: D, amount: 60_000_000 },
]

Outputs are added to recipient balances. They can also be reduced to pay fees (see Fee Strategy below).

Balance Equation

The fundamental constraint is:

sum(input_amounts) >= sum(output_amounts) + fees

If the inputs cannot cover both the outputs and the fees, the transition is rejected with AddressesNotEnoughFundsError.

Fee Strategy

Unlike identity-based transitions where fees are always deducted from the identity's balance, address-based transitions use an explicit fee strategy that the client includes in the transition. The fee strategy is an ordered sequence of steps that tells the platform where to find the fee credits:

#![allow(unused)]
fn main() {
pub enum AddressFundsFeeStrategyStep {
    /// Deduct fee from a specific input address's remaining balance.
    DeductFromInput(u16),
    /// Reduce a specific output's amount to cover the fee.
    ReduceOutput(u16),
}

pub type AddressFundsFeeStrategy = Vec<AddressFundsFeeStrategyStep>;
}

How It Works

The platform processes the steps in order. At each step, it deducts as much of the remaining fee as possible from the specified source:

fee_strategy: [DeductFromInput(0), ReduceOutput(0)]

Step 1: Try to deduct full fee from input 0's remaining balance
  - remaining_fee = 10,000,000
  - input_0_remaining = 25,000,000
  - deducted = 10,000,000
  - input_0_remaining = 15,000,000
  - remaining_fee = 0 → done

If the first source is insufficient, the algorithm moves to the next step:

fee_strategy: [DeductFromInput(0), ReduceOutput(1)]

Step 1: Try to deduct from input 0
  - remaining_fee = 10,000,000
  - input_0_remaining = 3,000,000
  - deducted = 3,000,000
  - input_0_remaining = 0 (removed)
  - remaining_fee = 7,000,000

Step 2: Try to reduce output 1
  - remaining_fee = 7,000,000
  - output_1_amount = 60,000,000
  - deducted = 7,000,000
  - output_1_amount = 53,000,000
  - remaining_fee = 0 → done

Index Stability

The indices in the fee strategy refer to the original BTreeMap iteration order. The implementation snapshots the address lists before processing any steps, so removing a drained entry at step 1 does not shift the indices for step 2. This is critical for correctness:

#![allow(unused)]
fn main() {
let input_addresses: Vec<PlatformAddress> = inputs.keys().copied().collect();
let output_addresses: Vec<PlatformAddress> = outputs.keys().copied().collect();

for step in fee_strategy {
    match step {
        DeductFromInput(index) => {
            let address = input_addresses[*index as usize];
            // look up by address, not by index into the live BTreeMap
        }
        // ...
    }
}
}

FeeDeductionResult

The deduction algorithm produces:

#![allow(unused)]
fn main() {
pub struct FeeDeductionResult {
    pub remaining_input_balances: BTreeMap<PlatformAddress, (AddressNonce, Credits)>,
    pub adjusted_outputs: BTreeMap<PlatformAddress, Credits>,
    pub fee_fully_covered: bool,
}
}

If fee_fully_covered is false, the transition is rejected.

Nonce System (Replay Protection)

Every platform address has a nonce that starts at 0 and increments by 1 with each transition that uses the address as an input. The transition must specify the expected nonce, which must be exactly current_nonce + 1:

#![allow(unused)]
fn main() {
let expected_next_nonce = state_nonce.saturating_add(1);
if provided_nonce != expected_next_nonce {
    return Err(AddressInvalidNonceError {
        expected: expected_next_nonce,
        provided: provided_nonce,
    });
}
}

This prevents:

  • Replay attacks — resubmitting an old transition (wrong nonce)
  • Double-spending — using the same balance twice (nonce already consumed)
  • Ordering attacks — submitting transitions out of order (nonce gap)

If a nonce reaches u32::MAX, the address is exhausted and cannot be used as an input anymore.

Multiple addresses can be used as inputs in a single transition, each with its own nonce. The platform enforces a maximum number of inputs per transition (configured in platform_version.dpp.state_transitions.max_address_inputs).

The PaidFromAddressInputs Event

When an address-based transition passes validation, the processor creates a PaidFromAddressInputs execution event:

#![allow(unused)]
fn main() {
ExecutionEvent::PaidFromAddressInputs {
    input_current_balances: BTreeMap<PlatformAddress, (AddressNonce, Credits)>,
    added_to_balance_outputs: BTreeMap<PlatformAddress, Credits>,
    fee_strategy: AddressFundsFeeStrategy,
    operations: Vec<DriveOperation>,
    execution_operations: Vec<ValidationOperation>,
    additional_fixed_fee_cost: Option<Credits>,
    user_fee_increase: UserFeeIncrease,
}
}
  • input_current_balances — the remaining balance of each input after consuming the requested amounts, plus the validated nonce
  • added_to_balance_outputs — the output amounts before any fee deductions
  • fee_strategy — the client's ordered fee deduction instructions
  • operations — the GroveDB operations (balance updates, nonce bumps)
  • execution_operations — validation operations that also incur fees
  • additional_fixed_fee_cost — optional fixed costs (e.g., registration fees)
  • user_fee_increase — voluntary processing fee multiplier

Fee Validation Pipeline

Address-based fee validation runs in two phases:

Phase 1: Pre-Check (Minimum Balance)

Before expensive state reads, a quick estimate verifies that the inputs have enough credits to cover the minimum possible fee:

#![allow(unused)]
fn main() {
fn validate_addresses_minimum_balance_pre_check(
    &self,
    remaining_address_balances: &BTreeMap<PlatformAddress, (AddressNonce, Credits)>,
    platform_version: &PlatformVersion,
) -> Result<SimpleConsensusValidationResult, Error>
}

This catches obviously insufficient balances early. Only AddressFundsTransfer, AddressCreditWithdrawal, IdentityCreateFromAddresses, and IdentityTopUpFromAddresses run this pre-check. AddressFundingFromAssetLock skips it because its funds come from the asset lock, not existing address balances.

Phase 2: Full Fee Validation

After the operations are determined, the platform:

  1. Applies all drive operations (in estimation mode) to calculate the actual FeeResult
  2. Adds validation operation costs
  3. Applies the user_fee_increase multiplier
  4. Adds any additional_fixed_fee_cost
  5. Runs the fee strategy to deduct the total from inputs/outputs
  6. Checks fee_fully_covered
#![allow(unused)]
fn main() {
let fee_deduction_result = deduct_fee_from_outputs_or_remaining_balance_of_inputs(
    input_current_balances.clone(),
    added_to_balance_outputs.clone(),
    fee_strategy,
    required_balance,
    platform_version,
)?;

if !fee_deduction_result.fee_fully_covered {
    return Err(AddressesNotEnoughFundsError);
}
}

Fee Execution

When the transition is executed, the fee deduction runs a second time with the real (not estimated) fee amount, and the adjusted balances are written to state:

  1. Apply all drive operations — the core state changes (transfers, nonce bumps, etc.)
  2. Calculate actual fee — from the FeeResult of the applied operations
  3. Deduct fee — run the fee strategy against the actual fee amount
  4. Adjust outputs — if any output was reduced, call remove_balance_from_address for the difference
  5. Adjust inputs — if any input's remaining balance was reduced, call set_balance_to_address with the adjusted amount
  6. Apply adjustment operations — batch the balance corrections into GroveDB

The fee goes into the epoch pool just like identity-based fees, and is distributed to proposers at epoch end.

Minimum Fee Calculation

For AddressFundsTransfer, the minimum fee scales with the number of inputs and outputs:

min_fee = num_inputs × address_funds_transfer_input_cost
        + num_outputs × address_funds_transfer_output_cost

With current constants:

InputsOutputsMinimum Fee
116,500,000
1212,500,000
217,000,000
2213,000,000

For IdentityCreateFromAddresses:

min_fee = identity_create_base_cost + num_keys × identity_key_in_creation_cost
KeysMinimum Fee
18,500,000
215,000,000
321,500,000

Key Differences from Identity Fees

AspectIdentity Credit FeesPlatform Address Fees
Fee sourceIdentity balance (single pool)Input addresses + output reduction
Fee deductionAutomatic from identityExplicit fee strategy
Debt allowedYes (negative balance tracked)No — must have funds
RefundsYes — deleted storage refunded to identityNo refund mechanism
NoncePer identity, per contractPer address, monotonic u32
User fee increaseApplied to processing feesApplied to processing fees
Minimum feePer transition typePer input + per output
ExecutionEventPaid / PaidFromAssetLockPaidFromAddressInputs
Error on insufficientBalanceIsNotEnoughErrorAddressesNotEnoughFundsError

No Debt, No Refunds

The most significant difference is that address-based transitions have no debt mechanism and no refund mechanism:

  • Identity fees can create a negative balance when the processing fee cannot be fully covered. This debt is tracked and must be repaid before the identity can submit new transitions: credits the identity receives while its balance is empty repay the debt first. From protocol version 14 the repaid part goes to the processing fee pool of the epoch it is repaid in, where the unpaid fee would have gone.

  • Address fees must be fully covered by available inputs and outputs. If the fee cannot be paid, the transition is rejected outright.

  • Identity refunds return credits to the identity when stored data is deleted. The refund is calculated per epoch based on when the data was originally stored.

  • Address-based transitions do not produce refundable storage. If an address-based transition stores data and that data is later deleted, no refund is issued to any address.

Example: AddressFundsTransfer

A concrete example of how fees flow for a two-input, two-output transfer:

Transition:
  inputs:  [{A, nonce: 5, amount: 1_000_000_000}, {B, nonce: 3, amount: 500_000_000}]
  outputs: [{C, amount: 800_000_000}, {D, amount: 600_000_000}]
  fee_strategy: [DeductFromInput(0), DeductFromInput(1)]
  user_fee_increase: 0

1. Validate inputs:
   A: state_nonce=4, balance=2_000_000_000 → nonce 5 ✓, balance ≥ 1B ✓
   B: state_nonce=2, balance=700_000_000  → nonce 3 ✓, balance ≥ 500M ✓

   Remaining: A=(5, 1_000_000_000), B=(3, 200_000_000)

2. Pre-check minimum fee:
   min = 2 × 500,000 + 2 × 6,000,000 = 13,000,000
   sum(remaining) = 1,200,000,000 >> 13M ✓

3. Create PaidFromAddressInputs event:
   input_current_balances: {A: (5, 1B), B: (3, 200M)}
   added_to_balance_outputs: {C: 800M, D: 600M}

4. Calculate actual fee:
   storage_fee = 14,000,000  (hypothetical)
   processing_fee = 2,500,000
   total = 16,500,000

5. Apply fee strategy:
   Step 1: DeductFromInput(0) → A: 1B - 16.5M = 983,500,000
   remaining_fee = 0 ✓

6. Execute:
   A.balance = 983,500,000, A.nonce = 5
   B.balance = 200,000,000, B.nonce = 3
   C.balance += 800,000,000
   D.balance += 600,000,000
   16,500,000 → epoch fee pool

Key Source Files

FileContents
rs-dpp/src/address_funds/fee_strategy/mod.rsAddressFundsFeeStrategy definition
rs-dpp/src/address_funds/fee_strategy/deduct_fee_from_inputs_and_outputs/Fee deduction algorithm
rs-drive-abci/src/execution/types/execution_event/mod.rsPaidFromAddressInputs variant
rs-drive-abci/src/execution/validation/.../processor/traits/address_balances_and_nonces.rsNonce and balance validation
rs-drive-abci/src/execution/validation/.../processor/traits/addresses_minimum_balance.rsMinimum balance pre-check
rs-drive-abci/src/execution/platform_events/.../validate_fees_of_event/Full fee validation
rs-drive-abci/src/execution/platform_events/.../execute_event/Fee execution with adjustments
rs-platform-version/src/version/fee/state_transition_min_fees/v1.rsAddress fee constants

Shielded Transaction Fees

Introduced in protocol version 12. For the general fee system overview, see Fee System Overview. For address-based fees (protocol versions 10--11), see Platform Address Fees.

Shielded transactions use the Orchard protocol's zero-knowledge proofs to hide transaction amounts. Because amounts are hidden, the platform cannot inspect the transaction to compute fees the way it does for transparent transitions. Instead, the fee is embedded into the cryptographic structure of the bundle itself, and the platform enforces a minimum.

This chapter explains the fee model, how it is validated, and how it differs from the transparent and address-based fee systems.

The Problem: Fees in a Privacy System

In a transparent state transition like AddressFundsTransfer, the platform can see the transfer amount, compute the cost of storage and processing, and deduct the fee from the sender's balance. The fee calculation happens after the transition is applied.

Shielded transitions break this model. The amounts inside the ZK proof are hidden. The platform cannot look inside the proof to determine how much was sent or received. It can only see two things from the public fields:

  1. value_balance — the net amount leaving the shielded pool (positive means credits flow out of the pool into the transparent world or to proposers as fees)
  2. num_actions — the number of spend+output pairs in the bundle

The fee must therefore be encoded into value_balance by the client and validated by the platform before execution.

Fee Extraction by Transition Type

The fee is derived differently depending on the shielded transition type:

TransitionFee FormulaExplanation
Shieldfee = metered(storage + processing) + shielded_verification_fee, paid from transparent address inputsCharged on the transparent side (not from value_balance), on top of the shielded amount. The storage and processing of the note/nullifier writes are metered by GroveDB; only the ZK compute fee (proof + num_actions × per_action_processing) is added on top. Skipped by the value_balance-based shielded fee validation; enforced through the address-input fee path. See Entry-Transition Fees.
ShieldedTransferfee = value_balance (pinned to the minimum)The entire value_balance is the fee and must equal compute_minimum_shielded_fee(num_actions) exactly (overpayment is rejected). Nothing leaves the pool except the fee.
Unshieldfee = compute_minimum_shielded_fee(num_actions) + unshield_address_storage_feevalue_balance (the transition's unshielding_amount) is the gross amount leaving the pool. The output address receives unshielding_amount − fee; validation requires unshielding_amount ≥ fee. Unshield also writes the net to the output platform address (AddBalanceToAddress), a real storage write priced on top of the base shielded minimum (unshield_address_storage_fee = 222 × per_byte_rate, ≈6.08M credits, flat regardless of action count — 222 bytes is the storage portion of the ≈6.24M metered address write) so the address write is covered and the proof fee isn't diverted to pay for it. See Per-Action Storage Fee.
ShieldedWithdrawalfee = compute_minimum_shielded_fee(num_actions) + withdrawal_document_storage_feevalue_balance (unshielding_amount) is the gross amount leaving the pool. The Core withdrawal document receives unshielding_amount − fee (which must also clear MIN_WITHDRAWAL_AMOUNT). Unlike the other pool-paid transitions, ShieldedWithdrawal also writes a Core withdrawal document — a real document insert into the withdrawals contract plus its index entries (AddWithdrawalDocument), with a real metered cost of ≈110M credits that is flat regardless of action count. That cost is priced on top of the base shielded minimum as a flat ~4,100-byte storage component (withdrawal_document_storage_fee = 4100 × per_byte_rate), so the document write is covered and the proof-verification fee isn't diverted from the proposer to pay for it. See Per-Action Storage Fee.
ShieldFromAssetLockpool_fee = compute_minimum_shielded_fee(num_actions) + asset_lock_base_cost, paid from the asset lockThe flat shielded minimum plus the asset-lock processing base cost is routed to the fee pools. Any remaining asset-lock value (the surplus) goes to an optional signed surplus_output platform address, or — if none is set — folds into the fee pools up to shielded_implicit_fee_cap. See Entry-Transition Fees.
IdentityCreateFromShieldedPooltotal_fee = metered(insert_nullifiers + AddNewIdentity(identity + N keys)) + shielded_verification_fee, moved from the new identity's balancevalue_balance is a fixed denomination (a member of the versioned set {0.1, 0.3, 0.5, 1.0} DASH) and must equal it EXACTLY. The new identity is created holding the full denomination, funded by decrementing the shielded pool by exactly that amount — a move between two balance trees (like Unshield's pool→address), so the global system-credit supply is unchanged (no AddToSystemCredits); the fee is then moved from that balance into the fee pools, so the identity ends with denomination − total_fee. Unlike the flat pool-paid transitions, the AddNewIdentity write grows with the key count, so the cost is metered (not a flat carve) — only the ZK compute fee (compute_shielded_verification_fee) is added on top, exactly like the transparent Shield. The client predicts it offline with compute_shielded_identity_create_fee(num_actions, num_keys); consensus rejects denomination < total_fee with IdentityInsufficientBalanceError.
ShieldFromIdentityfee = metered(storage + processing) + shielded_verification_fee, paid from the funding identity's balanceIdentity balance to pool (protocol version 14). Charged exactly like Shield, but on the identity side: the identity signature covers the whole outputs-only bundle, the metered note writes and identity writes go through the standard identity-paid path (IdentityCreditTransferToAddresses model), and only the ZK compute fee is added as additional_fixed_fee_cost. user_fee_increase applies. The identity must hold amount + fee; consensus rejects a short balance with IdentityInsufficientBalanceError. The pool and the identity are both balance trees, so no system-credit adjustment is emitted. See Entry-Transition Fees.
IdentityTopUpFromShieldedPoolfee = compute_shielded_identity_top_up_fee(num_actions) = compute_minimum_shielded_fee(num_actions) + identity_balance_storage_fee, carved from value_balanceShielded pool to an EXISTING identity's balance (protocol version 14). value_balance (the transition's topUpAmount) is the gross amount leaving the pool; the identity receives topUpAmount - fee and validation requires topUpAmount >= fee. Same flat pool-paid model as Unshield, with the identity balance write as a flat component built like Unshield's address write but calibrated to its measured cost: the top-up rewrites the existing identity's balance element and its Merk path (320 replaced bytes, 175,320 credits of processing, no storage), folded into one flat figure with headroom like the other shielded components, so identity_balance_storage_fee = 8 x per_byte_rate (SHIELDED_IDENTITY_TOP_UP_BALANCE_STORAGE_BYTES). The target identity and gross amount are bound into the Orchard sighash; the identity must already exist; no system-credit adjustment.

Token shielded pool fees

Token pools (protocol version 14, see Token Shielded Pools) hold tokens, and tokens cannot pay fees, so none of the three token pool transitions carves a fee from the bundle. They are TokenTransition variants inside a Batch, and the batch's signing identity pays in credits through the standard identity-paid path.

TransitionFee FormulaExplanation
TokenShieldfee = metered(storage + processing) + shielded_verification_fee, paid by the signing identitySame model as ShieldFromIdentity: the dummy nullifier inserts, the note appends, the identity token balance write and the pool balance write are metered, and compute_shielded_verification_fee(num_actions) is added as a precalculated operation before the proof is verified. value_balance is -amount in tokens and carries no fee.
TokenUnshieldfee = metered(storage + processing) + shielded_verification_fee, paid by the signing identityvalue_balance equals the unshielded token amount exactly; the recipient receives the full amount. Nullifier inserts, note appends and the two balance writes are metered.
TokenShieldedTransferfee = metered(storage + processing) + shielded_verification_fee, paid by the signing identityvalue_balance is exactly zero; consensus rejects any other value. Only the nullifier inserts and note appends are metered.
TokenMintToPool, TokenClaimToPool, TokenDirectPurchaseToPoolsame modelOutputs-only bundles; the dummy nullifier inserts, the note appends and the supply and pool balance writes are metered, the verification fee is charged by the action transformer so CheckTx and block execution price the bundle identically.
TokenBurnFromPoolsame modelNullifier inserts, change note appends and the supply and pool balance writes are metered.
Document with a TokenPaymentInfo::V1same model, on top of the document's own feeThe shielded payment bundle's verification fee is added by the batch transformer before anything can fail; the pool writes it causes are metered.

Because the verification fee is charged before the proof is checked, an invalid proof is a paid failure: the identity is charged, its identity contract nonce advances, and no token moves.

Identity-less token pool transitions

TokenShieldedTransferWithShieldedFee (26), TokenUnshieldWithShieldedFee (27) and TokenPurchaseFromShieldedPool (28) have no identity: the fee is carved from a second bundle spent in the credit shielded pool, exactly as ShieldedTransfer carves its own fee.

TransitionFee FormulaExplanation
TokenShieldedTransferWithShieldedFeecredit_amount == base(fee_actions) + base(token_actions)Two bundles are verified and stored, so the fee is the base shielded fee of each (compute_token_pool_paid_shielded_fee). Pure fee: overpayment is rejected.
TokenUnshieldWithShieldedFeecredit_amount == base(fee_actions) + base(token_actions) + identity balance bytesAdds the flat storage of the recipient's token balance item. Pure fee.
TokenPurchaseFromShieldedPoolcredit_amount == total_agreed_price + base(fee_actions) + base(token_actions) + balance write + supply bytesThe agreed price rides on top of the fee and is credited to the contract owner; credit_amount - total_agreed_price must equal the fee exactly.

The fee bundle's value balance is credit_amount; the minimum-fee validation, the SDK builders and the transformer all use the same compute_token_* function so the threshold never drifts from what is carved. An invalid proof or a spent nullifier is an unpaid rejection: nothing is committed and no fee is charged, the same as for the credit pool's pool-paid transitions.

For ShieldedTransfer, the client constructs the bundle so that total_spent − total_output = desired_fee. The Orchard circuit proves that value is conserved (inputs = outputs + value_balance), and the binding signature cryptographically commits to the value_balance. Mutating value_balance after signing will cause the binding signature to fail verification.

Entry-Transition Fees (Shield, ShieldFromAssetLock, and ShieldFromIdentity)

The entry transitions — Shield (transparent → shielded), ShieldFromAssetLock (Core asset lock → shielded), and, from protocol version 14, ShieldFromIdentity (identity balance → shielded) — move value into the pool, so there is no spent note from which value_balance could carry a fee. Their fees are therefore charged from the funding side, and both cover the same Halo 2 proof verification and per-action work the other shielded transitions pay for — but they account for it differently. Shield debits a state-queryable transparent address balance, so GroveDB meters its real storage/processing and only the compute portion (compute_shielded_verification_fee, no storage term) is added on top. ShieldFromAssetLock is funded by a consumed asset lock with no metering anchor, so it pays the flat compute_minimum_shielded_fee(num_actions) (plus the asset-lock base cost). num_actions is the on-wire action count of the bundle (a single-output, spends-disabled Orchard bundle pads to 2 actions, so the minimum is the 2-action fee).

Every action of an outputs-only bundle still reveals a nullifier, that of a dummy spend, which becomes the new note's rho. From protocol version 14 the entry transitions record those nullifiers and refuse one repeated inside the bundle or already recorded (NullifierAlreadySpentError), as the spends do, so every revealed nullifier is recorded once. Shield and ShieldFromIdentity meter the nullifier writes like the rest of their storage; ShieldFromAssetLock's flat fee already prices a note and a nullifier write per action (see Per-Action Storage Fee).

Shield

Shield is charged like any other address-funded transition: GroveDB meters the real storage and processing cost of applying it (the note-commitment and nullifier writes plus the address-balance updates), and the shielded compute fee is added on top:

fee = metered_storage + metered_processing + shielded_verification_fee
shielded_verification_fee = proof_verification_fee + num_actions × per_action_processing_fee

shielded_verification_fee is the ZK-verification cost (Halo 2 proof + per-action spend-auth verification) that GroveDB metering cannot see. It is added as the transition's additional_fixed_fee_cost — exactly the mechanism IdentityCreateFromAddresses uses for its registration cost. It carries no storage term: storage comes entirely from metering, so it is never double-counted. The address inputs must cover shield_amount + fee, and the booked storage/processing equals the deducted amount, so credits are conserved by the standard machinery (no special-case override).

Shield is skipped by the value_balance-based minimum-fee validation (its value_balance is the amount entering the pool, not a fee). The stateless structure floor requires only shield_amount + shielded_verification_fee (a conservative lower bound, since metered storage is unknowable without state); the authoritative metered + compute funding gate is validate_fees_of_event.

ShieldFromIdentity

ShieldFromIdentity (protocol version 14) is Shield with the identity balance as the funding side. It is identity-signed (TRANSFER key, identity nonce) like IdentityCreditTransferToAddresses, and carries the same outputs-only Orchard bundle as Shield. The fee model is identical to Shield's: GroveDB meters the note and nullifier inserts and the identity balance and nonce writes, and the shielded compute fee is added as additional_fixed_fee_cost:

fee = metered_storage + metered_processing + shielded_verification_fee
identity_balance_after = identity_balance_before - amount - fee

user_fee_increase applies to the metered processing portion. The stateless floor requires identity_balance >= amount + compute_shielded_identity_balance_write_fee, the conservative complete admission estimate: compute_minimum_shielded_fee, plus PV14's versioned shielded_identity_action_write_storage_bytes (400 effective bytes per action) and shielded_identity_balance_write_storage_bytes (500 flat bytes), each priced at the storage rate. The allowances cover the complete execution-event estimate: note/nullifier writes at the estimator's depth 16, identity nonce/balance writes at its maximum-element depth, and the signature and state-read validation context. They are admission reserves, not physical payload sizes or changes to the actual metered charge. For two actions the base wallet estimate is 114,140,000 + (2 × 400 + 500) × 27,400 = 149,760,000 credits. Historical tables preserve the preceding zero per-action and 20-byte flat allowance. An identity that could not pay the complete fee is therefore refused before the Orchard proof is verified. The authoritative gate is the identity-paid fee validation of the execution event (Paid), which rejects with IdentityInsufficientBalanceError. The identity balance and the pool total are both terms of the block conservation equation, so the converter emits no AddToSystemCredits (the same rule Unshield and IdentityCreateFromShieldedPool follow).

A failed Orchard proof is a paid failure, not a free rejection. Like ShieldFromAssetLock, the proof is verified inside the transition's own transform rather than in the shared stateless proof step: on failure the transition executes as a BumpIdentityNonceAction, consuming the identity nonce and charging the identity the versioned shielded_proof_verification_failure penalty on top of the processing metered so far. This closes the path where a funded identity could resubmit invalid proofs indefinitely with its nonce and balance left untouched. CheckTx never charges; it admits the verification under its node-local proof budget only after the cheap checks passed, and rejects on failure.

ShieldFromAssetLock

The asset lock funds the pool, so the fee is taken from the consumed asset-lock value. The pool fee is:

pool_fee = compute_minimum_shielded_fee(num_actions) + asset_lock_base_cost

asset_lock_base_cost is the same asset-lock-proof processing base cost charged to every asset-lock-funded transition (e.g. IdentityCreate): required_asset_lock_duff_balance_for_processing_start_for_address_funding (50,000 duffs) × CREDITS_PER_DUFF (1,000) = 50,000,000 credits. Adding it makes the ShieldFromAssetLock pool fee strictly greater than the bare F paid by the transparent Shield, pricing the extra cost of verifying the Core asset-lock proof.

The asset lock must cover shield_amount + pool_fee; the remainder is the surplus:

surplus = consumed_asset_lock_value − shield_amount − pool_fee   (always ≥ 0)

The surplus is disposed of in one of two ways:

  • surplus_output set — the transition carries an optional Option<PlatformAddress> surplus_output. When present, surplus is credited to that platform address (via an AddBalanceToAddress drive operation). This field is part of the signed payload (it sits before the signature field, which alone is excluded from the sighash), so a surplus recipient cannot be substituted or truncated after signing.
  • surplus_output unset — the surplus folds into the fee pools, but only up to shielded_implicit_fee_cap (20,000,000,000 credits = 0.2 DASH, a versioned constant). If the unclaimed surplus would exceed the cap, the transition is rejected with ShieldedImplicitFeeCapExceededError so a client cannot accidentally donate a large remainder to proposers. To intentionally over-fund, the client must set surplus_output (which has no cap).

Value conservation across the whole transition is exact:

consumed_asset_lock_value = shield_amount + surplus_amount + fee_amount

where surplus_amount is surplus when surplus_output is set and 0 otherwise (in which case the surplus is part of fee_amount).

The Three-Component Fee Model

The minimum shielded fee has three components:

min_fee = proof_verification_fee + num_actions × (processing_fee + storage_fee)

1. Proof Verification Fee (per bundle)

A single Halo 2 ZK proof covers the entire bundle regardless of action count. The bundle's base verification work is benchmarked at approximately 5 ms. This is a fixed cost per bundle.

Protocol version 14 value: 40,000,000 credits (40M)

2. Per-Action Processing Fee

The per-action processing fee prices the marginal Halo 2 verification work that each additional action adds to the bundle (≈1.1 ms/action measured against a ≈5 ms bundle base): a bundle with more actions is a larger circuit and a longer batch verification. For a spend-bearing action that marginal work includes:

  • RedPallas spend authorization signature verification
  • Nullifier duplicate check (hash + tree lookup)
  • Note commitment insertion into the Sinsemilla-based Merkle tree

Output-only entry transitions (Shield / ShieldFromAssetLock / ShieldFromIdentity) do no spends, but each output action still enlarges the proof and so carries the same per-action processing charge — this fee tracks the marginal verification work, not a fixed per-action checklist. From protocol version 14 it also prices the check of the nullifier each of their actions reveals.

At protocol version 14, the proof and per-action fees are versioned independently. Their numerical ratio is about 1.8:1 (40M : 22M).

Current value: 22,000,000 credits (22M)

3. Per-Action Storage Fee

Each action permanently stores data in two places:

StorageBytesContents
BulkAppendTree (commitment tree)31232 cmx + 32 rho + 32 cv_net + 216 encrypted note
Nullifier tree32nullifier key (value is empty)
Total physical payload344

Protocol version 14 prices a 550-byte allowance per action, covering the physical payload and database framing, at the platform's per-byte storage rates:

storage_fee_per_action = 550 × (storage_disk_usage_credit_per_byte
                              + storage_processing_credit_per_byte)
                       = 550 × (27,000 + 400)
                       = 550 × 27,400
                       = 15,070,000

The byte allowance is versioned. Its per-byte rates come from the storage fee version, so the fee tracks changes to those rates.

Fee Table

Combining all three components at protocol version 14:

ActionsProof FeeProcessingStorageTotal Minimum Fee
240,000,00044,000,00030,140,000114,140,000
340,000,00066,000,00045,210,000151,210,000
440,000,00088,000,00060,280,000188,280,000

Note: The Orchard protocol requires a minimum of 2 actions per bundle for privacy (even a single-input single-output transfer produces 2 actions with a dummy padding action). Bundles with 1 action are structurally invalid.

The totals above are the base compute_minimum_shielded_fee and apply directly to ShieldedTransfer. The three pool-paid transitions that write one extra per-transition output add a flat component on top of this base:

  • Unshield adds the output-address write cost: a flat unshield_address_storage_fee = 222 × per_byte_rate = 222 × 27,400 = 6,082,800 credits, independent of action count. So the 2-action Unshield fee is 114,140,000 + 6,082,800 = 120,222,800 credits (and likewise +6,082,800 at every action count). See the Fee Extraction Unshield row for why this component exists.
  • ShieldedWithdrawal adds the Core withdrawal-document storage cost: a flat withdrawal_document_storage_fee = 4100 × per_byte_rate = 4100 × 27,400 = 112,340,000 credits, independent of action count. So the 2-action ShieldedWithdrawal fee is 114,140,000 + 112,340,000 = 226,480,000 credits (and likewise +112,340,000 at every action count). See the Fee Extraction ShieldedWithdrawal row for why this component exists.
  • IdentityTopUpFromShieldedPool adds the identity balance write cost: a flat identity_balance_storage_fee = 8 × per_byte_rate = 8 × 27,400 = 219,200 credits, independent of action count, so the top-up fee at any action count is the base plus 219,200. See the Fee Extraction IdentityTopUpFromShieldedPool row for why this component is so much smaller than the address write: it rewrites an existing balance element instead of storing a new entry.

Where Fee Validation Runs

Fee validation is integrated into the processor pipeline (see Validation Pipeline) between basic structure validation and ZK proof verification:

... → Basic Structure → Minimum Fee Check → ZK Proof Verification → ...

The ordering is deliberate. The fee check is stateless and cheap — it only reads value_balance and actions.len() from the transition, with no GroveDB lookups. Placing it before proof verification means that bundles with insufficient fees are rejected instantly, without spending ~100ms on Halo 2 verification.

The implementation lives in packages/rs-drive-abci/src/execution/validation/state_transition/processor/traits/shielded_proof.rs:

#![allow(unused)]
fn main() {
pub(crate) trait StateTransitionShieldedMinimumFeeValidationV0 {
    fn validate_minimum_shielded_fee(
        &self,
        platform_version: &PlatformVersion,
    ) -> Result<SimpleConsensusValidationResult, Error>;
}
}

If the fee is below the minimum, the transition is rejected with InsufficientShieldedFeeError — an unpaid consensus error. The sender is not charged (there is no identity to charge), and the transition produces no execution event.

Fee Constants in the Version System

The fee parameters are stored in the platform version under drive_abci.validation_and_processing.event_constants:

#![allow(unused)]
fn main() {
pub struct DriveAbciValidationConstants {
    pub maximum_vote_polls_to_process: u16,
    pub maximum_contenders_to_consider: u16,
    pub minimum_pool_notes_for_outgoing: u64,
    pub shielded_anchor_retention_blocks: u64,
    pub shielded_anchor_pruning_interval: u64,
    pub shielded_proof_verification_fee: u64,      // 40_000_000 at protocol 14
    pub shielded_per_action_processing_fee: u64,    // 22_000_000
    pub shielded_storage_bytes_per_action: u64,     // 550 at protocol 14
    pub shielded_implicit_fee_cap: u64,             // 20_000_000_000 (0.2 DASH)
}
}

The shielded_implicit_fee_cap bounds the surplus that a ShieldFromAssetLock may implicitly donate to the fee pools when no surplus_output is set (see Entry-Transition Fees).

The storage component is derived at runtime from fee_version.storage.storage_disk_usage_credit_per_byte and fee_version.storage.storage_processing_credit_per_byte, multiplied by the versioned shielded_storage_bytes_per_action allowance.

This design means:

  • Proof and processing fees can be tuned independently via version bumps
  • Storage fees automatically track changes to the platform-wide storage rates
  • Storage allowances can be calibrated independently of the per-byte rates

How Fees Flow After Validation

Once the fee check passes and the transition is fully validated and executed, the shielded pool's total balance is decremented and the fee is booked via the PaidFromShieldedPool execution event:

ShieldedTransfer:              pool_balance -= fee_amount          // fee == value_balance
Unshield:                      pool_balance -= unshielding_amount   // gross
ShieldedWithdrawal:            pool_balance -= unshielding_amount   // gross
IdentityTopUpFromShieldedPool: pool_balance -= top_up_amount        // gross

For Unshield and ShieldedWithdrawal, unshielding_amount is the gross amount leaving the pool. Of that, unshielding_amount − fee_amount is credited to the output platform address (Unshield) or written into the Core withdrawal document (ShieldedWithdrawal), and fee_amount is booked as the transition fee. For Unshield that fee is compute_shielded_unshield_fee — the base fee plus the flat AddBalanceToAddress output-write storage cost (+6,082,800 credits), since Unshield also writes the net to a transparent platform address. For ShieldedWithdrawal it is compute_shielded_withdrawal_fee — the same base fee plus the flat Core withdrawal-document storage cost (+112,340,000 credits), since ShieldedWithdrawal also writes a real document into the withdrawals contract. Validation guarantees unshielding_amount ≥ fee_amount (and, for ShieldedWithdrawal, that the net also clears MIN_WITHDRAWAL_AMOUNT), so the subtraction never underflows. Because each fee prices its extra write, the booking split (storage routed to the storage pool, the remainder paid to the proposer) covers that write instead of zeroing the proposer's processing reward to cover it.

For IdentityTopUpFromShieldedPool, top_up_amount is likewise the gross amount leaving the pool: top_up_amount − fee_amount is added to the existing identity's balance and fee_amount (compute_shielded_identity_top_up_fee, the base fee plus the flat identity balance write component) is booked as the transition fee; validation guarantees top_up_amount ≥ fee_amount. The identity balance and the pool total are both terms of the block conservation equation, so no system-credit adjustment is emitted.

For ShieldedTransfer, the pool decreases by exactly the fee (the sender's notes are spent and the recipient's notes are created, but the pool's aggregate balance only drops by the fee).

In all cases the booked fee_amount is split the same way as every other transition's fee: the storage cost of the permanent shielded writes is routed to the storage pool (amortized across epochs and subject to the per-epoch fee multiplier at payout), and the remainder — proof verification plus per-action processing — is the processing fee paid to the current block proposer.

The three entry transitions (Shield, ShieldFromAssetLock, ShieldFromIdentity) do not decrement the pool (they add to it), so their fees are booked from the funding side instead:

Shield:              fee_amount = metered + shielded_verification_fee     // from transparent address inputs
ShieldFromAssetLock: fee_amount = pool_fee (+ unclaimed surplus)     // from the consumed asset lock

For Shield, the fee is deducted from the transparent address inputs and booked through the standard PaidFromAddressInputs event (deducted == booked, no override): metered storage and processing, plus the shielded_verification_fee folded into processing. For ShieldFromAssetLock, the consumed asset-lock value is partitioned into shield_amount (into the pool), surplus_amount (to surplus_output, or 0), and fee_amount (to the fee pools); see Entry-Transition Fees.

Cryptographic Binding

The fee is not just a field that the platform trusts. It is cryptographically bound to the ZK proof through two mechanisms:

  1. Value commitments (cv_net): Each action contains a Pedersen commitment to the note value. The sum of all value commitments must equal value_balance (modulo the blinding factors). The binding signature proves this relationship holds.

  2. Platform sighash: The bundle commitment (which includes value_balance) is hashed into the sighash that the spend authorization signatures sign over:

    sighash = SHA-256("DashPlatformSighash" || bundle_commitment || extra_data)
    

    Mutating value_balance after signing changes the sighash, invalidating all signatures. The BatchValidator checks both the Halo 2 proof and all signatures, so any tampering is caught.

This means a client cannot claim a lower fee than what the ZK proof actually commits to — the proof and signatures would fail verification.

Rules and Guidelines

Do:

  • Always set value_balance to at least the minimum fee when building a shielded bundle on the client side. Use min_fee = proof_verification_fee + num_actions × (processing_fee + storage_fee) with the current platform version constants.
  • Include the fee in the note arithmetic: total_spent = total_output + fee. The Orchard builder handles this when you set the output amount to spend_amount − desired_fee.
  • Remember that the minimum action count is 2 (Orchard privacy requirement).

Do not:

  • Assume the fee is free for Shield transitions — the fee comes from transparent address inputs and is validated through the address balance system, not here.
  • Mutate value_balance after building the bundle. The binding signature and sighash will be invalidated.
  • Hardcode fee amounts. Always read from PlatformVersion — the constants are versioned and will change as the protocol evolves.

What a Document Costs

drive::document::cost computes what creating one document costs, from the contract alone. The JavaScript SDKs expose it as documentCreateCost(contract, documentTypeName, options, platformVersion), which is what the contract visualizer shows per document type.

Amounts are in credits: 1 Dash is 100,000,000,000 credits.

Storage

The storage fee is the bytes the insert adds times storage_disk_usage_credit_per_byte (27,000 credits from protocol version 14). It is usually almost all of what a document costs.

The estimate is exact. It lists every element the insert writes (see GroveDB Structure for the layout), built with the functions the insert walkers use:

  • the document by id, or with documentsKeepHistory a tree of revisions and a pointer to the newest;
  • for each index, the value trees along its properties (created only when no earlier document has the value), the [0] terminal and the document's reference in it, or an indexOnly type's entry;
  • the rows a ranked index keeps for the value in its secondary trees, one per ranking axis;
  • the trees preallocated for other document types' preallocated indexes whose entries will reference the document, charged to its creator.

Each element is priced with GroveDB's byte formulas: the key with the 32-byte subtree prefix, the serialized element (or a fixed size standing for a tree), the value and node hashes, the aggregate feature of the tree it sits in (8 or 16 bytes in a count or sum tree), and the link its parent keeps to it. A document create's elements carry 35 bytes of storage flags (the owner and the epoch); index trees carry them only when the documents are mutable, the contract can be deleted, or the type is indexOnly and its documents can be deleted.

The storage depends on what is already stored, so it comes in two scenarios:

  • every value new: the first document with these index values creates their trees;
  • every value known: a later document with the same values adds only its own entries. A unique index still adds its value, which no earlier document can hold, as does every tree keyed by the document's own id (a preallocated index's, say) and a ttl document's expiration entry; a ranked index's row for an existing value only moves, which GroveDB bills as replaced bytes.

The estimate also splits the storage by index: the layers an index shares with other indexes (a common prefix of properties, paid once for the document) and the layers only it uses, so an index's cost on its own is its shared layers plus its own. A skipIfAbsent index that skips the document writes nothing, and adds no layer, just as Drive builds none for it; the rows of a ranked level under a time window with a ttl are priced like the window, as processing.

The test should_price_what_drive_charges inserts documents into 21 contracts covering every index shape and requires the estimate to equal the storage fee Drive charged, byte for byte, with the trees already stored read from GroveDB before each insert.

Processing

  • Exact: verifying the signature (15,000 credits for an ECDSA key, 300,000 for BLS) and fetching the signing key and the identity's balance (18,000).
  • Estimated: the work of the writes (seeks, hashing and rewriting the path to each new element in its tree and above), for an assumed number of stored documents, each with its own values (1,000 by default); and the small reads and writes around the insert (the document id check and the identity's contract nonce). The test should_estimate_the_processing_of_the_writes_within_a_factor_of_two holds the write estimate to Drive's processing fee. Processing is a few percent of a document's cost.
  • A userFeeIncrease raises the processing fee only.

The per-document "minimum fee" of a batch (document_batch_sub_transition) is a balance check before processing, not a charge, and the unique index checks and the fetch of the batch's own contract are not billed.

What the contract adds

  • Action fees (actionFees): the create's fee, in credits, scaled by the epoch's fee multiplier unless priced as fixed, paid into the contract's fee pots.
  • Token cost (tokenCost): tokens transferred to the contract owner or burned.
  • Contest fund: a contested index's vote fund (0.1 Dash, doubling past 250 contenders), paid when the value is contested.

Refunds

Deleting a document refunds the storage fee of its flagged elements, less what the epochs already passed were paid: about 99.9% in the epoch it was created, about 95% a year later, then less each year for fifty years.

Documents with a ttl

A document whose type declares a ttl is stored without flags and pays for its bytes by the lifetime it has left, all of its ttl when it is created, at the schedule's tier for that lifetime (from 1 credit per byte for an hour to 26 for a week, then 34 per 788,400 seconds) instead of the 27,000 of storage kept for good. It also adds its entry to the documents expirations tree, and prepays its deletion as processing (a base cost, a cost per index level and one per document byte). Nothing of it is refunded.

Not covered

  • A create whose value starts a contest (a DPNS name matching the contest rule, say) is stored in the contest's vote poll until the contest ends, not in the index. The estimate prices an uncontested create and lists the contest fund; the vote poll's storage is not priced.
  • A platform version whose insert methods differ from protocol version 14's is refused: other versions write other elements for some shapes.

Consensus Errors

When a Dash Platform node processes a state transition -- a document creation, an identity update, a credit withdrawal -- things can go wrong. The data contract might reference an invalid schema. The signature might not match. The identity might not have enough balance. These are not internal bugs; they are protocol-level validation failures that every node on the network must agree on. If one node rejects a transition for reason X and another rejects it for reason Y, the chain forks. If one node returns error code 40100 and another returns 40101, clients break.

This is why consensus errors in Dash Platform are not ordinary Rust errors. They are serializable, code-stable, network-transmitted data structures that must produce identical bytes on every node and remain decodable by every client version. The ConsensusError enum is the heart of this system.

The top-level enum

Open packages/rs-dpp/src/errors/consensus/consensus_error.rs and you will find the root of the hierarchy:

#![allow(unused)]
fn main() {
#[derive(
    thiserror::Error,
    Debug,
    Encode,
    Decode,
    PlatformSerialize,
    PlatformDeserialize,
    Clone,
    PartialEq,
)]
#[platform_serialize(limit = 2000)]
#[error(transparent)]
#[allow(clippy::large_enum_variant)]
pub enum ConsensusError {
    /*

    DO NOT CHANGE ORDER OF VARIANTS WITHOUT INTRODUCING OF NEW VERSION

    */
    #[error("default error")]
    DefaultError,

    #[error(transparent)]
    BasicError(BasicError),

    #[error(transparent)]
    StateError(StateError),

    #[error(transparent)]
    SignatureError(SignatureError),

    #[error(transparent)]
    FeeError(FeeError),

    #[cfg(test)]
    #[cfg_attr(test, error(transparent))]
    TestConsensusError(TestConsensusError),
}
}

There are five things worth understanding here.

The four categories

Every consensus error falls into one of four categories:

  • BasicError -- structural and syntactic validation failures. The state transition itself is malformed, references a nonexistent document type, has an invalid identifier, exceeds size limits, or fails schema validation. These are caught before the node ever checks persistent state. The BasicError enum in packages/rs-dpp/src/errors/consensus/basic/basic_error.rs contains over 130 variants organized into sub-groups: versioning errors, structure errors, data contract errors, group errors, document errors, token errors, identity errors, state transition errors, and address errors.

  • StateError -- the transition is structurally valid but conflicts with the current platform state. A document already exists, an identity nonce is wrong, a token account is frozen, a group action was already completed. The StateError enum in packages/rs-dpp/src/errors/consensus/state/state_error.rs contains roughly 80 variants covering data contracts, documents, identities, voting, tokens, groups, and address balances.

  • SignatureError -- the cryptographic signature on the transition is invalid. The identity was not found, the key type is wrong, the key is disabled, the security level is insufficient, or the raw signature verification failed.

  • FeeError -- the transition is valid in every other way, but the identity cannot pay for it. Currently this category has a single variant: BalanceIsNotEnoughError.

This layered structure mirrors the validation pipeline. Platform checks basic structure first, then state, then signatures, then fees. Each layer produces errors from its own category.

The DO NOT CHANGE ORDER rule

The comment at the top of every consensus error enum is not a suggestion:

#![allow(unused)]
fn main() {
/*

DO NOT CHANGE ORDER OF VARIANTS WITHOUT INTRODUCING OF NEW VERSION

*/
}

Why? Because ConsensusError derives Encode and Decode from bincode. Bincode serializes enum variants by their ordinal position -- the first variant is 0, the second is 1, and so on. If you reorder variants, the same bytes now decode to a different error on nodes running different code versions. This is a consensus failure.

The same rule applies to every nested error enum. Here is FeeError in packages/rs-dpp/src/errors/consensus/fee/fee_error.rs:

#![allow(unused)]
fn main() {
#[derive(
    Error, Debug, PartialEq, Encode, Decode, PlatformSerialize, PlatformDeserialize, Clone,
)]
pub enum FeeError {
    /*

    DO NOT CHANGE ORDER OF VARIANTS WITHOUT INTRODUCING OF NEW VERSION

    */
    #[error(transparent)]
    BalanceIsNotEnoughError(BalanceIsNotEnoughError),
}
}

And the same rule applies to individual error structs. Here is DocumentAlreadyPresentError in packages/rs-dpp/src/errors/consensus/state/document/document_already_present_error.rs:

#![allow(unused)]
fn main() {
#[derive(
    Error, Debug, Clone, PartialEq, Eq, Encode, Decode, PlatformSerialize, PlatformDeserialize,
)]
#[error("Document {document_id} is already present")]
#[platform_serialize(unversioned)]
pub struct DocumentAlreadyPresentError {
    /*

    DO NOT CHANGE ORDER OF FIELDS WITHOUT INTRODUCING OF NEW VERSION

    */
    document_id: Identifier,
}
}

Notice the comment says fields too, not just variants. Bincode serializes struct fields in declaration order. Swap two fields and the bytes change.

The derive stack

Every consensus error type carries the same set of derives:

#![allow(unused)]
fn main() {
#[derive(
    Error,              // thiserror -- provides Display and Error trait
    Debug,              // standard Rust debug formatting
    Clone,              // errors need to be cloneable
    PartialEq,          // errors need to be comparable
    Encode,             // bincode encoding (field-level)
    Decode,             // bincode decoding (field-level)
    PlatformSerialize,  // platform wrapper around bincode (size limits, etc.)
    PlatformDeserialize,// platform wrapper around bincode
)]
}

The thiserror::Error derive gives each variant a Display implementation. For enums, #[error(transparent)] delegates to the inner type's Display. For leaf structs, you write the message directly:

#![allow(unused)]
fn main() {
#[error("Document {document_id} is already present")]
pub struct DocumentAlreadyPresentError {
    document_id: Identifier,
}
}

The Encode and Decode derives come from bincode and handle the actual byte-level serialization of each field. The PlatformSerialize and PlatformDeserialize derives are the platform's own wrappers that add size limits and error type conversion on top of bincode. We will cover those in the serialization chapters.

The #[platform_serialize] attribute

On the top-level ConsensusError, you will notice:

#![allow(unused)]
fn main() {
#[platform_serialize(limit = 2000)]
}

This sets a maximum serialized size of 2000 bytes. If a consensus error somehow serializes to more than 2000 bytes, the platform will return a MaxEncodedBytesReachedError instead. This is a defense against oversized error payloads being transmitted across the network.

On individual leaf error structs, you will see a different attribute:

#![allow(unused)]
fn main() {
#[platform_serialize(unversioned)]
}

The unversioned flag means the struct does not need a PlatformVersion parameter for serialization. Most leaf error structs are simple enough that they do not change between protocol versions -- their serialization format is stable. The top-level ConsensusError enum does not use unversioned because it may need version-aware behavior as the protocol evolves.

How errors nest

The nesting is three levels deep:

  1. ConsensusError -- the top level, with four category variants
  2. Category enums (BasicError, StateError, SignatureError, FeeError) -- each holds dozens of specific error variants
  3. Leaf error structs -- the actual error data (identifiers, messages, amounts)

Each level has From implementations to make conversion ergonomic:

#![allow(unused)]
fn main() {
// Leaf -> Category
impl From<DocumentAlreadyPresentError> for ConsensusError {
    fn from(err: DocumentAlreadyPresentError) -> Self {
        Self::StateError(StateError::DocumentAlreadyPresentError(err))
    }
}

// Category -> Top-level
impl From<StateError> for ConsensusError {
    fn from(error: StateError) -> Self {
        Self::StateError(error)
    }
}
}

This means you can use ? to propagate any leaf error up to a ConsensusError:

#![allow(unused)]
fn main() {
fn validate_document(doc: &Document) -> Result<(), ConsensusError> {
    if document_exists(doc.id()) {
        return Err(DocumentAlreadyPresentError::new(doc.id()).into());
    }
    Ok(())
}
}

The .into() call walks the From chain: DocumentAlreadyPresentError -> ConsensusError::StateError(StateError::DocumentAlreadyPresentError(...)).

Anatomy of a leaf error

Every leaf error follows the same pattern. Let us look at the full DocumentAlreadyPresentError:

#![allow(unused)]
fn main() {
use crate::consensus::state::state_error::StateError;
use crate::consensus::ConsensusError;
use crate::errors::ProtocolError;
use bincode::{Decode, Encode};
use platform_serialization_derive::{PlatformDeserialize, PlatformSerialize};
use platform_value::Identifier;
use thiserror::Error;

#[derive(
    Error, Debug, Clone, PartialEq, Eq, Encode, Decode, PlatformSerialize, PlatformDeserialize,
)]
#[error("Document {document_id} is already present")]
#[platform_serialize(unversioned)]
pub struct DocumentAlreadyPresentError {
    /*

    DO NOT CHANGE ORDER OF FIELDS WITHOUT INTRODUCING OF NEW VERSION

    */
    document_id: Identifier,
}

impl DocumentAlreadyPresentError {
    pub fn new(document_id: Identifier) -> Self {
        Self { document_id }
    }

    pub fn document_id(&self) -> &Identifier {
        &self.document_id
    }
}

impl From<DocumentAlreadyPresentError> for ConsensusError {
    fn from(err: DocumentAlreadyPresentError) -> Self {
        Self::StateError(StateError::DocumentAlreadyPresentError(err))
    }
}
}

The pattern is:

  1. Fields are private with getter methods
  2. A new() constructor
  3. A From impl that chains through the category enum to ConsensusError
  4. The full derive stack including PlatformSerialize and PlatformDeserialize
  5. The #[platform_serialize(unversioned)] attribute
  6. The ordering warning comment

The test-only variant

You may have noticed this in ConsensusError:

#![allow(unused)]
fn main() {
#[cfg(test)]
#[cfg_attr(test, error(transparent))]
TestConsensusError(TestConsensusError),
}

This variant only exists in test builds. It allows tests to create synthetic consensus errors without depending on real validation logic. The #[cfg(test)] attribute ensures it is completely stripped from production builds and does not affect the binary encoding of the other variants.

Rules

Do:

  • Always add new variants at the end of the enum
  • Always add new fields at the end of the struct
  • Always include the ordering warning comment in new error types
  • Always implement From<YourError> for ConsensusError through the appropriate category
  • Always derive the full stack: Error, Debug, Clone, PartialEq, Encode, Decode, PlatformSerialize, PlatformDeserialize
  • Use #[platform_serialize(unversioned)] on leaf error structs
  • Keep fields private with getter methods

Do not:

  • Reorder existing variants or fields -- this breaks network serialization
  • Remove existing variants -- old nodes might still produce them
  • Change the #[error(...)] message without considering client-side parsing
  • Add large fields to error structs -- the 2000-byte limit on ConsensusError applies to the entire serialized tree
  • Use #[platform_serialize(limit = ...)] on leaf structs -- the limit is enforced at the ConsensusError level

Error Codes

In the previous chapter we saw how ConsensusError organizes errors into a tree of enums and structs. But when a client -- whether a JavaScript SDK, a mobile app, or a third-party tool -- receives a validation failure from the platform, it does not receive a Rust enum. It receives bytes on a wire. To make those bytes useful, every consensus error maps to a stable numeric code through the ErrorWithCode trait.

These codes are the public API of the error system. They appear in gRPC responses, they are documented for client developers, and they must never change once assigned. A client that handles error code 40100 ("document already present") today must still be able to handle that same code a year from now, even after dozens of protocol upgrades.

The ErrorWithCode trait

The trait lives in packages/rs-dpp/src/errors/consensus/codes.rs and is as simple as it gets:

#![allow(unused)]
fn main() {
pub trait ErrorWithCode {
    /// Returns the error code
    fn code(&self) -> u32;
}
}

One method, one number, no room for ambiguity. The trait is implemented for ConsensusError and for each of the four category enums.

How ConsensusError delegates

The top-level implementation just forwards to the appropriate category:

#![allow(unused)]
fn main() {
impl ErrorWithCode for ConsensusError {
    fn code(&self) -> u32 {
        match self {
            Self::BasicError(e) => e.code(),
            Self::SignatureError(e) => e.code(),
            Self::StateError(e) => e.code(),
            Self::FeeError(e) => e.code(),

            #[cfg(test)]
            ConsensusError::TestConsensusError(_) => 1000,
            ConsensusError::DefaultError => 1, // this should never happen
        }
    }
}
}

The DefaultError variant returns code 1 -- a sentinel value that should never appear in practice. The TestConsensusError returns 1000, safely outside any production range.

The code ranges

Error codes are organized into ranges that correspond to error categories and subcategories. Here is the complete map as it stands in the codebase:

BasicError codes (10000-10949)

RangeCategoryExamples
10000-10099VersioningUnsupportedVersionError (10000), ProtocolVersionParsingError (10001), IncompatibleProtocolVersionError (10004)
10100-10199StructureJsonSchemaCompilationError (10100), InvalidIdentifierError (10102), ValueError (10103)
10200-10277Data ContractDataContractMaxDepthExceedError (10200), DuplicateIndexError (10201), InvalidDataContractIdError (10204), DataContractInvalidRequiredFieldsUpdateError (10276), PreProgrammedDistributionAmountOverLimitError (10277)
10350-10359GroupsGroupPositionDoesNotExistError (10350), GroupExceedsMaxMembersError (10354)
10360-10367Contract GroupsContractGroupMembershipsOverLimitError (10360), InvalidContractGroupAdminsError (10364), InvalidContractGroupDescriptionLengthError (10367); 10365 unassigned
10400-10424DocumentsDataContractNotPresentError (10400), DuplicateDocumentTransitionsWithIdsError (10401), DocumentPropertyNotDistinctError (10419), InvalidEncryptedPropertyShapeError (10420), DocumentPropertyMaxBytesExceededError (10421), DocumentPropertyConstraintViolatedError (10422), DocumentReferencePreimageInvalidError (10423), DocumentPropertyNotGeneratedError (10424)
10450-10460TokensInvalidTokenIdError (10450), TokenTransferToOurselfError (10456)
10500-10535IdentityDuplicatedIdentityPublicKeyBasicError (10500), InvalidIdentityPublicKeyDataError (10511)
10600-10603State TransitionInvalidStateTransitionTypeError (10600), StateTransitionMaxSizeExceededError (10602)
10700-10700GeneralOverflowError (10700)
10800-10818AddressTransitionOverMaxInputsError (10800), WithdrawalBelowMinAmountError (10818)
10819-10827ShieldedShieldedNoActionsError (10819), ShieldedTooManyActionsError (10825), ShieldedImplicitFeeCapExceededError (10826), ShieldedInvalidDenominationError (10827 — IdentityCreateFromShieldedPool exit amount not a member of the versioned denomination set)
10900-10949Contract ModerationInvalidContractModerationConfigError (10900), ContractModerationSelfTargetError (10901), DocumentActionFeesWithoutModerationError (10902), ContractModerationReasonTooLongError (10903), InvalidContractModerationReasonDocumentsError (10904), InvalidContractModerationDocumentFieldsError (10905)

SignatureError codes (20000-20012)

#![allow(unused)]
fn main() {
impl ErrorWithCode for SignatureError {
    fn code(&self) -> u32 {
        match self {
            Self::IdentityNotFoundError { .. } => 20000,
            Self::InvalidIdentityPublicKeyTypeError { .. } => 20001,
            Self::InvalidStateTransitionSignatureError { .. } => 20002,
            Self::MissingPublicKeyError { .. } => 20003,
            Self::InvalidSignaturePublicKeySecurityLevelError { .. } => 20004,
            Self::WrongPublicKeyPurposeError { .. } => 20005,
            Self::PublicKeyIsDisabledError { .. } => 20006,
            Self::PublicKeySecurityLevelNotMetError { .. } => 20007,
            Self::SignatureShouldNotBePresentError(_) => 20008,
            Self::BasicECDSAError(_) => 20009,
            Self::BasicBLSError(_) => 20010,
            Self::InvalidSignaturePublicKeyPurposeError(_) => 20011,
            Self::UncompressedPublicKeyNotAllowedError(_) => 20012,
        }
    }
}
}

Signature errors occupy the 20000 range. There are only 13 of them -- cryptographic validation is relatively binary (the signature is valid or it is not), so there are fewer failure modes to distinguish.

FeeError codes (30000)

#![allow(unused)]
fn main() {
impl ErrorWithCode for FeeError {
    fn code(&self) -> u32 {
        match self {
            Self::BalanceIsNotEnoughError { .. } => 30000,
        }
    }
}
}

The fee category currently has a single code. The 30000 range is reserved for future fee-related errors.

StateError codes (40000-41299)

RangeCategoryExamples
40000-40009Data ContractDataContractAlreadyPresentError (40000), DataContractIsReadonlyError (40001), DataContractNotFoundError (40008)
40100-40147DocumentsDocumentAlreadyPresentError (40100), DocumentNotFoundError (40101), DuplicateUniqueIndexError (40105), DocumentActionFeeAgreementNotSetError (40132), DocumentActionFeeAgreementMismatchError (40133), DocumentActionFeeMultiplierNotToleratedError (40134), DocumentActionFeeModeratorsShareMismatchError (40139), DocumentExpiredError (40140), DocumentContestMaximumContendersReachedError (40141), ReferencedDocumentRequirementNotMetError (40142), ReferencedDocumentTypeNotModeratedError (40143), ReferencedDocumentTypeModeratedError (40144), ReferencedDocumentRemovedError (40145), ReferencedDocumentTypeIndexOnlyError (40146), DocumentDeleteConstraintViolatedError (40147)
40200-40217IdentityIdentityAlreadyExistsError (40200), InvalidIdentityRevisionError (40203), IdentityInsufficientBalanceError (40210)
40300-40307VotingMasternodeNotFoundError (40300), MasternodeVoteAlreadyPresentError (40304), VoteChoiceNotAllowedForVotePollError (40307)
40400-40401Prefunded BalancesPrefundedSpecializedBalanceInsufficientError (40400)
40500-40502Data TriggersDataTriggerConditionError (40500), DataTriggerExecutionError (40501)
40600-40603AddressesAddressDoesNotExistError (40600), AddressNotEnoughFundsError (40601)
40700-40721TokensIdentityDoesNotHaveEnoughTokenBalanceError (40700), UnauthorizedTokenActionError (40701)
40800-40804GroupsIdentityNotMemberOfGroupError (40800), GroupActionAlreadyCompletedError (40802)
40900-40904ShieldedInvalidAnchorError (40900), NullifierAlreadySpentError (40901), InsufficientShieldedFeeError (40904)
41000-41003Contract GroupsContractGroupAlreadyExistsError (41000), ContractGroupNotFoundError (41001), IdentityNotContractGroupOwnerOrAdminError (41002), ContractGroupAdminNotFoundError (41003)
41100-41124Contract ModerationContractModerationNotEnabledError (41100), IdentityNotContractModeratorError (41101), ContractUserBannedError (41107), ContractUserSuspendedError (41108), ContractModerationTargetNotFoundError (41109), ContractModeratorIdentityNotFoundError (41110), ContractFeesAlreadyClaimedThisEpochError (41111), ContractFeesNothingToClaimError (41112), ContractFeeClaimNotAllowedError (41113), ContractModerationCounterpartyBarredError (41114), DocumentTypeNotDeletableByModeratorsError (41115), DocumentModerationWindowElapsedError (41116), ContractUserNotWarnedError (41117), ContractUserWarningLimitReachedError (41118), ContractDocumentRemovalNotFoundError (41119), DocumentRestoreWindowElapsedError (41120), DocumentRestoreHashMismatchError (41121), ContractDocumentAlreadyRestoredError (41122), DocumentFieldNotChangeableByModeratorsError (41123), DocumentModeratorFieldNotWritableError (41124)
41200-41299Contract Moderation TeamsContractModeratedDocumentTypeNotYetUsableError (41200), ContractModerationAbilityNotGrantedError (41201), ModerationCharterAddedModeratorLimitReachedError (41202), ModerationReasonNotListedError (41203), DocumentTypeNotDeletableOnceSettledError (41204), ContractModerationTeamNotSeatedError (41205), DocumentNotSettledError (41206), ContractTeamActionDoesNotExistError (41207), ContractTeamActionAlreadySignedError (41208), SettledDeletionNotRestorableError (41209), ContractTeamActionAlreadyCompletedError (41210), ContractTeamActionDocumentChangedError (41211), ContractTeamMemberAddedAfterDocumentError (41212)

Notice how the DataTriggerError sub-enum has its own ErrorWithCode implementation that the StateError delegates to:

#![allow(unused)]
fn main() {
impl ErrorWithCode for StateError {
    fn code(&self) -> u32 {
        match self {
            // ...
            // Data trigger errors: 40500-40699
            Self::DataTriggerError(ref e) => e.code(),
            // ...
        }
    }
}

impl ErrorWithCode for DataTriggerError {
    fn code(&self) -> u32 {
        match self {
            Self::DataTriggerConditionError { .. } => 40500,
            Self::DataTriggerExecutionError { .. } => 40501,
            Self::DataTriggerInvalidResultError { .. } => 40502,
        }
    }
}
}

This is the only case where StateError delegates to a sub-enum rather than mapping directly. All other state error variants map to their codes inline.

The pattern: one variant, one code, one match arm

Look at how codes are assigned inside BasicError:

#![allow(unused)]
fn main() {
impl ErrorWithCode for BasicError {
    fn code(&self) -> u32 {
        match self {
            // Versioning Errors: 10000-10099
            Self::UnsupportedVersionError(_) => 10000,
            Self::ProtocolVersionParsingError { .. } => 10001,
            Self::SerializedObjectParsingError { .. } => 10002,
            Self::UnsupportedProtocolVersionError(_) => 10003,
            Self::IncompatibleProtocolVersionError(_) => 10004,
            Self::VersionError(_) => 10005,
            Self::UnsupportedFeatureError(_) => 10006,

            // Structure Errors: 10100-10199
            Self::JsonSchemaCompilationError(..) => 10100,
            Self::JsonSchemaError(_) => 10101,
            // ...
        }
    }
}
}

Every single variant gets its own match arm and its own code. There is no default case, no wildcard, no range-based mapping. This is deliberate -- if you add a new variant to the enum and forget to add a code for it, the Rust compiler will refuse to compile because the match is non-exhaustive. The type system enforces completeness.

Nested ContractError patterns

One interesting detail is how BasicError handles DataContractError variants. The ContractError wrapper contains a nested enum, so the code mapping matches through two levels:

#![allow(unused)]
fn main() {
Self::ContractError(DataContractError::DocumentTypesAreMissingError { .. }) => 10214,
Self::ContractError(DataContractError::DecodingContractError { .. }) => 10222,
Self::ContractError(DataContractError::DecodingDocumentError { .. }) => 10223,
Self::ContractError(DataContractError::InvalidDocumentTypeError { .. }) => 10224,
Self::ContractError(DataContractError::MissingRequiredKey(_)) => 10225,
Self::ContractError(DataContractError::FieldRequirementUnmet(_)) => 10226,
// ... many more
}

Each specific DataContractError variant gets its own distinct code. This ensures that clients can distinguish between "the contract is missing a required key" (10225) and "the contract has a wrong type for a value" (10228) even though both come wrapped in BasicError::ContractError.

Adding a new error code

Here is the procedure for adding a new consensus error:

  1. Choose the category. Is it a structural validation issue (BasicError), a state conflict (StateError), a signature problem (SignatureError), or a fee issue (FeeError)?

  2. Choose the subcategory range. Look at the existing ranges within that category. If your error is about tokens in StateError, you would use the 40700-40799 range.

  3. Pick the next available code. Within the range, find the highest existing code and add 1. Never reuse a code, even if the original error was removed.

  4. Add the variant to the appropriate enum (at the end -- remember the ordering rule).

  5. Add the match arm to the ErrorWithCode implementation.

  6. Document the code so client developers know what it means.

Here is what the diff would look like for adding a hypothetical new token state error:

#![allow(unused)]
fn main() {
// In state_error.rs -- add at the end of the enum:
#[error(transparent)]
TokenNewError(TokenNewError),

// In codes.rs -- add to the StateError match:
Self::TokenNewError(_) => 40722,  // next available in the 40700 range
}

Why codes can never change

Consider what happens if code 40100 is changed from DocumentAlreadyPresentError to something else:

  • A client SDK has a handler for code 40100 that displays "This document already exists" to the user.
  • After the change, the platform sends 40100 for a completely different error.
  • The user sees "This document already exists" when actually their token balance is insufficient.
  • Trust in the platform erodes.

Error codes are part of the platform's wire protocol. They are as immutable as protobuf field numbers or HTTP status codes. Once assigned, they live forever.

Rules

Do:

  • Assign new codes at the end of the appropriate range
  • Use the compiler's exhaustive match checking to ensure every variant has a code
  • Keep codes within their designated range (do not put a document error in the identity range)
  • Leave gaps between ranges for future expansion (this is already done)

Do not:

  • Change an existing error code -- ever
  • Reuse a code from a removed error variant
  • Add a wildcard (_) match arm to ErrorWithCode implementations
  • Assign codes outside the established ranges without allocating a new range first
  • Skip codes within a range (assign sequentially within each subcategory)

Drive Errors

The previous two chapters covered consensus errors -- the carefully serialized, code-stable errors that get sent across the network. Drive errors are a different beast entirely. They are internal errors that arise from the storage layer, the database, and the logic that sits between state transitions and GroveDB. They are not serialized as consensus errors, and they do not need stable numeric codes. The one place they reach a client is a query's gRPC status, described in Query errors on the wire.

But they do need to be well-organized, because Drive is where most of the platform's complexity lives. When something goes wrong in Drive, you need to know immediately whether it is a corrupted database, a protocol-level validation failure, a fee calculation error, or a bug in your own code.

Query errors on the wire

A gRPC query answers a failed request with a status code, and clients read that code to decide whose fault the failure is. rs-dapi-client takes INVALID_ARGUMENT as the caller's mistake: the error goes back to the caller, and nothing else happens. It takes UNKNOWN and INTERNAL as a fault of the node: the client stops using that node for a while (a ban) and sends the request to the next one. A request error answered as INTERNAL therefore gets every node banned in turn, since every node refuses the request the same way.

The mapping lives in packages/rs-drive-abci/src/query/service.rs:

ErrorStatus
QueryError::InvalidArgument, QueryError::TooManyElementsINVALID_ARGUMENT
A QuerySyntaxError, whether the handler reports it (QueryError::Query, QueryError::Drive(Error::Query)) or returns it (Error::Drive(Error::Query))INVALID_ARGUMENT
QueryError::NotFoundNOT_FOUND
QueryError::ResourceExhaustedRESOURCE_EXHAUSTED
A request without a version, or with a version the node does not serveUNKNOWN: a newer node may serve it
Any other error a handler returnsINTERNAL

A status message is cut at 1024 bytes, because it travels in the grpc-message HTTP/2 header and may echo a request field of any length.

A handler checks the request before Drive sees it, and refuses what no node can answer with a validation error, never with ?. Two rules decide what it refuses:

  • Refuse only what fails. A request that has an answer keeps it, even an empty one, so no caller that depends on it breaks.
  • GroveDB cannot prove an empty query, or one with a limit of 0. When a proof is asked, an empty id list or a limit of 0 is refused; without a proof, the answer is empty.
RequestWithout a proofWith a proof
Empty id list: data contracts, identities balances, identities contract keys (ids or purposes), evonode blocks by ids, identity and identities token balances and infos, token statuses, addresses infosEmpty answerINVALID_ARGUMENT
Identity keys with limit 0, epochs info with count 0Empty answerINVALID_ARGUMENT
Limit or count 0 on contract history (or above 10), protocol upgrade vote status, evonode blocks by range, identity votes, pre-programmed distributions, group infos, group actionsINVALID_ARGUMENTINVALID_ARGUMENT
Identity keys search without a limitINVALID_ARGUMENTServed
Specific identity keys with a limit below their number of idsThe first keys that exist, in id orderThe same keys, proved
Documents v0 with limit 0, shielded encrypted notes with count 00 means the default0 means the default

An empty identities balances request with a proof used to get a proof the client could not verify. A proof that fails to verify is treated as a node fault too, so refusing that request matters as much as the status codes.

Nodes released before these rules answer the same requests with UNKNOWN, INTERNAL, or that unverifiable proof.

The Drive Error enum

The top-level error type lives in packages/rs-drive/src/error/mod.rs:

#![allow(unused)]
fn main() {
/// Errors
#[derive(Debug, thiserror::Error)]
pub enum Error {
    /// Query error
    #[error("query: {0}")]
    Query(#[from] QuerySyntaxError),
    /// Storage Flags error
    #[error("storage flags: {0}")]
    StorageFlags(#[from] StorageFlagsError),
    /// Drive error
    #[error("drive: {0}")]
    Drive(#[from] DriveError),
    /// Proof error
    #[error("proof: {0}")]
    Proof(#[from] ProofError),
    /// GroveDB error
    #[error("grovedb: {0}")]
    GroveDB(Box<grovedb::Error>),
    /// Protocol error
    #[error("protocol: {0}")]
    Protocol(Box<ProtocolError>),
    /// Identity error
    #[error("identity: {0}")]
    Identity(#[from] IdentityError),
    /// Fee error
    #[error("fee: {0}")]
    Fee(#[from] FeeError),
    /// Document error
    #[error("document: {0}")]
    Document(#[from] DocumentError),
    /// Value error
    #[error("value: {0}")]
    Value(#[from] ValueError),
    /// DataContract error
    #[error("contract: {0}")]
    DataContract(#[from] DataContractError),
    /// Cache error
    #[error("contract: {0}")]
    Cache(#[from] CacheError),
    /// Protocol error with info string
    #[error("protocol: {0} ({1})")]
    ProtocolWithInfoString(Box<ProtocolError>, String),
    /// IO error with info string
    #[error("io: {0} ({1})")]
    IOErrorWithInfoString(Box<io::Error>, String),
}
}

This enum is a classic Rust error aggregator. It collects errors from every subsystem that Drive interacts with -- the query parser, the storage flags system, GroveDB, the protocol layer, identities, fees, documents, data contracts, and the cache. Each variant wraps a specific error type from that subsystem.

Notice the organizational difference from ConsensusError:

  • ConsensusError is organized by validation phase (basic, state, signature, fee)
  • Drive's Error is organized by subsystem (query, storage, grovedb, protocol, identity, etc.)

This makes sense. Consensus errors are about "what rule was violated." Drive errors are about "what component failed."

Box<ProtocolError> -- avoiding large enum variants

Two variants wrap their inner error in a Box:

#![allow(unused)]
fn main() {
/// GroveDB error
#[error("grovedb: {0}")]
GroveDB(Box<grovedb::Error>),

/// Protocol error
#[error("protocol: {0}")]
Protocol(Box<ProtocolError>),
}

Why? Because ProtocolError and grovedb::Error are large types. Without the Box, the entire Error enum would be as large as its largest variant, which could be hundreds of bytes. Since most Drive operations return Result<T, Error>, you would be paying that size cost on every Ok path too -- the Result itself is as large as the larger of T and Error.

Boxing the large variants means the Error enum stores only an 8-byte pointer for those cases, keeping the overall size reasonable. This is a common Rust pattern, and Clippy will warn you about it via the clippy::large_enum_variant lint.

The From trait chain

The #[from] attribute on most variants is a thiserror feature that automatically generates From implementations. For example, #[from] DriveError generates:

#![allow(unused)]
fn main() {
impl From<DriveError> for Error {
    fn from(value: DriveError) -> Self {
        Self::Drive(value)
    }
}
}

But look at the manually written From implementations at the bottom of the file:

#![allow(unused)]
fn main() {
impl From<ProtocolError> for Error {
    fn from(value: ProtocolError) -> Self {
        Self::Protocol(Box::new(value))
    }
}

impl From<grovedb::Error> for Error {
    fn from(value: grovedb::Error) -> Self {
        Self::GroveDB(Box::new(value))
    }
}

impl From<grovedb::element::error::ElementError> for Error {
    fn from(value: grovedb::element::error::ElementError) -> Self {
        Self::GroveDB(Box::new(grovedb::Error::ElementError(value)))
    }
}

impl From<ProtocolDataContractError> for Error {
    fn from(value: ProtocolDataContractError) -> Self {
        Self::Protocol(Box::new(ProtocolError::DataContractError(value)))
    }
}
}

These cannot use #[from] because they involve boxing or wrapping through an intermediate type. The ProtocolError conversion boxes the value. The grovedb::element::error::ElementError conversion wraps the error inside grovedb::Error::ElementError first, then boxes it. The ProtocolDataContractError conversion wraps through ProtocolError::DataContractError and then boxes.

These manual From implementations create multi-hop error conversion chains. When a function deep in Drive's GroveDB interaction layer returns a grovedb::element::error::ElementError, the ? operator can propagate it all the way up to a drive::Error in a single step, with the conversion chain handling the wrapping automatically.

The DriveError enum -- internal errors

The DriveError enum in packages/rs-drive/src/error/drive.rs is where Drive reports its own internal problems:

#![allow(unused)]
fn main() {
/// Drive errors
#[derive(Debug, thiserror::Error)]
pub enum DriveError {
    /// This error should never occur, it is the equivalent of a panic.
    #[error("corrupted code execution error: {0}")]
    CorruptedCodeExecution(&'static str),

    /// Platform expected some specific versions
    #[error("drive unknown version on {method}, received: {received}")]
    UnknownVersionMismatch {
        method: String,
        known_versions: Vec<FeatureVersion>,
        received: FeatureVersion,
    },

    /// A critical corrupted state should stall the chain.
    #[error("critical corrupted state error: {0}")]
    CriticalCorruptedState(&'static str),

    /// Error
    #[error("not supported error: {0}")]
    NotSupported(&'static str),

    // ... many more variants
}
}

Notice how DriveError uses &'static str for most of its error messages rather than String. This is deliberate -- these are internal error messages that should be known at compile time. A CorruptedCodeExecution("document tree missing expected root") is a fixed message that describes a specific bug or data corruption scenario. Using &'static str makes it clear that these are not user-facing messages and avoids allocations on error paths.

There are a few variants that do use String -- these are cases where the error message needs to include runtime data:

#![allow(unused)]
fn main() {
#[error("corrupted contract indexes error: {0}")]
CorruptedContractIndexes(String),

#[error("corrupted drive state error: {0}")]
CorruptedDriveState(String),
}

The severity spectrum

DriveError variants implicitly encode severity through naming conventions:

  • Corrupted* variants indicate data corruption. The database is in an unexpected state. These are serious problems that may indicate bugs or hardware failures:

    #![allow(unused)]
    fn main() {
    CorruptedCodeExecution(&'static str),
    CriticalCorruptedState(&'static str),
    CorruptedContractPath(&'static str),
    CorruptedDocumentPath(&'static str),
    CorruptedBalancePath(&'static str),
    CorruptedSerialization(String),
    CorruptedElementType(&'static str),
    CorruptedDriveState(String),
    }
  • Invalid* and NotSupported variants indicate logic errors -- the code is trying to do something that should not be possible:

    #![allow(unused)]
    fn main() {
    InvalidDeletionOfDocumentThatKeepsHistory(&'static str),
    InvalidContractHistoryFetchLimit(u16),
    NotSupported(&'static str),
    }
  • *NotFound and *DoesNotExist variants indicate missing data that was expected:

    #![allow(unused)]
    fn main() {
    DataContractNotFound(String),
    ElementNotFound(&'static str),
    PrefundedSpecializedBalanceDoesNotExist(String),
    }
  • Version mismatch variants indicate protocol version problems:

    #![allow(unused)]
    fn main() {
    UnknownVersionMismatch { method, known_versions, received },
    VersionNotActive { method, known_versions },
    }

When to use consensus errors vs drive errors

This is the key design decision you face when adding error handling to Drive code:

Use a consensus error when:

  • The error is caused by invalid user input (a malformed state transition)
  • The error needs to be communicated back to the client
  • The error needs a stable numeric code
  • Other nodes must produce the same error for the same input

Use a drive error when:

  • The error is caused by internal state (database corruption, missing paths)
  • The error indicates a bug in the platform code
  • The error is about version mismatches or unsupported features
  • The error involves the storage layer (GroveDB problems)

In practice, Drive functions often work with both. A typical pattern is a function that validates input (producing consensus errors) and then performs storage operations (which might produce drive errors). The return type is usually Result<T, Error> where Error is the Drive error, and consensus errors are embedded inside ProtocolError which is itself a variant of the Drive error.

The ? propagation pattern

Here is how error propagation typically works in Drive code:

#![allow(unused)]
fn main() {
fn apply_document_create(
    &self,
    document: &Document,
    contract: &DataContract,
    // ...
) -> Result<(), Error> {
    // This might return a grovedb::Error, which auto-converts via From
    let existing = self.grove_get(path, key, transaction)?;

    // This might return a DriveError
    if existing.is_some() {
        return Err(DriveError::CorruptedDocumentAlreadyExists(
            "document should not exist at this point"
        ).into());
    }

    // This might return a ProtocolError, which auto-converts via From + Box
    let serialized = document.serialize(platform_version)?;

    // This might return a grovedb::Error
    self.grove_insert(path, key, element, transaction)?;

    Ok(())
}
}

The ? operator handles all the conversions transparently. A grovedb::Error becomes Error::GroveDB(Box::new(...)). A DriveError becomes Error::Drive(...). A ProtocolError becomes Error::Protocol(Box::new(...)). The developer does not need to think about which From implementation is being invoked -- the type system figures it out.

Sub-module error types

Drive organizes its errors into submodules, each with its own error enum:

#![allow(unused)]
fn main() {
pub mod cache;      // CacheError
pub mod contract;   // DataContractError (Drive's own, distinct from DPP's)
pub mod document;   // DocumentError
pub mod drive;      // DriveError
pub mod fee;        // FeeError (Drive's own, distinct from consensus FeeError)
pub mod identity;   // IdentityError
pub mod proof;      // ProofError
pub mod query;      // QuerySyntaxError
}

Each of these has a #[from] conversion to the top-level Error, creating a clean hierarchy where subsystem-specific code can work with its own error type and callers can propagate through ? to the unified type.

Rules

Do:

  • Use Box for large error types (ProtocolError, grovedb::Error) to keep the enum small
  • Use &'static str for internal error messages that are known at compile time
  • Use String only when the message needs runtime data
  • Let #[from] generate From implementations where possible
  • Write manual From implementations when boxing or intermediate wrapping is needed
  • Follow the naming conventions: Corrupted* for data corruption, Invalid* for logic errors, *NotFound for missing data

Do not:

  • Use consensus errors for internal Drive problems
  • Use drive errors for user-facing validation failures
  • Forget to add a From implementation when introducing a new error type
  • Return a String error message when a typed error variant would be more informative
  • Panic in Drive code -- return a CorruptedCodeExecution or CriticalCorruptedState error instead

Platform Serialization

Dash Platform needs to serialize data structures -- state transitions, consensus errors, documents, identities -- into bytes and back. You might ask: why not just use bincode directly? Or serde? Or protobuf? The answer is that Platform has requirements that no off-the-shelf serialization library handles on its own:

  1. Version awareness. The serialization of a type may differ between protocol versions. A DataContract serialized under protocol version 3 might have different fields than under version 4. The serialization system needs to accept a PlatformVersion parameter and dispatch accordingly.

  2. Size limits. A malicious actor should not be able to send a 100 MB state transition that costs the node minutes to decode. Serialization must enforce configurable byte limits.

  3. Determinism. Every node must produce identical bytes for identical data. This rules out serialization formats that allow field reordering (like JSON) or that depend on hash map iteration order.

  4. Compatibility with bincode. Platform already uses bincode extensively for its compact binary format and deterministic encoding. The custom layer should wrap bincode, not replace it.

The rs-platform-serialization crate provides this layer. It sits between bincode and the rest of the platform, adding version-aware encoding/decoding while delegating the actual byte-level work to bincode.

The core traits

The crate defines two pairs of traits in packages/rs-platform-serialization/src/enc/mod.rs and packages/rs-platform-serialization/src/de/mod.rs.

Encoding: PlatformVersionEncode

#![allow(unused)]
fn main() {
pub trait PlatformVersionEncode {
    /// Encode a given type.
    fn platform_encode<E: Encoder>(
        &self,
        encoder: &mut E,
        platform_version: &PlatformVersion,
    ) -> Result<(), EncodeError>;
}
}

Compare this to bincode's standard Encode trait:

#![allow(unused)]
fn main() {
// bincode's Encode (for reference)
pub trait Encode {
    fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError>;
}
}

The only difference is the platform_version parameter. For types whose serialization does not change between versions, the implementation simply ignores it and delegates to bincode's Encode:

#![allow(unused)]
fn main() {
impl PlatformVersionEncode for String {
    fn platform_encode<E: Encoder>(
        &self,
        encoder: &mut E,
        _: &PlatformVersion,  // ignored -- String encoding never changes
    ) -> Result<(), EncodeError> {
        Encode::encode(self, encoder)
    }
}
}

For types that do change between versions, the implementation can dispatch to different encoding logic based on the version.

Decoding: PlatformVersionedDecode

#![allow(unused)]
fn main() {
pub trait PlatformVersionedDecode: Sized {
    fn platform_versioned_decode<D: Decoder<Context = crate::BincodeContext>>(
        decoder: &mut D,
        platform_version: &PlatformVersion,
    ) -> Result<Self, DecodeError>;
}
}

Again, this mirrors bincode's Decode but with the version parameter. There is also a borrowed variant for zero-copy decoding:

#![allow(unused)]
fn main() {
pub trait PlatformVersionedBorrowDecode<'de>: Sized {
    fn platform_versioned_borrow_decode<
        D: BorrowDecoder<'de, Context = crate::BincodeContext>,
    >(
        decoder: &mut D,
        platform_version: &PlatformVersion,
    ) -> Result<Self, DecodeError>;
}
}

And a convenience macro to implement PlatformVersionedBorrowDecode for any type that implements PlatformVersionedDecode:

#![allow(unused)]
fn main() {
#[macro_export]
macro_rules! impl_platform_versioned_borrow_decode {
    ($ty:ty) => {
        impl<'de> $crate::PlatformVersionedBorrowDecode<'de> for $ty {
            fn platform_versioned_borrow_decode<
                D: bincode::de::BorrowDecoder<'de, Context = $crate::BincodeContext>,
            >(
                decoder: &mut D,
                platform_version: &PlatformVersion,
            ) -> core::result::Result<Self, bincode::error::DecodeError> {
                <$ty as $crate::PlatformVersionedDecode>::platform_versioned_decode(
                    decoder,
                    platform_version,
                )
            }
        }
    };
}
}

The convenience functions

The crate provides top-level functions that mirror bincode's API but add version support. The most commonly used one is platform_encode_to_vec in packages/rs-platform-serialization/src/features/impl_alloc.rs:

#![allow(unused)]
fn main() {
pub fn platform_encode_to_vec<E: PlatformVersionEncode, C: Config>(
    val: E,
    config: C,
    platform_version: &PlatformVersion,
) -> Result<Vec<u8>, EncodeError> {
    let size = {
        let mut size_writer = enc::EncoderImpl::<_, C>::new(SizeWriter::default(), config);
        val.platform_encode(&mut size_writer, platform_version)?;
        size_writer.into_writer().bytes_written
    };
    let writer = VecWriter::with_capacity(size);
    let mut encoder = enc::EncoderImpl::<_, C>::new(writer, config);
    val.platform_encode(&mut encoder, platform_version)?;
    Ok(encoder.into_writer().inner)
}
}

This function does something clever: it encodes twice. The first pass uses a SizeWriter that counts bytes without allocating. The second pass uses a VecWriter pre-allocated to exactly the right size. This avoids reallocations during encoding.

For decoding:

#![allow(unused)]
fn main() {
pub fn platform_versioned_decode_from_slice<D: PlatformVersionedDecode, C: Config>(
    src: &[u8],
    config: C,
    platform_version: &PlatformVersion,
) -> Result<D, error::DecodeError> {
    let reader = read::SliceReader::new(src);
    let mut decoder = DecoderImpl::<_, C, crate::BincodeContext>::new(reader, config, ());
    D::platform_versioned_decode(&mut decoder, platform_version)
}
}

There are also platform_encode_into_slice for encoding into a pre-allocated buffer, encode_into_writer for encoding into arbitrary writers, and platform_versioned_decode_from_reader for decoding from arbitrary readers.

How it wraps bincode

The wrapping is lightweight. Platform serialization does not add a version prefix to the bytes by default. It does not change the wire format. When you call platform_encode_to_vec, the resulting bytes are identical to what bincode::encode_to_vec would produce -- the difference is that the encoding logic can vary based on the PlatformVersion parameter.

The bincode configuration used throughout the platform is:

#![allow(unused)]
fn main() {
let config = bincode::config::standard()
    .with_big_endian()  // network byte order for determinism
    .with_no_limit();   // or .with_limit::<{ N }>() for size-limited encoding
}

Big endian is used for deterministic cross-platform encoding. The limit is applied at the PlatformSerialize / PlatformDeserialize level (see the derive macros chapter) rather than at the raw bincode level.

Standard type implementations

The crate provides PlatformVersionEncode and PlatformVersionedDecode implementations for all standard Rust types. These live in packages/rs-platform-serialization/src/features/impl_alloc.rs and the other impl files. Here are the key patterns:

Types that ignore the version

Primitive types, strings, and simple wrappers just delegate to bincode:

#![allow(unused)]
fn main() {
impl PlatformVersionedDecode for String {
    fn platform_versioned_decode<D: Decoder<Context = crate::BincodeContext>>(
        decoder: &mut D,
        _: &PlatformVersion,
    ) -> Result<Self, DecodeError> {
        bincode::Decode::decode(decoder)
    }
}
}

Collections that propagate the version

Collections like Vec, BTreeMap, and BTreeSet encode their length, then encode each element with the platform version:

#![allow(unused)]
fn main() {
impl<T> PlatformVersionEncode for Vec<T>
where
    T: PlatformVersionEncode + 'static,
{
    fn platform_encode<E: Encoder>(
        &self,
        encoder: &mut E,
        platform_version: &PlatformVersion,
    ) -> Result<(), EncodeError> {
        crate::enc::encode_slice_len(encoder, self.len())?;
        // Optimization: byte slices are written directly
        if core::any::TypeId::of::<T>() == core::any::TypeId::of::<u8>() {
            let slice: &[u8] = unsafe { core::mem::transmute(self.as_slice()) };
            encoder.writer().write(slice)?;
            return Ok(());
        }
        for item in self.iter() {
            item.platform_encode(encoder, platform_version)?;
        }
        Ok(())
    }
}
}

Notice the Vec<u8> optimization -- byte vectors are written in bulk rather than element-by-element, which is significantly faster for large binary data.

Smart pointers

Box<T>, Rc<T>, Arc<T>, and Cow<T> all delegate to their inner type:

#![allow(unused)]
fn main() {
impl<T> PlatformVersionEncode for Box<T>
where
    T: PlatformVersionEncode + ?Sized,
{
    fn platform_encode<E: Encoder>(
        &self,
        encoder: &mut E,
        platform_version: &PlatformVersion,
    ) -> Result<(), EncodeError> {
        T::platform_encode(self, encoder, platform_version)
    }
}
}

Size limits and why they matter

Without size limits, a malicious state transition could contain a Vec claiming to have 2^64 elements, causing the node to allocate unbounded memory during decoding. Or a deeply nested document structure could produce a multi-megabyte serialized form that consumes excessive bandwidth.

Size limits are enforced at two levels:

  1. At the PlatformSerialize trait level -- the #[platform_serialize(limit = N)] attribute on a type causes the derive macro to use bincode::config::standard().with_big_endian().with_limit::<{ N }>(). If encoding exceeds N bytes, it returns a MaxEncodedBytesReachedError.

  2. At the bincode decoder level -- bincode's claim_container_read mechanism prevents allocating excessive memory for containers:

#![allow(unused)]
fn main() {
fn platform_versioned_decode<D: Decoder<Context = crate::BincodeContext>>(
    decoder: &mut D,
    platform_versioned: &PlatformVersion,
) -> Result<Self, DecodeError> {
    let len = crate::de::decode_slice_len(decoder)?;
    decoder.claim_container_read::<T>(len)?;  // checks memory budget

    let mut vec = Vec::with_capacity(len);
    for _ in 0..len {
        decoder.unclaim_bytes_read(core::mem::size_of::<T>());
        vec.push(T::platform_versioned_decode(decoder, platform_version)?);
    }
    Ok(vec)
}
}

The claim_container_read call tells the decoder "I am about to read len elements of type T" and the decoder checks whether this fits within the remaining byte budget. If not, it returns an error before any allocation happens.

The BincodeContext type alias

You will see crate::BincodeContext throughout the code:

#![allow(unused)]
fn main() {
pub type BincodeContext = ();
}

This is the bincode context type. Bincode supports contextual decoding where a context value is threaded through the decoder -- useful for things like string interning or reference resolution. Platform does not use this feature, so the context is the unit type (). The alias exists to make it easy to change later if needed.

Rules

Do:

  • Use PlatformVersionEncode / PlatformVersionedDecode for types whose serialization may change between protocol versions
  • Use standard bincode Encode / Decode for types whose format is permanently fixed
  • Use big-endian configuration everywhere for deterministic encoding
  • Apply size limits on types that are received from untrusted sources (state transitions, consensus errors)
  • Use the platform_encode_to_vec convenience function rather than constructing encoders manually

Do not:

  • Use little-endian or native-endian configuration -- this breaks determinism across architectures
  • Forget to propagate the platform_version parameter through collection types
  • Skip claim_container_read in custom collection decoders -- this is a denial-of-service defense
  • Add a version prefix to the bytes manually -- the platform version is a parameter, not part of the wire format
  • Use serde for consensus-critical serialization -- serde's flexibility (field names, human-readable formats) is a liability when you need deterministic bytes

Document Serialization

This chapter describes the binary wire format used to serialize and deserialize documents on Dash Platform. If you are building a tool that reads documents from gRPC responses, GroveDB storage, or any other source of raw document bytes, this is the specification you need.

Documents are not self-describing. The binary format does not include field names or type tags. To interpret the bytes, you must know which data contract and document type the document belongs to. The document type's schema defines the field order, types, and sizes that determine how the bytes map to values.

High-level structure

Every serialized document follows this layout:

┌──────────────────────┐
│  Serialization       │  varint (1-2 bytes)
│  Version             │  Currently: 0, 1, 2, or 3
├──────────────────────┤
│  Contract version    │  V3 only: varint
│  stamp               │  (0 = unstamped)
├──────────────────────┤
│  $id                 │  32 bytes
├──────────────────────┤
│  $ownerId            │  32 bytes
├──────────────────────┤
│  $creatorId          │  V2 only, if document type supports
│  (optional)          │  transfers/trading: 1 byte flag + 32 bytes
├──────────────────────┤
│  $revision           │  varint (if document type is mutable)
├──────────────────────┤
│  Time fields         │  2-byte bitfield + variable-length data
│  bitfield + data     │
├──────────────────────┤
│  Price               │  Only if document type supports trading
│  (optional)          │  1 byte flag + 8 bytes
├──────────────────────┤
│  User properties     │  Variable length, schema-dependent
│  (in schema order)   │
└──────────────────────┘

Source: packages/rs-dpp/src/document/v0/serialize.rs

Serialization versions

The first bytes of a serialized document are a varint encoding the serialization version. This is not the protocol version or the document struct version — it is the version of the serialization format itself.

Serialization VersionDescription
0Original format. All integers encoded as i64 (8 bytes big-endian) regardless of their schema type.
1Integers encoded at their native size (u8 = 1 byte, u16 = 2 bytes, u32 = 4 bytes, etc.). Otherwise identical to v0.
2Same as v1, but adds $creatorId field after $ownerId for document types that support transfers or trading.
3Same as v2, but adds a contract version stamp varint immediately after the version varint (protocol v14+). The stamp selects each property's layout when the document type carries requiredSince annotations — see below.

The varint encoding uses the integer-encoding crate's VarInt format. For values 0 through 3, the varint is a single byte: 0x00, 0x01, 0x02, or 0x03.

#![allow(unused)]
fn main() {
// Serialization version is written first
let mut buffer: Vec<u8> = 2u64.encode_var_vec(); // version 2 → byte 0x02
}

When deserializing, the version varint is read first, then the appropriate deserialization logic is dispatched:

#![allow(unused)]
fn main() {
let serialized_version: u64 = serialized_document.read_varint()?;
match serialized_version {
    0 => DocumentV0::from_bytes_v0(serialized_document, document_type, platform_version),
    1 => DocumentV0::from_bytes_v1(serialized_document, document_type, platform_version),
    2 => DocumentV0::from_bytes_v2(serialized_document, document_type, platform_version),
    3 => DocumentV0::from_bytes_v3(serialized_document, document_type, platform_version),
    _ => Err(/* unknown version */),
}
}

Note: version 0 has a fallback — if deserialization as v0 (all i64) fails, it retries as v1 (native integer types). This handles edge cases from protocol versions 1–8 where the version byte was 0 but non-i64 integer types may have been used.

The contract version stamp (v3)

Serialization version 3 (the default from protocol v14) writes one extra varint immediately after the version varint: the contract version stamp — the version of the data contract the document's bytes conform to. A value of 0 means unstamped: the document was originally serialized before format 3 existed and has merely been rewritten in the new envelope (for example by a transfer).

The stamp exists because contract updates may add new required properties from a specific contract version onward, using the requiredSince schema keyword:

"properties": {
  "newField": { "type": "string", "maxLength": 63, "position": 4, "requiredSince": 3 }
},
"required": ["existingField", "newField"]

Requiredness is baked into the wire format — a required property serializes raw while an optional one carries a presence flag — so a property whose requiredness varies by contract version needs the stamp to resolve its layout. The rule, per property:

A property is encoded as required (no presence flag) if it is listed in required and either it has no requiredSince annotation, or the document's stamp is at or above the annotation. Otherwise it is encoded as optional (presence-flagged).

An unstamped document (0) predates every requiredSince annotation, so only unconditionally required properties count as required for it. This means the latest contract alone reconstructs the byte layout of every document ever stored — no historical contract lookups are needed.

The stamp is platform-assigned: Drive sets it to the current contract version whenever document content is supplied (create and replace), and preserves it untouched through server-side rewrites that do not re-supply content (transfer and purchase). A document created before a contract update therefore keeps its old stamp — and may legitimately omit properties the newest schema requires — until a replace re-supplies its content and re-stamps it. Clients can also use the stamp as a staleness signal: a document stamped above the client's cached contract version means the contract needs refetching.

Formats 0–2 have no stamp; documents read from them deserialize with contract_version = None, equivalent to a 0 stamp.

Field-by-field breakdown

$id (32 bytes)

The document's unique identifier, written as raw bytes. This is a 256-bit value derived from the contract ID, owner ID, document type name, entropy and (protocol v14+) the identity contract nonce of the create transition via double SHA-256.

$ownerId (32 bytes)

The identity that currently owns the document, written as raw bytes.

$creatorId (v2 and v3, conditional)

Present only in serialization versions 2 and 3, and only if the document type supports transfers (documents_transferable) or trading (trade_mode != None).

0x01  [32 bytes creatorId]    — creator ID present
0x00                           — creator ID absent

$revision (varint, conditional)

Present only if the document type requires revisions (mutable documents). Encoded as a varint (u64). New documents start at revision 1 (INITIAL_REVISION).

Time fields (2-byte bitfield + data)

Time-related fields use a compact encoding with a bitfield to indicate which fields are present, followed by the data for each present field.

Bitfield (2 bytes, big-endian u16):

BitField
0 (0x0001)$createdAt
1 (0x0002)$updatedAt
2 (0x0004)$transferredAt
3 (0x0008)$createdAtBlockHeight
4 (0x0010)$updatedAtBlockHeight
5 (0x0020)$transferredAtBlockHeight
6 (0x0040)$createdAtCoreBlockHeight
7 (0x0080)$updatedAtCoreBlockHeight
8 (0x0100)$transferredAtCoreBlockHeight
9 (0x0200)$moderatedAt (version 3)
10 (0x0400)$moderatedBy (version 3)

Data: For each bit that is set (in the order above), the corresponding value is appended:

  • $createdAt, $updatedAt, $transferredAt: 8 bytes big-endian u64 — milliseconds since Unix epoch
  • $createdAtBlockHeight, $updatedAtBlockHeight, $transferredAtBlockHeight: 8 bytes big-endian u64 — platform block height
  • $createdAtCoreBlockHeight, $updatedAtCoreBlockHeight, $transferredAtCoreBlockHeight: 4 bytes big-endian u32 — core chain block height
  • $moderatedAt: 8 bytes big-endian u64, milliseconds since Unix epoch
  • $moderatedBy: 32 bytes, the moderator's identity id

Bits 9 and 10 are read only in version 3, the format of protocol version 14. They are set only on a document a moderator has written the fields only moderators write of (see System Properties), so every other document's bitfield is as it was before the bits had a meaning. An earlier format has no place for them: serializing a stamped document in format 0, 1 or 2 is refused rather than dropping the stamp.

For example, if a document has $createdAt and $updatedAt set, the bitfield would be 0x0003, followed by 16 bytes (8 for each timestamp).

Price (conditional)

If the document type's trade_mode allows seller-set pricing:

0x01  [8 bytes big-endian u64]   — price in credits
0x00                              — no price set

User-defined properties

Properties are serialized in schema position order — each property in the data contract schema has a position field, and document_type.properties() returns an IndexMap sorted by that position. This is not alphabetical order.

Each property is encoded based on its type and whether it is required. In serialization version 3, "required" means required at the document's contract version stamp (see above); in versions 0–2 — and for every property without a requiredSince annotation — it is simply whether the property is listed in required.

Required fields: The value is written directly with no prefix byte.

Optional fields: A 1-byte presence flag is written first:

  • 0x01 followed by the encoded value — field is present
  • 0x00 — field is absent

Transient fields: Always get a presence byte, even if marked as required. The serializer checks if !property.required || property.transient to decide whether to write the flag.

Value encoding by type

All numeric values use big-endian byte order.

TypeEncoding
u8 / i81 byte
u16 / i162 bytes big-endian
u32 / i324 bytes big-endian
u64 / i648 bytes big-endian
u128 / i12816 bytes big-endian
f648 bytes big-endian IEEE 754
boolean1 byte: 0x01 = true, 0x00 = false
stringvarint length prefix + UTF-8 bytes
byteArray (fixed size)raw bytes (no length prefix if min_size == max_size)
byteArray (variable size)varint length prefix + raw bytes
identifier32 bytes raw
date8 bytes big-endian f64 (when optional: 0xff prefix + 8 bytes)
array (typed array, protocol v14)varint element count + each element encoded exactly as a required property of the element's type (rows above): an identifier element is 32 raw bytes, an integer element takes the width its bounds give it, a fixed-size byte array element is raw, a string or variable-size byte array element has a varint length prefix. Elements never carry a presence byte
objectNested fields serialized recursively, in the order the schema lists them (not sorted by position)

Note on date types: User-property date fields are encoded as f64 (8 bytes). System timestamps ($createdAt, $updatedAt, $transferredAt) are u64 milliseconds. Both are 8 bytes big-endian but use different numeric representations.

Important: In serialization version 0, all integer types are encoded as i64 (8 bytes big-endian), regardless of the actual type in the schema. This means a u8 field that should be 1 byte is encoded as 8 bytes in v0.

Worked example: withdrawal document

The withdrawals contract defines a withdrawal document type with these properties (in schema position order):

PositionPropertyTypeRequired
0transactionIndexinteger (i64)no
1transactionSignHeightinteger (i64)no
2amountinteger (i64)yes
3coreFeePerByteinteger (i64)yes
4poolinginteger (i64)yes
5outputScriptbyteArray (23–25 bytes, variable)yes
6statusinteger (i64)yes

The document type requires $createdAt, $updatedAt, and $revision.

Here is a real serialized withdrawal document (hex), broken down byte by byte:

02                                      ← serialization version (varint: 2)
0222 9eda 94b3 5be5 5ac2 22ca 8cc4      ← $id (32 bytes)
631c 0717 c9ee 4a22 3f2a 269e 06c9
a1be 7c54
36b3 e63b a54a ba9b 7599 4128 d124      ← $ownerId (32 bytes)
e9e1 cebe 348c d304 15b5 098c 6052
6de0 157e
                                        ← no $creatorId: withdrawal document type
                                          does not support transfers/trading
c501                                    ← $revision (varint: 197)
0003                                    ← time bitfield: bits 0,1 set
                                          ($createdAt + $updatedAt)
0000019cd70f3323                        ← $createdAt: 1773134623523 ms
                                          (2026-03-10 09:23:43 UTC)
0000019d05406f0c                        ← $updatedAt: 1773909602060 ms
                                          (2026-03-19 08:40:02 UTC)
                                        ← user properties follow (schema-dependent)

To properly decode the properties section, you need the document type schema — field names, types, required flags, and order. This is why the decode-document CLI tool (in packages/rs-scripts) requires the contract and document type to be specified.

Using the tool on this document produces:

id:         9LSAr59Fw7A1PHvX9WV1RWHCjL4PrijrHZpwhYDPkMq
owner_id:   4gY7wFM4o53jc8PJZ9KNzqzaJhXhPVMivJREKVwihKVF
created_at: 2026-03-10 09:23:43 UTC
updated_at: 2026-03-19 08:40:02 UTC
revision:   197

properties:
  amount: (i64)191000
  coreFeePerByte: (i64)1
  outputScript: bytes 76a914...88ac
  pooling: (i64)0
  status: (i64)2
  transactionIndex: (i64)9815
  transactionSignHeight: (i64)2440497

The decode-document CLI tool

For convenience, the rs-scripts crate provides a decode-document binary that handles all of this deserialization automatically:

# Install
cargo install --path packages/rs-scripts

# Decode a withdrawal document from base64
decode-document -c withdrawals -d withdrawal "AgIintqUs1vl..."

# Decode from hex
decode-document -c withdrawals -d withdrawal "0202229eda94b35b..."

# Use a contract ID instead of a name
decode-document -c 4fJLR2GYTPFdomuTVvNy3VRrvWgvkKPzqehEBpNf2nk6 -d withdrawal "..."

See packages/rs-scripts/README.md for full usage details.

Common pitfalls for third-party deserializers

  1. The serialization version varint changed the layout. If your code was written for version 0 or 1, version 2 documents will have different field offsets due to the $creatorId field. Always read the version varint first and branch accordingly.

  2. Integer encoding differs between v0 and v1+. In version 0, a u8 field occupies 8 bytes (encoded as i64). In version 1+, it occupies 1 byte. Parsing with the wrong version assumption will shift every subsequent field.

  3. The time fields bitfield is variable-length data. The 2-byte bitfield tells you how many time fields follow. If you assume a fixed number of time fields, any document with a different set of time fields (e.g., one that includes $transferredAt or block heights) will be misaligned.

  4. Property order is schema position order, not alphabetical. Each property in the data contract schema has a position field. Properties are serialized in ascending position order (stored in an IndexMap). If you assume alphabetical order or JSON declaration order, fields will be read from the wrong positions.

  5. Optional fields have a presence byte. If you forget to read the 0x00/0x01 prefix for optional fields, every subsequent field will be shifted by one byte.

  6. ByteArray encoding depends on size constraints. Fixed-size byte arrays (where minItems == maxItems in the schema) have no length prefix. Variable-size byte arrays have a varint length prefix. Check the schema to know which encoding is used.

  7. A typed array's element width comes from its items schema. Each element is laid out as a required property of the element's type, so an integer element bounded 0..100 is 1 byte and an unbounded one 8, and a fixed-size byte array or identifier element has no length prefix. Parse the items schema exactly as a property schema to know the width, including the contract's sizedIntegerTypes setting.

  8. In version 3, the same document type can produce different property layouts. A property annotated with requiredSince is presence-flagged in documents stamped below the annotation and raw in documents stamped at or above it. Two version-3 documents of the same type may therefore differ in layout — always read the stamp varint and resolve each property's requiredness against it before decoding the properties section.

Derive Macros

In the previous chapter we looked at the PlatformVersionEncode and PlatformVersionedDecode traits and the platform_encode_to_vec / platform_versioned_decode_from_slice functions. But you rarely implement those traits by hand. Instead, you use three derive macros from the rs-platform-serialization-derive crate:

  • PlatformSerialize -- generates a high-level serialize_to_bytes() method
  • PlatformDeserialize -- generates a high-level deserialize_from_bytes() method
  • PlatformSignable -- generates a signable_bytes() method that excludes signature fields

These macros live in packages/rs-platform-serialization-derive/src/lib.rs and are the glue that connects Rust struct definitions to the platform serialization system.

PlatformSerialize and PlatformDeserialize

These two macros work as a pair. Let us start with how they are used on the ConsensusError type:

#![allow(unused)]
fn main() {
#[derive(
    thiserror::Error,
    Debug,
    Encode,
    Decode,
    PlatformSerialize,
    PlatformDeserialize,
    Clone,
    PartialEq,
)]
#[platform_serialize(limit = 2000)]
#[error(transparent)]
pub enum ConsensusError {
    // ...
}
}

And on a leaf error struct:

#![allow(unused)]
fn main() {
#[derive(
    Error, Debug, Clone, PartialEq, Eq, Encode, Decode, PlatformSerialize, PlatformDeserialize,
)]
#[error("Document {document_id} is already present")]
#[platform_serialize(unversioned)]
pub struct DocumentAlreadyPresentError {
    document_id: Identifier,
}
}

And on the StateTransition enum:

#![allow(unused)]
fn main() {
#[derive(
    Debug, Clone, Encode, Decode,
    PlatformSerialize, PlatformDeserialize, PlatformSignable,
    From, PartialEq,
)]
#[platform_serialize(unversioned)]
#[platform_serialize(limit = 100000)]
pub enum StateTransition {
    DataContractCreate(DataContractCreateTransition),
    DataContractUpdate(DataContractUpdateTransition),
    Batch(BatchTransition),
    // ...
}
}

Notice that StateTransition uses two #[platform_serialize] attributes -- one for unversioned and one for limit. These are combined internally.

What the derives generate

PlatformSerialize generates an implementation of one of two traits, depending on whether unversioned is set:

With unversioned -- implements PlatformSerializable:

#![allow(unused)]
fn main() {
// Generated code (simplified):
impl PlatformSerializable for DocumentAlreadyPresentError {
    type Error = ProtocolError;

    fn serialize_to_bytes(&self) -> Result<Vec<u8>, Self::Error> {
        let config = bincode::config::standard()
            .with_big_endian()
            .with_no_limit();
        bincode::encode_to_vec(self, config)
            .map_err(|e| {
                ProtocolError::PlatformSerializationError(
                    format!("unable to serialize DocumentAlreadyPresentError: {}", e)
                )
            })
    }

    fn serialize_consume_to_bytes(self) -> Result<Vec<u8>, Self::Error> {
        // same as above, taking self by value
    }
}
}

Without unversioned -- implements PlatformSerializableWithPlatformVersion:

#![allow(unused)]
fn main() {
// Generated code (simplified):
impl PlatformSerializableWithPlatformVersion for ConsensusError {
    type Error = ProtocolError;

    fn serialize_to_bytes_with_platform_version(
        &self,
        platform_version: &PlatformVersion,
    ) -> Result<Vec<u8>, ProtocolError> {
        let config = bincode::config::standard()
            .with_big_endian()
            .with_limit::<{ 2000 }>();
        platform_serialization::platform_encode_to_vec(self, config, platform_version)
            .map_err(|e| match e {
                bincode::error::EncodeError::Io { inner, index } =>
                    ProtocolError::MaxEncodedBytesReachedError {
                        max_size_kbytes: 2000,
                        size_hit: index,
                    },
                _ => ProtocolError::PlatformSerializationError(
                    format!("unable to serialize ConsensusError: {}", e)
                ),
            })
    }
}
}

The key differences: the unversioned variant uses plain bincode::encode_to_vec, while the versioned variant uses platform_serialization::platform_encode_to_vec which threads the PlatformVersion through the encode chain. When a limit is set, the config uses with_limit::<{ N }>() and the error mapping converts bincode IO errors to MaxEncodedBytesReachedError.

In both cases, the derive also generates bincode Encode and Decode implementations by internally calling into derive_bincode -- this is why you still need Encode and Decode in the derive list alongside PlatformSerialize and PlatformDeserialize.

The #[platform_serialize] attributes

The #[platform_serialize(...)] attribute accepts several parameters. Here is the full list from the derive macro source in packages/rs-platform-serialization-derive/src/lib.rs:

limit = N

Sets the maximum serialized size in bytes:

#![allow(unused)]
fn main() {
#[platform_serialize(limit = 2000)]
}

When encoding exceeds this limit, the error is MaxEncodedBytesReachedError. This is critical for types received from the network.

unversioned

Generates PlatformSerializable instead of PlatformSerializableWithPlatformVersion:

#![allow(unused)]
fn main() {
#[platform_serialize(unversioned)]
}

Use this when the type's serialization format does not change between protocol versions. Most leaf types (individual error structs, simple data holders) use this.

passthrough

For enums, serializes by delegating directly to the inner variant's serialization method:

#![allow(unused)]
fn main() {
#[platform_serialize(passthrough)]
}

When MyEnum::Variant(inner) is serialized with passthrough, it calls inner.serialize() directly rather than encoding the enum tag + inner data. This means the enum variant information is lost in serialization -- deserialization must use platform_version_path to know which variant to decode into.

Cannot be combined with limit, untagged, or into.

untagged

For enums, serializes without the variant tag number:

#![allow(unused)]
fn main() {
#[platform_serialize(untagged)]
}

Similar to passthrough but still uses the enum's own encoding logic rather than delegating to the inner type. The variant index is omitted from the output. Like passthrough, deserialization requires knowing which variant to expect.

Cannot be combined with passthrough.

into = "TypePath"

For structs, converts the value to another type before serialization:

#![allow(unused)]
fn main() {
#[platform_serialize(into = "DataContractInSerializationFormat")]
}

The generated code calls .into() to convert to the target type, then serializes that type. This is useful for types that have a different in-memory representation than their serialization format.

Cannot be used on enums.

platform_version_path = "..."

Used with passthrough or untagged to specify how to determine the correct variant during deserialization:

#![allow(unused)]
fn main() {
#[platform_serialize(
    passthrough,
    platform_version_path = "dpp.contract_versions.contract_serialization_version.default_current_version"
)]
}

The deserialization code reads this field path from the PlatformVersion to determine which variant index to decode.

crate_name = "..."

Overrides the default crate path (defaults to crate):

#![allow(unused)]
fn main() {
#[platform_serialize(crate_name = "dpp")]
}

allow_prepend_version and force_prepend_version

These flags are defined in the code but currently not actively used in the main serialization path. They were designed for prepending version bytes to serialized output. Only one can be used at a time.

The #[platform_error_type] attribute

By default, the derives use ProtocolError as the error type. You can override this:

#![allow(unused)]
fn main() {
#[derive(PlatformSerialize, PlatformDeserialize)]
#[platform_error_type(MyCustomError)]
pub struct MyType { ... }
}

The error type must have PlatformSerializationError(String) and MaxEncodedBytesReachedError { max_size_kbytes, size_hit } variants (or equivalent conversion paths).

PlatformSignable -- the signature hash derive

The PlatformSignable derive solves a specific problem: when you sign a state transition, you need to hash all the fields except the signature itself. You cannot include the signature in the data that was signed -- that would be circular.

Here is how it is used on DataContractCreateTransitionV0 in packages/rs-dpp/src/state_transition/state_transitions/contract/data_contract_create_transition/v0/mod.rs:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Encode, Decode, PartialEq, PlatformSignable)]
pub struct DataContractCreateTransitionV0 {
    pub data_contract: DataContractInSerializationFormat,
    pub identity_nonce: IdentityNonce,
    pub user_fee_increase: UserFeeIncrease,
    #[platform_signable(exclude_from_sig_hash)]
    pub signature_public_key_id: KeyID,
    #[platform_signable(exclude_from_sig_hash)]
    pub signature: BinaryData,
}
}

The #[platform_signable(exclude_from_sig_hash)] attribute marks fields that should be excluded from the signature hash. The derive generates:

  1. A new struct DataContractCreateTransitionV0Signable<'a> containing only the non-excluded fields as Cow references
  2. A From<&DataContractCreateTransitionV0> implementation for the signable struct
  3. An implementation of the Signable trait on the original struct

The generated code looks approximately like this:

#![allow(unused)]
fn main() {
// Generated by PlatformSignable derive:

#[derive(Debug, Clone, bincode::Encode)]
pub struct DataContractCreateTransitionV0Signable<'a> {
    data_contract: std::borrow::Cow<'a, DataContractInSerializationFormat>,
    identity_nonce: std::borrow::Cow<'a, IdentityNonce>,
    user_fee_increase: std::borrow::Cow<'a, UserFeeIncrease>,
    // signature_public_key_id -- excluded
    // signature -- excluded
}

impl<'a> From<&'a DataContractCreateTransitionV0>
    for DataContractCreateTransitionV0Signable<'a>
{
    fn from(original: &'a DataContractCreateTransitionV0) -> Self {
        DataContractCreateTransitionV0Signable {
            data_contract: std::borrow::Cow::Borrowed(&original.data_contract),
            identity_nonce: std::borrow::Cow::Borrowed(&original.identity_nonce),
            user_fee_increase: std::borrow::Cow::Borrowed(&original.user_fee_increase),
        }
    }
}

impl crate::serialization::Signable for DataContractCreateTransitionV0 {
    fn signable_bytes(&self) -> Result<Vec<u8>, ProtocolError> {
        let config = bincode::config::standard().with_big_endian();
        let intermediate: DataContractCreateTransitionV0Signable = self.into();
        bincode::encode_to_vec(intermediate, config).map_err(|e| {
            ProtocolError::PlatformSerializationError(
                format!("unable to serialize to produce sig hash \
                    DataContractCreateTransitionV0: {}", e)
            )
        })
    }
}
}

The Cow references avoid cloning the data just to produce a hash. The intermediate struct borrows from the original and only serializes the fields that matter for the signature.

PlatformSignable on enums

When used on an enum (like StateTransition), the derive generates a corresponding Signable enum where each variant wraps the signable version of its inner type:

#![allow(unused)]
fn main() {
// On StateTransition:
#[derive(PlatformSignable)]
pub enum StateTransition {
    DataContractCreate(DataContractCreateTransition),
    DataContractUpdate(DataContractUpdateTransition),
    // ...
}

// Generates:
#[derive(Debug, Clone, bincode::Encode, derive_more::From)]
pub enum StateTransitionSignable<'a> {
    DataContractCreate(DataContractCreateTransitionSignable<'a>),
    DataContractUpdate(DataContractUpdateTransitionSignable<'a>),
    // ...
}
}

The signable_bytes implementation on the enum encodes a variant index (as u16) followed by the inner type's signable bytes:

#![allow(unused)]
fn main() {
impl Signable for StateTransition {
    fn signable_bytes(&self) -> Result<Vec<u8>, ProtocolError> {
        let config = bincode::config::standard().with_big_endian();
        let signable_bytes = match self {
            StateTransition::DataContractCreate(ref inner) => {
                let mut buf = bincode::encode_to_vec(&(0u16), config).unwrap();
                let inner_signable_bytes = inner.signable_bytes()?;
                buf.extend(inner_signable_bytes);
                buf
            }
            StateTransition::DataContractUpdate(ref inner) => {
                let mut buf = bincode::encode_to_vec(&(1u16), config).unwrap();
                let inner_signable_bytes = inner.signable_bytes()?;
                buf.extend(inner_signable_bytes);
                buf
            }
            // ...
        };
        Ok(signable_bytes)
    }
}
}

PlatformSignable attributes

  • exclude_from_sig_hash -- marks a field to be excluded from the signable struct
  • into = "TypePath" -- converts a field to a different type in the signable struct (used for fields that need a different representation for hashing)
  • derive_into -- on enums, generates From conversions from the original enum to the signable enum
  • derive_bincode_with_borrowed_vec -- manually implements bincode::Encode for the signable struct instead of deriving it (needed when fields contain borrowed Vec types)

The relationship between Encode/Decode and PlatformSerialize/PlatformDeserialize

This is a common source of confusion. Here is how the pieces fit together:

  • Encode / Decode (from bincode) -- field-level encoding. These know how to write each field to bytes and read it back. They are the low-level building blocks.

  • PlatformVersionEncode / PlatformVersionedDecode (from rs-platform-serialization) -- version-aware field-level encoding. These wrap Encode/Decode with a PlatformVersion parameter.

  • PlatformSerialize / PlatformDeserialize (derive macros) -- high-level serialization. These generate the serialize_to_bytes() and deserialize_from_bytes() methods that configure bincode, enforce size limits, and convert errors.

When you derive all of them on a type, the call chain is:

your_type.serialize_to_bytes()           // PlatformSerialize-generated method
  -> platform_encode_to_vec(...)         // from rs-platform-serialization
    -> your_type.platform_encode(...)    // PlatformVersionEncode (auto-generated)
      -> field.encode(encoder)           // bincode Encode for each field

You need Encode and Decode in the derive list alongside PlatformSerialize and PlatformDeserialize because the platform derives internally delegate to bincode encoding.

Rules

Do:

  • Always derive Encode, Decode, PlatformSerialize, PlatformDeserialize together
  • Use #[platform_serialize(unversioned)] for types with a stable serialization format
  • Use #[platform_serialize(limit = N)] on types received from untrusted sources
  • Use #[platform_signable(exclude_from_sig_hash)] on signature and signature key ID fields
  • Keep the #[platform_error_type] attribute consistent with the crate's error type

Do not:

  • Use passthrough on structs (it is enum-only)
  • Use into on enums (it is struct-only)
  • Combine passthrough with limit, untagged, or into
  • Combine force_prepend_version with allow_prepend_version
  • Forget that PlatformSignable on enums requires each inner type to also implement Signable
  • Change the order of fields in a PlatformSignable struct -- this changes the signature hash, which invalidates existing signatures

Platform Addresses

Dash Platform has its own address system, independent of the legacy Base58Check addresses used on the Core chain. Platform addresses use Bech32m encoding (BIP-350), following the specification in DIP-0018. There are three address types: two transparent and one shielded.

If you are coming from Bitcoin or Dash Core, the key mental shift is this: platform addresses are not derived from a single private key via a single algorithm. They are typed containers for a hash or a public key, unified under a single encoding scheme with a shared human-readable prefix.

The Three Address Types

TypeInner DataBech32m Type BytePrefix (mainnet)Prefix (testnet)
P2PKH20-byte pubkey hash0xb0dash1k...tdash1k...
P2SH20-byte script hash0x80dash1s...tdash1s...
Orchard43-byte shielded address0x10dash1z...tdash1z...

The type byte is the first byte of the Bech32m data payload. It determines how the rest of the payload is interpreted. The type bytes were chosen so that the first character after dash1 (or tdash1) is a memorable letter: k for keys (P2PKH), s for scripts (P2SH), z for zero-knowledge (Orchard).

PlatformAddress

The transparent address type is defined in packages/rs-dpp/src/address_funds/platform_address.rs:

#![allow(unused)]
fn main() {
pub enum PlatformAddress {
    P2pkh([u8; 20]),
    P2sh([u8; 20]),
}
}

Two variants, both wrapping a 20-byte hash. A P2pkh address contains a Hash160(compressed_pubkey), just like on Core. A P2sh address contains a Hash160(redeem_script), supporting standard multisig scripts.

Bech32m Encoding

The human-readable part (HRP) depends on the network:

#![allow(unused)]
fn main() {
const PLATFORM_HRP_MAINNET: &str = "dash";
const PLATFORM_HRP_TESTNET: &str = "tdash";  // also devnet, regtest
}

Encoding produces addresses like:

  • dash1krma5z3ttj75la4m93xcndna9ullamq9y5e9n5rs (P2PKH, mainnet)
  • tdash1sppl5xpu70aka8nacc4kj2htflydspzkc8jtru5 (P2SH, testnet)

The wire format is type_byte || hash -- 21 bytes total, then Bech32m-encoded with the appropriate HRP.

Storage vs. Bech32m: Two Byte Schemes

There is an important distinction between how addresses are encoded for users and how they are serialized for storage. The Bech32m type bytes (0xb0, 0x80) are not the same as the bincode variant indices (0x00, 0x01) used in GroveDB keys:

ContextP2PKH byteP2SH byte
Bech32m (user-facing)0xb00x80
Bincode (storage/wire)0x000x01

This matters when you encounter raw bytes. If the leading byte is 0xb0 or 0x80, you are looking at a Bech32m payload. If it is 0x00 or 0x01, it is bincode-serialized. The to_bytes() method produces bincode format; to_bech32m_string() produces the user-facing string.

Conversion to Core Addresses

PlatformAddress can be converted to and from dashcore::Address, the type used by the Core chain RPC client. This allows the platform layer to interoperate with Core's address format when needed -- for example, when processing withdrawals that ultimately create Core chain transactions.

OrchardAddress

The shielded address type is defined in packages/rs-dpp/src/address_funds/orchard_address.rs:

#![allow(unused)]
fn main() {
pub struct OrchardAddress(grovedb_commitment_tree::PaymentAddress);
}

It wraps the Orchard protocol's native PaymentAddress, which consists of two components:

ComponentSizePurpose
Diversifier11 bytesEntropy for deriving unlinkable addresses from a single spending key
pk_d32 bytesDiversified transmission key (Pallas curve point)

Total: 43 bytes. The Bech32m payload is 0x10 || diversifier || pk_d -- 44 bytes.

Key Constants

#![allow(unused)]
fn main() {
const ORCHARD_DIVERSIFIER_SIZE: usize = 11;
const ORCHARD_PKD_SIZE: usize = 32;
const ORCHARD_ADDRESS_SIZE: usize = 43;
const ORCHARD_TYPE: u8 = 0x10;
}

Diversifiers and Unlinkability

A single Orchard spending key can derive an unlimited number of addresses by varying the diversifier. Each address looks completely unrelated to any other address derived from the same key. Only the holder of the corresponding Incoming Viewing Key (IVK) can link them.

This is the fundamental privacy property: a merchant can give every customer a unique address, and no observer can determine that all those addresses belong to the same wallet. The diversifier is the mechanism that makes this possible.

Differences from Zcash Encoding

Dash's OrchardAddress uses the same 43-byte raw format as Zcash Orchard addresses (identical diversifier + pk_d structure). But the Bech32m encoding is Dash-specific:

  • No F4Jumble: Zcash applies an F4Jumble permutation before encoding; Dash does not.
  • No Unified Address wrapper: Zcash wraps Orchard addresses in a Unified Address (UA) envelope with typecodes and length prefixes; Dash uses a simple type-byte scheme.
  • Different HRP: Zcash uses u1; Dash uses dash/tdash.

The result is simpler, shorter addresses that are not interoperable with Zcash wallets.

Converting to Orchard's Native Type

The builder functions that construct shielded transactions need the Orchard library's native PaymentAddress, not our wrapper. Conversion is straightforward:

#![allow(unused)]
fn main() {
impl From<&OrchardAddress> for PaymentAddress {
    fn from(address: &OrchardAddress) -> Self {
        *address.inner()
    }
}
}

This is used in the shielded builder module (packages/rs-dpp/src/shielded/builder/) when adding outputs to an Orchard bundle.

Address Witnesses

When a transparent address is used as an input (for example, funding a shield transition), the sender must prove ownership. This is done through an AddressWitness, defined in packages/rs-dpp/src/address_funds/witness.rs:

#![allow(unused)]
fn main() {
pub enum AddressWitness {
    P2pkh {
        signature: BinaryData,
    },
    P2sh {
        signatures: Vec<BinaryData>,
        redeem_script: BinaryData,
    },
}
}

P2PKH Witnesses

A P2PKH witness contains a single 65-byte recoverable ECDSA signature. The public key is recovered from the signature rather than transmitted alongside it -- saving 33 bytes per witness. Verification recovers the public key, hashes it with Hash160, and checks the result against the address hash.

P2SH Witnesses

A P2SH witness contains the signatures and the redeem script. Only standard bare multisig scripts are supported:

OP_M <pubkey1> <pubkey2> ... <pubkeyN> OP_N OP_CHECKMULTISIG

Constraints:

  • Maximum 17 signature entries (MAX_P2SH_SIGNATURES) -- 16 keys plus 1 dummy for the CHECKMULTISIG bug
  • Only compressed public keys (33 bytes)
  • No timelocks, hash puzzles, or custom scripts
  • The redeem script must hash to the address: Hash160(script) == address_hash

Verification Cost Tracking

Every witness verification has a measurable cost. The AddressWitnessVerificationOperations struct tracks the work:

#![allow(unused)]
fn main() {
pub struct AddressWitnessVerificationOperations {
    pub ecdsa_signature_verifications: u16,
    pub message_hash_count: u16,
    pub pubkey_hash_verifications: u16,
    pub script_hash_verifications: u16,
    pub signable_bytes_len: usize,
}
}

These counts feed into the fee system. A P2PKH witness costs one ECDSA verification. An M-of-N multisig costs N verifications (all public keys are checked against all signatures, per Bitcoin's CHECKMULTISIG semantics).

How Addresses Are Used in Shielded Transitions

Each of the five shielded transition types uses addresses differently:

Shield (Transparent to Shielded)

The sender provides one or more PlatformAddress inputs, each with a nonce and a maximum contribution amount. An AddressWitness proves ownership of each input. The funds enter the shielded pool and are received by an OrchardAddress embedded in the Orchard bundle's encrypted outputs.

#![allow(unused)]
fn main() {
pub struct ShieldTransitionV0 {
    pub inputs: BTreeMap<PlatformAddress, (AddressNonce, Credits)>,
    pub input_witnesses: Vec<AddressWitness>,
    pub actions: Vec<SerializedAction>,  // contains encrypted OrchardAddress recipient
    pub amount: u64,
    // ...
}
}

Shielded Transfer (Shielded to Shielded)

No transparent addresses are involved. The sender spends notes from the shielded pool and creates new notes for the recipient's OrchardAddress and a change OrchardAddress. Both addresses are hidden inside the Orchard bundle -- only the respective recipients can decrypt them.

#![allow(unused)]
fn main() {
pub struct ShieldedTransferTransitionV0 {
    pub actions: Vec<SerializedAction>,  // spend + output pairs
    pub value_balance: u64,             // fee extracted from pool
    // ...
}
}

Unshield (Shielded to Transparent)

The sender spends shielded notes and sends the funds to a PlatformAddress visible on-chain. The transparent output address is explicitly included in the transition and bound into the Orchard sighash to prevent substitution.

#![allow(unused)]
fn main() {
pub struct UnshieldTransitionV0 {
    pub output_address: PlatformAddress,  // visible transparent recipient
    pub unshielding_amount: u64,
    pub actions: Vec<SerializedAction>,   // spend + change output
    // ...
}
}

Shielded Withdrawal (Shielded to Core Chain)

Similar to unshield, but the destination is a Core chain script rather than a platform address. The CoreScript holds a raw P2PKH or P2SH script that will be used in a Core chain withdrawal transaction.

#![allow(unused)]
fn main() {
pub struct ShieldedWithdrawalTransitionV0 {
    pub output_script: CoreScript,  // Core chain P2PKH or P2SH
    pub core_fee_per_byte: u32,
    pub pooling: Pooling,
    // ...
}
}

Shield From Asset Lock (Core Chain to Shielded)

An asset lock proof from the Core chain funds the shielded pool directly. The recipient is an OrchardAddress inside the Orchard bundle. No PlatformAddress inputs are needed -- the asset lock proof substitutes for them.

The transition also carries an optional surplus_output: Option<PlatformAddress>. When the consumed asset lock exceeds shield_amount + pool_fee, the leftover surplus is credited to this platform address; if it is unset, the surplus folds into the fee pools (bounded by shielded_implicit_fee_cap). See Entry-Transition Fees. Unlike the transparent recipients of Unshield/ShieldedWithdrawal (which are bound through the Orchard sighash extra_data above), surplus_output is bound through the state transition's own platform_signable signature -- it sits before the signature field, so it is part of the signed payload and cannot be substituted or truncated after signing. The Orchard extra_data therefore does not carry it; what it binds instead is the asset lock itself (see below).

The Platform Sighash

When transparent fields need to be bound to an Orchard bundle's proof, the platform uses a custom sighash computation defined in packages/rs-dpp/src/shielded/mod.rs:

#![allow(unused)]
fn main() {
const SIGHASH_DOMAIN: &[u8] = b"DashPlatformSighash";

pub fn compute_platform_sighash(
    bundle_commitment: &[u8; 32],
    extra_data: &[u8],
) -> [u8; 32] {
    let mut hasher = Sha256::new();
    hasher.update(SIGHASH_DOMAIN);
    hasher.update(bundle_commitment);
    hasher.update(extra_data);
    hasher.finalize().into()
}
}

The bundle_commitment is a BLAKE2b-256 hash of the Orchard bundle (per ZIP-244). The extra_data binds transparent fields:

Transitionextra_data
Shield0x84 || SHA-256(input addresses)
Shield From Identity0x85 || identity_id
Shielded Transferempty
Unshieldoutput_address.to_bytes() || amount.to_le_bytes()
Shielded Withdrawaloutput_script.as_bytes()
Shield From Asset Lock0x86 || asset lock identifier

This binding is critical for security. Without it, an attacker who intercepts an unshield transition could substitute the output_address while reusing the valid Orchard proof and signatures. The sighash ensures the Orchard bundle's spend authorization signatures commit to the specific transparent recipient.

The three transitions that only create notes bind something for a different reason. Their bundles have no spends, so they carry no anchor of their own: the proof and the binding signature verify wherever the bundle is submitted. Left unbound, the authorized bundle is a self-contained object anybody could lift out of the mempool and wrap in a transition of their own, paid with their own credits, and any bundle ever published could be replayed. The copier gains nothing — they pay the full amount to the original recipient — but the copy lands a second note with the same commitment and the same nullifier in the credit pool, so only one of the two can ever be spent.

Each of these bundles therefore binds a kind tag and its owner, the thing that funds it and that a third party cannot authorize:

  • Shield: the SHA-256 of its input addresses, each in its 21-byte encoding (PlatformAddress::to_bytes), in the order the transition serializes its inputs. The nonces and contributed amounts are not included: the owner says who funds the bundle, not which transition carries it.
  • Shield From Identity: the identity id.
  • Shield From Asset Lock: the asset lock identifier, the double SHA-256 of the locked 36-byte outpoint (the same value an identity created from that lock would get as its id; the tag keeps the two apart). Because a successful shield consumes the whole lock, a bundle bound to it can land at most once, even when its own sender resubmits it. Binding only the transaction id would not do this, and would let the holder of another credit output of the same lock transaction lift the bundle.

The tags 0x84, 0x85 and 0x86 sit above the state transition type range, next to the token pools' outputs-only tags 0x80–0x83. The binding starts at protocol version 14; at earlier protocol versions Shield and Shield From Asset Lock bind nothing, and a client building for one of those versions must bind nothing too. Both sides read the choice from the same version field (dpp.methods.credit_pool_bundle_binding), so a client that builds with the network's protocol version produces what that network verifies.

Shield From Asset Lock also changes its transition version at protocol version 14: version 1 carries the bound bundle and is the only version 14 admits, while version 0, whose bundle binds nothing, is admitted only up to 13. A version 0 presented at 14, such as one still waiting in the mempool when 14 activates, is refused on its version byte when the transition is decoded, before any proof is verified: nothing is charged and its asset lock stays unspent, so its sender can resubmit it as version 1 from the same lock. Checking its unbound bundle against the bound preimage instead would fail the proof and burn the proof-failure penalty from an honest lock.

The binding does not prevent Faerie Gold. The rho of an outputs-only note is the nullifier of its bundle's dummy spend, so a sender who builds and signs a fresh bundle reusing the same dummy spend note and rseed gets the same commitment and the same nullifier whatever the preimage binds, and a recipient that counts deposits by commitment credits two payments where only one can be spent. Nor does it stop a Shield or Shield From Identity funder landing their own bundle twice through a new transition, as a client retrying with a fresh nonce does. Neither is closed: that would need the bundles' dummy nullifiers recorded and checked, which the platform does not do.

The same compute_platform_sighash function is used on both sides: the client uses it when signing the bundle, and the platform uses it when verifying.

Trial Decryption and Address Privacy

A shielded recipient does not appear anywhere in cleartext on-chain. To discover incoming payments, a wallet must attempt trial decryption of every new note using its Incoming Viewing Key (IVK).

The process works as follows:

  1. The wallet retrieves new note entries from the shielded pool (each entry contains a nullifier, cmx, and 216-byte encrypted_note).
  2. For each entry, the wallet constructs a CompactAction from the nullifier, commitment, ephemeral public key, and encrypted ciphertext.
  3. The wallet attempts trial decryption using try_note_decryption with its IVK.
  4. If decryption succeeds, the note was addressed to one of the wallet's OrchardAddress instances.

The nullifier stored alongside each note is essential -- it provides the Rho value needed for decryption. Without it, the Orchard protocol's forward secrecy mechanism would prevent the recipient from recovering the note.

Because a single spending key can generate unlimited OrchardAddress instances (via different diversifiers), the IVK-based scan catches all of them in a single pass. The diversifier that was used becomes apparent only after successful decryption.

Note Encryption Structure

Each SerializedAction contains an encrypted_note field of 216 bytes:

ComponentSizePurpose
epk32 bytesEphemeral public key for Diffie-Hellman key agreement
enc_ciphertext104 bytesNote plaintext encrypted to recipient (ChaCha20-Poly1305)
out_ciphertext80 bytesNote encrypted to sender for wallet recovery

The enc_ciphertext contains the note plaintext (52 bytes), a Dash-specific memo (36 bytes), and the AEAD authentication tag (16 bytes). The memo is smaller than Zcash's 512-byte memos -- Dash uses the DashMemo type (36 bytes) rather than ZcashMemo (512 bytes), keeping encrypted notes compact.

Storage in GroveDB

Notes are stored in a BulkAppendTree within the shielded credit pool:

AddressBalances / "s" (shielded_credit_pool) /
    [1] notes       -- CommitmentTree (BulkAppendTree)
    [2] nullifiers   -- Tree (spent note markers)
    [5] total_balance -- SumItem
    [6] anchors      -- Tree (block_height -> anchor)

Each note entry stores:

cmx (32 bytes) || nullifier (32 bytes) || encrypted_note (216 bytes) = 280 bytes

The nullifier is stored alongside the note (rather than separately) specifically to support trial decryption. A scanning client needs both the encrypted ciphertext and the nullifier to attempt decryption.

Rules and Guidelines

Do:

  • Use to_bech32m_string() for user-facing address display. Always include the network parameter.
  • Use to_bytes() (bincode format) for storage keys and wire serialization.
  • Generate a fresh diversifier for each new Orchard payment address to maximize unlinkability.
  • Always bind transparent fields into the platform sighash. Forgetting to include the output address in an unshield transition would be a critical vulnerability.

Do not:

  • Mix up Bech32m type bytes (0xb0, 0x80, 0x10) with bincode variant bytes (0x00, 0x01). They serve different purposes and are not interchangeable.
  • Assume Orchard addresses are interoperable with Zcash. The encoding differs (no F4Jumble, no Unified Address wrapper).
  • Use non-standard scripts in P2SH addresses. Only bare multisig (OP_M ... OP_N OP_CHECKMULTISIG) is supported.
  • Store or transmit the raw spending key. Use the Incoming Viewing Key for scanning and the Full Viewing Key for read-only wallet recovery.
  • Attempt trial decryption without the nullifier. The Orchard protocol requires the Rho derived from the nullifier to reconstruct the note.

Data Contracts

If you have ever built a traditional web application, you know the pattern: define a database schema, then write code that reads and writes data conforming to that schema. Dash Platform follows the same idea, but on a decentralized network. A data contract is the on-chain schema that defines what application data looks like, how it is indexed, and who can modify it.

Every application on Dash Platform -- whether it is DPNS (the naming service), DashPay (social payments), or your own custom dApp -- starts by registering a data contract. Once registered, users can create, update, and query documents that conform to that contract's schema. Think of the data contract as the CREATE TABLE statement and documents as the rows.

The DataContract Enum

Like almost every core type in Platform, DataContract is a versioned enum. You will find its definition in packages/rs-dpp/src/data_contract/mod.rs:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, PartialEq, From, PlatformVersioned)]
pub enum DataContract {
    V0(DataContractV0),
    V1(DataContractV1),
}
}

This is a pattern you will see over and over: the top-level type is an enum whose variants are concrete struct versions. The PlatformVersioned derive macro wires it into the protocol versioning system, and the From derive gives you free .into() conversions from either variant.

The enum also provides direct-access helpers for when you know exactly which version you are dealing with:

#![allow(unused)]
fn main() {
impl DataContract {
    pub fn as_v0(&self) -> Option<&DataContractV0> { ... }
    pub fn as_v1(&self) -> Option<&DataContractV1> { ... }
    pub fn into_v0(self) -> Option<DataContractV0> { ... }
    pub fn into_v1(self) -> Option<DataContractV1> { ... }
}
}

In tests, there are convenience methods as_latest() and into_latest() that always return the most recent variant. These are gated behind #[cfg(test)] because production code should never assume which version is "latest" -- the protocol version determines that.

What Lives Inside a Data Contract

The V0 struct contains the essentials. From packages/rs-dpp/src/data_contract/v0/data_contract.rs:

#![allow(unused)]
fn main() {
pub struct DataContractV0 {
    pub(crate) id: Identifier,
    pub(crate) version: u32,
    pub(crate) owner_id: Identifier,
    pub document_types: BTreeMap<DocumentName, DocumentType>,
    pub(crate) metadata: Option<Metadata>,
    pub(crate) config: DataContractConfig,
    pub(crate) schema_defs: Option<BTreeMap<DefinitionName, Value>>,
}
}

The key fields are:

  • id: A 32-byte identifier derived from the contract creation transaction. Globally unique.
  • version: A monotonically increasing counter. Every time the contract owner updates the contract, this increments.
  • owner_id: The identity that created (and can update) this contract.
  • document_types: A map from document type names (like "contactRequest" or "domain") to their DocumentType definitions, which include the JSON Schema, indexes, and mutability rules.
  • config: Contract-level configuration such as whether documents can be deleted, encryption key requirements, and so on.
  • schema_defs: Shared JSON Schema $defs that document types can reference.

What V1 Added

DataContractV1 extends V0 with several important capabilities. From packages/rs-dpp/src/data_contract/v1/data_contract.rs:

#![allow(unused)]
fn main() {
pub struct DataContractV1 {
    // All V0 fields...
    pub id: Identifier,
    pub version: u32,
    pub owner_id: Identifier,
    pub document_types: BTreeMap<DocumentName, DocumentType>,
    pub config: DataContractConfig,
    pub schema_defs: Option<BTreeMap<DefinitionName, Value>>,

    // New in V1:
    pub created_at: Option<TimestampMillis>,
    pub updated_at: Option<TimestampMillis>,
    pub created_at_block_height: Option<BlockHeight>,
    pub updated_at_block_height: Option<BlockHeight>,
    pub created_at_epoch: Option<EpochIndex>,
    pub updated_at_epoch: Option<EpochIndex>,
    pub groups: BTreeMap<GroupContractPosition, Group>,
    pub tokens: BTreeMap<TokenContractPosition, TokenConfiguration>,
    pub keywords: Vec<String>,
    pub description: Option<String>,
}
}

The additions fall into four categories:

  1. Timestamps and block tracking -- created_at, updated_at, created_at_block_height, updated_at_block_height, created_at_epoch, updated_at_epoch. These provide an immutable audit trail of when the contract was created and last modified.

  2. Groups -- BTreeMap<GroupContractPosition, Group>. Groups enable multiparty governance. Each group has a set of member identities with associated voting power and a required power threshold for actions.

  3. Tokens -- BTreeMap<TokenContractPosition, TokenConfiguration>. Contracts can now define and manage tokens with configurable supply limits, minting/burning rules, and governance controls.

  4. Searchability -- keywords and description make contracts discoverable through the platform's search system contract.

The Versioned Accessors Pattern

Here is where it gets interesting. You do not access data contract fields directly in most code. Instead, you go through accessor traits. These live in packages/rs-dpp/src/data_contract/accessors/.

The V0 getter trait, defined in accessors/v0/mod.rs:

#![allow(unused)]
fn main() {
pub trait DataContractV0Getters {
    fn id(&self) -> Identifier;
    fn id_ref(&self) -> &Identifier;
    fn version(&self) -> u32;
    fn owner_id(&self) -> Identifier;
    fn document_type_for_name(&self, name: &str)
        -> Result<DocumentTypeRef<'_>, DataContractError>;
    fn document_types(&self) -> &BTreeMap<DocumentName, DocumentType>;
    fn config(&self) -> &DataContractConfig;
    // ... and more
}

pub trait DataContractV0Setters {
    fn set_id(&mut self, id: Identifier);
    fn set_version(&mut self, version: u32);
    fn increment_version(&mut self);
    fn set_owner_id(&mut self, owner_id: Identifier);
    fn set_config(&mut self, config: DataContractConfig);
}
}

And the V1 getter trait extends V0, defined in accessors/v1/mod.rs:

#![allow(unused)]
fn main() {
pub trait DataContractV1Getters: DataContractV0Getters {
    fn groups(&self) -> &BTreeMap<GroupContractPosition, Group>;
    fn tokens(&self) -> &BTreeMap<TokenContractPosition, TokenConfiguration>;
    fn created_at(&self) -> Option<TimestampMillis>;
    fn updated_at(&self) -> Option<TimestampMillis>;
    fn keywords(&self) -> &Vec<String>;
    fn description(&self) -> Option<&String>;
    // ... and more
}
}

Notice that DataContractV1Getters has a supertrait bound on DataContractV0Getters. This means anything that implements V1 getters automatically has V0 getters too. You can use both interchangeably.

How the Enum Dispatches

The magic happens in packages/rs-dpp/src/data_contract/accessors/mod.rs, where the top-level DataContract enum implements both traits by dispatching to the inner variant:

#![allow(unused)]
fn main() {
impl DataContractV0Getters for DataContract {
    fn id(&self) -> Identifier {
        match self {
            DataContract::V0(v0) => v0.id(),
            DataContract::V1(v1) => v1.id(),
        }
    }

    fn version(&self) -> u32 {
        match self {
            DataContract::V0(v0) => v0.version(),
            DataContract::V1(v1) => v1.version(),
        }
    }
    // ... every method follows this pattern
}
}

For V1-only fields, the implementation gracefully handles V0 contracts:

#![allow(unused)]
fn main() {
impl DataContractV1Getters for DataContract {
    fn groups(&self) -> &BTreeMap<GroupContractPosition, Group> {
        match self {
            DataContract::V0(_) => &EMPTY_GROUPS,  // static empty map
            DataContract::V1(v1) => &v1.groups,
        }
    }

    fn tokens(&self) -> &BTreeMap<TokenContractPosition, TokenConfiguration> {
        match self {
            DataContract::V0(_) => &EMPTY_TOKENS,  // static empty map
            DataContract::V1(v1) => &v1.tokens,
        }
    }

    fn created_at(&self) -> Option<TimestampMillis> {
        match self {
            DataContract::V0(_) => None,
            DataContract::V1(v1) => v1.created_at,
        }
    }
}
}

This is a deliberate design choice. When code asks a V0 contract for its groups, it gets an empty map rather than an error. When it asks for a timestamp, it gets None. The calling code does not need to know or care which version it is working with -- it just checks whether the Option has a value or whether the map is empty.

Serialization Strategy

Data contracts use a two-step serialization approach. They are first converted to a DataContractInSerializationFormat (a common intermediate representation), then serialized to bytes using bincode with big-endian encoding:

#![allow(unused)]
fn main() {
impl PlatformSerializableWithPlatformVersion for DataContract {
    fn serialize_to_bytes_with_platform_version(
        &self,
        platform_version: &PlatformVersion,
    ) -> Result<Vec<u8>, ProtocolError> {
        let serialization_format: DataContractInSerializationFormat =
            self.try_into_platform_versioned(platform_version)?;
        let config = bincode::config::standard()
            .with_big_endian()
            .with_no_limit();
        bincode::encode_to_vec(serialization_format, config)
            .map_err(|e| PlatformSerializationError(
                format!("unable to serialize DataContract: {}", e)
            ))
    }
}
}

This intermediate format is important because serialization versions and code structure versions are independent. A contract serialized as V1 ten years ago must still be deserializable, even if the code structures have evolved to V5 by then. The serialization format acts as the bridge.

There is also versioned_limit_deserialize, which imposes a size limit and always performs full validation -- this is used for data coming from untrusted sources (anything not from Drive's own storage).

Evolving a Contract: Adding Required Fields

Contract updates are deliberately conservative: existing documents must stay valid and their stored bytes must stay readable, so most schema changes that would break either are rejected. Historically that froze the required set of a document type in both directions — requiredness is baked into the document wire format (required properties serialize raw, optional ones carry a presence flag), so changing it would desynchronize every stored document's bytes from the schema used to read them.

From protocol v14, an update may add a brand-new required property by annotating it with requiredSince equal to the contract version the update creates:

"properties": {
  "newField": { "type": "string", "maxLength": 63, "position": 4, "requiredSince": 3 }
},
"required": ["existingField", "newField"]

The rules, enforced by consensus (DataContractInvalidRequiredFieldsUpdateError, code 10276, on violation):

  • The annotation must name exactly the new contract version — requiredness can be neither pre-scheduled for a future version nor backdated.
  • Only brand-new properties can become required. Promoting an existing (optional) property is still rejected, as is removing anything from required or touching an existing requiredSince annotation.
  • On contract creation, requiredSince may only be 1.
  • Annotations sit on top-level properties listed in required; nested properties cannot carry them.

What happens to data:

  • Existing documents are grandfathered. Each document carries a contract version stamp recording the contract version its bytes conform to (see Document Serialization); a document stamped below a property's requiredSince may omit that property and still reads, transfers, and deletes normally.
  • New writes are held to the new schema. Creates must supply the property; replaces re-supply full content, so replacing a grandfathered document requires the new property and re-stamps the document at the current version — lazy migration, one document at a time.
  • Indexes are unaffected because index additions on update remain banned — a newly added required field cannot be indexed retroactively (there is no backfill).

Rules and Guidelines

Do:

  • Always access fields through the accessor traits (DataContractV0Getters, DataContractV1Getters), not by pattern-matching on the enum variant.
  • Handle V1-only fields being None or empty when the contract might be V0.
  • Use document_type_for_name() to retrieve document types -- it returns a proper error if the name does not exist.

Do not:

  • Use as_v0() / as_v1() in production code unless you genuinely need version-specific behavior. The trait accessors are the right abstraction.
  • Assume into_latest() exists outside of tests -- it is #[cfg(test)] only.
  • Forget that serialization versions persist forever. If you add a new field, old serialized contracts will not have it, and your deserialization must handle that gracefully.
  • Mutate a contract's id after creation -- it is derived from the creation transaction and must remain stable.

Contract Groups

An application often outgrows one data contract. Indexes cannot be added to a document type after the contract is registered, so a project that grows tends to ship a second contract beside the first rather than reshape the original. A token is usually best kept in a contract of its own, so its change-control groups and supply rules are not entangled with an application's document types. And a project that publishes a version 2 contract keeps version 1 alive for the documents already stored in it. The result is a family of contracts under one owner, and before protocol version 14 nothing in consensus state said they belonged together. A client could guess from the owner identity, but that identity might own unrelated contracts too, and a guess cannot be proved.

A contract group is the answer. It is an identity-owned set of contracts, contract document types and contract tokens. It is registered and grown through ordinary data contract create transitions, stored under its own root tree, and readable with one GroveDB proof. Anything that later needs to act on "the set as a whole", from an SDK deciding which contracts to preload to a future identity key bound to every contract in a group, gets a consensus-level answer to "which contracts are in this set".

Not the same thing as groups. A data contract declares groups for token change control (see Data Contracts). Those are multi-party sets of identities inside one contract, with voting power and a required threshold. A contract group is a set of contracts owned by identities. The two share a word and nothing else. The SystemLimits field that bounds a change-control group's members was renamed from max_contract_group_size to max_group_member_count in the same protocol version so the code does not confuse them either.

The Model

Three facts define a contract group:

  1. One identity owns it. The owner is the identity that signed the registering transition, and it is never on the wire. It can stand alone or name up to sixteen admins in the registration, existing identities other than itself that may add members alongside it. Admins act alone: there is no threshold and no vote.
  2. Its members are parts of contracts. A member is a whole contract, one document type of a contract, or one token of a contract. A contract can only enrol itself. Memberships are declared by the create transition of the contract that joins, never by a third party and never for someone else's contract.
  3. It is append-only. Memberships are recorded when the member contract is created. There is no update path and no leaving. Contracts are never deleted either. Together these two facts are what let the storage layout use plain references safely, as the storage section explains.

A group is registered inside a data contract create transition. The same transition may enrol the contract it creates, and later create transitions signed by any owner add more members. A group may also be registered empty and filled later.

How a Group Gets Its Id

The group id is never on the wire. It is derived from the registering identity and the transition's identity nonce, in packages/rs-dpp/src/contract_group/mod.rs:

#![allow(unused)]
fn main() {
pub const CONTRACT_GROUP_ID_DOMAIN: &[u8] = b"contract_group";

pub fn generate_contract_group_id(
    owner_id: &Identifier,
    identity_nonce: IdentityNonce,
) -> Identifier {
    let mut bytes = CONTRACT_GROUP_ID_DOMAIN.to_vec();
    bytes.extend_from_slice(owner_id.as_slice());
    bytes.extend_from_slice(&identity_nonce.to_be_bytes());
    Identifier::from(hash_double(bytes))
}
}

Compare the id of the contract created by the same transition: hash_double(owner_id || identity_nonce). The domain prefix keeps the two from ever colliding, and because both inputs are known before signing, a client knows the group id and the contract id before it broadcasts. That is what makes the one-transition case work: a create transition can register a group and, in the same membership list, join it by the id it is about to receive.

The Types

Everything a transition carries about contract groups lives in packages/rs-dpp/src/contract_group/mod.rs. There are four wire types and one stored type.

#![allow(unused)]
fn main() {
/// Who owns a contract group, and so who may add members to it.
pub enum ContractGroupOwner {
    SingleOwner(Identifier),
    OwnerAndAdmins {
        owner: Identifier,
        admins: BTreeSet<Identifier>,
    },
}

/// What part of the contract being created joins a contract group.
pub enum ContractGroupMember {
    Contract,
    DocumentType(DocumentName),
    Token(TokenContractPosition),
}

/// A declaration that a part of the created contract joins a contract group.
pub struct ContractGroupMembership {
    pub contract_group_id: Identifier,
    pub member: ContractGroupMember,
}

/// The registration of a new contract group. The owner is the signer and is not on the wire.
pub struct ContractGroupRegistration {
    pub admins: BTreeSet<Identifier>,
    pub name: Option<String>,
    pub description: Option<String>,
}
}

The owner is deliberately absent from the registration: the only valid value would be the transition's signer, so carrying it would add thirty-two signed bytes and an error for the one mistake that field could express. The admins are a BTreeSet, so a duplicate admin cannot be expressed and the encoding is canonical.

ContractGroupOwner is the stored form of ownership, built from the signer and the registration when the transition becomes an action: an empty admin set stores SingleOwner, anything else OwnerAndAdmins. It has the helpers validation needs: owner_id(), admin_ids(), admin_count() and may_add_members(&identity_id), which is true for the owner and for every admin.

The stored type wraps the owner with a version envelope, because it is persisted and must remain decodable forever:

#![allow(unused)]
fn main() {
#[platform_serialize(unversioned)]
pub enum ContractGroupInfo {
    V0(ContractGroupInfoV0),
}

pub struct ContractGroupInfoV0 {
    pub owner: ContractGroupOwner,
    pub name: Option<String>,
    pub description: Option<String>,
}
}

ContractGroupInfo derives PlatformSerialize, PlatformDeserializeTrusted and PlatformDeserializeUntrusted (see Derive Macros). The trusted decoder reads this node's own GroveDB; the untrusted one decodes bytes that arrived inside a proof. From<ContractGroupRegistration> builds the V0 variant, so the transition never carries a version tag for the info: Drive chooses the stored version.

Two details of the wire types are easy to miss. Both ContractGroupRegistration and ContractGroupMembership implement JsonSafeFields in packages/rs-dpp/src/serialization/json/safe_fields.rs, which the json_safe_fields derive on the transition requires for every field type. And the ContractGroupInfo module imports ProtocolError at the top even though no code in it names the type, because the PlatformSerialize derive expands to code that does.

Version 1 of the Create Transition

DataContractCreateTransition gains a second variant. Version 0 is unchanged and stays valid; version 1 is version 0 plus the two contract group fields. From packages/rs-dpp/src/state_transition/state_transitions/contract/data_contract_create_transition/v1/mod.rs:

#![allow(unused)]
fn main() {
pub struct DataContractCreateTransitionV1 {
    pub data_contract: DataContractInSerializationFormat,
    pub identity_nonce: IdentityNonce,
    pub contract_group: Option<ContractGroupRegistration>,
    pub contract_group_memberships: Vec<ContractGroupMembership>,
    pub user_fee_increase: UserFeeIncrease,
    #[platform_signable(exclude_from_sig_hash)]
    pub signature_public_key_id: KeyID,
    #[platform_signable(exclude_from_sig_hash)]
    pub signature: BinaryData,
}
}

Both new fields sit inside the signed payload. A plain contract creation in version 1 carries contract_group: None and an empty membership list, which is exactly what the TryFromPlatformVersioned<CreatedDataContract> conversion produces, so existing callers that build a transition from a contract keep working and simply emit version 1 under protocol version 14.

In JSON the transition looks like this (identifiers abbreviated):

{
  "$formatVersion": "1",
  "dataContract": { "...": "..." },
  "identityNonce": 7,
  "contractGroup": {
    "admins": ["GWRSAVFM…S31Ec"],
    "name": "cardgame",
    "description": "Rules, marketplace and token contracts of the card game"
  },
  "contractGroupMemberships": [
    { "contractGroupId": "8sJ6Rk…Q2mV", "member": "contract" },
    { "contractGroupId": "4hYb2N…kW9p", "member": { "documentType": "listing" } },
    { "contractGroupId": "4hYb2N…kW9p", "member": { "token": 0 } }
  ],
  "userFeeIncrease": 0,
  "signaturePublicKeyId": 1,
  "signature": "…"
}

The first membership joins the group this very transition registers, by its derived id. The other two join a group registered earlier, one the signer owns or administers. A group with a single owner leaves admins out or empty; the owner never appears, since it is the signer. Both group fields may be omitted from a version 1 object, which then reads as a plain creation.

Version Bounds

Which variant a client may send is governed by STATE_TRANSITION_SERIALIZATION_VERSIONS_V3, the table protocol version 14 selects (see Feature Versions):

#![allow(unused)]
fn main() {
contract_create_state_transition: FeatureVersionBounds {
    min_version: 0,
    max_version: 1,
    default_current_version: 1,
},
}

Below protocol version 14 a version 1 transition has to be refused before anything looks inside it, and the bounds table above does not do that on a node: only client-side code consults it. The gate is StateTransition::active_version_range, which answers 14 to the latest version for a version 1 create, so deserialize_from_bytes_untrusted_in_version fails with StateTransitionIsNotActiveError on a protocol version 13 node. Without that arm an upgraded node would create the contract, charge the fee and silently drop the group data while a pre-upgrade binary rejected the same bytes. At protocol version 14 the default moves to 1, which is why the factory test in packages/rs-dpp/src/data_contract/factory/v0/mod.rs compares the version of the transition it builds against default_current_version rather than a literal.

Accessors

Code reads the new fields through DataContractCreateTransitionAccessorsV1, following the same pattern the Data Contracts chapter describes for DataContractV1Getters:

#![allow(unused)]
fn main() {
pub trait DataContractCreateTransitionAccessorsV1 {
    fn contract_group(&self) -> Option<&ContractGroupRegistration>;
    fn contract_group_id(&self) -> Option<Identifier>;
    fn contract_group_memberships(&self) -> &[ContractGroupMembership];
}
}

A version 0 transition answers None, None and an empty slice. Validation and Drive never match on the variant; they ask the accessors and treat "no group data" as the ordinary case. contract_group_id() derives the id from the contract's owner and the transition's nonce, so nothing downstream recomputes the hash by hand.

Validation

Contract group checks slot into the existing validation pipeline at two stages, and the split between them is the important design decision. Everything that can be decided from the transition alone runs in basic structure, before the signature is checked, and costs nothing. Everything that needs a state lookup runs in state validation, after the signer is authenticated, and is paid for. Without that split an attacker could probe which group ids exist for free.

Basic Structure (Unpaid)

contract_group_basic_structure_error in packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/data_contract_create/basic_structure/v2/mod.rs runs after the existing contract checks and returns the first violation it finds, in this order:

  1. If the transition registers a group, its admins, when any, must number at most max_contract_group_admins (16) and must not include the signer, who is the owner.
  2. name, when present, must be 1 to max_contract_group_name_length (64) characters. description, when present, 1 to max_contract_group_description_length (256). Lengths count characters, not bytes.
  3. The membership list must hold at most max_contract_group_memberships_per_contract (16) entries.
  4. For each membership: a DocumentType member must name a document type of the created contract, and a Token member must name a token position the contract defines.
  5. No (group id, member) pair may repeat.
  6. A document type or token membership is redundant, and rejected, when the whole contract joins the same group in the same transition.

It is a free function rather than a method on the transition because drive-abci cannot add inherent methods to a type defined in dpp; the orphan rule forbids it.

State (Paid)

validate_contract_groups_against_state in .../data_contract_create/state/v1/mod.rs runs after the version 0 state checks (the contract must not already exist) and after the refersTo reference validation. It bills every lookup on the execution context and stops at the first failure:

  1. The group the transition registers must not exist yet. Its id was derived when the transition became an action, and that double hash is billed here, as the contract id derivation is.
  2. Every admin the registration names must be an existing non-masternode identity, the check change-control group members get; a missing one fails with ContractGroupAdminNotFoundError. Memberships are creation-only, so a mistyped admin could never be replaced.
  3. Every group a membership names must exist and must have the signer as its owner or one of its admins. The group registered by this same transition is skipped, since the signer owns it by construction, and each other group is fetched once however many memberships name it.

A failure here returns a BumpIdentityNonceAction carrying the errors: the identity pays for the lookups and its nonce advances, exactly as for any other paid validation failure.

The Errors

CodeErrorStage
10360ContractGroupMembershipsOverLimitErrorstructure
10361DuplicateContractGroupMembershipErrorstructure
10362RedundantContractGroupMembershipErrorstructure
10363ContractGroupMemberNotInContractErrorstructure
10364InvalidContractGroupAdminsErrorstructure
10366InvalidContractGroupNameLengthErrorstructure
10367InvalidContractGroupDescriptionLengthErrorstructure
41000ContractGroupAlreadyExistsErrorstate
41001ContractGroupNotFoundErrorstate
41002IdentityNotContractGroupOwnerOrAdminErrorstate
41003ContractGroupAdminNotFoundErrorstate

The basic errors live in packages/rs-dpp/src/errors/consensus/basic/contract_group/ and the state errors in .../consensus/state/contract_group/. Both sets were appended at the tail of their enums; StateError has a frozen-discriminant test that pins the four new variants, so an insertion before them fails it. Code 10365 is unassigned: the registrant-not-owner rule it served became inexpressible once the owner left the wire. The basic codes follow the change-control group range as their own block, and the state codes open a new hundred, 41000 to 41099, rather than borrowing from the identity range. See Error Codes for the code ranges.

From Transition to Action to Operations

The transform into action step gets a version 1 action to match the version 1 transition:

#![allow(unused)]
fn main() {
pub struct DataContractCreateTransitionActionV1 {
    pub data_contract: DataContract,
    pub identity_nonce: IdentityNonce,
    pub user_fee_increase: UserFeeIncrease,
    pub contract_group: Option<(Identifier, ContractGroupInfo)>,
    pub contract_group_memberships: Vec<ContractGroupMembership>,
}
}

Two things happen in the transformer (packages/rs-drive/src/state_transition_action/contract/data_contract_create/v1/transformer.rs). The registration becomes a ContractGroupInfo, the stored form, with the signer as owner, and the group id is derived once and carried alongside it, so nothing after this point needs the transition's owner and nonce; state validation reads the id from the action rather than hashing again. The BumpIdentityNonceAction transformer gained matching version 1 arms so a paid failure can still be turned into a nonce bump.

Converting the action into Drive operations is where ordering matters. The version 1 arm of into_high_level_drive_operations emits, in this order:

  1. UpdateIdentityNonce and UpdateIdentityContractNonce, as version 0 does.
  2. ApplyContract for the contract itself.
  3. RegisterContractGroup { contract_group_id, info }, if the transition registers a group.
  4. AddContractGroupMemberships { contract_id, memberships }, if it declares any.

The contract is applied before anything touches the group trees, and the registration comes before the memberships, so a contract may join the group its own transition registers: by the time the membership operation runs, the group's subtrees exist. Both operations are variants of ContractGroupOperationType in packages/rs-drive/src/util/batch/drive_op_batch/contract_group.rs, and they resolve to low-level GroveDB operations through the same DriveLowLevelOperationConverter machinery every other high-level operation uses (see Batch Operations).

Storage

Contract groups get their own root tree, RootTree::ContractGroups, at key 124. The number matters: a root key becomes a child of some existing node in the root Merk, and every write under that node's subtree then rewrites a slightly larger node. Key 124 attaches under Versions (120), which only the block-level app-version update writes, so no fee-bearing transition pays for it. A lower free key such as 68 would have attached under the asset-lock outpoints node and raised every identity create and top-up fee by 1480 credits for the life of the chain; the four conversion impls in packages/rs-drive/src/drive/mod.rs and the KnownPath mapping in the batch module were extended for it. Path helpers and the single-byte subtree keys live in packages/rs-drive/src/drive/contract_groups/paths.rs.

[124] ContractGroups
├── [0] Groups
│   └── <contract group id>
│       ├── [0] Info            -> Item(bincode ContractGroupInfo { owner, name?, description? })
│       ├── [1] Contracts       -> <contract id> -> Item([])
│       ├── [2] DocumentTypes   -> <contract id || document type name> -> Item([])
│       └── [3] Tokens          -> <contract id || token position, u16 BE> -> Item([])
└── [1] Members
    └── <contract id>
        ├── [0] Groups          -> <contract group id> -> Reference to Groups/<group>/[1]/<contract id>
        ├── [1] DocumentTypes   -> <document type name> -> <contract group id> -> Reference
        └── [2] Tokens          -> <token position> -> <contract group id> -> Reference

Groups is the forward store: everything about one group under its id. Members is the backwards index: everything one contract belongs to, under the contract id. Every leaf on the forward side is an empty item; the key carries all the information. Document type and token members sit on one level under a composite key, the contract id followed by the UTF-8 name or the two big-endian position bytes, rather than under a subtree per contract. The keys still sort by contract, and a flat level pages with a plain range after the cursor key. A subtree per contract would have forced a continuation page to descend into the cursor's contract, and GroveDB charges an empty descent against the page limit, which shortened every continuation page and ended a limit-one walk early. The decoders rebuild typed values from keys alone.

Why Plain References

The backwards entries are GroveDB Reference elements of type UpstreamRootHeightReference(1, …): keep the first path segment (the root tree key) and append the forward path, one hop. They are not GroveDB bidirectional references. Bidirectional references buy automatic cleanup when the target moves or disappears, at a cost on every write. Here neither can happen: memberships are append-only and contracts are never deleted, so a reference can never dangle. Plain references are cheaper and the invariant holds by construction.

Why Members Is Keyed by Contract First

The Members side is laid out so that "is document type D of contract C in group G" is one point lookup at [124, 1, C, 1, D, G], and the forward side answers it at [124, 0, G, 2, C || D]. That is the shape ContractBounds::ContractGroup on identity keys needs: for every member of a batch, one existence check under the contract the member belongs to. Today the batch transformer reads the contract's whole Members entry (at most max_contract_group_memberships_per_contract entries) into the action, for a batch signed by a group-bound key only, and the check runs in memory; the point lookup is there when a consumer wants it.

Writing

Registration (insert_contract_group_operations_v0) inserts the group's tree under Groups with batch_insert_empty_tree_if_not_exists, and if the tree was already there returns CorruptedDriveState. State validation guarantees the group is new, so hitting that branch means the two disagree, which is exactly what a corrupted-state error is for. It then writes the Info item and the three empty member subtrees.

Memberships (insert_contract_group_memberships_operations_v0) write one empty item on the forward side and one reference on the backwards side per membership, creating the intermediate trees on both sides as needed, each with an if-not-exists insert so a tree left by an earlier declaration of the same contract is kept rather than overwritten once memberships can be added after creation. Two memberships often need the same parent tree, for instance two document types of the same contract joining the same group both need Groups/<G>/DocumentTypes/<C>. GroveDB rejects a batch that operates twice on one slot under batching consistency verification, so the inserter keeps a HashSet of tree paths it has already created in this batch and inserts each parent once.

Cost Estimation

Both inserts have an estimation twin under estimated_costs/ that declares EstimatedLayerInformation for every layer the write can touch (see Cost Tracking): the root, the ContractGroups tree, the Groups and Members levels, a group's own tree as a mix of three subtrees and one item, and so on down. Three constants size the unknowns: an info item is estimated at 1024 bytes (a sixteen-owner group with a full-length name and description stays under it), a backwards reference at 128 bytes, and a document type name key at 16 bytes. The Drive test should_build_the_same_operations_for_estimation_and_for_apply pins the estimation path to the apply path so the two cannot drift apart silently.

Creating the Trees

One helper, Drive::insert_contract_groups_structure, creates the root tree and its two subtrees. A fresh chain calls it from create_initial_state_structure_v4 (packages/rs-drive/src/drive/initialization/v4/mod.rs), which DRIVE_VERSION_V9 selects; a chain upgrading to protocol version 14 calls it from transition_to_version_14, the migration hook the Versioned Dispatch chapter describes. Sharing the helper is what keeps a node started at 14 and a node upgraded to it byte-identical under this key, the same reason the shielded pool has insert_shielded_pool_structure. Both subtrees are created up front so a registration only ever writes under Groups and a membership only under Groups and Members; no write path has to check whether the top of the tree exists.

Reading and Proving

A group can be joined by any number of contracts, so nothing reads or proves a whole group at once. Drive splits reading into the group's information, a single item, and its members, read one kind at a time in pages bounded by a limit. Everything is versioned through DriveContractGroupMethodVersions:

  • fetch_contract_group_info(group_id) reads only the Info item. State validation uses its _with_fee variant, which returns the FeeResult for the lookup alongside the result. prove_contract_group_info and verify_contract_group_info do the same through a proof, with None for a group that is provably absent.
  • fetch_contract_group_members(group_id, &query, limit) returns one ContractGroupMembersPage of one kind. The ContractGroupMembersQuery names the kind, Contracts, DocumentTypes or Tokens, and carries a start_after cursor: a contract id alone, or a contract id with a document type name or a token position. Entries come back in key order, at most limit of them, and page.next_query() is the query for the page after it, None once a page is empty. limit must lie between 1 and the node's max_query_limit, on the fetch side and on the proof side, so no page and no proof grows with the size of the group. prove_contract_group_members and verify_contract_group_members take the same query and limit.
  • fetch_contract_group_memberships_for_contract(contract_id) returns a ContractGroupMembershipsForContract: the groups the whole contract joined, the groups each document type joined and the groups each token joined. It needs no limit: memberships are recorded at creation only and capped per transition, so a contract belongs to at most sixteen. prove_contract_group_memberships_for_contract and verify_contract_group_memberships_for_contract are its proof pair.

Three path queries in packages/rs-drive/src/drive/contract_groups/queries.rs drive all of them. contract_group_info_query asks for the one info key under the group's tree. contract_group_members_query reads one kind's level: the whole level on a first page, or a range after the cursor's key on a continuation, where the key is the contract id for Contracts and the contract id followed by the name or position for the other two kinds. There is no subquery to descend into, so a page can end anywhere and the next page resumes exactly after it. contract_group_memberships_for_contract_query reads a contract's Members entry with a conditional subquery per kind. The same PathQuery is used to fetch and to prove, so what a node reads locally and what a client verifies are the same set of elements.

Verification lives in packages/rs-drive/src/verify/contract_groups/. Each verify function rebuilds its result from the proved (path, key, element) triples with the same decoders the fetches use, in types.rs. Member pages carry no data in their elements: every entry is read from its path and key. The info item is the one thing that decodes bytes, and the verifier uses the untrusted ContractGroupInfo decoder because proof bytes came from someone else, where the fetch uses the trusted one.

Two GroveDB behaviours shape this code and are worth knowing before you touch it:

  • A path query over a tree that does not exist is an error, not an empty result. The fetches call grove_has_raw on the group's or contract's tree first and return None or an empty result when it is absent.
  • GroveDB cannot build a proof inside an open transaction. prove_* with Some(&transaction) fails with NotSupported. Tests commit first and prove with None.

The DAPI Queries

Three DAPI queries sit on top of these proofs, one per path query, so a client verifies exactly the elements a node read. Handlers live under packages/rs-drive-abci/src/query/contract_group_queries/ and are versioned by contract_group_queries in the query version table.

  • getContractGroupInfo(contractGroupId) answers with the group's owner, admins (empty for a single owner), name and description, or with no result when no group has the id. The proved form proves the info item or its absence; rs-drive-proof-verifier verifies it into a ContractGroupInfo.
  • getContractGroupMembers(contractGroupId, members, limit) answers with one page of one kind. members is a oneof that names the kind and carries its cursor: contracts { start_after: contract id }, document_types { start_after: contract id + name } or tokens { start_after: contract id + position }. limit defaults to and is capped by max_returned_elements (100) on the node and in the verifier alike, because a client rebuilds the page query from the request it sent: an omitted limit has to mean the same page everywhere. The response is the page in the same kind, or a proof, verified into a ContractGroupMembersPage.
  • getContractGroupsForContract(contractId) answers with the contract's memberships as a whole, by document type name and by token position, empty lists when it belongs to no group. The proved form proves that too, verified into a ContractGroupMembershipsForContract.

In rs-sdk, ContractGroupInfo, ContractGroupMembersPage and ContractGroupMembershipsForContract implement Fetch and FetchUnproved (packages/rs-sdk/src/platform/contract_groups.rs); a ContractGroupMembersPageQuery carries the group, kind, cursor and limit, and its after(&page) is the query for the next page. The wasm-sdk exposes getContractGroupInfo, getContractGroupMembers (a page object whose nextStartAfter is the next cursor) and getContractGroupsForContract, each with a WithProofInfo twin, and js-evo-sdk wraps them as sdk.contractGroups.

Versioning Touchpoints

Contract groups arrived with protocol version 14, which was unreleased while they were built, so every table below was edited in place rather than copied into a new version (the rule in Coding Conventions).

TableWhat changed
STATE_TRANSITION_SERIALIZATION_VERSIONS_V3contract_create_state_transition max version 0 → 1, default 0 → 1
DRIVE_STATE_TRANSITION_METHOD_VERSIONS_V4data_contract_create_transition converter 0 → 1 (emits the group operations)
DRIVE_VERSION_V9create_initial_state_structure 3 → 4 (creates the root tree)
DRIVE_ABCI_VALIDATION_VERSIONS_V10the contract create basic_structure v2 and state v1 modules gained the group checks
DriveMethodVersionsnew contract_group: DriveContractGroupMethodVersions slot (insert, fetch, prove, cost estimation), DRIVE_CONTRACT_GROUP_METHOD_VERSIONS_V1, every method at 0
DriveVerifyMethodVersionsnew contract_group: DriveVerifyContractGroupMethodVersions slot
SystemLimitsfour new limits, plus the max_group_member_count rename

The four limits:

LimitValue
max_contract_group_memberships_per_contract16
max_contract_group_admins16
max_contract_group_name_length64 characters
max_contract_group_description_length256 characters

Numbers go in SystemLimits and nowhere else, so validation reads them from platform_version.system_limits rather than from constants in the validation module.

Fees

Nothing about contract groups has a special fee. Storage is charged at the standard rates for what is written: the info item and three empty subtrees for a registration, one empty item plus one reference per membership, plus whichever intermediate trees a new contract needs on the backwards side. State validation bills each group lookup as a precalculated operation. A transition that fails basic structure costs nothing; one that fails state validation pays for the lookups it caused and has its identity nonce bumped.

Keys Bound to a Group

An AUTHENTICATION key can carry ContractBounds::ContractGroup { id } instead of a contract bound (protocol version 14; see docs/protocol/contract-bound-authentication-keys.md for the full rules). The key may then sign only batches, and a batch member on contract C is inside the bounds when C, the member's document type, or the member's token is a member of the group. The batch transformer (v2) first resolves the signer's bounds from the identity the signature was validated against (handed over through StateTransitionSignerAwareActionTransformer, since the unversioned transformer trait is not to be changed). Only when the signing key is an AUTHENTICATION key bound to a contract group does it read the group memberships of every contract the batch touches into the action with fetch_contract_group_memberships_for_contract_with_fee, keeping the fee of the read next to them the way a contract's fetch info does. Batch advanced-structure validation v1 then judges the key's bounds from the action alone: it checks contains for the whole-contract membership and for the exact one, and bills that fee. A batch signed by any other key is transformed exactly as before: no extra read, no extra fee, nothing new that can fail. A miss is the same paid ContractBoundedKeyOutOfBoundsError a contract bound produces. Registration goes through contract-bounds validation v2, which requires AUTHENTICATION, a non-MASTER level and an existing group (one billed fetch_contract_group_info_with_fee); encryption and decryption keys cannot be group-bound, since their bounds are opt-in per contract and a group has nothing to opt in with. Below protocol version 14 a transition carrying a group-bound key is not active (active_version_range), so it is rejected at decode without a charge, the way a binary that cannot decode the variant rejects it.

Drive keeps group-bound keys in the identity's contract-info level under the group id, next to contract ids, through the ContractGroupBased apply info and the ContractGroupBoundKey key request. Nothing new is written to the ContractGroups tree. Two consequences worth knowing: memberships are append-only, so a group-bound key's reach grows with the group; and IdentityCreateFromShieldedPool refuses group-bound keys (shielded-proof validation v1), because its sighash preimage layout predates them.

What Is Not There Yet

The first protocol version 14 change ships the consensus core only. Known gaps, all deliberate:

  • A group owned by a contract's change-control group. A third ContractGroupOwner variant, where the owner is a Group inside a contract rather than a set of identities, was specified and left out. The simple reading, "the signer is a member of that group with power above zero", is the recommended first step; full required-power group actions can follow.
  • Updating or leaving. Memberships are creation-only. A later DataContractUpdateTransition version would be needed to add members from an existing contract or to remove any.
  • Relaxing redundancy. A document type membership is refused when the whole contract already joins the same group. That could be allowed if a consumer wants the explicit entry.
  • An owners index. There is no identity → groups it owns tree. Finding the groups an identity owns means scanning Groups.
  • Creation surfaces. The DAPI queries and the Rust, wasm and JavaScript read bindings exist (see The DAPI Queries), but registering a group or joining one still needs a hand-built DataContractCreateTransitionV1: the wasm-dpp JSON fields and the JavaScript, Kotlin and Swift contract-create options follow separately.

Tests

Coverage sits at the three layers the feature touches (see Unit Tests):

  • dpp (contract_group/mod.rs): the id derivation differs from the contract id and depends on both inputs; both owner kinds resolve membership; the stored info round-trips through bincode with the untrusted decoder.
  • drive (drive/contract_groups/tests.rs): the root tree exists in the initial structure at protocol version 14 and not before; a group can be registered and proved present or absent; memberships land on both sides and prove; members page through with a cursor, a limit of one still reaches every entry past a contract with nothing left, and a zero or over-the-maximum page size is refused on both the fetch and the proof side; estimation and apply build the same operations; registering an existing group is refused.
  • drive-abci queries (query/contract_group_queries/*/v0/mod.rs): each handler rejects a malformed id, cursor or limit; an absent group or a contract in no group answers empty, unproved and proved; members page through with a cursor and come back in the requested kind; every proof verifies through the Drive::verify_* twin.
  • drive-proof-verifier (proof/contract_groups.rs, unproved.rs): request parsing fails before verification on a missing version, a short id, a missing kind or an out-of-bounds limit; unproved responses decode both owner kinds, reject a page of another kind, and read memberships into sets.
  • drive-abci (data_contract_create/contract_group_tests.rs): end-to-end through process_raw_state_transitions, covering register-and-join in one transition, an owner adding a later contract by document type and token, joining a group the identity does not own as a paid failure, joining an unknown group, a group with admins accepting the owner and each admin and refusing outsiders, a registration naming an unknown admin rejected as a paid failure, a version 0 create still processed at the latest version, and the full set of malformed registrations and memberships rejected in basic structure. The protocol upgrade hook has its own test proving the trees exist and accept a registration after the 13 to 14 transition.
cargo test -p dpp --all-features -- contract_group
cargo test -p drive --lib -- contract_groups
cargo test -p drive-abci -- contract_group data_contract_create
cargo test -p drive-abci --lib -- query::contract_group_queries
cargo test -p drive-proof-verifier --lib -- contract_groups

Rules and Guidelines

Do:

  • Read contract group data through DataContractCreateTransitionAccessorsV1 and treat "no registration, no memberships" as the normal case. Never match on V0 versus V1 to find out.
  • Derive the group id with generate_contract_group_id or contract_group_id(). Never hash by hand and never accept an id from the wire.
  • Keep lookups in state validation and everything else in basic structure. A check that needs Drive must be paid for.
  • Use the path helpers in paths.rs. The subtree keys are single bytes and easy to transpose.
  • Read members in pages with a limit. Never add a query that enumerates a whole group; the info item is the only thing read without a bound.

Do not:

  • Confuse contract groups with a contract's change-control groups. Different types, different limits, different trees.
  • Write to the Members side without the matching forward entry, or the other way round. Every membership is two writes and the decoders assume both exist.
  • Turn the backwards references into bidirectional references. The invariant that makes plain references safe (append-only, no deletion) is what the design relies on; if that invariant ever changes, the storage layout changes with it.
  • Add a membership path outside a create transition. The model is creation-only until an update transition version says otherwise.
  • Prove inside an open transaction, or path-query a tree you have not confirmed exists.

Contract Moderation

An application that stores user content needs a way to keep an abusive identity out. Before protocol version 14 nothing in consensus state could do that: the contract owner could delete nothing the user wrote, could stop nothing the user would write next, and a client-side blocklist bound nobody but the client that kept it. Every other user's node still accepted the identity's documents.

Contract moderation is the answer. A data contract may declare, in its config, that it keeps a banlist, a suspension list and/or a warning list of identities, and who may edit them. An identity on the banlist, or on the suspension list with a suspension that has not lapsed, cannot act on the contract at the document level: every document transition it signs against the contract is refused, paid, in the mempool and in a block. Token transitions are not affected. A warning bars nothing: it is a record, with a reason and a block time, of a step short of a suspension, which the warned identity and everyone else can read and which accumulates until a moderator clears it. The lists live under the contract's own subtree in Drive, are edited by one new state transition, and are readable with one GroveDB proof.

The Model

Four facts define a contract's moderation:

  1. The contract declares it. DataContractConfigV2::moderation is an optional ContractModerationConfig { banlist, suspensions, warnings, moderators }. At least one list must be kept, unless a document type lets the moderators delete its documents (see Deleting Documents below) or keeps fields only they write (see Changing Document Fields below), in which case the declaration may keep none. Which lists a contract keeps is decided when it is created and never changes: an update can not make an unmoderated contract moderated, turn a second list on, or turn a list off (validate_config_update 2, DataContractConfigUpdateError). Whoever writes documents under a contract knows from its first version whether and how they can be barred from it, which matters most where documents are assets: a ban also stops transfers and sales. And a list that is on may hold entries. Only the moderators may be changed by an update.
  2. The owner moderates, alone or with a fixed set, or an elected team does. ContractModerators is ContractOwner, AppointedModerators(set) or Elected(declaration); the third is its own section below. With the first two, a set is at most SystemLimits::max_contract_moderators (16) identities. The owner may always moderate and need not be named; it may be named, and then counts toward the 16. Naming it changes nothing about authority. Every identity named must exist: the contract create, and the contract update for the identities it adds, look each one up in state and refuse, paid, with ContractModeratorIdentityNotFoundError (41110). A moderator that does not exist can never sign, so naming one is a mistake, and catching it once at the declaration is cheaper than guarding every later reader of the set. Moderators act alone: there is no threshold and no vote. Neither the owner nor a moderator can be banned or suspended. An entry one of them already carries can still be lifted: a contract update may name as moderator an identity that is banned or suspended, the entry keeps binding it, and the owner or another moderator unbans or unsuspends it without demoting it first (never the identity itself: a moderation cannot target its own signer).
  3. A ban lasts until an unban; a suspension lasts until a block time. A suspension names the block time, in milliseconds, at which it lapses. A lapsed suspension is not deleted by the clock: the first document transition of the identity that runs at or after that time executes normally and, in the same execution, sweeps the stale entry. An explicit unsuspend deletes it too. A ban supersedes a suspension: banning a suspended identity removes the suspension, and suspending a banned identity is refused.
  4. A warning is a record, not a bar. A warn adds one warning, the block time and a reason, to the identity's entry on the warning list; the entry holds every warning the identity carries, oldest first, up to SystemLimits::max_contract_warnings_per_identity (16), past which a warn is refused until a clearWarnings deletes the entry whole. The document gate does not read the warning list, only the banlist and the suspension list (the warning list is read by a warn and a clearing alone); a ban leaves the warnings alone (they say how it came to that), and a banned or suspended identity may still be warned. What a warning is for is the client: an application shows the identity its warnings, and the moderators the count, before a suspension or a ban.

The Types

The declaration and the status live in packages/rs-dpp/src/data_contract/config/moderation/mod.rs:

#![allow(unused)]
fn main() {
pub struct ContractModerationConfig {
    pub banlist: bool,
    pub suspensions: bool,
    pub moderators: ContractModerators,
    pub warnings: bool,   // last: a declaration stored before the warning list existed does not decode
}

pub enum ContractModerators {
    ContractOwner,
    AppointedModerators(BTreeSet<Identifier>),
    Elected(Box<ElectedModerators>),      // see Elected Moderation below
}

pub enum ContractModerationList { Banlist, Suspensions, Warnings }

pub struct ContractModerationReason {
    pub code: Option<u16>,
    pub text: String,
    pub documents: Vec<ContractModerationDocument>,   // { document_type_name, document_id }
    pub reason_document_id: Option<Identifier>,       // a `reason` of the moderation charters contract
}

pub struct ContractBan { pub reason: ContractModerationReason }
pub struct ContractSuspension { pub until: TimestampMillis, pub reason: ContractModerationReason }
pub struct ContractWarning { pub warned_at: TimestampMillis, pub reason: ContractModerationReason }

pub struct ContractModerationStatus {
    pub ban: Option<ContractBan>,
    pub suspension: Option<ContractSuspension>,
    pub warnings: Vec<ContractWarning>,   // oldest first, empty when none
}
}

ContractModerationConfig::lists names the lists a contract keeps, in tree key order (banlist, suspension list, warning list); barring_lists the ones whose entries bar, which is every list but the warning list: what the document gate reads and what a ban's proof covers.

Every ban, every suspension and every warning carries a reason, stored with the entry so that whoever reads the list reads why. The text is free: at most SystemLimits::max_contract_moderation_reason_length (1024) bytes of UTF-8, possibly empty. The code is reserved for the ban codes a contract may declare in a later protocol version. No contract declares any today, so it is expected to be None; a moderator may still write any u16 there, and nothing checks it against anything. The reasonDocumentId names a reason document of the moderation charters contract, the ground the action is taken on: a seated elected team's ban, suspension, warning or deletion must name one its proposal lists (see The seated team below); for every other moderator it is stored as written, looked up nowhere. A reason may also cite the documents it is about, the posts a warning or a ban is for, each by its document type name and its id: at most SystemLimits::max_contract_moderation_reason_documents (16), none twice, with a type name a contract could admit (InvalidContractModerationReasonDocumentsError, 10904, unpaid). Nothing looks them up: a cited document may have been deleted since, by its author or by a moderator whose deletion left a record, or may never have existed, and a client that wants to show it fetches it or its removal record. The moderator pays the storage of the reason, documents included, byte for byte, and gets it back when the entry is removed.

The config is DataContractConfig::V2, a new variant of the config's own bincode enum inside the contract. The config version follows the platform version, as V1 did from protocol version 9: from protocol version 14 every new contract carries a V2 config, moderated or not (CONTRACT_VERSIONS_V6 sets both max_version and default_current_version to 2), and an existing V1 contract moves to V2 with its next update. config_valid_for_platform_version lowers a V2 only where the platform version does not admit it, never because of what it declares. Lowering would drop a moderation declaration, and moderation can never be turned on later, so that case is refused rather than dropped: serializing a contract whose config declares moderation at a platform version below 14 (ensure_admitted_by_platform_version), and parsing a config value with a moderation key at such a version, both fail with ProtocolError::NotSupported. For the same reason the declaration refuses an unknown key instead of skipping it (deny_unknown_fields, and the moderators' $type map likewise): a misspelled suspensions would otherwise leave the contract without the list for good. A contract create or update carrying a V2 config is active from protocol version 14 only (StateTransition::active_version_range): before that a node rejects it at decoding, unpaid, exactly as a binary that cannot decode the V2 discriminant does, so upgraded and older nodes agree on every block before activation. The JSON shape of the moderators is a flat {"$type": "contractOwner"}, {"$type": "appointedModerators", "identities": [...]} or {"$type": "elected", ...} with the declaration's keys beside its $type, the style of AuthorizedActionTakers.

The Transition

ContractUserModerationTransition (type 24) carries one action:

#![allow(unused)]
fn main() {
pub struct ContractUserModerationTransitionV0 {
    pub owner_id: Identifier,          // the moderator that signs
    pub data_contract_id: Identifier,
    pub identity_contract_nonce: IdentityNonce,
    pub action: ContractUserModerationAction,
    pub user_fee_increase: UserFeeIncrease,
    pub signature_public_key_id: KeyID,
    pub signature: BinaryData,
}

pub enum ContractUserModerationAction {
    Ban { identity_id, reason: ContractModerationReason },
    Unban { identity_id },
    Suspend { identity_id, until: TimestampMillis, reason: ContractModerationReason },
    Unsuspend { identity_id },
    Warn { identity_id, reason: ContractModerationReason },
    ClearWarnings { identity_id },
    DeleteDocument { document_type_name, document_id, reason: ContractModerationReason },
    RestoreDocument { document_type_name, document: BinaryData },
}
}

It is signed like a contract update: a CRITICAL authentication key without contract bounds, under the signer's contract-scoped nonce, and its minimum fee is the contract update floor. It activates with CONTRACT_USER_MODERATION_INITIAL_PROTOCOL_VERSION (14).

Validation

TierWhatCodes
Basic structure (unpaid)the target is not the signer; a suspension ends at or before SystemLimits::max_contract_suspension_until (2^53 - 1 ms, the largest value JSON clients read exactly); the text of a ban's, a suspension's or a warning's reason is at most SystemLimits::max_contract_moderation_reason_length bytes (its code is not checked) and the documents it cites are at most max_contract_moderation_reason_documents, well named and distinct10901, 10700, 10903, 10904
Signature and nonceCRITICAL key, contract nonceexisting
Transform (state, paid)the contract exists; it keeps the list the action edits; the signer is the owner or a moderator; the target of a ban, a suspend or a warn is neither; the target exists; the action fits the target's status41100-41106, 41109, 41117, 41118

The transform reads the contract and the target's status and refuses, paid, by bumping the signer's contract nonce. The action carries the status as read (and, for a warn, the warnings the target carries and the block time the new one is stamped with), so Drive edits the lists without reading them again, and the mempool, which transforms without a state validation stage, refuses with the same codes as a block. A suspend must end after the block time (41106). Suspending an identity that carries a suspension replaces it, longer or shorter. A warn is refused once the target carries max_contract_warnings_per_identity warnings (ContractUserWarningLimitReachedError, 41118), and a clearWarnings of an identity that carries none with ContractUserNotWarnedError (41117). A ban and a suspend read the barring lists the contract keeps; a warn and a clearing read the warning list alone.

The Document Gate

The gate sits in the batch transformer (a barred signer has every one of its transitions against the contract refused on its own, each with its nonce bump), transform_document_transitions_within_contract_v0, right after the contract is fetched, as its own versioned helper: contract_moderation_gate, selected by batch_state_transition.contract_moderation_gate (None up to protocol version 13, Some(0) from 14). That transformer is shared with every earlier protocol version, so the gate is a version-table fact there, not something inferred from contract data. A config that declares no moderation costs nothing: no read, no branch. Otherwise the transformer reads the owner's status on the lists the contract keeps, bills the read, and:

  • banned: every document transition of the batch on that contract except a deletion, and a replace on a type declaring retractedWhen, fails with ContractUserBannedError (41107), paid, each with its contract nonce bump;
  • suspended and not lapsed: the same with ContractUserSuspendedError (41108);
  • suspended and lapsed: the transitions go through and the batch action records the contract in lapsed_suspensions (the identity is always the batch owner, so it is not stored and nothing can queue another identity's); the batch converter (documents_batch_transition generation 1) appends one delete per contract.

Deletions (Delete and IndexOnlyDelete) are never refused: a barred identity can write nothing new, move nothing and sell nothing, but it may still take down what it wrote, under the document type's ordinary deletion rules. A batch of deletions alone carries on whole; in a mixed batch the deletions carry on next to the refusals. On a type whose documents the owner can not delete, the exit is a retraction instead: a type declaring retractedWhen has the gate let a barred signer's replaces through with its bar (ContractModerationRefusal::retraction_bar), and once the stored document is fetched the transformer (judge_barred_replace) refuses with that bar, paid, each replace whose written document does not meet the condition, judged as an immutable condition is. The type's other rules say what a retracted document may hold. Because the transformer runs in check_tx, a barred identity's documents never enter the mempool. The other party of a document transition is checked too: a transfer to a banned or live-suspended recipient, and a purchase from a banned or live-suspended seller, are refused with ContractModerationCounterpartyBarredError (41114), paid by the signer, so a barred identity collects neither assets nor proceeds on the contract. That read is billed to the batch; a counterparty's lapsed suspension is left for the counterparty's own next transition to sweep. No contract could declare moderation before protocol version 14, so older blocks replay unchanged through the same code.

The Errors

Basic, in their own band (10900-10949): InvalidContractModerationConfigError (10900), ContractModerationSelfTargetError (10901), DocumentActionFeesWithoutModerationError (10902, see Fee Pots and the Claim), ContractModerationReasonTooLongError (10903), InvalidContractModerationReasonDocumentsError (10904), InvalidContractModerationDocumentFieldsError (10905, a field change naming no field or a system property). State, in their own sub-band: ContractModerationNotEnabledError (41100), IdentityNotContractModeratorError (41101), ContractModerationTargetNotAllowedError (41102), ContractUserAlreadyBannedError (41103), ContractUserNotBannedError (41104), ContractUserNotSuspendedError (41105), ContractSuspensionNotInFutureError (41106), ContractUserBannedError (41107), ContractUserSuspendedError (41108), ContractModerationTargetNotFoundError (41109), ContractModeratorIdentityNotFoundError (41110, from the contract create and update, not from the moderation transition), ContractModerationCounterpartyBarredError (41114, from the document gate; 41111 to 41113 are reserved), ContractUserNotWarnedError (41117), ContractUserWarningLimitReachedError (41118), DocumentFieldNotChangeableByModeratorsError (41123) and DocumentModeratorFieldNotWritableError (41124) (see Changing Document Fields). A contract update that turns a list on or off is refused with the existing DataContractConfigUpdateError (40002). Elected moderation has its own band (41200-41299): ContractModeratedDocumentTypeNotYetUsableError (41200), ContractModerationAbilityNotGrantedError (41201), ModerationCharterAddedModeratorLimitReachedError (41202), ModerationReasonNotListedError (41203), and for the deletion of settled documents DocumentTypeNotDeletableOnceSettledError (41204), ContractModerationTeamNotSeatedError (41205), DocumentNotSettledError (41206), ContractTeamActionDoesNotExistError (41207), ContractTeamActionAlreadySignedError (41208), SettledDeletionNotRestorableError (41209), ContractTeamActionAlreadyCompletedError (41210), ContractTeamActionDocumentChangedError (41211) and ContractTeamMemberAddedAfterDocumentError (41212) (see Deleting Settled Documents). A discounted action fee the seated charter does not give is refused with DocumentActionFeeModeratorsShareMismatchError (40139), beside the other fee agreement errors.

Deleting Documents

A banlist keeps an identity out; it does not take down what the identity already wrote. A document type may let the contract's moderators do that:

"post": {
  "type": "object",
  "moderatorAbilities": { "delete": true },
  "properties": { "text": { "type": "string", "maxLength": 280, "position": 0 } },
  "additionalProperties": false
}

moderatorAbilities is a document type keyword of meta-schema v3, parsed by apply_moderator_abilities; its delete key is DocumentTypeV2::documents_can_be_deleted_by_moderators and its deleteWithin key documents_can_be_deleted_by_moderators_for. Its rules for deletion:

  • The contract declares moderation. The keyword on a contract without a moderation block is refused (InvalidContractStructure, 10231): moderation can not be switched on later, so nobody could ever delete anything. In return a moderation block may keep no list at all when at least one document type carries the keyword (ContractModerationConfig::validate takes that fact from the contract): a contract can moderate content without moderating users.
  • It is fixed with the type. A contract update can not add the keyword to an existing document type or take it away (DocumentTypeUpdateError, 40212): authors keep the rules they wrote under. A document type an update adds may carry it.
  • It is independent of canBeDeleted, which rules what a document's own owner may do. canBeDeleted: false with moderatorAbilities.delete: true is a post its author can not retract and moderation can remove.
  • Some types can not carry it: one that keeps history (Drive refuses to delete such documents), an indexOnly one (there is no stored row to name by id), and one that restricts creation (its documents are the contract owner's, which no moderator may delete). Transferable and tradeable types may, and so may a type with a deletion token cost, which a moderator does not pay.
  • It may come with a window. moderatorAbilities.deleteWithin: 86400 lets the moderators delete a document for that many seconds after its last modification ($updatedAt), and no longer: once block time is past $updatedAt plus the window the document is settled, and no moderator deletes it alone any more, the contract owner included (DocumentModerationWindowElapsedError, 41116); on a type that also sets deleteSettled, the seated team of an elected contract may still delete it together (see Deleting Settled Documents). At exactly $updatedAt plus the window the deletion still passes; the document's own owner still deletes it as canBeDeleted allows. Moderation acts on what was just written; it does not reach back alone into what has stood unchallenged. A replace or a price update moves $updatedAt, so new content opens the window again; a transfer or a purchase does not. The window needs the flag, at least one second, and the clock in the type's required, so that every document carries it: $updatedAt, or for a type with documentsMutable: false $createdAt instead, since nothing modifies such a document after its creation. A type whose documents can be replaced must require $updatedAt: measured from creation alone, an author could wait the window out and then rewrite a post into something no moderator can remove. The transform reads $updatedAt and falls back to $createdAt; like the flag it is fixed with the type, in both directions (a longer one would reopen documents that had settled). It is in seconds, as the other durations of a document type are, and it says nothing about a document's own owner, whose deletion canBeDeleted rules at any age.
  • For references it is no longer permanent. A permanentDocument reference refuses such a type (ReferencedDocumentTypeDeletableError, 40122) whatever its canBeDeleted says, so the guarantee that a validated permanent reference never dangles holds. When its owners can not delete its documents (canBeDeleted: false), it declares no ttl and it keeps removal records (the default), a document of it leaves state only on a moderator's record, and a moderatedDocument reference is the one that points at it (a deletableDocument reference is refused, ReferencedDocumentTypeModeratedError, 40144). A like or a reply that points at such a post declares refersTo: moderatedDocument: the reply stays valid through edits once a moderator removed the post, resolving to its removal record, and a join through it reports the removed post with its proven record. A type whose documents can also leave state otherwise, deleted by their owner, removed without a record or expiring, is a deletableDocument target, and a join through it reports a missing id. See References.
  • What a deletion leaves is the type's to say, and fixed with it. deleteKeepsRecord (default true, DocumentTypeV2::moderator_deletions_keep_records) says whether the deletion writes the removal record below, and deleteRefundsOwner (default false, moderator_deletions_refund_owner) whether the owner is refunded its storage. Both need delete: true and change with no update (40212). A type that keeps no record has no records subtree, is refused by getContractDocumentRemovals, and can never have a deletion restored (41119): its deletions are final. deleteKeepsFields (moderator_deletion_kept_fields) lists what of a deleted document stays public in its record: declared properties at any depth (meta.tags, or meta whole) and the timestamps and block heights the type requires, never a transient property, $id, $ownerId or a path inside another listed one. It needs a record, and is fixed with the type like the others.

The deletion is the seventh action of the same transition:

#![allow(unused)]
fn main() {
ContractUserModerationAction::DeleteDocument {
    document_type_name: String,
    document_id: Identifier,
    reason: ContractModerationReason,   // as on a ban: a code nothing checks, a text that may be empty
}
}

It names no identity (identity_id() is None): whose document it is is only known once the document is read. The transform checks, in order and each refusal paid: the document type exists (10406), it carries the keyword (DocumentTypeNotDeletableByModeratorsError, 41115), the signer is the owner or a moderator (41101), the document exists (DocumentNotFoundError), its owner is neither the contract owner nor a moderator (41102, the rule that protects them from a ban protects what they wrote), and block time is within the type's window after the document's last modification ($updatedAt, else $createdAt), when the type sets one (41116). The document is read the way a document's own deletion reads it, billed the same. The action carries the contract, the document's owner and the block time, so Drive reads nothing again. Nothing the document type prices is charged: neither its deletion token cost nor its actionFees deletion fee, both of which are what a document's own owner pays for deleting it.

Drive then runs DocumentOperationType::ForceDeleteDocument, the ordinary deletion (so every index and aggregate of the type stays right) without its canBeDeleted guard, which is the owner's rule and not the moderators', and writes a removal record:

#![allow(unused)]
fn main() {
pub struct ContractDocumentRemoval {
    pub document_owner_id: Identifier,
    pub moderator_id: Identifier,
    pub reason: ContractModerationReason,
    pub removed_at: TimestampMillis,   // the block time
    pub document_hash: [u8; 32],       // sha256d of the document as serialized under its type
    pub restoration: Option<ContractDocumentRestoration>, // { moderator_id, restored_at } once restored
    pub kept_fields: Vec<u8>, // what `deleteKeepsFields` keeps public, encoded as the document encodes its properties
}
}

The kept values are copied from the document as read by the transform and encoded the way the document encodes its properties (encode_kept_fields): for each path the type lists, in the list's order, 0 when the document holds no value there, or 1 and the value as an optional property of its type is written, an object length-prefixed with its members in order, a time or block height as eight big-endian bytes and a core block height as four. The paths are not written: the type lists them and never changes the list. So the record is read, as a document is, under its document type (ContractDocumentRemoval::kept_values), and each value comes back typed exactly as reading the document gives it; an object kept whole reads members an update added after the removal as absent, as a document written before them does. They are values, not index entries: nothing finds a record by them.

On a type that keeps records, the record is what is left to say that a document was removed, not lost (a type that keeps none writes nothing below, reads no existing record, and is proved by the document's absence, VerifiedDocuments with the id and none): a client holding a dangling id (a reply whose parent is gone) can prove who removed it, whose it was, when, why and what it was. The moderator pays for it, reason included, and nothing ever deletes it. From protocol version 14 a document id commits to the nonce of its create transition and is produced at most once, so the removed id can not be created again by anyone: the only way the document comes back is a moderator's restore (see Restoring Documents below), which marks the record restored and leaves it in place. A record and a live document of the same id therefore coexist exactly when the record is marked restored, and a restored document deleted again gets a fresh record in place of the marked one.

The hash is of the document as Document::serialize writes it under its document type and the protocol version of the deleting block, computed from the document as read, not from the bytes as stored: a document stored under an earlier version of its type or of the serialization would never re-serialize to its stored bytes, and a client keeping the document rather than the bytes could never match them. A client that may have to undo a deletion keeps the document it fetched before deleting; serialized under the same contract it gives the same bytes.

The deleted document's owner gets no storage refund, unless its type sets deleteRefundsOwner, in which case the batch refunds it as the owner's own deletion would, the moderator still paying for the transition and the record. Otherwise the batch carries ContractModerationOperationType::ForfeitStorageRefunds, a marker that writes nothing, and apply_drive_operations generation 1 turns every removal the batch's document operations cause into a removal attributed to nobody: the bytes still leave the system (FeeResult::removed_bytes_from_system), no refund is computed, and the credits stay in the storage pools they were distributed to. What the document operations remove is forfeited whoever paid for it: the document's bytes, its owner's or an earlier one's (a transferred document's bytes may still be its first owner's), and those of any index subtree the deletion empties, which an earlier author's document may have created. The rest of the batch is not: when it frees moderation storage someone is owed (a restored removal record a fresh one replaces, which may shrink, or the approvals and the info of a team action that the approval deleting the document moves to the closed actions or drops), the document operations are applied as a GroveDB batch of their own, after the rest and in the same transaction, so what the rest frees is refunded to whoever its flags name, a moderator who also paid for the document included, for those bytes alone; otherwise nothing else frees bytes (a fresh record is an insert, the nonce and the counts keep their size) and the batch is applied as one. An estimate carries no refund to begin with (refunds come from the flags of what is really removed), and prices the batch as it is applied, one GroveDB batch or two. An author who deletes the same document with an ordinary document transition is refunded as always.

Restoring Documents

A deletion can be undone. The eighth action of the same transition brings a deleted document back, as it was:

#![allow(unused)]
fn main() {
ContractUserModerationAction::RestoreDocument {
    document_type_name: String,
    document: BinaryData,   // the document serialized under its type, as it was when deleted
}
}

It names no identity and no id: the document is inside the bytes, and its id and owner with it. The transform checks, in order and each refusal paid: the document type exists (10406) and carries moderatorAbilities.delete (41115, only such a type keeps records), the signer is the owner or a moderator (41101, whoever deleted), the bytes decode under the type (a basic decoding error, never an execution error: the bytes are the signer's), the document has a removal record (ContractDocumentRemovalNotFoundError, 41119) that is not marked restored (ContractDocumentAlreadyRestoredError, 41122: the document is live), block time is within SystemLimits::contract_document_restore_window_ms of the removal, a week, the last millisecond included (DocumentRestoreWindowElapsedError, 41120), the bytes hash to what the record holds (DocumentRestoreHashMismatchError, 41121: the document as it was, not an edit of it), and no other document of the type holds a value of one of its unique indexes (DuplicateUniqueIndexError, 40105: the value was free while the document was gone, and someone may have taken it). The record read is billed; the hash pins everything else, so the owner, the revision, the timestamps, the references and the schema are not checked again. Whether the document's owner is banned is not asked either: a ban stops writes, it never removed documents.

The action carries the contract, the decoded document and the record marked restored by the signer at the block's time, so Drive puts the document back through the ordinary insert (DocumentOperationType::AddDocument, every index and aggregate of the type included) and replaces the record in place, without reading again. The restored document's storage flags name its owner, as they did before the deletion: the signer pays for the bytes, and the refund of a later deletion is the owner's, as it always was. Nothing the document type prices is charged, neither its creation token cost nor its actionFees creation fee, and no fee agreement is asked: a moderator undoes a moderation, it does not create content. The restore is proved by the record, now marked restored and holding the hash of the bytes the transition carried; the verifier reads the document's id out of those bytes under the contract's document type, so it needs the contract, which the SDKs register with their context provider before broadcasting.

Two consequences follow from the document coming back byte for byte. Its $updatedAt does not move, so a type with moderatorAbilities.deleteWithin may have settled it while it was gone: no moderator deletes it again alone until its author edits it, though the author still can, and on a type that sets deleteSettled the seated team can, by a fresh proposal (the action that deleted it ran and is closed). A deletion the team approved together is never restored: it is the only removal made after the document settled, since a moderator alone deletes up to the settling time and the team only after it, so the restore transform compares the removal time with the settling time of the document the bytes hash to (no read), and refuses the restore of a later removal, paid (SettledDeletionNotRestorableError, 41209), so no single moderator undoes what the leader and the members agreed on. And a type with a contested index can not carry moderatorAbilities.delete at all (InvalidContractStructure, 10231): a contested index only takes a document through a vote, which no restore can go through, so such a deletion could never be undone. The bytes are decoded under the type as the contract holds it when the restore is processed, and the hash is of the document serialized under the type as it was when the document was deleted: a contract update that changes the type's layout inside the window (a property added, say) leaves the record unrestorable, since bytes that decode under the new layout cannot hash to what the record holds. A client therefore serializes under the contract's current version, as Drive does, and keeps the document rather than the bytes.

Deleting Settled Documents

A document past its type's moderatorAbilities.deleteWithin window is settled: no moderator deletes it alone (41116). A type that also sets moderatorAbilities.deleteSettled lets the members of an elected contract's seated team delete it together, by approving its deletion one transition each; the approval that meets the rule deletes it. The rule is DocumentTypeV2::moderator_settled_deletion, a SettledDeletionRule { leader, approvals }: approvals members of the team must approve, the leader among them when leader is set. { leader: true } is the leader alone, { leader: true, approvals: 3 } the leader and two others.

The parser admits the key only beside deleteWithin, on a contract whose moderators are elected, with approvals of at least 1 and, at registration, at most the members the declared team can hold: its leader, SystemLimits::max_moderation_charter_elected_members (15, the maxItems of a charter's members) and the declaration's maxAddedModerators, so a rule no team could meet is refused rather than frozen with the type. All are InvalidContractStructure (10231) otherwise; the upper bound is a registration limit, not checked when a stored contract is read back, and the elected declaration must give the team deleteDocuments on the type (10900). It is fixed with the type in both directions (40212).

The team deletes a settled document the way a token group carries out an action: one member proposes it, the others approve it by its id, and it runs once the approvals meet the rule. What is kept meanwhile is a team action under the contract, a ContractTeamAction:

#![allow(unused)]
fn main() {
pub struct ContractTeamAction {
    pub proposer_id: Identifier,       // the member that proposed it, its first approval
    pub proposed_at: TimestampMillis,  // the block time of the proposal
    pub event: ContractTeamActionEvent,
}

pub enum ContractTeamActionEvent {
    DeleteSettledDocument {
        document_type_name: String,
        document_id: Identifier,
        document_last_modified_at: TimestampMillis, // the document's $updatedAt (else $createdAt) then
        document_revision: Option<Revision>,        // the document's $revision then, if it has one
        reason: ContractModerationReason,           // the proposer's
    },
}
}

The proposal is the transition's DeleteSettledDocument { document_type_name, document_id, reason }. Its id is computed, not carried: ContractTeamAction::settled_deletion_action_id, a double SHA-256 of the contract, the proposer, the proposer's nonce for the contract, the document type name, the document id and the reason (each part of it behind a presence byte or its length), which a proposer's nonce makes unique and the client knows before broadcasting; the reason is part of it so that the proof of the proposal is the proof of this one and not of another signed with the same nonce (ContractUserModerationTransition::team_action_id). The transform (transform_settled_deletion_proposal_v0) checks, in order and each refusal paid: the type exists (10406) and sets the rule (DocumentTypeNotDeletableOnceSettledError, 41204); a charter is seated (ContractModerationTeamNotSeatedError, 41205), since the rule names a seated team's leader and counts its members, so no interim moderator and no contract owner proposes; the signer is the leader or an active member (41101) and the declaration gives the team deleteDocuments on the type (41201); the reason names a reason document the team's proposal lists (ModerationReasonNotListedError, 41203); the document exists (40101) and its owner is not protected (41102); it is settled, block time past its last modification ($updatedAt, else $createdAt) plus the window (DocumentNotSettledError, 41206: within the window a moderator deletes it with DeleteDocument); and a signer the leader added was added before the document was created, unless the rule sets approversPredateDocument: false (ContractTeamMemberAddedAfterDocumentError, 41212: late_addition_refusal, which reads the addition's $createdAt from the seat the authority check already found, SeatedModerationCharter::seat_of, so no read is added). Both transforms take the last modification and the settling time from one helper (last_modified_and_settled_at), so a deletion alone and a proposal can never both pass, or both be refused, at one time. A rule the proposer meets alone runs at once.

An approval is the transition's ApproveTeamAction { action_id }: what it approves is the stored action's, so it carries nothing else. The transform (transform_team_action_approval_v0) checks, in order and each refusal paid: the contract keeps team actions, a document type of it setting deleteSettled (ContractTeamActionDoesNotExistError, 41207 otherwise); the action, read billed under the active and then the closed actions, exists (41207) and has not run (ContractTeamActionAlreadyCompletedError, 41210), which needs nothing of the team, since an action exists only once a seated member proposed it; a charter is seated (41205); the signer is on the team with deleteDocuments on the action's type (41101, 41201); its approvals are read, billed; the document exists (40101), its owner is not protected (41102), and it is still as proposed, its last modification and its revision unchanged (ContractTeamActionDocumentChangedError, 41211): every change of the document moves its revision, a moderator's change of its fields included, which leaves $updatedAt and so the settling time alone, and a transfer, a purchase or a price update moves it too; a signer the leader added was added before the document was created, as for a proposal (41212); and the approvals do not hold the signer's (ContractTeamActionAlreadySignedError, 41208), checked last so that a member taken off and added again too late, whose earlier approval no longer counts, is told 41212. Nothing lapses: a proposal the document never changed under stays active until it runs.

Whether it runs. The approvals given, this one counted, are compared with what the rule needs: its approvals, or all the seated team can hold when it asks for more (ElectedCharter::seats, the count the SDKs' ModerationTeam::seats gives a client too: the leader, the members the charter elected and the declaration's maxAddedModerators, the bound the moderators pot's settle reads the counts by, so a charter electing fewer members can still meet the rule; a member the leader removed still counts toward what the team can hold, so the leader can not lower the bar by removing members who would not approve, and gets the seat back by deleting the removal). Approvals too few to meet it read no seat and the approval is added. Otherwise the earlier approvers are checked against the seated team, one point read each (SeatedModerationCharter::seat_of, the leader needing none) while they are no more than the queries a read of the whole team makes (one for the removals when the charter elected anyone, one for the additions when the declaration allows any), and past that one read of the team (SeatedModerationCharter::fetch_active_seats), with the same answer for fewer billed reads (still_counted_approvers). The approvals of members who left no longer count and are dropped, refunded to them, and no longer prove: one who comes back after that approves again, while one back before any such read still has its approval counted. Under approversPredateDocument the same goes for a member the leader took off and added again since the document was created: on the team, but its seat (TeamSeat::Added { added_at }, the new addition's $createdAt) no longer counts for the document (TeamSeat::counts_toward, SettledDeletionRule::admits_addition), so its approval is dropped and a new one refused. When the approvals that count meet the rule (SettledDeletionRule::is_met_by, the leader read from the seated charter), the document is deleted as DeleteDocument deletes it (the shared document_deletion_context: ForceDeleteDocument, the removal record when the type keeps one, naming the member whose approval deleted it and the proposal's reason, and the refund as deleteRefundsOwner says), the action closes with the approvals that counted, and every counted approver's moderation action count goes up by one (next_moderation_action_count, the read count_for_signer makes too): a deletion signed by several counts for each of them, and approvals that fall short count for nobody, so that approving what never passes earns no share of the pot. The action carries what to write (ContractTeamActionWrite::Propose, Approve or Close), the deletion when there is one, and the counts, so Drive reads nothing again.

Storage mirrors a token group's actions (see the tree below): each approval is its own sum item under the action, flagged with the member that paid for it, so nothing is ever rewritten. The closing approval moves the action's info and the approvals that counted from the active actions (M) to the closed ones (X) without storage flags, which refunds each to its member, and deletes what is left under the active actions; the member whose approval closed the action pays for the closed copy, which nothing deletes. A closed action says who agreed to delete what and why, and when it was proposed (the block time of the deletion itself is in the removal record, when the type keeps one); an active one says who proposed it and who approved so far. Both are read with getContractTeamActions and getContractTeamActionSigners (see The DAPI Queries), which is how a member finds the actions to approve.

The proof of a proposal or an approval is the signer's approval wherever it is: the prover builds contract_team_action_signer_query, keys M and X with the subquery path [action id, S, signer], from the transition alone, and verify_contract_team_action_execution checks that the approval is under exactly one of them, returning VerifiedContractTeamActionSignature(contract, action id, status): active, or closed once the action ran and the document went. An approval moves only with its action, so the proof holds while the approval stands; only an approval dropped because its member left the team no longer proves. Closed says the action ran, by this approval or a later one: a proof taken after a later approval closed the action finds the signer's approval there.

Changing Document Fields

A deletion takes a document down; some moderation only annotates one. A report is handled, a post flagged, a ticket assigned, and the document should stay, with the moderators' word on it. A document type may keep fields for its moderators:

"report": {
  "type": "object",
  "documentsMutable": false,
  "moderatorAbilities": { "changeFields": ["status", "resolution"] },
  "properties": {
    "reason": { "type": "integer", "minimum": 0, "maximum": 8, "position": 0 },
    "status": { "type": "integer", "minimum": 1, "maximum": 3, "position": 1 },
    "resolution": { "type": "string", "maxLength": 200, "position": 2 }
  },
  "required": ["reason"],
  "additionalProperties": false
}

moderatorAbilities.changeFields is DocumentTypeV2::moderator_changeable_fields, parsed by apply_moderator_abilities. The listed properties are the moderators' to write, and only theirs:

  • A document's owner writes them only as a moderator. The batch transformer refuses a create that sets one, and a replace that changes, adds or removes one, with DocumentModeratorFieldNotWritableError (41124), unless the signer moderates the contract (transformer::v0::moderator_fields::judge_moderator_field_write, through common::moderators::moderator_field_write_refusal, which reads who moderates only when such a field is written). When the signer does moderate, the action is stamped as a moderator's: the create action's moderated flag, the replace action's moderated_at and moderated_by (which otherwise carry the stored document's stamp over, each half as stored), so the document is written with $moderatedAt the block's time and $moderatedBy the signer. It runs in the transformer, beside the aggregate reads, so the mempool refuses such a write as a block does; it is inert before protocol version 14, whose types keep no such field. A seated team's member must also hold changeDocumentFields on the type. So a report starts with no status, and its author can never mark it handled.
  • The fields are fixed with the type, in both directions (40212), and a type an update adds may declare them: a field only moderators write starts absent on every document, and one its owner could have set would no longer be theirs.
  • The parser keeps them out of what a change could break. Each must be a declared, optional, stored top-level property, not immutable, neither a reference nor read by one (a where entry's referring value, a findBy source or function param, a key id's identity property), neither generated nor a parameter of a generated property, and in no contested index; the type may not be indexOnly (10231). References are therefore never checked again when a moderator changes a document: what they read did not move. That matters for a report whose post a moderator already deleted: a replace re-checks every deletableDocument reference and would refuse it, while a moderator's change goes through. For the same reason a findBy key or an inList list on another type may not read a field moderators write (schema_property_is_fixed_once_written counts it as moving).
  • A type that lists any keeps a revision (DocumentTypeBasicMethods::requires_revision, through has_moderator_changeable_fields), even with documentsMutable: false: Drive stores the change as an update, which needs one, and the revision is what refuses an owner's replace built before the change.
  • The contract's moderation may keep no list, as with deletion, when a type keeps fields for its moderators.

The change is the ninth action of the transition:

#![allow(unused)]
fn main() {
ContractUserModerationAction::ChangeDocumentFields {
    document_type_name: String,
    document_id: Identifier,
    fields: BTreeMap<String, Value>,   // each field's new value, `Null` removing it
    reason: ContractModerationReason,
}
}

Basic structure refuses one that names no field or a system property (InvalidContractModerationDocumentFieldsError, 10905, unpaid). The transform (transform_document_fields_change_v0) checks, in order and each refusal paid: the document type exists (10406), every field is one it keeps for its moderators (DocumentFieldNotChangeableByModeratorsError, 41123), the signer moderates the contract (41101) and a seated team holds changeDocumentFields on the type (41201) and names a listed reason (41203), the document exists (40101) and has not expired (40140). It then builds the changed document: the stored one with each field set or removed, compared as a replace compares (Value::equal_underlying_data, so an integer sent at another width than the stored one is no change), $revision one higher (OverflowError, 10700, at u64::MAX), $moderatedAt the block's time and $moderatedBy the signer, everything else as stored, $updatedAt and its heights included. That document is judged as a replace judges one: validate_document_properties (the schema and propertyConstraints, with the countOf and sumOf totals read as they will be once it is stored, read_property_constraint_aggregates_for_moderator_change), distinctFrom and the shapes of encryptedFor properties, then, when a changed field is in a unique index, Drive::validate_moderated_document_uniqueness, the restore's check generalized to a changed document (only the changed fields are checked, and the document's own entries do not clash). Whoever owns the document is no protection: the fields are the moderators'. Nothing the document type prices is charged.

The action carries the contract and the changed document, and Drive stores it with DocumentOperationType::UpdateDocument, the replace's update, every index and aggregate of the type included. The document's storage flags name its owner, as a replace's do: the moderator pays for the bytes the change adds, and what the change frees (a smaller value, an index entry that moved) is refunded to the owner, who paid for it. A seated team's member does not have the change counted toward the team's action share: the changes of one document have no bound, and counting them would let a member farm it. A change that changes nothing is refused (10905) rather than written. $updatedAt does not move, so a change never opens a moderatorAbilities.deleteWithin window again: a moderator can not keep a post deletable by touching it.

The stamp is two system properties of document serialization format 3 (bits 9 and 10 of its time field flags): absent on a document no moderator has written, set only here and by the batch transformer, which stamps a moderator's create or replace of such a field (an action it did not judge is written unstamped), carried over by a replace that leaves the fields alone and by every transfer, purchase and price update, and put back by a restore with the rest of the bytes. A type that keeps fields for its moderators may index either, never in a unique index (apply_moderator_abilities, 10231; the core index check admits the two names from parser generation 3, ParserGeneration::admit_moderation_stamp_indexes). See System Properties.

The change is proved by the document itself: the prover builds the single-document query a replace is proved by, and verify_contract_document_change_execution checks that the document exists and holds each value the change set (and none of the removed ones), returning VerifiedDocuments. The stamp is not compared: a later moderator's write to another field moves it without undoing this change. Its other properties are the owner's and are not compared; the verifier reads the document under the contract, so the SDKs register the contract before broadcasting.

Storage

[64] DataContractDocuments
└── <contract id>
    ├── [0] the contract (or its history subtree)
    ├── [1] documents
    └── [2] other
        ├── [16]  document removals -> <document type name> -> <document id>
        │                            -> Item(owner id ‖ moderator id ‖ removed at ‖ document hash ‖ restored? [‖ restored by ‖ restored at] ‖ reason)   (with such a document type)
        ├── [24]  team actions -> M (active) | X (closed) -> <action id>                                  (with a deleteSettled type)
        │                            ├── I -> Item(tag ‖ proposer ‖ proposed at ‖ type ‖ document id ‖ last modified at ‖ revision ‖ reason)
        │                            └── S -> SumTree(<member id> -> SumItem(1))
        ├── [48]  moderation action counts -> <identity id> -> Item(count)        (elected contracts)
        ├── [64]  contract version item (every contract)
        ├── [128] banlist       -> <identity id> -> Item(reason)                 (when declared)
        ├── [192] suspensions   -> <identity id> -> Item(until ‖ reason)         (when declared)
        └── [224] warnings      -> <identity id> -> Item((warned at ‖ len ‖ reason)+)  (when declared)

until is a u64 of block time in milliseconds, big-endian. A reason is a tag byte (bit 0: a code follows, bit 1: documents follow, bit 2: a reason document follows), the code as a big-endian u16 when tagged, the 32-byte id of the reason document when tagged, then when tagged the documents it cites (their count in one byte, then each one's type name as a length byte and the name, and its 32-byte id), then the text as UTF-8 up to the end of the value, so an entry with an empty reason and no code costs one byte more than the bare entry would. A value without the tag byte is an entry written before entries carried a reason and reads as the empty reason; a tag without bit 1 is a reason from before documents could be cited, and one without bit 2 a reason naming no reason document. A warning list entry holds every warning the identity carries, oldest first, each warned at as a u64 of block time in milliseconds, big-endian, the length of its encoded reason as a big-endian u16 (the prefix that lets one value hold several reasons, each of which would otherwise run to the end), then the reason as above (types::encode_warnings). A warn rewrites the entry one warning longer; a clearWarnings deletes it, so a stored entry never holds fewer than one warning. A document removal is the document owner's id, the moderator's id, removed at as a u64 of block time in milliseconds, big-endian, the 32 bytes of the document hash, a tag byte (0: not restored, 1: restored) followed when restored by the restoring moderator's id and restored at as a u64, big-endian, then the reason the same way (types::encode_document_removal): 105 bytes before the reason, 145 once restored. The tag byte's bit 1 says kept fields follow the restoration, before the reason: their length as a big-endian u32, then the bytes encode_kept_fields wrote. Drive stores and proves them as they are, and reads the rest of the record without the document type; only the kept values need it. A record that keeps no field is written without them, as before fields could be kept. The records a write walks past are estimated from the document type, not from the record being written: every record of a type keeps the same paths, but a path the document held no value at is left out and values vary in length, so each kept path is estimated at its presence byte and its property's middle size, as a document of the type is, a time or height at its eight bytes and a core block height at its four (types::estimated_document_removal_kept_fields_size).

A removal record is replaced in place, as a suspension is, by the restore that marks it and by the deletion of a restored document, which writes a fresh record over the marked one; the deletion transform reads the record, billed, to know which of the two it writes, and a record that is not marked restored beside a live document is a state no transition produces. The replacement's flags follow GroveDB's flag merge like a suspension's: the restore adds forty bytes and passes the record, with the refund of its removal, to the restoring moderator, who pays for them. The fresh record of a second deletion loses the restoration but keeps the values the document holds then, so it may be shorter or longer: a longer one passes to the moderator deleting again, who pays for the bytes it adds; a shorter one refunds the bytes it frees to the restoring moderator, and passes to the moderator deleting again only within the epoch the record was paid in (or once the record spans epochs), keeping the restoring moderator's flags when shortened in a later epoch, as GroveDB merges the flags of a single-epoch item (the deletion's forfeiture takes its document operations alone, applied as a GroveDB batch of their own when the batch rewrites a record, so the record's holder is refunded whoever it is, the document's owner included, for the record's bytes only). A fee estimate prices a replacement as a fresh insert of what it adds to the record it replaces, never less than the bytes it adds, rather than GroveDB's average-case replace, which prices none, and the records a write walks past are estimated as restored, the larger shape, keeping what the type's records are estimated to keep.

The document removals tree exists exactly when the contract has a document type whose moderators' deletions keep records (moderatorAbilities.delete without deleteKeepsRecord: false): a contract without one keeps the other tree, and the shape, it would have had. One subtree per such document type is created with the type, by insert_contract generation 2 or by update_contract generation 2 for a type an update adds, and the tree above them with the first: whether it is there is read off the stored contract, since an existing type never changes the keyword, so the update needs no read. Nothing is created lazily by the first removal. The key sorts below 128, as a key added later should: a contract that keeps both lists, the one whose other tree then holds four keys, still has the banlist on top. With fewer keys the version item is on top, and the list one level down.

The team actions tree at 24 is shaped like a token group's actions: M holds the actions still gathering approvals and X those that ran, each an action id to a tree of the action's info (I) and the sum tree of its approvals (S, a member's id to a sum item of 1, so the sum is the number of approvals). It is created with the contract by insert_contract generation 2 when a document type sets moderatorAbilities.deleteSettled, with M and X under it, nothing lazily. No update adds such a type: it needs the elected declaration, fixed at creation, to give the team deleteDocuments on it, and the declaration can only name the types the contract was created with. An action's info is a tag byte (0: the deletion of a settled document), the proposer's id, the proposal's block time as a u64 big-endian, the document type name's length in one byte and the name, the document id, the document's last modification then and its revision (0 for a document that carries none; revisions start at 1), each a u64 big-endian, then the reason as a banlist entry holds it (encode_contract_team_action). Under M the info and each approval are flagged with the member that wrote them; the approval that closes the action moves them to X without flags, and deletes what is left under M. A fee estimate prices the move of the info by the size of the action as stored, which validation read. A sum item's storage is estimated by GroveDB at its serialized size and charged at its fixed size, a few bytes more per approval, as for a token group's approvals: the estimated fee still exceeds the one paid, the processing it over-estimates covering the difference.

The moderation action counts of an elected contract sit at 48: one item per member of the seated team who signed a counted action since the moderators pot was last settled, its identity id to its count as a big-endian u32 (see The seated team below). The tree is created with the contract, the only time elected moderation can be declared, so it is never made lazily. Each counted action reads its signer's count with one point read and writes it one higher: an insert of 36 bytes for the member's first action of a period, a replacement of the same size after. A settle reads the whole tree, at most one item per identity the team can hold (the leader, the elected members and the additions the target allows), and deletes every item. Per-member items keep the action, far more frequent than a settle, to a four-byte write; one packed item for the team would rewrite every member's count on every action. The items carry no storage flags: the member whose action writes one pays for it, the settle that deletes it refunds nobody, and a settle's fee result carries no refunds for anyone else. The key is below 64: created with two or three lists it leaves the banlist on top (with every list, where the version item and the suspension list alone had put the suspension list there), with or without the removal records tree; created with the banlist alone, or with the banlist and one other list beside removal records, it puts the version item on top and the banlist a level down. What sits there is read by the team's actions and by a settle, never by a document transition.

The contract's own subtree holds three keys whatever the contract keeps, so its Merk keeps 1, the documents, on top: every document proof and write goes through that key, and a fourth key beside it would have pushed it one level down (a Merk built from one sorted batch roots at the middle key). Everything else a contract keeps goes into 2, its other tree, which protocol version 14 introduces together with the version item. Inside, the keys are spread like the root tree's, so the tree stays balanced as it fills and the most read entry sits on top: the banlist at 128, read by every document transition on a moderated contract, the version item at 64, the suspension list at 192, the warning list at 224. A Merk built from one sorted batch roots at the middle key, the upper middle of an even count, so a key added later goes where it keeps 128 the median of the keys created together in the likely combinations: the warning list, which no document transition reads, sits above 192, which leaves the banlist on top for a contract keeping the banlist and a warning list, with or without the removal records tree at 16, and for one keeping all three lists with that tree; a contract keeping all three lists and no removal records has the suspension list on top and the banlist one level down.

The other tree is written by every contract insertion, and by the migration on the first block of protocol version 14 for the contracts stored before it. A contract update finds it there, so its fee estimate writes none; applied, the update reads key 2 once, billed, rather than fail inside a block: a tree is left alone (it may hold the lists), a missing one is written. The 4.2 betas kept the version item itself at key 2, before the other tree existed: an update of a contract that still holds that item puts the tree in its place and the item under it (add_contract_to_storage generation 1). Until such a contract is updated, the unproved getDataContractsLatestVersions reads its version from that item, and the proved form shows no version item for it.

The list trees are created by insert_contract generation 2 for a contract that declares them, and by nothing else: the lists are fixed at creation, so a contract update creates none and leaves the existing ones and their entries alone, and no tree is made lazily by the first ban. An entry's storage flags name the moderator that wrote it, so the storage refund of its deletion goes to that moderator whichever transition deletes it: the explicit unban or unsuspend, the ban over a suspension, or the document transition that sweeps a lapsed suspension. The sweep's processing fee is charged to the batch signer.

The writers, readers and provers live in packages/rs-drive/src/drive/contract/moderation/, versioned by DriveContractModerationMethodVersions. A suspend that replaces an entry is a batch_replace, because two operations on one key would fail the batch; so is a warn on an identity that already carries warnings, which rewrites the entry whole with the new warning last. The entry is then longer, so it passes, with the refund of its removal, to the moderator that warned last, who pays for the bytes the warning added; the same merge a longer suspension replacement goes through. The replacement brings its own reason, so the entry may change size: a longer replacement merges the flags as a document that changes hands does, the moderator that replaced it paying for the bytes it added and becoming the entry's owner, refunded when it is removed; a shorter or an equally long one stays the first moderator's, who is refunded the removed bytes at once and the rest on removal. A fee estimate prices a replacement as a fresh insert of the whole entry, because GroveDB's average-case replace assumes an item keeps its size and would price no storage for a longer reason; the entries a write walks past, and the one a delete removes, are estimated at a typical reason (128 bytes of text), not at the longest.

Reading and Proving

fetch_contract_moderation_status(contract, identity, lists) reads the identity's entry on each list named, the warning list included, and the verifier of its proof rebuilds the same merged path query from the same lists. The lists are the ones the contract's config declares; an undeclared list has no tree and cannot be queried, so a status query names the lists it wants and the node refuses one the contract does not keep. fetch_contract_moderation_entries pages one list in identity id order, bounded by the platform version's max_returned_elements (the default page size too, and the number the proof verifier assumes when a request names no limit), with the last identity as the cursor. A page shorter than its limit is the last one and carries no cursor.

The DAPI Queries

  • getContractModerationStatus(contract_id, identity_id, lists, prove): the identity's status on the lists named.
  • getContractModerationEntries(contract_id, list, start_after, limit, prove): one page of a list. An entry of the warning list carries every warning of the identity, and its reason is the latest warning's.
  • getContractTeamActions(contract_id, status, start_at_action_id, count, prove): one page of the actions a seated team votes on, active or closed, in action id order, bounded by max_returned_elements, on a contract that keeps them (a document type of it carries moderatorAbilities.deleteSettled: the node refuses any other). Each is the proposer, the proposal's time and what it does, today the deletion of a settled document with the document as proposed and the reason, and how many approvals it holds (approval_count, the proposer's included). The count is the sum of the action's approvals tree S, each approval a sum item of 1, so the page reads I and S of every action, two elements each, and its GroveDB limit is twice the page's. An active action's count is an upper bound: the approval of a member who left the team is dropped only when a later approval reads the team, and is counted until then. The exact figure is the action's signers (getContractTeamActionSigners) still on the team (the SDKs' ModerationTeam::contains), which a client reads only for an action whose count could meet its rule. Drive::verify_contract_team_actions rebuilds the path query from the same request.
  • getContractTeamActionSigners(contract_id, status, action_id, prove): who approved one action, active or closed as the request says, the proposer among them unless it left the team and its approval was dropped; none when there is no such action with that status. Drive::verify_contract_team_action_signers rebuilds the path query.
  • getContractDocumentRemovals(contract_id, document_type_name, document_ids | page, prove): the records of the documents moderators deleted, within one document type that carries moderatorAbilities.delete (no other keeps records, so the node refuses any other). By document ids, up to max_returned_elements of them and none twice: an id with no record is left out of the response, and proved absent by a proof. Or one page in document id order, with the last document id as the cursor. Each record carries the document hash, once restored who restored it and when, and the values it keeps (kept_fields, the bytes as the record stores them, which a client reads under the contract's document type as it reads documents; the SDKs do so, keptFields in wasm-sdk). Drive::verify_contract_document_removals rebuilds the path query from the same request.

A status query answers for the lists it names and no others: Drive::verify_contract_moderation_status and the SDK result both return ContractModerationListStatuses, one ContractModerationListStatus per list queried, so a list that was not read is absent rather than reported as empty (banned() is None unless the banlist was queried). ContractModerationStatusQuery::for_contract names every list the contract keeps; the wasm-sdk does the same, fetching the contract, when the query names no list. Both have Fetch and FetchUnproved impls in the Rust SDK (platform::contract_moderation), wasm-sdk functions and contracts.moderationStatus / contracts.moderationEntries on the JavaScript SDK. The proof of a moderation transition's execution covers the lists the moderation touched and is classified as affected state: an earlier or later moderation leaving the same entries verifies just the same. A ban does two things, adds the ban and removes a suspension, so its proof covers every barring list the contract keeps (the banlist entry present, the suspension absent), which the prover and the verifier both read from the contract's config (so the SDKs fetch and cache the contract before broadcasting a ban, as they do for the contracts a document batch touches); an unban, a suspend, an unsuspend, a warn and a clearWarnings prove the one entry they edit. A warn's proof shows the entry with the transition's reason as its last warning (the block time is the block's, which the verifier does not know); a clearing's shows the entry absent. The result, VerifiedContractModerationListStatuses, holds one ContractModerationListStatus per list proved, never a full status: a list that was not proved is left unknown rather than reported as empty. An identity whose unsuspend was just proved may be banned; the status query answers that.

Fee Pots and the Claim

A moderation team can be paid. A document type may charge a fixed fee in credits for an action on its documents (the actionFees keyword, see Document action fees), split in two parts. The owner parts collect in the contract's owner pot, the moderators parts in its moderators pot.

[40] PreFundedSpecializedBalances (sum tree)
├── [64]  owner fee pots      (sum tree) -> <contract id> -> SumItem(credits)
├── [128] voting balances
└── [192] moderators fee pots (sum tree) -> <contract id> -> SumItem(credits)

[64] DataContractDocuments -> <contract id> -> [2] other
    ├── [32] last claim of the owner pot       Item(epoch u16 BE | time u64 BE | claimant id)   (after a claim)
    └── [96] last claim of the moderators pot  Item(epoch u16 BE | time u64 BE | claimant id)   (after a claim)

The pots are not under the contract. The per-block total credits check (calculate_total_credits_balance) sums a fixed set of root sum trees, and DataContractDocuments is a normal tree: credits parked under a contract would leave that sum and fail every block with CorruptedCreditsNotBalanced. PreFundedSpecializedBalances is one of the summed trees, so the pots live there, in two sum trees beside the voting balances, created at genesis (state structure 4) and by the upgrade to protocol version 14 through the same helper, one after the other, so that both node populations build the same Merk. A pot is created by the first fee it receives, and so is its tree on a chain that reached protocol version 14 on a build from before the pots: that first fee checks, with a billed read, that the tree is there. The estimation of a voting balance write moves to generation 1 with them, because the prefunded balances layer now holds three trees instead of one. The two last claims are plain items of the contract's other tree, below 128 so the banlist stays on top, written by the first claim and replaced by every later one. A last claim (ContractFeePotLastClaim) is 42 bytes: the epoch of the claim, which the next claim is judged against, the time of its block in milliseconds, and the id of the identity that signed it. The owner pot's claimant is always the owner; the moderators pot's is whichever member of the team claimed for all of them, so the team can see who paid them and when. Every last claim has the same size, so a replacement never changes the size of the item, and the item carries no storage flags: it is never removed, and no claim adds bytes for anyone to own.

The team that shares the moderators pot is the set of identities the contract appoints, the owner among them only when appointed, and the owner alone when nobody is appointed (ContractModerators::team). It is about earnings, not authority: an owner who is not appointed still may moderate. For an elected contract it is the interim's team until a charter is seated, and from then on the seated team, which shares the pot by its proposal's reward split (see The seated team below). ContractFeePot::recipients names who a payout of a pot goes to: the contract owner for the owner pot, the team for the moderators pot, nobody for the moderators pot of a contract that declares no moderation.

ContractFeeClaim (state transition type 25) names a contract and a pot and pays the pot out. It is signed with a CRITICAL authentication key under the signer's contract nonce, and the claimant pays its gas like any other transition.

StageCheckError
Transform (state, paid)the contract existsDataContractNotPresentError (10400)
the signer is a recipient of the pot: the owner for the owner pot, a member of the team for the moderators pot (the leader or an active member once a charter is seated)41113
the pot was not paid out in this epoch yet41111
every recipient gets at least a credit41112

The owner pot goes to the owner whole. The moderators pot is split equally between the team, and what the split leaves over, less than a credit per member, stays in the pot for the next claim, so no member is favoured by the order of the identity ids. A seated team's pot is split by its proposal's reward split instead, rounded down the same way. Each pot is paid out at most once per epoch and the two are independent: the owner's claim does not use up the team's, nor the reverse. A refused claim is paid for by a nonce bump and leaves the pot and its last claim alone. As for moderation, state validation is the transform, so the mempool refuses with the same codes as a block.

The team is read when the claim executes. An owner who changes the appointed set by a contract update and then claims pays the new set: that follows from the owner controlling the contract's config, and is not prevented. The claim credits every recipient's balance, which is why a named moderator must exist (41110): crediting a balance that is not there is an internal error.

The proof of a claim's execution shows the pot with its last claim and the balance of every recipient, which the prover and the verifier both read from the contract. When the claimant is not among them, it shows the claimant's balance alone: the claimant is then on a seated elected team, which the charter contract names and the contract does not, so neither side could list the other payees (ContractFeePot::claim_proof_identities). An interim team's claim, before a charter is seated, is proved with every recipient's balance as before. VerifiedContractFeeClaim carries the contract id, the pot, that last claim (epoch, block time, claimant), the credits left in the pot and the balances. A pot that was never claimed proves no claim; a later claim of the same pot verifies just the same, so the result is classified as affected state.

Reading the Pots

  • getContractFeePots(contract_id, prove): both pots of the contract, each with its credits and its last claim: the epoch and the block time it was paid out in, and the identity that claimed.
  • getContractModerationActionCounts(contract_id, prove): how many counted actions (bans, suspensions, warnings and document deletions) each member of an elected contract's seated team signed since the moderators pot was last settled, in identity id order; a member that did not act since has no count. The node refuses a contract that is not elected, which has no counts tree, and an elected one stored before protocol version 14 counted actions (a development network's), which has none either (Drive::contract_keeps_moderation_action_counts), with or without a proof. The path query (Drive::contract_moderation_action_counts_query) reads the whole tree at 48 without a limit, which the prover and Drive::verify_contract_moderation_action_counts build alike, as the approvals of a team action are read: the tree holds at most the team, since every settle deletes the counts and a change of the team settles first, and a limit taken from today's SystemLimits could cut short a team registered under larger ones with a proof that still verifies.

The query always reads both pots, so its proof is one fixed path query (Drive::contract_fee_pots_query) that the prover and Drive::verify_contract_fee_pots build alike, with nothing in the request to get wrong. A pot nothing was paid into yet has no element and reads as zero credits, and a pot never paid out has no last claim, which is not a claim in epoch 0: a pot can have been paid out in epoch 0, so the last claim is a message of its own on the wire, unset when there is none, and the JavaScript fields (lastClaimEpoch, lastClaimTimeMs, lastClaimantId) are absent together. The proof says nothing about the contract itself, only about what is stored under its id, so the node refuses the query for a contract it does not hold before it proves anything, and a client that needs to know the contract exists fetches it.

The counts are what the action share of a claim splits by, so anyone can see who has been acting and a member sees its part of that share. A full preview of a claim also needs the team's submittedCharter: its rewardSplit, whose leader and equal shares are paid first, and its moderatorsShare, which sets what the pot collects. The counts are a snapshot of the period since the last settle, which resets them, so right after a claim they say little. The Rust SDK has Fetch and FetchUnproved impls for ContractModerationActionCounts (platform::contract_moderation, queried by the contract id or a ContractModerationActionCountsQuery), the wasm-sdk getContractModerationActionCounts and getContractModerationActionCountsWithProofInfo, and the evo-sdk contracts.moderationActionCounts and contracts.moderationActionCountsWithProof.

A recipient reads the pots to decide whether a claim is worth its gas: the credits are what it would pay, and a last claim epoch equal to the current epoch means the claim would be refused (41111). A member of the team also reads there which member last claimed for the team, and when. The Rust SDK has Fetch and FetchUnproved impls for ContractFeePots (platform::contract_fee_pots, queried by the contract id), the wasm-sdk getContractFeePots and contractClaimFees, and the JavaScript SDK contracts.feePots and contracts.claimFees.

The claim's proof is verified against the contract, which names who the pot pays, and the team can change by a contract update. So every client fetches the contract again before a claim instead of trusting a cached copy: ClaimContractFees in the Rust SDK, contractClaimFees in the wasm-sdk, and the wasm-sdk's generic broadcastAndWait for a ContractFeeClaim built by hand, which falls back to the cached copy when that fetch fails, because the transition is already broadcast by then.

Elected Moderation

A contract may hand the choice of its moderators to the network instead of keeping it: it declares that its moderators are a team elected by masternodes and evonodes. Teams apply with a charter in the moderation charters system contract (see docs/protocol/moderation-charters.md), masternodes elect one in a contest for the contract's seat, and the seated team moderates with the contract's declaration, charging at most what the contract declares. What the contract itself holds is the declaration, frozen at its creation, and the interim: how the contract is moderated until its first team is seated.

#![allow(unused)]
fn main() {
pub struct ElectedModerators {
    pub join_window: u32,                 // seconds; at most 4 weeks, at least 1 day on mainnet (0 elsewhere), 1 week by default
    pub vote_window: u32,                 // the same
    pub challenge_cool_down: Option<u32>, // Some = seat contestable, seconds, 2 weeks to 3 years; None = never contested again
    pub election_delay: Option<u32>,      // seconds after creation before the first charter; unbounded, none = at once
    pub max_added_moderators: u16,        // members the leader may add after the election; 0 to 15, 0 by default
    pub moderated_document_types: BTreeMap<DocumentName, BTreeSet<ModerationAbility>>, // per type: DeleteDocuments, Ban, Suspend, Warn, ChangeDocumentFields
    pub interim: InterimModerators,       // ContractOwner, AppointedModerators(set), NotYetUsable, NoModeration
    pub owner_protected: bool,            // false by default
}
}

The declaration lives in packages/rs-dpp/src/data_contract/config/moderation/elected.rs. On the wire it is the third $type of the moderators, flat: {"$type": "elected", "seatContestable": true, "challengeCoolDown": 1209600, "moderatedDocumentTypes": {"post": ["ban", "deleteDocuments"]}, "interim": {"$type": "notYetUsable"}}, or "seatContestable": false without a challengeCoolDown, with joinWindow, voteWindow, electionDelay, maxAddedModerators and ownerProtected optional. Its parts:

  • The election parameters are fixed once set (SystemLimits: max_contract_moderation_election_window_seconds caps both windows at four weeks, and on mainnet min_mainnet_contract_moderation_election_window_seconds keeps them at least a day; every other network takes a window of 0, so a test election can be run through in a block or two). The join window is how long applicants may join an election once the first one applied, the vote window how long masternodes then vote. The election delay is the one parameter the contract sets freely: how many seconds after its creation the first charter may be filed against it, the notice the contract gives before its first election can be called. It is optional and unbounded; left out, the election may be called at once. Because the declaration is made at the contract's creation and never changes, the creation is the declaration's own time. The charter contract's targetContractId reads it through the moderation: "electionOpen" requirement below. maxAddedModerators says how many members the leader of a seated team may add after the election, each one an identity that asked to join the team's proposal: additions ever filed, so a removal or a resignation frees no slot. It is 0 when left out, a team then being exactly what was elected, and at most SystemLimits::max_contract_moderation_added_moderators (15).
  • The seat is contestable or not, and the contract says which: seatContestable is required, with no default. A default of false would make every team permanent, leaving a contract nothing to do about a leader who lost its keys or went rogue, since the leader can not change and a challenge is the only remedy; a default of true would opt every contract into challenges without it asking. A contestable seat declares its challenge cool-down, how long a seated team is safe from a challenge after a seat change (min_contract_moderation_challenge_cool_down_seconds to max_contract_moderation_challenge_cool_down_seconds, two weeks to three years), and a seat that can not be contested declares none, since a cool-down means nothing there: a declaration missing the key, or seatContestable: true without the cool-down, or false with one, does not parse. In Rust the two are one field, challenge_cool_down: Option<u32> (ElectedModerators::seat_contestable is is_some), so a declaration can not disagree with itself however it is encoded. Nothing reads the seat yet: challenges come after protocol version 14, a challenge then being a new contest on the same byTargetContract index of the charter contract, allowed only when the target declares its seat contestable. The key is there now because the declaration is frozen at the contract's creation. Until challenges ship a seat is never contested again, whatever the key says.
  • The moderated set is the document types the team moderates, each with the abilities the seated team holds on it: non-empty, each type a document type of the contract, each ability set non-empty and backed by the contract (ban needs the banlist, suspend the suspension list, warn the warning list, deleteDocuments the type itself setting moderatorAbilities.delete, so deletions reach only such types, within their window, and changeDocumentFields the type listing moderatorAbilities.changeFields; and every type listing changeFields must be moderated with changeDocumentFields, since once a team is seated only it writes those fields). The charter of a team will say how those types are moderated, never which. The lists stay contract-wide: an ability on a type is what a team may do over the documents of that type. The set also bounds the interim block. A charter does not price the moderators part of an action: a type's own actionFees.moderators amount is the most a team may charge, a charter charges a share of it (the charter contract's business, not the declaration's), and the owner part stays what the type declares, immutable as before.
  • The interim says who moderates until a team is seated. ContractOwner and AppointedModerators(set) are the merged kinds, with their authority, their limit and their existence check (41110 at create): they moderate, they are protected, and they are the team that claims the moderators pot, all of it until a charter is seated and none of it after (see The seated team below). NotYetUsable names nobody: nobody moderates, nobody claims the pot (it accumulates for the team to come, ContractFeeClaimNotAllowedError for everyone), and the moderated document types can not be used. A contract that never attracts a team keeps those types unusable for good; the other types work as on an unmoderated contract. NoModeration names nobody too, with the moderated types usable meanwhile: nobody moderates and nobody claims the pot, and every type works as on an unmoderated contract until a team is seated.
  • The owner flag says whether the contract owner is protected from the team once one is seated, as the owner and the moderators of the merged kinds are (41102 on a ban, a suspension or a deletion of its documents). Not protected by default. During the interim the owner is protected whenever it moderates, flag or not: ContractModerationConfig::protects is what the moderation transition checks, and it is may_moderate or the flag for the owner.

Validation. validate_moderation_config v0 checks the declaration with the rest of the moderation config, from the raw document schemas of the create or update transition, and refuses with InvalidContractModerationConfigError (10900, unpaid): a window, or the cool-down of a contestable seat, outside its bounds, an empty moderated set, a moderated type the contract does not have, an empty ability set or an ability the type can not back, and an empty or oversized interim set. The create then checks that every interim moderator exists (41110), as it does for an appointed set.

The update. validate_config_update 2 refuses, with DataContractConfigUpdateError (40002), every change to the declaration, the interim included, and entering or leaving elected moderation. A contract that declares elected moderation is elected for good, and one that did not can not become so; the merged kinds still swap their moderators freely. An update may add document types; the declaration keeps naming the ones it named.

The interim block. The batch transformer's contract_moderation_gate v0 runs it before the lists: on an elected contract whose interim is NotYetUsable, every document transition of a moderated document type, deletions included (nothing of those types was ever written), is refused, paid, with ContractModeratedDocumentTypeNotYetUsableError (41200) and its contract nonce bump, in a block and in the mempool, until a charter is seated on the contract. Whether one is, is read (billed) only when a transition of the batch is on a type the block covers. The lists are read only for the transitions on the other types, and not at all when nothing is left. The interim moderators of the other two kinds moderate through the same transition, the same gate and the same claim as the merged kinds; a moderation transition against a NotYetUsable contract fails as by a non-moderator (41101).

The seated team. Seating writes nothing. A team applies with an electedCharter of the moderation charters contract, a create on its contested unique index byTargetContract, keyed by the target contract; awarding that contest writes the winner's document to the charter contract's storage, the only electedCharter ever written there for the target (contenders live in the contest, and in protocol version 14 a seat is never replaced, whether or not the declaration says it is contestable: another charter for a seated target is refused, paid, with DuplicateUniqueIndexError, 40105). So the charter seated on a contract is the one byTargetContract finds, and every moderation path reads it from there (execution/validation/state_transition/common/seated_moderation_charter in drive-abci): there is no block-end seating hook and no copy under the moderated contract. Its team is the charter's owner, the leader, plus the active members: its members less the memberId of every removedModerator for it, plus the memberId of every addedModerator for it (ElectedCharter::active_members), counting the documents that exist now, since the leader takes either back by deleting it. A resignationRequest changes nothing by itself; the leader acts on it by deleting the member's addition or removing an elected member.

  • Who moderates. Once a charter is seated, only its team moderates: the leader and the active members, each alone. The interim moderators, the owner among them, can no longer act (41101), whatever the interim was. Deciding whether the signer is on the team reads no list of it: the leader costs nothing beyond the charter lookup, an elected member one point read of its removal, anyone else one point read of its addition (both types are unique on the charter and the member). What the interim did stands: its bans, suspensions, warnings and removals, which the team may lift.
  • With what. The team holds the abilities the declaration gives it and no others. A deletion or a restore needs deleteDocuments on the document type, and a field change changeDocumentFields; a ban, a suspension or a warning, or lifting one, needs the ability on some moderated type, since the lists are contract-wide. Anything else is refused, paid, with ContractModerationAbilityNotGrantedError (41201).
  • Who is protected. The leader and the active members can be neither put on a list nor have their documents deleted (41102), and the owner too when the declaration sets ownerProtected. The interim moderators lose the protection they had.
  • How many join later. The leader adds members from the proposal's join requests, at most the target's maxAddedModerators at a time, counting the charter's additions that exist now: the leader takes an added member off by deleting its addition, which frees the slot. An elected member is taken off with a removedModerator, which may only name one of the charter's members and puts the member back when deleted. The schema cannot count documents, so the batch's state validation refuses the addition past the cap, paid, with ModerationCharterAddedModeratorLimitReachedError (41202), after reading the charter, its target and at most the cap's number of additions, all billed; additions an earlier create of the same batch was accepted for count too. Like a unique index conflict it is judged in the block, not in the mempool, which runs no state validation for a batch.
  • On what grounds. Every ban, suspension, warning, document deletion and field change of the team names, in its reason's reasonDocumentId, a reason document its proposal lists: the grounds it asked to be elected on. Any other is refused, paid, in a block and in the mempool (ModerationReasonNotListedError, 41203), a reason naming none included; a proposal listing no reason is a team that can take no such action. The proposal is read, billed, only when the reason names a document. Lifting a ban, a suspension or warnings and restoring a document carry no reason and are not checked, and the interim is not bound before the seating.
  • What it charges. An action on a moderated type may agree to the charter's moderatorsShare of the declared moderators part instead of the whole of it, and is then charged that (see Document action fees). An action agreeing to the declared amount reads no charter.
  • The pot. The interim team claims the moderators pot only until a charter is seated; its claim is refused after (41113), so the pot carries over to the seated team, unsettled. From then on the leader or an active member claims it for the team, at most once per epoch, and it is paid out by the proposal's rewardSplit: the leader share to the leader; the equal share in equal parts to the other active members, or to the leader when it has none; and the action share between the whole team, the leader included, in proportion to the bans, suspensions, warnings and document deletions each one signed since the last settle (not field changes, whose number on one document has no bound) (the counts of the other tree's key 48), or equally when nobody acted. Lifting and restoring do not count. Every share and every part rounds down to the credit; the few credits left stay in the pot for the next settle. A claim that would pay nobody a credit is refused (41112). The settle reads the team (the charter's removals and additions), the proposal and the counts, all billed, and deletes the counts.
  • Settled before every change. An addedModerator or removedModerator created or deleted pays the pot out to the team as it was first, the same way, and resets the counts: a removed member is paid for what it did, a new one shares nothing earned before it came, one coming back nothing earned while it was away. The settle ignores the once-per-epoch limit and is not a claim: it writes no last claim, and the team may still claim in the same epoch. It is an effect, never a refusal, judged by the batch's state validation once the change passed (the addition its cap too), which the mempool does not run; at most one per target contract per batch.

Referencing an elected contract. A document type that must point at a contract of this kind says so in its reference: "refersTo": { "type": "contract", "contractRequirements": { "moderation": "elected" } }. contractRequirements holds what the referenced contract must declare beyond existing, each key an aspect of the contract with a closed set of values or a bound: moderation: "elected", or moderation: "electionOpen", which also requires the contract's own election delay to have passed since its creation, or the contract to declare none (the delay between a contract's creation and the first charter against it, so a team cannot be seated before anyone has seen the contract, set by each contract for itself). Both have a user in the charter contract: a charter proposal only needs the target to be elected, so teams can form during the notice, and the charter that opens the contest needs its election electionOpen; minimumAgeSeconds, a number of seconds the reference fixes, which requires the contract's recorded creation time to be at least that far before the block time of the write; minimumSecondsSinceUpdate, the same of the later of the contract's creation and last update times (any update restarts the clock; an elected declaration can not be added by an update, so this one is for other uses than the charter); owner, "self" requiring the referenced contract to be owned by the writer of the referring document (its $ownerId, a write gate like the $ownerId property agreement of a document reference) and "other" by anyone else (so a charter may forbid an owner from chartering its own team); readonly: true, requiring the referenced contract's config to be read-only, one that can never be updated again (which makes minimumSecondsSinceUpdate moot for the same target); keepsHistory: true, requiring its config to keep history (only true is declarable for either flag); and ownerProtected, requiring the contract's elected moderation declaration to protect the owner from the team (true) or to leave it unprotected (false), which implies elected moderation without the schema having to say so, a contract without an elected declaration meeting neither value. A contract created before contracts recorded their creation time never meets a duration, its own election delay included. Consensus checks them when the referring document is written, against the contract it has already fetched for the existence check and the write itself (its owner and block time), so they cost no further read; a contract that exists but does not meet a requirement refuses the write, paid, with ReferencedContractRequirementNotMetError (40135) naming the requirement, where a contract that does not exist is still 40120. A replace re-checks them when it changes the reference. owner is the one requirement judged against the writer, and a transfer or a purchase changes the writer without any write, so on a document type whose documents can be transferred or traded a reference carrying owner is re-checked on every replace, touched or not, as the $ownerId writer gate is: after a transfer the new owner has to repoint it at a contract that meets the requirement for them, or clear it where it is optional. The reference is then checked whole, its other requirements included. Because the new owner must be able to repoint it, registration refuses such a reference held by an immutable property on a type whose documents can be transferred or traded (the property itself, inside an immutable object, or the elements of a typed array), with InvalidContractStructure. On a type whose documents stay with their owner the writer never changes and neither does a contract's owner, so nothing is re-checked. The other requirements are facts about the referenced contract, not the writer, and never bring a reference back on their own. A changed contractRequirements is an incompatible schema change on update, like the rest of a refersTo. The charter system contract's targetContractId is the first user.

Referencing an identity key with requirements. The same shape serves the key references the charter contract needs: "refersTo": { "type": "identityPublicKey", "keyIdProperty": "recipientKeyId", "keyRequirements": { "purpose": "decryption", "boundTo": "submittedCharter" } }. keyRequirements holds what the referenced key must be beyond existing and not being disabled, each key an aspect of the key: purpose, the key's purpose by its wire name (authentication, encryption, decryption, transfer, voting or owner; never system), and boundTo, the name of a document type of the declaring contract, which requires the key's contract bounds to be exactly the declaring contract and that document type; a whole-contract bound or a contract group bound never meets it, even where the group holds the type, since the check reads nothing beyond the key. Registration (create_document_types_from_document_schemas 1, a post-pass edited in place since it is inert before protocol version 14, under full validation like the meta-schema) checks that boundTo names a document type the contract has, so the write-time check never needs a second contract fetch, and that a key meeting the pair can exist at all: only authentication, encryption and decryption keys carry a document type bound, and Drive registers an encryption or decryption key bound to a document type only when that type declares requiresIdentityEncryptionBoundedKey or requiresIdentityDecryptionBoundedKey, so a boundTo paired with transfer, voting or owner, or with an encryption purpose on a type without the matching keyword, is refused as a requirement no key could ever meet. Consensus checks the requirements when the referring document is written, against the key it has already fetched for the existence check, so they cost no further read; a key that exists and is enabled but does not meet one refuses the write, paid, with ReferencedIdentityKeyRequirementNotMetError (40136) naming the document type, the property, the requirement and what the key has, where a missing key is still 40123 and a disabled one 40124. A replace that repoints the reference at another key, through either the identity id or the key id, re-checks them. A changed keyRequirements is an incompatible schema change on update, like the rest of a refersTo. New requirements (a security level, say) are new keys of the same object, never a new reference type. The charter contract's joinRequest.recipientId (a decryption key bound to submittedCharter) is the first user.

What comes next. After protocol version 14, challenges of a contestable seat, amendments and threshold actions beyond the deletion of settled documents. Issue #4865 holds the design.

Versioning Touchpoints

All in place for protocol version 14: CONTRACT_VERSIONS_V6 makes config V2 the config of every new contract (max_version and default_current_version 2) and validate_config_update 2; STATE_TRANSITION_SERIALIZATION_VERSIONS_V3 and DRIVE_ABCI_VALIDATION_VERSIONS_V10 carry the transition's slots and batch_state_transition.contract_moderation_gate, and the contract update's basic structure moves to 2 to validate the declaration; DRIVE_CONTRACT_METHOD_VERSIONS_V4 bumps insert_contract to 2 and adds the moderation table (its update_contract 2 belongs to token distribution and does nothing for moderation); DRIVE_STATE_TRANSITION_METHOD_VERSIONS_V4 adds the converter slot and bumps documents_batch_transition to 1 for the sweep; DRIVE_VERIFY_METHOD_VERSIONS and DRIVE_ABCI_QUERY_VERSIONS gain their moderation tables; SYSTEM_LIMITS_V4 gains max_contract_moderators, max_contract_suspension_until, max_contract_moderation_reason_length and max_contract_warnings_per_identity. The warning list adds two slots to DriveContractModerationMethodVersions (add_contract_warning, remove_contract_warnings) and nothing else: the list, the two actions and the two errors join a feature no release contains.

The document deletion adds, all for protocol version 14 as well: the moderatorAbilities.delete keyword in meta-schema v3 (CONTRACT_VERSIONS_V6 already selects it); five slots in DriveContractModerationMethodVersions and one in the verify and query tables; and batch_operations.apply_drive_operations = 1 in DRIVE_VERSION_V9, the generation that forfeits the refund. The transition's own tables do not move: the action joins a transition no release contains.

The field change moves no table either: the action, the transform, the converter, the prover's and the verifier's arms sit in generations only protocol version 14 selects, and the batch transformer's gate on an owner's create or replace (v0, edited in place) reads nothing for a type that keeps no such field, which no earlier version can register. The uniqueness check a restore and a field change share is validate_moderated_document_uniqueness in DriveDocumentIndexUniquenessMethodVersions, still 0, and takes the changed fields.

The deletion of settled documents adds, for protocol version 14 as well: the moderatorAbilities.deleteSettled key in meta-schema v3, the DeleteSettledDocument and ApproveTeamAction actions and the VerifiedContractTeamActionSignature proof result, all appended; eight slots in DriveContractModerationMethodVersions (add_contract_team_action_signature, fetch_contract_team_action, fetch_contract_team_actions, fetch_contract_team_action_signers, prove_contract_team_actions, prove_contract_team_action_signers, insert_contract_team_action_trees, add_estimation_costs_for_contract_team_action), three in the verify table (verify_contract_team_actions, verify_contract_team_action_signers, verify_contract_team_action_signature) and two in the query table (contract_team_actions, contract_team_action_signers); and max_moderation_charter_elected_members (15) in SYSTEM_LIMITS_V4, backfilled into the earlier tables, which nothing reads there. The transforms and the converter sit in generations only protocol version 14 selects. The prover's and the verifier's arms are in the shipped prove_state_transition v0 and verify_state_transition_was_executed_with_proof v0, which protocol versions 1 to 13 select too, edited in place: only a contract user moderation takes them, a transition inactive before protocol version 14 (should_not_be_active_before_protocol_version_14), so no earlier proof changes. The storage refund forfeiture of apply_drive_operations generation 1, which only protocol version 14 selects, now takes the document operations alone, applied as a GroveDB batch of their own when the batch also frees moderation storage someone is owed.

Elected moderation moves no table: the declaration is a variant of the same config V2, validate_moderation_config v0 and validate_config_update 2 take it on in place while protocol version 14 is unreleased, contract_moderation_gate v0 runs the interim block, and SYSTEM_LIMITS_V4 gains the four bounds of the windows and the cool-down. The seated team moves none either: the moderation transition's state v0, the claim's, the gate v0 and the batch transformer's state v2 read the charter in place, all generations no release selects; the cap on additions is a hook in the batch's shipped validate_state v0 that only a create of the charter contract reaches, a contract absent from state before protocol version 14. The pot, the counts and the reasons add four slots to DriveContractModerationMethodVersions (set_contract_moderation_action_count, fetch_contract_moderation_action_counts, remove_contract_moderation_action_counts and their estimation), 0 at every version. The forced settle is a second hook in the same validate_state v0, reached only by a create or delete of the charter contract's team changes; the claim's action, the moderation transition's action and the batch action carry what they settled into converters no release selects (contract_fee_claim_transition 0, contract_user_moderation_transition 0, documents_batch_transition 1); and the claim's arm of the shipped prove_state_transition v0 and verify_state_transition_was_executed_with_proof v0 proves the claimant alone for an elected contract, a transition no earlier version admits.

What Is Not There Yet

Deleting indexOnly documents (the action would have to carry the owner and the values), deleting every document of an identity at once, action fees on token transitions, group-based moderators (AuthorizedActionTakers::Group through group actions), keys bound to the contract allowed to sign its moderation, ban codes declared by the contract (the reason's code is where they will go), the moderator's id on a ban or a suspension (a warning carries its block time but not who issued it), retracting one warning rather than all, a warning that expires by the clock, a contract-declared strike count that turns warnings into a suspension, a query of the moderation action counts (a client reads them with a raw GroveDB proof today), challenges and amendments of a seated charter, and the Swift and Kotlin SDKs. The refusal a barred identity receives (41107, 41108, 41114) does not repeat the reason: the status query does.

Tests

  • packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/moderator_abilities_tests.rs: the object's shape on both paths, delete's rules and the window's (it needs delete and a clock, $updatedAt or for documents that never change $createdAt), deleteSettled's (its shape and defaults, the window and the elected declaration it needs, its bounds, the $createdAt approversPredateDocument needs), and every rule changeFields holds its properties to, the revision it keeps included; validate_update/v1: the abilities frozen across updates, the settled-deletion rule among them.
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests/moderator_fields.rs: a moderator's change stored, proved and paid for, a null removing a field, a report resolved after its post was deleted, every refusal of the transform, the create and replace gate for owners who do not moderate, a replace built before the change refused on its revision, and the unique indexes over the fields; seated_team.rs: a seated team writing the fields the declaration gives it changeDocumentFields on, and its members, not the off-team owner, writing them in their own documents; retraction.rs: a banned and a suspended author retracting a document it can not delete while its edits, a retraction breaking the type's own rule and taking a retraction back are refused, the same judgement in the mempool, and a type without retractedWhen refusing every replace of a barred author.
  • packages/rs-drive/src/drive/contract/moderation/document_removal_tests.rs: the trees created with the contract and with a document type an update adds, records written, replaced, read by ids and by page with proofs that verify to the same, the bounds of a read, estimate against applied cost, and a moderator's deletion refunding nobody where the author's own refunds the author.
  • packages/rs-drive/src/drive/contract/moderation/team_action_tests.rs: the team action trees created with an elected contract (the creation estimated first) and none without the rule, proposals and approvals written, an action closed with its approvals moved and nothing left active, read and paged with proofs that verify to the same, the proof of an approval before and after its action closed, the bounds of a page, the estimate against applied cost up to a team's worth of approvals and the longest reason, the approvals of members who left dropped and refunded, and the batch that deletes forfeiting the post's refund while refunding the proposer and the approvers the approvals it moves, the approver who wrote the post only its approval; types.rs: the action's encoding and every malformed one refused.
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests/seated_team/settled.rs: a settled story deleted once the leader and two members approve a member's proposal and not while three members without the leader have, the removal record with the proposal's reason, the counts of every approver and none before, the proof of the proposal before and after the action ran, an approval of an action that ran refused, the leader alone deleting a memo by its proposal, a proposal refused within the window, before a team is seated, on a type without the rule, off the team and for an unlisted reason, an approval of nothing proposed and off the team refused, the approvals of members who left dropped and refunded a month later and a member who came back approving again (on a type whose rule counts members whenever added), an approval refused once a moderator changed the document's fields or its author replaced it, the proposer refunded the action the deleting approval moves, a removed member's seat counted toward the whole team, a member added after a story (or in its block) refused as proposer and approver while one added before counts, and a member taken off and added again after the story dropped at the next team read (one point read each, and in one read of the team); query/contract_moderation_queries/contract_team_actions and contract_team_action_signers: the queries, their proofs, and the requests they refuse.
  • packages/rs-drive-abci/src/query/contract_moderation_queries/contract_document_removals: the query by ids and by page, its proof read back by the verifier, and every request it refuses.
  • packages/rs-dpp/src/data_contract/config/moderation/mod.rs and config/methods/validate_update/v2: the declaration's rules and the update rules, the elected declaration's among them (every bound, the seat and its cool-down, the moderated set, the envelope, the maximums, the interim set, the wire shape, and an update refused for each field and for entering or leaving); moderation/elected.rs: what each interim kind allows.
  • packages/rs-drive/src/drive/contract/moderation/tests.rs: tree creation on insert, the trees and their entries surviving a contract update, every writer with estimation, status and page proofs, paging, the refund going to the first moderator after another one replaces its suspension, a status proof over one list saying nothing about the other, the warning list tree created only when declared, warnings accumulating under the moderator that warned last and cleared with a refund to it, a warn never estimated below its cost up to the fullest entry, and the banlist on top of the other tree with every combination of lists.
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/transformer/v0/contract_moderation_gate/mod.rs: the gate is silent before protocol version 14 and for an unmoderated contract, refuses each barred operation of one batch on its own while keeping the deletions, passes a barred signer's replaces on a type declaring retractedWhen with the bar, and blocks the moderated types of an elected contract in its interim without reading the lists for them.
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests/seated_team/pot.rs: a seated team's claim split by its reward split with every part rounded down and the remainder left in the pot, the equal split of the action share when nobody acted, the counts per signer (counted actions only, not the interim's) reset by a claim and by a change of the team, the settle before an addition, before a removal and before either is undone (in an epoch already claimed, and leaving the epoch's claim to the team), and the claim's proof with the claimant's balance; seated_team/reasons.rs: a seated team's bound action refused without a listed reason in a block and in the mempool, a proposal with no reason, reversals and the interim unbound; packages/rs-dpp/src/moderation_charter/reward_split.rs: the split's arithmetic; packages/rs-drive/src/drive/contract/moderation/action_count_tests.rs: the counts tree created with an elected contract only, written, read, bounded and reset, and the banlist on top of an elected contract's other tree.
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests/seated_team.rs: a contest for an elected contract's seat awarded, then the leader and the members moderating instead of the interim, another charter for the seated target refused whether or not its seat is contestable, additions and removals, the protection of the team, the cap on additions, abilities the declaration does not give, the interim block ending, a discounted fee charged and read where the declared one reads nothing, every other discount refused in a block and on recheck, and the interim's claim refused once a charter is seated.
  • packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests.rs: the whole pipeline, including a warned user carrying on with its warnings accumulating, proved and cleared, the warning limit and the clearing that lifts it, warnings kept through a ban and out of the gate and the ban's proof, every refusal of a warn, and the warning list fixed at creation; the moderators' window (a deletion to the millisecond it ends on, refused one later for the contract owner too while the author's own still passes, reopened by a replace, measured from $createdAt on a type that never changes, fixed on update), a moderator deleting a post (record, execution proof, the author's balance unchanged, and the control where the author deletes it and is refunded), every refusal of a deletion, an update adding a document type moderators can delete from, a permanent reference to such a type refused, the mempool refusal, the lapse sweep, the moderator set, every refusal code, the lists staying as the contract was created with them, a barred identity deleting its own documents in a block and in the mempool, a barred identity refused as the recipient of a transfer and as the seller of a purchase, the ban's proof covering the suspension it removed, lifting the entry of an identity an update made moderator, the per-list execution proof, a named owner, a create or an update naming a moderator that does not exist, an update keeping its moderators, inactivity of the transition and of a moderated contract create or update before protocol version 14, and elected moderation (each interim kind moderating or not, the block of a moderated type next to an unmoderated one in a block and in the mempool, a create refused outside a bound and for an unknown type, and an update refused for a changed field and for entering or leaving).

Documents

If data contracts are the tables, then documents are the rows. A document is an instance of a document type defined within a data contract. When a user creates a profile on DashPay, submits a domain name on DPNS, or stores any application data on the platform, they are creating a document.

Documents are the most fundamental unit of user data on Dash Platform. They are stored in GroveDB (through Drive), indexed for efficient querying, and cryptographically provable. Understanding how documents work at the Rust level is essential for working with the platform codebase.

The Document Enum

Like DataContract and Identity, Document is a versioned enum. From packages/rs-dpp/src/document/mod.rs:

#![allow(unused)]
fn main() {
#[derive(Clone, Debug, PartialEq, From)]
pub enum Document {
    V0(DocumentV0),
}
}

Currently there is only one variant, V0. But the enum wrapper is already in place so that future protocol versions can introduce a V1 variant without breaking existing code. All code that works with documents goes through the accessor traits, so adding a new variant is purely additive.

What Lives Inside a Document

The DocumentV0 struct is defined in packages/rs-dpp/src/document/v0/mod.rs:

#![allow(unused)]
fn main() {
pub struct DocumentV0 {
    pub id: Identifier,
    pub owner_id: Identifier,
    pub properties: BTreeMap<String, Value>,
    pub revision: Option<Revision>,
    pub created_at: Option<TimestampMillis>,
    pub updated_at: Option<TimestampMillis>,
    pub transferred_at: Option<TimestampMillis>,
    pub created_at_block_height: Option<BlockHeight>,
    pub updated_at_block_height: Option<BlockHeight>,
    pub transferred_at_block_height: Option<BlockHeight>,
    pub created_at_core_block_height: Option<CoreBlockHeight>,
    pub updated_at_core_block_height: Option<CoreBlockHeight>,
    pub transferred_at_core_block_height: Option<CoreBlockHeight>,
    pub creator_id: Option<Identifier>,
    pub contract_version: Option<u32>,
}
}

Let us walk through the key fields:

  • id: A 32-byte unique identifier. Unlike contract IDs, document IDs are derived from a combination of the contract ID, owner ID, document type name, entropy and (protocol v14+) the identity contract nonce of the create transition. This makes them deterministic, unique, and impossible to produce twice.

  • owner_id: The identity that currently owns this document. Ownership can change if the document type supports transfers.

  • properties: The actual application data, stored as a BTreeMap<String, Value>. The Value type comes from platform-value and can represent strings, integers, byte arrays, nested maps, and arrays. The keys correspond to the property names defined in the document type's JSON Schema.

  • revision: An Option<Revision> (which is a u64). Mutable documents track revisions -- each update increments the revision. Immutable document types will have None here.

  • Timestamps: Six pairs of timestamp fields covering three events (creation, update, transfer) across three time references (milliseconds, block height, core block height). Whether these are populated depends on the document type schema -- if the schema requires $createdAt, the platform fills it in when the document is created.

  • creator_id: The original creator of the document. This differs from owner_id when a document has been transferred to a new owner.

  • contract_version: The data contract version this document's bytes conform to — the contract version stamp (protocol v14+, document serialization format 3). Drive assigns it whenever document content is supplied (create and replace) and preserves it through transfers and purchases. None means the document was serialized before format 3, which predates every requiredSince annotation. The stamp resolves per-property byte layouts when a document type gains required properties through contract updates — see the Document Serialization chapter.

Document ID Generation

Document IDs are not random -- they are derived deterministically. From packages/rs-dpp/src/document/generate_document_id.rs:

#![allow(unused)]
fn main() {
impl Document {
    pub fn generate_document_id_v1(
        contract_id: &Identifier,
        owner_id: &Identifier,
        document_type_name: &str,
        entropy: &[u8],
        identity_contract_nonce: IdentityNonce,
    ) -> Identifier {
        let mut buf: Vec<u8> = Vec::with_capacity(/* ... */);

        buf.extend_from_slice(DOCUMENT_ID_V1_DOMAIN_TAG); // b"dash:document-id:v1"
        buf.extend_from_slice(contract_id.as_slice());
        buf.extend_from_slice(owner_id.as_slice());
        buf.extend_from_slice(document_type_name.as_bytes());
        buf.extend_from_slice(entropy);
        buf.extend_from_slice(&identity_contract_nonce.to_be_bytes());

        Identifier::from(hash_double(&buf))
    }
}
}

The ID is a double SHA-256 hash of a domain tag, the contract ID, owner ID, document type name, client-provided entropy and the identity contract nonce of the create transition. Document::generate_document_id picks the derivation from the platform version; consensus recomputes it for every create and rejects a mismatch with InvalidDocumentTransitionIdError. This means:

  • The ID commits to the owner, so nobody else can take it, and to the contract and document type, preventing cross-contract collisions.
  • The ID is a deterministic function of its inputs: whoever knows the entropy and the nonce, which is the client building the create transition, can compute it before the document exists (see below). The entropy is the one input other parties can not guess, so as long as the client generates it unpredictably, nobody else can compute the ID of a document that has not been sent yet and point other documents at it in advance.
  • The nonce makes the ID single use. An identity contract nonce is consumed at most once, so an ID can be produced at most once.

Why the nonce is part of the ID

Up to protocol version 13 the ID was generate_document_id_v0: the same hash without the domain tag and the nonce. The create check only asks whether a document exists under the ID right now, so the owner of a deleted document could create a new document with the same entropy and get the same ID back, with different content. Everything that referenced the ID (likes, replies, a refersTo property, a moderation removal record) then pointed at the new content. For a document type with documentsMutable: false and canBeDeleted: true that is content substitution, the very thing immutability is supposed to rule out.

From protocol version 14 a reference to a document ID means that one document or nothing. This also holds for documents created before the upgrade: their entropy-only IDs can not be produced by the new derivation, and the old derivation is no longer accepted.

What this means for clients

The ID of a new document only exists once the nonce of its create transition is assigned, and it changes if the transition is rebuilt with another nonce:

  • The ID a Document carries before its create transition is built (for example the one create_document_from_data gives it) is a placeholder. DocumentCreateTransitionV0::from_document replaces it with the derived ID, so every transition built through dpp carries the right one.
  • On the Rust path (rs-sdk, or from_document directly) the Document you passed in keeps its placeholder: read the ID from the transition, or from the confirmed document put_to_platform_and_wait_for_response returns.
  • To know IDs up front (a chain of documents that reference each other), assign the nonces first: nonces may be used out of order within a window of 24.
  • JavaScript gets the same through wasm-dpp2, with one difference: new DocumentCreateTransition({ document, identityContractNonce }) derives the ID for the network's protocol version (platformVersion option, latest by default) and writes it both onto the transition and back onto document, so after construction document.id is the final ID and may be read from there. Document.generateId(type, owner, contract, entropy, identityContractNonce) and document.setIdForCreation(identityContractNonce) give the ID before the transition exists, and new Document({ ..., identityContractNonce }) derives it at construction (an explicit id passed alongside the nonce must equal the derived one). A Document built without a nonce carries the entropy-only placeholder until it is passed to DocumentCreateTransition. No app needs to reimplement the hash.

The Accessor Traits

Documents follow the same accessor-trait pattern as data contracts. The getter trait is defined in packages/rs-dpp/src/document/accessors/v0/mod.rs:

#![allow(unused)]
fn main() {
pub trait DocumentV0Getters {
    fn id(&self) -> Identifier;
    fn owner_id(&self) -> Identifier;
    fn properties(&self) -> &BTreeMap<String, Value>;
    fn properties_mut(&mut self) -> &mut BTreeMap<String, Value>;
    fn revision(&self) -> Option<Revision>;
    fn created_at(&self) -> Option<TimestampMillis>;
    fn updated_at(&self) -> Option<TimestampMillis>;
    fn transferred_at(&self) -> Option<TimestampMillis>;
    fn created_at_block_height(&self) -> Option<u64>;
    fn updated_at_block_height(&self) -> Option<u64>;
    fn creator_id(&self) -> Option<Identifier>;
    fn contract_version(&self) -> Option<u32>;
    // ... and more
}
}

The setter trait extends it with mutation methods and also provides convenient typed setters:

#![allow(unused)]
fn main() {
pub trait DocumentV0Setters: DocumentV0Getters {
    fn set_id(&mut self, id: Identifier);
    fn set_owner_id(&mut self, owner_id: Identifier);
    fn set_properties(&mut self, properties: BTreeMap<String, Value>);
    fn set_revision(&mut self, revision: Option<Revision>);
    fn set_created_at(&mut self, created_at: Option<TimestampMillis>);
    fn set_updated_at(&mut self, updated_at: Option<TimestampMillis>);

    // Generic property access via path syntax
    fn set(&mut self, path: &str, value: Value) { ... }
    fn remove(&mut self, path: &str) -> Option<Value> { ... }

    // Typed setters for common types
    fn set_u8(&mut self, property_name: &str, value: u8);
    fn set_u64(&mut self, property_name: &str, value: u64);
    fn set_bytes(&mut self, property_name: &str, value: Vec<u8>);
    // ... and more
}
}

Notice the set() method provides lodash-style path syntax: "root.people[0].name". Parents are created automatically if they do not exist.

The DocumentMethodsV0 Trait

Beyond simple field access, documents have behavior defined by the DocumentMethodsV0 trait in packages/rs-dpp/src/document/document_methods/mod.rs:

#![allow(unused)]
fn main() {
pub trait DocumentMethodsV0 {
    fn get_raw_for_contract(
        &self,
        key: &str,
        document_type_name: &str,
        contract: &DataContract,
        owner_id: Option<[u8; 32]>,
        platform_version: &PlatformVersion,
    ) -> Result<Option<Vec<u8>>, ProtocolError>;

    fn get_raw_for_document_type(
        &self,
        key_path: &str,
        document_type: DocumentTypeRef,
        owner_id: Option<[u8; 32]>,
        platform_version: &PlatformVersion,
    ) -> Result<Option<Vec<u8>>, ProtocolError>;

    fn hash(
        &self,
        contract: &DataContract,
        document_type: DocumentTypeRef,
        platform_version: &PlatformVersion,
    ) -> Result<Vec<u8>, ProtocolError>;

    fn increment_revision(&mut self) -> Result<(), ProtocolError>;

    fn is_equal_ignoring_time_based_fields(
        &self,
        rhs: &Self,
        also_ignore_fields: Option<Vec<&str>>,
        platform_version: &PlatformVersion,
    ) -> Result<bool, ProtocolError>;
}
}

The get_raw_for_contract and get_raw_for_document_type methods retrieve a document property as raw bytes, using the document type schema to determine how to serialize the value. This is critical for building index keys and storage operations.

The is_equal_ignoring_time_based_fields method is particularly useful in validation. Since timestamps and block heights are set by the network (not the client), you often want to compare two documents while ignoring those fields -- for example, to verify that a client's update only changed the fields it was supposed to change.

Version Dispatching in Methods

Every method in the Document implementation dispatches through the platform version, following the standard pattern:

#![allow(unused)]
fn main() {
impl DocumentMethodsV0 for Document {
    fn get_raw_for_contract(
        &self,
        key: &str,
        document_type_name: &str,
        contract: &DataContract,
        owner_id: Option<[u8; 32]>,
        platform_version: &PlatformVersion,
    ) -> Result<Option<Vec<u8>>, ProtocolError> {
        match self {
            Document::V0(document_v0) => {
                match platform_version
                    .dpp
                    .document_versions
                    .document_method_versions
                    .get_raw_for_contract
                {
                    0 => document_v0.get_raw_for_contract_v0(
                        key, document_type_name, contract,
                        owner_id, platform_version,
                    ),
                    version => Err(ProtocolError::UnknownVersionMismatch {
                        method: "DocumentMethodV0::get_raw_for_contract".to_string(),
                        known_versions: vec![0],
                        received: version,
                    }),
                }
            }
        }
    }
}
}

This is a double dispatch: first on the document variant (V0), then on the method version from the platform version configuration. This allows the platform to evolve both the document structure and the behavior of document methods independently.

How Documents Reference Their Contract

Documents do not carry a reference to their contract inside the struct itself. Instead, the relationship is established through context -- the document type name and contract are passed alongside the document whenever they are needed (for serialization, validation, hashing, and storage).

When serializing, a document is always serialized relative to its document type:

#![allow(unused)]
fn main() {
let serialized = <Document as DocumentPlatformConversionMethodsV0>::serialize(
    &document,
    document_type,  // the schema determines field order and encoding
    &contract,
    platform_version,
)?;

let deserialized = Document::from_bytes(
    &serialized,
    document_type,  // same schema needed for decoding
    platform_version,
)?;
}

This means a document's binary representation is not self-describing. You need the document type definition to interpret the bytes. This is a deliberate design choice for storage efficiency -- field names are not repeated in every serialized document.

The INITIAL_REVISION Constant

When a new document is created, it starts at revision 1:

#![allow(unused)]
fn main() {
pub const INITIAL_REVISION: u64 = 1;
}

Revision 0 is never used for active documents. This allows 0 to serve as a sentinel value meaning "no revision" in some contexts.

Document References (refersTo)

From protocol version 14 a property of a document type can declare what it points at, and consensus refuses a create or replace whose target does not exist when the document is written (the reference is a write-time constraint only; nothing resolves it for a reader). The keyword is refersTo on the property, its type one of identity, contract (optionally with contractRequirements, see Contract Moderation), token, permanentDocument, moderatedDocument (see A document only moderators remove), deletableDocument and identityPublicKey, or a reference expression combining several with anyOf and allOf (see Reference expressions). A document reference may find its document by the values of a unique index instead of by id (findBy, see Found by a unique index), check the document found (where, see Checked on the document found), and, on a permanentDocument, take its value as an element of a list the document holds (inList, see An element of a list). Every form sits on an identifier property, with one exception below. The parsed shape is DocumentPropertyType::IdentifierWithReference(target), and any change to a declaration on contract update is an incompatible schema change.

An identityPublicKey reference names one key of one identity, and comes in two forms that differ in which property carries what:

  • On the identity property. The identifier property carries the identity id and keyIdProperty names the sibling integer property carrying the key id: "refersTo": { "type": "identityPublicKey", "keyIdProperty": "toKeyIndex" }. Consensus fetches the named key of that identity.
  • On the key id property. The integer property carries the key id and identityProperty names whose key it is: "refersTo": { "type": "identityPublicKey", "identityProperty": "$ownerId" }. The property must declare exactly the range of a KeyID ("type": "integer", "minimum": 0, "maximum": 4294967295) and the declaration takes no keyIdProperty. identityProperty is $ownerId (the writer), $creatorId (the document's creator, only on a document type that records creator ids: a transferable or tradeable type of a format-1 contract) or the path of an identifier property of the same document type (which must exist, be an identifier and not carry an identityPublicKey reference of its own); the last two are checked at contract registration. keyRequirements sit on this form exactly as on the identifier form. The parsed shape is DocumentPropertyType::KeyIdWithReference(KeyIdReference), the identity source plus the requirements, sized, encoded and queried exactly as a plain u32.
"senderKeyId": {
  "type": "integer", "minimum": 0, "maximum": 4294967295,
  "refersTo": { "type": "identityPublicKey", "identityProperty": "$ownerId" },
  "position": 3
}

Both forms share the state check (validate_referenced_identity_key_v0 in the document reference validation): the key must exist and not be disabled, else the write is refused, paid, with ReferencedIdentityKeyNotFoundError (40123) or ReferencedIdentityKeyDisabledError (40124). Identity keys can be disabled but never removed, so a validated reference never dangles. The owner form's identity is the transition's signer, which the transition already proved exists, so the key fetch is its only read; a key id of some other identity's key is meaningless by construction, there is no property to name another identity. On replace the identity form is re-validated when either the identity property or its key id property changed. For the key id form it depends on where the identity comes from. $ownerId is the writer, transition metadata that never appears among the changed fields, and the document may have changed hands since the key id was written, so the reference is re-validated on every replace, touched or not, as the $ownerId writer gate is: after a transfer the new owner has to repoint the key id at one of its own keys. $creatorId never changes, so it is re-validated when the key id changed. A document written before its type recorded creator ids (no type did before protocol version 10, nor one of a contract whose config was still version 0) records none, and a contract update may add a $creatorId key id property to its type: setting that key id on such a document is refused (ReferencedKeyIdPropertyInvalidError, 40125), while a replace leaving it unset is not checked. A property path is re-validated when the key id or that property changed, and a key id set while the property is not is refused (ReferencedKeyIdPropertyInvalidError, 40125). A transfer itself is not checked in any form, so the reference governs writing, not holding. The declaring property must carry exactly the key id range in its schema, whatever the contract's integer sizing setting, and a keyIdProperty of the identity form may not name a property that carries this form, nor may a path name an identifier carrying an identityPublicKey reference: one pair is declared once (40125 at registration). The charter contract's joinRequest.senderKeyId, the owner's encryption key a shared secret is derived from, is the first user.

A document only moderators remove (moderatedDocument)

The three document references are disjoint, by what can make the referenced type's documents leave state (DocumentTypeV2Getters::document_reference_kind, a DocumentReferenceKind): nothing, for permanentDocument; only a removal by the contract's moderators that leaves a record, for moderatedDocument; anything else, for deletableDocument. A type is a moderated one when its owners can not delete its documents (canBeDeleted: false, not "onlyWhenConsumed", whose documents a consume deletes without a record), it declares no ttl, and moderatorAbilities.delete keeps a removal record (deleteKeepsRecord, true unless declared false). None of these can change on a contract update, so the kind a type admits holds for good. Registration (validate_data_contract_references 0) and the write (the document reference validation) refuse a declaration of another kind through one helper, document_reference_kind_mismatch: a moderatedDocument reference to any other type with ReferencedDocumentTypeNotModeratedError (state code 40143), a deletableDocument reference to a moderated type with ReferencedDocumentTypeModeratedError (40144), and the two earlier mismatches as before (40122, 40131).

In Rust the declaration is DocumentPropertyReferenceTarget::ModeratedDocument { contract_id, document_type_name, property_agreement }, appended to the enum so every earlier variant keeps its consensus encoding; as_any_document_reference returns it with kind: DocumentReferenceKind::Moderated. It is found by its id alone: the parser refuses findBy (a moderator's removal frees the document's unique index keys, so a key could come to find another document while the reference is held to the removed one's record), inList, an operand of a reference expression and a writer or creator reference.

The document must exist when the reference is written. A replace re-validates it as a permanentDocument reference (binds_a_changed_property): when the value changed, when the referring side of a where entry changed, and on every replace for a writer gate. When the value is one the stored document already held (a single reference whose path did not change, or an element the stored list held) and the document is not in state, the validation reads the removal record (fetch_contract_document_removal_with_fee, billed) and, when one stands unrestored, the reference holds through it (validate_pairs_against_removal): each where entry the replace asks about again compares the referenced $ownerId with the record's document_owner_id (a removed document can not be transferred, and a restore brings it back as it was) and $id with the id, and any other referenced property refuses the replace with ReferencedDocumentRemovedError (40145). An entry the replace does not ask about held when it was last checked. A value the write sets must name a document in state (ReferencedEntityNotFoundError, 40120). Whether a value is held is decided from the stored value (kept_value_changed_fields): a value under an unchanged top-level property is the stored one, and under a changed one (an object whose other property changed, or a typed array a replace changed) it is held when the stored value the replace action carries at the same path is it, or holds it. The parser admits a writer gate on a moderated reference against the referenced $ownerId or $id only (10231), since a writer gate is asked about on every replace. The write-time check lets a deletableDocument reference to a moderated type through, one a contract registered before the kind existed may hold; registration refuses it.

A chained query or a composite by-id join through a moderated reference proves the removal records of the joined ids beside their documents, one more component of the merged proof (query::moderated_join::removals_path_query: the ids under [64, contract, 2, 16, <document type>], walking as the join does, unlimited as the by-ids fetch is). A composite query builds one records component per referenced document type, over every id its moderated joins derived. The component names every joined id, not only the missing ones, since the verifier builds the merged query before it knows which are missing, and grovedb proves each queried key present or absent, so a prover can pass neither a removed document off as live nor a live one off as removed. A joined id with no document is reported with its record (ChainedDocumentsResult::removed_outer_documents, CompositeDocumentsResult::sub_result_removals, on the wire removed_outer_documents = 4 and removed = 4, clients removedOuterDocuments and removed); one with neither a document nor an unrestored record is refused, as a missing permanentDocument target is.

Checked on the document found (where)

A permanentDocument, moderatedDocument or deletableDocument reference may carry where: 1 to 10 entries { "<referenced property>": "<referring value>" }, each an equality the document found must meet when the referring document is written. The key is a property of the referenced document type, or its $ownerId, $creatorId or $id; the value is a property of the declaring document type, or "$ownerId", the writer, which makes the entry a write gate. A referring value may be compared once. where never finds the document: the value does, as its id, or findBy does (below), and where is checked against the document that was found. The moderation charters' joinRequest.submittedCharterId declares:

"refersTo": {
  "type": "permanentDocument",
  "documentType": "submittedCharter",
  "where": { "$ownerId": "recipientId" }
}

reads: the proposal this join request names must be owned by its recipientId. A document found that fails an entry refuses the write, paid, with ReferencedDocumentPropertyMismatchError (40127); no document found is ReferencedEntityNotFoundError (40120). At registration both sides must exist, be single values and share one value kind, and the referenced side may not be transient (ReferencedDocumentPropertyAgreementInvalidError, 40126); a where holding "." or a function is refused by the parser, since both find the document and belong in findBy. In Rust the parser holds where as property_agreement, keyed by the referring property ({referring property: referenced property}), so the parsed model reads the opposite way round from the schema.

Read as SQL, a document reference is a query that must return a row: documentType is the FROM, findBy the part of the WHERE that finds the row through a unique index (without it, $id = value), where the rest of the WHERE, checked on the row found, minimumAgeBlocks an age condition and consume a delete of the row found.

The spellings a protocol version 14 beta used, lookup (which named the index and mapped its properties under keys), propertyAgreement (pairs keyed by the referring property) and the type listElement, were replaced by findBy, where and inList on a permanentDocument before the version shipped. Meta-schema v3 does not admit them, and the parser refuses them on every parse (refuse_replaced_reference_keywords, InvalidContractStructure, 10231) with a message naming what replaced each, so a contract written with them never loads with another meaning.

Found by a unique index (findBy)

A permanentDocument reference normally holds the referenced document's id. It may instead carry findBy: the property's value, or on the elements of a typed array each element (see References on the Elements), is then one part of a key, and the referenced document is the one the unique index of the referenced document type over exactly the properties findBy names finds for that key. The reference holds if that document exists. The moderation charter's members is the first user:

"members": {
  "type": "array", "minItems": 0, "maxItems": 15, "uniqueItems": true,
  "items": {
    "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
    "contentMediaType": "application/x.dash.dpp.identifier",
    "distinctFrom": "$ownerId",
    "refersTo": {
      "type": "permanentDocument",
      "documentType": "joinRequest",
      "findBy": { "submittedCharterId": "submittedCharterId", "$ownerId": "." }
    }
  },
  "position": 2
}

reads: every member must be the owner of a joinRequest whose submittedCharterId equals this document's submittedCharterId. findBy names submittedCharterId and $ownerId, exactly the properties of joinRequest's unique bySubmittedCharter index, which finds it. Without findBy the list would have to hold the join requests' ids, which the writer would have to find first, and which say nothing about who asked to join. The same form works on a scalar identifier property, where "." is the property's own value.

A deletableDocument reference may carry findBy too, and then promises less than its id form. Once the document a key found is deleted, a new document with the same key makes the reference resolve again, to different content, where an id is produced at most once and a dead id reference stays dead. So a deletableDocument found by findBy means "a document with this key exists now", which is what a membership gate needs: the moderation charters' resignationRequest requires its writer to have an addedModerator for the charter now, and the leader takes an added member off by deleting that document. Every replace re-validates it, as it does a deletableDocument reference by id; an immutable property may not hold one, since the clearing a dead id reference allows reads an id, which a key is not; and it may be an operand of a reference expression and the target of an ownerRefersTo (see below).

findBy maps properties of the referenced document type, by their names there (system ones such as $ownerId included), in any order, to where their values come from on the referring side:

  • a property path of the referring document type ("submittedCharterId", "meta.charterId");
  • "$ownerId", the referring document's owner, the writer;
  • ".", the value of the property that carries the reference, or the element. It appears exactly once: without it every value would resolve to the same document;
  • a function, at most one (see Commit and reveal).

findBy names no index. The index is the unique one of the referenced document type whose properties are exactly those findBy names (DocumentReferenceLookup::resolve_index); two unique indexes over the same properties find the same document, so the first by name is taken. A type's indexes can never be added, removed or changed by a contract update, so the same findBy always resolves to the same index.

What is checked when the contract enters the chain, on registration and on update:

  • findBy is only allowed on permanentDocument and deletableDocument references, and where on those and moderatedDocument ones (meta-schema v3 and the parser, apply_property_reference 0), on the property or on the items of a typed array.
  • Each property an entry reads must exist on the referring type, be required (and so must every object around it), not be transient nor sit inside a transient object, and hold a single value, so findBy never runs with a missing key part and a reader can assemble the same key from the stored document. An entry that reads "$ownerId" needs a referring type whose documents can be neither transferred nor traded: the reference is judged when the document is written, and a transfer or purchase would move the writer part of its key without a write. These are properties of the referring type alone and are checked on every parse (generation 3).
  • The entries must name exactly the properties of a unique index of the referenced type, and nothing else. One naming no such index is refused with the unique indexes the type has ("joinRequest" has no unique index over exactly (submittedCharterId): findBy must name every property of one of its unique indexes and nothing else (bySubmittedCharter (submittedCharterId, $ownerId))), and one naming exactly the properties of an index that is not unique is refused too, since it could find several documents (index "byOwner" of "joinRequest" over ($ownerId) is not unique: findBy must find at most one document). The index may not bucket its first property with timeRange or integerRange, and the referenced type may not be indexOnly. Each source must hold the same kind of value as the property it fills (the rule of where, DocumentPropertyType::value_kind).
  • The key must stay with the document it found, or the reference could dangle without the document being deleted: every schema property of the index must be fixed once written (the referenced type is immutable, or the property, or the top-level object holding it, is listed under immutable and is no optional deletableDocument reference by id, which a replace may clear once its document is deleted), $ownerId is only a key part on a type whose documents can be neither transferred nor traded, and the update and transfer times are refused where a replace, transfer or purchase moves them. $id, $creatorId and the creation times are always fixed.
  • { "$id": <property> } is admitted only with inList, as its only entry (see An element of a list).
  • A changed, added or removed findBy is an incompatible schema change on update, like the rest of a refersTo.

The checks on the referenced type run where that type is in hand. For a document type of the same contract the contract parse runs them under full validation (create_document_types_from_document_schemas 1, next to the keyRequirements.boundTo check), once every document type is parsed; a target of the wrong kind is left to registration, which refuses a deletable type for a permanentDocument found by findBy (40122), and for a deletableDocument found by findBy one that forbids deletion (40131) or whose documents only moderators remove, on the record (40144). For a type of another contract (contractId) the registration state validation runs them against that contract, where the other refersTo checks into another contract run, and refuses a declaration that cannot resolve with ReferencedDocumentLookupInvalidError (state code 40137), whose message names the properties findBy names. Index definitions cannot change on a contract update from protocol version 14 (validate_update 1 compares them by name), and neither can the flags the permanence rule reads, so the answer holds.

When the referring document is created or replaced, the document reference validation assembles the key for each value and queries the index for at most one document, billed as a document fetch of the same kind as a fetch by id (fetch_document_through_lookup). No document, or a key it cannot assemble, refuses the write, paid, with ReferencedEntityNotFoundError (40120) naming the property, or the element by its list path (members[1]); its target reads "found by" and the properties findBy names. A where beside findBy is checked against the document the index found, exactly as for an id reference. A replace re-validates a permanentDocument found by findBy when the property itself changed (for a list, the elements the stored list did not hold), and every value, every element included, when a property an entry reads changed. Nothing else can move a key part: the writer is fixed on a type allowed to read it, and the referenced side's key is fixed by the rule above, so a validated permanent reference found by findBy never dangles. A deletableDocument found by findBy is re-validated on every replace, since the document it found may be gone.

Joins cannot go through a reference found by findBy: a chained query or a composite by-id join needs the join property's values to be the outer documents' ids, so both refuse such a property, and a preallocated index cannot be bound through one. In Rust the declaration is its own variant, DocumentPropertyReferenceTarget::PermanentDocumentLookup (and DeletableDocumentLookup for the deletable form, appended after it), rather than a field of PermanentDocument: the enum is embedded in the reference errors, so an id reference keeps its encoding, and code matching PermanentDocument as "the value is a document id" cannot mistake a key part for one. The rules are on DocumentReferenceLookup, which holds findBy as keys, each referenced property mapped to its LookupKeySource. as_document_reference returns only references whose value is a document id, the accessor for joins; the validators use as_any_document_reference, whose declaration carries the DocumentReferenceLookup in its lookup field.

Reference expressions (anyOf, allOf)

A refersTo may combine targets in place of naming one. { "anyOf": [...] } holds if at least one operand holds, { "allOf": [...] } if every operand holds for the same value. An operand is a leaf, an ordinary target with its own keys, or an expression of the other combinator, so the two nest:

"memberId": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "refersTo": {
    "anyOf": [
      {
        "type": "permanentDocument", "documentType": "addedModerator",
        "findBy": { "submittedCharterId": "submittedCharterId", "moderatorId": "." }
      },
      {
        "allOf": [
          { "type": "identity" },
          {
            "type": "permanentDocument", "documentType": "joinRequest",
            "findBy": { "submittedCharterId": "submittedCharterId", "$ownerId": "." }
          }
        ]
      }
    ]
  },
  "position": 2
}

reads: the member was added to the charter, or it is an identity that asked to join it. The same form sits on the items of a typed array, where each element meets the expression on its own.

What is checked when the contract enters the chain:

  • On every parse (meta-schema v3 and the parser, apply_property_reference 0): a combinator is the declaration's one key (a findBy, where or inList belongs to a leaf, inside it), a list names at least two operands (a single one is declared on its own), and an anyOf directly inside an anyOf (or an allOf inside an allOf) is refused, since it says what one flat list says.
  • Every leaf is an identity, a permanentDocument (by id, by findBy or with inList) or a deletableDocument found by findBy. The first two are existence checks against entities that are never deleted (inList reads a list that never changes on a document that is never deleted), so an expression of only them holds for good once it holds, as a single one of them does, and a replace re-validates it only when its value or a property one of its leaves binds changed. A deletableDocument leaf found by findBy may find nothing later, so an expression holding one is re-validated on every replace, and an immutable property may not hold it. A leaf whose key a findBy function computes is the exception: it is judged on the create alone and holds on a replace without a read, so an immutable property may hold it, and on a document type whose documents can be replaced it may not be an operand of an anyOf, which would then hold on every replace whichever operand held on the create. The other types do not compose with other operands and are refused, as is the key id form (identityProperty): deletableDocument by id is re-validated on every replace and may be cleared once its document is deleted (the immutable-property exception), which assumes the property refers to that one target; identityPublicKey pairs the value with a key id property no other operand reads; a contract target's requirements are gates judged against the block time and the writer rather than an existence check, and a contract or token id is never also an identity or document id. Admitting one later takes a new apply_property_reference generation.
  • Under full validation (registration): at most SystemLimits::max_reference_operands operands in one list and at most max_reference_expression_depth combinators on any path from the declaration to a leaf (4 and 4 at protocol version 14; the example above is 2 deep), no two alike operands in one list (a leaf naming the declaring contract explicitly is the same as one omitting it), and every leaf counted against max_references_per_document: an anyOf of two on a typed array of maxItems 15 counts 30, since each leaf may be read for each element.
  • Every leaf is checked exactly as the same target declared alone: the referenced document type, its permanence, the where sides and the findBy or list rules, at the same places (the contract parse for a type of the same contract, the registration state validation for another contract's). Every leaf must pass, since each has to be a declaration that could hold. An error names the failing leaf by where it sits, refersTo anyOf[1].allOf[1] findBy: ... from the parse and resignation.memberId.anyOf[1].allOf[1] from registration.
  • A changed expression (an operand added, removed, changed or moved, anyOf swapped for allOf, a single target turned into an expression or back) is an incompatible schema change on update, like the rest of a refersTo. Inside refersTo, anyOf and allOf are the declaration's data; the schema compatibility rules never read them as JSON Schema keywords.

When the referring document is created or replaced, the document reference validation evaluates each value (each element) operand by operand in declared order, a nested expression the same way. An anyOf stops at the first operand that holds; when none does, the write is refused, paid, with the last operand's result. An allOf stops at the first operand that fails and refuses the write with its result. A refusal is therefore always the error a leaf declared alone would give (for the example, ReferencedEntityNotFoundError (40120) for a findBy that found nothing, naming the property or the element), and the author's order decides which one a writer sees: put the most general operand of an anyOf last, and the cheapest or most telling one of an allOf first. There is no error of its own for "no operand held": each leaf's failure already has a precise error, and a combined one would have to nest one per leaf or lose their reasons. Every read is billed as it is made, so a value the second operand of an anyOf holds for pays for the first operand's query too, while an allOf whose first operand fails reads nothing more. A where is checked only against its own leaf's document: a value whose first leaf fails its where is still accepted through a second leaf without one.

Joins and preallocated indexes need one target: a chained query or a composite by-id join refuses an expression join property, and a preallocated index is never bound through one. In Rust the combinators are DocumentPropertyReferenceTarget::AnyOf(ReferenceOperands) and AllOf(ReferenceOperands), appended to the enum so every single target keeps its encoding. An expression is no document reference as a whole (as_any_document_reference is None); code that checks every declaration walks DocumentPropertyReferenceTarget::leaves (or leaves_with_paths), the leaves of an expression or the declaration itself. Since the enum is embedded in consensus errors, which clients decode from bytes a node sends, decoding refuses a nesting deeper than MAX_REFERENCE_EXPRESSION_DECODE_DEPTH (16, above every protocol version's registration limit, which a test holds it to), so no bytes can drive the decoder into unbounded recursion. A reference error never carries a combinator: a refusal is a leaf's error.

An element of a list (inList)

A permanentDocument reference with inList says the value must be one of the identifiers a list of another document holds, the document findBy names by its $id:

"resignation": {
  "type": "object",
  "properties": {
    "electedCharterId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "refersTo": { "type": "permanentDocument", "documentType": "electedCharter" },
      "position": 0
    },
    "memberId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "refersTo": {
        "type": "permanentDocument",
        "documentType": "electedCharter",
        "findBy": { "$id": "electedCharterId" },
        "inList": "members"
      },
      "position": 1
    }
  },
  "required": ["electedCharterId", "memberId"],
  "additionalProperties": false
}

reads: memberId must be one of the members of the electedCharter document whose $id this document's electedCharterId holds. The moderation charters, whose elected charter holds its members, are the first users. The declaration sits on an identifier property, on the items of a typed array of identifiers, where every element must be listed (see References on the Elements), as a leaf of a reference expression (see Reference expressions), or on the writer or the creator (see On the writer or the creator), where the value is that identity: "ownerRefersTo": { "anyOf": [<inList>, <addedModerator findBy>] } is the charters' rule that the writer is a seated or an added moderator.

A reference with inList is a permanentDocument reference in every respect but one. It takes the contractId, documentType and where of any permanentDocument reference, with the same checks (the referenced type must forbid deletion, every where entry must exist and share one value kind). What differs is what the value is: not the referenced document's id but an element of its list. The document is the one findBy: { "$id": <property> } names, read from an identifier property of the referring type (a schema property, never $ownerId: no document has the writer's id), and inList names the typed array of identifiers on the referenced type. { "$id": ... } is the only findBy entry naming $id, and only with inList; where may not compare $id again. What is checked when the contract enters the chain:

  • inList is only allowed on a permanentDocument reference, and needs findBy to be exactly { "$id": <property> } (meta-schema v3 and the parser).
  • The $id entry reads a stored identifier property of the referring type (it and every object around it not transient), so a reader can tell from the stored document which list the value was checked against. It may be optional, and it needs no refersTo of its own. Checked by the parse under full validation (generation 3, validate_list_element_sources), a leaf of an expression as it would be alone.
  • The list's document type forbids deletion, and inList is a stored typed array of identifiers on it that never changes once a document is written: the type is immutable (documentsMutable: false), or the list's top-level property is listed under immutable, the rule a findBy key's properties are judged by. An immutableAllowSetting entry can only be set on a document that has no value for it, against which no value was ever accepted, so it does not weaken the rule. For a type of the same contract the contract parse checks this (create_document_types_from_document_schemas 1); for a type of another contract, registration checks it against that contract in state and refuses a list that does not qualify with ReferencedDocumentListInvalidError (state code 40138). A missing document type is refused as for any document reference (40121), and a deletable one by the parse for the same contract (with the list reason) or by registration for another (40122).
  • Each value counts against SystemLimits::max_references_per_document, one for a property, maxItems for a typed array, like every other reference.
  • A changed, added or removed inList is an incompatible schema change on update.

When the referring document is created or replaced, the document reference validation fetches the document whose id the $id entry's property holds, by id, checks the where entries against it exactly as for a permanentDocument, and requires the value to be in its list. Every by-id document fetch of one write is shared: a charter that electedCharterId's own reference already fetched, or that the elements of one typed array all name, is fetched and billed once, and the list is collected once into a set, so each value is a set lookup. A value the list does not hold, a document the id names that does not exist, or a value set while the $id property is not, refuses the write, paid, with ReferencedEntityNotFoundError (40120) naming the property, or the element by its list path (witnesses[1]); its target reads "list element (<inList> of the <documentType> document <property> names)". A failing where entry is ReferencedDocumentPropertyMismatchError (40127), as always. A replace checks a list reference again when its value changed or when a property findBy or where reads changed (the $id property among them, since it may name another charter), and then every value, every element included; only a list that changed on its own leaves out the elements the stored list already held. Nothing else can make a validated value unlisted: the list's document can never be deleted and its list never changes.

For example:

electedCharter 7kX...: members [Alice, Bob]
resignation { electedCharterId: 7kX..., memberId: Alice }  -> accepted
resignation { electedCharterId: 7kX..., memberId: Carol }  -> refused, 40120:
  referenced list element (members of the electedCharter document electedCharterId names)
  <Carol> not found for path memberId

In Rust the declaration is the appended variant DocumentPropertyReferenceTarget::ListElement(ListElementReference), so every earlier variant keeps its encoding in the reference errors; the parser holds the $id entry among the where comparisons of its property_agreement, as {property: "$id"}, and the rules are on ListElementReference (document_id_property, referring_side_error, referenced_side_error, listed_values). as_any_document_reference carries it with in_list set, so the registration validator checks its contract, type and where through the same code as the other document references, while as_document_reference leaves it out, as it does a reference found by findBy: joins and preallocated indexes never go through it.

On the writer or the creator (ownerRefersTo, creatorRefersTo)

A property's reference constrains a value the writer chose. Some rules constrain the writer instead: in the moderation charters, a resignationRequest may only come from a moderator of the team it resigns from. A document type states that with the doctype-level ownerRefersTo keyword, one refersTo declaration whose value is the document's $ownerId, the writer, rather than a property's value:

"resignationRequest": {
  "type": "object",
  "ownerRefersTo": {
    "type": "deletableDocument",
    "documentType": "addedModerator",
    "findBy": { "electedCharterId": "electedCharterId", "memberId": "." }
  },
  "properties": { "electedCharterId": { "...": "..." } }
}

reads: the writer must be the memberId of an addedModerator for this document's electedCharterId, one that exists when the request is written (the leader takes an added member off by deleting it).

  • The declaration is the one an identifier property carries, read by the same code (apply_property_reference 0), but only these targets can hold a writer: identity, a permanentDocument or a deletableDocument found by findBy, and a permanentDocument with inList (the writer an element of the list, see An element of a list), alone or as the leaves of a reference expression (above). The rest are refused, as a leaf of an expression too. contract, token and a document by id would need the writer's identity id to be a contract, token or document id, which it never is, so a document type declaring one could never be written; identityPublicKey pairs the value with a key id the writer does not carry. Meta-schema v3 reuses the property declaration by $ref and admits only those forms; the parser (generation 3, parse_owner_reference, which reads the stored schema once the core parse has run the meta-schema) refuses the others on the stored path too. The parsed declaration is DocumentTypeV2::owner_reference, read through DocumentTypeV2Getters::owner_reference, and the property types are unchanged.
  • Only a document type whose documents can be neither transferred nor traded may declare it, checked on every parse. A transfer or a purchase is not a write, so it would hand the document to an owner the declaration never checked; with neither possible, the owner of every document is the writer that was checked.
  • In findBy, "." is the writer, and a "$ownerId" source is the writer as well. Every referring-side rule of a property's findBy applies unchanged (its "$ownerId" rule holds by the point above), and so does every referenced-side rule, for a type of the same contract at contract level and for one of another contract at registration.
  • A where works as on a property reference; its value may be "$ownerId", which is then the same writer as the reference's value.
  • It counts one against SystemLimits::max_references_per_document.
  • When a document is created, the document reference validation checks the writer against the target exactly as a property's value is checked, before the properties' references. A replace re-validates it under the rules of its target, as a property's: when a property its findBy or where reads changed, and on every replace for a where entry valued "$ownerId", a writer gate. Nothing else can change the outcome: the writer is the owner, the target can never be deleted and its key is fixed. A failure is the error the target reports for a property (ReferencedEntityNotFoundError, 40120, when findBy finds no document; ReferencedDocumentPropertyMismatchError, 40127, for a where entry; and the rest), with $ownerId as its path. An identity target reads nothing: the transition has already proved that the writer exists. At registration the contract reference validation checks the declaration as a property's, naming it <documentType>.$ownerId.
  • Adding, removing or changing it is an incompatible schema change on update (validate_schema_compatibility 1 freezes the keyword as the shared rule set freezes refersTo).
  • Every validator, the reference bound and the client bindings enumerate a type's references through DocumentTypeRef::reference_declarations, which yields the owner or creator reference first, as ReferenceHolder::Owner or ReferenceHolder::Creator, then each property's, so none can skip it.

A document type whose documents can be transferred or traded declares creatorRefersTo instead: the same declaration, whose value is the document's $creatorId, its creator, which a transfer or a purchase never changes. A marketplace item that only an elected moderator may mint, and anyone may then own, reads:

"moderatorBadge": {
  "type": "object",
  "transferable": 1,
  "creatorRefersTo": {
    "type": "permanentDocument",
    "documentType": "electedCharter",
    "findBy": { "$id": "electedCharterId" },
    "inList": "members"
  },
  "properties": { "electedCharterId": { "...": "..." } }
}
  • It takes the same targets but a deletableDocument (the creator never changes, and a document a transfer handed on could not be replaced once the one findBy found is deleted), except one found by a findBy function, which is judged on the create alone (see Commit and reveal), with "." the creator in findBy, and is refused where ownerRefersTo is admitted: only a document type that records creator ids may declare it, a transferable or tradeable type of a format-1 contract (should_use_creator_id), checked on every parse. A type therefore declares at most one of the two. A "$ownerId" source in its findBy is refused, as in a property's findBy on such a type, since the owner moves.
  • When a document is created its creator is the writer; on a replace, the value is the stored creator, whoever writes, and the replace rules are those of its target, as for the owner reference. A transfer or a purchase needs no check. A failure is the target's error at the path $creatorId, and registration names the declaration <documentType>.$creatorId. An identity target reads nothing: the creator existed when it wrote the document, and an identity is never removed. It counts one against max_references_per_document, and a change to it is an incompatible schema change on update.

Commit and reveal (a findBy function)

One findBy entry may hold a function instead of a source: "<referenced property>": { "function": "sys.hash.sha256d", "params": [...] } says the referenced document's property holds a hash of values the document being created reveals. The platform fills that property of the key with the hash to find the referenced document, so the function computes a key part. The document it finds is a commitment made earlier, so the reference is a commit and reveal: the document may be created only while a commitment to values it carries exists. The function is a system function, as in generatedFrom, from the sys.hash namespace; sys.hash.sha256d, the SHA-256 of the SHA-256, is the one there is. DPNS name registration is a commit and reveal of this kind, done today by its domain create trigger rather than by a declaration: a preorder holds saltedDomainHash under a unique index, and the trigger hashes what the domain reveals and looks for it. The same rule for a name under a parent, written as a declaration on the salt the domain reveals, reads:

"preorder": {
  "type": "object",
  "documentsMutable": false,
  "canBeDeleted": true,
  "properties": {
    "saltedDomainHash": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 0
    }
  },
  "indices": [
    { "name": "saltedHash", "properties": [{ "saltedDomainHash": "asc" }], "unique": true }
  ],
  "required": ["$createdAtBlockHeight", "saltedDomainHash"],
  "additionalProperties": false
},
"domain": {
  "type": "object",
  "properties": {
    "preorderSalt": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 4,
      "refersTo": {
        "type": "deletableDocument",
        "documentType": "preorder",
        "findBy": {
          "saltedDomainHash": {
            "function": "sys.hash.sha256d",
            "params": ["preorderSalt", "normalizedLabel", { "const": "." }, "parentDomainName"]
          }
        },
        "where": { "$ownerId": "$ownerId" },
        "minimumAgeBlocks": 1,
        "consume": true
      }
    }
  },
  "transient": ["preorderSalt"]
}

reads: the salt refers to a preorder, found by its saltedDomainHash (the whole of its unique saltedHash index), that the writer owns (the where entry), created in an earlier block, whose saltedDomainHash is the sha256d of the salt, the normalized label, a dot and the parent; creating the domain deletes that preorder. The hash is byte for byte the one create_domain_data_trigger_v1 computes for a name under a parent. On an index of several properties, findBy maps the others beside the function entry as usual.

A string or byte array property may carry a refersTo only this way: its value is not an id, so its declaration must be a permanentDocument or deletableDocument reference found by a findBy function, and the property keeps its type (the meta-schema admits a refersTo with findBy on a string or byte array). The value carrying the reference must fill the key exactly once: as a "." entry or as a param of the function. A param names a single property carrying the reference by its path, as the salt above; "." in the params stands only for a value without a path of its own: each element of a typed array, the writer of an ownerRefersTo, the creator of a creatorRefersTo. The writer and the creator may also be left out beside a function, since the declaration applies to every create.

The preimage is the bytes of the params joined in order with nothing between them:

  • a property path: that property of the document being created, a string's UTF-8, a byte array's bytes, an identifier's 32 bytes;
  • { "const": text }: fixed UTF-8 text, 1 to 64 bytes;
  • ".": the value carrying the reference, where it has no path, read the same way.

1 to 16 params. System values such as "$ownerId" are not params. Plain concatenation is ambiguous when two variable-length params meet ("ab" + "c" and "a" + "bc" are the same bytes), so every variable-length param (a string, or a byte array whose size is not fixed) with another variable-length param anywhere after it must be followed directly by a one-byte const, its separator, and a create whose value for that param holds the separator byte is refused. Each preimage then splits into its params one way only. Integers and other kinds are not params. A path that passes through a map holding a key more than once is refused, as for generatedFrom.

The document such a key finds is judged once, when the document is created:

  • A param may be transient (the DPNS salt is), since it is read from the create transition, and optional, since a create missing one is refused. Every stored value findBy reads, params and other entries alike, must be fixed once written (the type is immutable or lists the property under immutable), and so must a stored property carrying the reference, set when the document is created (not listed under immutableAllowSetting). Such a deletableDocument found by findBy may sit on an immutable property, which one re-validated on every replace may not. A replace leaves the reference alone: nothing it reads can have changed, and the commitment may be gone. The declaration's where entries are judged with it, on the create alone, so each referring property they name must be fixed once written the same way, or transient. On a document type whose documents can be replaced, such a reference may not be an operand of an anyOf: judged on the create alone, it would hold on every replace, whichever operand held on the create.
  • creatorRefersTo takes a deletableDocument target only through a function.
  • findBy holds at most one function. The property it fills must be a byte array of exactly the hash's size, 32 bytes for sys.hash.sha256d. Beside findBy, on the refersTo, the reference may then declare what it asks of the document findBy finds (both are refused without a function in findBy):
    • minimumAgeBlocks: the found document's $createdAtBlockHeight must be at least that many blocks below the height of the create. 1 means an earlier block, so a commitment and its reveal cannot share a block. The referenced type must list $createdAtBlockHeight in required; a document recording no creation height never meets it.
    • consume: true: the create deletes the found document in the same state transition, its storage refunded to its owner as a delete by that owner would be. Only on a deletableDocument reference whose where holds the entry "$ownerId": "$ownerId", into a document type of the declaring contract whose owner may delete its documents (canBeDeleted: true), or whose documents only a consume deletes (canBeDeleted: "onlyWhenConsumed", DocumentTypeV2Getters::documents_deleted_only_when_consumed). No delete transition is charged for the consumed document, so its type may declare no delete token cost and no delete action fee, and the key signing the create deletes it, so its type may not require a stricter signature security level than the declaring type.
  • Whose commitment it is, is the where entry "$ownerId": "$ownerId" beside the function: the writer must own the document found. Without it anyone who learns a preimage may reveal it.

What is refused, and where:

  • The shape, the params and the rules above: the contract parse on registration and update (meta-schema v3 findByFunction, apply_property_reference 0, DocumentReferenceLookup::referring_side_error and referenced_side_error), and for a findBy into another contract the registration state validation (ReferencedDocumentLookupInvalidError, 40137), which also refuses consume there. A function in where is refused (it finds the document, so it belongs in findBy), as are a second function in findBy and minimumAgeBlocks or consume without one. Any change to a function, its minimumAgeBlocks or its consume is an incompatible schema change on update, like the rest of a refersTo.
  • A create missing a param, or whose variable-length value holds its separator: document create structure validation 1, before any read (DocumentReferencePreimageInvalidError, 10423). A function in a leaf of a reference expression is left to the write-time check, where a key it cannot assemble finds no document, so that operand fails and the others still decide.
  • No document for the key: ReferencedEntityNotFoundError (40120), whose entity id for a string or byte array carrier is the hash. The hash is computed once and billed as the double SHA-256 it is (ValidationOperation::DoubleSha256, by the blocks it hashes: the padded preimage and the second pass), the index query as the document fetch it is. Another identity's commitment where the owner entry is declared: ReferencedDocumentPropertyMismatchError (40127). A commitment younger than minimumAgeBlocks: ReferencedDocumentRequirementNotMetError (40142). All paid, from document create state validation 2.
  • A create signed by a contract-bound key whose bounds leave out a document type the created type may consume: batch advanced structure validation 1, as a direct delete of that type would be (ContractBoundedKeyOutOfBoundsError, 20014, paid).
  • A batch in which a create consumes a document that another create of the batch consumes too, or that another transition of the batch deletes, replaces, transfers, reprices or buys, refuses that create with 40120 (ConsumedDocuments in the batch state validation). Today a batch holds one transition, so this only guards the day that cap is raised.

In Rust the parser holds the function entry in the DocumentReferenceLookup as the source of the property it names: LookupKeySource::Hash(LookupHashKey), appended to the key sources, with its function a HashFunction (the SystemFunction::Hash namespace, beside the string transformations generatedFrom takes) and its params LookupKeyParams. The parsed property_agreement holds the where entries alone. A string or byte array property's declaration is DocumentProperty::revealed_reference, listed by reference_declarations as PropertyReference::Revealed. DocumentReferenceLookup::is_checked_on_create_only tells a lookup holding a computed key, and the parser holds the reference's minimumAgeBlocks and consume on the lookup as minimum_age_blocks and consume. first_unrevealable_lookup_key is the structure check. A create that consumes carries its commitments in DocumentCreateTransitionAction::consumed_documents, turned into delete operations with the create's own.

Immutable Properties on Mutable Document Types

A document type either allows replaces (documentsMutable: true, the default) or freezes its documents entirely. Protocol version 14 adds a middle ground: the doctype-level immutable keyword lists top-level properties a replace may not change while the rest of the document stays replaceable. An entry naming a property freezes it at creation; an entry { "property", "when" } freezes it for any replace its condition holds for.

"post": {
  "type": "object",
  "documentsMutable": true,
  "properties": {
    "author": { "type": "string", "maxLength": 63, "position": 0 },
    "body": { "type": "string", "maxLength": 500, "position": 1 },
    "mood": { "type": "string", "maxLength": 30, "position": 2 }
  },
  "required": ["author", "body", "$createdAt", "$updatedAt"],
  "immutable": [
    "author",
    { "property": "body", "when": { "greaterThan": [{ "subtract": ["$updatedAt", "$createdAt"] }, 300000] } },
    { "property": "mood", "when": { "present": "$old.mood" } }
  ],
  "additionalProperties": false
}

author is frozen at creation, body five minutes after it, and mood once the stored post holds one, so an absent mood may be set by one replace.

The parser (generation 3, meta-schema v3) reads the list on every parse (parse_immutable_keyword, apply_immutable_fields):

  • The shape is checked on both paths: every entry is a string or an object with exactly property and when, and no property is listed twice. immutableAllowSetting, the keyword the present: "$old.<p>" condition replaces, is refused, naming its replacement, so that no contract written with it loads with another meaning.
  • A condition is parsed with the propertyConstraints grammar (parse_property_constraint_condition) and what it reads is checked as a rule's reads are (validate_property_constraint_reads), on both paths: a stored condition reading an undeclared or transient property could only read it as absent. A path through $old. (STORED_DOCUMENT_PREFIX) is judged as the path after it, and only a condition of immutable may use one. A condition may not read a countOf or sumOf, so a replace judges it without reading state.
  • Under full validation, when a contract enters the chain: the type is mutable; every listed property is a declared top-level property, neither a system property nor transient (list the containing object to freeze a nested value); a condition stays within a rule's node limit and lists no condition twice. Stored contracts skip these lints, so a later tightening can never make a committed contract unreadable.
  • A deletableDocument reference by id may be listed only without a condition. A replace may clear such a reference once its target is deleted (see References on the Elements), and a replace the condition then left free could set it to another document: the frozen reference would be repointed. The clear stays, because every replace re-validates the reference and without the clear the document could never be replaced again, so the condition is refused at registration instead, and on contract update, which parses the whole new contract the same way. Every other deletableDocument form is refused on any immutable property.
  • A property listed with a condition is not fixed once written (schema_property_is_fixed_once_written): a findBy key, an inList list and the values read beside a findBy function may not rely on it.
  • On contract update the properties listed without a condition may only grow, and a property listed with a condition keeps it as it is or loses it to be listed without one (validate_immutable_fields_update, DocumentTypeUpdateError otherwise). Whether one condition holds wherever another does cannot be told in general, so a changed condition is refused. The schema compatibility differ strips the key, like indices and required, so validate_update v1 is the single judge.

Enforcement lives in the replace action's state validation (generation 1). The action already records which top-level properties differ from the stored document in changed_data_fields (the same set that scopes refersTo re-validation). A changed property in the type's immutable_fields() fails the replace with DocumentImmutablePropertyChangedError (state code 40128) unless it is a deletableDocument reference by id that the replace removed after its target was deleted. A changed property in immutable_field_conditions() fails it with the same error when its condition holds, or faults, judged by PropertyConstraint::holds on the written properties with the stored ones under $old and the replace's system values (the stored creation and transfer, this block as the update). The stored properties are rebuilt from the written ones and stored_changed_values, the stored value of every changed property; a property the stored document did not hold is left out. Conditions are evaluated only for the properties the replace changes. The replace compares stored and supplied values by underlying data, recursing into objects regardless of member order and into arrays position by position, with integer widths ignored, since storage reorders object members by schema position and narrows integers. "Differ" covers a changed value, a property the stored document lacked, and a property the replace dropped. Transfers, price updates and purchases carry no property data and are unaffected.

In Rust the properties frozen at creation are DocumentTypeV2Getters::immutable_fields() and those frozen under a condition immutable_field_conditions(), property to PropertyConstraint. Earlier document type generations return empty ones.

Transient Properties

The doctype-level transient keyword lists top-level properties whose values are validated on the transition but never stored. DPNS uses it for the domain's preorderSalt: the write proves the salted preorder, and the salt is then dropped.

"transient": ["preorderSalt"]

A create drops the listed values before its document is built. Before protocol version 14 a replace stored whatever it carried, so a replaced document kept values its create had dropped; from protocol version 14 a replace drops them the same way (document_from_replace_transition_action 1). Values are dropped by top-level name, so a leaf of a transient object goes with the object.

Because no stored document carries a transient value, a rule that reads a stored value refuses a transient one, judged by the property's path and every object around it (is_transient). From protocol version 14, at registration:

  • Every transient entry names a top-level property. A nested path, a system property or an undeclared name would mark a property transient in the parsed type while its value was still stored.
  • No index reads a transient property. Every document would sit in the index's null branch, so a query by the value would find nothing and a unique index would enforce nothing.
  • A refersTo findBy reads no transient property to assemble its key (a findBy function's params excepted, which are read from the create), and its index keys documents by none. A where names none on its referenced side. The referring side may be transient: it is judged on the transition, a write gate like the writer's $ownerId.
  • A key reference does not store its key id with a transient identity, whichever side declares it (identityProperty on the key id, keyIdProperty on the identity): the key id alone names no key.
  • encryptedFor names no transient recipient or key id.

The list cannot change on contract update: it decides which values stored documents carry and how every property is encoded (a transient property takes a presence byte even when required). From protocol version 14 any change to the names it lists is an incompatible schema change (the list is compared as a set, so reordering or repeating a name is no change); before it, the schema compatibility check failed on the keyword as unsupported, an internal error.

Typed Arrays

Up to protocol version 13 a type: "array" property had to be a byte array (byteArray: true). Protocol version 14 adds typed arrays: a list whose items schema says what every element is.

"reasons": {
  "type": "array",
  "minItems": 0,
  "maxItems": 64,
  "uniqueItems": true,
  "items": {
    "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
    "contentMediaType": "application/x.dash.dpp.identifier"
  },
  "position": 2
}
  • An element is a scalar: an integer, a number, a string (with minLength / maxLength), a boolean, a byte array (byteArray: true, whose minItems / maxItems count bytes) or an identifier. Objects and arrays of arrays are refused. An identifier element may carry refersTo (see References on the Elements). An element may be limited to allowed values with enum; const is refused on elements, since a one-value enum does the same and a contract update can still widen it. The parser reads an element's enum, minimum and maximum onto the typed array (ArrayItemConstraints), refusing on both parse paths an enum with no member or a member of another type, an enum on a byte array or identifier element, and a minimum above the maximum, so random document generation stays inside them; the JSON schema validator enforces them on every document.
  • On the array itself minItems and maxItems count elements, not bytes. maxItems is required, minItems may not exceed it and contentMediaType belongs on the items; these hold on every parse. Contract registration also caps maxItems at SystemLimits::max_typed_array_items (1024), so a typed array's worst-case size, which fee estimation charges by, stays small. uniqueItems: true refuses a document that repeats an element.
  • A byte array keeps its form and takes no items. On a plain byte array uniqueItems keeps its old meaning, no repeated byte, but an identifier (a byte array with the identifier contentMediaType) refuses it: an identifier is one value, and "no repeated byte" would refuse most of them.
  • The document is validated against the JSON schema as always, so a list that is too long, too short, repeats an element under uniqueItems or holds a wrong-typed element fails with the usual JsonSchemaError.

The array is stored inline in the document, like any other property: a varint element count followed by the elements, each encoded exactly as a required property of the element's type (see Document Serialization). The reasons list above is therefore one count byte and 32 raw bytes per identifier, and an integer element bounded 0..100 takes one byte. Since the stored bytes depend on the element's type, a contract update may not change how an element encodes: raising an integer element's maximum (or adding an enum value) past its width, or unpinning a fixed-size byte array element, is refused with DocumentTypeUpdateError. A longer maxLength, a larger maxItems or a raised maximum that keeps the width are accepted. Identifier and byte array elements are conversion paths (reasons[], find_identifier_and_binary_paths 1), so a document built from JSON or a value map converts every element, as it converts a scalar identifier. ExtendedDocument::set_untrusted converts every member of a list set at such a path. From protocol version 14 a property name and a document type name are word characters only (^[a-zA-Z0-9_]{1,64}$): every earlier generation also admitted -, which the path syntax (a.b, list[]) was never written for, and a census of every contract on mainnet and testnet found none using it. Nothing is written per element, so a typed array cannot be an index property (InvalidIndexPropertyTypeError), an indexOnly terminal or entry payload property, or one side of a where entry.

In Rust a typed array parses to DocumentPropertyType::TypedArray(TypedArrayProperty), whose item_type is the DocumentPropertyType the items schema parses to as a property schema (try_from_value_map with the contract's parsing options). The parse is the versioned parse_typed_array (None before protocol version 14, where an array that is not a byte array is refused as it always was). The older DocumentPropertyType::Array variant, whose elements are an ArrayItemType in their own length-prefixed encoding, is never produced by the parser.

References on the Elements

An identifier element may carry a refersTo declaration, which every element of the list then declares. The moderation charters' reasons refers to reason documents this way:

"items": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "refersTo": { "type": "permanentDocument", "documentType": "reason" }
}
  • The declaration takes every target and key a single identifier property's refersTo takes (identity, contract with contractRequirements, token, and permanentDocument and deletableDocument with contractId, documentType, findBy and where, and inList on a permanentDocument), with the same checks at contract registration, except identityPublicKey in either form: its keyIdProperty names one sibling key id, and the identityProperty form sits on the key id itself, neither of which can pair with many elements, so an element may not declare it. The declaration belongs on the items; on the array itself it is refused.
  • When a document is created or replaced each element is checked as a single reference is, in list order: the target must exist, a referenced contract must meet the contractRequirements, a referenced document's type must be deletable or not as declared, and each where entry must hold. The referring value of an entry is still a property of the referring document or its $ownerId, the same for every element; the referenced side is a property of that element's referenced document. The first element that fails refuses the write with the error a single reference gives (ReferencedEntityNotFoundError 40120, ReferencedDocumentPropertyMismatchError 40127, ReferencedContractRequirementNotMetError 40135 and so on), whose path names the element by its list path: reasons[2] for the third. An empty or absent list checks nothing. Registration errors name the declaration submittedCharter.reasons[].
  • A replace follows the rules of a single reference, element by element. A changed list re-validates the elements the stored list did not hold (the replace action carries the stored value of each changed property, stored_changed_values); the ones it held are unchanged references and are left alone, as an unchanged single reference is. Every element is re-validated when a property a where or findBy reads changed, and on every replace when a where entry is valued "$ownerId", the elements are deletableDocument references, or they are contract references with an owner requirement on a document type whose documents can be transferred or traded. An element repeating an earlier one of the same list is not fetched again.
  • Every read is billed as a single reference's is. A foreign contract holding the referenced document type is fetched once per list, not once per element. Contract registration caps the references one document of a type can carry at SystemLimits::max_references_per_document (256), counting one per property with refersTo (an identifier, or a key id carrying a key reference), one for the type's ownerRefersTo and maxItems per typed array of referencing elements: maxItems alone would let a type declare many lists of up to 1024 references each, and each one is a read when a document is written.
  • An immutable property may not hold a deletableDocument reference a replace could not clear: a typed array of them, at the top level or inside an immutable object, or a single one inside an immutable object. Every replace re-validates them, so once a target is deleted the property would have to change, which an immutable property cannot. A single deletableDocument reference that is itself the immutable top-level property has a way out, a replace may remove it once its target is gone, and that exception reads the one identifier the removed top-level property held. That property may not also be listed under immutableAllowSetting: once it is cleared, the allowance would let the next replace set it to another document.
  • A changed element refersTo is an incompatible schema change on contract update, as a changed refersTo on a scalar is.

In Rust the element parses to DocumentPropertyType::IdentifierWithReference(target) inside item_type, through the same versioned apply_property_reference a scalar identifier goes through. DocumentPropertyType::reference() reports a property's declaration as PropertyReference::Value(target) for a scalar and PropertyReference::Elements(target) for a typed array; the registration check (validate_data_contract_references) and the write-time check (validate_document_references) both enumerate references through it.

Distinct Identifier Properties

Protocol version 14 adds the property-level distinctFrom keyword, a pure structure rule on identifier properties: the property's value must differ from the value of a named property of the same document, or from the document's $ownerId. It sits next to the reference keywords (refersTo and its where, which binds a property to another document's values) but reads nothing beyond the transition being written.

"delegateId": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "distinctFrom": "$ownerId",
  "position": 0
},
"backupId": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "distinctFrom": "delegateId",
  "position": 1
}

The value is "$ownerId" or the dotted path of another property of the document type ("meta.reviewerId" for a nested one). The parser (generation 3, meta-schema v3) checks the declaration when a contract enters the chain, on registration and on update:

  • The keyword is only allowed on identifier properties, enforced by the same dependent schema shape that restricts refersTo.
  • A named property must exist on the document type, must itself be an identifier (the only kind the value can be compared with), and must not be the declaring property. $ownerId needs no check; no other system property is accepted.
  • On contract update a changed, added or removed distinctFrom is an incompatible schema change, like a changed refersTo.
  • A typed array of identifiers declares it on its items, and every element must then differ from the named value; the declaration is refused on the array itself and on elements of any other type.

Enforcement lives in the structure validation of the document create and replace actions (create structure generation 1, introduced at protocol version 14, and replace structure generation 0, extended in place: the call is inert before 14, where no property can carry the keyword), after the schema validation of the document's properties, so every value compared is already a 32-byte identifier. The check reads the transition's data and the owner id it carries and never touches Drive; the declaring properties come from a list the parser built (distinct_from_fields), so a type without declarations costs nothing. An equal pair fails the write with DocumentPropertyNotDistinctError (basic code 10419), which names the document type, the property and what it collided with. When the named property is absent from the document there is nothing to differ from, so the rule passes.

A transfer or purchase changes $ownerId without touching the data, so the transfer and purchase structure validations (generation 0, extended in place: the call is inert before protocol version 14, where no property can carry the keyword) judge the stored document against its new owner: a transfer to, or a purchase by, the identity a $ownerId-distinct property names is refused with the same error. Price updates change neither owner nor data and are not judged.

In Rust the declaration is DocumentProperty::distinct_from (Option<DistinctFrom>, absent on every property parsed before protocol version 14), the document check is DocumentTypeV0Methods::validate_distinct_from_properties, and DistinctFrom::violation judges one value on its own, which is how the elements of a typed array are judged one by one.

Encrypted Properties (encryptedFor)

A byte array property may hold ciphertext that only one identity can read. Before protocol version 14 the contract said nothing about it, so every wallet had to learn the recipe (whose keys, which scheme, where the IV sits) from documentation or a side channel. From protocol version 14 the property declares it with the encryptedFor keyword, and wallets and SDKs read the recipe from the contract.

"encryptedMessage": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 1040,
  "encryptedFor": {
    "recipient": "recipientId",
    "recipientKey": "recipientKeyId",
    "senderKey": "senderKeyId",
    "scheme": "ecdh-secp256k1-aes256-cbc"
  },
  "position": 4
}

All four keys are required. recipient is the dotted path of an identifier property of the same document type whose value is the recipient identity's id, or $ownerId for a message the writer encrypts to themself. recipientKey and senderKey are dotted paths of integer properties of the same document type carrying the recipient's and the sender's identity key ids; each must declare minimum at least 0 and maximum at most 4294967295, read from the schema itself, so the rule holds whatever sizedIntegerTypes the contract sets. scheme is a closed set with one member today.

The parser (generation 3, meta-schema v3) admits the keyword on byte array properties only, never on an identifier (contentMediaType set) or any other type, and checks at contract registration that the three named properties exist with those types, that none of them is transient or sits inside a transient object (a transient value is stripped before storage, which would leave the stored ciphertext without its recipe), and that the byte array's own maxItems can hold the scheme's shortest ciphertext. A contract update that adds, removes or changes an encryptedFor declaration is an incompatible schema change (IncompatibleDocumentTypeSchemaError, 10246): documents already written under the old recipe could not be read under the new one. Contracts parsed before protocol version 14 ignore the keyword entirely.

The ecdh-secp256k1-aes256-cbc layout

This is the scheme the dashpay contact request already uses for encryptedPublicKey and encryptedAccountLabel (DIP-15), implemented in packages/rs-platform-encryption:

  1. The shared key is the libsecp256k1 ECDH of the sender's private key and the recipient's public key: SHA256(parity || x) of the product point, where parity is 0x02 for an even y and 0x03 for an odd one. The sender's key is the identity key with the id the senderKey property carries, on the document's $ownerId identity; the recipient's key is the one with the id the recipientKey property carries, on the identity the recipient property names (the owner itself for $ownerId). Either side derives the same 32 bytes from its own private key and the other's public key.
  2. The writer draws a random 16-byte IV.
  3. The value is the IV followed by the plaintext encrypted with AES-256-CBC under the shared key and that IV, with PKCS7 padding.

So a ciphertext is 16 + 16 * ceil((len(plaintext) + 1) / 16) bytes: at least 32, always a multiple of 16. The reader splits the first 16 bytes off as the IV, derives the shared key from its own private key and the sender's public key, and decrypts the rest.

What consensus checks, and what it cannot

Consensus sees bytes, not keys. On every document create and replace, after the JSON schema validation of the document's properties, the structure validation (create structure generation 1, introduced at protocol version 14, and replace structure generation 0, extended in place: the call is inert before 14, where no property can carry the keyword) walks the document type's declared properties and, for each one the transition supplies, checks that its length is at least the scheme's IV plus one block and a multiple of the block length. A value that is not refuses the transition with InvalidEncryptedPropertyShapeError (basic code 10420), which names the property path, the scheme and the lengths involved. No state is read; the check runs in the mempool as well as in the block. The JSON schema's own minItems and maxItems are checked first, so a value outside them (a lone 16-byte IV against minItems: 32, say) is refused with the schema's error rather than 10420; the shape check only sees values the bounds already admit.

Nothing else is verifiable on chain: not that the bytes decrypt, not that they decrypt under the keys the document names, not that the named key ids exist on the identities or have an encryption purpose, and not that the plaintext is what the document type means it to be. A writer can store any 32 bytes. Whether the keys exist and are of the right kind is what the reference keywords are for: a refersTo of type identityPublicKey with keyIdProperty on the recipient property makes consensus check that the recipient's key exists, and encryptedFor neither duplicates nor requires it. Readers must treat a value that fails to decrypt as a bad message, not as a protocol violation.

In Rust the declaration is DocumentProperty::encrypted_for (Option<EncryptedFor>), listed per document type by DocumentTypeV0Getters::encrypted_properties(), and the shape check is DocumentTypeBasicMethods::validate_encrypted_property_shapes(), versioned on the validate_encrypted_property_shapes method slot (None before protocol version 14, which is what keeps the in-place replace call inert). In JavaScript, contract.documentTypeEncryptedProperties(name) and contract.documentEncryptedProperties expose the same declarations, and the shape error reaches an app as DocumentEncryptionErrorCode.InvalidEncryptedPropertyShape.

Clients encrypt and decrypt through the declaration rather than a per-contract recipe. The Rust SDK's dash_sdk::platform::encrypted_for module has encrypt_property, which writes the ciphertext and both key id properties, and decrypt_property. EncryptedPropertyEnvelope::read names the identities and key ids a reader needs. select_encryption_keys picks the keys the document type's identityPublicKey references demand through their keyRequirements. In JavaScript the same helpers are sdk.encryptedFor.encrypt, decrypt and envelope (WasmSdk.encryptDocumentProperty, decryptDocumentProperty and encryptedPropertyEnvelope). The layout has no authentication tag, so a wrong key fails the padding check except about once in 256 attempts, when it yields garbage.

Byte Caps on Strings (maxBytes)

Protocol version 14 adds the property keyword maxBytes, a bound plain JSON Schema cannot count: the most bytes a string may take in UTF-8. maxLength counts characters, and a character is up to four bytes, so maxLength: 4096 alone admits values up to the 5120-byte cap every field has (SystemLimits::max_field_value_size), not 4096 bytes.

"description": {
  "type": "string", "minLength": 1, "maxLength": 4096,
  "maxBytes": 4096,
  "position": 1
}

The keyword goes on a string property, or on the items of a typed array of strings, where it bounds every element; it is refused on the array itself and on elements of any other type. It is an integer from 1 to 65535 and no lower than minLength, since a string of minLength characters is at least that many bytes. On contract update it moves like maxLength: raising or removing it is compatible, adding or lowering it is not.

The parser (generation 3, meta-schema v3, apply_max_bytes) folds the bound into the string's StringPropertySizes::max_bytes, next to max_length, so the sizes the type reports take it into account: max_byte_size is the smaller of maxBytes and four bytes a character, and random documents stay within it.

The check runs where the JSON schema validation of a document's properties runs, DataContract::validate_document_properties, right after it: on every document create and replace, and in every client that validates a document before sending it. A longer string is refused with DocumentPropertyMaxBytesExceededError (basic code 10421), which names the property (tags[2] for an element) and both lengths. The document validation (version 0, extended in place) is inert before protocol version 14, where no string carries a byte cap and the validate_max_bytes method slot is None. In Rust the check is DocumentTypeBasicMethods::validate_max_bytes_properties(); in JavaScript the error reaches an app as DocumentMaxBytesErrorCode.MaxBytesExceeded.

Generated Properties (generatedFrom)

Protocol version 14 adds the property keyword generatedFrom: the platform generates a string property's value with a system function of other properties of the same document, its params. The first system functions change the case of a string (lowercase, uppercase, capitalize, camelCase, snakeCase) or apply the DPNS domain rule (normalizedLabel is label lowercased, with o, i and l replaced by 0, 1 and 1), so any contract can build a unique index that treats look-alike names as one.

"normalizedLabel": {
  "type": "string", "maxLength": 63,
  "generatedFrom": {
    "function": "sys.stringTransformations.homographSafeASCII",
    "params": ["label"]
  },
  "position": 1
}

The functions are a closed list of system functions (SystemFunction), named under sys. so that functions a contract may bring later can be told apart by name; each declares how many params it takes. SystemFunction is an enum of namespaces, each an enum of its own in a module of its own: SystemFunction::StringTransformation(StringTransformation) for sys.stringTransformations, in system_function::string_transformations, whose six functions each take one string. lowercase, uppercase and capitalize change the case of ASCII letters; camelCase and snakeCase split the string into words (at every ASCII character that is neither a letter nor a digit, and before an ASCII uppercase letter that starts a word) and join them as helloWorld or hello_world; homographSafeASCII maps A to Z to lowercase, then o to 0 and i and l to 1. Every one changes ASCII characters only and keeps every other character, with no Unicode table, because Unicode case mappings differ between releases of the standard library, and two nodes must never generate different values; every one gives back its own output unchanged, and a test pins that the meta-schema lists exactly the functions the parser knows. On ASCII homographSafeASCII equals convert_to_homograph_safe_chars, the function the DPNS trigger uses; a test pins that over every ASCII character and over strings from the DPNS alphabets. It refuses nothing: the characters a value may hold are each param's pattern's to decide, and the generated property needs no pattern of its own, since every value it holds is generated from params that passed theirs. A param is a property path for now (GenerationParam::Property); literals, system values and nested calls can be added later without changing what parses today.

The parser (generation 3, meta-schema v3, apply_generated_from) reads the declaration onto DocumentProperty::generated_from (Option<GeneratedFrom>, absent on every property parsed before protocol version 14) and checks that params lists as many params as the function takes. validate_generated_from_declarations checks every param against the other properties on every parse: another string property, not generated itself, not transient nor inside a transient object (and neither is the declaring property), and inside every object that holds the declaring property. The meta-schema refuses the keyword beside $ref, whose definition replaces every keyword written next to it. On contract update a changed, added or removed generatedFrom is an incompatible schema change, and a property the update adds may declare it only when one of its params is new too (validate_update 1 refuses one whose params all existed with DocumentTypeUpdateError, 40212: documents stored before the update were never generated). The declaring paths and their declarations are cached on the document type (DocumentTypeV2Getters::generated_from_fields), so a write to a type without declarations pays nothing.

Three methods do the work at write time:

  • DocumentTypeBasicMethods::fill_generated_properties writes every declared property the document leaves out while supplying every param. The action transformers of document create, replace and index-only delete call it first, before the contest resolution and every check read the data, so the stored document, its indexes and its contest all hold the generated value. Document::try_from_create_transition and try_from_replace_transition call it too, and so does index_only_transition_entry_path_query, the one builder the prover and the verifier share for index-only entries, so proofs are built and checked against the document the platform stored. DocumentTypeBasicMethods::data_as_stored returns a transition's data with the generated properties written into a copy, or borrows it as it is on a type that declares none; the document subscription filter (DriveDocumentQueryFilter::matches_document_transition) reads a create's and a replace's data, and an index-only delete's values, through it, so a subscription on a generated property sees the value the platform stores.
  • DocumentTypeBasicMethods::regenerate_generated_properties is the client-side twin: it sets every declared property to what its params generate, replacing a value the document holds and removing it when a param is absent. The transition builders (from_document of create, replace and index-only delete), the SDK's contest fund lookup and the property-constraint pre-checks of the JavaScript and FFI SDKs call it, so a document fetched, edited and sent back carries the value of its new params rather than the stale one the platform would refuse, and its contest is detected from it. Random documents call it too, after drawing the params of every generated property they drew.
  • DocumentTypeBasicMethods::validate_generated_from_properties runs in DataContract::validate_document_properties, after the JSON schema and maxBytes: a declared property must equal what its function generates from its params, and be absent when a param is. A document that repeats a key in an object on the way to the property or to a param is refused too: the schema validation and the stored document keep the last of repeated keys, where the path reads the platform generates from find the first. A property that does not pass refuses the write with DocumentPropertyNotGeneratedError (basic code 10424). A document that arrived has been generated, so on the platform the check only refuses a value the client sent, a value sent without its params, or a repeated key.

Every call site was extended in place and is inert before protocol version 14: the apply_generated_from, fill_generated_properties and validate_generated_from slots are None there, and the meta-schemas refuse the keyword. In JavaScript the error reaches an app as DocumentGeneratedFromErrorCode.DocumentPropertyNotGenerated.

Property Constraints (propertyConstraints)

Protocol version 14 adds the doctype-level propertyConstraints keyword: named rules the properties of every created or replaced document must meet, where JSON Schema can only bound one property at a time. Each rule is a condition: a comparison of two integer expressions, a test of whether an integer expression takes one of listed values, a comparison of a string property with string constants, a test of whether the document holds a property, or anyOf, allOf or not over conditions.

"propertyConstraints": {
  "depositCoversOrder": {
    "lessThanOrEqual": [
      { "multiply": [{ "add": ["price", "fee"] }, "quantity"] },
      "deposit"
    ]
  },
  "wholeLots": { "equal": [{ "modulo": ["quantity", 10] }, 0] },
  "minimumOrder": {
    "greaterThanOrEqual": [{ "multiply": ["price", { "ifAbsent": ["quantity", 1] }] }, 100]
  },
  "feeWaivedOrAtLeastTen": {
    "anyOf": [{ "equal": ["fee", 0] }, { "greaterThanOrEqual": ["fee", 10] }]
  },
  "discountGivenAboveZero": {
    "anyOf": [{ "absent": "discount" }, { "greaterThan": ["discount", 0] }]
  },
  "tieredFee": { "in": ["fee", [0, 10, 25, 50]] },
  "closedNeedsClosedAt": {
    "anyOf": [{ "notEqual": ["status", { "const": "closed" }] }, { "present": "closedAt" }]
  }
}

The first rule reads ((price + fee) * quantity) <= deposit, feeWaivedOrAtLeastTen reads fee == 0 || fee >= 10, discountGivenAboveZero lets an offer leave its discount out but not give a discount of 0, tieredFee holds the fee to four tiers, and closedNeedsClosedAt says a closed offer carries a closedAt. A rule's name is 1 to 64 letters, digits or underscores, and the rule is a condition, an object with one key:

  • a comparison, equal, notEqual, lessThan, lessThanOrEqual, greaterThan or greaterThanOrEqual, listing the left and the right expression;
  • { "in": [expression, [values]] }, holding if the integer expression takes one of two or more distinct integer values. It says what an anyOf of equals says, in one node per value instead of three, so a set of up to 30 values fits the node limit where the anyOf fits 10. A value is a literal, never a path or an expression;
  • a string comparison: { "equal": [path, { "const": "closed" }] } or notEqual, with the constant on either side; { "notEqual": ["fromCurrency", "toCurrency"] }, two bare paths that both name string properties, which compares their strings; or { "in": [path, ["open", "pending"]] }, whose values are two or more distinct strings. The path names a string property, typically one with an enum. A string on its own is a path, so a constant is written as { "const": ... }, while the values an in lists are literals and need no wrapper. Strings are only compared for equality, never ordered or used in arithmetic. A string property the document leaves out equals no constant and no other string property, not even one also left out, so notEqual holds for it and equal and in do not, unless { "ifAbsent": [path, "open"] } gives it a string default, which it then reads as (it may stand wherever the bare path does, and makes the comparison one of strings); present and absent test it directly. When the property declares an enum, every constant compared with it must be one of the enum's values, so a misspelling is refused at registration rather than making the rule quietly never hold;
  • an identifier comparison, the same three forms for identifier properties, those declaring refersTo included: { "equal": ["paymentToken", { "const": "<base58>" }] } or notEqual, { "notEqual": ["buyerId", "sellerId"] }, or { "in": ["paymentToken", ["<base58>", "<base58>"]] }. Constants are base58 identifiers of 32 bytes, checked at registration, and compared by their bytes, whatever form the document gives the identifier in. An identifier property the document leaves out equals no identifier, not even another one left out; identifiers take no ifAbsent default and are never ordered. $ownerId, the document's owner, is an identifier operand too: { "equal": ["authorId", "$ownerId"] } holds the author to the owner, and { "in": ["$ownerId", ["<base58>", ...]] } lets only the identities listed own a document of the type. It is no property, so present or an integer operand refuses it, and comparing it with itself is refused. Since a transfer and a purchase give the document a new owner, each is judged against the rules reading $ownerId, with the new owner, and refused when it would break one; an indexOnly type refuses such a rule, since its deletes carry no owner;
  • { "startsWith": [text, affix] } and { "endsWith": [text, affix] }, holding if the first string starts or ends with the second, byte for byte with no case folding: each side a { "const": ... }, a string property or an ifAbsent string default, at least one a property, never the same one twice. { "startsWith": ["url", { "const": "https://" }] } holds a link to https, and { "startsWith": ["path", "parentPath"] } a path under its parent's. A string property left out without a default takes no string, and the condition does not hold for it; a constant tested against a property that declares an enum must start or end one of its values;
  • { "contains": [path, value] }, holding if the typed array property at the path holds an element equal to the value, looked for as the array's elements are: an integer expression among integers, a string constant, string property or ifAbsent string default among strings, an identifier constant, identifier property or $ownerId among identifiers. { "not": { "contains": ["labels", { "const": "used" }] } } refuses a "used" label, and { "contains": ["participants", "$ownerId"] } holds the owner to the participants (so a transfer or purchase to a non-participant is refused). An array the document leaves out holds nothing, a string or identifier property it leaves out is among no elements, and a string constant must be one of the elements' enum values when they declare one;
  • { "present": path }, holding if the document holds the property, and { "absent": path }, holding if it leaves it out (a property set to null counts as left out, and so does an object none of whose members is present, such as {}, which a stored document does not keep). An operand reads a property the document leaves out as 0, so only these tell "not given" from "given as 0". They may name a property of any type, an object or a member of one included, since they read no value;
  • { "anyOf": [...] }, holding if at least one of two or more conditions holds;
  • { "allOf": [...] }, holding if every one of two or more conditions holds;
  • { "not": condition }, holding if its one condition does not;
  • { "ifThen": [a, b] }, holding if b holds whenever a does: b is evaluated only when a holds, so { "ifThen": [{ "greaterThan": ["discount", 0] }, { "greaterThanOrEqual": [{ "divide": ["price", "discount"] }, 10] }] } never divides by zero, and a fault in either breaks the rule. It says what { "anyOf": [{ "not": a }, b] } says, in one node fewer. The two may not be alike;
  • { "ifThenElse": [a, b, c] }, holding if b holds when a does and c holds when it does not; only the branch a selects is evaluated. { "ifThenElse": [{ "greaterThanOrEqual": ["price", 1000] }, { "lessThanOrEqual": ["fee", 50] }, { "lessThanOrEqual": ["fee", 10] }] } allows a higher fee on an expensive offer. No two of the three may be alike;
  • { "notIn": [expression, [values]] }, an in negated, listed the same way and in as many nodes: { "notIn": ["fee", [7, 13]] } refuses two fees.

Conditions nest: { "not": { "allOf": [{ "equal": ["price", 0] }, { "greaterThan": ["quantity", 10] }] } } refuses a free order of more than 10. An anyOf or allOf may not list two alike conditions, nor hold one of its own kind directly (it says what one flat list says), and a not may not hold a not or a notIn directly. An expression is one of:

  • an integer value (100; a float with no fractional part, 100.0, reads as that integer, as the meta-schema's integer type admits it);
  • a string, the dotted path of an integer or boolean property of the document type ("price", "meta.total", "waiveFee"), whose value it takes, 0 when the document leaves the property out. A boolean reads as 1 for true and 0 for false, so { "equal": [{ "multiply": ["waiveFee", "fee"] }, 0] } says a waived fee is 0;
  • { "ifAbsent": [path, value] }, the property's value, or value when the document leaves it out (an integer value here; a string value gives a string property a default in a string comparison instead);
  • { "add": [...] } or { "multiply": [...] } over two or more operands;
  • { "min": [...] } or { "max": [...] }, the least or greatest of two or more operands, every one evaluated (a fault in any breaks the rule), and { "abs": a }, the absolute value of one: { "lessThanOrEqual": ["fee", { "max": [10, { "divide": ["price", 10] }] }] } caps a fee at 10 or a tenth of the price, whichever is more, and { "lessThanOrEqual": [{ "abs": { "subtract": ["a", "b"] } }, 5] } keeps two values within 5;
  • { "subtract": [a, b] }, { "divide": [a, b] }, { "modulo": [a, b] } or { "power": [a, b] };
  • a size: { "length": path }, the characters of a string property (counted as maxLength counts them), { "byteLength": path }, its UTF-8 bytes (as maxBytes counts them), or { "count": path }, the items of an array property or the bytes of a byte array property. Where maxLength, maxBytes and maxItems bound one property by a fixed number, a size can be compared with another property or bounded only under a condition: { "lessThanOrEqual": [{ "count": "tags" }, "maxTags"] } holds a list to its own limit, and { "anyOf": [{ "greaterThan": ["fee", 0] }, { "lessThanOrEqual": [{ "length": "title" }, 20] }] } keeps a free listing's title short. A property the document leaves out or sets to null has size 0, and a size never breaks a rule by itself (a value of another type would read as 0 too, but the schema validation refuses it first);
  • { "countPresent": [path, path, ...] }, how many of two or more properties, no two alike, the document holds, each as present tests it (any type, null and an object with no member present counting as absent), so a rule bounds how many of a group are set: { "equal": [{ "countPresent": ["email", "phone", "handle"] }, 1] } asks for exactly one, { "lessThanOrEqual": [..., 1] } for at most one, and { "in": [..., [1, 2]] } for one or two. It is one node plus one per path, where the same bound written with present takes an anyOf and a not of every pair;
  • a system time or height: "$createdAt", "$updatedAt" and "$transferredAt", block times in milliseconds, and each with BlockHeight or CoreBlockHeight appended, the Platform and Core block heights: those of the document's creation, of its last create, replace or price update, and of its last create, transfer or purchase. A rule may read one only on a type that records it by listing it in required, so every stored document holds it; it takes no ifAbsent, and an indexOnly type, whose deletes carry none, reads none. { "lessThanOrEqual": [{ "subtract": ["endsAt", "$createdAt"] }, 604800000] } keeps a listing to a week from its creation, and { "lessThanOrEqual": ["$updatedAt", "endsAt"] } refuses a replace or a price update after it ends;
  • a total read from state: { "countOf": [type] } or { "countOf": [type, filter] }, how many documents of a type of the same contract there are, or how many match the filter, and { "sumOf": [type, property] } or { "sumOf": [type, property, filter] }, the total of an integer property over them. A filter maps each key, a property of the counted type or $ownerId, to the value it must take, read from the document being written: one of its properties, $ownerId, $id (its id, so { "countOf": ["vote", { "pollId": "$id" }] } counts the votes naming it), an integer or a { "const": ... }. The total is the one a count or sum tree keeps, as it will be once the write is done (the document itself counted when the type is its own, at its new values, and moved to its new owner by a transfer or purchase), so { "lessThanOrEqual": [{ "countOf": ["listing", { "$ownerId": "$ownerId" }] }, 10] } on listing keeps every owner at ten listings or fewer. A whole-type total needs documentsCountable or documentsSummable, and a filtered one an index of the counted type whose properties are exactly the filter's keys, countable or summing the property. A rule is judged only when its own type is written, so a fact about another type can go stale after the write.

A JSON number is always a value and a string always a path, so a property named 100 is not confused with the number, and the rule is a tree the meta-schema can check rather than a string with precedence rules to parse. Consensus holds nothing but this tree; an SDK may offer an infix spelling that compiles to it.

The arithmetic is exact over i128. Operands are evaluated left to right, and every intermediate result must fit: an overflow, a divisor of 0, a negative exponent or a property value that is not an integer (a float with no fractional part passes the schema's integer type) breaks the rule instead of wrapping or truncating. divide and modulo are Euclidean, so the remainder is never negative and the quotient is the one that goes with it (-7 by 2 is -4 remainder 1); for operands that are not negative this is ordinary integer division. 0 to the power 0 is 1. There are no floats: a number property cannot be read, which keeps every node's result bit-identical.

Conditions are checked in declared order and no further than the outcome needs: a comparison evaluates its left side, then its right; anyOf stops at the first condition that holds and allOf at the first that fails. A fault in a condition that is checked breaks the rule whatever the others would say, and not does not turn it into a pass. So an earlier condition guards a later one: { "anyOf": [{ "equal": ["b", 0] }, { "equal": [{ "divide": ["a", "b"] }, 2] }] } holds for a b of 0 without dividing by it, while the same two conditions the other way round divide by zero and break the rule.

The parser (generation 3, meta-schema v3) checks the keyword on every parse, stored contracts included: the shape, that every path an operand reads names an integer or boolean property of the type, every path length or byteLength measures a string property, every path count counts an array or byte array property, every system time or height a rule reads is one the type lists in required (and an indexOnly type reads none), every path compared with a string names a string property (and every constant compared with one that declares an enum is one of its values) (a nested one by its dotted path) and every path present, absent or countPresent tests names a property of the type, and that neither is transient nor inside a transient object (a transient value is never stored, so a stored document could not be held to the rule), that every comparison and in reads at least one property (a constant one would make its rule, or an anyOf around it, hold for every document or for none), that no in lists a value twice and no countPresent a path twice, that no anyOf or allOf holds one of its own kind directly and no not a not, that no literal divisor is 0 and no literal exponent negative, and that no condition or operand nests deeper than MAX_PROPERTY_CONSTRAINT_PARSE_DEPTH (64), a constant that keeps a parse without full validation from recursing without bound and that no registrable rule comes near. Under full validation, when a contract registers or updates, it also holds the limits: at most SystemLimits::max_property_constraints rules per type (16) and max_property_constraint_nodes nodes per rule (32), counting every comparison and logical operator, every in and each value it lists, every const, every present or absent, every arithmetic operator and every operand (a size is one, a countPresent one plus one per path), that no anyOf or allOf lists the same condition twice (conditions that parse alike, so 1 and 1.0 are the same value), and that the rules of one type read at most max_property_constraint_aggregates (4) distinct totals. Once every document type of the contract is parsed, a registration also checks every countOf and sumOf: the type it counts is one of the contract's and not indexOnly, nor the declaring type when it has a contested index (a document a contest awards is stored without the rules judged), a tree of it keeps the total, the filter's keys and values are integers, strings or identifiers of the same kind, and every property a value reads is required. The rules are fixed when the document type is created: adding, removing or changing one is an incompatible schema change (IncompatibleDocumentTypeSchemaError, 10246), since stored documents were judged against the rules as they were.

Enforcement lives in DataContract::validate_document_properties (generation 0, extended in place: the call is inert before protocol version 14, where validate_property_constraints is None), after the schema validation. Document create and replace structure validation call it, so consensus applies the rules, and so does every client that validates a document before sending it. The rules are checked in name order against the document's properties, which for a replace is the whole document, and the first one broken fails with DocumentPropertyConstraintViolatedError (basic code 10422), naming the document type, the rule and why: the rule does not hold, or evaluating it overflowed, divided by zero, raised to a negative power or read a value that is not an integer. The check changes nothing stored. It reads state only for the countOf and sumOf totals, which the document batch transformer reads when it builds the action (Drive::fetch_property_constraint_aggregate, billed with the write) and hands the check in DocumentSystemValues::aggregates; the limits bound its cost. $ownerId and the system times and heights read the values of the document version being written (validate_document_properties takes them as DocumentSystemValues): on a create the writer and the block's time and heights, on a replace the writer, the stored creation and transfer values and the block's as the update. A client passes what it knows: an owner it does not know equals no identifier, and a rule reading a time, a height or a total it is not given is not judged. Consensus reads every total the rules it judges read (DocumentSystemValues::aggregates is Some), so one missing there is an error in the code building the write, never a skipped rule. The SDK pre-checks use the device clock for the times a write records, and leave a rule reading a block height unjudged, since the height is unknown until the block. Transfers and purchases change no property, only the owner and the transfer's time and heights, so only the rules reading those are judged again, with the new values (DocumentTypeV0Methods::validate_property_constraints_for_system_change, next to the distinctFrom check). Price updates change only the update's time and heights, and are judged against the rules reading those the same way.

In Rust the rules are DocumentTypeV2Getters::property_constraints (a map from name to PropertyConstraint: a comparison, an in, a string comparison (TextCompare, TextIn), a present or absent, or an anyOf, allOf or not of them, empty on types that predate the keyword; property_reads lists what a rule reads and how: by value, by presence, by size (Length, Count) or in a comparison of strings or identifiers, system_reads the system times and heights it reads, and aggregate_reads the totals, each an AggregateRead), each rule's holds and violation evaluate it against a document's data and DocumentSystemValues, and the document check is DocumentTypeV0Methods::validate_property_constraints.

Delete rules (deleteConstraints)

The same grammar gates the owner's delete. deleteConstraints is a doctype keyword of named rules the stored document must meet for its owner to delete it (see Deletion):

"deleteConstraints": {
  "noVotes": { "equal": [{ "countOf": ["vote", { "pollId": "$id" }] }, 0] }
}

The parser (apply_delete_constraints_v0, generation 3, called by apply_property_constraints so one dispatch on parse_property_constraints covers both keywords) reads the rules with parse_delete_constraints, the same parse as parse_property_constraints under another key, checks what they read with validate_property_constraint_reads and, under full validation, holds them to the same SystemLimits, counted apart from propertyConstraints. On every parse it refuses the keyword on a type whose owner never deletes a stored document: canBeDeleted: false, "onlyWhenConsumed" or indexOnly. validate_property_constraint_aggregates checks their countOf and sumOf with the contract's other types, except that a delete rule may total its own contested type, since the total it reads is the stored one. DocumentReferenceLookup::referenced_side_error refuses a consume into such a type, which would delete without a delete transition. The rules are frozen on update by the same compatibility rule as propertyConstraints (10246).

Enforcement is document delete state validation 1 (protocol version 14, DRIVE_ABCI_VALIDATION_VERSIONS_V10): the checks of version 0 (the base transition, the stored document, its owner), then, for a type with rules, read_delete_constraint_aggregates reads the totals (billed, each as it will be once the document is gone: the stored total less what the document adds to it) and DocumentTypeV0Methods::validate_delete_constraints judges each rule in name order on the stored document's properties and DocumentSystemValues::of_document. The first rule broken refuses the delete with DocumentDeleteConstraintViolatedError (state code 40147), naming the document, the rule and why, so the delete is paid and the nonce spent. In Rust the rules are DocumentTypeV2Getters::delete_constraints, and a filter value $id is AggregateBinding::Id, which AggregateRead::filter_values reads from the document's id.

Rules and Guidelines

Do:

  • Always serialize and deserialize documents using their document type definition. The type determines field layout.
  • Use is_equal_ignoring_time_based_fields() when comparing documents for validation purposes.
  • Use increment_revision() rather than manually manipulating the revision field -- it handles overflow checking.
  • Access properties through the accessor traits, not by reaching into the inner DocumentV0 struct.

Do not:

  • Assume a document carries its contract reference. The contract and document type are always passed as separate arguments.
  • Manually construct document IDs. Use Document::generate_document_id() with proper entropy, the nonce of the create transition and the network's platform version.
  • Rely on the ID of a document that has not been sent yet. It is a placeholder until the create transition is built.
  • Treat serialized document bytes as self-describing. Without the document type schema, the bytes are meaningless.
  • Set time-based fields from client code. The platform sets created_at, updated_at, block heights, and similar fields during state transition processing.

Document Time To Live

A document type may declare a time to live. Every document of such a type is deleted by the platform once its time to live has passed, whoever owns it and whatever the type says about who else may delete it. Its owner pays, when the document is written, for the time the document will occupy the state rather than for perpetual storage, and prepays its deletion. Available from protocol version 14.

"note": {
  "type": "object",
  "ttl": 1209600,
  "properties": { "text": { "type": "string", "maxLength": 280, "position": 0 } },
  "required": ["$createdAt", "text"],
  "additionalProperties": false
}

Documents of note above are deleted two weeks (1,209,600 seconds) after their creation.

This is a different feature from the ttl key of a timeRange index (see Time-Range Index TTL), which expires index entries and leaves the document in place.

Semantics

  • A document expires at its $createdAt plus ttl seconds. $createdAt is set from block time when the document is created, so nothing the writer sends moves the expiry, and every document of a type created in one block expires at the same time.
  • A replace, a transfer, a purchase or a price update never moves the expiry: $createdAt never changes. A buyer of an expiring document buys what is left of its life.
  • After each block's state transitions the platform deletes the expired documents, oldest first, at most max_document_expirations_per_block per block (128 at protocol version 14), and no more than max_document_expiration_weight_per_block (1,024) in weight, where a document weighs 1 plus the index levels of its type. The rest wait for the next block. A state transition in the block a document expires in still sees it; a document can therefore outlive its expiry by the part of a block before the cleanup, and by more when a backlog builds.
  • The expiry belongs to the document: it is its own $createdAt plus the type's ttl, both fixed, and the platform reads it from there. The expirations tree is only the index the cleanup finds documents through.
  • From its expiry on, a document can no longer be replaced, transferred, bought or repriced, and a moderator can no longer restore it: each is refused, paid, with DocumentExpiredError (40140), whether or not the cleanup has reached the document. Its owner may still delete it where canBeDeleted allows, which only removes it sooner. Until the cleanup deletes it, it can still be queried and referenced, and it keeps its values in the type's unique indexes: a create of the same unique value is refused as a duplicate until the cleanup has run. The cleanup deletes a fixed number per block, so a sustained flood of creations with a short time to live builds a backlog it drains at that rate, and that lag grows with it.
  • The document may be deleted earlier as usual: by its owner when canBeDeleted allows it, by the contract's moderators when moderatorAbilities.delete does. canBeDeleted: false only stops the owner; the platform still deletes the document when it expires.
  • The deletion is an ordinary deletion: the document and every index entry go, count and sum trees are decremented, and nothing is left behind.

Where a time to live is refused

The keyword is refused, on every parse of the contract (registration, update and a stored contract read back), when:

The document typeWhy
does not list $createdAt in requiredThe expiry is computed from it.
sets documentsKeepHistory: trueDrive refuses to delete a document whose type keeps history.
sets indexOnly: trueThere is no stored row to delete by id.
has a contested indexA contested document waits in its vote poll until the poll awards it, keeping the $createdAt of its create, so it could expire before it is stored.
declares ttl: 0A time to live lasts at least a second.

When a contract is registered or updated, a ttl below min_document_ttl_seconds (one hour at protocol version 14) or above max_document_ttl_seconds (one year) is refused too. The floor keeps a document in state well past the moment its writer fetches the proof of its create, which proves the document present; a document the cleanup had already deleted would fail that proof although the create succeeded. A summed property that admits values below 0 is refused there too, unless its minimum and maximum lie within ±2^27 (max_expiring_signed_summed_value_magnitude): the cleanup takes a document's value out of the type's sums with no transition to refuse, and removing a negative value raises them, so the values must keep every sum in the signed 64-bit range. Removing values that are never negative only lowers the sums, and values within ±2^27 keep them in range short of 2^36 documents. A contract update may not add, remove or change the ttl of an existing document type: every stored document carries the expiry it was written and paid with. A document type added by an update declares it freely.

The same holds for any write close to a document's expiry: a replace, transfer, purchase or price update accepted in the last block before the expiry proves the document present, and a proof fetched after the next block's cleanup finds it gone.

References treat a type with a ttl as deletable, like one with canBeDeleted or moderatorAbilities.delete. A permanentDocument reference, a lookup one included, and a list element reference may not target it; a deletableDocument reference may, a lookup one included. The check is DocumentTypeV2Getters::documents_can_disappear.

Everything else composes: mutable types, transferable, tradeMode, moderatorAbilities.delete (a moderator's restore puts the document back with its original $createdAt, and is refused once that document has expired), creationRestrictionMode, countable, summable and ranked indexes, timeRange indexes with or without their own ttl, refersTo declared on the type, action fees and token costs.

Storage

Nothing a document of such a type writes carries storage flags: its primary item, its index entries and the index trees it creates are flagless, and its deletion refunds nothing. Drive drops the writer's flags when the document is inserted or changed (DocumentAndContractInfo::without_storage_flags_if_expiring), before any element is sized, and the re-tagging described below strips any left.

Each document also gets an entry in the documents expirations tree under Misc:

Misc (104) / E / <expires at, u64 big endian ms> / <document id> -> contract id (32 bytes) ++ document type name

The entry is written with the document and removed with it, whoever deletes it (the hook is in force_delete_document_for_contract_operations, which the owner's, the moderators' and the cleanup's deletions share). The last entry of an expiry time takes the tree of that time with it, so every tree of an expiry time holds at least one entry, and a document deleted early leaves nothing the cleanup would have to read.

The storage fees of such documents wait in the lifetime storage fee pools under Pools, a sum tree so the pools' total counts them, one sum item per epoch they were collected in and number of epochs their storage lives:

Pools (48) / l / <collected in epoch, u16 big endian><lifetime in epochs, u16 big endian> -> credits

Both trees are created with the initial state structure of protocol version 14 and on the first block of protocol version 14, through one helper (Drive::insert_document_ttl_trees).

Fees

The fee schedule's document_ttl group (FeeDocumentTtlVersion, FEE_VERSION3) prices a document of such a type:

  • Bytes. Every byte the document writes, its expirations tree entry included, costs the price of the lifetime it has left. Up to seven days a tier applies; past that a price per pricing_period_seconds spanned, rounded up. The period is part of the schedule (788,400 seconds, mainnet's epoch length), not the node's epoch length, so a network configured with short epochs, like testnet's hour, prices a lifetime as mainnet does.

    LifetimeCredits per byte (protocol version 14)
    up to 1 hour1
    up to 1 day4
    up to 2 days8
    up to 4 days15
    up to 7 days26
    longer34 per 9.125 days spanned

    The values are the first year's share of the perpetual storage price (27,000 credits per byte, 5% of it paid out in the first year) pro rata, rounded up. A one-year ttl pays 40 × 34 = 1,360 credits per byte, about what a permanent document deleted after a year keeps paying net of its refund.

  • Payout. That amount is a storage fee, paid out to the epochs the document lives in rather than over the 50 eras of the perpetual storage distribution. Drive counts the epochs its remaining lifetime spans, rounded up and at most one era (40 epochs), with the network's epoch length (the node's epoch_time_length_s, handed to Drive through DriveConfig) and reports the amount under that count (FeeResult::lifetime_storage_fees). At the end of the block the amounts go to the lifetime storage fee pools of the current epoch (add_distribute_block_fees_into_pools_operations v1), and the next epoch change spreads each pool of an earlier epoch evenly over its number of epochs from the new epoch on, the remainder of the division to the new epoch, and removes it (add_distribute_storage_fee_to_epochs_operations v1). A block never adds to a pool its epoch change removes, so the first block of an epoch does both in one batch. A network with hour-long epochs therefore pays a year-long document out within 40 hours.

  • Deletion. Creating the document prepays, as processing, what its deletion will cost: cleanup_base_processing_cost (1,200,000), plus cleanup_processing_cost_per_index_level (400,000) per index level of the type, where an index counts its properties, times the overlapping windows of a timeRange index, plus cleanup_processing_cost_per_document_byte (420, what removing and reading a byte costs) per byte of the stored document. The cleanup itself bills nobody.

  • Changes. A replace, transfer, purchase or price update prices the bytes it adds by the lifetime left at that block, and prepays the deletion of the document bytes it adds at cleanup_processing_cost_per_document_byte. A deletion by the owner or a moderator pays its own processing like any deletion; the prepaid deletion is the platform's, and is not refunded.

The price never decreases with the lifetime, so an estimate made at an earlier block time (check_tx) stays an upper bound of the execution; where the amount is paid out does not change what the writer pays, and a fee increase the writer offers, which multiplies processing only, never multiplies it. A dry run estimates a change as an insert of the changed document (estimate_document_change_as_insert_operations_v1): without the entry and the deletion the creation prepaid, which a change never pays, and with the deletion of all the document's bytes, at least what the change adds. In Drive the document's grove operations are re-tagged EphemeralGroveOperation(_, EphemeralPricing::DocumentTtl { .. }) and applied as their own GroveDB batch, so their added bytes can be priced apart from the rest of the transition (see apply_batch_low_level_drive_operations); operations already tagged for a timeRange index's ttl keep that rule.

Expired documents

validate_document_not_expired (drive-abci, state_transition/common) is the one rule: a document of a type with a ttl has expired when block time is at or past its $createdAt plus the ttl (Drive's document_expires_at, the time its entry is keyed by), the same boundary the cleanup deletes at. The batch's advanced structure validation (v1, protocol version 14 only), which check_tx runs too, calls it for every document action through validate_document_action_not_expired, an exhaustive match that refuses, paid, a replace, transfer, purchase or price update of an expired document. The moderator restore calls it too, judging the $createdAt of the document the removal record's hash pins. A document deletion by its owner does not.

Cleanup

Platform::expire_documents runs after the block's state transitions, right after the address balance cleanup in run_block_proposal, and calls Drive::remove_expired_documents with max_document_expirations_per_block and max_document_expiration_weight_per_block:

  1. fetch_expired_documents reads, in one query, the entries of the expiry times at or before the block time, oldest first, at most the limit of them. Every tree of an expiry time holds an entry, so the query visits at most the limit of trees.
  2. Each expired document is read and checked against state (its contract, document type, ttl and stored document, and that the document expires when its entry says), then deleted from what was read: the deletion an owner runs, without the canBeDeleted guard (delete_read_document_for_contract_operations_v0). Its entry, and the tree of its expiry time when it was the last, go with it.
  3. An entry without a document to delete is logged and only removed. None is expected, and failing the block over one would halt the chain.
  4. The cleanup stops before a removal that would pass the weight budget (an entry's removal weighs 1), except the block's first, so the backlog always drains.

Every removal goes into one GroveDB batch, applied without a fee, each built against the removals queued before it, so every emptiness check sees them.

Versioning

Everything rides protocol version 14, unreleased when this landed: the keyword joined document meta-schema v3 and the generation 3 parser, the limits joined SYSTEM_LIMITS_V4, the fee group joined FEE_VERSION3, the update rule joined validate_update v1, and expire_documents is Some(0) in DRIVE_ABCI_METHOD_VERSIONS_V10 only. The shipped generations edited in place are inert before 14:

  • the deletion hook in delete_document_for_contract_operations v0, the reference checks (documents_can_disappear) and the restore check of contract_user_moderation state v0: documents_ttl_seconds is only ever Some on a document type parsed from the keyword, which no earlier protocol version reads. The deletion's post-read part moved into delete_read_document_for_contract_operations_v0 with its operations unchanged;
  • the expire_documents call in run_block_proposal v0: the method is None before 14;
  • the tree's creation in transition_to_version_14 (perform_events_on_first_block_of_protocol_change v0), which only an upgrade to 14 runs;
  • one batch per pricing rule in apply_batch_low_level_drive_operations v0 and the DocumentTtl arm of consume_to_fees_v0: nothing is tagged ephemeral before 14;
  • add_distribute_block_fees_into_pools_operations v0, split into a helper that v1 shares, with its operations unchanged, and fetch_pending_epoch_refunds v0, whose query and reading moved unchanged into a helper the lifetime storage fee pools share;
  • the pattern-only edits in batch_insert_empty_tree_if_not_exists v0 and convert_drive_operations_to_grove_operations v0, whose output is unchanged.

Contested Documents

A unique index may be declared contested. A document whose index values fall in the contested range is not stored outright: it opens or joins a contest, a ContestedDocumentResourceVotePoll that names the contract, the document type, the index and the index values, and masternodes and evonodes decide who gets the value. A masternode's vote counts once, an evonode's four times. Every vote is a MasternodeVote state transition carrying the poll and a ResourceVoteChoice.

From protocol version 14, a vote towards an identity must name a contender of the poll. Any other identity, including the reserved keys under which the poll keeps its stored result and its abstain and lock tallies, is refused with VoteChoiceNotAllowedForVotePollError (40307). Before 14 such a vote failed with an internal error, or counted as abstain or lock when it named a reserved key.

The contest is funded by the contenders' prefunded voting balances, and each vote costs a fixed amount from that balance. Contenders may join for the join window (one week on mainnet) after the first document; the contest runs for the poll duration (two weeks on mainnet). The first document's owner may not be joined by the same identity twice.

From protocol version 14, a contest accepts at most 1,000 contenders (max_contenders_per_contest): a document that would add one more is refused, paid, with DocumentContestMaximumContendersReachedError (40141). The bound is what lets one block end a contest: its end tallies up to maximum_contenders_to_consider contenders (10,000 from version 14) and removes the entries of those it tallied, so a contest within the cap is tallied and cleaned up whole. Before 14 a contest accepted any number of contenders and its end tallied at most 100; a contest that grew past 10,000 contenders before 14 is tallied and cleaned up for its first 10,000 only.

From protocol version 14, the fund a contender pays doubles once the contest holds 250 contenders (contested_document_contenders_before_fund_doubling) and again for every 50 more (contested_document_contenders_per_fund_doubling), so a contest stops growing long before its cap:

Contenders the contest holdsDPNS fund to joinModeration election fund to join
0 to 2490.1 Dash0.5 Dash
250 to 2990.2 Dash1 Dash
300 to 3490.4 Dash2 Dash
...doubles every 50doubles every 50
700 to 749102.4 Dash512 Dash
...doubles every 50doubles every 50
950 to 9993,276.8 Dash16,384 Dash

Filling a DPNS contest to 1,000 contenders costs 327,695 Dash (100 at a flat 0.1 Dash). A contender's prefunded voting balance is the most it is willing to pay, and it must hold that much: it is charged the fund to join the contest it joins, and what it stated beyond that stays with it. One stating less, the first contender of a new contest included, is refused, paid, with DocumentContestNotPaidForError, which carries the fund it has to pay. The SDKs read how many contenders a contest holds and state that fund unless the caller names the most it will pay, which lets a join go through while others join ahead of it. Before 14 every contender stated exactly the contest's fund, however many had joined, and paid what it stated.

The index's contested.resolution says how the contest is decided.

From protocol version 14, an identifier property among the index values is written as an identifier in the poll (Index::extract_contested_values), whether the document gave it as an identifier, as 32 bytes or as an array of byte values. The index keys store all of these alike, but a poll is hashed from its values, and that hash keys the contest's prefunded balance and end date: two contenders writing the same identifier two ways would otherwise split one contest into two polls. Before 14 the values are taken as given.

Resolution 0: masternode vote

The DPNS rule. The choices are a contender, abstain, or lock, which gives the value to nobody. The contender with the most votes wins unless the lock tally exceeds it, in which case the value is locked and may be contested again later. The contest always runs the full poll duration, even with a single contender, so the masternodes may lock the value.

Resolution 1: masternode vote without locking

ContestedIndexResolution::MasternodeVoteNoLocking, meta-schema v3 (protocol version 14). The choices are a contender or abstain. A Lock vote is refused with VoteChoiceNotAllowedForVotePollError (40307). The contest always ends with a winner: the contender with the most votes, no minimum.

A contest without locking ends when its join window closes while it still has a single contender, so that contender is awarded the value without a vote window. Its end-date entry is written at the end of the join window when the contest opens; the first additional contender moves it to the full poll duration, which opens the vote window, and removes the join window's end date when no other contest ends then. getVotePollsByEndDate shows whichever end applies.

The moderation charters contract uses this resolution to elect moderation teams.

Moderation elections

An electedCharter contest of the moderation charters contract (protocol version 14), keyed by the target contract id, is a moderation election and does not take the generic parameters:

  • Its join window and vote window are the joinWindow and voteWindow of the target contract's elected moderation declaration (at most four weeks each, one week by default; at least a day on mainnet, while any other network takes 0), in place of the generic windows of the network. A single applicant wins when the join window closes; a second applicant moves the end to the join window plus the vote window. A late applicant is refused with DocumentContestNotJoinableError naming the target's join window.
  • Each application prefunds the votes with the moderation fund, 0.5 Dash (moderation_vote_resolution_fund_required_amount), instead of the contested document fund.

The target's declaration is read, and billed, when an application opens the contest and when a later one joins it; it is frozen at the target's creation, so both reads agree. Nothing at the end of a contest reads the target: the end date was written when the contest opened or was joined. A target that is missing or declares something else leaves a contest on the generic windows instead of failing, and the application's own reference validation refuses it. Every other contest, DPNS included, keeps the generic windows and fund.

Ties

From protocol version 14, a tie among the top contenders goes to the earliest contender: creation time, then block height, then core block height, then document id, among every tied contender. This holds for both resolutions; contests ending before version 14 awarded the latest contender.

Storage

A contest's state lives under votes / contested_resource / active_polls, laid out like the contested index it decides: the contenders' documents, one votes sum tree per contender, and the abstain and lock tallies. From protocol version 14 the tree below a contest's last index value, which holds its contenders, stored result and tallies, is a count tree, so a join reads how many contenders there are in one fetch; a contest started before 14 keeps its plain tree, and a join counts its contenders by reading their keys. The masternodes' vote references live under votes / contested_resource / identity_votes, and the end dates under votes / end_date_queries, one tree per end date holding an entry for each contest ending then. Once the contest ends, the winning document is awarded, the losers are removed, and the stored result stays for the getContestedResourceVoteState query.

A block ends at most maximum_vote_polls_to_process contests (a drive-abci event_constants value, two at protocol version 14), the earliest end date first, so contests due together may end over several blocks. The cleanup removes each ended contest's end-date entry, and removes an end date only once none of its contests remain under it; the rest end in the next blocks. Before that read, a block removes any end date among the first due that holds no contest, since each would take a place in the read and end nothing.

Identities

Before a user can do anything on Dash Platform -- register a name, send a contact request, create a data contract -- they need an identity. An identity is the platform-level representation of a user. It is the anchor for everything: documents are owned by identities, state transitions are signed by identity keys, and fees are paid from identity balances.

If you are coming from Ethereum, think of an identity as an account. But unlike Ethereum's single-key accounts, a Dash Platform identity can have multiple public keys with different purposes and security levels, making it more flexible and more secure.

The Identity Enum

Following the standard versioning pattern, Identity is defined in packages/rs-dpp/src/identity/identity.rs:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, PartialEq, From)]
pub enum Identity {
    V0(IdentityV0),
}
}

Currently only V0 exists. The IdentityV0 struct lives in packages/rs-dpp/src/identity/v0/mod.rs:

#![allow(unused)]
fn main() {
pub struct IdentityV0 {
    pub id: Identifier,
    pub public_keys: BTreeMap<KeyID, IdentityPublicKey>,
    pub balance: u64,
    pub revision: Revision,
}
}

Four fields. That is it. An identity is remarkably simple:

  • id: A 32-byte unique identifier. For identities created via asset locks, this is derived from the locking transaction. For identities created via address-based funding, it is derived from the input addresses and nonces.

  • public_keys: A BTreeMap mapping key IDs (simple integers) to IdentityPublicKey objects. Each public key has a purpose (authentication, encryption, decryption, transfer, voting, owner), a security level (master, critical, high, medium), and the actual key data. An identity can have many keys for different scenarios.

  • balance: The identity's credit balance, measured in platform credits. Credits are the unit of account for fee payment on the platform. Users convert Dash into credits through a process called "topping up."

  • revision: A monotonically increasing counter that increments with every identity update (adding keys, disabling keys, etc.). This prevents replay attacks -- each update must reference the current revision.

The Credit System

Platform credits are the fuel that powers everything on the network. Every state transition (creating a document, updating a contract, transferring tokens) costs credits. The credit system is how the platform measures and charges for computational and storage resources.

The balance field is a simple u64 representing the number of credits an identity holds. When an operation is performed, the fee system calculates the cost (as we will see in the Cost Tracking chapter) and deducts it from the identity's balance. If the balance is insufficient, the operation is rejected.

Public Keys and Key Types

An identity's public keys are not just random cryptographic keys -- they are structured with purpose and security level. Each IdentityPublicKey carries:

  • Key ID (KeyID): A numeric identifier unique within the identity, used to reference the key.
  • Purpose: What the key is for -- authentication, encryption, decryption, transfer, voting, or owner operations.
  • Security Level: How sensitive operations signed by this key are -- master, critical, high, or medium.
  • Key Type: The cryptographic algorithm -- ECDSA_SECP256K1, BLS12_381, ECDSA_HASH160, BIP13_SCRIPT_HASH, or EDDSA_25519_HASH160.
  • Data: The actual public key bytes.
  • Disabled At: An optional timestamp marking when the key was disabled.

This multi-key design means an identity can have a master key stored in cold storage, a critical key for important operations, and a high-level key for everyday use -- all belonging to the same identity. If a day-to-day key is compromised, the master key can disable it and add a replacement without losing the identity.

The PartialIdentity Pattern

Here is where the codebase reveals a practical optimization. Loading a full identity from storage is expensive -- you need to fetch the balance, all the keys, the revision, and potentially more. But most operations do not need all of that. A balance transfer only needs the balance. A document creation only needs to verify one key.

Enter PartialIdentity, defined alongside Identity in packages/rs-dpp/src/identity/identity.rs:

#![allow(unused)]
fn main() {
pub struct PartialIdentity {
    pub id: Identifier,
    pub loaded_public_keys: BTreeMap<KeyID, IdentityPublicKey>,
    pub balance: Option<Credits>,
    pub revision: Option<Revision>,
    pub not_found_public_keys: BTreeSet<KeyID>,
}
}

A PartialIdentity is exactly what it sounds like -- a partially-loaded identity. Notice the differences from Identity:

  • balance is Option<Credits> rather than a bare u64. It might not have been loaded.
  • revision is Option<Revision>. Same story.
  • loaded_public_keys might only contain the specific keys that were requested.
  • not_found_public_keys tracks which keys were requested but did not exist on the identity.

You can convert a full Identity into a PartialIdentity:

#![allow(unused)]
fn main() {
impl IdentityV0 {
    pub fn into_partial_identity_info(self) -> PartialIdentity {
        let Self { id, public_keys, balance, revision, .. } = self;
        PartialIdentity {
            id,
            loaded_public_keys: public_keys,
            balance: Some(balance),
            revision: Some(revision),
            not_found_public_keys: Default::default(),
        }
    }

    pub fn into_partial_identity_info_no_balance(self) -> PartialIdentity {
        let Self { id, public_keys, revision, .. } = self;
        PartialIdentity {
            id,
            loaded_public_keys: public_keys,
            balance: None,  // explicitly not loaded
            revision: Some(revision),
            not_found_public_keys: Default::default(),
        }
    }
}
}

The PartialIdentity pattern is used extensively in Drive's query and validation code. When processing a state transition, Drive fetches only the identity fields it actually needs, wraps them in a PartialIdentity, and passes that through the validation pipeline. This avoids unnecessary storage reads and keeps things efficient.

Identity Nonces

Replay protection on Dash Platform uses nonces rather than sequential transaction counters. There are two kinds:

  1. Identity nonce: A per-identity counter used for identity-level operations (like key updates).
  2. Identity-contract nonce: A per-identity-per-contract counter used for document operations. This allows operations on different contracts to be submitted in parallel without conflicting.

The nonce system is defined in packages/rs-dpp/src/identity/identity_nonce.rs and is more sophisticated than a simple incrementing counter. The nonce value is actually a packed u64 that contains both the counter value and a bitfield tracking recently-used nonces:

#![allow(unused)]
fn main() {
pub const IDENTITY_NONCE_VALUE_FILTER: u64 = 0xFFFFFFFFFF;
pub const MISSING_IDENTITY_REVISIONS_FILTER: u64 = 0xFFFFFF0000000000;
pub const MAX_MISSING_IDENTITY_REVISIONS: u64 = 24;
}

The lower 40 bits hold the current nonce tip. The upper 24 bits form a bitfield that tracks which of the last 24 nonce values have been seen. This allows out-of-order submission within a window: if a user submits nonces 5, 7, and 6 in that order, all three are accepted. But nonce 5 cannot be submitted again because it is already marked in the bitfield.

The validation function checks several conditions:

#![allow(unused)]
fn main() {
pub fn validate_identity_nonce_update(
    existing_nonce: IdentityNonce,
    new_revision_nonce: IdentityNonce,
    identity_id: Identifier,
) -> SimpleConsensusValidationResult {
    let actual_existing_revision = existing_nonce & IDENTITY_NONCE_VALUE_FILTER;
    match actual_existing_revision.cmp(&new_revision_nonce) {
        std::cmp::Ordering::Equal => {
            // Nonce already used at the tip
            // -> NonceAlreadyPresentAtTip error
        }
        std::cmp::Ordering::Less => {
            // Nonce is in the future -- check it's within window
            // -> NonceTooFarInFuture if gap > 24
        }
        std::cmp::Ordering::Greater => {
            // Nonce is in the past -- check bitfield
            // -> NonceTooFarInPast if gap > 24
            // -> NonceAlreadyPresentInPast if bit is already set
        }
    }
}
}

This design balances several concerns:

  • Replay protection: A nonce cannot be reused.
  • Out-of-order tolerance: Within a 24-nonce window, transactions can arrive in any order.
  • Bounded storage: Only 8 bytes are needed to track the full nonce state (the packed u64).
  • Parallel submission: Identity-contract nonces let different contracts have independent nonce spaces.

Creating an Identity

The Identity enum provides versioned constructors:

#![allow(unused)]
fn main() {
impl Identity {
    pub fn new_with_id_and_keys(
        id: Identifier,
        public_keys: BTreeMap<KeyID, IdentityPublicKey>,
        platform_version: &PlatformVersion,
    ) -> Result<Identity, ProtocolError> {
        match platform_version
            .dpp
            .identity_versions
            .identity_structure_version
        {
            0 => {
                let identity_v0 = IdentityV0 {
                    id,
                    public_keys,
                    balance: 0,
                    revision: 0,
                };
                Ok(identity_v0.into())
            }
            version => Err(ProtocolError::UnknownVersionMismatch {
                method: "Identity::new_with_id_and_keys".to_string(),
                known_versions: vec![0],
                received: version,
            }),
        }
    }
}
}

New identities start with a balance of 0 and a revision of 0. The balance is filled by the identity creation state transition (which includes an asset lock or address-based funding), and the revision increments from there.

How Identity Differs from Other Types

One important thing to note: the identity is not stored as a single blob in Drive. Unlike documents and data contracts (which are serialized and stored as items), identity fields are stored in separate locations within GroveDB's tree structure. The balance is in one place, each key is in another, the revision somewhere else. This is because different operations need to update different parts of the identity independently and atomically.

The Identity struct is primarily used for:

  • Creating new identities (assembling all fields for the creation state transition)
  • Client-side representation (what the SDK returns when you query an identity)
  • Transport (serialized for gRPC responses)

Inside Drive and ABCI, you will more commonly see PartialIdentity or direct field access through Drive's identity methods.

Rules and Guidelines

Do:

  • Use PartialIdentity when you only need a subset of identity fields. It avoids unnecessary storage reads.
  • Validate nonces through the provided validate_identity_nonce_update function -- the bitfield logic is subtle.
  • Always go through PlatformVersion when constructing identities to ensure the correct structure version.

Do not:

  • Assume an identity has only one key. Identities commonly have multiple keys with different purposes and security levels.
  • Manually pack or unpack nonce bitfields. Use the provided constants and validation functions.
  • Store or cache full Identity objects when a PartialIdentity would suffice. The full identity can be large if it has many keys.
  • Treat identity balance as Dash amounts. Credits are the unit of account on the platform; the conversion to/from Dash happens at the protocol level.

Key Budgets and Expiry

An identity rarely signs everything itself. A wallet hands a key to a game, a social client, a bot, and from then on that application signs state transitions on the identity's behalf and the identity's balance pays for them. Contract bounds already limit where such a key may act: one contract, one document type, or one contract group. Until protocol version 14 nothing limited how much it could spend or for how long. A key bound to one contract could still burn the whole identity balance on that contract, and it kept working until somebody remembered to disable it.

A budget and an expiry close that gap. They are two optional properties of an AUTHENTICATION key. A budgeted key can take at most so many credits from its identity; an expiring key stops signing at a given block time. Either one, or both, and both combine freely with contract bounds: bounds say where, limits say how much and for how long. When a limit is reached nothing has to be revoked. The key simply stops being accepted.

This chapter calls the two properties together limits. It covers the key format that carries them, the rule that decides what counts against a budget, where in the validation pipeline each check runs and why it runs there, and how Drive keeps the running total.

The Model

Five facts define a limited key:

  1. Limits are opt-in per key and live on a new key version. IdentityPublicKey::V1 is the version 0 key followed by total_budget and expires_at. Every key that existed before, and every key without limits registered after, is still a version 0 key with the same bytes as ever.
  2. Only AUTHENTICATION keys below MASTER may carry them. The master key is what registers a replacement when a key runs out, so it must never run out itself. TRANSFER, ENCRYPTION and DECRYPTION keys cannot be limited.
  3. Limits are signed, and only ever loosened. They are part of the signable bytes of the transition that registers the key. The one transition that changes them, IdentityKeyLimitsUpdate (see Raising Limits below), raises a budget or moves an expiry later; to give an application less, disable the key and register another.
  4. A budget caps what leaves the identity, and only goes down. Fees, and credits the transition moves out (a document purchase, a prefunded voting balance), count against it. Storage refunds do not top it up. What is left is tracked by Drive next to the key, because the key itself never changes.
  5. An expiry is a block time. expires_at is an absolute timestamp in milliseconds, the same unit and clock as disabled_at. The key signs at expires_at - 1 and not at expires_at.

A limited key moves through a small set of states, and only the first one can sign:

stateDiagram-v2
    [*] --> Usable: registered
    Usable --> Usable: signs, budget goes down
    Usable --> Spent: budget reaches 0
    Usable --> Expired: block time reaches expires_at
    Usable --> Disabled: disabled by the master key
    Spent --> Disabled
    Expired --> Disabled

Spent and Expired are not written anywhere as a flag. Spent is the remaining budget in Drive being zero, and a spent key is refused at signature validation with PublicKeyBudgetExhaustedError (20015). Expired is a comparison against the block time, and an expired key is refused at fee validation with PublicKeyExpiredError (20016). Disabling works exactly as before and is independent of the limits: a master key can disable a limited key in any state.

The Version 1 Key

IdentityPublicKeyV1 lives in packages/rs-dpp/src/identity/identity_public_key/v1/mod.rs:

#![allow(unused)]
fn main() {
pub struct IdentityPublicKeyV1 {
    pub id: KeyID,
    pub purpose: Purpose,
    pub security_level: SecurityLevel,
    pub contract_bounds: Option<ContractBounds>,
    #[serde(rename = "type")]
    pub key_type: KeyType,
    pub read_only: bool,
    pub data: BinaryData,
    pub disabled_at: Option<TimestampMillis>,
    /// The total credits that state transitions signed with this key may take from the identity.
    pub total_budget: Option<Credits>,
    /// The block time, in milliseconds, from which the key can no longer sign.
    pub expires_at: Option<TimestampMillis>,
}
}

The field is called total_budget, not budget, because it is the fixed amount the identity granted and never changes. What is left of it is a different number, kept by Drive, and the code calls that one the remaining budget throughout.

The first eight fields are the version 0 fields in the same order, so the two encodings differ only by the variant byte and the two trailing options. Version 0 is every key from before protocol version 14 and every key without limits; version 1 is a key that may carry them:

flowchart TB
    subgraph V0["IdentityPublicKey::V0"]
        direction LR
        a0["variant<br/><b>0</b>"] --- a1["id · purpose · security level · contract bounds<br/>type · read only · data · disabled at"]
    end
    subgraph V1["IdentityPublicKey::V1"]
        direction LR
        b0["variant<br/><b>1</b>"] --- b1["the same eight fields<br/>in the same order"] --- b9["<b>total budget</b><br/>optional credits"] --- b10["<b>expires at</b><br/>optional milliseconds"]
    end
    V0 ~~~ V1
    style a0 fill:#2d3748,color:#e2e8f0
    style b0 fill:#2d3748,color:#e2e8f0
    style b9 fill:#c05621,color:#fff
    style b10 fill:#c05621,color:#fff

IdentityPublicKeyInCreationV1 mirrors it in public_key_in_creation/v1/mod.rs, with total_budget and expires_at placed before the signature. Only the signature is excluded from the signable bytes, so the identity signs the limits it grants and a test pins it: changing a budget changes the signable bytes, changing the key's own signature does not.

In JSON the key is tagged "$formatVersion": "1" and the two fields appear as totalBudget and expiresAt. Like disabledAt, they are left out when absent.

Reading and Building

Call sites never match on the variant. IdentityPublicKeyGettersV1 (accessors/v1/mod.rs) is implemented on the enum and on both structs, and a version 0 key answers None:

#![allow(unused)]
fn main() {
pub trait IdentityPublicKeyGettersV1 {
    fn total_budget(&self) -> Option<Credits>;
    fn expires_at(&self) -> Option<TimestampMillis>;

    /// The expiry instant itself is already expired.
    fn is_expired_at(&self, time_ms: TimestampMillis) -> bool { /* ... */ }
    fn has_limits(&self) -> bool { /* ... */ }
}
}

To build one, take any key and call with_limits. A version 0 key becomes a version 1 key and every other field is kept:

#![allow(unused)]
fn main() {
let app_key = key.with_limits(
    Some(dash_to_credits!(0.1)),          // total budget
    Some(now_ms + 30 * 24 * 3_600_000),   // expires in 30 days
);
}

The conversions between IdentityPublicKey and IdentityPublicKeyInCreation carry the limits in both directions, so the existing identity update builders register a limited key without knowing about limits. The one conversion that would lose them is building an IdentityPublicKeyInCreationV0 directly from a key; go through the enum instead.

The Protocol Version Gate

A binary from before this change cannot decode a version 1 key, so it rejects a transition carrying one while decoding. A new binary running protocol version 13 has to do the same, or the two would disagree during an upgrade window. StateTransition::active_version_range is where that is enforced: a transition carrying a version 1 key in creation is active from 14.

#![allow(unused)]
fn main() {
fn active_version_range_for_keys_in_creation(
    keys: &[IdentityPublicKeyInCreation],
    otherwise: RangeInclusive<ProtocolVersion>,
) -> RangeInclusive<ProtocolVersion> {
    if IdentityPublicKeyInCreation::first_bound_to_a_contract_group(keys).is_some()
        || IdentityPublicKeyInCreation::first_in_version_1_format(keys).is_some()
    {
        14..=LATEST_VERSION
    } else {
        otherwise
    }
}
}

The same function gates keys bound to a contract group. The predicates live on the key type (public_key_in_creation/mod.rs) so that every place that asks the question asks it the same way: first_in_version_1_format here, and first_with_limits for the shielded pool rule below, next to first_bound_to_a_contract_group. The gate is on the variant, not on whether limits are present: an old binary fails on the variant either way. It must also be this gate and not a structure rule, because identity create validates key structure in a paid stage. Refusing there would charge the asset lock on a new binary while an old one fails to decode, which is a state divergence.

Registering a Limited Key

A version 1 key arrives the way any key does, in the key list of a transition that registers keys: IdentityCreate, IdentityUpdate, IdentityCreateFromAddresses or IdentityCreateFromShieldedPool. Five things can stop it, in this order:

flowchart TD
    T["a transition registers<br/>a version 1 key"] --> PV{"protocol version<br/>at least 14?"}
    PV -->|no| NA["not active<br/>rejected while decoding"]
    PV -->|yes| SH{"has limits and is created<br/>from the shielded pool?"}
    SH -->|yes| E38["<b>10538</b><br/>refused"]
    SH -->|no| ST{"AUTHENTICATION and<br/>below MASTER?"}
    ST -->|no| E36["<b>10536</b><br/>limits not allowed"]
    ST -->|yes| BZ{"budget present<br/>and zero?"}
    BZ -->|yes| E37["<b>10537</b><br/>zero budget"]
    BZ -->|no| EX{"expires_at after<br/>the block time?"}
    EX -->|no| E19["<b>40219</b> already expired<br/>paid failure"]
    EX -->|yes| OK["key stored as version 1<br/>remaining budget = budget"]

    style NA fill:#c53030,color:#fff
    style E38 fill:#c53030,color:#fff
    style E36 fill:#c53030,color:#fff
    style E37 fill:#c53030,color:#fff
    style E19 fill:#c05621,color:#fff
    style OK fill:#276749,color:#fff

The checks sit in the tier their inputs dictate (see Where a check belongs):

RuleNeedsWhereError
Only AUTHENTICATION below MASTERthe keyrs-dpp validate_identity_public_keys_structure v1IdentityPublicKeyLimitsNotAllowedError 10536
Budget is not zerothe keysameInvalidIdentityPublicKeyBudgetError 10537
Expiry is after the block timethe block timedrive-abci validate_identity_public_keys_limits, called from the four identity create and update state/v1 validatorsIdentityPublicKeyAlreadyExpiredError 40219
No limits in a shielded identity creationthe transitiondrive-abci validate_shielded_proof v1, and the transition builderIdentityPublicKeyLimitsNotAllowedInShieldedIdentityCreationError 10538

The expiry rule exists because a key that is dead on arrival is not harmless. It still uses up its key id and, for a unique key type, registers its public key hash, which can then never be used on any identity again. The usual way to get there is passing seconds where milliseconds are expected, and the check turns that mistake into an error instead of a burnt key.

The Shielded Pool Exception

IdentityCreateFromShieldedPool has no identity signature. Its authorization is the Orchard proof, the spend authorization signatures, and a binding signature over a sighash. The new identity's keys are committed into that sighash by a hand-written preimage, identity_create_from_shielded_extra_sighash_data_v0, which lists the key fields it binds one by one: id, purpose, security level, type, data, read_only, contract bounds. That layout is frozen, and it predates the version 1 key. A budget or an expiry would simply not be in it.

How exposed would the limits be? Each ECDSA or BLS key carries a proof of possession, and that signature is over the signable bytes of the whole transition, which include every key with its variant and its limits. One such key is therefore enough to pin the limits of all of them. Hash based key types must carry an empty signature, so a transition whose keys are all hash based has nothing signing the limits. That is not an exotic case: hash based keys are the privacy-minded choice, and privacy is why one creates an identity from the shielded pool. For such a transition a relay or a proposer could strip a budget, or move an expiry, and everything would still verify. A test pins the cause: the preimage of a key with limits and of the same key without is byte for byte identical.

The preimage stays frozen and the key is refused instead, the same decision taken for keys bound to a contract group. validate_shielded_proof v1 refuses a key that carries a budget or an expiry before the preimage is built, and the transition builder refuses it before a proof is generated. Add the key with an identity update once the identity exists.

The refusal is about the limits, not about the format. A version 1 key without limits is accepted, because everything it holds is in the layout: it binds the same preimage bytes as the version 0 key with the same fields, and a second test pins that too. This keeps shielded identity creation working if version 1 ever becomes the default key format. What it leaves possible, and only when every key is hash based, is a relay re-encoding a limit-less key from one version to the other. The two are the same key, so nothing the identity granted changes; the visible effects are two more stored bytes and a different transition hash.

The general lesson. PlatformSignable covers a new key field automatically. The shielded preimage does not. Any future field on the key must be checked against packages/rs-dpp/src/shielded/sighash.rs as well.

Raising Limits

A spent key, or one about to expire, does not have to be replaced. IdentityKeyLimitsUpdate (state transition type 23) raises the limits of one key of the identity:

#![allow(unused)]
fn main() {
pub struct IdentityKeyLimitsUpdateTransitionV0 {
    pub identity_id: Identifier,
    pub nonce: IdentityNonce,
    pub key_id: KeyID,
    pub total_budget: Option<Credits>,        // the new total, None to leave it
    pub expires_at: Option<TimestampMillis>,  // the new expiry, None to leave it
    pub user_fee_increase: UserFeeIncrease,
    pub signature_public_key_id: KeyID,
    pub signature: BinaryData,
}
}

An update only ever loosens. A budget can grow, an expiry can move later, and that is all: a limit cannot be lowered, and a key that has no budget or no expiry cannot be given one. Tightening is what disabling is for. This one rule keeps the transition small, and it is what makes the arithmetic safe: the remaining budget never exceeds the total, both grow by the same amount, so the addition cannot overflow.

Both fields carry the new absolute value, not an amount to add; "top up by X" is SDK sugar that adds X to the total the client's copy of the key shows. What the wire carries is then exactly what a proof of the execution shows, and the proof binds it. No identity revision is claimed and none is bumped: an identity update needs one because clients allocate key ids, while this transition names an existing key and allocates nothing, so a stale copy of the identity cannot make it collide and the client only needs to hold the signing key. Two clients topping up the same key from the same stale copy find the second raise counted from the first's result, and a total that no longer raises the stored one is refused.

Who may sign. A MASTER key, or a CRITICAL authentication key that carries no limits itself and no contract bounds (a bound key may only sign batches, ContractBoundedKeyNonBatchError 20013). A key with limits can never raise limits, so it can never top itself up; the identity signature validation refuses it before the remaining budget is even read.

flowchart TD
    S["signer is MASTER,<br/>or CRITICAL without limits<br/>and without contract bounds"] --> K{"key exists<br/>and is enabled?"}
    K -->|no| E2["<b>40209</b> / <b>40208</b><br/>paid"]
    K -->|yes| L{"has each limit<br/>being raised?"}
    L -->|no| E3["<b>40220</b><br/>paid"]
    L -->|yes| G{"each new value<br/>greater?"}
    G -->|no| E4["<b>40221</b><br/>paid"]
    G -->|yes| X{"not expired<br/>afterwards?"}
    X -->|no| E5["<b>40219</b><br/>paid"]
    X -->|yes| OK["key rewritten,<br/>remaining raised"]

The last check is the one with a twist. An expired key may be revived by moving its expiry past the block time, but it cannot be topped up while it stays expired: after the update the key must be usable, or the update was pointless. The mempool asks the same questions of the key when it admits the transition, against the last block's time, so a bad target key is answered with these codes at admission; a top-up that arrives just before the key's expiry can still be admitted and then refused, paid, in the block where it lands. The two structural rules run unpaid before any of this: the transition must set at least one of the two fields (IdentityKeyLimitsUpdateEmptyError 10539), and a budget it sets is not zero (10537). A limited signer is refused unpaid with PublicKeyWithLimitsCannotUpdateKeyLimitsError 20017.

What Drive does. The key is read once, when the transition is validated, and carried in the action as it is stored: update_identity_key_limits sets the new limits on that copy and rewrites it in place with replace_key_in_storage_operations, the same patch a disable uses; the rewrite is priced by the byte delta, since a bigger total is a bigger varint and an expiry that was absent is new bytes. The references to the key in the purpose and security level trees carry its value hash, so they are refreshed as a disable refreshes them. Then add_to_identity_key_budget raises the remaining budget by the difference between the new total and the old one: a same-size replace of the eight byte entry. The nonce is updated first.

The proof. The proof is the rewritten key, nothing more. The verifier requires the key present and holding exactly the total budget and the expiry the transition asked for. That authenticates the state the update aimed at, not this exact transition: the nonce and the fee increase are signed but not stored, so any later state of the key with those limits would produce the same proof. The outcome is therefore classified as affected state, like a credit transfer, and the SDKs wait for it with the affected-state wait.

In the SDKs: Identity::update_key_limits, top_up_key_budget and extend_key_expiry (Rust, UpdateIdentityKeyLimits), identityUpdateKeyLimits({ identity, keyId, addBudget, expiresAt, signer }) (wasm-sdk), sdk.identities.updateKeyLimits (js-evo-sdk). All resolve to the key as stored after the update. A limited key is registered the ordinary way: an IdentityPublicKeyInCreation built with totalBudget or expiresAt (wasm-dpp2) passed to identityUpdate or sdk.identities.update, or an IdentityPublicKey::with_limits(..) key passed to the Rust identity update builder.

In the wallet and on mobile: IdentityWallet::update_identity_key_limits_with_external_signer (platform-wallet) raises the limits and lays the key as stored over the cached identity, so the client's key row follows through the persister; the FFI exposes it as platform_wallet_update_identity_key_limits_with_signer and the query as dash_sdk_identity_fetch_keys_remaining_budgets. Every key row that crosses the FFI (registration, update, the persisted key entry, the cold-restore row and the managed identity's key snapshot) carries total_budget and expires_at, so a limited key persists and restores as limited. Kotlin: IdentityUpdates.updateKeyLimits and Identities.fetchKeysRemainingBudgets, with IdentityPubkey.totalBudget / expiresAt on the rows it registers (Room schema 12). Swift: ManagedPlatformWallet.updateIdentityKeyLimits(identityId:keyId:addBudget:expiresAt:signer:) and SDK.fetchKeysRemainingBudgets(identityId:keyIds:), with the two limits on IdentityPublicKey, IdentityPublicKeyInfo, the IdentityPubkey row and PersistentPublicKey (SwiftData schema V5). The wallet's own signers prefer a key without limits and skip an expired one; what is left of a budget is only known through the query, so a spent key is refused by Platform.

The Budget Rule

A budget has one subtlety, and it is the same one identity balances have. Most of what a transition costs is known before it runs, but the metered processing fee is only known afterwards. A rule that demanded the whole fee fit up front would have to use the worst-case estimate, which can be fifty times the real processing cost, and would strand the tail of every budget. A rule that checked nothing up front would let a key spend without limit.

So the cost is split. Everything that is known, chosen or priced before the transition runs must fit in what is left. Only the metered processing fee may take the key over:

flowchart TD
    M["<b>Must fit in what is left of the budget</b><br/><br/>credits moved out of the identity<br/><i>a purchase price,<br/>a prefunded voting balance</i><br/><br/>storage fee<br/><i>estimated</i><br/><br/>fees priced up front<br/><i>a contract registration</i><br/><br/>user fee increase<br/><i>the processing the signer<br/>chose to add</i>"]
    M --> C{"required ≤ remaining?"}
    C -->|no| X["<b>40218</b> refused<br/>unpaid, nothing changes"]
    C -->|yes| RUN["the transition runs"]
    P["<b>May take the key over its budget</b><br/><br/>metered processing fee<br/><i>known only after the run</i>"] --> RUN
    RUN --> D["remaining = max(0, remaining − spent)"]

    style M fill:#1a365d,color:#e2e8f0
    style X fill:#c53030,color:#fff
    style RUN fill:#276749,color:#fff
    style P fill:#c05621,color:#fff

In code this is required_from_key_budget in validate_fees_of_event/v1/mod.rs:

#![allow(unused)]
fn main() {
removed_balance.unwrap_or_default()
    .saturating_add(storage_fee)
    .saturating_add(additional_fixed_fee_cost.unwrap_or_default())
    .saturating_add(user_fee_increase_amount)
}

Each term is there for a reason:

  • removed_balance: without it a budgeted key could drain the identity through document purchases, where the credits go to the seller and never show up as a fee.
  • Storage fee: the part of a fee that pays for permanent state. It is the part identity balances also treat as mandatory.
  • additional_fixed_fee_cost: technically a processing fee, but priced up front and large. A contract registration costs a tenth of a Dash. Letting it overshoot would make a budget of a thousand credits meaningless.
  • User fee increase: also processing, but it is the signer's choice, up to 65,535 percent. A key that is about to run out must not be able to burn several hundred times a processing fee on its way out.

What remains, the metered base processing fee, is bounded by what one transition can do. That is the "slightly over".

A Worked Example

Take a transition whose storage fee is 50,000,000 credits and whose metered processing fee turns out to be 2,000,000, with no user fee increase and nothing moved out of the identity. Required is 50,000,000.

Remaining beforeOutcomeIdentity paysRemaining after
60,000,000runs52,000,0008,000,000
50,000,001runs, processing overshoots52,000,0000
49,999,999refused with 40218, unpaid049,999,999
0refused with 20015 at signature validation00

The second row is the rule in one line. The storage fits with a credit to spare, so the transition runs. The identity pays the full 52,000,000, which is 1,999,999 more than the key had left, and the remaining budget stops at zero. The key is now spent, and the fourth row is what happens to its next transition, whatever that transition costs. should_let_only_metered_processing_take_a_key_over_its_budget pins rows two to four against real fees.

What Counts as Spent

After the transition runs, execute_event v1 deducts:

spent = removed_balance + the fee the identity owes, net of its own storage refunds

The second term is desired_removed_balance from FeeResult::into_balance_change, the same number the identity balance is charged. Three consequences:

  • A deletion whose refund exceeds its fee costs the budget nothing, and a refund never adds to it. A budget only goes down.
  • A failed transition that is still paid for (a penalty and a nonce bump) spends from the budget like a successful one. It goes through the same event.
  • The deduction saturates at zero. It never fails and never goes negative.

This mirrors the identity balance so closely on purpose. BalanceChange::RemoveFromBalance already carries a required_removed_balance (storage) and a desired_removed_balance (storage plus processing), and an identity that can cover the first but not the second goes into processing debt. A key budget is the same rule with the debt replaced by "and then the key is done".

Where the Checks Run

Three stages enforce limits at signing time. Which check goes where follows from what each stage knows:

flowchart TD
    A["is allowed"] --> B["<b>identity signature</b> v1<br/>has Drive and the signing key<br/>has no block time, no fee"]
    B --> C["nonce, basic structure,<br/>balance pre-check"]
    C --> D["advanced structure,<br/>transform into action,<br/>state validation"]
    D --> E["ExecutionEvent::Paid<br/>carries SigningKeyLimits"]
    E --> F["<b>validate_fees_of_event</b> v1<br/>has the block time<br/>and the estimated fee"]
    F --> G["<b>execute_event</b> v1<br/>has the actual fee"]
    G --> H["operations applied<br/>identity charged<br/>key budget deducted"]

    B -.->|"remaining budget is 0"| R1["<b>20015</b> budget exhausted<br/>unpaid"]
    F -.->|"block time ≥ expires_at"| R2["<b>20016</b> key expired<br/>unpaid"]
    F -.->|"required > remaining"| R3["<b>40218</b> budget exceeded<br/>unpaid"]

    style B fill:#744210,color:#fff
    style F fill:#744210,color:#fff
    style G fill:#744210,color:#fff
    style R1 fill:#c53030,color:#fff
    style R2 fill:#c53030,color:#fff
    style R3 fill:#c53030,color:#fff
    style H fill:#276749,color:#fff

The diagram shows the order in which the checks happen. In code the last two boxes are nested: during block execution execute_event calls validate_fees_of_event and then applies, while check tx calls validate_fees_of_event on its own and never executes.

A spent key is refused first, with the signature. Signature validation already reads the identity from Drive, so reading what is left of the budget there costs one more lookup, and an application that keeps trying with a spent key is turned away before any real work is done. This is the role the minimum balance pre-check plays for an empty identity. The read is billed as one extra key retrieval.

Expiry waits for fee validation. The natural place for it would be next to the disabled_at check, but signature validation is not given the block time, and the processor that calls it is a shipped v0. Threading a parameter through a shipped generation is exactly what the versioning rules forbid, and a new generation of the processor and of check tx would copy some fourteen hundred lines to pass one integer. validate_fees_of_event is the first stage that has the block time, runs for every paid transition, and runs in check tx as well as in block execution. The cost of checking late is that an expired key's transition is fully validated before it is refused. An insufficient balance has always had that same profile.

The budget comparison needs the estimated fee, so it can only be in fee validation. The deduction needs the actual fee, so it can only be in execution.

Carrying the Limits

The processor, check tx and the unversioned create_from_state_transition_action are all untouched. What connects the signature stage to the fee stage is the StateTransitionExecutionContext that already travels with every transition:

sequenceDiagram
    autonumber
    participant P as Processor v0
    participant S as Signature v1
    participant D as Drive
    participant E as execute_event v1

    P->>S: validate the signature
    S->>D: balance and signing key
    S->>D: remaining budget
    D-->>S: remaining
    alt remaining is 0
        S-->>P: 20015, unpaid
    else something is left
        Note over S: limits recorded in the<br/>execution context
        S-->>P: valid
    end
    Note over P: nonce, structure,<br/>transform, state
    Note over P: limits copied from the context<br/>onto ExecutionEvent Paid
    P->>E: execute_event
    Note over E: validate_fees_of_event v1<br/>expiry, then budget
    E->>D: apply the operations
    E->>D: charge the identity
    E->>D: deduct from the key budget

SigningKeyLimits (execution/types/signing_key_limits.rs) is three fields: the key id, expires_at, and the remaining budget as read at signature validation. It is None on the event for every transition signed by an ordinary key, and both v1 methods start by checking for it and handing everything else to their v0:

#![allow(unused)]
fn main() {
let ExecutionEvent::Paid { signing_key_limits: Some(signing_key_limits), .. } = event else {
    return self.validate_fees_of_event_v0(event, block_info, transaction, platform_version, previous_fee_versions);
};
}

An ordinary key therefore costs nothing new: no read, no write, no branch past that first line. execute_event v1 is narrower still. It only takes over for a budgeted signing key; a key that merely expires is executed by v0, which calls the fee validation dispatcher and so still reaches the expiry check.

The remaining budget read at signature time is safe to reuse at fee time because nothing can change it in between. State transitions in a block are processed one after another, each with its own signature validation, and the deduction itself re-reads the value inside the block transaction.

Check Tx

First time checkRecheckBlock execution
Signature validation, spent key refusedyesskippedyes
Expiryagainst the last committed block timenot checkedagainst the block's own time
Budget against the estimated feeyesnot checkedyes
Deductionnever, check tx does not writeneveryes

Recheck skips signature validation for every transition, so no key is loaded and no limits reach the event. A transition whose key expires while it waits in the mempool is therefore not evicted by a recheck. The proposer drops it at prepare_proposal, which is what happens to any transition that turned unpayable while it waited.

Refusals Are Not Charged

All three signing-time refusals return an unpaid result, like IdentityInsufficientBalanceError. Check tx rejects the transition, a proposer removes it from the block, and a block that contains one is rejected by validators. Nothing about the identity changes: not the balance, not the budget, not the nonce.

One case deserves a sentence. An invalid transition is normally a paid failure: the identity is charged a penalty and its nonce is bumped. If the key that signed it has expired or cannot cover the penalty, there is nobody to charge through that key, and the transition takes the path that already exists for an invalid transition from an identity that cannot afford the penalty. It is reported as an internal error and dropped from the block. A key that may not spend is never used to charge a penalty either.

Storage

The key is immutable, so the running total cannot live in it. It lives in a new subtree of the identity:

flowchart LR
    I["Identities"] --> ID["identity id<br/><i>32 bytes</i>"]
    ID --> CI["IdentityContractInfo <b>32</b>"]
    ID --> N["IdentityTreeNonce <b>64</b>"]
    ID --> NC["IdentityTreeNegativeCredit <b>96</b>"]
    ID --> K["IdentityTreeKeys <b>128</b><br/>key id → serialized key<br/><i>immutable: holds total_budget and expires_at</i>"]
    ID --> KR["IdentityTreeKeyReferences <b>160</b>"]
    ID --> REV["IdentityTreeRevision <b>192</b>"]
    ID --> KB["IdentityTreeKeyBudgets <b>224</b><br/><i>created with the first budgeted key</i>"]
    KB --> E1["key id 5 → 00 00 00 00 02 FA F0 80<br/><i>remaining credits, 8 bytes big endian</i>"]
    KB --> E2["key id 9 → ..."]

    style KB fill:#c05621,color:#fff
    style E1 fill:#744210,color:#fff
    style E2 fill:#744210,color:#fff

Three properties of this layout matter.

It is created lazily. New identities do not get the subtree, and identities from before protocol version 14 do not have it. insert_identity_key_budget_operations creates it with batch_insert_empty_tree_if_not_exists_check_existing_operations when the first budgeted key arrives, checking the pending operations as well as the state, because two budgeted keys can arrive in one transition. A key that only expires has no entry: there is nothing to count.

The value is fixed width. Eight big endian bytes, never a varint. Deducting from a budget then replaces the value without changing what is stored, so it produces no storage fee and no refund. That is what allows the deduction to be applied outside of the transition's fee, exactly like the balance write it follows. The identity's negative credit item uses the same trick for the same reason.

It is written once with the fee and then maintained without one. Whoever registers the key pays for the entry, and for the subtree the first time. insert_new_unique_key and insert_new_non_unique_key v1 add that write next to the key; keys cannot carry a budget before protocol version 14, so both v0s never do. From then on each deduction is an unbilled replace.

The methods live in packages/rs-drive/src/drive/identity/key/budget/:

MethodDoes
insert_identity_key_budget_operationswrites the full budget as the remaining budget of a new key, creating the subtree if needed
fetch_identity_key_remaining_budgetreads what is left; None for a key without a budget, including when the subtree does not exist
deduct_from_identity_key_budgetsubtracts, stopping at zero, and applies; errors if the key has no entry
add_estimation_costs_for_key_budgetsthe layer information for the subtree in estimation mode

Reading What Is Left

A spent key is refused, so a client wants to know where a budget stands before it signs. The getIdentityKeysRemainingBudgets query answers for several keys of one identity at once:

request:  identity_id, key_ids [3, 4, 5], prove
response: 3 -> 250000000    a budgeted key
          4 -> 0            a budgeted key that is spent
          5 -> (nothing)    no budget, or no such key

Every requested key id is answered. A key without a budget and a key that does not exist look the same here, because the query reads the budgets subtree and nothing else; getIdentityKeys tells them apart. The request names at least one key, none twice, and at most max_returned_elements of them.

With prove, the answer is a GroveDB proof of the path query [Identities, identity_id, IdentityTreeKeyBudgets] with one query item per key id, limited to their count. The interesting part is that "no budget" is provable in all three shapes state can have:

flowchart LR
    Q["prove the budgets<br/>of keys 3, 4, 5"] --> S{"budgets subtree<br/>exists?"}
    S -->|no| A["subtree proved absent<br/>every key is <b>None</b>"]
    S -->|yes| E{"entry for<br/>the key?"}
    E -->|yes| V["8 byte value proved<br/><b>Some(remaining)</b>"]
    E -->|no| N["entry proved absent<br/><b>None</b>"]

The verifier, Drive::verify_identity_keys_remaining_budgets, rebuilds the same path query from the request and uses verify_query_with_absence_proof, so it returns one entry per requested key and never an entry that was not asked for. packages/rs-drive/src/drive/identity/key/budget/mod.rs pins this with a test that forges a value inside a proof and expects it to no longer verify against the state root.

The same call is available at every layer:

LayerCall
Drivefetch_identity_keys_remaining_budgets, prove_identity_keys_remaining_budgets, verify_identity_keys_remaining_budgets
Rust SDKIdentityKeysRemainingBudgets::fetch(&sdk, IdentityKeysRemainingBudgetsQuery { identity_id, key_ids }), also fetch_unproved
wasm-sdkgetIdentityKeysRemainingBudgets(identityId, keyIds), ...WithProofInfo, returning Map<number, bigint | null>
js-evo-sdksdk.identities.keysRemainingBudgets(identityId, keyIds), keysRemainingBudgetsWithProof

What the query returns is the state as of the last committed block. A transition already in the mempool may spend from the budget before yours runs, so treat the number as an upper bound, not a reservation.

Versioning Touchpoints

Everything is gated to protocol version 14. Tables that protocol version 14 already owned (it was unreleased at the time) were amended in place; one new table was needed.

TableSlotChange
STATE_TRANSITION_METHOD_VERSIONS_V2 (new)validate_identity_public_keys_structure0 → 1
DRIVE_ABCI_VALIDATION_VERSIONS_V10validate_identity_public_keys_limitsnew, None → Some(0)
validate_state_transition_identity_signedstays 1, extended in place
validate_shielded_proofstays 1, extended in place
DRIVE_ABCI_METHOD_VERSIONS_V10validate_fees_of_event0 → 1
execute_event0 → 1
DRIVE_IDENTITY_METHOD_VERSIONS_V2keys.insert.insert_new_unique_key, insert_new_non_unique_key0 → 1
keys.budget.* (six slots, two of them for the query)new, None → Some(0)
DRIVE_ABCI_QUERY_VERSIONS_V0 and _V1identity_based_queries.keys_remaining_budgetsnew slot at 0; the Drive methods behind it are None before 14, which is what refuses the query there
DRIVE_VERIFY_METHOD_VERSIONS_V1identity.verify_identity_keys_remaining_budgetsnew, 0 (verification is client side and not gated)
DRIVE_ABCI_VALIDATION_VERSIONS_V10identity_key_limits_update_state_transitionnew slot; every gate None before V10, on from V10 (the transition is gated by is_allowed and active_version_range to 14 as well)
DRIVE_IDENTITY_METHOD_VERSIONS_V2update.update_identity_key_limits, keys.budget.add_to_identity_key_budgetnew, None → Some(0)
DRIVE_STATE_TRANSITION_METHOD_VERSIONS_V1 to _V4convert_to_high_level_operations.identity_key_limits_update_transitionnew, 0
STATE_TRANSITION_SERIALIZATION_VERSIONS_V1 to _V3identity_key_limits_update_state_transitionnew, 0

Two of these are worth a second look. Identity signature validation v1 and shielded proof validation v1 were introduced for protocol version 14 by the contract bounds work and had not shipped yet, so they were extended in place rather than given a v2. validate_fees_of_event and execute_event had shipped, so they got new generations, and those generations delegate to v0 for every event they do not handle instead of copying it.

Fees

Registering a budgeted key costs slightly more than registering an ordinary one: the eight byte entry, and the empty subtree the first time. Both are ordinary metered storage charged to the transition that adds the key. A key that only expires costs what a version 0 key costs plus the few bytes the two fields add to the stored key.

Signing with a budgeted key adds one fixed charge, the key retrieval that reads what is left (fetch_single_identity_key_processing_cost). The deduction write is not billed, like the balance write. Signing with a key that only expires adds nothing.

What Is Not There Yet

This change is the consensus core. Known gaps, all deliberate:

  • SDK surfaces. Rust callers can use with_limits today. The wasm, JavaScript, Swift and Kotlin bindings do not expose the two fields for creation, and SDK key selection does not skip an expired or spent key before signing, although the query above gives it what it needs. Swift and Kotlin do not expose that query yet.
  • Lowering limits. An update only raises a budget or extends an expiry (above). To take a key back, disable it.
  • Limits on other purposes. A TRANSFER key with a budget would cap transfers and withdrawals. The rule for it would differ (the amount moved is the point, not a side effect), so it was left out rather than half done.
  • Limits in a shielded identity creation. A key with a budget or an expiry is refused there, as explained above. Lifting that means a new generation of the sighash preimage.

Tests

  • dpp (identity_public_key/v1, public_key_in_creation/v1, validate_identity_public_keys_structure/v1, state_transition/mod.rs): the version 1 round trip; the version 0 encoding unchanged byte for byte; the JSON shape; the expiry boundary; limits surviving the conversions and being signed over; the structure rules; the protocol version 13 and 14 sides of the decode gate; frozen error discriminants.
  • drive (drive/identity/key/budget): the budget written on key add and on identity create; two budgeted keys in one batch; the deduction stopping at zero and never changing storage; estimated at least actual; protocol version 13 untouched.
  • drive-abci (batch/tests/key_limits.rs), end to end through process_raw_state_transitions and check_tx: the exact deduction; the rule pinned one credit either side of the storage fee; the user fee increase counted up front; the expiry boundary; both limits on one key; a paid failure spending from the budget; an invalid transition through an unusable key not being charged; mempool admission.
  • drive-abci (identity_key_limits_update/tests.rs), end to end: a spent key topped up and admitted again; an expired key revived by an extension and refused while it stays expired; each refusal pinned to its code, paid ones bumping the nonce; MASTER and unlimited CRITICAL signers accepted, a limited or HIGH one refused; the mempool; the execution proof; protocol version 13. drive (update_identity_key_limits): the total and the remaining budget growing by the same amount; the expiry alone; the estimate covering the rewrite; the whole database consistent after the reference refresh; the addition to the remaining budget; protocol version 13.
  • drive-abci (identity_update key_limits, identity_create_from_shielded_pool/tests.rs): registration storing a version 1 key with its whole budget left; an already expired key as a paid failure; limits on TRANSFER and MASTER keys and a zero budget as unpaid; the shielded creation refusing a key with either limit and accepting a version 1 key without any. The builder has the same pair of cases in rs-dpp (shielded/builder/identity_create_from_shielded_pool.rs).
cargo test -p dpp --all-features --lib -- identity_public_key public_key_in_creation should_only_admit
cargo test -p drive --lib -- identity::key::budget
cargo test -p drive-abci --lib -- key_limits validate_identity_public_keys_limits validate_fees_of_event
cargo test -p drive-abci --lib -- should_refuse_a_key_with_limits
cargo test -p dpp --all-features --lib -- identity_key_limits_update
cargo test -p drive --lib -- update_identity_key_limits
cargo test -p drive-abci --lib -- identity_key_limits_update

Rules and Guidelines

Do:

  • Read limits through IdentityPublicKeyGettersV1 and treat None as the normal case. Never match on V0 versus V1 to find out.
  • Build limited keys with with_limits, and convert between a key and a key in creation through the enums so the limits follow.
  • Give expires_at in milliseconds of block time. If an SDK offers a relative TTL, resolve it to an absolute time before signing, since the absolute time is what gets signed.
  • Keep every new cost term on the correct side of the budget rule. If a transition gains a cost that is chosen by the signer or priced up front, it belongs in required_from_key_budget.
  • Check any new field on the public key against the shielded sighash preimage, not only against PlatformSignable.

Do not:

  • Add a field to IdentityPublicKeyV0 or IdentityPublicKeyInCreationV0. Every key in state is one, and there is no version byte to tell old bytes from new.
  • Store the remaining budget in the key, or as a varint. The key is immutable, and a value that changes width changes storage, which would make the deduction a billed operation with a refund.
  • Credit a budget. Refunds go to the identity balance; a budget only goes down.
  • Move the expiry check into signature validation by threading the block time through the shipped processor. If that stage ever needs the time, it needs a new generation of what calls it.
  • Charge a penalty through a key that failed its limits. A refusal for a spent, exceeded or expired key is unpaid, whatever else is wrong with the transition.

Token Shielded Pools

From protocol version 14 a token can own a shielded pool: an Orchard pool that holds that token instead of credits. Holders move tokens between their identity balance and the pool with token transitions inside a batch, issuers mint, burn, release and sell straight into or out of it, document costs can be paid out of it, and three identity-less transitions move tokens with the fee paid from the credit shielded pool, so no identity appears at all. This chapter describes the storage, the configuration flag, the transitions, the validation rules, the block end bookkeeping, the queries and the client builders.

Why one pool per token

The Orchard construction Platform uses for credits has no asset base: a note carries a value but not an asset id, and the value balance the circuit proves is a single number. Mixing tokens in one pool would let a spend of token A create a note of token B. A token therefore gets its own pool, and each pool is a copy of the credit pool's layout rooted under the token.

One pool per token is what makes the arrangement safe, and it is also what it costs. A shielded pool hides a spend among the other notes in the same pool, so splitting the pools splits that crowd with them: a token's anonymity set is its own holders, not everyone on Platform who uses a shielded pool. A pool that holds one note hides nothing — the spend of that note names the shield that created it. A token's minimumPoolNotesForOutgoing is absent unless its issuer sets one, and absent reads as no threshold, so a new pool is spendable from its first note: that is where it is weakest rather than where it is strongest.

So the flag isolates a token's balances; on its own it does not make that token's transfers anonymous. An issuer turning it on is choosing a pool whose privacy grows with its use, and can require a floor of notes before tokens may leave it — see Configuration for the threshold and Validation for when it is read.

Storage layout

The credit shielded pool lives at [ShieldedBalances(52)]/"M". Token pools live under the tokens tree:

[Tokens(16)]
  [TOKEN_SHIELDED_POOLS_KEY(224)]            BigSumTree
    [token_id]                               SumTree (the pool)
      NOTES[128]            CommitmentTree, chunk power 11 (the note commitments and ciphertexts)
      NULLIFIERS[64]        ProvableCountTree (spent nullifiers)
      ANCHORS_IN_POOL[192]  anchor -> block height
      TOTAL_BALANCE[32]     SumItem (tokens currently shielded)
      ANCHORS_BY_HEIGHT[96] block height -> anchor (for pruning)

The five children use the same keys as the credit pool, so the drive primitives that insert notes and nullifiers, read balances and record anchors are shared: every credit pool method has a pool-agnostic form taking the pool path, and a token twin that supplies the token's path. The root of all token pools is a BigSumTree so the amount of every token that is shielded is one sum, which the token conservation check reads.

The root tree is created by the version 14 upgrade transition (transition_to_version_14) on an existing chain and by create_initial_state_structure version 4 on a new one. A pool's five trees are created when a contract with the flag is inserted or updated.

Configuration

TokenConfiguration gains a format version 1. It adds hasShieldedPool: bool, the optional minimumPoolNotesForOutgoing and the minimumPoolNotesForOutgoingChangeRules that govern it. A version 0 configuration behaves as hasShieldedPool: false.

minimumPoolNotesForOutgoing is how many notes a pool must hold before tokens may leave it. It is optional and absent by default, and absent reads as 0, no threshold. An issuer may set at most SystemLimits::max_token_pool_notes_for_outgoing (250), so none can name a floor its pool never reaches and strand every shielded balance. Unlike the flag, it is not immutable: it changes through TokenConfigUpdate under minimumPoolNotesForOutgoingChangeRules, which authorize no one when absent, so an issuer who wants to raise it later has to say so when the token is created. The format version is admitted by dpp.contract_versions.token_versions.token_configuration_format: protocol versions 13 and below allow only version 0, protocol version 14 allows versions 0 and 1. Contract create and update reject a token configuration outside the bounds with UnsupportedVersionError, so a pre-14 network never stores the flag.

The flag is immutable. A contract update that changes it is rejected with DataContractTokenConfigurationUpdateError for hasShieldedPool, because a pool that was enabled can hold notes that would become unspendable, and a pool that is enabled late would need a tree created under an existing token.

A pool also makes freezing and confiscation unenforceable: shielded notes belong to no identity account, so a holder who expects a freeze shields first and nothing can freeze or destroy those notes, because no account holds them for an issuer to act on. The notes themselves are not hidden — their commitments and ciphertexts are stored and queryable — but who owns one and how much it carries are. Rather than let an issuer advertise controls that cover only transparent balances, a token with hasShieldedPool must disable them permanently: freezeRules, unfreezeRules and destroyFrozenFundsRules must each authorize no one to take the action and have no admin action takers, so no later configuration update can switch them on. Contract create and update reject anything else with TokenShieldedPoolIncompatibleRulesError (10278). Pausing still works: every pool operation, inflows included (shield, mint, claim and purchase into the pool) and outflows (unshield, shielded transfer, burn from the pool), is rejected while the token is paused. A pooled token can never freeze an account, so the pool validators read and bill no frozen-account check.

Batch transitions

Seven operations are TokenTransition variants inside a Batch transition, like every other token operation. The identity signs the batch and pays the fee in credits. Tokens cannot pay fees, so unlike the credit pool nothing is carved from the bundle's value balance.

TransitionFlagsValue balanceExtra sighash dataEffect
TokenShieldoutputs only-amounttag(0x80), token_id, owner_idamount leaves the owner's balance and enters the pool as new notes.
TokenUnshieldspends and outputs+amounttoken_id, owner_id, recipient_id, amountNotes are spent; amount is credited to recipient_id; change comes back as new notes.
TokenShieldedTransferspends and outputs0token_id, owner_idNotes are spent and recreated; the pool balance is unchanged.
TokenMintToPooloutputs only-amounttag(0x81), token_id, minter_idAn authorized minter (manual minting rules, group actions supported) mints amount into new notes; the supply and the pool balance grow. Allowed only where mintingAllowChoosingDestination is set, since the notes' recipients are the minter's choice.
TokenBurnFromPoolspends and outputs+amounttoken_id, burner_id, amountAn authorized burner (manual burning rules, group actions supported) spends notes and destroys amount; the supply and the pool balance shrink. burner_id is the batch owner, or the proposer of a group action.
TokenClaimToPooloutputs only-amounttag(0x82), token_id, owner_idA distribution claim released into new notes instead of the claimant's balance; a perpetual claim names the cycle-aligned moment it claims up to so the amount is predictable.
TokenDirectPurchaseToPooloutputs only-token_counttag(0x83), token_id, owner_idThe buyer pays credits at the direct purchase price and the tokens are minted into new notes.

Each transition carries the Orchard bundle (actions, anchor, proof, binding_signature) next to the token base transition (token_id, contract id, contract position, identity contract nonce). The extra sighash data is bound into the Orchard sighash by the client and recomputed by consensus from the transition's own fields, so a bundle cannot be replayed against whatever its layout names. The layouts differ: the ones that spend bind the token, the owner and, where tokens leave the pool, the recipient and amount; the four that only create notes bind their kind, the token and the owner. The layouts are in dpp::shielded::sighash (token_unshield_extra_sighash_data, token_shielded_transfer_extra_sighash_data, token_burn_from_pool_extra_sighash_data and token_pool_output_only_extra_sighash_data).

Outputs-only bundles (shield, mint, claim, purchase) have no spends, so their anchor is not checked against the pool; the client builds them against the empty tree. Spending bundles must name an anchor the pool has recorded.

A group burn from the pool inherits a deadline from that requirement, and it is worth stating plainly because nothing in the group machinery announces it. The proposer's bundle spends notes, so it names an anchor, and the pool keeps an anchor only for shielded_anchor_retention_blocks. The bundle cannot be re-proved against a fresher anchor to buy more time: the action id covers the digest of the actions and the signatures are over a sighash that includes the anchor, so a bundle rebuilt on a new anchor is a different group action rather than the same one continued. Group actions have neither an expiry nor a cancellation, so a proposal that has not gathered enough signing power before its anchor is pruned can no longer close and no longer be withdrawn either — it simply remains open. A group intending to burn from a pool should therefore gather its signatures well inside the retention window, and an abandoned proposal should be treated as permanent. Closing this properly needs group actions to gain an expiry or a cancellation; until they do, the retention window is the real deadline.

The preimage of an outputs-only bundle binds its owner — the batch owner, or for a group action mint its proposer, whose bundle every other signer submits unchanged — so nobody else's transition can land a bundle lifted out of the mempool ahead of its author. What the preimage cannot stop is its owner submitting the same bundle again in a new transition. That repeat would land a second note with the same commitment and the same rho, hence the same nullifier, and only one of the two could ever be spent. The pool refuses it on the state side: each action's dummy nullifier, from which its note takes its rho, is recorded in the pool's nullifier tree when the bundle enters, and a bundle whose dummy nullifier is already there is rejected with NullifierAlreadySpentError. TokenPurchaseFromShieldedPool carries an outputs-only token bundle too and is checked the same way.

A mint or burn into the pool that goes through a group action stores TokenEvent::MintToPool / TokenEvent::BurnFromPool with a digest of the serialized actions (serialized_actions_digest), so every signer commits to exactly the same notes. Either bundle is therefore proven once, by the proposer: the digest fixes the bundle's actions and only the bundle's builder can sign it, so every other signer submits the proposer's bundle as it is and the sighash cannot depend on which signer's batch carries it. It binds the group action's proposer — as burner_id for a burn, as minter_id for a mint, and the batch owner when there is no group action. CheckTx verifies a batch's token bundles statelessly, seeing only the transition and not the stored group action, so it skips the bundle of a signer other than the proposer; state validation in the block verifies it against the proposer. No token history document is written for pool operations, so the ledger records no actor for them. The operations still leave public traces: spent nullifiers, the pool's note count and its anchors are all in state.

Documents paid from the pool

A document type whose action has a token cost can be paid out of the token's pool. TokenPaymentInfo gains a format version 1 that carries a TokenShieldedPayment: a spend bundle in the payment token's pool (amount, actions, anchor, proof, binding_signature) whose value balance is the document action's token cost. The identity still signs the batch and pays the credit fee; its token balance is never touched. The bundle's sighash binds the token id, the batch owner, the document's contract and id and the amount (document_token_payment_extra_sighash_data), so it cannot pay for another document.

The transformer rejects a payment whose amount is not the document type's cost (TokenShieldedPaymentAmountMismatchError, 40724) and a shielded payment on an action with no token cost (TokenShieldedPaymentNotRequiredError, 40725). Document base state validation version 1 skips the owner's balance and frozen-account checks for a shielded payment; the pool side (pool exists, token not paused, anchor, unspent nullifiers, pool balance, proof) is validated once the document action itself is valid. The lowering pays the cost from the pool: a TransferTokenToContractOwner effect is a token unshield into the contract owner's balance, a BurnToken effect a burn from the pool. The verification fee is charged like the batch pool transitions, and CheckTx admits the bundle under the identity contract nonce.

Identity-less transitions

Every batch transition above is signed by an identity that pays credits, so the chain sees which identity moved token T at height H. Three top-level state transitions remove the identity: each carries a bundle in the token's pool and a second spend bundle in the credit shielded pool that pays the fee, both authorized only by Orchard spend keys.

TypeTransitionToken bundleFee bundleEffect
26TokenShieldedTransferWithShieldedFeespends, value 0spends, value = feeNotes spent and recreated in the token pool; the fee leaves the credit pool.
27TokenUnshieldWithShieldedFeespends, value +amountspends, value = feeamount leaves the token pool into recipient_id's token balance.
28TokenPurchaseFromShieldedPooloutputs only, value -token_countspends, value = price + feetoken_count is minted into the token pool at the direct purchase price; the price is credited to the contract owner.

Every transition names the contract, the token position and the token id (which must derive from the two), the two bundles and credit_amount, the fee bundle's value balance. The token bundle's sighash binds the state transition type byte, the token id and the transparent fields (recipient and amount, or count and price); the fee bundle's sighash binds the type byte, the token id and a digest of the token bundle's actions (token_pool_fee_bundle_extra_sighash_data), so a fee bundle can only ever pay for that exact token bundle.

The processor treats them like the credit pool's own pool-paid transitions: structure validation checks both bundles, the minimum fee validation pins credit_amount to exactly the two-bundle fee (compute_token_pool_paid_shielded_fee: the base fee of each bundle plus the flat storage of what the transition writes outside the pools; a purchase adds the agreed price on top), both proofs are verified statelessly, and the transform validates the pools: the token owns a pool and is not paused, the token bundle's anchor is recorded and its nullifiers unspent in the token pool, the credit pool holds what leaves it and the fee bundle's anchor and nullifiers check out there, plus the transition's own rules (recipient exists for an unshield, the pricing schedule and the max supply for a purchase). Execution is a PaidFromShieldedPool event: the fee goes to the fee pools, the token side is the matching token operation, and a purchase credits the contract owner. Uniqueness is by the spent nullifiers of both bundles; a replay is an unpaid rejection. CheckTx admits their proofs under the generic pool-paid limiter. All three are gated on TOKEN_SHIELDED_POOL_INITIAL_PROTOCOL_VERSION.

A transfer proves its execution with the token pool nullifiers it spent, an unshield with the recipient's token balance, and a purchase with the token pool's total balance.

Validation

Structure validation checks the amount bounds, the action count against SystemLimits::max_shielded_transition_actions, the encrypted note sizes, a non-empty proof and a non-zero anchor.

The credit pool refuses an outgoing spend until it holds minimum_pool_notes_for_outgoing (250) notes, one floor for the whole network. A token pool's floor is the issuer's to choose: minimumPoolNotesForOutgoing is optional, and absent reads as 0, so by default a pool with a single note lets that note be spent at once. The price of that default is that a spend from a nearly empty pool is linkable to the shield that filled it; the price of a floor is that it traps a new pool's first depositors until enough notes accumulate, which is why none is set unless asked for. An issuer may set at most SystemLimits::max_token_pool_notes_for_outgoing (250), so no issuer can name a floor its pool never reaches and strand every shielded balance, and may change it later through TokenConfigUpdate under minimumPoolNotesForOutgoingChangeRules, which authorize no one when absent.

The floor counts note commitments, not holders. One bundle carries several actions, so a single depositor can reach a threshold alone: it tells holders how busy the pool should be before they leave it and guarantees no anonymity set. It applies to outflows with a visible destination: an unshield inside a batch, the identity-less TokenUnshieldWithShieldedFee, a burn from the pool, and a document's token cost paid from the pool. Transfers inside the pool are not limited.

State validation runs in this order, and the first failure is returned:

  1. The token base transition (contract exists, position valid, nonce).
  2. hasShieldedPool on the token's configuration, else TokenShieldedPoolNotEnabledError.
  3. The token is not paused, for every operation. Shield: the owner holds amount. Unshield: the recipient identity exists. Mint to pool: the configuration lets the minter choose the destination (mintingAllowChoosingDestination), the minting rules authorize the identity (or group) and the max supply is not exceeded. Burn from pool: the burning rules authorize the identity and the token is not paused. Claim to pool: the claim resolves exactly as a claim into a balance does (the shared resolve_token_claim). Purchase to pool: the pricing schedule and the max supply.
  4. Spending bundles: the anchor is recorded in the pool (InvalidAnchorError), no nullifier repeats within the bundle or is already spent (NullifierAlreadySpentError), and for an unshield the pool holds amount. Outputs-only bundles: no dummy nullifier repeats within the bundle or is already recorded in the pool (NullifierAlreadySpentError).
  5. Proof verification. The fee for it, compute_shielded_verification_fee(actions), is added as a precalculated operation before the Halo 2 proof and binding signature are checked, so a failed proof is a paid failure: the identity is charged, its nonce advances, and nothing else moves.

Batch state validation does not run in CheckTx. The mempool admission path (CheckTxProofVerifier) verifies the proofs of batch token transitions keyed by the identity contract nonce, so a proof is verified once per nonce before the block and not again for the same submission.

The transitions are gated on the protocol version: below 14 validate_is_allowed rejects a batch carrying any of them with StateTransitionNotActiveError.

Execution and conservation

The drive operations are composites of the credit pool primitives re-rooted under the token:

  • shield: remove amount from the owner's token balance, insert the bundle's dummy nullifiers, append the notes, add amount to the pool's TOTAL_BALANCE;
  • unshield: insert the nullifiers, append the notes, subtract amount from the pool balance, add amount to the recipient's token balance;
  • shielded transfer: insert the nullifiers, append the notes;
  • mint, claim and purchase to pool: insert the bundle's dummy nullifiers, append the notes, add amount to the pool balance and to the total supply (TokenMintToPool);
  • burn from pool: insert the nullifiers, append the change notes, subtract amount from the pool balance and from the total supply (TokenBurnFromPool).

Only a mint, claim, purchase or burn changes a token's total supply. calculate_total_tokens_balance version 1 reads the token pools BigSumTree and the block end conservation check requires identity balances + pool balances == total supply.

Block end

Every successfully executed token pool transition, document paid from a pool and identity-less token pool transition records its pool in StateTransitionsProcessingResult::token_shielded_pools_touched. A paid refusal records nothing: it bumps a nonce and writes to no pool, and the pool it named may not even exist, so letting one through would hand the anchor recorder a pool the chain does not hold. At block end record_token_shielded_pool_anchors (enabled by DRIVE_ABCI_METHOD_VERSIONS_V10) records each touched pool's current anchor if the commitment tree changed and prunes that pool's anchors older than shielded_anchor_retention_blocks, always keeping the newest one. Pruning is driven by touches rather than by an interval because there can be many pools; an idle pool keeps a valid anchor to spend against.

Queries

The six shielded pool queries (getShieldedPoolState, getShieldedNotesCount, getShieldedAnchors, getMostRecentShieldedAnchor, getShieldedEncryptedNotes, getShieldedNullifiers) take an optional token_id. Without it they target the credit pool; with a 32-byte token id they target that token's pool and answer with the same response shape. A token id is rejected with InvalidArgument before protocol version 14 or when it is not 32 bytes. The proof verifier routes on the same field to the token twins of the verify functions (verify_token_shielded_pool_state and the rest), and the Rust SDK exposes TokenShieldedPoolQuery, TokenShieldedEncryptedNotesQuery and TokenShieldedNullifiersQuery.

A token pool transition proves its execution like the token transfer it resembles: a shield proves the owner's balance, an unshield proves the recipient's balance, and a shielded transfer proves the spent nullifiers in the token's pool.

Client builders

dpp::shielded::builder provides build_token_shield_transition, build_token_unshield_transition, build_token_shielded_transfer_transition, build_token_mint_to_pool_transition, build_token_burn_from_pool_transition, build_token_claim_to_pool_transition and build_token_direct_purchase_to_pool_transition. They prove the bundle, bind the extra sighash data, and call the batch constructors (new_token_shield_transition and siblings) which sign with the identity key. build_document_shielded_token_payment builds the bundle of a TokenPaymentInfo::V1. For document creation, call Document::set_id_for_creation with the creation nonce before building the payment: the proof must bind the final document ID, not the factory placeholder. build_token_shielded_transfer_with_shielded_fee_transition, build_token_unshield_with_shielded_fee_transition and build_token_purchase_from_shielded_pool_transition build the identity-less transitions from a TokenPoolSpender (the token pool notes and keys) and a ShieldedFeePayer (the credit pool notes and keys). The wasm bindings expose a wrapper per transition, TokenPaymentInfo accepts a shieldedPayment, and TokenConfiguration accepts hasShieldedPool and reports formatVersion.

Fees

See Shielded Transaction Fees. In short: the identity pays the metered cost of the writes plus the proof verification fee, exactly like ShieldFromIdentity, for every batch transition and for a document paid from the pool; the identity-less transitions carve the two-bundle fee from the credit pool bundle.

Contract Keywords

A data contract describes its documents with a JSON schema per document type, and Platform reads a set of keywords in those schemas: some from JSON Schema, most of its own. The chapters of this part take each keyword, or a small group that works together, and say what it does, how to write it, what is checked when a contract is registered and when a document is written, what a later contract update may do with it, and which errors it produces. The chapters of the Data Model and Drive parts explain the internals behind them and are linked from each chapter.

Everything here follows the document meta-schema of protocol version 14, packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json. Every document type schema is validated against it when a contract is registered or updated, and the parser (try_from_schema) checks the rules a JSON schema cannot express. When a chapter and the meta-schema disagree, the meta-schema is right and the chapter is out of date.

Where keywords go

A contract's documentSchemas maps each document type name to its schema. Keywords sit at three levels:

"post": {
  "type": "object",
  "documentsMutable": true,
  "canBeDeleted": true,
  "indices": [
    { "name": "byOwner", "properties": [{ "$ownerId": "asc" }, { "$createdAt": "asc" }] }
  ],
  "properties": {
    "text": { "type": "string", "maxLength": 280, "maxBytes": 560, "position": 0 },
    "replyTo": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "refersTo": { "type": "deletableDocument", "documentType": "post" },
      "position": 1
    }
  },
  "required": ["$createdAt", "text"],
  "additionalProperties": false
}
  • Document type keywords sit at the top of the schema (documentsMutable, canBeDeleted, indices, required). They say what may happen to a document of the type and who may do it.
  • Index keywords sit inside an entry of indices (name, properties).
  • Property keywords sit inside a property's schema (type, maxLength, maxBytes, position, refersTo). Most are ordinary JSON Schema; the rest are Platform's own.

The contract around the document types has keys of its own, and a config object: see Contract-Level Keys and config.

Reading the chapters

Each chapter opens with a short table for each keyword:

  • Since is the first protocol version at which the keyword can be used.
  • On update is what a contract update may do with the keyword on a document type that already exists. Fixed means adding, removing and changing it are all refused. A document type the update adds may use any keyword, as a new contract may.
  • Errors are consensus errors, written ErrorName (code). See Error Codes for the code ranges.

A contract update that breaks an update rule is refused with one of two errors, depending on which check catches it. DocumentTypeUpdateError (40212) comes from the comparison of the parsed document types, which judges flags such as documentsMutable by their meaning. IncompatibleDocumentTypeSchemaError (10246) comes from the comparison of the two JSON schemas, which judges property keywords such as refersTo or maxLength by their text. Top-level required and indices have errors of their own (10276 and 10217). Because the schema comparison reads text, an edit that changes how a keyword is written but not what it means, such as writing out a default or switching to the documentsAverageable shorthand, is refused with 10246.

Protocol versions

The document meta-schema has changed three times:

Meta-schemaProtocol versionsWhat it added
v01 to 11The original keywords. A document type key the meta-schema did not know was ignored.
v112Unknown document type keys are refused. The count, sum and average keywords.
v213keepsTransferHistory, keepsPurchaseHistory, keepsPricingHistory.
v314References, typed arrays, requiredSince, immutable, ttl, propertyConstraints, deleteConstraints, actionFees, moderation deletion, ranked, time-range and integer-range indexes, index-only types, and the rest marked 14 in these chapters.

Most keywords of v0 took effect at protocol version 1. The exceptions are tokenCost (9) and the index keyword countable (12).

The complete language

Every key a contract can write, grouped by where it goes. Since is the protocol version from which the key works. Read more links to the key's section in this part and, where there is one, to the chapter with the internals. A dotted name such as tokenCost.<action>.amount is a key written inside the ones before it.

The contract

KeyTakesWhat it doesSinceRead more
$formatVersion"0" or "1"The contract's serialization format. "1", the default from 9, carries groups, tokens, keywords, description and the timestamps.1Contract keys
ididentifierThe contract's id: a hash of ownerId and the identity nonce of the create transition.1Contract keys
ownerIdidentifierThe identity that registers the contract, and the only one that may update it.1Contract keys
versioninteger1 at creation; every update raises it by exactly one.1Contract keys
configobjectContract-wide settings, below.1config
documentSchemasobject of document typesThe document types by name, each written with the document type keys.1documentSchemas
schemaDefsobjectDefinitions any property may point at with $ref.1Document Shape
groupsobjectSets of identities, each member with a voting power, whose approval some token actions need.9Data Contracts
tokensobjectThe contract's tokens, by position. Their configuration is not covered in this part.9Data Contracts · Creating a Basic Token
keywordsup to 50 strings of 3 to 50 bytesSearch keywords, for the keyword search contract.9keywords and description
descriptionstring of 3 to 100 bytesA short description, for the keyword search contract.9keywords and description
createdAt, updatedAt, createdAtBlockHeight, updatedAtBlockHeight, createdAtEpoch, updatedAtEpochnumbersWhen the contract was created and last updated. Set by the platform, never written.9Contract keys
contractGroup{ "admins", "name", "description" }On the create transition, beside the contract: registers a contract group, a set of contracts the signer owns, with up to 16 admins, a name of 1 to 64 characters and a description of 1 to 256.14Contract Groups
contractGroupMembershipsup to 16 { "contractGroupId", "member" }On the create transition: enrols the new contract ("contract"), one of its document types ({ "documentType": ... }) or one of its tokens ({ "token": ... }) in contract groups.14Contract Groups

config

KeyTakesWhat it doesSinceRead more
canBeDeletedboolean, default falseWhether the contract may ever be deleted. No transition deletes a contract today.1canBeDeleted
readonlyboolean, default falsetrue: the contract can never be updated.1readonly
keepsHistoryboolean, default falseDrive keeps every version of the contract.1keepsHistory
documentsKeepHistoryContractDefaultboolean, default falsedocumentsKeepHistory for a document type that does not say.1Document type defaults
documentsMutableContractDefaultboolean, default truedocumentsMutable for a document type that does not say.1Document type defaults
documentsCanBeDeletedContractDefaultboolean, default truecanBeDeleted for a document type that does not say.1Document type defaults
requiresIdentityEncryptionBoundedKey, requiresIdentityDecryptionBoundedKey0 unique, 1 multiple, 2 multiple with a pointer to the latestLets identities bind encryption or decryption keys to the whole contract, and says how they are kept.1Bounded key requirements · Contract Bounds
sizedIntegerTypesboolean, default trueStores each integer in the smallest width its bounds allow, instead of 8 bytes.9sizedIntegerTypes
moderationobjectMakes the contract moderated: which lists it keeps and who moderates.14moderation · Contract Moderation
moderation.banlist, .suspensions, .warningsboolean, default falseKeeps a banlist, a suspension list, a warning list. A banned or suspended identity cannot act on the contract's documents; a warning bars nothing.14The Model
moderation.moderators{ "$type": ... }Who moderates: "contractOwner", "appointedModerators" with identities (1 to 16), or "elected" with the keys below.14moderation
moderators.seatContestableboolean, required when electedWhether a seated team may later be challenged.14Elected Moderation
moderators.challengeCoolDownseconds, two weeks to three yearsHow long a seated team is safe from a challenge after a seat change. Required when the seat is contestable, refused when it is not.14Elected Moderation
moderators.moderatedDocumentTypesobject: document type → abilitiesThe document types the team moderates, each with its abilities: ban, suspend, warn, deleteDocuments.14Elected Moderation
moderators.interim{ "$type": ... }Who moderates until a team is seated: "contractOwner", "appointedModerators", "notYetUsable" (the moderated types cannot be used yet) or "noModeration".14Elected Moderation
moderators.joinWindow, .voteWindowsecondsHow long applicants may join an election, and how long masternodes then vote.14Elected Moderation
moderators.electionDelaysecondsHow long after the contract's creation the first election may be called.14Elected Moderation
moderators.maxAddedModerators0 to 15, default 0How many members the seated leader may add after the election.14Elected Moderation
moderators.ownerProtectedboolean, default falseProtects the contract owner from the seated team.14Elected Moderation

Document type

KeyTakesWhat it doesSinceRead more
type"object"Required. A document is an object.1type
propertiesobject of 1 to 100 propertiesThe document's properties, each written with the property keys.1properties
requiredarray of namesThe properties every document holds. A system time or height listed here is recorded.1required
additionalPropertiesfalseRequired: a document holds only the declared properties.1additionalProperties
minProperties, maxPropertiesintegerHow many properties a document holds.1minProperties and maxProperties
dependentRequiredobjectA property that requires others when present.1dependentRequired
$comment, descriptionstringNotes; consensus ignores them.1$comment and description
$schema, $defsadded by the platformThe meta-schema URL and the contract's schemaDefs. A document type writing either is refused.1$schema and $defs
transientarray of top-level namesProperties validated on the transition but never stored.1transient · internals
documentsMutableboolean, default truefalse: documents cannot be replaced.1documentsMutable
immutablearray of top-level names and { property, when }Properties frozen at creation, or while a condition holds, on a mutable type.14immutable · internals
canBeDeletedboolean or "onlyWhenConsumed", default truefalse: a document's owner cannot delete it. "onlyWhenConsumed" (14): only a create that consumes it deletes it.1canBeDeleted
deleteConstraintsobject of named rules, at most 16Rules, in the grammar of propertyConstraints, the stored document meets for its owner to delete it: a poll deleted only while no vote points at it.14deleteConstraints
retractedWhenone condition, as an immutable entry's whenThe replace a banned or suspended owner may still make on a moderated contract: one whose written document meets the condition.14retractedWhen · internals
moderatorAbilitiesobject: delete, deleteWithin (seconds), deleteKeepsRecord, deleteRefundsOwner, deleteSettled (leader, approvals, approversPredateDocument), deleteKeepsFields (array of property paths), changeFields (array of top-level names)What the contract's moderators may do to documents of the type: delete them, within a window after their last change, past it only when so many members of a seated team agree (the leader among them when the rule says so, members the leader added only for documents written after their addition unless the rule says otherwise), with or without a removal record keeping the fields that stay public and a refund to the owner, and write the fields only they write.14Moderator Abilities · delete · internals
ttlseconds, 3600 to 31536000The platform deletes each document this long after its creation.14Time To Live · internals
creationRestrictionMode0 anyone, 1 contract owner, 2 nobodyWho may create documents.1creationRestrictionMode
transferable0 never, 1 alwaysWhether an owner may give a document to another identity.1transferable
tradeMode0 none, 1 direct purchaseWhether an owner may set a price and anyone buy at it.1tradeMode
documentsKeepHistoryboolean, default falseDrive keeps every revision of every document.1documentsKeepHistory
keepsTransferHistory, keepsPurchaseHistory, keepsPricingHistoryboolean, default falseRecords every transfer, purchase or price update in the document history contract.13History
signatureSecurityLevelRequirement1 critical, 2 high (default), 3 mediumThe weakest key level that may sign a transition on the type.1signatureSecurityLevelRequirement · Security Level
requiresIdentityEncryptionBoundedKey, requiresIdentityDecryptionBoundedKey0 unique, 1 multiple, 2 multiple with a pointer to the latestLets identities bind encryption or decryption keys to the type, and says how they are kept.1Signing and Keys · Contract Bounds
ownerRefersToa refersTo declarationA reference the writer must meet, on types whose documents are never transferred or traded.14ownerRefersTo · internals
creatorRefersToa refersTo declarationA reference the creator must meet, on types whose documents can be transferred or traded.14creatorRefersTo
propertyConstraintsobject of named rules, at most 16Rules over several properties every created or replaced document meets. See the operators.14propertyConstraints · internals
tokenCostobject keyed by actionToken payments for actions on documents. See the keys.9Token Costs · Fees
actionFeesobject keyed by actionCredit fees for actions on documents, paid to the owner's and moderators' pots. See the keys.14Action Fees · Fees
indicesarray of 1 to 10 indexesThe indexes documents are queried by, each written with the index keys.1Indexes · internals
documentsCountablebooleanKeeps a count of the type's documents.12documentsCountable · internals
documentsSummableproperty nameKeeps the sum of one integer property over the type's documents.12documentsSummable · internals
documentsAverageableproperty nameShorthand for documentsCountable plus documentsSummable.12documentsAverageable
rangeCountable, rangeSummable, rangeAverageablebooleanProvable counts, sums or averages over ranges of document ids.12Document type range keys
indexOnlybooleanDocuments are never stored whole: the index entries are the rows.14indexOnly · internals
entryPayloadarray of 1 to 16 namesOn an index-only type, properties carried in each entry's value instead of a key.14entryPayload

Property

KeyTakesWhat it doesSinceRead more
typestring, integer, number, boolean, object, arrayThe kind of value. An array is a byte array or a typed array.1type
positionintegerThe property's place in the stored document. Required; top-level positions run 0, 1, 2 with no gap.1position · Document Serialization
minLength, maxLengthintegerA string's length in characters.1Strings
patternregular expressionA string must match it. Needs maxLength of at most 50000.1Strings
formatdate-time, date, time, email, idn-email, hostname, ipv4, ipv6, uri, regexA string must have this format. Needs maxLength of at most 50000.1Strings
maxBytes1 to 65535The most UTF-8 bytes a string may take.14maxBytes · internals
minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOfnumberNumeric bounds. minimum and maximum also decide an integer's stored width.1Numbers
enum, constvaluesThe values allowed, or the one value allowed.1enum and const
byteArraytrueMakes an array a string of bytes, stored raw.1Byte arrays and identifiers
contentMediaType"application/x.dash.dpp.identifier"Makes a 32-byte array an identifier.1Byte arrays and identifiers
minItems, maxItems, uniqueItems, containsinteger, boolean, schemaA byte array's length in bytes, or a typed array's number of elements (at most 1024); no repeats; an element that matches.1Arrays
itemsan element schemaMakes an array a typed array whose elements all follow this schema.14Typed Arrays · internals
properties, required, additionalProperties, minProperties, maxProperties, dependentRequiredas on a document typeA nested object's members and its bounds.1Objects
$ref"#/$defs/<name>"Uses a definition from the contract's schemaDefs.1$ref
$id, $comment, description, examplesannotationsNotes; consensus ignores them.1Annotations
requiredSincecontract versionLets an update add a required property that older documents may leave out.14requiredSince · internals
distinctFroma property path or "$ownerId"An identifier must differ from another identifier of the document, or from the owner.14distinctFrom · internals
encryptedFor{ "recipient", "recipientKey", "senderKey", "scheme" }Declares how an encrypted byte array was made: whose keys, which scheme.14encryptedFor · internals
encryptedFor.recipientidentifier property path or "$ownerId"The identity the value is encrypted to.14encryptedFor
encryptedFor.recipientKey, .senderKeyinteger property pathsThe properties holding the recipient's and the sender's key ids.14encryptedFor
encryptedFor.scheme"ecdh-secp256k1-aes256-cbc"How the ciphertext is made.14The scheme
generatedFrom{ "function", "params" }The platform generates the string from other properties of the document; on arrival when a document leaves it out.14generatedFrom · internals
generatedFrom.function"sys.stringTransformations.homographSafeASCII"The system function that generates the value: sys.stringTransformations. lowercase, uppercase, capitalize, camelCase, snakeCase or homographSafeASCII.14Functions
generatedFrom.paramsproperty pathsThe properties the function reads, in order.14Params
refersToa declarationWhat an identifier points at, checked when a document is written. See the keys.14References · internals

A typed array's element (items) takes type, enum, minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf, minLength, maxLength, pattern, format, minItems and maxItems (bytes of a byte array element), byteArray, contentMediaType, maxBytes, distinctFrom, refersTo, $comment and description. It takes no position, const, uniqueItems or examples.

refersTo

KeyTakesWhat it doesSinceRead more
typea target belowWhat the value points at.14Targets
type: "identity"The value is the id of an existing identity.14identity
type: "contract"The value is the id of an existing data contract.14contract
type: "token"The value is the id of an existing token.14token
type: "permanentDocument"The value is the id of a document whose type can never lose its documents; with findBy, part of the key that finds it; with inList, an element of its list.14permanentDocument
type: "deletableDocument"The value is the id of a document that can be deleted, or with findBy part of the key that finds it; checked again on every replace.14deletableDocument
type: "identityPublicKey"The value names an identity key that exists and is not disabled.14identityPublicKey
documentTypedocument type nameThe referenced document type.14documentType
contractIdidentifierThe contract holding documentType, when it is not this one.14contractId
findBy1 to 10 entries: referenced property → ".", "$ownerId", a path or a functionFinds the document through the unique index of documentType over exactly these properties, in any order; "." is the value itself. Without it the value is the document's $id.14findBy · internals
findBy.<property>{ "function": "sys.hash.sha256d", "params" }A function entry: the referenced property holds the hash of params (paths, { "const": text }, "." for a value without a path), which fills that part of the key, finding a commitment made earlier. At most one.14Commit and reveal · internals
where1 to 10 entries { "<referenced property>": "<referring value>" }Checked on the document found: each referenced property (or $ownerId, $creatorId, $id) must equal the referring property. A value of "$ownerId", the writer, makes a write gate. Never finds the document.14where
minimumAgeBlocks1 to 4294967295Beside a findBy function: the commitment was created at least this many blocks before the create.14Commit and reveal
consumetrueBeside a findBy function, on a deletableDocument with the where entry "$ownerId": "$ownerId", into a type with canBeDeleted true or "onlyWhenConsumed" and no deleteConstraints: the create deletes the writer's commitment.14Commit and reveal
inListtyped array pathOn a permanentDocument whose findBy is { "$id": <property> }: the list on that document the value must be in.14List Elements · internals
keyIdPropertyinteger property pathOn an identity property: the property holding the key id.14keyIdProperty and identityProperty
identityProperty"$ownerId", "$creatorId" or a pathOn a key id property: whose key it is.14keyIdProperty and identityProperty
keyRequirements.purposeauthentication, encryption, decryption, transfer, voting, ownerThe key's purpose.14keyRequirements
keyRequirements.boundTodocument type nameThe key must be bound to that document type of this contract.14keyRequirements
contractRequirements.moderation"elected", "electionOpen"The contract declares an elected team, or one whose election may be called.14contractRequirements · Elected Moderation
contractRequirements.minimumAgeSeconds, .minimumSecondsSinceUpdatesecondsThe contract was created, or last changed, at least this long ago.14contractRequirements
contractRequirements.owner"self", "other"The contract is owned by the writer, or by someone else.14contractRequirements
contractRequirements.readonly, .keepsHistorytrueThe contract can never be updated, or keeps history.14contractRequirements
contractRequirements.ownerProtectedbooleanThe contract's elected team does, or does not, protect its owner.14contractRequirements
anyOf, allOf2 to 4 operandsIn place of type: at least one, or every, operand holds. Nest at most 4 deep.14Expressions · internals

tokenCost and actionFees

<action> is one of create, replace, delete, transfer, update_price and purchase.

KeyTakesWhat it doesSinceRead more
tokenCost.<action>.tokenPosition0 to 65535, requiredWhich token is charged.9Token Costs
tokenCost.<action>.amountat least 1, requiredHow many tokens the action costs.9Token Costs
tokenCost.<action>.contractIdidentifierThe contract whose token is charged, when it is not this one.9contractId
tokenCost.<action>.effect0 to the contract owner (default), 1 burnWhat happens to the tokens paid.9effect
tokenCost.<action>.gasFeesPaidBy0 document owner (default), 1 contract owner, 2 prefer contract ownerWho the contract owner offers to have pay the gas. Accepted from 9, acted on from 14.14gasFeesPaidBy · Fees
tokenCost.<action>.optionalboolean, default falseA transition may skip the token and pay in credits.14Optional costs · Fees
actionFees.pricing"feeMultiplier" (default), "fixed"Whether the amounts scale with the epoch's fee multiplier.14Action Fees
actionFees.<action>.ownercreditsPaid into the contract owner's pot.14The pots and the claim
actionFees.<action>.moderatorscreditsPaid into the moderators' pot. Needs moderation.14The pots and the claim · Fee Pots

propertyConstraints

A rule is one condition, in propertyConstraints and in deleteConstraints alike. Conditions:

KeyTakesHolds whenSinceRead more
equal, notEqual[a, b]The two sides are equal, or differ: integer expressions, strings or identifiers.14Conditions
lessThan, lessThanOrEqual, greaterThan, greaterThanOrEqual[a, b]The integer comparison holds.14Conditions
in[a, [values]]a takes one of two or more listed integers, strings or identifiers.14Conditions
present, absenta pathThe document holds the property, or leaves it out (or null, or an object with no member present).14Conditions
anyOf, allOftwo or more conditionsAt least one, or every, condition holds, checked in order.14Evaluation order
nota conditionThe condition does not hold.14Conditions
ifThen, ifThenElse[if, then], [if, then, else]The second condition holds when the first does (and, for ifThenElse, the third when it does not); only the branch taken is evaluated.14Conditions
notIn[a, [values]]a takes none of the listed values.14Conditions
startsWith, endsWith[text, affix]A string starts or ends with another, byte for byte.14Conditions
contains[array, value]A typed array holds an element equal to the value.14Conditions

Expressions:

KeyTakesValueSinceRead more
an integer100Itself.14Expressions
a path"price", "meta.total"An integer or boolean property's value; 0 when left out.14Expressions
add, multiplytwo or more operandsThe sum or product.14Arithmetic
subtract, divide, modulo, power[a, b]The difference, Euclidean quotient or remainder, or power.14Arithmetic
min, max, abstwo or more operands, or one for absThe least, the greatest, or the absolute value.14Expressions
countOf, sumOf[type, filter?], [type, property, filter?]How many documents of a type of the contract match the filter, or the total of an integer property over them, from its count or sum trees.14Totals of other documents
ifAbsent[path, default]The property's value, or the default when left out (an integer, or a string for a string property).14Expressions
length, byteLengtha string pathA string's length in characters, or in UTF-8 bytes.14Expressions
countan array pathThe elements of a typed array, or the bytes of a byte array.14Expressions
countPresent[path, path, ...]How many of two or more properties the document holds, each as present tests it.14How many of a group
$createdAt, $updatedAt, $transferredAt, $createdAtBlockHeight, $updatedAtBlockHeight, $transferredAtBlockHeight, $createdAtCoreBlockHeight, $updatedAtCoreBlockHeight, $transferredAtCoreBlockHeighta pathA time or height the document records, when listed in required.14Times and heights
consta stringA string constant, or a base58 identifier, as one side of equal or notEqual.14Strings
$ownerIdThe document's owner, as an identifier side.14Identifiers and $ownerId
$idThe document's id, as the value a countOf or sumOf filter matches by: { "pollId": "$id" }, the documents pointing at it.14Totals of other documents

Index

KeyTakesWhat it doesSinceRead more
name1 to 32 characters, requiredThe index's name, unique in the type.1name
properties1 to 10 { "<path>": "asc" }The indexed properties, in order. A flat index of an index-only type leaves it out. From protocol version 14 a path may read through a reference, "<reference property>.<field>", a value of the referenced document the document does not store.1properties · Values of Referenced Documents · internals
uniquebooleanNo two documents share the indexed values.1unique
nullSearchableboolean, default truefalse leaves out documents whose indexed values are all null.1nullSearchable
contestedobjectMatching values are decided by a masternode vote, not first come.1Contested Indexes · internals
contested.resolution0 vote with lock, 1 vote without lockHow the contest is decided. 1 from 14.1The keys
contested.fieldMatches[{ "field", "regexPattern" }]Which values are contested.1The keys
contested.descriptionstringA note; consensus ignores it.1The keys
countable"notCountable", "countable", "countableAllowingOffset" or booleanKeeps a document count per indexed value.12countable · internals
summableproperty nameKeeps the sum of an integer property per indexed value.12summable · internals
averageableproperty nameShorthand for countable plus summable.12averageable
rangeCountable, rangeSummable, rangeAverageablebooleanProvable counts, sums or averages over ranges of the indexed value.12Index range keys
rankedCountableboolean or { "at": ... }Orders the indexed values by document count, for "top K" queries; at names the levels ranked.14rankedCountable · internals
rankedSummable, rankedAverageableboolean, or { "at": ... } on a summableOffCountIndex indexOrders them by sum, or by average.14Ranked Indexes
timeRange{ "on", "range", "step", "phase", "ttl" }Buckets a system timestamp into time windows, for trending queries.14Time-Range Indexes · internals
timeRange.on"$createdAt", "$updatedAt", "$transferredAt"The timestamp to bucket: the index's first property.14The keys
timeRange.range, .stepsecondsEach window's length, and the time between window starts.14The keys
timeRange.phaseseconds, default 0Shifts the window boundaries.14The keys
timeRange.ttlseconds, at most one weekExpires the index's entries after their window; on an index-only type, the rows leave this index.14The keys · internals
integerRange{ "on", "range", "step", "phase" }Buckets an integer property into value windows, for counts and rankings per band.14Integer-Range Indexes
integerRange.onproperty nameThe integer property to bucket: the index's first property, required.14The keys
integerRange.range, .step, .phaseintegers, phase default 0Each window's length, the distance between window starts, and the shift of the window boundaries.14The keys
terminalproperty name or listOn an index-only type, what keys each entry in place of the document id.14terminal
preallocatedbooleanOn an index-only type, creates the index's trees with the referenced document.14preallocated · internals
summableOffCountIndexindex nameOn an index-only type, keeps one counter per group of how many entries the named index keeps for it, in place of an entry per document: count(*) and the sum read the named index's entries, and the average divides them by the groups.14summableOffCountIndex
outlivesDeletebooleanOn an index-only type's time window with a ttl, a delete leaves the index's entries to expire, and a create writes over one already there.14outlivesDelete · internals
skipIfAbsenttrue or property namesA document missing a property of the skip set writes no entry into the index.14skipIfAbsent · internals

System properties

PropertyHoldsRecordedSinceRead more
$idThe document's id.always1$id
$ownerIdThe identity that owns the document.always1$ownerId
$revision1 at creation, raised by every replace, transfer, price update and purchase.on types whose documents can change hands or content1$revision
$createdAt, $updatedAt, $transferredAtBlock times, in milliseconds, of the creation, the last replace or price update, and the last transfer or purchase.when listed in required1Timestamps
$createdAtBlockHeight, $updatedAtBlockHeight, $transferredAtBlockHeightPlatform block heights of the same events.when listed in required1Block heights
$createdAtCoreBlockHeight, $updatedAtCoreBlockHeight, $transferredAtCoreBlockHeightCore chain block heights of the same events.when listed in required1Block heights
$creatorIdThe identity that created the document.on transferable or tradeable types of format-1 contracts10$creatorId
$moderatedAt, $moderatedByBlock time and moderator of the last write of the fields only moderators write.on types listing moderatorAbilities.changeFields, once a moderator writes them14$moderatedAt and $moderatedBy

Limits

The first three limits come from the meta-schema, the rest from protocol version 14's SystemLimits. A contract over a limit is refused at registration.

LimitValueApplies to
Properties per object100properties, at the top and in each nested object
Indexes per document type10indices
Properties per index10an index's properties
max_field_value_size5120 bytesany one value a document stores (DocumentFieldMaxSizeExceededError, 10417)
max_typed_array_items1024a typed array's maxItems
max_references_per_document256references one document carries
max_reference_operands4operands in one anyOf or allOf of a reference
max_reference_expression_depth4nesting of reference expressions
max_property_constraints16rules in one propertyConstraints
max_property_constraint_nodes32nodes in one rule
min_document_ttl_seconds, max_document_ttl_seconds3600, 31536000ttl
max_time_range_ttl_seconds604800a timeRange index's ttl
max_contested_summed_value_magnitude134217728 (2^27)the minimum and maximum of a summed property on a type with a contested index
max_expiring_signed_summed_value_magnitude134217728 (2^27)the minimum and maximum of a summed property that admits negative values, on a type with a ttl

Document Shape

A document type's schema is a JSON Schema object with some Platform keywords added. The keywords in this chapter give a document its outline: it is an object, it has these properties and no others, some of them must be present, and a few JSON Schema rules hold over the document as a whole. What a single property may hold is described in Property Schemas; what may happen to a document (replace, delete, transfer) has chapters of its own.

Example

The DashPay profile type, with its indexes left out:

"profile": {
  "type": "object",
  "properties": {
    "avatarUrl": { "type": "string", "format": "uri", "minLength": 1, "maxLength": 2048, "position": 0 },
    "avatarHash": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 1 },
    "avatarFingerprint": { "type": "array", "byteArray": true, "minItems": 8, "maxItems": 8, "position": 2 },
    "publicMessage": { "type": "string", "minLength": 1, "maxLength": 140, "position": 3 },
    "displayName": { "type": "string", "minLength": 1, "maxLength": 25, "position": 4 }
  },
  "minProperties": 1,
  "dependentRequired": {
    "avatarUrl": ["avatarHash", "avatarFingerprint"],
    "avatarHash": ["avatarUrl", "avatarFingerprint"],
    "avatarFingerprint": ["avatarUrl", "avatarHash"]
  },
  "required": ["$createdAt", "$updatedAt"],
  "additionalProperties": false
}

A profile holds at least one of its five properties, and never any other. The three avatar properties come together or not at all. No property of its own is required, but the platform records the time each profile was created and last updated, because required lists $createdAt and $updatedAt.

type

WhereThe top of a document type. Required.
Value"object", the only value allowed
Sinceprotocol version 1
On updateFixed (IncompatibleDocumentTypeSchemaError, 10246)
ErrorsJsonSchemaError (10101) at registration

A document is always an object: a set of named properties. Properties inside a document have their own type, described in Property Schemas.

properties

WhereThe top of a document type. Required.
ValueAn object mapping 1 to 100 property names to property schemas
Sinceprotocol version 1
On updateProperties may be added, never removed (IncompatibleDocumentTypeSchemaError, 10246). An added property is optional, or required with requiredSince.
ErrorsMissingPositionsInDocumentTypePropertiesError (10411), JsonSchemaError (10101), both at registration

properties declares every property a document of the type may hold. Each value is the property's schema: its type, its bounds and the Platform keywords that apply to it (see Property Schemas).

Rules at registration:

  • A document type has 1 to 100 properties. A nested object has the same limit.
  • A property name is 1 to 64 letters, digits or underscores (^[a-zA-Z0-9_]{1,64}$). Protocol versions 1 to 13 also admitted -; from protocol version 14 it is refused.
  • Every property has a position, a number. The top-level positions must run 0, 1, 2 and so on, with no gap and no number used twice (MissingPositionsInDocumentTypePropertiesError, 10411). A document is stored with its properties in position order and without their names, so the numbers are what tie the stored bytes to the schema.

On update, a new property takes the next free position. It is optional unless it is listed in required with requiredSince set to the version the update creates. A property that already exists can never be removed, since stored documents may hold it.

additionalProperties

WhereThe top of a document type (required), and every property of type object
Valuefalse, the only value allowed
Sinceprotocol version 1
On updateFixed (10246)
ErrorsJsonSchemaError (10101): a document holding a property its type does not declare

A document holds only the properties its type declares. Writing additionalProperties: false says so; Platform accepts no other value.

required

WhereThe top of a document type. A property of type object has its own required for its members (see Objects).
ValueAn array of names, none repeated: properties of the type, and the system timestamps and block heights
DefaultAbsent: every property is optional and no timestamp is recorded
Sinceprotocol version 1
On updateMay gain only a property the same update adds, annotated with requiredSince; may lose nothing (DataContractInvalidRequiredFieldsUpdateError, 10276)
ErrorsJsonSchemaError (10101): a created or replaced document leaves out a required property

required names two kinds of thing:

  • The type's own properties. Every created or replaced document must hold them. A required property is also stored without the presence byte an optional one carries (unless it is transient), which is why the list is so hard to change later.
  • System timestamps and block heights: $createdAt, $updatedAt, $transferredAt, and their BlockHeight and CoreBlockHeight forms. The writer does not supply these. Listing one tells the platform to record it on every document of the type; one that is not listed is never recorded. See System Properties.

Some keywords only work with a timestamp in the list: ttl needs $createdAt, and a time-range index needs the timestamp it buckets.

On a contract update, from protocol version 14:

  • A property the update adds may be required if it carries requiredSince equal to the contract version the update creates. See requiredSince.
  • An existing optional property may not become required.
  • Nothing may be removed from the list.
  • A system timestamp or block height may not be added. So the set of values a type records is fixed once the type exists.

Each of these is refused with DataContractInvalidRequiredFieldsUpdateError (10276). The required list of a nested object is fixed (IncompatibleDocumentTypeSchemaError, 10246).

minProperties and maxProperties

WhereThe top of a document type; also on properties of type object
ValueAn integer, 0 or more
Sinceprotocol version 1 (declared in the meta-schema from 12)
On updateFixed (IncompatibleDocumentTypeSchemaError, 10246)
ErrorsJsonSchemaError (10101)

These are the JSON Schema keywords: a document must hold at least minProperties and at most maxProperties of its own properties. System properties are not counted. The DashPay profile above uses minProperties: 1 so that an empty profile cannot be written.

Meta-schema v0, which covered protocol versions 1 to 11, did not list these keywords at the top of a document type, but it did not refuse keys it did not know either. A contract of that time could use them, and its documents were validated against them: the DashPay contract, registered at protocol version 1, uses minProperties and dependentRequired. From protocol version 12 the meta-schema lists them and checks their values.

dependentRequired

WhereThe top of a document type; also on properties of type object
ValueAn object mapping a property name to an array of property names
Sinceprotocol version 1 (declared in the meta-schema from 12)
On updateEntries, and names within an entry, may be removed, and so may the whole keyword; nothing may be added (IncompatibleDocumentTypeSchemaError, 10246)
ErrorsJsonSchemaError (10101)

The JSON Schema keyword: when a document holds the property named by a key, it must also hold every property in that key's array. In the example, a profile with an avatarUrl must also have an avatarHash and an avatarFingerprint. Removing an entry only lets more documents through, which is why removal is the only change allowed.

$comment and description

WhereThe top of a document type; also on any property
ValueA string
Sinceprotocol version 1
On updateFree: may be added, changed or removed
Errorsnone

Notes for people reading the contract. Consensus does not act on them.

$schema and $defs

WhereAdded by the platform; a document type does not write them
Value$schema: the document meta-schema's URL. $defs: the contract's schemaDefs.
Sinceprotocol version 1
ErrorsInvalidContractStructure (10231): a document type that writes either key

When Platform reads a document type, it adds two keys before validating the schema:

  • $schema, the URL of the document meta-schema, so the schema is checked against it.
  • $defs, holding the contract's schemaDefs: definitions shared by every document type of the contract. A property then refers to one with $ref, as "$ref": "#/$defs/<name>".

A document type that writes either key itself is refused. Definitions live only at the contract level, in schemaDefs, and are checked as part of each document type: 1 to 100 of them, each named like a property and each a property schema. A contract update may add definitions but not remove them, and a change to a definition follows the update rules of the keywords it holds (IncompatibleDataContractSchemaError, 10213). See Contract-Level Keys and config.

Limits on the whole type

  • Name. A document type's name, its key in documentSchemas, is 1 to 64 letters, digits or underscores (InvalidDocumentTypeNameError, 10415). Up to protocol version 13 it could also hold -.
  • Nesting. The schema, with the definitions it reaches through $ref, nests at most 256 levels of objects and arrays (DataContractMaxDepthExceedError, 10200). A $ref that does not resolve, or that leads back to itself, is refused (InvalidJsonSchemaRefError, 10207).
  • Known keys only. From protocol version 12 a key the meta-schema does not know is refused at the top of a document type (JsonSchemaError, 10101). Before 12 such a key was accepted without being checked. The keys are listed in the overview.

See also

Property Schemas

Each entry of a document type's properties is a property schema: JSON Schema (draft 2020-12), limited to the keywords in this chapter, plus three Platform keywords that say how a value is stored (position, byteArray and contentMediaType). The schema is checked when the contract is registered, and every created or replaced document is validated against it. Platform keywords with more to them, such as maxBytes, refersTo, distinctFrom, encryptedFor, generatedFrom, requiredSince and a typed array's items, have chapters of their own.

KeywordApplies toOn update
typeevery propertyFixed
positionevery propertyFixed
minLength, maxLength, pattern, formatstringsLoosened or removed only
minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOfintegers and numbersLoosened or removed only, keeping an integer's width; multipleOf fixed
enum, constanyenum may gain values; const may be removed
byteArray, contentMediaTypebyte arraysFixed
minItems, maxItems, uniqueItems, containsarraysLoosened or removed only; contains fixed
properties, required, additionalProperties, minProperties, maxProperties, dependentRequiredobjectsMembers may be added; the rest fixed, except dependentRequired may lose entries
$refanyFixed
$id, $comment, description, examplesanyFree ($id may only be added)

Each section gives the error a contract update gets for breaking its rules. Most are refused with IncompatibleDocumentTypeSchemaError (10246), from the comparison of the old and new schemas; a change to how a value is stored is refused with DocumentTypeUpdateError (40212).

How property schemas are checked

When a contract is registered or updated:

  • The schema is checked against the document meta-schema. A keyword the meta-schema does not allow where it is written, or a value of the wrong shape, is refused with JsonSchemaError (10101).
  • The parser reads each property's type and bounds, and refuses what the meta-schema cannot express (InvalidContractStructure, 10231).
  • The schema is compiled for validating documents. A pattern that is not a valid regular expression, or a format the validator does not know, is refused here (JsonSchemaError, 10101).

When a document is created or replaced:

  1. Every string and byte array value is at most 5120 bytes, whatever the schema allows (DocumentFieldMaxSizeExceededError, 10417). From protocol version 14 a value nested more than 256 levels deep is refused as well (ValueError, 10103).
  2. The document's properties are validated against the schema. Each failure is reported as a JsonSchemaError (10101) naming the keyword and the property.
  3. Platform's own checks, which JSON Schema cannot express, come next: maxBytes and propertyConstraints among them.

A transfer, a price update and a purchase carry no property values, so they are not validated against the schema again.

The schema also decides how each value is stored, since a stored document holds no property names or type tags:

PropertyStored as
integer1, 2, 4 or 8 bytes, chosen by its bounds (see Numbers)
number8 bytes, a 64-bit floating point number
boolean1 byte
stringa length prefix, then the UTF-8 bytes
byte array with minItems equal to maxItemsthe bytes, with no prefix
any other byte arraya length prefix, then the bytes
identifier32 bytes
objecta length prefix, then its members
typed arrayan element count, then the elements (see Typed Arrays)

An optional property adds one byte in front that says whether it is present. See Document Serialization for the exact encoding.

type

WhereEvery property; also the elements of a typed array
ValueOne of "string", "integer", "number", "boolean", "object", "array"
Sinceprotocol version 1
On updateFixed (IncompatibleDocumentTypeSchemaError, 10246)
ErrorsJsonSchemaError (10101): a document value of another type

type is a single name. A list of types, and "null", are refused at registration: every stored value needs one known encoding.

An array is one of two things:

  • a byte array, with byteArray: true: a string of bytes, such as a hash or an identifier;
  • from protocol version 14, a typed array, with an items schema: a list of values of one scalar type. See Typed Arrays.

An array that is neither is refused.

position

WhereEvery property, at every level. Not on the elements of a typed array.
ValueAn integer, 0 or more
Sinceprotocol version 1
On updateFixed (10246)
ErrorsMissingPositionsInDocumentTypePropertiesError (10411) at registration

A document is stored with its values one after another and no property names. position is the property's place in that sequence.

  • The top-level properties of a document type must use the positions 0, 1, 2 and so on, with no gap and no number used twice (10411). A top-level property without a position is refused.
  • The members of an object need a position too. Number them from 0 within the object; only the top level is checked for gaps.
  • A property that takes its schema from a $ref writes its position next to the $ref.
  • A property added by a contract update takes the next free position. An existing position can never change, or stored documents would be read in the wrong order.

Strings

KeywordsminLength, maxLength, pattern, format
WhereProperties of type string, and the string elements of a typed array
ValueminLength, maxLength: an integer, 0 or more, counting characters. pattern: a regular expression. format: the name of a JSON Schema format, such as "uri"
Sinceprotocol version 1
On updatemaxLength may be raised or removed, and minLength lowered or removed. pattern and format may be removed. None of them may be added, and no other change is allowed (10246).
ErrorsJsonSchemaError (10101)
"username": {
  "type": "string",
  "minLength": 3,
  "maxLength": 63,
  "pattern": "^[a-zA-Z0-9_]+$",
  "position": 0
}

A username is 3 to 63 letters, digits or underscores.

  • minLength and maxLength count characters, and a character takes 1 to 4 bytes in UTF-8. To cap the stored size, add maxBytes. Whatever maxLength says, no single string may exceed 5120 bytes (10417).
  • pattern is written in the syntax of Rust's regex crate, which has no lookaround and no backreferences. A pattern that does not compile is refused at registration (10101).
  • format is checked on every document. The validator knows date-time, date, time, email, idn-email, hostname, ipv4, ipv6, uri and regex. Any other format, uuid and uri-reference included, is refused at registration (10101).
  • A string with pattern or format must declare a maxLength of at most 50000, so that matching stays cheap.
  • A string used in an index needs a maxLength of at most 63 (InvalidIndexedPropertyConstraintError, 10205). See Indexes.

Numbers

Keywordsminimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf
WhereProperties of type integer or number, and such elements of a typed array
ValueA number. On an integer element of a typed array, minimum and maximum are integers.
Sinceprotocol version 1
On updatemaximum and exclusiveMaximum may be raised or removed, and minimum and exclusiveMinimum lowered or removed, unless that changes how an integer is stored (DocumentTypeUpdateError, 40212). None may be added, and multipleOf is fixed (10246).
ErrorsJsonSchemaError (10101)
"rating": { "type": "integer", "minimum": 1, "maximum": 5, "position": 2 }

A rating is a whole number from 1 to 5, and is stored in one byte.

A number is always stored in 8 bytes. An integer is stored in the smallest width its minimum and maximum allow, when the contract's config has sizedIntegerTypes on (the default; see Contract-Level Keys and config):

minimummaximumStored as
0 or moreup to 2551 byte, unsigned
0 or moreup to 655352 bytes, unsigned
0 or moreup to 42949672954 bytes, unsigned
0 or morehigher8 bytes, unsigned
below 0both bounds within -128 to 1271 byte, signed
below 0both bounds within -32768 to 327672 bytes, signed
below 0both bounds within -2147483648 to 21474836474 bytes, signed
below 0otherwise8 bytes, signed

With only a minimum, the integer takes 8 bytes, unsigned when the minimum is 0 or more. With only a maximum, it takes the unsigned width the maximum gives, so an integer that may be negative needs a minimum too. With neither, an enum of integers picks the width from its smallest and largest members; without one, the integer takes 8 bytes, signed. exclusiveMinimum and exclusiveMaximum do not affect the width. With sizedIntegerTypes off, every integer takes 8 bytes, signed.

Stored documents and index entries hold each integer at its width, so a contract update may not change the width or the sign. Raising maximum past the width, lowering minimum below 0, removing a bound, or adding an enum value outside the width is refused with DocumentTypeUpdateError (40212); so is turning sizedIntegerTypes on when it would change an existing integer's width. A change that keeps the width is accepted.

enum and const

WhereAny property. enum also on the string, integer, number and boolean elements of a typed array; const never on elements.
Valueenum: an array of one or more values, none repeated. const: one value.
Sinceprotocol version 1
On updateenum may gain values but not lose one; the keyword may be removed, not added. const may be removed, not added or changed (10246). An enum value that changes an integer's width is refused (DocumentTypeUpdateError, 40212).
ErrorsJsonSchemaError (10101)
"status": { "type": "string", "enum": ["open", "closed", "archived"], "position": 3 }

enum lists the values a property may take and const the one value it must take. Both only narrow what documents may hold, so an update may widen them (more enum values, or no keyword at all) but never narrow them, which could leave stored documents invalid. On an integer without bounds, enum also sets the stored width (see Numbers).

Byte arrays and identifiers

KeywordsbyteArray, contentMediaType
WhereProperties of type array, and the array elements of a typed array
ValuebyteArray: true, the only value. contentMediaType: "application/x.dash.dpp.identifier" makes the byte array an identifier.
Sinceprotocol version 1
On updateFixed (10246). The length bounds follow the rules of Arrays, but a byte array may not switch between a fixed and a variable length, or change its fixed length (DocumentTypeUpdateError, 40212).
ErrorsJsonSchemaError (10101)
"authorId": {
  "type": "array",
  "byteArray": true,
  "minItems": 32,
  "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "position": 1
}

authorId is an identifier: the 32-byte id of an identity, a document, a contract or a token.

  • byteArray: true makes an array a string of bytes. Its minItems and maxItems count bytes. When the two are equal the bytes are stored as they are; otherwise they carry a length prefix.
  • contentMediaType: "application/x.dash.dpp.identifier" makes a byte array an identifier. It must come with byteArray: true, minItems: 32 and maxItems: 32, and it may not carry uniqueItems. An identifier is shown in base58, and it is the kind of property distinctFrom and most refersTo targets are declared on.
  • A byte array used in an index needs a maxItems of at most 255 (InvalidIndexedPropertyConstraintError, 10205).
  • On a typed array, contentMediaType belongs on the items, not on the array.

Arrays

KeywordsminItems, maxItems, uniqueItems, contains
WhereProperties of type array. minItems and maxItems also on byte array elements of a typed array.
ValueminItems, maxItems: an integer, 0 or more. uniqueItems: a boolean. contains: a schema.
Sinceprotocol version 1
On updatemaxItems may be raised or removed and minItems lowered or removed (10246), within the byte array rule above (40212); a typed array keeps its maxItems. uniqueItems may be removed or set to false, not added (10246). contains is fixed (10246).
ErrorsJsonSchemaError (10101)
  • minItems and maxItems count bytes on a byte array and elements on a typed array. A typed array must declare maxItems, at most 1024.
  • uniqueItems: true on a typed array refuses a document that repeats an element. On a plain byte array it refuses a repeated byte. It is refused on an identifier, and on the elements of a typed array.
  • contains is the JSON Schema keyword: at least one element must match the schema it holds.

Objects

Keywordsproperties, required, additionalProperties, minProperties, maxProperties, dependentRequired
WhereProperties of type object
ValueThe same as at the top of a document type: see Document Shape
Sinceprotocol version 1
On updateMembers may be added, never removed; required and additionalProperties are fixed; dependentRequired may lose entries, not gain them (10246). minProperties and maxProperties are fixed (10246).
ErrorsJsonSchemaError (10101)
"records": {
  "type": "object",
  "properties": {
    "identity": {
      "type": "array",
      "byteArray": true,
      "minItems": 32,
      "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "position": 0
    }
  },
  "minProperties": 1,
  "additionalProperties": false,
  "position": 5
}

An object groups members under one property, as DPNS groups a name's records.

  • An object declares properties, 1 to 100 members each with a position, and additionalProperties: false, unless it takes its schema from a $ref.
  • Its required names the members every value of the object must hold. It cannot change after the type exists, so a member added by an update is optional.
  • An object is stored with its members inline. An object cannot be indexed, but a member can: an index names it by its dotted path, such as records.identity.

$ref

WhereAny property, except the elements of a typed array
Value"#/$defs/<name>": a definition in the contract's schemaDefs
Sinceprotocol version 1
On updateFixed (10246)
ErrorsInvalidJsonSchemaRefError (10207) at registration

$ref lets several document types share one schema, written once in the contract's schemaDefs:

"schemaDefs": {
  "address": {
    "type": "object",
    "properties": {
      "street": { "type": "string", "maxLength": 100, "position": 0 },
      "city": { "type": "string", "maxLength": 50, "position": 1 }
    },
    "required": ["street", "city"],
    "additionalProperties": false
  }
}

A property of any document type of the contract then reads "shippingAddress": { "$ref": "#/$defs/address", "position": 2 }.

  • Only local references, starting with #, are allowed. The platform places schemaDefs under $defs in every document type (see $schema and $defs).
  • A reference that does not resolve, or that leads back to itself, is refused (10207).
  • The property keeps its own position; the definition supplies everything else.
  • The $ref itself cannot change on update. The definition it points at can, under the rules of the keywords it holds (IncompatibleDataContractSchemaError, 10213).

Annotations

Keywords$id, $comment, description, examples
WhereAny property. $comment and description also on the elements of a typed array.
Value$id: a string starting with #. $comment, description: a string. examples: an array of values.
Sinceprotocol version 1
On update$comment, description and examples: free. $id: may be added, not removed or changed (10246).
Errorsnone

Notes for people and tools reading the contract. They do not change what a document may hold.

See also

Typed Arrays

A typed array is a list property: a document holds any number of values, up to a limit, all of one simple type. A post's tags, the scores of a match or the members of a team are typed arrays. The items keyword makes an array a typed array and gives the schema every element follows. Before protocol version 14 an array property had to be a byte array.

WhereA property of type array, at the top of a document type or inside an object, in place of byteArray: true
ValueThe schema of one element: an integer, a number, a string, a boolean, a byte array or an identifier
Sinceprotocol version 14
On updateitems may be neither added nor removed (IncompatibleDocumentTypeSchemaError, 10246). The keywords inside it follow their own update rules, and a change to how an element is stored is refused (DocumentTypeUpdateError, 40212).
ErrorsJsonSchemaError (10101); on elements, DocumentPropertyMaxBytesExceededError (10421), DocumentPropertyNotDistinctError (10419) and the reference errors of refersTo

Example

"tags": {
  "type": "array",
  "minItems": 0,
  "maxItems": 10,
  "uniqueItems": true,
  "items": { "type": "string", "minLength": 1, "maxLength": 32, "maxBytes": 64 },
  "position": 3
}

A post holds up to 10 tags, none repeated, each 1 to 32 characters and at most 64 bytes.

The elements may also be identifiers that point at other documents. The moderation charters contract lists the reasons a moderation team may act on like this:

"reasons": {
  "type": "array",
  "minItems": 0,
  "maxItems": 64,
  "uniqueItems": true,
  "items": {
    "type": "array",
    "byteArray": true,
    "minItems": 32,
    "maxItems": 32,
    "contentMediaType": "application/x.dash.dpp.identifier",
    "refersTo": { "type": "permanentDocument", "documentType": "reason" }
  },
  "position": 2
}

A charter names up to 64 distinct reason documents, and each one must exist when the charter is written.

How it works

On the array, minItems and maxItems count elements, and uniqueItems: true refuses a document that repeats one. A list that is too long or too short, repeats an element, or holds an element of the wrong type is refused with JsonSchemaError (10101).

On the elements, the items schema is checked for every element, as a property of that schema would be:

  • the JSON Schema keywords: an element's enum, bounds, length and pattern (JsonSchemaError, 10101);
  • maxBytes on string elements (DocumentPropertyMaxBytesExceededError, 10421), the error naming the element, such as tags[2];
  • distinctFrom on identifier elements: every element must differ from the named property (DocumentPropertyNotDistinctError, 10419);
  • refersTo on identifier elements: each element is checked as a single reference would be, in list order, and the first one that fails refuses the write with that reference's error, naming the element (reasons[2] for the third). An empty or absent list checks nothing.

A replace re-checks the references of the elements the stored list did not hold, and every element when the reference's rules call for it. See References on the Elements for the details.

Storage. A typed array is stored inline in the document: a count of its elements, then each element encoded as a required property of the element's type would be. An identifier element takes 32 bytes, an integer element the width its bounds give it (see Numbers), a fixed-size byte array element its bytes, and a string element a length prefix and its bytes. Elements never carry a presence byte. The reasons list above is therefore one count byte and 32 bytes per reason. The 5120-byte limit on a single value applies to each element, not to the list.

When a document is read from JSON, identifier and byte array elements are converted from their string form element by element, as a single identifier or byte array is.

Rules at registration

The array:

  • declares items and maxItems, and not byteArray;
  • has a maxItems of at most 1024, and a minItems no higher than its maxItems;
  • carries no contentMediaType, refersTo, distinctFrom or maxBytes of its own: these go on the items.

The element schema (items):

  • is written inline: a $ref is refused;
  • has a type of integer, number, string, boolean or array, and an array element must be a byte array (byteArray: true). Objects and lists of lists are refused;
  • takes these keywords: type, minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf, minLength, maxLength, pattern, format, minItems, maxItems (bytes, on a byte array element), enum, byteArray, contentMediaType, maxBytes, distinctFrom, refersTo, $comment and description. Anything else, position, const, uniqueItems and examples included, is refused. A one-value enum does what const would, and an update can still widen it;
  • may carry an enum only on a string, integer, number or boolean element, and every member must be a value of the element's type;
  • on an integer element, has an integer minimum and maximum, the minimum no higher than the maximum;
  • may carry refersTo only on an identifier element, and not with the identityPublicKey target: its key id is a single sibling property, which cannot pair with many elements.

Where a typed array cannot be used:

  • in an index (InvalidIndexPropertyTypeError, 10206), or as an index-only type's terminal or entry payload: nothing is written per element;
  • on either side of a where entry in a reference;
  • as an operand of a propertyConstraints rule, other than in a present or absent test;
  • with encryptedFor, which only a byte array takes.

References count against the limit of 256 per document at maxItems each: a list of up to 64 references counts as 64. An immutable property may not hold a list of deletableDocument references (see Mutability).

A meta-schema violation is refused with JsonSchemaError (10101) and the other rules with InvalidContractStructure (10231).

On update

  • items may not be added or removed, so a byte array cannot become a typed array or the reverse (10246).
  • Inside items, each keyword follows its own update rule (see Property Schemas): for example maxLength and maxBytes may be raised, and enum may gain values. refersTo and distinctFrom are fixed.
  • A change to how an element is stored is refused (DocumentTypeUpdateError, 40212): an integer element whose width or sign would change (by its bounds or its enum), or a byte array element that would switch between fixed and variable size or change its fixed size.
  • On the array, maxItems may be raised, up to 1024 and within the reference limit, and minItems lowered. uniqueItems may be removed, not added.

See also

System Properties

Every document carries a few values the platform manages rather than the writer: its id, its owner, and, on some types, its revision, its creator, the times it was created, updated and transferred, and who last moderated it and when. Their names start with $. A document type does not declare them in properties. It names them where it wants to use them: in required, to have a timestamp recorded, in indices, to query by them, and in the keywords that accept one, such as a reference's where.

PropertyHoldsRecorded
$idthe document's idalways
$ownerIdthe identity that owns the document nowalways
$revisionhow many times the document has changed, plus oneon types whose documents can be replaced, transferred or sold, or that keep fields for their moderators
$createdAt, $updatedAt, $transferredAtblock times of the creation, last update and last transferwhen listed in required
$createdAtBlockHeight and the other heightsPlatform and Core block heights of the same eventswhen listed in required
$creatorIdthe identity that created the documenton types whose documents can be transferred or sold
$moderatedAt, $moderatedBythe block time and the moderator of the last write of the fields only moderators writeon types that keep such fields, once a moderator writes them

Example

"listing": {
  "type": "object",
  "documentsMutable": true,
  "transferable": 1,
  "tradeMode": 1,
  "indices": [
    { "name": "byCreator", "properties": [{ "$creatorId": "asc" }, { "$createdAt": "asc" }] },
    { "name": "byOwner", "properties": [{ "$ownerId": "asc" }, { "$updatedAt": "asc" }] }
  ],
  "properties": {
    "title": { "type": "string", "minLength": 1, "maxLength": 100, "position": 0 }
  },
  "required": ["$createdAt", "$updatedAt", "$transferredAt", "title"],
  "additionalProperties": false
}

Every listing records when it was created, last updated and last transferred, because required lists the three times. Listings can be replaced, transferred and sold, so each one also carries a revision and the id of its creator, and the two indexes find them by who made them and by who holds them now.

$id

WhereEvery document
ValueAn identifier: 32 bytes
RecordedAlways
Sinceprotocol version 1
ErrorsInvalidDocumentTransitionIdError (10405): a create whose id is not the one derived for it. SystemPropertyIndexAlreadyPresentError (10208): an index that names $id.

A document's id is derived from the contract id, the owner's id, the document type's name, entropy chosen by the client and, from protocol version 14, the identity contract nonce of the create transition. Consensus derives it again for every create and refuses a transition that carries another. The id never changes. See Document ID Generation.

Documents are already stored by id, so an index may not name $id. A reference's where may name it as a key, the referenced side, and findBy names it to find the document holding a list (see References).

$ownerId

WhereEvery document
ValueAn identifier: the id of an identity
RecordedAlways
Sinceprotocol version 1
ErrorsDocumentOwnerIdMismatchError (40102): a replace, transfer, price update or delete signed by an identity that does not own the document

The owner is the identity that created the document, until a transfer or a purchase hands it to another. Only the owner may replace, transfer, reprice or delete it with a document transition. A document can also leave by other paths, a moderators' deletion or an expired ttl; see Deletion.

$ownerId may be indexed, and several keywords read it, where it usually stands for the writer of the document:

  • distinctFrom: "$ownerId" makes a property differ from the owner.
  • propertyConstraints: a rule may compare an identifier property with $ownerId.
  • References: a where entry, a findBy source and identityProperty may name it, and ownerRefersTo checks the owner itself.
  • encryptedFor: "recipient": "$ownerId" marks a message the writer encrypts to themself.

$revision

WhereDocuments of a type whose documents can be replaced (documentsMutable, true by default), transferred (transferable: 1) or sold (tradeMode: 1), or that keeps fields only its moderators write (moderatorAbilities.changeFields), even when its documents cannot be replaced
ValueAn integer, from 1
RecordedOn those types; absent on every other
Sinceprotocol version 1
ErrorsInvalidDocumentRevisionError (40106): a transition whose revision is not the stored one plus one

A new document has revision 1. Every replace, transfer, price update and purchase raises it by one, and the transition must state the new revision: the stored revision plus one. A transition built against an older copy of the document is refused, rather than silently overwriting a newer one. A moderator's change of the fields a type keeps for its moderators raises it by one too; the moderation transition states no revision, the platform sets it, and an owner's replace built before the change is refused. A type whose documents can never change after creation carries no revision. $revision is not one of the system properties an index may name.

Timestamps

Properties$createdAt, $updatedAt, $transferredAt
ValueA block time, in milliseconds since the Unix epoch
RecordedOnly when listed in the document type's required
Sinceprotocol version 1
On updateThe set is fixed: a contract update may not add one to required or remove one (DataContractInvalidRequiredFieldsUpdateError, 10276)

The platform sets these from the block that processes the transition; the writer never supplies them. A timestamp that is not in required is never recorded, and documents of the type do not have it.

EventSets
Createevery listed timestamp, so $updatedAt and $transferredAt start at the creation time
Replace$updatedAt
Price update$updatedAt
Transfer$transferredAt
Purchase$transferredAt

Timestamps may be indexed. Some keywords need one in required, since they read it:

  • ttl counts from $createdAt.
  • moderatorAbilities.deleteWithin counts from $updatedAt, or from $createdAt on a type that does not record $updatedAt (see Deletion).
  • A time-range index needs the timestamp it buckets.

Block heights

Properties$createdAtBlockHeight, $updatedAtBlockHeight, $transferredAtBlockHeight, $createdAtCoreBlockHeight, $updatedAtCoreBlockHeight, $transferredAtCoreBlockHeight
ValueThe BlockHeight forms: the Platform block height. The CoreBlockHeight forms: the Core chain height recorded with that block.
RecordedOnly when listed in the document type's required
Sinceprotocol version 1
On updateThe set is fixed, as for the timestamps (10276)

The same three events as the timestamps, measured in blocks instead of time. Each is set on the same events as the timestamp of its name, and may be indexed.

$creatorId

WhereDocuments of a type that sets transferable: 1 or tradeMode: 1, in a contract of format 1 whose config is version 1 or later
ValueAn identifier: the id of the identity that created the document
RecordedOn those types, from protocol version 10
Sinceprotocol version 10
ErrorsUndefinedIndexPropertyError (10209): an index naming $creatorId on a type that does not record it

On a type whose documents can change hands, $ownerId follows the document while $creatorId stays with the identity that created it: a transfer or a purchase never changes it. On a type whose documents never change hands the creator is always the owner, and no $creatorId is recorded. Neither is it on a contract of format 0 or with a config of version 0, whatever its types allow.

$creatorId is not written in required or properties. It may be named:

  • in an index;
  • as the key of a reference's where entry (the referenced side), and as a key reference's identityProperty;
  • by creatorRefersTo, which checks the creator. A type that does not record creators may not declare it (InvalidContractStructure, 10231).

$moderatedAt and $moderatedBy

WhereDocuments of a type that lists moderatorAbilities.changeFields
Value$moderatedAt: a block time, in milliseconds since the Unix epoch. $moderatedBy: an identifier, the id of the moderator.
RecordedOnce a moderator of the contract writes the fields the type keeps for its moderators; absent until then
Sinceprotocol version 14
ErrorsInvalidContractStructure (10231): an index naming either on a type that lists no changeFields, or in a unique index. UndefinedIndexPropertyError (10209): an index naming either before protocol version 14.

The two record the last time a moderator wrote the fields only moderators write, and who did. They are set together, from the block that processes the write, and never by the writer:

EventSets both
A moderator's field change (changeDocumentFields)to the block's time and the moderator who signed it
A create that sets such a field, by a document owner who moderates the contractto the block's time and the owner
A replace that changes, adds or removes such a field, by an owner who moderatesto the block's time and the owner

Nothing else moves them: a replace that leaves those fields as they were, a transfer, a purchase and a price update keep them, and a restore puts them back as they were. They do not change $updatedAt, which stays the owner's. A document no moderator has written carries neither, and a type that keeps no fields for its moderators never records them.

Both may be indexed, on a type that lists changeFields, so an application can find what its moderators handled, by whom and in what order:

"indices": [
  { "name": "byModerator", "properties": [{ "$moderatedBy": "asc" }, { "$moderatedAt": "asc" }] }
]

A unique index may not name them: a moderator's change would be refused because another document holds the same stamp. They are not written in required or properties, and no other keyword reads them.

See also

requiredSince

requiredSince lets a contract update add a property that every new document must hold. Documents already stored were written without it and stay valid: the property is required only of documents written under the contract version the annotation names, or a later one. Reach for it when an app needs a new mandatory field and the contract is already in use.

WhereA top-level property listed in the document type's required
ValueA contract version: an integer from 1 to 4294967295
DefaultAbsent: a property listed in required is required of every document
Sinceprotocol version 14
On updateMay only appear on a property the update adds, equal to the contract version the update creates (DataContractInvalidRequiredFieldsUpdateError, 10276). An existing annotation may not be added, changed or removed (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsDataContractInvalidRequiredFieldsUpdateError (10276), InvalidContractStructure (10231), IncompatibleDocumentTypeSchemaError (10246), all at registration; JsonSchemaError (10101) for a new document without the property

Example

Version 2 of a contract has a post type with one property, text. The update to version 3 adds a required language:

"post": {
  "type": "object",
  "properties": {
    "text": { "type": "string", "maxLength": 280, "position": 0 },
    "language": { "type": "string", "minLength": 2, "maxLength": 8, "requiredSince": 3, "position": 1 }
  },
  "required": ["text", "language"],
  "additionalProperties": false
}

Every post created or replaced under version 3 or later must have a language. Posts written under versions 1 and 2 have none, and remain valid as they are.

How it works

From protocol version 14, every create and replace stamps the document with the version of the contract it was written under. A transfer or a purchase keeps the stamp the document had, since it does not rewrite the document's properties. See the contract version stamp.

  • New writes are held to the new schema. A create must include the property. A replace sends the whole document again, so replacing a document written before the update must add the property too; the document is then stamped with the current version. Documents catch up one at a time, as they are replaced.
  • Older documents are left alone. A document stamped below the property's requiredSince may lack it. It can still be read, transferred, sold and deleted. A document last written before protocol version 14 has no stamp, and counts as older than every annotation.
  • Storage follows the stamp. A required property is stored without the presence byte an optional one carries. A property with requiredSince is stored as required in documents stamped at or above its version, and as optional in older ones. The latest contract alone therefore tells how to read every stored document.
  • Readers must expect gaps. An app that reads the type should handle documents without the property: every document stamped below the annotation may lack it.
  • No index on it. A contract update cannot add an index to an existing document type (see Indexes), so a property added this way cannot be indexed on that type.

Rules at registration

  • requiredSince sits only on a top-level property, and only on one listed in required. A nested property, or one that is not required, is refused (InvalidContractStructure, 10231). It cannot go on the elements of a typed array.
  • The value is never higher than the contract's own version (10276): a property cannot be scheduled to become required later.
  • On a new contract, which is version 1, the value may only be 1 (10276).

On an update to an existing document type, which always creates the version one above the current one:

  • A property the update adds may be required only if it carries requiredSince equal to that new version. Without the annotation, or with any other value, the update is refused (10276).
  • An existing property may not become required, with or without the annotation (10276).
  • No property may leave required (10276).
  • A property's existing requiredSince may not be changed or removed, and one may not be added to an existing property (IncompatibleDocumentTypeSchemaError, 10246).

A document type that the update adds has no older documents. Every requiredSince in it must still equal the version the update creates (10276).

See also

transient

transient lists properties that a transition must carry but a stored document never holds. The value is validated with the rest of the document, can be read by the checks that run on the transition, and is then dropped. Reach for it when a write has to prove something with a value that should not be kept, such as a secret salt.

WhereThe top of a document type
ValueAn array of names of top-level properties
DefaultAbsent: every property is stored
Sinceprotocol version 1. A replace drops the values from protocol version 14.
On updateFixed (IncompatibleDocumentTypeSchemaError, 10246). The list is compared as a set, so reordering or repeating a name is no change.
ErrorsInvalidContractStructure (10231) at registration

Example

The DPNS domain type, trimmed to two properties:

"domain": {
  "type": "object",
  "documentsMutable": false,
  "properties": {
    "label": { "type": "string", "minLength": 3, "maxLength": 63, "position": 0 },
    "preorderSalt": {
      "type": "array",
      "byteArray": true,
      "minItems": 32,
      "maxItems": 32,
      "description": "Salt used in the preorder document",
      "position": 1
    }
  },
  "required": ["label", "preorderSalt"],
  "transient": ["preorderSalt"],
  "additionalProperties": false
}

Registering a name takes two steps. A preorder document first commits to a hash of the salt and the name, and the domain document then reveals both. Every domain create must carry its 32-byte preorderSalt, and DPNS checks it against the preorder. Once the domain is created the salt has done its job, and the stored domain does not hold it.

How it works

  • Checked like any other value. A transient value is on the transition when the document is validated, so the schema holds it to its rules: a required transient property must be present, and its bounds apply. maxBytes and distinctFrom apply to it too, and so do checks such as the DPNS one above.
  • Dropped before storing. After validation, a create drops the transient values before the document is written. From protocol version 14 a replace drops them the same way. Before 14 a replace stored the values it carried, so a replaced document could hold values its create had dropped.
  • Dropped by top-level name. Only top-level properties can be transient. When a transient property is an object, the whole object is dropped with everything in it.
  • Never stored, never found. No stored document holds a transient value, so a query cannot find one and a later reader cannot see it.
  • Sent again on replace. A replace is validated as a whole document, so a required transient property must be carried by every replace, not only the create.
  • Storage. A transient property is stored as optional, with a presence byte, even when it is required: in a stored document it is always absent.

Rules at registration

From protocol version 14, a document type is refused (InvalidContractStructure, 10231) when:

  • an entry does not name a top-level property of the type. A nested path, a system property or an undeclared name is refused; to drop a nested value, list the object around it;
  • an index reads a transient property, or a property inside a transient object. Every stored document would lack the value, so the index could find nothing and a unique index would enforce nothing;
  • a reference reads one where the value would have to be stored: a refersTo findBy may not read one on either side (the params of a findBy function may, being read from the create), a where may not name one on its referenced side, a key reference may not store its key id with a transient identity, and an inList reference may not find its list's document through one. The referring side of a where entry may be transient: it is checked on the transition;
  • encryptedFor names one as its recipient or key id;
  • generatedFrom sits on one or names one as a param;
  • immutable lists one. A transient property is always absent from the stored document, so every replace that carries it would count as changing it;
  • a propertyConstraints rule reads one;
  • the type is indexOnly and declares any transient property.

Before protocol version 14 none of these was checked.

On a contract update the list is fixed. It decides which values stored documents hold and how every property is encoded, so documents written under one list could not be read under another.

See also

Mutability

These two keywords decide what a replace may change once a document exists. A replace is the transition an owner sends to overwrite a document with a new version of it. documentsMutable turns replaces on or off for the whole document type. immutable freezes chosen properties while the rest of the document stays editable: from the moment the document is created, or only while a condition holds, such as five minutes after creation, once the document is published, or once a value has been filled in.

Neither governs deletion, transfers or trading: see Deletion and Creation, Transfers and Trading.

documentsMutable

Whether the owner of a document may replace it. Set it to false for records that must never change after they are written: votes, receipts, name registrations.

Wheredocument type
Valueboolean
Defaultthe contract config's documentsMutableContractDefault, which is true unless the contract says otherwise
Sinceprotocol version 1
On updateFixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsInvalidDocumentTransitionActionError (10404) for a replace of a type set to false

Example

"vote": {
  "type": "object",
  "documentsMutable": false,
  "canBeDeleted": false,
  "properties": {
    "proposalId": {
      "type": "array",
      "byteArray": true,
      "minItems": 32,
      "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "position": 0
    },
    "choice": { "type": "string", "enum": ["yes", "no", "abstain"], "position": 1 }
  },
  "required": ["proposalId", "choice", "$createdAt"],
  "additionalProperties": false
}

A vote is cast once and stays as cast: no replace is accepted, and with canBeDeleted: false its owner cannot take it back either.

How it works

  • With true, the document's owner may replace it. The replace carries the whole new document, which is validated against the schema as a create is, and a $revision one higher than the stored one (InvalidDocumentRevisionError, 40106). Anyone other than the owner is refused (DocumentOwnerIdMismatchError, 40102). When the type lists $updatedAt in required, the replace sets it to the block's time, and likewise $updatedAtBlockHeight and $updatedAtCoreBlockHeight to the block heights.
  • With false, every replace is refused (InvalidDocumentTransitionActionError, 10404).
  • A type whose documents cannot be replaced may still let them be transferred or sold (transferable, tradeMode). Such a document changes owner and price, but its owner can never edit its properties. The DPNS domain type works this way.
  • A document stores a $revision when its type allows a replace, a transfer or trading, or keeps fields its moderators write (moderatorAbilities.changeFields), whose changes raise it even on a type whose documents cannot be replaced. A type that allows none of them stores none. See System Properties.

Rules at registration

  • A contested index needs a type whose documents cannot be replaced (ContestedUniqueIndexOnMutableDocumentTypeError, 10248). See Contested Indexes.
  • An indexOnly type must set documentsMutable: false. See Index-Only Types.
  • immutable is only accepted when the type's documents are mutable (InvalidContractStructure, 10231).
  • moderatorAbilities.deleteWithin on a mutable type needs $updatedAt in required. See Deletion.

immutable

The top-level properties a replace may not change, on a type whose documents can otherwise be replaced. Each entry is either a property name, frozen when the document is created, or a property with a condition, frozen for any replace the condition holds for. Reach for the first when part of a document is a commitment: the shop an order was placed with, the author of a post. Reach for the second when a value may change for a while and then must stand: a post's text for five minutes after it is published, an article's body once it is out of draft, a tracking code once it is filled in.

Wheredocument type, on a type with documentsMutable: true
Valuearray whose entries are a top-level property name, or { "property": <name>, "when": <condition> }; each property listed once
Defaultempty: every property may change
Sinceprotocol version 14
On updateMay gain entries, with a condition or without. A property listed with a condition keeps it, or loses it to be listed without one. A property listed without a condition stays, and no property is dropped (DocumentTypeUpdateError, 40212)
ErrorsDocumentImmutablePropertyChangedError (40128) for a replace that changes, adds or removes a frozen property

Example

"post": {
  "type": "object",
  "documentsMutable": true,
  "properties": {
    "author": { "type": "string", "maxLength": 63, "position": 0 },
    "text": { "type": "string", "maxLength": 500, "position": 1 },
    "status": { "type": "string", "enum": ["draft", "published"], "position": 2 },
    "body": { "type": "string", "maxLength": 5000, "position": 3 },
    "trackingCode": { "type": "string", "maxLength": 40, "position": 4 },
    "pinned": { "type": "boolean", "position": 5 }
  },
  "required": ["author", "text", "status", "$createdAt", "$updatedAt"],
  "immutable": [
    "author",
    {
      "property": "text",
      "when": { "greaterThan": [{ "subtract": ["$updatedAt", "$createdAt"] }, 300000] }
    },
    {
      "property": "body",
      "when": { "equal": ["$old.status", { "const": "published" }] }
    },
    { "property": "trackingCode", "when": { "present": "$old.trackingCode" } }
  ],
  "additionalProperties": false
}

author never changes. text can be corrected for five minutes after the post is created: during a replace $updatedAt is the replace's block time, so the difference is the post's age in milliseconds. body stays editable while the stored post is a draft, including in the replace that publishes it, and is frozen after that. trackingCode can be filled in by one replace while the stored post has none, and is frozen once it holds one. pinned is not listed, so it can change at any time.

How it works

  • On every replace, each property that differs from the stored document is checked against the list. A property listed by name is refused with DocumentImmutablePropertyChangedError (40128). A property listed with a condition is refused with the same error when its condition holds for this replace. "Differs" covers a changed value, a value the stored document did not have, and a value the replace leaves out.
  • A condition takes the grammar of a propertyConstraints rule: comparisons, arithmetic, in, present, absent, anyOf, allOf, not, ifThen, ifThenElse, the system times and heights the type records, and $ownerId. It is judged, as a rule judges a replace, on the document the replace writes: its properties as the replace sets them, the stored $createdAt and $transferredAt, and the replace's block as $updatedAt.
  • A path starting with $old. reads the stored document instead: $old.status is the status before the replace. Only a condition of immutable, or a type's retractedWhen, may read it, and only a schema property through it.
  • A property is frozen while its condition holds. If what the condition reads can change back, the property can become editable again: with body frozen by $old.status, a replace setting status back to draft still reads the stored published, so body stays frozen in that replace, and the next replace, reading the stored draft, may change it. For a freeze that lasts, list what the condition reads as well, or use a condition that cannot turn back, as a document's age or a filled-in value cannot.
  • A condition is evaluated only when its property changes. A condition that faults, dividing by zero or overflowing, counts as holding: a fault never frees a property.
  • Values are compared by their data, not their bytes: the order of an object's members and the width an integer is stored in do not count as changes. Freezing an object freezes everything inside it.
  • One change is always allowed: a replace may clear a deletableDocument reference by id, listed without a condition, once the document it points to has been deleted. Every replace checks such a reference again, so without this the document could never be replaced again. See References.
  • A replace judged by checkTx shortly before a time condition starts holding may still land in a block after that, and is then refused there and pays its fee, like any other state check.
  • Transfers, price updates and purchases carry no property values, so the list does not affect them.

Rules at registration

All refusals below are InvalidContractStructure (10231). The shape of the list, and what each condition reads, are checked on every parse. The other rules are checked when a contract is registered or updated.

  • Only on a type whose documents are mutable. On a type with documentsMutable: false every property is already frozen.
  • Every entry is a string or an object with exactly property and when, and no property is listed twice.
  • Every listed property is a declared top-level property. System properties ($ownerId, $createdAt and the rest) are refused, since the platform manages them. Nested paths such as meta.author are refused: list the object that contains them.
  • No listed property may be transient: it is never stored, so every replace that supplies it while it is frozen would count as a change.
  • A condition reads what a propertyConstraints rule may read of the type: declared properties of the right kind, neither transient nor inside a transient object, and system times and heights the type lists in required. It stays within the node limit of a rule, and lists no condition twice.
  • A condition may not read a countOf or sumOf total: it reads the document alone, so a replace judges it without reading state.
  • immutableAllowSetting, which earlier builds used to let a frozen property be filled in once, is refused and names the replacement: { "property": "p", "when": { "present": "$old.p" } }.
  • An immutable property may not hold a deletableDocument reference that a replace could not clear: a typed array of them, one inside an object, or one found by findBy whose key no function computes. A single reference by id held directly by the property is allowed, but only without a condition: once cleared, a replace the condition leaves free could set it to another document.
  • On a type whose documents can be transferred or traded, an immutable property may not hold a contract reference whose contractRequirements has an owner requirement: after a change of owner the new owner could neither meet it nor repoint it.
  • A listed property, with a condition or without, may not be one of the fields only moderators write (moderatorAbilities.changeFields).
  • A property listed with a condition is not fixed once written, so a findBy key, an inList list, a value read beside a findBy function, or a value another type indexes through a reference may not rely on it. See References.

On update

What immutable freezes may only tighten. A property may be added, with a condition or without, and documents already stored are held to it from then on. A property listed with a condition may lose it, which freezes it whatever the condition said. A condition cannot change: whether one condition holds wherever another does cannot be told in general. A property listed without a condition stays, and no property is dropped, since documents were written on the promise that those properties would not change. The comparison is of the parsed entries, so reordering them is no change.

See also

Deletion

A document can leave the state three ways: its owner deletes it, the contract's moderators delete it, or the platform deletes it when its time to live runs out. canBeDeleted rules the first, and deleteConstraints can narrow it to the documents that meet its rules; moderatorAbilities.delete, moderatorAbilities.deleteWithin and moderatorAbilities.deleteSettled the second, and ttl the third (see Time To Live). Each is independent of the others: a type may let moderators remove what its authors cannot retract, or expire documents that nobody may delete by hand.

canBeDeleted

Whether a document's owner may delete it. Set it to false for records that other documents or other people rely on staying put, and to "onlyWhenConsumed" for records only a create that consumes them may remove.

Wheredocument type
Valueboolean, or "onlyWhenConsumed"
Defaultthe contract config's documentsCanBeDeletedContractDefault, which is true unless the contract says otherwise
Sinceprotocol version 1; "onlyWhenConsumed" protocol version 14
On updateFixed (DocumentTypeUpdateError, 40212), except that a type which keeps history before and after the update may change it from true to false. Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsInvalidDocumentTransitionActionError (10404) for a delete of a type set to false or "onlyWhenConsumed", or, from protocol version 14, of a type that keeps history; DocumentOwnerIdMismatchError (40102) for a delete by anyone but the owner

Example

"comment": {
  "type": "object",
  "canBeDeleted": true,
  "properties": {
    "postId": {
      "type": "array",
      "byteArray": true,
      "minItems": 32,
      "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "position": 0
    },
    "text": { "type": "string", "maxLength": 500, "position": 1 }
  },
  "required": ["postId", "text"],
  "additionalProperties": false
}

A commenter may take a comment down at any time. Since true is the usual default, the key could be left out; writing it makes the intent plain.

How it works

  • The owner deletes a document with a delete transition that names its id. Anyone else is refused (DocumentOwnerIdMismatchError, 40102), and a document that does not exist is DocumentNotFoundError (40101).
  • The owner is refunded the part of the document's storage fee that has not yet been paid out to past epochs. A document of a type with a ttl refunds nothing. See Refunds.
  • A delete may carry a token cost or an action fee, like any document action. See Token Costs and Action Fees.
  • An identity that is banned or suspended on a moderated contract may still delete its own documents. See Contract Moderation. On a type set to false it can retract them instead, when the type declares retractedWhen.
  • false binds only the owner. The contract's moderators, when the type allows them, and the platform, when the type has a ttl, still delete such documents.
  • "onlyWhenConsumed" binds the owner as false does, and lets a create consume the document (see below).
  • Drive never deletes a document whose type keeps history (documentsKeepHistory). From protocol version 14 a delete of such a document is refused with 10404 whatever canBeDeleted says; before it, the delete failed inside Drive as an internal error.
  • Documents of an indexOnly type are deleted with an index-only delete transition that carries their values, since there is no stored row to name by id. A delete by id of such a document is refused (10404). See Index-Only Types.

Rules at registration

  • From protocol version 14, a type with documentsKeepHistory: true must set canBeDeleted: false (InvalidContractStructure, 10231). The default is true, so it has to be written out. A contract registered earlier with both flags on stays readable, but its next update is checked like a new contract, so that update must turn canBeDeleted off on the type. That is the one change to canBeDeleted an update may make.
  • For references, a type whose owner may delete its documents is deletable: a permanentDocument reference, inList included, may not point at it (ReferencedDocumentTypeDeletableError, 40122), and a deletableDocument reference may. See References.
  • "onlyWhenConsumed" is refused on a type that keeps history or is indexOnly (InvalidContractStructure, 10231): the storage layer never deletes a document that keeps history, and an indexOnly type has no stored row a reference finds, so nothing could consume either.

Deleted only when consumed

"onlyWhenConsumed" says the owner can not delete a document, as false does, but a create of the same contract whose refersTo declares consume can. Its owner never removes it: it leaves state when a create consumes it, or, as with false, when the contract's moderators delete it where the type allows them (moderatorAbilities.delete) or the platform deletes it when its ttl passes.

"preorder": {
  "type": "object",
  "documentsMutable": false,
  "canBeDeleted": "onlyWhenConsumed",
  "indices": [
    { "name": "saltedHash", "properties": [{ "saltedDomainHash": "asc" }], "unique": true }
  ],
  "properties": {
    "saltedDomainHash": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 0
    }
  },
  "required": ["$createdAtBlockHeight", "saltedDomainHash"],
  "additionalProperties": false
}

A preorder stays until the name registration that reveals it consumes it (the preorderSalt declaration in Commit and reveal). Its owner can not take it back with a delete.

  • A delete transition of such a document is refused (InvalidDocumentTransitionActionError, 10404), as for false.
  • A consume deletes it as a delete by its owner would, its storage refunded to the owner.
  • For references the type is deletable, as with true: its documents can leave state without a record. A permanentDocument reference to it is refused (40122), and so is a moderatedDocument one (40143), even when the type also lets its moderators delete with records; a deletableDocument reference is the one that points at it.
  • Fixed on update: an update may neither set nor remove it (DocumentTypeUpdateError, 40212). Setting it would let documents leave state under the permanentDocument references made to a type that promised they never would; removing it would leave the references that consume its documents nothing to delete.
  • Before protocol version 14 the meta-schemas accept only a boolean, so a contract carrying the string is refused (JsonSchemaError, 10101).

retractedWhen

The replace that retracts a document: what a banned or suspended author may still do on a type whose documents it can not delete. A barred identity can write nothing new, but deleting what it wrote is never refused. On a type set to canBeDeleted: false there is no delete, and the author's only way to take a document back is a replace that blanks it, which a bar refuses like any other write. retractedWhen names that replace, and lets it through.

Wheredocument type
Valueone condition, in the grammar of an immutable entry's when
Defaultnone: a barred author's replaces are all refused
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212): an update may not add it, remove it or change it
ErrorsContractUserBannedError (41107) or ContractUserSuspendedError (41108) for a barred author's replace whose written document does not meet the condition

Example

"post": {
  "type": "object",
  "canBeDeleted": false,
  "properties": {
    "text": { "type": "string", "maxLength": 500, "position": 0 },
    "deleted": { "type": "boolean", "position": 1 }
  },
  "additionalProperties": false,
  "retractedWhen": { "present": "deleted" },
  "propertyConstraints": {
    "retractedIsBlank": { "anyOf": [{ "absent": "deleted" }, { "absent": "text" }] }
  },
  "immutable": [
    { "property": "deleted", "when": { "present": "$old.deleted" } }
  ]
}

Replies point at posts, so a post stays in place. Its author retracts one by replacing it with { "deleted": true }. The three keywords split the work. retractedWhen says that a post carrying deleted is retracted, so a banned author may still write one. retractedIsBlank says that a retracted post carries no text, for every author. The immutable entry says that a post stays retracted once it is. A banned author's edit of its text is refused with 41107, and so is a replace that drops deleted again.

How it works

  • Only replaces are let through, and only those of a banned or suspended owner are judged. An identity that is not barred replaces its documents under the type's other rules alone.
  • The moderation gate lets every replace of a barred owner on the type through. Once the stored document is fetched, the transformer judges the condition on the document the replace writes, as an immutable condition is judged: its properties as the replace sets them, the replace's block as $updatedAt, and the stored document under $old.. A replace for which the condition does not hold is refused with the bar's error and its nonce bumped, in a block and in the mempool alike. A condition that faults, dividing by zero or overflowing, counts as not holding: a fault never lifts a bar.
  • The condition only picks the replaces a barred owner may make. Every other rule of the type still judges them: the schema, propertyConstraints, immutable, references, token costs and action fees. What a retracted document may hold is for those rules to say. Without a rule like retractedIsBlank above, a barred author could write anything into a document that meets the condition.
  • Creates, transfers, purchases and price updates by a barred owner are refused as before, and deletions pass as before.

Rules at registration

All refusals below are InvalidContractStructure (10231), checked on every parse.

  • Only on a type whose documents are mutable: without a replace there is nothing to retract with.
  • Only on a contract that keeps a banlist or a suspension list: otherwise no owner is ever barred.
  • The condition reads what an immutable entry's when may read: declared properties of the right kind, neither transient nor inside a transient object, the system times and heights the type lists in required, and the stored document through $old.. It reads no countOf or sumOf. When a contract is registered or updated, it also stays within the node limit of a rule and lists no condition twice.

deleteConstraints

Rules the stored document must meet for its owner to delete it. canBeDeleted: true lets an owner delete any of its documents at any time; deleteConstraints holds the delete to conditions, written in the grammar of propertyConstraints: a poll may be deleted only before its first vote, an order only while it is still open, an address only when its owner keeps another.

Wheredocument type
ValueAn object of rules, at least one. Each key is the rule's name (1 to 64 letters, digits or underscores); each value is a condition in the grammar of a propertyConstraints rule
DefaultAbsent: the owner deletes whenever canBeDeleted allows
Sinceprotocol version 14
On updateFixed: adding, removing or changing a rule is refused (IncompatibleDocumentTypeSchemaError, 10246)
ErrorsDocumentDeleteConstraintViolatedError (40147) on a delete; at registration JsonSchemaError (10101) or InvalidContractStructure (10231)

Example

"poll": {
  "type": "object",
  "canBeDeleted": true,
  "properties": {
    "question": { "type": "string", "maxLength": 280, "position": 0 },
    "status": { "type": "string", "enum": ["open", "locked"], "maxLength": 10, "position": 1 }
  },
  "required": ["$createdAt", "question"],
  "deleteConstraints": {
    "noVotes": { "equal": [{ "countOf": ["vote", { "pollId": "$id" }] }, 0] },
    "notLocked": { "notEqual": ["status", { "const": "locked" }] }
  },
  "additionalProperties": false
},
"vote": {
  "type": "object",
  "properties": {
    "pollId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "refersTo": { "type": "deletableDocument", "documentType": "poll" },
      "position": 0
    },
    "choice": { "type": "integer", "minimum": 0, "maximum": 9, "position": 1 }
  },
  "required": ["pollId", "choice"],
  "indices": [
    { "name": "byPoll", "properties": [{ "pollId": "asc" }], "countable": "countable" }
  ],
  "additionalProperties": false
}

noVotes counts the votes whose pollId is the poll's own id ($id), from the count the byPoll index keeps, and lets the poll go only while there are none. notLocked reads the stored poll: a locked one stays. Once a vote is cast, its author can no longer pull the poll out from under it; once every vote is gone, the poll can be deleted again.

Without the keyword, the poll had to be undeletable for its votes to be sure it stays. A propertyConstraints rule of the same shape on the poll, { "equal": [{ "countOf": ["vote", { "pollId": "$id" }] }, 0] }, refuses an edit after the first vote in the same way.

How it works

  • The owner's delete. When the owner deletes a document, consensus fetches it, checks the owner, then checks every rule in name order against the stored document, and refuses the delete at the first rule it breaks with DocumentDeleteConstraintViolatedError (40147). The error names the document, the rule, and why it failed, with the reasons of propertyConstraints. It is a state error, so the delete is paid for and the nonce is spent, and the document stays.
  • What a rule reads. The stored document's properties, its $ownerId (the owner deleting it), and the times and heights the type lists in required, as stored. A countOf or sumOf reads a total from state as it will be once the document is gone: a total over the type's own documents no longer counts it, so { "greaterThanOrEqual": [{ "countOf": ["address", { "$ownerId": "$ownerId" }] }, 1] } keeps the last address of each owner. Each total is a billed read.
  • $id in a filter. A filter may match a key by the document's own id, "$id", which is what makes "while no vote points at it" a rule. The key must be an identifier property of the counted type. $id is a filter value in propertyConstraints too, as the example shows; it is not an operand anywhere else.
  • Only the owner's delete. The contract's moderators delete as their abilities allow, and a ttl expires documents on time, whatever the rules say. A refersTo with consume may not target the type, since a consume deletes without a delete transition to judge. A banned or suspended owner, who may still delete its documents, is held to the rules like any other.
  • The totals are those of the block. Each state transition of a block is applied before the next is validated, so a vote earlier in the block counts, and a vote later in the block finds the poll gone (ReferencedEntityNotFoundError, 40120).
  • A total of another type is judged at the delete alone. The rules do not stop the counted documents from changing later, which is what the example wants: deleting the last vote frees the poll.

Rules at registration

All refusals below are InvalidContractStructure (10231) unless noted.

  • Only on a type whose owner deletes its stored documents: not canBeDeleted: false or "onlyWhenConsumed", where there is no delete to gate, and not indexOnly, whose delete carries the row's values and is judged by its propertyConstraints. Checked on every parse.
  • The grammar, the shape and the reads are those of a propertyConstraints rule (see Rules at registration): every path names a stored property of the kind it is read as, a time or height the type lists in required, no $old. path. Every countOf and sumOf counts a type of the contract with a tree that keeps the total. Unlike a propertyConstraints rule, a delete rule may total its own type when that type has a contested index: the total it reads at the delete is the stored one, which an awarded document is in like any other.
  • The same limits, counted apart from propertyConstraints: at most 16 rules, each of at most 32 nodes, reading at most 4 distinct totals.
  • A refersTo with consume may not target a type that declares the keyword.

moderatorAbilities.delete

Lets the contract's moderators delete documents of the type, whoever owns them. It is how an application takes down content that breaks its rules, where a ban only stops an identity from writing more. It is one key of the moderatorAbilities object, which also names the fields only moderators write.

WheremoderatorAbilities of a document type, in a contract whose config declares moderation
Valueboolean
Defaultfalse
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212)
ErrorsDocumentTypeNotDeletableByModeratorsError (41115), IdentityNotContractModeratorError (41101), ContractModerationTargetNotAllowedError (41102), DocumentNotFoundError (40101), and DocumentModerationWindowElapsedError (41116) with a window

Example

"post": {
  "type": "object",
  "documentsMutable": true,
  "canBeDeleted": false,
  "moderatorAbilities": { "delete": true, "deleteWithin": 604800 },
  "properties": {
    "text": { "type": "string", "maxLength": 280, "position": 0 }
  },
  "required": ["$createdAt", "$updatedAt", "text"],
  "additionalProperties": false
}

Authors cannot retract a post, but the contract's moderators can remove one for a week (604,800 seconds) after it was written or last edited. The contract around it must declare moderation in its config.

How it works

  • A moderator deletes a document with the contract user moderation transition, naming the document type, the document id and a reason. The moderators are the ones the contract's moderation config declares; see Contract Moderation.
  • The transition is checked in this order, each refusal paid: the document type exists (InvalidDocumentTypeError, 10406); it sets delete (41115); the signer is the contract owner or a moderator (41101); the document exists (40101); its owner is neither the contract owner nor a moderator (41102); and, when the type sets deleteWithin, the window has not passed (41116).
  • The document and all its index entries are deleted as an owner's delete would delete them, without the canBeDeleted check. A removal record is written under the contract: whose document it was, which moderator removed it, the reason, the block time and a hash of the document, and the values of any fields the type keeps public (see deleteKeepsFields). The record is never deleted. A type may leave no record: see deleteKeepsRecord.
  • The document's owner gets no storage refund unless the type says otherwise (see deleteRefundsOwner), and the moderator pays neither the type's delete token cost nor its delete action fee.
  • For a week after the deletion a moderator may restore the document exactly as it was. See Restoring Documents.

Rules at registration

All refusals below are InvalidContractStructure (10231).

  • The contract's config must declare moderation. Moderation cannot be added by a later update, so without it nobody could ever delete anything. In return the moderation block may keep no banlist, suspension list or warning list at all when a document type gives its moderators an ability.
  • Refused on a type that keeps history (Drive never deletes those documents), on an indexOnly type (there is no stored row to name), on a type with creationRestrictionMode 1 or 2 (its documents are the contract owner's or the platform's), and on a type with a contested index (a restore could not go through the vote the index requires).
  • For references, the type is no longer permanent even with canBeDeleted: false: a permanentDocument reference, inList included, may not point at it (40122). With canBeDeleted: false, no ttl and removal records kept (the default), its documents leave state only on a moderator's record, and a moderatedDocument reference is the one that points at it, resolving to the document or to its removal record; a deletableDocument reference is refused (40144). Otherwise a deletableDocument reference points at it.

moderatorAbilities.deleteWithin

Limits the moderators' deletion to a window after a document's last change. Once the window has passed the document is settled: moderation acts on what was just written and does not reach back into what has stood unchallenged.

WheremoderatorAbilities of a document type, with delete: true
Valueinteger, seconds, 1 to 4294967295
Defaultabsent: no limit
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212), in both directions: a longer window would reopen documents that had settled
ErrorsDocumentModerationWindowElapsedError (41116)

How it works

  • The window is measured from the document's $updatedAt, or from its $createdAt on a type that does not record $updatedAt. A deletion at exactly that time plus the window still passes; after it, every moderator is refused, the contract owner included. On a type that sets deleteSettled, the seated team may still delete it together.
  • A replace or a price update moves $updatedAt, so new content opens the window again. A transfer, a purchase or a moderator's field change does not move it.
  • A restored document comes back with its old $updatedAt, so it may already be settled.
  • The window says nothing about the document's own owner, whose deletion canBeDeleted rules at any age.

Rules at registration

  • Needs delete: true (InvalidContractStructure, 10231).
  • A type whose documents can be replaced must list $updatedAt in required: measured from creation alone, an author could wait the window out and then rewrite a post into something no moderator can remove. A type with documentsMutable: false must list $updatedAt or $createdAt. Both refusals are 10231.
  • A window of 0 is refused by the meta-schema (JsonSchemaError, 10101). A type that moderators may never delete from simply leaves delete out.

moderatorAbilities.deleteSettled

Who must agree to delete a settled document: one past its deleteWithin window, which no moderator deletes alone. It lets a contract keep settled content safe from any single moderator while leaving its elected moderation team a way to remove it when the team agrees: so many members together, the leader among them only when the rule says so (leader: true), which may be the leader alone.

WheremoderatorAbilities of a document type, with delete: true and deleteWithin, in a contract whose moderators are an elected team
Valueobject with leader (boolean, default false: whether the team's leader must be among the approvals), approvals (integer, default 1: how many members of the seated team must approve, the leader counted, at least 1 and at most the members the declared team can hold) and approversPredateDocument (boolean, default true when approvals is above 1 and false otherwise: whether a member the leader added counts only for documents created after its addition), at least one of them given
Defaultabsent: nobody deletes a settled document
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212), in both directions: fewer approvals would reach content written under more
ErrorsDocumentTypeNotDeletableOnceSettledError (41204), ContractModerationTeamNotSeatedError (41205), DocumentNotSettledError (41206), ContractTeamActionDoesNotExistError (41207), ContractTeamActionAlreadySignedError (41208), SettledDeletionNotRestorableError (41209), ContractTeamActionAlreadyCompletedError (41210), ContractTeamActionDocumentChangedError (41211), ContractTeamMemberAddedAfterDocumentError (41212), and those of a moderator's deletion (41101, 41102, 41201, 41203)

Example

"post": {
  "type": "object",
  "documentsMutable": true,
  "moderatorAbilities": {
    "delete": true,
    "deleteWithin": 86400,
    "deleteSettled": { "leader": true, "approvals": 3 }
  },
  "properties": {
    "text": { "type": "string", "maxLength": 280, "position": 0 }
  },
  "required": ["$createdAt", "$updatedAt", "text"],
  "additionalProperties": false
}

For a day after a post is written or edited, any moderator deletes it. After that, it is deleted only when the team's leader and two other members approve: one of them proposes the deletion, and the others approve the proposal. A member the leader added counts only for posts written after the leader added it. { "leader": true } would let the leader alone delete a settled post; { "approvals": 2 } any two members; { "leader": true, "approvals": 3, "approversPredateDocument": false } the leader and two others, whenever added.

How it works

The team deletes a settled document the way a token group carries out an action: one member proposes it, the others approve it by its id, and it runs once the approvals meet the rule.

  • Proposing. A member of the seated team sends the contract user moderation transition's deleteSettledDocument action, naming the document type, the document id and a reason. The proposal is its own approval. It is kept under the contract as a team action, naming the document as it is (its last modification and revision) and the reason, by an id the proposer's client computes from the contract, the proposer, its nonce, the document and the reason. A rule the proposer meets alone ({ "leader": true } proposed by the leader) deletes the document at once.
  • Approving. The other members send the approveTeamAction action with that id. What is deleted and why is the proposal's: an approval carries nothing else. The team's actions and who approved each are readable with getContractTeamActions and getContractTeamActionSigners, which is how a member finds what the others proposed.
  • Running. The approval that meets the rule deletes the document as a moderator's deleteDocument would: its removal record (unless the type sets deleteKeepsRecord: false, with the proposal's reason and the member whose approval deleted it), and the owner's refund as deleteRefundsOwner says. The action then moves, with the approvals that counted, from the contract's active team actions to its closed ones, for good.
  • Nothing lapses. A proposal stays active until it runs. Once the document changes (the author edits it, it changes hands, or a moderator changes its fields: anything that moves its $revision), an approval is refused (41211): the team would be approving the deletion of content it never saw. A member proposes afresh instead, a new action. Two proposals may name one document; once one runs, the other takes no approval (the document is gone, 40101) and stays active, as a token group's action that never gathers its power does. An edit by the author opens the deleteWithin window again anyway, in which any moderator deletes the document alone; a moderator's change of its fields does not.
  • Members the leader added. The leader names whom it adds (addedModerator), so without a limit it could add members who approve whatever it proposes, and take them off again once they had. Unless the rule sets approversPredateDocument: false, a member the leader added counts only for documents created after its addition (the default whenever more than one approval is needed; a rule one approval meets, the leader meets alone, so it dates nobody by default): its addedModerator's $createdAt is earlier than the document's $createdAt. One added in the same block as the document does not count. Its proposal or approval of an older document is refused (41212). The leader and the elected members always count, for documents older than the team too: the election seated them, not the leader. What the rule needs is not lowered for it: a team with too few members from before a document never deletes that document once settled. Every addition comes after the seat, so a document written before the seat counts only the leader and the elected members: under a rule asking for more approvals than those, no document older than the seat is ever deleted once settled, and the rule can not change. Pick approvals no higher than the leader plus the members a charter is expected to elect, or set approversPredateDocument: false.
  • Members who left. A member who left the team since approving no longer counts. When the approvals given could meet the rule, an approval reads the team and drops the approvals of members who are gone, refunded to them: one who comes back after that approves again, while one back before any such read still has its approval counted. A member the leader takes off and adds again sits in the new addition: under approversPredateDocument its approval of a document created before the new addition is dropped the same way, and it approves that document no more. A dropped approval no longer proves: the proof of that member's proposal or approval fails from then on.
  • The checks of a proposal, in order, each refusal paid: the document type exists (10406) and sets deleteSettled (41204); a team is seated (41205): the interim moderators and the contract owner never delete a settled document; the signer is on the team (41101) and the declaration gives the team deleteDocuments on the type (41201); the reason names a reason document the team's proposal lists (41203); the document exists (40101) and its owner is not protected (41102); the document is settled (41206: within the window, use deleteDocument); a signer the leader added was added before the document was created (41212).
  • The checks of an approval, in order, each refusal paid: the contract keeps team actions and the team proposed this one (41207); the action has not run (41210); a team is seated (41205); the signer is on the team with deleteDocuments on the type (41101, 41201); the document still exists (40101), its owner is not protected (41102), and it has not changed since the proposal (41211); a signer the leader added was added before the document was created (41212); the signer has not approved it already (41208): a member taken off and added again too late is told so, not that its earlier approval, which no longer counts, stands.
  • A deletion the team approved stands: no moderator restores it, the leader included (SettledDeletionNotRestorableError, 41209). A single moderator undoing what the leader and the members agreed on would defeat the rule; a deletion within the window is restored as before.
  • A deletion counts toward the moderators pot's action share for every approver whose approval counted, once it happens. Approvals that fall short count for nobody.
  • Each member pays for the transition and for its approval (the proposer for the action too), and is refunded its approval (the proposer the action's info too, but not the two trees that hold the action, which carry no storage flags and refund nobody) when the action runs and moves to the closed actions, or when a later approval drops it. The member whose approval runs the action pays for its closed copy, which nothing deletes.

Rules at registration

All refusals below are InvalidContractStructure (10231).

  • Needs deleteWithin: without a window nothing is ever settled.
  • Needs a contract whose moderation declares an elected team. The elected declaration must give the team deleteDocuments on the type (InvalidContractModerationConfigError, 10900), or no team could ever use the rule.
  • approvals is at least 1 and at most the members the declared team can hold: its leader, the 15 members a charter elects and the maxAddedModerators of the elected declaration (31 at most). A rule no team could meet would leave settled documents undeletable for good, since neither the rule nor the declaration can change. A seated team whose charter elects fewer than 15 members holds fewer: a rule asking for more than it can hold asks for all of them. A member the leader removed still counts toward what the team can hold: the leader can not lower the bar by removing members who would not approve, and gets the seat back by deleting the removal.
  • While approversPredateDocument is on (the default when approvals is above 1), the type must list $createdAt in required: who of the team predates a document is read from it. A type that does not record $createdAt sets approversPredateDocument: false. Checked at registration only, as the bound on approvals is: a stored type is read back as it is, and a document without $createdAt admits no added member.

moderatorAbilities.deleteKeepsRecord

Whether a moderator's deletion leaves a removal record under the contract. The record is what explains a missing document (who removed it, whose it was, why, when) and what a restore brings it back from. A contract that wants its moderators' deletions final and unrecorded, or does not want to pay for the records, turns it off.

WheremoderatorAbilities of a document type, with delete: true
Valueboolean
Defaulttrue
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212)
ErrorsContractDocumentRemovalNotFoundError (41119) for a restore when false

How it works

  • With false, the deletion writes no record, and the type gets no removal records tree: getContractDocumentRemovals refuses it as a type that keeps none. The document is gone for good: a restore is refused (41119), and a document id is never produced twice, so it cannot come back another way.
  • The proof of such a deletion is the document's absence, which the SDKs report as no record (delete_contract_document resolves with None, contractDeleteDocument with undefined). The verifier reads the type's setting from the contract, so for such a type it needs the contract, as a restore's does; a deletion that leaves a record is proved by the record alone.
  • The moderator pays less: no record is written and no hash computed.

Rules at registration

  • Needs delete: true (InvalidContractStructure, 10231).

moderatorAbilities.deleteKeepsFields

Which fields of a deleted document stay public in its removal record. The document is gone, but some of what it said may still matter to everyone else: the hashtag of a removed post keeps the hashtag's timeline honest ("a post here was removed"), the thread a removed reply belonged to, the time it was written. The record keeps a copy of those values; everything else leaves with the document.

WheremoderatorAbilities of a document type, with delete: true and a record (deleteKeepsRecord not false)
Valuearray of property paths, at least one, none twice
Defaultabsent: the record keeps no field of the document
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212), in both directions: which fields stay public is what an author was told when writing

Example

{
  "type": "object",
  "properties": {
    "text": { "type": "string", "maxLength": 500, "position": 0 },
    "hashtag": { "type": "string", "maxLength": 61, "position": 1 },
    "meta": {
      "type": "object",
      "position": 2,
      "properties": {
        "tags": { "type": "array", "items": { "type": "string", "maxLength": 20 }, "maxItems": 5, "position": 0 },
        "note": { "type": "string", "maxLength": 100, "position": 1 }
      },
      "additionalProperties": false
    }
  },
  "required": ["$createdAt", "text"],
  "canBeDeleted": false,
  "moderatorAbilities": {
    "delete": true,
    "deleteKeepsFields": ["hashtag", "meta.tags", "$createdAt"]
  },
  "additionalProperties": false
}

A moderator deleting a post of this type leaves a record that still says which hashtag and tags it carried and when it was written, while its text and its note are gone:

{
  "documentId": "…",
  "documentOwnerId": "…",
  "moderatorId": "…",
  "reason": { "text": "spam" },
  "removedAt": 1759200000000,
  "documentHash": "…",
  "keptFields": {
    "$createdAt": 1759100000000,
    "hashtag": "dash",
    "meta.tags": ["privacy", "payments"]
  }
}

How it works

  • The values are copied from the document as stored at the deletion, each under the path the type lists: a top-level property, a property inside an object (meta.tags), or a whole object (meta). A path the document holds no value at is left out of the record.
  • The record stores them as the document stores its properties, so they are read, as the document is, under its document type, and come back typed exactly as the document's values: the SDKs do this for you (keptFields). An object kept whole shows members an update added after the removal as absent, as an older document does.
  • They are read wherever the record is: getContractDocumentRemovals, by document id or by page, the proof of the deletion, and a join through a moderatedDocument reference. The records are not indexed by them: no query finds a record by a kept value.
  • A moderatedDocument reference to a removed document checks a where pair on a kept property against the kept value, as it would against the document.
  • An index of a referring type may hold a kept value, read through a moderatedDocument reference: a derived index property, such as a reply's postId.hashtag. The value must be one an index can key and fixed once written: the example above is not, since its documents are mutable and hashtag is not listed under immutable. Once the document is removed, Drive reads the value from the record.
  • A restore brings the document back and leaves the record, marked restored, with what it kept. A later deletion of the restored document writes a fresh record, with the values the document then held.
  • The moderator pays for the record, kept values included.

Rules at registration

All refusals below are InvalidContractStructure (10231).

  • Needs delete: true, and a record: refused beside deleteKeepsRecord: false.
  • Each entry is a declared property at any depth, stepping through objects by ., or one of the timestamps and block heights ($createdAt, $updatedAt, $transferredAt, and their BlockHeight and CoreBlockHeight forms) listed in required, without which no document carries it.
  • Refused: a transient property (no stored document holds it), $id and $ownerId (every record holds them already), any other system property, and a path inside another listed path (the object around it is kept whole already).

moderatorAbilities.deleteRefundsOwner

Whether the owner of a document a moderator deletes is refunded its storage. By default the owner forfeits it: removed content costs its author what they paid to store it. A contract whose moderation is housekeeping rather than sanction (clearing handled reports, expired listings) gives it back.

WheremoderatorAbilities of a document type, with delete: true
Valueboolean
Defaultfalse
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212)

How it works

  • With true, the owner is refunded as for their own deletion: the part of the storage fee not yet paid out to past epochs. The refund goes to the owner, not to the moderator, who still pays for the transition and the record. A document of a type with a ttl refunds nothing either way.
  • With false, the credits stay in the storage pools they were paid into.

Rules at registration

  • Needs delete: true (InvalidContractStructure, 10231).

How they combine

Who deletesAllowed byRefund to the owner
The document's ownercanBeDeleted: true, on a type that does not keep history, when the stored document meets every deleteConstraints ruleYes, except on a type with a ttl
The contract's moderatorsmoderatorAbilities.delete: true, within moderatorAbilities.deleteWithin when setOnly with moderatorAbilities.deleteRefundsOwner: true, except on a type with a ttl
The seated moderation team, togethermoderatorAbilities.deleteSettled, once moderatorAbilities.deleteWithin has passedAs for the moderators
The platformttl, once it has passedNo

Which reference may point at a type follows from which of the three it allows. A type that allows none of them is the target of a permanentDocument reference, with inList or without. A type that allows only the moderators' deletion, with removal records kept, is the target of a moderatedDocument reference: its documents never leave state without a record. Any other type is the target of a deletableDocument reference.

See also

Moderator Abilities

moderatorAbilities says what the contract's moderators may do to the documents of a type, whoever owns them: delete them, and write the fields that only they write. It is how an application gives its moderators a say over content without making them its authors: a report its moderators mark as handled, a post they flag, a ticket they assign.

Wheredocument type, in a contract whose config declares moderation
Valueobject with delete (boolean), deleteWithin (seconds), deleteKeepsRecord and deleteRefundsOwner (booleans), deleteKeepsFields (array of property paths), the last four only beside delete: true, deleteSettled (object, beside deleteWithin), and changeFields (array of top-level property names), at least one of them given
Defaultabsent: the moderators can do nothing to documents of the type
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212): a type can neither gain, lose nor change it. A type the update adds may declare it.

The keys:

KeyWhat it allowsDetails
deleteThe moderators delete documents of the type. By default each deletion leaves a removal record and can be restored within a week; with deleteKeepsRecord: false it leaves none and is final.Deletion
deleteWithinLimits delete to so many seconds after a document's last change.Deletion
deleteKeepsRecordWhether a deletion leaves a removal record, and so can be restored. Default true.Deletion
deleteRefundsOwnerWhether the deleted document's owner is refunded its storage. Default false.Deletion
deleteSettledPast deleteWithin, the seated team of an elected contract deletes a document together: one member proposes, and so many approve, the leader among them only when the rule says so (leader: true), and a member the leader added only for documents created after its addition unless the rule says otherwise (approversPredateDocument: false).Deletion
deleteKeepsFieldsThe fields of a deleted document whose values stay public in its removal record, such as a post's hashtag.Deletion
changeFieldsThe listed properties are written only by the moderators.below

The moderators are the ones the contract's moderation config declares: the contract owner, the identities it appoints, or the members of the seated team of an elected contract. A seated team holds an ability on a type only when the declaration's moderatedDocumentTypes gives it: deleteDocuments for delete, changeDocumentFields for changeFields. See Contract Moderation.

changeFields

The top-level properties of the type that only the contract's moderators write. A moderator sets or removes them on any document of the type with the contract user moderation transition. A document's owner can neither set them when creating the document nor change or remove them when replacing it, unless the owner is a moderator of the contract.

WheremoderatorAbilities of a document type
Valuearray of top-level property names, at least one, none twice
Defaultabsent: nobody but a document's owner writes its properties
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212)
ErrorsDocumentFieldNotChangeableByModeratorsError (41123), DocumentModeratorFieldNotWritableError (41124), InvalidContractModerationDocumentFieldsError (10905), IdentityNotContractModeratorError (41101), ContractModerationAbilityNotGrantedError (41201), ModerationReasonNotListedError (41203)

Example

"report": {
  "type": "object",
  "documentsMutable": false,
  "moderatorAbilities": { "delete": true, "changeFields": ["status", "resolution"] },
  "indices": [
    { "name": "byStatus", "properties": [{ "status": "asc" }, { "$createdAt": "asc" }] }
  ],
  "properties": {
    "postId": {
      "type": "array",
      "byteArray": true,
      "minItems": 32,
      "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "position": 0,
      "refersTo": { "type": "deletableDocument", "documentType": "post" }
    },
    "reason": { "type": "integer", "minimum": 0, "maximum": 8, "position": 1 },
    "status": { "type": "integer", "minimum": 1, "maximum": 3, "position": 2 },
    "resolution": { "type": "string", "maxLength": 200, "position": 3 }
  },
  "required": ["$createdAt", "postId", "reason"],
  "additionalProperties": false
}

A user files a report and can never edit it. A report starts with no status: the reporter cannot file one already marked handled. A moderator sets status and resolution when the report is dealt with, and queries the open ones through byStatus. The report stays, so the reporter and everyone else can see what became of it.

How it works

  • A moderator changes the fields with the contract user moderation transition's changeDocumentFields action, naming the document type, the document id, the new value of each field (null removes one; in the JavaScript SDKs a field set to undefined is left out, as JSON leaves it out) and a reason.
  • The transition is checked in this order, each refusal paid: the document type exists (InvalidDocumentTypeError, 10406); every field named is listed under changeFields (41123); the signer is a moderator of the contract (41101), and for a seated team the declaration gives it changeDocumentFields on the type (41201) and the reason names a reason document its proposal lists (41203); the document exists (40101) and has not expired (DocumentExpiredError, 40140); and the document as changed is still one of its type: its schema (JsonSchemaError, 10101), its propertyConstraints and its unique indexes (DuplicateUniqueIndexError, 40105). A change naming no field, or a system property, is refused before any state is read (10905).
  • Whoever owns the document may have it changed, the contract owner and the moderators included: the fields are the moderators', not the owner's, so the protection that keeps moderators from deleting each other's documents does not apply.
  • The document records who wrote the fields last, and when: $moderatedAt and $moderatedBy, which an index of the type may name to find what the moderators handled.
  • The document is updated in place. Every other property stays as its owner wrote it, $updatedAt among them, so a change never opens a deleteWithin window again. $revision goes up by one, so a replace its owner built on the earlier revision is refused (InvalidDocumentRevisionError, 40106) rather than writing over the change.
  • References are not checked again: no field a moderator writes holds one or is read by one (see the rules below). A report whose post a moderator has already deleted can still be marked handled.
  • The moderator pays for the transition and the bytes it adds. Storage the change frees, or index entries it moves, are refunded to the document's owner, who paid for them. Nothing the type prices is charged: a moderator's change is no action of the owner's.
  • A change that changes nothing, every field already holding the value named, is refused (10905): it would bump $revision for nothing.
  • A change needs a listed reason from a seated team, but does not count toward the team's action share: changes of one document have no bound, as deletions of the content that exists do, so counting them would let a member farm the share.
  • The proof of the change is the document as it now stands, holding the values the change set.

Owners who moderate

A create that sets a listed field, and a replace that changes, adds or removes one, is refused unless its signer is a moderator of the contract (41124), in a block and in the mempool alike. When the signer does moderate, the document is stamped as a moderator's: $moderatedAt the block's time, $moderatedBy the signer. A replace that leaves the listed fields alone keeps the stamp as it was. The contract owner is one under the appointed and owner-only kinds and during an interim that names it; once an elected contract has a seated team, only the team's leader and members are, and only where the declaration gives the team changeDocumentFields on the type. A replace carries the whole document, so an owner who is not a moderator must carry the listed fields exactly as the moderators left them.

Rules at registration

All refusals below are InvalidContractStructure (10231).

  • The contract's config must declare moderation, as for delete. A moderation block may then keep no list at all. An elected declaration must give the team changeDocumentFields on the type in moderatedDocumentTypes (InvalidContractModerationConfigError, 10900): once a team is seated only it writes the fields, and without the ability nobody could.
  • Refused on an indexOnly type: there is no stored row to change.
  • Refused on a type that requires a transient property: a transient value is never stored, and a moderator's change is judged against the schema, which would find it missing on every document.
  • Every entry must name a top-level property the type declares (list the object around a nested one), and that property must be:
    • optional: nobody but a moderator can set it, so it starts absent;
    • stored, so not transient;
    • not listed under immutable;
    • neither a refersTo reference nor read by one: not the referring value of a where entry, not a source findBy reads, not the identity property of a key id reference;
    • neither generatedFrom another property nor a parameter of one;
    • in no contested index.
  • A type that lists any keeps $revision on its documents, even when documentsMutable is false, because a moderator's change is stored as an update.
  • A refersTo findBy, or the list an inList reference reads, may not read a listed field of the type it refers to: such a field can change after the reference was checked.

See also

Time To Live (ttl)

ttl gives every document of a type a lifetime. The platform deletes each document once that many seconds have passed since it was created, whoever owns it and whatever the type says about who else may delete it. Use it for content that is meant to disappear: stories, invitations, offers, session records. Such documents are also cheaper: they pay for the time they occupy the state rather than for storage forever.

This is the document type keyword. An index can carry a ttl of its own inside timeRange, which expires index entries and leaves the documents in place; see Time-Range Indexes.

Wheredocument type
Valueinteger, seconds, 3600 (one hour) to 31536000 (one year)
Defaultabsent: documents live until someone deletes them
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212): an update may not add, remove or change it
ErrorsDocumentExpiredError (40140) for a replace, transfer, purchase or price update of a document past its expiry

Example

"story": {
  "type": "object",
  "ttl": 86400,
  "canBeDeleted": true,
  "properties": {
    "caption": { "type": "string", "maxLength": 200, "position": 0 },
    "mediaUrl": { "type": "string", "maxLength": 500, "position": 1 }
  },
  "required": ["$createdAt", "mediaUrl"],
  "additionalProperties": false
}

Each story is deleted one day (86,400 seconds) after its creation. Its author may delete it sooner. $createdAt must be in required, since the expiry is counted from it.

How it works

  • When a document expires. At its $createdAt plus ttl seconds. $createdAt is the block time of the create, so nothing the writer sends moves the expiry, and documents created in the same block expire together. A replace, transfer, purchase or price update never moves it either: a buyer of an expiring document buys what is left of its life.
  • When it is deleted. After each block's state transitions, the platform deletes expired documents, oldest first: at most 128 per block, and at most 1,024 in weight, where a document weighs 1 plus the index levels of its type (the values at protocol version 14). The rest wait for the next block. The deletion is an ordinary one: the document and every index entry go, and counts and sums are brought down.
  • Between expiry and deletion. A document past its expiry that the cleanup has not reached yet can still be queried and referenced, and it still holds its values in the type's unique indexes, so a create with the same unique value is refused as a duplicate until the cleanup has run. It can no longer be replaced, transferred, bought or repriced, and a moderator can no longer restore it: each is refused, paid, with DocumentExpiredError (40140). Its owner may still delete it where canBeDeleted allows.
  • Earlier deletion. The owner may delete a document before it expires when canBeDeleted allows it, and the contract's moderators when moderatorAbilities.delete does. canBeDeleted: false only stops the owner; the platform still deletes the document when it expires.
  • What it costs. The document is stored without storage flags and refunds nothing when it is deleted, by anyone. Instead of the price of permanent storage, each byte it writes pays a price for the time it will live: five tiers up to seven days, then a price per 9.125 days spanned. Creating it also prepays, as processing, the cost of its later deletion. A replace, transfer, purchase or price update pays for the bytes it adds at the price of the lifetime left. See Fees.
  • Proofs near the expiry. A write accepted in the last block before a document expires proves the document present. A proof fetched after the next block's cleanup finds it gone. The one-hour minimum keeps a newly created document in the state well past the moment its writer fetches the proof of the create.

Rules at registration

The refusals below are InvalidContractStructure (10231) unless a bullet says otherwise. They hold on every parse of the contract, except the bounds, which are checked when a contract is registered or updated.

  • $createdAt must be in required.
  • Refused together with documentsKeepHistory: true (Drive never deletes a document whose type keeps history), with indexOnly: true (there is no stored row to delete by id), and on a type with a contested index (a contested document waits in its vote poll, and could expire before it is stored).
  • A summed property (summable, averageable, documentsSummable or documentsAverageable) declares a minimum of at least 0, or a minimum of at least -134217728 and a maximum of at most 134217728 (±2^27, max_expiring_signed_summed_value_magnitude). Deleting an expired document takes its value out of the type's sums at the end of a block, with no transition to refuse: removing values that are never negative only lowers the sums, and values this small keep them in the signed 64-bit range short of 2^36 documents. Checked when a contract is registered or updated, like the bounds below.
  • At least min_document_ttl_seconds and at most max_document_ttl_seconds of SystemLimits: 3600 and 31536000 at protocol version 14. The meta-schema itself admits 1 to 4294967295, so a ttl of 0 is a JsonSchemaError (10101) and one outside the narrower bounds is 10231.
  • For references, a type with a ttl is deletable. A permanentDocument reference may not point at it, one found by findBy or with inList included (ReferencedDocumentTypeDeletableError, 40122); a deletableDocument reference may. See References.

Everything else combines with a ttl: mutable types, transferable, tradeMode, moderatorAbilities.delete, creationRestrictionMode, count, sum and ranked indexes, timeRange indexes with or without their own ttl, references declared on the type, action fees and token costs, with a summed property bounded as above.

On update

An update may not add, remove or change the ttl of an existing document type. Every stored document has the expiry it was written and paid with: adding one would leave stored documents that the cleanup cannot find, removing it would leave entries that delete documents the type says live forever, and changing it would move expiries away from what was paid for. A document type added by an update may declare a ttl freely.

See also

Creation, Transfers and Trading

A document belongs to the identity in its $ownerId: at first the one that created it. These three keywords decide who may create documents of a type, and whether a document may later change hands. creationRestrictionMode limits who creates. transferable lets an owner give a document away. tradeMode lets an owner put a price on a document and anyone else buy it at that price.

The example below is used throughout the chapter:

"ticket": {
  "type": "object",
  "documentsMutable": false,
  "canBeDeleted": false,
  "creationRestrictionMode": 1,
  "transferable": 1,
  "tradeMode": 1,
  "properties": {
    "event": { "type": "string", "maxLength": 63, "position": 0 },
    "seat": { "type": "string", "maxLength": 10, "position": 1 }
  },
  "required": ["event", "seat", "$createdAt"],
  "additionalProperties": false
}

Only the organiser, the identity that owns the contract, can issue tickets. A ticket's holder may give it to a friend, or list it for sale; anyone may then buy it at the listed price, with no approval from the seller. Nobody can edit a ticket or delete it.

creationRestrictionMode

Who may create documents of the type.

Wheredocument type
Value0 anyone, 1 the contract owner only, 2 nobody
Default0
Sinceprotocol version 1
On updateFixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsDocumentCreationNotAllowedError (10416)

How it works

  • 0: any identity may create documents of the type.
  • 1: only the identity that owns the contract may. A create signed by anyone else is refused (DocumentCreationNotAllowedError, 10416). Once created, a document may still pass to other identities if the type is transferable or tradeable.
  • 2: no create transition is ever accepted. This is for system contracts whose documents only the platform writes, such as the document history contract (see History) and the keyword search contract. On a user's contract the type would stay empty for good.
  • The mode rules creation only. Replaces, deletes, transfers and sales are decided by each document's own owner and by the other keywords.

Rules at registration

  • A type with mode 1 or 2 may not carry moderatorAbilities.delete (InvalidContractStructure, 10231): its documents belong to the contract owner or the platform, and no moderator may delete those. See Deletion.

transferable

Whether an owner may give a document to another identity.

Wheredocument type
Value0 never, 1 always
Default0
Sinceprotocol version 1
On updateFixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (10246).
ErrorsInvalidDocumentTransitionActionError (10404) for a transfer of a type set to 0

How it works

  • The owner sends a transfer transition naming the document, the recipient and a $revision one higher than the stored one (InvalidDocumentRevisionError, 40106). Only the owner may transfer (DocumentOwnerIdMismatchError, 40102). The recipient does not have to agree.
  • The document gets the recipient as its $ownerId and a new $revision. A price it was listed at is removed, so a transferred document is no longer for sale. When the type lists $transferredAt in required, it is set to the block's time, and likewise $transferredAtBlockHeight and $transferredAtCoreBlockHeight to the block heights.
  • A transfer carries no property values, and $creatorId keeps naming the identity that created the document. The one change the platform makes itself: from protocol version 13, a DPNS domain that is transferred or sold has its records.identity pointed at the new owner.
  • The document is checked as it will be stored, with its new owner: against the type's unique indexes (DuplicateUniqueIndexError, 40105), against a distinctFrom: "$ownerId" property (DocumentPropertyNotDistinctError, 10419), and against propertyConstraints rules that read $ownerId (DocumentPropertyConstraintViolatedError, 10422).
  • On a moderated contract, a transfer to a banned or suspended identity is refused (ContractModerationCounterpartyBarredError, 41114).
  • A transfer of a document past its ttl expiry is refused (DocumentExpiredError, 40140).
  • With keepsTransferHistory: true, each transfer is also recorded in the document history contract. See History.

tradeMode

Whether documents of the type can be sold through the platform's built-in marketplace. With 1, direct purchase, an owner sets a price and any other identity may buy the document at that price, with no approval.

Wheredocument type
Value0 none, 1 direct purchase
Default0
Sinceprotocol version 1
On updateFixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (10246).
ErrorsInvalidDocumentTransitionActionError (10404) for a price update or a purchase on a type set to 0, and for a purchase by the document's own owner; DocumentNotForSaleError (40108); DocumentIncorrectPurchasePriceError (40109)

How it works

  • Listing. The owner sends a price update transition with a price in credits and the next $revision. Only the owner may set the price (40102). The price is stored with the document, the revision goes up, and $updatedAt is set when the type lists it in required.
  • Buying. Any identity other than the owner sends a purchase transition naming the document, the next $revision and the price. A document with no price set is not for sale (DocumentNotForSaleError, 40108), and the price in the transition must equal the listed price exactly (DocumentIncorrectPurchasePriceError, 40109). An owner cannot buy its own document (10404).
  • What a purchase does. The price moves from the buyer's credit balance to the seller's. The document gets the buyer as its $ownerId and a new $revision, the listing is removed, and $transferredAt is set when the type lists it in required. The buyer's credit balance must cover the price.
  • The new owner is checked exactly as for a transfer: unique indexes (40105), distinctFrom: "$ownerId" (10419) and propertyConstraints rules that read $ownerId (10422).
  • On a moderated contract, a banned or suspended buyer is refused like any barred writer, and so is a purchase from a banned or suspended seller (ContractModerationCounterpartyBarredError, 41114).
  • A price update or a purchase of a document past its ttl expiry is refused (DocumentExpiredError, 40140).
  • With keepsPricingHistory and keepsPurchaseHistory, each price update and each purchase is also recorded in the document history contract. See History.

How they combine

  • transferable and tradeMode are independent. The DPNS domain type sets both, so a name can be given away or sold. A type may allow sales without gifts, or gifts without sales.
  • Neither needs documentsMutable. A document that cannot be replaced can still change owner and price, as the ticket above does.
  • A type that is transferable or tradeable stores a $revision on each document even when its documents cannot be replaced, and, from protocol version 10 on a format-1 contract whose config is version 1 or later, records each document's creator in $creatorId. See System Properties.
  • Every action has its own optional token cost and action fee: transfer, update_price and purchase, besides create. See Token Costs and Action Fees.
  • signatureSecurityLevelRequirement applies to all of these actions, so a buyer signs a purchase with a key at the level the type requires. See Signing and Keys.

Rules at registration

All refusals below are InvalidContractStructure (10231).

  • ownerRefersTo is refused on a type whose documents can be transferred or traded: a document could end up with an owner the declaration never checked. Such a type uses creatorRefersTo, which checks the creator, who never changes. creatorRefersTo is only accepted on such a type. See Writer and Creator References.
  • An indexOnly type can be neither transferable nor tradeable. See Index-Only Types.
  • On a transferable or tradeable type, an immutable property may not hold a contract reference with an owner requirement. See Mutability.
  • creationRestrictionMode 1 or 2 is refused together with moderatorAbilities.delete.

See also

History

Platform can keep two kinds of history for a document type. documentsKeepHistory keeps every version of each document in Drive, under the document itself, so an application can read what a document said at any earlier time. The three keeps*History flags record events instead: each transfer, purchase or price update of a document becomes a record in the document history system contract, where it can be queried by document, by contract, by identity and by time.

documentsKeepHistory

Keeps every version of every document of the type, not only the latest.

Wheredocument type
Valueboolean
Defaultthe contract config's documentsKeepHistoryContractDefault, which is false unless the contract says otherwise
Sinceprotocol version 1
On updateFixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsInvalidDocumentTransitionActionError (10404) for a delete of a document of the type, from protocol version 14

Example

"profile": {
  "type": "object",
  "documentsKeepHistory": true,
  "canBeDeleted": false,
  "properties": {
    "displayName": { "type": "string", "maxLength": 25, "position": 0 },
    "bio": { "type": "string", "maxLength": 140, "position": 1 }
  },
  "required": ["displayName"],
  "additionalProperties": false
}

Every edit of a profile adds a version, and every earlier version stays readable. canBeDeleted: false has to be written out, since a type that keeps history can never delete and the default is true.

How it works

  • Each document is stored as a small tree of its own: every version the document has had, keyed by the block time it was written at, and a pointer to the current one. A query by id or through an index sees the current version.
  • Every write that changes the document adds a version: a replace, and also a transfer, a price update or a purchase.
  • Nothing is ever removed. Drive refuses to delete a document whose type keeps history, so its owner's delete is refused (10404 from protocol version 14; before it, the delete failed inside Drive as an internal error), and the type can have neither moderator deletion nor a ttl.
  • The getDocumentHistory query returns a document's versions from a given time on, each with the block time it was written at, at most 10 per request, and with a proof when asked.
  • Every version stays stored, paid for by the write that added it.
  • A doctype-level sum or average (documentsSummable, documentsAverageable) counts only each document's current version. See Counts, Sums and Averages.

Rules at registration

All refusals below are InvalidContractStructure (10231).

  • From protocol version 14, the type must set canBeDeleted: false. A contract registered earlier with both flags on stays readable, and its next update must turn canBeDeleted off on that type. See Deletion.
  • Refused together with ttl, with moderatorAbilities.delete and with indexOnly.

keepsTransferHistory

Records every transfer of a document of the type in the document history contract.

Wheredocument type
Valueboolean
Defaultfalse
Sinceprotocol version 13
On updateFixed (DocumentTypeUpdateError, 40212)
Errorsnone of its own

keepsPurchaseHistory

Records every purchase of a document of the type in the document history contract.

Wheredocument type
Valueboolean
Defaultfalse
Sinceprotocol version 13
On updateFixed (DocumentTypeUpdateError, 40212)
Errorsnone of its own

keepsPricingHistory

Records every price update of a document of the type in the document history contract.

Wheredocument type
Valueboolean
Defaultfalse
Sinceprotocol version 13
On updateFixed (DocumentTypeUpdateError, 40212)
Errorsnone of its own

The document history contract

Example

"ticket": {
  "type": "object",
  "documentsMutable": false,
  "canBeDeleted": false,
  "transferable": 1,
  "tradeMode": 1,
  "keepsTransferHistory": true,
  "keepsPurchaseHistory": true,
  "keepsPricingHistory": true,
  "properties": {
    "event": { "type": "string", "maxLength": 63, "position": 0 },
    "seat": { "type": "string", "maxLength": 10, "position": 1 }
  },
  "required": ["event", "seat"],
  "additionalProperties": false
}

Each time a ticket is given away, listed or sold, a record of it is written to the document history contract, so anyone can look up who held a ticket, what it was offered at and what it sold for. The DPNS domain type sets all three flags from protocol version 13.

How records work

The document history contract is a system contract registered at protocol version 13, with the id 6voHRaoiPcfmMhbqCA9dixH98xcgPQ9UEcuaXjpVu3LD. It has one document type per flag:

FlagRecord typeWritten forProperties$ownerId of the record
keepsTransferHistorytransfereach transferdataContractId, documentTypeName, documentId, toIdentityIdthe sender
keepsPurchaseHistorypurchaseeach purchasedataContractId, documentTypeName, documentId, sellerId, pricethe buyer
keepsPricingHistorypriceUpdateeach price updatedataContractId, documentTypeName, documentId, pricethe owner who set the price
  • The platform writes the record as part of the transition that made the change, so a record exists exactly when the transfer, purchase or price update succeeded. Each record also carries $createdAt and $createdAtBlockHeight, the block's time and height.
  • The identity that signed the action owns the record, and writing it is part of that transition's fees.
  • Records can never be changed or deleted, and nobody can create one directly: the record types set documentsMutable: false, canBeDeleted: false and creationRestrictionMode: 2.
  • The record types are indexed for lookups by document (byDocument: contract, document, time) and by contract (byContract: contract, time). Transfers are also indexed by sender (from) and recipient (to), purchases by buyer (buyer), seller (seller) and price (byPrice).
  • They also keep provable aggregates. purchase keeps a count, total and average of price over all its records, and per contract or per document over a time range. priceUpdate keeps a count of all its records, and a count, total and average of asking prices per contract or per document over a time range, where a document listed three times counts three times. transfer keeps a count of all its records, and per contract over a time range. See Counts, Sums and Averages.
  • A flag only records the action the type allows. keepsTransferHistory on a type that is not transferable records nothing; no rule ties the flags to transferable or tradeMode.
  • The flags are fixed on update, so an existing type cannot start or stop recording. A type added by an update may set them.

Rules at registration

See also

Signing and Keys

These keywords tie a document type to identity keys. signatureSecurityLevelRequirement sets how strong a key must be to sign transitions on documents of the type. requiresIdentityEncryptionBoundedKey and requiresIdentityDecryptionBoundedKey let identities register encryption and decryption keys bound to the type, for applications that encrypt data between users, and say how many such keys an identity may hold.

Every identity key has a purpose (authentication, encryption, decryption and others) and a security level. The levels are, from strongest to weakest, MASTER (0), CRITICAL (1), HIGH (2) and MEDIUM (3): a lower number is a stronger key. See Identity Keys.

signatureSecurityLevelRequirement

The weakest key security level that may sign a transition on documents of the type. Raise it to 1 for documents whose forgery would be costly, so that a weaker everyday key cannot write them.

Wheredocument type
Value1 critical, 2 high, 3 medium
Default2 (high)
Sinceprotocol version 1
On updateFixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsInvalidSignaturePublicKeySecurityLevelError (20004)

Example

"payout": {
  "type": "object",
  "documentsMutable": false,
  "canBeDeleted": false,
  "signatureSecurityLevelRequirement": 1,
  "properties": {
    "recipient": {
      "type": "array",
      "byteArray": true,
      "minItems": 32,
      "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "position": 0
    },
    "amount": { "type": "integer", "minimum": 1, "position": 1 }
  },
  "required": ["recipient", "amount"],
  "additionalProperties": false
}

Only a critical key may sign a transition that creates a payout. A high or medium key of the same identity is refused.

How it works

The requirement admits the level it names and every stronger level except MASTER:

ValueKeys that may sign
1 criticalcritical
2 high (the default)critical, high
3 mediumcritical, high, medium
  • It applies to every document transition on the type: create, replace, delete, transfer, price update and purchase. A purchase is signed by the buyer, so the buyer needs a key at that level.
  • A batch transition signs all its transitions with one key. When it holds transitions on several document types, the key must satisfy the strictest of their requirements. A batch that also holds a token transition needs a critical key.
  • The key must be an authentication key. A master key never signs a document batch, whatever the requirement: the batch is refused at the signature check (20004).
  • A key whose level the requirement does not admit is refused with InvalidSignaturePublicKeySecurityLevelError (20004) after the signature has been verified. The failure is paid: the identity's nonce for the contract is bumped and the fees are charged.

Rules at registration

  • The meta-schema admits only 1, 2 and 3 (JsonSchemaError, 10101 otherwise). 0, master, cannot be required.

requiresIdentityEncryptionBoundedKey

Lets identities add encryption keys bound to this document type, and says how they are kept. Use it when documents of the type carry data encrypted to their readers, and each identity publishes the key others encrypt to.

Wheredocument type
Value0 unique, 1 multiple, 2 multiple with a pointer to the latest
Defaultabsent: no encryption key may be bound to the type
Sinceprotocol version 1
On updateFixed (DocumentTypeUpdateError, 40212)
ErrorsDataContractBoundsNotPresentError (10515) for an encryption key bound to a type that does not declare it; IdentityPublicKeyAlreadyExistsForUniqueContractBoundsError (40211) for a second key under 0

requiresIdentityDecryptionBoundedKey

The same for decryption keys.

Wheredocument type
Value0 unique, 1 multiple, 2 multiple with a pointer to the latest
Defaultabsent: no decryption key may be bound to the type
Sinceprotocol version 1
On updateFixed (DocumentTypeUpdateError, 40212)
ErrorsDataContractBoundsNotPresentError (10515) for a decryption key bound to a type that does not declare it; IdentityPublicKeyAlreadyExistsForUniqueContractBoundsError (40211) for a second key under 0

Example

Trimmed from the DashPay contract's contactRequest:

"contactRequest": {
  "type": "object",
  "documentsMutable": false,
  "canBeDeleted": false,
  "requiresIdentityEncryptionBoundedKey": 2,
  "requiresIdentityDecryptionBoundedKey": 2,
  "properties": {
    "toUserId": {
      "type": "array",
      "byteArray": true,
      "minItems": 32,
      "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "position": 0
    },
    "encryptedPublicKey": {
      "type": "array",
      "byteArray": true,
      "minItems": 96,
      "maxItems": 96,
      "position": 1
    },
    "senderKeyIndex": { "type": "integer", "minimum": 0, "position": 2 },
    "recipientKeyIndex": { "type": "integer", "minimum": 0, "position": 3 }
  },
  "required": ["toUserId", "encryptedPublicKey", "senderKeyIndex", "recipientKeyIndex"],
  "additionalProperties": false
}

An identity may bind any number of encryption and decryption keys to contactRequest, and the platform keeps a pointer to the newest of each. The request records the ids of the sender's and the recipient's keys in senderKeyIndex and recipientKeyIndex.

How it works

  • An identity key may carry contract bounds: a contract, or one document type of a contract. An encryption or decryption key bound to a document type is only accepted when the type declares the matching keyword; otherwise it is refused with DataContractBoundsNotPresentError (10515). The check runs when an identity is created with such a key, or updated to add one. The bound contract and document type must exist (DataContractNotPresentError, 10400; InvalidDocumentTypeError, 10406).
  • The value says how the identity's keys of that purpose, bound to the type, are kept:
    • 0, unique: the identity holds at most one, and it can never be replaced. A second is refused (IdentityPublicKeyAlreadyExistsForUniqueContractBoundsError, 40211).
    • 1, multiple: the identity may hold any number.
    • 2, multiple with a pointer to the latest: any number, and the platform keeps a pointer to the most recently added one, so a client reads the current key in one step.
  • Clients read keys bound to a contract or a document type with the getIdentitiesContractKeys query, which takes the identities, the contract, an optional document type name and the purposes.
  • Consensus reads these keywords only when keys are added. Nothing requires the writer of a document to hold such a key, and nothing checks which key encrypted a property. See encryptedFor for what consensus does check about encrypted values.
  • Encryption and decryption keys are MEDIUM keys. See Identity Keys.
  • Keys bound to the whole contract, rather than one document type, are governed by the contract config keys of the same names. See Contract-Level Keys and config.
  • Before protocol version 12, a decryption key bound to a document type was checked against requiresIdentityEncryptionBoundedKey by mistake. From 12 each purpose reads its own keyword.
  • From protocol version 14 an authentication key may also be bound to a contract or a document type, to limit what it may sign. That needs neither keyword. See Contract Bounds.

See also

References (refersTo)

An identifier (a 32-byte id) can hold any value. refersTo says what it points at, and Platform then checks, whenever a document is created or replaced, that the thing it names exists: an identity, a data contract, a token, a document, or one key of an identity. Reach for it when a document only makes sense next to something else: a reply needs its post, a join request needs the proposal it joins, an encrypted message needs the key it was encrypted to. The check runs when the document is written. Nothing checks the reference again when its target changes later, and nothing resolves it for a reader.

WhereAn identifier property, at the top level or inside an object; the items of a typed array of identifiers, where every element is checked; for one form of identityPublicKey, an integer key id property; and a string or byte array property whose value a findBy function reveals (see Commit and reveal). ownerRefersTo and creatorRefersTo carry the same declaration at the document type level.
ValueAn object: type, naming one target, with the keys that target takes; or an object holding only anyOf or only allOf (see Expressions).
DefaultAbsent: the identifier is not checked against anything.
Sinceprotocol version 14
On updateFixed: adding, removing or changing any part of a declaration is refused (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsReferencedEntityNotFoundError (40120) when the target does not exist; the full list is under Errors.

Example

"reply": {
  "type": "object",
  "documentsMutable": true,
  "canBeDeleted": true,
  "properties": {
    "postId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "refersTo": { "type": "deletableDocument", "documentType": "post" },
      "position": 0
    },
    "mentions": {
      "type": "array", "minItems": 0, "maxItems": 5, "uniqueItems": true,
      "items": {
        "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
        "contentMediaType": "application/x.dash.dpp.identifier",
        "refersTo": { "type": "identity" }
      },
      "position": 1
    },
    "text": { "type": "string", "minLength": 1, "maxLength": 280, "position": 2 }
  },
  "required": ["postId", "text"],
  "additionalProperties": false
}

postId must be the id of a post document of this contract that exists when the reply is written. Posts can be deleted, so the reference is a deletableDocument one. Each of the up to five mentions must be the id of an existing identity. The type carries six references at most: one for postId and five for mentions.

Targets

type names what the value points at.

typeThe value must beKeys it takes
identitythe id of an existing identitynone
contractthe id of an existing data contractcontractRequirements
tokenthe id of an existing tokennone
permanentDocumentthe id of an existing document of a type whose documents can never disappear; with findBy, one part of the key that finds it; with inList, one of the identifiers a list on it holdsdocumentType (required), contractId, findBy, where, inList, and beside a findBy function minimumAgeBlocks
moderatedDocumentthe id of an existing document of a type whose documents disappear only when the contract's moderators remove them, on the recorddocumentType (required), contractId, where
deletableDocumentthe id of an existing document of a type whose documents can disappear in any other way; with findBy, one part of the key that finds itdocumentType (required), contractId, findBy, where, and beside a findBy function minimumAgeBlocks and consume
identityPublicKeyan identity key that exists and is not disabledkeyIdProperty or identityProperty (one of them, required), keyRequirements

A key that belongs to another target is refused when the contract is registered.

identity

The value is the id of an identity that exists. The check reads the identity's revision. Identities are never removed, so a validated reference never dangles.

contract

The value is the id of a data contract that exists. contractRequirements can ask more of it: that it declares elected moderation, has reached a certain age, belongs to the writer, and so on. Contracts are never deleted.

token

The value is the id of a token. The check reads the token's record, which is written when its contract is registered and never removed.

permanentDocument

The value is the id of a document of documentType, in this contract or in the one contractId names. The referenced type must be one whose documents can never disappear: canBeDeleted: false (not "onlyWhenConsumed"), no moderatorAbilities.delete and no ttl. None of those can change on a contract update and document types are never removed, so a validated permanent reference never dangles.

"reasons": {
  "type": "array", "minItems": 0, "maxItems": 64, "uniqueItems": true,
  "items": {
    "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
    "contentMediaType": "application/x.dash.dpp.identifier",
    "refersTo": { "type": "permanentDocument", "documentType": "reason" }
  },
  "position": 2
}

This is the moderation charters contract's submittedCharter.reasons: every element must be the id of a reason document, a type that is immutable and can never be deleted. With findBy the value is instead one part of a unique index key that finds the document, and with inList one of the identifiers a list on the document holds.

moderatedDocument

The same for a type whose documents disappear only when the contract's moderators remove them, and never without a trace: canBeDeleted: false, no ttl, and moderatorAbilities.delete with deleteKeepsRecord left at its default, true. Every removal then leaves a removal record under the contract, holding the document's id, its owner, the moderator, the time, the reason and a hash of the document, and nothing ever deletes it. So a validated moderated reference always resolves: to the document, or to the record of its removal, from which a moderator can restore the document as it was.

"postId": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "refersTo": { "type": "moderatedDocument", "documentType": "post" },
  "position": 0
}

A reply to a post its author can not delete, which the moderators can take down. The post must exist when the reply is written. Once a moderator removes it, the reply keeps pointing at it, and a replace of the reply is checked as if the reference were permanent: an edit of the reply's text passes, and the reference now resolves to the removal record. What a replace can not do is point a reply at a removed post, or write a new reply to one: a value the write sets must name a document in state.

A removed document has no values left but those its record keeps: its id, its owner, and the fields its type lists under moderatorAbilities.deleteKeepsFields. A where entry the replace checks again (its referring property changed, or it is a writer gate valued "$ownerId") is checked against the record when it compares the referenced $ownerId, $id or a property the type keeps (a kept path, or one inside a kept object), with the same absence rule as against a document in state, and refused with ReferencedDocumentRemovedError (40145) when it compares any other property; the replace can repoint the reference at a document in state, or leave that property unchanged until the post is restored. A writer gate (a where entry valued "$ownerId") is checked on every replace, so on a moderatedDocument reference it may compare only the referenced $ownerId or $id (InvalidContractStructure, 10231, otherwise), the only values sure to be in every record: a gate on anything else could refuse every replace once the post is removed. The value is kept as long as it is the one the stored document held, whatever else the replace changed in the same object or list. The value is always the document's id: findBy, inList and operands of an expression are refused, since a removal frees the unique index keys a findBy finds by.

A chained or composite query that joins through a moderated reference proves each removed document's record beside the documents it joins, and reports the removal with it.

An index of an indexOnly type may be preallocated through a moderated reference when the removal record keeps every key of its path: each where entry the index uses compares the referenced $id, $ownerId or a property the type keeps. The trees then outlive a removed post the way its record does, and a restore finds them in place.

deletableDocument

The same for a type whose documents can disappear without a record: deleted by their owner (canBeDeleted), deleted by a create that consumes them (canBeDeleted: "onlyWhenConsumed"), removed by the contract's moderators when deleteKeepsRecord is false, or removed by the platform when their ttl passes. A type whose documents only moderators remove, keeping records, is a moderatedDocument target instead, and a deletableDocument reference to it is refused. The document must exist when the referring document is written, and may be deleted afterwards.

Because the target may be gone, every replace of the referring document checks the reference again, whether or not the replace touched it. Once the target is deleted, the replace has to point the property at a document that exists or remove it. A required property cannot be removed, so a document whose required reference has lost its target can be replaced only after it is repointed; it can still be deleted. A single deletableDocument reference held by an immutable top-level property may be removed by a replace once its target is gone, an exception to the immutability rule.

identityPublicKey

One key of one identity, which must exist and not be disabled. Identity keys can be disabled but never removed, so a validated key reference never dangles, while a disabled key refuses new writes. The declaration comes in two forms, which differ in which property carries it:

  • On the identity property. The identifier holds the identity's id and keyIdProperty names the sibling integer property holding the key id. The moderation charters contract's joinRequest.recipientId:

    "recipientId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "refersTo": {
        "type": "identityPublicKey",
        "keyIdProperty": "recipientKeyId",
        "keyRequirements": { "purpose": "decryption", "boundTo": "submittedCharter" }
      },
      "position": 1
    }
    
  • On the key id property. The integer holds the key id and identityProperty names whose key it is. The property must declare exactly the range of a key id, "minimum": 0 and "maximum": 4294967295. The same contract's joinRequest.senderKeyId, a key of the writer:

    "senderKeyId": {
      "type": "integer", "minimum": 0, "maximum": 4294967295,
      "refersTo": {
        "type": "identityPublicKey",
        "identityProperty": "$ownerId",
        "keyRequirements": { "purpose": "encryption", "boundTo": "joinRequest" }
      },
      "position": 3
    }
    

A key reference pairs the value with one key id, so it is refused on the elements of a typed array, as an operand of an expression and in ownerRefersTo or creatorRefersTo.

Keys

documentType

The name of the referenced document type, 1 to 64 letters, digits or underscores. Required on permanentDocument, moderatedDocument and deletableDocument, and refused on the other targets. For permanentDocument the type's documents must never disappear; for moderatedDocument they must disappear only through a moderator's recorded removal; for deletableDocument they must be able to disappear otherwise. With inList it is the type holding the list.

contractId

The contract holding documentType, as a base58 string or an array of 32 bytes. Absent means the declaring contract; naming the declaring contract's own id means the same. Only on the two document targets. A reference into another contract costs a billed fetch of that contract when a document is written, once per declaration: the elements of a typed array and the operands of an expression share it.

where

States what the referenced document must hold once found: each entry { "<referenced property>": "<referring value>" } must hold as an equality when the referring document is written. It takes 1 to 10 entries, on permanentDocument, moderatedDocument and deletableDocument references. where never finds the document: the value does, as its id, or findBy does. where is checked against the document found.

  • The key is the referenced side: a property of the referenced document type, a dotted path for a nested one, or one of the referenced document's own identifiers: $ownerId (its current owner, which follows it through transfers), $creatorId (its creator, which never changes, on types that record it, see System Properties) or $id (its id). A system name needs an identifier on the referring side.
  • The value is the referring side: a property of the declaring document type, a dotted path for a nested one, or "$ownerId", the writer. An entry whose value is "$ownerId" is a write gate: only an identity whose id equals the referenced side may create or replace the document. A referring value may appear once in where.
  • Absence counts. Both sides absent agree; one side absent is a mismatch, as a different value is.
  • Values, not keys. The two sides compare as values of their type: strings as text (so "" is not "\0"), byte arrays and identifiers as bytes (an identifier equals the same 32 bytes, however it is carried), integers as numbers whatever width they are carried at, floats exactly, bit for bit (so -0.0 is not 0.0). A value of any length compares, past the 255 bytes of an index key too.
  • The comparison reads the document already fetched for the existence check, so it costs nothing more. An entry that does not hold refuses the write with ReferencedDocumentPropertyMismatchError (40127); no document found is still ReferencedEntityNotFoundError (40120).
  • No "." and no functions. They find the document, so they belong in findBy, where a function is the key of a commit and reveal.
"submittedCharterId": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "refersTo": {
    "type": "permanentDocument",
    "documentType": "submittedCharter",
    "where": {
      "$ownerId": "$ownerId",
      "targetContractId": "targetContractId"
    }
  },
  "position": 1
}

This is the moderation charters contract's electedCharter.submittedCharterId: the proposal it names must be owned by the writer, and must be for the same targetContractId as the elected charter. The same contract's joinRequest.submittedCharterId declares "where": { "$ownerId": "recipientId" }: the proposal's owner must be the join request's recipientId. Other patterns: { "$ownerId": "authorId" } makes a like carry its post's owner in authorId, and { "$creatorId": "$ownerId" } lets only the referenced document's creator write.

Read as SQL, documentType is the FROM, the value (or findBy) the part of the WHERE that finds the row, and where the rest of the WHERE, checked on the row found:

SELECT * FROM submittedCharter
WHERE $id = :submittedCharterId                  -- the value, an id reference
  AND $ownerId = :writer                         -- where
  AND targetContractId = :targetContractId       -- where

At registration both sides must exist and hold the same type of value (for integers, the same stored integer type). Neither side may be an object or a typed array, the referring side may not be the reference property itself, and the referenced side may not be transient (no stored document carries it; the referring side may be). An entry through which a preallocated index is keyed needs a referenced property whose every value fits an index key, at most 255 bytes: a string of at most 63 characters or with maxBytes at most 255, or a byte array of at most 255 bytes. An entry breaking one of these is refused with ReferencedDocumentPropertyAgreementInvalidError (40126).

The keyword propertyAgreement, whose pairs were keyed the other way ({ "<referring property>": "<referenced property>" }), was replaced by where before protocol version 14 shipped. A declaration still carrying it is refused on every parse, with a message saying to key where by the referenced document's property. "propertyAgreement": { "recipientId": "$ownerId" } is now "where": { "$ownerId": "recipientId" }, and a symmetric pair such as "$ownerId": "$ownerId" reads the same.

contractRequirements

What a contract reference requires of the referenced contract beyond existing. It holds at least one of these keys:

KeyValueMet when
moderation"elected"the contract declares an elected moderation team
"electionOpen"the contract declares an elected team, and its own electionDelay, counted from the contract's creation, has passed at the block time of the write (or it declares no delay)
minimumAgeSecondsinteger, 1 to 4294967295the contract's recorded creation time is at least that many seconds before the block time of the write
minimumSecondsSinceUpdateinteger, 1 to 4294967295the same, counted from the later of its creation and its last update
owner"self"the contract is owned by the writer of the referring document
"other"the contract is owned by anyone else
readonlytruethe contract's config is readonly: it can never be updated again
keepsHistorytruethe contract's config keeps history
ownerProtectedtrue or falsethe contract's elected moderation protects, or does not protect, its owner from the team; a contract without elected moderation meets neither

A contract with no recorded creation time never meets minimumAgeSeconds or minimumSecondsSinceUpdate. The requirements are judged against the contract already fetched for the existence check, the writer and the block time, so they cost no further read. The first unmet one refuses the write with ReferencedContractRequirementNotMetError (40135); a contract that does not exist is still ReferencedEntityNotFoundError (40120).

The moderation charters contract uses both moderation values: a proposal (submittedCharter.targetContractId) needs a target that is elected, so teams can form while the target's election delay runs, and the charter that opens the election (electedCharter.targetContractId) needs it electionOpen:

"refersTo": { "type": "contract", "contractRequirements": { "moderation": "electionOpen" } }

owner is the one requirement judged against the writer, and a transfer or a purchase changes the owner without a write. On a type whose documents can be transferred or traded, a reference carrying owner is therefore checked again on every replace, so a new owner has to repoint it at a contract that meets the requirement for them, or remove it where it is optional. Such a reference may not sit under an immutable property of such a type, which could never be repointed. See Elected Moderation for the moderation values.

keyIdProperty and identityProperty

The two forms of identityPublicKey. A declaration takes exactly one of them.

  • keyIdProperty, on the identity property: the path of the integer property of the same document type holding the key id. It must exist, be an integer, and not carry a key reference of its own. If the identity is set and the key id is not, the write is refused with ReferencedKeyIdPropertyInvalidError (40125).
  • identityProperty, on the key id property: whose key the value is.
    • "$ownerId": the writer. The writer's existence is already proven, so the key fetch is the only read.
    • "$creatorId": the document's creator, only on a type that records creator ids. A document written before its type recorded them has none, and setting the key id on it is refused (40125).
    • The path of an identifier property of the same type, which must exist, be an identifier and not carry an identityPublicKey reference of its own. A key id set while that property is not set is refused (40125).

A stored key id may not be paired with a transient identity, since the key id alone names no key. Each rule is checked at registration and refused with ReferencedKeyIdPropertyInvalidError (40125).

keyRequirements

What an identityPublicKey reference requires of the key beyond existing and not being disabled. It holds at least one of:

  • purpose: the key's purpose, one of authentication, encryption, decryption, transfer, voting or owner.
  • boundTo: a document type of the declaring contract. The key must be bound to exactly this contract and that document type; a key bound to the whole contract or to a contract group does not meet it.

At registration boundTo must name a document type of the contract, and one a key of the required purpose can be bound to: only authentication, encryption and decryption keys carry a document type bound, an encryption key only where the type declares requiresIdentityEncryptionBoundedKey, and a decryption key only where it declares requiresIdentityDecryptionBoundedKey (see Signing and Keys). The requirements are judged against the key already fetched, and the first unmet one refuses the write with ReferencedIdentityKeyRequirementNotMetError (40136). A missing key is still 40123 and a disabled one 40124.

findBy, inList, anyOf and allOf

  • findBy finds a permanentDocument or deletableDocument through the unique index of its type over exactly the properties it names, with the value as one part of the key. One entry may be a function, for a commit and reveal, beside which the refersTo may declare minimumAgeBlocks and consume.
  • inList, on a permanentDocument whose findBy is { "$id": <property> }, names the list on that document the value must be in.
  • anyOf and allOf combine several targets in one declaration.

How it works

On create

When a document is created, every reference is checked against the current state: the ownerRefersTo or creatorRefersTo declaration first, then each property's. The first check that fails refuses the write with that check's error, naming the property by its path (postId, meta.charterId), an element by its list path (mentions[2] for the third) and the writer or creator reference as $ownerId or $creatorId. A refused write is still charged for the reads it made.

  • A reference property the document leaves out is not checked. Whether it may be left out is up to required.
  • Each check is a billed read: the identity, the contract (none for the declaring contract, which is already loaded), the token's record, the document by id or by findBy, or the key. One write fetches a given document by id once, however many references name it.
  • On a typed array each element is checked in list order as a single reference would be. An element repeating an earlier one is not checked twice.

On replace

A replace checks a reference again only when its outcome could have changed:

DeclarationChecked again on a replace when
identity, token, contractthe value changed
contract with an owner requirement, on a type whose documents can be transferred or tradedevery replace
permanentDocument, moderatedDocumentthe value changed, or the referring value of a where entry changed
any document reference with a where entry whose value is "$ownerId"every replace
permanentDocument with findBy, inList includedalso when a property findBy reads changed
deletableDocument, by id or with findByevery replace
a document reference with a findBy functionnever: the whole declaration, its where included, is judged on the create alone (see Commit and reveal)
identityPublicKey with keyIdPropertythe identity or the key id changed
key id with identityProperty: "$ownerId"every replace
key id with identityProperty: "$creatorId"the key id changed
key id with an identity property paththe key id or that property changed
anyOf or allOfone of its operands would be checked again; the whole expression is then checked

A value changed when the replace set it differently, added it or removed it. Changes are tracked per top-level property, so a change anywhere in an object checks again every reference inside that object. When a typed array changed, only the elements the stored list did not hold are checked, unless a rule above that applies to every element does (the referring value of a where entry changed, a where entry valued "$ownerId", an owner requirement on such a type, or a deletableDocument target); then every element is checked. The rules for ownerRefersTo and creatorRefersTo are in Writer and Creator References.

The "every replace" rows exist because something the reference depends on can change without a write to the referring document: the writer after a transfer or purchase, or the target's existence for a deletable one. A moderated target that a moderator removed is not one of them: the reference resolves to its removal record, as described above.

Transfers, purchases, deletes and restores

  • A transfer or a purchase checks no reference. A reference governs writing, not holding: a new owner meets the writer gates (a where entry valued "$ownerId", identityProperty: "$ownerId", an owner requirement) on their first replace.
  • Deleting a referring document checks nothing. Deleting a referenced document does not look for documents referring to it: a permanentDocument target cannot be deleted at all, a moderatedDocument target is removed by a moderator on the record the reference then resolves to, and a deletableDocument reference meets its missing target on the referring document's next replace. A deletableDocument target can still refuse its owner's delete while documents refer to it: a deleteConstraints rule counting them by the target's own id, { "equal": [{ "countOf": ["vote", { "pollId": "$id" }] }, 0] }, needs a countable index on the referring property, and holds the target in place until the last of them is gone.
  • A document a moderator removed and later restores comes back as it was, without its references being checked again (see Restoring Documents).

The reference budget

Every reference is a billed read when a document is written, so a document type may carry at most 256 references per document. Registration counts:

  • one for each property declaring refersTo, whether an identifier or a key id;
  • maxItems for each typed array whose elements declare it;
  • one for the type's ownerRefersTo or creatorRefersTo;
  • each of these multiplied by the number of leaves when the declaration is an expression.

The reply above counts 6. A typed array of maxItems 15 whose elements declare an anyOf of two targets counts 30. A type over the budget is refused at registration with InvalidContractStructure (10231).

Rules at registration

A contract's declarations are checked when it is registered, and again for the whole contract on every update. A declaration that is malformed is refused by the meta-schema (JsonSchemaError, 10101) or by the parser (InvalidContractStructure, 10231). A declaration that is well formed but cannot hold is refused with the reference errors below: those are judged against the contract itself for its own document types, and against the stored contract for another contract's.

  • refersTo sits on an identifier property or on the items of a typed array of identifiers. On the array itself it is refused: the declaration belongs on its items. The one exception is the key id form of identityPublicKey, on an integer property with exactly "minimum": 0 and "maximum": 4294967295.
  • A declaration holds type and the keys its target takes, or a single anyOf or allOf.
  • A referenced documentType must exist (ReferencedDocumentTypeNotFoundError, 40121). The three document references are disjoint, each type admitting exactly one: its documents must never disappear for permanentDocument (ReferencedDocumentTypeDeletableError, 40122), disappear only through a moderator's recorded removal for moderatedDocument (ReferencedDocumentTypeNotModeratedError, 40143), and be able to disappear otherwise for deletableDocument (ReferencedDocumentTypeNotDeletableError, 40131, for a type whose documents never disappear; ReferencedDocumentTypeModeratedError, 40144, for one whose documents only moderators remove on the record). The last is refused at registration only: a deletableDocument reference promises less than such a type keeps, and a contract registered on a network before moderatedDocument existed may hold one, which stays writable. A permanentDocument with inList whose list is in the declaring contract is the exception: the parser checks its type and reports a deletable one as InvalidContractStructure (10231).
  • A reference that finds its document by the document's id (permanentDocument, moderatedDocument or deletableDocument without findBy, or with inList, whose list's document is read by its id) may not name an indexOnly type (ReferencedDocumentTypeIndexOnlyError, 40146): its documents exist only as index entries and can not be fetched by $id, so no write could check the reference. This holds for a type of the declaring contract and of another one alike. A findBy into such a type is refused by its own rules instead, since the type has no unique index.
  • Every where entry must be one that can hold (40126), every key reference must fit the document type (40125), every boundTo must name a type a key can be bound to (10231), and every findBy and list must resolve.
  • An immutable property may not hold a deletableDocument reference that a replace could not remove: one inside an object, a typed array of them, or any deletableDocument found by findBy, except one whose key a findBy function computes, which is checked on the create alone. A single deletableDocument reference by id that is itself an immutable top-level property is allowed, but only listed without a condition: a replace the condition left free could set it to another document once it was cleared. A contract reference with an owner requirement may not sit under an immutable property of a type whose documents can be transferred or traded. All refused with InvalidContractStructure (10231); see Mutability.
  • The type stays within the reference budget.
  • The earlier spellings lookup, propertyAgreement and type: "listElement", replaced by findBy, where and inList before protocol version 14 shipped, are refused: by the meta-schema on registration and update (10101), and by the parser on every parse (10231), with a message naming what replaced each.
  • On an update, every existing declaration must be unchanged (10246). A document type the update adds may declare any reference.

Errors

ErrorCodeWhen
ReferencedEntityNotFoundError40120Write: the identity, contract, token or document does not exist, findBy finds no document, or a value is not in its list.
ReferencedDocumentTypeNotFoundError40121Registration: documentType does not exist, or contractId names no contract.
ReferencedDocumentTypeDeletableError40122Registration: a permanentDocument reference, with inList or without, names a type whose documents can disappear.
ReferencedIdentityKeyNotFoundError40123Write: the identity has no key with that id, or the identity does not exist.
ReferencedIdentityKeyDisabledError40124Write: the key is disabled.
ReferencedKeyIdPropertyInvalidError40125Registration: keyIdProperty or identityProperty names a property that does not fit, $creatorId on a type that records no creator ids, or a stored key id paired with a transient identity. Write: a key id without its identity, or an identity without its key id.
ReferencedDocumentPropertyAgreementInvalidError40126Registration: a where entry names a missing, transient, object or typed array property, or two properties of different types, or $creatorId on a type that does not record it, or keys a preallocated index by a property that can hold more than 255 bytes.
ReferencedDocumentPropertyMismatchError40127Write: the document found does not meet a where entry.
ReferencedDocumentTypeNotDeletableError40131Registration: a deletableDocument reference names a type whose documents can never disappear.
ReferencedContractRequirementNotMetError40135Write: the referenced contract exists but does not meet a contractRequirements entry.
ReferencedIdentityKeyRequirementNotMetError40136Write: the key exists and is enabled but does not meet a keyRequirements entry.
ReferencedDocumentLookupInvalidError40137Registration: a findBy into another contract's document type cannot resolve, for example because no unique index of it is over exactly the properties findBy names.
ReferencedDocumentListInvalidError40138Registration: an inList list on another contract's document type does not qualify.
ReferencedDocumentTypeNotModeratedError40143Registration: a moderatedDocument reference names a type whose documents can disappear otherwise than through a moderator's recorded removal, or never disappear.
ReferencedDocumentTypeModeratedError40144Registration: a deletableDocument reference names a type whose documents disappear only through a moderator's recorded removal.
ReferencedDocumentRemovedError40145Write: a replace kept a moderatedDocument reference whose document a moderator removed, and a where entry checked again compares a property the removal record does not keep.
ReferencedDocumentTypeIndexOnlyError40146Registration: a reference that finds its document by id (no findBy, or with inList) names an indexOnly type, whose documents can not be fetched by id.

The registration errors name the declaration as <documentType>.<property>, <documentType>.<property>[] for typed array elements, <documentType>.$ownerId or <documentType>.$creatorId for the writer and creator references, and add the operand for a leaf of an expression (resignation.memberId.anyOf[1]). When a document is written, the referenced document type is looked up and its kind checked again as a safeguard (40121, 40122, 40131, 40143), but a registered contract cannot fail those checks later: contracts and document types are never removed, and neither the deletion flags nor ttl nor moderatorAbilities can change. Nor can indexOnly, so the indexOnly rule (40146) is checked at registration only. The other codes in the range (40128 to 40130, 40132 to 40134, 40139 to 40142) belong to other keywords. See Error Codes.

More forms of reference

  • findBy: a permanentDocument or deletableDocument found through the unique index of its type over exactly the properties findBy names, with the value as one part of the key. The value can be an identity (a member, an author) instead of a document id.
  • Expressions: anyOf and allOf, several targets combined in one declaration.
  • List Elements: inList on a permanentDocument, a value that must be one of the identifiers a list on another document holds.
  • Writer and Creator References: ownerRefersTo and creatorRefersTo, the same declaration applied to the document's writer or creator instead of a property.
  • References on the elements of a typed array: a declaration on items, checked for every element.

See also

Finding a Document by Its Properties (findBy)

findBy lets a document reference find its target by the values of the target's properties instead of by its id. The property's value is then one part of a key, the rest comes from the referring document or its writer, and the reference holds if the referenced document type's unique index over exactly those properties finds a document. Reach for it when the natural value is an identity (a member, an author, a recipient) and the rule is "this identity has a document of that kind": with an id reference the writer would have to find the document's id first, and the id would say nothing about who it belongs to.

WhereInside a permanentDocument or deletableDocument refersTo declaration: on an identifier property, on the items of a typed array of identifiers, as an operand of an expression, or in ownerRefersTo and creatorRefersTo. With a function, also on a string or byte array property.
ValueAn object of 1 to 10 entries { "<referenced property>": <source> }. Each key is a property of documentType, by its name there (system ones such as $ownerId included). Each source is "." (the reference's own value), "$ownerId" (the writer), a property path of the referring document type, or at most one function. The keys must be exactly the properties of one unique index of documentType, in any order. Beside a function the refersTo may also declare minimumAgeBlocks and consume. With inList, findBy is { "$id": <property> } instead.
Sinceprotocol version 14
On updateFixed, like the rest of refersTo: adding, removing or changing findBy is refused (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsReferencedEntityNotFoundError (40120) when no document is found; at registration ReferencedDocumentLookupInvalidError (40137) for a document type of another contract, InvalidContractStructure (10231) for one of the same contract. A function adds DocumentReferencePreimageInvalidError (10423) and ReferencedDocumentRequirementNotMetError (40142).

Example

The moderation charters contract's joinRequest type, which is immutable and can never be deleted, has this unique index: one join request per proposal and owner.

"indices": [
  {
    "name": "bySubmittedCharter",
    "properties": [{ "submittedCharterId": "asc" }, { "$ownerId": "asc" }],
    "unique": true
  }
]

The same contract's electedCharter lists its team in members, each of whom must have asked to join:

"members": {
  "type": "array", "minItems": 0, "maxItems": 15, "uniqueItems": true,
  "items": {
    "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
    "contentMediaType": "application/x.dash.dpp.identifier",
    "distinctFrom": "$ownerId",
    "refersTo": {
      "type": "permanentDocument",
      "documentType": "joinRequest",
      "findBy": { "submittedCharterId": "submittedCharterId", "$ownerId": "." }
    }
  },
  "position": 2
}

Every member must be the owner of a joinRequest whose submittedCharterId equals this elected charter's submittedCharterId. findBy names submittedCharterId and $ownerId, exactly the properties of bySubmittedCharter, so that index finds the join request. For each element the key is submittedCharterId read from the elected charter and $ownerId filled with the element itself ("."). The same form works on a single identifier property, where "." is the property's value.

Read as SQL, a reference is a query that must return a row. documentType is the FROM, findBy the part of the WHERE that finds the row through a unique index, where the rest of the WHERE, checked on the row found, minimumAgeBlocks an age condition and consume a delete of the row. The declaration above reads, for each element:

SELECT * FROM joinRequest
WHERE submittedCharterId = :submittedCharterId  -- findBy, from the elected charter
  AND $ownerId = :element                       -- findBy, "."

How it works

Which index finds the document

findBy does not name an index. The index is the unique index of the referenced document type whose properties are exactly the properties findBy names, in any order. It must not bucket its first property with timeRange or integerRange, and the referenced type must not be indexOnly. A type's indexes can never be added, removed or changed by a contract update, so the same findBy always resolves to the same index.

Registration refuses a findBy that names no such index, listing the unique indexes the type has. Into the joinRequest type above, { "submittedCharterId": "submittedCharterId" } leaves out $ownerId:

"joinRequest" has no unique index over exactly (submittedCharterId): findBy must name every
property of one of its unique indexes and nothing else (bySubmittedCharter (submittedCharterId, $ownerId))

A findBy naming exactly the properties of an index that is not unique is refused too, since it could find several documents. joinRequest also has a byOwner index on $ownerId alone, which is not unique, so { "$ownerId": "." } is refused:

index "byOwner" of "joinRequest" over ($ownerId) is not unique: findBy must find at most one document

Assembling the key

When the referring document is created or replaced, a key is assembled for each value (each element of a typed array), one part per entry of findBy:

  • ".": the value being checked, the property's value or the element.
  • "$ownerId": the referring document's owner, the writer.
  • a property path ("submittedCharterId", "meta.charterId"): the referring document's value at that path.
  • a function: the hash of the params it reads.

The index is queried for at most one document, billed as a document fetch. If it finds one, the reference holds, and any where entries beside findBy are checked against that document (ReferencedDocumentPropertyMismatchError, 40127). If it finds none, the write is refused with ReferencedEntityNotFoundError (40120) naming the property, or the element by its list path (members[1]); the error's target reads "found by" and the properties findBy names.

Permanent and deletable references

A permanentDocument found by findBy never dangles. Its referenced documents are never deleted, and registration makes sure the key of every one of them is fixed once written (see below), so the document a key found stays there. A replace checks it again only:

  • when the property itself changed (for a typed array, only the elements the stored list did not hold);
  • when a property findBy reads changed, and then every value, every element included;
  • when the referring value of a where entry changed, or on every replace for a where entry whose value is "$ownerId".

A key part read from "$ownerId" never triggers a check: only a type whose documents keep their writer may read it.

A deletableDocument found by findBy promises less. Once the document a key found is deleted, a new document filed later under the same key makes the reference hold again, with different content, where an id reference to a deleted document stays dead. So it means "a document with this key exists now", which is what a membership gate needs. Every replace checks it again, whether or not the replace touched it, and no immutable property may hold one. The moderation charters contract's resignationRequest uses one to accept a writer who has an addedModerator document for the charter at the time of writing, as the alternative to being an elected member (see Writer and Creator References).

findBy can also reach into another contract. This recipientId must be an identity that has a DashPay profile when the document is written:

"recipientId": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "refersTo": {
    "type": "deletableDocument",
    "contractId": "Bwr4WHCPz5rFVAD87RqTs3izo4zpzwsEdKPWUT1NS1C7",
    "documentType": "profile",
    "findBy": { "$ownerId": "." }
  },
  "position": 0
}

DashPay's profile type has a unique index on $ownerId alone, which finds the profile, and profiles can be deleted, so the reference is a deletableDocument one.

What findBy cannot do

The value of a reference found by findBy is a key part, not a document id. So such a reference cannot be the join property of a chained query or of a composite join by id, and a preallocated index is never bound through one (see Index-Only Types). It counts as one reference per value against the reference budget, like any other.

Commit and reveal

One findBy entry may hold a function: "<referenced property>": { "function": "sys.hash.sha256d", "params": [...] } says the referenced document's property holds the SHA-256 of the SHA-256 of the params' bytes, joined in order. The platform fills that property of the key with the hash to find the document, like any other entry. The document it finds is a commitment made earlier, one that stored that hash, so the document being created may exist only while a commitment to values it carries exists. This is how a name registration that preorders a salted hash works, written as a declaration on the salt:

"preorderSalt": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 4,
  "refersTo": {
    "type": "deletableDocument",
    "documentType": "preorder",
    "findBy": {
      "saltedDomainHash": {
        "function": "sys.hash.sha256d",
        "params": ["preorderSalt", "normalizedLabel", { "const": "." }, "parentDomainName"]
      }
    },
    "where": { "$ownerId": "$ownerId" },
    "minimumAgeBlocks": 1,
    "consume": true
  }
}

The salt must reveal a preorder whose saltedDomainHash is the sha256d of the salt, the normalized label, a dot and the parent. findBy names saltedDomainHash alone, the whole of the preorder type's unique saltedHash index. The where entry makes it the writer's own preorder, minimumAgeBlocks: 1 one created in an earlier block, and consume: true deletes it with the create. As SQL:

SELECT * FROM preorder
WHERE saltedDomainHash =                                     -- findBy
      sha256d(:preorderSalt || :normalizedLabel || '.' || :parentDomainName)
  AND $ownerId = :writer                                     -- where
  AND $createdAtBlockHeight <= :createHeight - 1             -- minimumAgeBlocks
-- then the create deletes the row found                     -- consume
  • Params. A property path reads the document being created, the property carrying the reference included; { "const": text } is fixed text of 1 to 64 bytes; "." is the value carrying the reference where it has no path (each element of a typed array, the writer, the creator); 1 to 16 params. Strings count as their UTF-8, byte arrays as their bytes, identifiers as their 32 bytes. Two values of no fixed length must be separated by a one-byte const, and a value holding that byte is refused, so the joined bytes split back one way only.
  • One function. findBy holds at most one function, and the property it fills must be a byte array of exactly 32 bytes.
  • Carrier. A string or byte array property may carry a refersTo only this way, and keeps its type. The value carrying the reference fills the key exactly once, as a "." entry or a param; a param names a property by its path, never as ".". In ownerRefersTo and creatorRefersTo the value may be left out beside a function.
  • Judged once. A function's key is checked when the document is created, never on a replace, so every stored value it reads must be fixed once written, the carrier included (listed under immutable on a mutable type, and required when it is a deletableDocument reference by id, which a replace may otherwise clear once its document is deleted). The where entries are judged with it, so each referring property they name must be fixed once written too, or transient. On a type whose documents can be replaced it may not be an operand of an anyOf, which would then hold on every replace. Params may be transient: they are read from the create. The hash is billed as the double SHA-256 it is, by the blocks it hashes, beside the document fetch.

minimumAgeBlocks and consume sit on the refersTo, beside findBy: they describe the document findBy finds, and need a function in it.

  • minimumAgeBlocks: the commitment's $createdAtBlockHeight is at least that many blocks below the create's height. The commitment type must list $createdAtBlockHeight in required.
  • consume: true: the create deletes the commitment, its storage refunded to its owner. Only on a deletableDocument reference with the where entry "$ownerId": "$ownerId", into a type of the same contract whose owner may delete its documents (canBeDeleted: true) or whose documents only a consume deletes (canBeDeleted: "onlyWhenConsumed", see Deletion), that declares no delete token cost or delete action fee and no deleteConstraints (no delete transition charges or judges them), and that requires no stricter signature security level than the revealing type. A contract-bound key signing the create must be allowed to act on the consumed type too (ContractBoundedKeyOutOfBoundsError, 20014).

A create missing a param, or with a value holding its separator, is refused before any read (DocumentReferencePreimageInvalidError, 10423). No commitment is ReferencedEntityNotFoundError (40120), another identity's ReferencedDocumentPropertyMismatchError (40127), and one too young ReferencedDocumentRequirementNotMetError (40142). The Documents chapter has the full rules.

Rules at registration

On the referring side, checked for every contract by the parser (InvalidContractStructure, 10231):

  • findBy is only allowed on permanentDocument and deletableDocument references.
  • The reference's own value is read exactly once, as a "." entry or as a param of a function. Without it every value would find the same document. Only ownerRefersTo or creatorRefersTo may leave it out, beside a function.
  • Every other source is "$ownerId", a property path or a function. No other $ name is accepted, and a path may not name the reference property itself: write "." for that.
  • A property an entry reads must exist on the referring type, be required (and so must every object around it), not be transient or inside a transient object, and hold a single value, not an object or an array. findBy never runs with a missing key part, and a reader can assemble the same key from the stored document. A function's params follow their own rules: they may be transient or optional, since they are read from the create alone.
  • A "$ownerId" source needs a referring type whose documents can be neither transferred nor traded: a transfer or a purchase would move the writer part of the key without a write.
  • A deletableDocument found by findBy may not sit under an immutable property, alone or as an operand of an expression: every replace checks it again, so once its document is deleted the property would have to change. One whose key a function computes is the exception: it is checked on the create alone, and its carrier must be fixed once written.
  • { "$id": ... } is allowed only with inList, as the only entry.

On the referenced side, refused with InvalidContractStructure (10231) for a document type of the declaring contract and with ReferencedDocumentLookupInvalidError (40137) for one of another contract:

  • The keys are exactly the properties of one unique index of the type, by their names on the referenced side (system properties such as $ownerId included), in any order, and nothing else (see Which index finds the document). That index does not bucket its first property by a timeRange or an integerRange, the referenced type is not indexOnly, and no index property is transient.
  • Each source holds the same type of value as the property it fills. "." and "$ownerId" are identifiers.
  • The key stays with the document it found. Every schema property of the index must be fixed once written: the referenced type is immutable (documentsMutable: false), or the property's top-level property is listed under immutable and is no optional deletableDocument reference by id, which a replace may clear once its document is deleted. $ownerId may be a key part only where the referenced documents can be neither transferred nor traded. $updatedAt and its block height forms may be one only where they can be neither replaced, transferred nor traded, and $transferredAt and its forms only where they can be neither transferred nor traded. $id, $creatorId, $createdAt and its forms never change.

The referenced type must exist (ReferencedDocumentTypeNotFoundError, 40121). A permanentDocument found by findBy in a type whose documents can disappear is refused with ReferencedDocumentTypeDeletableError (40122), and a deletableDocument found by findBy in one whose documents cannot with ReferencedDocumentTypeNotDeletableError (40131).

Index definitions and the flags these rules read cannot change on a contract update, so a findBy that registered keeps resolving, through the same index.

The keyword lookup, which named the index and mapped its properties under keys, was replaced by findBy before protocol version 14 shipped. A declaration still carrying it is refused on every parse, with a message saying to map every property of the unique index to its source and leave the index name out.

See also

Expressions

A refersTo declaration may combine several targets instead of naming one. anyOf holds when at least one of its operands holds, and allOf when every operand holds for the same value. Reach for an expression when a value may point at one of several kinds of thing ("an elected member or an added one"), or must satisfy several references at once ("a member of the team who also has a profile"). Both combinators take the same operands, follow the same limits and are checked the same way; they differ only in when they stop.

anyOf

WhereIn place of a single target: in refersTo on an identifier property or on the items of a typed array of identifiers, in ownerRefersTo and creatorRefersTo, and as an operand of an allOf. Not on a key id property.
Value{ "anyOf": [ ... ] }, the declaration's only key: 2 to 4 distinct operands.
Sinceprotocol version 14
On updateFixed (IncompatibleDocumentTypeSchemaError, 10246), including a change of operand order.
ErrorsNone of its own: when no operand holds, the write is refused with the last operand's error, for example ReferencedEntityNotFoundError (40120).

The operands are checked in the order they are listed, and the first that holds decides: the rest are not read. When none holds, the write is refused with the error of the last one.

The moderation charters contract lets a member of a seated team resign with a resignationRequest. Its writer must be on the team, either elected or added later:

"ownerRefersTo": {
  "anyOf": [
    {
      "type": "permanentDocument",
      "documentType": "electedCharter",
      "findBy": { "$id": "electedCharterId" },
      "inList": "members"
    },
    {
      "type": "deletableDocument",
      "documentType": "addedModerator",
      "findBy": { "electedCharterId": "electedCharterId", "memberId": "." }
    }
  ]
}

The writer must be one of the members of the electedCharter this document's electedCharterId names, or have an addedModerator document for that charter that exists now. Here the expression is the writer's reference (see Writer and Creator References); the same declaration works on an identifier property, where it judges the property's value.

allOf

WhereIn place of a single target: in refersTo on an identifier property or on the items of a typed array of identifiers, in ownerRefersTo and creatorRefersTo, and as an operand of an anyOf. Not on a key id property.
Value{ "allOf": [ ... ] }, the declaration's only key: 2 to 4 distinct operands.
Sinceprotocol version 14
On updateFixed (IncompatibleDocumentTypeSchemaError, 10246), including a change of operand order.
ErrorsNone of its own: the write is refused with the first failing operand's error, for example ReferencedEntityNotFoundError (40120).

The operands are checked in the order they are listed, and the first that fails decides: the rest are not read, and the write is refused with that operand's error.

"memberId": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "refersTo": {
    "allOf": [
      {
        "type": "permanentDocument",
        "contractId": "EG7RGfV8fDTayC2FyVr8HwdpJh3fXDbVztcfE94UmN88",
        "documentType": "electedCharter",
        "findBy": { "$id": "electedCharterId" },
        "inList": "members"
      },
      {
        "type": "deletableDocument",
        "contractId": "Bwr4WHCPz5rFVAD87RqTs3izo4zpzwsEdKPWUT1NS1C7",
        "documentType": "profile",
        "findBy": { "$ownerId": "." }
      }
    ]
  },
  "position": 1
}

This property of a hypothetical moderator directory, next to an electedCharterId identifier property, must be one of the elected members of that charter in the moderation charters contract, and must have a DashPay profile when the document is written. The list check comes first, so a value that is not a member is refused without the profile being read.

Operands

An operand is a leaf or a nested expression.

Leaves. A leaf is an ordinary target declaration with its own keys, one of:

  • identity;
  • permanentDocument, by id, by findBy or with inList;
  • deletableDocument found by findBy.

The first two are existence checks against things that are never deleted (inList reads a list that never changes on a document that is never deleted), so an expression made only of them holds for good once it holds. A deletableDocument found by findBy may find nothing later, which is why an expression holding one is checked on every replace.

The other targets are refused as operands, since they do not compose with other operands:

  • deletableDocument by id: once its document is deleted a replace may clear the property, which assumes the property refers to that one target.
  • identityPublicKey, in either form: it pairs the value with a key id that no other operand reads.
  • contract: its requirements are gates judged against the block time and the writer, not an existence check, and a contract id is never also an identity or document id.
  • token: a token id is never also an identity or document id.

Nesting. An operand may be an expression of the other combinator: an allOf inside an anyOf, or an anyOf inside an allOf. An anyOf directly inside an anyOf, or an allOf inside an allOf, is refused, since it says what one flat list says.

A where, a findBy or an inList belongs to its leaf, inside it: the expression itself holds nothing but its combinator.

How it works

  • Order decides the error. A refusal is always the error a leaf declared alone would give, naming the property (or the element) as a single reference would. So the author's order decides which error a writer sees: put the most general operand of an anyOf last, and the cheapest or most telling operand of an allOf first.
  • Every read is billed. A value the second operand of an anyOf holds for also pays for the first operand's read. An allOf whose first operand fails reads nothing more.
  • where is per leaf. A where is checked only against its own leaf's document. A value whose first leaf fails its where can still be accepted through a second leaf that has none.
  • Typed arrays. On the items of a typed array, each element meets the expression on its own.
  • Replace. An expression is checked again when any of its leaves would be checked again alone (see On replace), and then it is evaluated whole, since which operands hold may have changed. An expression holding a deletableDocument found by findBy is therefore checked on every replace, except a leaf whose key a findBy function computes, which is judged on the create alone and holds on a replace without a read.
  • Budget. Every leaf counts against the reference budget, since each may be read for each value. An anyOf of two leaves on a typed array of maxItems 15 counts 30.
  • Queries. An expression cannot be the join property of a chained query or of a composite join by id, and a preallocated index is never bound through one.

Rules at registration

  • The combinator is the declaration's only key, and lists at least two operands: a single one is declared on its own. No two operands of one list may be alike; a leaf naming the declaring contract's id in contractId is the same as one leaving it out.
  • A list holds at most 4 operands, and any path from the declaration to a leaf passes through at most 4 combinators. The anyOf holding an allOf is 2 deep.
  • Every leaf is one of the admitted targets, and is checked exactly as the same target declared alone: its document type, the type's deletability, its where and its findBy or list. Every leaf must pass, since each has to be a declaration that could hold. The errors name the leaf by where it sits: refersTo anyOf[1].allOf[1] findBy: ... from the parser, resignation.memberId.anyOf[1].allOf[1] from the registration check.
  • An immutable property may not hold an expression with a deletableDocument leaf found by findBy, unless a findBy function computes that leaf's key (InvalidContractStructure, 10231).
  • On a document type whose documents can be replaced, a leaf whose key a findBy function computes may not be an operand of an anyOf: judged on the create alone, it would hold on every replace, whichever operand held on the create.
  • Every leaf counts against the reference budget of 256.

A malformed expression is refused by the meta-schema (JsonSchemaError, 10101) or the parser (InvalidContractStructure, 10231); a leaf that cannot hold, with the reference error it would get alone (see Errors). On an update, any change is refused (10246): an operand added, removed, changed or moved, anyOf swapped for allOf, or a single target turned into an expression or back.

See also

List Elements (inList)

A permanentDocument reference with inList says the value must be one of the identifiers a list on another document holds. inList names the list, and findBy: { "$id": <property> } names the document holding it, by the id another property of the referring document holds. Reach for it when membership is written down once as a list on a document that never changes, such as the members elected with a charter, and other documents must name one of them.

WhererefersTo on an identifier property, or on the items of a typed array of identifiers, where every element must be listed; an operand of an expression; or ownerRefersTo and creatorRefersTo, where the writer or the creator must be listed.
ValueinList, the path of a typed array of identifiers on documentType, in a "type": "permanentDocument" declaration with documentType (the type holding the list), findBy (exactly { "$id": "<the identifier property holding the document's id>" }), optionally where (up to 10 entries) and contractId.
Sinceprotocol version 14
On updateFixed, like the rest of refersTo (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsReferencedEntityNotFoundError (40120) when the value is not in the list or the list's document is not found; ReferencedDocumentPropertyMismatchError (40127) for a where entry; at registration ReferencedDocumentListInvalidError (40138) for a list of another contract.

Example

The moderation charters contract lets the leader of a seated team take an elected member off it with a removedModerator document:

"removedModerator": {
  "type": "object",
  "documentsMutable": false,
  "canBeDeleted": true,
  "properties": {
    "electedCharterId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "refersTo": {
        "type": "permanentDocument",
        "documentType": "electedCharter",
        "where": { "$ownerId": "$ownerId" }
      },
      "position": 0
    },
    "memberId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "distinctFrom": "$ownerId",
      "refersTo": {
        "type": "permanentDocument",
        "documentType": "electedCharter",
        "findBy": { "$id": "electedCharterId" },
        "inList": "members"
      },
      "position": 1
    }
  },
  "required": ["$createdAt", "electedCharterId", "memberId"],
  "additionalProperties": false
}

electedCharterId must name an elected charter the writer owns, so only the team's leader can write one. memberId must be one of the members of the electedCharter document whose $id this document's electedCharterId holds, so only an elected member can be removed. electedCharter is immutable and can never be deleted, so its members never change.

Read as SQL, with the list as a table of its elements:

SELECT * FROM electedCharter
WHERE $id = :electedCharterId      -- findBy
  AND :memberId IN (members)       -- inList

How it works

When the referring document is created or replaced:

  1. The document holding the list is the one whose $id equals the value of the property findBy reads. It is fetched by id, once per write: in the example the electedCharterId reference and the list reference share one fetch. The list is collected once, so checking many values against it costs one read.
  2. The where entries, if any, are checked against that document, as for any document reference (ReferencedDocumentPropertyMismatchError, 40127).
  3. The value must be in the list. On a typed array every element must be; on the writer's or creator's reference, the writer or creator must be.

A value not in the list, a list document that does not exist, or a value set while the $id property is not, refuses the write with ReferencedEntityNotFoundError (40120), naming the property, or an element by its list path (memberIds[1] for the second):

electedCharter 7kX...: members [Alice, Bob]
removedModerator { electedCharterId: 7kX..., memberId: Alice }  -> accepted
removedModerator { electedCharterId: 7kX..., memberId: Carol }  -> refused, 40120:
  referenced list element (members of the electedCharter document electedCharterId names)
  <Carol> not found for path memberId

A replace checks the reference again when its value changed, or when a property findBy or where reads changed: the $id property among them, since it may now name another document, whose list is then checked against every value. A where entry whose value is "$ownerId" is checked on every replace. When only a typed array of values changed, the elements the stored list already held are not checked again. Nothing else can make a validated value unlisted: the list's document is never deleted and its list never changes.

Rules at registration

  • inList is only allowed on a permanentDocument reference, and needs findBy.
  • findBy is exactly { "$id": "<property>" }, and { "$id": ... } is allowed only with inList. The property is an identifier property of the referring document type: not $ownerId (no document has the writer's id), not "." (the value is the list element) and not a typed array (one document holds the list). It must be stored, so it and every object around it are not transient, and a reader can tell from the stored document which list the value was checked against. It may be optional. It needs no refersTo of its own; if it has one, that must be a reference by id to documentType in the list's contract, or the value could never be in the list. Refused with InvalidContractStructure (10231) otherwise.
  • where may not compare $id again, nor the property findBy reads. Its entries follow the rules of every where (ReferencedDocumentPropertyAgreementInvalidError, 40126).
  • documentType must exist (ReferencedDocumentTypeNotFoundError, 40121), and its documents must never disappear: canBeDeleted: false, no moderatorAbilities.delete and no ttl.
  • inList is a property path (a dotted one for a nested list, never a $ name) of a stored typed array of identifiers on documentType, and the list must be fixed once a document is written: the type is immutable (documentsMutable: false), or the list's top-level property is listed under immutable without a condition. A list frozen only under a condition does not qualify: while the condition does not hold, a replace could change it.
  • The checks on documentType and inList are refused with InvalidContractStructure (10231) for a document type of the declaring contract. For one of another contract, a deletable type is refused with ReferencedDocumentTypeDeletableError (40122) and a list that does not qualify with ReferencedDocumentListInvalidError (40138).
  • Each value counts one against the reference budget, maxItems for a typed array.

The type listElement, which found the document holding the list through a propertyAgreement pair with $id on its referenced side, was replaced by inList on a permanentDocument before protocol version 14 shipped. A declaration still using it is refused on every parse, with a message saying to declare a permanentDocument with findBy: { "$id": ... } and inList.

See also

Writer and Creator References

A property's refersTo judges a value the writer chose. ownerRefersTo and creatorRefersTo judge an identity of the document instead: its writer ($ownerId) or its creator ($creatorId). Each takes the same declaration a property's refersTo takes, limited to the targets an identity's id can be. Reach for them to say who may write a document of a type: "only a member of this team", "only someone with a profile". A type declares at most one of the two: ownerRefersTo when its documents stay with their owner, creatorRefersTo when they can be transferred or traded.

Both are checked the way a property's reference is: the same targets, keys, errors and replace rules, with the identity as the value. They count as one reference each (times the leaves of an expression) against the reference budget, and the check runs before the properties' references.

ownerRefersTo

WhereThe document type, at the top level of its schema. Only on a type whose documents can be neither transferred nor traded.
ValueA refersTo declaration whose value is the writer: identity, a permanentDocument or deletableDocument found by findBy, a permanentDocument with inList, or an anyOf or allOf whose leaves are all of these.
Sinceprotocol version 14
On updateFixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsThe error the target reports for a property, at the path $ownerId: ReferencedEntityNotFoundError (40120) when findBy finds nothing or the writer is not in the list, ReferencedDocumentPropertyMismatchError (40127) for a where entry.

Example

"post": {
  "type": "object",
  "ownerRefersTo": {
    "type": "deletableDocument",
    "contractId": "Bwr4WHCPz5rFVAD87RqTs3izo4zpzwsEdKPWUT1NS1C7",
    "documentType": "profile",
    "findBy": { "$ownerId": "." }
  },
  "properties": {
    "text": { "type": "string", "minLength": 1, "maxLength": 280, "position": 0 }
  },
  "required": ["text"],
  "additionalProperties": false
}

Only an identity with a DashPay profile may create a post. In findBy, "." is the writer: DashPay's unique index on $ownerId must find a profile owned by them. Profiles can be deleted, so the target is a deletableDocument one and every replace asks again: a writer whose profile is gone can no longer edit their posts, though they can still delete them.

The moderation charters contract's resignationRequest combines two targets: its writer must be one of the elected members of the charter the request names, or have an addedModerator document for it that exists now. The declaration is shown under anyOf.

How it works

  • On create, the writer is checked against the target exactly as a property's value would be. An identity target always holds and reads nothing: the transition has already proved that the writer exists.
  • In findBy, "." is the writer, and so is a "$ownerId" source. In a where, an entry valued "$ownerId" names the same writer.
  • On replace, the declaration is checked again when a property its findBy or where reads changed, and on every replace when it has a where entry valued "$ownerId" or a deletableDocument found by findBy (alone or as a leaf), unless a findBy function computes the key, which is judged on the create alone. Otherwise nothing is read: the writer is always the owner, a permanent target is never deleted and its key never moves.
  • Transfers and purchases cannot happen on such a type, so the owner of every document is a writer that was checked.
  • A refusal names $ownerId as its path. Registration errors name the declaration <documentType>.$ownerId.

Rules at registration

  • The document type's documents can be neither transferred nor traded (transferable and tradeMode absent or 0). Otherwise a transfer or a purchase, which is not a write, would hand a document to an owner the declaration never checked; declare creatorRefersTo instead.
  • The target is one the writer's id can be. contract, token and a document by id are refused, since an identity's id is never one of those ids, and so is identityPublicKey, which pairs the value with a key id the writer does not carry. The same holds for every leaf of an expression.
  • Every other rule is a property reference's: those of findBy, the list and each where, checked against another contract's stored type where the declaration names one.

A malformed declaration is refused by the meta-schema (JsonSchemaError, 10101) or the parser (InvalidContractStructure, 10231); one that cannot hold, with the reference errors of its target (see Errors).

creatorRefersTo

WhereThe document type, at the top level of its schema. Only on a type that records creator ids: a transferable or tradeable type of a format-1 contract (see System Properties).
ValueA refersTo declaration whose value is the creator: identity, a permanentDocument found by findBy or with inList, a deletableDocument whose findBy holds a function, or an anyOf or allOf whose leaves are all of these.
Sinceprotocol version 14
On updateFixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246).
ErrorsThe error the target reports for a property, at the path $creatorId: ReferencedEntityNotFoundError (40120), ReferencedDocumentPropertyMismatchError (40127).

Example

"moderatorBadge": {
  "type": "object",
  "transferable": 1,
  "creatorRefersTo": {
    "type": "permanentDocument",
    "contractId": "EG7RGfV8fDTayC2FyVr8HwdpJh3fXDbVztcfE94UmN88",
    "documentType": "electedCharter",
    "findBy": { "$id": "electedCharterId" },
    "inList": "members"
  },
  "properties": {
    "electedCharterId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "position": 0
    }
  },
  "required": ["electedCharterId"],
  "additionalProperties": false
}

Only an elected member of the charter electedCharterId names, in the moderation charters contract, may mint a badge. Once minted, the badge can be transferred to anyone (transferable: 1), and it keeps its creator.

How it works

  • On create, the creator is the writer, and is checked as ownerRefersTo checks the writer. An identity target reads nothing.
  • On replace, the value is the stored creator, whoever writes. The declaration is checked again when a property its findBy or where reads changed, and on every replace when it has a where entry valued "$ownerId". Such an entry still names the writer, not the creator.
  • Transfers and purchases need no check: they do not change the creator.
  • In findBy, "." is the creator.
  • A refusal names $creatorId as its path. Registration errors name the declaration <documentType>.$creatorId.

Rules at registration

  • The document type records creator ids. Such a type's documents can be transferred or traded, so ownerRefersTo is refused on it, and a type declares at most one of the two.
  • The targets are those of ownerRefersTo except deletableDocument: the creator never changes, and a document a transfer handed on could not be replaced by its new owner once the document findBy found was deleted. A deletableDocument whose key a findBy function computes is the exception, since it is judged on the create alone.
  • findBy may not read "$ownerId": on a type whose documents can change owner, a transfer or a purchase would move that key part without a write.
  • It can only be declared on a type when the type is created, since an update may not add it. So every document of the type records its creator.
  • Every other rule is as for ownerRefersTo.

Choosing between them

The type's documentsDeclareJudges
stay with their owner (transferable and tradeMode absent or 0)ownerRefersTowhoever writes the document, who is always its owner
can be transferred or tradedcreatorRefersTothe identity that created the document, whoever writes it later

For a rule about the writer and a document one of its own properties already names, a where entry valued "$ownerId" on that property's reference is enough: { "$ownerId": "$ownerId" } requires the writer to own the referenced document. The type-level keywords are for rules no property carries, such as "the writer has a profile" or "the writer is on this list".

See also

distinctFrom

distinctFrom says that an identifier property must hold a different identity (or document) than another identifier of the same document, or than the document's owner. Reach for it when a document names two parties that must not be the same: a delegate who is not the delegator, a buyer who is not the seller, a team member who is not the team's leader. The check reads only the document being written, so it costs no state reads.

WhereAn identifier property, at the top level or inside an object; or the items of a typed array of identifiers, where it binds every element
Value"$ownerId", or the dotted path of another identifier property of the same document type ("meta.reviewerId" for a nested one), 1 to 256 characters
DefaultAbsent: no rule
Sinceprotocol version 14
On updateFixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246)
ErrorsDocumentPropertyNotDistinctError (10419) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231)

An identifier here is a 32-byte id, declared as a byte array with the identifier contentMediaType (see Property Schemas).

Example

"delegation": {
  "type": "object",
  "properties": {
    "delegateId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "distinctFrom": "$ownerId",
      "position": 0
    },
    "backupId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "distinctFrom": "delegateId",
      "position": 1
    }
  },
  "required": ["delegateId"],
  "additionalProperties": false
}

An identity cannot delegate to itself: delegateId must differ from the document's owner. The optional backupId must differ from delegateId; a delegation without a backup passes, since there is nothing to compare.

On a typed array the keyword goes on items, and every element must differ from the named value. The moderation charters system contract uses this for a team's members, none of whom may be the leader who writes the document:

"members": {
  "type": "array",
  "minItems": 0,
  "maxItems": 15,
  "uniqueItems": true,
  "items": {
    "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
    "contentMediaType": "application/x.dash.dpp.identifier",
    "distinctFrom": "$ownerId"
  },
  "position": 2
}

How it works

  • Create and replace. After the document's properties pass the JSON schema, each declaring property is compared with what it names: the other property's value in the same document, or the writer's identity for $ownerId. Equal values refuse the transition with DocumentPropertyNotDistinctError (10419), which names the document type, the property and what it collided with. For a typed array each element is compared on its own.
  • Absent values pass. When the declaring property or the property it names is left out of the document, there is nothing to compare, and the rule holds. Add the property to required when it must be there.
  • Transfer and purchase. These change the owner and nothing else. The stored document is judged against its new owner, so a transfer to, or a purchase by, the identity held in a property that must differ from $ownerId is refused with the same error. A delegation, say, cannot be transferred to its own delegate. A price update changes neither owner nor data and is not judged.
  • Deletes are never judged.

The check reads the transition (or, for a transfer or purchase, the stored document) and never looks anything else up, and a type without declarations pays nothing for it.

Rules at registration

  • The keyword is allowed only on an identifier property or on the identifier items of a typed array. On any other property, the typed array itself included, the meta-schema refuses it (JsonSchemaError, 10101).
  • The value must be "$ownerId" or name a property of the same document type. No other system property is accepted.
  • A named property must exist, must itself be an identifier (an identifier that carries refersTo counts), must not be an object, and must not be the declaring property.

The parser refuses a value that breaks the last two rules with InvalidContractStructure (10231).

See also

maxBytes

maxBytes caps how many bytes a string may take when it is encoded as UTF-8, which is how Platform stores it. JSON Schema's maxLength counts characters, and one character takes from one to four bytes, so maxLength alone does not bound the stored size. Reach for maxBytes when the size of a document matters: to keep storage fees predictable, or to stay under the 5120 bytes any one stored value may take.

WhereA string property, at the top level or inside an object; or the items of a typed array of strings, where it bounds every element
ValueAn integer from 1 to 65535, no lower than the property's minLength
DefaultAbsent: only maxLength and the 5120-byte cap on every value apply
Sinceprotocol version 14
On updateMay be raised or removed; adding it or lowering it is refused (IncompatibleDocumentTypeSchemaError, 10246)
ErrorsDocumentPropertyMaxBytesExceededError (10421) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231)

Example

"description": {
  "type": "string",
  "minLength": 1,
  "maxLength": 4096,
  "maxBytes": 4096,
  "position": 1
}

This is the description of a proposal in the moderation charters system contract. maxLength admits 4096 characters, which could take up to 16384 bytes (more than the 5120-byte cap on a stored value); maxBytes holds the stored text to 4096 bytes, so a description in plain ASCII can use every character and one written only in four-byte characters, such as most emoji, a quarter of them.

On a typed array the keyword goes on items:

"tags": {
  "type": "array",
  "maxItems": 8,
  "items": { "type": "string", "minLength": 1, "maxLength": 32, "maxBytes": 64 },
  "position": 2
}

Each of up to 8 tags is at most 32 characters and at most 64 bytes.

How it works

  • The check runs wherever a document's properties are validated: every create and replace in consensus, and every client that validates a document before sending it.
  • It runs after the JSON schema validation. A value the schema refuses (too many characters, the wrong type) is reported with the schema's error, not this one.
  • Every string the document holds for a property that declares maxBytes is measured in UTF-8 bytes. One that is longer refuses the transition with DocumentPropertyMaxBytesExceededError (10421), which names the property and both lengths. For a typed array the error names the element, as in tags[2].
  • A property the document leaves out is not checked.

The cap bounds the value only; maxLength and minLength still apply in characters. Setting both is normal: maxLength says what a user may type, maxBytes what the platform stores.

Rules at registration

  • The keyword is allowed only on a string property or on the string items of a typed array. On any other property, the typed array itself included, the meta-schema refuses it (JsonSchemaError, 10101).
  • The value is an integer from 1 to 65535 (10101).
  • It may not be lower than minLength: a string of minLength characters takes at least that many bytes, so a lower cap would refuse every value (InvalidContractStructure, 10231).

See also

generatedFrom

generatedFrom says that the platform generates a string property's value with a system function of other properties of the same document, its params. The functions change the case of a string (lowercase, uppercase, capitalize, camelCase, snakeCase), or fold a name so that names differing only by case or by look-alike characters read the same (homographSafeASCII): reach for that one when a unique index must treat Bob, BOB and B0B as one name. A client may leave the property out, and the platform generates it when the document arrives; a client that sends it has it checked. The check reads only the document being written, so it costs no state reads.

WhereA string property, at the top level or inside an object. Not on a typed array or its items, and not beside $ref
Value{ "function": <name>, "params": [<path>, ...] }: a system function, and as many params as it takes, each the dotted path of a property of the same document type ("profile.display" for a nested one), 1 to 256 characters (ASCII, as property names are, so as many bytes)
DefaultAbsent: no rule
Sinceprotocol version 14
On updateFixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246); a property an update adds may declare it only when one of its params is new too (DocumentTypeUpdateError, 40212)
ErrorsDocumentPropertyNotGeneratedError (10424) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231); on update IncompatibleDocumentTypeSchemaError (10246) or DocumentTypeUpdateError (40212)

Example

"handle": {
  "type": "object",
  "indices": [
    { "name": "byNormalizedLabel", "properties": [{ "normalizedLabel": "asc" }], "unique": true }
  ],
  "properties": {
    "label": {
      "type": "string", "pattern": "^[a-zA-Z0-9-]{3,63}$", "maxLength": 63,
      "position": 0
    },
    "normalizedLabel": {
      "type": "string", "maxLength": 63,
      "generatedFrom": {
        "function": "sys.stringTransformations.homographSafeASCII",
        "params": ["label"]
      },
      "position": 1
    }
  },
  "required": ["label", "normalizedLabel"],
  "additionalProperties": false
}

A client creates { "label": "Bob" } and the stored document holds { "label": "Bob", "normalizedLabel": "b0b" }. A second handle { "label": "B0B" } generates the same b0b and is refused by the unique index. A client that sends { "label": "Bob", "normalizedLabel": "b0b" } gets the same result; one that sends "normalizedLabel": "bob" is refused with DocumentPropertyNotGeneratedError.

The generated property needs no pattern of its own. Every value it can hold is what the function generates from params that passed their own patterns: here label admits ASCII letters, digits and -, so normalizedLabel can only ever hold a to z without i, l and o, digits and -. It keeps maxLength because it is indexed, and an indexed string declares one of at most 63.

This is the rule the DPNS domain type's data trigger checks today for normalizedLabel and normalizedParentDomainName, written into the schema.

Functions

Functions are system functions, built into the platform and named under sys.; any other name is refused at registration. The prefix keeps them apart from functions a contract may bring in a later protocol version. They are grouped in namespaces, and each one declares how many params it takes. The sys.stringTransformations functions each take one string:

FunctionReturnsExample
sys.stringTransformations.lowercaseThe string with A to Z lowercasedHello World to hello world
sys.stringTransformations.uppercaseThe string with a to z uppercasedHello World to HELLO WORLD
sys.stringTransformations.capitalizeThe first character uppercased and every other lowercasedhELLO wORLD to Hello world
sys.stringTransformations.camelCaseThe words joined, the first lowercased and every later one capitalizedHello World, hello_world and HelloWorld to helloWorld
sys.stringTransformations.snakeCaseThe words lowercased and joined with _Hello World, helloWorld and hello-world to hello_world
sys.stringTransformations.homographSafeASCIIThe string with A to Z lowercased, then o turned into 0 and i and l into 1Lil-Olive to 111-011ve

The string transformations change ASCII characters only and keep every other character as it is (Olé becomes 01é under homographSafeASCII, OLÉ becomes olÉ under lowercase). They use no Unicode tables: Unicode case mappings change between releases of the tools a node is built with, and two nodes that lowercased a character differently would disagree about a document. On ASCII, homographSafeASCII is exactly what the DPNS trigger computes.

camelCase and snakeCase split the string into words. Every ASCII character that is neither a letter nor a digit (a space, -, _, . and so on) separates words and is dropped. A word also starts at an ASCII uppercase letter that follows anything other than another ASCII uppercase letter (helloWorld is hello, World), or that follows one and is followed by an ASCII lowercase letter (XMLHttpRequest is XML, Http, Request, so it becomes xmlHttpRequest or xml_http_request). Characters outside ASCII belong to the word they are in. Every transformation gives back its own output unchanged.

lowercase, uppercase, capitalize and homographSafeASCII keep the length of the value, and camelCase never lengthens it, but snakeCase can: aBcDeF becomes a_bc_de_f. Give a generated property a maxLength that admits what its function can generate from its params' longest value.

A function refuses nothing. Which characters a value may hold is the job of each param's pattern: homographSafeASCII resists look-alike names only where that pattern admits ASCII alone, as DPNS's does. A contract that admits other scripts can hold two names that look alike but generate different values.

Params

For now a param is always a property of the same document, written as its dotted path. The form leaves room to add, in a later protocol version, literals ({ "const": ... }), system values such as "$ownerId" and nested calls, without changing what parses today.

How it works

  • Generated on arrival. When a document create or replace, or the values of an index-only delete, leaves the property out and supplies every param, the platform writes the generated value into the document before anything reads it: contest detection, the schema validation, the indexes and the stored document all see it. A property the document sends is left as sent. A generated value then goes through the property's own schema like a sent one, so any bound it declares, such as maxLength, should admit every value the function can generate from its params.
  • Checked after the JSON schema. Wherever a document's properties are validated, on every create and replace included and in a client that validates a document before sending it, the property must hold what the function generates from its params, and must be absent when a param is. A document that repeats a key in an object on the way to the property or to a param is refused too, since the value it holds there would be ambiguous. A property that breaks this refuses the transition with DocumentPropertyNotGeneratedError (10424), which names the document type, the property, the function and its params. A schema error on any of the values is reported first.
  • Absent params. A property one of whose params is absent must be absent too. To make the params required in effect, list the generated property in required: a document without them then fails the schema.
  • Replace. A replace is judged on the whole new document. Leave the property out to have it generated from the new params; a stale value sent with changed params is refused.
  • Transfers, purchases and deletes by id do not change the data and are not judged. An index-only delete is: its values are generated and checked like a create's, so a stale value, or one without its params, refuses it.

The SDK's transition builders generate the property from the document's params, replacing any value the document holds and leaving it out when a param is absent, so a transition built from a document carries the value the platform would generate, and contest detection sees it. A document fetched, edited and sent back through them therefore carries the value of its new params, not the stale one. The property-constraint pre-checks of the JavaScript and FFI SDKs judge the document the same way. A client that validates a document it built, before building a transition, should generate it first (in Rust, DocumentTypeBasicMethods::regenerate_generated_properties) or set the value; otherwise the local check reports the property missing. The proof a client verifies after a create or replace is checked against the document with the generated value, as the platform stored it.

Rules at registration

  • The keyword is allowed only on a string property, and not beside $ref, whose definition replaces every keyword written next to it (declare it in the definition instead). On any other property, a typed array and its items included, the meta-schema refuses it (JsonSchemaError, 10101).
  • function must name a system function, and params must list 1 to 16 params, as many as the function takes. The meta-schema refuses an unknown function or an empty or overlong list (JsonSchemaError, 10101).
  • Every param must name another string property of the same document type (not an object, not a system property, not the declaring property), and that property may not be generated itself.
  • Neither the declaring property nor a param may be transient or sit inside a transient object: a transient value is never stored.
  • Every param must sit inside every object that holds the declaring property: a top-level property may take any param, but profile.normalized must take params inside profile. A document that supplies the params then always holds the object the platform writes the value into.
  • On a contract update, a new property may declare generatedFrom only when one of its params is new too. Documents stored before the update were never generated, so a new generated property whose params all existed is refused (DocumentTypeUpdateError, 40212).

The meta-schema refuses the shape errors of the first two rules with JsonSchemaError (10101). The parser refuses a function with the wrong number of params, and a declaration that breaks the param rules (the third to fifth), with InvalidContractStructure (10231). The update rule refuses with DocumentTypeUpdateError (40212).

See also

encryptedFor

encryptedFor marks a byte array property as ciphertext that one identity can read, and writes the recipe into the contract: who the message is for, which identity keys were used, and which encryption scheme made the bytes. Wallets and SDKs read the recipe from the contract instead of from per-app documentation. Reach for it when a document carries a private message, a private note to self, or any other value only its recipient should read. Consensus cannot see inside the ciphertext: it checks only that the bytes have the length the scheme produces.

WhereA byte array property (byteArray: true) that is not an identifier, at the top level or inside an object. Not on the elements of a typed array
ValueAn object with exactly four keys, all required: recipient, recipientKey, senderKey, scheme (below)
DefaultAbsent: the property is plain bytes
Sinceprotocol version 14
On updateFixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246). Documents already written could not be read under another recipe
ErrorsInvalidEncryptedPropertyShapeError (10420) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231)

The four keys:

KeyValue
recipientThe dotted path of an identifier property of the same document type, whose value is the recipient identity's id; or "$ownerId" for a message the writer encrypts to themself
recipientKeyThe dotted path of an integer property of the same document type that carries the id of the recipient's identity key. Its schema must declare minimum of at least 0 and maximum of at most 4294967295
senderKeyThe same for the sender's identity key, a key of the document's owner ($ownerId)
scheme"ecdh-secp256k1-aes256-cbc", the only scheme today

The three paths are 1 to 256 characters each.

Example

"directMessage": {
  "type": "object",
  "documentsMutable": false,
  "properties": {
    "recipientId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "position": 0
    },
    "recipientKeyId": { "type": "integer", "minimum": 0, "maximum": 4294967295, "position": 1 },
    "senderKeyId": { "type": "integer", "minimum": 0, "maximum": 4294967295, "position": 2 },
    "encryptedMessage": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 1040,
      "encryptedFor": {
        "recipient": "recipientId",
        "recipientKey": "recipientKeyId",
        "senderKey": "senderKeyId",
        "scheme": "ecdh-secp256k1-aes256-cbc"
      },
      "position": 3
    }
  },
  "required": ["recipientId", "recipientKeyId", "senderKeyId", "encryptedMessage"],
  "additionalProperties": false
}

A message is written by its owner to the identity in recipientId. The owner used their key senderKeyId and the recipient's key recipientKeyId, and the ciphertext is 32 to 1040 bytes, room for a plaintext of up to 1023 bytes.

The scheme

ecdh-secp256k1-aes256-cbc is the scheme the DashPay contact request already uses for its encrypted fields:

  1. The shared key is the secp256k1 ECDH of the sender's private key and the recipient's public key: the SHA-256 of the product point's parity byte and x coordinate. The recipient derives the same 32 bytes from their own private key and the sender's public key.
  2. The writer draws a random 16-byte IV.
  3. The stored value is the IV followed by the plaintext encrypted with AES-256-CBC under the shared key and that IV, with PKCS7 padding.

A ciphertext is therefore 16 + 16 * ceil((plaintext length + 1) / 16) bytes: at least 32, and always a multiple of 16. There is no authentication tag, so a reader with the wrong key usually fails the padding check, but about once in 256 attempts gets garbage instead. Readers should treat a value that does not decrypt as a bad message, not as a protocol error.

The SDKs do this from the declaration. The Rust SDK's dash_sdk::platform::encrypted_for module has encrypt_property (which also fills in both key id properties) and decrypt_property; the JavaScript SDK has sdk.encryptedFor.encrypt, decrypt and envelope.

How it works

  • Create and replace. After the JSON schema validation, each property that declares encryptedFor and is present in the transition is checked for its shape: at least 32 bytes (the IV and one block) and a multiple of 16. A value that is not refuses the transition with InvalidEncryptedPropertyShapeError (10420), which names the property, the scheme and the lengths. The schema's own minItems and maxItems are checked first, so a value outside them gets the schema's error instead.
  • Nothing else is checkable on chain. Consensus does not know whether the bytes decrypt, whether the key ids exist on the identities, or whether those keys have an encryption purpose. A writer can store any 32 bytes.
  • Checking the keys. To have consensus check that the keys exist and are of the right kind, add references of type identityPublicKey next to the declaration. The moderation charters system contract does this: the recipient's key must be a decryption key and the sender's an encryption key.
"recipientId": {
  "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
  "contentMediaType": "application/x.dash.dpp.identifier",
  "refersTo": {
    "type": "identityPublicKey",
    "keyIdProperty": "recipientKeyId",
    "keyRequirements": { "purpose": "decryption" }
  },
  "position": 0
},
"senderKeyId": {
  "type": "integer", "minimum": 0, "maximum": 4294967295,
  "refersTo": {
    "type": "identityPublicKey",
    "identityProperty": "$ownerId",
    "keyRequirements": { "purpose": "encryption" }
  },
  "position": 2
}

encryptedFor neither requires nor duplicates these references: it describes the recipe, and the references hold the keys to it.

Rules at registration

  • The keyword is allowed only on a byte array that is not an identifier. On an identifier or any other property the meta-schema refuses it (JsonSchemaError, 10101), and so it does an unknown key, a missing key or another scheme.
  • recipient must name an identifier property of the document type (an identifier that carries refersTo counts) or be "$ownerId". No other $ name is accepted.
  • recipientKey and senderKey must name integer properties of the document type whose schemas declare minimum of at least 0 and maximum of at most 4294967295, the range of a key id. System properties are refused.
  • None of the three named properties may be transient or sit inside a transient object: a transient value is never stored, so a stored ciphertext would lose its recipe.
  • The byte array's maxItems must be at least 32, the shortest ciphertext the scheme produces.

A registration refusal from the parser is InvalidContractStructure (10231).

See also

propertyConstraints

propertyConstraints holds named rules that every created or replaced document of a type must meet. JSON Schema bounds one property at a time; these rules relate properties to each other: a deposit that covers price times quantity, percentages that add up to 100, a closed order that carries its closing time, a second party who is not the owner. Each rule is a small tree of comparisons, arithmetic and logic that consensus evaluates against the document. A rule can also read a total of other documents, how many there are or what an integer property adds up to, from the count and sum trees their indexes keep (see Totals of other documents).

WhereDocument type
ValueAn object of rules, at least one. Each key is the rule's name (1 to 64 letters, digits or underscores); each value is a condition (see Conditions)
DefaultAbsent: no rules
Sinceprotocol version 14
On updateFixed: adding, removing or changing a rule is refused (IncompatibleDocumentTypeSchemaError, 10246). Stored documents were judged against the rules as they were
ErrorsDocumentPropertyConstraintViolatedError (10422) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231)

Example

"order": {
  "type": "object",
  "properties": {
    "price": { "type": "integer", "minimum": 0, "maximum": 1000000000, "position": 0 },
    "fee": { "type": "integer", "minimum": 0, "maximum": 1000000, "position": 1 },
    "quantity": { "type": "integer", "minimum": 1, "maximum": 10000, "position": 2 },
    "deposit": { "type": "integer", "minimum": 0, "position": 3 },
    "status": { "type": "string", "enum": ["open", "pending", "closed"], "position": 4 },
    "closedAt": { "type": "integer", "minimum": 0, "position": 5 }
  },
  "required": ["price", "quantity", "deposit", "status"],
  "propertyConstraints": {
    "depositCoversOrder": {
      "lessThanOrEqual": [
        { "multiply": [{ "add": ["price", "fee"] }, "quantity"] },
        "deposit"
      ]
    },
    "feeWaivedOrAtLeastTen": {
      "anyOf": [{ "equal": ["fee", 0] }, { "greaterThanOrEqual": ["fee", 10] }]
    },
    "closedNeedsClosedAt": {
      "anyOf": [{ "notEqual": ["status", { "const": "closed" }] }, { "present": "closedAt" }]
    }
  },
  "additionalProperties": false
}

depositCoversOrder reads (price + fee) * quantity <= deposit. feeWaivedOrAtLeastTen reads fee == 0 || fee >= 10; an order that leaves fee out passes, since a missing integer reads as 0. closedNeedsClosedAt says an order whose status is closed carries a closedAt.

How it works

  • Create and replace. The rules run after the JSON schema validation of the document's properties (and after maxBytes), so every value a rule reads has passed its property's schema. A replace is judged on the whole new document, not only on what changed.
  • Name order, first failure. Rules are checked in the order of their names, and the first rule the document breaks refuses the transition with DocumentPropertyConstraintViolatedError (10422). The error names the document type, the rule, and why it failed (below).
  • Transfer and purchase. These change only the owner and the transfer's time and heights. Rules that read $ownerId, $transferredAt… or a total that depends on the owner are judged again, against the stored document with its new owner and transfer values; other rules are not, since nothing they read changed. A transfer or purchase that would break such a rule is refused with 10422.
  • Price updates change only the update's time and heights, so the rules that read $updatedAt… are judged again the same way; other rules are not.
  • Moderator restores. Every rule of the restored type is judged on the retained document, with its original owner, times and heights. Totals include the restored document as an insertion into the live trees; its removal record is not counted. Another write while it was deleted may make a cap, lower bound or cross-type prerequisite fail. Uniqueness is checked first. A refusal in a block charges the moderator and consumes its nonce, while the document stays deleted and its removal record stays unrestored. Mempool admission refuses the same invalid restore without persisting fees or a nonce change. Restore does not revalidate the full property schema or judge deleteConstraints.
  • Immutable properties. The same grammar is the condition of an immutable entry, which freezes a property while it holds. Only such a condition may read the stored document, through $old.<path>; a rule judges creates too, which have none.
  • Deletes are not judged, with one exception: a delete of an index-only document carries the row's values, which are validated like a create's, rules included. The delete carries neither the owner nor any time or height, which is why an index-only type may not have a rule reading $ownerId or a system time or height. What may be deleted is the job of deleteConstraints, rules in this grammar the stored document must meet for its owner to delete it.
  • State and fees. A rule reads the document, its owner and its times and heights, and a countOf or sumOf reads a total from state. Each such total is a state read billed with the write; nothing else a rule does adds a fee, and it changes nothing stored. The limits below bound its cost. SDKs that validate a document before sending it apply the same rules, except those reading a total, which they cannot read.

Why a rule fails, as the error reports it:

ReasonWhen
does not holdThe rule evaluates without a fault and comes out false
overflowA value it reads, or a result it computes on the way, does not fit a 128-bit signed integer
division by zeroA divide or modulo whose divisor evaluates to 0
negative exponentA power whose exponent evaluates to a negative number
not an integerA value it reads for an integer property is a float with no fractional part, such as 5.0, which the schema's integer type admits but an integer property cannot store

Conditions

A rule is a condition: a JSON object with exactly one key.

ConditionFormHolds when
equal, notEqual[left, right]The two sides are equal, or differ. The sides are two integer expressions, or a string property and a string constant or another string property, or an identifier property and an identifier constant, another identifier property or $ownerId
lessThan, lessThanOrEqual, greaterThan, greaterThanOrEqual[left, right]The left integer expression compares with the right one this way. Integers only
in[expression, [v1, v2, ...]]The expression takes one of the listed values: two or more, no two alike, all integers or all strings. With strings, the expression is a string property, or an identifier property or $ownerId with the strings as base58 identifiers
notIn[expression, [v1, v2, ...]]The expression takes none of the listed values: an in negated, listed the same way, in as many nodes. A string or identifier property the document leaves out takes none
startsWith, endsWith[text, affix]The first string starts, or ends, with the second, byte for byte with no case folding. Each side is a string constant, a string property or an ifAbsent string default, at least one a property and never the same one twice. A string property left out without a default takes no string, and the condition does not hold for it
contains["path", value]The typed array property at the path holds an element equal to the value: an integer expression among integers; a string constant, a string property or an ifAbsent string default among strings; an identifier constant, an identifier property or $ownerId among identifiers. An array the document leaves out holds nothing, and a string or identifier property it leaves out is among no elements
present"path"The document holds the property, with a value other than null and, for an object, with at least one member present
absent"path"The document leaves the property out, sets it to null, or gives an object no member that is present
anyOf[c1, c2, ...]At least one of two or more conditions holds
allOf[c1, c2, ...]Every one of two or more conditions holds
notconditionIts one condition does not hold
ifThen[if, then]If the first condition holds, the second must. The second is evaluated only when the first holds, and a fault in either breaks the rule. The two may not be alike
ifThenElse[if, then, else]If the first condition holds, the second must; if not, the third must. Only the branch the first selects is evaluated. No two of the three may be alike

Conditions nest: { "not": { "allOf": [{ "equal": ["price", 0] }, { "greaterThan": ["quantity", 10] }] } } refuses a free order of more than 10. An anyOf or allOf may not list the same condition twice, nor hold one of its own kind directly (it says what one flat list says), and a not may not hold a not or a notIn directly.

An in says what an anyOf of equal comparisons says, in far fewer nodes: { "in": ["fee", [0, 10, 25, 50]] } is 6 nodes where the anyOf is 13.

startsWith and endsWith test a string's ends: { "startsWith": ["url", { "const": "https://" }] } holds a link to https, { "endsWith": ["url", { "const": ".dash" }] } to a domain, and { "startsWith": ["path", "parentPath"] } holds a reply's path under its parent's. A constant tested against a property that declares an enum must start or end one of its values.

A contains looks the other way round, for one value among an array's elements:

  • { "not": { "contains": ["labels", { "const": "used" }] } } refuses a "used" label;
  • { "contains": ["participants", "$ownerId"] } holds the owner to the participants, and since it reads $ownerId, a transfer or purchase to someone else is refused;
  • { "contains": ["tiers", "quantity"] } holds the quantity to one of the tiers the document lists.

The kind of the array's elements decides what the value is: a { "const": "sale" } is a string among strings and a base58 identifier among identifiers.

Expressions

An integer expression is one of:

ExpressionFormValue
integer100Itself. A number written 100.0 reads as 100
path"price", "meta.total"The value of an integer or boolean property of the document type, 1 for true and 0 for false. A property the document leaves out, or sets to null, reads as 0
ifAbsent{ "ifAbsent": ["quantity", 1] }The property's value, or the given integer when the document leaves it out
add, multiply{ "add": [a, b, ...] }The sum or product of two or more operands
subtract{ "subtract": [a, b] }a - b
divide{ "divide": [a, b] }The Euclidean quotient of a by b
modulo{ "modulo": [a, b] }The Euclidean remainder of a by b, never negative
power{ "power": [a, b] }a to the power b
min, max{ "max": [a, b, ...] }The least or greatest of two or more operands, every one evaluated
abs{ "abs": a }The absolute value of its one operand
length, byteLength{ "length": "title" }The characters (as maxLength counts them) or UTF-8 bytes (as maxBytes counts them) of a string property, 0 when the document leaves it out
count{ "count": "tags" }The items of an array property, or the bytes of a byte array property, 0 when the document leaves it out
countPresent{ "countPresent": ["email", "phone"] }How many of two or more properties, no two alike, the document holds, each as present tests it (see How many of a group)
system time or height"$createdAt", "$updatedAtBlockHeight"A time or height the document records (see Times and heights)
countOf, sumOf{ "countOf": ["listing", { "$ownerId": "$ownerId" }] }A total of documents of a type of the same contract, read from state (see Totals of other documents)

Where maxLength, maxBytes and maxItems bound one property by a fixed number, a size can be compared with another property or bounded only under a condition: { "lessThanOrEqual": [{ "count": "tags" }, "maxTags"] } holds a list to its own limit. A size never breaks a rule by itself: a property left out or null has size 0, and so would a value of another type, which the schema validation refuses first.

Two more forms appear only in string and identifier comparisons, never inside arithmetic:

FormMeaning
{ "const": "closed" }A string constant, or, compared with an identifier property or $ownerId, a base58 identifier
{ "ifAbsent": ["status", "open"] }A string property, read as the given string when the document leaves it out

A bare JSON string is always a path and a bare JSON number always a value, so a constant string needs { "const": ... }. The values an in lists are literals and need no wrapper. A path is a property name, or names joined by dots for a nested property ("rewardSplit.leader"); the only $ names a rule accepts are $ownerId, the times and heights below, and, as the value a total's filter matches by, $id.

A number property (a float) cannot be read by a rule, which keeps every result exact.

Strings

A string property is compared for equality only, never ordered and never used in arithmetic:

  • { "equal": ["status", { "const": "closed" }] } or notEqual, with the constant on either side;
  • { "notEqual": ["fromCurrency", "toCurrency"] }, two bare paths that both name string properties, which compares their strings;
  • { "in": ["status", ["open", "pending"]] }, whose values are two or more distinct strings.

A string property the document leaves out equals no constant and no other string property, not even one also left out. So notEqual holds for it, and equal and in do not. { "ifAbsent": ["status", "open"] } gives it a default instead: it may stand wherever the bare path stands, and the property then reads as that string when it is left out. present and absent test it directly.

When the property declares an enum, every constant compared with it, and every ifAbsent default given to it, must be one of the enum's values. A misspelled constant is refused at registration instead of making the rule quietly never hold.

Identifiers and $ownerId

An identifier property compares in the same three ways: { "equal": ["paymentToken", { "const": "<base58>" }] } or notEqual, { "notEqual": ["buyerId", "sellerId"] }, and { "in": ["paymentToken", ["<base58>", "<base58>"]] }. Constants are base58 identifiers of 32 bytes, checked at registration and compared by their bytes, whatever form the document gives the identifier in. An identifier property the document leaves out equals no identifier, not even another one left out. Identifiers take no ifAbsent default and are never ordered.

$ownerId, the document's owner, is an identifier operand too:

  • { "equal": ["authorId", "$ownerId"] } holds the authorId property to the owner.
  • { "in": ["$ownerId", ["<base58>", "<base58>"]] } lets only the listed identities own a document of the type.

It is not a property: present, absent and integer expressions refuse it, and comparing it with itself is refused. On create and replace it is the writer. A transfer or purchase is judged with the new owner, as described in How it works. An index-only type may not declare a rule that reads it.

Times and heights

A rule can read when the document was created, last updated and last transferred, as an integer:

block time (ms)Platform block heightCore block height
creation$createdAt$createdAtBlockHeight$createdAtCoreBlockHeight
last update: a create, a replace or a price update$updatedAt$updatedAtBlockHeight$updatedAtCoreBlockHeight
last transfer: a create, a transfer or a purchase$transferredAt$transferredAtBlockHeight$transferredAtCoreBlockHeight
  • { "lessThanOrEqual": [{ "subtract": ["endsAt", "$createdAt"] }, 604800000] } keeps a listing to a week from its creation.
  • { "lessThanOrEqual": ["$updatedAt", "endsAt"] } refuses a replace or a price update after the listing ends.
  • { "lessThanOrEqual": ["$transferredAt", "endsAt"] } refuses a transfer or a purchase after it ends.

A rule may read one only when the document type records it by listing it in required, so every stored document holds it. None takes an ifAbsent default, present and absent refuse them, and an index-only type reads none. Each write is judged with the values the stored document ends up with: a create with its block's time and heights for all three events; a replace with the stored creation and transfer values and its block's as the update; a price update with its block's as the update; a transfer or a purchase with its block's as the transfer.

SDK pre-checks run before the block exists: they use the device clock for the times a write records, and do not judge a rule reading a block height, which is unknown until the block.

Totals of other documents

countOf and sumOf read a total from state: how many documents of a type of the same contract match a filter, or what one of their integer properties adds up to. The total is the one a count or sum tree keeps (Count Trees, Sum Trees), so reading it costs about the same however many documents match.

FormValueThe counted type needs
{ "countOf": ["listing"] }How many listing documents there aredocumentsCountable
{ "countOf": ["listing", { "$ownerId": "$ownerId" }] }How many of them match the filterA countable index whose properties are exactly the filter's keys
{ "sumOf": ["pledge", "amount"] }The total amount over every pledgedocumentsSummable: "amount"
{ "sumOf": ["pledge", "amount", { "campaignId": "campaignRef" }] }The total over those matching the filterAn index with summable: "amount" whose properties are exactly the filter's keys

A filter maps each key, a property of the counted type or $ownerId, to the value it must take, read from the document being written: one of its properties ("campaignRef"), $ownerId, $id (its own id, for an identifier key), an integer, or a { "const": ... } string or base58 identifier. The counted type may be the rule's own. $id counts the documents pointing at this one: { "countOf": ["vote", { "pollId": "$id" }] } is how many votes name the poll being written.

"propertyConstraints": {
  "atMostTenListings": {
    "lessThanOrEqual": [{ "countOf": ["listing", { "$ownerId": "$ownerId" }] }, 10]
  },
  "pledgesWithinGoal": {
    "lessThanOrEqual": [{ "sumOf": ["pledge", "amount", { "campaignId": "campaignRef" }] }, "goal"]
  }
}
  • As it will be after the write. The total is the stored one with the write applied. When the counted type is the rule's own, a create or moderator restore adds the document, a replace swaps its stored version for the new one, and a transfer or purchase moves it to its new owner. So atMostTenListings, declared on listing, keeps every owner at ten or fewer, and a replace of one of ten is allowed.
  • Judged when the rule's own type is written. A rule is never judged on writes of the type it counts. On its own type it holds for good, since every write that could raise the total is judged; a type with a contested index cannot total its own documents, since a document a contest awards is stored without any rule judged. On another type it is only checked when its own type is written, and can go stale later: deleting a profile does not undo a post that needed one. Deletes are not judged, so a lower bound can be broken by deleting documents, unless the counted type's deleteConstraints hold it.
  • Transfers, purchases and price updates. A total that depends on the owner (a filter value of $ownerId, or a $ownerId key on the rule's own type) is read again for a transfer or purchase, the document counted toward its new owner. A rule a price update judges, one reading $updatedAt…, reads its totals too.
  • Billed. Each total is a state read billed with the write. A total two rules read alike is read once.
  • Every earlier write counts. A document batch carries one transition, and each state transition of a block is applied before the next is validated, so a total includes every write before it.
  • SDK pre-checks cannot read state, so they do not judge a rule reading a total.

How many of a group

countPresent counts the properties of a group that the document holds, each as present tests it, so a rule compares the count with a number. A seller is reached by exactly one of three contacts:

"propertyConstraints": {
  "oneContact": { "equal": [{ "countPresent": ["email", "phone", "handle"] }, 1] }
}

The comparisons and in give every bound:

WantRule
exactly one{ "equal": [{ "countPresent": [...] }, 1] }
at most one{ "lessThanOrEqual": [{ "countPresent": [...] }, 1] }
at least two{ "greaterThanOrEqual": [{ "countPresent": [...] }, 2] }
one or two{ "in": [{ "countPresent": [...] }, [1, 2]] }
none or all three{ "in": [{ "countPresent": ["a", "b", "c"] }, [0, 3]] }

The count runs from 0 to the size of the group, so an in lists a range in a few values. oneContact is 6 nodes (the comparison, countPresent, three paths and 1); the same rule written with present alone needs an anyOf and a not of every pair, 17 nodes for three properties and more than 32 for five. Inside ifThen the bound applies only when another condition holds: { "ifThen": [{ "equal": ["kind", { "const": "shop" }] }, { "greaterThanOrEqual": [{ "countPresent": ["email", "phone", "handle"] }, 1] }] } asks a shop for at least one contact.

Evaluation order and short-circuiting

Conditions are checked in declared order and no further than the outcome needs. A comparison evaluates its left side, then its right. anyOf stops at the first condition that holds, allOf at the first that fails. Operands are evaluated left to right.

A fault (an overflow, a division by zero, a negative exponent, a value that is not an integer) in a condition that is evaluated breaks the rule, whatever the other conditions would say, and not does not turn a fault into a pass. String comparisons, present and absent never fault. So an earlier condition can guard a later one:

{ "anyOf": [{ "equal": ["b", 0] }, { "equal": [{ "divide": ["a", "b"] }, 2] }] }

holds for a b of 0 without dividing by it. The same two conditions the other way round divide by zero and break the rule.

Arithmetic

  • Integers are exact over 128-bit signed integers. Every intermediate result must fit, and one that does not breaks the rule instead of wrapping. add and multiply fold their operands from the left, so an overflow on the way is a fault even when a later operand would bring the total back in range.
  • divide and modulo are Euclidean: the remainder is never negative, and the quotient is the one that goes with it. -7 divided by 2 is -4, remainder 1. For operands that are not negative this is ordinary integer division.
  • A divisor that evaluates to 0 breaks the rule. A negative exponent breaks it too, since it has no integer result. 0 to the power 0 is 1.
  • There are no floats.

Rules at registration

The meta-schema checks the shape (JsonSchemaError, 10101):

  • the keyword is an object of one or more rules, named with 1 to 64 letters, digits or underscores;
  • every condition and every operator object has exactly one key;
  • a comparison, subtract, divide, modulo and power take exactly two operands; add and multiply two or more; anyOf and allOf two or more conditions, no two alike; an in two or more distinct values, all integers or all strings; a countPresent two or more distinct paths;
  • no anyOf or allOf holds its own kind directly, and no not holds a not or a notIn;
  • a path matches $ownerId, one of the nine times and heights, or dotted names of 1 to 64 letters, digits or underscores, so $revision and other system properties are refused;
  • a countOf lists a type name and optionally a filter, and a sumOf a type name, a property and optionally a filter; a filter has one or more keys, each $ownerId or a dotted path, and each value is a path, $ownerId, $id, an integer or a { "const": ... } string.

The parser then checks the rules against the document type (InvalidContractStructure, 10231):

  • every path an integer expression reads names an integer or boolean property; every path length or byteLength measures names a string property, and every path count counts an array or byte array property; every path a contains looks in names a typed array property whose elements are integers, strings or identifiers, of the kind of the value looked for (a string constant among them in the elements' enum when they declare one); every path compared with a string, or tested by startsWith or endsWith, names a string property, and a constant tested against one with an enum starts or ends one of its values; every path compared with an identifier names an identifier property; every path present, absent or countPresent tests names a property of any type, an object included;
  • no rule reads a property that is transient or inside a transient object, since a stored document could never be held to it;
  • every comparison and in reads at least one property: a comparison of constants would hold for every document or for none;
  • strings and identifiers are compared only with equal, notEqual and in; a string is never compared with an identifier; a property is never compared with itself;
  • string constants and ifAbsent defaults are in the property's enum when it has one; identifier constants are base58 identifiers of 32 bytes;
  • no literal divisor is 0 and no literal exponent is negative;
  • every time or height a rule reads is one the type lists in required, and takes no ifAbsent default;
  • present, absent and countPresent do not name $ownerId or a time or height, and an index-only type has no rule reading any of them;
  • no anyOf or allOf lists two conditions that parse alike, such as 1 and 1.0, or two in conditions listing the same values in another order, and no ifThen or ifThenElse holds two alike conditions;
  • no condition or operand nests more than 64 levels deep;
  • once every document type of the contract is parsed, every countOf and sumOf counts a type of the contract that is not index-only, and not its own type when that has a contested index, with a tree that keeps the total as set out in Totals of other documents. A unique, contested, ranked, time-range, integer-range or index-only-terminal index keeps no such total, nor does one with more properties than the filter has keys;
  • every key of a filter is $ownerId or an integer, string or identifier property of the counted type, and its value is of the same kind ($ownerId and $id are identifiers); a string constant is in the key's enum when it has one, and an identifier constant is base58;
  • every property a filter value reads is listed in required, with every object around it, so a write always has the value; an index-only type has no rule reading a total.

Three limits come from the protocol version 14 SystemLimits, and a rule over one is refused the same way:

  • at most 16 rules per document type (max_property_constraints);
  • at most 32 nodes per rule (max_property_constraint_nodes);
  • at most 4 distinct countOf and sumOf totals read by one document type's rules, a total read twice counting once (max_property_constraint_aggregates).

A rule within 32 nodes is never deep enough to reach the 64-level bound. Nodes are counted like this:

Part of a ruleNodes
A comparison of integers1, plus its two sides
An equal or notEqual of strings or identifiers, a startsWith or an endsWith3: the condition and its two sides
An in over integers1, plus its expression, plus 1 per value
An in over strings or identifiers2, plus 1 per value
contains2, plus the value it looks for
present, absent1
anyOf, allOf1, plus their conditions
not1, plus its condition
ifThen, ifThenElse1, plus their conditions
notInas the in it negates
An integer, a path, an ifAbsent, a size (length, byteLength, count) or a time or height1
countPresent1, plus 1 per path
add, multiply, subtract, divide, modulo, power, min, max, abs1, plus their operands
countOf, sumOf1, plus 1 per filter key

depositCoversOrder above is 7 nodes (the comparison, multiply, add and four paths), and closedNeedsClosedAt is 5. An in fits up to 30 values in 32 nodes.

Worked examples

Percentages that add up. The moderation charters system contract requires a proposal's reward split to be whole. The paths name members of the rewardSplit object, each an integer from 0 to 100:

"propertyConstraints": {
  "rewardSplitIsWhole": {
    "equal": [
      { "add": ["rewardSplit.leader", "rewardSplit.equal", "rewardSplit.actions"] },
      100
    ]
  }
}

Six nodes: the comparison, add, three paths and 100.

One of two, or both or neither. A contact card must give an email or a phone; a shipping block gives a street and a city together or not at all:

"propertyConstraints": {
  "reachable": { "anyOf": [{ "present": "email" }, { "present": "phone" }] },
  "addressComplete": {
    "anyOf": [
      { "allOf": [{ "present": "street" }, { "present": "city" }] },
      { "allOf": [{ "absent": "street" }, { "absent": "city" }] }
    ]
  }
}

present and absent work on properties of any type, strings and objects included. On an integer they are also the only way to tell "not given" from "given as 0", since a missing integer reads as 0 in an expression. An object with no member present, {} or { "inner": {} }, counts as absent: a stored document does not keep it, and a transfer, purchase or price update is judged on the stored document, so a create or replace is judged the same way.

A time window. An event ends after it starts, and lasts at most a week (startsAt and endsAt are required integer timestamps in milliseconds):

"propertyConstraints": {
  "endsAfterStart": { "lessThan": ["startsAt", "endsAt"] },
  "atMostAWeek": { "lessThanOrEqual": [{ "subtract": ["endsAt", "startsAt"] }, 604800000] }
}

A status workflow. On a ticket type, status is optional and means open when it is left out. A closed ticket names who closed it; an open or pending one does not:

"propertyConstraints": {
  "closedNamesCloser": {
    "anyOf": [{ "notEqual": ["status", { "const": "closed" }] }, { "present": "closedBy" }]
  },
  "activeHasNoCloser": {
    "anyOf": [
      { "not": { "in": [{ "ifAbsent": ["status", "open"] }, ["open", "pending"]] } },
      { "absent": "closedBy" }
    ]
  }
}

Without the ifAbsent, a ticket with no status would equal none of the listed strings, the in would not hold, and activeHasNoCloser would let it carry a closedBy. If status declares an enum, "closed", "open" and "pending" must all be in it.

Who may own a badge. On a transferable badge type, only two identities may ever hold one:

"propertyConstraints": {
  "knownHolder": {
    "in": [
      "$ownerId",
      ["HJtU46rVEkKJgevQhiVt2YhdHtDS3xtGzzJqit8en5mb", "GfRNCXeyuB3th33a6nkJKQJecKMdRzuBiPsgvSCyUTvK"]
    ]
  }
}

A create by anyone else is refused, and so is a transfer or sale of a badge to anyone else: the rule reads $ownerId, so it is judged again with the new owner.

A guarded division. The average unit price of a batch is at most 100, and a batch may be empty:

"propertyConstraints": {
  "unitPriceCapped": {
    "anyOf": [
      { "equal": ["quantity", 0] },
      { "lessThanOrEqual": [{ "divide": ["total", "quantity"] }, 100] }
    ]
  }
}

The equal comes first, so an empty batch never reaches the division. Written the other way round, an empty batch breaks the rule with a division by zero.

A flag in arithmetic. A boolean reads as 1 or 0, so a waived fee must be 0:

"propertyConstraints": {
  "waivedMeansFree": { "equal": [{ "multiply": ["waiveFee", "fee"] }, 0] }
}

See also

Token Costs (tokenCost)

tokenCost makes an action on a document cost tokens: creating a card costs 10 gems, deleting one costs 1. The tokens are taken from whoever signs the transition, and either go to the contract owner or are burned. Reach for it when an app has its own token and wants documents to be paid for with it, or wants to hand out tokens that let users act for free, with the contract owner paying the gas.

WhereDocument type
ValueAn object keyed by action: create, replace, delete, transfer, update_price, purchase. Each value is a cost object with the keys below. Actions left out cost no tokens
DefaultAbsent: no action costs tokens
Sinceprotocol version 9. gasFeesPaidBy is accepted from 9 and acted on from 14; optional is 14
On updateFixed: a cost may not be added, changed or removed on an existing document type (DocumentTypeUpdateError, 40212)
ErrorsOn a document transition: RequiredTokenPaymentInfoNotSetError (40115), IdentityHasNotAgreedToPayRequiredTokenAmountError (40116), IdentityTryingToPayWithWrongTokenError (40117), IdentityTokenAccountFrozenError (40702), IdentityDoesNotHaveEnoughTokenBalanceError (40700), GasFeesPaidByNotAllowedError (40129), InconsistentGasFeesPaidByInBatchError (40130), GasSponsorInsufficientBalanceError (40222). At registration: InvalidTokenPositionError (10451), RedundantDocumentPaidForByTokenWithContractId (10275), TokenPaymentByBurningOnlyAllowedOnInternalTokenError (10261), DataContractNotFoundError (40008), InvalidTokenPositionStateError (40009)

The keys of each cost object:

KeyValueDefaultMeaning
tokenPositioninteger, 0 to 65535requiredWhich token is charged: its position in this contract's tokens, or in the contract contractId names
amountinteger, 1 to 281474976710655requiredHow many tokens the action costs
contractIdidentifier (32 bytes)this contractThe contract whose token is charged, when it is not this one
effect0 transfer to the contract owner, 1 burn0What happens to the tokens paid
gasFeesPaidBy0 document owner, 1 contract owner, 2 prefer contract owner0Who the contract owner offers to have pay the gas of this action (acted on from protocol version 14)
optionalbooleanfalsetrue lets a transition skip the token and pay the gas in credits instead (protocol version 14)

Example

"card": {
  "type": "object",
  "documentsMutable": true,
  "canBeDeleted": true,
  "transferable": 1,
  "tradeMode": 1,
  "tokenCost": {
    "create": { "tokenPosition": 0, "amount": 10, "gasFeesPaidBy": 2, "optional": true },
    "replace": { "tokenPosition": 1, "amount": 1 },
    "delete": { "tokenPosition": 1, "amount": 1, "effect": 1 }
  },
  "properties": {
    "name": { "type": "string", "minLength": 1, "maxLength": 63, "position": 0 },
    "attack": { "type": "integer", "minimum": 0, "maximum": 100, "position": 1 }
  },
  "required": ["name", "attack"],
  "additionalProperties": false
}

The contract has two tokens. Creating a card costs 10 of token 0, which go to the contract owner. A player who pays with the token and asks for it has the gas paid by the contract owner, when the owner's balance covers it; a player may also skip the token and pay the gas in credits. Replacing a card costs 1 of token 1, also to the owner. Deleting one burns 1 of token 1. Transfers, price updates and purchases cost no tokens.

Paying: $tokenPaymentInfo

A document transition on an action with a token cost carries a $tokenPaymentInfo in its base, saying which token the signer agrees to pay with, how much at most, and who they ask to pay the gas:

"$tokenPaymentInfo": {
  "$formatVersion": "0",
  "tokenContractPosition": 0,
  "maximumTokenCost": 10,
  "gasFeesPaidBy": "PreferContractOwner"
}
FieldMeaning
paymentTokenContractIdThe contract of the token paid with. Leave it out for a token of the document's own contract: it must match the cost's contractId exactly, and a cost on the contract's own token has none
tokenContractPositionThe token's position in that contract
minimumTokenCost, maximumTokenCostOptional bounds on the amount the signer agrees to pay; a cost outside them refuses the transition
gasFeesPaidBy"DocumentOwner", "ContractOwner" or "PreferContractOwner": who the signer asks to pay the gas (see Who pays the gas)

When the transition is processed:

  1. A required cost with no $tokenPaymentInfo is refused (RequiredTokenPaymentInfoNotSetError, 40115).
  2. A payment info naming another token than the cost is refused (IdentityTryingToPayWithWrongTokenError, 40117).
  3. A cost outside the signer's minimumTokenCost and maximumTokenCost is refused (IdentityHasNotAgreedToPayRequiredTokenAmountError, 40116).
  4. The signer's gas request must be one the cost offers, and the whole batch must name one payer (40129, 40130).
  5. Against state: a signer whose account for the token is frozen is refused (IdentityTokenAccountFrozenError, 40702), and so is one whose balance is below amount (IdentityDoesNotHaveEnoughTokenBalanceError, 40700).
  6. From protocol version 14, a transparent payment that transfers or burns tokens is then refused if the token is paused (TokenIsPausedError, 40711).
  7. From protocol version 14, for a transfer to a different identity, the contract owner's account for the token is checked next. A frozen recipient is refused (40702) only when the token issuer's allowTransferToFrozenBalance is false; its default is true. For another contract's token, the issuing contract's policy applies, and the recipient remains the document contract's owner.

Every state read is charged, including on refusal. From protocol version 14, the issuer metadata and, for an external token, its contract are read only when the recipient is frozen. A transparent owner self-payment retains the payer freeze and balance checks and makes no transfer, so it performs neither the pause nor recipient checks. A payment from the shielded pool retains its separate pool validation.

The signer pays: the creator for a create, the owner for a replace, delete, transfer or price update, and the buyer for a purchase.

effect: transfer or burn

  • 0, the default, moves the tokens from the signer to the owner of the contract that holds the document type. When the contract owner performs the action themselves nothing moves, though their balance is still checked.
  • 1 burns the tokens from the signer's balance, lowering the token's supply. Only a token of the contract's own can be burned (TokenPaymentByBurningOnlyAllowedOnInternalTokenError, 10261, at registration).

Tokens of another contract: contractId

A document type may charge a token another contract defines, for example a shared currency. contractId names that contract and tokenPosition the token in it. The effect must then be 0: the tokens go to the owner of the contract holding the document type, not to the token's issuer. At registration the named contract must exist (DataContractNotFoundError, 40008) and have a token at that position (InvalidTokenPositionStateError, 40009), and it must not be the contract itself: leave contractId out for the contract's own tokens (RedundantDocumentPaidForByTokenWithContractId, 10275).

Who pays the gas: gasFeesPaidBy

From protocol version 14 the contract owner can pay the gas (the storage and processing fees) of an action paid with a token. The cost states what the contract owner offers; the transition's $tokenPaymentInfo.gasFeesPaidBy states what the signer asks for. The two resolve like this:

Cost offers / signer asksDocumentOwnerPreferContractOwnerContractOwner
0 document ownersigner payssigner paysrefused (40129)
2 prefer contract ownersigner payscontract owner, if their balance covers itrefused (40129)
1 contract ownersigner payscontract owner, if their balance covers itcontract owner
  • A signer can always pay for themself, can always state a preference, and can insist on the contract owner only where the cost offers 1. A request the cost does not cover is refused (GasFeesPaidByNotAllowedError, 40129). An action without a token cost offers 0, and a transition without $tokenPaymentInfo asks for DocumentOwner.
  • A batch has one payer. Transitions of one batch that resolve to different payers are refused (InconsistentGasFeesPaidByInBatchError, 40130); a token transition in the batch is never sponsored, so it counts as the signer paying.
  • When the contract owner's balance does not cover the gas (and any action fees they would owe), a batch that insisted is refused and charged nothing (GasSponsorInsufficientBalanceError, 40222), and a batch that only preferred falls back to the signer.
  • A transition that fails validation is never sponsored: its signer pays for the work that ran.
  • Storage refunds still go to the document's owner, whoever paid for the storage. Each token the contract owner hands out is therefore worth up to the storage fee of the largest document the type allows, so a type that offers to pay should bound its documents' size (maxLength, maxItems, maxBytes) and price the action to match.
  • A sponsor also pays any action fee the action charges.

Before protocol version 14 the key was accepted and stored but not acted on: the signer always paid.

Optional costs

With optional: true a transition may leave $tokenPaymentInfo out. It then pays no token, its signer pays the gas in credits as on an action without a token cost, and no sponsorship applies. With $tokenPaymentInfo present the token is charged exactly as for a required cost, sponsorship included, and an insufficient token balance is a rejection, never a fallback to credits: the client chooses between token and credits before signing.

Together with gasFeesPaidBy this gives a "free usage" pattern: an app hands out tokens, users act for free while their tokens last, and keep going on credits after.

Rules at registration

  • The meta-schema checks the shape (JsonSchemaError, 10101): only the six action keys; tokenPosition and amount required in each cost; values in the ranges above; no other key. Before protocol version 14 it also refuses optional.
  • Without contractId, tokenPosition must be a token of this contract (InvalidTokenPositionError, 10451).
  • With contractId: not this contract's own id (10275), no burn (10261), and a contract that exists with a token at that position (40008, 40009).

See also

Action Fees (actionFees)

actionFees charges a fixed fee in credits, on top of the gas, for an action on a document of the type. Each fee has two parts: one for the contract owner and one for the contract's moderators, each collected in a pot that its recipients claim. Reach for it when an app wants to earn from the documents written under it, or to pay the people who moderate it. The transition that pays must state the fee it agrees to, so a fee can never surprise a signer.

WhereDocument type
ValueAn object with an optional pricing, and one or more of the actions create, replace, delete, transfer, update_price, purchase, each an object with owner and/or moderators (below)
DefaultAbsent: no action charges a fee
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212): an update may not add, change or remove the fees of an existing document type, nor switch their pricing. A document type the update adds may declare its own
ErrorsOn a document transition: DocumentActionFeeAgreementNotSetError (40132), DocumentActionFeeAgreementMismatchError (40133), DocumentActionFeeMultiplierNotToleratedError (40134), DocumentActionFeeModeratorsShareMismatchError (40139). At registration: DocumentActionFeesWithoutModerationError (10902), JsonSchemaError (10101), InvalidContractStructure (10231)

The keys:

KeyValueDefaultMeaning
pricing"feeMultiplier" or "fixed""feeMultiplier"Whether the amounts follow the network's fee multiplier or are charged as written
<action>.ownercredits, 0 to 92233720368547758070Added to the contract's owner pot, which the contract owner claims
<action>.moderatorscredits, 0 to 92233720368547758070Added to the contract's moderators pot, which the moderation team shares. Needs moderation in the contract config

Amounts are in credits: 1 Dash is 100,000,000,000 credits (1000 credits per duff).

Example

"post": {
  "type": "object",
  "actionFees": {
    "pricing": "feeMultiplier",
    "create": { "moderators": 100000000, "owner": 10000000 }
  },
  "properties": {
    "text": { "type": "string", "minLength": 1, "maxLength": 280, "maxBytes": 560, "position": 0 }
  },
  "required": ["text"],
  "additionalProperties": false
}

In a contract that declares moderation, creating a post costs an extra 0.001 Dash for the moderation team and 0.0001 Dash for the contract owner, at a fee multiplier of 1. Replacing, deleting and every other action cost only their gas.

How it works

  • What is charged. With fixed pricing, the declared amounts. With feeMultiplier, the declared amounts scaled by the fee multiplier of the epoch the action executes in (declared * multiplier_permille / 1000, rounded down), so a fee follows the network's fees. A scaled amount is held at the maximum number of credits instead of overflowing; such a fee refuses the action for an insufficient balance.
  • Who pays. Whoever pays the gas pays the fee: the signer, or the contract owner when they pay the gas of a token-paid action (see Token Costs). The contract owner never pays the owner part, which would only travel through their pot back to them: a contract owner who pays, as the signer or as the gas sponsor, pays the moderators part only. A contract that sponsors gas should price that in: a sponsor's balance must cover the gas and those fees, or the transition falls back to the signer or is refused, as the token cost's rules say.
  • Only executed actions pay. A transition that fails, at any stage, owes no fee.
  • Where it goes. One removal from the payer's balance, and one addition to each pot the fee has a part for. The fee is not part of the gas: the fee pools and block proposers get none of it.

The agreement: $actionFeeAgreement

The contract is read when the transition executes, not when it was signed. So every transition on an action that charges a fee carries an action fee agreement in its base, naming the fee its signer saw (version 2 of the document base transition, the default from protocol version 14):

"$actionFeeAgreement": {
  "$formatVersion": "0",
  "owner": 10000000,
  "moderators": 100000000,
  "feeMultiplier": { "knownPermille": 1000, "increaseTolerancePercent": 20 }
}
FieldMeaning
owner, moderatorsThe amounts the document type declares for the action, before any multiplier. They must match exactly, each part on its own
feeMultiplierPresent for a feeMultiplier fee, left out for a fixed one. knownPermille is the fee multiplier the signer priced the fee with, in thousandths (1000 is 1x). increaseTolerancePercent is how far above it the executing epoch's multiplier may be, in percent of the known one: 20 accepts up to 1.2 times

A transition on an action that charges a fee is refused:

  • without an agreement (DocumentActionFeeAgreementNotSetError, 40132), whoever pays, a sponsored transition included;
  • with other amounts, parts moved between the pots, or the other pricing (DocumentActionFeeAgreementMismatchError, 40133). The signer reads the contract again;
  • when the executing epoch's multiplier is above what the agreement tolerates (DocumentActionFeeMultiplierNotToleratedError, 40134). A multiplier that fell is always accepted, and what is charged follows the epoch's multiplier, never the known one.

Each refusal bumps the signer's nonce, and no action fee is charged. The mempool applies the same checks on arrival and on every recheck, so a transition whose agreement no longer holds leaves the mempool with the same error. An agreement on an action that charges nothing is ignored.

A client should build the agreement from the contract it showed its user, never from a contract fetched behind their back at signing time. In Rust, DocumentActionFeeAgreement::for_document_type_action builds it from a document type, and the SDK's document transition builders take it with with_action_fee_agreement.

A seated team's discount. On a document type that an elected contract moderates, the moderators part of an agreement may name less than the declared amount: exactly the share the contract's seated moderation charter takes (its moderatorsShare, in percent, rounded down to the credit). Everything else must still match. The action is then charged the agreed amount. Any other amount below the declared one, including a discount on a contract with no seated charter yet, is refused (DocumentActionFeeModeratorsShareMismatchError, 40139). A lower amount anywhere else, on a type the contract does not moderate or a contract that is not elected, is the plain mismatch (40133). See Elected Moderation.

The pots and the claim

The owner parts collect in the contract's owner pot and the moderators parts in its moderators pot. A ContractFeeClaim state transition pays a pot out:

  • the owner pot goes whole to the contract owner, the only identity that may claim it;
  • the moderators pot is split equally between the moderation team (the identities the contract appoints, or the owner alone when it appoints none), and any member of the team may claim it for all of them. A seated elected team splits it by its charter's reward split instead. What a split leaves over, less than a credit per member, stays in the pot;
  • each pot is paid out at most once per epoch, and the two are independent.

A claim is refused when the signer is not a recipient of the pot (ContractFeeClaimNotAllowedError, 41113), when the pot was already paid out this epoch (ContractFeesAlreadyClaimedThisEpochError, 41111), or when a recipient would get less than a credit (ContractFeesNothingToClaimError, 41112). The JavaScript SDK reads the pots with contracts.feePots and claims with contracts.claimFees.

Rules at registration

  • The meta-schema checks the shape (JsonSchemaError, 10101): only pricing and the six action keys; pricing one of the two values; each action an object with owner, moderators or both, each an integer from 0 to 9223372036854775807.
  • At least one action is priced, and a priced action charges something: an action whose parts are all 0 is refused, leave it out instead. The two parts of an action may not add up to more than 9223372036854775807 credits (InvalidContractStructure, 10231).
  • A nonzero moderators part needs a contract whose config declares moderation (DocumentActionFeesWithoutModerationError, 10902): the moderation team is who that pot is for. This is checked when the contract is created and when it is updated.

See also

Indexes (indices)

A document type's indices list says which queries its documents can answer and which values must be unique. Without an index, a query can only address a document by its $id. With one, Drive keeps the documents sorted by the index's properties, so a query that fixes those properties reaches the matching documents directly instead of reading the whole type. Every index costs storage and processing on each write, and an index can never be added, removed or changed once the document type exists, so a contract author decides them before the document type is registered.

This chapter covers the keywords every index can use: name, properties, unique, nullSearchable and skipIfAbsent. The other index keywords, for contests, counts, sums, rankings, time windows and index-only types, have their own chapters (see More index keywords).

Example

"post": {
  "type": "object",
  "indices": [
    { "name": "byOwnerTime", "properties": [{ "$ownerId": "asc" }, { "$createdAt": "asc" }] },
    { "name": "bySlug", "properties": [{ "$ownerId": "asc" }, { "slug": "asc" }], "unique": true },
    { "name": "byTopic", "properties": [{ "meta.topic": "asc" }], "nullSearchable": false }
  ],
  "properties": {
    "slug": { "type": "string", "minLength": 1, "maxLength": 63, "position": 0 },
    "text": { "type": "string", "maxLength": 280, "position": 1 },
    "meta": {
      "type": "object",
      "properties": {
        "topic": { "type": "string", "minLength": 1, "maxLength": 32, "position": 0 }
      },
      "additionalProperties": false,
      "position": 2
    }
  },
  "required": ["$createdAt", "slug", "text"],
  "additionalProperties": false
}

byOwnerTime lists one author's posts in the order they were written. bySlug lets each author use a slug once: a second post by the same owner with the same slug is refused. byTopic finds posts by a property nested inside meta, and leaves out posts that have no topic.

indices

Wheredocument type
Valuearray of 1 to 10 index objects
Defaultabsent: the type has no index, and its documents can only be addressed by $id
Sinceprotocol version 1
On updateFixed: an index may not be added, removed or changed (DataContractInvalidIndexDefinitionUpdateError, 10217)
ErrorsDuplicateUniqueIndexError (40105)

Each entry of indices is one index. Drive builds a tree for it when the contract is registered and maintains it on every create, replace, transfer, purchase, price update and delete of a document of the type.

A query uses an index when its where clauses fix the index's leading properties, in order, and it orders by the properties that follow. An index on [a, b] answers a == x, a == x AND b == y, and a == x ordered by b, but not b == y alone. The query picker and the tree layout are described in Indexes.

Rules at registration:

  • At most 10 indexes, and at least one when the key is present (an empty indices array is refused by the meta-schema).
  • No two indexes may have the same properties in the same order (DuplicateIndexError, 10201).
  • At most one index may be contested, and a type with a contested index may have no other unique index (ContestedUniqueIndexWithUniqueIndexError, 10249). See Contested Indexes.

name

Whereindex
Valuestring, 1 to 32 characters
Defaultnone: required
Sinceprotocol version 1
On updateFixed: indexes are compared by name, so renaming an index is a removal plus an addition (10217)
ErrorsDuplicateIndexNameError (10211) at registration

The name identifies the index in queries that name one, in error messages and in the contract update rule. Two indexes of one document type may not share a name (DuplicateIndexNameError, 10211).

properties

Whereindex
Valuearray of 1 to 10 objects, each { "<property path>": "asc" } with exactly one key
Sinceprotocol version 1
On updateFixed (10217)
ErrorsUndefinedIndexPropertyError (10209), SystemPropertyIndexAlreadyPresentError (10208), InvalidIndexPropertyTypeError (10206), InvalidIndexedPropertyConstraintError (10205), all at registration

The indexed properties, in order. The order matters: a query uses the index through a prefix of the list. The only sort order the meta-schema accepts is "asc"; a query may still walk an index in descending order.

What may be indexed:

  • A top-level property of the type, by its name.
  • A property inside an object, by its dotted path. DPNS indexes records.identity, the identity property of a domain's records object.
  • System properties: $ownerId, $createdAt, $updatedAt, $transferredAt, their *BlockHeight and *CoreBlockHeight variants, $creatorId on a type that records it, and $moderatedAt and $moderatedBy on a type that lists moderatorAbilities.changeFields, in a non-unique index, from protocol version 14 (see System Properties). A timestamp or block height is only recorded when the type lists it in required; an index on one that is not required holds every document under null.
  • Not $id, which the document type's primary tree already indexes (SystemPropertyIndexAlreadyPresentError, 10208).
  • A value of the document a reference points at, from protocol version 14: "<reference property>.<field>", such as postId.$ownerId, read from the referenced document and never stored. See Values of Referenced Documents.

What each indexed property must be, because its value becomes a GroveDB key of at most 255 bytes:

  • A property the type defines (UndefinedIndexPropertyError, 10209).
  • Not an object, an array of values or a typed array (InvalidIndexPropertyTypeError, 10206). A byte array, including an identifier, is fine.
  • A string must declare maxLength of at most 63, since a character can take four bytes. A byte array must declare maxItems of at most 255. A missing or larger bound is refused (InvalidIndexedPropertyConstraintError, 10205). An index with a ranking has tighter bounds (see Ranked Indexes).
  • From protocol version 14, not a transient property, nor one inside a transient object: its value is never stored, so the index would never hold it. See transient.

unique

Whereindex
Valueboolean
Defaultfalse
Sinceprotocol version 1
On updateFixed (10217)
ErrorsDuplicateUniqueIndexError (40105)

On a unique index no two documents may hold the same values for all of the index's properties. A create, replace, transfer, purchase or price update that would make a second document with the same values is refused with DuplicateUniqueIndexError (40105); so is a moderator's restore of a deleted document whose values have been taken since. A replace that leaves the indexed values as they were is not held against the document itself.

A document that leaves any of the indexed properties out is not held to the index: uniqueness cannot be decided on a missing value, so two such documents may coexist. Make the properties required when every document must be unique.

Uniqueness is per index. bySlug above is unique over the pair ($ownerId, slug), so two authors may use the same slug; a unique index on slug alone would make each slug global.

A unique index with a timeRange or an integerRange treats two documents in the same window as equal on the bucketed property. A unique index cannot carry a ranking.

nullSearchable

Whereindex
Valueboolean
Defaulttrue
Sinceprotocol version 1
On updateFixed (10217)

With the default, a document whose indexed properties are all missing is still entered in the index, under null, so a query for the null value finds it. false leaves such a document out of this index: it exists, and other indexes and $id still reach it, but this index does not. A document with only some of the properties missing is entered either way.

DPNS sets "nullSearchable": false on its records.identity index, so domains that point at no identity take no room in it.

nullSearchable: false is refused on an index with a ranking, a timeRange or an integerRange, on an index of an index-only type, and together with skipIfAbsent.

skipIfAbsent

Whereindex
Valuetrue, or an array of 1 to 10 of the index's property names
Defaultfalse
Sinceprotocol version 14
On updateFixed (10217)

A document that leaves out a property of the index's skip set writes nothing into this index, and its delete looks for nothing there. true makes the skip set every optional property of the index; an array names it. A value of a referenced document is never in the true set, since whether it can be absent depends on the referenced type; the array may name one. The skip property can sit anywhere in the index, under a timeRange or integerRange window too. The index then holds only the documents carrying every skip property, and its counts and rankings are "among the documents that have them". A present but empty value is not absent and is indexed.

"post": {
  "type": "object",
  "documentsMutable": true,
  "indices": [
    {
      "name": "byHashtagLanguageTime",
      "properties": [{ "hashtag": "asc" }, { "language": "asc" }, { "$createdAt": "asc" }],
      "skipIfAbsent": ["hashtag"]
    },
    {
      "name": "byDayHashtag",
      "properties": [{ "$createdAt": "asc" }, { "hashtag": "asc" }],
      "rangeCountable": true,
      "rankedCountable": true,
      "timeRange": { "on": "$createdAt", "range": 86400, "step": 86400 },
      "skipIfAbsent": true
    }
  ],
  "properties": {
    "hashtag": { "type": "string", "maxLength": 61, "position": 0 },
    "language": { "type": "string", "maxLength": 8, "position": 1 },
    "text": { "type": "string", "maxLength": 280, "position": 2 }
  },
  "required": ["$createdAt", "text"],
  "additionalProperties": false
}

An untagged post writes nothing into either index: no entry under hashtag, and no day window in byDayHashtag, whose ranking of today's hashtags therefore never has to step over an empty hashtag. A tagged post without a language is entered into byHashtagLanguageTime under null for language, since the array leaves language out of the skip set. A replace that adds or drops the hashtag moves the post into or out of both indexes.

A query only uses a skip index when it constrains every skip property, so it cannot silently miss the documents the index skipped. On a stored type the constraint must be one no missing value can meet: an equality or in with values that are neither null nor empty (an empty byte array is keyed like null), a range with a non-empty lower bound, startsWith with a non-empty prefix, or ranking by the property. Ordering by the property alone, or a range with only an upper bound, does not count: an index that does not skip would return the documents without the property too (they sort first, under null).

Rules at registration:

  • The skip set is not empty: an index whose properties are all required could never skip.
  • Each skip property is a top-level property of the type, not a system property, and not listed in required, or a value of a referenced document that can be absent.
  • No rankedCountable at level sits above the index's deepest skip property: it would count only documents carrying that property, and no query could read it without binding the property.
  • The index is not contested, and does not also set nullSearchable: false (the skip already leaves out a document with every indexed value missing).
  • On a stored type, a skip property that is a byte array sets minItems to at least 1 (on the referenced type, for a value read from a referenced document): an empty byte array is keyed like a missing value.
  • On a stored type, a ranking at a skip property's level does not share that level with an index that keeps null for the property: that index would create the null value, and the ranking would show it as a group with no documents.
  • On an index-only type the skip set holds every optional property of the index, and each optional property needs a skip index of its own, without a timeRange (a window keeps the value only until the window drains).

Respelling true as the array of the same properties, or the other way round, is no change on a contract update.

Null handling

A property is null in an index when the document leaves it out. Putting the rules above together:

The document's indexed valuesEntered in the index?Held to unique?
all presentyesyes
some missingyes, under null for the missing onesno
all missingonly if nullSearchable is trueno

An index with skipIfAbsent leaves out a document missing any of its skip properties, whatever the other values are.

The storage shape of each case is in Null Handling.

Changing indexes

Drive builds an index's trees when the document type is registered and never backfills them, so an index added later would miss every existing document, and a removed or changed one would leave orphaned trees. A contract update may therefore not add, remove or change any index of an existing document type. From protocol version 14 the update is refused with DataContractInvalidIndexDefinitionUpdateError (10217), naming the first index that differs. Indexes are compared by name: reordering the indices array is no change, and renaming an index is a removal plus an addition. Earlier protocol versions refused every such change as well, though not always with this error.

A document type that the update adds may declare any indexes, as a new contract may.

Limits

LimitValue
Indexes per document type10
Properties per index10
Characters in an index name32
Characters in an index property path256
maxLength of an indexed string63 (lower on a ranked index)
maxItems of an indexed byte array255 (lower on a ranked index)
Contested indexes per document type1

More index keywords

An index entry may carry more keywords, each with its own chapter:

  • Contested Indexes: contested turns a unique index into a scarce resource that masternodes award by vote, the way DPNS gives out names.
  • Counts, Sums and Averages: countable, summable, averageable and their range* forms keep totals per indexed value, so counts, sums and averages are read without walking the documents.
  • Ranked Indexes: rankedCountable, rankedSummable and rankedAverageable order the indexed values by those totals, for "top 10" queries with proofs.
  • Time-Range Indexes: timeRange groups documents into time windows, for "trending this hour" queries.
  • Integer-Range Indexes: integerRange groups documents into windows of an integer property, for counts and rankings per price or score band.
  • Index-Only Types: terminal, preallocated, skipIfAbsent and outlivesDelete shape the indexes of a type whose documents live only in their indexes.
  • Values of Referenced Documents: an index property "<reference property>.<field>" holds a value of the document a reference points at, which the document does not store.

See also

Contested Indexes

A unique index normally works first come, first served: the first document to take a value keeps it, and every later one is refused as a duplicate. contested changes that for the values a contract considers valuable. A document whose values fall in the contested range opens a contest, or joins one already open for the same values, and masternodes and evonodes vote on which identity gets them. DPNS uses it so that a short name such as alice.dash cannot simply be taken by whoever submits first. A contract author reaches for it when a unique value is scarce and should be awarded rather than raced for.

Wherea unique index
Valueobject: resolution (required), fieldMatches, description
Sinceprotocol version 1; resolution: 1 from protocol version 14
On updateFixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217)
ErrorsDocumentContestNotPaidForError (40114), DocumentContestCurrentlyLockedError (40110), DocumentContestNotJoinableError (40111), DocumentContestIdentityAlreadyContestantError (40112), DocumentContestIndexMismatchError (40118), DocumentContestNotRequiredError (40119), DocumentContestMaximumContendersReachedError (40141); DuplicateUniqueIndexError (40105) for values outside the contested range

Example

The domain document type of the DPNS contract:

{
  "name": "parentNameAndLabel",
  "properties": [
    { "normalizedParentDomainName": "asc" },
    { "normalizedLabel": "asc" }
  ],
  "unique": true,
  "contested": {
    "fieldMatches": [
      { "field": "normalizedLabel", "regexPattern": "^[a-zA-Z01-]{3,19}$" }
    ],
    "resolution": 0,
    "description": "If the normalized label part of this index is less than 20 characters (all alphabet a-z, A-Z, 0, 1, and -) then a masternode vote contest takes place to give out the name"
  }
}

A name is unique under its parent domain. A label of 3 to 19 characters made of letters, 0, 1 and - is contested: registering it opens a masternode vote, which may give it to a contender or lock it. Any other label, a longer one or one with other digits, is registered first come, first served, and a second registration of it is a duplicate.

The keys

KeyValueWhat it does
resolution0 or 1, requiredHow the contest is decided. 0: masternodes vote for a contender, abstain, or lock the value so nobody gets it. 1 (from protocol version 14): masternodes vote for a contender or abstain; there is no lock, so the contest always ends with a winner.
fieldMatchesarray of at least one { "field", "regexPattern" }Which values are contested. field is a property path of the document; regexPattern a regular expression its value must match. Each is 1 to 256 characters.
descriptionstring, 1 to 256 charactersA note for readers. Consensus does not read it.

How it works

Which documents are contested. A document is contested when, for every fieldMatches entry, the document holds a string at field and regexPattern matches it. A missing value, a value that is not a string, or one entry that does not match makes the document an ordinary unique-index document. Without fieldMatches, every document the index covers is contested. The pattern uses the syntax of Rust's regex crate and matches anywhere in the value, so write ^ and $ to match the whole of it, as DPNS does.

Values outside the contested range behave exactly like any unique index: the first document takes the value, and a later create with the same values is refused with DuplicateUniqueIndexError (40105).

Opening or joining a contest. A create whose values are contested must carry $prefundedVotingBalance, a pair [indexName, amount]: the contested index's name and the most credits the contender will pay to fund the vote. The document is not stored under the index yet; it is held as a contender until the contest ends. A create is refused:

  • with DocumentContestNotPaidForError (40114) when it carries no prefunded balance, or less than the contest's fund. The fund is the contested document fund, 0.1 Dash. From protocol version 14 it doubles once the contest holds 250 contenders and again for every 50 more, and a contender is charged the fund and keeps what it stated beyond it; before 14 every contender stated and paid exactly the fund.
  • with DocumentContestIndexMismatchError (40118, from protocol version 14) when the pair names another index than the contested one its values match.
  • with DocumentContestNotRequiredError (40119, from protocol version 14) when the document is not contested but carries a prefunded balance.
  • with DocumentContestNotJoinableError (40111) when the contest's join window (one week on mainnet) has passed.
  • with DocumentContestIdentityAlreadyContestantError (40112) when its owner is already a contender.
  • with DocumentContestMaximumContendersReachedError (40141, from protocol version 14) when the contest already holds 1,000 contenders.
  • with DocumentContestCurrentlyLockedError (40110) when an earlier contest for these values ended locked.

The vote. Masternodes and evonodes vote with MasternodeVote transitions for the length of the poll (two weeks on mainnet). Under resolution: 0 the contender with the most votes wins unless the lock tally beats it, and the contest always runs its full length, even with one contender. Under resolution: 1 a contender always wins, and a contest that still has a single contender when its join window closes is awarded to it at once. From protocol version 14 a tie goes to the earliest contender.

After the contest. The winning document is stored and held by the index like any unique-index document; the other contenders' documents are removed. The contest's result stays readable.

The fund, the windows, the tallies and the special case of moderation elections are described in Contested Documents.

Rules at registration

  • The index is unique: true. A contested index that is not unique is refused (InvalidContractStructure, 10231).
  • The document type's documents cannot be replaced: documentsMutable: false (ContestedUniqueIndexOnMutableDocumentTypeError, 10248). They may still be transferred and sold, as DPNS domains are.
  • A document type has at most one contested index, and no other unique index beside it (ContestedUniqueIndexWithUniqueIndexError, 10249).
  • resolution is present and is 0 or 1; 1 is refused before protocol version 14.
  • fieldMatches, when present, holds at least one entry, and each regexPattern is a valid regular expression (RegexError, 10247).
  • A contested index cannot carry a timeRange, an integerRange or a ranking (a ranking needs a non-unique index), and an index-only type cannot have one.
  • A document type with a contested index cannot set ttl, nor moderatorAbilities.delete (see Deletion): a moderator's restore puts a document back by an ordinary insert, and a contested value is only awarded through a vote.
  • From protocol version 14, a document type with a contested index that sums a property (summable, averageable, documentsSummable or documentsAverageable) declares that property with a minimum of at least -134217728 and a maximum of at most 134217728 (±2^27, max_contested_summed_value_magnitude). The end of a contest adds the winner's value to the type's sums, after any number of other documents were written, and a sum that left the signed 64-bit range there could not be stored. Values this small keep the sums in range short of 2^36 documents. Checked when a contract is registered or updated, so contracts registered earlier keep their bounds.

See also

  • Contested Documents for the contest's lifecycle, fund, resolutions, ties and storage.
  • Indexes for unique and the other index keywords.
  • Mutability for documentsMutable, which a contested type sets to false.

Counts, Sums and Averages

Counting documents normally means fetching them and counting what comes back, which grows with the number of documents and proves every one of them. The keywords in this chapter make Drive keep running totals inside its trees instead, so a count, a sum or an average is read without visiting the documents, and proved with a short proof. The document type keywords keep totals over all of a type's documents. The index keywords keep totals per indexed value, and with their range* forms over ranges of values. An average is never stored as such: the platform returns the count and the sum of the same documents, and the client divides.

Every total is updated by every write that touches it, so each flag adds to the cost of writing documents of the type. The flags choose the layout of the type's trees, so all of them are fixed when the document type is created.

Example

"tip": {
  "type": "object",
  "documentsMutable": false,
  "documentsAverageable": "amount",
  "indices": [
    { "name": "byRecipient", "properties": [{ "recipient": "asc" }], "averageable": "amount" },
    { "name": "byDay", "properties": [{ "day": "asc" }], "summable": "amount", "rangeSummable": true }
  ],
  "properties": {
    "recipient": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "position": 0
    },
    "amount": { "type": "integer", "minimum": 1, "maximum": 4294967295, "position": 1 },
    "day": { "type": "integer", "minimum": 0, "maximum": 4294967295, "position": 2 }
  },
  "required": ["recipient", "amount", "day"],
  "additionalProperties": false
}

documentsAverageable keeps the number of tips and their total, so the average tip over the whole type is one read. byRecipient keeps the same pair per recipient, so "how many tips did this identity get, and how much on average" is one read. byDay keeps the sum per day and, with rangeSummable, answers "total tipped between day 100 and day 130" without visiting each day.

documentsCountable

Wheredocument type
Valueboolean
Defaultfalse
Sinceprotocol version 12
On updateFixed (DocumentTypeUpdateError, 40212)

Keeps the number of the type's documents in its primary tree, so a count query with no where clause is one read, with a proof. Without it, such a query is refused rather than answered slowly.

documentsSummable

Wheredocument type
Valuethe name of an integer property, 1 to 64 characters
Defaultabsent
Sinceprotocol version 12
On updateFixed (40212)

Keeps the sum of the named property over all the type's documents, so a sum query with no where clause is one read. On a type with documentsKeepHistory, the sum counts each document's current revision only.

The property must exist on the type, be listed in required (a missing value would leave the sum wrong when the document is deleted), and hold values that fit a signed 64-bit sum. Integers are stored in the smallest width their bounds allow when the contract uses sizedIntegerTypes, and a property that becomes an unsigned 64-bit integer is refused: "minimum": 0 with no maximum does, so add a maximum of at most 4294967295, or give no bounds at all.

documentsAverageable

Wheredocument type
Valuethe name of an integer property, 1 to 64 characters
Defaultabsent
Sinceprotocol version 12
On updateFixed (40212)

Shorthand for documentsCountable: true plus documentsSummable on the named property: the count and the sum an average is computed from. The storage is exactly that of the two flags. When documentsSummable is also written it must name the same property, and an explicit documentsCountable: false beside it is refused as a contradiction.

Document type rangeCountable, rangeSummable, rangeAverageable

Wheredocument type
Valueboolean each
Defaultfalse
Sinceprotocol version 12
On updateFixed (40212)

These keep the count, the sum, or both at every node of the primary tree rather than only at its root, which is what range and offset aggregates over the primary key need. They are rarely useful: a range count or sum is almost always wanted over an indexed property, which is what the index keywords of the same names give.

  • rangeCountable implies documentsCountable.
  • rangeSummable needs documentsSummable or documentsAverageable.
  • rangeAverageable is shorthand for rangeCountable plus rangeSummable, and needs documentsAverageable. An explicit false for either of the two beside it is refused as a contradiction.

countable

Whereindex
Value"notCountable", "countable", "countableAllowingOffset", or a boolean (true is "countable", false is "notCountable")
Default"notCountable"
Sinceprotocol version 12
On updateFixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217)

Keeps a document count for each value of the index, so "how many documents have these values" is one read per value, with a proof. A count query is served by a countable index whose properties exactly match its where clauses, each an equality or an in: a countable index on [brand, color] answers brand == "a" AND color == "red", and color in [...] under a fixed brand with one count per color, but not brand == "a" alone. Declare one countable index per shape of count the application needs.

"countableAllowingOffset" keeps a count at every node of the index's trees, not only per value. It costs more on every write and prepares the index for offset queries; use "countable" unless you need it. The boolean form is kept for contracts written before the string form existed.

On a unique index the flag changes almost nothing, since each value holds at most one document; it only counts the documents that leave an indexed property out.

summable

Whereindex
Valuethe name of an integer property, 1 to 64 characters
Defaultabsent
Sinceprotocol version 12
On updateFixed (10217)

Keeps the sum of the named property for each value of the index, the way countable keeps a count, and serves sum queries under the same exact-match rule. The property follows the rules of documentsSummable: it exists, is required, and fits a signed 64-bit sum.

averageable

Whereindex
Valuethe name of an integer property, 1 to 64 characters
Defaultabsent
Sinceprotocol version 12
On updateFixed (10217)

Shorthand for countable: "countable" plus summable on the named property, which is what an average query per value needs. When summable is also written it must name the same property. An explicit countable: "countableAllowingOffset" beside it is kept; an explicit "notCountable" (or false) is refused as a contradiction.

Index rangeCountable, rangeSummable, rangeAverageable

Whereindex
Valueboolean each
Defaultfalse
Sinceprotocol version 12
On updateFixed (10217)

These answer aggregates over a range of the index's last property: "how many reviews between grade 60 and 80", "total tipped from day 100 to day 130". The count or sum is kept at every node of the tree of that property's values, so a range total is read by walking the edges of the range, with a proof whose size grows with the logarithm of the number of values rather than with the documents. The same index also returns one total per distinct value in a range. The query fixes the properties before the last one, with equalities or an in.

  • rangeCountable makes the index countable: an omitted countable becomes "countable", an explicit "countableAllowingOffset" is kept, and an explicit "notCountable" is refused. Before protocol version 14, countable had to be written out beside it.
  • rangeSummable needs summable or averageable.
  • rangeAverageable is shorthand for rangeCountable plus rangeSummable, and needs averageable. An explicit false for either of the two beside it is refused.

A range total costs more on each write than a per-value total, since every node of the tree carries it. The ranked keywords build on these flags: see Ranked Indexes.

How they combine

  • Count and sum on one tree. An index with both a count and a sum (by averageable, or by countable plus summable) keeps both in one tree, and one proof returns the pair. The same holds for the document type flags.
  • One summed property per type. Every summable, averageable, documentsSummable and documentsAverageable of a document type must name the same property: the sums share their trees, which have no room to tell two properties apart. To sum two properties, use two document types.
  • Type and index flags are independent. documentsCountable gives the unfiltered total; a countable index gives filtered ones. A type may have both.
  • Index-only types take the index keywords but refuse the document type ones, since they have no primary tree. See Index-Only Types.
  • Queries must match. A count or sum query that no index serves exactly is refused; there is no slow fallback. Pick the indexes for the queries the application will make.

Rules at registration

  • A summed property exists on the type, is an integer that fits a signed 64-bit sum (not an unsigned 64-bit integer), and is listed in required.
  • All summed properties of a type are the same property.
  • From protocol version 14, on a type with a contested index, the summed property declares a minimum of at least -134217728 and a maximum of at most 134217728 (±2^27).
  • From protocol version 14, on a type with a ttl, the summed property declares a minimum of at least 0, or a minimum of at least -134217728 and a maximum of at most 134217728 (±2^27).
  • On other types, each value only has to fit a signed 64-bit integer.
  • The shorthands agree with their longhand where both are written, and no explicit false or "notCountable" contradicts them.
  • Each range* flag has its prerequisite, as listed above.
  • The meta-schema refuses a malformed value (JsonSchemaError, 10101).

Choosing what to set

The application needsSet
The number of documents of the typedocumentsCountable: true
The sum, or the average, of a property over the whole typedocumentsSummable or documentsAverageable
A count per value (where author == x)countable: "countable" on an index whose properties are exactly those of the query
A sum or an average per valuesummable or averageable on such an index
A count, sum or average over a range (where grade > 60)the matching range* flag on an index whose last property is the ranged one
The top or bottom values by count, sum or averagethe range* flags plus a ranking: see Ranked Indexes

See also

Ranked Indexes

A ranked index answers "which values score highest": the five restaurants with the best average grade, the ten hashtags with the most posts, the three sellers with the largest sales. The range aggregates of Counts, Sums and Averages keep a count or a sum for every value of an index, but in the order of the values, so finding the top five means reading every value. A ranking adds a second, ordered view of the same totals, and a "top K" query reads K entries from one end of it, with a proof that grows with K rather than with the number of values. Each ranking costs one more tree that every write under the index updates, so a contract declares only the rankings it will query.

The index's groups are the distinct values of its last property. The three keywords rank the groups by document count, by the sum of the summed property, or by its average.

Example

"review": {
  "type": "object",
  "indices": [
    {
      "name": "byRestaurant",
      "properties": [{ "restaurantId": "asc" }],
      "averageable": "grade",
      "rangeAverageable": true,
      "rankedAverageable": true,
      "rankedCountable": true
    }
  ],
  "properties": {
    "restaurantId": { "type": "string", "minLength": 1, "maxLength": 32, "position": 0 },
    "grade": { "type": "integer", "minimum": 0, "maximum": 100, "position": 1 }
  },
  "required": ["restaurantId", "grade"],
  "additionalProperties": false
}

averageable and rangeAverageable keep the count and the sum of grade per restaurant; rankedAverageable orders the restaurants by average grade and rankedCountable by number of reviews. The query

SELECT avg(grade) FROM review GROUP BY restaurantId ORDER BY avg(grade) DESC LIMIT 3

returns the three best-rated restaurants with their counts and sums, proved.

rankedCountable

Whereindex
Valueboolean, or { "at": <property name or array of 1 to 10 names> }
Defaultfalse
Sinceprotocol version 14
On updateFixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217)

Ranks groups by how many documents they hold. Needs rangeCountable: true on the index, or rangeAverageable: true, which implies it.

true ranks the values of the index's last property. On a compound index, the ranking is kept separately for each value of the properties before it: on [city, restaurantId] each city has its own ranking of restaurants, and a query names the city.

The object form { "at": ... } places the ranking at another level of the index. Naming an earlier property ranks that property's values by the number of documents beneath them, whatever the later properties hold:

{
  "name": "byHashtagPost",
  "properties": [{ "hashtag": "asc" }, { "postId": "asc" }],
  "countable": "countable",
  "rangeCountable": true,
  "rankedCountable": { "at": ["hashtag", "postId"] }
}

"hashtag" in at ranks hashtags by their total posts across all post ids; "postId", the last property, is the same as true and ranks the posts under one hashtag. An array declares several rankings on one index; each level named costs one more ordered tree to maintain on every write beneath it. A query addresses a ranking by the property it groups by, with every property before that one fixed.

at names only the index's own properties, each once. The object form cannot be combined with rankedSummable or rankedAverageable: a ranking at an earlier level is fed by a chain of counts that cannot also carry a sum. A summableOffCountIndex index is the exception: its counters feed a chain of sums, and of counts too from the shallowest average ranking down, so it takes all three at any level. An average reads a level only where that chain carries counts: above the shallowest average ranking its value trees sum but count nothing. There a document count is its sums (likes, not posts), so rankedCountable declares the same ranking as rankedSummable, which a ranked count(*) and a ranked sum of the source both read, and it needs no rangeCountable.

rankedSummable

Whereindex
Valueboolean, or { "at": <property name or array of 1 to 10 names> } on a summableOffCountIndex index
Defaultfalse
Sinceprotocol version 14
On updateFixed (10217)

Ranks the groups of the index's last property by the sum of the index's summable property: the recipients who received the most, the products that sold the most units. Needs rangeSummable: true, or rangeAverageable: true. On a summableOffCountIndex index the sum is the source's entries, and at ranks an earlier level by it: authors by the likes their posts received. A ranked count(*) over such an index reads this ranking too, since its sums are its document counts.

rankedAverageable

Whereindex
Valueboolean, or { "at": <property name or array of 1 to 10 names> } on a summableOffCountIndex index
Defaultfalse
Sinceprotocol version 14
On updateFixed (10217)

Ranks the groups of the index's last property by the average of the averageable property. On a summableOffCountIndex index the average is entries per group, and at ranks an earlier level by it: authors by likes per post. Needs both range totals: rangeAverageable: true, or rangeCountable: true with rangeSummable: true.

The three keywords are independent. rankedAverageable does not imply rankedCountable or rankedSummable, unlike averageable, which is shorthand for a count and a sum. Declare each ranking the application will query, and no other.

Queries

A ranked query names one aggregate, groups by the ranked property, orders by the aggregate and takes a limit, with an optional offset. On a compound index such as [city, restaurantId]:

SELECT count(*) FROM review WHERE city == "London" GROUP BY restaurantId ORDER BY count(*) DESC LIMIT 5

Every property before the grouped one must be fixed with an equality; at most one of them may instead be an in of 2 to 10 values, whose rankings are merged. There is no ranking across different values of those properties. A ranked index still answers the per-value range queries the same range* flags answer (one entry per value in the range). A range total, one count, sum or average over a whole range, is not available through an index whose path passes through a ranked level, one it ranks itself or one another index ranks at a level the two share: the ranked trees are indexed trees, which grovedb neither totals a range over nor proves a range total through, and an unproven total through a ranked level is refused as well so that a proved and an unproven read agree. Such a query is refused with a hint to group by the last property.

The request shape, ties, offsets and proofs are described in Ranked Index Examples.

Rules at registration

  • Its range totals. Each ranking needs the range flags listed above, checked both by the meta-schema and by the parser. A written-out false needs nothing.
  • A non-unique index. Every group of a unique index holds at most one document, so a ranking on one is refused, and a contested index, which is unique, cannot have one either.
  • nullSearchable left at true. With false, documents missing the property would still leave an empty group in the ranking.
  • A shorter key. The ranked property's value becomes part of the ordered tree's key, behind an 8-byte sort key (16 bytes when rankedAverageable ranks that property), so its worst case must fit in what is left of the 255-byte limit. A ranked string takes maxLength of at most 61, or 59 at a property rankedAverageable ranks; a ranked byte array takes maxItems of at most 247, or 239. A larger bound is refused (InvalidIndexedPropertyConstraintError, 10205). Identifiers, integers and other fixed-width values always fit. The bound applies to each ranked property, by the rankings at that property: the last one when a ranking is declared with true, and each property named in at.
  • Time and value windows. On a timeRange or integerRange index, a ranking must sit below the bucketed property, which gives one ranking per window. A single-property bucketed index cannot be ranked, and at cannot name the bucketed property.
  • Other indexes of the type. A compound ranked index [p1, ..., pn] is refused when another countable or summable index ends at exactly [p1, ..., pn-1]. A ranking at an earlier level (at) also restricts the other indexes that reach that level; the full table is in Shape Restrictions.
  • One index per property list. Two indexes with the same properties are a DuplicateIndexError (10201), so the rankings of one property list go on one index.
  • Protocol version 14. Earlier versions do not know the keywords and refuse them.

A broken rule other than the key length is refused as InvalidContractStructure (10231), or by the meta-schema as JsonSchemaError (10101).

Costs

Each ranking is one ordered tree, rewritten whenever a document under it is created, changed or deleted, on top of the range totals it is built from. Two rankings on one index cost two rewrites per write; a fully ranked at array costs one per ranked level. A ranking at the index's first property gets its tree when the contract is registered; a ranking at a deeper level gets one tree per value above it, as documents arrive.

See also

Time-Range Indexes

timeRange groups an index's documents into time windows by one of their timestamps, so that "the most used hashtags this hour" or "posts per day" is a question about one window rather than a scan over time. Each window is range seconds long and a new one starts every step seconds; when windows overlap, a document belongs to each window that contains its timestamp. Combined with the count and ranking keywords, a time-range index serves trending lists and leaderboards per window, with proofs. It costs one set of index entries per window a document falls in, and a ttl lets old windows expire so that this data does not stay in state forever.

Whereindex; buckets the index's first property
Valueobject: on, range, step (required), phase, ttl
Defaultabsent: the timestamp is indexed as it is
Sinceprotocol version 14 (ttl included)
On updateFixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217)
ErrorsDuplicateUniqueIndexError (40105) on a unique time-range index

Example

"post": {
  "type": "object",
  "indices": [
    {
      "name": "hourlyByHashtag",
      "properties": [{ "$createdAt": "asc" }, { "hashtag": "asc" }],
      "timeRange": { "on": "$createdAt", "range": 3600, "step": 900, "ttl": 86400 },
      "countable": "countable",
      "rangeCountable": true
    },
    {
      "name": "onePerDay",
      "properties": [{ "$createdAt": "asc" }, { "$ownerId": "asc" }],
      "unique": true,
      "timeRange": { "on": "$createdAt", "range": 86400, "step": 86400 }
    }
  ],
  "properties": {
    "hashtag": { "type": "string", "minLength": 1, "maxLength": 63, "position": 0 },
    "text": { "type": "string", "maxLength": 280, "position": 1 }
  },
  "required": ["$createdAt", "hashtag", "text"],
  "additionalProperties": false
}

hourlyByHashtag keeps one-hour windows starting every 15 minutes, so each post is counted in four windows, and the counts per hashtag in the window that covers the last hour are one query. Its entries expire a day after their window starts. onePerDay has non-overlapping daily windows and is unique, so an identity may post once per day: a second post in the same day is refused with DuplicateUniqueIndexError (40105).

The keys

KeyValueWhat it does
on"$createdAt", "$updatedAt" or "$transferredAt", requiredThe timestamp to bucket. It must be the index's first property and be listed in the type's required, since a timestamp that is not required is never recorded.
rangeinteger seconds, at least 1, requiredThe length of each window. An exact multiple of step.
stepinteger seconds, at least 1, requiredThe time between the starts of two windows.
phaseinteger seconds, default 0Moves the window boundaries: windows start at phase + k * step from the Unix epoch. Less than step and less than one year (31536000). Daily windows cut at 06:00 UTC take "phase": 21600.
ttlinteger seconds, at least 1How long the index's entries live after their window starts. At least range and at most 604800 (one week) at protocol version 14. Absent, entries live forever.

How it works

Buckets. For each document, the index stores the start of every window that contains the document's timestamp, in milliseconds, in place of the timestamp itself. With range equal to step the windows do not overlap and a document is in exactly one; otherwise it is in range / step of them, the index's overlap factor, and costs that many sets of entries. The overlap factor is at most 24 at protocol version 14. Windows are declared in seconds because block time, which the timestamps come from, moves in steps of about five seconds.

A source that changes, $updatedAt or $transferredAt, moves the document into the current windows each time it changes. Several indexes may bucket the same timestamp with different grids; each grid is stored apart from the others.

Queries. A query selects one window of the index with an IN_TIME_RANGE clause on the bucketed timestamp:

  • newest: the window that started most recently, which covers the latest slice of time (up to one step).
  • oldest: the oldest window still open, which covers nearly a full range of history: the one for "trending over the last hour".
  • byStart: the window starting at a given millisecond time on the grid, which reaches past windows.

newest and oldest are resolved against block time on the server and against the signed time of the response when verified. When the timestamp is bucketed by several grids, the query names the grid. A query may select at most one window. A window with no documents, including one that has not started, is a proved empty answer. Counts, sums and rankings declared on the index are then per window: a ranking below the bucketed timestamp is one leaderboard per window.

Uniqueness. On a unique time-range index, two documents conflict when they fall in the same window and agree on the index's other properties. That is only meaningful when each document is in one window and cannot move, so uniqueness needs non-overlapping windows over $createdAt.

Expiry (ttl). An entry can be queried until ttl seconds after its window's start; a query for an expired window is refused, so every window a query can reach is complete. The expired entries are removed lazily: each later write into the index removes some of the oldest expired entries, within a bounded budget, at no charge to the writer. An index that stops receiving writes keeps its expired entries.

Everything written under an index with a ttl is billed as processing, at an ephemeral-bytes rate, instead of as storage, and nothing is refunded when it is removed. The rate prices a lifetime of at most a week, which is why ttl is capped there.

A ttl removes entries from this index only. The documents stay, and so do their entries in the type's other indexes. To delete the documents themselves after a time, use the document type's ttl.

On an index-only type, a window with a ttl may also declare outlivesDelete: a delete of a document then leaves its entries in the window to expire, so the delete needs no $createdAt, and the document keeps counting there until the window moves past it.

Rules at registration

  • on names $createdAt, $updatedAt or $transferredAt, which is the index's first property and is listed in required. A user property cannot be bucketed by time; an integer one can be bucketed by value with integerRange.
  • range and step are at least 1, range is an exact multiple of step, and range / step is at most 24.
  • phase is less than step and less than 31536000.
  • ttl, when present, is at least range and at most 604800. Two indexes that bucket the same timestamp with the same range, step and phase share their storage and must declare the same ttl, or none.
  • A unique time-range index has range equal to step and on equal to $createdAt.
  • The index is not contested, does not set nullSearchable: false, and is not preallocated.
  • A ranking sits below the bucketed timestamp: a single-property time-range index cannot be ranked, and rankedCountable.at cannot name the timestamp. See Ranked Indexes.
  • A refersTo findBy cannot resolve through a time-range index. See findBy.
  • On an index-only type, only $createdAt can be bucketed, and a bucketed index cannot serve as the type's proof index.

A broken rule is refused as InvalidContractStructure (10231), or by the meta-schema as JsonSchemaError (10101). Before protocol version 14 the keyword is unknown and refused.

See also

Integer-Range Indexes

integerRange groups an index's documents into windows of an integer property, so that "listings priced 500 to 799 by category" or "players per score band" is a question about one window rather than a scan over values. Each window is range units long and a new one starts every step units; when windows overlap, a document belongs to each window that contains its value. Combined with the count and ranking keywords, an integer-range index serves counts, sums and leaderboards per window, with proofs. It is the integer counterpart of timeRange, and costs one set of index entries per window a document falls in.

Whereindex; buckets the index's first property
Valueobject: on, range, step (required), phase
Defaultabsent: the property is indexed as it is
Sinceprotocol version 14
On updateFixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217)
ErrorsDuplicateUniqueIndexError (40105) on a unique integer-range index

Example

"listing": {
  "type": "object",
  "indices": [
    {
      "name": "byPriceBand",
      "properties": [{ "price": "asc" }, { "category": "asc" }],
      "integerRange": { "on": "price", "range": 300, "step": 100 },
      "countable": "countable",
      "rangeCountable": true
    },
    {
      "name": "oneBidPerBand",
      "properties": [{ "price": "asc" }, { "$ownerId": "asc" }],
      "unique": true,
      "integerRange": { "on": "price", "range": 1000, "step": 1000 }
    }
  ],
  "properties": {
    "price": { "type": "integer", "minimum": 0, "maximum": 1000000, "position": 0 },
    "category": { "type": "string", "minLength": 1, "maxLength": 63, "position": 1 }
  },
  "required": ["price", "category"],
  "additionalProperties": false
}

byPriceBand keeps windows 300 wide starting every 100, so a listing priced 750 is counted in the windows starting at 500, 600 and 700, and the listings per category between 600 and 899 are one query. oneBidPerBand has non-overlapping windows of 1000 and is unique, so an identity may hold one listing per thousand: a second listing priced 1200 next to one priced 1900 is refused with DuplicateUniqueIndexError (40105).

The keys

KeyValueWhat it does
onproperty name, requiredThe integer property to bucket. It must be the index's first property, an integer property of the type of at most 64 bits, and listed in the type's required.
rangeinteger, at least 1, requiredThe length of each window, in the property's own units. An exact multiple of step.
stepinteger, at least 1, requiredThe distance between the starts of two windows.
phaseinteger, default 0Moves the window boundaries: windows start at phase + k * step for every integer k, negative ones included. Less than step. Windows of 100 cut at 50, 150, 250 take "phase": 50.

How it works

Windows. For each document, the index stores the start of every window that contains the document's value, in place of the value itself, encoded exactly like a value of the property. With range equal to step the windows do not overlap and a value is in exactly one; otherwise it is in range / step of them, the index's overlap factor, and the document costs that many sets of entries. The overlap factor is at most 24 at protocol version 14, the same cap as a time-range grid's.

The bottom window. The schema's minimum and maximum decide how many bytes the property takes and whether it can be negative. A property with a non-negative minimum, or with only a maximum, is unsigned and its lowest possible value is 0. Otherwise it is signed, and its lowest possible value is the smallest the chosen width holds: -128 for "minimum": -100, "maximum": 100, and -9223372036854775808 for an integer with no bounds. A window that would start below the lowest value the property's type can hold starts at that value instead. So on an unsigned property with "phase": 50, the values 0 to 49 are in a window that starts at 0, and with overlapping windows every window that would start below the lowest value merges into the one that starts at it. Every value the property can hold is in at least one window. When the lowest value is itself a window start, as it is on an unsigned property with no phase, those windows add nothing and simply do not exist.

A property that changes value moves the document into the windows of its new value. Several indexes may bucket the same property with different grids; each grid is stored apart from the others.

The bucketed property is required, so it is never a skipIfAbsent property, but the index's other properties can be: a document missing one of them writes no entry in any window, and an update that adds or drops it moves the document into or out of all of its windows.

Queries. A query selects one window of the index with an IN_INTEGER_RANGE clause on the bucketed property, naming the window by its start. The start must be a window start of the grid, or the lowest value of the property's type for a clamped bottom window; any other value is refused rather than rounded. The server and the proof verifier both read the window from the query itself, so no clock is involved. When the property is bucketed by several grids, the query names the grid. A query may select at most one window, of either kind. A window with no documents is a proved empty answer. Counts, sums and rankings declared on the index are then per window: a ranking below the bucketed property is one leaderboard per window.

A plain clause on the bucketed property, such as price == 700, never reads an integer-range index: the index holds window starts, not prices. Such a query needs an index that is not bucketed.

In the JavaScript SDK the selection is an integerRange entry of the query: { field: "price", start: 600 }, with grid: { range, step, phase } when the property has several grids. A start beyond Number.MAX_SAFE_INTEGER is given as a decimal string. In the Rust SDK it is DocumentQuery::with_integer_range("price", 600u64).

Uniqueness. On a unique integer-range index, two documents conflict when their values fall in the same window and they agree on the index's other properties. That is only meaningful when each value is in one window, so uniqueness needs non-overlapping windows. A document may change its value within its own window; moving into a window where another document holds the same other properties is refused.

Rules at registration

  • on names an integer property of the document type of at most 64 bits, which is the index's first property and is listed in required. A system property cannot be bucketed by value; for timestamps use timeRange.
  • range and step are at least 1, range is an exact multiple of step, and range / step is at most 24.
  • phase is less than step.
  • A unique integer-range index has range equal to step.
  • The index is not contested, does not set nullSearchable: false, is not preallocated, and does not also declare timeRange.
  • A ranking sits below the bucketed property: a single-property integer-range index cannot be ranked, and rankedCountable.at cannot name the bucketed property. See Ranked Indexes.
  • on is not inside an object that is left out of required: a document could otherwise omit the object and fall in no window.
  • The grid-qualified level key (on#range#step, with #phase when non-zero) is at most 255 bytes.
  • A refersTo findBy cannot resolve through an integer-range index, and a propertyConstraints count or sum does not read one. See findBy.
  • An index-only type cannot declare integerRange: its entries are keyed by the index's values and terminal, so two rows that differ only in the bucketed integer would claim the same entry in every window they share.

A broken rule is refused as InvalidContractStructure (10231), or by the meta-schema as JsonSchemaError (10101). Before protocol version 14 the keyword is unknown and refused.

See also

Index-Only Types

Some documents are nothing but a position: a like says which post, which hashtag and which identity, and nothing else. Stored as an ordinary document, a like pays for a serialized body, a row in the primary tree and a reference in every index, for a fact its index entries already hold. An index-only type stores no body and no row: its index entries are its documents. That cuts the storage of a small document by more than half, and makes each index a uniqueness rule. In exchange, its documents can only be created and deleted, every property must live in an index or in the entry's value, and a query returns documents rebuilt from index entries rather than fetched by $id.

Seven keywords shape an index-only type: indexOnly and entryPayload on the document type, and terminal, preallocated, summableOffCountIndex, skipIfAbsent and outlivesDelete on its indexes. All of them arrived at protocol version 14 and are fixed once the type exists.

Example

A like of a social contract whose post type cannot be deleted:

"like": {
  "type": "object",
  "indexOnly": true,
  "documentsMutable": false,
  "canBeDeleted": true,
  "indices": [
    {
      "name": "byHashtagPost",
      "properties": [{ "hashtag": "asc" }, { "postId": "asc" }],
      "countable": "countable",
      "rangeCountable": true,
      "rankedCountable": true,
      "skipIfAbsent": true
    },
    {
      "name": "byPost",
      "properties": [{ "postId": "asc" }],
      "countable": "countable",
      "rangeCountable": true,
      "rankedCountable": true
    },
    { "name": "byLiker", "properties": [{ "$ownerId": "asc" }], "terminal": "postId" }
  ],
  "properties": {
    "hashtag": { "type": "string", "minLength": 1, "maxLength": 63, "position": 0 },
    "postId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "refersTo": {
        "type": "permanentDocument",
        "documentType": "post",
        "where": { "hashtag": "hashtag" }
      },
      "position": 1
    }
  },
  "required": ["postId"],
  "additionalProperties": false
}

byPost holds one entry per post and liker (its terminal defaults to $ownerId), so an identity can like a post once, and it counts and ranks posts by likes. byHashtagPost ranks the posts under each hashtag, and only likes that carry a hashtag enter it. byLiker lists the posts one identity liked. The reference makes sure the post exists and that a like's hashtag is its post's.

indexOnly

Wheredocument type
Valueboolean
Defaultfalse
Sinceprotocol version 14
On updateFixed (DocumentTypeUpdateError, 40212)
ErrorsDuplicateUniqueIndexError (40105), DocumentNotFoundError (40101), InvalidDocumentTransitionActionError (10404)

true stores the type's documents only as index entries. Each entry sits under the index's values, is keyed by the terminal's values in place of the document id, and holds a 32-byte row commitment: a hash over all of the document's values that ties its entries in the different indexes together as one document.

Create. A create writes one entry into each index. If any of those entries already exists the create is refused with DuplicateUniqueIndexError (40105), so every index is a uniqueness rule over its properties and its terminal. refersTo and the other property checks run as on any document.

Delete. A document has no id to delete by. It is deleted with an indexOnlyDelete transition that carries all of its values (and $createdAt when the type requires it); every entry those values produce must exist and carry the matching row commitment, or the delete is refused with DocumentNotFoundError (40101). Deleting by id on an index-only type, or with indexOnlyDelete on an ordinary type, is refused with InvalidDocumentTransitionActionError (10404). The owner can only ever reach its own entries, since every index that keeps entries holds $ownerId; a summableOffCountIndex counter moves only with the entry the delete removes from its source. canBeDeleted: false forbids deletes as on any type.

No other action. A document cannot be replaced, transferred, sold or repriced.

Queries. A query goes through one index and returns documents rebuilt from its entries: the index's properties, the terminal, and $ownerId and $createdAt where the index holds them. A query through an index that holds only some of the properties yields only those with a proof; without one it is refused (Unsupported), since the documents could not be serialized, and so is a chained or composite read of them. The rebuilt $id is a hash of the entry's position and addresses nothing, so there is no fetch by $id and no startAt cursor; a query pages by the terminal instead (postId > <last seen>, with a limit). A query that sets the terminal with an equality can put an in on the index's last property, with a limit of at least its number of values; a range on an index property in such a query is refused, because its pages could hold fewer rows than exist. List the equality-bound properties first in an index, and page by a range on its terminal instead. The proof that a create or delete took effect is the presence or absence of its entry in the proof index, an index that involves no $createdAt, does not skip and keeps entries.

Rules at registration:

  • documentsMutable: false; transferable 0; tradeMode 0; no documentsKeepHistory or keeps*History; no transient properties.
  • None of the document type aggregate keywords (documentsCountable and the others): the type has no primary tree. The index keywords of Counts, Sums and Averages and Ranked Indexes are allowed.
  • At least one index. No index is unique or contested, and none sets nullSearchable: false.
  • Every index holds $ownerId, as a property or in its terminal, except a summableOffCountIndex index, which keeps counters instead of entries.
  • The only system properties an index may list are $ownerId and $createdAt, and an indexed $createdAt must be in required.
  • No index declares integerRange: rows that differ only in the bucketed integer would claim the same entry in the windows they share.
  • At least one index involves no $createdAt, does not set skipIfAbsent and keeps entries (it is no summableOffCountIndex index): the proof index.
  • Every property is in required, except a skip property of a skipIfAbsent index. An object holding an indexed property is required too.
  • Every required property appears in at least one index that does not skip and keeps entries, as a property or a terminal component, except the entryPayload properties and a property a summableOffCountIndex index's source fixes. Every optional property appears in a skip index without a timeRange whose skip set is that property alone, except, again, one such a source fixes.
  • The type cannot also set ttl or moderatorAbilities.delete, and no document reference can target it: a refersTo findBy finds no unique index in it, and a reference by id (or with inList) is refused when the referring contract is registered (ReferencedDocumentTypeIndexOnlyError, 40146), since its documents can not be fetched by $id.

A property a summableOffCountIndex index's source fixes may sit in no index that keeps entries: in the summableOffCountIndex example, a like's postAuthor, which the postId reference's where entry "$ownerId": "postAuthor" fixes to the post's owner, sits only in byAuthorPost. No query returns such a property: a proved read gives documents without it, required or not, and since no index of the type then holds every property, every documents read of the type without a proof is refused. Its value is the referenced document's (here the post's $ownerId), so a client building a delete reads it back from there.

entryPayload

Wheredocument type
Valuearray of 1 to 16 distinct property names, each 1 to 64 characters
Defaultabsent
Sinceprotocol version 14
On updateFixed (40212). The list is read as a set, so reordering it is no change.

The properties stored in each entry's value, after the row commitment, instead of in a key. They are for data the application reads but never queries by, such as a public key or a ciphertext: they need not be indexed, and they come back with every query result.

"indices": [{ "name": "byRequest", "terminal": ["appEphemeralPubKeyHash", "$ownerId"] }],
"entryPayload": ["walletEphemeralPubKey", "encryptedPayload"]

With a flat index keyed by a request hash and the responder, this is a key-value table: a query on the hash returns every responder with its public key and ciphertext.

Rules at registration:

  • Only on an indexOnly type.
  • Each entry names a top-level property that is required, a scalar (not an object or an array of values), and bounded: maxLength on a string, maxItems on a byte array.
  • A payload property appears in no index, neither as a property nor in a terminal.
  • The largest size each payload property can take, plus two bytes each, adds up to at most 5120 bytes. With several indexes, the payload is repeated in every entry.

terminal

Whereindex of an indexOnly type
Valuea property name, or an array of 1 to 10 distinct names
Default"$ownerId"
Sinceprotocol version 14
On updateFixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217)

Where an ordinary index keys each entry by the document id, an index-only index keys it by the terminal's values: the member key. There is one entry per index values and member key, so the terminal decides what the index makes unique. byPost above, with the default terminal, allows one like per post and owner; byLiker, with postId as terminal under $ownerId, holds the same pairs the other way round.

An array is a composite terminal whose values are joined in order. An index with no properties at all is a flat index, keyed by its terminal alone, as byRequest above is.

A query that fixes every property of the index can test one member key ("did I like this post") or walk the member keys in order, a page at a time.

Rules at registration:

  • Only on an indexOnly type.
  • Each component is $ownerId or a property of the type that could be indexed: not an object or an array of values, a string with maxLength of at most 63, a byte array with maxItems of at most 255. No other system property.
  • Every component but the last has a fixed width: a byte array with minItems equal to maxItems, an identifier, an integer or a boolean. A string can only be last.
  • The whole member key is at most 255 bytes. On a flat index, the level key, the component names each preceded by a zero byte, is at most 255 bytes as well.
  • A component is not one of the index's properties, and not an optional property.
  • A flat index takes no count, sum, ranking, timeRange, integerRange, skipIfAbsent or preallocated keyword.

preallocated

Whereindex of an indexOnly type
Valueboolean
Defaultfalse
Sinceprotocol version 14
On updateFixed (10217)

The first entry under a new set of values pays for every tree on its path; later ones pay for one entry. When the whole path is decided by a referenced document, preallocated: true creates the trees when that document is created, paid by its creator, so every entry costs the same from the first one on. Deleting the last entry keeps the trees, so a post with no likes still shows in the rankings with a count of zero, and a range count, sum or average grouped by the last property returns it with zero, counted in a page's limit, so without an IN a short page means the range ended (across an IN, a value whose range holds nothing takes a place in the limit too, so there a short page may not be the end). Only documents created once the index exists get the trees: when a contract update adds the referring type, the referenced documents already stored get theirs from their first entry, as without preallocated.

In the example, byHashtagPost could be preallocated: postId is the reference and hashtag agrees with the post's. byLiker could not, since no post decides who likes it.

Rules at registration:

  • Only on an indexOnly type.
  • Every index property is either a property with a permanentDocument or moderatedDocument reference to a type of the same contract, or a referring value of that reference's where. A deletableDocument reference does not qualify, since the trees would outlive a deleted target with nothing left to say what they were keyed by. $ownerId may only be the terminal.
  • Through a moderatedDocument reference, every key of the path must be kept by the referenced document's removal record: each where entry the index uses compares the referenced $id, $ownerId or a property the referenced type lists under moderatorAbilities.deleteKeepsFields (or one inside an object listed there), never $creatorId (InvalidContractStructure, 10231). The trees then outlive a removed document the way its record does, and a moderator's restore finds them in place. As through a permanentDocument reference, they are keyed by the values the document was created with: a value changed afterwards leaves them empty, and the first entry under the new value builds its own.
  • The referenced property of each such where entry holds at most 255 bytes, since creating a referenced document makes its value an index key (40126 when the contract is created or updated).
  • Not with timeRange or integerRange.

A referenced document whose agreed value takes more bytes than the referring property can hold preallocates nothing for that index, since no entry could agree with it.

summableOffCountIndex

Whereindex of an indexOnly type
Valuethe name of another index of the type, its source
Defaultnone
Sinceprotocol version 14
On updateFixed (10217)

A like counted by post, by author and by hashtag is written three times: once in byPost and once in each of the other two, which hold nothing byPost does not already hold. When every like of a post lands in the same author and the same hashtag, the other two only need to know how many likes each post has. An index with summableOffCountIndex keeps exactly that: one counter per group, holding the number of entries its source index keeps for it, in place of an entry per document. A like then adds one to two counters instead of writing two more entries.

Here the like of the example also carries its post's author, fixed through the reference's where like its hashtag:

"properties": {
  "postId": {
    "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
    "contentMediaType": "application/x.dash.dpp.identifier",
    "refersTo": {
      "type": "permanentDocument",
      "documentType": "post",
      "where": { "hashtag": "hashtag", "$ownerId": "postAuthor" }
    },
    "position": 0
  },
  "hashtag": { "type": "string", "minLength": 1, "maxLength": 59, "position": 1 },
  "postAuthor": {
    "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
    "contentMediaType": "application/x.dash.dpp.identifier",
    "position": 2
  }
},
"required": ["postId", "postAuthor"]

and counts its likes per author and post:

{
  "name": "byAuthorPost",
  "properties": [{ "postAuthor": "asc" }, { "postId": "asc" }],
  "summableOffCountIndex": "byPost",
  "rangeCountable": true,
  "rangeSummable": true,
  "rankedSummable": { "at": ["postAuthor", "postId"] },
  "rankedAverageable": { "at": ["postAuthor"] },
  "preallocated": true
}

Each counter adds its entries to the sums of the tree of the last property and, with rangeCountable, counts one group in its counts. A count query counts documents, so on this index it reads the sums, which hold the source's entries:

QueryReadsbyAuthorPost
count(*)the sumsan author's likes, or one post's
sum(byPost), named by the source indexthe sumsthe same
avg(byPost)the sums and the group countsan author's posts and likes, so likes per post

A ranking at an earlier level orders by these totals: rankedSummable: { "at": "postAuthor" } ranks authors by likes, rankedAverageable: { "at": "postAuthor" } by likes per post, and "postId", the last property, ranks an author's posts by likes. A count or sum point query may stop at any level from the shallowest sum or average ranking down, and an average point query at any level that also carries counts (from the shallowest average ranking down). Pinned on every property but the last, any of them reads the tree of the last property instead when that property is not ranked: postAuthor == A alone gives A's likes, and with rangeCountable A's posts for an average. A ranked or HAVING count reads the sum rankings the same way: count(*) grouped by postAuthor ranks authors by likes. So on this index rankedCountable declares the same ranking as rankedSummable, and needs no rangeCountable: rankedCountable: { "at": "postAuthor" } ranks authors by likes too. A range count reads the sums over the range too: count(*) with postAuthor == A and a range on postId, grouped by postId, gives each of A's posts in the range with its likes (on a preallocated index a post nobody liked comes back with zero, and takes its place in a page's limit), and across an IN on postAuthor each author's. A range total (no grouping, or one per IN value) needs an index whose path passes through no ranked level, as everywhere: grovedb proves a range total only through unranked trees, and the unproven read refuses one too so both agree, so on this index, which ranks postAuthor and postId, group by postId instead. When the index is preallocated, every post created once the index exists has a counter from its creation, so a post without likes counts as a post with zero likes (a post stored before a contract update added the like type gets its counter from its first like, and keeps it at zero after); otherwise a post shows once it is liked and leaves with its last like.

Rules at registration:

  • Only on an indexOnly type, with rangeSummable: true (the counters sit in the tree of the last property, which only rangeSummable makes a sum tree). No summable, averageable, terminal, countable: "countableAllowingOffset", timeRange, integerRange, outlivesDelete, unique or contested.
  • The source is another index of the type holding every document exactly once: it keeps entries (it is no summableOffCountIndex index itself), skips no document (skipIfAbsent), keeps no deleted one (outlivesDelete) and involves no $createdAt.
  • One summed value per type: no index of the type declares summable, every summableOffCountIndex index names the same source, and no property shares the source's name.
  • Every property of the source is a property of the index, so a group never counts two source groups.
  • Every other property is a referring value of a where on a permanentDocument or moderatedDocument reference by id (no findBy or inList, whose key could move to another document) to a type of the same contract, held by a property of the source, and the referenced value never changes once written: $id, $creatorId, an $ownerId no transfer or trade changes, or a property the referenced type never lets change (a where names only the referenced type's properties, $id, $creatorId and $ownerId). An immutable deletableDocument reference counts only when it is required: a replace may clear an optional one once its document is deleted. Through a moderatedDocument reference, the value must also stay on the removal record (deleteKeepsFields). Every like of one post then lands in one group.
  • No other index continues below the index's last property, where the counter stands.
  • rankedSummable and rankedAverageable take the { "at": ... } form only on such an index.

A broken rule is refused as InvalidContractStructure (10231), or by the meta-schema as JsonSchemaError (10101): the meta-schema refuses the keyword without rangeSummable: true or next to summable, averageable or terminal, the { "at": ... } form of rankedSummable or rankedAverageable without it, and a source name longer than 32 characters.

skipIfAbsent

Whereindex
Valuetrue, or an array of the index's property names
Defaultfalse
Sinceprotocol version 14
On updateFixed (10217)

The keyword is described in Indexes: a document that leaves out a property of the index's skip set writes nothing into the index, and its delete looks for nothing there. It is the one way a property of an index-only type can be optional: in the example, a like without a hashtag is not in byHashtagPost and pays nothing for it. A skip property may sit below the first position, which is what lets a windowed ranking skip it:

{
  "name": "byDayHashtagPost",
  "properties": [{ "$createdAt": "asc" }, { "hashtag": "asc" }, { "postId": "asc" }],
  "terminal": "$ownerId",
  "rangeCountable": true,
  "rankedCountable": { "at": ["hashtag", "postId"] },
  "timeRange": { "on": "$createdAt", "range": 86400, "step": 86400, "ttl": 604800 },
  "skipIfAbsent": true
}

An untagged like still enters the type's other indexes over the same day window, but writes nothing under hashtag in it.

What an index-only type adds to the rules of every type:

  • An index path has no representation for a missing value, so every index holding an optional property skips on it: the skip set is every optional property of the index, and an array must name them all.
  • Each optional property needs a skip index without a timeRange whose skip set is that property alone. A document carrying one optional property but missing another skips every index holding both, and its value would otherwise be written nowhere; a windowed index keeps it only until its windows drain, where document queries do not read it.
  • An optional property is never a terminal.
  • At least one index that involves no $createdAt does not skip: the proof index.

outlivesDelete

WheretimeRange index of an indexOnly type
Valueboolean
Defaultfalse
Sinceprotocol version 14
On updateFixed (10217)

A delete of an index-only document carries its values and removes the entries they address. A time window is keyed by the document's $createdAt, so without this keyword a delete must carry the exact timestamp to find the window's entries. With outlivesDelete: true, a delete leaves the index's entries where they are, and they expire with their window:

{
  "name": "byTrendPost",
  "properties": [{ "$createdAt": "asc" }, { "postId": "asc" }],
  "terminal": "$ownerId",
  "countable": "countable",
  "timeRange": { "on": "$createdAt", "range": 259200, "step": 86400, "ttl": 604800 },
  "outlivesDelete": true
}
  • A delete carries no $createdAt when every index involving it outlives deletes: the rows commit to no timestamp, and a delete that carries one is refused (InvalidDocumentTransitionActionError). An unlike needs only the like's other values, which any device knows.
  • A create writes over an entry already there. When the same owner writes the same values again while a deleted document's entry still stands in a window, the create writes its own entry over it: the owner counts once, and the create is not refused as a duplicate. Only the index's other entries decide whether a create is a duplicate.
  • A deleted document keeps counting in the window until the window moves past it. On a trending window, an unliked like still counts there for up to the window's range.
  • The executed-transition proof runs against an index that does not outlive deletes, so a delete still proves the document gone.

Rules at registration:

  • Only on an indexOnly type, on an index with a timeRange carrying a ttl, so the entries a delete leaves expire.
  • Not with a sum (summable), and not on a type with entryPayload: a kept entry holds the amount or payload of the document that wrote it.
  • Every schema property must also sit in an index that neither skips nor outlives deletes, which a delete checks.
  • The index's key (its properties but $createdAt, and its terminal) must hold the whole key of an index a delete clears and that skips nothing, so no two documents in state share one of its entries. [$createdAt, postId] → $ownerId holds byPost's [postId] → $ownerId; [$createdAt] → $ownerId holds no such key and is refused.
  • The proof index may not outlive deletes.

See also

Values of Referenced Documents

An index may hold a value that the document does not store, read from the document one of its references points at. A reply indexed by postId.$ownerId is filed under the owner of the post it replies to, so "every reply to my posts" is one query, and the reply never stores that owner. Drive reads the value from the post whenever it writes or removes the reply's index entries. This saves storing a copy of the value in every document (32 bytes for an identifier). In exchange, every write that moves the entries reads the referenced document again.

Example

A reply to a post only the moderators take down:

"post": {
  "type": "object",
  "canBeDeleted": false,
  "moderatorAbilities": { "delete": true },
  "properties": {
    "text": { "type": "string", "minLength": 1, "maxLength": 280, "position": 0 }
  },
  "required": ["text"],
  "additionalProperties": false
},
"reply": {
  "type": "object",
  "documentsMutable": true,
  "canBeDeleted": true,
  "immutable": ["postId"],
  "indices": [
    { "name": "toPostOwner", "properties": [{ "postId.$ownerId": "asc" }, { "$createdAt": "asc" }] }
  ],
  "properties": {
    "postId": {
      "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
      "contentMediaType": "application/x.dash.dpp.identifier",
      "refersTo": { "type": "moderatedDocument", "documentType": "post" },
      "position": 0
    },
    "body": { "type": "string", "minLength": 1, "maxLength": 280, "position": 1 }
  },
  "required": ["postId", "body", "$createdAt"],
  "additionalProperties": false
}

A reply is created with its own properties only, postId and body. Its author may edit body but never repoint postId. A query for the replies to one identity's posts, newest first:

{
  "where": [["postId.$ownerId", "==", "<identity id>"]],
  "orderBy": [["$createdAt", "desc"]]
}

The replies come back as stored, without the owner: the proof ties them to it through the index path.

A post whose removal record keeps a field can have its replies filed under that field too. With "documentsMutable": false and "moderatorAbilities": { "delete": true, "deleteKeepsFields": ["hashtag"] } on the post, a reply index on postId.hashtag files every reply under the hashtag of the post it replies to, before and after a moderator removes the post.

The index property

Wherea property of an index of a document type that is not indexOnly
Value"<reference property>.<field>"
Sinceprotocol version 14
On updateFixed, as every index is (10217)
ErrorsInvalidContractStructure (10231); for a field an index can not key, InvalidIndexPropertyTypeError (10206) or InvalidIndexedPropertyConstraintError (10205)

The name reads through a reference when its first segment names a top-level identifier property that declares a refersTo. An identifier has no nested properties, so the name can not mean a nested property. The field is what follows the first .:

  • $ownerId: the owner of the referenced document.
  • $creatorId: the identity that created the referenced document, on a type that records it (see System Properties).
  • A schema property of the referenced document type, by its dotted path, with that property's type.

What Drive does

  • Create. The reference is validated as usual, which reads the referenced document, and Drive writes the entries under the values it holds, without reading it again. A referenced document that has no value for the field, or a reference left out, puts the document under null, as a missing property would, or leaves it out of an index that skips on the property.
  • Replace, transfer, purchase, price update and a moderator's change of fields. Drive reads the referenced document once, for both versions of the document, since neither can point elsewhere. It reads it even when no index holding a derived value moves: every entry's reference is rewritten in place on an update, under the values it is filed by.
  • Delete, a moderator's removal, and expiry by ttl. Drive reads the referenced document to find the entries to remove.
  • A removed referenced document. Once a moderator removes a moderatedDocument target, Drive reads its owner, and any field its type keeps under deleteKeepsFields, from the removal record, which keeps them as the document held them. So the entries are found under the values they were written under. A restored document is read again.

Each read is billed as a processing fee with the write that makes it. A dry run prices the reads at their worst case and reads nothing, pricing the entries under a value of the field's type and typical size.

Rules at registration

A derived value must stay what it was when an entry was written, or Drive could not find the entry again. So:

  • The reference is a permanentDocument or a moderatedDocument one, by id, to a document type of the same contract. A deletableDocument target could leave state without a record, and a findBy, an inList or an operand of an expression does not name the document by id.
  • The reference property is fixed once written: the type's documents are immutable, or the property is listed under immutable without a condition, and it is not one of the fields only moderators write.
  • Through a moderatedDocument reference, the field is $ownerId, or a schema property the referenced type lists under moderatorAbilities.deleteKeepsFields, or one inside an object listed there: the removal record keeps these, and Drive reads them from it once the document is removed. Not $creatorId, which no record keeps. The list is fixed when the referenced type is registered, so an update adding a referring type can read only what it already lists.
  • $ownerId only of a type whose documents can not be transferred or traded, and $creatorId only of a type that records it.
  • A schema property must exist on the referenced type, be stored (not transient or inside a transient object), be fixed once written there under the same rule as the reference property, and be one an index can key: not an object or an array, a string of at most 63 characters, a byte array of at most 255 bytes, tighter on a ranked index.
  • Not $id of the referenced document, which is the reference property itself: index that instead. Nor any other system property.
  • The index is not unique or contested: a uniqueness check reads the values the create carries, and a derived value is not among them.
  • The derived property is not the source of a timeRange or an integerRange.
  • The type is not indexOnly: its entries hold every value of its documents, and its delete carries them.

Skipping a document without the value

A derived property can be a skip property of skipIfAbsent. The index then leaves out a document whose reference is absent, or whose referenced document has no value for the field, instead of filing it under null. A reply that may quote a post, filed by the owner of the post it quotes:

{
  "name": "byQuotedOwner",
  "properties": [{ "quoteId.$ownerId": "asc" }, { "$createdAt": "asc" }],
  "skipIfAbsent": ["quoteId.$ownerId"]
}

A reply quoting no post writes nothing into this index, and its delete looks for nothing there. Since neither the reference nor the field can change once written, a replace never moves a document into or out of such an index through its derived values.

Only the array form names a derived property. skipIfAbsent: true skips on the type's own optional properties: whether a derived value can be absent depends on the referenced type.

At registration, a derived skip property must be able to be absent: its reference property is not in required, or the field is a schema property the referenced type does not require, or one inside an object it does not require. $ownerId and $creatorId are absent only with the reference. A byte array the property reads must set minItems to at least 1 on the referenced type, since an empty one is keyed like a missing value. Every other skipIfAbsent rule applies as to any skip property.

Queries

A query uses a derived property as any other index property: ==, in, a range, orderBy. Its values encode as the field's type.

A startAt or startAfter cursor places the page by the values the document it names stores, and a derived value is not among them. A query paging with a cursor must therefore fix every derived property of its index with ==. Otherwise it is refused, and it can page by a range on another property of the index instead.

A subscription to state transitions filters by the values a transition carries, and a derived value is not among them either, so a subscription filter can not name a derived property. Filter by the reference property instead (postId), or query the index.

See also

Contract-Level Keys and config

The document types sit inside a data contract, which has keys of its own: its id, its owner, its version, the shared definitions its types may point at, its tokens and groups, and a search description. One of them, config, holds contract-wide settings: whether the contract can ever change, whether it keeps its history, the defaults its document types inherit, how integers are stored, and whether and how it is moderated. Almost every config setting is chosen once, at registration, and kept for the contract's life.

Example

{
  "$formatVersion": "1",
  "id": "AY6xWncZUFv2GCrS5seqKthUfbW9yYyUXtF8diSuHQ4g",
  "ownerId": "AtirhSVpAWF7dEt6dLAmesC4Sr1MsJ9bFC1nLAoNnq2S",
  "version": 1,
  "config": {
    "$formatVersion": "2",
    "canBeDeleted": false,
    "readonly": false,
    "keepsHistory": false,
    "documentsKeepHistoryContractDefault": false,
    "documentsMutableContractDefault": true,
    "documentsCanBeDeletedContractDefault": true,
    "sizedIntegerTypes": true,
    "moderation": {
      "banlist": true,
      "suspensions": true,
      "moderators": { "$type": "contractOwner" }
    }
  },
  "documentSchemas": {
    "post": {
      "type": "object",
      "properties": {
        "text": { "type": "string", "minLength": 1, "maxLength": 280, "position": 0 }
      },
      "required": ["text"],
      "additionalProperties": false
    }
  },
  "keywords": ["social", "microblog"],
  "description": "Short public posts"
}

A first version of a small social contract. Its config states the defaults explicitly, stores integers in the smallest width their bounds allow, and keeps a banlist and a suspension list that the owner edits. keywords and description make it findable through the keyword search contract.

Contract keys

KeyValueWhat it isSince
$formatVersion"0" or "1"The contract's serialization format. Format 1 is the default from protocol version 9 and is the one that carries groups, tokens, keywords, description and the timestamps.1
ididentifierThe contract's id: a hash of ownerId and the identity nonce of the create transition. A create whose id is not that hash is refused (InvalidDataContractIdError, 10204).1
ownerIdidentifierThe identity that registers the contract, and the only one that can update it.1
versioninteger1 when the contract is created; each update must raise it by exactly one (InvalidDataContractVersionError, 10212).1
configobjectContract-wide settings: see config below. Absent means the defaults.1
documentSchemasobjectThe document types, by name. See documentSchemas.1
schemaDefsobjectDefinitions every document type may point at with $ref. An update may add definitions, not remove them (IncompatibleDataContractSchemaError, 10213).1
groupsobjectGroups of identities that act together, each member with a voting power, whose approval some token actions need. See Data Contracts. Not the same thing as a contract group, a set of contracts.9
tokensobjectThe contract's tokens, keyed by position 0, 1, and so on. Document types may charge them with tokenCost.9
keywordsarray of stringsSearch keywords. See keywords and description.9
descriptionstringA short description for search. See keywords and description.9
createdAt, updatedAt, createdAtBlockHeight, updatedAtBlockHeight, createdAtEpoch, updatedAtEpochnumbersWhen the contract was created and last updated. The platform sets them; a contract does not write them.9

documentSchemas

Wherecontract
Valueobject mapping each document type name to its schema
Sinceprotocol version 1
On updateDocument types may be added; none may be removed (DocumentTypeUpdateError, 40212). Each existing type follows the update rules of its keywords.
ErrorsDocumentTypesAreMissingError (10214), InvalidDocumentTypeNameError (10415), at registration

A contract has at least one document type, unless it defines tokens (DocumentTypesAreMissingError, 10214). A name is 1 to 64 ASCII letters, digits, _ or -; from protocol version 14 a name may not contain - (InvalidDocumentTypeNameError, 10415). The keywords a schema takes are the subject of the rest of this part: see Contract Keywords.

keywords and description

Wherecontract
Valuekeywords: array of at most 50 strings; description: string
Defaultno keywords, no description
Sinceprotocol version 9
On updateMay be changed; the search entries are replaced
ErrorsTooManyKeywordsError (10262), InvalidKeywordLengthError (10270), InvalidKeywordCharacterError (10269), DuplicateKeywordsError (10263), InvalidDescriptionLengthError (10264)

The keyword search system contract indexes each contract by its keywords and description, so applications can find contracts by topic. The rules, checked on every create and update:

  • At most 50 keywords (10262).
  • Each keyword is 3 to 50 bytes of UTF-8 (10270) and contains no whitespace or control character (10269).
  • No keyword appears twice (10263).
  • A description is 3 to 100 bytes of UTF-8 (10264).

config

Wherecontract
Valueobject: $formatVersion and the keys below
Defaultabsent: every key takes its default
Sinceprotocol version 1
On updateThe keys below are fixed, with the exceptions each one names (DataContractConfigUpdateError, 40002)
ErrorsDataContractConfigUpdateError (40002), DataContractIsReadonlyError (40001)

$formatVersion is the config's own version: "0" before protocol version 9, "1" from 9, and "2" from 14, which adds moderation. From protocol version 14 every new contract carries config version 2, moderated or not, and an older contract moves to it with its next update.

canBeDeleted

Whereconfig
Valueboolean
Defaultfalse
Sinceprotocol version 1
On updateFixed (40002)

Whether the contract itself may ever be deleted. No transition deletes a contract today, so the flag has no effect yet beyond being recorded.

readonly

Whereconfig
Valueboolean
Defaultfalse
Sinceprotocol version 1
On updateCannot be set by an update (40002)
ErrorsDataContractIsReadonlyError (40001)

true freezes the contract at its first version: every update of it is refused with DataContractIsReadonlyError (40001). Only a create can set it, so a contract is read-only from the start or never. A document reference can require its target contract to be read-only (contractRequirements.readonly, see References).

keepsHistory

Whereconfig
Valueboolean
Defaultfalse
Sinceprotocol version 1
On updateFixed (40002)

true makes Drive keep every version of the contract, not only the latest, so a client can read and prove the contract as it was at an earlier version.

Document type defaults

Whereconfig
ValuedocumentsKeepHistoryContractDefault, documentsMutableContractDefault, documentsCanBeDeletedContractDefault: boolean each
Defaultfalse, true, true
Sinceprotocol version 1
On updateFixed (40002)

The value of documentsKeepHistory, documentsMutable and canBeDeleted for each document type that does not set it itself. A type that sets the keyword overrides the default. Changing a default would change every type that relies on it, so all three are fixed. See History, Mutability and Deletion.

Bounded key requirements

Whereconfig
ValuerequiresIdentityEncryptionBoundedKey, requiresIdentityDecryptionBoundedKey: 0 unique, 1 multiple, 2 multiple with a pointer to the latest
Defaultabsent
Sinceprotocol version 1
On updateFixed (40002)

Let identities add encryption, or decryption, keys bound to the whole contract, and say how they are kept: one key that cannot be replaced, several, or several with a pointer to the latest. The document type keywords of the same names do this for keys bound to one document type. See Signing and Keys and Contract Bounds.

sizedIntegerTypes

Whereconfig
Valueboolean
Defaulttrue in config version 1 and later; config version 0 has no such key and behaves as false
Sinceprotocol version 9
On updateMay be turned on, not off (40002)

With true, each integer property is stored in the smallest width its minimum, maximum or enum allow: "minimum": 0, "maximum": 100 takes one byte. With false, every integer is a signed 8-byte value. Turning it off would make stored documents unreadable, so it is refused. Turning it on is allowed by the config check, but it changes the width of every bounded integer of the existing document types, and an update that changes how an existing property's values are stored is refused (DocumentTypeUpdateError, 40212); in practice it can only be turned on when it leaves the width of every existing integer property unchanged.

The width also decides what a summed property may be: see Counts, Sums and Averages.

moderation

Whereconfig (config version 2)
Valueobject: banlist, suspensions, warnings (booleans, default false) and moderators
Defaultabsent: the contract is not moderated
Sinceprotocol version 14
On updateWhich lists are kept is fixed, and an elected team can be neither declared, changed nor left; otherwise the moderators may change (40002)
ErrorsInvalidContractModerationConfigError (10900), ContractModeratorIdentityNotFoundError (41110)

Declares which moderation lists the contract keeps and who edits them. An identity on the banlist, or suspended, cannot act on the contract's documents; a warning is a record that bars nothing. moderators is one of:

  • { "$type": "contractOwner" }: the owner moderates alone.
  • { "$type": "appointedModerators", "identities": [...] }: the owner and 1 to 16 named identities, each acting alone. Every named identity must exist (ContractModeratorIdentityNotFoundError, 41110).
  • { "$type": "elected", ... }: a team elected by masternodes moderates, with the abilities the declaration gives it. See Elected Moderation.

A declaration keeps at least one list, unless a document type gives its moderators an ability (moderatorAbilities: deleting its documents or writing the fields it keeps for them), and is refused otherwise (InvalidContractModerationConfigError, 10900). An unknown key is refused rather than ignored, so a misspelled list name cannot silently leave the contract without it. Because the lists are fixed, a contract that will ever need moderation declares it when it is created.

moderation is what moderatorAbilities and the moderators' share of actionFees require.

See also

The GroveDB Structure

Drive keeps everything in one GroveDB: a tree of trees. The root layer holds one subtree per RootTree variant, each of those holds fixed keys or one subtree per identity, contract or token, and so on down to the items. The other chapters of this section draw the part they are about. This chapter is about the whole, and about the two things that keep a picture of it honest: a description written as code, and a viewer that reads it.

Open the structure viewer

Dive into a tree to go a layer down, use the breadcrumb or Backspace to come back up, drag the protocol version at the bottom to see what each version added, and switch a layer to Merk tree to see its real binary tree.

The description is code

The structure is declared in Rust, in drive::structure (packages/rs-drive/src/structure). Each area declares its own part in a structure.rs beside its paths.rs, from the same constants the paths use:

#![allow(unused)]
fn main() {
StructureNode::fixed(
    "balances",
    &[TOKEN_BALANCES_KEY],
    "TokenBalances",
    "TOKEN_BALANCES_KEY",
)
.kind(ElementKind::BigSumTree)
.since(9)
.describe("Who holds how much of each token.")
.child(
    token("The balances of one token; the sum is its circulating supply.")
        .kind(ElementKind::SumTree)
        .child(
            StructureNode::identifier("identity", "identity_id", "The holder's identity id")
                .kind(ElementKind::SumItem)
                .value("token amount")
                .describe("One identity's balance of the token."),
        ),
)
}

A node says:

  • its key: fixed bytes and the constant they come from, or a template such as "identity id, 32 bytes" standing for many keys;
  • the element kinds that can sit there. Several when the code chooses, as the primary key tree of a document type does between eight tree kinds;
  • the element flags on it. GroveDB stores a byte string of flags with every element and never reads it; Drive keeps its storage flags there: the epoch the element's bytes were paid for in, the bytes added in later epochs if it grew, and for owned flags the identity that paid, which is who a refund goes to when the element is deleted or shrinks. A node says whether its element carries none, storage flags, or storage flags with an owner, and who that owner is. The exported file explains each kind and its byte layout once, under flag_kinds;
  • since, the first protocol version it exists in, which children inherit;
  • whether it is created with its parent, lazily on first use, or with its parent and deleted later while the parent stays, as an epoch's storage fee item is at payout;
  • what an item holds, what a reference points to, and recurse where the structure repeats to any depth, as index levels do;
  • its source file and, when there is one, its chapter in this book.

The identifier of a node is the dotted path of its segments (tokens.balances.token.identity). Identifiers are what two versions of the structure are compared by, so treat them as append-only: renaming one reads as a removal and an addition.

The module is compiled for tests and under the structure feature only. Nothing in the node depends on it.

What keeps it true

The tests in packages/rs-drive/src/structure/tests.rs stand between the description and drift. A lint first checks the description against itself: identifiers are unique, reference and recurse targets exist, every source file exists and names the constant a key claims to come from, and no two templates of a layer could claim the same element.

Conformance. check_conformance walks a real GroveDB layer by layer and reports every element no node describes, every element of a kind its node does not list, every element whose flags are of a kind its node does not list, every element outside its node's protocol versions, and every node that should have been created with its parent but is missing. Where several templates of a layer accept a key, the one whose description fits what is below the element wins: below a contested index a 32 byte key is a contender's identity id at the last level and an index value before it, and the key alone cannot tell. It runs against the initial state structure of every protocol version, which pins each since to what create_initial_state_structure really builds, and against populated state: identities, contracts with documents and history, tokens, group actions, address balances, an epoch before and after payout, contested documents. A change that adds a root tree, a subtree key or a level fails here until it is described.

The strategy tests. The drive-abci strategy tests check the state of every chain they run with the same walker (assert_state_conforms_to_structure), at whatever protocol version the chain ended on. They write far more than the fixtures do: votes, withdrawals, token distributions, epochs changing, protocol upgrades. Whatever a change's own strategy tests write is walked, so structure created only during some operation is caught too.

Coverage. Every node must be reached by some rs-drive fixture or be listed as reached by the strategy tests. UNVERIFIED, the list for nodes written from reading the code and never checked against a real GroveDB, is empty and meant to stay so: whoever describes a node can write a fixture that creates it. The test also fails when a listed node does get reached by a fixture, so the lists can only shrink.

The exported file. The description is serialized to packages/rs-drive/grovedb-structure.json, which is what the viewer reads. A test fails when the committed file is stale:

UPDATE_GROVEDB_STRUCTURE=1 cargo test -p drive --lib structure::tests

The shape of a layer

GroveDB stores each layer as a Merk, a balanced binary tree, and a proof of one element carries the hashes along the way from the layer's root. So where a key sits matters twice: keys near the top have shorter proofs, and every write below a node rewrites its ancestors in the layer. That is why the root keys are spread over the byte range with DataContractDocuments on top, and why ContractGroups took key 124, which hangs below Versions, a tree only the block-level version bookkeeping writes to (see Contract Groups).

The exported file records the exact Merk of every layer whose keys are all fixed, under layer_shapes. GroveDB does not expose the links between the nodes of a Merk, but a proof does: the proof of a query for everything in a layer lists every node and how they connect, so replaying its operations rebuilds the tree. The viewer draws it when you switch a layer to Merk tree.

The shape depends on the order of insertion, not only on the keys. For a layer reached through fixed keys only, the recorded shape is that of a fresh chain at the latest protocol version ("origin": "genesis@14"). A chain that upgraded through earlier versions inserted the same keys in another order and can differ.

A layer below a template exists once per identity, contract, epoch and so on. Its shape is recorded from one instance: the fullest one the test fixtures build ("origin": "fixture contracts_with_documents@14"). That is where layouts designed around the Merk show: a contract's layer keeps its documents on top with the contract and everything else below, the other tree keeps the banlist in the middle, and an identity's seven keys form a full tree with the keys at the root. Another instance can differ when it holds fewer keys or got them in another order.

Some layers gain and lose keys over their life, and then one shape is not enough. An epoch's layer is created at genesis holding only its storage fees; its first block adds the start fields, the proposers tree and the processing fees in one batch; and the batch that pays it out deletes the proposers and both fee items and writes the finished epoch info. No epoch ever holds all nine keys. A node declares such states in the order the layer goes through them, each with a title, what it means and what moves the layer into it, and the keys it holds then:

#![allow(unused)]
fn main() {
.state(
    "paid",
    "Paid out",
    "One batch pays the proposers, deletes the proposers tree and both fee \
     items, and writes the finished epoch info.",
    &["start_block_core_height", "finished_epoch_info", "start_block_height", ...],
)
}

The fixtures record one shape per state, and a state's shape must come from an instance holding exactly the keys the state declares. The viewer shows them as State 0, State 1, ... with their explanations. What matters for a faithful shape is that the fixture writes the same keys in the same batches as the block pipeline does, since a Merk batch of several keys gives another tree than the same keys written one at a time.

One document type's layout

The description covers every document type at once, so its document layers are templates: an index property, one of its values, the [0] where an index ends, and which tree types each of them can be. Which of those a given document type gets depends on its keywords (unique, countable, summable, the ranked keys, timeRange, indexOnly, documentsKeepHistory, ...), and the rules that pick them live in the index walkers.

drive::document::layout::document_type_layout applies those rules to one document type and returns its concrete layout: the document type tree, the documents by id, and for each index the property and value trees down to where it ends, each with the tree or element type Drive writes, the zero-contribution wrapper a continuation tree gets under an aggregating value tree, the ranking axes of an indexed tree, the indexes that use the layer, and conditions such as the tree a unique index falls back to when a value is null. Each layer names the node of the description it is an instance of, so a viewer can link to it. It needs only the contract, so it is compiled with the verify feature as well, and the JavaScript SDKs expose it as documentTypeLayout(contract, documentTypeName, platformVersion). It follows the v2 index walkers, so it refuses a platform version before 14.

The tree types come from the functions the walkers call, and the element choices the walkers make inline are held to them by should_lay_out_what_drive_writes. It applies 19 test contracts (plain, unique, compound, countable, summable, ranked and chained indexes, history, time windows, indexOnly types with flat and preallocated indexes), inserts documents, and fails on any element whose kind or wrapper the layout does not predict, and on any layer of the layout Drive did not write. It does not cover contested indexes, time windows with a ttl, or document updates and deletes.

Pull requests that change the structure

When a pull request changes grovedb-structure.json, the GroveDB Structure Preview workflow comments with a link that opens the viewer on the difference between the merge base and the head of the pull request: new nodes glow, removed ones stay as ghosts, every ancestor of a change carries a count so the trail is visible from the root, and a tour walks through each change. The viewer fetches both files from GitHub and compares them itself, so the link works before the merge and for forks.

To add structure, follow the checklist in the coding conventions.

Grove Operations

Drive is the storage layer of Dash Platform, and GroveDB is the authenticated data structure (a Merkle tree of trees) that Drive uses under the hood. But Drive never talks to GroveDB directly in its business logic. Instead, every single GroveDB call is wrapped in a versioned Drive method that follows a consistent pattern. This chapter explains why that wrapper layer exists and how it works.

The Problem: Raw GroveDB Is Too Low-Level

If you were to call GroveDB directly throughout Drive's codebase, you would face several problems:

  1. No cost tracking. GroveDB operations return a CostContext that wraps both the result and an OperationCost. If you forget to capture that cost, the fee system breaks.
  2. No version dispatch. Different protocol versions might need different behavior for the same logical operation (like how to handle estimated costs vs. actual costs).
  3. No consistent API. Each caller would need to handle cost capture, error conversion, and version checking independently.

The grove operations layer solves all three problems by providing a single, consistent abstraction. Every grove operation is a method on Drive that takes a path, a key, some type information, a transaction, and the mutable drive_operations accumulator.

The Module Structure

The grove operations live in packages/rs-drive/src/util/grove_operations/. Each operation is its own submodule:

grove_operations/
    mod.rs                    -- shared types and helpers
    grove_insert/
        mod.rs                -- version dispatcher
        v0/mod.rs             -- v0 implementation
    grove_get_raw/
        mod.rs                -- version dispatcher
        v0/mod.rs             -- v0 implementation
    grove_delete/
    grove_get/
    grove_get_raw_optional/
    grove_has_raw/
    batch_insert/
    batch_insert_empty_tree/
    batch_delete/
    ... (30+ more)

Each submodule follows the same structure: a mod.rs that dispatches on the version, and a v0/mod.rs (and potentially v1/, v2/, etc.) with the actual implementation.

The drive_operations Accumulator Pattern

This is the most important pattern to understand. Almost every grove operation method accepts a mutable reference to a Vec<LowLevelDriveOperation>:

#![allow(unused)]
fn main() {
drive_operations: &mut Vec<LowLevelDriveOperation>
}

Instead of returning costs directly, the method pushes the cost of its GroveDB call onto this vector. The caller passes the same vector through multiple operations, accumulating all costs. Later, the batch application system processes this vector to calculate the total fee.

Why accumulate rather than execute immediately? Two reasons:

  1. Fee estimation. When apply is false, Drive needs to estimate costs without actually writing to the database. The operations still accumulate cost information, but no state changes occur.
  2. Atomic batching. Multiple operations can be collected and then applied as a single atomic batch. More on this in the Batch Operations chapter.

A Concrete Example: grove_get_raw

Let us trace through a complete grove operation. The version dispatcher is in packages/rs-drive/src/util/grove_operations/grove_get_raw/mod.rs:

#![allow(unused)]
fn main() {
impl Drive {
    pub fn grove_get_raw<B: AsRef<[u8]>>(
        &self,
        path: SubtreePath<'_, B>,
        key: &[u8],
        direct_query_type: DirectQueryType,
        transaction: TransactionArg,
        drive_operations: &mut Vec<LowLevelDriveOperation>,
        drive_version: &DriveVersion,
    ) -> Result<Option<Element>, Error> {
        match drive_version.grove_methods.basic.grove_get_raw {
            0 => self.grove_get_raw_v0(
                path, key, direct_query_type,
                transaction, drive_operations, drive_version,
            ),
            version => Err(Error::Drive(DriveError::UnknownVersionMismatch {
                method: "grove_get_raw".to_string(),
                known_versions: vec![0],
                received: version,
            })),
        }
    }
}
}

The dispatcher consults drive_version.grove_methods.basic.grove_get_raw to determine which implementation version to call. If the version is unknown, it returns an error immediately.

Now the v0 implementation, from grove_get_raw/v0/mod.rs:

#![allow(unused)]
fn main() {
impl Drive {
    pub(super) fn grove_get_raw_v0<B: AsRef<[u8]>>(
        &self,
        path: SubtreePath<'_, B>,
        key: &[u8],
        direct_query_type: DirectQueryType,
        transaction: TransactionArg,
        drive_operations: &mut Vec<LowLevelDriveOperation>,
        drive_version: &DriveVersion,
    ) -> Result<Option<Element>, Error> {
        match direct_query_type {
            DirectQueryType::StatelessDirectQuery {
                in_tree_type,
                query_target,
            } => {
                let key_info_path = KeyInfoPath::from_known_owned_path(path.to_vec());
                let key_info = KeyInfo::KnownKey(key.to_vec());
                let cost = match query_target {
                    QueryTarget::QueryTargetTree(flags_size, tree_type) => {
                        GroveDb::average_case_for_get_tree(
                            &key_info_path, &key_info, flags_size,
                            tree_type, in_tree_type,
                            &drive_version.grove_version,
                        )
                    }
                    QueryTarget::QueryTargetValue(estimated_value_size) => {
                        GroveDb::average_case_for_get_raw(
                            &key_info_path, &key_info,
                            estimated_value_size, in_tree_type,
                            &drive_version.grove_version,
                        )
                    }
                }?;
                drive_operations.push(CalculatedCostOperation(cost));
                Ok(None) // No actual data -- just cost estimation
            }
            DirectQueryType::StatefulDirectQuery => {
                let CostContext { value, cost } = self.grove.get_raw(
                    path, key, transaction,
                    &drive_version.grove_version,
                );
                drive_operations.push(CalculatedCostOperation(cost));
                Ok(Some(value.map_err(Error::from)?))
            }
        }
    }
}
}

This reveals the dual nature of every grove operation: it can operate in stateless mode (for cost estimation) or stateful mode (for actual execution). In stateless mode, it calculates the average-case cost without touching the database and returns None. In stateful mode, it performs the actual GroveDB read and returns the element.

The DirectQueryType Enum

The DirectQueryType enum, defined in packages/rs-drive/src/util/grove_operations/mod.rs, controls this dual behavior:

#![allow(unused)]
fn main() {
pub enum DirectQueryType {
    StatelessDirectQuery {
        in_tree_type: TreeType,
        query_target: QueryTarget,
    },
    StatefulDirectQuery,
}
}
  • StatelessDirectQuery: Used for fee estimation. Provides the tree type and query target so the system can calculate costs without reading from disk. The QueryTarget specifies whether we are querying for a tree (with flags) or a value (with an estimated size).

  • StatefulDirectQuery: Used for actual execution. The system reads from GroveDB and returns real data.

There is also a more general QueryType enum that adds reference size estimation:

#![allow(unused)]
fn main() {
pub enum QueryType {
    StatelessQuery {
        in_tree_type: TreeType,
        query_target: QueryTarget,
        estimated_reference_sizes: Vec<u32>,
    },
    StatefulQuery,
}
}

And a QueryTarget enum that specifies what kind of element we expect to find:

#![allow(unused)]
fn main() {
pub enum QueryTarget {
    QueryTargetTree(FlagsLen, TreeType),
    QueryTargetValue(u32),  // estimated value size in bytes
}
}

Another Example: grove_insert

Inserts follow the same pattern. From packages/rs-drive/src/util/grove_operations/grove_insert/v0/mod.rs:

#![allow(unused)]
fn main() {
impl Drive {
    pub(super) fn grove_insert_v0<B: AsRef<[u8]>>(
        &self,
        path: SubtreePath<'_, B>,
        key: &[u8],
        element: Element,
        transaction: TransactionArg,
        options: Option<InsertOptions>,
        drive_operations: &mut Vec<LowLevelDriveOperation>,
        drive_version: &DriveVersion,
    ) -> Result<(), Error> {
        let cost_context = self.grove.insert(
            path, key, element, options, transaction,
            &drive_version.grove_version,
        );
        push_drive_operation_result(cost_context, drive_operations)
    }
}
}

This is simpler than the get because inserts are always stateful -- you cannot "estimate" an insert by not doing it. The push_drive_operation_result helper extracts the cost from GroveDB's CostContext and pushes it onto the operations vector:

#![allow(unused)]
fn main() {
fn push_drive_operation_result<T>(
    cost_context: CostContext<Result<T, GroveError>>,
    drive_operations: &mut Vec<LowLevelDriveOperation>,
) -> Result<T, Error> {
    let CostContext { value, cost } = cost_context;
    if !cost.is_nothing() {
        drive_operations.push(CalculatedCostOperation(cost));
    }
    value.map_err(Error::from)
}
}

Notice the is_nothing() check -- if an operation has zero cost (which can happen), we skip pushing to avoid cluttering the vector.

Batch Apply Types

For operations that work in batch mode (building up a batch of operations to apply atomically), there are corresponding apply-type enums. For example:

#![allow(unused)]
fn main() {
pub enum BatchDeleteApplyType {
    StatelessBatchDelete {
        in_tree_type: TreeType,
        estimated_key_size: u32,
        estimated_value_size: u32,
    },
    StatefulBatchDelete {
        is_known_to_be_subtree_with_sum: Option<MaybeTree>,
    },
}

pub enum BatchInsertTreeApplyType {
    StatelessBatchInsertTree {
        in_tree_type: TreeType,
        tree_type: TreeType,
        flags_len: FlagsLen,
    },
    StatefulBatchInsertTree,
}
}

These follow the same stateless/stateful split. The stateless variants carry enough information to estimate costs without touching the database, while the stateful variants trigger actual operations. Each batch apply type can be converted to a DirectQueryType for use with the lower-level grove operations.

The GroveDBToUse Enum

A recent addition supports querying different GroveDB instances:

#![allow(unused)]
fn main() {
pub enum GroveDBToUse {
    Current,
    LatestCheckpoint,
    Checkpoint(u64),
}
}

This enables queries against historical checkpoints -- useful for proof generation and state verification at specific block heights.

Method Signature Conventions

Across all grove operations, you will notice a consistent parameter ordering:

&self, path, key, [element], [query_type], transaction, drive_operations, drive_version
  1. &self -- the Drive instance (which holds the GroveDB handle)
  2. Path -- where in the tree
  3. Key -- which element at that path
  4. Element -- the data to write (for inserts/replaces)
  5. Query type -- stateless vs. stateful
  6. Transaction -- the GroveDB transaction context
  7. drive_operations -- the mutable cost accumulator
  8. drive_version -- for version dispatching

This consistency makes the codebase navigable even though there are 30+ different grove operations.

Rules and Guidelines

Do:

  • Always use the grove operation wrappers on Drive. Never call self.grove.insert() or self.grove.get() directly in business logic.
  • Pass the drive_operations vector through every call chain. It is how costs propagate upward.
  • Use StatelessDirectQuery for fee estimation and StatefulDirectQuery for actual execution.

Do not:

  • Ignore the cost returned by GroveDB operations. The push_drive_operation_result helper exists for this reason.
  • Mix stateful and stateless queries in a single estimation pass. Pick one mode and stick with it.
  • Create new grove operations without following the mod.rs + v0/mod.rs dispatcher pattern. Consistency is critical.

Batch Operations

Individual grove operations are the atoms. Batch operations are the molecules. Dash Platform never applies a single database write in isolation -- every state transition (a document creation, a contract update, a balance transfer) results in a batch of operations that are applied atomically. Either they all succeed, or none of them do. This chapter covers how that batching works at every level of the stack.

The Three Levels of Abstraction

Drive has three layers of operation abstraction, each serving a different purpose:

  1. DriveOperation -- High-level, domain-aware operations like "add this document" or "apply this contract."
  2. LowLevelDriveOperation -- Individual grove operations, cost calculations, and function costs.
  3. GroveDbOpBatch -- The final flat list of QualifiedGroveDbOp items that GroveDB applies atomically.

The flow is always top-down: DriveOperation -> LowLevelDriveOperation -> GroveDbOpBatch.

DriveOperation: The High-Level API

The DriveOperation enum lives in packages/rs-drive/src/util/batch/drive_op_batch/mod.rs and represents every kind of operation the platform can perform:

#![allow(unused)]
fn main() {
pub enum DriveOperation<'a> {
    DataContractOperation(DataContractOperationType<'a>),
    DocumentOperation(DocumentOperationType<'a>),
    TokenOperation(TokenOperationType),
    WithdrawalOperation(WithdrawalOperationType),
    IdentityOperation(IdentityOperationType),
    PrefundedSpecializedBalanceOperation(PrefundedSpecializedBalanceOperationType),
    SystemOperation(SystemOperationType),
    GroupOperation(GroupOperationType),
    AddressFundsOperation(AddressFundsOperationType),
    GroveDBOperation(QualifiedGroveDbOp),
    GroveDBOpBatch(GroveDbOpBatch),
}
}

Each variant wraps a domain-specific operation type. For example, a DocumentOperationType might be an AddDocument, UpdateDocument, or DeleteDocument. A DataContractOperationType might be ApplyContract. And so on.

The last two variants -- GroveDBOperation and GroveDBOpBatch -- are escape hatches for when code already has raw GroveDB operations and just wants to include them in the batch.

The DriveLowLevelOperationConverter Trait

Every DriveOperation variant knows how to convert itself into a list of low-level operations. This is defined by the DriveLowLevelOperationConverter trait:

#![allow(unused)]
fn main() {
pub trait DriveLowLevelOperationConverter {
    fn into_low_level_drive_operations(
        self,
        drive: &Drive,
        estimated_costs_only_with_layer_info: &mut Option<
            HashMap<KeyInfoPath, EstimatedLayerInformation>,
        >,
        block_info: &BlockInfo,
        transaction: TransactionArg,
        platform_version: &PlatformVersion,
    ) -> Result<Vec<LowLevelDriveOperation>, Error>;
}
}

The estimated_costs_only_with_layer_info parameter is key. When it is None, the converter performs actual operations (stateful mode). When it is Some(HashMap), the converter only estimates costs and fills in layer information for GroveDB's cost estimation (stateless mode).

The DriveOperation enum implements this trait by dispatching to each variant:

#![allow(unused)]
fn main() {
impl DriveLowLevelOperationConverter for DriveOperation<'_> {
    fn into_low_level_drive_operations(
        self, drive: &Drive,
        estimated_costs_only_with_layer_info: &mut Option<
            HashMap<KeyInfoPath, EstimatedLayerInformation>
        >,
        block_info: &BlockInfo,
        transaction: TransactionArg,
        platform_version: &PlatformVersion,
    ) -> Result<Vec<LowLevelDriveOperation>, Error> {
        match self {
            DriveOperation::DataContractOperation(op) =>
                op.into_low_level_drive_operations(
                    drive, estimated_costs_only_with_layer_info,
                    block_info, transaction, platform_version,
                ),
            DriveOperation::DocumentOperation(op) =>
                op.into_low_level_drive_operations(
                    drive, estimated_costs_only_with_layer_info,
                    block_info, transaction, platform_version,
                ),
            // ... each variant delegates to its own converter
            DriveOperation::GroveDBOperation(op) =>
                Ok(vec![GroveOperation(op)]),
            DriveOperation::GroveDBOpBatch(operations) =>
                Ok(operations.operations.into_iter()
                    .map(GroveOperation).collect()),
        }
    }
}
}

The apply_drive_operations Flow

The centerpiece of the batch system is Drive::apply_drive_operations. From packages/rs-drive/src/util/batch/drive_op_batch/drive_methods/apply_drive_operations/v0/mod.rs:

#![allow(unused)]
fn main() {
impl Drive {
    pub(crate) fn apply_drive_operations_v0(
        &self,
        operations: Vec<DriveOperation>,
        apply: bool,
        block_info: &BlockInfo,
        transaction: TransactionArg,
        platform_version: &PlatformVersion,
        previous_fee_versions: Option<&CachedEpochIndexFeeVersions>,
    ) -> Result<FeeResult, Error> {
        if operations.is_empty() {
            return Ok(FeeResult::default());
        }

        let mut low_level_operations = vec![];
        let mut estimated_costs_only_with_layer_info = if apply {
            None::<HashMap<KeyInfoPath, EstimatedLayerInformation>>
        } else {
            Some(HashMap::new())
        };

        let mut finalize_tasks: Vec<DriveOperationFinalizeTask> = Vec::new();

        for drive_op in operations {
            // Collect finalize tasks before conversion
            if let Some(tasks) = drive_op.finalization_tasks(platform_version)? {
                finalize_tasks.extend(tasks);
            }

            // Convert high-level to low-level operations
            low_level_operations.append(
                &mut drive_op.into_low_level_drive_operations(
                    self,
                    &mut estimated_costs_only_with_layer_info,
                    block_info,
                    transaction,
                    platform_version,
                )?
            );
        }

        let mut cost_operations = vec![];

        // Apply the batch atomically
        self.apply_batch_low_level_drive_operations(
            estimated_costs_only_with_layer_info,
            transaction,
            low_level_operations,
            &mut cost_operations,
            &platform_version.drive,
        )?;

        // Execute post-commit finalize tasks
        for task in finalize_tasks {
            task.execute(self, platform_version);
        }

        // Calculate total fee from accumulated costs
        Drive::calculate_fee(
            None,
            Some(cost_operations),
            &block_info.epoch,
            self.config.epochs_per_era,
            platform_version,
            previous_fee_versions,
        )
    }
}
}

Let us trace the flow step by step:

  1. Mode selection. If apply is true, estimated_costs_only_with_layer_info is None, triggering stateful execution. If false, it is Some(HashMap), triggering cost estimation only.

  2. Finalize task collection. Before converting each operation, we collect any finalize tasks it declares. These are post-commit callbacks (covered in the Finalize Tasks chapter).

  3. Conversion. Each DriveOperation is converted into zero or more LowLevelDriveOperation items. A single high-level operation like "add document" might produce dozens of low-level operations (insert the document itself, update each index, update the contract's document count, etc.).

  4. Batch application. All low-level operations are applied as a single atomic batch through apply_batch_low_level_drive_operations.

  5. Finalization. Post-commit tasks execute (like invalidating caches).

  6. Fee calculation. The accumulated cost operations are converted into a FeeResult.

GroveDbOpBatch: The Final Layer

Before operations hit GroveDB, they are split into two categories. From packages/rs-drive/src/util/operations/apply_batch_low_level_drive_operations/v0/mod.rs:

#![allow(unused)]
fn main() {
impl Drive {
    pub(crate) fn apply_batch_low_level_drive_operations_v0(
        &self,
        estimated_costs_only_with_layer_info: Option<
            HashMap<KeyInfoPath, EstimatedLayerInformation>,
        >,
        transaction: TransactionArg,
        batch_operations: Vec<LowLevelDriveOperation>,
        drive_operations: &mut Vec<LowLevelDriveOperation>,
        drive_version: &DriveVersion,
    ) -> Result<(), Error> {
        let (grove_db_operations, mut other_operations) =
            LowLevelDriveOperation::grovedb_operations_batch_consume_with_leftovers(
                batch_operations,
            );
        if !grove_db_operations.is_empty() {
            self.apply_batch_grovedb_operations(
                estimated_costs_only_with_layer_info,
                transaction,
                grove_db_operations,
                drive_operations,
                drive_version,
            )?;
        }
        drive_operations.append(&mut other_operations);
        Ok(())
    }
}
}

The grovedb_operations_batch_consume_with_leftovers method partitions the operations:

  • GroveOperation variants become a GroveDbOpBatch that is applied atomically to GroveDB.
  • Everything else (CalculatedCostOperation, FunctionOperation, PreCalculatedFeeResult) is kept as-is for fee calculation.

The GroveDbOpBatch itself is defined in packages/rs-drive/src/util/batch/grovedb_op_batch/mod.rs:

#![allow(unused)]
fn main() {
pub struct GroveDbOpBatch {
    pub(crate) operations: Vec<QualifiedGroveDbOp>,
}
}

It is a thin wrapper around a vector of QualifiedGroveDbOp -- GroveDB's native batch operation type. The wrapper provides convenience methods for building batches:

#![allow(unused)]
fn main() {
pub trait GroveDbOpBatchV0Methods {
    fn new() -> Self;
    fn push(&mut self, op: QualifiedGroveDbOp);
    fn add_insert_empty_tree(&mut self, path: Vec<Vec<u8>>, key: Vec<u8>);
    fn add_insert_empty_sum_tree(&mut self, path: Vec<Vec<u8>>, key: Vec<u8>);
    fn add_delete(&mut self, path: Vec<Vec<u8>>, key: Vec<u8>);
    fn add_insert(&mut self, path: Vec<Vec<u8>>, key: Vec<u8>, element: Element);
    fn verify_consistency_of_operations(&self) -> GroveDbOpConsistencyResults;
    fn contains<'c, P>(&self, path: P, key: &[u8]) -> Option<&GroveOp>;
    fn remove<'c, P>(&mut self, path: P, key: &[u8]) -> Option<GroveOp>;
    fn remove_if_insert(&mut self, path: Vec<Vec<u8>>, key: &[u8]) -> Option<GroveOp>;
}
}

The verify_consistency_of_operations method is particularly important -- it checks that the batch does not contain conflicting operations (like inserting and deleting the same key).

Building a Batch: A Real Example

The test code in drive_op_batch/mod.rs shows how a typical batch is assembled:

#![allow(unused)]
fn main() {
let mut drive_operations = vec![];

// Step 1: Apply a contract
drive_operations.push(DataContractOperation(ApplyContract {
    contract: Cow::Borrowed(&contract),
    storage_flags: None,
}));

// Step 2: Add a document
drive_operations.push(DocumentOperation(AddDocument {
    owned_document_info: OwnedDocumentInfo {
        document_info: DocumentRefInfo((
            &document,
            StorageFlags::optional_default_as_cow(),
        )),
        owner_id: None,
    },
    contract_info: DataContractInfo::BorrowedDataContract(&contract),
    document_type_info: DocumentTypeInfo::DocumentTypeRef(document_type),
    override_document: false,
}));

// Step 3: Apply everything atomically
drive.apply_drive_operations(
    drive_operations,
    true,  // actually apply, not just estimate
    &BlockInfo::default(),
    Some(&db_transaction),
    platform_version,
    None,
)?;
}

The contract application and document insertion happen in the same atomic batch. If the document insert fails (perhaps due to a uniqueness constraint violation), the contract application is also rolled back. This all-or-nothing guarantee is fundamental to platform correctness.

The Display Implementation

The GroveDbOpBatch has a custom Display implementation that produces human-readable output, mapping raw byte paths to meaningful names:

#![allow(unused)]
fn main() {
impl fmt::Display for GroveDbOpBatch {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        for op in &self.operations {
            let (path_string, known_path) = readable_path(&op.path);
            let (key_string, _) = readable_key_info(known_path, &op.key);
            writeln!(f, "   Path: {}", path_string)?;
            writeln!(f, "   Key: {}", key_string)?;
            // ... operation details
        }
    }
}
}

This translates paths like [0x03] into Identities(3) and keys like 32-byte arrays into IdentityId(bs58::...). Invaluable for debugging.

Rules and Guidelines

Do:

  • Always use apply_drive_operations for applying batches. It handles the full pipeline: conversion, application, finalization, and fee calculation.
  • Collect all operations for a state transition into a single Vec<DriveOperation> before applying.
  • Use the apply: false flag for dry-run fee estimation before committing.

Do not:

  • Apply operations one at a time. Always batch them for atomicity.
  • Mix stateful and stateless operations in the same batch application pass.
  • Forget to handle the FeeResult returned by apply_drive_operations. The fee system depends on it.
  • Manually construct GroveDbOpBatch objects unless you are working at the lowest level. Prefer DriveOperation for business logic.

Cost Tracking

Dash Platform is a fee-based system. Every operation -- reading a document, inserting a key, hashing a value -- has a cost measured in platform credits. This chapter explains how Drive tracks those costs from individual operations all the way through to the final fee result.

The Problem: Why Explicit Cost Tracking?

Some systems use gas metering: you start with a budget, and every opcode decrements it. Dash Platform takes a different approach. Instead of metering, it accumulates the costs of operations as they execute and then calculates the total fee at the end.

This matters because:

  1. Costs depend on what actually happened. A GroveDB insert into a deep tree costs more than one into a shallow tree because of the Merkle proof updates involved. You cannot know this in advance -- you have to do the insert and measure.
  2. Storage fees and processing fees are different. Storage fees are based on bytes added and removed. Processing fees cover computation: seeks, hash operations, byte loading. They need to be calculated separately.
  3. Refunds are possible. When data is removed from storage, the original storage fee can be partially refunded. This requires knowing when the data was originally stored (which epoch), making the calculation depend on historical state.

LowLevelDriveOperation: The Cost Carrier

The LowLevelDriveOperation enum is the vehicle that carries cost information through the system. Defined in packages/rs-drive/src/fees/op.rs:

#![allow(unused)]
fn main() {
pub enum LowLevelDriveOperation {
    GroveOperation(QualifiedGroveDbOp),
    FunctionOperation(FunctionOp),
    CalculatedCostOperation(OperationCost),
    PreCalculatedFeeResult(FeeResult),
}
}

Four variants, each representing a different kind of cost:

GroveOperation

A raw GroveDB operation (insert, delete, get, etc.) that has not yet been executed. When a batch is built up (as described in the Batch Operations chapter), individual grove operations accumulate as this variant. They carry no cost yet -- the cost is determined when the batch is applied to GroveDB.

CalculatedCostOperation

An OperationCost from a GroveDB operation that has already been executed (or estimated). This is what gets pushed onto the drive_operations vector by the grove operation wrappers:

#![allow(unused)]
fn main() {
fn push_drive_operation_result<T>(
    cost_context: CostContext<Result<T, GroveError>>,
    drive_operations: &mut Vec<LowLevelDriveOperation>,
) -> Result<T, Error> {
    let CostContext { value, cost } = cost_context;
    if !cost.is_nothing() {
        drive_operations.push(CalculatedCostOperation(cost));
    }
    value.map_err(Error::from)
}
}

The OperationCost (from grovedb_costs) tracks:

  • seek_count: Number of disk seeks performed
  • storage_cost: Bytes added, replaced, and removed (with per-epoch tracking for refunds)
  • storage_loaded_bytes: Bytes read from storage
  • hash_node_calls: Number of hash operations for Merkle tree updates

FunctionOperation

Represents the cost of a pure computation like hashing. Defined as:

#![allow(unused)]
fn main() {
pub struct FunctionOp {
    pub(crate) hash: HashFunction,
    pub(crate) rounds: u32,
}
}

With supported hash functions:

#![allow(unused)]
fn main() {
pub enum HashFunction {
    Sha256RipeMD160,
    Sha256,
    Sha256_2,  // Double SHA-256
    Blake3,
}
}

Each hash function has a base cost and a per-block cost. The total cost of a FunctionOp is:

#![allow(unused)]
fn main() {
impl FunctionOp {
    fn cost(&self, fee_version: &FeeVersion) -> Credits {
        let block_cost = (self.rounds as u64)
            .saturating_mul(self.hash.block_cost(fee_version));
        self.hash.base_cost(fee_version).saturating_add(block_cost)
    }
}
}

You can create a FunctionOp either by specifying the number of rounds directly or by providing the byte count (which calculates rounds based on the hash function's block size):

#![allow(unused)]
fn main() {
impl FunctionOp {
    pub fn new_with_round_count(hash: HashFunction, rounds: u32) -> Self {
        FunctionOp { hash, rounds }
    }

    pub fn new_with_byte_count(hash: HashFunction, byte_count: u16) -> Self {
        let blocks = byte_count / hash.block_size() + 1;
        let rounds = blocks + hash.rounds() - 1;
        FunctionOp { hash, rounds: rounds as u32 }
    }
}
}

PreCalculatedFeeResult

A fee result that was already computed elsewhere and just needs to be included in the total. This is a pass-through -- no further calculation needed.

BaseOp: Arithmetic Operation Costs

For simple computational operations (not storage-related), the BaseOp enum provides fixed costs:

#![allow(unused)]
fn main() {
pub enum BaseOp {
    Stop, Add, Mul, Sub, Div, Sdiv, Mod, Smod,
    Addmod, Mulmod, Signextend,
    Lt, Gt, Slt, Sgt, Eq, Iszero,
    And, Or, Xor, Not, Byte,
}

impl BaseOp {
    pub fn cost(&self) -> u64 {
        match self {
            BaseOp::Stop => 0,
            BaseOp::Add | BaseOp::Sub => 12,
            BaseOp::Mul | BaseOp::Div | BaseOp::Sdiv |
            BaseOp::Mod | BaseOp::Smod | BaseOp::Signextend => 20,
            BaseOp::Addmod | BaseOp::Mulmod => 32,
            BaseOp::Lt | BaseOp::Gt | BaseOp::Slt |
            BaseOp::Sgt | BaseOp::Eq | BaseOp::Iszero |
            BaseOp::And | BaseOp::Or | BaseOp::Xor |
            BaseOp::Not | BaseOp::Byte => 12,
        }
    }
}
}

These are EVM-inspired operation costs, adapted for the platform's fee model. Comparisons and bitwise operations cost 12 credits. Multiplication and division cost 20. Modular arithmetic costs 32.

The consume_to_fees_v0 Pipeline

When all operations are collected, they are converted into fee results through consume_to_fees_v0:

#![allow(unused)]
fn main() {
pub fn consume_to_fees_v0(
    drive_operations: Vec<LowLevelDriveOperation>,
    epoch: &Epoch,
    epochs_per_era: u16,
    fee_version: &FeeVersion,
    previous_fee_versions: Option<&CachedEpochIndexFeeVersions>,
) -> Result<Vec<FeeResult>, Error> {
    drive_operations.into_iter().map(|operation| match operation {
        PreCalculatedFeeResult(f) => Ok(f),

        FunctionOperation(op) => Ok(FeeResult {
            processing_fee: op.cost(fee_version),
            ..Default::default()
        }),

        _ => {
            let cost = operation.operation_cost()?;

            // Storage fee: bytes added * rate per byte
            let storage_fee = cost.storage_cost.added_bytes as u64
                * fee_version.storage.storage_disk_usage_credit_per_byte;

            // Processing fee: seeks + loaded bytes + hash calls + ...
            let processing_fee = cost.ephemeral_cost(fee_version)?;

            // Refunds from removed data
            let (fee_refunds, removed_bytes_from_system) =
                match cost.storage_cost.removed_bytes {
                    NoStorageRemoval => (FeeRefunds::default(), 0),
                    BasicStorageRemoval(amount) => (FeeRefunds::default(), amount),
                    SectionedStorageRemoval(removal_per_epoch_by_identifier) => {
                        // Calculate epoch-aware refunds
                        (FeeRefunds::from_storage_removal(
                            removal_per_epoch_by_identifier,
                            epoch.index,
                            epochs_per_era,
                            previous_fee_versions,
                        )?, system_amount)
                    }
                };

            Ok(FeeResult {
                storage_fee,
                processing_fee,
                fee_refunds,
                removed_bytes_from_system,
            })
        }
    }).collect()
}
}

Each operation produces a FeeResult with four components:

  • storage_fee: The cost of new bytes written to persistent storage.
  • processing_fee: The ephemeral cost of computation and I/O.
  • fee_refunds: Credits returned because previously-stored data was removed.
  • removed_bytes_from_system: Bytes removed that were stored by the system (not any particular identity), so no refund is issued.

Ephemeral Cost Calculation

The ephemeral_cost method on OperationCost computes the processing fee from the raw operation metrics:

#![allow(unused)]
fn main() {
impl DriveCost for OperationCost {
    fn ephemeral_cost(&self, fee_version: &FeeVersion) -> Result<Credits, Error> {
        let OperationCost {
            seek_count,
            storage_cost,
            storage_loaded_bytes,
            hash_node_calls,
        } = self;

        let seek_cost = (*seek_count as u64)
            .checked_mul(fee_version.storage.storage_seek_cost)?;

        let storage_added_bytes_ephemeral_cost = (storage_cost.added_bytes as u64)
            .checked_mul(fee_version.storage.storage_processing_credit_per_byte)?;

        let storage_replaced_bytes_ephemeral_cost = (storage_cost.replaced_bytes as u64)
            .checked_mul(fee_version.storage.storage_processing_credit_per_byte)?;

        let storage_loaded_bytes_cost = (*storage_loaded_bytes)
            .checked_mul(fee_version.storage.storage_load_credit_per_byte)?;

        let blake3_total = fee_version.hashing.blake3_base
            + fee_version.hashing.blake3_per_block;
        let hash_node_cost = blake3_total * (*hash_node_calls as u64);

        // Sum all costs with overflow checking
        seek_cost
            .checked_add(storage_added_bytes_ephemeral_cost)
            .and_then(|c| c.checked_add(storage_replaced_bytes_ephemeral_cost))
            .and_then(|c| c.checked_add(storage_loaded_bytes_cost))
            .and_then(|c| c.checked_add(hash_node_cost))
            .ok_or_else(|| get_overflow_error("ephemeral cost addition overflow"))
    }
}
}

Notice that every multiplication and addition uses checked arithmetic. In a fee system, overflows would be catastrophic -- an underflowed fee could let someone store unlimited data for free.

Helper Methods on LowLevelDriveOperation

The LowLevelDriveOperation type provides several methods for working with collections of operations:

#![allow(unused)]
fn main() {
impl LowLevelDriveOperation {
    // Combine all CalculatedCostOperation costs into one
    pub fn combine_cost_operations(
        operations: &[LowLevelDriveOperation]
    ) -> OperationCost { ... }

    // Extract GroveOperation variants into a batch
    pub fn grovedb_operations_batch(
        operations: &[LowLevelDriveOperation]
    ) -> GroveDbOpBatch { ... }

    // Same but consuming the vector
    pub fn grovedb_operations_batch_consume(
        operations: Vec<LowLevelDriveOperation>
    ) -> GroveDbOpBatch { ... }

    // Partition: grove ops go to batch, rest stays as leftovers
    pub fn grovedb_operations_batch_consume_with_leftovers(
        operations: Vec<LowLevelDriveOperation>,
    ) -> (GroveDbOpBatch, Vec<LowLevelDriveOperation>) { ... }
}
}

The grovedb_operations_batch_consume_with_leftovers method is particularly important -- it is used during batch application to separate the grove operations (which go to GroveDB) from the cost operations (which go to fee calculation).

Constructing Operations

LowLevelDriveOperation also has constructors for common operations:

#![allow(unused)]
fn main() {
impl LowLevelDriveOperation {
    pub fn for_known_path_key_empty_tree(
        path: Vec<Vec<u8>>, key: Vec<u8>,
        storage_flags: Option<&StorageFlags>,
    ) -> Self { ... }

    pub fn for_known_path_key_empty_sum_tree(
        path: Vec<Vec<u8>>, key: Vec<u8>,
        storage_flags: Option<&StorageFlags>,
    ) -> Self { ... }

    pub fn insert_for_known_path_key_element(
        path: Vec<Vec<u8>>, key: Vec<u8>, element: Element,
    ) -> Self {
        GroveOperation(QualifiedGroveDbOp::insert_or_replace_op(
            path, key, element
        ))
    }

    pub fn replace_for_known_path_key_element(
        path: Vec<Vec<u8>>, key: Vec<u8>, element: Element,
    ) -> Self {
        GroveOperation(QualifiedGroveDbOp::replace_op(
            path, key, element
        ))
    }
}
}

These provide a cleaner API than constructing QualifiedGroveDbOp directly, and they handle storage flags properly.

Rules and Guidelines

Do:

  • Use checked arithmetic everywhere in fee calculations. checked_mul, checked_add, and friends.
  • Construct FunctionOp with new_with_byte_count when you know the input size, and new_with_round_count when you know the rounds.
  • Let operations accumulate in the drive_operations vector throughout the call chain.

Do not:

  • Call operation_cost() on a GroveOperation -- it will return an error. Grove operations must be executed first; only CalculatedCostOperation carries a usable cost.
  • Forget that storage fees and processing fees are calculated differently. Storage fees are proportional to bytes. Processing fees are a complex function of seeks, loads, hashes, and byte movements.
  • Assume fee rates are constant. They are versioned through FeeVersion and can change between protocol versions.
  • Ignore removed_bytes_from_system. This tracks bytes removed that belong to the system rather than a specific identity, affecting the refund calculation.

Finalize Tasks

Most operations on Dash Platform follow a straightforward path: convert high-level operations to low-level ones, apply them atomically, calculate fees. But some operations need something to happen after the batch has been successfully committed. That is what finalize tasks are for.

The Problem: Post-Apply Side Effects

Consider what happens when a data contract is updated. The updated contract is written to GroveDB as part of the atomic batch. But Drive also caches contracts in memory for fast access. After the batch is applied, that cache entry is stale -- it still holds the old version of the contract.

You cannot refresh the cache before the batch is applied, because applying might fail (GroveDB could reject the batch due to a consistency error). And you cannot refresh it during the apply, because the batch application is a single atomic operation on GroveDB. You need a post-apply callback: "if the batch succeeds, do this."

That is exactly what DriveOperationFinalizeTask provides.

The DriveOperationFinalizeTask Enum

Defined in packages/rs-drive/src/util/batch/drive_op_batch/finalize_task.rs:

#![allow(unused)]
fn main() {
pub enum DriveOperationFinalizeTask {
    RefreshDataContractCache { contract_id: Identifier },
}
}

Currently there is only one variant: RefreshDataContractCache. When a data contract is created or updated, this task is registered. After the batch is applied successfully, it re-seeds Drive's in-memory cache from what state now holds for the contract, reading through the transaction the batch was applied in.

The execution delegates to Drive::refresh_data_contract_cache_from_state:

#![allow(unused)]
fn main() {
impl DriveOperationFinalizeTask {
    pub fn execute(
        self,
        drive: &Drive,
        transaction: TransactionArg,
        platform_version: &PlatformVersion,
    ) -> Result<(), Error> {
        match self {
            DriveOperationFinalizeTask::RefreshDataContractCache { contract_id } => drive
                .refresh_data_contract_cache_from_state(
                    contract_id.to_buffer(),
                    transaction,
                    platform_version,
                ),
        }
    }
}
}

The task takes the transaction because what it seeds must be what this transaction holds, not committed state. Inside a block, the batch was applied in the block transaction and the update is not committed yet: the refresh reads the updated contract through that transaction, marks the contract as modified in the block, and seeds the block cache with it. Outside a block (a caller that passed no transaction, so the batch committed on its own) the refresh only evicts the superseded copy, and the next reader reloads it from committed state.

The DriveOperationFinalizationTasks Trait

Not every DriveOperation has finalize tasks. The trait that declares them is:

#![allow(unused)]
fn main() {
pub trait DriveOperationFinalizationTasks {
    fn finalization_tasks(
        &self,
        platform_version: &PlatformVersion,
    ) -> Result<Option<Vec<DriveOperationFinalizeTask>>, Error>;
}
}

The return type is Option<Vec<...>> rather than just Vec<...>. This is a deliberate optimization -- since only one operation type currently has finalize tasks, returning None (rather than an empty Vec) avoids unnecessary heap allocations for the vast majority of operations.

The implementation on DriveOperation dispatches through versioning:

#![allow(unused)]
fn main() {
impl DriveOperationFinalizationTasks for DriveOperation<'_> {
    fn finalization_tasks(
        &self,
        platform_version: &PlatformVersion,
    ) -> Result<Option<Vec<DriveOperationFinalizeTask>>, Error> {
        match platform_version
            .drive
            .methods
            .state_transitions
            .operations
            .finalization_tasks
        {
            0 => self.finalization_tasks_v0(platform_version),
            version => Err(Error::Drive(DriveError::UnknownVersionMismatch {
                method: "DriveOperation.finalization_tasks".to_string(),
                known_versions: vec![0],
                received: version,
            })),
        }
    }
}
}

And the v0 implementation only checks data contract operations:

#![allow(unused)]
fn main() {
impl DriveOperation<'_> {
    fn finalization_tasks_v0(
        &self,
        platform_version: &PlatformVersion,
    ) -> Result<Option<Vec<DriveOperationFinalizeTask>>, Error> {
        match self {
            DriveOperation::DataContractOperation(o) =>
                o.finalization_tasks(platform_version),
            _ => Ok(None),
        }
    }
}
}

Every other operation variant -- documents, identities, tokens, withdrawals -- returns None. Only data contract operations can produce finalize tasks.

How Finalize Tasks Integrate with Batch Application

The integration point is in apply_drive_operations_v0, which we saw in the Batch Operations chapter. Here is the relevant excerpt:

#![allow(unused)]
fn main() {
pub(crate) fn apply_drive_operations_v0(
    &self,
    operations: Vec<DriveOperation>,
    apply: bool,
    block_info: &BlockInfo,
    transaction: TransactionArg,
    platform_version: &PlatformVersion,
    previous_fee_versions: Option<&CachedEpochIndexFeeVersions>,
) -> Result<FeeResult, Error> {
    // ...

    let mut finalize_tasks: Vec<DriveOperationFinalizeTask> = Vec::new();

    for drive_op in operations {
        // Step 1: Collect finalize tasks BEFORE converting the operation
        if let Some(tasks) = drive_op.finalization_tasks(platform_version)? {
            finalize_tasks.extend(tasks);
        }

        // Step 2: Convert to low-level operations (consumes drive_op)
        low_level_operations.append(
            &mut drive_op.into_low_level_drive_operations(/* ... */)?
        );
    }

    // Step 3: Apply the batch atomically
    self.apply_batch_low_level_drive_operations(/* ... */)?;

    // Step 4: Execute finalize tasks AFTER a successful apply, and only when
    // the batch was applied rather than estimated. They read through the
    // caller's transaction; `caller_transaction` is `None` exactly when the
    // batch committed on its own just above.
    if apply {
        for task in finalize_tasks {
            task.execute(self, caller_transaction, platform_version)?;
        }
    }

    // Step 5: Calculate fees
    Drive::calculate_fee(/* ... */)
}
}

The ordering is critical:

  1. Collect finalize tasks first. This happens before into_low_level_drive_operations because that method consumes the DriveOperation (it takes self, not &self). After conversion, the original operation is gone.

  2. Apply the batch. If this fails, we return the error immediately. The finalize tasks never execute.

  3. Execute finalize tasks only on success. By the time we reach step 4, we know the batch was applied successfully. Now it is safe to refresh caches and perform other side effects. An estimation-only call (apply == false) writes nothing, so it runs no finalize tasks either.

The Cache Refresh Pattern

Why a refresh rather than a plain eviction? Drive maintains an in-memory cache of frequently-accessed data contracts, and two kinds of reader share it: block execution, which reads through the block transaction on the consensus thread, and the query threads, which read committed state with no transaction, concurrently, and populate the global half of the cache with what they read.

Without any refresh, here is what would go wrong:

  1. Block N: Contract "foo" is at version 3 in GroveDB and cached.
  2. Block N+1: A state transition updates "foo" to version 4 in GroveDB.
  3. Block N+1: Without a refresh, the block still reads version 3 from the cache.
  4. Block N+1: Document validation uses the stale version 3 schema, potentially accepting invalid documents.

A plain eviction is not enough, because of the query threads. Between the eviction and the block's commit, a query can read committed state (still version 3) and put version 3 back into the global cache. A transactional read that missed the block cache and fell back to the global cache would then be handed version 3 again, while a validator whose cache happened to be cold reads version 4 from the transaction: the two validators execute the rest of the block against different contracts and compute different app hashes.

The refresh closes this in two ways, both implemented in DataContractCache (packages/rs-drive/src/cache/data_contract.rs):

  • It marks the contract as modified in the block. From then until the block cache is cleared or promoted, a transactional read of that contract is served from the block cache or from state through the transaction, never from the global cache. This holds even if the seeded block-cache entry is evicted, and it is what the proposer relies on when it rolls a transition back: the rollback drops the modified contracts from the block cache, and the next read goes to the rolled-back transaction.
  • The block cache is promoted into the global cache only after the block transaction is committed, by the finalize_block handler, and every committed-state read carries a CommittedGeneration snapshot taken before it read state. A read that straddles a commit cannot publish what it read, so a query thread descheduled across the commit cannot clobber the promoted definition with the pre-block one.

When to Use Finalize Tasks

Finalize tasks are the right tool when you need to perform side effects that:

  1. Must not happen if the batch fails. If you refresh a cache before the apply and the apply fails, you have seeded a definition the transaction does not hold.

  2. Are not idempotent with respect to partial application. A cache refresh is fine to do after the apply because the cache will self-heal on the next access. But if your side effect were "send a network message," you would want to be very sure the batch actually committed.

  3. Operate on data outside GroveDB. GroveDB's atomic batch guarantees only cover GroveDB state. In-memory caches, external systems, and non-transactional state all need explicit post-commit handling.

Extending Finalize Tasks

To add a new finalize task:

  1. Add a variant to the DriveOperationFinalizeTask enum in finalize_task.rs.
  2. Implement its execution in the execute method's match block.
  3. In the relevant DriveOperation variant's finalization_tasks implementation, return the new task when appropriate.

The design is intentionally simple and extensible. The enum + trait pattern means new finalize tasks do not affect existing code paths.

Rules and Guidelines

Do:

  • Collect finalize tasks before consuming DriveOperation via into_low_level_drive_operations.
  • Execute finalize tasks only after confirming the batch was applied successfully.
  • Keep finalize task execution fast. They run synchronously in the block processing pipeline.
  • Read through the transaction the batch was applied in. What a task seeds must be what that transaction holds, not committed state.

Do not:

  • Put business logic in finalize tasks. They are for side effects like cache management, not for state mutations. State mutations belong in the batch itself.
  • Execute finalize tasks if the batch application returns an error. The whole point is that they only run on success.
  • Rely on finalize tasks for exactly-once side effects. If the process crashes between the apply and the finalize task, the task will not run; the cache refresh survives this because Drive's caches are rebuilt from disk on restart.
  • Introduce finalize tasks with external side effects (like network calls) without careful consideration of failure modes. Keep them fast, local, and idempotent.

Indexes

Drive stores documents in GroveDB. Every document type has a primary-key tree (documents keyed by document ID), plus zero or more secondary indexes the contract author declares in the document schema. This chapter is a reference for the Index struct's fields, what they mean for the on-disk layout, and how Drive walks indexes during inserts and queries.

What an Index Is

A document type's indices array tells Drive: "for queries that filter or sort by these properties, build a sorted lookup so they don't have to enumerate every document." Each entry in indices becomes one secondary index; Drive maintains it on every insert/update/delete so that queries which match the index prefix are O(prefix walk) rather than O(documents).

Concrete example. Given:

{
  "person": {
    "type": "object",
    "indices": [
      {
        "name": "byLastName",
        "properties": [{ "lastName": "asc" }]
      }
    ],
    "properties": {
      "firstName": { "type": "string", "position": 0 },
      "lastName":  { "type": "string", "position": 1 }
    },
    "required": ["firstName", "lastName"],
    "additionalProperties": false
  }
}

— a query like where lastName = "Smith" reaches the matching documents through the byLastName index in O(log n) plus the per-result IO. Without that index it would be a full document-type scan.

The Index Struct

The compiled-Rust shape — the JSON schema fields are deserialized into this — lives in packages/rs-dpp/src/data_contract/document_type/index/mod.rs:

#![allow(unused)]
fn main() {
pub struct Index {
    pub name: String,
    pub properties: Vec<IndexProperty>,
    pub unique: bool,
    pub null_searchable: bool,
    pub contested_index: Option<ContestedIndexInformation>,
    pub countable: IndexCountability,
}

pub struct IndexProperty {
    pub name: String,
    pub ascending: bool,
}
}

name

A short, human-readable identifier for the index (e.g. "byOwnerAndType"). It shows up in error messages and is the key used in document_type.indexes() (BTreeMap<String, Index>). Every document meta-schema requires it. A parse that skips schema validation (check tx, test fixtures) and meets an unnamed index derives the name from the properties and their directions, joined with |, so every parse of the same contract agrees on it. Two indexes within the same document type cannot share a name.

properties: Vec<IndexProperty>

The ordered list of columns this index covers. Each IndexProperty is a (name, ascending) pair. Order matters: a query has to match a prefix of these properties for the index to be useful. An index [lastName, firstName] answers where lastName = X and where lastName = X AND firstName = Y but not where firstName = Y alone.

The schema form is:

"properties": [
  { "lastName":  "asc" },
  { "firstName": "asc" }
]

Every document meta-schema accepts only "asc". Drive stores index entries in ascending order; a query chooses its own result order.

unique: bool

If true, no two documents may share the same combination of values for the indexed properties. The platform enforces this on insert: a duplicate trips a DuplicateUniqueIndexError consensus error.

A unique index changes the on-disk layout at the terminal level: instead of a sub-tree of document references keyed by document ID, the terminal stores a single bare Reference element pointing at the one document that matched. See Tree Type at the Terminal Level below.

Uniqueness can't be enforced when an indexed property is null, so a document with any null in the index path falls back to the non-unique storage shape for that document. See Null Handling.

null_searchable: bool

Defaults to true. Controls what happens when all indexed properties of a document are null:

  • null_searchable: true — the document is still indexed at the all-null path, so a query against the all-null prefix can find it.
  • null_searchable: false — Drive skips the index insertion entirely. Documents with all-null index values exist (in the primary-key tree) but are not reachable via this index.

The flag only affects the all-null case. A document with some null values gets indexed regardless.

skip_if_absent: bool and skip_if_absent_properties: Vec<String>

From protocol version 14 an index may skip documents that omit a property of its skip set (skipIfAbsent: true for every optional property of the index, or an array naming them). A skipped document gets no reference in the index and builds none of the index's own trees, wherever the skip property sits in the property list. The other optional properties of a stored type's skip index keep the null layout below. skip_if_absent_properties holds the resolved set, so true and the array of the same properties parse to equal indexes; the level info at the index's end carries it too, since the walkers only see merged levels. See Null Handling for how the walkers apply it, and Conditional participation for the indexOnly rules.

contested_index: Option<ContestedIndexInformation>

When set, this index identifies a scarce, contested resource (the canonical example is a DPNS name like dash). Documents trying to register the same value under a contested index don't auto-fail with a uniqueness error — they enter a masternode-vote resolution where each contender's claim is held until voting concludes. Contested indexes must also be unique: true; the parser rejects the combination otherwise.

Out of scope for this chapter; see DPNS / contested-resource docs for the full lifecycle.

countable: IndexCountability

Controls whether the terminal tree under each indexed value carries a count, and which count-tree variant. Three variants:

ValueTree variantCapabilities
NotCountable (default)NormalTreeNo count fast path
CountableCountTreeO(1) totals at the root
CountableAllowingOffsetProvableCountTreeO(1) totals plus per-node counts that will enable future O(log n) range / offset queries

summable: Option<String> and range_summable: bool

The sum-side analog of countable / range_countable. When summable = Some(<property_name>), the terminal tree under each indexed value carries a running sum of the named property across the documents at that value — O(1) reads for SUM(<property>) WHERE <indexed_field> = X queries. The named property must be type: integer and listed in the document type's required array (the DPP validator enforces this at contract creation).

range_summable: true is the sum-side counterpart of range_countable: per-node aggregated sums committed to every internal merk node of the property-name tree, so SUM(<property>) WHERE <indexed_field> BETWEEN A AND B queries land on grovedb's AggregateSumOnRange primitive — O(log n), no document enumeration. Like range_countable, it requires summable to be set; it's additive, not a replacement.

summablerange_summableProperty-name treeValue treeCapabilities
None (default)–NormalTreeNormalTreeNo sum fast path
Some("amount")falseNormalTreeSumTreeO(1) sum(amount) WHERE field = X at the value-tree root
Some("amount")trueProvableSumTreeSumTreeO(1) point sum plus O(log n) range sums via AggregateSumOnRange

Compose orthogonally with the count flags. Combining countable and summable on the same index yields one of grovedb's combined-aggregation tree variants (CountSumTree, ProvableCountSumTree, or ProvableCountProvableSumTree) — one tree carries both metrics, queries on either axis read from the same merk root. See Range-Summable Indexes below for the storage layout, the ReferenceWithSumItem element type that makes per-document contributions land on the parent SumTree, and how range-summable composes with range-countable to produce PCPS trees backing the new AggregateCountAndSumOnRange combined-proof primitive.

For the conceptual treatment of sum trees and the full GetDocumentsSum query surface, see Document Sum Trees (paralleling Document Count Trees).

The schema accepts both the legacy boolean form (true → Countable, false → NotCountable) and the camelCase string form ("notCountable" / "countable" / "countableAllowingOffset"). For the full design rationale see Document Count Trees.

How Drive Builds the IndexLevel Trie

The flat list of Indexes declared on a document type is compiled, at contract-load time, into an IndexLevel trie (packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs):

#![allow(unused)]
fn main() {
pub struct IndexLevel {
    sub_index_levels: BTreeMap<String, IndexLevel>,
    has_index_with_type: Option<IndexLevelTypeInfo>,
    level_identifier: u64,
}
}

Each property name in any index becomes an edge in this trie; indexes that share a prefix share their initial path. An index "terminates" at a level by setting has_index_with_type = Some(...) — that's how the recursive insert / lookup code knows it's at the last property of a defined index, vs. just walking through a shared prefix.

Given two indexes:

  • byOwnerAndType = [ownerId, docType]
  • byOwnerAndStatus = [ownerId, status]

the trie that gets built is:

flowchart TD
    Root["(root)"]
    Owner["<b>ownerId</b><br/><i>shared prefix</i>"]
    DocType["<b>docType</b><br/>terminates <code>byOwnerAndType</code><br/><i>has_index_with_type = Some(...)</i>"]
    Status["<b>status</b><br/>terminates <code>byOwnerAndStatus</code><br/><i>has_index_with_type = Some(...)</i>"]

    Root --> Owner
    Owner --> DocType
    Owner --> Status

    style DocType fill:#e0f7fa,stroke:#006064,color:#000
    style Status fill:#e0f7fa,stroke:#006064,color:#000

The ownerId level is shared between both indexes. The docType and status levels each set has_index_with_type on themselves with their own unique / countable / null_searchable flags. A third index that also started with ownerId would attach further sub-levels under ownerId instead of duplicating it.

This trie shape directly mirrors the GroveDB path shape used at insert / query time.

GroveDB Layout

A document under contract C of type T with index property propA = vA, propB = vB lives at the grove path:

[ DataContractDocuments,  contract_id,  1,  doc_type_name,
  propA_name, vA,  propB_name, vB,  0  →  <terminal element> ]

Let's break that down:

  • DataContractDocuments — root tree byte (u8 constant) for "this is a document index, not a contract definition or identity record".
  • contract_id — 32-byte contract identifier.
  • 1 — separator distinguishing the document storage area from the contract definition area within contract_id. Its siblings under contract_id are 0, the serialized contract (or, for a contract that keeps history, the history subtree whose key 0 references the latest revision), and, from protocol version 14, 2, the contract's version number as a four-byte item that getDataContractsLatestVersions reads and proves without the contract.
  • doc_type_name — UTF-8 bytes of the document type ("person", "contactRequest", etc.).
  • propA_name, vA, propB_name, vB — alternating property key and serialized value, one pair per index property, in declaration order.
  • 0 — the conventional "terminal slot" byte under each value level; it's where the actual reference (or sub-tree-of-references) lives.

The intermediate levels (propA_name, vA, propB_name, vB) are all NormalTrees. The terminal element at [0] varies — see the next section.

Concretely, suppose a widget-store contract has document type widget with a non-unique countable index byColor = [color], and three documents stored: A and B with color = "red", C with color = "blue". Then drive lays out:

flowchart TD
    R["<b>DataContractDocuments</b><br/>(root tree byte)"]
    CID["<b>widget-store-id</b><br/>(32 bytes)"]
    Sep["<b>1</b><br/>(documents area)"]
    DT["<b>'widget'</b><br/>(document type)"]
    PK["primary-key tree<br/>(docs keyed by ID)<br/><i>not shown — own subtree</i>"]
    Color["<b>'color'</b><br/>(index property name)"]
    Red["<b>'red'</b><br/>(serialized value)"]
    Blue["<b>'blue'</b>"]
    RedT["<b>[0]: CountTree</b><br/>count = 2"]
    BlueT["<b>[0]: CountTree</b><br/>count = 1"]
    RA(["<b>doc_id_A</b><br/>Reference"])
    RB(["<b>doc_id_B</b><br/>Reference"])
    RC(["<b>doc_id_C</b><br/>Reference"])

    R --> CID --> Sep --> DT
    DT --> PK
    DT --> Color
    Color --> Red --> RedT
    Color --> Blue --> BlueT
    RedT --> RA
    RedT --> RB
    BlueT --> RC

    classDef countTree fill:#fff4e5,stroke:#bf6900,color:#000
    classDef reference fill:#e8f5e9,stroke:#1b5e20,color:#000
    classDef placeholder fill:#f5f5f5,stroke:#888,color:#000
    class RedT,BlueT countTree
    class RA,RB,RC reference
    class PK placeholder

Legend: rectangles are tree-type elements (intermediate NormalTrees holding sub-keys, terminal CountTrees holding per-doc refs); rounded green nodes are Reference elements (leaf pointers to documents). Amber highlight marks count-bearing trees specifically.

A query like where color = "red" walks the path down to the red value, opens the [0] CountTree, and either reads count_value (for GetDocumentsCount) or enumerates the inner references (for GetDocuments). Because the count is stored on the wrapping element, the count read is O(1) regardless of how many docs are inside.

Shared-Prefix Indexes

Now extend the same widget document type with a shape property and a second, compound, countable index byColorShape = [color, shape]. The IndexLevel trie is:

flowchart TD
    Root["(root)"]
    ColorLevel["<b>color</b><br/>terminates <code>byColor</code><br/><i>has_index_with_type = Some(...)</i><br/>+ sub-level <code>shape</code>"]
    ShapeLevel["<b>shape</b><br/>terminates <code>byColorShape</code><br/><i>has_index_with_type = Some(...)</i>"]

    Root --> ColorLevel --> ShapeLevel

    style ColorLevel fill:#e0f7fa,stroke:#006064,color:#000
    style ShapeLevel fill:#e0f7fa,stroke:#006064,color:#000

Importantly, color is a level that both terminates an index (byColor) and has a sub-level (shape) continuing past it. That dual role shows up directly in the on-disk path: at every [..., color, <value>] subtree, key [0] (the byColor terminal) and key 'shape' (the continuation into byColorShape) live as siblings.

With three documents — A: (red, circle), B: (red, square), C: (blue, square) — the layout is:

flowchart TD
    DT["<b>'widget'</b><br/>(document type)"]
    ColorKey["<b>'color'</b><br/>(index property)"]
    Red["<b>'red'</b>"]
    Blue["<b>'blue'</b>"]

    %% byColor terminals (at the color-value level)
    RedColorT["<b>[0]: CountTree</b><br/>count = 2<br/><i>byColor</i>"]
    BlueColorT["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColor</i>"]

    %% byColorShape continuation (sibling to [0] under each color value)
    RedShape["<b>'shape'</b>"]
    BlueShape["<b>'shape'</b>"]
    RedCircle["<b>'circle'</b>"]
    RedSquare["<b>'square'</b>"]
    BlueSquare["<b>'square'</b>"]

    %% byColorShape terminals
    RCT["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
    RST["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
    BST["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]

    %% References — each indexed path stores its own reference, so docs appear
    %% multiple times across the diagram (same key, same doc id, but a separate
    %% Reference element under each terminal that matches the document).
    RA1(["<b>doc_id_A</b><br/>Reference"])
    RB1(["<b>doc_id_B</b><br/>Reference"])
    RC1(["<b>doc_id_C</b><br/>Reference"])
    RA2(["<b>doc_id_A</b><br/>Reference"])
    RB2(["<b>doc_id_B</b><br/>Reference"])
    RC2(["<b>doc_id_C</b><br/>Reference"])

    DT --> ColorKey
    ColorKey --> Red
    ColorKey --> Blue

    Red --> RedColorT
    Red --> RedShape
    Blue --> BlueColorT
    Blue --> BlueShape

    RedColorT --> RA1
    RedColorT --> RB1
    BlueColorT --> RC1

    RedShape --> RedCircle
    RedShape --> RedSquare
    BlueShape --> BlueSquare

    RedCircle --> RCT --> RA2
    RedSquare --> RST --> RB2
    BlueSquare --> BST --> RC2

    classDef countTree fill:#fff4e5,stroke:#bf6900,color:#000
    classDef reference fill:#e8f5e9,stroke:#1b5e20,color:#000
    class RedColorT,BlueColorT,RCT,RST,BST countTree
    class RA1,RB1,RC1,RA2,RB2,RC2 reference

Two things to notice:

  • [0] and the sub-property name ('shape') are siblings under each color value. The [0] count tree is the byColor terminal at that color value; the 'shape' subtree is the continuation that byColorShape walks past for the next index property. Drive descends one or the other depending on which index covers the query.
  • The same document is stored as a separate Reference under every index path that matches it. Doc A appears under byColor[red] and under byColorShape[red, circle]; doc B under byColor[red] and byColorShape[red, square]; doc C under byColor[blue] and byColorShape[blue, square]. That's why each of A, B, C shows up twice in the diagram — once per index that covers the document. Insert/delete touches all of them; queries walk only the one path their picker selected.

A query like where color = "red" resolves through the byColor terminal ([0] under red) — count = 2, O(1). A query like where color = "red" AND shape = "circle" resolves through byColorShape instead, taking the 'shape' sub-tree past red and reading the terminal under circle — count = 1, also O(1). Both queries are served by the same shared-prefix layout, just descending different branches at the red node.

Range-Countable Indexes

Status: design. Not yet implemented at the time of writing. Depends on a parallel grovedb change that adds NonCounted<ElementType> element variants — element types that behave exactly like their counterparts except that their count value is not propagated to the parent count tree, and which are only insertable inside a CountTree / ProvableCountTree / CountSumTree / ProvableCountSumTree.

range_countable is a separate per-index property from countable. Where countable makes the count of docs at one specific value O(1), range_countable makes the count of docs between two values O(log n) — answering queries like "how many widgets have a color between red and tomato alphabetically" without enumerating every distinct color value.

Constraints

  • range_countable: true requires countable to be Countable or CountableAllowingOffset. It is additive to countability, not a replacement: range queries are useful only on indexes you'd already want to count by.
  • The combination is meaningful only on non-unique indexes (or unique indexes whose entries can be null-bearing), for the same reason countable is mostly inert on unique-with-required-fields: a unique non-null terminal is a bare Reference, with no tree to hang per-node counts off of.
  • Sibling sub-trees that share a prefix with a range-countable index — e.g., the 'shape' continuation when byColor is range-countable but byColorShape shares its color prefix — must use NonCounted<*> variants so their counts do not pollute the range-countable value tree's count.

Mechanism

Where today's countable upgrades only the terminal [0] element under each indexed value to a count tree, range_countable additionally upgrades two more levels:

LevelWithout range_countableWith range_countable
Property-name tree (e.g. 'color')NormalTreeProvableCountTree
Value tree (e.g. 'red', 'blue')NormalTreeCountTree
Terminal at [0] under each valueNormalTree / CountTree / ProvableCountTree (per countable)unchanged — still driven by countable
Sibling continuations inside the value tree (e.g. 'shape' for a compound index sharing the prefix)NormalTreeNonCounted<NormalTree>

The property-name tree is a ProvableCountTree rather than a plain CountTree because the merk-tree internal-node counts are exactly what makes range queries O(log n): walk the boundary path between the lower and upper bound, sum sub-counts at each internal node along the way. (See Document Count Trees for the underlying mechanic.)

The value trees become CountTrees because the property-name ProvableCountTree's aggregate is computed by summing each value tree's count_value. For that aggregate to mean "total docs at this property" rather than "number of distinct values", each value tree's count_value must equal "docs at this exact value" — which is only true if (a) the terminal [0] CountTree contributes its doc count, and (b) every sibling under the value tree (continuation sub-property names like 'shape', etc.) contributes zero rather than the default 1-per-Tree. That's what NonCounted<NormalTree> is for.

Layout

Same byColor + byColorShape example as before, with the same three documents (A: (red, circle), B: (red, square), C: (blue, square)), but now byColor.range_countable: true:

flowchart TD
    DT["<b>'widget'</b><br/>(document type)<br/>NormalTree"]
    ColorKey["<b>'color'</b><br/><b><i>ProvableCountTree</i></b><br/>count = 3"]
    Red["<b>'red'</b><br/><b><i>CountTree</i></b><br/>count = 2"]
    Blue["<b>'blue'</b><br/><b><i>CountTree</i></b><br/>count = 1"]

    %% byColor terminals (unchanged shape — same as before)
    RedColorT["<b>[0]: CountTree</b><br/>count = 2<br/><i>byColor terminal</i>"]
    BlueColorT["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColor terminal</i>"]

    %% byColorShape continuation — now NonCounted to avoid double-counting
    RedShape["<b>'shape'</b><br/><b><i>NonCounted&lt;NormalTree&gt;</i></b>"]
    BlueShape["<b>'shape'</b><br/><b><i>NonCounted&lt;NormalTree&gt;</i></b>"]
    RedCircle["<b>'circle'</b><br/>NormalTree"]
    RedSquare["<b>'square'</b><br/>NormalTree"]
    BlueSquare["<b>'square'</b><br/>NormalTree"]

    %% byColorShape terminals
    RCT["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
    RST["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
    BST["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]

    %% References (one per matching index path, per the earlier section)
    RA1(["<b>doc_id_A</b><br/>Reference"])
    RB1(["<b>doc_id_B</b><br/>Reference"])
    RC1(["<b>doc_id_C</b><br/>Reference"])
    RA2(["<b>doc_id_A</b><br/>Reference"])
    RB2(["<b>doc_id_B</b><br/>Reference"])
    RC2(["<b>doc_id_C</b><br/>Reference"])

    DT --> ColorKey
    ColorKey --> Red
    ColorKey --> Blue

    Red --> RedColorT
    Red --> RedShape
    Blue --> BlueColorT
    Blue --> BlueShape

    RedColorT --> RA1
    RedColorT --> RB1
    BlueColorT --> RC1

    RedShape --> RedCircle
    RedShape --> RedSquare
    BlueShape --> BlueSquare

    RedCircle --> RCT --> RA2
    RedSquare --> RST --> RB2
    BlueSquare --> BST --> RC2

    classDef provableCount fill:#ede7f6,stroke:#311b92,color:#000
    classDef countTree fill:#fff4e5,stroke:#bf6900,color:#000
    classDef nonCounted fill:#f3e5f5,stroke:#6a1b9a,color:#000,stroke-dasharray:5 5
    classDef reference fill:#e8f5e9,stroke:#1b5e20,color:#000
    class ColorKey provableCount
    class Red,Blue,RedColorT,BlueColorT,RCT,RST,BST countTree
    class RedShape,BlueShape nonCounted
    class RA1,RB1,RC1,RA2,RB2,RC2 reference

Legend additions for this diagram: purple = ProvableCountTree; amber = CountTree; dashed lavender = NonCounted<*> (the new grovedb variants); rounded green = Reference.

Walking through how the counts add up:

  • 'red' (CountTree, count=2) — its children are [0] (CountTree, contributes its count_value = 2) and 'shape' (NonCounted<NormalTree>, contributes 0 — that's the whole point of the new variant). Aggregate = 2. ✓
  • 'blue' (CountTree, count=1) — same shape, 1 doc + 0. ✓
  • 'color' (ProvableCountTree, count=3) — its children are 'red' (CountTree, contributes 2) and 'blue' (CountTree, contributes 1). Aggregate = 3. The provable variant additionally stores per-internal-node counts inside its merk structure, which is what enables the range walk.

If 'shape' were a plain NormalTree instead of NonCounted<NormalTree>, it would contribute 1 to 'red' (every non-count-tree element contributes 1 by default — see Document Count Trees § How Counts Aggregate). Then 'red' would read as 3, 'blue' as 2, 'color' as 5 — a count of "docs + sub-property-trees", not "docs". The NonCounted<*> variant exists exactly to fix this.

Query — "count between two values"

With the layout above, a query like WHERE color BETWEEN 'red' AND 'tomato' resolves at the 'color' ProvableCountTree level:

  1. Walk the merk tree from 'color''s root, finding the boundary node between 'red' (lower bound) and 'tomato' (upper bound) — O(log distinct color values).
  2. At each step, decide what to do with the off-boundary subtree using its pre-computed count: include its full count_value (subtree fully inside the range), exclude (fully outside), or recurse (straddles the boundary).
  3. Sum the contributions; the result is the count of all docs whose color falls in [red, tomato].

No leaf-level enumeration of distinct color values, no enumeration of individual documents — the count is computed entirely from the tree's pre-aggregated structure.

Compound indexes

range_countable: true on a compound index applies at the index's terminating level (its last property). For byColorShape = [color, shape] with range_countable: true:

  • 'shape' (the property-name tree under each color value) becomes a ProvableCountTree.
  • Each 'circle' / 'square' value tree becomes a CountTree.
  • Documents are referenced as Element::Reference leaves under those CountTrees, contributing 1 each to the count aggregate.

When the compound's leading prefix is also indexed by another range_countable index (e.g. byColor is also range_countable), sibling continuations under each color CountTree are wrapped with Element::NonCounted so a doc routed via byColorShape doesn't double-count under byColor's color aggregate. The walker (add_indices_for_index_level_for_contract_operations) threads a parent_value_tree_is_range_countable flag down the recursion to decide when to wrap, regardless of whether the inner tree is itself a ProvableCountTree, CountTree, or plain NormalTree.

End-to-end coverage in range_countable_index_e2e_tests (in packages/rs-drive/src/drive/contract/insert/insert_contract/v0/mod.rs) pins the storage layout against a real grovedb — including the count_tree_value_count_excludes_compound_continuation_via_non_counted test that proves NonCounted-wrapping is load-bearing for compound-index correctness.

Range-Summable Indexes

Status: live as of grovedb develop (PR #670 merged; head e98bab5f as of this PR) (feat: add Element::ProvableCountProvableSumTree + dual-axis crossover proofs). Uses two grovedb element variants from that PR: Element::ReferenceWithSumItem(ReferencePathType, MaxReferenceHop, SumValue, Option<ElementFlags>) — a reference that also contributes an i64 sum to its parent sum tree — and Element::NotSummed<*> / Element::NotCountedOrSummed<*> wrappers that opt out of sum (or both sum and count) propagation. The pure-sum side reuses the existing SumTree / ProvableSumTree variants; the combined-axis case uses ProvableCountProvableSumTree. Carrier-aggregate sum proofs work end-to-end via GroveDb::verify_aggregate_sum_query_per_key — see Sum Index Examples Query 9 for the byte-counts.

range_summable is the sum-side counterpart of range_countable. Where countable / summable make point-lookup aggregates O(1), range_summable makes range-sum queries O(log n) — answering "what's the sum of price for widgets with color between red and tomato?" without enumerating every distinct color value or every individual document.

The shape is structurally parallel to range-countable, but the per-element contribution rules are inverted, and that asymmetry shapes the storage layout in a subtle but load-bearing way.

Constraints

  • range_summable: true requires summable: Some(<property>). Same additive relationship as range_countable / countable.
  • The named property must be type: integer and listed in required on the document type. The DPP validator enforces this at contract-creation time — without it, a missing-or-null value at insert would leave the reference with no sum contribution and silently underflow the ancestor sums on delete.
  • The same property name must be used consistently across the doctype: documents_summable (if set) and every per-index summable must name the same property. Grovedb's sum trees aggregate i64 per merk node without a per-tree property tag, so mixing properties would feed inconsistent contributions into the same merk hierarchy.
  • Combining range_summable with range_countable on the same index promotes the property-name tree to ProvableCountProvableSumTree (PCPS) rather than nesting two trees — both metrics live on the same merk root and can be queried atomically. See Combined: range-countable + range-summable below.

Mechanism

range_summable upgrades the same three levels range_countable does, with the sum analogues at each level:

LevelWithout range_summableWith range_summable
Property-name tree (e.g. 'color')NormalTreeProvableSumTree
Value tree (e.g. 'red', 'blue')NormalTreeSumTree
Terminal at [0] under each valueNormalTree / SumTree (per summable)unchanged — still driven by summable
Sibling continuations (compound-index suffixes inside the value tree)NormalTreeNormalTree — usually unwrapped (see below)

The property-name tree is a ProvableSumTree rather than a plain SumTree for the same reason range_countable upgrades to ProvableCountTree: per-internal-node aggregated sums are what make range walks O(log n). Walk the boundary path between the lower and upper bound, sum sub-sums at each off-boundary internal node along the way. (See Document Sum Trees for the underlying mechanic.)

The value trees become SumTrees because the property-name ProvableSumTree's aggregate is computed by combining each value tree's sum_value. For that aggregate to mean "total <property> at this color" rather than "first-byte-of-some-i64-garbage", each value tree's sum_value must equal the documented sum — which requires the leaf elements stored under each value tree to be sum-bearing.

That's where the layout diverges from count.

The contribution asymmetry: count auto-propagates, sum requires sum-bearing elements

Count trees automatically count every child element. A NormalTree, an Item, a Reference — each contributes +1 to the parent's count_value by default. That's why range_countable needs NonCounted<*> wrappers everywhere: to suppress an aggregation that would otherwise happen.

Sum trees behave the opposite way. Only sum-bearing element variants — SumItem, ItemWithSumItem, ReferenceWithSumItem, and the sum-bearing tree variants themselves — contribute to a parent SumTree's running sum. Item, Reference, plain NormalTree, CountTree — all contribute 0 by default. That has two consequences:

  1. Per-document contributions don't appear automatically. A plain Element::Reference under a SumTree does not propagate any sum. We need a different reference element — Element::ReferenceWithSumItem(path, max_hops, sum_value, flags) — that carries an explicit i64 sum contribution (the document's value at the summable property, frozen at insert time) alongside the usual reference-path bytes. Grovedb PR 670 adds this variant; Drive's index walker constructs it via make_document_reference_with_sum_item under any index path with summable.is_some().
  2. Sibling continuations usually don't need a wrapper. A NormalTree continuation under a sum-bearing value tree contributes 0 by default — exactly what we want. No NotSummed wrap required. The exception is when the continuation is itself sum-bearing (e.g. a deeper compound index that's also range_summable); in that case wrap the continuation in Element::NotSummed<*> to keep its sum from leaking into the outer index's aggregate. Compare with range_countable, where every continuation needs NonCounted because every non-count-aware element auto-contributes 1.

Layout

Extend the widget contract with a numeric price property and promote both indexes to the sum surface:

{
  "widget": {
    "type": "object",
    "documentsCountable": true,                  // unchanged — total widget count fast path
    "properties": {
      "brand":  { "type": "string",  "position": 0, "maxLength": 32 },
      "color":  { "type": "string",  "position": 1, "maxLength": 32 },
      "shape":  { "type": "string",  "position": 2, "maxLength": 32 },
      "price":  { "type": "integer", "position": 3, "minimum": 0 } // ← new, summable target
    },
    "required": ["brand", "color", "shape", "price"],
    "indices": [
      {
        "name": "byColor",
        "properties": [{ "color": "asc" }],
        "summable": "price",                     // ← aggregate `price` per color
        "rangeSummable": true                    // ← per-node sums, range-queryable
      },
      {
        "name": "byColorShape",
        "properties": [{ "color": "asc" }, { "shape": "asc" }],
        "countable": "countable",                // ← per-(color, shape) doc count at O(1)
        "summable": "price",                     // ← aggregate `price` per (color, shape)
        "rangeSummable": true                    // ← per-node sums on the `shape` terminator
      }
    ],
    "additionalProperties": false
  }
}

Both indexes name the same sum property — summable: "price" in both. The DPP validator requires this: grovedb's sum trees aggregate i64 per merk node with no per-tree property tag, so a contract that mixed summable: "price" and summable: "fee" on the same doctype would feed inconsistent contributions into the same merk hierarchy. price is type: integer and listed in required — both also enforced at contract-creation time.

byColorShape combines countable (root-only doc count per (color, shape) pair) with summable + rangeSummable (per-node sums of price). Drive's dispatch table promotes this combination to ProvableCountProvableSumTree (PCPS) at the value-tree and [0] terminal levels — the only grovedb variant carrying per-node sums also carries per-node counts as a side effect, so the count side gets per-node tracking "for free" even though only the sum side was opted into provability. See DocumentTypePrimaryKeyTreeType::primary_key_tree_type's v1 dispatch table for the full mapping.

The two indexes share the color prefix exactly as the count examples did, so the same shared-prefix layout still applies. What changes is the element types at every level from 'color' downward — and the diagram below makes the compound case visible, because the 'shape' continuation under each color is now itself a sum-bearing tree (since byColorShape is rangeSummable) and needs Element::NotSummed<*>-wrapping to keep its aggregate from leaking into the outer byColor sum.

Document fixtures, three widgets: A: (brand_acme, red, circle, price=10), B: (brand_acme, red, square, price=20), C: (brand_acme, blue, square, price=30). The on-disk layout:

flowchart TD
    DT["<b>'widget'</b><br/>(document type)<br/>NormalTree"]
    ColorKey["<b>'color'</b><br/><b><i>ProvableSumTree</i></b><br/>sum = 60"]
    Red["<b>'red'</b><br/><b><i>SumTree</i></b><br/>sum = 30"]
    Blue["<b>'blue'</b><br/><b><i>SumTree</i></b><br/>sum = 30"]

    %% byColor terminals — SumTree, refs carry per-doc sum contributions
    RedColorT["<b>[0]: SumTree</b><br/>sum = 30<br/><i>byColor terminal</i>"]
    BlueColorT["<b>[0]: SumTree</b><br/>sum = 30<br/><i>byColor terminal</i>"]

    %% byColorShape continuation — now itself sum-bearing (rangeSummable),
    %% so it must be NotSummed-wrapped to contribute 0 to the parent
    %% byColor SumTree. The wrapped inner ProvableSumTree still works
    %% normally for byColorShape queries that descend through it.
    RedShape["<b>'shape'</b><br/><b><i>NotSummed&lt;ProvableSumTree&gt;</i></b><br/>contributes 0 to red's sum<br/>inner sum = 30 for byColorShape queries<br/>(no per-node count: rangeCountable not set)"]
    BlueShape["<b>'shape'</b><br/><b><i>NotSummed&lt;ProvableSumTree&gt;</i></b><br/>contributes 0 to blue's sum<br/>inner sum = 30 for byColorShape queries<br/>(no per-node count: rangeCountable not set)"]
    RedCircle["<b>'circle'</b><br/><b><i>PCPS</i></b><br/>count = 1, sum = 10"]
    RedSquare["<b>'square'</b><br/><b><i>PCPS</i></b><br/>count = 1, sum = 20"]
    BlueSquare["<b>'square'</b><br/><b><i>PCPS</i></b><br/>count = 1, sum = 30"]

    %% byColorShape terminals — now PCPS (carry both per-node count
    %% and per-node sum). References below contribute both axes.
    RCT["<b>[0]: PCPS</b><br/>count = 1, sum = 10<br/><i>byColorShape</i>"]
    RST["<b>[0]: PCPS</b><br/>count = 1, sum = 20<br/><i>byColorShape</i>"]
    BST["<b>[0]: PCPS</b><br/>count = 1, sum = 30<br/><i>byColorShape</i>"]

    %% References — every leaf is now ReferenceWithSumItem because both
    %% indexes are summable. Each document is stored under both
    %% byColor[color] and byColorShape[color, shape], so the same
    %% per-doc price contribution lands twice in the diagram — once
    %% per index that covers the document.
    RA1(["<b>doc_id_A</b><br/>ReferenceWithSumItem<br/>sum=10"])
    RB1(["<b>doc_id_B</b><br/>ReferenceWithSumItem<br/>sum=20"])
    RC1(["<b>doc_id_C</b><br/>ReferenceWithSumItem<br/>sum=30"])
    RA2(["<b>doc_id_A</b><br/>ReferenceWithSumItem<br/>sum=10"])
    RB2(["<b>doc_id_B</b><br/>ReferenceWithSumItem<br/>sum=20"])
    RC2(["<b>doc_id_C</b><br/>ReferenceWithSumItem<br/>sum=30"])

    DT --> ColorKey
    ColorKey --> Red
    ColorKey --> Blue

    Red --> RedColorT
    Red --> RedShape
    Blue --> BlueColorT
    Blue --> BlueShape

    RedColorT --> RA1
    RedColorT --> RB1
    BlueColorT --> RC1

    RedShape --> RedCircle
    RedShape --> RedSquare
    BlueShape --> BlueSquare

    RedCircle --> RCT --> RA2
    RedSquare --> RST --> RB2
    BlueSquare --> BST --> RC2

    classDef provableSum fill:#e3f2fd,stroke:#0d47a1,color:#000
    classDef sumTree fill:#e8eaf6,stroke:#1a237e,color:#000
    classDef pcps fill:#ede7f6,stroke:#311b92,color:#000,stroke-width:2px
    classDef notSummed fill:#fce4ec,stroke:#880e4f,color:#000,stroke-dasharray:5 5
    classDef refSum fill:#c8e6c9,stroke:#1b5e20,color:#000,stroke-width:2px
    class ColorKey provableSum
    class Red,Blue,RedColorT,BlueColorT sumTree
    class RedCircle,RedSquare,BlueSquare,RCT,RST,BST pcps
    class RedShape,BlueShape notSummed
    class RA1,RB1,RC1,RA2,RB2,RC2 refSum

Legend additions for this diagram: light blue = ProvableSumTree; indigo = SumTree; purple-outline = ProvableCountProvableSumTree (PCPS — per-node count and per-node sum); dashed pink = NotSummed<*> (contributes 0 to the parent's sum despite carrying its own internal aggregate); bold green = ReferenceWithSumItem.

Walking through how the aggregates layer:

byColor's view (read at the 'color' ProvableSumTree root, sum=60):

  • 'red' (SumTree, sum=30) — children are [0] (SumTree, contributes its sum_value = 30) and 'shape' (NotSummed<ProvableSumTree>, contributes 0 — that's the whole point of the wrapper, even though its own internal aggregate is also 30 for byColorShape queries). Aggregate = 30. ✓
  • 'blue' (SumTree, sum=30) — same shape: [0] contributes 30, 'shape' contributes 0. ✓
  • 'color' (ProvableSumTree, sum=60) — children are 'red' (SumTree, 30) and 'blue' (SumTree, 30). Aggregate = 60. The provable variant additionally stores per-internal-node sums inside its merk structure, which is what enables the range walk.

byColor is pure-sum (no countable flag) so the value trees here stay SumTree — there's no count aggregation at this layer.

byColorShape's view (descends through the NotSummed wrapper rather than reading it; the inner ProvableSumTree aggregates the PCPS value trees beneath):

  • 'red' → 'shape' (ProvableSumTree, inner sum=30) — children are 'circle' (PCPS, count=1 sum=10) and 'square' (PCPS, count=1 sum=20). Inner aggregate = 30. Note that 'shape' is ProvableSumTree rather than PCPS: only rangeSummable is set on byColorShape, not rangeCountable, so the property-name level ('shape') aggregates sums per-node but doesn't track per-node counts.
  • 'blue' → 'shape' (ProvableSumTree, inner sum=30) — single child 'square' (PCPS, count=1 sum=30). Inner aggregate = 30.
  • Point lookup SELECT COUNT(*), SUM(price) WHERE color = 'red' AND shape = 'circle' reads the PCPS value tree directly — both metrics in one element read (count=1, sum=10), no traversal.
  • Range query SELECT SUM(price) WHERE color = 'red' AND shape BETWEEN 'a' AND 'z' walks the red 'shape' ProvableSumTree's boundary and recovers sum=30 via AggregateSumOnRange in O(log distinct shape values). Range-count over the same boundary isn't supported (would need rangeCountable: true to promote 'shape' to PCPS at the property-name level); range-count proofs over shape would need to enumerate the value-tree count_values manually.
Why PCPS at the value level

PCPS is grovedb's only tree variant carrying per-node sums. When an index sets countable: "<tier>" + summable + rangeSummable, the dispatch table promotes the value tree to PCPS because there's no "ProvableSumCountTree" variant (per-node sum + root-only count) to land on. The count side gets per-node tracking "for free" — same storage cost as ProvableCountSumTree's count-half since PCPS commits the same per-node count metadata. See primary_key_tree_type.rs's v1 dispatch table for the full mapping.

byColor, by contrast, has only summable + rangeSummable (no countable), so its value trees stay SumTree — root-only sum, no count tracking, no upgrade. The two indexes living side by side on the same widget contract show both sides of the dispatch.

Why the NotSummed<*> wrap is still needed

The NotSummed<*> wrap is what keeps the two index views consistent. byColorShape's 'shape' subtree carries its own internal aggregate (30 at red, 30 at blue); byColor must not let those aggregates leak into its color sums. The wrapper makes 'shape' contribute exactly 0 to its parent 'red' / 'blue' SumTrees, so byColor reads from the [0] ref-bucket alone. Without the wrap, 'red' would read as 60 = 30 (refs) + 30 (the shape subtree's leaked aggregate), and any document covered by both indexes would be double-counted in byColor's aggregate.

Compare with the range-countable diagram above: there, the 'shape' continuations needed NonCounted<NormalTree> wrapping because a plain NormalTree auto-contributes +1 to a parent CountTree. Here the wrapper does conceptually the same job — suppress the would-be propagation — but for sum aggregation rather than count aggregation, and the wrapped variant is NotSummed<ProvableSumTree> because the continuation is itself sum-bearing (which is the only case where a sum wrapper is needed; plain NormalTree continuations naturally contribute 0 to a SumTree and don't need wrapping at all — see the asymmetry note above).

Query — "sum between two values"

A query like SELECT SUM(price) WHERE color BETWEEN 'red' AND 'tomato' resolves at the 'color' ProvableSumTree level via grovedb's AggregateSumOnRange primitive:

  1. Walk the merk tree from 'color''s root, finding the boundary node between 'red' (lower bound) and 'tomato' (upper bound) — O(log distinct color values).
  2. At each step, decide what to do with the off-boundary subtree using its pre-computed sum: include its full sum_value (subtree fully inside the range), exclude (fully outside), or recurse (straddles the boundary).
  3. Sum the contributions; the result is the total price across all docs whose color falls in [red, tomato].

No leaf-level enumeration of distinct color values, no enumeration of individual documents — the sum is computed entirely from the tree's pre-aggregated structure, exactly mirroring AggregateCountOnRange. The verifier counterpart is GroveDb::verify_aggregate_sum_query(proof, path_query, grove_version) -> Result<([u8; 32], i64), Error> returning (root_hash, aggregated_sum). (The sum is signed because grovedb's SumTree value type is i64. For tip-jar-style non-negative aggregations this stays ≥ 0 in practice; the verifier surfaces overflow into negative space as a distinct error rather than silently wrapping.)

Compound indexes

range_summable: true on a compound index applies at the index's terminating level (its last property). For an index byCategoryPrice = [category, price] with summable: "price" and range_summable: true:

  • 'price' (the property-name tree under each category value) becomes a ProvableSumTree.
  • Each price-value tree becomes a SumTree.
  • Documents are stored as Element::ReferenceWithSumItem leaves under those SumTrees, contributing their price to the sum aggregate.

When the compound's leading prefix is also an index that's range_summable (e.g. a separate byCategory index that's also summable on price), sibling continuations under each category SumTree need Element::NotSummed<*>-wrapping iff the continuation is itself sum-bearing — otherwise the inner sum-tree's aggregate would leak into the outer index's value-tree sum, double-counting documents that route through both indexes. The walker (add_indices_for_index_level_for_contract_operations) threads the parent value tree's aggregation flags down the recursion to decide when to wrap.

Combined: range-countable + range-summable

Setting both range_countable: true AND range_summable: true on the same index doesn't produce two separate trees — grovedb PR 670 adds a dedicated ProvableCountProvableSumTree (PCPS) variant that commits both per-node counts AND per-node sums to every internal merk node. A single tree carries both metrics, and three range primitives become available against it:

  • AggregateCountOnRange — recovers just the count
  • AggregateSumOnRange — recovers just the sum
  • AggregateCountAndSumOnRange (PCPS-only, new in PR 670) — recovers BOTH from a single merk traversal, verified via GroveDb::verify_aggregate_count_and_sum_query(...) -> Result<([u8; 32], u64, i64), Error> returning (root_hash, count, sum)

The combined primitive is strictly cheaper than running two separate range queries: one proof envelope, one merk walk, and both metrics atomically bound to the same root hash (so they can't drift relative to each other across a concurrent write).

The full dispatch table mapping (countable, range_countable, summable, range_summable) combinations to grovedb tree variants lives in DocumentTypePrimaryKeyTreeType::primary_key_tree_type's v1 arm; the index-walker dispatch in add_indices_for_index_level_for_contract_operations follows the same table at every recursion level.

End-to-end coverage for the sum surface lives in packages/rs-drive/benches/document_sum_worst_case.rs's tip-jar fixture (paralleling the count side's document_count_worst_case.rs widget bench), with the worked-example queries in Sum Index Examples.

Tree Type at the Terminal Level

The decision happens in add_reference_for_index_level_for_contract_operations/v0/mod.rs:

#![allow(unused)]
fn main() {
if !index_type.index_type.is_unique() || any_fields_null {
    // Non-unique branch: insert an empty tree at [0], then put
    // each document's reference inside that tree. The tree's variant
    // is governed by `countable`:
    //   NotCountable             → NormalTree
    //   Countable                → CountTree
    //   CountableAllowingOffset  → ProvableCountTree
} else {
    // Unique branch: store a single Reference element at [0] directly.
}
}

So the matrix:

uniqueany_fields_nullcountableWhat lives at [0]
false(any)NotCountableempty NormalTree containing per-doc references
false(any)Countableempty CountTree containing per-doc references
false(any)CountableAllowingOffsetempty ProvableCountTree containing per-doc references
truefalse(any)bare Reference to the one matching document
truetrueNotCountableempty NormalTree containing per-doc references
truetrueCountableempty CountTree containing per-doc references
truetrueCountableAllowingOffsetempty ProvableCountTree containing per-doc references

Note the last three rows: a unique index does go through the count-tree branch when any indexed field is null. That's why countable on a unique index is meaningful exactly when at least one of the indexed properties is optional in the schema.

Visualizing the three terminal shapes side by side:

flowchart TD
    subgraph SA["Non-unique, countable"]
        direction TB
        A1["[..., color, 'red']"]
        A2["<b>[0]: CountTree</b><br/>count = 2"]
        A3(["<b>doc_id_A</b><br/>Reference"])
        A4(["<b>doc_id_B</b><br/>Reference"])
        A1 --> A2 --> A3
        A2 --> A4
    end

    subgraph SB["Unique, all fields non-null"]
        direction TB
        B1["[..., email, 'alice@x']"]
        B2(["<b>[0]: Reference</b><br/>→ doc_id_X"])
        B1 --> B2
    end

    subgraph SC["Unique with null in path"]
        direction TB
        C1["[..., a, 'X', b, &lt;empty&gt;]"]
        C2["<b>[0]: CountTree</b><br/>count = 1"]
        C3(["<b>doc_id_W</b><br/>Reference"])
        C1 --> C2 --> C3
    end

    classDef countTree fill:#fff4e5,stroke:#bf6900,color:#000
    classDef reference fill:#e8f5e9,stroke:#1b5e20,color:#000
    class A2,C2 countTree
    class A3,A4,B2,C3 reference

Same convention as the layout diagram above: rectangles are tree-type elements, rounded green nodes are Reference elements. Same key ([0]) at the terminal in all three panels — what lives there is what differs. The middle case is the one that's "special" — a bare Reference directly at [0] instead of a sub-tree containing references — and it's specifically scoped to the unique-and-no-nulls scenario.

Null Handling

The any_fields_null and all_fields_null flags follow the path Drive descends during insertion: each sub-level gets its parent's flags combined with its own value (add_indices_for_index_level_for_contract_operations/v2/mod.rs):

#![allow(unused)]
fn main() {
let sub_level_any_fields_null = any_fields_null || document_index_field.is_empty();
let sub_level_all_fields_null = all_fields_null && document_index_field.is_empty();
}

any_fields_null becomes true the moment the walker hits any null/empty value on an index's path (first, middle, or last property) and stays true below it. all_fields_null only stays true if every value on the path so far is null. A sibling sub-level's value never enters them: under the index trie's shared levels, each index is judged by its own properties.

Before protocol version 14 the walkers updated the flags in place as they moved from one sibling sub-level to the next, so a sibling's missing value also marked every later sibling's path: a unique index could then hold its entry in the sub-tree shape with none of its own values missing, and a nullSearchable: false index could get an entry for a document missing all of its values because a sibling had one. The replace walker of those versions also chose a unique index's shape from all_fields_null, so a replace left an index with some of its values missing in the bare-Reference shape. From version 14, where one of those earlier writers could disagree with the rule, the delete walker and the replace walker read what is stored at [0] (and, for a nullSearchable: false index the rule skips, whether an entry is there) and remove or refresh the entry where it is. The read is unbilled bookkeeping, like the time-range walkers' removability reads, so a dry run and the applied operation bill the same; a type with a ttl, which exists only from version 14, skips it.

By the time the recursion reaches the terminal:

  • any_fields_null = false and the index is unique → unique branch (bare Reference).
  • any_fields_null = true (regardless of unique) → non-unique-style branch (sub-tree containing references).
  • all_fields_null = true AND null_searchable = false → the terminal call returns early without inserting anything; this document is not findable through this index.

This means different documents under the same unique index can land in different storage shapes depending on which of their indexed fields are null. A document with all required fields populated takes the bare-Reference shape; a document with a null in an optional indexed property takes the sub-tree shape, side by side under the same index.

A skipIfAbsent index is decided before any of this: a document missing a property of the index's skip set takes no part in it (document_takes_part_in_index), so the terminal call is not made for it at all. The walkers also build a level only when an index the document takes part in ends at or below it (level_reaches_entry), so a skipped index leaves no tree of its own behind, while a level it shares with a taking-part index is built as usual. The delete walkers apply the same predicates to the stored document. A replace compares the old and the new document's participation per index: it removes the old reference only if the old version took part and writes the new one (with the trees it hangs from) only if the new version does, so a replace that adds or drops a skip property moves the document into or out of the index and leaves exactly the tree a fresh insert of the new version would.

Insert Flow Summary

Putting it together, when Drive inserts a document into a contract C of type T:

  1. add_indices_for_top_index_level_for_contract_operations — for each top-level entry in the document type's index trie (each first-property of any declared index), pushes the property name and the document's value for that property onto the path, computes the initial any_fields_null / all_fields_null for that single value, and recurses.
  2. add_indices_for_index_level_for_contract_operations (recursive) — for each sub-level of the trie, pushes the property name and value onto the path, derives the sub-level's any_fields_null (OR) and all_fields_null (AND) from its parent's flags and its own value, and recurses. If the current level has has_index_with_type = Some(...), it also calls into step 3 before recursing further (because an index can terminate at a non-leaf trie level when another index continues past it).
  3. add_reference_for_index_level_for_contract_operations — the terminal call. Decides between unique and non-unique-style storage using the matrix above; for the non-unique-style path it picks a NormalTree / CountTree / ProvableCountTree based on countable; finally inserts the document reference (or sub-tree containing it).

Deletion mirrors the same walk in reverse — see packages/rs-drive/src/drive/document/delete/.

Query Traversal

When a query arrives at drive-abci, the document-query construction path picks one of the document type's indexes that "covers" the query — i.e., whose property prefix matches the query's equality clauses, in order. The picker is in packages/rs-drive/src/query/mod.rs (look for fn construct_path_query and the index-selection helpers it calls). For count queries specifically there's a separate, count-tree-aware picker (drive_document_count_query/mod.rs) — see Document Count Trees for that path.

Once an index is picked, the query-engine builds a PathQuery whose path is exactly the prefix shape the insert code produced: [DataContractDocuments, contract_id, 1, doc_type, prop, value, prop, value, …]. GroveDB then walks the path in O(log n per level), reading the terminal sub-tree (or single reference) and returning matching documents.

A query whose where-clauses don't form a prefix of any index can't take this fast path and falls back to a full-scan plan — which dapi-grpc surfaces as an error in most cases, since unbounded scans are deliberately discouraged.

Choosing Index Settings

Quick checklist for contract authors:

  • Don't index what you won't query. Each index costs storage on every insert/delete and counts against the per-document-type index limit (10 indexes per type currently).
  • Order index properties from most-selective to least-selective. A [country, city] index is more useful than [city, country] for queries like where country = "FR".
  • unique: true when the platform should reject duplicates at the consensus layer. This is the right place for "this should be unique" invariants — don't enforce them application-side.
  • countable: "countable" when you'll regularly call GetDocumentsCount with == (or in) clauses on exactly this index's properties. Adds a constant-factor overhead on insert/delete; reads become O(1). A countable: true index counts only queries whose where clauses match its properties exactly — partial-prefix queries are rejected with WhereClauseOnNonIndexedProperty, not falling through to a slow scan. Define a separate index per distinct count-query shape you want to support, or set documentsCountable: true on the document type for unfiltered totals.
  • countable: "countableAllowingOffset" when you'll also want offset / range queries on this index in a future release. Strictly more expensive than plain "countable"; only worth it if you need the capability.
  • null_searchable: true (the default) is right for almost all cases. Set to false only when documents with all-null indexed values shouldn't be findable through this index — typically a niche optimization to avoid a hot all-null prefix.

For specifically count-related concerns — primary-key-tree flags (documentsCountable / rangeCountable), the no-prove-vs-prove paths, and the operator restrictions — see the dedicated Document Count Trees chapter.

Document Count Trees

Counting the documents that match a query used to mean fetching them and calling .len(). From protocol v12 (Platform 3.1) onward, document types can opt into a different primary-key tree variant that maintains a running count inside the tree itself, turning count(*)-style queries into an O(1) lookup. This chapter explains the three tree variants, how a document type selects one, and the two query endpoints that expose the feature.

Why Count Trees Exist

The default primary-key tree for a document type is a NormalTree. To count the documents in it, Drive walks the subtree, deserializes every record, and returns the length of the resulting collection. That is fine for small types but becomes painful as soon as a UI needs "how many widgets are there?" on a contract with millions of widgets.

GroveDB has two count-aware tree variants. Both are provable — the count is committed to the Merkle root in each case — but they differ in where counts are stored inside the tree, and that controls which kinds of count queries can be answered without enumerating leaves:

  • CountTree — stores a single u64 count, at the root of the tree. The total document count is one read; any per-subtree count requires walking down to that subtree's root and reading its (separate) tree element.
  • ProvableCountTree — stores a u64 count at every internal node, not just the root. Each node's count covers everything in the subtree below it, so range queries like "how many items between key A and key B?" or "how many items per value of an indexed property?" can be answered by walking the boundary nodes and summing their pre-computed counts, without touching any leaf.

GroveDB merk trees are binary — each internal node has exactly a left and a right child:

The dashed box is the wrapping Element (the "tree" in grovedb terms) and contains the root node of the merk tree. Both variants store the total count on the wrapping element — that's the O(1) field Drive reads for total counts. The difference is what's inside: in a CountTree the merk root and the rest of the tree don't carry the count, so only the wrapper has it. In a ProvableCountTree the count is also stored on the root node itself and on every internal merk-tree node, so it's committed into the merk root hash and provable per-subtree.

flowchart LR
  subgraph CT ["CountTree"]
    direction TB
    subgraph CT_ELEM ["Tree element c=3"]
      direction TB
      A["root"]:::node
    end
    A --> B["·"]:::node
    A --> C["x"]:::leaf
    B --> D["x"]:::leaf
    B --> E["x"]:::leaf
  end

  subgraph PCT ["ProvableCountTree"]
    direction TB
    subgraph PCT_ELEM ["Tree element c=3"]
      direction TB
      H["root c=3"]:::countnode
    end
    H --> I["c=2"]:::countnode
    H --> J["c=1"]:::leaf
    I --> K["c=1"]:::leaf
    I --> L["c=1"]:::leaf
  end

  CT ~~~ PCT

  classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef countnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
  classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;

  style CT_ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb
  style PCT_ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb

In a CountTree, the only count-bearing node is the root. To compute "how many items per value of property P?" you'd have to navigate to each value-keyed subtree (a separate grovedb tree, not a child node of the binary structure above), read its root count, and pay for a separate proof per read — N reads for N distinct values. In a ProvableCountTree, every internal node along the binary path already carries the count of its left and right subtrees, so a range query like "items in [a, b]" or "items per value of P" walks only the boundary path and sums the pre-committed sub-counts in a single traversal and a single proof.

A document type opts in via two schema flags:

  • documentsCountable: true → primary-key tree is a CountTree. Enables O(1) total-count for the document type; sufficient for GetDocumentsCount with no where filter.
  • rangeCountable: true → primary-key tree is a ProvableCountTree. Implies documentsCountable. The same flag is also accepted per-index, where it controls range-count storage layout (see below) and is required for any GetDocumentsCount request that carries a range where-clause.

How a Document Type Picks Its Tree Variant

Selection lives in packages/rs-drive/src/drive/document/primary_key_tree_type.rs:

#![allow(unused)]
fn main() {
impl DocumentTypePrimaryKeyTreeType for DocumentTypeRef<'_> {
    fn primary_key_tree_type(
        &self,
        platform_version: &PlatformVersion,
    ) -> Result<TreeType, Error> {
        match platform_version
            .drive
            .methods
            .document
            .primary_key_tree_type
        {
            0 => {
                if self.range_countable() {
                    Ok(TreeType::ProvableCountTree)
                } else if self.documents_countable() {
                    Ok(TreeType::CountTree)
                } else {
                    Ok(TreeType::NormalTree)
                }
            }
            version => Err(Error::Drive(DriveError::UnknownVersionMismatch {
                method: "DocumentTypeRef::primary_key_tree_type".to_string(),
                known_versions: vec![0],
                received: version,
            })),
        }
    }
}
}

primary_key_tree_type() is the single source of truth — every Drive code path that needs to know which tree variant to read from or write to routes through this helper, including:

  • Contract insert and update (to CREATE the right tree element when the document type is added).
  • Document insert / delete (to know how to update the count alongside the document).
  • Cost estimation (so fees match the variant that will actually be used).

The contract insert/update paths use three thin Drive helpers parallel to the existing batch_insert_empty_tree / batch_insert_empty_sum_tree:

  • batch_insert_empty_tree — NormalTree.
  • batch_insert_empty_count_tree — CountTree, used when documents_countable() && !range_countable().
  • batch_insert_empty_provable_count_tree — ProvableCountTree, used when range_countable().

Each helper goes through LowLevelDriveOperation::for_known_path_key_empty_*_tree (or its _estimated_path_key_* cousin in cost-estimation paths), so the contract setup, document operations, and proof generation all see the same on-disk shape.

Storage-Layout Invariants

Because the tree variant is fixed at contract-creation time and baked into how the tree element is laid out on disk, two flags are immutable across a contract update:

  • Changing documents_countable from any state to any other state on a validate_config update returns DocumentTypeUpdateError.
  • Same for range_countable.

Tests pinning these guards live in packages/rs-dpp/src/data_contract/document_type/methods/validate_update/v0/mod.rs. Don't relax them: if a NormalTree-backed document type were silently switched to CountTree mid-contract, every subsequent insert or delete would update a count value attached to a tree element that physically isn't a count tree, leading to grovedb element-shape errors at best and consensus drift at worst.

Counting Documents at Query Time

A single unified gRPC endpoint exposes the feature: GetDocumentsCount. The response shape varies by request mode (total / per-In-value / per-distinct-value-in-range / total-over-range), see Range Modes below. The wire-level shape makes that split explicit: on the no-proof path the response's CountResults carries an inner oneof variant { uint64 aggregate_count; CountEntries entries; } — total-count and range-without-distinct modes return aggregate_count (a single u64), per-In-value and per-distinct-value-in-range modes return entries (a list of CountEntry { optional bytes in_key; bytes key; uint64 count } where in_key is the prefix value for compound In + range shapes and absent for flat queries). The endpoint has two underlying paths (prove vs. no-prove); every mode — including return_distinct_counts_in_range = true — is valid on both paths. The prove path uses two different proof shapes depending on whether you want a single aggregate or per-distinct-value entries (see Prove (Client-Side Verify-Then-Aggregate or Aggregate-Count Proof) below).

No-Prove (Server-Side O(1) or O(log n))

When prove=false, drive-abci calls into DriveDocumentCountQuery (in packages/rs-drive/src/query/drive_document_count_query/mod.rs). The handler picks a path based on the where clauses:

Unfiltered total (no where clauses) on a documentsCountable: true document type (read_primary_key_count_tree):

The doctype's primary-key tree at [contract_doc, contract_id, 1, doctype, 0] is itself a CountTree. One grovedb read gives count_value — the total document count. O(1).

Equal/In only (execute_no_proof):

  1. Pick a countable: true index whose properties exactly match the Equal/In where-clause fields — every index property has a matching clause, no orphan clauses, no uncovered properties. If no such index exists the request rejects with WhereClauseOnNonIndexedProperty (the strict-coverage contract; see "Index design" below).
  2. Walk the tree from the root down to the terminal level, pushing prop_name and serialize_value_for_key(prop_name, value) at each step. Equal extends one path; In clones the current path once per value in its array (a cartesian fork) and the per-branch counts are summed.
  3. Read the CountTree element at the resulting path and return its count_value. O(1) per branch.

If the request carries an In clause, the response is the entries variant — one CountEntry per In value (the per-value split mode). Otherwise the response is the aggregate_count variant — a single u64.

Index design contract: a countable: true index counts exactly its declared properties. Want count(*) WHERE color = X? Define a [color] countable index. Want count(*) WHERE color = X AND shape = Y? Define a [color, shape] countable index. Want both? Define both. Partial coverage (e.g. color = X against a [color, shape] index) is rejected — define a more specific countable index, or set documentsCountable: true on the document type for unfiltered total counts. The prove path enforces the same contract, so prove=true and prove=false reject in the same situations with the same error.

Range (execute_range_count_no_proof):

  1. Pick a range_countable: true index where the Equal/In clauses cover the prefix and the range operator hits the index's last property.
  2. Build the path [contract_doc, doctype, prefix..., range_prop_name] — pointing at the property-name ProvableCountTree.
  3. Issue a grovedb path query with the converted range QueryItem (>, >=, <, <=, Range, RangeInclusive, RangeAfter, RangeAfterTo, RangeAfterToInclusive) and walk the children whose keys lie inside the range.
  4. Each child's count_value_or_default() is the doc count at that property value. Either sum all per-value counts and return as the aggregate_count variant (summed mode), or emit them as per-value CountEntrys under the entries variant (distinct mode), then apply order / cursor / limit.

Prove (Client-Side Verify-Then-Aggregate or Aggregate-Count Proof)

When prove=true, the proof shape depends on whether the query carries a range clause.

With a range clause: the handler picks one of two prove sub-paths based on return_distinct_counts_in_range:

  • Aggregate (return_distinct_counts_in_range = false, default): drive-abci builds a grovedb AggregateCountOnRange path query against the property-name ProvableCountTree, and get_proved_path_query produces an aggregate-count proof. The client verifies via GroveDb::verify_aggregate_count_query and recovers (root_hash, count) directly — proof size is O(log n) regardless of how many keys match. No documents are ever materialized.

  • Distinct (return_distinct_counts_in_range = true): drive-abci builds a regular range path query (no AggregateCountOnRange wrapper) against the same ProvableCountTree. Because the leaf is a ProvableCountTree, merk emits one Node::KVCount(key, value, count) op per matched in-range key, with each count cryptographically committed to the merk root via node_hash_with_count(kv_hash, l_hash, r_hash, count) — same forge-resistance as the aggregate path's HashWithCount collapse. The SDK's drive_proof_verifier::verify_distinct_count_proof runs the standard hash-chain check, then walks the proof's op stream to extract the counts as a BTreeMap<Vec<u8>, u64>. Trade-off vs. the aggregate path: proof size is O(distinct values matched) rather than O(log n), because each distinct in-range key emits its own KVCount op instead of being collapsed into a boundary subtree. Acceptable for typical histograms (a few dozen distinct values in range); for "give me a single count" use the aggregate path instead.

Without a range clause (point-lookup with prove): two sub-paths based on the request shape.

  • Unfiltered total + documentsCountable: true: drive-abci proves the doctype's primary-key CountTree element at [contract_doc, contract_id, 1, doctype, 0]. One merk path proof; the SDK's drive_proof_verifier::verify_primary_key_count_tree_proof reads count_value off the verified element. O(log n) bytes.

  • Equal/In against a fully-covering countable: true index: drive-abci proves one Element::CountTree per covered branch. Two sub-shapes:

    • Equal-only fully-covered → one element at [..., last_field, last_value, 0].
    • In at any index position (with any number of trailing Equals) → one element per In value, fetched via outer Query + a subquery whose set_subquery_path carries the post-In Equal segments (zero of them when In is on the last property; one or more when In sits earlier in the index). The subquery's Key([0]) picks off the CountTree at [..., in_field, in_value, <trailing equals>, 0] for each matched In branch.

    The In position rule for count queries is more permissive than the regular document query path's Index::matches rule (which restricts In to last-or-before-last because of a positional path-construction assumption — see DriveDocumentQuery::get_non_primary_key_path_query for the layout that forces it). The count path doesn't have that constraint: there's no document-key terminator descent, no order_by interpretation, and no limit/offset semantics — it's a pure CountTree-element lookup, so set_subquery_path with an arbitrary trailing tail works. Both no-proof and prove count executors route through a single point_lookup_count_path_query builder (no-proof runs the path query via grove.query and sums the emitted CountTree elements' counts; prove signs the same path query via get_proved_path_query), so they accept the same query shapes by construction. The SDK's drive_proof_verifier::verify_point_lookup_count_proof verifies and extracts count_value_or_default() from each verified element.

Both sub-paths share the proof shape: each CountTree element's count_value is cryptographically bound to the merk root via node_hash_with_count(kv_hash, l_hash, r_hash, count), same forge-resistance guarantee the range-distinct path relies on. Neither materializes documents or runs per-key bookkeeping client-side.

Proof size: O(k × log n) where k is the number of covered branches (1 for the documents_countable fast path and Equal-only fully-covered case; ≤ |In values| for Equal-prefix + In-on-last).

Symmetric rejection contract: prove count requires a countable: true index whose properties exactly match the where clauses — same requirement as the no-proof Total / PerInValue modes. Partial coverage (where the where clauses are a strict prefix of the index, or the index has uncovered properties) rejects with a WhereClauseOnNonIndexedProperty-class error pointing the caller at the index-design fix. The documents_countable: true fast path handles unfiltered total counts in O(log n) proof bytes when set on the document type. No silent fallback to materializing matching documents — that path doesn't exist anymore.

Implementation reference:

Supported Where Operators

The no-prove fast path covers three operator shapes:

  • Equal (==) — single point lookup against the count tree at a fully-resolved index path. Picked by find_countable_index_for_where_clauses.
  • In (in) — cartesian fork. Each value in the In array becomes its own index path; their counts are summed (or, for split counts, merged by split key). An In clause with k values costs k point lookups, not a tree walk. The In clause also doubles as the per-value split signal in the unified GetDocumentsCount endpoint — at most one In per request.
  • Range (>, >=, <, <=, between*, startsWith) — walks the property-name ProvableCountTree's children whose keys lie inside the range, reading each child CountTree's count value. Picked by find_range_countable_index_for_where_clauses; requires the index to have range_countable: true AND the range property to be the index's last property (the IndexLevel terminator). startsWith "p" becomes the half-open range [serialize("p"), serialize("p") with last byte +1) — the same byte-incremented encoding the normal docs path uses (see conditions.rs's StartsWith arm), valid for UTF-8 string keys since UTF-8 never contains 0xFF.

Through the unified GetDocumentsCount request handler, range queries take a single range terminator clause plus a prefix of Equal clauses and/or one In clause. In on a prefix property exercises grovedb's native subquery primitive — each emitted entry then carries both the in_key (the In value for that fork) and the key (the terminator value within the range). Per-fork counts are NOT merged server-side — see No-Merge Compound Semantics below for rationale.

Range Modes

A range query in the unified endpoint produces one of two response shapes, controlled by return_distinct_counts_in_range:

  • return_distinct_counts_in_range = false (default) — CountResults.aggregate_count carrying the sum of the per-value CountTree counts within the range. Use for "how many widgets have color in [red, tomato]?".
  • return_distinct_counts_in_range = true — CountResults.entries with one CountEntry per distinct property value within the range (key = serialized terminator value, count = CountTree count for that value, in_key = the In-fork value for compound queries or absent for flat queries). Use for "show me a histogram of widgets by color in [red, tomato]".

No-Merge Compound Semantics

For compound queries (In on a prefix property + range on the terminator), the entries are returned unmerged — one CountEntry per emitted (in_key, key) pair. The server does NOT collapse them down to a flat histogram keyed only by key. This is a load-bearing design choice:

  1. Correctness under limit. Pushing a limit into grovedb's path query truncates the emitted elements before any merge could run. With cross-fork merging this can undercount the merged sums (e.g. brand IN (acme, contoso) + color > x + limit=1 could return acme/red, count=2 and silently drop contoso/red, count=3 so the merged red count comes out as 2 instead of 5). Without merge, limit and the user's "number of entries returned" mean the same thing.
  2. Proof verification stays straightforward. A malicious server omitting one In branch shows up as missing entries with that in_key rather than as a silent undercount in a merged total. The caller can detect "I asked for 3 In values but only got entries for 2" directly from the response shape.
  3. No information loss. A caller who wanted the merged histogram can compute result.fold(by=key, sum=count) client-side trivially. A caller who wanted per-(in_key, key) counts can't reverse a merged histogram.

The rs-sdk surfaces this via DocumentSplitCounts.0: Vec<VerifiedSplitCount>. Callers wanting the historical flat-map shape can call DocumentSplitCounts::into_flat_map() which sums across in_key forks.

Flat queries (no In on prefix) have in_key = None on every entry; for those callers the API behaves identically to the pre-no-merge shape.

Pagination

Distinct mode accepts pagination knobs:

FieldEffect
order_byCBOR-encoded list of [field, "asc"|"desc"] clauses, same shape as GetDocumentsRequestV0.order_by. First clause's direction controls split-mode entry ordering; ascending (default) walks the range in BTreeMap natural order, descending reverses. Only meaningful in split modes (per-In-value, per-distinct-value-in-range, prove-distinct); total-count and aggregate-prove responses are scalar and have no entry ordering.
limitTruncate after min(requested, max_query_limit) entries; applied last (after order). Unset (None) is normalized to default_query_limit before the cap is applied — the server never walks an unbounded distinct-mode result set, even if the client omits the field. Clients that want a tight working-set should still set this explicitly.

For pagination, clients narrow the underlying range itself rather than passing a cursor — page 2 is just color > <last-key-from-page-1> with the same limit. There's no cursor field on the request because a single-bytes cursor would be ambiguous for compound (In + range + distinct) queries whose natural sort is (in_key, key), and range narrowing has the same expressivity for the simple cases.

These knobs are ignored on summed mode (they have no defined meaning for a single aggregate).

Range Queries on the Prove Path

When prove = true and the query carries a range clause, the handler picks one of two prove sub-paths based on return_distinct_counts_in_range. The aggregate sub-path (default) builds a grovedb AggregateCountOnRange proof — verified via GroveDb::verify_aggregate_count_query, recovering (root_hash, count) without materializing any matching documents. Proof size is O(log n) regardless of how many documents match. The distinct sub-path (return_distinct_counts_in_range = true) builds a regular range proof against the property-name ProvableCountTree — the leaf merk emits per-(in_key, key) KVCount ops, each bound to the merk root via node_hash_with_count, and the SDK extracts them as a Vec<VerifiedSplitCount> (preserving the unmerged compound shape per No-Merge Compound Semantics). Distinct proof size is O(distinct (in_key, key) pairs matched) instead of the aggregate's O(log n) — pick the aggregate path when you want one number, the distinct path when you want a histogram.

In on a prefix property is supported on the distinct sub-path: grovedb's outer Query enumerates Key(in_value) entries at the In-bearing prop's property-name subtree, set_subquery_path carries any post-In Equal pairs + terminator name, and set_subquery is the range item. The aggregate sub-path still rejects In on prefix because AggregateCountOnRange is a single-range merk primitive that can't fork at the merk layer — for compound aggregates, callers use return_distinct_counts_in_range = true and reduce client-side via DocumentSplitCounts::into_flat_map.

A "desc" direction in the first order_by clause is supported on the distinct sub-path. The derived direction flows into grovedb's Query.left_to_right on both the outer In-keys Query and the inner range subquery, so descending iteration walks (in_key_desc, key_desc) tuples. The prover and verifier MUST agree on this direction — the path query bytes include it, and disagreement breaks merk-root recomputation. The SDK derives left_to_right from the first request.document_query.order_by_clauses direction, matching the server's derivation in drive_dispatcher, so the two stay in lockstep by construction. Combined with limit, descending order returns the LAST limit matched entries (the largest keys) rather than the first limit reversed — exactly what callers paginating from the end expect.

For point-lookup count proofs (no range clause), drive emits a CountTree element proof against the covering countable index — proof size is O(k × log n) where k is the number of covered branches, with no cap on the underlying document count. See the Prove path section above for the symmetric-rejection contract.

Range Queries and ProvableCountTree

Range count queries (>, <, between*) over an index with range_countable: true are answered in O(log n) by walking the property-name ProvableCountTree's boundary nodes. The proof path uses grovedb's AggregateCountOnRange, which lets clients verify a range count without ever materializing the underlying documents.

Offset-style queries ("the next 50 items starting after item 7") are a separate primitive that will likely build on the same ProvableCountTree shape. They are not exposed via GetDocumentsCount today — pagination of distinct-mode entries is done by narrowing the range itself (e.g. color > <last-key-from-previous-page>), not by offsetting into the underlying documents.

Why Internal-Node Counts Make Range Counts O(log n)

In a sorted merk tree the keys partition into a left (smaller) and right (larger) subtree at every internal node. To answer a question like "how many items have a key strictly greater than 7?" you walk the boundary between "below 7" and "above 7" from the root down, and at each step you can decide what to do with the other subtree — the one not on the boundary path — based on a single read:

  • If a subtree lies entirely above the cutoff, add its full count and don't descend into it.
  • If it lies entirely below, ignore it (contributes 0) and don't descend.
  • If it straddles the cutoff, recurse into it (it is then the next step on the boundary path).

On a ProvableCountTree every internal node carries the count of its left and right subtrees, so the "add the full count" step is a single O(1) read of the node we're already touching. The whole walk visits one node per tree level — O(log n) — and every visited node is on the boundary path. The total ends up as a sum of pre-committed sub-counts plus zero or one straddle leaf at the bottom.

Concretely, picture a ProvableCountTree of 8 items with sorted integer keys 1, 3, 5, 7, 9, 11, 13, 15 — three full levels of internal nodes plus a leaf row:

flowchart TB
  R["root c=8"]:::countroot
  R --> L1["c=4"]:::countnode
  R --> R1["c=4"]:::countnode
  L1 --> LL["c=2"]:::countnode
  L1 --> LR["c=2"]:::countnode
  R1 --> RL["c=2"]:::countnode
  R1 --> RR["c=2"]:::countnode
  LL --> x1["key=1, c=1"]:::leaf
  LL --> x3["key=3, c=1"]:::leaf
  LR --> x5["key=5, c=1"]:::leaf
  LR --> x7["key=7, c=1"]:::leaf
  RL --> x9["key=9, c=1"]:::leaf
  RL --> x11["key=11, c=1"]:::leaf
  RR --> x13["key=13, c=1"]:::leaf
  RR --> x15["key=15, c=1"]:::leaf

  classDef countroot fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef countnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
  classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;

For "give me the count of items with key > 6":

  • root (c=8): 6 falls inside the left subtree (which holds 1–7). Read both children's sub-counts. Right subtree's keys are all > 6 → take its full c=4 and don't descend. Recurse into left.
  • left (c=4): 6 falls inside its right subtree (which holds 5,7). Read both children. Left-left's keys (1,3) are both ≤ 6 → contribute 0 and don't descend. Recurse into left-right.
  • left-right (c=2): 6 splits this leaf-pair. Read both leaves. Key 5 ≤ 6 → contribute 0. Key 7 > 6 → contribute 1.
  • Total = 4 (right of root) + 0 (left-left) + 0 (key=5) + 1 (key=7) = 5.

We visited 4 internal nodes on the boundary path (root → left → left-right → key=7) and read sub-counts off 3 siblings (right, left-left, key=5) without descending into them. Six of the eight items were never enumerated: their counts were summed straight out of the committed sub-count fields. The walk is O(log n) in tree depth regardless of how many items live under each skipped subtree.

Why This Is Provable

A merk proof of the same boundary walk includes:

  1. The boundary path from root to the leaf adjacent to the cutoff.
  2. The siblings of every node on the boundary path (so the verifier can recompute hashes up to the merk root).

Each sibling node, on a ProvableCountTree, ships its committed sub-count alongside its hash. The verifier walks the same logic the server did — "this sibling lies entirely above 7, add its c=… value" — and ends up with the same total without enumerating the sibling subtrees. Verification is also O(log n).

The same primitive answers any range query of the form [A, B]: walk to the cutoff at A, then to the cutoff at B, and combine sub-counts along the way. [A, ∞) and (-∞, B] are special cases.

Authoring a Contract That Uses Count Trees

There are two opt-in surfaces in the document meta-schema. They're independent and can be used together:

  1. Top-level flags on the document type control the primary-key tree variant — the tree that stores documents keyed by document ID. This is what GetDocumentsCount (with no equality predicates) reads.
  2. A per-index countable: true flag controls whether that specific index's tree carries counts. This is what enables the no-prove fast path for queries that filter by the index's leading equality columns.

Primary-Key Tree Flags

Set at the same level as type / properties / indices on a document type:

{
  "widget": {
    "type": "object",
    "documentsCountable": true,
    "properties": {
      "name":  { "type": "string",  "position": 0, "maxLength": 64 },
      "color": { "type": "string",  "position": 1, "maxLength": 16 }
    },
    "additionalProperties": false
  }
}

That contract gets a CountTree for the widget primary-key tree. GetDocumentsCount for widget with no where filter is now an O(1) lookup of the tree element's count value.

To opt into a ProvableCountTree for the primary-key tree instead — useful if you want range queries on the primary key in the future, or if you intend to use this document type behind range proof primitives — set rangeCountable: true at the document-type level. It implies documentsCountable, so you don't need both:

{
  "widget": {
    "type": "object",
    "rangeCountable": true,
    "properties": {
      "name":  { "type": "string",  "position": 0, "maxLength": 64 },
      "color": { "type": "string",  "position": 1, "maxLength": 16 }
    },
    "additionalProperties": false
  }
}

These two flags are immutable across a contract update. You pick the tree variant at contract creation; you can't switch to a different one later without creating a new document type. (See Storage-Layout Invariants above.)

Per-Index Countable Flag

Set on a single entry in the document type's indices array:

{
  "widget": {
    "type": "object",
    "documentsCountable": true,
    "properties": {
      "name":  { "type": "string",  "position": 0, "maxLength": 64 },
      "color": { "type": "string",  "position": 1, "maxLength": 16 }
    },
    "indices": [
      {
        "name": "byColor",
        "properties": [{ "color": "asc" }],
        "countable": true
      }
    ],
    "additionalProperties": false
  }
}

With byColor.countable: true the byColor index's tree carries counts, so GetDocumentsCount with where: [["color", "==", "red"]] reaches the count via that index in O(1). Without the flag, find_countable_index_for_where_clauses skips the index and the query rejects with WhereClauseOnNonIndexedProperty — there's no slow fallback, only fast counts on properly-indexed properties.

The countable field accepts three forms:

JSON valueTree variantCapabilities
false (or omitted, or "notCountable")NormalTreeNo count fast path
true (or "countable")CountTreeO(1) totals at the root
"countableAllowingOffset"ProvableCountTreeO(1) totals plus per-node counts that will enable future O(log n) range / offset queries on this index

The boolean true / false form is kept for back-compat with contracts written before the enum form was introduced; new contracts should prefer the explicit string variants for clarity, especially "countableAllowingOffset" when range/offset queries are wanted.

A few notes about the index-level flag:

  • Setting any countable variant increases storage cost — every insert and delete updates the index tree's count alongside the document. "countableAllowingOffset" costs more than plain "countable" (every internal node carries count metadata, not just the root). Don't sprinkle it on every index; opt in for the ones you'll actually count by, and use the cheaper variant unless you specifically need the offset capability.
  • The flag is on the whole index, not per-property. The index handles count(*) queries whose equality where clauses cover the index's properties exactly — every index property has a matching == (or in) clause, and every clause's field appears in the index. A ["color", "size"] countable index gives you O(1) counts for WHERE color = X AND size = Y — but WHERE color = X alone is rejected with WhereClauseOnNonIndexedProperty because that index doesn't claim to count by color alone. If you want both single-column-by-color counts AND compound color+size counts, define both ["color"] and ["color", "size"] countable indexes (or just ["color"] if size-filtered counts aren't a hot path). The picker is strict by design: each countable index represents a deliberate decision about which count queries the contract supports.
  • Index-level countable is independent of the primary-key flags. You can have documentsCountable: true on the document type AND countable: true on a specific index — the first gives you fast totals, the second gives you fast filtered counts that match that index.
  • countable on a unique index is mostly a no-op, but not always. A unique index stores its terminal as a bare reference at key [0] rather than wrapping it in a count tree, so for documents whose indexed fields are all non-null the flag has no storage effect — insertion bypasses the count-tree code entirely. It does still do meaningful work for null-bearing entries: when a document has any null value among the indexed properties, insertion takes the same count-tree branch a non-unique index uses (because uniqueness can't be enforced on null), and the count tree at that path aggregates them. So countable on a unique index is worth setting when at least one of the indexed properties is optional in the schema and you expect null values; otherwise it's an inert flag. Counts on all-non-null exact matches still return correctly (1 if present, 0 if not) because the on-disk reference reads as count 1 via grovedb's default-aggregate semantics.

Choosing What to Set

You wantSet
Fast count(*) for the whole document typedocumentsCountable: true on the document type
O(1) filtered count: count(*) WHERE col = Xcountable: true on an index whose properties are exactly ["col"]. A composite index whose leading column is col (e.g. ["col", "other"]) does NOT answer this query — partial coverage rejects with WhereClauseOnNonIndexedProperty. Define a separate ["col"] countable index if you want this count.
Per-In-value sub-counts: one CountEntry per value in an In clausecountable: true on an index whose properties exactly match the query's == clauses plus the In field. The In field may sit at any position in the index — both the no-proof and prove count paths use set_subquery_path to descend through any trailing Equals after the In, which is strictly more permissive than the regular document query path's last-or-before-last rule. E.g. WHERE color IN [...] needs ["color"]; WHERE brand = X AND color IN [...] needs ["brand", "color"]; WHERE brand IN [...] AND model = X AND year = 2024 needs ["brand", "model", "year"] with In on brand (position 0 of 3).
O(log n) range count: count(*) WHERE col BETWEEN A AND BrangeCountable: true on an index whose last property is col and whose other properties cover any equality predicates as a prefix. Implies countable: true.
Per-distinct-value range histogram: one CountEntry per distinct value in a rangeSame rangeCountable: true index as above, plus return_distinct_counts_in_range = true on the request. Available on both prove and no-prove paths; the prove path returns a regular range proof against the property-name ProvableCountTree and the SDK extracts per-key counts from the proof's KVCount ops via drive_proof_verifier::verify_distinct_count_proof.
Range count proof (prove = true + range clause)Same rangeCountable: true index. The handler uses grovedb's AggregateCountOnRange proof primitive — proof is O(log n), no cap on matched docs.
Future offset-style range queries (not yet released — see above)rangeCountable: true on the document type
Nothing count-aware (default)Don't set any of these flags. Primary-key tree stays a NormalTree.

A migration check from dapi-grpc server logic: every count query requires either documentsCountable: true (for unfiltered totals) or a countable: true / rangeCountable: true index whose properties exactly match the query's where-clause fields. No covering index → the call returns a clear InvalidArgument describing what the picker was looking for ("requires a range_countable: true (or summableOffCountIndex) index whose last property matches the range field" for range queries, "requires a countable index whose properties exactly match the where clause fields" for Equal/In queries). Pick your indexes deliberately at contract creation time — per-index countable: true / rangeCountable: true flags can't be added later (contract indexes are immutable post-creation).

Counts Are Public

Anyone can run a count query and verify its proof, so a countable index publishes everything its counts reveal. Encrypting a document's fields does not hide its existence or its indexed values. A countable index keyed first by a recipient and then by the document owner answers "who sent documents to this recipient, and how many" for every recipient. On the DashPay contactRequest type, a countable [toUserId, $ownerId] index would publish every user's inbound contacts. If a UI only needs a badge, a countable index on the recipient alone (["toUserId"]) reveals a total and no per-sender edges.

SDK Access at Three Layers

rs-sdk (native Rust)

Both shapes land on the standard Fetch trait against a single DocumentCountQuery:

#![allow(unused)]
fn main() {
use dash_sdk::platform::documents::document_count_query::DocumentCountQuery;
use dash_sdk::platform::Fetch;
use drive::query::{WhereClause, WhereOperator};
use drive_proof_verifier::{DocumentCount, DocumentSplitCounts};

// Total count: no In clause.
let DocumentCount(count) = DocumentCount::fetch(
    &sdk,
    DocumentCountQuery::new(contract.clone(), "widget")?,
)
.await?
.expect("DocumentCount::fetch always returns a value on success");

// Split count: signal split by including an `In` clause whose field
// is the split property. The In's values enumerate the keys to count.
let split_query = DocumentCountQuery::new(contract, "widget")?
    .with_where(WhereClause {
        field: "color".to_string(),
        operator: WhereOperator::In,
        value: platform_value::Value::Array(vec![
            "red".into(),
            "blue".into(),
            "green".into(),
        ]),
    });
let splits = DocumentSplitCounts::fetch(&sdk, split_query)
    .await?
    .expect("DocumentSplitCounts::fetch always returns a value on success");
// `splits` is `DocumentSplitCounts(Vec<SplitCountEntry>)` — for the
// flat-histogram view, collapse via `splits.into_flat_map()`.
}

DocumentCountQuery wraps an internal DocumentQuery (so it reuses where-clause / order-by / contract-id machinery) and exposes with_where(WhereClause) + with_order_by(OrderClause) builders. The SDK picks the request mode (total / per-In-value / total-range / per-distinct-range) from query shape — Equal/In/range operators in the where clauses — plus explicit request flags. return_distinct_counts_in_range = true (set via .with_distinct_counts_in_range(true)) selects per-distinct-range over the default total-range when a range clause is present; without it a range query returns a single sum.

wasm-sdk (browser)

Two methods on the WasmSdk JS class — one entry per [plain | withProofInfo] variant covers every count mode, because the underlying DocumentSplitCounts::fetch dispatches on the query shape:

sdk.getDocumentsCount(
  query: DocumentsQuery,
): Promise<Map<string, bigint>>;

sdk.getDocumentsCountWithProofInfo(
  query: DocumentsQuery,
): Promise<ProofMetadataResponseTyped<Map<string, bigint>>>;

Result shapes:

  • No where, or Equal-only where — single map entry with the empty-string key carrying the total count.
  • where includes an In clause — one entry per (deduped) In value, keyed by the hex-encoded canonical bytes of that value.
  • where includes a range clause + returnDistinctCountsInRange: true — one entry per distinct property value in the range. For compound In + range + distinct queries, entries are summed by terminator key into a flat map (callers needing the unmerged per-(in_key, key) view should use a richer binding).

Map keys are always hex-encoded bytes matching the canonical serialize_value_for_key encoding of each property value, so callers that need a typed key ("red", 42, etc.) need to hex-decode and interpret per the contract's index-property type. The hex-encoded shape matches the no-prove server response, so merging or comparing count maps from prove and no-prove paths needs no transformation.

rs-sdk-ffi (iOS / native bindings)

#![allow(unused)]
fn main() {
dash_sdk_document_count(
    sdk,
    data_contract,
    document_type,
    where_json,                          // null or JSON [{field, operator, value}]
    order_by_json,                       // null or JSON [{field, direction}]
    return_distinct_counts_in_range,     // bool
    limit,                               // i64; -1 = server default, >= 0 = explicit cap
) -> JSON {"counts": {"<hex-key>": <u64>, ...}}
}

Single FFI entry covers every count mode — the result is always {"counts": {...}} with hex-encoded keys. For total counts (no where/In, distinct flag off), the map carries a single entry with the empty-string key. where_json is the same JSON shape dash_sdk_document_search already accepts ([{field, operator, value}]), so iOS callers can reuse their where-clause encoding. order_by_json is optional and controls split-mode entry ordering only (per-In-value and per-distinct-value-in-range results); pass null for total counts and aggregate range counts where ordering has no defined meaning. The endpoint returns its result as a JSON-encoded C string allocated on the heap — caller frees it via the standard SDK string-free routine.

Count Index Examples

This chapter walks through a representative contract and shows what a count-query proof actually proves — both the path query the prover signs and the verified element the verifier extracts. Every example uses the same widget contract (the same one the count-query bench at packages/rs-drive/benches/document_count_worst_case.rs populates) so the proof bytes, verified elements, and diagrams can all be cross-referenced against the same data.

The chapter assumes you've read Document Count Trees — that chapter explains the three tree variants (NormalTree / CountTree / ProvableCountTree), what Element::NonCounted does, and how the schema's documentsCountable / rangeCountable flags select between them. Here we take that machinery as given and trace what each query sees.

The Widget Contract

The widget document type carries three properties (brand, color, serial), opts into total counts at the doctype level via documentsCountable: true, and declares three indexes covering the count-query surface:

{
  "type": "object",
  "documentsCountable": true,
  "properties": {
    "brand":  { "type": "string", "position": 0, "maxLength": 32 },
    "color":  { "type": "string", "position": 1, "maxLength": 32 },
    "serial": { "type": "integer", "position": 2 }
  },
  "required": ["brand", "color", "serial"],
  "indices": [
    {
      "name": "byBrand",
      "properties": [{ "brand": "asc" }],
      "countable": "countable"
    },
    {
      "name": "byColor",
      "properties": [{ "color": "asc" }],
      "countable": "countable",
      "rangeCountable": true
    },
    {
      "name": "byBrandColor",
      "properties": [{ "brand": "asc" }, { "color": "asc" }],
      "countable": "countable",
      "rangeCountable": true
    }
  ],
  "additionalProperties": false
}

Three things to notice:

  1. documentsCountable: true at the document-type level upgrades the doctype's primary-key subtree (at widget/[0]) from NormalTree to CountTree. The unfiltered total count is one read against this element's count_value.
  2. byBrand is countable: "countable" only. It doesn't opt into rangeCountable, so brand > X range counts aren't supported. But every countable terminator's value tree is stored as a CountTree regardless of rangeCountable (see add_indices_for_index_level_for_contract_operations/v0/mod.rs), so point-lookup count proofs (e.g. brand == "X" or brand IN [...]) get the same compact value-tree-direct shape on byBrand that they do on rangeCountable indexes. rangeCountable is strictly an opt-in for AggregateCountOnRange support — orthogonal to proof-size shape.
  3. byColor and byBrandColor are rangeCountable: true. Their property-name subtrees (e.g. widget/color) are stored as ProvableCountTree rather than NormalTree, which is what AggregateCountOnRange walks for color > floor style queries.

The bench populates 100 000 documents under a deterministic schedule — row → (brand_(row % 100), color_(row / 100), serial=row). That gives exactly 1 000 docs per brand, exactly 100 docs per color, and exactly 1 doc per (brand, color) pair. Those numbers show up in every verified count below.

GroveDB Layout

The contract above produces this storage shape. Tree elements (the wrapping Element GroveDB stores under each key) are drawn as subgraphs; children inside each tree are merk-tree nodes. The doctype root and the per-property name subtrees are separate Element trees nested under the contract-documents prefix, just like every other index in Drive.

Diagram conventions: green nodes carry a count_value committed to the merk root; gray are regular subtrees; dashed boxes highlight Element::NonCounted wrappers (children that store data but contribute 0 to their parent CountTree's count).

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree

  WD --> PK["[0]: CountTree count=100000<br/>(documentsCountable primary key)"]:::countnode
  WD --> BR["brand: NormalTree<br/>(byBrand property-name)"]:::node
  WD --> CO["color: ProvableCountTree<br/>(byColor property-name)"]:::pctnode

  BR --> B000["brand_000: CountTree count=1000"]:::countnode
  BR --> B050["brand_050: CountTree count=1000"]:::countnode
  BR --> BMore["... brand_001 ... brand_099<br/>(all CountTree count=1000)"]:::countnode

  B050 --> B050_0["[0]: CountTree count=1000<br/>(byBrand refs)"]:::countnode
  B050 --> B050_C["color: NonCounted(ProvableCountTree)<br/>(byBrandColor continuation, contributes 0)"]:::noncounted

  B050_C --> B050_C_500["color_00000500: CountTree count=1<br/>(byBrandColor terminator)"]:::countnode
  B050_C_500 --> B050_C_500_0["[0]: CountTree count=1<br/>(byBrandColor ref)"]:::countnode

  CO --> C500["color_00000500: CountTree count=100<br/>(byColor terminator)"]:::countnode
  CO --> CMore["... color_00000000 ... color_00000999"]:::countnode
  C500 --> C500_0["[0]: CountTree count=100<br/>(byColor refs)"]:::countnode

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef countnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
  classDef pctnode fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef noncounted fill:#21262d,color:#c9d1d9,stroke:#fb8500,stroke-width:2px,stroke-dasharray: 6 4;

Three layout facts to internalize before reading the queries:

  • brand_050 is a CountTree with count_value = 1000. That's true because byBrand is countable; the rule applies uniformly to every countability tier (see add_indices_for_index_level_for_contract_operations/v0/mod.rs). The color continuation that branches off this value tree is NonCounted-wrapped so the parent's count equals exactly the 1 000 refs in [0].
  • widget/color is a ProvableCountTree, not a regular NormalTree. The yellow class above marks that — each internal merk node carries its subtree's count, which is what makes AggregateCountOnRange a single-pass primitive.
  • color_00000500 is a CountTree with count_value = 100 under either parent. The same element layout would result from a query against byColor or against byBrandColor's second level; the path that gets there differs, but the destination is structurally the same.

How To Read The Proofs

Every example below has four sections:

  1. Path query — the spec the prover hands GroveDB. path is the list of subtree segments to descend through (the proof carries merk-path bytes for each of these); query items is what to select once at the bottom; subquery items (when present) descends one more layer.
  2. Verified element — what GroveDB::verify_query (or verify_aggregate_count_query for the range primitive) returns after walking the proof bytes. The count_value_or_default field on a CountTree element is what the count surface ultimately surfaces to the caller.
  3. Proof display — the proof bytes, decoded via bincode into the structured GroveDBProof AST and rendered through its Display impl. This is the same view dash-evo-tool's Proof Log screen shows when its display mode is set to "JSON" — each layer is a separate LayerProof carrying its merk-tree operations (Push / Parent / Child over Hash / KVValueHash / KVHash) plus a lower_layers map naming the children to descend into. Wrapped in a collapsible block per example because the merk path through 4-5 grovedb layers makes for long output.
  4. Diagram — the path the proof walks through the layout. Blue arrows trace the descent; the cyan node is the verified element; faded gray nodes show context.

All proof-size numbers come from running the bench against a 100 000-row fixture; see document_count_worst_case.rs's report_proof_sizes / display_proofs / report_group_by_matrix helpers. The proof bytes are reproducible — run the bench, grep [proof] from stderr, and you'll get the same hashes shown here.

Queries in this Chapter

#QueryFilterComplexityAvg timeProof size
1Unfiltered Total Count(none — total at doctype level)O(1)22.5 µs585 B
2Equal on a Single Property (byBrand)brand == "brand_050"O(log B)35.7 µs1 041 B
3Equal on a RangeCountable Property (byColor)color == "color_00000500"O(log C)54.0 µs1 327 B
4Compound Equal-only (byBrandColor)brand == "brand_050" AND color == "color_00000500"O(log B + log C')71.4 µs1 911 B
5In on byBrandbrand IN ["brand_000", "brand_001"]O(k · log B)40.0 µs1 102 B
6In on byColor (RangeCountable)color IN ["color_00000000", "color_00000001"]O(k · log C)61.9 µs1 381 B
7Range Query (AggregateCountOnRange)color > "color_00000500"O(log C)69.2 µs2 072 B
8Compound == + Range (byBrandColor)brand == "brand_050" AND color > "color_00000500"O(log B + log C')84.9 µs2 656 B

Complexity variables. B = distinct brands in the byBrand merk-tree (≈ 100 in the fixture); C = distinct colors in the byColor merk-tree (≈ 1 000); C' = distinct colors per brand in byBrandColor's continuation (≈ 1 000 — every brand carries the full color namespace in this fixture); k = number of values in the IN clause (2 here). Notably absent: the total document count N (100 000 here). Count proofs read pre-committed count_values from CountTree merk roots — they never enumerate the underlying documents, so proof generation cost is polylog(distinct index values), independent of N. The grove-descent overhead (5–8 layers) is treated as a constant. The O() column captures shape only, not constants — for instance Q3's O(log C) is ~50% slower than Q2's O(log B) because in this fixture C ≈ 10 × B, and byColor's ProvableCountTree carries extra running-count metadata per merk node on top of that (37 merk ops in L6 vs 25 for byBrand — see Query 2 and Query 3's proof displays).

Avg time is the criterion-reported median of cargo bench --bench document_count_worst_case -- 'document_count_worst_case/query_' on a 100 000-row warmed fixture (no group_by — single-query latency on the prover side, including merk-proof construction and serialization). Each row reflects 10 samples × 67k–220k iterations per sample with 2 s warm-up and 5 s measurement; the median sits within ±2 % of the mean across reruns. For GROUP BY variants of these queries, see Count Index Group By Examples.

Each query has the same four sections (Path query, Verified element, Proof display, Diagram) plus a per-layer merk-tree diagram starting at Layer 5 (Layers 1–4 are byte-for-byte identical across every query — they're the root → @ → contract_id → 0x01 descent shown in full only on Query 1). The bottom of the chapter has an at-a-glance comparison summarizing the structural differences.

Query 1 — Unfiltered Total Count

select  = COUNT
where   = (empty)
prove   = true

Path query (primary-key CountTree fast path; no index walk needed):

path:         ["@", contract_id, 0x01, "widget"]
query items:  [Key(0x00)]

Verified element:

path:        ["@", contract_id, 0x01, "widget"]
key:         0x00
element:     CountTree { count_value_or_default: 100000 }

Proof size: 585 B.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (4 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(KVValueHashFeatureTypeWithChildHash(0x00, CountTree(0000000000010000fffffffffffeffff00000000000000000000000000000000, 100000), HASH[85843d8e6353dd6caf52f659c454b4a1352f510daa965df594b27319abf1d8a1], BasicMerkNode, HASH[0e6a5047f0600cafc385ed52b516c1fbbaf4994aa50dfcbd1e824b4ad9f55fa1]))
                              1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
                              2: Parent
                              3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
                              4: Child)
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. The descent goes: top-level GroveDB root → @ (DataContractDocuments root tree) → contract id → 0x01 (documents storage prefix) → widget doctype → finally the Key(0x00) payload at the bottom, where CountTree(…, 100000) is the verified element with its count_value of 100 000 visible inside.

The descent stops at the doctype's primary-key tree — the green node at the top of the layout. Because documentsCountable: true upgraded that tree to a CountTree, the count is one O(1) read.

Diagram: per-layer merk-tree structure

Each LayerProof above is its own GroveDB sub-tree whose contents form a merk binary tree. The merk-proof operations (Push / Parent / Child over KVValueHash / KVHash / Hash nodes) describe exactly which nodes of each layer's binary tree the proof reveals — the queried key gets its full kv-hash exposed; opaque siblings only commit their subtree-hash so the verifier can re-hash up to the merk root.

Cyan = the verified target. Blue = a kv-hash that's also a queried-key on the descent path (its value = Tree(...) is the merk-root pointer for the next layer). Gray = opaque sibling subtrees committed by hash only.

flowchart TB
  subgraph L1["Layer 1 — root GroveDB merk-tree"]
    direction TB
    L1_root["<b>@</b><br/>kv_hash=HASH[4a5a...]<br/>value: Tree(0x4ed2…)"]:::queried
    L1_left["HASH[bd29...]<br/>(left subtree, opaque)"]:::sibling
    L1_right["HASH[19c9...]<br/>(right subtree, opaque)"]:::sibling
    L1_root --> L1_left
    L1_root --> L1_right
  end

  subgraph L2["Layer 2 — @ subtree merk-tree (single key)"]
    direction TB
    L2_q["<b>contract_id 0x4ed2…</b><br/>kv_hash=HASH[5b90...]<br/>value: Tree(0x01)"]:::queried
  end

  subgraph L3["Layer 3 — contract_id subtree merk-tree"]
    direction TB
    L3_q["<b>0x01</b><br/>kv_hash=HASH[5d9a...]<br/>value: Tree(widget)"]:::queried
    L3_left["HASH[49e7...]<br/>(left subtree, opaque)"]:::sibling
    L3_q --> L3_left
  end

  subgraph L4["Layer 4 — 0x01 documents-prefix subtree (single key)"]
    direction TB
    L4_q["<b>widget</b><br/>kv_hash=HASH[6c50...]<br/>value: Tree(0x00/brand/color)"]:::queried
  end

  subgraph L5["Layer 5 — widget doctype merk-tree (TARGET layer)"]
    direction TB
    L5_root["KVHash[a29e...]<br/>(opaque internal kv: brand or color)"]:::sibling
    L5_target["<b>0x00</b><br/>kv_hash=HASH[8584...]<br/>value: <b>CountTree count=100000</b>"]:::target
    L5_right["HASH[6c36...]<br/>(right subtree, opaque)"]:::sibling
    L5_root --> L5_target
    L5_root --> L5_right
  end

  L1_root -. "value=Tree(merk_root[5b90…])" .-> L2_q
  L2_q -. "value=Tree(merk_root[5d9a…])" .-> L3_q
  L3_q -. "value=Tree(merk_root[6c50…])" .-> L4_q
  L4_q -. "value=Tree(merk_root[a29e…])" .-> L5_root

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

A few things this diagram makes explicit that the prose can't:

  • Each layer is its own merk binary tree, not a single graph. The 5 LayerProof blocks in the structured proof above each describe one of these binary trees. The hashes named in each block's Push(Hash(HASH[…])) ops are this diagram's opaque siblings; the Push(KVValueHash(K, …)) ops are this diagram's blue / cyan nodes.
  • The "descent" between layers is via a value: Tree(…). When a queried key's value is Tree(merk_root_hash), that hash IS the merk root of the next layer's binary tree. So Layer 1's @ doesn't descend to Layer 2's contract_id directly — it descends to Layer 2's merk root, which in this case happens to be the only node in Layer 2.
  • Single-key layers have a 1-node merk tree. Layers 2 and 4 contain exactly one entry (@ contains exactly one contract id; 0x01 contains exactly one doctype here), so their merk trees have no siblings to commit.
  • The merk root of a layer can be an opaque sibling, not the queried key. Layer 5's merk root is KVHash[a29e...] — a key (brand or color, we can't tell from the proof) whose kv_hash is committed but whose value isn't revealed. The queried 0x00 is reached as a child of that opaque root. This is why the merk-tree structure matters: the prover sometimes has to commit one merk-tree-depth's worth of hashes to prove the queried key's position, even if the verifier only cares about the target's value.

Query 2 — Equal on a Single Property (byBrand)

select  = COUNT
where   = brand == "brand_050"
prove   = true

Path query:

path:         ["@", contract_id, 0x01, "widget", "brand"]
query items:  [Key("brand_050")]

Verified element:

path:        ["@", contract_id, 0x01, "widget", "brand"]
key:         "brand_050"
element:     CountTree { count_value_or_default: 1000 }

Proof size: 1 041 B.

Proof display:

Expand to see the structured proof (verbatim, 5 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
                              1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
                              2: Parent
                              3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
                              4: Child)
                            lower_layers: {
                              brand => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[fb5eb23b3135d9c226e61f004ffb43abae104238d8a1ea7bc60e8ec6ba271596]))
                                    1: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
                                    2: Parent
                                    3: Push(Hash(HASH[19ec5730af134e9ac980bbea92c2978212c8efe750a467ab54f073626e0ca2f5]))
                                    4: Push(KVHash(HASH[87bc6e7e1e465b8dcdaf95db9957a455d6bd7c75976db122f33e592fe75f1e4a]))
                                    5: Parent
                                    6: Push(Hash(HASH[a0a354f2bb59b8169253aebabb52afcc3c59c4c60da203c8887abb679d747168]))
                                    7: Push(KVHash(HASH[fc6b1d0237f8ff89b555e9a14480ae1c5b80d529a0f9fb5e681ea7ecd157d3da]))
                                    8: Parent
                                    9: Push(KVValueHashFeatureTypeWithChildHash(brand_050, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[53dbd6216cccdddf16f3eb0f849aed0c0cea987a718f5b43493abf0a14e83eb9], BasicMerkNode, HASH[4947457e230f87ce0f75a7f1502f64f24ee4d3e27eb5d2210680822a3b17afa4]))
                                    10: Child
                                    11: Push(KVHash(HASH[027ac8b1bc9788118b27c13d0b3c3bd3661ef6a89a775a6b6bf78aa7e6f8ed3d]))
                                    12: Parent
                                    13: Push(Hash(HASH[7a5dc3002e6cb6c92e54d554e5af85e9c2ba64ee9c5f80e6489075cc5f3f0d55]))
                                    14: Child
                                    15: Push(KVHash(HASH[3363630479f1abe6e003b1e1d50b5118e55ad2efb7a3f4b3b6df902bea72ac9a]))
                                    16: Parent
                                    17: Push(Hash(HASH[3857faef5ddb06e201f1e65cf42f15d6c9b0dc67e7f73eb182b520854e9bb648]))
                                    18: Child
                                    19: Child
                                    20: Child
                                    21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
                                    22: Parent
                                    23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
                                    24: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

The bottom layer is the byBrand property-name tree; it has 100 distinct brand_NNN keys, so the merk path proves brand_050's position with 25 ops total (0–24). The verified payload is the inline KVValueHashFeatureTypeWithChildHash(brand_050, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), …) on op 9 — the 636f6c6f72 value slot is the ASCII bytes for "color" (the byBrandColor continuation pointer; NonCounted-wrapped at the storage layer so it contributes 0 to the parent count), and the 1000 is the doc count. The remaining 24 ops are the merk-binary boundary walk: each Push(Hash(…)) is an opaque subtree the proof commits but doesn't descend into, each Push(KVHash(…)) is an opaque internal sibling kv whose hash is committed, and each Parent / Child re-attaches them so the verifier can recompute the byBrand merk root.

brand_050 is itself a CountTree — every countable terminator's value tree carries the doc count directly, with sibling continuations wrapped NonCounted so they don't pollute the parent. The proof shape is the same as the rangeCountable case below, even though byBrand doesn't opt into rangeCountable: true. rangeCountable is the orthogonal opt-in for AggregateCountOnRange (Query 7), not for proof-size shape.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree"]:::path
  BR ==> B050["brand_050: CountTree count=1000"]:::target
  BR -.-> B000["brand_000"]:::faded
  BR -.-> BMore["..."]:::faded
  WD -.-> PK["[0]"]:::faded
  WD -.-> CO["color"]:::faded
  B050 -.-> B050_0["[0]: 1000 refs"]:::faded
  B050 -.-> B050_C["color (NonCounted)"]:::faded

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Layers 1–4 are byte-for-byte identical to Query 1's diagram (root → @ → contract_id → 0x01). The descent diverges at Layer 5, where this query takes the brand branch (rather than 0x00) and descends one extra grove layer to land on the verified target.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `brand`)"]
    direction TB
    L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
    L5_left["HASH[9862...]<br/>(left subtree, opaque)"]:::sibling
    L5_right["HASH[6c36...]<br/>(right subtree, opaque)"]:::sibling
    L5_q --> L5_left
    L5_q --> L5_right
  end

  subgraph L6["Layer 6 — byBrand merk-tree (TARGET layer)"]
    direction TB
    L6_target["<b>brand_050</b><br/>kv_hash=HASH[53db...]<br/>value: <b>CountTree count=1000</b><br/>child_hash=HASH[4947...]"]:::target
    L6_boundary["Boundary commitments (24 merk ops):<br/>6 KVHash opaque sibling brands<br/>+ 6 Hash subtree commitments<br/>(prove brand_050's position in byBrand's<br/>binary merk tree of ~100 brand entries)"]:::sibling
    L6_target --> L6_boundary
  end

  L5_q -. "value=Tree(merk_root[byBrand])" .-> L6_target

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The boundary commitments at L6 are what scale linearly with the byBrand tree's depth — they bind brand_050 to its claimed position so the verifier can recompute byBrand's merk root. The verified target itself is just one KVValueHashFeatureTypeWithChildHash op whose count_value_or_default = 1000 is the answer.

Query 3 — Equal on a RangeCountable Property (byColor)

select  = COUNT
where   = color == "color_00000500"
prove   = true

Path query:

path:         ["@", contract_id, 0x01, "widget", "color"]
query items:  [Key("color_00000500")]

Verified element:

path:        ["@", contract_id, 0x01, "widget", "color"]
key:         "color_00000500"
element:     CountTree { count_value_or_default: 100 }

Proof size: 1 327 B.

Proof display:

Expand to see the structured proof (verbatim, 5 layers; note `KVHashCount` ops in the byColor `ProvableCountTree` layer) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
                              1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
                              2: Parent
                              3: Push(KVValueHash(color, ProvableCountTree(636f6c6f725f3030303030353131, 100000), HASH[79569d595db75bbf2e9dca93a15c90b7eecf7b299632668ec410e2076d27f71c]))
                              4: Child)
                            lower_layers: {
                              color => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[864c8a53cdfc17560ea304fe40ae87570699a6920eae3dcb6075f71ca2d79b02]))
                                    1: Push(KVHashCount(HASH[3684347a67ceedad2ff4a7fce6ae303086543c1f146f5865dfdc23612308c05b], 51100))
                                    2: Parent
                                    3: Push(Hash(HASH[56422e033fcffda5514eaef88096da995646207f3f5e349a6840003b4297098e]))
                                    4: Push(KVHashCount(HASH[aa27604017cfc457ccd56aabeb4686a988b0b073d1c1c03a4fdf78164c31c8ea], 25500))
                                    5: Parent
                                    6: Push(Hash(HASH[09bcdaa37a5ae46f9059a7c026bf9cdf1c2d1ddecfcfe72fafe73f30abf2bccc]))
                                    7: Push(KVHashCount(HASH[525df42449bd5e881d55f94c11be2b1c95cd112123864fc249e6c170ea026f5a], 12700))
                                    8: Parent
                                    9: Push(Hash(HASH[ffe58ba46b2d1f91b04e9c78185b474828f8ad165757847d9178020e55ad6c26]))
                                    10: Push(KVHashCount(HASH[abbcbcef405f19e0a096a902993b3c76c77c59abdb8a3dcc95369e8c17b401c7], 6300))
                                    11: Parent
                                    12: Push(Hash(HASH[472879d66cf8e01e77bf4828d6a6f530a016cf7a99d712deb00c8fa5920b8495]))
                                    13: Push(KVHashCount(HASH[3ac3896404268efc1bbfc9a2a8925adcc9eff7248fc7ca3aaec6f62587cdaffd], 3100))
                                    14: Parent
                                    15: Push(Hash(HASH[1c40306956f164e416e74a69ce0fff8c7ca152904ad47f44c6142c7822d3d2fb]))
                                    16: Push(KVHashCount(HASH[494935a3d102495beb504953539d204ecd5b5ca8f5a03aa4a3cdbf16a3926335], 700))
                                    17: Parent
                                    18: Push(KVValueHashFeatureTypeWithChildHash(color_00000500, CountTree(00, 100, flags: [0, 0, 0]), HASH[47b0ade593a2e4e99e7d7363f5d1f692882007397f025226f19d097ca2f407fa], ProvableCountedMerkNode(100), HASH[4f7f13f56e087e7b19751c067671b75cda83156231cd3186f7c4172dccc8e97b]))
                                    19: Push(KVHashCount(HASH[4866192fb6beda0888f828d7bbf008fa725a1141cf19ae3b1e9d245c6cb12c7c], 300))
                                    20: Parent
                                    21: Push(Hash(HASH[f56dd41a87f9b487ee9893c310a8bdd2fe70eb573e2e22e048cef7e3dec5fc1d]))
                                    22: Child
                                    23: Child
                                    24: Push(KVHashCount(HASH[a646e152e4bfb609f5372833f5b8c001b4e523c3154f6fea43b154fe04c6e120], 1500))
                                    25: Parent
                                    26: Push(Hash(HASH[f434d46bb16f841310d2e120a259ad1aca2d679fd330ac0fd13d145c11a6b335]))
                                    27: Child
                                    28: Child
                                    29: Child
                                    30: Child
                                    31: Child
                                    32: Child
                                    33: Push(KVHashCount(HASH[c32ae0189f148c2390791534ff4bc205fabb53a7c7d15f109a4354170045308c], 100000))
                                    34: Parent
                                    35: Push(Hash(HASH[1a1c99166d7b1e1eb9087404f3bfae82d749a3a7a763da654f48c5d314e21e76]))
                                    36: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

This is the most interesting layer in the chapter. The byColor property-name tree (widget/color) is a ProvableCountTree, so every internal merk node carries its subtree's running count — visible here as KVHashCount(HASH[…], N) ops where N is the count contribution of that subtree (the values 51100, 25500, 12700, 6300, 3100, 700, 300, 1500, 100000 show up in the literal output above). The verified element (color_00000500) lands on op 18 in the middle of these KVHashCount ops, and the surrounding ops walk the binary boundary path so the prover can recompute the parent merk hash. For a point-lookup query like this, the ProvableCountTree machinery is overkill — it carries running counts the verifier doesn't need. Query 7 is where this pays off.

Structurally identical to Query 2 — only the property name and the count-tree depth differ. The intermediate widget/color tree is a ProvableCountTree here (vs NormalTree for byBrand), but the proof doesn't care about that: it descends through the property-name tree and surfaces the value-tree CountTree at the bottom. The ProvableCountTree upgrade matters for Query 7 (range aggregate), not for point lookup.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> CO["color: ProvableCountTree"]:::path
  CO ==> C500["color_00000500: CountTree count=100"]:::target
  CO -.-> C000["color_00000000"]:::faded
  CO -.-> CMore["..."]:::faded
  WD -.-> PK["[0]"]:::faded
  WD -.-> BR["brand"]:::faded
  C500 -.-> C500_0["[0]: 100 refs"]:::faded

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#d29922,color:#0d1117,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Layers 1–4 are byte-for-byte identical to Query 1. The L5 widget doctype merk-tree proof differs from Query 2: here color is the queried key (under an opaque KVHash[a29e...] root in the proof view), and the descent into Layer 6 enters a ProvableCountTree rather than a NormalTree.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `color`)"]
    direction TB
    L5_root["KVHash[a29e...]<br/>(opaque kv root)"]:::sibling
    L5_left["HASH[9862...]<br/>(left subtree, opaque)"]:::sibling
    L5_q["<b>color</b><br/>kv_hash=HASH[7956...]<br/>value: ProvableCountTree<br/>(descent into byColor)"]:::queried
    L5_root --> L5_left
    L5_root --> L5_q
  end

  subgraph L6["Layer 6 — byColor ProvableCountTree merk-tree (TARGET layer)"]
    direction TB
    L6_target["<b>color_00000500</b><br/>KVValueHashFeatureTypeWithChildHash:<br/>kv_hash=HASH[47b0...]<br/>value: <b>CountTree count=100</b><br/>feature: ProvableCountedMerkNode(100)<br/>child_hash=HASH[4f7f...]"]:::target
    L6_boundary["Boundary commitments (36 merk ops):<br/>9 KVHashCount running totals carrying<br/>per-subtree counts<br/>(700, 1500, 3100, 6300, 12700,<br/>25500, 51100, 100000, 300)<br/>+ 9 Hash subtree commitments"]:::sibling
    L6_target --> L6_boundary
  end

  L5_q -. "ProvableCountTree(merk_root[byColor])" .-> L6_target

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

Same structural shape as Query 2 (point lookup → CountTree), but the boundary commitments carry KVHashCount(HASH, N) ops instead of plain KVHash(HASH). The running totals are dead weight for this point lookup — the verifier never reads them — but they're the same commitments Query 7 will integrate over.

Query 4 — Compound Equal-only (byBrandColor)

select  = COUNT
where   = brand == "brand_050" AND color == "color_00000500"
prove   = true

Path query:

path:         ["@", contract_id, 0x01, "widget", "brand", "brand_050", "color"]
query items:  [Key("color_00000500")]

Verified element:

path:        ["@", contract_id, 0x01, "widget", "brand", "brand_050", "color"]
key:         "color_00000500"
element:     CountTree { count_value_or_default: 1 }

Proof size: 1 911 B.

Proof display:

Expand to see the structured proof (6 layers — the deepest descent in the chapter) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
                              1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
                              2: Parent
                              3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
                              4: Child)
                            lower_layers: {
                              brand => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[fb5eb23b3135d9c226e61f004ffb43abae104238d8a1ea7bc60e8ec6ba271596]))
                                    1: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
                                    2: Parent
                                    3: Push(Hash(HASH[19ec5730af134e9ac980bbea92c2978212c8efe750a467ab54f073626e0ca2f5]))
                                    4: Push(KVHash(HASH[87bc6e7e1e465b8dcdaf95db9957a455d6bd7c75976db122f33e592fe75f1e4a]))
                                    5: Parent
                                    6: Push(Hash(HASH[a0a354f2bb59b8169253aebabb52afcc3c59c4c60da203c8887abb679d747168]))
                                    7: Push(KVHash(HASH[fc6b1d0237f8ff89b555e9a14480ae1c5b80d529a0f9fb5e681ea7ecd157d3da]))
                                    8: Parent
                                    9: Push(KVValueHash(brand_050, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[53dbd6216cccdddf16f3eb0f849aed0c0cea987a718f5b43493abf0a14e83eb9]))
                                    10: Child
                                    11: Push(KVHash(HASH[027ac8b1bc9788118b27c13d0b3c3bd3661ef6a89a775a6b6bf78aa7e6f8ed3d]))
                                    12: Parent
                                    13: Push(Hash(HASH[7a5dc3002e6cb6c92e54d554e5af85e9c2ba64ee9c5f80e6489075cc5f3f0d55]))
                                    14: Child
                                    15: Push(KVHash(HASH[3363630479f1abe6e003b1e1d50b5118e55ad2efb7a3f4b3b6df902bea72ac9a]))
                                    16: Parent
                                    17: Push(Hash(HASH[3857faef5ddb06e201f1e65cf42f15d6c9b0dc67e7f73eb182b520854e9bb648]))
                                    18: Child
                                    19: Child
                                    20: Child
                                    21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
                                    22: Parent
                                    23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
                                    24: Child)
                                  lower_layers: {
                                    brand_050 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[2190c6fcd140792fd12be66cd631f97475b9ab3f19417a26d94798115ee46160]))
                                          1: Push(KVValueHash(color, NonCounted(ProvableCountTree(636f6c6f725f3030303030353131, 1000, flags: [0, 0, 0])), HASH[b1cedc48940faedea8b64bff8c8113344acdb1fd8eff37c567099b167b3c5861]))
                                          2: Parent)
                                        lower_layers: {
                                          color => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[7e2704a94ce3e08ee1e8249a7272e1860251f66b9816581f6191010bc0f15dfe]))
                                                1: Push(KVHashCount(HASH[ccfe3e95a84b22305b9815064d7d4e54b6ab0ca8efab26ca408391f2fad2b83e], 511))
                                                2: Parent
                                                3: Push(Hash(HASH[3dac20af894289bb36212087f48cfdd2c05713c5e134804638428a8d0ac8c905]))
                                                4: Push(KVHashCount(HASH[1a3db8540380b26ead4be363cd35a4a0036ec9a92cf3c9527db540f0878ab168], 255))
                                                5: Parent
                                                6: Push(Hash(HASH[61f333ba1ad78624c009fa514ac69407a1438cb7d7807e9d5de1d75223137235]))
                                                7: Push(KVHashCount(HASH[94e2ea0c17ffbf050dcba5e04928ae2ecf9fe21567e7fac5b93ad30040df8dfe], 127))
                                                8: Parent
                                                9: Push(Hash(HASH[a8571229cee7010a54ae9890a410edd246c079930d672fb4cdcd4a13c2bcc437]))
                                                10: Push(KVHashCount(HASH[6b04a6eb8e698272ec0ff801c76dc9d65a1d47ef5ae1beb9747058cdda05e2d6], 63))
                                                11: Parent
                                                12: Push(Hash(HASH[8c12a68cebf211bbbd3937519662a7c3ed5bce92cf1e99869548c06a5639de15]))
                                                13: Push(KVHashCount(HASH[9533ef417b8eed113b81bb2d7e56c81012d11835819a59769deebfc1a7e0eafd], 31))
                                                14: Parent
                                                15: Push(Hash(HASH[2f385d9fd5157a78a1cb1456050d3fb87f809e30b93a1963582edcedd31bfd0e]))
                                                16: Push(KVHashCount(HASH[74ad467d4132703ae845149ce86de7c71d9c6fc7472e76e9bb1b81bc182abe53], 7))
                                                17: Parent
                                                18: Push(KVValueHashFeatureTypeWithChildHash(color_00000500, CountTree(00, 1, flags: [0, 0, 0]), HASH[7f1d988845d9c82b9d1146f2188b09bf704d31647ee2a26054e69ed897de3750], ProvableCountedMerkNode(1), HASH[078e3476060013c48bfc77330dc75d4fedc585469f581a66dc6b7b32f6d4d60a]))
                                                19: Push(KVHashCount(HASH[48fc5c3cca2265eaeb5b86505f628660ffae9deee96cda8c26d7139f22ce0410], 3))
                                                20: Parent
                                                21: Push(Hash(HASH[c6e38bf64efcfd1d46d4ccd7937858870c7406f3abf31ca36148860d12c6b950]))
                                                22: Child
                                                23: Child
                                                24: Push(KVHashCount(HASH[67ab38f74160b7c15e62a37fee3d0c193156061f3b013190ce2a154e4164c7b0], 15))
                                                25: Parent
                                                26: Push(Hash(HASH[49e4ecf80eead3552c93208b39c4fa9a5a3b64b7c63b385e53e47cb6e7bd8759]))
                                                27: Child
                                                28: Child
                                                29: Child
                                                30: Child
                                                31: Child
                                                32: Child
                                                33: Push(KVHashCount(HASH[e735a44484a03e4f67ef4c79f370e2b2c4b0b98d942c5b1dca53039a031354b3], 1000))
                                                34: Parent
                                                35: Push(Hash(HASH[bf41c24632983b5858dcd20a04e0e0da6e7cacef58679e69e7859619dded444e]))
                                                36: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

This is the deepest descent in the chapter. The path threads:

  1. The two intermediate GroveDB-wrapper layers (@ and 0x01).
  2. The widget doctype.
  3. The byBrand property-name tree.
  4. The byBrand value tree for brand_050 (visible in Query 2 already, here it's an intermediate stop with CountTree(636f6c6f72, 1000, …) — same element, same count).
  5. The byBrandColor continuation (color — NonCounted(ProvableCountTree)).
  6. The byBrandColor terminator value tree, finally arriving at color_00000500 with CountTree(00, 1, …) — the bench's deterministic schedule gives exactly 1 doc per (brand, color) pair.

The proof descends through byBrandColor's prefix value tree (brand_050) into its continuation (color, the NonCounted-wrapped subtree shown earlier) and resolves at the terminator value tree color_00000500. The count is 1 because the bench's fixture has exactly one document per (brand, color) pair.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree"]:::path
  BR ==> B050["brand_050: CountTree count=1000"]:::path
  B050 ==> B050_C["color: NonCounted(ProvableCountTree)"]:::path
  B050_C ==> B050_C_500["color_00000500: CountTree count=1"]:::target
  B050_C -.-> Other["other colors"]:::faded
  B050 -.-> B050_0["[0]: 1000 byBrand refs"]:::faded
  BR -.-> Brands["other brands"]:::faded
  WD -.-> CO["color"]:::faded
  WD -.-> PK["[0]"]:::faded

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;
  linkStyle 3 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

This is the deepest descent in the chapter — four extra grove layers below the common Layers 1–4. Layer 5 enters brand (same as Query 2); Layer 6 lands on brand_050 but doesn't terminate (its CountTree has a continuation child); Layer 7 is brand_050's single-key sub-merk-tree carrying the byBrandColor continuation; Layer 8 is the byBrandColor terminator value tree where color_00000500 is the actual target.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree"]
    direction TB
    L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
    L5_left["HASH[9862...]"]:::sibling
    L5_right["HASH[6c36...]"]:::sibling
    L5_q --> L5_left
    L5_q --> L5_right
  end

  subgraph L6["Layer 6 — byBrand merk-tree (intermediate stop)"]
    direction TB
    L6_q["<b>brand_050</b><br/>kv_hash=HASH[53db...]<br/>value: CountTree count=1000<br/>(continuation child via lower_layer)"]:::queried
    L6_boundary["Boundary commitments (24 merk ops):<br/>6 KVHash sibling brands + 6 Hash subtrees"]:::sibling
    L6_q --> L6_boundary
  end

  subgraph L7["Layer 7 — brand_050's continuation merk-tree (single key)"]
    direction TB
    L7_q["<b>color</b><br/>kv_hash=HASH[b1ce...]<br/>value: NonCounted(ProvableCountTree)<br/>(descent into byBrandColor)"]:::queried
    L7_left["HASH[2190...]"]:::sibling
    L7_q --> L7_left
  end

  subgraph L8["Layer 8 — byBrandColor color sub-tree (TARGET layer)"]
    direction TB
    L8_target["<b>color_00000500</b><br/>KVValueHashFeatureTypeWithChildHash:<br/>kv_hash=HASH[7f1d...]<br/>value: <b>CountTree count=1</b><br/>feature: ProvableCountedMerkNode(1)<br/>child_hash=HASH[078e...]"]:::target
    L8_boundary["Boundary commitments (36 merk ops):<br/>KVHashCount running totals<br/>(3, 15, 1000, ...) + Hash subtrees<br/>covering ~1000 colors under brand_050"]:::sibling
    L8_target --> L8_boundary
  end

  L5_q -. "Tree(merk_root[byBrand])" .-> L6_q
  L6_q -. "CountTree continuation (child_hash)" .-> L7_q
  L7_q -. "NonCounted(ProvableCountTree)" .-> L8_target

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The two extra grove layers (L7 + L8) over Query 2's two-layer descent are what makes this the chapter's heaviest proof (1 911 B). The L7 layer is structurally trivial (one key) — its cost is the descent overhead, not the merk-tree boundary. L8 carries the same ProvableCountTree shape as Query 3's L6, but the contained key namespace is restricted to colors that co-occur with brand_050.

Query 5 — In on byBrand

select  = COUNT
where   = brand IN ["brand_000", "brand_001"]
prove   = true

Path query:

path:         ["@", contract_id, 0x01, "widget", "brand"]
query items:  [Key("brand_000"), Key("brand_001")]

Verified elements (one per In value, returned in lex-asc order):

path:        ["@", contract_id, 0x01, "widget", "brand"]
key:         "brand_000"
element:     CountTree { count_value_or_default: 1000 }

path:        ["@", contract_id, 0x01, "widget", "brand"]
key:         "brand_001"
element:     CountTree { count_value_or_default: 1000 }

Proof size: 1 102 B.

Proof display:

Expand to see the structured proof (5 layers, two `KVValueHash` items at the byBrand level) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
                              1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
                              2: Parent
                              3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
                              4: Child)
                            lower_layers: {
                              brand => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(KVValueHashFeatureTypeWithChildHash(brand_000, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[90ff6f6d9a3d901195982128130677243bfd27b75736206f3c8400966ef0d37b], BasicMerkNode, HASH[19b58883c492e746861db1e6ad07529a5a91cc8330af522682486db9346d6875]))
                                    1: Push(KVValueHashFeatureTypeWithChildHash(brand_001, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[484ca11fb4ec8f479be1f78af903ce0c9d4fe630517579fb0172c2576d6b9652], BasicMerkNode, HASH[0bf12023f8e067c12db4cec1583909a0283878d6d909c76196736299750b5879]))
                                    2: Parent
                                    3: Push(Hash(HASH[8ca09dadc802a7efe03534ce4ad991b2f191f368878754a37b5e5c03d9498dab]))
                                    4: Child
                                    5: Push(KVHash(HASH[e5297b3ebe81c6435c29f712074da5f7c90265e12ed3d4f5af1f6d900e50c9f1]))
                                    6: Parent
                                    7: Push(Hash(HASH[50f373fd01dea89c992779764dff82cc7200b492be8f5cf3721627d5323bcbff]))
                                    8: Child
                                    9: Push(KVHash(HASH[cf78c9f1b1a1204bb2e437806f52c21e331392de3436388572bd1fa4bce1cdc7]))
                                    10: Parent
                                    11: Push(Hash(HASH[4a8dc186a95c8c4a1252fb51dbc407727f588eb5bdc8313c96f5c29889e13926]))
                                    12: Child
                                    13: Push(KVHash(HASH[d00ee7653e34e47d46004929b13ded33dff069ed9cc88342cecdf66a65fd8401]))
                                    14: Parent
                                    15: Push(Hash(HASH[7f1d17b9632f0bd440dacf5e841025482bc1d8145df3650301a95a5ee71ce8c8]))
                                    16: Child
                                    17: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
                                    18: Parent
                                    19: Push(Hash(HASH[eaef9fc530408393bc321409414814b290309a861f474a925a922250327affc6]))
                                    20: Child
                                    21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
                                    22: Parent
                                    23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
                                    24: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

The two Push(KVValueHashFeatureTypeWithChildHash(brand_NNN, CountTree(…, 1000, …), …)) ops are the actual verified elements — both inlined in the byBrand layer's merk proof. They share the same parent path (@/.../widget/brand); the verifier-side verify_query returns both as siblings rather than descending one more layer per value (which is what the legacy Key([0]) shape would have forced for a normal-countable index, but no longer does — every countable terminator's value tree is a CountTree). The remaining 22 ops are the boundary-path hashes that prove brand_000 and brand_001 actually occupy the merk-tree positions claimed.

The outer query enumerates Key(in_value) items at the property-name subtree; each resolved element is itself a value-tree CountTree. No subquery is set — the In values' value trees are the count-bearing elements. The verifier reads the per-In value from grove_key (rather than from path[base_path_len], which is how it would for a trailing-Equal compound). The caller sums the two count_value_or_default reads (or surfaces them as per-group entries if group_by = ["brand"]).

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree"]:::path
  BR ==> B000["brand_000: CountTree count=1000"]:::target
  BR ==> B001["brand_001: CountTree count=1000"]:::target
  BR -.-> BMore["brand_002 ... brand_099"]:::faded
  B000 -.-> B000_0["[0]: 1000 refs"]:::faded
  B001 -.-> B001_0["[0]: 1000 refs"]:::faded
  WD -.-> PK["[0]"]:::faded
  WD -.-> CO["color"]:::faded

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Same L5 shape as Query 2 (brand queried, two opaque sibling subtrees). At Layer 6 the proof inlines two KVValueHashFeatureTypeWithChildHash(brand_NNN, ...) ops at the byBrand layer — both verified elements share the same parent path, so no extra grove descent is needed.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree"]
    direction TB
    L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
    L5_left["HASH[9862...]"]:::sibling
    L5_right["HASH[6c36...]"]:::sibling
    L5_q --> L5_left
    L5_q --> L5_right
  end

  subgraph L6["Layer 6 — byBrand merk-tree (TWO TARGETS)"]
    direction TB
    L6_t1["<b>brand_001</b><br/>kv_hash=HASH[484c...]<br/>value: <b>CountTree count=1000</b><br/>child_hash=HASH[0bf1...]"]:::target
    L6_t0["<b>brand_000</b><br/>kv_hash=HASH[90ff...]<br/>value: <b>CountTree count=1000</b><br/>child_hash=HASH[19b5...]"]:::target
    L6_boundary["Boundary commitments (22 merk ops):<br/>7 KVHash opaque sibling brands<br/>+ 7 Hash subtree commitments<br/>(prove the two targets' adjacent positions)"]:::sibling
    L6_t1 --> L6_t0
    L6_t1 --> L6_boundary
  end

  L5_q -. "Tree(merk_root[byBrand])" .-> L6_t1

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

Marginally larger than Query 2 (1 102 B vs 1 041 B) — the extra cost is one extra KVValueHashFeatureTypeWithChildHash op plus one merge op. The boundary commitments shrink slightly because the two adjacent targets share part of the merk-tree path.

Query 6 — In on byColor (RangeCountable)

select  = COUNT
where   = color IN ["color_00000000", "color_00000001"]
prove   = true

Path query:

path:         ["@", contract_id, 0x01, "widget", "color"]
query items:  [Key("color_00000000"), Key("color_00000001")]

Verified elements:

path:        ["@", contract_id, 0x01, "widget", "color"]
key:         "color_00000000"
element:     CountTree { count_value_or_default: 100 }

path:        ["@", contract_id, 0x01, "widget", "color"]
key:         "color_00000001"
element:     CountTree { count_value_or_default: 100 }

Proof size: 1 381 B.

Proof display:

Expand to see the structured proof (5 layers; bottom layer carries `KVHashCount` running totals from the `ProvableCountTree`) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
                              1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
                              2: Parent
                              3: Push(KVValueHash(color, ProvableCountTree(636f6c6f725f3030303030353131, 100000), HASH[79569d595db75bbf2e9dca93a15c90b7eecf7b299632668ec410e2076d27f71c]))
                              4: Child)
                            lower_layers: {
                              color => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(KVValueHashFeatureTypeWithChildHash(color_00000000, CountTree(00, 100, flags: [0, 0, 0]), HASH[ce582ad80dab7f822798cbdcd4a7e2d454339ef5da50af688e31acb463f13bc6], ProvableCountedMerkNode(100), HASH[ad2891a5a377d25ef300546faaa2acef14cb3431490a86ed1d16d5fd69ec9e3f]))
                                    1: Push(KVValueHashFeatureTypeWithChildHash(color_00000001, CountTree(00, 100, flags: [0, 0, 0]), HASH[c4024227f61350e128189bbfdb9cb3de893aef09626680a3d2336f991c1dbb14], ProvableCountedMerkNode(300), HASH[45e2452816d75b27baa9d1b8a82a251ce218d949d003bceb2e22ce1988312c4d]))
                                    2: Parent
                                    3: Push(Hash(HASH[cb34b6fa0bd36bf67c93768f3bdbadc7c5f4f143215222ff8bc8bbff5df0dc93]))
                                    4: Child
                                    5: Push(KVHashCount(HASH[2e045e449ad64fe27461182e3f335ee8fb65183c18a3fd3e4ff175c9e767b04b], 700))
                                    6: Parent
                                    7: Push(Hash(HASH[8d73c136c1428e6cca5c6579faeb12b9cc4e7094bdbdba383097d2d05032a414]))
                                    8: Child
                                    9: Push(KVHashCount(HASH[a9f7d6ebc19c3405af2ef32cbdf4f4ec0d4a96592bb5d389f9ab0462389c6fb5], 1500))
                                    10: Parent
                                    11: Push(Hash(HASH[e131726e58ca916c5d2c3fdff06be027b7bca567b45a1854b38774b7eb429b47]))
                                    12: Child
                                    13: Push(KVHashCount(HASH[c982b92207e31779affbc3c4495d175948ca647b9c15740c0cb0f6b7fede6d0d], 3100))
                                    14: Parent
                                    15: Push(Hash(HASH[c8f1d0d58823e8fb60dbd838fdd5b984c6940e1d4d4976473e8718a638dcd64c]))
                                    16: Child
                                    17: Push(KVHashCount(HASH[8dbbcf0d3b51cfa3f8c40c815b8904b650fd51e3bb55ae40f741f7341248ac38], 6300))
                                    18: Parent
                                    19: Push(Hash(HASH[28f1a2ab09b0920e50bdfd4d062412ba9c1d39d33579d485360e7a0941675a43]))
                                    20: Child
                                    21: Push(KVHashCount(HASH[6bf705340b0ff3872a4f692fc10bae0dd9e63fa2726bb3fd284fbfc273ef24af], 12700))
                                    22: Parent
                                    23: Push(Hash(HASH[8ebe73647e431636fe22547384c36bfd83d77a0e109dd3e3f5a69e691c860f9e]))
                                    24: Child
                                    25: Push(KVHashCount(HASH[b2fa1534ef346372a7d2df562fe4fc4938bd07bc72af5a147529478af878972d], 25500))
                                    26: Parent
                                    27: Push(Hash(HASH[db461b2f973111b65f34f31313ccff5530b24fa17bc7e5313d4794783336df24]))
                                    28: Child
                                    29: Push(KVHashCount(HASH[3684347a67ceedad2ff4a7fce6ae303086543c1f146f5865dfdc23612308c05b], 51100))
                                    30: Parent
                                    31: Push(Hash(HASH[e8c957f1d52f9ae3932f1f8d3e3d7f761569b52b29ffd7dc3f4c0c976405b3b4]))
                                    32: Child
                                    33: Push(KVHashCount(HASH[c32ae0189f148c2390791534ff4bc205fabb53a7c7d15f109a4354170045308c], 100000))
                                    34: Parent
                                    35: Push(Hash(HASH[1a1c99166d7b1e1eb9087404f3bfae82d749a3a7a763da654f48c5d314e21e76]))
                                    36: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Same layer count as Query 5 (5 layers) and same inline-two-elements pattern at the bottom. The difference is in the merk-tree node type surrounding the verified elements: byColor's bottom layer is a ProvableCountTree, so each sibling's merk-path operation is a KVHashCount(HASH[...], N) (carrying the sibling's running count) rather than the plain KVHash(HASH[...]) you see in Query 5's byBrand layer. The boundary-proof ops here read like a histogram of the byColor tree's per-subtree counts (700, 1500, 3100, 6300, 12700, 25500, 51100, 100000) — that's the same information Query 7 will sum over directly without descending to any specific value tree.

Same query shape as Query 5 — outer Key-per-In-value, no subquery, per-In CountTrees resolved at the bottom. The difference vs Query 5 is the property-name tree above is a ProvableCountTree instead of NormalTree. That doesn't change the proof's structural shape, but it does mean a future color > X range query against this property has a fast path Query 5's brand doesn't.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> CO["color: ProvableCountTree"]:::path
  CO ==> C000["color_00000000: CountTree count=100"]:::target
  CO ==> C001["color_00000001: CountTree count=100"]:::target
  CO -.-> CMore["color_00000002 ... color_00000999"]:::faded
  C000 -.-> C000_0["[0]: 100 refs"]:::faded
  C001 -.-> C001_0["[0]: 100 refs"]:::faded
  WD -.-> PK["[0]"]:::faded
  WD -.-> BR["brand"]:::faded

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#d29922,color:#0d1117,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Same L5 shape as Query 3 (color queried under an opaque kv root). L6 inlines two KVValueHashFeatureTypeWithChildHash targets in the byColor ProvableCountTree — the difference from Query 5's L6 is the surrounding boundary ops carry KVHashCount running totals instead of plain KVHash.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `color`)"]
    direction TB
    L5_root["KVHash[a29e...]<br/>(opaque kv root)"]:::sibling
    L5_left["HASH[9862...]"]:::sibling
    L5_q["<b>color</b><br/>kv_hash=HASH[7956...]<br/>value: ProvableCountTree (descent)"]:::queried
    L5_root --> L5_left
    L5_root --> L5_q
  end

  subgraph L6["Layer 6 — byColor ProvableCountTree merk-tree (TWO TARGETS)"]
    direction TB
    L6_t1["<b>color_00000001</b><br/>kv_hash=HASH[c402...]<br/>value: <b>CountTree count=100</b><br/>feature: ProvableCountedMerkNode(300)"]:::target
    L6_t0["<b>color_00000000</b><br/>kv_hash=HASH[ce58...]<br/>value: <b>CountTree count=100</b><br/>feature: ProvableCountedMerkNode(100)"]:::target
    L6_boundary["Boundary commitments (34 merk ops):<br/>8 KVHashCount running totals<br/>(700, 1500, 3100, 6300, 12700,<br/>25500, 51100, 100000)<br/>+ Hash subtree commitments"]:::sibling
    L6_t1 --> L6_t0
    L6_t1 --> L6_boundary
  end

  L5_q -. "ProvableCountTree(merk_root[byColor])" .-> L6_t1

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The two-In-values pattern carries over from Query 5; the per-subtree counts on the boundary KVHashCount ops are the same data Query 7 uses as the integrand for its range aggregate.

Query 7 — Range Query (AggregateCountOnRange)

select  = COUNT
where   = color > "color_00000500"
prove   = true

Path query (different primitive — note the AggregateCountOnRange query item):

path:         ["@", contract_id, 0x01, "widget", "color"]
query items:  [AggregateCountOnRange([RangeAfter("color_00000500"..)])]

Verified payload (different verifier — GroveDb::verify_aggregate_count_query returns a single u64, not an element list):

root_hash:   0x62ee7348f4d28dd9d7cf86a6c725fa8276cfd446f6007a6000fb0e1dfefa6468
count:       49900

Proof size: 2 072 B.

Proof display:

Expand to see the structured proof (5 layers; bottom layer uses `HashWithCount` + `KVDigestCount` ops instead of `KVValueHash` — the AggregateCountOnRange-specific merk primitive) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
                              1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
                              2: Parent
                              3: Push(KVValueHash(color, ProvableCountTree(636f6c6f725f3030303030353131, 100000), HASH[79569d595db75bbf2e9dca93a15c90b7eecf7b299632668ec410e2076d27f71c]))
                              4: Child)
                            lower_layers: {
                              color => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(HashWithCount(kv_hash=HASH[b2fa1534ef346372a7d2df562fe4fc4938bd07bc72af5a147529478af878972d], left=HASH[e8368be0ff72f87a2132f09d8d68d6dca140bc3c5b048d5f4f6fc8ab9b7bc554], right=HASH[db461b2f973111b65f34f31313ccff5530b24fa17bc7e5313d4794783336df24], count=25500))
                                    1: Push(KVDigestCount(color_00000255, HASH[adfb158116847927badc07be9745a21be7e2660a8b75f8a310aba9025f91feec], 51100))
                                    2: Parent
                                    3: Push(HashWithCount(kv_hash=HASH[e4f3a5c9fdf17ccb2c7508839b2fdcfd4cd878ed1d59270929ac69ef63179402], left=HASH[848d5873de457b1be03c8c7d74733b92874f2071028fdd6d30e8ca16c18a9770], right=HASH[676f04d3603911ecd1e0d2d01c2691b173df672b40daf8a7730f73c50d39e07e], count=12700))
                                    4: Push(KVDigestCount(color_00000383, HASH[14f48ee200148a9c4c673809297bdfb71e79fe9902b130e7842fbdb18c2e1a31], 25500))
                                    5: Parent
                                    6: Push(HashWithCount(kv_hash=HASH[42a257d9bc608c6b1a419f8e081b08df9056832c72e36b5dc07c4b724fb37578], left=HASH[65b3058c7b4d9bcfcf6022645f66bbaed9dbbdb74b7dbf367bbe2240263db767], right=HASH[315927383b45959aa67b32fb26b0b7c21baf6afbb1fcdc05e9c8c43a3c02b6c6], count=6300))
                                    7: Push(KVDigestCount(color_00000447, HASH[dcbfdf897e1b1d83a55172b6fa463446cd5e016331ba075440f7f1091d02467b], 12700))
                                    8: Parent
                                    9: Push(HashWithCount(kv_hash=HASH[ada831d9c38535694323d9092ab9c42e39949c9d2e4567fafd084b0f5754b0d9], left=HASH[09229789d4fdf4baba7646d3bd12e6b77b83ce19f7f1c0918b60b3c1de5bd8ea], right=HASH[ce92f20c6b464d3ff4c95f8f1ee49149aacc50298eaed2c6a2849d588bd4a667], count=3100))
                                    10: Push(KVDigestCount(color_00000479, HASH[1e6eb9e928e8bb229309db3a4a2c0f3041c63e90eb646061e4f5d82b1d65a1ac], 6300))
                                    11: Parent
                                    12: Push(HashWithCount(kv_hash=HASH[ae65499e6a1c105c878c418b09732df2dee29cf7db74c4b2e93b989710b449d0], left=HASH[94eac0807596d751092d12f27195dc72324f45999f4fc483688a9c15c554ecf3], right=HASH[fb4298cd62e8a90af17f9133fd4c106ec1da4b16be2954fc542af6ad0f6e316e], count=1500))
                                    13: Push(KVDigestCount(color_00000495, HASH[cca12136fed93b88094fc80ceb5722b752860000478404c62f7862eb652e268f], 3100))
                                    14: Parent
                                    15: Push(HashWithCount(kv_hash=HASH[db1493f4f683045aa7604c6a06c0280fecb34b352503b148eab16e245938492f], left=HASH[50f064fdcdd8e0f3e1eb86b98dc8eb6f7a8df0b26037df202b21726a05edeb79], right=HASH[d6e96c2078316fcd74e62265173c2bb52a94ad4ef0bccac569557f675307b382], count=300))
                                    16: Push(KVDigestCount(color_00000499, HASH[66e2d072be547070b1d433cb0f05f09ef508ec4d4f0702db4f49e71896ad91bc], 700))
                                    17: Parent
                                    18: Push(KVDigestCount(color_00000500, HASH[47b0ade593a2e4e99e7d7363f5d1f692882007397f025226f19d097ca2f407fa], 100))
                                    19: Push(KVDigestCount(color_00000501, HASH[9146433eb6d43db2f109f5f7714146624bd646b27c7310f3c2cad7155eb7c741], 300))
                                    20: Parent
                                    21: Push(HashWithCount(kv_hash=HASH[bbac5fc7646d820e2912c1771333ebc83b1012619347aa04cce3c4ad13c11eea], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=100))
                                    22: Child
                                    23: Child
                                    24: Push(KVDigestCount(color_00000503, HASH[66ea1280c29a6ea350e0c6695ab80430f5d3b5dc2df0f5a4d544a918d9fba29a], 1500))
                                    25: Parent
                                    26: Push(HashWithCount(kv_hash=HASH[4d7b5c895a6fb1e451ce85a522ecf18484fd1e406945cde8df9c75ec2152757e], left=HASH[6be0f9637caa5b6c09adb59618a8a90494e2f43a5e9948dc32d68af74528578a], right=HASH[ce1146de6de82a9767edf38a5cc11b5498e57023684acbe9e20bc3104ade94cf], count=700))
                                    27: Child
                                    28: Child
                                    29: Child
                                    30: Child
                                    31: Child
                                    32: Child
                                    33: Push(KVDigestCount(color_00000511, HASH[c7fdd609ef67f184976b1bdfeb97245fdfcb33e53ff6841277def88f55bc9c41], 100000))
                                    34: Parent
                                    35: Push(HashWithCount(kv_hash=HASH[6abc81973aeff51137a002d32ac447e6b91ebf507e34a4a13ec9d1bed4516d23], left=HASH[99323fb716110f45836334025ec154fcc56193c11ee0811bdd86320c0f8164ed], right=HASH[33b9e5cbdf27883150262112aaefda71c0b725a58c3f929ad1ce1cdd3f90aacd], count=48800))
                                    36: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

The bottom layer uses different merk-proof operations than every other query so far (Query 8 shares this shape one level deeper). AggregateCountOnRange doesn't return individual elements; it walks the boundary of the requested range (color > "color_00000500") over the ProvableCountTree's internal nodes and uses two specialized operations:

  • HashWithCount(kv_hash, left, right, count) — a boundary node that hides its full subtree behind a single hash + count. The count field is the load-bearing piece: the verifier just sums these without descending. In this proof you can see count=48800 at the bottom-right boundary node (everything to the right of the range cut, plus another count=100000 showing somewhere in the in-range path), and the prover walks the cut so each HashWithCount covers a different chunk of the range.
  • KVDigestCount(key, kv_hash, count) — a boundary key inside the in-range region; the prover names the key so the verifier knows exactly where the cut is, but only commits the hash + count, not the value. Note the keys here climb monotonically (color_00000255 → 383 → 447 → 479 → 495 → 499 → 500 → 501 → 511); each one names a binary-tree boundary node on the path from the range start (color_00000500) to the right edge of the tree.

The final summed count: 49900 is what the verifier returns. There's no CountTree(…) element in this proof — the running totals inside HashWithCount / KVDigestCount are the proof's count surface, committed into the ProvableCountTree's merk root at insertion time.

Together with Query 8 (the compound brand == X AND color > Y variant), this is one of two queries in the chapter that use a different GroveDB primitive. Instead of resolving N specific keys, AggregateCountOnRange walks the boundary of the requested range over widget/color's ProvableCountTree and sums the per-node counts already committed inside that tree. The proof carries the boundary merk path and the running total; the verifier returns just the count.

The reason this works only with rangeCountable: true (Query 5's byBrand couldn't do the equivalent) is that widget/color is a ProvableCountTree — its internal merk nodes carry running counts. widget/brand is a plain NormalTree; it would have to enumerate every brand and sum their counts (which is what brand IN [...] does, but for an unbounded range that's not a feasible proof shape).

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> CO["color: ProvableCountTree<br/>(internal merk nodes carry running counts)"]:::target
  CO -.-> C500["color_00000500 (boundary)"]:::faded
  CO -.-> CMore["color_00000501 ... color_00000999<br/>(in range, summed via merk-node counts)"]:::faded
  CO -.-> CBelow["color_00000000 ... color_00000499<br/>(below range, skipped)"]:::faded
  WD -.-> PK["[0]"]:::faded
  WD -.-> BR["brand"]:::faded

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Same L5 shape as Query 3 / Query 6 (color queried under an opaque kv root). L6 is fundamentally different from every other query: no individual elements are returned — the proof walks the boundary of the range color > "color_00000500" over the ProvableCountTree and sums per-node counts directly. The merk ops at L6 are HashWithCount (boundary subtree hashes carrying their full subtree count) and KVDigestCount (named boundary keys with hash + count, no value).

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `color`)"]
    direction TB
    L5_root["KVHash[a29e...]<br/>(opaque kv root)"]:::sibling
    L5_left["HASH[9862...]"]:::sibling
    L5_q["<b>color</b><br/>kv_hash=HASH[7956...]<br/>value: ProvableCountTree count=100000<br/>(descent into byColor)"]:::queried
    L5_root --> L5_left
    L5_root --> L5_q
  end

  subgraph L6["Layer 6 — byColor ProvableCountTree merk-tree (range-aggregate cut)"]
    direction TB
    L6_result["<b>Aggregate count = 49900</b><br/>(returned by verify_aggregate_count_query —<br/>a single u64, not an element list)"]:::target
    L6_inrange["KVDigestCount ops along the in-range path:<br/>color_00000500 (count=100), color_00000501 (300),<br/>color_00000503 (1500), color_00000511 (100000)"]:::sibling
    L6_boundary["HashWithCount boundary nodes covering<br/>chunks of the cut: counts<br/>(25500, 12700, 6300, 3100, 1500, 300, 100, 700, 48800)<br/>+ KVDigestCount path keys above the cut<br/>(color_00000255, 383, 447, 479, 495, 499)"]:::sibling
    L6_result --> L6_inrange
    L6_result --> L6_boundary
  end

  L5_q -. "ProvableCountTree(merk_root[byColor])" .-> L6_result

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The ProvableCountTree's value isn't to expose individual elements — it's to make the summation itself O(log n) instead of O(distinct values in range). The proof bytes are larger than Query 6's two-element point lookup (~2 KB vs ~1.4 KB) because the AggregateCountOnRange primitive has more structural overhead per result, but it scales to any size range in fixed proof bytes, where the point-lookup shape grows linearly with the number of resolved keys.

Query 8 — Compound Equal-plus-Range (byBrandColor)

select  = COUNT
where   = brand == "brand_050" AND color > "color_00000500"
prove   = true

Path query (the prefix brand == X fixes one byBrandColor leg; the range walks the terminator's ProvableCountTree):

path:         ["@", contract_id, 0x01, "widget", "brand", "brand_050", "color"]
query items:  [AggregateCountOnRange([RangeAfter("color_00000500"..)])]

Verified payload (same primitive as Query 7 — GroveDb::verify_aggregate_count_query returns a single u64):

root_hash:   0x62ee7348f4d28dd9d7cf86a6c725fa8276cfd446f6007a6000fb0e1dfefa6468
count:       499

The bench's deterministic schedule gives every brand all 1 000 colors; the strict > cut at color_00000500 leaves color_00000501..color_00000999 = 499 colors paired with brand_050, each contributing exactly 1 document.

Proof size: 2 656 B.

Proof display:

Expand to see the structured proof (8 layers — same descent as Query 4 down to `brand_050`'s color subtree, then `HashWithCount` / `KVDigestCount` ops over the byBrandColor terminator) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
                              1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
                              2: Parent
                              3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
                              4: Child)
                            lower_layers: {
                              brand => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[fb5eb23b3135d9c226e61f004ffb43abae104238d8a1ea7bc60e8ec6ba271596]))
                                    1: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
                                    2: Parent
                                    3: Push(Hash(HASH[19ec5730af134e9ac980bbea92c2978212c8efe750a467ab54f073626e0ca2f5]))
                                    4: Push(KVHash(HASH[87bc6e7e1e465b8dcdaf95db9957a455d6bd7c75976db122f33e592fe75f1e4a]))
                                    5: Parent
                                    6: Push(Hash(HASH[a0a354f2bb59b8169253aebabb52afcc3c59c4c60da203c8887abb679d747168]))
                                    7: Push(KVHash(HASH[fc6b1d0237f8ff89b555e9a14480ae1c5b80d529a0f9fb5e681ea7ecd157d3da]))
                                    8: Parent
                                    9: Push(KVValueHash(brand_050, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[53dbd6216cccdddf16f3eb0f849aed0c0cea987a718f5b43493abf0a14e83eb9]))
                                    10: Child
                                    11: Push(KVHash(HASH[027ac8b1bc9788118b27c13d0b3c3bd3661ef6a89a775a6b6bf78aa7e6f8ed3d]))
                                    12: Parent
                                    13: Push(Hash(HASH[7a5dc3002e6cb6c92e54d554e5af85e9c2ba64ee9c5f80e6489075cc5f3f0d55]))
                                    14: Child
                                    15: Push(KVHash(HASH[3363630479f1abe6e003b1e1d50b5118e55ad2efb7a3f4b3b6df902bea72ac9a]))
                                    16: Parent
                                    17: Push(Hash(HASH[3857faef5ddb06e201f1e65cf42f15d6c9b0dc67e7f73eb182b520854e9bb648]))
                                    18: Child
                                    19: Child
                                    20: Child
                                    21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
                                    22: Parent
                                    23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
                                    24: Child)
                                  lower_layers: {
                                    brand_050 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[2190c6fcd140792fd12be66cd631f97475b9ab3f19417a26d94798115ee46160]))
                                          1: Push(KVValueHash(color, NonCounted(ProvableCountTree(636f6c6f725f3030303030353131, 1000, flags: [0, 0, 0])), HASH[b1cedc48940faedea8b64bff8c8113344acdb1fd8eff37c567099b167b3c5861]))
                                          2: Parent)
                                        lower_layers: {
                                          color => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCount(kv_hash=HASH[4f8d29f51f626326fa5a3d4aa210a07eddf53121888aa5788625ae774be9bc37], left=HASH[ec92140543f4bd56112e8eaf4cb9796b1986d56b0bf721d81fc7d6a699d16a50], right=HASH[1eb29f80ffaac4878420ecfc9337e6181c9e6fc30608fc5475cf0b808f51a31d], count=255))
                                                1: Push(KVDigestCount(color_00000255, HASH[2ed4d50b30e917eceacb3356eb88057e490f9d98ebf6123d25535ff502d2da2b], 511))
                                                2: Parent
                                                3: Push(HashWithCount(kv_hash=HASH[80de09ce45f1c62d0532139ca67a93d88a293ce8139354e0e4751381346f64e7], left=HASH[32a8b4be78632242774668fd49f8db72d5f261d964f33ed9c0780ca99708ba20], right=HASH[af60a5ba4e39fa6326fd77dff0de304cc4cbac75c0702200a97d099f33617496], count=127))
                                                4: Push(KVDigestCount(color_00000383, HASH[fe27bc251ea815fbd838146098daf0662fe214425a5befaec84c960dadbff89b], 255))
                                                5: Parent
                                                6: Push(HashWithCount(kv_hash=HASH[827791c9001bdf85512aed74a917156299ad6b1a50abe27e03939cb745000dfa], left=HASH[4b5363b3bd01883530360ef09c2b645c6f744988e24c17e432bc2c3321d41541], right=HASH[c444ec932284bb1bae3b45f3f54ed9f3922fd85aeecea01dd10341b889c41137], count=63))
                                                7: Push(KVDigestCount(color_00000447, HASH[e7d9bb66af76a1b8600a9fb5d904f54825b61fb5e8004cdbfb4f42134455953a], 127))
                                                8: Parent
                                                9: Push(HashWithCount(kv_hash=HASH[11a374adf740d562dde32325c07b28949981033d310beafc1d90a3d44fb0bd6a], left=HASH[826dab51d3fd831414ae5344e837343104f0212e1c5ca57014951beba53f89d0], right=HASH[02ebfd24b8c7fd10b74bda8856344c4fd7287ce31ebdd6e9676c9a1a6e5943cc], count=31))
                                                10: Push(KVDigestCount(color_00000479, HASH[6151f4f40176302ed6a27f77fd687bbae015a09ca80ad4af6f80e7c29e8a3595], 63))
                                                11: Parent
                                                12: Push(HashWithCount(kv_hash=HASH[b93259768b6500a9b757c4a90e981f0e3a8a848b275b862f16f5b242310cb65a], left=HASH[b1f724e4b2546d1d92059a72076868112b5b6187b6d227fa834d3ff3579f7b8c], right=HASH[fd1765117ce2a3f6deca713039d81726b05eecab4119e223753dee5fd989d610], count=15))
                                                13: Push(KVDigestCount(color_00000495, HASH[034b88a8dfaf46db8b679fd72d342643d64b6937c44b06f983e5dbebd6f3b69b], 31))
                                                14: Parent
                                                15: Push(HashWithCount(kv_hash=HASH[bd58344e0fbbca9dee08997443550d1630adc59696701fb1f99c5a7e1fdb855a], left=HASH[22ef7d33de4e1d93a27009c5a3ae849ac8713c84dbac046dc615170a6b0e89a8], right=HASH[fbc94ef6e1255b8f0fffdb496258eb03a0b54c29d9f074830159d78a86e05621], count=3))
                                                16: Push(KVDigestCount(color_00000499, HASH[12672ddf0e18d172679f7ebf0ba5f6976b337066e0373ffdac8c176a6a160dcc], 7))
                                                17: Parent
                                                18: Push(KVDigestCount(color_00000500, HASH[7f1d988845d9c82b9d1146f2188b09bf704d31647ee2a26054e69ed897de3750], 1))
                                                19: Push(KVDigestCount(color_00000501, HASH[f0a8f993f517cee96055d69e48dfe51e70fe303424885194d3b7e71924af5df7], 3))
                                                20: Parent
                                                21: Push(HashWithCount(kv_hash=HASH[3b75b6239307e1a00f8596386421e623e365d4adc8451dae07cc3bcf589efc44], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=1))
                                                22: Child
                                                23: Child
                                                24: Push(KVDigestCount(color_00000503, HASH[aba3bbc16aa5a2413fd60261c5efe4d42c97f0f4b82fcc8e74af8562cc2fdfed], 15))
                                                25: Parent
                                                26: Push(HashWithCount(kv_hash=HASH[64d94410c9ae982091bff1d2fe0cf3edae7af54b43f24f613ea08f465e9fa29f], left=HASH[8d2afe8b42330b1bdd677daffde7238cf93a52146e60eeec8e08b4ce095a9ad1], right=HASH[663bf105cdfa9ffd5431d8190c55a87891da0c13c74eb6a16437526c74de889c], count=7))
                                                27: Child
                                                28: Child
                                                29: Child
                                                30: Child
                                                31: Child
                                                32: Child
                                                33: Push(KVDigestCount(color_00000511, HASH[fb4d7e1e5013a3c804045c72bd920ff81985ee986e87c9373c7041b78953d12e], 1000))
                                                34: Parent
                                                35: Push(HashWithCount(kv_hash=HASH[4ba23a437c91a135eb087602db30021bbbeeba7416d4af9317c2b1a7762ab0a4], left=HASH[cd9697f159ba87524f129190317680dd33f96cf5e8a444c9caaf264fe998746c], right=HASH[f4c9e984a836b6b3739392239b4e35c28c153dd513038a6da5294a4e327c07c0], count=488))
                                                36: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

The descent is identical to Query 4's first six layers (root → @ → contract_id → 0x01 → widget → byBrand → brand_050's value tree → color continuation), then forks at the deepest layer: where Query 4 read one KVValueHashFeatureTypeWithChildHash(color_00000500, CountTree count=1), Query 8 walks the ProvableCountTree boundary using the same HashWithCount / KVDigestCount ops Query 7 used at the doctype level. The boundary keys (color_00000255 → 383 → 447 → 479 → 495 → 499 → 500 → 501 → 503 → 511) name the same binary-merk-tree positions as Query 7's color subtree — predictably, since the bench's deterministic schedule means every brand's color subtree has the same shape.

The final count=488 at the bottom-right HashWithCount covers the upper portion of the in-range subtree (everything to the right of the visible cut); the in-range KVDigestCount ops (color_00000501 count=3, color_00000503 count=15, color_00000511 count=1000) cover named boundary positions inside that subtree. The count field on each merk node is the subtree count (including descendants), not just the named key's contribution — which is why color_00000511 count=1000 and not 1. Summing the boundary contributions yields the final count: 499.

Query 8 is the "compound == then range" shape — and the most expensive query in the chapter. It threads through the same 4 grove layers above the byBrand tree as every other query, descends through byBrand → brand_050's value tree → byBrandColor's color continuation (matching Query 4's path), then runs AggregateCountOnRange over brand_050's ProvableCountTree (matching Query 7's primitive). The result: 8 grove layers of merk-proof bytes — 2 656 B total, ~28 % larger than Query 7's single-leg range and ~39 % larger than Query 4's point-lookup compound.

The reason this even works is that byBrandColor's terminator (brand_X's color continuation) is itself a ProvableCountTree (see Document Count Trees). The compound index has rangeCountable: true, and the parent_value_tree_is_count_tree flag propagates through add_indices_for_index_level_for_contract_operations so the continuation becomes NonCounted(ProvableCountTree(...)) rather than NonCounted(NormalTree(...)). Without that, the boundary nodes wouldn't carry running counts and the verifier would have to enumerate every (brand_050, color) pair.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree"]:::path
  BR ==> B050["brand_050: CountTree count=1000"]:::path
  B050 ==> B050_C["color: NonCounted(ProvableCountTree count=1000)<br/>(internal merk nodes carry running counts)"]:::target
  B050_C -.-> CGT["color_00000501 ... color_00000999<br/>(in range, summed via merk-node counts)"]:::faded
  B050_C -.-> CBelow["color_00000000 ... color_00000500<br/>(below range, skipped)"]:::faded
  B050 -.-> B050_0["[0]: 1000 byBrand refs"]:::faded
  BR -.-> Brands["other brands"]:::faded
  WD -.-> CO["color"]:::faded
  WD -.-> PK["[0]"]:::faded

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;
  linkStyle 3 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Layers 5–7 are identical to Query 4 (widget → byBrand → brand_050's value tree → color continuation). The difference from Query 4 is entirely at Layer 8: Query 4 resolved a single color_X element with a KVValueHashFeatureTypeWithChildHash op, whereas Query 8 walks the boundary with HashWithCount and KVDigestCount ops (the same shape as Query 7's L6, just one grove layer deeper).

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree"]
    direction TB
    L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
    L5_left["HASH[9862...]"]:::sibling
    L5_right["HASH[6c36...]"]:::sibling
    L5_q --> L5_left
    L5_q --> L5_right
  end

  subgraph L6["Layer 6 — byBrand merk-tree (intermediate stop)"]
    direction TB
    L6_q["<b>brand_050</b><br/>kv_hash=HASH[53db...]<br/>value: CountTree count=1000<br/>(continuation child via lower_layer)"]:::queried
    L6_boundary["Boundary commitments (24 merk ops):<br/>6 KVHash sibling brands + 6 Hash subtrees"]:::sibling
    L6_q --> L6_boundary
  end

  subgraph L7["Layer 7 — brand_050's continuation merk-tree (single key)"]
    direction TB
    L7_q["<b>color</b><br/>kv_hash=HASH[b1ce...]<br/>value: NonCounted(ProvableCountTree count=1000)<br/>(descent into byBrandColor terminator)"]:::queried
    L7_left["HASH[2190...]"]:::sibling
    L7_q --> L7_left
  end

  subgraph L8["Layer 8 — byBrandColor color sub-tree (range-aggregate cut)"]
    direction TB
    L8_result["<b>Aggregate count = 499</b><br/>(returned by verify_aggregate_count_query)"]:::target
    L8_inrange["KVDigestCount ops in the in-range path:<br/>color_00000500 (count=1), color_00000501 (3),<br/>color_00000503 (15), color_00000511 (1000)"]:::sibling
    L8_boundary["HashWithCount boundary nodes covering<br/>chunks of the cut (counts):<br/>255, 127, 63, 31, 15, 3, 1, 7, 488<br/>+ KVDigestCount path keys above the cut:<br/>color_00000255, 383, 447, 479, 495, 499"]:::sibling
    L8_result --> L8_inrange
    L8_result --> L8_boundary
  end

  L5_q -. "Tree(merk_root[byBrand])" .-> L6_q
  L6_q -. "CountTree continuation (child_hash)" .-> L7_q
  L7_q -. "NonCounted(ProvableCountTree(merk_root))" .-> L8_result

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

Query 8 sits at the intersection of Query 4 (compound descent) and Query 7 (range-aggregate primitive). Its proof carries the descent overhead of both — the boundary commitments at L6 to position brand_050, plus the boundary commitments at L8 to position the color cut — explaining why it's the heaviest proof in the chapter despite verifying a smaller count (499) than Query 7 (49 900).

Diagram: Layer 8 binary merk-tree (the range-aggregate cut)

The 37 merk ops at Layer 8 reconstruct the entire boundary path through brand_050's color ProvableCountTree. Unlike every other query's bottom layer (which abstracts the binary tree into one "target + opaque siblings" box), the AggregateCountOnRange primitive forces the prover to reveal the structural shape of the in-range descent — so we can draw it literally.

Cyan = in-range contributions the verifier adds to the aggregate. Yellow-dashed = the boundary node color_00000500, named so the verifier can position the cut but excluded by the strict > operator. Gray = nodes/subtrees outside the range (boundary commitments needed to prove the rest of the tree's structure, but not summed).

The count field on each node is its subtree count (the node itself + all descendants in the binary merk tree), not just the named key's contribution. So color_00000511 (the root) carries count=1000 because every key in brand_050's color subtree is a descendant — not because there are 1 000 of that one key.

flowchart TB
  R["<b>color_00000511</b> (merk root)<br/>KVDigestCount, count=1000<br/>contributes <b>1</b> (itself, in-range)"]:::inrange
  R --> L1L["color_00000255<br/>KVDigestCount, count=511<br/>(out-of-range; descend right)"]:::outrange
  R --> L1R["HASH[4ba2...] (opaque subtree)<br/>HashWithCount, count=488<br/>(color_00000512 … color_00000999)<br/>contributes <b>488</b> (full subtree in-range)"]:::inrange

  L1L --> L2L["HASH[4f8d...] (opaque subtree)<br/>HashWithCount, count=255<br/>(color_00000000 … color_00000254,<br/>all out-of-range)"]:::outrange
  L1L --> L2R["color_00000383<br/>KVDigestCount, count=255<br/>(out-of-range)"]:::outrange

  L2R --> L3L["HASH[80de...]<br/>HashWithCount, count=127"]:::outrange
  L2R --> L3R["color_00000447<br/>KVDigestCount, count=127"]:::outrange

  L3R --> L4L["HASH[8277...]<br/>HashWithCount, count=63"]:::outrange
  L3R --> L4R["color_00000479<br/>KVDigestCount, count=63"]:::outrange

  L4R --> L5L["HASH[11a3...]<br/>HashWithCount, count=31"]:::outrange
  L4R --> L5R["color_00000495<br/>KVDigestCount, count=31"]:::outrange

  L5R --> L6L["HASH[b932...]<br/>HashWithCount, count=15"]:::outrange
  L5R --> L6R["color_00000503<br/>KVDigestCount, count=15<br/>contributes <b>1</b> (itself, in-range)"]:::inrange

  L6R --> L7L["color_00000499<br/>KVDigestCount, count=7<br/>(out-of-range; descend right)"]:::outrange
  L6R --> L7R["HASH[64d9...] (opaque subtree)<br/>HashWithCount, count=7<br/>(color_00000504 … color_00000510)<br/>contributes <b>7</b> (full subtree in-range)"]:::inrange

  L7L --> L8L["HASH[bd58...]<br/>HashWithCount, count=3<br/>(color_00000496 … color_00000498,<br/>all out-of-range)"]:::outrange
  L7L --> L8R["color_00000501<br/>KVDigestCount, count=3<br/>contributes <b>1</b> (itself, in-range)"]:::inrange

  L8R --> L9L["<b>color_00000500</b><br/>KVDigestCount, count=1<br/>boundary key — strict `&gt;` excludes it<br/>(named so the verifier can place the cut)"]:::boundary
  L8R --> L9R["HASH[3b75...] (opaque subtree)<br/>HashWithCount, count=1<br/>(color_00000502)<br/>contributes <b>1</b> (full subtree in-range)"]:::inrange

  classDef inrange fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:2px;
  classDef outrange fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef boundary fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px,stroke-dasharray: 6 3;

The verifier's aggregation walks this tree and sums the cyan nodes' contributions: 1 (color_00000511) + 488 (H_4ba2) + 1 (color_00000503) + 7 (H_64d9) + 1 (color_00000501) + 1 (H_3b75) = 499. Notice the asymmetry — the proof reveals a long boundary-descent path down the left of the tree (through KD_255 → KD_383 → KD_447 → KD_479 → KD_495 → KD_499) just to position the cut, even though none of those nodes contribute to the count. That's the structural floor for AggregateCountOnRange: the prover must commit one merk-binary-tree-depth's worth of boundary nodes per side of the range, regardless of how many keys actually fall inside it.

For a worst-case range (color > color_00000000, i.e. essentially the full tree), the boundary descent collapses to one node and H_4ba2-like fully-in-range subtree commitments dominate. For a narrow range like this one (cutting deep into the tree), the descent path costs more than the in-range commitments. Either way the total is O(log C') boundary nodes — Query 8 just happens to land at the unfavourable end of the constant factor.

The same in-order traversal also explains the keys' positions: a balanced merk binary tree over the 1 000 sorted color keys puts color_00000511 at the root (the ~midpoint by tree-depth, not by sort order — color_00000511 happens to land here because of grovedb-merk's AVL rotation rules on the insertion order), color_00000255 and color_00000383 (along with their descendants color_00000447 / 479 / 495 / 499) at progressive left-of-cut descents, and color_00000503 at the right child of color_00000495 (which is itself the right child of the descent path). The keys are sorted by in-order traversal, not by tree position, so don't expect them to look orderly in the diagram above.

Worked Example: How node_hash_with_count Rebuilds the Merk Root

This section uses Query 8's Layer 8 to make one thing concrete: what the verifier actually computes when it folds the proof's Push / Parent / Child ops up to the merk root. It's the same machinery underpinning every other query in the chapter — Q8 just exposes the most interesting node-hash variant (node_hash_with_count, used by ProvableCountTree).

All hashes are Blake3-256. The hash primitives live at merk/src/tree/hash.rs in grovedb. Six functions compose every node-hash in the chapter:

value_hash(v)                = Blake3( varint_len(v) || v )
kv_hash(k, v)                = Blake3( varint_len(k) || k || value_hash(v) )
kv_digest_to_kv_hash(k, vh)  = Blake3( varint_len(k) || k || vh )
combine_hash(h1, h2)         = Blake3( h1 || h2 )
node_hash(kv_h, l, r)        = Blake3( kv_h || l || r )
node_hash_with_count(kv_h, l, r, c)
                             = Blake3( kv_h || l || r || c.to_be_bytes() )

(varint_len is integer_encoding::VarInt::encode_var — the same unsigned-varint encoding used throughout grovedb. c.to_be_bytes() is the 8-byte big-endian encoding of the u64 count.)

Each proof op carries enough information to compute its subtree's node_hash. The reconstruction rule per op variant (from merk/src/proofs/tree.rs's compute_hash):

Proof opWhat's revealedSubtree node_hash formula
Hash(h)the subtree hash directlyh (no recompute)
KVHash(kv_h)the node's kv-hash onlynode_hash(kv_h, left_node_hash, right_node_hash)
KVHashCount(kv_h, c)kv-hash + node countnode_hash_with_count(kv_h, left, right, c)
HashWithCount(kv_h, left, right, c)kv-hash + both children's node-hashes + countnode_hash_with_count(kv_h, left, right, c) (no recursion — children are already pre-hashed)
KVValueHash(k, v, kv_h)full key+value + kv-hashnode_hash(kv_h, left, right)
KVDigest(k, vh)key + value-hashnode_hash(kv_digest_to_kv_hash(k, vh), left, right)
KVDigestCount(k, vh, c)key + value-hash + countnode_hash_with_count(kv_digest_to_kv_hash(k, vh), left, right, c)
KVValueHashFeatureTypeWithChildHash(k, v, kv_h, feature, child_hash)key + value + kv-hash + feature type + opaque child-layer hashcombines kv_h with child_hash (via combine_hash), then node_hash[_with_count] depending on feature

The left / right arguments are the children's reconstructed node-hashes (computed recursively from the stack as Parent / Child ops glue the subtrees together). For nodes at the leaf level of the proof's revealed structure, both children are NULL_HASH (32 zero bytes).

The example: rebuilding color_00000511's node_hash (Q8, Layer 8)

At the top of Layer 8's binary merk-tree, the proof has three ops that together produce the merk root hash for brand_050's color ProvableCountTree:

op 33:  Push(KVDigestCount(
            key   = "color_00000511",
            vh    = HASH[fb4d7e1e5013a3c804045c72bd920ff81985ee986e87c9373c7041b78953d12e],
            count = 1000))
op 34:  Parent     (links the running left-side subtree onto color_00000511 as its left child)
op 35:  Push(HashWithCount(
            kv_h  = HASH[4ba23a437c91a135eb087602db30021bbbeeba7416d4af9317c2b1a7762ab0a4],
            left  = HASH[cd9697f159ba87524f129190317680dd33f96cf5e8a444c9caaf264fe998746c],
            right = HASH[f4c9e984a836b6b3739392239b4e35c28c153dd513038a6da5294a4e327c07c0],
            count = 488))
op 36:  Child      (links the just-pushed HashWithCount as color_00000511's right child)

Call them KD_511, HWC_4ba2. By the time we reach op 33 the left-side stack already holds node_hash_left — the recursively-computed node-hash of the whole left subtree rooted at color_00000255 (255 + 1 + 255 = 511 keys, including the boundary-descent path). We'll trust that value here; it's the output of folding ops 0–32 with the same machinery applied recursively.

Step 1: Compute the right child's node_hash directly from HWC_4ba2.

Because HashWithCount already carries the children's node-hashes (cd96... and f4c9...) and the node's own kv-hash (4ba2...) and count (488), no recursion is needed — the verifier just plugs the four values into node_hash_with_count:

node_hash_right
  = node_hash_with_count(kv_h=4ba2..., left=cd96..., right=f4c9..., count=488)
  = Blake3( 4ba23a437c91a135eb087602db30021bbbeeba7416d4af9317c2b1a7762ab0a4
         || cd9697f159ba87524f129190317680dd33f96cf5e8a444c9caaf264fe998746c
         || f4c9e984a836b6b3739392239b4e35c28c153dd513038a6da5294a4e327c07c0
         || 0x00000000000001E0 )                          // 488 as big-endian u64

That's a single Blake3 call over 32 + 32 + 32 + 8 = 104 bytes.

Step 2: Compute color_00000511's kv-hash from its KVDigestCount op.

The proof reveals the key ("color_00000511", 14 bytes ASCII) and the value-hash (fb4d..., 32 bytes). kv_digest_to_kv_hash folds them into the node's kv-hash:

kv_h_511
  = kv_digest_to_kv_hash(key="color_00000511", value_hash=fb4d...)
  = Blake3( varint_len(14)                                // = 0x0E (one byte)
         || "color_00000511"                              // 14 ASCII bytes
         || fb4d7e1e5013a3c804045c72bd920ff81985ee986e87c9373c7041b78953d12e )

That's one Blake3 call over 1 + 14 + 32 = 47 bytes.

Step 3: Compute color_00000511's node_hash by folding the kv-hash with both children's node-hashes plus the running count.

node_hash_511
  = node_hash_with_count(
        kv_h  = kv_h_511,                                  // from Step 2
        left  = node_hash_left,                            // from ops 0..32 (the descent path)
        right = node_hash_right,                           // from Step 1
        count = 1000)
  = Blake3( kv_h_511 || node_hash_left || node_hash_right || 0x00000000000003E8 )
                                                          // 1000 as big-endian u64

Another single Blake3 call over 32 + 32 + 32 + 8 = 104 bytes.

Step 4: That's it. node_hash_511 is the merk root of Layer 8 — the byBrandColor color subtree for brand_050. The verifier then checks this against what Layer 7 claimed Layer 8's merk root would be (the NonCounted(ProvableCountTree(...)) value stored against the key "color" inside brand_050's value tree), and so on up the GroveDB layer stack until Layer 1 lands the entire chapter's root_hash = 0x62ee7348f4d28dd9d7cf86a6c725fa8276cfd446f6007a6000fb0e1dfefa6468.

Why the count is part of the hash

The crucial structural feature is the || count.to_be_bytes() tail in node_hash_with_count. Without it, the proof's running counts would be unsigned hints the verifier couldn't trust — a malicious prover could ship a KVDigestCount(color_00000511, fb4d..., 9_999_999_999) and there'd be no way to detect the lie without re-counting the documents (which is exactly what count proofs are supposed to avoid).

Binding the count into the merk root via node_hash_with_count is what lets AggregateCountOnRange skip enumeration: the verifier reads the count off the boundary commitments and trusts it because changing the count would change the merk root, which is consensus-committed.

Concretely, that's why every ProvableCountTree-derived op (KVHashCount, KVDigestCount, HashWithCount, KVValueHashFeatureTypeWithChildHash with feature ProvableCountedMerkNode(_)) routes through node_hash_with_count rather than the cheaper node_hash — see merk/src/proofs/tree.rs's compute_hash and the TreeFeatureType::ProvableCountedMerkNode branch in particular. NormalTree nodes (e.g. byBrand) use plain node_hash — their kv-hash + child hashes don't commit a count, which is why byBrand can't answer AggregateCountOnRange queries even though it's countable: "countable".

One last simplification

For nodes whose proof op already carries the kv-hash (the kvh-prefixed ops — KVHashCount, HashWithCount, KVHash), the verifier skips Step 2 entirely. For nodes whose proof op carries only the key and value-hash (KVDigest, KVDigestCount), the verifier folds them via kv_digest_to_kv_hash first (one extra Blake3 call). For nodes with KVValueHash (full key+value), the verifier recomputes the kv-hash all the way from scratch via kv_hash, which means it also re-hashes the value through value_hash. The choice is driven by how much of the node the proof needs to reveal:

  • A node on the descent path that the verifier doesn't need to materialize → emit Hash(node_hash) (1 hash, opaque).
  • A node whose existence the verifier must prove but whose value can stay opaque → emit KVHash(kv_h) or KVHashCount(kv_h, c) (kv-hash committed, value hidden).
  • A boundary node whose key the verifier needs to compare against the range → emit KVDigest(k, vh) or KVDigestCount(k, vh, c) (key revealed, value still digested).
  • A target whose full value the verifier must read → emit KVValueHash(k, v, kv_h) or the feature-typed variant.

The user-facing trade-off is proof bytes vs information revealed. The verifier's reconstruction logic is uniform: every op feeds the same node-hash formula one variant or another.

At-a-Glance Comparison

#QueryPrimitiveVerified shapeProof size
1(empty)primary-key CountTree1 CountTree, count=100000585 B
2brand == XPointLookupProof / byBrand1 CountTree, count=10001 041 B
3color == XPointLookupProof / byColor1 CountTree, count=1001 327 B
4brand == X AND color == YPointLookupProof / byBrandColor1 CountTree, count=11 911 B
5brand IN [b0, b1]PointLookupProof / byBrand2 CountTrees, sum=20001 102 B
6color IN [c0, c1]PointLookupProof / byColor2 CountTrees, sum=2001 381 B
7color > floorAggregateCountOnRange / byColoru64=499002 072 B
8brand == X AND color > floorAggregateCountOnRange / byBrandColoru64=4992 656 B

Four takeaways:

  • Query 1 is the cheapest. A doctype-level total count is one merk read; everything else descends through an index tree.
  • Query 2 and Query 6 are structurally identical despite covering different indexes (byBrand countable-only, byColor rangeCountable). The value-tree-direct shape is uniform across countability tiers — rangeCountable: true only matters for Queries 7 and 8.
  • Queries 7 and 8 use a fundamentally different verifier (verify_aggregate_count_query vs verify_query). Queries 1–6 return an element list and read count_value_or_default per branch; Queries 7 and 8 return a pre-summed u64. Query 8 is just Query 7 one grove layer deeper — same primitive, applied to byBrandColor's terminator rather than byColor's.
  • Query 8 is the most expensive. It pays for both the compound descent (Query 4's 4-extra-layer cost) and the range-aggregate boundary (Query 7's primitive). The verified count is far smaller than Query 7 (499 vs 49 900), but the proof bytes are 28 % larger because the merk-tree boundary at L8 has roughly the same shape regardless of how many keys the cut spans.

The path-query builder these examples decode lives at packages/rs-drive/src/query/drive_document_count_query/path_query.rs; the verifier mirror sits in packages/rs-drive/src/verify/document_count/. Both the prover and the verifier reconstruct the exact same PathQuery via the shared builder — touching one without the other is a Merkle-root mismatch waiting to happen, and the byte-identical contract is what makes the proof bytes here reproducible against the bench fixture.

Count Index Group By Examples

This chapter is the GROUP BY companion to Count Index Examples. It uses the same widget contract, the same 100 000-row fixture, and the same bench at packages/rs-drive/benches/document_count_worst_case.rs. Read chapter 29 first — most of the mechanics (CountTree variants, the merk-proof reconstruction algorithm, node_hash_with_count and friends) carry over unchanged.

What's different here:

  • Every query in chapter 29 returns either a single u64 aggregate or a small list of CountTrees the caller sums. The verifier-side payload shape is one count, total.
  • Every query in this chapter returns one count per group. The caller gets back a Vec<(group_key, count)> and can index it directly — no summation.

The most important thing to understand up front: group_by is two things at once — a result-shaping directive for the SDK and (for some queries) a proof-shaping directive for the prover. When you pass group_by = [...] in a count request, you're always telling the SDK "don't collapse the result into a single number — give me one count per group key." That result-shaping role is universal: it's what turns Aggregate(sum) into Entries([(key, count), …]).

Whether group_by also changes the proof bytes depends on the query shape. For queries where the underlying proof already commits one CountTree per matched key (single-property INs, for instance), the per-group breakdown is reconstructible from the existing bytes — the prover ships the same proof, the SDK just zips it with the group keys instead of summing. For range queries and certain compound shapes, the per-group breakdown can't be reconstructed from the aggregate-style proof (which commits opaque subtree counts rather than per-key counts), so passing group_by forces the prover to emit a structurally different, larger proof.

The interesting question this chapter answers is: which queries fall into which bucket, and why?

When group_by Changes the Proof (and When It Doesn't)

Filtergroup_byAggregate proof (no group_by)Group-By proofProof bytes change?
brand IN [b0, b1][brand]Q5 — 1 102 B1 102 B (2 entries)No — byte-identical
color IN [c0, c1][color]Q6 — 1 381 B1 381 B (2 entries)No — byte-identical
color > floor[color]Q7 — 2 072 B (1 u64)10 992 B (100 entries)Yes — different primitive
brand == X AND color > floor[brand, color]Q8 — 2 656 B (1 u64)not allowed in this form—

The key observation: IN clauses produce proofs that already commit one CountTree per resolved key, so adding group_by on the same property is purely a verifier-side relabel — the prover ships the same bytes, the verifier just returns them as Entries(...) instead of Aggregate(sum). This is why G1 and G2 below are not new proofs — they're Q5 and Q6 reinterpreted.

So why pass group_by at all if the proof bytes don't change? Because without it, the SDK has no way to know you want the per-key breakdown. The same brand IN ["brand_000", "brand_001"] proof can answer two different questions:

  • "How many widgets total are made by brand_000 or brand_001?" → caller passes no group_by, SDK returns Aggregate(2 000).
  • "How many widgets per brand?" → caller passes group_by = [brand], SDK returns Entries([("brand_000", 1 000), ("brand_001", 1 000)]).

The bytes on the wire and the cryptographic guarantees are identical; the only thing that changes is which result shape the SDK delivers. Think of group_by as the count-query equivalent of SELECT brand, COUNT(*) ... GROUP BY brand versus SELECT COUNT(*) ... in SQL — same scan plan, different projection.

Range queries are different. AggregateCountOnRange (chapter 29's Q7) walks the boundary of the range over a ProvableCountTree and sums per-subtree counts directly — it never resolves individual keys. GroupByRange (this chapter) has to enumerate the distinct in-range keys to label each group, so it produces a different proof shape with one CountTree (or CountTree-feature-typed element) per distinct key in the range. That's where group_by genuinely earns its bytes — the prover has to do additional work because the per-group breakdown can't be reconstructed from AggregateCountOnRange's opaque-subtree-count commitments.

Queries in this Chapter

All proof-size and behaviour numbers below come from the same bench helper (report_group_by_matrix) as chapter 29's. The dispatcher's group_by surface validation lives in DriveDocumentCountQuery::detect_mode; the per-mode path-query builders sit in packages/rs-drive/src/query/drive_document_count_query/path_query.rs (the group-by modes route to distinct_count_path_query and carrier_aggregate_count_path_query).

#QueryFilter + group_byComplexityAvg timeProof sizeVerified shapeNotes
G1In on byBrandbrand IN ["brand_000", "brand_001"]
group_by = [brand]
O(k · log B)38.6 µs1 102 BEntries(2 groups, sum = 2 000)Byte-identical to Q5
G1aIn on byBrand with an absent valuebrand IN ["brand_000", "brand_100"]
group_by = [brand]
O(k · log B)44.4 µs1 357 BEntries(1 group, sum = 1 000)One In value (brand_100) is absent — proof grows by 255 B for the absence subproof; verifier omits the absent branch from entries
G1bHigh-fanout In on byBrand (|IN| = B)brand IN [100 values]
group_by = [brand]
O(k · log B)1 532 µs10 038 BEntries(100 groups, sum = 100 000)Same shape as G1, scaled from |IN| = 2 → |IN| = 100; reveals every byBrand entry when |IN| = B
G2In on byColorcolor IN ["color_00000000", "color_00000001"]
group_by = [color]
O(k · log C)62.1 µs1 381 BEntries(2 groups, sum = 200)Byte-identical to Q6
G3Compound In + Equalbrand IN [...] AND color == Y
group_by = [brand]
O(k · (log B + log C'))106.2 µs2 842 BEntries(2 groups, sum = 2)Per-In compound resolution; two parallel Q4 descents sharing L1–L6
G4Range on byColorcolor > "color_00000500"
group_by = [color]
O(R · log C)762.9 µs10 992 BEntries(100 groups, sum = 10 000)GroupByRange: enumerates distinct in-range keys instead of Q7's boundary aggregate
G5Compound In + Rangebrand IN [...] AND color > "color_00000500"
group_by = [brand, color]
O(k · R' · log C')737.5 µs11 554 BEntries(100 groups, sum = 100)Compound In-fan-out × in-range distinct keys (G3 outer × G4 inner)
G7Carrier In + Range (byBrandColor)brand IN [...] AND color > "color_00000500"
group_by = [brand]
O(k · (log B + log C'))255.9 µs4 332 BEntries(2 groups, sum = 998)Per-In aggregate via AggregateCountOnRange as a carrier subquery; one u64 per branch
G8Carrier outer Range + Range (byBrandColor)brand > "brand_050" AND color > "color_00000500"
group_by = [brand]
O(L · (log B + log C'))523 µs18 022 BEntries(10 groups, sum = 4 990)Outer-Range carrier with a platform-max SizedQuery::limit of 10; caller may pass smaller, can't pass larger
G8aBounded carrier + bounded ACOR, descendingbrand > "brand_050" AND brand < "brand_065" AND color > "color_00000200" AND color < "color_00000400"
group_by = [brand], order_by = [(brand, desc)]
O(L · (log B + log C'))807 µs29 010 BEntries(10 groups, sum = 1 990)Bounded ranges on both axes + descending walk; same carrier shape as G8, different op variants on both range commitments
G8bSame carrier where but group_by = [brand, color]brand > "brand_050" AND color > "color_00000500"
group_by = [brand, color]
——rejectedInvalidWhereClauseComponents("count query supports at most one range where-clause; …or use group_by = [outer_range_field]withprove = true…")Two-range carrier is opened only for GroupByRange + single-field group_by; the compound shape can't fan over both ranges
G8cSame carrier where but group_by = []brand > "brand_050" AND color > "color_00000500"
group_by = []
——rejectedInvalidWhereClauseComponents("count query supports at most one range where-clause; …or use group_by = [outer_range_field]withprove = true…")Aggregate (no group_by) can't collapse the carrier's per-branch u64s into a single sum at the verifier

Complexity variables. B = distinct brands in the byBrand merk-tree (≈ 100); C = distinct colors in byColor (≈ 1 000); C' = distinct colors per brand in byBrandColor (≈ 1 000); R = distinct in-range values returned by GroupByRange (capped at 100 in this fixture by an implicit response-size limit); R' = distinct in-range values per fan-out branch (similarly capped); k = |IN| for the In-outer carrier shapes; L = the effective outer-walk limit for the Range-outer carrier shape (G8). The platform's MAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT = 10 is both the default (when the caller passes no limit) and a hard ceiling; callers may pass a smaller limit to truncate further. See G8 for the rationale. As in chapter 29, the total document count N doesn't appear — count proofs read pre-committed count_values rather than enumerating docs.

Avg time is the criterion-reported median of cargo bench --bench document_count_worst_case -- 'document_count_worst_case/query_g' on the same 100 000-row warmed fixture used by chapter 29's query_N_* cases. Each row reflects 10 samples × ~3 k–130 k iterations per sample with 2 s warm-up and 5 s measurement; the median sits within ±2 % of the mean across reruns. G1 and G2 match their Q5 / Q6 counterparts to within ~3 µs — the residual is the SDK-side zip-vs-sum cost. G4 is ~11 × Q7 because GroupByRange enumerates 100 distinct in-range CountTrees rather than walking O(log C) boundary nodes; the time difference is exactly the complexity difference predicted (O(R · log C) vs O(log C)).

Group-By Shapes That Are Not Allowed

Several plausible-looking (where, group_by) combinations are rejected by the dispatcher before any proof generation. The rejections fall into four buckets — operator/group_by mismatch, missing range window, no covering index, and one currently-deferred aggregate variant. All are surfaced as typed QuerySyntaxErrors; the precise error strings appear in the bench's [matrix] output.

1. group_by field constrained by == instead of In or range

where    = brand == "brand_050"
group_by = [brand]

count query supports only ... (rejected because == produces exactly one entry whose key equals the where-clause's value — grouping by a field that already has a single value contributes no extra information).

Why. GROUP BY [field] is meaningful only when field can take multiple values in the result set. An == clause pins the field to exactly one value, so the group_by is structurally redundant — the dispatcher rejects it rather than silently returning a single-entry response that would look like a bug. Use Q2 / Q3 (no group_by) for single-value == queries.

Applies symmetrically: where = color == X, group_by = [color] is rejected for the same reason.

2. group_by contains a range field but the where clause doesn't range over it

where    = brand IN[...] AND color == "color_00000500"
group_by = [brand, color]

GROUP BY on a range field requires a range where-clause; the range field must appear in where for the distinct walk to have a window to iterate over

Why. group_by = [in_field, range_field] (GroupByCompound) routes through distinct_count_path_query, which needs a range window on the second field to know what values to enumerate. With color == Y the second dimension collapses to a single value, so the compound walk degenerates to a point lookup — and that's what Q4 / G3 are for. For compound plus range, the where must carry a range on the second field (which is what G5 does).

3. group_by orders fields in a way no covering index can serve

where    = color IN[...] AND brand > "brand_050"
group_by = [color, brand]

where clause on non indexed property error: range count requires a range_countable: true (or summableOffCountIndex) index whose last property matches the range field

Why. The covering index for (group_by[0] = color, group_by[1] = brand) would need to be byColorBrand with rangeCountable: true on the brand terminator. The widget contract doesn't have that index — only byBrand, byColor, and byBrandColor. The dispatcher's index picker walks every declared index, finds none whose (properties, last_property_is_range_countable) shape matches the request, and rejects with the "non-indexed property" error.

The fix is contract-level: declare a byColorBrand index with rangeCountable: true if the application needs this group_by order. The dispatcher itself can't infer alternate index orders from the request alone — rangeCountable: true is an explicit opt-in on each index because it changes the on-disk tree shape (NormalTree → ProvableCountTree on the property-name subtree).


To put these three buckets in one place: every rejected (where, group_by) shape on this contract reduces to one of:

  • the group_by field's where operator doesn't admit multiple values (bucket 1),
  • the group_by has a range slot that the where doesn't fill with a range (bucket 2),
  • there's no covering rangeCountable index in property order (bucket 3).

All three checks happen at request validation, before any GroveDB work. The bench's report_group_by_matrix exercises one example of each and prints the exact error string, so adding a new contract or index shape is a quick way to see which checks each new query shape hits.

Historical note. A fourth bucket — group_by = [in_field] with where = in_field IN[...] AND range_field > floor — was rejected before grovedb PR #663. That PR added support for AggregateCountOnRange as a carrier subquery under outer Keys, which unblocked the natural single-field-group_by shape (one aggregate count per In branch) at the merk layer. The dispatcher now routes that shape to [DocumentCountMode::RangeAggregateCarrierProof]; the worked-out example is G7 below.

G1 — In on byBrand, Grouped By brand

select   = COUNT
where    = brand IN ["brand_000", "brand_001"]
group_by = [brand]
prove    = true

Path query (identical to Q5):

path:         ["@", contract_id, 0x01, "widget", "brand"]
query items:  [Key("brand_000"), Key("brand_001")]

Verified payload (the only thing that differs from Q5):

Entries([
  ("brand_000", CountTree { count_value_or_default: 1000 }),
  ("brand_001", CountTree { count_value_or_default: 1000 }),
])

The SDK zips the In values with the two resolved CountTree elements (in lex-asc order) rather than summing them as Q5's CountMode::Aggregate does.

Proof size: 1 102 B. Proof bytes are byte-identical to Q5 — same path query, same merk ops, same hash composition. The dispatcher recognises that CountMode::GroupByIn on a single-property In clause resolves through the same point_lookup_count_path_query as CountMode::Aggregate does; only the response-shaping at the very end differs.

For the verbatim proof display, see Q5 in chapter 29 — every byte of the 1 102-byte proof is the same. Or ▶ open the proof interactively in the visualizer ↗ (same encoded payload). The diagrams below show the result-shaping difference.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree"]:::path
  BR ==> B000["brand_000: CountTree count=1000"]:::target
  BR ==> B001["brand_001: CountTree count=1000"]:::target
  BR -.-> BMore["brand_002 ... brand_099"]:::faded

  SDK["Verifier returns Entries([<br/>(&quot;brand_000&quot;, 1000),<br/>(&quot;brand_001&quot;, 1000)<br/>])"]:::sdk

  B000 -.-> SDK
  B001 -.-> SDK

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Identical to Q5's Layer-5+ diagram — same merk ops, same byBrand binary tree, same two KVValueHashFeatureTypeWithChildHash targets. The only difference is what the verifier returns at the end (Entries(...) instead of Aggregate(2000)); the per-layer structure is unchanged. See chapter 29 for the diagram.

G1a — In on byBrand with one absent value, Grouped By brand

select   = COUNT
where    = brand IN ["brand_000", "brand_100"]
group_by = [brand]
prove    = true

The bench fixture has brands brand_000 … brand_099 (BRAND_COUNT = 100); brand_100 is deliberately outside that range. G1a is G1's same-shape sibling: same path query, same point_lookup_count_path_query builder, same CountMode::GroupByIn dispatch. The only structural difference is one of the In keys doesn't exist in the byBrand merk tree.

Path query (identical shape to G1; only the second key differs):

path:         ["@", contract_id, 0x01, "widget", "brand"]
query items:  [Key("brand_000"), Key("brand_100")]

Verified payload (note: only one entry — the absent branch is silently dropped):

Entries([
  ("brand_000", CountTree { count_value_or_default: 1000 }),
])

This is the load-bearing behaviour to know about: grovedb's verify_query without absence_proofs_for_non_existing_searched_keys: true drops absent-Key branches from the elements stream. The drive-side verifier (verify_point_lookup_count_proof_v0) uses the default (off) and so emits one entry per present In value, not one per requested In value. Test coverage: test_point_lookup_proof_omits_absent_in_branches_from_entries.

Caller implication. Callers MUST NOT assume entries.len() == |In|. To check whether a specific In value matched, demux entries by serialized key (the same serialize_value_for_key(field, value) the path-query builder uses for outer Keys) — see the test for the canonical pattern. A 0-count vs absent-key distinction would require passing absence_proofs_for_non_existing_searched_keys: true end-to-end, which the platform doesn't expose today.

Proof size: 1 357 B (+255 B over G1's 1 102 B). The delta is the absence subproof: grovedb walks the byBrand merk tree to commit the rightmost present key (brand_099) and the chain of Child ops that proves there's nothing between brand_099 and end-of-tree. Even though the verifier drops the absent entry, the prover must cryptographically commit to the absence — otherwise a malicious prover could omit a present branch by claiming it's absent.

Mode: CountMode::GroupByIn routed to DocumentCountMode::PointLookupProof — same as G1.

Proof display:

The absence-subproof shape is what makes G1a interesting. The L8 (byBrand value tree) layer commits both:

  1. The present branch (op 0): Push(KVValueHashFeatureTypeWithChildHash(brand_000, CountTree(636f6c6f72, 1000, …))) — brand_000 as a CountTree with count = 1000, exactly as in G1.
  2. The absence commitment (op 36): Push(KVDigest(brand_099, HASH[…])) — the rightmost present brand in the byBrand merk tree, paired with a chain of Child ops (37–42) that the verifier replays to confirm there's no key strictly between brand_099 and end-of-tree. brand_100 would have to sort after brand_099 (which is true: brand_099 < brand_100 lexicographically), so the verifier's merk-root recomputation succeeds with no brand_100 element emitted.

The bench's [gproof] G1a output dumps the full 1357-byte proof:

Expand to see the structured proof (L1–L8 for byBrand, with one present CountTree at L8 + one absence subproof at L8)
GroveDBProofV1 {
  LayerProof {                                          // L1: roots merk
    proof: Merk(
      0: Push(Hash(HASH[bd29…3b3]))                     // sibling: contracts subtree
      1: Push(KVValueHash(@, Tree(4ed2…289), HASH[…]))  // KVValueHash of `@` (data-contract subtree root) — descend
      2: Parent
      3: Push(Hash(HASH[19c9…b71]))                     // sibling
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {                                    // L2: `@` subtree
          proof: Merk(
            0: Push(KVValueHash(0x4ed2…289, Tree(01), HASH[…])))   // descend into contract-id subtree
          lower_layers: {
            0x4ed2…289 => {
              LayerProof {                              // L3: contract-id subtree
                proof: Merk(
                  0: Push(Hash(HASH[49e7…df8]))         // sibling
                  1: Push(KVValueHash(0x01, Tree(widget), HASH[…]))  // descend into doctype `widget`
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {                        // L4: doctype-prefix subtree
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(brand), HASH[…])))  // descend into byBrand index
                      lower_layers: {
                        widget => {
                          LayerProof {                  // L5: widget subtree
                            proof: Merk(
                              0: Push(Hash(HASH[9862…9d9]))                // sibling
                              1: Push(KVValueHash(brand, Tree(brand_063), HASH[…]))  // descend into byBrand value tree (rooted at `brand_063`)
                              2: Parent
                              3: Push(Hash(HASH[6c36…a86]))
                              4: Child)
                            lower_layers: {
                              brand => {
                                LayerProof {            // L6+L7+L8: byBrand value tree (binary search down to `brand_000` + absence walk to `brand_099`)
                                  proof: Merk(
                                    0: Push(KVValueHashFeatureTypeWithChildHash(brand_000, CountTree(color, 1000, flags), HASH[…], BasicMerkNode, HASH[…]))  // PRESENT — `brand_000` as CountTree(count=1000)
                                    1: Push(KVHash(HASH[…]))
                                    2: Parent
                                    3: Push(Hash(HASH[…]))
                                    4: Child
                                    … (24 intermediate `KVHash`/`Hash`/`Parent`/`Child` ops walking the binary search)
                                    35: Push(KVHash(HASH[…]))
                                    36: Push(KVDigest(brand_099, HASH[…]))    // ABSENCE COMMITMENT — rightmost present brand
                                    37: Child
                                    38: Child
                                    39: Child
                                    40: Child
                                    41: Child
                                    42: Child)
                                }}}}}}}}}}}}}}}}}

Op 36 (KVDigest(brand_099, …)) is the load-bearing piece. The verifier replays ops 37–42 (Childs) against the byBrand merk root committed at L5; any tampering — say, an honest brand_099 swapped for a malicious brand_100-shaped commitment — would change the merk root and the verification would fail.

Diagram: conceptual flow (where the absence proof sits)

flowchart TB
  RQ["IN [brand_000, brand_100]"]:::request
  RQ --> M["dispatcher → PointLookupProof<br/>(group_by = [brand])"]:::dispatch
  M --> P["point_lookup_count_path_query<br/>outer Keys = [brand_000, brand_100]"]:::path
  P --> V["grovedb walks byBrand merk tree"]:::engine
  V --> P1["brand_000 ✓ present<br/>commit CountTree(count=1000)"]:::present
  V --> P2["brand_100 ✗ absent<br/>commit rightmost present (brand_099)<br/>+ Child chain to end-of-tree"]:::absent
  P1 --> R["Proof bytes: 1357 B<br/>(1102 B for the present branch +<br/>~255 B for the absence subproof)"]:::result
  P2 --> R
  R --> SDK["verify_point_lookup_count_proof<br/>(absence_proofs_for_non_existing_searched_keys = false)"]:::verify
  SDK --> OUT["Entries([(brand_000, 1000)])<br/>brand_100 silently dropped"]:::sdk

  classDef request fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef dispatch fill:#21262d,color:#c9d1d9,stroke:#1f6feb;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb;
  classDef engine fill:#21262d,color:#c9d1d9,stroke:#39c5cf;
  classDef present fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef absent fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:3px,stroke-dasharray: 6 3;
  classDef result fill:#21262d,color:#c9d1d9,stroke:#39c5cf,stroke-width:2px;
  classDef verify fill:#21262d,color:#c9d1d9,stroke:#a371f7,stroke-width:2px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

Per-layer merk-tree structure (Layer 5+)

flowchart TB
  L5["L5 — widget subtree:<br/>KVValueHash(brand, Tree(brand_063))"]:::path
  L5 --> L6["L6 — byBrand value tree root:<br/>brand_063 (binary-search root)"]:::path
  L6 --> L7L["brand_031 (left subtree boundary)"]:::sibling
  L6 --> L7R["brand_095 (right subtree boundary)"]:::sibling
  L7L --> P000["brand_000<br/>(present, CountTree count=1000)"]:::target
  L7R --> A099["brand_099<br/>(rightmost present, absence-proof anchor)"]:::boundary
  L7R -.-> A100["brand_100 (not in tree — absence proven<br/>by Child chain to end-of-tree)"]:::absent

  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef boundary fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef absent fill:#21262d,color:#d29922,stroke:#d29922,stroke-width:2px,stroke-dasharray: 6 3;

Why absence-proof matters for count queries. The drive count fast path treats absent branches as 0, but it does NOT trust the SDK to apply that rule on un-committed data — every count or non-existence the verifier reports must be cryptographically committed by the prover. If absent branches were silently summed into 0 without a proof, a malicious prover could omit a present branch (with positive count) and claim it's absent, shrinking the result without detection. The 255-B absence-subproof overhead is the price of that integrity — small in absolute terms, but it scales linearly with the number of absent In values, so callers building queries with many speculative In values pay per-absence overhead.

G1b — High-fanout In on byBrand (|IN| = B), Grouped By brand

select   = COUNT
where    = brand IN ["brand_000", "brand_001", ..., "brand_099"]
group_by = [brand]
prove    = true

Path query (same shape as G1, scaled to |IN| = 100):

path:         ["@", contract_id, 0x01, "widget", "brand"]
query items:  [Key("brand_000"), Key("brand_001"), ..., Key("brand_099")]

Verified payload:

Entries(100 groups, sum = 100 000)

Every document in the fixture, partitioned by brand. Each Entries[i] carries (brand_NNN, CountTree count=1000).

Proof size: 10 038 B. Mode: CountMode::GroupByIn.

Same structural shape as G1, scaled from |IN| = 2 to |IN| = 100. The byBrand merk binary tree at L6 emits all 100 brands as KVValueHashFeatureTypeWithChildHash targets — each ~100 B (key + leaf kv-hash + CountTree(00, 1000, ...) + BasicMerkNode feature + child-hash) — plus minimal boundary glue at the binary-tree corners. The proof grows linearly with |IN|: G1 (|IN|=2) was 1 102 B; G1b (|IN|=100) is 10 038 B; the slope is ~99 B per additional In value.

Compare against the byColor equivalent (group_by_color_in_proof_100_rangecountable_branches, 10 512 B): the ProvableCountTree overhead from byColor's KVHashCount running counts adds ~5 % to the byBrand baseline, even though those running counts aren't consumed by a point-lookup group_by. This is the same ProvableCountTree overhead G2 carried at the smaller scale (|IN|=2).

Proof display:

Expand to see the structured proof (5 layers; bottom layer enumerates 100 brands as `KVValueHashFeatureTypeWithChildHash` targets — 192 merk ops total at L6 including binary-tree glue) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      // L2..L4 are byte-identical to every other query in this chapter
      // (the @ / contract_id / 0x01 descent into widget); see chapter 29's
      // Q1 verbatim for the full L1..L4 chain.
      ...
      widget => {
        LayerProof {
          proof: Merk(
            // L5 widget doctype — `brand` queried, opaque siblings 9862 / 6c36
            0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
            1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
            2: Parent
            3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
            4: Child)
          lower_layers: {
            brand => {
              LayerProof {
                proof: Merk(
                  // L6 byBrand merk-tree — 100 targets + binary-tree glue
                  // (192 merk ops total; structurally a fully-resolved in-order
                  // traversal of all 100 brand entries in the byBrand merk tree)
                  0: Push(KVValueHashFeatureTypeWithChildHash(brand_000, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[90ff6f6d9a3d901195982128130677243bfd27b75736206f3c8400966ef0d37b], BasicMerkNode, HASH[19b58883c492e746861db1e6ad07529a5a91cc8330af522682486db9346d6875]))
                  1: Push(KVValueHashFeatureTypeWithChildHash(brand_001, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[484ca11fb4ec8f479be1f78af903ce0c9d4fe630517579fb0172c2576d6b9652], BasicMerkNode, HASH[0bf12023f8e067c12db4cec1583909a0283878d6d909c76196736299750b5879]))
                  2: Parent
                  3: Push(KVValueHashFeatureTypeWithChildHash(brand_002, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[4c19f047068654e71813dce7839a579edfdcb446e3d70efa1b8592c73259da16], BasicMerkNode, HASH[e8d5372904b7f4ac9334aeb4ddab619d9ad7a308732a4f231416e10208a0a356]))
                  ...
                  // 97 more KVValueHashFeatureTypeWithChildHash targets following
                  // the same template — brand_003 ... brand_099 — interleaved with
                  // Parent/Child ops glueing them into the byBrand merk binary tree.
                  // Every target shares the structure:
                  //   Push(KVValueHashFeatureTypeWithChildHash(
                  //     brand_NNN,
                  //     CountTree(636f6c6f72, 1000, flags: [0, 0, 0]),   // count_value=1000
                  //     HASH[<per-brand leaf kv-hash>],
                  //     BasicMerkNode,                                  // NormalTree (no count on the merk node)
                  //     HASH[<per-brand subtree child hash>]
                  //   ))
                  ...
                  189: Push(KVValueHashFeatureTypeWithChildHash(brand_097, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[92adee932cc12927cd76ad9fd25906bbfe547df2bf21e826845bb4d3b47f5314], BasicMerkNode, HASH[34b69e1e424aa023c74f61554db2823da6c19dcbc51bdd5dece32e3f6f9fd219]))
                  190: Parent
                  191: Push(KVValueHashFeatureTypeWithChildHash(brand_098, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[68e02fcf66f86797035fbc8d53290185fe3fed7de897a8654743cae4007c47c3], BasicMerkNode, HASH[acfc3a88b852e8895449b4c7e01f4b1cc25028e6a80e4915cdde578ff6eb029b]))
                  192: Push(KVValueHashFeatureTypeWithChildHash(brand_099, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[af9667a8f2a10a9402b3d1fb0ac6e0b64d1e3dde5b8829c03b8d2c9cfc94e16d], BasicMerkNode, HASH[d049fe7e250b7dd763a4a5daa4227dcd2e41733dd95fd0758641ac06c63c3b51]))
                  // + closing Parent/Child ops binding the last few entries
                )
              }
            }
          }
        }
      }
    }
  }
}

The 254-line full verbatim sits in the bench's [gproof] G1b output — same template (one KVValueHashFeatureTypeWithChildHash per brand, all with CountTree count=1000 and BasicMerkNode feature) repeating 100 times. The schematic above shows the first 3 and last 3 targets so the structural pattern is clear without reproducing 100 near-identical lines.

Key observation: BasicMerkNode (not ProvableCountedMerkNode) is the feature type on each L6 op. byBrand is a NormalTree, so its merk binary tree's internal nodes don't carry running counts — only the per-brand CountTree count=1000 values stored inside each brand's element matter. Contrast this with G1b's byColor cousin (group_by_color_in_proof_100_rangecountable_branches, 10 512 B): there the L6 targets would carry ProvableCountedMerkNode(...) features because byColor IS a ProvableCountTree. The ~5 % size difference is exactly those count fields × 100 nodes.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree (100 entries)"]:::path
  BR ==> B000["brand_000: CountTree count=1000"]:::target
  BR ==> B001["brand_001: CountTree count=1000"]:::target
  BR ==> BMore["... 96 more in-range targets<br/>(brand_002 ... brand_097)"]:::target
  BR ==> B098["brand_098: CountTree count=1000"]:::target
  BR ==> B099["brand_099: CountTree count=1000"]:::target

  SDK["Entries(100 groups, sum=100 000):<br/>(&quot;brand_000&quot;, 1000),<br/>(&quot;brand_001&quot;, 1000),<br/>...<br/>(&quot;brand_099&quot;, 1000)"]:::sdk
  B000 -.-> SDK
  B099 -.-> SDK

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;
  linkStyle 3 stroke:#1f6feb,stroke-width:3px;
  linkStyle 4 stroke:#1f6feb,stroke-width:3px;
  linkStyle 5 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Identical to G1's L5–L6 shape, just with all 100 entries in the byBrand merk tree resolved as visible targets rather than just two. The byBrand binary tree has all 100 keys exposed — no opaque sibling subtrees (Hash ops) at all, only KVValueHashFeatureTypeWithChildHash (full reveal) plus Parent / Child glue.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree"]
    direction TB
    L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
    L5_left["HASH[9862...]"]:::sibling
    L5_right["HASH[6c36...]"]:::sibling
    L5_q --> L5_left
    L5_q --> L5_right
  end

  subgraph L6["Layer 6 — byBrand merk-tree (ALL 100 targets fully resolved)"]
    direction TB
    L6_t0["<b>brand_000</b><br/>CountTree count=1000<br/>BasicMerkNode"]:::target
    L6_t1["<b>brand_001</b><br/>CountTree count=1000"]:::target
    L6_tmid["... 97 more KVValueHashFeatureTypeWithChildHash<br/>targets, each CountTree count=1000<br/>(192 merk ops total: 100 Push + 92 Parent/Child)"]:::target
    L6_t99["<b>brand_099</b><br/>CountTree count=1000"]:::target

    L6_t0 --> L6_t1
    L6_t1 --> L6_tmid
    L6_tmid --> L6_t99
  end

  L5_q -. "Tree(merk_root[byBrand])" .-> L6_t0

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

Because the In set covers every brand in the fixture, the proof has zero opaque-sibling subtree commitments at L6 — every binary-tree node is revealed as a KVValueHashFeatureTypeWithChildHash target. That's the most efficient byte-per-key shape GroupByIn can hit: at |IN| = B (where B is the total entries in the property tree), the proof bytes ≈ B × (kv-hash + count + child-hash + glue) ≈ B × 100 B. For B = 100, that's exactly the 10 038 B we observe.

By contrast, smaller In sets (G1's |IN| = 2) pay the boundary-proof tax: the byBrand merk tree has ~98 unresolved entries, each contributing one KVHash (opaque-key commitment, ~33 B) or Hash (opaque-subtree commitment, ~33 B). The asymptotic crossover at which "reveal everything" becomes cheaper than "reveal-some-and-commit-the-rest" depends on the ratio of |IN| to B — for byBrand with B = 100, the crossover is around |IN| ≈ 50.

G2 — In on byColor, Grouped By color

select   = COUNT
where    = color IN ["color_00000000", "color_00000001"]
group_by = [color]
prove    = true

Path query (identical to Q6):

path:         ["@", contract_id, 0x01, "widget", "color"]
query items:  [Key("color_00000000"), Key("color_00000001")]

Verified payload:

Entries([
  ("color_00000000", CountTree { count_value_or_default: 100 }),
  ("color_00000001", CountTree { count_value_or_default: 100 }),
])

Proof size: 1 381 B. Byte-identical to Q6 — same path query, same ProvableCountTree-style boundary commitments (KVHashCount ops carry running counts even though the SDK doesn't read them for this point lookup). The single difference from G1 is the underlying property-name tree type (ProvableCountTree for byColor vs NormalTree for byBrand); that affects the merk-boundary commitments but not the dispatcher's GroupByIn-vs-Aggregate routing.

For the verbatim proof display, see Q6 in chapter 29 — or ▶ open it interactively in the visualizer ↗.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> CO["color: ProvableCountTree"]:::path
  CO ==> C000["color_00000000: CountTree count=100"]:::target
  CO ==> C001["color_00000001: CountTree count=100"]:::target
  CO -.-> CMore["color_00000002 ... color_00000999"]:::faded

  SDK["Verifier returns Entries([<br/>(&quot;color_00000000&quot;, 100),<br/>(&quot;color_00000001&quot;, 100)<br/>])"]:::sdk
  C000 -.-> SDK
  C001 -.-> SDK

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#d29922,color:#0d1117,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Identical to Q6's Layer-5+ diagram. The byColor ProvableCountTree at L6 carries the same KVHashCount running counts; the SDK ignores them for point-lookup group_by and reads only the two resolved targets' count_value_or_default.

G3 — Compound In + Equal, Grouped By brand

select   = COUNT
where    = brand IN ["brand_000", "brand_001"] AND color == "color_00000500"
group_by = [brand]
prove    = true

Path query (per-In compound resolution — outer Query on byBrand, inner subquery on byBrandColor's color terminator):

path:               ["@", contract_id, 0x01, "widget", "brand"]
query items:        [Key("brand_000"), Key("brand_001")]
subquery_path:      ["color"]
subquery items:     [Key("color_00000500")]

Verified payload:

Entries([
  ("brand_000", CountTree { count_value_or_default: 1 }),
  ("brand_001", CountTree { count_value_or_default: 1 }),
])

Each (brand, "color_00000500") pair has exactly 1 document in the bench's deterministic schedule.

Proof size: 2 842 B. Mode: CountMode::GroupByIn over the byBrandColor compound index.

Proof display:

Expand to see the structured proof (8 layers — two parallel brand-X → color → color_00000500 descents sharing L1–L6) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
                              1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
                              2: Parent
                              3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
                              4: Child)
                            lower_layers: {
                              brand => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(KVValueHash(brand_000, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[90ff6f6d9a3d901195982128130677243bfd27b75736206f3c8400966ef0d37b]))
                                    1: Push(KVValueHash(brand_001, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[484ca11fb4ec8f479be1f78af903ce0c9d4fe630517579fb0172c2576d6b9652]))
                                    2: Parent
                                    3: Push(Hash(HASH[8ca09dadc802a7efe03534ce4ad991b2f191f368878754a37b5e5c03d9498dab]))
                                    4: Child
                                    5: Push(KVHash(HASH[e5297b3ebe81c6435c29f712074da5f7c90265e12ed3d4f5af1f6d900e50c9f1]))
                                    6: Parent
                                    7: Push(Hash(HASH[50f373fd01dea89c992779764dff82cc7200b492be8f5cf3721627d5323bcbff]))
                                    8: Child
                                    9: Push(KVHash(HASH[cf78c9f1b1a1204bb2e437806f52c21e331392de3436388572bd1fa4bce1cdc7]))
                                    10: Parent
                                    11: Push(Hash(HASH[4a8dc186a95c8c4a1252fb51dbc407727f588eb5bdc8313c96f5c29889e13926]))
                                    12: Child
                                    13: Push(KVHash(HASH[d00ee7653e34e47d46004929b13ded33dff069ed9cc88342cecdf66a65fd8401]))
                                    14: Parent
                                    15: Push(Hash(HASH[7f1d17b9632f0bd440dacf5e841025482bc1d8145df3650301a95a5ee71ce8c8]))
                                    16: Child
                                    17: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
                                    18: Parent
                                    19: Push(Hash(HASH[eaef9fc530408393bc321409414814b290309a861f474a925a922250327affc6]))
                                    20: Child
                                    21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
                                    22: Parent
                                    23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
                                    24: Child)
                                  lower_layers: {
                                    brand_000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[d605b4b78e674fd77371ea6adb32ce3e58ee3b96d73c4d34df84159661634587]))
                                          1: Push(KVValueHash(color, NonCounted(ProvableCountTree(636f6c6f725f3030303030353131, 1000, flags: [0, 0, 0])), HASH[fccc0c94657f2a78084f789bb6f687c4bba295e3a062f3199bc33f14dd2b7fe2]))
                                          2: Parent)
                                        lower_layers: {
                                          color => {
                                            LayerProof {
                                              proof: Merk(
                                                ... 37 ops — same boundary shape as Q4 / Q8's L8,
                                                terminating at op 18 with
                                                Push(KVValueHashFeatureTypeWithChildHash(
                                                  color_00000500, CountTree(00, 1, ...),
                                                  HASH[6834...], ProvableCountedMerkNode(1),
                                                  HASH[840c...]))
                                                — TARGET 1
                                              )
                                            }
                                          }
                                        }
                                      }
                                    }
                                    brand_001 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[f54769bf6e9d24b9dba53ebd37c9ceb3485b3c6511f8de6f17860676fe4d9331]))
                                          1: Push(KVValueHash(color, NonCounted(ProvableCountTree(636f6c6f725f3030303030353131, 1000, flags: [0, 0, 0])), HASH[8f883171c33df0aba2541a5b9d6195faac7bd1ffef93e8ddcaf9d092f0fa5e19]))
                                          2: Parent)
                                        lower_layers: {
                                          color => {
                                            LayerProof {
                                              proof: Merk(
                                                ... 37 ops — same boundary shape as brand_000's
                                                color subtree, terminating at op 18 with
                                                Push(KVValueHashFeatureTypeWithChildHash(
                                                  color_00000500, CountTree(00, 1, ...),
                                                  HASH[881d...], ProvableCountedMerkNode(1),
                                                  HASH[a422...]))
                                                — TARGET 2
                                              )
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

The two parallel descents below brand are the structurally novel part — every other layer above brand is byte-identical to Q4. The byBrand layer (L6) inlines brand_000 and brand_001 as KVValueHash siblings (ops 0–2), then descends via the lower_layers map into each one's value-tree continuation. Each continuation (L7) carries a single color key whose value is NonCounted(ProvableCountTree(…)) — the byBrandColor terminator. The terminator (L8) walks the boundary path through its in-color binary merk tree to land at color_00000500 with CountTree count=1 and a feature-typed child hash.

The bulk of the proof bytes (≈ 2 × 1 100 B = 2 200 B) is the doubled L7+L8 descent. The L1–L6 prefix amortises across both branches (≈ 600 B shared), giving 2 842 B total — significantly less than 2× Q4's 1 911 B because the upper layers aren't repeated.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree"]:::path
  BR ==> B000["brand_000: CountTree count=1000"]:::path
  BR ==> B001["brand_001: CountTree count=1000"]:::path
  B000 ==> B000_C["color: NonCounted(ProvableCountTree)"]:::path
  B001 ==> B001_C["color: NonCounted(ProvableCountTree)"]:::path
  B000_C ==> T1["color_00000500: CountTree count=1"]:::target
  B001_C ==> T2["color_00000500: CountTree count=1"]:::target

  SDK["Verifier returns Entries([<br/>(&quot;brand_000&quot;, 1),<br/>(&quot;brand_001&quot;, 1)<br/>])"]:::sdk
  T1 -.-> SDK
  T2 -.-> SDK

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;
  linkStyle 3 stroke:#1f6feb,stroke-width:3px;
  linkStyle 4 stroke:#1f6feb,stroke-width:3px;
  linkStyle 5 stroke:#1f6feb,stroke-width:3px;
  linkStyle 6 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Layers 5–6 are like Q4's L5 + Q5's L6 combined (one KVValueHash per In brand at byBrand's binary tree); Layers 7–8 fork — one brand_000-rooted continuation chain and one brand_001-rooted chain — each shaped exactly like Q4's L7 + L8 descent.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree"]
    direction TB
    L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
    L5_left["HASH[9862...]"]:::sibling
    L5_right["HASH[6c36...]"]:::sibling
    L5_q --> L5_left
    L5_q --> L5_right
  end

  subgraph L6["Layer 6 — byBrand merk-tree (TWO INTERMEDIATE TARGETS)"]
    direction TB
    L6_t1["<b>brand_001</b><br/>kv_hash=HASH[484c...]<br/>value: CountTree count=1000"]:::queried
    L6_t0["<b>brand_000</b><br/>kv_hash=HASH[90ff...]<br/>value: CountTree count=1000"]:::queried
    L6_boundary["Boundary commitments (22 merk ops):<br/>7 KVHash sibling brands + 7 Hash subtrees"]:::sibling
    L6_t1 --> L6_t0
    L6_t1 --> L6_boundary
  end

  subgraph L7a["Layer 7a — brand_000's continuation merk-tree"]
    direction TB
    L7a_q["<b>color</b><br/>kv_hash=HASH[fccc...]<br/>value: NonCounted(ProvableCountTree)"]:::queried
    L7a_left["HASH[d605...]"]:::sibling
    L7a_q --> L7a_left
  end

  subgraph L7b["Layer 7b — brand_001's continuation merk-tree"]
    direction TB
    L7b_q["<b>color</b><br/>kv_hash=HASH[8f88...]<br/>value: NonCounted(ProvableCountTree)"]:::queried
    L7b_left["HASH[f547...]"]:::sibling
    L7b_q --> L7b_left
  end

  subgraph L8a["Layer 8a — brand_000's byBrandColor color subtree (TARGET 1)"]
    direction TB
    L8a_target["<b>color_00000500</b><br/>kv_hash=HASH[6834...]<br/>value: <b>CountTree count=1</b><br/>feature: ProvableCountedMerkNode(1)"]:::target
    L8a_boundary["37 merk ops:<br/>9 KVHashCount boundary commitments<br/>(running counts 3, 7, 15, 31, 63, 127, 255, 511, 1000)<br/>+ subtree hashes"]:::sibling
    L8a_target --> L8a_boundary
  end

  subgraph L8b["Layer 8b — brand_001's byBrandColor color subtree (TARGET 2)"]
    direction TB
    L8b_target["<b>color_00000500</b><br/>kv_hash=HASH[881d...]<br/>value: <b>CountTree count=1</b><br/>feature: ProvableCountedMerkNode(1)"]:::target
    L8b_boundary["37 merk ops:<br/>same boundary shape as L8a<br/>(different hashes — different brand's subtree)"]:::sibling
    L8b_target --> L8b_boundary
  end

  L5_q -. "Tree(merk_root[byBrand])" .-> L6_t1
  L6_t0 -. "CountTree continuation" .-> L7a_q
  L6_t1 -. "CountTree continuation" .-> L7b_q
  L7a_q -. "NonCounted(ProvableCountTree)" .-> L8a_target
  L7b_q -. "NonCounted(ProvableCountTree)" .-> L8b_target

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The two parallel byBrandColor descents share their L1–L6 commitments (the doctype prefix + byBrand merk root) but each gets its own L7 + L8 sub-proof. Proof bytes ≈ shared upper layers + 2 × per-brand byBrandColor descent ≈ 2 842 B.

G4 — Range on byColor, Grouped By color

GroupByRange is the proof primitive that enumerates distinct in-range keys with a count per key, as opposed to chapter 29's AggregateCountOnRange which collapses the same range to a single u64.

select   = COUNT
where    = color > "color_00000500"
group_by = [color]
prove    = true

Path query (uses distinct_count_path_query with limit=100, left_to_right=true):

path:         ["@", contract_id, 0x01, "widget", "color"]
query items:  [RangeAfter("color_00000500"..)]
limit:        100

Verified payload:

Entries(100 groups, sum = 10 000)

The 100 groups are color_00000501 through color_00000600 (the first 100 in-range colors in lex-asc order, capped by the limit). Each carries count_value_or_default = 100 since the fixture's deterministic schedule gives each color exactly 100 documents.

Wait — but Q7 said there are 499 distinct in-range colors and sum = 49 900 over the same color > "color_00000500" predicate. So why does G4 see only 100 groups summing to 10 000? Because GroupByRange's distinct_count_path_query applies the 100-entry response cap (Some(limit) in execute_distinct_count_with_proof). Without that cap the proof would scale linearly with the full in-range distinct count (~5.5 KB for the full 499 colors at ~110 B per resolved CountTree branch). The cap is a response-size safety control — the verifier ceases the walk once it has 100 entries.

Proof size: 10 992 B — ~5.3 × Q7. The structural reason:

  • Q7 (AggregateCountOnRange) walks the boundary of the range and emits one HashWithCount or KVDigestCount per merk-binary-tree boundary node. Total boundary nodes ≈ O(log C) (≈ 36 ops on the 1 000-color tree). The verifier sums subtree counts directly without descending into individual keys.
  • G4 (GroupByRange) walks the distinct in-range colors themselves — emitting one KVValueHashFeatureTypeWithChildHash(color_X, CountTree count=100, ProvableCountedMerkNode(…), …) per distinct color in the range, not just per merk-tree boundary node. Total ops ≈ O(R) where R is the distinct in-range colors (capped at 100 here).

The trade-off is exactly what you'd expect: AggregateCountOnRange is O(log C) in proof bytes but loses per-key resolution (returns one u64); GroupByRange is O(R) in proof bytes but preserves per-key counts.

Proof display:

Expand to see the structured proof (5 layers; bottom layer enumerates 100 distinct in-range colors as `KVValueHashFeatureTypeWithChildHash` targets, each carrying `CountTree count=100`) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
                  1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
                      lower_layers: {
                        widget => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
                              1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
                              2: Parent
                              3: Push(KVValueHash(color, ProvableCountTree(636f6c6f725f3030303030353131, 100000), HASH[79569d595db75bbf2e9dca93a15c90b7eecf7b299632668ec410e2076d27f71c]))
                              4: Child)
                            lower_layers: {
                              color => {
                                LayerProof {
                                  proof: Merk(
                                    ... 18 boundary-descent ops walking the binary tree from
                                    root (color_00000511) leftward to the cut point ...
                                    18: Push(KVDigestCount(color_00000500, HASH[47b0ade5...], 100))
                                       // op 18: BOUNDARY (excluded by strict `>`)
                                    19: Push(KVValueHashFeatureTypeWithChildHash(color_00000501,
                                       CountTree(00, 100, flags: [0, 0, 0]),
                                       HASH[9146433eb6d43db2f109f5f7714146624bd646b27c7310f3c2cad7155eb7c741],
                                       ProvableCountedMerkNode(300),
                                       HASH[c285efb8724a488de916ce8301b06c197fc687b5b9b83a04bf3a026f1098d17a]))
                                       // op 19: TARGET 1
                                    20: Parent
                                    21: Push(KVValueHashFeatureTypeWithChildHash(color_00000502, CountTree(00, 100, ...)))
                                       // op 21: TARGET 2
                                    ... 98 more KVValueHashFeatureTypeWithChildHash targets
                                    (color_00000503 ... color_00000600), each emitting
                                    `CountTree count=100` plus its merk feature/child-hash glue,
                                    interleaved with Parent/Child ops walking the binary tree
                                    in lex-asc order. Every target shares the same shape:
                                    Push(KVValueHashFeatureTypeWithChildHash(
                                      color_XXXXXXXX,
                                      CountTree(00, 100, flags: [0, 0, 0]),
                                      HASH[...],
                                      ProvableCountedMerkNode(running_count_at_this_node),
                                      HASH[...]
                                    )) ...
                                    220: Push(KVValueHashFeatureTypeWithChildHash(color_00000600,
                                       CountTree(00, 100, ...))) // op 220: TARGET 100 (LAST)
                                    221..244: closing boundary ops — KVHashCount running
                                    counts (300, 700, 6300, 25500, 48800) and Hash subtrees
                                    proving the still-out-of-range portion to the right of
                                    color_00000600 covers the remainder of the merk root.)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

That schematic gives the shape; the bench's [gproof] output (run cargo bench --bench document_count_worst_case and grep [gproof] G4) has all 245 ops verbatim. The compression in the chapter just elides the 100 KVValueHashFeatureTypeWithChildHash targets since they share the same structural template — only the key name, the leaf kv-hash, the running count, and the child-hash differ.

Why so many targets? Because GroupByRange must enumerate every in-range key with its CountTree value — the SDK needs each individual key→count pair, which the aggregate-style HashWithCount commitment hides. So the prover walks the merk binary tree's in-order traversal across the in-range portion (here, left-to-right starting just past color_00000500) and emits one KVValueHashFeatureTypeWithChildHash per distinct color it visits, until the response-size limit is reached.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> CO["color: ProvableCountTree count=100000"]:::path
  CO -.-> C500["color_00000500 (boundary, excluded)"]:::faded
  CO ==> C501["color_00000501: CountTree count=100"]:::target
  CO ==> CMore["color_00000502 ... color_00000600<br/>(98 more in-range targets,<br/>each CountTree count=100)"]:::target
  CO ==> C600["color_00000600: CountTree count=100"]:::target
  CO -.-> CRest["color_00000601 ... color_00000999<br/>(beyond limit — opaque)"]:::faded

  SDK["Verifier returns Entries(100 groups):<br/>(&quot;color_00000501&quot;, 100),<br/>(&quot;color_00000502&quot;, 100),<br/>... (&quot;color_00000600&quot;, 100)"]:::sdk
  C501 -.-> SDK
  CMore -.-> SDK
  C600 -.-> SDK

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#d29922,color:#0d1117,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;
  linkStyle 3 stroke:#1f6feb,stroke-width:3px;
  linkStyle 4 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

L5 is identical to Q3's / Q6's L5 (color queried under an opaque kv root in the widget doctype tree). L6 is the structural novelty: 245 merk ops, of which 100 are full KVValueHashFeatureTypeWithChildHash targets and the remaining 145 are boundary-walk glue (KVDigestCount / KVHashCount / HashWithCount / Hash + Parent/Child).

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `color`)"]
    direction TB
    L5_root["KVHash[a29e...]<br/>(opaque kv root)"]:::sibling
    L5_left["HASH[9862...]"]:::sibling
    L5_q["<b>color</b><br/>kv_hash=HASH[7956...]<br/>value: ProvableCountTree count=100000"]:::queried
    L5_root --> L5_left
    L5_root --> L5_q
  end

  subgraph L6["Layer 6 — byColor ProvableCountTree merk-tree (100 in-range targets)"]
    direction TB
    L6_boundary_l["Left boundary descent (18 ops):<br/>walks from merk root color_00000511<br/>through KVHashCount running counts<br/>(51100, 25500, 12700, 6300, 3100, 700)<br/>down to color_00000500"]:::sibling
    L6_cut["op 18: KVDigestCount(color_00000500, ..., 100)<br/>(boundary — excluded by strict `>`)"]:::boundary
    L6_targets["ops 19..220: 100 in-range targets<br/>color_00000501 (count=100), color_00000502 (100),<br/>color_00000503 (100), ... color_00000600 (100)<br/>each as KVValueHashFeatureTypeWithChildHash<br/>with ProvableCountedMerkNode(subtree_count)<br/>interleaved with Parent/Child glue"]:::target
    L6_boundary_r["Right closing boundary (24 ops):<br/>KVHashCount running counts<br/>(300, 700, 6300, 25500, 48800)<br/>+ Hash subtree commitments<br/>covering color_00000601 ... color_00000999"]:::sibling

    L6_boundary_l --> L6_cut
    L6_cut --> L6_targets
    L6_targets --> L6_boundary_r
  end

  L5_q -. "ProvableCountTree(merk_root[byColor])" .-> L6_boundary_l

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef boundary fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px,stroke-dasharray: 6 3;

Three things this diagram makes explicit:

  1. The cut is named. op 18: KVDigestCount(color_00000500, ..., 100) exposes the key at the boundary so the verifier knows the cut sits exactly between color_00000500 (excluded) and color_00000501 (first in-range). Without that named op, a malicious prover could shift the cut and the verifier wouldn't know.
  2. Targets carry their own count, not a running total. Unlike Q7's boundary commitments (where ProvableCountedMerkNode(N) carried a subtree count), G4's targets are individual keys with CountTree(00, 100, ...) — the count_value_or_default = 100 IS the per-key count, not a subtree aggregate. The ProvableCountedMerkNode(N) on the merk feature still carries the subtree count (e.g. 300 for color_00000501's subtree), but G4's verifier reads count_value_or_default directly from the CountTree element.
  3. The right closing boundary doesn't enumerate the rest. Once the limit is hit at color_00000600, the proof commits the remaining ~399 in-range colors as opaque subtree hashes (KVHashCount + Hash ops). The SDK returns only the 100 visible groups; the remainder are provably present but not enumerated. This is the limit's whole point — bound response size without sacrificing soundness on the visible groups.

G5 — Compound In + Range, Grouped By brand, color

select   = COUNT
where    = brand IN ["brand_000", "brand_001"] AND color > "color_00000500"
group_by = [brand, color]
prove    = true

Path query (outer In on byBrand fans out to per-brand distinct_count_path_query on byBrandColor's color terminator):

outer path:         ["@", contract_id, 0x01, "widget", "brand"]
outer query items:  [Key("brand_000"), Key("brand_001")]
subquery_path:      ["color"]
subquery items:     [RangeAfter("color_00000500"..)]
subquery limit:     100 (shared across both brands)

Verified payload:

Entries(100 groups, sum = 100)

Two brands × 50 in-range colors per brand = 100 distinct (brand, color) groups visible in the proof. Each (brand_X, color_Y) pair has exactly 1 document by the fixture's deterministic schedule.

Proof size: 11 554 B. Mode: CountMode::GroupByCompound.

This is the most general group-by shape supported on this contract: outer In fan-out × inner GroupByRange walk. Structurally it combines G3's two-branch descent with G4's in-range enumeration per branch. Proof bytes ≈ shared upper-layer descent + 2 × per-brand byBrandColor distinct-walk. The bench's group_by_compound_in_range_proof_limit_100 benchmark uses the same shape with |IN| = 100 brands instead of 2 — yielding 17 256 B at the much higher fan-out.

Proof display:

Expand to see the structured proof (8 layers — same descent skeleton as G3, but each brand's L8 enumerates 50 in-range colors instead of one point-lookup target) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => { LayerProof { ... contract_id descent ... } }
      // L2..L4 identical to G3 / Q4's first three subgroves
    }
  }
  // L5 widget doctype merk tree: same as G3 — `brand` queried, opaque siblings 9862 / 6c36
  // L6 byBrand merk tree: two KVValueHash targets (brand_000 + brand_001), 25 boundary ops
  // L7a brand_000's value tree: single key `color` with NonCounted(ProvableCountTree(...))
  //   L8a byBrandColor's color subtree (under brand_000):
  //     proof: Merk(
  //       ... 18 boundary-descent ops walking from the merk root down to color_00000500 ...
  //       18: Push(KVDigestCount(color_00000500, HASH[...], 1))     // BOUNDARY, excluded
  //       19: Push(KVValueHashFeatureTypeWithChildHash(color_00000501,
  //              CountTree(00, 1, flags: [0, 0, 0]),
  //              HASH[4192...], ProvableCountedMerkNode(3), HASH[c3b4...])) // TARGET (brand_000, color_00000501)
  //       21: Push(KVValueHashFeatureTypeWithChildHash(color_00000502, CountTree(00, 1, ...))) // TARGET 2
  //       24: Push(KVValueHashFeatureTypeWithChildHash(color_00000503, CountTree(00, 1, ...))) // TARGET 3
  //       ... 47 more KVValueHashFeatureTypeWithChildHash targets, each CountTree(00, 1, ...)
  //           — color_00000504 ... color_00000550 (50 per-brand_000 targets total) ...
  //       ... closing boundary ops covering color_00000551 ... color_00000999 for brand_000
  //     )
  //   end L8a
  // end L7a
  // L7b brand_001's value tree: identical structure to L7a, single key `color`
  //   L8b byBrandColor's color subtree (under brand_001):
  //     proof: Merk(
  //       ... 18 boundary-descent ops (different hashes — different brand's subtree) ...
  //       18: Push(KVDigestCount(color_00000500, HASH[...], 1))
  //       19..220: 50 in-range KVValueHashFeatureTypeWithChildHash(color_X, CountTree(00, 1, ...)) targets
  //                + interleaved Parent/Child glue + closing boundary ops
  //     )
  //   end L8b
  // end L7b
  // end L6
}

The 344-line verbatim is available via the bench's [gproof] G5 output. The schematic compresses the 50 per-brand KVValueHashFeatureTypeWithChildHash targets at L8a / L8b — they all share the same template (CountTree(00, 1, ...) since each (brand, color) pair has count=1), differing only in key, leaf kv-hash, running count, and child-hash. Once you've seen G3's L8 structure (single target) and G4's L6 structure (100 in-range targets at the doctype level), G5 is precisely the product: two parallel G3-shaped descents that each terminate in a G4-shaped distinct-walk.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree"]:::path
  BR ==> B000["brand_000: CountTree count=1000"]:::path
  BR ==> B001["brand_001: CountTree count=1000"]:::path

  B000 ==> B000_C["brand_000/color: NonCounted(ProvableCountTree)"]:::path
  B001 ==> B001_C["brand_001/color: NonCounted(ProvableCountTree)"]:::path

  B000_C ==> T000_501["color_00000501: CountTree count=1"]:::target
  B000_C ==> T000_more["... 48 more color targets<br/>(brand_000, color_00000502..550)"]:::target
  B000_C ==> T000_550["color_00000550: CountTree count=1"]:::target

  B001_C ==> T001_501["color_00000501: CountTree count=1"]:::target
  B001_C ==> T001_more["... 48 more color targets<br/>(brand_001, color_00000502..550)"]:::target
  B001_C ==> T001_550["color_00000550: CountTree count=1"]:::target

  SDK["Entries(100 groups, sum=100):<br/>(&quot;brand_000&quot;, &quot;color_00000501&quot;, 1),<br/>...<br/>(&quot;brand_001&quot;, &quot;color_00000550&quot;, 1)"]:::sdk

  T000_501 -.-> SDK
  T000_more -.-> SDK
  T000_550 -.-> SDK
  T001_501 -.-> SDK
  T001_more -.-> SDK
  T001_550 -.-> SDK

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;
  linkStyle 3 stroke:#1f6feb,stroke-width:3px;
  linkStyle 4 stroke:#1f6feb,stroke-width:3px;
  linkStyle 5 stroke:#1f6feb,stroke-width:3px;
  linkStyle 6 stroke:#1f6feb,stroke-width:3px;
  linkStyle 7 stroke:#1f6feb,stroke-width:3px;
  linkStyle 8 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

Layers 5–7 are exactly G3's L5–L7. The difference shows up at L8 — instead of a single target per brand (G3's compound point lookup), each brand's L8 walks 50 in-range colors via the same KVValueHashFeatureTypeWithChildHash enumeration G4 uses, plus the boundary descent / closing boundary glue.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree"]
    direction TB
    L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
  end

  subgraph L6["Layer 6 — byBrand merk-tree (two intermediate targets)"]
    direction TB
    L6_t0["<b>brand_000</b> (queried)<br/>CountTree count=1000"]:::queried
    L6_t1["<b>brand_001</b> (queried)<br/>CountTree count=1000"]:::queried
  end

  subgraph L7a["Layer 7a — brand_000's continuation"]
    direction TB
    L7a_q["<b>color</b> (queried)<br/>NonCounted(ProvableCountTree)"]:::queried
  end
  subgraph L7b["Layer 7b — brand_001's continuation"]
    direction TB
    L7b_q["<b>color</b> (queried)<br/>NonCounted(ProvableCountTree)"]:::queried
  end

  subgraph L8a["Layer 8a — brand_000's byBrandColor distinct-walk"]
    direction TB
    L8a_targets["50 KVValueHashFeatureTypeWithChildHash targets:<br/>color_00000501 ... color_00000550<br/>each CountTree(00, 1, ...)<br/>+ left/right boundary glue"]:::target
  end
  subgraph L8b["Layer 8b — brand_001's byBrandColor distinct-walk"]
    direction TB
    L8b_targets["50 KVValueHashFeatureTypeWithChildHash targets:<br/>color_00000501 ... color_00000550<br/>each CountTree(00, 1, ...)<br/>+ left/right boundary glue<br/>(different hashes — different brand subtree)"]:::target
  end

  L5_q -. "byBrand" .-> L6_t0
  L5_q -. "byBrand" .-> L6_t1
  L6_t0 -. "continuation" .-> L7a_q
  L6_t1 -. "continuation" .-> L7b_q
  L7a_q -. "byBrandColor distinct-range" .-> L8a_targets
  L7b_q -. "byBrandColor distinct-range" .-> L8b_targets

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The 50-targets-per-brand limit reflects the shared response-size cap. In the 2-brand case the cap kicks in at 50 colors per brand; if the In set had 1 brand it would be 100 colors; if it had 4 brands it would be 25 each. The dispatcher slices the cap evenly across the In fan-out so the total number of returned entries equals the limit, regardless of how many In branches share it. That's why the bench's [matrix] row for this case shows Entries(len=100, sum=100) rather than len=200, sum=200.

G7 — Carrier In + Range, Grouped By brand

select   = COUNT
where    = brand IN ["brand_000", "brand_001"] AND color > "color_00000500"
group_by = [brand]
prove    = true

Path query (carrier AggregateCountOnRange — outer Keys per In value, ACOR subquery over each brand's color subtree):

path:                  ["@", contract_id, 0x01, "widget", "brand"]
outer query items:     [Key("brand_000"), Key("brand_001")]
subquery_path:         ["color"]
subquery items:        [AggregateCountOnRange([RangeAfter("color_00000500"..)])]

Verified payload (verifier returns one (in_key, u64) per resolved In branch via GroveDb::verify_aggregate_count_query_per_key):

[("brand_000", 499), ("brand_001", 499)]

Each brand has all 1 000 colors in its byBrandColor terminator; the strict > cut at color_00000500 leaves color_00000501..color_00000999 = 499 in-range colors per brand. Total sum = 998 documents.

Proof size: 4 332 B. Mode: CountMode::GroupByIn routed to DocumentCountMode::RangeAggregateCarrierProof (the new dispatcher arm wired up against grovedb PR #663).

This is the natural answer to "give me a per-brand aggregate count over a colour range", verifiable in a single proof. Without a proof the same request runs the per-In fan-out and folds the brands into one total entry; only the proved answer keeps one entry per brand. Strictly smaller and asymptotically better than the alternative two-field shape G5:

  • G5 (compound distinct walk, group_by = [brand, color]): O(k · R' · log C') bytes; emits one KVValueHashFeatureTypeWithChildHash per resolved (brand, color) pair → 11 554 B for k=2, R'≈50. Carries per-pair granularity the caller may not want.
  • G7 (carrier aggregate, group_by = [brand]): O(k · (log B + log C')) bytes; emits one HashWithCount/KVDigestCount ACOR boundary walk per brand → 4 332 B for k=2, log C'≈10. ~2.7× smaller than G5 for the same input data, at the cost of losing per-color resolution (which the group_by = [brand] caller didn't ask for anyway).

The win vs Q8 (brand == X AND color > floor, the same shape with k=1 and group_by = []) is asymptotic: Q8 is 2 656 B, G7 is 4 332 B for k=2. The slope (G7 − Q8) / 1 = +1 676 B per additional In branch matches what you'd expect: each brand adds its own L6 commit + its own L7 + L8 ACOR boundary walk (≈ Q8's L7 + L8 ≈ ~1 700 B), with the L1–L5 prefix amortising once across all branches.

Proof display:

Expand to see the structured proof (8 layers — same skeleton as G5, but each brand's L8 is an ACOR boundary walk instead of a 50-target distinct-walk) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(... root-level descent, identical to every other chapter query ...)
    lower_layers: {
      @ => { ... contract_id descent ... }
      // L2..L4 byte-identical to G3 / G5 (the @/contract_id/0x01/widget chain)
    }
  }
  // L5 widget doctype: brand queried (same as G3 / G5 — opaque siblings 9862 / 6c36)
  // L6 byBrand merk-tree: two KVValueHash targets (brand_000 + brand_001), 25 ops
  //                       — same shape as G5's L6
  // L7a brand_000's value tree: single key `color` with NonCounted(ProvableCountTree)
  //   L8a byBrandColor color subtree under brand_000:
  //     proof: Merk(
  //       ... 36-37 ACOR boundary ops over color > color_00000500 ...
  //       18: Push(KVDigestCount(color_00000500, ..., 1))          // BOUNDARY (excluded)
  //       19..35: HashWithCount / KVDigestCount boundary walk
  //                 — same shape as Q8's L8, summing to count=499 for brand_000)
  //   end L8a
  // end L7a
  // L7b brand_001's value tree: same single-key shape, different hashes
  //   L8b byBrandColor color subtree under brand_001:
  //     proof: Merk(
  //       ... 36-37 ACOR boundary ops over color > color_00000500 ...
  //                 — same shape, different hashes, summing to count=499 for brand_001)
  //   end L8b
  // end L7b
}

The 186-line full verbatim is available via the bench's [gproof] G7 output. The schematic compresses the L1–L4 doctype prefix (byte-identical to every other 8-layer chapter query) and the two parallel L7+L8 descents (structurally identical to Q8's, with different hashes for each brand). Each brand's L8 contributes ~1 700 B of ACOR boundary commitments — exactly the predicted Q8 - L1..L5 overhead per branch.

Cryptographic guarantee (via grovedb PR #663): every per-brand count is independently committed to the merk root via node_hash_with_count. A malicious prover can't lie about brand_000's count without breaking brand_001's verification (and vice versa) because each carrier ACOR subquery has its own hash chain back to the merk root.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree"]:::path
  BR ==> B000["brand_000: CountTree count=1000"]:::path
  BR ==> B001["brand_001: CountTree count=1000"]:::path
  B000 ==> B000_C["brand_000/color: NonCounted(ProvableCountTree)<br/>ACOR boundary walk (color > color_00000500)"]:::target
  B001 ==> B001_C["brand_001/color: NonCounted(ProvableCountTree)<br/>ACOR boundary walk (color > color_00000500)"]:::target

  SDK["Entries(2 groups, sum=998):<br/>(&quot;brand_000&quot;, 499)<br/>(&quot;brand_001&quot;, 499)"]:::sdk
  B000_C -.-> SDK
  B001_C -.-> SDK

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;
  linkStyle 3 stroke:#1f6feb,stroke-width:3px;
  linkStyle 4 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

L5–L7 are exactly G5's L5–L7 (widget → byBrand → brand_X's continuation). The difference is at L8: G5 enumerates 50 distinct (brand_X, color_Y) pairs as KVValueHashFeatureTypeWithChildHash targets per brand; G7 walks the same color subtree as an ACOR boundary cut (like Q8's L8), emitting HashWithCount / KVDigestCount ops that commit a single aggregate u64 per brand.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree"]
    direction TB
    L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
  end

  subgraph L6["Layer 6 — byBrand merk-tree (two intermediate targets)"]
    direction TB
    L6_t0["<b>brand_000</b> (queried)<br/>CountTree count=1000"]:::queried
    L6_t1["<b>brand_001</b> (queried)<br/>CountTree count=1000"]:::queried
  end

  subgraph L7a["Layer 7a — brand_000's continuation"]
    direction TB
    L7a_q["<b>color</b> (queried)<br/>NonCounted(ProvableCountTree)"]:::queried
  end
  subgraph L7b["Layer 7b — brand_001's continuation"]
    direction TB
    L7b_q["<b>color</b> (queried)<br/>NonCounted(ProvableCountTree)"]:::queried
  end

  subgraph L8a["Layer 8a — brand_000's byBrandColor: ACOR cut"]
    direction TB
    L8a_target["<b>Aggregate count = 499</b><br/>(committed via node_hash_with_count)"]:::target
    L8a_ops["~37 merk ops:<br/>KVDigestCount(color_00000500, …) — boundary excluded<br/>+ HashWithCount/KVDigestCount boundary walk<br/>over the in-range portion"]:::sibling
    L8a_target --> L8a_ops
  end
  subgraph L8b["Layer 8b — brand_001's byBrandColor: ACOR cut"]
    direction TB
    L8b_target["<b>Aggregate count = 499</b><br/>(committed via node_hash_with_count)"]:::target
    L8b_ops["~37 merk ops:<br/>same boundary shape as L8a<br/>(different hashes — different brand subtree)"]:::sibling
    L8b_target --> L8b_ops
  end

  L5_q -. "byBrand" .-> L6_t0
  L5_q -. "byBrand" .-> L6_t1
  L6_t0 -. "continuation" .-> L7a_q
  L6_t1 -. "continuation" .-> L7b_q
  L7a_q -. "carrier ACOR subquery" .-> L8a_target
  L7b_q -. "carrier ACOR subquery" .-> L8b_target

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The "carrier" name comes from grovedb's PR #663 terminology: a carrier query is the outer multi-key query that carries an ACOR subquery into each branch. The ACOR primitive itself is unchanged — it still walks one range over one subtree per invocation — but it can now appear as a subquery item under outer Keys, which is what enables the per-brand aggregate proof shape G7 needs.

G8 — Carrier outer Range + Range, Grouped By brand

select   = COUNT
where    = brand > "brand_050" AND color > "color_00000500"
group_by = [brand]
limit    = (optional; ≤ 10)
prove    = true

The platform's MAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT = 10 is both the default (when the caller passes no limit) and a hard ceiling. Callers may pass a smaller limit (1 through 9) to truncate the outer walk further; passing 0 or any value > 10 is rejected with InvalidLimit. See the rationale below.

Path query (the same carrier-ACOR shape as G7, but with a range outer dimension and SizedQuery::limit bounded by the platform max):

path:                  ["@", contract_id, 0x01, "widget", "brand"]
outer query item:      RangeAfter("brand_050"..)
subquery_path:         ["color"]
subquery items:        [AggregateCountOnRange([RangeAfter("color_00000500"..)])]
SizedQuery::limit:     10  (platform default; caller may request smaller)

Verified payload (verifier returns one (in_key, u64) per in-range outer key, capped at limit, via GroveDb::verify_aggregate_count_query_per_key):

[("brand_051", 499), ("brand_052", 499), …, ("brand_060", 499)]

The bench's 100-brand fixture has 49 brands > "brand_050". The platform's default SizedQuery::limit = 10 caps the carrier at the first 10 (brand_051 … brand_060); each carries the per-brand ACOR count of 499 in-range colors (color_00000501 … color_00000999). Total sum = 10 × 499 = 4 990 documents.

Proof size: 18 022 B. Mode: CountMode::GroupByRange routed to DocumentCountMode::RangeAggregateCarrierProof (the dispatcher distinguishes G7's In-outer shape from G8's Range-outer shape by the carrier clause's operator).

G8 is G7's natural extension from "k specific outer keys" to "L outer keys from an in-range walk." Same carrier proof primitive, same node_hash_with_count commitments per branch, same one-u64-per-branch return shape. The structural differences are exactly two:

  • Outer dimension: G7 emits k Key(serialized_in_value) items in the carrier query; G8 emits a single RangeAfter(serialized_floor..) (or any Range* variant) and lets grovedb walk it.
  • Limit: G8 sets SizedQuery::limit = Some(L) where L is the smaller of the caller's request and the platform max. Per grovedb PR #664, this is the load-bearing relaxation — the predecessor PR #663 allowed Range outer items at the validator level but kept the leaf-ACOR rule rejecting SizedQuery::limit, which made unbounded range-outer carriers impractical at any reasonable dataset size (49 brands × ~1 700 B each ≈ 83 KB; with the platform default of 10 we land at 18 KB).

Why the cap exists and where the ceiling lives

The cap bounds the prove-path proof size; the ceiling is a hardcoded compile-time constant for prover/verifier-agreement reasons.

  1. Proof-size bounding. Proof bytes scale linearly with the limit (~1 700 B per outer match, exactly as for G7). 10 keeps the worst-case proof under 20 KB (Tier-1 for the GroveDB Proof Visualizer's shareable-link guidance — Tier-1 ≤ 20 KB works in every browser and link-preview surface; Tier-2 of 20–50 KB works in browsers but may be truncated in Slack/Discord previews; Tier-3 above 50 KB risks Safari's URL ceiling) — enough for typical "top-N brands by an outer range" queries while avoiding pathological proof sizes. Callers that want a window above 10 entries call repeatedly with disjoint outer-range bounds; callers that want fewer pass a smaller limit (1 through 9). Limit 0 is rejected to keep the response shape non-trivial.
  2. Prover/verifier byte-for-byte agreement. SizedQuery::limit is part of the serialized PathQuery and feeds the merk-root reconstruction; both prover and verifier must agree on its value. The caller's request carries limit over the wire, so its specific value (1..=10) is fine to vary. What can't vary is the platform's default when the caller passes nothing — that's why the ceiling is a hardcoded compile-time constant (MAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT) rather than an operator-tunable runtime value. Same rationale as RangeDistinctProof's use of crate::config::DEFAULT_QUERY_LIMIT rather than drive_config.default_query_limit.

Caller semantics summary:

Caller request.limitServer usesReason
None10 (the platform default)Default = ceiling
Some(1..=10)the caller's valueTruncates the walk further
Some(0)rejectedNon-trivial response required
Some(11+)rejectedAbove the ceiling

Complexity: O(L · (log B + log C')) where L = min(caller_limit, MAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT) — L outer-key descents in the byBrand layer + L leaf-ACOR boundary walks in each brand's color subtree. Independent of how many keys the outer range could have walked without the cap.

Proof display:

Expand to see the structured proof (8 layers — same skeleton as G7, but L8 contains 10 per-brand ACOR boundary walks instead of 2) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(... root-level descent, identical to every other chapter query ...)
    lower_layers: {
      @ => { ... contract_id descent ... }
      // L2..L4 byte-identical to G3 / G5 / G7 (the @/contract_id/0x01/widget chain)
    }
  }
  // L5 widget doctype: brand queried (same as G3 / G5 / G7)
  // L6 byBrand merk-tree: 10 outer-key matches inlined as KVValueHash items
  //                       (brand_051 ... brand_060), each descending into its
  //                       continuation. Boundary commitments cover the
  //                       brands_outside_the_limited_window.
  // L7 brand_NNN's value tree: single key `color` with NonCounted(ProvableCountTree)
  //    — repeated 10 times, once per resolved outer brand
  // L8 brand_NNN's byBrandColor color subtree:
  //    proof: Merk(
  //      ... 36-37 ACOR boundary ops over color > color_00000500,
  //          summing to count = 499 per brand ...
  //    )
  //    — repeated 10 times in parallel, each with its own per-brand boundary hashes
}

The 618-line full verbatim is available via the bench's [gproof] G8 output. The schematic compresses the 10 parallel L7+L8 descents — they share the same template (single-key continuation + 37-op ACOR boundary walk), differing only in per-brand kv-hashes and the resulting subtree commits. Each per-brand L8 contributes ~1 700 B of ACOR boundary commitments — exactly the predicted Q8 - L1..L5 overhead per outer match, scaling linearly: 18 022 B ≈ shared upper layers + 10 × ~1 700 B ≈ 18 KB (matches the per-In slope from G7 vs Q8).

Cryptographic guarantee (via grovedb PR #663 + PR #664): every per-brand count is independently committed to the merk root via node_hash_with_count. The SizedQuery::limit is part of the serialized PathQuery and is part of the merk-root reconstruction the verifier performs — a malicious prover can't truncate the outer walk at a different point without breaking the hash chain.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree"]:::path
  BR ==> B051["brand_051: CountTree count=1000"]:::path
  BR ==> BMore["… 8 more in-range brands (brand_052 … brand_059) …"]:::path
  BR ==> B060["brand_060: CountTree count=1000"]:::path
  BR -.-> BCapped["brand_061 … brand_099<br/>(beyond platform cap — opaque subtree commitments)"]:::faded
  BR -.-> BBelow["brand_000 … brand_050<br/>(below range floor — boundary commitments)"]:::faded

  B051 ==> B051_C["brand_051/color: NonCounted(ProvableCountTree)<br/>ACOR boundary walk (color > color_00000500)"]:::target
  BMore ==> BMore_C["8 parallel ACOR walks"]:::target
  B060 ==> B060_C["brand_060/color: NonCounted(ProvableCountTree)<br/>ACOR boundary walk (color > color_00000500)"]:::target

  SDK["Entries(10 groups, sum=4 990):<br/>(&quot;brand_051&quot;, 499)<br/>(&quot;brand_052&quot;, 499)<br/>…<br/>(&quot;brand_060&quot;, 499)"]:::sdk
  B051_C -.-> SDK
  BMore_C -.-> SDK
  B060_C -.-> SDK

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;
  linkStyle 3 stroke:#1f6feb,stroke-width:3px;
  linkStyle 6 stroke:#1f6feb,stroke-width:3px;
  linkStyle 7 stroke:#1f6feb,stroke-width:3px;
  linkStyle 8 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

L5 is identical to G7's L5 (widget doctype with brand queried). L6 differs: G7 inlined 2 KVValueHash targets for the In-bearing brands; G8 inlines 10 KVValueHash targets for the in-range brands the carrier walks (brand_051 through brand_060), with boundary commitments covering both the below-floor and beyond-cap portions of the byBrand merk tree. L7 + L8 fork into 10 parallel descents, each shaped exactly like G7's L7 + L8 — same NonCounted(ProvableCountTree) continuation, same 37-op ACOR boundary walk over color > color_00000500.

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree"]
    direction TB
    L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
  end

  subgraph L6["Layer 6 — byBrand merk-tree (10 outer-range targets)"]
    direction TB
    L6_t051["<b>brand_051</b><br/>CountTree count=1000"]:::queried
    L6_tmid["… 8 more in-range targets …<br/>(brand_052 … brand_059)"]:::queried
    L6_t060["<b>brand_060</b><br/>CountTree count=1000"]:::queried
    L6_capped["Beyond-cap commitments:<br/>brand_061 … brand_099<br/>(opaque KVHash / Hash ops)"]:::sibling
    L6_floor["Below-floor commitments:<br/>brand_000 … brand_050<br/>(opaque)"]:::sibling

    L6_t051 --> L6_tmid
    L6_tmid --> L6_t070
    L6_t070 --> L6_capped
    L6_t051 --> L6_floor
  end

  subgraph L7L8["Layers 7+8 — per-brand continuation + ACOR walk (×10)"]
    direction TB
    L7L8_each["For each of brand_051 … brand_060:<br/>L7: single-key `color` continuation (NonCounted(ProvableCountTree))<br/>L8: 37 merk ops — ACOR boundary walk for color > color_00000500<br/>committing one `u64 = 499` per brand"]:::target
  end

  L5_q -. "byBrand" .-> L6_t051
  L6_t051 -. "continuation × 20" .-> L7L8_each

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The slope vs G7 is the proof's whole story: G7's k = 2 outer matches → ~4 KB; G8's L = 10 outer matches → ~18 KB. The per-outer-match cost (~1 700 B) is the same; only the outer-walk count changes. The platform max of 10 keeps the worst-case proof under 20 KB (Tier-1 of the visualizer's shareable-link guidance); larger windows are unreachable without changing the constant — callers that want more results call repeatedly with disjoint outer-range windows.

G8a — Bounded carrier + bounded ACOR, grouped by brand, descending

select   = COUNT
where    = brand > "brand_050" AND brand < "brand_065"
       AND color > "color_00000200" AND color < "color_00000400"
group_by = [brand]
order_by = [(brand, desc)]
prove    = true

G8a stresses three carrier-ACOR dimensions G8 didn't: a bounded outer range (instead of half-open), a bounded inner ACOR (instead of > floor), and a descending walk (instead of left-to-right ascending). All three orthogonal. Same RangeAggregateCarrierProof mode, same path-query builder; the differences live entirely in the per-clause QueryItem variants and the carrier's left_to_right flag.

Path query (the carrier query items differ from G8 in three ways: outer item is RangeAfterTo instead of RangeAfter, inner ACOR item is RangeAfterTo instead of RangeAfter, and outer_query.left_to_right = false):

path:                  ["@", contract_id, 0x01, "widget", "brand"]
outer query item:      RangeAfterTo("brand_050".."brand_065")  // exclusive bounds
subquery_path:         ["color"]
subquery items:        [AggregateCountOnRange([RangeAfterTo("color_00000200".."color_00000400")])]
SizedQuery::limit:     10                                       // platform default
outer Query.left_to_right: false                                // from order_by [(brand, desc)]

Same-field range merging. The caller's wire shape carries four range clauses (brand >, brand <, color >, color <). The dispatcher merges each same-field pair into a single BetweenExcludeBounds clause via merge_same_field_range_pairs before mode detection runs. After merging, the structure is identical to G8's two-range shape; mode detection routes to RangeAggregateCarrierProof for the same reasons.

Verified payload (descending walk — outer keys come out from highest to lowest, capped at L = 10):

[("brand_064", 199), ("brand_063", 199), …, ("brand_055", 199)]

The bench's 100-brand fixture has 14 brands strictly between "brand_050" and "brand_065" (i.e. brand_051 through brand_064). The descending walk starts at brand_064 and runs left-to-right=false through the byBrand merk tree; the SizedQuery::limit = 10 halts the walk after 10 outer matches (brand_064 down to brand_055). Each brand's inner ACOR over color > "color_00000200" AND color < "color_00000400" sums to 199 documents (199 colors color_00000201 … color_00000399, one document per (brand, color) pair in the fixture). Total sum = 10 × 199 = 1 990.

Proof size: 29 010 B. Mode: CountMode::GroupByRange routed to DocumentCountMode::RangeAggregateCarrierProof.

G8a is structurally G8 with three independent variant changes, each adding a small amount of merk-proof overhead but no asymptotic complexity change:

  • Bounded outer range → the byBrand merk tree commits both bounds (brand_050 lower-exclusive + brand_065 upper-exclusive) as boundary KVDigest ops. G8's >-only outer commits one boundary; G8a's > AND < commits two. Modest size delta (~1 extra KVDigest per bound × the carrier's tree depth).
  • Bounded inner ACOR → each per-brand color subtree commits both bounds as KVDigestCount ops. G8's >-only ACOR walks O(log C') boundary nodes for the lower bound; G8a's two-sided ACOR walks O(log C') for both bounds. The asymptotic stays O(L · (log B + log C')); the constant roughly doubles for the per-brand boundary walk.
  • Descending walk → grovedb emits PushInverted(...) op variants instead of Push(...) and walks the binary merk tree right-to-left. Same op count as ascending, slightly different serialized encoding (~1–2 bytes per op for the PushInverted opcode discriminant). The verifier's reconstruction is byte-identical given the same left_to_right flag in the PathQuery.

Total proof bytes: 29 010 B vs G8's 18 022 B. Per-outer-match overhead: ~2 900 B (G8a) vs ~1 700 B (G8). The extra ~1 200 B per branch is the bounded-inner-ACOR cost — every per-brand subtree commits twice as many boundary KVDigestCount ops.

Proof display:

Expand to see the structured proof (8 layers; L8 uses two-sided ACOR boundary walks per brand, `PushInverted` outer-walk ops for descending direction) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(... root-level descent, identical to every other chapter query ...)
    lower_layers: {
      @ => { ... contract_id descent ... }
      // L2..L4 byte-identical to every 8-layer carrier query in this chapter
    }
  }
  // L5 widget doctype: brand queried (same as G3 / G5 / G7 / G8)
  // L6 byBrand merk-tree: walked LEFT-TO-RIGHT=FALSE (descending).
  //                       Outer query item: RangeAfterTo("brand_050".."brand_065")
  //                       Inlined targets: brand_064 → brand_063 → ... → brand_055
  //                       via `PushInverted(KVValueHash(brand_NNN, CountTree, ...))` ops.
  //                       Boundary KVDigest nodes name brand_065 (upper-exclusive cut)
  //                       and brand_050 (lower-exclusive cut, capped by SizedQuery::limit).
  // L7 brand_NNN's value tree: single key `color` with NonCounted(ProvableCountTree)
  //    — repeated 10 times, once per resolved outer brand (in descending order).
  // L8 brand_NNN's byBrandColor color subtree:
  //    proof: Merk(
  //      ... ACOR boundary walk for color > "color_00000200" AND color < "color_00000400"
  //          (two-sided cut, ~2× the boundary ops of G8's one-sided ACOR),
  //          summing to count = 199 per brand ...
  //    )
  //    — repeated 10 times in parallel, each with its own per-brand boundary hashes.
}

The 902-line full verbatim sits in the bench's [gproof] G8a output. The schematic compresses the 10 parallel L7+L8 descents and the per-brand boundary commitments — they share the same template (single-key continuation + ~50-op two-sided ACOR boundary walk), differing only in per-brand hashes and the resulting subtree commits. Each per-brand L8 contributes ~2 800 B of ACOR boundary commitments (~1.6× G8's ~1 700 B due to the two-sided range walking both bounds).

The most visually distinctive feature of the descending-walk proof: every L6 carrier op is PushInverted(...) rather than Push(...), signalling grovedb's right-to-left binary-merk-tree iteration. Identical merk-root reconstruction given the same Query.left_to_right = false flag — but the wire-level encoding diverges so the verifier knows which direction to walk.

flowchart TB
  WD["@/contract_id/0x01/widget"]:::tree
  WD ==> BR["brand: NormalTree (descending walk, left_to_right=false)"]:::path
  BR ==> B064["brand_064: CountTree count=1000"]:::path
  BR ==> BMore["brand_063 … brand_056<br/>(8 more in-range brands, descending)"]:::path
  BR ==> B055["brand_055: CountTree count=1000"]:::path
  BR -.-> BBelow["brand_051 … brand_054<br/>(in range but below cap — beyond limit, opaque)"]:::faded
  BR -.-> BAbove["brand_065 (boundary key, excluded by &lt;)"]:::faded
  BR -.-> BCapBelow["brand_000 … brand_050<br/>(below floor, opaque)"]:::faded

  B064 ==> B064_C["brand_064/color: NonCounted(ProvableCountTree)<br/>two-sided ACOR (color > 200 AND color < 400)"]:::target
  BMore ==> BMore_C["8 parallel two-sided ACOR walks<br/>(color > 200 AND color < 400)"]:::target
  B055 ==> B055_C["brand_055/color: NonCounted(ProvableCountTree)<br/>two-sided ACOR (color > 200 AND color < 400)"]:::target

  SDK["Entries(10 groups, sum=1 990) — DESCENDING:<br/>(&quot;brand_064&quot;, 199)<br/>(&quot;brand_063&quot;, 199)<br/>…<br/>(&quot;brand_055&quot;, 199)"]:::sdk
  B064_C -.-> SDK
  BMore_C -.-> SDK
  B055_C -.-> SDK

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;

  linkStyle 0 stroke:#1f6feb,stroke-width:3px;
  linkStyle 1 stroke:#1f6feb,stroke-width:3px;
  linkStyle 2 stroke:#1f6feb,stroke-width:3px;
  linkStyle 3 stroke:#1f6feb,stroke-width:3px;
  linkStyle 7 stroke:#1f6feb,stroke-width:3px;
  linkStyle 8 stroke:#1f6feb,stroke-width:3px;
  linkStyle 9 stroke:#1f6feb,stroke-width:3px;

Diagram: per-layer merk-tree structure (Layer 5+)

L5 is identical to G7 / G8 (widget doctype with brand queried). L6 differs from G8 in two ways: the outer query item is RangeAfterTo (bounded) rather than RangeAfter (half-open), and every op is PushInverted rather than Push because of left_to_right = false. L7 + L8 fork into 10 parallel descents, each carrying a two-sided ACOR boundary walk over color > "color_00000200" AND color < "color_00000400" instead of G8's one-sided color > "color_00000500".

flowchart TB
  subgraph L5["Layer 5 — widget doctype merk-tree"]
    direction TB
    L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
  end

  subgraph L6["Layer 6 — byBrand merk-tree (bounded outer range, descending walk, 10 targets)"]
    direction TB
    L6_t064["<b>brand_064</b><br/>PushInverted(KVValueHash …)<br/>CountTree count=1000"]:::queried
    L6_tmid["… 8 more in-range targets …<br/>(brand_063 → brand_056, descending)"]:::queried
    L6_t055["<b>brand_055</b><br/>PushInverted(KVValueHash …)<br/>CountTree count=1000"]:::queried
    L6_upper["Upper-bound commitment:<br/>KVDigest(brand_065, …) — excluded by &lt;"]:::boundary
    L6_lower["Below-cap + below-floor commitments:<br/>brand_051 … brand_054 (capped)<br/>+ brand_000 … brand_050 (below floor)<br/>(opaque KVHash / Hash ops)"]:::sibling

    L6_t064 --> L6_tmid
    L6_tmid --> L6_t055
    L6_t064 --> L6_upper
    L6_t055 --> L6_lower
  end

  subgraph L7L8["Layers 7+8 — per-brand continuation + two-sided ACOR walk (×10)"]
    direction TB
    L7L8_each["For each of brand_064 … brand_055 (descending):<br/>L7: single-key `color` continuation (NonCounted(ProvableCountTree))<br/>L8: ~50 merk ops — two-sided ACOR boundary walk<br/>for color > 200 AND color < 400<br/>committing one `u64 = 199` per brand"]:::target
  end

  L5_q -. "byBrand" .-> L6_t064
  L6_t064 -. "continuation × 10" .-> L7L8_each

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef boundary fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px,stroke-dasharray: 6 3;

The size delta between G8 and G8a, per outer match: ~1 700 B (G8) → ~2 800 B (G8a). The extra ~1 100 B per brand is roughly evenly split between (a) the bounded inner ACOR's second boundary walk and (b) the per-op PushInverted discriminant overhead. Both costs are linear in L (the platform-max outer cap), so doubling L doubles the delta. The asymptotic complexity stays O(L · (log B + log C')) — the bounded-vs-unbounded distinction is a constant-factor change in the per-walk boundary commit count, not a complexity-class change.

Reading the descending result: the SDK returns Vec<(Vec<u8>, u64)> in the same wire order grovedb walked the outer dimension. For left_to_right = false, that's lex-descending serialized brand keys (brand_064 before brand_063 before … before brand_055). Callers that expect ascending output sort the result client-side; the prove-path guarantee is on the contents (which brands and which counts), not the client-visible ordering — though for chapter-fixture-deterministic proofs the ordering IS visible in the proof bytes via Push vs PushInverted, so the verifier knows which direction grovedb walked.

G8b — Two-range carrier with group_by = [brand, color] (rejected)

select   = COUNT
where    = brand > "brand_050" AND color > "color_00000500"
group_by = [brand, color]
prove    = true

Outcome: Err(QuerySyntaxError::InvalidWhereClauseComponents("count query supports at most one range where-clause; combine two-sided ranges via between*instead of separate>/<clauses, or usegroup_by = [outer_range_field]withprove = true for the carrier-aggregate shape with one outer range and one inner ACOR range on a different field")) — at detect_mode's range_count > 1 short-circuit, before any index picking or path-query building.

Why. The two-range carrier shape (outer_range AND inner_range on distinct fields) is opened by mode detection only when mode == GroupByRange and group_by.len() == 1 and prove = true. G8b violates the first two: with group_by = [brand, color] the request maps to CountMode::GroupByCompound, which routes to distinct_count_path_query — a builder that knows how to walk an In + range fan-out but not a range + range cartesian product. Two design points:

  • GroupByCompound is specifically the (In, range) shape. Its path-query builder emits outer Key(serialized_in_value) items (one per In branch) and an inner Range* subquery; the walk is |In|-bounded by construction. Extending it to accept range + range would mean replacing the outer Keys with an outer Range* (and a SizedQuery::limit to bound the walk) and swapping the inner from "enumerate distinct values" to "single ACOR aggregate" — at which point the result shape stops being "per-distinct-value entries" and becomes "per-outer-key u64s," i.e. G8's shape with a redundant second group_by field. There's no information gain from adding color to the group_by — the carrier already commits one u64 per outer brand, and the inner range collapses into that u64 rather than being enumerated.
  • The carrier primitive returns one u64 per outer key, not per (outer, inner) pair. Per-distinct-color counts inside an outer-range brand walk would require the alternative RangeDistinctProof shape (the G5 compound-distinct path) running on a byBrandColor + rangeCountable: true cartesian fan-out — which works for In + range (a finite outer key set) but would explode for range + range (potentially B × C' distinct entries, dwarfing the MAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT = 10 cap that bounds G8). The dispatcher rejects rather than silently routing to a path that'd produce a proof orders of magnitude larger than the caller likely expected.

What to use instead.

  • If you want per-brand totals across an in-range color window (the most common interpretation of this request), use G8 (group_by = [brand]): one u64 per brand, capped at 10 outer matches.
  • If you want per-(brand, color) distinct counts across both ranges, the dispatcher has no path today — you'd need a byBrandColor + rangeCountable: true index plus a new mode that extends GroupByCompound to range + range with a per-pair SizedQuery::limit. Out of scope for this contract.
  • If you want a single sum across the whole brand > X AND color > Y window, you'd need to call G8 and sum the returned u64s client-side (server-side aggregation across the carrier's per-branch counts isn't supported on the prove path — see G8c below).

G8c — Two-range carrier with group_by = [] (rejected)

select   = COUNT
where    = brand > "brand_050" AND color > "color_00000500"
group_by = []
prove    = true

Outcome: same rejection as G8b — Err(QuerySyntaxError::InvalidWhereClauseComponents("count query supports at most one range where-clause; …")). Mode-detection's range_count > 1 short-circuit checks mode == GroupByRange, and the dispatcher maps group_by = [] to CountMode::Aggregate, so the check fails for the same structural reason as G8b.

Why. With no group_by the request asks for a single scalar u64 covering every document matching both ranges. The carrier-ACOR primitive emits one u64 per outer-range key (10 brands in G8's case), not a single sum across the whole walk. Two paths to a single sum, neither viable today:

  • Server-side sum across the carrier's branches. Would require a new grovedb primitive that takes the carrier shape and emits Σ branch_counts as a single ACOR-style aggregate. Not implemented — the carrier's commitment is per branch, which is what gives the verifier the cryptographic granularity to verify each entry independently. Summing in the server would lose that and force the verifier to trust the server's sum.
  • Client-side sum after running G8. Allowed and easy — call G8, get back Vec<(brand, u64)>, sum the u64s. The proof still cryptographically commits to each branch, and the client's sum is over verified data. This is the pragmatic path for "give me one number" callers; the chapter recommends it instead of opening up Aggregate for the two-range carrier shape.

The deeper reason Aggregate can't shortcut this. Per chapter 29's Q7 (Range Aggregate byColor), Aggregate + single range uses the leaf-level AggregateCountOnRange primitive directly, which DOES return a single u64. That works because the range is rooted at the index's terminator property — there's a single CountTree under which the boundary walk runs. With G8c's two ranges, the outer range walks the byBrand merk tree (no ProvableCountTree involved) and only the inner range hits the rangeCountable terminator. Collapsing across the outer walk would mean a ProvableCountTree over CountTrees, which grovedb's primitive set doesn't have. The walk could in principle compute and emit a sum at the outer layer, but the verifier wouldn't be able to recompute the per-branch counts to check the sum — defeating the prove-path's whole point.

Future Work

This chapter now mirrors chapter 29's per-query structure: every section above carries a path query, verified payload, proof size, verbatim or schematic proof display, narrative, conceptual flowchart, and per-layer merk-tree diagram.

Two pieces of infrastructure made this possible:

  • query_g1_* … query_g8_* criterion bench_function calls (the series skips g6) in document_count_worst_case.rs — produce the Avg time column in Queries in this Chapter.
  • display_group_by_proofs (a sibling of display_proofs in the same bench file) — emits each group_by shape's verbatim merk-proof structure via bincode decode + GroveDBProof::Display. Tagged with [gproof] prefix in stderr so reviewers can grep deterministically.

Open follow-ups:

  1. Inline the full G4 / G5 / G1b verbatim rather than the schematic-with-elision form. The bench captures every byte; the chapter's <details> blocks currently summarise the 100-target enumerations because reproducing 100 near-identical KVValueHashFeatureTypeWithChildHash lines per case is more noise than signal. If a reader needs byte-exact output, they can run the bench and grep [gproof].
  2. Wire path-query reconstruction + verified-payload printing into display_group_by_proofs. Today it only dumps the proof-display block; chapter 29's display_proofs also reconstructs the PathQuery and prints the verifier's structured result (the verified: block). Adding that to the group_by side would give the chapter parity with chapter 29's verified: sections — currently rendered manually from the [matrix] output's Entries(len=N, sum=M) figures.
  3. A high-fanout byColor variant of G1b (color IN [100 values], group_by = [color]) — captured implicitly in the bench's existing group_by_color_in_proof_100_rangecountable_branches (10 512 B) but not given its own G* section, since it's structurally G1b with ProvableCountTree overhead.

Cross-Reference to Chapter 29

For background on the building blocks every query in this chapter uses:

The path-query builder (packages/rs-drive/src/query/drive_document_count_query/path_query.rs) and verifier mirror (packages/rs-drive/src/verify/document_count/) live in the same modules for both chapters' queries — the only difference is which point_lookup_* / aggregate_* / distinct_count_* / carrier_aggregate_* function the dispatcher calls based on the CountMode carried in the request.

Document Sum Trees

Summing a numeric property across the documents that match a query used to mean fetching them all and adding values up client-side. The grovedb upgrade that landed alongside Document Count Trees adds provable sum trees and references with sum item as primitives — the building blocks Drive uses to turn sum(amount)-style queries into an O(log n) provable lookup. This chapter explains the three sum-tree variants, how a document type opts into one, the unified GetDocumentsSum endpoint that exposes the feature, and the parallels with the count-tree machinery.

Status: the grovedb-level sum-tree primitives (SumTree, ProvableSumTree, BigSumTree, and reference elements that carry a sum-item contribution) are in place. The Drive-level schema syntax, query handler, and SDK surfaces described below are the proposed design — the Sum Index Examples chapter is the worked-example companion, and the tip-jar contract fixture at packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json is the schema this design targets.

Why Sum Trees Exist

The default primary-key tree for a document type is a NormalTree. To total the amount field across its documents, Drive walks the subtree, deserializes every record, sums the property client-side, and returns the result. That is fine for small types but becomes painful as soon as a UI needs "how much has this creator received in tips?" on a tip jar with millions of entries — and worse if the caller wants a proof of the total, because the proof has to enumerate every contributing document.

GroveDB has two sum-aware tree variants. Both are provable — the running sum is committed to the merk root in each case — but they differ in where the sum is stored inside the tree, and that controls which kinds of sum queries can be answered without enumerating leaves:

  • SumTree — stores a single i64 sum at the root of the tree. The total sum is one read; any per-subtree sum requires walking down to that subtree's root and reading its (separate) tree element.
  • ProvableSumTree — stores an i64 sum at every internal node, not just the root. Each node's sum covers everything in the subtree below it, so range queries like "what's the total amount tipped between time A and time B?" or "what's the sum per recipient over time?" can be answered by walking the boundary nodes and combining their pre-computed sub-sums, without touching any leaf.

GroveDB merk trees are binary — each internal node has exactly a left and a right child:

The dashed box is the wrapping Element (the "tree" in grovedb terms) and contains the root node of the merk tree. Both variants store the total sum on the wrapping element — that's the O(1) field Drive reads for total sums. The difference is what's inside: in a SumTree the merk root and the rest of the tree don't carry the sum, so only the wrapper has it. In a ProvableSumTree the sum is also stored on the root node itself and on every internal merk-tree node, so it's committed into the merk root hash and provable per-subtree.

flowchart LR
  subgraph ST ["SumTree"]
    direction TB
    subgraph ST_ELEM ["Tree element s=18"]
      direction TB
      A["root"]:::node
    end
    A --> B["·"]:::node
    A --> C["amt=5"]:::leaf
    B --> D["amt=10"]:::leaf
    B --> E["amt=3"]:::leaf
  end

  subgraph PST ["ProvableSumTree"]
    direction TB
    subgraph PST_ELEM ["Tree element s=18"]
      direction TB
      H["root s=18"]:::sumnode
    end
    H --> I["s=13"]:::sumnode
    H --> J["amt=5"]:::leaf
    I --> K["amt=10"]:::leaf
    I --> L["amt=3"]:::leaf
  end

  ST ~~~ PST

  classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef sumnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
  classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;

  style ST_ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb
  style PST_ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb

In a SumTree, the only sum-bearing node is the root. To compute "total amount tipped per recipient" you'd have to navigate to each recipient-keyed subtree (a separate grovedb tree, not a child node of the binary structure above), read its root sum, and pay for a separate proof per read — N reads for N distinct recipients. In a ProvableSumTree, every internal node along the binary path already carries the sum of its left and right subtrees, so a range query like "amounts where sentAt ∈ [t1, t2]" walks only the boundary path and combines the pre-committed sub-sums in a single traversal and a single proof.

A document type opts in via two schema flags. Note that — unlike count, where the flag is a plain bool — sum needs to know which property to sum, so both flags carry a property name:

  • documentsSummable: "<property>" → primary-key tree is a SumTree summing the named property. Enables O(1) total-sum for the document type; sufficient for GetDocumentsSum with no where filter.
  • rangeSummable: true (paired with documentsSummable) → primary-key tree is a ProvableSumTree. The same flag is also accepted per-index, where it controls range-sum storage layout (see below) and is required for any GetDocumentsSum request that carries a range where-clause.

The named property must be type: integer and listed in the document type's required array. We don't define a null contribution rule — a missing-or-null value would have to either contribute 0 (and silently mask a misconfigured insert) or fail the insert (and invalidate documents that were valid at write time). Requiring the property avoids both choices.

How a Document Type Picks Its Tree Variant

Selection lives in DocumentTypePrimaryKeyTreeType::primary_key_tree_type — the same dispatcher that picks the count-tree variant — extended to consider sum flags alongside count flags:

#![allow(unused)]
fn main() {
// proposed v1 selection logic — **sum-only projection** for chapter clarity.
// The real dispatcher also picks the combined count+sum variants
// (`CountSumTree`, `ProvableCountSumTree`,
// `ProvableCountProvableSumTree`) when count flags are set alongside
// the sum flags — those branches are omitted here and covered in
// "Choosing What to Set" below. The v0 count-only logic stays in
// place behind a version bump.
match (range_summable, documents_summable, range_countable, documents_countable) {
    (true,  _,    _,    _)    => Ok(TreeType::ProvableSumTree),
    (false, true, _,    _)    => Ok(TreeType::SumTree),
    (false, false, true,  _)  => Ok(TreeType::ProvableCountTree),
    (false, false, false, true) => Ok(TreeType::CountTree),
    (false, false, false, false) => Ok(TreeType::NormalTree),
}
}

primary_key_tree_type() stays the single source of truth — every Drive code path that needs to know which tree variant to read from or write to routes through this helper, including:

  • Contract insert and update (to CREATE the right tree element when the document type is added).
  • Document insert / delete (to know how to update the sum alongside the document — adding amount to every ancestor sum field, decrementing on delete).
  • Cost estimation (so fees match the variant that will actually be used).

The contract insert/update paths use thin Drive helpers parallel to the existing count variants and the count chapter's batch_insert_empty_*_tree family:

  • batch_insert_empty_tree — NormalTree.
  • batch_insert_empty_sum_tree — SumTree, used when documents_summable.is_some() && !range_summable().
  • batch_insert_empty_provable_sum_tree — ProvableSumTree, used when range_summable().

A sum tree's contents under each value-keyed path are inserted via a different family of helpers — the reference-with-sum-item primitive. Where a non-summable index stores a reference at [index_value]/[0]/<doc_id>, a summable index stores a reference that also carries an i64 sum contribution (the document's amount value at write time). When the parent tree is a SumTree or ProvableSumTree, each insert's sum contribution propagates up the merk path, exactly as a count contribution does in the count case — but the contribution is amount rather than +1. Helpers:

  • batch_insert_sum_item — drops a bare sum item under a sum tree.
  • batch_insert_reference_with_sum_item — drops a reference that contributes a named amount to its parent sum tree. This is the helper non-primary-key sum indexes use.

Each helper goes through LowLevelDriveOperation::for_known_path_key_*_sum_* (or its _estimated_path_key_* cousin in cost-estimation paths), so the contract setup, document operations, and proof generation all see the same on-disk shape.

Storage-Layout Invariants

Because the tree variant is fixed at contract-creation time and baked into how the tree element is laid out on disk, both flags are immutable across a contract update — and the named summable property is too, since changing which property feeds the sum would silently invalidate every existing aggregation:

  • Changing documents_summable from any state to any other state (including changing the property name) on a validate_config update returns DocumentTypeUpdateError.
  • Same for range_summable.

Tests pinning these guards will live alongside the existing count-tree tests in packages/rs-dpp/src/data_contract/document_type/methods/validate_update/v0/mod.rs. Don't relax them: if a NormalTree-backed document type were silently switched to SumTree mid-contract, every subsequent insert or delete would update a sum value attached to a tree element that physically isn't a sum tree, leading to grovedb element-shape errors at best and consensus drift at worst.

The named property's value is read at insert time and frozen into the reference-with-sum-item — Drive doesn't re-read it on delete (it pulls the contribution from the reference itself). So changing the property's value would require a delete-then-reinsert; document mutability concerns apply normally.

Summing Documents at Query Time

A single unified gRPC endpoint exposes the feature: GetDocumentsSum — structurally identical to GetDocumentsCount. The response shape varies by request mode (total / per-In-value / per-distinct-value-in-range / total-over-range), see Range Modes below. The wire-level shape mirrors count: on the no-proof path the response's SumResults carries an inner oneof variant { sint64 aggregate_sum; SumEntries entries; } — total-sum and range-without-distinct modes return aggregate_sum (a single i64), per-In-value and per-distinct-value-in-range modes return entries (a list of SumEntry { optional bytes in_key; bytes key; sint64 sum } where in_key is the prefix value for compound In + range shapes and absent for flat queries). The endpoint has two underlying paths (prove vs. no-prove); every mode is valid on both paths.

The two-path / two-shape split is identical to the count endpoint's, and for the same reasons. What's new in the sum case:

  • Sums are signed (i64 / sint64 on the wire) — grovedb's sum trees model overflow into negative space rather than saturating, so a verifier extracting a sum from a proof can detect "this aggregation overflowed i64::MAX" by recovering a value that doesn't match the document set's expected magnitude. If you expect aggregations beyond i64::MAX, use a BigSumTree-backed variant (bigDocumentsSummable: "amount" — out of scope for this chapter; covered alongside the BigSumTree Drive plumbing).
  • Each sum query needs the property name to sum baked into the picker — there's no implicit "+1 per matched doc." The picker resolves the property from the covering index's summable: "<property>" flag and rejects queries whose target property isn't the same one any candidate index sums.

No-Prove (Server-Side O(1) or O(log n))

When prove=false, drive-abci calls into DriveDocumentSumQuery (the proposed analog of DriveDocumentCountQuery in packages/rs-drive/src/query/drive_document_count_query/mod.rs). The handler picks a path based on the where clauses:

Unfiltered total (no where clauses) on a documentsSummable: "amount" document type:

The doctype's primary-key tree at [contract_doc, contract_id, 1, doctype, 0] is itself a SumTree. One grovedb read gives sum_value — the total of amount across every document of this type. O(1).

Equal/In only:

  1. Pick a summable: "<prop>" index whose properties exactly match the Equal/In where-clause fields. <prop> must equal the request's target sum property. The same strict-coverage contract count uses applies — partial coverage rejects with WhereClauseOnNonIndexedProperty. (See "Index design" below.)
  2. Walk the tree from the root down to the terminal level, pushing prop_name and serialize_value_for_key(prop_name, value) at each step. Equal extends one path; In clones the current path once per value in its array (a cartesian fork) and the per-branch sums are summed.
  3. Read the SumTree element at the resulting path and return its sum_value. O(1) per branch.

If the request carries an In clause, the response is the entries variant — one SumEntry per In value. Otherwise the response is the aggregate_sum variant — a single i64.

Index design contract: a summable: "amount" index sums exactly its declared properties' coverage of amount. Want sum(amount) WHERE recipient = X? Define a [recipient] index with summable: "amount". Want sum(amount) WHERE recipient = X AND sentAt > T? Define a [recipient, sentAt] index with summable: "amount" AND rangeSummable: true. Partial coverage (e.g. recipient = X against a [recipient, sentAt] index without the range clause) is rejected — define a more specific summable index, or set documentsSummable: "amount" on the document type for unfiltered total sums. The prove path enforces the same contract, so prove=true and prove=false reject in the same situations with the same error.

Range:

  1. Pick a rangeSummable: true index where the Equal/In clauses cover the prefix and the range operator hits the index's last property.
  2. Build the path [contract_doc, doctype, prefix..., range_prop_name] — pointing at the property-name ProvableSumTree.
  3. Issue a grovedb path query with the converted range QueryItem (>, >=, <, <=, Range, RangeInclusive, RangeAfter, RangeAfterTo, RangeAfterToInclusive) and walk the children whose keys lie inside the range.
  4. Each child's sum_value_or_default() is the amount sum at that property value. Either combine all per-value sums and return as the aggregate_sum variant (summed mode), or emit them as per-value SumEntrys under the entries variant (distinct mode), then apply order / cursor / limit.

Prove (Client-Side Verify-Then-Aggregate or Aggregate-Sum Proof)

When prove=true, the proof shape depends on whether the query carries a range clause.

With a range clause: the handler picks one of two prove sub-paths based on return_distinct_sums_in_range:

  • Aggregate (return_distinct_sums_in_range = false, default): drive-abci builds a grovedb AggregateSumOnRange path query against the property-name ProvableSumTree, and get_proved_path_query produces an aggregate-sum proof. The client verifies via GroveDb::verify_aggregate_sum_query and recovers (root_hash, sum) directly — proof size is O(log n) regardless of how many keys match. No documents are ever materialized.

  • Distinct (return_distinct_sums_in_range = true): drive-abci builds a regular range path query (no AggregateSumOnRange wrapper) against the same ProvableSumTree. Because the leaf is a ProvableSumTree, merk emits one Node::KVSum(key, value, sum) op per matched in-range key, with each sum cryptographically committed to the merk root via node_hash_with_sum(kv_hash, l_hash, r_hash, sum) — same forge-resistance as the aggregate path's HashWithSum collapse. The SDK's drive_proof_verifier::verify_distinct_sum_proof runs the standard hash-chain check, then walks the proof's op stream to extract the sums as a BTreeMap<Vec<u8>, i64>. Trade-off vs. the aggregate path: proof size is O(distinct values matched) rather than O(log n).

Without a range clause (point-lookup with prove): two sub-paths based on the request shape.

  • Unfiltered total + documentsSummable: "amount": drive-abci proves the doctype's primary-key SumTree element at [contract_doc, contract_id, 1, doctype, 0]. One merk path proof; the SDK's drive_proof_verifier::verify_primary_key_sum_tree_proof reads sum_value off the verified element. O(log n) bytes.

  • Equal/In against a fully-covering summable: "amount" index: drive-abci proves one Element::SumTree per covered branch. Two sub-shapes parallel to count's:

    • Equal-only fully-covered → one element at [..., last_field, last_value, 0].
    • In at any index position (with any number of trailing Equals) → one element per In value, fetched via outer Query + a subquery whose set_subquery_path carries the post-In Equal segments.

    The In position rule and the set_subquery_path mechanics are byte-for-byte the same as the count case — see the count chapter's Prove section for the rationale. Sum picks up the same permissive layout because both paths use point_lookup_sum_path_query (no document-key terminator descent, no order_by interpretation, no limit/offset semantics — it's a pure SumTree-element lookup).

Both sub-paths share the proof shape: each SumTree element's sum_value is cryptographically bound to the merk root via node_hash_with_sum(kv_hash, l_hash, r_hash, sum). Neither materializes documents or runs per-key bookkeeping client-side.

Proof size: O(k × log n) where k is the number of covered branches (1 for the documents_summable fast path and Equal-only fully-covered case; ≤ |In values| for Equal-prefix + In-on-last).

Symmetric rejection contract: prove sum requires a summable: "<prop>" index whose properties exactly match the where clauses and whose summed property matches the request's target — same requirement as the no-proof Total / PerInValue modes. Partial coverage rejects with a WhereClauseOnNonIndexedProperty-class error. The documentsSummable: "<prop>" fast path handles unfiltered total sums in O(log n) proof bytes when set on the document type. No silent fallback to materializing matching documents.

Supported Where Operators

Identical to the count endpoint:

  • Equal (==) — single point lookup against the sum tree at a fully-resolved index path.
  • In (in) — cartesian fork. Each value in the In array becomes its own index path; their sums are combined (or, for split sums, merged by split key). An In clause with k values costs k point lookups, not a tree walk. The In clause also doubles as the per-value split signal in the unified GetDocumentsSum endpoint — at most one In per request.
  • Range (>, >=, <, <=, between*, startsWith) — walks the property-name ProvableSumTree's children whose keys lie inside the range, combining each child SumTree's sum value. Requires the index to have rangeSummable: true AND the range property to be the index's last property.

Range queries take a single range terminator clause plus a prefix of Equal clauses and/or one In clause. In on a prefix property exercises grovedb's native subquery primitive — each emitted entry carries both the in_key (the In value for that fork) and the key (the terminator value within the range). Per-fork sums are NOT merged server-side — same No-Merge Compound Semantics reasoning as count.

Range Modes

A range query produces one of two response shapes, controlled by return_distinct_sums_in_range:

  • return_distinct_sums_in_range = false (default) — SumResults.aggregate_sum carrying the sum of the per-value SumTree sums within the range. Use for "how much was tipped between t1 and t2?".
  • return_distinct_sums_in_range = true — SumResults.entries with one SumEntry per distinct property value within the range. Use for "show me the histogram of tip amounts per timestamp in [t1, t2]".

No-Merge Compound Semantics

For compound queries (In on a prefix property + range on the terminator), entries are returned unmerged — one SumEntry per emitted (in_key, key) pair. The server does NOT collapse them down to a flat histogram. Same three reasons as count:

  1. Correctness under limit. Pushing a limit into grovedb's path query truncates the emitted elements before any merge could run. With cross-fork merging this can undercount the merged sums.
  2. Proof verification stays straightforward. A malicious server omitting one In branch shows up as missing entries with that in_key rather than as a silent undercount in a merged total.
  3. No information loss. A caller who wanted the merged histogram can compute result.fold(by=key, sum=sum) client-side trivially.

The rs-sdk surfaces this via DocumentSplitSums.0: Vec<VerifiedSplitSum>. Callers wanting the historical flat-map shape can call DocumentSplitSums::into_flat_map() which combines across in_key forks.

Pagination

Identical to count's pagination — order_by controls split-mode entry ordering; limit truncates after min(requested, max_query_limit) with None normalized to default_query_limit. Pagination is by range narrowing, not cursor. Ignored on aggregate mode.

Range Queries on the Prove Path

Same shape as count's Range Queries on the Prove Path:

  • Aggregate sub-path (default) builds AggregateSumOnRange — proof size O(log n).
  • Distinct sub-path (return_distinct_sums_in_range = true) builds a regular range proof against the property-name ProvableSumTree. Per-(in_key, key) KVSum ops, each bound to the merk root via node_hash_with_sum.
  • In on a prefix property is supported on the distinct sub-path. The aggregate sub-path rejects In on prefix (single-range merk primitive can't fork at the merk layer).
  • "desc" direction in the first order_by clause flows through to grovedb's Query.left_to_right.

Range Queries and ProvableSumTree

Range sum queries (>, <, between*) over an index with rangeSummable: true are answered in O(log n) by walking the property-name ProvableSumTree's boundary nodes. The proof path uses grovedb's AggregateSumOnRange, which lets clients verify a range sum without ever materializing the underlying documents.

Why Internal-Node Sums Make Range Sums O(log n)

In a sorted merk tree the keys partition into a left (smaller) and right (larger) subtree at every internal node. To answer "what's the sum of amount for documents with sentAt > T?" you walk the boundary between "below T" and "above T" from the root down, and at each step you decide what to do with the other subtree based on a single read:

  • If a subtree lies entirely above the cutoff, add its full sum and don't descend.
  • If it lies entirely below, ignore it (contributes 0).
  • If it straddles, recurse.

On a ProvableSumTree every internal node carries the sum of its left and right subtrees, so the "add the full sum" step is a single O(1) read. The walk visits one node per tree level — O(log n).

Concretely, picture a ProvableSumTree of 8 tips with sorted sentAt keys and amount leaves:

flowchart TB
  R["root s=80"]:::sumroot
  R --> L1["s=30"]:::sumnode
  R --> R1["s=50"]:::sumnode
  L1 --> LL["s=15"]:::sumnode
  L1 --> LR["s=15"]:::sumnode
  R1 --> RL["s=20"]:::sumnode
  R1 --> RR["s=30"]:::sumnode
  LL --> x1["t=1, amt=5"]:::leaf
  LL --> x3["t=3, amt=10"]:::leaf
  LR --> x5["t=5, amt=7"]:::leaf
  LR --> x7["t=7, amt=8"]:::leaf
  RL --> x9["t=9, amt=12"]:::leaf
  RL --> x11["t=11, amt=8"]:::leaf
  RR --> x13["t=13, amt=15"]:::leaf
  RR --> x15["t=15, amt=15"]:::leaf

  classDef sumroot fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sumnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
  classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;

For "give me the sum of amount for items with sentAt > 6":

  • root (s=80): 6 falls inside the left subtree (which holds t=1..7). Read both children's sub-sums. Right subtree's keys are all > 6 → take its full s=50 and don't descend. Recurse into left.
  • left (s=30): 6 falls inside its right subtree (t=5,7). Read both children. Left-left's keys (1,3) are both ≤ 6 → contribute 0. Recurse into left-right.
  • left-right (s=15): 6 splits this leaf-pair. Read both leaves. Key 5 ≤ 6 → contribute 0. Key 7 > 6 → contribute its amt=8.
  • Total = 50 (right of root) + 0 (left-left) + 0 (t=5) + 8 (t=7) = 58.

We visited 4 internal nodes on the boundary path and read sub-sums off 3 siblings without descending. Six of the eight items were never enumerated: their contributions were combined straight out of the committed sub-sum fields.

Why This Is Provable

A merk proof of the same boundary walk includes:

  1. The boundary path from root to the leaf adjacent to the cutoff.
  2. The siblings of every node on the boundary path (so the verifier can recompute hashes up to the merk root).

Each sibling node, on a ProvableSumTree, ships its committed sub-sum alongside its hash. The verifier walks the same logic the server did — "this sibling lies entirely above 6, add its s=… value" — and ends up with the same total without enumerating the sibling subtrees. Verification is also O(log n).

The same primitive answers any range query of the form [A, B]: walk to the cutoff at A, then to the cutoff at B, and combine sub-sums along the way.

Authoring a Contract That Uses Sum Trees

Two opt-in surfaces, parallel to count. They're independent and can be used together:

  1. Top-level flags on the document type control the primary-key tree variant.
  2. A per-index summable: "<property>" flag controls whether that specific index's tree carries sums.

Primary-Key Tree Flags

Set at the same level as type / properties / indices on a document type:

{
  "tip": {
    "type": "object",
    "documentsSummable": "amount",
    "properties": {
      "recipient": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 0,
                     "contentMediaType": "application/x.dash.dpp.identifier" },
      "amount":    { "type": "integer", "minimum": 1, "maximum": 4294967295, "position": 1 },
      "sentAt":    { "type": "integer", "minimum": 0, "position": 2 }
    },
    "required": ["recipient", "amount", "sentAt"],
    "additionalProperties": false
  }
}

That contract gets a SumTree for the tip primary-key tree, summing amount. GetDocumentsSum for tip with no where filter is now an O(1) lookup of the tree element's sum value.

To opt into a ProvableSumTree for the primary-key tree instead — useful if you want range queries on the primary key or intend to use this document type behind range proof primitives — pair with rangeSummable: true:

{
  "tip": {
    "type": "object",
    "documentsSummable": "amount",
    "rangeSummable": true,
    ...
  }
}

Both flags are immutable across a contract update — you pick the tree variant at contract creation; you can't switch later without creating a new document type.

Per-Index Summable Flag

Set on a single entry in the document type's indices array:

{
  "indices": [
    {
      "name": "byRecipient",
      "properties": [{ "recipient": "asc" }],
      "summable": "amount"
    }
  ]
}

With byRecipient.summable: "amount" the byRecipient index's tree carries running sums of amount, so GetDocumentsSum with where: [["recipient", "==", X]] reaches the sum via that index in O(1). Without the flag the query rejects with WhereClauseOnNonIndexedProperty — there's no slow fallback, only fast sums on properly-indexed properties.

The summable field accepts a single shape: a string naming an integer property declared on the document type and listed in required. The named property must be the same one named at any other summable level (doctype documentsSummable and other indexes' summable) — multi-property summing on a single tree isn't supported and won't be; if you need to sum two properties, declare two separate aggregation surfaces.

A few notes about the index-level flag:

  • Setting summable increases storage cost — every insert and delete updates the index tree's sum alongside the document, and the reference under the index value tree carries an extra i64 sum-item contribution.
  • Setting rangeSummable: true increases storage cost further — every internal node of the property-name tree carries running-sum metadata, not just the root.
  • The flag is on the whole index, not per-property. Same strict-coverage rule as count: a ["recipient", "sentAt"] summable index gives O(1) sums for WHERE recipient = X AND sentAt = T but NOT for WHERE recipient = X alone. Define both indexes if you want both queries.
  • Index-level summable is independent of the primary-key flags. You can have documentsSummable: "amount" on the document type AND summable: "amount" on a specific index.
  • summable on a unique index is mostly a no-op, but not always, mirroring count's caveat. A unique index stores its terminal as a bare reference at key [0] rather than wrapping it in a sum tree, so for documents whose indexed fields are all non-null the flag has no storage effect. Null-bearing entries take the same sum-tree branch a non-unique index uses, and the sum tree at that path aggregates them. Sums on all-non-null exact matches still return correctly (the reference's stored sum contribution) because the on-disk reference reads as a sum item via grovedb's default-aggregate semantics.

Choosing What to Set

You wantSet
Fast sum(amount) for the whole document typedocumentsSummable: "amount" on the document type
O(1) filtered sum: sum(amount) WHERE col = Xsummable: "amount" on an index whose properties are exactly ["col"]. Partial coverage of a wider index rejects with WhereClauseOnNonIndexedProperty — define a dedicated index.
Per-In-value sub-sumssummable: "amount" on an index whose properties exactly match the query's == clauses plus the In field. The In field may sit at any position.
O(log n) range sum: sum(amount) WHERE col BETWEEN A AND BrangeSummable: true on an index whose last property is col and whose other properties cover any equality predicates as a prefix. Requires summable: "amount".
Per-distinct-value range histogramSame rangeSummable: true index as above, plus return_distinct_sums_in_range = true on the request.
Range sum proofSame rangeSummable: true index. Handler uses grovedb's AggregateSumOnRange — proof is O(log n), no cap on matched docs.
Aggregations beyond i64::MAXOut of scope for this chapter — see the BigSumTree variant.
Both a sum AND a count on the same treeCombine the count flags (documentsCountable / rangeCountable / per-index countable) with the sum flags. The dispatcher picks one of three combined variants depending on which axes opt into per-node aggregation: CountSumTree (both at root only), ProvableCountSumTree (per-node count, root-only sum — useful when range count is wanted but range sum isn't), or ProvableCountProvableSumTree / PCPS (both per-node — the grovedb PR 670 newcomer, enables AggregateCountAndSumOnRange recovering both metrics in a single range proof). One tree, two simultaneous queries, no double storage. The tip-jar contract above doesn't use these combinations (it's pure-sum to keep the introduction focused); a worked example using (count, sum) together is covered in a separate chapter alongside its own example contract.
Nothing sum-aware (default)Don't set any of these flags. Primary-key tree stays a NormalTree.

Every sum query requires either documentsSummable: "<prop>" (for unfiltered totals) or a summable: "<prop>" / rangeSummable: true index whose properties exactly match the query's where-clause fields. No covering index → the call returns a clear InvalidArgument describing what the picker was looking for. Pick your indexes deliberately at contract creation time — per-index summable / rangeSummable flags can't be added later (contract indexes are immutable post-creation).

SDK Access at Three Layers

rs-sdk (native Rust)

Both shapes will land on the standard Fetch trait against a single DocumentSumQuery:

#![allow(unused)]
fn main() {
use dash_sdk::platform::documents::document_sum_query::DocumentSumQuery;
use dash_sdk::platform::Fetch;
use drive::query::{WhereClause, WhereOperator};
use drive_proof_verifier::{DocumentSum, DocumentSplitSums};

// Total sum: no In clause.
let DocumentSum(sum) = DocumentSum::fetch(
    &sdk,
    DocumentSumQuery::new(contract.clone(), "tip", "amount")?,
)
.await?
.expect("DocumentSum::fetch always returns a value on success");

// Split sum: signal split by including an `In` clause whose field
// is the split property.
let split_query = DocumentSumQuery::new(contract, "tip", "amount")?
    .with_where(WhereClause {
        field: "recipient".to_string(),
        operator: WhereOperator::In,
        value: platform_value::Value::Array(vec![alice.into(), bob.into()]),
    });
let splits = DocumentSplitSums::fetch(&sdk, split_query)
    .await?
    .expect("DocumentSplitSums::fetch always returns a value on success");
// `splits` is `DocumentSplitSums(Vec<SplitSumEntry>)` — collapse via `splits.into_flat_map()`.
}

DocumentSumQuery wraps an internal DocumentQuery (reusing where-clause / order-by / contract-id machinery) and exposes with_where(WhereClause) + with_order_by(OrderClause) builders. The SDK picks the request mode from query shape plus explicit request flags. The target sum property is part of the query construction — the SDK validates against the contract that some covering index sums it.

wasm-sdk (browser)

Two methods on the WasmSdk JS class — one entry per [plain | withProofInfo] variant covers every sum mode:

sdk.getDocumentsSum(
  query: DocumentsQuery,
  sumProperty: string,
): Promise<Map<string, bigint>>;

sdk.getDocumentsSumWithProofInfo(
  query: DocumentsQuery,
  sumProperty: string,
): Promise<ProofMetadataResponseTyped<Map<string, bigint>>>;

Result shapes mirror count's wasm SDK, with bigint carrying a signed sum:

  • No where, or Equal-only where — single map entry with the empty-string key carrying the total sum.
  • where includes an In clause — one entry per (deduped) In value.
  • where includes a range clause + returnDistinctSumsInRange: true — one entry per distinct property value in the range.

Map keys are hex-encoded bytes matching the canonical serialize_value_for_key encoding of each property value, same convention as count.

rs-sdk-ffi (iOS / native bindings)

#![allow(unused)]
fn main() {
dash_sdk_document_sum(
    sdk,
    data_contract,
    document_type,
    sum_property,                      // name of the integer property to sum
    where_json,                        // null or JSON [{field, operator, value}]
    order_by_json,                     // null or JSON [{field, direction}]
    return_distinct_sums_in_range,     // bool
    limit,                             // i64; -1 = server default, >= 0 = explicit cap
) -> JSON {"sums": {"<hex-key>": <i64>, ...}}
}

Single FFI entry covers every sum mode — the result is always {"sums": {...}} with hex-encoded keys. For total sums (no where/In, distinct flag off), the map carries a single entry with the empty-string key. where_json is the same JSON shape dash_sdk_document_search already accepts. The endpoint returns its result as a JSON-encoded C string allocated on the heap — caller frees it via the standard SDK string-free routine.

Sum Index Examples

This chapter walks through a representative contract and shows what a sum-query proof actually proves — both the path query the prover signs and the verified element the verifier extracts. Every example uses the same tip document type on the tip-jar contract at packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json, so the proof bytes, verified elements, and diagrams can all be cross-referenced against the same data once the bench fixture lands.

The chapter assumes you've read Document Sum Trees — that chapter explains the three tree variants (NormalTree / SumTree / ProvableSumTree), how Element::NonCounted-style "doesn't contribute to my parent's aggregation" wrappers work (now for sums as well), and how the schema's documentsSummable / rangeSummable flags select between them. Here we take that machinery as given and trace what each query sees.

Status: the bench at packages/rs-drive/benches/document_sum_worst_case.rs lands the reproducible numbers below — same convention as the Count Index Examples chapter. All proof sizes are measured against a 100 000-row fixture; verified sum values are the actual sums the bench's matrix reports. The full surface — primary-key total, point lookups, In-fan-out, AggregateSumOnRange on both top-level and compound indexes, and the carrier-aggregate primitive from grovedb PR #670 — is fully wired and producing the byte counts in the table below.

The Tip Jar Contract

The tip document type carries four properties (recipient, amount, sentAt, note), opts into total sums at the doctype level via documentsSummable: "amount", and declares three indexes covering the sum-query surface:

{
  "type": "object",
  "documentsMutable": false,
  "documentsSummable": "amount",
  "properties": {
    "recipient": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
                   "position": 0, "contentMediaType": "application/x.dash.dpp.identifier" },
    "amount":    { "type": "integer", "minimum": 1, "maximum": 4294967295, "position": 1 },
    "sentAt":    { "type": "integer", "minimum": 0, "position": 2 },
    "note":      { "type": "string", "maxLength": 280, "position": 3 }
  },
  "required": ["recipient", "amount", "sentAt"],
  "indices": [
    {
      "name": "byRecipient",
      "properties": [{ "recipient": "asc" }],
      "summable": "amount"
    },
    {
      "name": "bySentAt",
      "properties": [{ "sentAt": "asc" }],
      "summable": "amount",
      "rangeSummable": true
    },
    {
      "name": "byRecipientTime",
      "properties": [{ "recipient": "asc" }, { "sentAt": "asc" }],
      "summable": "amount",
      "rangeSummable": true
    }
  ],
  "additionalProperties": false
}

Three things to notice:

  1. documentsSummable: "amount" at the document-type level upgrades the doctype's primary-key subtree (at tip/[0]) from NormalTree to SumTree. The unfiltered total sum is one read against this element's sum_value. The string-form value names the property each insert contributes to the tree — the picker uses it to validate that any future GetDocumentsSum request whose sum_property doesn't match "amount" is rejected at parse time.
  2. byRecipient is summable: "amount" only. It doesn't opt into rangeSummable, so recipient > X range sums aren't supported. Every summable terminator's value tree is stored as a SumTree regardless of rangeSummable, so point-lookup sum proofs (e.g. recipient == X or recipient IN [...]) get a compact value-tree-direct shape. rangeSummable is strictly an opt-in for AggregateSumOnRange support — orthogonal to proof-size shape on point queries.
  3. bySentAt and byRecipientTime are rangeSummable: true. Their property-name subtrees (e.g. tip/sentAt) are stored as ProvableSumTree rather than NormalTree, which is what AggregateSumOnRange walks for sentAt > floor style queries.

The bench populates 100 000 tips under a deterministic schedule: row → (recipient_(row % 100), sentAt = row, amount = (row % 10) + 1). That gives exactly 1 000 tips per recipient, but with an asymmetry worth flagging: since both recipient and amount are derived from row modulo (100 and 10 respectively), each recipient sees only one amount value across all their 1 000 tips. Recipient n always has amount = (n % 10) + 1:

  • recipient_000, recipient_010, …, recipient_090 → amount = 1, per-recipient sum = 1 000
  • recipient_001, recipient_011, …, recipient_091 → amount = 2, per-recipient sum = 2 000
  • …
  • recipient_009, recipient_019, …, recipient_099 → amount = 10, per-recipient sum = 10 000

The headline numbers:

  • Total sum(amount) across all tips: 550 000 (each amount value 1..10 appears 10 000 times → 10 000 × 55). The primary-key documentsSummable SumTree reports this directly via Query 1's O(1) read.
  • sum(amount) per recipient: varies 1 000–10 000 by recipient (see the cycle above).
  • sum(amount) for any contiguous sentAt range of length 10: exactly 55 (every 10-row window covers one full cycle of 1..10).
  • sum(amount) for the first half of the timeline (sentAt < 50 000): 275 000.

Those numbers appear in every verified-element block below.

GroveDB Layout

The contract above produces this storage shape. Tree elements (the wrapping Element GroveDB stores under each key) are drawn as subgraphs; children inside each tree are merk-tree nodes. The doctype root and the per-property name subtrees are separate Element trees nested under the contract-documents prefix, just like every other index in Drive.

Diagram conventions: green nodes carry a sum_value committed to the merk root; yellow nodes are ProvableSumTree (per-node sums); gray are regular subtrees; dashed boxes highlight Element::NonCounted-style wrappers (children that store data but contribute 0 to their parent's aggregation).

flowchart TB
  TD["@/contract_id/0x01/tip"]:::tree

  TD --> PK["[0]: SumTree sum=550000<br/>(documentsSummable primary key)"]:::sumnode
  TD --> RC["recipient: NormalTree<br/>(byRecipient property-name)"]:::node
  TD --> SA["sentAt: ProvableSumTree<br/>(bySentAt property-name)"]:::pstnode

  RC --> R000["recipient_000: SumTree sum=1000<br/>(amount cycle: 1)"]:::sumnode
  RC --> R050["recipient_050: SumTree sum=1000<br/>(amount cycle: 1)"]:::sumnode
  RC --> RMore["... recipient_001 sum=2000 ... recipient_009 sum=10000<br/>(per row%10 amount cycle)"]:::sumnode

  R050 --> R050_0["[0]: SumTree sum=1000<br/>(byRecipient refs)"]:::sumnode
  R050 --> R050_S["sentAt: NonCounted(ProvableSumTree)<br/>(byRecipientTime continuation, contributes 0)"]:::noncounted

  R050_S --> R050_S_500["sentAt_00050000: SumTree sum=1"]:::sumnode
  R050_S_500 --> R050_S_500_0["[0]: SumTree sum=1<br/>(byRecipientTime ref, amount=1)"]:::sumnode

  SA --> S500["sentAt_00050000: SumTree sum=1<br/>(bySentAt terminator, amount=1)"]:::sumnode
  SA --> SMore["... sentAt_00000000 ... sentAt_00099999"]:::sumnode
  S500 --> S500_0["[0]: SumTree sum=1<br/>(bySentAt ref)"]:::sumnode

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef sumnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
  classDef pstnode fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef noncounted fill:#21262d,color:#c9d1d9,stroke:#fb8500,stroke-width:2px,stroke-dasharray: 6 4;

Three layout facts to internalize before reading the queries:

  • recipient_050 is a SumTree with sum_value = 1000. That's true because byRecipient is summable; the rule applies uniformly to every summable tier. The sentAt continuation that branches off this value tree is NonCounted-wrapped so the parent's sum equals exactly the contribution in [0] (which for recipient_050 is 1 000 × 1 = 1 000 per the amount cycle). Other recipients get different value-tree sums per the per-recipient amount described above.
  • tip/sentAt is a ProvableSumTree, not a regular NormalTree. The yellow class above marks that — each internal merk node carries its subtree's sum, which is what makes AggregateSumOnRange a single-pass primitive.
  • sentAt_00050000 is a SumTree with sum_value = 1 under either parent (the global bySentAt path or the per-recipient byRecipientTime continuation). Same element layout under both; the path that gets there differs but the destination is structurally the same.

How To Read The Proofs

Every example below has four sections:

  1. Path query — the spec the prover hands GroveDB. path is the list of subtree segments to descend through; query items is what to select once at the bottom; subquery items (when present) descends one more layer.
  2. Verified element — what GroveDB::verify_query (or verify_aggregate_sum_query for the range primitive) returns after walking the proof bytes. The sum_value_or_default field on a SumTree element is what the sum surface ultimately surfaces to the caller.
  3. Proof display — the proof bytes decoded via bincode into the structured GroveDBProof AST and rendered through its Display impl, same convention as the count chapter. The bench's display_proofs block emits these inline; reproduce locally with DASH_PLATFORM_SUM_BENCH_REBUILD=1 cargo bench -p drive --bench document_sum_worst_case -- --test 2>&1 | grep -A 200 "^\[display\]".
  4. Diagram — the path the proof walks through the layout. Blue arrows trace the descent; the cyan node is the verified element; faded gray nodes show context.

Proof-size numbers below come from the 100 000-row bench run on the byRecipient / bySentAt / byRecipientTime indexes. Avg-time numbers are median-of-5 wall-clock measurements with one warmup discarded (see the methodology note under the queries table).

Queries in this Chapter

#QueryFilterComplexityAvg timeProof size
1Unfiltered Total Sum(none — total at doctype level)O(1)23.6 µs580 B
2Equal on a Single Property (byRecipient)recipient == "recipient_050"O(log R)37.2 µs1 087 B
3Equal on a RangeSummable Property (bySentAt)sentAt == 50000O(log T)72.1 µs1 706 B
4Compound Equal-only (byRecipientTime)recipient == "recipient_050" AND sentAt == 50000O(log R + log T')71.8 µs1 937 B
5In on byRecipientrecipient IN ["recipient_000", "recipient_001"]O(k · log R)42.8 µs (k=2) / 1 720 µs (k=100)1 168 B (k=2) / 12 064 B (k=100)
6In on bySentAt (RangeSummable)sentAt IN [0, 1]O(k · log T)81.2 µs (k=2) / 2 008 µs (k=100)1 756 B (k=2) / 9 784 B (k=100)
7Range Query (AggregateSumOnRange)sentAt > 50000O(log T)102.0 µs3 102 B
8Compound == + Range (byRecipientTime)recipient == "recipient_050" AND sentAt > 50000O(log R + log T')91.3 µs2 657 B
9Carrier-Aggregate (In + range)recipient IN [r000..r099] AND sentAt > 50000 (group_by = [recipient])O(k · log T')11 507.9 µs (k=100)169 064 B (k=100)

Timing methodology: median of 5 iterations after one warmup, measured against the bench's 100 000-row fixture on a warmed rocksdb cache. The figures reflect the drive-layer execute_document_sum_request call (executor + grovedb proof generation, no network or tenderdash signature compose). Reproduce with cargo bench -p drive --bench document_sum_worst_case -- --test; grep µs from stderr.

Complexity variables. R = distinct recipients in the byRecipient merk-tree (= 100 in the fixture); T = distinct timestamps in the bySentAt merk-tree (= 100 000); T' = distinct timestamps per recipient in byRecipientTime's continuation (= 1 000 per recipient); k = number of values in the IN clause (2 here). Notably absent: the total document count N (100 000 here). Sum proofs read pre-committed sum_values from SumTree merk roots — they never enumerate the underlying documents, so proof generation cost is polylog(distinct index values), independent of N. Same big-O story as count.

The bySentAt index's T = 100 000 is unusually large (one distinct timestamp per row in this fixture); real tip jars would bucket timestamps coarsely. A reproducible-numbers fixture deliberately maximizes distinct values to stress the prove paths' merk-traversal cost — the per-merk-op count and proof-size numbers will skew accordingly when the bench publishes them.

Query 1 — Unfiltered Total Sum

select       = SUM(amount)
where        = (empty)
sum_property = "amount"
prove        = true

Path query (primary-key SumTree fast path; no index walk needed):

path:         ["@", contract_id, 0x01, "tip"]
query items:  [Key(0x00)]

Verified element:

path:        ["@", contract_id, 0x01, "tip"]
key:         0x00
element:     SumTree { sum_value_or_default: 550000 }

Proof size: 580 bytes. Avg time: 23.6 µs. Verifier root hash: 95aa74708738c5254c706bbca3245b520022aad949674822218094bca15671b6 (same across every query in this chapter — same fixture state).

Proof display (GroveDBProof::Display):

Expand to see the structured proof (5 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
                  1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
                      lower_layers: {
                        tip => {
                          LayerProof {
                            proof: Merk(
                              0: Push(KVValueHashFeatureTypeWithChildHash(0x00, Tree(0000000000010000fffffffffffeffff00000000000000000000000000000000), HASH[f25f49fc619abc59d984dcb322947509eb2a34f02217fc31dfc11423bee001e8], BasicMerkNode, HASH[d8c56d5b5d11c2e30a70694fac85259bb3169854e474ae8056ef91d5cb877000]))
                              1: Push(KVHash(HASH[a17138f666ae4ab19c1c2930ad94c7e29a6a82789398fc4d3a0b053d3499ac68]))
                              2: Parent
                              3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
                              4: Child)
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. The descent goes: top-level GroveDB root → @ (the DataContractDocuments root tree) → contract id → 0x01 (documents storage prefix) → tip doctype → finally the Key(0x00) payload at the bottom. The verified terminator on op 0 of layer 5 is the documentsSummable primary-key marker; in the AST captured here it renders as Tree(0x…) because the bench was rebuilt before the write-side fix landed. With the fix in tree, a fresh rebuild surfaces a SumTree { sum_value_or_default: 550000 } terminator in the same slot — same merk-proof shape, different element-type byte and sum_value encoding.

The descent stops at the doctype's primary-key tree — the green node at the top of the layout. Because documentsSummable: "amount" upgraded that tree to a SumTree, the total is one O(1) read with an O(log n) proof.

Diagram: per-layer merk-tree structure

Each LayerProof above is its own GroveDB sub-tree whose contents form a merk binary tree. The merk-proof operations (Push / Parent / Child over KVValueHash / KVHash / Hash nodes) describe exactly which nodes of each layer's binary tree the proof reveals — the queried key gets its full kv-hash exposed; opaque siblings only commit their subtree-hash so the verifier can re-hash up to the merk root.

Cyan = the verified target. Blue = a kv-hash that's also a queried-key on the descent path (its value = Tree(...) is the merk-root pointer for the next layer). Gray = opaque sibling subtrees committed by hash only.

flowchart TB
  subgraph L1["Layer 1 — root GroveDB merk-tree"]
    direction TB
    L1_root["<b>@</b><br/>kv_hash=HASH[2f2b...]<br/>value: Tree(0x4ed2…)"]:::queried
    L1_left["HASH[bd29...]<br/>(left subtree, opaque)"]:::sibling
    L1_right["HASH[19c9...]<br/>(right subtree, opaque)"]:::sibling
    L1_root --> L1_left
    L1_root --> L1_right
  end

  subgraph L2["Layer 2 — @ subtree merk-tree (single key)"]
    direction TB
    L2_q["<b>contract_id 0x4ed2…</b><br/>kv_hash=HASH[5b1e...]<br/>value: Tree(0x01)"]:::queried
  end

  subgraph L3["Layer 3 — contract_id subtree merk-tree"]
    direction TB
    L3_q["<b>0x01</b><br/>kv_hash=HASH[fba9...]<br/>value: Tree(tip)"]:::queried
    L3_left["HASH[c53e...]<br/>(left subtree, opaque)"]:::sibling
    L3_q --> L3_left
  end

  subgraph L4["Layer 4 — 0x01 documents-prefix subtree (single key)"]
    direction TB
    L4_q["<b>tip</b><br/>kv_hash=HASH[0fe5...]<br/>value: Tree(recipient/sentAt/0x00)"]:::queried
  end

  subgraph L5["Layer 5 — tip doctype merk-tree (TARGET layer)"]
    direction TB
    L5_target["<b>0x00</b><br/>kv_hash=HASH[f25f...]<br/>value: <b>SumTree sum=550000</b><br/>child_hash=HASH[d8c5...]<br/>(captured AST shows Tree+sum=0; fresh rebuild refreshes)"]:::target
    L5_kv["KVHash[a171...]<br/>(opaque internal kv: sentAt or recipient)"]:::sibling
    L5_right["HASH[143e...]<br/>(right subtree, opaque)"]:::sibling
    L5_target --> L5_kv
    L5_target --> L5_right
  end

  L1_root -. "value=Tree(merk_root[5b1e…])" .-> L2_q
  L2_q -. "value=Tree(merk_root[fba9…])" .-> L3_q
  L3_q -. "value=Tree(merk_root[0fe5…])" .-> L4_q
  L4_q -. "value=Tree(merk_root[f25f…])" .-> L5_target

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

Layers 1–4 are the standard contract-documents descent every query in this chapter shares; the divergence starts at Layer 5. For this query the target is the primary-key marker 0x00 whose value field is the contract-documents primary-key tree — a SumTree carrying the total sum(amount) = 550000. Each of the next four queries diverges at this layer to a different doctype child (recipient, sentAt, or recipient → recipient_050 → sentAt).

Query 2 — Equal on a Single Property (byRecipient)

select       = SUM(amount)
where        = recipient == "recipient_050"
sum_property = "amount"
prove        = true

Path query:

path:         ["@", contract_id, 0x01, "tip", "recipient"]
query items:  [Key("recipient_050")]

Verified element:

path:        ["@", contract_id, 0x01, "tip", "recipient"]
key:         "recipient_050"
element:     SumTree { sum_value_or_default: 1000 }

Proof size: 1 087 bytes. Avg time: 37.2 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (6 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
                  1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
                      lower_layers: {
                        tip => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
                              1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
                              2: Parent
                              3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
                              4: Child)
                            lower_layers: {
                              recipient => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[b8804ea3f7998def339db7cadd6a6b29ba5bfadbdfa15b67802ef52e42adb68e]))
                                    1: Push(KVHash(HASH[3056a7c2daf102d31d0f461d45e661303b1562311971debbf15f40938727a074]))
                                    2: Parent
                                    3: Push(Hash(HASH[c406363887b632d071f2ee6ed83df682aace9a5456997aa4f4b944576476fbb6]))
                                    4: Push(KVHash(HASH[6b8ef1df5ba1299cd5b605dc7642be2ffb8356fd7f51c3a0498b6b657cddeebc]))
                                    5: Parent
                                    6: Push(Hash(HASH[347abc0b69e504bc619e9549e509c21be3c0ad1c9e03d8fb34bb5ed07e5cd26f]))
                                    7: Push(KVHash(HASH[ff9b5006130777d589b77e6f5bdda0f86879daace181cdc8e3d97bc3718e0a48]))
                                    8: Parent
                                    9: Push(KVValueHashFeatureTypeWithChildHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[264c6aac1a1acd864832a6f25ac42c626afb95dc79ac8c233d2b05c6df048f1c], BasicMerkNode, HASH[58680498d71fdda362f4a1815a0e0989b70a8cb287d8e40eaf84f8fe2cb48b6f]))
                                    10: Child
                                    11: Push(KVHash(HASH[5159e1ccad8c4ed1bbf7f52ec34e9ab4ee108559194e39b1af78cf42866b32cc]))
                                    12: Parent
                                    13: Push(Hash(HASH[4317777613ab1a0d6966670195ccce83dee1031fd37013c9ce625c016d1d6d17]))
                                    14: Child
                                    15: Push(KVHash(HASH[24f6646d8b75a6a428d05589c1cc1e80f1e52900924710ff6eafe9da0edc73d0]))
                                    16: Parent
                                    17: Push(Hash(HASH[14f8282bd0b5d9a8e7e471d3cfd215f02f5be2cfbf837c5e938c2871a2eddcb9]))
                                    18: Child
                                    19: Child
                                    20: Child
                                    21: Push(KVHash(HASH[cea6360efbf77b38d2ea206d508ea03c0d92fe7c02c5d9176aa322e8263a3acc]))
                                    22: Parent
                                    23: Push(Hash(HASH[209f3325816d5f02a1647311a4f1d9c68fcbacf1540be9b5ae65413f58ca3b26]))
                                    24: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. The descent diverges from Q1 at layer 5: the tip doctype layer commits the recipient property-name subtree on op 1 (cyan-blue blue queried, valued Tree(0x6263...) — the merk root of byRecipient), then descends into layer 6, byRecipient itself. There the queried recipient_050 key (0x0000000000000032ffffffffffffffcd...) is reached as op 9 of a 25-op merk path, and its value is SumTree(73656e744174, 1000) — the 73656e744174 bytes are ASCII for "sentAt" (the byRecipientTime continuation pointer, NonCounted-wrapped at storage so it contributes 0 to the parent SumTree), and the 1000 is the per-recipient sum_value for recipient_050 (1 000 tips × amount = 1 = 1 000). Verified root hash 95aa7470…71b6.

The descent walks one extra layer into the recipient property-name subtree and stops at recipient_050. Because byRecipient is summable: "amount", that node is a SumTree carrying the per-recipient sum directly — no need to step into [0] to look at individual references, exactly the same shortcut count proofs take. (recipient_050's amount = (50 % 10) + 1 = 1, so the per-recipient sum is 1 000 × 1 = 1 000. A recipient with a different n % 10 lands a proportionally larger sum — see the fixture narrative above.)

Diagram: per-layer merk-tree structure

Layers 1–4 are byte-for-byte identical to Q1's diagram (root → @ → contract_id → 0x01). The descent diverges at Layer 5, where this query takes the recipient branch (rather than 0x00) and descends one extra grove layer to land on the verified target inside byRecipient.

flowchart TB
  subgraph L5["Layer 5 — tip doctype merk-tree (proof view for `recipient`)"]
    direction TB
    L5_q["<b>recipient</b><br/>kv_hash=HASH[f580...]<br/>value: Tree (descent into byRecipient)"]:::queried
    L5_left["HASH[15fd...]<br/>(left subtree, opaque)"]:::sibling
    L5_right["HASH[143e...]<br/>(right subtree, opaque)"]:::sibling
    L5_q --> L5_left
    L5_q --> L5_right
  end

  subgraph L6["Layer 6 — byRecipient merk-tree (TARGET layer)"]
    direction TB
    L6_target["<b>recipient_050</b><br/>kv_hash=HASH[264c...]<br/>value: <b>SumTree sum=1000</b><br/>(value bytes 73656e744174 = `sentAt` continuation, NonCounted)<br/>child_hash=HASH[5868...]"]:::target
    L6_boundary["Boundary commitments (24 merk ops):<br/>7 KVHash opaque sibling recipient kvs<br/>+ 6 Hash subtree commitments<br/>(prove recipient_050's position in byRecipient's<br/>binary merk tree of ~100 recipient entries)"]:::sibling
    L6_target --> L6_boundary
  end

  L5_q -. "value=Tree(merk_root[byRecipient])" .-> L6_target

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The boundary commitments at L6 scale linearly with byRecipient's merk depth — they bind recipient_050 to its claimed position so the verifier can recompute byRecipient's merk root. The verified target itself is one KVValueHashFeatureTypeWithChildHash op whose SumTree(…, 1000) value carries the per-recipient sum directly; the continuation-pointer bytes 73656e744174 simply name the next layer's tree (which we never descend into for a Q2 point lookup).

Query 3 — Equal on a RangeSummable Property (bySentAt)

select       = SUM(amount)
where        = sentAt == 50000
sum_property = "amount"
prove        = true

Path query:

path:         ["@", contract_id, 0x01, "tip", "sentAt"]
query items:  [Key(serialize_value_for_key("sentAt", 50000))]

Verified element:

path:        ["@", contract_id, 0x01, "tip", "sentAt"]
key:         serialize_value_for_key("sentAt", 50000)
element:     SumTree { sum_value_or_default: 1 }

Proof size: 1 706 bytes — moderately larger than Query 2's 1 087 (a 619-byte delta) because the sentAt property-name subtree is a ProvableSumTree, so each merk node on the descent carries an extra i64 sum field versus byRecipient's plain NormalTree. (Same direction-and-magnitude delta the count chapter measures between byBrand and byColor.) Avg time: 72.1 µs (≈ 2× Query 2's 37.2 µs — the extra sum-field hash work on each layer dominates).

Proof display (GroveDBProof::Display):

Expand to see the structured proof (6 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
                  1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
                      lower_layers: {
                        tip => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
                              1: Push(KVHash(HASH[a17138f666ae4ab19c1c2930ad94c7e29a6a82789398fc4d3a0b053d3499ac68]))
                              2: Parent
                              3: Push(KVValueHash(sentAt, ProvableSumTree(800000000000ffff, 550000), HASH[3b3f5bf4e079c639895d84f8c5003fe135ccb46bf81b29fd3bdf415cddffe45c]))
                              4: Child)
                            lower_layers: {
                              sentAt => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[0a9e4f85b317569ed34bb8821fb5a7e4357e3a070413235d92ccfc09c4aae58f]))
                                    1: Push(KVHashSum(HASH[be99a0622f1400aaa04dc460566babac9c1a4955b3eeb1c5780dfd46ec716222], 360430))
                                    2: Parent
                                    3: Push(Hash(HASH[e8c7c1c4fbe14e2d5cc9e460788baad347ea50eed38e3bf0316288d3aa38c2d4]))
                                    4: Push(KVHashSum(HASH[053e0adbc34321342c734571ab822b3c9e2fd5aa2efc267f4e09abd69670dfad], 180214))
                                    5: Parent
                                    6: Push(Hash(HASH[b49dc66868027c23f0e5bfde974330d67f077636fa2f1a6c69d96a7db380e4f2]))
                                    7: Push(KVHashSum(HASH[86cb3966ee86191ce18f754b3c8492f9ad1929c9e540da193cf63899fd311d3b], 5622))
                                    8: Parent
                                    9: Push(Hash(HASH[41bf237b6f834b4271e191423907567fd3144d69bf6c75fd8d475807928063ee]))
                                    10: Push(KVHashSum(HASH[f1600f73cf84dc6ae4de0057887f1fe60a373073863ee352ebede717795b1c91], 2810))
                                    11: Parent
                                    12: Push(Hash(HASH[0765b948c17ac92fc10c83b4480283371bcd9c3a4745e62242d842f2f9125fe4]))
                                    13: Push(KVHashSum(HASH[a1d9da3b90a69411dcbba3934f3dd86daeeec542373c261ad36ddb072f0c61b8], 688))
                                    14: Parent
                                    15: Push(Hash(HASH[60ae1d862ec2ba4f63e041741bb93a539882e9620302cfc343ac5d0f339d8e21]))
                                    16: Push(KVHashSum(HASH[133fc6dec68084db2d8c5273011a000568f49447bbeeb1c4500776a42c6de434], 170))
                                    17: Parent
                                    18: Push(KVValueHashFeatureTypeWithChildHash(0x800000000000c350, SumTree(00, 1), HASH[b22d3790d948924dc6311518f2872b53c0590d3cc497d7a653f1ce7b9923e499], ProvableSummedMerkNode(1), HASH[f3c390a0a62080d8a85c55e5c08c01546f0086ae2456fb7f23cf88d7d3d0c44b]))
                                    19: Push(KVHashSum(HASH[4fa59ee6c08136a261a9b4dca592f5df25063b8f190543a9556946da3bf1d4d7], 6))
                                    20: Parent
                                    21: Push(Hash(HASH[6191f6b8bcfac3d6772193061c3b17dbc218a80657b018d8e9455ec571922d71]))
                                    22: Child
                                    23: Push(KVHashSum(HASH[5d412517c8ad8e784679852d2da663cf231d59f6e9f24bee36f26e9e6a1be900], 28))
                                    24: Parent
                                    25: Push(Hash(HASH[7af88dd3f234aa1e3bbf69e4543b63cd2920c16c4a04c7e2bbfef23405b0181c]))
                                    26: Child
                                    27: Push(KVHashSum(HASH[f412f555a36be8dc7b71d933ef60f51a34dcb3995aacf9ca82e6e5e31f2c0601], 70))
                                    28: Parent
                                    29: Push(Hash(HASH[7ece6d26bbaa79713cc72c05be520d699f98e6ffb66187d3d1a98f0de85db1dc]))
                                    30: Child
                                    31: Child
                                    32: Push(KVHashSum(HASH[758bd7f1ade8550bb59935448441bd81421f5c8b1d77da46581f576a326c791f], 348))
                                    33: Parent
                                    34: Push(Hash(HASH[b2b507154c77192abad2aebd28953ab177560b4c1d515c0630821dd0e6d22b67]))
                                    35: Child
                                    36: Child
                                    37: Push(KVHashSum(HASH[f2e039a1921a0d514ca714e7501d00268649420c96036fa4d6f803cb2838cdb7], 1390))
                                    38: Parent
                                    39: Push(Hash(HASH[145cbb78cc936b222cc9e6b06ae684aa47a1bf5479acdb35c92cbc57dc985a37]))
                                    40: Child
                                    41: Child
                                    42: Child
                                    43: Push(KVHashSum(HASH[36cf0712fbcca1239ce0cffb8b262b0a8b06a072015826c7f2a56d2bc7611169], 11262))
                                    44: Parent
                                    45: Push(Hash(HASH[5120fee7ab2fa02978f3dd43b6101dbc9b22de0d45c9e65976f5562a98ba4607]))
                                    46: Child
                                    47: Push(KVHashSum(HASH[ae76541450c5efae715f617db2d63a401da7bc2decaac3720127af252e99fa0d], 22520))
                                    48: Parent
                                    49: Push(Hash(HASH[5758f27a188b7d9a0b8c488374f80031671c662eb12828565146a6117344d5ea]))
                                    50: Child
                                    51: Push(KVHashSum(HASH[c60627bbfc24d9c01e6e7e6d8e79743884916f04904d08ca1d56e17ca8c52989], 45048))
                                    52: Parent
                                    53: Push(Hash(HASH[c7be4432e5634e8c8f2d0a2074b2f72a8f0fea8859a8b5f1bd6dd6556d16bf7e]))
                                    54: Child
                                    55: Push(KVHashSum(HASH[9f79592b39f89b6f6fed13d0fcd2e548d57735169a9e583960678dde3a5f0a05], 90102))
                                    56: Parent
                                    57: Push(Hash(HASH[cc7d10c23ef37f717891e6390c1fa52bf8133b230df6871c8cf85e1fcec2e9b2]))
                                    58: Child
                                    59: Child
                                    60: Child
                                    61: Push(KVHashSum(HASH[ca275ed3d5729250a4a67e2cc90fac909086fac99e386fc90688df2bb01b3eaa], 550000))
                                    62: Parent
                                    63: Push(Hash(HASH[8593bd2bc903da34da11e15f9a649cec37b39487ad59375020adef300a8b7482]))
                                    64: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. Same first four layers as Q1/Q2; at layer 5 (the tip doctype) the descent takes the sentAt branch on op 3 (its value is ProvableSumTree(800000000000ffff, 550000) — a ProvableSumTree whose root-committed sum is 550 000, the full timeline sum) and descends into layer 6, bySentAt. Layer 6 is structurally different from Q2's byRecipient layer: every internal kv op is a KVHashSum (a hash with its subtree's i64 sum), and every opaque subtree commit on the merk-boundary walk carries its own running sum (e.g. op 61's KVHashSum(…, 550000) is the full-tree root-commitment). The terminator on op 18 is KVValueHashFeatureTypeWithChildHash(0x800000000000c350, SumTree(00, 1), …, ProvableSummedMerkNode(1), …) — the ProvableSummedMerkNode(1) feature-type marks it as living inside a ProvableSumTree with its own contributing sum of 1, and SumTree(00, 1) is the per-timestamp value tree (sum_value_or_default = 1, since sentAt = 50 000 was assigned amount = (50000 % 10) + 1 = 1). Verified root hash 95aa7470…71b6.

Diagram: per-layer merk-tree structure

Layers 1–4 are byte-for-byte identical to Q1's. The descent diverges at layer 5 onto sentAt, then layer 6 walks the bySentAt ProvableSumTree — yellow rather than gray for sibling commits because every internal kv carries a sum field.

flowchart TB
  subgraph L5["Layer 5 — tip doctype merk-tree (proof view for `sentAt`)"]
    direction TB
    L5_q["<b>sentAt</b><br/>kv_hash=HASH[3b3f...]<br/>value: <b>ProvableSumTree</b> (root sum=550000)<br/>(descent into bySentAt)"]:::queried
    L5_left["HASH[15fd...]<br/>(left subtree, opaque)"]:::sibling
    L5_kv["KVHash[a171...]<br/>(opaque internal kv: 0x00 or recipient)"]:::sibling
    L5_q --> L5_left
    L5_q --> L5_kv
  end

  subgraph L6["Layer 6 — bySentAt ProvableSumTree merk-tree (TARGET layer)"]
    direction TB
    L6_target["<b>sentAt=50000 (0x800000000000c350)</b><br/>kv_hash=HASH[b22d...]<br/>value: <b>SumTree(00, sum=1)</b><br/>feature: ProvableSummedMerkNode(1)<br/>child_hash=HASH[f3c3...]"]:::target
    L6_pst["Boundary commitments (64 merk ops):<br/>every internal kv is a KVHashSum (hash + subtree sum)<br/>every opaque sibling is a Hash subtree commitment<br/>(both contribute to the per-node sum aggregation<br/>that makes AggregateSumOnRange a single-pass primitive)"]:::pst
    L6_target --> L6_pst
  end

  L5_q -. "value=ProvableSumTree(merk_root[bySentAt], sum=550000)" .-> L6_target

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The yellow pst node summarizes the 64-op ProvableSumTree merk descent: every kv-and-hash sibling carries an i64 sum field so the verifier can recompute the parent's sum, not just its hash. That extra hash work is what makes Q3 a ~2× longer wall-clock proof than Q2 even though the descent depth is identical.

Query 4 — Compound Equal-only (byRecipientTime)

select       = SUM(amount)
where        = recipient == "recipient_050" AND sentAt == 50000
sum_property = "amount"
prove        = true

Path query:

path:         ["@", contract_id, 0x01, "tip", "recipient", "recipient_050", "sentAt"]
query items:  [Key(serialize_value_for_key("sentAt", 50000))]

Verified element:

path:        ["@", contract_id, 0x01, "tip", "recipient", "recipient_050", "sentAt"]
key:         serialize_value_for_key("sentAt", 50000)
element:     SumTree { sum_value_or_default: 0 }   (no tip at this recipient×sentAt pair)

Proof size: 1 937 bytes. Avg time: 71.8 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (8 layers: 6 path layers + 2 byRecipientTime continuation layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
                  1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
                      lower_layers: {
                        tip => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
                              1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
                              2: Parent
                              3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
                              4: Child)
                            lower_layers: {
                              recipient => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[b8804ea3f7998def339db7cadd6a6b29ba5bfadbdfa15b67802ef52e42adb68e]))
                                    1: Push(KVHash(HASH[3056a7c2daf102d31d0f461d45e661303b1562311971debbf15f40938727a074]))
                                    2: Parent
                                    3: Push(Hash(HASH[c406363887b632d071f2ee6ed83df682aace9a5456997aa4f4b944576476fbb6]))
                                    4: Push(KVHash(HASH[6b8ef1df5ba1299cd5b605dc7642be2ffb8356fd7f51c3a0498b6b657cddeebc]))
                                    5: Parent
                                    6: Push(Hash(HASH[347abc0b69e504bc619e9549e509c21be3c0ad1c9e03d8fb34bb5ed07e5cd26f]))
                                    7: Push(KVHash(HASH[ff9b5006130777d589b77e6f5bdda0f86879daace181cdc8e3d97bc3718e0a48]))
                                    8: Parent
                                    9: Push(KVValueHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[264c6aac1a1acd864832a6f25ac42c626afb95dc79ac8c233d2b05c6df048f1c]))
                                    10: Child
                                    11: Push(KVHash(HASH[5159e1ccad8c4ed1bbf7f52ec34e9ab4ee108559194e39b1af78cf42866b32cc]))
                                    12: Parent
                                    13: Push(Hash(HASH[4317777613ab1a0d6966670195ccce83dee1031fd37013c9ce625c016d1d6d17]))
                                    14: Child
                                    15: Push(KVHash(HASH[24f6646d8b75a6a428d05589c1cc1e80f1e52900924710ff6eafe9da0edc73d0]))
                                    16: Parent
                                    17: Push(Hash(HASH[14f8282bd0b5d9a8e7e471d3cfd215f02f5be2cfbf837c5e938c2871a2eddcb9]))
                                    18: Child
                                    19: Child
                                    20: Child
                                    21: Push(KVHash(HASH[cea6360efbf77b38d2ea206d508ea03c0d92fe7c02c5d9176aa322e8263a3acc]))
                                    22: Parent
                                    23: Push(Hash(HASH[209f3325816d5f02a1647311a4f1d9c68fcbacf1540be9b5ae65413f58ca3b26]))
                                    24: Child)
                                  lower_layers: {
                                    0x0000000000000032ffffffffffffffcd00000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[ad21cf215639bca21e163333196de5b1774e0e91bf266a323c3a0b78c8cf7998]))
                                          1: Push(KVValueHash(sentAt, NotSummed(ProvableSumTree(800000000000c7ce, 1000)), HASH[fbe3df47d0e3ee0dbdb9a1491fc05d93ae26f8cb914fb240a42e9989a3752cbc]))
                                          2: Parent)
                                        lower_layers: {
                                          sentAt => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[585d1074d4c73df4e60784323fb6e1cd9a45a7f6fed92ca43ac78a1a575efa16]))
                                                1: Push(KVHashSum(HASH[471d8a8c670fb4bc4c30d251a7fb398f5946a076cfd3c734ce7c131a6a922ae7], 511))
                                                2: Parent
                                                3: Push(Hash(HASH[39bb4483b5c84d178863fb016add610ae44876b0450a9c2f75de97fcdd60c8b5]))
                                                4: Push(KVHashSum(HASH[59629e7456a1ef7da12d0d02c0c3501f68e27f92aa737f088b34a05ddebb42e7], 255))
                                                5: Parent
                                                6: Push(Hash(HASH[99363add734e45862d731777f87ca7a018109aa62aa489e3c5a5515dc965678b]))
                                                7: Push(KVHashSum(HASH[542ff7575944b1293665b9a4ea5a5a49945e2d3206280d9cbded80afe404a7de], 127))
                                                8: Parent
                                                9: Push(Hash(HASH[d1b19a359a223351d14e5f21b8fa3e49a012e0b4bb6237c5d73b08bdd9b4d9e8]))
                                                10: Push(KVHashSum(HASH[79ab80617285543900f09137ef8cc42c3f715a6be449c7453d6babceb602c36d], 63))
                                                11: Parent
                                                12: Push(Hash(HASH[497640ee5a5972a73c9484d129a9df3dc17c0e5f06a631d26e824f817706639a]))
                                                13: Push(KVHashSum(HASH[45ee1ce73fc4156e923393f2b7113f22ae3e10b9e74578c46e263c2fb8623af2], 31))
                                                14: Parent
                                                15: Push(Hash(HASH[97daec9f53ac09843034f165faf5410e54882561f3ee345281cf51f0972e29cc]))
                                                16: Push(KVDigestSum(0x800000000000c31e, HASH[110023b15d77b39d67044847913b994adf94cdb8ecfa5bce2abd4d37d6ac68c7], 7))
                                                17: Parent
                                                18: Push(KVDigestSum(0x800000000000c382, HASH[1ea127fb3003e8de7bcfbe341b29da4cff8b8c119dfa10d47eaca70fb7a62d54], 1))
                                                19: Push(KVHashSum(HASH[177c272d0a693d73e8543f41adbce6d964874573d9eacb6fa7bb8d8d5eefc4a3], 3))
                                                20: Parent
                                                21: Push(Hash(HASH[aad686ea6ee57db81ad554934281a5b4c383efd1dd2142011b11f2bf73872d7f]))
                                                22: Child
                                                23: Child
                                                24: Push(KVHashSum(HASH[d0c8fd2ef0a5ed9e51c335d6a7967e16bc0eb02c6a4fc8e7e1e86ec599717ede], 15))
                                                25: Parent
                                                26: Push(Hash(HASH[d13f24f322e026b7909002ca6b79b76c71aad9117601366b9bc867d649cb885b]))
                                                27: Child
                                                28: Child
                                                29: Child
                                                30: Child
                                                31: Child
                                                32: Child
                                                33: Push(KVHashSum(HASH[f94972b44e022eca9d291dda5a150bf75a9ab5cf759d4211bf5c8643ffbe7585], 1000))
                                                34: Parent
                                                35: Push(Hash(HASH[de4170da8225c95a1ffc8dbeb08e2a92add3ea7ac2b7972eb786bfdb1bee4207]))
                                                36: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. Q4 is the deepest single-key sum proof in the chapter: 8 LayerProofs because the byRecipientTime descent walks recipient → recipient_050 → sentAt → terminator. Layers 1–5 mirror Q2's descent verbatim (root → @ → contract → 0x01 → tip → recipient). Layer 6 lands on recipient_050 whose value is now plain SumTree(73656e744174, 1000) (no FeatureType wrapper because this proof descends into it rather than terminating there). Layer 7 is recipient_050's continuation merk-tree, a single-key tree whose only entry is the sentAt property name pointing at a NotSummed(ProvableSumTree(…, 1000)) — the NotSummed wrapper is the storage-layer signal that this continuation's sum doesn't propagate up to recipient_050's own SumTree (the parent's 1 000 already aggregates the value-tree references; this continuation is just a sibling index, contributing 0). Layer 8 walks the byRecipientTime ProvableSumTree (37 ops, the same per-node sum-bearing shape as Q3's bySentAt) but does NOT terminate at 0x800000000000c350: the queried sentAt-50000 key is absent under recipient_050 (since recipient_050 owns rows 50, 150, …, 99950, none of which are sentAt = 50000). The proof commits a complete merk path with no terminator op, which verify_query reports as an empty results vec (the verifier-side absence proof). Verified root hash 95aa7470…71b6.

Two property-name descents (recipient, then under recipient_050 the byRecipientTime continuation's sentAt). For this specific filter no row lands at recipient_050 ∧ sentAt = 50000 (recipient_050 owns rows {50, 150, …, 99 950}; sentAt = 50000 is owned by recipient_000), so the verified element is a SumTree with sum_value_or_default = 0 proving absence. The proof is still 1 937 bytes — absence proofs walk the same depth as present-key proofs, just with a different terminator merk node.

Diagram: per-layer merk-tree structure

Layers 1–5 are identical to Q2's; Q4's signature divergence happens at layers 6–8 where the byRecipientTime compound index threads through recipient_050's NotSummed(ProvableSumTree) continuation.

flowchart TB
  subgraph L6["Layer 6 — byRecipient merk-tree (mid-descent, NOT a target)"]
    direction TB
    L6_q["<b>recipient_050</b><br/>kv_hash=HASH[264c...]<br/>value: SumTree(73656e744174, sum=1000)<br/>(descends one more layer)"]:::queried
    L6_boundary["Boundary commitments (24 ops):<br/>same byRecipient path Q2 walks, just<br/>without the FeatureTypeWithChildHash wrapper"]:::sibling
    L6_q --> L6_boundary
  end

  subgraph L7["Layer 7 — recipient_050 continuation merk-tree (single key)"]
    direction TB
    L7_q["<b>sentAt</b><br/>kv_hash=HASH[fbe3...]<br/>value: <b>NotSummed(ProvableSumTree(…, 1000))</b><br/>(NotSummed = continuation, contributes 0 to L6's sum)"]:::queried
    L7_sib["HASH[ad21...]<br/>(left subtree, opaque)"]:::sibling
    L7_q --> L7_sib
  end

  subgraph L8["Layer 8 — byRecipientTime sentAt ProvableSumTree (TARGET layer, absent key)"]
    direction TB
    L8_absent["sentAt=50000 (queried key) — <b>absent</b><br/>(no terminator op committed; recipient_050<br/>owns rows 50, 150, …, 99950 only)"]:::target
    L8_neighbors["Boundary commitments (37 ops):<br/>KVDigestSum siblings at 0x800000000000c31e (sum=7) and<br/>0x800000000000c382 (sum=1) bracket the absent key<br/>+ 17 KVHashSum / Hash subtree commits across the<br/>ProvableSumTree's per-node sum-bearing structure"]:::pst
    L8_absent --> L8_neighbors
  end

  L6_q -. "value=SumTree(merk_root[r050-continuation])" .-> L7_q
  L7_q -. "value=NotSummed(ProvableSumTree(merk_root[byRT.sentAt]))" .-> L8_absent

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The absent-key shape is structurally important: the bracketing siblings 0x800000000000c31e and 0x800000000000c382 (op 16, op 18 in layer 8) are what convince the verifier the queried 0x800000000000c350 doesn't exist between them. The proof commits enough of the merk-tree to recompute the parent root with the gap intact — a non-membership witness in the same proof shape as a membership witness.

Query 5 — In on byRecipient

select       = SUM(amount)
where        = recipient IN ["recipient_000", "recipient_001"]
sum_property = "amount"
prove        = true

Path query (per-In-value point-lookup fan-out):

path:         ["@", contract_id, 0x01, "tip", "recipient"]
query items:  [Key("recipient_000"), Key("recipient_001")]

Verified entries (two SumEntrys under the entries variant):

SumEntry { in_key: None, key: "recipient_000", sum: 1000 }   (amount=1)
SumEntry { in_key: None, key: "recipient_001", sum: 2000 }   (amount=2)

Proof size: 1 168 bytes for k=2. The bench's report_proof_sizes measures the same shape at k=100 across all distinct recipients — 12 064 bytes, scaling roughly linearly with |In values| because each branch's merk descent is independent. The per-branch marginal cost is ≈ 109 bytes (10 880 / 99 marginal branches). Avg time: 42.8 µs (k=2) / 1 720 µs (k=100) — roughly 17 µs of marginal time per added In value, consistent with the per-branch merk-descent cost on byRecipient.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (6 layers, dual-target merk-path at the bottom) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
                  1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
                      lower_layers: {
                        tip => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
                              1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
                              2: Parent
                              3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
                              4: Child)
                            lower_layers: {
                              recipient => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(KVValueHashFeatureTypeWithChildHash(0x0000000000000000ffffffffffffffff00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[2c932396123fe6bae5fa3e4fa42225852e08a2dcf7600991e79fb9347de7506e], BasicMerkNode, HASH[307798135a4d282b0306f8f0a652c99391fb89a193b1f219632c956b31fb1351]))
                                    1: Push(KVValueHashFeatureTypeWithChildHash(0x0000000000000001fffffffffffffffe00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[43c65eed82b8e5d08b7fea673bf120a82332adbffb8582e0f8a3205190fab720], BasicMerkNode, HASH[8723320b1000d7ca62f0057154cc9ce7e79b31ceeec9e5ea16db382efdf6a81f]))
                                    2: Parent
                                    3: Push(Hash(HASH[e0442e4426d554662f91691fba1bec4c0eddac54662493b3964b08538fea33fe]))
                                    4: Child
                                    5: Push(KVHash(HASH[2f5f739085124d63217a3b5c8276da943d0901c1fc482010e5a50c2f73b36549]))
                                    6: Parent
                                    7: Push(Hash(HASH[93e907e7310d06719783bd32af4653a620b3d2f46c9b54bf1d5edb45928866f7]))
                                    8: Child
                                    9: Push(KVHash(HASH[ca38e3d00da85a2b31477b69b9f71e0c5535a1d9c1bca5b54c875444f7a0bad9]))
                                    10: Parent
                                    11: Push(Hash(HASH[274c7bf51d753c5ed1cc44997d2ed956e30c6574609c165b073a2c6f75e9af58]))
                                    12: Child
                                    13: Push(KVHash(HASH[db46825673ab3057db34992612299ceeca04f94152ac048f647cc92afbd1d771]))
                                    14: Parent
                                    15: Push(Hash(HASH[af91e1e902af6a64f73a4f4abd49ccb5893029568e23ec2e73452dd1fb58940a]))
                                    16: Child
                                    17: Push(KVHash(HASH[3056a7c2daf102d31d0f461d45e661303b1562311971debbf15f40938727a074]))
                                    18: Parent
                                    19: Push(Hash(HASH[22b84dcbedf3b829ad645fd68fa5eb1c59dadb2ab3cd098efd998ca247902c95]))
                                    20: Child
                                    21: Push(KVHash(HASH[cea6360efbf77b38d2ea206d508ea03c0d92fe7c02c5d9176aa322e8263a3acc]))
                                    22: Parent
                                    23: Push(Hash(HASH[209f3325816d5f02a1647311a4f1d9c68fcbacf1540be9b5ae65413f58ca3b26]))
                                    24: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. Same 5-layer descent as Q2 (root → @ → contract → 0x01 → tip → recipient); the divergence is at the bottom layer where the proof reveals two KVValueHashFeatureTypeWithChildHash terminator ops back-to-back (op 0 for recipient_000, op 1 for recipient_001), each carrying its own SumTree(73656e744174, …) with per-recipient sum (1000 and 2000 respectively). The 24-op boundary walk that surrounds them is the same shape as Q2's — proving the two queried keys' positions inside byRecipient's ~100-entry merk-tree by committing the bracketing opaque siblings. Verified entries via verify_query are two (path, key, SumTree { sum_value_or_default: … }) rows, one per In branch. Verified root hash 95aa7470…71b6.

The entries variant carries in_key: None because the In is itself the terminator (no compound-prefix prefix). Compare with Query 6, which has the same property-and-position In but on a rangeSummable index.

Diagram: per-layer merk-tree structure

Layers 1–5 mirror Q2's prefix exactly. The divergence is at layer 6: where Q2 had one cyan target, Q5 has two — and the boundary commits between them are shared by both descents.

flowchart TB
  subgraph L6["Layer 6 — byRecipient merk-tree (DUAL TARGET layer)"]
    direction TB
    L6_t0["<b>recipient_000</b><br/>kv_hash=HASH[2c93...]<br/>value: <b>SumTree sum=1000</b><br/>child_hash=HASH[3077...]"]:::target
    L6_t1["<b>recipient_001</b><br/>kv_hash=HASH[43c6...]<br/>value: <b>SumTree sum=2000</b><br/>child_hash=HASH[8723...]"]:::target
    L6_boundary["Boundary commitments (~22 ops shared across<br/>both descents):<br/>6 KVHash internal siblings + 6 Hash subtree commits<br/>(prove recipient_000/recipient_001 are adjacent<br/>in byRecipient's binary merk tree)"]:::sibling
    L6_t0 --> L6_boundary
    L6_t1 --> L6_boundary
  end

  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

Per-branch byte cost is sub-linear because the two targets share most of the merk-boundary walk (only their kv-hashes themselves are per-branch; siblings amortize). At k=100 with all 100 recipients covered by 2 In branches each, every byRecipient entry becomes a target and only the merk root's two-level boundary stays opaque — the most efficient byte-per-key shape an In on byRecipient can hit.

Query 6 — In on bySentAt (RangeSummable)

select       = SUM(amount)
where        = sentAt IN [0, 1]
sum_property = "amount"
prove        = true

Path query:

path:         ["@", contract_id, 0x01, "tip", "sentAt"]
query items:  [Key(serialize_value_for_key("sentAt", 0)),
               Key(serialize_value_for_key("sentAt", 1))]

Verified entries:

SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 0), sum: 1 }
SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 1), sum: 2 }

(Per the bench's amount = (row % 10) + 1 schedule, row 0 has amount = 1 and row 1 has amount = 2.)

Proof size: 1 756 bytes for k=2; 9 784 bytes at k=100. Per-branch marginal cost ≈ 81 bytes — somewhat less than Query 5's 109 bytes per branch, because the bySentAt subtree's distinct keys are dense integers (one per row) and share more merk-path prefix than byRecipient's hash-derived keys. Avg time: 81.2 µs (k=2) / 2 008 µs (k=100) — slightly higher per-branch time than Q5 despite smaller per-branch bytes; the ProvableSumTree's per-node sum-field hash work eats most of the prefix-sharing savings.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (6 layers, dual-target in a ProvableSumTree) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
                  1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
                      lower_layers: {
                        tip => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
                              1: Push(KVHash(HASH[a17138f666ae4ab19c1c2930ad94c7e29a6a82789398fc4d3a0b053d3499ac68]))
                              2: Parent
                              3: Push(KVValueHash(sentAt, ProvableSumTree(800000000000ffff, 550000), HASH[3b3f5bf4e079c639895d84f8c5003fe135ccb46bf81b29fd3bdf415cddffe45c]))
                              4: Child)
                            lower_layers: {
                              sentAt => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000000000, SumTree(00, 1), HASH[7196506382214367916d1e1e0cff254b14f89815f007840414956a077dd53148], ProvableSummedMerkNode(1), HASH[ef96a09b8f07fdbb7d17c3e86a7bdc954aae31c8e2a51900ed2b7d92a6ab9953]))
                                    1: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000000001, SumTree(00, 2), HASH[6b4e9f17f2121cb658ec356b970a91b5ad79032a57e304a3507f63d813f2da8d], ProvableSummedMerkNode(6), HASH[8abb1cbf9ae45b42ff519e6f187bd4fbadee93ab64afef59089aaa94ed46b502]))
                                    2: Parent
                                    3: Push(Hash(HASH[e03fc8c98c17d3574f4fe6a822916775844555dd63276f2e7289f11ec5aa043d]))
                                    4: Child
                                    5: Push(KVHashSum(HASH[4eb5727c0eb7018148785596abb4767ae93002c4ab76b4f565d31b5235b55a39], 28))
                                    6: Parent
                                    7: Push(Hash(HASH[843e47cd7806eb5309f780a529fe26b245b22cedd46ee3ea62ac23ab6767d59b]))
                                    8: Child
                                    9: Push(KVHashSum(HASH[af802589532730912220f760f96036c67bb64c60dc4cdc468d1c978cb78972fe], 70))
                                    10: Parent
                                    11: Push(Hash(HASH[9adc74e0427bccca23c5adbd648610c340edec52566da65d51f42bcfb1f78f93]))
                                    12: Child
                                    13: Push(KVHashSum(HASH[deb2eb99281aa70288d4accac22faba3681fcdd0258bd5c6d06a2b7ee6cb6624], 166))
                                    14: Parent
                                    15: Push(Hash(HASH[7cd9232374df6e9e98d64e199876edcc91d1016b6541acb867016bcdea24b7d3]))
                                    16: Child
                                    17: Push(KVHashSum(HASH[2acdf1c54d23ebf0874620d776ec72326bc88300a1fd724fbc24ed8c2d921b32], 336))
                                    18: Parent
                                    19: Push(Hash(HASH[2f62bfca826600367f5802e4ff86ac652a225e87dbddcea3d7b38fc922d18b59]))
                                    20: Child
                                    21: Push(KVHashSum(HASH[45f178d25bec1d9b81115e2f023db10ea0117d3c508409d61c2390a9244484e1], 688))
                                    22: Parent
                                    23: Push(Hash(HASH[f983a5b23027388a35323554689f394267399ad40071c6feaeeebd415d5e22d7]))
                                    24: Child
                                    25: Push(KVHashSum(HASH[fe3e84de3d4be2e935b0408a0ef427e277292d7675e88fabf0b9a71346f1ad7d], 1390))
                                    26: Parent
                                    27: Push(Hash(HASH[4552635b13dd23588d3fcdf3382a0a7fe0600f73d118f28b27507bbd7113842b]))
                                    28: Child
                                    29: Push(KVHashSum(HASH[d49e4a23ef475529c6b7116795a0f587d3dd191a6c51b42641e277623afa03e5], 2806))
                                    30: Parent
                                    31: Push(Hash(HASH[27cd36370fc15a994480197224b944ba24f775dcb6b34f28b74f25c0dd0582b9]))
                                    32: Child
                                    33: Push(KVHashSum(HASH[5e4ce1f7f36aca9f81c33b45d945c84488fdf1915402b07914cbbd0f35917af1], 5616))
                                    34: Parent
                                    35: Push(Hash(HASH[1ccfd53d7a30abdd7171a5a0ce63d22c7266ed0879e8353463fefb271112ba13]))
                                    36: Child
                                    37: Push(KVHashSum(HASH[26794659891cef5a5e0edf7947bbca061307b947783887b25ea3323277211af3], 11248))
                                    38: Parent
                                    39: Push(Hash(HASH[1a7593e8e1d66a041bf127649256d0b836368c3f41ec58fa714fbaceb6b5e9fb]))
                                    40: Child
                                    41: Push(KVHashSum(HASH[c42b2154308418eabefedc2b828dbc599009fbb2b0965f057480d8f9cf1e096d], 22510))
                                    42: Parent
                                    43: Push(Hash(HASH[3b81157cc8121f546f746342f60a4d6b325b965874342c094490f1d21f0cdef3]))
                                    44: Child
                                    45: Push(KVHashSum(HASH[2d2bd624f09591fdecf3068dbe47314ff108f3cb4f99482ebd0abbb9c947e257], 45046))
                                    46: Parent
                                    47: Push(Hash(HASH[71958692ac25e70f1d1df3fb5b85c2b90eac37ae745be90b931992b2f9dbfa94]))
                                    48: Child
                                    49: Push(KVHashSum(HASH[150a4c0a44464b24dbde8a45106e370fc189b31676bfd6284763923381a6d434], 90096))
                                    50: Parent
                                    51: Push(Hash(HASH[5e468c0f9805ee381d4c59dd961adc8dee1ab003834aa17fadd9f51411804147]))
                                    52: Child
                                    53: Push(KVHashSum(HASH[a133f55e1f33e2b3a2a78f9f89c3aff8fd06f0ad25254490a42beef2c130e9dd], 180208))
                                    54: Parent
                                    55: Push(Hash(HASH[620ccd98c998cb3a503ee12098a6617743a99ffeb644069927ce7ebd8b70f76a]))
                                    56: Child
                                    57: Push(KVHashSum(HASH[be99a0622f1400aaa04dc460566babac9c1a4955b3eeb1c5780dfd46ec716222], 360430))
                                    58: Parent
                                    59: Push(Hash(HASH[f404bfecb3e178393455544d654039f0cb77c0e0d918cc00d85d22d2707cbf0d]))
                                    60: Child
                                    61: Push(KVHashSum(HASH[ca275ed3d5729250a4a67e2cc90fac909086fac99e386fc90688df2bb01b3eaa], 550000))
                                    62: Parent
                                    63: Push(Hash(HASH[8593bd2bc903da34da11e15f9a649cec37b39487ad59375020adef300a8b7482]))
                                    64: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. Same first 5 layers as Q3 (the descent to bySentAt). The bottom layer reveals both queried timestamps as adjacent KVValueHashFeatureTypeWithChildHash terminator ops: op 0 for sentAt=0 (SumTree(00, 1), amount = 1) and op 1 for sentAt=1 (SumTree(00, 2), amount = 2). Each terminator's feature-type marker is ProvableSummedMerkNode(n) with n matching the per-node committed sum — this is the per-node-sum machinery that makes the surrounding bySentAt tree a ProvableSumTree. The 62-op boundary walk surrounds them with the same per-node sum-bearing structure as Q3, except now two leaves are revealed instead of one. Verified root hash 95aa7470…71b6.

Same structural shape as Query 5 — the difference is the property-name subtree is a ProvableSumTree rather than a NormalTree, so each descent step on the In branches carries a sum field. For point lookups that's pure overhead (we don't use the per-node sums); the payoff lands on Query 7.

Diagram: per-layer merk-tree structure

Layers 1–5 mirror Q3's; the difference is at layer 6 where two adjacent leaves are revealed inside the ProvableSumTree.

flowchart TB
  subgraph L6["Layer 6 — bySentAt ProvableSumTree merk-tree (DUAL TARGET layer)"]
    direction TB
    L6_t0["<b>sentAt=0 (0x8000000000000000)</b><br/>kv_hash=HASH[7196...]<br/>value: <b>SumTree(00, sum=1)</b><br/>feature: ProvableSummedMerkNode(1)<br/>child_hash=HASH[ef96...]"]:::target
    L6_t1["<b>sentAt=1 (0x8000000000000001)</b><br/>kv_hash=HASH[6b4e...]<br/>value: <b>SumTree(00, sum=2)</b><br/>feature: ProvableSummedMerkNode(6)<br/>child_hash=HASH[8abb...]"]:::target
    L6_pst["Boundary commitments (62 merk ops):<br/>every internal kv is a KVHashSum (hash + sum)<br/>every opaque sibling is a Hash subtree commitment<br/>(running sums on each boundary node:<br/>28 → 70 → 166 → 336 → 688 → 1390 → 2806 → 5616 → …<br/>→ 550000 at the merk root, the full timeline sum)"]:::pst
    L6_t0 --> L6_pst
    L6_t1 --> L6_pst
  end

  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The per-node sum chain 28 → 70 → 166 → … → 550000 is what makes this also a sum-bearing path: a verifier who walked the same descent for an AggregateSumOnRange instead would extract a running sum from the same KVHashSum ops rather than reading individual leaves. That's how Q3 / Q6 (point lookups on a ProvableSumTree) and Q7 (range collapse on the same tree) share infrastructure and differ only in which nodes the proof reveals as terminators.

Query 7 — Range Query (AggregateSumOnRange)

select       = SUM(amount)
where        = sentAt > 50000
sum_property = "amount"
prove        = true

Path query:

path:         ["@", contract_id, 0x01, "tip", "sentAt"]
query items:  AggregateSumOnRange(RangeAfter(serialize_value_for_key("sentAt", 50000)..))

Verified result (returned by GroveDb::verify_aggregate_sum_query):

(root_hash, sum) where sum = 274 999

49 999 rows have sentAt > 50 000 (the half-open (50000, ∞) range excludes sentAt = 50000 itself). Per the fixture's amount = (row % 10) + 1 schedule, rows 50 001..99 999 cycle through [2, 3, 4, 5, 6, 7, 8, 9, 10, 1] for 4 999 full cycles + a 9-element tail. The sum works out to 4 999 × 55 + (2 + 3 + … + 10) = 274 945 + 54 = 274 999, which matches the verified value byte-for-byte.

Proof size: 3 102 bytes. Avg time: 102.0 µs. Verifier root hash: 95aa74708738c5254c706bbca3245b520022aad949674822218094bca15671b6.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (6 layers, AggregateSumOnRange collapse at the bottom) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
                  1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
                      lower_layers: {
                        tip => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
                              1: Push(KVHash(HASH[a17138f666ae4ab19c1c2930ad94c7e29a6a82789398fc4d3a0b053d3499ac68]))
                              2: Parent
                              3: Push(KVValueHash(sentAt, ProvableSumTree(800000000000ffff, 550000), HASH[3b3f5bf4e079c639895d84f8c5003fe135ccb46bf81b29fd3bdf415cddffe45c]))
                              4: Child)
                            lower_layers: {
                              sentAt => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(HashWithSum(kv_hash=HASH[a133f55e1f33e2b3a2a78f9f89c3aff8fd06f0ad25254490a42beef2c130e9dd], left=HASH[d39ec024ef5f61890206e3c6d29638a77b94159869d3f7f2ff1941f2f5e87e25], right=HASH[620ccd98c998cb3a503ee12098a6617743a99ffeb644069927ce7ebd8b70f76a], sum=180208))
                                    1: Push(KVDigestSum(0x8000000000007fff, HASH[3e1b6b140f2f162e572401ab79584a57b7e6bd9aef2a3a11ffc94f983ed1e29e], 360430))
                                    2: Parent
                                    3: Push(HashWithSum(kv_hash=HASH[fe76ca504e6930ba89fde3579cc12e8b087398131edac88ced34c92dc89ad94b], left=HASH[f9636ca299ccd678c7e680dfee0ee20cfc97e7205ef085c0ca543f28b1cb5153], right=HASH[dd2ed3a63ec9da0bc56beae8bd1ce63e3b5873e4ae91105172a72c35ffab89d5], sum=90110))
                                    4: Push(KVDigestSum(0x800000000000bfff, HASH[85990c6978e9933c4c27bfebd6ad8b14293dc3dba72803a7d6052dbeb40fef62], 180214))
                                    5: Parent
                                    6: Push(HashWithSum(kv_hash=HASH[9464d86ed7482bea742921278b684b6b448c0c25353858d1d84d24bb99638373], left=HASH[1108c69f23ffac356496bc7bce37d06859d70fb1f4b8cbb7a1d87980919d4a61], right=HASH[680cdc0f17073459247d5782e373245d2247471f6fbfb600a33f71e8a1ab5480], sum=2808))
                                    7: Push(KVDigestSum(0x800000000000c1ff, HASH[04751262de120fb4c72492011ca10e081eb327ef49262fd03101d51c2f4649e6], 5622))
                                    8: Parent
                                    9: Push(HashWithSum(kv_hash=HASH[4c90aeb3a471fc6dd7f702c31133add5bacf8ad1e463d38b31bc9bee8a7b9c06], left=HASH[b8bde476ff8cb68fd463ba2834b30e2715c3bc6f95fa7b9cc93962b91aa6e6a5], right=HASH[63e03a72eed0862e13fedd63be7e1db08f66ef141a3d6071e38c66bc08504374], sum=1410))
                                    10: Push(KVDigestSum(0x800000000000c2ff, HASH[83fdac1310ddca8772a2f40e5cf1de0834d48ea2ac41832a9bb0893d8cf96464], 2810))
                                    11: Parent
                                    12: Push(HashWithSum(kv_hash=HASH[b7afd069ba319e087f24a7d84c05eeaee3676f751c8af09f62cda8966811ea3b], left=HASH[dc4cef21a317ba562e12fcd541a9f645d027ac9323e69868b82cf36625afef43], right=HASH[051f312ccfcbb07b2b99e78636b11050724c2a573ef8c257e54a3130c5501f00], sum=336))
                                    13: Push(KVDigestSum(0x800000000000c33f, HASH[9a94706e02ff4e42f2d7beb30a0f7747b5b31503a25f3c82be7136ef41f39433], 688))
                                    14: Parent
                                    15: Push(HashWithSum(kv_hash=HASH[4628fb01275e90713dacc4cb940b5c7ca5bf43690f1aee6a34bdf51114f016ef], left=HASH[11160330576883ac71e4842bc0ad1c73d28088b2ea54406c3f519e820ed5556f], right=HASH[73214eec9c3e28b1dd0e4b930e134c536d484f7ef3b64144e59a178c1bc375e8], sum=90))
                                    16: Push(KVDigestSum(0x800000000000c34f, HASH[c1ec8175bf1c39fc6543d151ee1d0f3663b80511139e486c5e21ef248cd291bd], 170))
                                    17: Parent
                                    18: Push(KVDigestSum(0x800000000000c350, HASH[b22d3790d948924dc6311518f2872b53c0590d3cc497d7a653f1ce7b9923e499], 1))
                                    19: Push(KVDigestSum(0x800000000000c351, HASH[af12fe5f400e96dc6517b898330583638bd8fe091ff8ee989a43b10848fe5f60], 6))
                                    20: Parent
                                    21: Push(HashWithSum(kv_hash=HASH[3fbfe732d4eb60c1611bf1658d092de8767129cb7a5bb688b02251d134039e11], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], sum=3))
                                    22: Child
                                    23: Push(KVDigestSum(0x800000000000c353, HASH[00536d93be0133a750702ef0ab5207da8c246fbb9ea731a74db9ac94884e919a], 28))
                                    24: Parent
                                    25: Push(HashWithSum(kv_hash=HASH[5f32751618bcf16785ecfd6e051158b4ebd6ed43432ffe44c4f4d26abf91524e], left=HASH[8ee930cb026de29ec8d652bba6b0644c7aed8bedec2da781805635aaeb61453c], right=HASH[d3d6c02c771eeac87ea43855db8535bb5f7e380046eb0089c7a52856bf120475], sum=18))
                                    26: Child
                                    27: Push(KVDigestSum(0x800000000000c357, HASH[c43d06166234e25f98140da5c573120a4ba81679559bb1876a71efbcabbbe459], 70))
                                    28: Parent
                                    29: Push(HashWithSum(kv_hash=HASH[cee1be881b3ee6049936fd08f6d6080d30209d400a009b8007a99be89c96002d], left=HASH[9f25c5bd27c6ec98ef1f74e31413413c76dd011377a07775817dd9d4f31bd95f], right=HASH[6896b09cfa0768780de9464637f284406ddc7c45e47f562518bf093876aa9a0f], sum=34))
                                    30: Child
                                    31: Child
                                    32: Push(KVDigestSum(0x800000000000c35f, HASH[0bee916b5ca009e480f594b34b4ff0023e003a520b9ae625beb84c53ab3bc93b], 348))
                                    33: Parent
                                    34: Push(HashWithSum(kv_hash=HASH[4ca53cefcf67c600b1c36334925073e1fcb1a3500dbc558428cb8476de92ba37], left=HASH[0f3cc2f97ad17f6ea81ea17d915947fdd47e4e20ccd1d2ff4bd4b81a22515938], right=HASH[4796a3bc0d36b5277ace3d37c112f6f5b6cb5945b1d5cc72bf7efec7bf42d987], sum=172))
                                    35: Child
                                    36: Child
                                    37: Push(KVDigestSum(0x800000000000c37f, HASH[fcb1700a1b6be13cd9f3c3d80abde188701aa1d01b25c24929588a2f858eae9c], 1390))
                                    38: Parent
                                    39: Push(HashWithSum(kv_hash=HASH[0e51a96e8e17135a01dce39490c7f55c49024de520f2870f84eedce77a65ecbb], left=HASH[215439487b3dcef7c0186e740ff1f2da27f8973466615691d6c2dc5db2134417], right=HASH[4911eb7861bbc7ea19c572eb9b76900bc2e0f80fbf178c131b30f6ca8969852a], sum=694))
                                    40: Child
                                    41: Child
                                    42: Child
                                    43: Push(KVDigestSum(0x800000000000c3ff, HASH[c875d92f9ae9adb882f612df44549ea07539a0d25fa915f7831a9d64993627a1], 11262))
                                    44: Parent
                                    45: Push(HashWithSum(kv_hash=HASH[3d4a96f576bd6fa7bf10947e8b6c3884306d374d499ab1f4b510d1d13c2b718b], left=HASH[9c4dc81fa4334b80fb36993450bc59dd16d9eb430e89a55e22678dd7be2c290b], right=HASH[db362866db8cdd17bc56f89f9f920895122e4bcb41b8897bac822d8895c520cc], sum=5634))
                                    46: Child
                                    47: Push(KVDigestSum(0x800000000000c7ff, HASH[277f41ec61fbf2c324169b989c07e1bbae573df54f10c0dce1220daf0ccd9b7d], 22520))
                                    48: Parent
                                    49: Push(HashWithSum(kv_hash=HASH[eeae2cee7b1a1141fbda7f7bce1062e969781102c4387ac4075e5702e7c1f608], left=HASH[9b7d2fd9115f3e799edc4c8bbfe0bca3b29ef9adad6c179984cbe2f632de949b], right=HASH[be02ecdf09c0b45fcd3246cb641fa7afa97bf0b1d10963aa337b93ead23cd74d], sum=11248))
                                    50: Child
                                    51: Push(KVDigestSum(0x800000000000cfff, HASH[e7d6b0ffd061d780d54d7d86cf76c401f35df152d58a362fe2129128aeb75fcd], 45048))
                                    52: Parent
                                    53: Push(HashWithSum(kv_hash=HASH[a641b281dca0faa95010eb5705db941b38c8008b38669c14e33ead3b9c341cf3], left=HASH[ff32269c72ee6af174b05a20db06bf57d766b05684208ff2cf676d0fa5e4b9d8], right=HASH[283d35cdf0761e087db90a9e8dfc9f816db0d3782125e8a3087f3e058b4a5ebb], sum=22520))
                                    54: Child
                                    55: Push(KVDigestSum(0x800000000000dfff, HASH[f3cd6aa655507ab09ee96baa7e1f951324ca052bfc39f14bd1f9443a6b8fc082], 90102))
                                    56: Parent
                                    57: Push(HashWithSum(kv_hash=HASH[5a331ce6ce57978d72fd13599ce6cb2e850c68c206e61941cfd374d6de22c514], left=HASH[b0acdc1413331ec55ef36da32fb0906e6dbc800d9ddb12041ab8ff9370d0bcf2], right=HASH[2753ae0214b4f860dc66dd4644f673f5fd48c13e483bd7088bbbcdac9774c5a0], sum=45050))
                                    58: Child
                                    59: Child
                                    60: Child
                                    61: Push(KVDigestSum(0x800000000000ffff, HASH[3326597e4515f458511d3ed24b5e50ce0b085821e1825d772fc13b500c0250d5], 550000))
                                    62: Parent
                                    63: Push(HashWithSum(kv_hash=HASH[576fea2411d44fafd6be01500f5fcca777ed0233b0d64577631c32f60119beed], left=HASH[f1119f240cf23919e3575090cb4b8999d5b9c6f58b24e0b20ac2b1837b9ec16c], right=HASH[00f34e547329b810797b6c891da04ac6bca733adea73b53eb0696acac942d413], sum=189564))
                                    64: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. Layers 1–5 mirror Q3/Q6's bySentAt descent. The bottom layer (layer 6) is structurally distinct from every previous query: it contains no KVValueHashFeatureTypeWithChildHash terminator ops at all. Instead the proof reveals only the merk-boundary structure of the range (HashWithSum for opaque subtrees that the range straddles, KVDigestSum for boundary kvs whose subtree-sums the verifier needs to sum-up). The verifier's range collapse works by traversing the merk boundary and accumulating the sum fields on HashWithSum / KVDigestSum ops for nodes the range covers, producing a single aggregate i64 = 274 999 along with the recomputed root hash. The op-numbered running sums in the proof above trace the merk's binary descent: op 0's sum=180208 is the merk root's left subtree (= 180 214 minus the boundary kv at 0x…7fff with sum 360 430 partially adjusted), and op 61's KVDigestSum(0x800000000000ffff, …, 550000) is the full-tree's max-sentinel committing the full timeline sum 550 000. The RangeAfter(serialize_value_for_key("sentAt", 50000)..) query item picks out the right-half subtrees from sentAt = 50 001 forward; their cumulative sum is 274 999. Verified via GroveDb::verify_aggregate_sum_query, root hash 95aa7470…71b6.

This is the headline payoff: 49 999 matched timestamps, zero documents materialized, a single committed sum verified in O(log T) bytes — same proof-size profile as count's Query 7. The proof is slightly larger than Query 8's 2 657 bytes because bySentAt covers ~100 000 distinct values (the full timeline) versus byRecipientTime's 1 000 distinct sentAts per recipient — wider merk-trees mean a deeper descent + more boundary commits in the proof.

Diagram: per-layer merk-tree structure

Layers 1–5 are identical to Q3/Q6. Layer 6 is where this query diverges sharply from the point-lookup queries: no individual leaf is the target. The "target" is the range collapse itself — a single committed sum the verifier extracts by walking the boundary and accumulating sum fields.

flowchart TB
  subgraph L6["Layer 6 — bySentAt ProvableSumTree (RANGE COLLAPSE, no individual target)"]
    direction TB
    L6_collapse["<b>AggregateSumOnRange(sentAt > 50000)</b><br/>verified sum = <b>274 999</b><br/>(49 999 timestamps covered, each contributing<br/>amount = (row % 10) + 1 per the fixture cycle)"]:::target
    L6_boundary["Range-boundary commitments (64 merk ops):<br/>HashWithSum for opaque subtree commits the range straddles<br/>(each carries the subtree's running sum_i64)<br/>+ KVDigestSum at boundary kvs where the range partially covers<br/>(e.g. op 18 KVDigestSum(0x…c350, …, 1) is the floor-exclusive boundary)<br/>+ op 61 KVDigestSum(0x…ffff, …, 550000) is the max-sentinel<br/>full-tree commitment, used by the verifier to validate completeness"]:::pst
    L6_collapse --> L6_boundary
  end

  classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

The cyan node here represents the aggregate output, not a single revealed kv. That's the structural payoff of rangeSummable: true: the merk-tree's per-node i64 sum fields turn a potentially N-leaf walk into a constant-depth boundary walk that produces a single verified sum. Q7's sum = 274 999 is byte-for-byte recoverable from the 64 boundary ops above without ever revealing the 49 999 individual sentAt leaves.

Query 8 — Compound == + Range (byRecipientTime)

select       = SUM(amount)
where        = recipient == "recipient_050" AND sentAt > 50000
sum_property = "amount"
prove        = true

Path query:

path:         ["@", contract_id, 0x01, "tip", "recipient", "recipient_050", "sentAt"]
query items:  AggregateSumOnRange(RangeAfter(serialize_value_for_key("sentAt", 50000)..))

Verified result:

(root_hash, sum) where sum = 500

(Per recipient_050, 500 of their 1 000 tips have sentAt > 50 000. Each contributes amount = 1 per the per-recipient cycle described in the fixture narrative, so the sum is 500 × 1 = 500. For a recipient with amount = 10, the same shape would land 500 × 10 = 5 000.)

Proof size: 2 657 bytes — the only end-to-end-verified AggregateSumOnRange proof on this fixture today, since Query 7 is blocked on the top-level promotion bug. Confirms the carrier path through byRecipientTime's NotSummed(ProvableSumTree) continuation works correctly. Avg time: 91.3 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (8 layers: 6 path layers + 2 byRecipientTime continuation layers, range collapse at the bottom) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
                  1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
                      lower_layers: {
                        tip => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
                              1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
                              2: Parent
                              3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
                              4: Child)
                            lower_layers: {
                              recipient => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[b8804ea3f7998def339db7cadd6a6b29ba5bfadbdfa15b67802ef52e42adb68e]))
                                    1: Push(KVHash(HASH[3056a7c2daf102d31d0f461d45e661303b1562311971debbf15f40938727a074]))
                                    2: Parent
                                    3: Push(Hash(HASH[c406363887b632d071f2ee6ed83df682aace9a5456997aa4f4b944576476fbb6]))
                                    4: Push(KVHash(HASH[6b8ef1df5ba1299cd5b605dc7642be2ffb8356fd7f51c3a0498b6b657cddeebc]))
                                    5: Parent
                                    6: Push(Hash(HASH[347abc0b69e504bc619e9549e509c21be3c0ad1c9e03d8fb34bb5ed07e5cd26f]))
                                    7: Push(KVHash(HASH[ff9b5006130777d589b77e6f5bdda0f86879daace181cdc8e3d97bc3718e0a48]))
                                    8: Parent
                                    9: Push(KVValueHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[264c6aac1a1acd864832a6f25ac42c626afb95dc79ac8c233d2b05c6df048f1c]))
                                    10: Child
                                    11: Push(KVHash(HASH[5159e1ccad8c4ed1bbf7f52ec34e9ab4ee108559194e39b1af78cf42866b32cc]))
                                    12: Parent
                                    13: Push(Hash(HASH[4317777613ab1a0d6966670195ccce83dee1031fd37013c9ce625c016d1d6d17]))
                                    14: Child
                                    15: Push(KVHash(HASH[24f6646d8b75a6a428d05589c1cc1e80f1e52900924710ff6eafe9da0edc73d0]))
                                    16: Parent
                                    17: Push(Hash(HASH[14f8282bd0b5d9a8e7e471d3cfd215f02f5be2cfbf837c5e938c2871a2eddcb9]))
                                    18: Child
                                    19: Child
                                    20: Child
                                    21: Push(KVHash(HASH[cea6360efbf77b38d2ea206d508ea03c0d92fe7c02c5d9176aa322e8263a3acc]))
                                    22: Parent
                                    23: Push(Hash(HASH[209f3325816d5f02a1647311a4f1d9c68fcbacf1540be9b5ae65413f58ca3b26]))
                                    24: Child)
                                  lower_layers: {
                                    0x0000000000000032ffffffffffffffcd00000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[ad21cf215639bca21e163333196de5b1774e0e91bf266a323c3a0b78c8cf7998]))
                                          1: Push(KVValueHash(sentAt, NotSummed(ProvableSumTree(800000000000c7ce, 1000)), HASH[fbe3df47d0e3ee0dbdb9a1491fc05d93ae26f8cb914fb240a42e9989a3752cbc]))
                                          2: Parent)
                                        lower_layers: {
                                          sentAt => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithSum(kv_hash=HASH[8a0594fb82ff688d061192d70597c544bc56c18badc2ba8e3ce83372adefa71a], left=HASH[04ea853c427ca0c50abed8b5e6349be94cf55f8988f1e6f314d5b15db57fc4e4], right=HASH[6c6832bf2d87daad24055a9195deeadfe50d4c1b816c79b1bbc6ec212ff94d62], sum=255))
                                                1: Push(KVDigestSum(0x80000000000063ce, HASH[7e46f009482a37f9e01a1a251291ae5bad2d1ffd6e04f1f88b44853c40373401], 511))
                                                2: Parent
                                                3: Push(HashWithSum(kv_hash=HASH[f9ac98fc04f30b63ec49c758c3d8fe7f3657182dab7d6255271d3540c521c3e1], left=HASH[c76899568ea6f6268413bd5c0096ea1a822c60c0e8e8d2ec5738b56e6a4cefed], right=HASH[953541994578d66b8d0d3ac32bef836f3c1d1a0f7a1e55cf99c5312df9f170c4], sum=127))
                                                4: Push(KVDigestSum(0x80000000000095ce, HASH[0e67180c6b3c0ff547de58f5c0b36d63fea5499b4146ebe315825a43590b9e53], 255))
                                                5: Parent
                                                6: Push(HashWithSum(kv_hash=HASH[d7245bd11393b60e38a551ae083e703e4a2584b6eaa70c1389934ae2fcb5f593], left=HASH[165d0e36dcabfbc3a1890a603eedfa51e0e170de6a6961f949031005746ac325], right=HASH[3aefae9c66dfb5bcbfbed5c59816a7ce7b447c7a9432d130e2b42a3c19c0d3d2], sum=63))
                                                7: Push(KVDigestSum(0x800000000000aece, HASH[d27cc74018d6d49e895ad0db3e93d0dd05619e0efa0bb50ffc6691f69a98aa90], 127))
                                                8: Parent
                                                9: Push(HashWithSum(kv_hash=HASH[dc56bbcb92f002e0a7e223cd2860a09121f117de4bcd725676d5b1ff48528d9f], left=HASH[eddcd28c36db5ecef8838d42845fee17a2087e0831ff6813ea2d38f3eaf1443f], right=HASH[bca8a2fc6c5a4c5025a65d845a5b54ac15bbfbbd3aebbba95c9dc38d0e94d3bb], sum=31))
                                                10: Push(KVDigestSum(0x800000000000bb4e, HASH[89151daf6d33cc9634cc0bcd93ffd0ab0be5aa9d4ddac159aa383b589432d14d], 63))
                                                11: Parent
                                                12: Push(HashWithSum(kv_hash=HASH[749de6348d21966ee040946142c7e4dce7b113041032dc9cc95d4c0e72bacdc9], left=HASH[32eee9d294a45839a77c1a0d6c05fa3bbebeb0435b4894409bb5863ac7e050d2], right=HASH[0b251e9efd6d616d3f9c5e361a5032b24d4526b556e16ee04a02984d9aa002d0], sum=15))
                                                13: Push(KVDigestSum(0x800000000000c18e, HASH[fa15c265143026dd1e34dbc5b986ba9839069a2c01692712954beb492a096d0e], 31))
                                                14: Parent
                                                15: Push(HashWithSum(kv_hash=HASH[6f8df1c4654e54ea20cd4dad686fc471cb8f94ee6cfc7c4a72bd00792b594138], left=HASH[3c80faf26e8f2a6a266234f2367a0ce908926b72a4092fc7c3b02bebe1c47e85], right=HASH[390d9a0a56902ab7b8611d68e07f0068667302802b4bcffe71796c01d0a0ccae], sum=3))
                                                16: Push(KVDigestSum(0x800000000000c31e, HASH[110023b15d77b39d67044847913b994adf94cdb8ecfa5bce2abd4d37d6ac68c7], 7))
                                                17: Parent
                                                18: Push(KVDigestSum(0x800000000000c382, HASH[1ea127fb3003e8de7bcfbe341b29da4cff8b8c119dfa10d47eaca70fb7a62d54], 1))
                                                19: Push(KVDigestSum(0x800000000000c3e6, HASH[35c7ea9920ba64ff253a6e16dd8ed0a38231221d2bdd6e769c35e96634ae994b], 3))
                                                20: Parent
                                                21: Push(HashWithSum(kv_hash=HASH[722fcc58984fccf6edb3dc0bb7bda4f6e35c57981fb6f70a937df696c079893f], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], sum=1))
                                                22: Child
                                                23: Child
                                                24: Push(KVDigestSum(0x800000000000c4ae, HASH[23944bfe2e2a1e79e00e267e0ab5ff93748f8e7273049e02f91dbf97892ccbdb], 15))
                                                25: Parent
                                                26: Push(HashWithSum(kv_hash=HASH[b59968c7e2c22578ed837d29716b533315b0a1717de0a3165126829aeeac00fc], left=HASH[1c2fcdb483a1ed6e433b0c42056c20b68502f6ac2f074daa3335cd3508d2b058], right=HASH[ecf421d6c9e3e68bafef2663530a1a87339ed7b7cd66539991d9c32051dc47f1], sum=7))
                                                27: Child
                                                28: Child
                                                29: Child
                                                30: Child
                                                31: Child
                                                32: Child
                                                33: Push(KVDigestSum(0x800000000000c7ce, HASH[978baee06712b4ec8bcb7c36e140dc5c4031e2d68b3aa15c2167870a2a45778a], 1000))
                                                34: Parent
                                                35: Push(HashWithSum(kv_hash=HASH[db8a2deb3aa116b1e0d5a1de03b8c4deed54d2ab06ad693238424a7a68288008], left=HASH[f58ece054ce9b084e4042e0ee2e03c8dddde53f1936c94e733c4ef5c146c597e], right=HASH[18f3c1450fce7a85cf5eb87ccc2c4008ef350b18d72de000f2e0606788862659], sum=488))
                                                36: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. The first 7 layers mirror Q4's compound prefix descent: root → @ → contract → 0x01 → tip → recipient → recipient_050 (with the NotSummed(ProvableSumTree) continuation pointer at layer 7). Layer 8 is the AggregateSumOnRange collapse on byRecipientTime's continuation merk-tree — same structural shape as Q7's bottom layer (HashWithSum opaque subtree commits + KVDigestSum boundary kvs), but covering only the 1 000 sentAt entries under recipient_050 instead of the global 100 000. The verifier walks the merk boundary and accumulates sum fields for the right-half range (RangeAfter), yielding sum = 500 (recipient_050 has 500 tips with sentAt > 50000, each contributing amount = 1). Note op 33's KVDigestSum(0x800000000000c7ce, …, 1000) is the max-sentinel committing recipient_050's full 1 000-tip sum; op 35's HashWithSum(…, sum=488) is the parent subtree commit on the right of the range floor. Verified via GroveDb::verify_aggregate_sum_query, root hash 95aa7470…71b6.

Two-prefix descent (recipient_050 → sentAt) followed by an AggregateSumOnRange over byRecipientTime's continuation ProvableSumTree. The same O(log T') range collapse the unfiltered range query (Query 7) is intended to use, just with a compound prefix paid for at the boundary subtrees on the way down.

Diagram: per-layer merk-tree structure

Layers 1–7 mirror Q4's compound-prefix descent (recipient → recipient_050 → sentAt-continuation). The divergence is at layer 8: instead of a single absent terminator op, the proof presents an AggregateSumOnRange boundary walk that collapses to a single aggregate sum.

flowchart TB
  subgraph L8["Layer 8 — byRecipientTime sentAt ProvableSumTree (RANGE COLLAPSE under recipient_050)"]
    direction TB
    L8_collapse["<b>AggregateSumOnRange(sentAt > 50000)</b><br/>verified sum = <b>500</b><br/>(500 of recipient_050's 1000 tips match,<br/>each contributing amount = 1 per the cycle)"]:::target
    L8_boundary["Range-boundary commitments (37 merk ops):<br/>same HashWithSum + KVDigestSum shape as Q7's layer 6,<br/>but on a 1000-leaf merk-tree (recipient_050's per-recipient<br/>continuation) instead of the global 100 000-leaf tree.<br/>Op 33 KVDigestSum(0x…c7ce, …, 1000) is the max-sentinel<br/>(per-recipient max sentAt). Op 35 HashWithSum(…, sum=488)<br/>is the parent commit on the right of the range floor."]:::pst
    L8_collapse --> L8_boundary
  end

  classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;

Q8 is smaller than Q7 (2 657 vs 3 102 bytes) for exactly one reason: the bottom layer's merk-tree is the per-recipient continuation (1 000 distinct sentAts under recipient_050) instead of the global timeline (100 000 distinct sentAts). Shallower merk-tree, shallower descent, fewer boundary commits. The Q7-Q8 byte delta is the structural cost of taking the global bySentAt path versus a compound-prefix descent.

Query 9 — Carrier-Aggregate (In plus range)

select       = SUM(amount)
where        = recipient IN ["recipient_000", "recipient_001", ..., "recipient_099"] AND sentAt > 50000
group_by     = [recipient]
limit        = 100
sum_property = "amount"
prove        = true

This is the carrier-aggregate sum shape: an In clause on the index's prefix property combined with a range on its terminator, returning one sum per resolved In-bucket rather than a single aggregate across all matches. The group_by = [recipient] (not [recipient, sentAt]) routes through SumMode::GroupByIn — the routing table at mode_detection/v0/mod.rs maps (GroupByIn, In, range, prove) → RangeAggregateCarrierProof, which is what the per-In-bucket aggregation needs. A group_by = [recipient, sentAt] (GroupByCompound) routes to RangeDistinctProof instead — per-(in_key, range_key) distinct walk, a different proof shape entirely. Sum analog of count's Range-Countable group-by carrier-aggregate. The primitive landed in grovedb PR #670 (head e98bab5f); the verifier is GroveDb::verify_aggregate_sum_query_per_key.

Path query (carrier-style: outer Query enumerates the In branches, subquery descends through the terminator's AggregateSumOnRange):

path:                ["@", contract_id, 0x01, "tip", "recipient"]
query items:         [Key("recipient_000"), Key("recipient_001"), ..., Key("recipient_099")]
subquery_path:       ["sentAt"]
subquery items:      AggregateSumOnRange(RangeAfter(serialize_value_for_key("sentAt", 50000)..))

Verified result (returned by GroveDb::verify_aggregate_sum_query_per_key):

(root_hash, entries) where entries =
  [ ("recipient_000", 499 × 1  = 499),
    ("recipient_001", 500 × 2  = 1000),
    ("recipient_002", 500 × 3  = 1500),
    ...
    ("recipient_009", 500 × 10 = 5000),
    ("recipient_010", 500 × 1  = 500),
    ...
    ("recipient_099", 500 × 10 = 5000) ]

(Per recipient, 500 of their 1 000 tips have sentAt > 50 000, and each recipient's per-doc amount is constant per the fixture's per-recipient cycle — so each outer-bucket sum is 500 × amount_for_recipient. recipient_000 is the lone exception: the boundary timestamp sentAt = 50 000 happens to be one of recipient_000's tips, so only 499 of recipient_000's tips satisfy sentAt > 50 000. The bench's verified entries lead with ("recipient_000", sum = 499).)

Proof size: 169 064 bytes (k=100 outer buckets). That's significantly larger than the non-carrier In shapes (Q5 12 064 B for k=100, Q6 9 784 B for k=100) because each of the 100 outer buckets walks an AggregateSumOnRange proof, whereas Q5/Q6 only commit a single-element value-tree sum_value per branch. The per-bucket marginal cost is ≈ 1 690 bytes (most of which is the inner range proof — the carrier composition itself adds only ≈ 100 bytes per outer Key versus the equivalent flat-In shape). Avg time: 11 507.9 µs (k=100) — ≈ 115 µs per outer bucket, consistent with Q8's 91.3 µs single-bucket AggregateSumOnRange plus the carrier descent's per-outer-key cost.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (206 LayerProofs total — recipient_000's full descent shown; recipient_001..recipient_099 abbreviated for readability) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
          lower_layers: {
            0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
                  1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
                      lower_layers: {
                        tip => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
                              1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
                              2: Parent
                              3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
                              4: Child)
                            lower_layers: {
                              recipient => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(KVValueHash(0x0000000000000000ffffffffffffffff00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[2c932396123fe6bae5fa3e4fa42225852e08a2dcf7600991e79fb9347de7506e]))
                                    1: Push(KVValueHash(0x0000000000000001fffffffffffffffe00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[43c65eed82b8e5d08b7fea673bf120a82332adbffb8582e0f8a3205190fab720]))
                                    2: Parent
                                    3: Push(KVValueHash(0x0000000000000002fffffffffffffffd00000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[e4cf97fee88b23d122998ea98dbc83fc662da5e0d80c1f854758b2ce56699289]))
                                    4: Child
                                    5: Push(KVValueHash(0x0000000000000003fffffffffffffffc00000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[b552d6302d6bd0f76277a1b56423f25efec67612d42c924e0d486202f31e3e44]))
                                    6: Parent
                                    7: Push(KVValueHash(0x0000000000000004fffffffffffffffb00000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[01be3636585f11d5a09875214cb8e7c68cbe462c739f012693ae3404c58f1179]))
                                    8: Push(KVValueHash(0x0000000000000005fffffffffffffffa00000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[2cd10cc36cb8056b17df3d117b6687d9bd857a79d1a73f2b85290d87a9ce4a24]))
                                    9: Parent
                                    10: Push(KVValueHash(0x0000000000000006fffffffffffffff900000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[f020781d83a04f027649d42a0dc92f91506ae35765844773f72e59071cb1f107]))
                                    11: Child
                                    12: Child
                                    13: Push(KVValueHash(0x0000000000000007fffffffffffffff800000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[9fba5594f7f0b758415ae7159919102c1a2c8aecc98af22c36da32d83199f475]))
                                    14: Parent
                                    15: Push(KVValueHash(0x0000000000000008fffffffffffffff700000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[64188af510ecd884d09350ae08a5f42d5922321408aa552ec9af6c847d264a7e]))
                                    16: Push(KVValueHash(0x0000000000000009fffffffffffffff600000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[304dc0b23c2eac4c2e8cf7eddd7b3c57bee411ae03ce59b9248c0b5ad89db242]))
                                    17: Parent
                                    18: Push(KVValueHash(0x000000000000000afffffffffffffff500000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[97b584dfeb399a5278c736c1c6668c12765926780a15e8e93678903734c61aa3]))
                                    19: Child
                                    20: Push(KVValueHash(0x000000000000000bfffffffffffffff400000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[76b3db0ddb6b2c5756cbc4d66cd37109501665ef122f6d1ce20c4a5566a0bf25]))
                                    21: Parent
                                    22: Push(KVValueHash(0x000000000000000cfffffffffffffff300000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[a81ae2020176545244f9cc4ef9b86f014802b6d11908985f38389f85b4222377]))
                                    23: Push(KVValueHash(0x000000000000000dfffffffffffffff200000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[d46254b6791a7c20ebcb43b2c13fbd73a07704cce56bc78357719d48ae27d1ef]))
                                    24: Parent
                                    25: Push(KVValueHash(0x000000000000000efffffffffffffff100000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[1e8cdba5aa96432bc4a5cd1bb4f2808f2c48debdc450d7f878076217b3fd38aa]))
                                    26: Child
                                    27: Child
                                    28: Child
                                    29: Push(KVValueHash(0x000000000000000ffffffffffffffff000000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[62d0fc12b138b76a1686f65ee269cbe91156aeb3da7b50dd3fd908eb1e6bf9c2]))
                                    30: Parent
                                    31: Push(KVValueHash(0x0000000000000010ffffffffffffffef00000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[c43bd23a1c31c624127621e4480ca21d5b2a03725035d5b74301f6780122af92]))
                                    32: Push(KVValueHash(0x0000000000000011ffffffffffffffee00000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[b9d2ff1ba31cc5ab8293e2a2c14609bf99adf6acc262c02801c15b04765c1857]))
                                    33: Parent
                                    34: Push(KVValueHash(0x0000000000000012ffffffffffffffed00000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[9fbbfcf2a78087d8f112050fa354e424f897638b570dc7afc82bf286a48aa2bd]))
                                    35: Child
                                    36: Push(KVValueHash(0x0000000000000013ffffffffffffffec00000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[929742b49a2ccf0ab75af768d9f37ac226212d4170f190e58f9f07f3e59b1500]))
                                    37: Parent
                                    38: Push(KVValueHash(0x0000000000000014ffffffffffffffeb00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[5131ff85f089920a94f6a9bf54f7db8bc753aa723485ae73f719acfb8992af87]))
                                    39: Push(KVValueHash(0x0000000000000015ffffffffffffffea00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[7a29693947430c79f6ecbd83cecc684e696b4303bdbbc110998408f57e72b6ac]))
                                    40: Parent
                                    41: Push(KVValueHash(0x0000000000000016ffffffffffffffe900000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[8b1b865f2a8118504ac42fbab6c2b621f155043eceb8fbc65b1ac60d698a2dd3]))
                                    42: Child
                                    43: Child
                                    44: Push(KVValueHash(0x0000000000000017ffffffffffffffe800000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[3f98dbb6d76a0eff5dd84171dc4b610ce2e4148f0dd2347a064eff194ed73878]))
                                    45: Parent
                                    46: Push(KVValueHash(0x0000000000000018ffffffffffffffe700000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[1e7835c334b773fff98899fcdeb173516c5f61640e96bbc54a9b2347b3496592]))
                                    47: Push(KVValueHash(0x0000000000000019ffffffffffffffe600000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[c77112c339a35d47922a4618d08315dab1a0d3cc2675d3416f20be077beffff6]))
                                    48: Parent
                                    49: Push(KVValueHash(0x000000000000001affffffffffffffe500000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[fa5fb0940cf60976030fa3ed1b215f2b8685bf61d5df0b1d071a0055a60b8669]))
                                    50: Child
                                    51: Push(KVValueHash(0x000000000000001bffffffffffffffe400000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[582775f7449c7c96988ea45a70744b17967586af24fc23254241961f04af89da]))
                                    52: Parent
                                    53: Push(KVValueHash(0x000000000000001cffffffffffffffe300000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[c71003aa7b862d8a1546fae82a4b0d40cd5bf00be0b3fa748295638262c6e97e]))
                                    54: Push(KVValueHash(0x000000000000001dffffffffffffffe200000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[d200a08e3400945ec481e35843f5970c2040c5d649c0936249ae223373f66933]))
                                    55: Parent
                                    56: Push(KVValueHash(0x000000000000001effffffffffffffe100000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[e399c91063655e659638f034cb37aa38be8aa6591826afb8150168e23bab062c]))
                                    57: Child
                                    58: Child
                                    59: Child
                                    60: Child
                                    61: Push(KVValueHash(0x000000000000001fffffffffffffffe000000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[6e393e0b7b898fae65df835b64af0060496d6b1f11e9eacbac8c3f5850136185]))
                                    62: Parent
                                    63: Push(KVValueHash(0x0000000000000020ffffffffffffffdf00000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[a39a5c32c8da4dd50b76788881dd61af87f88430984fcc1e3e3bda3f695060b7]))
                                    64: Push(KVValueHash(0x0000000000000021ffffffffffffffde00000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[655cc67f6eb581596c76e146f9ce55294d53bfcbec54646afde11c7adaebdcbf]))
                                    65: Parent
                                    66: Push(KVValueHash(0x0000000000000022ffffffffffffffdd00000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[a13be8916a62ae0d1b924b74b5e5524f905ed3f5d980bb0be9d53fe224b0b089]))
                                    67: Child
                                    68: Push(KVValueHash(0x0000000000000023ffffffffffffffdc00000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[2bf66732319328bcf5588bef2b78e33a0d0881e5cec062fa0b87aeda98f6e64a]))
                                    69: Parent
                                    70: Push(KVValueHash(0x0000000000000024ffffffffffffffdb00000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[df49aa0199c39cd22de27d4df6ff98438eb3e3b07b851cc48efb30fbd381c8d7]))
                                    71: Push(KVValueHash(0x0000000000000025ffffffffffffffda00000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[284d47e7e1d21fc8f37dac05a6deaf9f27506000ff5713c8936633174bacb660]))
                                    72: Parent
                                    73: Push(KVValueHash(0x0000000000000026ffffffffffffffd900000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[95bf9e3ea0a08e43d28e954311f6664532d015c8b5e1b0c0155327095aaf6858]))
                                    74: Child
                                    75: Child
                                    76: Push(KVValueHash(0x0000000000000027ffffffffffffffd800000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[bf951ec3b345040d169c258847c11f19c20e1579f492fba4ef478fbddc6c1a7f]))
                                    77: Parent
                                    78: Push(KVValueHash(0x0000000000000028ffffffffffffffd700000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[55ff1a89b12b01d1e31b9f599dec325ea52ada71065011ed2601e6a53373ce6d]))
                                    79: Push(KVValueHash(0x0000000000000029ffffffffffffffd600000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[5f38013b5c19b8272676fa4ad0ba58b3880b195d35b3d3ac2815b8233bf0de30]))
                                    80: Parent
                                    81: Push(KVValueHash(0x000000000000002affffffffffffffd500000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[0b443a9b31f304014595cdf5129f34da66e1a7274fc3548e8c799805232d3e23]))
                                    82: Child
                                    83: Push(KVValueHash(0x000000000000002bffffffffffffffd400000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[d349113a977fe2c1e5fdffe70a6fec8595da6fcea4882b63f2460516d141bc57]))
                                    84: Parent
                                    85: Push(KVValueHash(0x000000000000002cffffffffffffffd300000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[a198da76b8f9db38a4c80a602d8d2e9ac0ad7758ddb483450b4a3f13e3592479]))
                                    86: Push(KVValueHash(0x000000000000002dffffffffffffffd200000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[33b369e7c75c3d3d362dbf33965d1b19dc96216d7723bebb78f10e49547a56b8]))
                                    87: Parent
                                    88: Push(KVValueHash(0x000000000000002effffffffffffffd100000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[051215e98dc597dd3beec177f691301fd0607916809ed00c36be278bd91a8e65]))
                                    89: Child
                                    90: Child
                                    91: Child
                                    92: Push(KVValueHash(0x000000000000002fffffffffffffffd000000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[0cdd2e9e49edccb1edd6fecb4f2d1d8c9f1355db935a67747974fbd803bb4d76]))
                                    93: Parent
                                    94: Push(KVValueHash(0x0000000000000030ffffffffffffffcf00000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[e9a1c2651036336e091b4ba6f373f616c8409d54ab2d8b8949778d930204c551]))
                                    95: Push(KVValueHash(0x0000000000000031ffffffffffffffce00000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[74853523c4698c7e8cfa1f0f744a3da1480993f92f84805c34836cb0bfc31c26]))
                                    96: Parent
                                    97: Push(KVValueHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[264c6aac1a1acd864832a6f25ac42c626afb95dc79ac8c233d2b05c6df048f1c]))
                                    98: Child
                                    99: Push(KVValueHash(0x0000000000000033ffffffffffffffcc00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[b9cb74f5fabd2c1a81d0cf16213246acadcca2240d5910e4e2b22533d4f87067]))
                                    100: Parent
                                    101: Push(KVValueHash(0x0000000000000034ffffffffffffffcb00000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[257e1c20c28d5cf3bb2959b2c33e75ecbcf454188e2553200315693e7f2124bc]))
                                    102: Push(KVValueHash(0x0000000000000035ffffffffffffffca00000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[738ae56fcfe64ed942f759be3089512fb256097dd246da488998d00c13bb55ee]))
                                    103: Parent
                                    104: Push(KVValueHash(0x0000000000000036ffffffffffffffc900000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[9badbc44201ecd712314ec0086d0d47224d5dfab25b95149aa2fedf39989644c]))
                                    105: Child
                                    106: Child
                                    107: Push(KVValueHash(0x0000000000000037ffffffffffffffc800000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[167e2a54f09ffbf49d152a32371a8035596a3a9a2d839f055f46faa75ade51c3]))
                                    108: Parent
                                    109: Push(KVValueHash(0x0000000000000038ffffffffffffffc700000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[a447c9f23d474c77f7041c698f96a95aa8421d55e6fc7eee58f3b4be0d6caa9c]))
                                    110: Push(KVValueHash(0x0000000000000039ffffffffffffffc600000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[ced5fe912b9e12105bf8a733df895ed8bdd75cce53033a6cb51e78fc361f67a8]))
                                    111: Parent
                                    112: Push(KVValueHash(0x000000000000003affffffffffffffc500000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[f1ec66f96298db46aeceaa1785dd24e479005237456bf8ee6482a8731cac4dd2]))
                                    113: Child
                                    114: Push(KVValueHash(0x000000000000003bffffffffffffffc400000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[f354580129aa8e8c9d0df8ce9842e72c198f4865101825aa64efaba200a18425]))
                                    115: Parent
                                    116: Push(KVValueHash(0x000000000000003cffffffffffffffc300000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[bc33576ad567e77c108ce92c1aafa1122c8642e8849e93fca8e91b8f6532bdf5]))
                                    117: Push(KVValueHash(0x000000000000003dffffffffffffffc200000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[04bde39065544c84157f4c6248222af72678e1536735f870375c7e516d3f3fc4]))
                                    118: Parent
                                    119: Push(KVValueHash(0x000000000000003effffffffffffffc100000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[3bdadb83f854abf1c7aae1fdd86fd6e0c26753e2f813a1494f4ce6ee592371c9]))
                                    120: Child
                                    121: Child
                                    122: Child
                                    123: Child
                                    124: Child
                                    125: Push(KVValueHash(0x000000000000003fffffffffffffffc000000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[604318edaf54250e84ab7756467488be7cf9c6b3252942117fd36d6c53c2b872]))
                                    126: Parent
                                    127: Push(KVValueHash(0x0000000000000040ffffffffffffffbf00000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[c772aab9ebd16e076279e440adc1a8d365eefb36a6d25760a8fe0023b88b5904]))
                                    128: Push(KVValueHash(0x0000000000000041ffffffffffffffbe00000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[448ae13230a21a21f7df14295673b3dd7337536868d9a667db5a756e37be845f]))
                                    129: Parent
                                    130: Push(KVValueHash(0x0000000000000042ffffffffffffffbd00000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[1468ec171f1365f54e1845009bbd9105e86b0f9288a9b0df4789270269558168]))
                                    131: Child
                                    132: Push(KVValueHash(0x0000000000000043ffffffffffffffbc00000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[173dc47f3e1fce81d4bdfffad37bb2fcd7e08923454a07aa5318ac23ace64b00]))
                                    133: Parent
                                    134: Push(KVValueHash(0x0000000000000044ffffffffffffffbb00000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[abd15b12cf6f035db4e163162e765da775cfd78012ce5d20eaad3976eb80b291]))
                                    135: Push(KVValueHash(0x0000000000000045ffffffffffffffba00000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[ffa1ee3178182f5dc40a29afd3a236f8bba9892d1604d04114a32eb07c176f4f]))
                                    136: Parent
                                    137: Push(KVValueHash(0x0000000000000046ffffffffffffffb900000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[693437a92f6dfaee70b1cdb86fd6b208939bb431522a6f8321564f5f2e98b511]))
                                    138: Child
                                    139: Child
                                    140: Push(KVValueHash(0x0000000000000047ffffffffffffffb800000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[bf2a9fb3100355e398e9c869997ac02f9e3ba95a143a08b052adb1ab82450bdb]))
                                    141: Parent
                                    142: Push(KVValueHash(0x0000000000000048ffffffffffffffb700000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[a5824c1c11fb02112dff6b46b3e450abfd4cdff94151dae513348691febf7bcf]))
                                    143: Push(KVValueHash(0x0000000000000049ffffffffffffffb600000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[411d57a7e582d3f692ca98b3634e6aa75aba4681253d1bdb7144530a16bb546e]))
                                    144: Parent
                                    145: Push(KVValueHash(0x000000000000004affffffffffffffb500000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[e49198bc668baa08f93d66640ba7fdadfff9d59593f352bd628a3d6b57a2e53c]))
                                    146: Child
                                    147: Push(KVValueHash(0x000000000000004bffffffffffffffb400000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[c61be4cf68123d695433dc10c79d1600ff15306fd6123336b381f1918800434e]))
                                    148: Parent
                                    149: Push(KVValueHash(0x000000000000004cffffffffffffffb300000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[3a977675229ea451a61ac156f05fcbd91c85f08736666c27785513e98d0001e2]))
                                    150: Push(KVValueHash(0x000000000000004dffffffffffffffb200000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[f4b3e29e2113370f153076839d6159d2d21149779ce88dadcceac6217347bf4c]))
                                    151: Parent
                                    152: Push(KVValueHash(0x000000000000004effffffffffffffb100000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[a98df638bc8e3148d9a7e407f4c7a352d9a6f7713487b6d3e4c5af09a78126d0]))
                                    153: Child
                                    154: Child
                                    155: Child
                                    156: Push(KVValueHash(0x000000000000004fffffffffffffffb000000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[80b994258a9d0e7500bbaeb910405395f7c7d03006fb7a15156d01740205a364]))
                                    157: Parent
                                    158: Push(KVValueHash(0x0000000000000050ffffffffffffffaf00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[df44fba2712d3ba1d71ed1c1c586064197a41b7b903d3b5cae7f127c6c8a27e7]))
                                    159: Push(KVValueHash(0x0000000000000051ffffffffffffffae00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[2294f725ab5e79ad583cf60d7a942508fe292262b3fbfd3001aa916290a824d6]))
                                    160: Parent
                                    161: Push(KVValueHash(0x0000000000000052ffffffffffffffad00000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[79ca47b53c05f86e6c251af81b52999f032625e0c11eb0c06d41f2f3f83e1449]))
                                    162: Child
                                    163: Push(KVValueHash(0x0000000000000053ffffffffffffffac00000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[882db62836b8db91de36be15b3e85c22f30d2c7ffbf2357e4196104ea323713f]))
                                    164: Parent
                                    165: Push(KVValueHash(0x0000000000000054ffffffffffffffab00000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[2611abbb866d065a01f8d5dd51576365a84ff715b5cb9fd4b359f91393dd9b00]))
                                    166: Push(KVValueHash(0x0000000000000055ffffffffffffffaa00000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[364d73402be79726b57efabf3758cf40784aeccfbc0dccee7692af12fe66c279]))
                                    167: Parent
                                    168: Push(KVValueHash(0x0000000000000056ffffffffffffffa900000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[22a362b3398c53a32b94e2c916a91a8b5d59c0bce9613658021a02f23896da19]))
                                    169: Child
                                    170: Child
                                    171: Push(KVValueHash(0x0000000000000057ffffffffffffffa800000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[d8eb50c3ba501a828697da535d6a14665a12b9ea5e1214f37a738a27d7fe6ba1]))
                                    172: Parent
                                    173: Push(KVValueHash(0x0000000000000058ffffffffffffffa700000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[dada8ecba303e160deec29e7215e21572bdccc38647fd5550b0ac47af7ac52f4]))
                                    174: Push(KVValueHash(0x0000000000000059ffffffffffffffa600000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[cbeaf7428892a5ca259437414d3eda7fd4de6ef55eb3ef0195f37fddd695e893]))
                                    175: Parent
                                    176: Push(KVValueHash(0x000000000000005affffffffffffffa500000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[8320cbd66a3e07a381a989e260edf5b42579ea5951384c2280f037e698e95ff9]))
                                    177: Child
                                    178: Push(KVValueHash(0x000000000000005bffffffffffffffa400000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[44342c04d0b5a3ede78cd4ef036bb3b48b8f9e4da2a502e0b5e36874cab62588]))
                                    179: Parent
                                    180: Push(KVValueHash(0x000000000000005cffffffffffffffa300000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[547f96a911cb5f0657cad7e5ce0487fa1a8870b232c52ff3efac108e6978f9e9]))
                                    181: Push(KVValueHash(0x000000000000005dffffffffffffffa200000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[f8342da41347c35f4a966a1b9c4bbb684af653cd18c3bb200e630f2731266c06]))
                                    182: Parent
                                    183: Push(KVValueHash(0x000000000000005effffffffffffffa100000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[18b8d60c015a057419a6405275cd02d490ff7cab627a653ec603a97757a5f49f]))
                                    184: Child
                                    185: Child
                                    186: Push(KVValueHash(0x000000000000005fffffffffffffffa000000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[d5e7eb06c7bd04852d0b25bd5df0039f3ef20d34711ef121f00cce3a31349dd5]))
                                    187: Parent
                                    188: Push(KVValueHash(0x0000000000000060ffffffffffffff9f00000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[2462c14975dad4a210d556d0e579529e8a756b3b107cbbc75fe344b2a03b357b]))
                                    189: Push(KVValueHash(0x0000000000000061ffffffffffffff9e00000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[fe0d935c37c343340bc5f32dcedd4d3eb7c7cce20604065ff63bced0a951de75]))
                                    190: Parent
                                    191: Push(KVValueHash(0x0000000000000062ffffffffffffff9d00000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[bbc5e27f1fdbba68e332d6cc50790017f29b2878b21e49b401701deee9bb2948]))
                                    192: Push(KVValueHash(0x0000000000000063ffffffffffffff9c00000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[15ca1dc935c384f6fd904a3de0643e4af7b8c38fe3946379c182a54714786ac1]))
                                    193: Child
                                    194: Child
                                    195: Child
                                    196: Child
                                    197: Child
                                    198: Child)
                                  lower_layers: {
                                    0x0000000000000000ffffffffffffffff00000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[f3cd8b8f77e6c9473ba47bb5650e481537027a91bcea62c04e70d9c658b7ce87]))
                                          1: Push(KVValueHash(sentAt, NotSummed(ProvableSumTree(800000000000c79c, 1000)), HASH[92dc35643411662e66f2e8dc6b591cf7eed54f6dc4aaf69219c369c74188e8a7]))
                                          2: Parent)
                                        lower_layers: {
                                          sentAt => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithSum(kv_hash=HASH[622ec7c875ff1016840e660f388f97bc899c32726d6f18a5966da84ec63ea355], left=HASH[ff984fa6039c70cdb6a896d7d69ecf3271dc0b103ee44d5d8301365c110b23fd], right=HASH[c90f5696ed4eb077cc0b1cdb904b2af2a75fd9b07f60345925cdddf7f6b38f2a], sum=255))
                                                1: Push(KVDigestSum(0x800000000000639c, HASH[a9f7ac3993ee6e200d0f3d40697e009ce0bfe278009f77e3e82215ad7475c6e3], 511))
                                                2: Parent
                                                3: Push(HashWithSum(kv_hash=HASH[18cab59d8df3ddadeb0054a44540c64a916e022e3af157b38c175c7f9908b233], left=HASH[3f8f3e6a71375ccfae2714e6d9468ca8df8c914bcae7a5e67f2d167ccf82c6c6], right=HASH[fd4793282e7beeae5ffcc9ed6adf62ea213bbede0fc061ba5f5534303218a92f], sum=127))
                                                4: Push(KVDigestSum(0x800000000000959c, HASH[94924e9e1cb6b1851911f3290f93a745bc1e9751044e5e462e702eb9c48f6572], 255))
                                                5: Parent
                                                6: Push(HashWithSum(kv_hash=HASH[f03986bdca555701a548ebb6483b177814570cc7e8638bb312b7c6939504eb18], left=HASH[f171da996938e7ea3d5840ea2b3448e40c701c3d941e2e81ea45450055f6e5c8], right=HASH[8eccc3e0d53a1d8dabf858f17886a204728adff59e19346d61d227fafa86e08a], sum=63))
                                                7: Push(KVDigestSum(0x800000000000ae9c, HASH[a606c7eec7545c1124b0bccbcb29dedfdcba820f05446806c4eaf5c8106a667d], 127))
                                                8: Parent
                                                9: Push(HashWithSum(kv_hash=HASH[082aa2d5a4f27332961db4882ea673e1a9dc7c163eaf56c1ad0c2f876f8369c3], left=HASH[31d49f9b0d0c858fbe65650b5c4ffbfc86def44778a62253a2a81f92e60363dc], right=HASH[d52fc80e4e04177e9cab24953d10d870b73f5acd76f81076c784932b92f5cc4a], sum=31))
                                                10: Push(KVDigestSum(0x800000000000bb1c, HASH[01f81f07f05bdab399b7f9b3583316cae8227ee8ef6f38f1292510e3fcdf8759], 63))
                                                11: Parent
                                                12: Push(HashWithSum(kv_hash=HASH[1ed335fe242c26d4530101e609a6d5a52a32dddfdd9fed225de10a7039d19238], left=HASH[373889c6db5863abb87ef61b3ee479d9ebf13922571c2f7e24a3e3b1eac7fa16], right=HASH[e126395fd037d9f94216301144e57dfa044de771c999597f6fcf5e604c2ce2f1], sum=15))
                                                13: Push(KVDigestSum(0x800000000000c15c, HASH[7eb10dc2f4d81ce6c5df234cc54b19fd87b8b861fc5151dd0c19619171c7ce83], 31))
                                                14: Parent
                                                15: Push(HashWithSum(kv_hash=HASH[c59ef9246a6a7225fb2ef0558bf280f8df39ed99a17bb40812c639a18fd3af14], left=HASH[96e89928ae0de69be602723850619c0ebd3542da48ab32b8a1db5bc2e8603192], right=HASH[0cb2ba854b100015b09de5984a4103976d2d3b46900e9ddee76cf600234b5659], sum=3))
                                                16: Push(KVDigestSum(0x800000000000c2ec, HASH[c68b52e5b8036d991946274166ef630ef19aeafd0f38d2388e2f7423cda1045f], 7))
                                                17: Parent
                                                18: Push(KVDigestSum(0x800000000000c350, HASH[b22d3790d948924dc6311518f2872b53c0590d3cc497d7a653f1ce7b9923e499], 1))
                                                19: Push(KVDigestSum(0x800000000000c3b4, HASH[40b4e8cfba33cbdb2e6643b00f062c68ffd9786c156af78ba712b91a03af3ec0], 3))
                                                20: Parent
                                                21: Push(HashWithSum(kv_hash=HASH[38a6c1c68d6436d6059f514f0b532ffbb4d62788a72c283dea1f14e204399379], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], sum=1))
                                                22: Child
                                                23: Child
                                                24: Push(KVDigestSum(0x800000000000c47c, HASH[8c891440a3fdef244cefe3a928529f037fed7f6347509fc814f82430366f39df], 15))
                                                25: Parent
                                                26: Push(HashWithSum(kv_hash=HASH[b81a70230e6c91bdbcb631bed29cbb79815f3117da15b29f771a7f833d42090c], left=HASH[24c61905d7361140b4ca79506c2892a081daab4b471171b216f59968fd8855b5], right=HASH[8f4b5b0b1135c8b4c87597f33a968b8404bda5d2d88ee7d2b613d90d1795fb4a], sum=7))
                                                27: Child
                                                28: Child
                                                29: Child
                                                30: Child
                                                31: Child
                                                32: Child
                                                33: Push(KVDigestSum(0x800000000000c79c, HASH[387d1806a39adf033f4e7a7888970a627bfd803850eee18e8a277f45893f5e61], 1000))
                                                34: Parent
                                                35: Push(HashWithSum(kv_hash=HASH[a7d78a4571fc667589d9194ff86f9ebb7dc5c7b1c86241ee0fd1231fa877e47e], left=HASH[1bd878de2c7e1752d4e5521d36a8d3ab2854f652d6cf4be41cb1cd48cbb2cfe3], right=HASH[42073ef99149a312e57e2c165578c2ed4f62e9325f400102da26d3077916ca21], sum=488))
                                                36: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    // ↓↓↓ Q9 abbreviation ↓↓↓
                                    // 99 more recipient buckets follow here
                                    // (recipient_001 .. recipient_099), each a
                                    // LayerProof with the same structural shape as
                                    // recipient_000 above: a single-key continuation
                                    // Tree → NotSummed(ProvableSumTree) → AggregateSumOnRange
                                    // collapse. Per-bucket aggregate sums follow
                                    //   recipient_n.sum = 500 × ((n % 10) + 1)
                                    // with recipient_000.sum = 499 (the boundary
                                    // row 50000 is excluded by the half-open `>` range).
                                    // ↑↑↑ Q9 abbreviation ↑↑↑
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Each LayerProof is one GroveDB tree's merk proof. Q9 is the largest proof in the chapter and structurally distinct from every other query: the AST tree fans out at layer 6 (the byRecipient subtree) where the carrier outer walk reveals all 100 In-key terminators inline (each as a KVValueHash(recipient_NNN, SumTree(73656e744174, sum_NNN)) op), and each of those 100 keys' lower_layers entry then carries its own inner AggregateSumOnRange descent through that recipient's byRecipientTime continuation — a per-bucket Q8-shaped boundary walk producing one aggregate i64 per recipient. The total LayerProof count is 5 fixed (root → @ → contract → 0x01 → tip) + 1 outer byRecipient layer that lists all 100 In keys + 100 × 2 per-bucket layers (one for the recipient-continuation single-key tree, one for the byRecipientTime sentAt range collapse). The verifier (via GroveDb::verify_aggregate_sum_query_per_key) walks each branch, collapses its inner range to a single sum, and returns Vec<(in_key, i64)> with 100 entries:

(recipient_000, 499)        # 499 tips: rows 50100, 50200, …, 99900, each amount=1
(recipient_001, 1000)       # 500 × 2
(recipient_002, 1500)       # 500 × 3
…
(recipient_009, 5000)       # 500 × 10
(recipient_010, 500)        # 500 × 1 (cycle restarts)
…
(recipient_099, 5000)       # 500 × 10

Note recipient_000 gets 499 not 500: sentAt > 50000 excludes the boundary row (sentAt = 50000) which belongs to recipient_000 (row 50000 → recipient_(50000 % 100) = recipient_0), so its 500-row window loses one entry. Every other recipient is unaffected.

Verified root hash 95aa7470…71b6.

Diagram: per-layer merk-tree structure

Q9's structure is the chapter's only non-linear descent: instead of a single chain, layer 7 fans out into 100 parallel branches (one per resolved In-key), each running its own inner-range collapse. The diagram below shows one branch in detail and indicates the × 100 fan-out at the outer layer.

flowchart TB
  subgraph L6["Layer 6 — byRecipient merk-tree (CARRIER OUTER LAYER)"]
    direction TB
    L6_targets["100 KVValueHash outer-key terminators<br/>(recipient_000 … recipient_099)<br/>each value=<b>SumTree(73656e744174, per-recipient-sum)</b><br/>but here the values are mid-descent pointers, NOT the<br/>carrier-aggregate's final output — each one's<br/>lower_layers fans out to a per-bucket range walk."]:::queried
    L6_boundary["~22 merk-boundary ops shared across all 100 descents:<br/>since every byRecipient entry is revealed,<br/>only the merk-root spine stays opaque"]:::sibling
    L6_targets --> L6_boundary
  end

  subgraph L7a["Layer 7 — recipient_NNN continuation (× 100 parallel, one per bucket)"]
    direction TB
    L7a_q["<b>sentAt</b><br/>kv_hash=HASH[…]<br/>value: NotSummed(ProvableSumTree(…, 1000))<br/>(per-recipient continuation, single-key tree)"]:::queried
    L7a_sib["1 Hash opaque-sibling commit<br/>(per the per-bucket continuation merk-tree)"]:::sibling
    L7a_q --> L7a_sib
  end

  subgraph L8a["Layer 8 — byRecipientTime sentAt range collapse (× 100 parallel, one per bucket)"]
    direction TB
    L8a_collapse["<b>AggregateSumOnRange(sentAt > 50000)</b><br/>verified sum_NNN per recipient_NNN<br/>(499 for recipient_000;<br/>500 × ((NNN % 10) + 1) for the rest)"]:::target
    L8a_boundary["Range-boundary commitments (~36 ops per bucket):<br/>same HashWithSum + KVDigestSum shape as Q8's layer 8.<br/>This per-bucket cost × 100 buckets is what dominates Q9's<br/>169 064 B vs the non-carrier shapes (Q5 12 064 B, Q6 9 784 B)."]:::pst
    L8a_collapse --> L8a_boundary
  end

  L6_targets -. "100 fan-out: each value=SumTree(merk_root[per-bucket]) → L7a" .-> L7a_q
  L7a_q -. "value=NotSummed(ProvableSumTree(merk_root[byRT.sentAt])) → L8a" .-> L8a_collapse

  fanout["× 100 parallel descents<br/>(one per resolved In key)"]:::note
  L7a_q -.-> fanout

  classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
  classDef note fill:#21262d,color:#c9d1d9,stroke:#fb8500,stroke-width:2px,stroke-dasharray: 6 4;

This is the structural payoff of carrier composition: instead of 100 independent Q8-shaped proofs each running its own full root→@→contract→…→ recipient_NNN → sentAt descent (with the upper 5 fixed layers re-paid for each branch), Q9 amortizes the upper-5 layers across all 100 buckets and only pays the per-bucket cost (1 continuation layer + 1 range-collapse layer ≈ 1 690 bytes) per outer key. The 64% byte savings vs the N-independent baseline come almost entirely from that shared prefix amortization.

The carrier composition is the only way to get per-bucket range sums in a single proof. The alternative — issuing N independent AggregateSumOnRange proofs, one per In value — would take N × 2 657 bytes ≈ 265 700 bytes for k=100, plus N round-trips. Carrier collapses that to one proof, one round-trip, and ≈ 64% of the byte cost.

The two carrier-aggregate gates worth knowing:

  • SizedQuery::limit caps the outer walk (here limit = 100 accommodates all distinct recipients; for an open-ended outer Range clause it bounds how many In-branches the proof commits). The verifier rebuilds the same limit byte-for-byte; mismatched limits break the merk-root recomputation.
  • SizedQuery::offset is rejected for carrier-aggregate (would change which (outer_key, sum) pairs end up in the proof; the use case isn't designed yet). Mirrors the count-side carrier-ACOR contract.

The PCPS variant — AggregateCountAndSumOnRange on a carrier — exists too; same primitive, but returns (outer_key, u64 count, i64 sum) triples for indexes that opt into both rangeCountable: true and rangeSummable: true. Drive-side support is wired (DriveDocumentSumQuery::verify_carrier_aggregate_count_and_sum_proof); a worked PCPS-carrier example will live in a separate combined-feature chapter alongside its own contract.

Range Modes — Distinct Variant

Every range query above runs in aggregate mode (return_distinct_sums_in_range = false, the default). Setting it to true returns one SumEntry per distinct property value in the range — the histogram shape.

For Query 7 with return_distinct_sums_in_range = true:

SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 50001), sum: 2 }
SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 50002), sum: 3 }
...
SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 99999), sum: 10 }

49 999 entries (one per distinct sentAt in the range). The prove path's proof size in this mode is O(distinct values matched), not O(log n) — for tip-jar's per-row-distinct sentAt fixture that's a big difference. Pick the aggregate mode when you want one number, distinct when you want a histogram. Same trade-off as count's Range Queries on the Prove Path.

References With Sum Item — Where They Live

Every non-primary-key sum lookup above lands on a value-tree path like:

[..., "<index_property>", "<index_value>", 0]

where [0] (key zero) holds a SumTree of references — one per document — and the parent value-tree's sum is the aggregation of those references' sum-item contributions. The reference primitive grovedb exposes is Element::ReferenceWithSumItem(reference_path, sum_value, flags) (grovedb PR 670): a single element that dereferences to the document body in primary storage and carries an i64 (the amount value at insert time). When merk computes node hashes up the parent SumTree, the reference's sum-item is what propagates.

Two element types for two roles, kept distinct:

  • Primary storage at [doctype, 0, doc_id] uses Element::ItemWithSumItem(serialized_doc, sum_value, flags) when the doctype declares documentsSummable: "amount" — the document body lives there inline AND contributes to the primary-key SumTree's running aggregate. This is what makes the documentsSummable fast path (O(log n) total-sum proof, no index needed) actually work: the primary-key SumTree's root carries the real sum because every leaf is sum-bearing.
  • Index references at [index_path, value, 0, doc_id] use Element::ReferenceWithSumItem — a true reference (so document iteration via index walks still dereferences to the body in primary storage, exactly like Element::Reference does on the count side) AND a sum contribution that propagates to ancestor sum trees.

That's the load-bearing primitive for the design: without it, every non-primary-key sum index would need to either duplicate the document under the index (storage blowup) or refuse to participate in sum aggregation (no non-primary-key sums at all). With it, the existing index storage shape extends from "reference under [0]" to "reference with sum-item under [0]" — same descent, same proof verification, plus the running-sum propagation.

The reference's stored sum-item is fixed at insert time. On delete, Drive reads it back out of the reference (rather than re-reading the source document) and subtracts it from every ancestor sum tree. That's why the named summable property must be in required: a missing amount at insert time would leave the reference with no sum contribution, and a later delete would underflow the ancestor sums by zero — a quiet corruption that's never worth allowing.

At-a-Glance Comparison

QueryIndex usedElement shape at terminatorReturned variantProof primitive
1 — Total(doctype primary-key)SumTree at tip/[0]aggregate_summerk path
2 — Equal on byRecipientbyRecipientSumTree at recipient/recipient_050aggregate_summerk path
3 — Equal on bySentAtbySentAtSumTree at sentAt/serialize(50000)aggregate_summerk path
4 — Compound EqualbyRecipientTimeSumTree at recipient/recipient_050/sentAt/serialize(50000)aggregate_summerk path
5 — In on byRecipientbyRecipientk × SumTreesentriesk × merk path
6 — In on bySentAtbySentAtk × SumTreesentriesk × merk path
7 — Range on bySentAtbySentAt(collapsed boundary)aggregate_sumAggregateSumOnRange
8 — Compound + RangebyRecipientTime(collapsed boundary, prefix descent)aggregate_sumAggregateSumOnRange
9 — Carrier (In + Range)byRecipientTimek × (collapsed boundaries under outer Keys)per-key entriesverify_aggregate_sum_query_per_key (carrier composition)

The split is structurally the same as count's — single value-tree lookups for point queries (1–6), AggregateSumOnRange for range (7–8), and carrier composition for per-bucket range sums (9) — with SumTree/ProvableSumTree and aggregate_sum/SumEntry slotted in where count had CountTree/ProvableCountTree and aggregate_count/CountEntry. The Q9 carrier specifically wraps the leaf AggregateSumOnRange primitive once per outer In branch and stitches the per-bucket sums into a single verifiable response — saving N round-trips and ~36% of the bytes versus issuing N independent range-sum proofs.

What's Next

The full SUM surface is wired end-to-end: primary-key total (Query 1), point-lookup, In-fan-out, AggregateSumOnRange on both top-level and compound indexes, and the carrier-aggregate primitive. All nine queries verify against the shared root hash 95aa7470…71b6.

A Sum Index Group By Examples sibling chapter is queued mirroring the count GROUP BY chapter — the byRecipient index supports both per-recipient and group-by-recipient semantics, and the bench's report_group_by_matrix already publishes the full matrix (visible inline in bench -- --test output under [matrix]).

Average Index Examples

This chapter walks through a representative contract and shows how average queries work on Drive. Every example uses the same grade document type on the grades contract at packages/rs-drive/tests/supporting_files/contract/grades/grades-contract.json.

The chapter assumes you've read Document Count Trees and Document Sum Trees — averages are built directly on top of both, so understanding count + sum trees individually is the prerequisite. Here we take that machinery as given and look at the queries that need both.

Status: the document_average_worst_case bench lands the reproducible numbers below — same convention as the Count and Sum chapters. All proof sizes are measured against a 31 620-grade fixture; verified (count, sum) values are the actual numbers the bench's matrix reports. The full surface — primary-key global average, point lookups, range aggregates, and both carrier variants — is wired through to grovedb PR #670's verifiers end-to-end.

Why Averages Need a New Primitive

An average is sum / count. To prove an average against a single root-hash commit, you need both numbers from the same set in one proof — otherwise the client can't verify that the divisor and dividend describe the same documents.

Three options exist:

  1. Two separate proofs — one sum proof, one count proof, client divides. Burns 2× the proof bytes + 2× the round-trips. The bigger problem: the two proofs commit independently, so the client has to verify both root-hashes match (or the server could splice mismatched results).
  2. Materialize-and-divide — server walks the document set, sums + counts itself, returns one rational number. No O(log n) win, no cryptographic commit to the underlying count or sum; the client just trusts the server's reported quotient.
  3. A single dual-axis primitive — one proof commits both metrics from one merk traversal. The verifier returns (count, sum); the client divides. This is what this chapter is about.

The grovedb primitive is AggregateCountAndSumOnRange (added in grovedb PR #670) and its carrier extension verify_aggregate_count_and_sum_query_per_key (PCPS-carrier proofs for group-by averages). Both require the terminator tree to be a ProvableCountProvableSumTree (PCPS) — a single merk tree where each internal node carries both a count_value and a sum_value, committed to the parent node's hash. Lighter sum-bearing trees (SumTree, ProvableSumTree, CountSumTree, ProvableCountSumTree) all reject the combined primitive at the merk gate — you need both axes per-node for the proof's single traversal to commit both metrics.

The grades contract below opts two indexes into PCPS (byClassSemester and byStudentSemester); the other three are simpler CountSumTree shapes serving point-lookup averages.

The Grades Contract

The grade document type carries one grade for one student in one class during one semester. Five properties (student, class, semester, score, instructor), opts into global totals via documentsCountable: true + documentsSummable: "score", and declares five indexes spanning the average-query surface:

{
  "type": "object",
  "documentsMutable": false,
  "canBeDeleted": false,
  "documentsCountable": true,
  "documentsSummable": "score",
  "properties": {
    "student":    { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
                    "position": 0, "contentMediaType": "application/x.dash.dpp.identifier" },
    "class":      { "type": "string", "minLength": 1, "maxLength": 32, "position": 1 },
    "semester":   { "type": "integer", "minimum": 20000, "maximum": 99999, "position": 2 },
    "score":      { "type": "integer", "minimum": 0, "maximum": 100, "position": 3 },
    "instructor": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
                    "position": 4, "contentMediaType": "application/x.dash.dpp.identifier" }
  },
  "required": ["student", "class", "semester", "score", "instructor"],
  "indices": [
    { "name": "byClass",
      "properties": [{ "class": "asc" }],
      "countable": "countable", "summable": "score" },
    { "name": "byStudent",
      "properties": [{ "student": "asc" }],
      "countable": "countable", "summable": "score" },
    { "name": "bySemester",
      "properties": [{ "semester": "asc" }],
      "countable": "countable", "summable": "score" },
    { "name": "byClassSemester",
      "properties": [{ "class": "asc" }, { "semester": "asc" }],
      "countable": "countableAllowingOffset", "summable": "score",
      "rangeCountable": true, "rangeSummable": true },
    { "name": "byStudentSemester",
      "properties": [{ "student": "asc" }, { "semester": "asc" }],
      "countable": "countableAllowingOffset", "summable": "score",
      "rangeCountable": true, "rangeSummable": true }
  ],
  "additionalProperties": false
}

Five things to internalize before reading the queries:

  1. documentsCountable: true + documentsSummable: "score" at the document-type level upgrades the doctype's primary-key subtree (at grade/[0]) from NormalTree to CountSumTree. The unfiltered global average is one read against this element's (count_value, sum_value) pair, no index walk.
  2. byClass / byStudent / bySemester are countable: countable + summable: "score" (no range flags). Each per-key value-tree (one per class / student / semester) is a CountSumTree carrying both metrics at one merk lookup — point-lookup averages get the same shortcut count proofs and sum proofs do.
  3. byClassSemester and byStudentSemester set both range flags (rangeCountable: true AND rangeSummable: true). The semester continuation under each (class | student) value-tree is a ProvableCountProvableSumTree (PCPS), the structurally-richest tree variant — every internal merk node carries both a per-node count and a per-node sum. This is what AggregateCountAndSumOnRange walks for "average for class X in semester range [a..b]" style queries.
  4. Every summable index here is also countable. There's no summable-only index in this contract — averages need both axes, so a sum-only index would be unreachable from the average surface. (Pure-sum surfaces are covered by the tip-jar contract in the previous chapter; the grades contract is deliberately the dual-axis counterpart.)
  5. countableAllowingOffset on the PCPS indexes — rangeCountable: true implies a countable index (an omitted countable is promoted to "countable"; per the rule documented in Index::range_countable), so an explicit countableAllowingOffset is what picks the richer tier. The offset-allowing tier upgrades the property-name tree to a ProvableCountTree at minimum; combined with summable: "score" and rangeSummable: true the dispatcher resolves it to PCPS.

The bench populates 50 000 grades under a deterministic, realistic-data-shaped schedule: 500 students × 10 classes × 10 semesters = 50 000 grade documents. The score model layers three deterministic axes:

  • Per-class baseline + spread (see class_profile in the bench) — hard classes get low means and wide spreads; easy classes cluster near 85–90 with narrow spreads. The 10 classes have semantic names matching the chapter's references:
ClassBaseline meanSpreadProfile
PHYS1016012hard physics
CHEM1016510moderate chem
CALC2015813hardest math
ENGL101855easy english
HIST101788moderate history
BIOL101729moderate bio
ARTS101884easiest art
COMP101759moderate CS
MUSC101826easy music
SOCI101806easy social
  • Per-student skill — a deterministic FNV-1a hash of the student index, scaled to ≈ N(0, σ²) via central-limit-theorem (sum of three uniforms averaged), centered slightly below 0 to model "most students are average, a few are excellent, a few struggle." Spans roughly [-25, +15].
  • Per-grade noise — deterministic ±5 variation per (student, class, semester) so even one (student, class) pair has nontrivial semester-to-semester variation.

A skill score is amplified by class spread — skill × spread / 8 — so a +10-skill student in PHYS101 (spread=12) gains +15 over baseline, while the same student in ARTS101 (spread=4) only gains +5. This produces the realistic spread real transcripts exhibit: a strong student stands out more in hard classes, struggling students fall further behind in hard classes, easy classes flatten the curve.

Realistic enrollment. Not every student takes every class — that's not how transcripts work. The bench walks all 500 × 10 × 10 = 50 000 possible (student, class, semester) triples but only emits a grade when is_enrolled returns true, using a deterministic per-class popularity table:

ClassPopularityProfile
ENGL101100%required for everyone every semester
ARTS10190%very popular elective
MUSC10185%popular elective
HIST10170%common humanities
SOCI10170%common social science
BIOL10160%moderately popular
COMP10155%moderately popular
CHEM10145%moderately popular
PHYS10130%hard physics, smaller cohort
CALC20125%hardest math, smallest cohort

The total comes out to 31 620 actual grade documents (≈ 63% of the 50 000 possible triples — see the popularity table; the per-class actual rates match the documented popularities within ±0.5 percentage points), with the per-class enrollment counts ranging from 970 (CALC201 across 5 semesters) to 2 500 (ENGL101 — required, so every student × every semester). The expected per-student grade count is ≈ 6.3 classes per semester × 10 semesters = ≈ 63 grades per student.

Headline numbers from the bench's fixture (all verified end-to-end against the shared root hash 8b15f732af8f…ffc7):

  • Total count across all grades: 31 620 (not 50 000 — the enrollment filter removes ~29%).
  • Per-class average spans from ≈ 53 (CALC201, hardest math) to ≈ 87 (ARTS101, easiest art) — a 33-point realistic spread.
  • Per-class total count varies from 970 (CALC201) to 5 000 (ENGL101) across all 10 semesters — the enrollment differential surfaces in every per-class average proof.
  • Per cohort (one class in one semester): count varies, typically 125–500 depending on the class's popularity.
  • Per student (single student, all classes, all semesters): count varies by their enrolled mix, typically 55–70 grades. student_050 for instance verifies at count = 72, sum = 4 834 (avg = 67.14 — this student happens to have an above-average skill score).

GroveDB Layout

The contract above produces this storage shape. Tree elements are drawn as subgraphs; children inside each tree are merk-tree nodes. The doctype root and the per-property name subtrees are separate Element trees nested under the contract-documents prefix.

Diagram conventions: green nodes carry both a count_value and a sum_value (CountSumTree); yellow nodes carry both per node (PCPS); gray are regular subtrees; dashed boxes highlight wrapper elements (NotCountedOrSummed) that opt out of both axes from their parent's aggregation.

flowchart TB
  TD["@/contract_id/0x01/grade"]:::tree

  TD --> PK["[0]: CountSumTree count=31620 sum=2392808<br/>(documentsCountable + documentsSummable primary key)"]:::csnode
  TD --> CL["class: NormalTree<br/>(byClass property-name)"]:::node
  TD --> ST["student: NormalTree<br/>(byStudent property-name)"]:::node
  TD --> SM["semester: NormalTree<br/>(bySemester property-name)"]:::node

  CL --> CL_M["class_MATH101: CountSumTree count=1000 sum~50000"]:::csnode
  CL --> CL_P["... 9 more class value-trees<br/>(each CountSumTree count=1000)"]:::csnode

  CL_M --> CL_M_0["[0]: CountSumTree count=1000 sum~50000<br/>(byClass refs — one per grade)"]:::csnode
  CL_M --> CL_M_S["semester: NotCountedOrSummed(PCPS)<br/>(byClassSemester continuation, contributes 0 + 0)"]:::nonboth

  CL_M_S --> CL_M_S_241["semester_20241: CountSumTree count=100 sum~5000<br/>(byClassSemester cohort terminator)"]:::csnode
  CL_M_S --> CL_M_S_more["... 9 more semester buckets"]:::csnode

  ST --> ST_X["student_050: CountSumTree count=100 sum~5000"]:::csnode
  ST --> ST_more["... 99 more student value-trees"]:::csnode

  ST_X --> ST_X_0["[0]: CountSumTree count=100 sum~5000<br/>(byStudent refs)"]:::csnode
  ST_X --> ST_X_S["semester: NotCountedOrSummed(PCPS)<br/>(byStudentSemester continuation, contributes 0 + 0)"]:::nonboth

  SM --> SM_241["semester_20241: CountSumTree count=1000 sum~50000<br/>(bySemester all-grades terminator)"]:::csnode
  SM --> SM_more["... 9 more semester buckets"]:::csnode

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
  classDef csnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
  classDef pcpsnode fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
  classDef nonboth fill:#21262d,color:#c9d1d9,stroke:#fb8500,stroke-width:2px,stroke-dasharray: 6 4;

Three layout facts to internalize before reading the queries:

  • class_MATH101 is a CountSumTree with count = 1000 and sum ≈ 50 000. That's true because byClass declares both countable: countable and summable: "score". The average sum / count ≈ 50 is one merk lookup. The semester continuation that branches off this value tree is NotCountedOrSummed-wrapped so the parent's (count, sum) equals exactly the contribution from the 1 000 refs in [0] — without the wrapper, the compound byClassSemester continuation would double-count and double-sum into the parent.
  • The semester continuation under each class value-tree is a PCPS (ProvableCountProvableSumTree) wrapped in NotCountedOrSummed. Inside the wrapper, every internal merk node carries both a per-node count and a per-node sum — which is what makes AggregateCountAndSumOnRange a single-pass primitive. Wrapping it as NotCountedOrSummed is the load-bearing trick: the wrapper is invisible to the inner primitive (the merk walker descends into the PCPS unchanged) but opaque to the parent's aggregation (contributes 0 to both axes), keeping byClass's class-level (count, sum) clean.
  • bySemester's value trees are also CountSumTree (count + sum per semester across all students and classes). One semester's school-wide average is one merk lookup; the semester index doesn't have a byClassSemester-style continuation because there's no compound (semester, class) index in this contract — adding one would slot a parallel PCPS continuation here.

How To Read The Proofs

Every example below has four sections:

  1. Path query — the spec the prover hands GroveDB. path is the list of subtree segments to descend through; query items is what to select once at the bottom; subquery items (when present) descends one more layer.
  2. Verified result — what GroveDb::verify_query returns for point lookups, or GroveDb::verify_aggregate_count_and_sum_query / verify_aggregate_count_and_sum_query_per_key returns for the range and carrier primitives. For every query the return shape is (count, sum) (or Vec<(key, count, sum)> for carrier) — the client divides for the average. The chapter shows avg = sum / count derived inline.
  3. Proof display — the proof bytes decoded via bincode into the structured GroveDBProof AST and rendered through its Display impl, same convention as the count and sum chapters. Wrapped in a collapsible <details> block per example with a link to the visualizer.
  4. Diagram — per-layer merk-tree references back to the GroveDB Layout diagram above, with csnode (green) for CountSumTree terminators and pcpsnode (yellow) for ProvableCountProvableSumTree terminators where the dual (count, sum) per-node fields are visible.

All proof-size numbers and avg-times below come from the 10 000-row bench run; the methodology block under the queries table covers how to reproduce them.

Queries in this Chapter

#QueryFilter / Group-byComplexityAvg timeProof size
1Unfiltered Global Average(none — total at doctype level)O(1)25.3 µs622 B
2Average for One Class (byClass)class == "PHYS101"O(log C)32.1 µs871 B
3Student GPA (byStudent)student == student_050O(log S)42.0 µs1 227 B
4One Cohort (byClassSemester point)class == "PHYS101" AND semester == 20204O(log C + log T')51.0 µs1 304 B
5Class Trend (AggregateCountAndSumOnRange)class == "PHYS101" AND semester > 20204O(log C + log T')49.7 µs1 539 B
6Per-Student Averages for One Semester (carrier)student IN [0..9] AND semester == 20204 (group_by [student])O(k · log S + log T')304.4 µs6 581 B (k=10)
7Per-Class Trends (PCPS carrier)class IN [10 classes] AND semester > 20204 (group_by [class, semester])O(k · (log C + log T'))273.8 µs8 220 B (k=10)

Timing methodology: median of 5 iterations after one warmup, measured against the bench's 31 620-grade fixture on a warmed rocksdb cache (31 620 actual grades from 50 000 possible triples, filtered by per-class enrollment popularity). The figures reflect the drive-layer execute_* calls (path query build + grovedb proof generation, no network or tenderdash signature compose). Reproduce with DASH_PLATFORM_AVERAGE_BENCH_REBUILD=1 cargo bench -p drive --bench document_average_worst_case -- --test; grep µs from stderr.

Fixture-narrative cross-references: Q2/Q4/Q5/Q7 use the class name "PHYS101" (the first of 10 semantically-named classes — see the contract-narrative table above). Q3/Q6 reference student_050 (the midpoint student id). Q4/Q5/Q6/Q7 all use semester floor 20204 (the midpoint of the 10-semester range 20200..20209), so the range semester > 20204 matches exactly 5 semesters per class. The original chapter draft used "MATH101" / semester == 20241 / semester > 20210 placeholders; the bench substitutes deterministic-id names + arithmetic-midpoint values for verifiability.

Complexity variables. C = distinct classes (= 10 in the fixture); S = distinct students (= 100); T = distinct semesters (= 10); T' = distinct semesters per class or per student in the byClassSemester / byStudentSemester continuation (= 10); k = number of values in the IN clause. Notably absent: the total document count N (= 10 000 here). Average proofs read pre-committed count_value + sum_value from CountSumTree / PCPS merk roots — they never enumerate the underlying documents, so proof generation cost is polylog(distinct index values), independent of N. Same big-O story as count and sum individually; PCPS just commits both metrics per node, adding a small constant factor of per-node hash work vs. count-only / sum-only.

The first four queries (Q1–Q4) get their (count, sum) from a single CountSumTree element at the descent's terminator — same proof shape as a count point lookup, just with an extra 8 bytes per merk node for the sum field. Q5 uses AggregateCountAndSumOnRange against the byClassSemester PCPS continuation — one proof, single root-hash commit, returns (root_hash, count, sum). Q6 and Q7 are the carrier variants — outer In walk + inner per-bucket aggregation, returning Vec<(key, count, sum)>. Q7 specifically uses the PCPS carrier (verify_aggregate_count_and_sum_query_per_key); Q6 uses a CountSumTree-carrier on a point-inner subquery (since the per-student-per-semester cohort is a point, not a range — semester == 20241 doesn't need PCPS).

Query 1 — Unfiltered Global Average

select  = AVG(score)
where   = (empty)
prove   = true

Path query (primary-key CountSumTree fast path; no index walk needed):

path:         ["@", contract_id, 0x01, "grade"]
query items:  [Key(0x00)]

Verified result:

path:        ["@", contract_id, 0x01, "grade"]
key:         0x00
element:     CountSumTree { count_value_or_default: 31620, sum_value_or_default: 2392808 }
average:     2392808 / 31620 = 75.6739…

Proof size: 622 bytes. Avg time: 25.3 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (5 layers — primary-key fast path) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
          lower_layers: {
            0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
                  1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
                      lower_layers: {
                        grade => {
                          LayerProof {
                            proof: Merk(
                              0: Push(KVValueHashFeatureTypeWithChildHash(0x00, Tree(00000000000067cfffffffffffff983000000000000000000000000000000000), HASH[75d7cea7fe7cf4c112fe2d080d01417ade8169dffc00eac8cac7923fe4504951], BasicMerkNode, HASH[1f4bee393167bbeff921a8d577c20ab4939af57ce0f5835255ccc92538485f8d]))
                              1: Push(KVHash(HASH[08b88f8f4f1c20303d3be9c78935c7cdd6de33bdc4edc808ff8ddde0e2f3ec66]))
                              2: Parent
                              3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
                              4: Parent
                              5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
                              6: Child)
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Diagram: per-layer merk-tree structure

See the GroveDB Layout diagram for the overall storage shape. Q1's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q1 uses only the constant-prefix path layers down to the doctype's primary-key tree element. The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.

The descent stops at the doctype's primary-key tree — the green node at the top of the layout. Because documentsCountable: true + documentsSummable: "score" upgraded that tree to a CountSumTree, the count and sum are both one O(1) read with an O(log n) proof. The client divides locally to get the average. Same proof shape as count's Q1 and sum's Q1 individually — the CountSumTree just commits both fields at every merk node it walks, costing a constant ~8 extra bytes per descent layer vs. either single-axis variant.

Query 2 — Average for One Class (byClass)

select  = AVG(score)
where   = class == "MATH101"
prove   = true

Path query:

path:         ["@", contract_id, 0x01, "grade", "class"]
query items:  [Key("MATH101")]

Verified result:

path:        ["@", contract_id, 0x01, "grade", "class"]
key:         "MATH101"
element:     CountSumTree { count_value_or_default: 1000, sum_value_or_default: ≈50000 }
average:     ≈50000 / 1000 = ≈50.0

Proof size: 801 bytes. Avg time: 32.1 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (6 layers — byClass point lookup) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
          lower_layers: {
            0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
                  1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
                      lower_layers: {
                        grade => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[beefc1778aa2b24de9a979d69add4f02fe376983ff85d287462ec5e34dc1f764]))
                              1: Push(KVValueHash(class, Tree(4348454d313031), HASH[25ffaf63d65ed1b63f796004c15bdf33757a4b86e3bcde03f67df9c9d42d2168]))
                              2: Parent
                              3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
                              4: Parent
                              5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
                              6: Child)
                            lower_layers: {
                              class => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[221cc4921243629c99fbf517a7f8a93aa3d2f894537bb8a071ed1d46abe64a02]))
                                    1: Push(KVHash(HASH[97aae10e38ef5c482deddc58a643a50a9f27d23876d8327458c163cfbda2da9a]))
                                    2: Parent
                                    3: Push(Hash(HASH[c1b9be8b2629a4cfceb100f6c9e40a5e798dc0c5782e4ce9722f9a4747753711]))
                                    4: Push(KVHash(HASH[34ebec873b25c565a93d25e70507b749406b80b014cfa4ec70d1108e44a62cb0]))
                                    5: Parent
                                    6: Push(Hash(HASH[2e7c7f97470f615c1348e70489c8e1a25c823b88cbf4ecb20db8f05256231211]))
                                    7: Push(KVValueHashFeatureTypeWithChildHash(PHYS101, CountSumTree(73656d6573746572, 1508, 84598), HASH[ac1dccf6426a8467d1b923ebc24e15fb8ac504072fe20b136f1dee0ea7ec2073], BasicMerkNode, HASH[116a0e0b1328cc8529ed2cd4326afbf4e9ae37d15f178d582261d08c2a3894dc]))
                                    8: Parent
                                    9: Push(Hash(HASH[c9ef06be93f8ae382500c13ce025a9920ded466cd4794555d8eb1f6b5a4749e4]))
                                    10: Child
                                    11: Child
                                    12: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Diagram: per-layer merk-tree structure

See the GroveDB Layout diagram for the overall storage shape. Q2's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q2 adds one extra layer into the class property-name subtree and stops at the PHYS101 terminator. The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.

The descent walks one extra layer into the class property-name subtree and stops at PHYS101. The verified result is count=1 508, sum=84 598, avg=56.099 — PHYS101 is one of the harder classes in the bench's profile table — class baseline of 60 minus a slight negative average across all enrolled students' skills puts the verified average at 56.099. Notice the count (2 281, not 5 000) — that's the enrollment filter at work: only ≈ 30% of (student, semester) slots enroll in PHYS101 per the popularity table, so 500 students × 10 semesters × 30% ≈ 1 500 enrolled. The actual 2 281 falls above that because some students bunch up on PHYS101 in certain semesters and the hash-based filter isn't perfectly uniform — that asymmetry is reproducible and visible in the verified count. Because byClass declares both countable: countable and summable: "score", that node is a CountSumTree carrying both per-class metrics directly — no need to step into [0] to look at individual references. Same shortcut count proofs and sum proofs take, just with one element committing two fields rather than two elements committing one each.

Query 3 — Student GPA (byStudent)

select  = AVG(score)
where   = student == student_050
prove   = true

Path query:

path:         ["@", contract_id, 0x01, "grade", "student"]
query items:  [Key(student_050)]

Verified result:

path:        ["@", contract_id, 0x01, "grade", "student"]
key:         student_050
element:     CountSumTree { count_value_or_default: 100, sum_value_or_default: ≈5000 }
average:     ≈5000 / 100 = ≈50.0

Proof size: 1091 bytes. Avg time: 42.0 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (6 layers — byStudent point lookup) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
          lower_layers: {
            0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
                  1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
                      lower_layers: {
                        grade => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[2fa3be1c9771e5c4a10dfe3b5a7dbad43a2775c3822e77cb03ada370f92d608c]))
                              1: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
                              2: Parent
                              3: Push(KVValueHash(student, Tree(00000000000000d1ffffffffffffff2e00000000000000000000000000000000), HASH[24622f7d7a9da5318a2043bb2cb483c7222ad01aa2b24ebe967111ca5e7977cf]))
                              4: Child)
                            lower_layers: {
                              student => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[6759bd80220fbb50622101bf21cd6c3c2d9bd4832dbe04ef85bc8f673674ce39]))
                                    1: Push(KVHash(HASH[71b3c529ec4ef9a110626a15592a2f49cd362729d6c17772844efbfa184e9693]))
                                    2: Parent
                                    3: Push(Hash(HASH[3fb3fed5141b18c53c15769697eec36e067e94191e56d77fbb507239deacf868]))
                                    4: Push(KVHash(HASH[95a8d0469cc6e021c5db7c5d9b429531b948636b0209140cc1dc0a1b305af7f5]))
                                    5: Parent
                                    6: Push(Hash(HASH[dc602dc298843d5c96e4ff87d27ddeed9d9d3a5a0715c8ea96954c5dd075befc]))
                                    7: Push(KVHash(HASH[b862f6f003cc80580d7e980a229658d28ed75d9739c49b443b1d9fdfcf8ece19]))
                                    8: Parent
                                    9: Push(KVValueHashFeatureTypeWithChildHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 4289), HASH[1294d46e235be085d6fa9a0acd1beca9bb7e22e470e0705dd512c4cdbce5e312], BasicMerkNode, HASH[b4d5a93458113eb96afd5a80371df3cd7f8d8296de229a1e8b19dd7a7aec5ffe]))
                                    10: Push(KVHash(HASH[094f3bfd41b7722c90f35dcd4a5e1c68d1a7667043ae56059e232b1045852a45]))
                                    11: Parent
                                    12: Push(Hash(HASH[a73503263b84cc37101581a6eec94dd3dbeec4c437a7eac0ffded8e29d820567]))
                                    13: Child
                                    14: Push(KVHash(HASH[f4b374ae9d1f2bd74405661b8c79f1d1d4342670311b2a7244918af89142f721]))
                                    15: Parent
                                    16: Push(Hash(HASH[cf41f288ea7730eb52ad5547370c4340b836057822ca5ff52356091ae65e03ef]))
                                    17: Child
                                    18: Child
                                    19: Child
                                    20: Child
                                    21: Push(KVHash(HASH[d1bf190eafbd359612378043df0b00938c4070422ec3301d8160535ce62369d9]))
                                    22: Parent
                                    23: Push(Hash(HASH[6cc749152286a0f937958e75818a94f1506108cbee30147700f1233e9cf3684f]))
                                    24: Child
                                    25: Push(KVHash(HASH[93a9ff43d512bc697b94d3d13b8e4be942c6d2efab9e97ac4cd5b80404f9c84d]))
                                    26: Parent
                                    27: Push(Hash(HASH[fa08edb289995985b0f169c6eb2e073414994839941262ae58be3d90277e7029]))
                                    28: Child
                                    29: Push(KVHash(HASH[c7c39a5ae845f2ff491f91fe178d7230364a72db37cb13b3864809e2d8ca7041]))
                                    30: Parent
                                    31: Push(Hash(HASH[0cac6b8e6bd604e6dac04ea4bf84d0304e39e3a0e811eb257eac05587bdf23fa]))
                                    32: Child)
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Diagram: per-layer merk-tree structure

See the GroveDB Layout diagram for the overall storage shape. Q3's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q3 has the same shape as Q2, just over a different property-name subtree. The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.

Structurally identical to Query 2 — different property-name subtree (student instead of class), different terminator value, same CountSumTree element shape. Verified count=62, sum=4 289, avg=69.18 — student_050's GPA across the 62 grades they happen to be enrolled in. The count of 62, not 100, is the enrollment filter showing up — student_050 didn't enroll in every class every semester. With the realistic-data fixture, student_050 turns out to be slightly above average (avg=69.18 vs. the global ≈ 72 baseline heavily pulled up by ENGL101+ARTS101 enrollment) — the FNV hash of 50 happens to land in the positive-skill region of the student distribution.

Query 4 — One Cohort (byClassSemester point)

select  = AVG(score)
where   = class == "MATH101" AND semester == 20241
prove   = true

Path query:

path:         ["@", contract_id, 0x01, "grade", "class", "MATH101", "semester"]
query items:  [Key(serialize_value_for_key("semester", 20241))]

Verified result:

path:        ["@", contract_id, 0x01, "grade", "class", "PHYS101", "semester"]
key:         serialize_value_for_key("semester", 20204)
element:     ProvableCountProvableSumTree { count_value_or_default: 147, sum_value_or_default: 8114 }
average:     8 114 / 147 = 55.197

Proof size: 1233 bytes. Avg time: 51.0 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (8 layers — byClassSemester compound point lookup (PCPS terminator)) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
          lower_layers: {
            0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
                  1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
                      lower_layers: {
                        grade => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[beefc1778aa2b24de9a979d69add4f02fe376983ff85d287462ec5e34dc1f764]))
                              1: Push(KVValueHash(class, Tree(4348454d313031), HASH[25ffaf63d65ed1b63f796004c15bdf33757a4b86e3bcde03f67df9c9d42d2168]))
                              2: Parent
                              3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
                              4: Parent
                              5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
                              6: Child)
                            lower_layers: {
                              class => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[221cc4921243629c99fbf517a7f8a93aa3d2f894537bb8a071ed1d46abe64a02]))
                                    1: Push(KVHash(HASH[97aae10e38ef5c482deddc58a643a50a9f27d23876d8327458c163cfbda2da9a]))
                                    2: Parent
                                    3: Push(Hash(HASH[c1b9be8b2629a4cfceb100f6c9e40a5e798dc0c5782e4ce9722f9a4747753711]))
                                    4: Push(KVHash(HASH[34ebec873b25c565a93d25e70507b749406b80b014cfa4ec70d1108e44a62cb0]))
                                    5: Parent
                                    6: Push(Hash(HASH[2e7c7f97470f615c1348e70489c8e1a25c823b88cbf4ecb20db8f05256231211]))
                                    7: Push(KVValueHash(PHYS101, CountSumTree(73656d6573746572, 1508, 84598), HASH[ac1dccf6426a8467d1b923ebc24e15fb8ac504072fe20b136f1dee0ea7ec2073]))
                                    8: Parent
                                    9: Push(Hash(HASH[c9ef06be93f8ae382500c13ce025a9920ded466cd4794555d8eb1f6b5a4749e4]))
                                    10: Child
                                    11: Child
                                    12: Child)
                                  lower_layers: {
                                    PHYS101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[dd04eaab3acc2eb21101895d29f716fcc264adec3ee3c3d0828b68b2b1efb6a1]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 1508, 84598)), HASH[80f59d6ce839fd72da96b8d1b228172a9e80f4df9f8f9e078a87224943b8f816]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[b1b868b65a239d4256d28919204c86a5f26f9dea470eb984e4884c8801174265]))
                                                1: Push(KVHashCountSum(HASH[1d46fcc02a25b527028f49be8a58c891b3f19588348ac092a4bd446c85733c64], count=1508, sum=84598))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 147, 8114), HASH[ba3ae5cda415f7540cec982bd8403174f8b80ce2604463c630b6505692c4f0e4], ProvableCountedAndProvableSummedMerkNode(147, 8114), HASH[cc3c59428d6ace408733b850eaf8b58c974339d87dabd84f3efe6e62557ab17a]))
                                                4: Push(KVHashCountSum(HASH[2c145ca977d9a5a546df9eb3aaa67475e4004f60902c3dc4322ecbedafa88a6e], count=457, sum=25605))
                                                5: Parent
                                                6: Push(Hash(HASH[f8b2fa764a8fd989881e69b4e8570ada5d5846131b0af9c2770c5e9029b87f9c]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[798b0509467547a7cbc0e666a3afccc36a58c9552491816733ee23483830cad5], count=906, sum=50770))
                                                9: Parent
                                                10: Push(Hash(HASH[95c1dec4eaa2a4489d17d6d492030369abb7c7c2aa8328411017f70f31441e50]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Diagram: per-layer merk-tree structure

See the GroveDB Layout diagram for the overall storage shape. Q4's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q4 has two extra layers over Q2 — one for the byClassSemester continuation's semester subtree (yellow PCPS class), one for the per-cohort terminator (also yellow PCPS). The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.

Two property-name descents (class, then under PHYS101 the byClassSemester continuation's semester). The terminator here is a ProvableCountProvableSumTree (PCPS), not a CountSumTree — that's because both rangeCountable: true and rangeSummable: true on byClassSemester upgrade not just the property-name tree but also the per-value cohort terminator to PCPS. (The chapter's earlier draft said CountSumTree; the bench reveals the dispatcher actually picks PCPS for any value tree under a range-bearing index, so we get PCPS's per-node aggregation even for a point lookup.) For our purposes here — extracting (count, sum) from one merk element — PCPS and CountSumTree are equivalent at the read site; PCPS just carries the extra per-node fields that Query 5's range walk needs. Verified count=147, sum=8 114, avg=55.197.

Query 5 — Class Trend (AggregateCountAndSumOnRange)

select  = AVG(score)
where   = class == "MATH101" AND semester > 20210
prove   = true

Path query:

path:         ["@", contract_id, 0x01, "grade", "class", "MATH101", "semester"]
query items:  AggregateCountAndSumOnRange(RangeAfter(serialize_value_for_key("semester", 20210)..))

Verified result (returned by GroveDb::verify_aggregate_count_and_sum_query):

(root_hash, count, sum) where count = 759, sum = 42 656
average:                 42 656 / 759 = 56.200

(759 grades in range — 5 semesters × roughly 150 enrolled students per semester for PHYS101, matching the documented 30% popularity. The enrollment filter is visible in every range query.)

Proof size: 1469 bytes — O(log T') regardless of how many semesters lie in the range, because AggregateCountAndSumOnRange collapses the boundary walk into a single committed (count, sum) pair at proof-generation time. Same proof-size profile as count's Query 7 and sum's Query 7, with an extra i64 per merk node for the sum field on top of count's per-node count field. Strictly smaller than two independent proofs would be (which would each carry the merk descent overhead separately, plus the client would have to verify two root-hashes match).

Avg time: 49.7 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (8 layers — AggregateCountAndSumOnRange collapse on PCPS continuation) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
          lower_layers: {
            0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
                  1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
                      lower_layers: {
                        grade => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[beefc1778aa2b24de9a979d69add4f02fe376983ff85d287462ec5e34dc1f764]))
                              1: Push(KVValueHash(class, Tree(4348454d313031), HASH[25ffaf63d65ed1b63f796004c15bdf33757a4b86e3bcde03f67df9c9d42d2168]))
                              2: Parent
                              3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
                              4: Parent
                              5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
                              6: Child)
                            lower_layers: {
                              class => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(Hash(HASH[221cc4921243629c99fbf517a7f8a93aa3d2f894537bb8a071ed1d46abe64a02]))
                                    1: Push(KVHash(HASH[97aae10e38ef5c482deddc58a643a50a9f27d23876d8327458c163cfbda2da9a]))
                                    2: Parent
                                    3: Push(Hash(HASH[c1b9be8b2629a4cfceb100f6c9e40a5e798dc0c5782e4ce9722f9a4747753711]))
                                    4: Push(KVHash(HASH[34ebec873b25c565a93d25e70507b749406b80b014cfa4ec70d1108e44a62cb0]))
                                    5: Parent
                                    6: Push(Hash(HASH[2e7c7f97470f615c1348e70489c8e1a25c823b88cbf4ecb20db8f05256231211]))
                                    7: Push(KVValueHash(PHYS101, CountSumTree(73656d6573746572, 1508, 84598), HASH[ac1dccf6426a8467d1b923ebc24e15fb8ac504072fe20b136f1dee0ea7ec2073]))
                                    8: Parent
                                    9: Push(Hash(HASH[c9ef06be93f8ae382500c13ce025a9920ded466cd4794555d8eb1f6b5a4749e4]))
                                    10: Child
                                    11: Child
                                    12: Child)
                                  lower_layers: {
                                    PHYS101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[dd04eaab3acc2eb21101895d29f716fcc264adec3ee3c3d0828b68b2b1efb6a1]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 1508, 84598)), HASH[80f59d6ce839fd72da96b8d1b228172a9e80f4df9f8f9e078a87224943b8f816]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[4b85291fe8e5cae442614096553956521bb55510873f56ea4219d9a56d01408d], left=HASH[49700ad27bc9ac383b5bb5e867114f76acf4701ad7ea97311e12e877e92ffda9], right=HASH[5e9ff5741419a71daf0163e97389cf154aaea23144a589417a4a76e8df5904b4], count=455, sum=25572))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[fe6c93990c99748cec859c40fe458a191825c0941cd064d18eeaec17e52e0999], count=1508, sum=84598))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[ba3ae5cda415f7540cec982bd8403174f8b80ce2604463c630b6505692c4f0e4], count=147, sum=8114))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[e19d566cffe1e46af9eabb5b3b040898109a1d9d50a6a1bf87a04f421fd2617b], count=457, sum=25605))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[38be6684aff1201eeec45aedd505d3634fbcda546b158c5ae43e116819f2ea22], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=153, sum=8588))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[06ce5728c482b63134cf57461e9c4248638064a94393f9085842c830ae830381], count=906, sum=50770))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[8c2d715e4be3e576f5c4327d93f37192f71de69c46aa162ab9bcb725d53a3a46], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[de29287042c9ee0874e346cac4e947177e24129a9ec250bb1a31a575da222747], count=302, sum=16922))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Diagram: per-layer merk-tree structure

See the GroveDB Layout diagram for the overall storage shape. Q5's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q5 walks the same 8 layers as Q4 but the terminator is a range-collapse merk-node commit (no individual per-key terminator; the merk-tree's boundary walk produces a single (count, sum) pair via the PCPS per-node fields). The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.

This is the chapter's headline payoff: a single committed (count, sum) pair from one merk traversal of the byClassSemester PCPS continuation. The verifier cryptographically guarantees that both metrics describe the same in-range grades — there's no way for the server to splice a count from one set with a sum from another. The client divides locally to get the verified average. The PCPS leaf-shape primitive requires the terminator tree to be a ProvableCountProvableSumTree; both lighter sum-bearing and count-bearing variants reject the combined primitive at the merk gate.

Query 6 — Per-Student Averages for One Semester (carrier)

select   = AVG(score)
where    = student IN [student_000, student_001, ..., student_009] AND semester == 20241
group_by = [student]
limit    = 10
prove    = true

Path query (carrier-style: outer Query enumerates the In branches, subquery descends through the byStudentSemester semester == 20241 lookup):

path:                 ["@", contract_id, 0x01, "grade", "student"]
query items:          [Key(student_000), Key(student_001), ..., Key(student_009)]
subquery_path:        ["semester"]
subquery items:       [Key(serialize_value_for_key("semester", 20241))]

Because the inner where is semester == 20241 (a point, not a range), the per-bucket terminator is a CountSumTree element — not PCPS. This is the CountSumTree-carrier flavor that returns Vec<(key, count, sum)> by reading the (count_value, sum_value) off each per-bucket CountSumTree, not the AggregateCountAndSumOnRange flavor (which is reserved for range-bucket cases — see Query 7).

Verified result (returned by the carrier verifier):

(root_hash, entries) where entries =
  [
    (student_000, count=7, sum=477, avg=68.14)
    (student_001, count=6, sum=500, avg=83.33)
    (student_002, count=6, sum=496, avg=82.67)
    (student_003, count=6, sum=441, avg=73.50)
    (student_004, count=7, sum=418, avg=59.71)
    (student_005, count=6, sum=459, avg=76.50)
    (student_006, count=6, sum=445, avg=74.17)
    (student_007, count=6, sum=408, avg=68.00)
    (student_008, count=7, sum=587, avg=83.86)
    (student_009, count=6, sum=482, avg=80.33)
  ]
aggregate across all 10 buckets: count=63, sum=4713, avg=74.81

Per-bucket counts vary from 6 to 7 in this 10-student sample (students take a different number of classes per semester depending on which electives the enrollment filter accepts for them). The per-bucket sums spread from 408 (student_007 — drew lower-skill, mostly-hard classes that semester) to 587 (student_008 — drew higher-skill, mostly-easy classes that semester) — a real-data shape.

Proof size: 6581 bytes. Avg time: 304.4 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (8 layers — CountSumTree-carrier × 10 student buckets with point-inner subquery) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
          lower_layers: {
            0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
                  1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
                      lower_layers: {
                        grade => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[2fa3be1c9771e5c4a10dfe3b5a7dbad43a2775c3822e77cb03ada370f92d608c]))
                              1: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
                              2: Parent
                              3: Push(KVValueHash(student, Tree(00000000000000d1ffffffffffffff2e00000000000000000000000000000000), HASH[24622f7d7a9da5318a2043bb2cb483c7222ad01aa2b24ebe967111ca5e7977cf]))
                              4: Child)
                            lower_layers: {
                              student => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(KVValueHash(0x0000000000000000ffffffffffffffff00000000000000000000000000000000, CountSumTree(73656d6573746572, 63, 4228), HASH[cd6a37aaa2c84a9a441fea60b3a0467194fc626f67ec713780f0805c56309a31]))
                                    1: Push(KVValueHash(0x0000000000000001fffffffffffffffe00000000000000000000000000000000, CountSumTree(73656d6573746572, 63, 5010), HASH[967dbf74b3c2e23feee4745f809322c59c73dbad00fa2ceb1fa0d859a376096c]))
                                    2: Parent
                                    3: Push(KVValueHash(0x0000000000000002fffffffffffffffd00000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 5104), HASH[dca989d86ed13f79aecbc1aa4336b6453aa654573fa6746f2a5d58a20e4aa41c]))
                                    4: Push(KVValueHash(0x0000000000000003fffffffffffffffc00000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 4654), HASH[c6e417e5f392321939ad1398786db7d0006a304e6de3e1e0abf82501091633d5]))
                                    5: Parent
                                    6: Child
                                    7: Push(KVValueHash(0x0000000000000004fffffffffffffffb00000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 3652), HASH[68975d6209b052dbc23d7e1e6d9a8423adb3da71ed2fb0a7f634101627eff7f7]))
                                    8: Parent
                                    9: Push(KVValueHash(0x0000000000000005fffffffffffffffa00000000000000000000000000000000, CountSumTree(73656d6573746572, 65, 4664), HASH[5e0e79e266cc34ff7499dda1d82bc6e4f495b60fdb3c1697afdaa27ec5d28851]))
                                    10: Push(KVValueHash(0x0000000000000006fffffffffffffff900000000000000000000000000000000, CountSumTree(73656d6573746572, 63, 4717), HASH[ae1234b1a941bfc827eca9fc3a459f959d85dc54f269f7a7fb29470ae6419f35]))
                                    11: Child
                                    12: Push(KVValueHash(0x0000000000000007fffffffffffffff800000000000000000000000000000000, CountSumTree(73656d6573746572, 63, 4229), HASH[94475fc16687fe817e942bae9255abf688dbd01de1a2bf1d90331bee49e1a71a]))
                                    13: Parent
                                    14: Push(KVValueHash(0x0000000000000008fffffffffffffff700000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 5189), HASH[b7970fefd63e74384790758e760c74913bb83c56caab31cf2941623e4143d745]))
                                    15: Child
                                    16: Child
                                    17: Push(KVValueHash(0x0000000000000009fffffffffffffff600000000000000000000000000000000, CountSumTree(73656d6573746572, 64, 4871), HASH[77f98f78dab2a44de7c016599cf4fff25ede5872b01191241ce8c4dd0e2b4051]))
                                    18: Parent
                                    19: Push(Hash(HASH[eb09a430767a7d4cce9a6b13bcf5c969e10358520b9099433204178a94f77a1a]))
                                    20: Child
                                    21: Push(KVHash(HASH[71b3c529ec4ef9a110626a15592a2f49cd362729d6c17772844efbfa184e9693]))
                                    22: Parent
                                    23: Push(Hash(HASH[927e9b34e1e5f8bf58e89d4967357bb433e8bdd541da24a5e41ebca9ea53ae7a]))
                                    24: Child
                                    25: Push(KVHash(HASH[d1bf190eafbd359612378043df0b00938c4070422ec3301d8160535ce62369d9]))
                                    26: Parent
                                    27: Push(Hash(HASH[6cc749152286a0f937958e75818a94f1506108cbee30147700f1233e9cf3684f]))
                                    28: Child
                                    29: Push(KVHash(HASH[93a9ff43d512bc697b94d3d13b8e4be942c6d2efab9e97ac4cd5b80404f9c84d]))
                                    30: Parent
                                    31: Push(Hash(HASH[fa08edb289995985b0f169c6eb2e073414994839941262ae58be3d90277e7029]))
                                    32: Child
                                    33: Push(KVHash(HASH[c7c39a5ae845f2ff491f91fe178d7230364a72db37cb13b3864809e2d8ca7041]))
                                    34: Parent
                                    35: Push(Hash(HASH[0cac6b8e6bd604e6dac04ea4bf84d0304e39e3a0e811eb257eac05587bdf23fa]))
                                    36: Child)
                                  lower_layers: {
                                    0x0000000000000000ffffffffffffffff00000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[2c76195b7770957718fcb21d464f6229e1e2940005269dace3a4933319706658]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 63, 4228)), HASH[fe3cd77b60b86564f692f5204c324eb5f2a03af6d66273054c208fdaf1978484]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[bd4c9f525f3ac22ecc2f5ae6e00aaee9a32863a41add67f507d1578209e5ccb1]))
                                                1: Push(KVHashCountSum(HASH[ed8239c0bd92fe50785ea501400b4198acc410c2371602dbf13d76f44c16f7d5], count=63, sum=4228))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 7, 477), HASH[ef1d416d8fea679e43170c4d314d6f7523ca70544e575370c474547d542b3f6f], ProvableCountedAndProvableSummedMerkNode(7, 477), HASH[388656200c3ec466a3e066a3a658df8ea428dda1f7ace466a97a9ad4f64b4658]))
                                                4: Push(KVHashCountSum(HASH[2a3a332a185f435475e113e557765ad458171c09280d25940f3e748193690bd8], count=18, sum=1205))
                                                5: Parent
                                                6: Push(Hash(HASH[bddd5899ad6d572d8ff53a7535f1622430fde64dcd845800ef27c4b597f44b25]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[5290e5ef5976a00dc0f8e1e65aecbaaed0e1f98a63f4f572e414388748b6e75c], count=38, sum=2547))
                                                9: Parent
                                                10: Push(Hash(HASH[41e38b943c0bb7ad82fcb5b782c6ea2abc0e0d636f545658fe0892fa752d0255]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    0x0000000000000001fffffffffffffffe00000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[6b01cebbe317609614cd2032188e5ebe456919474223370041a1875ca102d725]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 63, 5010)), HASH[05908d7f182bb177509a9a62f2d8c274e6fae4a407c74f40d4af38bfc66c755b]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[3d4b43d9f0d56e8842381fdd29b46d7706084da492c5c4bcae4c1054938fb16e]))
                                                1: Push(KVHashCountSum(HASH[f2a00fc085b00b74daa4552f3cf56a94fc6b890410197e34af4cc9aa5b776176], count=63, sum=5010))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 500), HASH[d70c0a972023c06cfb5af447b578075b0ad9c62978366711fff09f4755687cb8], ProvableCountedAndProvableSummedMerkNode(6, 500), HASH[f179844799861dfd9d97f5b00088fceaf210e88c810e5cf019c7690d43edbbcc]))
                                                4: Push(KVHashCountSum(HASH[9c88005912fad6fa694c190694b47c990e87f03b76808257770d350622215563], count=18, sum=1443))
                                                5: Parent
                                                6: Push(Hash(HASH[d8f00e8973512018a5d313855bd512a1386db3df94bbe2880a36cfacdc5db8cb]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[79e976ee63058e02f844a3072df96fcb09d9b520dd473d7decf63b1a7f351a29], count=37, sum=2964))
                                                9: Parent
                                                10: Push(Hash(HASH[c3f3fb7bb10a6930ed23d7c10b1ea4c27194084468c3917a146ad506cd58c43e]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    0x0000000000000002fffffffffffffffd00000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[d625fc067da88f86b5d7bcaaef342e82df5146329ae2522d5806feaf48c5a9ed]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 62, 5104)), HASH[c894416ed05d217104355f7e5acc5deaf090f96158c168d8956c96814a82c763]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[3781ba0dbd202a90681259edd62e32d7914dde189d81c3a6b234b2b888d32bbe]))
                                                1: Push(KVHashCountSum(HASH[e349a23b2e008a6f9a3eb5ba687f84e5007f4bc42f4e1f7135612ed3321bb400], count=62, sum=5104))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 496), HASH[b08ccf15548715cc0ed999ef1ca9be1ecfb302fc3e39b600d1b089decb91e9c4], ProvableCountedAndProvableSummedMerkNode(6, 496), HASH[1b5687189cf542500b393af906e4709dcfe0157c6a5cbc2599b05807f52a3962]))
                                                4: Push(KVHashCountSum(HASH[a156211e0a1b4c14722886364a171f03e3a692fce93b279270486a3606023611], count=18, sum=1471))
                                                5: Parent
                                                6: Push(Hash(HASH[9c191caa12c32abb856347c6f590530642be1f12cce3c26a48f701fdacffcd74]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[28a46c7a36502f4688cda1fd8ebf4fac49a1796c31f5f644f8e9ab54461750a6], count=37, sum=3029))
                                                9: Parent
                                                10: Push(Hash(HASH[492b7dcf673e717285fe90683bb91958b9751170757f2527992d998a2a4d1934]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    0x0000000000000003fffffffffffffffc00000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[246fd08691b346678fb5c4f97cc6a74ad81a9901f15bee494a042c616ed1308a]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 62, 4654)), HASH[c4be1dbc0307777727c7d01d4c67cb4aead93899cef3c50b00dcb7c082d27a35]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[10e09dc243d6103d83938e3cedfd605cd5277d6ef3c0e4fc2da6bae7182ca44a]))
                                                1: Push(KVHashCountSum(HASH[759c768a1ff2180270af1e71443e480cb31a0d39a7b7df65827cd879045e3515], count=62, sum=4654))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 441), HASH[fd79150f43d983e625db22b2bcfb8a7eb320d82a2d70815b4dc16f09ce71eadf], ProvableCountedAndProvableSummedMerkNode(6, 441), HASH[a271f341e8a3ee353a157275c10a5b82e245519857f6873d4caa41ad2539926c]))
                                                4: Push(KVHashCountSum(HASH[9736bf15a3a713f92a9c13ba5f3fb8d3e54c8ef19b576fb850812afbbaf82a35], count=19, sum=1399))
                                                5: Parent
                                                6: Push(Hash(HASH[a748ca0363c702a22dcf7f8864d9aa45aea064db161e8a130b4e8d422f938b11]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[eede689eb26f342042ea8bac1bc9488d2c5eda1c95334ef5d47331f79f85638a], count=38, sum=2857))
                                                9: Parent
                                                10: Push(Hash(HASH[4637407982bd38e1f1ca8a8695616ca8ecb3f176aef0ac405857893f943c9da5]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    0x0000000000000004fffffffffffffffb00000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[058460675a738a0ac14d44dad1dc32d2fa85528394520a469f487d5cb4e26c0b]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 62, 3652)), HASH[4d4bf3754db9a87bf5afbaff143b0fc0082a786785a5d65ef211cadc975426cc]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[ac382aca72f4dc7d7ecd29f1a805ef731f459ae014d9436070b8f30a51ab251b]))
                                                1: Push(KVHashCountSum(HASH[bb7c46cfba7f651cb55026df9f02828eef7ff090bc7f3cbfca45084be6d2594d], count=62, sum=3652))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 7, 418), HASH[67fd6a495071fbad4f42632b871c4c3461291e6dd6a64d2032dc4e540040c0ff], ProvableCountedAndProvableSummedMerkNode(7, 418), HASH[d43bef142d6e74ae7699e5c8b62c08eb49ef652b87633ef6462b9403ccc78c91]))
                                                4: Push(KVHashCountSum(HASH[5341577287cc4b1dad09da63cde5957f85ab343451ce8910a016f624f77f50c3], count=18, sum=1050))
                                                5: Parent
                                                6: Push(Hash(HASH[3d96b9320c8177ea8533fe4f8be62b1cd83a0d058847717e0c466fb0e9d6b817]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[c5fb037894fdbb4f0ec82c7fcd24857003bc14f07b9c721ca7e8d073751c834f], count=38, sum=2209))
                                                9: Parent
                                                10: Push(Hash(HASH[276371d55550ba8e2fd6744b0d2a87f5849bf12cb2819e7fcf8cf9405acddc5a]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    0x0000000000000005fffffffffffffffa00000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[861d84a680ab44ccca0533eb5a1e90318e0e4161216e6a3292f01c37ae4d4286]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 65, 4664)), HASH[14cdcd21b12c66c58bcf74f04d2c714ff7d504c10dfd948e396ef2455dfc4ada]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[df54061a16d1fa7dc0ec2581d4221b76665abcf044b03ba0562b48484b78a1be]))
                                                1: Push(KVHashCountSum(HASH[0729dabe2fdce7dcd77ae5ba3a7e6b8e1377ed4264bf84e35eb029e89a8712d6], count=65, sum=4664))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 459), HASH[e89878a22d32aa9a3930dcaac98ae9c8713fe9826b77e4efc79bd3a70c934c52], ProvableCountedAndProvableSummedMerkNode(6, 459), HASH[1f098d52ad1de6f7e07b5fac53755156209b095a642511186a0caf236c0814d9]))
                                                4: Push(KVHashCountSum(HASH[bf3677858e1625685bb7bd9b5c038d95cb518ca2fda87d0dd3c9838f8c5341d7], count=20, sum=1438))
                                                5: Parent
                                                6: Push(Hash(HASH[1b300f7ced2cbf471e4c0092fe2658d339a54976aa26eb10844f8ad193d2994d]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[0ef7fea6dfc6ac9c5f60458487249c0cd7accfc498c7347074a8bab1ac455f77], count=40, sum=2887))
                                                9: Parent
                                                10: Push(Hash(HASH[1845b2da05e252a4f7f9b942ab4b196099da84795a8867fe9d61faf3e56c61a7]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    0x0000000000000006fffffffffffffff900000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[f084f3761d0377eaa616db0cec66b860f00ca16411d31607c03db41ec0e767ef]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 63, 4717)), HASH[9bd1ce241ecf93ef09d46632b83d86b1795532975cb6c2232eaad76e4f8fb706]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[d8cf1140b01c2b5af25587aa28a224dc43ea3fd590872bfa97ab3283a9cbf922]))
                                                1: Push(KVHashCountSum(HASH[30107a131a8e6fca131ca98ba42eb66ab46c2f7b118b563c41a7988a4c5b1ef0], count=63, sum=4717))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 445), HASH[148e907ea57790d36cb08b030a994e65d06370bf0afd1db9b53d4e3ceb66afdb], ProvableCountedAndProvableSummedMerkNode(6, 445), HASH[7422eef7fe3dbcf1333e6db4e467d593597863103e24afcbd9a2b2898e47a8d6]))
                                                4: Push(KVHashCountSum(HASH[93385238d3e372e4a276f9e69b38c5fc67a4b9ce1947b022a5ede6d14dfda46d], count=18, sum=1341))
                                                5: Parent
                                                6: Push(Hash(HASH[98c836393f4fde21842a04cd8ff9b6679bb1a0716c35bc1d8a0dd40174756ce5]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[8da039f749cbc46c3e4e520654c56615cac9b709fb44e547085bac85156f858c], count=38, sum=2831))
                                                9: Parent
                                                10: Push(Hash(HASH[1be2c53a013501c2a147eda502c2a25038ecf0b00f2cf2a9b864fcfb83b68990]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    0x0000000000000007fffffffffffffff800000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[6757edb3583519e40a8b5ff066873865d5ecc3b4b73d18a40591ee1d4bc0b8a0]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 63, 4229)), HASH[af29fceb2e25e354a76a08b00fd2ee00523380af35d2e8827f371318ac960978]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[516484dcfeb93b608305f68199d907e51637443c48fa3f0f6d865447e5fc82f0]))
                                                1: Push(KVHashCountSum(HASH[a86b417149127c4bcddf1f5dce95e9cb747073bec1b364b5cb332bc55880fa0f], count=63, sum=4229))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 408), HASH[48d6c9c3fbf427e0b8e1fba708a86268476615902653f46016a965100bffe86b], ProvableCountedAndProvableSummedMerkNode(6, 408), HASH[43c64ffaf3b718d8c591f6fbdb6360c3a4152dae040fe865cd53c46bc7561531]))
                                                4: Push(KVHashCountSum(HASH[c87a344cd9e3689337ccb065d53ea6e19bd3ce9de711086b7c08c574688dc1c7], count=19, sum=1261))
                                                5: Parent
                                                6: Push(Hash(HASH[b319c8cc34211d304bcbab053b0131d46ae7b6f22bfcbfebabdde0706e088395]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[90ddec9fcea82565519f269eae8ec8b79d88f2538a186fcad818c04d5d818ee2], count=39, sum=2615))
                                                9: Parent
                                                10: Push(Hash(HASH[abca4d83b4096d1e352c5e50e67359d48554427198f6ed516fdc8bcf7415f939]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    0x0000000000000008fffffffffffffff700000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[ccbdeec0f90a04cf1b65d705d8a07fa57f80e136a28597654daf58f791825228]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 62, 5189)), HASH[aaf52bcba6c0523789d5ace2627c84f70d71681e9a666dd16f16055cf2044f1b]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[ffcdc9eb6e6961c65debc526347912280d6fd20328dc419de0cc47c24d9a598b]))
                                                1: Push(KVHashCountSum(HASH[948c18e24a93d0cc101ad88f3de03078b3cac06141fa57c1f2fc2a4f985dbf28], count=62, sum=5189))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 7, 587), HASH[a061e5b6e0fc65b66786f63efd670aad881ef6a8731ffe84013e9829c715512b], ProvableCountedAndProvableSummedMerkNode(7, 587), HASH[ce13e20b17298a1ec9bf5cc7a6bcb1c7c41589592412eebfd18ea27bb4255a71]))
                                                4: Push(KVHashCountSum(HASH[55b70d14ab3825c49c1089d50278cc81c20e712b5c90cd11805963a52653daf0], count=18, sum=1519))
                                                5: Parent
                                                6: Push(Hash(HASH[5ccefdc0d8741cf7eea2af8c3f9ea8ea76d9b5392de3fb41f51f77e384ce0bd4]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[5a29557263bb375348d8f2f535fbe0b5c69bb9da18e05bb55ad2a6026a5a816d], count=38, sum=3180))
                                                9: Parent
                                                10: Push(Hash(HASH[d7f0f3a2db6f5b249dde2f48fde3ce72befbac78fa78cb369134626eba6396af]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    0x0000000000000009fffffffffffffff600000000000000000000000000000000 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[5cebb1f5c631c08b1b27a473ae8bd6b89596aee1ab4049a9652ac1da0829890d]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 64, 4871)), HASH[b66d1c5e201c8c81e8895b44dc42cbd4af377b2bc7028c02ca1ecca0d8f6ddf5]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(Hash(HASH[285b3c7ec0ff52c2ae2abcd18c4e174b55cc5fdbe6ae56669552c2b5c491413c]))
                                                1: Push(KVHashCountSum(HASH[f2a55f39d06bdd502fa50a8430176d11b3edfd7dce73fae1d535dcf59d5724ea], count=64, sum=4871))
                                                2: Parent
                                                3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 482), HASH[7b5b2db33170ef674e072a3d470d60d08e8af6dd30fc99f3817ceda5ba4330a4], ProvableCountedAndProvableSummedMerkNode(6, 482), HASH[2644743dddd29928dd9325b237025f14221b6470c45fde808dad21b55abc8d4b]))
                                                4: Push(KVHashCountSum(HASH[a5cc999850410d4709b4b7160234990c4d53ac46f1e442aad6bf7bd451f6a43a], count=20, sum=1522))
                                                5: Parent
                                                6: Push(Hash(HASH[6054de4db622ca55074de67018493815d0fff63fb2d2659b000c91f7a43560d1]))
                                                7: Child
                                                8: Push(KVHashCountSum(HASH[23c949136919913400a75afc81c327673017844848b17bd9b53ce8588fc9a09f], count=39, sum=2981))
                                                9: Parent
                                                10: Push(Hash(HASH[e7f34b24fd8bdddc86e68d6237e8fe8bbeaa15dd3cf86a746ad4fc6f68fdf06b]))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Diagram: per-layer merk-tree structure

See the GroveDB Layout diagram for the overall storage shape. Q6's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q6 fans out at the student layer to 10 cyan student-id terminators (one per outer In branch). Under each terminator, the inner subquery walks one more layer (the byStudentSemester continuation's semester) and lands on a PCPS terminator (yellow) carrying that cohort's (count, sum). The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.

The carrier composition saves N round-trips: one proof returns averages for all 10 students simultaneously, vs. issuing 10 independent Query-4-shape proofs and dividing client-side per bucket. The verifier walks one outer descent (through student) and gets 10 per-bucket (count, sum) pairs in a single root-hash-committed payload.

select   = AVG(score)
where    = class IN ["MATH101", "PHYS101", ..., "ENGL101"] AND semester > 20210
group_by = [class, semester]
limit    = 10
prove    = true

Path query (PCPS-carrier: outer In over class, inner AggregateCountAndSumOnRange over the byClassSemester semester continuation):

path:                 ["@", contract_id, 0x01, "grade", "class"]
query items:          [Key("MATH101"), Key("PHYS101"), ..., Key("ENGL101")]
subquery_path:        ["semester"]
subquery items:       AggregateCountAndSumOnRange(RangeAfter(serialize_value_for_key("semester", 20210)..))

Verified result (returned by GroveDb::verify_aggregate_count_and_sum_query_per_key):

(root_hash, entries) where entries =
  [
    ( 'ARTS101', count=2267, sum=197461, avg=  87.102)
    ( 'BIOL101', count=1493, sum=103334, avg=  69.212)
    ( 'CALC201', count= 629, sum= 33694, avg=  53.568)
    ( 'CHEM101', count=1130, sum= 69300, avg=  61.327)
    ( 'COMP101', count=1372, sum= 98873, avg=  72.065)
    ( 'ENGL101', count=2500, sum=208872, avg=  83.549)
    ( 'HIST101', count=1764, sum=133138, avg=  75.475)
    ( 'MUSC101', count=2141, sum=171548, avg=  80.125)
    ( 'PHYS101', count= 759, sum= 42656, avg=  56.200)
    ( 'SOCI101', count=1759, sum=137574, avg=  78.212)
  ]
aggregate across all 10 classes: count=15 814  sum=1 196 450  avg=75.658

Each class's bucket has a different count — 629 for CALC201 (hardest math, 25% enrollment), 2 500 for ENGL101 (everyone takes it). This is exactly what real-data carrier-aggregate output looks like: the per-class average is informative on its own, but the per-class count also tells you something — how many students chose that class. The verified per-bucket averages span from CALC201 (53.6 — hardest math) to ARTS101 (87.1 — easiest art), a realistic 34-point spread. This is the chapter's most striking payoff number: one carrier proof returns ten cryptographically-attested averages along with the enrollment-derived counts that contextualize them, all from the same root-hash commit. Doing the same query without the carrier primitive would burn 10 round-trips and 10 separate root-hash matches; the PCPS-carrier collapses it to one proof.

Proof size: 8220 bytes — measured against Query 5's 1 539 B baseline, that's ≈ 5.6× the bytes for k=10 buckets, better than the predicted 6×–10× envelope because the shared top-of-tree merk descent (the first 4 layers down to grade/class) is amortized across all 10 outer Keys rather than walked once per bucket. The per-bucket marginal cost works out to (8 220 − 1 539) / 9 ≈ 742 B per added carrier bucket.

Avg time: 273.8 µs.

Proof display (GroveDBProof::Display):

Expand to see the structured proof (8 layers — PCPS-carrier × 10 class buckets with range-inner subquery) — or open interactively in the visualizer ↗
GroveDBProofV1 {
  LayerProof {
    proof: Merk(
      0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
      1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
      2: Parent
      3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
      4: Child)
    lower_layers: {
      @ => {
        LayerProof {
          proof: Merk(
            0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
          lower_layers: {
            0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
              LayerProof {
                proof: Merk(
                  0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
                  1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
                  2: Parent)
                lower_layers: {
                  0x01 => {
                    LayerProof {
                      proof: Merk(
                        0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
                      lower_layers: {
                        grade => {
                          LayerProof {
                            proof: Merk(
                              0: Push(Hash(HASH[beefc1778aa2b24de9a979d69add4f02fe376983ff85d287462ec5e34dc1f764]))
                              1: Push(KVValueHash(class, Tree(4348454d313031), HASH[25ffaf63d65ed1b63f796004c15bdf33757a4b86e3bcde03f67df9c9d42d2168]))
                              2: Parent
                              3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
                              4: Parent
                              5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
                              6: Child)
                            lower_layers: {
                              class => {
                                LayerProof {
                                  proof: Merk(
                                    0: Push(KVValueHash(ARTS101, CountSumTree(73656d6573746572, 4527, 394132), HASH[6287666053158d702e1cac8b1e0a1b5d019016885d7213c8dc5d0c772079cf97]))
                                    1: Push(KVValueHash(BIOL101, CountSumTree(73656d6573746572, 3002, 207609), HASH[ba63cff57f365956744950b4c9e100ff34da3414eef0031e7260ad7609ad9522]))
                                    2: Parent
                                    3: Push(KVValueHash(CALC201, CountSumTree(73656d6573746572, 1254, 67097), HASH[352d89856ffbed603703a35b44f80c2a2426d71638574dd0d24c1f3a37ae749b]))
                                    4: Child
                                    5: Push(KVValueHash(CHEM101, CountSumTree(73656d6573746572, 2263, 139407), HASH[6180689ce9301e535a022141f4112ad99d0aada6dabb6de27b1e1e5e3462f9be]))
                                    6: Parent
                                    7: Push(KVValueHash(COMP101, CountSumTree(73656d6573746572, 2755, 198793), HASH[1204b674d76ba05a9bd9e32fe9fbb80894b88954394d28c9154d70ef72840e62]))
                                    8: Push(KVValueHash(ENGL101, CountSumTree(73656d6573746572, 5000, 417853), HASH[3abff4fc48017f2b57cd7711cf0076ac0f0cc4084e5add8c974ac33c1e02b6d1]))
                                    9: Parent
                                    10: Push(KVValueHash(HIST101, CountSumTree(73656d6573746572, 3526, 266216), HASH[fc7cc082d6312d5ebce80dc959b0b46829ecae8120b03ecad2e9c2fe55a1b269]))
                                    11: Parent
                                    12: Push(KVValueHash(MUSC101, CountSumTree(73656d6573746572, 4271, 342332), HASH[4ff44ad6d87bdb6e6544c3f98753f5060b0397cf91f34e10f64a30e5b82d1df4]))
                                    13: Push(KVValueHash(PHYS101, CountSumTree(73656d6573746572, 1508, 84598), HASH[ac1dccf6426a8467d1b923ebc24e15fb8ac504072fe20b136f1dee0ea7ec2073]))
                                    14: Parent
                                    15: Push(KVValueHash(SOCI101, CountSumTree(73656d6573746572, 3514, 274771), HASH[7f3e59b54da6cf457c87932ec5b767b1f46dd4f8d94be23adffb0fa507eacf70]))
                                    16: Child
                                    17: Child
                                    18: Child)
                                  lower_layers: {
                                    ARTS101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[b5edb782fcdb15baf23f26a117ead34c39d873294932cb8079f8534ceec5441a]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 4527, 394132)), HASH[523ac5176cd507c4f154e987241c1e9abecf31413ab4e12332b4ae71d0b1a817]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[153f2a3fec67aef426e26ac9744b79b77cd88baed3235fb7c312535c77599fb7], left=HASH[7c038d6e4ed27c1cdea438f56371cf663420601673ffd265a4daec852d212eb8], right=HASH[8cffbb97f331cf75df0742a765fd10bb9f165fbe5933554e1fcb9d90ae9e716e], count=1353, sum=117678))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[8c316d5faff7bfec194ca0b9ee0e27f79cd71b7782f0a88e6ee52a4b6f2c5246], count=4527, sum=394132))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[62afce744a06db0df95c8bd4baf4624759411cf4f5abee3343135ea310c935e4], count=459, sum=39982))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[80b92b87c27dbf6ad7f2d7e8149e48b384eab882f8a78c55bb4e614be6e4a508], count=1361, sum=118550))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[c7a90676ad2028cc2ff92d446e746878fb0f7a583e79a2e6b49b7ce6ac64519c], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=445, sum=38706))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[6cfca633536cf2b5aa9258f275930f8ba031d8ca06cb02444bb80b5b33ffe46a], count=2726, sum=237443))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[12f0875d8a8e80b0a13dd44009c9424948b23289e1d3993745ad1c1764ba9016], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[18853cfbdb647bdc009bd72671a952cf95c3fdac4c9ce373104a70a7d154431a], count=918, sum=79924))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    BIOL101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[badf193cdbc6e6c299c66c548e3ecd8c85708e02879f78ad285ea8aa045602c3]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 3002, 207609)), HASH[7e6c15bd061c84812e5d8670e033f987bb67ff5179614f49b36cc288d7c2eef2]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[edada11476903c0944f218315679a3005273ecf2b4662d7d06390ed9d3e813a3], left=HASH[5fb9a5d7009679ccbdf2917f99a477eace8ff7afcd8efa527b33e81534f3c722], right=HASH[2bab918fd202d2b14075963422be7733ddd6b5724e45d9ba2d6864a58a3aad9d], count=894, sum=61706))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[1099b82842eee73c68184a7ee59977086ae84d99f7e9bb2274566db7c793a35d], count=3002, sum=207609))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[a39eb3ce411c0cedfd5ea8b2cf4e5ffe6539f0d76377feddd33a8b347956f171], count=308, sum=21387))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[93e0ce27c896a0b4813858bb4d75a3e8a601fd03b2379de0bf39fbfd6582f57a], count=895, sum=61926))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[0a1ee4f1f3f89ab1e924cdedf4a51ed0d6f22e941294164fd4b6ce7d8fe49faf], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=293, sum=20246))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[d981cc2ffa042bf112eb9fdfeec150e4ceb00de7a7cb64af0bedb4b2252722fd], count=1801, sum=124721))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[2c034fa7d67d634e69a2ba7d8aa475f1e1db49efc9eebee0dfbf475f194cfd19], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[e2aef757bc42bd4e662b381af0bc697a108aa63b987680cede760afef97221fd], count=602, sum=41689))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    CALC201 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[b3775adbb191d4cbe78a1b86e8aef02cc0e0c8fdcb8dd3580a778792082ce62b]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 1254, 67097)), HASH[f4cb4daf3fdbe3b12a7976fc114cc8157ebe9fbf7c96d128303dff4a34586a7f]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[4df3da5ee89a6442394bc1802324940493b9728464e709f9d2bde478a970bd72], left=HASH[25dec93cd6d566d6c15b4cae3550d0260ba0722d1174959bc3689a7e7cdf1f29], right=HASH[6fcb1dfe38659e9b217175e9b8970b3c7156487fbcdcc9f04cde3758cacea1e5], count=375, sum=20121))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[f904010258c04d68609e279ccffe0de58be26b55ce6699c287a3974d1b1dc972], count=1254, sum=67097))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[baf373e749feedf3dbaea57838e995c3fe92261ed1eed197a2646d0812086369], count=125, sum=6595))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[4a8d8b11d641a21f1ac878e916aa8f3037dedfd78954d275cff2452281cb1560], count=378, sum=20177))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[89f6c530f62e9ebc271188c6b7dbc27f8cc7249c9679c62cf5e427e692446c6c], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=126, sum=6904))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[36949b12299e375b573ccc872578c824e0377b2859ad55640692a73e61c19814], count=754, sum=40289))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[c36238784a0d1a9fa93f17bdb7352b8db84c88af171e00984cc8406406def117], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[a2109aba1a7f9a0731d8386a8f7e670582fe2f9f1c7838a497c5d0f6e111441f], count=251, sum=13507))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    CHEM101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[c3620889aa20c51a24bc3752977653df1df0479f73dd119e19a64e942e0b8e6e]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 2263, 139407)), HASH[e60fa893a8789dbeadc4c882f63661c555f918df604503231038642a4171d890]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[8d9128eb523863f5e8b2a9e89e5dfb21ffde6bb7b577b4d234cee2c5d301b259], left=HASH[1e72254d8454ccc758f039cb1869840918b66671b18fcbe84a3139a53e7d433d], right=HASH[44014c573e0e11f43783c13309f92f2a1f9ac5b4cb4dc4ec7c267d7a89cf04e2], count=676, sum=41772))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[ea0da35647ac220198f5104e6e5ae7deeb5e9996baa7db88195ba87221538481], count=2263, sum=139407))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[13df40fc665d32c820baf598d6a15acb505e66b9a973191f89ded2b4e0c9eb89], count=226, sum=14044))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[a736fddb422d70bc326fed9d27c1cda6c83f59d9d6b18e4da02bb4c079e6fbe2], count=675, sum=41533))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[6a6bbf93e5786b93da68fbe6941f861856a4060c11bd8a4c15116824234e68ba], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=230, sum=14141))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[ab271c52dff2784bb05b5985d5597eac111736b3d65851effe9a05e5e2c2b906], count=1356, sum=83344))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[5de8d50527657bec30c7942113989d84aafae4ebe5ff643a8052950b1451a4c6], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[8ecff26ed936869396fc8c249fd824d1daf87a90721ec63146f3ce987562cd45], count=451, sum=27720))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    COMP101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[c6c4a95162a3df6f9e0c69f267068632bf13098481c94436baac801af7496d91]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 2755, 198793)), HASH[a600eb34c6fd3eb4bc03640c127920a7f5fe9cbecec8d743e1b3abcb50c5e5f7]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[4ebb6c39abe967f06b124c364c8da2ca549cdba6bb8a7d86ca6e121bf3fe2b62], left=HASH[446c3f351eb207fce31d2de9dc02e381a2a98a062bfbcf912d77350c87897b7b], right=HASH[768df3838abcd5784856a8efd8d62c6637a0c04f69077152f301761598a27764], count=828, sum=59850))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[85aa3be7ccddc565d626a7dcaaf2712ba4a584f089936ce912fac36f16b70341], count=2755, sum=198793))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[9ab629c3b6b8919ea13061d58eeaad21e4635757489dc9c05757621c30e0058e], count=281, sum=20237))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[5ab4bccc51b7d6dbefa38d39d8389172ce2544898015e7f58c5a9b133f0d67ed], count=825, sum=59412))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[8a65e954ff97fea886e1c3685e106b0d4b4826a069769607eb825262db5cf431], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=271, sum=19498))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[362a5fa9985d5a5cac8c3e442376abebe1080041ac7a49620107a30e27829125], count=1653, sum=119110))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[113376007a1dda22c2c63348d7e48448a8a990f5204d029a022a9891e919fb60], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0b1eddb8b5d52ed2b6b0a2911b30cd8b0f7126a5f47b104380ebbe6bddea95e8], count=556, sum=40073))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    ENGL101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[56ea1c0de96e4b9f47d107679a1132228ee66ebe5b74fdb4a23a3adaf2652e9f]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 5000, 417853)), HASH[6780d21bd4cd6541f2eb43896f9d39eee0089819b3c5d6392dd16a116ed65eaa]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[ea0fd84c9f08448e30b96634a6ded32cbcc818e672dc180799d0d5c8fe7020ae], left=HASH[8eb17d573a5319468f69f8c671293e1a134b929c6c5be7d426a470357b123aea], right=HASH[059de3a36af4da7fc91e2fd3520816beceb02051cba847c022773013e6437977], count=1500, sum=125427))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[629f76cabb9d728b1c4e25cea6b189815e704afc3a46281e0991c5ef5fc5ea2a], count=5000, sum=417853))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[e2a9429b263d3e81228017f77c47050d01c6ee129c26d5231aa321886de7f951], count=500, sum=41776))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[e8476c17b96c46ef951b68f0079075b5f0d214bfbe07f088a4a6248bc5cbe801], count=1500, sum=125320))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[2ab628820220c346ff82830a69e97b4b12ed1544871e07000382c56be157afc2], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=500, sum=41772))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[5c4b915ebf521938fda7637a08a80b59b7a2f674e57e9f68a6b5e111fd7bbc07], count=3000, sum=250648))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[ab118a65fce19cdea048af30abc34483c055c1be3e846dfa1d86bdb462d9b41f], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[c686c9e613b11d1f098458bfc3c52964abec500965dfa3aaa1ac06ca9903551d], count=1000, sum=83548))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    HIST101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[c6eb6168c1c93d6fa08bf9383d4c564c737d8a2b4a6e46b80155df24a41e27d5]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 3526, 266216)), HASH[79aa05b68fee276dd5cfe031e738d2e89eb58cf786e77e9c93c0dd684628530a]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[8a5495292f3e480d7b9c6bc5b5fde238d2b8c389e20644a0d97f3e481cedf5e1], left=HASH[fba8a121f5fb68782a0f0f43fc1bb8b2102fb164fa73c79e0c3b4adb2214c009], right=HASH[7dcaf93df62b9522a2ba6a2904b4996e0e54406b5f2e6ad0283016fce6752d2f], count=1052, sum=79538))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[c3cccf8d9661d57fda5183e9a79d2ebb59091576a598111836525564b41831ac], count=3526, sum=266216))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[e79e9fb4a9b1b2807225bd61b8e4f9eb86440d8f0e19b49877b338dc53af8a0d], count=356, sum=26850))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[5005889b9794bc0d1093470fec1c7f8a27f286f2093d83a4b7ac663dcfca6128], count=1056, sum=79698))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[324f62b0b57d244cd88260cd456f4dc3b8abc605646bd795b93e3522d8e9934a], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=345, sum=26047))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[64672e9fd6e2a88e83a6b2b7b8e9316ee00943f80518f576d2bd3a793d7af2ab], count=2120, sum=159988))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[f2f4bc42cdd1f7f361b4eca932e3c2ad76772241f487632f87c7e0a99c0d4774], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[e33a2bf146527bb951eee9255d1b4e4a9ffd77a1ad4d24b9ba17d271dec1121d], count=711, sum=53677))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    MUSC101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[44631d7a5550d36e07615836d360ae1b3b54d16ebc3b431b01c07e08779c4fe7]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 4271, 342332)), HASH[fac7326ded3beb763fefbb0854f76765519e1d6d7d63d8460f699354016321f8]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[bb4e98cc506773058c14f6a9910ecabc8c6964c2285d17563994c37239b9ef11], left=HASH[0126c120d418971205f42e999ed15fbaf71f0a592600c32758ae07eb4415171b], right=HASH[43ec6cd8e561f3825397415cd0faea6c8923f43e1d0fd9be2d6ebe6091f9506e], count=1283, sum=102886))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[f98fd031e5c2d2665bb313143f92b6d97ec4fac033c9e1359391f4b61f278e5c], count=4271, sum=342332))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[087aa3e650be9e1b897cb8a7926fafde85c0491a52b5268284ee405d38baa2a1], count=430, sum=34459))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[0d927909305aae61be3a3f80f1eee40894a8910f07bad3c66843fcbb60c5cf1d], count=1289, sum=103275))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[5226ae5de5d883845f2fded38618cd29ea3cc4e28a263dd115149633514179b7], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=423, sum=33900))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[fa8f8dd534ab83fd9a9ad352d7e63eadce07e3d4d95e846b6690a53e50f046e3], count=2571, sum=206007))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[5eca943d71f416b6111de36193238b60e1307591f4738c55b6e0b1e111dd3d53], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[bb11bd1cd03ee81736b06f3808f4d84c6c75435f9f10dd0cd94f5b1a39ff66e0], count=864, sum=69234))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    PHYS101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[dd04eaab3acc2eb21101895d29f716fcc264adec3ee3c3d0828b68b2b1efb6a1]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 1508, 84598)), HASH[80f59d6ce839fd72da96b8d1b228172a9e80f4df9f8f9e078a87224943b8f816]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[4b85291fe8e5cae442614096553956521bb55510873f56ea4219d9a56d01408d], left=HASH[49700ad27bc9ac383b5bb5e867114f76acf4701ad7ea97311e12e877e92ffda9], right=HASH[5e9ff5741419a71daf0163e97389cf154aaea23144a589417a4a76e8df5904b4], count=455, sum=25572))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[fe6c93990c99748cec859c40fe458a191825c0941cd064d18eeaec17e52e0999], count=1508, sum=84598))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[ba3ae5cda415f7540cec982bd8403174f8b80ce2604463c630b6505692c4f0e4], count=147, sum=8114))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[e19d566cffe1e46af9eabb5b3b040898109a1d9d50a6a1bf87a04f421fd2617b], count=457, sum=25605))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[38be6684aff1201eeec45aedd505d3634fbcda546b158c5ae43e116819f2ea22], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=153, sum=8588))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[06ce5728c482b63134cf57461e9c4248638064a94393f9085842c830ae830381], count=906, sum=50770))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[8c2d715e4be3e576f5c4327d93f37192f71de69c46aa162ab9bcb725d53a3a46], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[de29287042c9ee0874e346cac4e947177e24129a9ec250bb1a31a575da222747], count=302, sum=16922))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                    SOCI101 => {
                                      LayerProof {
                                        proof: Merk(
                                          0: Push(Hash(HASH[0d8cfa7c4466e01ca73d4f8ea0615a960ed422fd27b5b80c20dbc0ce854c7390]))
                                          1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 3514, 274771)), HASH[728c9ce73d533772a6e429bbd6319d054bf1e0b7367c3557f02b962911e49665]))
                                          2: Parent)
                                        lower_layers: {
                                          semester => {
                                            LayerProof {
                                              proof: Merk(
                                                0: Push(HashWithCountAndSum(kv_hash=HASH[5e787086a1b267e213e6c64ce21941786a5f2daf6979093884d1923c5110bae8], left=HASH[6f680062b47d1ea1a4826c8323ed0cc398f06a2a8132efbe640ee4504a20d6a5], right=HASH[2098e76b4f886cb4849224d2a6c37f5aba69265f21047475c1d0df6314e9fe05], count=1057, sum=82667))
                                                1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[307bbb9fac832c48ed1147e5d4ab751868c9faf1ab56c1c27f2813f4b27e83de], count=3514, sum=274771))
                                                2: Parent
                                                3: Push(KVDigestCountSum(0x8000000000004eec, HASH[1c92eaceff0d4774119c8cab00d0b8bc313306efbbe62b29b52f724e2ffaa21f], count=346, sum=27033))
                                                4: Push(KVDigestCountSum(0x8000000000004eed, HASH[97a045b4732eb325ea99f0583ac7643eb071320c077d7701e2828ba7342c782e], count=1056, sum=82592))
                                                5: Parent
                                                6: Push(HashWithCountAndSum(kv_hash=HASH[3203e7c385d9ce04e4f1c4c83861eaa5ba9a15df2aa138a21d738cc1cefb532b], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=355, sum=27805))
                                                7: Child
                                                8: Push(KVDigestCountSum(0x8000000000004eef, HASH[9e96b671df5054d6e040c859fe734c850e8101dde071e325a015af45871b2cca], count=2105, sum=164607))
                                                9: Parent
                                                10: Push(HashWithCountAndSum(kv_hash=HASH[7afdf59e0efd5e0695f40a1f1946b336e2304a52b08f5707f36e35f35f5fe6c3], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[21de4684adc3f1fcb126db75204aef9cbdb55ae381c3e37f511e7449b97acffa], count=695, sum=54406))
                                                11: Child
                                                12: Child)
                                            }
                                          }
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Diagram: per-layer merk-tree structure

See the GroveDB Layout diagram for the overall storage shape. Q7's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q7 fans out at the class layer to 10 cyan class-name terminators (one per outer In branch). Under each terminator, the inner subquery walks the byClassSemester continuation's semester (yellow PCPS) and emits an AggregateCountAndSumOnRange collapse for that class's in-range semesters. Each bucket emits one (count, sum) pair. The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.

This is the chapter's most expressive primitive: one proof, k cryptographically-committed (count, sum) triples, each describing a different class's semester-trend average. The client divides per bucket to get k verified per-class averages. Doing this without the PCPS carrier would require k × 2 independent proofs (one count, one sum per class) plus k root-hash matches the client must verify — the carrier collapses that to one proof, one root-hash, and roughly 1/3 to 1/2 the byte cost.

The two carrier-aggregate gates worth knowing (same as the sum chapter's Query 9):

  • SizedQuery::limit caps the outer walk. Mismatched limits between prover and verifier break the merk-root recomputation.
  • SizedQuery::offset is rejected for carrier-aggregate. Skipping outer matches changes which (outer_key, count, sum) triples end up in the proof; the use case isn't designed yet.

Numerical Considerations

A few facts about how Drive handles the arithmetic:

  • Count is u64, sum is i64. Reflects grovedb's per-node field types: count_value is unsigned (can't be negative), sum_value is signed (negative contributions are allowed in general, though the grades contract's score >= 0 constraint prevents them here).
  • No server-side division. The verifier returns (count, sum); the client divides. This is deliberate: integer division loses precision (the average of [1, 2] is 1.5, but 3 / 2 = 1 in integer math), and the right division precision depends on the client's use case (some want truncated integer, some want fixed-point, some want floating-point). The server doesn't pick for you.
  • Overflow risk for sum. Each grade contributes at most 100 to the sum (per the schema's score constraint). For 10 000 grades the maximum sum is 1 000 000 — well within i64::MAX (~9.2 × 10¹⁸). The contract's maxItems constraints on student and instructor cap document size; combined with grovedb's max-tree-size policies, you'd need on the order of 10¹⁶ grades to risk i64 overflow on the sum. For any realistic deployment, overflow is not a concern.
  • Division by zero when count = 0. Possible if the filter resolves to no matches (e.g., a class no one has taken yet in the requested range). The client must handle the zero-count case explicitly — typically by reporting "no grades" rather than computing sum / 0. The proof is still well-formed; it just commits (count=0, sum=0) and the verifier returns those values cleanly.

At-a-Glance Comparison

QueryIndex usedElement shape at terminatorReturned variantProof primitive
1 — Global Average(doctype primary-key)CountSumTree at grade/[0](count, sum)merk path
2 — Average for classbyClassCountSumTree at class/PHYS101(count, sum)merk path
3 — Student GPAbyStudentCountSumTree at student/student_050(count, sum)merk path
4 — One CohortbyClassSemesterProvableCountProvableSumTree at class/PHYS101/semester/serialize(20204)(count, sum)merk path
5 — Class TrendbyClassSemester (PCPS continuation)(collapsed boundary)(count, sum)AggregateCountAndSumOnRange
6 — Per-Student in SemesterbyStudentSemester (point inner)k × CountSumTreesper-key entriesCountSumTree-carrier (k × merk path)
7 — Per-Class TrendsbyClassSemester (PCPS continuation)k × (collapsed boundaries)per-key entriesverify_aggregate_count_and_sum_query_per_key (PCPS carrier)

The split closely parallels the count and sum chapters — point lookups for Q1–Q4, range-aggregate for Q5, carrier composition for Q6–Q7 — with the load-bearing difference that every query's returned shape carries both a count and a sum. The dual-axis primitive surfaces a payload ((count, sum)) that neither the count nor the sum chapter alone can produce in a single proof; the client computes avg = sum / count and gets a cryptographically-attested verified average from one root-hash commit.

What's Next

The chapter is grounded in the document_average_worst_case bench's measured numbers — Q1–Q7 verify cleanly end-to-end against the shared root hash 8b15f732…ffc7.

A natural expansion follow-up (out of scope here): a worked example of "exact-precision" averages — for callers that need fractional averages (e.g. avg = 50.7142857… rather than 50.99), the protocol-level approach is to return (count, sum) and let the client compute in its preferred numeric format (the chapter notes this in Numerical Considerations above; a future expansion could walk through the fixed-point vs. floating-point trade-offs).

No-proof path: joint count-and-sum dispatch

The no-proof AVG path lives in crate::query::drive_document_count_and_sum_query. It consumes the same DocumentAverageRequest the prove path uses and resolves routing through sum's versioned mode-detection table, so the (where_clauses × mode) → executor mapping has a single source of truth shared with the sum and count surfaces. The dispatcher splits on the resolved mode:

  • Total / PerInValue (no-range Equal/In on a summable + countable index) walks the point-lookup path query and decodes (count, sum) from each visited CountSumTree terminator in one call via Element::count_sum_value_or_default(). One grovedb call per In branch, both metrics together.
  • RangeNoProof distinct shapes (GroupByRange / GroupByCompound + range on an index that declares BOTH rangeCountable: true AND rangeSummable: true — DPP exposes rangeAverageable: true as shorthand for the pair) walk ProvableCountProvableSumTree terminators once via the same distinct_sum_path_query builder the sum surface uses, emitting one (count, sum) per distinct in-range key — strictly better than the count + sum surfaces' parallel walks because both metrics come from each visited element.
  • RangeNoProof aggregate shapes (Aggregate / GroupByIn + range) call grovedb's combined merk-internal accumulator directly: query_aggregate_count_and_sum against the PCPS path query, yielding (u64, i64) from a single O(log n) traversal. Compound In + range per-In fans out (≤100 branches per the In::in_values() validator cap) and issues one accumulator call per branch under a shared read transaction. Bounded regardless of how many documents the range matches — keeping the public DAPI endpoint closed against amplification.

The no-prove combined accumulator (query_aggregate_count_and_sum) is the symmetric counterpart of the prove-side AggregateCountAndSumOnRange primitive Q5 / Q7 above use. Both sides walk the same PCPS terminator shape with (u128, i128) accumulators (narrowing to (u64, i64) at the entry point) — the only difference is that the no-prove path returns the pair directly while the prove path emits proof bytes the client verifies via GroveDb::verify_aggregate_count_and_sum_query.

Document Ranked Trees

The aggregate surfaces that landed in Platform 4.0 answer "how many" and "how much" — per document type, per indexed value, per range of indexed values. What none of them answer is "which groups score highest". Counting the reviews of every restaurant is O(log n) per restaurant; finding the five best-rated restaurants meant enumerating all of them and sorting client-side, with a proof that grew with the number of restaurants rather than with the number you asked for.

From protocol v14 an index can declare that its groups are rankable by an aggregate. The terminal tree upgrades to grovedb's indexed-tree family (grovedb PR #657), which carries an ordered secondary Merk per declared ranking axis, and "top 5 restaurants by average grade" becomes an O(log n + k) read with an O(log n + k) proof. This chapter explains the three indexed tree variants, how an index opts into one, how the secondaries are keyed and maintained, and how the feature composes with the other v14 change. The Ranked Index Examples chapter is the worked-example companion.

The chapter assumes you've read Document Count Trees, Document Sum Trees, and the Average Index Examples chapter. Ranked trees are built directly on top of the range-aggregate layouts those chapters describe: a ranking secondary is an ordering over aggregates the range layout already maintains, so the range axis is a hard prerequisite for the ranked one.

Status: implemented and gated at protocol version 14. The storage layout is pinned end-to-end against a real grovedb by ranked_index_e2e_tests, which runs against the restaurants fixture at packages/rs-drive/tests/supporting_files/contract/restaurants/restaurants-contract.json — the same fixture the examples chapter walks through. The grovedb-side design lives in that project's book chapter The CountIndexedTree (docs/book/src/count-indexed-tree.md), which is the authoritative reference for the element layout, the hash composition, and the secondary-Merk storage prefixes summarised here.

Why Ranking Needs a New Primitive

A ProvableCountTree at the property-name level stores, per group, the group's document count — and stores it in a Merk keyed by the group key. That is exactly what you want for "how many reviews does restaurant alpha have?" and for "how many reviews do restaurants between alpha and mike have?": both questions are answered by walking a key-ordered boundary.

"Which five restaurants have the most reviews?" is a question about the aggregate, not about the key. Nothing in the key-ordered Merk correlates position with count, so the only honest answer is to visit every group. That is O(n) work and — worse — an O(n) proof, even though the answer is five entries. Three options exist:

  1. Enumerate and sort client-side. Correct, and the only thing 4.0 could do. Proof size scales with the number of groups, not with k; a contract with 50 000 restaurants pays 50 000 committed entries to learn about five.
  2. Have the server sort and return the top five. O(1) bytes, zero cryptographic value — the client has no way to check that the sixth-best restaurant wasn't quietly omitted.
  3. Maintain a second, aggregate-ordered view of the same groups, committed to the same root hash. Top-k becomes a bounded range read at one end of that view, and its proof is a standard Merk range proof over k entries. This is what this chapter is about.

The grovedb primitive is the indexed tree: a primary Merk that is a byte-compatible mirror of the tree it replaces, plus one ordered secondary Merk per declared axis, keyed so that the aggregate sorts first and the group key only breaks ties. The primary is byte-compatible, but grovedb's range aggregates (AggregateCountOnRange, AggregateSumOnRange, AggregateCountAndSumOnRange) take provable trees only and do not prove through an indexed tree, so Drive refuses a range total through any index whose path passes through a ranked level, its own or another index's at a level the two share (refuse_a_range_total_through_a_ranked_index); per-value range reads keep working.

The dashed box is the wrapping Element. The primary Merk (blue) is keyed by group key and is byte-identical to the ProvableCountTree it replaces. The secondary (orange) holds one entry per group, keyed by count_be ‖ group_key, so a descending walk of its right edge yields the highest-count groups first.

flowchart LR
  subgraph PCIT ["ProvableCountIndexedTree"]
    direction TB
    subgraph ELEM ["Tree element c=9"]
      direction TB
      P["primary root<br/>(keyed by group key)"]:::primary
      S["secondary root<br/>(keyed by count ‖ group key)"]:::secondary
    end
  end

  P --> PA["alpha c=2"]:::leaf
  P --> PM["mike c=3"]:::leaf
  P --> PZ["zulu c=4"]:::leaf

  S --> SA["0x…02 ‖ alpha"]:::leaf
  S --> SM["0x…03 ‖ mike"]:::leaf
  S --> SZ["0x…04 ‖ zulu"]:::leaf

  classDef primary fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef secondary fill:#fb8500,color:#0d1117,stroke:#fb8500,stroke-width:2px;
  classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;

  style ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb

ORDER BY … DESC LIMIT 2 reads the two right-most secondary entries (zulu, mike) and proves them with a standard Merk range proof — three levels of hashes, two committed entries. The same question against the primary alone would have to commit all nine groups.

Contract Grammar

Ranking is an index-level opt-in. Three independent keywords, one per axis:

KeywordRanks groups byRequires (in effect)Rust field
rankedCountableeach group's document countrangeCountable: trueIndex::ranked_countable / Index::ranked_countable_at
rankedSummableeach group's sum of the summable propertyrangeSummable: trueIndex::ranked_summable
rankedAverageableeach group's average of the averageable propertyrangeAverageable semantics — both rangeCountable and rangeSummableIndex::ranked_averageable

The three axes are independent opt-ins. rankedAverageable is not sugar for the other two, unlike averageable / rangeAverageable, which genuinely are sugar for their count+sum longhand. Each ranking axis costs its own ordered secondary Merk and its own maintenance on every write, so each is declared explicitly. Declaring rankedAverageable alone is a legal — and usually the right — choice.

rankedCountable alone has a second, level-addressed spelling. The boolean form ranks the terminal level's groups by direct member count; the object form names the level (or levels) the Count axis aggregates at:

"rankedCountable": true                                   // terminal groups by member count
"rankedCountable": { "at": "hashtag" }                    // hashtags by WHOLE-SUBTREE count
"rankedCountable": { "at": ["hashtag", "postId"] }        // both rankings on one index
"rankedCountable": { "at": ["tag", "region", "postId"] }  // every level of a 3-property index

at takes one property name or an array of them (each must be an index property; duplicates are rejected; the array is capped at 10 — the index property cap). Naming the last property is the boolean form spelled longhand and folds into it; every other name places a ranking at that prefix level, ordering its values by their whole-subtree document count — on [hashtag, postId], { "at": "hashtag" } ranks hashtags by total count across all their posts, where the boolean form ranks a pinned hashtag's posts. Any subset of levels may be named, and the parser stores the non-terminal ones in Index::ranked_countable_at in index-property order regardless of how the contract spelled them. The object form cannot be combined with rankedSummable / rankedAverageable: the chain of count-bearing levels a prefix ranking is built from (see Prefix-Level and Multi-Level Rankings) cannot carry a sum axis.

The range prerequisite is a hard requirement, not a convenience: a ranking secondary orders the per-group aggregates the range layout already maintains. Without the range axis the terminal property-name tree carries no per-group aggregate to sort by. The parser rejects the combination with a message naming the missing flag:

rankedCountable requires rangeCountable: true; ranking groups by count needs
the per-group counts the range-count layout maintains

Value-Sensitive Prerequisites in the Meta-Schema

The document meta-schema enforces the same prerequisites, in two forms. The range* rows are presence rules: a rangeSummable key, whatever its value, needs a summable (or an averageable) key beside it, and a rangeAverageable key needs an averageable. rangeCountable has no row: it implies countable, exactly as the doctype-level rangeCountable implies documentsCountable, and the parser promotes an omitted countable to "countable" (an explicit "countableAllowingOffset" is kept, an explicit "notCountable" is rejected as a contradiction). The ranked rules are value-sensitive if / then pairs, because a written-out opt-out, "rankedCountable": false, which the structural parser accepts as exactly that, must not be made to demand a rangeCountable the index does not need:

{
  "if": {
    "properties": {
      "rankedCountable": { "anyOf": [{ "const": true }, { "type": "object" }] }
    },
    "required": ["rankedCountable"],
    "not": { "required": ["rangeAverageable"] }
  },
  "then": { "required": ["rangeCountable"] }
}

(The rankedCountable conditional matches the object form as well as the literal true; both spellings need a range axis, and only a written-out false escapes the requirement.)

Every rule is sugar-aware, and the not clause above is what makes it so. JSON-schema validation runs over the index object exactly as authored, before averageable / rangeAverageable are expanded into their countable + summable longhand, so each rule spells the sugar out as an accepted alternative: rangeSummable accepts averageable in place of summable, rankedCountable and rankedSummable accept rangeAverageable in place of their own range axis, and rankedAverageable accepts the explicit rangeCountable + rangeSummable pair in place of rangeAverageable. The doctype-level rangeSummable row accepts documentsAverageable the same way. The parser then checks the same prerequisites on the resolved flags, so the two layers agree on every spelling, and the sugar form of a multi-axis index is the whole declaration:

{"name": "storeRating", "properties": [{"storeId": "asc"}],
 "averageable": "rating", "rangeAverageable": true,
 "rankedAverageable": true, "rankedCountable": true}

Up to 4.2.0-beta.1, meta-schema v3 demanded the literal key instead: this index failed registration with "rangeCountable" is a required property, and adding rangeCountable then pulled in a literal countable through a presence row, so the only accepted spellings carried keys the sugar already implied. The same release also made the index-level parser demand an explicit countable beside rangeCountable, while the doctype level had always treated rangeCountable as implying documentsCountable. v3 is editable until 4.2 is live on mainnet, so both rules were corrected in place rather than carried into a v4; below protocol version 14 nothing moves.

The two layers are not gated alike. The structural parser runs on every parse, full_validation or not, and is compiled into every build. The JSON-schema layer runs only under full_validation, and only in builds with rs-dpp's validation feature, which wasm-dpp2 (and so @dashevo/evo-sdk) does not enable. An SDK-side DataContract.fromJSON(json, true, pv) therefore sees the parser's verdict alone, which is why the two layers agreeing on every spelling matters: it is what keeps an offline acceptance from turning into a registration refusal.

Shape Restrictions

Two structural rules, both enforced at contract-parse time in rs-dpp:

  • No aggregating index on a compound ranked index's full prefix. Ranked flags are allowed on compound indexes, with per-prefix semantics: a ranked [identityId, class] puts the indexed tree at each prefix value's terminal class property-name level — one ordered secondary per identityId, each ranking only that identity's class groups. There is deliberately no global cross-prefix ordering; the query surfaces require every leading property to be pinned by a where clause — equalities, plus at most one IN that fans out across prefix branches and merges deterministically. The one shape that stays impossible — and is rejected per document type, where all indexes are visible (validate_no_ranked_prefix_overlap) — is a countable/summable index terminating at exactly the compound's leading prefix: its aggregating value trees would demand the NonCounted / NotSummed shell around the ranked terminal tree, and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralise the very aggregates the secondaries order by. (Drive's fail-closed guard behind the parse-time check is INDEXED_INNER_UNWRAPPABLE.) Only the exact n-1 prefix conflicts: an aggregating index at a shorter prefix wraps a plain intermediate tree, and one extending past the ranked terminal lives inside its value trees — both supported.

  • Non-unique indexes only. ranked aggregates are not supported on unique indexes: each group of a unique index contains at most one document, so there is nothing meaningful to rank. Contested indexes are covered transitively — a contested index is unique by construction, so it hits the same check rather than needing its own.

  • Prefix-level rankings add two rules of their own, both anchored on the index's shallowest ranked prefix level (every deeper ranked level sits inside its exclusive range). From that level down, the index's levels form one count-propagation chain, and the rule set over other indexes reaching the at level is a shape matrix:

    Other index S (vs. ranking at pk on [p1 … pn])Verdict
    Diverges before pk (level key differs at some position < k)✔ never conflicted
    Terminates above the at level, no countable/summable flags✔
    Aggregating, terminates at a prefix shorter than [p1 … p(k-1)]✔ wraps a plain intermediate tree
    Plain (no countable/summable/ranked flags, nothing contested), continues below the at level, and branches off the chain — by diverging below at or extending past the ranked terminal✔ laid out count-exempt — its branch trees are Element::NonCounted inside the chain's value trees
    Continues below the at level but is itself countable/summable/ranked✘ its aggregates would need the counts the wrapper suppresses
    Terminates exactly at the at level (any flags)✘ its member bucket and aggregates would sit on the grouping tree itself
    Plain, continues below at but never leaves the chain (a level-key prefix of [p1 … pn])✘ it has no branch to wrap — its member bucket would land on a grouping/propagating level, the terminal-plus-chain stamp the tree-type resolver fails closed on
    Aggregating, terminates at exactly [p1 … p(k-1)] (the prefix directly above at)✘ the grouping tree would need a NonCounted shell inside aggregating value trees — the wrapped-indexed impossibility, mirroring the first rule above

    The count-exempt row is the same demotion range-countable value trees already apply to their sibling continuations, one configuration over: the sibling's branch reads and proves normally, contributes zero to every subtree total the ranking keys on, and its writes skip the ranked re-key entirely. The derivation stamps the branch level (IndexLevel::count_exempt_branch) so every write path agrees on the wrapper. Level identity for all of these comparisons is Index::level_key, not the declared name: a time-range index's grid-qualified first key forks it into a sibling subtree, so a bucketed index whose declared properties match a ranked index's does not conflict.

Version Gate

The grammar activates with protocol version 14 through two independent gates that must agree:

  • CONTRACT_VERSIONS_V6 points document_type_schema at the v3 document meta-schema, which hosts the three keywords. v13 keeps validating against v2, where they fail an index entry's additionalProperties: false.
  • try_from_schema: 3 selects parser generation 3, which is the only generation that passes ranked_aggregates_allowed = true into Index::try_from_value_map. With that flag off, the three keywords are not part of the grammar at all: they fall through to the unknown-key arm and are rejected with exactly the error a pre-v14 node produced for them.

The doubled gate is load-bearing. The meta-schema only runs under full_validation, so a non-validating parse — check_tx, contract-cache warm-up, state restore — could otherwise smuggle a ranked index past a node whose protocol version has no idea how to lay one out on disk.

On the parser-generation pattern. Generation 3 is a full copy of generation 2 (and of the generation-1 core that wrapper delegates to), not a version gate threaded into those modules. That is the repository's standing rule for grammar introduced by a new protocol version: shipped generations stay byte-identical to the code consensus already ran, so replaying a historical block can never pick up grammar that did not exist when the block was produced. The copy is kept structurally line-for-line with its sources, so a diff against v1/mod.rs + v2/mod.rs shows only the ranked deltas.

How Drive Picks the Indexed Tree Variant

An index's terminal property-name tree is the level whose children are the last index property's value trees — one child per distinct value, i.e. one child per group. Without ranking flags its TreeType comes purely from the range flags:

range_countablerange_summableBase tree type
truetrueProvableCountProvableSumTree
truefalseProvableCountTree
falsetrueProvableSumTree
falsefalseNormalTree

A ranking flag upgrades that base to its indexed mirror:

BaseDeclared axesIndexed tree
any[]unchanged (no ranking declared)
ProvableCountTree[Count]ProvableCountIndexedTree
ProvableSumTree[Sum]ProvableSumIndexedTree
ProvableCountProvableSumTreeany non-emptyProvableCountProvableSumIndexedTree(axes)

The single-axis variants (PCIT / PSIT) carry no axis list on the element at all — their one secondary is implied by the variant — which is why they are only reachable when the base layout is already single-axis. An index that declares, say, rankedCountable alongside rangeSummable still lays out as ProvableCountProvableSumTree underneath, so it upgrades to the multi-axis PCPSIT carrying just the Count axis in its TLV.

The axes list is canonical: sorted by tag (0 = Count, 1 = Sum, 2 = Avg), deduped, one to three entries. Freshly created, the element's TLV is [(tag, None), …] — every secondary starts empty. grovedb validates the list on construction (Element::validate_pcpsit_axes), because an out-of-order, duplicated or empty TLV would still be hashed into the parent and produce a tree whose secondaries nothing can address.

Selection lives in packages/rs-drive/src/drive/document/ranked_index_tree_type.rs:

#![allow(unused)]
fn main() {
pub(crate) fn ranked_property_name_tree_type(
    base: TreeType,
    ranked_axes: &[IndexAxis],
) -> Result<TreeType, Error> {
    if ranked_axes.is_empty() {
        return Ok(base);
    }
    match (base, ranked_axes) {
        (TreeType::ProvableCountTree, [IndexAxis::Count]) => Ok(TreeType::ProvableCountIndexedTree),
        (TreeType::ProvableSumTree, [IndexAxis::Sum]) => Ok(TreeType::ProvableSumIndexedTree),
        (TreeType::ProvableCountProvableSumTree, _) => {
            Ok(TreeType::ProvableCountProvableSumIndexedTree)
        }
        _ => Err(Error::Drive(DriveError::CorruptedContractIndexes(/* … */))),
    }
}
}

The catch-all arm is unreachable given rs-dpp's parse-time invariants (ranked_countable ⇒ range_countable, ranked_summable ⇒ range_summable, ranked_averageable ⇒ both), which force base to be provable on every axis a ranking flag names. It is a typed error rather than a silent fallback so a future grammar change that breaks one of those implications surfaces here instead of quietly laying down a tree whose secondaries nothing maintains.

property_name_tree_type_and_ranked_axes() is the single source of truth — contract registration, contract update, the document insert / update / delete index walkers, and the cost-estimation layers all route through it, so the layouts they describe cannot drift apart. The contract insert/update paths reach grovedb through three helpers parallel to the count and sum families:

  • batch_insert_empty_provable_count_indexed_tree — PCIT.
  • batch_insert_empty_provable_sum_indexed_tree — PSIT.
  • batch_insert_empty_provable_count_provable_sum_indexed_tree — PCPSIT, taking the axes TLV.

The groups keep their ordinary value-tree types. An indexed primary is a mirror of the tree it replaces, so nothing changes one level down: with the restaurants fixture, review's groups are ProvableCountProvableSumTrees, visit's are CountTrees, and tip's are SumTrees — exactly the shapes the range flags alone would have produced.

The Secondary Merks

Each declared axis gets one ordered secondary Merk holding one entry per group, keyed by

axis_sort_key ‖ group_key

so the aggregate sorts first and the group key breaks ties. Ties therefore come back in group-key order in the direction of the walk — descending group-key order for TOP, ascending for BOTTOM.

AxisSort keyWidthEncoding
Countcount8 Bbig-endian u64
Sumsum8 Bbig-endian i64 with the sign bit flipped
Avgfloor(sum × SCALE / count)16 Bbig-endian i128 with the sign bit flipped

Sign-bit toggling is what makes the signed encoders order-preserving: flipping bit 63 of an i64 (or bit 127 of an i128) maps the two's-complement signed range onto an unsigned range with the same ordering, so plain byte comparison sorts negatives below positives.

The Avg Fixed-Point Sort Key

There is no fractional key type, so the Avg axis sorts by a fixed-point integer:

#![allow(unused)]
fn main() {
pub fn compute_avg_fixed_point(sum: i64, count: u64) -> i128 {
    if count == 0 {
        return 0;
    }
    (sum as i128)
        .saturating_mul(AVG_FIXED_POINT_SCALE)
        .div_euclid(count as i128)
}
}

Three properties to internalise:

  • The scale is grovedb's constant, not a wire constant. AVG_FIXED_POINT_SCALE is 10^19 today; it moved from 10^15 before release. Drive re-exports it (drive::query::RANKED_AVG_SCALE, itself re-exported by drive_proof_verifier::RANKED_AVG_SCALE) rather than re-declaring it, so the two can never drift — the encoded sort keys in storage are produced with grovedb's constant, and a platform-side copy that fell out of step would silently mis-scale every average a client renders. Never hardcode the literal.
  • Division is euclidean, i.e. floor toward −∞. Rust's / truncates signed integers toward zero, which would place negative averages one fixed-point bucket too high. The restaurants fixture carries a whole document type (adjustment, whose delta admits negative values) for the sole purpose of exercising signed sums and this rounding mode.
  • 0 / 0 is defined as 0. An empty group has no entry in the secondary at all, so this is a defensive definition rather than an observable one.

Maintenance

Secondaries are maintained through the normal batch write path — there is no separate reindex step and no background job. Every document insert, update and delete that changes a group's (count, sum) also rewrites that group's entry in each declared secondary: the old (sort_key ‖ group_key) entry is removed and the new one inserted, in the same grovedb batch, under the same block. Draining a group's last document removes the group from the primary and its entry from every secondary together.

grovedb's own consistency verifier enforces the invariant directly: every primary entry at key must have exactly one secondary entry at make_axis_secondary_key(axis, count, sum, key), and the secondary must contain nothing else.

Prefix-Level and Multi-Level Rankings

The boolean axes always place their secondaries on the terminal property-name tree. rankedCountable's at form places Count secondaries at any subset of the index's levels, and the storage answer to "what does a prefix level rank by?" is a count-propagation chain:

"hashtag"  property-name tree   ProvableCountIndexedTree   ← ranked level
  <value>  value tree           CountTree (count = subtree total)
    "postId" property-name tree ProvableCountTree — contributes its count
      <value> value tree        CountTree
        [0]  member bucket      counted references / Items

From the shallowest ranked level down to the terminal, every level is count-bearing, and — unlike the shared-prefix continuation case, where a sibling continuation is zero-wrapped — each level's chain continuation is inserted contributing, so its count IS the value tree's count. Every write's delta then propagates up the chain, and grovedb re-keys each ranked level's secondary entry on the way — one secondary rewrite per ranked level on the leaf-to-root path, nothing more.

A plain sibling index sharing the at level (see the shape matrix above) hangs its branch inside the same value trees, but zero-wrapped:

"postAuthor"  property-name tree  ProvableCountIndexedTree   ← ranked level
  <alice>     value tree          CountTree, count = 3       (alice's real likes, exact)
    "postId"      property-name tree  ProvableCountTree      ← chain: contributes, re-keys ranking
      <post7> …      (3 entries)
    "$createdAt"  property-name tree  NonCounted(Tree)       ← sibling: readable, count-invisible,
      <ts> …         (3 entries)                                writes skip the ranked re-key

The wrapped branch is a normal readable subtree — equality-prefix and range reads through it serve and prove exactly as under any merged index — while contributing zero to the value tree's count, so the ranking totals stay exact with the sibling fully populated. Sibling-only writes (an entry landing under the branch with the chain tuple unchanged) touch neither the counts nor the secondaries. A level named in at becomes the Count-axis indexed tree; an unnamed level between the shallowest ranked one and the terminal becomes a plain CountTree pair (count-propagating); the terminal keeps the layout its own flags give it, which is why the at form requires rangeCountable — that is what makes the terminal property-name tree count-bearing enough to feed the chain. When the terminal is also ranked (the boolean alongside at), the chain simply nests indexed trees: the terminal ProvableCountIndexedTree is itself the contributing child, and both secondaries maintain simultaneously — this composes at any depth, so a fully ranked index re-keys one secondary per level per write.

The per-level stamps live on IndexLevel (ranked_count_grouping / count_propagating), set by the index-level derivation and read by the same tree-type resolver every write path shares. An index ranked at its first property gets its indexed tree at contract registration (it is a top-level tree); deeper ranked levels materialize lazily per prefix, like every dynamic index tree, and drained chains prune out of their secondaries (or stay rankable at zero under preallocated).

On the query side each level serves its own ranking, addressed by the (group property, pin count) pair: GROUP BY names the ranked level's property, and every property before it must be pinned — properties after it never appear in the request at all, since they are interior to the subtrees being counted. { "at": ["tag", "region", "postId"] } therefore serves GROUP BY tag with nothing pinned (the global ranking), GROUP BY region with tag pinned, and GROUP BY postId with both pinned, each from its own secondary with its own proof.

Hash Composition

An indexed tree commits two Merks in one element. The composition grovedb uses — internally called H1-A — is a single three-input Blake3 call:

combined_value_hash = combine_hash_three(value_hash, primary_root_hash, axes_digest)

For the single-axis variants the third input is simply that axis's root hash. For PCPSIT it is axes_digest, a length-prefixed Blake3 over the canonical axes TLV — [n] followed by (tag, root_hash) per axis. The length prefix is what distinguishes a one-axis digest from a two-axis digest truncated to a single entry. An axis with no entries yet contributes NULL_HASH in its slot.

The three-input form is deliberate and regression-tested: it must not be equivalent to combine_hash(a, combine_hash(b, c)), the nested composition rejected during design review because it would have doubled the hash work per bubble-up.

What this buys the verifier: the parent's value_hash binds the primary root and every secondary root. A proof of a top-k walk over one secondary reconstructs that secondary's root, recombines it with the primary root the same way the writer did, and the result has to match what the parent committed. A server cannot serve a stale or forged secondary while presenting an honest primary.

The Grove Path

Every ranked read — and, on the prove path, every ranked proof — is issued against the path of the terminal property-name tree. For a single-property index that is:

[ RootTree::DataContractDocuments as u8 ]   // 0x01
  / <contract_id: 32 bytes>
  / [ 0x01 ]                                // "documents", not "contract"
  / <document_type_name: utf-8>             // e.g. b"review"
  / <last_index_property_name: utf-8>       // e.g. b"restaurantId"

The children of that tree are the groups: one value tree per distinct value of the last index property, keyed by the raw index-key bytes of that value (for a string property, its UTF-8 bytes — e.g. b"alpha"). The secondary entries a top-k read returns are keyed by those same group keys. A compound index [a, b] inserts <a> / <encoded pinned value of a> between the doctype and the terminal <b> level — the value segment comes from the request's where pin on a — an equality pin, or one element of the single permitted IN (one branch per element) — encoded with the same serialize_value_for_key the write path used to key that prefix's value tree, so the walk lands on that prefix's own indexed tree and secondary.

Prover and verifier build this path through the same function, DriveDocumentRankedQuery::indexed_property_name_tree_path (with the pinned prefix values encoded by the shared resolver, resolve_ranked_query_for_mode), which is why they agree on the root hash by construction.

Write-Path Cost: The Grove v4 Cleanup Gates

Indexed trees need two batch behaviours that earlier grove versions don't have, so protocol v14 also moves Drive from GROVE_V3 to GROVE_V4:

  • delete_tree_cleanup_type_source — a batch DeleteTree reads the stored element and uses its actual type to select cleanup namespaces, rejecting a declared/stored mismatch that involves an indexed tree. V1–V3 take the declared type at face value.
  • overwrite_indexed_cleanup_inspection — a batch overwrite of a non-reference element (with tree-override protection off) reads the stored element to detect an indexed tree being overwritten, scheduling its per-axis secondary storage for cleanup or refusing the ambiguous case.

Without them, a batch overwrite of a ranked index would orphan its per-axis secondary storage. Indexed trees only exist from protocol v14, so activating the stricter cleanup alongside them costs older versions nothing.

Both gates are cost-neutral: they derive the old element from data the merk apply already loads when it rewrites or deletes a key, so no extra stored-element read is charged and v14's fee constants match v13's. Three facts pin this:

  • Fees are identical across the boundary. The identity-balance, token-balance and state-transition processing-fee pins carry the same values at protocol v13 and v14, and run_chain_one_identity_in_solitude has a protocol-version-13 sibling asserting the identical end balance — the pair proves the v13 → v14 transition changes nothing about those runs' fees.
  • Storage fees are untouched everywhere — the gates only observe, never write.
  • Cleanup still happens. A batch overwrite of an indexed tree schedules its per-axis secondary storage for cleanup (or refuses the ambiguous case), and DeleteTree uses the actual stored type — covered by grovedb's own overwrite and delete-tree suites at the pinned revision.

Interaction With the Shared-Prefix Aggregate Fix

Protocol v14 hosts two consensus changes, and contracts exist where both apply to the same index. The shared-prefix aggregate fix makes a contract legal that previously registered but rejected every document insert: an aggregating index terminating at a property that is also the prefix of a compound index (e.g. summable [a] next to [a, b]).

The two changes are orthogonal by construction, and they live exactly one level apart:

  • The ranked upgrade decides the property-name tree type — plain → indexed mirror.
  • The continuation demotion decides the value tree type one level below it — a provable count-bearing value tree that has to host a compound continuation demotes to CountSumTree, since grovedb rejects count-suppressed children under provable count parents by design.

A demoted CountSumTree value tree contributes its (count, sum) to a ranked indexed parent exactly as the provable variant did, so a ranked index's secondaries keep ranking correctly over shared-prefix shapes. Concretely, for a dish doctype with a ranked [restaurantId] index alongside a plain compound [restaurantId, chefId]:

  • the property-name tree at restaurantId stays a ProvableCountProvableSumIndexedTree carrying the Avg axis;
  • each group's value tree demotes from ProvableCountProvableSumTree to CountSumTree;
  • the chefId continuation inside it goes in Element::NonCounted, contributing zero to the group's count and sum.

The one place the two changes genuinely collide is the case the prefix-overlap rule already forbids at contract-parse time: a ranked terminal level sitting inside an aggregating value tree would need a wrapper, and an indexed tree can never be wrapped. That is the INDEXED_INNER_UNWRAPPABLE guard, and it fails closed.

Storage-Layout Invariants

All three ranking flags are immutable across a contract update, for the same reason and with the same error as the count and sum flags. The set of declared axes picks the indexed tree variant and its ordered secondaries at contract creation; toggling any one of them would require rebuilding the secondaries for every existing group.

Rankings parse from protocol version 14, whose document type validate_update (v1) refuses any changed index by comparing whole Index definitions, so every ranking flag, at level and chain stamp (and a summableOffCountIndex index's source) is frozen with the rest of the index. IndexLevel::find_first_ranked_change, which walks the two index-level trees and returns the first path where ranked_countable, ranked_summable or ranked_averageable differs (e.g. restaurantId -> (ranked_averageable: false -> true)), serves the earlier validate_update (v0), which no contract with a ranking reaches. Either way the update fails with DataContractInvalidIndexDefinitionUpdateError:

Document with type {document_type} could not add or remove '{index_path}' during
data contract update as we do not allow modifications of data contract index paths

Adding a new ranked index on update is rejected by the same machinery. Don't relax these guards: an index whose declared axes disagree with the element on disk would have the write path maintaining secondaries the reader can't address, or the reader reading secondaries the write path never updates — consensus drift either way.

Authoring a Contract That Uses Ranked Trees

The restaurants fixture is the reference. Its review doctype ranks restaurants by average grade:

{
  "review": {
    "type": "object",
    "documentsMutable": true,
    "canBeDeleted": true,
    "indices": [
      {
        "name": "byRestaurant",
        "properties": [{ "restaurantId": "asc" }],
        "countable": "countable",
        "summable": "grade",
        "averageable": "grade",
        "rangeCountable": true,
        "rangeSummable": true,
        "rangeAverageable": true,
        "rankedAverageable": true
      }
    ],
    "properties": {
      "restaurantId": { "type": "string", "minLength": 1, "maxLength": 32, "position": 0 },
      "grade": { "type": "integer", "minimum": 0, "maximum": 100, "position": 1 }
    },
    "required": ["restaurantId", "grade"],
    "additionalProperties": false
  }
}

The countable / summable / averageable trio plus the three range* flags are the prerequisites; rankedAverageable: true is the one line that adds the ranking. That index's terminal property-name tree at restaurantId becomes a ProvableCountProvableSumIndexedTree carrying axes [Avg].

The count-only and sum-only shapes are correspondingly smaller:

{ "name": "byRestaurantVisits",
  "properties": [{ "restaurantId": "asc" }],
  "countable": "countable", "rangeCountable": true, "rankedCountable": true }
{ "name": "byRestaurantTips",
  "properties": [{ "restaurantId": "asc" }],
  "summable": "amount", "rangeSummable": true, "rankedSummable": true }

which lay down a ProvableCountIndexedTree and a ProvableSumIndexedTree respectively.

Note that the fixture puts each shape on its own document type. That's not an accident of style: two indexes over the same property set on one doctype is a DuplicateIndexError, so exercising all three variants needs three doctypes.

Choosing What to Set

You wantSet
Top / bottom K groups by document countrankedCountable: true on a non-unique index that already has countable + rangeCountable: true
Top / bottom K prefix values by whole-subtree count ("top hashtags by total likes" on [hashtag, postId])rankedCountable: { "at": "hashtag" } on the same prerequisites. The named level's values rank by their entire subtree's document count; the levels below become the count-propagation chain.
Both of the above on one index — and/or more levelsrankedCountable: { "at": [...] } naming every level you want rankable (the last property = the boolean form). Each named level costs one secondary rewrite per write beneath it.
Top / bottom K groups by sum of a propertyrankedSummable: true on an index with summable: "<prop>" + rangeSummable: true
Top / bottom K groups by average of a propertyrankedAverageable: true on an index with averageable: "<prop>" + rangeAverageable: true (or the count+sum longhand)
Two rankings on one index (e.g. by count and by average)Both keywords. The tree is a PCPSIT carrying both axes in its TLV; you pay one secondary Merk per axis on every write.
A ranking filtered by another property (top 5 restaurants in London)A compound ranked index with the filter property leading: [city, restaurantId] with the ranked flags. Each city gets its own secondary; the query pins the prefix with an equality where (WHERE city == "London" GROUP BY restaurantId ORDER BY <agg> DESC LIMIT 5). Equality pins select one prefix; at most one pin may be an IN (2..=10 distinct elements; a never-written element — or one whose deeper pinned path was never written — contributes an empty branch), which walks one secondary per element and merges by (aggregate, encoded prefix, group key), proved in one branched PathQuery envelope (shared ancestors proved once, per-element authenticated absence) — entries then carry in_key. A single == pin on a prefix no document has written yet (a timeRange window before its first document, a hashtag nobody has used) is likewise an authenticated empty page, at any OFFSET: GroveDB answers a single-path axis read over a path that does not exist with the traversal's empty result, the absence proved by the layers the walk emits rather than reported as an error. A single-element IN normalizes to the equality pin; a null pin stays legal on its own but cannot combine with an IN (null addresses its prefix through an empty path segment the branched proof cannot express); a non-zero OFFSET is rejected together with IN; and branched proofs are generated from committed state only. A range operator on the prefix stays rejected, and there is still no global cross-prefix ordering beyond that merge.
A ranking on a unique or contested indexNot available, and not meaningful: every group holds at most one document.
Range aggregates without ranking (the 4.0 surface)Just the range* flags. Ranking is strictly additive — adding it never changes what a range query returns.
Nothing ranking-aware (default)Don't set any ranked* flag. The terminal property-name tree keeps the type its range flags give it.

Every ranking axis is paid for on every write that touches a group, not just on the reads that use it. Two axes on one index means two secondary rewrites per document insert. Opt into the axes you will actually rank by.

What a Ranked Query Looks Like

The query surface is deliberately narrow — one aggregate select, one group_by, one ORDER BY naming that select's aggregate, and a LIMIT (plus an optional OFFSET). Nothing else:

SELECT avg(grade) FROM review
  GROUP BY restaurantId
  ORDER BY avg(grade) DESC
  LIMIT 3

That is ordinary SQL, deliberately. An earlier draft of this surface spelled the same question as HAVING avg(grade) IN TOP(3) — a TOP / BOTTOM primitive on the right of a HAVING clause. It was removed before release rather than deprecated: SQL already expresses "the n highest-scoring groups" with ORDER BY + LIMIT, and inventing a second, non-conformant spelling for it bought nothing but a grammar every client author would have to learn. HAVING is now purely a boolean per-group predicate, as in SQL.

The Ranked Index Examples chapter covers the wire shape, the SDK surface, the proof, the rank/offset semantics, and the full list of what is rejected and why.

Ranked Index Examples

This chapter walks through a representative contract and shows how ranked queries work on Drive. Every example uses the restaurants contract at packages/rs-drive/tests/supporting_files/contract/restaurants/restaurants-contract.json — the same fixture the write-path e2e suite and the query suite share.

The chapter assumes you've read Document Ranked Trees for the storage layout, and the Count / Sum / Average example chapters for the aggregate surfaces ranking builds on. Here we take the indexed-tree machinery as given and look at the query shape it enables: "which n groups score highest?"

Status: implemented and gated at protocol version 14. Unlike the count / sum / average chapters, this one is not backed by a worst-case bench — there is no ranked bench, so no measured proof sizes or timings appear below. Every value shown is instead taken from the end-to-end suite at packages/rs-drive/src/query/drive_document_ranked_query/tests.rs, which runs prover and verifier against a live Drive and asserts the reconstructed root hash equals the database's own. Where a size is stated it is an asymptotic, not a measurement.

Why Ranking Is a Different Query Shape

Every other aggregate query in Drive walks value trees under a property-name tree and aggregates what it finds: AggregateCountOnRange sums per-subtree counts along a boundary, the average carrier fans out over an In and aggregates each branch. A ranked query never touches the value trees at all. The answer already exists, pre-sorted, in the axis secondary described in the previous chapter — the query is a bounded scan of one end of it.

Three consequences shape the entire API, and they are the reason this chapter's "what is rejected" table is longer than its query list:

  1. No where clauses. Ranked indexes are single-property, so there is no equality prefix to narrow; and a where on the ranked property itself asks for a filtered ranking, which the secondary cannot express — it is sorted by aggregate, not by group key.
  2. No start_at cursor. A cursor names a document id, and document ids do not appear in a keyspace sorted by aggregate. LIMIT and OFFSET are honoured — they are how the ranking is sized and paged; see Ranks and Offsets.
  3. Entry order IS the ranking order. The executor returns entries in the order grovedb walked the secondary. Callers must not re-sort.

The rejections are rejections rather than silent ignores, on both the client and the server, because a ranked walk cannot honour them and silently answering a different question is worse than an error.

The grammar is plain SQL

A ranked query is spelled the way SQL already spells "the n highest-scoring groups":

SELECT avg(grade) FROM review
  GROUP BY restaurantId
  ORDER BY avg(grade) DESC
  LIMIT 3

DESC is the "top n" reading, ASC the "bottom n" reading. LIMIT is the ranking's size; OFFSET moves the window down the ranking.

This replaced a non-SQL spelling that never shipped. An earlier draft put the ranking on the right of a HAVING clause — HAVING avg(grade) IN TOP(3), with TOP / BOTTOM / MAX / MIN as cross-group primitives. It was removed before release rather than deprecated. The deliberate call: SQL conformance beats a bespoke primitive. Every client author already knows ORDER BY … LIMIT; nobody knows IN TOP(n), and the two express exactly the same thing. The retired spelling also had a rough edge the SQL one simply does not have — = MAX means every group tied at the extreme, which a bounded read cannot prove, so MAX / MIN had to be permanently refused. ORDER BY <agg> DESC LIMIT 1 is positional and has no such ambiguity.

HAVING survives as what it is in SQL: a boolean per-group predicate — and since protocol v14 it is evaluated. A grouped aggregate carrying exactly one having clause that bounds the selected aggregate (GROUP BY hashtag HAVING count(*) > 100 LIMIT 100) is served as a value-bounded range read of the same axis secondary the ranking walks, with the same completeness-proving envelope. An ORDER BY naming the selected aggregate may ride along to set the walk direction (HAVING avg(grade) > 80 ORDER BY avg(grade) DESC LIMIT 5 — the best matches first); what a having request cannot carry is rank-window pagination (OFFSET, starting_rank), because a value-bounded page has no rank base — its continuation is "tighten the bound past the last value seen". That continuation steps past distinct aggregate values only: if the LIMIT cuts inside a tie (several groups sharing the boundary aggregate), keeping the boundary value repeats the same page and moving past it permanently skips the remaining tied groups, so size the limit above the widest expected tie. The grammar's v1 boundaries: one clause only, on the aggregate the select projects, with a contiguous-range operator (=, >, >=, <, <=, BETWEEN variants; != and IN are non-contiguous and refused).

The Restaurants Contract

Four document types, one per ranking shape. They all group by the same property (restaurantId) and each carries exactly one single-property index — because two indexes over the same property set on one doctype is a DuplicateIndexError, so exercising all three axes needs one doctype apiece.

doctypeindexdeclaresterminal property-name tree
reviewbyRestaurantaverageable + rangeAverageable + rankedAverageableProvableCountProvableSumIndexedTree axes [Avg]
visitbyRestaurantVisitscountable + rangeCountable + rankedCountableProvableCountIndexedTree
tipbyRestaurantTipssummable + rangeSummable + rankedSummableProvableSumIndexedTree
adjustmentbyRestaurantAdjustmentssame as reviewProvableCountProvableSumIndexedTree axes [Avg]

adjustment duplicates review's shape for one reason: its aggregated property delta admits negative values, which grade (minimum: 0) does not, so it is the only doctype that can exercise signed sums and the floor-toward-negative-infinity rounding of the Avg sort key.

{
  "$formatVersion": "0",
  "id": "AY6xWncZUFv2GCrS5seqKthUfbW9yYyUXtF8diSuHQ3f",
  "ownerId": "AtirhSVpAWF7dEt6dLAmesC4Sr1MsJ9bFC1nLAoNnq2S",
  "version": 1,
  "documentSchemas": {
    "review": {
      "type": "object",
      "documentsMutable": true,
      "canBeDeleted": true,
      "indices": [
        {
          "name": "byRestaurant",
          "properties": [
            { "restaurantId": "asc" }
          ],
          "countable": "countable",
          "summable": "grade",
          "averageable": "grade",
          "rangeCountable": true,
          "rangeSummable": true,
          "rangeAverageable": true,
          "rankedAverageable": true
        }
      ],
      "properties": {
        "restaurantId": {
          "type": "string",
          "minLength": 1,
          "maxLength": 32,
          "position": 0
        },
        "grade": {
          "type": "integer",
          "minimum": 0,
          "maximum": 100,
          "position": 1
        }
      },
      "required": [
        "restaurantId",
        "grade"
      ],
      "additionalProperties": false
    },
    "visit": {
      "type": "object",
      "documentsMutable": true,
      "canBeDeleted": true,
      "indices": [
        {
          "name": "byRestaurantVisits",
          "properties": [
            { "restaurantId": "asc" }
          ],
          "countable": "countable",
          "rangeCountable": true,
          "rankedCountable": true
        }
      ],
      "properties": {
        "restaurantId": {
          "type": "string",
          "minLength": 1,
          "maxLength": 32,
          "position": 0
        },
        "guests": {
          "type": "integer",
          "minimum": 1,
          "maximum": 100,
          "position": 1
        }
      },
      "required": [
        "restaurantId",
        "guests"
      ],
      "additionalProperties": false
    },
    "tip": {
      "type": "object",
      "documentsMutable": true,
      "canBeDeleted": true,
      "indices": [
        {
          "name": "byRestaurantTips",
          "properties": [
            { "restaurantId": "asc" }
          ],
          "summable": "amount",
          "rangeSummable": true,
          "rankedSummable": true
        }
      ],
      "properties": {
        "restaurantId": {
          "type": "string",
          "minLength": 1,
          "maxLength": 32,
          "position": 0
        },
        "amount": {
          "type": "integer",
          "minimum": 0,
          "maximum": 1000000,
          "position": 1
        }
      },
      "required": [
        "restaurantId",
        "amount"
      ],
      "additionalProperties": false
    },
    "adjustment": {
      "type": "object",
      "documentsMutable": true,
      "canBeDeleted": true,
      "indices": [
        {
          "name": "byRestaurantAdjustments",
          "properties": [
            { "restaurantId": "asc" }
          ],
          "countable": "countable",
          "summable": "delta",
          "averageable": "delta",
          "rangeCountable": true,
          "rangeSummable": true,
          "rangeAverageable": true,
          "rankedAverageable": true
        }
      ],
      "properties": {
        "restaurantId": {
          "type": "string",
          "minLength": 1,
          "maxLength": 32,
          "position": 0
        },
        "delta": {
          "type": "integer",
          "minimum": -1000,
          "maximum": 1000,
          "position": 1
        }
      },
      "required": [
        "restaurantId",
        "delta"
      ],
      "additionalProperties": false
    }
  }
}

Three things to internalize before reading the queries:

  1. rankedAverageable: true is one line on top of six prerequisite flags. The countable / summable / averageable trio and the three range* flags are what give the terminal property-name tree its per-group (count, sum); the ranked flag adds the ordered secondary that sorts those groups. Drop any prerequisite and the contract is rejected at parse time.
  2. visit needs no aggregated property. The Count axis ranks by group cardinality, so COUNT(*) takes no field — and both the select and the having aggregate carry an empty field string on the wire.
  3. tip is sum-only. It declares no count flags, so its terminal tree is a ProvableSumIndexedTree — you can rank restaurants by total tips, but not by tip average, because there is no count axis to divide by. Averages need both.

GroveDB Layout

Each doctype's terminal property-name tree at restaurantId is an indexed tree: a primary Merk keyed by group key, plus one secondary per declared axis keyed by (sort_key ‖ group_key).

Diagram conventions: blue is the indexed element wrapper; the primary Merk's children are the ordinary group value trees (green); the orange secondary holds one entry per group, aggregate-ordered.

flowchart TB
  TD["@/contract_id/0x01/review"]:::tree
  TD --> RID["restaurantId:<br/>ProvableCountProvableSumIndexedTree axes [Avg]"]:::indexed

  RID --> PRIM["primary Merk<br/>(keyed by group key)"]:::primary
  RID --> SEC["Avg secondary Merk<br/>(keyed by avg_fp_be16 ‖ group key)"]:::secondary

  PRIM --> GA["alpha: PCPS count=2 sum=170"]:::group
  PRIM --> GB["beta: PCPS count=3 sum=180"]:::group
  PRIM --> GD["delta: PCPS count=2 sum=60"]:::group
  PRIM --> GG["gamma: PCPS count=1 sum=95"]:::group

  SEC --> SD["fp(30) ‖ delta"]:::leaf
  SEC --> SB["fp(60) ‖ beta"]:::leaf
  SEC --> SA["fp(85) ‖ alpha"]:::leaf
  SEC --> SG["fp(95) ‖ gamma"]:::leaf

  classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
  classDef indexed fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
  classDef primary fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
  classDef secondary fill:#fb8500,color:#0d1117,stroke:#fb8500,stroke-width:2px;
  classDef group fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
  classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;

The full grove path down to that indexed tree, for a single-property index:

[ RootTree::DataContractDocuments as u8 ]   // 0x01
  / <contract_id: 32 bytes>
  / [ 0x01 ]                                // "documents", not "contract"
  / <document_type_name: utf-8>             // e.g. b"review"
  / <last_index_property_name: utf-8>       // e.g. b"restaurantId"

Every ranked read and every ranked proof is issued against exactly that path, built by the same DriveDocumentRankedQuery::indexed_property_name_tree_path on both sides — which is why prover and verifier cannot drift on which ranking is being checked.

The Request on the Wire

Ranked queries ride the existing GetDocumentsRequestV1. No new request message and no new field were added: selects, group_by, order_by, limit and offset already existed, and the ranking is expressed by what goes in them.

message GetDocumentsRequestV1 {
  // …
  repeated OrderClause order_by = 4;
  optional uint32      limit    = 5;
  bool                 prove    = 8;
  repeated Select      selects  = 9;
  repeated string      group_by = 10;
  optional uint32      offset   = 12;
}

A well-formed ranked request carries exactly one select, group_by property and order_by clause, and everything else at its unset wire value:

FieldRanked value
selectsone Select { function: AVG, field: "grade" }
group_by["restaurantId"]
order_byone OrderClause { target: Field("grade"), ascending: false }
limitthe ranking's n, 1 ..= 100 — required
offsetranks to skip; unset = 0
where_clausesempty — rejected if not
havingempty — rejected if not
start_after / start_atunset — rejected if set

Naming the aggregate in ORDER BY

The order_by clause names the aggregate it orders by through the wire's field target, and the name has to be the one the select already fixed:

selectorder_by field
SUM(amount)"amount"
AVG(grade)"grade"
COUNT(*)"$count"

SUM(f) / AVG(f) are named by f — the same property the projection aggregates, which is how ORDER BY avg(grade) reads once SELECT has fixed the function. COUNT(*) aggregates no property, so it is named by the reserved sentinel $count (RANKED_COUNT_ORDER_KEY). The $ sigil is load-bearing: DPP reserves the $ prefix for system properties ($id, $ownerId, …) and a schema cannot declare a property starting with it, so $count can never collide with a real column. A bare count would silently hijack ordering for any schema that happens to have a count field.

An order_by naming anything else — a second clause, the GROUP BY property, an unrelated field — is rejected rather than normalized. The SDK's order_by_selected_aggregate() builder derives the name from the select through rs-drive's own mapping, so there is no way to get it wrong by hand.

The wire also carries an explicit aggregate target (OrderClause.target.aggregate, e.g. ORDER BY AVG(grade) spelled out). It is still Unsupported — the field-target spelling above is the one the ranked executor reads — and exists so the explicit form can start being evaluated without another version bump.

limit is bounded to 1 ..= 100 (MAX_RANKED_LIMIT). Out-of-range is rejected, not clamped — k is echoed inside the proof envelope and re-checked by the verifier, so a server-side clamp would produce a proof the client's own reconstruction rejects.

Ranks and Offsets

OFFSET is honoured on the ranked path — the one place in the v1 surface where it is — and it is how you ask for a rank rather than a prefix:

SELECT avg(grade) FROM review
  GROUP BY restaurantId
  ORDER BY avg(grade) DESC
  LIMIT 1 OFFSET 4        -- the 5th-best restaurant

The response carries the skip back in RankedEntries.skipped (see The Response), so the page is self-describing.

skipped is the page's starting rank base: entry i is the group at rank skipped + i (0-based). Without it, a caller who asked for LIMIT 1 OFFSET 4 receives one entry and has no way to tell that it really is the 5th-best group rather than the best. It is 0 for offset-less queries, which is what makes the field additive.

Three properties worth stating plainly:

  • The skip is counted, not walked. grovedb descends the secondary reading each subtree's aggregate count and collapses any subtree that fits entirely inside the remaining offset, instead of stepping through it. Both paths do this: the prover attests the skipped region from the counted subtree commitments (HashWithCount / HashWithCountAndSum), and the unproven read performs the same counted descent without building a proof. Work and proof size stay O(log n + k) at any offset.
  • There is therefore no offset ceiling. An offset of 4 and an offset of four billion cost the same order of work — on either path, the deeper one in fact cheaper, since a tree that fits entirely inside the offset collapses at the root. There is no denial-of-service lever a cap would close, and a cap would only stop honest deep pagination.
  • An offset past the end is a positive answer. entries comes back empty and skipped is the ranking's entire reported population. "There are only 12 groups" is more information than a bare empty list.

Both paths report the same skipped: the offset you asked for when the skip succeeded, and the ranking's total population when the walk ran out of groups first. What differs is the warrant, not the value. On the proved path it is cryptographically attested, re-derived by the verifier from the counted commitments. On the unproven path it is an unverified claim, exactly like the entries beside it — equal to the attested value on an honest node, with nothing forcing a node to be honest. Callers who need to trust the population, rather than merely receive it, must still prove.

The Response

The result is an additive ResultData.ranked variant:

message RankedEntry {
  bytes key = 1;
  oneof value {
    uint64 count = 2 [jstype = JS_STRING];
    sint64 sum   = 3 [jstype = JS_STRING];
    double avg   = 4;
  }
}

message RankedEntries {
  repeated RankedEntry entries = 1;
  optional uint64      skipped = 2 [jstype = JS_STRING];
}

Five properties of the payload, each of them load-bearing:

  • skipped is the page's starting rank — see Ranks and Offsets. 0 for an offset-less query.
  • key is raw index-key bytes, not a typed value — the same bytes that name the group's value tree under the index. For a string property that's its UTF-8 encoding (b"alpha"). Clients that want the typed value decode it with the document type's key deserialization; the wire carries bytes so prover and verifier agree without a schema round-trip.
  • Entry order IS the ranking order — best-first for DESC, worst-first for ASC. Clients must not re-sort. Ties come back in group-key order in the direction of the walk, which is descending group-key order for DESC.
  • Fewer than n entries is normal, not an error — the index simply has fewer groups than requested.
  • avg is a double approximation, and deliberately so. What grovedb sorts the Avg axis by is an exact i128 fixed point — floor(sum × SCALE / count) with euclidean (toward −∞) division — and this field is that integer divided by RANKED_AVG_SCALE in f64. The precision loss costs nothing because RankedEntry only exists on the no-proof path: a client that asked for a proof reconstructs each entry from the proof itself, where the exact fixed point lives, and never reads this field. So the wire says what it means — an approximation for the caller who already chose to trust the node — instead of dressing a trusted number up as an exact one. Two groups whose averages differ past f64's ~15–16 significant digits can compare equal here; anything needing the committed integer must request the proof. Entry order is exact regardless: the ranking happened over the i128 before the conversion. The decoder rejects a non-finite avg, or one that scales past i128, rather than casting it into a plausible-looking zero.

Queries in this Chapter

Every query below uses the same shape — one aggregate select, one group_by, one ORDER BY on that aggregate and a LIMIT — and differs only in the axis and the ranking. All seven come from the end-to-end suite; the "verified result" rows are what the unproven read returned and what the verifier recovered from the proof, both asserted equal against the live grovedb root hash.

#QueryDoctype / axisComplexityVerified result
1Top 3 by Average Gradereview / AvgO(log G + k)gamma(95), alpha(85), beta(60)
2The Worst Averagereview / AvgO(log G + 1)epsilon(10.5)
3Top 2 by Visit Countvisit / CountO(log G + k)delta(4), beta(3)
4The Quietest Restaurantvisit / CountO(log G + 1)alpha(1)
5Bottom 3 by Tip Totaltip / SumO(log G + k)delta(1), gamma(24), alpha(25)
6Four-Way Tietip / SumO(log G + k)gamma, delta, beta, alpha
7More Than Existtip / SumO(log G + G)beta, alpha (2 of a requested 100)

Complexity variable. G = the number of distinct groups (distinct values of the ranked property). Notably absent: the total document count N. A ranked walk reads pre-committed per-group aggregates out of the secondary and never enumerates documents, so both work and proof size are O(log G + k) — independent of how many documents sit inside the groups it returns. Proof bytes grow linearly in k (one committed secondary entry per returned group) plus the O(log G) ancestor path, which is exactly why k is capped at 100.

Query 1 — Top 3 Restaurants by Average Grade

Eight review documents across four restaurants:

restaurantgradescountsumaverage
alpha90, 80217085
beta60, 70, 50318060
gamma9519595
delta40, 2026030
select   = AVG(grade)
group_by = [restaurantId]
order_by = grade DESC
limit    = 3
prove    = true

Path query:

path:  ["@", contract_id, 0x01, "review", "restaurantId"]
read:  indexed_avg_top_k(k = 3, descending = true)

Verified result (returned by GroveDb::verify_indexed_axis_top_k, wrapped by DriveDocumentRankedQuery::verify_ranked_top_k_proof):

[ ("gamma", AvgFixedPoint(95 × SCALE)),
  ("alpha", AvgFixedPoint(85 × SCALE)),
  ("beta",  AvgFixedPoint(60 × SCALE)) ]

descending by average: gamma(95) > alpha(85) > beta(60) > delta(30)

delta is below the cut and never appears in the proof — that is the whole point. A count-tree walk would have had to commit all four groups to convince the client which three are highest; the secondary's ordering means committing three entries plus the boundary is sufficient.

The client divides: 95 × SCALE / SCALE = 95. It never sees a double off the wire.

Query 2 — The Worst Average (and the Fixed-Point Floor)

Add a fifth restaurant whose sum doesn't divide evenly: epsilon with grades 10 and 11.

select   = AVG(grade)
group_by = [restaurantId]
order_by = grade ASC
limit    = 1
prove    = true

Verified result:

[ ("epsilon", AvgFixedPoint(21 × SCALE / 2)) ]

21 / 2 = 10.5 is the lowest average

Three things this query pins:

  • ORDER BY … ASC LIMIT 1 is the positional single worst-ranked group. It walks the secondary from the smallest sort key up and stops after one entry.
  • The sort key is floor(sum × SCALE / count), computed with grovedb's own compute_avg_fixed_point — the test asserts the returned value against both the hand-written 21 × SCALE / 2 and grovedb's function, so a change to either the scale or the rounding shows up immediately.
  • as_f64() divides back down, returning exactly 10.5. It is a display helper: lossy for large counts and sums, and never to be used for consensus-relevant comparisons, since two groups whose fixed-point averages differ can round to the same f64.

For the negative half of the rounding story, the adjustment doctype exists: with delta admitting negatives, expected_avg_fixed_point(-11, 3) must come out exactly one fixed-point bucket below what truncating division would give, because euclidean division floors toward −∞ while Rust's / truncates toward zero.

Query 3 — Top 2 Restaurants by Visit Count

Ten visit documents. The guests values are stored but irrelevant here — the Count axis ranks by how many documents are in each group, not by anything inside them:

restaurantvisitscount
delta4 documents4
beta3 documents3
gamma2 documents2
alpha1 document1
select   = COUNT(*)
group_by = [restaurantId]
order_by = $count DESC
limit    = 2
prove    = true

Verified result:

[ ("delta", Count(4)),
  ("beta",  Count(3)) ]

descending by document count: delta(4) > beta(3) > gamma(2) > alpha(1)

On the wire, both the select and the having aggregate carry field: "". A non-empty field on a COUNT ranking is rejected — COUNT(field) (counting non-null values of a property) is a different aggregate and the ranked surface doesn't serve it.

Note that visit's index declares no sum flags at all, so its terminal tree is a plain ProvableCountIndexedTree — a single secondary, no axes TLV. Asking for SUM(guests) or AVG(guests) against it resolves no index and is rejected: the stored element could host that secondary, but the contract never declared it, and the write path therefore never maintained it.

Query 4 — The Quietest Restaurant

select   = COUNT(*)
group_by = [restaurantId]
order_by = $count ASC
limit    = 1
prove    = true

Verified result:

[ ("alpha", Count(1)) ]

Note what this query does not claim. ORDER BY $count ASC LIMIT 1 is the positional single worst-ranked group: if two restaurants tied at one visit each, one of them comes back and the other does not, deterministically (ties break by group key in the walk's direction). It is not "every group at the minimum" — that is a set a bounded read cannot attest, and it is why the retired grammar's = MIN spelling could never have been served. The positional reading documents dropping ties as its meaning; the value-based one would have had to lie about it.

See What Is Rejected and Why for the full reasoning.

Query 5 — Bottom 3 Restaurants by Tip Total

Seven tip documents:

restaurantamountssum
beta100100
alpha10, 1525
gamma7, 8, 924
delta11
select   = SUM(amount)
group_by = [restaurantId]
order_by = amount ASC
limit    = 3
prove    = true

Verified result:

[ ("delta", Sum(1)),
  ("gamma", Sum(24)),
  ("alpha", Sum(25)) ]

ascending by sum: delta(1) < gamma(24) < alpha(25) < beta(100)

ORDER BY amount DESC LIMIT 1 on the same data returns [("beta", Sum(100))].

Sums are signed (sint64 on the wire, i64 in the SDK) for the same reason the sum surface's are: grovedb's sum trees model overflow into negative space rather than saturating. The Sum axis's sort key is the i64 with its sign bit flipped, which is what makes plain byte comparison order negatives below positives.

Query 6 — A Four-Way Tie

Four groups, all summing to 50. Only the group key distinguishes them.

select   = SUM(amount)
group_by = [restaurantId]
order_by = amount DESC              // and the ASC mirror
limit    = 4

Verified results:

DESC LIMIT 4:  ["gamma", "delta", "beta", "alpha"]  // descending group key
ASC  LIMIT 4:  ["alpha", "beta", "delta", "gamma"]  // ascending group key

The two directions are exact reverses of each other under a full-width k. That falls out of the secondary's key layout rather than from a separate tie-break rule: keys are (sort_key ‖ group_key), and the walk is a plain directional scan of that keyspace, so equal sort keys come back in group-key order in the direction of the walk.

The consequence that matters is determinism under truncation. DESC LIMIT 2 on the same data returns ["gamma", "delta"] — a specific subset, not an arbitrary one, and the same subset on every node. That reproducibility is what makes a tie-truncating LIMIT k provable at all, and it is the reason the retired = MAX spelling could never have been served: = MAX means every group at the extreme, and a bounded read cannot attest that nothing else ties.

Query 7 — Asking For More Groups Than Exist

Two tip groups, LIMIT 100:

select   = SUM(amount)
group_by = [restaurantId]
order_by = amount DESC
limit    = 100
prove    = true

Verified result:

[ ("beta", Sum(20)), ("alpha", Sum(10)) ]

A short result is the index having fewer groups, not an error, and the proof round-trips just the same. The verifier enforces the bound in the other direction only: at most k entries, because more would mean the proof committed a longer walk than the request authorized.

Fetching From the Rust SDK

DocumentRankedEntries lands on the standard Fetch trait against a DocumentQuery, with order_by_selected_aggregate() building the one ordering clause the surface needs:

#![allow(unused)]
fn main() {
use dash_sdk::{Sdk, platform::{DataContract, DocumentQuery, Fetch, Identifier}};
use dash_sdk::drive::query::SelectProjection;
use dash_sdk::platform::documents::document_query::RankingDirection;
use drive_proof_verifier::{DocumentRankedEntries, RankedEntryValue, RANKED_AVG_SCALE};
use futures::executor::block_on;

const RESTAURANTS_CONTRACT_ID: [u8; 32] = [0; 32];
let sdk = Sdk::new_mock();
let contract = block_on(DataContract::fetch(&sdk, Identifier::new(RESTAURANTS_CONTRACT_ID)))
    .expect("fetch contract")
    .expect("contract exists");

let query = DocumentQuery::new(contract, "review")
    .expect("document type exists")
    .with_select(SelectProjection::avg("grade"))
    .with_group_by("restaurantId")
    .order_by_selected_aggregate(RankingDirection::Descending)
    .with_limit(5);

let ranked = block_on(DocumentRankedEntries::fetch(&sdk, query))
    .expect("fetch succeeds")
    .expect("a well-formed ranked query always answers");

// Entry order IS the ranking order — best first.
for (offset, entry) in ranked.entries.iter().enumerate() {
    let rank = ranked.starting_rank + offset as u64;
    let restaurant = String::from_utf8_lossy(&entry.key);
    if let RankedEntryValue::AvgFixedPoint(fixed_point) = entry.value {
        let average = (fixed_point as f64) / (RANKED_AVG_SCALE as f64);
        println!("#{}: {restaurant}: {average}", rank + 1);
    }
}
}

fixed_point is the exact integer the proof commits to on this (proved) path. On a prove = false fetch the wire carries only the double — the SDK re-scales it back into the same variant, so the low digits are reconstruction noise. See the response notes.

The 5th-best restaurant is the same query with the window moved down one rank at a time:

#![allow(unused)]
fn main() {
use dash_sdk::platform::{DataContract, DocumentQuery};
use dash_sdk::platform::documents::document_query::RankingDirection;
use dash_sdk::drive::query::SelectProjection;
fn example(contract: DataContract) -> Result<(), dash_sdk::Error> {
// SELECT avg(grade) GROUP BY restaurantId
//   ORDER BY avg(grade) DESC LIMIT 1 OFFSET 4
let query = DocumentQuery::new(contract, "review")?
    .with_select(SelectProjection::avg("grade"))
    .with_group_by("restaurantId")
    .order_by_selected_aggregate(RankingDirection::Descending)
    .with_limit(1)
    .with_offset(4);
Ok(())
}
}

Notes on the surface:

  • order_by_selected_aggregate() derives the ordered field from the select through rs-drive's own key mapping — "grade" for AVG(grade), the $count sentinel for COUNT(*). You never name it by hand, so client and server cannot disagree about what is being ordered. Set the select first; the builder reads it.
  • It replaces rather than appends. A ranked query takes exactly one ordering clause.
  • RANKED_AVG_SCALE is a re-export of grovedb's constant, which moved from 10^15 to 10^19 before release. Never hardcode the literal. RankedEntryValue::as_f64() does the same division for display purposes.
  • ranked.entries is Vec<RankedEntry> in ranking order. Do not re-sort it. ranked.starting_rank is the rank of entries[0], re-derived from the proof rather than taken from the node.
  • The Fetch path always requests a proof. There is no prove knob on it; if you need the unproven read, that's the DocumentRankedEntries::from_unproved_response path.
  • No JS/WASM or FFI binding exists yet. The ranked surface is Rust-SDK-only today; the generated gRPC types are present in the web client but nothing is hand-written on top of them.

Proof Notes

The root hash is the whole point. The merk-level verifier returning Ok is not by itself evidence of anything. A bit-flip sweep over a real ranked envelope shows that most mutations do error out — but roughly 9% of them (bytes of sibling-subtree hashes inside the ancestor layer proofs) verify cleanly and return the correct entries, under a different reconstructed root hash. What rejects those is the tenderdash composition: drive_proof_verifier::verify_ranked_top_k_proof checks the reconstructed root against the quorum-signed app hash for the response's block, and there is no path through it that yields entries without that check having run.

Three things grovedb checks before the entries come back:

  1. The envelope's (axis, k, descending, offset) match the query. They are echoed in the proof and compared against the arguments, so a proof generated for a different ranking — or for a different page of the same one — is rejected rather than silently reinterpreted.
  2. The result's axis shape matches the requested axis — a Count request must not come back holding Sum entries. Belt-and-braces on top of (1).
  3. At most k entries. Fewer is normal; more would mean the proof committed a longer walk than the request authorized.

Empty rankings prove. An earlier iteration of this surface could not prove one: grovedb's non-paginated prover had no absence-proof shape for "this axis secondary has no entries", so the merk layer failed with Cannot create proof for empty tree and the node had to map that to an invalid_argument telling the caller to retry unproved. It was reachable by anyone — querying a freshly registered contract with prove = true did it.

The paginated prover (prove_indexed_axis_top_k_paginated) closed that gap: against an empty axis secondary it emits a guaranteed-empty range rather than refusing. Proving a ranking over an index with no documents now succeeds and returns an empty page, so the proved and unproven paths agree on the one case where they used to diverge. There is no fallback to implement and no rejection to recognise; prove = true is always answerable.

The same mechanism is what makes an offset past the end provable — see Ranks and Offsets. The node keeps a narrow backstop mapping for the old merk-level error because it names a class of failure rather than a single call site, but it is no longer a live path.

Against a protocol-version-13 node, the whole request is rejected as Unsupported — v13's query table has no ranked path and refuses the aggregate ordering. That is the intended activation gate, not a bug: a v13 node and a v14 node must disagree here and nowhere else, which is what lets a mixed-version network run through the upgrade.

What Is Rejected and Why

Everything below is rejected before any grovedb work, and most of it is mirrored client-side so the caller learns without a round trip.

RejectedWhy
Compound (multi-property) ranked index — at contract-parse time, ranked aggregates are only supported on single-property indexes in this protocol versionTwo reasons. A compound index whose prefix level also terminates an aggregating index would need its ranked terminal tree wrapped in a NonCounted / NotSummed shell — and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralize the very aggregates the ranking indexes. Separately, the ranked query surface has no equality-prefix routing yet. Both are relaxable at a future protocol version. The query-side index picker is the backstop: it refuses compound indexes even if the flags are somehow present.
unique ranked index — ranked aggregates are not supported on unique indexes: each group of a unique index contains at most one document, so there is nothing meaningful to rankEvery ranking over a unique index degenerates to a constant-per-group ordering a plain range query already serves, while still paying for an indexed tree and its secondary maintenance on every write.
Contested ranked indexCovered transitively — a contested index is unique by construction, so it hits the check above.
where clauses — InvalidWhereClauseComponentsRanked indexes are single-property, so there is no equality prefix to narrow; and a clause on the ranked property itself asks for a ranking over a filtered subset, which the secondary cannot answer because it is ordered by aggregate rather than by group key. Silently dropping the filter would return the global ranking under the guise of a filtered one.
start_at / start_after — InvalidLimitThe cursor names a document id, but a ranked walk iterates an aggregate-ordered keyspace in which document ids do not appear.
order_by naming anything but the selected aggregate, or more than one clause — InvalidParameterThe single ordering clause is the ranking, and the secondary is sorted by one aggregate only. An ordering on the GROUP BY property, on an unrelated field, or a second tie-break clause names an order the secondary cannot produce. Accepting and silently ignoring it is the one genuinely dangerous option. Use the aggregate's own name ($count for COUNT(*)), or flip ASC ↔ DESC to reverse the ranking.
group_by with ≠ 1 property — InvalidParameterRanked indexes are single-property, so there is no compound grouping to rank over.
having that isn't one contiguous bound on the selected aggregateA grouped single-clause having bounding the selected aggregate is served since protocol v14 — it routes to the having-range executor, a value-bounded range read of the same axis secondary (see the HAVING paragraph above). What stays rejected: multiple clauses (a second predicate needs a per-candidate post-check no executor performs), a clause on a different aggregate than the select projects (same reason), non-contiguous operators (!=, IN), having without group_by (a single implicit group is a plain aggregate the client can bound itself), and OFFSET / start_at alongside having (a value-bounded page has no rank base; continuation is by tightening the bound). Protocol v13 and earlier reject every non-empty having unchanged.
no order_by at all, on a grouped aggregate — routed elsewhereWithout an ordering this is a plain grouped aggregate, not a ranking; the caller wanted the DocumentSplitCounts / DocumentSplitSums / DocumentSplitAverages surface.
COUNT(field) (non-*) — Unsupported; SUM / AVG with an empty field — InvalidParameterThe Count axis ranks group cardinality and takes no field; the Sum and Avg axes rank the property the index accumulates and require it.
limit unset, 0, or > 100 — InvalidLimitA ranking with no n has no size, and LIMIT 0 selects nothing. The ceiling is a hard limit, not a clamp, because k is echoed in the proof envelope and re-checked by the verifier — a silent clamp would produce a proof the client's own reconstruction rejects.
A SUM / AVG ranking on a field the index doesn't accumulate — no covering indexThe picker requires the select's field to be the index's summable property. Resolving anything else would answer about the wrong property with no indication that a substitution happened.
Proving a ranking over an empty indexNo longer rejected. The paginated prover emits a guaranteed-empty range against an empty axis secondary, so prove = true over an index with no documents returns an empty page. Listed here because it used to be a rejection and the old advice ("retry with prove = false") is now wrong.

At-a-Glance Comparison

QueryDoctypeTerminal treeAxisRankingReturned variant
1 — Top 3 by averagereviewProvableCountProvableSumIndexedTree [Avg]AvgORDER BY grade DESC LIMIT 3AvgFixedPoint(i128)
2 — Worst averagereviewsameAvgORDER BY grade ASC LIMIT 1AvgFixedPoint(i128)
3 — Top 2 by visitsvisitProvableCountIndexedTreeCountORDER BY $count DESC LIMIT 2Count(u64)
4 — QuietestvisitsameCountORDER BY $count ASC LIMIT 1Count(u64)
5 — Bottom 3 by tipstipProvableSumIndexedTreeSumORDER BY amount ASC LIMIT 3Sum(i64)
6 — Four-way tietipsameSumORDER BY amount DESC/ASC LIMIT 4Sum(i64)
7 — More than existtipsameSumORDER BY amount DESC LIMIT 100Sum(i64) (2 entries)

Every row is one bounded scan of one secondary Merk, one proof, one root-hash commit. The shape never varies with the axis — only the sort-key width (8 / 8 / 16 bytes) and the returned scalar type do.

What's Next

Two capabilities are deliberately deferred and would each land as a separate protocol-version change:

  • Compound ranked indexes. Both blockers are named above — the indexed-tree wrapper conflict and the missing equality-prefix routing. Lifting them would enable "top 5 chefs at restaurant alpha by average grade" without a client-side sort.
  • Offset-paginated rankings. grovedb already has the primitive (prove_indexed_axis_top_k_paginated); it is not exposed because a ranked query currently rejects every pagination knob, and wiring one in would need the result-size contract (result size == n) to be rewritten rather than extended.

For the shape of the tree these queries read, and the write-path cost of maintaining it, see Document Ranked Trees.

Time-Range Index TTL

Architecture reference for the ttl key of timeRange indexes: what it means, how expired windows are drained, and the invariants every walker shares. The storage primitive underneath is grovedb's flat-subtree drop (dashpay/grovedb#848, landed in grovedb PR #849); see the storage section.

A document type can also expire whole documents with its own ttl keyword; that is a different mechanism, described in Document Time To Live.

Motivation

A timeRange index stores every document once per containing window, and a ranked one additionally rewrites a per-window secondary on every write. All of those bytes are billed as storage — a price that prepays ~perpetual retention through the epoch-distribution model — even though windowed data is intrinsically ephemeral: a "posts liked this hour" bucket is worthless once the trending surface has moved past it. The result is that the flagship use case (likes feeding a trending index) pays perpetuity prices for state with a useful life measured in days, multiplied by the grid's overlap factor.

Nobody cleans this up, either. Deletion costs the deleter processing, refunds accrue to owners who have no reason to come back for entries this small, and the state lingers forever.

Semantics

A timeRange index may declare a time to live:

"timeRange": { "on": "$createdAt", "range": 3600, "step": 3600, "ttl": 604800 }

Entries under this index are queryable until ttl seconds past their bucket's start (the exact expiry boundary remains inclusive). Physical removal depends on subsequent writes and has no wall-clock deadline. Expired buckets are drained lazily, on write: every state transition that writes into the index continues draining the oldest expired bucket, deepest-first, under a per-write operation budget. A fully drained window is provably absent, exactly like a window that never held documents: a count over it proves zero, and a ranked or having-range query pinned to it proves an empty page (see the ranked chapter's pinned-prefix rules). An expired window is not queryable at all — byStart rejects starts past the horizon, so the drainage lag is purely internal: drainage only ever touches expired buckets, which makes every window a query can address complete. Everything written under the index's grid-qualified level bills as processing, not storage — including the transitional bytes — at an ephemeral-bytes rate.

Why the fee reclassification is honest, not a subsidy

Storage fees prepay retention distributed across future epochs. TTL indexes instead charge a flat per-byte processing surcharge for their transitional storage and write amplification. Version 1 caps the queryable lifetime at one week (SystemLimits::max_time_range_ttl_seconds = 604 800). Cleanup capacity exceeds the maximum rate at which continued writes can create trees. This is an amortized retention model, not a guarantee that physical bytes disappear within a week: bursts need subsequent writes to drain, and inactive indexes retain residue as described below.

The load-bearing simplification: TTL'd subtrees never create refundable storage. No StorageFlags, no owner/epoch refund entries. That single property pays off three times:

  1. the fee reroute needs no refund-ledger reconciliation;
  2. cleanup owes nobody anything;
  3. deletion needs no byte metering for consensus — which is what makes O(1) bucket drops possible at all (see the grovedb dependency).

Grammar and validation

  • ttl is an optional key of the timeRange map, in seconds, parsed into the transform. It is not part of the grid identity: [TimeRangeTransform::storage_key] excludes it, so declaring or changing a TTL never forks the storage level, and query-side grid matching ([TimeRangeGridSpec]) continues to compare (range, step, phase) only.
  • ttl ≥ range. $createdAt is consensus-assigned from block time, so writes only ever target windows containing now; this invariant guarantees no bucket that can still receive entries (or serve as the oldest selector's window) is ever dropped.
  • ttl ≤ SystemLimits::max_time_range_ttl_seconds (one week in v1). The cap is what makes the flat ephemeral-byte rate safe.
  • One TTL per grid per field. Two indexes bucketing the same field with the same grid share one storage level; a differing ttl would give the shared subtree two conflicting lifecycles. Rejected at contract validation.
  • Composes with everything the grid already composes with: countable, the range axes, ranked levels below the bucket, unique (range == step, $createdAt), indexOnly document types, and skipIfAbsent on a property below the bucket (a skipped document builds no window under a grid only skip indexes use, and no branch of its own in a window it shares). preallocated stays banned with timeRange for the pre-existing structural reason.

Cleanup

Trigger — deterministic and write-amortized: every write into a TTL'd index continues drainage of the oldest expired bucket (start < block_time − ttl), deepest-first. Each grid's per-write drop budget is max(SystemLimits::min_time_range_ttl_drop_operations_per_write, 2 × overlap × trees). The versioned floor is 32; trees bounds everything one document can create under one bucket: value trees, terminal [0] trees, and all property-name branches in the grid's merged index structure. Shared grids and deep suffixes are counted. Thus cleanup has capacity above the maximum tree creation rate, including at the supported overlap of 24. A fixed 32-drop cap cannot keep up with that overlap.

When nothing is expired, the check is a single bounded range read. Large expired buckets drain across writes. In a document batch, all cleanup runs before any document mutations are generated, including nested same-type document groups. Each document earns a budget; conversion then uses the prepared state without further direct drops. Estimation performs neither cleanup nor its bookkeeping reads. Drops share the caller's transaction, so rollback restores both the removed paths and their redo records.

Residue — an index that stops receiving writes retains its remaining buckets, including any expired backlog, indefinitely. That state owes no refund, but its size depends on past write volume; the TTL cap alone does not bound it. An epoch-transition sweep could provide a backstop, following check_for_ended_vote_polls / clean_up_after_vote_polls_end; it remains out of scope for v1.

User deletes and updates of expired documents — handled at full-path granularity, because a bucket drains piecewise: an entry whose bucket (or whose group's trees inside a standing bucket) the drain already took is skipped as cleanly removed; one whose trees still stand is removed normally, so a not-yet-drained expired bucket never carries dangling references. Every check is deterministic — it reads consensus state plus the carried $createdAt and block time. Writes never target expired windows, so an update of a fully expired document simply leaves it without entries under the TTL'd index. An index-only window declaring outlivesDelete is not touched by a user delete at all: its entries stay until this cleanup drops their bucket, and the delete carries no $createdAt when only such windows involve it (see Index-Only Document Types).

Per-index semantics — TTL removes entries from this index only. An indexOnly like whose windowed entries expire keeps counting in the all-time ranked byPost and in byLiker; permanence lives where the contract declares it. Ranked per-window secondaries die with their bucket — which also caps live leaderboard state at ~ttl / step windows per index.

grovedb dependency: flat-subtree drop

Dropping a bucket must never put user-scaled work on the consensus path. The primitive that landed (grovedb PR #849) is the flat-subtree drop: O(1) consensus removal of a subtree declared to contain no child subtrees — an ordinary parent-Merk element delete whose cost is independent of the subtree's contents — staging a durable redo record (atomically, outside the root hash) that names every storage prefix the drop orphaned: the subtree's own and, for indexed primaries, its three per-axis secondary prefixes. Reclamation is DB-level range tombstones, drained by GroveDb::flush_pending_prefix_drops — idempotent, crash-safe, snapshot-correct, and never part of consensus cost.

A time-range bucket is not flat, so the platform drains it deepest-first, one flat unit at a time (drain_expired_time_range_buckets):

  1. each group's [0] reference tree — flat by construction, and where the mass lives — is flat-dropped;
  2. the emptied group value tree leaves through the flat drop — or, under a ranked (indexed-primary) property-name tree, through grovedb's dedicated indexed-tree delete, which mirrors the group out of the ranking secondary;
  3. the drained property-name tree is flat-dropped (dooming its secondary prefixes when ranked);
  4. the emptied bucket is flat-dropped.

Every step is O(1); the number of steps scales with the window's distinct groups and is capped by the structure-derived per-write budget above. Between writes a bucket may stand partially drained. Removal walkers skip only paths already removed from expired buckets and delete standing entries normally. The indexOnly delete validation uses the same rule: every surviving entry must match the full row commitment, including entries in expired but standing trees. Missing live entries, missing terminal members in standing trees, and mismatched commitments fail. An indexOnly contract must retain a timestamp-independent proof index, which still has to prove the row's membership after all its TTL entries drain.

The flat-drop path-reuse contract (never re-create a dropped path before its record drains) holds by construction: bucket paths embed their window start, and writes never target expired windows. The host side: drive-abci calls flush_pending_prefix_drops after committing each block's transaction and once at startup, completing reclamation a crash may have interrupted.

Fee mechanics

Write operations targeting a TTL'd index's subtrees are classified ephemeral: the walkers route them into a separate operation batch (LowLevelDriveOperation::EphemeralGroveOperation), applied after the standing batch, whose captured cost is consumed on its own terms — added bytes bill to processing at FeeStorageVersion::ttl_ephemeral_disk_usage_credit_per_byte (270 credits/byte, 1% of the storage rate, ~27× a pro-rata week of retention) and the storage fee contribution is zero. The elements carry no storage flags, so deletion — the TTL drain or a user delete of a not-yet-expired document — is basic removal with no refund entries: there is nothing to refund, which is also where TTL writers collectively pre-pay the drainage described below. Cost estimation routes through the same split, so estimated and actual fees stay in the same class.

Drainage itself and the walkers' TTL bookkeeping reads are unbilled: their costs go to scratch accounting, never to the triggering user. That is load-bearing for the estimated >= actual fee invariant — the estimation dry run cannot read state and therefore cannot price state-dependent drainage, so billing it only on execution would let a transition pass validation and then overdraw on apply. The unbilled work is bounded: a capped count of O(1) drop operations plus a handful of bounded reads per write.

Queries

Unchanged in shape, with one hard rule: on a TTL'd index, expired windows are not queryable. byStart resolution rejects any start past the expiry horizon (the same strictly-below predicate the drain uses), on the server from committed block time and on the verifier from the quorum-signed response time_ms — so a node cannot serve an expired window's remnants past a verifying client. The point of the gate is that a mid-drainage window would otherwise serve a truncated answer that looks authoritative; rejecting the question is deterministic where "whatever the drain has left" is not. Because drainage only ever touches expired buckets, every window the resolver admits is complete, and a window a past drain fully emptied inside its lifetime never existed — absence proves normally. The relative selectors (newest / oldest) can never address an expired window at all (ttl >= range guarantees it).

Versioning

Everything rides the PV14 grammar, which was unreleased when this landed: the ttl key joined the meta-schema v3 timeRange map, the two limits joined SYSTEM_LIMITS_V4 in place, and the ephemeral-bytes rate joined the shared storage fee table directly — no fee-version fork, because the rate is dead below PV14 (the grammar does not parse, so no ephemeral-classified operation can exist to read it). No migration story exists or is needed.

Index-Only Document Types

Status: implemented and gated at protocol version 14 (meta-schema v3 / parser generation 3). The storage layout is pinned end-to-end against a real grovedb by index_only_e2e_tests, which runs against the yappr-likes fixture at packages/rs-drive/tests/supporting_files/contract/yappr-likes/yappr-likes-contract.json. The full ABCI pipeline (transitions, validation, executed-transition proofs) is exercised by the index_only test modules in rs-drive-abci's batch tests.

The problem

A minimal social interaction — a like — is fully expressed by where it sits: which post, which hashtag, which identity. Storing it as an ordinary document costs a serialized body in primary storage (~90–150 bytes plus element flags and tree-node overhead), a primary-tree insertion, and a ~70–90-byte reference per index, for a fact whose entire content is its index position.

An index-only document type (indexOnly: true on the doc-type schema) stores nothing in primary storage. The index entries ARE the rows:

[DataContractDocuments, contract_id, 1, <doctype>,
   <prop 1>, <val 1>, …, <prop K>, <val K>, 0, <terminal value>]
      → Item(<row commitment>, flags)

The terminal is the member key, sitting exactly where a normal non-unique index keys by document id; the element is an Item instead of a Reference because there is nothing to point at. It is a per-index keyword defaulting to $ownerId. It may name any schema property a prefix position could carry (an identifier with or without a refersTo, a bounded byte array or string, an integer, a boolean, a date), or an ordered list of such properties (a composite terminal, whose member key is their encoded values concatenated). The 0 storage marker, value-tree types, and the count/sum/ranked tree derivation are byte-identical to the ordinary non-unique layout, which is what lets the protocol v14 ranked machinery (see Document Ranked Trees) serve index-only types unchanged: "the five most-liked posts in #dash" is an O(log n + k) read with an O(log n + k) proof, and Items count in count/ranked trees exactly as References do.

The member key is the terminal value's tree-key encoding, produced by the same functions the prefix levels use (the walkers and probes through get_raw_for_document_type, queries and executed proofs through serialize_value_for_key, synthesis through decode_value_for_tree_keys), so nothing about it is specific to a 32-byte identifier: a 33-byte public key, a short string or an integer keys the 0 bucket exactly as it would key a prefix level, and fee estimation sizes the member key by the terminal property's declared bound (index_only_terminal_max_key_size) rather than by a fixed 32. Structural uniqueness spans the terminal value: one entry per (prefix values, terminal value), so two documents by one owner that differ only in a scalar terminal are two entries under the same prefix.

Composite terminals. "terminal": ["kind", "$ownerId"] keys the member by encode(kind) ‖ owner. Every component but the last must be fixed width (a byte array with minItems == maxItems, an identifier, an integer, a boolean, a date), so equality on the leading components is a clean key range and synthesis can split the key back; a string can only be the last component; the whole key is capped at 255 bytes. Uniqueness spans the whole key. Queries bind the components in order: equality clauses on the leading ones, then at most one range or in clause on the next (ordered by it), nothing on the rest. The lowering pads the bound prefix with 0xFF to the key cap for the upper bound of "every key under this prefix", and addresses the key itself when the bound component is the last one. After equality-bound components are ignored, orderBy must start at the first remaining component and follow component order without gaps, with the same direction for every listed component. A single member-key walk cannot sort by a later component alone or mix ascending and descending components.

Flat indexes. An index with no properties at all is flat: its entries live directly under a level of their own, keyed by a zero byte followed by each terminal component name preceded by a zero byte ("\0appEphemeralPubKeyHash\0$ownerId"), which no property-name tree can collide with since property names never contain a zero byte. This level key, including its separators, must also fit within 255 bytes:

[DataContractDocuments, contract_id, 1, <doctype>, "\0<c1>\0<c2>…", 0, <c1 ‖ c2 ‖ …>]
      → Item(<row commitment> [‖ <entry payload>], flags)

The flat level is registration-time structure, created with the property-name trees and kept when the last entry goes (the prune stops at its 0 bucket, as on a preallocated index), so every entry costs the same. There is no prefix level for an aggregate, a ranking, a time grid, a skip set or a preallocation to apply to, so a flat index admits none of those keywords. A clause-free query on a type with a flat index scans the flat level (every other indexOnly type refuses the by-id shape). Non-proof responses require this index to cover every property, including optional ones, just as filtered queries do; otherwise use a proved projection.

The entry payload. entryPayload: ["walletEphemeralPubKey", "encryptedPayload"] on the document type names top-level properties that live in no index: every entry's item carries them after the 32-byte row commitment, each length-framed (u16 big-endian), in property-name order: the type's value slot. Byte arrays store their raw bytes and strings store UTF-8; other scalars use their tree-key encoding. The length frame preserves empty byte arrays and strings without null sentinels, and distinguishes an empty string from a NUL string. A payload property must be required, scalar and bounded (the sum of the bounds is capped by the field value limit), and appears in no index as a property or a terminal component. It is still committed (the commitment hashes every present property, a payload value through the uncapped payload encoding), so the delete probes and the executed-transition verifier keep comparing the item's first 32 bytes only, and synthesis decodes the rest of the proved element. With more than one index the payload rides in every entry; fee estimation sizes the item by the commitment plus the payload bound. Together, a flat composite terminal and an entry payload make a key-value table:

"indices": [{ "name": "byRequest", "terminal": ["appEphemeralPubKeyHash", "$ownerId"] }],
"entryPayload": ["walletEphemeralPubKey", "encryptedPayload"]

lands at […, "\0appEphemeralPubKeyHash\0$ownerId", 0, hash ‖ owner] → Item(commitment ‖ len ‖ ciphertext ‖ len ‖ wallet key), and a query on the hash returns every responder's owner id with the payload decoded off the item, as one proof.

timeRange buckets compose too: a bucketed indexOnly index writes one commitment entry per containing bucket under the grid-qualified level, exactly as stored types do — the walkers' bucket fan-out, the probes' path derivation (entry_keys_for_raw, shared so probe and write paths cannot drift), and the IN_TIME_RANGE count aggregates are all the same machinery ("how many likes under #dash this hour"). The source can only be $createdAt (the prefix rule admits no other timestamp), which required must carry, so a delete's values reproduce the exact bucket set. A bucketed index involves $createdAt and therefore never serves as the proof index; and document synthesis over bucketed entries is refused with guidance — the bucket level carries bucket-start granularity, not the document's timestamp, and the raw entries are served by the type's non-bucketed indexes.

The sum axes compose the same way: a summable: "<prop>" index stores ItemWithSumItem(<row commitment>, <amount>) terminals — the same commitment payload, plus the summed property's value — so entries contribute to ancestor sum trees exactly as stored types' ReferenceWithSumItem references do ("total tipped to this post", "top posts by total tipped" via rankedSummable). The doctype-level summable cross-checks (one canonical summed property, i64-safe integer type, required membership) apply unchanged, and on delete grovedb reads the amount off the stored element and propagates the subtraction — the falsified-amount case dies on the commitment probe first, since the amount is one of the committed properties.

Governing principle: only what is in the indexes exists and is recoverable. Prefix property values live in the path, the terminal id in the member key, $ownerId and $createdAt wherever an index carries them. There is no document beyond that. One kind of property may sit in no entry-keeping index: one a summableOffCountIndex index's source fixes through a reference's where (a like's postAuthor, kept only by byAuthorPost's counters). Such a property, when no entry-keeping index holds it, is lacking from a document read back, and its value is the referenced document's, so a client rebuilding the row for a delete reads it from there.

The row commitment

Each entry's 32-byte payload is hash_double(owner ‖ (name ‖ length ‖ raw index bytes)* ‖ [$createdAt]) over the document's PRESENT properties in sorted-name order (index_only_row_commitment), $createdAt included unless only indexes whose entries outlive a delete involve it (index_only_row_commits_created_at, see below) — every required property must be present, and an optional property (a skip property of a skipIfAbsent index, the only optional kind) contributes nothing when absent, not even its name. It binds the independently stored index projections of one document back into one logical row: a delete recomputes the commitment from its submitted values, and every probed entry must carry it. A values tuple spliced from two different creates — even two creates by the same owner — fails the comparison on whichever entry belongs to the other row. Entry existence alone cannot make that distinction; the commitment is what does. The present-set is part of what is committed: absent (no name emitted) and present-but-empty (name ‖ 00000000) hash differently, and the variable present-set stays unambiguous because property names never contain a zero byte while every length prefix starts with one.

Constraint matrix (parse-time, apply_index_only)

The on-disk layout depends on every one of these, so they run regardless of full_validation — the same untrusted-boundary rule the doctype aggregate keywords follow:

ConstraintWhy
every property in required — except a skip property of a skipIfAbsent index; every ancestor of an indexed dotted path requiredthe index path is the storage; no null layout exists — the one sanctioned hole removes the whole entry instead
every index holding an optional property skips on it (its skip set is all of its optional properties)an absent value that did not skip would need the null layout this mode has no equivalent of
every required property appears in ≥ 1 non-skip index (prefix or terminal); every optional property appears in a skip index whose skip set is that property aloneonly indexed values exist, and a skip index carries no value for a document it skips — covered only there, a property would be validated and committed yet written nowhere
every index embeds $ownerId (prefix or terminal)entries are self-authorizing: a delete computed with owner = signer can only ever address the signer's own entries
≥ 1 index is $createdAt-free AND non-skipIfAbsent — the proof indexexecuted-transition proofs locate entries from the transition's values alone: they can neither reproduce a block timestamp nor anchor on an entry that may not exist
every terminal component is $ownerId or a schema property passing the indexed-shape limits (no arrays or objects; byte arrays ≤ 255 bytes, strings ≤ 63 characters); every component but the last is fixed width; the whole key ≤ 255 bytesthe member key is the components' tree-key encodings concatenated, derived by the same functions the prefix levels use; a leading component must be splittable back and rangeable; grovedb caps keys at 255 bytes; other system properties are refused because the $createdAt rules walk the prefix properties
no index declares integerRangean entry is keyed by the index's values and terminal, and a bucketed level holds window starts: rows differing only in the bucketed integer would claim the same entry in every window they share
a flat index (no properties) admits no countable / summable / ranked / timeRange / integerRange / skipIfAbsent / preallocated keywordthere is no prefix level for them to apply to
every entryPayload property is a required, bounded, top-level scalar in no indexthe entry value has no representation for an absent property, estimation sizes the item by the bounds, and a property is either a key or a value
indexed $createdAt requires $createdAt in requiredcreation only assigns timestamps for required system times
documentsMutable: false, no transfers/trading/history/transientno stored row, no revision
non-unique, non-contested, nullSearchable defaultv1 scope
preallocated requires a fully reference-determined, non-bucketed pathsee Preallocated index paths
a skip property is an optional, top-level schema property; no ranking sits above the index's deepest skip propertysee Conditional participation
a summableOffCountIndex index sums a source holding every document once, holds every source property, and fixes its other properties through unchanging where valuessee Counters (summableOffCountIndex)

indexOnly and the index set (terminals included, preallocated and skipIfAbsent flags included) are immutable across contract updates — a later-added index could never be backfilled, and the walkers derive the skip from each index's skip set, which therefore cannot drift from historical entries either. (Respelling skipIfAbsent: true as the array of the same properties is no change: both parse to the same skip set.)

Conditional participation (skipIfAbsent)

An index may declare skipIfAbsent: a document that omits a property of the index's skip set writes no entry into that index at all, and a delete recomputes the same skip from its carried values. true makes the skip set every optional property of the index (on an indexOnly type it must be that set anyway, so the array spelling, naming it, parses to the same index); a skip property may sit at any position, a timeRange window included. The skip properties are the only properties that may leave required, and the rules keep three views provably equivalent: the write walkers write an index's entry only for a document carrying its skip set (document_takes_part_in_index), the probes derive zero entry paths for the same documents, and the row commitment pins the exact present-set so a delete with a different absence pattern fails every probe — a skip index can neither be force-pruned nor left with an orphan entry.

No stranded trees. The merged index structure shares levels across indexes and prunes empty trees only upward from an entry, so the walkers build a level only when an index the document takes part in ends at or below it (level_reaches_entry). An untagged like under [$createdAt window, hashtag, postId] still builds the day window shared with byDayPost, but no hashtag branch under it; under a grid that only a skip index uses, it builds no window at all. On an indexOnly type every index holding an optional property skips on it, so a level keyed by an absent value is never reached.

The semantics are a sparse projection: the index holds exactly the documents carrying its skip set. Counts, ranked reads and absence proofs over it answer "among documents with these properties" — a proved empty position means "no tagged like", not "no like". The query router makes that opt-in (index_admissible_for_skip_if_absent, in every index picker, compiled into the verifier too): a skip index is admissible only when the query binds every skip property (equality, in, range, order-by, or the ranked property); the generic matcher alone would admit an unbound query within its difference budget, and an aggregate picker's prefix match could stop above a deep skip property, silently omitting every skipped row. A ranking never sits above a skip property (refused at registration: no query could read it). Structural uniqueness still spans the skip boundary — a skipped and a taking-part document colliding on any shared non-skip entry cannot coexist. An absent skip property is distinct from a present-but-empty value, which indexes normally under its encoded key.

Coverage. A document carrying one optional property but missing another skips every index holding both, so each optional property needs a skip index whose skip set is that property alone: its value is then written whenever the document carries it, and the document stays deletable by values.

The economics are the point: each ranked index costs roughly the same on every write, so a per-hashtag ranked index on a like doctype used to tax every like — tagged or not — and forced a '' sentinel onto the referenced post's hashtag. With byHashtagPost as a skip index, an untagged like pays only for the indexes it actually appears in, and the sentinel disappears (see the absence-aware where below).

Lifecycle

  • Create reuses DocumentCreateTransitionV0 unchanged. State validation probes every index's entry: ANY existing entry is a duplicate (DuplicateUniqueIndexError), which is also what makes a shorter index a uniqueness constraint over its value projection plus owner — for likes, the [postId] index is the one-like-per-(post, owner) rule. refersTo validation runs unchanged (it reads transition values, not storage), so a like on a nonexistent post is rejected, and a where declaration on the reference ({ "hashtag": "hashtag" }) binds the referenced post's property to the like's own: the referenced document is already fetched for the existence check, so the equality comparison adds no reads, and a like whose hashtag disagrees with its post's is refused. Absence is part of the comparison, strictly: both sides absent agree, one side absent is the same mismatch a differing value would be — a like may omit its hashtag exactly when its post has none (anything laxer would let likes on tagged posts silently deflate per-tag aggregates), which is what lets a compared property double as a skipIfAbsent skip property with both sides of the reference optional. The key of an entry, the referenced side, may also name the referenced document's $ownerId or $creatorId (the referring side must then be an identifier property): { "$ownerId": "authorId" } binds a like to its post's current owner, so an [authorId, postId] index can be preallocated and ranked per author. $ownerId follows the post through transfers and $creatorId never changes; either is checked when the like is written, not when the post later moves. $creatorId is only recorded by transferable or tradeable types of a format-1 contract, which contract registration checks before accepting the declaration. The referring side, an entry's value, may in turn be the like's own $ownerId, the writer: { "$ownerId": "$ownerId" } lets only the post's current owner create or replace a like on it, { "$creatorId": "$ownerId" } only its original creator. That is a write gate, checked on create and on every replace of the like, not only when its reference changes, since the post may have been transferred in between; a transfer itself is not re-checked, so on a transferable referring type it governs writing, not holding. A writer gate does not make an owner-prefixed index preallocatable.
  • Delete is its own transition kind, DocumentIndexOnlyDeleteTransition { base, data } ($action: "indexOnlyDelete"), carrying the full value tuple ($createdAt under its system key exactly when the row commits to it: index_only_row_commits_created_at). Delete-by-id and delete-by-values are different operations — different payload, authorization model and validation pipeline — so the factory picks the KIND from the doctype's storage mode. Validation and the storage layer both require every entry to exist AND match the row commitment. A by-id delete on an index-only type (and an indexOnlyDelete on a stored type) is rejected by the structure gates; below PV14 the kind is rejected at basic structure, keeping check_tx behavior aligned with pre-4.2 software.
  • Replace / transfer / purchase / price are structurally impossible.

Entries that outlive a delete (outlivesDelete)

A delete-by-values recomputes every entry of the document from its values, and a time-window entry is keyed by the bucket starts of $createdAt, which only the block that included the create assigned. A client that did not keep that timestamp could not delete the document. outlivesDelete: true on a timeRange index with a ttl takes the window out of the delete:

  • Delete. Neither the state validation probes nor Drive's row-integrity gate check the index, and the delete walkers do not descend into it: level_removes_entry skips a level whose indexes all outlive the delete (IndexLevel::outlives_delete_at_or_below keeps every other contract on the old path), and the terminating level of such an index removes nothing. Its entries stay until their window is dropped by the TTL cleanup, which drops whole buckets.
  • Commitment. When every index involving $createdAt outlives deletes, the row commitment leaves the timestamp out, and a delete carries none (structure validation refuses one that does). The executed-transition proof verifier computes the same commitment, so it no longer needs the block time for such a type.
  • Create. State validation does not probe the index for a duplicate, and the terminal insert writes over an entry already standing at the same key (left by a deleted document with the same key) without reading it: the count does not move, and the entry carries the commitment of the row writing it. Registration makes the index's key hold the key of an index a delete clears that skips nothing, so no two documents in state share an entry, and the within-batch collision tracker leaves these entries to that index.
  • Proofs. index_only_proof_index never picks such an index, and the parser requires another one to exist.

Registration admits the keyword only on an indexOnly timeRange index with a ttl, without a sum and on a type without entryPayload (a kept entry holds the first document's amount or payload), and requires every schema property to sit in an index that neither skips nor outlives deletes, and the index's key to hold the key of such an index. The flag is fixed with the index (find_first_outlives_delete_change). The index structure caches the decisions a delete reads (IndexLevel::cleared_on_delete_at_or_below, and at its root created_at_indexed_only_by_outliving), and Drive's delete refuses a document carrying a $createdAt its row does not commit to.

The cost is on the aggregates: a deleted document still counts in the windows it wrote until they move past it.

Preallocated index paths

The first entry under a fresh value tuple pays for every tree on its path — for a like that is the hashtag value tree, the postId property-name tree, the post's value tree and the 0 member bucket — while the second entry pays for one item insert. When the index path is a pure function of a refersTo-referenced document, that lopsidedness is avoidable: an index may declare preallocated: true iff every index property is either the referring property itself (its value is the referenced document's $id) or a referring value of that reference's where (consensus-equal to a referenced-document property, its $ownerId and $creatorId included), and the reference is a permanentDocument or moderatedDocument one targeting a document type of the same contract. A deletableDocument reference shapes the path the same way but does not qualify: its target can be deleted without a record, and the trees created alongside it would outlive it with other owners' entries inside and nothing left to say what they were keyed by. A moderatedDocument target leaves state only through a moderator's removal, whose record is never deleted, so its trees outlive it the way the record does, and a restore (which puts the document back through the create path) finds them in place. Through such a reference a binding counts only when the record keeps every key it binds: the referenced $id or $ownerId, or a property the referenced type lists under moderatorAbilities.deleteKeepsFields (PreallocationBinding::is_kept_on_removal). Registration refuses a preallocated index with no such binding (validate_preallocated_indexes_kept_on_removal, once every document type of the contract is parsed), and the insert path preallocates only through one (Index::preallocation_bindings_for_target, given the referenced type). byHashtagPost ([hashtag, postId]) qualifies: hashtag through where, postId as the reference; byLiker ([$ownerId]) cannot, since no referenced document determines the liker. An [authorId, postId] index whose authorId agrees with the post's $ownerId qualifies too: the poster is the one owner a referenced post does determine.

Three things change, all bit-compatible with the fallback layout:

  • Insert side (insert/add_preallocated_index_tree_operations): inserting the referenced document also emits if-not-exists creations of the referring index's dynamic trees, down to the empty 0 member bucket, derived through the same tree-type helper the entry walkers use — so a preallocated tree is byte-identical to the tree the first entry's create-on-insert path would have made. The poster pays for the structural bytes; storage flags ride only when the contract itself is deletable, because that is a preallocated tree's one deletion path — entry deletes retain it by design, so entry-level flags would be unrefundable dead weight. Shared prefixes (a second post under the same hashtag) deduplicate through the if-not-exists semantics.
  • Delete side: removing the last member entry stops the empty-tree-pruning climb at the member level, keeping the whole apparatus — the group stays in the ranked secondaries at count 0, and a re-entry is again a plain item insert. (Non-preallocated indexes of the same type keep pruning as before.)
  • Nothing else: entry insertion keeps its create-if-missing behavior, so correctness never depends on preallocation. Referenced documents created before a contract update introduced a referring type simply hand the first entry the old price, and their trees — created by the fallback — are retained on delete exactly like preallocated ones.

preallocated composes with skipIfAbsent: every bound key is resolved before any operation is emitted, and an absent bound value (an untagged post's hashtag, under the absence-aware agreement) bails without emitting anything — so a tagged post preallocates the skip index's trees, an untagged post preallocates only its id-bound indexes, and an untagged like skips exactly the trees that were never built.

The economics: the referenced document's creator pays for the trees whether or not anyone ever references it (which is why the flag is an explicit opt-in, per index), every entry from the first on costs the same, and "no entries yet" becomes a present-but-empty member bucket — provable as zero results, rankable as a zero-count group — instead of an absent tree.

An entry's proved (path, key) position IS the document, so queries and proofs synthesize documents through one shared builder (query/index_only_synthesis.rs, compiled for server and verify): prefix properties decoded from the path via decode_value_for_tree_keys (the inverse of the write path's key encoding), the terminal from the member key. A query through a subset index yields a documented projection. The synthesized $id is deterministic over the proved position (a domain-separated, length-framed hash covering every non-owner component, $createdAt included) — nothing on chain is ever addressed by it.

Executed-transition proofs (waitForStateTransitionResult) prove a create by the presence of the entry its values produce under the proof index (the first $ownerId-bearing, non-skipIfAbsent index not involving $createdAt — contract admission guarantees one exists) and a delete by its absence, with the proved entry's payload checked against the transition-derived row commitment (and, when the proof index is summable, the proved sum contribution against the created document's amount); prover and verifier build the same single-entry path query from the transition. The outcome is always AffectedState, never ExecutionProved: the commitment carries neither id, entropy nor nonce, so a snapshot cannot bind one specific transition's execution.

Where clauses on the terminal property lower directly onto the entry level's member keys once every prefix property carries an equality clause: an equality answers "did I like X" in one query, and a range ordered by the terminal (terminal > <last seen>, with a limit) walks the entries page by page — keyset pagination, the indexOnly replacement for id-shaped startAt cursors, which cannot address a position whose synthesized id is a one-way hash. Mixed shapes are served through a prefix pivot: one in clause may sit on the index's last prefix property instead of the terminal (hashtag == h AND postId IN [p, q] AND $ownerId == me), with everything above it equality-bound, the terminal clause an equality, and a limit of at least the number of in values.

A range pivot (postId > p in the same query), or an in pivot with prefix properties below it, is refused, and the error names the index shape that serves the query: one that lists the equality-bound properties, the terminal's included, before the ranged property. A pivot walk opens one branch per pivot value, and grovedb charges a branch that holds no row one slot of the limit, so a page of such a query could hold fewer rows than exist, and the response carries no cursor to say where it stopped. These shapes stay refused until the storage layer can report where a page stopped. An in pivot on the last prefix property opens at most one branch per value, so a limit that covers its values is never used up early. When another index serves the same query without an incomplete pivot, index selection prefers it over a pivot index that would win the name-order tie-break.

All shapes prove and verify through the same shared path-query builder.

Not supported on the read surface: by-$id fetches (no primary tree — rejected with guidance) and startAt cursors (rejected with the keyset guidance above); ranked / count / range-aggregate queries work unchanged since they never open value trees.

Counters (summableOffCountIndex)

A summableOffCountIndex index keeps no entry per document. At the value position of its last property, where another index grows a value tree, a 0 bucket and one entry per document, it keeps one Element::SumItem holding the number of entries its source index keeps for that group:

before: byAuthorPost → postAuthor → <author> → postId → <post> → 0 → <liker> = Item(commitment)
after:  byAuthorPost → postAuthor → <author> → postId → <post> = SumItem(likes)

The tree of the last property is a count-and-sum tree (rangeSummable, plus rangeCountable for the group count an average divides by), so each counter counts one group and adds its value to the sum. A level a { "at": ... } ranking names, and every level between it and the counters, carries those totals up: its value trees are CountSumTrees from the shallowest average ranking down and SumTrees above it (a rankedCountable on such an index is its sum ranking, so no count-only chain arises). The property-name tree of a level a ranking names is the indexed tree for the axes ranked at it, ProvableCountProvableSumIndexedTree for [Sum, Avg]; a level between two ranked levels keeps a plain count-and-sum or sum tree. Grovedb admits a bare SumItem under that indexed tree from grove version 4.

The write path (add_summable_off_count_counter_operations):

  • Create: reads the counter and writes it back one higher, or inserts it at one for the first document of the group. The create is refused before it gets here when the source already holds the entry, so the counter equals the source group's entry count.
  • Delete: writes it back one lower once the entries of the indexes that keep them matched the row commitment. A preallocated index keeps the counter at zero; any other removes it with its last document and prunes the trees it leaves empty, up to the document type.
  • Once per batch: a counter is written at most once per batch, because a documents batch carries one transition. A source keyed by more than its owner (a terminal such as ["$ownerId", "emoji"]) holds several entries of one owner in one group, and each document is converted on its own, so raising that cap needs the counter moves folded across documents first. A second write within one conversion is refused as corrupted code execution.
  • Storage: a SumItem is charged a fixed 11 bytes plus flags whatever its value, so a rewrite stores nothing new, and it keeps the flags of the first document that paid for it.
  • Preallocation: creating the referenced document creates the counter at zero, in place of the value tree and its empty 0 bucket.

The state probes and the duplicate check skip the index (it decides nothing about a create), it is never the proof index, and document queries never read it. The count, sum, average and ranked queries do: a sum query names the source index (sum(byPost)), a count query takes a counter's sum (its group's documents) where another index's read takes a count (a ranked or HAVING count walks the Sum secondaries, and its entries come back as counts), and a point query may stop at a level carrying the sums, reading that value tree's element. A range count reads the range sums (DriveDocumentCountQuery::counter_sums_query): every range count executor and verifier hands the same index and clauses to the sum surface's counterpart, summing the source index, and reads the sums back as counts, since grovedb's range count over the counters would count them, one per group. A range total over an index whose path passes through a ranked level (its own, or one another index ranks at a shared level) is refused with a hint to group by the last property, as everywhere: a ranked level's tree is indexed, and grovedb neither totals a range over an indexed tree nor proves a range total through one, so the unproven read refuses it too and the two agree.

What it costs and what it saves

Registration skips the [0] primary-key tree. Each document is exactly one […values, 0, terminal] → Item(32-byte commitment) per index — no primary row, no references — cutting storage well past half against a minimal stored document, with deletion refunds flowing from each entry's own element flags (the index walkers pass flags for immutable-yet-deletable index-only types specifically). Estimation pads the dry-run item above the real payload so estimated fees keep upper-bounding applied fees across the indexed-tree layers' documented under-count.

Unit Tests

If you have spent any time reading the Dash Platform codebase, you have probably noticed that test files are everywhere -- and they follow a very specific structure. This chapter walks through the patterns that Platform's unit tests use, why those patterns exist, and how to write your own tests that fit naturally into the codebase.

The Test Module Convention

Nearly every test file in Platform follows the same opening stanza:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;
    // ... additional imports ...
}
}

This is standard Rust, but the consistency matters. The #[cfg(test)] attribute means the entire module is compiled only when running cargo test. The use super::*; import pulls in everything from the parent module, so tests can access the types and functions they are testing without repeating import paths.

In Platform, tests that validate state transitions live in dedicated tests.rs files:

packages/rs-drive-abci/src/execution/validation/
  state_transition/state_transitions/
    address_credit_withdrawal/
      tests.rs            <-- unit tests for withdrawal transitions
    address_funds_transfer/
      tests.rs            <-- unit tests for address-to-address transfers

Each tests.rs file is a self-contained test module for one state transition type. Inside, tests are further organized into sub-modules by category:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    // ... imports and helpers ...

    mod structure_validation {
        use super::*;

        #[test]
        fn test_no_inputs_returns_error() { /* ... */ }

        #[test]
        fn test_too_many_inputs_returns_error() { /* ... */ }
    }

    mod address_state_validation {
        use super::*;
        // ...
    }

    mod witness_validation {
        use super::*;
        // ...
    }
}
}

This sub-module approach groups related tests, making cargo test output scannable. When a test fails, you immediately see tests::structure_validation::test_no_inputs_returns_error instead of a flat list.

TestPlatformBuilder: Setting Up the World

Most unit tests need a running Platform instance with a database, genesis state, and configuration. The TestPlatformBuilder provides a fluent API for this:

#![allow(unused)]
fn main() {
// File: packages/rs-drive-abci/src/test/helpers/setup.rs

pub struct TestPlatformBuilder {
    config: Option<PlatformConfig>,
    initial_protocol_version: Option<ProtocolVersion>,
    tempdir: TempDir,
}
}

The builder creates a TempPlatform -- a Platform instance backed by a temporary directory that is automatically cleaned up when the test finishes:

#![allow(unused)]
fn main() {
pub struct TempPlatform<C> {
    pub platform: Platform<C>,
    pub tempdir: TempDir,
}
}

Here is the typical setup pattern:

#![allow(unused)]
fn main() {
let platform = TestPlatformBuilder::new()
    .with_config(platform_config)
    .with_latest_protocol_version()
    .build_with_mock_rpc()
    .set_genesis_state();
}

Let's break this down:

  • new() creates a builder with a fresh TempDir.
  • with_config() injects a PlatformConfig (including test-specific overrides).
  • with_latest_protocol_version() pins the platform to the current protocol version.
  • build_with_mock_rpc() constructs the Platform with a MockCoreRPCLike -- no real Dash Core node needed.
  • set_genesis_state() writes the initial state tree (system data contracts, etc.) into the database.

Because TempPlatform implements Deref<Target = Platform<C>>, you can call Platform methods directly on it:

#![allow(unused)]
fn main() {
impl<C> Deref for TempPlatform<C> {
    type Target = Platform<C>;

    fn deref(&self) -> &Self::Target {
        &self.platform
    }
}
}

This means platform.drive, platform.state, and platform.config all work directly.

Helper Functions: Encapsulating Test Patterns

Each test file defines local helper functions that encapsulate repeated setup logic. For example, the withdrawal tests define helpers for creating transitions:

#![allow(unused)]
fn main() {
fn create_signed_address_credit_withdrawal_transition(
    signer: &TestAddressSigner,
    inputs: BTreeMap<PlatformAddress, (AddressNonce, u64)>,
    output: Option<(PlatformAddress, u64)>,
    fee_strategy: Vec<AddressFundsFeeStrategyStep>,
    output_script: CoreScript,
) -> StateTransition {
    AddressCreditWithdrawalTransitionV0::try_from_inputs_with_signer(
        inputs,
        output,
        AddressFundsFeeStrategy::from(fee_strategy),
        1, // core_fee_per_byte
        Pooling::Never,
        output_script,
        signer,
        0, // user_fee_increase
        PlatformVersion::latest(),
    )
    .expect("should create signed transition")
}
}

And helpers for submitting them to the platform:

#![allow(unused)]
fn main() {
fn check_tx_is_valid(
    platform: &TempPlatform<MockCoreRPCLike>,
    raw_tx: &[u8],
    platform_version: &PlatformVersion,
) -> bool {
    let platform_state = platform.state.load();
    let platform_ref = PlatformRef {
        drive: &platform.drive,
        state: &platform_state,
        config: &platform.config,
        core_rpc: &platform.core_rpc,
    };

    let check_result = platform
        .check_tx(raw_tx, CheckTxLevel::FirstTimeCheck,
                   &platform_ref, platform_version)
        .expect("expected to check tx");

    check_result.is_valid()
}
}

The key insight is that helpers should be specific to the test file. A withdrawal test's helper knows about AddressCreditWithdrawalTransition; it does not try to be a generic state transition factory.

assert_matches! for Error Checking

Platform tests lean heavily on the assert_matches! macro from the assert_matches crate. This is the idiomatic way to verify error variants in a deeply nested enum hierarchy:

#![allow(unused)]
fn main() {
use assert_matches::assert_matches;

assert_matches!(
    processing_result.execution_results().as_slice(),
    [StateTransitionExecutionResult::UnpaidConsensusError(
        ConsensusError::BasicError(
            BasicError::TransitionNoInputsError(_)
        )
    )]
);
}

Without assert_matches!, you would need a verbose match block or a chain of if let statements. The macro makes the expected shape of the error immediately visible in the test.

For cases where you need to inspect error fields, combine matches! with additional assertions:

#![allow(unused)]
fn main() {
let error = result.first_error().unwrap();
assert!(
    matches!(
        error,
        ConsensusError::BasicError(
            BasicError::TransitionOverMaxInputsError(e)
        ) if e.actual_inputs() == 17 && e.max_inputs() == 16
    ),
    "Expected TransitionOverMaxInputsError with 17/16, got {:?}",
    error
);
}

The guard clause (if e.actual_inputs() == 17) lets you verify both the variant and its contents in a single expression.

Processing State Transitions in Tests

The standard way to submit a state transition in unit tests is through process_raw_state_transitions:

#![allow(unused)]
fn main() {
let raw_bytes = transition.serialize_to_bytes().unwrap();

let processing_result = platform
    .platform
    .process_raw_state_transitions(
        &vec![raw_bytes],
        &platform_state,
        &BlockInfo::default(),
        &transaction,
        platform_version,
        false,   // not dry run
        None,    // no extra data
    )
    .expect("expected to process state transition");
}

This is the same code path that runs in production -- your test transition goes through the same validation pipeline that a real block proposer executes.

Deterministic Randomness

Tests that need random data always use a seeded RNG:

#![allow(unused)]
fn main() {
let mut rng = StdRng::seed_from_u64(567);
let output_script = CoreScript::random_p2pkh(&mut rng);
}

The seed ensures the test produces identical results every time. If a test fails, you can reproduce the exact same inputs. Never use thread_rng() or entropy-seeded RNGs in unit tests.

Feature-Gated Test Compilation

Some tests require features that are expensive or only available in certain contexts. The testing-config feature gate controls test-specific configuration:

#![allow(unused)]
fn main() {
#[cfg(feature = "testing-config")]
impl PlatformTestConfig {
    pub fn default_minimal_verifications() -> Self {
        Self {
            block_signing: false,
            store_platform_state: false,
            block_commit_signature_verification: false,
            disable_instant_lock_signature_verification: true,
            disable_checkpoints: true,
        }
    }
}
}

Tests that need the full platform test infrastructure will not compile without --features testing-config, keeping the main build clean.

OnceLock for Expensive Resources

When a test suite needs an expensive-to-create resource (like a cryptographic key that takes 30 seconds to build), the OnceLock pattern avoids rebuilding it for every test:

#![allow(unused)]
fn main() {
use std::sync::OnceLock;

static STATE_TRANSITION_TYPE_COUNTER: OnceLock<Mutex<BTreeMap<String, usize>>>
    = OnceLock::new();

fn state_transition_counter() -> &'static Mutex<BTreeMap<String, usize>> {
    STATE_TRANSITION_TYPE_COUNTER.get_or_init(|| Mutex::new(BTreeMap::new()))
}
}

OnceLock is initialized at most once, the first time any test calls it. Because test threads share the same process, all tests in the binary reuse the same instance. This pattern is essential for resources like cryptographic proving keys that are expensive to construct but immutable once built.

Rules

Do:

  • Follow the #[cfg(test)] mod tests { use super::*; } convention.
  • Group tests into sub-modules by validation category.
  • Use TestPlatformBuilder for any test that needs a platform instance.
  • Use StdRng::seed_from_u64() for deterministic randomness.
  • Use assert_matches! for checking error variants.
  • Use OnceLock for expensive, immutable test resources.
  • Process transitions through process_raw_state_transitions to test the real code path.

Don't:

  • Use thread_rng() or unseeded randomness in tests.
  • Create ad-hoc Platform instances without TestPlatformBuilder.
  • Write generic helper functions that try to handle all state transition types.
  • Skip set_genesis_state() unless you are specifically testing pre-genesis behavior.
  • Use unwrap() on validation results -- use assert_matches! to verify error shapes.

Strategy Tests

Unit tests verify that a single state transition behaves correctly. But what about testing an entire chain of blocks with hundreds of identities creating documents, transferring credits, and voting on contested resources -- all at the same time?

That is what strategy tests are for. They are Platform's integration-level simulation framework: you declare what should happen and let the framework simulate it across hundreds of blocks.

The Problem

Consider everything that happens in a real Dash Platform network over 100 blocks:

  • Masternodes join, leave, get banned, change IPs
  • Quorums rotate and sign blocks
  • Identities are created, topped up, and updated
  • Documents are inserted, replaced, deleted, and transferred
  • Contracts are deployed and updated
  • Withdrawals are processed and batched
  • Protocol upgrades happen mid-chain

Testing any of these in isolation is straightforward. Testing them together -- where the output of block 47 affects the input of block 48 -- requires something more powerful than a unit test.

Two-Layer Strategy Architecture

Strategy tests use a two-layer design:

Layer 1: Strategy (defined in packages/strategy-tests/src/lib.rs) describes what operations to perform:

#![allow(unused)]
fn main() {
pub struct Strategy {
    /// Identities to create on the first block.
    pub start_identities: StartIdentities,

    /// Platform addresses to fund on the first block.
    pub start_addresses: StartAddresses,

    /// Contracts to deploy on the second block,
    /// with optional scheduled updates.
    pub start_contracts: Vec<(
        CreatedDataContract,
        Option<BTreeMap<u64, CreatedDataContract>>,
    )>,

    /// Operations to execute each block.
    pub operations: Vec<Operation>,

    /// Configuration for ongoing identity creation.
    pub identity_inserts: IdentityInsertInfo,

    /// Optional nonce gaps for edge-case testing.
    pub identity_contract_nonce_gaps: Option<Frequency>,

    /// Key manager for signing state transitions.
    pub signer: Option<SimpleSigner>,
}
}

Layer 2: NetworkStrategy (defined in packages/rs-drive-abci/tests/strategy_tests/strategy.rs) wraps a Strategy with network-level configuration:

#![allow(unused)]
fn main() {
pub struct NetworkStrategy {
    pub strategy: Strategy,
    pub total_hpmns: u16,
    pub extra_normal_mns: u16,
    pub validator_quorum_count: u16,
    pub chain_lock_quorum_count: u16,
    pub instant_lock_quorum_count: u16,
    pub initial_core_height: u32,
    pub upgrading_info: Option<UpgradingInfo>,
    pub core_height_increase: CoreHeightIncrease,
    pub proposer_strategy: MasternodeListChangesStrategy,
    pub rotate_quorums: bool,
    pub failure_testing: Option<FailureStrategy>,
    pub query_testing: Option<QueryStrategy>,
    pub verify_state_transition_results: bool,
    pub max_tx_bytes_per_block: u64,
    pub independent_process_proposal_verification: bool,
    pub sign_chain_locks: bool,
    pub sign_instant_locks: bool,
    // ...
}
}

The separation is intentional. Strategy is about application-level behavior (documents, identities, contracts). NetworkStrategy is about network-level behavior (masternodes, quorums, block production). By composing them, you can test the same application strategy under different network conditions.

Operations and Frequency

Each operation in a strategy has a type and a frequency:

#![allow(unused)]
fn main() {
pub struct Operation {
    /// The type of operation to perform.
    pub op_type: OperationType,
    /// Configuration controlling how often this operation occurs.
    pub frequency: Frequency,
}
}

OperationType is an enum covering every kind of platform action:

#![allow(unused)]
fn main() {
pub enum OperationType {
    Document(DocumentOp),
    IdentityTopUp(AmountRange),
    IdentityUpdate(IdentityUpdateOp),
    IdentityWithdrawal(AmountRange),
    ContractCreate(RandomDocumentTypeParameters, DocumentTypeCount),
    ContractUpdate(DataContractUpdateOp),
    IdentityTransfer(Option<IdentityTransferInfo>),
    ResourceVote(ResourceVoteOp),
    // ... token operations, address operations, etc.
}
}

Frequency controls when and how many operations occur per block:

#![allow(unused)]
fn main() {
pub struct Frequency {
    /// Range for the number of events when a block is selected.
    pub times_per_block_range: Range<u16>,
    /// Probability (0.0 to 1.0) that events occur in a given block.
    pub chance_per_block: Option<f64>,
}
}

For example, Frequency { times_per_block_range: 1..4, chance_per_block: Some(0.5) } means: on each block, there is a 50% chance that 1-3 operations of this type will occur. This probabilistic scheduling creates realistic, varied block content.

Running a Strategy: run_chain_for_strategy

The engine that drives everything is run_chain_for_strategy, defined in packages/rs-drive-abci/tests/strategy_tests/execution.rs:

#![allow(unused)]
fn main() {
pub(crate) fn run_chain_for_strategy<'a>(
    platform: &'a mut Platform<MockCoreRPCLike>,
    block_count: u64,
    strategy: NetworkStrategy,
    config: PlatformConfig,
    seed: u64,
    add_voting_keys_to_signer: &mut Option<SimpleSigner>,
    add_payout_keys_to_signer: &mut Option<SimpleSigner>,
) -> ChainExecutionOutcome<'a> {
    // ...
}
}

This function:

  1. Generates a deterministic RNG from seed.
  2. Creates the specified number of masternodes and quorums.
  3. For each block (up to block_count):
    • Determines core height increases.
    • Generates state transitions based on the strategy's operations and frequencies.
    • Simulates ABCI PrepareProposal / ProcessProposal / FinalizeBlock.
    • Applies masternode list changes (joins, leaves, bans).
    • Rotates quorums if configured.
  4. Returns a ChainExecutionOutcome containing the final state.

The outcome struct captures everything you need to verify:

#![allow(unused)]
fn main() {
pub struct ChainExecutionOutcome<'a> {
    pub abci_app: FullAbciApplication<'a, MockCoreRPCLike>,
    pub masternode_identity_balances: BTreeMap<[u8; 32], Credits>,
    pub identities: Vec<Identity>,
    pub proposers: Vec<MasternodeListItemWithUpdates>,
    pub validator_quorums: BTreeMap<QuorumHash, TestQuorumInfo>,
    pub identity_nonce_counter: BTreeMap<Identifier, IdentityNonce>,
    pub end_epoch_index: u16,
    pub end_time_ms: u64,
    pub state_transition_results_per_block:
        BTreeMap<u64, Vec<(StateTransition, ExecTxResult)>>,
    // ...
}
}

Masternode List Changes

The MasternodeListChangesStrategy allows simulating a dynamic validator set:

#![allow(unused)]
fn main() {
pub struct MasternodeListChangesStrategy {
    pub new_hpmns: Frequency,
    pub removed_hpmns: Frequency,
    pub updated_hpmns: Frequency,
    pub banned_hpmns: Frequency,
    pub unbanned_hpmns: Frequency,
    pub changed_ip_hpmns: Frequency,
    pub changed_p2p_port_hpmns: Frequency,
    pub changed_http_port_hpmns: Frequency,
    pub new_masternodes: Frequency,
    pub removed_masternodes: Frequency,
    pub updated_masternodes: Frequency,
    pub banned_masternodes: Frequency,
    pub unbanned_masternodes: Frequency,
    pub changed_ip_masternodes: Frequency,
}
}

Each field uses Frequency, so you can say "ban 1-2 HPMNs per block with 10% probability" naturally.

Writing a Strategy Test

Here is a minimal strategy test from the codebase (packages/rs-drive-abci/tests/strategy_tests/test_cases/basic_tests.rs):

#![allow(unused)]
fn main() {
#[test]
fn run_chain_nothing_happening() {
    let strategy = NetworkStrategy {
        strategy: Strategy {
            start_contracts: vec![],
            operations: vec![],
            start_identities: StartIdentities::default(),
            start_addresses: StartAddresses::default(),
            identity_inserts: IdentityInsertInfo::default(),
            identity_contract_nonce_gaps: None,
            signer: None,
        },
        total_hpmns: 100,
        extra_normal_mns: 0,
        validator_quorum_count: 24,
        chain_lock_quorum_count: 24,
        upgrading_info: None,
        proposer_strategy: Default::default(),
        rotate_quorums: false,
        failure_testing: None,
        query_testing: None,
        verify_state_transition_results: false,
        ..Default::default()
    };

    let config = PlatformConfig {
        validator_set: ValidatorSetConfig::default_100_67(),
        chain_lock: ChainLockConfig::default_100_67(),
        instant_lock: InstantLockConfig::default_100_67(),
        execution: ExecutionConfig {
            verify_sum_trees: true,
            ..ExecutionConfig::default()
        },
        block_spacing_ms: 3000,
        testing_configs: PlatformTestConfig::default_minimal_verifications(),
        ..Default::default()
    };

    let mut platform = TestPlatformBuilder::new()
        .with_config(config.clone())
        .build_with_mock_rpc();

    run_chain_for_strategy(
        &mut platform, 100, strategy, config,
        15, &mut None, &mut None,
    );
}
}

This test runs 100 empty blocks with 100 masternodes and verifies that the chain progresses without errors. It is the "smoke test" for the strategy framework itself.

Continuing a Chain

Strategy tests support pausing and resuming with continue_chain_for_strategy:

#![allow(unused)]
fn main() {
let outcome = run_chain_for_strategy(
    &mut platform, 50, strategy.clone(), config.clone(),
    13, &mut None, &mut None,
);

// Later...
let continued = continue_chain_for_strategy(
    outcome, strategy, config,
    50, // 50 more blocks
    &mut None, &mut None,
);
}

This is invaluable for testing restart scenarios and verifying that state persists correctly across platform restarts.

How Strategy Tests Differ from Unit Tests

AspectUnit TestsStrategy Tests
ScopeOne state transitionHundreds across many blocks
SetupTestPlatformBuilderrun_chain_for_strategy
RandomnessSeeded per-testSeeded once, flows through blocks
MasternodesNot involvedFully simulated
QuorumsNot involvedRotated and signed
DeterminismYesYes (same seed = same outcome)
SpeedFast (seconds)Slow (minutes for large chains)

Rules

Do:

  • Use strategy tests for multi-block scenarios involving multiple participants.
  • Start with default_minimal_verifications() to speed up test execution.
  • Use small block counts (10-50) during development, increase for CI.
  • Check state_transition_results_per_block to verify specific block outcomes.
  • Use continue_chain_for_strategy for restart/persistence testing.

Don't:

  • Use strategy tests when a unit test would suffice -- they are much slower.
  • Forget to pass a deterministic seed -- non-deterministic strategy tests are useless.
  • Set verify_state_transition_results: true unless you need it; it adds overhead.
  • Create strategy tests with more than a few hundred blocks for regular CI runs.

Test Configuration

Platform tests need to run fast. A production node verifies block signatures, checks instant lock proofs, persists platform state to disk, and creates database checkpoints. All of that is essential for security -- and all of it makes tests slow.

This chapter covers the configuration system that lets tests disable expensive checks selectively, the builder that wires everything together, and the mock RPC layer that eliminates the need for a real Dash Core node.

PlatformTestConfig

The heart of test configuration is PlatformTestConfig, defined in packages/rs-drive-abci/src/config.rs:

#![allow(unused)]
fn main() {
#[cfg(feature = "testing-config")]
pub struct PlatformTestConfig {
    /// Whether to perform block signing.
    pub block_signing: bool,

    /// Whether to store platform state to disk.
    pub store_platform_state: bool,

    /// Whether to verify block commit signatures.
    pub block_commit_signature_verification: bool,

    /// Whether to disable instant lock signature verification.
    pub disable_instant_lock_signature_verification: bool,

    /// Whether to disable checkpoint creation during tests.
    pub disable_checkpoints: bool,
}
}

Notice the #[cfg(feature = "testing-config")] gate. This struct does not exist in production builds. You cannot accidentally ship code that disables signature verification.

Two Default Profiles

PlatformTestConfig provides two defaults, and choosing the right one matters:

Full defaults (Default::default()): Everything enabled. Tests run like a production node, just with a mock RPC backend:

#![allow(unused)]
fn main() {
#[cfg(feature = "testing-config")]
impl Default for PlatformTestConfig {
    fn default() -> Self {
        Self {
            block_signing: true,
            store_platform_state: true,
            block_commit_signature_verification: true,
            disable_instant_lock_signature_verification: false,
            disable_checkpoints: true,
        }
    }
}
}

Minimal verifications (default_minimal_verifications()): Disables everything that is not needed to test application logic:

#![allow(unused)]
fn main() {
impl PlatformTestConfig {
    pub fn default_minimal_verifications() -> Self {
        Self {
            block_signing: false,
            store_platform_state: false,
            block_commit_signature_verification: false,
            disable_instant_lock_signature_verification: true,
            disable_checkpoints: true,
        }
    }
}
}

Use default() when testing consensus-critical behavior (block signing, quorum verification). Use default_minimal_verifications() for everything else -- it is significantly faster because it skips cryptographic operations.

When to Override Individual Fields

Sometimes you need a custom combination. For example, testing withdrawal transitions requires disabling instant lock verification but keeping everything else:

#![allow(unused)]
fn main() {
let platform_config = PlatformConfig {
    testing_configs: PlatformTestConfig {
        disable_instant_lock_signature_verification: true,
        ..Default::default()
    },
    ..Default::default()
};
}

The ..Default::default() spread syntax fills in the remaining fields with their defaults. This pattern lets you express "default with one override" clearly.

TestPlatformBuilder

TestPlatformBuilder is the fluent API for constructing a test platform. It lives in packages/rs-drive-abci/src/test/helpers/setup.rs:

#![allow(unused)]
fn main() {
pub struct TestPlatformBuilder {
    config: Option<PlatformConfig>,
    initial_protocol_version: Option<ProtocolVersion>,
    tempdir: TempDir,
}
}

The Builder Chain

The builder supports three configuration methods:

#![allow(unused)]
fn main() {
impl TestPlatformBuilder {
    /// Create a new builder with a fresh temporary directory.
    pub fn new() -> Self { Self::default() }

    /// Override the platform configuration.
    pub fn with_config(mut self, config: PlatformConfig) -> Self {
        self.config = Some(config);
        self
    }

    /// Pin a specific protocol version.
    pub fn with_initial_protocol_version(
        mut self,
        initial_protocol_version: ProtocolVersion,
    ) -> Self {
        self.initial_protocol_version = Some(initial_protocol_version);
        self
    }

    /// Use the latest protocol version.
    pub fn with_latest_protocol_version(mut self) -> Self {
        self.initial_protocol_version =
            Some(PlatformVersion::latest().protocol_version);
        self
    }
}
}

Building

The builder has two build methods:

build_with_mock_rpc() creates a TempPlatform<MockCoreRPCLike> -- no real Dash Core node needed:

#![allow(unused)]
fn main() {
pub fn build_with_mock_rpc(self) -> TempPlatform<MockCoreRPCLike> {
    let config = self.config.map(|mut c| {
        c.db_path = self.tempdir.path().to_path_buf();
        c
    });

    let platform = Platform::<MockCoreRPCLike>::open(
        self.tempdir.path(),
        config,
        self.initial_protocol_version
            .or(Some(PlatformVersion::latest().protocol_version)),
    )
    .expect("should open Platform successfully");

    TempPlatform {
        platform,
        tempdir: self.tempdir,
    }
}
}

Notice how the builder automatically sets db_path to the temp directory -- you cannot accidentally write to a real database.

build_with_default_rpc() creates a TempPlatform<DefaultCoreRPC> for integration tests that need a real Dash Core connection.

Initializing State

After building, you choose what initial state to install:

#![allow(unused)]
fn main() {
// Minimal: just the GroveDB tree structure
let platform = TestPlatformBuilder::new()
    .build_with_mock_rpc()
    .set_initial_state_structure();

// Full: genesis state with system data contracts
let platform = TestPlatformBuilder::new()
    .with_latest_protocol_version()
    .build_with_mock_rpc()
    .set_genesis_state();

// Genesis with specific activation info
let platform = TestPlatformBuilder::new()
    .build_with_mock_rpc()
    .set_genesis_state_with_activation_info(
        genesis_time,
        start_core_block_height,
    );
}

Most tests want set_genesis_state(). Use set_initial_state_structure() only when testing the state structure itself.

Loading Test Data Contracts

TempPlatform provides convenience methods for loading test contracts:

#![allow(unused)]
fn main() {
let (platform, card_game_contract) = TestPlatformBuilder::new()
    .build_with_mock_rpc()
    .set_initial_state_structure()
    .with_crypto_card_game_transfer_only(Transferable::Always);
}

This loads a predefined "crypto card game" data contract from tests/supporting_files/contract/ and applies it to the platform. The returned DataContract can be used to create documents in subsequent test steps.

Mock RPC: Simulating Dash Core

The MockCoreRPCLike type (from the mockall crate) replaces the real Dash Core RPC client. It lets tests control exactly what Core "reports" -- which transactions are confirmed, what the current block height is, which asset locks exist, etc.

In strategy tests, the mock is configured automatically by run_chain_for_strategy. In unit tests, you typically let the default mock behavior handle things:

#![allow(unused)]
fn main() {
let platform = TestPlatformBuilder::new()
    .with_config(platform_config)
    .build_with_mock_rpc()  // <-- MockCoreRPCLike
    .set_genesis_state();
}

The mock RPC means unit tests require zero external services. They run in CI, on developer laptops, and in sandboxed environments with no network access.

PlatformConfig for Tests vs Production

PlatformConfig is a large struct with many subsections. Here is how tests typically configure it:

#![allow(unused)]
fn main() {
let config = PlatformConfig {
    // Validator set: 100 nodes, 67% threshold
    validator_set: ValidatorSetConfig::default_100_67(),

    // Chain lock quorum config
    chain_lock: ChainLockConfig::default_100_67(),

    // Instant lock quorum config
    instant_lock: InstantLockConfig::default_100_67(),

    // Execution settings
    execution: ExecutionConfig {
        verify_sum_trees: true,
        ..ExecutionConfig::default()
    },

    // Block timing
    block_spacing_ms: 3000,

    // Test-specific overrides
    testing_configs: PlatformTestConfig::default_minimal_verifications(),

    // Fill the rest with defaults
    ..Default::default()
};
}

The default_100_67() methods create configs for a 100-node network with a 67% signing threshold -- the standard test network size.

Platform Restart Testing

TempPlatform supports simulating a platform restart by reopening from the same temporary directory:

#![allow(unused)]
fn main() {
pub fn open_with_tempdir(
    tempdir: TempDir,
    mut config: PlatformConfig,
) -> Self {
    config.db_path = tempdir.path().to_path_buf();
    let platform = Platform::<MockCoreRPCLike>::open(
        tempdir.path(), Some(config), None,
    )
    .expect("should open Platform successfully");

    Self { platform, tempdir }
}
}

The pattern for restart testing:

#![allow(unused)]
fn main() {
// Run first phase
let outcome = run_chain_for_strategy(
    &mut platform, 50, strategy, config.clone(),
    seed, &mut None, &mut None,
);

// Extract tempdir (ownership transfer)
let tempdir = platform.tempdir;

// Reopen -- simulates restart
let platform = TempPlatform::open_with_tempdir(tempdir, config);

// Verify state survived the restart
}

Rules

Do:

  • Use default_minimal_verifications() for tests that do not need signature verification.
  • Use Default::default() for PlatformTestConfig when testing block signing or quorum logic.
  • Always build with build_with_mock_rpc() unless you specifically need Dash Core.
  • Let the builder manage db_path -- never set it manually in test configs.
  • Use set_genesis_state() for most tests; use set_initial_state_structure() only for low-level storage tests.

Don't:

  • Disable verifications in production code -- PlatformTestConfig is #[cfg(feature)] guarded.
  • Create Platform instances directly -- always use TestPlatformBuilder.
  • Share temporary directories between tests -- each test gets its own TempDir.
  • Forget with_latest_protocol_version() -- without it, the builder still defaults to latest, but being explicit prevents surprises during protocol upgrades.
  • Use build_with_default_rpc() in CI -- it requires a running Dash Core node.

Evo SDK Overview

The Evo SDK (@dashevo/evo-sdk) is the primary JavaScript/TypeScript SDK for building applications on Dash Platform. It provides a high-level, strongly-typed facade over the WebAssembly-based Rust SDK, working in both Node.js (≥ 18.18) and modern browsers.

API reference: For detailed per-method documentation with interactive examples, see the Evo SDK Docs.

How it works

┌──────────────────┐
│  Your TypeScript  │
│   Application     │
└────────┬─────────┘
         │  EvoSDK facades (identities, documents, tokens, …)
┌────────▼─────────┐
│   @dashevo/       │
│   evo-sdk         │  TypeScript wrapper layer
└────────┬─────────┘
         │  calls into compiled WASM module
┌────────▼─────────┐
│   @dashevo/       │
│   wasm-sdk        │  Rust SDK compiled to WebAssembly
└────────┬─────────┘
         │  gRPC over HTTPS
┌────────▼─────────┐
│  DAPI nodes       │  Dash Platform's decentralized API
└──────────────────┘

The Evo SDK does not use JSON-RPC or REST. Every request is a gRPC call to one of the Platform's DAPI nodes. Responses include cryptographic proofs that the SDK verifies against the platform state root, so you do not need to trust any single node.

Facades

The SDK organises its API into domain-specific facades, each accessible as a property on the EvoSDK instance:

FacadeDescription
sdk.identitiesFetch, create, update, and top up identities
sdk.contractsFetch, publish, and update data contracts
sdk.documentsQuery, create, replace, delete, and transfer documents
sdk.tokensMint, burn, transfer, freeze tokens and query balances
sdk.dpnsRegister and resolve Dash Platform names
sdk.addressesQuery balances, transfer credits, withdraw to L1
sdk.epochQuery epoch information and evonode proposed blocks
sdk.protocolProtocol version upgrade state and voting
sdk.stateTransitionsBroadcast and wait for state transitions
sdk.systemSystem status, quorum info, and total credits
sdk.groupGroup membership, actions, and contested resources
sdk.votingContested resource vote states and polls

A standalone wallet namespace is also exported for mnemonic generation, key derivation, address validation, and message signing — see the Wallet Utilities chapter.

What it covers

The SDK supports the full set of Platform operations:

  • Queries (read-only): fetch identities, contracts, documents, token balances, DPNS names, epoch info, vote states, and more. Every query can return a cryptographic proof.
  • State transitions (writes): create identities, deploy contracts, manage documents and tokens, register names, cast votes, and transfer credits.

See the API reference for the complete list of operations with interactive examples.

Getting Started

Installation

npm install @dashevo/evo-sdk

The package is ESM-only ("type": "module") and written in TypeScript with full type definitions included. In CommonJS projects use a dynamic import():

const { EvoSDK } = await import('@dashevo/evo-sdk');

Requirements: Node.js ≥ 18.18 or any modern browser with WebAssembly support.

Quick start

import { EvoSDK } from '@dashevo/evo-sdk';

// Pick your network: testnetTrusted() for development, mainnetTrusted() for production
const sdk = EvoSDK.testnetTrusted();

// Query the current epoch (connect() is called automatically on first use)
const epoch = await sdk.epoch.current();
console.log('Current epoch:', epoch.index);

// Fetch an existing identity by its base58 ID
const identity = await sdk.identities.fetch('4EfA9Jrvv3nnCFdSf7fad59851iiTRZ6Wcu6YVJ4iSeF');
console.log('Balance:', identity?.getBalance());

Connecting

Calling connect() explicitly is optional. The SDK connects automatically when you call any facade method for the first time. However, you can call connect() explicitly if you want to control when the WASM module is initialized and quorum keys are prefetched:

const sdk = EvoSDK.testnetTrusted();
await sdk.connect(); // optional — triggers WASM init and quorum prefetch now

Calling connect() more than once is a no-op.

Factory helpers

HelperEquivalent
EvoSDK.testnet()new EvoSDK({ network: 'testnet' })
EvoSDK.mainnet()new EvoSDK({ network: 'mainnet' })
EvoSDK.testnetTrusted()new EvoSDK({ network: 'testnet', trusted: true })
EvoSDK.mainnetTrusted()new EvoSDK({ network: 'mainnet', trusted: true })
EvoSDK.local()new EvoSDK({ network: 'local' })
EvoSDK.localTrusted()new EvoSDK({ network: 'local', trusted: true })

Custom addresses

To connect to specific masternodes (useful for testing or private networks):

const sdk = EvoSDK.withAddresses(
  ['https://52.12.176.90:1443'],
  'testnet',
);
await sdk.connect();

Connection options

All factory helpers and the constructor accept an optional ConnectionOptions object:

const sdk = EvoSDK.testnetTrusted({
  settings: {
    connectTimeoutMs: 5000,
    timeoutMs: 10000,
    retries: 3,
    banFailedAddress: true,
  },
  logs: 'info',         // 'off' | 'error' | 'warn' | 'info' | 'debug' | 'trace'
  proofs: true,          // request proofs with every query
  version: 8,           // pin a specific protocol version
});

Logging

The SDK delegates logging to the underlying Rust/WASM layer. Pass a simple level string or a full EnvFilter directive:

// Simple level
const sdk = EvoSDK.testnetTrusted({ logs: 'debug' });

// Granular filter
const sdk = EvoSDK.testnetTrusted({ logs: 'wasm_sdk=debug,rs_dapi_client=warn' });

// Change level at runtime (static, affects all instances)
await EvoSDK.setLogLevel('trace');

Trusted Mode and Proof Verification

The problem

Every Platform query response includes a cryptographic proof — a signed hash from the current validator quorum that attests the response matches the platform state tree. To verify these proofs the SDK needs the quorum public keys for the active validator set.

On a full node you can look up quorum keys directly from the Core chain. In a browser or lightweight environment you cannot. Trusted mode solves this.

How trusted mode works

When you create an SDK with trusted: true, the connect() call does an extra step before returning: it fetches the current quorum public keys from a well-known HTTPS endpoint and caches them in memory.

const sdk = EvoSDK.testnetTrusted();
await sdk.connect(); // fetches quorum keys, then connects to DAPI

The trust model:

  • You trust the HTTPS endpoint (operated by Dash Core Group) to return correct quorum public keys.
  • Once the keys are cached, every subsequent query response is verified against them — you do not trust individual DAPI nodes for the data itself.

This is a pragmatic trade-off: you trust one endpoint for the validator set, but verify all actual data cryptographically.

When to use trusted mode

ScenarioTrusted mode?Why
Browser appYesNo access to Core chain
Node.js scriptYesSimplest setup
Server with Core RPCOptionalCan fetch quorum keys from your own node
Local Docker setupYesUse EvoSDK.localTrusted()

If you do not use trusted mode, the SDK still works but cannot verify proofs. Queries return data but you are trusting the responding DAPI node.

Proofs in responses

By default, the SDK verifies proofs internally and returns just the data. To inspect the proof metadata yourself, use the WithProof variants:

// Standard — proof verified internally, returns data only
const identity = await sdk.identities.fetch(id);

// With proof — returns both data and proof metadata
const { data, proof, metadata } = await sdk.identities.fetchWithProof(id);
console.log('Block height:', metadata.height);
console.log('Core chain locked height:', metadata.coreChainLockedHeight);

Some methods also offer an Unproved variant that skips proof verification entirely, useful when you trust the node or want faster responses:

const identity = await sdk.identities.fetchUnproved(id);

State Transitions

State transitions are the write operations of Dash Platform. Unlike queries (which are free and instant), state transitions modify on-chain state, cost credits, and must be signed with a private key.

API reference: For the full list of state transition methods with parameters and examples, see the State Transitions section of the Evo SDK Docs.

How state transitions work

  1. Build — The SDK constructs the transition (e.g., "create identity", "register name") from the parameters you provide.
  2. Sign — You provide a private key (WIF format) and the SDK signs the transition.
  3. Broadcast — The signed transition is sent to a DAPI node, which propagates it to the Platform chain.
  4. Wait — The SDK waits for the transition to be included in a block and returns the result.

Identity operations

Create an identity

// Generate keys for the new identity
const keyPair = await wallet.generateKeyPair('testnet');

const identity = await sdk.identities.create({
  privateKeyWif: fundingKeyWif,     // key with Dash balance for the asset lock
  identityPublicKeys: [{
    type: 0,                         // ECDSA_SECP256K1
    purpose: 0,                      // AUTHENTICATION
    securityLevel: 0,                // MASTER
    publicKeyHex: keyPair.publicKeyHex,
  }],
});

Top up an identity

await sdk.identities.topUp({
  identityId: 'BxPVr5...',
  amount: 100000,                   // credits (1 credit = 1000 duffs)
  privateKeyWif: fundingKeyWif,
});

Transfer credits between identities

await sdk.identities.creditTransfer({
  identityId: senderIdentityId,
  recipientId: recipientIdentityId,
  amount: 50000,
  privateKeyWif: senderAuthKeyWif,
  signingKeyIndex: 0,
  nonce: await sdk.identities.nonce(senderIdentityId),
});

Register a key with a budget or an expiry

A key added with totalBudget or expiresAt is registered with those limits (protocol version 14): an application key that can spend at most so many credits, or that stops signing at a block time. Only AUTHENTICATION keys below MASTER may carry them. identityUpdate assigns the key id; the signer holds the identity's MASTER key and the new key's private key, since a new key signs its own registration.

const appKey = new IdentityPublicKeyInCreation({
  keyId: 0,                      // reassigned to the next free id
  purpose: 'AUTHENTICATION',
  securityLevel: 'CRITICAL',
  keyType: 'ECDSA_SECP256K1',
  data: appKeyPublicKeyBytes,
  totalBudget: 500000000n,       // credits this key may take from the identity over its lifetime
  expiresAt: 1800000000000n,     // optional: block time in milliseconds from which it stops signing
});

await sdk.identities.update({ identity, addPublicKeys: [appKey], signer });

Raise a key's limits

A key registered with a budget or an expiry can be topped up, or have its expiry moved later, without being replaced. The signer holds a MASTER key, or a CRITICAL authentication key without limits and without contract bounds; the budget is added to the total the passed identity's key shows.

const key = await sdk.identities.updateKeyLimits({
  identity,
  keyId: 5,
  addBudget: 100000000n,        // credits added to the budget and to what is left of it
  expiresAt: 1800000000000n,    // optional: a later expiry in milliseconds
  signer,
});

Document operations

Create a document

await sdk.documents.create({
  contractId: 'GWRSAVFMjXx8HpQFaNJMqBV7MBgMK4br5UESsB4S31Ec',
  documentType: 'domain',
  document: {
    label: 'my-username',
    normalizedLabel: 'my-username',
    normalizedParentDomainName: 'dash',
    records: { identity: identityId },
    subdomainRules: { allowSubdomains: false },
  },
  identityId,
  privateKeyWif: authKeyWif,
  signingKeyIndex: 0,
  nonce: await sdk.identities.contractNonce(identityId, dpnsContractId),
});

Replace, delete, transfer

The sdk.documents facade also provides replace(), delete(), transfer(), purchase(), and setPrice() methods. See the API reference for parameters.

Token operations

// Mint tokens (requires minting authority)
await sdk.tokens.mint({
  tokenId: '...',
  amount: 1000,
  recipientId: '...',
  identityId: minterIdentityId,
  privateKeyWif: minterKeyWif,
  signingKeyIndex: 0,
  nonce: await sdk.identities.nonce(minterIdentityId),
});

// Transfer tokens
await sdk.tokens.transfer({
  tokenId: '...',
  amount: 100,
  recipientId: '...',
  identityId: senderIdentityId,
  privateKeyWif: senderKeyWif,
  signingKeyIndex: 0,
  nonce: await sdk.identities.nonce(senderIdentityId),
});

DPNS name registration

await sdk.dpns.register({
  name: 'alice',
  identityId,
  privateKeyWif: authKeyWif,
  signingKeyIndex: 0,
  nonce: await sdk.identities.contractNonce(identityId, dpnsContractId),
});

Waiting for results

The sdk.stateTransitions facade provides low-level control:

// Broadcast a raw state transition and wait for confirmation
const result = await sdk.stateTransitions.waitForResult(stateTransitionHash);

Nonces

Every state transition requires a nonce to prevent replay attacks. There are two types:

  • Identity nonce — incremented per identity for identity-level transitions (top-ups, credit transfers, token operations)
  • Contract nonce — incremented per identity-contract pair for document and contract transitions
const identityNonce = await sdk.identities.nonce(identityId);
const contractNonce = await sdk.identities.contractNonce(identityId, contractId);

Always fetch the nonce immediately before broadcasting. If another transition lands between your fetch and broadcast, the nonce will be stale and the transition will be rejected.

Wallet Utilities

The Evo SDK exports a standalone wallet namespace with offline cryptographic utilities. These functions do not require a connected SDK instance — they initialise the WASM module on first call and work independently.

import { wallet } from '@dashevo/evo-sdk';

Mnemonic management

// Generate a new 12-word mnemonic
const mnemonic = await wallet.generateMnemonic();
// "abandon ability able about above absent ..."

// Validate an existing mnemonic
const valid = await wallet.validateMnemonic(mnemonic);

// Convert to seed bytes (with optional passphrase)
const seed = await wallet.mnemonicToSeed(mnemonic, 'optional-passphrase');

Key derivation

From seed phrase

const keyInfo = await wallet.deriveKeyFromSeedPhrase({
  mnemonic,
  network: 'testnet',
  derivationPath: "m/44'/1'/0'/0/0",
});
// keyInfo.privateKeyWif, keyInfo.publicKeyHex, keyInfo.address

From seed with path

const seed = await wallet.mnemonicToSeed(mnemonic);
const key = await wallet.deriveKeyFromSeedWithPath({
  seed,
  network: 'testnet',
  path: "m/44'/1'/0'/0/0",
});

Standard derivation paths

The SDK provides helpers for Dash-specific derivation paths:

// BIP-44 paths
const bip44 = await wallet.derivationPathBip44Testnet(0, 0, 0);
// "m/44'/1'/0'/0/0"

// DIP-9 Platform paths (identity authentication keys)
const dip9 = await wallet.derivationPathDip9Testnet(0, 0, 0);

// DIP-13 DashPay paths (contact encryption keys)
const dip13 = await wallet.derivationPathDip13Testnet(0);

Extended public key operations

// Convert xprv to xpub
const xpub = await wallet.xprvToXpub(xprv);

// Derive child public key
const childPub = await wallet.deriveChildPublicKey(xpub, 0, false);

Key pair generation

// Generate a random key pair
const keyPair = await wallet.generateKeyPair('testnet');
// keyPair.privateKeyWif, keyPair.publicKeyHex, keyPair.address

// Generate multiple key pairs
const pairs = await wallet.generateKeyPairs('testnet', 5);

// Import from WIF
const imported = await wallet.keyPairFromWif('cPrivateKeyWif...');

// Import from hex
const fromHex = await wallet.keyPairFromHex('abcd1234...', 'testnet');

Address utilities

// Derive address from public key
const address = await wallet.pubkeyToAddress(pubkeyHex, 'testnet');

// Validate an address for a network
const ok = await wallet.validateAddress('yWhatever...', 'testnet');

Message signing

const signature = await wallet.signMessage(
  'Hello Dash Platform',
  privateKeyWif,
);

DashPay contact keys

For DashPay encrypted messaging, derive contact-specific keys:

const contactKey = await wallet.deriveDashpayContactKey({
  mnemonic,
  network: 'testnet',
  senderIdentityId: '...',
  receiverIdentityId: '...',
  account: 0,
  addressIndex: 0,
});

The returned xpub is the contact payment public key that can be encrypted into a DashPay contactRequest.encryptedPublicKey field. See DashPay Contact Requests for the end-to-end document payload shape.

DashPay Contact Requests

DashPay contact requests are contactRequest documents in the DashPay data contract. The document links the sender identity to toUserId and carries the sender's DIP-15 contact payment public key encrypted for the recipient.

The Evo SDK exposes the DIP-15 derivation primitive through wallet.deriveDashpayContactKey. Applications still need to encrypt the derived contact xpub before submitting the document.

Document fields

The contactRequest document requires these fields:

type DashpayContactRequestDocument = {
  $createdAt: number;
  $createdAtCoreBlockHeight: number;
  toUserId: Uint8Array;
  encryptedPublicKey: Uint8Array;
  senderKeyIndex: number;
  recipientKeyIndex: number;
  accountReference: number;
};

The current DashPay contract schema requires the system field $createdAtCoreBlockHeight. Older external references may use coreHeightCreatedAt; do not submit that name to the current contract.

encryptedPublicKey is exactly 96 bytes:

  • 16 bytes: AES-CBC initialization vector
  • 80 bytes: AES-CBC ciphertext for the sender's 69-byte compact contact xpub

The compact xpub is the parent fingerprint (4 bytes), chain code (32 bytes) and public key (33 bytes), without the version, depth and child number of a full 78-byte BIP32 serialization. Receivers, including the reference mobile wallets and the Rust, Swift and Kotlin SDKs, refuse any other plaintext length. Both 69 and 78 bytes pad to the same 80-byte ciphertext, so the contract cannot catch the mistake: a request that encrypts the full 78 bytes is stored on chain, and the recipient's wallet then drops it.

The sender derives the contact xpub from the sender identity, recipient identity, account, and address index. The sender then encrypts that xpub with an ECDH shared secret from the sender's identity encryption private key and the recipient's identity decryption public key.

Derive the contact payment xpub

import { wallet } from '@dashevo/evo-sdk';

const senderKeyIndex = 0;
const recipientKeyIndex = 0;
const addressIndex = 0;

const contactKey = await wallet.deriveDashpayContactKey({
  mnemonic: senderMnemonic,
  network: 'testnet',
  senderIdentityId,
  receiverIdentityId: recipientIdentityId,
  account: 0,
  addressIndex,
});

// contactKey.xpub is encrypted into contactRequest.encryptedPublicKey.

senderKeyIndex and recipientKeyIndex identify the identity public keys used for ECDH. addressIndex is the DIP-15 child index used for contact payment key derivation and is independent from those identity key indexes.

Build the encrypted public key

The following helper shows the byte-level encryption shape. It uses the same secp256k1 primitives exposed by @dashevo/dashcore-lib and Node.js crypto for AES-CBC.

import crypto from 'node:crypto';
import dashcore from '@dashevo/dashcore-lib';

function fixed32(value): Buffer {
  return Buffer.from(value.toArray('be', 32));
}

function deriveSharedKey({
  privateKeyWif,
  publicKeyBytes,
}: {
  privateKeyWif: string;
  publicKeyBytes: Uint8Array;
}): Buffer {
  const privateKey = dashcore.PrivateKey.fromWIF(privateKeyWif);
  const publicKey = dashcore.PublicKey.fromBuffer(Buffer.from(publicKeyBytes));
  const sharedPoint = publicKey.point.mul(privateKey.toBigNumber());

  if (sharedPoint.isInfinity()) {
    throw new Error('ECDH shared point is invalid');
  }

  const x = fixed32(sharedPoint.getX());
  const y = fixed32(sharedPoint.getY());
  const compressedPrefix = Buffer.from([2 | (y[31] & 1)]);

  return crypto.createHash('sha256').update(Buffer.concat([compressedPrefix, x])).digest();
}

function compactXpubPayload(xpub: string): Buffer {
  const payload = dashcore.encoding.Base58Check.decode(xpub);

  if (payload.length !== 78) {
    throw new Error(`Invalid DashPay contact xpub length: ${payload.length}`);
  }

  // BIP32 layout: version(4) depth(1) parentFingerprint(4) childNumber(4)
  // chainCode(32) publicKey(33). DIP-15 encrypts only the parent
  // fingerprint, chain code and public key: 69 bytes.
  return Buffer.concat([payload.subarray(5, 9), payload.subarray(13, 78)]);
}

function encryptContactXpub({
  contactXpub,
  senderEncryptionPrivateKeyWif,
  recipientDecryptionPublicKeyBytes,
}: {
  contactXpub: string;
  senderEncryptionPrivateKeyWif: string;
  recipientDecryptionPublicKeyBytes: Uint8Array;
}): Uint8Array {
  const aesKey = deriveSharedKey({
    privateKeyWif: senderEncryptionPrivateKeyWif,
    publicKeyBytes: recipientDecryptionPublicKeyBytes,
  });
  const payload = compactXpubPayload(contactXpub);
  const iv = crypto.randomBytes(16);
  const cipher = crypto.createCipheriv('aes-256-cbc', aesKey, iv);
  const encrypted = Buffer.concat([
    cipher.update(payload),
    cipher.final(),
  ]);

  const encryptedPublicKey = Buffer.concat([iv, encrypted]);

  if (encryptedPublicKey.length !== 96) {
    throw new Error(`DashPay encryptedPublicKey must be 96 bytes, got ${encryptedPublicKey.length}`);
  }

  return encryptedPublicKey;
}

Submit the document

const encryptedPublicKey = encryptContactXpub({
  contactXpub: contactKey.xpub,
  senderEncryptionPrivateKeyWif,
  recipientDecryptionPublicKeyBytes,
});

const accountReference = 0;

const document = {
  $createdAt: Date.now(),
  $createdAtCoreBlockHeight: platformCoreHeight,
  toUserId: recipientIdentityIdBytes,
  encryptedPublicKey,
  senderKeyIndex,
  recipientKeyIndex,
  accountReference,
};

accountReference above is a placeholder for the current Platform field accepted by the contactRequest schema. It is not a complete implementation of any ASK/HMAC-based account-reference obfuscation described in older DIP text.

When querying received requests through the JavaScript SDK, pass identity IDs in the representation expected by the SDK call being used. The contract stores toUserId as a 32-byte identifier, while some high-level JavaScript query helpers accept the base58 identity string and perform the conversion internally.

Current SDK boundary

wallet.deriveDashpayContactKey handles DIP-15 path derivation. It does not currently submit DashPay documents or encrypt/decrypt encryptedPublicKey. Applications need to combine the wallet helper with identity encryption keys until a higher-level DashPay contact request helper is added to the JavaScript SDK.

Treat the example as a byte-level reference. A production application should add contract validation, decrypt round-trip tests, and checks that the selected identity keys are active secp256k1 keys bounded for DashPay contact requests.

Networks and Environments

The Evo SDK supports four built-in network configurations plus custom addresses for private or development networks.

Built-in networks

NetworkFactoryDAPI discoveryUse case
TestnetEvoSDK.testnetTrusted()Automatic via seed nodesDevelopment and testing
MainnetEvoSDK.mainnetTrusted()Automatic via seed nodesProduction applications
DevnetEvoSDK.devnetTrusted(name)Automatic via quorums serverLong-lived shared devnets (e.g. 'paloma')
LocalEvoSDK.localTrusted()127.0.0.1:1443Docker-based local development

For each network, the SDK discovers DAPI endpoints from seed nodes and rotates between them automatically. Failed nodes are temporarily banned so the SDK retries against healthy nodes.

Devnets

Devnets are long-lived shared development networks identified by a short name (e.g. 'paloma'). The trusted context derives the quorum base URL from the name as https://quorums.<name>.networks.dash.org:

const sdk = EvoSDK.devnetTrusted('paloma');
await sdk.connect();

If the public quorums DNS for a devnet isn't deployed yet, override the URL:

const sdk = EvoSDK.devnetTrusted('paloma', {
  quorumUrl: 'https://quorums.staging.example/',
});
await sdk.connect();

For a devnet without any trusted context (no proof verification), supply explicit DAPI addresses:

const sdk = EvoSDK.devnet('paloma', {
  addresses: ['https://10.0.0.5:1443'],
});
await sdk.connect();

Behind the scenes these factories call WasmTrustedContext.prefetchDevnet(name) or prefetchDevnetWithUrl(url); the same shape is available on prefetchMainnetWithUrl / prefetchTestnetWithUrl for staging endpoints (production networks must use https://).

The prefetch issues the current-quorum and previous-quorum requests concurrently. It also asks the quorum service for masternode addresses, unless the SDK was given explicit addresses: those take precedence in the builder, so connect() passes discoverAddresses: false and skips that request.

Local development with Docker

When running a local Platform network via dashmate, use the local network:

const sdk = EvoSDK.localTrusted();
await sdk.connect();

This connects to https://127.0.0.1:1443 by default. If your local setup uses different ports, use custom addresses:

const sdk = EvoSDK.withAddresses(
  ['https://127.0.0.1:2443'],
  'local',
);
await sdk.connect();

Custom masternode addresses

For private devnets, specific nodes, or debugging:

const sdk = EvoSDK.withAddresses(
  [
    'https://52.12.176.90:1443',
    'https://34.217.100.50:1443',
  ],
  'testnet',
);
await sdk.connect();

When custom addresses are provided, the SDK does not perform automatic node discovery — it uses only the addresses you supply.

Browser vs Node.js

The SDK works identically in both environments. The underlying WASM module handles platform differences transparently.

Node.js considerations:

  • Requires Node.js ≥ 18.18 (for WebAssembly and fetch support)
  • ESM-only package — use import, not require
  • No additional polyfills needed

Browser considerations:

  • Works in any browser with WebAssembly support (all modern browsers)
  • The WASM module is loaded asynchronously on first connect() call
  • Total bundle size includes the compiled Rust SDK (~2-4 MB gzipped)
  • gRPC calls use grpc-web over HTTPS, compatible with standard CORS

Tutorial: Car Sales Management

Build a decentralised car listing and sales application on Dash Platform. By the end you will have a data contract for vehicle listings, the ability to create/query/update listings, and a purchase flow using document transfers.

How this works in practice: Data contracts are deployed once using a Node.js script with a developer identity. After deployment, your browser app uses the published contract ID to create, query, and update documents. Steps 1-2 below are run from Node.js; steps 3 onward can run in either Node.js or the browser.

What you will learn

  • Designing a data contract with multiple document types
  • Publishing a contract to testnet from a Node.js deployment script
  • Creating, querying, and updating documents (Node.js or browser)
  • Using document pricing and purchase for a sales flow

Prerequisites

npm install @dashevo/evo-sdk

You need a funded testnet identity. See the Getting Started chapter for setup.

Step 1: Design the data contract

A car sales contract needs two document types: listings (vehicles for sale) and reviews (buyer reviews of sellers).

const carSalesSchema = {
  listing: {
    type: 'object',
    properties: {
      make:        { type: 'string', maxLength: 63, position: 0 },
      model:       { type: 'string', maxLength: 63, position: 1 },
      year:        { type: 'integer', minimum: 1900, maximum: 2100, position: 2 },
      mileageKm:   { type: 'integer', minimum: 0, position: 3 },
      priceUsd:    { type: 'integer', minimum: 0, position: 4 },
      description: { type: 'string', maxLength: 1024, position: 5 },
      imageUrl:    { type: 'string', maxLength: 512, format: 'uri', position: 6 },
      status:      { type: 'string', enum: ['available', 'pending', 'sold'], position: 7 },
    },
    required: ['make', 'model', 'year', 'priceUsd', 'status'],
    additionalProperties: false,
  },
  review: {
    type: 'object',
    properties: {
      sellerId:  { type: 'string', maxLength: 44, position: 0 },
      listingId: { type: 'string', maxLength: 44, position: 1 },
      rating:    { type: 'integer', minimum: 1, maximum: 5, position: 2 },
      comment:   { type: 'string', maxLength: 512, position: 3 },
    },
    required: ['sellerId', 'rating'],
    additionalProperties: false,
  },
};

Step 2: Connect and publish the contract

import { EvoSDK, DataContract, Document, Identifier, IdentitySigner } from '@dashevo/evo-sdk';

const sdk = EvoSDK.testnetTrusted();
await sdk.connect();

// Your identity credentials
const identityId = 'YOUR_IDENTITY_ID';
const privateKeyWif = 'YOUR_PRIVATE_KEY_WIF';
const signingKeyIndex = 0;

// Set up signing
const identity = await sdk.identities.fetch(identityId);
const identityKey = identity.publicKeys[signingKeyIndex];
const signer = new IdentitySigner();
signer.addKeyFromWif(privateKeyWif);

// Publish the data contract
const nonce = await sdk.identities.nonce(identityId);
const dataContract = new DataContract({
  ownerId: new Identifier(identityId),
  identityNonce: nonce + 1n,
  schemas: carSalesSchema,
});
const contract = await sdk.contracts.publish({ dataContract, identityKey, signer });

const contractId = contract.id.toString();
console.log('Contract published:', contractId);

Save the contractId — you will need it for all subsequent operations.

Step 3: Create a listing

const doc = new Document({
  documentTypeName: 'listing',
  dataContractId: new Identifier(contractId),
  ownerId: new Identifier(identityId),
  properties: {
    make: 'Toyota',
    model: 'Camry',
    year: 2021,
    mileageKm: 45000,
    priceUsd: 22500,
    description: 'Well-maintained, single owner, full service history.',
    status: 'available',
  },
});
await sdk.documents.create({ document: doc, identityKey, signer });

console.log('Listing created!');

Step 4: Query listings

// Fetch all available listings
const results = await sdk.documents.query({
  dataContractId: contractId,
  documentTypeName: 'listing',
  where: [['status', '==', 'available']],
  orderBy: [['priceUsd', 'asc']],
  limit: 20,
});

for (const [id, doc] of results) {
  if (!doc) continue;
  const data = doc.properties as Record<string, unknown>;
  console.log(`${data.year} ${data.make} ${data.model} — $${data.priceUsd}`);
  console.log(`  ID: ${id}`);
}

Search by make

const toyotas = await sdk.documents.query({
  dataContractId: contractId,
  documentTypeName: 'listing',
  where: [
    ['make', '==', 'Toyota'],
    ['status', '==', 'available'],
  ],
  limit: 10,
});

Step 5: Update a listing

Mark a listing as sold:

const listingId = 'THE_LISTING_DOCUMENT_ID';

// Fetch the existing document, modify it, and bump the revision
const existing = await sdk.documents.get(contractId, 'listing', listingId);
existing.properties = { ...existing.properties, status: 'sold' };
existing.revision = (existing.revision ?? 0n) + 1n;
await sdk.documents.replace({ document: existing, identityKey, signer });

console.log('Listing marked as sold');

Step 6: Leave a review

// Set up buyer signing
const buyerIdentity = await sdk.identities.fetch(buyerIdentityId);
const buyerKey = buyerIdentity.publicKeys[0];
const buyerSigner = new IdentitySigner();
buyerSigner.addKeyFromWif(buyerKeyWif);

const reviewDoc = new Document({
  documentTypeName: 'review',
  dataContractId: new Identifier(contractId),
  ownerId: new Identifier(buyerIdentityId),
  properties: {
    sellerId: 'SELLER_IDENTITY_ID',
    listingId: 'THE_LISTING_DOCUMENT_ID',
    rating: 5,
    comment: 'Great seller, car was exactly as described!',
  },
});
await sdk.documents.create({ document: reviewDoc, identityKey: buyerKey, signer: buyerSigner });

Query reviews for a seller

const reviews = await sdk.documents.query({
  dataContractId: contractId,
  documentTypeName: 'review',
  where: [['sellerId', '==', 'SELLER_IDENTITY_ID']],
  orderBy: [['rating', 'desc']],
  limit: 50,
});

let totalRating = 0;
let count = 0;
for (const [, doc] of reviews) {
  if (!doc) continue;
  const props = doc.properties as Record<string, unknown>;
  totalRating += props.rating as number;
  count++;
}
console.log(`Average rating: ${(totalRating / count).toFixed(1)} (${count} reviews)`);

Next steps

  • Add indexes to the contract schema for efficient queries on make, year, and priceUsd
  • Add a location field and query by region
  • Use document pricing (sdk.documents.setPrice / sdk.documents.purchase) to let buyers pay for premium listing details
  • Integrate with a frontend framework (React, Vue, etc.) for a full web app

Tutorial: Creating a Basic Token

Environment: This tutorial uses Node.js scripts to deploy a contract and perform token operations. For browser-based applications, deploy the contract from a Node.js script first, then use the contract ID in your frontend code.

Create a fungible token on Dash Platform with minting, transferring, and balance queries. This tutorial walks through the full lifecycle from contract deployment to token operations.

What you will learn

  • Defining a data contract with a token configuration
  • Minting tokens to an identity
  • Transferring tokens between identities
  • Querying balances and supply

Prerequisites

npm install @dashevo/evo-sdk

You need a funded testnet identity with enough credits to deploy a contract and perform token operations.

Step 1: Define the token contract

A token is defined as part of a data contract. The contract schema includes a tokens section alongside the usual document schemas.

import {
  EvoSDK, DataContract, Identifier, IdentitySigner,
  TokenConfigurationConvention, TokenConfigurationLocalization, TokenConfiguration,
  ChangeControlRules, AuthorizedActionTakers, TokenDistributionRules,
  TokenKeepsHistoryRules, TokenMarketplaceRules, TokenTradeMode,
} from '@dashevo/evo-sdk';

const sdk = EvoSDK.testnetTrusted();
await sdk.connect();

const identityId = 'YOUR_IDENTITY_ID';
const privateKeyWif = 'YOUR_PRIVATE_KEY_WIF';
const signingKeyIndex = 0;

// Define a contract with a token
const contractSchema = {
  // Document types (optional — a token-only contract can have none)
  tokenMetadata: {
    type: 'object',
    properties: {
      tokenName: { type: 'string', maxLength: 63, position: 0 },
      description: { type: 'string', maxLength: 256, position: 1 },
    },
    additionalProperties: false,
  },
};

// Build the token configuration using SDK classes
const localization = new TokenConfigurationLocalization(true, 'CoffeeCoin', 'CoffeeCoins');
const conventions = new TokenConfigurationConvention({ en: localization }, 2);

const ownerOnly = new ChangeControlRules({
  authorizedToMakeChange: AuthorizedActionTakers.ContractOwner(),
  adminActionTakers: AuthorizedActionTakers.ContractOwner(),
});
const noOne = new ChangeControlRules({
  authorizedToMakeChange: AuthorizedActionTakers.NoOne(),
  adminActionTakers: AuthorizedActionTakers.NoOne(),
});

const tokenConfig = new TokenConfiguration({
  conventions,
  conventionsChangeRules: noOne,
  baseSupply: 0n,
  maxSupply: 1_000_000_00n,  // 1,000,000.00 with 2 decimals
  maxSupplyChangeRules: noOne,
  keepsHistory: new TokenKeepsHistoryRules({
    isKeepingMintingHistory: true,
    isKeepingBurningHistory: true,
    isKeepingTransferHistory: true,
  }),
  distributionRules: new TokenDistributionRules({
    perpetualDistributionRules: noOne,
    newTokensDestinationIdentityRules: noOne,
    mintingAllowChoosingDestination: true,
    mintingAllowChoosingDestinationRules: noOne,
    changeDirectPurchasePricingRules: noOne,
  }),
  marketplaceRules: new TokenMarketplaceRules(TokenTradeMode.NotTradeable(), noOne),
  manualMintingRules: ownerOnly,
  manualBurningRules: ownerOnly,
  freezeRules: noOne,
  unfreezeRules: noOne,
  destroyFrozenFundsRules: noOne,
  emergencyActionRules: noOne,
  mainControlGroupCanBeModified: AuthorizedActionTakers.NoOne(),
});

Optional: let every identity claim a fixed amount once

A once-per-identity distribution turns the token into an open airdrop: any identity can claim the configured amount exactly once, and the total paid out is bounded only by maxSupply. The distribution is fixed at registration; there is no change control rule for it, and no configuration update changes it. Setting it makes the distribution rules serialize as format version 1, which needs protocol version 14, and adds 0.1 Dash to the contract's registration fee, like a perpetual or pre-programmed distribution does.

import { TokenOncePerIdentityDistribution } from '@dashevo/evo-sdk';

distributionRules: new TokenDistributionRules({
  perpetualDistributionRules: noOne,
  newTokensDestinationIdentityRules: noOne,
  mintingAllowChoosingDestination: true,
  mintingAllowChoosingDestinationRules: noOne,
  changeDirectPurchasePricingRules: noOne,
  oncePerIdentityDistribution: new TokenOncePerIdentityDistribution(100_00n), // 100.00 tokens
}),

Any identity then claims with a token claim of distribution type oncePerIdentity:

await sdk.tokens.claim({
  dataContractId: contract.id,
  tokenPosition: 0,
  identityId: claimantId,
  distributionType: 'oncePerIdentity',
  identityKey,
  signer,
});

A second claim by the same identity is rejected with TokenOncePerIdentityDistributionAlreadyClaimedError (code 40722), and a claim that would push the supply past maxSupply is rejected with TokenMintPastMaxSupplyError.

Step 2: Publish the contract

// Set up signing
const identity = await sdk.identities.fetch(identityId);
const identityKey = identity.publicKeys[signingKeyIndex];
const signer = new IdentitySigner();
signer.addKeyFromWif(privateKeyWif);

const nonce = await sdk.identities.nonce(identityId);
const dataContract = new DataContract({
  ownerId: new Identifier(identityId),
  identityNonce: nonce + 1n,
  schemas: contractSchema,
  tokens: { 0: tokenConfig },
});
const contract = await sdk.contracts.publish({ dataContract, identityKey, signer });

const contractId = contract.id.toString();
console.log('Contract published:', contractId);

Step 3: Mint tokens

The contract owner can mint tokens to any identity:

// Token operations require a CRITICAL security level key.
// Fetch a key with the appropriate security level from the identity.
const criticalKey = identity.publicKeys[signingKeyIndex];
const criticalSigner = new IdentitySigner();
criticalSigner.addKeyFromWif(privateKeyWif);

// Mint 10,000.00 CoffeeCoins to yourself
await sdk.tokens.mint({
  dataContractId: new Identifier(contractId),
  tokenPosition: 0,
  amount: 10_000_00n,          // 10,000.00 (2 decimal places) — must be bigint
  recipientId: new Identifier(identityId),
  identityId: new Identifier(identityId),
  identityKey: criticalKey,
  signer: criticalSigner,
});

console.log('Minted 10,000 CoffeeCoins');

Mint to another identity

await sdk.tokens.mint({
  dataContractId: new Identifier(contractId),
  tokenPosition: 0,
  amount: 500_00n,             // 500.00 CoffeeCoins
  recipientId: new Identifier('RECIPIENT_IDENTITY_ID'),
  identityId: new Identifier(identityId),
  identityKey: criticalKey,
  signer: criticalSigner,
});

Step 4: Check balances

// Check your own balance
const myBalances = await sdk.tokens.identityBalances(identityId, [contractId]);
let myBalance = 0n;
for (const [id, balance] of myBalances) {
  if (id.toString() === contractId) myBalance = balance;
}
console.log('My balance:', Number(myBalance) / 100, 'CoffeeCoins');

// Check multiple identities at once
const balances = await sdk.tokens.balances(
  [identityId, 'OTHER_IDENTITY_ID'],
  contractId,
);

for (const [id, balance] of balances) {
  console.log(`${id}: ${Number(balance) / 100} CoffeeCoins`);
}

Check total supply

const tokenId = await sdk.tokens.calculateId(contractId, 0);
const supply = await sdk.tokens.totalSupply(tokenId);
if (supply) {
  console.log('Total supply:', Number(supply.totalSupply) / 100, 'CoffeeCoins');
}

Step 5: Transfer tokens

await sdk.tokens.transfer({
  dataContractId: new Identifier(contractId),
  tokenPosition: 0,
  amount: 25_00n,              // 25.00 CoffeeCoins
  recipientId: new Identifier('RECIPIENT_IDENTITY_ID'),
  senderId: new Identifier(identityId),
  identityKey: criticalKey,
  signer: criticalSigner,
});

console.log('Transferred 25 CoffeeCoins');

Step 6: Burn tokens

Reduce the supply by burning tokens you own:

await sdk.tokens.burn({
  dataContractId: new Identifier(contractId),
  tokenPosition: 0,
  amount: 100_00n,             // 100.00 CoffeeCoins
  identityId: new Identifier(identityId),
  identityKey: criticalKey,
  signer: criticalSigner,
});

console.log('Burned 100 CoffeeCoins');

Full example

Putting it all together as a complete script:

import { EvoSDK, Identifier, IdentitySigner } from '@dashevo/evo-sdk';

async function main() {
  const sdk = EvoSDK.testnetTrusted();
  await sdk.connect();

  const identityId = 'YOUR_IDENTITY_ID';
  const privateKeyWif = 'YOUR_PRIVATE_KEY_WIF';
  const contractId = 'YOUR_CONTRACT_ID';  // from step 2

  // Set up signing (token ops require a CRITICAL security level key)
  const identity = await sdk.identities.fetch(identityId);
  const identityKey = identity.publicKeys[0];
  const signer = new IdentitySigner();
  signer.addKeyFromWif(privateKeyWif);

  // Check balance
  const balances = await sdk.tokens.identityBalances(identityId, [contractId]);
  for (const [id, balance] of balances) {
    if (id.toString() === contractId) console.log('Balance:', balance);
  }

  // Transfer
  await sdk.tokens.transfer({
    dataContractId: new Identifier(contractId),
    tokenPosition: 0,
    amount: 10_00n,
    recipientId: new Identifier('FRIEND_IDENTITY_ID'),
    senderId: new Identifier(identityId),
    identityKey,
    signer,
  });

  console.log('Transfer complete!');
}

main().catch(console.error);

Next steps

  • Add freeze/unfreeze capabilities for compliance scenarios
  • Set up a direct purchase price so anyone can buy tokens with credits
  • Create a distribution schedule for automatic token rewards
  • Use the tokenMetadata document type to store on-chain metadata

Tutorial: Card Game with Tokens

Build a collectible card game on Dash Platform where cards are documents that can be traded, and an in-game currency token is used for purchases. This tutorial combines data contracts, documents, and tokens into a cohesive application.

Environment: Steps 1-2 (contract design and deployment) are run from a Node.js script using a developer/operator identity. Steps 3 onward (minting, trading, querying) can run in either Node.js or a browser app using the published contract ID.

What you will learn

  • Designing a contract with both document types and tokens
  • Using documents as game items (cards) owned by identities
  • Token-based in-game economy (minting rewards, spending on packs)
  • Document transfers for card trading between players
  • Querying collections and leaderboards

Prerequisites

npm install @dashevo/evo-sdk

You need a funded testnet identity. This tutorial uses two identities to demonstrate trading.

Step 1: Design the game contract

The contract defines three document types and one token:

  • card — A collectible card with rarity, power, and element
  • deck — A player's active deck configuration
  • match — Match result history
  • GemToken — In-game currency for buying card packs
import {
  TokenConfigurationConvention, TokenConfigurationLocalization, TokenConfiguration,
  ChangeControlRules, AuthorizedActionTakers, TokenDistributionRules,
  TokenKeepsHistoryRules, TokenMarketplaceRules, TokenTradeMode,
} from '@dashevo/evo-sdk';

const gameSchema = {
  card: {
    type: 'object',
    properties: {
      name:    { type: 'string', maxLength: 63, position: 0 },
      element: { type: 'string', enum: ['fire', 'water', 'earth', 'air', 'shadow'], position: 1 },
      rarity:  { type: 'string', enum: ['common', 'uncommon', 'rare', 'legendary'], position: 2 },
      power:   { type: 'integer', minimum: 1, maximum: 100, position: 3 },
      defense: { type: 'integer', minimum: 1, maximum: 100, position: 4 },
      ability: { type: 'string', maxLength: 128, position: 5 },
      edition: { type: 'integer', minimum: 1, position: 6 },
    },
    required: ['name', 'element', 'rarity', 'power', 'defense', 'edition'],
    additionalProperties: false,
  },
  deck: {
    type: 'object',
    properties: {
      name:    { type: 'string', maxLength: 63, position: 0 },
      cardIds: {
        type: 'array',
        items: { type: 'string', maxLength: 44 },
        minItems: 5,
        maxItems: 10,
        position: 1,
      },
    },
    required: ['name', 'cardIds'],
    additionalProperties: false,
  },
  match: {
    type: 'object',
    properties: {
      player1Id:    { type: 'string', maxLength: 44, position: 0 },
      player2Id:    { type: 'string', maxLength: 44, position: 1 },
      winnerId:     { type: 'string', maxLength: 44, position: 2 },
      player1Score: { type: 'integer', minimum: 0, position: 3 },
      player2Score: { type: 'integer', minimum: 0, position: 4 },
      timestamp:    { type: 'integer', position: 5 },
    },
    required: ['player1Id', 'player2Id', 'winnerId', 'timestamp'],
    additionalProperties: false,
  },
};

// Build the token configuration using SDK classes
const localization = new TokenConfigurationLocalization(true, 'Gem', 'Gems');
const conventions = new TokenConfigurationConvention({ en: localization }, 0);

const ownerOnly = new ChangeControlRules({
  authorizedToMakeChange: AuthorizedActionTakers.ContractOwner(),
  adminActionTakers: AuthorizedActionTakers.ContractOwner(),
});
const noOne = new ChangeControlRules({
  authorizedToMakeChange: AuthorizedActionTakers.NoOne(),
  adminActionTakers: AuthorizedActionTakers.NoOne(),
});

const gemTokenConfig = new TokenConfiguration({
  conventions,
  conventionsChangeRules: noOne,
  baseSupply: 0n,
  maxSupply: 10_000_000n,  // 10 million Gems total
  maxSupplyChangeRules: noOne,
  keepsHistory: new TokenKeepsHistoryRules({
    isKeepingMintingHistory: true,
    isKeepingBurningHistory: true,
    isKeepingTransferHistory: true,
  }),
  distributionRules: new TokenDistributionRules({
    perpetualDistributionRules: noOne,
    newTokensDestinationIdentityRules: noOne,
    mintingAllowChoosingDestination: true,
    mintingAllowChoosingDestinationRules: noOne,
    changeDirectPurchasePricingRules: noOne,
  }),
  marketplaceRules: new TokenMarketplaceRules(TokenTradeMode.NotTradeable(), noOne),
  manualMintingRules: ownerOnly,
  manualBurningRules: ownerOnly,
  freezeRules: noOne,
  unfreezeRules: noOne,
  destroyFrozenFundsRules: noOne,
  emergencyActionRules: noOne,
  mainControlGroupCanBeModified: AuthorizedActionTakers.NoOne(),
});

Step 2: Deploy the contract

import { EvoSDK, DataContract, Document, Identifier, IdentitySigner } from '@dashevo/evo-sdk';

const sdk = EvoSDK.testnetTrusted();
await sdk.connect();

// Game operator identity
const operatorId = 'OPERATOR_IDENTITY_ID';
const operatorKey = 'OPERATOR_PRIVATE_KEY_WIF';

// Set up signing
const operatorIdentity = await sdk.identities.fetch(operatorId);
const operatorIdentityKey = operatorIdentity.publicKeys[0];
const operatorSigner = new IdentitySigner();
operatorSigner.addKeyFromWif(operatorKey);

const nonce = await sdk.identities.nonce(operatorId);
const dataContract = new DataContract({
  ownerId: new Identifier(operatorId),
  identityNonce: nonce + 1n,
  schemas: gameSchema,
  tokens: { 0: gemTokenConfig },
});
const contract = await sdk.contracts.publish({
  dataContract,
  identityKey: operatorIdentityKey,
  signer: operatorSigner,
});

const contractId = contract.id.toString();

console.log('Game contract:', contractId);

Step 3: Mint starter Gems for a new player

When a player joins, give them starter Gems:

async function onboardPlayer(playerId: string) {
  // Token operations require a CRITICAL security level key
  // Gift 100 Gems to the new player
  await sdk.tokens.mint({
    dataContractId: new Identifier(contractId),
    tokenPosition: 0,
    amount: 100n,
    recipientId: new Identifier(playerId),
    identityId: new Identifier(operatorId),
    identityKey: operatorIdentityKey,
    signer: operatorSigner,
  });

  console.log(`Welcomed ${playerId} with 100 Gems`);
}

Step 4: Create a card pack (operator mints cards)

The operator creates cards as documents. Each card is owned by the operator initially, then transferred to players when purchased.

// Define a set of cards for a pack
const starterPack = [
  { name: 'Flame Sprite',    element: 'fire',   rarity: 'common',   power: 15, defense: 10, edition: 1 },
  { name: 'Tidal Guardian',  element: 'water',  rarity: 'common',   power: 10, defense: 20, edition: 1 },
  { name: 'Stone Golem',     element: 'earth',  rarity: 'uncommon', power: 25, defense: 30, edition: 1 },
  { name: 'Wind Dancer',     element: 'air',    rarity: 'common',   power: 20, defense: 12, edition: 1 },
  { name: 'Shadow Wraith',   element: 'shadow', rarity: 'rare',     power: 40, defense: 15, edition: 1 },
];

async function createCards(cards: typeof starterPack) {
  for (const card of cards) {
    const cardDoc = new Document({
      documentTypeName: 'card',
      dataContractId: new Identifier(contractId),
      ownerId: new Identifier(operatorId),
      properties: card,
    });
    await sdk.documents.create({
      document: cardDoc,
      identityKey: operatorIdentityKey,
      signer: operatorSigner,
    });
    console.log(`Created: ${card.name} (${card.rarity})`);
  }
}

await createCards(starterPack);

Step 5: Player buys a card pack

The purchase flow:

  1. Player spends Gems (transfer to operator)
  2. Operator transfers card documents to the player
const PACK_PRICE = 50n; // 50 Gems per pack

async function buyPack(playerId: string, playerKey: string) {
  // Set up player signing (token ops require CRITICAL security level key)
  const playerIdentity = await sdk.identities.fetch(playerId);
  const playerIdentityKey = playerIdentity.publicKeys[0];
  const playerSigner = new IdentitySigner();
  playerSigner.addKeyFromWif(playerKey);

  // Player pays Gems to the operator
  await sdk.tokens.transfer({
    dataContractId: new Identifier(contractId),
    tokenPosition: 0,
    amount: PACK_PRICE,
    recipientId: new Identifier(operatorId),
    senderId: new Identifier(playerId),
    identityKey: playerIdentityKey,
    signer: playerSigner,
  });
  console.log(`Player paid ${PACK_PRICE} Gems`);

  // Operator transfers cards to the player
  // (In production, select random cards from available pool)
  const availableCards = await sdk.documents.query({
    dataContractId: contractId,
    documentTypeName: 'card',
    where: [['$ownerId', '==', operatorId]],
    limit: 5,
  });

  for (const [cardId, card] of availableCards) {
    if (!card) continue;
    await sdk.documents.transfer({
      document: card,
      recipientId: new Identifier(playerId),
      identityKey: operatorIdentityKey,
      signer: operatorSigner,
    });
    const props = card.properties as Record<string, unknown>;
    console.log(`Transferred ${props.name} to player`);
  }
}

Step 6: Query a player's collection

async function getCollection(playerId: string) {
  const cards = await sdk.documents.query({
    dataContractId: contractId,
    documentTypeName: 'card',
    where: [['$ownerId', '==', playerId]],
    orderBy: [['power', 'desc']],
    limit: 100,
  });

  console.log(`\n${playerId}'s collection:`);
  for (const [id, card] of cards) {
    if (!card) continue;
    const d = card.properties as Record<string, unknown>;
    console.log(`  [${d.rarity}] ${d.name} — ${d.element} — ATK:${d.power} DEF:${d.defense}`);
  }

  return cards;
}

Filter by rarity

const legendaries = await sdk.documents.query({
  dataContractId: contractId,
  documentTypeName: 'card',
  where: [
    ['$ownerId', '==', playerId],
    ['rarity', '==', 'legendary'],
  ],
  limit: 50,
});

Step 7: Trade cards between players

Player-to-player trading using document transfers:

async function tradeCards(
  fromId: string, fromKey: string, fromCardId: string,
  toId: string, toKey: string, toCardId: string,
) {
  // Set up signers for both players
  const fromIdentity = await sdk.identities.fetch(fromId);
  const fromIdentityKey = fromIdentity.publicKeys[0];
  const fromSigner = new IdentitySigner();
  fromSigner.addKeyFromWif(fromKey);

  const toIdentity = await sdk.identities.fetch(toId);
  const toIdentityKey = toIdentity.publicKeys[0];
  const toSigner = new IdentitySigner();
  toSigner.addKeyFromWif(toKey);

  // Fetch both card documents
  const fromCard = await sdk.documents.get(contractId, 'card', fromCardId);
  const toCard = await sdk.documents.get(contractId, 'card', toCardId);

  // Player A sends their card to Player B
  await sdk.documents.transfer({
    document: fromCard,
    recipientId: new Identifier(toId),
    identityKey: fromIdentityKey,
    signer: fromSigner,
  });

  // Player B sends their card to Player A
  await sdk.documents.transfer({
    document: toCard,
    recipientId: new Identifier(fromId),
    identityKey: toIdentityKey,
    signer: toSigner,
  });

  console.log('Trade complete!');
}

Step 8: Record a match result

async function recordMatch(
  player1Id: string, player2Id: string,
  winnerId: string,
  p1Score: number, p2Score: number,
) {
  const matchDoc = new Document({
    documentTypeName: 'match',
    dataContractId: new Identifier(contractId),
    ownerId: new Identifier(operatorId),
    properties: {
      player1Id,
      player2Id,
      winnerId,
      player1Score: p1Score,
      player2Score: p2Score,
      timestamp: Date.now(),
    },
  });
  await sdk.documents.create({
    document: matchDoc,
    identityKey: operatorIdentityKey,
    signer: operatorSigner,
  });

  // Reward the winner with Gems (token ops require CRITICAL security level key)
  await sdk.tokens.mint({
    dataContractId: new Identifier(contractId),
    tokenPosition: 0,
    amount: 10n,
    recipientId: new Identifier(winnerId),
    identityId: new Identifier(operatorId),
    identityKey: operatorIdentityKey,
    signer: operatorSigner,
  });

  console.log(`Match recorded. ${winnerId} wins and earns 10 Gems!`);
}

Step 9: Leaderboard

Query match history to build a win count:

async function getWinCounts() {
  const matches = await sdk.documents.query({
    dataContractId: contractId,
    documentTypeName: 'match',
    orderBy: [['timestamp', 'desc']],
    limit: 100,
  });

  const wins = new Map<string, number>();
  for (const [, doc] of matches) {
    if (!doc) continue;
    const props = doc.properties as Record<string, unknown>;
    const winner = props.winnerId as string;
    wins.set(winner, (wins.get(winner) ?? 0) + 1);
  }

  // Sort by wins descending
  const sorted = [...wins.entries()].sort((a, b) => b[1] - a[1]);
  console.log('\nLeaderboard:');
  sorted.forEach(([id, count], i) => {
    console.log(`  ${i + 1}. ${id.slice(0, 8)}... — ${count} wins`);
  });
}

Architecture recap

┌──────────────────────────────────────────────────┐
│                  Game Contract                    │
├──────────────────┬───────────────┬───────────────┤
│  card (document) │ deck (doc)    │ match (doc)   │
│  - name, element │ - cardIds[]   │ - players     │
│  - rarity, power │               │ - winner      │
│  - transferable  │               │ - scores      │
├──────────────────┴───────────────┴───────────────┤
│  GemToken (token position 0)                     │
│  - in-game currency                              │
│  - minted as rewards, spent on packs             │
└──────────────────────────────────────────────────┘

Next steps

  • Add deck validation — check that a deck only contains cards the player owns
  • Implement card pricing with sdk.documents.setPrice() for a marketplace
  • Add seasonal editions with different edition numbers
  • Build a real-time game client that listens for match results
  • Use groups for guild/clan systems with shared card pools

Tutorial: React Integration

Build a React application that connects to Dash Platform, queries data, and broadcasts state transitions. This tutorial covers SDK initialization in a React context, handling async WASM loading, and patterns for queries and mutations.

What you will learn

  • Initializing the Evo SDK in a React app with proper lifecycle management
  • Creating a React context/provider for SDK access
  • Building hooks for queries and state transitions
  • Handling loading, error, and connected states
  • Working with the SDK in both development and production builds

Prerequisites

npx create-vite@latest my-dash-app -- --template react-ts
cd my-dash-app
npm install @dashevo/evo-sdk

Vite is recommended because it handles WASM imports natively. Create React App (webpack 4) requires additional configuration for WASM — Vite works out of the box.

Step 1: Create the SDK provider

The SDK must be initialized once and shared across the app. A React context is the natural fit.

src/DashProvider.tsx

import { createContext, useContext, useEffect, useState, type ReactNode } from 'react';
import { EvoSDK } from '@dashevo/evo-sdk';

interface DashContextValue {
  sdk: EvoSDK | null;
  isConnecting: boolean;
  error: string | null;
}

const DashContext = createContext<DashContextValue>({
  sdk: null,
  isConnecting: true,
  error: null,
});

export function useDash() {
  return useContext(DashContext);
}

export function useSDK(): EvoSDK {
  const { sdk } = useDash();
  if (!sdk) throw new Error('SDK not connected. Wrap your app in <DashProvider>.');
  return sdk;
}

interface DashProviderProps {
  network?: 'testnet' | 'mainnet' | 'local';
  children: ReactNode;
}

export function DashProvider({ network = 'testnet', children }: DashProviderProps) {
  const [sdk, setSdk] = useState<EvoSDK | null>(null);
  const [isConnecting, setIsConnecting] = useState(true);
  const [error, setError] = useState<string | null>(null);

  useEffect(() => {
    let cancelled = false;

    async function connect() {
      try {
        setIsConnecting(true);
        setError(null);

        const instance = new EvoSDK({ network, trusted: true });
        await instance.connect();

        if (!cancelled) {
          setSdk(instance);
        }
      } catch (err) {
        if (!cancelled) {
          setError(err instanceof Error ? err.message : 'Failed to connect');
        }
      } finally {
        if (!cancelled) {
          setIsConnecting(false);
        }
      }
    }

    connect();

    return () => {
      cancelled = true;
    };
  }, [network]);

  return (
    <DashContext.Provider value={{ sdk, isConnecting, error }}>
      {children}
    </DashContext.Provider>
  );
}

src/main.tsx

import { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import { DashProvider } from './DashProvider';
import App from './App';

createRoot(document.getElementById('root')!).render(
  <StrictMode>
    <DashProvider network="testnet">
      <App />
    </DashProvider>
  </StrictMode>,
);

Step 2: Build query hooks

Create reusable hooks for common queries. These handle loading and error states automatically.

src/hooks/useDashQuery.ts

import { useEffect, useState } from 'react';
import { useDash } from '../DashProvider';
import type { EvoSDK } from '@dashevo/evo-sdk';

interface QueryState<T> {
  data: T | null;
  isLoading: boolean;
  error: string | null;
  refetch: () => void;
}

export function useDashQuery<T>(
  queryFn: (sdk: EvoSDK) => Promise<T>,
  deps: unknown[] = [],
): QueryState<T> {
  const { sdk } = useDash();
  const [data, setData] = useState<T | null>(null);
  const [isLoading, setIsLoading] = useState(true);
  const [error, setError] = useState<string | null>(null);
  const [trigger, setTrigger] = useState(0);

  useEffect(() => {
    if (!sdk) return;

    let cancelled = false;
    setIsLoading(true);

    queryFn(sdk)
      .then((result) => {
        if (!cancelled) setData(result);
      })
      .catch((err) => {
        if (!cancelled) setError(err.message);
      })
      .finally(() => {
        if (!cancelled) setIsLoading(false);
      });

    return () => { cancelled = true; };
  }, [sdk, trigger, ...deps]);

  return {
    data,
    isLoading,
    error,
    refetch: () => setTrigger((n) => n + 1),
  };
}

Specific query hooks

src/hooks/useIdentity.ts

import { useDashQuery } from './useDashQuery';

export function useIdentity(identityId: string) {
  return useDashQuery(
    (sdk) => sdk.identities.fetch(identityId),
    [identityId],
  );
}

src/hooks/useDocuments.ts

import { useDashQuery } from './useDashQuery';
import type { DocumentsQuery } from '@dashevo/evo-sdk';

export function useDocuments(query: DocumentsQuery) {
  return useDashQuery(
    (sdk) => sdk.documents.query(query),
    [JSON.stringify(query)],
  );
}

src/hooks/useTokenBalance.ts

import { useDashQuery } from './useDashQuery';

export function useTokenBalance(identityId: string, tokenId: string) {
  return useDashQuery(
    async (sdk) => {
      const balances = await sdk.tokens.identityBalances(identityId, [tokenId]);
      for (const [id, balance] of balances) {
        if (id.toString() === tokenId) return balance;
      }
      return 0n;
    },
    [identityId, tokenId],
  );
}

Step 3: Build a mutation hook

For state transitions (writes), create a hook that manages submission state:

src/hooks/useDashMutation.ts

import { useState, useCallback } from 'react';
import { useSDK } from '../DashProvider';
import type { EvoSDK } from '@dashevo/evo-sdk';

interface MutationState<T> {
  execute: () => Promise<T | undefined>;
  isSubmitting: boolean;
  error: string | null;
  reset: () => void;
}

export function useDashMutation<T>(
  mutationFn: (sdk: EvoSDK) => Promise<T>,
): MutationState<T> {
  const sdk = useSDK();
  const [isSubmitting, setIsSubmitting] = useState(false);
  const [error, setError] = useState<string | null>(null);

  const execute = useCallback(async () => {
    try {
      setIsSubmitting(true);
      setError(null);
      const result = await mutationFn(sdk);
      return result;
    } catch (err) {
      setError(err instanceof Error ? err.message : 'Transaction failed');
    } finally {
      setIsSubmitting(false);
    }
  }, [sdk, mutationFn]);

  return {
    execute,
    isSubmitting,
    error,
    reset: () => setError(null),
  };
}

Step 4: Build components

Connection status

src/components/ConnectionStatus.tsx

import { useDash } from '../DashProvider';

export function ConnectionStatus() {
  const { sdk, isConnecting, error } = useDash();

  if (isConnecting) return <span className="status connecting">Connecting...</span>;
  if (error) return <span className="status error">Error: {error}</span>;
  if (sdk) return <span className="status connected">Connected to testnet</span>;
  return null;
}

Identity viewer

src/components/IdentityViewer.tsx

import { useState } from 'react';
import { useIdentity } from '../hooks/useIdentity';

export function IdentityViewer() {
  const [identityId, setIdentityId] = useState('');
  const [searchId, setSearchId] = useState('');
  const { data: identity, isLoading, error } = useIdentity(searchId);

  return (
    <div>
      <h2>Fetch Identity</h2>
      <form onSubmit={(e) => { e.preventDefault(); setSearchId(identityId); }}>
        <input
          value={identityId}
          onChange={(e) => setIdentityId(e.target.value)}
          placeholder="Enter identity ID"
        />
        <button type="submit">Fetch</button>
      </form>

      {isLoading && searchId && <p>Loading...</p>}
      {error && <p className="error">{error}</p>}
      {identity && (
        <div className="identity-card">
          <p><strong>ID:</strong> {identity.id.toString()}</p>
          <p><strong>Balance:</strong> {identity.balance.toString()} credits</p>
          <p><strong>Public keys:</strong> {identity.publicKeys.length}</p>
        </div>
      )}
    </div>
  );
}

Document list (e.g., car listings from the previous tutorial)

src/components/ListingsList.tsx

import { useDocuments } from '../hooks/useDocuments';

const CONTRACT_ID = 'YOUR_CONTRACT_ID';

export function ListingsList() {
  const { data: results, isLoading, error, refetch } = useDocuments({
    dataContractId: CONTRACT_ID,
    documentTypeName: 'listing',
    where: [['status', '==', 'available']],
    orderBy: [['priceUsd', 'asc']],
    limit: 20,
  });

  if (isLoading) return <p>Loading listings...</p>;
  if (error) return <p className="error">{error}</p>;
  if (!results || results.size === 0) return <p>No listings found.</p>;

  return (
    <div>
      <h2>Available Cars</h2>
      <button onClick={refetch}>Refresh</button>
      <ul>
        {[...results.entries()].map(([id, doc]) => {
          if (!doc) return null;
          const d = doc.properties as Record<string, unknown>;
          return (
            <li key={id}>
              <strong>{d.year} {d.make} {d.model}</strong> — ${d.priceUsd}
            </li>
          );
        })}
      </ul>
    </div>
  );
}

Create document form

src/components/CreateListing.tsx

import { useState, useCallback } from 'react';
import { Document, Identifier, IdentitySigner } from '@dashevo/evo-sdk';
import { useDashMutation } from '../hooks/useDashMutation';

const CONTRACT_ID = 'YOUR_CONTRACT_ID';
const IDENTITY_ID = 'YOUR_IDENTITY_ID';
const PRIVATE_KEY = 'YOUR_PRIVATE_KEY_WIF';
const SIGNING_KEY_INDEX = 0;

export function CreateListing() {
  const [make, setMake] = useState('');
  const [model, setModel] = useState('');
  const [year, setYear] = useState(2024);
  const [price, setPrice] = useState(0);

  const mutation = useDashMutation(
    useCallback(
      async (sdk) => {
        const identity = await sdk.identities.fetch(IDENTITY_ID);
        const identityKey = identity.publicKeys[SIGNING_KEY_INDEX];
        const signer = new IdentitySigner();
        signer.addKeyFromWif(PRIVATE_KEY);

        const doc = new Document({
          documentTypeName: 'listing',
          dataContractId: new Identifier(CONTRACT_ID),
          ownerId: new Identifier(IDENTITY_ID),
          properties: {
            make,
            model,
            year,
            priceUsd: price,
            mileageKm: 0,
            status: 'available',
          },
        });
        return sdk.documents.create({ document: doc, identityKey, signer });
      },
      [make, model, year, price],
    ),
  );

  return (
    <form
      onSubmit={async (e) => {
        e.preventDefault();
        await mutation.execute();
      }}
    >
      <h2>Create Listing</h2>
      <input placeholder="Make" value={make} onChange={(e) => setMake(e.target.value)} />
      <input placeholder="Model" value={model} onChange={(e) => setModel(e.target.value)} />
      <input type="number" placeholder="Year" value={year} onChange={(e) => setYear(+e.target.value)} />
      <input type="number" placeholder="Price (USD)" value={price} onChange={(e) => setPrice(+e.target.value)} />
      <button type="submit" disabled={mutation.isSubmitting}>
        {mutation.isSubmitting ? 'Submitting...' : 'Create Listing'}
      </button>
      {mutation.error && <p className="error">{mutation.error}</p>}
    </form>
  );
}

Step 5: Assemble the app

src/App.tsx

import { ConnectionStatus } from './components/ConnectionStatus';
import { IdentityViewer } from './components/IdentityViewer';
import { ListingsList } from './components/ListingsList';
import { CreateListing } from './components/CreateListing';
import { useDash } from './DashProvider';

export default function App() {
  const { sdk } = useDash();

  return (
    <div className="app">
      <header>
        <h1>Dash Platform App</h1>
        <ConnectionStatus />
      </header>

      {sdk ? (
        <main>
          <IdentityViewer />
          <ListingsList />
          <CreateListing />
        </main>
      ) : (
        <p>Waiting for SDK connection...</p>
      )}
    </div>
  );
}

Production considerations

Private key management

The examples above hardcode private keys for clarity. In production:

  • Never ship private keys in frontend code
  • Use a backend service to sign state transitions, or
  • Prompt the user for their mnemonic/key at runtime and keep it in memory only
  • Consider the wallet namespace for key derivation from user-provided mnemonics
import { wallet } from '@dashevo/evo-sdk';

async function signWithUserMnemonic(mnemonic: string) {
  const keyInfo = await wallet.deriveKeyFromSeedPhrase({
    mnemonic,
    network: 'testnet',
    derivationPath: "m/9'/1'/0'/0/0",
  });
  return keyInfo.privateKeyWif;
}

Bundle size

The WASM module adds ~2-4 MB (gzipped) to your bundle. To optimise:

  • Use code splitting — the SDK module only loads when connect() is called
  • Vite handles WASM lazy loading automatically
  • Consider loading the SDK only on pages that need it

Error boundaries

Wrap SDK-dependent components in an error boundary to handle WASM initialization failures gracefully:

import { ErrorBoundary } from 'react-error-boundary';

<ErrorBoundary fallback={<p>Failed to load Dash SDK</p>}>
  <DashProvider>
    <App />
  </DashProvider>
</ErrorBoundary>

Network switching

To let users switch networks at runtime, key the provider on the network value:

const [network, setNetwork] = useState<'testnet' | 'mainnet'>('testnet');

<DashProvider network={network} key={network}>
  <App />
</DashProvider>

The key prop forces React to unmount and remount the provider, creating a fresh SDK connection for the new network.

Builder Pattern

The Dash Platform Rust SDK (packages/rs-sdk) is the primary way applications interact with Dash Platform. Before you can fetch identities, create documents, or broadcast state transitions, you need an Sdk instance. And to create an Sdk, you use the builder pattern.

This chapter covers SdkBuilder, the Sdk struct it produces, and the two modes of operation: normal (real network) and mock (testing).

The Problem

Creating an Sdk requires many pieces of configuration:

  • Network addresses (where are the DAPI nodes?)
  • Network type (mainnet, testnet, devnet, regtest?)
  • Request settings (timeouts, retries, ban policies)
  • Context provider (where do cached data contracts and quorum keys come from?)
  • Staleness checks (how old can metadata be before we reject it?)
  • Platform version (which protocol version should we use?)
  • Cancellation token (how do we abort pending requests?)
  • TLS certificates (for secure connections)

Most of these have sensible defaults. A constructor with 10 parameters would be unusable. The builder pattern lets you set only what you need.

SdkBuilder

SdkBuilder lives in packages/rs-sdk/src/sdk.rs:

#![allow(unused)]
fn main() {
pub struct SdkBuilder {
    addresses: Option<AddressList>,
    settings: Option<RequestSettings>,
    network: Network,
    core_ip: String,
    core_port: u16,
    core_user: String,
    core_password: Zeroizing<String>,
    proofs: bool,
    version: &'static PlatformVersion,
    context_provider: Option<Box<dyn ContextProvider>>,
    metadata_height_tolerance: Option<u64>,
    metadata_time_tolerance_ms: Option<u64>,
    cancel_token: CancellationToken,

    #[cfg(feature = "mocks")]
    data_contract_cache_size: NonZeroUsize,
    #[cfg(feature = "mocks")]
    token_config_cache_size: NonZeroUsize,
    #[cfg(feature = "mocks")]
    quorum_public_keys_cache_size: NonZeroUsize,
    #[cfg(feature = "mocks")]
    dump_dir: Option<PathBuf>,

    #[cfg(not(target_arch = "wasm32"))]
    ca_certificate: Option<Certificate>,
}
}

Constructor Methods

The builder offers several constructors for different scenarios:

#![allow(unused)]
fn main() {
// Normal mode: connect to specified DAPI nodes
let sdk = SdkBuilder::new(address_list)
    .with_network(Network::Testnet)
    .build()?;

// Mock mode: no network, useful for tests
let sdk = SdkBuilder::new_mock()
    .build()?;

// Convenience (not yet implemented):
let sdk = SdkBuilder::new_testnet().build()?;
let sdk = SdkBuilder::new_mainnet().build()?;
}

The key distinction: if addresses is Some, you get a real DapiClient that connects to DAPI nodes over gRPC. If addresses is None (the mock path), you get a MockDapiClient that responds with pre-programmed data.

Configuration Methods

Every builder method follows the same signature pattern: take mut self, modify a field, return self:

#![allow(unused)]
fn main() {
impl SdkBuilder {
    pub fn with_network(mut self, network: Network) -> Self {
        self.network = network;
        self
    }

    pub fn with_settings(mut self, settings: RequestSettings) -> Self {
        self.settings = Some(settings);
        self
    }

    pub fn with_version(mut self, version: &'static PlatformVersion) -> Self {
        self.version = version;
        self
    }

    pub fn with_context_provider<C: ContextProvider + 'static>(
        mut self,
        context_provider: C,
    ) -> Self {
        self.context_provider = Some(Box::new(context_provider));
        self
    }

    pub fn with_cancellation_token(
        mut self,
        cancel_token: CancellationToken,
    ) -> Self {
        self.cancel_token = cancel_token;
        self
    }
}
}

This lets you chain configuration fluently:

#![allow(unused)]
fn main() {
let sdk = SdkBuilder::new(addresses)
    .with_network(Network::Testnet)
    .with_version(PlatformVersion::latest())
    .with_settings(RequestSettings { retries: Some(5), ..Default::default() })
    .with_context_provider(my_provider)
    .with_cancellation_token(token)
    .build()?;
}

Staleness Configuration

The SDK protects against stale responses from out-of-date nodes:

#![allow(unused)]
fn main() {
// Reject responses whose height is behind by more than 1 block
let sdk = SdkBuilder::new(addresses)
    .with_height_tolerance(Some(1))     // default
    .with_time_tolerance(Some(360_000)) // 6 minutes
    .build()?;
}

Height tolerance defaults to Some(1) -- if a node returns metadata with a height more than 1 block behind the last seen height, the SDK considers it stale. Time tolerance defaults to None (disabled) because it requires synchronized clocks.

Dash Core Integration

For development, the SDK can use Dash Core as a wallet and context provider:

#![allow(unused)]
fn main() {
let sdk = SdkBuilder::new(addresses)
    .with_core("127.0.0.1", 19998, "user", "password")
    .build()?;
}

This is a convenience method that internally creates a GrpcContextProvider backed by Core's RPC interface. For production, you should implement ContextProvider yourself.

Dump Directory

For debugging, the SDK can record all gRPC requests and responses to disk:

#![allow(unused)]
fn main() {
let sdk = SdkBuilder::new(addresses)
    .with_dump_dir(Path::new("./sdk-dumps"))
    .build()?;
}

This creates files like msg-*.json, quorum_pubkey-*.json, and data_contract-*.json that can be replayed in mock mode.

The Sdk Struct

The build() method produces an Sdk:

#![allow(unused)]
fn main() {
pub struct Sdk {
    pub network: Network,
    inner: SdkInstance,
    proofs: bool,
    internal_cache: Arc<InternalSdkCache>,
    context_provider: ArcSwapOption<Box<dyn ContextProvider>>,
    metadata_last_seen_height: Arc<atomic::AtomicU64>,
    metadata_height_tolerance: Option<u64>,
    metadata_time_tolerance_ms: Option<u64>,
    pub(crate) cancel_token: CancellationToken,
    pub(crate) dapi_client_settings: RequestSettings,
}
}

SdkInstance: Normal vs Mock

The inner field is an enum that holds either a real or mock client:

#![allow(unused)]
fn main() {
enum SdkInstance {
    Dapi {
        dapi: DapiClient,
        version: &'static PlatformVersion,
    },
    #[cfg(feature = "mocks")]
    Mock {
        dapi: Arc<Mutex<MockDapiClient>>,
        mock: Arc<Mutex<MockDashPlatformSdk>>,
        address_list: AddressList,
        version: &'static PlatformVersion,
    },
}
}

All public Sdk methods work identically in both modes. Code that uses the SDK does not know (or care) whether it is talking to a real network or a mock.

Thread Safety

Sdk is Clone and thread-safe. It uses Arc for shared state and ArcSwapOption for the context provider (which allows lock-free reads). The mock mode uses tokio::Mutex for the mock client since mock state is modified in async contexts.

Nonce Management

The SDK maintains an internal cache of identity nonces to avoid querying the network on every state transition:

#![allow(unused)]
fn main() {
pub async fn get_identity_nonce(
    &self,
    identity_id: Identifier,
    bump_first: bool,
    settings: Option<PutSettings>,
) -> Result<IdentityNonce, Error> {
    // 1. Check cache
    // 2. If stale or absent, query Platform
    // 3. Optionally bump (increment) before returning
    // 4. Apply IDENTITY_NONCE_VALUE_FILTER mask
}
}

The cache has a staleness timeout (default: 20 minutes). When bump_first is true, the nonce is incremented before being returned -- this is used when creating new state transitions that need the next nonce value.

The Quick Mock Path

For tests that need a mock SDK immediately:

#![allow(unused)]
fn main() {
let sdk = Sdk::new_mock();
}

This is a shorthand for SdkBuilder::default().build().unwrap(). It creates an SDK in mock mode with all default settings. You can then configure expectations:

#![allow(unused)]
fn main() {
let mut sdk = Sdk::new_mock();
sdk.mock().expect_fetch(identity, None);
}

Request Settings

The SDK applies a chain of settings to every request:

#![allow(unused)]
fn main() {
const DEFAULT_REQUEST_SETTINGS: RequestSettings = RequestSettings {
    retries: Some(3),
    timeout: None,
    ban_failed_address: None,
    connect_timeout: None,
    max_decoding_message_size: None,
};
}

When building, user-provided settings override defaults:

#![allow(unused)]
fn main() {
let dapi_client_settings = match self.settings {
    Some(settings) => DEFAULT_REQUEST_SETTINGS.override_by(settings),
    None => DEFAULT_REQUEST_SETTINGS,
};
}

And when making individual requests, per-request settings override global settings:

#![allow(unused)]
fn main() {
let settings = sdk
    .dapi_client_settings
    .override_by(request_specific_settings);
}

This three-level cascade (defaults -> builder -> per-request) gives you control without verbosity.

Rules

Do:

  • Use SdkBuilder::new(addresses) for production code with real DAPI connections.
  • Use Sdk::new_mock() for quick unit tests.
  • Set with_context_provider() in production -- the fallback to Core RPC is for development only.
  • Use with_height_tolerance() to detect stale nodes.
  • Clone the SDK freely -- it is designed for shared ownership via Arc.

Don't:

  • Use new_testnet() or new_mainnet() -- they are not implemented yet.
  • Disable proofs (with_proofs(false)) in production -- proofs are the security model.
  • Set metadata_time_tolerance_ms too low -- network delays and time skew can cause false positives.
  • Forget the cancellation token in long-running applications -- without it, you cannot gracefully shut down pending requests.
  • Construct Sdk directly -- always use the builder.

Fetch Traits

Reading data from Dash Platform is the most common SDK operation. You need to look up an identity by its identifier, retrieve a data contract, query documents, check a balance. The SDK provides a unified abstraction for all of these: the Fetch and FetchMany traits.

This chapter covers how these traits work, how queries are formed, how proofs are verified, and how different Platform types plug into the system.

The Problem

Platform stores many different types of data: identities, data contracts, documents, balances, epoch info, votes, token configurations, and more. Each requires a different gRPC request, returns a different response, and needs different proof verification logic.

Without an abstraction, every type would need its own fetch function with duplicated retry logic, proof parsing, metadata validation, and error handling. The Fetch trait eliminates that duplication.

The Fetch Trait

Fetch is defined in packages/rs-sdk/src/platform/fetch.rs:

#![allow(unused)]
fn main() {
#[async_trait::async_trait]
pub trait Fetch
where
    Self: Sized
        + Debug
        + MockResponse
        + FromProof<
            <Self as Fetch>::Request,
            Request = <Self as Fetch>::Request,
            Response = <<Self as Fetch>::Request as DapiRequest>::Response,
        >,
{
    /// The gRPC request type used to fetch this object.
    type Request: TransportRequest
        + Into<<Self as FromProof<<Self as Fetch>::Request>>::Request>;

    /// Fetch a single object from Platform.
    async fn fetch<Q: Query<<Self as Fetch>::Request>>(
        sdk: &Sdk,
        query: Q,
    ) -> Result<Option<Self>, Error> {
        Self::fetch_with_settings(sdk, query, RequestSettings::default()).await
    }

    /// Fetch with metadata (block height, time, etc.)
    async fn fetch_with_metadata<Q: Query<<Self as Fetch>::Request>>(
        sdk: &Sdk,
        query: Q,
        settings: Option<RequestSettings>,
    ) -> Result<(Option<Self>, ResponseMetadata), Error> { /* ... */ }

    /// Fetch with metadata and the raw proof.
    async fn fetch_with_metadata_and_proof<Q: Query<<Self as Fetch>::Request>>(
        sdk: &Sdk,
        query: Q,
        settings: Option<RequestSettings>,
    ) -> Result<(Option<Self>, ResponseMetadata, Proof), Error> { /* ... */ }

    /// Fetch with custom request settings.
    async fn fetch_with_settings<Q: Query<<Self as Fetch>::Request>>(
        sdk: &Sdk,
        query: Q,
        settings: RequestSettings,
    ) -> Result<Option<Self>, Error> { /* ... */ }

    /// Convenience: fetch by identifier.
    async fn fetch_by_identifier(
        sdk: &Sdk,
        id: Identifier,
    ) -> Result<Option<Self>, Error>
    where
        Identifier: Query<<Self as Fetch>::Request>,
    {
        Self::fetch(sdk, id).await
    }
}
}

The Key Insight: Option Semantics

Notice the return type: Result<Option<Self>, Error>.

  • Ok(Some(item)) -- the object was found and verified.
  • Ok(None) -- the object was proven to not exist. This is not an error; it is a cryptographic proof of absence.
  • Err(error) -- something went wrong (network failure, proof verification failed, etc.)

This design means "not found" is a normal, expected outcome. Code that uses Fetch does not need to handle "not found" as an error case.

Usage

#![allow(unused)]
fn main() {
use dash_sdk::platform::{Fetch, Identifier, Identity};

// Fetch an identity
let identity = Identity::fetch(&sdk, some_identifier).await?;

match identity {
    Some(id) => println!("Found identity with balance: {}", id.balance()),
    None => println!("Identity does not exist"),
}
}

Implementing Fetch for a Type

For most types, implementing Fetch is a one-liner:

#![allow(unused)]
fn main() {
impl Fetch for Identity {
    type Request = IdentityRequest;
}

impl Fetch for dpp::prelude::DataContract {
    type Request = platform_proto::GetDataContractRequest;
}

impl Fetch for drive_proof_verifier::types::IdentityBalance {
    type Request = platform_proto::GetIdentityBalanceRequest;
}

impl Fetch for drive_proof_verifier::types::IdentityNonceFetcher {
    type Request = platform_proto::GetIdentityNonceRequest;
}

impl Fetch for ExtendedEpochInfo {
    type Request = platform_proto::GetEpochsInfoRequest;
}

impl Fetch for Vote {
    type Request = platform_proto::GetContestedResourceIdentityVotesRequest;
}

impl Fetch for drive_proof_verifier::types::TotalCreditsInPlatform {
    type Request = platform_proto::GetTotalCreditsInPlatformRequest;
}
}

The type Request associates each fetchable type with its gRPC request message. The default method implementations handle everything else -- sending the request, parsing the proof, verifying metadata. All you need to provide is the request type.

Document: A Custom Override

Documents are special because they depend on a data contract schema for deserialization. If the cached contract is outdated, deserialization fails. The Document implementation overrides the default to add retry logic:

#![allow(unused)]
fn main() {
#[async_trait::async_trait]
impl Fetch for Document {
    type Request = DocumentQuery;

    async fn fetch_with_metadata_and_proof<Q: Query<<Self as Fetch>::Request>>(
        sdk: &Sdk,
        query: Q,
        settings: Option<RequestSettings>,
    ) -> Result<(Option<Self>, ResponseMetadata, Proof), Error> {
        let document_query: DocumentQuery = query.query(sdk.prove())?;

        // First attempt with current (possibly cached) contract
        match fetch_request(sdk, &document_query, settings).await {
            Ok(result) => Ok(result),
            Err(e) if is_document_deserialization_error(&e) => {
                // Contract schema might have changed -- refetch it
                let fresh_query =
                    refetch_contract_for_query(sdk, &document_query).await?;
                fetch_request(sdk, &fresh_query, settings).await
            }
            Err(e) => Err(e),
        }
    }
}
}

If deserialization fails with a CorruptedSerialization error, the SDK refetches the data contract from the network, updates the cache, and retries. This handles the case where a contract was updated but the local cache still has the old version.

The Query Trait

Query converts user-friendly search criteria into gRPC request messages:

#![allow(unused)]
fn main() {
pub trait Query<T: TransportRequest + Mockable>: Send + Debug + Clone {
    fn query(self, prove: bool) -> Result<T, Error>;
}
}

The simplest implementation: any TransportRequest is a query for itself:

#![allow(unused)]
fn main() {
impl<T> Query<T> for T
where
    T: TransportRequest + Sized + Send + Sync + Clone + Debug,
{
    fn query(self, prove: bool) -> Result<T, Error> {
        if !prove {
            unimplemented!("queries without proofs are not supported");
        }
        Ok(self)
    }
}
}

But you can also implement Query for more ergonomic types. For example, Identifier implements Query<GetIdentityRequest>, so you can write:

#![allow(unused)]
fn main() {
let identity = Identity::fetch(&sdk, my_identifier).await?;
}

Instead of manually constructing a GetIdentityRequest proto message.

The FromProof Trait

Every Fetch implementation requires that the fetched type implements FromProof. This trait, defined in packages/rs-drive-proof-verifier/src/proof.rs, verifies the cryptographic proof returned by the Platform node:

#![allow(unused)]
fn main() {
pub trait FromProof<Req> {
    type Request;
    type Response;

    /// Parse and verify the proof, returning the requested object.
    ///
    /// Returns:
    /// - Ok(Some(object, metadata)) when found
    /// - Ok(None) when proven to not exist
    /// - Err when verification fails
    fn maybe_from_proof_with_metadata(
        request: Self::Request,
        response: Self::Response,
        network: Network,
        platform_version: &PlatformVersion,
        provider: &impl ContextProvider,
    ) -> Result<(Option<Self>, ResponseMetadata, Proof), Error>
    where
        Self: Sized;
}
}

The chain is: Query produces a request -> DAPI returns a response with a proof -> FromProof verifies the proof and extracts the object. Every step is type-safe and generic over the specific Platform type being fetched.

FetchMany: Retrieving Collections

FetchMany extends the pattern to collections:

#![allow(unused)]
fn main() {
pub trait FetchMany<K: Ord, O: FromIterator<(K, Option<Self>)>>
where
    Self: Sized,
    O: MockResponse
        + FromProof<Self::Request, ...>
        + Send
        + Default,
{
    type Request: TransportRequest;

    async fn fetch_many<Q: Query<Self::Request>>(
        sdk: &Sdk,
        query: Q,
    ) -> Result<O, Error> { /* ... */ }

    // ... with_settings, with_metadata, with_limit variants
}
}

The O type parameter is the output collection type. It must implement FromIterator<(K, Option<Self>)> -- a collection of key-value pairs where the value might be None (proven absent). This handles queries like "fetch documents matching these criteria" where some requested items might not exist.

The Internal Fetch Pipeline

When you call Identity::fetch(&sdk, id), here is what happens:

  1. Query conversion: id.query(true) produces a GetIdentityRequest with proofs enabled.

  2. Request execution with retry: The SDK sends the request to a DAPI node, with automatic retry logic:

    #![allow(unused)]
    fn main() {
    let fut = |settings: RequestSettings| async move {
        let response = request.clone().execute(sdk, settings).await?;
        let (object, metadata, proof) = sdk
            .parse_proof_with_metadata_and_proof(request, response)
            .await?;
        Ok((object, metadata, proof))
    };
    
    retry(sdk.address_list(), settings, fut).await
    }
  3. Proof verification: parse_proof_with_metadata_and_proof calls FromProof::maybe_from_proof_with_metadata, which verifies the GroveDB proof against quorum signatures.

  4. Metadata validation: The SDK checks that the response metadata (height, time) is fresh enough based on the configured tolerances.

  5. Result return: The verified object (or None) is returned to the caller.

Rules

Do:

  • Use Fetch::fetch() for single-object lookups by identifier.
  • Use FetchMany::fetch_many() for queries that return collections.
  • Handle Ok(None) as a normal case -- it means the object does not exist, proven cryptographically.
  • Implement Fetch for new types by specifying just type Request.
  • Override fetch_with_metadata_and_proof only when you need custom logic (like the Document retry pattern).

Don't:

  • Treat Ok(None) as an error -- "not found" is a valid, proven result.
  • Bypass the Fetch trait to make raw gRPC calls -- you would skip proof verification.
  • Forget to implement FromProof for new fetchable types -- without it, proofs cannot be verified.
  • Disable proofs in production -- query(prove: false) is not supported and will panic.
  • Implement Query conversions that lose information -- the query must fully specify what to fetch.

Put Operations

Reading data from Platform is handled by Fetch. Writing data -- creating documents, deploying contracts, registering identities -- is handled by the put operation traits. This chapter covers the write path through the SDK: how state transitions are built, signed, broadcast, and confirmed.

The Problem

Writing to Platform is fundamentally different from reading. A read is a simple request/response: send a query, get back a proof. A write involves multiple steps:

  1. Determine the correct nonce for the identity.
  2. Build a state transition from the data to be written.
  3. Sign the transition with the identity's private key.
  4. Broadcast the signed transition to the network.
  5. Wait for the transition to be included in a block.
  6. Verify the proof that the write was applied.

Each step can fail independently, and the SDK needs to handle all of them coherently.

The PutDocument Trait

The primary write trait for documents is PutDocument, defined in packages/rs-sdk/src/platform/transition/put_document.rs:

#![allow(unused)]
fn main() {
#[async_trait::async_trait]
pub trait PutDocument<S: Signer<IdentityPublicKey>>: Waitable {
    async fn put_to_platform(
        &self,
        sdk: &Sdk,
        document_type: DocumentType,
        document_state_transition_entropy: Option<[u8; 32]>,
        identity_public_key: IdentityPublicKey,
        token_payment_info: Option<TokenPaymentInfo>,
        signer: &S,
        settings: Option<PutSettings>,
    ) -> Result<StateTransition, Error>;

    async fn put_to_platform_and_wait_for_response(
        &self,
        sdk: &Sdk,
        document_type: DocumentType,
        document_state_transition_entropy: Option<[u8; 32]>,
        identity_public_key: IdentityPublicKey,
        token_payment_info: Option<TokenPaymentInfo>,
        signer: &S,
        settings: Option<PutSettings>,
    ) -> Result<Document, Error>;
}
}

There are two methods:

  • put_to_platform broadcasts the transition and returns immediately. You get back the StateTransition that was broadcast but no confirmation that it was applied.
  • put_to_platform_and_wait_for_response broadcasts and then waits for the platform to include the transition in a block, returning the confirmed Document.

The Nonce-Build-Broadcast-Wait Pipeline

Let's walk through put_to_platform step by step:

Step 1: Get the Nonce

#![allow(unused)]
fn main() {
let new_identity_contract_nonce = sdk
    .get_identity_contract_nonce(
        self.owner_id(),
        document_type.data_contract_id(),
        true,  // bump_first: increment the nonce
        settings,
    )
    .await?;
}

Every identity has a nonce that increments with each state transition targeting a specific contract. The SDK caches nonces internally and bumps them optimistically. bump_first: true means "give me the next unused nonce."

The SDK's nonce management is sophisticated:

  • Nonces are cached per (identity_id, contract_id) pair.
  • If the cache is older than the staleness timeout (default 20 minutes), the SDK re-fetches from Platform.
  • If Platform reports a higher nonce than the cache, the cache is updated.
  • A filter mask (IDENTITY_NONCE_VALUE_FILTER) is applied to keep the nonce in the valid range.

Step 2: Build the Transition

The SDK decides whether to create a new document or replace an existing one based on the document's revision:

#![allow(unused)]
fn main() {
let transition = if self.revision().is_some()
    && self.revision().unwrap() != INITIAL_REVISION
{
    // This is an update -- create a replacement transition
    BatchTransition::new_document_replacement_transition_from_document(
        self.clone(),
        document_type.as_ref(),
        &identity_public_key,
        new_identity_contract_nonce,
        settings.user_fee_increase.unwrap_or_default(),
        token_payment_info,
        signer,
        sdk.version(),
        settings.state_transition_creation_options,
    )
} else {
    // This is a new document -- generate entropy and create
    let (document, entropy) = match document_state_transition_entropy {
        Some(entropy) => (self.clone(), entropy),
        None => {
            let mut rng = StdRng::from_entropy();
            let mut document = self.clone();
            let entropy = rng.gen::<[u8; 32]>();
            document.set_id(Document::generate_document_id(
                &document_type.data_contract_id(),
                &document.owner_id(),
                document_type.name(),
                entropy.as_slice(),
                new_identity_contract_nonce,
                sdk.version(),
            )?);
            (document, entropy)
        }
    };

    BatchTransition::new_document_creation_transition_from_document(
        document,
        document_type.as_ref(),
        entropy,
        &identity_public_key,
        new_identity_contract_nonce,
        settings.user_fee_increase.unwrap_or_default(),
        token_payment_info,
        signer,
        sdk.version(),
        settings.state_transition_creation_options,
    )
}?;
}

For new documents, the SDK generates 32 bytes of entropy (unless you provide your own) and derives the document ID from it and, from protocol version 14, from the identity contract nonce it just fetched. sdk.version() tracks the network's protocol version, so the SDK switches derivation when the network does. Because the ID depends on the nonce, the ID on the document you pass in is a placeholder: use the ID of the confirmed document that put_to_platform_and_wait_for_response returns.

The JavaScript SDK follows the same pipeline. sdk.documents.create goes through put_to_platform_and_wait_for_response and hands the confirmed document back. An app that builds the transition itself (to sign it separately or cache the signed bytes) gets the derivation from wasm-dpp2: new DocumentCreateTransition({ document, identityContractNonce }) derives the ID from the document's entropy and the nonce for the network's protocol version (platformVersion option, latest by default), writes it onto the transition and back onto document, and the transition is then batched, signed and broadcast as before. The IDs such a transition carries are final; nothing has to be hashed on the app side.

Step 3: Validate Structure

Before broadcasting, the SDK validates the transition's basic structure:

#![allow(unused)]
fn main() {
ensure_valid_state_transition_structure(&transition, sdk.version())?;
}

This catches obvious errors (wrong field types, missing required fields) before the transition hits the network, saving a round-trip.

Step 4: Broadcast

#![allow(unused)]
fn main() {
transition.broadcast(sdk, Some(settings)).await?;
}

This sends the serialized transition to a DAPI node.

The BroadcastStateTransition Trait

Broadcasting is implemented as a trait on StateTransition:

#![allow(unused)]
fn main() {
#[async_trait::async_trait]
pub trait BroadcastStateTransition {
    async fn broadcast(
        &self,
        sdk: &Sdk,
        settings: Option<PutSettings>,
    ) -> Result<(), Error>;

    async fn wait_for_response<T: TryFrom<StateTransitionProofResult> + Send>(
        &self,
        sdk: &Sdk,
        settings: Option<PutSettings>,
    ) -> Result<T, Error>;

    async fn broadcast_and_wait<T: TryFrom<StateTransitionProofResult> + Send>(
        &self,
        sdk: &Sdk,
        settings: Option<PutSettings>,
    ) -> Result<T, Error>;
}
}

Three methods, three use cases:

  • broadcast: Fire-and-forget. Returns Ok(()) when the node accepts the transition. The response is always empty -- confirmation comes later.
  • wait_for_response: Poll until the transition is included in a block. Returns the proven result.
  • broadcast_and_wait: Combines both -- broadcast, then wait.

The Wait Mechanism

wait_for_response uses the WaitForStateTransitionResult gRPC endpoint. It sends the transition's hash and blocks until the platform includes it in a block:

#![allow(unused)]
fn main() {
async fn wait_for_response<T>(&self, sdk: &Sdk, settings: Option<PutSettings>)
    -> Result<T, Error>
{
    let factory = |request_settings: RequestSettings| async move {
        let request = self.wait_for_state_transition_result_request()?;
        let response = request.execute(sdk, request_settings).await?;

        // Check for broadcast errors
        if let Some(e) = state_transition_broadcast_error {
            return Err(Error::from(e));
        }

        // Extract and verify the proof
        let proof = grpc_response.proof()?;
        let (_, result) = Drive::verify_state_transition_was_executed_with_proof(
            self,
            &block_info,
            proof.grovedb_proof.as_slice(),
            &context_provider.as_contract_lookup_fn(sdk.version()),
            sdk.version(),
        )?;

        // Convert to the expected output type
        T::try_from(result)
    };

    retry(sdk.address_list(), retry_settings, factory).await
}
}

The wait includes full proof verification: the SDK verifies a GroveDB proof that the state transition was actually applied. This is not just checking a status flag -- it is cryptographic proof of inclusion.

From protocol version 14 the proof of an owned, fee-paying transition (document and token batches, contract creates and updates, identity updates and key limit updates, contract moderation) also carries the credit balance of the owner after it, read from the same state as the result, so a wallet learns what the write left it with without a second query. The verified StateTransitionProofOutcome hands it out through owner_balance() (None for a proof made at an earlier version or a transition without an owner); wait_for_document_and_owner_balance and put_to_platform_and_wait_for_response_with_owner_balance return it next to the document. The balance is a snapshot at the proof's block: it may already include later transitions of the same identity. A wait that asks for no proof but sets request_user_balance gets the owner's balance back unverified, as the response's success_with_owner_balance result, read from a Drive state at or past the block that executed the write; that works for any transition with an owner.

Timeout Handling

wait_for_response supports an optional timeout:

#![allow(unused)]
fn main() {
match wait_timeout {
    Some(timeout) => {
        tokio::time::timeout(timeout, future)
            .await
            .map_err(|_| Error::TimeoutReached(timeout, details))?
    }
    None => future.await,
}
}

Without a timeout, the wait is unbounded. For production use, always set a timeout via PutSettings.

The Waitable Trait

Waitable provides type-specific post-processing after a broadcast:

#![allow(unused)]
fn main() {
#[async_trait::async_trait]
pub trait Waitable: Sized {
    async fn wait_for_response(
        sdk: &Sdk,
        state_transition: StateTransition,
        settings: Option<PutSettings>,
    ) -> Result<Self, Error>;
}
}

Each type implements this differently:

DataContract and Vote: straightforward delegation:

#![allow(unused)]
fn main() {
impl Waitable for DataContract {
    async fn wait_for_response(
        sdk: &Sdk,
        state_transition: StateTransition,
        settings: Option<PutSettings>,
    ) -> Result<DataContract, Error> {
        state_transition.wait_for_response(sdk, settings).await
    }
}
}

Document: extracts the single document from the batch transition result:

#![allow(unused)]
fn main() {
impl Waitable for Document {
    async fn wait_for_response(
        sdk: &Sdk,
        state_transition: StateTransition,
        settings: Option<PutSettings>,
    ) -> Result<Self, Error> {
        // Verify this is a batch transition with exactly one document
        let doc_id = /* extract from transition */;

        let mut documents: BTreeMap<Identifier, Option<Document>> =
            state_transition.wait_for_response(sdk, settings).await?;

        documents.remove(&doc_id)
            .ok_or(Error::InvalidProvedResponse(...))?
            .ok_or(Error::InvalidProvedResponse(...))
    }
}
}

Identity: handles the "already exists" case specially by falling back to a fetch:

#![allow(unused)]
fn main() {
impl Waitable for Identity {
    async fn wait_for_response(
        sdk: &Sdk,
        state_transition: StateTransition,
        settings: Option<PutSettings>,
    ) -> Result<Self, Error> {
        match state_transition.wait_for_response(sdk, settings).await {
            Ok(identity) => Ok(identity),
            Err(Error::AlreadyExists(_)) => {
                // Identity already exists -- fetch it instead
                let identity_id = /* extract from transition */;
                Identity::fetch(sdk, identity_id).await?
                    .ok_or(Error::Generic("proved to not exist but said to exist"))
            }
            Err(e) => Err(e),
        }
    }
}
}

Error Handling at the SDK Level

Errors during put operations fall into several categories:

  • Nonce errors: The cached nonce was stale. The SDK refreshes nonces on broadcast failure: sdk.refresh_identity_nonce(&owner_id).await
  • Broadcast errors: The network rejected the transition. Returned as StateTransitionBroadcastError.
  • Proof errors: The proof verification failed. Returned as DriveProofError with the raw proof bytes and block info for debugging.
  • Timeout errors: The transition was not included in time. Returned as TimeoutReached with the timeout duration and a description.
  • Conversion errors: The proof result could not be converted to the expected type. Returned as InvalidProvedResponse.

PutSettings

All put operations accept optional PutSettings:

#![allow(unused)]
fn main() {
pub struct PutSettings {
    pub request_settings: RequestSettings,
    pub identity_nonce_stale_time_s: Option<u64>,
    pub user_fee_increase: Option<u16>,
    pub wait_timeout: Option<Duration>,
    pub state_transition_creation_options: Option<...>,
}
}

The most important field is wait_timeout. In production, always set it to avoid hanging indefinitely.

Rules

Do:

  • Use put_to_platform_and_wait_for_response when you need confirmation.
  • Use put_to_platform when you want fire-and-forget semantics.
  • Always set wait_timeout in production.
  • Let the SDK manage nonces -- do not manually set them.
  • Handle Error::AlreadyExists gracefully, especially for identity creation.

Don't:

  • Call broadcast without eventually calling wait_for_response -- you will not know if the transition succeeded.
  • Retry a failed transition without refreshing the nonce -- the old nonce may be consumed.
  • Set user_fee_increase to zero in congested networks -- your transition may be deprioritized.
  • Provide custom entropy unless you need deterministic document IDs for testing.
  • Ignore TimeoutReached errors -- they may indicate network issues that affect subsequent operations.

Identity Keys Deep Dive

Every identity on Dash Platform is controlled by a set of identity public keys. These keys determine what the identity can do: sign state transitions, encrypt messages, transfer credits, vote, or prove masternode ownership. The key system is designed around three axes -- purpose, security level, and key type -- that together define what a key is for, how sensitive it is, and what cryptographic algorithm it uses.

This chapter covers the full key lifecycle: structure, creation, storage, validation, rotation, and the GroveDB tree layout that makes lookups efficient.

Key Structure

An identity public key is represented by IdentityPublicKeyV0:

#![allow(unused)]
fn main() {
pub struct IdentityPublicKeyV0 {
    pub id: KeyID,                            // u32, unique within this identity
    pub purpose: Purpose,                      // what the key is used for
    pub security_level: SecurityLevel,         // how sensitive the key is
    pub key_type: KeyType,                     // cryptographic algorithm
    pub read_only: bool,                       // if true, cannot sign state transitions
    pub data: BinaryData,                      // the public key bytes
    pub disabled_at: Option<TimestampMillis>,  // None = active, Some = disabled
    pub contract_bounds: Option<ContractBounds>, // restrict to specific contract
}
}

Each field serves a specific role:

  • id (KeyID = u32): Sequential identifier assigned at creation. Key IDs are unique within a single identity but not globally. The ID is used to reference the key in state transitions and storage.

  • data (BinaryData): The raw public key bytes. Size depends on key_type: 33 bytes for ECDSA, 48 bytes for BLS, 20 bytes for hash-based types.

  • disabled_at: When set, the key can no longer be used to sign anything. This is a timestamp (milliseconds since epoch), not a boolean, so you know exactly when the key was disabled.

  • read_only: A read-only key can verify signatures but cannot be used to sign new state transitions. This is enforced at validation time.

Purpose

The Purpose enum defines what a key is authorized to do:

PurposeValueDescription
AUTHENTICATION0General-purpose signing. Every identity must have at least one MASTER-level authentication key.
ENCRYPTION1Encrypt data. Cannot sign documents or state transitions.
DECRYPTION2Decrypt data. Cannot sign documents or state transitions.
TRANSFER3Sign credit transfers, withdrawals, and token operations. Required at CRITICAL security level.
SYSTEM4System operations. Cannot sign documents.
VOTING5Cast masternode votes. Cannot sign documents.
OWNER6Prove ownership of a masternode or evonode.

Purposes are grouped by searchability in the storage layer:

  • Searchable: AUTHENTICATION, TRANSFER, VOTING -- these get indexed in the key reference tree so they can be looked up by purpose and security level.
  • Non-searchable: ENCRYPTION, DECRYPTION, SYSTEM, OWNER -- stored but not indexed for search.

The practical effect: if a Platform node needs to find "the TRANSFER key for identity X", it can do a direct tree lookup. But finding "the ENCRYPTION key for identity X" requires fetching all keys and filtering client-side.

Security Level

Security levels form a strict hierarchy:

MASTER (0)  >  CRITICAL (1)  >  HIGH (2)  >  MEDIUM (3)
  strongest                                    weakest

The numeric value is inverted from what you might expect: lower value = stronger security. This matters because many operations check key.security_level().stronger_or_equal_security_than(required_level).

What Security Level Controls

  1. What the key can sign. A data contract can require that documents be signed with at least a certain security level. A key at MEDIUM cannot sign a document that requires HIGH or above.

  2. What operations the key can perform. Some state transitions require specific security levels:

    • Adding/disabling other keys requires MASTER
    • Credit transfers require CRITICAL (enforced via the TRANSFER purpose)
    • Document operations accept CRITICAL down to the level each document type requires (signatureSecurityLevelRequirement, HIGH by default), so MEDIUM only where a type asks for it; a batch holding a token transition needs CRITICAL
  3. Which purposes allow which levels. Not all combinations are valid for externally added keys (i.e., keys added via identity create/update transitions):

    PurposeAllowed Security Levels
    AUTHENTICATIONMASTER, CRITICAL, HIGH, MEDIUM
    ENCRYPTIONMEDIUM only
    DECRYPTIONMEDIUM only
    TRANSFERCRITICAL only
    SYSTEMNot externally addable (platform-managed)
    VOTINGNot externally addable (platform-managed)
    OWNERNot externally addable (platform-managed)

    SYSTEM, VOTING, and OWNER keys are created automatically by the platform (e.g., during masternode registration) and cannot be added through state transitions. Attempting to add a key with one of these purposes will fail validation. Similarly, attempting to create a TRANSFER key at HIGH security level will fail because only CRITICAL is allowed for that purpose.

The Master Key Requirement

Every identity must have exactly one MASTER-level AUTHENTICATION key at creation time. This key is the identity's root of trust -- it can add new keys, disable other keys, and perform any operation. Losing access to the master key means losing the ability to manage the identity's key set.

Key Type

The KeyType enum determines the cryptographic algorithm and key size:

Key TypeValueSizeUniqueDescription
ECDSA_SECP256K1033 bytesYesStandard Bitcoin/Dash curve. Default.
BLS12_381148 bytesYesBLS signatures, used by masternodes.
ECDSA_HASH160220 bytesNoRIPEMD160(SHA256) of an ECDSA public key. Core address type.
BIP13_SCRIPT_HASH320 bytesNoScript hash. Core address type.
EDDSA_25519_HASH160420 bytesNoRIPEMD160(SHA256) of an Ed25519 public key.

Unique vs Non-Unique Keys

This distinction is critical for understanding how keys are stored and enforced:

  • Unique key types (ECDSA_SECP256K1, BLS12_381): The full public key is stored, and Platform enforces that no two identities can register the same public key. This is checked in both the unique and non-unique hash tables during insertion. If identity A registers an ECDSA key, identity B cannot register the same key bytes.

  • Non-unique key types (ECDSA_HASH160, BIP13_SCRIPT_HASH, EDDSA_25519_HASH160): Only a 20-byte hash is stored. Multiple identities can share the same hash. This makes sense for address-based key types where the same Dash address might legitimately be associated with multiple identities (e.g., through asset lock transactions).

Core Address Key Types

ECDSA_HASH160 and BIP13_SCRIPT_HASH are specifically for linking Platform identities to Layer 1 (Core) Dash addresses. They store the same 20-byte hash used in Core addresses, enabling cross-layer identity verification without revealing the full public key.

Contract Bounds

A key can optionally be restricted to one data contract, one document type of a contract, or every member of a contract group:

#![allow(unused)]
fn main() {
pub enum ContractBounds {
    /// Key can only be used within a specific contract
    SingleContract { id: Identifier },

    /// Key can only be used within a specific contract and document type
    SingleContractDocumentType {
        id: Identifier,
        document_type_name: String,
    },

    /// Key can only be used within the members of a contract group (protocol version 14)
    ContractGroup { id: Identifier },
}
}

What the bounds mean depends on the key's purpose:

  • ENCRYPTION and DECRYPTION keys: a hint to clients about which contract the key serves. The bound contract, or document type, must opt in with requiresIdentityEncryptionBoundedKey or requiresIdentityDecryptionBoundedKey, which also fixes the storage rule (unique, multiple, or multiple with a pointer to the latest). These keys cannot be bound to a group.
  • AUTHENTICATION keys (protocol version 14): a restriction consensus enforces. The key may sign only batch transitions, and every member must be inside the bounds: on the bound contract, of the bound document type, or, for a group bound, a member of the group (the whole contract, the document type, or the token). Any contract or group may be bound, the key cannot be MASTER, and a member outside the bounds is a paid failure.

See docs/protocol/contract-bound-authentication-keys.md for the exact rules and errors.

This enables fine-grained delegation: an identity owner can create a key that is only allowed to interact with one specific dApp, or with one project's set of contracts, limiting exposure if that key is compromised.

Budget and Expiry

Bounds limit where a key may act. From protocol version 14 an AUTHENTICATION key below the MASTER level can also be limited in how much it may spend and for how long it may sign. The limits live on the version 1 key, IdentityPublicKeyV1, which is the version 0 key followed by two optional fields:

#![allow(unused)]
fn main() {
pub struct IdentityPublicKeyV1 {
    // ... the IdentityPublicKeyV0 fields, in the same order ...
    pub total_budget: Option<Credits>,        // total credits the key may take from the identity
    pub expires_at: Option<TimestampMillis>,  // block time from which the key no longer signs
}
}

Version 0 keys are untouched: keys already in state decode as before and a key without limits is still written as version 0. Read the limits through IdentityPublicKeyGettersV1 (total_budget(), expires_at(), is_expired_at(time_ms), has_limits()), which answers None for a version 0 key, and turn a key into a limited one with IdentityPublicKey::with_limits.

  • total_budget caps everything state transitions signed with the key take from the identity: fees (net of the storage refunds the same transition returns) and credits moved out, such as a document purchase price. What is left is tracked by Drive next to the key and only goes down. Before a transition runs, everything but the metered processing fee must fit in what is left; that processing fee may take the key over its budget once, after which the key can no longer sign. This is the rule identity balances follow (storage must be covered, processing may leave a debt), applied to the key.
  • expires_at is compared with the time of the block the transition executes in. The key signs at expires_at - 1 and not at expires_at. A key cannot be registered already expired.

The limits are part of the signable bytes of the transition that registers the key and cannot be changed afterwards. An identity created from the shielded pool is the exception: it has no identity signature and its sighash does not cover the limits, so a key that carries a budget or an expiry is refused there and has to be added with an identity update (a version 1 key without limits is accepted). Refusals for a spent, exceeded or expired key leave the transition unpaid, like an identity that cannot afford its fee. Key Budgets and Expiry explains the design: the budget rule, where each check runs in the validation pipeline, and how Drive keeps the running total. The rules and error codes as a reference are in docs/protocol/authentication-key-limits.md.

Storage in GroveDB

Identity keys are stored across multiple trees in GroveDB for efficient access patterns.

Identity-Level Trees

Each identity has its own subtree under the root Identities tree:

Identities [RootTree::Identities]
└── {identity_id (32 bytes)}
    ├── IdentityTreeKeys [128]
    │   └── {key_id (varint)} → serialized IdentityPublicKey
    │
    ├── IdentityTreeKeyReferences [160]
    │   ├── AUTHENTICATION [0]
    │   │   ├── MASTER [0]
    │   │   │   └── {key_id} → reference to IdentityTreeKeys
    │   │   ├── CRITICAL [1]
    │   │   │   └── ...
    │   │   ├── HIGH [2]
    │   │   │   └── ...
    │   │   └── MEDIUM [3]    ← pre-created at identity creation
    │   │       └── ...
    │   ├── TRANSFER [3]
    │   │   └── {key_id} → reference
    │   └── VOTING [5]
    │       └── {key_id} → reference
    │
    ├── IdentityTreeRevision [192]
    ├── IdentityTreeNonce [64]
    ├── IdentityContractInfo [32]
    └── IdentityTreeKeyBudgets [224]    ← created with the first budgeted key
        └── {key_id (varint)} → remaining budget (u64, 8 bytes big endian)

IdentityTreeKeys stores the actual serialized key data, keyed by the key ID encoded as a varint.

IdentityTreeKeyReferences provides a searchable index organized by purpose and (for AUTHENTICATION) security level. Each entry is a GroveDB reference pointing back to the actual key in IdentityTreeKeys.

The MEDIUM security level subtree under AUTHENTICATION is pre-created during identity initialization, even if no MEDIUM keys exist yet. Other security level subtrees are created on-demand when a key with that level is first added.

IdentityTreeKeyBudgets holds what is left of the budget of each budgeted key. The key itself is immutable, so the running value lives here. It is fixed width so that spending from a budget replaces the value without changing what is stored, which lets the deduction be applied outside of the fee, like the balance change it accompanies.

Global Key Hash Tables

Two root-level trees provide reverse lookups from key hashes to identity IDs:

UniquePublicKeyHashesToIdentities [24]
└── {key_hash (20 bytes)} → identity_id (32 bytes)

NonUniquePublicKeyKeyHashesToIdentities [8]
└── {key_hash (20 bytes)}
    └── {identity_id (32 bytes)} → empty item

The unique table is a flat mapping: one hash to one identity. Insertion fails if the hash already exists in either table.

The non-unique table uses a nested structure: each key hash has a subtree containing identity IDs as keys. This allows multiple identities to share the same key hash.

Key Hash Computation

All key types are hashed to 20 bytes for storage in the hash tables:

  • ECDSA_SECP256K1 (33 bytes): RIPEMD160(SHA256(pubkey))
  • BLS12_381 (48 bytes): RIPEMD160(SHA256(pubkey))
  • ECDSA_HASH160 (20 bytes): stored as-is (already a hash)
  • BIP13_SCRIPT_HASH (20 bytes): stored as-is
  • EDDSA_25519_HASH160 (20 bytes): stored as-is

Key Lifecycle

Creation

Keys are added to an identity either at identity creation or via an IdentityUpdate state transition.

At Identity Creation:

  • The state transition includes IdentityPublicKeyInCreation objects
  • Validation enforces exactly one MASTER-level AUTHENTICATION key
  • Each key also carries a signature field (proving the creator holds the private key)
  • After validation, keys are converted to IdentityPublicKey with disabled_at = None

Via IdentityUpdate:

  • The add_public_keys field carries new IdentityPublicKeyInCreation objects
  • Multiple keys can be added in a single transition
  • The transition must be signed by a key with sufficient security level
  • New key IDs must not collide with existing keys on the identity

The insertion process differs by key type:

  1. Unique keys: Check both hash tables for conflicts, insert into UniquePublicKeyHashesToIdentities, insert key data, create references.
  2. Non-unique keys: Create subtree under hash if needed, insert identity ID, insert key data, create references.

Disabling

Keys are disabled (not deleted) via the disable_public_keys field of IdentityUpdate:

#![allow(unused)]
fn main() {
// IdentityUpdateTransition fields (simplified)
pub add_public_keys: Vec<IdentityPublicKeyInCreation>,
pub disable_public_keys: Vec<KeyID>,
}

When a key is disabled:

  1. The key is fetched from storage.
  2. disabled_at is set to the current block timestamp (milliseconds).
  3. The serialized key is replaced in IdentityTreeKeys.
  4. Key references in IdentityTreeKeyReferences are refreshed.

A disabled key:

  • Cannot be used to sign any state transition (checked during signature verification)
  • Remains in storage (can still be read)
  • Can be re-enabled in the future

Re-enabling

Keys can be re-enabled by clearing the disabled_at field:

  1. The key is fetched from storage.
  2. disabled_at is set to None.
  3. The serialized key is replaced.
  4. References are refreshed.

Masternode Keys

Masternode identities have a special rule: all their keys are registered as non-unique, regardless of key type. This allows the same BLS key to be used across multiple masternode identities (e.g., during key rotation or when the same operator runs multiple masternodes).

Signing and Verification

When a state transition is signed:

  1. The transition specifies which key ID it was signed with.
  2. The key is looked up on the signing identity.
  3. Validation checks:
    • The key exists and is not disabled (disabled_at must be None)
    • The key's purpose allows this type of state transition
    • The key's security level meets the minimum required
    • If the key has contract_bounds, the transition targets the bound contract
    • If the key is read_only, it cannot sign
    • If the key has a total_budget, some of it is left Two more checks on a limited key need the block time and the fee, so they run with fee validation instead: the key has not expired, and what the transition requires from the budget fits in what is left.
  4. The signature is verified using the appropriate algorithm:
    • ECDSA_SECP256K1: standard secp256k1 signature verification
    • BLS12_381: BLS signature verification
    • ECDSA_HASH160: ECDSA verification (key data is a hash, so verification uses the hash comparison path)

Validation Rules Summary

At identity creation:

  • Exactly 1 MASTER-level AUTHENTICATION key required
  • No duplicate key IDs in the transition
  • No duplicate key data for unique key types
  • Key count must not exceed max_public_keys_in_creation (platform config)
  • Each key's purpose/security level combination must be in the allowed set

At key addition (IdentityUpdate):

  • New key IDs must not conflict with existing keys
  • Unique key hashes must not exist in either the unique or non-unique global tables
  • The signing key must have sufficient security level to add keys
  • All purpose/security level constraints apply
  • A budget or an expiry is only allowed on an AUTHENTICATION key below MASTER; the budget is not zero and the expiry is after the block time

At signing time:

  • Key must not be disabled
  • Key purpose must match the operation
  • Key security level must be >= the required level
  • Contract bounds must match (if set)
  • Key must not be read-only
  • Key must not be expired, and its budget must cover the transition (if set)

Querying Keys

Find identity by public key hash:

#![allow(unused)]
fn main() {
// Returns the identity ID that owns this unique public key
let identity_id = drive.fetch_identity_id_by_unique_public_key_hash(
    key_hash, transaction, platform_version
)?;
}

Find all identities sharing a non-unique key hash:

#![allow(unused)]
fn main() {
// Returns all identity IDs registered under this non-unique key hash
let identity_ids = drive.fetch_identity_ids_by_non_unique_public_key_hash(
    key_hash, transaction, platform_version
)?;
}

Fetch all keys for an identity:

#![allow(unused)]
fn main() {
let keys = identity.public_keys();  // BTreeMap<KeyID, IdentityPublicKey>
}

Fetch by purpose and security level: Uses the IdentityTreeKeyReferences tree to efficiently look up keys without scanning all keys on the identity.

Design Rationale

Why separate purpose and security level? Purpose defines what a key can do; security level defines how sensitive it is. An AUTHENTICATION key at MEDIUM can sign low-sensitivity documents. An AUTHENTICATION key at MASTER can manage the identity itself. This separation lets identities create keys with exactly the right capabilities -- not too much, not too little.

Why disable instead of delete? Disabled keys remain in storage so that historical signatures can still be verified. If a key were deleted, past state transitions signed by that key would become unverifiable.

Why unique vs non-unique hash tables? Full public keys (ECDSA, BLS) must be globally unique to prevent impersonation. But hash-based keys (ECDSA_HASH160, BIP13_SCRIPT_HASH) represent Dash addresses that may legitimately appear in multiple identities -- for example, when the same address is used in multiple asset lock transactions.

Why pre-create the MEDIUM AUTHENTICATION subtree? This is the most commonly used security level for document signing. Pre-creating its tree at identity creation avoids the cost of creating it on the first document submission.

Why contract bounds? They enable the principle of least privilege. An identity can create a key specifically for interacting with one dApp. If that key is compromised, the damage is limited to that single contract -- the attacker cannot use it to transfer credits or interact with other contracts.

BLAST Sync

Blockchain Layered Address Sync Tree (BLAST) is a privacy-preserving synchronization algorithm used by the Dash Platform SDK. It allows wallets to discover which of their keys exist in a server-side Merkle tree without revealing the specific keys being queried.

BLAST is used for two distinct sync tasks:

  • Address balance sync: Discovering which platform addresses have balances and what those balances are.
  • Nullifier sync: Checking which nullifiers have been spent in the shielded pool.

Both follow the same trunk/branch tree-scan pattern, extracted into a shared generic algorithm.

Why BLAST Preserves Privacy

The naive approach to syncing — querying each address individually — leaks the wallet's full key set to the server. Even batching all keys into a single request reveals the exact set. The server learns which addresses belong to the same wallet, enabling transaction graph analysis and balance tracking.

BLAST solves this by never querying individual keys. Instead, the wallet queries subtrees of the Merkle tree. Each query returns a chunk of the tree containing the target key alongside many other unrelated keys. The server sees a request for a region of the tree but cannot determine which specific key within that region the wallet cares about.

The privacy guarantee comes from three mechanisms:

  1. Trunk query hides intent entirely. The initial trunk query fetches the top levels of the entire tree — the same request regardless of which keys the wallet holds. The server learns nothing from this query.

  2. Branch queries request subtrees, not keys. When the wallet needs to resolve a key that traced to a truncated leaf, it requests the subtree rooted at that leaf. The response contains all keys in that subtree, not just the target. The server knows the wallet is interested in some key in the subtree, but not which one.

  3. Privacy adjustment enlarges small subtrees. If a leaf's subtree contains fewer than min_privacy_count elements (default: 32), the wallet queries an ancestor subtree instead — one that contains at least 32 keys. This prevents the server from narrowing down the target when a subtree is too small. For example, if the target key's leaf subtree has only 3 elements, querying it would reveal the target with ~33% probability. By querying an ancestor with 50+ elements, the probability drops to ~2%.

  4. Incremental catch-up queries block ranges, not addresses. The recent and compacted change queries ask "what changed since block height X?" — they return all address balance changes for all addresses in those blocks, not just the wallet's addresses. The server sees a generic block-range query identical to what any other wallet would send. The wallet filters the results client-side, discarding changes for addresses it doesn't own.

The result: for each sync, the server sees a trunk query (identical for all wallets) plus a small number of branch queries for subtrees containing 32+ keys each, plus block-range queries that reveal no address-specific information. It cannot distinguish which specific keys the wallet owns, how many keys it has, or whether two branch queries target different keys or are privacy-adjusted queries for the same key.

Performance

BLAST sync scales logarithmically with the total number of addresses in the platform state tree. A wallet with 20 addresses performs roughly the same number of queries whether the tree contains 1,000 or 1,000,000,000 addresses.

Estimated Performance for 20 Wallet Addresses

The total query count is Q = 1 (trunk) + 2k × iterations + 1 (recent) where each key requires 2 branch queries (left + right child) per iteration. See the Appendix: Query Count Analysis for the full derivation.

xychart-beta
    title "Total Queries (full scan, 20 wallet addresses)"
    x-axis "Total addresses in tree" ["1K", "10K", "100K", "1M", "10M", "100M", "1B"]
    y-axis "Queries" 0 --> 100
    bar [42, 42, 42, 62, 72, 82, 82]
xychart-beta
    title "Estimated Sync Time (10 concurrent, ~330ms/round)"
    x-axis "Total addresses in tree" ["1K", "10K", "100K", "1M", "10M", "100M", "1B"]
    y-axis "Seconds" 0 --> 7
    line [2.0, 2.0, 2.0, 4.0, 4.6, 5.3, 5.3]
Total addressesTree depthd_trunkd_branchBranch itersBranch queriesTotal queriesParallel roundsEst. sync time
1,0001099140426~2.0s
10,0001499140426~2.0s
100,0001799140426~2.0s
1,000,0002099260628~4.0s
10,000,0002399270729~4.6s
100,000,00027993808210~5.3s
1,000,000,00030993808210~5.3s

How to read the table: At 100K addresses (depth 17), the trunk covers 9 levels and one branch iteration covers up to 9 more levels (total 18 > 17). All 20 keys resolve in a single round of 40 branch queries. At 1M (depth 20), trunk + one iteration covers 18 levels — 2 levels short, so ~50% of keys need a second round, adding ~20 queries.

Assumptions: d_trunk = 9, d_branch = 9 (platform version max_depth), P = 32, 2 branch queries per key per iteration (left + right child), 10 concurrent branch queries, ~330ms per parallel round (measured empirically: ~4s for 1M addresses). Sync time = rounds × 330ms where rounds = 1 (trunk) + ⌈branch_queries / 10⌉ + 1 (recent). Note: iteration transitions add overhead, so multi-iteration syncs are slightly slower than the formula suggests.

Incremental sync (subsequent 15-second re-syncs): 1 query, ~330ms — only the recent changes query, no trunk/branch. If compaction occurred: +1 compacted query.

Key Insight

Going from 1,000 to 1,000,000,000 addresses — a million-fold increase — adds only ~40 queries and ~2.6 seconds. The query count grows as O(k · ⌈log₂(N) / d_branch⌉) where each step up in tree depth beyond d_trunk + d_branch adds one iteration of ~20 queries (diminishing as keys resolve). With 10-way parallelism, each additional iteration costs only ~2 extra parallel rounds (~660ms).

Full Sync Flowchart

flowchart TD
    Start["<b>sync_address_balances</b><br/>called by wallet"] --> CheckTS{"last_sync_timestamp<br/>provided?"}

    CheckTS -->|"No / zero"| FullScan1["<b>FULL SCAN</b><br/>First sync ever"]
    CheckTS -->|"Yes"| CheckElapsed{"Elapsed time ><br/>full_rescan_after?<br/><i>default: 7 days</i>"}

    CheckElapsed -->|"Yes - stale"| FullScan2["<b>FULL SCAN</b><br/>Data too old"]
    CheckElapsed -->|"No - recent"| Incremental["<b>INCREMENTAL ONLY</b><br/>Seed from current_balances<br/>start_height = last_sync_height"]

    FullScan1 --> Trunk
    FullScan2 --> Trunk

    Trunk["<b>Trunk Query</b><br/>Fetch top N levels of Merk tree<br/>Verify proof against quorum-signed root<br/>Sets checkpoint_height from metadata"] --> Classify["<b>Classify Keys</b><br/>BST traversal for each target address<br/><i>Found</i> → record balance<br/><i>Traced to leaf</i> → add to tracker<br/><i>Absent</i> → proven not in tree"]
    Classify --> Privacy["<b>Privacy Adjust</b><br/>Leaves with count < 32:<br/>query ancestor subtree instead<br/>Dedup: multiple keys → one query"]
    Privacy --> Branch["<b>Branch Queries</b><br/>Parallel: up to 10 concurrent<br/>Iterative: up to 50 rounds<br/>Each round resolves deeper leaves<br/>Gap limit extension after each round"]
    Branch -->|"catch_up_from =<br/>checkpoint_height"| Recent

    Incremental --> Recent

    Recent["<b>Query Recent</b><br/>Single query - no pagination needed<br/>Recent tree max 64 blocks < 100 limit<br/><b>Hold results</b> - do not apply yet<br/>Record observed_tip from metadata"] --> CompCheck{"<b>Compaction Check</b><br/>Has last_known_recent_block?"}

    CompCheck -->|"Yes: use RangeAfter<br/>key_exists_as_boundary<br/>on last_known_recent_block"| BoundaryCheck{"Boundary exists<br/>in proof?"}
    CompCheck -->|"No: first sync or<br/>empty recent tree"| Compacted

    BoundaryCheck -->|"Yes: not compacted"| ApplyRecent
    BoundaryCheck -->|"No: compacted away"| Compacted

    Compacted["<b>Query Compacted</b><br/>Paginated: 25 ranges per batch<br/>Aggregated block ranges with<br/>SetCredits or AddToCreditsOperations<br/>Apply immediately, advance height"] --> ApplyRecent

    ApplyRecent["<b>Apply Held Recent</b><br/>Process per-block entries from recent query<br/>Update balances, call on_address_found<br/>Advance current_height per block"] --> Finalize

    Finalize["<b>Finalize</b><br/>new_sync_height = max of current and observed_tip<br/>new_sync_timestamp = latest metadata time<br/>checkpoint_height = trunk query height<br/><i>Caller persists for next sync</i>"]

    style FullScan1 fill:#c53030,color:#fff
    style FullScan2 fill:#c53030,color:#fff
    style Incremental fill:#2b6cb0,color:#fff
    style Trunk fill:#1a365d,color:#e2e8f0
    style Classify fill:#1a365d,color:#e2e8f0
    style Privacy fill:#1a365d,color:#e2e8f0
    style Branch fill:#1a365d,color:#e2e8f0
    style Recent fill:#1a3a2a,color:#e2e8f0
    style CompCheck fill:#744210,color:#fff
    style Compacted fill:#c05621,color:#fff
    style ApplyRecent fill:#1a3a2a,color:#e2e8f0
    style Finalize fill:#2d3748,color:#e2e8f0

The Problem

A wallet holds a set of keys (addresses or nullifiers) and needs to learn which ones exist in a Merkle tree stored by Platform nodes. The naive approach -- querying each key individually -- leaks the wallet's full key set to the server. Even batching the keys into a single request reveals the exact set.

BLAST solves this by querying subtrees of the Merkle tree rather than individual keys. The server returns a chunk of the tree that contains the target key along with many other keys, making it impossible for the server to determine which specific key the wallet cares about.

Algorithm Overview

The sync has two phases: a tree scan for bulk discovery, and incremental catch-up for staying current between scans.

Phase 1: Tree Scan (Trunk/Branch)

The Underlying Data Structure

Platform stores addresses and nullifiers in a Merk tree -- a balanced binary search tree (BST) where each node is keyed and ordered. Every internal node has a left child (keys less than this node) and a right child (keys greater than this node). Each node also carries a Merkle hash of its subtree, making the entire structure cryptographically verifiable.

The trunk query returns a partial view of this BST: the top N levels are fully expanded (you can see the actual keys and values), while deeper subtrees are truncated to hash placeholders. The boundary between "expanded" and "truncated" defines the leaf nodes of the trunk result.

                          ┌──────┐
                          │  30  │   ← Root node (key=30)
                          └──┬───┘
                      ┌──────┴──────┐
                      ▼             ▼
                   ┌──────┐     ┌──────┐
                   │  15  │     │  45  │   ← Internal nodes (expanded)
                   └──┬───┘     └──┬───┘
                 ┌────┴────┐  ┌────┴────┐
                 ▼         ▼  ▼         ▼
              ┌─────┐  ┌─────┐ ┌─────┐  ┌─────┐
              │  7  │  │ 22  │ │ 38  │  │ 55  │  ← Leaf nodes
              │▓▓▓▓▓│  │▓▓▓▓▓│ │▓▓▓▓▓│  │▓▓▓▓▓│     (children are
              └─────┘  └─────┘ └─────┘  └─────┘      hash placeholders)

              ▓▓▓ = truncated subtree (only hash known, not contents)

The trunk result contains three key pieces of data:

  • elements: A BTreeMap<Vec<u8>, Element> of key-value pairs at expanded nodes. These are fully resolved -- the wallet can read their values directly.
  • leaf_keys: A BTreeMap<Vec<u8>, LeafInfo> of nodes at the truncation boundary. Each LeafInfo has a hash (for verifying subsequent branch queries) and an optional count (number of elements in the truncated subtree).
  • tree: The reconstructed BST structure from the proof, used for key tracing.

Step 1: The Trunk Query

The wallet sends a single trunk query to a Platform node. The request specifies a max_depth (how many levels of the BST to expand). The server returns the trunk elements, the leaf boundary information, and a Merkle proof covering the entire result.

The proof is verified against the quorum-signed root hash, ensuring the server cannot lie about what the tree contains.

#![allow(unused)]
fn main() {
let (trunk_result, metadata) =
    PlatformAddressTrunkState::fetch_with_metadata(sdk, (), Some(settings)).await?;
}

Step 2: Classifying Target Keys via BST Traversal

After receiving the trunk, the wallet classifies each of its target keys by traversing the BST structure. The trace_key_to_leaf method performs a standard binary search:

  1. Start at the root node.
  2. Compare the target key against the current node's key.
  3. If equal: the key is found in the trunk elements.
  4. If less: follow the left child.
  5. If greater: follow the right child.
  6. If the current node is a leaf (its children are hash placeholders): the target key is somewhere in this leaf's truncated subtree, but we can't resolve it yet.
  7. If there is no child to follow: the key is proven absent.

This produces exactly three outcomes for each target key:

OutcomeWhat it meansAction
FoundKey exists in trunk elementsRecord the value (balance, spent status)
Traced to leafKey is in a truncated subtreeAdd to KeyLeafTracker for branch querying
AbsentNo path exists in the BSTKey is cryptographically proven to not exist
#![allow(unused)]
fn main() {
for key in target_keys {
    if trunk_result.elements.contains_key(&key) {
        // Found directly in trunk -- record it
        result.found.insert(key);
    } else if let Some((leaf_key, info)) = trunk_result.trace_key_to_leaf(&key) {
        // Traces to a leaf subtree -- need a branch query
        tracker.add_key(key, leaf_key, info);
    } else {
        // Proven absent from the tree
        result.absent.insert(key);
    }
}
}

A concrete example: suppose the wallet is looking for key 20 in the tree above. The BST traversal goes: root 30 (20 < 30, go left) -> node 15 (20 > 15, go right) -> leaf 22 (children are hash placeholders). Key 20 traces to leaf 22 because it would be in leaf 22's left subtree. The wallet now knows it needs to query leaf 22's subtree to determine whether key 20 actually exists.

Note that each target key traces to exactly one leaf -- the BST path is deterministic. Multiple target keys may trace to the same leaf if they are close together in the key space.

Step 3: Privacy Adjustment

Before querying leaf subtrees, the algorithm applies privacy adjustment to prevent the server from learning which specific keys the wallet cares about.

Each leaf in the trunk result has an optional count -- the number of elements in its truncated subtree. If this count is small (below min_privacy_count, default 32), then querying that specific leaf reveals too much: the server knows the wallet is interested in one of only a few keys.

The fix is to query an ancestor higher in the tree that has enough elements to provide cover:

    Suppose leaf "22" has count=5 (too small for privacy).
    Its parent "15" has count=50 (enough).

    Instead of asking: "give me the subtree rooted at 22"
    The wallet asks:   "give me the subtree rooted at 15"

    Now the server sees a query for a 50-element subtree and cannot
    tell whether the wallet wants key 20 (in 22's subtree) or
    key 10 (in 7's subtree) or any other key under 15.

The get_ancestor method walks up the BST path from the leaf to the root, stopping at the first ancestor whose count exceeds min_privacy_count. It never returns the root itself (that would be equivalent to re-fetching the entire trunk). If no ancestor has enough count, it falls back to the node one level below the root.

The query depth is adjusted when using an ancestor: since the ancestor is higher in the tree, its subtree is deeper, so the depth parameter is reduced by the number of levels climbed.

Deduplication ensures that if multiple target keys expand to the same ancestor, only one branch query is sent.

Step 4: Iterative Branch Queries

For each leaf (or privacy-adjusted ancestor) with unresolved keys, the wallet sends a branch query specifying:

  • The leaf's key (identifies which subtree to expand)
  • The query depth (how many levels to expand)
  • The expected root hash (the leaf's hash from the trunk, used for verification)
  • The checkpoint height (ensures the branch matches the same tree snapshot as the trunk)

The server returns a GroveBranchQueryResult with the same structure as the trunk: expanded elements, new leaf keys at the next truncation boundary, and a Merk proof. The wallet verifies the proof against the expected hash from the parent query.

Each target key in the queried subtree is classified again:

  • Found in the branch elements -- resolved.
  • Traced to a deeper leaf -- the key is in an even deeper truncated subtree. The KeyLeafTracker is updated to point to the new, deeper leaf.
  • Absent -- proven to not exist within this subtree.
    Iteration 1: Trunk query
    ┌────────────────────────────────────────────────────┐
    │  Key 20 traces to leaf 22                          │
    │  Key 41 traces to leaf 38                          │
    └────────────────────────────────────────────────────┘
                            │
                            ▼
    Iteration 2: Branch queries for leaves 22 and 38
    ┌────────────────────────────────────────────────────┐
    │  Key 20: found in leaf 22's subtree → RESOLVED     │
    │  Key 41: traces to deeper leaf 40 → CONTINUE       │
    └────────────────────────────────────────────────────┘
                            │
                            ▼
    Iteration 3: Branch query for leaf 40
    ┌────────────────────────────────────────────────────┐
    │  Key 41: proven absent in leaf 40's subtree → DONE │
    └────────────────────────────────────────────────────┘

Branch queries run in parallel using FuturesUnordered with configurable concurrency (max_concurrent_requests, default: 10). The iteration loop continues until all keys are resolved or max_iterations (default: 50) is reached. In practice, most keys resolve within 2-3 iterations because each branch query expands several levels of the tree.

Phase 2: Incremental Catch-Up

After the tree scan produces a snapshot at some checkpoint height, the wallet needs to catch up to the chain tip. The incremental phase also runs on its own for frequent re-syncs (when the elapsed time since the last sync is within full_rescan_after_time_s, default 7 days).

Platform stores per-block address balance changes in a "recent" tree. Every 64 blocks (or 2048 address entries), these per-block entries are compacted into aggregated range entries covering the merged block span. The compacted entries have an expiration TTL and are eventually cleaned up. This means:

  • The recent tree never has more than 64 block entries at any time.
  • The compacted tree contains historical aggregated ranges for blocks that were compacted away from the recent tree.
  checkpoint_height                              chain_tip
        │                                            │
        ▼                                            ▼
  ──────┬────────────────────────────┬───────────────┤
        │   Compacted ranges         │ Recent blocks │
        │   (aggregated, ≤25/page)   │ (per-block,   │
        │   Only exist after         │  always ≤64)  │
        │   compaction events        │               │
        └────────────────────────────┴───────────────┘

Step 1: Query Recent First

The wallet always queries recent changes first:

#![allow(unused)]
fn main() {
GetRecentAddressBalanceChanges { start_height, prove: true }
}

Since the recent tree can never exceed 64 entries and the server limit is 100 per request, one query always covers the entire uncompacted range. No pagination is needed for recent changes.

The response contains Vec<BlockAddressBalanceChanges>, where each entry has a block_height and a map of PlatformAddress → CreditOperation (either SetCredits or AddToCredits).

Important: The results are held but not applied yet. They may need to be applied after compacted data if compaction occurred.

Step 2: Detect Compaction via Proof

The wallet tracks last_known_recent_block — the highest block height from the entries returned by the previous recent query. This is a block that was actually present in the recent tree.

When last_known_recent_block > 0, the recent query uses exclusive start (RangeAfter) with that height. This causes the height to appear as a boundary node in the proof. The wallet then uses key_exists_as_boundary to check whether the block is still in the recent tree:

#![allow(unused)]
fn main() {
let cursor_exists = Drive::verify_key_exists_as_boundary(
    proof,
    &[SavedBlockTransactions, ADDRESS_BALANCES_KEY],
    &last_known_recent_block.to_be_bytes(),
    platform_version,
)?;
}
cursor_existsMeaningAction
trueBlock still in recent tree — no compaction happenedApply held recent results directly (Step 4)
falseBlock was compacted awayQuery compacted first (Step 3), then apply recent

When last_known_recent_block == 0: This occurs on the first sync after a tree scan, or when the recent tree had no entries (no platform address activity on chain). In this case, the query falls back to inclusive start (RangeFrom) and the compacted phase always runs. This is the safe default — the compacted query returns empty quickly if there is no compacted data.

Step 3: Query Compacted (only if needed)

Only runs when compaction is detected or on the first catch-up after a tree scan:

#![allow(unused)]
fn main() {
GetRecentCompactedAddressBalanceChanges { start_block_height, prove: true }
}

Returns Vec<CompactedBlockAddressBalanceChanges>, where each entry has:

  • start_block_height and end_block_height (the compacted range)
  • changes: BTreeMap<PlatformAddress, BlockAwareCreditOperation>
    • SetCredits(u64) — absolute balance
    • AddToCreditsOperations(Vec<(height, credits)>) — deltas by height

The server returns up to 25 ranges per request. If the response contains exactly 25 entries, the client paginates by setting start_block_height = last_end_height + 1 and querying again.

Compacted changes are applied to the result immediately, updating balances and advancing the pagination cursor.

Step 4: Apply Held Recent Results

After compacted data (if any) is applied, the held recent results from Step 1 are processed:

  • For each BlockAddressBalanceChanges entry, for each address change, if the address matches one of the wallet's target addresses, update the balance and call provider.on_address_found().
  • Advance current_height to block_height + 1 for each processed entry.

Finalization

#![allow(unused)]
fn main() {
result.new_sync_height = max(current_height, observed_tip_height);
result.new_sync_timestamp = latest_metadata.time_ms / 1000;
}

The caller persists new_sync_height and new_sync_timestamp for the next sync call. On the next incremental sync, new_sync_height becomes start_height for the recent query, and new_sync_timestamp determines whether a full tree rescan is needed.

The TrunkBranchSyncOps Trait

The shared algorithm is parameterized by the TrunkBranchSyncOps trait, defined in packages/rs-sdk/src/platform/trunk_branch_sync/mod.rs. Each sync module implements this trait to plug in its specific query construction, result processing, and depth limits.

#![allow(unused)]
fn main() {
pub trait TrunkBranchSyncOps {
    /// Module-specific mutable state carried through the scan.
    type Context<'a>: Send where Self: 'a;

    /// Immutable config for parallel branch queries (cloned into each task).
    type BranchQueryConfig: Clone + Send + Sync + 'static;

    // Trunk
    async fn execute_trunk_query(sdk, settings, context)
        -> Result<(GroveTrunkQueryResult, u64, u64), Error>;
    fn process_trunk_result(trunk_result, context, tracker) -> Result<(), Error>;

    // Branch
    fn branch_query_config(context) -> Self::BranchQueryConfig;
    async fn execute_single_branch_query(sdk, config, key, depth, ...)
        -> Result<GroveBranchQueryResult, Error>;
    fn process_branch_result(branch_result, leaf_key, context, tracker)
        -> Result<(), Error>;

    // Limits and hooks
    fn depth_limits(platform_version) -> (u8, u8);
    fn after_branch_iteration(trunk_result, context, tracker) { }
    fn on_branch_query(context);
    fn on_branch_failure(context);
    fn on_elements_seen(context, count);
    fn on_iteration(context, iteration);
    fn set_checkpoint_height(context, height);
}
}

The two associated types deserve attention:

  • Context<'a> is a GAT (generic associated type) that carries mutable state through the algorithm. For nullifiers, this holds the input keys and result sets. For addresses, it holds the address provider, key-to-index mapping, and result.

  • BranchQueryConfig holds immutable parameters needed to construct branch queries that must be sent to async tasks. For nullifiers, this is (pool_type, pool_identifier). For addresses, it is () since no extra parameters are needed.

The after_branch_iteration hook allows the address sync module to implement gap-limit behavior: after each branch iteration, it checks if the provider has extended its pending address list and adds newly pending keys to the tracker.

KeyLeafTracker

The KeyLeafTracker (in trunk_branch_sync/tracker.rs) maintains the mapping between target keys and the leaf subtrees they reside in. It supports:

  • Adding keys: When a key traces to a leaf during trunk processing
  • Updating keys: When a branch query reveals the key is in a deeper subtree
  • Removing keys: When a key is found or proven absent
  • Reference counting: Multiple target keys can map to the same leaf; the leaf stays active until all its keys are resolved
#![allow(unused)]
fn main() {
let mut tracker = KeyLeafTracker::new();

// After trunk query: key traces to leaf subtree
tracker.add_key(target_key, leaf_boundary_key, leaf_info);

// After branch query: key found in subtree
tracker.key_found(&target_key);

// After branch query: key in even deeper subtree
tracker.update_leaf(&target_key, deeper_leaf_key, deeper_info);

// Check what still needs querying
let active = tracker.active_leaves(); // leaves with unresolved keys
let remaining = tracker.remaining_count();
}

Privacy-Adjusted Leaves (Detail)

The get_privacy_adjusted_leaves function (in trunk_branch_sync/mod.rs) implements the privacy adjustment described in Phase 1, Step 3. The full logic for each active leaf:

  1. Calculate the query depth from the leaf's element count using calculate_max_tree_depth_from_count(count), clamped to platform-version bounds [min_query_depth, max_query_depth].
  2. If count >= min_privacy_count: query this leaf directly at the calculated depth.
  3. If count < min_privacy_count: call trunk_result.get_ancestor(&leaf_key, min_privacy_count) to find a higher node. Reduce depth by levels_up (the number of tree levels climbed) so the total subtree size returned stays reasonable.
  4. If no suitable ancestor exists (rare -- means the entire tree is small): query the leaf anyway, accepting reduced privacy.
  5. Deduplicate: if two target keys expand to the same ancestor, only one branch query is emitted (tracked via a BTreeSet<LeafBoundaryKey>).

Concrete Implementations

Address Balance Sync

The address sync module (platform/address_sync/) implements TrunkBranchSyncOps as AddressOps<P> where P: AddressProvider.

The AddressProvider trait is implemented by wallets to supply:

  • The list of pending addresses to check
  • Callbacks when addresses are found or proven absent
  • Gap-limit extension (generating new addresses when prior ones are found)
  • Current balances for incremental-only mode
#![allow(unused)]
fn main() {
// First sync -- full tree scan + incremental catch-up
let result = sdk.sync_address_balances(&mut wallet, None, None).await?;

// Store for next call
let height = result.new_sync_height;
let timestamp = result.new_sync_timestamp;

// Subsequent sync -- incremental only if within 7-day threshold
let result = sdk.sync_address_balances(&mut wallet, None, Some(timestamp)).await?;
}

Address balance sync uses ItemWithSumItem GroveDB elements where the item value contains the nonce (4 bytes big-endian) and the sum value contains the credit balance.

Nullifier Sync

The nullifier sync module (platform/nullifier_sync/) implements TrunkBranchSyncOps as NullifierOps.

Nullifier sync differs from address sync in several ways:

  • Target keys are fixed 32-byte arrays ([u8; 32])
  • Branch queries carry extra config: (pool_type, pool_identifier) to identify the shielded pool
  • No gap-limit behavior (the after_branch_iteration hook is not overridden)
  • Branch query failures are tracked in metrics
#![allow(unused)]
fn main() {
let nullifiers: Vec<[u8; 32]> = vec![/* ... */];

// First sync -- full tree scan + incremental catch-up
let result = sdk.sync_nullifiers(&nullifiers, None, None, None).await?;

// Store for next call
let height = result.new_sync_height;
let timestamp = result.new_sync_timestamp;

// Subsequent sync -- incremental only if within 7-day threshold
let result = sdk.sync_nullifiers(&nullifiers, None, Some(height), Some(timestamp)).await?;
}

Found nullifiers indicate spent notes; absent nullifiers indicate unspent notes.

Sync Mode Decision

Both sync modules use the same logic to decide between full scan and incremental-only:

last_sync_timestampElapsed timeMode
None--Full tree scan + catch-up
Some(ts)< full_rescan_after_time_sIncremental only
Some(ts)>= full_rescan_after_time_sFull tree scan + catch-up

The default full_rescan_after_time_s is 604800 (7 days). Setting it to 0 forces a full tree scan on every call.

Configuration

Both modules expose configuration structs with sensible defaults:

ParameterDefaultDescription
min_privacy_count32Minimum elements in a queried subtree
max_concurrent_requests10Parallel branch queries
max_iterations50Safety limit for branch iteration depth
full_rescan_after_time_s604800Seconds before forcing a full rescan

Module Structure

packages/rs-sdk/src/platform/
├── trunk_branch_sync/
│   ├── mod.rs        # TrunkBranchSyncOps trait, run_full_tree_scan(),
│   │                 #   get_privacy_adjusted_leaves(), parallel execution
│   └── tracker.rs    # KeyLeafTracker with reference counting
├── address_sync/
│   ├── mod.rs        # AddressOps<P> impl, sync_address_balances(),
│   │                 #   incremental_catch_up()
│   ├── provider.rs   # AddressProvider trait
│   └── types.rs      # AddressSyncConfig, AddressSyncResult, AddressFunds
└── nullifier_sync/
    ├── mod.rs        # NullifierOps impl, sync_nullifiers(),
    │                 #   incremental_catch_up()
    ├── provider.rs   # NullifierProvider trait
    └── types.rs      # NullifierSyncConfig, NullifierSyncResult

Rules

Do:

  • Use sdk.sync_address_balances() or sdk.sync_nullifiers() as the entry points.
  • Persist new_sync_height and new_sync_timestamp from the result and pass them back on the next sync call. This enables incremental-only mode.
  • Implement AddressProvider to integrate with your wallet's key derivation and storage.
  • Set min_privacy_count high enough that individual key lookups cannot be distinguished. The default of 32 is a reasonable minimum.

Don't:

  • Query individual keys directly via the trunk/branch RPCs -- use the sync functions which handle privacy adjustment, iteration, and proof verification.
  • Set max_iterations too low -- complex trees may need many rounds. The default of 50 handles trees with millions of entries.
  • Ignore the full_rescan_after_time_s threshold -- without periodic full rescans, the incremental phase could miss changes that occurred before the last known height.
  • Skip the incremental catch-up phase -- the tree scan snapshot may be slightly stale (the trunk is captured at a specific block height), and the catch-up brings it current.

Appendix: Query Count Analysis

Let N = total addresses in the tree, k = wallet addresses (e.g. 20), d_trunk = trunk depth (levels expanded), d_branch = levels per branch query, and P = privacy minimum (default 32).

The Merk tree is a balanced BST with depth D = ⌈log₂(N)⌉.

Trunk query: Always exactly 1 query. Returns the top d_trunk levels, where d_trunk is between min_depth (6) and max_depth (9) depending on tree size. Exposes ~2^d_trunk nodes. Cost is independent of k or N.

Key classification: After the trunk, each of the k target keys is classified by BST traversal (local computation, no queries). Keys found directly in trunk elements are resolved immediately.

Branch queries per iteration: Unresolved keys trace to trunk leaves. With privacy adjustment, keys mapping to subtrees smaller than P are promoted to ancestor subtrees. Multiple keys may share an ancestor, reducing the query count. Expected unique leaves after deduplication:

L ≈ min(k, 2^d_trunk) · (1 - e^(-k / 2^d_trunk))

Total branch queries: Each target key that traces to a leaf requires 2 branch queries per iteration — one for the left child subtree and one for the right child — to determine which side the key falls on. With k = 20 wallet keys, the first iteration produces 2k = 40 branch queries. If the tree is deeper than d_trunk + d_branch, some keys trace to deeper leaves and require a second iteration with fewer keys (typically ~50% resolve per round).

Iterations needed: I = ⌈(D - d_trunk) / d_branch⌉ where D = ⌈log₂(N)⌉. Both d_trunk and d_branch range from 6 to 9 (configured per platform version). For trees up to depth ~18 (N ≤ ~260K), one iteration suffices with max depth 9. Deeper trees need additional iterations.

Incremental catch-up: 1 recent query + 0–1 compacted queries.

Total:

Q_total = 1 (trunk) + 2k × I_eff + Q_incremental

where I_eff accounts for decreasing keys per iteration (not all keys need every round). In practice, I_eff ≈ I × 0.7 because ~50% of keys resolve per iteration and later rounds have progressively fewer queries.

Binding Patterns

Dash Platform's core logic is written in Rust. But many developers build applications in JavaScript -- browser-based wallets, Node.js services, React Native apps. The wasm-dpp package bridges these worlds by compiling Rust types to WebAssembly and exposing them as JavaScript classes.

This chapter covers the patterns used to create these bindings: the wrapper struct pattern, naming conventions, buffer handling at the boundary, getter/setter patterns, and the Inner trait.

The Problem

Rust and JavaScript have fundamentally different type systems. Rust has ownership, lifetimes, and zero-cost abstractions. JavaScript has garbage collection, prototype chains, and dynamic types. wasm-bindgen bridges the gap, but it requires careful manual work to make the resulting JavaScript API feel natural.

The core challenge: you have a Rust type like Identity with methods like id(), balance(), and public_keys(). You need to expose it to JavaScript as a class with methods like getId(), getBalance(), and getPublicKeys() -- following JavaScript naming conventions while maintaining Rust's safety guarantees.

The Wrapper Struct Pattern

Every Rust type exposed to JavaScript gets a wrapper struct. Here is IdentityWasm from packages/wasm-dpp/src/identity/identity.rs:

#![allow(unused)]
fn main() {
#[wasm_bindgen(js_name=Identity)]
#[derive(Clone)]
pub struct IdentityWasm {
    inner: Identity,
    metadata: Option<Metadata>,
}
}

The pattern has three parts:

  1. #[wasm_bindgen(js_name=Identity)] -- tells wasm_bindgen to expose this struct as Identity in JavaScript, not IdentityWasm.
  2. inner: Identity -- the real Rust type, hidden from JavaScript.
  3. Additional fields -- any extra state needed at the WASM boundary (like metadata here, which is managed separately in JS).

Why a Wrapper?

You cannot put #[wasm_bindgen] directly on Identity for several reasons:

  • Identity is defined in rs-dpp, a different crate. You cannot add attributes to types in other crates.
  • Identity may contain types that wasm_bindgen cannot handle (nested enums, complex generics, trait objects).
  • The JavaScript API should have camelCase methods (getId), not Rust-style snake_case (id).
  • Some conversions (like turning Identifier into a Buffer) only make sense at the WASM boundary.

From/Into Conversions

Every wrapper implements bidirectional conversion:

#![allow(unused)]
fn main() {
impl From<IdentityWasm> for Identity {
    fn from(identity: IdentityWasm) -> Self {
        identity.inner
    }
}

impl From<Identity> for IdentityWasm {
    fn from(identity: Identity) -> Self {
        Self {
            inner: identity,
            metadata: None,
        }
    }
}
}

This lets internal Rust code work with the real Identity type while the WASM boundary works with IdentityWasm. The conversion is zero-cost -- it just moves the inner value.

JavaScript Method Naming

Methods use #[wasm_bindgen(js_name=...)] to follow JavaScript conventions:

#![allow(unused)]
fn main() {
#[wasm_bindgen(js_class=Identity)]
impl IdentityWasm {
    #[wasm_bindgen(js_name=getId)]
    pub fn get_id(&self) -> IdentifierWrapper {
        self.inner.id().into()
    }

    #[wasm_bindgen(js_name=setId)]
    pub fn set_id(&mut self, id: IdentifierWrapper) {
        self.inner.set_id(id.into());
    }

    #[wasm_bindgen(js_name=getBalance)]
    pub fn get_balance(&self) -> u64 {
        self.inner.balance()
    }

    #[wasm_bindgen(js_name=setBalance)]
    pub fn set_balance(&mut self, balance: u64) {
        self.inner.set_balance(balance);
    }
}
}

Note the #[wasm_bindgen(js_class=Identity)] on the impl block -- this associates the methods with the Identity JavaScript class (the js_name from the struct).

In JavaScript, these become:

const identity = new Identity(platformVersion);
const id = identity.getId();
identity.setBalance(1000n);

Getter Properties

For simple values, you can use JavaScript getter syntax:

#![allow(unused)]
fn main() {
#[wasm_bindgen(getter)]
pub fn balance(&self) -> u64 {
    self.inner.balance()
}
}

In JavaScript, this becomes a property access:

const bal = identity.balance;  // no parentheses

Platform uses both patterns -- getBalance() method and balance getter -- for the same value. This provides flexibility: the getter is concise for reading, the method is consistent for tools that expect a getter/setter pair.

Constructors

The #[wasm_bindgen(constructor)] attribute creates a JavaScript constructor:

#![allow(unused)]
fn main() {
#[wasm_bindgen(constructor)]
pub fn new(platform_version: u32) -> Result<IdentityWasm, JsValue> {
    let platform_version = &PlatformVersion::get(platform_version)
        .map_err(|e| JsValue::from(e.to_string()))?;

    Identity::default_versioned(platform_version)
        .map(Into::into)
        .map_err(from_dpp_err)
}
}

Notice the error handling: Rust Result becomes a JavaScript throw. The from_dpp_err function converts Rust ProtocolError into JavaScript error objects.

Buffer Handling at the Boundary

Binary data crosses the WASM boundary as Buffer (a custom type that wraps Uint8Array):

#![allow(unused)]
fn main() {
#[wasm_bindgen(js_name=toBuffer)]
pub fn to_buffer(&self) -> Result<Buffer, JsValue> {
    let bytes = PlatformSerializable::serialize_to_bytes(
        &self.inner.clone()
    ).with_js_error()?;
    Ok(Buffer::from_bytes(&bytes))
}

#[wasm_bindgen(js_name=fromBuffer)]
pub fn from_buffer(buffer: Vec<u8>) -> Result<IdentityWasm, JsValue> {
    let identity: Identity =
        PlatformDeserializable::deserialize_from_bytes(buffer.as_slice())
            .with_js_error()?;
    Ok(identity.into())
}
}

The toBuffer/fromBuffer pair is the standard serialization interface. Every WASM type that needs to be stored or transmitted implements this pair.

Handling Complex Types: Arrays and Objects

JavaScript arrays and objects require special handling. For collections of public keys:

#![allow(unused)]
fn main() {
#[wasm_bindgen(js_name=getPublicKeys)]
pub fn get_public_keys(&self) -> Vec<JsValue> {
    self.inner
        .public_keys()
        .values()
        .cloned()
        .map(IdentityPublicKeyWasm::from)  // Rust -> Wrapper
        .map(JsValue::from)               // Wrapper -> JsValue
        .collect()
}

#[wasm_bindgen(js_name=setPublicKeys)]
pub fn set_public_keys(&mut self, public_keys: js_sys::Array)
    -> Result<usize, JsValue>
{
    if public_keys.length() == 0 {
        return Err("Must use array of PublicKeys".into());
    }

    let public_keys = public_keys
        .iter()
        .map(|key| {
            key.to_wasm::<IdentityPublicKeyWasm>("IdentityPublicKey")
                .map(|key| {
                    let key = IdentityPublicKey::from(key.to_owned());
                    (key.id(), key)
                })
        })
        .collect::<Result<_, _>>()?;

    self.inner.set_public_keys(public_keys);
    Ok(self.inner.public_keys().len())
}
}

The to_wasm::<T>("TypeName") helper extracts a Rust wrapper from a JsValue, validating that it is the correct type.

JSON Serialization

Most WASM types provide toJSON and toObject methods:

#![allow(unused)]
fn main() {
#[wasm_bindgen(js_name=toJSON)]
pub fn to_json(&self) -> Result<JsValue, JsValue> {
    let mut value = self.inner.to_object().with_js_error()?;

    // Convert identifiers to Base58 strings for readability
    value.replace_at_paths(
        dpp::identity::IDENTIFIER_FIELDS_RAW_OBJECT,
        ReplacementType::TextBase58,
    ).map_err(|e| e.to_string())?;

    // Convert binary key data to Base64
    let public_keys = value
        .get_array_mut_ref(dpp::identity::property_names::PUBLIC_KEYS)
        .map_err(|e| e.to_string())?;

    for key in public_keys.iter_mut() {
        key.replace_at_paths(
            dpp::identity::identity_public_key::BINARY_DATA_FIELDS,
            ReplacementType::TextBase64,
        ).map_err(|e| e.to_string())?;
    }

    let json = value.try_into_validating_json()
        .map_err(|e| e.to_string())?
        .to_string();

    js_sys::JSON::parse(&json)
}
}

The toJSON method applies human-readable encoding (Base58 for identifiers, Base64 for binary data) before converting to a JavaScript object. This is the format used in APIs and debugging tools.

The Inner Trait

To standardize wrapper access, Platform defines an Inner trait:

#![allow(unused)]
fn main() {
impl Inner for IdentityWasm {
    type InnerItem = Identity;

    fn into_inner(self) -> Self::InnerItem {
        self.inner
    }

    fn inner(&self) -> &Self::InnerItem {
        &self.inner
    }

    fn inner_mut(&mut self) -> &mut Self::InnerItem {
        &mut self.inner
    }
}
}

This trait provides a consistent way for other Rust code in the WASM layer to access the unwrapped type without knowing the wrapper's internal structure.

Rules

Do:

  • Follow the wrapper struct pattern: TypeWasm { inner: Type }.
  • Use js_name on the struct and js_class on the impl block.
  • Implement From<Type> for TypeWasm and From<TypeWasm> for Type.
  • Use Buffer::from_bytes for binary data crossing the boundary.
  • Implement toBuffer/fromBuffer for any type that needs serialization.
  • Use from_dpp_err or with_js_error() for error conversion.
  • Implement the Inner trait for consistent wrapper access.

Don't:

  • Expose Rust types directly to WASM -- always wrap them.
  • Use Rust naming conventions (get_id) in the JavaScript API -- use js_name=getId.
  • Return Result<T, ProtocolError> from WASM methods -- convert to Result<T, JsValue>.
  • Forget to handle empty arrays and invalid inputs with clear error messages.
  • Expose internal fields that do not make sense in JavaScript (like Arc or Mutex).

Error Macros

Dash Platform defines hundreds of consensus errors. Each one needs a WASM binding so JavaScript code can inspect error codes, read messages, and serialize errors for transport. Writing a wrapper struct for every single error by hand would be tedious, error-prone, and a maintenance burden.

This chapter covers the generic_consensus_error! macro that generates WASM bindings automatically, the paste! crate that makes it work, the manual binding pattern for errors that need custom methods, and how the two approaches coexist.

The Problem

Consider Platform's error hierarchy. At the top level:

ConsensusError
  BasicError (dozens of variants)
  StateError (dozens of variants)
  SignatureError (handful of variants)
  FeeError (one variant)

Each variant wraps a specific error struct -- TransitionNoInputsError, DocumentNotFoundError, MasternodeNotFoundError, and so on. There are well over a hundred of these.

Every one needs a JavaScript class with:

  • A getCode() method returning the numeric error code.
  • A message getter returning the human-readable error string.
  • A serialize() method for wire encoding.
  • A From<&RustType> implementation for conversion.

Writing all of this for each error would mean hundreds of nearly identical files.

The generic_consensus_error! Macro

The macro lives in packages/wasm-dpp/src/errors/generic_consensus_error.rs:

#![allow(unused)]
fn main() {
#[macro_export]
macro_rules! generic_consensus_error {
    ($error_type:ident, $error_instance:expr) => {{
        use {
            dpp::{
                consensus::{codes::ErrorWithCode, ConsensusError},
                serialization::PlatformSerializableWithPlatformVersion,
                version::PlatformVersion,
            },
            paste::paste,
            wasm_bindgen::prelude::wasm_bindgen,
            $crate::buffer::Buffer,
        };

        paste! {
            #[derive(Debug)]
            #[wasm_bindgen(js_name=$error_type)]
            pub struct [<$error_type Wasm>] {
                inner: $error_type
            }

            impl From<&$error_type> for [<$error_type Wasm>] {
                fn from(e: &$error_type) -> Self {
                    Self {
                        inner: e.clone()
                    }
                }
            }

            #[wasm_bindgen(js_class=$error_type)]
            impl [<$error_type Wasm>] {
                #[wasm_bindgen(js_name=getCode)]
                pub fn get_code(&self) -> u32 {
                    ConsensusError::from(self.inner.clone()).code()
                }

                #[wasm_bindgen(getter)]
                pub fn message(&self) -> String {
                    self.inner.to_string()
                }

                pub fn serialize(&self) -> Result<Buffer, JsError> {
                    let bytes = ConsensusError::from(self.inner.clone())
                        .serialize_to_bytes_with_platform_version(
                            PlatformVersion::first(),
                        )
                        .map_err(JsError::from)?;

                    Ok(Buffer::from_bytes(bytes.as_slice()))
                }
            }

            [<$error_type Wasm>]::from($error_instance)
        }
    }};
}
}

What It Generates

For a call like generic_consensus_error!(MasternodeNotFoundError, e), the macro generates:

  1. A wrapper struct: MasternodeNotFoundErrorWasm with inner: MasternodeNotFoundError
  2. A From impl: From<&MasternodeNotFoundError> for MasternodeNotFoundErrorWasm
  3. Three methods:
    • get_code() -- returns the numeric error code
    • message -- returns the Display string
    • serialize() -- encodes the error to bytes
  4. An instantiation: Creates a MasternodeNotFoundErrorWasm from the error reference

The paste! Crate

The magic of [<$error_type Wasm>] comes from the paste crate. Standard Rust macros cannot concatenate identifiers -- you cannot write $error_type ## Wasm to create a new identifier. The paste! macro provides this:

#![allow(unused)]
fn main() {
paste! {
    pub struct [<$error_type Wasm>] { ... }
    //         ^^^^^^^^^^^^^^^^^
    //         pastes to: MasternodeNotFoundErrorWasm
}
}

Inside paste! { ... }, [<token1 token2>] concatenates tokens into a single identifier. This is what allows the macro to generate both the JavaScript name (MasternodeNotFoundError via js_name) and the Rust struct name (MasternodeNotFoundErrorWasm via paste).

How It Is Used

The macro is used inline within the from_consensus_error_ref function and its helpers in packages/wasm-dpp/src/errors/consensus/consensus_error.rs. Here is a representative excerpt:

#![allow(unused)]
fn main() {
pub fn from_state_error(state_error: &StateError) -> JsValue {
    match state_error {
        // Manual wrappers (have custom methods)
        StateError::DocumentAlreadyPresentError(e) => {
            DocumentAlreadyPresentErrorWasm::from(e).into()
        }
        StateError::DocumentNotFoundError(e) => {
            DocumentNotFoundErrorWasm::from(e).into()
        }

        // Macro-generated wrappers (standard interface only)
        StateError::MasternodeNotFoundError(e) => {
            generic_consensus_error!(MasternodeNotFoundError, e).into()
        }
        StateError::DocumentContestCurrentlyLockedError(e) => {
            generic_consensus_error!(
                DocumentContestCurrentlyLockedError, e
            ).into()
        }
        StateError::TokenIsPausedError(e) => {
            generic_consensus_error!(TokenIsPausedError, e).into()
        }
        // ... dozens more
    }
}
}

The pattern is clear: use the macro for errors that need only getCode(), message, and serialize(). Use manual wrappers for errors that need custom accessors.

Manual Wrappers: When You Need More

Some errors expose domain-specific data that JavaScript code needs to access. For these, you write a full wrapper by hand. Here is DataContractMaxDepthExceedError from packages/wasm-dpp/src/errors/consensus/basic/data_contract/:

#![allow(unused)]
fn main() {
#[wasm_bindgen(js_name=DataContractMaxDepthExceedError)]
pub struct DataContractMaxDepthExceedErrorWasm {
    inner: DataContractMaxDepthExceedError,
}

impl From<&DataContractMaxDepthExceedError>
    for DataContractMaxDepthExceedErrorWasm
{
    fn from(e: &DataContractMaxDepthExceedError) -> Self {
        Self { inner: e.clone() }
    }
}

#[wasm_bindgen(js_class=DataContractMaxDepthError)]
impl DataContractMaxDepthExceedErrorWasm {
    #[wasm_bindgen(js_name=getMaxDepth)]
    pub fn get_max_depth(&self) -> usize {
        self.inner.max_depth()
    }

    #[wasm_bindgen(js_name=getCode)]
    pub fn get_code(&self) -> u32 {
        ConsensusError::from(self.inner.clone()).code()
    }

    #[wasm_bindgen(getter)]
    pub fn message(&self) -> String {
        self.inner.to_string()
    }
}
}

The custom get_max_depth() method lets JavaScript inspect the specific limit that was exceeded. The macro cannot generate these domain-specific accessors -- it only knows about the three standard methods.

The from_dpp_err Pattern

At the top level, Rust's ProtocolError is converted to a JsValue through from_dpp_err in packages/wasm-dpp/src/errors/from.rs:

#![allow(unused)]
fn main() {
pub fn from_dpp_err(pe: ProtocolError) -> JsValue {
    match pe {
        ProtocolError::ConsensusError(consensus_error) => {
            from_consensus_error(*consensus_error)
        }
        ProtocolError::DataContractError(e) => {
            from_data_contract_to_js_error(e)
        }
        ProtocolError::Document(e) => {
            from_document_to_js_error(*e)
        }
        ProtocolError::DataContractNotPresentError(err) => {
            DataContractNotPresentNotConsensusErrorWasm::new(
                err.data_contract_id()
            ).into()
        }
        ProtocolError::ValueError(value_error) => {
            PlatformValueErrorWasm::from(value_error).into()
        }
        _ => JsValue::from_str(
            &format!("Error conversion not implemented: {pe:#}")
        ),
    }
}
}

This is the entry point for error conversion. It dispatches to the appropriate conversion function based on the error variant. The fallback case converts unhandled errors to a string -- not ideal, but it ensures no errors are silently swallowed.

The Consensus Error Dispatch

The from_consensus_error_ref function dispatches across the entire consensus error hierarchy:

#![allow(unused)]
fn main() {
pub fn from_consensus_error_ref(e: &DPPConsensusError) -> JsValue {
    match e {
        DPPConsensusError::FeeError(e) => match e {
            FeeError::BalanceIsNotEnoughError(e) =>
                BalanceIsNotEnoughErrorWasm::from(e).into(),
        },
        DPPConsensusError::SignatureError(e) =>
            from_signature_error(e),
        DPPConsensusError::StateError(state_error) =>
            from_state_error(state_error),
        DPPConsensusError::BasicError(basic_error) =>
            from_basic_error(basic_error),
        DPPConsensusError::DefaultError =>
            JsError::new("DefaultError").into(),
    }
}
}

Each sub-function (from_state_error, from_basic_error, from_signature_error) handles its category, using either manual wrappers or the macro as appropriate.

Adding a New WASM Error Binding

When a new consensus error is added to rs-dpp, you need to add its WASM binding. Here is the checklist:

If the error only needs getCode(), message, and serialize():

  1. Import the error type in consensus_error.rs.
  2. Add a match arm using the macro:
#![allow(unused)]
fn main() {
StateError::YourNewError(e) => {
    generic_consensus_error!(YourNewError, e).into()
}
}

That is it. The macro handles everything else.

If the error needs custom accessors:

  1. Create a new file in the appropriate subdirectory under packages/wasm-dpp/src/errors/consensus/.
  2. Define the wrapper struct, From impl, and methods following the manual pattern.
  3. Add the wrapper to the mod.rs file.
  4. Import it in consensus_error.rs.
  5. Add a match arm using the manual wrapper:
#![allow(unused)]
fn main() {
StateError::YourNewError(e) => {
    YourNewErrorWasm::from(e).into()
}
}

Rules

Do:

  • Use generic_consensus_error! for errors that need only the standard three methods.
  • Use manual wrappers when JavaScript needs to access error-specific fields.
  • Follow the existing file organization: one file per manual error, grouped by category.
  • Always provide getCode(), message, and serialize() -- these are the standard interface.
  • Test that new errors convert correctly in both directions.

Don't:

  • Write manual wrappers when the macro would suffice -- it just creates maintenance debt.
  • Forget to add the match arm in consensus_error.rs -- unhandled errors will fall through to the DefaultError case or an unreachable_patterns guard.
  • Use js_name values that differ from the Rust error type name -- JavaScript developers should see the same name they find in documentation.
  • Skip the serialize() method -- it is needed for error transport across process boundaries.
  • Modify generic_consensus_error! without understanding that every call site will be affected -- it generates code in every match arm that uses it.

API Reference

Auto-generated API documentation for the Dash Platform developer ecosystem. Each section is built from source and updated by CI when relevant source files change on v*-dev branches.


Rust Crate Docs

Full rustdoc for the Dash Platform workspace — every public type, trait, function, and module across all crates.

Key crates:

CrateDescription
dash-sdkHigh-level client SDK with builder pattern and fetch traits
dppDash Platform Protocol — data contracts, documents, identities, state transitions
driveDecentralized storage engine built on GroveDB
drive-abciABCI application connecting Tenderdash to Drive
dapi-grpcRust types generated from the gRPC protocol definitions
rs-dapi-clientLow-level DAPI client with retries and load balancing
platform-valueCross-language value representation
platform-versionProtocol versioning and feature version dispatch

gRPC API

Protocol Buffer service definitions for the three gRPC endpoints:

  • Platform — identity, data contract, document, and token operations
  • Core — block, transaction, and masternode queries
  • Drive — internal node-to-node replication

Documents every RPC method, request/response message, and field type.

JavaScript / TypeScript API

TypeDoc for the JavaScript and TypeScript client libraries:

  • dash (js-dash-sdk) — main SDK for Node.js and browser applications
  • wasm-dpp — WebAssembly bindings for Dash Platform Protocol
  • wasm-sdk — Browser-facing WASM SDK with TypeScript type definitions