Introduction
Welcome to The Dash Platform Book. This is a guide to the design philosophy, architectural patterns, and engineering conventions that shape the Dash Platform Rust codebase. It is not a user manual or an API reference -- it is the book you read before you open your editor, so that the thousands of files in the monorepo start to make sense.
Who This Book Is For
- Contributors who want to add a new state transition, query endpoint, or Drive operation and need to know where things go and why.
- Auditors reviewing the codebase for correctness, who need a map of the security-critical paths (validation pipelines, fee calculations, proof verification).
- Architects evaluating the system design, particularly the versioning strategy, the ABCI integration, and the GroveDB storage model.
If you have written Rust before and understand traits, enums, and feature flags, you are ready.
The Design Philosophy
Four principles recur throughout the codebase. Understanding them up front will save you hours of "why is it done this way?" confusion.
1. Version Everything
Every method that touches consensus has a version number. The version is not
embedded in the code path -- it lives in a central PlatformVersion struct that
is threaded through the entire call stack:
#![allow(unused)] fn main() { // From packages/rs-platform-version/src/version/protocol_version.rs #[derive(Clone, Debug)] pub struct PlatformVersion { pub protocol_version: ProtocolVersion, pub dpp: DPPVersion, pub drive: DriveVersion, pub drive_abci: DriveAbciVersion, pub consensus: ConsensusVersions, pub fee_version: FeeVersion, pub system_data_contracts: SystemDataContractVersions, pub system_limits: SystemLimits, } }
When you see a method that dispatches on a version number, this is the canonical pattern:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/execution/engine/finalize_block_proposal/mod.rs pub(crate) fn finalize_block_proposal( &self, request_finalize_block: FinalizeBlockCleanedRequest, block_execution_context: BlockExecutionContext, transaction: &Transaction, platform_version: &PlatformVersion, ) -> Result<block_execution_outcome::v0::BlockFinalizationOutcome, Error> { match platform_version .drive_abci .methods .engine .finalize_block_proposal { 0 => self.finalize_block_proposal_v0( request_finalize_block, block_execution_context, transaction, platform_version, ), version => Err(Error::Execution(ExecutionError::UnknownVersionMismatch { method: "finalize_block_proposal".to_string(), known_versions: vec![0], received: version, })), } } }
This pattern allows the network to upgrade without hard forks. Every node on the network agrees on which version to run for each method at each block height, and the version table is the single source of truth. The Versioning section covers this in depth.
2. Explicit Costs
Platform is a metered system. Every operation -- inserting a document, updating a
contract, creating an identity -- has a fee expressed in credits. Costs are not
estimated after the fact; they are tracked as operations execute, through
GroveDB's cost accounting layer. The FeeResult type flows alongside the data,
and sum trees in GroveDB let the platform verify fee totals against the
Merkle-proven state.
The fee version itself is part of PlatformVersion, which means the network can
adjust fee schedules across protocol upgrades without any node disagreeing on
how much an operation costs at a given block height.
3. Clear Transformation Stages
A state transition does not go from "bytes on the wire" to "committed to disk" in one step. It passes through well-defined stages:
- Decode -- raw bytes become a
StateTransitionenum. - Structure validation -- syntactic checks (field lengths, required fields).
- State validation -- checks against current platform state (does this identity exist? does the nonce match?).
- Transform into action -- the validated transition becomes a
StateTransitionAction, a representation of what to do rather than what was requested. - Convert to operations -- the action becomes a list of
DriveOperationvalues (GroveDB inserts, deletes, replacements). - Apply -- the operations execute inside a GroveDB transaction.
Each stage is a separate module with its own versioned dispatch. This separation means you can audit validation independently from execution, and you can test each stage in isolation.
4. Trait-Based Polymorphism
The codebase avoids dynamic dispatch where performance matters and embraces it
where extensibility matters. ABCI handlers are defined against traits like
PlatformApplication, TransactionalApplication, and
BlockExecutionApplication:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/abci/app/mod.rs pub trait PlatformApplication<C = DefaultCoreRPC> { fn platform(&self) -> &Platform<C>; } pub trait TransactionalApplication<'a> { fn start_transaction(&self); fn transaction(&self) -> &RwLock<Option<Transaction<'a>>>; fn commit_transaction(&self, platform_version: &PlatformVersion) -> Result<(), Error>; } pub trait BlockExecutionApplication { fn block_execution_context(&self) -> &RwLock<Option<BlockExecutionContext>>; fn unsigned_withdrawal_txs_by_round(&self) -> &RwLock<UnsignedWithdrawalTxsByRound>; } }
This makes it possible to swap in mock implementations for testing while keeping the production path zero-cost.
How This Book Is Organized
The book follows the data as it flows through the system, from the outside in:
| Section | What You Will Learn |
|---|---|
| Architecture | The monorepo layout, crate responsibilities, and the request pipeline from client to GroveDB. |
| Versioning | How PlatformVersion controls every consensus-critical code path, and how upgrades propagate. |
| State Transitions | The lifecycle of a state transition: validation, transformation, operation generation, and application. |
| Error Handling | The split between consensus errors (returned to users) and execution errors (node-level panics). |
| Serialization | The platform-serialization crate and its derive macros for versioned binary encoding. |
| Data Model | Data contracts, documents, identities, and tokens as Rust types. |
| Drive | GroveDB operations, batch processing, cost tracking, and finalize tasks. |
| Testing | Unit test patterns, strategy tests for randomized multi-block scenarios, and test configuration. |
| SDK | The client-side dash-sdk crate: builder patterns, fetch traits, and proof verification. |
| WASM | Binding patterns for the browser-facing wasm-dpp and wasm-sdk crates. |
Each chapter follows the same arc: why the pattern exists (the problem it solves), what the pattern is (the types and modules involved), how it works (real code from the repository), and rules (the do's and don'ts that keep the codebase consistent).
Coding Conventions
A few conventions appear consistently across the codebase and are worth calling out early:
#![forbid(unsafe_code)]is set in bothdppanddrive. The platform avoids unsafe Rust entirely in its core logic. Unsafe operations are confined to external dependencies (RocksDB, cryptographic libraries).#![deny(missing_docs)]is enabled indrive, enforcing doc comments on every public item. DPP has this commented out but is moving toward it.- Feature-gated compilation is pervasive. A typical crate has 20-80 Cargo features controlling which modules, serialization formats, and integrations are compiled. This keeps binary sizes small and compile times manageable for downstream consumers that only need a subset of functionality.
- Versioned module layout: When a function has multiple versions, they live
in sibling directories named
v0/,v1/, etc., with a parentmod.rsthat dispatches based onPlatformVersion. This is the dominant structural pattern in Drive-ABCI.
A Note on Reading the Source
The codebase lives in a monorepo at packages/. The Rust crates are prefixed
with rs- on disk but have shorter names in Cargo.toml:
| Disk path | Crate name |
|---|---|
packages/rs-dpp | dpp |
packages/rs-drive | drive |
packages/rs-drive-abci | drive-abci |
packages/rs-sdk | dash-sdk |
packages/rs-platform-version | platform-version |
packages/rs-platform-value | platform-value |
packages/rs-platform-serialization | platform-serialization |
packages/rs-drive-proof-verifier | drive-proof-verifier |
The workspace currently targets Rust 1.98 and protocol version 14 (as of
4.2.0-dev). The workspace Cargo.toml lists 47 member crates, but the core
platform logic lives in the first eight listed above.
Let's begin with the architecture.
Platform Comparison
How Dash Platform compares to other blockchain networks across architecture,
features, and developer experience. Ratings from - (not supported) through
+ (basic), ++ (good), to +++ (best in class) reflect relative strength
in each dimension.
Overview
| Bitcoin | Ethereum | Solana | Polkadot | NEAR | Cosmos SDK | Avalanche | Dash Platform | |
|---|---|---|---|---|---|---|---|---|
| Primary purpose | Payments | General-purpose smart contracts | High-throughput smart contracts | Multi-chain shared security | Sharded smart contracts | App-chain framework | Multi-chain smart contracts | Decentralized data storage and querying |
| Consensus | Nakamoto (PoW) | Gasper (PoS) | Tower BFT (PoS) | GRANDPA + BABE (PoS) | Nightshade (PoS) | CometBFT (PoS) | Snowman (PoS) | Tenderdash SBFT (masternode quorums, BLS threshold signatures) |
| Finality | - Probabilistic (~60 min) | + ~13 min (2 epochs) | +++ ~0.4s (optimistic) | + ~12-60s (2 rounds) | ++ ~1-2s | +++ Instant (1 block) | ++ ~1-2s | +++ Instant (1 block) |
| Throughput (simple tx) | - ~7 tx/s | + ~15-30 tx/s | +++ ~65,000 tx/s | +++ Scales per parachain | +++ ~100,000 tx/s (sharded) | +++ Per-chain | ++ ~4,500 tx/s | ++ ~1,000 tx/s |
Data and Querying
| Bitcoin | Ethereum | Solana | Polkadot | NEAR | Cosmos SDK | Avalanche | Dash Platform | |
|---|---|---|---|---|---|---|---|---|
| Data model | - UTXOs | + Account / key-value | + Account / key-value | + Account / key-value | + Account / key-value | + App-defined | + Account / key-value | +++ Structured documents with secondary indexes |
| Decentralized querying | - Keys only (UTXO lookup) | + Keys only (no native indexing) | + Keys only (via RPC, no proofs) | + Keys only (per parachain) | + Keys only (via RPC, no proofs) | + Keys only (app-specific) | + Keys only (via RPC, no proofs) | +++ Rich queries with indexes, ordering, and ranges -- all with proofs |
| State proofs | + SPV (block headers) | ++ Merkle-Patricia proofs | - No native proofs | + Merkle proofs (per parachain) | ++ Merkle-Patricia proofs | + IAVL proofs | + Merkle proofs | +++ GroveDB Merkle proofs for every query |
| Light client trust | + Follows longest chain | + Needs sync committee | - Trusts RPC provider | + Trusts relay chain | - Trusts RPC provider | + Trusts IBC relayer | - Trusts RPC provider | +++ Cryptographic proof per response -- same security as a full node |
The standout difference is light client verification. Most chains either offer no state proofs (Solana, Avalanche), require trusting intermediaries (Polkadot's relay chain, Cosmos IBC relayers, NEAR's RPC providers), or give proofs that are expensive to verify (Ethereum's sync committee). Dash Platform serves a cryptographic proof with every query response, and a single BLS threshold signature is all a client needs to verify it. A mobile wallet gets the same security guarantees as a full node.
Smart Contracts and Programmability
| Bitcoin | Ethereum | Solana | Polkadot | NEAR | Cosmos SDK | Avalanche | Dash Platform | |
|---|---|---|---|---|---|---|---|---|
| Smart contracts | - Limited Script opcodes | +++ Solidity / Vyper on EVM | +++ Rust / C on SVM | ++ Per-parachain, typically Wasm | ++ Rust / JS / AssemblyScript on Wasm VM | + App-specific (Go) | ++ Solidity on EVM, Rust on Wasm | - Coming in v4.0 |
| VM / execution | - Script interpreter | +++ EVM | +++ SVM (eBPF) | ++ Wasm (per parachain) | ++ Wasm VM | + No VM (compiled Go) | ++ EVM + Wasm subnets | - No VM (data contracts; VM planned for v4.0) |
| Developer languages | - Script | +++ Solidity, Vyper | ++ Rust, C | ++ Rust (Substrate) | ++ Rust, JS, AssemblyScript | + Go | ++ Solidity, Rust | + JSON Schema (data contracts), Rust/JS/Swift (SDKs) |
| Smart contract security | N/A | + Reentrancy, gas exploits | ++ No reentrancy, but complexity | ++ Sandboxed per parachain | ++ Wasm sandboxing | N/A | + Inherits EVM risks | N/A (data contracts are declarative) |
Dash Platform takes a fundamentally different approach: instead of a VM that executes arbitrary code, developers define data contracts -- JSON Schema-based specifications that describe the structure and validation rules for their application data. The network stores, indexes, and enforces these schemas directly. This eliminates entire classes of smart contract vulnerabilities (reentrancy, unchecked external calls, gas manipulation). Smart contract support is planned for Platform v4.0 (targeted for mainnet in 2027).
Token Support
| Bitcoin | Ethereum | Solana | Polkadot | NEAR | Cosmos SDK | Avalanche | Dash Platform | |
|---|---|---|---|---|---|---|---|---|
| Native token standard | - BRC-20 via inscriptions | +++ ERC-20 / ERC-721 / ERC-1155 | ++ SPL Token | + Per-parachain | ++ NEP-141 / NEP-171 | + Per-chain | ++ ERC-20 (C-Chain) | +++ Protocol-native tokens with declarative rules |
| Token creation | - Requires third-party indexer | ++ Deploy smart contract | ++ Deploy program | + Deploy parachain | ++ Deploy smart contract | + Build app-chain | ++ Deploy smart contract | +++ Declare in data contract -- no code deployment |
| Freeze / pause | - No | + Only if contract implements it | ++ Mint authority can freeze | + Per-parachain | + Only if contract implements it | + Per-chain | + Only if contract implements it | +++ Protocol-level freeze, pause, and destroy |
| Minting authority | - N/A | + Contract owner / governance | + Mint authority | + Per-parachain | + Contract owner | + Per-chain | + Contract owner | +++ Individuals, groups with threshold signing, or pre-programmed schedules |
| Pre-programmed distributions | - No | + Requires contract logic | + Requires program logic | - No | + Requires contract logic | + Requires app logic | + Requires contract logic | +++ Native: time-based, epoch-based perpetual distributions |
Dash Platform tokens are first-class protocol objects rather than smart contract deployments. Token behavior (minting rules, supply caps, freeze authority, distribution schedules) is configured declaratively in data contracts and enforced by the protocol itself.
Project and Ecosystem
| Bitcoin | Ethereum | Solana | Polkadot | NEAR | Cosmos SDK | Avalanche | Dash Platform | |
|---|---|---|---|---|---|---|---|---|
| License | MIT | Various (GPL, Apache, MIT) | Apache 2.0 | GPL 3.0 | Apache 2.0 / MIT | Apache 2.0 | BSD 3-Clause | MIT |
| Open source | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Core language | C++ | Go, Rust | Rust | Rust | Rust | Go | Go | Rust |
| Client SDKs | + Multiple (community) | +++ web3.js, ethers.js, viem | ++ @solana/web3.js | + Polkadot.js | + near-api-js | + CosmJS | ++ ethers.js (C-Chain) | ++ Rust, JavaScript, Swift (iOS), Android (coming) |
| Launched | 2009 | 2015 | 2020 | 2020 | 2020 | 2019 (SDK) | 2020 | 2024 (v1.0 mainnet) |
| Ecosystem maturity | +++ Largest, most established | +++ Largest smart contract ecosystem | ++ Fast-growing DeFi ecosystem | + Growing parachain ecosystem | + Growing dApp ecosystem | ++ Many sovereign chains | ++ Growing subnet ecosystem | + Early stage, growing |
| Identity system | - Addresses only | + ENS (contract-based) | - No native identity | - No native identity | + Named accounts | - No native identity | - No native identity | +++ Protocol-native identities with hierarchical keys and DPNS usernames |
| Native token privacy | + UTXO model allows address rotation | - Account model, all activity linked to one address | - Account model, fully transparent | - Per-parachain, generally transparent | - Account model, fully transparent | - Generally transparent | - Account model, fully transparent | +++ Shielded pool with Orchard/Halo2 ZK proofs |
SDK Support
Dash Platform provides SDKs for multiple languages and environments so developers can build applications on whatever stack they prefer.
Available SDKs
| SDK | Language | Status | Package | Use case |
|---|---|---|---|---|
| Rust SDK | Rust | Available now | rs-sdk | Server-side applications, full-node tooling, direct protocol access |
| JavaScript SDK | JavaScript / TypeScript | Available now | js-evo-sdk | Node.js backends, scripts, CLI tools |
| iOS SDK | Swift | Coming in v3.1 | swift-sdk | iOS and macOS applications |
| Android SDK | Kotlin | Coming in v3.2 | -- | Android applications |
Supporting packages
| Package | Purpose |
|---|---|
rs-sdk-ffi | C FFI layer over the Rust SDK; used by the Swift SDK, the Android SDK, and any language that can call C |
Choosing an SDK
Building a server or CLI tool? Use the Rust SDK for maximum performance and direct access to all protocol features, or the JavaScript SDK if your stack is Node.js.
Building an iOS or macOS app? Use the Swift SDK (v3.1+), which wraps the Rust SDK through an FFI layer and provides native Swift types.
Building an Android app? The Android SDK (v3.2+) will wrap the same FFI layer with native Kotlin types.
Building for another language? The FFI layer (rs-sdk-ffi) exposes a
C-compatible interface that can be called from Python, C#, or any language
with C interop support.
What every SDK provides
All SDKs share the same underlying Rust implementation, so behavior is consistent across platforms:
- Identity management -- create, top up, and manage identities with hierarchical key support
- Data contract deployment -- define and publish JSON Schema-based data contracts
- Document operations -- create, update, delete, and query documents with proof verification
- Token operations -- query balances, supply, statuses, and pre-programmed distributions
- Name registration -- register and resolve DPNS usernames
- Proof verification -- every query response can be cryptographically verified against the platform state root
Getting Started
This guide covers prerequisites and local development setup for the Dash Platform monorepo.
Prerequisites
-
Node.js v20+
-
Docker v20.10+
-
Rust v1.92+, with the wasm32 target:
rustup target add wasm32-unknown-unknown -
protoc (Protocol Buffers compiler) v32.0+. If
protocis not on yourPATH, set thePROTOCenvironment variable to the binary location. -
cargo install wasm-bindgen-cli@0.2.103Important: the
wasm-bindgen-cliversion must match thewasm-bindgenversion inCargo.lock. Check withgrep 'name = "wasm-bindgen"' Cargo.lock.Depending on your system, you may need additional packages before
wasm-bindgen-cliwill compile (e.g.clang,llvm,libssl-dev). -
curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh -
Build essentials (Debian / Ubuntu):
apt install -y build-essential libssl-dev pkg-config clang cmake llvm
macOS-specific notes
The built-in Apple llvm toolchain does not work for WASM compilation.
Install LLVM from Homebrew and put it on your PATH:
brew install llvm
echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc
If you use Bash, replace ~/.zshrc with ~/.bash_profile. You can check your
default shell with echo $SHELL.
Setup
# Enable corepack (ships with Node.js) to get the correct yarn version
corepack enable
# Install dependencies, configure, and build all packages
yarn setup
# Start the local development environment (runs a local Dash network in Docker)
yarn start
# Run the full test suite (requires a running local node)
yarn test
# Stop the local environment (frees system resources)
yarn stop
# Rebuild after making changes
yarn build
# If you need to restart services after a rebuild
yarn restart
# Complete reset of all local data and builds
yarn reset
Running package-level tests
You can run tests for a single package instead of the entire suite:
yarn workspace <package_name> test
For example:
yarn workspace @dashevo/dapi-client test
See the packages directory for the full list of available packages.
Rust development
# Run tests for a specific Rust crate
cargo test -p <crate_name>
# Run all Rust workspace tests
cargo test --workspace
# Check compilation without building
cargo check --workspace
# Run the clippy linter
cargo clippy --workspace
# Format all Rust code
cargo fmt --all
Monorepo Overview
Dash Platform ships as a single Git repository containing 47 Rust crates, a handful of JavaScript/TypeScript packages, and supporting tooling. This chapter maps the territory: what each crate owns, how they depend on one another, and where the boundaries are drawn.
The Workspace
The top-level Cargo.toml declares a workspace with resolver = "2". It pins
a few critical external dependencies at the workspace level -- most notably
dashcore (the Rust Dash Core library) and the GroveDB family of crates:
# From Cargo.toml (workspace root)
[workspace.dependencies]
dashcore = { git = "https://github.com/dashpay/rust-dashcore", rev = "53d699c..." }
The workspace version (4.2.0-dev at time of writing, Rust edition 2021, MSRV
1.98) is shared by all member crates through version.workspace = true.
The Core Dependency Chain
Four crates form the spine of the platform. Understanding their dependency order is the single most important thing for navigating the codebase:
dpp (rs-dpp)
|
v
drive (rs-drive)
|
v
drive-abci (rs-drive-abci)
|
v
dash-sdk (rs-sdk)
Each layer adds a concern:
dpp -- Dash Platform Protocol
Crate name: dpp
Path: packages/rs-dpp
DPP defines the data model of the platform. This is where you will find:
Identity,IdentityPublicKey, and identity state transitionsDataContractand the JSON Schema validation logicDocumentand document state transitionsStateTransition-- the enum that unifies all transition types- Prelude types used everywhere:
Identifier,BlockHeight,IdentityNonce - Token, voting, and withdrawal types
#![allow(unused)] fn main() { // From packages/rs-dpp/src/lib.rs pub mod data_contract; pub mod document; pub mod identifier; pub mod identity; pub mod state_transition; pub mod tokens; pub mod voting; pub mod fee; pub mod validation; }
DPP is deliberately storage-agnostic. It knows nothing about GroveDB, ABCI,
or gRPC. It depends on platform-version, platform-value, and
platform-serialization for versioned encoding, but it never opens a database.
A key design choice: DPP uses feature flags extensively. The Cargo.toml
lists over 80 features that control which serialization formats, validation
paths, and system contracts are compiled in. The abci feature, for instance,
pulls in the subset needed by Drive-ABCI without dragging in client-side
concerns:
# From packages/rs-dpp/Cargo.toml
[features]
abci = [
"state-transitions",
"state-transition-validation",
"validation",
"random-public-keys",
"identity-serialization",
"vote-serialization",
"platform-value-cbor",
"core-types",
"core-types-serialization",
"core-types-serde-conversion",
"core_rpc_client",
]
Rule: If you are adding a new type that both the server and the SDK need, put it in DPP. If it needs GroveDB, it belongs in Drive.
drive -- Dash Drive
Crate name: drive
Path: packages/rs-drive
Drive is the storage engine. It wraps GroveDB (a Merkle-tree-based key-value store) and provides domain-specific operations: inserting documents, managing identity balances, tracking fee pools, building cryptographic proofs.
#![allow(unused)] fn main() { // From packages/rs-drive/src/lib.rs #[cfg(any(feature = "server", feature = "verify"))] pub mod drive; #[cfg(any(feature = "server", feature = "verify"))] pub mod query; #[cfg(feature = "server")] pub mod state_transition_action; #[cfg(any(feature = "server", feature = "verify"))] pub mod verify; #[cfg(feature = "server")] pub mod fees; }
Notice the feature split: server includes everything needed to write to the
database, while verify includes only what is needed to prove and verify
queries. The SDK only enables verify, which means it can check GroveDB proofs
without linking in the full storage engine.
Drive depends on DPP for type definitions and on the GroveDB family of crates for storage:
# From packages/rs-drive/Cargo.toml
dpp = { path = "../rs-dpp", features = ["state-transitions"], default-features = false }
grovedb = { git = "https://github.com/dashpay/grovedb", rev = "33dfd48...", optional = true }
grovedb-path = { git = "https://github.com/dashpay/grovedb", rev = "33dfd48..." }
grovedb-costs = { git = "https://github.com/dashpay/grovedb", rev = "33dfd48...", optional = true }
Rule: Never import drive with the server feature from a client-side
crate. Use verify only.
drive-abci -- The ABCI Application
Crate name: drive-abci
Path: packages/rs-drive-abci
This is the application server. It implements the Tenderdash ABCI interface, orchestrates block processing, validates state transitions, manages platform state, and serves gRPC queries. It is the binary that masternodes run.
Drive-ABCI depends on both DPP and Drive, plus the Tenderdash ABCI library and the dapi-grpc protobuf definitions:
# From packages/rs-drive-abci/Cargo.toml
drive = { path = "../rs-drive", default-features = false, features = ["server"] }
dpp = { path = "../rs-dpp", default-features = false, features = ["abci"] }
tenderdash-abci = { git = "https://github.com/dashpay/rs-tenderdash-abci", tag = "v1.5.0" }
dapi-grpc = { path = "../dapi-grpc", default-features = false, features = ["server", "platform"] }
The crate is organized into three major subsystems:
abci/-- Tenderdash handler functions (prepare_proposal,process_proposal,finalize_block,check_tx, etc.)execution/-- Block processing engine, state transition validation, platform events (epoch changes, withdrawals, voting)query/-- gRPC query service implementing the Platform API
Rule: Business logic goes in execution/. The abci/ handlers should be
thin wrappers that delegate to the execution engine.
dash-sdk -- The Client SDK
Crate name: dash-sdk
Path: packages/rs-sdk
The SDK is what application developers use. It provides high-level methods for
fetching documents, creating identities, broadcasting state transitions, and
verifying proofs. It depends on DPP for types, Drive (with verify only) for
proof verification, and rs-dapi-client for network communication:
# From packages/rs-sdk/Cargo.toml
dpp = { path = "../rs-dpp", default-features = false, features = ["dash-sdk-features"] }
drive = { path = "../rs-drive", default-features = false, features = ["verify"] }
drive-proof-verifier = { path = "../rs-drive-proof-verifier", default-features = false }
rs-dapi-client = { path = "../rs-dapi-client", default-features = false }
Supporting Crates
Several smaller crates provide cross-cutting infrastructure:
platform-version
Path: packages/rs-platform-version
The versioning backbone. Defines PlatformVersion, ProtocolVersion, and the
version tables for every consensus-critical method across DPP, Drive, and
Drive-ABCI. Currently tracks 14 protocol versions (v1 through v14).
#![allow(unused)] fn main() { // From packages/rs-platform-version/src/version/mod.rs pub type ProtocolVersion = u32; pub const LATEST_VERSION: ProtocolVersion = PROTOCOL_VERSION_14; pub const INITIAL_PROTOCOL_VERSION: ProtocolVersion = 1; }
Every crate in the dependency chain depends on platform-version. It is the
root of the version tree.
platform-serialization
Path: packages/rs-platform-serialization
A thin wrapper around bincode that adds platform-version-aware serialization.
Paired with platform-serialization-derive for derive macros that generate
versioned Encode/Decode implementations.
platform-value
Path: packages/rs-platform-value
A dynamically-typed value type (think serde_json::Value but with binary
support). Used as the interchange format when converting between JSON, CBOR, and
Rust types, especially in data contract and document processing.
drive-proof-verifier
Path: packages/rs-drive-proof-verifier
Client-side proof verification. Takes a GroveDB proof returned by a platform query and verifies it against a known root hash. Used by the SDK and the WASM bindings.
dapi-grpc
Path: packages/dapi-grpc
Protobuf definitions and generated Rust code for the Platform gRPC API. Both
server-side (Drive-ABCI) and client-side (SDK, DAPI client) depend on this
crate, using the server and client features respectively.
What Is GroveDB?
GroveDB is an external dependency -- a Merkle-tree-based authenticated data structure built on RocksDB. It is not part of this repository but is central to understanding Drive.
Key properties:
- Authenticated: Every read can produce a cryptographic proof that the data (or its absence) is consistent with the root hash stored in the block header.
- Hierarchical: Data is organized into nested trees (subtrees), addressed by
paths. A document lives at a path like
[contract_id, document_type, document_id]. - Sum trees: Some subtrees track the sum of their leaf values, used for balance accounting and fee verification.
- Transactional: All writes happen inside a transaction that can be committed or rolled back atomically.
- Cost-tracking: Every operation returns a
CostResultthat records storage and processing costs.
GroveDB is pinned to a specific Git revision in the workspace Cargo.toml and
referenced by five sub-crates: grovedb, grovedb-path, grovedb-costs,
grovedb-storage, and grovedb-version.
The Full Crate Map
Here is a simplified view of every Rust workspace member, grouped by role:
| Role | Crates |
|---|---|
| Protocol types | dpp, platform-value, platform-serialization, platform-serialization-derive, platform-versioning, platform-value-convertible |
| Storage | drive |
| Application server | drive-abci |
| Client SDK | dash-sdk, rs-dapi-client, dash-context-provider, rs-sdk-trusted-context-provider |
| Proof verification | drive-proof-verifier |
| gRPC definitions | dapi-grpc |
| WASM bindings | wasm-dpp, wasm-dpp2, wasm-sdk, wasm-drive-verify |
| iOS/FFI | rs-sdk-ffi |
| System contracts | dpns-contract, dashpay-contract, withdrawals-contract, masternode-reward-shares-contract, wallet-utils-contract, token-history-contract, keyword-search-contract, document-history-contract, app-connect-contract, moderation-charters-contract, data-contracts |
| Tooling | dashmate (JS), strategy-tests, simple-signer, check-features, json-schema-compatibility-validator |
| Other | dash-platform-macros, rs-dash-event-bus, rs-platform-wallet, dash-platform-balance-checker, rs-dapi |
Rules
Do:
- Follow the dependency direction. DPP never imports Drive. Drive never imports Drive-ABCI.
- Use feature flags to keep compilation lean. The SDK should never compile server-side code.
- Put new domain types in DPP. Put new storage operations in Drive. Put new validation logic in Drive-ABCI.
Don't:
- Add GroveDB as a dependency to DPP. If you need tree structure knowledge in a type definition, use a path abstraction.
- Enable the
serverfeature of Drive in client-facing crates. This pulls in RocksDB and doubles compile times. - Create new top-level crates without updating the workspace
Cargo.tomland ensuring the dependency direction is maintained.
Component Pipeline
This chapter traces a request from the moment it leaves a client application to the moment its effects are committed to GroveDB. Understanding this pipeline is essential because every bug, every audit finding, and every performance issue lives somewhere along this path.
The Big Picture
Client App
|
| gRPC (protobuf)
v
DAPI (rs-dapi) -----------> Drive-ABCI query service (reads)
|
| BroadcastStateTransition
v
Tenderdash mempool
|
| ABCI (check_tx, prepare_proposal, process_proposal, finalize_block)
v
Drive-ABCI (rs-drive-abci)
|
| DriveOperations
v
Drive (rs-drive)
|
| GroveDB transaction
v
GroveDB -> RocksDB
There are two fundamentally different paths through this pipeline:
-
Reads (queries): Client sends a gRPC query, DAPI forwards it to Drive-ABCI's query service, which reads from GroveDB and returns data with a Merkle proof. No consensus involved.
-
Writes (state transitions): Client broadcasts a state transition, it enters the Tenderdash mempool, goes through consensus, and is applied to GroveDB during block processing.
Let's trace the write path in detail -- it is where all the complexity lives.
DAPI: The Entry Point
DAPI (packages/rs-dapi) is the internet-facing gRPC server. It implements two
roles:
- Platform queries: Forwarded directly to Drive-ABCI's gRPC service (which runs as a separate listener within the same process).
- State transition broadcast: Submitted to Tenderdash via its RPC interface.
// Simplified from packages/rs-dapi/src/services/platform_service/broadcast_state_transition.rs
Client --gRPC--> DAPI --Tenderdash RPC--> Tenderdash mempool
DAPI itself does minimal validation. It is a routing layer. The heavy lifting happens in Drive-ABCI.
The ABCI Handler Layer
When Tenderdash needs the application to do something -- check a transaction,
build a block, validate a proposal, or finalize a block -- it calls an ABCI
method. Drive-ABCI implements these in packages/rs-drive-abci/src/abci/handler/:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/abci/handler/mod.rs mod check_tx; mod echo; mod extend_vote; mod finalize_block; mod info; mod init_chain; mod prepare_proposal; mod process_proposal; mod verify_vote_extension; }
These handlers are implemented against trait bounds, not concrete types:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/abci/handler/finalize_block.rs pub fn finalize_block<'a, A, C>( app: &A, request: proto::RequestFinalizeBlock, ) -> Result<proto::ResponseFinalizeBlock, Error> where A: PlatformApplication<C> + TransactionalApplication<'a> + BlockExecutionApplication, C: CoreRPCLike, { ... } }
The FullAbciApplication struct wires everything together. It holds a reference
to Platform, a GroveDB transaction, the current block execution context, and
the withdrawal transactions of every proposal accepted at the current height,
which vote extensions are verified against:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/abci/app/full.rs pub struct FullAbciApplication<'a, C> { pub platform: &'a Platform<C>, pub transaction: RwLock<Option<Transaction<'a>>>, pub block_execution_context: RwLock<Option<BlockExecutionContext>>, pub unsigned_withdrawal_txs_by_round: RwLock<UnsignedWithdrawalTxsByRound>, } }
It implements the tenderdash_abci::Application trait, delegating each method
to the corresponding handler function.
Block Processing Lifecycle
Tenderdash uses a proposal-based consensus model. Here is the sequence of ABCI calls for a single block:
1. check_tx -- Mempool Gatekeeper
Before a state transition enters the mempool, Tenderdash calls check_tx. This
is a lightweight validation that runs outside of consensus:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/abci/handler/check_tx.rs pub fn check_tx<C>( platform: &Platform<C>, core_rpc: &C, request: proto::RequestCheckTx, ) -> Result<proto::ResponseCheckTx, Error> where C: CoreRPCLike, { let platform_state = platform.state.load(); let platform_version = platform_state.current_platform_version()?; let validation_result = platform.check_tx( tx.as_slice(), r#type.try_into()?, &platform_ref, platform_version, ); // ... } }
check_tx operates in two modes: mode 0 (new transaction) and mode 1
(re-check existing mempool transactions after a new block). It returns a fee
estimate (gas_wanted), a priority for ordering, and a sender identifier for
deduplication. Importantly, check_tx does not run inside a GroveDB
transaction -- it reads committed state only.
2. prepare_proposal -- The Proposer Builds a Block
The block proposer calls prepare_proposal with a list of candidate
transactions. The handler starts a GroveDB transaction and runs the full block
proposal:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/abci/handler/prepare_proposal.rs // Start a GroveDB transaction app.start_transaction(); // Run the full proposal (validates + executes all state transitions) let mut run_result = app.platform().run_block_proposal( block_proposal, true, // known_from_us = true (we are the proposer) &platform_state, transaction, Some(&timer), )?; }
The response tells Tenderdash which transactions to keep, remove, or delay:
TxAction::Unmodified-- valid transition, include in blockTxAction::Removed-- unpaid error or internal error, strip from blockTxAction::Delayed-- exceeded max block size, try next block
3. process_proposal -- Validators Verify the Block
Non-proposing validators receive the block and call process_proposal. This
runs the same run_block_proposal logic but with known_from_us = false:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/abci/handler/process_proposal.rs let run_result = app.platform().run_block_proposal( (&request).try_into()?, false, // known_from_us = false (we are validating someone else's proposal) &platform_state, transaction, None, )?; }
A key optimization: if the validator was also the proposer for this round (same
height and round), the cached result from prepare_proposal is reused:
#![allow(unused)] fn main() { // From process_proposal.rs -- cache hit path if let Some(proposal_info) = block_execution_context.proposer_results() { return Ok(proto::ResponseProcessProposal { status: proto::response_process_proposal::ProposalStatus::Accept.into(), app_hash: proposal_info.app_hash.clone(), tx_results: proposal_info.tx_results.clone(), // ... }); } }
If the proposal contains failed or unpaid transitions, the validator rejects it.
4. finalize_block -- Commit
After consensus is reached, Tenderdash calls finalize_block. This is where
the GroveDB transaction is committed to disk:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/abci/handler/finalize_block.rs let block_finalization_outcome = app.platform().finalize_block_proposal( request_finalize_block, block_execution_context, transaction, platform_version, )?; // Commit the GroveDB transaction let result = app.commit_transaction(platform_version); }
After commit, the block height counter is updated and, if needed, a GroveDB checkpoint is created for crash recovery.
The finalize response also carries a proposer hint,
propose_next_block_immediately (Tenderdash 1.8.0, ABCI 1.4.0). Drive sets it
when the block leaves
withdrawal work for the next block: untied withdrawal transactions waiting in
the queue to be signed. Tenderdash then proposes round 0 of the next height without waiting
for transactions or the empty-block interval, so a withdrawal is signed one
block after it was pooled instead of one interval later. The hint is local to
the node and never part of consensus: it is read from the same GroveDB
transaction the block committed, but it does not change the state or the app
hash. See has_pending_withdrawal_work under
packages/rs-drive-abci/src/execution/platform_events/withdrawals/.
Inside run_block_proposal
The run_block_proposal method in packages/rs-drive-abci/src/execution/engine/run_block_proposal/v0/mod.rs
is the heart of the block processing engine. It orchestrates everything that
happens within a single block. Here is the sequence, in order:
1. Verify protocol version matches expected version
2. Validate block follows previous block (height, core height)
3. Clear drive block cache
4. Verify chain lock (if core chain lock update present)
5. Update core info (masternode list, quorums)
6. Update validator proposed app version
7. Rebroadcast expired withdrawals
8. Update broadcasted withdrawal statuses
9. Dequeue and build unsigned withdrawal transactions
10. Run DAO platform events (vote tallying, contested documents)
11. Process raw state transitions <-- the main work
12. Store address balance changes
13. Clean up expired address balance entries
14. Pool withdrawals into transaction queue
15. Clean up expired withdrawal amount locks
16. Process block fees and validate sum trees
17. Compute root hash (app_hash)
18. Determine validator set update
Steps 1-10 are "block-level housekeeping." Step 11 is where individual state transitions are decoded, validated, transformed into actions, and applied to GroveDB. Steps 12-18 finalize the block's effects.
State Transition Processing
State transition processing (step 11 above) follows its own pipeline within
packages/rs-drive-abci/src/execution/platform_events/state_transition_processing/:
decode_raw_state_transitions
|
v
process_raw_state_transitions
|
v
For each transition:
|
+-> validate (structure, state, signatures)
+-> transform_into_action
+-> validate_fees_of_event
+-> execute_event (apply DriveOperations to GroveDB)
The validation itself is split into modules per transition type:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/mod.rs pub mod batch; pub mod identity_create; pub mod identity_credit_transfer; pub mod identity_credit_withdrawal; pub mod identity_top_up; pub mod identity_update; pub mod data_contract_create; pub mod data_contract_update; pub mod masternode_vote; // ... and more }
Each module provides versioned validation methods dispatched through
PlatformVersion, following the same pattern shown in the Introduction.
The Query Path
Queries bypass consensus entirely. Drive-ABCI implements the Platform gRPC service directly:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/query/service.rs // Implements dapi_grpc::platform::v0::platform_server::Platform // with methods like: // get_identity() // get_documents() // get_data_contract() // get_identity_balance() // ... (50+ query endpoints) }
Each query reads from the committed GroveDB state (no transaction), generates
a Merkle proof, and returns both the data and the proof. The client SDK uses
drive-proof-verifier to independently verify that the returned data matches
the proof against a known root hash.
Protocol Version Upgrades During Block Processing
One subtlety worth highlighting: protocol version changes happen at epoch
boundaries. The run_block_proposal method checks whether the current block is
the first block of a new epoch and whether the locked-in next protocol version
differs from the current one:
#![allow(unused)] fn main() { // From packages/rs-drive-abci/src/execution/engine/run_block_proposal/mod.rs let block_platform_version = if epoch_info.is_epoch_change_but_not_genesis() && platform_state.next_epoch_protocol_version() != platform_state.current_protocol_version_in_consensus() { let next_protocol_version = platform_state.next_epoch_protocol_version(); let next_platform_version = PlatformVersion::get(next_protocol_version)?; // Perform structural changes for the new protocol version self.perform_events_on_first_block_of_protocol_change( platform_state, &block_info, transaction, old_protocol_version, next_platform_version, )?; next_platform_version } else { last_committed_platform_version }; }
This ensures that all nodes switch protocol versions at exactly the same block, and that any structural migrations (new GroveDB trees, schema changes) are applied atomically as part of that block's transaction.
Rules
Do:
- Keep ABCI handlers thin. They should parse the request, delegate to the execution engine, and format the response. No business logic.
- Always pass
platform_versionthrough the call stack. Never hard-code a version number in execution logic. - Run
check_txvalidation as a strict subset of proposal validation. Ifcheck_txaccepts a transition,process_proposalshould not reject it (modulo state changes between the two calls).
Don't:
- Read uncommitted state in
check_tx. It reads committed state only because it runs outside the block transaction. - Assume
prepare_proposalandprocess_proposalsee the same state. Another block may have been committed between the two calls if a round change occurs. - Add new block-level events without inserting them in the correct position in
the
run_block_proposalsequence. Order matters -- withdrawals must be processed before state transitions, fee accounting must happen after. - Panic in ABCI handlers except for truly unrecoverable situations (like app hash mismatches, which indicate data corruption).
Coding Conventions
The other chapters of this book explain how each subsystem works. This chapter is the prescriptive companion: the rules a change has to follow to keep the codebase maintainable, and the reason behind each one. Most of these rules were learned in code review rather than designed up front, so each entry says what goes wrong when the rule is skipped. Where a rule has mechanics, it links to the chapter that shows them.
Read this before your first non-trivial PR. Skim the checklists at the end before every PR.
The one constraint behind every rule
Dash Platform is a replicated state machine. Every masternode must produce a byte-identical state root for every block, including blocks from years ago that a fresh node replays from genesis. That means code that shipped inside a released protocol version must keep behaving exactly as it did on the day it shipped, forever, while the same binary also runs the newest behaviour for current blocks.
Almost everything below follows from that constraint: why behaviour is split
into frozen generations, why numbers live in version tables, why a stray
unwrap is a network outage and not a bug report, and why a check has to sit
in the right validation tier. When a rule here seems pedantic, the question to
ask is "could two nodes disagree because of this?"
Where code goes
The crates are layered, and dependencies only point downward:
platform-version at the bottom, then dpp, then drive. Above drive the
graph forks: drive-abci (the node) and dash-sdk (the client) both depend on
drive, the SDK with the verify feature only, and dash-sdk does not depend
on drive-abci at all (see the
Monorepo Overview). Put a change in the lowest
crate that has what the change needs, and no lower.
| You are adding… | It lives in | Notes |
|---|---|---|
| A protocol type, its wire shape, or a pure-data invariant | packages/rs-dpp | Knows nothing about storage. Structural validation that needs only the data itself goes here. |
| Storage layout, GroveDB operations, index maintenance, query lowering, proof generation | packages/rs-drive (server feature) | One method per directory with versioned dispatch. |
| Proof verification a client runs | packages/rs-drive/src/verify/ | Must compile with --no-default-features --features verify. |
| A validation step that reads platform state, block execution, an ABCI handler | packages/rs-drive-abci/src/execution/ | Handlers under abci/ stay thin and delegate. |
| A gRPC query handler | packages/rs-drive-abci/src/query/<name>/ plus the proto in packages/dapi-grpc | Request version dispatch mirrors method version dispatch. |
| A version number, a limit, a fee rate | packages/rs-platform-version | Never a literal in a method body. |
Client-side proof composition (FromProof, Tenderdash signature check) | packages/rs-drive-proof-verifier | Calls into drive::verify, never into GroveDB directly. |
| A client API | packages/rs-sdk | Follow the query checklist in packages/rs-sdk/README.md. |
| A JavaScript binding | packages/wasm-dpp2, packages/wasm-sdk | Mirror the Rust shape; never validate. See packages/wasm-dpp2/CONVENTIONS.md. |
| Mobile orchestration (sync, identity registration, DashPay) | packages/rs-platform-wallet | The FFI crates and the Swift and Kotlin SDKs marshal; they do not decide. |
Three boundaries are enforced by CI and worth knowing by name:
- The verify-only cut.
packages/wasm-drive-verifybuildsdrivewithdefault-features = false, features = ["verify"]. Anything undersrc/verify/**that reaches aserver-gated helper breaks the JavaScript build on a Rust-only PR. A helper both the prover and the verifier need goes indrive::util::common, and the server-side code delegates to it. - The transport-free cut.
drive-proof-verifier,dash-platform-queries, and the wasm32 dependency trees ofdash-sdkandwasm-sdkmust not pull inhyper,rustls,tower, ormio. Embedders with their own networking consume verification through these crates, so a dependency added for convenience in a shared crate can fail a job that has nothing to do with the change. - The wallet closure. The wallet CI fast path compiles only the wallet
crates and their known dependents. Adding a dependency on a wallet crate from
a new place fails the build until the closure list in
.github/scripts/check-wallet-closure.pyis updated deliberately.
Versioned behaviour
The mechanics of PlatformVersion, feature version tables, and the dispatcher
shape are covered in the Versioning
chapters. The rules here are about what to do with those mechanics.
Shipped generations are frozen unless the change cannot modify consensus
A behaviour change to a versioned method means a new vN module selected only
by the tables of the unreleased protocol version. A shipped vN may be edited
in place only when we are sure the edit cannot modify consensus at any protocol
version that selects it: the new code is unreachable there by construction
(the data it acts on cannot exist under those versions, such as a keyword every
one of their meta-schemas refuses and their parser ignores, judged through a
dpp method whose own gate is None there), or the edit is a pure refactor with
identical output. "Probably inert" is not enough. If the argument takes more
than a sentence, or rests on a runtime check inside the shipped module, add a
generation instead. Inside a new generation the capability is a constant fact
(Index::try_from_value_map(map, true)), not a runtime check. For new
changes, the patterns that compare protocol_version on purpose are the
protocol upgrade ladder, the is_allowed gate for new transition kinds, and
chain-creation content (create_genesis_state v1 and the Drive helpers that
build the initial state structure); the Versioned
Dispatch chapter describes them. Older
comparisons elsewhere (for example the pre-version-9 arithmetic in
DocumentPropertyType) stay because history replays through them; they are not
a pattern to copy.
Why: replay safety is structural when a shipped file stays byte-identical, and
becomes a proof the reviewer has to check the moment it does not. An in-place
edit is acceptable when that proof is short and written down; a dead version
check inside v1 that misleads the next reader into thinking v1 can take
that path is not.
How, new generation: copy the previous generation into the new module, make the change there, move the tests that exercise the new behaviour into the new module, and bump the method's number in the new protocol version's tables only. Duplication between generations is the accepted cost; it is cheaper than a drift-prone flag.
How, in place: make the edit, leave a comment at the edited lines naming why they are inert for every protocol version that selects the module, and give the pull request description an "In-place changes to shipped generations" section that lists each edited generation, the protocol versions that select it, and the reason consensus cannot change there. Reviewers read that section first.
Table versions follow protocol-version boundaries, not PRs
Version-table constants (DRIVE_ABCI_VALIDATION_VERSIONS_V10,
CONTRACT_VERSIONS_V6, SYSTEM_LIMITS_V4, and so on) exist to mark the point
where a released protocol version froze them. If the unreleased protocol
version already introduced a new table version, a second feature landing in
the same protocol version amends that table in place. Adding another table
version would leave one referenced by no PLATFORM_V* at all.
Leaf implementation modules are different: each feature still gets its own new
vN implementation module under the method directory. The rule is about the
tables.
Shared helpers get a versioned method, not a capability flag
When new behaviour lives in a helper reached from several generations (a value
walker, a property-reference resolver, a shared insert path), the fix is the
standard versioned-method pattern: an OptionalFeatureVersion field in the
tables (None for versions that predate the feature, Some(0) to dispatch to
a _v0 implementation), and the helper takes &PlatformVersion. A
admit_property_references: bool on a shared context struct is the wrong
shape, because it moves the version decision away from the tables into
whichever caller happened to set the flag.
Generation-owned grammar constants may live on a per-generation struct. Feature gates reachable from more than one generation may not.
Numbers live in the version tables
If a protocol version changes only a number (a cap, a floor, a count), the
number goes into SystemLimits or the relevant *_constants table, and the
versioned method reads it. Do not add a vN method module whose entire body is
return NEW_CONSTANT.
How: add the field with a doc comment naming the method version that reads it.
Backfill every shipped SYSTEM_LIMITS_V* (and the mock tables under
version/mocks/) with the old value so shipped protocol versions are
behaviour-preserving. Add the next SYSTEM_LIMITS_V{n+1} for the unreleased
protocol version with the new value. Keep a real method version only where the
logic differs: in packages/rs-dpp/src/withdrawal/daily_withdrawal_limit/,
v0 computes a tiered percentage of total credits and v2 reads
daily_withdrawal_limit_percent from SystemLimits; that is a logic change
and earns its own version. Raising the
percentage later would be a table edit, not a v3.
Why: reviewers look for limits in the tables. A constant hidden in a method module is a second place to check and a dead module per bump.
Fees change through named schedules
A fee change for a new protocol version is a new named schedule
(FEE_VERSION2 in packages/rs-platform-version/src/version/fee/v2.rs, and
the next one for the next protocol version) referenced from that
PLATFORM_V*, with versioned fee-group constants where a group changed. Never
inline a nested FeeVersion { .. } override inside the platform version
constant. Preserve earlier schedules. If the unreleased protocol version
already owns a new schedule, amend that schedule instead of creating another
generation nobody references.
fee_version_number is a separate concept from the schedule generation: it
keys persisted fee history and storage-refund behaviour, and it is stored in
platform state. Inspect its consumers before changing it. FEE_VERSION1 and
FEE_VERSION2 both carry fee_version_number: 1 because storage rates did not
change between them, only a fee group did. See the
Fee System Overview.
Consensus code reads the version from state
PlatformVersion::latest() is what the binary supports. The active version is
what the network agreed on, obtained from
platform_state.current_platform_version(). During an upgrade window they
differ. latest() belongs in tests and in client code that is choosing which
format to emit; it never belongs on a path that decides what a block does.
Serde-based conversions of DataContract read a process-wide current version
set through PlatformVersionCurrentVersion::set_current. Tests that build
platforms at different protocol versions race on it. Prefer the explicit
to_value(platform_version) style API wherever a version is in hand.
Dispatcher shape
The Versioned Dispatch chapter shows the
canonical match and its rules. Three details that come up in review:
- The implementation is
pub(super) fn name_v0inv0/mod.rs, usually#[inline(always)]. Nothing outside the method directory calls it. - The catch-all arm is
version => Err(...UnknownVersionMismatch { method, known_versions, received: version }), using the crate's own error (DriveError,ExecutionError,ProtocolError). Optional stages add aNone => Err(...VersionNotActive { .. })arm. #[allow(clippy::too_many_arguments)]on a dispatcher and its implementations is accepted. Do not invent a one-off parameter struct only to silence the lint.
platform_version is the last parameter
The observed order in drive and drive-abci is: &self, domain inputs,
block_info, mode flags (apply, stateful), the estimation map,
transaction, the operations accumulator, platform_version. A new parameter
goes up with the context items, never after platform_version. The same holds
for fields on context structs such as parse contexts. The known exception is
previous_fee_versions, which trails it on fee-returning methods.
Why: readers of a few hundred versioned signatures expect the closing parameter to be the version. Appending after it breaks that scan.
Versioned types
A protocol type that may evolve is an enum over per-version structs
(DataContract { V0(DataContractV0), V1(DataContractV1) }) with accessor
traits (DataContractV0Getters) implemented on the enum, so call sites never
match on the variant. The serde tag for versioned enums is $formatVersion
with variants renamed to "0", "1"; $version is reserved for the legacy
protocol-version fields and must not be reused. New variants are appended.
The full pattern, including the derive stack and the round-trip test template,
is in docs/json-value-conversion-canonical-pattern.md and the
Serialization chapters.
Where a check belongs
A new validation rule has exactly one correct tier, determined by what it needs to read. The Validation Pipeline chapter walks the stages in order; this table is the placement guide.
| Tier | Location | May read | Error class |
|---|---|---|---|
| Pure-data invariants | rs-dpp (validate_basic_structure, document type and contract self-validation) | The value itself and PlatformVersion | BasicError |
| Basic structure | drive-abci <transition>/basic_structure/vN/ | Network and PlatformVersion only | BasicError |
| Signature and nonces | drive-abci processor traits | The signing identity, nonces | SignatureError, unpaid rejection |
| Advanced structure without state | drive-abci <transition>/advanced_structure/vN/ (validate_advanced_structure) | The transition, the fetched PartialIdentity, and PlatformVersion; no Drive reads | ConsensusError, paid |
| Advanced structure with state | drive-abci batch/advanced_structure/vN/ and the per-action validators under batch/action_validation/*/advanced_structure_vN/ (validate_advanced_structure_from_state) | The action produced by transform_into_action, including the contracts it fetched, plus block, network, and identity context; no further Drive reads in the validator | ConsensusError, paid |
| State | drive-abci <transition>/state/vN/ | Drive through PlatformRef and the transaction | StateError, paid |
| Execution | drive operations | GroveDB | Internal Error only |
Rules that fall out of the table:
- Basic structure runs inside
check_txon every mempool entry. Keep it cheap and stateless. - Advanced structure without state uses the transition and the already fetched
signing identity. Advanced structure with state uses data fetched by
transform_into_action, such as the contract schema carried by a document action: the transformer performs the Drive reads, and the advanced validator consumes the resulting action. A transition opts into the second variant throughhas_advanced_structure_validation_with_state; today onlyBatchdoes. - Checks that require additional Drive queries, such as uniqueness checks, belong in state validation. Needing a fetched contract does not by itself make a structural check a state-validation check.
- Preserve mempool coverage.
Batchruns advanced structure with state duringcheck_tx, while full state validation is skipped there (validates_full_state_on_check_txdefaults tofalse; masternode votes are the one transition that opts in, because a block refuses them unpaid, and they run advanced structure with state there too). Moving a contract-dependent structural check into state validation would remove that rejection from mempool admission. - Validation outcomes are
ConsensusValidationResult, returned asOk. AResult::Errfrom a validation function means the node is broken, not that the transition is invalid. The block loop converts it into an internal-error result instead of halting, andprocess_proposalrejects any block that contains one. - Validation lives once.
wasm-dpp2, the SDKs, and the FFI layer never duplicate a length, range, or count check thatrs-dppenforces, even when the duplicate would give a friendlier early error. Two definitions of "valid" drift the day one of them changes. - Consensus errors carry numeric codes in fixed bands
(
packages/rs-dpp/src/errors/consensus/codes.rs): basic errors in 10000-19999, signature in 20000-29999, fee in 30000-39999, state in 40000-49999, with sub-bands per domain. A new error takes the next free code in its band and is appended to its enum. See Consensus Errors and Error Codes.
Errors, panics, and arithmetic
A panic in block execution halts the chain
packages/rs-drive-abci/src/main.rs installs a panic hook that cancels the
node, and there is no catch_unwind anywhere in the crate. A panic reachable
from process_proposal or finalize_block is deterministic, so every
validator hits it on the same block and the network stops. This is the
strongest rule in the codebase:
- On any path reachable from block execution, no
unwrap, noexpectwithout a proof, no slice indexing without a bounds check, nounreachable!, no division by a value that could be zero. expectis allowed when the message states the invariant established immediately above it, in the style the codebase already uses:.expect("check above enforces it exists"),.expect("we have already shown there is 1 byte"). If you cannot write that sentence, return an error.- A condition that "cannot happen" but is reachable returns
DriveError::CorruptedCodeExecution(orCriticalCorruptedStatewhen the right response is to stall) orExecutionError::CorruptedCodeExecution. See Drive Errors. - Action transformers under
state_transition_action/**/transformer.rsrun before fee and funding checks, so they are the most exposed surface. Treat every conversion there as untrusted input.
The *_blocking accessors in rs-platform-wallet are a related hazard on the
client side: tokio::sync::RwLock::blocking_read panics on a runtime worker,
and the iOS profiles build with panic = "abort". When the current thread
may be inside a runtime, use try_read and return a typed busy error.
Integer arithmetic wraps in release
The root Cargo.toml does not enable overflow-checks, so a release build
wraps silently and a debug build panics. Neither is acceptable in credit, fee,
or balance math. Use the checked forms and surface Overflow; FeeResult
already exposes checked_add_assign for the common case.
Append-only structures are enforced by CI
Structures marked // @append_only (the drive-abci error enums, the data
contract error enums, document property types, storage key requirements,
CheckTxLevel) accept new variants at the end and nothing else. Structures marked // @immutable (Epoch, BlockInfo) accept no
change at all. The Detect immutable structure changes step in
.github/workflows/tests-rs-workspace.yml diffs the tagged block against the
base branch and fails the PR on a deletion or, for @immutable, any change.
The reason is serialization: these types are encoded into proofs, responses,
and stored state with positional discriminants. Reordering or removing a
variant changes what old bytes mean.
The same rule applies without a marker to anything with a DO NOT CHANGE ORDER banner, to StateTransitionType discriminants, and to
SystemDataContract slots (note the reserved FeatureFlags = 2).
Consensus errors versus internal errors
A ConsensusError is something two nodes must agree on: it is deterministic,
carries a code, and is serialized to the client. A ProtocolError, DriveError
or ExecutionError means the build is wrong, data is corrupt, or an API was
misused. Never return a consensus error for an internal failure or an internal
error for a user mistake; the block loop treats the two classes differently.
Module layout and style
- One method, one directory.
method_name/mod.rsholds the public dispatcher and its doc comment;method_name/v0/mod.rsholds the implementation. The dispatcher's doc comment has# Parametersand# Returnssections.driveanddrive-abcicompile with#![deny(missing_docs)], so every public item needs a doc line. - Drive's naming triad.
*_operationsgathersVec<LowLevelDriveOperation>without applying anything and takes theestimated_costs_only_with_layer_infomap for dry runs;*_apply_and_add_to_operationsappends into a caller-owned accumulator and applies; the bare public method owns the accumulator and returns aFeeResult. Keep new storage methods in this shape so cost estimation and real execution share one code path. See Drive Operations. - Tests sit with the implementation. A
#[cfg(test)] mod testsblock at the bottom ofvN/mod.rs, orvN/tests/when it outgrows the file. The dispatcher'smod.rsmay carry end-to-end tests that need every version. - Imports at the top. No fully qualified
crate::a::b::c::function(...)at a call site or in a signature. Add auseand call it bare, or keep at most one module qualifier when the bare name would be ambiguous. Inline paths are fine in doc comments and inside macros that cannot see imports. - No unsafe.
dpp,drive, anddrive-abcicompile with#![forbid(unsafe_code)]. Unsafe stays in the FFI crates and in dependencies. - Test-code lints are real lints. CI runs
cargo clippy --workspace --all-targets --all-features -- -D warnings, soclippy::type_complexityin a test helper or&vec![..]passed as a slice in a test fails the build.drive-abciallows a fixed list of test lints at the crate root; do not extend it for convenience. - State access is snapshot-based.
Platform.stateis anArcSwap; read withload()and passPlatformReforPlatformStateRefdown. The only locks are the ABCI application'stransaction,block_execution_contextandunsigned_withdrawal_txs_by_round, taken in that order. In async code, a guard's scope, not adrop()call, is what clearsclippy::await_holding_lock; wrap the guard in a block that ends before the first.await.
Tests
- The latest generation tests against
PlatformVersion::latest(). Only frozen older generations pin an explicit version withPlatformVersion::get(n). When you introducev(N+1), that is the moment to pin the now-frozenvNmodule's tests. Pinning the current generation to today's number only creates churn when the next version lands. - Test behaviour through the dispatcher, on both sides of the gate. "PV13
rejects this query shape, PV14 accepts it" is a real test. "Slot
foois 0 inPLATFORM_V13and 1 inPLATFORM_V14" restates the table literal and cannot fail without the edit being deliberate; do not write it. The exception is a historical freeze test that guards a shipped on-chain table with a replay rationale, likehistorical_method_table_freezein the drive document method versions. - Names start with
should, unit tests never touch the network, and randomness is seeded (StdRng::seed_from_u64). Platforms come fromTestPlatformBuilder; Drives come fromsetup_drive_with_initial_state_structure. See Unit Tests. - Multi-block behaviour is a strategy test. Anything involving epochs,
upgrades, withdrawals over time, or proposer churn belongs in
packages/rs-drive-abci/tests/strategy_tests/. See Strategy Tests. - Fixtures are shared.
rs-dppexposes its fixtures under thefixtures-and-mocksfeature;rs-sdkrecords test vectors from a devnet and replays them offline. Add to those rather than hand-rolling payloads. - Struct-literal churn is caught by
--all-targets. A new required field compiles clean under a lib-onlycargo checkand then fails in a test module in CI. After adding a field or changing a signature, runcargo check --workspace --all-targets.
Client layers
Proof verification is layered
drive::verify does the GroveDB work: each verifier is a method on the
Drive*Query type, builds the path query with the same builder the prover
uses, and returns (RootHash, T). drive-proof-verifier takes the full
Proof message, calls into that method, and layers the Tenderdash signature
check on top; the FromProof impl is the SDK's entry point. Never call
GroveDb::verify_* from the proof-verifier crate and never build the path
query at the SDK call site; both create a "these bytes must match the prover"
invariant that nothing checks. Entry types live in drive and are re-exported
so SDK consumers need no direct drive dependency.
The wasm layer mirrors, it does not reshape
rs-dpp serde defines the canonical wire shape and wasm-dpp2 mirrors it one
to one, differing only in primitive encoding (Uint8Array versus base64,
bigint versus number-or-string). Sum types are internally tagged with type,
never wrapped in data. Field-like accessors are properties, verbs are
methods. The rules and their reasons are in packages/wasm-dpp2/CONVENTIONS.md
and the WASM chapters.
FFI and mobile SDKs hold no business logic
The FFI crates (rs-sdk-ffi, rs-platform-wallet-ffi, key-wallet-ffi) and
the Swift and Kotlin SDKs do three things: persist Rust-emitted state, load it
for views, and call a function that already exists in a Rust library. If the
wrapper is deciding anything (an index, a derivation path, which key, how
many), that decision moves into Rust, usually rs-platform-wallet. An FFI
function is named for the user-facing verb (sendContactRequest,
sendToContact), not for a cryptographic or wallet building block; DIP-14 and
DIP-15 derivation primitives stay internal. The motivating case was a Swift
view that iterated the gap limit, built the DIP-9 path, and pulled the mnemonic
across the FFI twice to do what one preview_identity_registration_keys call
now does.
Commits, pull requests, and what to run before pushing
-
Conventional Commits, with the repository's own scope list. Types and scopes are enforced by
.github/workflows/pr.yml. The subject starts lowercase. A change that spans packages uses no scope rather than an invented one (depsis not a scope;rs-platform-wallet-ffichanges useplatform-wallet). -
!means consensus-breaking.feat!:andfix!:are reserved for changes that would fork the network if rolled out unevenly: protocol rules, state-transition shapes, the fee model, anything nodes must agree on. Removing a public Swift symbol or anextern "C"function is ordinaryfeat:orrefactor:work described in the PR body. -
Design documents are working artifacts. A spec or plan written to drive a change stays in the working tree and is never staged. Stage with explicit paths. Facts worth keeping after merge go into a code comment, a test name, or the PR description.
-
Local gate before pushing. The exhaustive pass is CI's job; the local pass is scoped to what changed:
cargo fmt --all cargo clippy -p <crate> --all-features --all-targets -- -D warnings cargo check --workspace --all-targets # after any field or signature change cargo check -p drive --no-default-features --features verify # when touching src/verify/** cargo test -p <crate> <filter> # the tests for the change, not the suiteDo not block a push on the full
drive-abcisuite; it runs for a quarter of an hour and CI runs it anyway.
Checklists
Adding a new generation of a versioned method
- Copy
vN/tov(N+1)/, make the change there, leavevN/byte-identical. - Add the
N+1 =>arm to the dispatcher and extendknown_versions. - Bump the method's number in the unreleased protocol version's tables only. If that protocol version already has a new table constant, amend it.
- Move or write the behaviour tests in
v(N+1)/againstPlatformVersion::latest(); pinvN/'s tests toPlatformVersion::get(n). - Add a test that runs both versions through the dispatcher.
Editing a shipped generation in place
- Only when the edit cannot modify consensus at any protocol version that selects the module: unreachable by construction there, or output-identical.
- Comment the edited lines with why they are inert for those versions.
- Add a test that runs the module at the last shipped protocol version and shows the outcome unchanged.
- Add an "In-place changes to shipped generations" section to the pull request description: each edited generation, the versions that select it, and the reason consensus cannot change there.
Changing a limit or a fee
- A number: add or update the
SystemLimits(or*_constants) field, backfill shipped tables with the old value, add the next table for the unreleased protocol version. - A fee: add or amend the unreleased protocol version's named
FEE_VERSION*schedule; do not touchfee_version_numberunless storage rates changed and you have read its consumers. - The method reads the table. No new
vNunless the logic changed.
Adding a consensus error
- New file under
errors/consensus/{basic,state,signature,fee}/<domain>/with the standard derive stack, private fields,new(), getters, and the ordering banner. - Append the variant to its sub-enum; add the
Fromimpl forConsensusError. - Assign the next free code in the band in
codes.rs. - If JavaScript branches on it, mirror the code in
packages/wasm-dpp2/src/consensus_error.rs.
Adding a check to a state transition
- Pick the tier from the table above by what the check reads.
- Put it in
<transition>/<tier>/v(N+1)/if the transition has shipped, or in the existingvNif that generation is still unreleased. - Return it through
ConsensusValidationResult; neverErr. - Test the rejection through
process_raw_state_transitions, and test that the previous protocol version still accepts the input.
Adding GroveDB structure
- Put the key constant and the
*_path()/*_path_vec()pair in the area'spaths.rs. A new root key goes where writes below it are rare; see The GroveDB Structure. - Build it in one function that both
create_initial_state_structureand the protocol upgrade (transition_to_version_N) call, in the same order, so a fresh chain and an upgraded one hold the same state. - Describe it in the
structure.rsbeside thatpaths.rs, from the same constant, withsinceset to the protocol version that introduces it and the element flags it is written with. - Regenerate the exported file and commit it:
UPDATE_GROVEDB_STRUCTURE=1 cargo test -p drive --lib structure::tests. - Make sure something writes to the new nodes under test: a fixture in
structure/tests.rs, or a strategy test (every chain they run is checked against the description).UNVERIFIEDis empty; keep it that way. - The pull request gets a comment with a link to the structure viewer showing the new nodes. Check that it shows what you meant to add.
Adding a query end-to-end
- Proto message in
packages/dapi-grpc/protos/platform/v0/platform.proto, registered inbuild.rs's versioned request and response lists. - Drive query type and prover, then
drive::verifymethod with itsFeatureVersionslot, with a prover-verifier round trip test. drive-abcihandler underquery/<name>/{mod.rs,v0/}dispatching on the request version againstdrive_abci.querybounds.drive-proof-verifierFromProofwrapper, then thers-sdkQueryandFetch/FetchManyimpls per the checklist inpackages/rs-sdk/README.md.- Genesis test data and a recorded test vector so the SDK test runs offline.
Platform Version
The Problem: Deterministic Upgrades in a Distributed System
Dash Platform is a replicated state machine. Every masternode in the network processes the same transactions and must arrive at the exact same state. If even one node computes a fee differently, serializes a document with one extra byte, or validates a field that others skip, the chain forks.
Now imagine you need to ship a bug fix. In a normal application you deploy the new binary and move on. In a blockchain you have a harder constraint: not every node upgrades at the same moment. Some masternodes will still be running the old code when the new protocol version activates. The system needs a way to say "at protocol version N, use these exact behaviors" -- and it needs to be impossible for a developer to accidentally mix behaviors from different versions.
The answer is the PlatformVersion struct: a single, massive, immutable
snapshot that pins every versioned behavior in the entire platform to a
concrete value.
The PlatformVersion Struct
Open packages/rs-platform-version/src/version/protocol_version.rs and you
will find the heart of the system:
#![allow(unused)] fn main() { #[derive(Clone, Debug)] pub struct PlatformVersion { pub protocol_version: ProtocolVersion, pub dpp: DPPVersion, pub drive: DriveVersion, pub drive_abci: DriveAbciVersion, pub consensus: ConsensusVersions, pub fee_version: FeeVersion, pub system_data_contracts: SystemDataContractVersions, pub system_limits: SystemLimits, } }
Where ProtocolVersion is simply:
#![allow(unused)] fn main() { pub type ProtocolVersion = u32; }
Every field inside PlatformVersion is itself a version struct -- and those
structs contain more version structs, all the way down to individual method
version numbers. We will explore that nesting in the next chapter. For now,
the key insight is that PlatformVersion is the root of a tree. Given a
single protocol version number (like 7), you can resolve the exact version
of every method, every fee parameter, every system limit, and every data
contract schema across the entire platform.
Think of it like a lockfile in a package manager. Cargo.lock pins every
transitive dependency to a specific version so that builds are reproducible.
PlatformVersion does the same thing for runtime behavior: it pins every
function version so that execution is deterministic.
The Version Array
Each protocol version gets its own constant, defined in a separate file. At the time of writing, the platform has fourteen versions:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/version/mod.rs pub type ProtocolVersion = u32; pub const LATEST_VERSION: ProtocolVersion = PROTOCOL_VERSION_14; pub const INITIAL_PROTOCOL_VERSION: ProtocolVersion = 1; pub const ALL_VERSIONS: RangeInclusive<ProtocolVersion> = 1..=LATEST_VERSION; }
These fourteen snapshots are collected into a single static array in
protocol_version.rs:
#![allow(unused)] fn main() { pub const PLATFORM_VERSIONS: &[PlatformVersion] = &[ PLATFORM_V1, PLATFORM_V2, PLATFORM_V3, PLATFORM_V4, PLATFORM_V5, PLATFORM_V6, PLATFORM_V7, PLATFORM_V8, PLATFORM_V9, PLATFORM_V10, PLATFORM_V11, PLATFORM_V12, PLATFORM_V13, PLATFORM_V14, ]; pub const LATEST_PLATFORM_VERSION: &PlatformVersion = &PLATFORM_V14; pub const DESIRED_PLATFORM_VERSION: &PlatformVersion = LATEST_PLATFORM_VERSION; }
The array is indexed by protocol version number minus one (since versions are
1-indexed). PLATFORM_V1 sits at index 0, PLATFORM_V14 at index 13. This
simple layout is what makes the get function so fast.
One file, one protocol version. v14.rs was created when the first consensus
change after version 13 shipped needed somewhere to live, and every later
change destined for version 14 amends that same file. The day version 14 is
released the file freezes: from then on it is part of the chain's historical
record, and the next consensus change creates v15.rs. There is never a
v14.rs that means one thing on a node built last month and another on a node
built today.
What a Version Snapshot Looks Like
Here is the very first version, PLATFORM_V1, slightly abbreviated:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/version/v1.rs pub const PROTOCOL_VERSION_1: ProtocolVersion = 1; pub const PLATFORM_V1: PlatformVersion = PlatformVersion { protocol_version: 1, drive: DRIVE_VERSION_V1, drive_abci: DriveAbciVersion { structs: DRIVE_ABCI_STRUCTURE_VERSIONS_V1, methods: DRIVE_ABCI_METHOD_VERSIONS_V1, validation_and_processing: DRIVE_ABCI_VALIDATION_VERSIONS_V1, withdrawal_constants: DRIVE_ABCI_WITHDRAWAL_CONSTANTS_V1, query: DRIVE_ABCI_QUERY_VERSIONS_V1, checkpoints: DRIVE_ABCI_CHECKPOINT_PARAMETERS_V1, }, dpp: DPPVersion { costs: DPP_COSTS_VERSIONS_V1, validation: DPP_VALIDATION_VERSIONS_V1, state_transition_serialization_versions: STATE_TRANSITION_SERIALIZATION_VERSIONS_V1, state_transition_conversion_versions: STATE_TRANSITION_CONVERSION_VERSIONS_V1, state_transition_method_versions: STATE_TRANSITION_METHOD_VERSIONS_V1, state_transitions: STATE_TRANSITION_VERSIONS_V1, contract_versions: CONTRACT_VERSIONS_V1, document_versions: DOCUMENT_VERSIONS_V1, identity_versions: IDENTITY_VERSIONS_V1, voting_versions: VOTING_VERSION_V1, token_versions: TOKEN_VERSIONS_V1, asset_lock_versions: DPP_ASSET_LOCK_VERSIONS_V1, methods: DPP_METHOD_VERSIONS_V1, factory_versions: DPP_FACTORY_VERSIONS_V1, }, system_data_contracts: SYSTEM_DATA_CONTRACT_VERSIONS_V1, fee_version: FEE_VERSION1, system_limits: SYSTEM_LIMITS_V1, consensus: ConsensusVersions { tenderdash_consensus_version: 0, }, }; }
Now compare with PLATFORM_V14, the latest at the time of writing. By
convention, each sub-constant slot that was bumped carries a trailing
// changed: comment saying what changed. The protocol_version field is the
snapshot's identity and is never annotated. One bumped slot in this snapshot,
validation (DPP_VALIDATION_VERSIONS_V4 to V5), is missing its comment,
which is exactly the omission the convention exists to prevent:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/version/v14.rs pub const PLATFORM_V14: PlatformVersion = PlatformVersion { protocol_version: PROTOCOL_VERSION_14, drive: DRIVE_VERSION_V9, // changed: drive document method versions v4 (v2 index walkers, detect_ranked_mode slot) drive_abci: DriveAbciVersion { structs: DRIVE_ABCI_STRUCTURE_VERSIONS_V1, methods: DRIVE_ABCI_METHOD_VERSIONS_V10, // changed: records the per-block total credits history validation_and_processing: DRIVE_ABCI_VALIDATION_VERSIONS_V10, // changed: contested-index cross-check + refersTo validation withdrawal_constants: DRIVE_ABCI_WITHDRAWAL_CONSTANTS_V3, // changed: prune bound for the total credits history query: DRIVE_ABCI_QUERY_VERSIONS_V2, // changed: ranked + boolean-HAVING routing gate checkpoints: DRIVE_ABCI_CHECKPOINT_PARAMETERS_V1, }, dpp: DPPVersion { costs: DPP_COSTS_VERSIONS_V1, validation: DPP_VALIDATION_VERSIONS_V5, state_transition_serialization_versions: STATE_TRANSITION_SERIALIZATION_VERSIONS_V3, // changed: documentIndexOnlyDelete joins the wire state_transition_conversion_versions: STATE_TRANSITION_CONVERSION_VERSIONS_V2, state_transition_method_versions: STATE_TRANSITION_METHOD_VERSIONS_V1, state_transitions: STATE_TRANSITION_VERSIONS_V3, contract_versions: CONTRACT_VERSIONS_V6, // changed: v3 document meta-schema (ranked, refersTo, requiredSince, timeRange) document_versions: DOCUMENT_VERSIONS_V4, // changed: document serialization format 3 identity_versions: IDENTITY_VERSIONS_V1, voting_versions: VOTING_VERSION_V2, token_versions: TOKEN_VERSIONS_V2, asset_lock_versions: DPP_ASSET_LOCK_VERSIONS_V1, methods: DPP_METHOD_VERSIONS_V3, // changed: daily_withdrawal_limit v2 factory_versions: DPP_FACTORY_VERSIONS_V1, }, system_data_contracts: SYSTEM_DATA_CONTRACT_VERSIONS_V3, // changed: DashPay v2 profile payment address fields fee_version: FEE_VERSION2, system_limits: SYSTEM_LIMITS_V4, // changed: relative daily withdrawal limit + time-range overlap cap consensus: ConsensusVersions { tenderdash_consensus_version: 1, }, }; }
Notice how only some subsystem versions change between V1 and V14. The ABCI structure versions and checkpoint parameters are still at V1 because nothing in them ever changed. The ABCI method versions, on the other hand, went from V1 to V10 -- ten revisions of the block processing logic -- and the query versions from V1 to V3.
This is the power of the snapshot model: each subsystem version evolves at its own pace. A new protocol version does not require bumping everything. You only change the sub-constants that actually differ.
Two annotation conventions make a version file reviewable:
- The file's doc comment is the changelog.
v14.rsopens with a numbered list of every consensus change the version hosts, each with a paragraph on what it does and why it needed a protocol version. A change landing in the unreleased version adds an item there. - Every bumped slot gets a
// changed:comment saying what the new sub-constant does differently. A reviewer readingPLATFORM_V14top to bottom sees every behaviour difference fromPLATFORM_V13without opening another file.
What Belongs in the Snapshot
The snapshot holds three kinds of values, and the third is the one people forget:
- Method versions.
FeatureVersionnumbers that select av0,v1, ... implementation. The next two chapters are about these. - Format bounds.
FeatureVersionBoundsfor serialized structures: which structure versions a node accepts and which one it writes. - Protocol parameters. Plain numbers the code reads at runtime: size
caps, per-block limits, penalty amounts, fee rates, retention windows,
expiry heights.
SystemLimits,FeeVersion,DriveAbciWithdrawalConstants,DriveAbciValidationConstantsandPenaltyAmountsare all tables of these.
The rule for the third kind: a constant that might change in a future
protocol version is declared in the version tables, not in an implementation
file. Code reads it through platform_version. A const MAX_SOMETHING: u16 = 50; at the top of a Drive module is a value that cannot change without
editing shipped code, which is exactly what the versioning system exists to
avoid. A field on SystemLimits can change at a protocol-version boundary
with the old value preserved for replay.
Two consequences:
- Changing a number never creates a new method version. You change the table, and the method that reads the table keeps its version. See Non-Method Version Fields for the recipe.
- Constants that genuinely cannot change stay in code: key and hash lengths, encoding widths, tree key bytes, anything whose change would be a new storage format rather than a new parameter. The test is whether a future protocol version could plausibly want a different value. If it could, it belongs in the tables from the start; moving it later means touching shipped code.
The Get Dispatch
The most important function on PlatformVersion is get:
#![allow(unused)] fn main() { impl PlatformVersion { pub fn get<'a>(version: ProtocolVersion) -> Result<&'a Self, PlatformVersionError> { if version > 0 { PLATFORM_VERSIONS.get(version as usize - 1).ok_or_else(|| { PlatformVersionError::UnknownVersionError( format!("no platform version {version}") ) }) } else { Err(PlatformVersionError::UnknownVersionError( format!("no platform version {version}") )) } } } }
This is a simple array lookup. Protocol version 1 maps to index 0, version 14 to index 13. If the version number is out of range, you get a clear error. No hash maps, no runtime registration, no dynamic dispatch -- just a static array of compile-time constants.
There are also convenience methods:
#![allow(unused)] fn main() { impl PlatformVersion { pub fn first<'a>() -> &'a Self { PLATFORM_VERSIONS.first() .expect("expected to have a platform version") } pub fn latest<'a>() -> &'a Self { PLATFORM_VERSIONS.last() .expect("expected to have a platform version") } pub fn desired<'a>() -> &'a Self { DESIRED_PLATFORM_VERSION } } }
first() is used in tests that need to verify behavior under the initial
protocol. latest() is the default for new code. desired() returns the
version that nodes want to upgrade to -- it equals latest() during normal
operation but could theoretically differ during a staged rollout.
Version-Aware Traits
The rs-platform-version crate also defines traits that thread the platform
version through standard Rust conversion patterns:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/lib.rs pub trait TryFromPlatformVersioned<T>: Sized { type Error; fn try_from_platform_versioned( value: T, platform_version: &PlatformVersion, ) -> Result<Self, Self::Error>; } pub trait DefaultForPlatformVersion: Sized { type Error; fn default_for_platform_version( platform_version: &PlatformVersion, ) -> Result<Self, Self::Error>; } }
These are the versioned equivalents of TryFrom and Default. When you
convert a data structure, you pass the platform version so the implementation
can pick the right serialization format, the right field set, or the right
validation rules. There is also FromPlatformVersioned for infallible
conversions, and blanket IntoPlatformVersioned implementations that mirror
the standard library pattern.
Mock Versions for Testing
The version system supports a mock-versions feature flag for tests:
#![allow(unused)] fn main() { #[cfg(feature = "mock-versions")] pub static PLATFORM_TEST_VERSIONS: OnceLock<Vec<PlatformVersion>> = OnceLock::new(); }
When this feature is enabled, PlatformVersion::get checks for a special bit
in the version number. If set, it routes to the test version array instead of
the production one. This lets tests create synthetic platform versions with
specific behaviors without polluting the production constants:
#![allow(unused)] fn main() { #[cfg(feature = "mock-versions")] { if version >> TEST_PROTOCOL_VERSION_SHIFT_BYTES > 0 { let test_version = version - (1 << TEST_PROTOCOL_VERSION_SHIFT_BYTES); let versions = PLATFORM_TEST_VERSIONS .get_or_init(|| vec![TEST_PLATFORM_V2, TEST_PLATFORM_V3]); return versions.get(test_version as usize - 2).ok_or(/* ... */); } } }
This is a clever design: tests can exercise version upgrade logic (like "what happens when we transition from test version 2 to test version 3?") without needing to create real protocol versions.
Why Immutable Snapshots?
You might wonder: why not use a mutable configuration object? Why not a
HashMap<&str, u16> that maps method names to versions?
Three reasons:
-
Determinism. A
constvalue is baked into the binary at compile time. There is no way to accidentally modify it at runtime. Every node running the same binary with the same protocol version will use the exact same values. -
Exhaustiveness. Because the version struct has named fields for every subsystem, adding a new versioned method forces you to set its version in every platform version constant. The compiler will refuse to compile if you forget one. A hash map cannot give you this guarantee.
-
Performance. Looking up a version number is a struct field access -- zero overhead at runtime. The entire version tree lives in static memory. No allocations, no lookups, no indirection.
The cost is verbosity. Each new platform version file is large and repetitive.
But this is a deliberate trade-off: the system favors correctness and
auditability over conciseness. When you read PLATFORM_V14, you can see
every single version number in one place. There is no mystery about what
version 14 means.
Rules
Do:
- Always pass
&PlatformVersion(or&DriveVersion, etc.) to functions that have versioned behavior. Never hardcode a version number at a call site. - Use
PlatformVersion::latest()for tests of the current generation. Pin an explicit version withPlatformVersion::get(n)only for tests of a frozen, older generation, and usePlatformVersion::first()when you need the initial protocol behavior. - Create
vN.rsonce, when the first consensus change after version N-1 ships needs a home: copy the previous version file and change only the constants that differ. Every later change destined for version N amends that file: add a numbered item to its doc comment and a// changed:comment on each slot it bumps. - Declare any constant that might change in a future protocol version in the
version tables (
SystemLimits,FeeVersion, the*Constantsstructs) and read it throughplatform_version. Keep only genuinely invariant values asconstitems in implementation files.
Do not:
- Never mutate platform version data at runtime. The constants are
constfor a reason. - Never edit a released
vN.rs, or any table constant it references. Once a protocol version has run on the network its snapshot is part of the chain's history; a change there makes a fresh node replay history differently from every node that was there at the time. - Never add a new field to
PlatformVersionwithout also updating everyPLATFORM_V*constant. The compiler will enforce this, but be aware that the fix is updating fourteen files, not one. - Never use
PlatformVersion::latest()in consensus-critical code paths. Always use the version from the current platform state, obtained viaplatform_state.current_platform_version(). The "latest" version is what the binary supports; the active version is what the network has agreed upon -- and they may differ during an upgrade window.
Feature Versions
The Problem: Granularity
The previous chapter showed how PlatformVersion is an immutable snapshot of
the entire platform's behavior at a given protocol version. But a snapshot is
only useful if it can describe behavior at a fine enough granularity.
Consider the Drive storage layer. It has dozens of grove operations, hundreds
of document methods, contract methods, identity methods, and more. When you
fix a bug in update_contract, you need to bump that one method's version
without affecting insert_contract or prove_contract. The system needs a
way to assign a version number to individual methods and then compose those
numbers into larger subsystem snapshots.
This is where FeatureVersion and the nested version structs come in.
The FeatureVersion Type
At the very bottom of the version tree is a single type, defined in the
external versioned-feature-core crate:
#![allow(unused)] fn main() { // versioned-feature-core/src/lib.rs pub type FeatureVersion = u16; pub type OptionalFeatureVersion = Option<u16>; }
That is it. A FeatureVersion is a u16 -- a number that says "use version N
of this particular function." The value 0 means "use the v0 implementation,"
1 means "use v1," and so on.
OptionalFeatureVersion is Option<u16>. It represents a feature that did not
exist in earlier protocol versions. When the value is None, the feature is
not active -- calling it returns a VersionNotActive error. When it is
Some(0), the feature exists and should use its v0 implementation.
There is also a bounds type for serialization format versions:
#![allow(unused)] fn main() { #[derive(Clone, Debug, Default)] pub struct FeatureVersionBounds { pub min_version: FeatureVersion, pub max_version: FeatureVersion, pub default_current_version: FeatureVersion, } }
This is used when a field can accept a range of versions -- for example, a data contract serialization format where the system can read versions 0 through 2 but writes version 2 by default.
Version Structs: The Middle of the Tree
Between the top-level PlatformVersion and the leaf-level FeatureVersion
numbers sit dozens of intermediate structs. These structs group related method
versions together, forming a hierarchy that mirrors the codebase's module
structure.
Let us trace a path from the top down.
Level 1: PlatformVersion
#![allow(unused)] fn main() { pub struct PlatformVersion { pub protocol_version: ProtocolVersion, pub dpp: DPPVersion, pub drive: DriveVersion, pub drive_abci: DriveAbciVersion, pub consensus: ConsensusVersions, pub fee_version: FeeVersion, pub system_data_contracts: SystemDataContractVersions, pub system_limits: SystemLimits, } }
Level 2: DriveVersion
The drive field contains DriveVersion, which groups all storage layer
versions:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/version/drive_versions/mod.rs #[derive(Clone, Debug, Default)] pub struct DriveVersion { pub structure: DriveStructureVersion, pub methods: DriveMethodVersions, pub grove_methods: DriveGroveMethodVersions, pub grove_version: GroveVersion, } }
Level 3: DriveMethodVersions
The methods field expands into every category of Drive operation:
#![allow(unused)] fn main() { #[derive(Clone, Debug, Default)] pub struct DriveMethodVersions { pub initialization: DriveInitializationMethodVersions, pub credit_pools: DriveCreditPoolMethodVersions, pub protocol_upgrade: DriveProtocolUpgradeVersions, pub prefunded_specialized_balances: DrivePrefundedSpecializedMethodVersions, pub balances: DriveBalancesMethodVersions, pub document: DriveDocumentMethodVersions, pub vote: DriveVoteMethodVersions, pub contract: DriveContractMethodVersions, pub fees: DriveFeesMethodVersions, pub estimated_costs: DriveEstimatedCostsMethodVersions, pub asset_lock: DriveAssetLockMethodVersions, pub verify: DriveVerifyMethodVersions, pub identity: DriveIdentityMethodVersions, pub token: DriveTokenMethodVersions, pub platform_system: DrivePlatformSystemMethodVersions, pub operations: DriveOperationsMethodVersion, pub batch_operations: DriveBatchOperationsMethodVersion, pub fetch: DriveFetchMethodVersions, pub prove: DriveProveMethodVersions, pub state_transitions: DriveStateTransitionMethodVersions, pub platform_state: DrivePlatformStateMethodVersions, pub group: DriveGroupMethodVersions, pub address_funds: DriveAddressFundsMethodVersions, pub saved_block_transactions: DriveSavedBlockTransactionsMethodVersions, } }
Level 4: Individual Method Categories
Each category struct contains FeatureVersion fields for individual methods.
For example, the contract method versions:
#![allow(unused)] fn main() { #[derive(Clone, Debug, Default)] pub struct DriveContractMethodVersions { pub prove: DriveContractProveMethodVersions, pub apply: DriveContractApplyMethodVersions, pub insert: DriveContractInsertMethodVersions, pub update: DriveContractUpdateMethodVersions, pub costs: DriveContractCostsMethodVersions, pub get: DriveContractGetMethodVersions, } #[derive(Clone, Debug, Default)] pub struct DriveContractUpdateMethodVersions { pub update_contract: FeatureVersion, pub update_description: FeatureVersion, pub update_keywords: FeatureVersion, } }
So the full path to read "which version of update_contract should I use?" is:
#![allow(unused)] fn main() { platform_version.drive.methods.contract.update.update_contract }
That is a five-level deep field access, and it resolves to a plain u16.
The Grove Methods Branch
Let us trace a different path. The grove_methods field on DriveVersion
holds versions for low-level GroveDB operations:
#![allow(unused)] fn main() { #[derive(Clone, Debug, Default)] pub struct DriveGroveMethodVersions { pub basic: DriveGroveBasicMethodVersions, pub batch: DriveGroveBatchMethodVersions, pub apply: DriveGroveApplyMethodVersions, pub costs: DriveGroveCostMethodVersions, } }
The basic struct is where individual grove operations live:
#![allow(unused)] fn main() { #[derive(Clone, Debug, Default)] pub struct DriveGroveBasicMethodVersions { pub grove_insert: FeatureVersion, pub grove_insert_empty_tree: FeatureVersion, pub grove_insert_if_not_exists: FeatureVersion, pub grove_clear: FeatureVersion, pub grove_delete: FeatureVersion, pub grove_get_raw: FeatureVersion, pub grove_get_raw_optional: FeatureVersion, pub grove_get: FeatureVersion, pub grove_get_path_query: FeatureVersion, pub grove_get_proved_path_query: FeatureVersion, pub grove_get_sum_tree_total_value: FeatureVersion, pub grove_has_raw: FeatureVersion, // ... and many more } }
So the path for grove_get_raw is:
#![allow(unused)] fn main() { drive_version.grove_methods.basic.grove_get_raw }
Notice something: grove operations take a &DriveVersion rather than
&PlatformVersion. This is a minor optimization -- when you are deep in the
Drive layer, you only need the drive-specific version numbers, not the entire
platform snapshot. The caller extracts &platform_version.drive once and
passes it down.
The DPP Branch
The Dash Platform Protocol has its own deep tree. DPPVersion contains
fourteen sub-version structs:
#![allow(unused)] fn main() { #[derive(Clone, Debug, Default)] pub struct DPPVersion { pub costs: DPPCostsVersions, pub validation: DPPValidationVersions, pub state_transition_serialization_versions: DPPStateTransitionSerializationVersions, pub state_transition_conversion_versions: DPPStateTransitionConversionVersions, pub state_transition_method_versions: DPPStateTransitionMethodVersions, pub state_transitions: DPPStateTransitionVersions, pub contract_versions: DPPContractVersions, pub document_versions: DPPDocumentVersions, pub identity_versions: DPPIdentityVersions, pub voting_versions: DPPVotingVersions, pub token_versions: DPPTokenVersions, pub asset_lock_versions: DPPAssetLockVersions, pub methods: DPPMethodVersions, pub factory_versions: DPPFactoryVersions, } }
And those go deeper. For example, DPPContractVersions contains not just
FeatureVersion values but also FeatureVersionBounds and further nesting:
#![allow(unused)] fn main() { #[derive(Clone, Debug, Default)] pub struct DPPContractVersions { pub max_serialized_size: u32, pub contract_serialization_version: FeatureVersionBounds, pub contract_structure_version: FeatureVersion, pub created_data_contract_structure: FeatureVersion, pub config: FeatureVersionBounds, pub methods: DataContractMethodVersions, pub document_type_versions: DocumentTypeVersions, pub token_versions: TokenVersions, } }
Notice max_serialized_size: u32. Not every field is a FeatureVersion. Some
are configuration values -- limits, thresholds, constants -- that change between
protocol versions. The version struct is flexible enough to hold both "which
implementation to use" and "what parameters to use."
The Drive ABCI Branch
The DriveAbciVersion struct covers the application blockchain interface --
the layer that processes blocks, validates state transitions, and handles
protocol upgrades:
#![allow(unused)] fn main() { #[derive(Clone, Debug, Default)] pub struct DriveAbciVersion { pub structs: DriveAbciStructureVersions, pub methods: DriveAbciMethodVersions, pub validation_and_processing: DriveAbciValidationVersions, pub withdrawal_constants: DriveAbciWithdrawalConstants, pub query: DriveAbciQueryVersions, pub checkpoints: DriveAbciCheckpointParameters, } }
The validation_and_processing field is where state transition validation
versions live. This is where OptionalFeatureVersion becomes important:
#![allow(unused)] fn main() { #[derive(Clone, Debug, Default)] pub struct DriveAbciStateTransitionValidationVersion { pub basic_structure: OptionalFeatureVersion, pub advanced_structure: OptionalFeatureVersion, pub identity_signatures: OptionalFeatureVersion, pub nonce: OptionalFeatureVersion, pub state: FeatureVersion, pub transform_into_action: FeatureVersion, } }
basic_structure is OptionalFeatureVersion -- in some protocol versions,
basic structure validation may not exist for a particular state transition. The
dispatch code handles this with a three-arm match:
#![allow(unused)] fn main() { match platform_version .drive_abci .validation_and_processing .state_transitions .identity_create_state_transition .basic_structure { Some(0) => self.validate_basic_structure_v0(platform_version), Some(version) => Err(Error::Execution( ExecutionError::UnknownVersionMismatch { /* ... */ } )), None => Err(Error::Execution( ExecutionError::VersionNotActive { /* ... */ } )), } }
Compare with state and transform_into_action which are plain
FeatureVersion -- those validations always exist, so there is no None arm.
Non-Method Version Fields
Some version structs contain values that are not method versions at all, but
protocol parameters. SystemLimits is the main one:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/version/system_limits/mod.rs #[derive(Clone, Debug, Default)] pub struct SystemLimits { pub estimated_contract_max_serialized_size: u16, pub max_field_value_size: u32, /// `None` preserves the behavior of protocol versions that predate this limit. pub max_document_value_depth: Option<u16>, pub max_state_transition_size: u64, pub max_transitions_in_documents_batch: u16, pub withdrawal_transactions_per_block_limit: u16, pub retry_signing_expired_withdrawal_documents_per_block_limit: u16, pub max_withdrawal_amount: u64, /// `None` for the protocol versions that predate the relative rule. pub daily_withdrawal_limit_percent: Option<u8>, pub core_credit_pool_unlock_limit_percent: Option<u8>, pub core_credit_pool_unlock_limit_floor: Option<u64>, pub core_credit_pool_window_blocks: Option<u32>, pub regtest_core_credit_pool_window_blocks: Option<u32>, pub min_withdrawal_amount: u64, pub max_contract_group_size: u16, pub max_token_redemption_cycles: u32, pub max_shielded_transition_actions: u16, pub max_time_range_overlap_factor: Option<u64>, } }
There are four SYSTEM_LIMITS_V* constants, one for each protocol version at
which a limit changed. The Option fields show the idiom for a parameter that
did not exist before some version: None in the tables of the versions that
predate the rule, Some(value) from the version that introduced it. It is the
parameter-shaped twin of OptionalFeatureVersion.
The same shape recurs wherever a subsystem owns tunables:
#![allow(unused)] fn main() { // drive_abci_versions/drive_abci_withdrawal_constants/mod.rs pub struct DriveAbciWithdrawalConstants { pub core_expiration_blocks: u32, pub cleanup_expired_locks_of_withdrawal_amounts_limit: u16, pub total_credits_history_prune_limit: u16, pub core_blocks_scanned_per_block_limit: u16, } // drive_abci_versions/drive_abci_validation_versions/mod.rs pub struct PenaltyAmounts { pub identity_id_not_correct: u64, pub unique_key_already_present: u64, // ... pub shielded_proof_verification_failure: u64, } pub struct DriveAbciCoreChainLockMethodVersionsAndConstants { pub choose_quorum: FeatureVersion, pub verify_chain_lock: FeatureVersion, // ... pub recent_block_count_amount: u32, } }
The chain lock struct mixes method versions (choose_quorum: FeatureVersion)
with protocol constants (recent_block_count_amount: u32). This is perfectly
fine -- the version snapshot captures all protocol-specific values, whether
they control dispatch or configure behavior. Fee rates follow the same idea
one level up: FeeVersion is a table of numbers, and a fee change is a new
named FEE_VERSION* schedule referenced from the platform version, never an
inline override inside PLATFORM_V*.
Changing a number
Because parameters live in tables, changing one is a table edit and not a method version:
- If the field does not exist yet, add it to the struct with a doc comment naming the method version that reads it.
- Backfill every shipped table constant (and the mock tables under
version/mocks/) with the old value, so released protocol versions keep their behaviour. The compiler forces this step. - Add the next table constant for the unreleased protocol version with the
new value, and point that version's
PLATFORM_V*at it. If the unreleased version already introduced a new table constant, edit that one in place instead. - Make the method read the field through
platform_version. The method keeps its version number: shipped versions read their old value from their own table, so their behaviour is unchanged.
Do not create a vN method module whose entire body returns the new
constant. That hides a protocol parameter in an implementation file, which is
the situation the tables exist to prevent, and it leaves a dead module behind
every time the number moves. A new method version is warranted only when the
logic changes. daily_withdrawal_limit in rs-dpp is the reference case:
v0 derives the limit from the current total credits, v2 reads
daily_withdrawal_limit_percent from SystemLimits. Raising the percentage
later is a SYSTEM_LIMITS_V5, not a v3.
How Subsystem Version Constants Compose
Each subsystem version constant (like DRIVE_VERSION_V1) is assembled from
smaller constants:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/version/drive_versions/v1.rs pub const DRIVE_VERSION_V1: DriveVersion = DriveVersion { structure: DRIVE_STRUCTURE_V1, methods: DriveMethodVersions { initialization: DriveInitializationMethodVersions { create_initial_state_structure: 0, }, credit_pools: CREDIT_POOL_METHOD_VERSIONS_V1, protocol_upgrade: DriveProtocolUpgradeVersions { clear_version_information: 0, fetch_versions_with_counter: 0, // ... }, balances: DriveBalancesMethodVersions { add_to_system_credits: 0, remove_from_system_credits: 0, calculate_total_credits_balance: 0, // ... }, contract: DRIVE_CONTRACT_METHOD_VERSIONS_V1, // ... }, grove_methods: DRIVE_GROVE_METHOD_VERSIONS_V1, grove_version: GROVE_V1, }; }
Notice the mix of inline construction and named constants. Small structs like
DriveProtocolUpgradeVersions are often written inline because all their
fields are 0 in every version. Larger, frequently-changing structs like
DRIVE_CONTRACT_METHOD_VERSIONS_V1 get their own named constant so they can
be reused or overridden in later versions.
When DRIVE_VERSION_V9 (used in PLATFORM_V14) needs new document method
versions, it references DRIVE_DOCUMENT_METHOD_VERSIONS_V4 where V8
referenced V3, and annotates the slot:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/version/drive_versions/v9.rs pub const DRIVE_VERSION_V9: DriveVersion = DriveVersion { structure: DRIVE_STRUCTURE_V1, methods: DriveMethodVersions { // ... document: DRIVE_DOCUMENT_METHOD_VERSIONS_V4, // changed in v9: v2 index walkers + v1 update walker contract: DRIVE_CONTRACT_METHOD_VERSIONS_V3, // changed in v8: count-tree-aware contract-insertion cost estimation // ... verify: DRIVE_VERIFY_METHOD_VERSIONS_V2, // changed in v8: compacted address-balance proof envelope identity: DRIVE_IDENTITY_METHOD_VERSIONS_V2, // changed in v9: v1 withdrawal-by-transaction-index query builder // ... }, grove_methods: DRIVE_GROVE_METHOD_VERSIONS_V1, // changed in v9: GROVE_V4 activates the indexed-tree batch cleanup grove_version: GROVE_V4, }; }
The unchanged parts reference the same constants they always did. Only the
parts that actually changed get new constants. grove_version is how GroveDB
behaviour is versioned from Platform's side: a new DRIVE_VERSION_V* points
at a new GROVE_V*, and every GroveDB call receives it.
Two ways to write a new table constant
Older table files are full copies of the previous version with the changed slots edited. Newer ones use struct update syntax, so that the file is the diff:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/version/drive_abci_versions/drive_abci_query_versions/v2.rs /// Differs from v1 in two slots. /// `document_query_helpers.compute_aggregate_mode_and_check_limit` is 2 /// rather than 0: the ranked and boolean-`HAVING` routing gate. ... pub const DRIVE_ABCI_QUERY_VERSIONS_V2: DriveAbciQueryVersions = DriveAbciQueryVersions { document_query_helpers: DriveAbciDocumentQueryHelperVersions { compute_aggregate_mode_and_check_limit: 2, }, data_contract_query_helpers: DriveAbciDataContractQueryHelperVersions { latest_versions_read: 1, }, ..DRIVE_ABCI_QUERY_VERSIONS_V1 }; }
It nests, so one changed leaf deep in a table still reads as one line:
#![allow(unused)] fn main() { // packages/rs-platform-version/src/version/dpp_versions/dpp_validation_versions/v5.rs pub const DPP_VALIDATION_VERSIONS_V5: DPPValidationVersions = DPPValidationVersions { document_type: DocumentTypeValidationVersions { validate_update: 1, ..DPP_VALIDATION_VERSIONS_V4.document_type }, ..DPP_VALIDATION_VERSIONS_V4 }; }
Use the struct update form for new table constants. Either way, the doc
comment on the constant names the slots that differ from the previous table
and why, in the same spirit as the // changed: comments on PLATFORM_V*.
When to create a new table constant
Table constants mark the boundaries between released protocol versions. Create
DRIVE_ABCI_QUERY_VERSIONS_V4 when the constant you would otherwise edit is
referenced by a released PLATFORM_V*. If the unreleased protocol version
already introduced a new constant for that table, a second feature landing in
the same protocol version amends it in place. Otherwise the crate accumulates
table versions that no protocol version references, and the numbering stops
saying anything about what shipped when.
The File Layout
The version structs follow a consistent directory layout in
packages/rs-platform-version/src/version/:
version/
mod.rs # ProtocolVersion type alias, LATEST_VERSION, module declarations
protocol_version.rs # PlatformVersion struct, PLATFORM_VERSIONS array, get()
consensus_versions.rs # ConsensusVersions (Tenderdash consensus version)
feature_initial_protocol_versions.rs # first protocol version of whole new transition kinds
v1.rs .. v14.rs # PLATFORM_V* snapshot constants, one per protocol version
drive_versions/
mod.rs # DriveVersion, DriveMethodVersions, etc.
v1.rs .. v9.rs # DRIVE_VERSION_V* constants
drive_grove_method_versions/
mod.rs # DriveGroveMethodVersions struct
v1.rs # DRIVE_GROVE_METHOD_VERSIONS_V1
drive_contract_method_versions/
mod.rs # DriveContractMethodVersions struct
v1.rs .. v3.rs # versioned constants
drive_document_method_versions/
mod.rs
v1.rs .. v4.rs
drive_verify_method_versions/
mod.rs
v1.rs, v2.rs
...
drive_abci_versions/
mod.rs # DriveAbciVersion struct
drive_abci_method_versions/
mod.rs # DriveAbciMethodVersions and sub-structs
v1.rs .. v10.rs # versioned constants
drive_abci_validation_versions/
mod.rs # DriveAbciValidationVersions, PenaltyAmounts, validation constants
v1.rs .. v10.rs
drive_abci_query_versions/
mod.rs
v0.rs .. v2.rs
drive_abci_withdrawal_constants/
mod.rs # DriveAbciWithdrawalConstants (parameters, not method versions)
v1.rs .. v3.rs
drive_abci_structure_versions/ v1.rs
drive_abci_checkpoint_parameters/ v1.rs
dpp_versions/
mod.rs # DPPVersion struct
dpp_contract_versions/
mod.rs # DPPContractVersions struct
v1.rs .. v6.rs
dpp_validation_versions/
mod.rs
v1.rs .. v5.rs
...
fee/
mod.rs # FeeVersion struct, FEE_VERSIONS
v1.rs, v2.rs # FEE_VERSION* schedules
storage/, signature/, processing/, ... # per-group tables, each with its own v*.rs
system_limits/
mod.rs # SystemLimits struct
v1.rs .. v4.rs
system_data_contract_versions/
mod.rs
v1.rs .. v3.rs
mocks/
v2_test.rs, v3_test.rs # TEST_PLATFORM_V* behind the mock-versions feature
The pattern is: mod.rs defines the struct, and v*.rs files define the
concrete constants. The struct definition is the schema. The version files
are the data. Note that each table's v* numbering is its own:
DRIVE_VERSION_V9 is what PLATFORM_V14 uses, and SYSTEM_LIMITS_V4
likewise. The number says how many times that table has changed, not which
protocol version it belongs to.
Rules
Do:
- When adding a new method to Drive, DPP, or Drive ABCI, add a corresponding
FeatureVersionfield to the appropriate version struct. Then set its value in everyv*.rsconstant -- the compiler will force you. - Use
OptionalFeatureVersionfor features that are being introduced in a non-initial protocol version. Set them toNonein earlier versions andSome(0)in the version that introduces the feature. Use anOptionfield the same way for a parameter that did not exist before some version. - Declare every protocol parameter that might change (limits, penalties,
retention windows, fee rates) as a table field and read it through
platform_version. Change a number by adding a table version, not a method version. - Write new table constants with struct update syntax (
..PREVIOUS) and a doc comment naming the slots that differ and why. - Group related methods into their own sub-struct when the parent struct
grows too large. Follow the existing naming pattern:
Drive<Category>MethodVersions.
Do not:
- Never use a raw
u16where you meanFeatureVersion. The type alias exists for readability and future-proofing -- if we ever need to change the underlying type, the alias is the single point of change. - Never put runtime-computed values into a version struct. Every field must be a compile-time constant. This is what makes the snapshot deterministic.
- Never reuse a version constant with different semantics. If
DRIVE_CONTRACT_METHOD_VERSIONS_V1means something, creating a V2 that changes one field is correct. Silently modifying V1 is not -- it would change the behavior of every platform version that references it. - Never create a new table constant for a second change landing in the same unreleased protocol version. Table versions mark released boundaries; amend the unreleased version's own constant in place.
- Never leave a
constin an implementation file that a future protocol version might want to change. Move it to the tables the first time it is touched.
Versioned Dispatch
The Problem: Running the Right Code
The previous two chapters explained what gets versioned (every method in the
platform) and how version numbers are stored (nested structs inside an
immutable PlatformVersion snapshot). This chapter covers the most important
part: how those version numbers actually select which code runs.
The core idea is simple. Every versioned function has a dispatch method that
reads a FeatureVersion value and calls the corresponding implementation. But
the way this dispatch is organized across files, the error handling conventions,
and the step-by-step process of adding a new version -- these are the details
that make the pattern work at scale.
The Canonical Match Pattern
Here is the most common pattern in the codebase. This is from
packages/rs-drive/src/util/grove_operations/grove_get_raw/mod.rs:
#![allow(unused)] fn main() { impl Drive { pub fn grove_get_raw<B: AsRef<[u8]>>( &self, path: SubtreePath<'_, B>, key: &[u8], direct_query_type: DirectQueryType, transaction: TransactionArg, drive_operations: &mut Vec<LowLevelDriveOperation>, drive_version: &DriveVersion, ) -> Result<Option<Element>, Error> { match drive_version.grove_methods.basic.grove_get_raw { 0 => self.grove_get_raw_v0( path, key, direct_query_type, transaction, drive_operations, drive_version, ), version => Err(Error::Drive(DriveError::UnknownVersionMismatch { method: "grove_get_raw".to_string(), known_versions: vec![0], received: version, })), } } } }
Let us break down what is happening:
-
The public method (
grove_get_raw) is the entry point. It takes all the business parameters plus a version reference (drive_version: &DriveVersion). -
The version lookup reads the specific
FeatureVersionfor this method:drive_version.grove_methods.basic.grove_get_raw. This resolves to au16. -
The match dispatches to the right implementation. Version
0callsgrove_get_raw_v0. The catch-all arm (version =>) returns an error. -
The error (
UnknownVersionMismatch) includes the method name, the list of known versions, and the version that was actually received. This makes debugging version mismatches trivial.
This pattern appears hundreds of times across the codebase. It is the fundamental building block of versioned execution.
Multiple Versions
When a method has been revised, the match grows. Here is update_contract
from packages/rs-drive/src/drive/contract/update/update_contract/mod.rs:
#![allow(unused)] fn main() { impl Drive { pub fn update_contract( &self, contract: &DataContract, block_info: BlockInfo, apply: bool, transaction: TransactionArg, platform_version: &PlatformVersion, previous_fee_versions: Option<&CachedEpochIndexFeeVersions>, ) -> Result<FeeResult, Error> { match platform_version .drive .methods .contract .update .update_contract { 0 => self.update_contract_v0( contract, block_info, apply, transaction, platform_version, previous_fee_versions, ), 1 => self.update_contract_v1( contract, block_info, apply, transaction, platform_version, previous_fee_versions, ), version => Err(Error::Drive(DriveError::UnknownVersionMismatch { method: "update_contract".to_string(), known_versions: vec![0, 1], received: version, })), } } } }
The structure is identical. The only differences are: there are now two known
versions (0 and 1), and the known_versions vector in the error arm lists
both. When a node running protocol version 1 processes a block, the version
number is 0 and update_contract_v0 runs. When the network upgrades and the
version number becomes 1, update_contract_v1 runs instead.
Both v0 and v1 implementations coexist in the binary. Old code is never deleted (at least not until a version is permanently retired from the network). This is critical for replaying historical blocks -- a node syncing from genesis needs to execute v0 for early blocks and v1 for later ones.
OptionalFeatureVersion Dispatch
For features introduced after the initial protocol version, the dispatch
handles a None case:
#![allow(unused)] fn main() { // From identity_create/mod.rs match platform_version .drive_abci .validation_and_processing .state_transitions .identity_create_state_transition .basic_structure { Some(0) => { self.validate_basic_structure_v0(platform_version) } Some(version) => Err(Error::Execution( ExecutionError::UnknownVersionMismatch { method: "identity create transition: validate_basic_structure" .to_string(), known_versions: vec![0], received: version, } )), None => Err(Error::Execution( ExecutionError::VersionNotActive { method: "identity create transition: validate_basic_structure" .to_string(), known_versions: vec![0], } )), } }
Three arms instead of two:
Some(0)-- the feature exists, use v0.Some(version)-- the feature exists but the version is unrecognized.None-- the feature does not exist in this protocol version.
The VersionNotActive error is different from UnknownVersionMismatch. It
means "this feature is legitimately not available," not "something went wrong."
This distinction matters for callers that need to handle graceful degradation.
The Directory Convention
Versioned methods follow a strict directory layout. Let us use grove_get_raw
as the example:
packages/rs-drive/src/util/grove_operations/
grove_get_raw/
mod.rs # dispatch method (the match statement)
v0/
mod.rs # grove_get_raw_v0 implementation
The dispatch method lives in grove_get_raw/mod.rs. Each implementation
version gets its own subdirectory: v0/mod.rs, v1/mod.rs, etc. The dispatch
file declares the version modules:
#![allow(unused)] fn main() { // grove_get_raw/mod.rs mod v0; }
And each version module provides the actual implementation as a method on
Drive:
#![allow(unused)] fn main() { // grove_get_raw/v0/mod.rs impl Drive { pub(super) fn grove_get_raw_v0<B: AsRef<[u8]>>( &self, path: SubtreePath<'_, B>, key: &[u8], direct_query_type: DirectQueryType, transaction: TransactionArg, drive_operations: &mut Vec<LowLevelDriveOperation>, drive_version: &DriveVersion, ) -> Result<Option<Element>, Error> { // actual implementation match direct_query_type { DirectQueryType::StatelessDirectQuery { /* ... */ } => { // estimate costs } DirectQueryType::StatefulDirectQuery => { let CostContext { value, cost } = self.grove.get_raw(path, key, transaction, &drive_version.grove_version); drive_operations.push(CalculatedCostOperation(cost)); Ok(Some(value.map_err(Error::from)?)) } } } } }
Notice the visibility: pub(super). The v0 function is only visible to its
parent module (the dispatch file). External code calls the public dispatch
method, never the versioned implementation directly.
The layout is the versioning contract made physical, and three rules follow from it:
- One directory per generation. A behaviour change to a versioned method
is a new
v1/(orv2/, ...) directory with its ownmod.rs, plus a new match arm. An edit inside a shippedv0/is allowed only when it cannot modify consensus at any protocol version that selectsv0/, because the code it adds is unreachable there by construction or its output is identical; the edited lines say why, and the pull request description carries an "In-place changes to shipped generations" section (see the coding conventions). Anif platform_version.protocol_version >= 14insidev0/is not that: it is a runtime check the reader has to trust, so it gets a generation. - Inside a generation, a capability is a constant fact, not a check. If
v1admits a new keyword,v1admits it unconditionally (Index::try_from_value_map(map, true)). The decision of whether the keyword is allowed was made by the table that selectedv1. Old generations cannot reach the new path at all, so there is nothing for them to check. - Start the new generation as a copy of the old one. Duplicate
v0/intov1/, rename the function, make the change, and move the tests that exercise the new behaviour across. Duplication between generations is the accepted cost; a shared helper with a flag is the thing it replaces.
When new behaviour lives in a helper reached from several generations (a
value walker, a property-reference resolver), the helper itself becomes a
versioned method: an OptionalFeatureVersion slot in the tables (None for
versions that predate the feature, Some(0) to dispatch to _v0), and the
helper takes &PlatformVersion. A bool on a shared context struct is the
wrong shape, because it moves the version decision from the tables to whoever
set the flag. Per-generation grammar constants may live on a generation-owned
struct; feature gates reachable from more than one generation may not.
Tests live with the generation they test: a #[cfg(test)] mod tests at the
bottom of vN/mod.rs, or vN/tests/ when it grows. The dispatcher's mod.rs
may carry end-to-end tests that need every generation.
For state transitions in Drive ABCI, the same pattern applies but with trait implementations:
packages/rs-drive-abci/src/execution/validation/state_transition/
state_transitions/
identity_create/
mod.rs # dispatch traits and match statements
basic_structure/
mod.rs # just declares v0
v0/
mod.rs # BasicStructureValidationV0 implementation
advanced_structure/
mod.rs
v0/
mod.rs
state/
mod.rs
v0/
mod.rs
The Error Types
There are two UnknownVersionMismatch error variants in the codebase -- one
for Drive and one for Drive ABCI -- but they have the same shape:
#![allow(unused)] fn main() { // packages/rs-drive/src/error/drive.rs #[derive(Debug, thiserror::Error)] pub enum DriveError { #[error("drive unknown version on {method}, received: {received}")] UnknownVersionMismatch { method: String, known_versions: Vec<FeatureVersion>, received: FeatureVersion, }, #[error("{method} not active for drive version")] VersionNotActive { method: String, known_versions: Vec<FeatureVersion>, }, // ... } }
#![allow(unused)] fn main() { // packages/rs-drive-abci/src/error/execution.rs #[derive(Debug, thiserror::Error)] pub enum ExecutionError { #[error("platform unknown version on {method}, received: {received}")] UnknownVersionMismatch { method: String, known_versions: Vec<FeatureVersion>, received: FeatureVersion, }, #[error("{method} not active for drive version")] VersionNotActive { method: String, known_versions: Vec<FeatureVersion>, }, // ... } }
Both carry three pieces of information:
method: A human-readable name identifying which dispatch failed.known_versions: The versions this binary knows how to handle.received: The version number that was actually in the platform version.
This makes the error message self-diagnosing. If you see "drive unknown version on update_contract, received: 2, known versions: [0, 1]", you immediately know that the binary is too old to handle the active protocol version.
How to Add a New Version: Step by Step
Let us walk through the exact steps to add a v1 implementation of a method
that currently only has v0. We will use a fictional example:
my_grove_operation.
Step 1: Write the new implementation
Create the v1 module:
my_grove_operation/
mod.rs # existing dispatch
v0/
mod.rs # existing v0
v1/
mod.rs # NEW: v1 implementation
#![allow(unused)] fn main() { // my_grove_operation/v1/mod.rs impl Drive { pub(super) fn my_grove_operation_v1( &self, // same signature as v0, or possibly different ) -> Result<SomeResult, Error> { // new implementation with bug fix or feature } } }
Step 2: Update the dispatch
In my_grove_operation/mod.rs, declare the new module and add the match arm:
#![allow(unused)] fn main() { mod v0; mod v1; // NEW impl Drive { pub fn my_grove_operation( &self, // ... drive_version: &DriveVersion, ) -> Result<SomeResult, Error> { match drive_version.grove_methods.basic.my_grove_operation { 0 => self.my_grove_operation_v0(/* ... */), 1 => self.my_grove_operation_v1(/* ... */), // NEW version => Err(Error::Drive(DriveError::UnknownVersionMismatch { method: "my_grove_operation".to_string(), known_versions: vec![0, 1], // UPDATED received: version, })), } } } }
Step 3: Bump the slot in the unreleased protocol version's tables
The new arm is dead until a table selects it. Which table you edit depends on whether the unreleased protocol version already owns one.
At the time of writing the latest released version is 13 and version 14 is in
development. PLATFORM_V14 already references DRIVE_VERSION_V9, which was
created for version 14, so a version-14 change edits v9.rs in place:
#![allow(unused)] fn main() { // drive_versions/v9.rs (existing file, amended) pub const DRIVE_VERSION_V9: DriveVersion = DriveVersion { // ... grove_methods: DRIVE_GROVE_METHOD_VERSIONS_V2, // changed in v9: my_grove_operation v1 // ... }; }
DRIVE_GROVE_METHOD_VERSIONS_V1 is referenced by released versions, so it
cannot be edited. Create the next one with struct update syntax:
#![allow(unused)] fn main() { // drive_grove_method_versions/v2.rs (NEW file) /// Differs from v1 in one slot: `basic.my_grove_operation` is 1 rather /// than 0. v1 of the operation <what changed and why>. pub const DRIVE_GROVE_METHOD_VERSIONS_V2: DriveGroveMethodVersions = DriveGroveMethodVersions { basic: DriveGroveBasicMethodVersions { my_grove_operation: 1, ..DRIVE_GROVE_METHOD_VERSIONS_V1.basic }, ..DRIVE_GROVE_METHOD_VERSIONS_V1 }; }
Had DRIVE_VERSION_V9 also been shared with a released version, the same
logic would apply one level up: a new drive_versions/v10.rs pointing at
DRIVE_GROVE_METHOD_VERSIONS_V2, and PLATFORM_V14 pointing at
DRIVE_VERSION_V10. The rule at every level is the same: edit in place if
the constant belongs only to the unreleased version; create the next constant
if a released version references it.
Step 4: Annotate the platform version file
Add or extend the // changed: comment on the affected slot of the unreleased
PLATFORM_V*, and add a numbered item to the file's doc comment describing
the consensus change. That doc comment is the release changelog for the
protocol version.
Step 5: If this is the first change after a release, create the version
Only the first consensus change after a release creates a new protocol version. If version 14 had already shipped, the change above would start version 15:
#![allow(unused)] fn main() { // version/v15.rs (NEW file, copied from v14.rs) pub const PROTOCOL_VERSION_15: ProtocolVersion = 15; /// v15 hosts one consensus change so far: /// /// 1. **my_grove_operation v1**: ... pub const PLATFORM_V15: PlatformVersion = PlatformVersion { protocol_version: PROTOCOL_VERSION_15, drive: DRIVE_VERSION_V10, // changed: my_grove_operation v1 // ... everything else unchanged from V14 }; }
Then register it:
#![allow(unused)] fn main() { // version/mod.rs pub mod v15; pub const LATEST_VERSION: ProtocolVersion = PROTOCOL_VERSION_15; // version/protocol_version.rs pub const PLATFORM_VERSIONS: &[PlatformVersion] = &[ PLATFORM_V1, // ... PLATFORM_V14, PLATFORM_V15, // NEW ]; pub const LATEST_PLATFORM_VERSION: &PlatformVersion = &PLATFORM_V15; }
A test in system_limits/mod.rs asserts that PLATFORM_VERSIONS.len()
equals LATEST_VERSION, so a version that is declared but not registered
fails the test run rather than silently resolving to the previous one. Every
subsequent change destined for version 15 amends v15.rs and the constants it
introduced, as in step 3.
Step 6: Write tests
Test both generations through the dispatcher, and put each test with the generation it exercises:
#![allow(unused)] fn main() { // my_grove_operation/v1/mod.rs #[cfg(test)] mod tests { #[test] fn should_apply_new_behaviour() { let platform_version = PlatformVersion::latest(); // ... drive.my_grove_operation(..., &platform_version.drive) } } // my_grove_operation/v0/mod.rs #[cfg(test)] mod tests { #[test] fn should_keep_old_behaviour() { // frozen generation: pin the last protocol version that selected it let platform_version = PlatformVersion::get(13).expect("known version"); // ... } } }
The current generation tests against PlatformVersion::latest() so it keeps
tracking the tip; the moment v1 is introduced is the moment v0's tests get
pinned to an explicit version. Do not write a test that merely asserts the
table slot's value (assert_eq!(PLATFORM_V14.drive.grove_methods.basic.my_grove_operation, 1));
it restates the literal and cannot fail without the edit being deliberate. A
behaviour test that runs both versions through the dispatcher pins the gate
meaningfully.
This is a lot of steps, but each one is mechanical and the compiler guides you through most of it. If you add a field to a version struct and forget to set it in one of the fourteen platform version constants, the build fails.
Passing Version References
A subtle but important convention is which version reference a function receives. There are three patterns:
&PlatformVersion -- used by high-level code that might need any part of
the version tree. State transition processing, block execution, and similar
entry points take this.
&DriveVersion -- used by mid-level Drive code that only needs drive-
specific versions. The caller extracts &platform_version.drive once.
&GroveVersion -- used by the lowest-level GroveDB operations. Extracted
from &drive_version.grove_version.
This layering avoids passing the entire PlatformVersion into the deepest
functions. It also makes the dependency explicit: a function taking
&DriveVersion cannot accidentally use a DPP version number.
The Version Flow in Block Processing
Here is how the version flows through a real execution path:
Block arrives from Tenderdash
|
v
PlatformState has the current protocol_version (e.g., 14)
|
v
PlatformVersion::get(14) -> &PLATFORM_V14
|
v
process_raw_state_transitions(&platform_version)
|
v
validate_state_for_identity_create_transition()
reads: platform_version.drive_abci.validation_and_processing
.state_transitions.identity_create_state_transition.state
dispatches to: validate_state_v0()
|
v
drive.update_contract(&platform_version)
reads: platform_version.drive.methods.contract.update.update_contract
dispatches to: update_contract_v1()
|
v
drive.grove_get_raw(&platform_version.drive)
reads: drive_version.grove_methods.basic.grove_get_raw
dispatches to: grove_get_raw_v0()
The protocol version number enters at the top and the correct implementation is selected at every level. No function chooses its own version -- it is always determined by the version reference passed from above.
The First Block of a New Protocol Version
The version flow above assumes the protocol version is already known. The
switch itself happens in run_block_proposal
(packages/rs-drive-abci/src/execution/engine/run_block_proposal/mod.rs).
On the first block of an epoch, if the protocol version the network locked in
during the previous epoch differs from the one in consensus, the block runs
under the new version:
#![allow(unused)] fn main() { // abbreviated let block_platform_version = if epoch_info.is_epoch_change_but_not_genesis() && platform_state.next_epoch_protocol_version() != platform_state.current_protocol_version_in_consensus() { let next_protocol_version = platform_state.next_epoch_protocol_version(); // We should panic if this node is not supported a new protocol version let Ok(next_platform_version) = PlatformVersion::get(next_protocol_version) else { panic!("Failed to upgrade the network protocol version {next_protocol_version}. ..."); }; let old_protocol_version = block_platform_state.current_protocol_version_in_consensus(); block_platform_state.set_current_protocol_version_in_consensus(next_protocol_version); // This is for events like adding stuff to the root tree, or making structural changes/fixes self.perform_events_on_first_block_of_protocol_change( platform_state, &block_info, transaction, old_protocol_version, next_platform_version, )?; next_platform_version } else { last_committed_platform_version }; }
Three things to take from this:
- Epoch info is computed with the old version, before the switch. The new version applies to everything after it.
- A binary that does not know the new version panics with an upgrade message. That is deliberate: a node that cannot run the agreed protocol must stop rather than produce a divergent state root.
perform_events_on_first_block_of_protocol_changeis where state migrations live. Anything that needs to be done to the tree once for a new protocol version, before its first state transition runs, goes here: creating a new root-tree subtree, rewriting a system contract, back-filling a sum tree.
The migration hook is itself a versioned method
(drive_abci.methods.protocol_upgrade.perform_events_on_first_block_of_protocol_change:
None in the earliest method tables, Some(0) once the first migration
existed, Some(1) in the two most recent tables). Its v0 is a ladder of
guarded rungs, one per protocol version that needed a migration:
#![allow(unused)] fn main() { // packages/rs-drive-abci/src/execution/platform_events/protocol_upgrade/ // perform_events_on_first_block_of_protocol_change/v0/mod.rs if previous_protocol_version < 4 && platform_version.protocol_version >= 4 { self.transition_to_version_4(platform_state, block_info, transaction, platform_version)?; } if previous_protocol_version < 6 && platform_version.protocol_version >= 6 { self.transition_to_version_6(block_info, transaction, platform_version)?; } // ... 8, 9, 11, 12, 13 ... if previous_protocol_version < 14 && platform_version.protocol_version >= 14 { self.transition_to_version_14(block_info, transaction, platform_version)?; } }
The guard shape matters. A node can cross more than one protocol version in a
single switch (a network that skipped a version, or a devnet started at an old
one), and the previous < N && new >= N form runs every rung it crossed, in
order. A rung written as new == N would be skipped by such a node, and its
state tree would be missing a subtree every other node has.
v1 exists because the migrations write system contracts through a path that
bypasses the drive operation batch's cache invalidation; it runs the same
ladder and then refreshes the cached contract definitions so that a validator
with a warm cache and one with a cold cache serialize the same bytes. Read its
doc comment before touching the hook: it is a worked example of a
consensus-critical cache bug.
To add a migration for a new protocol version: add a rung at the bottom of the
ladder in the current generation of the hook, guarded by that version, with a
transition_to_version_N helper next to the others. A failure in a rung
returns Err from block processing on every node, so a rung either succeeds
deterministically or halts the network. The version 8 rung's or_else that
logs and continues is the exception for a migration that does not touch the
state structure, not a pattern to copy.
Whole new state transition kinds are gated separately, by the is_allowed
stage of the validation pipeline reading the constants in
feature_initial_protocol_versions.rs
(ADDRESS_FUNDS_INITIAL_PROTOCOL_VERSION = 11,
SHIELDED_POOL_INITIAL_PROTOCOL_VERSION = 12). A transition submitted before
its initial version is rejected with a StateTransitionNotActiveError rather
than an unknown-version dispatch error.
Genesis content is the other place a protocol_version comparison is the
intended shape. create_genesis_state runs once per chain, under the protocol
version the chain is born at. Mainnet and testnet were born at protocol
version 1 and replay generation 0, which stays frozen. Generation 1 is selected
only for chains born at protocol version 9 or later, and every such chain is a
devnet, a local network or a test suite that is created again for each
release, so no live node reproduces its genesis. When a protocol version adds
a system contract or other genesis content, it goes into
create_genesis_state_v1 behind if platform_version.protocol_version >= N
(document history at 13, app connect and moderation charters at 14), and a
chain that already exists gets the same content from its
transition_to_version_N rung. A new genesis generation would add code for no
replay benefit. Never edit generation 0.
The Drive helpers that build the initial state structure follow the same rule
for the same reason: they run once, at chain creation, under the chain's
initial protocol version, and a chain that already exists gets the same trees
from its upgrade rung. add_initial_withdrawal_state_structure_operations
adds the withdrawal sum trees behind >= 4, which replaying mainnet's genesis at
protocol version 1 does not take. The withdrawal limit trees of protocol version
14 are not in that batch: genesis and transition_to_version_14 both add them
one insert at a time through Drive::insert_withdrawal_limit_trees, so both
build the withdrawals Merk in the same shape.
Rules
Do:
- Always include the method name in the
UnknownVersionMismatcherror. Use the same string format you see in existing code: the plain method name for Drive methods ("grove_get_raw"), and a descriptive path for ABCI methods ("identity create transition: validate_basic_structure"). - Keep the
known_versionsvector in the error arm up to date. When you add version 2, the vector should bevec![0, 1, 2]. - Make versioned implementation methods
pub(super)-- visible to the dispatch module but not to external code. - Put every new generation in its own
vN/directory, started as a copy of the previous one, with its tests inside it. - Bump the slot in the unreleased protocol version's tables only: amend a constant that only the unreleased version references, create the next constant when a released version references it.
- Test the current generation against
PlatformVersion::latest()and pin the previous generation's tests to the last protocol version that selected it. - Put one-time state changes a protocol version needs in a guarded rung of
perform_events_on_first_block_of_protocol_change.
Do not:
- Never call a versioned implementation directly (e.g.,
grove_get_raw_v0). Always go through the dispatch method. Direct calls bypass version control and break determinism. - Never add a version to the match without also adding the corresponding
FeatureVersionfield value in the version constants. The dispatch will never be reached if no platform version sets that number. - Never use
_ =>as the catch-all arm in a version dispatch. Always useversion =>so the variable is available for the error message. And never silently ignore unknown versions -- always return an error. - Never modify a shipped generation, not even by threading a parameter or a version check through it. If v0 takes five parameters and v1 needs six, that is fine -- v0 keeps its original signature and body forever.
- Never gate new behaviour with a
boolon a shared context struct. A helper reached from several generations gets anOptionalFeatureVersionslot and takes&PlatformVersion. - Never write a test that only asserts a table slot's value. Test the behaviour through the dispatcher on both sides of the gate.
The State Transition Lifecycle
Every change to Dash Platform -- creating an identity, registering a data contract, storing a document, casting a masternode vote -- follows the same fundamental pattern: a state transition. If you understand this one concept, you understand the heartbeat of the entire platform.
What Is a State Transition?
A state transition is the atomic unit of state change on Dash Platform. Think of the platform's state as a database. You cannot write to that database directly. Instead, you construct a state transition object that describes what you want to change, sign it with your private key, serialize it to bytes, and broadcast it to the network. Validators receive it, validate it through a multi-stage pipeline, and -- if everything checks out -- apply it to their copy of the state.
This is fundamentally different from a smart contract model. There is no arbitrary code execution. Every possible mutation is one of a fixed set of state transition types, each with its own validation rules hardcoded into the platform. The benefit is predictability: you can reason about fees, security, and correctness without worrying about Turing-complete execution.
The StateTransition Enum
At the Rust level, every state transition is a variant of a single enum. This is defined
in packages/rs-dpp/src/state_transition/mod.rs:
#![allow(unused)] fn main() { #[derive(Debug, Clone, Encode, Decode, PlatformSerialize, PlatformDeserialize, PlatformSignable, From, PartialEq)] #[platform_serialize(unversioned)] #[platform_serialize(limit = 100000)] pub enum StateTransition { DataContractCreate(DataContractCreateTransition), DataContractUpdate(DataContractUpdateTransition), Batch(BatchTransition), IdentityCreate(IdentityCreateTransition), IdentityTopUp(IdentityTopUpTransition), IdentityCreditWithdrawal(IdentityCreditWithdrawalTransition), IdentityUpdate(IdentityUpdateTransition), IdentityCreditTransfer(IdentityCreditTransferTransition), MasternodeVote(MasternodeVoteTransition), IdentityCreditTransferToAddresses(IdentityCreditTransferToAddressesTransition), IdentityCreateFromAddresses(IdentityCreateFromAddressesTransition), IdentityTopUpFromAddresses(IdentityTopUpFromAddressesTransition), AddressFundsTransfer(AddressFundsTransferTransition), AddressFundingFromAssetLock(AddressFundingFromAssetLockTransition), AddressCreditWithdrawal(AddressCreditWithdrawalTransition), Shield(ShieldTransition), ShieldedTransfer(ShieldedTransferTransition), Unshield(UnshieldTransition), ShieldFromAssetLock(ShieldFromAssetLockTransition), ShieldedWithdrawal(ShieldedWithdrawalTransition), IdentityCreateFromShieldedPool(IdentityCreateFromShieldedPoolTransition), ShieldFromIdentity(ShieldFromIdentityTransition), IdentityTopUpFromShieldedPool(IdentityTopUpFromShieldedPoolTransition), IdentityKeyLimitsUpdate(IdentityKeyLimitsUpdateTransition), } }
Each variant wraps a dedicated struct. Notice the derive macros: Encode and Decode
for bincode serialization, PlatformSerialize and PlatformDeserialize for the
platform's own serialization layer, and PlatformSignable for generating the
"signable bytes" that get signed.
These variants fall into natural groups:
Identity lifecycle:
IdentityCreate-- Register a new identity (funded by an asset lock on the Dash core chain)IdentityTopUp-- Add credits to an existing identity (also asset-lock funded)IdentityUpdate-- Add or disable public keys on an identityIdentityCreditWithdrawal-- Withdraw credits back to the core chainIdentityCreditTransfer-- Transfer credits between identitiesIdentityKeyLimitsUpdate-- Raise the budget or extend the expiry of one of the identity's keys
Data contracts and documents:
DataContractCreate-- Register a new data contract (schema)DataContractUpdate-- Update an existing data contractBatch-- Create, replace, delete, or transfer documents; mint, burn, transfer, or freeze tokens
Governance:
MasternodeVote-- Cast a vote in a contested resource election
Address-based (newer):
IdentityCreateFromAddresses,IdentityTopUpFromAddresses,AddressFundsTransfer,AddressFundingFromAssetLock,AddressCreditWithdrawal-- Operations that use platform addresses instead of (or in addition to) identity-based authentication
Shielded pool:
Shield,ShieldedTransfer,Unshield,ShieldFromAssetLock,ShieldedWithdrawal,IdentityCreateFromShieldedPool,ShieldFromIdentity,IdentityTopUpFromShieldedPool-- Operations that move credits into, inside, and out of the shielded pool
Each variant has its own numeric discriminant, defined in
packages/rs-dpp/src/state_transition/state_transition_types.rs:
#![allow(unused)] fn main() { #[repr(u8)] pub enum StateTransitionType { DataContractCreate = 0, Batch = 1, IdentityCreate = 2, IdentityTopUp = 3, DataContractUpdate = 4, IdentityUpdate = 5, IdentityCreditWithdrawal = 6, IdentityCreditTransfer = 7, MasternodeVote = 8, IdentityCreditTransferToAddresses = 9, IdentityCreateFromAddresses = 10, IdentityTopUpFromAddresses = 11, AddressFundsTransfer = 12, AddressFundingFromAssetLock = 13, AddressCreditWithdrawal = 14, Shield = 15, ShieldedTransfer = 16, Unshield = 17, ShieldFromAssetLock = 18, ShieldedWithdrawal = 19, IdentityCreateFromShieldedPool = 20, ShieldFromIdentity = 21, IdentityTopUpFromShieldedPool = 22, IdentityKeyLimitsUpdate = 23, } }
This type tag is what allows the platform to deserialize a raw byte buffer into the correct variant. When bytes arrive over the wire, the first byte tells the deserializer which struct to decode into.
The Batch Transition: A Swiss Army Knife
The Batch variant deserves special attention because it is the most complex. A single
BatchTransition can contain multiple sub-transitions, each operating on a different
document or token. The sub-transitions include:
- Document operations: Create, Replace, Delete, Transfer, UpdatePrice, Purchase
- Token operations: Transfer, Mint, Burn, Freeze, Unfreeze, DestroyFrozenFunds, EmergencyAction, ConfigUpdate, Claim, DirectPurchase, SetPriceForDirectPurchase
This batching is important for atomicity: either all operations in the batch succeed or none of them do. It also means a single identity nonce covers the entire batch, preventing partial replay attacks.
Signatures and Authentication
State transitions carry cryptographic signatures that prove authorization. There are two fundamentally different authentication models:
Identity-signed transitions -- The majority of transition types. The signer is an
identity that already exists on the platform. The transition carries a
signature_public_key_id referencing a key in the identity's key set, plus a signature
over the signable bytes.
Asset-lock transitions -- IdentityCreate and IdentityTopUp are special because
the identity may not exist yet (or the transition does not require an identity signature).
Instead, they carry an AssetLockProof that references a transaction on the Dash core
chain. The signature proves ownership of the funds being locked.
Address-based transitions -- Newer transition types like IdentityCreateFromAddresses
use platform address inputs with their own nonces and balances, rather than identity-based
authentication.
The sign method on StateTransition handles this:
#![allow(unused)] fn main() { pub fn sign( &mut self, identity_public_key: &IdentityPublicKey, private_key: &[u8], bls: &impl BlsModule, ) -> Result<(), ProtocolError> { ... } }
It first verifies that the key's purpose and security level are appropriate for
this transition type, then signs the signable_bytes() with the private key.
Supported key types are ECDSA_SECP256K1, ECDSA_HASH160, and BLS12_381.
The signable bytes are produced by the PlatformSignable derive macro. It serializes
the transition with the signature field zeroed out, producing a deterministic byte
sequence that both client and platform can independently compute.
Serialization
The platform uses bincode for wire serialization of state transitions, wrapped
in the PlatformSerialize / PlatformDeserialize layer. This gives us:
- Compact binary encoding (no field names, no JSON overhead)
- Deterministic output (critical for signature verification)
- Size limits (
#[platform_serialize(limit = 100000)]-- 100KB max) - Version awareness through the
#[platform_serialize(unversioned)]annotation
Deserialization includes version-range checks:
#![allow(unused)] fn main() { pub fn deserialize_from_bytes_in_version( bytes: &[u8], platform_version: &PlatformVersion, ) -> Result<Self, ProtocolError> { let state_transition = StateTransition::deserialize_from_bytes(bytes)?; let active_version_range = state_transition.active_version_range(); if active_version_range.contains(&platform_version.protocol_version) { Ok(state_transition) } else { Err(ProtocolError::StateTransitionError( StateTransitionIsNotActiveError { ... }, )) } } }
This ensures that a transition type introduced in protocol version 9 cannot be submitted to a node running protocol version 8.
The Full Journey
Here is the end-to-end lifecycle of a state transition, from a client's perspective all the way through to state application:
1. Client constructs the transition. Using the Rust SDK (rs-sdk), JavaScript
SDK (js-dash-sdk), or any client library, the application builds a concrete transition
struct -- for example, a DataContractCreateTransition containing the new contract's schema.
2. Client signs the transition. The client calls sign() or sign_external(),
which computes signable bytes, signs them with the appropriate private key, and attaches
the signature and key ID to the transition struct.
3. Client serializes and broadcasts. The signed StateTransition is serialized
to bytes via serialize_to_bytes() and sent to the network through DAPI (the
Decentralized API).
4. Platform receives the bytes in check_tx. Before a state transition enters the
mempool, it goes through a lighter validation pass. The platform deserializes the bytes,
checks the signature, verifies the identity has sufficient balance, and validates basic
structure. This is a gatekeeper -- it rejects obviously invalid transitions cheaply.
5. Platform processes during process_proposal. When a block is proposed, each
state transition goes through the full validation pipeline: is_allowed, signature
verification, nonce validation, basic structure, balance checks, advanced structure,
transform_into_action, and state validation. (We will cover this pipeline in detail in
the next chapter.)
6. Platform applies the action. If validation succeeds, the resulting
StateTransitionAction is converted into DriveOperations, which are converted
into LowLevelDriveOperations, which are applied atomically to GroveDB. Fees are
calculated and deducted. The state has changed.
7. Result is returned. The client receives confirmation (or rejection) through DAPI.
The Versioning Pattern
You will notice that almost every method on StateTransition dispatches through
a version table. For example:
#![allow(unused)] fn main() { match platform_version .drive_abci .validation_and_processing .process_state_transition { 0 => v0::process_state_transition_v0(...), version => Err(Error::Execution(ExecutionError::UnknownVersionMismatch { ... })), } }
This is the protocol versioning pattern. Every behavior that could conceivably change
between protocol versions is gated behind a version number looked up from
PlatformVersion. This means a node running protocol version 9 and a node running
protocol version 10 can both validate the same block correctly, each using its own
version's logic. It is how the platform achieves hard-fork-free upgrades.
The call_method Macro
Since StateTransition is an enum with 15 variants, dispatching a method call to
the inner type would require writing out a 15-arm match statement every time. The
codebase solves this with a family of macros:
#![allow(unused)] fn main() { macro_rules! call_method { ($state_transition:expr, $method:ident) => { match $state_transition { StateTransition::DataContractCreate(st) => st.$method(), StateTransition::DataContractUpdate(st) => st.$method(), StateTransition::Batch(st) => st.$method(), // ... all 15 variants } }; } }
There are several variants: call_method for universal methods,
call_getter_method_identity_signed for methods that return Option (returning None
for non-identity-signed transitions like IdentityCreate), and
call_errorable_method_identity_signed for methods that return Result (returning
an error for inapplicable variants).
Rules and Guidelines
Do:
- Always deserialize with
deserialize_from_bytes_in_versionin production code, to enforce version-range checks. - Use
signable_bytes()when computing what to sign or verify -- never serialize the full transition (which includes the signature itself). - Check
active_version_range()before processing a transition to ensure it is valid for the current protocol version.
Do not:
- Assume all transitions have signatures.
IdentityCreateFromAddressesandAddressFundsTransferreturnNonefromsignature(). - Assume all transitions have an
owner_id. Address-based transitions do not. - Modify the
StateTransitionTypediscriminant values -- they are part of the wire format and changing them would break all existing serialized data. - Add new variants without also updating every
call_methodmacro and every match statement in the validation pipeline.
The Validation Pipeline
When a state transition arrives at a Dash Platform node, it does not get applied immediately. It must survive a gauntlet of validation stages, each designed to catch a different category of error. Understanding this pipeline is essential to understanding how the platform maintains security, prevents abuse, and ensures deterministic state across all validators.
Two Entry Points: check_tx and process_proposal
State transitions enter the pipeline through two different doors, depending on when they are being validated.
check_tx runs when a transition first arrives at a node (before entering the mempool) and again when rechecking mempool contents. It performs a lighter validation: signature verification, basic structure, and balance checks. The goal is to filter out obvious garbage cheaply, without doing expensive state lookups.
A MasternodeVote is the exception. A block refuses a failed vote without charging anyone,
and the proposer drops it from the block without a trace, so check_tx runs the vote's advanced
structure and state validation as well. A vote that a block would refuse, such as a Lock vote
on a contest without locking, is refused when it is broadcast, with its error.
This is implemented in
packages/rs-drive-abci/src/execution/validation/state_transition/check_tx_verification/mod.rs:
#![allow(unused)] fn main() { pub(in crate::execution) fn state_transition_to_execution_event_for_check_tx<'a, C: CoreRPCLike>( platform: &'a PlatformRef<C>, state_transition: StateTransition, check_tx_level: CheckTxLevel, platform_version: &PlatformVersion, ) -> Result<ConsensusValidationResult<Option<ExecutionEvent<'a>>>, Error> { ... } }
process_proposal (also called the "Validator" path) runs during block execution.
This is the full pipeline. It is implemented in
packages/rs-drive-abci/src/execution/validation/state_transition/processor/v0/mod.rs:
#![allow(unused)] fn main() { pub(super) fn process_state_transition_v0<'a, C: CoreRPCLike>( platform: &'a PlatformRef<C>, block_info: &BlockInfo, state_transition: StateTransition, transaction: TransactionArg, platform_version: &PlatformVersion, ) -> Result<ConsensusValidationResult<ExecutionEvent<'a>>, Error> { ... } }
Let us walk through the full pipeline, stage by stage.
Stage 1: Is Allowed
Some state transition types are only available starting from a certain protocol version.
For example, address-based transitions like IdentityCreateFromAddresses require
protocol version 11 or higher. The first check asks: is this transition type even
permitted on the current network?
#![allow(unused)] fn main() { if state_transition.has_is_allowed_validation()? { let result = state_transition.validate_is_allowed(platform_version)?; if !result.is_valid() { return Ok(ConsensusValidationResult::new_with_errors(result.errors)); } } }
The trait is defined in
packages/rs-drive-abci/src/execution/validation/state_transition/processor/traits/is_allowed.rs:
#![allow(unused)] fn main() { pub(crate) trait StateTransitionIsAllowedValidationV0 { fn has_is_allowed_validation(&self) -> Result<bool, Error>; fn validate_is_allowed( &self, platform_version: &PlatformVersion, ) -> Result<ConsensusValidationResult<()>, Error>; } }
Transitions available since protocol version 1, like DataContractCreate,
IdentityCreate and Batch, skip this check entirely.
Stage 2: Identity Signature Verification
For identity-signed transitions (everything except IdentityCreate, IdentityTopUp,
and the address-based types), the platform fetches the signer's identity from state
and verifies the signature against one of its registered public keys.
#![allow(unused)] fn main() { let mut maybe_identity = if state_transition.uses_identity_in_state() { let result = if state_transition.validates_signature_based_on_identity_info() { state_transition.validate_identity_signed_state_transition( platform.drive, transaction, &mut state_transition_execution_context, platform_version, ) } else { state_transition.retrieve_identity_info(...) }?; if !result.is_valid() { return Ok(ConsensusValidationResult::new_with_errors(result.errors)); } Some(result.into_data()?) } else { None }; }
If the signature does not match, the transition is rejected without charging a fee. This is critical: if someone forges a transition with your identity ID but a wrong signature, you should not pay for their garbage.
Stage 3: Address Witness Validation
For address-based transitions, instead of identity signatures, the platform validates witnesses (proofs of ownership) for the input addresses:
#![allow(unused)] fn main() { if state_transition.has_address_witness_validation(platform_version)? { let result = state_transition.validate_address_witnesses( &mut state_transition_execution_context, platform_version, )?; if !result.is_valid() { return Ok(ConsensusValidationResult::new_with_errors(result.errors)); } } }
Stage 4: Address Balances and Nonces
For transitions that use platform addresses as inputs, the platform verifies that each input address has sufficient balance and that the provided nonces are correct (preventing replay):
#![allow(unused)] fn main() { let remaining_address_balances = if state_transition.has_addresses_balances_and_nonces_validation() { let result = state_transition.validate_address_balances_and_nonces( platform.drive, &mut state_transition_execution_context, transaction, platform_version, )?; if !result.is_valid() { return Ok(ConsensusValidationResult::new_with_errors(result.errors)); } Some(result.into_data()?) } else { None }; }
Stage 5: Identity Nonce Validation
Nonces prevent replay attacks. Each identity-signed transition carries a nonce that must be strictly greater than the last used nonce for that identity (or identity-contract pair). The platform checks this against the stored nonce in state.
#![allow(unused)] fn main() { if state_transition.has_identity_nonce_validation(platform_version)? { let result = state_transition.validate_identity_nonces( &platform.into(), platform.state.last_block_info(), transaction, &mut state_transition_execution_context, platform_version, )?; if !result.is_valid() { return Ok(ConsensusValidationResult::new_with_errors(result.errors)); } } }
Identity create and identity top up skip this check -- they do not have nonces because the identity may not exist yet.
Stage 6: Basic Structure Validation
This stage checks that the transition's data is well-formed without looking at platform state. For example: are all required fields present? Are field values within allowed ranges? Is the data contract schema valid JSON Schema?
#![allow(unused)] fn main() { if state_transition.has_basic_structure_validation(platform_version) { let consensus_result = state_transition.validate_basic_structure( platform.config.network, platform_version, )?; if !consensus_result.is_valid() { return Ok(ConsensusValidationResult::new_with_errors( consensus_result.errors, )); } } }
The trait lives in processor/traits/basic_structure.rs:
#![allow(unused)] fn main() { pub(crate) trait StateTransitionBasicStructureValidationV0 { fn validate_basic_structure( &self, network_type: Network, platform_version: &PlatformVersion, ) -> Result<SimpleConsensusValidationResult, Error>; fn has_basic_structure_validation(&self, _platform_version: &PlatformVersion) -> bool { true } } }
MasternodeVote skips basic structure validation entirely. DataContractCreate and
DataContractUpdate conditionally enable it based on the platform version.
Stage 7: Balance Pre-Check
Before doing expensive state validation, the platform checks that the identity has enough credits to plausibly pay for this transition. For credit transfers and withdrawals, this includes checking that the transfer amount plus estimated fees does not exceed the balance.
#![allow(unused)] fn main() { if state_transition.has_identity_minimum_balance_pre_check_validation() { let result = state_transition.validate_identity_minimum_balance_pre_check( identity, platform_version, )?; if !result.is_valid() { return Ok(ConsensusValidationResult::new_with_errors(result.errors)); } } }
A MasternodeVote is paid by its vote poll's prefunded specialized balance, not by
the voter, so its pre-check is on that pot: a vote on a poll whose pot does not exist,
or holds less than the single vote cost, is refused unpaid with
PrefundedSpecializedBalanceNotFoundError or
PrefundedSpecializedBalanceInsufficientError.
Stage 8: Advanced Structure Validation (without State)
Some transitions need structural validation that goes beyond basic checks but does not
require reading from platform state. For example, IdentityUpdate verifies that
added public keys have valid signatures. DataContractCreate validates the contract
schema more deeply.
#![allow(unused)] fn main() { if state_transition.has_advanced_structure_validation_without_state() { let consensus_result = state_transition.validate_advanced_structure( identity, &mut state_transition_execution_context, platform_version, )?; if !consensus_result.is_valid() { // Note: this returns an action so the nonce gets bumped even on failure return consensus_result.map_result(|action| { ExecutionEvent::create_from_state_transition_action(...) }); } } }
An important detail: if advanced structure validation fails, the identity's nonce is still bumped. This prevents an attacker from replaying a structurally invalid transition over and over without cost.
Stage 9: Transform Into Action + Advanced Structure (with State)
For certain transition types (Batch, IdentityCreate, MasternodeVote,
AddressFundingFromAssetLock, IdentityCreateFromAddresses), the platform needs
to read state before it can finish structure validation. For example, a document
create transition needs to fetch the data contract to validate the document against
its schema.
This stage first transforms the raw state transition into an action (reading state in the process), then validates the action's structure:
#![allow(unused)] fn main() { let action = if state_transition.has_advanced_structure_validation_with_state() { let state_transition_action_result = state_transition.transform_into_action( platform, block_info, &remaining_address_balances, ValidationMode::Validator, &mut state_transition_execution_context, transaction, )?; if !state_transition_action_result.is_valid_with_data() { return state_transition_action_result.map_result(|action| { ... }); } let action = state_transition_action_result.into_data()?; let result = state_transition.validate_advanced_structure_from_state( block_info, platform.config.network, &action, maybe_identity.as_ref(), &mut state_transition_execution_context, platform_version, )?; if !result.is_valid() { return result.map_result(|action| { ... }); } Some(action) } else { None }; }
We will cover transform_into_action in detail in the next chapter.
For IdentityCreateFromAddresses this stage also checks the proof of possession of each key
the transition registers. The address witnesses cannot sign those signatures (both sign the
same bytes), so the owners of the inputs never signed what this check judges. From protocol
version 14 a failure is therefore refused without a fee, like a failed witness, instead of
charging the inputs a penalty; check_tx runs this stage for the transition too, so such a
transition does not reach a block.
Stage 10: State Validation
The final validation stage checks for state-level conflicts. Does a data contract with this ID already exist? Is there already a document with this unique index value? Has this asset lock already been spent?
#![allow(unused)] fn main() { let result = if state_transition.has_state_validation() { state_transition.validate_state( action, platform, ValidationMode::Validator, block_info, &mut state_transition_execution_context, transaction, )? } else if let Some(action) = action { ConsensusValidationResult::new_with_data(action) } else { state_transition.transform_into_action(...)? // For transitions that skipped earlier }; }
Not all transitions need state validation. IdentityTopUp, IdentityCreditWithdrawal,
AddressFundsTransfer, and several others skip it -- their validation is fully covered
by the earlier stages.
ConsensusValidationResult: How Errors Accumulate
Throughout the pipeline, errors are communicated through ConsensusValidationResult,
defined in packages/rs-dpp/src/validation/validation_result.rs:
#![allow(unused)] fn main() { pub type ConsensusValidationResult<TData> = ValidationResult<TData, ConsensusError>; pub type SimpleConsensusValidationResult = ConsensusValidationResult<()>; pub struct ValidationResult<TData: Clone, E: Debug> { pub errors: Vec<E>, pub data: Option<TData>, } }
Key properties of this type:
- It can carry data alongside errors. When
transform_into_actionpartially succeeds but has warnings, the action is indataand the warnings are inerrors. is_valid()checks iferrorsis empty. Any error means failure.is_valid_with_data()checks both. Valid and has associated data.- Errors can be merged from multiple validation steps using
add_errors()ormerge(). - Chainable via
and_then_validation()for pipeline-style composition.
The SimpleConsensusValidationResult alias (where TData = ()) is used by validation
stages that just check pass/fail without producing a transformed object.
ValidationMode
The pipeline behavior varies based on context:
#![allow(unused)] fn main() { pub enum ValidationMode { CheckTx, // Mempool admission -- lighter checks RecheckTx, // Periodic mempool revalidation Validator, // Full block execution NoValidation, // Testing/tooling only } }
For example, should_fully_validate_contract_on_transform_into_action() returns
true only in Validator mode. During CheckTx, the platform skips expensive
contract validation to keep mempool admission fast.
The Early-Return Pattern
You will notice a consistent pattern throughout the pipeline: each stage checks
is_valid() and returns early if validation failed. This is intentional.
When the pipeline returns early due to a signature failure or invalid nonce, the transition produces no execution event -- the user is not charged. This protects users from being billed for transitions they did not actually submit (forged signatures) or that are replays (invalid nonces).
But when the pipeline returns early after the nonce has been validated -- for example, during advanced structure validation or state validation -- it returns a nonce-bump action so the identity still pays a small fee. This prevents a different attack: submitting thousands of structurally invalid transitions to waste validator resources for free.
Rules and Guidelines
Do:
- Run the full pipeline in
Validatormode during block execution. Skipping stages can lead to consensus divergence. - Return early on signature/nonce failures without charging the user.
- Bump the nonce (and charge) when structure or state validation fails after the nonce check has passed.
- Use
ConsensusValidationResulteverywhere -- do not useResult<(), Error>for validation outcomes, because a validation failure is not a system error.
Do not:
- Add expensive state reads to basic structure validation. It runs during
check_txand must be fast. - Skip
is_allowedvalidation -- it is the mechanism that enables protocol upgrades to gate new transition types. - Assume the pipeline order is arbitrary. Each stage depends on information established by previous stages (e.g., the identity fetched during signature validation is used in balance checks).
- Confuse
ConsensusError(a validation failure that the user caused) withError(a system error like a database failure). The former accumulates inValidationResult::errors; the latter propagates viaResult::Err.
Transform Into Action
After a state transition passes the early validation stages -- signature verification, nonce checks, basic structure -- the platform needs to translate it from a raw "what the client sent" representation into a "what the platform should do" representation. This translation step is called transform into action, and it is where the platform reads state, resolves references, and prepares a validated, self-contained instruction for the Drive storage layer.
Why Actions Exist
Consider a DataContractCreateTransition. The client sends the contract in its
serialized form. But before the platform can store it, it needs to:
- Deserialize the contract from its wire format
- Validate the contract schema (JSON Schema validation, index rules, etc.)
- Compute the contract ID from the owner ID and nonce
- Check version compatibility
The transition is what the client sends. The action is what the platform will execute. The action is a validated, resolved, ready-to-apply object. It has already been through deserialization, its references have been resolved against current state, and it carries exactly the information needed to generate Drive operations.
This separation is a deliberate design choice. The transition type lives in rs-dpp
(the protocol library, shared between client and platform). The action type lives in
rs-drive (the storage layer, platform-only). By keeping them separate:
- Clients never need to depend on storage internals
- The platform can evolve its internal representation without changing the wire format
- Validation logic lives close to state access, not scattered across the protocol layer
The StateTransitionActionTransformer Trait
The transformation is defined by a trait in
packages/rs-drive-abci/src/execution/validation/state_transition/transformer/mod.rs:
#![allow(unused)] fn main() { pub trait StateTransitionActionTransformer { fn transform_into_action<C: CoreRPCLike>( &self, platform: &PlatformRef<C>, block_info: &BlockInfo, remaining_address_input_balances: &Option< BTreeMap<PlatformAddress, (AddressNonce, Credits)>, >, validation_mode: ValidationMode, execution_context: &mut StateTransitionExecutionContext, tx: TransactionArg, ) -> Result<ConsensusValidationResult<StateTransitionAction>, Error>; } }
The key parameters:
platform-- Access to the full platform state, including Drive (the storage layer), the current platform state, and configuration.block_info-- The current block height, time, epoch. Some validations are time-dependent.remaining_address_input_balances-- For address-based transitions, the balances remaining after earlier deductions.validation_mode-- Controls how thorough the validation is (lighter forCheckTx, full forValidator).execution_context-- Accumulates information about what operations were performed during validation (used for fee estimation).tx-- The GroveDB transaction handle for consistent state reads.
The return type is ConsensusValidationResult<StateTransitionAction> -- it can carry
both the resulting action and any validation errors encountered during transformation.
The Top-Level Dispatch
The StateTransition enum implements StateTransitionActionTransformer by dispatching
to each variant's own implementation:
#![allow(unused)] fn main() { impl StateTransitionActionTransformer for StateTransition { fn transform_into_action<C: CoreRPCLike>( &self, platform: &PlatformRef<C>, block_info: &BlockInfo, remaining_address_input_balances: &Option<BTreeMap<PlatformAddress, (AddressNonce, Credits)>>, validation_mode: ValidationMode, execution_context: &mut StateTransitionExecutionContext, tx: TransactionArg, ) -> Result<ConsensusValidationResult<StateTransitionAction>, Error> { match self { StateTransition::DataContractCreate(st) => st.transform_into_action( platform, block_info, remaining_address_input_balances, validation_mode, execution_context, tx, ), StateTransition::Batch(st) => st.transform_into_action( platform, block_info, remaining_address_input_balances, validation_mode, execution_context, tx, ), StateTransition::IdentityCreate(st) => { let signable_bytes = self.signable_bytes()?; st.transform_into_action_for_identity_create_transition( platform, signable_bytes, validation_mode, execution_context, tx, ) }, StateTransition::IdentityTopUp(st) => { let signable_bytes = self.signable_bytes()?; st.transform_into_action_for_identity_top_up_transition( platform, signable_bytes, validation_mode, execution_context, tx, ) }, // ... remaining variants } } } }
Notice that IdentityCreate and IdentityTopUp use specialized method names
(transform_into_action_for_identity_create_transition) and pass signable_bytes
explicitly. This is because these transitions need the signable bytes to verify
the asset lock proof signature, and computing them at the StateTransition level
(before the inner struct is extracted) ensures the correct bytes are used.
The StateTransitionAction Enum
The output of transformation is a StateTransitionAction, defined in
packages/rs-drive/src/state_transition_action/mod.rs:
#![allow(unused)] fn main() { pub enum StateTransitionAction { DataContractCreateAction(DataContractCreateTransitionAction), DataContractUpdateAction(DataContractUpdateTransitionAction), BatchAction(BatchTransitionAction), IdentityCreateAction(IdentityCreateTransitionAction), IdentityTopUpAction(IdentityTopUpTransitionAction), IdentityCreditWithdrawalAction(IdentityCreditWithdrawalTransitionAction), IdentityUpdateAction(IdentityUpdateTransitionAction), IdentityCreditTransferAction(IdentityCreditTransferTransitionAction), MasternodeVoteAction(MasternodeVoteTransitionAction), // ... address-based actions ... BumpIdentityNonceAction(BumpIdentityNonceAction), BumpIdentityDataContractNonceAction(BumpIdentityDataContractNonceAction), PartiallyUseAssetLockAction(PartiallyUseAssetLockAction), BumpAddressInputNoncesAction(BumpAddressInputNoncesAction), } }
Notice the system actions at the bottom: BumpIdentityNonceAction,
BumpIdentityDataContractNonceAction, PartiallyUseAssetLockAction, and
BumpAddressInputNoncesAction. These are not user-requested actions. They are
generated by the platform when a transition fails validation after the nonce check.
The platform still needs to bump the nonce (so the same invalid transition cannot
be replayed) and charge a fee. These system actions handle that housekeeping.
Versioned Dispatch Within transform_into_action
Each transition type's transform_into_action implementation uses the standard
versioned dispatch pattern. Here is the data contract create example from
packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/data_contract_create/mod.rs:
#![allow(unused)] fn main() { impl StateTransitionActionTransformer for DataContractCreateTransition { fn transform_into_action<C: CoreRPCLike>( &self, platform: &PlatformRef<C>, block_info: &BlockInfo, _remaining_address_input_balances: &Option< BTreeMap<PlatformAddress, (AddressNonce, Credits)>, >, validation_mode: ValidationMode, execution_context: &mut StateTransitionExecutionContext, _tx: TransactionArg, ) -> Result<ConsensusValidationResult<StateTransitionAction>, Error> { let platform_version = platform.state.current_platform_version()?; match platform_version .drive_abci .validation_and_processing .state_transitions .contract_create_state_transition .transform_into_action { 0 => self.transform_into_action_v0::<C>( block_info, validation_mode, execution_context, platform_version, ), version => Err(Error::Execution(ExecutionError::UnknownVersionMismatch { method: "data contract create transition: transform_into_action".to_string(), known_versions: vec![0], received: version, })), } } } }
The version number (0 here) is looked up from the platform version struct, allowing
the logic to change in future protocol upgrades without modifying the dispatch layer.
What Happens Inside: Reading State
The core work inside transform_into_action varies by transition type, but the
common thread is resolving references against current state. Here are the key
patterns for different transition types:
DataContractCreate: Deserializes the contract from its wire format, validates the
schema, and wraps it in a DataContractCreateTransitionAction. The validation depth
depends on ValidationMode -- CheckTx mode skips full schema validation for speed.
DataContractUpdate: Fetches the existing contract from Drive to compare against the proposed update. Validates that schema changes are backward-compatible (e.g., you can add fields but not remove required ones).
Batch (Documents): This is the most complex transformation. For each document sub-transition in the batch, the platform must:
- Fetch the data contract that the document belongs to
- Look up the document type within that contract
- For creates: validate the document against the type's schema
- For replaces/deletes: fetch the existing document to verify ownership and revision
- For token operations: validate token configuration and authorization
IdentityCreate: Validates the asset lock proof against the core chain (via RPC), verifies the signature on the transition using keys embedded in the transition itself, and constructs the identity that will be created.
MasternodeVote: Fetches the contested resource being voted on, verifies that the masternode is eligible to vote, and constructs the vote action.
The Batch Transition: A Deeper Look
The BatchTransition is worth examining more closely because it demonstrates how
transform_into_action handles multiple sub-operations. A single batch can contain
dozens of document and token transitions targeting different contracts and document
types.
During transformation, the platform:
- Groups transitions by contract. This allows fetching each contract only once.
- Fetches all needed contracts. Each contract is loaded from Drive (with caching).
- Resolves each sub-transition. Document creates get validated against their type's schema. Replaces fetch the existing document. Token operations check authorization rules.
- Produces a
BatchTransitionActioncontaining individualBatchedTransitionActionitems -- each of which is either aDocumentActionor aTokenAction.
If any sub-transition fails validation, the entire batch fails, and the platform
produces a BumpIdentityDataContractNonceAction instead (bumping the nonce to
prevent replay, while charging the user).
When Transformation Fails
Transformation can fail in two ways:
System error (Result::Err): Something unexpected happened -- a database read
failed, a version mismatch, corrupted state. These propagate up as Error and
typically halt block processing.
Consensus error (ValidationResult with errors): The transition itself is invalid.
The document does not match its schema, the contract does not exist, the asset lock is
already spent. In this case, the result still carries data -- typically a nonce-bump
action -- so the pipeline can charge the user and move on.
This distinction is reflected in the return type:
Result<ConsensusValidationResult<StateTransitionAction>, Error>. The outer Result
is for system errors. The inner ConsensusValidationResult is for user errors.
Rules and Guidelines
Do:
- Always respect
ValidationMode. Skip expensive work duringCheckTxbut never skip it duringValidatormode. - Accumulate validation costs in
execution_context-- every state read, every schema validation -- so that fee calculation is accurate. - Return a nonce-bump action when transformation fails for user-caused reasons. Never silently swallow the transition.
- Fetch contracts through Drive's caching layer. A batch transition might reference the same contract dozens of times; fetching it from disk each time would be prohibitively expensive.
Do not:
- Perform state writes during transform_into_action. This is a read-only phase. State is only modified when the resulting action is applied later.
- Mix up the transition and action types. The transition is
DataContractCreateTransition(fromrs-dpp). The action isDataContractCreateTransitionAction(fromrs-drive). They live in different crates for good reason. - Forget to handle the
remaining_address_input_balancesparameter for address-based transitions. It isNonefor identity-based transitions but must beSomefor address-based ones -- failing to provide it is a corrupted code execution error. - Add new transition types without implementing both the transformer trait and
adding the variant to the
StateTransitionActionenum. These must stay in sync.
Drive Operations: From Action to Storage
By this point in the pipeline, a state transition has been validated and transformed
into a StateTransitionAction. But the action is still an abstract description of
what should change. The final step is translating it into concrete storage mutations
that get applied atomically to GroveDB, the platform's Merkle tree database. This
translation happens through a three-tier pipeline of progressively lower-level
operation types.
The Three-Tier Pipeline
The pipeline looks like this:
StateTransitionAction
|
| DriveHighLevelOperationConverter::into_high_level_drive_operations()
v
Vec<DriveOperation>
|
| DriveLowLevelOperationConverter::into_low_level_drive_operations()
v
Vec<LowLevelDriveOperation>
|
| apply_batch_low_level_drive_operations()
v
GroveDB (applied atomically)
Each tier exists for a reason:
- StateTransitionAction speaks the language of the protocol: "create this identity," "store this document," "transfer these credits."
- DriveOperation speaks the language of Drive's domain model: "insert a document into this contract's document type tree," "update an identity's balance."
- LowLevelDriveOperation speaks the language of GroveDB: "insert this key-value pair at this path," "delete this element."
This layering allows each tier to be tested independently and evolved separately.
A change to how documents are indexed in GroveDB only affects the
DriveLowLevelOperationConverter implementation for DocumentOperationType -- it does
not ripple up to the action layer.
Tier 1: DriveHighLevelOperationConverter
The first conversion step is defined in
packages/rs-drive/src/state_transition_action/action_convert_to_operations/mod.rs:
#![allow(unused)] fn main() { pub trait DriveHighLevelOperationConverter { fn into_high_level_drive_operations<'a>( self, epoch: &Epoch, platform_version: &PlatformVersion, ) -> Result<Vec<DriveOperation<'a>>, Error>; } }
The StateTransitionAction enum implements this trait by dispatching to each variant:
#![allow(unused)] fn main() { impl DriveHighLevelOperationConverter for StateTransitionAction { fn into_high_level_drive_operations<'a>( self, epoch: &Epoch, platform_version: &PlatformVersion, ) -> Result<Vec<DriveOperation<'a>>, Error> { match self { StateTransitionAction::DataContractCreateAction(action) => { action.into_high_level_drive_operations(epoch, platform_version) } StateTransitionAction::DataContractUpdateAction(action) => { action.into_high_level_drive_operations(epoch, platform_version) } StateTransitionAction::BatchAction(action) => { action.into_high_level_drive_operations(epoch, platform_version) } StateTransitionAction::IdentityCreateAction(action) => { action.into_high_level_drive_operations(epoch, platform_version) } // ... all other variants } } } }
Each action type knows how to decompose itself into the appropriate DriveOperation
variants. A single action often produces multiple drive operations. For example,
IdentityCreateAction might produce:
- An
IdentityOperationto insert the identity - An
IdentityOperationto set the initial balance - Multiple
IdentityOperations to add each public key - A
SystemOperationto update system credit tracking
For the BatchTransitionAction, there is an additional layer of delegation. The batch
contains multiple BatchedTransitionAction items, each of which implements
DriveHighLevelBatchOperationConverter:
#![allow(unused)] fn main() { pub trait DriveHighLevelBatchOperationConverter { fn into_high_level_batch_drive_operations<'a>( self, epoch: &Epoch, owner_id: Identifier, platform_version: &PlatformVersion, ) -> Result<Vec<DriveOperation<'a>>, Error>; } }
Notice the extra owner_id parameter -- batch operations need to know which identity
owns the documents being created or modified.
Tier 2: The DriveOperation Enum
The DriveOperation enum represents domain-level storage operations. It is defined in
packages/rs-drive/src/util/batch/drive_op_batch/mod.rs:
#![allow(unused)] fn main() { pub enum DriveOperation<'a> { DataContractOperation(DataContractOperationType<'a>), DocumentOperation(DocumentOperationType<'a>), TokenOperation(TokenOperationType), WithdrawalOperation(WithdrawalOperationType), IdentityOperation(IdentityOperationType), PrefundedSpecializedBalanceOperation(PrefundedSpecializedBalanceOperationType), SystemOperation(SystemOperationType), GroupOperation(GroupOperationType), AddressFundsOperation(AddressFundsOperationType), GroveDBOperation(QualifiedGroveDbOp), GroveDBOpBatch(GroveDbOpBatch), } }
Each variant wraps a type-specific operation enum. For example, DocumentOperationType
includes operations like AddDocument, UpdateDocument, DeleteDocument -- each
carrying the document data, contract reference, and storage flags needed for insertion.
The last two variants -- GroveDBOperation and GroveDBOpBatch -- are escape hatches
for when higher-level abstractions are not needed. They wrap raw GroveDB operations
directly.
The DriveOperation enum implements the DriveLowLevelOperationConverter trait to
convert itself into the next tier:
#![allow(unused)] fn main() { pub trait DriveLowLevelOperationConverter { fn into_low_level_drive_operations( self, drive: &Drive, estimated_costs_only_with_layer_info: &mut Option< HashMap<KeyInfoPath, EstimatedLayerInformation>, >, block_info: &BlockInfo, transaction: TransactionArg, platform_version: &PlatformVersion, ) -> Result<Vec<LowLevelDriveOperation>, Error>; } }
Two important parameters here:
estimated_costs_only_with_layer_info: When this isSome, the converter does not actually read from or write to GroveDB. Instead, it estimates the cost of the operations using layer information. This is used for fee estimation before execution.transaction: The GroveDB transaction handle. All reads and writes within a single block happen within one transaction, ensuring atomicity.
Tier 3: LowLevelDriveOperation
The lowest tier is defined in packages/rs-drive/src/fees/op.rs:
#![allow(unused)] fn main() { pub enum LowLevelDriveOperation { GroveOperation(QualifiedGroveDbOp), FunctionOperation(FunctionOp), CalculatedCostOperation(OperationCost), PreCalculatedFeeResult(FeeResult), } }
At this level, there are only four kinds of things:
- GroveOperation: A concrete GroveDB operation -- insert, delete, or update a key-value pair at a specific path in the Merkle tree.
- FunctionOperation: A CPU-bound operation with a pre-defined cost (like hashing or signature verification). These do not touch storage but still cost processing fees.
- CalculatedCostOperation: A pre-computed cost that gets folded into the fee calculation.
- PreCalculatedFeeResult: An already-computed fee result, used when the fee for an operation was determined earlier in the pipeline.
The GroveOperation variant is where the rubber meets the road.
QualifiedGroveDbOp is GroveDB's own batch operation type -- it specifies a path
(a vector of byte-string segments navigating the tree), a key, and an operation
(insert element, delete, replace, etc.).
Applying the Batch
The entire sequence -- from DriveOperation collection to GroveDB application --
is orchestrated by Drive::apply_drive_operations, defined in
packages/rs-drive/src/util/batch/drive_op_batch/drive_methods/apply_drive_operations/v0/mod.rs:
#![allow(unused)] fn main() { pub(crate) fn apply_drive_operations_v0( &self, operations: Vec<DriveOperation>, apply: bool, block_info: &BlockInfo, transaction: TransactionArg, platform_version: &PlatformVersion, previous_fee_versions: Option<&CachedEpochIndexFeeVersions>, ) -> Result<FeeResult, Error> { if operations.is_empty() { return Ok(FeeResult::default()); } let mut low_level_operations = vec![]; let mut estimated_costs_only_with_layer_info = if apply { None } else { Some(HashMap::new()) }; let mut finalize_tasks: Vec<DriveOperationFinalizeTask> = Vec::new(); for drive_op in operations { if let Some(tasks) = drive_op.finalization_tasks(platform_version)? { finalize_tasks.extend(tasks); } low_level_operations.append( &mut drive_op.into_low_level_drive_operations( self, &mut estimated_costs_only_with_layer_info, block_info, transaction, platform_version, )? ); } let mut cost_operations = vec![]; self.apply_batch_low_level_drive_operations( estimated_costs_only_with_layer_info, transaction, low_level_operations, &mut cost_operations, &platform_version.drive, )?; for task in finalize_tasks { task.execute(self, platform_version); } Drive::calculate_fee( None, Some(cost_operations), &block_info.epoch, self.config.epochs_per_era, platform_version, previous_fee_versions, ) } }
Let us break this down:
-
Collect finalization tasks. Some operations need post-processing. For example,
DataContractOperationmay produce aRecordShieldedAnchorfinalization task that runs after the batch is committed. Tasks are collected first, executed last. -
Convert to low-level operations. Each
DriveOperationis expanded into one or moreLowLevelDriveOperations. A single document insertion might produce dozens of GroveDB operations (one for each index, plus the document itself, plus metadata). -
Apply the batch atomically.
apply_batch_low_level_drive_operationscollects allGroveOperationitems into a single GroveDB batch and applies them in one atomic write. This is critical -- if the node crashes mid-application, either all operations succeed or none do. -
Execute finalization tasks. Post-commit callbacks run (e.g., updating caches).
-
Calculate fees. The cost of every operation (storage bytes written, bytes read, processing time) is tallied into a
FeeResultthat determines how many credits the user pays.
The apply Parameter
Notice the apply: bool parameter. When apply is false, the entire pipeline runs
in estimation mode: operations are not actually written to GroveDB. Instead, the
estimated_costs_only_with_layer_info map is populated with what would be written,
and fees are estimated from that.
This is used during check_tx and fee estimation. The platform needs to know
approximately how much a transition will cost before actually applying it, both for
balance pre-checks and for returning fee estimates to clients.
The Fee Calculation
At the end of apply_drive_operations, fees are calculated from the accumulated
cost operations:
#![allow(unused)] fn main() { Drive::calculate_fee( None, Some(cost_operations), &block_info.epoch, self.config.epochs_per_era, platform_version, previous_fee_versions, ) }
The fee has two components:
- Storage fee: Proportional to the bytes written to disk. Stored bytes have an ongoing cost because they consume space in the state tree indefinitely (until deleted). When bytes are later removed, a portion of the storage fee is refunded.
- Processing fee: Proportional to the CPU work performed -- hashing, signature verification, tree traversal. This is ephemeral and not refundable.
The user's user_fee_increase (a percentage multiplier) applies to the processing fee,
allowing users to bid higher for priority.
Finalization Tasks
Some DriveOperation variants carry finalization tasks -- callbacks that run after
the batch is committed. These are defined in
packages/rs-drive/src/util/batch/drive_op_batch/finalize_task.rs:
#![allow(unused)] fn main() { pub(crate) trait DriveOperationFinalizationTasks { fn finalization_tasks( &self, platform_version: &PlatformVersion, ) -> Result<Option<Vec<DriveOperationFinalizeTask>>, Error>; } }
The most common finalization task is RecordShieldedAnchor, used by shielded
transaction operations to record Merkle tree anchors after the state changes are
committed. Finalization tasks are intentionally limited -- they must be deterministic
and must not fail, since they run after the batch is already committed.
Putting It All Together: A Document Create
Let us trace a document creation through the entire three-tier pipeline:
-
Action:
BatchTransitionActioncontaining aDocumentAction::CreateActionwith the document data, its type, and the contract reference. -
DriveHighLevelOperationConverter: The document create action produces a
DriveOperation::DocumentOperation(AddDocument { ... })containing the owned document, contract info, document type info, and storage flags. -
DriveLowLevelOperationConverter: The
AddDocumentoperation produces multipleLowLevelDriveOperation::GroveOperationitems:- Insert the serialized document at its primary key path
- Insert index entries for each indexed property
- Update the document type's document count
- Record storage flags for fee tracking
-
GroveDB batch: All the
GroveOperationitems from all documents in the batch are collected into a singleGroveDbOpBatchand applied atomically. -
Fee calculation: The total bytes written, bytes read, and processing operations are summed to produce the
FeeResult.
Rules and Guidelines
Do:
- Implement
DriveHighLevelOperationConverterfor new action types. This is the contract between the validation layer and the storage layer. - Keep
into_low_level_drive_operationsdeterministic. Given the same inputs and state, it must always produce the same operations. Non-determinism causes consensus failures. - Use the estimation mode (
apply = false) for fee pre-checks. Do not skip fee estimation -- users need accurate cost information before committing to a transition. - Test both the estimation path and the application path. They can diverge if the estimation layer info is stale or incomplete.
Do not:
- Access GroveDB directly from action conversion code. Always go through Drive's methods, which handle versioning, caching, and error translation.
- Produce side effects in
into_high_level_drive_operations. This conversion must be pure -- it maps data, it does not read or write state. - Assume a 1:1 mapping between actions and GroveDB operations. A single document create can produce 10+ GroveDB operations (one per index). A batch with 50 documents can produce hundreds.
- Forget finalization tasks when adding new operation types that require post-commit
work. If your operation needs to update a cache or record an anchor, implement
DriveOperationFinalizationTasks. - Mix up
DriveOperation(high-level, domain-aware) withLowLevelDriveOperation(low-level, GroveDB-aware). The naming can be confusing, but the distinction is important: the former knows about documents and identities, the latter knows about tree paths and elements.
Fee System Overview
Every state transition on Dash Platform costs credits. Credits are the internal unit of account (1 Dash = 100,000,000,000 credits). The fee system ensures that validators are compensated for computation and storage, that spam is economically infeasible, and that the platform's state does not grow unboundedly without payment.
This section covers the three fee eras that the platform has gone through:
- Identity Credit Fees (protocol versions 1--9) — Fees paid from an identity's credit balance, funded by asset lock transactions on Core.
- Platform Address Fees (protocol versions 10--11) — Fees paid from platform address balances using a UTXO-like input/output model.
- Shielded Transaction Fees (protocol version 12) — Fees embedded in zero-knowledge proofs and cryptographically bound to the Orchard bundle.
Each era introduced new ExecutionEvent variants and fee validation logic, but
the underlying cost accounting (storage fees, processing fees, epoch
distribution) is shared across all three.
Credits and Denomination
Platform credits are the smallest unit of value:
| Unit | Credits |
|---|---|
| 1 credit | 1 |
| 1 mDash | 100,000,000 |
| 1 Dash | 100,000,000,000 |
All fee constants in the codebase are denominated in credits.
Cost Components
The platform distinguishes two fundamental kinds of cost:
Storage Fees
Storage fees pay for bytes that persist in GroveDB indefinitely. The rate is set
in FeeStorageVersion:
| Parameter | Value | Description |
|---|---|---|
storage_disk_usage_credit_per_byte | 27,000 | Permanent disk storage cost |
storage_processing_credit_per_byte | 400 | I/O cost to write the bytes |
storage_load_credit_per_byte | 20 | I/O cost to read stored bytes |
non_storage_load_credit_per_byte | 10 | I/O cost for ephemeral reads |
storage_seek_cost | 2,000 | Cost of a single disk seek |
Storage fees are refundable: when data is deleted, a portion of the original storage fee is returned to the identity that paid it (see Refunds below).
The documents of a type that declares a ttl (protocol version 14) are the exception: they
carry no storage flags and refund nothing, their bytes are priced for the time they live, and
their storage fees are paid out to the epochs they live in, through the lifetime storage fee
pools, instead of over the perpetual distribution. See
Document Time To Live.
Processing Fees
Processing fees pay for computation that does not leave a permanent trace in storage: signature verification, hashing, tree traversal, and so on. These are non-refundable — the computation has already been performed.
Processing costs are built up from individual operations:
processing_fee =
seek_count × storage_seek_cost
+ added_bytes × storage_processing_credit_per_byte
+ replaced_bytes × storage_processing_credit_per_byte
+ loaded_bytes × storage_load_credit_per_byte
+ hash_node_calls × (blake3_base + blake3_per_block)
Signature verification adds a fixed cost per algorithm:
| Algorithm | Cost (credits) |
|---|---|
| ECDSA secp256k1 | 15,000 |
| BLS12-381 | 300,000 |
| ECDSA hash160 | 15,500 |
| BIP13 script hash | 300,000 |
| EdDSA ed25519 hash160 | 3,500 |
Hashing costs scale with the number of blocks processed:
| Hash Function | Base | Per Block |
|---|---|---|
| SHA-256 | 100 | 5,000 |
| Blake3 | 100 | 300 |
| SHA-256 + RIPEMD-160 | 6,000 | 5,000 |
Minimum Fees
Every state transition type has a minimum fee that must be met regardless of the
actual computation cost. This prevents zero-cost spam. The minimums are defined
in StateTransitionMinFees:
Identity-Based Transitions (protocol versions 1--9)
| Transition | Minimum Fee (credits) |
|---|---|
| Credit Transfer | 100,000 |
| Credit Transfer to Addresses | 500,000 |
| Credit Withdrawal | 400,000,000 |
| Identity Update | 100,000 |
| Document Batch (per sub-transition) | 100,000 |
| Contract Create | 100,000 |
| Contract Update | 100,000 |
| Masternode Vote | 100,000 |
Address-Based Transitions (protocol versions 10--11)
| Transition | Minimum Fee (credits) |
|---|---|
| Address Funds Transfer (per input) | 500,000 |
| Address Funds Transfer (per output) | 6,000,000 |
| Address Credit Withdrawal | 400,000,000 |
| Identity Create (base) | 2,000,000 |
| Identity Create (per key) | 6,500,000 |
| Identity Top-Up (base) | 500,000 |
Data Contract Registration Fees (protocol version 9+)
Protocol version 9 introduced significant registration fees for data contracts to prevent namespace squatting:
| Component | Fee | Dash Equivalent |
|---|---|---|
| Base contract registration | 10,000,000,000 | 0.1 Dash |
| Document type registration | 2,000,000,000 | 0.02 Dash |
| Non-unique index | 1,000,000,000 | 0.01 Dash |
| Unique index | 1,000,000,000 | 0.01 Dash |
| Contested index | 100,000,000,000 | 1.0 Dash |
| Token registration | 10,000,000,000 | 0.1 Dash |
| Token uses a perpetual distribution | 10,000,000,000 | 0.1 Dash |
| Token uses a pre-programmed distribution | 10,000,000,000 | 0.1 Dash |
| Token uses a once-per-identity distribution (protocol version 14+) | 10,000,000,000 | 0.1 Dash |
| Search keyword | 10,000,000,000 | 0.1 Dash |
Before protocol version 9, all registration fees were zero.
Contest Funds
A document create that opens or joins a contest (a contested unique index)
prefunds the masternode votes: the amount leaves the contender's balance for the
contest's prefunded balance, each vote takes a fixed cost from it, and what is
left when the contest ends is released as processing fees. The amounts are
VoteResolutionFundFees in the fee version:
| Component | Protocol versions 1 to 13 | Protocol version 14 |
|---|---|---|
| Contested document fund (DPNS and every other contest) | 0.2 Dash | 0.1 Dash |
Moderation election fund (an electedCharter application) | none exist | 0.5 Dash |
| One vote | 0.0001 Dash | 0.00002 Dash |
required_vote_resolution_fund in rs-dpp picks between the two funds; the
schedules before 14 carry the contested document amount in the moderation
field, so the choice changes nothing there.
User Fee Increase
Every state transition carries a user_fee_increase field (a UserFeeIncrease
value). This allows the sender to voluntarily pay more than the base fee to
prioritize their transition. The multiplier works as follows:
0= 100% of base fee (no increase)1= 101% of base fee10= 110% of base fee100= 200% of base fee
The increase applies only to the processing fee component, not to storage fees. This is because storage fees are a direct function of bytes stored and should not be inflated.
#![allow(unused)] fn main() { fn apply_user_fee_increase(&mut self, user_fee_increase: UserFeeIncrease) { let increase = self.processing_fee * user_fee_increase as u64 / 100; self.processing_fee = self.processing_fee.saturating_add(increase); } }
ExecutionEvent Variants
The ExecutionEvent enum (in rs-drive-abci) determines how fees are collected
for each state transition. There are eight variants:
| Variant | Fee Source | Used By |
|---|---|---|
Paid | Identity credit balance, or the contract owner's for a sponsored document batch (below) | Most identity-based transitions |
PaidFromAssetLock | Asset lock transaction value | IdentityCreate, IdentityTopUp |
PaidFromAssetLockWithoutIdentity | Asset lock (fixed amount) | PartiallyUseAssetLock |
PaidFromAssetLockToPool | Asset lock value; fee routed to the fee pools | ShieldFromAssetLock |
PaidFromAddressInputs | Platform address balances | All address-based transitions; Shield (metered + a ZK compute fee via additional_fixed_fee_cost) |
PaidFixedCost | Fixed fee to pool | MasternodeVote |
PaidFromShieldedPool | Shielded pool value_balance | ShieldedTransfer, Unshield, ShieldedWithdrawal |
PaidFromShieldedPoolToNewIdentity | Shielded pool (the fixed denomination); the metered write + ZK compute fee is moved from the new identity's balance into the fee pools | IdentityCreateFromShieldedPool |
Each variant carries the operations to execute and enough context for the fee validation and execution pipeline to deduct the correct amount from the correct source.
Gas paid by the contract owner
From protocol version 14 a document action that is paid for with a token can
have its gas (the storage and processing fee) paid by the contract owner. The
document type's token cost offers it (gasFeesPaidBy: DocumentOwner,
ContractOwner or PreferContractOwner) and the transition's
$tokenPaymentInfo asks for it with the same enum; GasFeesPaidBy::resolve
in rs-dpp names the payer. A document owner can always opt out, can always
state a preference, and can insist (ContractOwner) only on a document type
that commits to paying. Only a token-paid action can be sponsored, so every
sponsored transition is backed by a token the contract owner chose to hand out.
The batch transformer (v2) resolves one payer for the whole batch, reads the
contract owner's balance into the action (ResolvedGasSponsor, billed to the
batch) and the execution event carries it in Paid.gas_sponsor. Fee
validation v1 judges the fee against the sponsor's balance and the signer only
has to fund removed_balance; when the sponsor's balance falls short a batch
that insists is refused unpaid (GasSponsorInsufficientBalanceError, 40222)
and a batch that prefers falls back to the signer's balance. Execution v1 then
charges whoever fee validation admitted. A batch that fails validation is never
sponsored: its signer pays for the work that ran, and a request the document
type does not offer is a paid rejection (GasFeesPaidByNotAllowedError,
40129). Storage refunds still go to the document's owner, whoever paid the
storage: a sponsored document refunds its owner when it is deleted or replaced
by a smaller one, even when the sponsor pays for that transition too. Each
token the sponsor hands out is therefore worth up to the storage fee of the
largest document the type allows, so a document type that offers sponsorship
should bound its documents' size (maxLength, maxItems) and price the
action accordingly.
The signer's minimum balance pre-check runs before the contracts are loaded; its v1 asks a batch that requests sponsorship for its principal only (purchases, contest collateral) and leaves the gas to fee validation, so an identity without credits can act on tokens it was given. Such a signer could not pay for a failed batch, and a failed batch is never sponsored, so check tx validates the batch of a signer under the fee minimum against the state in full, on the first check and on every recheck, as it does a masternode vote: what nobody could be charged for is refused there, or leaves the mempool once the tokens it counted on are spent, instead of being executed for free by a proposer.
Optional token costs
A token cost may declare optional: true (v3 meta-schema, protocol version
14). A transition on such an action may leave $tokenPaymentInfo out: it then
pays no token, its signer pays the gas in credits as on an action without a
token cost, and no sponsorship applies. With $tokenPaymentInfo present the
token is charged exactly as for a required cost, sponsorship included, and an
insufficient token balance is a rejection rather than a fallback to credits:
the client chooses between token and credits before signing. Together with
contract-owner gas this is the "free usage" pattern: an app hands out tokens,
a user posts for free while they last, and keeps posting on credits after.
Document action fees
From protocol version 14 a document type may charge a fixed fee in credits for
an action on one of its documents, on top of the gas. The actionFees keyword
(v3 document meta-schema) sits beside tokenCost and prices the same six
actions:
"post": {
"type": "object",
"actionFees": {
"pricing": "feeMultiplier",
"create": { "moderators": 100000000, "owner": 10000000 }
}
}
Creating a post here costs an extra 0.001 Dash for the contract's moderation
team and 0.0001 Dash for its owner. Each fee has those two parts, either of
which may be left out. The owner parts collect in the contract's owner
pot and the moderators parts in its moderators pot, and a
ContractFeeClaim state transition pays a pot out (see
Contract Moderation).
A moderators part needs a contract that declares moderation
(DocumentActionFeesWithoutModerationError, 10902): the moderation team is
who that pot is for.
Pricing. fixed charges the declared amounts as written. feeMultiplier,
the default, scales them by the fee multiplier of the epoch the action
executes in (declared * multiplier_permille / 1000, rounded down), so a fee
follows the network's fees. The multiplier is the item every epoch tree
records, read once per batch and billed to it
(fetch_action_fee_multiplier_with_fee). In the first block of an epoch that
item is not there yet, because state transitions execute before the end of the
block, where the epoch is initialized; the multiplier the epoch is about to be
initialized with, the fee schedule's, is used then. Nothing else reads the
epoch multiplier today: the metered fees do not scale with it. A scaled
amount is held at the maximum number of credits rather than overflowing: a fee
nobody can pay refuses the action for an insufficient balance, a consensus
error, where an overflow would have failed every transition on the action with
an internal one.
The transition agrees to the fee. The contract is read when the action
executes, not when the transition was signed, so a transition that said
nothing would pay whatever the contract declares by then. Every transition on
an action that charges a fee therefore carries an action fee agreement
($actionFeeAgreement, on version 2 of the document base transition, the
default from protocol version 14):
"$actionFeeAgreement": {
"$formatVersion": "0",
"owner": 10000000,
"moderators": 100000000,
"feeMultiplier": { "knownPermille": 1000, "increaseTolerancePercent": 20 }
}
ownerandmoderatorsare the amounts the document type declares for the action, before any multiplier. They must match exactly, each pot on its own: a fee that was raised, lowered, or moved between the pots since the signer read the contract refuses the action (DocumentActionFeeAgreementMismatchError, 40133), and the signer reads the contract again.feeMultipliersays how the fee is priced. It is named for afeeMultiplierfee and left out for afixedone, and an agreement to the other pricing is the same mismatch.knownPermilleis the fee multiplier the signer priced the fee with, andincreaseTolerancePercenthow far above it the multiplier of the executing epoch may be, in percent of the known one: 20 accepts up to 1.2 times. A transition signed just before an epoch boundary is then not refused for a small move; one the multiplier outran is (DocumentActionFeeMultiplierNotToleratedError, 40134). A multiplier that fell is always accepted. What is charged follows the epoch's multiplier, never the known one.- A transition without an agreement on an action that charges a fee is
refused (
DocumentActionFeeAgreementNotSetError, 40132), whoever pays: a sponsored action says what it agrees to as well, since a preferred sponsor can hand the fee back to the signer. An agreement on an action that charges nothing is ignored.
All three are paid refusals that bump the nonce and charge no fee. They are
judged in the batch's advanced structure validation
(BatchTransitionAction::validate_action_fee_agreements), off the action
alone: the base action carries the declaration beside the agreement, and the
batch action the multiplier its transformer read. The mempool runs the same
check when a transition arrives and again on every recheck, so a transition
whose agreement no longer holds leaves the mempool with the same error instead
of failing in the block that would have refused it. A client builds the
agreement from the contract it showed its user with
DocumentActionFeeAgreement::for_document_type_action, never from a contract
fetched behind their back at signing time.
A seated team's discount. On a document type an elected contract
moderates, the moderators part of an agreement may name less than the
declared amount: the share the contract's seated moderation charter takes
(its proposal's moderatorsShare, a percentage; none declared is the full
amount), applied to the declared amount and rounded down to the credit
(moderation_charter::moderators_share_of). Everything else must still match:
the owner part and the pricing. With a share of 60, the post above admits
exactly 60000000 for the moderators:
"$actionFeeAgreement": {
"$formatVersion": "0",
"owner": 10000000,
"moderators": 60000000,
"feeMultiplier": { "knownPermille": 1000, "increaseTolerancePercent": 20 }
}
The action is then charged the agreed amount, which is what reaches the
moderators pot (scaled by the multiplier for a feeMultiplier fee, like the
declared amount). An agreement to the declared amount stays valid whatever the
team charges and reads no charter; only one that names less has the batch
transformer read the seated charter (the byTargetContract index of the
moderation charters contract) and the proposal it runs on, billed to the
batch. Any other amount below the declared one, including a discount on a
contract with no seated charter yet, is refused like a mismatch, paid and
without a fee (DocumentActionFeeModeratorsShareMismatchError, 40139). A
lower amount anywhere else (a type the contract does not moderate, a contract
that is not elected) is the plain mismatch (40133). The mempool judges it
the same way on arrival and on every recheck, since the recheck transforms the
batch anew.
The amounts do not change yet. A contract update may not add, change or
remove the actionFees of an existing document type, nor switch their pricing
(DocumentTypeUpdateError). A document type added by an update may declare
its own, so a live contract gets fees through new document types only. The
agreement is what makes lifting this safe later: an owner who changes a fee
cannot make a transition signed against the old one pay the new one.
Who pays. Whoever pays the gas pays the action fee: the signer, or the
contract owner when they sponsor the gas. A sponsor's balance has to cover the
gas and the fees they would owe; one that insisted and falls short is the
same unpaid refusal as before (40222), and one that only preferred hands the
gas and the fees back to the signer. The gas is estimated for the payer:
first with the sponsor paying, then, when the sponsor does not pay, with the
signer paying. Fee validation settles the payer through one function
(gas_sponsor_pays) and hands it to execution, which charges that payer, so
they always name the same one. The contract owner never pays the owner part:
it would travel through the owner pot back to them and only cost writes. A
sponsor is always the contract owner, so a sponsored action pays into the
moderators pot only, which a contract that sponsors gas should price in. The
fee counts against the budget of a budgeted signing key when its identity pays
it.
Only an action that executes is charged. A transition that fails, in the transformer or later in state validation, becomes a nonce bump, and a bump owes nothing. The fees are therefore read off the transitions when the execution event is built, after state validation had its say, not when the transformer ran.
The fee is not part of the FeeResult. It moves as balance operations in
the batch's own operation list: one removal from the payer, one addition per
pot. The transition may already write the payer's balance: a purchase moves
its price, a contested document its voting fund, and a sale pays a contract
owner who may be sponsoring the gas. A balance operation computes the new
balance from the one committed before its batch and GroveDB keeps only the
last write of a key, so apply_drive_operations (generation 1, protocol
version 14) merges every write of one identity balance, one fee pot or one
prefunded specialized balance in a batch into a single net operation. Token
writes cannot be merged that way (a transfer writes two balances, a mint or a
burn a balance and the supply), so a batch that writes one token balance or
supply twice is refused; no state transition makes one. The fee pools and the proposers see
exactly what they saw before. The
pots sit under the PreFundedSpecializedBalances root sum tree, which the
per-block total credits check already sums, so the credits stay accounted for
while they wait to be claimed.
FeeResult
All fee calculations produce a FeeResult:
#![allow(unused)] fn main() { pub struct FeeResult { pub storage_fee: Credits, pub processing_fee: Credits, pub fee_refunds: FeeRefunds, pub removed_bytes_from_system: u32, } }
storage_fee— credits for new bytes written to persistent storageprocessing_fee— credits for computation and I/Ofee_refunds— credits returned because previously stored data was deletedremoved_bytes_from_system— bytes removed that were stored by the system (not by any identity), so no refund is issued
The total base fee is storage_fee + processing_fee. The FeeResult is
produced by Drive::apply_drive_operations(), which executes the GroveDB
operations and measures the actual cost of each insert, delete, and query.
Refunds
When data is removed from GroveDB (a document is deleted, a key is removed), the system calculates a refund of the original storage fee. Refunds are tracked per identity per epoch:
#![allow(unused)] fn main() { pub struct FeeRefunds(pub CreditsPerEpochByIdentifier); // BTreeMap<IdentifierBytes32, BTreeMap<EpochIndex, Credits>> }
Refunds are not 1:1 with the original fee because storage fees are distributed across future epochs (see below). The refund amount depends on how many epochs have elapsed since the data was stored — the longer the data has been stored, the smaller the refund, because more of the distributed fees have already been paid out to proposers.
There is a dust limit: refunds below 32 bytes worth of storage credits are discarded to prevent micro-refund spam.
Epoch-Based Fee Distribution
Fees do not go directly to the block proposer. Instead, they accumulate in epoch-specific pools and are distributed to proposers at epoch boundaries.
How Epochs Work
- An epoch is a fixed window of blocks
- An era consists of 40 epochs
- Storage fees are distributed across 50 eras (2,000 epochs, roughly 50 years) using a declining schedule
The distribution table allocates percentages per era:
| Era | Percentage | Cumulative |
|---|---|---|
| 0 | 5.000% | 5.0% |
| 1 | 4.800% | 9.8% |
| 2 | 4.600% | 14.4% |
| ... | ... | ... |
| 49 | 0.125% | 100.0% |
Within each era, the percentage is divided equally among the era's epochs. For example, a 1,000,000-credit storage fee distributes 50,000 credits (5%) to era 0, split evenly across 40 epochs = 1,250 credits per epoch.
Fee Flow
Block execution
└→ FeeResult (storage + processing)
└→ End of block: add_distribute_block_fees_into_pools()
├→ Processing fees → current epoch pool
└→ Storage fees → global distribution pool → spread across future epochs
Epoch change
└→ add_distribute_fees_from_oldest_unpaid_epoch_pool_to_proposers()
├→ Calculate Core block rewards for the epoch
├→ Add Core rewards to system credits
└→ Distribute (Platform fees + Core rewards) to proposers
Processing fees are paid to proposers at the end of the epoch in which they were collected. Storage fees are spread across 50 eras of future epochs, providing a long-term revenue stream for validators.
Fee Versioning
All fee parameters are versioned through FeeVersion, stored in
PlatformVersion. This allows the protocol to adjust fee rates without a hard
fork — a new protocol version simply references different fee constants.
The current fee version structure:
#![allow(unused)] fn main() { pub struct FeeVersion { pub fee_version_number: FeeVersionNumber, pub uses_version_fee_multiplier_permille: Option<u64>, pub storage: FeeStorageVersion, pub signature: FeeSignatureVersion, pub hashing: FeeHashingVersion, pub processing: FeeProcessingVersion, pub data_contract_validation: FeeDataContractValidationVersion, pub data_contract_registration: FeeDataContractRegistrationVersion, pub state_transition_min_fees: StateTransitionMinFees, pub vote_resolution_fund_fees: VoteResolutionFundFees, } }
Fee versions are stored in the FEE_VERSIONS array and looked up by number. The
uses_version_fee_multiplier_permille field allows a global scaling factor
(permille = divide by 1000; a value of 1000 means no change).
Key Source Files
| File | Contents |
|---|---|
rs-platform-version/src/version/fee/ | All fee version definitions |
rs-platform-version/src/version/fee/storage/v1.rs | Storage fee rates |
rs-platform-version/src/version/fee/signature/v1.rs | Signature verification costs |
rs-platform-version/src/version/fee/state_transition_min_fees/v1.rs | Minimum fees per transition |
rs-platform-version/src/version/fee/data_contract_registration/v2.rs | Contract registration fees |
rs-platform-version/src/version/fee/data_contract_registration/v3.rs | Protocol version 14 addition: once-per-identity distribution surcharge |
rs-drive/src/fees/op.rs | LowLevelDriveOperation and cost calculation |
rs-dpp/src/fee/fee_result/mod.rs | FeeResult, BalanceChangeForIdentity |
rs-dpp/src/fee/epoch/distribution.rs | Epoch distribution table and refund logic |
rs-drive-abci/src/execution/types/execution_event/mod.rs | ExecutionEvent enum |
rs-drive-abci/src/execution/platform_events/fee_pool_inwards_distribution/ | Block fee collection |
rs-drive-abci/src/execution/platform_events/fee_pool_outwards_distribution/ | Proposer payout |
Platform Address Fees
Protocol versions 10 and 11 introduced a new class of state transitions that operate on platform addresses rather than identities. Platform addresses are derived from public keys (similar to Bitcoin addresses) and hold a credit balance directly, without requiring an identity to be registered.
This chapter explains how fees work for address-based transitions, how they differ from the identity credit model, and how the fee strategy mechanism gives clients control over fee deduction.
Background: Why Platform Addresses?
Before protocol version 10, every action on the platform required an identity — a registered entity funded by an asset lock transaction. Creating an identity required a Core transaction, waiting for confirmations, and then submitting a state transition. This was a multi-step process that created friction for simple operations like "send credits to an address."
Platform addresses solve this by allowing credits to exist at addresses without a full identity. Users can fund an address (via an asset lock or a transfer from another address) and then spend from it directly using a signature from the address's key pair.
Address-Based State Transitions
Protocol versions 10 and 11 added the following transition types:
| Transition | Protocol Version | Description |
|---|---|---|
IdentityCreateFromAddresses | 10 | Create an identity funded from platform address balances |
IdentityTopUpFromAddresses | 10 | Add credits to an existing identity from address balances |
AddressFundsTransfer | 11 | Transfer credits between platform addresses |
AddressFundingFromAssetLock | 11 | Fund an address directly from an asset lock |
AddressCreditWithdrawal | 11 | Withdraw credits from an address back to Core |
All of these use the PaidFromAddressInputs execution event variant.
The Input/Output Model
Address-based transitions follow a UTXO-inspired model with inputs and outputs:
Inputs
Each input specifies a platform address, the expected nonce, and the amount to consume from that address's balance:
inputs: [
{ address: A, nonce: 5, amount: 100_000_000 },
{ address: B, nonce: 3, amount: 50_000_000 },
]
The platform validates each input by:
- Checking the address exists in state
- Verifying the nonce is exactly
current_nonce + 1(replay protection) - Confirming the address has sufficient balance for the requested amount
After validation, the remaining balance of each input is tracked:
remaining = actual_balance - requested_amount
This remaining balance is what is available to pay fees from (after the requested amount is committed to outputs).
Outputs
Outputs specify destination addresses and the credits to send:
outputs: [
{ address: C, amount: 80_000_000 },
{ address: D, amount: 60_000_000 },
]
Outputs are added to recipient balances. They can also be reduced to pay fees (see Fee Strategy below).
Balance Equation
The fundamental constraint is:
sum(input_amounts) >= sum(output_amounts) + fees
If the inputs cannot cover both the outputs and the fees, the transition is
rejected with AddressesNotEnoughFundsError.
Fee Strategy
Unlike identity-based transitions where fees are always deducted from the identity's balance, address-based transitions use an explicit fee strategy that the client includes in the transition. The fee strategy is an ordered sequence of steps that tells the platform where to find the fee credits:
#![allow(unused)] fn main() { pub enum AddressFundsFeeStrategyStep { /// Deduct fee from a specific input address's remaining balance. DeductFromInput(u16), /// Reduce a specific output's amount to cover the fee. ReduceOutput(u16), } pub type AddressFundsFeeStrategy = Vec<AddressFundsFeeStrategyStep>; }
How It Works
The platform processes the steps in order. At each step, it deducts as much of the remaining fee as possible from the specified source:
fee_strategy: [DeductFromInput(0), ReduceOutput(0)]
Step 1: Try to deduct full fee from input 0's remaining balance
- remaining_fee = 10,000,000
- input_0_remaining = 25,000,000
- deducted = 10,000,000
- input_0_remaining = 15,000,000
- remaining_fee = 0 → done
If the first source is insufficient, the algorithm moves to the next step:
fee_strategy: [DeductFromInput(0), ReduceOutput(1)]
Step 1: Try to deduct from input 0
- remaining_fee = 10,000,000
- input_0_remaining = 3,000,000
- deducted = 3,000,000
- input_0_remaining = 0 (removed)
- remaining_fee = 7,000,000
Step 2: Try to reduce output 1
- remaining_fee = 7,000,000
- output_1_amount = 60,000,000
- deducted = 7,000,000
- output_1_amount = 53,000,000
- remaining_fee = 0 → done
Index Stability
The indices in the fee strategy refer to the original BTreeMap iteration order. The implementation snapshots the address lists before processing any steps, so removing a drained entry at step 1 does not shift the indices for step 2. This is critical for correctness:
#![allow(unused)] fn main() { let input_addresses: Vec<PlatformAddress> = inputs.keys().copied().collect(); let output_addresses: Vec<PlatformAddress> = outputs.keys().copied().collect(); for step in fee_strategy { match step { DeductFromInput(index) => { let address = input_addresses[*index as usize]; // look up by address, not by index into the live BTreeMap } // ... } } }
FeeDeductionResult
The deduction algorithm produces:
#![allow(unused)] fn main() { pub struct FeeDeductionResult { pub remaining_input_balances: BTreeMap<PlatformAddress, (AddressNonce, Credits)>, pub adjusted_outputs: BTreeMap<PlatformAddress, Credits>, pub fee_fully_covered: bool, } }
If fee_fully_covered is false, the transition is rejected.
Nonce System (Replay Protection)
Every platform address has a nonce that starts at 0 and increments by 1 with
each transition that uses the address as an input. The transition must specify
the expected nonce, which must be exactly current_nonce + 1:
#![allow(unused)] fn main() { let expected_next_nonce = state_nonce.saturating_add(1); if provided_nonce != expected_next_nonce { return Err(AddressInvalidNonceError { expected: expected_next_nonce, provided: provided_nonce, }); } }
This prevents:
- Replay attacks — resubmitting an old transition (wrong nonce)
- Double-spending — using the same balance twice (nonce already consumed)
- Ordering attacks — submitting transitions out of order (nonce gap)
If a nonce reaches u32::MAX, the address is exhausted and cannot be used as
an input anymore.
Multiple addresses can be used as inputs in a single transition, each with its
own nonce. The platform enforces a maximum number of inputs per transition
(configured in platform_version.dpp.state_transitions.max_address_inputs).
The PaidFromAddressInputs Event
When an address-based transition passes validation, the processor creates a
PaidFromAddressInputs execution event:
#![allow(unused)] fn main() { ExecutionEvent::PaidFromAddressInputs { input_current_balances: BTreeMap<PlatformAddress, (AddressNonce, Credits)>, added_to_balance_outputs: BTreeMap<PlatformAddress, Credits>, fee_strategy: AddressFundsFeeStrategy, operations: Vec<DriveOperation>, execution_operations: Vec<ValidationOperation>, additional_fixed_fee_cost: Option<Credits>, user_fee_increase: UserFeeIncrease, } }
input_current_balances— the remaining balance of each input after consuming the requested amounts, plus the validated nonceadded_to_balance_outputs— the output amounts before any fee deductionsfee_strategy— the client's ordered fee deduction instructionsoperations— the GroveDB operations (balance updates, nonce bumps)execution_operations— validation operations that also incur feesadditional_fixed_fee_cost— optional fixed costs (e.g., registration fees)user_fee_increase— voluntary processing fee multiplier
Fee Validation Pipeline
Address-based fee validation runs in two phases:
Phase 1: Pre-Check (Minimum Balance)
Before expensive state reads, a quick estimate verifies that the inputs have enough credits to cover the minimum possible fee:
#![allow(unused)] fn main() { fn validate_addresses_minimum_balance_pre_check( &self, remaining_address_balances: &BTreeMap<PlatformAddress, (AddressNonce, Credits)>, platform_version: &PlatformVersion, ) -> Result<SimpleConsensusValidationResult, Error> }
This catches obviously insufficient balances early. Only AddressFundsTransfer,
AddressCreditWithdrawal, IdentityCreateFromAddresses, and
IdentityTopUpFromAddresses run this pre-check. AddressFundingFromAssetLock
skips it because its funds come from the asset lock, not existing address
balances.
Phase 2: Full Fee Validation
After the operations are determined, the platform:
- Applies all drive operations (in estimation mode) to calculate the actual
FeeResult - Adds validation operation costs
- Applies the
user_fee_increasemultiplier - Adds any
additional_fixed_fee_cost - Runs the fee strategy to deduct the total from inputs/outputs
- Checks
fee_fully_covered
#![allow(unused)] fn main() { let fee_deduction_result = deduct_fee_from_outputs_or_remaining_balance_of_inputs( input_current_balances.clone(), added_to_balance_outputs.clone(), fee_strategy, required_balance, platform_version, )?; if !fee_deduction_result.fee_fully_covered { return Err(AddressesNotEnoughFundsError); } }
Fee Execution
When the transition is executed, the fee deduction runs a second time with the real (not estimated) fee amount, and the adjusted balances are written to state:
- Apply all drive operations — the core state changes (transfers, nonce bumps, etc.)
- Calculate actual fee — from the FeeResult of the applied operations
- Deduct fee — run the fee strategy against the actual fee amount
- Adjust outputs — if any output was reduced, call
remove_balance_from_addressfor the difference - Adjust inputs — if any input's remaining balance was reduced, call
set_balance_to_addresswith the adjusted amount - Apply adjustment operations — batch the balance corrections into GroveDB
The fee goes into the epoch pool just like identity-based fees, and is distributed to proposers at epoch end.
Minimum Fee Calculation
For AddressFundsTransfer, the minimum fee scales with the number of inputs
and outputs:
min_fee = num_inputs × address_funds_transfer_input_cost
+ num_outputs × address_funds_transfer_output_cost
With current constants:
| Inputs | Outputs | Minimum Fee |
|---|---|---|
| 1 | 1 | 6,500,000 |
| 1 | 2 | 12,500,000 |
| 2 | 1 | 7,000,000 |
| 2 | 2 | 13,000,000 |
For IdentityCreateFromAddresses:
min_fee = identity_create_base_cost + num_keys × identity_key_in_creation_cost
| Keys | Minimum Fee |
|---|---|
| 1 | 8,500,000 |
| 2 | 15,000,000 |
| 3 | 21,500,000 |
Key Differences from Identity Fees
| Aspect | Identity Credit Fees | Platform Address Fees |
|---|---|---|
| Fee source | Identity balance (single pool) | Input addresses + output reduction |
| Fee deduction | Automatic from identity | Explicit fee strategy |
| Debt allowed | Yes (negative balance tracked) | No — must have funds |
| Refunds | Yes — deleted storage refunded to identity | No refund mechanism |
| Nonce | Per identity, per contract | Per address, monotonic u32 |
| User fee increase | Applied to processing fees | Applied to processing fees |
| Minimum fee | Per transition type | Per input + per output |
| ExecutionEvent | Paid / PaidFromAssetLock | PaidFromAddressInputs |
| Error on insufficient | BalanceIsNotEnoughError | AddressesNotEnoughFundsError |
No Debt, No Refunds
The most significant difference is that address-based transitions have no debt mechanism and no refund mechanism:
-
Identity fees can create a negative balance when the processing fee cannot be fully covered. This debt is tracked and must be repaid before the identity can submit new transitions: credits the identity receives while its balance is empty repay the debt first. From protocol version 14 the repaid part goes to the processing fee pool of the epoch it is repaid in, where the unpaid fee would have gone.
-
Address fees must be fully covered by available inputs and outputs. If the fee cannot be paid, the transition is rejected outright.
-
Identity refunds return credits to the identity when stored data is deleted. The refund is calculated per epoch based on when the data was originally stored.
-
Address-based transitions do not produce refundable storage. If an address-based transition stores data and that data is later deleted, no refund is issued to any address.
Example: AddressFundsTransfer
A concrete example of how fees flow for a two-input, two-output transfer:
Transition:
inputs: [{A, nonce: 5, amount: 1_000_000_000}, {B, nonce: 3, amount: 500_000_000}]
outputs: [{C, amount: 800_000_000}, {D, amount: 600_000_000}]
fee_strategy: [DeductFromInput(0), DeductFromInput(1)]
user_fee_increase: 0
1. Validate inputs:
A: state_nonce=4, balance=2_000_000_000 → nonce 5 ✓, balance ≥ 1B ✓
B: state_nonce=2, balance=700_000_000 → nonce 3 ✓, balance ≥ 500M ✓
Remaining: A=(5, 1_000_000_000), B=(3, 200_000_000)
2. Pre-check minimum fee:
min = 2 × 500,000 + 2 × 6,000,000 = 13,000,000
sum(remaining) = 1,200,000,000 >> 13M ✓
3. Create PaidFromAddressInputs event:
input_current_balances: {A: (5, 1B), B: (3, 200M)}
added_to_balance_outputs: {C: 800M, D: 600M}
4. Calculate actual fee:
storage_fee = 14,000,000 (hypothetical)
processing_fee = 2,500,000
total = 16,500,000
5. Apply fee strategy:
Step 1: DeductFromInput(0) → A: 1B - 16.5M = 983,500,000
remaining_fee = 0 ✓
6. Execute:
A.balance = 983,500,000, A.nonce = 5
B.balance = 200,000,000, B.nonce = 3
C.balance += 800,000,000
D.balance += 600,000,000
16,500,000 → epoch fee pool
Key Source Files
| File | Contents |
|---|---|
rs-dpp/src/address_funds/fee_strategy/mod.rs | AddressFundsFeeStrategy definition |
rs-dpp/src/address_funds/fee_strategy/deduct_fee_from_inputs_and_outputs/ | Fee deduction algorithm |
rs-drive-abci/src/execution/types/execution_event/mod.rs | PaidFromAddressInputs variant |
rs-drive-abci/src/execution/validation/.../processor/traits/address_balances_and_nonces.rs | Nonce and balance validation |
rs-drive-abci/src/execution/validation/.../processor/traits/addresses_minimum_balance.rs | Minimum balance pre-check |
rs-drive-abci/src/execution/platform_events/.../validate_fees_of_event/ | Full fee validation |
rs-drive-abci/src/execution/platform_events/.../execute_event/ | Fee execution with adjustments |
rs-platform-version/src/version/fee/state_transition_min_fees/v1.rs | Address fee constants |
Shielded Transaction Fees
Introduced in protocol version 12. For the general fee system overview, see Fee System Overview. For address-based fees (protocol versions 10--11), see Platform Address Fees.
Shielded transactions use the Orchard protocol's zero-knowledge proofs to hide transaction amounts. Because amounts are hidden, the platform cannot inspect the transaction to compute fees the way it does for transparent transitions. Instead, the fee is embedded into the cryptographic structure of the bundle itself, and the platform enforces a minimum.
This chapter explains the fee model, how it is validated, and how it differs from the transparent and address-based fee systems.
The Problem: Fees in a Privacy System
In a transparent state transition like AddressFundsTransfer, the platform can see
the transfer amount, compute the cost of storage and processing, and deduct the fee
from the sender's balance. The fee calculation happens after the transition is
applied.
Shielded transitions break this model. The amounts inside the ZK proof are hidden. The platform cannot look inside the proof to determine how much was sent or received. It can only see two things from the public fields:
value_balance— the net amount leaving the shielded pool (positive means credits flow out of the pool into the transparent world or to proposers as fees)num_actions— the number of spend+output pairs in the bundle
The fee must therefore be encoded into value_balance by the client and validated
by the platform before execution.
Fee Extraction by Transition Type
The fee is derived differently depending on the shielded transition type:
| Transition | Fee Formula | Explanation |
|---|---|---|
| Shield | fee = metered(storage + processing) + shielded_verification_fee, paid from transparent address inputs | Charged on the transparent side (not from value_balance), on top of the shielded amount. The storage and processing of the note/nullifier writes are metered by GroveDB; only the ZK compute fee (proof + num_actions × per_action_processing) is added on top. Skipped by the value_balance-based shielded fee validation; enforced through the address-input fee path. See Entry-Transition Fees. |
| ShieldedTransfer | fee = value_balance (pinned to the minimum) | The entire value_balance is the fee and must equal compute_minimum_shielded_fee(num_actions) exactly (overpayment is rejected). Nothing leaves the pool except the fee. |
| Unshield | fee = compute_minimum_shielded_fee(num_actions) + unshield_address_storage_fee | value_balance (the transition's unshielding_amount) is the gross amount leaving the pool. The output address receives unshielding_amount − fee; validation requires unshielding_amount ≥ fee. Unshield also writes the net to the output platform address (AddBalanceToAddress), a real storage write priced on top of the base shielded minimum (unshield_address_storage_fee = 222 × per_byte_rate, ≈6.08M credits, flat regardless of action count — 222 bytes is the storage portion of the ≈6.24M metered address write) so the address write is covered and the proof fee isn't diverted to pay for it. See Per-Action Storage Fee. |
| ShieldedWithdrawal | fee = compute_minimum_shielded_fee(num_actions) + withdrawal_document_storage_fee | value_balance (unshielding_amount) is the gross amount leaving the pool. The Core withdrawal document receives unshielding_amount − fee (which must also clear MIN_WITHDRAWAL_AMOUNT). Unlike the other pool-paid transitions, ShieldedWithdrawal also writes a Core withdrawal document — a real document insert into the withdrawals contract plus its index entries (AddWithdrawalDocument), with a real metered cost of ≈110M credits that is flat regardless of action count. That cost is priced on top of the base shielded minimum as a flat ~4,100-byte storage component (withdrawal_document_storage_fee = 4100 × per_byte_rate), so the document write is covered and the proof-verification fee isn't diverted from the proposer to pay for it. See Per-Action Storage Fee. |
| ShieldFromAssetLock | pool_fee = compute_minimum_shielded_fee(num_actions) + asset_lock_base_cost, paid from the asset lock | The flat shielded minimum plus the asset-lock processing base cost is routed to the fee pools. Any remaining asset-lock value (the surplus) goes to an optional signed surplus_output platform address, or — if none is set — folds into the fee pools up to shielded_implicit_fee_cap. See Entry-Transition Fees. |
| IdentityCreateFromShieldedPool | total_fee = metered(insert_nullifiers + AddNewIdentity(identity + N keys)) + shielded_verification_fee, moved from the new identity's balance | value_balance is a fixed denomination (a member of the versioned set {0.1, 0.3, 0.5, 1.0} DASH) and must equal it EXACTLY. The new identity is created holding the full denomination, funded by decrementing the shielded pool by exactly that amount — a move between two balance trees (like Unshield's pool→address), so the global system-credit supply is unchanged (no AddToSystemCredits); the fee is then moved from that balance into the fee pools, so the identity ends with denomination − total_fee. Unlike the flat pool-paid transitions, the AddNewIdentity write grows with the key count, so the cost is metered (not a flat carve) — only the ZK compute fee (compute_shielded_verification_fee) is added on top, exactly like the transparent Shield. The client predicts it offline with compute_shielded_identity_create_fee(num_actions, num_keys); consensus rejects denomination < total_fee with IdentityInsufficientBalanceError. |
| ShieldFromIdentity | fee = metered(storage + processing) + shielded_verification_fee, paid from the funding identity's balance | Identity balance to pool (protocol version 14). Charged exactly like Shield, but on the identity side: the identity signature covers the whole outputs-only bundle, the metered note writes and identity writes go through the standard identity-paid path (IdentityCreditTransferToAddresses model), and only the ZK compute fee is added as additional_fixed_fee_cost. user_fee_increase applies. The identity must hold amount + fee; consensus rejects a short balance with IdentityInsufficientBalanceError. The pool and the identity are both balance trees, so no system-credit adjustment is emitted. See Entry-Transition Fees. |
| IdentityTopUpFromShieldedPool | fee = compute_shielded_identity_top_up_fee(num_actions) = compute_minimum_shielded_fee(num_actions) + identity_balance_storage_fee, carved from value_balance | Shielded pool to an EXISTING identity's balance (protocol version 14). value_balance (the transition's topUpAmount) is the gross amount leaving the pool; the identity receives topUpAmount - fee and validation requires topUpAmount >= fee. Same flat pool-paid model as Unshield, with the identity balance write as a flat component built like Unshield's address write but calibrated to its measured cost: the top-up rewrites the existing identity's balance element and its Merk path (320 replaced bytes, 175,320 credits of processing, no storage), folded into one flat figure with headroom like the other shielded components, so identity_balance_storage_fee = 8 x per_byte_rate (SHIELDED_IDENTITY_TOP_UP_BALANCE_STORAGE_BYTES). The target identity and gross amount are bound into the Orchard sighash; the identity must already exist; no system-credit adjustment. |
Token shielded pool fees
Token pools (protocol version 14, see Token Shielded Pools)
hold tokens, and tokens cannot pay fees, so none of the three token pool transitions carves a
fee from the bundle. They are TokenTransition variants inside a Batch, and the batch's
signing identity pays in credits through the standard identity-paid path.
| Transition | Fee Formula | Explanation |
|---|---|---|
| TokenShield | fee = metered(storage + processing) + shielded_verification_fee, paid by the signing identity | Same model as ShieldFromIdentity: the dummy nullifier inserts, the note appends, the identity token balance write and the pool balance write are metered, and compute_shielded_verification_fee(num_actions) is added as a precalculated operation before the proof is verified. value_balance is -amount in tokens and carries no fee. |
| TokenUnshield | fee = metered(storage + processing) + shielded_verification_fee, paid by the signing identity | value_balance equals the unshielded token amount exactly; the recipient receives the full amount. Nullifier inserts, note appends and the two balance writes are metered. |
| TokenShieldedTransfer | fee = metered(storage + processing) + shielded_verification_fee, paid by the signing identity | value_balance is exactly zero; consensus rejects any other value. Only the nullifier inserts and note appends are metered. |
| TokenMintToPool, TokenClaimToPool, TokenDirectPurchaseToPool | same model | Outputs-only bundles; the dummy nullifier inserts, the note appends and the supply and pool balance writes are metered, the verification fee is charged by the action transformer so CheckTx and block execution price the bundle identically. |
| TokenBurnFromPool | same model | Nullifier inserts, change note appends and the supply and pool balance writes are metered. |
| Document with a TokenPaymentInfo::V1 | same model, on top of the document's own fee | The shielded payment bundle's verification fee is added by the batch transformer before anything can fail; the pool writes it causes are metered. |
Because the verification fee is charged before the proof is checked, an invalid proof is a paid failure: the identity is charged, its identity contract nonce advances, and no token moves.
Identity-less token pool transitions
TokenShieldedTransferWithShieldedFee (26), TokenUnshieldWithShieldedFee (27) and
TokenPurchaseFromShieldedPool (28) have no identity: the fee is carved from a second bundle
spent in the credit shielded pool, exactly as ShieldedTransfer carves its own fee.
| Transition | Fee Formula | Explanation |
|---|---|---|
| TokenShieldedTransferWithShieldedFee | credit_amount == base(fee_actions) + base(token_actions) | Two bundles are verified and stored, so the fee is the base shielded fee of each (compute_token_pool_paid_shielded_fee). Pure fee: overpayment is rejected. |
| TokenUnshieldWithShieldedFee | credit_amount == base(fee_actions) + base(token_actions) + identity balance bytes | Adds the flat storage of the recipient's token balance item. Pure fee. |
| TokenPurchaseFromShieldedPool | credit_amount == total_agreed_price + base(fee_actions) + base(token_actions) + balance write + supply bytes | The agreed price rides on top of the fee and is credited to the contract owner; credit_amount - total_agreed_price must equal the fee exactly. |
The fee bundle's value balance is credit_amount; the minimum-fee validation, the SDK builders
and the transformer all use the same compute_token_* function so the threshold never drifts
from what is carved. An invalid proof or a spent nullifier is an unpaid rejection: nothing is
committed and no fee is charged, the same as for the credit pool's pool-paid transitions.
For ShieldedTransfer, the client constructs the bundle so that total_spent − total_output = desired_fee. The Orchard circuit proves that value is conserved
(inputs = outputs + value_balance), and the binding signature cryptographically
commits to the value_balance. Mutating value_balance after signing will cause
the binding signature to fail verification.
Entry-Transition Fees (Shield, ShieldFromAssetLock, and ShieldFromIdentity)
The entry transitions — Shield (transparent → shielded), ShieldFromAssetLock
(Core asset lock → shielded), and, from protocol version 14, ShieldFromIdentity
(identity balance → shielded) — move value into the pool, so
there is no spent note from which value_balance could carry a fee. Their fees are
therefore charged from the funding side, and both cover the same Halo 2 proof
verification and per-action work the other shielded transitions pay for — but they
account for it differently. Shield debits a state-queryable transparent address
balance, so GroveDB meters its real storage/processing and only the compute portion
(compute_shielded_verification_fee, no storage term) is added on top. ShieldFromAssetLock
is funded by a consumed asset lock with no metering anchor, so it pays the flat
compute_minimum_shielded_fee(num_actions) (plus the asset-lock base cost). num_actions
is the on-wire action count of the bundle (a single-output, spends-disabled Orchard bundle
pads to 2 actions, so the minimum is the 2-action fee).
Every action of an outputs-only bundle still reveals a nullifier, that of a dummy spend,
which becomes the new note's rho. From protocol version 14 the entry transitions record
those nullifiers and refuse one repeated inside the bundle or already recorded
(NullifierAlreadySpentError), as the spends do, so every revealed nullifier is recorded
once. Shield and ShieldFromIdentity meter the nullifier writes like the rest of their
storage; ShieldFromAssetLock's flat fee already prices a note and a nullifier write per
action (see Per-Action Storage Fee).
Shield
Shield is charged like any other address-funded transition: GroveDB meters the
real storage and processing cost of applying it (the note-commitment and nullifier
writes plus the address-balance updates), and the shielded compute fee is added on
top:
fee = metered_storage + metered_processing + shielded_verification_fee
shielded_verification_fee = proof_verification_fee + num_actions × per_action_processing_fee
shielded_verification_fee is the ZK-verification cost (Halo 2 proof + per-action spend-auth
verification) that GroveDB metering cannot see. It is added as the transition's
additional_fixed_fee_cost — exactly the mechanism IdentityCreateFromAddresses uses
for its registration cost. It carries no storage term: storage comes entirely from
metering, so it is never double-counted. The address inputs must cover shield_amount + fee, and the booked storage/processing equals the deducted amount, so credits are
conserved by the standard machinery (no special-case override).
Shield is skipped by the value_balance-based minimum-fee validation (its
value_balance is the amount entering the pool, not a fee). The stateless structure
floor requires only shield_amount + shielded_verification_fee (a conservative lower bound,
since metered storage is unknowable without state); the authoritative metered + compute funding gate is validate_fees_of_event.
ShieldFromIdentity
ShieldFromIdentity (protocol version 14) is Shield with the identity balance as
the funding side. It is identity-signed (TRANSFER key, identity nonce) like
IdentityCreditTransferToAddresses, and carries the same outputs-only Orchard bundle
as Shield. The fee model is identical to Shield's: GroveDB meters the note and
nullifier inserts and the identity balance and nonce writes, and the shielded compute fee is
added as additional_fixed_fee_cost:
fee = metered_storage + metered_processing + shielded_verification_fee
identity_balance_after = identity_balance_before - amount - fee
user_fee_increase applies to the metered processing portion. The stateless
floor requires identity_balance >= amount + compute_shielded_identity_balance_write_fee,
the conservative complete admission estimate: compute_minimum_shielded_fee, plus
PV14's versioned shielded_identity_action_write_storage_bytes (400 effective
bytes per action) and shielded_identity_balance_write_storage_bytes (500 flat
bytes), each priced at the storage rate. The allowances cover the complete
execution-event estimate: note/nullifier writes at the estimator's depth 16,
identity nonce/balance writes at its maximum-element depth, and the signature
and state-read validation context. They are admission reserves, not physical
payload sizes or changes to the actual metered charge. For two actions the
base wallet estimate is 114,140,000 + (2 × 400 + 500) × 27,400 = 149,760,000
credits. Historical tables preserve the preceding zero per-action and 20-byte
flat allowance. An identity that could not pay the
complete fee is therefore refused before the Orchard proof is verified. The authoritative gate is the identity-paid fee
validation of the execution event (Paid), which rejects with
IdentityInsufficientBalanceError. The identity balance and the pool total are
both terms of the block conservation equation, so the converter emits no
AddToSystemCredits (the same rule Unshield and IdentityCreateFromShieldedPool
follow).
A failed Orchard proof is a paid failure, not a free rejection. Like
ShieldFromAssetLock, the proof is verified inside the transition's own transform
rather than in the shared stateless proof step: on failure the transition executes
as a BumpIdentityNonceAction, consuming the identity nonce and charging the
identity the versioned shielded_proof_verification_failure penalty on top of the
processing metered so far. This closes the path where a funded identity could
resubmit invalid proofs indefinitely with its nonce and balance left untouched.
CheckTx never charges; it admits the verification under its node-local proof
budget only after the cheap checks passed, and rejects on failure.
ShieldFromAssetLock
The asset lock funds the pool, so the fee is taken from the consumed asset-lock value. The pool fee is:
pool_fee = compute_minimum_shielded_fee(num_actions) + asset_lock_base_cost
asset_lock_base_cost is the same asset-lock-proof processing base cost charged to
every asset-lock-funded transition (e.g. IdentityCreate):
required_asset_lock_duff_balance_for_processing_start_for_address_funding
(50,000 duffs) × CREDITS_PER_DUFF (1,000) = 50,000,000 credits. Adding it makes
the ShieldFromAssetLock pool fee strictly greater than the bare F paid by the
transparent Shield, pricing the extra cost of verifying the Core asset-lock proof.
The asset lock must cover shield_amount + pool_fee; the remainder is the surplus:
surplus = consumed_asset_lock_value − shield_amount − pool_fee (always ≥ 0)
The surplus is disposed of in one of two ways:
surplus_outputset — the transition carries an optionalOption<PlatformAddress>surplus_output. When present,surplusis credited to that platform address (via anAddBalanceToAddressdrive operation). This field is part of the signed payload (it sits before thesignaturefield, which alone is excluded from the sighash), so a surplus recipient cannot be substituted or truncated after signing.surplus_outputunset — the surplus folds into the fee pools, but only up toshielded_implicit_fee_cap(20,000,000,000 credits = 0.2 DASH, a versioned constant). If the unclaimed surplus would exceed the cap, the transition is rejected withShieldedImplicitFeeCapExceededErrorso a client cannot accidentally donate a large remainder to proposers. To intentionally over-fund, the client must setsurplus_output(which has no cap).
Value conservation across the whole transition is exact:
consumed_asset_lock_value = shield_amount + surplus_amount + fee_amount
where surplus_amount is surplus when surplus_output is set and 0 otherwise (in
which case the surplus is part of fee_amount).
The Three-Component Fee Model
The minimum shielded fee has three components:
min_fee = proof_verification_fee + num_actions × (processing_fee + storage_fee)
1. Proof Verification Fee (per bundle)
A single Halo 2 ZK proof covers the entire bundle regardless of action count. The bundle's base verification work is benchmarked at approximately 5 ms. This is a fixed cost per bundle.
Protocol version 14 value: 40,000,000 credits (40M)
2. Per-Action Processing Fee
The per-action processing fee prices the marginal Halo 2 verification work that each additional action adds to the bundle (≈1.1 ms/action measured against a ≈5 ms bundle base): a bundle with more actions is a larger circuit and a longer batch verification. For a spend-bearing action that marginal work includes:
- RedPallas spend authorization signature verification
- Nullifier duplicate check (hash + tree lookup)
- Note commitment insertion into the Sinsemilla-based Merkle tree
Output-only entry transitions (Shield / ShieldFromAssetLock / ShieldFromIdentity) do no spends, but each output action still enlarges the proof and so carries the same per-action processing charge — this fee tracks the marginal verification work, not a fixed per-action checklist. From protocol version 14 it also prices the check of the nullifier each of their actions reveals.
At protocol version 14, the proof and per-action fees are versioned independently. Their numerical ratio is about 1.8:1 (40M : 22M).
Current value: 22,000,000 credits (22M)
3. Per-Action Storage Fee
Each action permanently stores data in two places:
| Storage | Bytes | Contents |
|---|---|---|
| BulkAppendTree (commitment tree) | 312 | 32 cmx + 32 rho + 32 cv_net + 216 encrypted note |
| Nullifier tree | 32 | nullifier key (value is empty) |
| Total physical payload | 344 |
Protocol version 14 prices a 550-byte allowance per action, covering the physical payload and database framing, at the platform's per-byte storage rates:
storage_fee_per_action = 550 × (storage_disk_usage_credit_per_byte
+ storage_processing_credit_per_byte)
= 550 × (27,000 + 400)
= 550 × 27,400
= 15,070,000
The byte allowance is versioned. Its per-byte rates come from the storage fee version, so the fee tracks changes to those rates.
Fee Table
Combining all three components at protocol version 14:
| Actions | Proof Fee | Processing | Storage | Total Minimum Fee |
|---|---|---|---|---|
| 2 | 40,000,000 | 44,000,000 | 30,140,000 | 114,140,000 |
| 3 | 40,000,000 | 66,000,000 | 45,210,000 | 151,210,000 |
| 4 | 40,000,000 | 88,000,000 | 60,280,000 | 188,280,000 |
Note: The Orchard protocol requires a minimum of 2 actions per bundle for privacy (even a single-input single-output transfer produces 2 actions with a dummy padding action). Bundles with 1 action are structurally invalid.
The totals above are the base compute_minimum_shielded_fee and apply directly to
ShieldedTransfer. The three pool-paid transitions that write one extra per-transition output
add a flat component on top of this base:
Unshieldadds the output-address write cost: a flatunshield_address_storage_fee = 222 × per_byte_rate = 222 × 27,400 = 6,082,800credits, independent of action count. So the 2-action Unshield fee is114,140,000 + 6,082,800 = 120,222,800credits (and likewise+6,082,800at every action count). See the Fee Extraction Unshield row for why this component exists.ShieldedWithdrawaladds the Core withdrawal-document storage cost: a flatwithdrawal_document_storage_fee = 4100 × per_byte_rate = 4100 × 27,400 = 112,340,000credits, independent of action count. So the 2-action ShieldedWithdrawal fee is114,140,000 + 112,340,000 = 226,480,000credits (and likewise+112,340,000at every action count). See the Fee Extraction ShieldedWithdrawal row for why this component exists.IdentityTopUpFromShieldedPooladds the identity balance write cost: a flatidentity_balance_storage_fee = 8 × per_byte_rate = 8 × 27,400 = 219,200credits, independent of action count, so the top-up fee at any action count is the base plus219,200. See the Fee Extraction IdentityTopUpFromShieldedPool row for why this component is so much smaller than the address write: it rewrites an existing balance element instead of storing a new entry.
Where Fee Validation Runs
Fee validation is integrated into the processor pipeline (see Validation Pipeline) between basic structure validation and ZK proof verification:
... → Basic Structure → Minimum Fee Check → ZK Proof Verification → ...
The ordering is deliberate. The fee check is stateless and cheap — it only reads
value_balance and actions.len() from the transition, with no GroveDB lookups.
Placing it before proof verification means that bundles with insufficient fees are
rejected instantly, without spending ~100ms on Halo 2 verification.
The implementation lives in
packages/rs-drive-abci/src/execution/validation/state_transition/processor/traits/shielded_proof.rs:
#![allow(unused)] fn main() { pub(crate) trait StateTransitionShieldedMinimumFeeValidationV0 { fn validate_minimum_shielded_fee( &self, platform_version: &PlatformVersion, ) -> Result<SimpleConsensusValidationResult, Error>; } }
If the fee is below the minimum, the transition is rejected with
InsufficientShieldedFeeError — an unpaid consensus error. The sender is not
charged (there is no identity to charge), and the transition produces no execution
event.
Fee Constants in the Version System
The fee parameters are stored in the platform version under
drive_abci.validation_and_processing.event_constants:
#![allow(unused)] fn main() { pub struct DriveAbciValidationConstants { pub maximum_vote_polls_to_process: u16, pub maximum_contenders_to_consider: u16, pub minimum_pool_notes_for_outgoing: u64, pub shielded_anchor_retention_blocks: u64, pub shielded_anchor_pruning_interval: u64, pub shielded_proof_verification_fee: u64, // 40_000_000 at protocol 14 pub shielded_per_action_processing_fee: u64, // 22_000_000 pub shielded_storage_bytes_per_action: u64, // 550 at protocol 14 pub shielded_implicit_fee_cap: u64, // 20_000_000_000 (0.2 DASH) } }
The shielded_implicit_fee_cap bounds the surplus that a ShieldFromAssetLock may
implicitly donate to the fee pools when no surplus_output is set (see
Entry-Transition Fees).
The storage component is derived at runtime from
fee_version.storage.storage_disk_usage_credit_per_byte and
fee_version.storage.storage_processing_credit_per_byte, multiplied by the
versioned shielded_storage_bytes_per_action allowance.
This design means:
- Proof and processing fees can be tuned independently via version bumps
- Storage fees automatically track changes to the platform-wide storage rates
- Storage allowances can be calibrated independently of the per-byte rates
How Fees Flow After Validation
Once the fee check passes and the transition is fully validated and executed, the
shielded pool's total balance is decremented and the fee is booked via the
PaidFromShieldedPool execution event:
ShieldedTransfer: pool_balance -= fee_amount // fee == value_balance
Unshield: pool_balance -= unshielding_amount // gross
ShieldedWithdrawal: pool_balance -= unshielding_amount // gross
IdentityTopUpFromShieldedPool: pool_balance -= top_up_amount // gross
For Unshield and ShieldedWithdrawal, unshielding_amount is the gross amount
leaving the pool. Of that, unshielding_amount − fee_amount is credited to the output
platform address (Unshield) or written into the Core withdrawal document
(ShieldedWithdrawal), and fee_amount is booked as the transition fee. For Unshield
that fee is compute_shielded_unshield_fee — the base fee plus the flat
AddBalanceToAddress output-write storage cost (+6,082,800 credits), since Unshield
also writes the net to a transparent platform address. For ShieldedWithdrawal it is
compute_shielded_withdrawal_fee — the same base fee plus the flat Core
withdrawal-document storage cost (+112,340,000 credits), since ShieldedWithdrawal also
writes a real document into the withdrawals contract. Validation guarantees
unshielding_amount ≥ fee_amount (and, for ShieldedWithdrawal, that the net also clears
MIN_WITHDRAWAL_AMOUNT), so the subtraction never underflows. Because each fee prices its
extra write, the booking split (storage routed to the storage pool, the remainder paid to
the proposer) covers that write instead of zeroing the proposer's processing reward to
cover it.
For IdentityTopUpFromShieldedPool, top_up_amount is likewise the gross amount leaving the
pool: top_up_amount − fee_amount is added to the existing identity's balance and
fee_amount (compute_shielded_identity_top_up_fee, the base fee plus the flat identity
balance write component) is booked as the transition fee; validation guarantees
top_up_amount ≥ fee_amount. The identity balance and the pool total are both terms of the
block conservation equation, so no system-credit adjustment is emitted.
For ShieldedTransfer, the pool decreases by exactly the fee (the sender's notes are
spent and the recipient's notes are created, but the pool's aggregate balance only drops
by the fee).
In all cases the booked fee_amount is split the same way as every other transition's
fee: the storage cost of the permanent shielded writes is routed to the storage pool
(amortized across epochs and subject to the per-epoch fee multiplier at payout), and the
remainder — proof verification plus per-action processing — is the processing fee paid to
the current block proposer.
The three entry transitions (Shield, ShieldFromAssetLock, ShieldFromIdentity) do not decrement the pool (they add to it), so their fees are
booked from the funding side instead:
Shield: fee_amount = metered + shielded_verification_fee // from transparent address inputs
ShieldFromAssetLock: fee_amount = pool_fee (+ unclaimed surplus) // from the consumed asset lock
For Shield, the fee is deducted from the transparent address inputs and booked through
the standard PaidFromAddressInputs event (deducted == booked, no override): metered
storage and processing, plus the shielded_verification_fee folded into processing. For
ShieldFromAssetLock, the consumed asset-lock value is partitioned into shield_amount
(into the pool), surplus_amount (to surplus_output, or 0), and fee_amount (to the
fee pools); see Entry-Transition Fees.
Cryptographic Binding
The fee is not just a field that the platform trusts. It is cryptographically bound to the ZK proof through two mechanisms:
-
Value commitments (cv_net): Each action contains a Pedersen commitment to the note value. The sum of all value commitments must equal
value_balance(modulo the blinding factors). The binding signature proves this relationship holds. -
Platform sighash: The bundle commitment (which includes
value_balance) is hashed into the sighash that the spend authorization signatures sign over:sighash = SHA-256("DashPlatformSighash" || bundle_commitment || extra_data)Mutating
value_balanceafter signing changes the sighash, invalidating all signatures. TheBatchValidatorchecks both the Halo 2 proof and all signatures, so any tampering is caught.
This means a client cannot claim a lower fee than what the ZK proof actually commits to — the proof and signatures would fail verification.
Rules and Guidelines
Do:
- Always set
value_balanceto at least the minimum fee when building a shielded bundle on the client side. Usemin_fee = proof_verification_fee + num_actions × (processing_fee + storage_fee)with the current platform version constants. - Include the fee in the note arithmetic:
total_spent = total_output + fee. The Orchard builder handles this when you set the output amount tospend_amount − desired_fee. - Remember that the minimum action count is 2 (Orchard privacy requirement).
Do not:
- Assume the fee is free for
Shieldtransitions — the fee comes from transparent address inputs and is validated through the address balance system, not here. - Mutate
value_balanceafter building the bundle. The binding signature and sighash will be invalidated. - Hardcode fee amounts. Always read from
PlatformVersion— the constants are versioned and will change as the protocol evolves.
What a Document Costs
drive::document::cost computes what creating one document costs, from the
contract alone. The JavaScript SDKs expose it as
documentCreateCost(contract, documentTypeName, options, platformVersion),
which is what the contract visualizer shows per document type.
Amounts are in credits: 1 Dash is 100,000,000,000 credits.
Storage
The storage fee is the bytes the insert adds times
storage_disk_usage_credit_per_byte (27,000 credits from protocol version 14).
It is usually almost all of what a document costs.
The estimate is exact. It lists every element the insert writes (see GroveDB Structure for the layout), built with the functions the insert walkers use:
- the document by id, or with
documentsKeepHistorya tree of revisions and a pointer to the newest; - for each index, the value trees along its properties (created only when no
earlier document has the value), the
[0]terminal and the document's reference in it, or an indexOnly type's entry; - the rows a ranked index keeps for the value in its secondary trees, one per ranking axis;
- the trees preallocated for other document types'
preallocatedindexes whose entries will reference the document, charged to its creator.
Each element is priced with GroveDB's byte formulas: the key with the 32-byte subtree prefix, the serialized element (or a fixed size standing for a tree), the value and node hashes, the aggregate feature of the tree it sits in (8 or 16 bytes in a count or sum tree), and the link its parent keeps to it. A document create's elements carry 35 bytes of storage flags (the owner and the epoch); index trees carry them only when the documents are mutable, the contract can be deleted, or the type is indexOnly and its documents can be deleted.
The storage depends on what is already stored, so it comes in two scenarios:
- every value new: the first document with these index values creates their trees;
- every value known: a later document with the same values adds only its
own entries. A unique index still adds its value, which no earlier document
can hold, as does every tree keyed by the document's own id (a preallocated
index's, say) and a
ttldocument's expiration entry; a ranked index's row for an existing value only moves, which GroveDB bills as replaced bytes.
The estimate also splits the storage by index: the layers an index shares with
other indexes (a common prefix of properties, paid once for the document) and
the layers only it uses, so an index's cost on its own is its shared layers
plus its own. A skipIfAbsent index that skips the document writes nothing,
and adds no layer, just as Drive builds none for it; the rows of a ranked
level under a time window with a ttl are priced like the window, as
processing.
The test should_price_what_drive_charges inserts documents into 21 contracts
covering every index shape and requires the estimate to equal the storage fee
Drive charged, byte for byte, with the trees already stored read from GroveDB
before each insert.
Processing
- Exact: verifying the signature (15,000 credits for an ECDSA key, 300,000 for BLS) and fetching the signing key and the identity's balance (18,000).
- Estimated: the work of the writes (seeks, hashing and rewriting the path
to each new element in its tree and above), for an assumed number of stored
documents, each with its own values (1,000 by default); and the small reads
and writes around the insert (the document id check and the identity's
contract nonce). The test
should_estimate_the_processing_of_the_writes_within_a_factor_of_twoholds the write estimate to Drive's processing fee. Processing is a few percent of a document's cost. - A
userFeeIncreaseraises the processing fee only.
The per-document "minimum fee" of a batch (document_batch_sub_transition) is
a balance check before processing, not a charge, and the unique index checks
and the fetch of the batch's own contract are not billed.
What the contract adds
- Action fees (
actionFees): the create's fee, in credits, scaled by the epoch's fee multiplier unless priced as fixed, paid into the contract's fee pots. - Token cost (
tokenCost): tokens transferred to the contract owner or burned. - Contest fund: a contested index's vote fund (0.1 Dash, doubling past 250 contenders), paid when the value is contested.
Refunds
Deleting a document refunds the storage fee of its flagged elements, less what the epochs already passed were paid: about 99.9% in the epoch it was created, about 95% a year later, then less each year for fifty years.
Documents with a ttl
A document whose type declares a ttl is
stored without flags and pays for its bytes by the lifetime it has left, all of
its ttl when it is created, at the schedule's tier for that lifetime (from 1
credit per byte for an hour to 26 for a week, then 34 per 788,400 seconds)
instead of the 27,000 of storage kept for good. It also adds its entry to the
documents expirations tree, and prepays its deletion as processing (a base
cost, a cost per index level and one per document byte). Nothing of it is
refunded.
Not covered
- A create whose value starts a contest (a DPNS name matching the contest rule, say) is stored in the contest's vote poll until the contest ends, not in the index. The estimate prices an uncontested create and lists the contest fund; the vote poll's storage is not priced.
- A platform version whose insert methods differ from protocol version 14's is refused: other versions write other elements for some shapes.
Consensus Errors
When a Dash Platform node processes a state transition -- a document creation, an identity update, a credit withdrawal -- things can go wrong. The data contract might reference an invalid schema. The signature might not match. The identity might not have enough balance. These are not internal bugs; they are protocol-level validation failures that every node on the network must agree on. If one node rejects a transition for reason X and another rejects it for reason Y, the chain forks. If one node returns error code 40100 and another returns 40101, clients break.
This is why consensus errors in Dash Platform are not ordinary Rust errors. They are serializable, code-stable, network-transmitted data structures that must produce identical bytes on every node and remain decodable by every client version. The ConsensusError enum is the heart of this system.
The top-level enum
Open packages/rs-dpp/src/errors/consensus/consensus_error.rs and you will find the root of the hierarchy:
#![allow(unused)] fn main() { #[derive( thiserror::Error, Debug, Encode, Decode, PlatformSerialize, PlatformDeserialize, Clone, PartialEq, )] #[platform_serialize(limit = 2000)] #[error(transparent)] #[allow(clippy::large_enum_variant)] pub enum ConsensusError { /* DO NOT CHANGE ORDER OF VARIANTS WITHOUT INTRODUCING OF NEW VERSION */ #[error("default error")] DefaultError, #[error(transparent)] BasicError(BasicError), #[error(transparent)] StateError(StateError), #[error(transparent)] SignatureError(SignatureError), #[error(transparent)] FeeError(FeeError), #[cfg(test)] #[cfg_attr(test, error(transparent))] TestConsensusError(TestConsensusError), } }
There are five things worth understanding here.
The four categories
Every consensus error falls into one of four categories:
-
BasicError -- structural and syntactic validation failures. The state transition itself is malformed, references a nonexistent document type, has an invalid identifier, exceeds size limits, or fails schema validation. These are caught before the node ever checks persistent state. The
BasicErrorenum inpackages/rs-dpp/src/errors/consensus/basic/basic_error.rscontains over 130 variants organized into sub-groups: versioning errors, structure errors, data contract errors, group errors, document errors, token errors, identity errors, state transition errors, and address errors. -
StateError -- the transition is structurally valid but conflicts with the current platform state. A document already exists, an identity nonce is wrong, a token account is frozen, a group action was already completed. The
StateErrorenum inpackages/rs-dpp/src/errors/consensus/state/state_error.rscontains roughly 80 variants covering data contracts, documents, identities, voting, tokens, groups, and address balances. -
SignatureError -- the cryptographic signature on the transition is invalid. The identity was not found, the key type is wrong, the key is disabled, the security level is insufficient, or the raw signature verification failed.
-
FeeError -- the transition is valid in every other way, but the identity cannot pay for it. Currently this category has a single variant:
BalanceIsNotEnoughError.
This layered structure mirrors the validation pipeline. Platform checks basic structure first, then state, then signatures, then fees. Each layer produces errors from its own category.
The DO NOT CHANGE ORDER rule
The comment at the top of every consensus error enum is not a suggestion:
#![allow(unused)] fn main() { /* DO NOT CHANGE ORDER OF VARIANTS WITHOUT INTRODUCING OF NEW VERSION */ }
Why? Because ConsensusError derives Encode and Decode from bincode. Bincode serializes enum variants by their ordinal position -- the first variant is 0, the second is 1, and so on. If you reorder variants, the same bytes now decode to a different error on nodes running different code versions. This is a consensus failure.
The same rule applies to every nested error enum. Here is FeeError in packages/rs-dpp/src/errors/consensus/fee/fee_error.rs:
#![allow(unused)] fn main() { #[derive( Error, Debug, PartialEq, Encode, Decode, PlatformSerialize, PlatformDeserialize, Clone, )] pub enum FeeError { /* DO NOT CHANGE ORDER OF VARIANTS WITHOUT INTRODUCING OF NEW VERSION */ #[error(transparent)] BalanceIsNotEnoughError(BalanceIsNotEnoughError), } }
And the same rule applies to individual error structs. Here is DocumentAlreadyPresentError in packages/rs-dpp/src/errors/consensus/state/document/document_already_present_error.rs:
#![allow(unused)] fn main() { #[derive( Error, Debug, Clone, PartialEq, Eq, Encode, Decode, PlatformSerialize, PlatformDeserialize, )] #[error("Document {document_id} is already present")] #[platform_serialize(unversioned)] pub struct DocumentAlreadyPresentError { /* DO NOT CHANGE ORDER OF FIELDS WITHOUT INTRODUCING OF NEW VERSION */ document_id: Identifier, } }
Notice the comment says fields too, not just variants. Bincode serializes struct fields in declaration order. Swap two fields and the bytes change.
The derive stack
Every consensus error type carries the same set of derives:
#![allow(unused)] fn main() { #[derive( Error, // thiserror -- provides Display and Error trait Debug, // standard Rust debug formatting Clone, // errors need to be cloneable PartialEq, // errors need to be comparable Encode, // bincode encoding (field-level) Decode, // bincode decoding (field-level) PlatformSerialize, // platform wrapper around bincode (size limits, etc.) PlatformDeserialize,// platform wrapper around bincode )] }
The thiserror::Error derive gives each variant a Display implementation. For enums, #[error(transparent)] delegates to the inner type's Display. For leaf structs, you write the message directly:
#![allow(unused)] fn main() { #[error("Document {document_id} is already present")] pub struct DocumentAlreadyPresentError { document_id: Identifier, } }
The Encode and Decode derives come from bincode and handle the actual byte-level serialization of each field. The PlatformSerialize and PlatformDeserialize derives are the platform's own wrappers that add size limits and error type conversion on top of bincode. We will cover those in the serialization chapters.
The #[platform_serialize] attribute
On the top-level ConsensusError, you will notice:
#![allow(unused)] fn main() { #[platform_serialize(limit = 2000)] }
This sets a maximum serialized size of 2000 bytes. If a consensus error somehow serializes to more than 2000 bytes, the platform will return a MaxEncodedBytesReachedError instead. This is a defense against oversized error payloads being transmitted across the network.
On individual leaf error structs, you will see a different attribute:
#![allow(unused)] fn main() { #[platform_serialize(unversioned)] }
The unversioned flag means the struct does not need a PlatformVersion parameter for serialization. Most leaf error structs are simple enough that they do not change between protocol versions -- their serialization format is stable. The top-level ConsensusError enum does not use unversioned because it may need version-aware behavior as the protocol evolves.
How errors nest
The nesting is three levels deep:
- ConsensusError -- the top level, with four category variants
- Category enums (BasicError, StateError, SignatureError, FeeError) -- each holds dozens of specific error variants
- Leaf error structs -- the actual error data (identifiers, messages, amounts)
Each level has From implementations to make conversion ergonomic:
#![allow(unused)] fn main() { // Leaf -> Category impl From<DocumentAlreadyPresentError> for ConsensusError { fn from(err: DocumentAlreadyPresentError) -> Self { Self::StateError(StateError::DocumentAlreadyPresentError(err)) } } // Category -> Top-level impl From<StateError> for ConsensusError { fn from(error: StateError) -> Self { Self::StateError(error) } } }
This means you can use ? to propagate any leaf error up to a ConsensusError:
#![allow(unused)] fn main() { fn validate_document(doc: &Document) -> Result<(), ConsensusError> { if document_exists(doc.id()) { return Err(DocumentAlreadyPresentError::new(doc.id()).into()); } Ok(()) } }
The .into() call walks the From chain: DocumentAlreadyPresentError -> ConsensusError::StateError(StateError::DocumentAlreadyPresentError(...)).
Anatomy of a leaf error
Every leaf error follows the same pattern. Let us look at the full DocumentAlreadyPresentError:
#![allow(unused)] fn main() { use crate::consensus::state::state_error::StateError; use crate::consensus::ConsensusError; use crate::errors::ProtocolError; use bincode::{Decode, Encode}; use platform_serialization_derive::{PlatformDeserialize, PlatformSerialize}; use platform_value::Identifier; use thiserror::Error; #[derive( Error, Debug, Clone, PartialEq, Eq, Encode, Decode, PlatformSerialize, PlatformDeserialize, )] #[error("Document {document_id} is already present")] #[platform_serialize(unversioned)] pub struct DocumentAlreadyPresentError { /* DO NOT CHANGE ORDER OF FIELDS WITHOUT INTRODUCING OF NEW VERSION */ document_id: Identifier, } impl DocumentAlreadyPresentError { pub fn new(document_id: Identifier) -> Self { Self { document_id } } pub fn document_id(&self) -> &Identifier { &self.document_id } } impl From<DocumentAlreadyPresentError> for ConsensusError { fn from(err: DocumentAlreadyPresentError) -> Self { Self::StateError(StateError::DocumentAlreadyPresentError(err)) } } }
The pattern is:
- Fields are private with getter methods
- A
new()constructor - A
Fromimpl that chains through the category enum toConsensusError - The full derive stack including
PlatformSerializeandPlatformDeserialize - The
#[platform_serialize(unversioned)]attribute - The ordering warning comment
The test-only variant
You may have noticed this in ConsensusError:
#![allow(unused)] fn main() { #[cfg(test)] #[cfg_attr(test, error(transparent))] TestConsensusError(TestConsensusError), }
This variant only exists in test builds. It allows tests to create synthetic consensus errors without depending on real validation logic. The #[cfg(test)] attribute ensures it is completely stripped from production builds and does not affect the binary encoding of the other variants.
Rules
Do:
- Always add new variants at the end of the enum
- Always add new fields at the end of the struct
- Always include the ordering warning comment in new error types
- Always implement
From<YourError> for ConsensusErrorthrough the appropriate category - Always derive the full stack:
Error, Debug, Clone, PartialEq, Encode, Decode, PlatformSerialize, PlatformDeserialize - Use
#[platform_serialize(unversioned)]on leaf error structs - Keep fields private with getter methods
Do not:
- Reorder existing variants or fields -- this breaks network serialization
- Remove existing variants -- old nodes might still produce them
- Change the
#[error(...)]message without considering client-side parsing - Add large fields to error structs -- the 2000-byte limit on
ConsensusErrorapplies to the entire serialized tree - Use
#[platform_serialize(limit = ...)]on leaf structs -- the limit is enforced at theConsensusErrorlevel
Error Codes
In the previous chapter we saw how ConsensusError organizes errors into a tree of enums and structs. But when a client -- whether a JavaScript SDK, a mobile app, or a third-party tool -- receives a validation failure from the platform, it does not receive a Rust enum. It receives bytes on a wire. To make those bytes useful, every consensus error maps to a stable numeric code through the ErrorWithCode trait.
These codes are the public API of the error system. They appear in gRPC responses, they are documented for client developers, and they must never change once assigned. A client that handles error code 40100 ("document already present") today must still be able to handle that same code a year from now, even after dozens of protocol upgrades.
The ErrorWithCode trait
The trait lives in packages/rs-dpp/src/errors/consensus/codes.rs and is as simple as it gets:
#![allow(unused)] fn main() { pub trait ErrorWithCode { /// Returns the error code fn code(&self) -> u32; } }
One method, one number, no room for ambiguity. The trait is implemented for ConsensusError and for each of the four category enums.
How ConsensusError delegates
The top-level implementation just forwards to the appropriate category:
#![allow(unused)] fn main() { impl ErrorWithCode for ConsensusError { fn code(&self) -> u32 { match self { Self::BasicError(e) => e.code(), Self::SignatureError(e) => e.code(), Self::StateError(e) => e.code(), Self::FeeError(e) => e.code(), #[cfg(test)] ConsensusError::TestConsensusError(_) => 1000, ConsensusError::DefaultError => 1, // this should never happen } } } }
The DefaultError variant returns code 1 -- a sentinel value that should never appear in practice. The TestConsensusError returns 1000, safely outside any production range.
The code ranges
Error codes are organized into ranges that correspond to error categories and subcategories. Here is the complete map as it stands in the codebase:
BasicError codes (10000-10949)
| Range | Category | Examples |
|---|---|---|
| 10000-10099 | Versioning | UnsupportedVersionError (10000), ProtocolVersionParsingError (10001), IncompatibleProtocolVersionError (10004) |
| 10100-10199 | Structure | JsonSchemaCompilationError (10100), InvalidIdentifierError (10102), ValueError (10103) |
| 10200-10277 | Data Contract | DataContractMaxDepthExceedError (10200), DuplicateIndexError (10201), InvalidDataContractIdError (10204), DataContractInvalidRequiredFieldsUpdateError (10276), PreProgrammedDistributionAmountOverLimitError (10277) |
| 10350-10359 | Groups | GroupPositionDoesNotExistError (10350), GroupExceedsMaxMembersError (10354) |
| 10360-10367 | Contract Groups | ContractGroupMembershipsOverLimitError (10360), InvalidContractGroupAdminsError (10364), InvalidContractGroupDescriptionLengthError (10367); 10365 unassigned |
| 10400-10424 | Documents | DataContractNotPresentError (10400), DuplicateDocumentTransitionsWithIdsError (10401), DocumentPropertyNotDistinctError (10419), InvalidEncryptedPropertyShapeError (10420), DocumentPropertyMaxBytesExceededError (10421), DocumentPropertyConstraintViolatedError (10422), DocumentReferencePreimageInvalidError (10423), DocumentPropertyNotGeneratedError (10424) |
| 10450-10460 | Tokens | InvalidTokenIdError (10450), TokenTransferToOurselfError (10456) |
| 10500-10535 | Identity | DuplicatedIdentityPublicKeyBasicError (10500), InvalidIdentityPublicKeyDataError (10511) |
| 10600-10603 | State Transition | InvalidStateTransitionTypeError (10600), StateTransitionMaxSizeExceededError (10602) |
| 10700-10700 | General | OverflowError (10700) |
| 10800-10818 | Address | TransitionOverMaxInputsError (10800), WithdrawalBelowMinAmountError (10818) |
| 10819-10827 | Shielded | ShieldedNoActionsError (10819), ShieldedTooManyActionsError (10825), ShieldedImplicitFeeCapExceededError (10826), ShieldedInvalidDenominationError (10827 — IdentityCreateFromShieldedPool exit amount not a member of the versioned denomination set) |
| 10900-10949 | Contract Moderation | InvalidContractModerationConfigError (10900), ContractModerationSelfTargetError (10901), DocumentActionFeesWithoutModerationError (10902), ContractModerationReasonTooLongError (10903), InvalidContractModerationReasonDocumentsError (10904), InvalidContractModerationDocumentFieldsError (10905) |
SignatureError codes (20000-20012)
#![allow(unused)] fn main() { impl ErrorWithCode for SignatureError { fn code(&self) -> u32 { match self { Self::IdentityNotFoundError { .. } => 20000, Self::InvalidIdentityPublicKeyTypeError { .. } => 20001, Self::InvalidStateTransitionSignatureError { .. } => 20002, Self::MissingPublicKeyError { .. } => 20003, Self::InvalidSignaturePublicKeySecurityLevelError { .. } => 20004, Self::WrongPublicKeyPurposeError { .. } => 20005, Self::PublicKeyIsDisabledError { .. } => 20006, Self::PublicKeySecurityLevelNotMetError { .. } => 20007, Self::SignatureShouldNotBePresentError(_) => 20008, Self::BasicECDSAError(_) => 20009, Self::BasicBLSError(_) => 20010, Self::InvalidSignaturePublicKeyPurposeError(_) => 20011, Self::UncompressedPublicKeyNotAllowedError(_) => 20012, } } } }
Signature errors occupy the 20000 range. There are only 13 of them -- cryptographic validation is relatively binary (the signature is valid or it is not), so there are fewer failure modes to distinguish.
FeeError codes (30000)
#![allow(unused)] fn main() { impl ErrorWithCode for FeeError { fn code(&self) -> u32 { match self { Self::BalanceIsNotEnoughError { .. } => 30000, } } } }
The fee category currently has a single code. The 30000 range is reserved for future fee-related errors.
StateError codes (40000-41299)
| Range | Category | Examples |
|---|---|---|
| 40000-40009 | Data Contract | DataContractAlreadyPresentError (40000), DataContractIsReadonlyError (40001), DataContractNotFoundError (40008) |
| 40100-40147 | Documents | DocumentAlreadyPresentError (40100), DocumentNotFoundError (40101), DuplicateUniqueIndexError (40105), DocumentActionFeeAgreementNotSetError (40132), DocumentActionFeeAgreementMismatchError (40133), DocumentActionFeeMultiplierNotToleratedError (40134), DocumentActionFeeModeratorsShareMismatchError (40139), DocumentExpiredError (40140), DocumentContestMaximumContendersReachedError (40141), ReferencedDocumentRequirementNotMetError (40142), ReferencedDocumentTypeNotModeratedError (40143), ReferencedDocumentTypeModeratedError (40144), ReferencedDocumentRemovedError (40145), ReferencedDocumentTypeIndexOnlyError (40146), DocumentDeleteConstraintViolatedError (40147) |
| 40200-40217 | Identity | IdentityAlreadyExistsError (40200), InvalidIdentityRevisionError (40203), IdentityInsufficientBalanceError (40210) |
| 40300-40307 | Voting | MasternodeNotFoundError (40300), MasternodeVoteAlreadyPresentError (40304), VoteChoiceNotAllowedForVotePollError (40307) |
| 40400-40401 | Prefunded Balances | PrefundedSpecializedBalanceInsufficientError (40400) |
| 40500-40502 | Data Triggers | DataTriggerConditionError (40500), DataTriggerExecutionError (40501) |
| 40600-40603 | Addresses | AddressDoesNotExistError (40600), AddressNotEnoughFundsError (40601) |
| 40700-40721 | Tokens | IdentityDoesNotHaveEnoughTokenBalanceError (40700), UnauthorizedTokenActionError (40701) |
| 40800-40804 | Groups | IdentityNotMemberOfGroupError (40800), GroupActionAlreadyCompletedError (40802) |
| 40900-40904 | Shielded | InvalidAnchorError (40900), NullifierAlreadySpentError (40901), InsufficientShieldedFeeError (40904) |
| 41000-41003 | Contract Groups | ContractGroupAlreadyExistsError (41000), ContractGroupNotFoundError (41001), IdentityNotContractGroupOwnerOrAdminError (41002), ContractGroupAdminNotFoundError (41003) |
| 41100-41124 | Contract Moderation | ContractModerationNotEnabledError (41100), IdentityNotContractModeratorError (41101), ContractUserBannedError (41107), ContractUserSuspendedError (41108), ContractModerationTargetNotFoundError (41109), ContractModeratorIdentityNotFoundError (41110), ContractFeesAlreadyClaimedThisEpochError (41111), ContractFeesNothingToClaimError (41112), ContractFeeClaimNotAllowedError (41113), ContractModerationCounterpartyBarredError (41114), DocumentTypeNotDeletableByModeratorsError (41115), DocumentModerationWindowElapsedError (41116), ContractUserNotWarnedError (41117), ContractUserWarningLimitReachedError (41118), ContractDocumentRemovalNotFoundError (41119), DocumentRestoreWindowElapsedError (41120), DocumentRestoreHashMismatchError (41121), ContractDocumentAlreadyRestoredError (41122), DocumentFieldNotChangeableByModeratorsError (41123), DocumentModeratorFieldNotWritableError (41124) |
| 41200-41299 | Contract Moderation Teams | ContractModeratedDocumentTypeNotYetUsableError (41200), ContractModerationAbilityNotGrantedError (41201), ModerationCharterAddedModeratorLimitReachedError (41202), ModerationReasonNotListedError (41203), DocumentTypeNotDeletableOnceSettledError (41204), ContractModerationTeamNotSeatedError (41205), DocumentNotSettledError (41206), ContractTeamActionDoesNotExistError (41207), ContractTeamActionAlreadySignedError (41208), SettledDeletionNotRestorableError (41209), ContractTeamActionAlreadyCompletedError (41210), ContractTeamActionDocumentChangedError (41211), ContractTeamMemberAddedAfterDocumentError (41212) |
Notice how the DataTriggerError sub-enum has its own ErrorWithCode implementation that the StateError delegates to:
#![allow(unused)] fn main() { impl ErrorWithCode for StateError { fn code(&self) -> u32 { match self { // ... // Data trigger errors: 40500-40699 Self::DataTriggerError(ref e) => e.code(), // ... } } } impl ErrorWithCode for DataTriggerError { fn code(&self) -> u32 { match self { Self::DataTriggerConditionError { .. } => 40500, Self::DataTriggerExecutionError { .. } => 40501, Self::DataTriggerInvalidResultError { .. } => 40502, } } } }
This is the only case where StateError delegates to a sub-enum rather than mapping directly. All other state error variants map to their codes inline.
The pattern: one variant, one code, one match arm
Look at how codes are assigned inside BasicError:
#![allow(unused)] fn main() { impl ErrorWithCode for BasicError { fn code(&self) -> u32 { match self { // Versioning Errors: 10000-10099 Self::UnsupportedVersionError(_) => 10000, Self::ProtocolVersionParsingError { .. } => 10001, Self::SerializedObjectParsingError { .. } => 10002, Self::UnsupportedProtocolVersionError(_) => 10003, Self::IncompatibleProtocolVersionError(_) => 10004, Self::VersionError(_) => 10005, Self::UnsupportedFeatureError(_) => 10006, // Structure Errors: 10100-10199 Self::JsonSchemaCompilationError(..) => 10100, Self::JsonSchemaError(_) => 10101, // ... } } } }
Every single variant gets its own match arm and its own code. There is no default case, no wildcard, no range-based mapping. This is deliberate -- if you add a new variant to the enum and forget to add a code for it, the Rust compiler will refuse to compile because the match is non-exhaustive. The type system enforces completeness.
Nested ContractError patterns
One interesting detail is how BasicError handles DataContractError variants. The ContractError wrapper contains a nested enum, so the code mapping matches through two levels:
#![allow(unused)] fn main() { Self::ContractError(DataContractError::DocumentTypesAreMissingError { .. }) => 10214, Self::ContractError(DataContractError::DecodingContractError { .. }) => 10222, Self::ContractError(DataContractError::DecodingDocumentError { .. }) => 10223, Self::ContractError(DataContractError::InvalidDocumentTypeError { .. }) => 10224, Self::ContractError(DataContractError::MissingRequiredKey(_)) => 10225, Self::ContractError(DataContractError::FieldRequirementUnmet(_)) => 10226, // ... many more }
Each specific DataContractError variant gets its own distinct code. This ensures that clients can distinguish between "the contract is missing a required key" (10225) and "the contract has a wrong type for a value" (10228) even though both come wrapped in BasicError::ContractError.
Adding a new error code
Here is the procedure for adding a new consensus error:
-
Choose the category. Is it a structural validation issue (BasicError), a state conflict (StateError), a signature problem (SignatureError), or a fee issue (FeeError)?
-
Choose the subcategory range. Look at the existing ranges within that category. If your error is about tokens in
StateError, you would use the 40700-40799 range. -
Pick the next available code. Within the range, find the highest existing code and add 1. Never reuse a code, even if the original error was removed.
-
Add the variant to the appropriate enum (at the end -- remember the ordering rule).
-
Add the match arm to the
ErrorWithCodeimplementation. -
Document the code so client developers know what it means.
Here is what the diff would look like for adding a hypothetical new token state error:
#![allow(unused)] fn main() { // In state_error.rs -- add at the end of the enum: #[error(transparent)] TokenNewError(TokenNewError), // In codes.rs -- add to the StateError match: Self::TokenNewError(_) => 40722, // next available in the 40700 range }
Why codes can never change
Consider what happens if code 40100 is changed from DocumentAlreadyPresentError to something else:
- A client SDK has a handler for code 40100 that displays "This document already exists" to the user.
- After the change, the platform sends 40100 for a completely different error.
- The user sees "This document already exists" when actually their token balance is insufficient.
- Trust in the platform erodes.
Error codes are part of the platform's wire protocol. They are as immutable as protobuf field numbers or HTTP status codes. Once assigned, they live forever.
Rules
Do:
- Assign new codes at the end of the appropriate range
- Use the compiler's exhaustive match checking to ensure every variant has a code
- Keep codes within their designated range (do not put a document error in the identity range)
- Leave gaps between ranges for future expansion (this is already done)
Do not:
- Change an existing error code -- ever
- Reuse a code from a removed error variant
- Add a wildcard (
_) match arm toErrorWithCodeimplementations - Assign codes outside the established ranges without allocating a new range first
- Skip codes within a range (assign sequentially within each subcategory)
Drive Errors
The previous two chapters covered consensus errors -- the carefully serialized, code-stable errors that get sent across the network. Drive errors are a different beast entirely. They are internal errors that arise from the storage layer, the database, and the logic that sits between state transitions and GroveDB. They are not serialized as consensus errors, and they do not need stable numeric codes. The one place they reach a client is a query's gRPC status, described in Query errors on the wire.
But they do need to be well-organized, because Drive is where most of the platform's complexity lives. When something goes wrong in Drive, you need to know immediately whether it is a corrupted database, a protocol-level validation failure, a fee calculation error, or a bug in your own code.
Query errors on the wire
A gRPC query answers a failed request with a status code, and clients read that code to decide whose fault the failure is. rs-dapi-client takes INVALID_ARGUMENT as the caller's mistake: the error goes back to the caller, and nothing else happens. It takes UNKNOWN and INTERNAL as a fault of the node: the client stops using that node for a while (a ban) and sends the request to the next one. A request error answered as INTERNAL therefore gets every node banned in turn, since every node refuses the request the same way.
The mapping lives in packages/rs-drive-abci/src/query/service.rs:
| Error | Status |
|---|---|
QueryError::InvalidArgument, QueryError::TooManyElements | INVALID_ARGUMENT |
A QuerySyntaxError, whether the handler reports it (QueryError::Query, QueryError::Drive(Error::Query)) or returns it (Error::Drive(Error::Query)) | INVALID_ARGUMENT |
QueryError::NotFound | NOT_FOUND |
QueryError::ResourceExhausted | RESOURCE_EXHAUSTED |
| A request without a version, or with a version the node does not serve | UNKNOWN: a newer node may serve it |
| Any other error a handler returns | INTERNAL |
A status message is cut at 1024 bytes, because it travels in the grpc-message HTTP/2 header and may echo a request field of any length.
A handler checks the request before Drive sees it, and refuses what no node can answer with a validation error, never with ?. Two rules decide what it refuses:
- Refuse only what fails. A request that has an answer keeps it, even an empty one, so no caller that depends on it breaks.
- GroveDB cannot prove an empty query, or one with a limit of 0. When a proof is asked, an empty id list or a limit of 0 is refused; without a proof, the answer is empty.
| Request | Without a proof | With a proof |
|---|---|---|
| Empty id list: data contracts, identities balances, identities contract keys (ids or purposes), evonode blocks by ids, identity and identities token balances and infos, token statuses, addresses infos | Empty answer | INVALID_ARGUMENT |
| Identity keys with limit 0, epochs info with count 0 | Empty answer | INVALID_ARGUMENT |
| Limit or count 0 on contract history (or above 10), protocol upgrade vote status, evonode blocks by range, identity votes, pre-programmed distributions, group infos, group actions | INVALID_ARGUMENT | INVALID_ARGUMENT |
| Identity keys search without a limit | INVALID_ARGUMENT | Served |
| Specific identity keys with a limit below their number of ids | The first keys that exist, in id order | The same keys, proved |
| Documents v0 with limit 0, shielded encrypted notes with count 0 | 0 means the default | 0 means the default |
An empty identities balances request with a proof used to get a proof the client could not verify. A proof that fails to verify is treated as a node fault too, so refusing that request matters as much as the status codes.
Nodes released before these rules answer the same requests with UNKNOWN, INTERNAL, or that unverifiable proof.
The Drive Error enum
The top-level error type lives in packages/rs-drive/src/error/mod.rs:
#![allow(unused)] fn main() { /// Errors #[derive(Debug, thiserror::Error)] pub enum Error { /// Query error #[error("query: {0}")] Query(#[from] QuerySyntaxError), /// Storage Flags error #[error("storage flags: {0}")] StorageFlags(#[from] StorageFlagsError), /// Drive error #[error("drive: {0}")] Drive(#[from] DriveError), /// Proof error #[error("proof: {0}")] Proof(#[from] ProofError), /// GroveDB error #[error("grovedb: {0}")] GroveDB(Box<grovedb::Error>), /// Protocol error #[error("protocol: {0}")] Protocol(Box<ProtocolError>), /// Identity error #[error("identity: {0}")] Identity(#[from] IdentityError), /// Fee error #[error("fee: {0}")] Fee(#[from] FeeError), /// Document error #[error("document: {0}")] Document(#[from] DocumentError), /// Value error #[error("value: {0}")] Value(#[from] ValueError), /// DataContract error #[error("contract: {0}")] DataContract(#[from] DataContractError), /// Cache error #[error("contract: {0}")] Cache(#[from] CacheError), /// Protocol error with info string #[error("protocol: {0} ({1})")] ProtocolWithInfoString(Box<ProtocolError>, String), /// IO error with info string #[error("io: {0} ({1})")] IOErrorWithInfoString(Box<io::Error>, String), } }
This enum is a classic Rust error aggregator. It collects errors from every subsystem that Drive interacts with -- the query parser, the storage flags system, GroveDB, the protocol layer, identities, fees, documents, data contracts, and the cache. Each variant wraps a specific error type from that subsystem.
Notice the organizational difference from ConsensusError:
ConsensusErroris organized by validation phase (basic, state, signature, fee)- Drive's
Erroris organized by subsystem (query, storage, grovedb, protocol, identity, etc.)
This makes sense. Consensus errors are about "what rule was violated." Drive errors are about "what component failed."
Box<ProtocolError> -- avoiding large enum variants
Two variants wrap their inner error in a Box:
#![allow(unused)] fn main() { /// GroveDB error #[error("grovedb: {0}")] GroveDB(Box<grovedb::Error>), /// Protocol error #[error("protocol: {0}")] Protocol(Box<ProtocolError>), }
Why? Because ProtocolError and grovedb::Error are large types. Without the Box, the entire Error enum would be as large as its largest variant, which could be hundreds of bytes. Since most Drive operations return Result<T, Error>, you would be paying that size cost on every Ok path too -- the Result itself is as large as the larger of T and Error.
Boxing the large variants means the Error enum stores only an 8-byte pointer for those cases, keeping the overall size reasonable. This is a common Rust pattern, and Clippy will warn you about it via the clippy::large_enum_variant lint.
The From trait chain
The #[from] attribute on most variants is a thiserror feature that automatically generates From implementations. For example, #[from] DriveError generates:
#![allow(unused)] fn main() { impl From<DriveError> for Error { fn from(value: DriveError) -> Self { Self::Drive(value) } } }
But look at the manually written From implementations at the bottom of the file:
#![allow(unused)] fn main() { impl From<ProtocolError> for Error { fn from(value: ProtocolError) -> Self { Self::Protocol(Box::new(value)) } } impl From<grovedb::Error> for Error { fn from(value: grovedb::Error) -> Self { Self::GroveDB(Box::new(value)) } } impl From<grovedb::element::error::ElementError> for Error { fn from(value: grovedb::element::error::ElementError) -> Self { Self::GroveDB(Box::new(grovedb::Error::ElementError(value))) } } impl From<ProtocolDataContractError> for Error { fn from(value: ProtocolDataContractError) -> Self { Self::Protocol(Box::new(ProtocolError::DataContractError(value))) } } }
These cannot use #[from] because they involve boxing or wrapping through an intermediate type. The ProtocolError conversion boxes the value. The grovedb::element::error::ElementError conversion wraps the error inside grovedb::Error::ElementError first, then boxes it. The ProtocolDataContractError conversion wraps through ProtocolError::DataContractError and then boxes.
These manual From implementations create multi-hop error conversion chains. When a function deep in Drive's GroveDB interaction layer returns a grovedb::element::error::ElementError, the ? operator can propagate it all the way up to a drive::Error in a single step, with the conversion chain handling the wrapping automatically.
The DriveError enum -- internal errors
The DriveError enum in packages/rs-drive/src/error/drive.rs is where Drive reports its own internal problems:
#![allow(unused)] fn main() { /// Drive errors #[derive(Debug, thiserror::Error)] pub enum DriveError { /// This error should never occur, it is the equivalent of a panic. #[error("corrupted code execution error: {0}")] CorruptedCodeExecution(&'static str), /// Platform expected some specific versions #[error("drive unknown version on {method}, received: {received}")] UnknownVersionMismatch { method: String, known_versions: Vec<FeatureVersion>, received: FeatureVersion, }, /// A critical corrupted state should stall the chain. #[error("critical corrupted state error: {0}")] CriticalCorruptedState(&'static str), /// Error #[error("not supported error: {0}")] NotSupported(&'static str), // ... many more variants } }
Notice how DriveError uses &'static str for most of its error messages rather than String. This is deliberate -- these are internal error messages that should be known at compile time. A CorruptedCodeExecution("document tree missing expected root") is a fixed message that describes a specific bug or data corruption scenario. Using &'static str makes it clear that these are not user-facing messages and avoids allocations on error paths.
There are a few variants that do use String -- these are cases where the error message needs to include runtime data:
#![allow(unused)] fn main() { #[error("corrupted contract indexes error: {0}")] CorruptedContractIndexes(String), #[error("corrupted drive state error: {0}")] CorruptedDriveState(String), }
The severity spectrum
DriveError variants implicitly encode severity through naming conventions:
-
Corrupted*variants indicate data corruption. The database is in an unexpected state. These are serious problems that may indicate bugs or hardware failures:#![allow(unused)] fn main() { CorruptedCodeExecution(&'static str), CriticalCorruptedState(&'static str), CorruptedContractPath(&'static str), CorruptedDocumentPath(&'static str), CorruptedBalancePath(&'static str), CorruptedSerialization(String), CorruptedElementType(&'static str), CorruptedDriveState(String), } -
Invalid*andNotSupportedvariants indicate logic errors -- the code is trying to do something that should not be possible:#![allow(unused)] fn main() { InvalidDeletionOfDocumentThatKeepsHistory(&'static str), InvalidContractHistoryFetchLimit(u16), NotSupported(&'static str), } -
*NotFoundand*DoesNotExistvariants indicate missing data that was expected:#![allow(unused)] fn main() { DataContractNotFound(String), ElementNotFound(&'static str), PrefundedSpecializedBalanceDoesNotExist(String), } -
Version mismatch variants indicate protocol version problems:
#![allow(unused)] fn main() { UnknownVersionMismatch { method, known_versions, received }, VersionNotActive { method, known_versions }, }
When to use consensus errors vs drive errors
This is the key design decision you face when adding error handling to Drive code:
Use a consensus error when:
- The error is caused by invalid user input (a malformed state transition)
- The error needs to be communicated back to the client
- The error needs a stable numeric code
- Other nodes must produce the same error for the same input
Use a drive error when:
- The error is caused by internal state (database corruption, missing paths)
- The error indicates a bug in the platform code
- The error is about version mismatches or unsupported features
- The error involves the storage layer (GroveDB problems)
In practice, Drive functions often work with both. A typical pattern is a function that validates input (producing consensus errors) and then performs storage operations (which might produce drive errors). The return type is usually Result<T, Error> where Error is the Drive error, and consensus errors are embedded inside ProtocolError which is itself a variant of the Drive error.
The ? propagation pattern
Here is how error propagation typically works in Drive code:
#![allow(unused)] fn main() { fn apply_document_create( &self, document: &Document, contract: &DataContract, // ... ) -> Result<(), Error> { // This might return a grovedb::Error, which auto-converts via From let existing = self.grove_get(path, key, transaction)?; // This might return a DriveError if existing.is_some() { return Err(DriveError::CorruptedDocumentAlreadyExists( "document should not exist at this point" ).into()); } // This might return a ProtocolError, which auto-converts via From + Box let serialized = document.serialize(platform_version)?; // This might return a grovedb::Error self.grove_insert(path, key, element, transaction)?; Ok(()) } }
The ? operator handles all the conversions transparently. A grovedb::Error becomes Error::GroveDB(Box::new(...)). A DriveError becomes Error::Drive(...). A ProtocolError becomes Error::Protocol(Box::new(...)). The developer does not need to think about which From implementation is being invoked -- the type system figures it out.
Sub-module error types
Drive organizes its errors into submodules, each with its own error enum:
#![allow(unused)] fn main() { pub mod cache; // CacheError pub mod contract; // DataContractError (Drive's own, distinct from DPP's) pub mod document; // DocumentError pub mod drive; // DriveError pub mod fee; // FeeError (Drive's own, distinct from consensus FeeError) pub mod identity; // IdentityError pub mod proof; // ProofError pub mod query; // QuerySyntaxError }
Each of these has a #[from] conversion to the top-level Error, creating a clean hierarchy where subsystem-specific code can work with its own error type and callers can propagate through ? to the unified type.
Rules
Do:
- Use
Boxfor large error types (ProtocolError, grovedb::Error) to keep the enum small - Use
&'static strfor internal error messages that are known at compile time - Use
Stringonly when the message needs runtime data - Let
#[from]generateFromimplementations where possible - Write manual
Fromimplementations when boxing or intermediate wrapping is needed - Follow the naming conventions:
Corrupted*for data corruption,Invalid*for logic errors,*NotFoundfor missing data
Do not:
- Use consensus errors for internal Drive problems
- Use drive errors for user-facing validation failures
- Forget to add a
Fromimplementation when introducing a new error type - Return a
Stringerror message when a typed error variant would be more informative - Panic in Drive code -- return a
CorruptedCodeExecutionorCriticalCorruptedStateerror instead
Platform Serialization
Dash Platform needs to serialize data structures -- state transitions, consensus errors, documents, identities -- into bytes and back. You might ask: why not just use bincode directly? Or serde? Or protobuf? The answer is that Platform has requirements that no off-the-shelf serialization library handles on its own:
-
Version awareness. The serialization of a type may differ between protocol versions. A
DataContractserialized under protocol version 3 might have different fields than under version 4. The serialization system needs to accept aPlatformVersionparameter and dispatch accordingly. -
Size limits. A malicious actor should not be able to send a 100 MB state transition that costs the node minutes to decode. Serialization must enforce configurable byte limits.
-
Determinism. Every node must produce identical bytes for identical data. This rules out serialization formats that allow field reordering (like JSON) or that depend on hash map iteration order.
-
Compatibility with bincode. Platform already uses bincode extensively for its compact binary format and deterministic encoding. The custom layer should wrap bincode, not replace it.
The rs-platform-serialization crate provides this layer. It sits between bincode and the rest of the platform, adding version-aware encoding/decoding while delegating the actual byte-level work to bincode.
The core traits
The crate defines two pairs of traits in packages/rs-platform-serialization/src/enc/mod.rs and packages/rs-platform-serialization/src/de/mod.rs.
Encoding: PlatformVersionEncode
#![allow(unused)] fn main() { pub trait PlatformVersionEncode { /// Encode a given type. fn platform_encode<E: Encoder>( &self, encoder: &mut E, platform_version: &PlatformVersion, ) -> Result<(), EncodeError>; } }
Compare this to bincode's standard Encode trait:
#![allow(unused)] fn main() { // bincode's Encode (for reference) pub trait Encode { fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError>; } }
The only difference is the platform_version parameter. For types whose serialization does not change between versions, the implementation simply ignores it and delegates to bincode's Encode:
#![allow(unused)] fn main() { impl PlatformVersionEncode for String { fn platform_encode<E: Encoder>( &self, encoder: &mut E, _: &PlatformVersion, // ignored -- String encoding never changes ) -> Result<(), EncodeError> { Encode::encode(self, encoder) } } }
For types that do change between versions, the implementation can dispatch to different encoding logic based on the version.
Decoding: PlatformVersionedDecode
#![allow(unused)] fn main() { pub trait PlatformVersionedDecode: Sized { fn platform_versioned_decode<D: Decoder<Context = crate::BincodeContext>>( decoder: &mut D, platform_version: &PlatformVersion, ) -> Result<Self, DecodeError>; } }
Again, this mirrors bincode's Decode but with the version parameter. There is also a borrowed variant for zero-copy decoding:
#![allow(unused)] fn main() { pub trait PlatformVersionedBorrowDecode<'de>: Sized { fn platform_versioned_borrow_decode< D: BorrowDecoder<'de, Context = crate::BincodeContext>, >( decoder: &mut D, platform_version: &PlatformVersion, ) -> Result<Self, DecodeError>; } }
And a convenience macro to implement PlatformVersionedBorrowDecode for any type that implements PlatformVersionedDecode:
#![allow(unused)] fn main() { #[macro_export] macro_rules! impl_platform_versioned_borrow_decode { ($ty:ty) => { impl<'de> $crate::PlatformVersionedBorrowDecode<'de> for $ty { fn platform_versioned_borrow_decode< D: bincode::de::BorrowDecoder<'de, Context = $crate::BincodeContext>, >( decoder: &mut D, platform_version: &PlatformVersion, ) -> core::result::Result<Self, bincode::error::DecodeError> { <$ty as $crate::PlatformVersionedDecode>::platform_versioned_decode( decoder, platform_version, ) } } }; } }
The convenience functions
The crate provides top-level functions that mirror bincode's API but add version support. The most commonly used one is platform_encode_to_vec in packages/rs-platform-serialization/src/features/impl_alloc.rs:
#![allow(unused)] fn main() { pub fn platform_encode_to_vec<E: PlatformVersionEncode, C: Config>( val: E, config: C, platform_version: &PlatformVersion, ) -> Result<Vec<u8>, EncodeError> { let size = { let mut size_writer = enc::EncoderImpl::<_, C>::new(SizeWriter::default(), config); val.platform_encode(&mut size_writer, platform_version)?; size_writer.into_writer().bytes_written }; let writer = VecWriter::with_capacity(size); let mut encoder = enc::EncoderImpl::<_, C>::new(writer, config); val.platform_encode(&mut encoder, platform_version)?; Ok(encoder.into_writer().inner) } }
This function does something clever: it encodes twice. The first pass uses a SizeWriter that counts bytes without allocating. The second pass uses a VecWriter pre-allocated to exactly the right size. This avoids reallocations during encoding.
For decoding:
#![allow(unused)] fn main() { pub fn platform_versioned_decode_from_slice<D: PlatformVersionedDecode, C: Config>( src: &[u8], config: C, platform_version: &PlatformVersion, ) -> Result<D, error::DecodeError> { let reader = read::SliceReader::new(src); let mut decoder = DecoderImpl::<_, C, crate::BincodeContext>::new(reader, config, ()); D::platform_versioned_decode(&mut decoder, platform_version) } }
There are also platform_encode_into_slice for encoding into a pre-allocated buffer, encode_into_writer for encoding into arbitrary writers, and platform_versioned_decode_from_reader for decoding from arbitrary readers.
How it wraps bincode
The wrapping is lightweight. Platform serialization does not add a version prefix to the bytes by default. It does not change the wire format. When you call platform_encode_to_vec, the resulting bytes are identical to what bincode::encode_to_vec would produce -- the difference is that the encoding logic can vary based on the PlatformVersion parameter.
The bincode configuration used throughout the platform is:
#![allow(unused)] fn main() { let config = bincode::config::standard() .with_big_endian() // network byte order for determinism .with_no_limit(); // or .with_limit::<{ N }>() for size-limited encoding }
Big endian is used for deterministic cross-platform encoding. The limit is applied at the PlatformSerialize / PlatformDeserialize level (see the derive macros chapter) rather than at the raw bincode level.
Standard type implementations
The crate provides PlatformVersionEncode and PlatformVersionedDecode implementations for all standard Rust types. These live in packages/rs-platform-serialization/src/features/impl_alloc.rs and the other impl files. Here are the key patterns:
Types that ignore the version
Primitive types, strings, and simple wrappers just delegate to bincode:
#![allow(unused)] fn main() { impl PlatformVersionedDecode for String { fn platform_versioned_decode<D: Decoder<Context = crate::BincodeContext>>( decoder: &mut D, _: &PlatformVersion, ) -> Result<Self, DecodeError> { bincode::Decode::decode(decoder) } } }
Collections that propagate the version
Collections like Vec, BTreeMap, and BTreeSet encode their length, then encode each element with the platform version:
#![allow(unused)] fn main() { impl<T> PlatformVersionEncode for Vec<T> where T: PlatformVersionEncode + 'static, { fn platform_encode<E: Encoder>( &self, encoder: &mut E, platform_version: &PlatformVersion, ) -> Result<(), EncodeError> { crate::enc::encode_slice_len(encoder, self.len())?; // Optimization: byte slices are written directly if core::any::TypeId::of::<T>() == core::any::TypeId::of::<u8>() { let slice: &[u8] = unsafe { core::mem::transmute(self.as_slice()) }; encoder.writer().write(slice)?; return Ok(()); } for item in self.iter() { item.platform_encode(encoder, platform_version)?; } Ok(()) } } }
Notice the Vec<u8> optimization -- byte vectors are written in bulk rather than element-by-element, which is significantly faster for large binary data.
Smart pointers
Box<T>, Rc<T>, Arc<T>, and Cow<T> all delegate to their inner type:
#![allow(unused)] fn main() { impl<T> PlatformVersionEncode for Box<T> where T: PlatformVersionEncode + ?Sized, { fn platform_encode<E: Encoder>( &self, encoder: &mut E, platform_version: &PlatformVersion, ) -> Result<(), EncodeError> { T::platform_encode(self, encoder, platform_version) } } }
Size limits and why they matter
Without size limits, a malicious state transition could contain a Vec claiming to have 2^64 elements, causing the node to allocate unbounded memory during decoding. Or a deeply nested document structure could produce a multi-megabyte serialized form that consumes excessive bandwidth.
Size limits are enforced at two levels:
-
At the
PlatformSerializetrait level -- the#[platform_serialize(limit = N)]attribute on a type causes the derive macro to usebincode::config::standard().with_big_endian().with_limit::<{ N }>(). If encoding exceeds N bytes, it returns aMaxEncodedBytesReachedError. -
At the bincode decoder level -- bincode's
claim_container_readmechanism prevents allocating excessive memory for containers:
#![allow(unused)] fn main() { fn platform_versioned_decode<D: Decoder<Context = crate::BincodeContext>>( decoder: &mut D, platform_versioned: &PlatformVersion, ) -> Result<Self, DecodeError> { let len = crate::de::decode_slice_len(decoder)?; decoder.claim_container_read::<T>(len)?; // checks memory budget let mut vec = Vec::with_capacity(len); for _ in 0..len { decoder.unclaim_bytes_read(core::mem::size_of::<T>()); vec.push(T::platform_versioned_decode(decoder, platform_version)?); } Ok(vec) } }
The claim_container_read call tells the decoder "I am about to read len elements of type T" and the decoder checks whether this fits within the remaining byte budget. If not, it returns an error before any allocation happens.
The BincodeContext type alias
You will see crate::BincodeContext throughout the code:
#![allow(unused)] fn main() { pub type BincodeContext = (); }
This is the bincode context type. Bincode supports contextual decoding where a context value is threaded through the decoder -- useful for things like string interning or reference resolution. Platform does not use this feature, so the context is the unit type (). The alias exists to make it easy to change later if needed.
Rules
Do:
- Use
PlatformVersionEncode/PlatformVersionedDecodefor types whose serialization may change between protocol versions - Use standard bincode
Encode/Decodefor types whose format is permanently fixed - Use big-endian configuration everywhere for deterministic encoding
- Apply size limits on types that are received from untrusted sources (state transitions, consensus errors)
- Use the
platform_encode_to_vecconvenience function rather than constructing encoders manually
Do not:
- Use little-endian or native-endian configuration -- this breaks determinism across architectures
- Forget to propagate the
platform_versionparameter through collection types - Skip
claim_container_readin custom collection decoders -- this is a denial-of-service defense - Add a version prefix to the bytes manually -- the platform version is a parameter, not part of the wire format
- Use serde for consensus-critical serialization -- serde's flexibility (field names, human-readable formats) is a liability when you need deterministic bytes
Document Serialization
This chapter describes the binary wire format used to serialize and deserialize documents on Dash Platform. If you are building a tool that reads documents from gRPC responses, GroveDB storage, or any other source of raw document bytes, this is the specification you need.
Documents are not self-describing. The binary format does not include field names or type tags. To interpret the bytes, you must know which data contract and document type the document belongs to. The document type's schema defines the field order, types, and sizes that determine how the bytes map to values.
High-level structure
Every serialized document follows this layout:
┌──────────────────────┐
│ Serialization │ varint (1-2 bytes)
│ Version │ Currently: 0, 1, 2, or 3
├──────────────────────┤
│ Contract version │ V3 only: varint
│ stamp │ (0 = unstamped)
├──────────────────────┤
│ $id │ 32 bytes
├──────────────────────┤
│ $ownerId │ 32 bytes
├──────────────────────┤
│ $creatorId │ V2 only, if document type supports
│ (optional) │ transfers/trading: 1 byte flag + 32 bytes
├──────────────────────┤
│ $revision │ varint (if document type is mutable)
├──────────────────────┤
│ Time fields │ 2-byte bitfield + variable-length data
│ bitfield + data │
├──────────────────────┤
│ Price │ Only if document type supports trading
│ (optional) │ 1 byte flag + 8 bytes
├──────────────────────┤
│ User properties │ Variable length, schema-dependent
│ (in schema order) │
└──────────────────────┘
Source: packages/rs-dpp/src/document/v0/serialize.rs
Serialization versions
The first bytes of a serialized document are a varint encoding the serialization version. This is not the protocol version or the document struct version — it is the version of the serialization format itself.
| Serialization Version | Description |
|---|---|
| 0 | Original format. All integers encoded as i64 (8 bytes big-endian) regardless of their schema type. |
| 1 | Integers encoded at their native size (u8 = 1 byte, u16 = 2 bytes, u32 = 4 bytes, etc.). Otherwise identical to v0. |
| 2 | Same as v1, but adds $creatorId field after $ownerId for document types that support transfers or trading. |
| 3 | Same as v2, but adds a contract version stamp varint immediately after the version varint (protocol v14+). The stamp selects each property's layout when the document type carries requiredSince annotations — see below. |
The varint encoding uses the integer-encoding crate's VarInt format. For values 0 through 3, the varint is a single byte: 0x00, 0x01, 0x02, or 0x03.
#![allow(unused)] fn main() { // Serialization version is written first let mut buffer: Vec<u8> = 2u64.encode_var_vec(); // version 2 → byte 0x02 }
When deserializing, the version varint is read first, then the appropriate deserialization logic is dispatched:
#![allow(unused)] fn main() { let serialized_version: u64 = serialized_document.read_varint()?; match serialized_version { 0 => DocumentV0::from_bytes_v0(serialized_document, document_type, platform_version), 1 => DocumentV0::from_bytes_v1(serialized_document, document_type, platform_version), 2 => DocumentV0::from_bytes_v2(serialized_document, document_type, platform_version), 3 => DocumentV0::from_bytes_v3(serialized_document, document_type, platform_version), _ => Err(/* unknown version */), } }
Note: version 0 has a fallback — if deserialization as v0 (all i64) fails, it retries as v1 (native integer types). This handles edge cases from protocol versions 1–8 where the version byte was 0 but non-i64 integer types may have been used.
The contract version stamp (v3)
Serialization version 3 (the default from protocol v14) writes one extra varint immediately after the version varint: the contract version stamp — the version of the data contract the document's bytes conform to. A value of 0 means unstamped: the document was originally serialized before format 3 existed and has merely been rewritten in the new envelope (for example by a transfer).
The stamp exists because contract updates may add new required properties from a specific contract version onward, using the requiredSince schema keyword:
"properties": {
"newField": { "type": "string", "maxLength": 63, "position": 4, "requiredSince": 3 }
},
"required": ["existingField", "newField"]
Requiredness is baked into the wire format — a required property serializes raw while an optional one carries a presence flag — so a property whose requiredness varies by contract version needs the stamp to resolve its layout. The rule, per property:
A property is encoded as required (no presence flag) if it is listed in
requiredand either it has norequiredSinceannotation, or the document's stamp is at or above the annotation. Otherwise it is encoded as optional (presence-flagged).
An unstamped document (0) predates every requiredSince annotation, so only unconditionally required properties count as required for it. This means the latest contract alone reconstructs the byte layout of every document ever stored — no historical contract lookups are needed.
The stamp is platform-assigned: Drive sets it to the current contract version whenever document content is supplied (create and replace), and preserves it untouched through server-side rewrites that do not re-supply content (transfer and purchase). A document created before a contract update therefore keeps its old stamp — and may legitimately omit properties the newest schema requires — until a replace re-supplies its content and re-stamps it. Clients can also use the stamp as a staleness signal: a document stamped above the client's cached contract version means the contract needs refetching.
Formats 0–2 have no stamp; documents read from them deserialize with contract_version = None, equivalent to a 0 stamp.
Field-by-field breakdown
$id (32 bytes)
The document's unique identifier, written as raw bytes. This is a 256-bit value derived from the contract ID, owner ID, document type name, entropy and (protocol v14+) the identity contract nonce of the create transition via double SHA-256.
$ownerId (32 bytes)
The identity that currently owns the document, written as raw bytes.
$creatorId (v2 and v3, conditional)
Present only in serialization versions 2 and 3, and only if the document type supports transfers (documents_transferable) or trading (trade_mode != None).
0x01 [32 bytes creatorId] — creator ID present
0x00 — creator ID absent
$revision (varint, conditional)
Present only if the document type requires revisions (mutable documents). Encoded as a varint (u64). New documents start at revision 1 (INITIAL_REVISION).
Time fields (2-byte bitfield + data)
Time-related fields use a compact encoding with a bitfield to indicate which fields are present, followed by the data for each present field.
Bitfield (2 bytes, big-endian u16):
| Bit | Field |
|---|---|
| 0 (0x0001) | $createdAt |
| 1 (0x0002) | $updatedAt |
| 2 (0x0004) | $transferredAt |
| 3 (0x0008) | $createdAtBlockHeight |
| 4 (0x0010) | $updatedAtBlockHeight |
| 5 (0x0020) | $transferredAtBlockHeight |
| 6 (0x0040) | $createdAtCoreBlockHeight |
| 7 (0x0080) | $updatedAtCoreBlockHeight |
| 8 (0x0100) | $transferredAtCoreBlockHeight |
| 9 (0x0200) | $moderatedAt (version 3) |
| 10 (0x0400) | $moderatedBy (version 3) |
Data: For each bit that is set (in the order above), the corresponding value is appended:
$createdAt,$updatedAt,$transferredAt: 8 bytes big-endian u64 — milliseconds since Unix epoch$createdAtBlockHeight,$updatedAtBlockHeight,$transferredAtBlockHeight: 8 bytes big-endian u64 — platform block height$createdAtCoreBlockHeight,$updatedAtCoreBlockHeight,$transferredAtCoreBlockHeight: 4 bytes big-endian u32 — core chain block height$moderatedAt: 8 bytes big-endian u64, milliseconds since Unix epoch$moderatedBy: 32 bytes, the moderator's identity id
Bits 9 and 10 are read only in version 3, the format of protocol version 14. They are set only on a document a moderator has written the fields only moderators write of (see System Properties), so every other document's bitfield is as it was before the bits had a meaning. An earlier format has no place for them: serializing a stamped document in format 0, 1 or 2 is refused rather than dropping the stamp.
For example, if a document has $createdAt and $updatedAt set, the bitfield would be 0x0003, followed by 16 bytes (8 for each timestamp).
Price (conditional)
If the document type's trade_mode allows seller-set pricing:
0x01 [8 bytes big-endian u64] — price in credits
0x00 — no price set
User-defined properties
Properties are serialized in schema position order — each property in the data contract schema has a position field, and document_type.properties() returns an IndexMap sorted by that position. This is not alphabetical order.
Each property is encoded based on its type and whether it is required. In serialization version 3, "required" means required at the document's contract version stamp (see above); in versions 0–2 — and for every property without a requiredSince annotation — it is simply whether the property is listed in required.
Required fields: The value is written directly with no prefix byte.
Optional fields: A 1-byte presence flag is written first:
0x01followed by the encoded value — field is present0x00— field is absent
Transient fields: Always get a presence byte, even if marked as required. The serializer checks if !property.required || property.transient to decide whether to write the flag.
Value encoding by type
All numeric values use big-endian byte order.
| Type | Encoding |
|---|---|
u8 / i8 | 1 byte |
u16 / i16 | 2 bytes big-endian |
u32 / i32 | 4 bytes big-endian |
u64 / i64 | 8 bytes big-endian |
u128 / i128 | 16 bytes big-endian |
f64 | 8 bytes big-endian IEEE 754 |
boolean | 1 byte: 0x01 = true, 0x00 = false |
string | varint length prefix + UTF-8 bytes |
byteArray (fixed size) | raw bytes (no length prefix if min_size == max_size) |
byteArray (variable size) | varint length prefix + raw bytes |
identifier | 32 bytes raw |
date | 8 bytes big-endian f64 (when optional: 0xff prefix + 8 bytes) |
array (typed array, protocol v14) | varint element count + each element encoded exactly as a required property of the element's type (rows above): an identifier element is 32 raw bytes, an integer element takes the width its bounds give it, a fixed-size byte array element is raw, a string or variable-size byte array element has a varint length prefix. Elements never carry a presence byte |
object | Nested fields serialized recursively, in the order the schema lists them (not sorted by position) |
Note on date types: User-property date fields are encoded as f64 (8 bytes). System timestamps ($createdAt, $updatedAt, $transferredAt) are u64 milliseconds. Both are 8 bytes big-endian but use different numeric representations.
Important: In serialization version 0, all integer types are encoded as i64 (8 bytes big-endian), regardless of the actual type in the schema. This means a u8 field that should be 1 byte is encoded as 8 bytes in v0.
Worked example: withdrawal document
The withdrawals contract defines a withdrawal document type with these properties (in schema position order):
| Position | Property | Type | Required |
|---|---|---|---|
| 0 | transactionIndex | integer (i64) | no |
| 1 | transactionSignHeight | integer (i64) | no |
| 2 | amount | integer (i64) | yes |
| 3 | coreFeePerByte | integer (i64) | yes |
| 4 | pooling | integer (i64) | yes |
| 5 | outputScript | byteArray (23–25 bytes, variable) | yes |
| 6 | status | integer (i64) | yes |
The document type requires $createdAt, $updatedAt, and $revision.
Here is a real serialized withdrawal document (hex), broken down byte by byte:
02 ← serialization version (varint: 2)
0222 9eda 94b3 5be5 5ac2 22ca 8cc4 ← $id (32 bytes)
631c 0717 c9ee 4a22 3f2a 269e 06c9
a1be 7c54
36b3 e63b a54a ba9b 7599 4128 d124 ← $ownerId (32 bytes)
e9e1 cebe 348c d304 15b5 098c 6052
6de0 157e
← no $creatorId: withdrawal document type
does not support transfers/trading
c501 ← $revision (varint: 197)
0003 ← time bitfield: bits 0,1 set
($createdAt + $updatedAt)
0000019cd70f3323 ← $createdAt: 1773134623523 ms
(2026-03-10 09:23:43 UTC)
0000019d05406f0c ← $updatedAt: 1773909602060 ms
(2026-03-19 08:40:02 UTC)
← user properties follow (schema-dependent)
To properly decode the properties section, you need the document type schema — field names, types, required flags, and order. This is why the decode-document CLI tool (in packages/rs-scripts) requires the contract and document type to be specified.
Using the tool on this document produces:
id: 9LSAr59Fw7A1PHvX9WV1RWHCjL4PrijrHZpwhYDPkMq
owner_id: 4gY7wFM4o53jc8PJZ9KNzqzaJhXhPVMivJREKVwihKVF
created_at: 2026-03-10 09:23:43 UTC
updated_at: 2026-03-19 08:40:02 UTC
revision: 197
properties:
amount: (i64)191000
coreFeePerByte: (i64)1
outputScript: bytes 76a914...88ac
pooling: (i64)0
status: (i64)2
transactionIndex: (i64)9815
transactionSignHeight: (i64)2440497
The decode-document CLI tool
For convenience, the rs-scripts crate provides a decode-document binary that handles all of this deserialization automatically:
# Install
cargo install --path packages/rs-scripts
# Decode a withdrawal document from base64
decode-document -c withdrawals -d withdrawal "AgIintqUs1vl..."
# Decode from hex
decode-document -c withdrawals -d withdrawal "0202229eda94b35b..."
# Use a contract ID instead of a name
decode-document -c 4fJLR2GYTPFdomuTVvNy3VRrvWgvkKPzqehEBpNf2nk6 -d withdrawal "..."
See packages/rs-scripts/README.md for full usage details.
Common pitfalls for third-party deserializers
-
The serialization version varint changed the layout. If your code was written for version 0 or 1, version 2 documents will have different field offsets due to the
$creatorIdfield. Always read the version varint first and branch accordingly. -
Integer encoding differs between v0 and v1+. In version 0, a
u8field occupies 8 bytes (encoded as i64). In version 1+, it occupies 1 byte. Parsing with the wrong version assumption will shift every subsequent field. -
The time fields bitfield is variable-length data. The 2-byte bitfield tells you how many time fields follow. If you assume a fixed number of time fields, any document with a different set of time fields (e.g., one that includes
$transferredAtor block heights) will be misaligned. -
Property order is schema position order, not alphabetical. Each property in the data contract schema has a
positionfield. Properties are serialized in ascending position order (stored in anIndexMap). If you assume alphabetical order or JSON declaration order, fields will be read from the wrong positions. -
Optional fields have a presence byte. If you forget to read the
0x00/0x01prefix for optional fields, every subsequent field will be shifted by one byte. -
ByteArray encoding depends on size constraints. Fixed-size byte arrays (where
minItems == maxItemsin the schema) have no length prefix. Variable-size byte arrays have a varint length prefix. Check the schema to know which encoding is used. -
A typed array's element width comes from its
itemsschema. Each element is laid out as a required property of the element's type, so an integer element bounded0..100is 1 byte and an unbounded one 8, and a fixed-size byte array or identifier element has no length prefix. Parse theitemsschema exactly as a property schema to know the width, including the contract'ssizedIntegerTypessetting. -
In version 3, the same document type can produce different property layouts. A property annotated with
requiredSinceis presence-flagged in documents stamped below the annotation and raw in documents stamped at or above it. Two version-3 documents of the same type may therefore differ in layout — always read the stamp varint and resolve each property's requiredness against it before decoding the properties section.
Derive Macros
In the previous chapter we looked at the PlatformVersionEncode and PlatformVersionedDecode traits and the platform_encode_to_vec / platform_versioned_decode_from_slice functions. But you rarely implement those traits by hand. Instead, you use three derive macros from the rs-platform-serialization-derive crate:
PlatformSerialize-- generates a high-levelserialize_to_bytes()methodPlatformDeserialize-- generates a high-leveldeserialize_from_bytes()methodPlatformSignable-- generates asignable_bytes()method that excludes signature fields
These macros live in packages/rs-platform-serialization-derive/src/lib.rs and are the glue that connects Rust struct definitions to the platform serialization system.
PlatformSerialize and PlatformDeserialize
These two macros work as a pair. Let us start with how they are used on the ConsensusError type:
#![allow(unused)] fn main() { #[derive( thiserror::Error, Debug, Encode, Decode, PlatformSerialize, PlatformDeserialize, Clone, PartialEq, )] #[platform_serialize(limit = 2000)] #[error(transparent)] pub enum ConsensusError { // ... } }
And on a leaf error struct:
#![allow(unused)] fn main() { #[derive( Error, Debug, Clone, PartialEq, Eq, Encode, Decode, PlatformSerialize, PlatformDeserialize, )] #[error("Document {document_id} is already present")] #[platform_serialize(unversioned)] pub struct DocumentAlreadyPresentError { document_id: Identifier, } }
And on the StateTransition enum:
#![allow(unused)] fn main() { #[derive( Debug, Clone, Encode, Decode, PlatformSerialize, PlatformDeserialize, PlatformSignable, From, PartialEq, )] #[platform_serialize(unversioned)] #[platform_serialize(limit = 100000)] pub enum StateTransition { DataContractCreate(DataContractCreateTransition), DataContractUpdate(DataContractUpdateTransition), Batch(BatchTransition), // ... } }
Notice that StateTransition uses two #[platform_serialize] attributes -- one for unversioned and one for limit. These are combined internally.
What the derives generate
PlatformSerialize generates an implementation of one of two traits, depending on whether unversioned is set:
With unversioned -- implements PlatformSerializable:
#![allow(unused)] fn main() { // Generated code (simplified): impl PlatformSerializable for DocumentAlreadyPresentError { type Error = ProtocolError; fn serialize_to_bytes(&self) -> Result<Vec<u8>, Self::Error> { let config = bincode::config::standard() .with_big_endian() .with_no_limit(); bincode::encode_to_vec(self, config) .map_err(|e| { ProtocolError::PlatformSerializationError( format!("unable to serialize DocumentAlreadyPresentError: {}", e) ) }) } fn serialize_consume_to_bytes(self) -> Result<Vec<u8>, Self::Error> { // same as above, taking self by value } } }
Without unversioned -- implements PlatformSerializableWithPlatformVersion:
#![allow(unused)] fn main() { // Generated code (simplified): impl PlatformSerializableWithPlatformVersion for ConsensusError { type Error = ProtocolError; fn serialize_to_bytes_with_platform_version( &self, platform_version: &PlatformVersion, ) -> Result<Vec<u8>, ProtocolError> { let config = bincode::config::standard() .with_big_endian() .with_limit::<{ 2000 }>(); platform_serialization::platform_encode_to_vec(self, config, platform_version) .map_err(|e| match e { bincode::error::EncodeError::Io { inner, index } => ProtocolError::MaxEncodedBytesReachedError { max_size_kbytes: 2000, size_hit: index, }, _ => ProtocolError::PlatformSerializationError( format!("unable to serialize ConsensusError: {}", e) ), }) } } }
The key differences: the unversioned variant uses plain bincode::encode_to_vec, while the versioned variant uses platform_serialization::platform_encode_to_vec which threads the PlatformVersion through the encode chain. When a limit is set, the config uses with_limit::<{ N }>() and the error mapping converts bincode IO errors to MaxEncodedBytesReachedError.
In both cases, the derive also generates bincode Encode and Decode implementations by internally calling into derive_bincode -- this is why you still need Encode and Decode in the derive list alongside PlatformSerialize and PlatformDeserialize.
The #[platform_serialize] attributes
The #[platform_serialize(...)] attribute accepts several parameters. Here is the full list from the derive macro source in packages/rs-platform-serialization-derive/src/lib.rs:
limit = N
Sets the maximum serialized size in bytes:
#![allow(unused)] fn main() { #[platform_serialize(limit = 2000)] }
When encoding exceeds this limit, the error is MaxEncodedBytesReachedError. This is critical for types received from the network.
unversioned
Generates PlatformSerializable instead of PlatformSerializableWithPlatformVersion:
#![allow(unused)] fn main() { #[platform_serialize(unversioned)] }
Use this when the type's serialization format does not change between protocol versions. Most leaf types (individual error structs, simple data holders) use this.
passthrough
For enums, serializes by delegating directly to the inner variant's serialization method:
#![allow(unused)] fn main() { #[platform_serialize(passthrough)] }
When MyEnum::Variant(inner) is serialized with passthrough, it calls inner.serialize() directly rather than encoding the enum tag + inner data. This means the enum variant information is lost in serialization -- deserialization must use platform_version_path to know which variant to decode into.
Cannot be combined with limit, untagged, or into.
untagged
For enums, serializes without the variant tag number:
#![allow(unused)] fn main() { #[platform_serialize(untagged)] }
Similar to passthrough but still uses the enum's own encoding logic rather than delegating to the inner type. The variant index is omitted from the output. Like passthrough, deserialization requires knowing which variant to expect.
Cannot be combined with passthrough.
into = "TypePath"
For structs, converts the value to another type before serialization:
#![allow(unused)] fn main() { #[platform_serialize(into = "DataContractInSerializationFormat")] }
The generated code calls .into() to convert to the target type, then serializes that type. This is useful for types that have a different in-memory representation than their serialization format.
Cannot be used on enums.
platform_version_path = "..."
Used with passthrough or untagged to specify how to determine the correct variant during deserialization:
#![allow(unused)] fn main() { #[platform_serialize( passthrough, platform_version_path = "dpp.contract_versions.contract_serialization_version.default_current_version" )] }
The deserialization code reads this field path from the PlatformVersion to determine which variant index to decode.
crate_name = "..."
Overrides the default crate path (defaults to crate):
#![allow(unused)] fn main() { #[platform_serialize(crate_name = "dpp")] }
allow_prepend_version and force_prepend_version
These flags are defined in the code but currently not actively used in the main serialization path. They were designed for prepending version bytes to serialized output. Only one can be used at a time.
The #[platform_error_type] attribute
By default, the derives use ProtocolError as the error type. You can override this:
#![allow(unused)] fn main() { #[derive(PlatformSerialize, PlatformDeserialize)] #[platform_error_type(MyCustomError)] pub struct MyType { ... } }
The error type must have PlatformSerializationError(String) and MaxEncodedBytesReachedError { max_size_kbytes, size_hit } variants (or equivalent conversion paths).
PlatformSignable -- the signature hash derive
The PlatformSignable derive solves a specific problem: when you sign a state transition, you need to hash all the fields except the signature itself. You cannot include the signature in the data that was signed -- that would be circular.
Here is how it is used on DataContractCreateTransitionV0 in packages/rs-dpp/src/state_transition/state_transitions/contract/data_contract_create_transition/v0/mod.rs:
#![allow(unused)] fn main() { #[derive(Debug, Clone, Encode, Decode, PartialEq, PlatformSignable)] pub struct DataContractCreateTransitionV0 { pub data_contract: DataContractInSerializationFormat, pub identity_nonce: IdentityNonce, pub user_fee_increase: UserFeeIncrease, #[platform_signable(exclude_from_sig_hash)] pub signature_public_key_id: KeyID, #[platform_signable(exclude_from_sig_hash)] pub signature: BinaryData, } }
The #[platform_signable(exclude_from_sig_hash)] attribute marks fields that should be excluded from the signature hash. The derive generates:
- A new struct
DataContractCreateTransitionV0Signable<'a>containing only the non-excluded fields asCowreferences - A
From<&DataContractCreateTransitionV0>implementation for the signable struct - An implementation of the
Signabletrait on the original struct
The generated code looks approximately like this:
#![allow(unused)] fn main() { // Generated by PlatformSignable derive: #[derive(Debug, Clone, bincode::Encode)] pub struct DataContractCreateTransitionV0Signable<'a> { data_contract: std::borrow::Cow<'a, DataContractInSerializationFormat>, identity_nonce: std::borrow::Cow<'a, IdentityNonce>, user_fee_increase: std::borrow::Cow<'a, UserFeeIncrease>, // signature_public_key_id -- excluded // signature -- excluded } impl<'a> From<&'a DataContractCreateTransitionV0> for DataContractCreateTransitionV0Signable<'a> { fn from(original: &'a DataContractCreateTransitionV0) -> Self { DataContractCreateTransitionV0Signable { data_contract: std::borrow::Cow::Borrowed(&original.data_contract), identity_nonce: std::borrow::Cow::Borrowed(&original.identity_nonce), user_fee_increase: std::borrow::Cow::Borrowed(&original.user_fee_increase), } } } impl crate::serialization::Signable for DataContractCreateTransitionV0 { fn signable_bytes(&self) -> Result<Vec<u8>, ProtocolError> { let config = bincode::config::standard().with_big_endian(); let intermediate: DataContractCreateTransitionV0Signable = self.into(); bincode::encode_to_vec(intermediate, config).map_err(|e| { ProtocolError::PlatformSerializationError( format!("unable to serialize to produce sig hash \ DataContractCreateTransitionV0: {}", e) ) }) } } }
The Cow references avoid cloning the data just to produce a hash. The intermediate struct borrows from the original and only serializes the fields that matter for the signature.
PlatformSignable on enums
When used on an enum (like StateTransition), the derive generates a corresponding Signable enum where each variant wraps the signable version of its inner type:
#![allow(unused)] fn main() { // On StateTransition: #[derive(PlatformSignable)] pub enum StateTransition { DataContractCreate(DataContractCreateTransition), DataContractUpdate(DataContractUpdateTransition), // ... } // Generates: #[derive(Debug, Clone, bincode::Encode, derive_more::From)] pub enum StateTransitionSignable<'a> { DataContractCreate(DataContractCreateTransitionSignable<'a>), DataContractUpdate(DataContractUpdateTransitionSignable<'a>), // ... } }
The signable_bytes implementation on the enum encodes a variant index (as u16) followed by the inner type's signable bytes:
#![allow(unused)] fn main() { impl Signable for StateTransition { fn signable_bytes(&self) -> Result<Vec<u8>, ProtocolError> { let config = bincode::config::standard().with_big_endian(); let signable_bytes = match self { StateTransition::DataContractCreate(ref inner) => { let mut buf = bincode::encode_to_vec(&(0u16), config).unwrap(); let inner_signable_bytes = inner.signable_bytes()?; buf.extend(inner_signable_bytes); buf } StateTransition::DataContractUpdate(ref inner) => { let mut buf = bincode::encode_to_vec(&(1u16), config).unwrap(); let inner_signable_bytes = inner.signable_bytes()?; buf.extend(inner_signable_bytes); buf } // ... }; Ok(signable_bytes) } } }
PlatformSignable attributes
exclude_from_sig_hash-- marks a field to be excluded from the signable structinto = "TypePath"-- converts a field to a different type in the signable struct (used for fields that need a different representation for hashing)derive_into-- on enums, generatesFromconversions from the original enum to the signable enumderive_bincode_with_borrowed_vec-- manually implementsbincode::Encodefor the signable struct instead of deriving it (needed when fields contain borrowedVectypes)
The relationship between Encode/Decode and PlatformSerialize/PlatformDeserialize
This is a common source of confusion. Here is how the pieces fit together:
-
Encode/Decode(from bincode) -- field-level encoding. These know how to write each field to bytes and read it back. They are the low-level building blocks. -
PlatformVersionEncode/PlatformVersionedDecode(from rs-platform-serialization) -- version-aware field-level encoding. These wrapEncode/Decodewith aPlatformVersionparameter. -
PlatformSerialize/PlatformDeserialize(derive macros) -- high-level serialization. These generate theserialize_to_bytes()anddeserialize_from_bytes()methods that configure bincode, enforce size limits, and convert errors.
When you derive all of them on a type, the call chain is:
your_type.serialize_to_bytes() // PlatformSerialize-generated method
-> platform_encode_to_vec(...) // from rs-platform-serialization
-> your_type.platform_encode(...) // PlatformVersionEncode (auto-generated)
-> field.encode(encoder) // bincode Encode for each field
You need Encode and Decode in the derive list alongside PlatformSerialize and PlatformDeserialize because the platform derives internally delegate to bincode encoding.
Rules
Do:
- Always derive
Encode, Decode, PlatformSerialize, PlatformDeserializetogether - Use
#[platform_serialize(unversioned)]for types with a stable serialization format - Use
#[platform_serialize(limit = N)]on types received from untrusted sources - Use
#[platform_signable(exclude_from_sig_hash)]on signature and signature key ID fields - Keep the
#[platform_error_type]attribute consistent with the crate's error type
Do not:
- Use
passthroughon structs (it is enum-only) - Use
intoon enums (it is struct-only) - Combine
passthroughwithlimit,untagged, orinto - Combine
force_prepend_versionwithallow_prepend_version - Forget that
PlatformSignableon enums requires each inner type to also implementSignable - Change the order of fields in a
PlatformSignablestruct -- this changes the signature hash, which invalidates existing signatures
Platform Addresses
Dash Platform has its own address system, independent of the legacy Base58Check addresses used on the Core chain. Platform addresses use Bech32m encoding (BIP-350), following the specification in DIP-0018. There are three address types: two transparent and one shielded.
If you are coming from Bitcoin or Dash Core, the key mental shift is this: platform addresses are not derived from a single private key via a single algorithm. They are typed containers for a hash or a public key, unified under a single encoding scheme with a shared human-readable prefix.
The Three Address Types
| Type | Inner Data | Bech32m Type Byte | Prefix (mainnet) | Prefix (testnet) |
|---|---|---|---|---|
| P2PKH | 20-byte pubkey hash | 0xb0 | dash1k... | tdash1k... |
| P2SH | 20-byte script hash | 0x80 | dash1s... | tdash1s... |
| Orchard | 43-byte shielded address | 0x10 | dash1z... | tdash1z... |
The type byte is the first byte of the Bech32m data payload. It determines how the rest of the payload is interpreted. The type bytes were chosen so that the first character after dash1 (or tdash1) is a memorable letter: k for keys (P2PKH), s for scripts (P2SH), z for zero-knowledge (Orchard).
PlatformAddress
The transparent address type is defined in packages/rs-dpp/src/address_funds/platform_address.rs:
#![allow(unused)] fn main() { pub enum PlatformAddress { P2pkh([u8; 20]), P2sh([u8; 20]), } }
Two variants, both wrapping a 20-byte hash. A P2pkh address contains a Hash160(compressed_pubkey), just like on Core. A P2sh address contains a Hash160(redeem_script), supporting standard multisig scripts.
Bech32m Encoding
The human-readable part (HRP) depends on the network:
#![allow(unused)] fn main() { const PLATFORM_HRP_MAINNET: &str = "dash"; const PLATFORM_HRP_TESTNET: &str = "tdash"; // also devnet, regtest }
Encoding produces addresses like:
dash1krma5z3ttj75la4m93xcndna9ullamq9y5e9n5rs(P2PKH, mainnet)tdash1sppl5xpu70aka8nacc4kj2htflydspzkc8jtru5(P2SH, testnet)
The wire format is type_byte || hash -- 21 bytes total, then Bech32m-encoded with the appropriate HRP.
Storage vs. Bech32m: Two Byte Schemes
There is an important distinction between how addresses are encoded for users and how they are serialized for storage. The Bech32m type bytes (0xb0, 0x80) are not the same as the bincode variant indices (0x00, 0x01) used in GroveDB keys:
| Context | P2PKH byte | P2SH byte |
|---|---|---|
| Bech32m (user-facing) | 0xb0 | 0x80 |
| Bincode (storage/wire) | 0x00 | 0x01 |
This matters when you encounter raw bytes. If the leading byte is 0xb0 or 0x80, you are looking at a Bech32m payload. If it is 0x00 or 0x01, it is bincode-serialized. The to_bytes() method produces bincode format; to_bech32m_string() produces the user-facing string.
Conversion to Core Addresses
PlatformAddress can be converted to and from dashcore::Address, the type used by the Core chain RPC client. This allows the platform layer to interoperate with Core's address format when needed -- for example, when processing withdrawals that ultimately create Core chain transactions.
OrchardAddress
The shielded address type is defined in packages/rs-dpp/src/address_funds/orchard_address.rs:
#![allow(unused)] fn main() { pub struct OrchardAddress(grovedb_commitment_tree::PaymentAddress); }
It wraps the Orchard protocol's native PaymentAddress, which consists of two components:
| Component | Size | Purpose |
|---|---|---|
| Diversifier | 11 bytes | Entropy for deriving unlinkable addresses from a single spending key |
| pk_d | 32 bytes | Diversified transmission key (Pallas curve point) |
Total: 43 bytes. The Bech32m payload is 0x10 || diversifier || pk_d -- 44 bytes.
Key Constants
#![allow(unused)] fn main() { const ORCHARD_DIVERSIFIER_SIZE: usize = 11; const ORCHARD_PKD_SIZE: usize = 32; const ORCHARD_ADDRESS_SIZE: usize = 43; const ORCHARD_TYPE: u8 = 0x10; }
Diversifiers and Unlinkability
A single Orchard spending key can derive an unlimited number of addresses by varying the diversifier. Each address looks completely unrelated to any other address derived from the same key. Only the holder of the corresponding Incoming Viewing Key (IVK) can link them.
This is the fundamental privacy property: a merchant can give every customer a unique address, and no observer can determine that all those addresses belong to the same wallet. The diversifier is the mechanism that makes this possible.
Differences from Zcash Encoding
Dash's OrchardAddress uses the same 43-byte raw format as Zcash Orchard addresses (identical diversifier + pk_d structure). But the Bech32m encoding is Dash-specific:
- No F4Jumble: Zcash applies an F4Jumble permutation before encoding; Dash does not.
- No Unified Address wrapper: Zcash wraps Orchard addresses in a Unified Address (UA) envelope with typecodes and length prefixes; Dash uses a simple type-byte scheme.
- Different HRP: Zcash uses
u1; Dash usesdash/tdash.
The result is simpler, shorter addresses that are not interoperable with Zcash wallets.
Converting to Orchard's Native Type
The builder functions that construct shielded transactions need the Orchard library's native PaymentAddress, not our wrapper. Conversion is straightforward:
#![allow(unused)] fn main() { impl From<&OrchardAddress> for PaymentAddress { fn from(address: &OrchardAddress) -> Self { *address.inner() } } }
This is used in the shielded builder module (packages/rs-dpp/src/shielded/builder/) when adding outputs to an Orchard bundle.
Address Witnesses
When a transparent address is used as an input (for example, funding a shield transition), the sender must prove ownership. This is done through an AddressWitness, defined in packages/rs-dpp/src/address_funds/witness.rs:
#![allow(unused)] fn main() { pub enum AddressWitness { P2pkh { signature: BinaryData, }, P2sh { signatures: Vec<BinaryData>, redeem_script: BinaryData, }, } }
P2PKH Witnesses
A P2PKH witness contains a single 65-byte recoverable ECDSA signature. The public key is recovered from the signature rather than transmitted alongside it -- saving 33 bytes per witness. Verification recovers the public key, hashes it with Hash160, and checks the result against the address hash.
P2SH Witnesses
A P2SH witness contains the signatures and the redeem script. Only standard bare multisig scripts are supported:
OP_M <pubkey1> <pubkey2> ... <pubkeyN> OP_N OP_CHECKMULTISIG
Constraints:
- Maximum 17 signature entries (
MAX_P2SH_SIGNATURES) -- 16 keys plus 1 dummy for the CHECKMULTISIG bug - Only compressed public keys (33 bytes)
- No timelocks, hash puzzles, or custom scripts
- The redeem script must hash to the address:
Hash160(script) == address_hash
Verification Cost Tracking
Every witness verification has a measurable cost. The AddressWitnessVerificationOperations struct tracks the work:
#![allow(unused)] fn main() { pub struct AddressWitnessVerificationOperations { pub ecdsa_signature_verifications: u16, pub message_hash_count: u16, pub pubkey_hash_verifications: u16, pub script_hash_verifications: u16, pub signable_bytes_len: usize, } }
These counts feed into the fee system. A P2PKH witness costs one ECDSA verification. An M-of-N multisig costs N verifications (all public keys are checked against all signatures, per Bitcoin's CHECKMULTISIG semantics).
How Addresses Are Used in Shielded Transitions
Each of the five shielded transition types uses addresses differently:
Shield (Transparent to Shielded)
The sender provides one or more PlatformAddress inputs, each with a nonce and a maximum contribution amount. An AddressWitness proves ownership of each input. The funds enter the shielded pool and are received by an OrchardAddress embedded in the Orchard bundle's encrypted outputs.
#![allow(unused)] fn main() { pub struct ShieldTransitionV0 { pub inputs: BTreeMap<PlatformAddress, (AddressNonce, Credits)>, pub input_witnesses: Vec<AddressWitness>, pub actions: Vec<SerializedAction>, // contains encrypted OrchardAddress recipient pub amount: u64, // ... } }
Shielded Transfer (Shielded to Shielded)
No transparent addresses are involved. The sender spends notes from the shielded pool and creates new notes for the recipient's OrchardAddress and a change OrchardAddress. Both addresses are hidden inside the Orchard bundle -- only the respective recipients can decrypt them.
#![allow(unused)] fn main() { pub struct ShieldedTransferTransitionV0 { pub actions: Vec<SerializedAction>, // spend + output pairs pub value_balance: u64, // fee extracted from pool // ... } }
Unshield (Shielded to Transparent)
The sender spends shielded notes and sends the funds to a PlatformAddress visible on-chain. The transparent output address is explicitly included in the transition and bound into the Orchard sighash to prevent substitution.
#![allow(unused)] fn main() { pub struct UnshieldTransitionV0 { pub output_address: PlatformAddress, // visible transparent recipient pub unshielding_amount: u64, pub actions: Vec<SerializedAction>, // spend + change output // ... } }
Shielded Withdrawal (Shielded to Core Chain)
Similar to unshield, but the destination is a Core chain script rather than a platform address. The CoreScript holds a raw P2PKH or P2SH script that will be used in a Core chain withdrawal transaction.
#![allow(unused)] fn main() { pub struct ShieldedWithdrawalTransitionV0 { pub output_script: CoreScript, // Core chain P2PKH or P2SH pub core_fee_per_byte: u32, pub pooling: Pooling, // ... } }
Shield From Asset Lock (Core Chain to Shielded)
An asset lock proof from the Core chain funds the shielded pool directly. The recipient is an OrchardAddress inside the Orchard bundle. No PlatformAddress inputs are needed -- the asset lock proof substitutes for them.
The transition also carries an optional surplus_output: Option<PlatformAddress>. When the consumed asset lock exceeds shield_amount + pool_fee, the leftover surplus is credited to this platform address; if it is unset, the surplus folds into the fee pools (bounded by shielded_implicit_fee_cap). See Entry-Transition Fees. Unlike the transparent recipients of Unshield/ShieldedWithdrawal (which are bound through the Orchard sighash extra_data above), surplus_output is bound through the state transition's own platform_signable signature -- it sits before the signature field, so it is part of the signed payload and cannot be substituted or truncated after signing. The Orchard extra_data therefore does not carry it; what it binds instead is the asset lock itself (see below).
The Platform Sighash
When transparent fields need to be bound to an Orchard bundle's proof, the platform uses a custom sighash computation defined in packages/rs-dpp/src/shielded/mod.rs:
#![allow(unused)] fn main() { const SIGHASH_DOMAIN: &[u8] = b"DashPlatformSighash"; pub fn compute_platform_sighash( bundle_commitment: &[u8; 32], extra_data: &[u8], ) -> [u8; 32] { let mut hasher = Sha256::new(); hasher.update(SIGHASH_DOMAIN); hasher.update(bundle_commitment); hasher.update(extra_data); hasher.finalize().into() } }
The bundle_commitment is a BLAKE2b-256 hash of the Orchard bundle (per ZIP-244). The extra_data binds transparent fields:
| Transition | extra_data |
|---|---|
| Shield | 0x84 || SHA-256(input addresses) |
| Shield From Identity | 0x85 || identity_id |
| Shielded Transfer | empty |
| Unshield | output_address.to_bytes() || amount.to_le_bytes() |
| Shielded Withdrawal | output_script.as_bytes() |
| Shield From Asset Lock | 0x86 || asset lock identifier |
This binding is critical for security. Without it, an attacker who intercepts an unshield transition could substitute the output_address while reusing the valid Orchard proof and signatures. The sighash ensures the Orchard bundle's spend authorization signatures commit to the specific transparent recipient.
The three transitions that only create notes bind something for a different reason. Their bundles have no spends, so they carry no anchor of their own: the proof and the binding signature verify wherever the bundle is submitted. Left unbound, the authorized bundle is a self-contained object anybody could lift out of the mempool and wrap in a transition of their own, paid with their own credits, and any bundle ever published could be replayed. The copier gains nothing — they pay the full amount to the original recipient — but the copy lands a second note with the same commitment and the same nullifier in the credit pool, so only one of the two can ever be spent.
Each of these bundles therefore binds a kind tag and its owner, the thing that funds it and that a third party cannot authorize:
- Shield: the SHA-256 of its input addresses, each in its 21-byte encoding (
PlatformAddress::to_bytes), in the order the transition serializes its inputs. The nonces and contributed amounts are not included: the owner says who funds the bundle, not which transition carries it. - Shield From Identity: the identity id.
- Shield From Asset Lock: the asset lock identifier, the double SHA-256 of the locked 36-byte outpoint (the same value an identity created from that lock would get as its id; the tag keeps the two apart). Because a successful shield consumes the whole lock, a bundle bound to it can land at most once, even when its own sender resubmits it. Binding only the transaction id would not do this, and would let the holder of another credit output of the same lock transaction lift the bundle.
The tags 0x84, 0x85 and 0x86 sit above the state transition type range, next to the token pools' outputs-only tags 0x80–0x83. The binding starts at protocol version 14; at earlier protocol versions Shield and Shield From Asset Lock bind nothing, and a client building for one of those versions must bind nothing too. Both sides read the choice from the same version field (dpp.methods.credit_pool_bundle_binding), so a client that builds with the network's protocol version produces what that network verifies.
Shield From Asset Lock also changes its transition version at protocol version 14: version 1 carries the bound bundle and is the only version 14 admits, while version 0, whose bundle binds nothing, is admitted only up to 13. A version 0 presented at 14, such as one still waiting in the mempool when 14 activates, is refused on its version byte when the transition is decoded, before any proof is verified: nothing is charged and its asset lock stays unspent, so its sender can resubmit it as version 1 from the same lock. Checking its unbound bundle against the bound preimage instead would fail the proof and burn the proof-failure penalty from an honest lock.
The binding does not prevent Faerie Gold. The rho of an outputs-only note is the nullifier of its bundle's dummy spend, so a sender who builds and signs a fresh bundle reusing the same dummy spend note and rseed gets the same commitment and the same nullifier whatever the preimage binds, and a recipient that counts deposits by commitment credits two payments where only one can be spent. Nor does it stop a Shield or Shield From Identity funder landing their own bundle twice through a new transition, as a client retrying with a fresh nonce does. Neither is closed: that would need the bundles' dummy nullifiers recorded and checked, which the platform does not do.
The same compute_platform_sighash function is used on both sides: the client uses it when signing the bundle, and the platform uses it when verifying.
Trial Decryption and Address Privacy
A shielded recipient does not appear anywhere in cleartext on-chain. To discover incoming payments, a wallet must attempt trial decryption of every new note using its Incoming Viewing Key (IVK).
The process works as follows:
- The wallet retrieves new note entries from the shielded pool (each entry contains a
nullifier,cmx, and 216-byteencrypted_note). - For each entry, the wallet constructs a
CompactActionfrom the nullifier, commitment, ephemeral public key, and encrypted ciphertext. - The wallet attempts trial decryption using
try_note_decryptionwith its IVK. - If decryption succeeds, the note was addressed to one of the wallet's
OrchardAddressinstances.
The nullifier stored alongside each note is essential -- it provides the Rho value needed for decryption. Without it, the Orchard protocol's forward secrecy mechanism would prevent the recipient from recovering the note.
Because a single spending key can generate unlimited OrchardAddress instances (via different diversifiers), the IVK-based scan catches all of them in a single pass. The diversifier that was used becomes apparent only after successful decryption.
Note Encryption Structure
Each SerializedAction contains an encrypted_note field of 216 bytes:
| Component | Size | Purpose |
|---|---|---|
| epk | 32 bytes | Ephemeral public key for Diffie-Hellman key agreement |
| enc_ciphertext | 104 bytes | Note plaintext encrypted to recipient (ChaCha20-Poly1305) |
| out_ciphertext | 80 bytes | Note encrypted to sender for wallet recovery |
The enc_ciphertext contains the note plaintext (52 bytes), a Dash-specific memo (36 bytes), and the AEAD authentication tag (16 bytes). The memo is smaller than Zcash's 512-byte memos -- Dash uses the DashMemo type (36 bytes) rather than ZcashMemo (512 bytes), keeping encrypted notes compact.
Storage in GroveDB
Notes are stored in a BulkAppendTree within the shielded credit pool:
AddressBalances / "s" (shielded_credit_pool) /
[1] notes -- CommitmentTree (BulkAppendTree)
[2] nullifiers -- Tree (spent note markers)
[5] total_balance -- SumItem
[6] anchors -- Tree (block_height -> anchor)
Each note entry stores:
cmx (32 bytes) || nullifier (32 bytes) || encrypted_note (216 bytes) = 280 bytes
The nullifier is stored alongside the note (rather than separately) specifically to support trial decryption. A scanning client needs both the encrypted ciphertext and the nullifier to attempt decryption.
Rules and Guidelines
Do:
- Use
to_bech32m_string()for user-facing address display. Always include the network parameter. - Use
to_bytes()(bincode format) for storage keys and wire serialization. - Generate a fresh diversifier for each new Orchard payment address to maximize unlinkability.
- Always bind transparent fields into the platform sighash. Forgetting to include the output address in an unshield transition would be a critical vulnerability.
Do not:
- Mix up Bech32m type bytes (
0xb0,0x80,0x10) with bincode variant bytes (0x00,0x01). They serve different purposes and are not interchangeable. - Assume Orchard addresses are interoperable with Zcash. The encoding differs (no F4Jumble, no Unified Address wrapper).
- Use non-standard scripts in P2SH addresses. Only bare multisig (
OP_M ... OP_N OP_CHECKMULTISIG) is supported. - Store or transmit the raw spending key. Use the Incoming Viewing Key for scanning and the Full Viewing Key for read-only wallet recovery.
- Attempt trial decryption without the nullifier. The Orchard protocol requires the
Rhoderived from the nullifier to reconstruct the note.
Data Contracts
If you have ever built a traditional web application, you know the pattern: define a database schema, then write code that reads and writes data conforming to that schema. Dash Platform follows the same idea, but on a decentralized network. A data contract is the on-chain schema that defines what application data looks like, how it is indexed, and who can modify it.
Every application on Dash Platform -- whether it is DPNS (the naming service), DashPay (social payments), or your own custom dApp -- starts by registering a data contract. Once registered, users can create, update, and query documents that conform to that contract's schema. Think of the data contract as the CREATE TABLE statement and documents as the rows.
The DataContract Enum
Like almost every core type in Platform, DataContract is a versioned enum. You will find its definition in packages/rs-dpp/src/data_contract/mod.rs:
#![allow(unused)] fn main() { #[derive(Debug, Clone, PartialEq, From, PlatformVersioned)] pub enum DataContract { V0(DataContractV0), V1(DataContractV1), } }
This is a pattern you will see over and over: the top-level type is an enum whose variants are concrete struct versions. The PlatformVersioned derive macro wires it into the protocol versioning system, and the From derive gives you free .into() conversions from either variant.
The enum also provides direct-access helpers for when you know exactly which version you are dealing with:
#![allow(unused)] fn main() { impl DataContract { pub fn as_v0(&self) -> Option<&DataContractV0> { ... } pub fn as_v1(&self) -> Option<&DataContractV1> { ... } pub fn into_v0(self) -> Option<DataContractV0> { ... } pub fn into_v1(self) -> Option<DataContractV1> { ... } } }
In tests, there are convenience methods as_latest() and into_latest() that always return the most recent variant. These are gated behind #[cfg(test)] because production code should never assume which version is "latest" -- the protocol version determines that.
What Lives Inside a Data Contract
The V0 struct contains the essentials. From packages/rs-dpp/src/data_contract/v0/data_contract.rs:
#![allow(unused)] fn main() { pub struct DataContractV0 { pub(crate) id: Identifier, pub(crate) version: u32, pub(crate) owner_id: Identifier, pub document_types: BTreeMap<DocumentName, DocumentType>, pub(crate) metadata: Option<Metadata>, pub(crate) config: DataContractConfig, pub(crate) schema_defs: Option<BTreeMap<DefinitionName, Value>>, } }
The key fields are:
id: A 32-byte identifier derived from the contract creation transaction. Globally unique.version: A monotonically increasing counter. Every time the contract owner updates the contract, this increments.owner_id: The identity that created (and can update) this contract.document_types: A map from document type names (like"contactRequest"or"domain") to theirDocumentTypedefinitions, which include the JSON Schema, indexes, and mutability rules.config: Contract-level configuration such as whether documents can be deleted, encryption key requirements, and so on.schema_defs: Shared JSON Schema$defsthat document types can reference.
What V1 Added
DataContractV1 extends V0 with several important capabilities. From packages/rs-dpp/src/data_contract/v1/data_contract.rs:
#![allow(unused)] fn main() { pub struct DataContractV1 { // All V0 fields... pub id: Identifier, pub version: u32, pub owner_id: Identifier, pub document_types: BTreeMap<DocumentName, DocumentType>, pub config: DataContractConfig, pub schema_defs: Option<BTreeMap<DefinitionName, Value>>, // New in V1: pub created_at: Option<TimestampMillis>, pub updated_at: Option<TimestampMillis>, pub created_at_block_height: Option<BlockHeight>, pub updated_at_block_height: Option<BlockHeight>, pub created_at_epoch: Option<EpochIndex>, pub updated_at_epoch: Option<EpochIndex>, pub groups: BTreeMap<GroupContractPosition, Group>, pub tokens: BTreeMap<TokenContractPosition, TokenConfiguration>, pub keywords: Vec<String>, pub description: Option<String>, } }
The additions fall into four categories:
-
Timestamps and block tracking --
created_at,updated_at,created_at_block_height,updated_at_block_height,created_at_epoch,updated_at_epoch. These provide an immutable audit trail of when the contract was created and last modified. -
Groups --
BTreeMap<GroupContractPosition, Group>. Groups enable multiparty governance. Each group has a set of member identities with associated voting power and a required power threshold for actions. -
Tokens --
BTreeMap<TokenContractPosition, TokenConfiguration>. Contracts can now define and manage tokens with configurable supply limits, minting/burning rules, and governance controls. -
Searchability --
keywordsanddescriptionmake contracts discoverable through the platform'ssearchsystem contract.
The Versioned Accessors Pattern
Here is where it gets interesting. You do not access data contract fields directly in most code. Instead, you go through accessor traits. These live in packages/rs-dpp/src/data_contract/accessors/.
The V0 getter trait, defined in accessors/v0/mod.rs:
#![allow(unused)] fn main() { pub trait DataContractV0Getters { fn id(&self) -> Identifier; fn id_ref(&self) -> &Identifier; fn version(&self) -> u32; fn owner_id(&self) -> Identifier; fn document_type_for_name(&self, name: &str) -> Result<DocumentTypeRef<'_>, DataContractError>; fn document_types(&self) -> &BTreeMap<DocumentName, DocumentType>; fn config(&self) -> &DataContractConfig; // ... and more } pub trait DataContractV0Setters { fn set_id(&mut self, id: Identifier); fn set_version(&mut self, version: u32); fn increment_version(&mut self); fn set_owner_id(&mut self, owner_id: Identifier); fn set_config(&mut self, config: DataContractConfig); } }
And the V1 getter trait extends V0, defined in accessors/v1/mod.rs:
#![allow(unused)] fn main() { pub trait DataContractV1Getters: DataContractV0Getters { fn groups(&self) -> &BTreeMap<GroupContractPosition, Group>; fn tokens(&self) -> &BTreeMap<TokenContractPosition, TokenConfiguration>; fn created_at(&self) -> Option<TimestampMillis>; fn updated_at(&self) -> Option<TimestampMillis>; fn keywords(&self) -> &Vec<String>; fn description(&self) -> Option<&String>; // ... and more } }
Notice that DataContractV1Getters has a supertrait bound on DataContractV0Getters. This means anything that implements V1 getters automatically has V0 getters too. You can use both interchangeably.
How the Enum Dispatches
The magic happens in packages/rs-dpp/src/data_contract/accessors/mod.rs, where the top-level DataContract enum implements both traits by dispatching to the inner variant:
#![allow(unused)] fn main() { impl DataContractV0Getters for DataContract { fn id(&self) -> Identifier { match self { DataContract::V0(v0) => v0.id(), DataContract::V1(v1) => v1.id(), } } fn version(&self) -> u32 { match self { DataContract::V0(v0) => v0.version(), DataContract::V1(v1) => v1.version(), } } // ... every method follows this pattern } }
For V1-only fields, the implementation gracefully handles V0 contracts:
#![allow(unused)] fn main() { impl DataContractV1Getters for DataContract { fn groups(&self) -> &BTreeMap<GroupContractPosition, Group> { match self { DataContract::V0(_) => &EMPTY_GROUPS, // static empty map DataContract::V1(v1) => &v1.groups, } } fn tokens(&self) -> &BTreeMap<TokenContractPosition, TokenConfiguration> { match self { DataContract::V0(_) => &EMPTY_TOKENS, // static empty map DataContract::V1(v1) => &v1.tokens, } } fn created_at(&self) -> Option<TimestampMillis> { match self { DataContract::V0(_) => None, DataContract::V1(v1) => v1.created_at, } } } }
This is a deliberate design choice. When code asks a V0 contract for its groups, it gets an empty map rather than an error. When it asks for a timestamp, it gets None. The calling code does not need to know or care which version it is working with -- it just checks whether the Option has a value or whether the map is empty.
Serialization Strategy
Data contracts use a two-step serialization approach. They are first converted to a DataContractInSerializationFormat (a common intermediate representation), then serialized to bytes using bincode with big-endian encoding:
#![allow(unused)] fn main() { impl PlatformSerializableWithPlatformVersion for DataContract { fn serialize_to_bytes_with_platform_version( &self, platform_version: &PlatformVersion, ) -> Result<Vec<u8>, ProtocolError> { let serialization_format: DataContractInSerializationFormat = self.try_into_platform_versioned(platform_version)?; let config = bincode::config::standard() .with_big_endian() .with_no_limit(); bincode::encode_to_vec(serialization_format, config) .map_err(|e| PlatformSerializationError( format!("unable to serialize DataContract: {}", e) )) } } }
This intermediate format is important because serialization versions and code structure versions are independent. A contract serialized as V1 ten years ago must still be deserializable, even if the code structures have evolved to V5 by then. The serialization format acts as the bridge.
There is also versioned_limit_deserialize, which imposes a size limit and always performs full validation -- this is used for data coming from untrusted sources (anything not from Drive's own storage).
Evolving a Contract: Adding Required Fields
Contract updates are deliberately conservative: existing documents must stay valid and their stored bytes must stay readable, so most schema changes that would break either are rejected. Historically that froze the required set of a document type in both directions — requiredness is baked into the document wire format (required properties serialize raw, optional ones carry a presence flag), so changing it would desynchronize every stored document's bytes from the schema used to read them.
From protocol v14, an update may add a brand-new required property by annotating it with requiredSince equal to the contract version the update creates:
"properties": {
"newField": { "type": "string", "maxLength": 63, "position": 4, "requiredSince": 3 }
},
"required": ["existingField", "newField"]
The rules, enforced by consensus (DataContractInvalidRequiredFieldsUpdateError, code 10276, on violation):
- The annotation must name exactly the new contract version — requiredness can be neither pre-scheduled for a future version nor backdated.
- Only brand-new properties can become required. Promoting an existing (optional) property is still rejected, as is removing anything from
requiredor touching an existingrequiredSinceannotation. - On contract creation,
requiredSincemay only be1. - Annotations sit on top-level properties listed in
required; nested properties cannot carry them.
What happens to data:
- Existing documents are grandfathered. Each document carries a contract version stamp recording the contract version its bytes conform to (see Document Serialization); a document stamped below a property's
requiredSincemay omit that property and still reads, transfers, and deletes normally. - New writes are held to the new schema. Creates must supply the property; replaces re-supply full content, so replacing a grandfathered document requires the new property and re-stamps the document at the current version — lazy migration, one document at a time.
- Indexes are unaffected because index additions on update remain banned — a newly added required field cannot be indexed retroactively (there is no backfill).
Rules and Guidelines
Do:
- Always access fields through the accessor traits (
DataContractV0Getters,DataContractV1Getters), not by pattern-matching on the enum variant. - Handle V1-only fields being
Noneor empty when the contract might be V0. - Use
document_type_for_name()to retrieve document types -- it returns a proper error if the name does not exist.
Do not:
- Use
as_v0()/as_v1()in production code unless you genuinely need version-specific behavior. The trait accessors are the right abstraction. - Assume
into_latest()exists outside of tests -- it is#[cfg(test)]only. - Forget that serialization versions persist forever. If you add a new field, old serialized contracts will not have it, and your deserialization must handle that gracefully.
- Mutate a contract's
idafter creation -- it is derived from the creation transaction and must remain stable.
Contract Groups
An application often outgrows one data contract. Indexes cannot be added to a document type after the contract is registered, so a project that grows tends to ship a second contract beside the first rather than reshape the original. A token is usually best kept in a contract of its own, so its change-control groups and supply rules are not entangled with an application's document types. And a project that publishes a version 2 contract keeps version 1 alive for the documents already stored in it. The result is a family of contracts under one owner, and before protocol version 14 nothing in consensus state said they belonged together. A client could guess from the owner identity, but that identity might own unrelated contracts too, and a guess cannot be proved.
A contract group is the answer. It is an identity-owned set of contracts, contract document types and contract tokens. It is registered and grown through ordinary data contract create transitions, stored under its own root tree, and readable with one GroveDB proof. Anything that later needs to act on "the set as a whole", from an SDK deciding which contracts to preload to a future identity key bound to every contract in a group, gets a consensus-level answer to "which contracts are in this set".
Not the same thing as
groups. A data contract declaresgroupsfor token change control (see Data Contracts). Those are multi-party sets of identities inside one contract, with voting power and a required threshold. A contract group is a set of contracts owned by identities. The two share a word and nothing else. TheSystemLimitsfield that bounds a change-control group's members was renamed frommax_contract_group_sizetomax_group_member_countin the same protocol version so the code does not confuse them either.
The Model
Three facts define a contract group:
- One identity owns it. The owner is the identity that signed the registering transition, and it is never on the wire. It can stand alone or name up to sixteen admins in the registration, existing identities other than itself that may add members alongside it. Admins act alone: there is no threshold and no vote.
- Its members are parts of contracts. A member is a whole contract, one document type of a contract, or one token of a contract. A contract can only enrol itself. Memberships are declared by the create transition of the contract that joins, never by a third party and never for someone else's contract.
- It is append-only. Memberships are recorded when the member contract is created. There is no update path and no leaving. Contracts are never deleted either. Together these two facts are what let the storage layout use plain references safely, as the storage section explains.
A group is registered inside a data contract create transition. The same transition may enrol the contract it creates, and later create transitions signed by any owner add more members. A group may also be registered empty and filled later.
How a Group Gets Its Id
The group id is never on the wire. It is derived from the registering identity and the transition's identity nonce, in packages/rs-dpp/src/contract_group/mod.rs:
#![allow(unused)] fn main() { pub const CONTRACT_GROUP_ID_DOMAIN: &[u8] = b"contract_group"; pub fn generate_contract_group_id( owner_id: &Identifier, identity_nonce: IdentityNonce, ) -> Identifier { let mut bytes = CONTRACT_GROUP_ID_DOMAIN.to_vec(); bytes.extend_from_slice(owner_id.as_slice()); bytes.extend_from_slice(&identity_nonce.to_be_bytes()); Identifier::from(hash_double(bytes)) } }
Compare the id of the contract created by the same transition: hash_double(owner_id || identity_nonce). The domain prefix keeps the two from ever colliding, and because both inputs are known before signing, a client knows the group id and the contract id before it broadcasts. That is what makes the one-transition case work: a create transition can register a group and, in the same membership list, join it by the id it is about to receive.
The Types
Everything a transition carries about contract groups lives in packages/rs-dpp/src/contract_group/mod.rs. There are four wire types and one stored type.
#![allow(unused)] fn main() { /// Who owns a contract group, and so who may add members to it. pub enum ContractGroupOwner { SingleOwner(Identifier), OwnerAndAdmins { owner: Identifier, admins: BTreeSet<Identifier>, }, } /// What part of the contract being created joins a contract group. pub enum ContractGroupMember { Contract, DocumentType(DocumentName), Token(TokenContractPosition), } /// A declaration that a part of the created contract joins a contract group. pub struct ContractGroupMembership { pub contract_group_id: Identifier, pub member: ContractGroupMember, } /// The registration of a new contract group. The owner is the signer and is not on the wire. pub struct ContractGroupRegistration { pub admins: BTreeSet<Identifier>, pub name: Option<String>, pub description: Option<String>, } }
The owner is deliberately absent from the registration: the only valid value would be the transition's signer, so carrying it would add thirty-two signed bytes and an error for the one mistake that field could express. The admins are a BTreeSet, so a duplicate admin cannot be expressed and the encoding is canonical.
ContractGroupOwner is the stored form of ownership, built from the signer and the registration when the transition becomes an action: an empty admin set stores SingleOwner, anything else OwnerAndAdmins. It has the helpers validation needs: owner_id(), admin_ids(), admin_count() and may_add_members(&identity_id), which is true for the owner and for every admin.
The stored type wraps the owner with a version envelope, because it is persisted and must remain decodable forever:
#![allow(unused)] fn main() { #[platform_serialize(unversioned)] pub enum ContractGroupInfo { V0(ContractGroupInfoV0), } pub struct ContractGroupInfoV0 { pub owner: ContractGroupOwner, pub name: Option<String>, pub description: Option<String>, } }
ContractGroupInfo derives PlatformSerialize, PlatformDeserializeTrusted and PlatformDeserializeUntrusted (see Derive Macros). The trusted decoder reads this node's own GroveDB; the untrusted one decodes bytes that arrived inside a proof. From<ContractGroupRegistration> builds the V0 variant, so the transition never carries a version tag for the info: Drive chooses the stored version.
Two details of the wire types are easy to miss. Both ContractGroupRegistration and ContractGroupMembership implement JsonSafeFields in packages/rs-dpp/src/serialization/json/safe_fields.rs, which the json_safe_fields derive on the transition requires for every field type. And the ContractGroupInfo module imports ProtocolError at the top even though no code in it names the type, because the PlatformSerialize derive expands to code that does.
Version 1 of the Create Transition
DataContractCreateTransition gains a second variant. Version 0 is unchanged and stays valid; version 1 is version 0 plus the two contract group fields. From packages/rs-dpp/src/state_transition/state_transitions/contract/data_contract_create_transition/v1/mod.rs:
#![allow(unused)] fn main() { pub struct DataContractCreateTransitionV1 { pub data_contract: DataContractInSerializationFormat, pub identity_nonce: IdentityNonce, pub contract_group: Option<ContractGroupRegistration>, pub contract_group_memberships: Vec<ContractGroupMembership>, pub user_fee_increase: UserFeeIncrease, #[platform_signable(exclude_from_sig_hash)] pub signature_public_key_id: KeyID, #[platform_signable(exclude_from_sig_hash)] pub signature: BinaryData, } }
Both new fields sit inside the signed payload. A plain contract creation in version 1 carries contract_group: None and an empty membership list, which is exactly what the TryFromPlatformVersioned<CreatedDataContract> conversion produces, so existing callers that build a transition from a contract keep working and simply emit version 1 under protocol version 14.
In JSON the transition looks like this (identifiers abbreviated):
{
"$formatVersion": "1",
"dataContract": { "...": "..." },
"identityNonce": 7,
"contractGroup": {
"admins": ["GWRSAVFM…S31Ec"],
"name": "cardgame",
"description": "Rules, marketplace and token contracts of the card game"
},
"contractGroupMemberships": [
{ "contractGroupId": "8sJ6Rk…Q2mV", "member": "contract" },
{ "contractGroupId": "4hYb2N…kW9p", "member": { "documentType": "listing" } },
{ "contractGroupId": "4hYb2N…kW9p", "member": { "token": 0 } }
],
"userFeeIncrease": 0,
"signaturePublicKeyId": 1,
"signature": "…"
}
The first membership joins the group this very transition registers, by its derived id. The other two join a group registered earlier, one the signer owns or administers. A group with a single owner leaves admins out or empty; the owner never appears, since it is the signer. Both group fields may be omitted from a version 1 object, which then reads as a plain creation.
Version Bounds
Which variant a client may send is governed by STATE_TRANSITION_SERIALIZATION_VERSIONS_V3, the table protocol version 14 selects (see Feature Versions):
#![allow(unused)] fn main() { contract_create_state_transition: FeatureVersionBounds { min_version: 0, max_version: 1, default_current_version: 1, }, }
Below protocol version 14 a version 1 transition has to be refused before anything looks inside it, and the bounds table above does not do that on a node: only client-side code consults it. The gate is StateTransition::active_version_range, which answers 14 to the latest version for a version 1 create, so deserialize_from_bytes_untrusted_in_version fails with StateTransitionIsNotActiveError on a protocol version 13 node. Without that arm an upgraded node would create the contract, charge the fee and silently drop the group data while a pre-upgrade binary rejected the same bytes. At protocol version 14 the default moves to 1, which is why the factory test in packages/rs-dpp/src/data_contract/factory/v0/mod.rs compares the version of the transition it builds against default_current_version rather than a literal.
Accessors
Code reads the new fields through DataContractCreateTransitionAccessorsV1, following the same pattern the Data Contracts chapter describes for DataContractV1Getters:
#![allow(unused)] fn main() { pub trait DataContractCreateTransitionAccessorsV1 { fn contract_group(&self) -> Option<&ContractGroupRegistration>; fn contract_group_id(&self) -> Option<Identifier>; fn contract_group_memberships(&self) -> &[ContractGroupMembership]; } }
A version 0 transition answers None, None and an empty slice. Validation and Drive never match on the variant; they ask the accessors and treat "no group data" as the ordinary case. contract_group_id() derives the id from the contract's owner and the transition's nonce, so nothing downstream recomputes the hash by hand.
Validation
Contract group checks slot into the existing validation pipeline at two stages, and the split between them is the important design decision. Everything that can be decided from the transition alone runs in basic structure, before the signature is checked, and costs nothing. Everything that needs a state lookup runs in state validation, after the signer is authenticated, and is paid for. Without that split an attacker could probe which group ids exist for free.
Basic Structure (Unpaid)
contract_group_basic_structure_error in packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/data_contract_create/basic_structure/v2/mod.rs runs after the existing contract checks and returns the first violation it finds, in this order:
- If the transition registers a group, its
admins, when any, must number at mostmax_contract_group_admins(16) and must not include the signer, who is the owner. name, when present, must be 1 tomax_contract_group_name_length(64) characters.description, when present, 1 tomax_contract_group_description_length(256). Lengths count characters, not bytes.- The membership list must hold at most
max_contract_group_memberships_per_contract(16) entries. - For each membership: a
DocumentTypemember must name a document type of the created contract, and aTokenmember must name a token position the contract defines. - No
(group id, member)pair may repeat. - A document type or token membership is redundant, and rejected, when the whole contract joins the same group in the same transition.
It is a free function rather than a method on the transition because drive-abci cannot add inherent methods to a type defined in dpp; the orphan rule forbids it.
State (Paid)
validate_contract_groups_against_state in .../data_contract_create/state/v1/mod.rs runs after the version 0 state checks (the contract must not already exist) and after the refersTo reference validation. It bills every lookup on the execution context and stops at the first failure:
- The group the transition registers must not exist yet. Its id was derived when the transition became an action, and that double hash is billed here, as the contract id derivation is.
- Every admin the registration names must be an existing non-masternode identity, the check change-control group members get; a missing one fails with
ContractGroupAdminNotFoundError. Memberships are creation-only, so a mistyped admin could never be replaced. - Every group a membership names must exist and must have the signer as its owner or one of its admins. The group registered by this same transition is skipped, since the signer owns it by construction, and each other group is fetched once however many memberships name it.
A failure here returns a BumpIdentityNonceAction carrying the errors: the identity pays for the lookups and its nonce advances, exactly as for any other paid validation failure.
The Errors
| Code | Error | Stage |
|---|---|---|
| 10360 | ContractGroupMembershipsOverLimitError | structure |
| 10361 | DuplicateContractGroupMembershipError | structure |
| 10362 | RedundantContractGroupMembershipError | structure |
| 10363 | ContractGroupMemberNotInContractError | structure |
| 10364 | InvalidContractGroupAdminsError | structure |
| 10366 | InvalidContractGroupNameLengthError | structure |
| 10367 | InvalidContractGroupDescriptionLengthError | structure |
| 41000 | ContractGroupAlreadyExistsError | state |
| 41001 | ContractGroupNotFoundError | state |
| 41002 | IdentityNotContractGroupOwnerOrAdminError | state |
| 41003 | ContractGroupAdminNotFoundError | state |
The basic errors live in packages/rs-dpp/src/errors/consensus/basic/contract_group/ and the state errors in .../consensus/state/contract_group/. Both sets were appended at the tail of their enums; StateError has a frozen-discriminant test that pins the four new variants, so an insertion before them fails it. Code 10365 is unassigned: the registrant-not-owner rule it served became inexpressible once the owner left the wire. The basic codes follow the change-control group range as their own block, and the state codes open a new hundred, 41000 to 41099, rather than borrowing from the identity range. See Error Codes for the code ranges.
From Transition to Action to Operations
The transform into action step gets a version 1 action to match the version 1 transition:
#![allow(unused)] fn main() { pub struct DataContractCreateTransitionActionV1 { pub data_contract: DataContract, pub identity_nonce: IdentityNonce, pub user_fee_increase: UserFeeIncrease, pub contract_group: Option<(Identifier, ContractGroupInfo)>, pub contract_group_memberships: Vec<ContractGroupMembership>, } }
Two things happen in the transformer (packages/rs-drive/src/state_transition_action/contract/data_contract_create/v1/transformer.rs). The registration becomes a ContractGroupInfo, the stored form, with the signer as owner, and the group id is derived once and carried alongside it, so nothing after this point needs the transition's owner and nonce; state validation reads the id from the action rather than hashing again. The BumpIdentityNonceAction transformer gained matching version 1 arms so a paid failure can still be turned into a nonce bump.
Converting the action into Drive operations is where ordering matters. The version 1 arm of into_high_level_drive_operations emits, in this order:
UpdateIdentityNonceandUpdateIdentityContractNonce, as version 0 does.ApplyContractfor the contract itself.RegisterContractGroup { contract_group_id, info }, if the transition registers a group.AddContractGroupMemberships { contract_id, memberships }, if it declares any.
The contract is applied before anything touches the group trees, and the registration comes before the memberships, so a contract may join the group its own transition registers: by the time the membership operation runs, the group's subtrees exist. Both operations are variants of ContractGroupOperationType in packages/rs-drive/src/util/batch/drive_op_batch/contract_group.rs, and they resolve to low-level GroveDB operations through the same DriveLowLevelOperationConverter machinery every other high-level operation uses (see Batch Operations).
Storage
Contract groups get their own root tree, RootTree::ContractGroups, at key 124. The number matters: a root key becomes a child of some existing node in the root Merk, and every write under that node's subtree then rewrites a slightly larger node. Key 124 attaches under Versions (120), which only the block-level app-version update writes, so no fee-bearing transition pays for it. A lower free key such as 68 would have attached under the asset-lock outpoints node and raised every identity create and top-up fee by 1480 credits for the life of the chain; the four conversion impls in packages/rs-drive/src/drive/mod.rs and the KnownPath mapping in the batch module were extended for it. Path helpers and the single-byte subtree keys live in packages/rs-drive/src/drive/contract_groups/paths.rs.
[124] ContractGroups
├── [0] Groups
│ └── <contract group id>
│ ├── [0] Info -> Item(bincode ContractGroupInfo { owner, name?, description? })
│ ├── [1] Contracts -> <contract id> -> Item([])
│ ├── [2] DocumentTypes -> <contract id || document type name> -> Item([])
│ └── [3] Tokens -> <contract id || token position, u16 BE> -> Item([])
└── [1] Members
└── <contract id>
├── [0] Groups -> <contract group id> -> Reference to Groups/<group>/[1]/<contract id>
├── [1] DocumentTypes -> <document type name> -> <contract group id> -> Reference
└── [2] Tokens -> <token position> -> <contract group id> -> Reference
Groups is the forward store: everything about one group under its id. Members is the backwards index: everything one contract belongs to, under the contract id. Every leaf on the forward side is an empty item; the key carries all the information. Document type and token members sit on one level under a composite key, the contract id followed by the UTF-8 name or the two big-endian position bytes, rather than under a subtree per contract. The keys still sort by contract, and a flat level pages with a plain range after the cursor key. A subtree per contract would have forced a continuation page to descend into the cursor's contract, and GroveDB charges an empty descent against the page limit, which shortened every continuation page and ended a limit-one walk early. The decoders rebuild typed values from keys alone.
Why Plain References
The backwards entries are GroveDB Reference elements of type UpstreamRootHeightReference(1, …): keep the first path segment (the root tree key) and append the forward path, one hop. They are not GroveDB bidirectional references. Bidirectional references buy automatic cleanup when the target moves or disappears, at a cost on every write. Here neither can happen: memberships are append-only and contracts are never deleted, so a reference can never dangle. Plain references are cheaper and the invariant holds by construction.
Why Members Is Keyed by Contract First
The Members side is laid out so that "is document type D of contract C in group G" is one point lookup at [124, 1, C, 1, D, G], and the forward side answers it at [124, 0, G, 2, C || D]. That is the shape ContractBounds::ContractGroup on identity keys needs: for every member of a batch, one existence check under the contract the member belongs to. Today the batch transformer reads the contract's whole Members entry (at most max_contract_group_memberships_per_contract entries) into the action, for a batch signed by a group-bound key only, and the check runs in memory; the point lookup is there when a consumer wants it.
Writing
Registration (insert_contract_group_operations_v0) inserts the group's tree under Groups with batch_insert_empty_tree_if_not_exists, and if the tree was already there returns CorruptedDriveState. State validation guarantees the group is new, so hitting that branch means the two disagree, which is exactly what a corrupted-state error is for. It then writes the Info item and the three empty member subtrees.
Memberships (insert_contract_group_memberships_operations_v0) write one empty item on the forward side and one reference on the backwards side per membership, creating the intermediate trees on both sides as needed, each with an if-not-exists insert so a tree left by an earlier declaration of the same contract is kept rather than overwritten once memberships can be added after creation. Two memberships often need the same parent tree, for instance two document types of the same contract joining the same group both need Groups/<G>/DocumentTypes/<C>. GroveDB rejects a batch that operates twice on one slot under batching consistency verification, so the inserter keeps a HashSet of tree paths it has already created in this batch and inserts each parent once.
Cost Estimation
Both inserts have an estimation twin under estimated_costs/ that declares EstimatedLayerInformation for every layer the write can touch (see Cost Tracking): the root, the ContractGroups tree, the Groups and Members levels, a group's own tree as a mix of three subtrees and one item, and so on down. Three constants size the unknowns: an info item is estimated at 1024 bytes (a sixteen-owner group with a full-length name and description stays under it), a backwards reference at 128 bytes, and a document type name key at 16 bytes. The Drive test should_build_the_same_operations_for_estimation_and_for_apply pins the estimation path to the apply path so the two cannot drift apart silently.
Creating the Trees
One helper, Drive::insert_contract_groups_structure, creates the root tree and its two subtrees. A fresh chain calls it from create_initial_state_structure_v4 (packages/rs-drive/src/drive/initialization/v4/mod.rs), which DRIVE_VERSION_V9 selects; a chain upgrading to protocol version 14 calls it from transition_to_version_14, the migration hook the Versioned Dispatch chapter describes. Sharing the helper is what keeps a node started at 14 and a node upgraded to it byte-identical under this key, the same reason the shielded pool has insert_shielded_pool_structure. Both subtrees are created up front so a registration only ever writes under Groups and a membership only under Groups and Members; no write path has to check whether the top of the tree exists.
Reading and Proving
A group can be joined by any number of contracts, so nothing reads or proves a whole group at once. Drive splits reading into the group's information, a single item, and its members, read one kind at a time in pages bounded by a limit. Everything is versioned through DriveContractGroupMethodVersions:
fetch_contract_group_info(group_id)reads only theInfoitem. State validation uses its_with_feevariant, which returns theFeeResultfor the lookup alongside the result.prove_contract_group_infoandverify_contract_group_infodo the same through a proof, withNonefor a group that is provably absent.fetch_contract_group_members(group_id, &query, limit)returns oneContractGroupMembersPageof one kind. TheContractGroupMembersQuerynames the kind,Contracts,DocumentTypesorTokens, and carries astart_aftercursor: a contract id alone, or a contract id with a document type name or a token position. Entries come back in key order, at mostlimitof them, andpage.next_query()is the query for the page after it,Noneonce a page is empty.limitmust lie between 1 and the node'smax_query_limit, on the fetch side and on the proof side, so no page and no proof grows with the size of the group.prove_contract_group_membersandverify_contract_group_memberstake the same query and limit.fetch_contract_group_memberships_for_contract(contract_id)returns aContractGroupMembershipsForContract: the groups the whole contract joined, the groups each document type joined and the groups each token joined. It needs no limit: memberships are recorded at creation only and capped per transition, so a contract belongs to at most sixteen.prove_contract_group_memberships_for_contractandverify_contract_group_memberships_for_contractare its proof pair.
Three path queries in packages/rs-drive/src/drive/contract_groups/queries.rs drive all of them. contract_group_info_query asks for the one info key under the group's tree. contract_group_members_query reads one kind's level: the whole level on a first page, or a range after the cursor's key on a continuation, where the key is the contract id for Contracts and the contract id followed by the name or position for the other two kinds. There is no subquery to descend into, so a page can end anywhere and the next page resumes exactly after it. contract_group_memberships_for_contract_query reads a contract's Members entry with a conditional subquery per kind. The same PathQuery is used to fetch and to prove, so what a node reads locally and what a client verifies are the same set of elements.
Verification lives in packages/rs-drive/src/verify/contract_groups/. Each verify function rebuilds its result from the proved (path, key, element) triples with the same decoders the fetches use, in types.rs. Member pages carry no data in their elements: every entry is read from its path and key. The info item is the one thing that decodes bytes, and the verifier uses the untrusted ContractGroupInfo decoder because proof bytes came from someone else, where the fetch uses the trusted one.
Two GroveDB behaviours shape this code and are worth knowing before you touch it:
- A path query over a tree that does not exist is an error, not an empty result. The fetches call
grove_has_rawon the group's or contract's tree first and returnNoneor an empty result when it is absent. - GroveDB cannot build a proof inside an open transaction.
prove_*withSome(&transaction)fails withNotSupported. Tests commit first and prove withNone.
The DAPI Queries
Three DAPI queries sit on top of these proofs, one per path query, so a client verifies exactly the elements a node read. Handlers live under packages/rs-drive-abci/src/query/contract_group_queries/ and are versioned by contract_group_queries in the query version table.
getContractGroupInfo(contractGroupId)answers with the group's owner, admins (empty for a single owner), name and description, or with no result when no group has the id. The proved form proves the info item or its absence;rs-drive-proof-verifierverifies it into aContractGroupInfo.getContractGroupMembers(contractGroupId, members, limit)answers with one page of one kind.membersis aoneofthat names the kind and carries its cursor:contracts { start_after: contract id },document_types { start_after: contract id + name }ortokens { start_after: contract id + position }.limitdefaults to and is capped bymax_returned_elements(100) on the node and in the verifier alike, because a client rebuilds the page query from the request it sent: an omitted limit has to mean the same page everywhere. The response is the page in the same kind, or a proof, verified into aContractGroupMembersPage.getContractGroupsForContract(contractId)answers with the contract's memberships as a whole, by document type name and by token position, empty lists when it belongs to no group. The proved form proves that too, verified into aContractGroupMembershipsForContract.
In rs-sdk, ContractGroupInfo, ContractGroupMembersPage and ContractGroupMembershipsForContract implement Fetch and FetchUnproved (packages/rs-sdk/src/platform/contract_groups.rs); a ContractGroupMembersPageQuery carries the group, kind, cursor and limit, and its after(&page) is the query for the next page. The wasm-sdk exposes getContractGroupInfo, getContractGroupMembers (a page object whose nextStartAfter is the next cursor) and getContractGroupsForContract, each with a WithProofInfo twin, and js-evo-sdk wraps them as sdk.contractGroups.
Versioning Touchpoints
Contract groups arrived with protocol version 14, which was unreleased while they were built, so every table below was edited in place rather than copied into a new version (the rule in Coding Conventions).
| Table | What changed |
|---|---|
STATE_TRANSITION_SERIALIZATION_VERSIONS_V3 | contract_create_state_transition max version 0 → 1, default 0 → 1 |
DRIVE_STATE_TRANSITION_METHOD_VERSIONS_V4 | data_contract_create_transition converter 0 → 1 (emits the group operations) |
DRIVE_VERSION_V9 | create_initial_state_structure 3 → 4 (creates the root tree) |
DRIVE_ABCI_VALIDATION_VERSIONS_V10 | the contract create basic_structure v2 and state v1 modules gained the group checks |
DriveMethodVersions | new contract_group: DriveContractGroupMethodVersions slot (insert, fetch, prove, cost estimation), DRIVE_CONTRACT_GROUP_METHOD_VERSIONS_V1, every method at 0 |
DriveVerifyMethodVersions | new contract_group: DriveVerifyContractGroupMethodVersions slot |
SystemLimits | four new limits, plus the max_group_member_count rename |
The four limits:
| Limit | Value |
|---|---|
max_contract_group_memberships_per_contract | 16 |
max_contract_group_admins | 16 |
max_contract_group_name_length | 64 characters |
max_contract_group_description_length | 256 characters |
Numbers go in SystemLimits and nowhere else, so validation reads them from platform_version.system_limits rather than from constants in the validation module.
Fees
Nothing about contract groups has a special fee. Storage is charged at the standard rates for what is written: the info item and three empty subtrees for a registration, one empty item plus one reference per membership, plus whichever intermediate trees a new contract needs on the backwards side. State validation bills each group lookup as a precalculated operation. A transition that fails basic structure costs nothing; one that fails state validation pays for the lookups it caused and has its identity nonce bumped.
Keys Bound to a Group
An AUTHENTICATION key can carry ContractBounds::ContractGroup { id } instead of a contract bound (protocol version 14; see docs/protocol/contract-bound-authentication-keys.md for the full rules). The key may then sign only batches, and a batch member on contract C is inside the bounds when C, the member's document type, or the member's token is a member of the group. The batch transformer (v2) first resolves the signer's bounds from the identity the signature was validated against (handed over through StateTransitionSignerAwareActionTransformer, since the unversioned transformer trait is not to be changed). Only when the signing key is an AUTHENTICATION key bound to a contract group does it read the group memberships of every contract the batch touches into the action with fetch_contract_group_memberships_for_contract_with_fee, keeping the fee of the read next to them the way a contract's fetch info does. Batch advanced-structure validation v1 then judges the key's bounds from the action alone: it checks contains for the whole-contract membership and for the exact one, and bills that fee. A batch signed by any other key is transformed exactly as before: no extra read, no extra fee, nothing new that can fail. A miss is the same paid ContractBoundedKeyOutOfBoundsError a contract bound produces. Registration goes through contract-bounds validation v2, which requires AUTHENTICATION, a non-MASTER level and an existing group (one billed fetch_contract_group_info_with_fee); encryption and decryption keys cannot be group-bound, since their bounds are opt-in per contract and a group has nothing to opt in with. Below protocol version 14 a transition carrying a group-bound key is not active (active_version_range), so it is rejected at decode without a charge, the way a binary that cannot decode the variant rejects it.
Drive keeps group-bound keys in the identity's contract-info level under the group id, next to contract ids, through the ContractGroupBased apply info and the ContractGroupBoundKey key request. Nothing new is written to the ContractGroups tree. Two consequences worth knowing: memberships are append-only, so a group-bound key's reach grows with the group; and IdentityCreateFromShieldedPool refuses group-bound keys (shielded-proof validation v1), because its sighash preimage layout predates them.
What Is Not There Yet
The first protocol version 14 change ships the consensus core only. Known gaps, all deliberate:
- A group owned by a contract's change-control group. A third
ContractGroupOwnervariant, where the owner is aGroupinside a contract rather than a set of identities, was specified and left out. The simple reading, "the signer is a member of that group with power above zero", is the recommended first step; full required-power group actions can follow. - Updating or leaving. Memberships are creation-only. A later
DataContractUpdateTransitionversion would be needed to add members from an existing contract or to remove any. - Relaxing redundancy. A document type membership is refused when the whole contract already joins the same group. That could be allowed if a consumer wants the explicit entry.
- An owners index. There is no
identity → groups it ownstree. Finding the groups an identity owns means scanningGroups. - Creation surfaces. The DAPI queries and the Rust, wasm and JavaScript read bindings exist (see The DAPI Queries), but registering a group or joining one still needs a hand-built
DataContractCreateTransitionV1: the wasm-dpp JSON fields and the JavaScript, Kotlin and Swift contract-create options follow separately.
Tests
Coverage sits at the three layers the feature touches (see Unit Tests):
- dpp (
contract_group/mod.rs): the id derivation differs from the contract id and depends on both inputs; both owner kinds resolve membership; the stored info round-trips through bincode with the untrusted decoder. - drive (
drive/contract_groups/tests.rs): the root tree exists in the initial structure at protocol version 14 and not before; a group can be registered and proved present or absent; memberships land on both sides and prove; members page through with a cursor, a limit of one still reaches every entry past a contract with nothing left, and a zero or over-the-maximum page size is refused on both the fetch and the proof side; estimation and apply build the same operations; registering an existing group is refused. - drive-abci queries (
query/contract_group_queries/*/v0/mod.rs): each handler rejects a malformed id, cursor or limit; an absent group or a contract in no group answers empty, unproved and proved; members page through with a cursor and come back in the requested kind; every proof verifies through theDrive::verify_*twin. - drive-proof-verifier (
proof/contract_groups.rs,unproved.rs): request parsing fails before verification on a missing version, a short id, a missing kind or an out-of-bounds limit; unproved responses decode both owner kinds, reject a page of another kind, and read memberships into sets. - drive-abci (
data_contract_create/contract_group_tests.rs): end-to-end throughprocess_raw_state_transitions, covering register-and-join in one transition, an owner adding a later contract by document type and token, joining a group the identity does not own as a paid failure, joining an unknown group, a group with admins accepting the owner and each admin and refusing outsiders, a registration naming an unknown admin rejected as a paid failure, a version 0 create still processed at the latest version, and the full set of malformed registrations and memberships rejected in basic structure. The protocol upgrade hook has its own test proving the trees exist and accept a registration after the 13 to 14 transition.
cargo test -p dpp --all-features -- contract_group
cargo test -p drive --lib -- contract_groups
cargo test -p drive-abci -- contract_group data_contract_create
cargo test -p drive-abci --lib -- query::contract_group_queries
cargo test -p drive-proof-verifier --lib -- contract_groups
Rules and Guidelines
Do:
- Read contract group data through
DataContractCreateTransitionAccessorsV1and treat "no registration, no memberships" as the normal case. Never match onV0versusV1to find out. - Derive the group id with
generate_contract_group_idorcontract_group_id(). Never hash by hand and never accept an id from the wire. - Keep lookups in state validation and everything else in basic structure. A check that needs Drive must be paid for.
- Use the path helpers in
paths.rs. The subtree keys are single bytes and easy to transpose. - Read members in pages with a limit. Never add a query that enumerates a whole group; the info item is the only thing read without a bound.
Do not:
- Confuse contract groups with a contract's change-control
groups. Different types, different limits, different trees. - Write to the
Membersside without the matching forward entry, or the other way round. Every membership is two writes and the decoders assume both exist. - Turn the backwards references into bidirectional references. The invariant that makes plain references safe (append-only, no deletion) is what the design relies on; if that invariant ever changes, the storage layout changes with it.
- Add a membership path outside a create transition. The model is creation-only until an update transition version says otherwise.
- Prove inside an open transaction, or path-query a tree you have not confirmed exists.
Contract Moderation
An application that stores user content needs a way to keep an abusive identity out. Before protocol version 14 nothing in consensus state could do that: the contract owner could delete nothing the user wrote, could stop nothing the user would write next, and a client-side blocklist bound nobody but the client that kept it. Every other user's node still accepted the identity's documents.
Contract moderation is the answer. A data contract may declare, in its config, that it keeps a banlist, a suspension list and/or a warning list of identities, and who may edit them. An identity on the banlist, or on the suspension list with a suspension that has not lapsed, cannot act on the contract at the document level: every document transition it signs against the contract is refused, paid, in the mempool and in a block. Token transitions are not affected. A warning bars nothing: it is a record, with a reason and a block time, of a step short of a suspension, which the warned identity and everyone else can read and which accumulates until a moderator clears it. The lists live under the contract's own subtree in Drive, are edited by one new state transition, and are readable with one GroveDB proof.
The Model
Four facts define a contract's moderation:
- The contract declares it.
DataContractConfigV2::moderationis an optionalContractModerationConfig { banlist, suspensions, warnings, moderators }. At least one list must be kept, unless a document type lets the moderators delete its documents (see Deleting Documents below) or keeps fields only they write (see Changing Document Fields below), in which case the declaration may keep none. Which lists a contract keeps is decided when it is created and never changes: an update can not make an unmoderated contract moderated, turn a second list on, or turn a list off (validate_config_update2,DataContractConfigUpdateError). Whoever writes documents under a contract knows from its first version whether and how they can be barred from it, which matters most where documents are assets: a ban also stops transfers and sales. And a list that is on may hold entries. Only the moderators may be changed by an update. - The owner moderates, alone or with a fixed set, or an elected team does.
ContractModeratorsisContractOwner,AppointedModerators(set)orElected(declaration); the third is its own section below. With the first two, a set is at mostSystemLimits::max_contract_moderators(16) identities. The owner may always moderate and need not be named; it may be named, and then counts toward the 16. Naming it changes nothing about authority. Every identity named must exist: the contract create, and the contract update for the identities it adds, look each one up in state and refuse, paid, withContractModeratorIdentityNotFoundError(41110). A moderator that does not exist can never sign, so naming one is a mistake, and catching it once at the declaration is cheaper than guarding every later reader of the set. Moderators act alone: there is no threshold and no vote. Neither the owner nor a moderator can be banned or suspended. An entry one of them already carries can still be lifted: a contract update may name as moderator an identity that is banned or suspended, the entry keeps binding it, and the owner or another moderator unbans or unsuspends it without demoting it first (never the identity itself: a moderation cannot target its own signer). - A ban lasts until an unban; a suspension lasts until a block time. A suspension names the block time, in milliseconds, at which it lapses. A lapsed suspension is not deleted by the clock: the first document transition of the identity that runs at or after that time executes normally and, in the same execution, sweeps the stale entry. An explicit unsuspend deletes it too. A ban supersedes a suspension: banning a suspended identity removes the suspension, and suspending a banned identity is refused.
- A warning is a record, not a bar. A warn adds one warning, the block time and a reason, to the identity's entry on the warning list; the entry holds every warning the identity carries, oldest first, up to
SystemLimits::max_contract_warnings_per_identity(16), past which a warn is refused until a clearWarnings deletes the entry whole. The document gate does not read the warning list, only the banlist and the suspension list (the warning list is read by a warn and a clearing alone); a ban leaves the warnings alone (they say how it came to that), and a banned or suspended identity may still be warned. What a warning is for is the client: an application shows the identity its warnings, and the moderators the count, before a suspension or a ban.
The Types
The declaration and the status live in packages/rs-dpp/src/data_contract/config/moderation/mod.rs:
#![allow(unused)] fn main() { pub struct ContractModerationConfig { pub banlist: bool, pub suspensions: bool, pub moderators: ContractModerators, pub warnings: bool, // last: a declaration stored before the warning list existed does not decode } pub enum ContractModerators { ContractOwner, AppointedModerators(BTreeSet<Identifier>), Elected(Box<ElectedModerators>), // see Elected Moderation below } pub enum ContractModerationList { Banlist, Suspensions, Warnings } pub struct ContractModerationReason { pub code: Option<u16>, pub text: String, pub documents: Vec<ContractModerationDocument>, // { document_type_name, document_id } pub reason_document_id: Option<Identifier>, // a `reason` of the moderation charters contract } pub struct ContractBan { pub reason: ContractModerationReason } pub struct ContractSuspension { pub until: TimestampMillis, pub reason: ContractModerationReason } pub struct ContractWarning { pub warned_at: TimestampMillis, pub reason: ContractModerationReason } pub struct ContractModerationStatus { pub ban: Option<ContractBan>, pub suspension: Option<ContractSuspension>, pub warnings: Vec<ContractWarning>, // oldest first, empty when none } }
ContractModerationConfig::lists names the lists a contract keeps, in tree key order (banlist, suspension list, warning list); barring_lists the ones whose entries bar, which is every list but the warning list: what the document gate reads and what a ban's proof covers.
Every ban, every suspension and every warning carries a reason, stored with the entry so that whoever reads the list reads why. The text is free: at most SystemLimits::max_contract_moderation_reason_length (1024) bytes of UTF-8, possibly empty. The code is reserved for the ban codes a contract may declare in a later protocol version. No contract declares any today, so it is expected to be None; a moderator may still write any u16 there, and nothing checks it against anything. The reasonDocumentId names a reason document of the moderation charters contract, the ground the action is taken on: a seated elected team's ban, suspension, warning or deletion must name one its proposal lists (see The seated team below); for every other moderator it is stored as written, looked up nowhere. A reason may also cite the documents it is about, the posts a warning or a ban is for, each by its document type name and its id: at most SystemLimits::max_contract_moderation_reason_documents (16), none twice, with a type name a contract could admit (InvalidContractModerationReasonDocumentsError, 10904, unpaid). Nothing looks them up: a cited document may have been deleted since, by its author or by a moderator whose deletion left a record, or may never have existed, and a client that wants to show it fetches it or its removal record. The moderator pays the storage of the reason, documents included, byte for byte, and gets it back when the entry is removed.
The config is DataContractConfig::V2, a new variant of the config's own bincode enum inside the contract. The config version follows the platform version, as V1 did from protocol version 9: from protocol version 14 every new contract carries a V2 config, moderated or not (CONTRACT_VERSIONS_V6 sets both max_version and default_current_version to 2), and an existing V1 contract moves to V2 with its next update. config_valid_for_platform_version lowers a V2 only where the platform version does not admit it, never because of what it declares. Lowering would drop a moderation declaration, and moderation can never be turned on later, so that case is refused rather than dropped: serializing a contract whose config declares moderation at a platform version below 14 (ensure_admitted_by_platform_version), and parsing a config value with a moderation key at such a version, both fail with ProtocolError::NotSupported. For the same reason the declaration refuses an unknown key instead of skipping it (deny_unknown_fields, and the moderators' $type map likewise): a misspelled suspensions would otherwise leave the contract without the list for good. A contract create or update carrying a V2 config is active from protocol version 14 only (StateTransition::active_version_range): before that a node rejects it at decoding, unpaid, exactly as a binary that cannot decode the V2 discriminant does, so upgraded and older nodes agree on every block before activation. The JSON shape of the moderators is a flat {"$type": "contractOwner"}, {"$type": "appointedModerators", "identities": [...]} or {"$type": "elected", ...} with the declaration's keys beside its $type, the style of AuthorizedActionTakers.
The Transition
ContractUserModerationTransition (type 24) carries one action:
#![allow(unused)] fn main() { pub struct ContractUserModerationTransitionV0 { pub owner_id: Identifier, // the moderator that signs pub data_contract_id: Identifier, pub identity_contract_nonce: IdentityNonce, pub action: ContractUserModerationAction, pub user_fee_increase: UserFeeIncrease, pub signature_public_key_id: KeyID, pub signature: BinaryData, } pub enum ContractUserModerationAction { Ban { identity_id, reason: ContractModerationReason }, Unban { identity_id }, Suspend { identity_id, until: TimestampMillis, reason: ContractModerationReason }, Unsuspend { identity_id }, Warn { identity_id, reason: ContractModerationReason }, ClearWarnings { identity_id }, DeleteDocument { document_type_name, document_id, reason: ContractModerationReason }, RestoreDocument { document_type_name, document: BinaryData }, } }
It is signed like a contract update: a CRITICAL authentication key without contract bounds, under the signer's contract-scoped nonce, and its minimum fee is the contract update floor. It activates with CONTRACT_USER_MODERATION_INITIAL_PROTOCOL_VERSION (14).
Validation
| Tier | What | Codes |
|---|---|---|
| Basic structure (unpaid) | the target is not the signer; a suspension ends at or before SystemLimits::max_contract_suspension_until (2^53 - 1 ms, the largest value JSON clients read exactly); the text of a ban's, a suspension's or a warning's reason is at most SystemLimits::max_contract_moderation_reason_length bytes (its code is not checked) and the documents it cites are at most max_contract_moderation_reason_documents, well named and distinct | 10901, 10700, 10903, 10904 |
| Signature and nonce | CRITICAL key, contract nonce | existing |
| Transform (state, paid) | the contract exists; it keeps the list the action edits; the signer is the owner or a moderator; the target of a ban, a suspend or a warn is neither; the target exists; the action fits the target's status | 41100-41106, 41109, 41117, 41118 |
The transform reads the contract and the target's status and refuses, paid, by bumping the signer's contract nonce. The action carries the status as read (and, for a warn, the warnings the target carries and the block time the new one is stamped with), so Drive edits the lists without reading them again, and the mempool, which transforms without a state validation stage, refuses with the same codes as a block. A suspend must end after the block time (41106). Suspending an identity that carries a suspension replaces it, longer or shorter. A warn is refused once the target carries max_contract_warnings_per_identity warnings (ContractUserWarningLimitReachedError, 41118), and a clearWarnings of an identity that carries none with ContractUserNotWarnedError (41117). A ban and a suspend read the barring lists the contract keeps; a warn and a clearing read the warning list alone.
The Document Gate
The gate sits in the batch transformer (a barred signer has every one of its transitions against the contract refused on its own, each with its nonce bump), transform_document_transitions_within_contract_v0, right after the contract is fetched, as its own versioned helper: contract_moderation_gate, selected by batch_state_transition.contract_moderation_gate (None up to protocol version 13, Some(0) from 14). That transformer is shared with every earlier protocol version, so the gate is a version-table fact there, not something inferred from contract data. A config that declares no moderation costs nothing: no read, no branch. Otherwise the transformer reads the owner's status on the lists the contract keeps, bills the read, and:
- banned: every document transition of the batch on that contract except a deletion, and a replace on a type declaring
retractedWhen, fails withContractUserBannedError(41107), paid, each with its contract nonce bump; - suspended and not lapsed: the same with
ContractUserSuspendedError(41108); - suspended and lapsed: the transitions go through and the batch action records the contract in
lapsed_suspensions(the identity is always the batch owner, so it is not stored and nothing can queue another identity's); the batch converter (documents_batch_transitiongeneration 1) appends one delete per contract.
Deletions (Delete and IndexOnlyDelete) are never refused: a barred identity can write nothing new, move nothing and sell nothing, but it may still take down what it wrote, under the document type's ordinary deletion rules. A batch of deletions alone carries on whole; in a mixed batch the deletions carry on next to the refusals. On a type whose documents the owner can not delete, the exit is a retraction instead: a type declaring retractedWhen has the gate let a barred signer's replaces through with its bar (ContractModerationRefusal::retraction_bar), and once the stored document is fetched the transformer (judge_barred_replace) refuses with that bar, paid, each replace whose written document does not meet the condition, judged as an immutable condition is. The type's other rules say what a retracted document may hold. Because the transformer runs in check_tx, a barred identity's documents never enter the mempool. The other party of a document transition is checked too: a transfer to a banned or live-suspended recipient, and a purchase from a banned or live-suspended seller, are refused with ContractModerationCounterpartyBarredError (41114), paid by the signer, so a barred identity collects neither assets nor proceeds on the contract. That read is billed to the batch; a counterparty's lapsed suspension is left for the counterparty's own next transition to sweep. No contract could declare moderation before protocol version 14, so older blocks replay unchanged through the same code.
The Errors
Basic, in their own band (10900-10949): InvalidContractModerationConfigError (10900), ContractModerationSelfTargetError (10901), DocumentActionFeesWithoutModerationError (10902, see Fee Pots and the Claim), ContractModerationReasonTooLongError (10903), InvalidContractModerationReasonDocumentsError (10904), InvalidContractModerationDocumentFieldsError (10905, a field change naming no field or a system property). State, in their own sub-band: ContractModerationNotEnabledError (41100), IdentityNotContractModeratorError (41101), ContractModerationTargetNotAllowedError (41102), ContractUserAlreadyBannedError (41103), ContractUserNotBannedError (41104), ContractUserNotSuspendedError (41105), ContractSuspensionNotInFutureError (41106), ContractUserBannedError (41107), ContractUserSuspendedError (41108), ContractModerationTargetNotFoundError (41109), ContractModeratorIdentityNotFoundError (41110, from the contract create and update, not from the moderation transition), ContractModerationCounterpartyBarredError (41114, from the document gate; 41111 to 41113 are reserved), ContractUserNotWarnedError (41117), ContractUserWarningLimitReachedError (41118), DocumentFieldNotChangeableByModeratorsError (41123) and DocumentModeratorFieldNotWritableError (41124) (see Changing Document Fields). A contract update that turns a list on or off is refused with the existing DataContractConfigUpdateError (40002). Elected moderation has its own band (41200-41299): ContractModeratedDocumentTypeNotYetUsableError (41200), ContractModerationAbilityNotGrantedError (41201), ModerationCharterAddedModeratorLimitReachedError (41202), ModerationReasonNotListedError (41203), and for the deletion of settled documents DocumentTypeNotDeletableOnceSettledError (41204), ContractModerationTeamNotSeatedError (41205), DocumentNotSettledError (41206), ContractTeamActionDoesNotExistError (41207), ContractTeamActionAlreadySignedError (41208), SettledDeletionNotRestorableError (41209), ContractTeamActionAlreadyCompletedError (41210), ContractTeamActionDocumentChangedError (41211) and ContractTeamMemberAddedAfterDocumentError (41212) (see Deleting Settled Documents). A discounted action fee the seated charter does not give is refused with DocumentActionFeeModeratorsShareMismatchError (40139), beside the other fee agreement errors.
Deleting Documents
A banlist keeps an identity out; it does not take down what the identity already wrote. A document type may let the contract's moderators do that:
"post": {
"type": "object",
"moderatorAbilities": { "delete": true },
"properties": { "text": { "type": "string", "maxLength": 280, "position": 0 } },
"additionalProperties": false
}
moderatorAbilities is a document type keyword of meta-schema v3, parsed by apply_moderator_abilities; its delete key is DocumentTypeV2::documents_can_be_deleted_by_moderators and its deleteWithin key documents_can_be_deleted_by_moderators_for. Its rules for deletion:
- The contract declares moderation. The keyword on a contract without a
moderationblock is refused (InvalidContractStructure, 10231): moderation can not be switched on later, so nobody could ever delete anything. In return amoderationblock may keep no list at all when at least one document type carries the keyword (ContractModerationConfig::validatetakes that fact from the contract): a contract can moderate content without moderating users. - It is fixed with the type. A contract update can not add the keyword to an existing document type or take it away (
DocumentTypeUpdateError, 40212): authors keep the rules they wrote under. A document type an update adds may carry it. - It is independent of
canBeDeleted, which rules what a document's own owner may do.canBeDeleted: falsewithmoderatorAbilities.delete: trueis a post its author can not retract and moderation can remove. - Some types can not carry it: one that keeps history (Drive refuses to delete such documents), an indexOnly one (there is no stored row to name by id), and one that restricts creation (its documents are the contract owner's, which no moderator may delete). Transferable and tradeable types may, and so may a type with a deletion token cost, which a moderator does not pay.
- It may come with a window.
moderatorAbilities.deleteWithin: 86400lets the moderators delete a document for that many seconds after its last modification ($updatedAt), and no longer: once block time is past$updatedAtplus the window the document is settled, and no moderator deletes it alone any more, the contract owner included (DocumentModerationWindowElapsedError, 41116); on a type that also setsdeleteSettled, the seated team of an elected contract may still delete it together (see Deleting Settled Documents). At exactly$updatedAtplus the window the deletion still passes; the document's own owner still deletes it ascanBeDeletedallows. Moderation acts on what was just written; it does not reach back alone into what has stood unchallenged. A replace or a price update moves$updatedAt, so new content opens the window again; a transfer or a purchase does not. The window needs the flag, at least one second, and the clock in the type'srequired, so that every document carries it:$updatedAt, or for a type withdocumentsMutable: false$createdAtinstead, since nothing modifies such a document after its creation. A type whose documents can be replaced must require$updatedAt: measured from creation alone, an author could wait the window out and then rewrite a post into something no moderator can remove. The transform reads$updatedAtand falls back to$createdAt; like the flag it is fixed with the type, in both directions (a longer one would reopen documents that had settled). It is in seconds, as the other durations of a document type are, and it says nothing about a document's own owner, whose deletioncanBeDeletedrules at any age. - For references it is no longer permanent. A
permanentDocumentreference refuses such a type (ReferencedDocumentTypeDeletableError, 40122) whatever itscanBeDeletedsays, so the guarantee that a validated permanent reference never dangles holds. When its owners can not delete its documents (canBeDeleted: false), it declares nottland it keeps removal records (the default), a document of it leaves state only on a moderator's record, and amoderatedDocumentreference is the one that points at it (adeletableDocumentreference is refused,ReferencedDocumentTypeModeratedError, 40144). A like or a reply that points at such a post declaresrefersTo: moderatedDocument: the reply stays valid through edits once a moderator removed the post, resolving to its removal record, and a join through it reports the removed post with its proven record. A type whose documents can also leave state otherwise, deleted by their owner, removed without a record or expiring, is adeletableDocumenttarget, and a join through it reports a missing id. See References. - What a deletion leaves is the type's to say, and fixed with it.
deleteKeepsRecord(defaulttrue,DocumentTypeV2::moderator_deletions_keep_records) says whether the deletion writes the removal record below, anddeleteRefundsOwner(defaultfalse,moderator_deletions_refund_owner) whether the owner is refunded its storage. Both needdelete: trueand change with no update (40212). A type that keeps no record has no records subtree, is refused bygetContractDocumentRemovals, and can never have a deletion restored (41119): its deletions are final.deleteKeepsFields(moderator_deletion_kept_fields) lists what of a deleted document stays public in its record: declared properties at any depth (meta.tags, ormetawhole) and the timestamps and block heights the type requires, never a transient property,$id,$ownerIdor a path inside another listed one. It needs a record, and is fixed with the type like the others.
The deletion is the seventh action of the same transition:
#![allow(unused)] fn main() { ContractUserModerationAction::DeleteDocument { document_type_name: String, document_id: Identifier, reason: ContractModerationReason, // as on a ban: a code nothing checks, a text that may be empty } }
It names no identity (identity_id() is None): whose document it is is only known once the document is read. The transform checks, in order and each refusal paid: the document type exists (10406), it carries the keyword (DocumentTypeNotDeletableByModeratorsError, 41115), the signer is the owner or a moderator (41101), the document exists (DocumentNotFoundError), its owner is neither the contract owner nor a moderator (41102, the rule that protects them from a ban protects what they wrote), and block time is within the type's window after the document's last modification ($updatedAt, else $createdAt), when the type sets one (41116). The document is read the way a document's own deletion reads it, billed the same. The action carries the contract, the document's owner and the block time, so Drive reads nothing again. Nothing the document type prices is charged: neither its deletion token cost nor its actionFees deletion fee, both of which are what a document's own owner pays for deleting it.
Drive then runs DocumentOperationType::ForceDeleteDocument, the ordinary deletion (so every index and aggregate of the type stays right) without its canBeDeleted guard, which is the owner's rule and not the moderators', and writes a removal record:
#![allow(unused)] fn main() { pub struct ContractDocumentRemoval { pub document_owner_id: Identifier, pub moderator_id: Identifier, pub reason: ContractModerationReason, pub removed_at: TimestampMillis, // the block time pub document_hash: [u8; 32], // sha256d of the document as serialized under its type pub restoration: Option<ContractDocumentRestoration>, // { moderator_id, restored_at } once restored pub kept_fields: Vec<u8>, // what `deleteKeepsFields` keeps public, encoded as the document encodes its properties } }
The kept values are copied from the document as read by the transform and encoded the way the document encodes its properties (encode_kept_fields): for each path the type lists, in the list's order, 0 when the document holds no value there, or 1 and the value as an optional property of its type is written, an object length-prefixed with its members in order, a time or block height as eight big-endian bytes and a core block height as four. The paths are not written: the type lists them and never changes the list. So the record is read, as a document is, under its document type (ContractDocumentRemoval::kept_values), and each value comes back typed exactly as reading the document gives it; an object kept whole reads members an update added after the removal as absent, as a document written before them does. They are values, not index entries: nothing finds a record by them.
On a type that keeps records, the record is what is left to say that a document was removed, not lost (a type that keeps none writes nothing below, reads no existing record, and is proved by the document's absence, VerifiedDocuments with the id and none): a client holding a dangling id (a reply whose parent is gone) can prove who removed it, whose it was, when, why and what it was. The moderator pays for it, reason included, and nothing ever deletes it. From protocol version 14 a document id commits to the nonce of its create transition and is produced at most once, so the removed id can not be created again by anyone: the only way the document comes back is a moderator's restore (see Restoring Documents below), which marks the record restored and leaves it in place. A record and a live document of the same id therefore coexist exactly when the record is marked restored, and a restored document deleted again gets a fresh record in place of the marked one.
The hash is of the document as Document::serialize writes it under its document type and the protocol version of the deleting block, computed from the document as read, not from the bytes as stored: a document stored under an earlier version of its type or of the serialization would never re-serialize to its stored bytes, and a client keeping the document rather than the bytes could never match them. A client that may have to undo a deletion keeps the document it fetched before deleting; serialized under the same contract it gives the same bytes.
The deleted document's owner gets no storage refund, unless its type sets deleteRefundsOwner, in which case the batch refunds it as the owner's own deletion would, the moderator still paying for the transition and the record. Otherwise the batch carries ContractModerationOperationType::ForfeitStorageRefunds, a marker that writes nothing, and apply_drive_operations generation 1 turns every removal the batch's document operations cause into a removal attributed to nobody: the bytes still leave the system (FeeResult::removed_bytes_from_system), no refund is computed, and the credits stay in the storage pools they were distributed to. What the document operations remove is forfeited whoever paid for it: the document's bytes, its owner's or an earlier one's (a transferred document's bytes may still be its first owner's), and those of any index subtree the deletion empties, which an earlier author's document may have created. The rest of the batch is not: when it frees moderation storage someone is owed (a restored removal record a fresh one replaces, which may shrink, or the approvals and the info of a team action that the approval deleting the document moves to the closed actions or drops), the document operations are applied as a GroveDB batch of their own, after the rest and in the same transaction, so what the rest frees is refunded to whoever its flags name, a moderator who also paid for the document included, for those bytes alone; otherwise nothing else frees bytes (a fresh record is an insert, the nonce and the counts keep their size) and the batch is applied as one. An estimate carries no refund to begin with (refunds come from the flags of what is really removed), and prices the batch as it is applied, one GroveDB batch or two. An author who deletes the same document with an ordinary document transition is refunded as always.
Restoring Documents
A deletion can be undone. The eighth action of the same transition brings a deleted document back, as it was:
#![allow(unused)] fn main() { ContractUserModerationAction::RestoreDocument { document_type_name: String, document: BinaryData, // the document serialized under its type, as it was when deleted } }
It names no identity and no id: the document is inside the bytes, and its id and owner with it. The transform checks, in order and each refusal paid: the document type exists (10406) and carries moderatorAbilities.delete (41115, only such a type keeps records), the signer is the owner or a moderator (41101, whoever deleted), the bytes decode under the type (a basic decoding error, never an execution error: the bytes are the signer's), the document has a removal record (ContractDocumentRemovalNotFoundError, 41119) that is not marked restored (ContractDocumentAlreadyRestoredError, 41122: the document is live), block time is within SystemLimits::contract_document_restore_window_ms of the removal, a week, the last millisecond included (DocumentRestoreWindowElapsedError, 41120), the bytes hash to what the record holds (DocumentRestoreHashMismatchError, 41121: the document as it was, not an edit of it), and no other document of the type holds a value of one of its unique indexes (DuplicateUniqueIndexError, 40105: the value was free while the document was gone, and someone may have taken it). The record read is billed; the hash pins everything else, so the owner, the revision, the timestamps, the references and the schema are not checked again. Whether the document's owner is banned is not asked either: a ban stops writes, it never removed documents.
The action carries the contract, the decoded document and the record marked restored by the signer at the block's time, so Drive puts the document back through the ordinary insert (DocumentOperationType::AddDocument, every index and aggregate of the type included) and replaces the record in place, without reading again. The restored document's storage flags name its owner, as they did before the deletion: the signer pays for the bytes, and the refund of a later deletion is the owner's, as it always was. Nothing the document type prices is charged, neither its creation token cost nor its actionFees creation fee, and no fee agreement is asked: a moderator undoes a moderation, it does not create content. The restore is proved by the record, now marked restored and holding the hash of the bytes the transition carried; the verifier reads the document's id out of those bytes under the contract's document type, so it needs the contract, which the SDKs register with their context provider before broadcasting.
Two consequences follow from the document coming back byte for byte. Its $updatedAt does not move, so a type with moderatorAbilities.deleteWithin may have settled it while it was gone: no moderator deletes it again alone until its author edits it, though the author still can, and on a type that sets deleteSettled the seated team can, by a fresh proposal (the action that deleted it ran and is closed). A deletion the team approved together is never restored: it is the only removal made after the document settled, since a moderator alone deletes up to the settling time and the team only after it, so the restore transform compares the removal time with the settling time of the document the bytes hash to (no read), and refuses the restore of a later removal, paid (SettledDeletionNotRestorableError, 41209), so no single moderator undoes what the leader and the members agreed on. And a type with a contested index can not carry moderatorAbilities.delete at all (InvalidContractStructure, 10231): a contested index only takes a document through a vote, which no restore can go through, so such a deletion could never be undone. The bytes are decoded under the type as the contract holds it when the restore is processed, and the hash is of the document serialized under the type as it was when the document was deleted: a contract update that changes the type's layout inside the window (a property added, say) leaves the record unrestorable, since bytes that decode under the new layout cannot hash to what the record holds. A client therefore serializes under the contract's current version, as Drive does, and keeps the document rather than the bytes.
Deleting Settled Documents
A document past its type's moderatorAbilities.deleteWithin window is settled: no moderator deletes it alone (41116). A type that also sets moderatorAbilities.deleteSettled lets the members of an elected contract's seated team delete it together, by approving its deletion one transition each; the approval that meets the rule deletes it. The rule is DocumentTypeV2::moderator_settled_deletion, a SettledDeletionRule { leader, approvals }: approvals members of the team must approve, the leader among them when leader is set. { leader: true } is the leader alone, { leader: true, approvals: 3 } the leader and two others.
The parser admits the key only beside deleteWithin, on a contract whose moderators are elected, with approvals of at least 1 and, at registration, at most the members the declared team can hold: its leader, SystemLimits::max_moderation_charter_elected_members (15, the maxItems of a charter's members) and the declaration's maxAddedModerators, so a rule no team could meet is refused rather than frozen with the type. All are InvalidContractStructure (10231) otherwise; the upper bound is a registration limit, not checked when a stored contract is read back, and the elected declaration must give the team deleteDocuments on the type (10900). It is fixed with the type in both directions (40212).
The team deletes a settled document the way a token group carries out an action: one member proposes it, the others approve it by its id, and it runs once the approvals meet the rule. What is kept meanwhile is a team action under the contract, a ContractTeamAction:
#![allow(unused)] fn main() { pub struct ContractTeamAction { pub proposer_id: Identifier, // the member that proposed it, its first approval pub proposed_at: TimestampMillis, // the block time of the proposal pub event: ContractTeamActionEvent, } pub enum ContractTeamActionEvent { DeleteSettledDocument { document_type_name: String, document_id: Identifier, document_last_modified_at: TimestampMillis, // the document's $updatedAt (else $createdAt) then document_revision: Option<Revision>, // the document's $revision then, if it has one reason: ContractModerationReason, // the proposer's }, } }
The proposal is the transition's DeleteSettledDocument { document_type_name, document_id, reason }. Its id is computed, not carried: ContractTeamAction::settled_deletion_action_id, a double SHA-256 of the contract, the proposer, the proposer's nonce for the contract, the document type name, the document id and the reason (each part of it behind a presence byte or its length), which a proposer's nonce makes unique and the client knows before broadcasting; the reason is part of it so that the proof of the proposal is the proof of this one and not of another signed with the same nonce (ContractUserModerationTransition::team_action_id). The transform (transform_settled_deletion_proposal_v0) checks, in order and each refusal paid: the type exists (10406) and sets the rule (DocumentTypeNotDeletableOnceSettledError, 41204); a charter is seated (ContractModerationTeamNotSeatedError, 41205), since the rule names a seated team's leader and counts its members, so no interim moderator and no contract owner proposes; the signer is the leader or an active member (41101) and the declaration gives the team deleteDocuments on the type (41201); the reason names a reason document the team's proposal lists (ModerationReasonNotListedError, 41203); the document exists (40101) and its owner is not protected (41102); it is settled, block time past its last modification ($updatedAt, else $createdAt) plus the window (DocumentNotSettledError, 41206: within the window a moderator deletes it with DeleteDocument); and a signer the leader added was added before the document was created, unless the rule sets approversPredateDocument: false (ContractTeamMemberAddedAfterDocumentError, 41212: late_addition_refusal, which reads the addition's $createdAt from the seat the authority check already found, SeatedModerationCharter::seat_of, so no read is added). Both transforms take the last modification and the settling time from one helper (last_modified_and_settled_at), so a deletion alone and a proposal can never both pass, or both be refused, at one time. A rule the proposer meets alone runs at once.
An approval is the transition's ApproveTeamAction { action_id }: what it approves is the stored action's, so it carries nothing else. The transform (transform_team_action_approval_v0) checks, in order and each refusal paid: the contract keeps team actions, a document type of it setting deleteSettled (ContractTeamActionDoesNotExistError, 41207 otherwise); the action, read billed under the active and then the closed actions, exists (41207) and has not run (ContractTeamActionAlreadyCompletedError, 41210), which needs nothing of the team, since an action exists only once a seated member proposed it; a charter is seated (41205); the signer is on the team with deleteDocuments on the action's type (41101, 41201); its approvals are read, billed; the document exists (40101), its owner is not protected (41102), and it is still as proposed, its last modification and its revision unchanged (ContractTeamActionDocumentChangedError, 41211): every change of the document moves its revision, a moderator's change of its fields included, which leaves $updatedAt and so the settling time alone, and a transfer, a purchase or a price update moves it too; a signer the leader added was added before the document was created, as for a proposal (41212); and the approvals do not hold the signer's (ContractTeamActionAlreadySignedError, 41208), checked last so that a member taken off and added again too late, whose earlier approval no longer counts, is told 41212. Nothing lapses: a proposal the document never changed under stays active until it runs.
Whether it runs. The approvals given, this one counted, are compared with what the rule needs: its approvals, or all the seated team can hold when it asks for more (ElectedCharter::seats, the count the SDKs' ModerationTeam::seats gives a client too: the leader, the members the charter elected and the declaration's maxAddedModerators, the bound the moderators pot's settle reads the counts by, so a charter electing fewer members can still meet the rule; a member the leader removed still counts toward what the team can hold, so the leader can not lower the bar by removing members who would not approve, and gets the seat back by deleting the removal). Approvals too few to meet it read no seat and the approval is added. Otherwise the earlier approvers are checked against the seated team, one point read each (SeatedModerationCharter::seat_of, the leader needing none) while they are no more than the queries a read of the whole team makes (one for the removals when the charter elected anyone, one for the additions when the declaration allows any), and past that one read of the team (SeatedModerationCharter::fetch_active_seats), with the same answer for fewer billed reads (still_counted_approvers). The approvals of members who left no longer count and are dropped, refunded to them, and no longer prove: one who comes back after that approves again, while one back before any such read still has its approval counted. Under approversPredateDocument the same goes for a member the leader took off and added again since the document was created: on the team, but its seat (TeamSeat::Added { added_at }, the new addition's $createdAt) no longer counts for the document (TeamSeat::counts_toward, SettledDeletionRule::admits_addition), so its approval is dropped and a new one refused. When the approvals that count meet the rule (SettledDeletionRule::is_met_by, the leader read from the seated charter), the document is deleted as DeleteDocument deletes it (the shared document_deletion_context: ForceDeleteDocument, the removal record when the type keeps one, naming the member whose approval deleted it and the proposal's reason, and the refund as deleteRefundsOwner says), the action closes with the approvals that counted, and every counted approver's moderation action count goes up by one (next_moderation_action_count, the read count_for_signer makes too): a deletion signed by several counts for each of them, and approvals that fall short count for nobody, so that approving what never passes earns no share of the pot. The action carries what to write (ContractTeamActionWrite::Propose, Approve or Close), the deletion when there is one, and the counts, so Drive reads nothing again.
Storage mirrors a token group's actions (see the tree below): each approval is its own sum item under the action, flagged with the member that paid for it, so nothing is ever rewritten. The closing approval moves the action's info and the approvals that counted from the active actions (M) to the closed ones (X) without storage flags, which refunds each to its member, and deletes what is left under the active actions; the member whose approval closed the action pays for the closed copy, which nothing deletes. A closed action says who agreed to delete what and why, and when it was proposed (the block time of the deletion itself is in the removal record, when the type keeps one); an active one says who proposed it and who approved so far. Both are read with getContractTeamActions and getContractTeamActionSigners (see The DAPI Queries), which is how a member finds the actions to approve.
The proof of a proposal or an approval is the signer's approval wherever it is: the prover builds contract_team_action_signer_query, keys M and X with the subquery path [action id, S, signer], from the transition alone, and verify_contract_team_action_execution checks that the approval is under exactly one of them, returning VerifiedContractTeamActionSignature(contract, action id, status): active, or closed once the action ran and the document went. An approval moves only with its action, so the proof holds while the approval stands; only an approval dropped because its member left the team no longer proves. Closed says the action ran, by this approval or a later one: a proof taken after a later approval closed the action finds the signer's approval there.
Changing Document Fields
A deletion takes a document down; some moderation only annotates one. A report is handled, a post flagged, a ticket assigned, and the document should stay, with the moderators' word on it. A document type may keep fields for its moderators:
"report": {
"type": "object",
"documentsMutable": false,
"moderatorAbilities": { "changeFields": ["status", "resolution"] },
"properties": {
"reason": { "type": "integer", "minimum": 0, "maximum": 8, "position": 0 },
"status": { "type": "integer", "minimum": 1, "maximum": 3, "position": 1 },
"resolution": { "type": "string", "maxLength": 200, "position": 2 }
},
"required": ["reason"],
"additionalProperties": false
}
moderatorAbilities.changeFields is DocumentTypeV2::moderator_changeable_fields, parsed by apply_moderator_abilities. The listed properties are the moderators' to write, and only theirs:
- A document's owner writes them only as a moderator. The batch transformer refuses a create that sets one, and a replace that changes, adds or removes one, with
DocumentModeratorFieldNotWritableError(41124), unless the signer moderates the contract (transformer::v0::moderator_fields::judge_moderator_field_write, throughcommon::moderators::moderator_field_write_refusal, which reads who moderates only when such a field is written). When the signer does moderate, the action is stamped as a moderator's: the create action'smoderatedflag, the replace action'smoderated_atandmoderated_by(which otherwise carry the stored document's stamp over, each half as stored), so the document is written with$moderatedAtthe block's time and$moderatedBythe signer. It runs in the transformer, beside the aggregate reads, so the mempool refuses such a write as a block does; it is inert before protocol version 14, whose types keep no such field. A seated team's member must also holdchangeDocumentFieldson the type. So a report starts with nostatus, and its author can never mark it handled. - The fields are fixed with the type, in both directions (40212), and a type an update adds may declare them: a field only moderators write starts absent on every document, and one its owner could have set would no longer be theirs.
- The parser keeps them out of what a change could break. Each must be a declared, optional, stored top-level property, not
immutable, neither a reference nor read by one (awhereentry's referring value, afindBysource or function param, a key id's identity property), neither generated nor a parameter of a generated property, and in no contested index; the type may not be indexOnly (10231). References are therefore never checked again when a moderator changes a document: what they read did not move. That matters for a report whose post a moderator already deleted: a replace re-checks everydeletableDocumentreference and would refuse it, while a moderator's change goes through. For the same reason afindBykey or aninListlist on another type may not read a field moderators write (schema_property_is_fixed_once_writtencounts it as moving). - A type that lists any keeps a revision (
DocumentTypeBasicMethods::requires_revision, throughhas_moderator_changeable_fields), even withdocumentsMutable: false: Drive stores the change as an update, which needs one, and the revision is what refuses an owner's replace built before the change. - The contract's moderation may keep no list, as with deletion, when a type keeps fields for its moderators.
The change is the ninth action of the transition:
#![allow(unused)] fn main() { ContractUserModerationAction::ChangeDocumentFields { document_type_name: String, document_id: Identifier, fields: BTreeMap<String, Value>, // each field's new value, `Null` removing it reason: ContractModerationReason, } }
Basic structure refuses one that names no field or a system property (InvalidContractModerationDocumentFieldsError, 10905, unpaid). The transform (transform_document_fields_change_v0) checks, in order and each refusal paid: the document type exists (10406), every field is one it keeps for its moderators (DocumentFieldNotChangeableByModeratorsError, 41123), the signer moderates the contract (41101) and a seated team holds changeDocumentFields on the type (41201) and names a listed reason (41203), the document exists (40101) and has not expired (40140). It then builds the changed document: the stored one with each field set or removed, compared as a replace compares (Value::equal_underlying_data, so an integer sent at another width than the stored one is no change), $revision one higher (OverflowError, 10700, at u64::MAX), $moderatedAt the block's time and $moderatedBy the signer, everything else as stored, $updatedAt and its heights included. That document is judged as a replace judges one: validate_document_properties (the schema and propertyConstraints, with the countOf and sumOf totals read as they will be once it is stored, read_property_constraint_aggregates_for_moderator_change), distinctFrom and the shapes of encryptedFor properties, then, when a changed field is in a unique index, Drive::validate_moderated_document_uniqueness, the restore's check generalized to a changed document (only the changed fields are checked, and the document's own entries do not clash). Whoever owns the document is no protection: the fields are the moderators'. Nothing the document type prices is charged.
The action carries the contract and the changed document, and Drive stores it with DocumentOperationType::UpdateDocument, the replace's update, every index and aggregate of the type included. The document's storage flags name its owner, as a replace's do: the moderator pays for the bytes the change adds, and what the change frees (a smaller value, an index entry that moved) is refunded to the owner, who paid for it. A seated team's member does not have the change counted toward the team's action share: the changes of one document have no bound, and counting them would let a member farm it. A change that changes nothing is refused (10905) rather than written. $updatedAt does not move, so a change never opens a moderatorAbilities.deleteWithin window again: a moderator can not keep a post deletable by touching it.
The stamp is two system properties of document serialization format 3 (bits 9 and 10 of its time field flags): absent on a document no moderator has written, set only here and by the batch transformer, which stamps a moderator's create or replace of such a field (an action it did not judge is written unstamped), carried over by a replace that leaves the fields alone and by every transfer, purchase and price update, and put back by a restore with the rest of the bytes. A type that keeps fields for its moderators may index either, never in a unique index (apply_moderator_abilities, 10231; the core index check admits the two names from parser generation 3, ParserGeneration::admit_moderation_stamp_indexes). See System Properties.
The change is proved by the document itself: the prover builds the single-document query a replace is proved by, and verify_contract_document_change_execution checks that the document exists and holds each value the change set (and none of the removed ones), returning VerifiedDocuments. The stamp is not compared: a later moderator's write to another field moves it without undoing this change. Its other properties are the owner's and are not compared; the verifier reads the document under the contract, so the SDKs register the contract before broadcasting.
Storage
[64] DataContractDocuments
└── <contract id>
├── [0] the contract (or its history subtree)
├── [1] documents
└── [2] other
├── [16] document removals -> <document type name> -> <document id>
│ -> Item(owner id ‖ moderator id ‖ removed at ‖ document hash ‖ restored? [‖ restored by ‖ restored at] ‖ reason) (with such a document type)
├── [24] team actions -> M (active) | X (closed) -> <action id> (with a deleteSettled type)
│ ├── I -> Item(tag ‖ proposer ‖ proposed at ‖ type ‖ document id ‖ last modified at ‖ revision ‖ reason)
│ └── S -> SumTree(<member id> -> SumItem(1))
├── [48] moderation action counts -> <identity id> -> Item(count) (elected contracts)
├── [64] contract version item (every contract)
├── [128] banlist -> <identity id> -> Item(reason) (when declared)
├── [192] suspensions -> <identity id> -> Item(until ‖ reason) (when declared)
└── [224] warnings -> <identity id> -> Item((warned at ‖ len ‖ reason)+) (when declared)
until is a u64 of block time in milliseconds, big-endian. A reason is a tag byte (bit 0: a code follows, bit 1: documents follow, bit 2: a reason document follows), the code as a big-endian u16 when tagged, the 32-byte id of the reason document when tagged, then when tagged the documents it cites (their count in one byte, then each one's type name as a length byte and the name, and its 32-byte id), then the text as UTF-8 up to the end of the value, so an entry with an empty reason and no code costs one byte more than the bare entry would. A value without the tag byte is an entry written before entries carried a reason and reads as the empty reason; a tag without bit 1 is a reason from before documents could be cited, and one without bit 2 a reason naming no reason document. A warning list entry holds every warning the identity carries, oldest first, each warned at as a u64 of block time in milliseconds, big-endian, the length of its encoded reason as a big-endian u16 (the prefix that lets one value hold several reasons, each of which would otherwise run to the end), then the reason as above (types::encode_warnings). A warn rewrites the entry one warning longer; a clearWarnings deletes it, so a stored entry never holds fewer than one warning. A document removal is the document owner's id, the moderator's id, removed at as a u64 of block time in milliseconds, big-endian, the 32 bytes of the document hash, a tag byte (0: not restored, 1: restored) followed when restored by the restoring moderator's id and restored at as a u64, big-endian, then the reason the same way (types::encode_document_removal): 105 bytes before the reason, 145 once restored. The tag byte's bit 1 says kept fields follow the restoration, before the reason: their length as a big-endian u32, then the bytes encode_kept_fields wrote. Drive stores and proves them as they are, and reads the rest of the record without the document type; only the kept values need it. A record that keeps no field is written without them, as before fields could be kept. The records a write walks past are estimated from the document type, not from the record being written: every record of a type keeps the same paths, but a path the document held no value at is left out and values vary in length, so each kept path is estimated at its presence byte and its property's middle size, as a document of the type is, a time or height at its eight bytes and a core block height at its four (types::estimated_document_removal_kept_fields_size).
A removal record is replaced in place, as a suspension is, by the restore that marks it and by the deletion of a restored document, which writes a fresh record over the marked one; the deletion transform reads the record, billed, to know which of the two it writes, and a record that is not marked restored beside a live document is a state no transition produces. The replacement's flags follow GroveDB's flag merge like a suspension's: the restore adds forty bytes and passes the record, with the refund of its removal, to the restoring moderator, who pays for them. The fresh record of a second deletion loses the restoration but keeps the values the document holds then, so it may be shorter or longer: a longer one passes to the moderator deleting again, who pays for the bytes it adds; a shorter one refunds the bytes it frees to the restoring moderator, and passes to the moderator deleting again only within the epoch the record was paid in (or once the record spans epochs), keeping the restoring moderator's flags when shortened in a later epoch, as GroveDB merges the flags of a single-epoch item (the deletion's forfeiture takes its document operations alone, applied as a GroveDB batch of their own when the batch rewrites a record, so the record's holder is refunded whoever it is, the document's owner included, for the record's bytes only). A fee estimate prices a replacement as a fresh insert of what it adds to the record it replaces, never less than the bytes it adds, rather than GroveDB's average-case replace, which prices none, and the records a write walks past are estimated as restored, the larger shape, keeping what the type's records are estimated to keep.
The document removals tree exists exactly when the contract has a document type whose moderators' deletions keep records (moderatorAbilities.delete without deleteKeepsRecord: false): a contract without one keeps the other tree, and the shape, it would have had. One subtree per such document type is created with the type, by insert_contract generation 2 or by update_contract generation 2 for a type an update adds, and the tree above them with the first: whether it is there is read off the stored contract, since an existing type never changes the keyword, so the update needs no read. Nothing is created lazily by the first removal. The key sorts below 128, as a key added later should: a contract that keeps both lists, the one whose other tree then holds four keys, still has the banlist on top. With fewer keys the version item is on top, and the list one level down.
The team actions tree at 24 is shaped like a token group's actions: M holds the actions still gathering approvals and X those that ran, each an action id to a tree of the action's info (I) and the sum tree of its approvals (S, a member's id to a sum item of 1, so the sum is the number of approvals). It is created with the contract by insert_contract generation 2 when a document type sets moderatorAbilities.deleteSettled, with M and X under it, nothing lazily. No update adds such a type: it needs the elected declaration, fixed at creation, to give the team deleteDocuments on it, and the declaration can only name the types the contract was created with. An action's info is a tag byte (0: the deletion of a settled document), the proposer's id, the proposal's block time as a u64 big-endian, the document type name's length in one byte and the name, the document id, the document's last modification then and its revision (0 for a document that carries none; revisions start at 1), each a u64 big-endian, then the reason as a banlist entry holds it (encode_contract_team_action). Under M the info and each approval are flagged with the member that wrote them; the approval that closes the action moves them to X without flags, and deletes what is left under M. A fee estimate prices the move of the info by the size of the action as stored, which validation read. A sum item's storage is estimated by GroveDB at its serialized size and charged at its fixed size, a few bytes more per approval, as for a token group's approvals: the estimated fee still exceeds the one paid, the processing it over-estimates covering the difference.
The moderation action counts of an elected contract sit at 48: one item per member of the seated team who signed a counted action since the moderators pot was last settled, its identity id to its count as a big-endian u32 (see The seated team below). The tree is created with the contract, the only time elected moderation can be declared, so it is never made lazily. Each counted action reads its signer's count with one point read and writes it one higher: an insert of 36 bytes for the member's first action of a period, a replacement of the same size after. A settle reads the whole tree, at most one item per identity the team can hold (the leader, the elected members and the additions the target allows), and deletes every item. Per-member items keep the action, far more frequent than a settle, to a four-byte write; one packed item for the team would rewrite every member's count on every action. The items carry no storage flags: the member whose action writes one pays for it, the settle that deletes it refunds nobody, and a settle's fee result carries no refunds for anyone else. The key is below 64: created with two or three lists it leaves the banlist on top (with every list, where the version item and the suspension list alone had put the suspension list there), with or without the removal records tree; created with the banlist alone, or with the banlist and one other list beside removal records, it puts the version item on top and the banlist a level down. What sits there is read by the team's actions and by a settle, never by a document transition.
The contract's own subtree holds three keys whatever the contract keeps, so its Merk keeps 1, the documents, on top: every document proof and write goes through that key, and a fourth key beside it would have pushed it one level down (a Merk built from one sorted batch roots at the middle key). Everything else a contract keeps goes into 2, its other tree, which protocol version 14 introduces together with the version item. Inside, the keys are spread like the root tree's, so the tree stays balanced as it fills and the most read entry sits on top: the banlist at 128, read by every document transition on a moderated contract, the version item at 64, the suspension list at 192, the warning list at 224. A Merk built from one sorted batch roots at the middle key, the upper middle of an even count, so a key added later goes where it keeps 128 the median of the keys created together in the likely combinations: the warning list, which no document transition reads, sits above 192, which leaves the banlist on top for a contract keeping the banlist and a warning list, with or without the removal records tree at 16, and for one keeping all three lists with that tree; a contract keeping all three lists and no removal records has the suspension list on top and the banlist one level down.
The other tree is written by every contract insertion, and by the migration on the first block of protocol version 14 for the contracts stored before it. A contract update finds it there, so its fee estimate writes none; applied, the update reads key 2 once, billed, rather than fail inside a block: a tree is left alone (it may hold the lists), a missing one is written. The 4.2 betas kept the version item itself at key 2, before the other tree existed: an update of a contract that still holds that item puts the tree in its place and the item under it (add_contract_to_storage generation 1). Until such a contract is updated, the unproved getDataContractsLatestVersions reads its version from that item, and the proved form shows no version item for it.
The list trees are created by insert_contract generation 2 for a contract that declares them, and by nothing else: the lists are fixed at creation, so a contract update creates none and leaves the existing ones and their entries alone, and no tree is made lazily by the first ban. An entry's storage flags name the moderator that wrote it, so the storage refund of its deletion goes to that moderator whichever transition deletes it: the explicit unban or unsuspend, the ban over a suspension, or the document transition that sweeps a lapsed suspension. The sweep's processing fee is charged to the batch signer.
The writers, readers and provers live in packages/rs-drive/src/drive/contract/moderation/, versioned by DriveContractModerationMethodVersions. A suspend that replaces an entry is a batch_replace, because two operations on one key would fail the batch; so is a warn on an identity that already carries warnings, which rewrites the entry whole with the new warning last. The entry is then longer, so it passes, with the refund of its removal, to the moderator that warned last, who pays for the bytes the warning added; the same merge a longer suspension replacement goes through. The replacement brings its own reason, so the entry may change size: a longer replacement merges the flags as a document that changes hands does, the moderator that replaced it paying for the bytes it added and becoming the entry's owner, refunded when it is removed; a shorter or an equally long one stays the first moderator's, who is refunded the removed bytes at once and the rest on removal. A fee estimate prices a replacement as a fresh insert of the whole entry, because GroveDB's average-case replace assumes an item keeps its size and would price no storage for a longer reason; the entries a write walks past, and the one a delete removes, are estimated at a typical reason (128 bytes of text), not at the longest.
Reading and Proving
fetch_contract_moderation_status(contract, identity, lists) reads the identity's entry on each list named, the warning list included, and the verifier of its proof rebuilds the same merged path query from the same lists. The lists are the ones the contract's config declares; an undeclared list has no tree and cannot be queried, so a status query names the lists it wants and the node refuses one the contract does not keep. fetch_contract_moderation_entries pages one list in identity id order, bounded by the platform version's max_returned_elements (the default page size too, and the number the proof verifier assumes when a request names no limit), with the last identity as the cursor. A page shorter than its limit is the last one and carries no cursor.
The DAPI Queries
getContractModerationStatus(contract_id, identity_id, lists, prove): the identity's status on the lists named.getContractModerationEntries(contract_id, list, start_after, limit, prove): one page of a list. An entry of the warning list carries every warning of the identity, and itsreasonis the latest warning's.getContractTeamActions(contract_id, status, start_at_action_id, count, prove): one page of the actions a seated team votes on, active or closed, in action id order, bounded bymax_returned_elements, on a contract that keeps them (a document type of it carriesmoderatorAbilities.deleteSettled: the node refuses any other). Each is the proposer, the proposal's time and what it does, today the deletion of a settled document with the document as proposed and the reason, and how many approvals it holds (approval_count, the proposer's included). The count is the sum of the action's approvals treeS, each approval a sum item of 1, so the page readsIandSof every action, two elements each, and its GroveDB limit is twice the page's. An active action's count is an upper bound: the approval of a member who left the team is dropped only when a later approval reads the team, and is counted until then. The exact figure is the action's signers (getContractTeamActionSigners) still on the team (the SDKs'ModerationTeam::contains), which a client reads only for an action whose count could meet its rule.Drive::verify_contract_team_actionsrebuilds the path query from the same request.getContractTeamActionSigners(contract_id, status, action_id, prove): who approved one action, active or closed as the request says, the proposer among them unless it left the team and its approval was dropped; none when there is no such action with that status.Drive::verify_contract_team_action_signersrebuilds the path query.getContractDocumentRemovals(contract_id, document_type_name, document_ids | page, prove): the records of the documents moderators deleted, within one document type that carriesmoderatorAbilities.delete(no other keeps records, so the node refuses any other). By document ids, up tomax_returned_elementsof them and none twice: an id with no record is left out of the response, and proved absent by a proof. Or one page in document id order, with the last document id as the cursor. Each record carries the document hash, once restored who restored it and when, and the values it keeps (kept_fields, the bytes as the record stores them, which a client reads under the contract's document type as it reads documents; the SDKs do so,keptFieldsin wasm-sdk).Drive::verify_contract_document_removalsrebuilds the path query from the same request.
A status query answers for the lists it names and no others: Drive::verify_contract_moderation_status and the SDK result both return ContractModerationListStatuses, one ContractModerationListStatus per list queried, so a list that was not read is absent rather than reported as empty (banned() is None unless the banlist was queried). ContractModerationStatusQuery::for_contract names every list the contract keeps; the wasm-sdk does the same, fetching the contract, when the query names no list. Both have Fetch and FetchUnproved impls in the Rust SDK (platform::contract_moderation), wasm-sdk functions and contracts.moderationStatus / contracts.moderationEntries on the JavaScript SDK. The proof of a moderation transition's execution covers the lists the moderation touched and is classified as affected state: an earlier or later moderation leaving the same entries verifies just the same. A ban does two things, adds the ban and removes a suspension, so its proof covers every barring list the contract keeps (the banlist entry present, the suspension absent), which the prover and the verifier both read from the contract's config (so the SDKs fetch and cache the contract before broadcasting a ban, as they do for the contracts a document batch touches); an unban, a suspend, an unsuspend, a warn and a clearWarnings prove the one entry they edit. A warn's proof shows the entry with the transition's reason as its last warning (the block time is the block's, which the verifier does not know); a clearing's shows the entry absent. The result, VerifiedContractModerationListStatuses, holds one ContractModerationListStatus per list proved, never a full status: a list that was not proved is left unknown rather than reported as empty. An identity whose unsuspend was just proved may be banned; the status query answers that.
Fee Pots and the Claim
A moderation team can be paid. A document type may charge a fixed fee in credits for an action on its documents (the actionFees keyword, see Document action fees), split in two parts. The owner parts collect in the contract's owner pot, the moderators parts in its moderators pot.
[40] PreFundedSpecializedBalances (sum tree)
├── [64] owner fee pots (sum tree) -> <contract id> -> SumItem(credits)
├── [128] voting balances
└── [192] moderators fee pots (sum tree) -> <contract id> -> SumItem(credits)
[64] DataContractDocuments -> <contract id> -> [2] other
├── [32] last claim of the owner pot Item(epoch u16 BE | time u64 BE | claimant id) (after a claim)
└── [96] last claim of the moderators pot Item(epoch u16 BE | time u64 BE | claimant id) (after a claim)
The pots are not under the contract. The per-block total credits check (calculate_total_credits_balance) sums a fixed set of root sum trees, and DataContractDocuments is a normal tree: credits parked under a contract would leave that sum and fail every block with CorruptedCreditsNotBalanced. PreFundedSpecializedBalances is one of the summed trees, so the pots live there, in two sum trees beside the voting balances, created at genesis (state structure 4) and by the upgrade to protocol version 14 through the same helper, one after the other, so that both node populations build the same Merk. A pot is created by the first fee it receives, and so is its tree on a chain that reached protocol version 14 on a build from before the pots: that first fee checks, with a billed read, that the tree is there. The estimation of a voting balance write moves to generation 1 with them, because the prefunded balances layer now holds three trees instead of one. The two last claims are plain items of the contract's other tree, below 128 so the banlist stays on top, written by the first claim and replaced by every later one. A last claim (ContractFeePotLastClaim) is 42 bytes: the epoch of the claim, which the next claim is judged against, the time of its block in milliseconds, and the id of the identity that signed it. The owner pot's claimant is always the owner; the moderators pot's is whichever member of the team claimed for all of them, so the team can see who paid them and when. Every last claim has the same size, so a replacement never changes the size of the item, and the item carries no storage flags: it is never removed, and no claim adds bytes for anyone to own.
The team that shares the moderators pot is the set of identities the contract appoints, the owner among them only when appointed, and the owner alone when nobody is appointed (ContractModerators::team). It is about earnings, not authority: an owner who is not appointed still may moderate. For an elected contract it is the interim's team until a charter is seated, and from then on the seated team, which shares the pot by its proposal's reward split (see The seated team below). ContractFeePot::recipients names who a payout of a pot goes to: the contract owner for the owner pot, the team for the moderators pot, nobody for the moderators pot of a contract that declares no moderation.
ContractFeeClaim (state transition type 25) names a contract and a pot and pays the pot out. It is signed with a CRITICAL authentication key under the signer's contract nonce, and the claimant pays its gas like any other transition.
| Stage | Check | Error |
|---|---|---|
| Transform (state, paid) | the contract exists | DataContractNotPresentError (10400) |
| the signer is a recipient of the pot: the owner for the owner pot, a member of the team for the moderators pot (the leader or an active member once a charter is seated) | 41113 | |
| the pot was not paid out in this epoch yet | 41111 | |
| every recipient gets at least a credit | 41112 |
The owner pot goes to the owner whole. The moderators pot is split equally between the team, and what the split leaves over, less than a credit per member, stays in the pot for the next claim, so no member is favoured by the order of the identity ids. A seated team's pot is split by its proposal's reward split instead, rounded down the same way. Each pot is paid out at most once per epoch and the two are independent: the owner's claim does not use up the team's, nor the reverse. A refused claim is paid for by a nonce bump and leaves the pot and its last claim alone. As for moderation, state validation is the transform, so the mempool refuses with the same codes as a block.
The team is read when the claim executes. An owner who changes the appointed set by a contract update and then claims pays the new set: that follows from the owner controlling the contract's config, and is not prevented. The claim credits every recipient's balance, which is why a named moderator must exist (41110): crediting a balance that is not there is an internal error.
The proof of a claim's execution shows the pot with its last claim and the balance of every recipient, which the prover and the verifier both read from the contract. When the claimant is not among them, it shows the claimant's balance alone: the claimant is then on a seated elected team, which the charter contract names and the contract does not, so neither side could list the other payees (ContractFeePot::claim_proof_identities). An interim team's claim, before a charter is seated, is proved with every recipient's balance as before. VerifiedContractFeeClaim carries the contract id, the pot, that last claim (epoch, block time, claimant), the credits left in the pot and the balances. A pot that was never claimed proves no claim; a later claim of the same pot verifies just the same, so the result is classified as affected state.
Reading the Pots
getContractFeePots(contract_id, prove): both pots of the contract, each with its credits and its last claim: the epoch and the block time it was paid out in, and the identity that claimed.getContractModerationActionCounts(contract_id, prove): how many counted actions (bans, suspensions, warnings and document deletions) each member of an elected contract's seated team signed since the moderators pot was last settled, in identity id order; a member that did not act since has no count. The node refuses a contract that is not elected, which has no counts tree, and an elected one stored before protocol version 14 counted actions (a development network's), which has none either (Drive::contract_keeps_moderation_action_counts), with or without a proof. The path query (Drive::contract_moderation_action_counts_query) reads the whole tree at48without a limit, which the prover andDrive::verify_contract_moderation_action_countsbuild alike, as the approvals of a team action are read: the tree holds at most the team, since every settle deletes the counts and a change of the team settles first, and a limit taken from today'sSystemLimitscould cut short a team registered under larger ones with a proof that still verifies.
The query always reads both pots, so its proof is one fixed path query (Drive::contract_fee_pots_query) that the prover and Drive::verify_contract_fee_pots build alike, with nothing in the request to get wrong. A pot nothing was paid into yet has no element and reads as zero credits, and a pot never paid out has no last claim, which is not a claim in epoch 0: a pot can have been paid out in epoch 0, so the last claim is a message of its own on the wire, unset when there is none, and the JavaScript fields (lastClaimEpoch, lastClaimTimeMs, lastClaimantId) are absent together. The proof says nothing about the contract itself, only about what is stored under its id, so the node refuses the query for a contract it does not hold before it proves anything, and a client that needs to know the contract exists fetches it.
The counts are what the action share of a claim splits by, so anyone can see who has been acting and a member sees its part of that share. A full preview of a claim also needs the team's submittedCharter: its rewardSplit, whose leader and equal shares are paid first, and its moderatorsShare, which sets what the pot collects. The counts are a snapshot of the period since the last settle, which resets them, so right after a claim they say little. The Rust SDK has Fetch and FetchUnproved impls for ContractModerationActionCounts (platform::contract_moderation, queried by the contract id or a ContractModerationActionCountsQuery), the wasm-sdk getContractModerationActionCounts and getContractModerationActionCountsWithProofInfo, and the evo-sdk contracts.moderationActionCounts and contracts.moderationActionCountsWithProof.
A recipient reads the pots to decide whether a claim is worth its gas: the credits are what it would pay, and a last claim epoch equal to the current epoch means the claim would be refused (41111). A member of the team also reads there which member last claimed for the team, and when. The Rust SDK has Fetch and FetchUnproved impls for ContractFeePots (platform::contract_fee_pots, queried by the contract id), the wasm-sdk getContractFeePots and contractClaimFees, and the JavaScript SDK contracts.feePots and contracts.claimFees.
The claim's proof is verified against the contract, which names who the pot pays, and the team can change by a contract update. So every client fetches the contract again before a claim instead of trusting a cached copy: ClaimContractFees in the Rust SDK, contractClaimFees in the wasm-sdk, and the wasm-sdk's generic broadcastAndWait for a ContractFeeClaim built by hand, which falls back to the cached copy when that fetch fails, because the transition is already broadcast by then.
Elected Moderation
A contract may hand the choice of its moderators to the network instead of keeping it: it declares that its moderators are a team elected by masternodes and evonodes. Teams apply with a charter in the moderation charters system contract (see docs/protocol/moderation-charters.md), masternodes elect one in a contest for the contract's seat, and the seated team moderates with the contract's declaration, charging at most what the contract declares. What the contract itself holds is the declaration, frozen at its creation, and the interim: how the contract is moderated until its first team is seated.
#![allow(unused)] fn main() { pub struct ElectedModerators { pub join_window: u32, // seconds; at most 4 weeks, at least 1 day on mainnet (0 elsewhere), 1 week by default pub vote_window: u32, // the same pub challenge_cool_down: Option<u32>, // Some = seat contestable, seconds, 2 weeks to 3 years; None = never contested again pub election_delay: Option<u32>, // seconds after creation before the first charter; unbounded, none = at once pub max_added_moderators: u16, // members the leader may add after the election; 0 to 15, 0 by default pub moderated_document_types: BTreeMap<DocumentName, BTreeSet<ModerationAbility>>, // per type: DeleteDocuments, Ban, Suspend, Warn, ChangeDocumentFields pub interim: InterimModerators, // ContractOwner, AppointedModerators(set), NotYetUsable, NoModeration pub owner_protected: bool, // false by default } }
The declaration lives in packages/rs-dpp/src/data_contract/config/moderation/elected.rs. On the wire it is the third $type of the moderators, flat: {"$type": "elected", "seatContestable": true, "challengeCoolDown": 1209600, "moderatedDocumentTypes": {"post": ["ban", "deleteDocuments"]}, "interim": {"$type": "notYetUsable"}}, or "seatContestable": false without a challengeCoolDown, with joinWindow, voteWindow, electionDelay, maxAddedModerators and ownerProtected optional. Its parts:
- The election parameters are fixed once set (
SystemLimits:max_contract_moderation_election_window_secondscaps both windows at four weeks, and on mainnetmin_mainnet_contract_moderation_election_window_secondskeeps them at least a day; every other network takes a window of 0, so a test election can be run through in a block or two). The join window is how long applicants may join an election once the first one applied, the vote window how long masternodes then vote. The election delay is the one parameter the contract sets freely: how many seconds after its creation the first charter may be filed against it, the notice the contract gives before its first election can be called. It is optional and unbounded; left out, the election may be called at once. Because the declaration is made at the contract's creation and never changes, the creation is the declaration's own time. The charter contract'stargetContractIdreads it through themoderation: "electionOpen"requirement below.maxAddedModeratorssays how many members the leader of a seated team may add after the election, each one an identity that asked to join the team's proposal: additions ever filed, so a removal or a resignation frees no slot. It is 0 when left out, a team then being exactly what was elected, and at mostSystemLimits::max_contract_moderation_added_moderators(15). - The seat is contestable or not, and the contract says which:
seatContestableis required, with no default. A default of false would make every team permanent, leaving a contract nothing to do about a leader who lost its keys or went rogue, since the leader can not change and a challenge is the only remedy; a default of true would opt every contract into challenges without it asking. A contestable seat declares its challenge cool-down, how long a seated team is safe from a challenge after a seat change (min_contract_moderation_challenge_cool_down_secondstomax_contract_moderation_challenge_cool_down_seconds, two weeks to three years), and a seat that can not be contested declares none, since a cool-down means nothing there: a declaration missing the key, orseatContestable: truewithout the cool-down, orfalsewith one, does not parse. In Rust the two are one field,challenge_cool_down: Option<u32>(ElectedModerators::seat_contestableisis_some), so a declaration can not disagree with itself however it is encoded. Nothing reads the seat yet: challenges come after protocol version 14, a challenge then being a new contest on the samebyTargetContractindex of the charter contract, allowed only when the target declares its seat contestable. The key is there now because the declaration is frozen at the contract's creation. Until challenges ship a seat is never contested again, whatever the key says. - The moderated set is the document types the team moderates, each with the abilities the seated team holds on it: non-empty, each type a document type of the contract, each ability set non-empty and backed by the contract (
banneeds the banlist,suspendthe suspension list,warnthe warning list,deleteDocumentsthe type itself settingmoderatorAbilities.delete, so deletions reach only such types, within their window, andchangeDocumentFieldsthe type listingmoderatorAbilities.changeFields; and every type listingchangeFieldsmust be moderated withchangeDocumentFields, since once a team is seated only it writes those fields). The charter of a team will say how those types are moderated, never which. The lists stay contract-wide: an ability on a type is what a team may do over the documents of that type. The set also bounds the interim block. A charter does not price the moderators part of an action: a type's ownactionFees.moderatorsamount is the most a team may charge, a charter charges a share of it (the charter contract's business, not the declaration's), and the owner part stays what the type declares, immutable as before. - The interim says who moderates until a team is seated.
ContractOwnerandAppointedModerators(set)are the merged kinds, with their authority, their limit and their existence check (41110 at create): they moderate, they are protected, and they are the team that claims the moderators pot, all of it until a charter is seated and none of it after (see The seated team below).NotYetUsablenames nobody: nobody moderates, nobody claims the pot (it accumulates for the team to come,ContractFeeClaimNotAllowedErrorfor everyone), and the moderated document types can not be used. A contract that never attracts a team keeps those types unusable for good; the other types work as on an unmoderated contract.NoModerationnames nobody too, with the moderated types usable meanwhile: nobody moderates and nobody claims the pot, and every type works as on an unmoderated contract until a team is seated. - The owner flag says whether the contract owner is protected from the team once one is seated, as the owner and the moderators of the merged kinds are (41102 on a ban, a suspension or a deletion of its documents). Not protected by default. During the interim the owner is protected whenever it moderates, flag or not:
ContractModerationConfig::protectsis what the moderation transition checks, and it ismay_moderateor the flag for the owner.
Validation. validate_moderation_config v0 checks the declaration with the rest of the moderation config, from the raw document schemas of the create or update transition, and refuses with InvalidContractModerationConfigError (10900, unpaid): a window, or the cool-down of a contestable seat, outside its bounds, an empty moderated set, a moderated type the contract does not have, an empty ability set or an ability the type can not back, and an empty or oversized interim set. The create then checks that every interim moderator exists (41110), as it does for an appointed set.
The update. validate_config_update 2 refuses, with DataContractConfigUpdateError (40002), every change to the declaration, the interim included, and entering or leaving elected moderation. A contract that declares elected moderation is elected for good, and one that did not can not become so; the merged kinds still swap their moderators freely. An update may add document types; the declaration keeps naming the ones it named.
The interim block. The batch transformer's contract_moderation_gate v0 runs it before the lists: on an elected contract whose interim is NotYetUsable, every document transition of a moderated document type, deletions included (nothing of those types was ever written), is refused, paid, with ContractModeratedDocumentTypeNotYetUsableError (41200) and its contract nonce bump, in a block and in the mempool, until a charter is seated on the contract. Whether one is, is read (billed) only when a transition of the batch is on a type the block covers. The lists are read only for the transitions on the other types, and not at all when nothing is left. The interim moderators of the other two kinds moderate through the same transition, the same gate and the same claim as the merged kinds; a moderation transition against a NotYetUsable contract fails as by a non-moderator (41101).
The seated team. Seating writes nothing. A team applies with an electedCharter of the moderation charters contract, a create on its contested unique index byTargetContract, keyed by the target contract; awarding that contest writes the winner's document to the charter contract's storage, the only electedCharter ever written there for the target (contenders live in the contest, and in protocol version 14 a seat is never replaced, whether or not the declaration says it is contestable: another charter for a seated target is refused, paid, with DuplicateUniqueIndexError, 40105). So the charter seated on a contract is the one byTargetContract finds, and every moderation path reads it from there (execution/validation/state_transition/common/seated_moderation_charter in drive-abci): there is no block-end seating hook and no copy under the moderated contract. Its team is the charter's owner, the leader, plus the active members: its members less the memberId of every removedModerator for it, plus the memberId of every addedModerator for it (ElectedCharter::active_members), counting the documents that exist now, since the leader takes either back by deleting it. A resignationRequest changes nothing by itself; the leader acts on it by deleting the member's addition or removing an elected member.
- Who moderates. Once a charter is seated, only its team moderates: the leader and the active members, each alone. The interim moderators, the owner among them, can no longer act (41101), whatever the interim was. Deciding whether the signer is on the team reads no list of it: the leader costs nothing beyond the charter lookup, an elected member one point read of its removal, anyone else one point read of its addition (both types are unique on the charter and the member). What the interim did stands: its bans, suspensions, warnings and removals, which the team may lift.
- With what. The team holds the abilities the declaration gives it and no others. A deletion or a restore needs
deleteDocumentson the document type, and a field changechangeDocumentFields; a ban, a suspension or a warning, or lifting one, needs the ability on some moderated type, since the lists are contract-wide. Anything else is refused, paid, withContractModerationAbilityNotGrantedError(41201). - Who is protected. The leader and the active members can be neither put on a list nor have their documents deleted (41102), and the owner too when the declaration sets
ownerProtected. The interim moderators lose the protection they had. - How many join later. The leader adds members from the proposal's join requests, at most the target's
maxAddedModeratorsat a time, counting the charter's additions that exist now: the leader takes an added member off by deleting its addition, which frees the slot. An elected member is taken off with aremovedModerator, which may only name one of the charter'smembersand puts the member back when deleted. The schema cannot count documents, so the batch's state validation refuses the addition past the cap, paid, withModerationCharterAddedModeratorLimitReachedError(41202), after reading the charter, its target and at most the cap's number of additions, all billed; additions an earlier create of the same batch was accepted for count too. Like a unique index conflict it is judged in the block, not in the mempool, which runs no state validation for a batch. - On what grounds. Every ban, suspension, warning, document deletion and field change of the team names, in its reason's
reasonDocumentId, areasondocument its proposal lists: the grounds it asked to be elected on. Any other is refused, paid, in a block and in the mempool (ModerationReasonNotListedError, 41203), a reason naming none included; a proposal listing no reason is a team that can take no such action. The proposal is read, billed, only when the reason names a document. Lifting a ban, a suspension or warnings and restoring a document carry no reason and are not checked, and the interim is not bound before the seating. - What it charges. An action on a moderated type may agree to the charter's
moderatorsShareof the declared moderators part instead of the whole of it, and is then charged that (see Document action fees). An action agreeing to the declared amount reads no charter. - The pot. The interim team claims the moderators pot only until a charter is seated; its claim is refused after (41113), so the pot carries over to the seated team, unsettled. From then on the leader or an active member claims it for the team, at most once per epoch, and it is paid out by the proposal's
rewardSplit: the leader share to the leader; the equal share in equal parts to the other active members, or to the leader when it has none; and the action share between the whole team, the leader included, in proportion to the bans, suspensions, warnings and document deletions each one signed since the last settle (not field changes, whose number on one document has no bound) (the counts of the other tree's key48), or equally when nobody acted. Lifting and restoring do not count. Every share and every part rounds down to the credit; the few credits left stay in the pot for the next settle. A claim that would pay nobody a credit is refused (41112). The settle reads the team (the charter's removals and additions), the proposal and the counts, all billed, and deletes the counts. - Settled before every change. An
addedModeratororremovedModeratorcreated or deleted pays the pot out to the team as it was first, the same way, and resets the counts: a removed member is paid for what it did, a new one shares nothing earned before it came, one coming back nothing earned while it was away. The settle ignores the once-per-epoch limit and is not a claim: it writes no last claim, and the team may still claim in the same epoch. It is an effect, never a refusal, judged by the batch's state validation once the change passed (the addition its cap too), which the mempool does not run; at most one per target contract per batch.
Referencing an elected contract. A document type that must point at a contract of this kind says so in its reference: "refersTo": { "type": "contract", "contractRequirements": { "moderation": "elected" } }. contractRequirements holds what the referenced contract must declare beyond existing, each key an aspect of the contract with a closed set of values or a bound: moderation: "elected", or moderation: "electionOpen", which also requires the contract's own election delay to have passed since its creation, or the contract to declare none (the delay between a contract's creation and the first charter against it, so a team cannot be seated before anyone has seen the contract, set by each contract for itself). Both have a user in the charter contract: a charter proposal only needs the target to be elected, so teams can form during the notice, and the charter that opens the contest needs its election electionOpen; minimumAgeSeconds, a number of seconds the reference fixes, which requires the contract's recorded creation time to be at least that far before the block time of the write; minimumSecondsSinceUpdate, the same of the later of the contract's creation and last update times (any update restarts the clock; an elected declaration can not be added by an update, so this one is for other uses than the charter); owner, "self" requiring the referenced contract to be owned by the writer of the referring document (its $ownerId, a write gate like the $ownerId property agreement of a document reference) and "other" by anyone else (so a charter may forbid an owner from chartering its own team); readonly: true, requiring the referenced contract's config to be read-only, one that can never be updated again (which makes minimumSecondsSinceUpdate moot for the same target); keepsHistory: true, requiring its config to keep history (only true is declarable for either flag); and ownerProtected, requiring the contract's elected moderation declaration to protect the owner from the team (true) or to leave it unprotected (false), which implies elected moderation without the schema having to say so, a contract without an elected declaration meeting neither value. A contract created before contracts recorded their creation time never meets a duration, its own election delay included. Consensus checks them when the referring document is written, against the contract it has already fetched for the existence check and the write itself (its owner and block time), so they cost no further read; a contract that exists but does not meet a requirement refuses the write, paid, with ReferencedContractRequirementNotMetError (40135) naming the requirement, where a contract that does not exist is still 40120. A replace re-checks them when it changes the reference. owner is the one requirement judged against the writer, and a transfer or a purchase changes the writer without any write, so on a document type whose documents can be transferred or traded a reference carrying owner is re-checked on every replace, touched or not, as the $ownerId writer gate is: after a transfer the new owner has to repoint it at a contract that meets the requirement for them, or clear it where it is optional. The reference is then checked whole, its other requirements included. Because the new owner must be able to repoint it, registration refuses such a reference held by an immutable property on a type whose documents can be transferred or traded (the property itself, inside an immutable object, or the elements of a typed array), with InvalidContractStructure. On a type whose documents stay with their owner the writer never changes and neither does a contract's owner, so nothing is re-checked. The other requirements are facts about the referenced contract, not the writer, and never bring a reference back on their own. A changed contractRequirements is an incompatible schema change on update, like the rest of a refersTo. The charter system contract's targetContractId is the first user.
Referencing an identity key with requirements. The same shape serves the key references the charter contract needs: "refersTo": { "type": "identityPublicKey", "keyIdProperty": "recipientKeyId", "keyRequirements": { "purpose": "decryption", "boundTo": "submittedCharter" } }. keyRequirements holds what the referenced key must be beyond existing and not being disabled, each key an aspect of the key: purpose, the key's purpose by its wire name (authentication, encryption, decryption, transfer, voting or owner; never system), and boundTo, the name of a document type of the declaring contract, which requires the key's contract bounds to be exactly the declaring contract and that document type; a whole-contract bound or a contract group bound never meets it, even where the group holds the type, since the check reads nothing beyond the key. Registration (create_document_types_from_document_schemas 1, a post-pass edited in place since it is inert before protocol version 14, under full validation like the meta-schema) checks that boundTo names a document type the contract has, so the write-time check never needs a second contract fetch, and that a key meeting the pair can exist at all: only authentication, encryption and decryption keys carry a document type bound, and Drive registers an encryption or decryption key bound to a document type only when that type declares requiresIdentityEncryptionBoundedKey or requiresIdentityDecryptionBoundedKey, so a boundTo paired with transfer, voting or owner, or with an encryption purpose on a type without the matching keyword, is refused as a requirement no key could ever meet. Consensus checks the requirements when the referring document is written, against the key it has already fetched for the existence check, so they cost no further read; a key that exists and is enabled but does not meet one refuses the write, paid, with ReferencedIdentityKeyRequirementNotMetError (40136) naming the document type, the property, the requirement and what the key has, where a missing key is still 40123 and a disabled one 40124. A replace that repoints the reference at another key, through either the identity id or the key id, re-checks them. A changed keyRequirements is an incompatible schema change on update, like the rest of a refersTo. New requirements (a security level, say) are new keys of the same object, never a new reference type. The charter contract's joinRequest.recipientId (a decryption key bound to submittedCharter) is the first user.
What comes next. After protocol version 14, challenges of a contestable seat, amendments and threshold actions beyond the deletion of settled documents. Issue #4865 holds the design.
Versioning Touchpoints
All in place for protocol version 14: CONTRACT_VERSIONS_V6 makes config V2 the config of every new contract (max_version and default_current_version 2) and validate_config_update 2; STATE_TRANSITION_SERIALIZATION_VERSIONS_V3 and DRIVE_ABCI_VALIDATION_VERSIONS_V10 carry the transition's slots and batch_state_transition.contract_moderation_gate, and the contract update's basic structure moves to 2 to validate the declaration; DRIVE_CONTRACT_METHOD_VERSIONS_V4 bumps insert_contract to 2 and adds the moderation table (its update_contract 2 belongs to token distribution and does nothing for moderation); DRIVE_STATE_TRANSITION_METHOD_VERSIONS_V4 adds the converter slot and bumps documents_batch_transition to 1 for the sweep; DRIVE_VERIFY_METHOD_VERSIONS and DRIVE_ABCI_QUERY_VERSIONS gain their moderation tables; SYSTEM_LIMITS_V4 gains max_contract_moderators, max_contract_suspension_until, max_contract_moderation_reason_length and max_contract_warnings_per_identity. The warning list adds two slots to DriveContractModerationMethodVersions (add_contract_warning, remove_contract_warnings) and nothing else: the list, the two actions and the two errors join a feature no release contains.
The document deletion adds, all for protocol version 14 as well: the moderatorAbilities.delete keyword in meta-schema v3 (CONTRACT_VERSIONS_V6 already selects it); five slots in DriveContractModerationMethodVersions and one in the verify and query tables; and batch_operations.apply_drive_operations = 1 in DRIVE_VERSION_V9, the generation that forfeits the refund. The transition's own tables do not move: the action joins a transition no release contains.
The field change moves no table either: the action, the transform, the converter, the prover's and the verifier's arms sit in generations only protocol version 14 selects, and the batch transformer's gate on an owner's create or replace (v0, edited in place) reads nothing for a type that keeps no such field, which no earlier version can register. The uniqueness check a restore and a field change share is validate_moderated_document_uniqueness in DriveDocumentIndexUniquenessMethodVersions, still 0, and takes the changed fields.
The deletion of settled documents adds, for protocol version 14 as well: the moderatorAbilities.deleteSettled key in meta-schema v3, the DeleteSettledDocument and ApproveTeamAction actions and the VerifiedContractTeamActionSignature proof result, all appended; eight slots in DriveContractModerationMethodVersions (add_contract_team_action_signature, fetch_contract_team_action, fetch_contract_team_actions, fetch_contract_team_action_signers, prove_contract_team_actions, prove_contract_team_action_signers, insert_contract_team_action_trees, add_estimation_costs_for_contract_team_action), three in the verify table (verify_contract_team_actions, verify_contract_team_action_signers, verify_contract_team_action_signature) and two in the query table (contract_team_actions, contract_team_action_signers); and max_moderation_charter_elected_members (15) in SYSTEM_LIMITS_V4, backfilled into the earlier tables, which nothing reads there. The transforms and the converter sit in generations only protocol version 14 selects. The prover's and the verifier's arms are in the shipped prove_state_transition v0 and verify_state_transition_was_executed_with_proof v0, which protocol versions 1 to 13 select too, edited in place: only a contract user moderation takes them, a transition inactive before protocol version 14 (should_not_be_active_before_protocol_version_14), so no earlier proof changes. The storage refund forfeiture of apply_drive_operations generation 1, which only protocol version 14 selects, now takes the document operations alone, applied as a GroveDB batch of their own when the batch also frees moderation storage someone is owed.
Elected moderation moves no table: the declaration is a variant of the same config V2, validate_moderation_config v0 and validate_config_update 2 take it on in place while protocol version 14 is unreleased, contract_moderation_gate v0 runs the interim block, and SYSTEM_LIMITS_V4 gains the four bounds of the windows and the cool-down. The seated team moves none either: the moderation transition's state v0, the claim's, the gate v0 and the batch transformer's state v2 read the charter in place, all generations no release selects; the cap on additions is a hook in the batch's shipped validate_state v0 that only a create of the charter contract reaches, a contract absent from state before protocol version 14. The pot, the counts and the reasons add four slots to DriveContractModerationMethodVersions (set_contract_moderation_action_count, fetch_contract_moderation_action_counts, remove_contract_moderation_action_counts and their estimation), 0 at every version. The forced settle is a second hook in the same validate_state v0, reached only by a create or delete of the charter contract's team changes; the claim's action, the moderation transition's action and the batch action carry what they settled into converters no release selects (contract_fee_claim_transition 0, contract_user_moderation_transition 0, documents_batch_transition 1); and the claim's arm of the shipped prove_state_transition v0 and verify_state_transition_was_executed_with_proof v0 proves the claimant alone for an elected contract, a transition no earlier version admits.
What Is Not There Yet
Deleting indexOnly documents (the action would have to carry the owner and the values), deleting every document of an identity at once, action fees on token transitions, group-based moderators (AuthorizedActionTakers::Group through group actions), keys bound to the contract allowed to sign its moderation, ban codes declared by the contract (the reason's code is where they will go), the moderator's id on a ban or a suspension (a warning carries its block time but not who issued it), retracting one warning rather than all, a warning that expires by the clock, a contract-declared strike count that turns warnings into a suspension, a query of the moderation action counts (a client reads them with a raw GroveDB proof today), challenges and amendments of a seated charter, and the Swift and Kotlin SDKs. The refusal a barred identity receives (41107, 41108, 41114) does not repeat the reason: the status query does.
Tests
packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/moderator_abilities_tests.rs: the object's shape on both paths,delete's rules and the window's (it needsdeleteand a clock,$updatedAtor for documents that never change$createdAt),deleteSettled's (its shape and defaults, the window and the elected declaration it needs, its bounds, the$createdAtapproversPredateDocumentneeds), and every rulechangeFieldsholds its properties to, the revision it keeps included;validate_update/v1: the abilities frozen across updates, the settled-deletion rule among them.packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests/moderator_fields.rs: a moderator's change stored, proved and paid for, anullremoving a field, a report resolved after its post was deleted, every refusal of the transform, the create and replace gate for owners who do not moderate, a replace built before the change refused on its revision, and the unique indexes over the fields;seated_team.rs: a seated team writing the fields the declaration gives itchangeDocumentFieldson, and its members, not the off-team owner, writing them in their own documents;retraction.rs: a banned and a suspended author retracting a document it can not delete while its edits, a retraction breaking the type's own rule and taking a retraction back are refused, the same judgement in the mempool, and a type withoutretractedWhenrefusing every replace of a barred author.packages/rs-drive/src/drive/contract/moderation/document_removal_tests.rs: the trees created with the contract and with a document type an update adds, records written, replaced, read by ids and by page with proofs that verify to the same, the bounds of a read, estimate against applied cost, and a moderator's deletion refunding nobody where the author's own refunds the author.packages/rs-drive/src/drive/contract/moderation/team_action_tests.rs: the team action trees created with an elected contract (the creation estimated first) and none without the rule, proposals and approvals written, an action closed with its approvals moved and nothing left active, read and paged with proofs that verify to the same, the proof of an approval before and after its action closed, the bounds of a page, the estimate against applied cost up to a team's worth of approvals and the longest reason, the approvals of members who left dropped and refunded, and the batch that deletes forfeiting the post's refund while refunding the proposer and the approvers the approvals it moves, the approver who wrote the post only its approval;types.rs: the action's encoding and every malformed one refused.packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests/seated_team/settled.rs: a settled story deleted once the leader and two members approve a member's proposal and not while three members without the leader have, the removal record with the proposal's reason, the counts of every approver and none before, the proof of the proposal before and after the action ran, an approval of an action that ran refused, the leader alone deleting a memo by its proposal, a proposal refused within the window, before a team is seated, on a type without the rule, off the team and for an unlisted reason, an approval of nothing proposed and off the team refused, the approvals of members who left dropped and refunded a month later and a member who came back approving again (on a type whose rule counts members whenever added), an approval refused once a moderator changed the document's fields or its author replaced it, the proposer refunded the action the deleting approval moves, a removed member's seat counted toward the whole team, a member added after a story (or in its block) refused as proposer and approver while one added before counts, and a member taken off and added again after the story dropped at the next team read (one point read each, and in one read of the team);query/contract_moderation_queries/contract_team_actionsandcontract_team_action_signers: the queries, their proofs, and the requests they refuse.packages/rs-drive-abci/src/query/contract_moderation_queries/contract_document_removals: the query by ids and by page, its proof read back by the verifier, and every request it refuses.packages/rs-dpp/src/data_contract/config/moderation/mod.rsandconfig/methods/validate_update/v2: the declaration's rules and the update rules, the elected declaration's among them (every bound, the seat and its cool-down, the moderated set, the envelope, the maximums, the interim set, the wire shape, and an update refused for each field and for entering or leaving);moderation/elected.rs: what each interim kind allows.packages/rs-drive/src/drive/contract/moderation/tests.rs: tree creation on insert, the trees and their entries surviving a contract update, every writer with estimation, status and page proofs, paging, the refund going to the first moderator after another one replaces its suspension, a status proof over one list saying nothing about the other, the warning list tree created only when declared, warnings accumulating under the moderator that warned last and cleared with a refund to it, a warn never estimated below its cost up to the fullest entry, and the banlist on top of the other tree with every combination of lists.packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/batch/transformer/v0/contract_moderation_gate/mod.rs: the gate is silent before protocol version 14 and for an unmoderated contract, refuses each barred operation of one batch on its own while keeping the deletions, passes a barred signer's replaces on a type declaringretractedWhenwith the bar, and blocks the moderated types of an elected contract in its interim without reading the lists for them.packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests/seated_team/pot.rs: a seated team's claim split by its reward split with every part rounded down and the remainder left in the pot, the equal split of the action share when nobody acted, the counts per signer (counted actions only, not the interim's) reset by a claim and by a change of the team, the settle before an addition, before a removal and before either is undone (in an epoch already claimed, and leaving the epoch's claim to the team), and the claim's proof with the claimant's balance;seated_team/reasons.rs: a seated team's bound action refused without a listed reason in a block and in the mempool, a proposal with no reason, reversals and the interim unbound;packages/rs-dpp/src/moderation_charter/reward_split.rs: the split's arithmetic;packages/rs-drive/src/drive/contract/moderation/action_count_tests.rs: the counts tree created with an elected contract only, written, read, bounded and reset, and the banlist on top of an elected contract's other tree.packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests/seated_team.rs: a contest for an elected contract's seat awarded, then the leader and the members moderating instead of the interim, another charter for the seated target refused whether or not its seat is contestable, additions and removals, the protection of the team, the cap on additions, abilities the declaration does not give, the interim block ending, a discounted fee charged and read where the declared one reads nothing, every other discount refused in a block and on recheck, and the interim's claim refused once a charter is seated.packages/rs-drive-abci/src/execution/validation/state_transition/state_transitions/contract_user_moderation/tests.rs: the whole pipeline, including a warned user carrying on with its warnings accumulating, proved and cleared, the warning limit and the clearing that lifts it, warnings kept through a ban and out of the gate and the ban's proof, every refusal of a warn, and the warning list fixed at creation; the moderators' window (a deletion to the millisecond it ends on, refused one later for the contract owner too while the author's own still passes, reopened by a replace, measured from$createdAton a type that never changes, fixed on update), a moderator deleting a post (record, execution proof, the author's balance unchanged, and the control where the author deletes it and is refunded), every refusal of a deletion, an update adding a document type moderators can delete from, a permanent reference to such a type refused, the mempool refusal, the lapse sweep, the moderator set, every refusal code, the lists staying as the contract was created with them, a barred identity deleting its own documents in a block and in the mempool, a barred identity refused as the recipient of a transfer and as the seller of a purchase, the ban's proof covering the suspension it removed, lifting the entry of an identity an update made moderator, the per-list execution proof, a named owner, a create or an update naming a moderator that does not exist, an update keeping its moderators, inactivity of the transition and of a moderated contract create or update before protocol version 14, and elected moderation (each interim kind moderating or not, the block of a moderated type next to an unmoderated one in a block and in the mempool, a create refused outside a bound and for an unknown type, and an update refused for a changed field and for entering or leaving).
Documents
If data contracts are the tables, then documents are the rows. A document is an instance of a document type defined within a data contract. When a user creates a profile on DashPay, submits a domain name on DPNS, or stores any application data on the platform, they are creating a document.
Documents are the most fundamental unit of user data on Dash Platform. They are stored in GroveDB (through Drive), indexed for efficient querying, and cryptographically provable. Understanding how documents work at the Rust level is essential for working with the platform codebase.
The Document Enum
Like DataContract and Identity, Document is a versioned enum. From packages/rs-dpp/src/document/mod.rs:
#![allow(unused)] fn main() { #[derive(Clone, Debug, PartialEq, From)] pub enum Document { V0(DocumentV0), } }
Currently there is only one variant, V0. But the enum wrapper is already in place so that future protocol versions can introduce a V1 variant without breaking existing code. All code that works with documents goes through the accessor traits, so adding a new variant is purely additive.
What Lives Inside a Document
The DocumentV0 struct is defined in packages/rs-dpp/src/document/v0/mod.rs:
#![allow(unused)] fn main() { pub struct DocumentV0 { pub id: Identifier, pub owner_id: Identifier, pub properties: BTreeMap<String, Value>, pub revision: Option<Revision>, pub created_at: Option<TimestampMillis>, pub updated_at: Option<TimestampMillis>, pub transferred_at: Option<TimestampMillis>, pub created_at_block_height: Option<BlockHeight>, pub updated_at_block_height: Option<BlockHeight>, pub transferred_at_block_height: Option<BlockHeight>, pub created_at_core_block_height: Option<CoreBlockHeight>, pub updated_at_core_block_height: Option<CoreBlockHeight>, pub transferred_at_core_block_height: Option<CoreBlockHeight>, pub creator_id: Option<Identifier>, pub contract_version: Option<u32>, } }
Let us walk through the key fields:
-
id: A 32-byte unique identifier. Unlike contract IDs, document IDs are derived from a combination of the contract ID, owner ID, document type name, entropy and (protocol v14+) the identity contract nonce of the create transition. This makes them deterministic, unique, and impossible to produce twice. -
owner_id: The identity that currently owns this document. Ownership can change if the document type supports transfers. -
properties: The actual application data, stored as aBTreeMap<String, Value>. TheValuetype comes fromplatform-valueand can represent strings, integers, byte arrays, nested maps, and arrays. The keys correspond to the property names defined in the document type's JSON Schema. -
revision: AnOption<Revision>(which is au64). Mutable documents track revisions -- each update increments the revision. Immutable document types will haveNonehere. -
Timestamps: Six pairs of timestamp fields covering three events (creation, update, transfer) across three time references (milliseconds, block height, core block height). Whether these are populated depends on the document type schema -- if the schema requires
$createdAt, the platform fills it in when the document is created. -
creator_id: The original creator of the document. This differs fromowner_idwhen a document has been transferred to a new owner. -
contract_version: The data contract version this document's bytes conform to — the contract version stamp (protocol v14+, document serialization format 3). Drive assigns it whenever document content is supplied (create and replace) and preserves it through transfers and purchases.Nonemeans the document was serialized before format 3, which predates everyrequiredSinceannotation. The stamp resolves per-property byte layouts when a document type gains required properties through contract updates — see the Document Serialization chapter.
Document ID Generation
Document IDs are not random -- they are derived deterministically. From packages/rs-dpp/src/document/generate_document_id.rs:
#![allow(unused)] fn main() { impl Document { pub fn generate_document_id_v1( contract_id: &Identifier, owner_id: &Identifier, document_type_name: &str, entropy: &[u8], identity_contract_nonce: IdentityNonce, ) -> Identifier { let mut buf: Vec<u8> = Vec::with_capacity(/* ... */); buf.extend_from_slice(DOCUMENT_ID_V1_DOMAIN_TAG); // b"dash:document-id:v1" buf.extend_from_slice(contract_id.as_slice()); buf.extend_from_slice(owner_id.as_slice()); buf.extend_from_slice(document_type_name.as_bytes()); buf.extend_from_slice(entropy); buf.extend_from_slice(&identity_contract_nonce.to_be_bytes()); Identifier::from(hash_double(&buf)) } } }
The ID is a double SHA-256 hash of a domain tag, the contract ID, owner ID, document type name, client-provided entropy and the identity contract nonce of the create transition. Document::generate_document_id picks the derivation from the platform version; consensus recomputes it for every create and rejects a mismatch with InvalidDocumentTransitionIdError. This means:
- The ID commits to the owner, so nobody else can take it, and to the contract and document type, preventing cross-contract collisions.
- The ID is a deterministic function of its inputs: whoever knows the entropy and the nonce, which is the client building the create transition, can compute it before the document exists (see below). The entropy is the one input other parties can not guess, so as long as the client generates it unpredictably, nobody else can compute the ID of a document that has not been sent yet and point other documents at it in advance.
- The nonce makes the ID single use. An identity contract nonce is consumed at most once, so an ID can be produced at most once.
Why the nonce is part of the ID
Up to protocol version 13 the ID was generate_document_id_v0: the same hash without the domain tag and the nonce. The create check only asks whether a document exists under the ID right now, so the owner of a deleted document could create a new document with the same entropy and get the same ID back, with different content. Everything that referenced the ID (likes, replies, a refersTo property, a moderation removal record) then pointed at the new content. For a document type with documentsMutable: false and canBeDeleted: true that is content substitution, the very thing immutability is supposed to rule out.
From protocol version 14 a reference to a document ID means that one document or nothing. This also holds for documents created before the upgrade: their entropy-only IDs can not be produced by the new derivation, and the old derivation is no longer accepted.
What this means for clients
The ID of a new document only exists once the nonce of its create transition is assigned, and it changes if the transition is rebuilt with another nonce:
- The ID a
Documentcarries before its create transition is built (for example the onecreate_document_from_datagives it) is a placeholder.DocumentCreateTransitionV0::from_documentreplaces it with the derived ID, so every transition built through dpp carries the right one. - On the Rust path (rs-sdk, or
from_documentdirectly) theDocumentyou passed in keeps its placeholder: read the ID from the transition, or from the confirmed documentput_to_platform_and_wait_for_responsereturns. - To know IDs up front (a chain of documents that reference each other), assign the nonces first: nonces may be used out of order within a window of 24.
- JavaScript gets the same through
wasm-dpp2, with one difference:new DocumentCreateTransition({ document, identityContractNonce })derives the ID for the network's protocol version (platformVersionoption, latest by default) and writes it both onto the transition and back ontodocument, so after constructiondocument.idis the final ID and may be read from there.Document.generateId(type, owner, contract, entropy, identityContractNonce)anddocument.setIdForCreation(identityContractNonce)give the ID before the transition exists, andnew Document({ ..., identityContractNonce })derives it at construction (an explicitidpassed alongside the nonce must equal the derived one). ADocumentbuilt without a nonce carries the entropy-only placeholder until it is passed toDocumentCreateTransition. No app needs to reimplement the hash.
The Accessor Traits
Documents follow the same accessor-trait pattern as data contracts. The getter trait is defined in packages/rs-dpp/src/document/accessors/v0/mod.rs:
#![allow(unused)] fn main() { pub trait DocumentV0Getters { fn id(&self) -> Identifier; fn owner_id(&self) -> Identifier; fn properties(&self) -> &BTreeMap<String, Value>; fn properties_mut(&mut self) -> &mut BTreeMap<String, Value>; fn revision(&self) -> Option<Revision>; fn created_at(&self) -> Option<TimestampMillis>; fn updated_at(&self) -> Option<TimestampMillis>; fn transferred_at(&self) -> Option<TimestampMillis>; fn created_at_block_height(&self) -> Option<u64>; fn updated_at_block_height(&self) -> Option<u64>; fn creator_id(&self) -> Option<Identifier>; fn contract_version(&self) -> Option<u32>; // ... and more } }
The setter trait extends it with mutation methods and also provides convenient typed setters:
#![allow(unused)] fn main() { pub trait DocumentV0Setters: DocumentV0Getters { fn set_id(&mut self, id: Identifier); fn set_owner_id(&mut self, owner_id: Identifier); fn set_properties(&mut self, properties: BTreeMap<String, Value>); fn set_revision(&mut self, revision: Option<Revision>); fn set_created_at(&mut self, created_at: Option<TimestampMillis>); fn set_updated_at(&mut self, updated_at: Option<TimestampMillis>); // Generic property access via path syntax fn set(&mut self, path: &str, value: Value) { ... } fn remove(&mut self, path: &str) -> Option<Value> { ... } // Typed setters for common types fn set_u8(&mut self, property_name: &str, value: u8); fn set_u64(&mut self, property_name: &str, value: u64); fn set_bytes(&mut self, property_name: &str, value: Vec<u8>); // ... and more } }
Notice the set() method provides lodash-style path syntax: "root.people[0].name". Parents are created automatically if they do not exist.
The DocumentMethodsV0 Trait
Beyond simple field access, documents have behavior defined by the DocumentMethodsV0 trait in packages/rs-dpp/src/document/document_methods/mod.rs:
#![allow(unused)] fn main() { pub trait DocumentMethodsV0 { fn get_raw_for_contract( &self, key: &str, document_type_name: &str, contract: &DataContract, owner_id: Option<[u8; 32]>, platform_version: &PlatformVersion, ) -> Result<Option<Vec<u8>>, ProtocolError>; fn get_raw_for_document_type( &self, key_path: &str, document_type: DocumentTypeRef, owner_id: Option<[u8; 32]>, platform_version: &PlatformVersion, ) -> Result<Option<Vec<u8>>, ProtocolError>; fn hash( &self, contract: &DataContract, document_type: DocumentTypeRef, platform_version: &PlatformVersion, ) -> Result<Vec<u8>, ProtocolError>; fn increment_revision(&mut self) -> Result<(), ProtocolError>; fn is_equal_ignoring_time_based_fields( &self, rhs: &Self, also_ignore_fields: Option<Vec<&str>>, platform_version: &PlatformVersion, ) -> Result<bool, ProtocolError>; } }
The get_raw_for_contract and get_raw_for_document_type methods retrieve a document property as raw bytes, using the document type schema to determine how to serialize the value. This is critical for building index keys and storage operations.
The is_equal_ignoring_time_based_fields method is particularly useful in validation. Since timestamps and block heights are set by the network (not the client), you often want to compare two documents while ignoring those fields -- for example, to verify that a client's update only changed the fields it was supposed to change.
Version Dispatching in Methods
Every method in the Document implementation dispatches through the platform version, following the standard pattern:
#![allow(unused)] fn main() { impl DocumentMethodsV0 for Document { fn get_raw_for_contract( &self, key: &str, document_type_name: &str, contract: &DataContract, owner_id: Option<[u8; 32]>, platform_version: &PlatformVersion, ) -> Result<Option<Vec<u8>>, ProtocolError> { match self { Document::V0(document_v0) => { match platform_version .dpp .document_versions .document_method_versions .get_raw_for_contract { 0 => document_v0.get_raw_for_contract_v0( key, document_type_name, contract, owner_id, platform_version, ), version => Err(ProtocolError::UnknownVersionMismatch { method: "DocumentMethodV0::get_raw_for_contract".to_string(), known_versions: vec![0], received: version, }), } } } } } }
This is a double dispatch: first on the document variant (V0), then on the method version from the platform version configuration. This allows the platform to evolve both the document structure and the behavior of document methods independently.
How Documents Reference Their Contract
Documents do not carry a reference to their contract inside the struct itself. Instead, the relationship is established through context -- the document type name and contract are passed alongside the document whenever they are needed (for serialization, validation, hashing, and storage).
When serializing, a document is always serialized relative to its document type:
#![allow(unused)] fn main() { let serialized = <Document as DocumentPlatformConversionMethodsV0>::serialize( &document, document_type, // the schema determines field order and encoding &contract, platform_version, )?; let deserialized = Document::from_bytes( &serialized, document_type, // same schema needed for decoding platform_version, )?; }
This means a document's binary representation is not self-describing. You need the document type definition to interpret the bytes. This is a deliberate design choice for storage efficiency -- field names are not repeated in every serialized document.
The INITIAL_REVISION Constant
When a new document is created, it starts at revision 1:
#![allow(unused)] fn main() { pub const INITIAL_REVISION: u64 = 1; }
Revision 0 is never used for active documents. This allows 0 to serve as a sentinel value meaning "no revision" in some contexts.
Document References (refersTo)
From protocol version 14 a property of a document type can declare what it points at, and consensus refuses a create or replace whose target does not exist when the document is written (the reference is a write-time constraint only; nothing resolves it for a reader). The keyword is refersTo on the property, its type one of identity, contract (optionally with contractRequirements, see Contract Moderation), token, permanentDocument, moderatedDocument (see A document only moderators remove), deletableDocument and identityPublicKey, or a reference expression combining several with anyOf and allOf (see Reference expressions). A document reference may find its document by the values of a unique index instead of by id (findBy, see Found by a unique index), check the document found (where, see Checked on the document found), and, on a permanentDocument, take its value as an element of a list the document holds (inList, see An element of a list). Every form sits on an identifier property, with one exception below. The parsed shape is DocumentPropertyType::IdentifierWithReference(target), and any change to a declaration on contract update is an incompatible schema change.
An identityPublicKey reference names one key of one identity, and comes in two forms that differ in which property carries what:
- On the identity property. The identifier property carries the identity id and
keyIdPropertynames the sibling integer property carrying the key id:"refersTo": { "type": "identityPublicKey", "keyIdProperty": "toKeyIndex" }. Consensus fetches the named key of that identity. - On the key id property. The integer property carries the key id and
identityPropertynames whose key it is:"refersTo": { "type": "identityPublicKey", "identityProperty": "$ownerId" }. The property must declare exactly the range of aKeyID("type": "integer", "minimum": 0, "maximum": 4294967295) and the declaration takes nokeyIdProperty.identityPropertyis$ownerId(the writer),$creatorId(the document's creator, only on a document type that records creator ids: a transferable or tradeable type of a format-1 contract) or the path of an identifier property of the same document type (which must exist, be an identifier and not carry anidentityPublicKeyreference of its own); the last two are checked at contract registration.keyRequirementssit on this form exactly as on the identifier form. The parsed shape isDocumentPropertyType::KeyIdWithReference(KeyIdReference), the identity source plus the requirements, sized, encoded and queried exactly as a plainu32.
"senderKeyId": {
"type": "integer", "minimum": 0, "maximum": 4294967295,
"refersTo": { "type": "identityPublicKey", "identityProperty": "$ownerId" },
"position": 3
}
Both forms share the state check (validate_referenced_identity_key_v0 in the document reference validation): the key must exist and not be disabled, else the write is refused, paid, with ReferencedIdentityKeyNotFoundError (40123) or ReferencedIdentityKeyDisabledError (40124). Identity keys can be disabled but never removed, so a validated reference never dangles. The owner form's identity is the transition's signer, which the transition already proved exists, so the key fetch is its only read; a key id of some other identity's key is meaningless by construction, there is no property to name another identity. On replace the identity form is re-validated when either the identity property or its key id property changed. For the key id form it depends on where the identity comes from. $ownerId is the writer, transition metadata that never appears among the changed fields, and the document may have changed hands since the key id was written, so the reference is re-validated on every replace, touched or not, as the $ownerId writer gate is: after a transfer the new owner has to repoint the key id at one of its own keys. $creatorId never changes, so it is re-validated when the key id changed. A document written before its type recorded creator ids (no type did before protocol version 10, nor one of a contract whose config was still version 0) records none, and a contract update may add a $creatorId key id property to its type: setting that key id on such a document is refused (ReferencedKeyIdPropertyInvalidError, 40125), while a replace leaving it unset is not checked. A property path is re-validated when the key id or that property changed, and a key id set while the property is not is refused (ReferencedKeyIdPropertyInvalidError, 40125). A transfer itself is not checked in any form, so the reference governs writing, not holding. The declaring property must carry exactly the key id range in its schema, whatever the contract's integer sizing setting, and a keyIdProperty of the identity form may not name a property that carries this form, nor may a path name an identifier carrying an identityPublicKey reference: one pair is declared once (40125 at registration). The charter contract's joinRequest.senderKeyId, the owner's encryption key a shared secret is derived from, is the first user.
A document only moderators remove (moderatedDocument)
The three document references are disjoint, by what can make the referenced type's documents leave state (DocumentTypeV2Getters::document_reference_kind, a DocumentReferenceKind): nothing, for permanentDocument; only a removal by the contract's moderators that leaves a record, for moderatedDocument; anything else, for deletableDocument. A type is a moderated one when its owners can not delete its documents (canBeDeleted: false, not "onlyWhenConsumed", whose documents a consume deletes without a record), it declares no ttl, and moderatorAbilities.delete keeps a removal record (deleteKeepsRecord, true unless declared false). None of these can change on a contract update, so the kind a type admits holds for good. Registration (validate_data_contract_references 0) and the write (the document reference validation) refuse a declaration of another kind through one helper, document_reference_kind_mismatch: a moderatedDocument reference to any other type with ReferencedDocumentTypeNotModeratedError (state code 40143), a deletableDocument reference to a moderated type with ReferencedDocumentTypeModeratedError (40144), and the two earlier mismatches as before (40122, 40131).
In Rust the declaration is DocumentPropertyReferenceTarget::ModeratedDocument { contract_id, document_type_name, property_agreement }, appended to the enum so every earlier variant keeps its consensus encoding; as_any_document_reference returns it with kind: DocumentReferenceKind::Moderated. It is found by its id alone: the parser refuses findBy (a moderator's removal frees the document's unique index keys, so a key could come to find another document while the reference is held to the removed one's record), inList, an operand of a reference expression and a writer or creator reference.
The document must exist when the reference is written. A replace re-validates it as a permanentDocument reference (binds_a_changed_property): when the value changed, when the referring side of a where entry changed, and on every replace for a writer gate. When the value is one the stored document already held (a single reference whose path did not change, or an element the stored list held) and the document is not in state, the validation reads the removal record (fetch_contract_document_removal_with_fee, billed) and, when one stands unrestored, the reference holds through it (validate_pairs_against_removal): each where entry the replace asks about again compares the referenced $ownerId with the record's document_owner_id (a removed document can not be transferred, and a restore brings it back as it was) and $id with the id, and any other referenced property refuses the replace with ReferencedDocumentRemovedError (40145). An entry the replace does not ask about held when it was last checked. A value the write sets must name a document in state (ReferencedEntityNotFoundError, 40120). Whether a value is held is decided from the stored value (kept_value_changed_fields): a value under an unchanged top-level property is the stored one, and under a changed one (an object whose other property changed, or a typed array a replace changed) it is held when the stored value the replace action carries at the same path is it, or holds it. The parser admits a writer gate on a moderated reference against the referenced $ownerId or $id only (10231), since a writer gate is asked about on every replace. The write-time check lets a deletableDocument reference to a moderated type through, one a contract registered before the kind existed may hold; registration refuses it.
A chained query or a composite by-id join through a moderated reference proves the removal records of the joined ids beside their documents, one more component of the merged proof (query::moderated_join::removals_path_query: the ids under [64, contract, 2, 16, <document type>], walking as the join does, unlimited as the by-ids fetch is). A composite query builds one records component per referenced document type, over every id its moderated joins derived. The component names every joined id, not only the missing ones, since the verifier builds the merged query before it knows which are missing, and grovedb proves each queried key present or absent, so a prover can pass neither a removed document off as live nor a live one off as removed. A joined id with no document is reported with its record (ChainedDocumentsResult::removed_outer_documents, CompositeDocumentsResult::sub_result_removals, on the wire removed_outer_documents = 4 and removed = 4, clients removedOuterDocuments and removed); one with neither a document nor an unrestored record is refused, as a missing permanentDocument target is.
Checked on the document found (where)
A permanentDocument, moderatedDocument or deletableDocument reference may carry where: 1 to 10 entries { "<referenced property>": "<referring value>" }, each an equality the document found must meet when the referring document is written. The key is a property of the referenced document type, or its $ownerId, $creatorId or $id; the value is a property of the declaring document type, or "$ownerId", the writer, which makes the entry a write gate. A referring value may be compared once. where never finds the document: the value does, as its id, or findBy does (below), and where is checked against the document that was found. The moderation charters' joinRequest.submittedCharterId declares:
"refersTo": {
"type": "permanentDocument",
"documentType": "submittedCharter",
"where": { "$ownerId": "recipientId" }
}
reads: the proposal this join request names must be owned by its recipientId. A document found that fails an entry refuses the write, paid, with ReferencedDocumentPropertyMismatchError (40127); no document found is ReferencedEntityNotFoundError (40120). At registration both sides must exist, be single values and share one value kind, and the referenced side may not be transient (ReferencedDocumentPropertyAgreementInvalidError, 40126); a where holding "." or a function is refused by the parser, since both find the document and belong in findBy. In Rust the parser holds where as property_agreement, keyed by the referring property ({referring property: referenced property}), so the parsed model reads the opposite way round from the schema.
Read as SQL, a document reference is a query that must return a row: documentType is the FROM, findBy the part of the WHERE that finds the row through a unique index (without it, $id = value), where the rest of the WHERE, checked on the row found, minimumAgeBlocks an age condition and consume a delete of the row found.
The spellings a protocol version 14 beta used, lookup (which named the index and mapped its properties under keys), propertyAgreement (pairs keyed by the referring property) and the type listElement, were replaced by findBy, where and inList on a permanentDocument before the version shipped. Meta-schema v3 does not admit them, and the parser refuses them on every parse (refuse_replaced_reference_keywords, InvalidContractStructure, 10231) with a message naming what replaced each, so a contract written with them never loads with another meaning.
Found by a unique index (findBy)
A permanentDocument reference normally holds the referenced document's id. It may instead carry findBy: the property's value, or on the elements of a typed array each element (see References on the Elements), is then one part of a key, and the referenced document is the one the unique index of the referenced document type over exactly the properties findBy names finds for that key. The reference holds if that document exists. The moderation charter's members is the first user:
"members": {
"type": "array", "minItems": 0, "maxItems": 15, "uniqueItems": true,
"items": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"distinctFrom": "$ownerId",
"refersTo": {
"type": "permanentDocument",
"documentType": "joinRequest",
"findBy": { "submittedCharterId": "submittedCharterId", "$ownerId": "." }
}
},
"position": 2
}
reads: every member must be the owner of a joinRequest whose submittedCharterId equals this document's submittedCharterId. findBy names submittedCharterId and $ownerId, exactly the properties of joinRequest's unique bySubmittedCharter index, which finds it. Without findBy the list would have to hold the join requests' ids, which the writer would have to find first, and which say nothing about who asked to join. The same form works on a scalar identifier property, where "." is the property's own value.
A deletableDocument reference may carry findBy too, and then promises less than its id form. Once the document a key found is deleted, a new document with the same key makes the reference resolve again, to different content, where an id is produced at most once and a dead id reference stays dead. So a deletableDocument found by findBy means "a document with this key exists now", which is what a membership gate needs: the moderation charters' resignationRequest requires its writer to have an addedModerator for the charter now, and the leader takes an added member off by deleting that document. Every replace re-validates it, as it does a deletableDocument reference by id; an immutable property may not hold one, since the clearing a dead id reference allows reads an id, which a key is not; and it may be an operand of a reference expression and the target of an ownerRefersTo (see below).
findBy maps properties of the referenced document type, by their names there (system ones such as $ownerId included), in any order, to where their values come from on the referring side:
- a property path of the referring document type (
"submittedCharterId","meta.charterId"); "$ownerId", the referring document's owner, the writer;".", the value of the property that carries the reference, or the element. It appears exactly once: without it every value would resolve to the same document;- a function, at most one (see Commit and reveal).
findBy names no index. The index is the unique one of the referenced document type whose properties are exactly those findBy names (DocumentReferenceLookup::resolve_index); two unique indexes over the same properties find the same document, so the first by name is taken. A type's indexes can never be added, removed or changed by a contract update, so the same findBy always resolves to the same index.
What is checked when the contract enters the chain, on registration and on update:
findByis only allowed onpermanentDocumentanddeletableDocumentreferences, andwhereon those andmoderatedDocumentones (meta-schema v3 and the parser,apply_property_reference0), on the property or on theitemsof a typed array.- Each property an entry reads must exist on the referring type, be required (and so must every object around it), not be transient nor sit inside a transient object, and hold a single value, so
findBynever runs with a missing key part and a reader can assemble the same key from the stored document. An entry that reads"$ownerId"needs a referring type whose documents can be neither transferred nor traded: the reference is judged when the document is written, and a transfer or purchase would move the writer part of its key without a write. These are properties of the referring type alone and are checked on every parse (generation 3). - The entries must name exactly the properties of a unique index of the referenced type, and nothing else. One naming no such index is refused with the unique indexes the type has (
"joinRequest" has no unique index over exactly (submittedCharterId): findBy must name every property of one of its unique indexes and nothing else (bySubmittedCharter (submittedCharterId, $ownerId))), and one naming exactly the properties of an index that is not unique is refused too, since it could find several documents (index "byOwner" of "joinRequest" over ($ownerId) is not unique: findBy must find at most one document). The index may not bucket its first property withtimeRangeorintegerRange, and the referenced type may not beindexOnly. Each source must hold the same kind of value as the property it fills (the rule ofwhere,DocumentPropertyType::value_kind). - The key must stay with the document it found, or the reference could dangle without the document being deleted: every schema property of the index must be fixed once written (the referenced type is immutable, or the property, or the top-level object holding it, is listed under
immutableand is no optionaldeletableDocumentreference by id, which a replace may clear once its document is deleted),$ownerIdis only a key part on a type whose documents can be neither transferred nor traded, and the update and transfer times are refused where a replace, transfer or purchase moves them.$id,$creatorIdand the creation times are always fixed. { "$id": <property> }is admitted only withinList, as its only entry (see An element of a list).- A changed, added or removed
findByis an incompatible schema change on update, like the rest of arefersTo.
The checks on the referenced type run where that type is in hand. For a document type of the same contract the contract parse runs them under full validation (create_document_types_from_document_schemas 1, next to the keyRequirements.boundTo check), once every document type is parsed; a target of the wrong kind is left to registration, which refuses a deletable type for a permanentDocument found by findBy (40122), and for a deletableDocument found by findBy one that forbids deletion (40131) or whose documents only moderators remove, on the record (40144). For a type of another contract (contractId) the registration state validation runs them against that contract, where the other refersTo checks into another contract run, and refuses a declaration that cannot resolve with ReferencedDocumentLookupInvalidError (state code 40137), whose message names the properties findBy names. Index definitions cannot change on a contract update from protocol version 14 (validate_update 1 compares them by name), and neither can the flags the permanence rule reads, so the answer holds.
When the referring document is created or replaced, the document reference validation assembles the key for each value and queries the index for at most one document, billed as a document fetch of the same kind as a fetch by id (fetch_document_through_lookup). No document, or a key it cannot assemble, refuses the write, paid, with ReferencedEntityNotFoundError (40120) naming the property, or the element by its list path (members[1]); its target reads "found by" and the properties findBy names. A where beside findBy is checked against the document the index found, exactly as for an id reference. A replace re-validates a permanentDocument found by findBy when the property itself changed (for a list, the elements the stored list did not hold), and every value, every element included, when a property an entry reads changed. Nothing else can move a key part: the writer is fixed on a type allowed to read it, and the referenced side's key is fixed by the rule above, so a validated permanent reference found by findBy never dangles. A deletableDocument found by findBy is re-validated on every replace, since the document it found may be gone.
Joins cannot go through a reference found by findBy: a chained query or a composite by-id join needs the join property's values to be the outer documents' ids, so both refuse such a property, and a preallocated index cannot be bound through one. In Rust the declaration is its own variant, DocumentPropertyReferenceTarget::PermanentDocumentLookup (and DeletableDocumentLookup for the deletable form, appended after it), rather than a field of PermanentDocument: the enum is embedded in the reference errors, so an id reference keeps its encoding, and code matching PermanentDocument as "the value is a document id" cannot mistake a key part for one. The rules are on DocumentReferenceLookup, which holds findBy as keys, each referenced property mapped to its LookupKeySource. as_document_reference returns only references whose value is a document id, the accessor for joins; the validators use as_any_document_reference, whose declaration carries the DocumentReferenceLookup in its lookup field.
Reference expressions (anyOf, allOf)
A refersTo may combine targets in place of naming one. { "anyOf": [...] } holds if at least one operand holds, { "allOf": [...] } if every operand holds for the same value. An operand is a leaf, an ordinary target with its own keys, or an expression of the other combinator, so the two nest:
"memberId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": {
"anyOf": [
{
"type": "permanentDocument", "documentType": "addedModerator",
"findBy": { "submittedCharterId": "submittedCharterId", "moderatorId": "." }
},
{
"allOf": [
{ "type": "identity" },
{
"type": "permanentDocument", "documentType": "joinRequest",
"findBy": { "submittedCharterId": "submittedCharterId", "$ownerId": "." }
}
]
}
]
},
"position": 2
}
reads: the member was added to the charter, or it is an identity that asked to join it. The same form sits on the items of a typed array, where each element meets the expression on its own.
What is checked when the contract enters the chain:
- On every parse (meta-schema v3 and the parser,
apply_property_reference0): a combinator is the declaration's one key (afindBy,whereorinListbelongs to a leaf, inside it), a list names at least two operands (a single one is declared on its own), and ananyOfdirectly inside ananyOf(or anallOfinside anallOf) is refused, since it says what one flat list says. - Every leaf is an
identity, apermanentDocument(by id, byfindByor withinList) or adeletableDocumentfound byfindBy. The first two are existence checks against entities that are never deleted (inListreads a list that never changes on a document that is never deleted), so an expression of only them holds for good once it holds, as a single one of them does, and a replace re-validates it only when its value or a property one of its leaves binds changed. AdeletableDocumentleaf found byfindBymay find nothing later, so an expression holding one is re-validated on every replace, and an immutable property may not hold it. A leaf whose key afindByfunction computes is the exception: it is judged on the create alone and holds on a replace without a read, so an immutable property may hold it, and on a document type whose documents can be replaced it may not be an operand of ananyOf, which would then hold on every replace whichever operand held on the create. The other types do not compose with other operands and are refused, as is the key id form (identityProperty):deletableDocumentby id is re-validated on every replace and may be cleared once its document is deleted (the immutable-property exception), which assumes the property refers to that one target;identityPublicKeypairs the value with a key id property no other operand reads; acontracttarget's requirements are gates judged against the block time and the writer rather than an existence check, and a contract or token id is never also an identity or document id. Admitting one later takes a newapply_property_referencegeneration. - Under full validation (registration): at most
SystemLimits::max_reference_operandsoperands in one list and at mostmax_reference_expression_depthcombinators on any path from the declaration to a leaf (4 and 4 at protocol version 14; the example above is 2 deep), no two alike operands in one list (a leaf naming the declaring contract explicitly is the same as one omitting it), and every leaf counted againstmax_references_per_document: ananyOfof two on a typed array ofmaxItems15 counts 30, since each leaf may be read for each element. - Every leaf is checked exactly as the same target declared alone: the referenced document type, its permanence, the
wheresides and thefindByor list rules, at the same places (the contract parse for a type of the same contract, the registration state validation for another contract's). Every leaf must pass, since each has to be a declaration that could hold. An error names the failing leaf by where it sits,refersTo anyOf[1].allOf[1] findBy: ...from the parse andresignation.memberId.anyOf[1].allOf[1]from registration. - A changed expression (an operand added, removed, changed or moved,
anyOfswapped forallOf, a single target turned into an expression or back) is an incompatible schema change on update, like the rest of arefersTo. InsiderefersTo,anyOfandallOfare the declaration's data; the schema compatibility rules never read them as JSON Schema keywords.
When the referring document is created or replaced, the document reference validation evaluates each value (each element) operand by operand in declared order, a nested expression the same way. An anyOf stops at the first operand that holds; when none does, the write is refused, paid, with the last operand's result. An allOf stops at the first operand that fails and refuses the write with its result. A refusal is therefore always the error a leaf declared alone would give (for the example, ReferencedEntityNotFoundError (40120) for a findBy that found nothing, naming the property or the element), and the author's order decides which one a writer sees: put the most general operand of an anyOf last, and the cheapest or most telling one of an allOf first. There is no error of its own for "no operand held": each leaf's failure already has a precise error, and a combined one would have to nest one per leaf or lose their reasons. Every read is billed as it is made, so a value the second operand of an anyOf holds for pays for the first operand's query too, while an allOf whose first operand fails reads nothing more. A where is checked only against its own leaf's document: a value whose first leaf fails its where is still accepted through a second leaf without one.
Joins and preallocated indexes need one target: a chained query or a composite by-id join refuses an expression join property, and a preallocated index is never bound through one. In Rust the combinators are DocumentPropertyReferenceTarget::AnyOf(ReferenceOperands) and AllOf(ReferenceOperands), appended to the enum so every single target keeps its encoding. An expression is no document reference as a whole (as_any_document_reference is None); code that checks every declaration walks DocumentPropertyReferenceTarget::leaves (or leaves_with_paths), the leaves of an expression or the declaration itself. Since the enum is embedded in consensus errors, which clients decode from bytes a node sends, decoding refuses a nesting deeper than MAX_REFERENCE_EXPRESSION_DECODE_DEPTH (16, above every protocol version's registration limit, which a test holds it to), so no bytes can drive the decoder into unbounded recursion. A reference error never carries a combinator: a refusal is a leaf's error.
An element of a list (inList)
A permanentDocument reference with inList says the value must be one of the identifiers a list of another document holds, the document findBy names by its $id:
"resignation": {
"type": "object",
"properties": {
"electedCharterId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "permanentDocument", "documentType": "electedCharter" },
"position": 0
},
"memberId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": {
"type": "permanentDocument",
"documentType": "electedCharter",
"findBy": { "$id": "electedCharterId" },
"inList": "members"
},
"position": 1
}
},
"required": ["electedCharterId", "memberId"],
"additionalProperties": false
}
reads: memberId must be one of the members of the electedCharter document whose $id this document's electedCharterId holds. The moderation charters, whose elected charter holds its members, are the first users. The declaration sits on an identifier property, on the items of a typed array of identifiers, where every element must be listed (see References on the Elements), as a leaf of a reference expression (see Reference expressions), or on the writer or the creator (see On the writer or the creator), where the value is that identity: "ownerRefersTo": { "anyOf": [<inList>, <addedModerator findBy>] } is the charters' rule that the writer is a seated or an added moderator.
A reference with inList is a permanentDocument reference in every respect but one. It takes the contractId, documentType and where of any permanentDocument reference, with the same checks (the referenced type must forbid deletion, every where entry must exist and share one value kind). What differs is what the value is: not the referenced document's id but an element of its list. The document is the one findBy: { "$id": <property> } names, read from an identifier property of the referring type (a schema property, never $ownerId: no document has the writer's id), and inList names the typed array of identifiers on the referenced type. { "$id": ... } is the only findBy entry naming $id, and only with inList; where may not compare $id again. What is checked when the contract enters the chain:
inListis only allowed on apermanentDocumentreference, and needsfindByto be exactly{ "$id": <property> }(meta-schema v3 and the parser).- The
$identry reads a stored identifier property of the referring type (it and every object around it not transient), so a reader can tell from the stored document which list the value was checked against. It may be optional, and it needs norefersToof its own. Checked by the parse under full validation (generation 3,validate_list_element_sources), a leaf of an expression as it would be alone. - The list's document type forbids deletion, and
inListis a stored typed array of identifiers on it that never changes once a document is written: the type is immutable (documentsMutable: false), or the list's top-level property is listed underimmutable, the rule afindBykey's properties are judged by. AnimmutableAllowSettingentry can only be set on a document that has no value for it, against which no value was ever accepted, so it does not weaken the rule. For a type of the same contract the contract parse checks this (create_document_types_from_document_schemas1); for a type of another contract, registration checks it against that contract in state and refuses a list that does not qualify withReferencedDocumentListInvalidError(state code 40138). A missing document type is refused as for any document reference (40121), and a deletable one by the parse for the same contract (with the list reason) or by registration for another (40122). - Each value counts against
SystemLimits::max_references_per_document, one for a property,maxItemsfor a typed array, like every other reference. - A changed, added or removed
inListis an incompatible schema change on update.
When the referring document is created or replaced, the document reference validation fetches the document whose id the $id entry's property holds, by id, checks the where entries against it exactly as for a permanentDocument, and requires the value to be in its list. Every by-id document fetch of one write is shared: a charter that electedCharterId's own reference already fetched, or that the elements of one typed array all name, is fetched and billed once, and the list is collected once into a set, so each value is a set lookup. A value the list does not hold, a document the id names that does not exist, or a value set while the $id property is not, refuses the write, paid, with ReferencedEntityNotFoundError (40120) naming the property, or the element by its list path (witnesses[1]); its target reads "list element (<inList> of the <documentType> document <property> names)". A failing where entry is ReferencedDocumentPropertyMismatchError (40127), as always. A replace checks a list reference again when its value changed or when a property findBy or where reads changed (the $id property among them, since it may name another charter), and then every value, every element included; only a list that changed on its own leaves out the elements the stored list already held. Nothing else can make a validated value unlisted: the list's document can never be deleted and its list never changes.
For example:
electedCharter 7kX...: members [Alice, Bob]
resignation { electedCharterId: 7kX..., memberId: Alice } -> accepted
resignation { electedCharterId: 7kX..., memberId: Carol } -> refused, 40120:
referenced list element (members of the electedCharter document electedCharterId names)
<Carol> not found for path memberId
In Rust the declaration is the appended variant DocumentPropertyReferenceTarget::ListElement(ListElementReference), so every earlier variant keeps its encoding in the reference errors; the parser holds the $id entry among the where comparisons of its property_agreement, as {property: "$id"}, and the rules are on ListElementReference (document_id_property, referring_side_error, referenced_side_error, listed_values). as_any_document_reference carries it with in_list set, so the registration validator checks its contract, type and where through the same code as the other document references, while as_document_reference leaves it out, as it does a reference found by findBy: joins and preallocated indexes never go through it.
On the writer or the creator (ownerRefersTo, creatorRefersTo)
A property's reference constrains a value the writer chose. Some rules constrain the writer instead: in the moderation charters, a resignationRequest may only come from a moderator of the team it resigns from. A document type states that with the doctype-level ownerRefersTo keyword, one refersTo declaration whose value is the document's $ownerId, the writer, rather than a property's value:
"resignationRequest": {
"type": "object",
"ownerRefersTo": {
"type": "deletableDocument",
"documentType": "addedModerator",
"findBy": { "electedCharterId": "electedCharterId", "memberId": "." }
},
"properties": { "electedCharterId": { "...": "..." } }
}
reads: the writer must be the memberId of an addedModerator for this document's electedCharterId, one that exists when the request is written (the leader takes an added member off by deleting it).
- The declaration is the one an identifier property carries, read by the same code (
apply_property_reference0), but only these targets can hold a writer:identity, apermanentDocumentor adeletableDocumentfound byfindBy, and apermanentDocumentwithinList(the writer an element of the list, see An element of a list), alone or as the leaves of a reference expression (above). The rest are refused, as a leaf of an expression too.contract,tokenand a document by id would need the writer's identity id to be a contract, token or document id, which it never is, so a document type declaring one could never be written;identityPublicKeypairs the value with a key id the writer does not carry. Meta-schema v3 reuses the property declaration by$refand admits only those forms; the parser (generation 3,parse_owner_reference, which reads the stored schema once the core parse has run the meta-schema) refuses the others on the stored path too. The parsed declaration isDocumentTypeV2::owner_reference, read throughDocumentTypeV2Getters::owner_reference, and the property types are unchanged. - Only a document type whose documents can be neither transferred nor traded may declare it, checked on every parse. A transfer or a purchase is not a write, so it would hand the document to an owner the declaration never checked; with neither possible, the owner of every document is the writer that was checked.
- In
findBy,"."is the writer, and a"$ownerId"source is the writer as well. Every referring-side rule of a property'sfindByapplies unchanged (its"$ownerId"rule holds by the point above), and so does every referenced-side rule, for a type of the same contract at contract level and for one of another contract at registration. - A
whereworks as on a property reference; its value may be"$ownerId", which is then the same writer as the reference's value. - It counts one against
SystemLimits::max_references_per_document. - When a document is created, the document reference validation checks the writer against the target exactly as a property's value is checked, before the properties' references. A replace re-validates it under the rules of its target, as a property's: when a property its
findByorwherereads changed, and on every replace for awhereentry valued"$ownerId", a writer gate. Nothing else can change the outcome: the writer is the owner, the target can never be deleted and its key is fixed. A failure is the error the target reports for a property (ReferencedEntityNotFoundError, 40120, whenfindByfinds no document;ReferencedDocumentPropertyMismatchError, 40127, for awhereentry; and the rest), with$ownerIdas its path. Anidentitytarget reads nothing: the transition has already proved that the writer exists. At registration the contract reference validation checks the declaration as a property's, naming it<documentType>.$ownerId. - Adding, removing or changing it is an incompatible schema change on update (
validate_schema_compatibility1 freezes the keyword as the shared rule set freezesrefersTo). - Every validator, the reference bound and the client bindings enumerate a type's references through
DocumentTypeRef::reference_declarations, which yields the owner or creator reference first, asReferenceHolder::OwnerorReferenceHolder::Creator, then each property's, so none can skip it.
A document type whose documents can be transferred or traded declares creatorRefersTo instead: the same declaration, whose value is the document's $creatorId, its creator, which a transfer or a purchase never changes. A marketplace item that only an elected moderator may mint, and anyone may then own, reads:
"moderatorBadge": {
"type": "object",
"transferable": 1,
"creatorRefersTo": {
"type": "permanentDocument",
"documentType": "electedCharter",
"findBy": { "$id": "electedCharterId" },
"inList": "members"
},
"properties": { "electedCharterId": { "...": "..." } }
}
- It takes the same targets but a
deletableDocument(the creator never changes, and a document a transfer handed on could not be replaced once the onefindByfound is deleted), except one found by afindByfunction, which is judged on the create alone (see Commit and reveal), with"."the creator infindBy, and is refused whereownerRefersTois admitted: only a document type that records creator ids may declare it, a transferable or tradeable type of a format-1 contract (should_use_creator_id), checked on every parse. A type therefore declares at most one of the two. A"$ownerId"source in itsfindByis refused, as in a property'sfindByon such a type, since the owner moves. - When a document is created its creator is the writer; on a replace, the value is the stored creator, whoever writes, and the replace rules are those of its target, as for the owner reference. A transfer or a purchase needs no check. A failure is the target's error at the path
$creatorId, and registration names the declaration<documentType>.$creatorId. Anidentitytarget reads nothing: the creator existed when it wrote the document, and an identity is never removed. It counts one againstmax_references_per_document, and a change to it is an incompatible schema change on update.
Commit and reveal (a findBy function)
One findBy entry may hold a function instead of a source: "<referenced property>": { "function": "sys.hash.sha256d", "params": [...] } says the referenced document's property holds a hash of values the document being created reveals. The platform fills that property of the key with the hash to find the referenced document, so the function computes a key part. The document it finds is a commitment made earlier, so the reference is a commit and reveal: the document may be created only while a commitment to values it carries exists. The function is a system function, as in generatedFrom, from the sys.hash namespace; sys.hash.sha256d, the SHA-256 of the SHA-256, is the one there is. DPNS name registration is a commit and reveal of this kind, done today by its domain create trigger rather than by a declaration: a preorder holds saltedDomainHash under a unique index, and the trigger hashes what the domain reveals and looks for it. The same rule for a name under a parent, written as a declaration on the salt the domain reveals, reads:
"preorder": {
"type": "object",
"documentsMutable": false,
"canBeDeleted": true,
"properties": {
"saltedDomainHash": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 0
}
},
"indices": [
{ "name": "saltedHash", "properties": [{ "saltedDomainHash": "asc" }], "unique": true }
],
"required": ["$createdAtBlockHeight", "saltedDomainHash"],
"additionalProperties": false
},
"domain": {
"type": "object",
"properties": {
"preorderSalt": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 4,
"refersTo": {
"type": "deletableDocument",
"documentType": "preorder",
"findBy": {
"saltedDomainHash": {
"function": "sys.hash.sha256d",
"params": ["preorderSalt", "normalizedLabel", { "const": "." }, "parentDomainName"]
}
},
"where": { "$ownerId": "$ownerId" },
"minimumAgeBlocks": 1,
"consume": true
}
}
},
"transient": ["preorderSalt"]
}
reads: the salt refers to a preorder, found by its saltedDomainHash (the whole of its unique saltedHash index), that the writer owns (the where entry), created in an earlier block, whose saltedDomainHash is the sha256d of the salt, the normalized label, a dot and the parent; creating the domain deletes that preorder. The hash is byte for byte the one create_domain_data_trigger_v1 computes for a name under a parent. On an index of several properties, findBy maps the others beside the function entry as usual.
A string or byte array property may carry a refersTo only this way: its value is not an id, so its declaration must be a permanentDocument or deletableDocument reference found by a findBy function, and the property keeps its type (the meta-schema admits a refersTo with findBy on a string or byte array). The value carrying the reference must fill the key exactly once: as a "." entry or as a param of the function. A param names a single property carrying the reference by its path, as the salt above; "." in the params stands only for a value without a path of its own: each element of a typed array, the writer of an ownerRefersTo, the creator of a creatorRefersTo. The writer and the creator may also be left out beside a function, since the declaration applies to every create.
The preimage is the bytes of the params joined in order with nothing between them:
- a property path: that property of the document being created, a string's UTF-8, a byte array's bytes, an identifier's 32 bytes;
{ "const": text }: fixed UTF-8 text, 1 to 64 bytes;".": the value carrying the reference, where it has no path, read the same way.
1 to 16 params. System values such as "$ownerId" are not params. Plain concatenation is ambiguous when two variable-length params meet ("ab" + "c" and "a" + "bc" are the same bytes), so every variable-length param (a string, or a byte array whose size is not fixed) with another variable-length param anywhere after it must be followed directly by a one-byte const, its separator, and a create whose value for that param holds the separator byte is refused. Each preimage then splits into its params one way only. Integers and other kinds are not params. A path that passes through a map holding a key more than once is refused, as for generatedFrom.
The document such a key finds is judged once, when the document is created:
- A param may be transient (the DPNS salt is), since it is read from the create transition, and optional, since a create missing one is refused. Every stored value
findByreads, params and other entries alike, must be fixed once written (the type is immutable or lists the property underimmutable), and so must a stored property carrying the reference, set when the document is created (not listed underimmutableAllowSetting). Such adeletableDocumentfound byfindBymay sit on animmutableproperty, which one re-validated on every replace may not. A replace leaves the reference alone: nothing it reads can have changed, and the commitment may be gone. The declaration'swhereentries are judged with it, on the create alone, so each referring property they name must be fixed once written the same way, or transient. On a document type whose documents can be replaced, such a reference may not be an operand of ananyOf: judged on the create alone, it would hold on every replace, whichever operand held on the create. creatorRefersTotakes adeletableDocumenttarget only through a function.findByholds at most one function. The property it fills must be a byte array of exactly the hash's size, 32 bytes forsys.hash.sha256d. BesidefindBy, on therefersTo, the reference may then declare what it asks of the documentfindByfinds (both are refused without a function infindBy):minimumAgeBlocks: the found document's$createdAtBlockHeightmust be at least that many blocks below the height of the create.1means an earlier block, so a commitment and its reveal cannot share a block. The referenced type must list$createdAtBlockHeightinrequired; a document recording no creation height never meets it.consume: true: the create deletes the found document in the same state transition, its storage refunded to its owner as a delete by that owner would be. Only on adeletableDocumentreference whosewhereholds the entry"$ownerId": "$ownerId", into a document type of the declaring contract whose owner may delete its documents (canBeDeleted: true), or whose documents only a consume deletes (canBeDeleted: "onlyWhenConsumed",DocumentTypeV2Getters::documents_deleted_only_when_consumed). No delete transition is charged for the consumed document, so its type may declare no delete token cost and no delete action fee, and the key signing the create deletes it, so its type may not require a stricter signature security level than the declaring type.
- Whose commitment it is, is the
whereentry"$ownerId": "$ownerId"beside the function: the writer must own the document found. Without it anyone who learns a preimage may reveal it.
What is refused, and where:
- The shape, the params and the rules above: the contract parse on registration and update (meta-schema v3
findByFunction,apply_property_reference0,DocumentReferenceLookup::referring_side_errorandreferenced_side_error), and for afindByinto another contract the registration state validation (ReferencedDocumentLookupInvalidError, 40137), which also refusesconsumethere. A function inwhereis refused (it finds the document, so it belongs infindBy), as are a second function infindByandminimumAgeBlocksorconsumewithout one. Any change to a function, itsminimumAgeBlocksor itsconsumeis an incompatible schema change on update, like the rest of arefersTo. - A create missing a param, or whose variable-length value holds its separator: document create structure validation 1, before any read (
DocumentReferencePreimageInvalidError, 10423). A function in a leaf of a reference expression is left to the write-time check, where a key it cannot assemble finds no document, so that operand fails and the others still decide. - No document for the key:
ReferencedEntityNotFoundError(40120), whose entity id for a string or byte array carrier is the hash. The hash is computed once and billed as the double SHA-256 it is (ValidationOperation::DoubleSha256, by the blocks it hashes: the padded preimage and the second pass), the index query as the document fetch it is. Another identity's commitment where the owner entry is declared:ReferencedDocumentPropertyMismatchError(40127). A commitment younger thanminimumAgeBlocks:ReferencedDocumentRequirementNotMetError(40142). All paid, from document create state validation 2. - A create signed by a contract-bound key whose bounds leave out a document type the created type may consume: batch advanced structure validation 1, as a direct delete of that type would be (
ContractBoundedKeyOutOfBoundsError, 20014, paid). - A batch in which a create consumes a document that another create of the batch consumes too, or that another transition of the batch deletes, replaces, transfers, reprices or buys, refuses that create with 40120 (
ConsumedDocumentsin the batch state validation). Today a batch holds one transition, so this only guards the day that cap is raised.
In Rust the parser holds the function entry in the DocumentReferenceLookup as the source of the property it names: LookupKeySource::Hash(LookupHashKey), appended to the key sources, with its function a HashFunction (the SystemFunction::Hash namespace, beside the string transformations generatedFrom takes) and its params LookupKeyParams. The parsed property_agreement holds the where entries alone. A string or byte array property's declaration is DocumentProperty::revealed_reference, listed by reference_declarations as PropertyReference::Revealed. DocumentReferenceLookup::is_checked_on_create_only tells a lookup holding a computed key, and the parser holds the reference's minimumAgeBlocks and consume on the lookup as minimum_age_blocks and consume. first_unrevealable_lookup_key is the structure check. A create that consumes carries its commitments in DocumentCreateTransitionAction::consumed_documents, turned into delete operations with the create's own.
Immutable Properties on Mutable Document Types
A document type either allows replaces (documentsMutable: true, the default) or freezes its documents entirely. Protocol version 14 adds a middle ground: the doctype-level immutable keyword lists top-level properties a replace may not change while the rest of the document stays replaceable. An entry naming a property freezes it at creation; an entry { "property", "when" } freezes it for any replace its condition holds for.
"post": {
"type": "object",
"documentsMutable": true,
"properties": {
"author": { "type": "string", "maxLength": 63, "position": 0 },
"body": { "type": "string", "maxLength": 500, "position": 1 },
"mood": { "type": "string", "maxLength": 30, "position": 2 }
},
"required": ["author", "body", "$createdAt", "$updatedAt"],
"immutable": [
"author",
{ "property": "body", "when": { "greaterThan": [{ "subtract": ["$updatedAt", "$createdAt"] }, 300000] } },
{ "property": "mood", "when": { "present": "$old.mood" } }
],
"additionalProperties": false
}
author is frozen at creation, body five minutes after it, and mood once the stored post holds one, so an absent mood may be set by one replace.
The parser (generation 3, meta-schema v3) reads the list on every parse (parse_immutable_keyword, apply_immutable_fields):
- The shape is checked on both paths: every entry is a string or an object with exactly
propertyandwhen, and no property is listed twice.immutableAllowSetting, the keyword thepresent: "$old.<p>"condition replaces, is refused, naming its replacement, so that no contract written with it loads with another meaning. - A condition is parsed with the
propertyConstraintsgrammar (parse_property_constraint_condition) and what it reads is checked as a rule's reads are (validate_property_constraint_reads), on both paths: a stored condition reading an undeclared or transient property could only read it as absent. A path through$old.(STORED_DOCUMENT_PREFIX) is judged as the path after it, and only a condition ofimmutablemay use one. A condition may not read acountOforsumOf, so a replace judges it without reading state. - Under full validation, when a contract enters the chain: the type is mutable; every listed property is a declared top-level property, neither a system property nor transient (list the containing object to freeze a nested value); a condition stays within a rule's node limit and lists no condition twice. Stored contracts skip these lints, so a later tightening can never make a committed contract unreadable.
- A
deletableDocumentreference by id may be listed only without a condition. A replace may clear such a reference once its target is deleted (see References on the Elements), and a replace the condition then left free could set it to another document: the frozen reference would be repointed. The clear stays, because every replace re-validates the reference and without the clear the document could never be replaced again, so the condition is refused at registration instead, and on contract update, which parses the whole new contract the same way. Every otherdeletableDocumentform is refused on any immutable property. - A property listed with a condition is not fixed once written (
schema_property_is_fixed_once_written): afindBykey, aninListlist and the values read beside afindByfunction may not rely on it. - On contract update the properties listed without a condition may only grow, and a property listed with a condition keeps it as it is or loses it to be listed without one (
validate_immutable_fields_update,DocumentTypeUpdateErrorotherwise). Whether one condition holds wherever another does cannot be told in general, so a changed condition is refused. The schema compatibility differ strips the key, likeindicesandrequired, sovalidate_updatev1 is the single judge.
Enforcement lives in the replace action's state validation (generation 1). The action already records which top-level properties differ from the stored document in changed_data_fields (the same set that scopes refersTo re-validation). A changed property in the type's immutable_fields() fails the replace with DocumentImmutablePropertyChangedError (state code 40128) unless it is a deletableDocument reference by id that the replace removed after its target was deleted. A changed property in immutable_field_conditions() fails it with the same error when its condition holds, or faults, judged by PropertyConstraint::holds on the written properties with the stored ones under $old and the replace's system values (the stored creation and transfer, this block as the update). The stored properties are rebuilt from the written ones and stored_changed_values, the stored value of every changed property; a property the stored document did not hold is left out. Conditions are evaluated only for the properties the replace changes. The replace compares stored and supplied values by underlying data, recursing into objects regardless of member order and into arrays position by position, with integer widths ignored, since storage reorders object members by schema position and narrows integers. "Differ" covers a changed value, a property the stored document lacked, and a property the replace dropped. Transfers, price updates and purchases carry no property data and are unaffected.
In Rust the properties frozen at creation are DocumentTypeV2Getters::immutable_fields() and those frozen under a condition immutable_field_conditions(), property to PropertyConstraint. Earlier document type generations return empty ones.
Transient Properties
The doctype-level transient keyword lists top-level properties whose values are validated on the transition but never stored. DPNS uses it for the domain's preorderSalt: the write proves the salted preorder, and the salt is then dropped.
"transient": ["preorderSalt"]
A create drops the listed values before its document is built. Before protocol version 14 a replace stored whatever it carried, so a replaced document kept values its create had dropped; from protocol version 14 a replace drops them the same way (document_from_replace_transition_action 1). Values are dropped by top-level name, so a leaf of a transient object goes with the object.
Because no stored document carries a transient value, a rule that reads a stored value refuses a transient one, judged by the property's path and every object around it (is_transient). From protocol version 14, at registration:
- Every
transiententry names a top-level property. A nested path, a system property or an undeclared name would mark a property transient in the parsed type while its value was still stored. - No index reads a transient property. Every document would sit in the index's null branch, so a query by the value would find nothing and a unique index would enforce nothing.
- A
refersTofindByreads no transient property to assemble its key (afindByfunction's params excepted, which are read from the create), and its index keys documents by none. Awherenames none on its referenced side. The referring side may be transient: it is judged on the transition, a write gate like the writer's$ownerId. - A key reference does not store its key id with a transient identity, whichever side declares it (
identityPropertyon the key id,keyIdPropertyon the identity): the key id alone names no key. encryptedFornames no transient recipient or key id.
The list cannot change on contract update: it decides which values stored documents carry and how every property is encoded (a transient property takes a presence byte even when required). From protocol version 14 any change to the names it lists is an incompatible schema change (the list is compared as a set, so reordering or repeating a name is no change); before it, the schema compatibility check failed on the keyword as unsupported, an internal error.
Typed Arrays
Up to protocol version 13 a type: "array" property had to be a byte array (byteArray: true). Protocol version 14 adds typed arrays: a list whose items schema says what every element is.
"reasons": {
"type": "array",
"minItems": 0,
"maxItems": 64,
"uniqueItems": true,
"items": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier"
},
"position": 2
}
- An element is a scalar: an integer, a number, a string (with
minLength/maxLength), a boolean, a byte array (byteArray: true, whoseminItems/maxItemscount bytes) or an identifier. Objects and arrays of arrays are refused. An identifier element may carryrefersTo(see References on the Elements). An element may be limited to allowed values withenum;constis refused on elements, since a one-valueenumdoes the same and a contract update can still widen it. The parser reads an element'senum,minimumandmaximumonto the typed array (ArrayItemConstraints), refusing on both parse paths anenumwith no member or a member of another type, anenumon a byte array or identifier element, and aminimumabove themaximum, so random document generation stays inside them; the JSON schema validator enforces them on every document. - On the array itself
minItemsandmaxItemscount elements, not bytes.maxItemsis required,minItemsmay not exceed it andcontentMediaTypebelongs on the items; these hold on every parse. Contract registration also capsmaxItemsatSystemLimits::max_typed_array_items(1024), so a typed array's worst-case size, which fee estimation charges by, stays small.uniqueItems: truerefuses a document that repeats an element. - A byte array keeps its form and takes no
items. On a plain byte arrayuniqueItemskeeps its old meaning, no repeated byte, but an identifier (a byte array with the identifiercontentMediaType) refuses it: an identifier is one value, and "no repeated byte" would refuse most of them. - The document is validated against the JSON schema as always, so a list that is too long, too short, repeats an element under
uniqueItemsor holds a wrong-typed element fails with the usualJsonSchemaError.
The array is stored inline in the document, like any other property: a varint element count followed by the elements, each encoded exactly as a required property of the element's type (see Document Serialization). The reasons list above is therefore one count byte and 32 raw bytes per identifier, and an integer element bounded 0..100 takes one byte. Since the stored bytes depend on the element's type, a contract update may not change how an element encodes: raising an integer element's maximum (or adding an enum value) past its width, or unpinning a fixed-size byte array element, is refused with DocumentTypeUpdateError. A longer maxLength, a larger maxItems or a raised maximum that keeps the width are accepted. Identifier and byte array elements are conversion paths (reasons[], find_identifier_and_binary_paths 1), so a document built from JSON or a value map converts every element, as it converts a scalar identifier. ExtendedDocument::set_untrusted converts every member of a list set at such a path. From protocol version 14 a property name and a document type name are word characters only (^[a-zA-Z0-9_]{1,64}$): every earlier generation also admitted -, which the path syntax (a.b, list[]) was never written for, and a census of every contract on mainnet and testnet found none using it. Nothing is written per element, so a typed array cannot be an index property (InvalidIndexPropertyTypeError), an indexOnly terminal or entry payload property, or one side of a where entry.
In Rust a typed array parses to DocumentPropertyType::TypedArray(TypedArrayProperty), whose item_type is the DocumentPropertyType the items schema parses to as a property schema (try_from_value_map with the contract's parsing options). The parse is the versioned parse_typed_array (None before protocol version 14, where an array that is not a byte array is refused as it always was). The older DocumentPropertyType::Array variant, whose elements are an ArrayItemType in their own length-prefixed encoding, is never produced by the parser.
References on the Elements
An identifier element may carry a refersTo declaration, which every element of the list then declares. The moderation charters' reasons refers to reason documents this way:
"items": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "permanentDocument", "documentType": "reason" }
}
- The declaration takes every target and key a single identifier property's
refersTotakes (identity,contractwithcontractRequirements,token, andpermanentDocumentanddeletableDocumentwithcontractId,documentType,findByandwhere, andinListon apermanentDocument), with the same checks at contract registration, exceptidentityPublicKeyin either form: itskeyIdPropertynames one sibling key id, and theidentityPropertyform sits on the key id itself, neither of which can pair with many elements, so an element may not declare it. The declaration belongs on theitems; on the array itself it is refused. - When a document is created or replaced each element is checked as a single reference is, in list order: the target must exist, a referenced contract must meet the
contractRequirements, a referenced document's type must be deletable or not as declared, and eachwhereentry must hold. The referring value of an entry is still a property of the referring document or its$ownerId, the same for every element; the referenced side is a property of that element's referenced document. The first element that fails refuses the write with the error a single reference gives (ReferencedEntityNotFoundError40120,ReferencedDocumentPropertyMismatchError40127,ReferencedContractRequirementNotMetError40135 and so on), whosepathnames the element by its list path:reasons[2]for the third. An empty or absent list checks nothing. Registration errors name the declarationsubmittedCharter.reasons[]. - A replace follows the rules of a single reference, element by element. A changed list re-validates the elements the stored list did not hold (the replace action carries the stored value of each changed property,
stored_changed_values); the ones it held are unchanged references and are left alone, as an unchanged single reference is. Every element is re-validated when a property awhereorfindByreads changed, and on every replace when awhereentry is valued"$ownerId", the elements aredeletableDocumentreferences, or they arecontractreferences with anownerrequirement on a document type whose documents can be transferred or traded. An element repeating an earlier one of the same list is not fetched again. - Every read is billed as a single reference's is. A foreign contract holding the referenced document type is fetched once per list, not once per element. Contract registration caps the references one document of a type can carry at
SystemLimits::max_references_per_document(256), counting one per property withrefersTo(an identifier, or a key id carrying a key reference), one for the type'sownerRefersToandmaxItemsper typed array of referencing elements:maxItemsalone would let a type declare many lists of up to 1024 references each, and each one is a read when a document is written. - An
immutableproperty may not hold adeletableDocumentreference a replace could not clear: a typed array of them, at the top level or inside an immutable object, or a single one inside an immutable object. Every replace re-validates them, so once a target is deleted the property would have to change, which an immutable property cannot. A singledeletableDocumentreference that is itself the immutable top-level property has a way out, a replace may remove it once its target is gone, and that exception reads the one identifier the removed top-level property held. That property may not also be listed underimmutableAllowSetting: once it is cleared, the allowance would let the next replace set it to another document. - A changed element
refersTois an incompatible schema change on contract update, as a changedrefersToon a scalar is.
In Rust the element parses to DocumentPropertyType::IdentifierWithReference(target) inside item_type, through the same versioned apply_property_reference a scalar identifier goes through. DocumentPropertyType::reference() reports a property's declaration as PropertyReference::Value(target) for a scalar and PropertyReference::Elements(target) for a typed array; the registration check (validate_data_contract_references) and the write-time check (validate_document_references) both enumerate references through it.
Distinct Identifier Properties
Protocol version 14 adds the property-level distinctFrom keyword, a pure structure rule on identifier properties: the property's value must differ from the value of a named property of the same document, or from the document's $ownerId. It sits next to the reference keywords (refersTo and its where, which binds a property to another document's values) but reads nothing beyond the transition being written.
"delegateId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"distinctFrom": "$ownerId",
"position": 0
},
"backupId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"distinctFrom": "delegateId",
"position": 1
}
The value is "$ownerId" or the dotted path of another property of the document type ("meta.reviewerId" for a nested one). The parser (generation 3, meta-schema v3) checks the declaration when a contract enters the chain, on registration and on update:
- The keyword is only allowed on identifier properties, enforced by the same dependent schema shape that restricts
refersTo. - A named property must exist on the document type, must itself be an identifier (the only kind the value can be compared with), and must not be the declaring property.
$ownerIdneeds no check; no other system property is accepted. - On contract update a changed, added or removed
distinctFromis an incompatible schema change, like a changedrefersTo. - A typed array of identifiers declares it on its
items, and every element must then differ from the named value; the declaration is refused on the array itself and on elements of any other type.
Enforcement lives in the structure validation of the document create and replace actions (create structure generation 1, introduced at protocol version 14, and replace structure generation 0, extended in place: the call is inert before 14, where no property can carry the keyword), after the schema validation of the document's properties, so every value compared is already a 32-byte identifier. The check reads the transition's data and the owner id it carries and never touches Drive; the declaring properties come from a list the parser built (distinct_from_fields), so a type without declarations costs nothing. An equal pair fails the write with DocumentPropertyNotDistinctError (basic code 10419), which names the document type, the property and what it collided with. When the named property is absent from the document there is nothing to differ from, so the rule passes.
A transfer or purchase changes $ownerId without touching the data, so the transfer and purchase structure validations (generation 0, extended in place: the call is inert before protocol version 14, where no property can carry the keyword) judge the stored document against its new owner: a transfer to, or a purchase by, the identity a $ownerId-distinct property names is refused with the same error. Price updates change neither owner nor data and are not judged.
In Rust the declaration is DocumentProperty::distinct_from (Option<DistinctFrom>, absent on every property parsed before protocol version 14), the document check is DocumentTypeV0Methods::validate_distinct_from_properties, and DistinctFrom::violation judges one value on its own, which is how the elements of a typed array are judged one by one.
Encrypted Properties (encryptedFor)
A byte array property may hold ciphertext that only one identity can read. Before protocol version 14 the contract said nothing about it, so every wallet had to learn the recipe (whose keys, which scheme, where the IV sits) from documentation or a side channel. From protocol version 14 the property declares it with the encryptedFor keyword, and wallets and SDKs read the recipe from the contract.
"encryptedMessage": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 1040,
"encryptedFor": {
"recipient": "recipientId",
"recipientKey": "recipientKeyId",
"senderKey": "senderKeyId",
"scheme": "ecdh-secp256k1-aes256-cbc"
},
"position": 4
}
All four keys are required. recipient is the dotted path of an identifier property of the same document type whose value is the recipient identity's id, or $ownerId for a message the writer encrypts to themself. recipientKey and senderKey are dotted paths of integer properties of the same document type carrying the recipient's and the sender's identity key ids; each must declare minimum at least 0 and maximum at most 4294967295, read from the schema itself, so the rule holds whatever sizedIntegerTypes the contract sets. scheme is a closed set with one member today.
The parser (generation 3, meta-schema v3) admits the keyword on byte array properties only, never on an identifier (contentMediaType set) or any other type, and checks at contract registration that the three named properties exist with those types, that none of them is transient or sits inside a transient object (a transient value is stripped before storage, which would leave the stored ciphertext without its recipe), and that the byte array's own maxItems can hold the scheme's shortest ciphertext. A contract update that adds, removes or changes an encryptedFor declaration is an incompatible schema change (IncompatibleDocumentTypeSchemaError, 10246): documents already written under the old recipe could not be read under the new one. Contracts parsed before protocol version 14 ignore the keyword entirely.
The ecdh-secp256k1-aes256-cbc layout
This is the scheme the dashpay contact request already uses for encryptedPublicKey and encryptedAccountLabel (DIP-15), implemented in packages/rs-platform-encryption:
- The shared key is the libsecp256k1 ECDH of the sender's private key and the recipient's public key:
SHA256(parity || x)of the product point, whereparityis0x02for an evenyand0x03for an odd one. The sender's key is the identity key with the id thesenderKeyproperty carries, on the document's$ownerIdidentity; the recipient's key is the one with the id therecipientKeyproperty carries, on the identity therecipientproperty names (the owner itself for$ownerId). Either side derives the same 32 bytes from its own private key and the other's public key. - The writer draws a random 16-byte IV.
- The value is the IV followed by the plaintext encrypted with AES-256-CBC under the shared key and that IV, with PKCS7 padding.
So a ciphertext is 16 + 16 * ceil((len(plaintext) + 1) / 16) bytes: at least 32, always a multiple of 16. The reader splits the first 16 bytes off as the IV, derives the shared key from its own private key and the sender's public key, and decrypts the rest.
What consensus checks, and what it cannot
Consensus sees bytes, not keys. On every document create and replace, after the JSON schema validation of the document's properties, the structure validation (create structure generation 1, introduced at protocol version 14, and replace structure generation 0, extended in place: the call is inert before 14, where no property can carry the keyword) walks the document type's declared properties and, for each one the transition supplies, checks that its length is at least the scheme's IV plus one block and a multiple of the block length. A value that is not refuses the transition with InvalidEncryptedPropertyShapeError (basic code 10420), which names the property path, the scheme and the lengths involved. No state is read; the check runs in the mempool as well as in the block. The JSON schema's own minItems and maxItems are checked first, so a value outside them (a lone 16-byte IV against minItems: 32, say) is refused with the schema's error rather than 10420; the shape check only sees values the bounds already admit.
Nothing else is verifiable on chain: not that the bytes decrypt, not that they decrypt under the keys the document names, not that the named key ids exist on the identities or have an encryption purpose, and not that the plaintext is what the document type means it to be. A writer can store any 32 bytes. Whether the keys exist and are of the right kind is what the reference keywords are for: a refersTo of type identityPublicKey with keyIdProperty on the recipient property makes consensus check that the recipient's key exists, and encryptedFor neither duplicates nor requires it. Readers must treat a value that fails to decrypt as a bad message, not as a protocol violation.
In Rust the declaration is DocumentProperty::encrypted_for (Option<EncryptedFor>), listed per document type by DocumentTypeV0Getters::encrypted_properties(), and the shape check is DocumentTypeBasicMethods::validate_encrypted_property_shapes(), versioned on the validate_encrypted_property_shapes method slot (None before protocol version 14, which is what keeps the in-place replace call inert). In JavaScript, contract.documentTypeEncryptedProperties(name) and contract.documentEncryptedProperties expose the same declarations, and the shape error reaches an app as DocumentEncryptionErrorCode.InvalidEncryptedPropertyShape.
Clients encrypt and decrypt through the declaration rather than a per-contract recipe. The Rust SDK's dash_sdk::platform::encrypted_for module has encrypt_property, which writes the ciphertext and both key id properties, and decrypt_property. EncryptedPropertyEnvelope::read names the identities and key ids a reader needs. select_encryption_keys picks the keys the document type's identityPublicKey references demand through their keyRequirements. In JavaScript the same helpers are sdk.encryptedFor.encrypt, decrypt and envelope (WasmSdk.encryptDocumentProperty, decryptDocumentProperty and encryptedPropertyEnvelope). The layout has no authentication tag, so a wrong key fails the padding check except about once in 256 attempts, when it yields garbage.
Byte Caps on Strings (maxBytes)
Protocol version 14 adds the property keyword maxBytes, a bound plain JSON Schema cannot count: the most bytes a string may take in UTF-8. maxLength counts characters, and a character is up to four bytes, so maxLength: 4096 alone admits values up to the 5120-byte cap every field has (SystemLimits::max_field_value_size), not 4096 bytes.
"description": {
"type": "string", "minLength": 1, "maxLength": 4096,
"maxBytes": 4096,
"position": 1
}
The keyword goes on a string property, or on the items of a typed array of strings, where it bounds every element; it is refused on the array itself and on elements of any other type. It is an integer from 1 to 65535 and no lower than minLength, since a string of minLength characters is at least that many bytes. On contract update it moves like maxLength: raising or removing it is compatible, adding or lowering it is not.
The parser (generation 3, meta-schema v3, apply_max_bytes) folds the bound into the string's StringPropertySizes::max_bytes, next to max_length, so the sizes the type reports take it into account: max_byte_size is the smaller of maxBytes and four bytes a character, and random documents stay within it.
The check runs where the JSON schema validation of a document's properties runs, DataContract::validate_document_properties, right after it: on every document create and replace, and in every client that validates a document before sending it. A longer string is refused with DocumentPropertyMaxBytesExceededError (basic code 10421), which names the property (tags[2] for an element) and both lengths. The document validation (version 0, extended in place) is inert before protocol version 14, where no string carries a byte cap and the validate_max_bytes method slot is None. In Rust the check is DocumentTypeBasicMethods::validate_max_bytes_properties(); in JavaScript the error reaches an app as DocumentMaxBytesErrorCode.MaxBytesExceeded.
Generated Properties (generatedFrom)
Protocol version 14 adds the property keyword generatedFrom: the platform generates a string property's value with a system function of other properties of the same document, its params. The first system functions change the case of a string (lowercase, uppercase, capitalize, camelCase, snakeCase) or apply the DPNS domain rule (normalizedLabel is label lowercased, with o, i and l replaced by 0, 1 and 1), so any contract can build a unique index that treats look-alike names as one.
"normalizedLabel": {
"type": "string", "maxLength": 63,
"generatedFrom": {
"function": "sys.stringTransformations.homographSafeASCII",
"params": ["label"]
},
"position": 1
}
The functions are a closed list of system functions (SystemFunction), named under sys. so that functions a contract may bring later can be told apart by name; each declares how many params it takes. SystemFunction is an enum of namespaces, each an enum of its own in a module of its own: SystemFunction::StringTransformation(StringTransformation) for sys.stringTransformations, in system_function::string_transformations, whose six functions each take one string. lowercase, uppercase and capitalize change the case of ASCII letters; camelCase and snakeCase split the string into words (at every ASCII character that is neither a letter nor a digit, and before an ASCII uppercase letter that starts a word) and join them as helloWorld or hello_world; homographSafeASCII maps A to Z to lowercase, then o to 0 and i and l to 1. Every one changes ASCII characters only and keeps every other character, with no Unicode table, because Unicode case mappings differ between releases of the standard library, and two nodes must never generate different values; every one gives back its own output unchanged, and a test pins that the meta-schema lists exactly the functions the parser knows. On ASCII homographSafeASCII equals convert_to_homograph_safe_chars, the function the DPNS trigger uses; a test pins that over every ASCII character and over strings from the DPNS alphabets. It refuses nothing: the characters a value may hold are each param's pattern's to decide, and the generated property needs no pattern of its own, since every value it holds is generated from params that passed theirs. A param is a property path for now (GenerationParam::Property); literals, system values and nested calls can be added later without changing what parses today.
The parser (generation 3, meta-schema v3, apply_generated_from) reads the declaration onto DocumentProperty::generated_from (Option<GeneratedFrom>, absent on every property parsed before protocol version 14) and checks that params lists as many params as the function takes. validate_generated_from_declarations checks every param against the other properties on every parse: another string property, not generated itself, not transient nor inside a transient object (and neither is the declaring property), and inside every object that holds the declaring property. The meta-schema refuses the keyword beside $ref, whose definition replaces every keyword written next to it. On contract update a changed, added or removed generatedFrom is an incompatible schema change, and a property the update adds may declare it only when one of its params is new too (validate_update 1 refuses one whose params all existed with DocumentTypeUpdateError, 40212: documents stored before the update were never generated). The declaring paths and their declarations are cached on the document type (DocumentTypeV2Getters::generated_from_fields), so a write to a type without declarations pays nothing.
Three methods do the work at write time:
DocumentTypeBasicMethods::fill_generated_propertieswrites every declared property the document leaves out while supplying every param. The action transformers of document create, replace and index-only delete call it first, before the contest resolution and every check read the data, so the stored document, its indexes and its contest all hold the generated value.Document::try_from_create_transitionandtry_from_replace_transitioncall it too, and so doesindex_only_transition_entry_path_query, the one builder the prover and the verifier share for index-only entries, so proofs are built and checked against the document the platform stored.DocumentTypeBasicMethods::data_as_storedreturns a transition's data with the generated properties written into a copy, or borrows it as it is on a type that declares none; the document subscription filter (DriveDocumentQueryFilter::matches_document_transition) reads a create's and a replace's data, and an index-only delete's values, through it, so a subscription on a generated property sees the value the platform stores.DocumentTypeBasicMethods::regenerate_generated_propertiesis the client-side twin: it sets every declared property to what its params generate, replacing a value the document holds and removing it when a param is absent. The transition builders (from_documentof create, replace and index-only delete), the SDK's contest fund lookup and the property-constraint pre-checks of the JavaScript and FFI SDKs call it, so a document fetched, edited and sent back carries the value of its new params rather than the stale one the platform would refuse, and its contest is detected from it. Random documents call it too, after drawing the params of every generated property they drew.DocumentTypeBasicMethods::validate_generated_from_propertiesruns inDataContract::validate_document_properties, after the JSON schema andmaxBytes: a declared property must equal what its function generates from its params, and be absent when a param is. A document that repeats a key in an object on the way to the property or to a param is refused too: the schema validation and the stored document keep the last of repeated keys, where the path reads the platform generates from find the first. A property that does not pass refuses the write withDocumentPropertyNotGeneratedError(basic code 10424). A document that arrived has been generated, so on the platform the check only refuses a value the client sent, a value sent without its params, or a repeated key.
Every call site was extended in place and is inert before protocol version 14: the apply_generated_from, fill_generated_properties and validate_generated_from slots are None there, and the meta-schemas refuse the keyword. In JavaScript the error reaches an app as DocumentGeneratedFromErrorCode.DocumentPropertyNotGenerated.
Property Constraints (propertyConstraints)
Protocol version 14 adds the doctype-level propertyConstraints keyword: named rules the properties of every created or replaced document must meet, where JSON Schema can only bound one property at a time. Each rule is a condition: a comparison of two integer expressions, a test of whether an integer expression takes one of listed values, a comparison of a string property with string constants, a test of whether the document holds a property, or anyOf, allOf or not over conditions.
"propertyConstraints": {
"depositCoversOrder": {
"lessThanOrEqual": [
{ "multiply": [{ "add": ["price", "fee"] }, "quantity"] },
"deposit"
]
},
"wholeLots": { "equal": [{ "modulo": ["quantity", 10] }, 0] },
"minimumOrder": {
"greaterThanOrEqual": [{ "multiply": ["price", { "ifAbsent": ["quantity", 1] }] }, 100]
},
"feeWaivedOrAtLeastTen": {
"anyOf": [{ "equal": ["fee", 0] }, { "greaterThanOrEqual": ["fee", 10] }]
},
"discountGivenAboveZero": {
"anyOf": [{ "absent": "discount" }, { "greaterThan": ["discount", 0] }]
},
"tieredFee": { "in": ["fee", [0, 10, 25, 50]] },
"closedNeedsClosedAt": {
"anyOf": [{ "notEqual": ["status", { "const": "closed" }] }, { "present": "closedAt" }]
}
}
The first rule reads ((price + fee) * quantity) <= deposit, feeWaivedOrAtLeastTen reads fee == 0 || fee >= 10, discountGivenAboveZero lets an offer leave its discount out but not give a discount of 0, tieredFee holds the fee to four tiers, and closedNeedsClosedAt says a closed offer carries a closedAt. A rule's name is 1 to 64 letters, digits or underscores, and the rule is a condition, an object with one key:
- a comparison,
equal,notEqual,lessThan,lessThanOrEqual,greaterThanorgreaterThanOrEqual, listing the left and the right expression; { "in": [expression, [values]] }, holding if the integer expression takes one of two or more distinct integer values. It says what ananyOfofequals says, in one node per value instead of three, so a set of up to 30 values fits the node limit where theanyOffits 10. A value is a literal, never a path or an expression;- a string comparison:
{ "equal": [path, { "const": "closed" }] }ornotEqual, with the constant on either side;{ "notEqual": ["fromCurrency", "toCurrency"] }, two bare paths that both name string properties, which compares their strings; or{ "in": [path, ["open", "pending"]] }, whose values are two or more distinct strings. The path names a string property, typically one with anenum. A string on its own is a path, so a constant is written as{ "const": ... }, while the values aninlists are literals and need no wrapper. Strings are only compared for equality, never ordered or used in arithmetic. A string property the document leaves out equals no constant and no other string property, not even one also left out, sonotEqualholds for it andequalandindo not, unless{ "ifAbsent": [path, "open"] }gives it a string default, which it then reads as (it may stand wherever the bare path does, and makes the comparison one of strings);presentandabsenttest it directly. When the property declares anenum, every constant compared with it must be one of the enum's values, so a misspelling is refused at registration rather than making the rule quietly never hold; - an identifier comparison, the same three forms for identifier properties, those declaring
refersToincluded:{ "equal": ["paymentToken", { "const": "<base58>" }] }ornotEqual,{ "notEqual": ["buyerId", "sellerId"] }, or{ "in": ["paymentToken", ["<base58>", "<base58>"]] }. Constants are base58 identifiers of 32 bytes, checked at registration, and compared by their bytes, whatever form the document gives the identifier in. An identifier property the document leaves out equals no identifier, not even another one left out; identifiers take noifAbsentdefault and are never ordered.$ownerId, the document's owner, is an identifier operand too:{ "equal": ["authorId", "$ownerId"] }holds the author to the owner, and{ "in": ["$ownerId", ["<base58>", ...]] }lets only the identities listed own a document of the type. It is no property, sopresentor an integer operand refuses it, and comparing it with itself is refused. Since a transfer and a purchase give the document a new owner, each is judged against the rules reading$ownerId, with the new owner, and refused when it would break one; an indexOnly type refuses such a rule, since its deletes carry no owner; { "startsWith": [text, affix] }and{ "endsWith": [text, affix] }, holding if the first string starts or ends with the second, byte for byte with no case folding: each side a{ "const": ... }, a string property or anifAbsentstring default, at least one a property, never the same one twice.{ "startsWith": ["url", { "const": "https://" }] }holds a link to https, and{ "startsWith": ["path", "parentPath"] }a path under its parent's. A string property left out without a default takes no string, and the condition does not hold for it; a constant tested against a property that declares anenummust start or end one of its values;{ "contains": [path, value] }, holding if the typed array property at the path holds an element equal to the value, looked for as the array's elements are: an integer expression among integers, a string constant, string property orifAbsentstring default among strings, an identifier constant, identifier property or$ownerIdamong identifiers.{ "not": { "contains": ["labels", { "const": "used" }] } }refuses a"used"label, and{ "contains": ["participants", "$ownerId"] }holds the owner to the participants (so a transfer or purchase to a non-participant is refused). An array the document leaves out holds nothing, a string or identifier property it leaves out is among no elements, and a string constant must be one of the elements'enumvalues when they declare one;{ "present": path }, holding if the document holds the property, and{ "absent": path }, holding if it leaves it out (a property set to null counts as left out, and so does an object none of whose members is present, such as{}, which a stored document does not keep). An operand reads a property the document leaves out as 0, so only these tell "not given" from "given as 0". They may name a property of any type, an object or a member of one included, since they read no value;{ "anyOf": [...] }, holding if at least one of two or more conditions holds;{ "allOf": [...] }, holding if every one of two or more conditions holds;{ "not": condition }, holding if its one condition does not;{ "ifThen": [a, b] }, holding ifbholds wheneveradoes:bis evaluated only whenaholds, so{ "ifThen": [{ "greaterThan": ["discount", 0] }, { "greaterThanOrEqual": [{ "divide": ["price", "discount"] }, 10] }] }never divides by zero, and a fault in either breaks the rule. It says what{ "anyOf": [{ "not": a }, b] }says, in one node fewer. The two may not be alike;{ "ifThenElse": [a, b, c] }, holding ifbholds whenadoes andcholds when it does not; only the branchaselects is evaluated.{ "ifThenElse": [{ "greaterThanOrEqual": ["price", 1000] }, { "lessThanOrEqual": ["fee", 50] }, { "lessThanOrEqual": ["fee", 10] }] }allows a higher fee on an expensive offer. No two of the three may be alike;{ "notIn": [expression, [values]] }, aninnegated, listed the same way and in as many nodes:{ "notIn": ["fee", [7, 13]] }refuses two fees.
Conditions nest: { "not": { "allOf": [{ "equal": ["price", 0] }, { "greaterThan": ["quantity", 10] }] } } refuses a free order of more than 10. An anyOf or allOf may not list two alike conditions, nor hold one of its own kind directly (it says what one flat list says), and a not may not hold a not or a notIn directly. An expression is one of:
- an integer value (
100; a float with no fractional part,100.0, reads as that integer, as the meta-schema'sintegertype admits it); - a string, the dotted path of an integer or boolean property of the document type (
"price","meta.total","waiveFee"), whose value it takes, 0 when the document leaves the property out. A boolean reads as 1 for true and 0 for false, so{ "equal": [{ "multiply": ["waiveFee", "fee"] }, 0] }says a waived fee is 0; { "ifAbsent": [path, value] }, the property's value, orvaluewhen the document leaves it out (an integer value here; a string value gives a string property a default in a string comparison instead);{ "add": [...] }or{ "multiply": [...] }over two or more operands;{ "min": [...] }or{ "max": [...] }, the least or greatest of two or more operands, every one evaluated (a fault in any breaks the rule), and{ "abs": a }, the absolute value of one:{ "lessThanOrEqual": ["fee", { "max": [10, { "divide": ["price", 10] }] }] }caps a fee at 10 or a tenth of the price, whichever is more, and{ "lessThanOrEqual": [{ "abs": { "subtract": ["a", "b"] } }, 5] }keeps two values within 5;{ "subtract": [a, b] },{ "divide": [a, b] },{ "modulo": [a, b] }or{ "power": [a, b] };- a size:
{ "length": path }, the characters of a string property (counted asmaxLengthcounts them),{ "byteLength": path }, its UTF-8 bytes (asmaxBytescounts them), or{ "count": path }, the items of an array property or the bytes of a byte array property. WheremaxLength,maxBytesandmaxItemsbound one property by a fixed number, a size can be compared with another property or bounded only under a condition:{ "lessThanOrEqual": [{ "count": "tags" }, "maxTags"] }holds a list to its own limit, and{ "anyOf": [{ "greaterThan": ["fee", 0] }, { "lessThanOrEqual": [{ "length": "title" }, 20] }] }keeps a free listing's title short. A property the document leaves out or sets to null has size 0, and a size never breaks a rule by itself (a value of another type would read as 0 too, but the schema validation refuses it first); { "countPresent": [path, path, ...] }, how many of two or more properties, no two alike, the document holds, each aspresenttests it (any type, null and an object with no member present counting as absent), so a rule bounds how many of a group are set:{ "equal": [{ "countPresent": ["email", "phone", "handle"] }, 1] }asks for exactly one,{ "lessThanOrEqual": [..., 1] }for at most one, and{ "in": [..., [1, 2]] }for one or two. It is one node plus one per path, where the same bound written withpresenttakes ananyOfand anotof every pair;- a system time or height:
"$createdAt","$updatedAt"and"$transferredAt", block times in milliseconds, and each withBlockHeightorCoreBlockHeightappended, the Platform and Core block heights: those of the document's creation, of its last create, replace or price update, and of its last create, transfer or purchase. A rule may read one only on a type that records it by listing it inrequired, so every stored document holds it; it takes noifAbsent, and an indexOnly type, whose deletes carry none, reads none.{ "lessThanOrEqual": [{ "subtract": ["endsAt", "$createdAt"] }, 604800000] }keeps a listing to a week from its creation, and{ "lessThanOrEqual": ["$updatedAt", "endsAt"] }refuses a replace or a price update after it ends; - a total read from state:
{ "countOf": [type] }or{ "countOf": [type, filter] }, how many documents of a type of the same contract there are, or how many match the filter, and{ "sumOf": [type, property] }or{ "sumOf": [type, property, filter] }, the total of an integer property over them. A filter maps each key, a property of the counted type or$ownerId, to the value it must take, read from the document being written: one of its properties,$ownerId,$id(its id, so{ "countOf": ["vote", { "pollId": "$id" }] }counts the votes naming it), an integer or a{ "const": ... }. The total is the one a count or sum tree keeps, as it will be once the write is done (the document itself counted when the type is its own, at its new values, and moved to its new owner by a transfer or purchase), so{ "lessThanOrEqual": [{ "countOf": ["listing", { "$ownerId": "$ownerId" }] }, 10] }onlistingkeeps every owner at ten listings or fewer. A whole-type total needsdocumentsCountableordocumentsSummable, and a filtered one an index of the counted type whose properties are exactly the filter's keys, countable or summing the property. A rule is judged only when its own type is written, so a fact about another type can go stale after the write.
A JSON number is always a value and a string always a path, so a property named 100 is not confused with the number, and the rule is a tree the meta-schema can check rather than a string with precedence rules to parse. Consensus holds nothing but this tree; an SDK may offer an infix spelling that compiles to it.
The arithmetic is exact over i128. Operands are evaluated left to right, and every intermediate result must fit: an overflow, a divisor of 0, a negative exponent or a property value that is not an integer (a float with no fractional part passes the schema's integer type) breaks the rule instead of wrapping or truncating. divide and modulo are Euclidean, so the remainder is never negative and the quotient is the one that goes with it (-7 by 2 is -4 remainder 1); for operands that are not negative this is ordinary integer division. 0 to the power 0 is 1. There are no floats: a number property cannot be read, which keeps every node's result bit-identical.
Conditions are checked in declared order and no further than the outcome needs: a comparison evaluates its left side, then its right; anyOf stops at the first condition that holds and allOf at the first that fails. A fault in a condition that is checked breaks the rule whatever the others would say, and not does not turn it into a pass. So an earlier condition guards a later one: { "anyOf": [{ "equal": ["b", 0] }, { "equal": [{ "divide": ["a", "b"] }, 2] }] } holds for a b of 0 without dividing by it, while the same two conditions the other way round divide by zero and break the rule.
The parser (generation 3, meta-schema v3) checks the keyword on every parse, stored contracts included: the shape, that every path an operand reads names an integer or boolean property of the type, every path length or byteLength measures a string property, every path count counts an array or byte array property, every system time or height a rule reads is one the type lists in required (and an indexOnly type reads none), every path compared with a string names a string property (and every constant compared with one that declares an enum is one of its values) (a nested one by its dotted path) and every path present, absent or countPresent tests names a property of the type, and that neither is transient nor inside a transient object (a transient value is never stored, so a stored document could not be held to the rule), that every comparison and in reads at least one property (a constant one would make its rule, or an anyOf around it, hold for every document or for none), that no in lists a value twice and no countPresent a path twice, that no anyOf or allOf holds one of its own kind directly and no not a not, that no literal divisor is 0 and no literal exponent negative, and that no condition or operand nests deeper than MAX_PROPERTY_CONSTRAINT_PARSE_DEPTH (64), a constant that keeps a parse without full validation from recursing without bound and that no registrable rule comes near. Under full validation, when a contract registers or updates, it also holds the limits: at most SystemLimits::max_property_constraints rules per type (16) and max_property_constraint_nodes nodes per rule (32), counting every comparison and logical operator, every in and each value it lists, every const, every present or absent, every arithmetic operator and every operand (a size is one, a countPresent one plus one per path), that no anyOf or allOf lists the same condition twice (conditions that parse alike, so 1 and 1.0 are the same value), and that the rules of one type read at most max_property_constraint_aggregates (4) distinct totals. Once every document type of the contract is parsed, a registration also checks every countOf and sumOf: the type it counts is one of the contract's and not indexOnly, nor the declaring type when it has a contested index (a document a contest awards is stored without the rules judged), a tree of it keeps the total, the filter's keys and values are integers, strings or identifiers of the same kind, and every property a value reads is required. The rules are fixed when the document type is created: adding, removing or changing one is an incompatible schema change (IncompatibleDocumentTypeSchemaError, 10246), since stored documents were judged against the rules as they were.
Enforcement lives in DataContract::validate_document_properties (generation 0, extended in place: the call is inert before protocol version 14, where validate_property_constraints is None), after the schema validation. Document create and replace structure validation call it, so consensus applies the rules, and so does every client that validates a document before sending it. The rules are checked in name order against the document's properties, which for a replace is the whole document, and the first one broken fails with DocumentPropertyConstraintViolatedError (basic code 10422), naming the document type, the rule and why: the rule does not hold, or evaluating it overflowed, divided by zero, raised to a negative power or read a value that is not an integer. The check changes nothing stored. It reads state only for the countOf and sumOf totals, which the document batch transformer reads when it builds the action (Drive::fetch_property_constraint_aggregate, billed with the write) and hands the check in DocumentSystemValues::aggregates; the limits bound its cost. $ownerId and the system times and heights read the values of the document version being written (validate_document_properties takes them as DocumentSystemValues): on a create the writer and the block's time and heights, on a replace the writer, the stored creation and transfer values and the block's as the update. A client passes what it knows: an owner it does not know equals no identifier, and a rule reading a time, a height or a total it is not given is not judged. Consensus reads every total the rules it judges read (DocumentSystemValues::aggregates is Some), so one missing there is an error in the code building the write, never a skipped rule. The SDK pre-checks use the device clock for the times a write records, and leave a rule reading a block height unjudged, since the height is unknown until the block. Transfers and purchases change no property, only the owner and the transfer's time and heights, so only the rules reading those are judged again, with the new values (DocumentTypeV0Methods::validate_property_constraints_for_system_change, next to the distinctFrom check). Price updates change only the update's time and heights, and are judged against the rules reading those the same way.
In Rust the rules are DocumentTypeV2Getters::property_constraints (a map from name to PropertyConstraint: a comparison, an in, a string comparison (TextCompare, TextIn), a present or absent, or an anyOf, allOf or not of them, empty on types that predate the keyword; property_reads lists what a rule reads and how: by value, by presence, by size (Length, Count) or in a comparison of strings or identifiers, system_reads the system times and heights it reads, and aggregate_reads the totals, each an AggregateRead), each rule's holds and violation evaluate it against a document's data and DocumentSystemValues, and the document check is DocumentTypeV0Methods::validate_property_constraints.
Delete rules (deleteConstraints)
The same grammar gates the owner's delete. deleteConstraints is a doctype keyword of named rules the stored document must meet for its owner to delete it (see Deletion):
"deleteConstraints": {
"noVotes": { "equal": [{ "countOf": ["vote", { "pollId": "$id" }] }, 0] }
}
The parser (apply_delete_constraints_v0, generation 3, called by apply_property_constraints so one dispatch on parse_property_constraints covers both keywords) reads the rules with parse_delete_constraints, the same parse as parse_property_constraints under another key, checks what they read with validate_property_constraint_reads and, under full validation, holds them to the same SystemLimits, counted apart from propertyConstraints. On every parse it refuses the keyword on a type whose owner never deletes a stored document: canBeDeleted: false, "onlyWhenConsumed" or indexOnly. validate_property_constraint_aggregates checks their countOf and sumOf with the contract's other types, except that a delete rule may total its own contested type, since the total it reads is the stored one. DocumentReferenceLookup::referenced_side_error refuses a consume into such a type, which would delete without a delete transition. The rules are frozen on update by the same compatibility rule as propertyConstraints (10246).
Enforcement is document delete state validation 1 (protocol version 14, DRIVE_ABCI_VALIDATION_VERSIONS_V10): the checks of version 0 (the base transition, the stored document, its owner), then, for a type with rules, read_delete_constraint_aggregates reads the totals (billed, each as it will be once the document is gone: the stored total less what the document adds to it) and DocumentTypeV0Methods::validate_delete_constraints judges each rule in name order on the stored document's properties and DocumentSystemValues::of_document. The first rule broken refuses the delete with DocumentDeleteConstraintViolatedError (state code 40147), naming the document, the rule and why, so the delete is paid and the nonce spent. In Rust the rules are DocumentTypeV2Getters::delete_constraints, and a filter value $id is AggregateBinding::Id, which AggregateRead::filter_values reads from the document's id.
Rules and Guidelines
Do:
- Always serialize and deserialize documents using their document type definition. The type determines field layout.
- Use
is_equal_ignoring_time_based_fields()when comparing documents for validation purposes. - Use
increment_revision()rather than manually manipulating the revision field -- it handles overflow checking. - Access properties through the accessor traits, not by reaching into the inner
DocumentV0struct.
Do not:
- Assume a document carries its contract reference. The contract and document type are always passed as separate arguments.
- Manually construct document IDs. Use
Document::generate_document_id()with proper entropy, the nonce of the create transition and the network's platform version. - Rely on the ID of a document that has not been sent yet. It is a placeholder until the create transition is built.
- Treat serialized document bytes as self-describing. Without the document type schema, the bytes are meaningless.
- Set time-based fields from client code. The platform sets
created_at,updated_at, block heights, and similar fields during state transition processing.
Document Time To Live
A document type may declare a time to live. Every document of such a type is deleted by the platform once its time to live has passed, whoever owns it and whatever the type says about who else may delete it. Its owner pays, when the document is written, for the time the document will occupy the state rather than for perpetual storage, and prepays its deletion. Available from protocol version 14.
"note": {
"type": "object",
"ttl": 1209600,
"properties": { "text": { "type": "string", "maxLength": 280, "position": 0 } },
"required": ["$createdAt", "text"],
"additionalProperties": false
}
Documents of note above are deleted two weeks (1,209,600 seconds) after their creation.
This is a different feature from the ttl key of a timeRange index (see
Time-Range Index TTL), which expires index entries and leaves
the document in place.
Semantics
- A document expires at its
$createdAtplusttlseconds.$createdAtis set from block time when the document is created, so nothing the writer sends moves the expiry, and every document of a type created in one block expires at the same time. - A replace, a transfer, a purchase or a price update never moves the expiry:
$createdAtnever changes. A buyer of an expiring document buys what is left of its life. - After each block's state transitions the platform deletes the expired documents, oldest
first, at most
max_document_expirations_per_blockper block (128 at protocol version 14), and no more thanmax_document_expiration_weight_per_block(1,024) in weight, where a document weighs 1 plus the index levels of its type. The rest wait for the next block. A state transition in the block a document expires in still sees it; a document can therefore outlive its expiry by the part of a block before the cleanup, and by more when a backlog builds. - The expiry belongs to the document: it is its own
$createdAtplus the type'sttl, both fixed, and the platform reads it from there. The expirations tree is only the index the cleanup finds documents through. - From its expiry on, a document can no longer be replaced, transferred, bought or repriced,
and a moderator can no longer restore it: each is refused, paid, with
DocumentExpiredError(40140), whether or not the cleanup has reached the document. Its owner may still delete it wherecanBeDeletedallows, which only removes it sooner. Until the cleanup deletes it, it can still be queried and referenced, and it keeps its values in the type's unique indexes: a create of the same unique value is refused as a duplicate until the cleanup has run. The cleanup deletes a fixed number per block, so a sustained flood of creations with a short time to live builds a backlog it drains at that rate, and that lag grows with it. - The document may be deleted earlier as usual: by its owner when
canBeDeletedallows it, by the contract's moderators whenmoderatorAbilities.deletedoes.canBeDeleted: falseonly stops the owner; the platform still deletes the document when it expires. - The deletion is an ordinary deletion: the document and every index entry go, count and sum trees are decremented, and nothing is left behind.
Where a time to live is refused
The keyword is refused, on every parse of the contract (registration, update and a stored contract read back), when:
| The document type | Why |
|---|---|
does not list $createdAt in required | The expiry is computed from it. |
sets documentsKeepHistory: true | Drive refuses to delete a document whose type keeps history. |
sets indexOnly: true | There is no stored row to delete by id. |
| has a contested index | A contested document waits in its vote poll until the poll awards it, keeping the $createdAt of its create, so it could expire before it is stored. |
declares ttl: 0 | A time to live lasts at least a second. |
When a contract is registered or updated, a ttl below min_document_ttl_seconds (one hour at
protocol version 14) or above max_document_ttl_seconds (one year) is refused too. The floor
keeps a document in state well past the moment its writer fetches the proof of its create, which
proves the document present; a document the cleanup had already deleted would fail that proof
although the create succeeded. A summed property that admits values below 0 is refused there
too, unless its minimum and maximum lie within ±2^27
(max_expiring_signed_summed_value_magnitude): the cleanup takes a document's value out of the
type's sums with no transition to refuse, and removing a negative value raises them, so the
values must keep every sum in the signed 64-bit range. Removing values that are never negative
only lowers the sums, and values within ±2^27 keep them in range short of 2^36 documents. A
contract update may not add, remove or change the ttl of an existing document type: every
stored document carries the expiry it was written and paid with. A document type added by an
update declares it freely.
The same holds for any write close to a document's expiry: a replace, transfer, purchase or price update accepted in the last block before the expiry proves the document present, and a proof fetched after the next block's cleanup finds it gone.
References treat a type with a ttl as deletable, like one with canBeDeleted or
moderatorAbilities.delete. A permanentDocument reference, a lookup one included, and a list
element reference may not target it; a deletableDocument reference may, a lookup one included.
The check is DocumentTypeV2Getters::documents_can_disappear.
Everything else composes: mutable types, transferable, tradeMode,
moderatorAbilities.delete (a moderator's restore puts the document back with its original
$createdAt, and is refused once that document has expired),
creationRestrictionMode, countable, summable and ranked indexes, timeRange indexes with
or without their own ttl, refersTo declared on the type, action fees and token costs.
Storage
Nothing a document of such a type writes carries storage flags: its primary item, its index
entries and the index trees it creates are flagless, and its deletion refunds nothing.
Drive drops the writer's flags when the document is inserted or changed
(DocumentAndContractInfo::without_storage_flags_if_expiring), before any element is sized,
and the re-tagging described below strips any left.
Each document also gets an entry in the documents expirations tree under Misc:
Misc (104) / E / <expires at, u64 big endian ms> / <document id> -> contract id (32 bytes) ++ document type name
The entry is written with the document and removed with it, whoever deletes it (the hook is
in force_delete_document_for_contract_operations, which the owner's, the moderators' and the
cleanup's deletions share). The last entry of an expiry time takes the tree of that time with
it, so every tree of an expiry time holds at least one entry, and a document deleted early
leaves nothing the cleanup would have to read.
The storage fees of such documents wait in the lifetime storage fee pools under Pools,
a sum tree so the pools' total counts them, one sum item per epoch they were collected in and
number of epochs their storage lives:
Pools (48) / l / <collected in epoch, u16 big endian><lifetime in epochs, u16 big endian> -> credits
Both trees are created with the initial state structure of protocol version 14 and on the
first block of protocol version 14, through one helper (Drive::insert_document_ttl_trees).
Fees
The fee schedule's document_ttl group (FeeDocumentTtlVersion, FEE_VERSION3) prices a
document of such a type:
-
Bytes. Every byte the document writes, its expirations tree entry included, costs the price of the lifetime it has left. Up to seven days a tier applies; past that a price per
pricing_period_secondsspanned, rounded up. The period is part of the schedule (788,400 seconds, mainnet's epoch length), not the node's epoch length, so a network configured with short epochs, like testnet's hour, prices a lifetime as mainnet does.Lifetime Credits per byte (protocol version 14) up to 1 hour 1 up to 1 day 4 up to 2 days 8 up to 4 days 15 up to 7 days 26 longer 34 per 9.125 days spanned The values are the first year's share of the perpetual storage price (27,000 credits per byte, 5% of it paid out in the first year) pro rata, rounded up. A one-year
ttlpays 40 × 34 = 1,360 credits per byte, about what a permanent document deleted after a year keeps paying net of its refund. -
Payout. That amount is a storage fee, paid out to the epochs the document lives in rather than over the 50 eras of the perpetual storage distribution. Drive counts the epochs its remaining lifetime spans, rounded up and at most one era (40 epochs), with the network's epoch length (the node's
epoch_time_length_s, handed to Drive throughDriveConfig) and reports the amount under that count (FeeResult::lifetime_storage_fees). At the end of the block the amounts go to the lifetime storage fee pools of the current epoch (add_distribute_block_fees_into_pools_operationsv1), and the next epoch change spreads each pool of an earlier epoch evenly over its number of epochs from the new epoch on, the remainder of the division to the new epoch, and removes it (add_distribute_storage_fee_to_epochs_operationsv1). A block never adds to a pool its epoch change removes, so the first block of an epoch does both in one batch. A network with hour-long epochs therefore pays a year-long document out within 40 hours. -
Deletion. Creating the document prepays, as processing, what its deletion will cost:
cleanup_base_processing_cost(1,200,000), pluscleanup_processing_cost_per_index_level(400,000) per index level of the type, where an index counts its properties, times the overlapping windows of atimeRangeindex, pluscleanup_processing_cost_per_document_byte(420, what removing and reading a byte costs) per byte of the stored document. The cleanup itself bills nobody. -
Changes. A replace, transfer, purchase or price update prices the bytes it adds by the lifetime left at that block, and prepays the deletion of the document bytes it adds at
cleanup_processing_cost_per_document_byte. A deletion by the owner or a moderator pays its own processing like any deletion; the prepaid deletion is the platform's, and is not refunded.
The price never decreases with the lifetime, so an estimate made at an earlier block time
(check_tx) stays an upper bound of the execution; where the amount is paid out does not change
what the writer pays, and a fee increase the writer offers, which multiplies processing only,
never multiplies it. A dry
run estimates a change as an insert of the changed document
(estimate_document_change_as_insert_operations_v1): without the entry and the deletion the
creation prepaid, which a change never pays, and with the deletion of all the document's
bytes, at least what the change adds. In Drive the document's grove operations are
re-tagged EphemeralGroveOperation(_, EphemeralPricing::DocumentTtl { .. }) and applied as
their own GroveDB batch, so their added bytes can be priced apart from the rest of the
transition (see apply_batch_low_level_drive_operations); operations already tagged for a
timeRange index's ttl keep that rule.
Expired documents
validate_document_not_expired (drive-abci, state_transition/common) is the one rule: a
document of a type with a ttl has expired when block time is at or past its $createdAt
plus the ttl (Drive's document_expires_at, the time its entry is keyed by), the same
boundary the cleanup deletes at. The batch's advanced structure validation (v1, protocol
version 14 only), which check_tx runs too, calls it for every document action through
validate_document_action_not_expired, an exhaustive match that refuses, paid, a replace,
transfer, purchase or price update of an expired document. The moderator restore calls it too, judging the
$createdAt of the document the removal record's hash pins. A document deletion by its owner
does not.
Cleanup
Platform::expire_documents runs after the block's state transitions, right after the address
balance cleanup in run_block_proposal, and calls Drive::remove_expired_documents with
max_document_expirations_per_block and max_document_expiration_weight_per_block:
fetch_expired_documentsreads, in one query, the entries of the expiry times at or before the block time, oldest first, at most the limit of them. Every tree of an expiry time holds an entry, so the query visits at most the limit of trees.- Each expired document is read and checked against state (its contract, document type,
ttland stored document, and that the document expires when its entry says), then deleted from what was read: the deletion an owner runs, without thecanBeDeletedguard (delete_read_document_for_contract_operations_v0). Its entry, and the tree of its expiry time when it was the last, go with it. - An entry without a document to delete is logged and only removed. None is expected, and failing the block over one would halt the chain.
- The cleanup stops before a removal that would pass the weight budget (an entry's removal weighs 1), except the block's first, so the backlog always drains.
Every removal goes into one GroveDB batch, applied without a fee, each built against the removals queued before it, so every emptiness check sees them.
Versioning
Everything rides protocol version 14, unreleased when this landed: the keyword joined document
meta-schema v3 and the generation 3 parser, the limits joined SYSTEM_LIMITS_V4, the fee group
joined FEE_VERSION3, the update rule joined validate_update v1, and expire_documents is
Some(0) in DRIVE_ABCI_METHOD_VERSIONS_V10 only. The shipped generations edited in place are
inert before 14:
- the deletion hook in
delete_document_for_contract_operationsv0, the reference checks (documents_can_disappear) and the restore check ofcontract_user_moderationstate v0:documents_ttl_secondsis only everSomeon a document type parsed from the keyword, which no earlier protocol version reads. The deletion's post-read part moved intodelete_read_document_for_contract_operations_v0with its operations unchanged; - the
expire_documentscall inrun_block_proposalv0: the method isNonebefore 14; - the tree's creation in
transition_to_version_14(perform_events_on_first_block_of_protocol_changev0), which only an upgrade to 14 runs; - one batch per pricing rule in
apply_batch_low_level_drive_operationsv0 and theDocumentTtlarm ofconsume_to_fees_v0: nothing is tagged ephemeral before 14; add_distribute_block_fees_into_pools_operationsv0, split into a helper that v1 shares, with its operations unchanged, andfetch_pending_epoch_refundsv0, whose query and reading moved unchanged into a helper the lifetime storage fee pools share;- the pattern-only edits in
batch_insert_empty_tree_if_not_existsv0 andconvert_drive_operations_to_grove_operationsv0, whose output is unchanged.
Contested Documents
A unique index may be declared contested. A document whose index values fall in the contested
range is not stored outright: it opens or joins a contest, a ContestedDocumentResourceVotePoll
that names the contract, the document type, the index and the index values, and masternodes and
evonodes decide who gets the value. A masternode's vote counts once, an evonode's four times. Every
vote is a MasternodeVote state transition carrying the poll and a ResourceVoteChoice.
From protocol version 14, a vote towards an identity must name a contender of the poll. Any other
identity, including the reserved keys under which the poll keeps its stored result and its abstain
and lock tallies, is refused with VoteChoiceNotAllowedForVotePollError (40307). Before 14 such a
vote failed with an internal error, or counted as abstain or lock when it named a reserved key.
The contest is funded by the contenders' prefunded voting balances, and each vote costs a fixed amount from that balance. Contenders may join for the join window (one week on mainnet) after the first document; the contest runs for the poll duration (two weeks on mainnet). The first document's owner may not be joined by the same identity twice.
From protocol version 14, a contest accepts at most 1,000 contenders
(max_contenders_per_contest): a document that would add one more is refused, paid, with
DocumentContestMaximumContendersReachedError (40141). The bound is what lets one block end a
contest: its end tallies up to maximum_contenders_to_consider contenders (10,000 from version 14)
and removes the entries of those it tallied, so a contest within the cap is tallied and cleaned up
whole. Before 14 a contest accepted any number of contenders and its end tallied at most 100; a
contest that grew past 10,000 contenders before 14 is tallied and cleaned up for its first 10,000
only.
From protocol version 14, the fund a contender pays doubles once the contest holds 250 contenders
(contested_document_contenders_before_fund_doubling) and again for every 50 more
(contested_document_contenders_per_fund_doubling), so a contest stops growing long before its
cap:
| Contenders the contest holds | DPNS fund to join | Moderation election fund to join |
|---|---|---|
| 0 to 249 | 0.1 Dash | 0.5 Dash |
| 250 to 299 | 0.2 Dash | 1 Dash |
| 300 to 349 | 0.4 Dash | 2 Dash |
| ... | doubles every 50 | doubles every 50 |
| 700 to 749 | 102.4 Dash | 512 Dash |
| ... | doubles every 50 | doubles every 50 |
| 950 to 999 | 3,276.8 Dash | 16,384 Dash |
Filling a DPNS contest to 1,000 contenders costs 327,695 Dash (100 at a flat 0.1 Dash). A contender's
prefunded voting balance is the most it is willing to pay, and it must hold that much: it is
charged the fund to join the contest it joins, and what it stated beyond that stays with it. One stating less, the first
contender of a new contest included, is refused, paid, with DocumentContestNotPaidForError,
which carries the fund it has to pay. The SDKs read how many contenders a contest holds and state
that fund unless the caller names the most it will pay, which lets a join go through while others
join ahead of it. Before 14 every contender stated exactly the contest's fund, however many had
joined, and paid what it stated.
The index's contested.resolution says how the contest is decided.
From protocol version 14, an identifier property among the index values is written as an
identifier in the poll (Index::extract_contested_values), whether the document gave it as an
identifier, as 32 bytes or as an array of byte values. The index keys store all of these alike,
but a poll is hashed from its values, and that hash keys the contest's prefunded balance and end
date: two contenders writing the same identifier two ways would otherwise split one contest into
two polls. Before 14 the values are taken as given.
Resolution 0: masternode vote
The DPNS rule. The choices are a contender, abstain, or lock, which gives the value to nobody. The contender with the most votes wins unless the lock tally exceeds it, in which case the value is locked and may be contested again later. The contest always runs the full poll duration, even with a single contender, so the masternodes may lock the value.
Resolution 1: masternode vote without locking
ContestedIndexResolution::MasternodeVoteNoLocking, meta-schema v3 (protocol version 14). The
choices are a contender or abstain. A Lock vote is refused with VoteChoiceNotAllowedForVotePollError
(40307). The contest always ends with a winner: the contender with the most votes, no minimum.
A contest without locking ends when its join window closes while it still has a single contender,
so that contender is awarded the value without a vote window. Its end-date entry is written at the
end of the join window when the contest opens; the first additional contender moves it to the full
poll duration, which opens the vote window, and removes the join window's end date when no other
contest ends then. getVotePollsByEndDate shows whichever end applies.
The moderation charters contract uses this resolution to elect moderation teams.
Moderation elections
An electedCharter contest of the moderation charters contract (protocol version 14), keyed by
the target contract id, is a moderation election and does not take the generic parameters:
- Its join window and vote window are the
joinWindowandvoteWindowof the target contract's elected moderation declaration (at most four weeks each, one week by default; at least a day on mainnet, while any other network takes 0), in place of the generic windows of the network. A single applicant wins when the join window closes; a second applicant moves the end to the join window plus the vote window. A late applicant is refused withDocumentContestNotJoinableErrornaming the target's join window. - Each application prefunds the votes with the moderation fund, 0.5 Dash
(
moderation_vote_resolution_fund_required_amount), instead of the contested document fund.
The target's declaration is read, and billed, when an application opens the contest and when a later one joins it; it is frozen at the target's creation, so both reads agree. Nothing at the end of a contest reads the target: the end date was written when the contest opened or was joined. A target that is missing or declares something else leaves a contest on the generic windows instead of failing, and the application's own reference validation refuses it. Every other contest, DPNS included, keeps the generic windows and fund.
Ties
From protocol version 14, a tie among the top contenders goes to the earliest contender: creation time, then block height, then core block height, then document id, among every tied contender. This holds for both resolutions; contests ending before version 14 awarded the latest contender.
Storage
A contest's state lives under votes / contested_resource / active_polls, laid out like the
contested index it decides: the contenders' documents, one votes sum tree per contender, and the
abstain and lock tallies. From protocol version 14 the tree below a contest's last index value,
which holds its contenders, stored result and tallies, is a count tree, so a join reads how many
contenders there are in one fetch; a contest started before 14 keeps its plain tree, and a join
counts its contenders by reading their keys. The masternodes' vote references live under
votes / contested_resource / identity_votes, and the end dates under votes / end_date_queries,
one tree per end date holding an entry for each contest ending then.
Once the contest ends, the winning document is awarded, the losers are removed, and the stored
result stays for the getContestedResourceVoteState query.
A block ends at most maximum_vote_polls_to_process contests (a drive-abci event_constants
value, two at protocol version 14), the earliest end date first, so contests due together may end
over several blocks. The cleanup removes each ended contest's
end-date entry, and removes an end date only once none of its contests remain under it; the rest
end in the next blocks. Before that read, a block removes any end date among the first due that
holds no contest, since each would take a place in the read and end nothing.
Identities
Before a user can do anything on Dash Platform -- register a name, send a contact request, create a data contract -- they need an identity. An identity is the platform-level representation of a user. It is the anchor for everything: documents are owned by identities, state transitions are signed by identity keys, and fees are paid from identity balances.
If you are coming from Ethereum, think of an identity as an account. But unlike Ethereum's single-key accounts, a Dash Platform identity can have multiple public keys with different purposes and security levels, making it more flexible and more secure.
The Identity Enum
Following the standard versioning pattern, Identity is defined in packages/rs-dpp/src/identity/identity.rs:
#![allow(unused)] fn main() { #[derive(Debug, Clone, PartialEq, From)] pub enum Identity { V0(IdentityV0), } }
Currently only V0 exists. The IdentityV0 struct lives in packages/rs-dpp/src/identity/v0/mod.rs:
#![allow(unused)] fn main() { pub struct IdentityV0 { pub id: Identifier, pub public_keys: BTreeMap<KeyID, IdentityPublicKey>, pub balance: u64, pub revision: Revision, } }
Four fields. That is it. An identity is remarkably simple:
-
id: A 32-byte unique identifier. For identities created via asset locks, this is derived from the locking transaction. For identities created via address-based funding, it is derived from the input addresses and nonces. -
public_keys: ABTreeMapmapping key IDs (simple integers) toIdentityPublicKeyobjects. Each public key has a purpose (authentication, encryption, decryption, transfer, voting, owner), a security level (master, critical, high, medium), and the actual key data. An identity can have many keys for different scenarios. -
balance: The identity's credit balance, measured in platform credits. Credits are the unit of account for fee payment on the platform. Users convert Dash into credits through a process called "topping up." -
revision: A monotonically increasing counter that increments with every identity update (adding keys, disabling keys, etc.). This prevents replay attacks -- each update must reference the current revision.
The Credit System
Platform credits are the fuel that powers everything on the network. Every state transition (creating a document, updating a contract, transferring tokens) costs credits. The credit system is how the platform measures and charges for computational and storage resources.
The balance field is a simple u64 representing the number of credits an identity holds. When an operation is performed, the fee system calculates the cost (as we will see in the Cost Tracking chapter) and deducts it from the identity's balance. If the balance is insufficient, the operation is rejected.
Public Keys and Key Types
An identity's public keys are not just random cryptographic keys -- they are structured with purpose and security level. Each IdentityPublicKey carries:
- Key ID (
KeyID): A numeric identifier unique within the identity, used to reference the key. - Purpose: What the key is for -- authentication, encryption, decryption, transfer, voting, or owner operations.
- Security Level: How sensitive operations signed by this key are -- master, critical, high, or medium.
- Key Type: The cryptographic algorithm -- ECDSA_SECP256K1, BLS12_381, ECDSA_HASH160, BIP13_SCRIPT_HASH, or EDDSA_25519_HASH160.
- Data: The actual public key bytes.
- Disabled At: An optional timestamp marking when the key was disabled.
This multi-key design means an identity can have a master key stored in cold storage, a critical key for important operations, and a high-level key for everyday use -- all belonging to the same identity. If a day-to-day key is compromised, the master key can disable it and add a replacement without losing the identity.
The PartialIdentity Pattern
Here is where the codebase reveals a practical optimization. Loading a full identity from storage is expensive -- you need to fetch the balance, all the keys, the revision, and potentially more. But most operations do not need all of that. A balance transfer only needs the balance. A document creation only needs to verify one key.
Enter PartialIdentity, defined alongside Identity in packages/rs-dpp/src/identity/identity.rs:
#![allow(unused)] fn main() { pub struct PartialIdentity { pub id: Identifier, pub loaded_public_keys: BTreeMap<KeyID, IdentityPublicKey>, pub balance: Option<Credits>, pub revision: Option<Revision>, pub not_found_public_keys: BTreeSet<KeyID>, } }
A PartialIdentity is exactly what it sounds like -- a partially-loaded identity. Notice the differences from Identity:
balanceisOption<Credits>rather than a bareu64. It might not have been loaded.revisionisOption<Revision>. Same story.loaded_public_keysmight only contain the specific keys that were requested.not_found_public_keystracks which keys were requested but did not exist on the identity.
You can convert a full Identity into a PartialIdentity:
#![allow(unused)] fn main() { impl IdentityV0 { pub fn into_partial_identity_info(self) -> PartialIdentity { let Self { id, public_keys, balance, revision, .. } = self; PartialIdentity { id, loaded_public_keys: public_keys, balance: Some(balance), revision: Some(revision), not_found_public_keys: Default::default(), } } pub fn into_partial_identity_info_no_balance(self) -> PartialIdentity { let Self { id, public_keys, revision, .. } = self; PartialIdentity { id, loaded_public_keys: public_keys, balance: None, // explicitly not loaded revision: Some(revision), not_found_public_keys: Default::default(), } } } }
The PartialIdentity pattern is used extensively in Drive's query and validation code. When processing a state transition, Drive fetches only the identity fields it actually needs, wraps them in a PartialIdentity, and passes that through the validation pipeline. This avoids unnecessary storage reads and keeps things efficient.
Identity Nonces
Replay protection on Dash Platform uses nonces rather than sequential transaction counters. There are two kinds:
- Identity nonce: A per-identity counter used for identity-level operations (like key updates).
- Identity-contract nonce: A per-identity-per-contract counter used for document operations. This allows operations on different contracts to be submitted in parallel without conflicting.
The nonce system is defined in packages/rs-dpp/src/identity/identity_nonce.rs and is more sophisticated than a simple incrementing counter. The nonce value is actually a packed u64 that contains both the counter value and a bitfield tracking recently-used nonces:
#![allow(unused)] fn main() { pub const IDENTITY_NONCE_VALUE_FILTER: u64 = 0xFFFFFFFFFF; pub const MISSING_IDENTITY_REVISIONS_FILTER: u64 = 0xFFFFFF0000000000; pub const MAX_MISSING_IDENTITY_REVISIONS: u64 = 24; }
The lower 40 bits hold the current nonce tip. The upper 24 bits form a bitfield that tracks which of the last 24 nonce values have been seen. This allows out-of-order submission within a window: if a user submits nonces 5, 7, and 6 in that order, all three are accepted. But nonce 5 cannot be submitted again because it is already marked in the bitfield.
The validation function checks several conditions:
#![allow(unused)] fn main() { pub fn validate_identity_nonce_update( existing_nonce: IdentityNonce, new_revision_nonce: IdentityNonce, identity_id: Identifier, ) -> SimpleConsensusValidationResult { let actual_existing_revision = existing_nonce & IDENTITY_NONCE_VALUE_FILTER; match actual_existing_revision.cmp(&new_revision_nonce) { std::cmp::Ordering::Equal => { // Nonce already used at the tip // -> NonceAlreadyPresentAtTip error } std::cmp::Ordering::Less => { // Nonce is in the future -- check it's within window // -> NonceTooFarInFuture if gap > 24 } std::cmp::Ordering::Greater => { // Nonce is in the past -- check bitfield // -> NonceTooFarInPast if gap > 24 // -> NonceAlreadyPresentInPast if bit is already set } } } }
This design balances several concerns:
- Replay protection: A nonce cannot be reused.
- Out-of-order tolerance: Within a 24-nonce window, transactions can arrive in any order.
- Bounded storage: Only 8 bytes are needed to track the full nonce state (the packed
u64). - Parallel submission: Identity-contract nonces let different contracts have independent nonce spaces.
Creating an Identity
The Identity enum provides versioned constructors:
#![allow(unused)] fn main() { impl Identity { pub fn new_with_id_and_keys( id: Identifier, public_keys: BTreeMap<KeyID, IdentityPublicKey>, platform_version: &PlatformVersion, ) -> Result<Identity, ProtocolError> { match platform_version .dpp .identity_versions .identity_structure_version { 0 => { let identity_v0 = IdentityV0 { id, public_keys, balance: 0, revision: 0, }; Ok(identity_v0.into()) } version => Err(ProtocolError::UnknownVersionMismatch { method: "Identity::new_with_id_and_keys".to_string(), known_versions: vec![0], received: version, }), } } } }
New identities start with a balance of 0 and a revision of 0. The balance is filled by the identity creation state transition (which includes an asset lock or address-based funding), and the revision increments from there.
How Identity Differs from Other Types
One important thing to note: the identity is not stored as a single blob in Drive. Unlike documents and data contracts (which are serialized and stored as items), identity fields are stored in separate locations within GroveDB's tree structure. The balance is in one place, each key is in another, the revision somewhere else. This is because different operations need to update different parts of the identity independently and atomically.
The Identity struct is primarily used for:
- Creating new identities (assembling all fields for the creation state transition)
- Client-side representation (what the SDK returns when you query an identity)
- Transport (serialized for gRPC responses)
Inside Drive and ABCI, you will more commonly see PartialIdentity or direct field access through Drive's identity methods.
Rules and Guidelines
Do:
- Use
PartialIdentitywhen you only need a subset of identity fields. It avoids unnecessary storage reads. - Validate nonces through the provided
validate_identity_nonce_updatefunction -- the bitfield logic is subtle. - Always go through
PlatformVersionwhen constructing identities to ensure the correct structure version.
Do not:
- Assume an identity has only one key. Identities commonly have multiple keys with different purposes and security levels.
- Manually pack or unpack nonce bitfields. Use the provided constants and validation functions.
- Store or cache full
Identityobjects when aPartialIdentitywould suffice. The full identity can be large if it has many keys. - Treat identity balance as Dash amounts. Credits are the unit of account on the platform; the conversion to/from Dash happens at the protocol level.
Key Budgets and Expiry
An identity rarely signs everything itself. A wallet hands a key to a game, a social client, a bot, and from then on that application signs state transitions on the identity's behalf and the identity's balance pays for them. Contract bounds already limit where such a key may act: one contract, one document type, or one contract group. Until protocol version 14 nothing limited how much it could spend or for how long. A key bound to one contract could still burn the whole identity balance on that contract, and it kept working until somebody remembered to disable it.
A budget and an expiry close that gap. They are two optional properties of an AUTHENTICATION key. A budgeted key can take at most so many credits from its identity; an expiring key stops signing at a given block time. Either one, or both, and both combine freely with contract bounds: bounds say where, limits say how much and for how long. When a limit is reached nothing has to be revoked. The key simply stops being accepted.
This chapter calls the two properties together limits. It covers the key format that carries them, the rule that decides what counts against a budget, where in the validation pipeline each check runs and why it runs there, and how Drive keeps the running total.
The Model
Five facts define a limited key:
- Limits are opt-in per key and live on a new key version.
IdentityPublicKey::V1is the version 0 key followed bytotal_budgetandexpires_at. Every key that existed before, and every key without limits registered after, is still a version 0 key with the same bytes as ever. - Only AUTHENTICATION keys below MASTER may carry them. The master key is what registers a replacement when a key runs out, so it must never run out itself. TRANSFER, ENCRYPTION and DECRYPTION keys cannot be limited.
- Limits are signed, and only ever loosened. They are part of the signable bytes of the transition that registers the key. The one transition that changes them,
IdentityKeyLimitsUpdate(see Raising Limits below), raises a budget or moves an expiry later; to give an application less, disable the key and register another. - A budget caps what leaves the identity, and only goes down. Fees, and credits the transition moves out (a document purchase, a prefunded voting balance), count against it. Storage refunds do not top it up. What is left is tracked by Drive next to the key, because the key itself never changes.
- An expiry is a block time.
expires_atis an absolute timestamp in milliseconds, the same unit and clock asdisabled_at. The key signs atexpires_at - 1and not atexpires_at.
A limited key moves through a small set of states, and only the first one can sign:
stateDiagram-v2
[*] --> Usable: registered
Usable --> Usable: signs, budget goes down
Usable --> Spent: budget reaches 0
Usable --> Expired: block time reaches expires_at
Usable --> Disabled: disabled by the master key
Spent --> Disabled
Expired --> Disabled
Spent and Expired are not written anywhere as a flag. Spent is the remaining budget in Drive being zero, and a spent key is refused at signature validation with PublicKeyBudgetExhaustedError (20015). Expired is a comparison against the block time, and an expired key is refused at fee validation with PublicKeyExpiredError (20016). Disabling works exactly as before and is independent of the limits: a master key can disable a limited key in any state.
The Version 1 Key
IdentityPublicKeyV1 lives in packages/rs-dpp/src/identity/identity_public_key/v1/mod.rs:
#![allow(unused)] fn main() { pub struct IdentityPublicKeyV1 { pub id: KeyID, pub purpose: Purpose, pub security_level: SecurityLevel, pub contract_bounds: Option<ContractBounds>, #[serde(rename = "type")] pub key_type: KeyType, pub read_only: bool, pub data: BinaryData, pub disabled_at: Option<TimestampMillis>, /// The total credits that state transitions signed with this key may take from the identity. pub total_budget: Option<Credits>, /// The block time, in milliseconds, from which the key can no longer sign. pub expires_at: Option<TimestampMillis>, } }
The field is called total_budget, not budget, because it is the fixed amount the identity granted and never changes. What is left of it is a different number, kept by Drive, and the code calls that one the remaining budget throughout.
The first eight fields are the version 0 fields in the same order, so the two encodings differ only by the variant byte and the two trailing options. Version 0 is every key from before protocol version 14 and every key without limits; version 1 is a key that may carry them:
flowchart TB
subgraph V0["IdentityPublicKey::V0"]
direction LR
a0["variant<br/><b>0</b>"] --- a1["id · purpose · security level · contract bounds<br/>type · read only · data · disabled at"]
end
subgraph V1["IdentityPublicKey::V1"]
direction LR
b0["variant<br/><b>1</b>"] --- b1["the same eight fields<br/>in the same order"] --- b9["<b>total budget</b><br/>optional credits"] --- b10["<b>expires at</b><br/>optional milliseconds"]
end
V0 ~~~ V1
style a0 fill:#2d3748,color:#e2e8f0
style b0 fill:#2d3748,color:#e2e8f0
style b9 fill:#c05621,color:#fff
style b10 fill:#c05621,color:#fff
IdentityPublicKeyInCreationV1 mirrors it in public_key_in_creation/v1/mod.rs, with total_budget and expires_at placed before the signature. Only the signature is excluded from the signable bytes, so the identity signs the limits it grants and a test pins it: changing a budget changes the signable bytes, changing the key's own signature does not.
In JSON the key is tagged "$formatVersion": "1" and the two fields appear as totalBudget and expiresAt. Like disabledAt, they are left out when absent.
Reading and Building
Call sites never match on the variant. IdentityPublicKeyGettersV1 (accessors/v1/mod.rs) is implemented on the enum and on both structs, and a version 0 key answers None:
#![allow(unused)] fn main() { pub trait IdentityPublicKeyGettersV1 { fn total_budget(&self) -> Option<Credits>; fn expires_at(&self) -> Option<TimestampMillis>; /// The expiry instant itself is already expired. fn is_expired_at(&self, time_ms: TimestampMillis) -> bool { /* ... */ } fn has_limits(&self) -> bool { /* ... */ } } }
To build one, take any key and call with_limits. A version 0 key becomes a version 1 key and every other field is kept:
#![allow(unused)] fn main() { let app_key = key.with_limits( Some(dash_to_credits!(0.1)), // total budget Some(now_ms + 30 * 24 * 3_600_000), // expires in 30 days ); }
The conversions between IdentityPublicKey and IdentityPublicKeyInCreation carry the limits in both directions, so the existing identity update builders register a limited key without knowing about limits. The one conversion that would lose them is building an IdentityPublicKeyInCreationV0 directly from a key; go through the enum instead.
The Protocol Version Gate
A binary from before this change cannot decode a version 1 key, so it rejects a transition carrying one while decoding. A new binary running protocol version 13 has to do the same, or the two would disagree during an upgrade window. StateTransition::active_version_range is where that is enforced: a transition carrying a version 1 key in creation is active from 14.
#![allow(unused)] fn main() { fn active_version_range_for_keys_in_creation( keys: &[IdentityPublicKeyInCreation], otherwise: RangeInclusive<ProtocolVersion>, ) -> RangeInclusive<ProtocolVersion> { if IdentityPublicKeyInCreation::first_bound_to_a_contract_group(keys).is_some() || IdentityPublicKeyInCreation::first_in_version_1_format(keys).is_some() { 14..=LATEST_VERSION } else { otherwise } } }
The same function gates keys bound to a contract group. The predicates live on the key type (public_key_in_creation/mod.rs) so that every place that asks the question asks it the same way: first_in_version_1_format here, and first_with_limits for the shielded pool rule below, next to first_bound_to_a_contract_group. The gate is on the variant, not on whether limits are present: an old binary fails on the variant either way. It must also be this gate and not a structure rule, because identity create validates key structure in a paid stage. Refusing there would charge the asset lock on a new binary while an old one fails to decode, which is a state divergence.
Registering a Limited Key
A version 1 key arrives the way any key does, in the key list of a transition that registers keys: IdentityCreate, IdentityUpdate, IdentityCreateFromAddresses or IdentityCreateFromShieldedPool. Five things can stop it, in this order:
flowchart TD
T["a transition registers<br/>a version 1 key"] --> PV{"protocol version<br/>at least 14?"}
PV -->|no| NA["not active<br/>rejected while decoding"]
PV -->|yes| SH{"has limits and is created<br/>from the shielded pool?"}
SH -->|yes| E38["<b>10538</b><br/>refused"]
SH -->|no| ST{"AUTHENTICATION and<br/>below MASTER?"}
ST -->|no| E36["<b>10536</b><br/>limits not allowed"]
ST -->|yes| BZ{"budget present<br/>and zero?"}
BZ -->|yes| E37["<b>10537</b><br/>zero budget"]
BZ -->|no| EX{"expires_at after<br/>the block time?"}
EX -->|no| E19["<b>40219</b> already expired<br/>paid failure"]
EX -->|yes| OK["key stored as version 1<br/>remaining budget = budget"]
style NA fill:#c53030,color:#fff
style E38 fill:#c53030,color:#fff
style E36 fill:#c53030,color:#fff
style E37 fill:#c53030,color:#fff
style E19 fill:#c05621,color:#fff
style OK fill:#276749,color:#fff
The checks sit in the tier their inputs dictate (see Where a check belongs):
| Rule | Needs | Where | Error |
|---|---|---|---|
| Only AUTHENTICATION below MASTER | the key | rs-dpp validate_identity_public_keys_structure v1 | IdentityPublicKeyLimitsNotAllowedError 10536 |
| Budget is not zero | the key | same | InvalidIdentityPublicKeyBudgetError 10537 |
| Expiry is after the block time | the block time | drive-abci validate_identity_public_keys_limits, called from the four identity create and update state/v1 validators | IdentityPublicKeyAlreadyExpiredError 40219 |
| No limits in a shielded identity creation | the transition | drive-abci validate_shielded_proof v1, and the transition builder | IdentityPublicKeyLimitsNotAllowedInShieldedIdentityCreationError 10538 |
The expiry rule exists because a key that is dead on arrival is not harmless. It still uses up its key id and, for a unique key type, registers its public key hash, which can then never be used on any identity again. The usual way to get there is passing seconds where milliseconds are expected, and the check turns that mistake into an error instead of a burnt key.
The Shielded Pool Exception
IdentityCreateFromShieldedPool has no identity signature. Its authorization is the Orchard proof, the spend authorization signatures, and a binding signature over a sighash. The new identity's keys are committed into that sighash by a hand-written preimage, identity_create_from_shielded_extra_sighash_data_v0, which lists the key fields it binds one by one: id, purpose, security level, type, data, read_only, contract bounds. That layout is frozen, and it predates the version 1 key. A budget or an expiry would simply not be in it.
How exposed would the limits be? Each ECDSA or BLS key carries a proof of possession, and that signature is over the signable bytes of the whole transition, which include every key with its variant and its limits. One such key is therefore enough to pin the limits of all of them. Hash based key types must carry an empty signature, so a transition whose keys are all hash based has nothing signing the limits. That is not an exotic case: hash based keys are the privacy-minded choice, and privacy is why one creates an identity from the shielded pool. For such a transition a relay or a proposer could strip a budget, or move an expiry, and everything would still verify. A test pins the cause: the preimage of a key with limits and of the same key without is byte for byte identical.
The preimage stays frozen and the key is refused instead, the same decision taken for keys bound to a contract group. validate_shielded_proof v1 refuses a key that carries a budget or an expiry before the preimage is built, and the transition builder refuses it before a proof is generated. Add the key with an identity update once the identity exists.
The refusal is about the limits, not about the format. A version 1 key without limits is accepted, because everything it holds is in the layout: it binds the same preimage bytes as the version 0 key with the same fields, and a second test pins that too. This keeps shielded identity creation working if version 1 ever becomes the default key format. What it leaves possible, and only when every key is hash based, is a relay re-encoding a limit-less key from one version to the other. The two are the same key, so nothing the identity granted changes; the visible effects are two more stored bytes and a different transition hash.
The general lesson.
PlatformSignablecovers a new key field automatically. The shielded preimage does not. Any future field on the key must be checked againstpackages/rs-dpp/src/shielded/sighash.rsas well.
Raising Limits
A spent key, or one about to expire, does not have to be replaced. IdentityKeyLimitsUpdate (state transition type 23) raises the limits of one key of the identity:
#![allow(unused)] fn main() { pub struct IdentityKeyLimitsUpdateTransitionV0 { pub identity_id: Identifier, pub nonce: IdentityNonce, pub key_id: KeyID, pub total_budget: Option<Credits>, // the new total, None to leave it pub expires_at: Option<TimestampMillis>, // the new expiry, None to leave it pub user_fee_increase: UserFeeIncrease, pub signature_public_key_id: KeyID, pub signature: BinaryData, } }
An update only ever loosens. A budget can grow, an expiry can move later, and that is all: a limit cannot be lowered, and a key that has no budget or no expiry cannot be given one. Tightening is what disabling is for. This one rule keeps the transition small, and it is what makes the arithmetic safe: the remaining budget never exceeds the total, both grow by the same amount, so the addition cannot overflow.
Both fields carry the new absolute value, not an amount to add; "top up by X" is SDK sugar that adds X to the total the client's copy of the key shows. What the wire carries is then exactly what a proof of the execution shows, and the proof binds it. No identity revision is claimed and none is bumped: an identity update needs one because clients allocate key ids, while this transition names an existing key and allocates nothing, so a stale copy of the identity cannot make it collide and the client only needs to hold the signing key. Two clients topping up the same key from the same stale copy find the second raise counted from the first's result, and a total that no longer raises the stored one is refused.
Who may sign. A MASTER key, or a CRITICAL authentication key that carries no limits itself and no contract bounds (a bound key may only sign batches, ContractBoundedKeyNonBatchError 20013). A key with limits can never raise limits, so it can never top itself up; the identity signature validation refuses it before the remaining budget is even read.
flowchart TD
S["signer is MASTER,<br/>or CRITICAL without limits<br/>and without contract bounds"] --> K{"key exists<br/>and is enabled?"}
K -->|no| E2["<b>40209</b> / <b>40208</b><br/>paid"]
K -->|yes| L{"has each limit<br/>being raised?"}
L -->|no| E3["<b>40220</b><br/>paid"]
L -->|yes| G{"each new value<br/>greater?"}
G -->|no| E4["<b>40221</b><br/>paid"]
G -->|yes| X{"not expired<br/>afterwards?"}
X -->|no| E5["<b>40219</b><br/>paid"]
X -->|yes| OK["key rewritten,<br/>remaining raised"]
The last check is the one with a twist. An expired key may be revived by moving its expiry past the block time, but it cannot be topped up while it stays expired: after the update the key must be usable, or the update was pointless. The mempool asks the same questions of the key when it admits the transition, against the last block's time, so a bad target key is answered with these codes at admission; a top-up that arrives just before the key's expiry can still be admitted and then refused, paid, in the block where it lands. The two structural rules run unpaid before any of this: the transition must set at least one of the two fields (IdentityKeyLimitsUpdateEmptyError 10539), and a budget it sets is not zero (10537). A limited signer is refused unpaid with PublicKeyWithLimitsCannotUpdateKeyLimitsError 20017.
What Drive does. The key is read once, when the transition is validated, and carried in the action as it is stored: update_identity_key_limits sets the new limits on that copy and rewrites it in place with replace_key_in_storage_operations, the same patch a disable uses; the rewrite is priced by the byte delta, since a bigger total is a bigger varint and an expiry that was absent is new bytes. The references to the key in the purpose and security level trees carry its value hash, so they are refreshed as a disable refreshes them. Then add_to_identity_key_budget raises the remaining budget by the difference between the new total and the old one: a same-size replace of the eight byte entry. The nonce is updated first.
The proof. The proof is the rewritten key, nothing more. The verifier requires the key present and holding exactly the total budget and the expiry the transition asked for. That authenticates the state the update aimed at, not this exact transition: the nonce and the fee increase are signed but not stored, so any later state of the key with those limits would produce the same proof. The outcome is therefore classified as affected state, like a credit transfer, and the SDKs wait for it with the affected-state wait.
In the SDKs: Identity::update_key_limits, top_up_key_budget and extend_key_expiry (Rust, UpdateIdentityKeyLimits), identityUpdateKeyLimits({ identity, keyId, addBudget, expiresAt, signer }) (wasm-sdk), sdk.identities.updateKeyLimits (js-evo-sdk). All resolve to the key as stored after the update. A limited key is registered the ordinary way: an IdentityPublicKeyInCreation built with totalBudget or expiresAt (wasm-dpp2) passed to identityUpdate or sdk.identities.update, or an IdentityPublicKey::with_limits(..) key passed to the Rust identity update builder.
In the wallet and on mobile: IdentityWallet::update_identity_key_limits_with_external_signer (platform-wallet) raises the limits and lays the key as stored over the cached identity, so the client's key row follows through the persister; the FFI exposes it as platform_wallet_update_identity_key_limits_with_signer and the query as dash_sdk_identity_fetch_keys_remaining_budgets. Every key row that crosses the FFI (registration, update, the persisted key entry, the cold-restore row and the managed identity's key snapshot) carries total_budget and expires_at, so a limited key persists and restores as limited. Kotlin: IdentityUpdates.updateKeyLimits and Identities.fetchKeysRemainingBudgets, with IdentityPubkey.totalBudget / expiresAt on the rows it registers (Room schema 12). Swift: ManagedPlatformWallet.updateIdentityKeyLimits(identityId:keyId:addBudget:expiresAt:signer:) and SDK.fetchKeysRemainingBudgets(identityId:keyIds:), with the two limits on IdentityPublicKey, IdentityPublicKeyInfo, the IdentityPubkey row and PersistentPublicKey (SwiftData schema V5). The wallet's own signers prefer a key without limits and skip an expired one; what is left of a budget is only known through the query, so a spent key is refused by Platform.
The Budget Rule
A budget has one subtlety, and it is the same one identity balances have. Most of what a transition costs is known before it runs, but the metered processing fee is only known afterwards. A rule that demanded the whole fee fit up front would have to use the worst-case estimate, which can be fifty times the real processing cost, and would strand the tail of every budget. A rule that checked nothing up front would let a key spend without limit.
So the cost is split. Everything that is known, chosen or priced before the transition runs must fit in what is left. Only the metered processing fee may take the key over:
flowchart TD
M["<b>Must fit in what is left of the budget</b><br/><br/>credits moved out of the identity<br/><i>a purchase price,<br/>a prefunded voting balance</i><br/><br/>storage fee<br/><i>estimated</i><br/><br/>fees priced up front<br/><i>a contract registration</i><br/><br/>user fee increase<br/><i>the processing the signer<br/>chose to add</i>"]
M --> C{"required ≤ remaining?"}
C -->|no| X["<b>40218</b> refused<br/>unpaid, nothing changes"]
C -->|yes| RUN["the transition runs"]
P["<b>May take the key over its budget</b><br/><br/>metered processing fee<br/><i>known only after the run</i>"] --> RUN
RUN --> D["remaining = max(0, remaining − spent)"]
style M fill:#1a365d,color:#e2e8f0
style X fill:#c53030,color:#fff
style RUN fill:#276749,color:#fff
style P fill:#c05621,color:#fff
In code this is required_from_key_budget in validate_fees_of_event/v1/mod.rs:
#![allow(unused)] fn main() { removed_balance.unwrap_or_default() .saturating_add(storage_fee) .saturating_add(additional_fixed_fee_cost.unwrap_or_default()) .saturating_add(user_fee_increase_amount) }
Each term is there for a reason:
removed_balance: without it a budgeted key could drain the identity through document purchases, where the credits go to the seller and never show up as a fee.- Storage fee: the part of a fee that pays for permanent state. It is the part identity balances also treat as mandatory.
additional_fixed_fee_cost: technically a processing fee, but priced up front and large. A contract registration costs a tenth of a Dash. Letting it overshoot would make a budget of a thousand credits meaningless.- User fee increase: also processing, but it is the signer's choice, up to 65,535 percent. A key that is about to run out must not be able to burn several hundred times a processing fee on its way out.
What remains, the metered base processing fee, is bounded by what one transition can do. That is the "slightly over".
A Worked Example
Take a transition whose storage fee is 50,000,000 credits and whose metered processing fee turns out to be 2,000,000, with no user fee increase and nothing moved out of the identity. Required is 50,000,000.
| Remaining before | Outcome | Identity pays | Remaining after |
|---|---|---|---|
| 60,000,000 | runs | 52,000,000 | 8,000,000 |
| 50,000,001 | runs, processing overshoots | 52,000,000 | 0 |
| 49,999,999 | refused with 40218, unpaid | 0 | 49,999,999 |
| 0 | refused with 20015 at signature validation | 0 | 0 |
The second row is the rule in one line. The storage fits with a credit to spare, so the transition runs. The identity pays the full 52,000,000, which is 1,999,999 more than the key had left, and the remaining budget stops at zero. The key is now spent, and the fourth row is what happens to its next transition, whatever that transition costs. should_let_only_metered_processing_take_a_key_over_its_budget pins rows two to four against real fees.
What Counts as Spent
After the transition runs, execute_event v1 deducts:
spent = removed_balance + the fee the identity owes, net of its own storage refunds
The second term is desired_removed_balance from FeeResult::into_balance_change, the same number the identity balance is charged. Three consequences:
- A deletion whose refund exceeds its fee costs the budget nothing, and a refund never adds to it. A budget only goes down.
- A failed transition that is still paid for (a penalty and a nonce bump) spends from the budget like a successful one. It goes through the same event.
- The deduction saturates at zero. It never fails and never goes negative.
This mirrors the identity balance so closely on purpose. BalanceChange::RemoveFromBalance already carries a required_removed_balance (storage) and a desired_removed_balance (storage plus processing), and an identity that can cover the first but not the second goes into processing debt. A key budget is the same rule with the debt replaced by "and then the key is done".
Where the Checks Run
Three stages enforce limits at signing time. Which check goes where follows from what each stage knows:
flowchart TD
A["is allowed"] --> B["<b>identity signature</b> v1<br/>has Drive and the signing key<br/>has no block time, no fee"]
B --> C["nonce, basic structure,<br/>balance pre-check"]
C --> D["advanced structure,<br/>transform into action,<br/>state validation"]
D --> E["ExecutionEvent::Paid<br/>carries SigningKeyLimits"]
E --> F["<b>validate_fees_of_event</b> v1<br/>has the block time<br/>and the estimated fee"]
F --> G["<b>execute_event</b> v1<br/>has the actual fee"]
G --> H["operations applied<br/>identity charged<br/>key budget deducted"]
B -.->|"remaining budget is 0"| R1["<b>20015</b> budget exhausted<br/>unpaid"]
F -.->|"block time ≥ expires_at"| R2["<b>20016</b> key expired<br/>unpaid"]
F -.->|"required > remaining"| R3["<b>40218</b> budget exceeded<br/>unpaid"]
style B fill:#744210,color:#fff
style F fill:#744210,color:#fff
style G fill:#744210,color:#fff
style R1 fill:#c53030,color:#fff
style R2 fill:#c53030,color:#fff
style R3 fill:#c53030,color:#fff
style H fill:#276749,color:#fff
The diagram shows the order in which the checks happen. In code the last two boxes are nested: during block execution execute_event calls validate_fees_of_event and then applies, while check tx calls validate_fees_of_event on its own and never executes.
A spent key is refused first, with the signature. Signature validation already reads the identity from Drive, so reading what is left of the budget there costs one more lookup, and an application that keeps trying with a spent key is turned away before any real work is done. This is the role the minimum balance pre-check plays for an empty identity. The read is billed as one extra key retrieval.
Expiry waits for fee validation. The natural place for it would be next to the disabled_at check, but signature validation is not given the block time, and the processor that calls it is a shipped v0. Threading a parameter through a shipped generation is exactly what the versioning rules forbid, and a new generation of the processor and of check tx would copy some fourteen hundred lines to pass one integer. validate_fees_of_event is the first stage that has the block time, runs for every paid transition, and runs in check tx as well as in block execution. The cost of checking late is that an expired key's transition is fully validated before it is refused. An insufficient balance has always had that same profile.
The budget comparison needs the estimated fee, so it can only be in fee validation. The deduction needs the actual fee, so it can only be in execution.
Carrying the Limits
The processor, check tx and the unversioned create_from_state_transition_action are all untouched. What connects the signature stage to the fee stage is the StateTransitionExecutionContext that already travels with every transition:
sequenceDiagram
autonumber
participant P as Processor v0
participant S as Signature v1
participant D as Drive
participant E as execute_event v1
P->>S: validate the signature
S->>D: balance and signing key
S->>D: remaining budget
D-->>S: remaining
alt remaining is 0
S-->>P: 20015, unpaid
else something is left
Note over S: limits recorded in the<br/>execution context
S-->>P: valid
end
Note over P: nonce, structure,<br/>transform, state
Note over P: limits copied from the context<br/>onto ExecutionEvent Paid
P->>E: execute_event
Note over E: validate_fees_of_event v1<br/>expiry, then budget
E->>D: apply the operations
E->>D: charge the identity
E->>D: deduct from the key budget
SigningKeyLimits (execution/types/signing_key_limits.rs) is three fields: the key id, expires_at, and the remaining budget as read at signature validation. It is None on the event for every transition signed by an ordinary key, and both v1 methods start by checking for it and handing everything else to their v0:
#![allow(unused)] fn main() { let ExecutionEvent::Paid { signing_key_limits: Some(signing_key_limits), .. } = event else { return self.validate_fees_of_event_v0(event, block_info, transaction, platform_version, previous_fee_versions); }; }
An ordinary key therefore costs nothing new: no read, no write, no branch past that first line. execute_event v1 is narrower still. It only takes over for a budgeted signing key; a key that merely expires is executed by v0, which calls the fee validation dispatcher and so still reaches the expiry check.
The remaining budget read at signature time is safe to reuse at fee time because nothing can change it in between. State transitions in a block are processed one after another, each with its own signature validation, and the deduction itself re-reads the value inside the block transaction.
Check Tx
| First time check | Recheck | Block execution | |
|---|---|---|---|
| Signature validation, spent key refused | yes | skipped | yes |
| Expiry | against the last committed block time | not checked | against the block's own time |
| Budget against the estimated fee | yes | not checked | yes |
| Deduction | never, check tx does not write | never | yes |
Recheck skips signature validation for every transition, so no key is loaded and no limits reach the event. A transition whose key expires while it waits in the mempool is therefore not evicted by a recheck. The proposer drops it at prepare_proposal, which is what happens to any transition that turned unpayable while it waited.
Refusals Are Not Charged
All three signing-time refusals return an unpaid result, like IdentityInsufficientBalanceError. Check tx rejects the transition, a proposer removes it from the block, and a block that contains one is rejected by validators. Nothing about the identity changes: not the balance, not the budget, not the nonce.
One case deserves a sentence. An invalid transition is normally a paid failure: the identity is charged a penalty and its nonce is bumped. If the key that signed it has expired or cannot cover the penalty, there is nobody to charge through that key, and the transition takes the path that already exists for an invalid transition from an identity that cannot afford the penalty. It is reported as an internal error and dropped from the block. A key that may not spend is never used to charge a penalty either.
Storage
The key is immutable, so the running total cannot live in it. It lives in a new subtree of the identity:
flowchart LR
I["Identities"] --> ID["identity id<br/><i>32 bytes</i>"]
ID --> CI["IdentityContractInfo <b>32</b>"]
ID --> N["IdentityTreeNonce <b>64</b>"]
ID --> NC["IdentityTreeNegativeCredit <b>96</b>"]
ID --> K["IdentityTreeKeys <b>128</b><br/>key id → serialized key<br/><i>immutable: holds total_budget and expires_at</i>"]
ID --> KR["IdentityTreeKeyReferences <b>160</b>"]
ID --> REV["IdentityTreeRevision <b>192</b>"]
ID --> KB["IdentityTreeKeyBudgets <b>224</b><br/><i>created with the first budgeted key</i>"]
KB --> E1["key id 5 → 00 00 00 00 02 FA F0 80<br/><i>remaining credits, 8 bytes big endian</i>"]
KB --> E2["key id 9 → ..."]
style KB fill:#c05621,color:#fff
style E1 fill:#744210,color:#fff
style E2 fill:#744210,color:#fff
Three properties of this layout matter.
It is created lazily. New identities do not get the subtree, and identities from before protocol version 14 do not have it. insert_identity_key_budget_operations creates it with batch_insert_empty_tree_if_not_exists_check_existing_operations when the first budgeted key arrives, checking the pending operations as well as the state, because two budgeted keys can arrive in one transition. A key that only expires has no entry: there is nothing to count.
The value is fixed width. Eight big endian bytes, never a varint. Deducting from a budget then replaces the value without changing what is stored, so it produces no storage fee and no refund. That is what allows the deduction to be applied outside of the transition's fee, exactly like the balance write it follows. The identity's negative credit item uses the same trick for the same reason.
It is written once with the fee and then maintained without one. Whoever registers the key pays for the entry, and for the subtree the first time. insert_new_unique_key and insert_new_non_unique_key v1 add that write next to the key; keys cannot carry a budget before protocol version 14, so both v0s never do. From then on each deduction is an unbilled replace.
The methods live in packages/rs-drive/src/drive/identity/key/budget/:
| Method | Does |
|---|---|
insert_identity_key_budget_operations | writes the full budget as the remaining budget of a new key, creating the subtree if needed |
fetch_identity_key_remaining_budget | reads what is left; None for a key without a budget, including when the subtree does not exist |
deduct_from_identity_key_budget | subtracts, stopping at zero, and applies; errors if the key has no entry |
add_estimation_costs_for_key_budgets | the layer information for the subtree in estimation mode |
Reading What Is Left
A spent key is refused, so a client wants to know where a budget stands before it signs. The getIdentityKeysRemainingBudgets query answers for several keys of one identity at once:
request: identity_id, key_ids [3, 4, 5], prove
response: 3 -> 250000000 a budgeted key
4 -> 0 a budgeted key that is spent
5 -> (nothing) no budget, or no such key
Every requested key id is answered. A key without a budget and a key that does not exist look the same here, because the query reads the budgets subtree and nothing else; getIdentityKeys tells them apart. The request names at least one key, none twice, and at most max_returned_elements of them.
With prove, the answer is a GroveDB proof of the path query [Identities, identity_id, IdentityTreeKeyBudgets] with one query item per key id, limited to their count. The interesting part is that "no budget" is provable in all three shapes state can have:
flowchart LR
Q["prove the budgets<br/>of keys 3, 4, 5"] --> S{"budgets subtree<br/>exists?"}
S -->|no| A["subtree proved absent<br/>every key is <b>None</b>"]
S -->|yes| E{"entry for<br/>the key?"}
E -->|yes| V["8 byte value proved<br/><b>Some(remaining)</b>"]
E -->|no| N["entry proved absent<br/><b>None</b>"]
The verifier, Drive::verify_identity_keys_remaining_budgets, rebuilds the same path query from the request and uses verify_query_with_absence_proof, so it returns one entry per requested key and never an entry that was not asked for. packages/rs-drive/src/drive/identity/key/budget/mod.rs pins this with a test that forges a value inside a proof and expects it to no longer verify against the state root.
The same call is available at every layer:
| Layer | Call |
|---|---|
| Drive | fetch_identity_keys_remaining_budgets, prove_identity_keys_remaining_budgets, verify_identity_keys_remaining_budgets |
| Rust SDK | IdentityKeysRemainingBudgets::fetch(&sdk, IdentityKeysRemainingBudgetsQuery { identity_id, key_ids }), also fetch_unproved |
| wasm-sdk | getIdentityKeysRemainingBudgets(identityId, keyIds), ...WithProofInfo, returning Map<number, bigint | null> |
| js-evo-sdk | sdk.identities.keysRemainingBudgets(identityId, keyIds), keysRemainingBudgetsWithProof |
What the query returns is the state as of the last committed block. A transition already in the mempool may spend from the budget before yours runs, so treat the number as an upper bound, not a reservation.
Versioning Touchpoints
Everything is gated to protocol version 14. Tables that protocol version 14 already owned (it was unreleased at the time) were amended in place; one new table was needed.
| Table | Slot | Change |
|---|---|---|
STATE_TRANSITION_METHOD_VERSIONS_V2 (new) | validate_identity_public_keys_structure | 0 → 1 |
DRIVE_ABCI_VALIDATION_VERSIONS_V10 | validate_identity_public_keys_limits | new, None → Some(0) |
validate_state_transition_identity_signed | stays 1, extended in place | |
validate_shielded_proof | stays 1, extended in place | |
DRIVE_ABCI_METHOD_VERSIONS_V10 | validate_fees_of_event | 0 → 1 |
execute_event | 0 → 1 | |
DRIVE_IDENTITY_METHOD_VERSIONS_V2 | keys.insert.insert_new_unique_key, insert_new_non_unique_key | 0 → 1 |
keys.budget.* (six slots, two of them for the query) | new, None → Some(0) | |
DRIVE_ABCI_QUERY_VERSIONS_V0 and _V1 | identity_based_queries.keys_remaining_budgets | new slot at 0; the Drive methods behind it are None before 14, which is what refuses the query there |
DRIVE_VERIFY_METHOD_VERSIONS_V1 | identity.verify_identity_keys_remaining_budgets | new, 0 (verification is client side and not gated) |
DRIVE_ABCI_VALIDATION_VERSIONS_V10 | identity_key_limits_update_state_transition | new slot; every gate None before V10, on from V10 (the transition is gated by is_allowed and active_version_range to 14 as well) |
DRIVE_IDENTITY_METHOD_VERSIONS_V2 | update.update_identity_key_limits, keys.budget.add_to_identity_key_budget | new, None → Some(0) |
DRIVE_STATE_TRANSITION_METHOD_VERSIONS_V1 to _V4 | convert_to_high_level_operations.identity_key_limits_update_transition | new, 0 |
STATE_TRANSITION_SERIALIZATION_VERSIONS_V1 to _V3 | identity_key_limits_update_state_transition | new, 0 |
Two of these are worth a second look. Identity signature validation v1 and shielded proof validation v1 were introduced for protocol version 14 by the contract bounds work and had not shipped yet, so they were extended in place rather than given a v2. validate_fees_of_event and execute_event had shipped, so they got new generations, and those generations delegate to v0 for every event they do not handle instead of copying it.
Fees
Registering a budgeted key costs slightly more than registering an ordinary one: the eight byte entry, and the empty subtree the first time. Both are ordinary metered storage charged to the transition that adds the key. A key that only expires costs what a version 0 key costs plus the few bytes the two fields add to the stored key.
Signing with a budgeted key adds one fixed charge, the key retrieval that reads what is left (fetch_single_identity_key_processing_cost). The deduction write is not billed, like the balance write. Signing with a key that only expires adds nothing.
What Is Not There Yet
This change is the consensus core. Known gaps, all deliberate:
- SDK surfaces. Rust callers can use
with_limitstoday. The wasm, JavaScript, Swift and Kotlin bindings do not expose the two fields for creation, and SDK key selection does not skip an expired or spent key before signing, although the query above gives it what it needs. Swift and Kotlin do not expose that query yet. - Lowering limits. An update only raises a budget or extends an expiry (above). To take a key back, disable it.
- Limits on other purposes. A TRANSFER key with a budget would cap transfers and withdrawals. The rule for it would differ (the amount moved is the point, not a side effect), so it was left out rather than half done.
- Limits in a shielded identity creation. A key with a budget or an expiry is refused there, as explained above. Lifting that means a new generation of the sighash preimage.
Tests
- dpp (
identity_public_key/v1,public_key_in_creation/v1,validate_identity_public_keys_structure/v1,state_transition/mod.rs): the version 1 round trip; the version 0 encoding unchanged byte for byte; the JSON shape; the expiry boundary; limits surviving the conversions and being signed over; the structure rules; the protocol version 13 and 14 sides of the decode gate; frozen error discriminants. - drive (
drive/identity/key/budget): the budget written on key add and on identity create; two budgeted keys in one batch; the deduction stopping at zero and never changing storage; estimated at least actual; protocol version 13 untouched. - drive-abci (
batch/tests/key_limits.rs), end to end throughprocess_raw_state_transitionsandcheck_tx: the exact deduction; the rule pinned one credit either side of the storage fee; the user fee increase counted up front; the expiry boundary; both limits on one key; a paid failure spending from the budget; an invalid transition through an unusable key not being charged; mempool admission. - drive-abci (
identity_key_limits_update/tests.rs), end to end: a spent key topped up and admitted again; an expired key revived by an extension and refused while it stays expired; each refusal pinned to its code, paid ones bumping the nonce; MASTER and unlimited CRITICAL signers accepted, a limited or HIGH one refused; the mempool; the execution proof; protocol version 13. drive (update_identity_key_limits): the total and the remaining budget growing by the same amount; the expiry alone; the estimate covering the rewrite; the whole database consistent after the reference refresh; the addition to the remaining budget; protocol version 13. - drive-abci (
identity_updatekey_limits,identity_create_from_shielded_pool/tests.rs): registration storing a version 1 key with its whole budget left; an already expired key as a paid failure; limits on TRANSFER and MASTER keys and a zero budget as unpaid; the shielded creation refusing a key with either limit and accepting a version 1 key without any. The builder has the same pair of cases inrs-dpp(shielded/builder/identity_create_from_shielded_pool.rs).
cargo test -p dpp --all-features --lib -- identity_public_key public_key_in_creation should_only_admit
cargo test -p drive --lib -- identity::key::budget
cargo test -p drive-abci --lib -- key_limits validate_identity_public_keys_limits validate_fees_of_event
cargo test -p drive-abci --lib -- should_refuse_a_key_with_limits
cargo test -p dpp --all-features --lib -- identity_key_limits_update
cargo test -p drive --lib -- update_identity_key_limits
cargo test -p drive-abci --lib -- identity_key_limits_update
Rules and Guidelines
Do:
- Read limits through
IdentityPublicKeyGettersV1and treatNoneas the normal case. Never match onV0versusV1to find out. - Build limited keys with
with_limits, and convert between a key and a key in creation through the enums so the limits follow. - Give
expires_atin milliseconds of block time. If an SDK offers a relative TTL, resolve it to an absolute time before signing, since the absolute time is what gets signed. - Keep every new cost term on the correct side of the budget rule. If a transition gains a cost that is chosen by the signer or priced up front, it belongs in
required_from_key_budget. - Check any new field on the public key against the shielded sighash preimage, not only against
PlatformSignable.
Do not:
- Add a field to
IdentityPublicKeyV0orIdentityPublicKeyInCreationV0. Every key in state is one, and there is no version byte to tell old bytes from new. - Store the remaining budget in the key, or as a varint. The key is immutable, and a value that changes width changes storage, which would make the deduction a billed operation with a refund.
- Credit a budget. Refunds go to the identity balance; a budget only goes down.
- Move the expiry check into signature validation by threading the block time through the shipped processor. If that stage ever needs the time, it needs a new generation of what calls it.
- Charge a penalty through a key that failed its limits. A refusal for a spent, exceeded or expired key is unpaid, whatever else is wrong with the transition.
Token Shielded Pools
From protocol version 14 a token can own a shielded pool: an Orchard pool that holds that token instead of credits. Holders move tokens between their identity balance and the pool with token transitions inside a batch, issuers mint, burn, release and sell straight into or out of it, document costs can be paid out of it, and three identity-less transitions move tokens with the fee paid from the credit shielded pool, so no identity appears at all. This chapter describes the storage, the configuration flag, the transitions, the validation rules, the block end bookkeeping, the queries and the client builders.
Why one pool per token
The Orchard construction Platform uses for credits has no asset base: a note carries a value but not an asset id, and the value balance the circuit proves is a single number. Mixing tokens in one pool would let a spend of token A create a note of token B. A token therefore gets its own pool, and each pool is a copy of the credit pool's layout rooted under the token.
One pool per token is what makes the arrangement safe, and it is also what it costs. A shielded
pool hides a spend among the other notes in the same pool, so splitting the pools splits that
crowd with them: a token's anonymity set is its own holders, not everyone on Platform who uses a
shielded pool. A pool that holds one note hides nothing — the spend of that note names the shield
that created it. A token's minimumPoolNotesForOutgoing is absent unless its issuer sets one, and
absent reads as no threshold, so a new pool is spendable from its first note: that is where it is
weakest rather than where it is strongest.
So the flag isolates a token's balances; on its own it does not make that token's transfers anonymous. An issuer turning it on is choosing a pool whose privacy grows with its use, and can require a floor of notes before tokens may leave it — see Configuration for the threshold and Validation for when it is read.
Storage layout
The credit shielded pool lives at [ShieldedBalances(52)]/"M". Token pools live under the
tokens tree:
[Tokens(16)]
[TOKEN_SHIELDED_POOLS_KEY(224)] BigSumTree
[token_id] SumTree (the pool)
NOTES[128] CommitmentTree, chunk power 11 (the note commitments and ciphertexts)
NULLIFIERS[64] ProvableCountTree (spent nullifiers)
ANCHORS_IN_POOL[192] anchor -> block height
TOTAL_BALANCE[32] SumItem (tokens currently shielded)
ANCHORS_BY_HEIGHT[96] block height -> anchor (for pruning)
The five children use the same keys as the credit pool, so the drive primitives that insert notes and nullifiers, read balances and record anchors are shared: every credit pool method has a pool-agnostic form taking the pool path, and a token twin that supplies the token's path. The root of all token pools is a BigSumTree so the amount of every token that is shielded is one sum, which the token conservation check reads.
The root tree is created by the version 14 upgrade transition (transition_to_version_14) on
an existing chain and by create_initial_state_structure version 4 on a new one. A pool's five
trees are created when a contract with the flag is inserted or updated.
Configuration
TokenConfiguration gains a format version 1. It adds hasShieldedPool: bool, the optional
minimumPoolNotesForOutgoing and the minimumPoolNotesForOutgoingChangeRules that govern it.
A version 0 configuration behaves as hasShieldedPool: false.
minimumPoolNotesForOutgoing is how many notes a pool must hold before tokens may leave it.
It is optional and absent by default, and absent reads as 0, no threshold. An issuer may set at
most SystemLimits::max_token_pool_notes_for_outgoing (250), so none can name a floor its pool
never reaches and strand every shielded balance. Unlike the flag, it is not immutable: it
changes through TokenConfigUpdate under minimumPoolNotesForOutgoingChangeRules, which
authorize no one when absent, so an issuer who wants to raise it later has to say so when the
token is created.
The format version is admitted by
dpp.contract_versions.token_versions.token_configuration_format: protocol versions 13 and
below allow only version 0, protocol version 14 allows versions 0 and 1. Contract create and
update reject a token configuration outside the bounds with UnsupportedVersionError, so a
pre-14 network never stores the flag.
The flag is immutable. A contract update that changes it is rejected with
DataContractTokenConfigurationUpdateError for hasShieldedPool, because a pool that was
enabled can hold notes that would become unspendable, and a pool that is enabled late would
need a tree created under an existing token.
A pool also makes freezing and confiscation unenforceable: shielded notes belong to no identity
account, so a holder who expects a freeze shields first and nothing can freeze or destroy those
notes, because no account holds them for an issuer to act on. The notes themselves are not
hidden — their commitments and ciphertexts are stored and queryable — but who owns one and how
much it carries are. Rather than let an issuer advertise controls that cover only transparent
balances, a token with hasShieldedPool must disable them permanently: freezeRules,
unfreezeRules and destroyFrozenFundsRules must each authorize no one to take the action and
have no admin action takers, so no later configuration update can switch them on. Contract
create and update reject anything else with TokenShieldedPoolIncompatibleRulesError (10278).
Pausing still works: every pool operation, inflows included (shield, mint, claim and purchase
into the pool) and outflows (unshield, shielded transfer, burn from the pool), is rejected while
the token is paused. A pooled token can never freeze an account, so the pool validators read and bill no
frozen-account check.
Batch transitions
Seven operations are TokenTransition variants inside a Batch transition, like every other
token operation. The identity signs the batch and pays the fee in credits. Tokens cannot pay
fees, so unlike the credit pool nothing is carved from the bundle's value balance.
| Transition | Flags | Value balance | Extra sighash data | Effect |
|---|---|---|---|---|
TokenShield | outputs only | -amount | tag(0x80), token_id, owner_id | amount leaves the owner's balance and enters the pool as new notes. |
TokenUnshield | spends and outputs | +amount | token_id, owner_id, recipient_id, amount | Notes are spent; amount is credited to recipient_id; change comes back as new notes. |
TokenShieldedTransfer | spends and outputs | 0 | token_id, owner_id | Notes are spent and recreated; the pool balance is unchanged. |
TokenMintToPool | outputs only | -amount | tag(0x81), token_id, minter_id | An authorized minter (manual minting rules, group actions supported) mints amount into new notes; the supply and the pool balance grow. Allowed only where mintingAllowChoosingDestination is set, since the notes' recipients are the minter's choice. |
TokenBurnFromPool | spends and outputs | +amount | token_id, burner_id, amount | An authorized burner (manual burning rules, group actions supported) spends notes and destroys amount; the supply and the pool balance shrink. burner_id is the batch owner, or the proposer of a group action. |
TokenClaimToPool | outputs only | -amount | tag(0x82), token_id, owner_id | A distribution claim released into new notes instead of the claimant's balance; a perpetual claim names the cycle-aligned moment it claims up to so the amount is predictable. |
TokenDirectPurchaseToPool | outputs only | -token_count | tag(0x83), token_id, owner_id | The buyer pays credits at the direct purchase price and the tokens are minted into new notes. |
Each transition carries the Orchard bundle (actions, anchor, proof, binding_signature)
next to the token base transition (token_id, contract id, contract position, identity
contract nonce). The extra sighash data is bound into the Orchard sighash by the client and
recomputed by consensus from the transition's own fields, so a bundle cannot be replayed
against whatever its layout names. The layouts differ: the ones that spend bind the token, the
owner and, where tokens leave the pool, the recipient and amount; the four that only create
notes bind their kind, the token and the owner. The layouts are in
dpp::shielded::sighash (token_unshield_extra_sighash_data,
token_shielded_transfer_extra_sighash_data, token_burn_from_pool_extra_sighash_data and
token_pool_output_only_extra_sighash_data).
Outputs-only bundles (shield, mint, claim, purchase) have no spends, so their anchor is not checked against the pool; the client builds them against the empty tree. Spending bundles must name an anchor the pool has recorded.
A group burn from the pool inherits a deadline from that requirement, and it is worth stating
plainly because nothing in the group machinery announces it. The proposer's bundle spends notes,
so it names an anchor, and the pool keeps an anchor only for shielded_anchor_retention_blocks.
The bundle cannot be re-proved against a fresher anchor to buy more time: the action id covers
the digest of the actions and the signatures are over a sighash that includes the anchor, so a
bundle rebuilt on a new anchor is a different group action rather than the same one continued.
Group actions have neither an expiry nor a cancellation, so a proposal that has not gathered
enough signing power before its anchor is pruned can no longer close and no longer be withdrawn
either — it simply remains open. A group intending to burn from a pool should therefore gather
its signatures well inside the retention window, and an abandoned proposal should be treated as
permanent. Closing this properly needs group actions to gain an expiry or a cancellation; until
they do, the retention window is the real deadline.
The preimage of an outputs-only bundle binds its owner — the batch owner, or for a group action
mint its proposer, whose bundle every other signer submits unchanged — so nobody else's
transition can land a bundle lifted out of the mempool ahead of its author. What the preimage
cannot stop is its owner submitting the same bundle again in a new transition. That repeat
would land a second note with the same commitment and the same rho, hence the same nullifier,
and only one of the two could ever be spent. The pool refuses it on the state side: each
action's dummy nullifier, from which its note takes its rho, is recorded in the pool's
nullifier tree when the bundle enters, and a bundle whose dummy nullifier is already there is
rejected with NullifierAlreadySpentError. TokenPurchaseFromShieldedPool carries an
outputs-only token bundle too and is checked the same way.
A mint or burn into the pool that goes through a group action stores TokenEvent::MintToPool /
TokenEvent::BurnFromPool with a digest of the serialized actions
(serialized_actions_digest), so every signer commits to exactly the same notes. Either bundle
is therefore proven once, by the proposer: the digest fixes the bundle's actions and only the
bundle's builder can sign it, so every other signer submits the proposer's bundle as it is and
the sighash cannot depend on which signer's batch carries it. It binds the group action's
proposer — as burner_id for a burn, as minter_id for a mint, and the batch owner when there
is no group action. CheckTx verifies a batch's token bundles statelessly, seeing only the
transition and not the stored group action, so it skips the bundle of a signer other than the
proposer; state validation in the block verifies it against the proposer. No token history
document is written for pool operations, so the ledger records no actor for them. The
operations still leave public traces: spent nullifiers, the pool's note count and its anchors
are all in state.
Documents paid from the pool
A document type whose action has a token cost can be paid out of the token's pool.
TokenPaymentInfo gains a format version 1 that carries a TokenShieldedPayment: a spend
bundle in the payment token's pool (amount, actions, anchor, proof,
binding_signature) whose value balance is the document action's token cost. The identity still
signs the batch and pays the credit fee; its token balance is never touched. The bundle's sighash
binds the token id, the batch owner, the document's contract and id and the amount
(document_token_payment_extra_sighash_data), so it cannot pay for another document.
The transformer rejects a payment whose amount is not the document type's cost
(TokenShieldedPaymentAmountMismatchError, 40724) and a shielded payment on an action with no
token cost (TokenShieldedPaymentNotRequiredError, 40725). Document base state validation
version 1 skips the owner's balance and frozen-account checks for a shielded payment; the pool
side (pool exists, token not paused, anchor, unspent nullifiers, pool balance, proof) is
validated once the document action itself is valid. The lowering pays the cost from the pool: a
TransferTokenToContractOwner effect is a token unshield into the contract owner's balance, a
BurnToken effect a burn from the pool. The verification fee is charged like the batch pool
transitions, and CheckTx admits the bundle under the identity contract nonce.
Identity-less transitions
Every batch transition above is signed by an identity that pays credits, so the chain sees
which identity moved token T at height H. Three top-level state transitions remove the
identity: each carries a bundle in the token's pool and a second spend bundle in the credit
shielded pool that pays the fee, both authorized only by Orchard spend keys.
| Type | Transition | Token bundle | Fee bundle | Effect |
|---|---|---|---|---|
| 26 | TokenShieldedTransferWithShieldedFee | spends, value 0 | spends, value = fee | Notes spent and recreated in the token pool; the fee leaves the credit pool. |
| 27 | TokenUnshieldWithShieldedFee | spends, value +amount | spends, value = fee | amount leaves the token pool into recipient_id's token balance. |
| 28 | TokenPurchaseFromShieldedPool | outputs only, value -token_count | spends, value = price + fee | token_count is minted into the token pool at the direct purchase price; the price is credited to the contract owner. |
Every transition names the contract, the token position and the token id (which must derive
from the two), the two bundles and credit_amount, the fee bundle's value balance. The token
bundle's sighash binds the state transition type byte, the token id and the transparent fields
(recipient and amount, or count and price); the fee bundle's sighash binds the type byte, the
token id and a digest of the token bundle's actions (token_pool_fee_bundle_extra_sighash_data),
so a fee bundle can only ever pay for that exact token bundle.
The processor treats them like the credit pool's own pool-paid transitions: structure
validation checks both bundles, the minimum fee validation pins credit_amount to exactly the
two-bundle fee (compute_token_pool_paid_shielded_fee: the base fee of each bundle plus the
flat storage of what the transition writes outside the pools; a purchase adds the agreed price
on top), both proofs are verified statelessly, and the transform validates the pools: the
token owns a pool and is not paused, the token bundle's anchor is recorded and its nullifiers
unspent in the token pool, the credit pool holds what leaves it and the fee bundle's anchor and
nullifiers check out there, plus the transition's own rules (recipient exists for an
unshield, the pricing schedule and the max supply for a purchase). Execution is a
PaidFromShieldedPool event: the fee goes to the fee pools, the token side is the matching
token operation, and a purchase credits the contract owner. Uniqueness is by the spent
nullifiers of both bundles; a replay is an unpaid rejection. CheckTx admits their proofs under
the generic pool-paid limiter. All three are gated on TOKEN_SHIELDED_POOL_INITIAL_PROTOCOL_VERSION.
A transfer proves its execution with the token pool nullifiers it spent, an unshield with the recipient's token balance, and a purchase with the token pool's total balance.
Validation
Structure validation checks the amount bounds, the action count against
SystemLimits::max_shielded_transition_actions, the encrypted note sizes, a non-empty proof and
a non-zero anchor.
The credit pool refuses an outgoing spend until it holds minimum_pool_notes_for_outgoing
(250) notes, one floor for the whole network. A token pool's floor is the issuer's to choose:
minimumPoolNotesForOutgoing is optional, and absent reads as 0, so by default a pool with a
single note lets that note be spent at once. The price of that default is that a spend from a
nearly empty pool is linkable to the shield that filled it; the price of a floor is that it
traps a new pool's first depositors until enough notes accumulate, which is why none is set
unless asked for. An issuer may set at most SystemLimits::max_token_pool_notes_for_outgoing
(250), so no issuer can name a floor its pool never reaches and strand every shielded balance,
and may change it later through TokenConfigUpdate under
minimumPoolNotesForOutgoingChangeRules, which authorize no one when absent.
The floor counts note commitments, not holders. One bundle carries several actions, so a single
depositor can reach a threshold alone: it tells holders how busy the pool should be before they
leave it and guarantees no anonymity set. It applies to outflows with a visible destination:
an unshield inside a batch, the identity-less TokenUnshieldWithShieldedFee, a burn from the
pool, and a document's token cost paid from the pool. Transfers inside the pool are not
limited.
State validation runs in this order, and the first failure is returned:
- The token base transition (contract exists, position valid, nonce).
hasShieldedPoolon the token's configuration, elseTokenShieldedPoolNotEnabledError.- The token is not paused, for every operation. Shield: the owner holds
amount. Unshield: the recipient identity exists. Mint to pool: the configuration lets the minter choose the destination (mintingAllowChoosingDestination), the minting rules authorize the identity (or group) and the max supply is not exceeded. Burn from pool: the burning rules authorize the identity and the token is not paused. Claim to pool: the claim resolves exactly as a claim into a balance does (the sharedresolve_token_claim). Purchase to pool: the pricing schedule and the max supply. - Spending bundles: the anchor is recorded in the pool (
InvalidAnchorError), no nullifier repeats within the bundle or is already spent (NullifierAlreadySpentError), and for an unshield the pool holdsamount. Outputs-only bundles: no dummy nullifier repeats within the bundle or is already recorded in the pool (NullifierAlreadySpentError). - Proof verification. The fee for it,
compute_shielded_verification_fee(actions), is added as a precalculated operation before the Halo 2 proof and binding signature are checked, so a failed proof is a paid failure: the identity is charged, its nonce advances, and nothing else moves.
Batch state validation does not run in CheckTx. The mempool admission path
(CheckTxProofVerifier) verifies the proofs of batch token transitions keyed by the identity
contract nonce, so a proof is verified once per nonce before the block and not again for the
same submission.
The transitions are gated on the protocol version: below 14 validate_is_allowed rejects a
batch carrying any of them with StateTransitionNotActiveError.
Execution and conservation
The drive operations are composites of the credit pool primitives re-rooted under the token:
- shield: remove
amountfrom the owner's token balance, insert the bundle's dummy nullifiers, append the notes, addamountto the pool'sTOTAL_BALANCE; - unshield: insert the nullifiers, append the notes, subtract
amountfrom the pool balance, addamountto the recipient's token balance; - shielded transfer: insert the nullifiers, append the notes;
- mint, claim and purchase to pool: insert the bundle's dummy nullifiers, append the notes,
add
amountto the pool balance and to the total supply (TokenMintToPool); - burn from pool: insert the nullifiers, append the change notes, subtract
amountfrom the pool balance and from the total supply (TokenBurnFromPool).
Only a mint, claim, purchase or burn changes a token's total supply. calculate_total_tokens_balance version 1 reads
the token pools BigSumTree and the block end conservation check requires
identity balances + pool balances == total supply.
Block end
Every successfully executed token pool transition, document paid from a pool and
identity-less token pool transition records its pool in
StateTransitionsProcessingResult::token_shielded_pools_touched. A paid refusal records
nothing: it bumps a nonce and writes to no pool, and the pool it named may not even exist,
so letting one through would hand the anchor recorder a pool the chain does not hold. At
block end
record_token_shielded_pool_anchors (enabled by DRIVE_ABCI_METHOD_VERSIONS_V10) records
each touched pool's current anchor if the commitment tree changed and prunes that pool's
anchors older than shielded_anchor_retention_blocks, always keeping the newest one. Pruning
is driven by touches rather than by an interval because there can be many pools; an idle pool
keeps a valid anchor to spend against.
Queries
The six shielded pool queries (getShieldedPoolState, getShieldedNotesCount,
getShieldedAnchors, getMostRecentShieldedAnchor, getShieldedEncryptedNotes,
getShieldedNullifiers) take an optional token_id. Without it they target the credit pool;
with a 32-byte token id they target that token's pool and answer with the same response shape.
A token id is rejected with InvalidArgument before protocol version 14 or when it is not 32
bytes. The proof verifier routes on the same field to the token twins of the verify functions
(verify_token_shielded_pool_state and the rest), and the Rust SDK exposes
TokenShieldedPoolQuery, TokenShieldedEncryptedNotesQuery and TokenShieldedNullifiersQuery.
A token pool transition proves its execution like the token transfer it resembles: a shield proves the owner's balance, an unshield proves the recipient's balance, and a shielded transfer proves the spent nullifiers in the token's pool.
Client builders
dpp::shielded::builder provides build_token_shield_transition,
build_token_unshield_transition, build_token_shielded_transfer_transition,
build_token_mint_to_pool_transition, build_token_burn_from_pool_transition,
build_token_claim_to_pool_transition and build_token_direct_purchase_to_pool_transition.
They prove the bundle, bind the extra sighash data, and call the batch constructors
(new_token_shield_transition and siblings) which sign with the identity key.
build_document_shielded_token_payment builds the bundle of a TokenPaymentInfo::V1.
For document creation, call Document::set_id_for_creation with the creation nonce before
building the payment: the proof must bind the final document ID, not the factory placeholder.
build_token_shielded_transfer_with_shielded_fee_transition,
build_token_unshield_with_shielded_fee_transition and
build_token_purchase_from_shielded_pool_transition build the identity-less transitions from a
TokenPoolSpender (the token pool notes and keys) and a ShieldedFeePayer (the credit pool
notes and keys). The wasm bindings expose a wrapper per transition, TokenPaymentInfo accepts
a shieldedPayment, and TokenConfiguration accepts hasShieldedPool and reports
formatVersion.
Fees
See Shielded Transaction Fees. In short:
the identity pays the metered cost of the writes plus the proof verification fee, exactly like
ShieldFromIdentity, for every batch transition and for a document paid from the pool; the
identity-less transitions carve the two-bundle fee from the credit pool bundle.
Contract Keywords
A data contract describes its documents with a JSON schema per document type, and Platform reads a set of keywords in those schemas: some from JSON Schema, most of its own. The chapters of this part take each keyword, or a small group that works together, and say what it does, how to write it, what is checked when a contract is registered and when a document is written, what a later contract update may do with it, and which errors it produces. The chapters of the Data Model and Drive parts explain the internals behind them and are linked from each chapter.
Everything here follows the document meta-schema of protocol version 14, packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json. Every document type schema is validated against it when a contract is registered or updated, and the parser (try_from_schema) checks the rules a JSON schema cannot express. When a chapter and the meta-schema disagree, the meta-schema is right and the chapter is out of date.
Where keywords go
A contract's documentSchemas maps each document type name to its schema. Keywords sit at three levels:
"post": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": true,
"indices": [
{ "name": "byOwner", "properties": [{ "$ownerId": "asc" }, { "$createdAt": "asc" }] }
],
"properties": {
"text": { "type": "string", "maxLength": 280, "maxBytes": 560, "position": 0 },
"replyTo": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "deletableDocument", "documentType": "post" },
"position": 1
}
},
"required": ["$createdAt", "text"],
"additionalProperties": false
}
- Document type keywords sit at the top of the schema (
documentsMutable,canBeDeleted,indices,required). They say what may happen to a document of the type and who may do it. - Index keywords sit inside an entry of
indices(name,properties). - Property keywords sit inside a property's schema (
type,maxLength,maxBytes,position,refersTo). Most are ordinary JSON Schema; the rest are Platform's own.
The contract around the document types has keys of its own, and a config object: see Contract-Level Keys and config.
Reading the chapters
Each chapter opens with a short table for each keyword:
- Since is the first protocol version at which the keyword can be used.
- On update is what a contract update may do with the keyword on a document type that already exists. Fixed means adding, removing and changing it are all refused. A document type the update adds may use any keyword, as a new contract may.
- Errors are consensus errors, written
ErrorName(code). See Error Codes for the code ranges.
A contract update that breaks an update rule is refused with one of two errors, depending on which check catches it. DocumentTypeUpdateError (40212) comes from the comparison of the parsed document types, which judges flags such as documentsMutable by their meaning. IncompatibleDocumentTypeSchemaError (10246) comes from the comparison of the two JSON schemas, which judges property keywords such as refersTo or maxLength by their text. Top-level required and indices have errors of their own (10276 and 10217). Because the schema comparison reads text, an edit that changes how a keyword is written but not what it means, such as writing out a default or switching to the documentsAverageable shorthand, is refused with 10246.
Protocol versions
The document meta-schema has changed three times:
| Meta-schema | Protocol versions | What it added |
|---|---|---|
| v0 | 1 to 11 | The original keywords. A document type key the meta-schema did not know was ignored. |
| v1 | 12 | Unknown document type keys are refused. The count, sum and average keywords. |
| v2 | 13 | keepsTransferHistory, keepsPurchaseHistory, keepsPricingHistory. |
| v3 | 14 | References, typed arrays, requiredSince, immutable, ttl, propertyConstraints, deleteConstraints, actionFees, moderation deletion, ranked, time-range and integer-range indexes, index-only types, and the rest marked 14 in these chapters. |
Most keywords of v0 took effect at protocol version 1. The exceptions are tokenCost (9) and the index keyword countable (12).
The complete language
Every key a contract can write, grouped by where it goes. Since is the protocol version from which the key works. Read more links to the key's section in this part and, where there is one, to the chapter with the internals. A dotted name such as tokenCost.<action>.amount is a key written inside the ones before it.
The contract
| Key | Takes | What it does | Since | Read more |
|---|---|---|---|---|
$formatVersion | "0" or "1" | The contract's serialization format. "1", the default from 9, carries groups, tokens, keywords, description and the timestamps. | 1 | Contract keys |
id | identifier | The contract's id: a hash of ownerId and the identity nonce of the create transition. | 1 | Contract keys |
ownerId | identifier | The identity that registers the contract, and the only one that may update it. | 1 | Contract keys |
version | integer | 1 at creation; every update raises it by exactly one. | 1 | Contract keys |
config | object | Contract-wide settings, below. | 1 | config |
documentSchemas | object of document types | The document types by name, each written with the document type keys. | 1 | documentSchemas |
schemaDefs | object | Definitions any property may point at with $ref. | 1 | Document Shape |
groups | object | Sets of identities, each member with a voting power, whose approval some token actions need. | 9 | Data Contracts |
tokens | object | The contract's tokens, by position. Their configuration is not covered in this part. | 9 | Data Contracts · Creating a Basic Token |
keywords | up to 50 strings of 3 to 50 bytes | Search keywords, for the keyword search contract. | 9 | keywords and description |
description | string of 3 to 100 bytes | A short description, for the keyword search contract. | 9 | keywords and description |
createdAt, updatedAt, createdAtBlockHeight, updatedAtBlockHeight, createdAtEpoch, updatedAtEpoch | numbers | When the contract was created and last updated. Set by the platform, never written. | 9 | Contract keys |
contractGroup | { "admins", "name", "description" } | On the create transition, beside the contract: registers a contract group, a set of contracts the signer owns, with up to 16 admins, a name of 1 to 64 characters and a description of 1 to 256. | 14 | Contract Groups |
contractGroupMemberships | up to 16 { "contractGroupId", "member" } | On the create transition: enrols the new contract ("contract"), one of its document types ({ "documentType": ... }) or one of its tokens ({ "token": ... }) in contract groups. | 14 | Contract Groups |
config
| Key | Takes | What it does | Since | Read more |
|---|---|---|---|---|
canBeDeleted | boolean, default false | Whether the contract may ever be deleted. No transition deletes a contract today. | 1 | canBeDeleted |
readonly | boolean, default false | true: the contract can never be updated. | 1 | readonly |
keepsHistory | boolean, default false | Drive keeps every version of the contract. | 1 | keepsHistory |
documentsKeepHistoryContractDefault | boolean, default false | documentsKeepHistory for a document type that does not say. | 1 | Document type defaults |
documentsMutableContractDefault | boolean, default true | documentsMutable for a document type that does not say. | 1 | Document type defaults |
documentsCanBeDeletedContractDefault | boolean, default true | canBeDeleted for a document type that does not say. | 1 | Document type defaults |
requiresIdentityEncryptionBoundedKey, requiresIdentityDecryptionBoundedKey | 0 unique, 1 multiple, 2 multiple with a pointer to the latest | Lets identities bind encryption or decryption keys to the whole contract, and says how they are kept. | 1 | Bounded key requirements · Contract Bounds |
sizedIntegerTypes | boolean, default true | Stores each integer in the smallest width its bounds allow, instead of 8 bytes. | 9 | sizedIntegerTypes |
moderation | object | Makes the contract moderated: which lists it keeps and who moderates. | 14 | moderation · Contract Moderation |
moderation.banlist, .suspensions, .warnings | boolean, default false | Keeps a banlist, a suspension list, a warning list. A banned or suspended identity cannot act on the contract's documents; a warning bars nothing. | 14 | The Model |
moderation.moderators | { "$type": ... } | Who moderates: "contractOwner", "appointedModerators" with identities (1 to 16), or "elected" with the keys below. | 14 | moderation |
moderators.seatContestable | boolean, required when elected | Whether a seated team may later be challenged. | 14 | Elected Moderation |
moderators.challengeCoolDown | seconds, two weeks to three years | How long a seated team is safe from a challenge after a seat change. Required when the seat is contestable, refused when it is not. | 14 | Elected Moderation |
moderators.moderatedDocumentTypes | object: document type → abilities | The document types the team moderates, each with its abilities: ban, suspend, warn, deleteDocuments. | 14 | Elected Moderation |
moderators.interim | { "$type": ... } | Who moderates until a team is seated: "contractOwner", "appointedModerators", "notYetUsable" (the moderated types cannot be used yet) or "noModeration". | 14 | Elected Moderation |
moderators.joinWindow, .voteWindow | seconds | How long applicants may join an election, and how long masternodes then vote. | 14 | Elected Moderation |
moderators.electionDelay | seconds | How long after the contract's creation the first election may be called. | 14 | Elected Moderation |
moderators.maxAddedModerators | 0 to 15, default 0 | How many members the seated leader may add after the election. | 14 | Elected Moderation |
moderators.ownerProtected | boolean, default false | Protects the contract owner from the seated team. | 14 | Elected Moderation |
Document type
| Key | Takes | What it does | Since | Read more |
|---|---|---|---|---|
type | "object" | Required. A document is an object. | 1 | type |
properties | object of 1 to 100 properties | The document's properties, each written with the property keys. | 1 | properties |
required | array of names | The properties every document holds. A system time or height listed here is recorded. | 1 | required |
additionalProperties | false | Required: a document holds only the declared properties. | 1 | additionalProperties |
minProperties, maxProperties | integer | How many properties a document holds. | 1 | minProperties and maxProperties |
dependentRequired | object | A property that requires others when present. | 1 | dependentRequired |
$comment, description | string | Notes; consensus ignores them. | 1 | $comment and description |
$schema, $defs | added by the platform | The meta-schema URL and the contract's schemaDefs. A document type writing either is refused. | 1 | $schema and $defs |
transient | array of top-level names | Properties validated on the transition but never stored. | 1 | transient · internals |
documentsMutable | boolean, default true | false: documents cannot be replaced. | 1 | documentsMutable |
immutable | array of top-level names and { property, when } | Properties frozen at creation, or while a condition holds, on a mutable type. | 14 | immutable · internals |
canBeDeleted | boolean or "onlyWhenConsumed", default true | false: a document's owner cannot delete it. "onlyWhenConsumed" (14): only a create that consumes it deletes it. | 1 | canBeDeleted |
deleteConstraints | object of named rules, at most 16 | Rules, in the grammar of propertyConstraints, the stored document meets for its owner to delete it: a poll deleted only while no vote points at it. | 14 | deleteConstraints |
retractedWhen | one condition, as an immutable entry's when | The replace a banned or suspended owner may still make on a moderated contract: one whose written document meets the condition. | 14 | retractedWhen · internals |
moderatorAbilities | object: delete, deleteWithin (seconds), deleteKeepsRecord, deleteRefundsOwner, deleteSettled (leader, approvals, approversPredateDocument), deleteKeepsFields (array of property paths), changeFields (array of top-level names) | What the contract's moderators may do to documents of the type: delete them, within a window after their last change, past it only when so many members of a seated team agree (the leader among them when the rule says so, members the leader added only for documents written after their addition unless the rule says otherwise), with or without a removal record keeping the fields that stay public and a refund to the owner, and write the fields only they write. | 14 | Moderator Abilities · delete · internals |
ttl | seconds, 3600 to 31536000 | The platform deletes each document this long after its creation. | 14 | Time To Live · internals |
creationRestrictionMode | 0 anyone, 1 contract owner, 2 nobody | Who may create documents. | 1 | creationRestrictionMode |
transferable | 0 never, 1 always | Whether an owner may give a document to another identity. | 1 | transferable |
tradeMode | 0 none, 1 direct purchase | Whether an owner may set a price and anyone buy at it. | 1 | tradeMode |
documentsKeepHistory | boolean, default false | Drive keeps every revision of every document. | 1 | documentsKeepHistory |
keepsTransferHistory, keepsPurchaseHistory, keepsPricingHistory | boolean, default false | Records every transfer, purchase or price update in the document history contract. | 13 | History |
signatureSecurityLevelRequirement | 1 critical, 2 high (default), 3 medium | The weakest key level that may sign a transition on the type. | 1 | signatureSecurityLevelRequirement · Security Level |
requiresIdentityEncryptionBoundedKey, requiresIdentityDecryptionBoundedKey | 0 unique, 1 multiple, 2 multiple with a pointer to the latest | Lets identities bind encryption or decryption keys to the type, and says how they are kept. | 1 | Signing and Keys · Contract Bounds |
ownerRefersTo | a refersTo declaration | A reference the writer must meet, on types whose documents are never transferred or traded. | 14 | ownerRefersTo · internals |
creatorRefersTo | a refersTo declaration | A reference the creator must meet, on types whose documents can be transferred or traded. | 14 | creatorRefersTo |
propertyConstraints | object of named rules, at most 16 | Rules over several properties every created or replaced document meets. See the operators. | 14 | propertyConstraints · internals |
tokenCost | object keyed by action | Token payments for actions on documents. See the keys. | 9 | Token Costs · Fees |
actionFees | object keyed by action | Credit fees for actions on documents, paid to the owner's and moderators' pots. See the keys. | 14 | Action Fees · Fees |
indices | array of 1 to 10 indexes | The indexes documents are queried by, each written with the index keys. | 1 | Indexes · internals |
documentsCountable | boolean | Keeps a count of the type's documents. | 12 | documentsCountable · internals |
documentsSummable | property name | Keeps the sum of one integer property over the type's documents. | 12 | documentsSummable · internals |
documentsAverageable | property name | Shorthand for documentsCountable plus documentsSummable. | 12 | documentsAverageable |
rangeCountable, rangeSummable, rangeAverageable | boolean | Provable counts, sums or averages over ranges of document ids. | 12 | Document type range keys |
indexOnly | boolean | Documents are never stored whole: the index entries are the rows. | 14 | indexOnly · internals |
entryPayload | array of 1 to 16 names | On an index-only type, properties carried in each entry's value instead of a key. | 14 | entryPayload |
Property
| Key | Takes | What it does | Since | Read more |
|---|---|---|---|---|
type | string, integer, number, boolean, object, array | The kind of value. An array is a byte array or a typed array. | 1 | type |
position | integer | The property's place in the stored document. Required; top-level positions run 0, 1, 2 with no gap. | 1 | position · Document Serialization |
minLength, maxLength | integer | A string's length in characters. | 1 | Strings |
pattern | regular expression | A string must match it. Needs maxLength of at most 50000. | 1 | Strings |
format | date-time, date, time, email, idn-email, hostname, ipv4, ipv6, uri, regex | A string must have this format. Needs maxLength of at most 50000. | 1 | Strings |
maxBytes | 1 to 65535 | The most UTF-8 bytes a string may take. | 14 | maxBytes · internals |
minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf | number | Numeric bounds. minimum and maximum also decide an integer's stored width. | 1 | Numbers |
enum, const | values | The values allowed, or the one value allowed. | 1 | enum and const |
byteArray | true | Makes an array a string of bytes, stored raw. | 1 | Byte arrays and identifiers |
contentMediaType | "application/x.dash.dpp.identifier" | Makes a 32-byte array an identifier. | 1 | Byte arrays and identifiers |
minItems, maxItems, uniqueItems, contains | integer, boolean, schema | A byte array's length in bytes, or a typed array's number of elements (at most 1024); no repeats; an element that matches. | 1 | Arrays |
items | an element schema | Makes an array a typed array whose elements all follow this schema. | 14 | Typed Arrays · internals |
properties, required, additionalProperties, minProperties, maxProperties, dependentRequired | as on a document type | A nested object's members and its bounds. | 1 | Objects |
$ref | "#/$defs/<name>" | Uses a definition from the contract's schemaDefs. | 1 | $ref |
$id, $comment, description, examples | annotations | Notes; consensus ignores them. | 1 | Annotations |
requiredSince | contract version | Lets an update add a required property that older documents may leave out. | 14 | requiredSince · internals |
distinctFrom | a property path or "$ownerId" | An identifier must differ from another identifier of the document, or from the owner. | 14 | distinctFrom · internals |
encryptedFor | { "recipient", "recipientKey", "senderKey", "scheme" } | Declares how an encrypted byte array was made: whose keys, which scheme. | 14 | encryptedFor · internals |
encryptedFor.recipient | identifier property path or "$ownerId" | The identity the value is encrypted to. | 14 | encryptedFor |
encryptedFor.recipientKey, .senderKey | integer property paths | The properties holding the recipient's and the sender's key ids. | 14 | encryptedFor |
encryptedFor.scheme | "ecdh-secp256k1-aes256-cbc" | How the ciphertext is made. | 14 | The scheme |
generatedFrom | { "function", "params" } | The platform generates the string from other properties of the document; on arrival when a document leaves it out. | 14 | generatedFrom · internals |
generatedFrom.function | "sys.stringTransformations.homographSafeASCII" | The system function that generates the value: sys.stringTransformations. lowercase, uppercase, capitalize, camelCase, snakeCase or homographSafeASCII. | 14 | Functions |
generatedFrom.params | property paths | The properties the function reads, in order. | 14 | Params |
refersTo | a declaration | What an identifier points at, checked when a document is written. See the keys. | 14 | References · internals |
A typed array's element (items) takes type, enum, minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf, minLength, maxLength, pattern, format, minItems and maxItems (bytes of a byte array element), byteArray, contentMediaType, maxBytes, distinctFrom, refersTo, $comment and description. It takes no position, const, uniqueItems or examples.
refersTo
| Key | Takes | What it does | Since | Read more |
|---|---|---|---|---|
type | a target below | What the value points at. | 14 | Targets |
type: "identity" | The value is the id of an existing identity. | 14 | identity | |
type: "contract" | The value is the id of an existing data contract. | 14 | contract | |
type: "token" | The value is the id of an existing token. | 14 | token | |
type: "permanentDocument" | The value is the id of a document whose type can never lose its documents; with findBy, part of the key that finds it; with inList, an element of its list. | 14 | permanentDocument | |
type: "deletableDocument" | The value is the id of a document that can be deleted, or with findBy part of the key that finds it; checked again on every replace. | 14 | deletableDocument | |
type: "identityPublicKey" | The value names an identity key that exists and is not disabled. | 14 | identityPublicKey | |
documentType | document type name | The referenced document type. | 14 | documentType |
contractId | identifier | The contract holding documentType, when it is not this one. | 14 | contractId |
findBy | 1 to 10 entries: referenced property → ".", "$ownerId", a path or a function | Finds the document through the unique index of documentType over exactly these properties, in any order; "." is the value itself. Without it the value is the document's $id. | 14 | findBy · internals |
findBy.<property> | { "function": "sys.hash.sha256d", "params" } | A function entry: the referenced property holds the hash of params (paths, { "const": text }, "." for a value without a path), which fills that part of the key, finding a commitment made earlier. At most one. | 14 | Commit and reveal · internals |
where | 1 to 10 entries { "<referenced property>": "<referring value>" } | Checked on the document found: each referenced property (or $ownerId, $creatorId, $id) must equal the referring property. A value of "$ownerId", the writer, makes a write gate. Never finds the document. | 14 | where |
minimumAgeBlocks | 1 to 4294967295 | Beside a findBy function: the commitment was created at least this many blocks before the create. | 14 | Commit and reveal |
consume | true | Beside a findBy function, on a deletableDocument with the where entry "$ownerId": "$ownerId", into a type with canBeDeleted true or "onlyWhenConsumed" and no deleteConstraints: the create deletes the writer's commitment. | 14 | Commit and reveal |
inList | typed array path | On a permanentDocument whose findBy is { "$id": <property> }: the list on that document the value must be in. | 14 | List Elements · internals |
keyIdProperty | integer property path | On an identity property: the property holding the key id. | 14 | keyIdProperty and identityProperty |
identityProperty | "$ownerId", "$creatorId" or a path | On a key id property: whose key it is. | 14 | keyIdProperty and identityProperty |
keyRequirements.purpose | authentication, encryption, decryption, transfer, voting, owner | The key's purpose. | 14 | keyRequirements |
keyRequirements.boundTo | document type name | The key must be bound to that document type of this contract. | 14 | keyRequirements |
contractRequirements.moderation | "elected", "electionOpen" | The contract declares an elected team, or one whose election may be called. | 14 | contractRequirements · Elected Moderation |
contractRequirements.minimumAgeSeconds, .minimumSecondsSinceUpdate | seconds | The contract was created, or last changed, at least this long ago. | 14 | contractRequirements |
contractRequirements.owner | "self", "other" | The contract is owned by the writer, or by someone else. | 14 | contractRequirements |
contractRequirements.readonly, .keepsHistory | true | The contract can never be updated, or keeps history. | 14 | contractRequirements |
contractRequirements.ownerProtected | boolean | The contract's elected team does, or does not, protect its owner. | 14 | contractRequirements |
anyOf, allOf | 2 to 4 operands | In place of type: at least one, or every, operand holds. Nest at most 4 deep. | 14 | Expressions · internals |
tokenCost and actionFees
<action> is one of create, replace, delete, transfer, update_price and purchase.
| Key | Takes | What it does | Since | Read more |
|---|---|---|---|---|
tokenCost.<action>.tokenPosition | 0 to 65535, required | Which token is charged. | 9 | Token Costs |
tokenCost.<action>.amount | at least 1, required | How many tokens the action costs. | 9 | Token Costs |
tokenCost.<action>.contractId | identifier | The contract whose token is charged, when it is not this one. | 9 | contractId |
tokenCost.<action>.effect | 0 to the contract owner (default), 1 burn | What happens to the tokens paid. | 9 | effect |
tokenCost.<action>.gasFeesPaidBy | 0 document owner (default), 1 contract owner, 2 prefer contract owner | Who the contract owner offers to have pay the gas. Accepted from 9, acted on from 14. | 14 | gasFeesPaidBy · Fees |
tokenCost.<action>.optional | boolean, default false | A transition may skip the token and pay in credits. | 14 | Optional costs · Fees |
actionFees.pricing | "feeMultiplier" (default), "fixed" | Whether the amounts scale with the epoch's fee multiplier. | 14 | Action Fees |
actionFees.<action>.owner | credits | Paid into the contract owner's pot. | 14 | The pots and the claim |
actionFees.<action>.moderators | credits | Paid into the moderators' pot. Needs moderation. | 14 | The pots and the claim · Fee Pots |
propertyConstraints
A rule is one condition, in propertyConstraints and in deleteConstraints alike. Conditions:
| Key | Takes | Holds when | Since | Read more |
|---|---|---|---|---|
equal, notEqual | [a, b] | The two sides are equal, or differ: integer expressions, strings or identifiers. | 14 | Conditions |
lessThan, lessThanOrEqual, greaterThan, greaterThanOrEqual | [a, b] | The integer comparison holds. | 14 | Conditions |
in | [a, [values]] | a takes one of two or more listed integers, strings or identifiers. | 14 | Conditions |
present, absent | a path | The document holds the property, or leaves it out (or null, or an object with no member present). | 14 | Conditions |
anyOf, allOf | two or more conditions | At least one, or every, condition holds, checked in order. | 14 | Evaluation order |
not | a condition | The condition does not hold. | 14 | Conditions |
ifThen, ifThenElse | [if, then], [if, then, else] | The second condition holds when the first does (and, for ifThenElse, the third when it does not); only the branch taken is evaluated. | 14 | Conditions |
notIn | [a, [values]] | a takes none of the listed values. | 14 | Conditions |
startsWith, endsWith | [text, affix] | A string starts or ends with another, byte for byte. | 14 | Conditions |
contains | [array, value] | A typed array holds an element equal to the value. | 14 | Conditions |
Expressions:
| Key | Takes | Value | Since | Read more |
|---|---|---|---|---|
| an integer | 100 | Itself. | 14 | Expressions |
| a path | "price", "meta.total" | An integer or boolean property's value; 0 when left out. | 14 | Expressions |
add, multiply | two or more operands | The sum or product. | 14 | Arithmetic |
subtract, divide, modulo, power | [a, b] | The difference, Euclidean quotient or remainder, or power. | 14 | Arithmetic |
min, max, abs | two or more operands, or one for abs | The least, the greatest, or the absolute value. | 14 | Expressions |
countOf, sumOf | [type, filter?], [type, property, filter?] | How many documents of a type of the contract match the filter, or the total of an integer property over them, from its count or sum trees. | 14 | Totals of other documents |
ifAbsent | [path, default] | The property's value, or the default when left out (an integer, or a string for a string property). | 14 | Expressions |
length, byteLength | a string path | A string's length in characters, or in UTF-8 bytes. | 14 | Expressions |
count | an array path | The elements of a typed array, or the bytes of a byte array. | 14 | Expressions |
countPresent | [path, path, ...] | How many of two or more properties the document holds, each as present tests it. | 14 | How many of a group |
$createdAt, $updatedAt, $transferredAt, $createdAtBlockHeight, $updatedAtBlockHeight, $transferredAtBlockHeight, $createdAtCoreBlockHeight, $updatedAtCoreBlockHeight, $transferredAtCoreBlockHeight | a path | A time or height the document records, when listed in required. | 14 | Times and heights |
const | a string | A string constant, or a base58 identifier, as one side of equal or notEqual. | 14 | Strings |
$ownerId | The document's owner, as an identifier side. | 14 | Identifiers and $ownerId | |
$id | The document's id, as the value a countOf or sumOf filter matches by: { "pollId": "$id" }, the documents pointing at it. | 14 | Totals of other documents |
Index
| Key | Takes | What it does | Since | Read more |
|---|---|---|---|---|
name | 1 to 32 characters, required | The index's name, unique in the type. | 1 | name |
properties | 1 to 10 { "<path>": "asc" } | The indexed properties, in order. A flat index of an index-only type leaves it out. From protocol version 14 a path may read through a reference, "<reference property>.<field>", a value of the referenced document the document does not store. | 1 | properties · Values of Referenced Documents · internals |
unique | boolean | No two documents share the indexed values. | 1 | unique |
nullSearchable | boolean, default true | false leaves out documents whose indexed values are all null. | 1 | nullSearchable |
contested | object | Matching values are decided by a masternode vote, not first come. | 1 | Contested Indexes · internals |
contested.resolution | 0 vote with lock, 1 vote without lock | How the contest is decided. 1 from 14. | 1 | The keys |
contested.fieldMatches | [{ "field", "regexPattern" }] | Which values are contested. | 1 | The keys |
contested.description | string | A note; consensus ignores it. | 1 | The keys |
countable | "notCountable", "countable", "countableAllowingOffset" or boolean | Keeps a document count per indexed value. | 12 | countable · internals |
summable | property name | Keeps the sum of an integer property per indexed value. | 12 | summable · internals |
averageable | property name | Shorthand for countable plus summable. | 12 | averageable |
rangeCountable, rangeSummable, rangeAverageable | boolean | Provable counts, sums or averages over ranges of the indexed value. | 12 | Index range keys |
rankedCountable | boolean or { "at": ... } | Orders the indexed values by document count, for "top K" queries; at names the levels ranked. | 14 | rankedCountable · internals |
rankedSummable, rankedAverageable | boolean, or { "at": ... } on a summableOffCountIndex index | Orders them by sum, or by average. | 14 | Ranked Indexes |
timeRange | { "on", "range", "step", "phase", "ttl" } | Buckets a system timestamp into time windows, for trending queries. | 14 | Time-Range Indexes · internals |
timeRange.on | "$createdAt", "$updatedAt", "$transferredAt" | The timestamp to bucket: the index's first property. | 14 | The keys |
timeRange.range, .step | seconds | Each window's length, and the time between window starts. | 14 | The keys |
timeRange.phase | seconds, default 0 | Shifts the window boundaries. | 14 | The keys |
timeRange.ttl | seconds, at most one week | Expires the index's entries after their window; on an index-only type, the rows leave this index. | 14 | The keys · internals |
integerRange | { "on", "range", "step", "phase" } | Buckets an integer property into value windows, for counts and rankings per band. | 14 | Integer-Range Indexes |
integerRange.on | property name | The integer property to bucket: the index's first property, required. | 14 | The keys |
integerRange.range, .step, .phase | integers, phase default 0 | Each window's length, the distance between window starts, and the shift of the window boundaries. | 14 | The keys |
terminal | property name or list | On an index-only type, what keys each entry in place of the document id. | 14 | terminal |
preallocated | boolean | On an index-only type, creates the index's trees with the referenced document. | 14 | preallocated · internals |
summableOffCountIndex | index name | On an index-only type, keeps one counter per group of how many entries the named index keeps for it, in place of an entry per document: count(*) and the sum read the named index's entries, and the average divides them by the groups. | 14 | summableOffCountIndex |
outlivesDelete | boolean | On an index-only type's time window with a ttl, a delete leaves the index's entries to expire, and a create writes over one already there. | 14 | outlivesDelete · internals |
skipIfAbsent | true or property names | A document missing a property of the skip set writes no entry into the index. | 14 | skipIfAbsent · internals |
System properties
| Property | Holds | Recorded | Since | Read more |
|---|---|---|---|---|
$id | The document's id. | always | 1 | $id |
$ownerId | The identity that owns the document. | always | 1 | $ownerId |
$revision | 1 at creation, raised by every replace, transfer, price update and purchase. | on types whose documents can change hands or content | 1 | $revision |
$createdAt, $updatedAt, $transferredAt | Block times, in milliseconds, of the creation, the last replace or price update, and the last transfer or purchase. | when listed in required | 1 | Timestamps |
$createdAtBlockHeight, $updatedAtBlockHeight, $transferredAtBlockHeight | Platform block heights of the same events. | when listed in required | 1 | Block heights |
$createdAtCoreBlockHeight, $updatedAtCoreBlockHeight, $transferredAtCoreBlockHeight | Core chain block heights of the same events. | when listed in required | 1 | Block heights |
$creatorId | The identity that created the document. | on transferable or tradeable types of format-1 contracts | 10 | $creatorId |
$moderatedAt, $moderatedBy | Block time and moderator of the last write of the fields only moderators write. | on types listing moderatorAbilities.changeFields, once a moderator writes them | 14 | $moderatedAt and $moderatedBy |
Limits
The first three limits come from the meta-schema, the rest from protocol version 14's SystemLimits. A contract over a limit is refused at registration.
| Limit | Value | Applies to |
|---|---|---|
| Properties per object | 100 | properties, at the top and in each nested object |
| Indexes per document type | 10 | indices |
| Properties per index | 10 | an index's properties |
max_field_value_size | 5120 bytes | any one value a document stores (DocumentFieldMaxSizeExceededError, 10417) |
max_typed_array_items | 1024 | a typed array's maxItems |
max_references_per_document | 256 | references one document carries |
max_reference_operands | 4 | operands in one anyOf or allOf of a reference |
max_reference_expression_depth | 4 | nesting of reference expressions |
max_property_constraints | 16 | rules in one propertyConstraints |
max_property_constraint_nodes | 32 | nodes in one rule |
min_document_ttl_seconds, max_document_ttl_seconds | 3600, 31536000 | ttl |
max_time_range_ttl_seconds | 604800 | a timeRange index's ttl |
max_contested_summed_value_magnitude | 134217728 (2^27) | the minimum and maximum of a summed property on a type with a contested index |
max_expiring_signed_summed_value_magnitude | 134217728 (2^27) | the minimum and maximum of a summed property that admits negative values, on a type with a ttl |
Document Shape
A document type's schema is a JSON Schema object with some Platform keywords added. The keywords in this chapter give a document its outline: it is an object, it has these properties and no others, some of them must be present, and a few JSON Schema rules hold over the document as a whole. What a single property may hold is described in Property Schemas; what may happen to a document (replace, delete, transfer) has chapters of its own.
Example
The DashPay profile type, with its indexes left out:
"profile": {
"type": "object",
"properties": {
"avatarUrl": { "type": "string", "format": "uri", "minLength": 1, "maxLength": 2048, "position": 0 },
"avatarHash": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 1 },
"avatarFingerprint": { "type": "array", "byteArray": true, "minItems": 8, "maxItems": 8, "position": 2 },
"publicMessage": { "type": "string", "minLength": 1, "maxLength": 140, "position": 3 },
"displayName": { "type": "string", "minLength": 1, "maxLength": 25, "position": 4 }
},
"minProperties": 1,
"dependentRequired": {
"avatarUrl": ["avatarHash", "avatarFingerprint"],
"avatarHash": ["avatarUrl", "avatarFingerprint"],
"avatarFingerprint": ["avatarUrl", "avatarHash"]
},
"required": ["$createdAt", "$updatedAt"],
"additionalProperties": false
}
A profile holds at least one of its five properties, and never any other. The three avatar properties come together or not at all. No property of its own is required, but the platform records the time each profile was created and last updated, because required lists $createdAt and $updatedAt.
type
| Where | The top of a document type. Required. |
| Value | "object", the only value allowed |
| Since | protocol version 1 |
| On update | Fixed (IncompatibleDocumentTypeSchemaError, 10246) |
| Errors | JsonSchemaError (10101) at registration |
A document is always an object: a set of named properties. Properties inside a document have their own type, described in Property Schemas.
properties
| Where | The top of a document type. Required. |
| Value | An object mapping 1 to 100 property names to property schemas |
| Since | protocol version 1 |
| On update | Properties may be added, never removed (IncompatibleDocumentTypeSchemaError, 10246). An added property is optional, or required with requiredSince. |
| Errors | MissingPositionsInDocumentTypePropertiesError (10411), JsonSchemaError (10101), both at registration |
properties declares every property a document of the type may hold. Each value is the property's schema: its type, its bounds and the Platform keywords that apply to it (see Property Schemas).
Rules at registration:
- A document type has 1 to 100 properties. A nested object has the same limit.
- A property name is 1 to 64 letters, digits or underscores (
^[a-zA-Z0-9_]{1,64}$). Protocol versions 1 to 13 also admitted-; from protocol version 14 it is refused. - Every property has a
position, a number. The top-level positions must run 0, 1, 2 and so on, with no gap and no number used twice (MissingPositionsInDocumentTypePropertiesError, 10411). A document is stored with its properties inpositionorder and without their names, so the numbers are what tie the stored bytes to the schema.
On update, a new property takes the next free position. It is optional unless it is listed in required with requiredSince set to the version the update creates. A property that already exists can never be removed, since stored documents may hold it.
additionalProperties
| Where | The top of a document type (required), and every property of type object |
| Value | false, the only value allowed |
| Since | protocol version 1 |
| On update | Fixed (10246) |
| Errors | JsonSchemaError (10101): a document holding a property its type does not declare |
A document holds only the properties its type declares. Writing additionalProperties: false says so; Platform accepts no other value.
required
| Where | The top of a document type. A property of type object has its own required for its members (see Objects). |
| Value | An array of names, none repeated: properties of the type, and the system timestamps and block heights |
| Default | Absent: every property is optional and no timestamp is recorded |
| Since | protocol version 1 |
| On update | May gain only a property the same update adds, annotated with requiredSince; may lose nothing (DataContractInvalidRequiredFieldsUpdateError, 10276) |
| Errors | JsonSchemaError (10101): a created or replaced document leaves out a required property |
required names two kinds of thing:
- The type's own properties. Every created or replaced document must hold them. A required property is also stored without the presence byte an optional one carries (unless it is transient), which is why the list is so hard to change later.
- System timestamps and block heights:
$createdAt,$updatedAt,$transferredAt, and theirBlockHeightandCoreBlockHeightforms. The writer does not supply these. Listing one tells the platform to record it on every document of the type; one that is not listed is never recorded. See System Properties.
Some keywords only work with a timestamp in the list: ttl needs $createdAt, and a time-range index needs the timestamp it buckets.
On a contract update, from protocol version 14:
- A property the update adds may be required if it carries
requiredSinceequal to the contract version the update creates. See requiredSince. - An existing optional property may not become required.
- Nothing may be removed from the list.
- A system timestamp or block height may not be added. So the set of values a type records is fixed once the type exists.
Each of these is refused with DataContractInvalidRequiredFieldsUpdateError (10276). The required list of a nested object is fixed (IncompatibleDocumentTypeSchemaError, 10246).
minProperties and maxProperties
| Where | The top of a document type; also on properties of type object |
| Value | An integer, 0 or more |
| Since | protocol version 1 (declared in the meta-schema from 12) |
| On update | Fixed (IncompatibleDocumentTypeSchemaError, 10246) |
| Errors | JsonSchemaError (10101) |
These are the JSON Schema keywords: a document must hold at least minProperties and at most maxProperties of its own properties. System properties are not counted. The DashPay profile above uses minProperties: 1 so that an empty profile cannot be written.
Meta-schema v0, which covered protocol versions 1 to 11, did not list these keywords at the top of a document type, but it did not refuse keys it did not know either. A contract of that time could use them, and its documents were validated against them: the DashPay contract, registered at protocol version 1, uses minProperties and dependentRequired. From protocol version 12 the meta-schema lists them and checks their values.
dependentRequired
| Where | The top of a document type; also on properties of type object |
| Value | An object mapping a property name to an array of property names |
| Since | protocol version 1 (declared in the meta-schema from 12) |
| On update | Entries, and names within an entry, may be removed, and so may the whole keyword; nothing may be added (IncompatibleDocumentTypeSchemaError, 10246) |
| Errors | JsonSchemaError (10101) |
The JSON Schema keyword: when a document holds the property named by a key, it must also hold every property in that key's array. In the example, a profile with an avatarUrl must also have an avatarHash and an avatarFingerprint. Removing an entry only lets more documents through, which is why removal is the only change allowed.
$comment and description
| Where | The top of a document type; also on any property |
| Value | A string |
| Since | protocol version 1 |
| On update | Free: may be added, changed or removed |
| Errors | none |
Notes for people reading the contract. Consensus does not act on them.
$schema and $defs
| Where | Added by the platform; a document type does not write them |
| Value | $schema: the document meta-schema's URL. $defs: the contract's schemaDefs. |
| Since | protocol version 1 |
| Errors | InvalidContractStructure (10231): a document type that writes either key |
When Platform reads a document type, it adds two keys before validating the schema:
$schema, the URL of the document meta-schema, so the schema is checked against it.$defs, holding the contract'sschemaDefs: definitions shared by every document type of the contract. A property then refers to one with$ref, as"$ref": "#/$defs/<name>".
A document type that writes either key itself is refused. Definitions live only at the contract level, in schemaDefs, and are checked as part of each document type: 1 to 100 of them, each named like a property and each a property schema. A contract update may add definitions but not remove them, and a change to a definition follows the update rules of the keywords it holds (IncompatibleDataContractSchemaError, 10213). See Contract-Level Keys and config.
Limits on the whole type
- Name. A document type's name, its key in
documentSchemas, is 1 to 64 letters, digits or underscores (InvalidDocumentTypeNameError, 10415). Up to protocol version 13 it could also hold-. - Nesting. The schema, with the definitions it reaches through
$ref, nests at most 256 levels of objects and arrays (DataContractMaxDepthExceedError, 10200). A$refthat does not resolve, or that leads back to itself, is refused (InvalidJsonSchemaRefError, 10207). - Known keys only. From protocol version 12 a key the meta-schema does not know is refused at the top of a document type (
JsonSchemaError, 10101). Before 12 such a key was accepted without being checked. The keys are listed in the overview.
See also
- Property Schemas, for what goes inside
properties - System Properties, for the timestamps
requiredcan record - requiredSince, for adding a required property in an update
- Evolving a Contract: Adding Required Fields
- Document Serialization, for how
positionandrequiredshape the stored bytes
Property Schemas
Each entry of a document type's properties is a property schema: JSON Schema (draft 2020-12), limited to the keywords in this chapter, plus three Platform keywords that say how a value is stored (position, byteArray and contentMediaType). The schema is checked when the contract is registered, and every created or replaced document is validated against it. Platform keywords with more to them, such as maxBytes, refersTo, distinctFrom, encryptedFor, generatedFrom, requiredSince and a typed array's items, have chapters of their own.
| Keyword | Applies to | On update |
|---|---|---|
type | every property | Fixed |
position | every property | Fixed |
minLength, maxLength, pattern, format | strings | Loosened or removed only |
minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf | integers and numbers | Loosened or removed only, keeping an integer's width; multipleOf fixed |
enum, const | any | enum may gain values; const may be removed |
byteArray, contentMediaType | byte arrays | Fixed |
minItems, maxItems, uniqueItems, contains | arrays | Loosened or removed only; contains fixed |
properties, required, additionalProperties, minProperties, maxProperties, dependentRequired | objects | Members may be added; the rest fixed, except dependentRequired may lose entries |
$ref | any | Fixed |
$id, $comment, description, examples | any | Free ($id may only be added) |
Each section gives the error a contract update gets for breaking its rules. Most are refused with IncompatibleDocumentTypeSchemaError (10246), from the comparison of the old and new schemas; a change to how a value is stored is refused with DocumentTypeUpdateError (40212).
How property schemas are checked
When a contract is registered or updated:
- The schema is checked against the document meta-schema. A keyword the meta-schema does not allow where it is written, or a value of the wrong shape, is refused with
JsonSchemaError(10101). - The parser reads each property's type and bounds, and refuses what the meta-schema cannot express (
InvalidContractStructure, 10231). - The schema is compiled for validating documents. A
patternthat is not a valid regular expression, or aformatthe validator does not know, is refused here (JsonSchemaError, 10101).
When a document is created or replaced:
- Every string and byte array value is at most 5120 bytes, whatever the schema allows (
DocumentFieldMaxSizeExceededError, 10417). From protocol version 14 a value nested more than 256 levels deep is refused as well (ValueError, 10103). - The document's properties are validated against the schema. Each failure is reported as a
JsonSchemaError(10101) naming the keyword and the property. - Platform's own checks, which JSON Schema cannot express, come next:
maxBytesandpropertyConstraintsamong them.
A transfer, a price update and a purchase carry no property values, so they are not validated against the schema again.
The schema also decides how each value is stored, since a stored document holds no property names or type tags:
| Property | Stored as |
|---|---|
integer | 1, 2, 4 or 8 bytes, chosen by its bounds (see Numbers) |
number | 8 bytes, a 64-bit floating point number |
boolean | 1 byte |
string | a length prefix, then the UTF-8 bytes |
byte array with minItems equal to maxItems | the bytes, with no prefix |
| any other byte array | a length prefix, then the bytes |
| identifier | 32 bytes |
object | a length prefix, then its members |
| typed array | an element count, then the elements (see Typed Arrays) |
An optional property adds one byte in front that says whether it is present. See Document Serialization for the exact encoding.
type
| Where | Every property; also the elements of a typed array |
| Value | One of "string", "integer", "number", "boolean", "object", "array" |
| Since | protocol version 1 |
| On update | Fixed (IncompatibleDocumentTypeSchemaError, 10246) |
| Errors | JsonSchemaError (10101): a document value of another type |
type is a single name. A list of types, and "null", are refused at registration: every stored value needs one known encoding.
An array is one of two things:
- a byte array, with
byteArray: true: a string of bytes, such as a hash or an identifier; - from protocol version 14, a typed array, with an
itemsschema: a list of values of one scalar type. See Typed Arrays.
An array that is neither is refused.
position
| Where | Every property, at every level. Not on the elements of a typed array. |
| Value | An integer, 0 or more |
| Since | protocol version 1 |
| On update | Fixed (10246) |
| Errors | MissingPositionsInDocumentTypePropertiesError (10411) at registration |
A document is stored with its values one after another and no property names. position is the property's place in that sequence.
- The top-level properties of a document type must use the positions 0, 1, 2 and so on, with no gap and no number used twice (10411). A top-level property without a
positionis refused. - The members of an object need a
positiontoo. Number them from 0 within the object; only the top level is checked for gaps. - A property that takes its schema from a
$refwrites itspositionnext to the$ref. - A property added by a contract update takes the next free position. An existing position can never change, or stored documents would be read in the wrong order.
Strings
| Keywords | minLength, maxLength, pattern, format |
| Where | Properties of type string, and the string elements of a typed array |
| Value | minLength, maxLength: an integer, 0 or more, counting characters. pattern: a regular expression. format: the name of a JSON Schema format, such as "uri" |
| Since | protocol version 1 |
| On update | maxLength may be raised or removed, and minLength lowered or removed. pattern and format may be removed. None of them may be added, and no other change is allowed (10246). |
| Errors | JsonSchemaError (10101) |
"username": {
"type": "string",
"minLength": 3,
"maxLength": 63,
"pattern": "^[a-zA-Z0-9_]+$",
"position": 0
}
A username is 3 to 63 letters, digits or underscores.
minLengthandmaxLengthcount characters, and a character takes 1 to 4 bytes in UTF-8. To cap the stored size, addmaxBytes. WhatevermaxLengthsays, no single string may exceed 5120 bytes (10417).patternis written in the syntax of Rust'sregexcrate, which has no lookaround and no backreferences. A pattern that does not compile is refused at registration (10101).formatis checked on every document. The validator knowsdate-time,date,time,email,idn-email,hostname,ipv4,ipv6,uriandregex. Any other format,uuidanduri-referenceincluded, is refused at registration (10101).- A string with
patternorformatmust declare amaxLengthof at most 50000, so that matching stays cheap. - A string used in an index needs a
maxLengthof at most 63 (InvalidIndexedPropertyConstraintError, 10205). See Indexes.
Numbers
| Keywords | minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf |
| Where | Properties of type integer or number, and such elements of a typed array |
| Value | A number. On an integer element of a typed array, minimum and maximum are integers. |
| Since | protocol version 1 |
| On update | maximum and exclusiveMaximum may be raised or removed, and minimum and exclusiveMinimum lowered or removed, unless that changes how an integer is stored (DocumentTypeUpdateError, 40212). None may be added, and multipleOf is fixed (10246). |
| Errors | JsonSchemaError (10101) |
"rating": { "type": "integer", "minimum": 1, "maximum": 5, "position": 2 }
A rating is a whole number from 1 to 5, and is stored in one byte.
A number is always stored in 8 bytes. An integer is stored in the smallest width its minimum and maximum allow, when the contract's config has sizedIntegerTypes on (the default; see Contract-Level Keys and config):
minimum | maximum | Stored as |
|---|---|---|
| 0 or more | up to 255 | 1 byte, unsigned |
| 0 or more | up to 65535 | 2 bytes, unsigned |
| 0 or more | up to 4294967295 | 4 bytes, unsigned |
| 0 or more | higher | 8 bytes, unsigned |
| below 0 | both bounds within -128 to 127 | 1 byte, signed |
| below 0 | both bounds within -32768 to 32767 | 2 bytes, signed |
| below 0 | both bounds within -2147483648 to 2147483647 | 4 bytes, signed |
| below 0 | otherwise | 8 bytes, signed |
With only a minimum, the integer takes 8 bytes, unsigned when the minimum is 0 or more. With only a maximum, it takes the unsigned width the maximum gives, so an integer that may be negative needs a minimum too. With neither, an enum of integers picks the width from its smallest and largest members; without one, the integer takes 8 bytes, signed. exclusiveMinimum and exclusiveMaximum do not affect the width. With sizedIntegerTypes off, every integer takes 8 bytes, signed.
Stored documents and index entries hold each integer at its width, so a contract update may not change the width or the sign. Raising maximum past the width, lowering minimum below 0, removing a bound, or adding an enum value outside the width is refused with DocumentTypeUpdateError (40212); so is turning sizedIntegerTypes on when it would change an existing integer's width. A change that keeps the width is accepted.
enum and const
| Where | Any property. enum also on the string, integer, number and boolean elements of a typed array; const never on elements. |
| Value | enum: an array of one or more values, none repeated. const: one value. |
| Since | protocol version 1 |
| On update | enum may gain values but not lose one; the keyword may be removed, not added. const may be removed, not added or changed (10246). An enum value that changes an integer's width is refused (DocumentTypeUpdateError, 40212). |
| Errors | JsonSchemaError (10101) |
"status": { "type": "string", "enum": ["open", "closed", "archived"], "position": 3 }
enum lists the values a property may take and const the one value it must take. Both only narrow what documents may hold, so an update may widen them (more enum values, or no keyword at all) but never narrow them, which could leave stored documents invalid. On an integer without bounds, enum also sets the stored width (see Numbers).
Byte arrays and identifiers
| Keywords | byteArray, contentMediaType |
| Where | Properties of type array, and the array elements of a typed array |
| Value | byteArray: true, the only value. contentMediaType: "application/x.dash.dpp.identifier" makes the byte array an identifier. |
| Since | protocol version 1 |
| On update | Fixed (10246). The length bounds follow the rules of Arrays, but a byte array may not switch between a fixed and a variable length, or change its fixed length (DocumentTypeUpdateError, 40212). |
| Errors | JsonSchemaError (10101) |
"authorId": {
"type": "array",
"byteArray": true,
"minItems": 32,
"maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 1
}
authorId is an identifier: the 32-byte id of an identity, a document, a contract or a token.
byteArray: truemakes an array a string of bytes. ItsminItemsandmaxItemscount bytes. When the two are equal the bytes are stored as they are; otherwise they carry a length prefix.contentMediaType: "application/x.dash.dpp.identifier"makes a byte array an identifier. It must come withbyteArray: true,minItems: 32andmaxItems: 32, and it may not carryuniqueItems. An identifier is shown in base58, and it is the kind of propertydistinctFromand mostrefersTotargets are declared on.- A byte array used in an index needs a
maxItemsof at most 255 (InvalidIndexedPropertyConstraintError, 10205). - On a typed array,
contentMediaTypebelongs on theitems, not on the array.
Arrays
| Keywords | minItems, maxItems, uniqueItems, contains |
| Where | Properties of type array. minItems and maxItems also on byte array elements of a typed array. |
| Value | minItems, maxItems: an integer, 0 or more. uniqueItems: a boolean. contains: a schema. |
| Since | protocol version 1 |
| On update | maxItems may be raised or removed and minItems lowered or removed (10246), within the byte array rule above (40212); a typed array keeps its maxItems. uniqueItems may be removed or set to false, not added (10246). contains is fixed (10246). |
| Errors | JsonSchemaError (10101) |
minItemsandmaxItemscount bytes on a byte array and elements on a typed array. A typed array must declaremaxItems, at most 1024.uniqueItems: trueon a typed array refuses a document that repeats an element. On a plain byte array it refuses a repeated byte. It is refused on an identifier, and on the elements of a typed array.containsis the JSON Schema keyword: at least one element must match the schema it holds.
Objects
| Keywords | properties, required, additionalProperties, minProperties, maxProperties, dependentRequired |
| Where | Properties of type object |
| Value | The same as at the top of a document type: see Document Shape |
| Since | protocol version 1 |
| On update | Members may be added, never removed; required and additionalProperties are fixed; dependentRequired may lose entries, not gain them (10246). minProperties and maxProperties are fixed (10246). |
| Errors | JsonSchemaError (10101) |
"records": {
"type": "object",
"properties": {
"identity": {
"type": "array",
"byteArray": true,
"minItems": 32,
"maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 0
}
},
"minProperties": 1,
"additionalProperties": false,
"position": 5
}
An object groups members under one property, as DPNS groups a name's records.
- An object declares
properties, 1 to 100 members each with aposition, andadditionalProperties: false, unless it takes its schema from a$ref. - Its
requirednames the members every value of the object must hold. It cannot change after the type exists, so a member added by an update is optional. - An object is stored with its members inline. An object cannot be indexed, but a member can: an index names it by its dotted path, such as
records.identity.
$ref
| Where | Any property, except the elements of a typed array |
| Value | "#/$defs/<name>": a definition in the contract's schemaDefs |
| Since | protocol version 1 |
| On update | Fixed (10246) |
| Errors | InvalidJsonSchemaRefError (10207) at registration |
$ref lets several document types share one schema, written once in the contract's schemaDefs:
"schemaDefs": {
"address": {
"type": "object",
"properties": {
"street": { "type": "string", "maxLength": 100, "position": 0 },
"city": { "type": "string", "maxLength": 50, "position": 1 }
},
"required": ["street", "city"],
"additionalProperties": false
}
}
A property of any document type of the contract then reads "shippingAddress": { "$ref": "#/$defs/address", "position": 2 }.
- Only local references, starting with
#, are allowed. The platform placesschemaDefsunder$defsin every document type (see$schemaand$defs). - A reference that does not resolve, or that leads back to itself, is refused (10207).
- The property keeps its own
position; the definition supplies everything else. - The
$refitself cannot change on update. The definition it points at can, under the rules of the keywords it holds (IncompatibleDataContractSchemaError, 10213).
Annotations
| Keywords | $id, $comment, description, examples |
| Where | Any property. $comment and description also on the elements of a typed array. |
| Value | $id: a string starting with #. $comment, description: a string. examples: an array of values. |
| Since | protocol version 1 |
| On update | $comment, description and examples: free. $id: may be added, not removed or changed (10246). |
| Errors | none |
Notes for people and tools reading the contract. They do not change what a document may hold.
See also
- Document Shape, for the keywords at the top of a document type
- Typed Arrays, for arrays of values
- maxBytes, for capping a string's size in bytes
- Document Serialization, for the stored form of every type
- Indexes, for the limits on indexed properties
- Error Codes
Typed Arrays
A typed array is a list property: a document holds any number of values, up to a limit, all of one simple type. A post's tags, the scores of a match or the members of a team are typed arrays. The items keyword makes an array a typed array and gives the schema every element follows. Before protocol version 14 an array property had to be a byte array.
| Where | A property of type array, at the top of a document type or inside an object, in place of byteArray: true |
| Value | The schema of one element: an integer, a number, a string, a boolean, a byte array or an identifier |
| Since | protocol version 14 |
| On update | items may be neither added nor removed (IncompatibleDocumentTypeSchemaError, 10246). The keywords inside it follow their own update rules, and a change to how an element is stored is refused (DocumentTypeUpdateError, 40212). |
| Errors | JsonSchemaError (10101); on elements, DocumentPropertyMaxBytesExceededError (10421), DocumentPropertyNotDistinctError (10419) and the reference errors of refersTo |
Example
"tags": {
"type": "array",
"minItems": 0,
"maxItems": 10,
"uniqueItems": true,
"items": { "type": "string", "minLength": 1, "maxLength": 32, "maxBytes": 64 },
"position": 3
}
A post holds up to 10 tags, none repeated, each 1 to 32 characters and at most 64 bytes.
The elements may also be identifiers that point at other documents. The moderation charters contract lists the reasons a moderation team may act on like this:
"reasons": {
"type": "array",
"minItems": 0,
"maxItems": 64,
"uniqueItems": true,
"items": {
"type": "array",
"byteArray": true,
"minItems": 32,
"maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "permanentDocument", "documentType": "reason" }
},
"position": 2
}
A charter names up to 64 distinct reason documents, and each one must exist when the charter is written.
How it works
On the array, minItems and maxItems count elements, and uniqueItems: true refuses a document that repeats one. A list that is too long or too short, repeats an element, or holds an element of the wrong type is refused with JsonSchemaError (10101).
On the elements, the items schema is checked for every element, as a property of that schema would be:
- the JSON Schema keywords: an element's
enum, bounds, length andpattern(JsonSchemaError, 10101); maxByteson string elements (DocumentPropertyMaxBytesExceededError, 10421), the error naming the element, such astags[2];distinctFromon identifier elements: every element must differ from the named property (DocumentPropertyNotDistinctError, 10419);refersToon identifier elements: each element is checked as a single reference would be, in list order, and the first one that fails refuses the write with that reference's error, naming the element (reasons[2]for the third). An empty or absent list checks nothing.
A replace re-checks the references of the elements the stored list did not hold, and every element when the reference's rules call for it. See References on the Elements for the details.
Storage. A typed array is stored inline in the document: a count of its elements, then each element encoded as a required property of the element's type would be. An identifier element takes 32 bytes, an integer element the width its bounds give it (see Numbers), a fixed-size byte array element its bytes, and a string element a length prefix and its bytes. Elements never carry a presence byte. The reasons list above is therefore one count byte and 32 bytes per reason. The 5120-byte limit on a single value applies to each element, not to the list.
When a document is read from JSON, identifier and byte array elements are converted from their string form element by element, as a single identifier or byte array is.
Rules at registration
The array:
- declares
itemsandmaxItems, and notbyteArray; - has a
maxItemsof at most 1024, and aminItemsno higher than itsmaxItems; - carries no
contentMediaType,refersTo,distinctFromormaxBytesof its own: these go on theitems.
The element schema (items):
- is written inline: a
$refis refused; - has a
typeofinteger,number,string,booleanorarray, and anarrayelement must be a byte array (byteArray: true). Objects and lists of lists are refused; - takes these keywords:
type,minimum,maximum,exclusiveMinimum,exclusiveMaximum,multipleOf,minLength,maxLength,pattern,format,minItems,maxItems(bytes, on a byte array element),enum,byteArray,contentMediaType,maxBytes,distinctFrom,refersTo,$commentanddescription. Anything else,position,const,uniqueItemsandexamplesincluded, is refused. A one-valueenumdoes whatconstwould, and an update can still widen it; - may carry an
enumonly on a string, integer, number or boolean element, and every member must be a value of the element's type; - on an integer element, has an integer
minimumandmaximum, the minimum no higher than the maximum; - may carry
refersToonly on an identifier element, and not with theidentityPublicKeytarget: its key id is a single sibling property, which cannot pair with many elements.
Where a typed array cannot be used:
- in an index (
InvalidIndexPropertyTypeError, 10206), or as an index-only type's terminal or entry payload: nothing is written per element; - on either side of a
whereentry in a reference; - as an operand of a propertyConstraints rule, other than in a
presentorabsenttest; - with
encryptedFor, which only a byte array takes.
References count against the limit of 256 per document at maxItems each: a list of up to 64 references counts as 64. An immutable property may not hold a list of deletableDocument references (see Mutability).
A meta-schema violation is refused with JsonSchemaError (10101) and the other rules with InvalidContractStructure (10231).
On update
itemsmay not be added or removed, so a byte array cannot become a typed array or the reverse (10246).- Inside
items, each keyword follows its own update rule (see Property Schemas): for examplemaxLengthandmaxBytesmay be raised, andenummay gain values.refersToanddistinctFromare fixed. - A change to how an element is stored is refused (
DocumentTypeUpdateError, 40212): an integer element whose width or sign would change (by its bounds or itsenum), or a byte array element that would switch between fixed and variable size or change its fixed size. - On the array,
maxItemsmay be raised, up to 1024 and within the reference limit, andminItemslowered.uniqueItemsmay be removed, not added.
See also
- Typed Arrays and References on the Elements in the Documents chapter
- Property Schemas, for the keywords an element takes
- References (refersTo), distinctFrom and maxBytes, which apply to every element
- Document Serialization, for the stored form
System Properties
Every document carries a few values the platform manages rather than the writer: its id, its owner, and, on some types, its revision, its creator, the times it was created, updated and transferred, and who last moderated it and when. Their names start with $. A document type does not declare them in properties. It names them where it wants to use them: in required, to have a timestamp recorded, in indices, to query by them, and in the keywords that accept one, such as a reference's where.
| Property | Holds | Recorded |
|---|---|---|
$id | the document's id | always |
$ownerId | the identity that owns the document now | always |
$revision | how many times the document has changed, plus one | on types whose documents can be replaced, transferred or sold, or that keep fields for their moderators |
$createdAt, $updatedAt, $transferredAt | block times of the creation, last update and last transfer | when listed in required |
$createdAtBlockHeight and the other heights | Platform and Core block heights of the same events | when listed in required |
$creatorId | the identity that created the document | on types whose documents can be transferred or sold |
$moderatedAt, $moderatedBy | the block time and the moderator of the last write of the fields only moderators write | on types that keep such fields, once a moderator writes them |
Example
"listing": {
"type": "object",
"documentsMutable": true,
"transferable": 1,
"tradeMode": 1,
"indices": [
{ "name": "byCreator", "properties": [{ "$creatorId": "asc" }, { "$createdAt": "asc" }] },
{ "name": "byOwner", "properties": [{ "$ownerId": "asc" }, { "$updatedAt": "asc" }] }
],
"properties": {
"title": { "type": "string", "minLength": 1, "maxLength": 100, "position": 0 }
},
"required": ["$createdAt", "$updatedAt", "$transferredAt", "title"],
"additionalProperties": false
}
Every listing records when it was created, last updated and last transferred, because required lists the three times. Listings can be replaced, transferred and sold, so each one also carries a revision and the id of its creator, and the two indexes find them by who made them and by who holds them now.
$id
| Where | Every document |
| Value | An identifier: 32 bytes |
| Recorded | Always |
| Since | protocol version 1 |
| Errors | InvalidDocumentTransitionIdError (10405): a create whose id is not the one derived for it. SystemPropertyIndexAlreadyPresentError (10208): an index that names $id. |
A document's id is derived from the contract id, the owner's id, the document type's name, entropy chosen by the client and, from protocol version 14, the identity contract nonce of the create transition. Consensus derives it again for every create and refuses a transition that carries another. The id never changes. See Document ID Generation.
Documents are already stored by id, so an index may not name $id. A reference's where may name it as a key, the referenced side, and findBy names it to find the document holding a list (see References).
$ownerId
| Where | Every document |
| Value | An identifier: the id of an identity |
| Recorded | Always |
| Since | protocol version 1 |
| Errors | DocumentOwnerIdMismatchError (40102): a replace, transfer, price update or delete signed by an identity that does not own the document |
The owner is the identity that created the document, until a transfer or a purchase hands it to another. Only the owner may replace, transfer, reprice or delete it with a document transition. A document can also leave by other paths, a moderators' deletion or an expired ttl; see Deletion.
$ownerId may be indexed, and several keywords read it, where it usually stands for the writer of the document:
distinctFrom:"$ownerId"makes a property differ from the owner.propertyConstraints: a rule may compare an identifier property with$ownerId.- References: a
whereentry, afindBysource andidentityPropertymay name it, andownerRefersTochecks the owner itself. encryptedFor:"recipient": "$ownerId"marks a message the writer encrypts to themself.
$revision
| Where | Documents of a type whose documents can be replaced (documentsMutable, true by default), transferred (transferable: 1) or sold (tradeMode: 1), or that keeps fields only its moderators write (moderatorAbilities.changeFields), even when its documents cannot be replaced |
| Value | An integer, from 1 |
| Recorded | On those types; absent on every other |
| Since | protocol version 1 |
| Errors | InvalidDocumentRevisionError (40106): a transition whose revision is not the stored one plus one |
A new document has revision 1. Every replace, transfer, price update and purchase raises it by one, and the transition must state the new revision: the stored revision plus one. A transition built against an older copy of the document is refused, rather than silently overwriting a newer one. A moderator's change of the fields a type keeps for its moderators raises it by one too; the moderation transition states no revision, the platform sets it, and an owner's replace built before the change is refused. A type whose documents can never change after creation carries no revision. $revision is not one of the system properties an index may name.
Timestamps
| Properties | $createdAt, $updatedAt, $transferredAt |
| Value | A block time, in milliseconds since the Unix epoch |
| Recorded | Only when listed in the document type's required |
| Since | protocol version 1 |
| On update | The set is fixed: a contract update may not add one to required or remove one (DataContractInvalidRequiredFieldsUpdateError, 10276) |
The platform sets these from the block that processes the transition; the writer never supplies them. A timestamp that is not in required is never recorded, and documents of the type do not have it.
| Event | Sets |
|---|---|
| Create | every listed timestamp, so $updatedAt and $transferredAt start at the creation time |
| Replace | $updatedAt |
| Price update | $updatedAt |
| Transfer | $transferredAt |
| Purchase | $transferredAt |
Timestamps may be indexed. Some keywords need one in required, since they read it:
ttlcounts from$createdAt.moderatorAbilities.deleteWithincounts from$updatedAt, or from$createdAton a type that does not record$updatedAt(see Deletion).- A time-range index needs the timestamp it buckets.
Block heights
| Properties | $createdAtBlockHeight, $updatedAtBlockHeight, $transferredAtBlockHeight, $createdAtCoreBlockHeight, $updatedAtCoreBlockHeight, $transferredAtCoreBlockHeight |
| Value | The BlockHeight forms: the Platform block height. The CoreBlockHeight forms: the Core chain height recorded with that block. |
| Recorded | Only when listed in the document type's required |
| Since | protocol version 1 |
| On update | The set is fixed, as for the timestamps (10276) |
The same three events as the timestamps, measured in blocks instead of time. Each is set on the same events as the timestamp of its name, and may be indexed.
$creatorId
| Where | Documents of a type that sets transferable: 1 or tradeMode: 1, in a contract of format 1 whose config is version 1 or later |
| Value | An identifier: the id of the identity that created the document |
| Recorded | On those types, from protocol version 10 |
| Since | protocol version 10 |
| Errors | UndefinedIndexPropertyError (10209): an index naming $creatorId on a type that does not record it |
On a type whose documents can change hands, $ownerId follows the document while $creatorId stays with the identity that created it: a transfer or a purchase never changes it. On a type whose documents never change hands the creator is always the owner, and no $creatorId is recorded. Neither is it on a contract of format 0 or with a config of version 0, whatever its types allow.
$creatorId is not written in required or properties. It may be named:
- in an index;
- as the key of a reference's
whereentry (the referenced side), and as a key reference'sidentityProperty; - by
creatorRefersTo, which checks the creator. A type that does not record creators may not declare it (InvalidContractStructure, 10231).
$moderatedAt and $moderatedBy
| Where | Documents of a type that lists moderatorAbilities.changeFields |
| Value | $moderatedAt: a block time, in milliseconds since the Unix epoch. $moderatedBy: an identifier, the id of the moderator. |
| Recorded | Once a moderator of the contract writes the fields the type keeps for its moderators; absent until then |
| Since | protocol version 14 |
| Errors | InvalidContractStructure (10231): an index naming either on a type that lists no changeFields, or in a unique index. UndefinedIndexPropertyError (10209): an index naming either before protocol version 14. |
The two record the last time a moderator wrote the fields only moderators write, and who did. They are set together, from the block that processes the write, and never by the writer:
| Event | Sets both |
|---|---|
A moderator's field change (changeDocumentFields) | to the block's time and the moderator who signed it |
| A create that sets such a field, by a document owner who moderates the contract | to the block's time and the owner |
| A replace that changes, adds or removes such a field, by an owner who moderates | to the block's time and the owner |
Nothing else moves them: a replace that leaves those fields as they were, a transfer, a purchase and a price update keep them, and a restore puts them back as they were. They do not change $updatedAt, which stays the owner's. A document no moderator has written carries neither, and a type that keeps no fields for its moderators never records them.
Both may be indexed, on a type that lists changeFields, so an application can find what its moderators handled, by whom and in what order:
"indices": [
{ "name": "byModerator", "properties": [{ "$moderatedBy": "asc" }, { "$moderatedAt": "asc" }] }
]
A unique index may not name them: a moderator's change would be refused because another document holds the same stamp. They are not written in required or properties, and no other keyword reads them.
See also
- What Lives Inside a Document, for the fields of a document
- Document Shape, for the
requiredlist that records timestamps - Document Serialization, for where each system property sits in the stored bytes
- Creation, Transfers and Trading, for the flags that decide
$revisionand$creatorId - Moderator Abilities, for the fields whose writes
$moderatedAtand$moderatedByrecord
requiredSince
requiredSince lets a contract update add a property that every new document must hold. Documents already stored were written without it and stay valid: the property is required only of documents written under the contract version the annotation names, or a later one. Reach for it when an app needs a new mandatory field and the contract is already in use.
| Where | A top-level property listed in the document type's required |
| Value | A contract version: an integer from 1 to 4294967295 |
| Default | Absent: a property listed in required is required of every document |
| Since | protocol version 14 |
| On update | May only appear on a property the update adds, equal to the contract version the update creates (DataContractInvalidRequiredFieldsUpdateError, 10276). An existing annotation may not be added, changed or removed (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | DataContractInvalidRequiredFieldsUpdateError (10276), InvalidContractStructure (10231), IncompatibleDocumentTypeSchemaError (10246), all at registration; JsonSchemaError (10101) for a new document without the property |
Example
Version 2 of a contract has a post type with one property, text. The update to version 3 adds a required language:
"post": {
"type": "object",
"properties": {
"text": { "type": "string", "maxLength": 280, "position": 0 },
"language": { "type": "string", "minLength": 2, "maxLength": 8, "requiredSince": 3, "position": 1 }
},
"required": ["text", "language"],
"additionalProperties": false
}
Every post created or replaced under version 3 or later must have a language. Posts written under versions 1 and 2 have none, and remain valid as they are.
How it works
From protocol version 14, every create and replace stamps the document with the version of the contract it was written under. A transfer or a purchase keeps the stamp the document had, since it does not rewrite the document's properties. See the contract version stamp.
- New writes are held to the new schema. A create must include the property. A replace sends the whole document again, so replacing a document written before the update must add the property too; the document is then stamped with the current version. Documents catch up one at a time, as they are replaced.
- Older documents are left alone. A document stamped below the property's
requiredSincemay lack it. It can still be read, transferred, sold and deleted. A document last written before protocol version 14 has no stamp, and counts as older than every annotation. - Storage follows the stamp. A required property is stored without the presence byte an optional one carries. A property with
requiredSinceis stored as required in documents stamped at or above its version, and as optional in older ones. The latest contract alone therefore tells how to read every stored document. - Readers must expect gaps. An app that reads the type should handle documents without the property: every document stamped below the annotation may lack it.
- No index on it. A contract update cannot add an index to an existing document type (see Indexes), so a property added this way cannot be indexed on that type.
Rules at registration
requiredSincesits only on a top-level property, and only on one listed inrequired. A nested property, or one that is not required, is refused (InvalidContractStructure, 10231). It cannot go on the elements of a typed array.- The value is never higher than the contract's own version (10276): a property cannot be scheduled to become required later.
- On a new contract, which is version 1, the value may only be 1 (10276).
On an update to an existing document type, which always creates the version one above the current one:
- A property the update adds may be required only if it carries
requiredSinceequal to that new version. Without the annotation, or with any other value, the update is refused (10276). - An existing property may not become required, with or without the annotation (10276).
- No property may leave
required(10276). - A property's existing
requiredSincemay not be changed or removed, and one may not be added to an existing property (IncompatibleDocumentTypeSchemaError, 10246).
A document type that the update adds has no older documents. Every requiredSince in it must still equal the version the update creates (10276).
See also
- Evolving a Contract: Adding Required Fields
- The contract version stamp, for how the stamp decides each property's layout
- Document Shape, for the other rules of
required
transient
transient lists properties that a transition must carry but a stored document never holds. The value is validated with the rest of the document, can be read by the checks that run on the transition, and is then dropped. Reach for it when a write has to prove something with a value that should not be kept, such as a secret salt.
| Where | The top of a document type |
| Value | An array of names of top-level properties |
| Default | Absent: every property is stored |
| Since | protocol version 1. A replace drops the values from protocol version 14. |
| On update | Fixed (IncompatibleDocumentTypeSchemaError, 10246). The list is compared as a set, so reordering or repeating a name is no change. |
| Errors | InvalidContractStructure (10231) at registration |
Example
The DPNS domain type, trimmed to two properties:
"domain": {
"type": "object",
"documentsMutable": false,
"properties": {
"label": { "type": "string", "minLength": 3, "maxLength": 63, "position": 0 },
"preorderSalt": {
"type": "array",
"byteArray": true,
"minItems": 32,
"maxItems": 32,
"description": "Salt used in the preorder document",
"position": 1
}
},
"required": ["label", "preorderSalt"],
"transient": ["preorderSalt"],
"additionalProperties": false
}
Registering a name takes two steps. A preorder document first commits to a hash of the salt and the name, and the domain document then reveals both. Every domain create must carry its 32-byte preorderSalt, and DPNS checks it against the preorder. Once the domain is created the salt has done its job, and the stored domain does not hold it.
How it works
- Checked like any other value. A transient value is on the transition when the document is validated, so the schema holds it to its rules: a required transient property must be present, and its bounds apply.
maxBytesanddistinctFromapply to it too, and so do checks such as the DPNS one above. - Dropped before storing. After validation, a create drops the transient values before the document is written. From protocol version 14 a replace drops them the same way. Before 14 a replace stored the values it carried, so a replaced document could hold values its create had dropped.
- Dropped by top-level name. Only top-level properties can be transient. When a transient property is an object, the whole object is dropped with everything in it.
- Never stored, never found. No stored document holds a transient value, so a query cannot find one and a later reader cannot see it.
- Sent again on replace. A replace is validated as a whole document, so a required transient property must be carried by every replace, not only the create.
- Storage. A transient property is stored as optional, with a presence byte, even when it is required: in a stored document it is always absent.
Rules at registration
From protocol version 14, a document type is refused (InvalidContractStructure, 10231) when:
- an entry does not name a top-level property of the type. A nested path, a system property or an undeclared name is refused; to drop a nested value, list the object around it;
- an index reads a transient property, or a property inside a transient object. Every stored document would lack the value, so the index could find nothing and a unique index would enforce nothing;
- a reference reads one where the value would have to be stored: a
refersTofindBymay not read one on either side (the params of afindByfunction may, being read from the create), awheremay not name one on its referenced side, a key reference may not store its key id with a transient identity, and aninListreference may not find its list's document through one. The referring side of awhereentry may be transient: it is checked on the transition; encryptedFornames one as its recipient or key id;generatedFromsits on one or names one as a param;immutablelists one. A transient property is always absent from the stored document, so every replace that carries it would count as changing it;- a
propertyConstraintsrule reads one; - the type is
indexOnlyand declares any transient property.
Before protocol version 14 none of these was checked.
On a contract update the list is fixed. It decides which values stored documents hold and how every property is encoded, so documents written under one list could not be read under another.
See also
- Transient Properties in the Documents chapter
- Document Serialization, for the presence byte a transient property always takes
- Document Shape, for
required - References (refersTo), encryptedFor, generatedFrom, Mutability, propertyConstraints and Index-Only Types, whose rules refuse transient properties
Mutability
These two keywords decide what a replace may change once a document exists. A replace is the transition an owner sends to overwrite a document with a new version of it. documentsMutable turns replaces on or off for the whole document type. immutable freezes chosen properties while the rest of the document stays editable: from the moment the document is created, or only while a condition holds, such as five minutes after creation, once the document is published, or once a value has been filled in.
Neither governs deletion, transfers or trading: see Deletion and Creation, Transfers and Trading.
documentsMutable
Whether the owner of a document may replace it. Set it to false for records that must never change after they are written: votes, receipts, name registrations.
| Where | document type |
| Value | boolean |
| Default | the contract config's documentsMutableContractDefault, which is true unless the contract says otherwise |
| Since | protocol version 1 |
| On update | Fixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | InvalidDocumentTransitionActionError (10404) for a replace of a type set to false |
Example
"vote": {
"type": "object",
"documentsMutable": false,
"canBeDeleted": false,
"properties": {
"proposalId": {
"type": "array",
"byteArray": true,
"minItems": 32,
"maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 0
},
"choice": { "type": "string", "enum": ["yes", "no", "abstain"], "position": 1 }
},
"required": ["proposalId", "choice", "$createdAt"],
"additionalProperties": false
}
A vote is cast once and stays as cast: no replace is accepted, and with canBeDeleted: false its owner cannot take it back either.
How it works
- With
true, the document's owner may replace it. The replace carries the whole new document, which is validated against the schema as a create is, and a$revisionone higher than the stored one (InvalidDocumentRevisionError, 40106). Anyone other than the owner is refused (DocumentOwnerIdMismatchError, 40102). When the type lists$updatedAtinrequired, the replace sets it to the block's time, and likewise$updatedAtBlockHeightand$updatedAtCoreBlockHeightto the block heights. - With
false, every replace is refused (InvalidDocumentTransitionActionError, 10404). - A type whose documents cannot be replaced may still let them be transferred or sold (
transferable,tradeMode). Such a document changes owner and price, but its owner can never edit its properties. The DPNSdomaintype works this way. - A document stores a
$revisionwhen its type allows a replace, a transfer or trading, or keeps fields its moderators write (moderatorAbilities.changeFields), whose changes raise it even on a type whose documents cannot be replaced. A type that allows none of them stores none. See System Properties.
Rules at registration
- A contested index needs a type whose documents cannot be replaced (
ContestedUniqueIndexOnMutableDocumentTypeError, 10248). See Contested Indexes. - An
indexOnlytype must setdocumentsMutable: false. See Index-Only Types. immutableis only accepted when the type's documents are mutable (InvalidContractStructure, 10231).moderatorAbilities.deleteWithinon a mutable type needs$updatedAtinrequired. See Deletion.
immutable
The top-level properties a replace may not change, on a type whose documents can otherwise be replaced. Each entry is either a property name, frozen when the document is created, or a property with a condition, frozen for any replace the condition holds for. Reach for the first when part of a document is a commitment: the shop an order was placed with, the author of a post. Reach for the second when a value may change for a while and then must stand: a post's text for five minutes after it is published, an article's body once it is out of draft, a tracking code once it is filled in.
| Where | document type, on a type with documentsMutable: true |
| Value | array whose entries are a top-level property name, or { "property": <name>, "when": <condition> }; each property listed once |
| Default | empty: every property may change |
| Since | protocol version 14 |
| On update | May gain entries, with a condition or without. A property listed with a condition keeps it, or loses it to be listed without one. A property listed without a condition stays, and no property is dropped (DocumentTypeUpdateError, 40212) |
| Errors | DocumentImmutablePropertyChangedError (40128) for a replace that changes, adds or removes a frozen property |
Example
"post": {
"type": "object",
"documentsMutable": true,
"properties": {
"author": { "type": "string", "maxLength": 63, "position": 0 },
"text": { "type": "string", "maxLength": 500, "position": 1 },
"status": { "type": "string", "enum": ["draft", "published"], "position": 2 },
"body": { "type": "string", "maxLength": 5000, "position": 3 },
"trackingCode": { "type": "string", "maxLength": 40, "position": 4 },
"pinned": { "type": "boolean", "position": 5 }
},
"required": ["author", "text", "status", "$createdAt", "$updatedAt"],
"immutable": [
"author",
{
"property": "text",
"when": { "greaterThan": [{ "subtract": ["$updatedAt", "$createdAt"] }, 300000] }
},
{
"property": "body",
"when": { "equal": ["$old.status", { "const": "published" }] }
},
{ "property": "trackingCode", "when": { "present": "$old.trackingCode" } }
],
"additionalProperties": false
}
author never changes. text can be corrected for five minutes after the post is created: during a replace $updatedAt is the replace's block time, so the difference is the post's age in milliseconds. body stays editable while the stored post is a draft, including in the replace that publishes it, and is frozen after that. trackingCode can be filled in by one replace while the stored post has none, and is frozen once it holds one. pinned is not listed, so it can change at any time.
How it works
- On every replace, each property that differs from the stored document is checked against the list. A property listed by name is refused with
DocumentImmutablePropertyChangedError(40128). A property listed with a condition is refused with the same error when its condition holds for this replace. "Differs" covers a changed value, a value the stored document did not have, and a value the replace leaves out. - A condition takes the grammar of a
propertyConstraintsrule: comparisons, arithmetic,in,present,absent,anyOf,allOf,not,ifThen,ifThenElse, the system times and heights the type records, and$ownerId. It is judged, as a rule judges a replace, on the document the replace writes: its properties as the replace sets them, the stored$createdAtand$transferredAt, and the replace's block as$updatedAt. - A path starting with
$old.reads the stored document instead:$old.statusis the status before the replace. Only a condition ofimmutable, or a type'sretractedWhen, may read it, and only a schema property through it. - A property is frozen while its condition holds. If what the condition reads can change back, the property can become editable again: with
bodyfrozen by$old.status, a replace settingstatusback todraftstill reads the storedpublished, sobodystays frozen in that replace, and the next replace, reading the storeddraft, may change it. For a freeze that lasts, list what the condition reads as well, or use a condition that cannot turn back, as a document's age or a filled-in value cannot. - A condition is evaluated only when its property changes. A condition that faults, dividing by zero or overflowing, counts as holding: a fault never frees a property.
- Values are compared by their data, not their bytes: the order of an object's members and the width an integer is stored in do not count as changes. Freezing an object freezes everything inside it.
- One change is always allowed: a replace may clear a
deletableDocumentreference by id, listed without a condition, once the document it points to has been deleted. Every replace checks such a reference again, so without this the document could never be replaced again. See References. - A replace judged by
checkTxshortly before a time condition starts holding may still land in a block after that, and is then refused there and pays its fee, like any other state check. - Transfers, price updates and purchases carry no property values, so the list does not affect them.
Rules at registration
All refusals below are InvalidContractStructure (10231). The shape of the list, and what each condition reads, are checked on every parse. The other rules are checked when a contract is registered or updated.
- Only on a type whose documents are mutable. On a type with
documentsMutable: falseevery property is already frozen. - Every entry is a string or an object with exactly
propertyandwhen, and no property is listed twice. - Every listed property is a declared top-level property. System properties (
$ownerId,$createdAtand the rest) are refused, since the platform manages them. Nested paths such asmeta.authorare refused: list the object that contains them. - No listed property may be
transient: it is never stored, so every replace that supplies it while it is frozen would count as a change. - A condition reads what a
propertyConstraintsrule may read of the type: declared properties of the right kind, neither transient nor inside a transient object, and system times and heights the type lists inrequired. It stays within the node limit of a rule, and lists no condition twice. - A condition may not read a
countOforsumOftotal: it reads the document alone, so a replace judges it without reading state. immutableAllowSetting, which earlier builds used to let a frozen property be filled in once, is refused and names the replacement:{ "property": "p", "when": { "present": "$old.p" } }.- An immutable property may not hold a
deletableDocumentreference that a replace could not clear: a typed array of them, one inside an object, or one found byfindBywhose key no function computes. A single reference by id held directly by the property is allowed, but only without a condition: once cleared, a replace the condition leaves free could set it to another document. - On a type whose documents can be transferred or traded, an immutable property may not hold a
contractreference whosecontractRequirementshas anownerrequirement: after a change of owner the new owner could neither meet it nor repoint it. - A listed property, with a condition or without, may not be one of the fields only moderators write (
moderatorAbilities.changeFields). - A property listed with a condition is not fixed once written, so a
findBykey, aninListlist, a value read beside afindByfunction, or a value another type indexes through a reference may not rely on it. See References.
On update
What immutable freezes may only tighten. A property may be added, with a condition or without, and documents already stored are held to it from then on. A property listed with a condition may lose it, which freezes it whatever the condition said. A condition cannot change: whether one condition holds wherever another does cannot be told in general. A property listed without a condition stays, and no property is dropped, since documents were written on the promise that those properties would not change. The comparison is of the parsed entries, so reordering them is no change.
See also
- Immutable Properties on Mutable Document Types, for how the replace compares values
- Deletion, Creation, Transfers and Trading and History, the other keywords on what may happen to a document
- System Properties, for
$revisionand$updatedAt - transient and References, for the properties
immutablerefuses - propertyConstraints, for the grammar of a condition
- Contract Keywords, for how the summary tables read
Deletion
A document can leave the state three ways: its owner deletes it, the contract's moderators delete it, or the platform deletes it when its time to live runs out. canBeDeleted rules the first, and deleteConstraints can narrow it to the documents that meet its rules; moderatorAbilities.delete, moderatorAbilities.deleteWithin and moderatorAbilities.deleteSettled the second, and ttl the third (see Time To Live). Each is independent of the others: a type may let moderators remove what its authors cannot retract, or expire documents that nobody may delete by hand.
canBeDeleted
Whether a document's owner may delete it. Set it to false for records that other documents or other people rely on staying put, and to "onlyWhenConsumed" for records only a create that consumes them may remove.
| Where | document type |
| Value | boolean, or "onlyWhenConsumed" |
| Default | the contract config's documentsCanBeDeletedContractDefault, which is true unless the contract says otherwise |
| Since | protocol version 1; "onlyWhenConsumed" protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212), except that a type which keeps history before and after the update may change it from true to false. Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | InvalidDocumentTransitionActionError (10404) for a delete of a type set to false or "onlyWhenConsumed", or, from protocol version 14, of a type that keeps history; DocumentOwnerIdMismatchError (40102) for a delete by anyone but the owner |
Example
"comment": {
"type": "object",
"canBeDeleted": true,
"properties": {
"postId": {
"type": "array",
"byteArray": true,
"minItems": 32,
"maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 0
},
"text": { "type": "string", "maxLength": 500, "position": 1 }
},
"required": ["postId", "text"],
"additionalProperties": false
}
A commenter may take a comment down at any time. Since true is the usual default, the key could be left out; writing it makes the intent plain.
How it works
- The owner deletes a document with a delete transition that names its id. Anyone else is refused (
DocumentOwnerIdMismatchError, 40102), and a document that does not exist isDocumentNotFoundError(40101). - The owner is refunded the part of the document's storage fee that has not yet been paid out to past epochs. A document of a type with a
ttlrefunds nothing. See Refunds. - A delete may carry a token cost or an action fee, like any document action. See Token Costs and Action Fees.
- An identity that is banned or suspended on a moderated contract may still delete its own documents. See Contract Moderation. On a type set to
falseit can retract them instead, when the type declaresretractedWhen. falsebinds only the owner. The contract's moderators, when the type allows them, and the platform, when the type has attl, still delete such documents."onlyWhenConsumed"binds the owner asfalsedoes, and lets a create consume the document (see below).- Drive never deletes a document whose type keeps history (
documentsKeepHistory). From protocol version 14 a delete of such a document is refused with 10404 whatevercanBeDeletedsays; before it, the delete failed inside Drive as an internal error. - Documents of an
indexOnlytype are deleted with an index-only delete transition that carries their values, since there is no stored row to name by id. A delete by id of such a document is refused (10404). See Index-Only Types.
Rules at registration
- From protocol version 14, a type with
documentsKeepHistory: truemust setcanBeDeleted: false(InvalidContractStructure, 10231). The default istrue, so it has to be written out. A contract registered earlier with both flags on stays readable, but its next update is checked like a new contract, so that update must turncanBeDeletedoff on the type. That is the one change tocanBeDeletedan update may make. - For references, a type whose owner may delete its documents is deletable: a
permanentDocumentreference,inListincluded, may not point at it (ReferencedDocumentTypeDeletableError, 40122), and adeletableDocumentreference may. See References. "onlyWhenConsumed"is refused on a type that keeps history or isindexOnly(InvalidContractStructure, 10231): the storage layer never deletes a document that keeps history, and anindexOnlytype has no stored row a reference finds, so nothing could consume either.
Deleted only when consumed
"onlyWhenConsumed" says the owner can not delete a document, as false does, but a create of the same contract whose refersTo declares consume can. Its owner never removes it: it leaves state when a create consumes it, or, as with false, when the contract's moderators delete it where the type allows them (moderatorAbilities.delete) or the platform deletes it when its ttl passes.
"preorder": {
"type": "object",
"documentsMutable": false,
"canBeDeleted": "onlyWhenConsumed",
"indices": [
{ "name": "saltedHash", "properties": [{ "saltedDomainHash": "asc" }], "unique": true }
],
"properties": {
"saltedDomainHash": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 0
}
},
"required": ["$createdAtBlockHeight", "saltedDomainHash"],
"additionalProperties": false
}
A preorder stays until the name registration that reveals it consumes it (the preorderSalt declaration in Commit and reveal). Its owner can not take it back with a delete.
- A delete transition of such a document is refused (
InvalidDocumentTransitionActionError, 10404), as forfalse. - A consume deletes it as a delete by its owner would, its storage refunded to the owner.
- For references the type is deletable, as with
true: its documents can leave state without a record. ApermanentDocumentreference to it is refused (40122), and so is amoderatedDocumentone (40143), even when the type also lets its moderators delete with records; adeletableDocumentreference is the one that points at it. - Fixed on update: an update may neither set nor remove it (
DocumentTypeUpdateError, 40212). Setting it would let documents leave state under thepermanentDocumentreferences made to a type that promised they never would; removing it would leave the references that consume its documents nothing to delete. - Before protocol version 14 the meta-schemas accept only a boolean, so a contract carrying the string is refused (
JsonSchemaError, 10101).
retractedWhen
The replace that retracts a document: what a banned or suspended author may still do on a type whose documents it can not delete. A barred identity can write nothing new, but deleting what it wrote is never refused. On a type set to canBeDeleted: false there is no delete, and the author's only way to take a document back is a replace that blanks it, which a bar refuses like any other write. retractedWhen names that replace, and lets it through.
| Where | document type |
| Value | one condition, in the grammar of an immutable entry's when |
| Default | none: a barred author's replaces are all refused |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212): an update may not add it, remove it or change it |
| Errors | ContractUserBannedError (41107) or ContractUserSuspendedError (41108) for a barred author's replace whose written document does not meet the condition |
Example
"post": {
"type": "object",
"canBeDeleted": false,
"properties": {
"text": { "type": "string", "maxLength": 500, "position": 0 },
"deleted": { "type": "boolean", "position": 1 }
},
"additionalProperties": false,
"retractedWhen": { "present": "deleted" },
"propertyConstraints": {
"retractedIsBlank": { "anyOf": [{ "absent": "deleted" }, { "absent": "text" }] }
},
"immutable": [
{ "property": "deleted", "when": { "present": "$old.deleted" } }
]
}
Replies point at posts, so a post stays in place. Its author retracts one by replacing it with { "deleted": true }. The three keywords split the work. retractedWhen says that a post carrying deleted is retracted, so a banned author may still write one. retractedIsBlank says that a retracted post carries no text, for every author. The immutable entry says that a post stays retracted once it is. A banned author's edit of its text is refused with 41107, and so is a replace that drops deleted again.
How it works
- Only replaces are let through, and only those of a banned or suspended owner are judged. An identity that is not barred replaces its documents under the type's other rules alone.
- The moderation gate lets every replace of a barred owner on the type through. Once the stored document is fetched, the transformer judges the condition on the document the replace writes, as an
immutablecondition is judged: its properties as the replace sets them, the replace's block as$updatedAt, and the stored document under$old.. A replace for which the condition does not hold is refused with the bar's error and its nonce bumped, in a block and in the mempool alike. A condition that faults, dividing by zero or overflowing, counts as not holding: a fault never lifts a bar. - The condition only picks the replaces a barred owner may make. Every other rule of the type still judges them: the schema,
propertyConstraints,immutable, references, token costs and action fees. What a retracted document may hold is for those rules to say. Without a rule likeretractedIsBlankabove, a barred author could write anything into a document that meets the condition. - Creates, transfers, purchases and price updates by a barred owner are refused as before, and deletions pass as before.
Rules at registration
All refusals below are InvalidContractStructure (10231), checked on every parse.
- Only on a type whose documents are mutable: without a replace there is nothing to retract with.
- Only on a contract that keeps a banlist or a suspension list: otherwise no owner is ever barred.
- The condition reads what an
immutableentry'swhenmay read: declared properties of the right kind, neither transient nor inside a transient object, the system times and heights the type lists inrequired, and the stored document through$old.. It reads nocountOforsumOf. When a contract is registered or updated, it also stays within the node limit of a rule and lists no condition twice.
deleteConstraints
Rules the stored document must meet for its owner to delete it. canBeDeleted: true lets an owner delete any of its documents at any time; deleteConstraints holds the delete to conditions, written in the grammar of propertyConstraints: a poll may be deleted only before its first vote, an order only while it is still open, an address only when its owner keeps another.
| Where | document type |
| Value | An object of rules, at least one. Each key is the rule's name (1 to 64 letters, digits or underscores); each value is a condition in the grammar of a propertyConstraints rule |
| Default | Absent: the owner deletes whenever canBeDeleted allows |
| Since | protocol version 14 |
| On update | Fixed: adding, removing or changing a rule is refused (IncompatibleDocumentTypeSchemaError, 10246) |
| Errors | DocumentDeleteConstraintViolatedError (40147) on a delete; at registration JsonSchemaError (10101) or InvalidContractStructure (10231) |
Example
"poll": {
"type": "object",
"canBeDeleted": true,
"properties": {
"question": { "type": "string", "maxLength": 280, "position": 0 },
"status": { "type": "string", "enum": ["open", "locked"], "maxLength": 10, "position": 1 }
},
"required": ["$createdAt", "question"],
"deleteConstraints": {
"noVotes": { "equal": [{ "countOf": ["vote", { "pollId": "$id" }] }, 0] },
"notLocked": { "notEqual": ["status", { "const": "locked" }] }
},
"additionalProperties": false
},
"vote": {
"type": "object",
"properties": {
"pollId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "deletableDocument", "documentType": "poll" },
"position": 0
},
"choice": { "type": "integer", "minimum": 0, "maximum": 9, "position": 1 }
},
"required": ["pollId", "choice"],
"indices": [
{ "name": "byPoll", "properties": [{ "pollId": "asc" }], "countable": "countable" }
],
"additionalProperties": false
}
noVotes counts the votes whose pollId is the poll's own id ($id), from the count the byPoll index keeps, and lets the poll go only while there are none. notLocked reads the stored poll: a locked one stays. Once a vote is cast, its author can no longer pull the poll out from under it; once every vote is gone, the poll can be deleted again.
Without the keyword, the poll had to be undeletable for its votes to be sure it stays. A propertyConstraints rule of the same shape on the poll, { "equal": [{ "countOf": ["vote", { "pollId": "$id" }] }, 0] }, refuses an edit after the first vote in the same way.
How it works
- The owner's delete. When the owner deletes a document, consensus fetches it, checks the owner, then checks every rule in name order against the stored document, and refuses the delete at the first rule it breaks with
DocumentDeleteConstraintViolatedError(40147). The error names the document, the rule, and why it failed, with the reasons ofpropertyConstraints. It is a state error, so the delete is paid for and the nonce is spent, and the document stays. - What a rule reads. The stored document's properties, its
$ownerId(the owner deleting it), and the times and heights the type lists inrequired, as stored. AcountOforsumOfreads a total from state as it will be once the document is gone: a total over the type's own documents no longer counts it, so{ "greaterThanOrEqual": [{ "countOf": ["address", { "$ownerId": "$ownerId" }] }, 1] }keeps the last address of each owner. Each total is a billed read. $idin a filter. A filter may match a key by the document's own id,"$id", which is what makes "while no vote points at it" a rule. The key must be an identifier property of the counted type.$idis a filter value inpropertyConstraintstoo, as the example shows; it is not an operand anywhere else.- Only the owner's delete. The contract's moderators delete as their abilities allow, and a
ttlexpires documents on time, whatever the rules say. ArefersTowithconsumemay not target the type, since a consume deletes without a delete transition to judge. A banned or suspended owner, who may still delete its documents, is held to the rules like any other. - The totals are those of the block. Each state transition of a block is applied before the next is validated, so a vote earlier in the block counts, and a vote later in the block finds the poll gone (
ReferencedEntityNotFoundError, 40120). - A total of another type is judged at the delete alone. The rules do not stop the counted documents from changing later, which is what the example wants: deleting the last vote frees the poll.
Rules at registration
All refusals below are InvalidContractStructure (10231) unless noted.
- Only on a type whose owner deletes its stored documents: not
canBeDeleted: falseor"onlyWhenConsumed", where there is no delete to gate, and notindexOnly, whose delete carries the row's values and is judged by itspropertyConstraints. Checked on every parse. - The grammar, the shape and the reads are those of a
propertyConstraintsrule (see Rules at registration): every path names a stored property of the kind it is read as, a time or height the type lists inrequired, no$old.path. EverycountOfandsumOfcounts a type of the contract with a tree that keeps the total. Unlike apropertyConstraintsrule, a delete rule may total its own type when that type has a contested index: the total it reads at the delete is the stored one, which an awarded document is in like any other. - The same limits, counted apart from
propertyConstraints: at most 16 rules, each of at most 32 nodes, reading at most 4 distinct totals. - A
refersTowithconsumemay not target a type that declares the keyword.
moderatorAbilities.delete
Lets the contract's moderators delete documents of the type, whoever owns them. It is how an application takes down content that breaks its rules, where a ban only stops an identity from writing more. It is one key of the moderatorAbilities object, which also names the fields only moderators write.
| Where | moderatorAbilities of a document type, in a contract whose config declares moderation |
| Value | boolean |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
| Errors | DocumentTypeNotDeletableByModeratorsError (41115), IdentityNotContractModeratorError (41101), ContractModerationTargetNotAllowedError (41102), DocumentNotFoundError (40101), and DocumentModerationWindowElapsedError (41116) with a window |
Example
"post": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": false,
"moderatorAbilities": { "delete": true, "deleteWithin": 604800 },
"properties": {
"text": { "type": "string", "maxLength": 280, "position": 0 }
},
"required": ["$createdAt", "$updatedAt", "text"],
"additionalProperties": false
}
Authors cannot retract a post, but the contract's moderators can remove one for a week (604,800 seconds) after it was written or last edited. The contract around it must declare moderation in its config.
How it works
- A moderator deletes a document with the contract user moderation transition, naming the document type, the document id and a reason. The moderators are the ones the contract's
moderationconfig declares; see Contract Moderation. - The transition is checked in this order, each refusal paid: the document type exists (
InvalidDocumentTypeError, 10406); it setsdelete(41115); the signer is the contract owner or a moderator (41101); the document exists (40101); its owner is neither the contract owner nor a moderator (41102); and, when the type setsdeleteWithin, the window has not passed (41116). - The document and all its index entries are deleted as an owner's delete would delete them, without the
canBeDeletedcheck. A removal record is written under the contract: whose document it was, which moderator removed it, the reason, the block time and a hash of the document, and the values of any fields the type keeps public (seedeleteKeepsFields). The record is never deleted. A type may leave no record: seedeleteKeepsRecord. - The document's owner gets no storage refund unless the type says otherwise (see
deleteRefundsOwner), and the moderator pays neither the type's delete token cost nor its delete action fee. - For a week after the deletion a moderator may restore the document exactly as it was. See Restoring Documents.
Rules at registration
All refusals below are InvalidContractStructure (10231).
- The contract's config must declare
moderation. Moderation cannot be added by a later update, so without it nobody could ever delete anything. In return themoderationblock may keep no banlist, suspension list or warning list at all when a document type gives its moderators an ability. - Refused on a type that keeps history (Drive never deletes those documents), on an
indexOnlytype (there is no stored row to name), on a type withcreationRestrictionMode1 or 2 (its documents are the contract owner's or the platform's), and on a type with a contested index (a restore could not go through the vote the index requires). - For references, the type is no longer permanent even with
canBeDeleted: false: apermanentDocumentreference,inListincluded, may not point at it (40122). WithcanBeDeleted: false, nottland removal records kept (the default), its documents leave state only on a moderator's record, and amoderatedDocumentreference is the one that points at it, resolving to the document or to its removal record; adeletableDocumentreference is refused (40144). Otherwise adeletableDocumentreference points at it.
moderatorAbilities.deleteWithin
Limits the moderators' deletion to a window after a document's last change. Once the window has passed the document is settled: moderation acts on what was just written and does not reach back into what has stood unchallenged.
| Where | moderatorAbilities of a document type, with delete: true |
| Value | integer, seconds, 1 to 4294967295 |
| Default | absent: no limit |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212), in both directions: a longer window would reopen documents that had settled |
| Errors | DocumentModerationWindowElapsedError (41116) |
How it works
- The window is measured from the document's
$updatedAt, or from its$createdAton a type that does not record$updatedAt. A deletion at exactly that time plus the window still passes; after it, every moderator is refused, the contract owner included. On a type that setsdeleteSettled, the seated team may still delete it together. - A replace or a price update moves
$updatedAt, so new content opens the window again. A transfer, a purchase or a moderator's field change does not move it. - A restored document comes back with its old
$updatedAt, so it may already be settled. - The window says nothing about the document's own owner, whose deletion
canBeDeletedrules at any age.
Rules at registration
- Needs
delete: true(InvalidContractStructure, 10231). - A type whose documents can be replaced must list
$updatedAtinrequired: measured from creation alone, an author could wait the window out and then rewrite a post into something no moderator can remove. A type withdocumentsMutable: falsemust list$updatedAtor$createdAt. Both refusals are 10231. - A window of 0 is refused by the meta-schema (
JsonSchemaError, 10101). A type that moderators may never delete from simply leavesdeleteout.
moderatorAbilities.deleteSettled
Who must agree to delete a settled document: one past its deleteWithin window, which no moderator deletes alone. It lets a contract keep settled content safe from any single moderator while leaving its elected moderation team a way to remove it when the team agrees: so many members together, the leader among them only when the rule says so (leader: true), which may be the leader alone.
| Where | moderatorAbilities of a document type, with delete: true and deleteWithin, in a contract whose moderators are an elected team |
| Value | object with leader (boolean, default false: whether the team's leader must be among the approvals), approvals (integer, default 1: how many members of the seated team must approve, the leader counted, at least 1 and at most the members the declared team can hold) and approversPredateDocument (boolean, default true when approvals is above 1 and false otherwise: whether a member the leader added counts only for documents created after its addition), at least one of them given |
| Default | absent: nobody deletes a settled document |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212), in both directions: fewer approvals would reach content written under more |
| Errors | DocumentTypeNotDeletableOnceSettledError (41204), ContractModerationTeamNotSeatedError (41205), DocumentNotSettledError (41206), ContractTeamActionDoesNotExistError (41207), ContractTeamActionAlreadySignedError (41208), SettledDeletionNotRestorableError (41209), ContractTeamActionAlreadyCompletedError (41210), ContractTeamActionDocumentChangedError (41211), ContractTeamMemberAddedAfterDocumentError (41212), and those of a moderator's deletion (41101, 41102, 41201, 41203) |
Example
"post": {
"type": "object",
"documentsMutable": true,
"moderatorAbilities": {
"delete": true,
"deleteWithin": 86400,
"deleteSettled": { "leader": true, "approvals": 3 }
},
"properties": {
"text": { "type": "string", "maxLength": 280, "position": 0 }
},
"required": ["$createdAt", "$updatedAt", "text"],
"additionalProperties": false
}
For a day after a post is written or edited, any moderator deletes it. After that, it is deleted only when the team's leader and two other members approve: one of them proposes the deletion, and the others approve the proposal. A member the leader added counts only for posts written after the leader added it. { "leader": true } would let the leader alone delete a settled post; { "approvals": 2 } any two members; { "leader": true, "approvals": 3, "approversPredateDocument": false } the leader and two others, whenever added.
How it works
The team deletes a settled document the way a token group carries out an action: one member proposes it, the others approve it by its id, and it runs once the approvals meet the rule.
- Proposing. A member of the seated team sends the contract user moderation transition's
deleteSettledDocumentaction, naming the document type, the document id and a reason. The proposal is its own approval. It is kept under the contract as a team action, naming the document as it is (its last modification and revision) and the reason, by an id the proposer's client computes from the contract, the proposer, its nonce, the document and the reason. A rule the proposer meets alone ({ "leader": true }proposed by the leader) deletes the document at once. - Approving. The other members send the
approveTeamActionaction with that id. What is deleted and why is the proposal's: an approval carries nothing else. The team's actions and who approved each are readable withgetContractTeamActionsandgetContractTeamActionSigners, which is how a member finds what the others proposed. - Running. The approval that meets the rule deletes the document as a moderator's
deleteDocumentwould: its removal record (unless the type setsdeleteKeepsRecord: false, with the proposal's reason and the member whose approval deleted it), and the owner's refund asdeleteRefundsOwnersays. The action then moves, with the approvals that counted, from the contract's active team actions to its closed ones, for good. - Nothing lapses. A proposal stays active until it runs. Once the document changes (the author edits it, it changes hands, or a moderator changes its fields: anything that moves its
$revision), an approval is refused (41211): the team would be approving the deletion of content it never saw. A member proposes afresh instead, a new action. Two proposals may name one document; once one runs, the other takes no approval (the document is gone, 40101) and stays active, as a token group's action that never gathers its power does. An edit by the author opens thedeleteWithinwindow again anyway, in which any moderator deletes the document alone; a moderator's change of its fields does not. - Members the leader added. The leader names whom it adds (
addedModerator), so without a limit it could add members who approve whatever it proposes, and take them off again once they had. Unless the rule setsapproversPredateDocument: false, a member the leader added counts only for documents created after its addition (the default whenever more than one approval is needed; a rule one approval meets, the leader meets alone, so it dates nobody by default): itsaddedModerator's$createdAtis earlier than the document's$createdAt. One added in the same block as the document does not count. Its proposal or approval of an older document is refused (41212). The leader and the elected members always count, for documents older than the team too: the election seated them, not the leader. What the rule needs is not lowered for it: a team with too few members from before a document never deletes that document once settled. Every addition comes after the seat, so a document written before the seat counts only the leader and the elected members: under a rule asking for more approvals than those, no document older than the seat is ever deleted once settled, and the rule can not change. Pickapprovalsno higher than the leader plus the members a charter is expected to elect, or setapproversPredateDocument: false. - Members who left. A member who left the team since approving no longer counts. When the approvals given could meet the rule, an approval reads the team and drops the approvals of members who are gone, refunded to them: one who comes back after that approves again, while one back before any such read still has its approval counted. A member the leader takes off and adds again sits in the new addition: under
approversPredateDocumentits approval of a document created before the new addition is dropped the same way, and it approves that document no more. A dropped approval no longer proves: the proof of that member's proposal or approval fails from then on. - The checks of a proposal, in order, each refusal paid: the document type exists (10406) and sets
deleteSettled(41204); a team is seated (41205): the interim moderators and the contract owner never delete a settled document; the signer is on the team (41101) and the declaration gives the teamdeleteDocumentson the type (41201); the reason names a reason document the team's proposal lists (41203); the document exists (40101) and its owner is not protected (41102); the document is settled (41206: within the window, usedeleteDocument); a signer the leader added was added before the document was created (41212). - The checks of an approval, in order, each refusal paid: the contract keeps team actions and the team proposed this one (41207); the action has not run (41210); a team is seated (41205); the signer is on the team with
deleteDocumentson the type (41101, 41201); the document still exists (40101), its owner is not protected (41102), and it has not changed since the proposal (41211); a signer the leader added was added before the document was created (41212); the signer has not approved it already (41208): a member taken off and added again too late is told so, not that its earlier approval, which no longer counts, stands. - A deletion the team approved stands: no moderator restores it, the leader included (
SettledDeletionNotRestorableError, 41209). A single moderator undoing what the leader and the members agreed on would defeat the rule; a deletion within the window is restored as before. - A deletion counts toward the moderators pot's action share for every approver whose approval counted, once it happens. Approvals that fall short count for nobody.
- Each member pays for the transition and for its approval (the proposer for the action too), and is refunded its approval (the proposer the action's info too, but not the two trees that hold the action, which carry no storage flags and refund nobody) when the action runs and moves to the closed actions, or when a later approval drops it. The member whose approval runs the action pays for its closed copy, which nothing deletes.
Rules at registration
All refusals below are InvalidContractStructure (10231).
- Needs
deleteWithin: without a window nothing is ever settled. - Needs a contract whose
moderationdeclares an elected team. The elected declaration must give the teamdeleteDocumentson the type (InvalidContractModerationConfigError, 10900), or no team could ever use the rule. approvalsis at least 1 and at most the members the declared team can hold: its leader, the 15 members a charter elects and themaxAddedModeratorsof the elected declaration (31 at most). A rule no team could meet would leave settled documents undeletable for good, since neither the rule nor the declaration can change. A seated team whose charter elects fewer than 15 members holds fewer: a rule asking for more than it can hold asks for all of them. A member the leader removed still counts toward what the team can hold: the leader can not lower the bar by removing members who would not approve, and gets the seat back by deleting the removal.- While
approversPredateDocumentis on (the default whenapprovalsis above 1), the type must list$createdAtinrequired: who of the team predates a document is read from it. A type that does not record$createdAtsetsapproversPredateDocument: false. Checked at registration only, as the bound onapprovalsis: a stored type is read back as it is, and a document without$createdAtadmits no added member.
moderatorAbilities.deleteKeepsRecord
Whether a moderator's deletion leaves a removal record under the contract. The record is what explains a missing document (who removed it, whose it was, why, when) and what a restore brings it back from. A contract that wants its moderators' deletions final and unrecorded, or does not want to pay for the records, turns it off.
| Where | moderatorAbilities of a document type, with delete: true |
| Value | boolean |
| Default | true |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
| Errors | ContractDocumentRemovalNotFoundError (41119) for a restore when false |
How it works
- With
false, the deletion writes no record, and the type gets no removal records tree:getContractDocumentRemovalsrefuses it as a type that keeps none. The document is gone for good: a restore is refused (41119), and a document id is never produced twice, so it cannot come back another way. - The proof of such a deletion is the document's absence, which the SDKs report as no record (
delete_contract_documentresolves withNone,contractDeleteDocumentwithundefined). The verifier reads the type's setting from the contract, so for such a type it needs the contract, as a restore's does; a deletion that leaves a record is proved by the record alone. - The moderator pays less: no record is written and no hash computed.
Rules at registration
- Needs
delete: true(InvalidContractStructure, 10231).
moderatorAbilities.deleteKeepsFields
Which fields of a deleted document stay public in its removal record. The document is gone, but some of what it said may still matter to everyone else: the hashtag of a removed post keeps the hashtag's timeline honest ("a post here was removed"), the thread a removed reply belonged to, the time it was written. The record keeps a copy of those values; everything else leaves with the document.
| Where | moderatorAbilities of a document type, with delete: true and a record (deleteKeepsRecord not false) |
| Value | array of property paths, at least one, none twice |
| Default | absent: the record keeps no field of the document |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212), in both directions: which fields stay public is what an author was told when writing |
Example
{
"type": "object",
"properties": {
"text": { "type": "string", "maxLength": 500, "position": 0 },
"hashtag": { "type": "string", "maxLength": 61, "position": 1 },
"meta": {
"type": "object",
"position": 2,
"properties": {
"tags": { "type": "array", "items": { "type": "string", "maxLength": 20 }, "maxItems": 5, "position": 0 },
"note": { "type": "string", "maxLength": 100, "position": 1 }
},
"additionalProperties": false
}
},
"required": ["$createdAt", "text"],
"canBeDeleted": false,
"moderatorAbilities": {
"delete": true,
"deleteKeepsFields": ["hashtag", "meta.tags", "$createdAt"]
},
"additionalProperties": false
}
A moderator deleting a post of this type leaves a record that still says which hashtag and tags it carried and when it was written, while its text and its note are gone:
{
"documentId": "…",
"documentOwnerId": "…",
"moderatorId": "…",
"reason": { "text": "spam" },
"removedAt": 1759200000000,
"documentHash": "…",
"keptFields": {
"$createdAt": 1759100000000,
"hashtag": "dash",
"meta.tags": ["privacy", "payments"]
}
}
How it works
- The values are copied from the document as stored at the deletion, each under the path the type lists: a top-level property, a property inside an object (
meta.tags), or a whole object (meta). A path the document holds no value at is left out of the record. - The record stores them as the document stores its properties, so they are read, as the document is, under its document type, and come back typed exactly as the document's values: the SDKs do this for you (
keptFields). An object kept whole shows members an update added after the removal as absent, as an older document does. - They are read wherever the record is:
getContractDocumentRemovals, by document id or by page, the proof of the deletion, and a join through amoderatedDocumentreference. The records are not indexed by them: no query finds a record by a kept value. - A
moderatedDocumentreference to a removed document checks awherepair on a kept property against the kept value, as it would against the document. - An index of a referring type may hold a kept value, read through a
moderatedDocumentreference: a derived index property, such as a reply'spostId.hashtag. The value must be one an index can key and fixed once written: the example above is not, since its documents are mutable andhashtagis not listed underimmutable. Once the document is removed, Drive reads the value from the record. - A restore brings the document back and leaves the record, marked restored, with what it kept. A later deletion of the restored document writes a fresh record, with the values the document then held.
- The moderator pays for the record, kept values included.
Rules at registration
All refusals below are InvalidContractStructure (10231).
- Needs
delete: true, and a record: refused besidedeleteKeepsRecord: false. - Each entry is a declared property at any depth, stepping through objects by
., or one of the timestamps and block heights ($createdAt,$updatedAt,$transferredAt, and theirBlockHeightandCoreBlockHeightforms) listed inrequired, without which no document carries it. - Refused: a transient property (no stored document holds it),
$idand$ownerId(every record holds them already), any other system property, and a path inside another listed path (the object around it is kept whole already).
moderatorAbilities.deleteRefundsOwner
Whether the owner of a document a moderator deletes is refunded its storage. By default the owner forfeits it: removed content costs its author what they paid to store it. A contract whose moderation is housekeeping rather than sanction (clearing handled reports, expired listings) gives it back.
| Where | moderatorAbilities of a document type, with delete: true |
| Value | boolean |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
How it works
- With
true, the owner is refunded as for their own deletion: the part of the storage fee not yet paid out to past epochs. The refund goes to the owner, not to the moderator, who still pays for the transition and the record. A document of a type with attlrefunds nothing either way. - With
false, the credits stay in the storage pools they were paid into.
Rules at registration
- Needs
delete: true(InvalidContractStructure, 10231).
How they combine
| Who deletes | Allowed by | Refund to the owner |
|---|---|---|
| The document's owner | canBeDeleted: true, on a type that does not keep history, when the stored document meets every deleteConstraints rule | Yes, except on a type with a ttl |
| The contract's moderators | moderatorAbilities.delete: true, within moderatorAbilities.deleteWithin when set | Only with moderatorAbilities.deleteRefundsOwner: true, except on a type with a ttl |
| The seated moderation team, together | moderatorAbilities.deleteSettled, once moderatorAbilities.deleteWithin has passed | As for the moderators |
| The platform | ttl, once it has passed | No |
Which reference may point at a type follows from which of the three it allows. A type that allows none of them is the target of a permanentDocument reference, with inList or without. A type that allows only the moderators' deletion, with removal records kept, is the target of a moderatedDocument reference: its documents never leave state without a record. Any other type is the target of a deletableDocument reference.
See also
- Deleting Documents, Deleting Settled Documents and Restoring Documents, for the moderation transition, the approvals and the removal record
- Moderator Abilities, for the
moderatorAbilitiesobject and the fields only moderators write - Time To Live, the third way a document leaves the state
- History, for why a type that keeps history can never delete
- Mutability, for
immutableand the conditionsretractedWhenshares with it, and Creation, Transfers and Trading - propertyConstraints, for the grammar of
deleteConstraintsand the totals its rules read - References, for
permanentDocument,moderatedDocumentanddeletableDocument - Contract-Level Keys and config, for
documentsCanBeDeletedContractDefaultandmoderation
Moderator Abilities
moderatorAbilities says what the contract's moderators may do to the documents of a type, whoever owns them: delete them, and write the fields that only they write. It is how an application gives its moderators a say over content without making them its authors: a report its moderators mark as handled, a post they flag, a ticket they assign.
| Where | document type, in a contract whose config declares moderation |
| Value | object with delete (boolean), deleteWithin (seconds), deleteKeepsRecord and deleteRefundsOwner (booleans), deleteKeepsFields (array of property paths), the last four only beside delete: true, deleteSettled (object, beside deleteWithin), and changeFields (array of top-level property names), at least one of them given |
| Default | absent: the moderators can do nothing to documents of the type |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212): a type can neither gain, lose nor change it. A type the update adds may declare it. |
The keys:
| Key | What it allows | Details |
|---|---|---|
delete | The moderators delete documents of the type. By default each deletion leaves a removal record and can be restored within a week; with deleteKeepsRecord: false it leaves none and is final. | Deletion |
deleteWithin | Limits delete to so many seconds after a document's last change. | Deletion |
deleteKeepsRecord | Whether a deletion leaves a removal record, and so can be restored. Default true. | Deletion |
deleteRefundsOwner | Whether the deleted document's owner is refunded its storage. Default false. | Deletion |
deleteSettled | Past deleteWithin, the seated team of an elected contract deletes a document together: one member proposes, and so many approve, the leader among them only when the rule says so (leader: true), and a member the leader added only for documents created after its addition unless the rule says otherwise (approversPredateDocument: false). | Deletion |
deleteKeepsFields | The fields of a deleted document whose values stay public in its removal record, such as a post's hashtag. | Deletion |
changeFields | The listed properties are written only by the moderators. | below |
The moderators are the ones the contract's moderation config declares: the contract owner, the identities it appoints, or the members of the seated team of an elected contract. A seated team holds an ability on a type only when the declaration's moderatedDocumentTypes gives it: deleteDocuments for delete, changeDocumentFields for changeFields. See Contract Moderation.
changeFields
The top-level properties of the type that only the contract's moderators write. A moderator sets or removes them on any document of the type with the contract user moderation transition. A document's owner can neither set them when creating the document nor change or remove them when replacing it, unless the owner is a moderator of the contract.
| Where | moderatorAbilities of a document type |
| Value | array of top-level property names, at least one, none twice |
| Default | absent: nobody but a document's owner writes its properties |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
| Errors | DocumentFieldNotChangeableByModeratorsError (41123), DocumentModeratorFieldNotWritableError (41124), InvalidContractModerationDocumentFieldsError (10905), IdentityNotContractModeratorError (41101), ContractModerationAbilityNotGrantedError (41201), ModerationReasonNotListedError (41203) |
Example
"report": {
"type": "object",
"documentsMutable": false,
"moderatorAbilities": { "delete": true, "changeFields": ["status", "resolution"] },
"indices": [
{ "name": "byStatus", "properties": [{ "status": "asc" }, { "$createdAt": "asc" }] }
],
"properties": {
"postId": {
"type": "array",
"byteArray": true,
"minItems": 32,
"maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 0,
"refersTo": { "type": "deletableDocument", "documentType": "post" }
},
"reason": { "type": "integer", "minimum": 0, "maximum": 8, "position": 1 },
"status": { "type": "integer", "minimum": 1, "maximum": 3, "position": 2 },
"resolution": { "type": "string", "maxLength": 200, "position": 3 }
},
"required": ["$createdAt", "postId", "reason"],
"additionalProperties": false
}
A user files a report and can never edit it. A report starts with no status: the reporter cannot file one already marked handled. A moderator sets status and resolution when the report is dealt with, and queries the open ones through byStatus. The report stays, so the reporter and everyone else can see what became of it.
How it works
- A moderator changes the fields with the contract user moderation transition's
changeDocumentFieldsaction, naming the document type, the document id, the new value of each field (nullremoves one; in the JavaScript SDKs a field set toundefinedis left out, as JSON leaves it out) and a reason. - The transition is checked in this order, each refusal paid: the document type exists (
InvalidDocumentTypeError, 10406); every field named is listed underchangeFields(41123); the signer is a moderator of the contract (41101), and for a seated team the declaration gives itchangeDocumentFieldson the type (41201) and the reason names a reason document its proposal lists (41203); the document exists (40101) and has not expired (DocumentExpiredError, 40140); and the document as changed is still one of its type: its schema (JsonSchemaError, 10101), itspropertyConstraintsand its unique indexes (DuplicateUniqueIndexError, 40105). A change naming no field, or a system property, is refused before any state is read (10905). - Whoever owns the document may have it changed, the contract owner and the moderators included: the fields are the moderators', not the owner's, so the protection that keeps moderators from deleting each other's documents does not apply.
- The document records who wrote the fields last, and when:
$moderatedAtand$moderatedBy, which an index of the type may name to find what the moderators handled. - The document is updated in place. Every other property stays as its owner wrote it,
$updatedAtamong them, so a change never opens adeleteWithinwindow again.$revisiongoes up by one, so a replace its owner built on the earlier revision is refused (InvalidDocumentRevisionError, 40106) rather than writing over the change. - References are not checked again: no field a moderator writes holds one or is read by one (see the rules below). A report whose post a moderator has already deleted can still be marked handled.
- The moderator pays for the transition and the bytes it adds. Storage the change frees, or index entries it moves, are refunded to the document's owner, who paid for them. Nothing the type prices is charged: a moderator's change is no action of the owner's.
- A change that changes nothing, every field already holding the value named, is refused (10905): it would bump
$revisionfor nothing. - A change needs a listed reason from a seated team, but does not count toward the team's action share: changes of one document have no bound, as deletions of the content that exists do, so counting them would let a member farm the share.
- The proof of the change is the document as it now stands, holding the values the change set.
Owners who moderate
A create that sets a listed field, and a replace that changes, adds or removes one, is refused unless its signer is a moderator of the contract (41124), in a block and in the mempool alike. When the signer does moderate, the document is stamped as a moderator's: $moderatedAt the block's time, $moderatedBy the signer. A replace that leaves the listed fields alone keeps the stamp as it was. The contract owner is one under the appointed and owner-only kinds and during an interim that names it; once an elected contract has a seated team, only the team's leader and members are, and only where the declaration gives the team changeDocumentFields on the type. A replace carries the whole document, so an owner who is not a moderator must carry the listed fields exactly as the moderators left them.
Rules at registration
All refusals below are InvalidContractStructure (10231).
- The contract's config must declare
moderation, as fordelete. Amoderationblock may then keep no list at all. An elected declaration must give the teamchangeDocumentFieldson the type inmoderatedDocumentTypes(InvalidContractModerationConfigError, 10900): once a team is seated only it writes the fields, and without the ability nobody could. - Refused on an
indexOnlytype: there is no stored row to change. - Refused on a type that requires a
transientproperty: a transient value is never stored, and a moderator's change is judged against the schema, which would find it missing on every document. - Every entry must name a top-level property the type declares (list the object around a nested one), and that property must be:
- optional: nobody but a moderator can set it, so it starts absent;
- stored, so not
transient; - not listed under
immutable; - neither a
refersToreference nor read by one: not the referring value of awhereentry, not a sourcefindByreads, not the identity property of a key id reference; - neither
generatedFromanother property nor a parameter of one; - in no contested index.
- A type that lists any keeps
$revisionon its documents, even whendocumentsMutableisfalse, because a moderator's change is stored as an update. - A
refersTofindBy, or the list aninListreference reads, may not read a listed field of the type it refers to: such a field can change after the reference was checked.
See also
- Deletion, for
delete,deleteWithinanddeleteSettled - Contract Moderation, for the transition, the checks and the proof
- Mutability, for what a document's own owner may change
- Contract-Level Keys and config, for
moderation
Time To Live (ttl)
ttl gives every document of a type a lifetime. The platform deletes each document once that many seconds have passed since it was created, whoever owns it and whatever the type says about who else may delete it. Use it for content that is meant to disappear: stories, invitations, offers, session records. Such documents are also cheaper: they pay for the time they occupy the state rather than for storage forever.
This is the document type keyword. An index can carry a ttl of its own inside timeRange, which expires index entries and leaves the documents in place; see Time-Range Indexes.
| Where | document type |
| Value | integer, seconds, 3600 (one hour) to 31536000 (one year) |
| Default | absent: documents live until someone deletes them |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212): an update may not add, remove or change it |
| Errors | DocumentExpiredError (40140) for a replace, transfer, purchase or price update of a document past its expiry |
Example
"story": {
"type": "object",
"ttl": 86400,
"canBeDeleted": true,
"properties": {
"caption": { "type": "string", "maxLength": 200, "position": 0 },
"mediaUrl": { "type": "string", "maxLength": 500, "position": 1 }
},
"required": ["$createdAt", "mediaUrl"],
"additionalProperties": false
}
Each story is deleted one day (86,400 seconds) after its creation. Its author may delete it sooner. $createdAt must be in required, since the expiry is counted from it.
How it works
- When a document expires. At its
$createdAtplusttlseconds.$createdAtis the block time of the create, so nothing the writer sends moves the expiry, and documents created in the same block expire together. A replace, transfer, purchase or price update never moves it either: a buyer of an expiring document buys what is left of its life. - When it is deleted. After each block's state transitions, the platform deletes expired documents, oldest first: at most 128 per block, and at most 1,024 in weight, where a document weighs 1 plus the index levels of its type (the values at protocol version 14). The rest wait for the next block. The deletion is an ordinary one: the document and every index entry go, and counts and sums are brought down.
- Between expiry and deletion. A document past its expiry that the cleanup has not reached yet can still be queried and referenced, and it still holds its values in the type's unique indexes, so a create with the same unique value is refused as a duplicate until the cleanup has run. It can no longer be replaced, transferred, bought or repriced, and a moderator can no longer restore it: each is refused, paid, with
DocumentExpiredError(40140). Its owner may still delete it wherecanBeDeletedallows. - Earlier deletion. The owner may delete a document before it expires when
canBeDeletedallows it, and the contract's moderators whenmoderatorAbilities.deletedoes.canBeDeleted: falseonly stops the owner; the platform still deletes the document when it expires. - What it costs. The document is stored without storage flags and refunds nothing when it is deleted, by anyone. Instead of the price of permanent storage, each byte it writes pays a price for the time it will live: five tiers up to seven days, then a price per 9.125 days spanned. Creating it also prepays, as processing, the cost of its later deletion. A replace, transfer, purchase or price update pays for the bytes it adds at the price of the lifetime left. See Fees.
- Proofs near the expiry. A write accepted in the last block before a document expires proves the document present. A proof fetched after the next block's cleanup finds it gone. The one-hour minimum keeps a newly created document in the state well past the moment its writer fetches the proof of the create.
Rules at registration
The refusals below are InvalidContractStructure (10231) unless a bullet says otherwise. They hold on every parse of the contract, except the bounds, which are checked when a contract is registered or updated.
$createdAtmust be inrequired.- Refused together with
documentsKeepHistory: true(Drive never deletes a document whose type keeps history), withindexOnly: true(there is no stored row to delete by id), and on a type with a contested index (a contested document waits in its vote poll, and could expire before it is stored). - A summed property (
summable,averageable,documentsSummableordocumentsAverageable) declares aminimumof at least 0, or aminimumof at least -134217728 and amaximumof at most 134217728 (±2^27,max_expiring_signed_summed_value_magnitude). Deleting an expired document takes its value out of the type's sums at the end of a block, with no transition to refuse: removing values that are never negative only lowers the sums, and values this small keep them in the signed 64-bit range short of 2^36 documents. Checked when a contract is registered or updated, like the bounds below. - At least
min_document_ttl_secondsand at mostmax_document_ttl_secondsofSystemLimits: 3600 and 31536000 at protocol version 14. The meta-schema itself admits 1 to 4294967295, so attlof 0 is aJsonSchemaError(10101) and one outside the narrower bounds is 10231. - For references, a type with a
ttlis deletable. ApermanentDocumentreference may not point at it, one found byfindByor withinListincluded (ReferencedDocumentTypeDeletableError, 40122); adeletableDocumentreference may. See References.
Everything else combines with a ttl: mutable types, transferable, tradeMode, moderatorAbilities.delete, creationRestrictionMode, count, sum and ranked indexes, timeRange indexes with or without their own ttl, references declared on the type, action fees and token costs, with a summed property bounded as above.
On update
An update may not add, remove or change the ttl of an existing document type. Every stored document has the expiry it was written and paid with: adding one would leave stored documents that the cleanup cannot find, removing it would leave entries that delete documents the type says live forever, and changing it would move expiries away from what was paid for. A document type added by an update may declare a ttl freely.
See also
- Document Time To Live, for the expirations tree, the fee tiers, the payout to epochs and the cleanup
- Deletion, for the other two ways a document is deleted
- Time-Range Indexes, for the index
ttl - History, for why a type that keeps history cannot expire
- System Properties, for
$createdAt
Creation, Transfers and Trading
A document belongs to the identity in its $ownerId: at first the one that created it. These three keywords decide who may create documents of a type, and whether a document may later change hands. creationRestrictionMode limits who creates. transferable lets an owner give a document away. tradeMode lets an owner put a price on a document and anyone else buy it at that price.
The example below is used throughout the chapter:
"ticket": {
"type": "object",
"documentsMutable": false,
"canBeDeleted": false,
"creationRestrictionMode": 1,
"transferable": 1,
"tradeMode": 1,
"properties": {
"event": { "type": "string", "maxLength": 63, "position": 0 },
"seat": { "type": "string", "maxLength": 10, "position": 1 }
},
"required": ["event", "seat", "$createdAt"],
"additionalProperties": false
}
Only the organiser, the identity that owns the contract, can issue tickets. A ticket's holder may give it to a friend, or list it for sale; anyone may then buy it at the listed price, with no approval from the seller. Nobody can edit a ticket or delete it.
creationRestrictionMode
Who may create documents of the type.
| Where | document type |
| Value | 0 anyone, 1 the contract owner only, 2 nobody |
| Default | 0 |
| Since | protocol version 1 |
| On update | Fixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | DocumentCreationNotAllowedError (10416) |
How it works
0: any identity may create documents of the type.1: only the identity that owns the contract may. A create signed by anyone else is refused (DocumentCreationNotAllowedError, 10416). Once created, a document may still pass to other identities if the type is transferable or tradeable.2: no create transition is ever accepted. This is for system contracts whose documents only the platform writes, such as the document history contract (see History) and the keyword search contract. On a user's contract the type would stay empty for good.- The mode rules creation only. Replaces, deletes, transfers and sales are decided by each document's own owner and by the other keywords.
Rules at registration
- A type with mode
1or2may not carrymoderatorAbilities.delete(InvalidContractStructure, 10231): its documents belong to the contract owner or the platform, and no moderator may delete those. See Deletion.
transferable
Whether an owner may give a document to another identity.
| Where | document type |
| Value | 0 never, 1 always |
| Default | 0 |
| Since | protocol version 1 |
| On update | Fixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (10246). |
| Errors | InvalidDocumentTransitionActionError (10404) for a transfer of a type set to 0 |
How it works
- The owner sends a transfer transition naming the document, the recipient and a
$revisionone higher than the stored one (InvalidDocumentRevisionError, 40106). Only the owner may transfer (DocumentOwnerIdMismatchError, 40102). The recipient does not have to agree. - The document gets the recipient as its
$ownerIdand a new$revision. A price it was listed at is removed, so a transferred document is no longer for sale. When the type lists$transferredAtinrequired, it is set to the block's time, and likewise$transferredAtBlockHeightand$transferredAtCoreBlockHeightto the block heights. - A transfer carries no property values, and
$creatorIdkeeps naming the identity that created the document. The one change the platform makes itself: from protocol version 13, a DPNSdomainthat is transferred or sold has itsrecords.identitypointed at the new owner. - The document is checked as it will be stored, with its new owner: against the type's unique indexes (
DuplicateUniqueIndexError, 40105), against adistinctFrom: "$ownerId"property (DocumentPropertyNotDistinctError, 10419), and againstpropertyConstraintsrules that read$ownerId(DocumentPropertyConstraintViolatedError, 10422). - On a moderated contract, a transfer to a banned or suspended identity is refused (
ContractModerationCounterpartyBarredError, 41114). - A transfer of a document past its
ttlexpiry is refused (DocumentExpiredError, 40140). - With
keepsTransferHistory: true, each transfer is also recorded in the document history contract. See History.
tradeMode
Whether documents of the type can be sold through the platform's built-in marketplace. With 1, direct purchase, an owner sets a price and any other identity may buy the document at that price, with no approval.
| Where | document type |
| Value | 0 none, 1 direct purchase |
| Default | 0 |
| Since | protocol version 1 |
| On update | Fixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (10246). |
| Errors | InvalidDocumentTransitionActionError (10404) for a price update or a purchase on a type set to 0, and for a purchase by the document's own owner; DocumentNotForSaleError (40108); DocumentIncorrectPurchasePriceError (40109) |
How it works
- Listing. The owner sends a price update transition with a price in credits and the next
$revision. Only the owner may set the price (40102). The price is stored with the document, the revision goes up, and$updatedAtis set when the type lists it inrequired. - Buying. Any identity other than the owner sends a purchase transition naming the document, the next
$revisionand the price. A document with no price set is not for sale (DocumentNotForSaleError, 40108), and the price in the transition must equal the listed price exactly (DocumentIncorrectPurchasePriceError, 40109). An owner cannot buy its own document (10404). - What a purchase does. The price moves from the buyer's credit balance to the seller's. The document gets the buyer as its
$ownerIdand a new$revision, the listing is removed, and$transferredAtis set when the type lists it inrequired. The buyer's credit balance must cover the price. - The new owner is checked exactly as for a transfer: unique indexes (40105),
distinctFrom: "$ownerId"(10419) andpropertyConstraintsrules that read$ownerId(10422). - On a moderated contract, a banned or suspended buyer is refused like any barred writer, and so is a purchase from a banned or suspended seller (
ContractModerationCounterpartyBarredError, 41114). - A price update or a purchase of a document past its
ttlexpiry is refused (DocumentExpiredError, 40140). - With
keepsPricingHistoryandkeepsPurchaseHistory, each price update and each purchase is also recorded in the document history contract. See History.
How they combine
transferableandtradeModeare independent. The DPNSdomaintype sets both, so a name can be given away or sold. A type may allow sales without gifts, or gifts without sales.- Neither needs
documentsMutable. A document that cannot be replaced can still change owner and price, as the ticket above does. - A type that is transferable or tradeable stores a
$revisionon each document even when its documents cannot be replaced, and, from protocol version 10 on a format-1 contract whose config is version 1 or later, records each document's creator in$creatorId. See System Properties. - Every action has its own optional token cost and action fee:
transfer,update_priceandpurchase, besidescreate. See Token Costs and Action Fees. signatureSecurityLevelRequirementapplies to all of these actions, so a buyer signs a purchase with a key at the level the type requires. See Signing and Keys.
Rules at registration
All refusals below are InvalidContractStructure (10231).
ownerRefersTois refused on a type whose documents can be transferred or traded: a document could end up with an owner the declaration never checked. Such a type usescreatorRefersTo, which checks the creator, who never changes.creatorRefersTois only accepted on such a type. See Writer and Creator References.- An
indexOnlytype can be neither transferable nor tradeable. See Index-Only Types. - On a transferable or tradeable type, an
immutableproperty may not hold acontractreference with anownerrequirement. See Mutability. creationRestrictionMode1or2is refused together withmoderatorAbilities.delete.
See also
- History, for recording transfers, purchases and price updates
- Mutability and Deletion, the other keywords on what may happen to a document
- System Properties, for
$ownerId,$creatorId,$revisionand$transferredAt - distinctFrom and propertyConstraints, which also judge a new owner
- Contract Moderation, for barred counterparties
History
Platform can keep two kinds of history for a document type. documentsKeepHistory keeps every version of each document in Drive, under the document itself, so an application can read what a document said at any earlier time. The three keeps*History flags record events instead: each transfer, purchase or price update of a document becomes a record in the document history system contract, where it can be queried by document, by contract, by identity and by time.
documentsKeepHistory
Keeps every version of every document of the type, not only the latest.
| Where | document type |
| Value | boolean |
| Default | the contract config's documentsKeepHistoryContractDefault, which is false unless the contract says otherwise |
| Since | protocol version 1 |
| On update | Fixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | InvalidDocumentTransitionActionError (10404) for a delete of a document of the type, from protocol version 14 |
Example
"profile": {
"type": "object",
"documentsKeepHistory": true,
"canBeDeleted": false,
"properties": {
"displayName": { "type": "string", "maxLength": 25, "position": 0 },
"bio": { "type": "string", "maxLength": 140, "position": 1 }
},
"required": ["displayName"],
"additionalProperties": false
}
Every edit of a profile adds a version, and every earlier version stays readable. canBeDeleted: false has to be written out, since a type that keeps history can never delete and the default is true.
How it works
- Each document is stored as a small tree of its own: every version the document has had, keyed by the block time it was written at, and a pointer to the current one. A query by id or through an index sees the current version.
- Every write that changes the document adds a version: a replace, and also a transfer, a price update or a purchase.
- Nothing is ever removed. Drive refuses to delete a document whose type keeps history, so its owner's delete is refused (10404 from protocol version 14; before it, the delete failed inside Drive as an internal error), and the type can have neither moderator deletion nor a
ttl. - The
getDocumentHistoryquery returns a document's versions from a given time on, each with the block time it was written at, at most 10 per request, and with a proof when asked. - Every version stays stored, paid for by the write that added it.
- A doctype-level sum or average (
documentsSummable,documentsAverageable) counts only each document's current version. See Counts, Sums and Averages.
Rules at registration
All refusals below are InvalidContractStructure (10231).
- From protocol version 14, the type must set
canBeDeleted: false. A contract registered earlier with both flags on stays readable, and its next update must turncanBeDeletedoff on that type. See Deletion. - Refused together with
ttl, withmoderatorAbilities.deleteand withindexOnly.
keepsTransferHistory
Records every transfer of a document of the type in the document history contract.
| Where | document type |
| Value | boolean |
| Default | false |
| Since | protocol version 13 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
| Errors | none of its own |
keepsPurchaseHistory
Records every purchase of a document of the type in the document history contract.
| Where | document type |
| Value | boolean |
| Default | false |
| Since | protocol version 13 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
| Errors | none of its own |
keepsPricingHistory
Records every price update of a document of the type in the document history contract.
| Where | document type |
| Value | boolean |
| Default | false |
| Since | protocol version 13 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
| Errors | none of its own |
The document history contract
Example
"ticket": {
"type": "object",
"documentsMutable": false,
"canBeDeleted": false,
"transferable": 1,
"tradeMode": 1,
"keepsTransferHistory": true,
"keepsPurchaseHistory": true,
"keepsPricingHistory": true,
"properties": {
"event": { "type": "string", "maxLength": 63, "position": 0 },
"seat": { "type": "string", "maxLength": 10, "position": 1 }
},
"required": ["event", "seat"],
"additionalProperties": false
}
Each time a ticket is given away, listed or sold, a record of it is written to the document history contract, so anyone can look up who held a ticket, what it was offered at and what it sold for. The DPNS domain type sets all three flags from protocol version 13.
How records work
The document history contract is a system contract registered at protocol version 13, with the id 6voHRaoiPcfmMhbqCA9dixH98xcgPQ9UEcuaXjpVu3LD. It has one document type per flag:
| Flag | Record type | Written for | Properties | $ownerId of the record |
|---|---|---|---|---|
keepsTransferHistory | transfer | each transfer | dataContractId, documentTypeName, documentId, toIdentityId | the sender |
keepsPurchaseHistory | purchase | each purchase | dataContractId, documentTypeName, documentId, sellerId, price | the buyer |
keepsPricingHistory | priceUpdate | each price update | dataContractId, documentTypeName, documentId, price | the owner who set the price |
- The platform writes the record as part of the transition that made the change, so a record exists exactly when the transfer, purchase or price update succeeded. Each record also carries
$createdAtand$createdAtBlockHeight, the block's time and height. - The identity that signed the action owns the record, and writing it is part of that transition's fees.
- Records can never be changed or deleted, and nobody can create one directly: the record types set
documentsMutable: false,canBeDeleted: falseandcreationRestrictionMode: 2. - The record types are indexed for lookups by document (
byDocument: contract, document, time) and by contract (byContract: contract, time). Transfers are also indexed by sender (from) and recipient (to), purchases by buyer (buyer), seller (seller) and price (byPrice). - They also keep provable aggregates.
purchasekeeps a count, total and average ofpriceover all its records, and per contract or per document over a time range.priceUpdatekeeps a count of all its records, and a count, total and average of asking prices per contract or per document over a time range, where a document listed three times counts three times.transferkeeps a count of all its records, and per contract over a time range. See Counts, Sums and Averages. - A flag only records the action the type allows.
keepsTransferHistoryon a type that is not transferable records nothing; no rule ties the flags totransferableortradeMode. - The flags are fixed on update, so an existing type cannot start or stop recording. A type added by an update may set them.
Rules at registration
- An
indexOnlytype may keep no history of either kind. See Index-Only Types.
See also
- Creation, Transfers and Trading, for the actions the three flags record
- Deletion and Time To Live, for why a type that keeps history can never lose a document
- Counts, Sums and Averages, for the aggregates of the history records
- Contract-Level Keys and config, for
documentsKeepHistoryContractDefaultand the contract's ownkeepsHistory
Signing and Keys
These keywords tie a document type to identity keys. signatureSecurityLevelRequirement sets how strong a key must be to sign transitions on documents of the type. requiresIdentityEncryptionBoundedKey and requiresIdentityDecryptionBoundedKey let identities register encryption and decryption keys bound to the type, for applications that encrypt data between users, and say how many such keys an identity may hold.
Every identity key has a purpose (authentication, encryption, decryption and others) and a security level. The levels are, from strongest to weakest, MASTER (0), CRITICAL (1), HIGH (2) and MEDIUM (3): a lower number is a stronger key. See Identity Keys.
signatureSecurityLevelRequirement
The weakest key security level that may sign a transition on documents of the type. Raise it to 1 for documents whose forgery would be costly, so that a weaker everyday key cannot write them.
| Where | document type |
| Value | 1 critical, 2 high, 3 medium |
| Default | 2 (high) |
| Since | protocol version 1 |
| On update | Fixed (DocumentTypeUpdateError, 40212). Adding or removing the key without changing its value is refused too, as a schema change (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | InvalidSignaturePublicKeySecurityLevelError (20004) |
Example
"payout": {
"type": "object",
"documentsMutable": false,
"canBeDeleted": false,
"signatureSecurityLevelRequirement": 1,
"properties": {
"recipient": {
"type": "array",
"byteArray": true,
"minItems": 32,
"maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 0
},
"amount": { "type": "integer", "minimum": 1, "position": 1 }
},
"required": ["recipient", "amount"],
"additionalProperties": false
}
Only a critical key may sign a transition that creates a payout. A high or medium key of the same identity is refused.
How it works
The requirement admits the level it names and every stronger level except MASTER:
| Value | Keys that may sign |
|---|---|
1 critical | critical |
2 high (the default) | critical, high |
3 medium | critical, high, medium |
- It applies to every document transition on the type: create, replace, delete, transfer, price update and purchase. A purchase is signed by the buyer, so the buyer needs a key at that level.
- A batch transition signs all its transitions with one key. When it holds transitions on several document types, the key must satisfy the strictest of their requirements. A batch that also holds a token transition needs a critical key.
- The key must be an authentication key. A master key never signs a document batch, whatever the requirement: the batch is refused at the signature check (20004).
- A key whose level the requirement does not admit is refused with
InvalidSignaturePublicKeySecurityLevelError(20004) after the signature has been verified. The failure is paid: the identity's nonce for the contract is bumped and the fees are charged.
Rules at registration
- The meta-schema admits only
1,2and3(JsonSchemaError, 10101 otherwise).0, master, cannot be required.
requiresIdentityEncryptionBoundedKey
Lets identities add encryption keys bound to this document type, and says how they are kept. Use it when documents of the type carry data encrypted to their readers, and each identity publishes the key others encrypt to.
| Where | document type |
| Value | 0 unique, 1 multiple, 2 multiple with a pointer to the latest |
| Default | absent: no encryption key may be bound to the type |
| Since | protocol version 1 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
| Errors | DataContractBoundsNotPresentError (10515) for an encryption key bound to a type that does not declare it; IdentityPublicKeyAlreadyExistsForUniqueContractBoundsError (40211) for a second key under 0 |
requiresIdentityDecryptionBoundedKey
The same for decryption keys.
| Where | document type |
| Value | 0 unique, 1 multiple, 2 multiple with a pointer to the latest |
| Default | absent: no decryption key may be bound to the type |
| Since | protocol version 1 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
| Errors | DataContractBoundsNotPresentError (10515) for a decryption key bound to a type that does not declare it; IdentityPublicKeyAlreadyExistsForUniqueContractBoundsError (40211) for a second key under 0 |
Example
Trimmed from the DashPay contract's contactRequest:
"contactRequest": {
"type": "object",
"documentsMutable": false,
"canBeDeleted": false,
"requiresIdentityEncryptionBoundedKey": 2,
"requiresIdentityDecryptionBoundedKey": 2,
"properties": {
"toUserId": {
"type": "array",
"byteArray": true,
"minItems": 32,
"maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 0
},
"encryptedPublicKey": {
"type": "array",
"byteArray": true,
"minItems": 96,
"maxItems": 96,
"position": 1
},
"senderKeyIndex": { "type": "integer", "minimum": 0, "position": 2 },
"recipientKeyIndex": { "type": "integer", "minimum": 0, "position": 3 }
},
"required": ["toUserId", "encryptedPublicKey", "senderKeyIndex", "recipientKeyIndex"],
"additionalProperties": false
}
An identity may bind any number of encryption and decryption keys to contactRequest, and the platform keeps a pointer to the newest of each. The request records the ids of the sender's and the recipient's keys in senderKeyIndex and recipientKeyIndex.
How it works
- An identity key may carry contract bounds: a contract, or one document type of a contract. An encryption or decryption key bound to a document type is only accepted when the type declares the matching keyword; otherwise it is refused with
DataContractBoundsNotPresentError(10515). The check runs when an identity is created with such a key, or updated to add one. The bound contract and document type must exist (DataContractNotPresentError, 10400;InvalidDocumentTypeError, 10406). - The value says how the identity's keys of that purpose, bound to the type, are kept:
0, unique: the identity holds at most one, and it can never be replaced. A second is refused (IdentityPublicKeyAlreadyExistsForUniqueContractBoundsError, 40211).1, multiple: the identity may hold any number.2, multiple with a pointer to the latest: any number, and the platform keeps a pointer to the most recently added one, so a client reads the current key in one step.
- Clients read keys bound to a contract or a document type with the
getIdentitiesContractKeysquery, which takes the identities, the contract, an optional document type name and the purposes. - Consensus reads these keywords only when keys are added. Nothing requires the writer of a document to hold such a key, and nothing checks which key encrypted a property. See encryptedFor for what consensus does check about encrypted values.
- Encryption and decryption keys are
MEDIUMkeys. See Identity Keys. - Keys bound to the whole contract, rather than one document type, are governed by the contract config keys of the same names. See Contract-Level Keys and config.
- Before protocol version 12, a decryption key bound to a document type was checked against
requiresIdentityEncryptionBoundedKeyby mistake. From 12 each purpose reads its own keyword. - From protocol version 14 an authentication key may also be bound to a contract or a document type, to limit what it may sign. That needs neither keyword. See Contract Bounds.
See also
- Identity Keys Deep Dive, for purposes, security levels and contract bounds
- encryptedFor, for declaring how an encrypted property was made
- Creation, Transfers and Trading, for the actions a key signs
- Contract-Level Keys and config, for the contract-wide key requirements
References (refersTo)
An identifier (a 32-byte id) can hold any value. refersTo says what it points at, and Platform then checks, whenever a document is created or replaced, that the thing it names exists: an identity, a data contract, a token, a document, or one key of an identity. Reach for it when a document only makes sense next to something else: a reply needs its post, a join request needs the proposal it joins, an encrypted message needs the key it was encrypted to. The check runs when the document is written. Nothing checks the reference again when its target changes later, and nothing resolves it for a reader.
| Where | An identifier property, at the top level or inside an object; the items of a typed array of identifiers, where every element is checked; for one form of identityPublicKey, an integer key id property; and a string or byte array property whose value a findBy function reveals (see Commit and reveal). ownerRefersTo and creatorRefersTo carry the same declaration at the document type level. |
| Value | An object: type, naming one target, with the keys that target takes; or an object holding only anyOf or only allOf (see Expressions). |
| Default | Absent: the identifier is not checked against anything. |
| Since | protocol version 14 |
| On update | Fixed: adding, removing or changing any part of a declaration is refused (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | ReferencedEntityNotFoundError (40120) when the target does not exist; the full list is under Errors. |
Example
"reply": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": true,
"properties": {
"postId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "deletableDocument", "documentType": "post" },
"position": 0
},
"mentions": {
"type": "array", "minItems": 0, "maxItems": 5, "uniqueItems": true,
"items": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "identity" }
},
"position": 1
},
"text": { "type": "string", "minLength": 1, "maxLength": 280, "position": 2 }
},
"required": ["postId", "text"],
"additionalProperties": false
}
postId must be the id of a post document of this contract that exists when the reply is written. Posts can be deleted, so the reference is a deletableDocument one. Each of the up to five mentions must be the id of an existing identity. The type carries six references at most: one for postId and five for mentions.
Targets
type names what the value points at.
type | The value must be | Keys it takes |
|---|---|---|
identity | the id of an existing identity | none |
contract | the id of an existing data contract | contractRequirements |
token | the id of an existing token | none |
permanentDocument | the id of an existing document of a type whose documents can never disappear; with findBy, one part of the key that finds it; with inList, one of the identifiers a list on it holds | documentType (required), contractId, findBy, where, inList, and beside a findBy function minimumAgeBlocks |
moderatedDocument | the id of an existing document of a type whose documents disappear only when the contract's moderators remove them, on the record | documentType (required), contractId, where |
deletableDocument | the id of an existing document of a type whose documents can disappear in any other way; with findBy, one part of the key that finds it | documentType (required), contractId, findBy, where, and beside a findBy function minimumAgeBlocks and consume |
identityPublicKey | an identity key that exists and is not disabled | keyIdProperty or identityProperty (one of them, required), keyRequirements |
A key that belongs to another target is refused when the contract is registered.
identity
The value is the id of an identity that exists. The check reads the identity's revision. Identities are never removed, so a validated reference never dangles.
contract
The value is the id of a data contract that exists. contractRequirements can ask more of it: that it declares elected moderation, has reached a certain age, belongs to the writer, and so on. Contracts are never deleted.
token
The value is the id of a token. The check reads the token's record, which is written when its contract is registered and never removed.
permanentDocument
The value is the id of a document of documentType, in this contract or in the one contractId names. The referenced type must be one whose documents can never disappear: canBeDeleted: false (not "onlyWhenConsumed"), no moderatorAbilities.delete and no ttl. None of those can change on a contract update and document types are never removed, so a validated permanent reference never dangles.
"reasons": {
"type": "array", "minItems": 0, "maxItems": 64, "uniqueItems": true,
"items": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "permanentDocument", "documentType": "reason" }
},
"position": 2
}
This is the moderation charters contract's submittedCharter.reasons: every element must be the id of a reason document, a type that is immutable and can never be deleted. With findBy the value is instead one part of a unique index key that finds the document, and with inList one of the identifiers a list on the document holds.
moderatedDocument
The same for a type whose documents disappear only when the contract's moderators remove them, and never without a trace: canBeDeleted: false, no ttl, and moderatorAbilities.delete with deleteKeepsRecord left at its default, true. Every removal then leaves a removal record under the contract, holding the document's id, its owner, the moderator, the time, the reason and a hash of the document, and nothing ever deletes it. So a validated moderated reference always resolves: to the document, or to the record of its removal, from which a moderator can restore the document as it was.
"postId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "moderatedDocument", "documentType": "post" },
"position": 0
}
A reply to a post its author can not delete, which the moderators can take down. The post must exist when the reply is written. Once a moderator removes it, the reply keeps pointing at it, and a replace of the reply is checked as if the reference were permanent: an edit of the reply's text passes, and the reference now resolves to the removal record. What a replace can not do is point a reply at a removed post, or write a new reply to one: a value the write sets must name a document in state.
A removed document has no values left but those its record keeps: its id, its owner, and the fields its type lists under moderatorAbilities.deleteKeepsFields. A where entry the replace checks again (its referring property changed, or it is a writer gate valued "$ownerId") is checked against the record when it compares the referenced $ownerId, $id or a property the type keeps (a kept path, or one inside a kept object), with the same absence rule as against a document in state, and refused with ReferencedDocumentRemovedError (40145) when it compares any other property; the replace can repoint the reference at a document in state, or leave that property unchanged until the post is restored. A writer gate (a where entry valued "$ownerId") is checked on every replace, so on a moderatedDocument reference it may compare only the referenced $ownerId or $id (InvalidContractStructure, 10231, otherwise), the only values sure to be in every record: a gate on anything else could refuse every replace once the post is removed. The value is kept as long as it is the one the stored document held, whatever else the replace changed in the same object or list. The value is always the document's id: findBy, inList and operands of an expression are refused, since a removal frees the unique index keys a findBy finds by.
A chained or composite query that joins through a moderated reference proves each removed document's record beside the documents it joins, and reports the removal with it.
An index of an indexOnly type may be preallocated through a moderated reference when the removal record keeps every key of its path: each where entry the index uses compares the referenced $id, $ownerId or a property the type keeps. The trees then outlive a removed post the way its record does, and a restore finds them in place.
deletableDocument
The same for a type whose documents can disappear without a record: deleted by their owner (canBeDeleted), deleted by a create that consumes them (canBeDeleted: "onlyWhenConsumed"), removed by the contract's moderators when deleteKeepsRecord is false, or removed by the platform when their ttl passes. A type whose documents only moderators remove, keeping records, is a moderatedDocument target instead, and a deletableDocument reference to it is refused. The document must exist when the referring document is written, and may be deleted afterwards.
Because the target may be gone, every replace of the referring document checks the reference again, whether or not the replace touched it. Once the target is deleted, the replace has to point the property at a document that exists or remove it. A required property cannot be removed, so a document whose required reference has lost its target can be replaced only after it is repointed; it can still be deleted. A single deletableDocument reference held by an immutable top-level property may be removed by a replace once its target is gone, an exception to the immutability rule.
identityPublicKey
One key of one identity, which must exist and not be disabled. Identity keys can be disabled but never removed, so a validated key reference never dangles, while a disabled key refuses new writes. The declaration comes in two forms, which differ in which property carries it:
-
On the identity property. The identifier holds the identity's id and
keyIdPropertynames the sibling integer property holding the key id. The moderation charters contract'sjoinRequest.recipientId:"recipientId": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "contentMediaType": "application/x.dash.dpp.identifier", "refersTo": { "type": "identityPublicKey", "keyIdProperty": "recipientKeyId", "keyRequirements": { "purpose": "decryption", "boundTo": "submittedCharter" } }, "position": 1 } -
On the key id property. The integer holds the key id and
identityPropertynames whose key it is. The property must declare exactly the range of a key id,"minimum": 0and"maximum": 4294967295. The same contract'sjoinRequest.senderKeyId, a key of the writer:"senderKeyId": { "type": "integer", "minimum": 0, "maximum": 4294967295, "refersTo": { "type": "identityPublicKey", "identityProperty": "$ownerId", "keyRequirements": { "purpose": "encryption", "boundTo": "joinRequest" } }, "position": 3 }
A key reference pairs the value with one key id, so it is refused on the elements of a typed array, as an operand of an expression and in ownerRefersTo or creatorRefersTo.
Keys
documentType
The name of the referenced document type, 1 to 64 letters, digits or underscores. Required on permanentDocument, moderatedDocument and deletableDocument, and refused on the other targets. For permanentDocument the type's documents must never disappear; for moderatedDocument they must disappear only through a moderator's recorded removal; for deletableDocument they must be able to disappear otherwise. With inList it is the type holding the list.
contractId
The contract holding documentType, as a base58 string or an array of 32 bytes. Absent means the declaring contract; naming the declaring contract's own id means the same. Only on the two document targets. A reference into another contract costs a billed fetch of that contract when a document is written, once per declaration: the elements of a typed array and the operands of an expression share it.
where
States what the referenced document must hold once found: each entry { "<referenced property>": "<referring value>" } must hold as an equality when the referring document is written. It takes 1 to 10 entries, on permanentDocument, moderatedDocument and deletableDocument references. where never finds the document: the value does, as its id, or findBy does. where is checked against the document found.
- The key is the referenced side: a property of the referenced document type, a dotted path for a nested one, or one of the referenced document's own identifiers:
$ownerId(its current owner, which follows it through transfers),$creatorId(its creator, which never changes, on types that record it, see System Properties) or$id(its id). A system name needs an identifier on the referring side. - The value is the referring side: a property of the declaring document type, a dotted path for a nested one, or
"$ownerId", the writer. An entry whose value is"$ownerId"is a write gate: only an identity whose id equals the referenced side may create or replace the document. A referring value may appear once inwhere. - Absence counts. Both sides absent agree; one side absent is a mismatch, as a different value is.
- Values, not keys. The two sides compare as values of their type: strings as text (so
""is not"\0"), byte arrays and identifiers as bytes (an identifier equals the same 32 bytes, however it is carried), integers as numbers whatever width they are carried at, floats exactly, bit for bit (so-0.0is not0.0). A value of any length compares, past the 255 bytes of an index key too. - The comparison reads the document already fetched for the existence check, so it costs nothing more. An entry that does not hold refuses the write with
ReferencedDocumentPropertyMismatchError(40127); no document found is stillReferencedEntityNotFoundError(40120). - No
"."and no functions. They find the document, so they belong infindBy, where a function is the key of a commit and reveal.
"submittedCharterId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": {
"type": "permanentDocument",
"documentType": "submittedCharter",
"where": {
"$ownerId": "$ownerId",
"targetContractId": "targetContractId"
}
},
"position": 1
}
This is the moderation charters contract's electedCharter.submittedCharterId: the proposal it names must be owned by the writer, and must be for the same targetContractId as the elected charter. The same contract's joinRequest.submittedCharterId declares "where": { "$ownerId": "recipientId" }: the proposal's owner must be the join request's recipientId. Other patterns: { "$ownerId": "authorId" } makes a like carry its post's owner in authorId, and { "$creatorId": "$ownerId" } lets only the referenced document's creator write.
Read as SQL, documentType is the FROM, the value (or findBy) the part of the WHERE that finds the row, and where the rest of the WHERE, checked on the row found:
SELECT * FROM submittedCharter
WHERE $id = :submittedCharterId -- the value, an id reference
AND $ownerId = :writer -- where
AND targetContractId = :targetContractId -- where
At registration both sides must exist and hold the same type of value (for integers, the same stored integer type). Neither side may be an object or a typed array, the referring side may not be the reference property itself, and the referenced side may not be transient (no stored document carries it; the referring side may be). An entry through which a preallocated index is keyed needs a referenced property whose every value fits an index key, at most 255 bytes: a string of at most 63 characters or with maxBytes at most 255, or a byte array of at most 255 bytes. An entry breaking one of these is refused with ReferencedDocumentPropertyAgreementInvalidError (40126).
The keyword propertyAgreement, whose pairs were keyed the other way ({ "<referring property>": "<referenced property>" }), was replaced by where before protocol version 14 shipped. A declaration still carrying it is refused on every parse, with a message saying to key where by the referenced document's property. "propertyAgreement": { "recipientId": "$ownerId" } is now "where": { "$ownerId": "recipientId" }, and a symmetric pair such as "$ownerId": "$ownerId" reads the same.
contractRequirements
What a contract reference requires of the referenced contract beyond existing. It holds at least one of these keys:
| Key | Value | Met when |
|---|---|---|
moderation | "elected" | the contract declares an elected moderation team |
"electionOpen" | the contract declares an elected team, and its own electionDelay, counted from the contract's creation, has passed at the block time of the write (or it declares no delay) | |
minimumAgeSeconds | integer, 1 to 4294967295 | the contract's recorded creation time is at least that many seconds before the block time of the write |
minimumSecondsSinceUpdate | integer, 1 to 4294967295 | the same, counted from the later of its creation and its last update |
owner | "self" | the contract is owned by the writer of the referring document |
"other" | the contract is owned by anyone else | |
readonly | true | the contract's config is readonly: it can never be updated again |
keepsHistory | true | the contract's config keeps history |
ownerProtected | true or false | the contract's elected moderation protects, or does not protect, its owner from the team; a contract without elected moderation meets neither |
A contract with no recorded creation time never meets minimumAgeSeconds or minimumSecondsSinceUpdate. The requirements are judged against the contract already fetched for the existence check, the writer and the block time, so they cost no further read. The first unmet one refuses the write with ReferencedContractRequirementNotMetError (40135); a contract that does not exist is still ReferencedEntityNotFoundError (40120).
The moderation charters contract uses both moderation values: a proposal (submittedCharter.targetContractId) needs a target that is elected, so teams can form while the target's election delay runs, and the charter that opens the election (electedCharter.targetContractId) needs it electionOpen:
"refersTo": { "type": "contract", "contractRequirements": { "moderation": "electionOpen" } }
owner is the one requirement judged against the writer, and a transfer or a purchase changes the owner without a write. On a type whose documents can be transferred or traded, a reference carrying owner is therefore checked again on every replace, so a new owner has to repoint it at a contract that meets the requirement for them, or remove it where it is optional. Such a reference may not sit under an immutable property of such a type, which could never be repointed. See Elected Moderation for the moderation values.
keyIdProperty and identityProperty
The two forms of identityPublicKey. A declaration takes exactly one of them.
keyIdProperty, on the identity property: the path of the integer property of the same document type holding the key id. It must exist, be an integer, and not carry a key reference of its own. If the identity is set and the key id is not, the write is refused withReferencedKeyIdPropertyInvalidError(40125).identityProperty, on the key id property: whose key the value is."$ownerId": the writer. The writer's existence is already proven, so the key fetch is the only read."$creatorId": the document's creator, only on a type that records creator ids. A document written before its type recorded them has none, and setting the key id on it is refused (40125).- The path of an identifier property of the same type, which must exist, be an identifier and not carry an
identityPublicKeyreference of its own. A key id set while that property is not set is refused (40125).
A stored key id may not be paired with a transient identity, since the key id alone names no key. Each rule is checked at registration and refused with ReferencedKeyIdPropertyInvalidError (40125).
keyRequirements
What an identityPublicKey reference requires of the key beyond existing and not being disabled. It holds at least one of:
purpose: the key's purpose, one ofauthentication,encryption,decryption,transfer,votingorowner.boundTo: a document type of the declaring contract. The key must be bound to exactly this contract and that document type; a key bound to the whole contract or to a contract group does not meet it.
At registration boundTo must name a document type of the contract, and one a key of the required purpose can be bound to: only authentication, encryption and decryption keys carry a document type bound, an encryption key only where the type declares requiresIdentityEncryptionBoundedKey, and a decryption key only where it declares requiresIdentityDecryptionBoundedKey (see Signing and Keys). The requirements are judged against the key already fetched, and the first unmet one refuses the write with ReferencedIdentityKeyRequirementNotMetError (40136). A missing key is still 40123 and a disabled one 40124.
findBy, inList, anyOf and allOf
findByfinds apermanentDocumentordeletableDocumentthrough the unique index of its type over exactly the properties it names, with the value as one part of the key. One entry may be a function, for a commit and reveal, beside which therefersTomay declareminimumAgeBlocksandconsume.inList, on apermanentDocumentwhosefindByis{ "$id": <property> }, names the list on that document the value must be in.anyOfandallOfcombine several targets in one declaration.
How it works
On create
When a document is created, every reference is checked against the current state: the ownerRefersTo or creatorRefersTo declaration first, then each property's. The first check that fails refuses the write with that check's error, naming the property by its path (postId, meta.charterId), an element by its list path (mentions[2] for the third) and the writer or creator reference as $ownerId or $creatorId. A refused write is still charged for the reads it made.
- A reference property the document leaves out is not checked. Whether it may be left out is up to
required. - Each check is a billed read: the identity, the contract (none for the declaring contract, which is already loaded), the token's record, the document by id or by
findBy, or the key. One write fetches a given document by id once, however many references name it. - On a typed array each element is checked in list order as a single reference would be. An element repeating an earlier one is not checked twice.
On replace
A replace checks a reference again only when its outcome could have changed:
| Declaration | Checked again on a replace when |
|---|---|
identity, token, contract | the value changed |
contract with an owner requirement, on a type whose documents can be transferred or traded | every replace |
permanentDocument, moderatedDocument | the value changed, or the referring value of a where entry changed |
any document reference with a where entry whose value is "$ownerId" | every replace |
permanentDocument with findBy, inList included | also when a property findBy reads changed |
deletableDocument, by id or with findBy | every replace |
a document reference with a findBy function | never: the whole declaration, its where included, is judged on the create alone (see Commit and reveal) |
identityPublicKey with keyIdProperty | the identity or the key id changed |
key id with identityProperty: "$ownerId" | every replace |
key id with identityProperty: "$creatorId" | the key id changed |
| key id with an identity property path | the key id or that property changed |
anyOf or allOf | one of its operands would be checked again; the whole expression is then checked |
A value changed when the replace set it differently, added it or removed it. Changes are tracked per top-level property, so a change anywhere in an object checks again every reference inside that object. When a typed array changed, only the elements the stored list did not hold are checked, unless a rule above that applies to every element does (the referring value of a where entry changed, a where entry valued "$ownerId", an owner requirement on such a type, or a deletableDocument target); then every element is checked. The rules for ownerRefersTo and creatorRefersTo are in Writer and Creator References.
The "every replace" rows exist because something the reference depends on can change without a write to the referring document: the writer after a transfer or purchase, or the target's existence for a deletable one. A moderated target that a moderator removed is not one of them: the reference resolves to its removal record, as described above.
Transfers, purchases, deletes and restores
- A transfer or a purchase checks no reference. A reference governs writing, not holding: a new owner meets the writer gates (a
whereentry valued"$ownerId",identityProperty: "$ownerId", anownerrequirement) on their first replace. - Deleting a referring document checks nothing. Deleting a referenced document does not look for documents referring to it: a
permanentDocumenttarget cannot be deleted at all, amoderatedDocumenttarget is removed by a moderator on the record the reference then resolves to, and adeletableDocumentreference meets its missing target on the referring document's next replace. AdeletableDocumenttarget can still refuse its owner's delete while documents refer to it: adeleteConstraintsrule counting them by the target's own id,{ "equal": [{ "countOf": ["vote", { "pollId": "$id" }] }, 0] }, needs a countable index on the referring property, and holds the target in place until the last of them is gone. - A document a moderator removed and later restores comes back as it was, without its references being checked again (see Restoring Documents).
The reference budget
Every reference is a billed read when a document is written, so a document type may carry at most 256 references per document. Registration counts:
- one for each property declaring
refersTo, whether an identifier or a key id; maxItemsfor each typed array whose elements declare it;- one for the type's
ownerRefersToorcreatorRefersTo; - each of these multiplied by the number of leaves when the declaration is an expression.
The reply above counts 6. A typed array of maxItems 15 whose elements declare an anyOf of two targets counts 30. A type over the budget is refused at registration with InvalidContractStructure (10231).
Rules at registration
A contract's declarations are checked when it is registered, and again for the whole contract on every update. A declaration that is malformed is refused by the meta-schema (JsonSchemaError, 10101) or by the parser (InvalidContractStructure, 10231). A declaration that is well formed but cannot hold is refused with the reference errors below: those are judged against the contract itself for its own document types, and against the stored contract for another contract's.
refersTosits on an identifier property or on theitemsof a typed array of identifiers. On the array itself it is refused: the declaration belongs on itsitems. The one exception is the key id form ofidentityPublicKey, on an integer property with exactly"minimum": 0and"maximum": 4294967295.- A declaration holds
typeand the keys its target takes, or a singleanyOforallOf. - A referenced
documentTypemust exist (ReferencedDocumentTypeNotFoundError, 40121). The three document references are disjoint, each type admitting exactly one: its documents must never disappear forpermanentDocument(ReferencedDocumentTypeDeletableError, 40122), disappear only through a moderator's recorded removal formoderatedDocument(ReferencedDocumentTypeNotModeratedError, 40143), and be able to disappear otherwise fordeletableDocument(ReferencedDocumentTypeNotDeletableError, 40131, for a type whose documents never disappear;ReferencedDocumentTypeModeratedError, 40144, for one whose documents only moderators remove on the record). The last is refused at registration only: adeletableDocumentreference promises less than such a type keeps, and a contract registered on a network beforemoderatedDocumentexisted may hold one, which stays writable. ApermanentDocumentwithinListwhose list is in the declaring contract is the exception: the parser checks its type and reports a deletable one asInvalidContractStructure(10231). - A reference that finds its document by the document's id (
permanentDocument,moderatedDocumentordeletableDocumentwithoutfindBy, or withinList, whose list's document is read by its id) may not name anindexOnlytype (ReferencedDocumentTypeIndexOnlyError, 40146): its documents exist only as index entries and can not be fetched by$id, so no write could check the reference. This holds for a type of the declaring contract and of another one alike. AfindByinto such a type is refused by its own rules instead, since the type has no unique index. - Every
whereentry must be one that can hold (40126), every key reference must fit the document type (40125), everyboundTomust name a type a key can be bound to (10231), and everyfindByand list must resolve. - An
immutableproperty may not hold adeletableDocumentreference that a replace could not remove: one inside an object, a typed array of them, or anydeletableDocumentfound byfindBy, except one whose key afindByfunction computes, which is checked on the create alone. A singledeletableDocumentreference by id that is itself an immutable top-level property is allowed, but only listed without a condition: a replace the condition left free could set it to another document once it was cleared. Acontractreference with anownerrequirement may not sit under an immutable property of a type whose documents can be transferred or traded. All refused withInvalidContractStructure(10231); see Mutability. - The type stays within the reference budget.
- The earlier spellings
lookup,propertyAgreementandtype: "listElement", replaced byfindBy,whereandinListbefore protocol version 14 shipped, are refused: by the meta-schema on registration and update (10101), and by the parser on every parse (10231), with a message naming what replaced each. - On an update, every existing declaration must be unchanged (10246). A document type the update adds may declare any reference.
Errors
| Error | Code | When |
|---|---|---|
ReferencedEntityNotFoundError | 40120 | Write: the identity, contract, token or document does not exist, findBy finds no document, or a value is not in its list. |
ReferencedDocumentTypeNotFoundError | 40121 | Registration: documentType does not exist, or contractId names no contract. |
ReferencedDocumentTypeDeletableError | 40122 | Registration: a permanentDocument reference, with inList or without, names a type whose documents can disappear. |
ReferencedIdentityKeyNotFoundError | 40123 | Write: the identity has no key with that id, or the identity does not exist. |
ReferencedIdentityKeyDisabledError | 40124 | Write: the key is disabled. |
ReferencedKeyIdPropertyInvalidError | 40125 | Registration: keyIdProperty or identityProperty names a property that does not fit, $creatorId on a type that records no creator ids, or a stored key id paired with a transient identity. Write: a key id without its identity, or an identity without its key id. |
ReferencedDocumentPropertyAgreementInvalidError | 40126 | Registration: a where entry names a missing, transient, object or typed array property, or two properties of different types, or $creatorId on a type that does not record it, or keys a preallocated index by a property that can hold more than 255 bytes. |
ReferencedDocumentPropertyMismatchError | 40127 | Write: the document found does not meet a where entry. |
ReferencedDocumentTypeNotDeletableError | 40131 | Registration: a deletableDocument reference names a type whose documents can never disappear. |
ReferencedContractRequirementNotMetError | 40135 | Write: the referenced contract exists but does not meet a contractRequirements entry. |
ReferencedIdentityKeyRequirementNotMetError | 40136 | Write: the key exists and is enabled but does not meet a keyRequirements entry. |
ReferencedDocumentLookupInvalidError | 40137 | Registration: a findBy into another contract's document type cannot resolve, for example because no unique index of it is over exactly the properties findBy names. |
ReferencedDocumentListInvalidError | 40138 | Registration: an inList list on another contract's document type does not qualify. |
ReferencedDocumentTypeNotModeratedError | 40143 | Registration: a moderatedDocument reference names a type whose documents can disappear otherwise than through a moderator's recorded removal, or never disappear. |
ReferencedDocumentTypeModeratedError | 40144 | Registration: a deletableDocument reference names a type whose documents disappear only through a moderator's recorded removal. |
ReferencedDocumentRemovedError | 40145 | Write: a replace kept a moderatedDocument reference whose document a moderator removed, and a where entry checked again compares a property the removal record does not keep. |
ReferencedDocumentTypeIndexOnlyError | 40146 | Registration: a reference that finds its document by id (no findBy, or with inList) names an indexOnly type, whose documents can not be fetched by id. |
The registration errors name the declaration as <documentType>.<property>, <documentType>.<property>[] for typed array elements, <documentType>.$ownerId or <documentType>.$creatorId for the writer and creator references, and add the operand for a leaf of an expression (resignation.memberId.anyOf[1]). When a document is written, the referenced document type is looked up and its kind checked again as a safeguard (40121, 40122, 40131, 40143), but a registered contract cannot fail those checks later: contracts and document types are never removed, and neither the deletion flags nor ttl nor moderatorAbilities can change. Nor can indexOnly, so the indexOnly rule (40146) is checked at registration only. The other codes in the range (40128 to 40130, 40132 to 40134, 40139 to 40142) belong to other keywords. See Error Codes.
More forms of reference
- findBy: a
permanentDocumentordeletableDocumentfound through the unique index of its type over exactly the propertiesfindBynames, with the value as one part of the key. The value can be an identity (a member, an author) instead of a document id. - Expressions:
anyOfandallOf, several targets combined in one declaration. - List Elements:
inListon apermanentDocument, a value that must be one of the identifiers a list on another document holds. - Writer and Creator References:
ownerRefersToandcreatorRefersTo, the same declaration applied to the document's writer or creator instead of a property. - References on the elements of a typed array: a declaration on
items, checked for every element.
See also
- Document References in the Documents chapter, with the internals.
- Elected Moderation for the moderation charters contract, the first user of most reference forms.
- Typed Arrays, System Properties, Mutability, Deletion, Time To Live (ttl) and Creation, Transfers and Trading for the keywords references read.
- distinctFrom, which requires an identifier to differ from another, and encryptedFor, which names the key references an encrypted property was made with.
- Values of Referenced Documents, where an index holds a value of the document a
permanentDocumentormoderatedDocumentreference points at. - Contract Keywords for the conventions these pages use.
Finding a Document by Its Properties (findBy)
findBy lets a document reference find its target by the values of the target's properties instead of by its id. The property's value is then one part of a key, the rest comes from the referring document or its writer, and the reference holds if the referenced document type's unique index over exactly those properties finds a document. Reach for it when the natural value is an identity (a member, an author, a recipient) and the rule is "this identity has a document of that kind": with an id reference the writer would have to find the document's id first, and the id would say nothing about who it belongs to.
| Where | Inside a permanentDocument or deletableDocument refersTo declaration: on an identifier property, on the items of a typed array of identifiers, as an operand of an expression, or in ownerRefersTo and creatorRefersTo. With a function, also on a string or byte array property. |
| Value | An object of 1 to 10 entries { "<referenced property>": <source> }. Each key is a property of documentType, by its name there (system ones such as $ownerId included). Each source is "." (the reference's own value), "$ownerId" (the writer), a property path of the referring document type, or at most one function. The keys must be exactly the properties of one unique index of documentType, in any order. Beside a function the refersTo may also declare minimumAgeBlocks and consume. With inList, findBy is { "$id": <property> } instead. |
| Since | protocol version 14 |
| On update | Fixed, like the rest of refersTo: adding, removing or changing findBy is refused (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | ReferencedEntityNotFoundError (40120) when no document is found; at registration ReferencedDocumentLookupInvalidError (40137) for a document type of another contract, InvalidContractStructure (10231) for one of the same contract. A function adds DocumentReferencePreimageInvalidError (10423) and ReferencedDocumentRequirementNotMetError (40142). |
Example
The moderation charters contract's joinRequest type, which is immutable and can never be deleted, has this unique index: one join request per proposal and owner.
"indices": [
{
"name": "bySubmittedCharter",
"properties": [{ "submittedCharterId": "asc" }, { "$ownerId": "asc" }],
"unique": true
}
]
The same contract's electedCharter lists its team in members, each of whom must have asked to join:
"members": {
"type": "array", "minItems": 0, "maxItems": 15, "uniqueItems": true,
"items": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"distinctFrom": "$ownerId",
"refersTo": {
"type": "permanentDocument",
"documentType": "joinRequest",
"findBy": { "submittedCharterId": "submittedCharterId", "$ownerId": "." }
}
},
"position": 2
}
Every member must be the owner of a joinRequest whose submittedCharterId equals this elected charter's submittedCharterId. findBy names submittedCharterId and $ownerId, exactly the properties of bySubmittedCharter, so that index finds the join request. For each element the key is submittedCharterId read from the elected charter and $ownerId filled with the element itself ("."). The same form works on a single identifier property, where "." is the property's value.
Read as SQL, a reference is a query that must return a row. documentType is the FROM, findBy the part of the WHERE that finds the row through a unique index, where the rest of the WHERE, checked on the row found, minimumAgeBlocks an age condition and consume a delete of the row. The declaration above reads, for each element:
SELECT * FROM joinRequest
WHERE submittedCharterId = :submittedCharterId -- findBy, from the elected charter
AND $ownerId = :element -- findBy, "."
How it works
Which index finds the document
findBy does not name an index. The index is the unique index of the referenced document type whose properties are exactly the properties findBy names, in any order. It must not bucket its first property with timeRange or integerRange, and the referenced type must not be indexOnly. A type's indexes can never be added, removed or changed by a contract update, so the same findBy always resolves to the same index.
Registration refuses a findBy that names no such index, listing the unique indexes the type has. Into the joinRequest type above, { "submittedCharterId": "submittedCharterId" } leaves out $ownerId:
"joinRequest" has no unique index over exactly (submittedCharterId): findBy must name every
property of one of its unique indexes and nothing else (bySubmittedCharter (submittedCharterId, $ownerId))
A findBy naming exactly the properties of an index that is not unique is refused too, since it could find several documents. joinRequest also has a byOwner index on $ownerId alone, which is not unique, so { "$ownerId": "." } is refused:
index "byOwner" of "joinRequest" over ($ownerId) is not unique: findBy must find at most one document
Assembling the key
When the referring document is created or replaced, a key is assembled for each value (each element of a typed array), one part per entry of findBy:
".": the value being checked, the property's value or the element."$ownerId": the referring document's owner, the writer.- a property path (
"submittedCharterId","meta.charterId"): the referring document's value at that path. - a function: the hash of the params it reads.
The index is queried for at most one document, billed as a document fetch. If it finds one, the reference holds, and any where entries beside findBy are checked against that document (ReferencedDocumentPropertyMismatchError, 40127). If it finds none, the write is refused with ReferencedEntityNotFoundError (40120) naming the property, or the element by its list path (members[1]); the error's target reads "found by" and the properties findBy names.
Permanent and deletable references
A permanentDocument found by findBy never dangles. Its referenced documents are never deleted, and registration makes sure the key of every one of them is fixed once written (see below), so the document a key found stays there. A replace checks it again only:
- when the property itself changed (for a typed array, only the elements the stored list did not hold);
- when a property
findByreads changed, and then every value, every element included; - when the referring value of a
whereentry changed, or on every replace for awhereentry whose value is"$ownerId".
A key part read from "$ownerId" never triggers a check: only a type whose documents keep their writer may read it.
A deletableDocument found by findBy promises less. Once the document a key found is deleted, a new document filed later under the same key makes the reference hold again, with different content, where an id reference to a deleted document stays dead. So it means "a document with this key exists now", which is what a membership gate needs. Every replace checks it again, whether or not the replace touched it, and no immutable property may hold one. The moderation charters contract's resignationRequest uses one to accept a writer who has an addedModerator document for the charter at the time of writing, as the alternative to being an elected member (see Writer and Creator References).
findBy can also reach into another contract. This recipientId must be an identity that has a DashPay profile when the document is written:
"recipientId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": {
"type": "deletableDocument",
"contractId": "Bwr4WHCPz5rFVAD87RqTs3izo4zpzwsEdKPWUT1NS1C7",
"documentType": "profile",
"findBy": { "$ownerId": "." }
},
"position": 0
}
DashPay's profile type has a unique index on $ownerId alone, which finds the profile, and profiles can be deleted, so the reference is a deletableDocument one.
What findBy cannot do
The value of a reference found by findBy is a key part, not a document id. So such a reference cannot be the join property of a chained query or of a composite join by id, and a preallocated index is never bound through one (see Index-Only Types). It counts as one reference per value against the reference budget, like any other.
Commit and reveal
One findBy entry may hold a function: "<referenced property>": { "function": "sys.hash.sha256d", "params": [...] } says the referenced document's property holds the SHA-256 of the SHA-256 of the params' bytes, joined in order. The platform fills that property of the key with the hash to find the document, like any other entry. The document it finds is a commitment made earlier, one that stored that hash, so the document being created may exist only while a commitment to values it carries exists. This is how a name registration that preorders a salted hash works, written as a declaration on the salt:
"preorderSalt": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 4,
"refersTo": {
"type": "deletableDocument",
"documentType": "preorder",
"findBy": {
"saltedDomainHash": {
"function": "sys.hash.sha256d",
"params": ["preorderSalt", "normalizedLabel", { "const": "." }, "parentDomainName"]
}
},
"where": { "$ownerId": "$ownerId" },
"minimumAgeBlocks": 1,
"consume": true
}
}
The salt must reveal a preorder whose saltedDomainHash is the sha256d of the salt, the normalized label, a dot and the parent. findBy names saltedDomainHash alone, the whole of the preorder type's unique saltedHash index. The where entry makes it the writer's own preorder, minimumAgeBlocks: 1 one created in an earlier block, and consume: true deletes it with the create. As SQL:
SELECT * FROM preorder
WHERE saltedDomainHash = -- findBy
sha256d(:preorderSalt || :normalizedLabel || '.' || :parentDomainName)
AND $ownerId = :writer -- where
AND $createdAtBlockHeight <= :createHeight - 1 -- minimumAgeBlocks
-- then the create deletes the row found -- consume
- Params. A property path reads the document being created, the property carrying the reference included;
{ "const": text }is fixed text of 1 to 64 bytes;"."is the value carrying the reference where it has no path (each element of a typed array, the writer, the creator); 1 to 16 params. Strings count as their UTF-8, byte arrays as their bytes, identifiers as their 32 bytes. Two values of no fixed length must be separated by a one-byteconst, and a value holding that byte is refused, so the joined bytes split back one way only. - One function.
findByholds at most one function, and the property it fills must be a byte array of exactly 32 bytes. - Carrier. A string or byte array property may carry a
refersToonly this way, and keeps its type. The value carrying the reference fills the key exactly once, as a"."entry or a param; a param names a property by its path, never as".". InownerRefersToandcreatorRefersTothe value may be left out beside a function. - Judged once. A function's key is checked when the document is created, never on a replace, so every stored value it reads must be fixed once written, the carrier included (listed under
immutableon a mutable type, and required when it is adeletableDocumentreference by id, which a replace may otherwise clear once its document is deleted). Thewhereentries are judged with it, so each referring property they name must be fixed once written too, or transient. On a type whose documents can be replaced it may not be an operand of ananyOf, which would then hold on every replace. Params may be transient: they are read from the create. The hash is billed as the double SHA-256 it is, by the blocks it hashes, beside the document fetch.
minimumAgeBlocks and consume sit on the refersTo, beside findBy: they describe the document findBy finds, and need a function in it.
minimumAgeBlocks: the commitment's$createdAtBlockHeightis at least that many blocks below the create's height. The commitment type must list$createdAtBlockHeightinrequired.consume: true: the create deletes the commitment, its storage refunded to its owner. Only on adeletableDocumentreference with thewhereentry"$ownerId": "$ownerId", into a type of the same contract whose owner may delete its documents (canBeDeleted: true) or whose documents only a consume deletes (canBeDeleted: "onlyWhenConsumed", see Deletion), that declares no delete token cost or delete action fee and nodeleteConstraints(no delete transition charges or judges them), and that requires no stricter signature security level than the revealing type. A contract-bound key signing the create must be allowed to act on the consumed type too (ContractBoundedKeyOutOfBoundsError, 20014).
A create missing a param, or with a value holding its separator, is refused before any read (DocumentReferencePreimageInvalidError, 10423). No commitment is ReferencedEntityNotFoundError (40120), another identity's ReferencedDocumentPropertyMismatchError (40127), and one too young ReferencedDocumentRequirementNotMetError (40142). The Documents chapter has the full rules.
Rules at registration
On the referring side, checked for every contract by the parser (InvalidContractStructure, 10231):
findByis only allowed onpermanentDocumentanddeletableDocumentreferences.- The reference's own value is read exactly once, as a
"."entry or as a param of a function. Without it every value would find the same document. OnlyownerRefersToorcreatorRefersTomay leave it out, beside a function. - Every other source is
"$ownerId", a property path or a function. No other$name is accepted, and a path may not name the reference property itself: write"."for that. - A property an entry reads must exist on the referring type, be required (and so must every object around it), not be
transientor inside a transient object, and hold a single value, not an object or an array.findBynever runs with a missing key part, and a reader can assemble the same key from the stored document. A function's params follow their own rules: they may be transient or optional, since they are read from the create alone. - A
"$ownerId"source needs a referring type whose documents can be neither transferred nor traded: a transfer or a purchase would move the writer part of the key without a write. - A
deletableDocumentfound byfindBymay not sit under animmutableproperty, alone or as an operand of an expression: every replace checks it again, so once its document is deleted the property would have to change. One whose key a function computes is the exception: it is checked on the create alone, and its carrier must be fixed once written. { "$id": ... }is allowed only withinList, as the only entry.
On the referenced side, refused with InvalidContractStructure (10231) for a document type of the declaring contract and with ReferencedDocumentLookupInvalidError (40137) for one of another contract:
- The keys are exactly the properties of one unique index of the type, by their names on the referenced side (system properties such as
$ownerIdincluded), in any order, and nothing else (see Which index finds the document). That index does not bucket its first property by atimeRangeor anintegerRange, the referenced type is notindexOnly, and no index property istransient. - Each source holds the same type of value as the property it fills.
"."and"$ownerId"are identifiers. - The key stays with the document it found. Every schema property of the index must be fixed once written: the referenced type is immutable (
documentsMutable: false), or the property's top-level property is listed underimmutableand is no optionaldeletableDocumentreference by id, which a replace may clear once its document is deleted.$ownerIdmay be a key part only where the referenced documents can be neither transferred nor traded.$updatedAtand its block height forms may be one only where they can be neither replaced, transferred nor traded, and$transferredAtand its forms only where they can be neither transferred nor traded.$id,$creatorId,$createdAtand its forms never change.
The referenced type must exist (ReferencedDocumentTypeNotFoundError, 40121). A permanentDocument found by findBy in a type whose documents can disappear is refused with ReferencedDocumentTypeDeletableError (40122), and a deletableDocument found by findBy in one whose documents cannot with ReferencedDocumentTypeNotDeletableError (40131).
Index definitions and the flags these rules read cannot change on a contract update, so a findBy that registered keeps resolving, through the same index.
The keyword lookup, which named the index and mapped its properties under keys, was replaced by findBy before protocol version 14 shipped. A declaration still carrying it is refused on every parse, with a message saying to map every property of the unique index to its source and leave the index name out.
See also
- References (refersTo) for the targets,
whereand the replace rules. - Expressions, where a reference found by
findBymay be an operand, and Writer and Creator References, where"."is the writer or the creator. - Found by a unique index in the Documents chapter, with the internals.
- Indexes (indices), Time-Range Indexes, Mutability and transient for the keywords the rules read.
Expressions
A refersTo declaration may combine several targets instead of naming one. anyOf holds when at least one of its operands holds, and allOf when every operand holds for the same value. Reach for an expression when a value may point at one of several kinds of thing ("an elected member or an added one"), or must satisfy several references at once ("a member of the team who also has a profile"). Both combinators take the same operands, follow the same limits and are checked the same way; they differ only in when they stop.
anyOf
| Where | In place of a single target: in refersTo on an identifier property or on the items of a typed array of identifiers, in ownerRefersTo and creatorRefersTo, and as an operand of an allOf. Not on a key id property. |
| Value | { "anyOf": [ ... ] }, the declaration's only key: 2 to 4 distinct operands. |
| Since | protocol version 14 |
| On update | Fixed (IncompatibleDocumentTypeSchemaError, 10246), including a change of operand order. |
| Errors | None of its own: when no operand holds, the write is refused with the last operand's error, for example ReferencedEntityNotFoundError (40120). |
The operands are checked in the order they are listed, and the first that holds decides: the rest are not read. When none holds, the write is refused with the error of the last one.
The moderation charters contract lets a member of a seated team resign with a resignationRequest. Its writer must be on the team, either elected or added later:
"ownerRefersTo": {
"anyOf": [
{
"type": "permanentDocument",
"documentType": "electedCharter",
"findBy": { "$id": "electedCharterId" },
"inList": "members"
},
{
"type": "deletableDocument",
"documentType": "addedModerator",
"findBy": { "electedCharterId": "electedCharterId", "memberId": "." }
}
]
}
The writer must be one of the members of the electedCharter this document's electedCharterId names, or have an addedModerator document for that charter that exists now. Here the expression is the writer's reference (see Writer and Creator References); the same declaration works on an identifier property, where it judges the property's value.
allOf
| Where | In place of a single target: in refersTo on an identifier property or on the items of a typed array of identifiers, in ownerRefersTo and creatorRefersTo, and as an operand of an anyOf. Not on a key id property. |
| Value | { "allOf": [ ... ] }, the declaration's only key: 2 to 4 distinct operands. |
| Since | protocol version 14 |
| On update | Fixed (IncompatibleDocumentTypeSchemaError, 10246), including a change of operand order. |
| Errors | None of its own: the write is refused with the first failing operand's error, for example ReferencedEntityNotFoundError (40120). |
The operands are checked in the order they are listed, and the first that fails decides: the rest are not read, and the write is refused with that operand's error.
"memberId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": {
"allOf": [
{
"type": "permanentDocument",
"contractId": "EG7RGfV8fDTayC2FyVr8HwdpJh3fXDbVztcfE94UmN88",
"documentType": "electedCharter",
"findBy": { "$id": "electedCharterId" },
"inList": "members"
},
{
"type": "deletableDocument",
"contractId": "Bwr4WHCPz5rFVAD87RqTs3izo4zpzwsEdKPWUT1NS1C7",
"documentType": "profile",
"findBy": { "$ownerId": "." }
}
]
},
"position": 1
}
This property of a hypothetical moderator directory, next to an electedCharterId identifier property, must be one of the elected members of that charter in the moderation charters contract, and must have a DashPay profile when the document is written. The list check comes first, so a value that is not a member is refused without the profile being read.
Operands
An operand is a leaf or a nested expression.
Leaves. A leaf is an ordinary target declaration with its own keys, one of:
The first two are existence checks against things that are never deleted (inList reads a list that never changes on a document that is never deleted), so an expression made only of them holds for good once it holds. A deletableDocument found by findBy may find nothing later, which is why an expression holding one is checked on every replace.
The other targets are refused as operands, since they do not compose with other operands:
deletableDocumentby id: once its document is deleted a replace may clear the property, which assumes the property refers to that one target.identityPublicKey, in either form: it pairs the value with a key id that no other operand reads.contract: its requirements are gates judged against the block time and the writer, not an existence check, and a contract id is never also an identity or document id.token: a token id is never also an identity or document id.
Nesting. An operand may be an expression of the other combinator: an allOf inside an anyOf, or an anyOf inside an allOf. An anyOf directly inside an anyOf, or an allOf inside an allOf, is refused, since it says what one flat list says.
A where, a findBy or an inList belongs to its leaf, inside it: the expression itself holds nothing but its combinator.
How it works
- Order decides the error. A refusal is always the error a leaf declared alone would give, naming the property (or the element) as a single reference would. So the author's order decides which error a writer sees: put the most general operand of an
anyOflast, and the cheapest or most telling operand of anallOffirst. - Every read is billed. A value the second operand of an
anyOfholds for also pays for the first operand's read. AnallOfwhose first operand fails reads nothing more. whereis per leaf. Awhereis checked only against its own leaf's document. A value whose first leaf fails itswherecan still be accepted through a second leaf that has none.- Typed arrays. On the
itemsof a typed array, each element meets the expression on its own. - Replace. An expression is checked again when any of its leaves would be checked again alone (see On replace), and then it is evaluated whole, since which operands hold may have changed. An expression holding a
deletableDocumentfound byfindByis therefore checked on every replace, except a leaf whose key afindByfunction computes, which is judged on the create alone and holds on a replace without a read. - Budget. Every leaf counts against the reference budget, since each may be read for each value. An
anyOfof two leaves on a typed array ofmaxItems15 counts 30. - Queries. An expression cannot be the join property of a chained query or of a composite join by id, and a
preallocatedindex is never bound through one.
Rules at registration
- The combinator is the declaration's only key, and lists at least two operands: a single one is declared on its own. No two operands of one list may be alike; a leaf naming the declaring contract's id in
contractIdis the same as one leaving it out. - A list holds at most 4 operands, and any path from the declaration to a leaf passes through at most 4 combinators. The
anyOfholding anallOfis 2 deep. - Every leaf is one of the admitted targets, and is checked exactly as the same target declared alone: its document type, the type's deletability, its
whereand itsfindByor list. Every leaf must pass, since each has to be a declaration that could hold. The errors name the leaf by where it sits:refersTo anyOf[1].allOf[1] findBy: ...from the parser,resignation.memberId.anyOf[1].allOf[1]from the registration check. - An
immutableproperty may not hold an expression with adeletableDocumentleaf found byfindBy, unless afindByfunction computes that leaf's key (InvalidContractStructure, 10231). - On a document type whose documents can be replaced, a leaf whose key a
findByfunction computes may not be an operand of ananyOf: judged on the create alone, it would hold on every replace, whichever operand held on the create. - Every leaf counts against the reference budget of 256.
A malformed expression is refused by the meta-schema (JsonSchemaError, 10101) or the parser (InvalidContractStructure, 10231); a leaf that cannot hold, with the reference error it would get alone (see Errors). On an update, any change is refused (10246): an operand added, removed, changed or moved, anyOf swapped for allOf, or a single target turned into an expression or back.
See also
- References (refersTo) for the targets and the replace rules.
- findBy, List Elements and Writer and Creator References, whose declarations are the usual leaves and holders of an expression.
- Reference expressions in the Documents chapter, with the internals.
- propertyConstraints, whose rules also combine with
anyOfandallOfbut compare the document's own values instead of reading other state.
List Elements (inList)
A permanentDocument reference with inList says the value must be one of the identifiers a list on another document holds. inList names the list, and findBy: { "$id": <property> } names the document holding it, by the id another property of the referring document holds. Reach for it when membership is written down once as a list on a document that never changes, such as the members elected with a charter, and other documents must name one of them.
| Where | refersTo on an identifier property, or on the items of a typed array of identifiers, where every element must be listed; an operand of an expression; or ownerRefersTo and creatorRefersTo, where the writer or the creator must be listed. |
| Value | inList, the path of a typed array of identifiers on documentType, in a "type": "permanentDocument" declaration with documentType (the type holding the list), findBy (exactly { "$id": "<the identifier property holding the document's id>" }), optionally where (up to 10 entries) and contractId. |
| Since | protocol version 14 |
| On update | Fixed, like the rest of refersTo (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | ReferencedEntityNotFoundError (40120) when the value is not in the list or the list's document is not found; ReferencedDocumentPropertyMismatchError (40127) for a where entry; at registration ReferencedDocumentListInvalidError (40138) for a list of another contract. |
Example
The moderation charters contract lets the leader of a seated team take an elected member off it with a removedModerator document:
"removedModerator": {
"type": "object",
"documentsMutable": false,
"canBeDeleted": true,
"properties": {
"electedCharterId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": {
"type": "permanentDocument",
"documentType": "electedCharter",
"where": { "$ownerId": "$ownerId" }
},
"position": 0
},
"memberId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"distinctFrom": "$ownerId",
"refersTo": {
"type": "permanentDocument",
"documentType": "electedCharter",
"findBy": { "$id": "electedCharterId" },
"inList": "members"
},
"position": 1
}
},
"required": ["$createdAt", "electedCharterId", "memberId"],
"additionalProperties": false
}
electedCharterId must name an elected charter the writer owns, so only the team's leader can write one. memberId must be one of the members of the electedCharter document whose $id this document's electedCharterId holds, so only an elected member can be removed. electedCharter is immutable and can never be deleted, so its members never change.
Read as SQL, with the list as a table of its elements:
SELECT * FROM electedCharter
WHERE $id = :electedCharterId -- findBy
AND :memberId IN (members) -- inList
How it works
When the referring document is created or replaced:
- The document holding the list is the one whose
$idequals the value of the propertyfindByreads. It is fetched by id, once per write: in the example theelectedCharterIdreference and the list reference share one fetch. The list is collected once, so checking many values against it costs one read. - The
whereentries, if any, are checked against that document, as for any document reference (ReferencedDocumentPropertyMismatchError, 40127). - The value must be in the list. On a typed array every element must be; on the writer's or creator's reference, the writer or creator must be.
A value not in the list, a list document that does not exist, or a value set while the $id property is not, refuses the write with ReferencedEntityNotFoundError (40120), naming the property, or an element by its list path (memberIds[1] for the second):
electedCharter 7kX...: members [Alice, Bob]
removedModerator { electedCharterId: 7kX..., memberId: Alice } -> accepted
removedModerator { electedCharterId: 7kX..., memberId: Carol } -> refused, 40120:
referenced list element (members of the electedCharter document electedCharterId names)
<Carol> not found for path memberId
A replace checks the reference again when its value changed, or when a property findBy or where reads changed: the $id property among them, since it may now name another document, whose list is then checked against every value. A where entry whose value is "$ownerId" is checked on every replace. When only a typed array of values changed, the elements the stored list already held are not checked again. Nothing else can make a validated value unlisted: the list's document is never deleted and its list never changes.
Rules at registration
inListis only allowed on apermanentDocumentreference, and needsfindBy.findByis exactly{ "$id": "<property>" }, and{ "$id": ... }is allowed only withinList. The property is an identifier property of the referring document type: not$ownerId(no document has the writer's id), not"."(the value is the list element) and not a typed array (one document holds the list). It must be stored, so it and every object around it are nottransient, and a reader can tell from the stored document which list the value was checked against. It may be optional. It needs norefersToof its own; if it has one, that must be a reference by id todocumentTypein the list's contract, or the value could never be in the list. Refused withInvalidContractStructure(10231) otherwise.wheremay not compare$idagain, nor the propertyfindByreads. Its entries follow the rules of everywhere(ReferencedDocumentPropertyAgreementInvalidError, 40126).documentTypemust exist (ReferencedDocumentTypeNotFoundError, 40121), and its documents must never disappear:canBeDeleted: false, nomoderatorAbilities.deleteand nottl.inListis a property path (a dotted one for a nested list, never a$name) of a stored typed array of identifiers ondocumentType, and the list must be fixed once a document is written: the type is immutable (documentsMutable: false), or the list's top-level property is listed underimmutablewithout a condition. A list frozen only under a condition does not qualify: while the condition does not hold, a replace could change it.- The checks on
documentTypeandinListare refused withInvalidContractStructure(10231) for a document type of the declaring contract. For one of another contract, a deletable type is refused withReferencedDocumentTypeDeletableError(40122) and a list that does not qualify withReferencedDocumentListInvalidError(40138). - Each value counts one against the reference budget,
maxItemsfor a typed array.
The type listElement, which found the document holding the list through a propertyAgreement pair with $id on its referenced side, was replaced by inList on a permanentDocument before protocol version 14 shipped. A declaration still using it is refused on every parse, with a message saying to declare a permanentDocument with findBy: { "$id": ... } and inList.
See also
- References (refersTo) for
documentType,contractId,whereand the replace rules. - findBy, which finds a referenced document by the values of a unique index instead of by its id.
- Expressions and Writer and Creator References, where a list reference is a common operand or target.
- An element of a list in the Documents chapter, with the internals.
- Typed Arrays and Mutability for the list and the rules that keep it fixed.
Writer and Creator References
A property's refersTo judges a value the writer chose. ownerRefersTo and creatorRefersTo judge an identity of the document instead: its writer ($ownerId) or its creator ($creatorId). Each takes the same declaration a property's refersTo takes, limited to the targets an identity's id can be. Reach for them to say who may write a document of a type: "only a member of this team", "only someone with a profile". A type declares at most one of the two: ownerRefersTo when its documents stay with their owner, creatorRefersTo when they can be transferred or traded.
Both are checked the way a property's reference is: the same targets, keys, errors and replace rules, with the identity as the value. They count as one reference each (times the leaves of an expression) against the reference budget, and the check runs before the properties' references.
ownerRefersTo
| Where | The document type, at the top level of its schema. Only on a type whose documents can be neither transferred nor traded. |
| Value | A refersTo declaration whose value is the writer: identity, a permanentDocument or deletableDocument found by findBy, a permanentDocument with inList, or an anyOf or allOf whose leaves are all of these. |
| Since | protocol version 14 |
| On update | Fixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | The error the target reports for a property, at the path $ownerId: ReferencedEntityNotFoundError (40120) when findBy finds nothing or the writer is not in the list, ReferencedDocumentPropertyMismatchError (40127) for a where entry. |
Example
"post": {
"type": "object",
"ownerRefersTo": {
"type": "deletableDocument",
"contractId": "Bwr4WHCPz5rFVAD87RqTs3izo4zpzwsEdKPWUT1NS1C7",
"documentType": "profile",
"findBy": { "$ownerId": "." }
},
"properties": {
"text": { "type": "string", "minLength": 1, "maxLength": 280, "position": 0 }
},
"required": ["text"],
"additionalProperties": false
}
Only an identity with a DashPay profile may create a post. In findBy, "." is the writer: DashPay's unique index on $ownerId must find a profile owned by them. Profiles can be deleted, so the target is a deletableDocument one and every replace asks again: a writer whose profile is gone can no longer edit their posts, though they can still delete them.
The moderation charters contract's resignationRequest combines two targets: its writer must be one of the elected members of the charter the request names, or have an addedModerator document for it that exists now. The declaration is shown under anyOf.
How it works
- On create, the writer is checked against the target exactly as a property's value would be. An
identitytarget always holds and reads nothing: the transition has already proved that the writer exists. - In
findBy,"."is the writer, and so is a"$ownerId"source. In awhere, an entry valued"$ownerId"names the same writer. - On replace, the declaration is checked again when a property its
findByorwherereads changed, and on every replace when it has awhereentry valued"$ownerId"or adeletableDocumentfound byfindBy(alone or as a leaf), unless afindByfunction computes the key, which is judged on the create alone. Otherwise nothing is read: the writer is always the owner, a permanent target is never deleted and its key never moves. - Transfers and purchases cannot happen on such a type, so the owner of every document is a writer that was checked.
- A refusal names
$ownerIdas its path. Registration errors name the declaration<documentType>.$ownerId.
Rules at registration
- The document type's documents can be neither transferred nor traded (
transferableandtradeModeabsent or0). Otherwise a transfer or a purchase, which is not a write, would hand a document to an owner the declaration never checked; declarecreatorRefersToinstead. - The target is one the writer's id can be.
contract,tokenand a document by id are refused, since an identity's id is never one of those ids, and so isidentityPublicKey, which pairs the value with a key id the writer does not carry. The same holds for every leaf of an expression. - Every other rule is a property reference's: those of
findBy, the list and eachwhere, checked against another contract's stored type where the declaration names one.
A malformed declaration is refused by the meta-schema (JsonSchemaError, 10101) or the parser (InvalidContractStructure, 10231); one that cannot hold, with the reference errors of its target (see Errors).
creatorRefersTo
| Where | The document type, at the top level of its schema. Only on a type that records creator ids: a transferable or tradeable type of a format-1 contract (see System Properties). |
| Value | A refersTo declaration whose value is the creator: identity, a permanentDocument found by findBy or with inList, a deletableDocument whose findBy holds a function, or an anyOf or allOf whose leaves are all of these. |
| Since | protocol version 14 |
| On update | Fixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246). |
| Errors | The error the target reports for a property, at the path $creatorId: ReferencedEntityNotFoundError (40120), ReferencedDocumentPropertyMismatchError (40127). |
Example
"moderatorBadge": {
"type": "object",
"transferable": 1,
"creatorRefersTo": {
"type": "permanentDocument",
"contractId": "EG7RGfV8fDTayC2FyVr8HwdpJh3fXDbVztcfE94UmN88",
"documentType": "electedCharter",
"findBy": { "$id": "electedCharterId" },
"inList": "members"
},
"properties": {
"electedCharterId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 0
}
},
"required": ["electedCharterId"],
"additionalProperties": false
}
Only an elected member of the charter electedCharterId names, in the moderation charters contract, may mint a badge. Once minted, the badge can be transferred to anyone (transferable: 1), and it keeps its creator.
How it works
- On create, the creator is the writer, and is checked as
ownerRefersTochecks the writer. Anidentitytarget reads nothing. - On replace, the value is the stored creator, whoever writes. The declaration is checked again when a property its
findByorwherereads changed, and on every replace when it has awhereentry valued"$ownerId". Such an entry still names the writer, not the creator. - Transfers and purchases need no check: they do not change the creator.
- In
findBy,"."is the creator. - A refusal names
$creatorIdas its path. Registration errors name the declaration<documentType>.$creatorId.
Rules at registration
- The document type records creator ids. Such a type's documents can be transferred or traded, so
ownerRefersTois refused on it, and a type declares at most one of the two. - The targets are those of
ownerRefersToexceptdeletableDocument: the creator never changes, and a document a transfer handed on could not be replaced by its new owner once the documentfindByfound was deleted. AdeletableDocumentwhose key afindByfunction computes is the exception, since it is judged on the create alone. findBymay not read"$ownerId": on a type whose documents can change owner, a transfer or a purchase would move that key part without a write.- It can only be declared on a type when the type is created, since an update may not add it. So every document of the type records its creator.
- Every other rule is as for
ownerRefersTo.
Choosing between them
| The type's documents | Declare | Judges |
|---|---|---|
stay with their owner (transferable and tradeMode absent or 0) | ownerRefersTo | whoever writes the document, who is always its owner |
| can be transferred or traded | creatorRefersTo | the identity that created the document, whoever writes it later |
For a rule about the writer and a document one of its own properties already names, a where entry valued "$ownerId" on that property's reference is enough: { "$ownerId": "$ownerId" } requires the writer to own the referenced document. The type-level keywords are for rules no property carries, such as "the writer has a profile" or "the writer is on this list".
See also
- References (refersTo) for the targets, keys and errors.
- findBy, List Elements and Expressions, the targets a writer or creator reference usually takes.
- On the writer or the creator in the Documents chapter, with the internals.
- System Properties for
$ownerIdand$creatorId, and Creation, Transfers and Trading fortransferableandtradeMode.
distinctFrom
distinctFrom says that an identifier property must hold a different identity (or document) than another identifier of the same document, or than the document's owner. Reach for it when a document names two parties that must not be the same: a delegate who is not the delegator, a buyer who is not the seller, a team member who is not the team's leader. The check reads only the document being written, so it costs no state reads.
| Where | An identifier property, at the top level or inside an object; or the items of a typed array of identifiers, where it binds every element |
| Value | "$ownerId", or the dotted path of another identifier property of the same document type ("meta.reviewerId" for a nested one), 1 to 256 characters |
| Default | Absent: no rule |
| Since | protocol version 14 |
| On update | Fixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246) |
| Errors | DocumentPropertyNotDistinctError (10419) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231) |
An identifier here is a 32-byte id, declared as a byte array with the identifier contentMediaType (see Property Schemas).
Example
"delegation": {
"type": "object",
"properties": {
"delegateId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"distinctFrom": "$ownerId",
"position": 0
},
"backupId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"distinctFrom": "delegateId",
"position": 1
}
},
"required": ["delegateId"],
"additionalProperties": false
}
An identity cannot delegate to itself: delegateId must differ from the document's owner. The optional backupId must differ from delegateId; a delegation without a backup passes, since there is nothing to compare.
On a typed array the keyword goes on items, and every element must differ from the named value. The moderation charters system contract uses this for a team's members, none of whom may be the leader who writes the document:
"members": {
"type": "array",
"minItems": 0,
"maxItems": 15,
"uniqueItems": true,
"items": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"distinctFrom": "$ownerId"
},
"position": 2
}
How it works
- Create and replace. After the document's properties pass the JSON schema, each declaring property is compared with what it names: the other property's value in the same document, or the writer's identity for
$ownerId. Equal values refuse the transition withDocumentPropertyNotDistinctError(10419), which names the document type, the property and what it collided with. For a typed array each element is compared on its own. - Absent values pass. When the declaring property or the property it names is left out of the document, there is nothing to compare, and the rule holds. Add the property to
requiredwhen it must be there. - Transfer and purchase. These change the owner and nothing else. The stored document is judged against its new owner, so a transfer to, or a purchase by, the identity held in a property that must differ from
$ownerIdis refused with the same error. A delegation, say, cannot be transferred to its own delegate. A price update changes neither owner nor data and is not judged. - Deletes are never judged.
The check reads the transition (or, for a transfer or purchase, the stored document) and never looks anything else up, and a type without declarations pays nothing for it.
Rules at registration
- The keyword is allowed only on an identifier property or on the identifier
itemsof a typed array. On any other property, the typed array itself included, the meta-schema refuses it (JsonSchemaError, 10101). - The value must be
"$ownerId"or name a property of the same document type. No other system property is accepted. - A named property must exist, must itself be an identifier (an identifier that carries
refersTocounts), must not be an object, and must not be the declaring property.
The parser refuses a value that breaks the last two rules with InvalidContractStructure (10231).
See also
- Distinct Identifier Properties, the deep dive
- Typed Arrays, for keywords on
items - propertyConstraints: a rule such as
{ "notEqual": ["buyerId", "sellerId"] }says the same, and can be combined with other conditions - Writer and Creator References, for rules that look the writer up in state
- Contract Keywords overview, for the conventions of these tables
maxBytes
maxBytes caps how many bytes a string may take when it is encoded as UTF-8, which is how Platform stores it. JSON Schema's maxLength counts characters, and one character takes from one to four bytes, so maxLength alone does not bound the stored size. Reach for maxBytes when the size of a document matters: to keep storage fees predictable, or to stay under the 5120 bytes any one stored value may take.
| Where | A string property, at the top level or inside an object; or the items of a typed array of strings, where it bounds every element |
| Value | An integer from 1 to 65535, no lower than the property's minLength |
| Default | Absent: only maxLength and the 5120-byte cap on every value apply |
| Since | protocol version 14 |
| On update | May be raised or removed; adding it or lowering it is refused (IncompatibleDocumentTypeSchemaError, 10246) |
| Errors | DocumentPropertyMaxBytesExceededError (10421) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231) |
Example
"description": {
"type": "string",
"minLength": 1,
"maxLength": 4096,
"maxBytes": 4096,
"position": 1
}
This is the description of a proposal in the moderation charters system contract. maxLength admits 4096 characters, which could take up to 16384 bytes (more than the 5120-byte cap on a stored value); maxBytes holds the stored text to 4096 bytes, so a description in plain ASCII can use every character and one written only in four-byte characters, such as most emoji, a quarter of them.
On a typed array the keyword goes on items:
"tags": {
"type": "array",
"maxItems": 8,
"items": { "type": "string", "minLength": 1, "maxLength": 32, "maxBytes": 64 },
"position": 2
}
Each of up to 8 tags is at most 32 characters and at most 64 bytes.
How it works
- The check runs wherever a document's properties are validated: every create and replace in consensus, and every client that validates a document before sending it.
- It runs after the JSON schema validation. A value the schema refuses (too many characters, the wrong type) is reported with the schema's error, not this one.
- Every string the document holds for a property that declares
maxBytesis measured in UTF-8 bytes. One that is longer refuses the transition withDocumentPropertyMaxBytesExceededError(10421), which names the property and both lengths. For a typed array the error names the element, as intags[2]. - A property the document leaves out is not checked.
The cap bounds the value only; maxLength and minLength still apply in characters. Setting both is normal: maxLength says what a user may type, maxBytes what the platform stores.
Rules at registration
- The keyword is allowed only on a string property or on the string
itemsof a typed array. On any other property, the typed array itself included, the meta-schema refuses it (JsonSchemaError, 10101). - The value is an integer from 1 to 65535 (10101).
- It may not be lower than
minLength: a string ofminLengthcharacters takes at least that many bytes, so a lower cap would refuse every value (InvalidContractStructure, 10231).
See also
- Byte Caps on Strings, the deep dive
- Property Schemas, for
maxLengthandminLength - Typed Arrays, for keywords on
items - Contract Keywords overview, for the 5120-byte value limit and the conventions of these tables
generatedFrom
generatedFrom says that the platform generates a string property's value with a system function of other properties of the same document, its params. The functions change the case of a string (lowercase, uppercase, capitalize, camelCase, snakeCase), or fold a name so that names differing only by case or by look-alike characters read the same (homographSafeASCII): reach for that one when a unique index must treat Bob, BOB and B0B as one name. A client may leave the property out, and the platform generates it when the document arrives; a client that sends it has it checked. The check reads only the document being written, so it costs no state reads.
| Where | A string property, at the top level or inside an object. Not on a typed array or its items, and not beside $ref |
| Value | { "function": <name>, "params": [<path>, ...] }: a system function, and as many params as it takes, each the dotted path of a property of the same document type ("profile.display" for a nested one), 1 to 256 characters (ASCII, as property names are, so as many bytes) |
| Default | Absent: no rule |
| Since | protocol version 14 |
| On update | Fixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246); a property an update adds may declare it only when one of its params is new too (DocumentTypeUpdateError, 40212) |
| Errors | DocumentPropertyNotGeneratedError (10424) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231); on update IncompatibleDocumentTypeSchemaError (10246) or DocumentTypeUpdateError (40212) |
Example
"handle": {
"type": "object",
"indices": [
{ "name": "byNormalizedLabel", "properties": [{ "normalizedLabel": "asc" }], "unique": true }
],
"properties": {
"label": {
"type": "string", "pattern": "^[a-zA-Z0-9-]{3,63}$", "maxLength": 63,
"position": 0
},
"normalizedLabel": {
"type": "string", "maxLength": 63,
"generatedFrom": {
"function": "sys.stringTransformations.homographSafeASCII",
"params": ["label"]
},
"position": 1
}
},
"required": ["label", "normalizedLabel"],
"additionalProperties": false
}
A client creates { "label": "Bob" } and the stored document holds { "label": "Bob", "normalizedLabel": "b0b" }. A second handle { "label": "B0B" } generates the same b0b and is refused by the unique index. A client that sends { "label": "Bob", "normalizedLabel": "b0b" } gets the same result; one that sends "normalizedLabel": "bob" is refused with DocumentPropertyNotGeneratedError.
The generated property needs no pattern of its own. Every value it can hold is what the function generates from params that passed their own patterns: here label admits ASCII letters, digits and -, so normalizedLabel can only ever hold a to z without i, l and o, digits and -. It keeps maxLength because it is indexed, and an indexed string declares one of at most 63.
This is the rule the DPNS domain type's data trigger checks today for normalizedLabel and normalizedParentDomainName, written into the schema.
Functions
Functions are system functions, built into the platform and named under sys.; any other name is refused at registration. The prefix keeps them apart from functions a contract may bring in a later protocol version. They are grouped in namespaces, and each one declares how many params it takes. The sys.stringTransformations functions each take one string:
| Function | Returns | Example |
|---|---|---|
sys.stringTransformations.lowercase | The string with A to Z lowercased | Hello World to hello world |
sys.stringTransformations.uppercase | The string with a to z uppercased | Hello World to HELLO WORLD |
sys.stringTransformations.capitalize | The first character uppercased and every other lowercased | hELLO wORLD to Hello world |
sys.stringTransformations.camelCase | The words joined, the first lowercased and every later one capitalized | Hello World, hello_world and HelloWorld to helloWorld |
sys.stringTransformations.snakeCase | The words lowercased and joined with _ | Hello World, helloWorld and hello-world to hello_world |
sys.stringTransformations.homographSafeASCII | The string with A to Z lowercased, then o turned into 0 and i and l into 1 | Lil-Olive to 111-011ve |
The string transformations change ASCII characters only and keep every other character as it is (Olé becomes 01é under homographSafeASCII, OLÉ becomes olÉ under lowercase). They use no Unicode tables: Unicode case mappings change between releases of the tools a node is built with, and two nodes that lowercased a character differently would disagree about a document. On ASCII, homographSafeASCII is exactly what the DPNS trigger computes.
camelCase and snakeCase split the string into words. Every ASCII character that is neither a letter nor a digit (a space, -, _, . and so on) separates words and is dropped. A word also starts at an ASCII uppercase letter that follows anything other than another ASCII uppercase letter (helloWorld is hello, World), or that follows one and is followed by an ASCII lowercase letter (XMLHttpRequest is XML, Http, Request, so it becomes xmlHttpRequest or xml_http_request). Characters outside ASCII belong to the word they are in. Every transformation gives back its own output unchanged.
lowercase, uppercase, capitalize and homographSafeASCII keep the length of the value, and camelCase never lengthens it, but snakeCase can: aBcDeF becomes a_bc_de_f. Give a generated property a maxLength that admits what its function can generate from its params' longest value.
A function refuses nothing. Which characters a value may hold is the job of each param's pattern: homographSafeASCII resists look-alike names only where that pattern admits ASCII alone, as DPNS's does. A contract that admits other scripts can hold two names that look alike but generate different values.
Params
For now a param is always a property of the same document, written as its dotted path. The form leaves room to add, in a later protocol version, literals ({ "const": ... }), system values such as "$ownerId" and nested calls, without changing what parses today.
How it works
- Generated on arrival. When a document create or replace, or the values of an index-only delete, leaves the property out and supplies every param, the platform writes the generated value into the document before anything reads it: contest detection, the schema validation, the indexes and the stored document all see it. A property the document sends is left as sent. A generated value then goes through the property's own schema like a sent one, so any bound it declares, such as
maxLength, should admit every value the function can generate from its params. - Checked after the JSON schema. Wherever a document's properties are validated, on every create and replace included and in a client that validates a document before sending it, the property must hold what the function generates from its params, and must be absent when a param is. A document that repeats a key in an object on the way to the property or to a param is refused too, since the value it holds there would be ambiguous. A property that breaks this refuses the transition with
DocumentPropertyNotGeneratedError(10424), which names the document type, the property, the function and its params. A schema error on any of the values is reported first. - Absent params. A property one of whose params is absent must be absent too. To make the params required in effect, list the generated property in
required: a document without them then fails the schema. - Replace. A replace is judged on the whole new document. Leave the property out to have it generated from the new params; a stale value sent with changed params is refused.
- Transfers, purchases and deletes by id do not change the data and are not judged. An index-only delete is: its values are generated and checked like a create's, so a stale value, or one without its params, refuses it.
The SDK's transition builders generate the property from the document's params, replacing any value the document holds and leaving it out when a param is absent, so a transition built from a document carries the value the platform would generate, and contest detection sees it. A document fetched, edited and sent back through them therefore carries the value of its new params, not the stale one. The property-constraint pre-checks of the JavaScript and FFI SDKs judge the document the same way. A client that validates a document it built, before building a transition, should generate it first (in Rust, DocumentTypeBasicMethods::regenerate_generated_properties) or set the value; otherwise the local check reports the property missing. The proof a client verifies after a create or replace is checked against the document with the generated value, as the platform stored it.
Rules at registration
- The keyword is allowed only on a string property, and not beside
$ref, whose definition replaces every keyword written next to it (declare it in the definition instead). On any other property, a typed array and itsitemsincluded, the meta-schema refuses it (JsonSchemaError, 10101). functionmust name a system function, andparamsmust list 1 to 16 params, as many as the function takes. The meta-schema refuses an unknown function or an empty or overlong list (JsonSchemaError, 10101).- Every param must name another string property of the same document type (not an object, not a system property, not the declaring property), and that property may not be generated itself.
- Neither the declaring property nor a param may be transient or sit inside a transient object: a transient value is never stored.
- Every param must sit inside every object that holds the declaring property: a top-level property may take any param, but
profile.normalizedmust take params insideprofile. A document that supplies the params then always holds the object the platform writes the value into. - On a contract update, a new property may declare
generatedFromonly when one of its params is new too. Documents stored before the update were never generated, so a new generated property whose params all existed is refused (DocumentTypeUpdateError, 40212).
The meta-schema refuses the shape errors of the first two rules with JsonSchemaError (10101). The parser refuses a function with the wrong number of params, and a declaration that breaks the param rules (the third to fifth), with InvalidContractStructure (10231). The update rule refuses with DocumentTypeUpdateError (40212).
See also
- Generated Properties, the deep dive
- Property Schemas, for
patternandrequired - Indexes (indices), for the unique index that usually reads the generated property
- Contract Keywords overview, for the conventions of these tables
encryptedFor
encryptedFor marks a byte array property as ciphertext that one identity can read, and writes the recipe into the contract: who the message is for, which identity keys were used, and which encryption scheme made the bytes. Wallets and SDKs read the recipe from the contract instead of from per-app documentation. Reach for it when a document carries a private message, a private note to self, or any other value only its recipient should read. Consensus cannot see inside the ciphertext: it checks only that the bytes have the length the scheme produces.
| Where | A byte array property (byteArray: true) that is not an identifier, at the top level or inside an object. Not on the elements of a typed array |
| Value | An object with exactly four keys, all required: recipient, recipientKey, senderKey, scheme (below) |
| Default | Absent: the property is plain bytes |
| Since | protocol version 14 |
| On update | Fixed: adding, removing or changing it is refused (IncompatibleDocumentTypeSchemaError, 10246). Documents already written could not be read under another recipe |
| Errors | InvalidEncryptedPropertyShapeError (10420) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231) |
The four keys:
| Key | Value |
|---|---|
recipient | The dotted path of an identifier property of the same document type, whose value is the recipient identity's id; or "$ownerId" for a message the writer encrypts to themself |
recipientKey | The dotted path of an integer property of the same document type that carries the id of the recipient's identity key. Its schema must declare minimum of at least 0 and maximum of at most 4294967295 |
senderKey | The same for the sender's identity key, a key of the document's owner ($ownerId) |
scheme | "ecdh-secp256k1-aes256-cbc", the only scheme today |
The three paths are 1 to 256 characters each.
Example
"directMessage": {
"type": "object",
"documentsMutable": false,
"properties": {
"recipientId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 0
},
"recipientKeyId": { "type": "integer", "minimum": 0, "maximum": 4294967295, "position": 1 },
"senderKeyId": { "type": "integer", "minimum": 0, "maximum": 4294967295, "position": 2 },
"encryptedMessage": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 1040,
"encryptedFor": {
"recipient": "recipientId",
"recipientKey": "recipientKeyId",
"senderKey": "senderKeyId",
"scheme": "ecdh-secp256k1-aes256-cbc"
},
"position": 3
}
},
"required": ["recipientId", "recipientKeyId", "senderKeyId", "encryptedMessage"],
"additionalProperties": false
}
A message is written by its owner to the identity in recipientId. The owner used their key senderKeyId and the recipient's key recipientKeyId, and the ciphertext is 32 to 1040 bytes, room for a plaintext of up to 1023 bytes.
The scheme
ecdh-secp256k1-aes256-cbc is the scheme the DashPay contact request already uses for its encrypted fields:
- The shared key is the secp256k1 ECDH of the sender's private key and the recipient's public key: the SHA-256 of the product point's parity byte and x coordinate. The recipient derives the same 32 bytes from their own private key and the sender's public key.
- The writer draws a random 16-byte IV.
- The stored value is the IV followed by the plaintext encrypted with AES-256-CBC under the shared key and that IV, with PKCS7 padding.
A ciphertext is therefore 16 + 16 * ceil((plaintext length + 1) / 16) bytes: at least 32, and always a multiple of 16. There is no authentication tag, so a reader with the wrong key usually fails the padding check, but about once in 256 attempts gets garbage instead. Readers should treat a value that does not decrypt as a bad message, not as a protocol error.
The SDKs do this from the declaration. The Rust SDK's dash_sdk::platform::encrypted_for module has encrypt_property (which also fills in both key id properties) and decrypt_property; the JavaScript SDK has sdk.encryptedFor.encrypt, decrypt and envelope.
How it works
- Create and replace. After the JSON schema validation, each property that declares
encryptedForand is present in the transition is checked for its shape: at least 32 bytes (the IV and one block) and a multiple of 16. A value that is not refuses the transition withInvalidEncryptedPropertyShapeError(10420), which names the property, the scheme and the lengths. The schema's ownminItemsandmaxItemsare checked first, so a value outside them gets the schema's error instead. - Nothing else is checkable on chain. Consensus does not know whether the bytes decrypt, whether the key ids exist on the identities, or whether those keys have an encryption purpose. A writer can store any 32 bytes.
- Checking the keys. To have consensus check that the keys exist and are of the right kind, add references of type
identityPublicKeynext to the declaration. The moderation charters system contract does this: the recipient's key must be a decryption key and the sender's an encryption key.
"recipientId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": {
"type": "identityPublicKey",
"keyIdProperty": "recipientKeyId",
"keyRequirements": { "purpose": "decryption" }
},
"position": 0
},
"senderKeyId": {
"type": "integer", "minimum": 0, "maximum": 4294967295,
"refersTo": {
"type": "identityPublicKey",
"identityProperty": "$ownerId",
"keyRequirements": { "purpose": "encryption" }
},
"position": 2
}
encryptedFor neither requires nor duplicates these references: it describes the recipe, and the references hold the keys to it.
Rules at registration
- The keyword is allowed only on a byte array that is not an identifier. On an identifier or any other property the meta-schema refuses it (
JsonSchemaError, 10101), and so it does an unknown key, a missing key or anotherscheme. recipientmust name an identifier property of the document type (an identifier that carriesrefersTocounts) or be"$ownerId". No other$name is accepted.recipientKeyandsenderKeymust name integer properties of the document type whose schemas declareminimumof at least 0 andmaximumof at most 4294967295, the range of a key id. System properties are refused.- None of the three named properties may be
transientor sit inside a transient object: a transient value is never stored, so a stored ciphertext would lose its recipe. - The byte array's
maxItemsmust be at least 32, the shortest ciphertext the scheme produces.
A registration refusal from the parser is InvalidContractStructure (10231).
See also
- Encrypted Properties, the deep dive, with the scheme's layout and what consensus checks, and what it cannot
- References (refersTo), for
identityPublicKeyreferences - Signing and Keys, for encryption and decryption keys bound to a document type
- transient
- Contract Keywords overview, for the conventions of these tables
propertyConstraints
propertyConstraints holds named rules that every created or replaced document of a type must meet. JSON Schema bounds one property at a time; these rules relate properties to each other: a deposit that covers price times quantity, percentages that add up to 100, a closed order that carries its closing time, a second party who is not the owner. Each rule is a small tree of comparisons, arithmetic and logic that consensus evaluates against the document. A rule can also read a total of other documents, how many there are or what an integer property adds up to, from the count and sum trees their indexes keep (see Totals of other documents).
| Where | Document type |
| Value | An object of rules, at least one. Each key is the rule's name (1 to 64 letters, digits or underscores); each value is a condition (see Conditions) |
| Default | Absent: no rules |
| Since | protocol version 14 |
| On update | Fixed: adding, removing or changing a rule is refused (IncompatibleDocumentTypeSchemaError, 10246). Stored documents were judged against the rules as they were |
| Errors | DocumentPropertyConstraintViolatedError (10422) on a document; at registration JsonSchemaError (10101) or InvalidContractStructure (10231) |
Example
"order": {
"type": "object",
"properties": {
"price": { "type": "integer", "minimum": 0, "maximum": 1000000000, "position": 0 },
"fee": { "type": "integer", "minimum": 0, "maximum": 1000000, "position": 1 },
"quantity": { "type": "integer", "minimum": 1, "maximum": 10000, "position": 2 },
"deposit": { "type": "integer", "minimum": 0, "position": 3 },
"status": { "type": "string", "enum": ["open", "pending", "closed"], "position": 4 },
"closedAt": { "type": "integer", "minimum": 0, "position": 5 }
},
"required": ["price", "quantity", "deposit", "status"],
"propertyConstraints": {
"depositCoversOrder": {
"lessThanOrEqual": [
{ "multiply": [{ "add": ["price", "fee"] }, "quantity"] },
"deposit"
]
},
"feeWaivedOrAtLeastTen": {
"anyOf": [{ "equal": ["fee", 0] }, { "greaterThanOrEqual": ["fee", 10] }]
},
"closedNeedsClosedAt": {
"anyOf": [{ "notEqual": ["status", { "const": "closed" }] }, { "present": "closedAt" }]
}
},
"additionalProperties": false
}
depositCoversOrder reads (price + fee) * quantity <= deposit. feeWaivedOrAtLeastTen reads fee == 0 || fee >= 10; an order that leaves fee out passes, since a missing integer reads as 0. closedNeedsClosedAt says an order whose status is closed carries a closedAt.
How it works
- Create and replace. The rules run after the JSON schema validation of the document's properties (and after maxBytes), so every value a rule reads has passed its property's schema. A replace is judged on the whole new document, not only on what changed.
- Name order, first failure. Rules are checked in the order of their names, and the first rule the document breaks refuses the transition with
DocumentPropertyConstraintViolatedError(10422). The error names the document type, the rule, and why it failed (below). - Transfer and purchase. These change only the owner and the transfer's time and heights. Rules that read
$ownerId,$transferredAt…or a total that depends on the owner are judged again, against the stored document with its new owner and transfer values; other rules are not, since nothing they read changed. A transfer or purchase that would break such a rule is refused with 10422. - Price updates change only the update's time and heights, so the rules that read
$updatedAt…are judged again the same way; other rules are not. - Moderator restores. Every rule of the restored type is judged on the retained document, with its original owner, times and heights. Totals include the restored document as an insertion into the live trees; its removal record is not counted. Another write while it was deleted may make a cap, lower bound or cross-type prerequisite fail. Uniqueness is checked first. A refusal in a block charges the moderator and consumes its nonce, while the document stays deleted and its removal record stays unrestored. Mempool admission refuses the same invalid restore without persisting fees or a nonce change. Restore does not revalidate the full property schema or judge
deleteConstraints. - Immutable properties. The same grammar is the condition of an
immutableentry, which freezes a property while it holds. Only such a condition may read the stored document, through$old.<path>; a rule judges creates too, which have none. - Deletes are not judged, with one exception: a delete of an index-only document carries the row's values, which are validated like a create's, rules included. The delete carries neither the owner nor any time or height, which is why an index-only type may not have a rule reading
$ownerIdor a system time or height. What may be deleted is the job ofdeleteConstraints, rules in this grammar the stored document must meet for its owner to delete it. - State and fees. A rule reads the document, its owner and its times and heights, and a
countOforsumOfreads a total from state. Each such total is a state read billed with the write; nothing else a rule does adds a fee, and it changes nothing stored. The limits below bound its cost. SDKs that validate a document before sending it apply the same rules, except those reading a total, which they cannot read.
Why a rule fails, as the error reports it:
| Reason | When |
|---|---|
| does not hold | The rule evaluates without a fault and comes out false |
| overflow | A value it reads, or a result it computes on the way, does not fit a 128-bit signed integer |
| division by zero | A divide or modulo whose divisor evaluates to 0 |
| negative exponent | A power whose exponent evaluates to a negative number |
| not an integer | A value it reads for an integer property is a float with no fractional part, such as 5.0, which the schema's integer type admits but an integer property cannot store |
Conditions
A rule is a condition: a JSON object with exactly one key.
| Condition | Form | Holds when |
|---|---|---|
equal, notEqual | [left, right] | The two sides are equal, or differ. The sides are two integer expressions, or a string property and a string constant or another string property, or an identifier property and an identifier constant, another identifier property or $ownerId |
lessThan, lessThanOrEqual, greaterThan, greaterThanOrEqual | [left, right] | The left integer expression compares with the right one this way. Integers only |
in | [expression, [v1, v2, ...]] | The expression takes one of the listed values: two or more, no two alike, all integers or all strings. With strings, the expression is a string property, or an identifier property or $ownerId with the strings as base58 identifiers |
notIn | [expression, [v1, v2, ...]] | The expression takes none of the listed values: an in negated, listed the same way, in as many nodes. A string or identifier property the document leaves out takes none |
startsWith, endsWith | [text, affix] | The first string starts, or ends, with the second, byte for byte with no case folding. Each side is a string constant, a string property or an ifAbsent string default, at least one a property and never the same one twice. A string property left out without a default takes no string, and the condition does not hold for it |
contains | ["path", value] | The typed array property at the path holds an element equal to the value: an integer expression among integers; a string constant, a string property or an ifAbsent string default among strings; an identifier constant, an identifier property or $ownerId among identifiers. An array the document leaves out holds nothing, and a string or identifier property it leaves out is among no elements |
present | "path" | The document holds the property, with a value other than null and, for an object, with at least one member present |
absent | "path" | The document leaves the property out, sets it to null, or gives an object no member that is present |
anyOf | [c1, c2, ...] | At least one of two or more conditions holds |
allOf | [c1, c2, ...] | Every one of two or more conditions holds |
not | condition | Its one condition does not hold |
ifThen | [if, then] | If the first condition holds, the second must. The second is evaluated only when the first holds, and a fault in either breaks the rule. The two may not be alike |
ifThenElse | [if, then, else] | If the first condition holds, the second must; if not, the third must. Only the branch the first selects is evaluated. No two of the three may be alike |
Conditions nest: { "not": { "allOf": [{ "equal": ["price", 0] }, { "greaterThan": ["quantity", 10] }] } } refuses a free order of more than 10. An anyOf or allOf may not list the same condition twice, nor hold one of its own kind directly (it says what one flat list says), and a not may not hold a not or a notIn directly.
An in says what an anyOf of equal comparisons says, in far fewer nodes: { "in": ["fee", [0, 10, 25, 50]] } is 6 nodes where the anyOf is 13.
startsWith and endsWith test a string's ends: { "startsWith": ["url", { "const": "https://" }] } holds a link to https, { "endsWith": ["url", { "const": ".dash" }] } to a domain, and { "startsWith": ["path", "parentPath"] } holds a reply's path under its parent's. A constant tested against a property that declares an enum must start or end one of its values.
A contains looks the other way round, for one value among an array's elements:
{ "not": { "contains": ["labels", { "const": "used" }] } }refuses a"used"label;{ "contains": ["participants", "$ownerId"] }holds the owner to the participants, and since it reads$ownerId, a transfer or purchase to someone else is refused;{ "contains": ["tiers", "quantity"] }holds the quantity to one of the tiers the document lists.
The kind of the array's elements decides what the value is: a { "const": "sale" } is a string among strings and a base58 identifier among identifiers.
Expressions
An integer expression is one of:
| Expression | Form | Value |
|---|---|---|
| integer | 100 | Itself. A number written 100.0 reads as 100 |
| path | "price", "meta.total" | The value of an integer or boolean property of the document type, 1 for true and 0 for false. A property the document leaves out, or sets to null, reads as 0 |
ifAbsent | { "ifAbsent": ["quantity", 1] } | The property's value, or the given integer when the document leaves it out |
add, multiply | { "add": [a, b, ...] } | The sum or product of two or more operands |
subtract | { "subtract": [a, b] } | a - b |
divide | { "divide": [a, b] } | The Euclidean quotient of a by b |
modulo | { "modulo": [a, b] } | The Euclidean remainder of a by b, never negative |
power | { "power": [a, b] } | a to the power b |
min, max | { "max": [a, b, ...] } | The least or greatest of two or more operands, every one evaluated |
abs | { "abs": a } | The absolute value of its one operand |
length, byteLength | { "length": "title" } | The characters (as maxLength counts them) or UTF-8 bytes (as maxBytes counts them) of a string property, 0 when the document leaves it out |
count | { "count": "tags" } | The items of an array property, or the bytes of a byte array property, 0 when the document leaves it out |
countPresent | { "countPresent": ["email", "phone"] } | How many of two or more properties, no two alike, the document holds, each as present tests it (see How many of a group) |
| system time or height | "$createdAt", "$updatedAtBlockHeight" | A time or height the document records (see Times and heights) |
countOf, sumOf | { "countOf": ["listing", { "$ownerId": "$ownerId" }] } | A total of documents of a type of the same contract, read from state (see Totals of other documents) |
Where maxLength, maxBytes and maxItems bound one property by a fixed number, a size can be compared with another property or bounded only under a condition: { "lessThanOrEqual": [{ "count": "tags" }, "maxTags"] } holds a list to its own limit. A size never breaks a rule by itself: a property left out or null has size 0, and so would a value of another type, which the schema validation refuses first.
Two more forms appear only in string and identifier comparisons, never inside arithmetic:
| Form | Meaning |
|---|---|
{ "const": "closed" } | A string constant, or, compared with an identifier property or $ownerId, a base58 identifier |
{ "ifAbsent": ["status", "open"] } | A string property, read as the given string when the document leaves it out |
A bare JSON string is always a path and a bare JSON number always a value, so a constant string needs { "const": ... }. The values an in lists are literals and need no wrapper. A path is a property name, or names joined by dots for a nested property ("rewardSplit.leader"); the only $ names a rule accepts are $ownerId, the times and heights below, and, as the value a total's filter matches by, $id.
A number property (a float) cannot be read by a rule, which keeps every result exact.
Strings
A string property is compared for equality only, never ordered and never used in arithmetic:
{ "equal": ["status", { "const": "closed" }] }ornotEqual, with the constant on either side;{ "notEqual": ["fromCurrency", "toCurrency"] }, two bare paths that both name string properties, which compares their strings;{ "in": ["status", ["open", "pending"]] }, whose values are two or more distinct strings.
A string property the document leaves out equals no constant and no other string property, not even one also left out. So notEqual holds for it, and equal and in do not. { "ifAbsent": ["status", "open"] } gives it a default instead: it may stand wherever the bare path stands, and the property then reads as that string when it is left out. present and absent test it directly.
When the property declares an enum, every constant compared with it, and every ifAbsent default given to it, must be one of the enum's values. A misspelled constant is refused at registration instead of making the rule quietly never hold.
Identifiers and $ownerId
An identifier property compares in the same three ways: { "equal": ["paymentToken", { "const": "<base58>" }] } or notEqual, { "notEqual": ["buyerId", "sellerId"] }, and { "in": ["paymentToken", ["<base58>", "<base58>"]] }. Constants are base58 identifiers of 32 bytes, checked at registration and compared by their bytes, whatever form the document gives the identifier in. An identifier property the document leaves out equals no identifier, not even another one left out. Identifiers take no ifAbsent default and are never ordered.
$ownerId, the document's owner, is an identifier operand too:
{ "equal": ["authorId", "$ownerId"] }holds theauthorIdproperty to the owner.{ "in": ["$ownerId", ["<base58>", "<base58>"]] }lets only the listed identities own a document of the type.
It is not a property: present, absent and integer expressions refuse it, and comparing it with itself is refused. On create and replace it is the writer. A transfer or purchase is judged with the new owner, as described in How it works. An index-only type may not declare a rule that reads it.
Times and heights
A rule can read when the document was created, last updated and last transferred, as an integer:
| block time (ms) | Platform block height | Core block height | |
|---|---|---|---|
| creation | $createdAt | $createdAtBlockHeight | $createdAtCoreBlockHeight |
| last update: a create, a replace or a price update | $updatedAt | $updatedAtBlockHeight | $updatedAtCoreBlockHeight |
| last transfer: a create, a transfer or a purchase | $transferredAt | $transferredAtBlockHeight | $transferredAtCoreBlockHeight |
{ "lessThanOrEqual": [{ "subtract": ["endsAt", "$createdAt"] }, 604800000] }keeps a listing to a week from its creation.{ "lessThanOrEqual": ["$updatedAt", "endsAt"] }refuses a replace or a price update after the listing ends.{ "lessThanOrEqual": ["$transferredAt", "endsAt"] }refuses a transfer or a purchase after it ends.
A rule may read one only when the document type records it by listing it in required, so every stored document holds it. None takes an ifAbsent default, present and absent refuse them, and an index-only type reads none. Each write is judged with the values the stored document ends up with: a create with its block's time and heights for all three events; a replace with the stored creation and transfer values and its block's as the update; a price update with its block's as the update; a transfer or a purchase with its block's as the transfer.
SDK pre-checks run before the block exists: they use the device clock for the times a write records, and do not judge a rule reading a block height, which is unknown until the block.
Totals of other documents
countOf and sumOf read a total from state: how many documents of a type of the same contract match a filter, or what one of their integer properties adds up to. The total is the one a count or sum tree keeps (Count Trees, Sum Trees), so reading it costs about the same however many documents match.
| Form | Value | The counted type needs |
|---|---|---|
{ "countOf": ["listing"] } | How many listing documents there are | documentsCountable |
{ "countOf": ["listing", { "$ownerId": "$ownerId" }] } | How many of them match the filter | A countable index whose properties are exactly the filter's keys |
{ "sumOf": ["pledge", "amount"] } | The total amount over every pledge | documentsSummable: "amount" |
{ "sumOf": ["pledge", "amount", { "campaignId": "campaignRef" }] } | The total over those matching the filter | An index with summable: "amount" whose properties are exactly the filter's keys |
A filter maps each key, a property of the counted type or $ownerId, to the value it must take, read from the document being written: one of its properties ("campaignRef"), $ownerId, $id (its own id, for an identifier key), an integer, or a { "const": ... } string or base58 identifier. The counted type may be the rule's own. $id counts the documents pointing at this one: { "countOf": ["vote", { "pollId": "$id" }] } is how many votes name the poll being written.
"propertyConstraints": {
"atMostTenListings": {
"lessThanOrEqual": [{ "countOf": ["listing", { "$ownerId": "$ownerId" }] }, 10]
},
"pledgesWithinGoal": {
"lessThanOrEqual": [{ "sumOf": ["pledge", "amount", { "campaignId": "campaignRef" }] }, "goal"]
}
}
- As it will be after the write. The total is the stored one with the write applied. When the counted type is the rule's own, a create or moderator restore adds the document, a replace swaps its stored version for the new one, and a transfer or purchase moves it to its new owner. So
atMostTenListings, declared onlisting, keeps every owner at ten or fewer, and a replace of one of ten is allowed. - Judged when the rule's own type is written. A rule is never judged on writes of the type it counts. On its own type it holds for good, since every write that could raise the total is judged; a type with a contested index cannot total its own documents, since a document a contest awards is stored without any rule judged. On another type it is only checked when its own type is written, and can go stale later: deleting a
profiledoes not undo apostthat needed one. Deletes are not judged, so a lower bound can be broken by deleting documents, unless the counted type'sdeleteConstraintshold it. - Transfers, purchases and price updates. A total that depends on the owner (a filter value of
$ownerId, or a$ownerIdkey on the rule's own type) is read again for a transfer or purchase, the document counted toward its new owner. A rule a price update judges, one reading$updatedAt…, reads its totals too. - Billed. Each total is a state read billed with the write. A total two rules read alike is read once.
- Every earlier write counts. A document batch carries one transition, and each state transition of a block is applied before the next is validated, so a total includes every write before it.
- SDK pre-checks cannot read state, so they do not judge a rule reading a total.
How many of a group
countPresent counts the properties of a group that the document holds, each as present tests it, so a rule compares the count with a number. A seller is reached by exactly one of three contacts:
"propertyConstraints": {
"oneContact": { "equal": [{ "countPresent": ["email", "phone", "handle"] }, 1] }
}
The comparisons and in give every bound:
| Want | Rule |
|---|---|
| exactly one | { "equal": [{ "countPresent": [...] }, 1] } |
| at most one | { "lessThanOrEqual": [{ "countPresent": [...] }, 1] } |
| at least two | { "greaterThanOrEqual": [{ "countPresent": [...] }, 2] } |
| one or two | { "in": [{ "countPresent": [...] }, [1, 2]] } |
| none or all three | { "in": [{ "countPresent": ["a", "b", "c"] }, [0, 3]] } |
The count runs from 0 to the size of the group, so an in lists a range in a few values. oneContact is 6 nodes (the comparison, countPresent, three paths and 1); the same rule written with present alone needs an anyOf and a not of every pair, 17 nodes for three properties and more than 32 for five. Inside ifThen the bound applies only when another condition holds: { "ifThen": [{ "equal": ["kind", { "const": "shop" }] }, { "greaterThanOrEqual": [{ "countPresent": ["email", "phone", "handle"] }, 1] }] } asks a shop for at least one contact.
Evaluation order and short-circuiting
Conditions are checked in declared order and no further than the outcome needs. A comparison evaluates its left side, then its right. anyOf stops at the first condition that holds, allOf at the first that fails. Operands are evaluated left to right.
A fault (an overflow, a division by zero, a negative exponent, a value that is not an integer) in a condition that is evaluated breaks the rule, whatever the other conditions would say, and not does not turn a fault into a pass. String comparisons, present and absent never fault. So an earlier condition can guard a later one:
{ "anyOf": [{ "equal": ["b", 0] }, { "equal": [{ "divide": ["a", "b"] }, 2] }] }
holds for a b of 0 without dividing by it. The same two conditions the other way round divide by zero and break the rule.
Arithmetic
- Integers are exact over 128-bit signed integers. Every intermediate result must fit, and one that does not breaks the rule instead of wrapping.
addandmultiplyfold their operands from the left, so an overflow on the way is a fault even when a later operand would bring the total back in range. divideandmoduloare Euclidean: the remainder is never negative, and the quotient is the one that goes with it.-7divided by2is-4, remainder1. For operands that are not negative this is ordinary integer division.- A divisor that evaluates to 0 breaks the rule. A negative exponent breaks it too, since it has no integer result.
0to the power0is1. - There are no floats.
Rules at registration
The meta-schema checks the shape (JsonSchemaError, 10101):
- the keyword is an object of one or more rules, named with 1 to 64 letters, digits or underscores;
- every condition and every operator object has exactly one key;
- a comparison,
subtract,divide,moduloandpowertake exactly two operands;addandmultiplytwo or more;anyOfandallOftwo or more conditions, no two alike; anintwo or more distinct values, all integers or all strings; acountPresenttwo or more distinct paths; - no
anyOforallOfholds its own kind directly, and nonotholds anotor anotIn; - a path matches
$ownerId, one of the nine times and heights, or dotted names of 1 to 64 letters, digits or underscores, so$revisionand other system properties are refused; - a
countOflists a type name and optionally a filter, and asumOfa type name, a property and optionally a filter; a filter has one or more keys, each$ownerIdor a dotted path, and each value is a path,$ownerId,$id, an integer or a{ "const": ... }string.
The parser then checks the rules against the document type (InvalidContractStructure, 10231):
- every path an integer expression reads names an integer or boolean property; every path
lengthorbyteLengthmeasures names a string property, and every pathcountcounts an array or byte array property; every path acontainslooks in names a typed array property whose elements are integers, strings or identifiers, of the kind of the value looked for (a string constant among them in the elements'enumwhen they declare one); every path compared with a string, or tested bystartsWithorendsWith, names a string property, and a constant tested against one with anenumstarts or ends one of its values; every path compared with an identifier names an identifier property; every pathpresent,absentorcountPresenttests names a property of any type, an object included; - no rule reads a property that is
transientor inside a transient object, since a stored document could never be held to it; - every comparison and
inreads at least one property: a comparison of constants would hold for every document or for none; - strings and identifiers are compared only with
equal,notEqualandin; a string is never compared with an identifier; a property is never compared with itself; - string constants and
ifAbsentdefaults are in the property'senumwhen it has one; identifier constants are base58 identifiers of 32 bytes; - no literal divisor is 0 and no literal exponent is negative;
- every time or height a rule reads is one the type lists in
required, and takes noifAbsentdefault; present,absentandcountPresentdo not name$ownerIdor a time or height, and an index-only type has no rule reading any of them;- no
anyOforallOflists two conditions that parse alike, such as1and1.0, or twoinconditions listing the same values in another order, and noifThenorifThenElseholds two alike conditions; - no condition or operand nests more than 64 levels deep;
- once every document type of the contract is parsed, every
countOfandsumOfcounts a type of the contract that is not index-only, and not its own type when that has a contested index, with a tree that keeps the total as set out in Totals of other documents. A unique, contested, ranked, time-range, integer-range or index-only-terminal index keeps no such total, nor does one with more properties than the filter has keys; - every key of a filter is
$ownerIdor an integer, string or identifier property of the counted type, and its value is of the same kind ($ownerIdand$idare identifiers); a string constant is in the key'senumwhen it has one, and an identifier constant is base58; - every property a filter value reads is listed in
required, with every object around it, so a write always has the value; an index-only type has no rule reading a total.
Three limits come from the protocol version 14 SystemLimits, and a rule over one is refused the same way:
- at most 16 rules per document type (
max_property_constraints); - at most 32 nodes per rule (
max_property_constraint_nodes); - at most 4 distinct
countOfandsumOftotals read by one document type's rules, a total read twice counting once (max_property_constraint_aggregates).
A rule within 32 nodes is never deep enough to reach the 64-level bound. Nodes are counted like this:
| Part of a rule | Nodes |
|---|---|
| A comparison of integers | 1, plus its two sides |
An equal or notEqual of strings or identifiers, a startsWith or an endsWith | 3: the condition and its two sides |
An in over integers | 1, plus its expression, plus 1 per value |
An in over strings or identifiers | 2, plus 1 per value |
contains | 2, plus the value it looks for |
present, absent | 1 |
anyOf, allOf | 1, plus their conditions |
not | 1, plus its condition |
ifThen, ifThenElse | 1, plus their conditions |
notIn | as the in it negates |
An integer, a path, an ifAbsent, a size (length, byteLength, count) or a time or height | 1 |
countPresent | 1, plus 1 per path |
add, multiply, subtract, divide, modulo, power, min, max, abs | 1, plus their operands |
countOf, sumOf | 1, plus 1 per filter key |
depositCoversOrder above is 7 nodes (the comparison, multiply, add and four paths), and closedNeedsClosedAt is 5. An in fits up to 30 values in 32 nodes.
Worked examples
Percentages that add up. The moderation charters system contract requires a proposal's reward split to be whole. The paths name members of the rewardSplit object, each an integer from 0 to 100:
"propertyConstraints": {
"rewardSplitIsWhole": {
"equal": [
{ "add": ["rewardSplit.leader", "rewardSplit.equal", "rewardSplit.actions"] },
100
]
}
}
Six nodes: the comparison, add, three paths and 100.
One of two, or both or neither. A contact card must give an email or a phone; a shipping block gives a street and a city together or not at all:
"propertyConstraints": {
"reachable": { "anyOf": [{ "present": "email" }, { "present": "phone" }] },
"addressComplete": {
"anyOf": [
{ "allOf": [{ "present": "street" }, { "present": "city" }] },
{ "allOf": [{ "absent": "street" }, { "absent": "city" }] }
]
}
}
present and absent work on properties of any type, strings and objects included. On an integer they are also the only way to tell "not given" from "given as 0", since a missing integer reads as 0 in an expression. An object with no member present, {} or { "inner": {} }, counts as absent: a stored document does not keep it, and a transfer, purchase or price update is judged on the stored document, so a create or replace is judged the same way.
A time window. An event ends after it starts, and lasts at most a week (startsAt and endsAt are required integer timestamps in milliseconds):
"propertyConstraints": {
"endsAfterStart": { "lessThan": ["startsAt", "endsAt"] },
"atMostAWeek": { "lessThanOrEqual": [{ "subtract": ["endsAt", "startsAt"] }, 604800000] }
}
A status workflow. On a ticket type, status is optional and means open when it is left out. A closed ticket names who closed it; an open or pending one does not:
"propertyConstraints": {
"closedNamesCloser": {
"anyOf": [{ "notEqual": ["status", { "const": "closed" }] }, { "present": "closedBy" }]
},
"activeHasNoCloser": {
"anyOf": [
{ "not": { "in": [{ "ifAbsent": ["status", "open"] }, ["open", "pending"]] } },
{ "absent": "closedBy" }
]
}
}
Without the ifAbsent, a ticket with no status would equal none of the listed strings, the in would not hold, and activeHasNoCloser would let it carry a closedBy. If status declares an enum, "closed", "open" and "pending" must all be in it.
Who may own a badge. On a transferable badge type, only two identities may ever hold one:
"propertyConstraints": {
"knownHolder": {
"in": [
"$ownerId",
["HJtU46rVEkKJgevQhiVt2YhdHtDS3xtGzzJqit8en5mb", "GfRNCXeyuB3th33a6nkJKQJecKMdRzuBiPsgvSCyUTvK"]
]
}
}
A create by anyone else is refused, and so is a transfer or sale of a badge to anyone else: the rule reads $ownerId, so it is judged again with the new owner.
A guarded division. The average unit price of a batch is at most 100, and a batch may be empty:
"propertyConstraints": {
"unitPriceCapped": {
"anyOf": [
{ "equal": ["quantity", 0] },
{ "lessThanOrEqual": [{ "divide": ["total", "quantity"] }, 100] }
]
}
}
The equal comes first, so an empty batch never reaches the division. Written the other way round, an empty batch breaks the rule with a division by zero.
A flag in arithmetic. A boolean reads as 1 or 0, so a waived fee must be 0:
"propertyConstraints": {
"waivedMeansFree": { "equal": [{ "multiply": ["waiveFee", "fee"] }, 0] }
}
See also
- Property Constraints, the deep dive
- deleteConstraints, rules in this grammar judged on the owner's delete
- distinctFrom, a single-keyword way to keep two identifiers apart
- Property Schemas, for the one-property bounds JSON Schema gives
- transient, Index-Only Types
- Contract Keywords overview, for the limits and the conventions of these tables
Token Costs (tokenCost)
tokenCost makes an action on a document cost tokens: creating a card costs 10 gems, deleting one costs 1. The tokens are taken from whoever signs the transition, and either go to the contract owner or are burned. Reach for it when an app has its own token and wants documents to be paid for with it, or wants to hand out tokens that let users act for free, with the contract owner paying the gas.
| Where | Document type |
| Value | An object keyed by action: create, replace, delete, transfer, update_price, purchase. Each value is a cost object with the keys below. Actions left out cost no tokens |
| Default | Absent: no action costs tokens |
| Since | protocol version 9. gasFeesPaidBy is accepted from 9 and acted on from 14; optional is 14 |
| On update | Fixed: a cost may not be added, changed or removed on an existing document type (DocumentTypeUpdateError, 40212) |
| Errors | On a document transition: RequiredTokenPaymentInfoNotSetError (40115), IdentityHasNotAgreedToPayRequiredTokenAmountError (40116), IdentityTryingToPayWithWrongTokenError (40117), IdentityTokenAccountFrozenError (40702), IdentityDoesNotHaveEnoughTokenBalanceError (40700), GasFeesPaidByNotAllowedError (40129), InconsistentGasFeesPaidByInBatchError (40130), GasSponsorInsufficientBalanceError (40222). At registration: InvalidTokenPositionError (10451), RedundantDocumentPaidForByTokenWithContractId (10275), TokenPaymentByBurningOnlyAllowedOnInternalTokenError (10261), DataContractNotFoundError (40008), InvalidTokenPositionStateError (40009) |
The keys of each cost object:
| Key | Value | Default | Meaning |
|---|---|---|---|
tokenPosition | integer, 0 to 65535 | required | Which token is charged: its position in this contract's tokens, or in the contract contractId names |
amount | integer, 1 to 281474976710655 | required | How many tokens the action costs |
contractId | identifier (32 bytes) | this contract | The contract whose token is charged, when it is not this one |
effect | 0 transfer to the contract owner, 1 burn | 0 | What happens to the tokens paid |
gasFeesPaidBy | 0 document owner, 1 contract owner, 2 prefer contract owner | 0 | Who the contract owner offers to have pay the gas of this action (acted on from protocol version 14) |
optional | boolean | false | true lets a transition skip the token and pay the gas in credits instead (protocol version 14) |
Example
"card": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": true,
"transferable": 1,
"tradeMode": 1,
"tokenCost": {
"create": { "tokenPosition": 0, "amount": 10, "gasFeesPaidBy": 2, "optional": true },
"replace": { "tokenPosition": 1, "amount": 1 },
"delete": { "tokenPosition": 1, "amount": 1, "effect": 1 }
},
"properties": {
"name": { "type": "string", "minLength": 1, "maxLength": 63, "position": 0 },
"attack": { "type": "integer", "minimum": 0, "maximum": 100, "position": 1 }
},
"required": ["name", "attack"],
"additionalProperties": false
}
The contract has two tokens. Creating a card costs 10 of token 0, which go to the contract owner. A player who pays with the token and asks for it has the gas paid by the contract owner, when the owner's balance covers it; a player may also skip the token and pay the gas in credits. Replacing a card costs 1 of token 1, also to the owner. Deleting one burns 1 of token 1. Transfers, price updates and purchases cost no tokens.
Paying: $tokenPaymentInfo
A document transition on an action with a token cost carries a $tokenPaymentInfo in its base, saying which token the signer agrees to pay with, how much at most, and who they ask to pay the gas:
"$tokenPaymentInfo": {
"$formatVersion": "0",
"tokenContractPosition": 0,
"maximumTokenCost": 10,
"gasFeesPaidBy": "PreferContractOwner"
}
| Field | Meaning |
|---|---|
paymentTokenContractId | The contract of the token paid with. Leave it out for a token of the document's own contract: it must match the cost's contractId exactly, and a cost on the contract's own token has none |
tokenContractPosition | The token's position in that contract |
minimumTokenCost, maximumTokenCost | Optional bounds on the amount the signer agrees to pay; a cost outside them refuses the transition |
gasFeesPaidBy | "DocumentOwner", "ContractOwner" or "PreferContractOwner": who the signer asks to pay the gas (see Who pays the gas) |
When the transition is processed:
- A required cost with no
$tokenPaymentInfois refused (RequiredTokenPaymentInfoNotSetError, 40115). - A payment info naming another token than the cost is refused (
IdentityTryingToPayWithWrongTokenError, 40117). - A cost outside the signer's
minimumTokenCostandmaximumTokenCostis refused (IdentityHasNotAgreedToPayRequiredTokenAmountError, 40116). - The signer's gas request must be one the cost offers, and the whole batch must name one payer (40129, 40130).
- Against state: a signer whose account for the token is frozen is refused (
IdentityTokenAccountFrozenError, 40702), and so is one whose balance is belowamount(IdentityDoesNotHaveEnoughTokenBalanceError, 40700). - From protocol version 14, a transparent payment that transfers or burns tokens is then refused if the token is paused (
TokenIsPausedError, 40711). - From protocol version 14, for a transfer to a different identity, the contract owner's account for the token is checked next. A frozen recipient is refused (40702) only when the token issuer's
allowTransferToFrozenBalanceisfalse; its default istrue. For another contract's token, the issuing contract's policy applies, and the recipient remains the document contract's owner.
Every state read is charged, including on refusal. From protocol version 14, the issuer metadata and, for an external token, its contract are read only when the recipient is frozen. A transparent owner self-payment retains the payer freeze and balance checks and makes no transfer, so it performs neither the pause nor recipient checks. A payment from the shielded pool retains its separate pool validation.
The signer pays: the creator for a create, the owner for a replace, delete, transfer or price update, and the buyer for a purchase.
effect: transfer or burn
0, the default, moves the tokens from the signer to the owner of the contract that holds the document type. When the contract owner performs the action themselves nothing moves, though their balance is still checked.1burns the tokens from the signer's balance, lowering the token's supply. Only a token of the contract's own can be burned (TokenPaymentByBurningOnlyAllowedOnInternalTokenError, 10261, at registration).
Tokens of another contract: contractId
A document type may charge a token another contract defines, for example a shared currency. contractId names that contract and tokenPosition the token in it. The effect must then be 0: the tokens go to the owner of the contract holding the document type, not to the token's issuer. At registration the named contract must exist (DataContractNotFoundError, 40008) and have a token at that position (InvalidTokenPositionStateError, 40009), and it must not be the contract itself: leave contractId out for the contract's own tokens (RedundantDocumentPaidForByTokenWithContractId, 10275).
Who pays the gas: gasFeesPaidBy
From protocol version 14 the contract owner can pay the gas (the storage and processing fees) of an action paid with a token. The cost states what the contract owner offers; the transition's $tokenPaymentInfo.gasFeesPaidBy states what the signer asks for. The two resolve like this:
| Cost offers / signer asks | DocumentOwner | PreferContractOwner | ContractOwner |
|---|---|---|---|
0 document owner | signer pays | signer pays | refused (40129) |
2 prefer contract owner | signer pays | contract owner, if their balance covers it | refused (40129) |
1 contract owner | signer pays | contract owner, if their balance covers it | contract owner |
- A signer can always pay for themself, can always state a preference, and can insist on the contract owner only where the cost offers
1. A request the cost does not cover is refused (GasFeesPaidByNotAllowedError, 40129). An action without a token cost offers0, and a transition without$tokenPaymentInfoasks forDocumentOwner. - A batch has one payer. Transitions of one batch that resolve to different payers are refused (
InconsistentGasFeesPaidByInBatchError, 40130); a token transition in the batch is never sponsored, so it counts as the signer paying. - When the contract owner's balance does not cover the gas (and any action fees they would owe), a batch that insisted is refused and charged nothing (
GasSponsorInsufficientBalanceError, 40222), and a batch that only preferred falls back to the signer. - A transition that fails validation is never sponsored: its signer pays for the work that ran.
- Storage refunds still go to the document's owner, whoever paid for the storage. Each token the contract owner hands out is therefore worth up to the storage fee of the largest document the type allows, so a type that offers to pay should bound its documents' size (
maxLength,maxItems, maxBytes) and price the action to match. - A sponsor also pays any action fee the action charges.
Before protocol version 14 the key was accepted and stored but not acted on: the signer always paid.
Optional costs
With optional: true a transition may leave $tokenPaymentInfo out. It then pays no token, its signer pays the gas in credits as on an action without a token cost, and no sponsorship applies. With $tokenPaymentInfo present the token is charged exactly as for a required cost, sponsorship included, and an insufficient token balance is a rejection, never a fallback to credits: the client chooses between token and credits before signing.
Together with gasFeesPaidBy this gives a "free usage" pattern: an app hands out tokens, users act for free while their tokens last, and keep going on credits after.
Rules at registration
- The meta-schema checks the shape (
JsonSchemaError, 10101): only the six action keys;tokenPositionandamountrequired in each cost; values in the ranges above; no other key. Before protocol version 14 it also refusesoptional. - Without
contractId,tokenPositionmust be a token of this contract (InvalidTokenPositionError, 10451). - With
contractId: not this contract's own id (10275), no burn (10261), and a contract that exists with a token at that position (40008, 40009).
See also
- Gas paid by the contract owner and Optional token costs, the deep dive
- Action Fees (actionFees), fixed credit fees on the same six actions
- Creation, Transfers and Trading, for the actions a type allows
- Contract-Level Keys and config, for the contract's
tokens - Contract Keywords overview, for the conventions of these tables
Action Fees (actionFees)
actionFees charges a fixed fee in credits, on top of the gas, for an action on a document of the type. Each fee has two parts: one for the contract owner and one for the contract's moderators, each collected in a pot that its recipients claim. Reach for it when an app wants to earn from the documents written under it, or to pay the people who moderate it. The transition that pays must state the fee it agrees to, so a fee can never surprise a signer.
| Where | Document type |
| Value | An object with an optional pricing, and one or more of the actions create, replace, delete, transfer, update_price, purchase, each an object with owner and/or moderators (below) |
| Default | Absent: no action charges a fee |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212): an update may not add, change or remove the fees of an existing document type, nor switch their pricing. A document type the update adds may declare its own |
| Errors | On a document transition: DocumentActionFeeAgreementNotSetError (40132), DocumentActionFeeAgreementMismatchError (40133), DocumentActionFeeMultiplierNotToleratedError (40134), DocumentActionFeeModeratorsShareMismatchError (40139). At registration: DocumentActionFeesWithoutModerationError (10902), JsonSchemaError (10101), InvalidContractStructure (10231) |
The keys:
| Key | Value | Default | Meaning |
|---|---|---|---|
pricing | "feeMultiplier" or "fixed" | "feeMultiplier" | Whether the amounts follow the network's fee multiplier or are charged as written |
<action>.owner | credits, 0 to 9223372036854775807 | 0 | Added to the contract's owner pot, which the contract owner claims |
<action>.moderators | credits, 0 to 9223372036854775807 | 0 | Added to the contract's moderators pot, which the moderation team shares. Needs moderation in the contract config |
Amounts are in credits: 1 Dash is 100,000,000,000 credits (1000 credits per duff).
Example
"post": {
"type": "object",
"actionFees": {
"pricing": "feeMultiplier",
"create": { "moderators": 100000000, "owner": 10000000 }
},
"properties": {
"text": { "type": "string", "minLength": 1, "maxLength": 280, "maxBytes": 560, "position": 0 }
},
"required": ["text"],
"additionalProperties": false
}
In a contract that declares moderation, creating a post costs an extra 0.001 Dash for the moderation team and 0.0001 Dash for the contract owner, at a fee multiplier of 1. Replacing, deleting and every other action cost only their gas.
How it works
- What is charged. With
fixedpricing, the declared amounts. WithfeeMultiplier, the declared amounts scaled by the fee multiplier of the epoch the action executes in (declared * multiplier_permille / 1000, rounded down), so a fee follows the network's fees. A scaled amount is held at the maximum number of credits instead of overflowing; such a fee refuses the action for an insufficient balance. - Who pays. Whoever pays the gas pays the fee: the signer, or the contract owner when they pay the gas of a token-paid action (see Token Costs). The contract owner never pays the
ownerpart, which would only travel through their pot back to them: a contract owner who pays, as the signer or as the gas sponsor, pays themoderatorspart only. A contract that sponsors gas should price that in: a sponsor's balance must cover the gas and those fees, or the transition falls back to the signer or is refused, as the token cost's rules say. - Only executed actions pay. A transition that fails, at any stage, owes no fee.
- Where it goes. One removal from the payer's balance, and one addition to each pot the fee has a part for. The fee is not part of the gas: the fee pools and block proposers get none of it.
The agreement: $actionFeeAgreement
The contract is read when the transition executes, not when it was signed. So every transition on an action that charges a fee carries an action fee agreement in its base, naming the fee its signer saw (version 2 of the document base transition, the default from protocol version 14):
"$actionFeeAgreement": {
"$formatVersion": "0",
"owner": 10000000,
"moderators": 100000000,
"feeMultiplier": { "knownPermille": 1000, "increaseTolerancePercent": 20 }
}
| Field | Meaning |
|---|---|
owner, moderators | The amounts the document type declares for the action, before any multiplier. They must match exactly, each part on its own |
feeMultiplier | Present for a feeMultiplier fee, left out for a fixed one. knownPermille is the fee multiplier the signer priced the fee with, in thousandths (1000 is 1x). increaseTolerancePercent is how far above it the executing epoch's multiplier may be, in percent of the known one: 20 accepts up to 1.2 times |
A transition on an action that charges a fee is refused:
- without an agreement (
DocumentActionFeeAgreementNotSetError, 40132), whoever pays, a sponsored transition included; - with other amounts, parts moved between the pots, or the other pricing (
DocumentActionFeeAgreementMismatchError, 40133). The signer reads the contract again; - when the executing epoch's multiplier is above what the agreement tolerates (
DocumentActionFeeMultiplierNotToleratedError, 40134). A multiplier that fell is always accepted, and what is charged follows the epoch's multiplier, never the known one.
Each refusal bumps the signer's nonce, and no action fee is charged. The mempool applies the same checks on arrival and on every recheck, so a transition whose agreement no longer holds leaves the mempool with the same error. An agreement on an action that charges nothing is ignored.
A client should build the agreement from the contract it showed its user, never from a contract fetched behind their back at signing time. In Rust, DocumentActionFeeAgreement::for_document_type_action builds it from a document type, and the SDK's document transition builders take it with with_action_fee_agreement.
A seated team's discount. On a document type that an elected contract moderates, the moderators part of an agreement may name less than the declared amount: exactly the share the contract's seated moderation charter takes (its moderatorsShare, in percent, rounded down to the credit). Everything else must still match. The action is then charged the agreed amount. Any other amount below the declared one, including a discount on a contract with no seated charter yet, is refused (DocumentActionFeeModeratorsShareMismatchError, 40139). A lower amount anywhere else, on a type the contract does not moderate or a contract that is not elected, is the plain mismatch (40133). See Elected Moderation.
The pots and the claim
The owner parts collect in the contract's owner pot and the moderators parts in its moderators pot. A ContractFeeClaim state transition pays a pot out:
- the owner pot goes whole to the contract owner, the only identity that may claim it;
- the moderators pot is split equally between the moderation team (the identities the contract appoints, or the owner alone when it appoints none), and any member of the team may claim it for all of them. A seated elected team splits it by its charter's reward split instead. What a split leaves over, less than a credit per member, stays in the pot;
- each pot is paid out at most once per epoch, and the two are independent.
A claim is refused when the signer is not a recipient of the pot (ContractFeeClaimNotAllowedError, 41113), when the pot was already paid out this epoch (ContractFeesAlreadyClaimedThisEpochError, 41111), or when a recipient would get less than a credit (ContractFeesNothingToClaimError, 41112). The JavaScript SDK reads the pots with contracts.feePots and claims with contracts.claimFees.
Rules at registration
- The meta-schema checks the shape (
JsonSchemaError, 10101): onlypricingand the six action keys;pricingone of the two values; each action an object withowner,moderatorsor both, each an integer from 0 to 9223372036854775807. - At least one action is priced, and a priced action charges something: an action whose parts are all 0 is refused, leave it out instead. The two parts of an action may not add up to more than 9223372036854775807 credits (
InvalidContractStructure, 10231). - A nonzero
moderatorspart needs a contract whose config declaresmoderation(DocumentActionFeesWithoutModerationError, 10902): the moderation team is who that pot is for. This is checked when the contract is created and when it is updated.
See also
- Document action fees, the deep dive
- Fee Pots and the Claim, for the pots, the claim and its proofs
- Token Costs (tokenCost), which prices the same six actions in tokens
- Contract-Level Keys and config, for
moderation - Contract Keywords overview, for the conventions of these tables
Indexes (indices)
A document type's indices list says which queries its documents can answer and which values must be unique. Without an index, a query can only address a document by its $id. With one, Drive keeps the documents sorted by the index's properties, so a query that fixes those properties reaches the matching documents directly instead of reading the whole type. Every index costs storage and processing on each write, and an index can never be added, removed or changed once the document type exists, so a contract author decides them before the document type is registered.
This chapter covers the keywords every index can use: name, properties, unique, nullSearchable and skipIfAbsent. The other index keywords, for contests, counts, sums, rankings, time windows and index-only types, have their own chapters (see More index keywords).
Example
"post": {
"type": "object",
"indices": [
{ "name": "byOwnerTime", "properties": [{ "$ownerId": "asc" }, { "$createdAt": "asc" }] },
{ "name": "bySlug", "properties": [{ "$ownerId": "asc" }, { "slug": "asc" }], "unique": true },
{ "name": "byTopic", "properties": [{ "meta.topic": "asc" }], "nullSearchable": false }
],
"properties": {
"slug": { "type": "string", "minLength": 1, "maxLength": 63, "position": 0 },
"text": { "type": "string", "maxLength": 280, "position": 1 },
"meta": {
"type": "object",
"properties": {
"topic": { "type": "string", "minLength": 1, "maxLength": 32, "position": 0 }
},
"additionalProperties": false,
"position": 2
}
},
"required": ["$createdAt", "slug", "text"],
"additionalProperties": false
}
byOwnerTime lists one author's posts in the order they were written. bySlug lets each author use a slug once: a second post by the same owner with the same slug is refused. byTopic finds posts by a property nested inside meta, and leaves out posts that have no topic.
indices
| Where | document type |
| Value | array of 1 to 10 index objects |
| Default | absent: the type has no index, and its documents can only be addressed by $id |
| Since | protocol version 1 |
| On update | Fixed: an index may not be added, removed or changed (DataContractInvalidIndexDefinitionUpdateError, 10217) |
| Errors | DuplicateUniqueIndexError (40105) |
Each entry of indices is one index. Drive builds a tree for it when the contract is registered and maintains it on every create, replace, transfer, purchase, price update and delete of a document of the type.
A query uses an index when its where clauses fix the index's leading properties, in order, and it orders by the properties that follow. An index on [a, b] answers a == x, a == x AND b == y, and a == x ordered by b, but not b == y alone. The query picker and the tree layout are described in Indexes.
Rules at registration:
- At most 10 indexes, and at least one when the key is present (an empty
indicesarray is refused by the meta-schema). - No two indexes may have the same properties in the same order (
DuplicateIndexError, 10201). - At most one index may be contested, and a type with a contested index may have no other unique index (
ContestedUniqueIndexWithUniqueIndexError, 10249). See Contested Indexes.
name
| Where | index |
| Value | string, 1 to 32 characters |
| Default | none: required |
| Since | protocol version 1 |
| On update | Fixed: indexes are compared by name, so renaming an index is a removal plus an addition (10217) |
| Errors | DuplicateIndexNameError (10211) at registration |
The name identifies the index in queries that name one, in error messages and in the contract update rule. Two indexes of one document type may not share a name (DuplicateIndexNameError, 10211).
properties
| Where | index |
| Value | array of 1 to 10 objects, each { "<property path>": "asc" } with exactly one key |
| Since | protocol version 1 |
| On update | Fixed (10217) |
| Errors | UndefinedIndexPropertyError (10209), SystemPropertyIndexAlreadyPresentError (10208), InvalidIndexPropertyTypeError (10206), InvalidIndexedPropertyConstraintError (10205), all at registration |
The indexed properties, in order. The order matters: a query uses the index through a prefix of the list. The only sort order the meta-schema accepts is "asc"; a query may still walk an index in descending order.
What may be indexed:
- A top-level property of the type, by its name.
- A property inside an object, by its dotted path. DPNS indexes
records.identity, theidentityproperty of a domain'srecordsobject. - System properties:
$ownerId,$createdAt,$updatedAt,$transferredAt, their*BlockHeightand*CoreBlockHeightvariants,$creatorIdon a type that records it, and$moderatedAtand$moderatedByon a type that listsmoderatorAbilities.changeFields, in a non-unique index, from protocol version 14 (see System Properties). A timestamp or block height is only recorded when the type lists it inrequired; an index on one that is not required holds every document under null. - Not
$id, which the document type's primary tree already indexes (SystemPropertyIndexAlreadyPresentError, 10208). - A value of the document a reference points at, from protocol version 14:
"<reference property>.<field>", such aspostId.$ownerId, read from the referenced document and never stored. See Values of Referenced Documents.
What each indexed property must be, because its value becomes a GroveDB key of at most 255 bytes:
- A property the type defines (
UndefinedIndexPropertyError, 10209). - Not an object, an array of values or a typed array (
InvalidIndexPropertyTypeError, 10206). A byte array, including an identifier, is fine. - A string must declare
maxLengthof at most 63, since a character can take four bytes. A byte array must declaremaxItemsof at most 255. A missing or larger bound is refused (InvalidIndexedPropertyConstraintError, 10205). An index with a ranking has tighter bounds (see Ranked Indexes). - From protocol version 14, not a transient property, nor one inside a transient object: its value is never stored, so the index would never hold it. See transient.
unique
| Where | index |
| Value | boolean |
| Default | false |
| Since | protocol version 1 |
| On update | Fixed (10217) |
| Errors | DuplicateUniqueIndexError (40105) |
On a unique index no two documents may hold the same values for all of the index's properties. A create, replace, transfer, purchase or price update that would make a second document with the same values is refused with DuplicateUniqueIndexError (40105); so is a moderator's restore of a deleted document whose values have been taken since. A replace that leaves the indexed values as they were is not held against the document itself.
A document that leaves any of the indexed properties out is not held to the index: uniqueness cannot be decided on a missing value, so two such documents may coexist. Make the properties required when every document must be unique.
Uniqueness is per index. bySlug above is unique over the pair ($ownerId, slug), so two authors may use the same slug; a unique index on slug alone would make each slug global.
A unique index with a timeRange or an integerRange treats two documents in the same window as equal on the bucketed property. A unique index cannot carry a ranking.
nullSearchable
| Where | index |
| Value | boolean |
| Default | true |
| Since | protocol version 1 |
| On update | Fixed (10217) |
With the default, a document whose indexed properties are all missing is still entered in the index, under null, so a query for the null value finds it. false leaves such a document out of this index: it exists, and other indexes and $id still reach it, but this index does not. A document with only some of the properties missing is entered either way.
DPNS sets "nullSearchable": false on its records.identity index, so domains that point at no identity take no room in it.
nullSearchable: false is refused on an index with a ranking, a timeRange or an integerRange, on an index of an index-only type, and together with skipIfAbsent.
skipIfAbsent
| Where | index |
| Value | true, or an array of 1 to 10 of the index's property names |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed (10217) |
A document that leaves out a property of the index's skip set writes nothing into this index, and its delete looks for nothing there. true makes the skip set every optional property of the index; an array names it. A value of a referenced document is never in the true set, since whether it can be absent depends on the referenced type; the array may name one. The skip property can sit anywhere in the index, under a timeRange or integerRange window too. The index then holds only the documents carrying every skip property, and its counts and rankings are "among the documents that have them". A present but empty value is not absent and is indexed.
"post": {
"type": "object",
"documentsMutable": true,
"indices": [
{
"name": "byHashtagLanguageTime",
"properties": [{ "hashtag": "asc" }, { "language": "asc" }, { "$createdAt": "asc" }],
"skipIfAbsent": ["hashtag"]
},
{
"name": "byDayHashtag",
"properties": [{ "$createdAt": "asc" }, { "hashtag": "asc" }],
"rangeCountable": true,
"rankedCountable": true,
"timeRange": { "on": "$createdAt", "range": 86400, "step": 86400 },
"skipIfAbsent": true
}
],
"properties": {
"hashtag": { "type": "string", "maxLength": 61, "position": 0 },
"language": { "type": "string", "maxLength": 8, "position": 1 },
"text": { "type": "string", "maxLength": 280, "position": 2 }
},
"required": ["$createdAt", "text"],
"additionalProperties": false
}
An untagged post writes nothing into either index: no entry under hashtag, and no day window in byDayHashtag, whose ranking of today's hashtags therefore never has to step over an empty hashtag. A tagged post without a language is entered into byHashtagLanguageTime under null for language, since the array leaves language out of the skip set. A replace that adds or drops the hashtag moves the post into or out of both indexes.
A query only uses a skip index when it constrains every skip property, so it cannot silently miss the documents the index skipped. On a stored type the constraint must be one no missing value can meet: an equality or in with values that are neither null nor empty (an empty byte array is keyed like null), a range with a non-empty lower bound, startsWith with a non-empty prefix, or ranking by the property. Ordering by the property alone, or a range with only an upper bound, does not count: an index that does not skip would return the documents without the property too (they sort first, under null).
Rules at registration:
- The skip set is not empty: an index whose properties are all required could never skip.
- Each skip property is a top-level property of the type, not a system property, and not listed in
required, or a value of a referenced document that can be absent. - No
rankedCountableatlevel sits above the index's deepest skip property: it would count only documents carrying that property, and no query could read it without binding the property. - The index is not contested, and does not also set
nullSearchable: false(the skip already leaves out a document with every indexed value missing). - On a stored type, a skip property that is a byte array sets
minItemsto at least 1 (on the referenced type, for a value read from a referenced document): an empty byte array is keyed like a missing value. - On a stored type, a ranking at a skip property's level does not share that level with an index that keeps null for the property: that index would create the null value, and the ranking would show it as a group with no documents.
- On an index-only type the skip set holds every optional property of the index, and each optional property needs a skip index of its own, without a
timeRange(a window keeps the value only until the window drains).
Respelling true as the array of the same properties, or the other way round, is no change on a contract update.
Null handling
A property is null in an index when the document leaves it out. Putting the rules above together:
| The document's indexed values | Entered in the index? | Held to unique? |
|---|---|---|
| all present | yes | yes |
| some missing | yes, under null for the missing ones | no |
| all missing | only if nullSearchable is true | no |
An index with skipIfAbsent leaves out a document missing any of its skip properties, whatever the other values are.
The storage shape of each case is in Null Handling.
Changing indexes
Drive builds an index's trees when the document type is registered and never backfills them, so an index added later would miss every existing document, and a removed or changed one would leave orphaned trees. A contract update may therefore not add, remove or change any index of an existing document type. From protocol version 14 the update is refused with DataContractInvalidIndexDefinitionUpdateError (10217), naming the first index that differs. Indexes are compared by name: reordering the indices array is no change, and renaming an index is a removal plus an addition. Earlier protocol versions refused every such change as well, though not always with this error.
A document type that the update adds may declare any indexes, as a new contract may.
Limits
| Limit | Value |
|---|---|
| Indexes per document type | 10 |
| Properties per index | 10 |
| Characters in an index name | 32 |
| Characters in an index property path | 256 |
maxLength of an indexed string | 63 (lower on a ranked index) |
maxItems of an indexed byte array | 255 (lower on a ranked index) |
| Contested indexes per document type | 1 |
More index keywords
An index entry may carry more keywords, each with its own chapter:
- Contested Indexes:
contestedturns a unique index into a scarce resource that masternodes award by vote, the way DPNS gives out names. - Counts, Sums and Averages:
countable,summable,averageableand theirrange*forms keep totals per indexed value, so counts, sums and averages are read without walking the documents. - Ranked Indexes:
rankedCountable,rankedSummableandrankedAverageableorder the indexed values by those totals, for "top 10" queries with proofs. - Time-Range Indexes:
timeRangegroups documents into time windows, for "trending this hour" queries. - Integer-Range Indexes:
integerRangegroups documents into windows of an integer property, for counts and rankings per price or score band. - Index-Only Types:
terminal,preallocated,skipIfAbsentandoutlivesDeleteshape the indexes of a type whose documents live only in their indexes. - Values of Referenced Documents: an index property
"<reference property>.<field>"holds a value of the document a reference points at, which the document does not store.
See also
- Indexes in the Drive part: the index trie, the GroveDB layout and the query picker.
- Contested Indexes, Counts, Sums and Averages, Ranked Indexes, Time-Range Indexes, Integer-Range Indexes, Index-Only Types.
- System Properties for what
$ownerId,$createdAtand the others hold. - Contract Keywords for the conventions of these chapters.
Contested Indexes
A unique index normally works first come, first served: the first document to take a value keeps it, and every later one is refused as a duplicate. contested changes that for the values a contract considers valuable. A document whose values fall in the contested range opens a contest, or joins one already open for the same values, and masternodes and evonodes vote on which identity gets them. DPNS uses it so that a short name such as alice.dash cannot simply be taken by whoever submits first. A contract author reaches for it when a unique value is scarce and should be awarded rather than raced for.
| Where | a unique index |
| Value | object: resolution (required), fieldMatches, description |
| Since | protocol version 1; resolution: 1 from protocol version 14 |
| On update | Fixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217) |
| Errors | DocumentContestNotPaidForError (40114), DocumentContestCurrentlyLockedError (40110), DocumentContestNotJoinableError (40111), DocumentContestIdentityAlreadyContestantError (40112), DocumentContestIndexMismatchError (40118), DocumentContestNotRequiredError (40119), DocumentContestMaximumContendersReachedError (40141); DuplicateUniqueIndexError (40105) for values outside the contested range |
Example
The domain document type of the DPNS contract:
{
"name": "parentNameAndLabel",
"properties": [
{ "normalizedParentDomainName": "asc" },
{ "normalizedLabel": "asc" }
],
"unique": true,
"contested": {
"fieldMatches": [
{ "field": "normalizedLabel", "regexPattern": "^[a-zA-Z01-]{3,19}$" }
],
"resolution": 0,
"description": "If the normalized label part of this index is less than 20 characters (all alphabet a-z, A-Z, 0, 1, and -) then a masternode vote contest takes place to give out the name"
}
}
A name is unique under its parent domain. A label of 3 to 19 characters made of letters, 0, 1 and - is contested: registering it opens a masternode vote, which may give it to a contender or lock it. Any other label, a longer one or one with other digits, is registered first come, first served, and a second registration of it is a duplicate.
The keys
| Key | Value | What it does |
|---|---|---|
resolution | 0 or 1, required | How the contest is decided. 0: masternodes vote for a contender, abstain, or lock the value so nobody gets it. 1 (from protocol version 14): masternodes vote for a contender or abstain; there is no lock, so the contest always ends with a winner. |
fieldMatches | array of at least one { "field", "regexPattern" } | Which values are contested. field is a property path of the document; regexPattern a regular expression its value must match. Each is 1 to 256 characters. |
description | string, 1 to 256 characters | A note for readers. Consensus does not read it. |
How it works
Which documents are contested. A document is contested when, for every fieldMatches entry, the document holds a string at field and regexPattern matches it. A missing value, a value that is not a string, or one entry that does not match makes the document an ordinary unique-index document. Without fieldMatches, every document the index covers is contested. The pattern uses the syntax of Rust's regex crate and matches anywhere in the value, so write ^ and $ to match the whole of it, as DPNS does.
Values outside the contested range behave exactly like any unique index: the first document takes the value, and a later create with the same values is refused with DuplicateUniqueIndexError (40105).
Opening or joining a contest. A create whose values are contested must carry $prefundedVotingBalance, a pair [indexName, amount]: the contested index's name and the most credits the contender will pay to fund the vote. The document is not stored under the index yet; it is held as a contender until the contest ends. A create is refused:
- with
DocumentContestNotPaidForError(40114) when it carries no prefunded balance, or less than the contest's fund. The fund is the contested document fund, 0.1 Dash. From protocol version 14 it doubles once the contest holds 250 contenders and again for every 50 more, and a contender is charged the fund and keeps what it stated beyond it; before 14 every contender stated and paid exactly the fund. - with
DocumentContestIndexMismatchError(40118, from protocol version 14) when the pair names another index than the contested one its values match. - with
DocumentContestNotRequiredError(40119, from protocol version 14) when the document is not contested but carries a prefunded balance. - with
DocumentContestNotJoinableError(40111) when the contest's join window (one week on mainnet) has passed. - with
DocumentContestIdentityAlreadyContestantError(40112) when its owner is already a contender. - with
DocumentContestMaximumContendersReachedError(40141, from protocol version 14) when the contest already holds 1,000 contenders. - with
DocumentContestCurrentlyLockedError(40110) when an earlier contest for these values ended locked.
The vote. Masternodes and evonodes vote with MasternodeVote transitions for the length of the poll (two weeks on mainnet). Under resolution: 0 the contender with the most votes wins unless the lock tally beats it, and the contest always runs its full length, even with one contender. Under resolution: 1 a contender always wins, and a contest that still has a single contender when its join window closes is awarded to it at once. From protocol version 14 a tie goes to the earliest contender.
After the contest. The winning document is stored and held by the index like any unique-index document; the other contenders' documents are removed. The contest's result stays readable.
The fund, the windows, the tallies and the special case of moderation elections are described in Contested Documents.
Rules at registration
- The index is
unique: true. A contested index that is not unique is refused (InvalidContractStructure, 10231). - The document type's documents cannot be replaced:
documentsMutable: false(ContestedUniqueIndexOnMutableDocumentTypeError, 10248). They may still be transferred and sold, as DPNS domains are. - A document type has at most one contested index, and no other unique index beside it (
ContestedUniqueIndexWithUniqueIndexError, 10249). resolutionis present and is0or1;1is refused before protocol version 14.fieldMatches, when present, holds at least one entry, and eachregexPatternis a valid regular expression (RegexError, 10247).- A contested index cannot carry a
timeRange, anintegerRangeor a ranking (a ranking needs a non-unique index), and an index-only type cannot have one. - A document type with a contested index cannot set
ttl, normoderatorAbilities.delete(see Deletion): a moderator's restore puts a document back by an ordinary insert, and a contested value is only awarded through a vote. - From protocol version 14, a document type with a contested index that sums a property (
summable,averageable,documentsSummableordocumentsAverageable) declares that property with aminimumof at least -134217728 and amaximumof at most 134217728 (±2^27,max_contested_summed_value_magnitude). The end of a contest adds the winner's value to the type's sums, after any number of other documents were written, and a sum that left the signed 64-bit range there could not be stored. Values this small keep the sums in range short of 2^36 documents. Checked when a contract is registered or updated, so contracts registered earlier keep their bounds.
See also
- Contested Documents for the contest's lifecycle, fund, resolutions, ties and storage.
- Indexes for
uniqueand the other index keywords. - Mutability for
documentsMutable, which a contested type sets tofalse.
Counts, Sums and Averages
Counting documents normally means fetching them and counting what comes back, which grows with the number of documents and proves every one of them. The keywords in this chapter make Drive keep running totals inside its trees instead, so a count, a sum or an average is read without visiting the documents, and proved with a short proof. The document type keywords keep totals over all of a type's documents. The index keywords keep totals per indexed value, and with their range* forms over ranges of values. An average is never stored as such: the platform returns the count and the sum of the same documents, and the client divides.
Every total is updated by every write that touches it, so each flag adds to the cost of writing documents of the type. The flags choose the layout of the type's trees, so all of them are fixed when the document type is created.
Example
"tip": {
"type": "object",
"documentsMutable": false,
"documentsAverageable": "amount",
"indices": [
{ "name": "byRecipient", "properties": [{ "recipient": "asc" }], "averageable": "amount" },
{ "name": "byDay", "properties": [{ "day": "asc" }], "summable": "amount", "rangeSummable": true }
],
"properties": {
"recipient": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 0
},
"amount": { "type": "integer", "minimum": 1, "maximum": 4294967295, "position": 1 },
"day": { "type": "integer", "minimum": 0, "maximum": 4294967295, "position": 2 }
},
"required": ["recipient", "amount", "day"],
"additionalProperties": false
}
documentsAverageable keeps the number of tips and their total, so the average tip over the whole type is one read. byRecipient keeps the same pair per recipient, so "how many tips did this identity get, and how much on average" is one read. byDay keeps the sum per day and, with rangeSummable, answers "total tipped between day 100 and day 130" without visiting each day.
documentsCountable
| Where | document type |
| Value | boolean |
| Default | false |
| Since | protocol version 12 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
Keeps the number of the type's documents in its primary tree, so a count query with no where clause is one read, with a proof. Without it, such a query is refused rather than answered slowly.
documentsSummable
| Where | document type |
| Value | the name of an integer property, 1 to 64 characters |
| Default | absent |
| Since | protocol version 12 |
| On update | Fixed (40212) |
Keeps the sum of the named property over all the type's documents, so a sum query with no where clause is one read. On a type with documentsKeepHistory, the sum counts each document's current revision only.
The property must exist on the type, be listed in required (a missing value would leave the sum wrong when the document is deleted), and hold values that fit a signed 64-bit sum. Integers are stored in the smallest width their bounds allow when the contract uses sizedIntegerTypes, and a property that becomes an unsigned 64-bit integer is refused: "minimum": 0 with no maximum does, so add a maximum of at most 4294967295, or give no bounds at all.
documentsAverageable
| Where | document type |
| Value | the name of an integer property, 1 to 64 characters |
| Default | absent |
| Since | protocol version 12 |
| On update | Fixed (40212) |
Shorthand for documentsCountable: true plus documentsSummable on the named property: the count and the sum an average is computed from. The storage is exactly that of the two flags. When documentsSummable is also written it must name the same property, and an explicit documentsCountable: false beside it is refused as a contradiction.
Document type rangeCountable, rangeSummable, rangeAverageable
| Where | document type |
| Value | boolean each |
| Default | false |
| Since | protocol version 12 |
| On update | Fixed (40212) |
These keep the count, the sum, or both at every node of the primary tree rather than only at its root, which is what range and offset aggregates over the primary key need. They are rarely useful: a range count or sum is almost always wanted over an indexed property, which is what the index keywords of the same names give.
rangeCountableimpliesdocumentsCountable.rangeSummableneedsdocumentsSummableordocumentsAverageable.rangeAverageableis shorthand forrangeCountableplusrangeSummable, and needsdocumentsAverageable. An explicitfalsefor either of the two beside it is refused as a contradiction.
countable
| Where | index |
| Value | "notCountable", "countable", "countableAllowingOffset", or a boolean (true is "countable", false is "notCountable") |
| Default | "notCountable" |
| Since | protocol version 12 |
| On update | Fixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217) |
Keeps a document count for each value of the index, so "how many documents have these values" is one read per value, with a proof. A count query is served by a countable index whose properties exactly match its where clauses, each an equality or an in: a countable index on [brand, color] answers brand == "a" AND color == "red", and color in [...] under a fixed brand with one count per color, but not brand == "a" alone. Declare one countable index per shape of count the application needs.
"countableAllowingOffset" keeps a count at every node of the index's trees, not only per value. It costs more on every write and prepares the index for offset queries; use "countable" unless you need it. The boolean form is kept for contracts written before the string form existed.
On a unique index the flag changes almost nothing, since each value holds at most one document; it only counts the documents that leave an indexed property out.
summable
| Where | index |
| Value | the name of an integer property, 1 to 64 characters |
| Default | absent |
| Since | protocol version 12 |
| On update | Fixed (10217) |
Keeps the sum of the named property for each value of the index, the way countable keeps a count, and serves sum queries under the same exact-match rule. The property follows the rules of documentsSummable: it exists, is required, and fits a signed 64-bit sum.
averageable
| Where | index |
| Value | the name of an integer property, 1 to 64 characters |
| Default | absent |
| Since | protocol version 12 |
| On update | Fixed (10217) |
Shorthand for countable: "countable" plus summable on the named property, which is what an average query per value needs. When summable is also written it must name the same property. An explicit countable: "countableAllowingOffset" beside it is kept; an explicit "notCountable" (or false) is refused as a contradiction.
Index rangeCountable, rangeSummable, rangeAverageable
| Where | index |
| Value | boolean each |
| Default | false |
| Since | protocol version 12 |
| On update | Fixed (10217) |
These answer aggregates over a range of the index's last property: "how many reviews between grade 60 and 80", "total tipped from day 100 to day 130". The count or sum is kept at every node of the tree of that property's values, so a range total is read by walking the edges of the range, with a proof whose size grows with the logarithm of the number of values rather than with the documents. The same index also returns one total per distinct value in a range. The query fixes the properties before the last one, with equalities or an in.
rangeCountablemakes the index countable: an omittedcountablebecomes"countable", an explicit"countableAllowingOffset"is kept, and an explicit"notCountable"is refused. Before protocol version 14,countablehad to be written out beside it.rangeSummableneedssummableoraverageable.rangeAverageableis shorthand forrangeCountableplusrangeSummable, and needsaverageable. An explicitfalsefor either of the two beside it is refused.
A range total costs more on each write than a per-value total, since every node of the tree carries it. The ranked keywords build on these flags: see Ranked Indexes.
How they combine
- Count and sum on one tree. An index with both a count and a sum (by
averageable, or bycountableplussummable) keeps both in one tree, and one proof returns the pair. The same holds for the document type flags. - One summed property per type. Every
summable,averageable,documentsSummableanddocumentsAverageableof a document type must name the same property: the sums share their trees, which have no room to tell two properties apart. To sum two properties, use two document types. - Type and index flags are independent.
documentsCountablegives the unfiltered total; a countable index gives filtered ones. A type may have both. - Index-only types take the index keywords but refuse the document type ones, since they have no primary tree. See Index-Only Types.
- Queries must match. A count or sum query that no index serves exactly is refused; there is no slow fallback. Pick the indexes for the queries the application will make.
Rules at registration
- A summed property exists on the type, is an integer that fits a signed 64-bit sum (not an unsigned 64-bit integer), and is listed in
required. - All summed properties of a type are the same property.
- From protocol version 14, on a type with a contested index, the summed property declares a
minimumof at least -134217728 and amaximumof at most 134217728 (±2^27). - From protocol version 14, on a type with a
ttl, the summed property declares aminimumof at least 0, or aminimumof at least -134217728 and amaximumof at most 134217728 (±2^27). - On other types, each value only has to fit a signed 64-bit integer.
- The shorthands agree with their longhand where both are written, and no explicit
falseor"notCountable"contradicts them. - Each
range*flag has its prerequisite, as listed above. - The meta-schema refuses a malformed value (
JsonSchemaError, 10101).
Choosing what to set
| The application needs | Set |
|---|---|
| The number of documents of the type | documentsCountable: true |
| The sum, or the average, of a property over the whole type | documentsSummable or documentsAverageable |
A count per value (where author == x) | countable: "countable" on an index whose properties are exactly those of the query |
| A sum or an average per value | summable or averageable on such an index |
A count, sum or average over a range (where grade > 60) | the matching range* flag on an index whose last property is the ranged one |
| The top or bottom values by count, sum or average | the range* flags plus a ranking: see Ranked Indexes |
See also
- Document Count Trees and Document Sum Trees for the tree variants, the query endpoints and what each flag costs.
- Count Index Examples, Sum Index Examples and Average Index Examples for worked queries and proof sizes.
- Indexes for the index keywords every index uses.
- Ranked Indexes for ordering values by these totals.
Ranked Indexes
A ranked index answers "which values score highest": the five restaurants with the best average grade, the ten hashtags with the most posts, the three sellers with the largest sales. The range aggregates of Counts, Sums and Averages keep a count or a sum for every value of an index, but in the order of the values, so finding the top five means reading every value. A ranking adds a second, ordered view of the same totals, and a "top K" query reads K entries from one end of it, with a proof that grows with K rather than with the number of values. Each ranking costs one more tree that every write under the index updates, so a contract declares only the rankings it will query.
The index's groups are the distinct values of its last property. The three keywords rank the groups by document count, by the sum of the summed property, or by its average.
Example
"review": {
"type": "object",
"indices": [
{
"name": "byRestaurant",
"properties": [{ "restaurantId": "asc" }],
"averageable": "grade",
"rangeAverageable": true,
"rankedAverageable": true,
"rankedCountable": true
}
],
"properties": {
"restaurantId": { "type": "string", "minLength": 1, "maxLength": 32, "position": 0 },
"grade": { "type": "integer", "minimum": 0, "maximum": 100, "position": 1 }
},
"required": ["restaurantId", "grade"],
"additionalProperties": false
}
averageable and rangeAverageable keep the count and the sum of grade per restaurant; rankedAverageable orders the restaurants by average grade and rankedCountable by number of reviews. The query
SELECT avg(grade) FROM review GROUP BY restaurantId ORDER BY avg(grade) DESC LIMIT 3
returns the three best-rated restaurants with their counts and sums, proved.
rankedCountable
| Where | index |
| Value | boolean, or { "at": <property name or array of 1 to 10 names> } |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217) |
Ranks groups by how many documents they hold. Needs rangeCountable: true on the index, or rangeAverageable: true, which implies it.
true ranks the values of the index's last property. On a compound index, the ranking is kept separately for each value of the properties before it: on [city, restaurantId] each city has its own ranking of restaurants, and a query names the city.
The object form { "at": ... } places the ranking at another level of the index. Naming an earlier property ranks that property's values by the number of documents beneath them, whatever the later properties hold:
{
"name": "byHashtagPost",
"properties": [{ "hashtag": "asc" }, { "postId": "asc" }],
"countable": "countable",
"rangeCountable": true,
"rankedCountable": { "at": ["hashtag", "postId"] }
}
"hashtag" in at ranks hashtags by their total posts across all post ids; "postId", the last property, is the same as true and ranks the posts under one hashtag. An array declares several rankings on one index; each level named costs one more ordered tree to maintain on every write beneath it. A query addresses a ranking by the property it groups by, with every property before that one fixed.
at names only the index's own properties, each once. The object form cannot be combined with rankedSummable or rankedAverageable: a ranking at an earlier level is fed by a chain of counts that cannot also carry a sum. A summableOffCountIndex index is the exception: its counters feed a chain of sums, and of counts too from the shallowest average ranking down, so it takes all three at any level. An average reads a level only where that chain carries counts: above the shallowest average ranking its value trees sum but count nothing. There a document count is its sums (likes, not posts), so rankedCountable declares the same ranking as rankedSummable, which a ranked count(*) and a ranked sum of the source both read, and it needs no rangeCountable.
rankedSummable
| Where | index |
| Value | boolean, or { "at": <property name or array of 1 to 10 names> } on a summableOffCountIndex index |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed (10217) |
Ranks the groups of the index's last property by the sum of the index's summable property: the recipients who received the most, the products that sold the most units. Needs rangeSummable: true, or rangeAverageable: true. On a summableOffCountIndex index the sum is the source's entries, and at ranks an earlier level by it: authors by the likes their posts received. A ranked count(*) over such an index reads this ranking too, since its sums are its document counts.
rankedAverageable
| Where | index |
| Value | boolean, or { "at": <property name or array of 1 to 10 names> } on a summableOffCountIndex index |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed (10217) |
Ranks the groups of the index's last property by the average of the averageable property. On a summableOffCountIndex index the average is entries per group, and at ranks an earlier level by it: authors by likes per post. Needs both range totals: rangeAverageable: true, or rangeCountable: true with rangeSummable: true.
The three keywords are independent. rankedAverageable does not imply rankedCountable or rankedSummable, unlike averageable, which is shorthand for a count and a sum. Declare each ranking the application will query, and no other.
Queries
A ranked query names one aggregate, groups by the ranked property, orders by the aggregate and takes a limit, with an optional offset. On a compound index such as [city, restaurantId]:
SELECT count(*) FROM review WHERE city == "London" GROUP BY restaurantId ORDER BY count(*) DESC LIMIT 5
Every property before the grouped one must be fixed with an equality; at most one of them may instead be an in of 2 to 10 values, whose rankings are merged. There is no ranking across different values of those properties. A ranked index still answers the per-value range queries the same range* flags answer (one entry per value in the range). A range total, one count, sum or average over a whole range, is not available through an index whose path passes through a ranked level, one it ranks itself or one another index ranks at a level the two share: the ranked trees are indexed trees, which grovedb neither totals a range over nor proves a range total through, and an unproven total through a ranked level is refused as well so that a proved and an unproven read agree. Such a query is refused with a hint to group by the last property.
The request shape, ties, offsets and proofs are described in Ranked Index Examples.
Rules at registration
- Its range totals. Each ranking needs the range flags listed above, checked both by the meta-schema and by the parser. A written-out
falseneeds nothing. - A non-unique index. Every group of a unique index holds at most one document, so a ranking on one is refused, and a contested index, which is unique, cannot have one either.
nullSearchableleft attrue. Withfalse, documents missing the property would still leave an empty group in the ranking.- A shorter key. The ranked property's value becomes part of the ordered tree's key, behind an 8-byte sort key (16 bytes when
rankedAverageableranks that property), so its worst case must fit in what is left of the 255-byte limit. A ranked string takesmaxLengthof at most 61, or 59 at a propertyrankedAverageableranks; a ranked byte array takesmaxItemsof at most 247, or 239. A larger bound is refused (InvalidIndexedPropertyConstraintError, 10205). Identifiers, integers and other fixed-width values always fit. The bound applies to each ranked property, by the rankings at that property: the last one when a ranking is declared withtrue, and each property named inat. - Time and value windows. On a
timeRangeorintegerRangeindex, a ranking must sit below the bucketed property, which gives one ranking per window. A single-property bucketed index cannot be ranked, andatcannot name the bucketed property. - Other indexes of the type. A compound ranked index
[p1, ..., pn]is refused when another countable or summable index ends at exactly[p1, ..., pn-1]. A ranking at an earlier level (at) also restricts the other indexes that reach that level; the full table is in Shape Restrictions. - One index per property list. Two indexes with the same properties are a
DuplicateIndexError(10201), so the rankings of one property list go on one index. - Protocol version 14. Earlier versions do not know the keywords and refuse them.
A broken rule other than the key length is refused as InvalidContractStructure (10231), or by the meta-schema as JsonSchemaError (10101).
Costs
Each ranking is one ordered tree, rewritten whenever a document under it is created, changed or deleted, on top of the range totals it is built from. Two rankings on one index cost two rewrites per write; a fully ranked at array costs one per ranked level. A ranking at the index's first property gets its tree when the contract is registered; a ranking at a deeper level gets one tree per value above it, as documents arrive.
See also
- Document Ranked Trees for the tree variants, prefix-level rankings and how the rankings are maintained.
- Ranked Index Examples for worked queries and proofs.
- Counts, Sums and Averages for the range totals a ranking builds on.
- Time-Range Indexes for rankings per time window.
Time-Range Indexes
timeRange groups an index's documents into time windows by one of their timestamps, so that "the most used hashtags this hour" or "posts per day" is a question about one window rather than a scan over time. Each window is range seconds long and a new one starts every step seconds; when windows overlap, a document belongs to each window that contains its timestamp. Combined with the count and ranking keywords, a time-range index serves trending lists and leaderboards per window, with proofs. It costs one set of index entries per window a document falls in, and a ttl lets old windows expire so that this data does not stay in state forever.
| Where | index; buckets the index's first property |
| Value | object: on, range, step (required), phase, ttl |
| Default | absent: the timestamp is indexed as it is |
| Since | protocol version 14 (ttl included) |
| On update | Fixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217) |
| Errors | DuplicateUniqueIndexError (40105) on a unique time-range index |
Example
"post": {
"type": "object",
"indices": [
{
"name": "hourlyByHashtag",
"properties": [{ "$createdAt": "asc" }, { "hashtag": "asc" }],
"timeRange": { "on": "$createdAt", "range": 3600, "step": 900, "ttl": 86400 },
"countable": "countable",
"rangeCountable": true
},
{
"name": "onePerDay",
"properties": [{ "$createdAt": "asc" }, { "$ownerId": "asc" }],
"unique": true,
"timeRange": { "on": "$createdAt", "range": 86400, "step": 86400 }
}
],
"properties": {
"hashtag": { "type": "string", "minLength": 1, "maxLength": 63, "position": 0 },
"text": { "type": "string", "maxLength": 280, "position": 1 }
},
"required": ["$createdAt", "hashtag", "text"],
"additionalProperties": false
}
hourlyByHashtag keeps one-hour windows starting every 15 minutes, so each post is counted in four windows, and the counts per hashtag in the window that covers the last hour are one query. Its entries expire a day after their window starts. onePerDay has non-overlapping daily windows and is unique, so an identity may post once per day: a second post in the same day is refused with DuplicateUniqueIndexError (40105).
The keys
| Key | Value | What it does |
|---|---|---|
on | "$createdAt", "$updatedAt" or "$transferredAt", required | The timestamp to bucket. It must be the index's first property and be listed in the type's required, since a timestamp that is not required is never recorded. |
range | integer seconds, at least 1, required | The length of each window. An exact multiple of step. |
step | integer seconds, at least 1, required | The time between the starts of two windows. |
phase | integer seconds, default 0 | Moves the window boundaries: windows start at phase + k * step from the Unix epoch. Less than step and less than one year (31536000). Daily windows cut at 06:00 UTC take "phase": 21600. |
ttl | integer seconds, at least 1 | How long the index's entries live after their window starts. At least range and at most 604800 (one week) at protocol version 14. Absent, entries live forever. |
How it works
Buckets. For each document, the index stores the start of every window that contains the document's timestamp, in milliseconds, in place of the timestamp itself. With range equal to step the windows do not overlap and a document is in exactly one; otherwise it is in range / step of them, the index's overlap factor, and costs that many sets of entries. The overlap factor is at most 24 at protocol version 14. Windows are declared in seconds because block time, which the timestamps come from, moves in steps of about five seconds.
A source that changes, $updatedAt or $transferredAt, moves the document into the current windows each time it changes. Several indexes may bucket the same timestamp with different grids; each grid is stored apart from the others.
Queries. A query selects one window of the index with an IN_TIME_RANGE clause on the bucketed timestamp:
newest: the window that started most recently, which covers the latest slice of time (up to onestep).oldest: the oldest window still open, which covers nearly a fullrangeof history: the one for "trending over the last hour".byStart: the window starting at a given millisecond time on the grid, which reaches past windows.
newest and oldest are resolved against block time on the server and against the signed time of the response when verified. When the timestamp is bucketed by several grids, the query names the grid. A query may select at most one window. A window with no documents, including one that has not started, is a proved empty answer. Counts, sums and rankings declared on the index are then per window: a ranking below the bucketed timestamp is one leaderboard per window.
Uniqueness. On a unique time-range index, two documents conflict when they fall in the same window and agree on the index's other properties. That is only meaningful when each document is in one window and cannot move, so uniqueness needs non-overlapping windows over $createdAt.
Expiry (ttl). An entry can be queried until ttl seconds after its window's start; a query for an expired window is refused, so every window a query can reach is complete. The expired entries are removed lazily: each later write into the index removes some of the oldest expired entries, within a bounded budget, at no charge to the writer. An index that stops receiving writes keeps its expired entries.
Everything written under an index with a ttl is billed as processing, at an ephemeral-bytes rate, instead of as storage, and nothing is refunded when it is removed. The rate prices a lifetime of at most a week, which is why ttl is capped there.
A ttl removes entries from this index only. The documents stay, and so do their entries in the type's other indexes. To delete the documents themselves after a time, use the document type's ttl.
On an index-only type, a window with a ttl may also declare outlivesDelete: a delete of a document then leaves its entries in the window to expire, so the delete needs no $createdAt, and the document keeps counting there until the window moves past it.
Rules at registration
onnames$createdAt,$updatedAtor$transferredAt, which is the index's first property and is listed inrequired. A user property cannot be bucketed by time; an integer one can be bucketed by value withintegerRange.rangeandstepare at least 1,rangeis an exact multiple ofstep, andrange / stepis at most 24.phaseis less thanstepand less than 31536000.ttl, when present, is at leastrangeand at most 604800. Two indexes that bucket the same timestamp with the samerange,stepandphaseshare their storage and must declare the samettl, or none.- A unique time-range index has
rangeequal tostepandonequal to$createdAt. - The index is not contested, does not set
nullSearchable: false, and is notpreallocated. - A ranking sits below the bucketed timestamp: a single-property time-range index cannot be ranked, and
rankedCountable.atcannot name the timestamp. See Ranked Indexes. - A
refersTofindBycannot resolve through a time-range index. See findBy. - On an index-only type, only
$createdAtcan be bucketed, and a bucketed index cannot serve as the type's proof index.
A broken rule is refused as InvalidContractStructure (10231), or by the meta-schema as JsonSchemaError (10101). Before protocol version 14 the keyword is unknown and refused.
See also
- Time-Range Index TTL for how expired windows are drained, the fee rate and the query gate.
- Integer-Range Indexes for the same windows over an integer property.
- Counts, Sums and Averages and Ranked Indexes for totals and leaderboards per window.
- Time To Live (ttl) for expiring whole documents.
- Indexes for the index keywords every index uses.
Integer-Range Indexes
integerRange groups an index's documents into windows of an integer property, so that "listings priced 500 to 799 by category" or "players per score band" is a question about one window rather than a scan over values. Each window is range units long and a new one starts every step units; when windows overlap, a document belongs to each window that contains its value. Combined with the count and ranking keywords, an integer-range index serves counts, sums and leaderboards per window, with proofs. It is the integer counterpart of timeRange, and costs one set of index entries per window a document falls in.
| Where | index; buckets the index's first property |
| Value | object: on, range, step (required), phase |
| Default | absent: the property is indexed as it is |
| Since | protocol version 14 |
| On update | Fixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217) |
| Errors | DuplicateUniqueIndexError (40105) on a unique integer-range index |
Example
"listing": {
"type": "object",
"indices": [
{
"name": "byPriceBand",
"properties": [{ "price": "asc" }, { "category": "asc" }],
"integerRange": { "on": "price", "range": 300, "step": 100 },
"countable": "countable",
"rangeCountable": true
},
{
"name": "oneBidPerBand",
"properties": [{ "price": "asc" }, { "$ownerId": "asc" }],
"unique": true,
"integerRange": { "on": "price", "range": 1000, "step": 1000 }
}
],
"properties": {
"price": { "type": "integer", "minimum": 0, "maximum": 1000000, "position": 0 },
"category": { "type": "string", "minLength": 1, "maxLength": 63, "position": 1 }
},
"required": ["price", "category"],
"additionalProperties": false
}
byPriceBand keeps windows 300 wide starting every 100, so a listing priced 750 is counted in the windows starting at 500, 600 and 700, and the listings per category between 600 and 899 are one query. oneBidPerBand has non-overlapping windows of 1000 and is unique, so an identity may hold one listing per thousand: a second listing priced 1200 next to one priced 1900 is refused with DuplicateUniqueIndexError (40105).
The keys
| Key | Value | What it does |
|---|---|---|
on | property name, required | The integer property to bucket. It must be the index's first property, an integer property of the type of at most 64 bits, and listed in the type's required. |
range | integer, at least 1, required | The length of each window, in the property's own units. An exact multiple of step. |
step | integer, at least 1, required | The distance between the starts of two windows. |
phase | integer, default 0 | Moves the window boundaries: windows start at phase + k * step for every integer k, negative ones included. Less than step. Windows of 100 cut at 50, 150, 250 take "phase": 50. |
How it works
Windows. For each document, the index stores the start of every window that contains the document's value, in place of the value itself, encoded exactly like a value of the property. With range equal to step the windows do not overlap and a value is in exactly one; otherwise it is in range / step of them, the index's overlap factor, and the document costs that many sets of entries. The overlap factor is at most 24 at protocol version 14, the same cap as a time-range grid's.
The bottom window. The schema's minimum and maximum decide how many bytes the property takes and whether it can be negative. A property with a non-negative minimum, or with only a maximum, is unsigned and its lowest possible value is 0. Otherwise it is signed, and its lowest possible value is the smallest the chosen width holds: -128 for "minimum": -100, "maximum": 100, and -9223372036854775808 for an integer with no bounds. A window that would start below the lowest value the property's type can hold starts at that value instead. So on an unsigned property with "phase": 50, the values 0 to 49 are in a window that starts at 0, and with overlapping windows every window that would start below the lowest value merges into the one that starts at it. Every value the property can hold is in at least one window. When the lowest value is itself a window start, as it is on an unsigned property with no phase, those windows add nothing and simply do not exist.
A property that changes value moves the document into the windows of its new value. Several indexes may bucket the same property with different grids; each grid is stored apart from the others.
The bucketed property is required, so it is never a skipIfAbsent property, but the index's other properties can be: a document missing one of them writes no entry in any window, and an update that adds or drops it moves the document into or out of all of its windows.
Queries. A query selects one window of the index with an IN_INTEGER_RANGE clause on the bucketed property, naming the window by its start. The start must be a window start of the grid, or the lowest value of the property's type for a clamped bottom window; any other value is refused rather than rounded. The server and the proof verifier both read the window from the query itself, so no clock is involved. When the property is bucketed by several grids, the query names the grid. A query may select at most one window, of either kind. A window with no documents is a proved empty answer. Counts, sums and rankings declared on the index are then per window: a ranking below the bucketed property is one leaderboard per window.
A plain clause on the bucketed property, such as price == 700, never reads an integer-range index: the index holds window starts, not prices. Such a query needs an index that is not bucketed.
In the JavaScript SDK the selection is an integerRange entry of the query: { field: "price", start: 600 }, with grid: { range, step, phase } when the property has several grids. A start beyond Number.MAX_SAFE_INTEGER is given as a decimal string. In the Rust SDK it is DocumentQuery::with_integer_range("price", 600u64).
Uniqueness. On a unique integer-range index, two documents conflict when their values fall in the same window and they agree on the index's other properties. That is only meaningful when each value is in one window, so uniqueness needs non-overlapping windows. A document may change its value within its own window; moving into a window where another document holds the same other properties is refused.
Rules at registration
onnames an integer property of the document type of at most 64 bits, which is the index's first property and is listed inrequired. A system property cannot be bucketed by value; for timestamps usetimeRange.rangeandstepare at least 1,rangeis an exact multiple ofstep, andrange / stepis at most 24.phaseis less thanstep.- A unique integer-range index has
rangeequal tostep. - The index is not contested, does not set
nullSearchable: false, is notpreallocated, and does not also declaretimeRange. - A ranking sits below the bucketed property: a single-property integer-range index cannot be ranked, and
rankedCountable.atcannot name the bucketed property. See Ranked Indexes. onis not inside an object that is left out ofrequired: a document could otherwise omit the object and fall in no window.- The grid-qualified level key (
on#range#step, with#phasewhen non-zero) is at most 255 bytes. - A
refersTofindBycannot resolve through an integer-range index, and apropertyConstraintscount or sum does not read one. See findBy. - An index-only type cannot declare
integerRange: its entries are keyed by the index's values and terminal, so two rows that differ only in the bucketed integer would claim the same entry in every window they share.
A broken rule is refused as InvalidContractStructure (10231), or by the meta-schema as JsonSchemaError (10101). Before protocol version 14 the keyword is unknown and refused.
See also
- Time-Range Indexes for windows over timestamps.
- Counts, Sums and Averages and Ranked Indexes for totals and leaderboards per window.
- Indexes for the index keywords every index uses.
Index-Only Types
Some documents are nothing but a position: a like says which post, which hashtag and which identity, and nothing else. Stored as an ordinary document, a like pays for a serialized body, a row in the primary tree and a reference in every index, for a fact its index entries already hold. An index-only type stores no body and no row: its index entries are its documents. That cuts the storage of a small document by more than half, and makes each index a uniqueness rule. In exchange, its documents can only be created and deleted, every property must live in an index or in the entry's value, and a query returns documents rebuilt from index entries rather than fetched by $id.
Seven keywords shape an index-only type: indexOnly and entryPayload on the document type, and terminal, preallocated, summableOffCountIndex, skipIfAbsent and outlivesDelete on its indexes. All of them arrived at protocol version 14 and are fixed once the type exists.
Example
A like of a social contract whose post type cannot be deleted:
"like": {
"type": "object",
"indexOnly": true,
"documentsMutable": false,
"canBeDeleted": true,
"indices": [
{
"name": "byHashtagPost",
"properties": [{ "hashtag": "asc" }, { "postId": "asc" }],
"countable": "countable",
"rangeCountable": true,
"rankedCountable": true,
"skipIfAbsent": true
},
{
"name": "byPost",
"properties": [{ "postId": "asc" }],
"countable": "countable",
"rangeCountable": true,
"rankedCountable": true
},
{ "name": "byLiker", "properties": [{ "$ownerId": "asc" }], "terminal": "postId" }
],
"properties": {
"hashtag": { "type": "string", "minLength": 1, "maxLength": 63, "position": 0 },
"postId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": {
"type": "permanentDocument",
"documentType": "post",
"where": { "hashtag": "hashtag" }
},
"position": 1
}
},
"required": ["postId"],
"additionalProperties": false
}
byPost holds one entry per post and liker (its terminal defaults to $ownerId), so an identity can like a post once, and it counts and ranks posts by likes. byHashtagPost ranks the posts under each hashtag, and only likes that carry a hashtag enter it. byLiker lists the posts one identity liked. The reference makes sure the post exists and that a like's hashtag is its post's.
indexOnly
| Where | document type |
| Value | boolean |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed (DocumentTypeUpdateError, 40212) |
| Errors | DuplicateUniqueIndexError (40105), DocumentNotFoundError (40101), InvalidDocumentTransitionActionError (10404) |
true stores the type's documents only as index entries. Each entry sits under the index's values, is keyed by the terminal's values in place of the document id, and holds a 32-byte row commitment: a hash over all of the document's values that ties its entries in the different indexes together as one document.
Create. A create writes one entry into each index. If any of those entries already exists the create is refused with DuplicateUniqueIndexError (40105), so every index is a uniqueness rule over its properties and its terminal. refersTo and the other property checks run as on any document.
Delete. A document has no id to delete by. It is deleted with an indexOnlyDelete transition that carries all of its values (and $createdAt when the type requires it); every entry those values produce must exist and carry the matching row commitment, or the delete is refused with DocumentNotFoundError (40101). Deleting by id on an index-only type, or with indexOnlyDelete on an ordinary type, is refused with InvalidDocumentTransitionActionError (10404). The owner can only ever reach its own entries, since every index that keeps entries holds $ownerId; a summableOffCountIndex counter moves only with the entry the delete removes from its source. canBeDeleted: false forbids deletes as on any type.
No other action. A document cannot be replaced, transferred, sold or repriced.
Queries. A query goes through one index and returns documents rebuilt from its entries: the index's properties, the terminal, and $ownerId and $createdAt where the index holds them. A query through an index that holds only some of the properties yields only those with a proof; without one it is refused (Unsupported), since the documents could not be serialized, and so is a chained or composite read of them. The rebuilt $id is a hash of the entry's position and addresses nothing, so there is no fetch by $id and no startAt cursor; a query pages by the terminal instead (postId > <last seen>, with a limit). A query that sets the terminal with an equality can put an in on the index's last property, with a limit of at least its number of values; a range on an index property in such a query is refused, because its pages could hold fewer rows than exist. List the equality-bound properties first in an index, and page by a range on its terminal instead. The proof that a create or delete took effect is the presence or absence of its entry in the proof index, an index that involves no $createdAt, does not skip and keeps entries.
Rules at registration:
documentsMutable: false;transferable0;tradeMode0; nodocumentsKeepHistoryorkeeps*History; notransientproperties.- None of the document type aggregate keywords (
documentsCountableand the others): the type has no primary tree. The index keywords of Counts, Sums and Averages and Ranked Indexes are allowed. - At least one index. No index is
uniqueorcontested, and none setsnullSearchable: false. - Every index holds
$ownerId, as a property or in its terminal, except asummableOffCountIndexindex, which keeps counters instead of entries. - The only system properties an index may list are
$ownerIdand$createdAt, and an indexed$createdAtmust be inrequired. - No index declares
integerRange: rows that differ only in the bucketed integer would claim the same entry in the windows they share. - At least one index involves no
$createdAt, does not setskipIfAbsentand keeps entries (it is nosummableOffCountIndexindex): the proof index. - Every property is in
required, except a skip property of askipIfAbsentindex. An object holding an indexed property is required too. - Every required property appears in at least one index that does not skip and keeps entries, as a property or a terminal component, except the
entryPayloadproperties and a property asummableOffCountIndexindex's source fixes. Every optional property appears in a skip index without atimeRangewhose skip set is that property alone, except, again, one such a source fixes. - The type cannot also set
ttlormoderatorAbilities.delete, and no document reference can target it: arefersTofindByfinds no unique index in it, and a reference by id (or withinList) is refused when the referring contract is registered (ReferencedDocumentTypeIndexOnlyError, 40146), since its documents can not be fetched by$id.
A property a summableOffCountIndex index's source fixes may sit in no index that keeps entries: in the summableOffCountIndex example, a like's postAuthor, which the postId reference's where entry "$ownerId": "postAuthor" fixes to the post's owner, sits only in byAuthorPost. No query returns such a property: a proved read gives documents without it, required or not, and since no index of the type then holds every property, every documents read of the type without a proof is refused. Its value is the referenced document's (here the post's $ownerId), so a client building a delete reads it back from there.
entryPayload
| Where | document type |
| Value | array of 1 to 16 distinct property names, each 1 to 64 characters |
| Default | absent |
| Since | protocol version 14 |
| On update | Fixed (40212). The list is read as a set, so reordering it is no change. |
The properties stored in each entry's value, after the row commitment, instead of in a key. They are for data the application reads but never queries by, such as a public key or a ciphertext: they need not be indexed, and they come back with every query result.
"indices": [{ "name": "byRequest", "terminal": ["appEphemeralPubKeyHash", "$ownerId"] }],
"entryPayload": ["walletEphemeralPubKey", "encryptedPayload"]
With a flat index keyed by a request hash and the responder, this is a key-value table: a query on the hash returns every responder with its public key and ciphertext.
Rules at registration:
- Only on an
indexOnlytype. - Each entry names a top-level property that is
required, a scalar (not an object or an array of values), and bounded:maxLengthon a string,maxItemson a byte array. - A payload property appears in no index, neither as a property nor in a terminal.
- The largest size each payload property can take, plus two bytes each, adds up to at most 5120 bytes. With several indexes, the payload is repeated in every entry.
terminal
| Where | index of an indexOnly type |
| Value | a property name, or an array of 1 to 10 distinct names |
| Default | "$ownerId" |
| Since | protocol version 14 |
| On update | Fixed, like every index (DataContractInvalidIndexDefinitionUpdateError, 10217) |
Where an ordinary index keys each entry by the document id, an index-only index keys it by the terminal's values: the member key. There is one entry per index values and member key, so the terminal decides what the index makes unique. byPost above, with the default terminal, allows one like per post and owner; byLiker, with postId as terminal under $ownerId, holds the same pairs the other way round.
An array is a composite terminal whose values are joined in order. An index with no properties at all is a flat index, keyed by its terminal alone, as byRequest above is.
A query that fixes every property of the index can test one member key ("did I like this post") or walk the member keys in order, a page at a time.
Rules at registration:
- Only on an
indexOnlytype. - Each component is
$ownerIdor a property of the type that could be indexed: not an object or an array of values, a string withmaxLengthof at most 63, a byte array withmaxItemsof at most 255. No other system property. - Every component but the last has a fixed width: a byte array with
minItemsequal tomaxItems, an identifier, an integer or a boolean. A string can only be last. - The whole member key is at most 255 bytes. On a flat index, the level key, the component names each preceded by a zero byte, is at most 255 bytes as well.
- A component is not one of the index's
properties, and not an optional property. - A flat index takes no count, sum, ranking,
timeRange,integerRange,skipIfAbsentorpreallocatedkeyword.
preallocated
| Where | index of an indexOnly type |
| Value | boolean |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed (10217) |
The first entry under a new set of values pays for every tree on its path; later ones pay for one entry. When the whole path is decided by a referenced document, preallocated: true creates the trees when that document is created, paid by its creator, so every entry costs the same from the first one on. Deleting the last entry keeps the trees, so a post with no likes still shows in the rankings with a count of zero, and a range count, sum or average grouped by the last property returns it with zero, counted in a page's limit, so without an IN a short page means the range ended (across an IN, a value whose range holds nothing takes a place in the limit too, so there a short page may not be the end). Only documents created once the index exists get the trees: when a contract update adds the referring type, the referenced documents already stored get theirs from their first entry, as without preallocated.
In the example, byHashtagPost could be preallocated: postId is the reference and hashtag agrees with the post's. byLiker could not, since no post decides who likes it.
Rules at registration:
- Only on an
indexOnlytype. - Every index property is either a property with a
permanentDocumentormoderatedDocumentreference to a type of the same contract, or a referring value of that reference'swhere. AdeletableDocumentreference does not qualify, since the trees would outlive a deleted target with nothing left to say what they were keyed by.$ownerIdmay only be the terminal. - Through a
moderatedDocumentreference, every key of the path must be kept by the referenced document's removal record: eachwhereentry the index uses compares the referenced$id,$ownerIdor a property the referenced type lists undermoderatorAbilities.deleteKeepsFields(or one inside an object listed there), never$creatorId(InvalidContractStructure, 10231). The trees then outlive a removed document the way its record does, and a moderator's restore finds them in place. As through apermanentDocumentreference, they are keyed by the values the document was created with: a value changed afterwards leaves them empty, and the first entry under the new value builds its own. - The referenced property of each such
whereentry holds at most 255 bytes, since creating a referenced document makes its value an index key (40126 when the contract is created or updated). - Not with
timeRangeorintegerRange.
A referenced document whose agreed value takes more bytes than the referring property can hold preallocates nothing for that index, since no entry could agree with it.
summableOffCountIndex
| Where | index of an indexOnly type |
| Value | the name of another index of the type, its source |
| Default | none |
| Since | protocol version 14 |
| On update | Fixed (10217) |
A like counted by post, by author and by hashtag is written three times: once in byPost and once in each of the other two, which hold nothing byPost does not already hold. When every like of a post lands in the same author and the same hashtag, the other two only need to know how many likes each post has. An index with summableOffCountIndex keeps exactly that: one counter per group, holding the number of entries its source index keeps for it, in place of an entry per document. A like then adds one to two counters instead of writing two more entries.
Here the like of the example also carries its post's author, fixed through the reference's where like its hashtag:
"properties": {
"postId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": {
"type": "permanentDocument",
"documentType": "post",
"where": { "hashtag": "hashtag", "$ownerId": "postAuthor" }
},
"position": 0
},
"hashtag": { "type": "string", "minLength": 1, "maxLength": 59, "position": 1 },
"postAuthor": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"position": 2
}
},
"required": ["postId", "postAuthor"]
and counts its likes per author and post:
{
"name": "byAuthorPost",
"properties": [{ "postAuthor": "asc" }, { "postId": "asc" }],
"summableOffCountIndex": "byPost",
"rangeCountable": true,
"rangeSummable": true,
"rankedSummable": { "at": ["postAuthor", "postId"] },
"rankedAverageable": { "at": ["postAuthor"] },
"preallocated": true
}
Each counter adds its entries to the sums of the tree of the last property and, with rangeCountable, counts one group in its counts. A count query counts documents, so on this index it reads the sums, which hold the source's entries:
| Query | Reads | byAuthorPost |
|---|---|---|
count(*) | the sums | an author's likes, or one post's |
sum(byPost), named by the source index | the sums | the same |
avg(byPost) | the sums and the group counts | an author's posts and likes, so likes per post |
A ranking at an earlier level orders by these totals: rankedSummable: { "at": "postAuthor" } ranks authors by likes, rankedAverageable: { "at": "postAuthor" } by likes per post, and "postId", the last property, ranks an author's posts by likes. A count or sum point query may stop at any level from the shallowest sum or average ranking down, and an average point query at any level that also carries counts (from the shallowest average ranking down). Pinned on every property but the last, any of them reads the tree of the last property instead when that property is not ranked: postAuthor == A alone gives A's likes, and with rangeCountable A's posts for an average. A ranked or HAVING count reads the sum rankings the same way: count(*) grouped by postAuthor ranks authors by likes. So on this index rankedCountable declares the same ranking as rankedSummable, and needs no rangeCountable: rankedCountable: { "at": "postAuthor" } ranks authors by likes too. A range count reads the sums over the range too: count(*) with postAuthor == A and a range on postId, grouped by postId, gives each of A's posts in the range with its likes (on a preallocated index a post nobody liked comes back with zero, and takes its place in a page's limit), and across an IN on postAuthor each author's. A range total (no grouping, or one per IN value) needs an index whose path passes through no ranked level, as everywhere: grovedb proves a range total only through unranked trees, and the unproven read refuses one too so both agree, so on this index, which ranks postAuthor and postId, group by postId instead. When the index is preallocated, every post created once the index exists has a counter from its creation, so a post without likes counts as a post with zero likes (a post stored before a contract update added the like type gets its counter from its first like, and keeps it at zero after); otherwise a post shows once it is liked and leaves with its last like.
Rules at registration:
- Only on an
indexOnlytype, withrangeSummable: true(the counters sit in the tree of the last property, which onlyrangeSummablemakes a sum tree). Nosummable,averageable,terminal,countable: "countableAllowingOffset",timeRange,integerRange,outlivesDelete,uniqueorcontested. - The source is another index of the type holding every document exactly once: it keeps entries (it is no
summableOffCountIndexindex itself), skips no document (skipIfAbsent), keeps no deleted one (outlivesDelete) and involves no$createdAt. - One summed value per type: no index of the type declares
summable, everysummableOffCountIndexindex names the same source, and no property shares the source's name. - Every property of the source is a property of the index, so a group never counts two source groups.
- Every other property is a referring value of a
whereon apermanentDocumentormoderatedDocumentreference by id (nofindByorinList, whose key could move to another document) to a type of the same contract, held by a property of the source, and the referenced value never changes once written:$id,$creatorId, an$ownerIdno transfer or trade changes, or a property the referenced type never lets change (awherenames only the referenced type's properties,$id,$creatorIdand$ownerId). An immutabledeletableDocumentreference counts only when it is required: a replace may clear an optional one once its document is deleted. Through amoderatedDocumentreference, the value must also stay on the removal record (deleteKeepsFields). Every like of one post then lands in one group. - No other index continues below the index's last property, where the counter stands.
rankedSummableandrankedAverageabletake the{ "at": ... }form only on such an index.
A broken rule is refused as InvalidContractStructure (10231), or by the meta-schema as JsonSchemaError (10101): the meta-schema refuses the keyword without rangeSummable: true or next to summable, averageable or terminal, the { "at": ... } form of rankedSummable or rankedAverageable without it, and a source name longer than 32 characters.
skipIfAbsent
| Where | index |
| Value | true, or an array of the index's property names |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed (10217) |
The keyword is described in Indexes: a document that leaves out a property of the index's skip set writes nothing into the index, and its delete looks for nothing there. It is the one way a property of an index-only type can be optional: in the example, a like without a hashtag is not in byHashtagPost and pays nothing for it. A skip property may sit below the first position, which is what lets a windowed ranking skip it:
{
"name": "byDayHashtagPost",
"properties": [{ "$createdAt": "asc" }, { "hashtag": "asc" }, { "postId": "asc" }],
"terminal": "$ownerId",
"rangeCountable": true,
"rankedCountable": { "at": ["hashtag", "postId"] },
"timeRange": { "on": "$createdAt", "range": 86400, "step": 86400, "ttl": 604800 },
"skipIfAbsent": true
}
An untagged like still enters the type's other indexes over the same day window, but writes nothing under hashtag in it.
What an index-only type adds to the rules of every type:
- An index path has no representation for a missing value, so every index holding an optional property skips on it: the skip set is every optional property of the index, and an array must name them all.
- Each optional property needs a skip index without a
timeRangewhose skip set is that property alone. A document carrying one optional property but missing another skips every index holding both, and its value would otherwise be written nowhere; a windowed index keeps it only until its windows drain, where document queries do not read it. - An optional property is never a terminal.
- At least one index that involves no
$createdAtdoes not skip: the proof index.
outlivesDelete
| Where | timeRange index of an indexOnly type |
| Value | boolean |
| Default | false |
| Since | protocol version 14 |
| On update | Fixed (10217) |
A delete of an index-only document carries its values and removes the entries they address. A time window is keyed by the document's $createdAt, so without this keyword a delete must carry the exact timestamp to find the window's entries. With outlivesDelete: true, a delete leaves the index's entries where they are, and they expire with their window:
{
"name": "byTrendPost",
"properties": [{ "$createdAt": "asc" }, { "postId": "asc" }],
"terminal": "$ownerId",
"countable": "countable",
"timeRange": { "on": "$createdAt", "range": 259200, "step": 86400, "ttl": 604800 },
"outlivesDelete": true
}
- A delete carries no
$createdAtwhen every index involving it outlives deletes: the rows commit to no timestamp, and a delete that carries one is refused (InvalidDocumentTransitionActionError). An unlike needs only the like's other values, which any device knows. - A create writes over an entry already there. When the same owner writes the same values again while a deleted document's entry still stands in a window, the create writes its own entry over it: the owner counts once, and the create is not refused as a duplicate. Only the index's other entries decide whether a create is a duplicate.
- A deleted document keeps counting in the window until the window moves past it. On a trending window, an unliked like still counts there for up to the window's
range. - The executed-transition proof runs against an index that does not outlive deletes, so a delete still proves the document gone.
Rules at registration:
- Only on an
indexOnlytype, on an index with atimeRangecarrying attl, so the entries a delete leaves expire. - Not with a sum (
summable), and not on a type withentryPayload: a kept entry holds the amount or payload of the document that wrote it. - Every schema property must also sit in an index that neither skips nor outlives deletes, which a delete checks.
- The index's key (its properties but
$createdAt, and its terminal) must hold the whole key of an index a delete clears and that skips nothing, so no two documents in state share one of its entries.[$createdAt, postId] → $ownerIdholdsbyPost's[postId] → $ownerId;[$createdAt] → $ownerIdholds no such key and is refused. - The proof index may not outlive deletes.
See also
- Index-Only Document Types for the entry layout, the row commitment, the full constraint list and the query surface.
- Time Range for the windows an
outlivesDeleteindex needs. - Indexes, Counts, Sums and Averages and Ranked Indexes for the index keywords an index-only type uses.
- References (refersTo) for
permanentDocumentandmoderatedDocumentreferences andwhere, whichpreallocatedrelies on. - Mutability, Deletion and Creation, Transfers and Trading for the flags an index-only type must set.
Values of Referenced Documents
An index may hold a value that the document does not store, read from the document one of its references points at. A reply indexed by postId.$ownerId is filed under the owner of the post it replies to, so "every reply to my posts" is one query, and the reply never stores that owner. Drive reads the value from the post whenever it writes or removes the reply's index entries. This saves storing a copy of the value in every document (32 bytes for an identifier). In exchange, every write that moves the entries reads the referenced document again.
Example
A reply to a post only the moderators take down:
"post": {
"type": "object",
"canBeDeleted": false,
"moderatorAbilities": { "delete": true },
"properties": {
"text": { "type": "string", "minLength": 1, "maxLength": 280, "position": 0 }
},
"required": ["text"],
"additionalProperties": false
},
"reply": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": true,
"immutable": ["postId"],
"indices": [
{ "name": "toPostOwner", "properties": [{ "postId.$ownerId": "asc" }, { "$createdAt": "asc" }] }
],
"properties": {
"postId": {
"type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"contentMediaType": "application/x.dash.dpp.identifier",
"refersTo": { "type": "moderatedDocument", "documentType": "post" },
"position": 0
},
"body": { "type": "string", "minLength": 1, "maxLength": 280, "position": 1 }
},
"required": ["postId", "body", "$createdAt"],
"additionalProperties": false
}
A reply is created with its own properties only, postId and body. Its author may edit body but never repoint postId. A query for the replies to one identity's posts, newest first:
{
"where": [["postId.$ownerId", "==", "<identity id>"]],
"orderBy": [["$createdAt", "desc"]]
}
The replies come back as stored, without the owner: the proof ties them to it through the index path.
A post whose removal record keeps a field can have its replies filed under that field too. With "documentsMutable": false and "moderatorAbilities": { "delete": true, "deleteKeepsFields": ["hashtag"] } on the post, a reply index on postId.hashtag files every reply under the hashtag of the post it replies to, before and after a moderator removes the post.
The index property
| Where | a property of an index of a document type that is not indexOnly |
| Value | "<reference property>.<field>" |
| Since | protocol version 14 |
| On update | Fixed, as every index is (10217) |
| Errors | InvalidContractStructure (10231); for a field an index can not key, InvalidIndexPropertyTypeError (10206) or InvalidIndexedPropertyConstraintError (10205) |
The name reads through a reference when its first segment names a top-level identifier property that declares a refersTo. An identifier has no nested properties, so the name can not mean a nested property. The field is what follows the first .:
$ownerId: the owner of the referenced document.$creatorId: the identity that created the referenced document, on a type that records it (see System Properties).- A schema property of the referenced document type, by its dotted path, with that property's type.
What Drive does
- Create. The reference is validated as usual, which reads the referenced document, and Drive writes the entries under the values it holds, without reading it again. A referenced document that has no value for the field, or a reference left out, puts the document under null, as a missing property would, or leaves it out of an index that skips on the property.
- Replace, transfer, purchase, price update and a moderator's change of fields. Drive reads the referenced document once, for both versions of the document, since neither can point elsewhere. It reads it even when no index holding a derived value moves: every entry's reference is rewritten in place on an update, under the values it is filed by.
- Delete, a moderator's removal, and expiry by
ttl. Drive reads the referenced document to find the entries to remove. - A removed referenced document. Once a moderator removes a
moderatedDocumenttarget, Drive reads its owner, and any field its type keeps underdeleteKeepsFields, from the removal record, which keeps them as the document held them. So the entries are found under the values they were written under. A restored document is read again.
Each read is billed as a processing fee with the write that makes it. A dry run prices the reads at their worst case and reads nothing, pricing the entries under a value of the field's type and typical size.
Rules at registration
A derived value must stay what it was when an entry was written, or Drive could not find the entry again. So:
- The reference is a
permanentDocumentor amoderatedDocumentone, by id, to a document type of the same contract. AdeletableDocumenttarget could leave state without a record, and afindBy, aninListor an operand of an expression does not name the document by id. - The reference property is fixed once written: the type's documents are immutable, or the property is listed under
immutablewithout a condition, and it is not one of the fields only moderators write. - Through a
moderatedDocumentreference, the field is$ownerId, or a schema property the referenced type lists undermoderatorAbilities.deleteKeepsFields, or one inside an object listed there: the removal record keeps these, and Drive reads them from it once the document is removed. Not$creatorId, which no record keeps. The list is fixed when the referenced type is registered, so an update adding a referring type can read only what it already lists. $ownerIdonly of a type whose documents can not be transferred or traded, and$creatorIdonly of a type that records it.- A schema property must exist on the referenced type, be stored (not
transientor inside a transient object), be fixed once written there under the same rule as the reference property, and be one an index can key: not an object or an array, a string of at most 63 characters, a byte array of at most 255 bytes, tighter on a ranked index. - Not
$idof the referenced document, which is the reference property itself: index that instead. Nor any other system property. - The index is not
uniqueor contested: a uniqueness check reads the values the create carries, and a derived value is not among them. - The derived property is not the source of a
timeRangeor anintegerRange. - The type is not
indexOnly: its entries hold every value of its documents, and its delete carries them.
Skipping a document without the value
A derived property can be a skip property of skipIfAbsent. The index then leaves out a document whose reference is absent, or whose referenced document has no value for the field, instead of filing it under null. A reply that may quote a post, filed by the owner of the post it quotes:
{
"name": "byQuotedOwner",
"properties": [{ "quoteId.$ownerId": "asc" }, { "$createdAt": "asc" }],
"skipIfAbsent": ["quoteId.$ownerId"]
}
A reply quoting no post writes nothing into this index, and its delete looks for nothing there. Since neither the reference nor the field can change once written, a replace never moves a document into or out of such an index through its derived values.
Only the array form names a derived property. skipIfAbsent: true skips on the type's own optional properties: whether a derived value can be absent depends on the referenced type.
At registration, a derived skip property must be able to be absent: its reference property is not in required, or the field is a schema property the referenced type does not require, or one inside an object it does not require. $ownerId and $creatorId are absent only with the reference. A byte array the property reads must set minItems to at least 1 on the referenced type, since an empty one is keyed like a missing value. Every other skipIfAbsent rule applies as to any skip property.
Queries
A query uses a derived property as any other index property: ==, in, a range, orderBy. Its values encode as the field's type.
A startAt or startAfter cursor places the page by the values the document it names stores, and a derived value is not among them. A query paging with a cursor must therefore fix every derived property of its index with ==. Otherwise it is refused, and it can page by a range on another property of the index instead.
A subscription to state transitions filters by the values a transition carries, and a derived value is not among them either, so a subscription filter can not name a derived property. Filter by the reference property instead (postId), or query the index.
See also
- Indexes (indices) for the index keywords every index takes.
- References (refersTo) for
permanentDocumentandmoderatedDocument. - Mutability for
immutable, and Moderator Abilities for removal records.
Contract-Level Keys and config
The document types sit inside a data contract, which has keys of its own: its id, its owner, its version, the shared definitions its types may point at, its tokens and groups, and a search description. One of them, config, holds contract-wide settings: whether the contract can ever change, whether it keeps its history, the defaults its document types inherit, how integers are stored, and whether and how it is moderated. Almost every config setting is chosen once, at registration, and kept for the contract's life.
Example
{
"$formatVersion": "1",
"id": "AY6xWncZUFv2GCrS5seqKthUfbW9yYyUXtF8diSuHQ4g",
"ownerId": "AtirhSVpAWF7dEt6dLAmesC4Sr1MsJ9bFC1nLAoNnq2S",
"version": 1,
"config": {
"$formatVersion": "2",
"canBeDeleted": false,
"readonly": false,
"keepsHistory": false,
"documentsKeepHistoryContractDefault": false,
"documentsMutableContractDefault": true,
"documentsCanBeDeletedContractDefault": true,
"sizedIntegerTypes": true,
"moderation": {
"banlist": true,
"suspensions": true,
"moderators": { "$type": "contractOwner" }
}
},
"documentSchemas": {
"post": {
"type": "object",
"properties": {
"text": { "type": "string", "minLength": 1, "maxLength": 280, "position": 0 }
},
"required": ["text"],
"additionalProperties": false
}
},
"keywords": ["social", "microblog"],
"description": "Short public posts"
}
A first version of a small social contract. Its config states the defaults explicitly, stores integers in the smallest width their bounds allow, and keeps a banlist and a suspension list that the owner edits. keywords and description make it findable through the keyword search contract.
Contract keys
| Key | Value | What it is | Since |
|---|---|---|---|
$formatVersion | "0" or "1" | The contract's serialization format. Format 1 is the default from protocol version 9 and is the one that carries groups, tokens, keywords, description and the timestamps. | 1 |
id | identifier | The contract's id: a hash of ownerId and the identity nonce of the create transition. A create whose id is not that hash is refused (InvalidDataContractIdError, 10204). | 1 |
ownerId | identifier | The identity that registers the contract, and the only one that can update it. | 1 |
version | integer | 1 when the contract is created; each update must raise it by exactly one (InvalidDataContractVersionError, 10212). | 1 |
config | object | Contract-wide settings: see config below. Absent means the defaults. | 1 |
documentSchemas | object | The document types, by name. See documentSchemas. | 1 |
schemaDefs | object | Definitions every document type may point at with $ref. An update may add definitions, not remove them (IncompatibleDataContractSchemaError, 10213). | 1 |
groups | object | Groups of identities that act together, each member with a voting power, whose approval some token actions need. See Data Contracts. Not the same thing as a contract group, a set of contracts. | 9 |
tokens | object | The contract's tokens, keyed by position 0, 1, and so on. Document types may charge them with tokenCost. | 9 |
keywords | array of strings | Search keywords. See keywords and description. | 9 |
description | string | A short description for search. See keywords and description. | 9 |
createdAt, updatedAt, createdAtBlockHeight, updatedAtBlockHeight, createdAtEpoch, updatedAtEpoch | numbers | When the contract was created and last updated. The platform sets them; a contract does not write them. | 9 |
documentSchemas
| Where | contract |
| Value | object mapping each document type name to its schema |
| Since | protocol version 1 |
| On update | Document types may be added; none may be removed (DocumentTypeUpdateError, 40212). Each existing type follows the update rules of its keywords. |
| Errors | DocumentTypesAreMissingError (10214), InvalidDocumentTypeNameError (10415), at registration |
A contract has at least one document type, unless it defines tokens (DocumentTypesAreMissingError, 10214). A name is 1 to 64 ASCII letters, digits, _ or -; from protocol version 14 a name may not contain - (InvalidDocumentTypeNameError, 10415). The keywords a schema takes are the subject of the rest of this part: see Contract Keywords.
keywords and description
| Where | contract |
| Value | keywords: array of at most 50 strings; description: string |
| Default | no keywords, no description |
| Since | protocol version 9 |
| On update | May be changed; the search entries are replaced |
| Errors | TooManyKeywordsError (10262), InvalidKeywordLengthError (10270), InvalidKeywordCharacterError (10269), DuplicateKeywordsError (10263), InvalidDescriptionLengthError (10264) |
The keyword search system contract indexes each contract by its keywords and description, so applications can find contracts by topic. The rules, checked on every create and update:
- At most 50 keywords (10262).
- Each keyword is 3 to 50 bytes of UTF-8 (10270) and contains no whitespace or control character (10269).
- No keyword appears twice (10263).
- A description is 3 to 100 bytes of UTF-8 (10264).
config
| Where | contract |
| Value | object: $formatVersion and the keys below |
| Default | absent: every key takes its default |
| Since | protocol version 1 |
| On update | The keys below are fixed, with the exceptions each one names (DataContractConfigUpdateError, 40002) |
| Errors | DataContractConfigUpdateError (40002), DataContractIsReadonlyError (40001) |
$formatVersion is the config's own version: "0" before protocol version 9, "1" from 9, and "2" from 14, which adds moderation. From protocol version 14 every new contract carries config version 2, moderated or not, and an older contract moves to it with its next update.
canBeDeleted
| Where | config |
| Value | boolean |
| Default | false |
| Since | protocol version 1 |
| On update | Fixed (40002) |
Whether the contract itself may ever be deleted. No transition deletes a contract today, so the flag has no effect yet beyond being recorded.
readonly
| Where | config |
| Value | boolean |
| Default | false |
| Since | protocol version 1 |
| On update | Cannot be set by an update (40002) |
| Errors | DataContractIsReadonlyError (40001) |
true freezes the contract at its first version: every update of it is refused with DataContractIsReadonlyError (40001). Only a create can set it, so a contract is read-only from the start or never. A document reference can require its target contract to be read-only (contractRequirements.readonly, see References).
keepsHistory
| Where | config |
| Value | boolean |
| Default | false |
| Since | protocol version 1 |
| On update | Fixed (40002) |
true makes Drive keep every version of the contract, not only the latest, so a client can read and prove the contract as it was at an earlier version.
Document type defaults
| Where | config |
| Value | documentsKeepHistoryContractDefault, documentsMutableContractDefault, documentsCanBeDeletedContractDefault: boolean each |
| Default | false, true, true |
| Since | protocol version 1 |
| On update | Fixed (40002) |
The value of documentsKeepHistory, documentsMutable and canBeDeleted for each document type that does not set it itself. A type that sets the keyword overrides the default. Changing a default would change every type that relies on it, so all three are fixed. See History, Mutability and Deletion.
Bounded key requirements
| Where | config |
| Value | requiresIdentityEncryptionBoundedKey, requiresIdentityDecryptionBoundedKey: 0 unique, 1 multiple, 2 multiple with a pointer to the latest |
| Default | absent |
| Since | protocol version 1 |
| On update | Fixed (40002) |
Let identities add encryption, or decryption, keys bound to the whole contract, and say how they are kept: one key that cannot be replaced, several, or several with a pointer to the latest. The document type keywords of the same names do this for keys bound to one document type. See Signing and Keys and Contract Bounds.
sizedIntegerTypes
| Where | config |
| Value | boolean |
| Default | true in config version 1 and later; config version 0 has no such key and behaves as false |
| Since | protocol version 9 |
| On update | May be turned on, not off (40002) |
With true, each integer property is stored in the smallest width its minimum, maximum or enum allow: "minimum": 0, "maximum": 100 takes one byte. With false, every integer is a signed 8-byte value. Turning it off would make stored documents unreadable, so it is refused. Turning it on is allowed by the config check, but it changes the width of every bounded integer of the existing document types, and an update that changes how an existing property's values are stored is refused (DocumentTypeUpdateError, 40212); in practice it can only be turned on when it leaves the width of every existing integer property unchanged.
The width also decides what a summed property may be: see Counts, Sums and Averages.
moderation
| Where | config (config version 2) |
| Value | object: banlist, suspensions, warnings (booleans, default false) and moderators |
| Default | absent: the contract is not moderated |
| Since | protocol version 14 |
| On update | Which lists are kept is fixed, and an elected team can be neither declared, changed nor left; otherwise the moderators may change (40002) |
| Errors | InvalidContractModerationConfigError (10900), ContractModeratorIdentityNotFoundError (41110) |
Declares which moderation lists the contract keeps and who edits them. An identity on the banlist, or suspended, cannot act on the contract's documents; a warning is a record that bars nothing. moderators is one of:
{ "$type": "contractOwner" }: the owner moderates alone.{ "$type": "appointedModerators", "identities": [...] }: the owner and 1 to 16 named identities, each acting alone. Every named identity must exist (ContractModeratorIdentityNotFoundError, 41110).{ "$type": "elected", ... }: a team elected by masternodes moderates, with the abilities the declaration gives it. See Elected Moderation.
A declaration keeps at least one list, unless a document type gives its moderators an ability (moderatorAbilities: deleting its documents or writing the fields it keeps for them), and is refused otherwise (InvalidContractModerationConfigError, 10900). An unknown key is refused rather than ignored, so a misspelled list name cannot silently leave the contract without it. Because the lists are fixed, a contract that will ever need moderation declares it when it is created.
moderation is what moderatorAbilities and the moderators' share of actionFees require.
See also
- Data Contracts for the contract structure and its versions.
- Contract Moderation for the lists, the moderation transition and elected teams.
- Contract Groups for contract groups, sets of contracts, which a create transition may register or join (not the contract's
groups). - Contract Keywords for the document type keywords.
The GroveDB Structure
Drive keeps everything in one GroveDB: a tree of trees. The root layer holds
one subtree per RootTree variant, each of those holds fixed keys or one
subtree per identity, contract or token, and so on down to the items. The other
chapters of this section draw the part they are about. This chapter is about
the whole, and about the two things that keep a picture of it honest: a
description written as code, and a viewer that reads it.
Dive into a tree to go a layer down, use the breadcrumb or Backspace to come back up, drag the protocol version at the bottom to see what each version added, and switch a layer to Merk tree to see its real binary tree.
The description is code
The structure is declared in Rust, in drive::structure
(packages/rs-drive/src/structure). Each area declares its own part in a
structure.rs beside its paths.rs, from the same constants the paths use:
#![allow(unused)] fn main() { StructureNode::fixed( "balances", &[TOKEN_BALANCES_KEY], "TokenBalances", "TOKEN_BALANCES_KEY", ) .kind(ElementKind::BigSumTree) .since(9) .describe("Who holds how much of each token.") .child( token("The balances of one token; the sum is its circulating supply.") .kind(ElementKind::SumTree) .child( StructureNode::identifier("identity", "identity_id", "The holder's identity id") .kind(ElementKind::SumItem) .value("token amount") .describe("One identity's balance of the token."), ), ) }
A node says:
- its key: fixed bytes and the constant they come from, or a template such as "identity id, 32 bytes" standing for many keys;
- the element kinds that can sit there. Several when the code chooses, as the primary key tree of a document type does between eight tree kinds;
- the element flags on it. GroveDB stores a byte string of flags with
every element and never reads it; Drive keeps its storage flags there: the
epoch the element's bytes were paid for in, the bytes added in later epochs
if it grew, and for owned flags the identity that paid, which is who a
refund goes to when the element is deleted or shrinks. A node says whether
its element carries none, storage flags, or storage flags with an owner, and
who that owner is. The exported file explains each kind and its byte layout
once, under
flag_kinds; since, the first protocol version it exists in, which children inherit;- whether it is created with its parent, lazily on first use, or with its parent and deleted later while the parent stays, as an epoch's storage fee item is at payout;
- what an item holds, what a reference points to, and
recursewhere the structure repeats to any depth, as index levels do; - its source file and, when there is one, its chapter in this book.
The identifier of a node is the dotted path of its segments
(tokens.balances.token.identity). Identifiers are what two versions of the
structure are compared by, so treat them as append-only: renaming one reads as
a removal and an addition.
The module is compiled for tests and under the structure feature only.
Nothing in the node depends on it.
What keeps it true
The tests in packages/rs-drive/src/structure/tests.rs stand between the
description and drift. A lint first checks the description against itself:
identifiers are unique, reference and recurse targets exist, every source
file exists and names the constant a key claims to come from, and no two
templates of a layer could claim the same element.
Conformance. check_conformance walks a real GroveDB layer by layer and
reports every element no node describes, every element of a kind its node
does not list, every element whose flags are of a kind its node does not list, every element outside its node's protocol versions, and every
node that should have been created with its parent but is missing. Where
several templates of a layer accept a key, the one whose description fits what
is below the element wins: below a contested index a 32 byte key is a
contender's identity id at the last level and an index value before it, and
the key alone cannot tell. It runs against the initial state structure of
every protocol version, which pins each since to what
create_initial_state_structure really builds, and against populated state:
identities, contracts with documents and history, tokens, group actions,
address balances, an epoch before and after payout, contested documents. A
change that adds a root tree, a subtree key or a level fails here until it is
described.
The strategy tests. The drive-abci strategy tests check the state of every
chain they run with the same walker (assert_state_conforms_to_structure), at
whatever protocol version the chain ended on. They write far more than the
fixtures do: votes, withdrawals, token distributions, epochs changing, protocol
upgrades. Whatever a change's own strategy tests write is walked, so structure
created only during some operation is caught too.
Coverage. Every node must be reached by some rs-drive fixture or be listed
as reached by the strategy tests. UNVERIFIED, the list for nodes written from
reading the code and never checked against a real GroveDB, is empty and meant
to stay so: whoever describes a node can write a fixture that creates it. The
test also fails when a listed node does get reached by a fixture, so the lists
can only shrink.
The exported file. The description is serialized to
packages/rs-drive/grovedb-structure.json, which is what the viewer reads. A
test fails when the committed file is stale:
UPDATE_GROVEDB_STRUCTURE=1 cargo test -p drive --lib structure::tests
The shape of a layer
GroveDB stores each layer as a Merk, a balanced binary tree, and a proof of
one element carries the hashes along the way from the layer's root. So where a
key sits matters twice: keys near the top have shorter proofs, and every write
below a node rewrites its ancestors in the layer. That is why the root keys
are spread over the byte range with DataContractDocuments on top, and why
ContractGroups took key 124, which hangs below Versions, a tree only the
block-level version bookkeeping writes to (see
Contract Groups).
The exported file records the exact Merk of every layer whose keys are all
fixed, under layer_shapes. GroveDB does not expose the links between the
nodes of a Merk, but a proof does: the proof of a query for everything in a
layer lists every node and how they connect, so replaying its operations
rebuilds the tree. The viewer draws it when you switch a layer to Merk tree.
The shape depends on the order of insertion, not only on the keys. For a
layer reached through fixed keys only, the recorded shape is that of a fresh
chain at the latest protocol version ("origin": "genesis@14"). A chain that
upgraded through earlier versions inserted the same keys in another order and
can differ.
A layer below a template exists once per identity, contract, epoch and so on.
Its shape is recorded from one instance: the fullest one the test fixtures
build ("origin": "fixture contracts_with_documents@14"). That is where
layouts designed around the Merk show: a contract's layer keeps its documents
on top with the contract and everything else below, the other tree keeps
the banlist in the middle, and an identity's seven keys form a full tree with
the keys at the root. Another instance can differ when it holds fewer keys or
got them in another order.
Some layers gain and lose keys over their life, and then one shape is not enough. An epoch's layer is created at genesis holding only its storage fees; its first block adds the start fields, the proposers tree and the processing fees in one batch; and the batch that pays it out deletes the proposers and both fee items and writes the finished epoch info. No epoch ever holds all nine keys. A node declares such states in the order the layer goes through them, each with a title, what it means and what moves the layer into it, and the keys it holds then:
#![allow(unused)] fn main() { .state( "paid", "Paid out", "One batch pays the proposers, deletes the proposers tree and both fee \ items, and writes the finished epoch info.", &["start_block_core_height", "finished_epoch_info", "start_block_height", ...], ) }
The fixtures record one shape per state, and a state's shape must come from an instance holding exactly the keys the state declares. The viewer shows them as State 0, State 1, ... with their explanations. What matters for a faithful shape is that the fixture writes the same keys in the same batches as the block pipeline does, since a Merk batch of several keys gives another tree than the same keys written one at a time.
One document type's layout
The description covers every document type at once, so its document layers
are templates: an index property, one of its values, the [0] where an index
ends, and which tree types each of them can be. Which of those a given
document type gets depends on its keywords (unique, countable,
summable, the ranked keys, timeRange, indexOnly, documentsKeepHistory,
...), and the rules that pick them live in the index walkers.
drive::document::layout::document_type_layout applies those rules to one
document type and returns its concrete layout: the document type tree, the
documents by id, and for each index the property and value trees down to
where it ends, each with the tree or element type Drive writes, the
zero-contribution wrapper a continuation tree gets under an aggregating value
tree, the ranking axes of an indexed tree, the indexes that use the layer,
and conditions such as the tree a unique index falls back to when a value is
null. Each layer names the node of the description it is an instance of, so a
viewer can link to it. It needs only the contract, so it is compiled with the
verify feature as well, and the JavaScript SDKs expose it as
documentTypeLayout(contract, documentTypeName, platformVersion). It follows
the v2 index walkers, so it refuses a platform version before 14.
The tree types come from the functions the walkers call, and the element
choices the walkers make inline are held to them by
should_lay_out_what_drive_writes. It applies 19 test contracts (plain,
unique, compound, countable, summable, ranked and chained indexes, history,
time windows, indexOnly types with flat and preallocated indexes), inserts
documents, and fails on any element whose kind or wrapper the layout does not
predict, and on any layer of the layout Drive did not write. It does not cover
contested indexes, time windows with a ttl, or document updates and
deletes.
Pull requests that change the structure
When a pull request changes grovedb-structure.json, the
GroveDB Structure Preview workflow comments with a link that opens the viewer
on the difference between the merge base and the head of the pull request:
new nodes glow, removed ones stay as ghosts, every ancestor of a change
carries a count so the trail is visible from the root, and a tour walks
through each change. The viewer fetches both files from GitHub and compares
them itself, so the link works before the merge and for forks.
To add structure, follow the checklist in the coding conventions.
Grove Operations
Drive is the storage layer of Dash Platform, and GroveDB is the authenticated data structure (a Merkle tree of trees) that Drive uses under the hood. But Drive never talks to GroveDB directly in its business logic. Instead, every single GroveDB call is wrapped in a versioned Drive method that follows a consistent pattern. This chapter explains why that wrapper layer exists and how it works.
The Problem: Raw GroveDB Is Too Low-Level
If you were to call GroveDB directly throughout Drive's codebase, you would face several problems:
- No cost tracking. GroveDB operations return a
CostContextthat wraps both the result and anOperationCost. If you forget to capture that cost, the fee system breaks. - No version dispatch. Different protocol versions might need different behavior for the same logical operation (like how to handle estimated costs vs. actual costs).
- No consistent API. Each caller would need to handle cost capture, error conversion, and version checking independently.
The grove operations layer solves all three problems by providing a single, consistent abstraction. Every grove operation is a method on Drive that takes a path, a key, some type information, a transaction, and the mutable drive_operations accumulator.
The Module Structure
The grove operations live in packages/rs-drive/src/util/grove_operations/. Each operation is its own submodule:
grove_operations/
mod.rs -- shared types and helpers
grove_insert/
mod.rs -- version dispatcher
v0/mod.rs -- v0 implementation
grove_get_raw/
mod.rs -- version dispatcher
v0/mod.rs -- v0 implementation
grove_delete/
grove_get/
grove_get_raw_optional/
grove_has_raw/
batch_insert/
batch_insert_empty_tree/
batch_delete/
... (30+ more)
Each submodule follows the same structure: a mod.rs that dispatches on the version, and a v0/mod.rs (and potentially v1/, v2/, etc.) with the actual implementation.
The drive_operations Accumulator Pattern
This is the most important pattern to understand. Almost every grove operation method accepts a mutable reference to a Vec<LowLevelDriveOperation>:
#![allow(unused)] fn main() { drive_operations: &mut Vec<LowLevelDriveOperation> }
Instead of returning costs directly, the method pushes the cost of its GroveDB call onto this vector. The caller passes the same vector through multiple operations, accumulating all costs. Later, the batch application system processes this vector to calculate the total fee.
Why accumulate rather than execute immediately? Two reasons:
- Fee estimation. When
applyis false, Drive needs to estimate costs without actually writing to the database. The operations still accumulate cost information, but no state changes occur. - Atomic batching. Multiple operations can be collected and then applied as a single atomic batch. More on this in the Batch Operations chapter.
A Concrete Example: grove_get_raw
Let us trace through a complete grove operation. The version dispatcher is in packages/rs-drive/src/util/grove_operations/grove_get_raw/mod.rs:
#![allow(unused)] fn main() { impl Drive { pub fn grove_get_raw<B: AsRef<[u8]>>( &self, path: SubtreePath<'_, B>, key: &[u8], direct_query_type: DirectQueryType, transaction: TransactionArg, drive_operations: &mut Vec<LowLevelDriveOperation>, drive_version: &DriveVersion, ) -> Result<Option<Element>, Error> { match drive_version.grove_methods.basic.grove_get_raw { 0 => self.grove_get_raw_v0( path, key, direct_query_type, transaction, drive_operations, drive_version, ), version => Err(Error::Drive(DriveError::UnknownVersionMismatch { method: "grove_get_raw".to_string(), known_versions: vec![0], received: version, })), } } } }
The dispatcher consults drive_version.grove_methods.basic.grove_get_raw to determine which implementation version to call. If the version is unknown, it returns an error immediately.
Now the v0 implementation, from grove_get_raw/v0/mod.rs:
#![allow(unused)] fn main() { impl Drive { pub(super) fn grove_get_raw_v0<B: AsRef<[u8]>>( &self, path: SubtreePath<'_, B>, key: &[u8], direct_query_type: DirectQueryType, transaction: TransactionArg, drive_operations: &mut Vec<LowLevelDriveOperation>, drive_version: &DriveVersion, ) -> Result<Option<Element>, Error> { match direct_query_type { DirectQueryType::StatelessDirectQuery { in_tree_type, query_target, } => { let key_info_path = KeyInfoPath::from_known_owned_path(path.to_vec()); let key_info = KeyInfo::KnownKey(key.to_vec()); let cost = match query_target { QueryTarget::QueryTargetTree(flags_size, tree_type) => { GroveDb::average_case_for_get_tree( &key_info_path, &key_info, flags_size, tree_type, in_tree_type, &drive_version.grove_version, ) } QueryTarget::QueryTargetValue(estimated_value_size) => { GroveDb::average_case_for_get_raw( &key_info_path, &key_info, estimated_value_size, in_tree_type, &drive_version.grove_version, ) } }?; drive_operations.push(CalculatedCostOperation(cost)); Ok(None) // No actual data -- just cost estimation } DirectQueryType::StatefulDirectQuery => { let CostContext { value, cost } = self.grove.get_raw( path, key, transaction, &drive_version.grove_version, ); drive_operations.push(CalculatedCostOperation(cost)); Ok(Some(value.map_err(Error::from)?)) } } } } }
This reveals the dual nature of every grove operation: it can operate in stateless mode (for cost estimation) or stateful mode (for actual execution). In stateless mode, it calculates the average-case cost without touching the database and returns None. In stateful mode, it performs the actual GroveDB read and returns the element.
The DirectQueryType Enum
The DirectQueryType enum, defined in packages/rs-drive/src/util/grove_operations/mod.rs, controls this dual behavior:
#![allow(unused)] fn main() { pub enum DirectQueryType { StatelessDirectQuery { in_tree_type: TreeType, query_target: QueryTarget, }, StatefulDirectQuery, } }
-
StatelessDirectQuery: Used for fee estimation. Provides the tree type and query target so the system can calculate costs without reading from disk. TheQueryTargetspecifies whether we are querying for a tree (with flags) or a value (with an estimated size). -
StatefulDirectQuery: Used for actual execution. The system reads from GroveDB and returns real data.
There is also a more general QueryType enum that adds reference size estimation:
#![allow(unused)] fn main() { pub enum QueryType { StatelessQuery { in_tree_type: TreeType, query_target: QueryTarget, estimated_reference_sizes: Vec<u32>, }, StatefulQuery, } }
And a QueryTarget enum that specifies what kind of element we expect to find:
#![allow(unused)] fn main() { pub enum QueryTarget { QueryTargetTree(FlagsLen, TreeType), QueryTargetValue(u32), // estimated value size in bytes } }
Another Example: grove_insert
Inserts follow the same pattern. From packages/rs-drive/src/util/grove_operations/grove_insert/v0/mod.rs:
#![allow(unused)] fn main() { impl Drive { pub(super) fn grove_insert_v0<B: AsRef<[u8]>>( &self, path: SubtreePath<'_, B>, key: &[u8], element: Element, transaction: TransactionArg, options: Option<InsertOptions>, drive_operations: &mut Vec<LowLevelDriveOperation>, drive_version: &DriveVersion, ) -> Result<(), Error> { let cost_context = self.grove.insert( path, key, element, options, transaction, &drive_version.grove_version, ); push_drive_operation_result(cost_context, drive_operations) } } }
This is simpler than the get because inserts are always stateful -- you cannot "estimate" an insert by not doing it. The push_drive_operation_result helper extracts the cost from GroveDB's CostContext and pushes it onto the operations vector:
#![allow(unused)] fn main() { fn push_drive_operation_result<T>( cost_context: CostContext<Result<T, GroveError>>, drive_operations: &mut Vec<LowLevelDriveOperation>, ) -> Result<T, Error> { let CostContext { value, cost } = cost_context; if !cost.is_nothing() { drive_operations.push(CalculatedCostOperation(cost)); } value.map_err(Error::from) } }
Notice the is_nothing() check -- if an operation has zero cost (which can happen), we skip pushing to avoid cluttering the vector.
Batch Apply Types
For operations that work in batch mode (building up a batch of operations to apply atomically), there are corresponding apply-type enums. For example:
#![allow(unused)] fn main() { pub enum BatchDeleteApplyType { StatelessBatchDelete { in_tree_type: TreeType, estimated_key_size: u32, estimated_value_size: u32, }, StatefulBatchDelete { is_known_to_be_subtree_with_sum: Option<MaybeTree>, }, } pub enum BatchInsertTreeApplyType { StatelessBatchInsertTree { in_tree_type: TreeType, tree_type: TreeType, flags_len: FlagsLen, }, StatefulBatchInsertTree, } }
These follow the same stateless/stateful split. The stateless variants carry enough information to estimate costs without touching the database, while the stateful variants trigger actual operations. Each batch apply type can be converted to a DirectQueryType for use with the lower-level grove operations.
The GroveDBToUse Enum
A recent addition supports querying different GroveDB instances:
#![allow(unused)] fn main() { pub enum GroveDBToUse { Current, LatestCheckpoint, Checkpoint(u64), } }
This enables queries against historical checkpoints -- useful for proof generation and state verification at specific block heights.
Method Signature Conventions
Across all grove operations, you will notice a consistent parameter ordering:
&self, path, key, [element], [query_type], transaction, drive_operations, drive_version
&self-- the Drive instance (which holds the GroveDB handle)- Path -- where in the tree
- Key -- which element at that path
- Element -- the data to write (for inserts/replaces)
- Query type -- stateless vs. stateful
- Transaction -- the GroveDB transaction context
drive_operations-- the mutable cost accumulatordrive_version-- for version dispatching
This consistency makes the codebase navigable even though there are 30+ different grove operations.
Rules and Guidelines
Do:
- Always use the grove operation wrappers on
Drive. Never callself.grove.insert()orself.grove.get()directly in business logic. - Pass the
drive_operationsvector through every call chain. It is how costs propagate upward. - Use
StatelessDirectQueryfor fee estimation andStatefulDirectQueryfor actual execution.
Do not:
- Ignore the cost returned by GroveDB operations. The
push_drive_operation_resulthelper exists for this reason. - Mix stateful and stateless queries in a single estimation pass. Pick one mode and stick with it.
- Create new grove operations without following the
mod.rs+v0/mod.rsdispatcher pattern. Consistency is critical.
Batch Operations
Individual grove operations are the atoms. Batch operations are the molecules. Dash Platform never applies a single database write in isolation -- every state transition (a document creation, a contract update, a balance transfer) results in a batch of operations that are applied atomically. Either they all succeed, or none of them do. This chapter covers how that batching works at every level of the stack.
The Three Levels of Abstraction
Drive has three layers of operation abstraction, each serving a different purpose:
DriveOperation-- High-level, domain-aware operations like "add this document" or "apply this contract."LowLevelDriveOperation-- Individual grove operations, cost calculations, and function costs.GroveDbOpBatch-- The final flat list ofQualifiedGroveDbOpitems that GroveDB applies atomically.
The flow is always top-down: DriveOperation -> LowLevelDriveOperation -> GroveDbOpBatch.
DriveOperation: The High-Level API
The DriveOperation enum lives in packages/rs-drive/src/util/batch/drive_op_batch/mod.rs and represents every kind of operation the platform can perform:
#![allow(unused)] fn main() { pub enum DriveOperation<'a> { DataContractOperation(DataContractOperationType<'a>), DocumentOperation(DocumentOperationType<'a>), TokenOperation(TokenOperationType), WithdrawalOperation(WithdrawalOperationType), IdentityOperation(IdentityOperationType), PrefundedSpecializedBalanceOperation(PrefundedSpecializedBalanceOperationType), SystemOperation(SystemOperationType), GroupOperation(GroupOperationType), AddressFundsOperation(AddressFundsOperationType), GroveDBOperation(QualifiedGroveDbOp), GroveDBOpBatch(GroveDbOpBatch), } }
Each variant wraps a domain-specific operation type. For example, a DocumentOperationType might be an AddDocument, UpdateDocument, or DeleteDocument. A DataContractOperationType might be ApplyContract. And so on.
The last two variants -- GroveDBOperation and GroveDBOpBatch -- are escape hatches for when code already has raw GroveDB operations and just wants to include them in the batch.
The DriveLowLevelOperationConverter Trait
Every DriveOperation variant knows how to convert itself into a list of low-level operations. This is defined by the DriveLowLevelOperationConverter trait:
#![allow(unused)] fn main() { pub trait DriveLowLevelOperationConverter { fn into_low_level_drive_operations( self, drive: &Drive, estimated_costs_only_with_layer_info: &mut Option< HashMap<KeyInfoPath, EstimatedLayerInformation>, >, block_info: &BlockInfo, transaction: TransactionArg, platform_version: &PlatformVersion, ) -> Result<Vec<LowLevelDriveOperation>, Error>; } }
The estimated_costs_only_with_layer_info parameter is key. When it is None, the converter performs actual operations (stateful mode). When it is Some(HashMap), the converter only estimates costs and fills in layer information for GroveDB's cost estimation (stateless mode).
The DriveOperation enum implements this trait by dispatching to each variant:
#![allow(unused)] fn main() { impl DriveLowLevelOperationConverter for DriveOperation<'_> { fn into_low_level_drive_operations( self, drive: &Drive, estimated_costs_only_with_layer_info: &mut Option< HashMap<KeyInfoPath, EstimatedLayerInformation> >, block_info: &BlockInfo, transaction: TransactionArg, platform_version: &PlatformVersion, ) -> Result<Vec<LowLevelDriveOperation>, Error> { match self { DriveOperation::DataContractOperation(op) => op.into_low_level_drive_operations( drive, estimated_costs_only_with_layer_info, block_info, transaction, platform_version, ), DriveOperation::DocumentOperation(op) => op.into_low_level_drive_operations( drive, estimated_costs_only_with_layer_info, block_info, transaction, platform_version, ), // ... each variant delegates to its own converter DriveOperation::GroveDBOperation(op) => Ok(vec![GroveOperation(op)]), DriveOperation::GroveDBOpBatch(operations) => Ok(operations.operations.into_iter() .map(GroveOperation).collect()), } } } }
The apply_drive_operations Flow
The centerpiece of the batch system is Drive::apply_drive_operations. From packages/rs-drive/src/util/batch/drive_op_batch/drive_methods/apply_drive_operations/v0/mod.rs:
#![allow(unused)] fn main() { impl Drive { pub(crate) fn apply_drive_operations_v0( &self, operations: Vec<DriveOperation>, apply: bool, block_info: &BlockInfo, transaction: TransactionArg, platform_version: &PlatformVersion, previous_fee_versions: Option<&CachedEpochIndexFeeVersions>, ) -> Result<FeeResult, Error> { if operations.is_empty() { return Ok(FeeResult::default()); } let mut low_level_operations = vec![]; let mut estimated_costs_only_with_layer_info = if apply { None::<HashMap<KeyInfoPath, EstimatedLayerInformation>> } else { Some(HashMap::new()) }; let mut finalize_tasks: Vec<DriveOperationFinalizeTask> = Vec::new(); for drive_op in operations { // Collect finalize tasks before conversion if let Some(tasks) = drive_op.finalization_tasks(platform_version)? { finalize_tasks.extend(tasks); } // Convert high-level to low-level operations low_level_operations.append( &mut drive_op.into_low_level_drive_operations( self, &mut estimated_costs_only_with_layer_info, block_info, transaction, platform_version, )? ); } let mut cost_operations = vec![]; // Apply the batch atomically self.apply_batch_low_level_drive_operations( estimated_costs_only_with_layer_info, transaction, low_level_operations, &mut cost_operations, &platform_version.drive, )?; // Execute post-commit finalize tasks for task in finalize_tasks { task.execute(self, platform_version); } // Calculate total fee from accumulated costs Drive::calculate_fee( None, Some(cost_operations), &block_info.epoch, self.config.epochs_per_era, platform_version, previous_fee_versions, ) } } }
Let us trace the flow step by step:
-
Mode selection. If
applyis true,estimated_costs_only_with_layer_infoisNone, triggering stateful execution. If false, it isSome(HashMap), triggering cost estimation only. -
Finalize task collection. Before converting each operation, we collect any finalize tasks it declares. These are post-commit callbacks (covered in the Finalize Tasks chapter).
-
Conversion. Each
DriveOperationis converted into zero or moreLowLevelDriveOperationitems. A single high-level operation like "add document" might produce dozens of low-level operations (insert the document itself, update each index, update the contract's document count, etc.). -
Batch application. All low-level operations are applied as a single atomic batch through
apply_batch_low_level_drive_operations. -
Finalization. Post-commit tasks execute (like invalidating caches).
-
Fee calculation. The accumulated cost operations are converted into a
FeeResult.
GroveDbOpBatch: The Final Layer
Before operations hit GroveDB, they are split into two categories. From packages/rs-drive/src/util/operations/apply_batch_low_level_drive_operations/v0/mod.rs:
#![allow(unused)] fn main() { impl Drive { pub(crate) fn apply_batch_low_level_drive_operations_v0( &self, estimated_costs_only_with_layer_info: Option< HashMap<KeyInfoPath, EstimatedLayerInformation>, >, transaction: TransactionArg, batch_operations: Vec<LowLevelDriveOperation>, drive_operations: &mut Vec<LowLevelDriveOperation>, drive_version: &DriveVersion, ) -> Result<(), Error> { let (grove_db_operations, mut other_operations) = LowLevelDriveOperation::grovedb_operations_batch_consume_with_leftovers( batch_operations, ); if !grove_db_operations.is_empty() { self.apply_batch_grovedb_operations( estimated_costs_only_with_layer_info, transaction, grove_db_operations, drive_operations, drive_version, )?; } drive_operations.append(&mut other_operations); Ok(()) } } }
The grovedb_operations_batch_consume_with_leftovers method partitions the operations:
GroveOperationvariants become aGroveDbOpBatchthat is applied atomically to GroveDB.- Everything else (
CalculatedCostOperation,FunctionOperation,PreCalculatedFeeResult) is kept as-is for fee calculation.
The GroveDbOpBatch itself is defined in packages/rs-drive/src/util/batch/grovedb_op_batch/mod.rs:
#![allow(unused)] fn main() { pub struct GroveDbOpBatch { pub(crate) operations: Vec<QualifiedGroveDbOp>, } }
It is a thin wrapper around a vector of QualifiedGroveDbOp -- GroveDB's native batch operation type. The wrapper provides convenience methods for building batches:
#![allow(unused)] fn main() { pub trait GroveDbOpBatchV0Methods { fn new() -> Self; fn push(&mut self, op: QualifiedGroveDbOp); fn add_insert_empty_tree(&mut self, path: Vec<Vec<u8>>, key: Vec<u8>); fn add_insert_empty_sum_tree(&mut self, path: Vec<Vec<u8>>, key: Vec<u8>); fn add_delete(&mut self, path: Vec<Vec<u8>>, key: Vec<u8>); fn add_insert(&mut self, path: Vec<Vec<u8>>, key: Vec<u8>, element: Element); fn verify_consistency_of_operations(&self) -> GroveDbOpConsistencyResults; fn contains<'c, P>(&self, path: P, key: &[u8]) -> Option<&GroveOp>; fn remove<'c, P>(&mut self, path: P, key: &[u8]) -> Option<GroveOp>; fn remove_if_insert(&mut self, path: Vec<Vec<u8>>, key: &[u8]) -> Option<GroveOp>; } }
The verify_consistency_of_operations method is particularly important -- it checks that the batch does not contain conflicting operations (like inserting and deleting the same key).
Building a Batch: A Real Example
The test code in drive_op_batch/mod.rs shows how a typical batch is assembled:
#![allow(unused)] fn main() { let mut drive_operations = vec![]; // Step 1: Apply a contract drive_operations.push(DataContractOperation(ApplyContract { contract: Cow::Borrowed(&contract), storage_flags: None, })); // Step 2: Add a document drive_operations.push(DocumentOperation(AddDocument { owned_document_info: OwnedDocumentInfo { document_info: DocumentRefInfo(( &document, StorageFlags::optional_default_as_cow(), )), owner_id: None, }, contract_info: DataContractInfo::BorrowedDataContract(&contract), document_type_info: DocumentTypeInfo::DocumentTypeRef(document_type), override_document: false, })); // Step 3: Apply everything atomically drive.apply_drive_operations( drive_operations, true, // actually apply, not just estimate &BlockInfo::default(), Some(&db_transaction), platform_version, None, )?; }
The contract application and document insertion happen in the same atomic batch. If the document insert fails (perhaps due to a uniqueness constraint violation), the contract application is also rolled back. This all-or-nothing guarantee is fundamental to platform correctness.
The Display Implementation
The GroveDbOpBatch has a custom Display implementation that produces human-readable output, mapping raw byte paths to meaningful names:
#![allow(unused)] fn main() { impl fmt::Display for GroveDbOpBatch { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { for op in &self.operations { let (path_string, known_path) = readable_path(&op.path); let (key_string, _) = readable_key_info(known_path, &op.key); writeln!(f, " Path: {}", path_string)?; writeln!(f, " Key: {}", key_string)?; // ... operation details } } } }
This translates paths like [0x03] into Identities(3) and keys like 32-byte arrays into IdentityId(bs58::...). Invaluable for debugging.
Rules and Guidelines
Do:
- Always use
apply_drive_operationsfor applying batches. It handles the full pipeline: conversion, application, finalization, and fee calculation. - Collect all operations for a state transition into a single
Vec<DriveOperation>before applying. - Use the
apply: falseflag for dry-run fee estimation before committing.
Do not:
- Apply operations one at a time. Always batch them for atomicity.
- Mix stateful and stateless operations in the same batch application pass.
- Forget to handle the
FeeResultreturned byapply_drive_operations. The fee system depends on it. - Manually construct
GroveDbOpBatchobjects unless you are working at the lowest level. PreferDriveOperationfor business logic.
Cost Tracking
Dash Platform is a fee-based system. Every operation -- reading a document, inserting a key, hashing a value -- has a cost measured in platform credits. This chapter explains how Drive tracks those costs from individual operations all the way through to the final fee result.
The Problem: Why Explicit Cost Tracking?
Some systems use gas metering: you start with a budget, and every opcode decrements it. Dash Platform takes a different approach. Instead of metering, it accumulates the costs of operations as they execute and then calculates the total fee at the end.
This matters because:
- Costs depend on what actually happened. A GroveDB insert into a deep tree costs more than one into a shallow tree because of the Merkle proof updates involved. You cannot know this in advance -- you have to do the insert and measure.
- Storage fees and processing fees are different. Storage fees are based on bytes added and removed. Processing fees cover computation: seeks, hash operations, byte loading. They need to be calculated separately.
- Refunds are possible. When data is removed from storage, the original storage fee can be partially refunded. This requires knowing when the data was originally stored (which epoch), making the calculation depend on historical state.
LowLevelDriveOperation: The Cost Carrier
The LowLevelDriveOperation enum is the vehicle that carries cost information through the system. Defined in packages/rs-drive/src/fees/op.rs:
#![allow(unused)] fn main() { pub enum LowLevelDriveOperation { GroveOperation(QualifiedGroveDbOp), FunctionOperation(FunctionOp), CalculatedCostOperation(OperationCost), PreCalculatedFeeResult(FeeResult), } }
Four variants, each representing a different kind of cost:
GroveOperation
A raw GroveDB operation (insert, delete, get, etc.) that has not yet been executed. When a batch is built up (as described in the Batch Operations chapter), individual grove operations accumulate as this variant. They carry no cost yet -- the cost is determined when the batch is applied to GroveDB.
CalculatedCostOperation
An OperationCost from a GroveDB operation that has already been executed (or estimated). This is what gets pushed onto the drive_operations vector by the grove operation wrappers:
#![allow(unused)] fn main() { fn push_drive_operation_result<T>( cost_context: CostContext<Result<T, GroveError>>, drive_operations: &mut Vec<LowLevelDriveOperation>, ) -> Result<T, Error> { let CostContext { value, cost } = cost_context; if !cost.is_nothing() { drive_operations.push(CalculatedCostOperation(cost)); } value.map_err(Error::from) } }
The OperationCost (from grovedb_costs) tracks:
seek_count: Number of disk seeks performedstorage_cost: Bytes added, replaced, and removed (with per-epoch tracking for refunds)storage_loaded_bytes: Bytes read from storagehash_node_calls: Number of hash operations for Merkle tree updates
FunctionOperation
Represents the cost of a pure computation like hashing. Defined as:
#![allow(unused)] fn main() { pub struct FunctionOp { pub(crate) hash: HashFunction, pub(crate) rounds: u32, } }
With supported hash functions:
#![allow(unused)] fn main() { pub enum HashFunction { Sha256RipeMD160, Sha256, Sha256_2, // Double SHA-256 Blake3, } }
Each hash function has a base cost and a per-block cost. The total cost of a FunctionOp is:
#![allow(unused)] fn main() { impl FunctionOp { fn cost(&self, fee_version: &FeeVersion) -> Credits { let block_cost = (self.rounds as u64) .saturating_mul(self.hash.block_cost(fee_version)); self.hash.base_cost(fee_version).saturating_add(block_cost) } } }
You can create a FunctionOp either by specifying the number of rounds directly or by providing the byte count (which calculates rounds based on the hash function's block size):
#![allow(unused)] fn main() { impl FunctionOp { pub fn new_with_round_count(hash: HashFunction, rounds: u32) -> Self { FunctionOp { hash, rounds } } pub fn new_with_byte_count(hash: HashFunction, byte_count: u16) -> Self { let blocks = byte_count / hash.block_size() + 1; let rounds = blocks + hash.rounds() - 1; FunctionOp { hash, rounds: rounds as u32 } } } }
PreCalculatedFeeResult
A fee result that was already computed elsewhere and just needs to be included in the total. This is a pass-through -- no further calculation needed.
BaseOp: Arithmetic Operation Costs
For simple computational operations (not storage-related), the BaseOp enum provides fixed costs:
#![allow(unused)] fn main() { pub enum BaseOp { Stop, Add, Mul, Sub, Div, Sdiv, Mod, Smod, Addmod, Mulmod, Signextend, Lt, Gt, Slt, Sgt, Eq, Iszero, And, Or, Xor, Not, Byte, } impl BaseOp { pub fn cost(&self) -> u64 { match self { BaseOp::Stop => 0, BaseOp::Add | BaseOp::Sub => 12, BaseOp::Mul | BaseOp::Div | BaseOp::Sdiv | BaseOp::Mod | BaseOp::Smod | BaseOp::Signextend => 20, BaseOp::Addmod | BaseOp::Mulmod => 32, BaseOp::Lt | BaseOp::Gt | BaseOp::Slt | BaseOp::Sgt | BaseOp::Eq | BaseOp::Iszero | BaseOp::And | BaseOp::Or | BaseOp::Xor | BaseOp::Not | BaseOp::Byte => 12, } } } }
These are EVM-inspired operation costs, adapted for the platform's fee model. Comparisons and bitwise operations cost 12 credits. Multiplication and division cost 20. Modular arithmetic costs 32.
The consume_to_fees_v0 Pipeline
When all operations are collected, they are converted into fee results through consume_to_fees_v0:
#![allow(unused)] fn main() { pub fn consume_to_fees_v0( drive_operations: Vec<LowLevelDriveOperation>, epoch: &Epoch, epochs_per_era: u16, fee_version: &FeeVersion, previous_fee_versions: Option<&CachedEpochIndexFeeVersions>, ) -> Result<Vec<FeeResult>, Error> { drive_operations.into_iter().map(|operation| match operation { PreCalculatedFeeResult(f) => Ok(f), FunctionOperation(op) => Ok(FeeResult { processing_fee: op.cost(fee_version), ..Default::default() }), _ => { let cost = operation.operation_cost()?; // Storage fee: bytes added * rate per byte let storage_fee = cost.storage_cost.added_bytes as u64 * fee_version.storage.storage_disk_usage_credit_per_byte; // Processing fee: seeks + loaded bytes + hash calls + ... let processing_fee = cost.ephemeral_cost(fee_version)?; // Refunds from removed data let (fee_refunds, removed_bytes_from_system) = match cost.storage_cost.removed_bytes { NoStorageRemoval => (FeeRefunds::default(), 0), BasicStorageRemoval(amount) => (FeeRefunds::default(), amount), SectionedStorageRemoval(removal_per_epoch_by_identifier) => { // Calculate epoch-aware refunds (FeeRefunds::from_storage_removal( removal_per_epoch_by_identifier, epoch.index, epochs_per_era, previous_fee_versions, )?, system_amount) } }; Ok(FeeResult { storage_fee, processing_fee, fee_refunds, removed_bytes_from_system, }) } }).collect() } }
Each operation produces a FeeResult with four components:
storage_fee: The cost of new bytes written to persistent storage.processing_fee: The ephemeral cost of computation and I/O.fee_refunds: Credits returned because previously-stored data was removed.removed_bytes_from_system: Bytes removed that were stored by the system (not any particular identity), so no refund is issued.
Ephemeral Cost Calculation
The ephemeral_cost method on OperationCost computes the processing fee from the raw operation metrics:
#![allow(unused)] fn main() { impl DriveCost for OperationCost { fn ephemeral_cost(&self, fee_version: &FeeVersion) -> Result<Credits, Error> { let OperationCost { seek_count, storage_cost, storage_loaded_bytes, hash_node_calls, } = self; let seek_cost = (*seek_count as u64) .checked_mul(fee_version.storage.storage_seek_cost)?; let storage_added_bytes_ephemeral_cost = (storage_cost.added_bytes as u64) .checked_mul(fee_version.storage.storage_processing_credit_per_byte)?; let storage_replaced_bytes_ephemeral_cost = (storage_cost.replaced_bytes as u64) .checked_mul(fee_version.storage.storage_processing_credit_per_byte)?; let storage_loaded_bytes_cost = (*storage_loaded_bytes) .checked_mul(fee_version.storage.storage_load_credit_per_byte)?; let blake3_total = fee_version.hashing.blake3_base + fee_version.hashing.blake3_per_block; let hash_node_cost = blake3_total * (*hash_node_calls as u64); // Sum all costs with overflow checking seek_cost .checked_add(storage_added_bytes_ephemeral_cost) .and_then(|c| c.checked_add(storage_replaced_bytes_ephemeral_cost)) .and_then(|c| c.checked_add(storage_loaded_bytes_cost)) .and_then(|c| c.checked_add(hash_node_cost)) .ok_or_else(|| get_overflow_error("ephemeral cost addition overflow")) } } }
Notice that every multiplication and addition uses checked arithmetic. In a fee system, overflows would be catastrophic -- an underflowed fee could let someone store unlimited data for free.
Helper Methods on LowLevelDriveOperation
The LowLevelDriveOperation type provides several methods for working with collections of operations:
#![allow(unused)] fn main() { impl LowLevelDriveOperation { // Combine all CalculatedCostOperation costs into one pub fn combine_cost_operations( operations: &[LowLevelDriveOperation] ) -> OperationCost { ... } // Extract GroveOperation variants into a batch pub fn grovedb_operations_batch( operations: &[LowLevelDriveOperation] ) -> GroveDbOpBatch { ... } // Same but consuming the vector pub fn grovedb_operations_batch_consume( operations: Vec<LowLevelDriveOperation> ) -> GroveDbOpBatch { ... } // Partition: grove ops go to batch, rest stays as leftovers pub fn grovedb_operations_batch_consume_with_leftovers( operations: Vec<LowLevelDriveOperation>, ) -> (GroveDbOpBatch, Vec<LowLevelDriveOperation>) { ... } } }
The grovedb_operations_batch_consume_with_leftovers method is particularly important -- it is used during batch application to separate the grove operations (which go to GroveDB) from the cost operations (which go to fee calculation).
Constructing Operations
LowLevelDriveOperation also has constructors for common operations:
#![allow(unused)] fn main() { impl LowLevelDriveOperation { pub fn for_known_path_key_empty_tree( path: Vec<Vec<u8>>, key: Vec<u8>, storage_flags: Option<&StorageFlags>, ) -> Self { ... } pub fn for_known_path_key_empty_sum_tree( path: Vec<Vec<u8>>, key: Vec<u8>, storage_flags: Option<&StorageFlags>, ) -> Self { ... } pub fn insert_for_known_path_key_element( path: Vec<Vec<u8>>, key: Vec<u8>, element: Element, ) -> Self { GroveOperation(QualifiedGroveDbOp::insert_or_replace_op( path, key, element )) } pub fn replace_for_known_path_key_element( path: Vec<Vec<u8>>, key: Vec<u8>, element: Element, ) -> Self { GroveOperation(QualifiedGroveDbOp::replace_op( path, key, element )) } } }
These provide a cleaner API than constructing QualifiedGroveDbOp directly, and they handle storage flags properly.
Rules and Guidelines
Do:
- Use checked arithmetic everywhere in fee calculations.
checked_mul,checked_add, and friends. - Construct
FunctionOpwithnew_with_byte_countwhen you know the input size, andnew_with_round_countwhen you know the rounds. - Let operations accumulate in the
drive_operationsvector throughout the call chain.
Do not:
- Call
operation_cost()on aGroveOperation-- it will return an error. Grove operations must be executed first; onlyCalculatedCostOperationcarries a usable cost. - Forget that storage fees and processing fees are calculated differently. Storage fees are proportional to bytes. Processing fees are a complex function of seeks, loads, hashes, and byte movements.
- Assume fee rates are constant. They are versioned through
FeeVersionand can change between protocol versions. - Ignore
removed_bytes_from_system. This tracks bytes removed that belong to the system rather than a specific identity, affecting the refund calculation.
Finalize Tasks
Most operations on Dash Platform follow a straightforward path: convert high-level operations to low-level ones, apply them atomically, calculate fees. But some operations need something to happen after the batch has been successfully committed. That is what finalize tasks are for.
The Problem: Post-Apply Side Effects
Consider what happens when a data contract is updated. The updated contract is written to GroveDB as part of the atomic batch. But Drive also caches contracts in memory for fast access. After the batch is applied, that cache entry is stale -- it still holds the old version of the contract.
You cannot refresh the cache before the batch is applied, because applying might fail (GroveDB could reject the batch due to a consistency error). And you cannot refresh it during the apply, because the batch application is a single atomic operation on GroveDB. You need a post-apply callback: "if the batch succeeds, do this."
That is exactly what DriveOperationFinalizeTask provides.
The DriveOperationFinalizeTask Enum
Defined in packages/rs-drive/src/util/batch/drive_op_batch/finalize_task.rs:
#![allow(unused)] fn main() { pub enum DriveOperationFinalizeTask { RefreshDataContractCache { contract_id: Identifier }, } }
Currently there is only one variant: RefreshDataContractCache. When a data contract is created or updated, this task is registered. After the batch is applied successfully, it re-seeds Drive's in-memory cache from what state now holds for the contract, reading through the transaction the batch was applied in.
The execution delegates to Drive::refresh_data_contract_cache_from_state:
#![allow(unused)] fn main() { impl DriveOperationFinalizeTask { pub fn execute( self, drive: &Drive, transaction: TransactionArg, platform_version: &PlatformVersion, ) -> Result<(), Error> { match self { DriveOperationFinalizeTask::RefreshDataContractCache { contract_id } => drive .refresh_data_contract_cache_from_state( contract_id.to_buffer(), transaction, platform_version, ), } } } }
The task takes the transaction because what it seeds must be what this transaction holds, not committed state. Inside a block, the batch was applied in the block transaction and the update is not committed yet: the refresh reads the updated contract through that transaction, marks the contract as modified in the block, and seeds the block cache with it. Outside a block (a caller that passed no transaction, so the batch committed on its own) the refresh only evicts the superseded copy, and the next reader reloads it from committed state.
The DriveOperationFinalizationTasks Trait
Not every DriveOperation has finalize tasks. The trait that declares them is:
#![allow(unused)] fn main() { pub trait DriveOperationFinalizationTasks { fn finalization_tasks( &self, platform_version: &PlatformVersion, ) -> Result<Option<Vec<DriveOperationFinalizeTask>>, Error>; } }
The return type is Option<Vec<...>> rather than just Vec<...>. This is a deliberate optimization -- since only one operation type currently has finalize tasks, returning None (rather than an empty Vec) avoids unnecessary heap allocations for the vast majority of operations.
The implementation on DriveOperation dispatches through versioning:
#![allow(unused)] fn main() { impl DriveOperationFinalizationTasks for DriveOperation<'_> { fn finalization_tasks( &self, platform_version: &PlatformVersion, ) -> Result<Option<Vec<DriveOperationFinalizeTask>>, Error> { match platform_version .drive .methods .state_transitions .operations .finalization_tasks { 0 => self.finalization_tasks_v0(platform_version), version => Err(Error::Drive(DriveError::UnknownVersionMismatch { method: "DriveOperation.finalization_tasks".to_string(), known_versions: vec![0], received: version, })), } } } }
And the v0 implementation only checks data contract operations:
#![allow(unused)] fn main() { impl DriveOperation<'_> { fn finalization_tasks_v0( &self, platform_version: &PlatformVersion, ) -> Result<Option<Vec<DriveOperationFinalizeTask>>, Error> { match self { DriveOperation::DataContractOperation(o) => o.finalization_tasks(platform_version), _ => Ok(None), } } } }
Every other operation variant -- documents, identities, tokens, withdrawals -- returns None. Only data contract operations can produce finalize tasks.
How Finalize Tasks Integrate with Batch Application
The integration point is in apply_drive_operations_v0, which we saw in the Batch Operations chapter. Here is the relevant excerpt:
#![allow(unused)] fn main() { pub(crate) fn apply_drive_operations_v0( &self, operations: Vec<DriveOperation>, apply: bool, block_info: &BlockInfo, transaction: TransactionArg, platform_version: &PlatformVersion, previous_fee_versions: Option<&CachedEpochIndexFeeVersions>, ) -> Result<FeeResult, Error> { // ... let mut finalize_tasks: Vec<DriveOperationFinalizeTask> = Vec::new(); for drive_op in operations { // Step 1: Collect finalize tasks BEFORE converting the operation if let Some(tasks) = drive_op.finalization_tasks(platform_version)? { finalize_tasks.extend(tasks); } // Step 2: Convert to low-level operations (consumes drive_op) low_level_operations.append( &mut drive_op.into_low_level_drive_operations(/* ... */)? ); } // Step 3: Apply the batch atomically self.apply_batch_low_level_drive_operations(/* ... */)?; // Step 4: Execute finalize tasks AFTER a successful apply, and only when // the batch was applied rather than estimated. They read through the // caller's transaction; `caller_transaction` is `None` exactly when the // batch committed on its own just above. if apply { for task in finalize_tasks { task.execute(self, caller_transaction, platform_version)?; } } // Step 5: Calculate fees Drive::calculate_fee(/* ... */) } }
The ordering is critical:
-
Collect finalize tasks first. This happens before
into_low_level_drive_operationsbecause that method consumes theDriveOperation(it takesself, not&self). After conversion, the original operation is gone. -
Apply the batch. If this fails, we return the error immediately. The finalize tasks never execute.
-
Execute finalize tasks only on success. By the time we reach step 4, we know the batch was applied successfully. Now it is safe to refresh caches and perform other side effects. An estimation-only call (
apply == false) writes nothing, so it runs no finalize tasks either.
The Cache Refresh Pattern
Why a refresh rather than a plain eviction? Drive maintains an in-memory cache of frequently-accessed data contracts, and two kinds of reader share it: block execution, which reads through the block transaction on the consensus thread, and the query threads, which read committed state with no transaction, concurrently, and populate the global half of the cache with what they read.
Without any refresh, here is what would go wrong:
- Block N: Contract "foo" is at version 3 in GroveDB and cached.
- Block N+1: A state transition updates "foo" to version 4 in GroveDB.
- Block N+1: Without a refresh, the block still reads version 3 from the cache.
- Block N+1: Document validation uses the stale version 3 schema, potentially accepting invalid documents.
A plain eviction is not enough, because of the query threads. Between the eviction and the block's commit, a query can read committed state (still version 3) and put version 3 back into the global cache. A transactional read that missed the block cache and fell back to the global cache would then be handed version 3 again, while a validator whose cache happened to be cold reads version 4 from the transaction: the two validators execute the rest of the block against different contracts and compute different app hashes.
The refresh closes this in two ways, both implemented in DataContractCache (packages/rs-drive/src/cache/data_contract.rs):
- It marks the contract as modified in the block. From then until the block cache is cleared or promoted, a transactional read of that contract is served from the block cache or from state through the transaction, never from the global cache. This holds even if the seeded block-cache entry is evicted, and it is what the proposer relies on when it rolls a transition back: the rollback drops the modified contracts from the block cache, and the next read goes to the rolled-back transaction.
- The block cache is promoted into the global cache only after the block transaction is committed, by the
finalize_blockhandler, and every committed-state read carries aCommittedGenerationsnapshot taken before it read state. A read that straddles a commit cannot publish what it read, so a query thread descheduled across the commit cannot clobber the promoted definition with the pre-block one.
When to Use Finalize Tasks
Finalize tasks are the right tool when you need to perform side effects that:
-
Must not happen if the batch fails. If you refresh a cache before the apply and the apply fails, you have seeded a definition the transaction does not hold.
-
Are not idempotent with respect to partial application. A cache refresh is fine to do after the apply because the cache will self-heal on the next access. But if your side effect were "send a network message," you would want to be very sure the batch actually committed.
-
Operate on data outside GroveDB. GroveDB's atomic batch guarantees only cover GroveDB state. In-memory caches, external systems, and non-transactional state all need explicit post-commit handling.
Extending Finalize Tasks
To add a new finalize task:
- Add a variant to the
DriveOperationFinalizeTaskenum infinalize_task.rs. - Implement its execution in the
executemethod's match block. - In the relevant
DriveOperationvariant'sfinalization_tasksimplementation, return the new task when appropriate.
The design is intentionally simple and extensible. The enum + trait pattern means new finalize tasks do not affect existing code paths.
Rules and Guidelines
Do:
- Collect finalize tasks before consuming
DriveOperationviainto_low_level_drive_operations. - Execute finalize tasks only after confirming the batch was applied successfully.
- Keep finalize task execution fast. They run synchronously in the block processing pipeline.
- Read through the transaction the batch was applied in. What a task seeds must be what that transaction holds, not committed state.
Do not:
- Put business logic in finalize tasks. They are for side effects like cache management, not for state mutations. State mutations belong in the batch itself.
- Execute finalize tasks if the batch application returns an error. The whole point is that they only run on success.
- Rely on finalize tasks for exactly-once side effects. If the process crashes between the apply and the finalize task, the task will not run; the cache refresh survives this because Drive's caches are rebuilt from disk on restart.
- Introduce finalize tasks with external side effects (like network calls) without careful consideration of failure modes. Keep them fast, local, and idempotent.
Indexes
Drive stores documents in GroveDB. Every document type has a primary-key tree (documents keyed by document ID), plus zero or more secondary indexes the contract author declares in the document schema. This chapter is a reference for the Index struct's fields, what they mean for the on-disk layout, and how Drive walks indexes during inserts and queries.
What an Index Is
A document type's indices array tells Drive: "for queries that filter or sort by these properties, build a sorted lookup so they don't have to enumerate every document." Each entry in indices becomes one secondary index; Drive maintains it on every insert/update/delete so that queries which match the index prefix are O(prefix walk) rather than O(documents).
Concrete example. Given:
{
"person": {
"type": "object",
"indices": [
{
"name": "byLastName",
"properties": [{ "lastName": "asc" }]
}
],
"properties": {
"firstName": { "type": "string", "position": 0 },
"lastName": { "type": "string", "position": 1 }
},
"required": ["firstName", "lastName"],
"additionalProperties": false
}
}
— a query like where lastName = "Smith" reaches the matching documents through the byLastName index in O(log n) plus the per-result IO. Without that index it would be a full document-type scan.
The Index Struct
The compiled-Rust shape — the JSON schema fields are deserialized into this — lives in packages/rs-dpp/src/data_contract/document_type/index/mod.rs:
#![allow(unused)] fn main() { pub struct Index { pub name: String, pub properties: Vec<IndexProperty>, pub unique: bool, pub null_searchable: bool, pub contested_index: Option<ContestedIndexInformation>, pub countable: IndexCountability, } pub struct IndexProperty { pub name: String, pub ascending: bool, } }
name
A short, human-readable identifier for the index (e.g. "byOwnerAndType"). It shows up in error messages and is the key used in document_type.indexes() (BTreeMap<String, Index>). Every document meta-schema requires it. A parse that skips schema validation (check tx, test fixtures) and meets an unnamed index derives the name from the properties and their directions, joined with |, so every parse of the same contract agrees on it. Two indexes within the same document type cannot share a name.
properties: Vec<IndexProperty>
The ordered list of columns this index covers. Each IndexProperty is a (name, ascending) pair. Order matters: a query has to match a prefix of these properties for the index to be useful. An index [lastName, firstName] answers where lastName = X and where lastName = X AND firstName = Y but not where firstName = Y alone.
The schema form is:
"properties": [
{ "lastName": "asc" },
{ "firstName": "asc" }
]
Every document meta-schema accepts only "asc". Drive stores index entries in ascending order; a query chooses its own result order.
unique: bool
If true, no two documents may share the same combination of values for the indexed properties. The platform enforces this on insert: a duplicate trips a DuplicateUniqueIndexError consensus error.
A unique index changes the on-disk layout at the terminal level: instead of a sub-tree of document references keyed by document ID, the terminal stores a single bare Reference element pointing at the one document that matched. See Tree Type at the Terminal Level below.
Uniqueness can't be enforced when an indexed property is null, so a document with any null in the index path falls back to the non-unique storage shape for that document. See Null Handling.
null_searchable: bool
Defaults to true. Controls what happens when all indexed properties of a document are null:
null_searchable: true— the document is still indexed at the all-null path, so a query against the all-null prefix can find it.null_searchable: false— Drive skips the index insertion entirely. Documents with all-null index values exist (in the primary-key tree) but are not reachable via this index.
The flag only affects the all-null case. A document with some null values gets indexed regardless.
skip_if_absent: bool and skip_if_absent_properties: Vec<String>
From protocol version 14 an index may skip documents that omit a property of its skip set (skipIfAbsent: true for every optional property of the index, or an array naming them). A skipped document gets no reference in the index and builds none of the index's own trees, wherever the skip property sits in the property list. The other optional properties of a stored type's skip index keep the null layout below. skip_if_absent_properties holds the resolved set, so true and the array of the same properties parse to equal indexes; the level info at the index's end carries it too, since the walkers only see merged levels. See Null Handling for how the walkers apply it, and Conditional participation for the indexOnly rules.
contested_index: Option<ContestedIndexInformation>
When set, this index identifies a scarce, contested resource (the canonical example is a DPNS name like dash). Documents trying to register the same value under a contested index don't auto-fail with a uniqueness error — they enter a masternode-vote resolution where each contender's claim is held until voting concludes. Contested indexes must also be unique: true; the parser rejects the combination otherwise.
Out of scope for this chapter; see DPNS / contested-resource docs for the full lifecycle.
countable: IndexCountability
Controls whether the terminal tree under each indexed value carries a count, and which count-tree variant. Three variants:
| Value | Tree variant | Capabilities |
|---|---|---|
NotCountable (default) | NormalTree | No count fast path |
Countable | CountTree | O(1) totals at the root |
CountableAllowingOffset | ProvableCountTree | O(1) totals plus per-node counts that will enable future O(log n) range / offset queries |
summable: Option<String> and range_summable: bool
The sum-side analog of countable / range_countable. When summable = Some(<property_name>), the terminal tree under each indexed value carries a running sum of the named property across the documents at that value — O(1) reads for SUM(<property>) WHERE <indexed_field> = X queries. The named property must be type: integer and listed in the document type's required array (the DPP validator enforces this at contract creation).
range_summable: true is the sum-side counterpart of range_countable: per-node aggregated sums committed to every internal merk node of the property-name tree, so SUM(<property>) WHERE <indexed_field> BETWEEN A AND B queries land on grovedb's AggregateSumOnRange primitive — O(log n), no document enumeration. Like range_countable, it requires summable to be set; it's additive, not a replacement.
summable | range_summable | Property-name tree | Value tree | Capabilities |
|---|---|---|---|---|
None (default) | – | NormalTree | NormalTree | No sum fast path |
Some("amount") | false | NormalTree | SumTree | O(1) sum(amount) WHERE field = X at the value-tree root |
Some("amount") | true | ProvableSumTree | SumTree | O(1) point sum plus O(log n) range sums via AggregateSumOnRange |
Compose orthogonally with the count flags. Combining countable and summable on the same index yields one of grovedb's combined-aggregation tree variants (CountSumTree, ProvableCountSumTree, or ProvableCountProvableSumTree) — one tree carries both metrics, queries on either axis read from the same merk root. See Range-Summable Indexes below for the storage layout, the ReferenceWithSumItem element type that makes per-document contributions land on the parent SumTree, and how range-summable composes with range-countable to produce PCPS trees backing the new AggregateCountAndSumOnRange combined-proof primitive.
For the conceptual treatment of sum trees and the full GetDocumentsSum query surface, see Document Sum Trees (paralleling Document Count Trees).
The schema accepts both the legacy boolean form (true → Countable, false → NotCountable) and the camelCase string form ("notCountable" / "countable" / "countableAllowingOffset"). For the full design rationale see Document Count Trees.
How Drive Builds the IndexLevel Trie
The flat list of Indexes declared on a document type is compiled, at contract-load time, into an IndexLevel trie (packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs):
#![allow(unused)] fn main() { pub struct IndexLevel { sub_index_levels: BTreeMap<String, IndexLevel>, has_index_with_type: Option<IndexLevelTypeInfo>, level_identifier: u64, } }
Each property name in any index becomes an edge in this trie; indexes that share a prefix share their initial path. An index "terminates" at a level by setting has_index_with_type = Some(...) — that's how the recursive insert / lookup code knows it's at the last property of a defined index, vs. just walking through a shared prefix.
Given two indexes:
byOwnerAndType = [ownerId, docType]byOwnerAndStatus = [ownerId, status]
the trie that gets built is:
flowchart TD
Root["(root)"]
Owner["<b>ownerId</b><br/><i>shared prefix</i>"]
DocType["<b>docType</b><br/>terminates <code>byOwnerAndType</code><br/><i>has_index_with_type = Some(...)</i>"]
Status["<b>status</b><br/>terminates <code>byOwnerAndStatus</code><br/><i>has_index_with_type = Some(...)</i>"]
Root --> Owner
Owner --> DocType
Owner --> Status
style DocType fill:#e0f7fa,stroke:#006064,color:#000
style Status fill:#e0f7fa,stroke:#006064,color:#000
The ownerId level is shared between both indexes. The docType and status levels each set has_index_with_type on themselves with their own unique / countable / null_searchable flags. A third index that also started with ownerId would attach further sub-levels under ownerId instead of duplicating it.
This trie shape directly mirrors the GroveDB path shape used at insert / query time.
GroveDB Layout
A document under contract C of type T with index property propA = vA, propB = vB lives at the grove path:
[ DataContractDocuments, contract_id, 1, doc_type_name,
propA_name, vA, propB_name, vB, 0 → <terminal element> ]
Let's break that down:
DataContractDocuments— root tree byte (u8constant) for "this is a document index, not a contract definition or identity record".contract_id— 32-byte contract identifier.1— separator distinguishing the document storage area from the contract definition area withincontract_id. Its siblings undercontract_idare0, the serialized contract (or, for a contract that keeps history, the history subtree whose key0references the latest revision), and, from protocol version 14,2, the contract's version number as a four-byte item thatgetDataContractsLatestVersionsreads and proves without the contract.doc_type_name— UTF-8 bytes of the document type ("person","contactRequest", etc.).propA_name, vA, propB_name, vB— alternating property key and serialized value, one pair per index property, in declaration order.0— the conventional "terminal slot" byte under each value level; it's where the actual reference (or sub-tree-of-references) lives.
The intermediate levels (propA_name, vA, propB_name, vB) are all NormalTrees. The terminal element at [0] varies — see the next section.
Concretely, suppose a widget-store contract has document type widget with a non-unique countable index byColor = [color], and three documents stored: A and B with color = "red", C with color = "blue". Then drive lays out:
flowchart TD
R["<b>DataContractDocuments</b><br/>(root tree byte)"]
CID["<b>widget-store-id</b><br/>(32 bytes)"]
Sep["<b>1</b><br/>(documents area)"]
DT["<b>'widget'</b><br/>(document type)"]
PK["primary-key tree<br/>(docs keyed by ID)<br/><i>not shown — own subtree</i>"]
Color["<b>'color'</b><br/>(index property name)"]
Red["<b>'red'</b><br/>(serialized value)"]
Blue["<b>'blue'</b>"]
RedT["<b>[0]: CountTree</b><br/>count = 2"]
BlueT["<b>[0]: CountTree</b><br/>count = 1"]
RA(["<b>doc_id_A</b><br/>Reference"])
RB(["<b>doc_id_B</b><br/>Reference"])
RC(["<b>doc_id_C</b><br/>Reference"])
R --> CID --> Sep --> DT
DT --> PK
DT --> Color
Color --> Red --> RedT
Color --> Blue --> BlueT
RedT --> RA
RedT --> RB
BlueT --> RC
classDef countTree fill:#fff4e5,stroke:#bf6900,color:#000
classDef reference fill:#e8f5e9,stroke:#1b5e20,color:#000
classDef placeholder fill:#f5f5f5,stroke:#888,color:#000
class RedT,BlueT countTree
class RA,RB,RC reference
class PK placeholder
Legend: rectangles are tree-type elements (intermediate NormalTrees holding sub-keys, terminal CountTrees holding per-doc refs); rounded green nodes are Reference elements (leaf pointers to documents). Amber highlight marks count-bearing trees specifically.
A query like where color = "red" walks the path down to the red value, opens the [0] CountTree, and either reads count_value (for GetDocumentsCount) or enumerates the inner references (for GetDocuments). Because the count is stored on the wrapping element, the count read is O(1) regardless of how many docs are inside.
Shared-Prefix Indexes
Now extend the same widget document type with a shape property and a second, compound, countable index byColorShape = [color, shape]. The IndexLevel trie is:
flowchart TD
Root["(root)"]
ColorLevel["<b>color</b><br/>terminates <code>byColor</code><br/><i>has_index_with_type = Some(...)</i><br/>+ sub-level <code>shape</code>"]
ShapeLevel["<b>shape</b><br/>terminates <code>byColorShape</code><br/><i>has_index_with_type = Some(...)</i>"]
Root --> ColorLevel --> ShapeLevel
style ColorLevel fill:#e0f7fa,stroke:#006064,color:#000
style ShapeLevel fill:#e0f7fa,stroke:#006064,color:#000
Importantly, color is a level that both terminates an index (byColor) and has a sub-level (shape) continuing past it. That dual role shows up directly in the on-disk path: at every [..., color, <value>] subtree, key [0] (the byColor terminal) and key 'shape' (the continuation into byColorShape) live as siblings.
With three documents — A: (red, circle), B: (red, square), C: (blue, square) — the layout is:
flowchart TD
DT["<b>'widget'</b><br/>(document type)"]
ColorKey["<b>'color'</b><br/>(index property)"]
Red["<b>'red'</b>"]
Blue["<b>'blue'</b>"]
%% byColor terminals (at the color-value level)
RedColorT["<b>[0]: CountTree</b><br/>count = 2<br/><i>byColor</i>"]
BlueColorT["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColor</i>"]
%% byColorShape continuation (sibling to [0] under each color value)
RedShape["<b>'shape'</b>"]
BlueShape["<b>'shape'</b>"]
RedCircle["<b>'circle'</b>"]
RedSquare["<b>'square'</b>"]
BlueSquare["<b>'square'</b>"]
%% byColorShape terminals
RCT["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
RST["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
BST["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
%% References — each indexed path stores its own reference, so docs appear
%% multiple times across the diagram (same key, same doc id, but a separate
%% Reference element under each terminal that matches the document).
RA1(["<b>doc_id_A</b><br/>Reference"])
RB1(["<b>doc_id_B</b><br/>Reference"])
RC1(["<b>doc_id_C</b><br/>Reference"])
RA2(["<b>doc_id_A</b><br/>Reference"])
RB2(["<b>doc_id_B</b><br/>Reference"])
RC2(["<b>doc_id_C</b><br/>Reference"])
DT --> ColorKey
ColorKey --> Red
ColorKey --> Blue
Red --> RedColorT
Red --> RedShape
Blue --> BlueColorT
Blue --> BlueShape
RedColorT --> RA1
RedColorT --> RB1
BlueColorT --> RC1
RedShape --> RedCircle
RedShape --> RedSquare
BlueShape --> BlueSquare
RedCircle --> RCT --> RA2
RedSquare --> RST --> RB2
BlueSquare --> BST --> RC2
classDef countTree fill:#fff4e5,stroke:#bf6900,color:#000
classDef reference fill:#e8f5e9,stroke:#1b5e20,color:#000
class RedColorT,BlueColorT,RCT,RST,BST countTree
class RA1,RB1,RC1,RA2,RB2,RC2 reference
Two things to notice:
[0]and the sub-property name ('shape') are siblings under each color value. The[0]count tree is thebyColorterminal at that color value; the'shape'subtree is the continuation thatbyColorShapewalks past for the next index property. Drive descends one or the other depending on which index covers the query.- The same document is stored as a separate
Referenceunder every index path that matches it. Doc A appears underbyColor[red]and underbyColorShape[red, circle]; doc B underbyColor[red]andbyColorShape[red, square]; doc C underbyColor[blue]andbyColorShape[blue, square]. That's why each of A, B, C shows up twice in the diagram — once per index that covers the document. Insert/delete touches all of them; queries walk only the one path their picker selected.
A query like where color = "red" resolves through the byColor terminal ([0] under red) — count = 2, O(1). A query like where color = "red" AND shape = "circle" resolves through byColorShape instead, taking the 'shape' sub-tree past red and reading the terminal under circle — count = 1, also O(1). Both queries are served by the same shared-prefix layout, just descending different branches at the red node.
Range-Countable Indexes
Status: design. Not yet implemented at the time of writing. Depends on a parallel grovedb change that adds
NonCounted<ElementType>element variants — element types that behave exactly like their counterparts except that their count value is not propagated to the parent count tree, and which are only insertable inside aCountTree/ProvableCountTree/CountSumTree/ProvableCountSumTree.
range_countable is a separate per-index property from countable. Where countable makes the count of docs at one specific value O(1), range_countable makes the count of docs between two values O(log n) — answering queries like "how many widgets have a color between red and tomato alphabetically" without enumerating every distinct color value.
Constraints
range_countable: truerequirescountableto beCountableorCountableAllowingOffset. It is additive to countability, not a replacement: range queries are useful only on indexes you'd already want to count by.- The combination is meaningful only on non-unique indexes (or unique indexes whose entries can be null-bearing), for the same reason
countableis mostly inert on unique-with-required-fields: a unique non-null terminal is a bareReference, with no tree to hang per-node counts off of. - Sibling sub-trees that share a prefix with a range-countable index — e.g., the
'shape'continuation whenbyColoris range-countable butbyColorShapeshares itscolorprefix — must useNonCounted<*>variants so their counts do not pollute the range-countable value tree's count.
Mechanism
Where today's countable upgrades only the terminal [0] element under each indexed value to a count tree, range_countable additionally upgrades two more levels:
| Level | Without range_countable | With range_countable |
|---|---|---|
Property-name tree (e.g. 'color') | NormalTree | ProvableCountTree |
Value tree (e.g. 'red', 'blue') | NormalTree | CountTree |
Terminal at [0] under each value | NormalTree / CountTree / ProvableCountTree (per countable) | unchanged — still driven by countable |
Sibling continuations inside the value tree (e.g. 'shape' for a compound index sharing the prefix) | NormalTree | NonCounted<NormalTree> |
The property-name tree is a ProvableCountTree rather than a plain CountTree because the merk-tree internal-node counts are exactly what makes range queries O(log n): walk the boundary path between the lower and upper bound, sum sub-counts at each internal node along the way. (See Document Count Trees for the underlying mechanic.)
The value trees become CountTrees because the property-name ProvableCountTree's aggregate is computed by summing each value tree's count_value. For that aggregate to mean "total docs at this property" rather than "number of distinct values", each value tree's count_value must equal "docs at this exact value" — which is only true if (a) the terminal [0] CountTree contributes its doc count, and (b) every sibling under the value tree (continuation sub-property names like 'shape', etc.) contributes zero rather than the default 1-per-Tree. That's what NonCounted<NormalTree> is for.
Layout
Same byColor + byColorShape example as before, with the same three documents (A: (red, circle), B: (red, square), C: (blue, square)), but now byColor.range_countable: true:
flowchart TD
DT["<b>'widget'</b><br/>(document type)<br/>NormalTree"]
ColorKey["<b>'color'</b><br/><b><i>ProvableCountTree</i></b><br/>count = 3"]
Red["<b>'red'</b><br/><b><i>CountTree</i></b><br/>count = 2"]
Blue["<b>'blue'</b><br/><b><i>CountTree</i></b><br/>count = 1"]
%% byColor terminals (unchanged shape — same as before)
RedColorT["<b>[0]: CountTree</b><br/>count = 2<br/><i>byColor terminal</i>"]
BlueColorT["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColor terminal</i>"]
%% byColorShape continuation — now NonCounted to avoid double-counting
RedShape["<b>'shape'</b><br/><b><i>NonCounted<NormalTree></i></b>"]
BlueShape["<b>'shape'</b><br/><b><i>NonCounted<NormalTree></i></b>"]
RedCircle["<b>'circle'</b><br/>NormalTree"]
RedSquare["<b>'square'</b><br/>NormalTree"]
BlueSquare["<b>'square'</b><br/>NormalTree"]
%% byColorShape terminals
RCT["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
RST["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
BST["<b>[0]: CountTree</b><br/>count = 1<br/><i>byColorShape</i>"]
%% References (one per matching index path, per the earlier section)
RA1(["<b>doc_id_A</b><br/>Reference"])
RB1(["<b>doc_id_B</b><br/>Reference"])
RC1(["<b>doc_id_C</b><br/>Reference"])
RA2(["<b>doc_id_A</b><br/>Reference"])
RB2(["<b>doc_id_B</b><br/>Reference"])
RC2(["<b>doc_id_C</b><br/>Reference"])
DT --> ColorKey
ColorKey --> Red
ColorKey --> Blue
Red --> RedColorT
Red --> RedShape
Blue --> BlueColorT
Blue --> BlueShape
RedColorT --> RA1
RedColorT --> RB1
BlueColorT --> RC1
RedShape --> RedCircle
RedShape --> RedSquare
BlueShape --> BlueSquare
RedCircle --> RCT --> RA2
RedSquare --> RST --> RB2
BlueSquare --> BST --> RC2
classDef provableCount fill:#ede7f6,stroke:#311b92,color:#000
classDef countTree fill:#fff4e5,stroke:#bf6900,color:#000
classDef nonCounted fill:#f3e5f5,stroke:#6a1b9a,color:#000,stroke-dasharray:5 5
classDef reference fill:#e8f5e9,stroke:#1b5e20,color:#000
class ColorKey provableCount
class Red,Blue,RedColorT,BlueColorT,RCT,RST,BST countTree
class RedShape,BlueShape nonCounted
class RA1,RB1,RC1,RA2,RB2,RC2 reference
Legend additions for this diagram: purple = ProvableCountTree; amber = CountTree; dashed lavender = NonCounted<*> (the new grovedb variants); rounded green = Reference.
Walking through how the counts add up:
'red'(CountTree, count=2) — its children are[0](CountTree, contributes itscount_value= 2) and'shape'(NonCounted<NormalTree>, contributes 0 — that's the whole point of the new variant). Aggregate = 2. ✓'blue'(CountTree, count=1) — same shape, 1 doc + 0. ✓'color'(ProvableCountTree, count=3) — its children are'red'(CountTree, contributes 2) and'blue'(CountTree, contributes 1). Aggregate = 3. The provable variant additionally stores per-internal-node counts inside its merk structure, which is what enables the range walk.
If 'shape' were a plain NormalTree instead of NonCounted<NormalTree>, it would contribute 1 to 'red' (every non-count-tree element contributes 1 by default — see Document Count Trees § How Counts Aggregate). Then 'red' would read as 3, 'blue' as 2, 'color' as 5 — a count of "docs + sub-property-trees", not "docs". The NonCounted<*> variant exists exactly to fix this.
Query — "count between two values"
With the layout above, a query like WHERE color BETWEEN 'red' AND 'tomato' resolves at the 'color' ProvableCountTree level:
- Walk the merk tree from
'color''s root, finding the boundary node between'red'(lower bound) and'tomato'(upper bound) — O(log distinct color values). - At each step, decide what to do with the off-boundary subtree using its pre-computed count: include its full
count_value(subtree fully inside the range), exclude (fully outside), or recurse (straddles the boundary). - Sum the contributions; the result is the count of all docs whose color falls in
[red, tomato].
No leaf-level enumeration of distinct color values, no enumeration of individual documents — the count is computed entirely from the tree's pre-aggregated structure.
Compound indexes
range_countable: true on a compound index applies at the index's terminating level (its last property). For byColorShape = [color, shape] with range_countable: true:
'shape'(the property-name tree under each color value) becomes aProvableCountTree.- Each
'circle'/'square'value tree becomes aCountTree. - Documents are referenced as
Element::Referenceleaves under thoseCountTrees, contributing 1 each to the count aggregate.
When the compound's leading prefix is also indexed by another range_countable index (e.g. byColor is also range_countable), sibling continuations under each color CountTree are wrapped with Element::NonCounted so a doc routed via byColorShape doesn't double-count under byColor's color aggregate. The walker (add_indices_for_index_level_for_contract_operations) threads a parent_value_tree_is_range_countable flag down the recursion to decide when to wrap, regardless of whether the inner tree is itself a ProvableCountTree, CountTree, or plain NormalTree.
End-to-end coverage in range_countable_index_e2e_tests (in packages/rs-drive/src/drive/contract/insert/insert_contract/v0/mod.rs) pins the storage layout against a real grovedb — including the count_tree_value_count_excludes_compound_continuation_via_non_counted test that proves NonCounted-wrapping is load-bearing for compound-index correctness.
Range-Summable Indexes
Status: live as of grovedb develop (PR #670 merged; head
e98bab5fas of this PR) (feat: add Element::ProvableCountProvableSumTree + dual-axis crossover proofs). Uses two grovedb element variants from that PR:Element::ReferenceWithSumItem(ReferencePathType, MaxReferenceHop, SumValue, Option<ElementFlags>)— a reference that also contributes ani64sum to its parent sum tree — andElement::NotSummed<*>/Element::NotCountedOrSummed<*>wrappers that opt out of sum (or both sum and count) propagation. The pure-sum side reuses the existingSumTree/ProvableSumTreevariants; the combined-axis case usesProvableCountProvableSumTree. Carrier-aggregate sum proofs work end-to-end viaGroveDb::verify_aggregate_sum_query_per_key— see Sum Index Examples Query 9 for the byte-counts.
range_summable is the sum-side counterpart of range_countable. Where countable / summable make point-lookup aggregates O(1), range_summable makes range-sum queries O(log n) — answering "what's the sum of price for widgets with color between red and tomato?" without enumerating every distinct color value or every individual document.
The shape is structurally parallel to range-countable, but the per-element contribution rules are inverted, and that asymmetry shapes the storage layout in a subtle but load-bearing way.
Constraints
range_summable: truerequiressummable: Some(<property>). Same additive relationship asrange_countable/countable.- The named property must be
type: integerand listed inrequiredon the document type. The DPP validator enforces this at contract-creation time — without it, a missing-or-null value at insert would leave the reference with no sum contribution and silently underflow the ancestor sums on delete. - The same property name must be used consistently across the doctype:
documents_summable(if set) and every per-indexsummablemust name the same property. Grovedb's sum trees aggregatei64per merk node without a per-tree property tag, so mixing properties would feed inconsistent contributions into the same merk hierarchy. - Combining
range_summablewithrange_countableon the same index promotes the property-name tree toProvableCountProvableSumTree(PCPS) rather than nesting two trees — both metrics live on the same merk root and can be queried atomically. See Combined: range-countable + range-summable below.
Mechanism
range_summable upgrades the same three levels range_countable does, with the sum analogues at each level:
| Level | Without range_summable | With range_summable |
|---|---|---|
Property-name tree (e.g. 'color') | NormalTree | ProvableSumTree |
Value tree (e.g. 'red', 'blue') | NormalTree | SumTree |
Terminal at [0] under each value | NormalTree / SumTree (per summable) | unchanged — still driven by summable |
| Sibling continuations (compound-index suffixes inside the value tree) | NormalTree | NormalTree — usually unwrapped (see below) |
The property-name tree is a ProvableSumTree rather than a plain SumTree for the same reason range_countable upgrades to ProvableCountTree: per-internal-node aggregated sums are what make range walks O(log n). Walk the boundary path between the lower and upper bound, sum sub-sums at each off-boundary internal node along the way. (See Document Sum Trees for the underlying mechanic.)
The value trees become SumTrees because the property-name ProvableSumTree's aggregate is computed by combining each value tree's sum_value. For that aggregate to mean "total <property> at this color" rather than "first-byte-of-some-i64-garbage", each value tree's sum_value must equal the documented sum — which requires the leaf elements stored under each value tree to be sum-bearing.
That's where the layout diverges from count.
The contribution asymmetry: count auto-propagates, sum requires sum-bearing elements
Count trees automatically count every child element. A NormalTree, an Item, a Reference — each contributes +1 to the parent's count_value by default. That's why range_countable needs NonCounted<*> wrappers everywhere: to suppress an aggregation that would otherwise happen.
Sum trees behave the opposite way. Only sum-bearing element variants — SumItem, ItemWithSumItem, ReferenceWithSumItem, and the sum-bearing tree variants themselves — contribute to a parent SumTree's running sum. Item, Reference, plain NormalTree, CountTree — all contribute 0 by default. That has two consequences:
- Per-document contributions don't appear automatically. A plain
Element::Referenceunder aSumTreedoes not propagate any sum. We need a different reference element —Element::ReferenceWithSumItem(path, max_hops, sum_value, flags)— that carries an expliciti64sum contribution (the document's value at thesummableproperty, frozen at insert time) alongside the usual reference-path bytes. Grovedb PR 670 adds this variant; Drive's index walker constructs it viamake_document_reference_with_sum_itemunder any index path withsummable.is_some(). - Sibling continuations usually don't need a wrapper. A
NormalTreecontinuation under a sum-bearing value tree contributes 0 by default — exactly what we want. NoNotSummedwrap required. The exception is when the continuation is itself sum-bearing (e.g. a deeper compound index that's alsorange_summable); in that case wrap the continuation inElement::NotSummed<*>to keep its sum from leaking into the outer index's aggregate. Compare withrange_countable, where every continuation needsNonCountedbecause every non-count-aware element auto-contributes 1.
Layout
Extend the widget contract with a numeric price property and promote both indexes to the sum surface:
{
"widget": {
"type": "object",
"documentsCountable": true, // unchanged — total widget count fast path
"properties": {
"brand": { "type": "string", "position": 0, "maxLength": 32 },
"color": { "type": "string", "position": 1, "maxLength": 32 },
"shape": { "type": "string", "position": 2, "maxLength": 32 },
"price": { "type": "integer", "position": 3, "minimum": 0 } // ← new, summable target
},
"required": ["brand", "color", "shape", "price"],
"indices": [
{
"name": "byColor",
"properties": [{ "color": "asc" }],
"summable": "price", // ← aggregate `price` per color
"rangeSummable": true // ← per-node sums, range-queryable
},
{
"name": "byColorShape",
"properties": [{ "color": "asc" }, { "shape": "asc" }],
"countable": "countable", // ← per-(color, shape) doc count at O(1)
"summable": "price", // ← aggregate `price` per (color, shape)
"rangeSummable": true // ← per-node sums on the `shape` terminator
}
],
"additionalProperties": false
}
}
Both indexes name the same sum property — summable: "price" in both. The DPP validator requires this: grovedb's sum trees aggregate i64 per merk node with no per-tree property tag, so a contract that mixed summable: "price" and summable: "fee" on the same doctype would feed inconsistent contributions into the same merk hierarchy. price is type: integer and listed in required — both also enforced at contract-creation time.
byColorShape combines countable (root-only doc count per (color, shape) pair) with summable + rangeSummable (per-node sums of price). Drive's dispatch table promotes this combination to ProvableCountProvableSumTree (PCPS) at the value-tree and [0] terminal levels — the only grovedb variant carrying per-node sums also carries per-node counts as a side effect, so the count side gets per-node tracking "for free" even though only the sum side was opted into provability. See DocumentTypePrimaryKeyTreeType::primary_key_tree_type's v1 dispatch table for the full mapping.
The two indexes share the color prefix exactly as the count examples did, so the same shared-prefix layout still applies. What changes is the element types at every level from 'color' downward — and the diagram below makes the compound case visible, because the 'shape' continuation under each color is now itself a sum-bearing tree (since byColorShape is rangeSummable) and needs Element::NotSummed<*>-wrapping to keep its aggregate from leaking into the outer byColor sum.
Document fixtures, three widgets: A: (brand_acme, red, circle, price=10), B: (brand_acme, red, square, price=20), C: (brand_acme, blue, square, price=30). The on-disk layout:
flowchart TD
DT["<b>'widget'</b><br/>(document type)<br/>NormalTree"]
ColorKey["<b>'color'</b><br/><b><i>ProvableSumTree</i></b><br/>sum = 60"]
Red["<b>'red'</b><br/><b><i>SumTree</i></b><br/>sum = 30"]
Blue["<b>'blue'</b><br/><b><i>SumTree</i></b><br/>sum = 30"]
%% byColor terminals — SumTree, refs carry per-doc sum contributions
RedColorT["<b>[0]: SumTree</b><br/>sum = 30<br/><i>byColor terminal</i>"]
BlueColorT["<b>[0]: SumTree</b><br/>sum = 30<br/><i>byColor terminal</i>"]
%% byColorShape continuation — now itself sum-bearing (rangeSummable),
%% so it must be NotSummed-wrapped to contribute 0 to the parent
%% byColor SumTree. The wrapped inner ProvableSumTree still works
%% normally for byColorShape queries that descend through it.
RedShape["<b>'shape'</b><br/><b><i>NotSummed<ProvableSumTree></i></b><br/>contributes 0 to red's sum<br/>inner sum = 30 for byColorShape queries<br/>(no per-node count: rangeCountable not set)"]
BlueShape["<b>'shape'</b><br/><b><i>NotSummed<ProvableSumTree></i></b><br/>contributes 0 to blue's sum<br/>inner sum = 30 for byColorShape queries<br/>(no per-node count: rangeCountable not set)"]
RedCircle["<b>'circle'</b><br/><b><i>PCPS</i></b><br/>count = 1, sum = 10"]
RedSquare["<b>'square'</b><br/><b><i>PCPS</i></b><br/>count = 1, sum = 20"]
BlueSquare["<b>'square'</b><br/><b><i>PCPS</i></b><br/>count = 1, sum = 30"]
%% byColorShape terminals — now PCPS (carry both per-node count
%% and per-node sum). References below contribute both axes.
RCT["<b>[0]: PCPS</b><br/>count = 1, sum = 10<br/><i>byColorShape</i>"]
RST["<b>[0]: PCPS</b><br/>count = 1, sum = 20<br/><i>byColorShape</i>"]
BST["<b>[0]: PCPS</b><br/>count = 1, sum = 30<br/><i>byColorShape</i>"]
%% References — every leaf is now ReferenceWithSumItem because both
%% indexes are summable. Each document is stored under both
%% byColor[color] and byColorShape[color, shape], so the same
%% per-doc price contribution lands twice in the diagram — once
%% per index that covers the document.
RA1(["<b>doc_id_A</b><br/>ReferenceWithSumItem<br/>sum=10"])
RB1(["<b>doc_id_B</b><br/>ReferenceWithSumItem<br/>sum=20"])
RC1(["<b>doc_id_C</b><br/>ReferenceWithSumItem<br/>sum=30"])
RA2(["<b>doc_id_A</b><br/>ReferenceWithSumItem<br/>sum=10"])
RB2(["<b>doc_id_B</b><br/>ReferenceWithSumItem<br/>sum=20"])
RC2(["<b>doc_id_C</b><br/>ReferenceWithSumItem<br/>sum=30"])
DT --> ColorKey
ColorKey --> Red
ColorKey --> Blue
Red --> RedColorT
Red --> RedShape
Blue --> BlueColorT
Blue --> BlueShape
RedColorT --> RA1
RedColorT --> RB1
BlueColorT --> RC1
RedShape --> RedCircle
RedShape --> RedSquare
BlueShape --> BlueSquare
RedCircle --> RCT --> RA2
RedSquare --> RST --> RB2
BlueSquare --> BST --> RC2
classDef provableSum fill:#e3f2fd,stroke:#0d47a1,color:#000
classDef sumTree fill:#e8eaf6,stroke:#1a237e,color:#000
classDef pcps fill:#ede7f6,stroke:#311b92,color:#000,stroke-width:2px
classDef notSummed fill:#fce4ec,stroke:#880e4f,color:#000,stroke-dasharray:5 5
classDef refSum fill:#c8e6c9,stroke:#1b5e20,color:#000,stroke-width:2px
class ColorKey provableSum
class Red,Blue,RedColorT,BlueColorT sumTree
class RedCircle,RedSquare,BlueSquare,RCT,RST,BST pcps
class RedShape,BlueShape notSummed
class RA1,RB1,RC1,RA2,RB2,RC2 refSum
Legend additions for this diagram: light blue = ProvableSumTree; indigo = SumTree; purple-outline = ProvableCountProvableSumTree (PCPS — per-node count and per-node sum); dashed pink = NotSummed<*> (contributes 0 to the parent's sum despite carrying its own internal aggregate); bold green = ReferenceWithSumItem.
Walking through how the aggregates layer:
byColor's view (read at the 'color' ProvableSumTree root, sum=60):
'red'(SumTree, sum=30) — children are[0](SumTree, contributes itssum_value= 30) and'shape'(NotSummed<ProvableSumTree>, contributes 0 — that's the whole point of the wrapper, even though its own internal aggregate is also 30 forbyColorShapequeries). Aggregate = 30. ✓'blue'(SumTree, sum=30) — same shape:[0]contributes 30,'shape'contributes 0. ✓'color'(ProvableSumTree, sum=60) — children are'red'(SumTree, 30) and'blue'(SumTree, 30). Aggregate = 60. The provable variant additionally stores per-internal-node sums inside its merk structure, which is what enables the range walk.
byColor is pure-sum (no countable flag) so the value trees here stay SumTree — there's no count aggregation at this layer.
byColorShape's view (descends through the NotSummed wrapper rather than reading it; the inner ProvableSumTree aggregates the PCPS value trees beneath):
'red' → 'shape'(ProvableSumTree, inner sum=30) — children are'circle'(PCPS, count=1 sum=10) and'square'(PCPS, count=1 sum=20). Inner aggregate = 30. Note that'shape'isProvableSumTreerather than PCPS: onlyrangeSummableis set onbyColorShape, notrangeCountable, so the property-name level ('shape') aggregates sums per-node but doesn't track per-node counts.'blue' → 'shape'(ProvableSumTree, inner sum=30) — single child'square'(PCPS, count=1 sum=30). Inner aggregate = 30.- Point lookup
SELECT COUNT(*), SUM(price) WHERE color = 'red' AND shape = 'circle'reads the PCPS value tree directly — both metrics in one element read (count=1, sum=10), no traversal. - Range query
SELECT SUM(price) WHERE color = 'red' AND shape BETWEEN 'a' AND 'z'walks the red'shape'ProvableSumTree's boundary and recovers sum=30 viaAggregateSumOnRangein O(log distinct shape values). Range-count over the same boundary isn't supported (would needrangeCountable: trueto promote'shape'to PCPS at the property-name level); range-count proofs overshapewould need to enumerate the value-tree count_values manually.
Why PCPS at the value level
PCPS is grovedb's only tree variant carrying per-node sums. When an index sets countable: "<tier>" + summable + rangeSummable, the dispatch table promotes the value tree to PCPS because there's no "ProvableSumCountTree" variant (per-node sum + root-only count) to land on. The count side gets per-node tracking "for free" — same storage cost as ProvableCountSumTree's count-half since PCPS commits the same per-node count metadata. See primary_key_tree_type.rs's v1 dispatch table for the full mapping.
byColor, by contrast, has only summable + rangeSummable (no countable), so its value trees stay SumTree — root-only sum, no count tracking, no upgrade. The two indexes living side by side on the same widget contract show both sides of the dispatch.
Why the NotSummed<*> wrap is still needed
The NotSummed<*> wrap is what keeps the two index views consistent. byColorShape's 'shape' subtree carries its own internal aggregate (30 at red, 30 at blue); byColor must not let those aggregates leak into its color sums. The wrapper makes 'shape' contribute exactly 0 to its parent 'red' / 'blue' SumTrees, so byColor reads from the [0] ref-bucket alone. Without the wrap, 'red' would read as 60 = 30 (refs) + 30 (the shape subtree's leaked aggregate), and any document covered by both indexes would be double-counted in byColor's aggregate.
Compare with the range-countable diagram above: there, the 'shape' continuations needed NonCounted<NormalTree> wrapping because a plain NormalTree auto-contributes +1 to a parent CountTree. Here the wrapper does conceptually the same job — suppress the would-be propagation — but for sum aggregation rather than count aggregation, and the wrapped variant is NotSummed<ProvableSumTree> because the continuation is itself sum-bearing (which is the only case where a sum wrapper is needed; plain NormalTree continuations naturally contribute 0 to a SumTree and don't need wrapping at all — see the asymmetry note above).
Query — "sum between two values"
A query like SELECT SUM(price) WHERE color BETWEEN 'red' AND 'tomato' resolves at the 'color' ProvableSumTree level via grovedb's AggregateSumOnRange primitive:
- Walk the merk tree from
'color''s root, finding the boundary node between'red'(lower bound) and'tomato'(upper bound) — O(log distinct color values). - At each step, decide what to do with the off-boundary subtree using its pre-computed sum: include its full
sum_value(subtree fully inside the range), exclude (fully outside), or recurse (straddles the boundary). - Sum the contributions; the result is the total
priceacross all docs whose color falls in[red, tomato].
No leaf-level enumeration of distinct color values, no enumeration of individual documents — the sum is computed entirely from the tree's pre-aggregated structure, exactly mirroring AggregateCountOnRange. The verifier counterpart is GroveDb::verify_aggregate_sum_query(proof, path_query, grove_version) -> Result<([u8; 32], i64), Error> returning (root_hash, aggregated_sum). (The sum is signed because grovedb's SumTree value type is i64. For tip-jar-style non-negative aggregations this stays ≥ 0 in practice; the verifier surfaces overflow into negative space as a distinct error rather than silently wrapping.)
Compound indexes
range_summable: true on a compound index applies at the index's terminating level (its last property). For an index byCategoryPrice = [category, price] with summable: "price" and range_summable: true:
'price'(the property-name tree under each category value) becomes aProvableSumTree.- Each price-value tree becomes a
SumTree. - Documents are stored as
Element::ReferenceWithSumItemleaves under thoseSumTrees, contributing theirpriceto the sum aggregate.
When the compound's leading prefix is also an index that's range_summable (e.g. a separate byCategory index that's also summable on price), sibling continuations under each category SumTree need Element::NotSummed<*>-wrapping iff the continuation is itself sum-bearing — otherwise the inner sum-tree's aggregate would leak into the outer index's value-tree sum, double-counting documents that route through both indexes. The walker (add_indices_for_index_level_for_contract_operations) threads the parent value tree's aggregation flags down the recursion to decide when to wrap.
Combined: range-countable + range-summable
Setting both range_countable: true AND range_summable: true on the same index doesn't produce two separate trees — grovedb PR 670 adds a dedicated ProvableCountProvableSumTree (PCPS) variant that commits both per-node counts AND per-node sums to every internal merk node. A single tree carries both metrics, and three range primitives become available against it:
AggregateCountOnRange— recovers just the countAggregateSumOnRange— recovers just the sumAggregateCountAndSumOnRange(PCPS-only, new in PR 670) — recovers BOTH from a single merk traversal, verified viaGroveDb::verify_aggregate_count_and_sum_query(...) -> Result<([u8; 32], u64, i64), Error>returning(root_hash, count, sum)
The combined primitive is strictly cheaper than running two separate range queries: one proof envelope, one merk walk, and both metrics atomically bound to the same root hash (so they can't drift relative to each other across a concurrent write).
The full dispatch table mapping (countable, range_countable, summable, range_summable) combinations to grovedb tree variants lives in DocumentTypePrimaryKeyTreeType::primary_key_tree_type's v1 arm; the index-walker dispatch in add_indices_for_index_level_for_contract_operations follows the same table at every recursion level.
End-to-end coverage for the sum surface lives in packages/rs-drive/benches/document_sum_worst_case.rs's tip-jar fixture (paralleling the count side's document_count_worst_case.rs widget bench), with the worked-example queries in Sum Index Examples.
Tree Type at the Terminal Level
The decision happens in add_reference_for_index_level_for_contract_operations/v0/mod.rs:
#![allow(unused)] fn main() { if !index_type.index_type.is_unique() || any_fields_null { // Non-unique branch: insert an empty tree at [0], then put // each document's reference inside that tree. The tree's variant // is governed by `countable`: // NotCountable → NormalTree // Countable → CountTree // CountableAllowingOffset → ProvableCountTree } else { // Unique branch: store a single Reference element at [0] directly. } }
So the matrix:
unique | any_fields_null | countable | What lives at [0] |
|---|---|---|---|
| false | (any) | NotCountable | empty NormalTree containing per-doc references |
| false | (any) | Countable | empty CountTree containing per-doc references |
| false | (any) | CountableAllowingOffset | empty ProvableCountTree containing per-doc references |
| true | false | (any) | bare Reference to the one matching document |
| true | true | NotCountable | empty NormalTree containing per-doc references |
| true | true | Countable | empty CountTree containing per-doc references |
| true | true | CountableAllowingOffset | empty ProvableCountTree containing per-doc references |
Note the last three rows: a unique index does go through the count-tree branch when any indexed field is null. That's why countable on a unique index is meaningful exactly when at least one of the indexed properties is optional in the schema.
Visualizing the three terminal shapes side by side:
flowchart TD
subgraph SA["Non-unique, countable"]
direction TB
A1["[..., color, 'red']"]
A2["<b>[0]: CountTree</b><br/>count = 2"]
A3(["<b>doc_id_A</b><br/>Reference"])
A4(["<b>doc_id_B</b><br/>Reference"])
A1 --> A2 --> A3
A2 --> A4
end
subgraph SB["Unique, all fields non-null"]
direction TB
B1["[..., email, 'alice@x']"]
B2(["<b>[0]: Reference</b><br/>→ doc_id_X"])
B1 --> B2
end
subgraph SC["Unique with null in path"]
direction TB
C1["[..., a, 'X', b, <empty>]"]
C2["<b>[0]: CountTree</b><br/>count = 1"]
C3(["<b>doc_id_W</b><br/>Reference"])
C1 --> C2 --> C3
end
classDef countTree fill:#fff4e5,stroke:#bf6900,color:#000
classDef reference fill:#e8f5e9,stroke:#1b5e20,color:#000
class A2,C2 countTree
class A3,A4,B2,C3 reference
Same convention as the layout diagram above: rectangles are tree-type elements, rounded green nodes are Reference elements. Same key ([0]) at the terminal in all three panels — what lives there is what differs. The middle case is the one that's "special" — a bare Reference directly at [0] instead of a sub-tree containing references — and it's specifically scoped to the unique-and-no-nulls scenario.
Null Handling
The any_fields_null and all_fields_null flags follow the path Drive descends during insertion: each sub-level gets its parent's flags combined with its own value (add_indices_for_index_level_for_contract_operations/v2/mod.rs):
#![allow(unused)] fn main() { let sub_level_any_fields_null = any_fields_null || document_index_field.is_empty(); let sub_level_all_fields_null = all_fields_null && document_index_field.is_empty(); }
any_fields_null becomes true the moment the walker hits any null/empty value on an index's path (first, middle, or last property) and stays true below it. all_fields_null only stays true if every value on the path so far is null. A sibling sub-level's value never enters them: under the index trie's shared levels, each index is judged by its own properties.
Before protocol version 14 the walkers updated the flags in place as they moved from one sibling sub-level to the next, so a sibling's missing value also marked every later sibling's path: a unique index could then hold its entry in the sub-tree shape with none of its own values missing, and a nullSearchable: false index could get an entry for a document missing all of its values because a sibling had one. The replace walker of those versions also chose a unique index's shape from all_fields_null, so a replace left an index with some of its values missing in the bare-Reference shape. From version 14, where one of those earlier writers could disagree with the rule, the delete walker and the replace walker read what is stored at [0] (and, for a nullSearchable: false index the rule skips, whether an entry is there) and remove or refresh the entry where it is. The read is unbilled bookkeeping, like the time-range walkers' removability reads, so a dry run and the applied operation bill the same; a type with a ttl, which exists only from version 14, skips it.
By the time the recursion reaches the terminal:
any_fields_null = falseand the index is unique → unique branch (bare Reference).any_fields_null = true(regardless of unique) → non-unique-style branch (sub-tree containing references).all_fields_null = trueANDnull_searchable = false→ the terminal call returns early without inserting anything; this document is not findable through this index.
This means different documents under the same unique index can land in different storage shapes depending on which of their indexed fields are null. A document with all required fields populated takes the bare-Reference shape; a document with a null in an optional indexed property takes the sub-tree shape, side by side under the same index.
A skipIfAbsent index is decided before any of this: a document missing a property of the index's skip set takes no part in it (document_takes_part_in_index), so the terminal call is not made for it at all. The walkers also build a level only when an index the document takes part in ends at or below it (level_reaches_entry), so a skipped index leaves no tree of its own behind, while a level it shares with a taking-part index is built as usual. The delete walkers apply the same predicates to the stored document. A replace compares the old and the new document's participation per index: it removes the old reference only if the old version took part and writes the new one (with the trees it hangs from) only if the new version does, so a replace that adds or drops a skip property moves the document into or out of the index and leaves exactly the tree a fresh insert of the new version would.
Insert Flow Summary
Putting it together, when Drive inserts a document into a contract C of type T:
add_indices_for_top_index_level_for_contract_operations— for each top-level entry in the document type's index trie (each first-property of any declared index), pushes the property name and the document's value for that property onto the path, computes the initialany_fields_null/all_fields_nullfor that single value, and recurses.add_indices_for_index_level_for_contract_operations(recursive) — for each sub-level of the trie, pushes the property name and value onto the path, derives the sub-level'sany_fields_null(OR) andall_fields_null(AND) from its parent's flags and its own value, and recurses. If the current level hashas_index_with_type = Some(...), it also calls into step 3 before recursing further (because an index can terminate at a non-leaf trie level when another index continues past it).add_reference_for_index_level_for_contract_operations— the terminal call. Decides between unique and non-unique-style storage using the matrix above; for the non-unique-style path it picks aNormalTree/CountTree/ProvableCountTreebased oncountable; finally inserts the document reference (or sub-tree containing it).
Deletion mirrors the same walk in reverse — see packages/rs-drive/src/drive/document/delete/.
Query Traversal
When a query arrives at drive-abci, the document-query construction path picks one of the document type's indexes that "covers" the query — i.e., whose property prefix matches the query's equality clauses, in order. The picker is in packages/rs-drive/src/query/mod.rs (look for fn construct_path_query and the index-selection helpers it calls). For count queries specifically there's a separate, count-tree-aware picker (drive_document_count_query/mod.rs) — see Document Count Trees for that path.
Once an index is picked, the query-engine builds a PathQuery whose path is exactly the prefix shape the insert code produced: [DataContractDocuments, contract_id, 1, doc_type, prop, value, prop, value, …]. GroveDB then walks the path in O(log n per level), reading the terminal sub-tree (or single reference) and returning matching documents.
A query whose where-clauses don't form a prefix of any index can't take this fast path and falls back to a full-scan plan — which dapi-grpc surfaces as an error in most cases, since unbounded scans are deliberately discouraged.
Choosing Index Settings
Quick checklist for contract authors:
- Don't index what you won't query. Each index costs storage on every insert/delete and counts against the per-document-type index limit (10 indexes per type currently).
- Order index properties from most-selective to least-selective. A
[country, city]index is more useful than[city, country]for queries likewhere country = "FR". unique: truewhen the platform should reject duplicates at the consensus layer. This is the right place for "this should be unique" invariants — don't enforce them application-side.countable: "countable"when you'll regularly callGetDocumentsCountwith==(orin) clauses on exactly this index's properties. Adds a constant-factor overhead on insert/delete; reads become O(1). Acountable: trueindex counts only queries whose where clauses match its properties exactly — partial-prefix queries are rejected withWhereClauseOnNonIndexedProperty, not falling through to a slow scan. Define a separate index per distinct count-query shape you want to support, or setdocumentsCountable: trueon the document type for unfiltered totals.countable: "countableAllowingOffset"when you'll also want offset / range queries on this index in a future release. Strictly more expensive than plain"countable"; only worth it if you need the capability.null_searchable: true(the default) is right for almost all cases. Set tofalseonly when documents with all-null indexed values shouldn't be findable through this index — typically a niche optimization to avoid a hot all-null prefix.
For specifically count-related concerns — primary-key-tree flags (documentsCountable / rangeCountable), the no-prove-vs-prove paths, and the operator restrictions — see the dedicated Document Count Trees chapter.
Document Count Trees
Counting the documents that match a query used to mean fetching them and calling .len(). From protocol v12 (Platform 3.1) onward, document types can opt into a different primary-key tree variant that maintains a running count inside the tree itself, turning count(*)-style queries into an O(1) lookup. This chapter explains the three tree variants, how a document type selects one, and the two query endpoints that expose the feature.
Why Count Trees Exist
The default primary-key tree for a document type is a NormalTree. To count the documents in it, Drive walks the subtree, deserializes every record, and returns the length of the resulting collection. That is fine for small types but becomes painful as soon as a UI needs "how many widgets are there?" on a contract with millions of widgets.
GroveDB has two count-aware tree variants. Both are provable — the count is committed to the Merkle root in each case — but they differ in where counts are stored inside the tree, and that controls which kinds of count queries can be answered without enumerating leaves:
CountTree— stores a singleu64count, at the root of the tree. The total document count is one read; any per-subtree count requires walking down to that subtree's root and reading its (separate) tree element.ProvableCountTree— stores au64count at every internal node, not just the root. Each node's count covers everything in the subtree below it, so range queries like "how many items between key A and key B?" or "how many items per value of an indexed property?" can be answered by walking the boundary nodes and summing their pre-computed counts, without touching any leaf.
GroveDB merk trees are binary — each internal node has exactly a left and a right child:
The dashed box is the wrapping Element (the "tree" in grovedb terms) and contains the root node of the merk tree. Both variants store the total count on the wrapping element — that's the O(1) field Drive reads for total counts. The difference is what's inside: in a CountTree the merk root and the rest of the tree don't carry the count, so only the wrapper has it. In a ProvableCountTree the count is also stored on the root node itself and on every internal merk-tree node, so it's committed into the merk root hash and provable per-subtree.
flowchart LR
subgraph CT ["CountTree"]
direction TB
subgraph CT_ELEM ["Tree element c=3"]
direction TB
A["root"]:::node
end
A --> B["·"]:::node
A --> C["x"]:::leaf
B --> D["x"]:::leaf
B --> E["x"]:::leaf
end
subgraph PCT ["ProvableCountTree"]
direction TB
subgraph PCT_ELEM ["Tree element c=3"]
direction TB
H["root c=3"]:::countnode
end
H --> I["c=2"]:::countnode
H --> J["c=1"]:::leaf
I --> K["c=1"]:::leaf
I --> L["c=1"]:::leaf
end
CT ~~~ PCT
classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
classDef countnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;
style CT_ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb
style PCT_ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb
In a CountTree, the only count-bearing node is the root. To compute "how many items per value of property P?" you'd have to navigate to each value-keyed subtree (a separate grovedb tree, not a child node of the binary structure above), read its root count, and pay for a separate proof per read — N reads for N distinct values. In a ProvableCountTree, every internal node along the binary path already carries the count of its left and right subtrees, so a range query like "items in [a, b]" or "items per value of P" walks only the boundary path and sums the pre-committed sub-counts in a single traversal and a single proof.
A document type opts in via two schema flags:
documentsCountable: true→ primary-key tree is aCountTree. Enables O(1) total-count for the document type; sufficient forGetDocumentsCountwith nowherefilter.rangeCountable: true→ primary-key tree is aProvableCountTree. ImpliesdocumentsCountable. The same flag is also accepted per-index, where it controls range-count storage layout (see below) and is required for anyGetDocumentsCountrequest that carries a range where-clause.
How a Document Type Picks Its Tree Variant
Selection lives in packages/rs-drive/src/drive/document/primary_key_tree_type.rs:
#![allow(unused)] fn main() { impl DocumentTypePrimaryKeyTreeType for DocumentTypeRef<'_> { fn primary_key_tree_type( &self, platform_version: &PlatformVersion, ) -> Result<TreeType, Error> { match platform_version .drive .methods .document .primary_key_tree_type { 0 => { if self.range_countable() { Ok(TreeType::ProvableCountTree) } else if self.documents_countable() { Ok(TreeType::CountTree) } else { Ok(TreeType::NormalTree) } } version => Err(Error::Drive(DriveError::UnknownVersionMismatch { method: "DocumentTypeRef::primary_key_tree_type".to_string(), known_versions: vec![0], received: version, })), } } } }
primary_key_tree_type() is the single source of truth — every Drive code path that needs to know which tree variant to read from or write to routes through this helper, including:
- Contract insert and update (to
CREATEthe right tree element when the document type is added). - Document insert / delete (to know how to update the count alongside the document).
- Cost estimation (so fees match the variant that will actually be used).
The contract insert/update paths use three thin Drive helpers parallel to the existing batch_insert_empty_tree / batch_insert_empty_sum_tree:
batch_insert_empty_tree— NormalTree.batch_insert_empty_count_tree— CountTree, used whendocuments_countable() && !range_countable().batch_insert_empty_provable_count_tree— ProvableCountTree, used whenrange_countable().
Each helper goes through LowLevelDriveOperation::for_known_path_key_empty_*_tree (or its _estimated_path_key_* cousin in cost-estimation paths), so the contract setup, document operations, and proof generation all see the same on-disk shape.
Storage-Layout Invariants
Because the tree variant is fixed at contract-creation time and baked into how the tree element is laid out on disk, two flags are immutable across a contract update:
- Changing
documents_countablefrom any state to any other state on avalidate_configupdate returnsDocumentTypeUpdateError. - Same for
range_countable.
Tests pinning these guards live in packages/rs-dpp/src/data_contract/document_type/methods/validate_update/v0/mod.rs. Don't relax them: if a NormalTree-backed document type were silently switched to CountTree mid-contract, every subsequent insert or delete would update a count value attached to a tree element that physically isn't a count tree, leading to grovedb element-shape errors at best and consensus drift at worst.
Counting Documents at Query Time
A single unified gRPC endpoint exposes the feature: GetDocumentsCount. The response shape varies by request mode (total / per-In-value / per-distinct-value-in-range / total-over-range), see Range Modes below. The wire-level shape makes that split explicit: on the no-proof path the response's CountResults carries an inner oneof variant { uint64 aggregate_count; CountEntries entries; } — total-count and range-without-distinct modes return aggregate_count (a single u64), per-In-value and per-distinct-value-in-range modes return entries (a list of CountEntry { optional bytes in_key; bytes key; uint64 count } where in_key is the prefix value for compound In + range shapes and absent for flat queries). The endpoint has two underlying paths (prove vs. no-prove); every mode — including return_distinct_counts_in_range = true — is valid on both paths. The prove path uses two different proof shapes depending on whether you want a single aggregate or per-distinct-value entries (see Prove (Client-Side Verify-Then-Aggregate or Aggregate-Count Proof) below).
No-Prove (Server-Side O(1) or O(log n))
When prove=false, drive-abci calls into DriveDocumentCountQuery (in packages/rs-drive/src/query/drive_document_count_query/mod.rs). The handler picks a path based on the where clauses:
Unfiltered total (no where clauses) on a documentsCountable: true document type (read_primary_key_count_tree):
The doctype's primary-key tree at [contract_doc, contract_id, 1, doctype, 0] is itself a CountTree. One grovedb read gives count_value — the total document count. O(1).
Equal/In only (execute_no_proof):
- Pick a
countable: trueindex whose properties exactly match the Equal/In where-clause fields — every index property has a matching clause, no orphan clauses, no uncovered properties. If no such index exists the request rejects withWhereClauseOnNonIndexedProperty(the strict-coverage contract; see "Index design" below). - Walk the tree from the root down to the terminal level, pushing
prop_nameandserialize_value_for_key(prop_name, value)at each step.Equalextends one path;Inclones the current path once per value in its array (a cartesian fork) and the per-branch counts are summed. - Read the
CountTreeelement at the resulting path and return itscount_value. O(1) per branch.
If the request carries an In clause, the response is the entries variant — one CountEntry per In value (the per-value split mode). Otherwise the response is the aggregate_count variant — a single u64.
Index design contract: a countable: true index counts exactly its declared properties. Want count(*) WHERE color = X? Define a [color] countable index. Want count(*) WHERE color = X AND shape = Y? Define a [color, shape] countable index. Want both? Define both. Partial coverage (e.g. color = X against a [color, shape] index) is rejected — define a more specific countable index, or set documentsCountable: true on the document type for unfiltered total counts. The prove path enforces the same contract, so prove=true and prove=false reject in the same situations with the same error.
Range (execute_range_count_no_proof):
- Pick a
range_countable: trueindex where the Equal/In clauses cover the prefix and the range operator hits the index's last property. - Build the path
[contract_doc, doctype, prefix..., range_prop_name]— pointing at the property-nameProvableCountTree. - Issue a grovedb path query with the converted range
QueryItem(>,>=,<,<=,Range,RangeInclusive,RangeAfter,RangeAfterTo,RangeAfterToInclusive) and walk the children whose keys lie inside the range. - Each child's
count_value_or_default()is the doc count at that property value. Either sum all per-value counts and return as theaggregate_countvariant (summed mode), or emit them as per-valueCountEntrys under theentriesvariant (distinct mode), then apply order / cursor / limit.
Prove (Client-Side Verify-Then-Aggregate or Aggregate-Count Proof)
When prove=true, the proof shape depends on whether the query carries a range clause.
With a range clause: the handler picks one of two prove sub-paths based on return_distinct_counts_in_range:
-
Aggregate (
return_distinct_counts_in_range = false, default): drive-abci builds a grovedbAggregateCountOnRangepath query against the property-nameProvableCountTree, andget_proved_path_queryproduces an aggregate-count proof. The client verifies viaGroveDb::verify_aggregate_count_queryand recovers(root_hash, count)directly — proof size is O(log n) regardless of how many keys match. No documents are ever materialized. -
Distinct (
return_distinct_counts_in_range = true): drive-abci builds a regular range path query (noAggregateCountOnRangewrapper) against the sameProvableCountTree. Because the leaf is aProvableCountTree, merk emits oneNode::KVCount(key, value, count)op per matched in-range key, with eachcountcryptographically committed to the merk root vianode_hash_with_count(kv_hash, l_hash, r_hash, count)— same forge-resistance as the aggregate path'sHashWithCountcollapse. The SDK'sdrive_proof_verifier::verify_distinct_count_proofruns the standard hash-chain check, then walks the proof's op stream to extract the counts as aBTreeMap<Vec<u8>, u64>. Trade-off vs. the aggregate path: proof size is O(distinct values matched) rather than O(log n), because each distinct in-range key emits its ownKVCountop instead of being collapsed into a boundary subtree. Acceptable for typical histograms (a few dozen distinct values in range); for "give me a single count" use the aggregate path instead.
Without a range clause (point-lookup with prove): two sub-paths based on the request shape.
-
Unfiltered total +
documentsCountable: true: drive-abci proves the doctype's primary-keyCountTreeelement at[contract_doc, contract_id, 1, doctype, 0]. One merk path proof; the SDK'sdrive_proof_verifier::verify_primary_key_count_tree_proofreadscount_valueoff the verified element. O(log n) bytes. -
Equal/In against a fully-covering
countable: trueindex: drive-abci proves oneElement::CountTreeper covered branch. Two sub-shapes:- Equal-only fully-covered → one element at
[..., last_field, last_value, 0]. Inat any index position (with any number of trailing Equals) → one element per In value, fetched via outer Query + a subquery whoseset_subquery_pathcarries the post-In Equal segments (zero of them when In is on the last property; one or more when In sits earlier in the index). The subquery'sKey([0])picks off the CountTree at[..., in_field, in_value, <trailing equals>, 0]for each matched In branch.
The In position rule for count queries is more permissive than the regular document query path's
Index::matchesrule (which restricts In to last-or-before-last because of a positional path-construction assumption — seeDriveDocumentQuery::get_non_primary_key_path_queryfor the layout that forces it). The count path doesn't have that constraint: there's no document-key terminator descent, noorder_byinterpretation, and nolimit/offsetsemantics — it's a pure CountTree-element lookup, soset_subquery_pathwith an arbitrary trailing tail works. Both no-proof and prove count executors route through a singlepoint_lookup_count_path_querybuilder (no-proof runs the path query viagrove.queryand sums the emittedCountTreeelements' counts; prove signs the same path query viaget_proved_path_query), so they accept the same query shapes by construction. The SDK'sdrive_proof_verifier::verify_point_lookup_count_proofverifies and extractscount_value_or_default()from each verified element. - Equal-only fully-covered → one element at
Both sub-paths share the proof shape: each CountTree element's count_value is cryptographically bound to the merk root via node_hash_with_count(kv_hash, l_hash, r_hash, count), same forge-resistance guarantee the range-distinct path relies on. Neither materializes documents or runs per-key bookkeeping client-side.
Proof size: O(k × log n) where k is the number of covered branches (1 for the documents_countable fast path and Equal-only fully-covered case; ≤ |In values| for Equal-prefix + In-on-last).
Symmetric rejection contract: prove count requires a countable: true index whose properties exactly match the where clauses — same requirement as the no-proof Total / PerInValue modes. Partial coverage (where the where clauses are a strict prefix of the index, or the index has uncovered properties) rejects with a WhereClauseOnNonIndexedProperty-class error pointing the caller at the index-design fix. The documents_countable: true fast path handles unfiltered total counts in O(log n) proof bytes when set on the document type. No silent fallback to materializing matching documents — that path doesn't exist anymore.
Implementation reference:
- Path query:
DriveDocumentCountQuery::point_lookup_count_path_query— shared by prover and verifier. - Server executor:
DriveDocumentCountQuery::execute_point_lookup_count_with_proof. - Verifier:
DriveDocumentCountQuery::verify_point_lookup_count_proof; SDK wrapperdrive_proof_verifier::verify_point_lookup_count_proofcomposes tenderdash signature verification on top.
Supported Where Operators
The no-prove fast path covers three operator shapes:
Equal(==) — single point lookup against the count tree at a fully-resolved index path. Picked byfind_countable_index_for_where_clauses.In(in) — cartesian fork. Each value in theInarray becomes its own index path; their counts are summed (or, for split counts, merged by split key). AnInclause withkvalues costskpoint lookups, not a tree walk. TheInclause also doubles as the per-value split signal in the unifiedGetDocumentsCountendpoint — at most oneInper request.- Range (
>,>=,<,<=,between*,startsWith) — walks the property-nameProvableCountTree's children whose keys lie inside the range, reading each childCountTree's count value. Picked byfind_range_countable_index_for_where_clauses; requires the index to haverange_countable: trueAND the range property to be the index's last property (the IndexLevel terminator).startsWith "p"becomes the half-open range[serialize("p"), serialize("p") with last byte +1)— the same byte-incremented encoding the normal docs path uses (seeconditions.rs'sStartsWitharm), valid for UTF-8 string keys since UTF-8 never contains0xFF.
Through the unified GetDocumentsCount request handler, range queries take a single range terminator clause plus a prefix of Equal clauses and/or one In clause. In on a prefix property exercises grovedb's native subquery primitive — each emitted entry then carries both the in_key (the In value for that fork) and the key (the terminator value within the range). Per-fork counts are NOT merged server-side — see No-Merge Compound Semantics below for rationale.
Range Modes
A range query in the unified endpoint produces one of two response shapes, controlled by return_distinct_counts_in_range:
return_distinct_counts_in_range = false(default) —CountResults.aggregate_countcarrying the sum of the per-valueCountTreecounts within the range. Use for "how many widgets have color in[red, tomato]?".return_distinct_counts_in_range = true—CountResults.entrieswith oneCountEntryper distinct property value within the range (key= serialized terminator value,count=CountTreecount for that value,in_key= the In-fork value for compound queries or absent for flat queries). Use for "show me a histogram of widgets by color in[red, tomato]".
No-Merge Compound Semantics
For compound queries (In on a prefix property + range on the terminator), the entries are returned unmerged — one CountEntry per emitted (in_key, key) pair. The server does NOT collapse them down to a flat histogram keyed only by key. This is a load-bearing design choice:
- Correctness under
limit. Pushing alimitinto grovedb's path query truncates the emitted elements before any merge could run. With cross-fork merging this can undercount the merged sums (e.g.brand IN (acme, contoso)+color > x+limit=1could returnacme/red, count=2and silently dropcontoso/red, count=3so the mergedredcount comes out as2instead of5). Without merge,limitand the user's "number of entries returned" mean the same thing. - Proof verification stays straightforward. A malicious server omitting one
Inbranch shows up as missing entries with thatin_keyrather than as a silent undercount in a merged total. The caller can detect "I asked for 3 In values but only got entries for 2" directly from the response shape. - No information loss. A caller who wanted the merged histogram can compute
result.fold(by=key, sum=count)client-side trivially. A caller who wanted per-(in_key, key)counts can't reverse a merged histogram.
The rs-sdk surfaces this via DocumentSplitCounts.0: Vec<VerifiedSplitCount>. Callers wanting the historical flat-map shape can call DocumentSplitCounts::into_flat_map() which sums across in_key forks.
Flat queries (no In on prefix) have in_key = None on every entry; for those callers the API behaves identically to the pre-no-merge shape.
Pagination
Distinct mode accepts pagination knobs:
| Field | Effect |
|---|---|
order_by | CBOR-encoded list of [field, "asc"|"desc"] clauses, same shape as GetDocumentsRequestV0.order_by. First clause's direction controls split-mode entry ordering; ascending (default) walks the range in BTreeMap natural order, descending reverses. Only meaningful in split modes (per-In-value, per-distinct-value-in-range, prove-distinct); total-count and aggregate-prove responses are scalar and have no entry ordering. |
limit | Truncate after min(requested, max_query_limit) entries; applied last (after order). Unset (None) is normalized to default_query_limit before the cap is applied — the server never walks an unbounded distinct-mode result set, even if the client omits the field. Clients that want a tight working-set should still set this explicitly. |
For pagination, clients narrow the underlying range itself rather than passing a cursor — page 2 is just color > <last-key-from-page-1> with the same limit. There's no cursor field on the request because a single-bytes cursor would be ambiguous for compound (In + range + distinct) queries whose natural sort is (in_key, key), and range narrowing has the same expressivity for the simple cases.
These knobs are ignored on summed mode (they have no defined meaning for a single aggregate).
Range Queries on the Prove Path
When prove = true and the query carries a range clause, the handler picks one of two prove sub-paths based on return_distinct_counts_in_range. The aggregate sub-path (default) builds a grovedb AggregateCountOnRange proof — verified via GroveDb::verify_aggregate_count_query, recovering (root_hash, count) without materializing any matching documents. Proof size is O(log n) regardless of how many documents match. The distinct sub-path (return_distinct_counts_in_range = true) builds a regular range proof against the property-name ProvableCountTree — the leaf merk emits per-(in_key, key) KVCount ops, each bound to the merk root via node_hash_with_count, and the SDK extracts them as a Vec<VerifiedSplitCount> (preserving the unmerged compound shape per No-Merge Compound Semantics). Distinct proof size is O(distinct (in_key, key) pairs matched) instead of the aggregate's O(log n) — pick the aggregate path when you want one number, the distinct path when you want a histogram.
In on a prefix property is supported on the distinct sub-path: grovedb's outer Query enumerates Key(in_value) entries at the In-bearing prop's property-name subtree, set_subquery_path carries any post-In Equal pairs + terminator name, and set_subquery is the range item. The aggregate sub-path still rejects In on prefix because AggregateCountOnRange is a single-range merk primitive that can't fork at the merk layer — for compound aggregates, callers use return_distinct_counts_in_range = true and reduce client-side via DocumentSplitCounts::into_flat_map.
A "desc" direction in the first order_by clause is supported on the distinct sub-path. The derived direction flows into grovedb's Query.left_to_right on both the outer In-keys Query and the inner range subquery, so descending iteration walks (in_key_desc, key_desc) tuples. The prover and verifier MUST agree on this direction — the path query bytes include it, and disagreement breaks merk-root recomputation. The SDK derives left_to_right from the first request.document_query.order_by_clauses direction, matching the server's derivation in drive_dispatcher, so the two stay in lockstep by construction. Combined with limit, descending order returns the LAST limit matched entries (the largest keys) rather than the first limit reversed — exactly what callers paginating from the end expect.
For point-lookup count proofs (no range clause), drive emits a CountTree element proof against the covering countable index — proof size is O(k × log n) where k is the number of covered branches, with no cap on the underlying document count. See the Prove path section above for the symmetric-rejection contract.
Range Queries and ProvableCountTree
Range count queries (>, <, between*) over an index with range_countable: true are answered in O(log n) by walking the property-name ProvableCountTree's boundary nodes. The proof path uses grovedb's AggregateCountOnRange, which lets clients verify a range count without ever materializing the underlying documents.
Offset-style queries ("the next 50 items starting after item 7") are a separate primitive that will likely build on the same
ProvableCountTreeshape. They are not exposed viaGetDocumentsCounttoday — pagination of distinct-mode entries is done by narrowing the range itself (e.g.color > <last-key-from-previous-page>), not by offsetting into the underlying documents.
Why Internal-Node Counts Make Range Counts O(log n)
In a sorted merk tree the keys partition into a left (smaller) and right (larger) subtree at every internal node. To answer a question like "how many items have a key strictly greater than 7?" you walk the boundary between "below 7" and "above 7" from the root down, and at each step you can decide what to do with the other subtree — the one not on the boundary path — based on a single read:
- If a subtree lies entirely above the cutoff, add its full count and don't descend into it.
- If it lies entirely below, ignore it (contributes 0) and don't descend.
- If it straddles the cutoff, recurse into it (it is then the next step on the boundary path).
On a ProvableCountTree every internal node carries the count of its left and right subtrees, so the "add the full count" step is a single O(1) read of the node we're already touching. The whole walk visits one node per tree level — O(log n) — and every visited node is on the boundary path. The total ends up as a sum of pre-committed sub-counts plus zero or one straddle leaf at the bottom.
Concretely, picture a ProvableCountTree of 8 items with sorted integer keys 1, 3, 5, 7, 9, 11, 13, 15 — three full levels of internal nodes plus a leaf row:
flowchart TB
R["root c=8"]:::countroot
R --> L1["c=4"]:::countnode
R --> R1["c=4"]:::countnode
L1 --> LL["c=2"]:::countnode
L1 --> LR["c=2"]:::countnode
R1 --> RL["c=2"]:::countnode
R1 --> RR["c=2"]:::countnode
LL --> x1["key=1, c=1"]:::leaf
LL --> x3["key=3, c=1"]:::leaf
LR --> x5["key=5, c=1"]:::leaf
LR --> x7["key=7, c=1"]:::leaf
RL --> x9["key=9, c=1"]:::leaf
RL --> x11["key=11, c=1"]:::leaf
RR --> x13["key=13, c=1"]:::leaf
RR --> x15["key=15, c=1"]:::leaf
classDef countroot fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef countnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;
For "give me the count of items with key > 6":
- root (c=8): 6 falls inside the left subtree (which holds 1–7). Read both children's sub-counts. Right subtree's keys are all > 6 → take its full
c=4and don't descend. Recurse into left. - left (c=4): 6 falls inside its right subtree (which holds 5,7). Read both children. Left-left's keys (1,3) are both ≤ 6 → contribute 0 and don't descend. Recurse into left-right.
- left-right (c=2): 6 splits this leaf-pair. Read both leaves. Key 5 ≤ 6 → contribute 0. Key 7 > 6 → contribute 1.
- Total = 4 (right of root) + 0 (left-left) + 0 (key=5) + 1 (key=7) = 5.
We visited 4 internal nodes on the boundary path (root → left → left-right → key=7) and read sub-counts off 3 siblings (right, left-left, key=5) without descending into them. Six of the eight items were never enumerated: their counts were summed straight out of the committed sub-count fields. The walk is O(log n) in tree depth regardless of how many items live under each skipped subtree.
Why This Is Provable
A merk proof of the same boundary walk includes:
- The boundary path from root to the leaf adjacent to the cutoff.
- The siblings of every node on the boundary path (so the verifier can recompute hashes up to the merk root).
Each sibling node, on a ProvableCountTree, ships its committed sub-count alongside its hash. The verifier walks the same logic the server did — "this sibling lies entirely above 7, add its c=… value" — and ends up with the same total without enumerating the sibling subtrees. Verification is also O(log n).
The same primitive answers any range query of the form [A, B]: walk to the cutoff at A, then to the cutoff at B, and combine sub-counts along the way. [A, ∞) and (-∞, B] are special cases.
Authoring a Contract That Uses Count Trees
There are two opt-in surfaces in the document meta-schema. They're independent and can be used together:
- Top-level flags on the document type control the primary-key tree variant — the tree that stores documents keyed by document ID. This is what
GetDocumentsCount(with no equality predicates) reads. - A per-index
countable: trueflag controls whether that specific index's tree carries counts. This is what enables the no-prove fast path for queries that filter by the index's leading equality columns.
Primary-Key Tree Flags
Set at the same level as type / properties / indices on a document type:
{
"widget": {
"type": "object",
"documentsCountable": true,
"properties": {
"name": { "type": "string", "position": 0, "maxLength": 64 },
"color": { "type": "string", "position": 1, "maxLength": 16 }
},
"additionalProperties": false
}
}
That contract gets a CountTree for the widget primary-key tree. GetDocumentsCount for widget with no where filter is now an O(1) lookup of the tree element's count value.
To opt into a ProvableCountTree for the primary-key tree instead — useful if you want range queries on the primary key in the future, or if you intend to use this document type behind range proof primitives — set rangeCountable: true at the document-type level. It implies documentsCountable, so you don't need both:
{
"widget": {
"type": "object",
"rangeCountable": true,
"properties": {
"name": { "type": "string", "position": 0, "maxLength": 64 },
"color": { "type": "string", "position": 1, "maxLength": 16 }
},
"additionalProperties": false
}
}
These two flags are immutable across a contract update. You pick the tree variant at contract creation; you can't switch to a different one later without creating a new document type. (See Storage-Layout Invariants above.)
Per-Index Countable Flag
Set on a single entry in the document type's indices array:
{
"widget": {
"type": "object",
"documentsCountable": true,
"properties": {
"name": { "type": "string", "position": 0, "maxLength": 64 },
"color": { "type": "string", "position": 1, "maxLength": 16 }
},
"indices": [
{
"name": "byColor",
"properties": [{ "color": "asc" }],
"countable": true
}
],
"additionalProperties": false
}
}
With byColor.countable: true the byColor index's tree carries counts, so GetDocumentsCount with where: [["color", "==", "red"]] reaches the count via that index in O(1). Without the flag, find_countable_index_for_where_clauses skips the index and the query rejects with WhereClauseOnNonIndexedProperty — there's no slow fallback, only fast counts on properly-indexed properties.
The countable field accepts three forms:
| JSON value | Tree variant | Capabilities |
|---|---|---|
false (or omitted, or "notCountable") | NormalTree | No count fast path |
true (or "countable") | CountTree | O(1) totals at the root |
"countableAllowingOffset" | ProvableCountTree | O(1) totals plus per-node counts that will enable future O(log n) range / offset queries on this index |
The boolean true / false form is kept for back-compat with contracts written before the enum form was introduced; new contracts should prefer the explicit string variants for clarity, especially "countableAllowingOffset" when range/offset queries are wanted.
A few notes about the index-level flag:
- Setting any countable variant increases storage cost — every insert and delete updates the index tree's count alongside the document.
"countableAllowingOffset"costs more than plain"countable"(every internal node carries count metadata, not just the root). Don't sprinkle it on every index; opt in for the ones you'll actually count by, and use the cheaper variant unless you specifically need the offset capability. - The flag is on the whole index, not per-property. The index handles
count(*)queries whose equalitywhereclauses cover the index's properties exactly — every index property has a matching==(orin) clause, and every clause's field appears in the index. A["color", "size"]countable index gives you O(1) counts forWHERE color = X AND size = Y— butWHERE color = Xalone is rejected withWhereClauseOnNonIndexedPropertybecause that index doesn't claim to count by color alone. If you want both single-column-by-color counts AND compound color+size counts, define both["color"]and["color", "size"]countable indexes (or just["color"]if size-filtered counts aren't a hot path). The picker is strict by design: each countable index represents a deliberate decision about which count queries the contract supports. - Index-level countable is independent of the primary-key flags. You can have
documentsCountable: trueon the document type ANDcountable: trueon a specific index — the first gives you fast totals, the second gives you fast filtered counts that match that index. countableon auniqueindex is mostly a no-op, but not always. A unique index stores its terminal as a bare reference at key[0]rather than wrapping it in a count tree, so for documents whose indexed fields are all non-null the flag has no storage effect — insertion bypasses the count-tree code entirely. It does still do meaningful work for null-bearing entries: when a document has any null value among the indexed properties, insertion takes the same count-tree branch a non-unique index uses (because uniqueness can't be enforced on null), and the count tree at that path aggregates them. Socountableon a unique index is worth setting when at least one of the indexed properties is optional in the schema and you expect null values; otherwise it's an inert flag. Counts on all-non-null exact matches still return correctly (1 if present, 0 if not) because the on-disk reference reads as count 1 via grovedb's default-aggregate semantics.
Choosing What to Set
| You want | Set |
|---|---|
Fast count(*) for the whole document type | documentsCountable: true on the document type |
O(1) filtered count: count(*) WHERE col = X | countable: true on an index whose properties are exactly ["col"]. A composite index whose leading column is col (e.g. ["col", "other"]) does NOT answer this query — partial coverage rejects with WhereClauseOnNonIndexedProperty. Define a separate ["col"] countable index if you want this count. |
Per-In-value sub-counts: one CountEntry per value in an In clause | countable: true on an index whose properties exactly match the query's == clauses plus the In field. The In field may sit at any position in the index — both the no-proof and prove count paths use set_subquery_path to descend through any trailing Equals after the In, which is strictly more permissive than the regular document query path's last-or-before-last rule. E.g. WHERE color IN [...] needs ["color"]; WHERE brand = X AND color IN [...] needs ["brand", "color"]; WHERE brand IN [...] AND model = X AND year = 2024 needs ["brand", "model", "year"] with In on brand (position 0 of 3). |
O(log n) range count: count(*) WHERE col BETWEEN A AND B | rangeCountable: true on an index whose last property is col and whose other properties cover any equality predicates as a prefix. Implies countable: true. |
Per-distinct-value range histogram: one CountEntry per distinct value in a range | Same rangeCountable: true index as above, plus return_distinct_counts_in_range = true on the request. Available on both prove and no-prove paths; the prove path returns a regular range proof against the property-name ProvableCountTree and the SDK extracts per-key counts from the proof's KVCount ops via drive_proof_verifier::verify_distinct_count_proof. |
Range count proof (prove = true + range clause) | Same rangeCountable: true index. The handler uses grovedb's AggregateCountOnRange proof primitive — proof is O(log n), no cap on matched docs. |
| Future offset-style range queries (not yet released — see above) | rangeCountable: true on the document type |
| Nothing count-aware (default) | Don't set any of these flags. Primary-key tree stays a NormalTree. |
A migration check from dapi-grpc server logic: every count query requires either documentsCountable: true (for unfiltered totals) or a countable: true / rangeCountable: true index whose properties exactly match the query's where-clause fields. No covering index → the call returns a clear InvalidArgument describing what the picker was looking for ("requires a range_countable: true (or summableOffCountIndex) index whose last property matches the range field" for range queries, "requires a countable index whose properties exactly match the where clause fields" for Equal/In queries). Pick your indexes deliberately at contract creation time — per-index countable: true / rangeCountable: true flags can't be added later (contract indexes are immutable post-creation).
Counts Are Public
Anyone can run a count query and verify its proof, so a countable index publishes everything its counts reveal. Encrypting a document's fields does not hide its existence or its indexed values. A countable index keyed first by a recipient and then by the document owner answers "who sent documents to this recipient, and how many" for every recipient. On the DashPay contactRequest type, a countable [toUserId, $ownerId] index would publish every user's inbound contacts. If a UI only needs a badge, a countable index on the recipient alone (["toUserId"]) reveals a total and no per-sender edges.
SDK Access at Three Layers
rs-sdk (native Rust)
Both shapes land on the standard Fetch trait against a single DocumentCountQuery:
#![allow(unused)] fn main() { use dash_sdk::platform::documents::document_count_query::DocumentCountQuery; use dash_sdk::platform::Fetch; use drive::query::{WhereClause, WhereOperator}; use drive_proof_verifier::{DocumentCount, DocumentSplitCounts}; // Total count: no In clause. let DocumentCount(count) = DocumentCount::fetch( &sdk, DocumentCountQuery::new(contract.clone(), "widget")?, ) .await? .expect("DocumentCount::fetch always returns a value on success"); // Split count: signal split by including an `In` clause whose field // is the split property. The In's values enumerate the keys to count. let split_query = DocumentCountQuery::new(contract, "widget")? .with_where(WhereClause { field: "color".to_string(), operator: WhereOperator::In, value: platform_value::Value::Array(vec![ "red".into(), "blue".into(), "green".into(), ]), }); let splits = DocumentSplitCounts::fetch(&sdk, split_query) .await? .expect("DocumentSplitCounts::fetch always returns a value on success"); // `splits` is `DocumentSplitCounts(Vec<SplitCountEntry>)` — for the // flat-histogram view, collapse via `splits.into_flat_map()`. }
DocumentCountQuery wraps an internal DocumentQuery (so it reuses where-clause / order-by / contract-id machinery) and exposes with_where(WhereClause) + with_order_by(OrderClause) builders. The SDK picks the request mode (total / per-In-value / total-range / per-distinct-range) from query shape — Equal/In/range operators in the where clauses — plus explicit request flags. return_distinct_counts_in_range = true (set via .with_distinct_counts_in_range(true)) selects per-distinct-range over the default total-range when a range clause is present; without it a range query returns a single sum.
wasm-sdk (browser)
Two methods on the WasmSdk JS class — one entry per [plain | withProofInfo] variant covers every count mode, because the underlying DocumentSplitCounts::fetch dispatches on the query shape:
sdk.getDocumentsCount(
query: DocumentsQuery,
): Promise<Map<string, bigint>>;
sdk.getDocumentsCountWithProofInfo(
query: DocumentsQuery,
): Promise<ProofMetadataResponseTyped<Map<string, bigint>>>;
Result shapes:
- No
where, or Equal-onlywhere— single map entry with the empty-string key carrying the total count. whereincludes anInclause — one entry per (deduped) In value, keyed by the hex-encoded canonical bytes of that value.whereincludes a range clause +returnDistinctCountsInRange: true— one entry per distinct property value in the range. For compoundIn + range + distinctqueries, entries are summed by terminatorkeyinto a flat map (callers needing the unmerged per-(in_key, key) view should use a richer binding).
Map keys are always hex-encoded bytes matching the canonical serialize_value_for_key encoding of each property value, so callers that need a typed key ("red", 42, etc.) need to hex-decode and interpret per the contract's index-property type. The hex-encoded shape matches the no-prove server response, so merging or comparing count maps from prove and no-prove paths needs no transformation.
rs-sdk-ffi (iOS / native bindings)
#![allow(unused)] fn main() { dash_sdk_document_count( sdk, data_contract, document_type, where_json, // null or JSON [{field, operator, value}] order_by_json, // null or JSON [{field, direction}] return_distinct_counts_in_range, // bool limit, // i64; -1 = server default, >= 0 = explicit cap ) -> JSON {"counts": {"<hex-key>": <u64>, ...}} }
Single FFI entry covers every count mode — the result is always {"counts": {...}} with hex-encoded keys. For total counts (no where/In, distinct flag off), the map carries a single entry with the empty-string key. where_json is the same JSON shape dash_sdk_document_search already accepts ([{field, operator, value}]), so iOS callers can reuse their where-clause encoding. order_by_json is optional and controls split-mode entry ordering only (per-In-value and per-distinct-value-in-range results); pass null for total counts and aggregate range counts where ordering has no defined meaning. The endpoint returns its result as a JSON-encoded C string allocated on the heap — caller frees it via the standard SDK string-free routine.
Count Index Examples
This chapter walks through a representative contract and shows what a count-query proof actually proves — both the path query the prover signs and the verified element the verifier extracts. Every example uses the same widget contract (the same one the count-query bench at packages/rs-drive/benches/document_count_worst_case.rs populates) so the proof bytes, verified elements, and diagrams can all be cross-referenced against the same data.
The chapter assumes you've read Document Count Trees — that chapter explains the three tree variants (NormalTree / CountTree / ProvableCountTree), what Element::NonCounted does, and how the schema's documentsCountable / rangeCountable flags select between them. Here we take that machinery as given and trace what each query sees.
The Widget Contract
The widget document type carries three properties (brand, color, serial), opts into total counts at the doctype level via documentsCountable: true, and declares three indexes covering the count-query surface:
{
"type": "object",
"documentsCountable": true,
"properties": {
"brand": { "type": "string", "position": 0, "maxLength": 32 },
"color": { "type": "string", "position": 1, "maxLength": 32 },
"serial": { "type": "integer", "position": 2 }
},
"required": ["brand", "color", "serial"],
"indices": [
{
"name": "byBrand",
"properties": [{ "brand": "asc" }],
"countable": "countable"
},
{
"name": "byColor",
"properties": [{ "color": "asc" }],
"countable": "countable",
"rangeCountable": true
},
{
"name": "byBrandColor",
"properties": [{ "brand": "asc" }, { "color": "asc" }],
"countable": "countable",
"rangeCountable": true
}
],
"additionalProperties": false
}
Three things to notice:
documentsCountable: trueat the document-type level upgrades the doctype's primary-key subtree (atwidget/[0]) fromNormalTreetoCountTree. The unfiltered total count is one read against this element'scount_value.byBrandiscountable: "countable"only. It doesn't opt intorangeCountable, sobrand > Xrange counts aren't supported. But every countable terminator's value tree is stored as aCountTreeregardless ofrangeCountable(seeadd_indices_for_index_level_for_contract_operations/v0/mod.rs), so point-lookup count proofs (e.g.brand == "X"orbrand IN [...]) get the same compact value-tree-direct shape on byBrand that they do on rangeCountable indexes.rangeCountableis strictly an opt-in forAggregateCountOnRangesupport — orthogonal to proof-size shape.byColorandbyBrandColorarerangeCountable: true. Their property-name subtrees (e.g.widget/color) are stored asProvableCountTreerather thanNormalTree, which is whatAggregateCountOnRangewalks forcolor > floorstyle queries.
The bench populates 100 000 documents under a deterministic schedule — row → (brand_(row % 100), color_(row / 100), serial=row). That gives exactly 1 000 docs per brand, exactly 100 docs per color, and exactly 1 doc per (brand, color) pair. Those numbers show up in every verified count below.
GroveDB Layout
The contract above produces this storage shape. Tree elements (the wrapping Element GroveDB stores under each key) are drawn as subgraphs; children inside each tree are merk-tree nodes. The doctype root and the per-property name subtrees are separate Element trees nested under the contract-documents prefix, just like every other index in Drive.
Diagram conventions: green nodes carry a count_value committed to the merk root; gray are regular subtrees; dashed boxes highlight Element::NonCounted wrappers (children that store data but contribute 0 to their parent CountTree's count).
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD --> PK["[0]: CountTree count=100000<br/>(documentsCountable primary key)"]:::countnode
WD --> BR["brand: NormalTree<br/>(byBrand property-name)"]:::node
WD --> CO["color: ProvableCountTree<br/>(byColor property-name)"]:::pctnode
BR --> B000["brand_000: CountTree count=1000"]:::countnode
BR --> B050["brand_050: CountTree count=1000"]:::countnode
BR --> BMore["... brand_001 ... brand_099<br/>(all CountTree count=1000)"]:::countnode
B050 --> B050_0["[0]: CountTree count=1000<br/>(byBrand refs)"]:::countnode
B050 --> B050_C["color: NonCounted(ProvableCountTree)<br/>(byBrandColor continuation, contributes 0)"]:::noncounted
B050_C --> B050_C_500["color_00000500: CountTree count=1<br/>(byBrandColor terminator)"]:::countnode
B050_C_500 --> B050_C_500_0["[0]: CountTree count=1<br/>(byBrandColor ref)"]:::countnode
CO --> C500["color_00000500: CountTree count=100<br/>(byColor terminator)"]:::countnode
CO --> CMore["... color_00000000 ... color_00000999"]:::countnode
C500 --> C500_0["[0]: CountTree count=100<br/>(byColor refs)"]:::countnode
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
classDef countnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
classDef pctnode fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef noncounted fill:#21262d,color:#c9d1d9,stroke:#fb8500,stroke-width:2px,stroke-dasharray: 6 4;
Three layout facts to internalize before reading the queries:
brand_050is aCountTreewithcount_value = 1000. That's true becausebyBrandis countable; the rule applies uniformly to every countability tier (seeadd_indices_for_index_level_for_contract_operations/v0/mod.rs). Thecolorcontinuation that branches off this value tree isNonCounted-wrapped so the parent's count equals exactly the 1 000 refs in[0].widget/coloris aProvableCountTree, not a regularNormalTree. The yellow class above marks that — each internal merk node carries its subtree's count, which is what makesAggregateCountOnRangea single-pass primitive.color_00000500is aCountTreewithcount_value = 100under either parent. The same element layout would result from a query againstbyColoror againstbyBrandColor's second level; the path that gets there differs, but the destination is structurally the same.
How To Read The Proofs
Every example below has four sections:
- Path query — the spec the prover hands GroveDB.
pathis the list of subtree segments to descend through (the proof carries merk-path bytes for each of these);query itemsis what to select once at the bottom;subquery items(when present) descends one more layer. - Verified element — what
GroveDB::verify_query(orverify_aggregate_count_queryfor the range primitive) returns after walking the proof bytes. Thecount_value_or_defaultfield on aCountTreeelement is what the count surface ultimately surfaces to the caller. - Proof display — the proof bytes, decoded via
bincodeinto the structuredGroveDBProofAST and rendered through itsDisplayimpl. This is the same view dash-evo-tool's Proof Log screen shows when its display mode is set to "JSON" — each layer is a separateLayerProofcarrying its merk-tree operations (Push/Parent/ChildoverHash/KVValueHash/KVHash) plus alower_layersmap naming the children to descend into. Wrapped in a collapsible block per example because the merk path through 4-5 grovedb layers makes for long output. - Diagram — the path the proof walks through the layout. Blue arrows trace the descent; the cyan node is the verified element; faded gray nodes show context.
All proof-size numbers come from running the bench against a 100 000-row fixture; see document_count_worst_case.rs's report_proof_sizes / display_proofs / report_group_by_matrix helpers. The proof bytes are reproducible — run the bench, grep [proof] from stderr, and you'll get the same hashes shown here.
Queries in this Chapter
| # | Query | Filter | Complexity | Avg time | Proof size |
|---|---|---|---|---|---|
| 1 | Unfiltered Total Count | (none — total at doctype level) | O(1) | 22.5 µs | 585 B |
| 2 | Equal on a Single Property (byBrand) | brand == "brand_050" | O(log B) | 35.7 µs | 1 041 B |
| 3 | Equal on a RangeCountable Property (byColor) | color == "color_00000500" | O(log C) | 54.0 µs | 1 327 B |
| 4 | Compound Equal-only (byBrandColor) | brand == "brand_050" AND color == "color_00000500" | O(log B + log C') | 71.4 µs | 1 911 B |
| 5 | In on byBrand | brand IN ["brand_000", "brand_001"] | O(k · log B) | 40.0 µs | 1 102 B |
| 6 | In on byColor (RangeCountable) | color IN ["color_00000000", "color_00000001"] | O(k · log C) | 61.9 µs | 1 381 B |
| 7 | Range Query (AggregateCountOnRange) | color > "color_00000500" | O(log C) | 69.2 µs | 2 072 B |
| 8 | Compound == + Range (byBrandColor) | brand == "brand_050" AND color > "color_00000500" | O(log B + log C') | 84.9 µs | 2 656 B |
Complexity variables. B = distinct brands in the byBrand merk-tree (≈ 100 in the fixture); C = distinct colors in the byColor merk-tree (≈ 1 000); C' = distinct colors per brand in byBrandColor's continuation (≈ 1 000 — every brand carries the full color namespace in this fixture); k = number of values in the IN clause (2 here). Notably absent: the total document count N (100 000 here). Count proofs read pre-committed count_values from CountTree merk roots — they never enumerate the underlying documents, so proof generation cost is polylog(distinct index values), independent of N. The grove-descent overhead (5–8 layers) is treated as a constant. The O() column captures shape only, not constants — for instance Q3's O(log C) is ~50% slower than Q2's O(log B) because in this fixture C ≈ 10 × B, and byColor's ProvableCountTree carries extra running-count metadata per merk node on top of that (37 merk ops in L6 vs 25 for byBrand — see Query 2 and Query 3's proof displays).
Avg time is the criterion-reported median of cargo bench --bench document_count_worst_case -- 'document_count_worst_case/query_' on a 100 000-row warmed fixture (no group_by — single-query latency on the prover side, including merk-proof construction and serialization). Each row reflects 10 samples × 67k–220k iterations per sample with 2 s warm-up and 5 s measurement; the median sits within ±2 % of the mean across reruns. For GROUP BY variants of these queries, see Count Index Group By Examples.
Each query has the same four sections (Path query, Verified element, Proof display, Diagram) plus a per-layer merk-tree diagram starting at Layer 5 (Layers 1–4 are byte-for-byte identical across every query — they're the root → @ → contract_id → 0x01 descent shown in full only on Query 1). The bottom of the chapter has an at-a-glance comparison summarizing the structural differences.
Query 1 — Unfiltered Total Count
select = COUNT
where = (empty)
prove = true
Path query (primary-key CountTree fast path; no index walk needed):
path: ["@", contract_id, 0x01, "widget"]
query items: [Key(0x00)]
Verified element:
path: ["@", contract_id, 0x01, "widget"]
key: 0x00
element: CountTree { count_value_or_default: 100000 }
Proof size: 585 B.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (4 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(KVValueHashFeatureTypeWithChildHash(0x00, CountTree(0000000000010000fffffffffffeffff00000000000000000000000000000000, 100000), HASH[85843d8e6353dd6caf52f659c454b4a1352f510daa965df594b27319abf1d8a1], BasicMerkNode, HASH[0e6a5047f0600cafc385ed52b516c1fbbaf4994aa50dfcbd1e824b4ad9f55fa1]))
1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
2: Parent
3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
4: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. The descent goes: top-level GroveDB root → @ (DataContractDocuments root tree) → contract id → 0x01 (documents storage prefix) → widget doctype → finally the Key(0x00) payload at the bottom, where CountTree(…, 100000) is the verified element with its count_value of 100 000 visible inside.
The descent stops at the doctype's primary-key tree — the green node at the top of the layout. Because documentsCountable: true upgraded that tree to a CountTree, the count is one O(1) read.
Diagram: per-layer merk-tree structure
Each LayerProof above is its own GroveDB sub-tree whose contents form a merk binary tree. The merk-proof operations (Push / Parent / Child over KVValueHash / KVHash / Hash nodes) describe exactly which nodes of each layer's binary tree the proof reveals — the queried key gets its full kv-hash exposed; opaque siblings only commit their subtree-hash so the verifier can re-hash up to the merk root.
Cyan = the verified target. Blue = a kv-hash that's also a queried-key on the descent path (its value = Tree(...) is the merk-root pointer for the next layer). Gray = opaque sibling subtrees committed by hash only.
flowchart TB
subgraph L1["Layer 1 — root GroveDB merk-tree"]
direction TB
L1_root["<b>@</b><br/>kv_hash=HASH[4a5a...]<br/>value: Tree(0x4ed2…)"]:::queried
L1_left["HASH[bd29...]<br/>(left subtree, opaque)"]:::sibling
L1_right["HASH[19c9...]<br/>(right subtree, opaque)"]:::sibling
L1_root --> L1_left
L1_root --> L1_right
end
subgraph L2["Layer 2 — @ subtree merk-tree (single key)"]
direction TB
L2_q["<b>contract_id 0x4ed2…</b><br/>kv_hash=HASH[5b90...]<br/>value: Tree(0x01)"]:::queried
end
subgraph L3["Layer 3 — contract_id subtree merk-tree"]
direction TB
L3_q["<b>0x01</b><br/>kv_hash=HASH[5d9a...]<br/>value: Tree(widget)"]:::queried
L3_left["HASH[49e7...]<br/>(left subtree, opaque)"]:::sibling
L3_q --> L3_left
end
subgraph L4["Layer 4 — 0x01 documents-prefix subtree (single key)"]
direction TB
L4_q["<b>widget</b><br/>kv_hash=HASH[6c50...]<br/>value: Tree(0x00/brand/color)"]:::queried
end
subgraph L5["Layer 5 — widget doctype merk-tree (TARGET layer)"]
direction TB
L5_root["KVHash[a29e...]<br/>(opaque internal kv: brand or color)"]:::sibling
L5_target["<b>0x00</b><br/>kv_hash=HASH[8584...]<br/>value: <b>CountTree count=100000</b>"]:::target
L5_right["HASH[6c36...]<br/>(right subtree, opaque)"]:::sibling
L5_root --> L5_target
L5_root --> L5_right
end
L1_root -. "value=Tree(merk_root[5b90…])" .-> L2_q
L2_q -. "value=Tree(merk_root[5d9a…])" .-> L3_q
L3_q -. "value=Tree(merk_root[6c50…])" .-> L4_q
L4_q -. "value=Tree(merk_root[a29e…])" .-> L5_root
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
A few things this diagram makes explicit that the prose can't:
- Each layer is its own merk binary tree, not a single graph. The 5
LayerProofblocks in the structured proof above each describe one of these binary trees. The hashes named in each block'sPush(Hash(HASH[…]))ops are this diagram's opaque siblings; thePush(KVValueHash(K, …))ops are this diagram's blue / cyan nodes. - The "descent" between layers is via a
value: Tree(…). When a queried key's value isTree(merk_root_hash), that hash IS the merk root of the next layer's binary tree. So Layer 1's@doesn't descend to Layer 2'scontract_iddirectly — it descends to Layer 2's merk root, which in this case happens to be the only node in Layer 2. - Single-key layers have a 1-node merk tree. Layers 2 and 4 contain exactly one entry (
@contains exactly one contract id;0x01contains exactly one doctype here), so their merk trees have no siblings to commit. - The merk root of a layer can be an opaque sibling, not the queried key. Layer 5's merk root is
KVHash[a29e...]— a key (brandorcolor, we can't tell from the proof) whose kv_hash is committed but whose value isn't revealed. The queried0x00is reached as a child of that opaque root. This is why the merk-tree structure matters: the prover sometimes has to commit one merk-tree-depth's worth of hashes to prove the queried key's position, even if the verifier only cares about the target's value.
Query 2 — Equal on a Single Property (byBrand)
select = COUNT
where = brand == "brand_050"
prove = true
Path query:
path: ["@", contract_id, 0x01, "widget", "brand"]
query items: [Key("brand_050")]
Verified element:
path: ["@", contract_id, 0x01, "widget", "brand"]
key: "brand_050"
element: CountTree { count_value_or_default: 1000 }
Proof size: 1 041 B.
Proof display:
Expand to see the structured proof (verbatim, 5 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
2: Parent
3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
4: Child)
lower_layers: {
brand => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[fb5eb23b3135d9c226e61f004ffb43abae104238d8a1ea7bc60e8ec6ba271596]))
1: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
2: Parent
3: Push(Hash(HASH[19ec5730af134e9ac980bbea92c2978212c8efe750a467ab54f073626e0ca2f5]))
4: Push(KVHash(HASH[87bc6e7e1e465b8dcdaf95db9957a455d6bd7c75976db122f33e592fe75f1e4a]))
5: Parent
6: Push(Hash(HASH[a0a354f2bb59b8169253aebabb52afcc3c59c4c60da203c8887abb679d747168]))
7: Push(KVHash(HASH[fc6b1d0237f8ff89b555e9a14480ae1c5b80d529a0f9fb5e681ea7ecd157d3da]))
8: Parent
9: Push(KVValueHashFeatureTypeWithChildHash(brand_050, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[53dbd6216cccdddf16f3eb0f849aed0c0cea987a718f5b43493abf0a14e83eb9], BasicMerkNode, HASH[4947457e230f87ce0f75a7f1502f64f24ee4d3e27eb5d2210680822a3b17afa4]))
10: Child
11: Push(KVHash(HASH[027ac8b1bc9788118b27c13d0b3c3bd3661ef6a89a775a6b6bf78aa7e6f8ed3d]))
12: Parent
13: Push(Hash(HASH[7a5dc3002e6cb6c92e54d554e5af85e9c2ba64ee9c5f80e6489075cc5f3f0d55]))
14: Child
15: Push(KVHash(HASH[3363630479f1abe6e003b1e1d50b5118e55ad2efb7a3f4b3b6df902bea72ac9a]))
16: Parent
17: Push(Hash(HASH[3857faef5ddb06e201f1e65cf42f15d6c9b0dc67e7f73eb182b520854e9bb648]))
18: Child
19: Child
20: Child
21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
22: Parent
23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
24: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
The bottom layer is the byBrand property-name tree; it has 100 distinct brand_NNN keys, so the merk path proves brand_050's position with 25 ops total (0–24). The verified payload is the inline KVValueHashFeatureTypeWithChildHash(brand_050, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), …) on op 9 — the 636f6c6f72 value slot is the ASCII bytes for "color" (the byBrandColor continuation pointer; NonCounted-wrapped at the storage layer so it contributes 0 to the parent count), and the 1000 is the doc count. The remaining 24 ops are the merk-binary boundary walk: each Push(Hash(…)) is an opaque subtree the proof commits but doesn't descend into, each Push(KVHash(…)) is an opaque internal sibling kv whose hash is committed, and each Parent / Child re-attaches them so the verifier can recompute the byBrand merk root.
brand_050 is itself a CountTree — every countable terminator's value tree carries the doc count directly, with sibling continuations wrapped NonCounted so they don't pollute the parent. The proof shape is the same as the rangeCountable case below, even though byBrand doesn't opt into rangeCountable: true. rangeCountable is the orthogonal opt-in for AggregateCountOnRange (Query 7), not for proof-size shape.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree"]:::path
BR ==> B050["brand_050: CountTree count=1000"]:::target
BR -.-> B000["brand_000"]:::faded
BR -.-> BMore["..."]:::faded
WD -.-> PK["[0]"]:::faded
WD -.-> CO["color"]:::faded
B050 -.-> B050_0["[0]: 1000 refs"]:::faded
B050 -.-> B050_C["color (NonCounted)"]:::faded
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Layers 1–4 are byte-for-byte identical to Query 1's diagram (root → @ → contract_id → 0x01). The descent diverges at Layer 5, where this query takes the brand branch (rather than 0x00) and descends one extra grove layer to land on the verified target.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `brand`)"]
direction TB
L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
L5_left["HASH[9862...]<br/>(left subtree, opaque)"]:::sibling
L5_right["HASH[6c36...]<br/>(right subtree, opaque)"]:::sibling
L5_q --> L5_left
L5_q --> L5_right
end
subgraph L6["Layer 6 — byBrand merk-tree (TARGET layer)"]
direction TB
L6_target["<b>brand_050</b><br/>kv_hash=HASH[53db...]<br/>value: <b>CountTree count=1000</b><br/>child_hash=HASH[4947...]"]:::target
L6_boundary["Boundary commitments (24 merk ops):<br/>6 KVHash opaque sibling brands<br/>+ 6 Hash subtree commitments<br/>(prove brand_050's position in byBrand's<br/>binary merk tree of ~100 brand entries)"]:::sibling
L6_target --> L6_boundary
end
L5_q -. "value=Tree(merk_root[byBrand])" .-> L6_target
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The boundary commitments at L6 are what scale linearly with the byBrand tree's depth — they bind brand_050 to its claimed position so the verifier can recompute byBrand's merk root. The verified target itself is just one KVValueHashFeatureTypeWithChildHash op whose count_value_or_default = 1000 is the answer.
Query 3 — Equal on a RangeCountable Property (byColor)
select = COUNT
where = color == "color_00000500"
prove = true
Path query:
path: ["@", contract_id, 0x01, "widget", "color"]
query items: [Key("color_00000500")]
Verified element:
path: ["@", contract_id, 0x01, "widget", "color"]
key: "color_00000500"
element: CountTree { count_value_or_default: 100 }
Proof size: 1 327 B.
Proof display:
Expand to see the structured proof (verbatim, 5 layers; note `KVHashCount` ops in the byColor `ProvableCountTree` layer) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
2: Parent
3: Push(KVValueHash(color, ProvableCountTree(636f6c6f725f3030303030353131, 100000), HASH[79569d595db75bbf2e9dca93a15c90b7eecf7b299632668ec410e2076d27f71c]))
4: Child)
lower_layers: {
color => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[864c8a53cdfc17560ea304fe40ae87570699a6920eae3dcb6075f71ca2d79b02]))
1: Push(KVHashCount(HASH[3684347a67ceedad2ff4a7fce6ae303086543c1f146f5865dfdc23612308c05b], 51100))
2: Parent
3: Push(Hash(HASH[56422e033fcffda5514eaef88096da995646207f3f5e349a6840003b4297098e]))
4: Push(KVHashCount(HASH[aa27604017cfc457ccd56aabeb4686a988b0b073d1c1c03a4fdf78164c31c8ea], 25500))
5: Parent
6: Push(Hash(HASH[09bcdaa37a5ae46f9059a7c026bf9cdf1c2d1ddecfcfe72fafe73f30abf2bccc]))
7: Push(KVHashCount(HASH[525df42449bd5e881d55f94c11be2b1c95cd112123864fc249e6c170ea026f5a], 12700))
8: Parent
9: Push(Hash(HASH[ffe58ba46b2d1f91b04e9c78185b474828f8ad165757847d9178020e55ad6c26]))
10: Push(KVHashCount(HASH[abbcbcef405f19e0a096a902993b3c76c77c59abdb8a3dcc95369e8c17b401c7], 6300))
11: Parent
12: Push(Hash(HASH[472879d66cf8e01e77bf4828d6a6f530a016cf7a99d712deb00c8fa5920b8495]))
13: Push(KVHashCount(HASH[3ac3896404268efc1bbfc9a2a8925adcc9eff7248fc7ca3aaec6f62587cdaffd], 3100))
14: Parent
15: Push(Hash(HASH[1c40306956f164e416e74a69ce0fff8c7ca152904ad47f44c6142c7822d3d2fb]))
16: Push(KVHashCount(HASH[494935a3d102495beb504953539d204ecd5b5ca8f5a03aa4a3cdbf16a3926335], 700))
17: Parent
18: Push(KVValueHashFeatureTypeWithChildHash(color_00000500, CountTree(00, 100, flags: [0, 0, 0]), HASH[47b0ade593a2e4e99e7d7363f5d1f692882007397f025226f19d097ca2f407fa], ProvableCountedMerkNode(100), HASH[4f7f13f56e087e7b19751c067671b75cda83156231cd3186f7c4172dccc8e97b]))
19: Push(KVHashCount(HASH[4866192fb6beda0888f828d7bbf008fa725a1141cf19ae3b1e9d245c6cb12c7c], 300))
20: Parent
21: Push(Hash(HASH[f56dd41a87f9b487ee9893c310a8bdd2fe70eb573e2e22e048cef7e3dec5fc1d]))
22: Child
23: Child
24: Push(KVHashCount(HASH[a646e152e4bfb609f5372833f5b8c001b4e523c3154f6fea43b154fe04c6e120], 1500))
25: Parent
26: Push(Hash(HASH[f434d46bb16f841310d2e120a259ad1aca2d679fd330ac0fd13d145c11a6b335]))
27: Child
28: Child
29: Child
30: Child
31: Child
32: Child
33: Push(KVHashCount(HASH[c32ae0189f148c2390791534ff4bc205fabb53a7c7d15f109a4354170045308c], 100000))
34: Parent
35: Push(Hash(HASH[1a1c99166d7b1e1eb9087404f3bfae82d749a3a7a763da654f48c5d314e21e76]))
36: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
This is the most interesting layer in the chapter. The byColor property-name tree (widget/color) is a ProvableCountTree, so every internal merk node carries its subtree's running count — visible here as KVHashCount(HASH[…], N) ops where N is the count contribution of that subtree (the values 51100, 25500, 12700, 6300, 3100, 700, 300, 1500, 100000 show up in the literal output above). The verified element (color_00000500) lands on op 18 in the middle of these KVHashCount ops, and the surrounding ops walk the binary boundary path so the prover can recompute the parent merk hash. For a point-lookup query like this, the ProvableCountTree machinery is overkill — it carries running counts the verifier doesn't need. Query 7 is where this pays off.
Structurally identical to Query 2 — only the property name and the count-tree depth differ. The intermediate widget/color tree is a ProvableCountTree here (vs NormalTree for byBrand), but the proof doesn't care about that: it descends through the property-name tree and surfaces the value-tree CountTree at the bottom. The ProvableCountTree upgrade matters for Query 7 (range aggregate), not for point lookup.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> CO["color: ProvableCountTree"]:::path
CO ==> C500["color_00000500: CountTree count=100"]:::target
CO -.-> C000["color_00000000"]:::faded
CO -.-> CMore["..."]:::faded
WD -.-> PK["[0]"]:::faded
WD -.-> BR["brand"]:::faded
C500 -.-> C500_0["[0]: 100 refs"]:::faded
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#d29922,color:#0d1117,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Layers 1–4 are byte-for-byte identical to Query 1. The L5 widget doctype merk-tree proof differs from Query 2: here color is the queried key (under an opaque KVHash[a29e...] root in the proof view), and the descent into Layer 6 enters a ProvableCountTree rather than a NormalTree.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `color`)"]
direction TB
L5_root["KVHash[a29e...]<br/>(opaque kv root)"]:::sibling
L5_left["HASH[9862...]<br/>(left subtree, opaque)"]:::sibling
L5_q["<b>color</b><br/>kv_hash=HASH[7956...]<br/>value: ProvableCountTree<br/>(descent into byColor)"]:::queried
L5_root --> L5_left
L5_root --> L5_q
end
subgraph L6["Layer 6 — byColor ProvableCountTree merk-tree (TARGET layer)"]
direction TB
L6_target["<b>color_00000500</b><br/>KVValueHashFeatureTypeWithChildHash:<br/>kv_hash=HASH[47b0...]<br/>value: <b>CountTree count=100</b><br/>feature: ProvableCountedMerkNode(100)<br/>child_hash=HASH[4f7f...]"]:::target
L6_boundary["Boundary commitments (36 merk ops):<br/>9 KVHashCount running totals carrying<br/>per-subtree counts<br/>(700, 1500, 3100, 6300, 12700,<br/>25500, 51100, 100000, 300)<br/>+ 9 Hash subtree commitments"]:::sibling
L6_target --> L6_boundary
end
L5_q -. "ProvableCountTree(merk_root[byColor])" .-> L6_target
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
Same structural shape as Query 2 (point lookup → CountTree), but the boundary commitments carry KVHashCount(HASH, N) ops instead of plain KVHash(HASH). The running totals are dead weight for this point lookup — the verifier never reads them — but they're the same commitments Query 7 will integrate over.
Query 4 — Compound Equal-only (byBrandColor)
select = COUNT
where = brand == "brand_050" AND color == "color_00000500"
prove = true
Path query:
path: ["@", contract_id, 0x01, "widget", "brand", "brand_050", "color"]
query items: [Key("color_00000500")]
Verified element:
path: ["@", contract_id, 0x01, "widget", "brand", "brand_050", "color"]
key: "color_00000500"
element: CountTree { count_value_or_default: 1 }
Proof size: 1 911 B.
Proof display:
Expand to see the structured proof (6 layers — the deepest descent in the chapter) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
2: Parent
3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
4: Child)
lower_layers: {
brand => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[fb5eb23b3135d9c226e61f004ffb43abae104238d8a1ea7bc60e8ec6ba271596]))
1: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
2: Parent
3: Push(Hash(HASH[19ec5730af134e9ac980bbea92c2978212c8efe750a467ab54f073626e0ca2f5]))
4: Push(KVHash(HASH[87bc6e7e1e465b8dcdaf95db9957a455d6bd7c75976db122f33e592fe75f1e4a]))
5: Parent
6: Push(Hash(HASH[a0a354f2bb59b8169253aebabb52afcc3c59c4c60da203c8887abb679d747168]))
7: Push(KVHash(HASH[fc6b1d0237f8ff89b555e9a14480ae1c5b80d529a0f9fb5e681ea7ecd157d3da]))
8: Parent
9: Push(KVValueHash(brand_050, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[53dbd6216cccdddf16f3eb0f849aed0c0cea987a718f5b43493abf0a14e83eb9]))
10: Child
11: Push(KVHash(HASH[027ac8b1bc9788118b27c13d0b3c3bd3661ef6a89a775a6b6bf78aa7e6f8ed3d]))
12: Parent
13: Push(Hash(HASH[7a5dc3002e6cb6c92e54d554e5af85e9c2ba64ee9c5f80e6489075cc5f3f0d55]))
14: Child
15: Push(KVHash(HASH[3363630479f1abe6e003b1e1d50b5118e55ad2efb7a3f4b3b6df902bea72ac9a]))
16: Parent
17: Push(Hash(HASH[3857faef5ddb06e201f1e65cf42f15d6c9b0dc67e7f73eb182b520854e9bb648]))
18: Child
19: Child
20: Child
21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
22: Parent
23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
24: Child)
lower_layers: {
brand_050 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[2190c6fcd140792fd12be66cd631f97475b9ab3f19417a26d94798115ee46160]))
1: Push(KVValueHash(color, NonCounted(ProvableCountTree(636f6c6f725f3030303030353131, 1000, flags: [0, 0, 0])), HASH[b1cedc48940faedea8b64bff8c8113344acdb1fd8eff37c567099b167b3c5861]))
2: Parent)
lower_layers: {
color => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[7e2704a94ce3e08ee1e8249a7272e1860251f66b9816581f6191010bc0f15dfe]))
1: Push(KVHashCount(HASH[ccfe3e95a84b22305b9815064d7d4e54b6ab0ca8efab26ca408391f2fad2b83e], 511))
2: Parent
3: Push(Hash(HASH[3dac20af894289bb36212087f48cfdd2c05713c5e134804638428a8d0ac8c905]))
4: Push(KVHashCount(HASH[1a3db8540380b26ead4be363cd35a4a0036ec9a92cf3c9527db540f0878ab168], 255))
5: Parent
6: Push(Hash(HASH[61f333ba1ad78624c009fa514ac69407a1438cb7d7807e9d5de1d75223137235]))
7: Push(KVHashCount(HASH[94e2ea0c17ffbf050dcba5e04928ae2ecf9fe21567e7fac5b93ad30040df8dfe], 127))
8: Parent
9: Push(Hash(HASH[a8571229cee7010a54ae9890a410edd246c079930d672fb4cdcd4a13c2bcc437]))
10: Push(KVHashCount(HASH[6b04a6eb8e698272ec0ff801c76dc9d65a1d47ef5ae1beb9747058cdda05e2d6], 63))
11: Parent
12: Push(Hash(HASH[8c12a68cebf211bbbd3937519662a7c3ed5bce92cf1e99869548c06a5639de15]))
13: Push(KVHashCount(HASH[9533ef417b8eed113b81bb2d7e56c81012d11835819a59769deebfc1a7e0eafd], 31))
14: Parent
15: Push(Hash(HASH[2f385d9fd5157a78a1cb1456050d3fb87f809e30b93a1963582edcedd31bfd0e]))
16: Push(KVHashCount(HASH[74ad467d4132703ae845149ce86de7c71d9c6fc7472e76e9bb1b81bc182abe53], 7))
17: Parent
18: Push(KVValueHashFeatureTypeWithChildHash(color_00000500, CountTree(00, 1, flags: [0, 0, 0]), HASH[7f1d988845d9c82b9d1146f2188b09bf704d31647ee2a26054e69ed897de3750], ProvableCountedMerkNode(1), HASH[078e3476060013c48bfc77330dc75d4fedc585469f581a66dc6b7b32f6d4d60a]))
19: Push(KVHashCount(HASH[48fc5c3cca2265eaeb5b86505f628660ffae9deee96cda8c26d7139f22ce0410], 3))
20: Parent
21: Push(Hash(HASH[c6e38bf64efcfd1d46d4ccd7937858870c7406f3abf31ca36148860d12c6b950]))
22: Child
23: Child
24: Push(KVHashCount(HASH[67ab38f74160b7c15e62a37fee3d0c193156061f3b013190ce2a154e4164c7b0], 15))
25: Parent
26: Push(Hash(HASH[49e4ecf80eead3552c93208b39c4fa9a5a3b64b7c63b385e53e47cb6e7bd8759]))
27: Child
28: Child
29: Child
30: Child
31: Child
32: Child
33: Push(KVHashCount(HASH[e735a44484a03e4f67ef4c79f370e2b2c4b0b98d942c5b1dca53039a031354b3], 1000))
34: Parent
35: Push(Hash(HASH[bf41c24632983b5858dcd20a04e0e0da6e7cacef58679e69e7859619dded444e]))
36: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
This is the deepest descent in the chapter. The path threads:
- The two intermediate GroveDB-wrapper layers (
@and0x01). - The widget doctype.
- The byBrand property-name tree.
- The byBrand value tree for
brand_050(visible in Query 2 already, here it's an intermediate stop withCountTree(636f6c6f72, 1000, …)— same element, same count). - The byBrandColor continuation (
color—NonCounted(ProvableCountTree)). - The byBrandColor terminator value tree, finally arriving at
color_00000500withCountTree(00, 1, …)— the bench's deterministic schedule gives exactly 1 doc per(brand, color)pair.
The proof descends through byBrandColor's prefix value tree (brand_050) into its continuation (color, the NonCounted-wrapped subtree shown earlier) and resolves at the terminator value tree color_00000500. The count is 1 because the bench's fixture has exactly one document per (brand, color) pair.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree"]:::path
BR ==> B050["brand_050: CountTree count=1000"]:::path
B050 ==> B050_C["color: NonCounted(ProvableCountTree)"]:::path
B050_C ==> B050_C_500["color_00000500: CountTree count=1"]:::target
B050_C -.-> Other["other colors"]:::faded
B050 -.-> B050_0["[0]: 1000 byBrand refs"]:::faded
BR -.-> Brands["other brands"]:::faded
WD -.-> CO["color"]:::faded
WD -.-> PK["[0]"]:::faded
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
linkStyle 3 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
This is the deepest descent in the chapter — four extra grove layers below the common Layers 1–4. Layer 5 enters brand (same as Query 2); Layer 6 lands on brand_050 but doesn't terminate (its CountTree has a continuation child); Layer 7 is brand_050's single-key sub-merk-tree carrying the byBrandColor continuation; Layer 8 is the byBrandColor terminator value tree where color_00000500 is the actual target.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree"]
direction TB
L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
L5_left["HASH[9862...]"]:::sibling
L5_right["HASH[6c36...]"]:::sibling
L5_q --> L5_left
L5_q --> L5_right
end
subgraph L6["Layer 6 — byBrand merk-tree (intermediate stop)"]
direction TB
L6_q["<b>brand_050</b><br/>kv_hash=HASH[53db...]<br/>value: CountTree count=1000<br/>(continuation child via lower_layer)"]:::queried
L6_boundary["Boundary commitments (24 merk ops):<br/>6 KVHash sibling brands + 6 Hash subtrees"]:::sibling
L6_q --> L6_boundary
end
subgraph L7["Layer 7 — brand_050's continuation merk-tree (single key)"]
direction TB
L7_q["<b>color</b><br/>kv_hash=HASH[b1ce...]<br/>value: NonCounted(ProvableCountTree)<br/>(descent into byBrandColor)"]:::queried
L7_left["HASH[2190...]"]:::sibling
L7_q --> L7_left
end
subgraph L8["Layer 8 — byBrandColor color sub-tree (TARGET layer)"]
direction TB
L8_target["<b>color_00000500</b><br/>KVValueHashFeatureTypeWithChildHash:<br/>kv_hash=HASH[7f1d...]<br/>value: <b>CountTree count=1</b><br/>feature: ProvableCountedMerkNode(1)<br/>child_hash=HASH[078e...]"]:::target
L8_boundary["Boundary commitments (36 merk ops):<br/>KVHashCount running totals<br/>(3, 15, 1000, ...) + Hash subtrees<br/>covering ~1000 colors under brand_050"]:::sibling
L8_target --> L8_boundary
end
L5_q -. "Tree(merk_root[byBrand])" .-> L6_q
L6_q -. "CountTree continuation (child_hash)" .-> L7_q
L7_q -. "NonCounted(ProvableCountTree)" .-> L8_target
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The two extra grove layers (L7 + L8) over Query 2's two-layer descent are what makes this the chapter's heaviest proof (1 911 B). The L7 layer is structurally trivial (one key) — its cost is the descent overhead, not the merk-tree boundary. L8 carries the same ProvableCountTree shape as Query 3's L6, but the contained key namespace is restricted to colors that co-occur with brand_050.
Query 5 — In on byBrand
select = COUNT
where = brand IN ["brand_000", "brand_001"]
prove = true
Path query:
path: ["@", contract_id, 0x01, "widget", "brand"]
query items: [Key("brand_000"), Key("brand_001")]
Verified elements (one per In value, returned in lex-asc order):
path: ["@", contract_id, 0x01, "widget", "brand"]
key: "brand_000"
element: CountTree { count_value_or_default: 1000 }
path: ["@", contract_id, 0x01, "widget", "brand"]
key: "brand_001"
element: CountTree { count_value_or_default: 1000 }
Proof size: 1 102 B.
Proof display:
Expand to see the structured proof (5 layers, two `KVValueHash` items at the byBrand level) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
2: Parent
3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
4: Child)
lower_layers: {
brand => {
LayerProof {
proof: Merk(
0: Push(KVValueHashFeatureTypeWithChildHash(brand_000, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[90ff6f6d9a3d901195982128130677243bfd27b75736206f3c8400966ef0d37b], BasicMerkNode, HASH[19b58883c492e746861db1e6ad07529a5a91cc8330af522682486db9346d6875]))
1: Push(KVValueHashFeatureTypeWithChildHash(brand_001, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[484ca11fb4ec8f479be1f78af903ce0c9d4fe630517579fb0172c2576d6b9652], BasicMerkNode, HASH[0bf12023f8e067c12db4cec1583909a0283878d6d909c76196736299750b5879]))
2: Parent
3: Push(Hash(HASH[8ca09dadc802a7efe03534ce4ad991b2f191f368878754a37b5e5c03d9498dab]))
4: Child
5: Push(KVHash(HASH[e5297b3ebe81c6435c29f712074da5f7c90265e12ed3d4f5af1f6d900e50c9f1]))
6: Parent
7: Push(Hash(HASH[50f373fd01dea89c992779764dff82cc7200b492be8f5cf3721627d5323bcbff]))
8: Child
9: Push(KVHash(HASH[cf78c9f1b1a1204bb2e437806f52c21e331392de3436388572bd1fa4bce1cdc7]))
10: Parent
11: Push(Hash(HASH[4a8dc186a95c8c4a1252fb51dbc407727f588eb5bdc8313c96f5c29889e13926]))
12: Child
13: Push(KVHash(HASH[d00ee7653e34e47d46004929b13ded33dff069ed9cc88342cecdf66a65fd8401]))
14: Parent
15: Push(Hash(HASH[7f1d17b9632f0bd440dacf5e841025482bc1d8145df3650301a95a5ee71ce8c8]))
16: Child
17: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
18: Parent
19: Push(Hash(HASH[eaef9fc530408393bc321409414814b290309a861f474a925a922250327affc6]))
20: Child
21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
22: Parent
23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
24: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
The two Push(KVValueHashFeatureTypeWithChildHash(brand_NNN, CountTree(…, 1000, …), …)) ops are the actual verified elements — both inlined in the byBrand layer's merk proof. They share the same parent path (@/.../widget/brand); the verifier-side verify_query returns both as siblings rather than descending one more layer per value (which is what the legacy Key([0]) shape would have forced for a normal-countable index, but no longer does — every countable terminator's value tree is a CountTree). The remaining 22 ops are the boundary-path hashes that prove brand_000 and brand_001 actually occupy the merk-tree positions claimed.
The outer query enumerates Key(in_value) items at the property-name subtree; each resolved element is itself a value-tree CountTree. No subquery is set — the In values' value trees are the count-bearing elements. The verifier reads the per-In value from grove_key (rather than from path[base_path_len], which is how it would for a trailing-Equal compound). The caller sums the two count_value_or_default reads (or surfaces them as per-group entries if group_by = ["brand"]).
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree"]:::path
BR ==> B000["brand_000: CountTree count=1000"]:::target
BR ==> B001["brand_001: CountTree count=1000"]:::target
BR -.-> BMore["brand_002 ... brand_099"]:::faded
B000 -.-> B000_0["[0]: 1000 refs"]:::faded
B001 -.-> B001_0["[0]: 1000 refs"]:::faded
WD -.-> PK["[0]"]:::faded
WD -.-> CO["color"]:::faded
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Same L5 shape as Query 2 (brand queried, two opaque sibling subtrees). At Layer 6 the proof inlines two KVValueHashFeatureTypeWithChildHash(brand_NNN, ...) ops at the byBrand layer — both verified elements share the same parent path, so no extra grove descent is needed.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree"]
direction TB
L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
L5_left["HASH[9862...]"]:::sibling
L5_right["HASH[6c36...]"]:::sibling
L5_q --> L5_left
L5_q --> L5_right
end
subgraph L6["Layer 6 — byBrand merk-tree (TWO TARGETS)"]
direction TB
L6_t1["<b>brand_001</b><br/>kv_hash=HASH[484c...]<br/>value: <b>CountTree count=1000</b><br/>child_hash=HASH[0bf1...]"]:::target
L6_t0["<b>brand_000</b><br/>kv_hash=HASH[90ff...]<br/>value: <b>CountTree count=1000</b><br/>child_hash=HASH[19b5...]"]:::target
L6_boundary["Boundary commitments (22 merk ops):<br/>7 KVHash opaque sibling brands<br/>+ 7 Hash subtree commitments<br/>(prove the two targets' adjacent positions)"]:::sibling
L6_t1 --> L6_t0
L6_t1 --> L6_boundary
end
L5_q -. "Tree(merk_root[byBrand])" .-> L6_t1
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
Marginally larger than Query 2 (1 102 B vs 1 041 B) — the extra cost is one extra KVValueHashFeatureTypeWithChildHash op plus one merge op. The boundary commitments shrink slightly because the two adjacent targets share part of the merk-tree path.
Query 6 — In on byColor (RangeCountable)
select = COUNT
where = color IN ["color_00000000", "color_00000001"]
prove = true
Path query:
path: ["@", contract_id, 0x01, "widget", "color"]
query items: [Key("color_00000000"), Key("color_00000001")]
Verified elements:
path: ["@", contract_id, 0x01, "widget", "color"]
key: "color_00000000"
element: CountTree { count_value_or_default: 100 }
path: ["@", contract_id, 0x01, "widget", "color"]
key: "color_00000001"
element: CountTree { count_value_or_default: 100 }
Proof size: 1 381 B.
Proof display:
Expand to see the structured proof (5 layers; bottom layer carries `KVHashCount` running totals from the `ProvableCountTree`) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
2: Parent
3: Push(KVValueHash(color, ProvableCountTree(636f6c6f725f3030303030353131, 100000), HASH[79569d595db75bbf2e9dca93a15c90b7eecf7b299632668ec410e2076d27f71c]))
4: Child)
lower_layers: {
color => {
LayerProof {
proof: Merk(
0: Push(KVValueHashFeatureTypeWithChildHash(color_00000000, CountTree(00, 100, flags: [0, 0, 0]), HASH[ce582ad80dab7f822798cbdcd4a7e2d454339ef5da50af688e31acb463f13bc6], ProvableCountedMerkNode(100), HASH[ad2891a5a377d25ef300546faaa2acef14cb3431490a86ed1d16d5fd69ec9e3f]))
1: Push(KVValueHashFeatureTypeWithChildHash(color_00000001, CountTree(00, 100, flags: [0, 0, 0]), HASH[c4024227f61350e128189bbfdb9cb3de893aef09626680a3d2336f991c1dbb14], ProvableCountedMerkNode(300), HASH[45e2452816d75b27baa9d1b8a82a251ce218d949d003bceb2e22ce1988312c4d]))
2: Parent
3: Push(Hash(HASH[cb34b6fa0bd36bf67c93768f3bdbadc7c5f4f143215222ff8bc8bbff5df0dc93]))
4: Child
5: Push(KVHashCount(HASH[2e045e449ad64fe27461182e3f335ee8fb65183c18a3fd3e4ff175c9e767b04b], 700))
6: Parent
7: Push(Hash(HASH[8d73c136c1428e6cca5c6579faeb12b9cc4e7094bdbdba383097d2d05032a414]))
8: Child
9: Push(KVHashCount(HASH[a9f7d6ebc19c3405af2ef32cbdf4f4ec0d4a96592bb5d389f9ab0462389c6fb5], 1500))
10: Parent
11: Push(Hash(HASH[e131726e58ca916c5d2c3fdff06be027b7bca567b45a1854b38774b7eb429b47]))
12: Child
13: Push(KVHashCount(HASH[c982b92207e31779affbc3c4495d175948ca647b9c15740c0cb0f6b7fede6d0d], 3100))
14: Parent
15: Push(Hash(HASH[c8f1d0d58823e8fb60dbd838fdd5b984c6940e1d4d4976473e8718a638dcd64c]))
16: Child
17: Push(KVHashCount(HASH[8dbbcf0d3b51cfa3f8c40c815b8904b650fd51e3bb55ae40f741f7341248ac38], 6300))
18: Parent
19: Push(Hash(HASH[28f1a2ab09b0920e50bdfd4d062412ba9c1d39d33579d485360e7a0941675a43]))
20: Child
21: Push(KVHashCount(HASH[6bf705340b0ff3872a4f692fc10bae0dd9e63fa2726bb3fd284fbfc273ef24af], 12700))
22: Parent
23: Push(Hash(HASH[8ebe73647e431636fe22547384c36bfd83d77a0e109dd3e3f5a69e691c860f9e]))
24: Child
25: Push(KVHashCount(HASH[b2fa1534ef346372a7d2df562fe4fc4938bd07bc72af5a147529478af878972d], 25500))
26: Parent
27: Push(Hash(HASH[db461b2f973111b65f34f31313ccff5530b24fa17bc7e5313d4794783336df24]))
28: Child
29: Push(KVHashCount(HASH[3684347a67ceedad2ff4a7fce6ae303086543c1f146f5865dfdc23612308c05b], 51100))
30: Parent
31: Push(Hash(HASH[e8c957f1d52f9ae3932f1f8d3e3d7f761569b52b29ffd7dc3f4c0c976405b3b4]))
32: Child
33: Push(KVHashCount(HASH[c32ae0189f148c2390791534ff4bc205fabb53a7c7d15f109a4354170045308c], 100000))
34: Parent
35: Push(Hash(HASH[1a1c99166d7b1e1eb9087404f3bfae82d749a3a7a763da654f48c5d314e21e76]))
36: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Same layer count as Query 5 (5 layers) and same inline-two-elements pattern at the bottom. The difference is in the merk-tree node type surrounding the verified elements: byColor's bottom layer is a ProvableCountTree, so each sibling's merk-path operation is a KVHashCount(HASH[...], N) (carrying the sibling's running count) rather than the plain KVHash(HASH[...]) you see in Query 5's byBrand layer. The boundary-proof ops here read like a histogram of the byColor tree's per-subtree counts (700, 1500, 3100, 6300, 12700, 25500, 51100, 100000) — that's the same information Query 7 will sum over directly without descending to any specific value tree.
Same query shape as Query 5 — outer Key-per-In-value, no subquery, per-In CountTrees resolved at the bottom. The difference vs Query 5 is the property-name tree above is a ProvableCountTree instead of NormalTree. That doesn't change the proof's structural shape, but it does mean a future color > X range query against this property has a fast path Query 5's brand doesn't.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> CO["color: ProvableCountTree"]:::path
CO ==> C000["color_00000000: CountTree count=100"]:::target
CO ==> C001["color_00000001: CountTree count=100"]:::target
CO -.-> CMore["color_00000002 ... color_00000999"]:::faded
C000 -.-> C000_0["[0]: 100 refs"]:::faded
C001 -.-> C001_0["[0]: 100 refs"]:::faded
WD -.-> PK["[0]"]:::faded
WD -.-> BR["brand"]:::faded
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#d29922,color:#0d1117,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Same L5 shape as Query 3 (color queried under an opaque kv root). L6 inlines two KVValueHashFeatureTypeWithChildHash targets in the byColor ProvableCountTree — the difference from Query 5's L6 is the surrounding boundary ops carry KVHashCount running totals instead of plain KVHash.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `color`)"]
direction TB
L5_root["KVHash[a29e...]<br/>(opaque kv root)"]:::sibling
L5_left["HASH[9862...]"]:::sibling
L5_q["<b>color</b><br/>kv_hash=HASH[7956...]<br/>value: ProvableCountTree (descent)"]:::queried
L5_root --> L5_left
L5_root --> L5_q
end
subgraph L6["Layer 6 — byColor ProvableCountTree merk-tree (TWO TARGETS)"]
direction TB
L6_t1["<b>color_00000001</b><br/>kv_hash=HASH[c402...]<br/>value: <b>CountTree count=100</b><br/>feature: ProvableCountedMerkNode(300)"]:::target
L6_t0["<b>color_00000000</b><br/>kv_hash=HASH[ce58...]<br/>value: <b>CountTree count=100</b><br/>feature: ProvableCountedMerkNode(100)"]:::target
L6_boundary["Boundary commitments (34 merk ops):<br/>8 KVHashCount running totals<br/>(700, 1500, 3100, 6300, 12700,<br/>25500, 51100, 100000)<br/>+ Hash subtree commitments"]:::sibling
L6_t1 --> L6_t0
L6_t1 --> L6_boundary
end
L5_q -. "ProvableCountTree(merk_root[byColor])" .-> L6_t1
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The two-In-values pattern carries over from Query 5; the per-subtree counts on the boundary KVHashCount ops are the same data Query 7 uses as the integrand for its range aggregate.
Query 7 — Range Query (AggregateCountOnRange)
select = COUNT
where = color > "color_00000500"
prove = true
Path query (different primitive — note the AggregateCountOnRange query item):
path: ["@", contract_id, 0x01, "widget", "color"]
query items: [AggregateCountOnRange([RangeAfter("color_00000500"..)])]
Verified payload (different verifier — GroveDb::verify_aggregate_count_query returns a single u64, not an element list):
root_hash: 0x62ee7348f4d28dd9d7cf86a6c725fa8276cfd446f6007a6000fb0e1dfefa6468
count: 49900
Proof size: 2 072 B.
Proof display:
Expand to see the structured proof (5 layers; bottom layer uses `HashWithCount` + `KVDigestCount` ops instead of `KVValueHash` — the AggregateCountOnRange-specific merk primitive) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
2: Parent
3: Push(KVValueHash(color, ProvableCountTree(636f6c6f725f3030303030353131, 100000), HASH[79569d595db75bbf2e9dca93a15c90b7eecf7b299632668ec410e2076d27f71c]))
4: Child)
lower_layers: {
color => {
LayerProof {
proof: Merk(
0: Push(HashWithCount(kv_hash=HASH[b2fa1534ef346372a7d2df562fe4fc4938bd07bc72af5a147529478af878972d], left=HASH[e8368be0ff72f87a2132f09d8d68d6dca140bc3c5b048d5f4f6fc8ab9b7bc554], right=HASH[db461b2f973111b65f34f31313ccff5530b24fa17bc7e5313d4794783336df24], count=25500))
1: Push(KVDigestCount(color_00000255, HASH[adfb158116847927badc07be9745a21be7e2660a8b75f8a310aba9025f91feec], 51100))
2: Parent
3: Push(HashWithCount(kv_hash=HASH[e4f3a5c9fdf17ccb2c7508839b2fdcfd4cd878ed1d59270929ac69ef63179402], left=HASH[848d5873de457b1be03c8c7d74733b92874f2071028fdd6d30e8ca16c18a9770], right=HASH[676f04d3603911ecd1e0d2d01c2691b173df672b40daf8a7730f73c50d39e07e], count=12700))
4: Push(KVDigestCount(color_00000383, HASH[14f48ee200148a9c4c673809297bdfb71e79fe9902b130e7842fbdb18c2e1a31], 25500))
5: Parent
6: Push(HashWithCount(kv_hash=HASH[42a257d9bc608c6b1a419f8e081b08df9056832c72e36b5dc07c4b724fb37578], left=HASH[65b3058c7b4d9bcfcf6022645f66bbaed9dbbdb74b7dbf367bbe2240263db767], right=HASH[315927383b45959aa67b32fb26b0b7c21baf6afbb1fcdc05e9c8c43a3c02b6c6], count=6300))
7: Push(KVDigestCount(color_00000447, HASH[dcbfdf897e1b1d83a55172b6fa463446cd5e016331ba075440f7f1091d02467b], 12700))
8: Parent
9: Push(HashWithCount(kv_hash=HASH[ada831d9c38535694323d9092ab9c42e39949c9d2e4567fafd084b0f5754b0d9], left=HASH[09229789d4fdf4baba7646d3bd12e6b77b83ce19f7f1c0918b60b3c1de5bd8ea], right=HASH[ce92f20c6b464d3ff4c95f8f1ee49149aacc50298eaed2c6a2849d588bd4a667], count=3100))
10: Push(KVDigestCount(color_00000479, HASH[1e6eb9e928e8bb229309db3a4a2c0f3041c63e90eb646061e4f5d82b1d65a1ac], 6300))
11: Parent
12: Push(HashWithCount(kv_hash=HASH[ae65499e6a1c105c878c418b09732df2dee29cf7db74c4b2e93b989710b449d0], left=HASH[94eac0807596d751092d12f27195dc72324f45999f4fc483688a9c15c554ecf3], right=HASH[fb4298cd62e8a90af17f9133fd4c106ec1da4b16be2954fc542af6ad0f6e316e], count=1500))
13: Push(KVDigestCount(color_00000495, HASH[cca12136fed93b88094fc80ceb5722b752860000478404c62f7862eb652e268f], 3100))
14: Parent
15: Push(HashWithCount(kv_hash=HASH[db1493f4f683045aa7604c6a06c0280fecb34b352503b148eab16e245938492f], left=HASH[50f064fdcdd8e0f3e1eb86b98dc8eb6f7a8df0b26037df202b21726a05edeb79], right=HASH[d6e96c2078316fcd74e62265173c2bb52a94ad4ef0bccac569557f675307b382], count=300))
16: Push(KVDigestCount(color_00000499, HASH[66e2d072be547070b1d433cb0f05f09ef508ec4d4f0702db4f49e71896ad91bc], 700))
17: Parent
18: Push(KVDigestCount(color_00000500, HASH[47b0ade593a2e4e99e7d7363f5d1f692882007397f025226f19d097ca2f407fa], 100))
19: Push(KVDigestCount(color_00000501, HASH[9146433eb6d43db2f109f5f7714146624bd646b27c7310f3c2cad7155eb7c741], 300))
20: Parent
21: Push(HashWithCount(kv_hash=HASH[bbac5fc7646d820e2912c1771333ebc83b1012619347aa04cce3c4ad13c11eea], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=100))
22: Child
23: Child
24: Push(KVDigestCount(color_00000503, HASH[66ea1280c29a6ea350e0c6695ab80430f5d3b5dc2df0f5a4d544a918d9fba29a], 1500))
25: Parent
26: Push(HashWithCount(kv_hash=HASH[4d7b5c895a6fb1e451ce85a522ecf18484fd1e406945cde8df9c75ec2152757e], left=HASH[6be0f9637caa5b6c09adb59618a8a90494e2f43a5e9948dc32d68af74528578a], right=HASH[ce1146de6de82a9767edf38a5cc11b5498e57023684acbe9e20bc3104ade94cf], count=700))
27: Child
28: Child
29: Child
30: Child
31: Child
32: Child
33: Push(KVDigestCount(color_00000511, HASH[c7fdd609ef67f184976b1bdfeb97245fdfcb33e53ff6841277def88f55bc9c41], 100000))
34: Parent
35: Push(HashWithCount(kv_hash=HASH[6abc81973aeff51137a002d32ac447e6b91ebf507e34a4a13ec9d1bed4516d23], left=HASH[99323fb716110f45836334025ec154fcc56193c11ee0811bdd86320c0f8164ed], right=HASH[33b9e5cbdf27883150262112aaefda71c0b725a58c3f929ad1ce1cdd3f90aacd], count=48800))
36: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
The bottom layer uses different merk-proof operations than every other query so far (Query 8 shares this shape one level deeper). AggregateCountOnRange doesn't return individual elements; it walks the boundary of the requested range (color > "color_00000500") over the ProvableCountTree's internal nodes and uses two specialized operations:
HashWithCount(kv_hash, left, right, count)— a boundary node that hides its full subtree behind a single hash + count. Thecountfield is the load-bearing piece: the verifier just sums these without descending. In this proof you can seecount=48800at the bottom-right boundary node (everything to the right of the range cut, plus anothercount=100000showing somewhere in the in-range path), and the prover walks the cut so eachHashWithCountcovers a different chunk of the range.KVDigestCount(key, kv_hash, count)— a boundary key inside the in-range region; the prover names the key so the verifier knows exactly where the cut is, but only commits the hash + count, not the value. Note the keys here climb monotonically (color_00000255 → 383 → 447 → 479 → 495 → 499 → 500 → 501 → 511); each one names a binary-tree boundary node on the path from the range start (color_00000500) to the right edge of the tree.
The final summed count: 49900 is what the verifier returns. There's no CountTree(…) element in this proof — the running totals inside HashWithCount / KVDigestCount are the proof's count surface, committed into the ProvableCountTree's merk root at insertion time.
Together with Query 8 (the compound brand == X AND color > Y variant), this is one of two queries in the chapter that use a different GroveDB primitive. Instead of resolving N specific keys, AggregateCountOnRange walks the boundary of the requested range over widget/color's ProvableCountTree and sums the per-node counts already committed inside that tree. The proof carries the boundary merk path and the running total; the verifier returns just the count.
The reason this works only with rangeCountable: true (Query 5's byBrand couldn't do the equivalent) is that widget/color is a ProvableCountTree — its internal merk nodes carry running counts. widget/brand is a plain NormalTree; it would have to enumerate every brand and sum their counts (which is what brand IN [...] does, but for an unbounded range that's not a feasible proof shape).
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> CO["color: ProvableCountTree<br/>(internal merk nodes carry running counts)"]:::target
CO -.-> C500["color_00000500 (boundary)"]:::faded
CO -.-> CMore["color_00000501 ... color_00000999<br/>(in range, summed via merk-node counts)"]:::faded
CO -.-> CBelow["color_00000000 ... color_00000499<br/>(below range, skipped)"]:::faded
WD -.-> PK["[0]"]:::faded
WD -.-> BR["brand"]:::faded
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Same L5 shape as Query 3 / Query 6 (color queried under an opaque kv root). L6 is fundamentally different from every other query: no individual elements are returned — the proof walks the boundary of the range color > "color_00000500" over the ProvableCountTree and sums per-node counts directly. The merk ops at L6 are HashWithCount (boundary subtree hashes carrying their full subtree count) and KVDigestCount (named boundary keys with hash + count, no value).
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `color`)"]
direction TB
L5_root["KVHash[a29e...]<br/>(opaque kv root)"]:::sibling
L5_left["HASH[9862...]"]:::sibling
L5_q["<b>color</b><br/>kv_hash=HASH[7956...]<br/>value: ProvableCountTree count=100000<br/>(descent into byColor)"]:::queried
L5_root --> L5_left
L5_root --> L5_q
end
subgraph L6["Layer 6 — byColor ProvableCountTree merk-tree (range-aggregate cut)"]
direction TB
L6_result["<b>Aggregate count = 49900</b><br/>(returned by verify_aggregate_count_query —<br/>a single u64, not an element list)"]:::target
L6_inrange["KVDigestCount ops along the in-range path:<br/>color_00000500 (count=100), color_00000501 (300),<br/>color_00000503 (1500), color_00000511 (100000)"]:::sibling
L6_boundary["HashWithCount boundary nodes covering<br/>chunks of the cut: counts<br/>(25500, 12700, 6300, 3100, 1500, 300, 100, 700, 48800)<br/>+ KVDigestCount path keys above the cut<br/>(color_00000255, 383, 447, 479, 495, 499)"]:::sibling
L6_result --> L6_inrange
L6_result --> L6_boundary
end
L5_q -. "ProvableCountTree(merk_root[byColor])" .-> L6_result
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The ProvableCountTree's value isn't to expose individual elements — it's to make the summation itself O(log n) instead of O(distinct values in range). The proof bytes are larger than Query 6's two-element point lookup (~2 KB vs ~1.4 KB) because the AggregateCountOnRange primitive has more structural overhead per result, but it scales to any size range in fixed proof bytes, where the point-lookup shape grows linearly with the number of resolved keys.
Query 8 — Compound Equal-plus-Range (byBrandColor)
select = COUNT
where = brand == "brand_050" AND color > "color_00000500"
prove = true
Path query (the prefix brand == X fixes one byBrandColor leg; the range walks the terminator's ProvableCountTree):
path: ["@", contract_id, 0x01, "widget", "brand", "brand_050", "color"]
query items: [AggregateCountOnRange([RangeAfter("color_00000500"..)])]
Verified payload (same primitive as Query 7 — GroveDb::verify_aggregate_count_query returns a single u64):
root_hash: 0x62ee7348f4d28dd9d7cf86a6c725fa8276cfd446f6007a6000fb0e1dfefa6468
count: 499
The bench's deterministic schedule gives every brand all 1 000 colors; the strict > cut at color_00000500 leaves color_00000501..color_00000999 = 499 colors paired with brand_050, each contributing exactly 1 document.
Proof size: 2 656 B.
Proof display:
Expand to see the structured proof (8 layers — same descent as Query 4 down to `brand_050`'s color subtree, then `HashWithCount` / `KVDigestCount` ops over the byBrandColor terminator) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
2: Parent
3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
4: Child)
lower_layers: {
brand => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[fb5eb23b3135d9c226e61f004ffb43abae104238d8a1ea7bc60e8ec6ba271596]))
1: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
2: Parent
3: Push(Hash(HASH[19ec5730af134e9ac980bbea92c2978212c8efe750a467ab54f073626e0ca2f5]))
4: Push(KVHash(HASH[87bc6e7e1e465b8dcdaf95db9957a455d6bd7c75976db122f33e592fe75f1e4a]))
5: Parent
6: Push(Hash(HASH[a0a354f2bb59b8169253aebabb52afcc3c59c4c60da203c8887abb679d747168]))
7: Push(KVHash(HASH[fc6b1d0237f8ff89b555e9a14480ae1c5b80d529a0f9fb5e681ea7ecd157d3da]))
8: Parent
9: Push(KVValueHash(brand_050, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[53dbd6216cccdddf16f3eb0f849aed0c0cea987a718f5b43493abf0a14e83eb9]))
10: Child
11: Push(KVHash(HASH[027ac8b1bc9788118b27c13d0b3c3bd3661ef6a89a775a6b6bf78aa7e6f8ed3d]))
12: Parent
13: Push(Hash(HASH[7a5dc3002e6cb6c92e54d554e5af85e9c2ba64ee9c5f80e6489075cc5f3f0d55]))
14: Child
15: Push(KVHash(HASH[3363630479f1abe6e003b1e1d50b5118e55ad2efb7a3f4b3b6df902bea72ac9a]))
16: Parent
17: Push(Hash(HASH[3857faef5ddb06e201f1e65cf42f15d6c9b0dc67e7f73eb182b520854e9bb648]))
18: Child
19: Child
20: Child
21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
22: Parent
23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
24: Child)
lower_layers: {
brand_050 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[2190c6fcd140792fd12be66cd631f97475b9ab3f19417a26d94798115ee46160]))
1: Push(KVValueHash(color, NonCounted(ProvableCountTree(636f6c6f725f3030303030353131, 1000, flags: [0, 0, 0])), HASH[b1cedc48940faedea8b64bff8c8113344acdb1fd8eff37c567099b167b3c5861]))
2: Parent)
lower_layers: {
color => {
LayerProof {
proof: Merk(
0: Push(HashWithCount(kv_hash=HASH[4f8d29f51f626326fa5a3d4aa210a07eddf53121888aa5788625ae774be9bc37], left=HASH[ec92140543f4bd56112e8eaf4cb9796b1986d56b0bf721d81fc7d6a699d16a50], right=HASH[1eb29f80ffaac4878420ecfc9337e6181c9e6fc30608fc5475cf0b808f51a31d], count=255))
1: Push(KVDigestCount(color_00000255, HASH[2ed4d50b30e917eceacb3356eb88057e490f9d98ebf6123d25535ff502d2da2b], 511))
2: Parent
3: Push(HashWithCount(kv_hash=HASH[80de09ce45f1c62d0532139ca67a93d88a293ce8139354e0e4751381346f64e7], left=HASH[32a8b4be78632242774668fd49f8db72d5f261d964f33ed9c0780ca99708ba20], right=HASH[af60a5ba4e39fa6326fd77dff0de304cc4cbac75c0702200a97d099f33617496], count=127))
4: Push(KVDigestCount(color_00000383, HASH[fe27bc251ea815fbd838146098daf0662fe214425a5befaec84c960dadbff89b], 255))
5: Parent
6: Push(HashWithCount(kv_hash=HASH[827791c9001bdf85512aed74a917156299ad6b1a50abe27e03939cb745000dfa], left=HASH[4b5363b3bd01883530360ef09c2b645c6f744988e24c17e432bc2c3321d41541], right=HASH[c444ec932284bb1bae3b45f3f54ed9f3922fd85aeecea01dd10341b889c41137], count=63))
7: Push(KVDigestCount(color_00000447, HASH[e7d9bb66af76a1b8600a9fb5d904f54825b61fb5e8004cdbfb4f42134455953a], 127))
8: Parent
9: Push(HashWithCount(kv_hash=HASH[11a374adf740d562dde32325c07b28949981033d310beafc1d90a3d44fb0bd6a], left=HASH[826dab51d3fd831414ae5344e837343104f0212e1c5ca57014951beba53f89d0], right=HASH[02ebfd24b8c7fd10b74bda8856344c4fd7287ce31ebdd6e9676c9a1a6e5943cc], count=31))
10: Push(KVDigestCount(color_00000479, HASH[6151f4f40176302ed6a27f77fd687bbae015a09ca80ad4af6f80e7c29e8a3595], 63))
11: Parent
12: Push(HashWithCount(kv_hash=HASH[b93259768b6500a9b757c4a90e981f0e3a8a848b275b862f16f5b242310cb65a], left=HASH[b1f724e4b2546d1d92059a72076868112b5b6187b6d227fa834d3ff3579f7b8c], right=HASH[fd1765117ce2a3f6deca713039d81726b05eecab4119e223753dee5fd989d610], count=15))
13: Push(KVDigestCount(color_00000495, HASH[034b88a8dfaf46db8b679fd72d342643d64b6937c44b06f983e5dbebd6f3b69b], 31))
14: Parent
15: Push(HashWithCount(kv_hash=HASH[bd58344e0fbbca9dee08997443550d1630adc59696701fb1f99c5a7e1fdb855a], left=HASH[22ef7d33de4e1d93a27009c5a3ae849ac8713c84dbac046dc615170a6b0e89a8], right=HASH[fbc94ef6e1255b8f0fffdb496258eb03a0b54c29d9f074830159d78a86e05621], count=3))
16: Push(KVDigestCount(color_00000499, HASH[12672ddf0e18d172679f7ebf0ba5f6976b337066e0373ffdac8c176a6a160dcc], 7))
17: Parent
18: Push(KVDigestCount(color_00000500, HASH[7f1d988845d9c82b9d1146f2188b09bf704d31647ee2a26054e69ed897de3750], 1))
19: Push(KVDigestCount(color_00000501, HASH[f0a8f993f517cee96055d69e48dfe51e70fe303424885194d3b7e71924af5df7], 3))
20: Parent
21: Push(HashWithCount(kv_hash=HASH[3b75b6239307e1a00f8596386421e623e365d4adc8451dae07cc3bcf589efc44], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=1))
22: Child
23: Child
24: Push(KVDigestCount(color_00000503, HASH[aba3bbc16aa5a2413fd60261c5efe4d42c97f0f4b82fcc8e74af8562cc2fdfed], 15))
25: Parent
26: Push(HashWithCount(kv_hash=HASH[64d94410c9ae982091bff1d2fe0cf3edae7af54b43f24f613ea08f465e9fa29f], left=HASH[8d2afe8b42330b1bdd677daffde7238cf93a52146e60eeec8e08b4ce095a9ad1], right=HASH[663bf105cdfa9ffd5431d8190c55a87891da0c13c74eb6a16437526c74de889c], count=7))
27: Child
28: Child
29: Child
30: Child
31: Child
32: Child
33: Push(KVDigestCount(color_00000511, HASH[fb4d7e1e5013a3c804045c72bd920ff81985ee986e87c9373c7041b78953d12e], 1000))
34: Parent
35: Push(HashWithCount(kv_hash=HASH[4ba23a437c91a135eb087602db30021bbbeeba7416d4af9317c2b1a7762ab0a4], left=HASH[cd9697f159ba87524f129190317680dd33f96cf5e8a444c9caaf264fe998746c], right=HASH[f4c9e984a836b6b3739392239b4e35c28c153dd513038a6da5294a4e327c07c0], count=488))
36: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
The descent is identical to Query 4's first six layers (root → @ → contract_id → 0x01 → widget → byBrand → brand_050's value tree → color continuation), then forks at the deepest layer: where Query 4 read one KVValueHashFeatureTypeWithChildHash(color_00000500, CountTree count=1), Query 8 walks the ProvableCountTree boundary using the same HashWithCount / KVDigestCount ops Query 7 used at the doctype level. The boundary keys (color_00000255 → 383 → 447 → 479 → 495 → 499 → 500 → 501 → 503 → 511) name the same binary-merk-tree positions as Query 7's color subtree — predictably, since the bench's deterministic schedule means every brand's color subtree has the same shape.
The final count=488 at the bottom-right HashWithCount covers the upper portion of the in-range subtree (everything to the right of the visible cut); the in-range KVDigestCount ops (color_00000501 count=3, color_00000503 count=15, color_00000511 count=1000) cover named boundary positions inside that subtree. The count field on each merk node is the subtree count (including descendants), not just the named key's contribution — which is why color_00000511 count=1000 and not 1. Summing the boundary contributions yields the final count: 499.
Query 8 is the "compound == then range" shape — and the most expensive query in the chapter. It threads through the same 4 grove layers above the byBrand tree as every other query, descends through byBrand → brand_050's value tree → byBrandColor's color continuation (matching Query 4's path), then runs AggregateCountOnRange over brand_050's ProvableCountTree (matching Query 7's primitive). The result: 8 grove layers of merk-proof bytes — 2 656 B total, ~28 % larger than Query 7's single-leg range and ~39 % larger than Query 4's point-lookup compound.
The reason this even works is that byBrandColor's terminator (brand_X's color continuation) is itself a ProvableCountTree (see Document Count Trees). The compound index has rangeCountable: true, and the parent_value_tree_is_count_tree flag propagates through add_indices_for_index_level_for_contract_operations so the continuation becomes NonCounted(ProvableCountTree(...)) rather than NonCounted(NormalTree(...)). Without that, the boundary nodes wouldn't carry running counts and the verifier would have to enumerate every (brand_050, color) pair.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree"]:::path
BR ==> B050["brand_050: CountTree count=1000"]:::path
B050 ==> B050_C["color: NonCounted(ProvableCountTree count=1000)<br/>(internal merk nodes carry running counts)"]:::target
B050_C -.-> CGT["color_00000501 ... color_00000999<br/>(in range, summed via merk-node counts)"]:::faded
B050_C -.-> CBelow["color_00000000 ... color_00000500<br/>(below range, skipped)"]:::faded
B050 -.-> B050_0["[0]: 1000 byBrand refs"]:::faded
BR -.-> Brands["other brands"]:::faded
WD -.-> CO["color"]:::faded
WD -.-> PK["[0]"]:::faded
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
linkStyle 3 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Layers 5–7 are identical to Query 4 (widget → byBrand → brand_050's value tree → color continuation). The difference from Query 4 is entirely at Layer 8: Query 4 resolved a single color_X element with a KVValueHashFeatureTypeWithChildHash op, whereas Query 8 walks the boundary with HashWithCount and KVDigestCount ops (the same shape as Query 7's L6, just one grove layer deeper).
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree"]
direction TB
L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
L5_left["HASH[9862...]"]:::sibling
L5_right["HASH[6c36...]"]:::sibling
L5_q --> L5_left
L5_q --> L5_right
end
subgraph L6["Layer 6 — byBrand merk-tree (intermediate stop)"]
direction TB
L6_q["<b>brand_050</b><br/>kv_hash=HASH[53db...]<br/>value: CountTree count=1000<br/>(continuation child via lower_layer)"]:::queried
L6_boundary["Boundary commitments (24 merk ops):<br/>6 KVHash sibling brands + 6 Hash subtrees"]:::sibling
L6_q --> L6_boundary
end
subgraph L7["Layer 7 — brand_050's continuation merk-tree (single key)"]
direction TB
L7_q["<b>color</b><br/>kv_hash=HASH[b1ce...]<br/>value: NonCounted(ProvableCountTree count=1000)<br/>(descent into byBrandColor terminator)"]:::queried
L7_left["HASH[2190...]"]:::sibling
L7_q --> L7_left
end
subgraph L8["Layer 8 — byBrandColor color sub-tree (range-aggregate cut)"]
direction TB
L8_result["<b>Aggregate count = 499</b><br/>(returned by verify_aggregate_count_query)"]:::target
L8_inrange["KVDigestCount ops in the in-range path:<br/>color_00000500 (count=1), color_00000501 (3),<br/>color_00000503 (15), color_00000511 (1000)"]:::sibling
L8_boundary["HashWithCount boundary nodes covering<br/>chunks of the cut (counts):<br/>255, 127, 63, 31, 15, 3, 1, 7, 488<br/>+ KVDigestCount path keys above the cut:<br/>color_00000255, 383, 447, 479, 495, 499"]:::sibling
L8_result --> L8_inrange
L8_result --> L8_boundary
end
L5_q -. "Tree(merk_root[byBrand])" .-> L6_q
L6_q -. "CountTree continuation (child_hash)" .-> L7_q
L7_q -. "NonCounted(ProvableCountTree(merk_root))" .-> L8_result
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
Query 8 sits at the intersection of Query 4 (compound descent) and Query 7 (range-aggregate primitive). Its proof carries the descent overhead of both — the boundary commitments at L6 to position brand_050, plus the boundary commitments at L8 to position the color cut — explaining why it's the heaviest proof in the chapter despite verifying a smaller count (499) than Query 7 (49 900).
Diagram: Layer 8 binary merk-tree (the range-aggregate cut)
The 37 merk ops at Layer 8 reconstruct the entire boundary path through brand_050's color ProvableCountTree. Unlike every other query's bottom layer (which abstracts the binary tree into one "target + opaque siblings" box), the AggregateCountOnRange primitive forces the prover to reveal the structural shape of the in-range descent — so we can draw it literally.
Cyan = in-range contributions the verifier adds to the aggregate. Yellow-dashed = the boundary node color_00000500, named so the verifier can position the cut but excluded by the strict > operator. Gray = nodes/subtrees outside the range (boundary commitments needed to prove the rest of the tree's structure, but not summed).
The count field on each node is its subtree count (the node itself + all descendants in the binary merk tree), not just the named key's contribution. So color_00000511 (the root) carries count=1000 because every key in brand_050's color subtree is a descendant — not because there are 1 000 of that one key.
flowchart TB
R["<b>color_00000511</b> (merk root)<br/>KVDigestCount, count=1000<br/>contributes <b>1</b> (itself, in-range)"]:::inrange
R --> L1L["color_00000255<br/>KVDigestCount, count=511<br/>(out-of-range; descend right)"]:::outrange
R --> L1R["HASH[4ba2...] (opaque subtree)<br/>HashWithCount, count=488<br/>(color_00000512 … color_00000999)<br/>contributes <b>488</b> (full subtree in-range)"]:::inrange
L1L --> L2L["HASH[4f8d...] (opaque subtree)<br/>HashWithCount, count=255<br/>(color_00000000 … color_00000254,<br/>all out-of-range)"]:::outrange
L1L --> L2R["color_00000383<br/>KVDigestCount, count=255<br/>(out-of-range)"]:::outrange
L2R --> L3L["HASH[80de...]<br/>HashWithCount, count=127"]:::outrange
L2R --> L3R["color_00000447<br/>KVDigestCount, count=127"]:::outrange
L3R --> L4L["HASH[8277...]<br/>HashWithCount, count=63"]:::outrange
L3R --> L4R["color_00000479<br/>KVDigestCount, count=63"]:::outrange
L4R --> L5L["HASH[11a3...]<br/>HashWithCount, count=31"]:::outrange
L4R --> L5R["color_00000495<br/>KVDigestCount, count=31"]:::outrange
L5R --> L6L["HASH[b932...]<br/>HashWithCount, count=15"]:::outrange
L5R --> L6R["color_00000503<br/>KVDigestCount, count=15<br/>contributes <b>1</b> (itself, in-range)"]:::inrange
L6R --> L7L["color_00000499<br/>KVDigestCount, count=7<br/>(out-of-range; descend right)"]:::outrange
L6R --> L7R["HASH[64d9...] (opaque subtree)<br/>HashWithCount, count=7<br/>(color_00000504 … color_00000510)<br/>contributes <b>7</b> (full subtree in-range)"]:::inrange
L7L --> L8L["HASH[bd58...]<br/>HashWithCount, count=3<br/>(color_00000496 … color_00000498,<br/>all out-of-range)"]:::outrange
L7L --> L8R["color_00000501<br/>KVDigestCount, count=3<br/>contributes <b>1</b> (itself, in-range)"]:::inrange
L8R --> L9L["<b>color_00000500</b><br/>KVDigestCount, count=1<br/>boundary key — strict `>` excludes it<br/>(named so the verifier can place the cut)"]:::boundary
L8R --> L9R["HASH[3b75...] (opaque subtree)<br/>HashWithCount, count=1<br/>(color_00000502)<br/>contributes <b>1</b> (full subtree in-range)"]:::inrange
classDef inrange fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:2px;
classDef outrange fill:#6e7681,color:#fff,stroke:#6e7681;
classDef boundary fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px,stroke-dasharray: 6 3;
The verifier's aggregation walks this tree and sums the cyan nodes' contributions: 1 (color_00000511) + 488 (H_4ba2) + 1 (color_00000503) + 7 (H_64d9) + 1 (color_00000501) + 1 (H_3b75) = 499. Notice the asymmetry — the proof reveals a long boundary-descent path down the left of the tree (through KD_255 → KD_383 → KD_447 → KD_479 → KD_495 → KD_499) just to position the cut, even though none of those nodes contribute to the count. That's the structural floor for AggregateCountOnRange: the prover must commit one merk-binary-tree-depth's worth of boundary nodes per side of the range, regardless of how many keys actually fall inside it.
For a worst-case range (color > color_00000000, i.e. essentially the full tree), the boundary descent collapses to one node and H_4ba2-like fully-in-range subtree commitments dominate. For a narrow range like this one (cutting deep into the tree), the descent path costs more than the in-range commitments. Either way the total is O(log C') boundary nodes — Query 8 just happens to land at the unfavourable end of the constant factor.
The same in-order traversal also explains the keys' positions: a balanced merk binary tree over the 1 000 sorted color keys puts color_00000511 at the root (the ~midpoint by tree-depth, not by sort order — color_00000511 happens to land here because of grovedb-merk's AVL rotation rules on the insertion order), color_00000255 and color_00000383 (along with their descendants color_00000447 / 479 / 495 / 499) at progressive left-of-cut descents, and color_00000503 at the right child of color_00000495 (which is itself the right child of the descent path). The keys are sorted by in-order traversal, not by tree position, so don't expect them to look orderly in the diagram above.
Worked Example: How node_hash_with_count Rebuilds the Merk Root
This section uses Query 8's Layer 8 to make one thing concrete: what the verifier actually computes when it folds the proof's Push / Parent / Child ops up to the merk root. It's the same machinery underpinning every other query in the chapter — Q8 just exposes the most interesting node-hash variant (node_hash_with_count, used by ProvableCountTree).
All hashes are Blake3-256. The hash primitives live at merk/src/tree/hash.rs in grovedb. Six functions compose every node-hash in the chapter:
value_hash(v) = Blake3( varint_len(v) || v )
kv_hash(k, v) = Blake3( varint_len(k) || k || value_hash(v) )
kv_digest_to_kv_hash(k, vh) = Blake3( varint_len(k) || k || vh )
combine_hash(h1, h2) = Blake3( h1 || h2 )
node_hash(kv_h, l, r) = Blake3( kv_h || l || r )
node_hash_with_count(kv_h, l, r, c)
= Blake3( kv_h || l || r || c.to_be_bytes() )
(varint_len is integer_encoding::VarInt::encode_var — the same unsigned-varint encoding used throughout grovedb. c.to_be_bytes() is the 8-byte big-endian encoding of the u64 count.)
Each proof op carries enough information to compute its subtree's node_hash. The reconstruction rule per op variant (from merk/src/proofs/tree.rs's compute_hash):
| Proof op | What's revealed | Subtree node_hash formula |
|---|---|---|
Hash(h) | the subtree hash directly | h (no recompute) |
KVHash(kv_h) | the node's kv-hash only | node_hash(kv_h, left_node_hash, right_node_hash) |
KVHashCount(kv_h, c) | kv-hash + node count | node_hash_with_count(kv_h, left, right, c) |
HashWithCount(kv_h, left, right, c) | kv-hash + both children's node-hashes + count | node_hash_with_count(kv_h, left, right, c) (no recursion — children are already pre-hashed) |
KVValueHash(k, v, kv_h) | full key+value + kv-hash | node_hash(kv_h, left, right) |
KVDigest(k, vh) | key + value-hash | node_hash(kv_digest_to_kv_hash(k, vh), left, right) |
KVDigestCount(k, vh, c) | key + value-hash + count | node_hash_with_count(kv_digest_to_kv_hash(k, vh), left, right, c) |
KVValueHashFeatureTypeWithChildHash(k, v, kv_h, feature, child_hash) | key + value + kv-hash + feature type + opaque child-layer hash | combines kv_h with child_hash (via combine_hash), then node_hash[_with_count] depending on feature |
The left / right arguments are the children's reconstructed node-hashes (computed recursively from the stack as Parent / Child ops glue the subtrees together). For nodes at the leaf level of the proof's revealed structure, both children are NULL_HASH (32 zero bytes).
The example: rebuilding color_00000511's node_hash (Q8, Layer 8)
At the top of Layer 8's binary merk-tree, the proof has three ops that together produce the merk root hash for brand_050's color ProvableCountTree:
op 33: Push(KVDigestCount(
key = "color_00000511",
vh = HASH[fb4d7e1e5013a3c804045c72bd920ff81985ee986e87c9373c7041b78953d12e],
count = 1000))
op 34: Parent (links the running left-side subtree onto color_00000511 as its left child)
op 35: Push(HashWithCount(
kv_h = HASH[4ba23a437c91a135eb087602db30021bbbeeba7416d4af9317c2b1a7762ab0a4],
left = HASH[cd9697f159ba87524f129190317680dd33f96cf5e8a444c9caaf264fe998746c],
right = HASH[f4c9e984a836b6b3739392239b4e35c28c153dd513038a6da5294a4e327c07c0],
count = 488))
op 36: Child (links the just-pushed HashWithCount as color_00000511's right child)
Call them KD_511, HWC_4ba2. By the time we reach op 33 the left-side stack already holds node_hash_left — the recursively-computed node-hash of the whole left subtree rooted at color_00000255 (255 + 1 + 255 = 511 keys, including the boundary-descent path). We'll trust that value here; it's the output of folding ops 0–32 with the same machinery applied recursively.
Step 1: Compute the right child's node_hash directly from HWC_4ba2.
Because HashWithCount already carries the children's node-hashes (cd96... and f4c9...) and the node's own kv-hash (4ba2...) and count (488), no recursion is needed — the verifier just plugs the four values into node_hash_with_count:
node_hash_right
= node_hash_with_count(kv_h=4ba2..., left=cd96..., right=f4c9..., count=488)
= Blake3( 4ba23a437c91a135eb087602db30021bbbeeba7416d4af9317c2b1a7762ab0a4
|| cd9697f159ba87524f129190317680dd33f96cf5e8a444c9caaf264fe998746c
|| f4c9e984a836b6b3739392239b4e35c28c153dd513038a6da5294a4e327c07c0
|| 0x00000000000001E0 ) // 488 as big-endian u64
That's a single Blake3 call over 32 + 32 + 32 + 8 = 104 bytes.
Step 2: Compute color_00000511's kv-hash from its KVDigestCount op.
The proof reveals the key ("color_00000511", 14 bytes ASCII) and the value-hash (fb4d..., 32 bytes). kv_digest_to_kv_hash folds them into the node's kv-hash:
kv_h_511
= kv_digest_to_kv_hash(key="color_00000511", value_hash=fb4d...)
= Blake3( varint_len(14) // = 0x0E (one byte)
|| "color_00000511" // 14 ASCII bytes
|| fb4d7e1e5013a3c804045c72bd920ff81985ee986e87c9373c7041b78953d12e )
That's one Blake3 call over 1 + 14 + 32 = 47 bytes.
Step 3: Compute color_00000511's node_hash by folding the kv-hash with both children's node-hashes plus the running count.
node_hash_511
= node_hash_with_count(
kv_h = kv_h_511, // from Step 2
left = node_hash_left, // from ops 0..32 (the descent path)
right = node_hash_right, // from Step 1
count = 1000)
= Blake3( kv_h_511 || node_hash_left || node_hash_right || 0x00000000000003E8 )
// 1000 as big-endian u64
Another single Blake3 call over 32 + 32 + 32 + 8 = 104 bytes.
Step 4: That's it. node_hash_511 is the merk root of Layer 8 — the byBrandColor color subtree for brand_050. The verifier then checks this against what Layer 7 claimed Layer 8's merk root would be (the NonCounted(ProvableCountTree(...)) value stored against the key "color" inside brand_050's value tree), and so on up the GroveDB layer stack until Layer 1 lands the entire chapter's root_hash = 0x62ee7348f4d28dd9d7cf86a6c725fa8276cfd446f6007a6000fb0e1dfefa6468.
Why the count is part of the hash
The crucial structural feature is the || count.to_be_bytes() tail in node_hash_with_count. Without it, the proof's running counts would be unsigned hints the verifier couldn't trust — a malicious prover could ship a KVDigestCount(color_00000511, fb4d..., 9_999_999_999) and there'd be no way to detect the lie without re-counting the documents (which is exactly what count proofs are supposed to avoid).
Binding the count into the merk root via node_hash_with_count is what lets AggregateCountOnRange skip enumeration: the verifier reads the count off the boundary commitments and trusts it because changing the count would change the merk root, which is consensus-committed.
Concretely, that's why every ProvableCountTree-derived op (KVHashCount, KVDigestCount, HashWithCount, KVValueHashFeatureTypeWithChildHash with feature ProvableCountedMerkNode(_)) routes through node_hash_with_count rather than the cheaper node_hash — see merk/src/proofs/tree.rs's compute_hash and the TreeFeatureType::ProvableCountedMerkNode branch in particular. NormalTree nodes (e.g. byBrand) use plain node_hash — their kv-hash + child hashes don't commit a count, which is why byBrand can't answer AggregateCountOnRange queries even though it's countable: "countable".
One last simplification
For nodes whose proof op already carries the kv-hash (the kvh-prefixed ops — KVHashCount, HashWithCount, KVHash), the verifier skips Step 2 entirely. For nodes whose proof op carries only the key and value-hash (KVDigest, KVDigestCount), the verifier folds them via kv_digest_to_kv_hash first (one extra Blake3 call). For nodes with KVValueHash (full key+value), the verifier recomputes the kv-hash all the way from scratch via kv_hash, which means it also re-hashes the value through value_hash. The choice is driven by how much of the node the proof needs to reveal:
- A node on the descent path that the verifier doesn't need to materialize → emit
Hash(node_hash)(1 hash, opaque). - A node whose existence the verifier must prove but whose value can stay opaque → emit
KVHash(kv_h)orKVHashCount(kv_h, c)(kv-hash committed, value hidden). - A boundary node whose key the verifier needs to compare against the range → emit
KVDigest(k, vh)orKVDigestCount(k, vh, c)(key revealed, value still digested). - A target whose full value the verifier must read → emit
KVValueHash(k, v, kv_h)or the feature-typed variant.
The user-facing trade-off is proof bytes vs information revealed. The verifier's reconstruction logic is uniform: every op feeds the same node-hash formula one variant or another.
At-a-Glance Comparison
| # | Query | Primitive | Verified shape | Proof size |
|---|---|---|---|---|
| 1 | (empty) | primary-key CountTree | 1 CountTree, count=100000 | 585 B |
| 2 | brand == X | PointLookupProof / byBrand | 1 CountTree, count=1000 | 1 041 B |
| 3 | color == X | PointLookupProof / byColor | 1 CountTree, count=100 | 1 327 B |
| 4 | brand == X AND color == Y | PointLookupProof / byBrandColor | 1 CountTree, count=1 | 1 911 B |
| 5 | brand IN [b0, b1] | PointLookupProof / byBrand | 2 CountTrees, sum=2000 | 1 102 B |
| 6 | color IN [c0, c1] | PointLookupProof / byColor | 2 CountTrees, sum=200 | 1 381 B |
| 7 | color > floor | AggregateCountOnRange / byColor | u64=49900 | 2 072 B |
| 8 | brand == X AND color > floor | AggregateCountOnRange / byBrandColor | u64=499 | 2 656 B |
Four takeaways:
- Query 1 is the cheapest. A doctype-level total count is one merk read; everything else descends through an index tree.
- Query 2 and Query 6 are structurally identical despite covering different indexes (
byBrandcountable-only,byColorrangeCountable). The value-tree-direct shape is uniform across countability tiers —rangeCountable: trueonly matters for Queries 7 and 8. - Queries 7 and 8 use a fundamentally different verifier (
verify_aggregate_count_queryvsverify_query). Queries 1–6 return an element list and readcount_value_or_defaultper branch; Queries 7 and 8 return a pre-summedu64. Query 8 is just Query 7 one grove layer deeper — same primitive, applied to byBrandColor's terminator rather than byColor's. - Query 8 is the most expensive. It pays for both the compound descent (Query 4's 4-extra-layer cost) and the range-aggregate boundary (Query 7's primitive). The verified count is far smaller than Query 7 (499 vs 49 900), but the proof bytes are 28 % larger because the merk-tree boundary at L8 has roughly the same shape regardless of how many keys the cut spans.
The path-query builder these examples decode lives at packages/rs-drive/src/query/drive_document_count_query/path_query.rs; the verifier mirror sits in packages/rs-drive/src/verify/document_count/. Both the prover and the verifier reconstruct the exact same PathQuery via the shared builder — touching one without the other is a Merkle-root mismatch waiting to happen, and the byte-identical contract is what makes the proof bytes here reproducible against the bench fixture.
Count Index Group By Examples
This chapter is the GROUP BY companion to Count Index Examples. It uses the same widget contract, the same 100 000-row fixture, and the same bench at packages/rs-drive/benches/document_count_worst_case.rs. Read chapter 29 first — most of the mechanics (CountTree variants, the merk-proof reconstruction algorithm, node_hash_with_count and friends) carry over unchanged.
What's different here:
- Every query in chapter 29 returns either a single
u64aggregate or a small list ofCountTrees the caller sums. The verifier-side payload shape is one count, total. - Every query in this chapter returns one count per group. The caller gets back a
Vec<(group_key, count)>and can index it directly — no summation.
The most important thing to understand up front: group_by is two things at once — a result-shaping directive for the SDK and (for some queries) a proof-shaping directive for the prover. When you pass group_by = [...] in a count request, you're always telling the SDK "don't collapse the result into a single number — give me one count per group key." That result-shaping role is universal: it's what turns Aggregate(sum) into Entries([(key, count), …]).
Whether group_by also changes the proof bytes depends on the query shape. For queries where the underlying proof already commits one CountTree per matched key (single-property INs, for instance), the per-group breakdown is reconstructible from the existing bytes — the prover ships the same proof, the SDK just zips it with the group keys instead of summing. For range queries and certain compound shapes, the per-group breakdown can't be reconstructed from the aggregate-style proof (which commits opaque subtree counts rather than per-key counts), so passing group_by forces the prover to emit a structurally different, larger proof.
The interesting question this chapter answers is: which queries fall into which bucket, and why?
When group_by Changes the Proof (and When It Doesn't)
| Filter | group_by | Aggregate proof (no group_by) | Group-By proof | Proof bytes change? |
|---|---|---|---|---|
brand IN [b0, b1] | [brand] | Q5 — 1 102 B | 1 102 B (2 entries) | No — byte-identical |
color IN [c0, c1] | [color] | Q6 — 1 381 B | 1 381 B (2 entries) | No — byte-identical |
color > floor | [color] | Q7 — 2 072 B (1 u64) | 10 992 B (100 entries) | Yes — different primitive |
brand == X AND color > floor | [brand, color] | Q8 — 2 656 B (1 u64) | not allowed in this form | — |
The key observation: IN clauses produce proofs that already commit one CountTree per resolved key, so adding group_by on the same property is purely a verifier-side relabel — the prover ships the same bytes, the verifier just returns them as Entries(...) instead of Aggregate(sum). This is why G1 and G2 below are not new proofs — they're Q5 and Q6 reinterpreted.
So why pass group_by at all if the proof bytes don't change? Because without it, the SDK has no way to know you want the per-key breakdown. The same brand IN ["brand_000", "brand_001"] proof can answer two different questions:
- "How many widgets total are made by brand_000 or brand_001?" → caller passes no
group_by, SDK returnsAggregate(2 000). - "How many widgets per brand?" → caller passes
group_by = [brand], SDK returnsEntries([("brand_000", 1 000), ("brand_001", 1 000)]).
The bytes on the wire and the cryptographic guarantees are identical; the only thing that changes is which result shape the SDK delivers. Think of group_by as the count-query equivalent of SELECT brand, COUNT(*) ... GROUP BY brand versus SELECT COUNT(*) ... in SQL — same scan plan, different projection.
Range queries are different. AggregateCountOnRange (chapter 29's Q7) walks the boundary of the range over a ProvableCountTree and sums per-subtree counts directly — it never resolves individual keys. GroupByRange (this chapter) has to enumerate the distinct in-range keys to label each group, so it produces a different proof shape with one CountTree (or CountTree-feature-typed element) per distinct key in the range. That's where group_by genuinely earns its bytes — the prover has to do additional work because the per-group breakdown can't be reconstructed from AggregateCountOnRange's opaque-subtree-count commitments.
Queries in this Chapter
All proof-size and behaviour numbers below come from the same bench helper (report_group_by_matrix) as chapter 29's. The dispatcher's group_by surface validation lives in DriveDocumentCountQuery::detect_mode; the per-mode path-query builders sit in packages/rs-drive/src/query/drive_document_count_query/path_query.rs (the group-by modes route to distinct_count_path_query and carrier_aggregate_count_path_query).
| # | Query | Filter + group_by | Complexity | Avg time | Proof size | Verified shape | Notes |
|---|---|---|---|---|---|---|---|
| G1 | In on byBrand | brand IN ["brand_000", "brand_001"] group_by = [brand] | O(k · log B) | 38.6 µs | 1 102 B | Entries(2 groups, sum = 2 000) | Byte-identical to Q5 |
| G1a | In on byBrand with an absent value | brand IN ["brand_000", "brand_100"] group_by = [brand] | O(k · log B) | 44.4 µs | 1 357 B | Entries(1 group, sum = 1 000) | One In value (brand_100) is absent — proof grows by 255 B for the absence subproof; verifier omits the absent branch from entries |
| G1b | High-fanout In on byBrand (|IN| = B) | brand IN [100 values] group_by = [brand] | O(k · log B) | 1 532 µs | 10 038 B | Entries(100 groups, sum = 100 000) | Same shape as G1, scaled from |IN| = 2 → |IN| = 100; reveals every byBrand entry when |IN| = B |
| G2 | In on byColor | color IN ["color_00000000", "color_00000001"] group_by = [color] | O(k · log C) | 62.1 µs | 1 381 B | Entries(2 groups, sum = 200) | Byte-identical to Q6 |
| G3 | Compound In + Equal | brand IN [...] AND color == Y group_by = [brand] | O(k · (log B + log C')) | 106.2 µs | 2 842 B | Entries(2 groups, sum = 2) | Per-In compound resolution; two parallel Q4 descents sharing L1–L6 |
| G4 | Range on byColor | color > "color_00000500" group_by = [color] | O(R · log C) | 762.9 µs | 10 992 B | Entries(100 groups, sum = 10 000) | GroupByRange: enumerates distinct in-range keys instead of Q7's boundary aggregate |
| G5 | Compound In + Range | brand IN [...] AND color > "color_00000500" group_by = [brand, color] | O(k · R' · log C') | 737.5 µs | 11 554 B | Entries(100 groups, sum = 100) | Compound In-fan-out × in-range distinct keys (G3 outer × G4 inner) |
| G7 | Carrier In + Range (byBrandColor) | brand IN [...] AND color > "color_00000500" group_by = [brand] | O(k · (log B + log C')) | 255.9 µs | 4 332 B | Entries(2 groups, sum = 998) | Per-In aggregate via AggregateCountOnRange as a carrier subquery; one u64 per branch |
| G8 | Carrier outer Range + Range (byBrandColor) | brand > "brand_050" AND color > "color_00000500" group_by = [brand] | O(L · (log B + log C')) | 523 µs | 18 022 B | Entries(10 groups, sum = 4 990) | Outer-Range carrier with a platform-max SizedQuery::limit of 10; caller may pass smaller, can't pass larger |
| G8a | Bounded carrier + bounded ACOR, descending | brand > "brand_050" AND brand < "brand_065" AND color > "color_00000200" AND color < "color_00000400" group_by = [brand], order_by = [(brand, desc)] | O(L · (log B + log C')) | 807 µs | 29 010 B | Entries(10 groups, sum = 1 990) | Bounded ranges on both axes + descending walk; same carrier shape as G8, different op variants on both range commitments |
| G8b | Same carrier where but group_by = [brand, color] | brand > "brand_050" AND color > "color_00000500" group_by = [brand, color] | — | — | rejected | InvalidWhereClauseComponents("count query supports at most one range where-clause; …or use group_by = [outer_range_field]withprove = true…") | Two-range carrier is opened only for GroupByRange + single-field group_by; the compound shape can't fan over both ranges |
| G8c | Same carrier where but group_by = [] | brand > "brand_050" AND color > "color_00000500" group_by = [] | — | — | rejected | InvalidWhereClauseComponents("count query supports at most one range where-clause; …or use group_by = [outer_range_field]withprove = true…") | Aggregate (no group_by) can't collapse the carrier's per-branch u64s into a single sum at the verifier |
Complexity variables. B = distinct brands in the byBrand merk-tree (≈ 100); C = distinct colors in byColor (≈ 1 000); C' = distinct colors per brand in byBrandColor (≈ 1 000); R = distinct in-range values returned by GroupByRange (capped at 100 in this fixture by an implicit response-size limit); R' = distinct in-range values per fan-out branch (similarly capped); k = |IN| for the In-outer carrier shapes; L = the effective outer-walk limit for the Range-outer carrier shape (G8). The platform's MAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT = 10 is both the default (when the caller passes no limit) and a hard ceiling; callers may pass a smaller limit to truncate further. See G8 for the rationale. As in chapter 29, the total document count N doesn't appear — count proofs read pre-committed count_values rather than enumerating docs.
Avg time is the criterion-reported median of cargo bench --bench document_count_worst_case -- 'document_count_worst_case/query_g' on the same 100 000-row warmed fixture used by chapter 29's query_N_* cases. Each row reflects 10 samples × ~3 k–130 k iterations per sample with 2 s warm-up and 5 s measurement; the median sits within ±2 % of the mean across reruns. G1 and G2 match their Q5 / Q6 counterparts to within ~3 µs — the residual is the SDK-side zip-vs-sum cost. G4 is ~11 × Q7 because GroupByRange enumerates 100 distinct in-range CountTrees rather than walking O(log C) boundary nodes; the time difference is exactly the complexity difference predicted (O(R · log C) vs O(log C)).
Group-By Shapes That Are Not Allowed
Several plausible-looking (where, group_by) combinations are rejected by the dispatcher before any proof generation. The rejections fall into four buckets — operator/group_by mismatch, missing range window, no covering index, and one currently-deferred aggregate variant. All are surfaced as typed QuerySyntaxErrors; the precise error strings appear in the bench's [matrix] output.
1. group_by field constrained by == instead of In or range
where = brand == "brand_050"
group_by = [brand]
count query supports only ...(rejected because==produces exactly one entry whose key equals the where-clause's value — grouping by a field that already has a single value contributes no extra information).
Why. GROUP BY [field] is meaningful only when field can take multiple values in the result set. An == clause pins the field to exactly one value, so the group_by is structurally redundant — the dispatcher rejects it rather than silently returning a single-entry response that would look like a bug. Use Q2 / Q3 (no group_by) for single-value == queries.
Applies symmetrically: where = color == X, group_by = [color] is rejected for the same reason.
2. group_by contains a range field but the where clause doesn't range over it
where = brand IN[...] AND color == "color_00000500"
group_by = [brand, color]
GROUP BY on a range field requires a range where-clause; the range field must appear inwherefor the distinct walk to have a window to iterate over
Why. group_by = [in_field, range_field] (GroupByCompound) routes through distinct_count_path_query, which needs a range window on the second field to know what values to enumerate. With color == Y the second dimension collapses to a single value, so the compound walk degenerates to a point lookup — and that's what Q4 / G3 are for. For compound plus range, the where must carry a range on the second field (which is what G5 does).
3. group_by orders fields in a way no covering index can serve
where = color IN[...] AND brand > "brand_050"
group_by = [color, brand]
where clause on non indexed property error: range count requires arange_countable: true(or summableOffCountIndex) index whose last property matches the range field
Why. The covering index for (group_by[0] = color, group_by[1] = brand) would need to be byColorBrand with rangeCountable: true on the brand terminator. The widget contract doesn't have that index — only byBrand, byColor, and byBrandColor. The dispatcher's index picker walks every declared index, finds none whose (properties, last_property_is_range_countable) shape matches the request, and rejects with the "non-indexed property" error.
The fix is contract-level: declare a byColorBrand index with rangeCountable: true if the application needs this group_by order. The dispatcher itself can't infer alternate index orders from the request alone — rangeCountable: true is an explicit opt-in on each index because it changes the on-disk tree shape (NormalTree → ProvableCountTree on the property-name subtree).
To put these three buckets in one place: every rejected (where, group_by) shape on this contract reduces to one of:
- the
group_byfield'swhereoperator doesn't admit multiple values (bucket 1), - the
group_byhas a range slot that thewheredoesn't fill with a range (bucket 2), - there's no covering
rangeCountableindex in property order (bucket 3).
All three checks happen at request validation, before any GroveDB work. The bench's report_group_by_matrix exercises one example of each and prints the exact error string, so adding a new contract or index shape is a quick way to see which checks each new query shape hits.
Historical note. A fourth bucket —
group_by = [in_field]withwhere = in_field IN[...] AND range_field > floor— was rejected before grovedb PR #663. That PR added support forAggregateCountOnRangeas a carrier subquery under outerKeys, which unblocked the natural single-field-group_by shape (one aggregate count per In branch) at the merk layer. The dispatcher now routes that shape to [DocumentCountMode::RangeAggregateCarrierProof]; the worked-out example is G7 below.
G1 — In on byBrand, Grouped By brand
select = COUNT
where = brand IN ["brand_000", "brand_001"]
group_by = [brand]
prove = true
Path query (identical to Q5):
path: ["@", contract_id, 0x01, "widget", "brand"]
query items: [Key("brand_000"), Key("brand_001")]
Verified payload (the only thing that differs from Q5):
Entries([
("brand_000", CountTree { count_value_or_default: 1000 }),
("brand_001", CountTree { count_value_or_default: 1000 }),
])
The SDK zips the In values with the two resolved CountTree elements (in lex-asc order) rather than summing them as Q5's CountMode::Aggregate does.
Proof size: 1 102 B. Proof bytes are byte-identical to Q5 — same path query, same merk ops, same hash composition. The dispatcher recognises that CountMode::GroupByIn on a single-property In clause resolves through the same point_lookup_count_path_query as CountMode::Aggregate does; only the response-shaping at the very end differs.
For the verbatim proof display, see Q5 in chapter 29 — every byte of the 1 102-byte proof is the same. Or ▶ open the proof interactively in the visualizer ↗ (same encoded payload). The diagrams below show the result-shaping difference.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree"]:::path
BR ==> B000["brand_000: CountTree count=1000"]:::target
BR ==> B001["brand_001: CountTree count=1000"]:::target
BR -.-> BMore["brand_002 ... brand_099"]:::faded
SDK["Verifier returns Entries([<br/>("brand_000", 1000),<br/>("brand_001", 1000)<br/>])"]:::sdk
B000 -.-> SDK
B001 -.-> SDK
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Identical to Q5's Layer-5+ diagram — same merk ops, same byBrand binary tree, same two KVValueHashFeatureTypeWithChildHash targets. The only difference is what the verifier returns at the end (Entries(...) instead of Aggregate(2000)); the per-layer structure is unchanged. See chapter 29 for the diagram.
G1a — In on byBrand with one absent value, Grouped By brand
select = COUNT
where = brand IN ["brand_000", "brand_100"]
group_by = [brand]
prove = true
The bench fixture has brands brand_000 … brand_099 (BRAND_COUNT = 100); brand_100 is deliberately outside that range. G1a is G1's same-shape sibling: same path query, same point_lookup_count_path_query builder, same CountMode::GroupByIn dispatch. The only structural difference is one of the In keys doesn't exist in the byBrand merk tree.
Path query (identical shape to G1; only the second key differs):
path: ["@", contract_id, 0x01, "widget", "brand"]
query items: [Key("brand_000"), Key("brand_100")]
Verified payload (note: only one entry — the absent branch is silently dropped):
Entries([
("brand_000", CountTree { count_value_or_default: 1000 }),
])
This is the load-bearing behaviour to know about: grovedb's verify_query without absence_proofs_for_non_existing_searched_keys: true drops absent-Key branches from the elements stream. The drive-side verifier (verify_point_lookup_count_proof_v0) uses the default (off) and so emits one entry per present In value, not one per requested In value. Test coverage: test_point_lookup_proof_omits_absent_in_branches_from_entries.
Caller implication. Callers MUST NOT assume entries.len() == |In|. To check whether a specific In value matched, demux entries by serialized key (the same serialize_value_for_key(field, value) the path-query builder uses for outer Keys) — see the test for the canonical pattern. A 0-count vs absent-key distinction would require passing absence_proofs_for_non_existing_searched_keys: true end-to-end, which the platform doesn't expose today.
Proof size: 1 357 B (+255 B over G1's 1 102 B). The delta is the absence subproof: grovedb walks the byBrand merk tree to commit the rightmost present key (brand_099) and the chain of Child ops that proves there's nothing between brand_099 and end-of-tree. Even though the verifier drops the absent entry, the prover must cryptographically commit to the absence — otherwise a malicious prover could omit a present branch by claiming it's absent.
Mode: CountMode::GroupByIn routed to DocumentCountMode::PointLookupProof — same as G1.
Proof display:
The absence-subproof shape is what makes G1a interesting. The L8 (byBrand value tree) layer commits both:
- The present branch (op 0):
Push(KVValueHashFeatureTypeWithChildHash(brand_000, CountTree(636f6c6f72, 1000, …)))—brand_000as a CountTree with count = 1000, exactly as in G1. - The absence commitment (op 36):
Push(KVDigest(brand_099, HASH[…]))— the rightmost present brand in the byBrand merk tree, paired with a chain ofChildops (37–42) that the verifier replays to confirm there's no key strictly betweenbrand_099and end-of-tree.brand_100would have to sort afterbrand_099(which is true:brand_099<brand_100lexicographically), so the verifier's merk-root recomputation succeeds with nobrand_100element emitted.
The bench's [gproof] G1a output dumps the full 1357-byte proof:
Expand to see the structured proof (L1–L8 for byBrand, with one present CountTree at L8 + one absence subproof at L8)
GroveDBProofV1 {
LayerProof { // L1: roots merk
proof: Merk(
0: Push(Hash(HASH[bd29…3b3])) // sibling: contracts subtree
1: Push(KVValueHash(@, Tree(4ed2…289), HASH[…])) // KVValueHash of `@` (data-contract subtree root) — descend
2: Parent
3: Push(Hash(HASH[19c9…b71])) // sibling
4: Child)
lower_layers: {
@ => {
LayerProof { // L2: `@` subtree
proof: Merk(
0: Push(KVValueHash(0x4ed2…289, Tree(01), HASH[…]))) // descend into contract-id subtree
lower_layers: {
0x4ed2…289 => {
LayerProof { // L3: contract-id subtree
proof: Merk(
0: Push(Hash(HASH[49e7…df8])) // sibling
1: Push(KVValueHash(0x01, Tree(widget), HASH[…])) // descend into doctype `widget`
2: Parent)
lower_layers: {
0x01 => {
LayerProof { // L4: doctype-prefix subtree
proof: Merk(
0: Push(KVValueHash(widget, Tree(brand), HASH[…]))) // descend into byBrand index
lower_layers: {
widget => {
LayerProof { // L5: widget subtree
proof: Merk(
0: Push(Hash(HASH[9862…9d9])) // sibling
1: Push(KVValueHash(brand, Tree(brand_063), HASH[…])) // descend into byBrand value tree (rooted at `brand_063`)
2: Parent
3: Push(Hash(HASH[6c36…a86]))
4: Child)
lower_layers: {
brand => {
LayerProof { // L6+L7+L8: byBrand value tree (binary search down to `brand_000` + absence walk to `brand_099`)
proof: Merk(
0: Push(KVValueHashFeatureTypeWithChildHash(brand_000, CountTree(color, 1000, flags), HASH[…], BasicMerkNode, HASH[…])) // PRESENT — `brand_000` as CountTree(count=1000)
1: Push(KVHash(HASH[…]))
2: Parent
3: Push(Hash(HASH[…]))
4: Child
… (24 intermediate `KVHash`/`Hash`/`Parent`/`Child` ops walking the binary search)
35: Push(KVHash(HASH[…]))
36: Push(KVDigest(brand_099, HASH[…])) // ABSENCE COMMITMENT — rightmost present brand
37: Child
38: Child
39: Child
40: Child
41: Child
42: Child)
}}}}}}}}}}}}}}}}}
Op 36 (KVDigest(brand_099, …)) is the load-bearing piece. The verifier replays ops 37–42 (Childs) against the byBrand merk root committed at L5; any tampering — say, an honest brand_099 swapped for a malicious brand_100-shaped commitment — would change the merk root and the verification would fail.
Diagram: conceptual flow (where the absence proof sits)
flowchart TB
RQ["IN [brand_000, brand_100]"]:::request
RQ --> M["dispatcher → PointLookupProof<br/>(group_by = [brand])"]:::dispatch
M --> P["point_lookup_count_path_query<br/>outer Keys = [brand_000, brand_100]"]:::path
P --> V["grovedb walks byBrand merk tree"]:::engine
V --> P1["brand_000 ✓ present<br/>commit CountTree(count=1000)"]:::present
V --> P2["brand_100 ✗ absent<br/>commit rightmost present (brand_099)<br/>+ Child chain to end-of-tree"]:::absent
P1 --> R["Proof bytes: 1357 B<br/>(1102 B for the present branch +<br/>~255 B for the absence subproof)"]:::result
P2 --> R
R --> SDK["verify_point_lookup_count_proof<br/>(absence_proofs_for_non_existing_searched_keys = false)"]:::verify
SDK --> OUT["Entries([(brand_000, 1000)])<br/>brand_100 silently dropped"]:::sdk
classDef request fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef dispatch fill:#21262d,color:#c9d1d9,stroke:#1f6feb;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb;
classDef engine fill:#21262d,color:#c9d1d9,stroke:#39c5cf;
classDef present fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef absent fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:3px,stroke-dasharray: 6 3;
classDef result fill:#21262d,color:#c9d1d9,stroke:#39c5cf,stroke-width:2px;
classDef verify fill:#21262d,color:#c9d1d9,stroke:#a371f7,stroke-width:2px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
Per-layer merk-tree structure (Layer 5+)
flowchart TB
L5["L5 — widget subtree:<br/>KVValueHash(brand, Tree(brand_063))"]:::path
L5 --> L6["L6 — byBrand value tree root:<br/>brand_063 (binary-search root)"]:::path
L6 --> L7L["brand_031 (left subtree boundary)"]:::sibling
L6 --> L7R["brand_095 (right subtree boundary)"]:::sibling
L7L --> P000["brand_000<br/>(present, CountTree count=1000)"]:::target
L7R --> A099["brand_099<br/>(rightmost present, absence-proof anchor)"]:::boundary
L7R -.-> A100["brand_100 (not in tree — absence proven<br/>by Child chain to end-of-tree)"]:::absent
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef boundary fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef absent fill:#21262d,color:#d29922,stroke:#d29922,stroke-width:2px,stroke-dasharray: 6 3;
Why absence-proof matters for count queries. The drive count fast path treats absent branches as 0, but it does NOT trust the SDK to apply that rule on un-committed data — every count or non-existence the verifier reports must be cryptographically committed by the prover. If absent branches were silently summed into 0 without a proof, a malicious prover could omit a present branch (with positive count) and claim it's absent, shrinking the result without detection. The 255-B absence-subproof overhead is the price of that integrity — small in absolute terms, but it scales linearly with the number of absent In values, so callers building queries with many speculative In values pay per-absence overhead.
G1b — High-fanout In on byBrand (|IN| = B), Grouped By brand
select = COUNT
where = brand IN ["brand_000", "brand_001", ..., "brand_099"]
group_by = [brand]
prove = true
Path query (same shape as G1, scaled to |IN| = 100):
path: ["@", contract_id, 0x01, "widget", "brand"]
query items: [Key("brand_000"), Key("brand_001"), ..., Key("brand_099")]
Verified payload:
Entries(100 groups, sum = 100 000)
Every document in the fixture, partitioned by brand. Each Entries[i] carries (brand_NNN, CountTree count=1000).
Proof size: 10 038 B. Mode: CountMode::GroupByIn.
Same structural shape as G1, scaled from |IN| = 2 to |IN| = 100. The byBrand merk binary tree at L6 emits all 100 brands as KVValueHashFeatureTypeWithChildHash targets — each ~100 B (key + leaf kv-hash + CountTree(00, 1000, ...) + BasicMerkNode feature + child-hash) — plus minimal boundary glue at the binary-tree corners. The proof grows linearly with |IN|: G1 (|IN|=2) was 1 102 B; G1b (|IN|=100) is 10 038 B; the slope is ~99 B per additional In value.
Compare against the byColor equivalent (group_by_color_in_proof_100_rangecountable_branches, 10 512 B): the ProvableCountTree overhead from byColor's KVHashCount running counts adds ~5 % to the byBrand baseline, even though those running counts aren't consumed by a point-lookup group_by. This is the same ProvableCountTree overhead G2 carried at the smaller scale (|IN|=2).
Proof display:
Expand to see the structured proof (5 layers; bottom layer enumerates 100 brands as `KVValueHashFeatureTypeWithChildHash` targets — 192 merk ops total at L6 including binary-tree glue) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
// L2..L4 are byte-identical to every other query in this chapter
// (the @ / contract_id / 0x01 descent into widget); see chapter 29's
// Q1 verbatim for the full L1..L4 chain.
...
widget => {
LayerProof {
proof: Merk(
// L5 widget doctype — `brand` queried, opaque siblings 9862 / 6c36
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
2: Parent
3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
4: Child)
lower_layers: {
brand => {
LayerProof {
proof: Merk(
// L6 byBrand merk-tree — 100 targets + binary-tree glue
// (192 merk ops total; structurally a fully-resolved in-order
// traversal of all 100 brand entries in the byBrand merk tree)
0: Push(KVValueHashFeatureTypeWithChildHash(brand_000, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[90ff6f6d9a3d901195982128130677243bfd27b75736206f3c8400966ef0d37b], BasicMerkNode, HASH[19b58883c492e746861db1e6ad07529a5a91cc8330af522682486db9346d6875]))
1: Push(KVValueHashFeatureTypeWithChildHash(brand_001, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[484ca11fb4ec8f479be1f78af903ce0c9d4fe630517579fb0172c2576d6b9652], BasicMerkNode, HASH[0bf12023f8e067c12db4cec1583909a0283878d6d909c76196736299750b5879]))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(brand_002, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[4c19f047068654e71813dce7839a579edfdcb446e3d70efa1b8592c73259da16], BasicMerkNode, HASH[e8d5372904b7f4ac9334aeb4ddab619d9ad7a308732a4f231416e10208a0a356]))
...
// 97 more KVValueHashFeatureTypeWithChildHash targets following
// the same template — brand_003 ... brand_099 — interleaved with
// Parent/Child ops glueing them into the byBrand merk binary tree.
// Every target shares the structure:
// Push(KVValueHashFeatureTypeWithChildHash(
// brand_NNN,
// CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), // count_value=1000
// HASH[<per-brand leaf kv-hash>],
// BasicMerkNode, // NormalTree (no count on the merk node)
// HASH[<per-brand subtree child hash>]
// ))
...
189: Push(KVValueHashFeatureTypeWithChildHash(brand_097, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[92adee932cc12927cd76ad9fd25906bbfe547df2bf21e826845bb4d3b47f5314], BasicMerkNode, HASH[34b69e1e424aa023c74f61554db2823da6c19dcbc51bdd5dece32e3f6f9fd219]))
190: Parent
191: Push(KVValueHashFeatureTypeWithChildHash(brand_098, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[68e02fcf66f86797035fbc8d53290185fe3fed7de897a8654743cae4007c47c3], BasicMerkNode, HASH[acfc3a88b852e8895449b4c7e01f4b1cc25028e6a80e4915cdde578ff6eb029b]))
192: Push(KVValueHashFeatureTypeWithChildHash(brand_099, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[af9667a8f2a10a9402b3d1fb0ac6e0b64d1e3dde5b8829c03b8d2c9cfc94e16d], BasicMerkNode, HASH[d049fe7e250b7dd763a4a5daa4227dcd2e41733dd95fd0758641ac06c63c3b51]))
// + closing Parent/Child ops binding the last few entries
)
}
}
}
}
}
}
}
}
The 254-line full verbatim sits in the bench's [gproof] G1b output — same template (one KVValueHashFeatureTypeWithChildHash per brand, all with CountTree count=1000 and BasicMerkNode feature) repeating 100 times. The schematic above shows the first 3 and last 3 targets so the structural pattern is clear without reproducing 100 near-identical lines.
Key observation: BasicMerkNode (not ProvableCountedMerkNode) is the feature type on each L6 op. byBrand is a NormalTree, so its merk binary tree's internal nodes don't carry running counts — only the per-brand CountTree count=1000 values stored inside each brand's element matter. Contrast this with G1b's byColor cousin (group_by_color_in_proof_100_rangecountable_branches, 10 512 B): there the L6 targets would carry ProvableCountedMerkNode(...) features because byColor IS a ProvableCountTree. The ~5 % size difference is exactly those count fields × 100 nodes.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree (100 entries)"]:::path
BR ==> B000["brand_000: CountTree count=1000"]:::target
BR ==> B001["brand_001: CountTree count=1000"]:::target
BR ==> BMore["... 96 more in-range targets<br/>(brand_002 ... brand_097)"]:::target
BR ==> B098["brand_098: CountTree count=1000"]:::target
BR ==> B099["brand_099: CountTree count=1000"]:::target
SDK["Entries(100 groups, sum=100 000):<br/>("brand_000", 1000),<br/>("brand_001", 1000),<br/>...<br/>("brand_099", 1000)"]:::sdk
B000 -.-> SDK
B099 -.-> SDK
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
linkStyle 3 stroke:#1f6feb,stroke-width:3px;
linkStyle 4 stroke:#1f6feb,stroke-width:3px;
linkStyle 5 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Identical to G1's L5–L6 shape, just with all 100 entries in the byBrand merk tree resolved as visible targets rather than just two. The byBrand binary tree has all 100 keys exposed — no opaque sibling subtrees (Hash ops) at all, only KVValueHashFeatureTypeWithChildHash (full reveal) plus Parent / Child glue.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree"]
direction TB
L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
L5_left["HASH[9862...]"]:::sibling
L5_right["HASH[6c36...]"]:::sibling
L5_q --> L5_left
L5_q --> L5_right
end
subgraph L6["Layer 6 — byBrand merk-tree (ALL 100 targets fully resolved)"]
direction TB
L6_t0["<b>brand_000</b><br/>CountTree count=1000<br/>BasicMerkNode"]:::target
L6_t1["<b>brand_001</b><br/>CountTree count=1000"]:::target
L6_tmid["... 97 more KVValueHashFeatureTypeWithChildHash<br/>targets, each CountTree count=1000<br/>(192 merk ops total: 100 Push + 92 Parent/Child)"]:::target
L6_t99["<b>brand_099</b><br/>CountTree count=1000"]:::target
L6_t0 --> L6_t1
L6_t1 --> L6_tmid
L6_tmid --> L6_t99
end
L5_q -. "Tree(merk_root[byBrand])" .-> L6_t0
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
Because the In set covers every brand in the fixture, the proof has zero opaque-sibling subtree commitments at L6 — every binary-tree node is revealed as a KVValueHashFeatureTypeWithChildHash target. That's the most efficient byte-per-key shape GroupByIn can hit: at |IN| = B (where B is the total entries in the property tree), the proof bytes ≈ B × (kv-hash + count + child-hash + glue) ≈ B × 100 B. For B = 100, that's exactly the 10 038 B we observe.
By contrast, smaller In sets (G1's |IN| = 2) pay the boundary-proof tax: the byBrand merk tree has ~98 unresolved entries, each contributing one KVHash (opaque-key commitment, ~33 B) or Hash (opaque-subtree commitment, ~33 B). The asymptotic crossover at which "reveal everything" becomes cheaper than "reveal-some-and-commit-the-rest" depends on the ratio of |IN| to B — for byBrand with B = 100, the crossover is around |IN| ≈ 50.
G2 — In on byColor, Grouped By color
select = COUNT
where = color IN ["color_00000000", "color_00000001"]
group_by = [color]
prove = true
Path query (identical to Q6):
path: ["@", contract_id, 0x01, "widget", "color"]
query items: [Key("color_00000000"), Key("color_00000001")]
Verified payload:
Entries([
("color_00000000", CountTree { count_value_or_default: 100 }),
("color_00000001", CountTree { count_value_or_default: 100 }),
])
Proof size: 1 381 B. Byte-identical to Q6 — same path query, same ProvableCountTree-style boundary commitments (KVHashCount ops carry running counts even though the SDK doesn't read them for this point lookup). The single difference from G1 is the underlying property-name tree type (ProvableCountTree for byColor vs NormalTree for byBrand); that affects the merk-boundary commitments but not the dispatcher's GroupByIn-vs-Aggregate routing.
For the verbatim proof display, see Q6 in chapter 29 — or ▶ open it interactively in the visualizer ↗.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> CO["color: ProvableCountTree"]:::path
CO ==> C000["color_00000000: CountTree count=100"]:::target
CO ==> C001["color_00000001: CountTree count=100"]:::target
CO -.-> CMore["color_00000002 ... color_00000999"]:::faded
SDK["Verifier returns Entries([<br/>("color_00000000", 100),<br/>("color_00000001", 100)<br/>])"]:::sdk
C000 -.-> SDK
C001 -.-> SDK
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#d29922,color:#0d1117,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Identical to Q6's Layer-5+ diagram. The byColor ProvableCountTree at L6 carries the same KVHashCount running counts; the SDK ignores them for point-lookup group_by and reads only the two resolved targets' count_value_or_default.
G3 — Compound In + Equal, Grouped By brand
select = COUNT
where = brand IN ["brand_000", "brand_001"] AND color == "color_00000500"
group_by = [brand]
prove = true
Path query (per-In compound resolution — outer Query on byBrand, inner subquery on byBrandColor's color terminator):
path: ["@", contract_id, 0x01, "widget", "brand"]
query items: [Key("brand_000"), Key("brand_001")]
subquery_path: ["color"]
subquery items: [Key("color_00000500")]
Verified payload:
Entries([
("brand_000", CountTree { count_value_or_default: 1 }),
("brand_001", CountTree { count_value_or_default: 1 }),
])
Each (brand, "color_00000500") pair has exactly 1 document in the bench's deterministic schedule.
Proof size: 2 842 B. Mode: CountMode::GroupByIn over the byBrandColor compound index.
Proof display:
Expand to see the structured proof (8 layers — two parallel brand-X → color → color_00000500 descents sharing L1–L6) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVValueHash(brand, Tree(6272616e645f303633), HASH[68b697da99d6ea70a83eb41794dca7ba3938d0ba98fbfaeb3cd0c19b3b5d0ff2]))
2: Parent
3: Push(Hash(HASH[6c36729e93b1a316cbf60fe282eb630c0ed6e45db088e365110302b6c9caba86]))
4: Child)
lower_layers: {
brand => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(brand_000, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[90ff6f6d9a3d901195982128130677243bfd27b75736206f3c8400966ef0d37b]))
1: Push(KVValueHash(brand_001, CountTree(636f6c6f72, 1000, flags: [0, 0, 0]), HASH[484ca11fb4ec8f479be1f78af903ce0c9d4fe630517579fb0172c2576d6b9652]))
2: Parent
3: Push(Hash(HASH[8ca09dadc802a7efe03534ce4ad991b2f191f368878754a37b5e5c03d9498dab]))
4: Child
5: Push(KVHash(HASH[e5297b3ebe81c6435c29f712074da5f7c90265e12ed3d4f5af1f6d900e50c9f1]))
6: Parent
7: Push(Hash(HASH[50f373fd01dea89c992779764dff82cc7200b492be8f5cf3721627d5323bcbff]))
8: Child
9: Push(KVHash(HASH[cf78c9f1b1a1204bb2e437806f52c21e331392de3436388572bd1fa4bce1cdc7]))
10: Parent
11: Push(Hash(HASH[4a8dc186a95c8c4a1252fb51dbc407727f588eb5bdc8313c96f5c29889e13926]))
12: Child
13: Push(KVHash(HASH[d00ee7653e34e47d46004929b13ded33dff069ed9cc88342cecdf66a65fd8401]))
14: Parent
15: Push(Hash(HASH[7f1d17b9632f0bd440dacf5e841025482bc1d8145df3650301a95a5ee71ce8c8]))
16: Child
17: Push(KVHash(HASH[3ed48a5e35cb7546d329487b0e1ab8a81d7c5bec358c37449e6cbd956e3bb069]))
18: Parent
19: Push(Hash(HASH[eaef9fc530408393bc321409414814b290309a861f474a925a922250327affc6]))
20: Child
21: Push(KVHash(HASH[f776417ede76e6194706e483ac14ab7b3db6aa0461ec14ed5f8e5d20071363af]))
22: Parent
23: Push(Hash(HASH[b3fccba79c14fcc5e97ff6a3cd051228dc755e6de147bef690ba9681264b2b9f]))
24: Child)
lower_layers: {
brand_000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[d605b4b78e674fd77371ea6adb32ce3e58ee3b96d73c4d34df84159661634587]))
1: Push(KVValueHash(color, NonCounted(ProvableCountTree(636f6c6f725f3030303030353131, 1000, flags: [0, 0, 0])), HASH[fccc0c94657f2a78084f789bb6f687c4bba295e3a062f3199bc33f14dd2b7fe2]))
2: Parent)
lower_layers: {
color => {
LayerProof {
proof: Merk(
... 37 ops — same boundary shape as Q4 / Q8's L8,
terminating at op 18 with
Push(KVValueHashFeatureTypeWithChildHash(
color_00000500, CountTree(00, 1, ...),
HASH[6834...], ProvableCountedMerkNode(1),
HASH[840c...]))
— TARGET 1
)
}
}
}
}
}
brand_001 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[f54769bf6e9d24b9dba53ebd37c9ceb3485b3c6511f8de6f17860676fe4d9331]))
1: Push(KVValueHash(color, NonCounted(ProvableCountTree(636f6c6f725f3030303030353131, 1000, flags: [0, 0, 0])), HASH[8f883171c33df0aba2541a5b9d6195faac7bd1ffef93e8ddcaf9d092f0fa5e19]))
2: Parent)
lower_layers: {
color => {
LayerProof {
proof: Merk(
... 37 ops — same boundary shape as brand_000's
color subtree, terminating at op 18 with
Push(KVValueHashFeatureTypeWithChildHash(
color_00000500, CountTree(00, 1, ...),
HASH[881d...], ProvableCountedMerkNode(1),
HASH[a422...]))
— TARGET 2
)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
The two parallel descents below brand are the structurally novel part — every other layer above brand is byte-identical to Q4. The byBrand layer (L6) inlines brand_000 and brand_001 as KVValueHash siblings (ops 0–2), then descends via the lower_layers map into each one's value-tree continuation. Each continuation (L7) carries a single color key whose value is NonCounted(ProvableCountTree(…)) — the byBrandColor terminator. The terminator (L8) walks the boundary path through its in-color binary merk tree to land at color_00000500 with CountTree count=1 and a feature-typed child hash.
The bulk of the proof bytes (≈ 2 × 1 100 B = 2 200 B) is the doubled L7+L8 descent. The L1–L6 prefix amortises across both branches (≈ 600 B shared), giving 2 842 B total — significantly less than 2× Q4's 1 911 B because the upper layers aren't repeated.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree"]:::path
BR ==> B000["brand_000: CountTree count=1000"]:::path
BR ==> B001["brand_001: CountTree count=1000"]:::path
B000 ==> B000_C["color: NonCounted(ProvableCountTree)"]:::path
B001 ==> B001_C["color: NonCounted(ProvableCountTree)"]:::path
B000_C ==> T1["color_00000500: CountTree count=1"]:::target
B001_C ==> T2["color_00000500: CountTree count=1"]:::target
SDK["Verifier returns Entries([<br/>("brand_000", 1),<br/>("brand_001", 1)<br/>])"]:::sdk
T1 -.-> SDK
T2 -.-> SDK
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
linkStyle 3 stroke:#1f6feb,stroke-width:3px;
linkStyle 4 stroke:#1f6feb,stroke-width:3px;
linkStyle 5 stroke:#1f6feb,stroke-width:3px;
linkStyle 6 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Layers 5–6 are like Q4's L5 + Q5's L6 combined (one KVValueHash per In brand at byBrand's binary tree); Layers 7–8 fork — one brand_000-rooted continuation chain and one brand_001-rooted chain — each shaped exactly like Q4's L7 + L8 descent.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree"]
direction TB
L5_q["<b>brand</b><br/>kv_hash=HASH[68b6...]<br/>value: Tree (descent into byBrand)"]:::queried
L5_left["HASH[9862...]"]:::sibling
L5_right["HASH[6c36...]"]:::sibling
L5_q --> L5_left
L5_q --> L5_right
end
subgraph L6["Layer 6 — byBrand merk-tree (TWO INTERMEDIATE TARGETS)"]
direction TB
L6_t1["<b>brand_001</b><br/>kv_hash=HASH[484c...]<br/>value: CountTree count=1000"]:::queried
L6_t0["<b>brand_000</b><br/>kv_hash=HASH[90ff...]<br/>value: CountTree count=1000"]:::queried
L6_boundary["Boundary commitments (22 merk ops):<br/>7 KVHash sibling brands + 7 Hash subtrees"]:::sibling
L6_t1 --> L6_t0
L6_t1 --> L6_boundary
end
subgraph L7a["Layer 7a — brand_000's continuation merk-tree"]
direction TB
L7a_q["<b>color</b><br/>kv_hash=HASH[fccc...]<br/>value: NonCounted(ProvableCountTree)"]:::queried
L7a_left["HASH[d605...]"]:::sibling
L7a_q --> L7a_left
end
subgraph L7b["Layer 7b — brand_001's continuation merk-tree"]
direction TB
L7b_q["<b>color</b><br/>kv_hash=HASH[8f88...]<br/>value: NonCounted(ProvableCountTree)"]:::queried
L7b_left["HASH[f547...]"]:::sibling
L7b_q --> L7b_left
end
subgraph L8a["Layer 8a — brand_000's byBrandColor color subtree (TARGET 1)"]
direction TB
L8a_target["<b>color_00000500</b><br/>kv_hash=HASH[6834...]<br/>value: <b>CountTree count=1</b><br/>feature: ProvableCountedMerkNode(1)"]:::target
L8a_boundary["37 merk ops:<br/>9 KVHashCount boundary commitments<br/>(running counts 3, 7, 15, 31, 63, 127, 255, 511, 1000)<br/>+ subtree hashes"]:::sibling
L8a_target --> L8a_boundary
end
subgraph L8b["Layer 8b — brand_001's byBrandColor color subtree (TARGET 2)"]
direction TB
L8b_target["<b>color_00000500</b><br/>kv_hash=HASH[881d...]<br/>value: <b>CountTree count=1</b><br/>feature: ProvableCountedMerkNode(1)"]:::target
L8b_boundary["37 merk ops:<br/>same boundary shape as L8a<br/>(different hashes — different brand's subtree)"]:::sibling
L8b_target --> L8b_boundary
end
L5_q -. "Tree(merk_root[byBrand])" .-> L6_t1
L6_t0 -. "CountTree continuation" .-> L7a_q
L6_t1 -. "CountTree continuation" .-> L7b_q
L7a_q -. "NonCounted(ProvableCountTree)" .-> L8a_target
L7b_q -. "NonCounted(ProvableCountTree)" .-> L8b_target
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The two parallel byBrandColor descents share their L1–L6 commitments (the doctype prefix + byBrand merk root) but each gets its own L7 + L8 sub-proof. Proof bytes ≈ shared upper layers + 2 × per-brand byBrandColor descent ≈ 2 842 B.
G4 — Range on byColor, Grouped By color
GroupByRange is the proof primitive that enumerates distinct in-range keys with a count per key, as opposed to chapter 29's AggregateCountOnRange which collapses the same range to a single u64.
select = COUNT
where = color > "color_00000500"
group_by = [color]
prove = true
Path query (uses distinct_count_path_query with limit=100, left_to_right=true):
path: ["@", contract_id, 0x01, "widget", "color"]
query items: [RangeAfter("color_00000500"..)]
limit: 100
Verified payload:
Entries(100 groups, sum = 10 000)
The 100 groups are color_00000501 through color_00000600 (the first 100 in-range colors in lex-asc order, capped by the limit). Each carries count_value_or_default = 100 since the fixture's deterministic schedule gives each color exactly 100 documents.
Wait — but Q7 said there are 499 distinct in-range colors and sum = 49 900 over the same color > "color_00000500" predicate. So why does G4 see only 100 groups summing to 10 000? Because GroupByRange's distinct_count_path_query applies the 100-entry response cap (Some(limit) in execute_distinct_count_with_proof). Without that cap the proof would scale linearly with the full in-range distinct count (~5.5 KB for the full 499 colors at ~110 B per resolved CountTree branch). The cap is a response-size safety control — the verifier ceases the walk once it has 100 entries.
Proof size: 10 992 B — ~5.3 × Q7. The structural reason:
- Q7 (
AggregateCountOnRange) walks the boundary of the range and emits oneHashWithCountorKVDigestCountper merk-binary-tree boundary node. Total boundary nodes ≈O(log C)(≈ 36 ops on the 1 000-color tree). The verifier sums subtree counts directly without descending into individual keys. - G4 (
GroupByRange) walks the distinct in-range colors themselves — emitting oneKVValueHashFeatureTypeWithChildHash(color_X, CountTree count=100, ProvableCountedMerkNode(…), …)per distinct color in the range, not just per merk-tree boundary node. Total ops ≈O(R)whereRis the distinct in-range colors (capped at 100 here).
The trade-off is exactly what you'd expect: AggregateCountOnRange is O(log C) in proof bytes but loses per-key resolution (returns one u64); GroupByRange is O(R) in proof bytes but preserves per-key counts.
Proof display:
Expand to see the structured proof (5 layers; bottom layer enumerates 100 distinct in-range colors as `KVValueHashFeatureTypeWithChildHash` targets, each carrying `CountTree count=100`) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b90e1e952b7eef903cc9db2d9098e334a37f7e08cade52c6b2ea3bf4b56b645])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[49e7191075272395ed72cf03e973987ede6e4945e08574fe77d725f4ce7ecdf8]))
1: Push(KVValueHash(0x01, Tree(776964676574), HASH[5d9a0fad8a3f32560f8e8950c1e84a7feabaab21b79bc72fec4482442844e2ef]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(widget, Tree(6272616e64), HASH[6c505f53f2ebf3de030cc2aca463d4b429aeb320a9fadb8ae68bb7903a22bb68])))
lower_layers: {
widget => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[9862894b16a0792688fdcf64edcb2ceade5c8b234649bfc6cfc6426869b0e9d9]))
1: Push(KVHash(HASH[a29ee8f206a253362b6da4fcacf8643ee8e5925cd979fcd449e5906f0f9f8be3]))
2: Parent
3: Push(KVValueHash(color, ProvableCountTree(636f6c6f725f3030303030353131, 100000), HASH[79569d595db75bbf2e9dca93a15c90b7eecf7b299632668ec410e2076d27f71c]))
4: Child)
lower_layers: {
color => {
LayerProof {
proof: Merk(
... 18 boundary-descent ops walking the binary tree from
root (color_00000511) leftward to the cut point ...
18: Push(KVDigestCount(color_00000500, HASH[47b0ade5...], 100))
// op 18: BOUNDARY (excluded by strict `>`)
19: Push(KVValueHashFeatureTypeWithChildHash(color_00000501,
CountTree(00, 100, flags: [0, 0, 0]),
HASH[9146433eb6d43db2f109f5f7714146624bd646b27c7310f3c2cad7155eb7c741],
ProvableCountedMerkNode(300),
HASH[c285efb8724a488de916ce8301b06c197fc687b5b9b83a04bf3a026f1098d17a]))
// op 19: TARGET 1
20: Parent
21: Push(KVValueHashFeatureTypeWithChildHash(color_00000502, CountTree(00, 100, ...)))
// op 21: TARGET 2
... 98 more KVValueHashFeatureTypeWithChildHash targets
(color_00000503 ... color_00000600), each emitting
`CountTree count=100` plus its merk feature/child-hash glue,
interleaved with Parent/Child ops walking the binary tree
in lex-asc order. Every target shares the same shape:
Push(KVValueHashFeatureTypeWithChildHash(
color_XXXXXXXX,
CountTree(00, 100, flags: [0, 0, 0]),
HASH[...],
ProvableCountedMerkNode(running_count_at_this_node),
HASH[...]
)) ...
220: Push(KVValueHashFeatureTypeWithChildHash(color_00000600,
CountTree(00, 100, ...))) // op 220: TARGET 100 (LAST)
221..244: closing boundary ops — KVHashCount running
counts (300, 700, 6300, 25500, 48800) and Hash subtrees
proving the still-out-of-range portion to the right of
color_00000600 covers the remainder of the merk root.)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
That schematic gives the shape; the bench's [gproof] output (run cargo bench --bench document_count_worst_case and grep [gproof] G4) has all 245 ops verbatim. The compression in the chapter just elides the 100 KVValueHashFeatureTypeWithChildHash targets since they share the same structural template — only the key name, the leaf kv-hash, the running count, and the child-hash differ.
Why so many targets? Because GroupByRange must enumerate every in-range key with its CountTree value — the SDK needs each individual key→count pair, which the aggregate-style HashWithCount commitment hides. So the prover walks the merk binary tree's in-order traversal across the in-range portion (here, left-to-right starting just past color_00000500) and emits one KVValueHashFeatureTypeWithChildHash per distinct color it visits, until the response-size limit is reached.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> CO["color: ProvableCountTree count=100000"]:::path
CO -.-> C500["color_00000500 (boundary, excluded)"]:::faded
CO ==> C501["color_00000501: CountTree count=100"]:::target
CO ==> CMore["color_00000502 ... color_00000600<br/>(98 more in-range targets,<br/>each CountTree count=100)"]:::target
CO ==> C600["color_00000600: CountTree count=100"]:::target
CO -.-> CRest["color_00000601 ... color_00000999<br/>(beyond limit — opaque)"]:::faded
SDK["Verifier returns Entries(100 groups):<br/>("color_00000501", 100),<br/>("color_00000502", 100),<br/>... ("color_00000600", 100)"]:::sdk
C501 -.-> SDK
CMore -.-> SDK
C600 -.-> SDK
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#d29922,color:#0d1117,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
linkStyle 3 stroke:#1f6feb,stroke-width:3px;
linkStyle 4 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
L5 is identical to Q3's / Q6's L5 (color queried under an opaque kv root in the widget doctype tree). L6 is the structural novelty: 245 merk ops, of which 100 are full KVValueHashFeatureTypeWithChildHash targets and the remaining 145 are boundary-walk glue (KVDigestCount / KVHashCount / HashWithCount / Hash + Parent/Child).
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree (proof view for `color`)"]
direction TB
L5_root["KVHash[a29e...]<br/>(opaque kv root)"]:::sibling
L5_left["HASH[9862...]"]:::sibling
L5_q["<b>color</b><br/>kv_hash=HASH[7956...]<br/>value: ProvableCountTree count=100000"]:::queried
L5_root --> L5_left
L5_root --> L5_q
end
subgraph L6["Layer 6 — byColor ProvableCountTree merk-tree (100 in-range targets)"]
direction TB
L6_boundary_l["Left boundary descent (18 ops):<br/>walks from merk root color_00000511<br/>through KVHashCount running counts<br/>(51100, 25500, 12700, 6300, 3100, 700)<br/>down to color_00000500"]:::sibling
L6_cut["op 18: KVDigestCount(color_00000500, ..., 100)<br/>(boundary — excluded by strict `>`)"]:::boundary
L6_targets["ops 19..220: 100 in-range targets<br/>color_00000501 (count=100), color_00000502 (100),<br/>color_00000503 (100), ... color_00000600 (100)<br/>each as KVValueHashFeatureTypeWithChildHash<br/>with ProvableCountedMerkNode(subtree_count)<br/>interleaved with Parent/Child glue"]:::target
L6_boundary_r["Right closing boundary (24 ops):<br/>KVHashCount running counts<br/>(300, 700, 6300, 25500, 48800)<br/>+ Hash subtree commitments<br/>covering color_00000601 ... color_00000999"]:::sibling
L6_boundary_l --> L6_cut
L6_cut --> L6_targets
L6_targets --> L6_boundary_r
end
L5_q -. "ProvableCountTree(merk_root[byColor])" .-> L6_boundary_l
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef boundary fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px,stroke-dasharray: 6 3;
Three things this diagram makes explicit:
- The cut is named.
op 18: KVDigestCount(color_00000500, ..., 100)exposes the key at the boundary so the verifier knows the cut sits exactly betweencolor_00000500(excluded) andcolor_00000501(first in-range). Without that named op, a malicious prover could shift the cut and the verifier wouldn't know. - Targets carry their own count, not a running total. Unlike Q7's boundary commitments (where
ProvableCountedMerkNode(N)carried a subtree count), G4's targets are individual keys withCountTree(00, 100, ...)— thecount_value_or_default = 100IS the per-key count, not a subtree aggregate. TheProvableCountedMerkNode(N)on the merk feature still carries the subtree count (e.g.300forcolor_00000501's subtree), but G4's verifier readscount_value_or_defaultdirectly from the CountTree element. - The right closing boundary doesn't enumerate the rest. Once the limit is hit at
color_00000600, the proof commits the remaining ~399 in-range colors as opaque subtree hashes (KVHashCount+Hashops). The SDK returns only the 100 visible groups; the remainder are provably present but not enumerated. This is the limit's whole point — bound response size without sacrificing soundness on the visible groups.
G5 — Compound In + Range, Grouped By brand, color
select = COUNT
where = brand IN ["brand_000", "brand_001"] AND color > "color_00000500"
group_by = [brand, color]
prove = true
Path query (outer In on byBrand fans out to per-brand distinct_count_path_query on byBrandColor's color terminator):
outer path: ["@", contract_id, 0x01, "widget", "brand"]
outer query items: [Key("brand_000"), Key("brand_001")]
subquery_path: ["color"]
subquery items: [RangeAfter("color_00000500"..)]
subquery limit: 100 (shared across both brands)
Verified payload:
Entries(100 groups, sum = 100)
Two brands × 50 in-range colors per brand = 100 distinct (brand, color) groups visible in the proof. Each (brand_X, color_Y) pair has exactly 1 document by the fixture's deterministic schedule.
Proof size: 11 554 B. Mode: CountMode::GroupByCompound.
This is the most general group-by shape supported on this contract: outer In fan-out × inner GroupByRange walk. Structurally it combines G3's two-branch descent with G4's in-range enumeration per branch. Proof bytes ≈ shared upper-layer descent + 2 × per-brand byBrandColor distinct-walk. The bench's group_by_compound_in_range_proof_limit_100 benchmark uses the same shape with |IN| = 100 brands instead of 2 — yielding 17 256 B at the much higher fan-out.
Proof display:
Expand to see the structured proof (8 layers — same descent skeleton as G3, but each brand's L8 enumerates 50 in-range colors instead of one point-lookup target) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[4a5a28cb1b40226aa35b2f0d502767df13268bdf4678627dbfde26a557acdf73]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => { LayerProof { ... contract_id descent ... } }
// L2..L4 identical to G3 / Q4's first three subgroves
}
}
// L5 widget doctype merk tree: same as G3 — `brand` queried, opaque siblings 9862 / 6c36
// L6 byBrand merk tree: two KVValueHash targets (brand_000 + brand_001), 25 boundary ops
// L7a brand_000's value tree: single key `color` with NonCounted(ProvableCountTree(...))
// L8a byBrandColor's color subtree (under brand_000):
// proof: Merk(
// ... 18 boundary-descent ops walking from the merk root down to color_00000500 ...
// 18: Push(KVDigestCount(color_00000500, HASH[...], 1)) // BOUNDARY, excluded
// 19: Push(KVValueHashFeatureTypeWithChildHash(color_00000501,
// CountTree(00, 1, flags: [0, 0, 0]),
// HASH[4192...], ProvableCountedMerkNode(3), HASH[c3b4...])) // TARGET (brand_000, color_00000501)
// 21: Push(KVValueHashFeatureTypeWithChildHash(color_00000502, CountTree(00, 1, ...))) // TARGET 2
// 24: Push(KVValueHashFeatureTypeWithChildHash(color_00000503, CountTree(00, 1, ...))) // TARGET 3
// ... 47 more KVValueHashFeatureTypeWithChildHash targets, each CountTree(00, 1, ...)
// — color_00000504 ... color_00000550 (50 per-brand_000 targets total) ...
// ... closing boundary ops covering color_00000551 ... color_00000999 for brand_000
// )
// end L8a
// end L7a
// L7b brand_001's value tree: identical structure to L7a, single key `color`
// L8b byBrandColor's color subtree (under brand_001):
// proof: Merk(
// ... 18 boundary-descent ops (different hashes — different brand's subtree) ...
// 18: Push(KVDigestCount(color_00000500, HASH[...], 1))
// 19..220: 50 in-range KVValueHashFeatureTypeWithChildHash(color_X, CountTree(00, 1, ...)) targets
// + interleaved Parent/Child glue + closing boundary ops
// )
// end L8b
// end L7b
// end L6
}
The 344-line verbatim is available via the bench's [gproof] G5 output. The schematic compresses the 50 per-brand KVValueHashFeatureTypeWithChildHash targets at L8a / L8b — they all share the same template (CountTree(00, 1, ...) since each (brand, color) pair has count=1), differing only in key, leaf kv-hash, running count, and child-hash. Once you've seen G3's L8 structure (single target) and G4's L6 structure (100 in-range targets at the doctype level), G5 is precisely the product: two parallel G3-shaped descents that each terminate in a G4-shaped distinct-walk.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree"]:::path
BR ==> B000["brand_000: CountTree count=1000"]:::path
BR ==> B001["brand_001: CountTree count=1000"]:::path
B000 ==> B000_C["brand_000/color: NonCounted(ProvableCountTree)"]:::path
B001 ==> B001_C["brand_001/color: NonCounted(ProvableCountTree)"]:::path
B000_C ==> T000_501["color_00000501: CountTree count=1"]:::target
B000_C ==> T000_more["... 48 more color targets<br/>(brand_000, color_00000502..550)"]:::target
B000_C ==> T000_550["color_00000550: CountTree count=1"]:::target
B001_C ==> T001_501["color_00000501: CountTree count=1"]:::target
B001_C ==> T001_more["... 48 more color targets<br/>(brand_001, color_00000502..550)"]:::target
B001_C ==> T001_550["color_00000550: CountTree count=1"]:::target
SDK["Entries(100 groups, sum=100):<br/>("brand_000", "color_00000501", 1),<br/>...<br/>("brand_001", "color_00000550", 1)"]:::sdk
T000_501 -.-> SDK
T000_more -.-> SDK
T000_550 -.-> SDK
T001_501 -.-> SDK
T001_more -.-> SDK
T001_550 -.-> SDK
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
linkStyle 3 stroke:#1f6feb,stroke-width:3px;
linkStyle 4 stroke:#1f6feb,stroke-width:3px;
linkStyle 5 stroke:#1f6feb,stroke-width:3px;
linkStyle 6 stroke:#1f6feb,stroke-width:3px;
linkStyle 7 stroke:#1f6feb,stroke-width:3px;
linkStyle 8 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
Layers 5–7 are exactly G3's L5–L7. The difference shows up at L8 — instead of a single target per brand (G3's compound point lookup), each brand's L8 walks 50 in-range colors via the same KVValueHashFeatureTypeWithChildHash enumeration G4 uses, plus the boundary descent / closing boundary glue.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree"]
direction TB
L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
end
subgraph L6["Layer 6 — byBrand merk-tree (two intermediate targets)"]
direction TB
L6_t0["<b>brand_000</b> (queried)<br/>CountTree count=1000"]:::queried
L6_t1["<b>brand_001</b> (queried)<br/>CountTree count=1000"]:::queried
end
subgraph L7a["Layer 7a — brand_000's continuation"]
direction TB
L7a_q["<b>color</b> (queried)<br/>NonCounted(ProvableCountTree)"]:::queried
end
subgraph L7b["Layer 7b — brand_001's continuation"]
direction TB
L7b_q["<b>color</b> (queried)<br/>NonCounted(ProvableCountTree)"]:::queried
end
subgraph L8a["Layer 8a — brand_000's byBrandColor distinct-walk"]
direction TB
L8a_targets["50 KVValueHashFeatureTypeWithChildHash targets:<br/>color_00000501 ... color_00000550<br/>each CountTree(00, 1, ...)<br/>+ left/right boundary glue"]:::target
end
subgraph L8b["Layer 8b — brand_001's byBrandColor distinct-walk"]
direction TB
L8b_targets["50 KVValueHashFeatureTypeWithChildHash targets:<br/>color_00000501 ... color_00000550<br/>each CountTree(00, 1, ...)<br/>+ left/right boundary glue<br/>(different hashes — different brand subtree)"]:::target
end
L5_q -. "byBrand" .-> L6_t0
L5_q -. "byBrand" .-> L6_t1
L6_t0 -. "continuation" .-> L7a_q
L6_t1 -. "continuation" .-> L7b_q
L7a_q -. "byBrandColor distinct-range" .-> L8a_targets
L7b_q -. "byBrandColor distinct-range" .-> L8b_targets
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The 50-targets-per-brand limit reflects the shared response-size cap. In the 2-brand case the cap kicks in at 50 colors per brand; if the In set had 1 brand it would be 100 colors; if it had 4 brands it would be 25 each. The dispatcher slices the cap evenly across the In fan-out so the total number of returned entries equals the limit, regardless of how many In branches share it. That's why the bench's [matrix] row for this case shows Entries(len=100, sum=100) rather than len=200, sum=200.
G7 — Carrier In + Range, Grouped By brand
select = COUNT
where = brand IN ["brand_000", "brand_001"] AND color > "color_00000500"
group_by = [brand]
prove = true
Path query (carrier AggregateCountOnRange — outer Keys per In value, ACOR subquery over each brand's color subtree):
path: ["@", contract_id, 0x01, "widget", "brand"]
outer query items: [Key("brand_000"), Key("brand_001")]
subquery_path: ["color"]
subquery items: [AggregateCountOnRange([RangeAfter("color_00000500"..)])]
Verified payload (verifier returns one (in_key, u64) per resolved In branch via GroveDb::verify_aggregate_count_query_per_key):
[("brand_000", 499), ("brand_001", 499)]
Each brand has all 1 000 colors in its byBrandColor terminator; the strict > cut at color_00000500 leaves color_00000501..color_00000999 = 499 in-range colors per brand. Total sum = 998 documents.
Proof size: 4 332 B. Mode: CountMode::GroupByIn routed to DocumentCountMode::RangeAggregateCarrierProof (the new dispatcher arm wired up against grovedb PR #663).
This is the natural answer to "give me a per-brand aggregate count over a colour range", verifiable in a single proof. Without a proof the same request runs the per-In fan-out and folds the brands into one total entry; only the proved answer keeps one entry per brand. Strictly smaller and asymptotically better than the alternative two-field shape G5:
- G5 (compound distinct walk,
group_by = [brand, color]):O(k · R' · log C')bytes; emits oneKVValueHashFeatureTypeWithChildHashper resolved(brand, color)pair → 11 554 B fork=2, R'≈50. Carries per-pair granularity the caller may not want. - G7 (carrier aggregate,
group_by = [brand]):O(k · (log B + log C'))bytes; emits oneHashWithCount/KVDigestCountACOR boundary walk per brand → 4 332 B fork=2, log C'≈10. ~2.7× smaller than G5 for the same input data, at the cost of losing per-color resolution (which thegroup_by = [brand]caller didn't ask for anyway).
The win vs Q8 (brand == X AND color > floor, the same shape with k=1 and group_by = []) is asymptotic: Q8 is 2 656 B, G7 is 4 332 B for k=2. The slope (G7 − Q8) / 1 = +1 676 B per additional In branch matches what you'd expect: each brand adds its own L6 commit + its own L7 + L8 ACOR boundary walk (≈ Q8's L7 + L8 ≈ ~1 700 B), with the L1–L5 prefix amortising once across all branches.
Proof display:
Expand to see the structured proof (8 layers — same skeleton as G5, but each brand's L8 is an ACOR boundary walk instead of a 50-target distinct-walk) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(... root-level descent, identical to every other chapter query ...)
lower_layers: {
@ => { ... contract_id descent ... }
// L2..L4 byte-identical to G3 / G5 (the @/contract_id/0x01/widget chain)
}
}
// L5 widget doctype: brand queried (same as G3 / G5 — opaque siblings 9862 / 6c36)
// L6 byBrand merk-tree: two KVValueHash targets (brand_000 + brand_001), 25 ops
// — same shape as G5's L6
// L7a brand_000's value tree: single key `color` with NonCounted(ProvableCountTree)
// L8a byBrandColor color subtree under brand_000:
// proof: Merk(
// ... 36-37 ACOR boundary ops over color > color_00000500 ...
// 18: Push(KVDigestCount(color_00000500, ..., 1)) // BOUNDARY (excluded)
// 19..35: HashWithCount / KVDigestCount boundary walk
// — same shape as Q8's L8, summing to count=499 for brand_000)
// end L8a
// end L7a
// L7b brand_001's value tree: same single-key shape, different hashes
// L8b byBrandColor color subtree under brand_001:
// proof: Merk(
// ... 36-37 ACOR boundary ops over color > color_00000500 ...
// — same shape, different hashes, summing to count=499 for brand_001)
// end L8b
// end L7b
}
The 186-line full verbatim is available via the bench's [gproof] G7 output. The schematic compresses the L1–L4 doctype prefix (byte-identical to every other 8-layer chapter query) and the two parallel L7+L8 descents (structurally identical to Q8's, with different hashes for each brand). Each brand's L8 contributes ~1 700 B of ACOR boundary commitments — exactly the predicted Q8 - L1..L5 overhead per branch.
Cryptographic guarantee (via grovedb PR #663): every per-brand count is independently committed to the merk root via node_hash_with_count. A malicious prover can't lie about brand_000's count without breaking brand_001's verification (and vice versa) because each carrier ACOR subquery has its own hash chain back to the merk root.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree"]:::path
BR ==> B000["brand_000: CountTree count=1000"]:::path
BR ==> B001["brand_001: CountTree count=1000"]:::path
B000 ==> B000_C["brand_000/color: NonCounted(ProvableCountTree)<br/>ACOR boundary walk (color > color_00000500)"]:::target
B001 ==> B001_C["brand_001/color: NonCounted(ProvableCountTree)<br/>ACOR boundary walk (color > color_00000500)"]:::target
SDK["Entries(2 groups, sum=998):<br/>("brand_000", 499)<br/>("brand_001", 499)"]:::sdk
B000_C -.-> SDK
B001_C -.-> SDK
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
linkStyle 3 stroke:#1f6feb,stroke-width:3px;
linkStyle 4 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
L5–L7 are exactly G5's L5–L7 (widget → byBrand → brand_X's continuation). The difference is at L8: G5 enumerates 50 distinct (brand_X, color_Y) pairs as KVValueHashFeatureTypeWithChildHash targets per brand; G7 walks the same color subtree as an ACOR boundary cut (like Q8's L8), emitting HashWithCount / KVDigestCount ops that commit a single aggregate u64 per brand.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree"]
direction TB
L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
end
subgraph L6["Layer 6 — byBrand merk-tree (two intermediate targets)"]
direction TB
L6_t0["<b>brand_000</b> (queried)<br/>CountTree count=1000"]:::queried
L6_t1["<b>brand_001</b> (queried)<br/>CountTree count=1000"]:::queried
end
subgraph L7a["Layer 7a — brand_000's continuation"]
direction TB
L7a_q["<b>color</b> (queried)<br/>NonCounted(ProvableCountTree)"]:::queried
end
subgraph L7b["Layer 7b — brand_001's continuation"]
direction TB
L7b_q["<b>color</b> (queried)<br/>NonCounted(ProvableCountTree)"]:::queried
end
subgraph L8a["Layer 8a — brand_000's byBrandColor: ACOR cut"]
direction TB
L8a_target["<b>Aggregate count = 499</b><br/>(committed via node_hash_with_count)"]:::target
L8a_ops["~37 merk ops:<br/>KVDigestCount(color_00000500, …) — boundary excluded<br/>+ HashWithCount/KVDigestCount boundary walk<br/>over the in-range portion"]:::sibling
L8a_target --> L8a_ops
end
subgraph L8b["Layer 8b — brand_001's byBrandColor: ACOR cut"]
direction TB
L8b_target["<b>Aggregate count = 499</b><br/>(committed via node_hash_with_count)"]:::target
L8b_ops["~37 merk ops:<br/>same boundary shape as L8a<br/>(different hashes — different brand subtree)"]:::sibling
L8b_target --> L8b_ops
end
L5_q -. "byBrand" .-> L6_t0
L5_q -. "byBrand" .-> L6_t1
L6_t0 -. "continuation" .-> L7a_q
L6_t1 -. "continuation" .-> L7b_q
L7a_q -. "carrier ACOR subquery" .-> L8a_target
L7b_q -. "carrier ACOR subquery" .-> L8b_target
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The "carrier" name comes from grovedb's PR #663 terminology: a carrier query is the outer multi-key query that carries an ACOR subquery into each branch. The ACOR primitive itself is unchanged — it still walks one range over one subtree per invocation — but it can now appear as a subquery item under outer Keys, which is what enables the per-brand aggregate proof shape G7 needs.
G8 — Carrier outer Range + Range, Grouped By brand
select = COUNT
where = brand > "brand_050" AND color > "color_00000500"
group_by = [brand]
limit = (optional; ≤ 10)
prove = true
The platform's MAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT = 10 is both the default (when the caller passes no limit) and a hard ceiling. Callers may pass a smaller limit (1 through 9) to truncate the outer walk further; passing 0 or any value > 10 is rejected with InvalidLimit. See the rationale below.
Path query (the same carrier-ACOR shape as G7, but with a range outer dimension and SizedQuery::limit bounded by the platform max):
path: ["@", contract_id, 0x01, "widget", "brand"]
outer query item: RangeAfter("brand_050"..)
subquery_path: ["color"]
subquery items: [AggregateCountOnRange([RangeAfter("color_00000500"..)])]
SizedQuery::limit: 10 (platform default; caller may request smaller)
Verified payload (verifier returns one (in_key, u64) per in-range outer key, capped at limit, via GroveDb::verify_aggregate_count_query_per_key):
[("brand_051", 499), ("brand_052", 499), …, ("brand_060", 499)]
The bench's 100-brand fixture has 49 brands > "brand_050". The platform's default SizedQuery::limit = 10 caps the carrier at the first 10 (brand_051 … brand_060); each carries the per-brand ACOR count of 499 in-range colors (color_00000501 … color_00000999). Total sum = 10 × 499 = 4 990 documents.
Proof size: 18 022 B. Mode: CountMode::GroupByRange routed to DocumentCountMode::RangeAggregateCarrierProof (the dispatcher distinguishes G7's In-outer shape from G8's Range-outer shape by the carrier clause's operator).
G8 is G7's natural extension from "k specific outer keys" to "L outer keys from an in-range walk." Same carrier proof primitive, same node_hash_with_count commitments per branch, same one-u64-per-branch return shape. The structural differences are exactly two:
- Outer dimension: G7 emits
kKey(serialized_in_value)items in the carrier query; G8 emits a singleRangeAfter(serialized_floor..)(or anyRange*variant) and lets grovedb walk it. - Limit: G8 sets
SizedQuery::limit = Some(L)whereLis the smaller of the caller's request and the platform max. Per grovedb PR #664, this is the load-bearing relaxation — the predecessor PR #663 allowed Range outer items at the validator level but kept the leaf-ACOR rule rejectingSizedQuery::limit, which made unbounded range-outer carriers impractical at any reasonable dataset size (49 brands × ~1 700 B each ≈ 83 KB; with the platform default of 10 we land at 18 KB).
Why the cap exists and where the ceiling lives
The cap bounds the prove-path proof size; the ceiling is a hardcoded compile-time constant for prover/verifier-agreement reasons.
- Proof-size bounding. Proof bytes scale linearly with the limit (~1 700 B per outer match, exactly as for G7). 10 keeps the worst-case proof under 20 KB (Tier-1 for the GroveDB Proof Visualizer's shareable-link guidance — Tier-1 ≤ 20 KB works in every browser and link-preview surface; Tier-2 of 20–50 KB works in browsers but may be truncated in Slack/Discord previews; Tier-3 above 50 KB risks Safari's URL ceiling) — enough for typical "top-N brands by an outer range" queries while avoiding pathological proof sizes. Callers that want a window above 10 entries call repeatedly with disjoint outer-range bounds; callers that want fewer pass a smaller
limit(1 through 9). Limit 0 is rejected to keep the response shape non-trivial. - Prover/verifier byte-for-byte agreement.
SizedQuery::limitis part of the serializedPathQueryand feeds the merk-root reconstruction; both prover and verifier must agree on its value. The caller's request carrieslimitover the wire, so its specific value (1..=10) is fine to vary. What can't vary is the platform's default when the caller passes nothing — that's why the ceiling is a hardcoded compile-time constant (MAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT) rather than an operator-tunable runtime value. Same rationale asRangeDistinctProof's use ofcrate::config::DEFAULT_QUERY_LIMITrather thandrive_config.default_query_limit.
Caller semantics summary:
Caller request.limit | Server uses | Reason |
|---|---|---|
None | 10 (the platform default) | Default = ceiling |
Some(1..=10) | the caller's value | Truncates the walk further |
Some(0) | rejected | Non-trivial response required |
Some(11+) | rejected | Above the ceiling |
Complexity: O(L · (log B + log C')) where L = min(caller_limit, MAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT) — L outer-key descents in the byBrand layer + L leaf-ACOR boundary walks in each brand's color subtree. Independent of how many keys the outer range could have walked without the cap.
Proof display:
Expand to see the structured proof (8 layers — same skeleton as G7, but L8 contains 10 per-brand ACOR boundary walks instead of 2) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(... root-level descent, identical to every other chapter query ...)
lower_layers: {
@ => { ... contract_id descent ... }
// L2..L4 byte-identical to G3 / G5 / G7 (the @/contract_id/0x01/widget chain)
}
}
// L5 widget doctype: brand queried (same as G3 / G5 / G7)
// L6 byBrand merk-tree: 10 outer-key matches inlined as KVValueHash items
// (brand_051 ... brand_060), each descending into its
// continuation. Boundary commitments cover the
// brands_outside_the_limited_window.
// L7 brand_NNN's value tree: single key `color` with NonCounted(ProvableCountTree)
// — repeated 10 times, once per resolved outer brand
// L8 brand_NNN's byBrandColor color subtree:
// proof: Merk(
// ... 36-37 ACOR boundary ops over color > color_00000500,
// summing to count = 499 per brand ...
// )
// — repeated 10 times in parallel, each with its own per-brand boundary hashes
}
The 618-line full verbatim is available via the bench's [gproof] G8 output. The schematic compresses the 10 parallel L7+L8 descents — they share the same template (single-key continuation + 37-op ACOR boundary walk), differing only in per-brand kv-hashes and the resulting subtree commits. Each per-brand L8 contributes ~1 700 B of ACOR boundary commitments — exactly the predicted Q8 - L1..L5 overhead per outer match, scaling linearly: 18 022 B ≈ shared upper layers + 10 × ~1 700 B ≈ 18 KB (matches the per-In slope from G7 vs Q8).
Cryptographic guarantee (via grovedb PR #663 + PR #664): every per-brand count is independently committed to the merk root via node_hash_with_count. The SizedQuery::limit is part of the serialized PathQuery and is part of the merk-root reconstruction the verifier performs — a malicious prover can't truncate the outer walk at a different point without breaking the hash chain.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree"]:::path
BR ==> B051["brand_051: CountTree count=1000"]:::path
BR ==> BMore["… 8 more in-range brands (brand_052 … brand_059) …"]:::path
BR ==> B060["brand_060: CountTree count=1000"]:::path
BR -.-> BCapped["brand_061 … brand_099<br/>(beyond platform cap — opaque subtree commitments)"]:::faded
BR -.-> BBelow["brand_000 … brand_050<br/>(below range floor — boundary commitments)"]:::faded
B051 ==> B051_C["brand_051/color: NonCounted(ProvableCountTree)<br/>ACOR boundary walk (color > color_00000500)"]:::target
BMore ==> BMore_C["8 parallel ACOR walks"]:::target
B060 ==> B060_C["brand_060/color: NonCounted(ProvableCountTree)<br/>ACOR boundary walk (color > color_00000500)"]:::target
SDK["Entries(10 groups, sum=4 990):<br/>("brand_051", 499)<br/>("brand_052", 499)<br/>…<br/>("brand_060", 499)"]:::sdk
B051_C -.-> SDK
BMore_C -.-> SDK
B060_C -.-> SDK
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
linkStyle 3 stroke:#1f6feb,stroke-width:3px;
linkStyle 6 stroke:#1f6feb,stroke-width:3px;
linkStyle 7 stroke:#1f6feb,stroke-width:3px;
linkStyle 8 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
L5 is identical to G7's L5 (widget doctype with brand queried). L6 differs: G7 inlined 2 KVValueHash targets for the In-bearing brands; G8 inlines 10 KVValueHash targets for the in-range brands the carrier walks (brand_051 through brand_060), with boundary commitments covering both the below-floor and beyond-cap portions of the byBrand merk tree. L7 + L8 fork into 10 parallel descents, each shaped exactly like G7's L7 + L8 — same NonCounted(ProvableCountTree) continuation, same 37-op ACOR boundary walk over color > color_00000500.
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree"]
direction TB
L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
end
subgraph L6["Layer 6 — byBrand merk-tree (10 outer-range targets)"]
direction TB
L6_t051["<b>brand_051</b><br/>CountTree count=1000"]:::queried
L6_tmid["… 8 more in-range targets …<br/>(brand_052 … brand_059)"]:::queried
L6_t060["<b>brand_060</b><br/>CountTree count=1000"]:::queried
L6_capped["Beyond-cap commitments:<br/>brand_061 … brand_099<br/>(opaque KVHash / Hash ops)"]:::sibling
L6_floor["Below-floor commitments:<br/>brand_000 … brand_050<br/>(opaque)"]:::sibling
L6_t051 --> L6_tmid
L6_tmid --> L6_t070
L6_t070 --> L6_capped
L6_t051 --> L6_floor
end
subgraph L7L8["Layers 7+8 — per-brand continuation + ACOR walk (×10)"]
direction TB
L7L8_each["For each of brand_051 … brand_060:<br/>L7: single-key `color` continuation (NonCounted(ProvableCountTree))<br/>L8: 37 merk ops — ACOR boundary walk for color > color_00000500<br/>committing one `u64 = 499` per brand"]:::target
end
L5_q -. "byBrand" .-> L6_t051
L6_t051 -. "continuation × 20" .-> L7L8_each
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The slope vs G7 is the proof's whole story: G7's k = 2 outer matches → ~4 KB; G8's L = 10 outer matches → ~18 KB. The per-outer-match cost (~1 700 B) is the same; only the outer-walk count changes. The platform max of 10 keeps the worst-case proof under 20 KB (Tier-1 of the visualizer's shareable-link guidance); larger windows are unreachable without changing the constant — callers that want more results call repeatedly with disjoint outer-range windows.
G8a — Bounded carrier + bounded ACOR, grouped by brand, descending
select = COUNT
where = brand > "brand_050" AND brand < "brand_065"
AND color > "color_00000200" AND color < "color_00000400"
group_by = [brand]
order_by = [(brand, desc)]
prove = true
G8a stresses three carrier-ACOR dimensions G8 didn't: a bounded outer range (instead of half-open), a bounded inner ACOR (instead of > floor), and a descending walk (instead of left-to-right ascending). All three orthogonal. Same RangeAggregateCarrierProof mode, same path-query builder; the differences live entirely in the per-clause QueryItem variants and the carrier's left_to_right flag.
Path query (the carrier query items differ from G8 in three ways: outer item is RangeAfterTo instead of RangeAfter, inner ACOR item is RangeAfterTo instead of RangeAfter, and outer_query.left_to_right = false):
path: ["@", contract_id, 0x01, "widget", "brand"]
outer query item: RangeAfterTo("brand_050".."brand_065") // exclusive bounds
subquery_path: ["color"]
subquery items: [AggregateCountOnRange([RangeAfterTo("color_00000200".."color_00000400")])]
SizedQuery::limit: 10 // platform default
outer Query.left_to_right: false // from order_by [(brand, desc)]
Same-field range merging. The caller's wire shape carries four range clauses (brand >, brand <, color >, color <). The dispatcher merges each same-field pair into a single BetweenExcludeBounds clause via merge_same_field_range_pairs before mode detection runs. After merging, the structure is identical to G8's two-range shape; mode detection routes to RangeAggregateCarrierProof for the same reasons.
Verified payload (descending walk — outer keys come out from highest to lowest, capped at L = 10):
[("brand_064", 199), ("brand_063", 199), …, ("brand_055", 199)]
The bench's 100-brand fixture has 14 brands strictly between "brand_050" and "brand_065" (i.e. brand_051 through brand_064). The descending walk starts at brand_064 and runs left-to-right=false through the byBrand merk tree; the SizedQuery::limit = 10 halts the walk after 10 outer matches (brand_064 down to brand_055). Each brand's inner ACOR over color > "color_00000200" AND color < "color_00000400" sums to 199 documents (199 colors color_00000201 … color_00000399, one document per (brand, color) pair in the fixture). Total sum = 10 × 199 = 1 990.
Proof size: 29 010 B. Mode: CountMode::GroupByRange routed to DocumentCountMode::RangeAggregateCarrierProof.
G8a is structurally G8 with three independent variant changes, each adding a small amount of merk-proof overhead but no asymptotic complexity change:
- Bounded outer range → the byBrand merk tree commits both bounds (
brand_050lower-exclusive +brand_065upper-exclusive) as boundaryKVDigestops. G8's>-only outer commits one boundary; G8a's>AND<commits two. Modest size delta (~1 extraKVDigestper bound × the carrier's tree depth). - Bounded inner ACOR → each per-brand color subtree commits both bounds as
KVDigestCountops. G8's>-only ACOR walksO(log C')boundary nodes for the lower bound; G8a's two-sided ACOR walksO(log C')for both bounds. The asymptotic staysO(L · (log B + log C')); the constant roughly doubles for the per-brand boundary walk. - Descending walk → grovedb emits
PushInverted(...)op variants instead ofPush(...)and walks the binary merk tree right-to-left. Same op count as ascending, slightly different serialized encoding (~1–2 bytes per op for thePushInvertedopcode discriminant). The verifier's reconstruction is byte-identical given the sameleft_to_rightflag in thePathQuery.
Total proof bytes: 29 010 B vs G8's 18 022 B. Per-outer-match overhead: ~2 900 B (G8a) vs ~1 700 B (G8). The extra ~1 200 B per branch is the bounded-inner-ACOR cost — every per-brand subtree commits twice as many boundary KVDigestCount ops.
Proof display:
Expand to see the structured proof (8 layers; L8 uses two-sided ACOR boundary walks per brand, `PushInverted` outer-walk ops for descending direction) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(... root-level descent, identical to every other chapter query ...)
lower_layers: {
@ => { ... contract_id descent ... }
// L2..L4 byte-identical to every 8-layer carrier query in this chapter
}
}
// L5 widget doctype: brand queried (same as G3 / G5 / G7 / G8)
// L6 byBrand merk-tree: walked LEFT-TO-RIGHT=FALSE (descending).
// Outer query item: RangeAfterTo("brand_050".."brand_065")
// Inlined targets: brand_064 → brand_063 → ... → brand_055
// via `PushInverted(KVValueHash(brand_NNN, CountTree, ...))` ops.
// Boundary KVDigest nodes name brand_065 (upper-exclusive cut)
// and brand_050 (lower-exclusive cut, capped by SizedQuery::limit).
// L7 brand_NNN's value tree: single key `color` with NonCounted(ProvableCountTree)
// — repeated 10 times, once per resolved outer brand (in descending order).
// L8 brand_NNN's byBrandColor color subtree:
// proof: Merk(
// ... ACOR boundary walk for color > "color_00000200" AND color < "color_00000400"
// (two-sided cut, ~2× the boundary ops of G8's one-sided ACOR),
// summing to count = 199 per brand ...
// )
// — repeated 10 times in parallel, each with its own per-brand boundary hashes.
}
The 902-line full verbatim sits in the bench's [gproof] G8a output. The schematic compresses the 10 parallel L7+L8 descents and the per-brand boundary commitments — they share the same template (single-key continuation + ~50-op two-sided ACOR boundary walk), differing only in per-brand hashes and the resulting subtree commits. Each per-brand L8 contributes ~2 800 B of ACOR boundary commitments (~1.6× G8's ~1 700 B due to the two-sided range walking both bounds).
The most visually distinctive feature of the descending-walk proof: every L6 carrier op is PushInverted(...) rather than Push(...), signalling grovedb's right-to-left binary-merk-tree iteration. Identical merk-root reconstruction given the same Query.left_to_right = false flag — but the wire-level encoding diverges so the verifier knows which direction to walk.
flowchart TB
WD["@/contract_id/0x01/widget"]:::tree
WD ==> BR["brand: NormalTree (descending walk, left_to_right=false)"]:::path
BR ==> B064["brand_064: CountTree count=1000"]:::path
BR ==> BMore["brand_063 … brand_056<br/>(8 more in-range brands, descending)"]:::path
BR ==> B055["brand_055: CountTree count=1000"]:::path
BR -.-> BBelow["brand_051 … brand_054<br/>(in range but below cap — beyond limit, opaque)"]:::faded
BR -.-> BAbove["brand_065 (boundary key, excluded by <)"]:::faded
BR -.-> BCapBelow["brand_000 … brand_050<br/>(below floor, opaque)"]:::faded
B064 ==> B064_C["brand_064/color: NonCounted(ProvableCountTree)<br/>two-sided ACOR (color > 200 AND color < 400)"]:::target
BMore ==> BMore_C["8 parallel two-sided ACOR walks<br/>(color > 200 AND color < 400)"]:::target
B055 ==> B055_C["brand_055/color: NonCounted(ProvableCountTree)<br/>two-sided ACOR (color > 200 AND color < 400)"]:::target
SDK["Entries(10 groups, sum=1 990) — DESCENDING:<br/>("brand_064", 199)<br/>("brand_063", 199)<br/>…<br/>("brand_055", 199)"]:::sdk
B064_C -.-> SDK
BMore_C -.-> SDK
B055_C -.-> SDK
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef path fill:#6e7681,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef faded fill:#21262d,color:#6e7681,stroke:#484f58;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef sdk fill:#21262d,color:#39c5cf,stroke:#39c5cf,stroke-width:2px,stroke-dasharray: 4 2;
linkStyle 0 stroke:#1f6feb,stroke-width:3px;
linkStyle 1 stroke:#1f6feb,stroke-width:3px;
linkStyle 2 stroke:#1f6feb,stroke-width:3px;
linkStyle 3 stroke:#1f6feb,stroke-width:3px;
linkStyle 7 stroke:#1f6feb,stroke-width:3px;
linkStyle 8 stroke:#1f6feb,stroke-width:3px;
linkStyle 9 stroke:#1f6feb,stroke-width:3px;
Diagram: per-layer merk-tree structure (Layer 5+)
L5 is identical to G7 / G8 (widget doctype with brand queried). L6 differs from G8 in two ways: the outer query item is RangeAfterTo (bounded) rather than RangeAfter (half-open), and every op is PushInverted rather than Push because of left_to_right = false. L7 + L8 fork into 10 parallel descents, each carrying a two-sided ACOR boundary walk over color > "color_00000200" AND color < "color_00000400" instead of G8's one-sided color > "color_00000500".
flowchart TB
subgraph L5["Layer 5 — widget doctype merk-tree"]
direction TB
L5_q["<b>brand</b> (queried)<br/>kv_hash=HASH[68b6...]"]:::queried
end
subgraph L6["Layer 6 — byBrand merk-tree (bounded outer range, descending walk, 10 targets)"]
direction TB
L6_t064["<b>brand_064</b><br/>PushInverted(KVValueHash …)<br/>CountTree count=1000"]:::queried
L6_tmid["… 8 more in-range targets …<br/>(brand_063 → brand_056, descending)"]:::queried
L6_t055["<b>brand_055</b><br/>PushInverted(KVValueHash …)<br/>CountTree count=1000"]:::queried
L6_upper["Upper-bound commitment:<br/>KVDigest(brand_065, …) — excluded by <"]:::boundary
L6_lower["Below-cap + below-floor commitments:<br/>brand_051 … brand_054 (capped)<br/>+ brand_000 … brand_050 (below floor)<br/>(opaque KVHash / Hash ops)"]:::sibling
L6_t064 --> L6_tmid
L6_tmid --> L6_t055
L6_t064 --> L6_upper
L6_t055 --> L6_lower
end
subgraph L7L8["Layers 7+8 — per-brand continuation + two-sided ACOR walk (×10)"]
direction TB
L7L8_each["For each of brand_064 … brand_055 (descending):<br/>L7: single-key `color` continuation (NonCounted(ProvableCountTree))<br/>L8: ~50 merk ops — two-sided ACOR boundary walk<br/>for color > 200 AND color < 400<br/>committing one `u64 = 199` per brand"]:::target
end
L5_q -. "byBrand" .-> L6_t064
L6_t064 -. "continuation × 10" .-> L7L8_each
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef boundary fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px,stroke-dasharray: 6 3;
The size delta between G8 and G8a, per outer match: ~1 700 B (G8) → ~2 800 B (G8a). The extra ~1 100 B per brand is roughly evenly split between (a) the bounded inner ACOR's second boundary walk and (b) the per-op PushInverted discriminant overhead. Both costs are linear in L (the platform-max outer cap), so doubling L doubles the delta. The asymptotic complexity stays O(L · (log B + log C')) — the bounded-vs-unbounded distinction is a constant-factor change in the per-walk boundary commit count, not a complexity-class change.
Reading the descending result: the SDK returns Vec<(Vec<u8>, u64)> in the same wire order grovedb walked the outer dimension. For left_to_right = false, that's lex-descending serialized brand keys (brand_064 before brand_063 before … before brand_055). Callers that expect ascending output sort the result client-side; the prove-path guarantee is on the contents (which brands and which counts), not the client-visible ordering — though for chapter-fixture-deterministic proofs the ordering IS visible in the proof bytes via Push vs PushInverted, so the verifier knows which direction grovedb walked.
G8b — Two-range carrier with group_by = [brand, color] (rejected)
select = COUNT
where = brand > "brand_050" AND color > "color_00000500"
group_by = [brand, color]
prove = true
Outcome: Err(QuerySyntaxError::InvalidWhereClauseComponents("count query supports at most one range where-clause; combine two-sided ranges via between*instead of separate>/<clauses, or usegroup_by = [outer_range_field]withprove = true for the carrier-aggregate shape with one outer range and one inner ACOR range on a different field")) — at detect_mode's range_count > 1 short-circuit, before any index picking or path-query building.
Why. The two-range carrier shape (outer_range AND inner_range on distinct fields) is opened by mode detection only when mode == GroupByRange and group_by.len() == 1 and prove = true. G8b violates the first two: with group_by = [brand, color] the request maps to CountMode::GroupByCompound, which routes to distinct_count_path_query — a builder that knows how to walk an In + range fan-out but not a range + range cartesian product. Two design points:
GroupByCompoundis specifically the(In, range)shape. Its path-query builder emits outerKey(serialized_in_value)items (one per In branch) and an innerRange*subquery; the walk is|In|-bounded by construction. Extending it to acceptrange + rangewould mean replacing the outerKeys with an outerRange*(and aSizedQuery::limitto bound the walk) and swapping the inner from "enumerate distinct values" to "single ACOR aggregate" — at which point the result shape stops being "per-distinct-value entries" and becomes "per-outer-keyu64s," i.e. G8's shape with a redundant second group_by field. There's no information gain from addingcolorto the group_by — the carrier already commits oneu64per outerbrand, and the inner range collapses into thatu64rather than being enumerated.- The carrier primitive returns one
u64per outer key, not per(outer, inner)pair. Per-distinct-color counts inside an outer-range brand walk would require the alternativeRangeDistinctProofshape (the G5 compound-distinct path) running on abyBrandColor + rangeCountable: truecartesian fan-out — which works forIn + range(a finite outer key set) but would explode forrange + range(potentiallyB × C'distinct entries, dwarfing theMAX_CARRIER_AGGREGATE_OUTER_RANGE_LIMIT = 10cap that bounds G8). The dispatcher rejects rather than silently routing to a path that'd produce a proof orders of magnitude larger than the caller likely expected.
What to use instead.
- If you want per-brand totals across an in-range color window (the most common interpretation of this request), use G8 (
group_by = [brand]): oneu64per brand, capped at 10 outer matches. - If you want per-
(brand, color)distinct counts across both ranges, the dispatcher has no path today — you'd need abyBrandColor + rangeCountable: trueindex plus a new mode that extendsGroupByCompoundtorange + rangewith a per-pairSizedQuery::limit. Out of scope for this contract. - If you want a single sum across the whole
brand > X AND color > Ywindow, you'd need to call G8 and sum the returnedu64s client-side (server-side aggregation across the carrier's per-branch counts isn't supported on the prove path — see G8c below).
G8c — Two-range carrier with group_by = [] (rejected)
select = COUNT
where = brand > "brand_050" AND color > "color_00000500"
group_by = []
prove = true
Outcome: same rejection as G8b — Err(QuerySyntaxError::InvalidWhereClauseComponents("count query supports at most one range where-clause; …")). Mode-detection's range_count > 1 short-circuit checks mode == GroupByRange, and the dispatcher maps group_by = [] to CountMode::Aggregate, so the check fails for the same structural reason as G8b.
Why. With no group_by the request asks for a single scalar u64 covering every document matching both ranges. The carrier-ACOR primitive emits one u64 per outer-range key (10 brands in G8's case), not a single sum across the whole walk. Two paths to a single sum, neither viable today:
- Server-side sum across the carrier's branches. Would require a new grovedb primitive that takes the carrier shape and emits
Σ branch_countsas a single ACOR-style aggregate. Not implemented — the carrier's commitment is per branch, which is what gives the verifier the cryptographic granularity to verify each entry independently. Summing in the server would lose that and force the verifier to trust the server's sum. - Client-side sum after running G8. Allowed and easy — call G8, get back
Vec<(brand, u64)>, sum theu64s. The proof still cryptographically commits to each branch, and the client's sum is over verified data. This is the pragmatic path for "give me one number" callers; the chapter recommends it instead of opening upAggregatefor the two-range carrier shape.
The deeper reason Aggregate can't shortcut this. Per chapter 29's Q7 (Range Aggregate byColor), Aggregate + single range uses the leaf-level AggregateCountOnRange primitive directly, which DOES return a single u64. That works because the range is rooted at the index's terminator property — there's a single CountTree under which the boundary walk runs. With G8c's two ranges, the outer range walks the byBrand merk tree (no ProvableCountTree involved) and only the inner range hits the rangeCountable terminator. Collapsing across the outer walk would mean a ProvableCountTree over CountTrees, which grovedb's primitive set doesn't have. The walk could in principle compute and emit a sum at the outer layer, but the verifier wouldn't be able to recompute the per-branch counts to check the sum — defeating the prove-path's whole point.
Future Work
This chapter now mirrors chapter 29's per-query structure: every section above carries a path query, verified payload, proof size, verbatim or schematic proof display, narrative, conceptual flowchart, and per-layer merk-tree diagram.
Two pieces of infrastructure made this possible:
query_g1_*…query_g8_*criterionbench_functioncalls (the series skipsg6) indocument_count_worst_case.rs— produce the Avg time column in Queries in this Chapter.display_group_by_proofs(a sibling ofdisplay_proofsin the same bench file) — emits eachgroup_byshape's verbatim merk-proof structure via bincode decode +GroveDBProof::Display. Tagged with[gproof]prefix in stderr so reviewers can grep deterministically.
Open follow-ups:
- Inline the full G4 / G5 / G1b verbatim rather than the schematic-with-elision form. The bench captures every byte; the chapter's
<details>blocks currently summarise the 100-target enumerations because reproducing 100 near-identicalKVValueHashFeatureTypeWithChildHashlines per case is more noise than signal. If a reader needs byte-exact output, they can run the bench and grep[gproof]. - Wire path-query reconstruction + verified-payload printing into
display_group_by_proofs. Today it only dumps the proof-display block; chapter 29'sdisplay_proofsalso reconstructs thePathQueryand prints the verifier's structured result (theverified:block). Adding that to the group_by side would give the chapter parity with chapter 29'sverified:sections — currently rendered manually from the[matrix]output'sEntries(len=N, sum=M)figures. - A high-fanout byColor variant of G1b (
color IN [100 values],group_by = [color]) — captured implicitly in the bench's existinggroup_by_color_in_proof_100_rangecountable_branches(10 512 B) but not given its own G* section, since it's structurally G1b withProvableCountTreeoverhead.
Cross-Reference to Chapter 29
For background on the building blocks every query in this chapter uses:
- Document Count Trees —
CountTree/ProvableCountTree/NormalTreemechanics. - Count Index Examples § How To Read The Proofs — the four-section per-query template plus the
LayerProof/Merk/Push/Parent/Childop grammar. - Count Index Examples § Worked Example: How
node_hash_with_countRebuilds the Merk Root — exact Blake3 formulas underpinning every count proof in either chapter.
The path-query builder (packages/rs-drive/src/query/drive_document_count_query/path_query.rs) and verifier mirror (packages/rs-drive/src/verify/document_count/) live in the same modules for both chapters' queries — the only difference is which point_lookup_* / aggregate_* / distinct_count_* / carrier_aggregate_* function the dispatcher calls based on the CountMode carried in the request.
Document Sum Trees
Summing a numeric property across the documents that match a query used to mean fetching them all and adding values up client-side. The grovedb upgrade that landed alongside Document Count Trees adds provable sum trees and references with sum item as primitives — the building blocks Drive uses to turn sum(amount)-style queries into an O(log n) provable lookup. This chapter explains the three sum-tree variants, how a document type opts into one, the unified GetDocumentsSum endpoint that exposes the feature, and the parallels with the count-tree machinery.
Status: the grovedb-level sum-tree primitives (
SumTree,ProvableSumTree,BigSumTree, and reference elements that carry a sum-item contribution) are in place. The Drive-level schema syntax, query handler, and SDK surfaces described below are the proposed design — the Sum Index Examples chapter is the worked-example companion, and the tip-jar contract fixture atpackages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.jsonis the schema this design targets.
Why Sum Trees Exist
The default primary-key tree for a document type is a NormalTree. To total the amount field across its documents, Drive walks the subtree, deserializes every record, sums the property client-side, and returns the result. That is fine for small types but becomes painful as soon as a UI needs "how much has this creator received in tips?" on a tip jar with millions of entries — and worse if the caller wants a proof of the total, because the proof has to enumerate every contributing document.
GroveDB has two sum-aware tree variants. Both are provable — the running sum is committed to the merk root in each case — but they differ in where the sum is stored inside the tree, and that controls which kinds of sum queries can be answered without enumerating leaves:
SumTree— stores a singlei64sum at the root of the tree. The total sum is one read; any per-subtree sum requires walking down to that subtree's root and reading its (separate) tree element.ProvableSumTree— stores ani64sum at every internal node, not just the root. Each node's sum covers everything in the subtree below it, so range queries like "what's the total amount tipped between time A and time B?" or "what's the sum per recipient over time?" can be answered by walking the boundary nodes and combining their pre-computed sub-sums, without touching any leaf.
GroveDB merk trees are binary — each internal node has exactly a left and a right child:
The dashed box is the wrapping Element (the "tree" in grovedb terms) and contains the root node of the merk tree. Both variants store the total sum on the wrapping element — that's the O(1) field Drive reads for total sums. The difference is what's inside: in a SumTree the merk root and the rest of the tree don't carry the sum, so only the wrapper has it. In a ProvableSumTree the sum is also stored on the root node itself and on every internal merk-tree node, so it's committed into the merk root hash and provable per-subtree.
flowchart LR
subgraph ST ["SumTree"]
direction TB
subgraph ST_ELEM ["Tree element s=18"]
direction TB
A["root"]:::node
end
A --> B["·"]:::node
A --> C["amt=5"]:::leaf
B --> D["amt=10"]:::leaf
B --> E["amt=3"]:::leaf
end
subgraph PST ["ProvableSumTree"]
direction TB
subgraph PST_ELEM ["Tree element s=18"]
direction TB
H["root s=18"]:::sumnode
end
H --> I["s=13"]:::sumnode
H --> J["amt=5"]:::leaf
I --> K["amt=10"]:::leaf
I --> L["amt=3"]:::leaf
end
ST ~~~ PST
classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
classDef sumnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;
style ST_ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb
style PST_ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb
In a SumTree, the only sum-bearing node is the root. To compute "total amount tipped per recipient" you'd have to navigate to each recipient-keyed subtree (a separate grovedb tree, not a child node of the binary structure above), read its root sum, and pay for a separate proof per read — N reads for N distinct recipients. In a ProvableSumTree, every internal node along the binary path already carries the sum of its left and right subtrees, so a range query like "amounts where sentAt ∈ [t1, t2]" walks only the boundary path and combines the pre-committed sub-sums in a single traversal and a single proof.
A document type opts in via two schema flags. Note that — unlike count, where the flag is a plain bool — sum needs to know which property to sum, so both flags carry a property name:
documentsSummable: "<property>"→ primary-key tree is aSumTreesumming the named property. Enables O(1) total-sum for the document type; sufficient forGetDocumentsSumwith nowherefilter.rangeSummable: true(paired withdocumentsSummable) → primary-key tree is aProvableSumTree. The same flag is also accepted per-index, where it controls range-sum storage layout (see below) and is required for anyGetDocumentsSumrequest that carries a range where-clause.
The named property must be type: integer and listed in the document type's required array. We don't define a null contribution rule — a missing-or-null value would have to either contribute 0 (and silently mask a misconfigured insert) or fail the insert (and invalidate documents that were valid at write time). Requiring the property avoids both choices.
How a Document Type Picks Its Tree Variant
Selection lives in DocumentTypePrimaryKeyTreeType::primary_key_tree_type — the same dispatcher that picks the count-tree variant — extended to consider sum flags alongside count flags:
#![allow(unused)] fn main() { // proposed v1 selection logic — **sum-only projection** for chapter clarity. // The real dispatcher also picks the combined count+sum variants // (`CountSumTree`, `ProvableCountSumTree`, // `ProvableCountProvableSumTree`) when count flags are set alongside // the sum flags — those branches are omitted here and covered in // "Choosing What to Set" below. The v0 count-only logic stays in // place behind a version bump. match (range_summable, documents_summable, range_countable, documents_countable) { (true, _, _, _) => Ok(TreeType::ProvableSumTree), (false, true, _, _) => Ok(TreeType::SumTree), (false, false, true, _) => Ok(TreeType::ProvableCountTree), (false, false, false, true) => Ok(TreeType::CountTree), (false, false, false, false) => Ok(TreeType::NormalTree), } }
primary_key_tree_type() stays the single source of truth — every Drive code path that needs to know which tree variant to read from or write to routes through this helper, including:
- Contract insert and update (to
CREATEthe right tree element when the document type is added). - Document insert / delete (to know how to update the sum alongside the document — adding
amountto every ancestor sum field, decrementing on delete). - Cost estimation (so fees match the variant that will actually be used).
The contract insert/update paths use thin Drive helpers parallel to the existing count variants and the count chapter's batch_insert_empty_*_tree family:
batch_insert_empty_tree— NormalTree.batch_insert_empty_sum_tree— SumTree, used whendocuments_summable.is_some() && !range_summable().batch_insert_empty_provable_sum_tree— ProvableSumTree, used whenrange_summable().
A sum tree's contents under each value-keyed path are inserted via a different family of helpers — the reference-with-sum-item primitive. Where a non-summable index stores a reference at [index_value]/[0]/<doc_id>, a summable index stores a reference that also carries an i64 sum contribution (the document's amount value at write time). When the parent tree is a SumTree or ProvableSumTree, each insert's sum contribution propagates up the merk path, exactly as a count contribution does in the count case — but the contribution is amount rather than +1. Helpers:
batch_insert_sum_item— drops a bare sum item under a sum tree.batch_insert_reference_with_sum_item— drops a reference that contributes a named amount to its parent sum tree. This is the helper non-primary-key sum indexes use.
Each helper goes through LowLevelDriveOperation::for_known_path_key_*_sum_* (or its _estimated_path_key_* cousin in cost-estimation paths), so the contract setup, document operations, and proof generation all see the same on-disk shape.
Storage-Layout Invariants
Because the tree variant is fixed at contract-creation time and baked into how the tree element is laid out on disk, both flags are immutable across a contract update — and the named summable property is too, since changing which property feeds the sum would silently invalidate every existing aggregation:
- Changing
documents_summablefrom any state to any other state (including changing the property name) on avalidate_configupdate returnsDocumentTypeUpdateError. - Same for
range_summable.
Tests pinning these guards will live alongside the existing count-tree tests in packages/rs-dpp/src/data_contract/document_type/methods/validate_update/v0/mod.rs. Don't relax them: if a NormalTree-backed document type were silently switched to SumTree mid-contract, every subsequent insert or delete would update a sum value attached to a tree element that physically isn't a sum tree, leading to grovedb element-shape errors at best and consensus drift at worst.
The named property's value is read at insert time and frozen into the reference-with-sum-item — Drive doesn't re-read it on delete (it pulls the contribution from the reference itself). So changing the property's value would require a delete-then-reinsert; document mutability concerns apply normally.
Summing Documents at Query Time
A single unified gRPC endpoint exposes the feature: GetDocumentsSum — structurally identical to GetDocumentsCount. The response shape varies by request mode (total / per-In-value / per-distinct-value-in-range / total-over-range), see Range Modes below. The wire-level shape mirrors count: on the no-proof path the response's SumResults carries an inner oneof variant { sint64 aggregate_sum; SumEntries entries; } — total-sum and range-without-distinct modes return aggregate_sum (a single i64), per-In-value and per-distinct-value-in-range modes return entries (a list of SumEntry { optional bytes in_key; bytes key; sint64 sum } where in_key is the prefix value for compound In + range shapes and absent for flat queries). The endpoint has two underlying paths (prove vs. no-prove); every mode is valid on both paths.
The two-path / two-shape split is identical to the count endpoint's, and for the same reasons. What's new in the sum case:
- Sums are signed (
i64/sint64on the wire) — grovedb's sum trees model overflow into negative space rather than saturating, so a verifier extracting a sum from a proof can detect "this aggregation overflowedi64::MAX" by recovering a value that doesn't match the document set's expected magnitude. If you expect aggregations beyondi64::MAX, use aBigSumTree-backed variant (bigDocumentsSummable: "amount"— out of scope for this chapter; covered alongside theBigSumTreeDrive plumbing). - Each sum query needs the property name to sum baked into the picker — there's no implicit "+1 per matched doc." The picker resolves the property from the covering index's
summable: "<property>"flag and rejects queries whose target property isn't the same one any candidate index sums.
No-Prove (Server-Side O(1) or O(log n))
When prove=false, drive-abci calls into DriveDocumentSumQuery (the proposed analog of DriveDocumentCountQuery in packages/rs-drive/src/query/drive_document_count_query/mod.rs). The handler picks a path based on the where clauses:
Unfiltered total (no where clauses) on a documentsSummable: "amount" document type:
The doctype's primary-key tree at [contract_doc, contract_id, 1, doctype, 0] is itself a SumTree. One grovedb read gives sum_value — the total of amount across every document of this type. O(1).
Equal/In only:
- Pick a
summable: "<prop>"index whose properties exactly match the Equal/In where-clause fields.<prop>must equal the request's target sum property. The same strict-coverage contract count uses applies — partial coverage rejects withWhereClauseOnNonIndexedProperty. (See "Index design" below.) - Walk the tree from the root down to the terminal level, pushing
prop_nameandserialize_value_for_key(prop_name, value)at each step.Equalextends one path;Inclones the current path once per value in its array (a cartesian fork) and the per-branch sums are summed. - Read the
SumTreeelement at the resulting path and return itssum_value. O(1) per branch.
If the request carries an In clause, the response is the entries variant — one SumEntry per In value. Otherwise the response is the aggregate_sum variant — a single i64.
Index design contract: a summable: "amount" index sums exactly its declared properties' coverage of amount. Want sum(amount) WHERE recipient = X? Define a [recipient] index with summable: "amount". Want sum(amount) WHERE recipient = X AND sentAt > T? Define a [recipient, sentAt] index with summable: "amount" AND rangeSummable: true. Partial coverage (e.g. recipient = X against a [recipient, sentAt] index without the range clause) is rejected — define a more specific summable index, or set documentsSummable: "amount" on the document type for unfiltered total sums. The prove path enforces the same contract, so prove=true and prove=false reject in the same situations with the same error.
Range:
- Pick a
rangeSummable: trueindex where the Equal/In clauses cover the prefix and the range operator hits the index's last property. - Build the path
[contract_doc, doctype, prefix..., range_prop_name]— pointing at the property-nameProvableSumTree. - Issue a grovedb path query with the converted range
QueryItem(>,>=,<,<=,Range,RangeInclusive,RangeAfter,RangeAfterTo,RangeAfterToInclusive) and walk the children whose keys lie inside the range. - Each child's
sum_value_or_default()is theamountsum at that property value. Either combine all per-value sums and return as theaggregate_sumvariant (summed mode), or emit them as per-valueSumEntrys under theentriesvariant (distinct mode), then apply order / cursor / limit.
Prove (Client-Side Verify-Then-Aggregate or Aggregate-Sum Proof)
When prove=true, the proof shape depends on whether the query carries a range clause.
With a range clause: the handler picks one of two prove sub-paths based on return_distinct_sums_in_range:
-
Aggregate (
return_distinct_sums_in_range = false, default): drive-abci builds a grovedbAggregateSumOnRangepath query against the property-nameProvableSumTree, andget_proved_path_queryproduces an aggregate-sum proof. The client verifies viaGroveDb::verify_aggregate_sum_queryand recovers(root_hash, sum)directly — proof size is O(log n) regardless of how many keys match. No documents are ever materialized. -
Distinct (
return_distinct_sums_in_range = true): drive-abci builds a regular range path query (noAggregateSumOnRangewrapper) against the sameProvableSumTree. Because the leaf is aProvableSumTree, merk emits oneNode::KVSum(key, value, sum)op per matched in-range key, with eachsumcryptographically committed to the merk root vianode_hash_with_sum(kv_hash, l_hash, r_hash, sum)— same forge-resistance as the aggregate path'sHashWithSumcollapse. The SDK'sdrive_proof_verifier::verify_distinct_sum_proofruns the standard hash-chain check, then walks the proof's op stream to extract the sums as aBTreeMap<Vec<u8>, i64>. Trade-off vs. the aggregate path: proof size is O(distinct values matched) rather than O(log n).
Without a range clause (point-lookup with prove): two sub-paths based on the request shape.
-
Unfiltered total +
documentsSummable: "amount": drive-abci proves the doctype's primary-keySumTreeelement at[contract_doc, contract_id, 1, doctype, 0]. One merk path proof; the SDK'sdrive_proof_verifier::verify_primary_key_sum_tree_proofreadssum_valueoff the verified element. O(log n) bytes. -
Equal/In against a fully-covering
summable: "amount"index: drive-abci proves oneElement::SumTreeper covered branch. Two sub-shapes parallel to count's:- Equal-only fully-covered → one element at
[..., last_field, last_value, 0]. Inat any index position (with any number of trailing Equals) → one element per In value, fetched via outer Query + a subquery whoseset_subquery_pathcarries the post-In Equal segments.
The In position rule and the
set_subquery_pathmechanics are byte-for-byte the same as the count case — see the count chapter's Prove section for the rationale. Sum picks up the same permissive layout because both paths usepoint_lookup_sum_path_query(no document-key terminator descent, noorder_byinterpretation, nolimit/offsetsemantics — it's a pure SumTree-element lookup). - Equal-only fully-covered → one element at
Both sub-paths share the proof shape: each SumTree element's sum_value is cryptographically bound to the merk root via node_hash_with_sum(kv_hash, l_hash, r_hash, sum). Neither materializes documents or runs per-key bookkeeping client-side.
Proof size: O(k × log n) where k is the number of covered branches (1 for the documents_summable fast path and Equal-only fully-covered case; ≤ |In values| for Equal-prefix + In-on-last).
Symmetric rejection contract: prove sum requires a summable: "<prop>" index whose properties exactly match the where clauses and whose summed property matches the request's target — same requirement as the no-proof Total / PerInValue modes. Partial coverage rejects with a WhereClauseOnNonIndexedProperty-class error. The documentsSummable: "<prop>" fast path handles unfiltered total sums in O(log n) proof bytes when set on the document type. No silent fallback to materializing matching documents.
Supported Where Operators
Identical to the count endpoint:
Equal(==) — single point lookup against the sum tree at a fully-resolved index path.In(in) — cartesian fork. Each value in theInarray becomes its own index path; their sums are combined (or, for split sums, merged by split key). AnInclause withkvalues costskpoint lookups, not a tree walk. TheInclause also doubles as the per-value split signal in the unifiedGetDocumentsSumendpoint — at most oneInper request.- Range (
>,>=,<,<=,between*,startsWith) — walks the property-nameProvableSumTree's children whose keys lie inside the range, combining each childSumTree's sum value. Requires the index to haverangeSummable: trueAND the range property to be the index's last property.
Range queries take a single range terminator clause plus a prefix of Equal clauses and/or one In clause. In on a prefix property exercises grovedb's native subquery primitive — each emitted entry carries both the in_key (the In value for that fork) and the key (the terminator value within the range). Per-fork sums are NOT merged server-side — same No-Merge Compound Semantics reasoning as count.
Range Modes
A range query produces one of two response shapes, controlled by return_distinct_sums_in_range:
return_distinct_sums_in_range = false(default) —SumResults.aggregate_sumcarrying the sum of the per-valueSumTreesums within the range. Use for "how much was tipped between t1 and t2?".return_distinct_sums_in_range = true—SumResults.entrieswith oneSumEntryper distinct property value within the range. Use for "show me the histogram of tip amounts per timestamp in [t1, t2]".
No-Merge Compound Semantics
For compound queries (In on a prefix property + range on the terminator), entries are returned unmerged — one SumEntry per emitted (in_key, key) pair. The server does NOT collapse them down to a flat histogram. Same three reasons as count:
- Correctness under
limit. Pushing alimitinto grovedb's path query truncates the emitted elements before any merge could run. With cross-fork merging this can undercount the merged sums. - Proof verification stays straightforward. A malicious server omitting one
Inbranch shows up as missing entries with thatin_keyrather than as a silent undercount in a merged total. - No information loss. A caller who wanted the merged histogram can compute
result.fold(by=key, sum=sum)client-side trivially.
The rs-sdk surfaces this via DocumentSplitSums.0: Vec<VerifiedSplitSum>. Callers wanting the historical flat-map shape can call DocumentSplitSums::into_flat_map() which combines across in_key forks.
Pagination
Identical to count's pagination — order_by controls split-mode entry ordering; limit truncates after min(requested, max_query_limit) with None normalized to default_query_limit. Pagination is by range narrowing, not cursor. Ignored on aggregate mode.
Range Queries on the Prove Path
Same shape as count's Range Queries on the Prove Path:
- Aggregate sub-path (default) builds
AggregateSumOnRange— proof size O(log n). - Distinct sub-path (
return_distinct_sums_in_range = true) builds a regular range proof against the property-nameProvableSumTree. Per-(in_key, key)KVSumops, each bound to the merk root vianode_hash_with_sum. Inon a prefix property is supported on the distinct sub-path. The aggregate sub-path rejectsInon prefix (single-range merk primitive can't fork at the merk layer)."desc"direction in the firstorder_byclause flows through to grovedb'sQuery.left_to_right.
Range Queries and ProvableSumTree
Range sum queries (>, <, between*) over an index with rangeSummable: true are answered in O(log n) by walking the property-name ProvableSumTree's boundary nodes. The proof path uses grovedb's AggregateSumOnRange, which lets clients verify a range sum without ever materializing the underlying documents.
Why Internal-Node Sums Make Range Sums O(log n)
In a sorted merk tree the keys partition into a left (smaller) and right (larger) subtree at every internal node. To answer "what's the sum of amount for documents with sentAt > T?" you walk the boundary between "below T" and "above T" from the root down, and at each step you decide what to do with the other subtree based on a single read:
- If a subtree lies entirely above the cutoff, add its full sum and don't descend.
- If it lies entirely below, ignore it (contributes 0).
- If it straddles, recurse.
On a ProvableSumTree every internal node carries the sum of its left and right subtrees, so the "add the full sum" step is a single O(1) read. The walk visits one node per tree level — O(log n).
Concretely, picture a ProvableSumTree of 8 tips with sorted sentAt keys and amount leaves:
flowchart TB
R["root s=80"]:::sumroot
R --> L1["s=30"]:::sumnode
R --> R1["s=50"]:::sumnode
L1 --> LL["s=15"]:::sumnode
L1 --> LR["s=15"]:::sumnode
R1 --> RL["s=20"]:::sumnode
R1 --> RR["s=30"]:::sumnode
LL --> x1["t=1, amt=5"]:::leaf
LL --> x3["t=3, amt=10"]:::leaf
LR --> x5["t=5, amt=7"]:::leaf
LR --> x7["t=7, amt=8"]:::leaf
RL --> x9["t=9, amt=12"]:::leaf
RL --> x11["t=11, amt=8"]:::leaf
RR --> x13["t=13, amt=15"]:::leaf
RR --> x15["t=15, amt=15"]:::leaf
classDef sumroot fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sumnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;
For "give me the sum of amount for items with sentAt > 6":
- root (s=80): 6 falls inside the left subtree (which holds t=1..7). Read both children's sub-sums. Right subtree's keys are all > 6 → take its full
s=50and don't descend. Recurse into left. - left (s=30): 6 falls inside its right subtree (t=5,7). Read both children. Left-left's keys (1,3) are both ≤ 6 → contribute 0. Recurse into left-right.
- left-right (s=15): 6 splits this leaf-pair. Read both leaves. Key 5 ≤ 6 → contribute 0. Key 7 > 6 → contribute its
amt=8. - Total = 50 (right of root) + 0 (left-left) + 0 (t=5) + 8 (t=7) = 58.
We visited 4 internal nodes on the boundary path and read sub-sums off 3 siblings without descending. Six of the eight items were never enumerated: their contributions were combined straight out of the committed sub-sum fields.
Why This Is Provable
A merk proof of the same boundary walk includes:
- The boundary path from root to the leaf adjacent to the cutoff.
- The siblings of every node on the boundary path (so the verifier can recompute hashes up to the merk root).
Each sibling node, on a ProvableSumTree, ships its committed sub-sum alongside its hash. The verifier walks the same logic the server did — "this sibling lies entirely above 6, add its s=… value" — and ends up with the same total without enumerating the sibling subtrees. Verification is also O(log n).
The same primitive answers any range query of the form [A, B]: walk to the cutoff at A, then to the cutoff at B, and combine sub-sums along the way.
Authoring a Contract That Uses Sum Trees
Two opt-in surfaces, parallel to count. They're independent and can be used together:
- Top-level flags on the document type control the primary-key tree variant.
- A per-index
summable: "<property>"flag controls whether that specific index's tree carries sums.
Primary-Key Tree Flags
Set at the same level as type / properties / indices on a document type:
{
"tip": {
"type": "object",
"documentsSummable": "amount",
"properties": {
"recipient": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32, "position": 0,
"contentMediaType": "application/x.dash.dpp.identifier" },
"amount": { "type": "integer", "minimum": 1, "maximum": 4294967295, "position": 1 },
"sentAt": { "type": "integer", "minimum": 0, "position": 2 }
},
"required": ["recipient", "amount", "sentAt"],
"additionalProperties": false
}
}
That contract gets a SumTree for the tip primary-key tree, summing amount. GetDocumentsSum for tip with no where filter is now an O(1) lookup of the tree element's sum value.
To opt into a ProvableSumTree for the primary-key tree instead — useful if you want range queries on the primary key or intend to use this document type behind range proof primitives — pair with rangeSummable: true:
{
"tip": {
"type": "object",
"documentsSummable": "amount",
"rangeSummable": true,
...
}
}
Both flags are immutable across a contract update — you pick the tree variant at contract creation; you can't switch later without creating a new document type.
Per-Index Summable Flag
Set on a single entry in the document type's indices array:
{
"indices": [
{
"name": "byRecipient",
"properties": [{ "recipient": "asc" }],
"summable": "amount"
}
]
}
With byRecipient.summable: "amount" the byRecipient index's tree carries running sums of amount, so GetDocumentsSum with where: [["recipient", "==", X]] reaches the sum via that index in O(1). Without the flag the query rejects with WhereClauseOnNonIndexedProperty — there's no slow fallback, only fast sums on properly-indexed properties.
The summable field accepts a single shape: a string naming an integer property declared on the document type and listed in required. The named property must be the same one named at any other summable level (doctype documentsSummable and other indexes' summable) — multi-property summing on a single tree isn't supported and won't be; if you need to sum two properties, declare two separate aggregation surfaces.
A few notes about the index-level flag:
- Setting
summableincreases storage cost — every insert and delete updates the index tree's sum alongside the document, and the reference under the index value tree carries an extrai64sum-item contribution. - Setting
rangeSummable: trueincreases storage cost further — every internal node of the property-name tree carries running-sum metadata, not just the root. - The flag is on the whole index, not per-property. Same strict-coverage rule as count: a
["recipient", "sentAt"]summable index gives O(1) sums forWHERE recipient = X AND sentAt = Tbut NOT forWHERE recipient = Xalone. Define both indexes if you want both queries. - Index-level
summableis independent of the primary-key flags. You can havedocumentsSummable: "amount"on the document type ANDsummable: "amount"on a specific index. summableon auniqueindex is mostly a no-op, but not always, mirroring count's caveat. A unique index stores its terminal as a bare reference at key[0]rather than wrapping it in a sum tree, so for documents whose indexed fields are all non-null the flag has no storage effect. Null-bearing entries take the same sum-tree branch a non-unique index uses, and the sum tree at that path aggregates them. Sums on all-non-null exact matches still return correctly (the reference's stored sum contribution) because the on-disk reference reads as a sum item via grovedb's default-aggregate semantics.
Choosing What to Set
| You want | Set |
|---|---|
Fast sum(amount) for the whole document type | documentsSummable: "amount" on the document type |
O(1) filtered sum: sum(amount) WHERE col = X | summable: "amount" on an index whose properties are exactly ["col"]. Partial coverage of a wider index rejects with WhereClauseOnNonIndexedProperty — define a dedicated index. |
Per-In-value sub-sums | summable: "amount" on an index whose properties exactly match the query's == clauses plus the In field. The In field may sit at any position. |
O(log n) range sum: sum(amount) WHERE col BETWEEN A AND B | rangeSummable: true on an index whose last property is col and whose other properties cover any equality predicates as a prefix. Requires summable: "amount". |
| Per-distinct-value range histogram | Same rangeSummable: true index as above, plus return_distinct_sums_in_range = true on the request. |
| Range sum proof | Same rangeSummable: true index. Handler uses grovedb's AggregateSumOnRange — proof is O(log n), no cap on matched docs. |
Aggregations beyond i64::MAX | Out of scope for this chapter — see the BigSumTree variant. |
| Both a sum AND a count on the same tree | Combine the count flags (documentsCountable / rangeCountable / per-index countable) with the sum flags. The dispatcher picks one of three combined variants depending on which axes opt into per-node aggregation: CountSumTree (both at root only), ProvableCountSumTree (per-node count, root-only sum — useful when range count is wanted but range sum isn't), or ProvableCountProvableSumTree / PCPS (both per-node — the grovedb PR 670 newcomer, enables AggregateCountAndSumOnRange recovering both metrics in a single range proof). One tree, two simultaneous queries, no double storage. The tip-jar contract above doesn't use these combinations (it's pure-sum to keep the introduction focused); a worked example using (count, sum) together is covered in a separate chapter alongside its own example contract. |
| Nothing sum-aware (default) | Don't set any of these flags. Primary-key tree stays a NormalTree. |
Every sum query requires either documentsSummable: "<prop>" (for unfiltered totals) or a summable: "<prop>" / rangeSummable: true index whose properties exactly match the query's where-clause fields. No covering index → the call returns a clear InvalidArgument describing what the picker was looking for. Pick your indexes deliberately at contract creation time — per-index summable / rangeSummable flags can't be added later (contract indexes are immutable post-creation).
SDK Access at Three Layers
rs-sdk (native Rust)
Both shapes will land on the standard Fetch trait against a single DocumentSumQuery:
#![allow(unused)] fn main() { use dash_sdk::platform::documents::document_sum_query::DocumentSumQuery; use dash_sdk::platform::Fetch; use drive::query::{WhereClause, WhereOperator}; use drive_proof_verifier::{DocumentSum, DocumentSplitSums}; // Total sum: no In clause. let DocumentSum(sum) = DocumentSum::fetch( &sdk, DocumentSumQuery::new(contract.clone(), "tip", "amount")?, ) .await? .expect("DocumentSum::fetch always returns a value on success"); // Split sum: signal split by including an `In` clause whose field // is the split property. let split_query = DocumentSumQuery::new(contract, "tip", "amount")? .with_where(WhereClause { field: "recipient".to_string(), operator: WhereOperator::In, value: platform_value::Value::Array(vec![alice.into(), bob.into()]), }); let splits = DocumentSplitSums::fetch(&sdk, split_query) .await? .expect("DocumentSplitSums::fetch always returns a value on success"); // `splits` is `DocumentSplitSums(Vec<SplitSumEntry>)` — collapse via `splits.into_flat_map()`. }
DocumentSumQuery wraps an internal DocumentQuery (reusing where-clause / order-by / contract-id machinery) and exposes with_where(WhereClause) + with_order_by(OrderClause) builders. The SDK picks the request mode from query shape plus explicit request flags. The target sum property is part of the query construction — the SDK validates against the contract that some covering index sums it.
wasm-sdk (browser)
Two methods on the WasmSdk JS class — one entry per [plain | withProofInfo] variant covers every sum mode:
sdk.getDocumentsSum(
query: DocumentsQuery,
sumProperty: string,
): Promise<Map<string, bigint>>;
sdk.getDocumentsSumWithProofInfo(
query: DocumentsQuery,
sumProperty: string,
): Promise<ProofMetadataResponseTyped<Map<string, bigint>>>;
Result shapes mirror count's wasm SDK, with bigint carrying a signed sum:
- No
where, or Equal-onlywhere— single map entry with the empty-string key carrying the total sum. whereincludes anInclause — one entry per (deduped) In value.whereincludes a range clause +returnDistinctSumsInRange: true— one entry per distinct property value in the range.
Map keys are hex-encoded bytes matching the canonical serialize_value_for_key encoding of each property value, same convention as count.
rs-sdk-ffi (iOS / native bindings)
#![allow(unused)] fn main() { dash_sdk_document_sum( sdk, data_contract, document_type, sum_property, // name of the integer property to sum where_json, // null or JSON [{field, operator, value}] order_by_json, // null or JSON [{field, direction}] return_distinct_sums_in_range, // bool limit, // i64; -1 = server default, >= 0 = explicit cap ) -> JSON {"sums": {"<hex-key>": <i64>, ...}} }
Single FFI entry covers every sum mode — the result is always {"sums": {...}} with hex-encoded keys. For total sums (no where/In, distinct flag off), the map carries a single entry with the empty-string key. where_json is the same JSON shape dash_sdk_document_search already accepts. The endpoint returns its result as a JSON-encoded C string allocated on the heap — caller frees it via the standard SDK string-free routine.
Sum Index Examples
This chapter walks through a representative contract and shows what a sum-query proof actually proves — both the path query the prover signs and the verified element the verifier extracts. Every example uses the same tip document type on the tip-jar contract at packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json, so the proof bytes, verified elements, and diagrams can all be cross-referenced against the same data once the bench fixture lands.
The chapter assumes you've read Document Sum Trees — that chapter explains the three tree variants (NormalTree / SumTree / ProvableSumTree), how Element::NonCounted-style "doesn't contribute to my parent's aggregation" wrappers work (now for sums as well), and how the schema's documentsSummable / rangeSummable flags select between them. Here we take that machinery as given and trace what each query sees.
Status: the bench at
packages/rs-drive/benches/document_sum_worst_case.rslands the reproducible numbers below — same convention as the Count Index Examples chapter. All proof sizes are measured against a 100 000-row fixture; verifiedsumvalues are the actual sums the bench's matrix reports. The full surface — primary-key total, point lookups, In-fan-out,AggregateSumOnRangeon both top-level and compound indexes, and the carrier-aggregate primitive from grovedb PR #670 — is fully wired and producing the byte counts in the table below.
The Tip Jar Contract
The tip document type carries four properties (recipient, amount, sentAt, note), opts into total sums at the doctype level via documentsSummable: "amount", and declares three indexes covering the sum-query surface:
{
"type": "object",
"documentsMutable": false,
"documentsSummable": "amount",
"properties": {
"recipient": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"position": 0, "contentMediaType": "application/x.dash.dpp.identifier" },
"amount": { "type": "integer", "minimum": 1, "maximum": 4294967295, "position": 1 },
"sentAt": { "type": "integer", "minimum": 0, "position": 2 },
"note": { "type": "string", "maxLength": 280, "position": 3 }
},
"required": ["recipient", "amount", "sentAt"],
"indices": [
{
"name": "byRecipient",
"properties": [{ "recipient": "asc" }],
"summable": "amount"
},
{
"name": "bySentAt",
"properties": [{ "sentAt": "asc" }],
"summable": "amount",
"rangeSummable": true
},
{
"name": "byRecipientTime",
"properties": [{ "recipient": "asc" }, { "sentAt": "asc" }],
"summable": "amount",
"rangeSummable": true
}
],
"additionalProperties": false
}
Three things to notice:
documentsSummable: "amount"at the document-type level upgrades the doctype's primary-key subtree (attip/[0]) fromNormalTreetoSumTree. The unfiltered total sum is one read against this element'ssum_value. The string-form value names the property each insert contributes to the tree — the picker uses it to validate that any futureGetDocumentsSumrequest whosesum_propertydoesn't match"amount"is rejected at parse time.byRecipientissummable: "amount"only. It doesn't opt intorangeSummable, sorecipient > Xrange sums aren't supported. Every summable terminator's value tree is stored as aSumTreeregardless ofrangeSummable, so point-lookup sum proofs (e.g.recipient == Xorrecipient IN [...]) get a compact value-tree-direct shape.rangeSummableis strictly an opt-in forAggregateSumOnRangesupport — orthogonal to proof-size shape on point queries.bySentAtandbyRecipientTimearerangeSummable: true. Their property-name subtrees (e.g.tip/sentAt) are stored asProvableSumTreerather thanNormalTree, which is whatAggregateSumOnRangewalks forsentAt > floorstyle queries.
The bench populates 100 000 tips under a deterministic schedule: row → (recipient_(row % 100), sentAt = row, amount = (row % 10) + 1). That gives exactly 1 000 tips per recipient, but with an asymmetry worth flagging: since both recipient and amount are derived from row modulo (100 and 10 respectively), each recipient sees only one amount value across all their 1 000 tips. Recipient n always has amount = (n % 10) + 1:
recipient_000,recipient_010, …,recipient_090→amount = 1, per-recipient sum = 1 000recipient_001,recipient_011, …,recipient_091→amount = 2, per-recipient sum = 2 000- …
recipient_009,recipient_019, …,recipient_099→amount = 10, per-recipient sum = 10 000
The headline numbers:
- Total
sum(amount)across all tips: 550 000 (each amount value 1..10 appears 10 000 times →10 000 × 55). The primary-keydocumentsSummableSumTree reports this directly via Query 1's O(1) read. sum(amount)per recipient: varies 1 000–10 000 by recipient (see the cycle above).sum(amount)for any contiguoussentAtrange of length 10: exactly 55 (every 10-row window covers one full cycle of1..10).sum(amount)for the first half of the timeline (sentAt < 50 000): 275 000.
Those numbers appear in every verified-element block below.
GroveDB Layout
The contract above produces this storage shape. Tree elements (the wrapping Element GroveDB stores under each key) are drawn as subgraphs; children inside each tree are merk-tree nodes. The doctype root and the per-property name subtrees are separate Element trees nested under the contract-documents prefix, just like every other index in Drive.
Diagram conventions: green nodes carry a sum_value committed to the merk root; yellow nodes are ProvableSumTree (per-node sums); gray are regular subtrees; dashed boxes highlight Element::NonCounted-style wrappers (children that store data but contribute 0 to their parent's aggregation).
flowchart TB
TD["@/contract_id/0x01/tip"]:::tree
TD --> PK["[0]: SumTree sum=550000<br/>(documentsSummable primary key)"]:::sumnode
TD --> RC["recipient: NormalTree<br/>(byRecipient property-name)"]:::node
TD --> SA["sentAt: ProvableSumTree<br/>(bySentAt property-name)"]:::pstnode
RC --> R000["recipient_000: SumTree sum=1000<br/>(amount cycle: 1)"]:::sumnode
RC --> R050["recipient_050: SumTree sum=1000<br/>(amount cycle: 1)"]:::sumnode
RC --> RMore["... recipient_001 sum=2000 ... recipient_009 sum=10000<br/>(per row%10 amount cycle)"]:::sumnode
R050 --> R050_0["[0]: SumTree sum=1000<br/>(byRecipient refs)"]:::sumnode
R050 --> R050_S["sentAt: NonCounted(ProvableSumTree)<br/>(byRecipientTime continuation, contributes 0)"]:::noncounted
R050_S --> R050_S_500["sentAt_00050000: SumTree sum=1"]:::sumnode
R050_S_500 --> R050_S_500_0["[0]: SumTree sum=1<br/>(byRecipientTime ref, amount=1)"]:::sumnode
SA --> S500["sentAt_00050000: SumTree sum=1<br/>(bySentAt terminator, amount=1)"]:::sumnode
SA --> SMore["... sentAt_00000000 ... sentAt_00099999"]:::sumnode
S500 --> S500_0["[0]: SumTree sum=1<br/>(bySentAt ref)"]:::sumnode
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
classDef sumnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
classDef pstnode fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef noncounted fill:#21262d,color:#c9d1d9,stroke:#fb8500,stroke-width:2px,stroke-dasharray: 6 4;
Three layout facts to internalize before reading the queries:
recipient_050is aSumTreewithsum_value = 1000. That's true becausebyRecipientis summable; the rule applies uniformly to every summable tier. ThesentAtcontinuation that branches off this value tree isNonCounted-wrapped so the parent's sum equals exactly the contribution in[0](which for recipient_050 is1 000 × 1 = 1 000per the amount cycle). Other recipients get different value-tree sums per the per-recipient amount described above.tip/sentAtis aProvableSumTree, not a regularNormalTree. The yellow class above marks that — each internal merk node carries its subtree's sum, which is what makesAggregateSumOnRangea single-pass primitive.sentAt_00050000is aSumTreewithsum_value = 1under either parent (the globalbySentAtpath or the per-recipientbyRecipientTimecontinuation). Same element layout under both; the path that gets there differs but the destination is structurally the same.
How To Read The Proofs
Every example below has four sections:
- Path query — the spec the prover hands GroveDB.
pathis the list of subtree segments to descend through;query itemsis what to select once at the bottom;subquery items(when present) descends one more layer. - Verified element — what
GroveDB::verify_query(orverify_aggregate_sum_queryfor the range primitive) returns after walking the proof bytes. Thesum_value_or_defaultfield on aSumTreeelement is what the sum surface ultimately surfaces to the caller. - Proof display — the proof bytes decoded via
bincodeinto the structuredGroveDBProofAST and rendered through itsDisplayimpl, same convention as the count chapter. The bench'sdisplay_proofsblock emits these inline; reproduce locally withDASH_PLATFORM_SUM_BENCH_REBUILD=1 cargo bench -p drive --bench document_sum_worst_case -- --test 2>&1 | grep -A 200 "^\[display\]". - Diagram — the path the proof walks through the layout. Blue arrows trace the descent; the cyan node is the verified element; faded gray nodes show context.
Proof-size numbers below come from the 100 000-row bench run on the byRecipient / bySentAt / byRecipientTime indexes. Avg-time numbers are median-of-5 wall-clock measurements with one warmup discarded (see the methodology note under the queries table).
Queries in this Chapter
| # | Query | Filter | Complexity | Avg time | Proof size |
|---|---|---|---|---|---|
| 1 | Unfiltered Total Sum | (none — total at doctype level) | O(1) | 23.6 µs | 580 B |
| 2 | Equal on a Single Property (byRecipient) | recipient == "recipient_050" | O(log R) | 37.2 µs | 1 087 B |
| 3 | Equal on a RangeSummable Property (bySentAt) | sentAt == 50000 | O(log T) | 72.1 µs | 1 706 B |
| 4 | Compound Equal-only (byRecipientTime) | recipient == "recipient_050" AND sentAt == 50000 | O(log R + log T') | 71.8 µs | 1 937 B |
| 5 | In on byRecipient | recipient IN ["recipient_000", "recipient_001"] | O(k · log R) | 42.8 µs (k=2) / 1 720 µs (k=100) | 1 168 B (k=2) / 12 064 B (k=100) |
| 6 | In on bySentAt (RangeSummable) | sentAt IN [0, 1] | O(k · log T) | 81.2 µs (k=2) / 2 008 µs (k=100) | 1 756 B (k=2) / 9 784 B (k=100) |
| 7 | Range Query (AggregateSumOnRange) | sentAt > 50000 | O(log T) | 102.0 µs | 3 102 B |
| 8 | Compound == + Range (byRecipientTime) | recipient == "recipient_050" AND sentAt > 50000 | O(log R + log T') | 91.3 µs | 2 657 B |
| 9 | Carrier-Aggregate (In + range) | recipient IN [r000..r099] AND sentAt > 50000 (group_by = [recipient]) | O(k · log T') | 11 507.9 µs (k=100) | 169 064 B (k=100) |
Timing methodology: median of 5 iterations after one warmup, measured against the bench's 100 000-row fixture on a warmed rocksdb cache. The figures reflect the drive-layer execute_document_sum_request call (executor + grovedb proof generation, no network or tenderdash signature compose). Reproduce with cargo bench -p drive --bench document_sum_worst_case -- --test; grep µs from stderr.
Complexity variables. R = distinct recipients in the byRecipient merk-tree (= 100 in the fixture); T = distinct timestamps in the bySentAt merk-tree (= 100 000); T' = distinct timestamps per recipient in byRecipientTime's continuation (= 1 000 per recipient); k = number of values in the IN clause (2 here). Notably absent: the total document count N (100 000 here). Sum proofs read pre-committed sum_values from SumTree merk roots — they never enumerate the underlying documents, so proof generation cost is polylog(distinct index values), independent of N. Same big-O story as count.
The bySentAt index's T = 100 000 is unusually large (one distinct timestamp per row in this fixture); real tip jars would bucket timestamps coarsely. A reproducible-numbers fixture deliberately maximizes distinct values to stress the prove paths' merk-traversal cost — the per-merk-op count and proof-size numbers will skew accordingly when the bench publishes them.
Query 1 — Unfiltered Total Sum
select = SUM(amount)
where = (empty)
sum_property = "amount"
prove = true
Path query (primary-key SumTree fast path; no index walk needed):
path: ["@", contract_id, 0x01, "tip"]
query items: [Key(0x00)]
Verified element:
path: ["@", contract_id, 0x01, "tip"]
key: 0x00
element: SumTree { sum_value_or_default: 550000 }
Proof size: 580 bytes. Avg time: 23.6 µs. Verifier root hash: 95aa74708738c5254c706bbca3245b520022aad949674822218094bca15671b6 (same across every query in this chapter — same fixture state).
Proof display (GroveDBProof::Display):
Expand to see the structured proof (5 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
lower_layers: {
tip => {
LayerProof {
proof: Merk(
0: Push(KVValueHashFeatureTypeWithChildHash(0x00, Tree(0000000000010000fffffffffffeffff00000000000000000000000000000000), HASH[f25f49fc619abc59d984dcb322947509eb2a34f02217fc31dfc11423bee001e8], BasicMerkNode, HASH[d8c56d5b5d11c2e30a70694fac85259bb3169854e474ae8056ef91d5cb877000]))
1: Push(KVHash(HASH[a17138f666ae4ab19c1c2930ad94c7e29a6a82789398fc4d3a0b053d3499ac68]))
2: Parent
3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
4: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. The descent goes: top-level GroveDB root → @ (the DataContractDocuments root tree) → contract id → 0x01 (documents storage prefix) → tip doctype → finally the Key(0x00) payload at the bottom. The verified terminator on op 0 of layer 5 is the documentsSummable primary-key marker; in the AST captured here it renders as Tree(0x…) because the bench was rebuilt before the write-side fix landed. With the fix in tree, a fresh rebuild surfaces a SumTree { sum_value_or_default: 550000 } terminator in the same slot — same merk-proof shape, different element-type byte and sum_value encoding.
The descent stops at the doctype's primary-key tree — the green node at the top of the layout. Because documentsSummable: "amount" upgraded that tree to a SumTree, the total is one O(1) read with an O(log n) proof.
Diagram: per-layer merk-tree structure
Each LayerProof above is its own GroveDB sub-tree whose contents form a merk binary tree. The merk-proof operations (Push / Parent / Child over KVValueHash / KVHash / Hash nodes) describe exactly which nodes of each layer's binary tree the proof reveals — the queried key gets its full kv-hash exposed; opaque siblings only commit their subtree-hash so the verifier can re-hash up to the merk root.
Cyan = the verified target. Blue = a kv-hash that's also a queried-key on the descent path (its value = Tree(...) is the merk-root pointer for the next layer). Gray = opaque sibling subtrees committed by hash only.
flowchart TB
subgraph L1["Layer 1 — root GroveDB merk-tree"]
direction TB
L1_root["<b>@</b><br/>kv_hash=HASH[2f2b...]<br/>value: Tree(0x4ed2…)"]:::queried
L1_left["HASH[bd29...]<br/>(left subtree, opaque)"]:::sibling
L1_right["HASH[19c9...]<br/>(right subtree, opaque)"]:::sibling
L1_root --> L1_left
L1_root --> L1_right
end
subgraph L2["Layer 2 — @ subtree merk-tree (single key)"]
direction TB
L2_q["<b>contract_id 0x4ed2…</b><br/>kv_hash=HASH[5b1e...]<br/>value: Tree(0x01)"]:::queried
end
subgraph L3["Layer 3 — contract_id subtree merk-tree"]
direction TB
L3_q["<b>0x01</b><br/>kv_hash=HASH[fba9...]<br/>value: Tree(tip)"]:::queried
L3_left["HASH[c53e...]<br/>(left subtree, opaque)"]:::sibling
L3_q --> L3_left
end
subgraph L4["Layer 4 — 0x01 documents-prefix subtree (single key)"]
direction TB
L4_q["<b>tip</b><br/>kv_hash=HASH[0fe5...]<br/>value: Tree(recipient/sentAt/0x00)"]:::queried
end
subgraph L5["Layer 5 — tip doctype merk-tree (TARGET layer)"]
direction TB
L5_target["<b>0x00</b><br/>kv_hash=HASH[f25f...]<br/>value: <b>SumTree sum=550000</b><br/>child_hash=HASH[d8c5...]<br/>(captured AST shows Tree+sum=0; fresh rebuild refreshes)"]:::target
L5_kv["KVHash[a171...]<br/>(opaque internal kv: sentAt or recipient)"]:::sibling
L5_right["HASH[143e...]<br/>(right subtree, opaque)"]:::sibling
L5_target --> L5_kv
L5_target --> L5_right
end
L1_root -. "value=Tree(merk_root[5b1e…])" .-> L2_q
L2_q -. "value=Tree(merk_root[fba9…])" .-> L3_q
L3_q -. "value=Tree(merk_root[0fe5…])" .-> L4_q
L4_q -. "value=Tree(merk_root[f25f…])" .-> L5_target
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
Layers 1–4 are the standard contract-documents descent every query in this chapter shares; the divergence starts at Layer 5. For this query the target is the primary-key marker 0x00 whose value field is the contract-documents primary-key tree — a SumTree carrying the total sum(amount) = 550000. Each of the next four queries diverges at this layer to a different doctype child (recipient, sentAt, or recipient → recipient_050 → sentAt).
Query 2 — Equal on a Single Property (byRecipient)
select = SUM(amount)
where = recipient == "recipient_050"
sum_property = "amount"
prove = true
Path query:
path: ["@", contract_id, 0x01, "tip", "recipient"]
query items: [Key("recipient_050")]
Verified element:
path: ["@", contract_id, 0x01, "tip", "recipient"]
key: "recipient_050"
element: SumTree { sum_value_or_default: 1000 }
Proof size: 1 087 bytes. Avg time: 37.2 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (6 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
lower_layers: {
tip => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
2: Parent
3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
4: Child)
lower_layers: {
recipient => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[b8804ea3f7998def339db7cadd6a6b29ba5bfadbdfa15b67802ef52e42adb68e]))
1: Push(KVHash(HASH[3056a7c2daf102d31d0f461d45e661303b1562311971debbf15f40938727a074]))
2: Parent
3: Push(Hash(HASH[c406363887b632d071f2ee6ed83df682aace9a5456997aa4f4b944576476fbb6]))
4: Push(KVHash(HASH[6b8ef1df5ba1299cd5b605dc7642be2ffb8356fd7f51c3a0498b6b657cddeebc]))
5: Parent
6: Push(Hash(HASH[347abc0b69e504bc619e9549e509c21be3c0ad1c9e03d8fb34bb5ed07e5cd26f]))
7: Push(KVHash(HASH[ff9b5006130777d589b77e6f5bdda0f86879daace181cdc8e3d97bc3718e0a48]))
8: Parent
9: Push(KVValueHashFeatureTypeWithChildHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[264c6aac1a1acd864832a6f25ac42c626afb95dc79ac8c233d2b05c6df048f1c], BasicMerkNode, HASH[58680498d71fdda362f4a1815a0e0989b70a8cb287d8e40eaf84f8fe2cb48b6f]))
10: Child
11: Push(KVHash(HASH[5159e1ccad8c4ed1bbf7f52ec34e9ab4ee108559194e39b1af78cf42866b32cc]))
12: Parent
13: Push(Hash(HASH[4317777613ab1a0d6966670195ccce83dee1031fd37013c9ce625c016d1d6d17]))
14: Child
15: Push(KVHash(HASH[24f6646d8b75a6a428d05589c1cc1e80f1e52900924710ff6eafe9da0edc73d0]))
16: Parent
17: Push(Hash(HASH[14f8282bd0b5d9a8e7e471d3cfd215f02f5be2cfbf837c5e938c2871a2eddcb9]))
18: Child
19: Child
20: Child
21: Push(KVHash(HASH[cea6360efbf77b38d2ea206d508ea03c0d92fe7c02c5d9176aa322e8263a3acc]))
22: Parent
23: Push(Hash(HASH[209f3325816d5f02a1647311a4f1d9c68fcbacf1540be9b5ae65413f58ca3b26]))
24: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. The descent diverges from Q1 at layer 5: the tip doctype layer commits the recipient property-name subtree on op 1 (cyan-blue blue queried, valued Tree(0x6263...) — the merk root of byRecipient), then descends into layer 6, byRecipient itself. There the queried recipient_050 key (0x0000000000000032ffffffffffffffcd...) is reached as op 9 of a 25-op merk path, and its value is SumTree(73656e744174, 1000) — the 73656e744174 bytes are ASCII for "sentAt" (the byRecipientTime continuation pointer, NonCounted-wrapped at storage so it contributes 0 to the parent SumTree), and the 1000 is the per-recipient sum_value for recipient_050 (1 000 tips × amount = 1 = 1 000). Verified root hash 95aa7470…71b6.
The descent walks one extra layer into the recipient property-name subtree and stops at recipient_050. Because byRecipient is summable: "amount", that node is a SumTree carrying the per-recipient sum directly — no need to step into [0] to look at individual references, exactly the same shortcut count proofs take. (recipient_050's amount = (50 % 10) + 1 = 1, so the per-recipient sum is 1 000 × 1 = 1 000. A recipient with a different n % 10 lands a proportionally larger sum — see the fixture narrative above.)
Diagram: per-layer merk-tree structure
Layers 1–4 are byte-for-byte identical to Q1's diagram (root → @ → contract_id → 0x01). The descent diverges at Layer 5, where this query takes the recipient branch (rather than 0x00) and descends one extra grove layer to land on the verified target inside byRecipient.
flowchart TB
subgraph L5["Layer 5 — tip doctype merk-tree (proof view for `recipient`)"]
direction TB
L5_q["<b>recipient</b><br/>kv_hash=HASH[f580...]<br/>value: Tree (descent into byRecipient)"]:::queried
L5_left["HASH[15fd...]<br/>(left subtree, opaque)"]:::sibling
L5_right["HASH[143e...]<br/>(right subtree, opaque)"]:::sibling
L5_q --> L5_left
L5_q --> L5_right
end
subgraph L6["Layer 6 — byRecipient merk-tree (TARGET layer)"]
direction TB
L6_target["<b>recipient_050</b><br/>kv_hash=HASH[264c...]<br/>value: <b>SumTree sum=1000</b><br/>(value bytes 73656e744174 = `sentAt` continuation, NonCounted)<br/>child_hash=HASH[5868...]"]:::target
L6_boundary["Boundary commitments (24 merk ops):<br/>7 KVHash opaque sibling recipient kvs<br/>+ 6 Hash subtree commitments<br/>(prove recipient_050's position in byRecipient's<br/>binary merk tree of ~100 recipient entries)"]:::sibling
L6_target --> L6_boundary
end
L5_q -. "value=Tree(merk_root[byRecipient])" .-> L6_target
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The boundary commitments at L6 scale linearly with byRecipient's merk depth — they bind recipient_050 to its claimed position so the verifier can recompute byRecipient's merk root. The verified target itself is one KVValueHashFeatureTypeWithChildHash op whose SumTree(…, 1000) value carries the per-recipient sum directly; the continuation-pointer bytes 73656e744174 simply name the next layer's tree (which we never descend into for a Q2 point lookup).
Query 3 — Equal on a RangeSummable Property (bySentAt)
select = SUM(amount)
where = sentAt == 50000
sum_property = "amount"
prove = true
Path query:
path: ["@", contract_id, 0x01, "tip", "sentAt"]
query items: [Key(serialize_value_for_key("sentAt", 50000))]
Verified element:
path: ["@", contract_id, 0x01, "tip", "sentAt"]
key: serialize_value_for_key("sentAt", 50000)
element: SumTree { sum_value_or_default: 1 }
Proof size: 1 706 bytes — moderately larger than Query 2's 1 087 (a 619-byte delta) because the sentAt property-name subtree is a ProvableSumTree, so each merk node on the descent carries an extra i64 sum field versus byRecipient's plain NormalTree. (Same direction-and-magnitude delta the count chapter measures between byBrand and byColor.) Avg time: 72.1 µs (≈ 2× Query 2's 37.2 µs — the extra sum-field hash work on each layer dominates).
Proof display (GroveDBProof::Display):
Expand to see the structured proof (6 layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
lower_layers: {
tip => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
1: Push(KVHash(HASH[a17138f666ae4ab19c1c2930ad94c7e29a6a82789398fc4d3a0b053d3499ac68]))
2: Parent
3: Push(KVValueHash(sentAt, ProvableSumTree(800000000000ffff, 550000), HASH[3b3f5bf4e079c639895d84f8c5003fe135ccb46bf81b29fd3bdf415cddffe45c]))
4: Child)
lower_layers: {
sentAt => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[0a9e4f85b317569ed34bb8821fb5a7e4357e3a070413235d92ccfc09c4aae58f]))
1: Push(KVHashSum(HASH[be99a0622f1400aaa04dc460566babac9c1a4955b3eeb1c5780dfd46ec716222], 360430))
2: Parent
3: Push(Hash(HASH[e8c7c1c4fbe14e2d5cc9e460788baad347ea50eed38e3bf0316288d3aa38c2d4]))
4: Push(KVHashSum(HASH[053e0adbc34321342c734571ab822b3c9e2fd5aa2efc267f4e09abd69670dfad], 180214))
5: Parent
6: Push(Hash(HASH[b49dc66868027c23f0e5bfde974330d67f077636fa2f1a6c69d96a7db380e4f2]))
7: Push(KVHashSum(HASH[86cb3966ee86191ce18f754b3c8492f9ad1929c9e540da193cf63899fd311d3b], 5622))
8: Parent
9: Push(Hash(HASH[41bf237b6f834b4271e191423907567fd3144d69bf6c75fd8d475807928063ee]))
10: Push(KVHashSum(HASH[f1600f73cf84dc6ae4de0057887f1fe60a373073863ee352ebede717795b1c91], 2810))
11: Parent
12: Push(Hash(HASH[0765b948c17ac92fc10c83b4480283371bcd9c3a4745e62242d842f2f9125fe4]))
13: Push(KVHashSum(HASH[a1d9da3b90a69411dcbba3934f3dd86daeeec542373c261ad36ddb072f0c61b8], 688))
14: Parent
15: Push(Hash(HASH[60ae1d862ec2ba4f63e041741bb93a539882e9620302cfc343ac5d0f339d8e21]))
16: Push(KVHashSum(HASH[133fc6dec68084db2d8c5273011a000568f49447bbeeb1c4500776a42c6de434], 170))
17: Parent
18: Push(KVValueHashFeatureTypeWithChildHash(0x800000000000c350, SumTree(00, 1), HASH[b22d3790d948924dc6311518f2872b53c0590d3cc497d7a653f1ce7b9923e499], ProvableSummedMerkNode(1), HASH[f3c390a0a62080d8a85c55e5c08c01546f0086ae2456fb7f23cf88d7d3d0c44b]))
19: Push(KVHashSum(HASH[4fa59ee6c08136a261a9b4dca592f5df25063b8f190543a9556946da3bf1d4d7], 6))
20: Parent
21: Push(Hash(HASH[6191f6b8bcfac3d6772193061c3b17dbc218a80657b018d8e9455ec571922d71]))
22: Child
23: Push(KVHashSum(HASH[5d412517c8ad8e784679852d2da663cf231d59f6e9f24bee36f26e9e6a1be900], 28))
24: Parent
25: Push(Hash(HASH[7af88dd3f234aa1e3bbf69e4543b63cd2920c16c4a04c7e2bbfef23405b0181c]))
26: Child
27: Push(KVHashSum(HASH[f412f555a36be8dc7b71d933ef60f51a34dcb3995aacf9ca82e6e5e31f2c0601], 70))
28: Parent
29: Push(Hash(HASH[7ece6d26bbaa79713cc72c05be520d699f98e6ffb66187d3d1a98f0de85db1dc]))
30: Child
31: Child
32: Push(KVHashSum(HASH[758bd7f1ade8550bb59935448441bd81421f5c8b1d77da46581f576a326c791f], 348))
33: Parent
34: Push(Hash(HASH[b2b507154c77192abad2aebd28953ab177560b4c1d515c0630821dd0e6d22b67]))
35: Child
36: Child
37: Push(KVHashSum(HASH[f2e039a1921a0d514ca714e7501d00268649420c96036fa4d6f803cb2838cdb7], 1390))
38: Parent
39: Push(Hash(HASH[145cbb78cc936b222cc9e6b06ae684aa47a1bf5479acdb35c92cbc57dc985a37]))
40: Child
41: Child
42: Child
43: Push(KVHashSum(HASH[36cf0712fbcca1239ce0cffb8b262b0a8b06a072015826c7f2a56d2bc7611169], 11262))
44: Parent
45: Push(Hash(HASH[5120fee7ab2fa02978f3dd43b6101dbc9b22de0d45c9e65976f5562a98ba4607]))
46: Child
47: Push(KVHashSum(HASH[ae76541450c5efae715f617db2d63a401da7bc2decaac3720127af252e99fa0d], 22520))
48: Parent
49: Push(Hash(HASH[5758f27a188b7d9a0b8c488374f80031671c662eb12828565146a6117344d5ea]))
50: Child
51: Push(KVHashSum(HASH[c60627bbfc24d9c01e6e7e6d8e79743884916f04904d08ca1d56e17ca8c52989], 45048))
52: Parent
53: Push(Hash(HASH[c7be4432e5634e8c8f2d0a2074b2f72a8f0fea8859a8b5f1bd6dd6556d16bf7e]))
54: Child
55: Push(KVHashSum(HASH[9f79592b39f89b6f6fed13d0fcd2e548d57735169a9e583960678dde3a5f0a05], 90102))
56: Parent
57: Push(Hash(HASH[cc7d10c23ef37f717891e6390c1fa52bf8133b230df6871c8cf85e1fcec2e9b2]))
58: Child
59: Child
60: Child
61: Push(KVHashSum(HASH[ca275ed3d5729250a4a67e2cc90fac909086fac99e386fc90688df2bb01b3eaa], 550000))
62: Parent
63: Push(Hash(HASH[8593bd2bc903da34da11e15f9a649cec37b39487ad59375020adef300a8b7482]))
64: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. Same first four layers as Q1/Q2; at layer 5 (the tip doctype) the descent takes the sentAt branch on op 3 (its value is ProvableSumTree(800000000000ffff, 550000) — a ProvableSumTree whose root-committed sum is 550 000, the full timeline sum) and descends into layer 6, bySentAt. Layer 6 is structurally different from Q2's byRecipient layer: every internal kv op is a KVHashSum (a hash with its subtree's i64 sum), and every opaque subtree commit on the merk-boundary walk carries its own running sum (e.g. op 61's KVHashSum(…, 550000) is the full-tree root-commitment). The terminator on op 18 is KVValueHashFeatureTypeWithChildHash(0x800000000000c350, SumTree(00, 1), …, ProvableSummedMerkNode(1), …) — the ProvableSummedMerkNode(1) feature-type marks it as living inside a ProvableSumTree with its own contributing sum of 1, and SumTree(00, 1) is the per-timestamp value tree (sum_value_or_default = 1, since sentAt = 50 000 was assigned amount = (50000 % 10) + 1 = 1). Verified root hash 95aa7470…71b6.
Diagram: per-layer merk-tree structure
Layers 1–4 are byte-for-byte identical to Q1's. The descent diverges at layer 5 onto sentAt, then layer 6 walks the bySentAt ProvableSumTree — yellow rather than gray for sibling commits because every internal kv carries a sum field.
flowchart TB
subgraph L5["Layer 5 — tip doctype merk-tree (proof view for `sentAt`)"]
direction TB
L5_q["<b>sentAt</b><br/>kv_hash=HASH[3b3f...]<br/>value: <b>ProvableSumTree</b> (root sum=550000)<br/>(descent into bySentAt)"]:::queried
L5_left["HASH[15fd...]<br/>(left subtree, opaque)"]:::sibling
L5_kv["KVHash[a171...]<br/>(opaque internal kv: 0x00 or recipient)"]:::sibling
L5_q --> L5_left
L5_q --> L5_kv
end
subgraph L6["Layer 6 — bySentAt ProvableSumTree merk-tree (TARGET layer)"]
direction TB
L6_target["<b>sentAt=50000 (0x800000000000c350)</b><br/>kv_hash=HASH[b22d...]<br/>value: <b>SumTree(00, sum=1)</b><br/>feature: ProvableSummedMerkNode(1)<br/>child_hash=HASH[f3c3...]"]:::target
L6_pst["Boundary commitments (64 merk ops):<br/>every internal kv is a KVHashSum (hash + subtree sum)<br/>every opaque sibling is a Hash subtree commitment<br/>(both contribute to the per-node sum aggregation<br/>that makes AggregateSumOnRange a single-pass primitive)"]:::pst
L6_target --> L6_pst
end
L5_q -. "value=ProvableSumTree(merk_root[bySentAt], sum=550000)" .-> L6_target
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The yellow pst node summarizes the 64-op ProvableSumTree merk descent: every kv-and-hash sibling carries an i64 sum field so the verifier can recompute the parent's sum, not just its hash. That extra hash work is what makes Q3 a ~2× longer wall-clock proof than Q2 even though the descent depth is identical.
Query 4 — Compound Equal-only (byRecipientTime)
select = SUM(amount)
where = recipient == "recipient_050" AND sentAt == 50000
sum_property = "amount"
prove = true
Path query:
path: ["@", contract_id, 0x01, "tip", "recipient", "recipient_050", "sentAt"]
query items: [Key(serialize_value_for_key("sentAt", 50000))]
Verified element:
path: ["@", contract_id, 0x01, "tip", "recipient", "recipient_050", "sentAt"]
key: serialize_value_for_key("sentAt", 50000)
element: SumTree { sum_value_or_default: 0 } (no tip at this recipient×sentAt pair)
Proof size: 1 937 bytes. Avg time: 71.8 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (8 layers: 6 path layers + 2 byRecipientTime continuation layers) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
lower_layers: {
tip => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
2: Parent
3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
4: Child)
lower_layers: {
recipient => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[b8804ea3f7998def339db7cadd6a6b29ba5bfadbdfa15b67802ef52e42adb68e]))
1: Push(KVHash(HASH[3056a7c2daf102d31d0f461d45e661303b1562311971debbf15f40938727a074]))
2: Parent
3: Push(Hash(HASH[c406363887b632d071f2ee6ed83df682aace9a5456997aa4f4b944576476fbb6]))
4: Push(KVHash(HASH[6b8ef1df5ba1299cd5b605dc7642be2ffb8356fd7f51c3a0498b6b657cddeebc]))
5: Parent
6: Push(Hash(HASH[347abc0b69e504bc619e9549e509c21be3c0ad1c9e03d8fb34bb5ed07e5cd26f]))
7: Push(KVHash(HASH[ff9b5006130777d589b77e6f5bdda0f86879daace181cdc8e3d97bc3718e0a48]))
8: Parent
9: Push(KVValueHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[264c6aac1a1acd864832a6f25ac42c626afb95dc79ac8c233d2b05c6df048f1c]))
10: Child
11: Push(KVHash(HASH[5159e1ccad8c4ed1bbf7f52ec34e9ab4ee108559194e39b1af78cf42866b32cc]))
12: Parent
13: Push(Hash(HASH[4317777613ab1a0d6966670195ccce83dee1031fd37013c9ce625c016d1d6d17]))
14: Child
15: Push(KVHash(HASH[24f6646d8b75a6a428d05589c1cc1e80f1e52900924710ff6eafe9da0edc73d0]))
16: Parent
17: Push(Hash(HASH[14f8282bd0b5d9a8e7e471d3cfd215f02f5be2cfbf837c5e938c2871a2eddcb9]))
18: Child
19: Child
20: Child
21: Push(KVHash(HASH[cea6360efbf77b38d2ea206d508ea03c0d92fe7c02c5d9176aa322e8263a3acc]))
22: Parent
23: Push(Hash(HASH[209f3325816d5f02a1647311a4f1d9c68fcbacf1540be9b5ae65413f58ca3b26]))
24: Child)
lower_layers: {
0x0000000000000032ffffffffffffffcd00000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[ad21cf215639bca21e163333196de5b1774e0e91bf266a323c3a0b78c8cf7998]))
1: Push(KVValueHash(sentAt, NotSummed(ProvableSumTree(800000000000c7ce, 1000)), HASH[fbe3df47d0e3ee0dbdb9a1491fc05d93ae26f8cb914fb240a42e9989a3752cbc]))
2: Parent)
lower_layers: {
sentAt => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[585d1074d4c73df4e60784323fb6e1cd9a45a7f6fed92ca43ac78a1a575efa16]))
1: Push(KVHashSum(HASH[471d8a8c670fb4bc4c30d251a7fb398f5946a076cfd3c734ce7c131a6a922ae7], 511))
2: Parent
3: Push(Hash(HASH[39bb4483b5c84d178863fb016add610ae44876b0450a9c2f75de97fcdd60c8b5]))
4: Push(KVHashSum(HASH[59629e7456a1ef7da12d0d02c0c3501f68e27f92aa737f088b34a05ddebb42e7], 255))
5: Parent
6: Push(Hash(HASH[99363add734e45862d731777f87ca7a018109aa62aa489e3c5a5515dc965678b]))
7: Push(KVHashSum(HASH[542ff7575944b1293665b9a4ea5a5a49945e2d3206280d9cbded80afe404a7de], 127))
8: Parent
9: Push(Hash(HASH[d1b19a359a223351d14e5f21b8fa3e49a012e0b4bb6237c5d73b08bdd9b4d9e8]))
10: Push(KVHashSum(HASH[79ab80617285543900f09137ef8cc42c3f715a6be449c7453d6babceb602c36d], 63))
11: Parent
12: Push(Hash(HASH[497640ee5a5972a73c9484d129a9df3dc17c0e5f06a631d26e824f817706639a]))
13: Push(KVHashSum(HASH[45ee1ce73fc4156e923393f2b7113f22ae3e10b9e74578c46e263c2fb8623af2], 31))
14: Parent
15: Push(Hash(HASH[97daec9f53ac09843034f165faf5410e54882561f3ee345281cf51f0972e29cc]))
16: Push(KVDigestSum(0x800000000000c31e, HASH[110023b15d77b39d67044847913b994adf94cdb8ecfa5bce2abd4d37d6ac68c7], 7))
17: Parent
18: Push(KVDigestSum(0x800000000000c382, HASH[1ea127fb3003e8de7bcfbe341b29da4cff8b8c119dfa10d47eaca70fb7a62d54], 1))
19: Push(KVHashSum(HASH[177c272d0a693d73e8543f41adbce6d964874573d9eacb6fa7bb8d8d5eefc4a3], 3))
20: Parent
21: Push(Hash(HASH[aad686ea6ee57db81ad554934281a5b4c383efd1dd2142011b11f2bf73872d7f]))
22: Child
23: Child
24: Push(KVHashSum(HASH[d0c8fd2ef0a5ed9e51c335d6a7967e16bc0eb02c6a4fc8e7e1e86ec599717ede], 15))
25: Parent
26: Push(Hash(HASH[d13f24f322e026b7909002ca6b79b76c71aad9117601366b9bc867d649cb885b]))
27: Child
28: Child
29: Child
30: Child
31: Child
32: Child
33: Push(KVHashSum(HASH[f94972b44e022eca9d291dda5a150bf75a9ab5cf759d4211bf5c8643ffbe7585], 1000))
34: Parent
35: Push(Hash(HASH[de4170da8225c95a1ffc8dbeb08e2a92add3ea7ac2b7972eb786bfdb1bee4207]))
36: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. Q4 is the deepest single-key sum proof in the chapter: 8 LayerProofs because the byRecipientTime descent walks recipient → recipient_050 → sentAt → terminator. Layers 1–5 mirror Q2's descent verbatim (root → @ → contract → 0x01 → tip → recipient). Layer 6 lands on recipient_050 whose value is now plain SumTree(73656e744174, 1000) (no FeatureType wrapper because this proof descends into it rather than terminating there). Layer 7 is recipient_050's continuation merk-tree, a single-key tree whose only entry is the sentAt property name pointing at a NotSummed(ProvableSumTree(…, 1000)) — the NotSummed wrapper is the storage-layer signal that this continuation's sum doesn't propagate up to recipient_050's own SumTree (the parent's 1 000 already aggregates the value-tree references; this continuation is just a sibling index, contributing 0). Layer 8 walks the byRecipientTime ProvableSumTree (37 ops, the same per-node sum-bearing shape as Q3's bySentAt) but does NOT terminate at 0x800000000000c350: the queried sentAt-50000 key is absent under recipient_050 (since recipient_050 owns rows 50, 150, …, 99950, none of which are sentAt = 50000). The proof commits a complete merk path with no terminator op, which verify_query reports as an empty results vec (the verifier-side absence proof). Verified root hash 95aa7470…71b6.
Two property-name descents (recipient, then under recipient_050 the byRecipientTime continuation's sentAt). For this specific filter no row lands at recipient_050 ∧ sentAt = 50000 (recipient_050 owns rows {50, 150, …, 99 950}; sentAt = 50000 is owned by recipient_000), so the verified element is a SumTree with sum_value_or_default = 0 proving absence. The proof is still 1 937 bytes — absence proofs walk the same depth as present-key proofs, just with a different terminator merk node.
Diagram: per-layer merk-tree structure
Layers 1–5 are identical to Q2's; Q4's signature divergence happens at layers 6–8 where the byRecipientTime compound index threads through recipient_050's NotSummed(ProvableSumTree) continuation.
flowchart TB
subgraph L6["Layer 6 — byRecipient merk-tree (mid-descent, NOT a target)"]
direction TB
L6_q["<b>recipient_050</b><br/>kv_hash=HASH[264c...]<br/>value: SumTree(73656e744174, sum=1000)<br/>(descends one more layer)"]:::queried
L6_boundary["Boundary commitments (24 ops):<br/>same byRecipient path Q2 walks, just<br/>without the FeatureTypeWithChildHash wrapper"]:::sibling
L6_q --> L6_boundary
end
subgraph L7["Layer 7 — recipient_050 continuation merk-tree (single key)"]
direction TB
L7_q["<b>sentAt</b><br/>kv_hash=HASH[fbe3...]<br/>value: <b>NotSummed(ProvableSumTree(…, 1000))</b><br/>(NotSummed = continuation, contributes 0 to L6's sum)"]:::queried
L7_sib["HASH[ad21...]<br/>(left subtree, opaque)"]:::sibling
L7_q --> L7_sib
end
subgraph L8["Layer 8 — byRecipientTime sentAt ProvableSumTree (TARGET layer, absent key)"]
direction TB
L8_absent["sentAt=50000 (queried key) — <b>absent</b><br/>(no terminator op committed; recipient_050<br/>owns rows 50, 150, …, 99950 only)"]:::target
L8_neighbors["Boundary commitments (37 ops):<br/>KVDigestSum siblings at 0x800000000000c31e (sum=7) and<br/>0x800000000000c382 (sum=1) bracket the absent key<br/>+ 17 KVHashSum / Hash subtree commits across the<br/>ProvableSumTree's per-node sum-bearing structure"]:::pst
L8_absent --> L8_neighbors
end
L6_q -. "value=SumTree(merk_root[r050-continuation])" .-> L7_q
L7_q -. "value=NotSummed(ProvableSumTree(merk_root[byRT.sentAt]))" .-> L8_absent
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The absent-key shape is structurally important: the bracketing siblings 0x800000000000c31e and 0x800000000000c382 (op 16, op 18 in layer 8) are what convince the verifier the queried 0x800000000000c350 doesn't exist between them. The proof commits enough of the merk-tree to recompute the parent root with the gap intact — a non-membership witness in the same proof shape as a membership witness.
Query 5 — In on byRecipient
select = SUM(amount)
where = recipient IN ["recipient_000", "recipient_001"]
sum_property = "amount"
prove = true
Path query (per-In-value point-lookup fan-out):
path: ["@", contract_id, 0x01, "tip", "recipient"]
query items: [Key("recipient_000"), Key("recipient_001")]
Verified entries (two SumEntrys under the entries variant):
SumEntry { in_key: None, key: "recipient_000", sum: 1000 } (amount=1)
SumEntry { in_key: None, key: "recipient_001", sum: 2000 } (amount=2)
Proof size: 1 168 bytes for k=2. The bench's report_proof_sizes measures the same shape at k=100 across all distinct recipients — 12 064 bytes, scaling roughly linearly with |In values| because each branch's merk descent is independent. The per-branch marginal cost is ≈ 109 bytes (10 880 / 99 marginal branches). Avg time: 42.8 µs (k=2) / 1 720 µs (k=100) — roughly 17 µs of marginal time per added In value, consistent with the per-branch merk-descent cost on byRecipient.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (6 layers, dual-target merk-path at the bottom) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
lower_layers: {
tip => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
2: Parent
3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
4: Child)
lower_layers: {
recipient => {
LayerProof {
proof: Merk(
0: Push(KVValueHashFeatureTypeWithChildHash(0x0000000000000000ffffffffffffffff00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[2c932396123fe6bae5fa3e4fa42225852e08a2dcf7600991e79fb9347de7506e], BasicMerkNode, HASH[307798135a4d282b0306f8f0a652c99391fb89a193b1f219632c956b31fb1351]))
1: Push(KVValueHashFeatureTypeWithChildHash(0x0000000000000001fffffffffffffffe00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[43c65eed82b8e5d08b7fea673bf120a82332adbffb8582e0f8a3205190fab720], BasicMerkNode, HASH[8723320b1000d7ca62f0057154cc9ce7e79b31ceeec9e5ea16db382efdf6a81f]))
2: Parent
3: Push(Hash(HASH[e0442e4426d554662f91691fba1bec4c0eddac54662493b3964b08538fea33fe]))
4: Child
5: Push(KVHash(HASH[2f5f739085124d63217a3b5c8276da943d0901c1fc482010e5a50c2f73b36549]))
6: Parent
7: Push(Hash(HASH[93e907e7310d06719783bd32af4653a620b3d2f46c9b54bf1d5edb45928866f7]))
8: Child
9: Push(KVHash(HASH[ca38e3d00da85a2b31477b69b9f71e0c5535a1d9c1bca5b54c875444f7a0bad9]))
10: Parent
11: Push(Hash(HASH[274c7bf51d753c5ed1cc44997d2ed956e30c6574609c165b073a2c6f75e9af58]))
12: Child
13: Push(KVHash(HASH[db46825673ab3057db34992612299ceeca04f94152ac048f647cc92afbd1d771]))
14: Parent
15: Push(Hash(HASH[af91e1e902af6a64f73a4f4abd49ccb5893029568e23ec2e73452dd1fb58940a]))
16: Child
17: Push(KVHash(HASH[3056a7c2daf102d31d0f461d45e661303b1562311971debbf15f40938727a074]))
18: Parent
19: Push(Hash(HASH[22b84dcbedf3b829ad645fd68fa5eb1c59dadb2ab3cd098efd998ca247902c95]))
20: Child
21: Push(KVHash(HASH[cea6360efbf77b38d2ea206d508ea03c0d92fe7c02c5d9176aa322e8263a3acc]))
22: Parent
23: Push(Hash(HASH[209f3325816d5f02a1647311a4f1d9c68fcbacf1540be9b5ae65413f58ca3b26]))
24: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. Same 5-layer descent as Q2 (root → @ → contract → 0x01 → tip → recipient); the divergence is at the bottom layer where the proof reveals two KVValueHashFeatureTypeWithChildHash terminator ops back-to-back (op 0 for recipient_000, op 1 for recipient_001), each carrying its own SumTree(73656e744174, …) with per-recipient sum (1000 and 2000 respectively). The 24-op boundary walk that surrounds them is the same shape as Q2's — proving the two queried keys' positions inside byRecipient's ~100-entry merk-tree by committing the bracketing opaque siblings. Verified entries via verify_query are two (path, key, SumTree { sum_value_or_default: … }) rows, one per In branch. Verified root hash 95aa7470…71b6.
The entries variant carries in_key: None because the In is itself the terminator (no compound-prefix prefix). Compare with Query 6, which has the same property-and-position In but on a rangeSummable index.
Diagram: per-layer merk-tree structure
Layers 1–5 mirror Q2's prefix exactly. The divergence is at layer 6: where Q2 had one cyan target, Q5 has two — and the boundary commits between them are shared by both descents.
flowchart TB
subgraph L6["Layer 6 — byRecipient merk-tree (DUAL TARGET layer)"]
direction TB
L6_t0["<b>recipient_000</b><br/>kv_hash=HASH[2c93...]<br/>value: <b>SumTree sum=1000</b><br/>child_hash=HASH[3077...]"]:::target
L6_t1["<b>recipient_001</b><br/>kv_hash=HASH[43c6...]<br/>value: <b>SumTree sum=2000</b><br/>child_hash=HASH[8723...]"]:::target
L6_boundary["Boundary commitments (~22 ops shared across<br/>both descents):<br/>6 KVHash internal siblings + 6 Hash subtree commits<br/>(prove recipient_000/recipient_001 are adjacent<br/>in byRecipient's binary merk tree)"]:::sibling
L6_t0 --> L6_boundary
L6_t1 --> L6_boundary
end
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
Per-branch byte cost is sub-linear because the two targets share most of the merk-boundary walk (only their kv-hashes themselves are per-branch; siblings amortize). At k=100 with all 100 recipients covered by 2 In branches each, every byRecipient entry becomes a target and only the merk root's two-level boundary stays opaque — the most efficient byte-per-key shape an In on byRecipient can hit.
Query 6 — In on bySentAt (RangeSummable)
select = SUM(amount)
where = sentAt IN [0, 1]
sum_property = "amount"
prove = true
Path query:
path: ["@", contract_id, 0x01, "tip", "sentAt"]
query items: [Key(serialize_value_for_key("sentAt", 0)),
Key(serialize_value_for_key("sentAt", 1))]
Verified entries:
SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 0), sum: 1 }
SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 1), sum: 2 }
(Per the bench's amount = (row % 10) + 1 schedule, row 0 has amount = 1 and row 1 has amount = 2.)
Proof size: 1 756 bytes for k=2; 9 784 bytes at k=100. Per-branch marginal cost ≈ 81 bytes — somewhat less than Query 5's 109 bytes per branch, because the bySentAt subtree's distinct keys are dense integers (one per row) and share more merk-path prefix than byRecipient's hash-derived keys. Avg time: 81.2 µs (k=2) / 2 008 µs (k=100) — slightly higher per-branch time than Q5 despite smaller per-branch bytes; the ProvableSumTree's per-node sum-field hash work eats most of the prefix-sharing savings.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (6 layers, dual-target in a ProvableSumTree) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
lower_layers: {
tip => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
1: Push(KVHash(HASH[a17138f666ae4ab19c1c2930ad94c7e29a6a82789398fc4d3a0b053d3499ac68]))
2: Parent
3: Push(KVValueHash(sentAt, ProvableSumTree(800000000000ffff, 550000), HASH[3b3f5bf4e079c639895d84f8c5003fe135ccb46bf81b29fd3bdf415cddffe45c]))
4: Child)
lower_layers: {
sentAt => {
LayerProof {
proof: Merk(
0: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000000000, SumTree(00, 1), HASH[7196506382214367916d1e1e0cff254b14f89815f007840414956a077dd53148], ProvableSummedMerkNode(1), HASH[ef96a09b8f07fdbb7d17c3e86a7bdc954aae31c8e2a51900ed2b7d92a6ab9953]))
1: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000000001, SumTree(00, 2), HASH[6b4e9f17f2121cb658ec356b970a91b5ad79032a57e304a3507f63d813f2da8d], ProvableSummedMerkNode(6), HASH[8abb1cbf9ae45b42ff519e6f187bd4fbadee93ab64afef59089aaa94ed46b502]))
2: Parent
3: Push(Hash(HASH[e03fc8c98c17d3574f4fe6a822916775844555dd63276f2e7289f11ec5aa043d]))
4: Child
5: Push(KVHashSum(HASH[4eb5727c0eb7018148785596abb4767ae93002c4ab76b4f565d31b5235b55a39], 28))
6: Parent
7: Push(Hash(HASH[843e47cd7806eb5309f780a529fe26b245b22cedd46ee3ea62ac23ab6767d59b]))
8: Child
9: Push(KVHashSum(HASH[af802589532730912220f760f96036c67bb64c60dc4cdc468d1c978cb78972fe], 70))
10: Parent
11: Push(Hash(HASH[9adc74e0427bccca23c5adbd648610c340edec52566da65d51f42bcfb1f78f93]))
12: Child
13: Push(KVHashSum(HASH[deb2eb99281aa70288d4accac22faba3681fcdd0258bd5c6d06a2b7ee6cb6624], 166))
14: Parent
15: Push(Hash(HASH[7cd9232374df6e9e98d64e199876edcc91d1016b6541acb867016bcdea24b7d3]))
16: Child
17: Push(KVHashSum(HASH[2acdf1c54d23ebf0874620d776ec72326bc88300a1fd724fbc24ed8c2d921b32], 336))
18: Parent
19: Push(Hash(HASH[2f62bfca826600367f5802e4ff86ac652a225e87dbddcea3d7b38fc922d18b59]))
20: Child
21: Push(KVHashSum(HASH[45f178d25bec1d9b81115e2f023db10ea0117d3c508409d61c2390a9244484e1], 688))
22: Parent
23: Push(Hash(HASH[f983a5b23027388a35323554689f394267399ad40071c6feaeeebd415d5e22d7]))
24: Child
25: Push(KVHashSum(HASH[fe3e84de3d4be2e935b0408a0ef427e277292d7675e88fabf0b9a71346f1ad7d], 1390))
26: Parent
27: Push(Hash(HASH[4552635b13dd23588d3fcdf3382a0a7fe0600f73d118f28b27507bbd7113842b]))
28: Child
29: Push(KVHashSum(HASH[d49e4a23ef475529c6b7116795a0f587d3dd191a6c51b42641e277623afa03e5], 2806))
30: Parent
31: Push(Hash(HASH[27cd36370fc15a994480197224b944ba24f775dcb6b34f28b74f25c0dd0582b9]))
32: Child
33: Push(KVHashSum(HASH[5e4ce1f7f36aca9f81c33b45d945c84488fdf1915402b07914cbbd0f35917af1], 5616))
34: Parent
35: Push(Hash(HASH[1ccfd53d7a30abdd7171a5a0ce63d22c7266ed0879e8353463fefb271112ba13]))
36: Child
37: Push(KVHashSum(HASH[26794659891cef5a5e0edf7947bbca061307b947783887b25ea3323277211af3], 11248))
38: Parent
39: Push(Hash(HASH[1a7593e8e1d66a041bf127649256d0b836368c3f41ec58fa714fbaceb6b5e9fb]))
40: Child
41: Push(KVHashSum(HASH[c42b2154308418eabefedc2b828dbc599009fbb2b0965f057480d8f9cf1e096d], 22510))
42: Parent
43: Push(Hash(HASH[3b81157cc8121f546f746342f60a4d6b325b965874342c094490f1d21f0cdef3]))
44: Child
45: Push(KVHashSum(HASH[2d2bd624f09591fdecf3068dbe47314ff108f3cb4f99482ebd0abbb9c947e257], 45046))
46: Parent
47: Push(Hash(HASH[71958692ac25e70f1d1df3fb5b85c2b90eac37ae745be90b931992b2f9dbfa94]))
48: Child
49: Push(KVHashSum(HASH[150a4c0a44464b24dbde8a45106e370fc189b31676bfd6284763923381a6d434], 90096))
50: Parent
51: Push(Hash(HASH[5e468c0f9805ee381d4c59dd961adc8dee1ab003834aa17fadd9f51411804147]))
52: Child
53: Push(KVHashSum(HASH[a133f55e1f33e2b3a2a78f9f89c3aff8fd06f0ad25254490a42beef2c130e9dd], 180208))
54: Parent
55: Push(Hash(HASH[620ccd98c998cb3a503ee12098a6617743a99ffeb644069927ce7ebd8b70f76a]))
56: Child
57: Push(KVHashSum(HASH[be99a0622f1400aaa04dc460566babac9c1a4955b3eeb1c5780dfd46ec716222], 360430))
58: Parent
59: Push(Hash(HASH[f404bfecb3e178393455544d654039f0cb77c0e0d918cc00d85d22d2707cbf0d]))
60: Child
61: Push(KVHashSum(HASH[ca275ed3d5729250a4a67e2cc90fac909086fac99e386fc90688df2bb01b3eaa], 550000))
62: Parent
63: Push(Hash(HASH[8593bd2bc903da34da11e15f9a649cec37b39487ad59375020adef300a8b7482]))
64: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. Same first 5 layers as Q3 (the descent to bySentAt). The bottom layer reveals both queried timestamps as adjacent KVValueHashFeatureTypeWithChildHash terminator ops: op 0 for sentAt=0 (SumTree(00, 1), amount = 1) and op 1 for sentAt=1 (SumTree(00, 2), amount = 2). Each terminator's feature-type marker is ProvableSummedMerkNode(n) with n matching the per-node committed sum — this is the per-node-sum machinery that makes the surrounding bySentAt tree a ProvableSumTree. The 62-op boundary walk surrounds them with the same per-node sum-bearing structure as Q3, except now two leaves are revealed instead of one. Verified root hash 95aa7470…71b6.
Same structural shape as Query 5 — the difference is the property-name subtree is a ProvableSumTree rather than a NormalTree, so each descent step on the In branches carries a sum field. For point lookups that's pure overhead (we don't use the per-node sums); the payoff lands on Query 7.
Diagram: per-layer merk-tree structure
Layers 1–5 mirror Q3's; the difference is at layer 6 where two adjacent leaves are revealed inside the ProvableSumTree.
flowchart TB
subgraph L6["Layer 6 — bySentAt ProvableSumTree merk-tree (DUAL TARGET layer)"]
direction TB
L6_t0["<b>sentAt=0 (0x8000000000000000)</b><br/>kv_hash=HASH[7196...]<br/>value: <b>SumTree(00, sum=1)</b><br/>feature: ProvableSummedMerkNode(1)<br/>child_hash=HASH[ef96...]"]:::target
L6_t1["<b>sentAt=1 (0x8000000000000001)</b><br/>kv_hash=HASH[6b4e...]<br/>value: <b>SumTree(00, sum=2)</b><br/>feature: ProvableSummedMerkNode(6)<br/>child_hash=HASH[8abb...]"]:::target
L6_pst["Boundary commitments (62 merk ops):<br/>every internal kv is a KVHashSum (hash + sum)<br/>every opaque sibling is a Hash subtree commitment<br/>(running sums on each boundary node:<br/>28 → 70 → 166 → 336 → 688 → 1390 → 2806 → 5616 → …<br/>→ 550000 at the merk root, the full timeline sum)"]:::pst
L6_t0 --> L6_pst
L6_t1 --> L6_pst
end
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The per-node sum chain 28 → 70 → 166 → … → 550000 is what makes this also a sum-bearing path: a verifier who walked the same descent for an AggregateSumOnRange instead would extract a running sum from the same KVHashSum ops rather than reading individual leaves. That's how Q3 / Q6 (point lookups on a ProvableSumTree) and Q7 (range collapse on the same tree) share infrastructure and differ only in which nodes the proof reveals as terminators.
Query 7 — Range Query (AggregateSumOnRange)
select = SUM(amount)
where = sentAt > 50000
sum_property = "amount"
prove = true
Path query:
path: ["@", contract_id, 0x01, "tip", "sentAt"]
query items: AggregateSumOnRange(RangeAfter(serialize_value_for_key("sentAt", 50000)..))
Verified result (returned by GroveDb::verify_aggregate_sum_query):
(root_hash, sum) where sum = 274 999
49 999 rows have sentAt > 50 000 (the half-open (50000, ∞) range excludes sentAt = 50000 itself). Per the fixture's amount = (row % 10) + 1 schedule, rows 50 001..99 999 cycle through [2, 3, 4, 5, 6, 7, 8, 9, 10, 1] for 4 999 full cycles + a 9-element tail. The sum works out to 4 999 × 55 + (2 + 3 + … + 10) = 274 945 + 54 = 274 999, which matches the verified value byte-for-byte.
Proof size: 3 102 bytes. Avg time: 102.0 µs. Verifier root hash: 95aa74708738c5254c706bbca3245b520022aad949674822218094bca15671b6.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (6 layers, AggregateSumOnRange collapse at the bottom) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
lower_layers: {
tip => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
1: Push(KVHash(HASH[a17138f666ae4ab19c1c2930ad94c7e29a6a82789398fc4d3a0b053d3499ac68]))
2: Parent
3: Push(KVValueHash(sentAt, ProvableSumTree(800000000000ffff, 550000), HASH[3b3f5bf4e079c639895d84f8c5003fe135ccb46bf81b29fd3bdf415cddffe45c]))
4: Child)
lower_layers: {
sentAt => {
LayerProof {
proof: Merk(
0: Push(HashWithSum(kv_hash=HASH[a133f55e1f33e2b3a2a78f9f89c3aff8fd06f0ad25254490a42beef2c130e9dd], left=HASH[d39ec024ef5f61890206e3c6d29638a77b94159869d3f7f2ff1941f2f5e87e25], right=HASH[620ccd98c998cb3a503ee12098a6617743a99ffeb644069927ce7ebd8b70f76a], sum=180208))
1: Push(KVDigestSum(0x8000000000007fff, HASH[3e1b6b140f2f162e572401ab79584a57b7e6bd9aef2a3a11ffc94f983ed1e29e], 360430))
2: Parent
3: Push(HashWithSum(kv_hash=HASH[fe76ca504e6930ba89fde3579cc12e8b087398131edac88ced34c92dc89ad94b], left=HASH[f9636ca299ccd678c7e680dfee0ee20cfc97e7205ef085c0ca543f28b1cb5153], right=HASH[dd2ed3a63ec9da0bc56beae8bd1ce63e3b5873e4ae91105172a72c35ffab89d5], sum=90110))
4: Push(KVDigestSum(0x800000000000bfff, HASH[85990c6978e9933c4c27bfebd6ad8b14293dc3dba72803a7d6052dbeb40fef62], 180214))
5: Parent
6: Push(HashWithSum(kv_hash=HASH[9464d86ed7482bea742921278b684b6b448c0c25353858d1d84d24bb99638373], left=HASH[1108c69f23ffac356496bc7bce37d06859d70fb1f4b8cbb7a1d87980919d4a61], right=HASH[680cdc0f17073459247d5782e373245d2247471f6fbfb600a33f71e8a1ab5480], sum=2808))
7: Push(KVDigestSum(0x800000000000c1ff, HASH[04751262de120fb4c72492011ca10e081eb327ef49262fd03101d51c2f4649e6], 5622))
8: Parent
9: Push(HashWithSum(kv_hash=HASH[4c90aeb3a471fc6dd7f702c31133add5bacf8ad1e463d38b31bc9bee8a7b9c06], left=HASH[b8bde476ff8cb68fd463ba2834b30e2715c3bc6f95fa7b9cc93962b91aa6e6a5], right=HASH[63e03a72eed0862e13fedd63be7e1db08f66ef141a3d6071e38c66bc08504374], sum=1410))
10: Push(KVDigestSum(0x800000000000c2ff, HASH[83fdac1310ddca8772a2f40e5cf1de0834d48ea2ac41832a9bb0893d8cf96464], 2810))
11: Parent
12: Push(HashWithSum(kv_hash=HASH[b7afd069ba319e087f24a7d84c05eeaee3676f751c8af09f62cda8966811ea3b], left=HASH[dc4cef21a317ba562e12fcd541a9f645d027ac9323e69868b82cf36625afef43], right=HASH[051f312ccfcbb07b2b99e78636b11050724c2a573ef8c257e54a3130c5501f00], sum=336))
13: Push(KVDigestSum(0x800000000000c33f, HASH[9a94706e02ff4e42f2d7beb30a0f7747b5b31503a25f3c82be7136ef41f39433], 688))
14: Parent
15: Push(HashWithSum(kv_hash=HASH[4628fb01275e90713dacc4cb940b5c7ca5bf43690f1aee6a34bdf51114f016ef], left=HASH[11160330576883ac71e4842bc0ad1c73d28088b2ea54406c3f519e820ed5556f], right=HASH[73214eec9c3e28b1dd0e4b930e134c536d484f7ef3b64144e59a178c1bc375e8], sum=90))
16: Push(KVDigestSum(0x800000000000c34f, HASH[c1ec8175bf1c39fc6543d151ee1d0f3663b80511139e486c5e21ef248cd291bd], 170))
17: Parent
18: Push(KVDigestSum(0x800000000000c350, HASH[b22d3790d948924dc6311518f2872b53c0590d3cc497d7a653f1ce7b9923e499], 1))
19: Push(KVDigestSum(0x800000000000c351, HASH[af12fe5f400e96dc6517b898330583638bd8fe091ff8ee989a43b10848fe5f60], 6))
20: Parent
21: Push(HashWithSum(kv_hash=HASH[3fbfe732d4eb60c1611bf1658d092de8767129cb7a5bb688b02251d134039e11], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], sum=3))
22: Child
23: Push(KVDigestSum(0x800000000000c353, HASH[00536d93be0133a750702ef0ab5207da8c246fbb9ea731a74db9ac94884e919a], 28))
24: Parent
25: Push(HashWithSum(kv_hash=HASH[5f32751618bcf16785ecfd6e051158b4ebd6ed43432ffe44c4f4d26abf91524e], left=HASH[8ee930cb026de29ec8d652bba6b0644c7aed8bedec2da781805635aaeb61453c], right=HASH[d3d6c02c771eeac87ea43855db8535bb5f7e380046eb0089c7a52856bf120475], sum=18))
26: Child
27: Push(KVDigestSum(0x800000000000c357, HASH[c43d06166234e25f98140da5c573120a4ba81679559bb1876a71efbcabbbe459], 70))
28: Parent
29: Push(HashWithSum(kv_hash=HASH[cee1be881b3ee6049936fd08f6d6080d30209d400a009b8007a99be89c96002d], left=HASH[9f25c5bd27c6ec98ef1f74e31413413c76dd011377a07775817dd9d4f31bd95f], right=HASH[6896b09cfa0768780de9464637f284406ddc7c45e47f562518bf093876aa9a0f], sum=34))
30: Child
31: Child
32: Push(KVDigestSum(0x800000000000c35f, HASH[0bee916b5ca009e480f594b34b4ff0023e003a520b9ae625beb84c53ab3bc93b], 348))
33: Parent
34: Push(HashWithSum(kv_hash=HASH[4ca53cefcf67c600b1c36334925073e1fcb1a3500dbc558428cb8476de92ba37], left=HASH[0f3cc2f97ad17f6ea81ea17d915947fdd47e4e20ccd1d2ff4bd4b81a22515938], right=HASH[4796a3bc0d36b5277ace3d37c112f6f5b6cb5945b1d5cc72bf7efec7bf42d987], sum=172))
35: Child
36: Child
37: Push(KVDigestSum(0x800000000000c37f, HASH[fcb1700a1b6be13cd9f3c3d80abde188701aa1d01b25c24929588a2f858eae9c], 1390))
38: Parent
39: Push(HashWithSum(kv_hash=HASH[0e51a96e8e17135a01dce39490c7f55c49024de520f2870f84eedce77a65ecbb], left=HASH[215439487b3dcef7c0186e740ff1f2da27f8973466615691d6c2dc5db2134417], right=HASH[4911eb7861bbc7ea19c572eb9b76900bc2e0f80fbf178c131b30f6ca8969852a], sum=694))
40: Child
41: Child
42: Child
43: Push(KVDigestSum(0x800000000000c3ff, HASH[c875d92f9ae9adb882f612df44549ea07539a0d25fa915f7831a9d64993627a1], 11262))
44: Parent
45: Push(HashWithSum(kv_hash=HASH[3d4a96f576bd6fa7bf10947e8b6c3884306d374d499ab1f4b510d1d13c2b718b], left=HASH[9c4dc81fa4334b80fb36993450bc59dd16d9eb430e89a55e22678dd7be2c290b], right=HASH[db362866db8cdd17bc56f89f9f920895122e4bcb41b8897bac822d8895c520cc], sum=5634))
46: Child
47: Push(KVDigestSum(0x800000000000c7ff, HASH[277f41ec61fbf2c324169b989c07e1bbae573df54f10c0dce1220daf0ccd9b7d], 22520))
48: Parent
49: Push(HashWithSum(kv_hash=HASH[eeae2cee7b1a1141fbda7f7bce1062e969781102c4387ac4075e5702e7c1f608], left=HASH[9b7d2fd9115f3e799edc4c8bbfe0bca3b29ef9adad6c179984cbe2f632de949b], right=HASH[be02ecdf09c0b45fcd3246cb641fa7afa97bf0b1d10963aa337b93ead23cd74d], sum=11248))
50: Child
51: Push(KVDigestSum(0x800000000000cfff, HASH[e7d6b0ffd061d780d54d7d86cf76c401f35df152d58a362fe2129128aeb75fcd], 45048))
52: Parent
53: Push(HashWithSum(kv_hash=HASH[a641b281dca0faa95010eb5705db941b38c8008b38669c14e33ead3b9c341cf3], left=HASH[ff32269c72ee6af174b05a20db06bf57d766b05684208ff2cf676d0fa5e4b9d8], right=HASH[283d35cdf0761e087db90a9e8dfc9f816db0d3782125e8a3087f3e058b4a5ebb], sum=22520))
54: Child
55: Push(KVDigestSum(0x800000000000dfff, HASH[f3cd6aa655507ab09ee96baa7e1f951324ca052bfc39f14bd1f9443a6b8fc082], 90102))
56: Parent
57: Push(HashWithSum(kv_hash=HASH[5a331ce6ce57978d72fd13599ce6cb2e850c68c206e61941cfd374d6de22c514], left=HASH[b0acdc1413331ec55ef36da32fb0906e6dbc800d9ddb12041ab8ff9370d0bcf2], right=HASH[2753ae0214b4f860dc66dd4644f673f5fd48c13e483bd7088bbbcdac9774c5a0], sum=45050))
58: Child
59: Child
60: Child
61: Push(KVDigestSum(0x800000000000ffff, HASH[3326597e4515f458511d3ed24b5e50ce0b085821e1825d772fc13b500c0250d5], 550000))
62: Parent
63: Push(HashWithSum(kv_hash=HASH[576fea2411d44fafd6be01500f5fcca777ed0233b0d64577631c32f60119beed], left=HASH[f1119f240cf23919e3575090cb4b8999d5b9c6f58b24e0b20ac2b1837b9ec16c], right=HASH[00f34e547329b810797b6c891da04ac6bca733adea73b53eb0696acac942d413], sum=189564))
64: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. Layers 1–5 mirror Q3/Q6's bySentAt descent. The bottom layer (layer 6) is structurally distinct from every previous query: it contains no KVValueHashFeatureTypeWithChildHash terminator ops at all. Instead the proof reveals only the merk-boundary structure of the range (HashWithSum for opaque subtrees that the range straddles, KVDigestSum for boundary kvs whose subtree-sums the verifier needs to sum-up). The verifier's range collapse works by traversing the merk boundary and accumulating the sum fields on HashWithSum / KVDigestSum ops for nodes the range covers, producing a single aggregate i64 = 274 999 along with the recomputed root hash. The op-numbered running sums in the proof above trace the merk's binary descent: op 0's sum=180208 is the merk root's left subtree (= 180 214 minus the boundary kv at 0x…7fff with sum 360 430 partially adjusted), and op 61's KVDigestSum(0x800000000000ffff, …, 550000) is the full-tree's max-sentinel committing the full timeline sum 550 000. The RangeAfter(serialize_value_for_key("sentAt", 50000)..) query item picks out the right-half subtrees from sentAt = 50 001 forward; their cumulative sum is 274 999. Verified via GroveDb::verify_aggregate_sum_query, root hash 95aa7470…71b6.
This is the headline payoff: 49 999 matched timestamps, zero documents materialized, a single committed sum verified in O(log T) bytes — same proof-size profile as count's Query 7. The proof is slightly larger than Query 8's 2 657 bytes because bySentAt covers ~100 000 distinct values (the full timeline) versus byRecipientTime's 1 000 distinct sentAts per recipient — wider merk-trees mean a deeper descent + more boundary commits in the proof.
Diagram: per-layer merk-tree structure
Layers 1–5 are identical to Q3/Q6. Layer 6 is where this query diverges sharply from the point-lookup queries: no individual leaf is the target. The "target" is the range collapse itself — a single committed sum the verifier extracts by walking the boundary and accumulating sum fields.
flowchart TB
subgraph L6["Layer 6 — bySentAt ProvableSumTree (RANGE COLLAPSE, no individual target)"]
direction TB
L6_collapse["<b>AggregateSumOnRange(sentAt > 50000)</b><br/>verified sum = <b>274 999</b><br/>(49 999 timestamps covered, each contributing<br/>amount = (row % 10) + 1 per the fixture cycle)"]:::target
L6_boundary["Range-boundary commitments (64 merk ops):<br/>HashWithSum for opaque subtree commits the range straddles<br/>(each carries the subtree's running sum_i64)<br/>+ KVDigestSum at boundary kvs where the range partially covers<br/>(e.g. op 18 KVDigestSum(0x…c350, …, 1) is the floor-exclusive boundary)<br/>+ op 61 KVDigestSum(0x…ffff, …, 550000) is the max-sentinel<br/>full-tree commitment, used by the verifier to validate completeness"]:::pst
L6_collapse --> L6_boundary
end
classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
The cyan node here represents the aggregate output, not a single revealed kv. That's the structural payoff of rangeSummable: true: the merk-tree's per-node i64 sum fields turn a potentially N-leaf walk into a constant-depth boundary walk that produces a single verified sum. Q7's sum = 274 999 is byte-for-byte recoverable from the 64 boundary ops above without ever revealing the 49 999 individual sentAt leaves.
Query 8 — Compound == + Range (byRecipientTime)
select = SUM(amount)
where = recipient == "recipient_050" AND sentAt > 50000
sum_property = "amount"
prove = true
Path query:
path: ["@", contract_id, 0x01, "tip", "recipient", "recipient_050", "sentAt"]
query items: AggregateSumOnRange(RangeAfter(serialize_value_for_key("sentAt", 50000)..))
Verified result:
(root_hash, sum) where sum = 500
(Per recipient_050, 500 of their 1 000 tips have sentAt > 50 000. Each contributes amount = 1 per the per-recipient cycle described in the fixture narrative, so the sum is 500 × 1 = 500. For a recipient with amount = 10, the same shape would land 500 × 10 = 5 000.)
Proof size: 2 657 bytes — the only end-to-end-verified AggregateSumOnRange proof on this fixture today, since Query 7 is blocked on the top-level promotion bug. Confirms the carrier path through byRecipientTime's NotSummed(ProvableSumTree) continuation works correctly. Avg time: 91.3 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (8 layers: 6 path layers + 2 byRecipientTime continuation layers, range collapse at the bottom) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
lower_layers: {
tip => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
2: Parent
3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
4: Child)
lower_layers: {
recipient => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[b8804ea3f7998def339db7cadd6a6b29ba5bfadbdfa15b67802ef52e42adb68e]))
1: Push(KVHash(HASH[3056a7c2daf102d31d0f461d45e661303b1562311971debbf15f40938727a074]))
2: Parent
3: Push(Hash(HASH[c406363887b632d071f2ee6ed83df682aace9a5456997aa4f4b944576476fbb6]))
4: Push(KVHash(HASH[6b8ef1df5ba1299cd5b605dc7642be2ffb8356fd7f51c3a0498b6b657cddeebc]))
5: Parent
6: Push(Hash(HASH[347abc0b69e504bc619e9549e509c21be3c0ad1c9e03d8fb34bb5ed07e5cd26f]))
7: Push(KVHash(HASH[ff9b5006130777d589b77e6f5bdda0f86879daace181cdc8e3d97bc3718e0a48]))
8: Parent
9: Push(KVValueHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[264c6aac1a1acd864832a6f25ac42c626afb95dc79ac8c233d2b05c6df048f1c]))
10: Child
11: Push(KVHash(HASH[5159e1ccad8c4ed1bbf7f52ec34e9ab4ee108559194e39b1af78cf42866b32cc]))
12: Parent
13: Push(Hash(HASH[4317777613ab1a0d6966670195ccce83dee1031fd37013c9ce625c016d1d6d17]))
14: Child
15: Push(KVHash(HASH[24f6646d8b75a6a428d05589c1cc1e80f1e52900924710ff6eafe9da0edc73d0]))
16: Parent
17: Push(Hash(HASH[14f8282bd0b5d9a8e7e471d3cfd215f02f5be2cfbf837c5e938c2871a2eddcb9]))
18: Child
19: Child
20: Child
21: Push(KVHash(HASH[cea6360efbf77b38d2ea206d508ea03c0d92fe7c02c5d9176aa322e8263a3acc]))
22: Parent
23: Push(Hash(HASH[209f3325816d5f02a1647311a4f1d9c68fcbacf1540be9b5ae65413f58ca3b26]))
24: Child)
lower_layers: {
0x0000000000000032ffffffffffffffcd00000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[ad21cf215639bca21e163333196de5b1774e0e91bf266a323c3a0b78c8cf7998]))
1: Push(KVValueHash(sentAt, NotSummed(ProvableSumTree(800000000000c7ce, 1000)), HASH[fbe3df47d0e3ee0dbdb9a1491fc05d93ae26f8cb914fb240a42e9989a3752cbc]))
2: Parent)
lower_layers: {
sentAt => {
LayerProof {
proof: Merk(
0: Push(HashWithSum(kv_hash=HASH[8a0594fb82ff688d061192d70597c544bc56c18badc2ba8e3ce83372adefa71a], left=HASH[04ea853c427ca0c50abed8b5e6349be94cf55f8988f1e6f314d5b15db57fc4e4], right=HASH[6c6832bf2d87daad24055a9195deeadfe50d4c1b816c79b1bbc6ec212ff94d62], sum=255))
1: Push(KVDigestSum(0x80000000000063ce, HASH[7e46f009482a37f9e01a1a251291ae5bad2d1ffd6e04f1f88b44853c40373401], 511))
2: Parent
3: Push(HashWithSum(kv_hash=HASH[f9ac98fc04f30b63ec49c758c3d8fe7f3657182dab7d6255271d3540c521c3e1], left=HASH[c76899568ea6f6268413bd5c0096ea1a822c60c0e8e8d2ec5738b56e6a4cefed], right=HASH[953541994578d66b8d0d3ac32bef836f3c1d1a0f7a1e55cf99c5312df9f170c4], sum=127))
4: Push(KVDigestSum(0x80000000000095ce, HASH[0e67180c6b3c0ff547de58f5c0b36d63fea5499b4146ebe315825a43590b9e53], 255))
5: Parent
6: Push(HashWithSum(kv_hash=HASH[d7245bd11393b60e38a551ae083e703e4a2584b6eaa70c1389934ae2fcb5f593], left=HASH[165d0e36dcabfbc3a1890a603eedfa51e0e170de6a6961f949031005746ac325], right=HASH[3aefae9c66dfb5bcbfbed5c59816a7ce7b447c7a9432d130e2b42a3c19c0d3d2], sum=63))
7: Push(KVDigestSum(0x800000000000aece, HASH[d27cc74018d6d49e895ad0db3e93d0dd05619e0efa0bb50ffc6691f69a98aa90], 127))
8: Parent
9: Push(HashWithSum(kv_hash=HASH[dc56bbcb92f002e0a7e223cd2860a09121f117de4bcd725676d5b1ff48528d9f], left=HASH[eddcd28c36db5ecef8838d42845fee17a2087e0831ff6813ea2d38f3eaf1443f], right=HASH[bca8a2fc6c5a4c5025a65d845a5b54ac15bbfbbd3aebbba95c9dc38d0e94d3bb], sum=31))
10: Push(KVDigestSum(0x800000000000bb4e, HASH[89151daf6d33cc9634cc0bcd93ffd0ab0be5aa9d4ddac159aa383b589432d14d], 63))
11: Parent
12: Push(HashWithSum(kv_hash=HASH[749de6348d21966ee040946142c7e4dce7b113041032dc9cc95d4c0e72bacdc9], left=HASH[32eee9d294a45839a77c1a0d6c05fa3bbebeb0435b4894409bb5863ac7e050d2], right=HASH[0b251e9efd6d616d3f9c5e361a5032b24d4526b556e16ee04a02984d9aa002d0], sum=15))
13: Push(KVDigestSum(0x800000000000c18e, HASH[fa15c265143026dd1e34dbc5b986ba9839069a2c01692712954beb492a096d0e], 31))
14: Parent
15: Push(HashWithSum(kv_hash=HASH[6f8df1c4654e54ea20cd4dad686fc471cb8f94ee6cfc7c4a72bd00792b594138], left=HASH[3c80faf26e8f2a6a266234f2367a0ce908926b72a4092fc7c3b02bebe1c47e85], right=HASH[390d9a0a56902ab7b8611d68e07f0068667302802b4bcffe71796c01d0a0ccae], sum=3))
16: Push(KVDigestSum(0x800000000000c31e, HASH[110023b15d77b39d67044847913b994adf94cdb8ecfa5bce2abd4d37d6ac68c7], 7))
17: Parent
18: Push(KVDigestSum(0x800000000000c382, HASH[1ea127fb3003e8de7bcfbe341b29da4cff8b8c119dfa10d47eaca70fb7a62d54], 1))
19: Push(KVDigestSum(0x800000000000c3e6, HASH[35c7ea9920ba64ff253a6e16dd8ed0a38231221d2bdd6e769c35e96634ae994b], 3))
20: Parent
21: Push(HashWithSum(kv_hash=HASH[722fcc58984fccf6edb3dc0bb7bda4f6e35c57981fb6f70a937df696c079893f], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], sum=1))
22: Child
23: Child
24: Push(KVDigestSum(0x800000000000c4ae, HASH[23944bfe2e2a1e79e00e267e0ab5ff93748f8e7273049e02f91dbf97892ccbdb], 15))
25: Parent
26: Push(HashWithSum(kv_hash=HASH[b59968c7e2c22578ed837d29716b533315b0a1717de0a3165126829aeeac00fc], left=HASH[1c2fcdb483a1ed6e433b0c42056c20b68502f6ac2f074daa3335cd3508d2b058], right=HASH[ecf421d6c9e3e68bafef2663530a1a87339ed7b7cd66539991d9c32051dc47f1], sum=7))
27: Child
28: Child
29: Child
30: Child
31: Child
32: Child
33: Push(KVDigestSum(0x800000000000c7ce, HASH[978baee06712b4ec8bcb7c36e140dc5c4031e2d68b3aa15c2167870a2a45778a], 1000))
34: Parent
35: Push(HashWithSum(kv_hash=HASH[db8a2deb3aa116b1e0d5a1de03b8c4deed54d2ab06ad693238424a7a68288008], left=HASH[f58ece054ce9b084e4042e0ee2e03c8dddde53f1936c94e733c4ef5c146c597e], right=HASH[18f3c1450fce7a85cf5eb87ccc2c4008ef350b18d72de000f2e0606788862659], sum=488))
36: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. The first 7 layers mirror Q4's compound prefix descent: root → @ → contract → 0x01 → tip → recipient → recipient_050 (with the NotSummed(ProvableSumTree) continuation pointer at layer 7). Layer 8 is the AggregateSumOnRange collapse on byRecipientTime's continuation merk-tree — same structural shape as Q7's bottom layer (HashWithSum opaque subtree commits + KVDigestSum boundary kvs), but covering only the 1 000 sentAt entries under recipient_050 instead of the global 100 000. The verifier walks the merk boundary and accumulates sum fields for the right-half range (RangeAfter), yielding sum = 500 (recipient_050 has 500 tips with sentAt > 50000, each contributing amount = 1). Note op 33's KVDigestSum(0x800000000000c7ce, …, 1000) is the max-sentinel committing recipient_050's full 1 000-tip sum; op 35's HashWithSum(…, sum=488) is the parent subtree commit on the right of the range floor. Verified via GroveDb::verify_aggregate_sum_query, root hash 95aa7470…71b6.
Two-prefix descent (recipient_050 → sentAt) followed by an AggregateSumOnRange over byRecipientTime's continuation ProvableSumTree. The same O(log T') range collapse the unfiltered range query (Query 7) is intended to use, just with a compound prefix paid for at the boundary subtrees on the way down.
Diagram: per-layer merk-tree structure
Layers 1–7 mirror Q4's compound-prefix descent (recipient → recipient_050 → sentAt-continuation). The divergence is at layer 8: instead of a single absent terminator op, the proof presents an AggregateSumOnRange boundary walk that collapses to a single aggregate sum.
flowchart TB
subgraph L8["Layer 8 — byRecipientTime sentAt ProvableSumTree (RANGE COLLAPSE under recipient_050)"]
direction TB
L8_collapse["<b>AggregateSumOnRange(sentAt > 50000)</b><br/>verified sum = <b>500</b><br/>(500 of recipient_050's 1000 tips match,<br/>each contributing amount = 1 per the cycle)"]:::target
L8_boundary["Range-boundary commitments (37 merk ops):<br/>same HashWithSum + KVDigestSum shape as Q7's layer 6,<br/>but on a 1000-leaf merk-tree (recipient_050's per-recipient<br/>continuation) instead of the global 100 000-leaf tree.<br/>Op 33 KVDigestSum(0x…c7ce, …, 1000) is the max-sentinel<br/>(per-recipient max sentAt). Op 35 HashWithSum(…, sum=488)<br/>is the parent commit on the right of the range floor."]:::pst
L8_collapse --> L8_boundary
end
classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
Q8 is smaller than Q7 (2 657 vs 3 102 bytes) for exactly one reason: the bottom layer's merk-tree is the per-recipient continuation (1 000 distinct sentAts under recipient_050) instead of the global timeline (100 000 distinct sentAts). Shallower merk-tree, shallower descent, fewer boundary commits. The Q7-Q8 byte delta is the structural cost of taking the global bySentAt path versus a compound-prefix descent.
Query 9 — Carrier-Aggregate (In plus range)
select = SUM(amount)
where = recipient IN ["recipient_000", "recipient_001", ..., "recipient_099"] AND sentAt > 50000
group_by = [recipient]
limit = 100
sum_property = "amount"
prove = true
This is the carrier-aggregate sum shape: an In clause on the index's prefix property combined with a range on its terminator, returning one sum per resolved In-bucket rather than a single aggregate across all matches. The group_by = [recipient] (not [recipient, sentAt]) routes through SumMode::GroupByIn — the routing table at mode_detection/v0/mod.rs maps (GroupByIn, In, range, prove) → RangeAggregateCarrierProof, which is what the per-In-bucket aggregation needs. A group_by = [recipient, sentAt] (GroupByCompound) routes to RangeDistinctProof instead — per-(in_key, range_key) distinct walk, a different proof shape entirely. Sum analog of count's Range-Countable group-by carrier-aggregate. The primitive landed in grovedb PR #670 (head e98bab5f); the verifier is GroveDb::verify_aggregate_sum_query_per_key.
Path query (carrier-style: outer Query enumerates the In branches, subquery descends through the terminator's AggregateSumOnRange):
path: ["@", contract_id, 0x01, "tip", "recipient"]
query items: [Key("recipient_000"), Key("recipient_001"), ..., Key("recipient_099")]
subquery_path: ["sentAt"]
subquery items: AggregateSumOnRange(RangeAfter(serialize_value_for_key("sentAt", 50000)..))
Verified result (returned by GroveDb::verify_aggregate_sum_query_per_key):
(root_hash, entries) where entries =
[ ("recipient_000", 499 × 1 = 499),
("recipient_001", 500 × 2 = 1000),
("recipient_002", 500 × 3 = 1500),
...
("recipient_009", 500 × 10 = 5000),
("recipient_010", 500 × 1 = 500),
...
("recipient_099", 500 × 10 = 5000) ]
(Per recipient, 500 of their 1 000 tips have sentAt > 50 000, and each recipient's per-doc amount is constant per the fixture's per-recipient cycle — so each outer-bucket sum is 500 × amount_for_recipient. recipient_000 is the lone exception: the boundary timestamp sentAt = 50 000 happens to be one of recipient_000's tips, so only 499 of recipient_000's tips satisfy sentAt > 50 000. The bench's verified entries lead with ("recipient_000", sum = 499).)
Proof size: 169 064 bytes (k=100 outer buckets). That's significantly larger than the non-carrier In shapes (Q5 12 064 B for k=100, Q6 9 784 B for k=100) because each of the 100 outer buckets walks an AggregateSumOnRange proof, whereas Q5/Q6 only commit a single-element value-tree sum_value per branch. The per-bucket marginal cost is ≈ 1 690 bytes (most of which is the inner range proof — the carrier composition itself adds only ≈ 100 bytes per outer Key versus the equivalent flat-In shape). Avg time: 11 507.9 µs (k=100) — ≈ 115 µs per outer bucket, consistent with Q8's 91.3 µs single-bucket AggregateSumOnRange plus the carrier descent's per-outer-key cost.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (206 LayerProofs total — recipient_000's full descent shown; recipient_001..recipient_099 abbreviated for readability) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289), HASH[2f2bb4914c2a17715b3084357a87925d56371820af54019ed45f53b0f2ff352f]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289, Tree(01), HASH[5b1ed2f146918bb1d0f6fafa85f17558e030a4337c9cb85d9364da64ae1801ec])))
lower_layers: {
0x4ed22624752972af97fb71abf4067b23e6d296a61a02f35b2098819fde39d289 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c53ef5d17d01aba0f72670d88a09905db45e8639ffe72a77ab68e00770b1334b]))
1: Push(KVValueHash(0x01, Tree(746970), HASH[fba96529e5faf688cd9f3544b9f7979fde626476f4022e4cee50138ceb0295d2]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(tip, Tree(726563697069656e74), HASH[0fe5cf57426e356c1cfcc44a0c35c1dabf78aebdd7ef26d070950a2c7ff86800])))
lower_layers: {
tip => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15fd7f83e57e9b83533d5df4eacabf0e99a861c6cc59a539b8f486a02414babd]))
1: Push(KVValueHash(recipient, Tree(000000000000003fffffffffffffffc000000000000000000000000000000000), HASH[f5801a1723ac6dfd5ff650eaa97d8b134255eef3ab8429dd516690dcfac74221]))
2: Parent
3: Push(Hash(HASH[143e80400b4fe5e0de4201bdf02853ac92347483a4a559040aa4ccb1ff4e3b03]))
4: Child)
lower_layers: {
recipient => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x0000000000000000ffffffffffffffff00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[2c932396123fe6bae5fa3e4fa42225852e08a2dcf7600991e79fb9347de7506e]))
1: Push(KVValueHash(0x0000000000000001fffffffffffffffe00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[43c65eed82b8e5d08b7fea673bf120a82332adbffb8582e0f8a3205190fab720]))
2: Parent
3: Push(KVValueHash(0x0000000000000002fffffffffffffffd00000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[e4cf97fee88b23d122998ea98dbc83fc662da5e0d80c1f854758b2ce56699289]))
4: Child
5: Push(KVValueHash(0x0000000000000003fffffffffffffffc00000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[b552d6302d6bd0f76277a1b56423f25efec67612d42c924e0d486202f31e3e44]))
6: Parent
7: Push(KVValueHash(0x0000000000000004fffffffffffffffb00000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[01be3636585f11d5a09875214cb8e7c68cbe462c739f012693ae3404c58f1179]))
8: Push(KVValueHash(0x0000000000000005fffffffffffffffa00000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[2cd10cc36cb8056b17df3d117b6687d9bd857a79d1a73f2b85290d87a9ce4a24]))
9: Parent
10: Push(KVValueHash(0x0000000000000006fffffffffffffff900000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[f020781d83a04f027649d42a0dc92f91506ae35765844773f72e59071cb1f107]))
11: Child
12: Child
13: Push(KVValueHash(0x0000000000000007fffffffffffffff800000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[9fba5594f7f0b758415ae7159919102c1a2c8aecc98af22c36da32d83199f475]))
14: Parent
15: Push(KVValueHash(0x0000000000000008fffffffffffffff700000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[64188af510ecd884d09350ae08a5f42d5922321408aa552ec9af6c847d264a7e]))
16: Push(KVValueHash(0x0000000000000009fffffffffffffff600000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[304dc0b23c2eac4c2e8cf7eddd7b3c57bee411ae03ce59b9248c0b5ad89db242]))
17: Parent
18: Push(KVValueHash(0x000000000000000afffffffffffffff500000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[97b584dfeb399a5278c736c1c6668c12765926780a15e8e93678903734c61aa3]))
19: Child
20: Push(KVValueHash(0x000000000000000bfffffffffffffff400000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[76b3db0ddb6b2c5756cbc4d66cd37109501665ef122f6d1ce20c4a5566a0bf25]))
21: Parent
22: Push(KVValueHash(0x000000000000000cfffffffffffffff300000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[a81ae2020176545244f9cc4ef9b86f014802b6d11908985f38389f85b4222377]))
23: Push(KVValueHash(0x000000000000000dfffffffffffffff200000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[d46254b6791a7c20ebcb43b2c13fbd73a07704cce56bc78357719d48ae27d1ef]))
24: Parent
25: Push(KVValueHash(0x000000000000000efffffffffffffff100000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[1e8cdba5aa96432bc4a5cd1bb4f2808f2c48debdc450d7f878076217b3fd38aa]))
26: Child
27: Child
28: Child
29: Push(KVValueHash(0x000000000000000ffffffffffffffff000000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[62d0fc12b138b76a1686f65ee269cbe91156aeb3da7b50dd3fd908eb1e6bf9c2]))
30: Parent
31: Push(KVValueHash(0x0000000000000010ffffffffffffffef00000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[c43bd23a1c31c624127621e4480ca21d5b2a03725035d5b74301f6780122af92]))
32: Push(KVValueHash(0x0000000000000011ffffffffffffffee00000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[b9d2ff1ba31cc5ab8293e2a2c14609bf99adf6acc262c02801c15b04765c1857]))
33: Parent
34: Push(KVValueHash(0x0000000000000012ffffffffffffffed00000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[9fbbfcf2a78087d8f112050fa354e424f897638b570dc7afc82bf286a48aa2bd]))
35: Child
36: Push(KVValueHash(0x0000000000000013ffffffffffffffec00000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[929742b49a2ccf0ab75af768d9f37ac226212d4170f190e58f9f07f3e59b1500]))
37: Parent
38: Push(KVValueHash(0x0000000000000014ffffffffffffffeb00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[5131ff85f089920a94f6a9bf54f7db8bc753aa723485ae73f719acfb8992af87]))
39: Push(KVValueHash(0x0000000000000015ffffffffffffffea00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[7a29693947430c79f6ecbd83cecc684e696b4303bdbbc110998408f57e72b6ac]))
40: Parent
41: Push(KVValueHash(0x0000000000000016ffffffffffffffe900000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[8b1b865f2a8118504ac42fbab6c2b621f155043eceb8fbc65b1ac60d698a2dd3]))
42: Child
43: Child
44: Push(KVValueHash(0x0000000000000017ffffffffffffffe800000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[3f98dbb6d76a0eff5dd84171dc4b610ce2e4148f0dd2347a064eff194ed73878]))
45: Parent
46: Push(KVValueHash(0x0000000000000018ffffffffffffffe700000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[1e7835c334b773fff98899fcdeb173516c5f61640e96bbc54a9b2347b3496592]))
47: Push(KVValueHash(0x0000000000000019ffffffffffffffe600000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[c77112c339a35d47922a4618d08315dab1a0d3cc2675d3416f20be077beffff6]))
48: Parent
49: Push(KVValueHash(0x000000000000001affffffffffffffe500000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[fa5fb0940cf60976030fa3ed1b215f2b8685bf61d5df0b1d071a0055a60b8669]))
50: Child
51: Push(KVValueHash(0x000000000000001bffffffffffffffe400000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[582775f7449c7c96988ea45a70744b17967586af24fc23254241961f04af89da]))
52: Parent
53: Push(KVValueHash(0x000000000000001cffffffffffffffe300000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[c71003aa7b862d8a1546fae82a4b0d40cd5bf00be0b3fa748295638262c6e97e]))
54: Push(KVValueHash(0x000000000000001dffffffffffffffe200000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[d200a08e3400945ec481e35843f5970c2040c5d649c0936249ae223373f66933]))
55: Parent
56: Push(KVValueHash(0x000000000000001effffffffffffffe100000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[e399c91063655e659638f034cb37aa38be8aa6591826afb8150168e23bab062c]))
57: Child
58: Child
59: Child
60: Child
61: Push(KVValueHash(0x000000000000001fffffffffffffffe000000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[6e393e0b7b898fae65df835b64af0060496d6b1f11e9eacbac8c3f5850136185]))
62: Parent
63: Push(KVValueHash(0x0000000000000020ffffffffffffffdf00000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[a39a5c32c8da4dd50b76788881dd61af87f88430984fcc1e3e3bda3f695060b7]))
64: Push(KVValueHash(0x0000000000000021ffffffffffffffde00000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[655cc67f6eb581596c76e146f9ce55294d53bfcbec54646afde11c7adaebdcbf]))
65: Parent
66: Push(KVValueHash(0x0000000000000022ffffffffffffffdd00000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[a13be8916a62ae0d1b924b74b5e5524f905ed3f5d980bb0be9d53fe224b0b089]))
67: Child
68: Push(KVValueHash(0x0000000000000023ffffffffffffffdc00000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[2bf66732319328bcf5588bef2b78e33a0d0881e5cec062fa0b87aeda98f6e64a]))
69: Parent
70: Push(KVValueHash(0x0000000000000024ffffffffffffffdb00000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[df49aa0199c39cd22de27d4df6ff98438eb3e3b07b851cc48efb30fbd381c8d7]))
71: Push(KVValueHash(0x0000000000000025ffffffffffffffda00000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[284d47e7e1d21fc8f37dac05a6deaf9f27506000ff5713c8936633174bacb660]))
72: Parent
73: Push(KVValueHash(0x0000000000000026ffffffffffffffd900000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[95bf9e3ea0a08e43d28e954311f6664532d015c8b5e1b0c0155327095aaf6858]))
74: Child
75: Child
76: Push(KVValueHash(0x0000000000000027ffffffffffffffd800000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[bf951ec3b345040d169c258847c11f19c20e1579f492fba4ef478fbddc6c1a7f]))
77: Parent
78: Push(KVValueHash(0x0000000000000028ffffffffffffffd700000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[55ff1a89b12b01d1e31b9f599dec325ea52ada71065011ed2601e6a53373ce6d]))
79: Push(KVValueHash(0x0000000000000029ffffffffffffffd600000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[5f38013b5c19b8272676fa4ad0ba58b3880b195d35b3d3ac2815b8233bf0de30]))
80: Parent
81: Push(KVValueHash(0x000000000000002affffffffffffffd500000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[0b443a9b31f304014595cdf5129f34da66e1a7274fc3548e8c799805232d3e23]))
82: Child
83: Push(KVValueHash(0x000000000000002bffffffffffffffd400000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[d349113a977fe2c1e5fdffe70a6fec8595da6fcea4882b63f2460516d141bc57]))
84: Parent
85: Push(KVValueHash(0x000000000000002cffffffffffffffd300000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[a198da76b8f9db38a4c80a602d8d2e9ac0ad7758ddb483450b4a3f13e3592479]))
86: Push(KVValueHash(0x000000000000002dffffffffffffffd200000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[33b369e7c75c3d3d362dbf33965d1b19dc96216d7723bebb78f10e49547a56b8]))
87: Parent
88: Push(KVValueHash(0x000000000000002effffffffffffffd100000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[051215e98dc597dd3beec177f691301fd0607916809ed00c36be278bd91a8e65]))
89: Child
90: Child
91: Child
92: Push(KVValueHash(0x000000000000002fffffffffffffffd000000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[0cdd2e9e49edccb1edd6fecb4f2d1d8c9f1355db935a67747974fbd803bb4d76]))
93: Parent
94: Push(KVValueHash(0x0000000000000030ffffffffffffffcf00000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[e9a1c2651036336e091b4ba6f373f616c8409d54ab2d8b8949778d930204c551]))
95: Push(KVValueHash(0x0000000000000031ffffffffffffffce00000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[74853523c4698c7e8cfa1f0f744a3da1480993f92f84805c34836cb0bfc31c26]))
96: Parent
97: Push(KVValueHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[264c6aac1a1acd864832a6f25ac42c626afb95dc79ac8c233d2b05c6df048f1c]))
98: Child
99: Push(KVValueHash(0x0000000000000033ffffffffffffffcc00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[b9cb74f5fabd2c1a81d0cf16213246acadcca2240d5910e4e2b22533d4f87067]))
100: Parent
101: Push(KVValueHash(0x0000000000000034ffffffffffffffcb00000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[257e1c20c28d5cf3bb2959b2c33e75ecbcf454188e2553200315693e7f2124bc]))
102: Push(KVValueHash(0x0000000000000035ffffffffffffffca00000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[738ae56fcfe64ed942f759be3089512fb256097dd246da488998d00c13bb55ee]))
103: Parent
104: Push(KVValueHash(0x0000000000000036ffffffffffffffc900000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[9badbc44201ecd712314ec0086d0d47224d5dfab25b95149aa2fedf39989644c]))
105: Child
106: Child
107: Push(KVValueHash(0x0000000000000037ffffffffffffffc800000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[167e2a54f09ffbf49d152a32371a8035596a3a9a2d839f055f46faa75ade51c3]))
108: Parent
109: Push(KVValueHash(0x0000000000000038ffffffffffffffc700000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[a447c9f23d474c77f7041c698f96a95aa8421d55e6fc7eee58f3b4be0d6caa9c]))
110: Push(KVValueHash(0x0000000000000039ffffffffffffffc600000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[ced5fe912b9e12105bf8a733df895ed8bdd75cce53033a6cb51e78fc361f67a8]))
111: Parent
112: Push(KVValueHash(0x000000000000003affffffffffffffc500000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[f1ec66f96298db46aeceaa1785dd24e479005237456bf8ee6482a8731cac4dd2]))
113: Child
114: Push(KVValueHash(0x000000000000003bffffffffffffffc400000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[f354580129aa8e8c9d0df8ce9842e72c198f4865101825aa64efaba200a18425]))
115: Parent
116: Push(KVValueHash(0x000000000000003cffffffffffffffc300000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[bc33576ad567e77c108ce92c1aafa1122c8642e8849e93fca8e91b8f6532bdf5]))
117: Push(KVValueHash(0x000000000000003dffffffffffffffc200000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[04bde39065544c84157f4c6248222af72678e1536735f870375c7e516d3f3fc4]))
118: Parent
119: Push(KVValueHash(0x000000000000003effffffffffffffc100000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[3bdadb83f854abf1c7aae1fdd86fd6e0c26753e2f813a1494f4ce6ee592371c9]))
120: Child
121: Child
122: Child
123: Child
124: Child
125: Push(KVValueHash(0x000000000000003fffffffffffffffc000000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[604318edaf54250e84ab7756467488be7cf9c6b3252942117fd36d6c53c2b872]))
126: Parent
127: Push(KVValueHash(0x0000000000000040ffffffffffffffbf00000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[c772aab9ebd16e076279e440adc1a8d365eefb36a6d25760a8fe0023b88b5904]))
128: Push(KVValueHash(0x0000000000000041ffffffffffffffbe00000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[448ae13230a21a21f7df14295673b3dd7337536868d9a667db5a756e37be845f]))
129: Parent
130: Push(KVValueHash(0x0000000000000042ffffffffffffffbd00000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[1468ec171f1365f54e1845009bbd9105e86b0f9288a9b0df4789270269558168]))
131: Child
132: Push(KVValueHash(0x0000000000000043ffffffffffffffbc00000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[173dc47f3e1fce81d4bdfffad37bb2fcd7e08923454a07aa5318ac23ace64b00]))
133: Parent
134: Push(KVValueHash(0x0000000000000044ffffffffffffffbb00000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[abd15b12cf6f035db4e163162e765da775cfd78012ce5d20eaad3976eb80b291]))
135: Push(KVValueHash(0x0000000000000045ffffffffffffffba00000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[ffa1ee3178182f5dc40a29afd3a236f8bba9892d1604d04114a32eb07c176f4f]))
136: Parent
137: Push(KVValueHash(0x0000000000000046ffffffffffffffb900000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[693437a92f6dfaee70b1cdb86fd6b208939bb431522a6f8321564f5f2e98b511]))
138: Child
139: Child
140: Push(KVValueHash(0x0000000000000047ffffffffffffffb800000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[bf2a9fb3100355e398e9c869997ac02f9e3ba95a143a08b052adb1ab82450bdb]))
141: Parent
142: Push(KVValueHash(0x0000000000000048ffffffffffffffb700000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[a5824c1c11fb02112dff6b46b3e450abfd4cdff94151dae513348691febf7bcf]))
143: Push(KVValueHash(0x0000000000000049ffffffffffffffb600000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[411d57a7e582d3f692ca98b3634e6aa75aba4681253d1bdb7144530a16bb546e]))
144: Parent
145: Push(KVValueHash(0x000000000000004affffffffffffffb500000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[e49198bc668baa08f93d66640ba7fdadfff9d59593f352bd628a3d6b57a2e53c]))
146: Child
147: Push(KVValueHash(0x000000000000004bffffffffffffffb400000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[c61be4cf68123d695433dc10c79d1600ff15306fd6123336b381f1918800434e]))
148: Parent
149: Push(KVValueHash(0x000000000000004cffffffffffffffb300000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[3a977675229ea451a61ac156f05fcbd91c85f08736666c27785513e98d0001e2]))
150: Push(KVValueHash(0x000000000000004dffffffffffffffb200000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[f4b3e29e2113370f153076839d6159d2d21149779ce88dadcceac6217347bf4c]))
151: Parent
152: Push(KVValueHash(0x000000000000004effffffffffffffb100000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[a98df638bc8e3148d9a7e407f4c7a352d9a6f7713487b6d3e4c5af09a78126d0]))
153: Child
154: Child
155: Child
156: Push(KVValueHash(0x000000000000004fffffffffffffffb000000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[80b994258a9d0e7500bbaeb910405395f7c7d03006fb7a15156d01740205a364]))
157: Parent
158: Push(KVValueHash(0x0000000000000050ffffffffffffffaf00000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[df44fba2712d3ba1d71ed1c1c586064197a41b7b903d3b5cae7f127c6c8a27e7]))
159: Push(KVValueHash(0x0000000000000051ffffffffffffffae00000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[2294f725ab5e79ad583cf60d7a942508fe292262b3fbfd3001aa916290a824d6]))
160: Parent
161: Push(KVValueHash(0x0000000000000052ffffffffffffffad00000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[79ca47b53c05f86e6c251af81b52999f032625e0c11eb0c06d41f2f3f83e1449]))
162: Child
163: Push(KVValueHash(0x0000000000000053ffffffffffffffac00000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[882db62836b8db91de36be15b3e85c22f30d2c7ffbf2357e4196104ea323713f]))
164: Parent
165: Push(KVValueHash(0x0000000000000054ffffffffffffffab00000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[2611abbb866d065a01f8d5dd51576365a84ff715b5cb9fd4b359f91393dd9b00]))
166: Push(KVValueHash(0x0000000000000055ffffffffffffffaa00000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[364d73402be79726b57efabf3758cf40784aeccfbc0dccee7692af12fe66c279]))
167: Parent
168: Push(KVValueHash(0x0000000000000056ffffffffffffffa900000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[22a362b3398c53a32b94e2c916a91a8b5d59c0bce9613658021a02f23896da19]))
169: Child
170: Child
171: Push(KVValueHash(0x0000000000000057ffffffffffffffa800000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[d8eb50c3ba501a828697da535d6a14665a12b9ea5e1214f37a738a27d7fe6ba1]))
172: Parent
173: Push(KVValueHash(0x0000000000000058ffffffffffffffa700000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[dada8ecba303e160deec29e7215e21572bdccc38647fd5550b0ac47af7ac52f4]))
174: Push(KVValueHash(0x0000000000000059ffffffffffffffa600000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[cbeaf7428892a5ca259437414d3eda7fd4de6ef55eb3ef0195f37fddd695e893]))
175: Parent
176: Push(KVValueHash(0x000000000000005affffffffffffffa500000000000000000000000000000000, SumTree(73656e744174, 1000), HASH[8320cbd66a3e07a381a989e260edf5b42579ea5951384c2280f037e698e95ff9]))
177: Child
178: Push(KVValueHash(0x000000000000005bffffffffffffffa400000000000000000000000000000000, SumTree(73656e744174, 2000), HASH[44342c04d0b5a3ede78cd4ef036bb3b48b8f9e4da2a502e0b5e36874cab62588]))
179: Parent
180: Push(KVValueHash(0x000000000000005cffffffffffffffa300000000000000000000000000000000, SumTree(73656e744174, 3000), HASH[547f96a911cb5f0657cad7e5ce0487fa1a8870b232c52ff3efac108e6978f9e9]))
181: Push(KVValueHash(0x000000000000005dffffffffffffffa200000000000000000000000000000000, SumTree(73656e744174, 4000), HASH[f8342da41347c35f4a966a1b9c4bbb684af653cd18c3bb200e630f2731266c06]))
182: Parent
183: Push(KVValueHash(0x000000000000005effffffffffffffa100000000000000000000000000000000, SumTree(73656e744174, 5000), HASH[18b8d60c015a057419a6405275cd02d490ff7cab627a653ec603a97757a5f49f]))
184: Child
185: Child
186: Push(KVValueHash(0x000000000000005fffffffffffffffa000000000000000000000000000000000, SumTree(73656e744174, 6000), HASH[d5e7eb06c7bd04852d0b25bd5df0039f3ef20d34711ef121f00cce3a31349dd5]))
187: Parent
188: Push(KVValueHash(0x0000000000000060ffffffffffffff9f00000000000000000000000000000000, SumTree(73656e744174, 7000), HASH[2462c14975dad4a210d556d0e579529e8a756b3b107cbbc75fe344b2a03b357b]))
189: Push(KVValueHash(0x0000000000000061ffffffffffffff9e00000000000000000000000000000000, SumTree(73656e744174, 8000), HASH[fe0d935c37c343340bc5f32dcedd4d3eb7c7cce20604065ff63bced0a951de75]))
190: Parent
191: Push(KVValueHash(0x0000000000000062ffffffffffffff9d00000000000000000000000000000000, SumTree(73656e744174, 9000), HASH[bbc5e27f1fdbba68e332d6cc50790017f29b2878b21e49b401701deee9bb2948]))
192: Push(KVValueHash(0x0000000000000063ffffffffffffff9c00000000000000000000000000000000, SumTree(73656e744174, 10000), HASH[15ca1dc935c384f6fd904a3de0643e4af7b8c38fe3946379c182a54714786ac1]))
193: Child
194: Child
195: Child
196: Child
197: Child
198: Child)
lower_layers: {
0x0000000000000000ffffffffffffffff00000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[f3cd8b8f77e6c9473ba47bb5650e481537027a91bcea62c04e70d9c658b7ce87]))
1: Push(KVValueHash(sentAt, NotSummed(ProvableSumTree(800000000000c79c, 1000)), HASH[92dc35643411662e66f2e8dc6b591cf7eed54f6dc4aaf69219c369c74188e8a7]))
2: Parent)
lower_layers: {
sentAt => {
LayerProof {
proof: Merk(
0: Push(HashWithSum(kv_hash=HASH[622ec7c875ff1016840e660f388f97bc899c32726d6f18a5966da84ec63ea355], left=HASH[ff984fa6039c70cdb6a896d7d69ecf3271dc0b103ee44d5d8301365c110b23fd], right=HASH[c90f5696ed4eb077cc0b1cdb904b2af2a75fd9b07f60345925cdddf7f6b38f2a], sum=255))
1: Push(KVDigestSum(0x800000000000639c, HASH[a9f7ac3993ee6e200d0f3d40697e009ce0bfe278009f77e3e82215ad7475c6e3], 511))
2: Parent
3: Push(HashWithSum(kv_hash=HASH[18cab59d8df3ddadeb0054a44540c64a916e022e3af157b38c175c7f9908b233], left=HASH[3f8f3e6a71375ccfae2714e6d9468ca8df8c914bcae7a5e67f2d167ccf82c6c6], right=HASH[fd4793282e7beeae5ffcc9ed6adf62ea213bbede0fc061ba5f5534303218a92f], sum=127))
4: Push(KVDigestSum(0x800000000000959c, HASH[94924e9e1cb6b1851911f3290f93a745bc1e9751044e5e462e702eb9c48f6572], 255))
5: Parent
6: Push(HashWithSum(kv_hash=HASH[f03986bdca555701a548ebb6483b177814570cc7e8638bb312b7c6939504eb18], left=HASH[f171da996938e7ea3d5840ea2b3448e40c701c3d941e2e81ea45450055f6e5c8], right=HASH[8eccc3e0d53a1d8dabf858f17886a204728adff59e19346d61d227fafa86e08a], sum=63))
7: Push(KVDigestSum(0x800000000000ae9c, HASH[a606c7eec7545c1124b0bccbcb29dedfdcba820f05446806c4eaf5c8106a667d], 127))
8: Parent
9: Push(HashWithSum(kv_hash=HASH[082aa2d5a4f27332961db4882ea673e1a9dc7c163eaf56c1ad0c2f876f8369c3], left=HASH[31d49f9b0d0c858fbe65650b5c4ffbfc86def44778a62253a2a81f92e60363dc], right=HASH[d52fc80e4e04177e9cab24953d10d870b73f5acd76f81076c784932b92f5cc4a], sum=31))
10: Push(KVDigestSum(0x800000000000bb1c, HASH[01f81f07f05bdab399b7f9b3583316cae8227ee8ef6f38f1292510e3fcdf8759], 63))
11: Parent
12: Push(HashWithSum(kv_hash=HASH[1ed335fe242c26d4530101e609a6d5a52a32dddfdd9fed225de10a7039d19238], left=HASH[373889c6db5863abb87ef61b3ee479d9ebf13922571c2f7e24a3e3b1eac7fa16], right=HASH[e126395fd037d9f94216301144e57dfa044de771c999597f6fcf5e604c2ce2f1], sum=15))
13: Push(KVDigestSum(0x800000000000c15c, HASH[7eb10dc2f4d81ce6c5df234cc54b19fd87b8b861fc5151dd0c19619171c7ce83], 31))
14: Parent
15: Push(HashWithSum(kv_hash=HASH[c59ef9246a6a7225fb2ef0558bf280f8df39ed99a17bb40812c639a18fd3af14], left=HASH[96e89928ae0de69be602723850619c0ebd3542da48ab32b8a1db5bc2e8603192], right=HASH[0cb2ba854b100015b09de5984a4103976d2d3b46900e9ddee76cf600234b5659], sum=3))
16: Push(KVDigestSum(0x800000000000c2ec, HASH[c68b52e5b8036d991946274166ef630ef19aeafd0f38d2388e2f7423cda1045f], 7))
17: Parent
18: Push(KVDigestSum(0x800000000000c350, HASH[b22d3790d948924dc6311518f2872b53c0590d3cc497d7a653f1ce7b9923e499], 1))
19: Push(KVDigestSum(0x800000000000c3b4, HASH[40b4e8cfba33cbdb2e6643b00f062c68ffd9786c156af78ba712b91a03af3ec0], 3))
20: Parent
21: Push(HashWithSum(kv_hash=HASH[38a6c1c68d6436d6059f514f0b532ffbb4d62788a72c283dea1f14e204399379], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], sum=1))
22: Child
23: Child
24: Push(KVDigestSum(0x800000000000c47c, HASH[8c891440a3fdef244cefe3a928529f037fed7f6347509fc814f82430366f39df], 15))
25: Parent
26: Push(HashWithSum(kv_hash=HASH[b81a70230e6c91bdbcb631bed29cbb79815f3117da15b29f771a7f833d42090c], left=HASH[24c61905d7361140b4ca79506c2892a081daab4b471171b216f59968fd8855b5], right=HASH[8f4b5b0b1135c8b4c87597f33a968b8404bda5d2d88ee7d2b613d90d1795fb4a], sum=7))
27: Child
28: Child
29: Child
30: Child
31: Child
32: Child
33: Push(KVDigestSum(0x800000000000c79c, HASH[387d1806a39adf033f4e7a7888970a627bfd803850eee18e8a277f45893f5e61], 1000))
34: Parent
35: Push(HashWithSum(kv_hash=HASH[a7d78a4571fc667589d9194ff86f9ebb7dc5c7b1c86241ee0fd1231fa877e47e], left=HASH[1bd878de2c7e1752d4e5521d36a8d3ab2854f652d6cf4be41cb1cd48cbb2cfe3], right=HASH[42073ef99149a312e57e2c165578c2ed4f62e9325f400102da26d3077916ca21], sum=488))
36: Child)
}
}
}
}
}
// ↓↓↓ Q9 abbreviation ↓↓↓
// 99 more recipient buckets follow here
// (recipient_001 .. recipient_099), each a
// LayerProof with the same structural shape as
// recipient_000 above: a single-key continuation
// Tree → NotSummed(ProvableSumTree) → AggregateSumOnRange
// collapse. Per-bucket aggregate sums follow
// recipient_n.sum = 500 × ((n % 10) + 1)
// with recipient_000.sum = 499 (the boundary
// row 50000 is excluded by the half-open `>` range).
// ↑↑↑ Q9 abbreviation ↑↑↑
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Each LayerProof is one GroveDB tree's merk proof. Q9 is the largest proof in the chapter and structurally distinct from every other query: the AST tree fans out at layer 6 (the byRecipient subtree) where the carrier outer walk reveals all 100 In-key terminators inline (each as a KVValueHash(recipient_NNN, SumTree(73656e744174, sum_NNN)) op), and each of those 100 keys' lower_layers entry then carries its own inner AggregateSumOnRange descent through that recipient's byRecipientTime continuation — a per-bucket Q8-shaped boundary walk producing one aggregate i64 per recipient. The total LayerProof count is 5 fixed (root → @ → contract → 0x01 → tip) + 1 outer byRecipient layer that lists all 100 In keys + 100 × 2 per-bucket layers (one for the recipient-continuation single-key tree, one for the byRecipientTime sentAt range collapse). The verifier (via GroveDb::verify_aggregate_sum_query_per_key) walks each branch, collapses its inner range to a single sum, and returns Vec<(in_key, i64)> with 100 entries:
(recipient_000, 499) # 499 tips: rows 50100, 50200, …, 99900, each amount=1
(recipient_001, 1000) # 500 × 2
(recipient_002, 1500) # 500 × 3
…
(recipient_009, 5000) # 500 × 10
(recipient_010, 500) # 500 × 1 (cycle restarts)
…
(recipient_099, 5000) # 500 × 10
Note recipient_000 gets 499 not 500: sentAt > 50000 excludes the boundary row (sentAt = 50000) which belongs to recipient_000 (row 50000 → recipient_(50000 % 100) = recipient_0), so its 500-row window loses one entry. Every other recipient is unaffected.
Verified root hash 95aa7470…71b6.
Diagram: per-layer merk-tree structure
Q9's structure is the chapter's only non-linear descent: instead of a single chain, layer 7 fans out into 100 parallel branches (one per resolved In-key), each running its own inner-range collapse. The diagram below shows one branch in detail and indicates the × 100 fan-out at the outer layer.
flowchart TB
subgraph L6["Layer 6 — byRecipient merk-tree (CARRIER OUTER LAYER)"]
direction TB
L6_targets["100 KVValueHash outer-key terminators<br/>(recipient_000 … recipient_099)<br/>each value=<b>SumTree(73656e744174, per-recipient-sum)</b><br/>but here the values are mid-descent pointers, NOT the<br/>carrier-aggregate's final output — each one's<br/>lower_layers fans out to a per-bucket range walk."]:::queried
L6_boundary["~22 merk-boundary ops shared across all 100 descents:<br/>since every byRecipient entry is revealed,<br/>only the merk-root spine stays opaque"]:::sibling
L6_targets --> L6_boundary
end
subgraph L7a["Layer 7 — recipient_NNN continuation (× 100 parallel, one per bucket)"]
direction TB
L7a_q["<b>sentAt</b><br/>kv_hash=HASH[…]<br/>value: NotSummed(ProvableSumTree(…, 1000))<br/>(per-recipient continuation, single-key tree)"]:::queried
L7a_sib["1 Hash opaque-sibling commit<br/>(per the per-bucket continuation merk-tree)"]:::sibling
L7a_q --> L7a_sib
end
subgraph L8a["Layer 8 — byRecipientTime sentAt range collapse (× 100 parallel, one per bucket)"]
direction TB
L8a_collapse["<b>AggregateSumOnRange(sentAt > 50000)</b><br/>verified sum_NNN per recipient_NNN<br/>(499 for recipient_000;<br/>500 × ((NNN % 10) + 1) for the rest)"]:::target
L8a_boundary["Range-boundary commitments (~36 ops per bucket):<br/>same HashWithSum + KVDigestSum shape as Q8's layer 8.<br/>This per-bucket cost × 100 buckets is what dominates Q9's<br/>169 064 B vs the non-carrier shapes (Q5 12 064 B, Q6 9 784 B)."]:::pst
L8a_collapse --> L8a_boundary
end
L6_targets -. "100 fan-out: each value=SumTree(merk_root[per-bucket]) → L7a" .-> L7a_q
L7a_q -. "value=NotSummed(ProvableSumTree(merk_root[byRT.sentAt])) → L8a" .-> L8a_collapse
fanout["× 100 parallel descents<br/>(one per resolved In key)"]:::note
L7a_q -.-> fanout
classDef queried fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef sibling fill:#6e7681,color:#fff,stroke:#6e7681;
classDef pst fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef target fill:#39c5cf,color:#0d1117,stroke:#39c5cf,stroke-width:3px;
classDef note fill:#21262d,color:#c9d1d9,stroke:#fb8500,stroke-width:2px,stroke-dasharray: 6 4;
This is the structural payoff of carrier composition: instead of 100 independent Q8-shaped proofs each running its own full root→@→contract→…→ recipient_NNN → sentAt descent (with the upper 5 fixed layers re-paid for each branch), Q9 amortizes the upper-5 layers across all 100 buckets and only pays the per-bucket cost (1 continuation layer + 1 range-collapse layer ≈ 1 690 bytes) per outer key. The 64% byte savings vs the N-independent baseline come almost entirely from that shared prefix amortization.
The carrier composition is the only way to get per-bucket range sums in a single proof. The alternative — issuing N independent AggregateSumOnRange proofs, one per In value — would take N × 2 657 bytes ≈ 265 700 bytes for k=100, plus N round-trips. Carrier collapses that to one proof, one round-trip, and ≈ 64% of the byte cost.
The two carrier-aggregate gates worth knowing:
SizedQuery::limitcaps the outer walk (herelimit = 100accommodates all distinct recipients; for an open-ended outerRangeclause it bounds how many In-branches the proof commits). The verifier rebuilds the samelimitbyte-for-byte; mismatched limits break the merk-root recomputation.SizedQuery::offsetis rejected for carrier-aggregate (would change which(outer_key, sum)pairs end up in the proof; the use case isn't designed yet). Mirrors the count-side carrier-ACOR contract.
The PCPS variant — AggregateCountAndSumOnRange on a carrier — exists too; same primitive, but returns (outer_key, u64 count, i64 sum) triples for indexes that opt into both rangeCountable: true and rangeSummable: true. Drive-side support is wired (DriveDocumentSumQuery::verify_carrier_aggregate_count_and_sum_proof); a worked PCPS-carrier example will live in a separate combined-feature chapter alongside its own contract.
Range Modes — Distinct Variant
Every range query above runs in aggregate mode (return_distinct_sums_in_range = false, the default). Setting it to true returns one SumEntry per distinct property value in the range — the histogram shape.
For Query 7 with return_distinct_sums_in_range = true:
SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 50001), sum: 2 }
SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 50002), sum: 3 }
...
SumEntry { in_key: None, key: serialize_value_for_key("sentAt", 99999), sum: 10 }
49 999 entries (one per distinct sentAt in the range). The prove path's proof size in this mode is O(distinct values matched), not O(log n) — for tip-jar's per-row-distinct sentAt fixture that's a big difference. Pick the aggregate mode when you want one number, distinct when you want a histogram. Same trade-off as count's Range Queries on the Prove Path.
References With Sum Item — Where They Live
Every non-primary-key sum lookup above lands on a value-tree path like:
[..., "<index_property>", "<index_value>", 0]
where [0] (key zero) holds a SumTree of references — one per document — and the parent value-tree's sum is the aggregation of those references' sum-item contributions. The reference primitive grovedb exposes is Element::ReferenceWithSumItem(reference_path, sum_value, flags) (grovedb PR 670): a single element that dereferences to the document body in primary storage and carries an i64 (the amount value at insert time). When merk computes node hashes up the parent SumTree, the reference's sum-item is what propagates.
Two element types for two roles, kept distinct:
- Primary storage at
[doctype, 0, doc_id]usesElement::ItemWithSumItem(serialized_doc, sum_value, flags)when the doctype declaresdocumentsSummable: "amount"— the document body lives there inline AND contributes to the primary-key SumTree's running aggregate. This is what makes thedocumentsSummablefast path (O(log n) total-sum proof, no index needed) actually work: the primary-key SumTree's root carries the real sum because every leaf is sum-bearing. - Index references at
[index_path, value, 0, doc_id]useElement::ReferenceWithSumItem— a true reference (so document iteration via index walks still dereferences to the body in primary storage, exactly likeElement::Referencedoes on the count side) AND a sum contribution that propagates to ancestor sum trees.
That's the load-bearing primitive for the design: without it, every non-primary-key sum index would need to either duplicate the document under the index (storage blowup) or refuse to participate in sum aggregation (no non-primary-key sums at all). With it, the existing index storage shape extends from "reference under [0]" to "reference with sum-item under [0]" — same descent, same proof verification, plus the running-sum propagation.
The reference's stored sum-item is fixed at insert time. On delete, Drive reads it back out of the reference (rather than re-reading the source document) and subtracts it from every ancestor sum tree. That's why the named summable property must be in required: a missing amount at insert time would leave the reference with no sum contribution, and a later delete would underflow the ancestor sums by zero — a quiet corruption that's never worth allowing.
At-a-Glance Comparison
| Query | Index used | Element shape at terminator | Returned variant | Proof primitive |
|---|---|---|---|---|
| 1 — Total | (doctype primary-key) | SumTree at tip/[0] | aggregate_sum | merk path |
| 2 — Equal on byRecipient | byRecipient | SumTree at recipient/recipient_050 | aggregate_sum | merk path |
| 3 — Equal on bySentAt | bySentAt | SumTree at sentAt/serialize(50000) | aggregate_sum | merk path |
| 4 — Compound Equal | byRecipientTime | SumTree at recipient/recipient_050/sentAt/serialize(50000) | aggregate_sum | merk path |
| 5 — In on byRecipient | byRecipient | k × SumTrees | entries | k × merk path |
| 6 — In on bySentAt | bySentAt | k × SumTrees | entries | k × merk path |
| 7 — Range on bySentAt | bySentAt | (collapsed boundary) | aggregate_sum | AggregateSumOnRange |
| 8 — Compound + Range | byRecipientTime | (collapsed boundary, prefix descent) | aggregate_sum | AggregateSumOnRange |
| 9 — Carrier (In + Range) | byRecipientTime | k × (collapsed boundaries under outer Keys) | per-key entries | verify_aggregate_sum_query_per_key (carrier composition) |
The split is structurally the same as count's — single value-tree lookups for point queries (1–6), AggregateSumOnRange for range (7–8), and carrier composition for per-bucket range sums (9) — with SumTree/ProvableSumTree and aggregate_sum/SumEntry slotted in where count had CountTree/ProvableCountTree and aggregate_count/CountEntry. The Q9 carrier specifically wraps the leaf AggregateSumOnRange primitive once per outer In branch and stitches the per-bucket sums into a single verifiable response — saving N round-trips and ~36% of the bytes versus issuing N independent range-sum proofs.
What's Next
The full SUM surface is wired end-to-end: primary-key total (Query 1), point-lookup, In-fan-out, AggregateSumOnRange on both top-level and compound indexes, and the carrier-aggregate primitive. All nine queries verify against the shared root hash 95aa7470…71b6.
A Sum Index Group By Examples sibling chapter is queued mirroring the count GROUP BY chapter — the byRecipient index supports both per-recipient and group-by-recipient semantics, and the bench's report_group_by_matrix already publishes the full matrix (visible inline in bench -- --test output under [matrix]).
Average Index Examples
This chapter walks through a representative contract and shows how average queries work on Drive. Every example uses the same grade document type on the grades contract at packages/rs-drive/tests/supporting_files/contract/grades/grades-contract.json.
The chapter assumes you've read Document Count Trees and Document Sum Trees — averages are built directly on top of both, so understanding count + sum trees individually is the prerequisite. Here we take that machinery as given and look at the queries that need both.
Status: the
document_average_worst_casebench lands the reproducible numbers below — same convention as the Count and Sum chapters. All proof sizes are measured against a 31 620-grade fixture; verified(count, sum)values are the actual numbers the bench's matrix reports. The full surface — primary-key global average, point lookups, range aggregates, and both carrier variants — is wired through to grovedb PR #670's verifiers end-to-end.
Why Averages Need a New Primitive
An average is sum / count. To prove an average against a single root-hash commit, you need both numbers from the same set in one proof — otherwise the client can't verify that the divisor and dividend describe the same documents.
Three options exist:
- Two separate proofs — one sum proof, one count proof, client divides. Burns 2× the proof bytes + 2× the round-trips. The bigger problem: the two proofs commit independently, so the client has to verify both root-hashes match (or the server could splice mismatched results).
- Materialize-and-divide — server walks the document set, sums + counts itself, returns one rational number. No
O(log n)win, no cryptographic commit to the underlying count or sum; the client just trusts the server's reported quotient. - A single dual-axis primitive — one proof commits both metrics from one merk traversal. The verifier returns
(count, sum); the client divides. This is what this chapter is about.
The grovedb primitive is AggregateCountAndSumOnRange (added in grovedb PR #670) and its carrier extension verify_aggregate_count_and_sum_query_per_key (PCPS-carrier proofs for group-by averages). Both require the terminator tree to be a ProvableCountProvableSumTree (PCPS) — a single merk tree where each internal node carries both a count_value and a sum_value, committed to the parent node's hash. Lighter sum-bearing trees (SumTree, ProvableSumTree, CountSumTree, ProvableCountSumTree) all reject the combined primitive at the merk gate — you need both axes per-node for the proof's single traversal to commit both metrics.
The grades contract below opts two indexes into PCPS (byClassSemester and byStudentSemester); the other three are simpler CountSumTree shapes serving point-lookup averages.
The Grades Contract
The grade document type carries one grade for one student in one class during one semester. Five properties (student, class, semester, score, instructor), opts into global totals via documentsCountable: true + documentsSummable: "score", and declares five indexes spanning the average-query surface:
{
"type": "object",
"documentsMutable": false,
"canBeDeleted": false,
"documentsCountable": true,
"documentsSummable": "score",
"properties": {
"student": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"position": 0, "contentMediaType": "application/x.dash.dpp.identifier" },
"class": { "type": "string", "minLength": 1, "maxLength": 32, "position": 1 },
"semester": { "type": "integer", "minimum": 20000, "maximum": 99999, "position": 2 },
"score": { "type": "integer", "minimum": 0, "maximum": 100, "position": 3 },
"instructor": { "type": "array", "byteArray": true, "minItems": 32, "maxItems": 32,
"position": 4, "contentMediaType": "application/x.dash.dpp.identifier" }
},
"required": ["student", "class", "semester", "score", "instructor"],
"indices": [
{ "name": "byClass",
"properties": [{ "class": "asc" }],
"countable": "countable", "summable": "score" },
{ "name": "byStudent",
"properties": [{ "student": "asc" }],
"countable": "countable", "summable": "score" },
{ "name": "bySemester",
"properties": [{ "semester": "asc" }],
"countable": "countable", "summable": "score" },
{ "name": "byClassSemester",
"properties": [{ "class": "asc" }, { "semester": "asc" }],
"countable": "countableAllowingOffset", "summable": "score",
"rangeCountable": true, "rangeSummable": true },
{ "name": "byStudentSemester",
"properties": [{ "student": "asc" }, { "semester": "asc" }],
"countable": "countableAllowingOffset", "summable": "score",
"rangeCountable": true, "rangeSummable": true }
],
"additionalProperties": false
}
Five things to internalize before reading the queries:
documentsCountable: true+documentsSummable: "score"at the document-type level upgrades the doctype's primary-key subtree (atgrade/[0]) fromNormalTreetoCountSumTree. The unfiltered global average is one read against this element's(count_value, sum_value)pair, no index walk.byClass/byStudent/bySemesterarecountable: countable+summable: "score"(no range flags). Each per-key value-tree (one per class / student / semester) is aCountSumTreecarrying both metrics at one merk lookup — point-lookup averages get the same shortcut count proofs and sum proofs do.byClassSemesterandbyStudentSemesterset both range flags (rangeCountable: trueANDrangeSummable: true). Thesemestercontinuation under each (class | student) value-tree is aProvableCountProvableSumTree(PCPS), the structurally-richest tree variant — every internal merk node carries both a per-node count and a per-node sum. This is whatAggregateCountAndSumOnRangewalks for "average for class X in semester range [a..b]" style queries.- Every
summableindex here is alsocountable. There's nosummable-only index in this contract — averages need both axes, so a sum-only index would be unreachable from the average surface. (Pure-sum surfaces are covered by the tip-jar contract in the previous chapter; the grades contract is deliberately the dual-axis counterpart.) countableAllowingOffseton the PCPS indexes —rangeCountable: trueimplies a countable index (an omittedcountableis promoted to"countable"; per the rule documented inIndex::range_countable), so an explicitcountableAllowingOffsetis what picks the richer tier. The offset-allowing tier upgrades the property-name tree to aProvableCountTreeat minimum; combined withsummable: "score"andrangeSummable: truethe dispatcher resolves it to PCPS.
The bench populates 50 000 grades under a deterministic, realistic-data-shaped schedule: 500 students × 10 classes × 10 semesters = 50 000 grade documents. The score model layers three deterministic axes:
- Per-class baseline + spread (see
class_profilein the bench) — hard classes get low means and wide spreads; easy classes cluster near 85–90 with narrow spreads. The 10 classes have semantic names matching the chapter's references:
| Class | Baseline mean | Spread | Profile |
|---|---|---|---|
PHYS101 | 60 | 12 | hard physics |
CHEM101 | 65 | 10 | moderate chem |
CALC201 | 58 | 13 | hardest math |
ENGL101 | 85 | 5 | easy english |
HIST101 | 78 | 8 | moderate history |
BIOL101 | 72 | 9 | moderate bio |
ARTS101 | 88 | 4 | easiest art |
COMP101 | 75 | 9 | moderate CS |
MUSC101 | 82 | 6 | easy music |
SOCI101 | 80 | 6 | easy social |
- Per-student skill — a deterministic FNV-1a hash of the student index, scaled to ≈ N(0, σ²) via central-limit-theorem (sum of three uniforms averaged), centered slightly below 0 to model "most students are average, a few are excellent, a few struggle." Spans roughly
[-25, +15]. - Per-grade noise — deterministic ±5 variation per
(student, class, semester)so even one(student, class)pair has nontrivial semester-to-semester variation.
A skill score is amplified by class spread — skill × spread / 8 — so a +10-skill student in PHYS101 (spread=12) gains +15 over baseline, while the same student in ARTS101 (spread=4) only gains +5. This produces the realistic spread real transcripts exhibit: a strong student stands out more in hard classes, struggling students fall further behind in hard classes, easy classes flatten the curve.
Realistic enrollment. Not every student takes every class — that's not how transcripts work. The bench walks all 500 × 10 × 10 = 50 000 possible (student, class, semester) triples but only emits a grade when is_enrolled returns true, using a deterministic per-class popularity table:
| Class | Popularity | Profile |
|---|---|---|
ENGL101 | 100% | required for everyone every semester |
ARTS101 | 90% | very popular elective |
MUSC101 | 85% | popular elective |
HIST101 | 70% | common humanities |
SOCI101 | 70% | common social science |
BIOL101 | 60% | moderately popular |
COMP101 | 55% | moderately popular |
CHEM101 | 45% | moderately popular |
PHYS101 | 30% | hard physics, smaller cohort |
CALC201 | 25% | hardest math, smallest cohort |
The total comes out to 31 620 actual grade documents (≈ 63% of the 50 000 possible triples — see the popularity table; the per-class actual rates match the documented popularities within ±0.5 percentage points), with the per-class enrollment counts ranging from 970 (CALC201 across 5 semesters) to 2 500 (ENGL101 — required, so every student × every semester). The expected per-student grade count is ≈ 6.3 classes per semester × 10 semesters = ≈ 63 grades per student.
Headline numbers from the bench's fixture (all verified end-to-end against the shared root hash 8b15f732af8f…ffc7):
- Total
countacross all grades: 31 620 (not 50 000 — the enrollment filter removes ~29%). - Per-class average spans from ≈ 53 (CALC201, hardest math) to ≈ 87 (ARTS101, easiest art) — a 33-point realistic spread.
- Per-class total count varies from 970 (CALC201) to 5 000 (ENGL101) across all 10 semesters — the enrollment differential surfaces in every per-class average proof.
- Per cohort (one class in one semester): count varies, typically 125–500 depending on the class's popularity.
- Per student (single student, all classes, all semesters): count varies by their enrolled mix, typically 55–70 grades.
student_050for instance verifies at count = 72, sum = 4 834 (avg = 67.14 — this student happens to have an above-average skill score).
GroveDB Layout
The contract above produces this storage shape. Tree elements are drawn as subgraphs; children inside each tree are merk-tree nodes. The doctype root and the per-property name subtrees are separate Element trees nested under the contract-documents prefix.
Diagram conventions: green nodes carry both a count_value and a sum_value (CountSumTree); yellow nodes carry both per node (PCPS); gray are regular subtrees; dashed boxes highlight wrapper elements (NotCountedOrSummed) that opt out of both axes from their parent's aggregation.
flowchart TB
TD["@/contract_id/0x01/grade"]:::tree
TD --> PK["[0]: CountSumTree count=31620 sum=2392808<br/>(documentsCountable + documentsSummable primary key)"]:::csnode
TD --> CL["class: NormalTree<br/>(byClass property-name)"]:::node
TD --> ST["student: NormalTree<br/>(byStudent property-name)"]:::node
TD --> SM["semester: NormalTree<br/>(bySemester property-name)"]:::node
CL --> CL_M["class_MATH101: CountSumTree count=1000 sum~50000"]:::csnode
CL --> CL_P["... 9 more class value-trees<br/>(each CountSumTree count=1000)"]:::csnode
CL_M --> CL_M_0["[0]: CountSumTree count=1000 sum~50000<br/>(byClass refs — one per grade)"]:::csnode
CL_M --> CL_M_S["semester: NotCountedOrSummed(PCPS)<br/>(byClassSemester continuation, contributes 0 + 0)"]:::nonboth
CL_M_S --> CL_M_S_241["semester_20241: CountSumTree count=100 sum~5000<br/>(byClassSemester cohort terminator)"]:::csnode
CL_M_S --> CL_M_S_more["... 9 more semester buckets"]:::csnode
ST --> ST_X["student_050: CountSumTree count=100 sum~5000"]:::csnode
ST --> ST_more["... 99 more student value-trees"]:::csnode
ST_X --> ST_X_0["[0]: CountSumTree count=100 sum~5000<br/>(byStudent refs)"]:::csnode
ST_X --> ST_X_S["semester: NotCountedOrSummed(PCPS)<br/>(byStudentSemester continuation, contributes 0 + 0)"]:::nonboth
SM --> SM_241["semester_20241: CountSumTree count=1000 sum~50000<br/>(bySemester all-grades terminator)"]:::csnode
SM --> SM_more["... 9 more semester buckets"]:::csnode
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef node fill:#6e7681,color:#fff,stroke:#6e7681;
classDef csnode fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
classDef pcpsnode fill:#d29922,color:#0d1117,stroke:#d29922,stroke-width:2px;
classDef nonboth fill:#21262d,color:#c9d1d9,stroke:#fb8500,stroke-width:2px,stroke-dasharray: 6 4;
Three layout facts to internalize before reading the queries:
class_MATH101is aCountSumTreewithcount = 1000andsum ≈ 50 000. That's true becausebyClassdeclares bothcountable: countableandsummable: "score". The averagesum / count ≈ 50is one merk lookup. Thesemestercontinuation that branches off this value tree isNotCountedOrSummed-wrapped so the parent's(count, sum)equals exactly the contribution from the 1 000 refs in[0]— without the wrapper, the compoundbyClassSemestercontinuation would double-count and double-sum into the parent.- The
semestercontinuation under each class value-tree is a PCPS (ProvableCountProvableSumTree) wrapped inNotCountedOrSummed. Inside the wrapper, every internal merk node carries both a per-node count and a per-node sum — which is what makesAggregateCountAndSumOnRangea single-pass primitive. Wrapping it asNotCountedOrSummedis the load-bearing trick: the wrapper is invisible to the inner primitive (the merk walker descends into the PCPS unchanged) but opaque to the parent's aggregation (contributes 0 to both axes), keepingbyClass's class-level(count, sum)clean. bySemester's value trees are alsoCountSumTree(count + sum per semester across all students and classes). One semester's school-wide average is one merk lookup; thesemesterindex doesn't have abyClassSemester-style continuation because there's no compound(semester, class)index in this contract — adding one would slot a parallel PCPS continuation here.
How To Read The Proofs
Every example below has four sections:
- Path query — the spec the prover hands GroveDB.
pathis the list of subtree segments to descend through;query itemsis what to select once at the bottom;subquery items(when present) descends one more layer. - Verified result — what
GroveDb::verify_queryreturns for point lookups, orGroveDb::verify_aggregate_count_and_sum_query/verify_aggregate_count_and_sum_query_per_keyreturns for the range and carrier primitives. For every query the return shape is(count, sum)(orVec<(key, count, sum)>for carrier) — the client divides for the average. The chapter showsavg = sum / countderived inline. - Proof display — the proof bytes decoded via
bincodeinto the structuredGroveDBProofAST and rendered through itsDisplayimpl, same convention as the count and sum chapters. Wrapped in a collapsible<details>block per example with a link to the visualizer. - Diagram — per-layer merk-tree references back to the GroveDB Layout diagram above, with
csnode(green) forCountSumTreeterminators andpcpsnode(yellow) forProvableCountProvableSumTreeterminators where the dual(count, sum)per-node fields are visible.
All proof-size numbers and avg-times below come from the 10 000-row bench run; the methodology block under the queries table covers how to reproduce them.
Queries in this Chapter
| # | Query | Filter / Group-by | Complexity | Avg time | Proof size |
|---|---|---|---|---|---|
| 1 | Unfiltered Global Average | (none — total at doctype level) | O(1) | 25.3 µs | 622 B |
| 2 | Average for One Class (byClass) | class == "PHYS101" | O(log C) | 32.1 µs | 871 B |
| 3 | Student GPA (byStudent) | student == student_050 | O(log S) | 42.0 µs | 1 227 B |
| 4 | One Cohort (byClassSemester point) | class == "PHYS101" AND semester == 20204 | O(log C + log T') | 51.0 µs | 1 304 B |
| 5 | Class Trend (AggregateCountAndSumOnRange) | class == "PHYS101" AND semester > 20204 | O(log C + log T') | 49.7 µs | 1 539 B |
| 6 | Per-Student Averages for One Semester (carrier) | student IN [0..9] AND semester == 20204 (group_by [student]) | O(k · log S + log T') | 304.4 µs | 6 581 B (k=10) |
| 7 | Per-Class Trends (PCPS carrier) | class IN [10 classes] AND semester > 20204 (group_by [class, semester]) | O(k · (log C + log T')) | 273.8 µs | 8 220 B (k=10) |
Timing methodology: median of 5 iterations after one warmup, measured against the bench's 31 620-grade fixture on a warmed rocksdb cache (31 620 actual grades from 50 000 possible triples, filtered by per-class enrollment popularity). The figures reflect the drive-layer execute_* calls (path query build + grovedb proof generation, no network or tenderdash signature compose). Reproduce with DASH_PLATFORM_AVERAGE_BENCH_REBUILD=1 cargo bench -p drive --bench document_average_worst_case -- --test; grep µs from stderr.
Fixture-narrative cross-references: Q2/Q4/Q5/Q7 use the class name "PHYS101" (the first of 10 semantically-named classes — see the contract-narrative table above). Q3/Q6 reference student_050 (the midpoint student id). Q4/Q5/Q6/Q7 all use semester floor 20204 (the midpoint of the 10-semester range 20200..20209), so the range semester > 20204 matches exactly 5 semesters per class. The original chapter draft used "MATH101" / semester == 20241 / semester > 20210 placeholders; the bench substitutes deterministic-id names + arithmetic-midpoint values for verifiability.
Complexity variables. C = distinct classes (= 10 in the fixture); S = distinct students (= 100); T = distinct semesters (= 10); T' = distinct semesters per class or per student in the byClassSemester / byStudentSemester continuation (= 10); k = number of values in the IN clause. Notably absent: the total document count N (= 10 000 here). Average proofs read pre-committed count_value + sum_value from CountSumTree / PCPS merk roots — they never enumerate the underlying documents, so proof generation cost is polylog(distinct index values), independent of N. Same big-O story as count and sum individually; PCPS just commits both metrics per node, adding a small constant factor of per-node hash work vs. count-only / sum-only.
The first four queries (Q1–Q4) get their (count, sum) from a single CountSumTree element at the descent's terminator — same proof shape as a count point lookup, just with an extra 8 bytes per merk node for the sum field. Q5 uses AggregateCountAndSumOnRange against the byClassSemester PCPS continuation — one proof, single root-hash commit, returns (root_hash, count, sum). Q6 and Q7 are the carrier variants — outer In walk + inner per-bucket aggregation, returning Vec<(key, count, sum)>. Q7 specifically uses the PCPS carrier (verify_aggregate_count_and_sum_query_per_key); Q6 uses a CountSumTree-carrier on a point-inner subquery (since the per-student-per-semester cohort is a point, not a range — semester == 20241 doesn't need PCPS).
Query 1 — Unfiltered Global Average
select = AVG(score)
where = (empty)
prove = true
Path query (primary-key CountSumTree fast path; no index walk needed):
path: ["@", contract_id, 0x01, "grade"]
query items: [Key(0x00)]
Verified result:
path: ["@", contract_id, 0x01, "grade"]
key: 0x00
element: CountSumTree { count_value_or_default: 31620, sum_value_or_default: 2392808 }
average: 2392808 / 31620 = 75.6739…
Proof size: 622 bytes. Avg time: 25.3 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (5 layers — primary-key fast path) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
lower_layers: {
0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
lower_layers: {
grade => {
LayerProof {
proof: Merk(
0: Push(KVValueHashFeatureTypeWithChildHash(0x00, Tree(00000000000067cfffffffffffff983000000000000000000000000000000000), HASH[75d7cea7fe7cf4c112fe2d080d01417ade8169dffc00eac8cac7923fe4504951], BasicMerkNode, HASH[1f4bee393167bbeff921a8d577c20ab4939af57ce0f5835255ccc92538485f8d]))
1: Push(KVHash(HASH[08b88f8f4f1c20303d3be9c78935c7cdd6de33bdc4edc808ff8ddde0e2f3ec66]))
2: Parent
3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
4: Parent
5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
6: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Diagram: per-layer merk-tree structure
See the GroveDB Layout diagram for the overall storage shape. Q1's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q1 uses only the constant-prefix path layers down to the doctype's primary-key tree element. The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.
The descent stops at the doctype's primary-key tree — the green node at the top of the layout. Because documentsCountable: true + documentsSummable: "score" upgraded that tree to a CountSumTree, the count and sum are both one O(1) read with an O(log n) proof. The client divides locally to get the average. Same proof shape as count's Q1 and sum's Q1 individually — the CountSumTree just commits both fields at every merk node it walks, costing a constant ~8 extra bytes per descent layer vs. either single-axis variant.
Query 2 — Average for One Class (byClass)
select = AVG(score)
where = class == "MATH101"
prove = true
Path query:
path: ["@", contract_id, 0x01, "grade", "class"]
query items: [Key("MATH101")]
Verified result:
path: ["@", contract_id, 0x01, "grade", "class"]
key: "MATH101"
element: CountSumTree { count_value_or_default: 1000, sum_value_or_default: ≈50000 }
average: ≈50000 / 1000 = ≈50.0
Proof size: 801 bytes. Avg time: 32.1 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (6 layers — byClass point lookup) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
lower_layers: {
0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
lower_layers: {
grade => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[beefc1778aa2b24de9a979d69add4f02fe376983ff85d287462ec5e34dc1f764]))
1: Push(KVValueHash(class, Tree(4348454d313031), HASH[25ffaf63d65ed1b63f796004c15bdf33757a4b86e3bcde03f67df9c9d42d2168]))
2: Parent
3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
4: Parent
5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
6: Child)
lower_layers: {
class => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[221cc4921243629c99fbf517a7f8a93aa3d2f894537bb8a071ed1d46abe64a02]))
1: Push(KVHash(HASH[97aae10e38ef5c482deddc58a643a50a9f27d23876d8327458c163cfbda2da9a]))
2: Parent
3: Push(Hash(HASH[c1b9be8b2629a4cfceb100f6c9e40a5e798dc0c5782e4ce9722f9a4747753711]))
4: Push(KVHash(HASH[34ebec873b25c565a93d25e70507b749406b80b014cfa4ec70d1108e44a62cb0]))
5: Parent
6: Push(Hash(HASH[2e7c7f97470f615c1348e70489c8e1a25c823b88cbf4ecb20db8f05256231211]))
7: Push(KVValueHashFeatureTypeWithChildHash(PHYS101, CountSumTree(73656d6573746572, 1508, 84598), HASH[ac1dccf6426a8467d1b923ebc24e15fb8ac504072fe20b136f1dee0ea7ec2073], BasicMerkNode, HASH[116a0e0b1328cc8529ed2cd4326afbf4e9ae37d15f178d582261d08c2a3894dc]))
8: Parent
9: Push(Hash(HASH[c9ef06be93f8ae382500c13ce025a9920ded466cd4794555d8eb1f6b5a4749e4]))
10: Child
11: Child
12: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Diagram: per-layer merk-tree structure
See the GroveDB Layout diagram for the overall storage shape. Q2's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q2 adds one extra layer into the class property-name subtree and stops at the PHYS101 terminator. The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.
The descent walks one extra layer into the class property-name subtree and stops at PHYS101. The verified result is count=1 508, sum=84 598, avg=56.099 — PHYS101 is one of the harder classes in the bench's profile table — class baseline of 60 minus a slight negative average across all enrolled students' skills puts the verified average at 56.099. Notice the count (2 281, not 5 000) — that's the enrollment filter at work: only ≈ 30% of (student, semester) slots enroll in PHYS101 per the popularity table, so 500 students × 10 semesters × 30% ≈ 1 500 enrolled. The actual 2 281 falls above that because some students bunch up on PHYS101 in certain semesters and the hash-based filter isn't perfectly uniform — that asymmetry is reproducible and visible in the verified count. Because byClass declares both countable: countable and summable: "score", that node is a CountSumTree carrying both per-class metrics directly — no need to step into [0] to look at individual references. Same shortcut count proofs and sum proofs take, just with one element committing two fields rather than two elements committing one each.
Query 3 — Student GPA (byStudent)
select = AVG(score)
where = student == student_050
prove = true
Path query:
path: ["@", contract_id, 0x01, "grade", "student"]
query items: [Key(student_050)]
Verified result:
path: ["@", contract_id, 0x01, "grade", "student"]
key: student_050
element: CountSumTree { count_value_or_default: 100, sum_value_or_default: ≈5000 }
average: ≈5000 / 100 = ≈50.0
Proof size: 1091 bytes. Avg time: 42.0 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (6 layers — byStudent point lookup) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
lower_layers: {
0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
lower_layers: {
grade => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[2fa3be1c9771e5c4a10dfe3b5a7dbad43a2775c3822e77cb03ada370f92d608c]))
1: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
2: Parent
3: Push(KVValueHash(student, Tree(00000000000000d1ffffffffffffff2e00000000000000000000000000000000), HASH[24622f7d7a9da5318a2043bb2cb483c7222ad01aa2b24ebe967111ca5e7977cf]))
4: Child)
lower_layers: {
student => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[6759bd80220fbb50622101bf21cd6c3c2d9bd4832dbe04ef85bc8f673674ce39]))
1: Push(KVHash(HASH[71b3c529ec4ef9a110626a15592a2f49cd362729d6c17772844efbfa184e9693]))
2: Parent
3: Push(Hash(HASH[3fb3fed5141b18c53c15769697eec36e067e94191e56d77fbb507239deacf868]))
4: Push(KVHash(HASH[95a8d0469cc6e021c5db7c5d9b429531b948636b0209140cc1dc0a1b305af7f5]))
5: Parent
6: Push(Hash(HASH[dc602dc298843d5c96e4ff87d27ddeed9d9d3a5a0715c8ea96954c5dd075befc]))
7: Push(KVHash(HASH[b862f6f003cc80580d7e980a229658d28ed75d9739c49b443b1d9fdfcf8ece19]))
8: Parent
9: Push(KVValueHashFeatureTypeWithChildHash(0x0000000000000032ffffffffffffffcd00000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 4289), HASH[1294d46e235be085d6fa9a0acd1beca9bb7e22e470e0705dd512c4cdbce5e312], BasicMerkNode, HASH[b4d5a93458113eb96afd5a80371df3cd7f8d8296de229a1e8b19dd7a7aec5ffe]))
10: Push(KVHash(HASH[094f3bfd41b7722c90f35dcd4a5e1c68d1a7667043ae56059e232b1045852a45]))
11: Parent
12: Push(Hash(HASH[a73503263b84cc37101581a6eec94dd3dbeec4c437a7eac0ffded8e29d820567]))
13: Child
14: Push(KVHash(HASH[f4b374ae9d1f2bd74405661b8c79f1d1d4342670311b2a7244918af89142f721]))
15: Parent
16: Push(Hash(HASH[cf41f288ea7730eb52ad5547370c4340b836057822ca5ff52356091ae65e03ef]))
17: Child
18: Child
19: Child
20: Child
21: Push(KVHash(HASH[d1bf190eafbd359612378043df0b00938c4070422ec3301d8160535ce62369d9]))
22: Parent
23: Push(Hash(HASH[6cc749152286a0f937958e75818a94f1506108cbee30147700f1233e9cf3684f]))
24: Child
25: Push(KVHash(HASH[93a9ff43d512bc697b94d3d13b8e4be942c6d2efab9e97ac4cd5b80404f9c84d]))
26: Parent
27: Push(Hash(HASH[fa08edb289995985b0f169c6eb2e073414994839941262ae58be3d90277e7029]))
28: Child
29: Push(KVHash(HASH[c7c39a5ae845f2ff491f91fe178d7230364a72db37cb13b3864809e2d8ca7041]))
30: Parent
31: Push(Hash(HASH[0cac6b8e6bd604e6dac04ea4bf84d0304e39e3a0e811eb257eac05587bdf23fa]))
32: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Diagram: per-layer merk-tree structure
See the GroveDB Layout diagram for the overall storage shape. Q3's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q3 has the same shape as Q2, just over a different property-name subtree. The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.
Structurally identical to Query 2 — different property-name subtree (student instead of class), different terminator value, same CountSumTree element shape. Verified count=62, sum=4 289, avg=69.18 — student_050's GPA across the 62 grades they happen to be enrolled in. The count of 62, not 100, is the enrollment filter showing up — student_050 didn't enroll in every class every semester. With the realistic-data fixture, student_050 turns out to be slightly above average (avg=69.18 vs. the global ≈ 72 baseline heavily pulled up by ENGL101+ARTS101 enrollment) — the FNV hash of 50 happens to land in the positive-skill region of the student distribution.
Query 4 — One Cohort (byClassSemester point)
select = AVG(score)
where = class == "MATH101" AND semester == 20241
prove = true
Path query:
path: ["@", contract_id, 0x01, "grade", "class", "MATH101", "semester"]
query items: [Key(serialize_value_for_key("semester", 20241))]
Verified result:
path: ["@", contract_id, 0x01, "grade", "class", "PHYS101", "semester"]
key: serialize_value_for_key("semester", 20204)
element: ProvableCountProvableSumTree { count_value_or_default: 147, sum_value_or_default: 8114 }
average: 8 114 / 147 = 55.197
Proof size: 1233 bytes. Avg time: 51.0 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (8 layers — byClassSemester compound point lookup (PCPS terminator)) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
lower_layers: {
0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
lower_layers: {
grade => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[beefc1778aa2b24de9a979d69add4f02fe376983ff85d287462ec5e34dc1f764]))
1: Push(KVValueHash(class, Tree(4348454d313031), HASH[25ffaf63d65ed1b63f796004c15bdf33757a4b86e3bcde03f67df9c9d42d2168]))
2: Parent
3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
4: Parent
5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
6: Child)
lower_layers: {
class => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[221cc4921243629c99fbf517a7f8a93aa3d2f894537bb8a071ed1d46abe64a02]))
1: Push(KVHash(HASH[97aae10e38ef5c482deddc58a643a50a9f27d23876d8327458c163cfbda2da9a]))
2: Parent
3: Push(Hash(HASH[c1b9be8b2629a4cfceb100f6c9e40a5e798dc0c5782e4ce9722f9a4747753711]))
4: Push(KVHash(HASH[34ebec873b25c565a93d25e70507b749406b80b014cfa4ec70d1108e44a62cb0]))
5: Parent
6: Push(Hash(HASH[2e7c7f97470f615c1348e70489c8e1a25c823b88cbf4ecb20db8f05256231211]))
7: Push(KVValueHash(PHYS101, CountSumTree(73656d6573746572, 1508, 84598), HASH[ac1dccf6426a8467d1b923ebc24e15fb8ac504072fe20b136f1dee0ea7ec2073]))
8: Parent
9: Push(Hash(HASH[c9ef06be93f8ae382500c13ce025a9920ded466cd4794555d8eb1f6b5a4749e4]))
10: Child
11: Child
12: Child)
lower_layers: {
PHYS101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[dd04eaab3acc2eb21101895d29f716fcc264adec3ee3c3d0828b68b2b1efb6a1]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 1508, 84598)), HASH[80f59d6ce839fd72da96b8d1b228172a9e80f4df9f8f9e078a87224943b8f816]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[b1b868b65a239d4256d28919204c86a5f26f9dea470eb984e4884c8801174265]))
1: Push(KVHashCountSum(HASH[1d46fcc02a25b527028f49be8a58c891b3f19588348ac092a4bd446c85733c64], count=1508, sum=84598))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 147, 8114), HASH[ba3ae5cda415f7540cec982bd8403174f8b80ce2604463c630b6505692c4f0e4], ProvableCountedAndProvableSummedMerkNode(147, 8114), HASH[cc3c59428d6ace408733b850eaf8b58c974339d87dabd84f3efe6e62557ab17a]))
4: Push(KVHashCountSum(HASH[2c145ca977d9a5a546df9eb3aaa67475e4004f60902c3dc4322ecbedafa88a6e], count=457, sum=25605))
5: Parent
6: Push(Hash(HASH[f8b2fa764a8fd989881e69b4e8570ada5d5846131b0af9c2770c5e9029b87f9c]))
7: Child
8: Push(KVHashCountSum(HASH[798b0509467547a7cbc0e666a3afccc36a58c9552491816733ee23483830cad5], count=906, sum=50770))
9: Parent
10: Push(Hash(HASH[95c1dec4eaa2a4489d17d6d492030369abb7c7c2aa8328411017f70f31441e50]))
11: Child
12: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Diagram: per-layer merk-tree structure
See the GroveDB Layout diagram for the overall storage shape. Q4's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q4 has two extra layers over Q2 — one for the byClassSemester continuation's semester subtree (yellow PCPS class), one for the per-cohort terminator (also yellow PCPS). The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.
Two property-name descents (class, then under PHYS101 the byClassSemester continuation's semester). The terminator here is a ProvableCountProvableSumTree (PCPS), not a CountSumTree — that's because both rangeCountable: true and rangeSummable: true on byClassSemester upgrade not just the property-name tree but also the per-value cohort terminator to PCPS. (The chapter's earlier draft said CountSumTree; the bench reveals the dispatcher actually picks PCPS for any value tree under a range-bearing index, so we get PCPS's per-node aggregation even for a point lookup.) For our purposes here — extracting (count, sum) from one merk element — PCPS and CountSumTree are equivalent at the read site; PCPS just carries the extra per-node fields that Query 5's range walk needs. Verified count=147, sum=8 114, avg=55.197.
Query 5 — Class Trend (AggregateCountAndSumOnRange)
select = AVG(score)
where = class == "MATH101" AND semester > 20210
prove = true
Path query:
path: ["@", contract_id, 0x01, "grade", "class", "MATH101", "semester"]
query items: AggregateCountAndSumOnRange(RangeAfter(serialize_value_for_key("semester", 20210)..))
Verified result (returned by GroveDb::verify_aggregate_count_and_sum_query):
(root_hash, count, sum) where count = 759, sum = 42 656
average: 42 656 / 759 = 56.200
(759 grades in range — 5 semesters × roughly 150 enrolled students per semester for PHYS101, matching the documented 30% popularity. The enrollment filter is visible in every range query.)
Proof size: 1469 bytes — O(log T') regardless of how many semesters lie in the range, because AggregateCountAndSumOnRange collapses the boundary walk into a single committed (count, sum) pair at proof-generation time. Same proof-size profile as count's Query 7 and sum's Query 7, with an extra i64 per merk node for the sum field on top of count's per-node count field. Strictly smaller than two independent proofs would be (which would each carry the merk descent overhead separately, plus the client would have to verify two root-hashes match).
Avg time: 49.7 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (8 layers — AggregateCountAndSumOnRange collapse on PCPS continuation) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
lower_layers: {
0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
lower_layers: {
grade => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[beefc1778aa2b24de9a979d69add4f02fe376983ff85d287462ec5e34dc1f764]))
1: Push(KVValueHash(class, Tree(4348454d313031), HASH[25ffaf63d65ed1b63f796004c15bdf33757a4b86e3bcde03f67df9c9d42d2168]))
2: Parent
3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
4: Parent
5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
6: Child)
lower_layers: {
class => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[221cc4921243629c99fbf517a7f8a93aa3d2f894537bb8a071ed1d46abe64a02]))
1: Push(KVHash(HASH[97aae10e38ef5c482deddc58a643a50a9f27d23876d8327458c163cfbda2da9a]))
2: Parent
3: Push(Hash(HASH[c1b9be8b2629a4cfceb100f6c9e40a5e798dc0c5782e4ce9722f9a4747753711]))
4: Push(KVHash(HASH[34ebec873b25c565a93d25e70507b749406b80b014cfa4ec70d1108e44a62cb0]))
5: Parent
6: Push(Hash(HASH[2e7c7f97470f615c1348e70489c8e1a25c823b88cbf4ecb20db8f05256231211]))
7: Push(KVValueHash(PHYS101, CountSumTree(73656d6573746572, 1508, 84598), HASH[ac1dccf6426a8467d1b923ebc24e15fb8ac504072fe20b136f1dee0ea7ec2073]))
8: Parent
9: Push(Hash(HASH[c9ef06be93f8ae382500c13ce025a9920ded466cd4794555d8eb1f6b5a4749e4]))
10: Child
11: Child
12: Child)
lower_layers: {
PHYS101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[dd04eaab3acc2eb21101895d29f716fcc264adec3ee3c3d0828b68b2b1efb6a1]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 1508, 84598)), HASH[80f59d6ce839fd72da96b8d1b228172a9e80f4df9f8f9e078a87224943b8f816]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[4b85291fe8e5cae442614096553956521bb55510873f56ea4219d9a56d01408d], left=HASH[49700ad27bc9ac383b5bb5e867114f76acf4701ad7ea97311e12e877e92ffda9], right=HASH[5e9ff5741419a71daf0163e97389cf154aaea23144a589417a4a76e8df5904b4], count=455, sum=25572))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[fe6c93990c99748cec859c40fe458a191825c0941cd064d18eeaec17e52e0999], count=1508, sum=84598))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[ba3ae5cda415f7540cec982bd8403174f8b80ce2604463c630b6505692c4f0e4], count=147, sum=8114))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[e19d566cffe1e46af9eabb5b3b040898109a1d9d50a6a1bf87a04f421fd2617b], count=457, sum=25605))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[38be6684aff1201eeec45aedd505d3634fbcda546b158c5ae43e116819f2ea22], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=153, sum=8588))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[06ce5728c482b63134cf57461e9c4248638064a94393f9085842c830ae830381], count=906, sum=50770))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[8c2d715e4be3e576f5c4327d93f37192f71de69c46aa162ab9bcb725d53a3a46], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[de29287042c9ee0874e346cac4e947177e24129a9ec250bb1a31a575da222747], count=302, sum=16922))
11: Child
12: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Diagram: per-layer merk-tree structure
See the GroveDB Layout diagram for the overall storage shape. Q5's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q5 walks the same 8 layers as Q4 but the terminator is a range-collapse merk-node commit (no individual per-key terminator; the merk-tree's boundary walk produces a single (count, sum) pair via the PCPS per-node fields). The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.
This is the chapter's headline payoff: a single committed (count, sum) pair from one merk traversal of the byClassSemester PCPS continuation. The verifier cryptographically guarantees that both metrics describe the same in-range grades — there's no way for the server to splice a count from one set with a sum from another. The client divides locally to get the verified average. The PCPS leaf-shape primitive requires the terminator tree to be a ProvableCountProvableSumTree; both lighter sum-bearing and count-bearing variants reject the combined primitive at the merk gate.
Query 6 — Per-Student Averages for One Semester (carrier)
select = AVG(score)
where = student IN [student_000, student_001, ..., student_009] AND semester == 20241
group_by = [student]
limit = 10
prove = true
Path query (carrier-style: outer Query enumerates the In branches, subquery descends through the byStudentSemester semester == 20241 lookup):
path: ["@", contract_id, 0x01, "grade", "student"]
query items: [Key(student_000), Key(student_001), ..., Key(student_009)]
subquery_path: ["semester"]
subquery items: [Key(serialize_value_for_key("semester", 20241))]
Because the inner where is semester == 20241 (a point, not a range), the per-bucket terminator is a CountSumTree element — not PCPS. This is the CountSumTree-carrier flavor that returns Vec<(key, count, sum)> by reading the (count_value, sum_value) off each per-bucket CountSumTree, not the AggregateCountAndSumOnRange flavor (which is reserved for range-bucket cases — see Query 7).
Verified result (returned by the carrier verifier):
(root_hash, entries) where entries =
[
(student_000, count=7, sum=477, avg=68.14)
(student_001, count=6, sum=500, avg=83.33)
(student_002, count=6, sum=496, avg=82.67)
(student_003, count=6, sum=441, avg=73.50)
(student_004, count=7, sum=418, avg=59.71)
(student_005, count=6, sum=459, avg=76.50)
(student_006, count=6, sum=445, avg=74.17)
(student_007, count=6, sum=408, avg=68.00)
(student_008, count=7, sum=587, avg=83.86)
(student_009, count=6, sum=482, avg=80.33)
]
aggregate across all 10 buckets: count=63, sum=4713, avg=74.81
Per-bucket counts vary from 6 to 7 in this 10-student sample (students take a different number of classes per semester depending on which electives the enrollment filter accepts for them). The per-bucket sums spread from 408 (student_007 — drew lower-skill, mostly-hard classes that semester) to 587 (student_008 — drew higher-skill, mostly-easy classes that semester) — a real-data shape.
Proof size: 6581 bytes. Avg time: 304.4 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (8 layers — CountSumTree-carrier × 10 student buckets with point-inner subquery) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
lower_layers: {
0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
lower_layers: {
grade => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[2fa3be1c9771e5c4a10dfe3b5a7dbad43a2775c3822e77cb03ada370f92d608c]))
1: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
2: Parent
3: Push(KVValueHash(student, Tree(00000000000000d1ffffffffffffff2e00000000000000000000000000000000), HASH[24622f7d7a9da5318a2043bb2cb483c7222ad01aa2b24ebe967111ca5e7977cf]))
4: Child)
lower_layers: {
student => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x0000000000000000ffffffffffffffff00000000000000000000000000000000, CountSumTree(73656d6573746572, 63, 4228), HASH[cd6a37aaa2c84a9a441fea60b3a0467194fc626f67ec713780f0805c56309a31]))
1: Push(KVValueHash(0x0000000000000001fffffffffffffffe00000000000000000000000000000000, CountSumTree(73656d6573746572, 63, 5010), HASH[967dbf74b3c2e23feee4745f809322c59c73dbad00fa2ceb1fa0d859a376096c]))
2: Parent
3: Push(KVValueHash(0x0000000000000002fffffffffffffffd00000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 5104), HASH[dca989d86ed13f79aecbc1aa4336b6453aa654573fa6746f2a5d58a20e4aa41c]))
4: Push(KVValueHash(0x0000000000000003fffffffffffffffc00000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 4654), HASH[c6e417e5f392321939ad1398786db7d0006a304e6de3e1e0abf82501091633d5]))
5: Parent
6: Child
7: Push(KVValueHash(0x0000000000000004fffffffffffffffb00000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 3652), HASH[68975d6209b052dbc23d7e1e6d9a8423adb3da71ed2fb0a7f634101627eff7f7]))
8: Parent
9: Push(KVValueHash(0x0000000000000005fffffffffffffffa00000000000000000000000000000000, CountSumTree(73656d6573746572, 65, 4664), HASH[5e0e79e266cc34ff7499dda1d82bc6e4f495b60fdb3c1697afdaa27ec5d28851]))
10: Push(KVValueHash(0x0000000000000006fffffffffffffff900000000000000000000000000000000, CountSumTree(73656d6573746572, 63, 4717), HASH[ae1234b1a941bfc827eca9fc3a459f959d85dc54f269f7a7fb29470ae6419f35]))
11: Child
12: Push(KVValueHash(0x0000000000000007fffffffffffffff800000000000000000000000000000000, CountSumTree(73656d6573746572, 63, 4229), HASH[94475fc16687fe817e942bae9255abf688dbd01de1a2bf1d90331bee49e1a71a]))
13: Parent
14: Push(KVValueHash(0x0000000000000008fffffffffffffff700000000000000000000000000000000, CountSumTree(73656d6573746572, 62, 5189), HASH[b7970fefd63e74384790758e760c74913bb83c56caab31cf2941623e4143d745]))
15: Child
16: Child
17: Push(KVValueHash(0x0000000000000009fffffffffffffff600000000000000000000000000000000, CountSumTree(73656d6573746572, 64, 4871), HASH[77f98f78dab2a44de7c016599cf4fff25ede5872b01191241ce8c4dd0e2b4051]))
18: Parent
19: Push(Hash(HASH[eb09a430767a7d4cce9a6b13bcf5c969e10358520b9099433204178a94f77a1a]))
20: Child
21: Push(KVHash(HASH[71b3c529ec4ef9a110626a15592a2f49cd362729d6c17772844efbfa184e9693]))
22: Parent
23: Push(Hash(HASH[927e9b34e1e5f8bf58e89d4967357bb433e8bdd541da24a5e41ebca9ea53ae7a]))
24: Child
25: Push(KVHash(HASH[d1bf190eafbd359612378043df0b00938c4070422ec3301d8160535ce62369d9]))
26: Parent
27: Push(Hash(HASH[6cc749152286a0f937958e75818a94f1506108cbee30147700f1233e9cf3684f]))
28: Child
29: Push(KVHash(HASH[93a9ff43d512bc697b94d3d13b8e4be942c6d2efab9e97ac4cd5b80404f9c84d]))
30: Parent
31: Push(Hash(HASH[fa08edb289995985b0f169c6eb2e073414994839941262ae58be3d90277e7029]))
32: Child
33: Push(KVHash(HASH[c7c39a5ae845f2ff491f91fe178d7230364a72db37cb13b3864809e2d8ca7041]))
34: Parent
35: Push(Hash(HASH[0cac6b8e6bd604e6dac04ea4bf84d0304e39e3a0e811eb257eac05587bdf23fa]))
36: Child)
lower_layers: {
0x0000000000000000ffffffffffffffff00000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[2c76195b7770957718fcb21d464f6229e1e2940005269dace3a4933319706658]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 63, 4228)), HASH[fe3cd77b60b86564f692f5204c324eb5f2a03af6d66273054c208fdaf1978484]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd4c9f525f3ac22ecc2f5ae6e00aaee9a32863a41add67f507d1578209e5ccb1]))
1: Push(KVHashCountSum(HASH[ed8239c0bd92fe50785ea501400b4198acc410c2371602dbf13d76f44c16f7d5], count=63, sum=4228))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 7, 477), HASH[ef1d416d8fea679e43170c4d314d6f7523ca70544e575370c474547d542b3f6f], ProvableCountedAndProvableSummedMerkNode(7, 477), HASH[388656200c3ec466a3e066a3a658df8ea428dda1f7ace466a97a9ad4f64b4658]))
4: Push(KVHashCountSum(HASH[2a3a332a185f435475e113e557765ad458171c09280d25940f3e748193690bd8], count=18, sum=1205))
5: Parent
6: Push(Hash(HASH[bddd5899ad6d572d8ff53a7535f1622430fde64dcd845800ef27c4b597f44b25]))
7: Child
8: Push(KVHashCountSum(HASH[5290e5ef5976a00dc0f8e1e65aecbaaed0e1f98a63f4f572e414388748b6e75c], count=38, sum=2547))
9: Parent
10: Push(Hash(HASH[41e38b943c0bb7ad82fcb5b782c6ea2abc0e0d636f545658fe0892fa752d0255]))
11: Child
12: Child)
}
}
}
}
}
0x0000000000000001fffffffffffffffe00000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[6b01cebbe317609614cd2032188e5ebe456919474223370041a1875ca102d725]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 63, 5010)), HASH[05908d7f182bb177509a9a62f2d8c274e6fae4a407c74f40d4af38bfc66c755b]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[3d4b43d9f0d56e8842381fdd29b46d7706084da492c5c4bcae4c1054938fb16e]))
1: Push(KVHashCountSum(HASH[f2a00fc085b00b74daa4552f3cf56a94fc6b890410197e34af4cc9aa5b776176], count=63, sum=5010))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 500), HASH[d70c0a972023c06cfb5af447b578075b0ad9c62978366711fff09f4755687cb8], ProvableCountedAndProvableSummedMerkNode(6, 500), HASH[f179844799861dfd9d97f5b00088fceaf210e88c810e5cf019c7690d43edbbcc]))
4: Push(KVHashCountSum(HASH[9c88005912fad6fa694c190694b47c990e87f03b76808257770d350622215563], count=18, sum=1443))
5: Parent
6: Push(Hash(HASH[d8f00e8973512018a5d313855bd512a1386db3df94bbe2880a36cfacdc5db8cb]))
7: Child
8: Push(KVHashCountSum(HASH[79e976ee63058e02f844a3072df96fcb09d9b520dd473d7decf63b1a7f351a29], count=37, sum=2964))
9: Parent
10: Push(Hash(HASH[c3f3fb7bb10a6930ed23d7c10b1ea4c27194084468c3917a146ad506cd58c43e]))
11: Child
12: Child)
}
}
}
}
}
0x0000000000000002fffffffffffffffd00000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[d625fc067da88f86b5d7bcaaef342e82df5146329ae2522d5806feaf48c5a9ed]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 62, 5104)), HASH[c894416ed05d217104355f7e5acc5deaf090f96158c168d8956c96814a82c763]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[3781ba0dbd202a90681259edd62e32d7914dde189d81c3a6b234b2b888d32bbe]))
1: Push(KVHashCountSum(HASH[e349a23b2e008a6f9a3eb5ba687f84e5007f4bc42f4e1f7135612ed3321bb400], count=62, sum=5104))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 496), HASH[b08ccf15548715cc0ed999ef1ca9be1ecfb302fc3e39b600d1b089decb91e9c4], ProvableCountedAndProvableSummedMerkNode(6, 496), HASH[1b5687189cf542500b393af906e4709dcfe0157c6a5cbc2599b05807f52a3962]))
4: Push(KVHashCountSum(HASH[a156211e0a1b4c14722886364a171f03e3a692fce93b279270486a3606023611], count=18, sum=1471))
5: Parent
6: Push(Hash(HASH[9c191caa12c32abb856347c6f590530642be1f12cce3c26a48f701fdacffcd74]))
7: Child
8: Push(KVHashCountSum(HASH[28a46c7a36502f4688cda1fd8ebf4fac49a1796c31f5f644f8e9ab54461750a6], count=37, sum=3029))
9: Parent
10: Push(Hash(HASH[492b7dcf673e717285fe90683bb91958b9751170757f2527992d998a2a4d1934]))
11: Child
12: Child)
}
}
}
}
}
0x0000000000000003fffffffffffffffc00000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[246fd08691b346678fb5c4f97cc6a74ad81a9901f15bee494a042c616ed1308a]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 62, 4654)), HASH[c4be1dbc0307777727c7d01d4c67cb4aead93899cef3c50b00dcb7c082d27a35]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[10e09dc243d6103d83938e3cedfd605cd5277d6ef3c0e4fc2da6bae7182ca44a]))
1: Push(KVHashCountSum(HASH[759c768a1ff2180270af1e71443e480cb31a0d39a7b7df65827cd879045e3515], count=62, sum=4654))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 441), HASH[fd79150f43d983e625db22b2bcfb8a7eb320d82a2d70815b4dc16f09ce71eadf], ProvableCountedAndProvableSummedMerkNode(6, 441), HASH[a271f341e8a3ee353a157275c10a5b82e245519857f6873d4caa41ad2539926c]))
4: Push(KVHashCountSum(HASH[9736bf15a3a713f92a9c13ba5f3fb8d3e54c8ef19b576fb850812afbbaf82a35], count=19, sum=1399))
5: Parent
6: Push(Hash(HASH[a748ca0363c702a22dcf7f8864d9aa45aea064db161e8a130b4e8d422f938b11]))
7: Child
8: Push(KVHashCountSum(HASH[eede689eb26f342042ea8bac1bc9488d2c5eda1c95334ef5d47331f79f85638a], count=38, sum=2857))
9: Parent
10: Push(Hash(HASH[4637407982bd38e1f1ca8a8695616ca8ecb3f176aef0ac405857893f943c9da5]))
11: Child
12: Child)
}
}
}
}
}
0x0000000000000004fffffffffffffffb00000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[058460675a738a0ac14d44dad1dc32d2fa85528394520a469f487d5cb4e26c0b]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 62, 3652)), HASH[4d4bf3754db9a87bf5afbaff143b0fc0082a786785a5d65ef211cadc975426cc]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[ac382aca72f4dc7d7ecd29f1a805ef731f459ae014d9436070b8f30a51ab251b]))
1: Push(KVHashCountSum(HASH[bb7c46cfba7f651cb55026df9f02828eef7ff090bc7f3cbfca45084be6d2594d], count=62, sum=3652))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 7, 418), HASH[67fd6a495071fbad4f42632b871c4c3461291e6dd6a64d2032dc4e540040c0ff], ProvableCountedAndProvableSummedMerkNode(7, 418), HASH[d43bef142d6e74ae7699e5c8b62c08eb49ef652b87633ef6462b9403ccc78c91]))
4: Push(KVHashCountSum(HASH[5341577287cc4b1dad09da63cde5957f85ab343451ce8910a016f624f77f50c3], count=18, sum=1050))
5: Parent
6: Push(Hash(HASH[3d96b9320c8177ea8533fe4f8be62b1cd83a0d058847717e0c466fb0e9d6b817]))
7: Child
8: Push(KVHashCountSum(HASH[c5fb037894fdbb4f0ec82c7fcd24857003bc14f07b9c721ca7e8d073751c834f], count=38, sum=2209))
9: Parent
10: Push(Hash(HASH[276371d55550ba8e2fd6744b0d2a87f5849bf12cb2819e7fcf8cf9405acddc5a]))
11: Child
12: Child)
}
}
}
}
}
0x0000000000000005fffffffffffffffa00000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[861d84a680ab44ccca0533eb5a1e90318e0e4161216e6a3292f01c37ae4d4286]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 65, 4664)), HASH[14cdcd21b12c66c58bcf74f04d2c714ff7d504c10dfd948e396ef2455dfc4ada]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[df54061a16d1fa7dc0ec2581d4221b76665abcf044b03ba0562b48484b78a1be]))
1: Push(KVHashCountSum(HASH[0729dabe2fdce7dcd77ae5ba3a7e6b8e1377ed4264bf84e35eb029e89a8712d6], count=65, sum=4664))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 459), HASH[e89878a22d32aa9a3930dcaac98ae9c8713fe9826b77e4efc79bd3a70c934c52], ProvableCountedAndProvableSummedMerkNode(6, 459), HASH[1f098d52ad1de6f7e07b5fac53755156209b095a642511186a0caf236c0814d9]))
4: Push(KVHashCountSum(HASH[bf3677858e1625685bb7bd9b5c038d95cb518ca2fda87d0dd3c9838f8c5341d7], count=20, sum=1438))
5: Parent
6: Push(Hash(HASH[1b300f7ced2cbf471e4c0092fe2658d339a54976aa26eb10844f8ad193d2994d]))
7: Child
8: Push(KVHashCountSum(HASH[0ef7fea6dfc6ac9c5f60458487249c0cd7accfc498c7347074a8bab1ac455f77], count=40, sum=2887))
9: Parent
10: Push(Hash(HASH[1845b2da05e252a4f7f9b942ab4b196099da84795a8867fe9d61faf3e56c61a7]))
11: Child
12: Child)
}
}
}
}
}
0x0000000000000006fffffffffffffff900000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[f084f3761d0377eaa616db0cec66b860f00ca16411d31607c03db41ec0e767ef]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 63, 4717)), HASH[9bd1ce241ecf93ef09d46632b83d86b1795532975cb6c2232eaad76e4f8fb706]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[d8cf1140b01c2b5af25587aa28a224dc43ea3fd590872bfa97ab3283a9cbf922]))
1: Push(KVHashCountSum(HASH[30107a131a8e6fca131ca98ba42eb66ab46c2f7b118b563c41a7988a4c5b1ef0], count=63, sum=4717))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 445), HASH[148e907ea57790d36cb08b030a994e65d06370bf0afd1db9b53d4e3ceb66afdb], ProvableCountedAndProvableSummedMerkNode(6, 445), HASH[7422eef7fe3dbcf1333e6db4e467d593597863103e24afcbd9a2b2898e47a8d6]))
4: Push(KVHashCountSum(HASH[93385238d3e372e4a276f9e69b38c5fc67a4b9ce1947b022a5ede6d14dfda46d], count=18, sum=1341))
5: Parent
6: Push(Hash(HASH[98c836393f4fde21842a04cd8ff9b6679bb1a0716c35bc1d8a0dd40174756ce5]))
7: Child
8: Push(KVHashCountSum(HASH[8da039f749cbc46c3e4e520654c56615cac9b709fb44e547085bac85156f858c], count=38, sum=2831))
9: Parent
10: Push(Hash(HASH[1be2c53a013501c2a147eda502c2a25038ecf0b00f2cf2a9b864fcfb83b68990]))
11: Child
12: Child)
}
}
}
}
}
0x0000000000000007fffffffffffffff800000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[6757edb3583519e40a8b5ff066873865d5ecc3b4b73d18a40591ee1d4bc0b8a0]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 63, 4229)), HASH[af29fceb2e25e354a76a08b00fd2ee00523380af35d2e8827f371318ac960978]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[516484dcfeb93b608305f68199d907e51637443c48fa3f0f6d865447e5fc82f0]))
1: Push(KVHashCountSum(HASH[a86b417149127c4bcddf1f5dce95e9cb747073bec1b364b5cb332bc55880fa0f], count=63, sum=4229))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 408), HASH[48d6c9c3fbf427e0b8e1fba708a86268476615902653f46016a965100bffe86b], ProvableCountedAndProvableSummedMerkNode(6, 408), HASH[43c64ffaf3b718d8c591f6fbdb6360c3a4152dae040fe865cd53c46bc7561531]))
4: Push(KVHashCountSum(HASH[c87a344cd9e3689337ccb065d53ea6e19bd3ce9de711086b7c08c574688dc1c7], count=19, sum=1261))
5: Parent
6: Push(Hash(HASH[b319c8cc34211d304bcbab053b0131d46ae7b6f22bfcbfebabdde0706e088395]))
7: Child
8: Push(KVHashCountSum(HASH[90ddec9fcea82565519f269eae8ec8b79d88f2538a186fcad818c04d5d818ee2], count=39, sum=2615))
9: Parent
10: Push(Hash(HASH[abca4d83b4096d1e352c5e50e67359d48554427198f6ed516fdc8bcf7415f939]))
11: Child
12: Child)
}
}
}
}
}
0x0000000000000008fffffffffffffff700000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[ccbdeec0f90a04cf1b65d705d8a07fa57f80e136a28597654daf58f791825228]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 62, 5189)), HASH[aaf52bcba6c0523789d5ace2627c84f70d71681e9a666dd16f16055cf2044f1b]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[ffcdc9eb6e6961c65debc526347912280d6fd20328dc419de0cc47c24d9a598b]))
1: Push(KVHashCountSum(HASH[948c18e24a93d0cc101ad88f3de03078b3cac06141fa57c1f2fc2a4f985dbf28], count=62, sum=5189))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 7, 587), HASH[a061e5b6e0fc65b66786f63efd670aad881ef6a8731ffe84013e9829c715512b], ProvableCountedAndProvableSummedMerkNode(7, 587), HASH[ce13e20b17298a1ec9bf5cc7a6bcb1c7c41589592412eebfd18ea27bb4255a71]))
4: Push(KVHashCountSum(HASH[55b70d14ab3825c49c1089d50278cc81c20e712b5c90cd11805963a52653daf0], count=18, sum=1519))
5: Parent
6: Push(Hash(HASH[5ccefdc0d8741cf7eea2af8c3f9ea8ea76d9b5392de3fb41f51f77e384ce0bd4]))
7: Child
8: Push(KVHashCountSum(HASH[5a29557263bb375348d8f2f535fbe0b5c69bb9da18e05bb55ad2a6026a5a816d], count=38, sum=3180))
9: Parent
10: Push(Hash(HASH[d7f0f3a2db6f5b249dde2f48fde3ce72befbac78fa78cb369134626eba6396af]))
11: Child
12: Child)
}
}
}
}
}
0x0000000000000009fffffffffffffff600000000000000000000000000000000 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[5cebb1f5c631c08b1b27a473ae8bd6b89596aee1ab4049a9652ac1da0829890d]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 64, 4871)), HASH[b66d1c5e201c8c81e8895b44dc42cbd4af377b2bc7028c02ca1ecca0d8f6ddf5]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[285b3c7ec0ff52c2ae2abcd18c4e174b55cc5fdbe6ae56669552c2b5c491413c]))
1: Push(KVHashCountSum(HASH[f2a55f39d06bdd502fa50a8430176d11b3edfd7dce73fae1d535dcf59d5724ea], count=64, sum=4871))
2: Parent
3: Push(KVValueHashFeatureTypeWithChildHash(0x8000000000004eec, ProvableCountProvableSumTree(00, 6, 482), HASH[7b5b2db33170ef674e072a3d470d60d08e8af6dd30fc99f3817ceda5ba4330a4], ProvableCountedAndProvableSummedMerkNode(6, 482), HASH[2644743dddd29928dd9325b237025f14221b6470c45fde808dad21b55abc8d4b]))
4: Push(KVHashCountSum(HASH[a5cc999850410d4709b4b7160234990c4d53ac46f1e442aad6bf7bd451f6a43a], count=20, sum=1522))
5: Parent
6: Push(Hash(HASH[6054de4db622ca55074de67018493815d0fff63fb2d2659b000c91f7a43560d1]))
7: Child
8: Push(KVHashCountSum(HASH[23c949136919913400a75afc81c327673017844848b17bd9b53ce8588fc9a09f], count=39, sum=2981))
9: Parent
10: Push(Hash(HASH[e7f34b24fd8bdddc86e68d6237e8fe8bbeaa15dd3cf86a746ad4fc6f68fdf06b]))
11: Child
12: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Diagram: per-layer merk-tree structure
See the GroveDB Layout diagram for the overall storage shape. Q6's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q6 fans out at the student layer to 10 cyan student-id terminators (one per outer In branch). Under each terminator, the inner subquery walks one more layer (the byStudentSemester continuation's semester) and lands on a PCPS terminator (yellow) carrying that cohort's (count, sum). The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.
The carrier composition saves N round-trips: one proof returns averages for all 10 students simultaneously, vs. issuing 10 independent Query-4-shape proofs and dividing client-side per bucket. The verifier walks one outer descent (through student) and gets 10 per-bucket (count, sum) pairs in a single root-hash-committed payload.
Query 7 — Per-Class Trends (PCPS carrier)
select = AVG(score)
where = class IN ["MATH101", "PHYS101", ..., "ENGL101"] AND semester > 20210
group_by = [class, semester]
limit = 10
prove = true
Path query (PCPS-carrier: outer In over class, inner AggregateCountAndSumOnRange over the byClassSemester semester continuation):
path: ["@", contract_id, 0x01, "grade", "class"]
query items: [Key("MATH101"), Key("PHYS101"), ..., Key("ENGL101")]
subquery_path: ["semester"]
subquery items: AggregateCountAndSumOnRange(RangeAfter(serialize_value_for_key("semester", 20210)..))
Verified result (returned by GroveDb::verify_aggregate_count_and_sum_query_per_key):
(root_hash, entries) where entries =
[
( 'ARTS101', count=2267, sum=197461, avg= 87.102)
( 'BIOL101', count=1493, sum=103334, avg= 69.212)
( 'CALC201', count= 629, sum= 33694, avg= 53.568)
( 'CHEM101', count=1130, sum= 69300, avg= 61.327)
( 'COMP101', count=1372, sum= 98873, avg= 72.065)
( 'ENGL101', count=2500, sum=208872, avg= 83.549)
( 'HIST101', count=1764, sum=133138, avg= 75.475)
( 'MUSC101', count=2141, sum=171548, avg= 80.125)
( 'PHYS101', count= 759, sum= 42656, avg= 56.200)
( 'SOCI101', count=1759, sum=137574, avg= 78.212)
]
aggregate across all 10 classes: count=15 814 sum=1 196 450 avg=75.658
Each class's bucket has a different count — 629 for CALC201 (hardest math, 25% enrollment), 2 500 for ENGL101 (everyone takes it). This is exactly what real-data carrier-aggregate output looks like: the per-class average is informative on its own, but the per-class count also tells you something — how many students chose that class. The verified per-bucket averages span from CALC201 (53.6 — hardest math) to ARTS101 (87.1 — easiest art), a realistic 34-point spread. This is the chapter's most striking payoff number: one carrier proof returns ten cryptographically-attested averages along with the enrollment-derived counts that contextualize them, all from the same root-hash commit. Doing the same query without the carrier primitive would burn 10 round-trips and 10 separate root-hash matches; the PCPS-carrier collapses it to one proof.
Proof size: 8220 bytes — measured against Query 5's 1 539 B baseline, that's ≈ 5.6× the bytes for k=10 buckets, better than the predicted 6×–10× envelope because the shared top-of-tree merk descent (the first 4 layers down to grade/class) is amortized across all 10 outer Keys rather than walked once per bucket. The per-bucket marginal cost works out to (8 220 − 1 539) / 9 ≈ 742 B per added carrier bucket.
Avg time: 273.8 µs.
Proof display (GroveDBProof::Display):
Expand to see the structured proof (8 layers — PCPS-carrier × 10 class buckets with range-inner subquery) — or open interactively in the visualizer ↗
GroveDBProofV1 {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[bd291f29893fb6f6d6201087746ca1f23a178dd08e1346cb6c127e91ae3623b3]))
1: Push(KVValueHash(@, Tree(723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40), HASH[10a56c2707b7fcc97700cfa5dd2bfca4b881f975ded9b0f715bb99926d44a068]))
2: Parent
3: Push(Hash(HASH[19c924989e473a90d0848277d0b1498ccc8db3dc870cbc130e773f3d79ea5b71]))
4: Child)
lower_layers: {
@ => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40, Tree(01), HASH[42578c0f835a2d91d84b25beb8d49ceea8fbc926f9f3c8c8d3a0fb7af3d75f92])))
lower_layers: {
0x723785299b6682e8f4f4483423d95e2b67bc3d9a1bd09a5f864fa0703dfe8c40 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[15ab0920bb39de98aa007cd0ad5f8a263158849580c31827434d4fc976199579]))
1: Push(KVValueHash(0x01, Tree(6772616465), HASH[095a879a3c1f5de343d16aa5ef0c87063f7973b1fe8d250f8f2cb595891ba293]))
2: Parent)
lower_layers: {
0x01 => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(grade, Tree(73656d6573746572), HASH[2c67e58cbe8fa4f6c0c5e892141aee8642822ff5e014d5a8afd847b42dd155da])))
lower_layers: {
grade => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[beefc1778aa2b24de9a979d69add4f02fe376983ff85d287462ec5e34dc1f764]))
1: Push(KVValueHash(class, Tree(4348454d313031), HASH[25ffaf63d65ed1b63f796004c15bdf33757a4b86e3bcde03f67df9c9d42d2168]))
2: Parent
3: Push(KVHash(HASH[7df65880d4adce28f836f4f28c419efa34b96d614e7d90ffb66faf24bd0ed861]))
4: Parent
5: Push(Hash(HASH[dd444dce979bb4b3b5e66dea7deadecfbf4b7059551ce384cf645245772e4646]))
6: Child)
lower_layers: {
class => {
LayerProof {
proof: Merk(
0: Push(KVValueHash(ARTS101, CountSumTree(73656d6573746572, 4527, 394132), HASH[6287666053158d702e1cac8b1e0a1b5d019016885d7213c8dc5d0c772079cf97]))
1: Push(KVValueHash(BIOL101, CountSumTree(73656d6573746572, 3002, 207609), HASH[ba63cff57f365956744950b4c9e100ff34da3414eef0031e7260ad7609ad9522]))
2: Parent
3: Push(KVValueHash(CALC201, CountSumTree(73656d6573746572, 1254, 67097), HASH[352d89856ffbed603703a35b44f80c2a2426d71638574dd0d24c1f3a37ae749b]))
4: Child
5: Push(KVValueHash(CHEM101, CountSumTree(73656d6573746572, 2263, 139407), HASH[6180689ce9301e535a022141f4112ad99d0aada6dabb6de27b1e1e5e3462f9be]))
6: Parent
7: Push(KVValueHash(COMP101, CountSumTree(73656d6573746572, 2755, 198793), HASH[1204b674d76ba05a9bd9e32fe9fbb80894b88954394d28c9154d70ef72840e62]))
8: Push(KVValueHash(ENGL101, CountSumTree(73656d6573746572, 5000, 417853), HASH[3abff4fc48017f2b57cd7711cf0076ac0f0cc4084e5add8c974ac33c1e02b6d1]))
9: Parent
10: Push(KVValueHash(HIST101, CountSumTree(73656d6573746572, 3526, 266216), HASH[fc7cc082d6312d5ebce80dc959b0b46829ecae8120b03ecad2e9c2fe55a1b269]))
11: Parent
12: Push(KVValueHash(MUSC101, CountSumTree(73656d6573746572, 4271, 342332), HASH[4ff44ad6d87bdb6e6544c3f98753f5060b0397cf91f34e10f64a30e5b82d1df4]))
13: Push(KVValueHash(PHYS101, CountSumTree(73656d6573746572, 1508, 84598), HASH[ac1dccf6426a8467d1b923ebc24e15fb8ac504072fe20b136f1dee0ea7ec2073]))
14: Parent
15: Push(KVValueHash(SOCI101, CountSumTree(73656d6573746572, 3514, 274771), HASH[7f3e59b54da6cf457c87932ec5b767b1f46dd4f8d94be23adffb0fa507eacf70]))
16: Child
17: Child
18: Child)
lower_layers: {
ARTS101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[b5edb782fcdb15baf23f26a117ead34c39d873294932cb8079f8534ceec5441a]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 4527, 394132)), HASH[523ac5176cd507c4f154e987241c1e9abecf31413ab4e12332b4ae71d0b1a817]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[153f2a3fec67aef426e26ac9744b79b77cd88baed3235fb7c312535c77599fb7], left=HASH[7c038d6e4ed27c1cdea438f56371cf663420601673ffd265a4daec852d212eb8], right=HASH[8cffbb97f331cf75df0742a765fd10bb9f165fbe5933554e1fcb9d90ae9e716e], count=1353, sum=117678))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[8c316d5faff7bfec194ca0b9ee0e27f79cd71b7782f0a88e6ee52a4b6f2c5246], count=4527, sum=394132))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[62afce744a06db0df95c8bd4baf4624759411cf4f5abee3343135ea310c935e4], count=459, sum=39982))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[80b92b87c27dbf6ad7f2d7e8149e48b384eab882f8a78c55bb4e614be6e4a508], count=1361, sum=118550))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[c7a90676ad2028cc2ff92d446e746878fb0f7a583e79a2e6b49b7ce6ac64519c], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=445, sum=38706))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[6cfca633536cf2b5aa9258f275930f8ba031d8ca06cb02444bb80b5b33ffe46a], count=2726, sum=237443))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[12f0875d8a8e80b0a13dd44009c9424948b23289e1d3993745ad1c1764ba9016], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[18853cfbdb647bdc009bd72671a952cf95c3fdac4c9ce373104a70a7d154431a], count=918, sum=79924))
11: Child
12: Child)
}
}
}
}
}
BIOL101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[badf193cdbc6e6c299c66c548e3ecd8c85708e02879f78ad285ea8aa045602c3]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 3002, 207609)), HASH[7e6c15bd061c84812e5d8670e033f987bb67ff5179614f49b36cc288d7c2eef2]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[edada11476903c0944f218315679a3005273ecf2b4662d7d06390ed9d3e813a3], left=HASH[5fb9a5d7009679ccbdf2917f99a477eace8ff7afcd8efa527b33e81534f3c722], right=HASH[2bab918fd202d2b14075963422be7733ddd6b5724e45d9ba2d6864a58a3aad9d], count=894, sum=61706))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[1099b82842eee73c68184a7ee59977086ae84d99f7e9bb2274566db7c793a35d], count=3002, sum=207609))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[a39eb3ce411c0cedfd5ea8b2cf4e5ffe6539f0d76377feddd33a8b347956f171], count=308, sum=21387))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[93e0ce27c896a0b4813858bb4d75a3e8a601fd03b2379de0bf39fbfd6582f57a], count=895, sum=61926))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[0a1ee4f1f3f89ab1e924cdedf4a51ed0d6f22e941294164fd4b6ce7d8fe49faf], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=293, sum=20246))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[d981cc2ffa042bf112eb9fdfeec150e4ceb00de7a7cb64af0bedb4b2252722fd], count=1801, sum=124721))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[2c034fa7d67d634e69a2ba7d8aa475f1e1db49efc9eebee0dfbf475f194cfd19], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[e2aef757bc42bd4e662b381af0bc697a108aa63b987680cede760afef97221fd], count=602, sum=41689))
11: Child
12: Child)
}
}
}
}
}
CALC201 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[b3775adbb191d4cbe78a1b86e8aef02cc0e0c8fdcb8dd3580a778792082ce62b]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 1254, 67097)), HASH[f4cb4daf3fdbe3b12a7976fc114cc8157ebe9fbf7c96d128303dff4a34586a7f]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[4df3da5ee89a6442394bc1802324940493b9728464e709f9d2bde478a970bd72], left=HASH[25dec93cd6d566d6c15b4cae3550d0260ba0722d1174959bc3689a7e7cdf1f29], right=HASH[6fcb1dfe38659e9b217175e9b8970b3c7156487fbcdcc9f04cde3758cacea1e5], count=375, sum=20121))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[f904010258c04d68609e279ccffe0de58be26b55ce6699c287a3974d1b1dc972], count=1254, sum=67097))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[baf373e749feedf3dbaea57838e995c3fe92261ed1eed197a2646d0812086369], count=125, sum=6595))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[4a8d8b11d641a21f1ac878e916aa8f3037dedfd78954d275cff2452281cb1560], count=378, sum=20177))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[89f6c530f62e9ebc271188c6b7dbc27f8cc7249c9679c62cf5e427e692446c6c], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=126, sum=6904))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[36949b12299e375b573ccc872578c824e0377b2859ad55640692a73e61c19814], count=754, sum=40289))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[c36238784a0d1a9fa93f17bdb7352b8db84c88af171e00984cc8406406def117], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[a2109aba1a7f9a0731d8386a8f7e670582fe2f9f1c7838a497c5d0f6e111441f], count=251, sum=13507))
11: Child
12: Child)
}
}
}
}
}
CHEM101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c3620889aa20c51a24bc3752977653df1df0479f73dd119e19a64e942e0b8e6e]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 2263, 139407)), HASH[e60fa893a8789dbeadc4c882f63661c555f918df604503231038642a4171d890]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[8d9128eb523863f5e8b2a9e89e5dfb21ffde6bb7b577b4d234cee2c5d301b259], left=HASH[1e72254d8454ccc758f039cb1869840918b66671b18fcbe84a3139a53e7d433d], right=HASH[44014c573e0e11f43783c13309f92f2a1f9ac5b4cb4dc4ec7c267d7a89cf04e2], count=676, sum=41772))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[ea0da35647ac220198f5104e6e5ae7deeb5e9996baa7db88195ba87221538481], count=2263, sum=139407))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[13df40fc665d32c820baf598d6a15acb505e66b9a973191f89ded2b4e0c9eb89], count=226, sum=14044))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[a736fddb422d70bc326fed9d27c1cda6c83f59d9d6b18e4da02bb4c079e6fbe2], count=675, sum=41533))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[6a6bbf93e5786b93da68fbe6941f861856a4060c11bd8a4c15116824234e68ba], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=230, sum=14141))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[ab271c52dff2784bb05b5985d5597eac111736b3d65851effe9a05e5e2c2b906], count=1356, sum=83344))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[5de8d50527657bec30c7942113989d84aafae4ebe5ff643a8052950b1451a4c6], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[8ecff26ed936869396fc8c249fd824d1daf87a90721ec63146f3ce987562cd45], count=451, sum=27720))
11: Child
12: Child)
}
}
}
}
}
COMP101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c6c4a95162a3df6f9e0c69f267068632bf13098481c94436baac801af7496d91]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 2755, 198793)), HASH[a600eb34c6fd3eb4bc03640c127920a7f5fe9cbecec8d743e1b3abcb50c5e5f7]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[4ebb6c39abe967f06b124c364c8da2ca549cdba6bb8a7d86ca6e121bf3fe2b62], left=HASH[446c3f351eb207fce31d2de9dc02e381a2a98a062bfbcf912d77350c87897b7b], right=HASH[768df3838abcd5784856a8efd8d62c6637a0c04f69077152f301761598a27764], count=828, sum=59850))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[85aa3be7ccddc565d626a7dcaaf2712ba4a584f089936ce912fac36f16b70341], count=2755, sum=198793))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[9ab629c3b6b8919ea13061d58eeaad21e4635757489dc9c05757621c30e0058e], count=281, sum=20237))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[5ab4bccc51b7d6dbefa38d39d8389172ce2544898015e7f58c5a9b133f0d67ed], count=825, sum=59412))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[8a65e954ff97fea886e1c3685e106b0d4b4826a069769607eb825262db5cf431], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=271, sum=19498))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[362a5fa9985d5a5cac8c3e442376abebe1080041ac7a49620107a30e27829125], count=1653, sum=119110))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[113376007a1dda22c2c63348d7e48448a8a990f5204d029a022a9891e919fb60], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0b1eddb8b5d52ed2b6b0a2911b30cd8b0f7126a5f47b104380ebbe6bddea95e8], count=556, sum=40073))
11: Child
12: Child)
}
}
}
}
}
ENGL101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[56ea1c0de96e4b9f47d107679a1132228ee66ebe5b74fdb4a23a3adaf2652e9f]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 5000, 417853)), HASH[6780d21bd4cd6541f2eb43896f9d39eee0089819b3c5d6392dd16a116ed65eaa]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[ea0fd84c9f08448e30b96634a6ded32cbcc818e672dc180799d0d5c8fe7020ae], left=HASH[8eb17d573a5319468f69f8c671293e1a134b929c6c5be7d426a470357b123aea], right=HASH[059de3a36af4da7fc91e2fd3520816beceb02051cba847c022773013e6437977], count=1500, sum=125427))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[629f76cabb9d728b1c4e25cea6b189815e704afc3a46281e0991c5ef5fc5ea2a], count=5000, sum=417853))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[e2a9429b263d3e81228017f77c47050d01c6ee129c26d5231aa321886de7f951], count=500, sum=41776))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[e8476c17b96c46ef951b68f0079075b5f0d214bfbe07f088a4a6248bc5cbe801], count=1500, sum=125320))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[2ab628820220c346ff82830a69e97b4b12ed1544871e07000382c56be157afc2], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=500, sum=41772))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[5c4b915ebf521938fda7637a08a80b59b7a2f674e57e9f68a6b5e111fd7bbc07], count=3000, sum=250648))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[ab118a65fce19cdea048af30abc34483c055c1be3e846dfa1d86bdb462d9b41f], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[c686c9e613b11d1f098458bfc3c52964abec500965dfa3aaa1ac06ca9903551d], count=1000, sum=83548))
11: Child
12: Child)
}
}
}
}
}
HIST101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[c6eb6168c1c93d6fa08bf9383d4c564c737d8a2b4a6e46b80155df24a41e27d5]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 3526, 266216)), HASH[79aa05b68fee276dd5cfe031e738d2e89eb58cf786e77e9c93c0dd684628530a]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[8a5495292f3e480d7b9c6bc5b5fde238d2b8c389e20644a0d97f3e481cedf5e1], left=HASH[fba8a121f5fb68782a0f0f43fc1bb8b2102fb164fa73c79e0c3b4adb2214c009], right=HASH[7dcaf93df62b9522a2ba6a2904b4996e0e54406b5f2e6ad0283016fce6752d2f], count=1052, sum=79538))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[c3cccf8d9661d57fda5183e9a79d2ebb59091576a598111836525564b41831ac], count=3526, sum=266216))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[e79e9fb4a9b1b2807225bd61b8e4f9eb86440d8f0e19b49877b338dc53af8a0d], count=356, sum=26850))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[5005889b9794bc0d1093470fec1c7f8a27f286f2093d83a4b7ac663dcfca6128], count=1056, sum=79698))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[324f62b0b57d244cd88260cd456f4dc3b8abc605646bd795b93e3522d8e9934a], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=345, sum=26047))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[64672e9fd6e2a88e83a6b2b7b8e9316ee00943f80518f576d2bd3a793d7af2ab], count=2120, sum=159988))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[f2f4bc42cdd1f7f361b4eca932e3c2ad76772241f487632f87c7e0a99c0d4774], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[e33a2bf146527bb951eee9255d1b4e4a9ffd77a1ad4d24b9ba17d271dec1121d], count=711, sum=53677))
11: Child
12: Child)
}
}
}
}
}
MUSC101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[44631d7a5550d36e07615836d360ae1b3b54d16ebc3b431b01c07e08779c4fe7]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 4271, 342332)), HASH[fac7326ded3beb763fefbb0854f76765519e1d6d7d63d8460f699354016321f8]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[bb4e98cc506773058c14f6a9910ecabc8c6964c2285d17563994c37239b9ef11], left=HASH[0126c120d418971205f42e999ed15fbaf71f0a592600c32758ae07eb4415171b], right=HASH[43ec6cd8e561f3825397415cd0faea6c8923f43e1d0fd9be2d6ebe6091f9506e], count=1283, sum=102886))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[f98fd031e5c2d2665bb313143f92b6d97ec4fac033c9e1359391f4b61f278e5c], count=4271, sum=342332))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[087aa3e650be9e1b897cb8a7926fafde85c0491a52b5268284ee405d38baa2a1], count=430, sum=34459))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[0d927909305aae61be3a3f80f1eee40894a8910f07bad3c66843fcbb60c5cf1d], count=1289, sum=103275))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[5226ae5de5d883845f2fded38618cd29ea3cc4e28a263dd115149633514179b7], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=423, sum=33900))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[fa8f8dd534ab83fd9a9ad352d7e63eadce07e3d4d95e846b6690a53e50f046e3], count=2571, sum=206007))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[5eca943d71f416b6111de36193238b60e1307591f4738c55b6e0b1e111dd3d53], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[bb11bd1cd03ee81736b06f3808f4d84c6c75435f9f10dd0cd94f5b1a39ff66e0], count=864, sum=69234))
11: Child
12: Child)
}
}
}
}
}
PHYS101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[dd04eaab3acc2eb21101895d29f716fcc264adec3ee3c3d0828b68b2b1efb6a1]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 1508, 84598)), HASH[80f59d6ce839fd72da96b8d1b228172a9e80f4df9f8f9e078a87224943b8f816]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[4b85291fe8e5cae442614096553956521bb55510873f56ea4219d9a56d01408d], left=HASH[49700ad27bc9ac383b5bb5e867114f76acf4701ad7ea97311e12e877e92ffda9], right=HASH[5e9ff5741419a71daf0163e97389cf154aaea23144a589417a4a76e8df5904b4], count=455, sum=25572))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[fe6c93990c99748cec859c40fe458a191825c0941cd064d18eeaec17e52e0999], count=1508, sum=84598))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[ba3ae5cda415f7540cec982bd8403174f8b80ce2604463c630b6505692c4f0e4], count=147, sum=8114))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[e19d566cffe1e46af9eabb5b3b040898109a1d9d50a6a1bf87a04f421fd2617b], count=457, sum=25605))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[38be6684aff1201eeec45aedd505d3634fbcda546b158c5ae43e116819f2ea22], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=153, sum=8588))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[06ce5728c482b63134cf57461e9c4248638064a94393f9085842c830ae830381], count=906, sum=50770))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[8c2d715e4be3e576f5c4327d93f37192f71de69c46aa162ab9bcb725d53a3a46], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[de29287042c9ee0874e346cac4e947177e24129a9ec250bb1a31a575da222747], count=302, sum=16922))
11: Child
12: Child)
}
}
}
}
}
SOCI101 => {
LayerProof {
proof: Merk(
0: Push(Hash(HASH[0d8cfa7c4466e01ca73d4f8ea0615a960ed422fd27b5b80c20dbc0ce854c7390]))
1: Push(KVValueHash(semester, NotCountedOrSummed(ProvableCountProvableSumTree(8000000000004eeb, 3514, 274771)), HASH[728c9ce73d533772a6e429bbd6319d054bf1e0b7367c3557f02b962911e49665]))
2: Parent)
lower_layers: {
semester => {
LayerProof {
proof: Merk(
0: Push(HashWithCountAndSum(kv_hash=HASH[5e787086a1b267e213e6c64ce21941786a5f2daf6979093884d1923c5110bae8], left=HASH[6f680062b47d1ea1a4826c8323ed0cc398f06a2a8132efbe640ee4504a20d6a5], right=HASH[2098e76b4f886cb4849224d2a6c37f5aba69265f21047475c1d0df6314e9fe05], count=1057, sum=82667))
1: Push(KVDigestCountSum(0x8000000000004eeb, HASH[307bbb9fac832c48ed1147e5d4ab751868c9faf1ab56c1c27f2813f4b27e83de], count=3514, sum=274771))
2: Parent
3: Push(KVDigestCountSum(0x8000000000004eec, HASH[1c92eaceff0d4774119c8cab00d0b8bc313306efbbe62b29b52f724e2ffaa21f], count=346, sum=27033))
4: Push(KVDigestCountSum(0x8000000000004eed, HASH[97a045b4732eb325ea99f0583ac7643eb071320c077d7701e2828ba7342c782e], count=1056, sum=82592))
5: Parent
6: Push(HashWithCountAndSum(kv_hash=HASH[3203e7c385d9ce04e4f1c4c83861eaa5ba9a15df2aa138a21d738cc1cefb532b], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[0000000000000000000000000000000000000000000000000000000000000000], count=355, sum=27805))
7: Child
8: Push(KVDigestCountSum(0x8000000000004eef, HASH[9e96b671df5054d6e040c859fe734c850e8101dde071e325a015af45871b2cca], count=2105, sum=164607))
9: Parent
10: Push(HashWithCountAndSum(kv_hash=HASH[7afdf59e0efd5e0695f40a1f1946b336e2304a52b08f5707f36e35f35f5fe6c3], left=HASH[0000000000000000000000000000000000000000000000000000000000000000], right=HASH[21de4684adc3f1fcb126db75204aef9cbdb55ae381c3e37f511e7449b97acffa], count=695, sum=54406))
11: Child
12: Child)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Diagram: per-layer merk-tree structure
See the GroveDB Layout diagram for the overall storage shape. Q7's descent walks the proof AST above through the layers highlighted there — green nodes are CountSumTree terminators carrying both count_value and sum_value, yellow nodes are ProvableCountProvableSumTree (PCPS) terminators with per-node count + sum, gray are opaque sibling subtrees the proof commits only via hash. Q7 fans out at the class layer to 10 cyan class-name terminators (one per outer In branch). Under each terminator, the inner subquery walks the byClassSemester continuation's semester (yellow PCPS) and emits an AggregateCountAndSumOnRange collapse for that class's in-range semesters. Each bucket emits one (count, sum) pair. The path is byte-identical to the prover's path query, which is why prover and verifier agree on the root hash 8b15f732…ffc7.
This is the chapter's most expressive primitive: one proof, k cryptographically-committed (count, sum) triples, each describing a different class's semester-trend average. The client divides per bucket to get k verified per-class averages. Doing this without the PCPS carrier would require k × 2 independent proofs (one count, one sum per class) plus k root-hash matches the client must verify — the carrier collapses that to one proof, one root-hash, and roughly 1/3 to 1/2 the byte cost.
The two carrier-aggregate gates worth knowing (same as the sum chapter's Query 9):
SizedQuery::limitcaps the outer walk. Mismatched limits between prover and verifier break the merk-root recomputation.SizedQuery::offsetis rejected for carrier-aggregate. Skipping outer matches changes which(outer_key, count, sum)triples end up in the proof; the use case isn't designed yet.
Numerical Considerations
A few facts about how Drive handles the arithmetic:
- Count is
u64, sum isi64. Reflects grovedb's per-node field types:count_valueis unsigned (can't be negative),sum_valueis signed (negative contributions are allowed in general, though the grades contract'sscore >= 0constraint prevents them here). - No server-side division. The verifier returns
(count, sum); the client divides. This is deliberate: integer division loses precision (the average of[1, 2]is1.5, but3 / 2 = 1in integer math), and the right division precision depends on the client's use case (some want truncated integer, some want fixed-point, some want floating-point). The server doesn't pick for you. - Overflow risk for
sum. Each grade contributes at most 100 to the sum (per the schema'sscoreconstraint). For 10 000 grades the maximum sum is 1 000 000 — well withini64::MAX(~9.2 × 10¹⁸). The contract'smaxItemsconstraints onstudentandinstructorcap document size; combined with grovedb's max-tree-size policies, you'd need on the order of 10¹⁶ grades to risk i64 overflow on the sum. For any realistic deployment, overflow is not a concern. - Division by zero when count = 0. Possible if the filter resolves to no matches (e.g., a class no one has taken yet in the requested range). The client must handle the zero-count case explicitly — typically by reporting "no grades" rather than computing
sum / 0. The proof is still well-formed; it just commits(count=0, sum=0)and the verifier returns those values cleanly.
At-a-Glance Comparison
| Query | Index used | Element shape at terminator | Returned variant | Proof primitive |
|---|---|---|---|---|
| 1 — Global Average | (doctype primary-key) | CountSumTree at grade/[0] | (count, sum) | merk path |
| 2 — Average for class | byClass | CountSumTree at class/PHYS101 | (count, sum) | merk path |
| 3 — Student GPA | byStudent | CountSumTree at student/student_050 | (count, sum) | merk path |
| 4 — One Cohort | byClassSemester | ProvableCountProvableSumTree at class/PHYS101/semester/serialize(20204) | (count, sum) | merk path |
| 5 — Class Trend | byClassSemester (PCPS continuation) | (collapsed boundary) | (count, sum) | AggregateCountAndSumOnRange |
| 6 — Per-Student in Semester | byStudentSemester (point inner) | k × CountSumTrees | per-key entries | CountSumTree-carrier (k × merk path) |
| 7 — Per-Class Trends | byClassSemester (PCPS continuation) | k × (collapsed boundaries) | per-key entries | verify_aggregate_count_and_sum_query_per_key (PCPS carrier) |
The split closely parallels the count and sum chapters — point lookups for Q1–Q4, range-aggregate for Q5, carrier composition for Q6–Q7 — with the load-bearing difference that every query's returned shape carries both a count and a sum. The dual-axis primitive surfaces a payload ((count, sum)) that neither the count nor the sum chapter alone can produce in a single proof; the client computes avg = sum / count and gets a cryptographically-attested verified average from one root-hash commit.
What's Next
The chapter is grounded in the document_average_worst_case bench's measured numbers — Q1–Q7 verify cleanly end-to-end against the shared root hash 8b15f732…ffc7.
A natural expansion follow-up (out of scope here): a worked example of "exact-precision" averages — for callers that need fractional averages (e.g. avg = 50.7142857… rather than 50.99), the protocol-level approach is to return (count, sum) and let the client compute in its preferred numeric format (the chapter notes this in Numerical Considerations above; a future expansion could walk through the fixed-point vs. floating-point trade-offs).
No-proof path: joint count-and-sum dispatch
The no-proof AVG path lives in crate::query::drive_document_count_and_sum_query. It consumes the same DocumentAverageRequest the prove path uses and resolves routing through sum's versioned mode-detection table, so the (where_clauses × mode) → executor mapping has a single source of truth shared with the sum and count surfaces. The dispatcher splits on the resolved mode:
Total/PerInValue(no-rangeEqual/Inon asummable + countableindex) walks the point-lookup path query and decodes(count, sum)from each visitedCountSumTreeterminator in one call viaElement::count_sum_value_or_default(). One grovedb call perInbranch, both metrics together.RangeNoProofdistinct shapes (GroupByRange/GroupByCompound+ range on an index that declares BOTHrangeCountable: trueANDrangeSummable: true— DPP exposesrangeAverageable: trueas shorthand for the pair) walkProvableCountProvableSumTreeterminators once via the samedistinct_sum_path_querybuilder the sum surface uses, emitting one(count, sum)per distinct in-range key — strictly better than the count + sum surfaces' parallel walks because both metrics come from each visited element.RangeNoProofaggregate shapes (Aggregate/GroupByIn+ range) call grovedb's combined merk-internal accumulator directly:query_aggregate_count_and_sumagainst the PCPS path query, yielding(u64, i64)from a single O(log n) traversal. CompoundIn + rangeper-In fans out (≤100 branches per theIn::in_values()validator cap) and issues one accumulator call per branch under a shared read transaction. Bounded regardless of how many documents the range matches — keeping the public DAPI endpoint closed against amplification.
The no-prove combined accumulator (query_aggregate_count_and_sum) is the symmetric counterpart of the prove-side AggregateCountAndSumOnRange primitive Q5 / Q7 above use. Both sides walk the same PCPS terminator shape with (u128, i128) accumulators (narrowing to (u64, i64) at the entry point) — the only difference is that the no-prove path returns the pair directly while the prove path emits proof bytes the client verifies via GroveDb::verify_aggregate_count_and_sum_query.
Document Ranked Trees
The aggregate surfaces that landed in Platform 4.0 answer "how many" and "how much" — per document type, per indexed value, per range of indexed values. What none of them answer is "which groups score highest". Counting the reviews of every restaurant is O(log n) per restaurant; finding the five best-rated restaurants meant enumerating all of them and sorting client-side, with a proof that grew with the number of restaurants rather than with the number you asked for.
From protocol v14 an index can declare that its groups are rankable by an aggregate. The terminal tree upgrades to grovedb's indexed-tree family (grovedb PR #657), which carries an ordered secondary Merk per declared ranking axis, and "top 5 restaurants by average grade" becomes an O(log n + k) read with an O(log n + k) proof. This chapter explains the three indexed tree variants, how an index opts into one, how the secondaries are keyed and maintained, and how the feature composes with the other v14 change. The Ranked Index Examples chapter is the worked-example companion.
The chapter assumes you've read Document Count Trees, Document Sum Trees, and the Average Index Examples chapter. Ranked trees are built directly on top of the range-aggregate layouts those chapters describe: a ranking secondary is an ordering over aggregates the range layout already maintains, so the range axis is a hard prerequisite for the ranked one.
Status: implemented and gated at protocol version 14. The storage layout is pinned end-to-end against a real grovedb by
ranked_index_e2e_tests, which runs against the restaurants fixture atpackages/rs-drive/tests/supporting_files/contract/restaurants/restaurants-contract.json— the same fixture the examples chapter walks through. The grovedb-side design lives in that project's book chapter The CountIndexedTree (docs/book/src/count-indexed-tree.md), which is the authoritative reference for the element layout, the hash composition, and the secondary-Merk storage prefixes summarised here.
Why Ranking Needs a New Primitive
A ProvableCountTree at the property-name level stores, per group, the group's document count — and stores it in a Merk keyed by the group key. That is exactly what you want for "how many reviews does restaurant alpha have?" and for "how many reviews do restaurants between alpha and mike have?": both questions are answered by walking a key-ordered boundary.
"Which five restaurants have the most reviews?" is a question about the aggregate, not about the key. Nothing in the key-ordered Merk correlates position with count, so the only honest answer is to visit every group. That is O(n) work and — worse — an O(n) proof, even though the answer is five entries. Three options exist:
- Enumerate and sort client-side. Correct, and the only thing 4.0 could do. Proof size scales with the number of groups, not with
k; a contract with 50 000 restaurants pays 50 000 committed entries to learn about five. - Have the server sort and return the top five. O(1) bytes, zero cryptographic value — the client has no way to check that the sixth-best restaurant wasn't quietly omitted.
- Maintain a second, aggregate-ordered view of the same groups, committed to the same root hash. Top-k becomes a bounded range read at one end of that view, and its proof is a standard Merk range proof over
kentries. This is what this chapter is about.
The grovedb primitive is the indexed tree: a primary Merk that is a byte-compatible mirror of the tree it replaces, plus one ordered secondary Merk per declared axis, keyed so that the aggregate sorts first and the group key only breaks ties. The primary is byte-compatible, but grovedb's range aggregates (AggregateCountOnRange, AggregateSumOnRange, AggregateCountAndSumOnRange) take provable trees only and do not prove through an indexed tree, so Drive refuses a range total through any index whose path passes through a ranked level, its own or another index's at a level the two share (refuse_a_range_total_through_a_ranked_index); per-value range reads keep working.
The dashed box is the wrapping Element. The primary Merk (blue) is keyed by group key and is byte-identical to the ProvableCountTree it replaces. The secondary (orange) holds one entry per group, keyed by count_be ‖ group_key, so a descending walk of its right edge yields the highest-count groups first.
flowchart LR
subgraph PCIT ["ProvableCountIndexedTree"]
direction TB
subgraph ELEM ["Tree element c=9"]
direction TB
P["primary root<br/>(keyed by group key)"]:::primary
S["secondary root<br/>(keyed by count ‖ group key)"]:::secondary
end
end
P --> PA["alpha c=2"]:::leaf
P --> PM["mike c=3"]:::leaf
P --> PZ["zulu c=4"]:::leaf
S --> SA["0x…02 ‖ alpha"]:::leaf
S --> SM["0x…03 ‖ mike"]:::leaf
S --> SZ["0x…04 ‖ zulu"]:::leaf
classDef primary fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef secondary fill:#fb8500,color:#0d1117,stroke:#fb8500,stroke-width:2px;
classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;
style ELEM fill:none,stroke:#1f6feb,stroke-width:2px,stroke-dasharray: 6 4,color:#1f6feb
ORDER BY … DESC LIMIT 2 reads the two right-most secondary entries (zulu, mike) and proves them with a standard Merk range proof — three levels of hashes, two committed entries. The same question against the primary alone would have to commit all nine groups.
Contract Grammar
Ranking is an index-level opt-in. Three independent keywords, one per axis:
| Keyword | Ranks groups by | Requires (in effect) | Rust field |
|---|---|---|---|
rankedCountable | each group's document count | rangeCountable: true | Index::ranked_countable / Index::ranked_countable_at |
rankedSummable | each group's sum of the summable property | rangeSummable: true | Index::ranked_summable |
rankedAverageable | each group's average of the averageable property | rangeAverageable semantics — both rangeCountable and rangeSummable | Index::ranked_averageable |
The three axes are independent opt-ins. rankedAverageable is not sugar for the other two, unlike averageable / rangeAverageable, which genuinely are sugar for their count+sum longhand. Each ranking axis costs its own ordered secondary Merk and its own maintenance on every write, so each is declared explicitly. Declaring rankedAverageable alone is a legal — and usually the right — choice.
rankedCountable alone has a second, level-addressed spelling. The boolean form ranks the terminal level's groups by direct member count; the object form names the level (or levels) the Count axis aggregates at:
"rankedCountable": true // terminal groups by member count
"rankedCountable": { "at": "hashtag" } // hashtags by WHOLE-SUBTREE count
"rankedCountable": { "at": ["hashtag", "postId"] } // both rankings on one index
"rankedCountable": { "at": ["tag", "region", "postId"] } // every level of a 3-property index
at takes one property name or an array of them (each must be an index property; duplicates are rejected; the array is capped at 10 — the index property cap). Naming the last property is the boolean form spelled longhand and folds into it; every other name places a ranking at that prefix level, ordering its values by their whole-subtree document count — on [hashtag, postId], { "at": "hashtag" } ranks hashtags by total count across all their posts, where the boolean form ranks a pinned hashtag's posts. Any subset of levels may be named, and the parser stores the non-terminal ones in Index::ranked_countable_at in index-property order regardless of how the contract spelled them. The object form cannot be combined with rankedSummable / rankedAverageable: the chain of count-bearing levels a prefix ranking is built from (see Prefix-Level and Multi-Level Rankings) cannot carry a sum axis.
The range prerequisite is a hard requirement, not a convenience: a ranking secondary orders the per-group aggregates the range layout already maintains. Without the range axis the terminal property-name tree carries no per-group aggregate to sort by. The parser rejects the combination with a message naming the missing flag:
rankedCountable requires rangeCountable: true; ranking groups by count needs
the per-group counts the range-count layout maintains
Value-Sensitive Prerequisites in the Meta-Schema
The document meta-schema enforces the same prerequisites, in two forms. The range* rows are presence rules: a rangeSummable key, whatever its value, needs a summable (or an averageable) key beside it, and a rangeAverageable key needs an averageable. rangeCountable has no row: it implies countable, exactly as the doctype-level rangeCountable implies documentsCountable, and the parser promotes an omitted countable to "countable" (an explicit "countableAllowingOffset" is kept, an explicit "notCountable" is rejected as a contradiction). The ranked rules are value-sensitive if / then pairs, because a written-out opt-out, "rankedCountable": false, which the structural parser accepts as exactly that, must not be made to demand a rangeCountable the index does not need:
{
"if": {
"properties": {
"rankedCountable": { "anyOf": [{ "const": true }, { "type": "object" }] }
},
"required": ["rankedCountable"],
"not": { "required": ["rangeAverageable"] }
},
"then": { "required": ["rangeCountable"] }
}
(The rankedCountable conditional matches the object form as well as the literal true; both spellings need a range axis, and only a written-out false escapes the requirement.)
Every rule is sugar-aware, and the not clause above is what makes it so. JSON-schema validation runs over the index object exactly as authored, before averageable / rangeAverageable are expanded into their countable + summable longhand, so each rule spells the sugar out as an accepted alternative: rangeSummable accepts averageable in place of summable, rankedCountable and rankedSummable accept rangeAverageable in place of their own range axis, and rankedAverageable accepts the explicit rangeCountable + rangeSummable pair in place of rangeAverageable. The doctype-level rangeSummable row accepts documentsAverageable the same way. The parser then checks the same prerequisites on the resolved flags, so the two layers agree on every spelling, and the sugar form of a multi-axis index is the whole declaration:
{"name": "storeRating", "properties": [{"storeId": "asc"}],
"averageable": "rating", "rangeAverageable": true,
"rankedAverageable": true, "rankedCountable": true}
Up to 4.2.0-beta.1, meta-schema v3 demanded the literal key instead: this index failed registration with "rangeCountable" is a required property, and adding rangeCountable then pulled in a literal countable through a presence row, so the only accepted spellings carried keys the sugar already implied. The same release also made the index-level parser demand an explicit countable beside rangeCountable, while the doctype level had always treated rangeCountable as implying documentsCountable. v3 is editable until 4.2 is live on mainnet, so both rules were corrected in place rather than carried into a v4; below protocol version 14 nothing moves.
The two layers are not gated alike. The structural parser runs on every parse, full_validation or not, and is compiled into every build. The JSON-schema layer runs only under full_validation, and only in builds with rs-dpp's validation feature, which wasm-dpp2 (and so @dashevo/evo-sdk) does not enable. An SDK-side DataContract.fromJSON(json, true, pv) therefore sees the parser's verdict alone, which is why the two layers agreeing on every spelling matters: it is what keeps an offline acceptance from turning into a registration refusal.
Shape Restrictions
Two structural rules, both enforced at contract-parse time in rs-dpp:
-
No aggregating index on a compound ranked index's full prefix. Ranked flags are allowed on compound indexes, with per-prefix semantics: a ranked
[identityId, class]puts the indexed tree at each prefix value's terminalclassproperty-name level — one ordered secondary peridentityId, each ranking only that identity'sclassgroups. There is deliberately no global cross-prefix ordering; the query surfaces require every leading property to be pinned by awhereclause — equalities, plus at most oneINthat fans out across prefix branches and merges deterministically. The one shape that stays impossible — and is rejected per document type, where all indexes are visible (validate_no_ranked_prefix_overlap) — is a countable/summable index terminating at exactly the compound's leading prefix: its aggregating value trees would demand theNonCounted/NotSummedshell around the ranked terminal tree, and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralise the very aggregates the secondaries order by. (Drive's fail-closed guard behind the parse-time check isINDEXED_INNER_UNWRAPPABLE.) Only the exactn-1prefix conflicts: an aggregating index at a shorter prefix wraps a plain intermediate tree, and one extending past the ranked terminal lives inside its value trees — both supported. -
Non-unique indexes only.
ranked aggregates are not supported on unique indexes: each group of a unique index contains at most one document, so there is nothing meaningful to rank. Contested indexes are covered transitively — a contested index is unique by construction, so it hits the same check rather than needing its own. -
Prefix-level rankings add two rules of their own, both anchored on the index's shallowest ranked prefix level (every deeper ranked level sits inside its exclusive range). From that level down, the index's levels form one count-propagation chain, and the rule set over other indexes reaching the
atlevel is a shape matrix:Other index S(vs. ranking atpkon[p1 … pn])Verdict Diverges before pk(level key differs at some position< k)✔ never conflicted Terminates above the atlevel, no countable/summable flags✔ Aggregating, terminates at a prefix shorter than [p1 … p(k-1)]✔ wraps a plain intermediate tree Plain (no countable/summable/ranked flags, nothing contested), continues below the atlevel, and branches off the chain — by diverging belowator extending past the ranked terminal✔ laid out count-exempt — its branch trees are Element::NonCountedinside the chain's value treesContinues below the atlevel but is itself countable/summable/ranked✘ its aggregates would need the counts the wrapper suppresses Terminates exactly at the atlevel (any flags)✘ its member bucket and aggregates would sit on the grouping tree itself Plain, continues below atbut never leaves the chain (a level-key prefix of[p1 … pn])✘ it has no branch to wrap — its member bucket would land on a grouping/propagating level, the terminal-plus-chain stamp the tree-type resolver fails closed on Aggregating, terminates at exactly [p1 … p(k-1)](the prefix directly aboveat)✘ the grouping tree would need a NonCountedshell inside aggregating value trees — the wrapped-indexed impossibility, mirroring the first rule aboveThe count-exempt row is the same demotion range-countable value trees already apply to their sibling continuations, one configuration over: the sibling's branch reads and proves normally, contributes zero to every subtree total the ranking keys on, and its writes skip the ranked re-key entirely. The derivation stamps the branch level (
IndexLevel::count_exempt_branch) so every write path agrees on the wrapper. Level identity for all of these comparisons isIndex::level_key, not the declared name: a time-range index's grid-qualified first key forks it into a sibling subtree, so a bucketed index whose declared properties match a ranked index's does not conflict.
Version Gate
The grammar activates with protocol version 14 through two independent gates that must agree:
CONTRACT_VERSIONS_V6pointsdocument_type_schemaat the v3 document meta-schema, which hosts the three keywords. v13 keeps validating against v2, where they fail an index entry'sadditionalProperties: false.try_from_schema: 3selects parser generation 3, which is the only generation that passesranked_aggregates_allowed = trueintoIndex::try_from_value_map. With that flag off, the three keywords are not part of the grammar at all: they fall through to the unknown-key arm and are rejected with exactly the error a pre-v14 node produced for them.
The doubled gate is load-bearing. The meta-schema only runs under full_validation, so a non-validating parse — check_tx, contract-cache warm-up, state restore — could otherwise smuggle a ranked index past a node whose protocol version has no idea how to lay one out on disk.
On the parser-generation pattern. Generation 3 is a full copy of generation 2 (and of the generation-1 core that wrapper delegates to), not a version gate threaded into those modules. That is the repository's standing rule for grammar introduced by a new protocol version: shipped generations stay byte-identical to the code consensus already ran, so replaying a historical block can never pick up grammar that did not exist when the block was produced. The copy is kept structurally line-for-line with its sources, so a diff against v1/mod.rs + v2/mod.rs shows only the ranked deltas.
How Drive Picks the Indexed Tree Variant
An index's terminal property-name tree is the level whose children are the last index property's value trees — one child per distinct value, i.e. one child per group. Without ranking flags its TreeType comes purely from the range flags:
range_countable | range_summable | Base tree type |
|---|---|---|
true | true | ProvableCountProvableSumTree |
true | false | ProvableCountTree |
false | true | ProvableSumTree |
false | false | NormalTree |
A ranking flag upgrades that base to its indexed mirror:
| Base | Declared axes | Indexed tree |
|---|---|---|
| any | [] | unchanged (no ranking declared) |
ProvableCountTree | [Count] | ProvableCountIndexedTree |
ProvableSumTree | [Sum] | ProvableSumIndexedTree |
ProvableCountProvableSumTree | any non-empty | ProvableCountProvableSumIndexedTree(axes) |
The single-axis variants (PCIT / PSIT) carry no axis list on the element at all — their one secondary is implied by the variant — which is why they are only reachable when the base layout is already single-axis. An index that declares, say, rankedCountable alongside rangeSummable still lays out as ProvableCountProvableSumTree underneath, so it upgrades to the multi-axis PCPSIT carrying just the Count axis in its TLV.
The axes list is canonical: sorted by tag (0 = Count, 1 = Sum, 2 = Avg), deduped, one to three entries. Freshly created, the element's TLV is [(tag, None), …] — every secondary starts empty. grovedb validates the list on construction (Element::validate_pcpsit_axes), because an out-of-order, duplicated or empty TLV would still be hashed into the parent and produce a tree whose secondaries nothing can address.
Selection lives in packages/rs-drive/src/drive/document/ranked_index_tree_type.rs:
#![allow(unused)] fn main() { pub(crate) fn ranked_property_name_tree_type( base: TreeType, ranked_axes: &[IndexAxis], ) -> Result<TreeType, Error> { if ranked_axes.is_empty() { return Ok(base); } match (base, ranked_axes) { (TreeType::ProvableCountTree, [IndexAxis::Count]) => Ok(TreeType::ProvableCountIndexedTree), (TreeType::ProvableSumTree, [IndexAxis::Sum]) => Ok(TreeType::ProvableSumIndexedTree), (TreeType::ProvableCountProvableSumTree, _) => { Ok(TreeType::ProvableCountProvableSumIndexedTree) } _ => Err(Error::Drive(DriveError::CorruptedContractIndexes(/* … */))), } } }
The catch-all arm is unreachable given rs-dpp's parse-time invariants (ranked_countable ⇒ range_countable, ranked_summable ⇒ range_summable, ranked_averageable ⇒ both), which force base to be provable on every axis a ranking flag names. It is a typed error rather than a silent fallback so a future grammar change that breaks one of those implications surfaces here instead of quietly laying down a tree whose secondaries nothing maintains.
property_name_tree_type_and_ranked_axes() is the single source of truth — contract registration, contract update, the document insert / update / delete index walkers, and the cost-estimation layers all route through it, so the layouts they describe cannot drift apart. The contract insert/update paths reach grovedb through three helpers parallel to the count and sum families:
batch_insert_empty_provable_count_indexed_tree— PCIT.batch_insert_empty_provable_sum_indexed_tree— PSIT.batch_insert_empty_provable_count_provable_sum_indexed_tree— PCPSIT, taking the axes TLV.
The groups keep their ordinary value-tree types. An indexed primary is a mirror of the tree it replaces, so nothing changes one level down: with the restaurants fixture, review's groups are ProvableCountProvableSumTrees, visit's are CountTrees, and tip's are SumTrees — exactly the shapes the range flags alone would have produced.
The Secondary Merks
Each declared axis gets one ordered secondary Merk holding one entry per group, keyed by
axis_sort_key ‖ group_key
so the aggregate sorts first and the group key breaks ties. Ties therefore come back in group-key order in the direction of the walk — descending group-key order for TOP, ascending for BOTTOM.
| Axis | Sort key | Width | Encoding |
|---|---|---|---|
| Count | count | 8 B | big-endian u64 |
| Sum | sum | 8 B | big-endian i64 with the sign bit flipped |
| Avg | floor(sum × SCALE / count) | 16 B | big-endian i128 with the sign bit flipped |
Sign-bit toggling is what makes the signed encoders order-preserving: flipping bit 63 of an i64 (or bit 127 of an i128) maps the two's-complement signed range onto an unsigned range with the same ordering, so plain byte comparison sorts negatives below positives.
The Avg Fixed-Point Sort Key
There is no fractional key type, so the Avg axis sorts by a fixed-point integer:
#![allow(unused)] fn main() { pub fn compute_avg_fixed_point(sum: i64, count: u64) -> i128 { if count == 0 { return 0; } (sum as i128) .saturating_mul(AVG_FIXED_POINT_SCALE) .div_euclid(count as i128) } }
Three properties to internalise:
- The scale is grovedb's constant, not a wire constant.
AVG_FIXED_POINT_SCALEis10^19today; it moved from10^15before release. Drive re-exports it (drive::query::RANKED_AVG_SCALE, itself re-exported bydrive_proof_verifier::RANKED_AVG_SCALE) rather than re-declaring it, so the two can never drift — the encoded sort keys in storage are produced with grovedb's constant, and a platform-side copy that fell out of step would silently mis-scale every average a client renders. Never hardcode the literal. - Division is euclidean, i.e. floor toward −∞. Rust's
/truncates signed integers toward zero, which would place negative averages one fixed-point bucket too high. The restaurants fixture carries a whole document type (adjustment, whosedeltaadmits negative values) for the sole purpose of exercising signed sums and this rounding mode. 0 / 0is defined as0. An empty group has no entry in the secondary at all, so this is a defensive definition rather than an observable one.
Maintenance
Secondaries are maintained through the normal batch write path — there is no separate reindex step and no background job. Every document insert, update and delete that changes a group's (count, sum) also rewrites that group's entry in each declared secondary: the old (sort_key ‖ group_key) entry is removed and the new one inserted, in the same grovedb batch, under the same block. Draining a group's last document removes the group from the primary and its entry from every secondary together.
grovedb's own consistency verifier enforces the invariant directly: every primary entry at key must have exactly one secondary entry at make_axis_secondary_key(axis, count, sum, key), and the secondary must contain nothing else.
Prefix-Level and Multi-Level Rankings
The boolean axes always place their secondaries on the terminal property-name tree. rankedCountable's at form places Count secondaries at any subset of the index's levels, and the storage answer to "what does a prefix level rank by?" is a count-propagation chain:
"hashtag" property-name tree ProvableCountIndexedTree ← ranked level
<value> value tree CountTree (count = subtree total)
"postId" property-name tree ProvableCountTree — contributes its count
<value> value tree CountTree
[0] member bucket counted references / Items
From the shallowest ranked level down to the terminal, every level is count-bearing, and — unlike the shared-prefix continuation case, where a sibling continuation is zero-wrapped — each level's chain continuation is inserted contributing, so its count IS the value tree's count. Every write's delta then propagates up the chain, and grovedb re-keys each ranked level's secondary entry on the way — one secondary rewrite per ranked level on the leaf-to-root path, nothing more.
A plain sibling index sharing the at level (see the shape matrix above) hangs its branch inside the same value trees, but zero-wrapped:
"postAuthor" property-name tree ProvableCountIndexedTree ← ranked level
<alice> value tree CountTree, count = 3 (alice's real likes, exact)
"postId" property-name tree ProvableCountTree ← chain: contributes, re-keys ranking
<post7> … (3 entries)
"$createdAt" property-name tree NonCounted(Tree) ← sibling: readable, count-invisible,
<ts> … (3 entries) writes skip the ranked re-key
The wrapped branch is a normal readable subtree — equality-prefix and range reads through it serve and prove exactly as under any merged index — while contributing zero to the value tree's count, so the ranking totals stay exact with the sibling fully populated. Sibling-only writes (an entry landing under the branch with the chain tuple unchanged) touch neither the counts nor the secondaries. A level named in at becomes the Count-axis indexed tree; an unnamed level between the shallowest ranked one and the terminal becomes a plain CountTree pair (count-propagating); the terminal keeps the layout its own flags give it, which is why the at form requires rangeCountable — that is what makes the terminal property-name tree count-bearing enough to feed the chain. When the terminal is also ranked (the boolean alongside at), the chain simply nests indexed trees: the terminal ProvableCountIndexedTree is itself the contributing child, and both secondaries maintain simultaneously — this composes at any depth, so a fully ranked index re-keys one secondary per level per write.
The per-level stamps live on IndexLevel (ranked_count_grouping / count_propagating), set by the index-level derivation and read by the same tree-type resolver every write path shares. An index ranked at its first property gets its indexed tree at contract registration (it is a top-level tree); deeper ranked levels materialize lazily per prefix, like every dynamic index tree, and drained chains prune out of their secondaries (or stay rankable at zero under preallocated).
On the query side each level serves its own ranking, addressed by the (group property, pin count) pair: GROUP BY names the ranked level's property, and every property before it must be pinned — properties after it never appear in the request at all, since they are interior to the subtrees being counted. { "at": ["tag", "region", "postId"] } therefore serves GROUP BY tag with nothing pinned (the global ranking), GROUP BY region with tag pinned, and GROUP BY postId with both pinned, each from its own secondary with its own proof.
Hash Composition
An indexed tree commits two Merks in one element. The composition grovedb uses — internally called H1-A — is a single three-input Blake3 call:
combined_value_hash = combine_hash_three(value_hash, primary_root_hash, axes_digest)
For the single-axis variants the third input is simply that axis's root hash. For PCPSIT it is axes_digest, a length-prefixed Blake3 over the canonical axes TLV — [n] followed by (tag, root_hash) per axis. The length prefix is what distinguishes a one-axis digest from a two-axis digest truncated to a single entry. An axis with no entries yet contributes NULL_HASH in its slot.
The three-input form is deliberate and regression-tested: it must not be equivalent to combine_hash(a, combine_hash(b, c)), the nested composition rejected during design review because it would have doubled the hash work per bubble-up.
What this buys the verifier: the parent's value_hash binds the primary root and every secondary root. A proof of a top-k walk over one secondary reconstructs that secondary's root, recombines it with the primary root the same way the writer did, and the result has to match what the parent committed. A server cannot serve a stale or forged secondary while presenting an honest primary.
The Grove Path
Every ranked read — and, on the prove path, every ranked proof — is issued against the path of the terminal property-name tree. For a single-property index that is:
[ RootTree::DataContractDocuments as u8 ] // 0x01
/ <contract_id: 32 bytes>
/ [ 0x01 ] // "documents", not "contract"
/ <document_type_name: utf-8> // e.g. b"review"
/ <last_index_property_name: utf-8> // e.g. b"restaurantId"
The children of that tree are the groups: one value tree per distinct value of the last index property, keyed by the raw index-key bytes of that value (for a string property, its UTF-8 bytes — e.g. b"alpha"). The secondary entries a top-k read returns are keyed by those same group keys. A compound index [a, b] inserts <a> / <encoded pinned value of a> between the doctype and the terminal <b> level — the value segment comes from the request's where pin on a — an equality pin, or one element of the single permitted IN (one branch per element) — encoded with the same serialize_value_for_key the write path used to key that prefix's value tree, so the walk lands on that prefix's own indexed tree and secondary.
Prover and verifier build this path through the same function, DriveDocumentRankedQuery::indexed_property_name_tree_path (with the pinned prefix values encoded by the shared resolver, resolve_ranked_query_for_mode), which is why they agree on the root hash by construction.
Write-Path Cost: The Grove v4 Cleanup Gates
Indexed trees need two batch behaviours that earlier grove versions don't have, so protocol v14 also moves Drive from GROVE_V3 to GROVE_V4:
delete_tree_cleanup_type_source— a batchDeleteTreereads the stored element and uses its actual type to select cleanup namespaces, rejecting a declared/stored mismatch that involves an indexed tree. V1–V3 take the declared type at face value.overwrite_indexed_cleanup_inspection— a batch overwrite of a non-reference element (with tree-override protection off) reads the stored element to detect an indexed tree being overwritten, scheduling its per-axis secondary storage for cleanup or refusing the ambiguous case.
Without them, a batch overwrite of a ranked index would orphan its per-axis secondary storage. Indexed trees only exist from protocol v14, so activating the stricter cleanup alongside them costs older versions nothing.
Both gates are cost-neutral: they derive the old element from data the merk apply already loads when it rewrites or deletes a key, so no extra stored-element read is charged and v14's fee constants match v13's. Three facts pin this:
- Fees are identical across the boundary. The identity-balance, token-balance and state-transition processing-fee pins carry the same values at protocol v13 and v14, and
run_chain_one_identity_in_solitudehas a protocol-version-13 sibling asserting the identical end balance — the pair proves the v13 → v14 transition changes nothing about those runs' fees. - Storage fees are untouched everywhere — the gates only observe, never write.
- Cleanup still happens. A batch overwrite of an indexed tree schedules its per-axis secondary storage for cleanup (or refuses the ambiguous case), and
DeleteTreeuses the actual stored type — covered by grovedb's own overwrite and delete-tree suites at the pinned revision.
Interaction With the Shared-Prefix Aggregate Fix
Protocol v14 hosts two consensus changes, and contracts exist where both apply to the same index. The shared-prefix aggregate fix makes a contract legal that previously registered but rejected every document insert: an aggregating index terminating at a property that is also the prefix of a compound index (e.g. summable [a] next to [a, b]).
The two changes are orthogonal by construction, and they live exactly one level apart:
- The ranked upgrade decides the property-name tree type — plain → indexed mirror.
- The continuation demotion decides the value tree type one level below it — a provable count-bearing value tree that has to host a compound continuation demotes to
CountSumTree, since grovedb rejects count-suppressed children under provable count parents by design.
A demoted CountSumTree value tree contributes its (count, sum) to a ranked indexed parent exactly as the provable variant did, so a ranked index's secondaries keep ranking correctly over shared-prefix shapes. Concretely, for a dish doctype with a ranked [restaurantId] index alongside a plain compound [restaurantId, chefId]:
- the property-name tree at
restaurantIdstays aProvableCountProvableSumIndexedTreecarrying the Avg axis; - each group's value tree demotes from
ProvableCountProvableSumTreetoCountSumTree; - the
chefIdcontinuation inside it goes inElement::NonCounted, contributing zero to the group's count and sum.
The one place the two changes genuinely collide is the case the prefix-overlap rule already forbids at contract-parse time: a ranked terminal level sitting inside an aggregating value tree would need a wrapper, and an indexed tree can never be wrapped. That is the INDEXED_INNER_UNWRAPPABLE guard, and it fails closed.
Storage-Layout Invariants
All three ranking flags are immutable across a contract update, for the same reason and with the same error as the count and sum flags. The set of declared axes picks the indexed tree variant and its ordered secondaries at contract creation; toggling any one of them would require rebuilding the secondaries for every existing group.
Rankings parse from protocol version 14, whose document type validate_update (v1) refuses any changed index by comparing whole Index definitions, so every ranking flag, at level and chain stamp (and a summableOffCountIndex index's source) is frozen with the rest of the index. IndexLevel::find_first_ranked_change, which walks the two index-level trees and returns the first path where ranked_countable, ranked_summable or ranked_averageable differs (e.g. restaurantId -> (ranked_averageable: false -> true)), serves the earlier validate_update (v0), which no contract with a ranking reaches. Either way the update fails with DataContractInvalidIndexDefinitionUpdateError:
Document with type {document_type} could not add or remove '{index_path}' during
data contract update as we do not allow modifications of data contract index paths
Adding a new ranked index on update is rejected by the same machinery. Don't relax these guards: an index whose declared axes disagree with the element on disk would have the write path maintaining secondaries the reader can't address, or the reader reading secondaries the write path never updates — consensus drift either way.
Authoring a Contract That Uses Ranked Trees
The restaurants fixture is the reference. Its review doctype ranks restaurants by average grade:
{
"review": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": true,
"indices": [
{
"name": "byRestaurant",
"properties": [{ "restaurantId": "asc" }],
"countable": "countable",
"summable": "grade",
"averageable": "grade",
"rangeCountable": true,
"rangeSummable": true,
"rangeAverageable": true,
"rankedAverageable": true
}
],
"properties": {
"restaurantId": { "type": "string", "minLength": 1, "maxLength": 32, "position": 0 },
"grade": { "type": "integer", "minimum": 0, "maximum": 100, "position": 1 }
},
"required": ["restaurantId", "grade"],
"additionalProperties": false
}
}
The countable / summable / averageable trio plus the three range* flags are the prerequisites; rankedAverageable: true is the one line that adds the ranking. That index's terminal property-name tree at restaurantId becomes a ProvableCountProvableSumIndexedTree carrying axes [Avg].
The count-only and sum-only shapes are correspondingly smaller:
{ "name": "byRestaurantVisits",
"properties": [{ "restaurantId": "asc" }],
"countable": "countable", "rangeCountable": true, "rankedCountable": true }
{ "name": "byRestaurantTips",
"properties": [{ "restaurantId": "asc" }],
"summable": "amount", "rangeSummable": true, "rankedSummable": true }
which lay down a ProvableCountIndexedTree and a ProvableSumIndexedTree respectively.
Note that the fixture puts each shape on its own document type. That's not an accident of style: two indexes over the same property set on one doctype is a DuplicateIndexError, so exercising all three variants needs three doctypes.
Choosing What to Set
| You want | Set |
|---|---|
| Top / bottom K groups by document count | rankedCountable: true on a non-unique index that already has countable + rangeCountable: true |
Top / bottom K prefix values by whole-subtree count ("top hashtags by total likes" on [hashtag, postId]) | rankedCountable: { "at": "hashtag" } on the same prerequisites. The named level's values rank by their entire subtree's document count; the levels below become the count-propagation chain. |
| Both of the above on one index — and/or more levels | rankedCountable: { "at": [...] } naming every level you want rankable (the last property = the boolean form). Each named level costs one secondary rewrite per write beneath it. |
| Top / bottom K groups by sum of a property | rankedSummable: true on an index with summable: "<prop>" + rangeSummable: true |
| Top / bottom K groups by average of a property | rankedAverageable: true on an index with averageable: "<prop>" + rangeAverageable: true (or the count+sum longhand) |
| Two rankings on one index (e.g. by count and by average) | Both keywords. The tree is a PCPSIT carrying both axes in its TLV; you pay one secondary Merk per axis on every write. |
A ranking filtered by another property (top 5 restaurants in London) | A compound ranked index with the filter property leading: [city, restaurantId] with the ranked flags. Each city gets its own secondary; the query pins the prefix with an equality where (WHERE city == "London" GROUP BY restaurantId ORDER BY <agg> DESC LIMIT 5). Equality pins select one prefix; at most one pin may be an IN (2..=10 distinct elements; a never-written element — or one whose deeper pinned path was never written — contributes an empty branch), which walks one secondary per element and merges by (aggregate, encoded prefix, group key), proved in one branched PathQuery envelope (shared ancestors proved once, per-element authenticated absence) — entries then carry in_key. A single == pin on a prefix no document has written yet (a timeRange window before its first document, a hashtag nobody has used) is likewise an authenticated empty page, at any OFFSET: GroveDB answers a single-path axis read over a path that does not exist with the traversal's empty result, the absence proved by the layers the walk emits rather than reported as an error. A single-element IN normalizes to the equality pin; a null pin stays legal on its own but cannot combine with an IN (null addresses its prefix through an empty path segment the branched proof cannot express); a non-zero OFFSET is rejected together with IN; and branched proofs are generated from committed state only. A range operator on the prefix stays rejected, and there is still no global cross-prefix ordering beyond that merge. |
| A ranking on a unique or contested index | Not available, and not meaningful: every group holds at most one document. |
| Range aggregates without ranking (the 4.0 surface) | Just the range* flags. Ranking is strictly additive — adding it never changes what a range query returns. |
| Nothing ranking-aware (default) | Don't set any ranked* flag. The terminal property-name tree keeps the type its range flags give it. |
Every ranking axis is paid for on every write that touches a group, not just on the reads that use it. Two axes on one index means two secondary rewrites per document insert. Opt into the axes you will actually rank by.
What a Ranked Query Looks Like
The query surface is deliberately narrow — one aggregate select, one group_by, one ORDER BY naming that select's aggregate, and a LIMIT (plus an optional OFFSET). Nothing else:
SELECT avg(grade) FROM review
GROUP BY restaurantId
ORDER BY avg(grade) DESC
LIMIT 3
That is ordinary SQL, deliberately. An earlier draft of this surface spelled the same question as HAVING avg(grade) IN TOP(3) — a TOP / BOTTOM primitive on the right of a HAVING clause. It was removed before release rather than deprecated: SQL already expresses "the n highest-scoring groups" with ORDER BY + LIMIT, and inventing a second, non-conformant spelling for it bought nothing but a grammar every client author would have to learn. HAVING is now purely a boolean per-group predicate, as in SQL.
The Ranked Index Examples chapter covers the wire shape, the SDK surface, the proof, the rank/offset semantics, and the full list of what is rejected and why.
Ranked Index Examples
This chapter walks through a representative contract and shows how ranked queries work on Drive. Every example uses the restaurants contract at packages/rs-drive/tests/supporting_files/contract/restaurants/restaurants-contract.json — the same fixture the write-path e2e suite and the query suite share.
The chapter assumes you've read Document Ranked Trees for the storage layout, and the Count / Sum / Average example chapters for the aggregate surfaces ranking builds on. Here we take the indexed-tree machinery as given and look at the query shape it enables: "which n groups score highest?"
Status: implemented and gated at protocol version 14. Unlike the count / sum / average chapters, this one is not backed by a worst-case bench — there is no ranked bench, so no measured proof sizes or timings appear below. Every value shown is instead taken from the end-to-end suite at
packages/rs-drive/src/query/drive_document_ranked_query/tests.rs, which runs prover and verifier against a live Drive and asserts the reconstructed root hash equals the database's own. Where a size is stated it is an asymptotic, not a measurement.
Why Ranking Is a Different Query Shape
Every other aggregate query in Drive walks value trees under a property-name tree and aggregates what it finds: AggregateCountOnRange sums per-subtree counts along a boundary, the average carrier fans out over an In and aggregates each branch. A ranked query never touches the value trees at all. The answer already exists, pre-sorted, in the axis secondary described in the previous chapter — the query is a bounded scan of one end of it.
Three consequences shape the entire API, and they are the reason this chapter's "what is rejected" table is longer than its query list:
- No
whereclauses. Ranked indexes are single-property, so there is no equality prefix to narrow; and awhereon the ranked property itself asks for a filtered ranking, which the secondary cannot express — it is sorted by aggregate, not by group key. - No
start_atcursor. A cursor names a document id, and document ids do not appear in a keyspace sorted by aggregate.LIMITandOFFSETare honoured — they are how the ranking is sized and paged; see Ranks and Offsets. - Entry order IS the ranking order. The executor returns entries in the order grovedb walked the secondary. Callers must not re-sort.
The rejections are rejections rather than silent ignores, on both the client and the server, because a ranked walk cannot honour them and silently answering a different question is worse than an error.
The grammar is plain SQL
A ranked query is spelled the way SQL already spells "the n highest-scoring groups":
SELECT avg(grade) FROM review
GROUP BY restaurantId
ORDER BY avg(grade) DESC
LIMIT 3
DESC is the "top n" reading, ASC the "bottom n" reading. LIMIT is the ranking's size; OFFSET moves the window down the ranking.
This replaced a non-SQL spelling that never shipped. An earlier draft put the ranking on the right of a HAVING clause — HAVING avg(grade) IN TOP(3), with TOP / BOTTOM / MAX / MIN as cross-group primitives. It was removed before release rather than deprecated. The deliberate call: SQL conformance beats a bespoke primitive. Every client author already knows ORDER BY … LIMIT; nobody knows IN TOP(n), and the two express exactly the same thing. The retired spelling also had a rough edge the SQL one simply does not have — = MAX means every group tied at the extreme, which a bounded read cannot prove, so MAX / MIN had to be permanently refused. ORDER BY <agg> DESC LIMIT 1 is positional and has no such ambiguity.
HAVING survives as what it is in SQL: a boolean per-group predicate — and since protocol v14 it is evaluated. A grouped aggregate carrying exactly one having clause that bounds the selected aggregate (GROUP BY hashtag HAVING count(*) > 100 LIMIT 100) is served as a value-bounded range read of the same axis secondary the ranking walks, with the same completeness-proving envelope. An ORDER BY naming the selected aggregate may ride along to set the walk direction (HAVING avg(grade) > 80 ORDER BY avg(grade) DESC LIMIT 5 — the best matches first); what a having request cannot carry is rank-window pagination (OFFSET, starting_rank), because a value-bounded page has no rank base — its continuation is "tighten the bound past the last value seen". That continuation steps past distinct aggregate values only: if the LIMIT cuts inside a tie (several groups sharing the boundary aggregate), keeping the boundary value repeats the same page and moving past it permanently skips the remaining tied groups, so size the limit above the widest expected tie. The grammar's v1 boundaries: one clause only, on the aggregate the select projects, with a contiguous-range operator (=, >, >=, <, <=, BETWEEN variants; != and IN are non-contiguous and refused).
The Restaurants Contract
Four document types, one per ranking shape. They all group by the same property (restaurantId) and each carries exactly one single-property index — because two indexes over the same property set on one doctype is a DuplicateIndexError, so exercising all three axes needs one doctype apiece.
| doctype | index | declares | terminal property-name tree |
|---|---|---|---|
review | byRestaurant | averageable + rangeAverageable + rankedAverageable | ProvableCountProvableSumIndexedTree axes [Avg] |
visit | byRestaurantVisits | countable + rangeCountable + rankedCountable | ProvableCountIndexedTree |
tip | byRestaurantTips | summable + rangeSummable + rankedSummable | ProvableSumIndexedTree |
adjustment | byRestaurantAdjustments | same as review | ProvableCountProvableSumIndexedTree axes [Avg] |
adjustment duplicates review's shape for one reason: its aggregated property delta admits negative values, which grade (minimum: 0) does not, so it is the only doctype that can exercise signed sums and the floor-toward-negative-infinity rounding of the Avg sort key.
{
"$formatVersion": "0",
"id": "AY6xWncZUFv2GCrS5seqKthUfbW9yYyUXtF8diSuHQ3f",
"ownerId": "AtirhSVpAWF7dEt6dLAmesC4Sr1MsJ9bFC1nLAoNnq2S",
"version": 1,
"documentSchemas": {
"review": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": true,
"indices": [
{
"name": "byRestaurant",
"properties": [
{ "restaurantId": "asc" }
],
"countable": "countable",
"summable": "grade",
"averageable": "grade",
"rangeCountable": true,
"rangeSummable": true,
"rangeAverageable": true,
"rankedAverageable": true
}
],
"properties": {
"restaurantId": {
"type": "string",
"minLength": 1,
"maxLength": 32,
"position": 0
},
"grade": {
"type": "integer",
"minimum": 0,
"maximum": 100,
"position": 1
}
},
"required": [
"restaurantId",
"grade"
],
"additionalProperties": false
},
"visit": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": true,
"indices": [
{
"name": "byRestaurantVisits",
"properties": [
{ "restaurantId": "asc" }
],
"countable": "countable",
"rangeCountable": true,
"rankedCountable": true
}
],
"properties": {
"restaurantId": {
"type": "string",
"minLength": 1,
"maxLength": 32,
"position": 0
},
"guests": {
"type": "integer",
"minimum": 1,
"maximum": 100,
"position": 1
}
},
"required": [
"restaurantId",
"guests"
],
"additionalProperties": false
},
"tip": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": true,
"indices": [
{
"name": "byRestaurantTips",
"properties": [
{ "restaurantId": "asc" }
],
"summable": "amount",
"rangeSummable": true,
"rankedSummable": true
}
],
"properties": {
"restaurantId": {
"type": "string",
"minLength": 1,
"maxLength": 32,
"position": 0
},
"amount": {
"type": "integer",
"minimum": 0,
"maximum": 1000000,
"position": 1
}
},
"required": [
"restaurantId",
"amount"
],
"additionalProperties": false
},
"adjustment": {
"type": "object",
"documentsMutable": true,
"canBeDeleted": true,
"indices": [
{
"name": "byRestaurantAdjustments",
"properties": [
{ "restaurantId": "asc" }
],
"countable": "countable",
"summable": "delta",
"averageable": "delta",
"rangeCountable": true,
"rangeSummable": true,
"rangeAverageable": true,
"rankedAverageable": true
}
],
"properties": {
"restaurantId": {
"type": "string",
"minLength": 1,
"maxLength": 32,
"position": 0
},
"delta": {
"type": "integer",
"minimum": -1000,
"maximum": 1000,
"position": 1
}
},
"required": [
"restaurantId",
"delta"
],
"additionalProperties": false
}
}
}
Three things to internalize before reading the queries:
rankedAverageable: trueis one line on top of six prerequisite flags. Thecountable/summable/averageabletrio and the threerange*flags are what give the terminal property-name tree its per-group(count, sum); the ranked flag adds the ordered secondary that sorts those groups. Drop any prerequisite and the contract is rejected at parse time.visitneeds no aggregated property. The Count axis ranks by group cardinality, soCOUNT(*)takes no field — and both theselectand thehavingaggregate carry an empty field string on the wire.tipis sum-only. It declares no count flags, so its terminal tree is aProvableSumIndexedTree— you can rank restaurants by total tips, but not by tip average, because there is no count axis to divide by. Averages need both.
GroveDB Layout
Each doctype's terminal property-name tree at restaurantId is an indexed tree: a primary Merk keyed by group key, plus one secondary per declared axis keyed by (sort_key ‖ group_key).
Diagram conventions: blue is the indexed element wrapper; the primary Merk's children are the ordinary group value trees (green); the orange secondary holds one entry per group, aggregate-ordered.
flowchart TB
TD["@/contract_id/0x01/review"]:::tree
TD --> RID["restaurantId:<br/>ProvableCountProvableSumIndexedTree axes [Avg]"]:::indexed
RID --> PRIM["primary Merk<br/>(keyed by group key)"]:::primary
RID --> SEC["Avg secondary Merk<br/>(keyed by avg_fp_be16 ‖ group key)"]:::secondary
PRIM --> GA["alpha: PCPS count=2 sum=170"]:::group
PRIM --> GB["beta: PCPS count=3 sum=180"]:::group
PRIM --> GD["delta: PCPS count=2 sum=60"]:::group
PRIM --> GG["gamma: PCPS count=1 sum=95"]:::group
SEC --> SD["fp(30) ‖ delta"]:::leaf
SEC --> SB["fp(60) ‖ beta"]:::leaf
SEC --> SA["fp(85) ‖ alpha"]:::leaf
SEC --> SG["fp(95) ‖ gamma"]:::leaf
classDef tree fill:#21262d,color:#c9d1d9,stroke:#1f6feb,stroke-width:2px;
classDef indexed fill:#1f6feb,color:#fff,stroke:#1f6feb,stroke-width:2px;
classDef primary fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
classDef secondary fill:#fb8500,color:#0d1117,stroke:#fb8500,stroke-width:2px;
classDef group fill:#3fb950,color:#0d1117,stroke:#3fb950,stroke-width:2px;
classDef leaf fill:#21262d,color:#c9d1d9,stroke:#484f58;
The full grove path down to that indexed tree, for a single-property index:
[ RootTree::DataContractDocuments as u8 ] // 0x01
/ <contract_id: 32 bytes>
/ [ 0x01 ] // "documents", not "contract"
/ <document_type_name: utf-8> // e.g. b"review"
/ <last_index_property_name: utf-8> // e.g. b"restaurantId"
Every ranked read and every ranked proof is issued against exactly that path, built by the same DriveDocumentRankedQuery::indexed_property_name_tree_path on both sides — which is why prover and verifier cannot drift on which ranking is being checked.
The Request on the Wire
Ranked queries ride the existing GetDocumentsRequestV1. No new request message and no new field were added: selects, group_by, order_by, limit and offset already existed, and the ranking is expressed by what goes in them.
message GetDocumentsRequestV1 {
// …
repeated OrderClause order_by = 4;
optional uint32 limit = 5;
bool prove = 8;
repeated Select selects = 9;
repeated string group_by = 10;
optional uint32 offset = 12;
}
A well-formed ranked request carries exactly one select, group_by property and order_by clause, and everything else at its unset wire value:
| Field | Ranked value |
|---|---|
selects | one Select { function: AVG, field: "grade" } |
group_by | ["restaurantId"] |
order_by | one OrderClause { target: Field("grade"), ascending: false } |
limit | the ranking's n, 1 ..= 100 — required |
offset | ranks to skip; unset = 0 |
where_clauses | empty — rejected if not |
having | empty — rejected if not |
start_after / start_at | unset — rejected if set |
Naming the aggregate in ORDER BY
The order_by clause names the aggregate it orders by through the wire's field target, and the name has to be the one the select already fixed:
select | order_by field |
|---|---|
SUM(amount) | "amount" |
AVG(grade) | "grade" |
COUNT(*) | "$count" |
SUM(f) / AVG(f) are named by f — the same property the projection aggregates, which is how ORDER BY avg(grade) reads once SELECT has fixed the function. COUNT(*) aggregates no property, so it is named by the reserved sentinel $count (RANKED_COUNT_ORDER_KEY). The $ sigil is load-bearing: DPP reserves the $ prefix for system properties ($id, $ownerId, …) and a schema cannot declare a property starting with it, so $count can never collide with a real column. A bare count would silently hijack ordering for any schema that happens to have a count field.
An order_by naming anything else — a second clause, the GROUP BY property, an unrelated field — is rejected rather than normalized. The SDK's order_by_selected_aggregate() builder derives the name from the select through rs-drive's own mapping, so there is no way to get it wrong by hand.
The wire also carries an explicit aggregate target (OrderClause.target.aggregate, e.g. ORDER BY AVG(grade) spelled out). It is still Unsupported — the field-target spelling above is the one the ranked executor reads — and exists so the explicit form can start being evaluated without another version bump.
limit is bounded to 1 ..= 100 (MAX_RANKED_LIMIT). Out-of-range is rejected, not clamped — k is echoed inside the proof envelope and re-checked by the verifier, so a server-side clamp would produce a proof the client's own reconstruction rejects.
Ranks and Offsets
OFFSET is honoured on the ranked path — the one place in the v1 surface where it is — and it is how you ask for a rank rather than a prefix:
SELECT avg(grade) FROM review
GROUP BY restaurantId
ORDER BY avg(grade) DESC
LIMIT 1 OFFSET 4 -- the 5th-best restaurant
The response carries the skip back in RankedEntries.skipped (see The Response), so the page is self-describing.
skipped is the page's starting rank base: entry i is the group at rank skipped + i (0-based). Without it, a caller who asked for LIMIT 1 OFFSET 4 receives one entry and has no way to tell that it really is the 5th-best group rather than the best. It is 0 for offset-less queries, which is what makes the field additive.
Three properties worth stating plainly:
- The skip is counted, not walked. grovedb descends the secondary reading each subtree's aggregate count and collapses any subtree that fits entirely inside the remaining offset, instead of stepping through it. Both paths do this: the prover attests the skipped region from the counted subtree commitments (
HashWithCount/HashWithCountAndSum), and the unproven read performs the same counted descent without building a proof. Work and proof size stayO(log n + k)at any offset. - There is therefore no offset ceiling. An offset of 4 and an offset of four billion cost the same order of work — on either path, the deeper one in fact cheaper, since a tree that fits entirely inside the offset collapses at the root. There is no denial-of-service lever a cap would close, and a cap would only stop honest deep pagination.
- An offset past the end is a positive answer.
entriescomes back empty andskippedis the ranking's entire reported population. "There are only 12 groups" is more information than a bare empty list.
Both paths report the same skipped: the offset you asked for when the skip succeeded, and the ranking's total population when the walk ran out of groups first. What differs is the warrant, not the value. On the proved path it is cryptographically attested, re-derived by the verifier from the counted commitments. On the unproven path it is an unverified claim, exactly like the entries beside it — equal to the attested value on an honest node, with nothing forcing a node to be honest. Callers who need to trust the population, rather than merely receive it, must still prove.
The Response
The result is an additive ResultData.ranked variant:
message RankedEntry {
bytes key = 1;
oneof value {
uint64 count = 2 [jstype = JS_STRING];
sint64 sum = 3 [jstype = JS_STRING];
double avg = 4;
}
}
message RankedEntries {
repeated RankedEntry entries = 1;
optional uint64 skipped = 2 [jstype = JS_STRING];
}
Five properties of the payload, each of them load-bearing:
skippedis the page's starting rank — see Ranks and Offsets.0for an offset-less query.keyis raw index-key bytes, not a typed value — the same bytes that name the group's value tree under the index. For astringproperty that's its UTF-8 encoding (b"alpha"). Clients that want the typed value decode it with the document type's key deserialization; the wire carries bytes so prover and verifier agree without a schema round-trip.- Entry order IS the ranking order — best-first for
DESC, worst-first forASC. Clients must not re-sort. Ties come back in group-key order in the direction of the walk, which is descending group-key order forDESC. - Fewer than
nentries is normal, not an error — the index simply has fewer groups than requested. avgis adoubleapproximation, and deliberately so. What grovedb sorts the Avg axis by is an exacti128fixed point —floor(sum × SCALE / count)with euclidean (toward −∞) division — and this field is that integer divided byRANKED_AVG_SCALEinf64. The precision loss costs nothing becauseRankedEntryonly exists on the no-proof path: a client that asked for a proof reconstructs each entry from the proof itself, where the exact fixed point lives, and never reads this field. So the wire says what it means — an approximation for the caller who already chose to trust the node — instead of dressing a trusted number up as an exact one. Two groups whose averages differ pastf64's ~15–16 significant digits can compare equal here; anything needing the committed integer must request the proof. Entry order is exact regardless: the ranking happened over thei128before the conversion. The decoder rejects a non-finiteavg, or one that scales pasti128, rather than casting it into a plausible-looking zero.
Queries in this Chapter
Every query below uses the same shape — one aggregate select, one group_by, one ORDER BY on that aggregate and a LIMIT — and differs only in the axis and the ranking. All seven come from the end-to-end suite; the "verified result" rows are what the unproven read returned and what the verifier recovered from the proof, both asserted equal against the live grovedb root hash.
| # | Query | Doctype / axis | Complexity | Verified result |
|---|---|---|---|---|
| 1 | Top 3 by Average Grade | review / Avg | O(log G + k) | gamma(95), alpha(85), beta(60) |
| 2 | The Worst Average | review / Avg | O(log G + 1) | epsilon(10.5) |
| 3 | Top 2 by Visit Count | visit / Count | O(log G + k) | delta(4), beta(3) |
| 4 | The Quietest Restaurant | visit / Count | O(log G + 1) | alpha(1) |
| 5 | Bottom 3 by Tip Total | tip / Sum | O(log G + k) | delta(1), gamma(24), alpha(25) |
| 6 | Four-Way Tie | tip / Sum | O(log G + k) | gamma, delta, beta, alpha |
| 7 | More Than Exist | tip / Sum | O(log G + G) | beta, alpha (2 of a requested 100) |
Complexity variable. G = the number of distinct groups (distinct values of the ranked property). Notably absent: the total document count N. A ranked walk reads pre-committed per-group aggregates out of the secondary and never enumerates documents, so both work and proof size are O(log G + k) — independent of how many documents sit inside the groups it returns. Proof bytes grow linearly in k (one committed secondary entry per returned group) plus the O(log G) ancestor path, which is exactly why k is capped at 100.
Query 1 — Top 3 Restaurants by Average Grade
Eight review documents across four restaurants:
| restaurant | grades | count | sum | average |
|---|---|---|---|---|
alpha | 90, 80 | 2 | 170 | 85 |
beta | 60, 70, 50 | 3 | 180 | 60 |
gamma | 95 | 1 | 95 | 95 |
delta | 40, 20 | 2 | 60 | 30 |
select = AVG(grade)
group_by = [restaurantId]
order_by = grade DESC
limit = 3
prove = true
Path query:
path: ["@", contract_id, 0x01, "review", "restaurantId"]
read: indexed_avg_top_k(k = 3, descending = true)
Verified result (returned by GroveDb::verify_indexed_axis_top_k, wrapped by DriveDocumentRankedQuery::verify_ranked_top_k_proof):
[ ("gamma", AvgFixedPoint(95 × SCALE)),
("alpha", AvgFixedPoint(85 × SCALE)),
("beta", AvgFixedPoint(60 × SCALE)) ]
descending by average: gamma(95) > alpha(85) > beta(60) > delta(30)
delta is below the cut and never appears in the proof — that is the whole point. A count-tree walk would have had to commit all four groups to convince the client which three are highest; the secondary's ordering means committing three entries plus the boundary is sufficient.
The client divides: 95 × SCALE / SCALE = 95. It never sees a double off the wire.
Query 2 — The Worst Average (and the Fixed-Point Floor)
Add a fifth restaurant whose sum doesn't divide evenly: epsilon with grades 10 and 11.
select = AVG(grade)
group_by = [restaurantId]
order_by = grade ASC
limit = 1
prove = true
Verified result:
[ ("epsilon", AvgFixedPoint(21 × SCALE / 2)) ]
21 / 2 = 10.5 is the lowest average
Three things this query pins:
ORDER BY … ASC LIMIT 1is the positional single worst-ranked group. It walks the secondary from the smallest sort key up and stops after one entry.- The sort key is
floor(sum × SCALE / count), computed with grovedb's owncompute_avg_fixed_point— the test asserts the returned value against both the hand-written21 × SCALE / 2and grovedb's function, so a change to either the scale or the rounding shows up immediately. as_f64()divides back down, returning exactly10.5. It is a display helper: lossy for large counts and sums, and never to be used for consensus-relevant comparisons, since two groups whose fixed-point averages differ can round to the samef64.
For the negative half of the rounding story, the adjustment doctype exists: with delta admitting negatives, expected_avg_fixed_point(-11, 3) must come out exactly one fixed-point bucket below what truncating division would give, because euclidean division floors toward −∞ while Rust's / truncates toward zero.
Query 3 — Top 2 Restaurants by Visit Count
Ten visit documents. The guests values are stored but irrelevant here — the Count axis ranks by how many documents are in each group, not by anything inside them:
| restaurant | visits | count |
|---|---|---|
delta | 4 documents | 4 |
beta | 3 documents | 3 |
gamma | 2 documents | 2 |
alpha | 1 document | 1 |
select = COUNT(*)
group_by = [restaurantId]
order_by = $count DESC
limit = 2
prove = true
Verified result:
[ ("delta", Count(4)),
("beta", Count(3)) ]
descending by document count: delta(4) > beta(3) > gamma(2) > alpha(1)
On the wire, both the select and the having aggregate carry field: "". A non-empty field on a COUNT ranking is rejected — COUNT(field) (counting non-null values of a property) is a different aggregate and the ranked surface doesn't serve it.
Note that visit's index declares no sum flags at all, so its terminal tree is a plain ProvableCountIndexedTree — a single secondary, no axes TLV. Asking for SUM(guests) or AVG(guests) against it resolves no index and is rejected: the stored element could host that secondary, but the contract never declared it, and the write path therefore never maintained it.
Query 4 — The Quietest Restaurant
select = COUNT(*)
group_by = [restaurantId]
order_by = $count ASC
limit = 1
prove = true
Verified result:
[ ("alpha", Count(1)) ]
Note what this query does not claim. ORDER BY $count ASC LIMIT 1 is the positional single worst-ranked group: if two restaurants tied at one visit each, one of them comes back and the other does not, deterministically (ties break by group key in the walk's direction). It is not "every group at the minimum" — that is a set a bounded read cannot attest, and it is why the retired grammar's = MIN spelling could never have been served. The positional reading documents dropping ties as its meaning; the value-based one would have had to lie about it.
See What Is Rejected and Why for the full reasoning.
Query 5 — Bottom 3 Restaurants by Tip Total
Seven tip documents:
| restaurant | amounts | sum |
|---|---|---|
beta | 100 | 100 |
alpha | 10, 15 | 25 |
gamma | 7, 8, 9 | 24 |
delta | 1 | 1 |
select = SUM(amount)
group_by = [restaurantId]
order_by = amount ASC
limit = 3
prove = true
Verified result:
[ ("delta", Sum(1)),
("gamma", Sum(24)),
("alpha", Sum(25)) ]
ascending by sum: delta(1) < gamma(24) < alpha(25) < beta(100)
ORDER BY amount DESC LIMIT 1 on the same data returns [("beta", Sum(100))].
Sums are signed (sint64 on the wire, i64 in the SDK) for the same reason the sum surface's are: grovedb's sum trees model overflow into negative space rather than saturating. The Sum axis's sort key is the i64 with its sign bit flipped, which is what makes plain byte comparison order negatives below positives.
Query 6 — A Four-Way Tie
Four groups, all summing to 50. Only the group key distinguishes them.
select = SUM(amount)
group_by = [restaurantId]
order_by = amount DESC // and the ASC mirror
limit = 4
Verified results:
DESC LIMIT 4: ["gamma", "delta", "beta", "alpha"] // descending group key
ASC LIMIT 4: ["alpha", "beta", "delta", "gamma"] // ascending group key
The two directions are exact reverses of each other under a full-width k. That falls out of the secondary's key layout rather than from a separate tie-break rule: keys are (sort_key ‖ group_key), and the walk is a plain directional scan of that keyspace, so equal sort keys come back in group-key order in the direction of the walk.
The consequence that matters is determinism under truncation. DESC LIMIT 2 on the same data returns ["gamma", "delta"] — a specific subset, not an arbitrary one, and the same subset on every node. That reproducibility is what makes a tie-truncating LIMIT k provable at all, and it is the reason the retired = MAX spelling could never have been served: = MAX means every group at the extreme, and a bounded read cannot attest that nothing else ties.
Query 7 — Asking For More Groups Than Exist
Two tip groups, LIMIT 100:
select = SUM(amount)
group_by = [restaurantId]
order_by = amount DESC
limit = 100
prove = true
Verified result:
[ ("beta", Sum(20)), ("alpha", Sum(10)) ]
A short result is the index having fewer groups, not an error, and the proof round-trips just the same. The verifier enforces the bound in the other direction only: at most k entries, because more would mean the proof committed a longer walk than the request authorized.
Fetching From the Rust SDK
DocumentRankedEntries lands on the standard Fetch trait against a DocumentQuery, with order_by_selected_aggregate() building the one ordering clause the surface needs:
#![allow(unused)] fn main() { use dash_sdk::{Sdk, platform::{DataContract, DocumentQuery, Fetch, Identifier}}; use dash_sdk::drive::query::SelectProjection; use dash_sdk::platform::documents::document_query::RankingDirection; use drive_proof_verifier::{DocumentRankedEntries, RankedEntryValue, RANKED_AVG_SCALE}; use futures::executor::block_on; const RESTAURANTS_CONTRACT_ID: [u8; 32] = [0; 32]; let sdk = Sdk::new_mock(); let contract = block_on(DataContract::fetch(&sdk, Identifier::new(RESTAURANTS_CONTRACT_ID))) .expect("fetch contract") .expect("contract exists"); let query = DocumentQuery::new(contract, "review") .expect("document type exists") .with_select(SelectProjection::avg("grade")) .with_group_by("restaurantId") .order_by_selected_aggregate(RankingDirection::Descending) .with_limit(5); let ranked = block_on(DocumentRankedEntries::fetch(&sdk, query)) .expect("fetch succeeds") .expect("a well-formed ranked query always answers"); // Entry order IS the ranking order — best first. for (offset, entry) in ranked.entries.iter().enumerate() { let rank = ranked.starting_rank + offset as u64; let restaurant = String::from_utf8_lossy(&entry.key); if let RankedEntryValue::AvgFixedPoint(fixed_point) = entry.value { let average = (fixed_point as f64) / (RANKED_AVG_SCALE as f64); println!("#{}: {restaurant}: {average}", rank + 1); } } }
fixed_point is the exact integer the proof commits to on this (proved) path. On a prove = false fetch the wire carries only the double — the SDK re-scales it back into the same variant, so the low digits are reconstruction noise. See the response notes.
The 5th-best restaurant is the same query with the window moved down one rank at a time:
#![allow(unused)] fn main() { use dash_sdk::platform::{DataContract, DocumentQuery}; use dash_sdk::platform::documents::document_query::RankingDirection; use dash_sdk::drive::query::SelectProjection; fn example(contract: DataContract) -> Result<(), dash_sdk::Error> { // SELECT avg(grade) GROUP BY restaurantId // ORDER BY avg(grade) DESC LIMIT 1 OFFSET 4 let query = DocumentQuery::new(contract, "review")? .with_select(SelectProjection::avg("grade")) .with_group_by("restaurantId") .order_by_selected_aggregate(RankingDirection::Descending) .with_limit(1) .with_offset(4); Ok(()) } }
Notes on the surface:
order_by_selected_aggregate()derives the ordered field from theselectthrough rs-drive's own key mapping —"grade"forAVG(grade), the$countsentinel forCOUNT(*). You never name it by hand, so client and server cannot disagree about what is being ordered. Set theselectfirst; the builder reads it.- It replaces rather than appends. A ranked query takes exactly one ordering clause.
RANKED_AVG_SCALEis a re-export of grovedb's constant, which moved from10^15to10^19before release. Never hardcode the literal.RankedEntryValue::as_f64()does the same division for display purposes.ranked.entriesisVec<RankedEntry>in ranking order. Do not re-sort it.ranked.starting_rankis the rank ofentries[0], re-derived from the proof rather than taken from the node.- The
Fetchpath always requests a proof. There is noproveknob on it; if you need the unproven read, that's theDocumentRankedEntries::from_unproved_responsepath. - No JS/WASM or FFI binding exists yet. The ranked surface is Rust-SDK-only today; the generated gRPC types are present in the web client but nothing is hand-written on top of them.
Proof Notes
The root hash is the whole point. The merk-level verifier returning Ok is not by itself evidence of anything. A bit-flip sweep over a real ranked envelope shows that most mutations do error out — but roughly 9% of them (bytes of sibling-subtree hashes inside the ancestor layer proofs) verify cleanly and return the correct entries, under a different reconstructed root hash. What rejects those is the tenderdash composition: drive_proof_verifier::verify_ranked_top_k_proof checks the reconstructed root against the quorum-signed app hash for the response's block, and there is no path through it that yields entries without that check having run.
Three things grovedb checks before the entries come back:
- The envelope's
(axis, k, descending, offset)match the query. They are echoed in the proof and compared against the arguments, so a proof generated for a different ranking — or for a different page of the same one — is rejected rather than silently reinterpreted. - The result's axis shape matches the requested axis — a
Countrequest must not come back holdingSumentries. Belt-and-braces on top of (1). - At most
kentries. Fewer is normal; more would mean the proof committed a longer walk than the request authorized.
Empty rankings prove. An earlier iteration of this surface could not prove one: grovedb's non-paginated prover had no absence-proof shape for "this axis secondary has no entries", so the merk layer failed with Cannot create proof for empty tree and the node had to map that to an invalid_argument telling the caller to retry unproved. It was reachable by anyone — querying a freshly registered contract with prove = true did it.
The paginated prover (prove_indexed_axis_top_k_paginated) closed that gap: against an empty axis secondary it emits a guaranteed-empty range rather than refusing. Proving a ranking over an index with no documents now succeeds and returns an empty page, so the proved and unproven paths agree on the one case where they used to diverge. There is no fallback to implement and no rejection to recognise; prove = true is always answerable.
The same mechanism is what makes an offset past the end provable — see Ranks and Offsets. The node keeps a narrow backstop mapping for the old merk-level error because it names a class of failure rather than a single call site, but it is no longer a live path.
Against a protocol-version-13 node, the whole request is rejected as Unsupported — v13's query table has no ranked path and refuses the aggregate ordering. That is the intended activation gate, not a bug: a v13 node and a v14 node must disagree here and nowhere else, which is what lets a mixed-version network run through the upgrade.
What Is Rejected and Why
Everything below is rejected before any grovedb work, and most of it is mirrored client-side so the caller learns without a round trip.
| Rejected | Why |
|---|---|
Compound (multi-property) ranked index — at contract-parse time, ranked aggregates are only supported on single-property indexes in this protocol version | Two reasons. A compound index whose prefix level also terminates an aggregating index would need its ranked terminal tree wrapped in a NonCounted / NotSummed shell — and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralize the very aggregates the ranking indexes. Separately, the ranked query surface has no equality-prefix routing yet. Both are relaxable at a future protocol version. The query-side index picker is the backstop: it refuses compound indexes even if the flags are somehow present. |
unique ranked index — ranked aggregates are not supported on unique indexes: each group of a unique index contains at most one document, so there is nothing meaningful to rank | Every ranking over a unique index degenerates to a constant-per-group ordering a plain range query already serves, while still paying for an indexed tree and its secondary maintenance on every write. |
| Contested ranked index | Covered transitively — a contested index is unique by construction, so it hits the check above. |
where clauses — InvalidWhereClauseComponents | Ranked indexes are single-property, so there is no equality prefix to narrow; and a clause on the ranked property itself asks for a ranking over a filtered subset, which the secondary cannot answer because it is ordered by aggregate rather than by group key. Silently dropping the filter would return the global ranking under the guise of a filtered one. |
start_at / start_after — InvalidLimit | The cursor names a document id, but a ranked walk iterates an aggregate-ordered keyspace in which document ids do not appear. |
order_by naming anything but the selected aggregate, or more than one clause — InvalidParameter | The single ordering clause is the ranking, and the secondary is sorted by one aggregate only. An ordering on the GROUP BY property, on an unrelated field, or a second tie-break clause names an order the secondary cannot produce. Accepting and silently ignoring it is the one genuinely dangerous option. Use the aggregate's own name ($count for COUNT(*)), or flip ASC ↔ DESC to reverse the ranking. |
group_by with ≠ 1 property — InvalidParameter | Ranked indexes are single-property, so there is no compound grouping to rank over. |
having that isn't one contiguous bound on the selected aggregate | A grouped single-clause having bounding the selected aggregate is served since protocol v14 — it routes to the having-range executor, a value-bounded range read of the same axis secondary (see the HAVING paragraph above). What stays rejected: multiple clauses (a second predicate needs a per-candidate post-check no executor performs), a clause on a different aggregate than the select projects (same reason), non-contiguous operators (!=, IN), having without group_by (a single implicit group is a plain aggregate the client can bound itself), and OFFSET / start_at alongside having (a value-bounded page has no rank base; continuation is by tightening the bound). Protocol v13 and earlier reject every non-empty having unchanged. |
no order_by at all, on a grouped aggregate — routed elsewhere | Without an ordering this is a plain grouped aggregate, not a ranking; the caller wanted the DocumentSplitCounts / DocumentSplitSums / DocumentSplitAverages surface. |
COUNT(field) (non-*) — Unsupported; SUM / AVG with an empty field — InvalidParameter | The Count axis ranks group cardinality and takes no field; the Sum and Avg axes rank the property the index accumulates and require it. |
limit unset, 0, or > 100 — InvalidLimit | A ranking with no n has no size, and LIMIT 0 selects nothing. The ceiling is a hard limit, not a clamp, because k is echoed in the proof envelope and re-checked by the verifier — a silent clamp would produce a proof the client's own reconstruction rejects. |
A SUM / AVG ranking on a field the index doesn't accumulate — no covering index | The picker requires the select's field to be the index's summable property. Resolving anything else would answer about the wrong property with no indication that a substitution happened. |
No longer rejected. The paginated prover emits a guaranteed-empty range against an empty axis secondary, so prove = true over an index with no documents returns an empty page. Listed here because it used to be a rejection and the old advice ("retry with prove = false") is now wrong. |
At-a-Glance Comparison
| Query | Doctype | Terminal tree | Axis | Ranking | Returned variant |
|---|---|---|---|---|---|
| 1 — Top 3 by average | review | ProvableCountProvableSumIndexedTree [Avg] | Avg | ORDER BY grade DESC LIMIT 3 | AvgFixedPoint(i128) |
| 2 — Worst average | review | same | Avg | ORDER BY grade ASC LIMIT 1 | AvgFixedPoint(i128) |
| 3 — Top 2 by visits | visit | ProvableCountIndexedTree | Count | ORDER BY $count DESC LIMIT 2 | Count(u64) |
| 4 — Quietest | visit | same | Count | ORDER BY $count ASC LIMIT 1 | Count(u64) |
| 5 — Bottom 3 by tips | tip | ProvableSumIndexedTree | Sum | ORDER BY amount ASC LIMIT 3 | Sum(i64) |
| 6 — Four-way tie | tip | same | Sum | ORDER BY amount DESC/ASC LIMIT 4 | Sum(i64) |
| 7 — More than exist | tip | same | Sum | ORDER BY amount DESC LIMIT 100 | Sum(i64) (2 entries) |
Every row is one bounded scan of one secondary Merk, one proof, one root-hash commit. The shape never varies with the axis — only the sort-key width (8 / 8 / 16 bytes) and the returned scalar type do.
What's Next
Two capabilities are deliberately deferred and would each land as a separate protocol-version change:
- Compound ranked indexes. Both blockers are named above — the indexed-tree wrapper conflict and the missing equality-prefix routing. Lifting them would enable "top 5 chefs at restaurant
alphaby average grade" without a client-side sort. - Offset-paginated rankings. grovedb already has the primitive (
prove_indexed_axis_top_k_paginated); it is not exposed because a ranked query currently rejects every pagination knob, and wiring one in would need the result-size contract (result size == n) to be rewritten rather than extended.
For the shape of the tree these queries read, and the write-path cost of maintaining it, see Document Ranked Trees.
Time-Range Index TTL
Architecture reference for the ttl key of timeRange indexes: what it
means, how expired windows are drained, and the invariants every walker
shares. The storage primitive underneath is grovedb's flat-subtree drop
(dashpay/grovedb#848,
landed in grovedb PR #849); see
the storage section.
A document type can also expire whole documents with its own ttl keyword; that is a
different mechanism, described in Document Time To Live.
Motivation
A timeRange index stores every document once per containing window, and
a ranked one additionally rewrites a per-window secondary on every write.
All of those bytes are billed as storage — a price that prepays
~perpetual retention through the epoch-distribution model — even though
windowed data is intrinsically ephemeral: a "posts liked this hour"
bucket is worthless once the trending surface has moved past it. The
result is that the flagship use case (likes feeding a trending index)
pays perpetuity prices for state with a useful life measured in days,
multiplied by the grid's overlap factor.
Nobody cleans this up, either. Deletion costs the deleter processing, refunds accrue to owners who have no reason to come back for entries this small, and the state lingers forever.
Semantics
A timeRange index may declare a time to live:
"timeRange": { "on": "$createdAt", "range": 3600, "step": 3600, "ttl": 604800 }
Entries under this index are queryable until ttl seconds past their
bucket's start (the exact expiry boundary remains inclusive). Physical
removal depends on subsequent writes and has no wall-clock deadline. Expired
buckets are drained lazily, on write: every state transition that
writes into the index continues draining the oldest expired bucket,
deepest-first, under a per-write operation budget. A fully drained
window is provably absent, exactly like a window that never held
documents: a count over it proves zero, and a ranked or having-range
query pinned to it proves an empty page (see the ranked chapter's
pinned-prefix rules). An expired window is not queryable at all — byStart
rejects starts past the horizon, so the drainage lag is purely internal:
drainage only ever touches expired buckets, which makes every window a
query can address complete. Everything written under the index's
grid-qualified level
bills as processing, not storage — including the transitional bytes
— at an ephemeral-bytes rate.
Why the fee reclassification is honest, not a subsidy
Storage fees prepay retention distributed across future epochs. TTL
indexes instead charge a flat per-byte processing surcharge for their
transitional storage and write amplification. Version 1 caps the queryable
lifetime at one week (SystemLimits::max_time_range_ttl_seconds = 604 800).
Cleanup capacity exceeds the maximum rate at which continued writes can
create trees. This is an amortized retention model, not a guarantee that
physical bytes disappear within a week: bursts need subsequent writes to
drain, and inactive indexes retain residue as described below.
The load-bearing simplification: TTL'd subtrees never create
refundable storage. No StorageFlags, no owner/epoch refund entries.
That single property pays off three times:
- the fee reroute needs no refund-ledger reconciliation;
- cleanup owes nobody anything;
- deletion needs no byte metering for consensus — which is what makes O(1) bucket drops possible at all (see the grovedb dependency).
Grammar and validation
ttlis an optional key of thetimeRangemap, in seconds, parsed into the transform. It is not part of the grid identity: [TimeRangeTransform::storage_key] excludes it, so declaring or changing a TTL never forks the storage level, and query-side grid matching ([TimeRangeGridSpec]) continues to compare(range, step, phase)only.ttl ≥ range.$createdAtis consensus-assigned from block time, so writes only ever target windows containing now; this invariant guarantees no bucket that can still receive entries (or serve as theoldestselector's window) is ever dropped.ttl ≤ SystemLimits::max_time_range_ttl_seconds(one week in v1). The cap is what makes the flat ephemeral-byte rate safe.- One TTL per grid per field. Two indexes bucketing the same field
with the same grid share one storage level; a differing
ttlwould give the shared subtree two conflicting lifecycles. Rejected at contract validation. - Composes with everything the grid already composes with:
countable, the range axes, ranked levels below the bucket,unique(range == step,$createdAt), indexOnly document types, andskipIfAbsenton a property below the bucket (a skipped document builds no window under a grid only skip indexes use, and no branch of its own in a window it shares).preallocatedstays banned withtimeRangefor the pre-existing structural reason.
Cleanup
Trigger — deterministic and write-amortized: every write into a
TTL'd index continues drainage of the oldest expired bucket (start
< block_time − ttl), deepest-first. Each grid's per-write drop budget is
max(SystemLimits::min_time_range_ttl_drop_operations_per_write, 2 × overlap × trees).
The versioned floor is 32; trees bounds everything one document can
create under one bucket: value trees, terminal [0] trees, and all
property-name branches in the grid's merged index structure. Shared grids
and deep suffixes are counted. Thus cleanup has capacity above the maximum
tree creation rate, including at the supported overlap of 24. A fixed
32-drop cap cannot keep up with that overlap.
When nothing is expired, the check is a single bounded range read. Large expired buckets drain across writes. In a document batch, all cleanup runs before any document mutations are generated, including nested same-type document groups. Each document earns a budget; conversion then uses the prepared state without further direct drops. Estimation performs neither cleanup nor its bookkeeping reads. Drops share the caller's transaction, so rollback restores both the removed paths and their redo records.
Residue — an index that stops receiving writes retains its remaining
buckets, including any expired backlog, indefinitely. That state owes no
refund, but its size depends on past write volume; the TTL cap alone does
not bound it. An epoch-transition sweep could provide a backstop, following
check_for_ended_vote_polls / clean_up_after_vote_polls_end; it remains
out of scope for v1.
User deletes and updates of expired documents — handled at
full-path granularity, because a bucket drains piecewise: an entry
whose bucket (or whose group's trees inside a standing bucket) the drain
already took is skipped as cleanly removed; one whose trees still stand
is removed normally, so a not-yet-drained expired bucket never carries
dangling references. Every check is deterministic — it reads consensus
state plus the carried $createdAt and block time. Writes never target
expired windows, so an update of a fully expired document simply leaves
it without entries under the TTL'd index.
An index-only window declaring outlivesDelete is not touched by a user
delete at all: its entries stay until this cleanup drops their bucket, and
the delete carries no $createdAt when only such windows involve it (see
Index-Only Document Types).
Per-index semantics — TTL removes entries from this index only.
An indexOnly like whose windowed entries expire keeps counting in the
all-time ranked byPost and in byLiker; permanence lives where the
contract declares it. Ranked per-window secondaries die with their
bucket — which also caps live leaderboard state at ~ttl / step windows
per index.
grovedb dependency: flat-subtree drop
Dropping a bucket must never put user-scaled work on the consensus path.
The primitive that landed (grovedb PR #849) is the flat-subtree drop:
O(1) consensus removal of a subtree declared to contain no child
subtrees — an ordinary parent-Merk element delete whose cost is
independent of the subtree's contents — staging a durable redo record
(atomically, outside the root hash) that names every storage prefix the
drop orphaned: the subtree's own and, for indexed primaries, its three
per-axis secondary prefixes. Reclamation is DB-level range tombstones,
drained by GroveDb::flush_pending_prefix_drops — idempotent,
crash-safe, snapshot-correct, and never part of consensus cost.
A time-range bucket is not flat, so the platform drains it
deepest-first, one flat unit at a time (drain_expired_time_range_buckets):
- each group's
[0]reference tree — flat by construction, and where the mass lives — is flat-dropped; - the emptied group value tree leaves through the flat drop — or, under a ranked (indexed-primary) property-name tree, through grovedb's dedicated indexed-tree delete, which mirrors the group out of the ranking secondary;
- the drained property-name tree is flat-dropped (dooming its secondary prefixes when ranked);
- the emptied bucket is flat-dropped.
Every step is O(1); the number of steps scales with the window's distinct groups and is capped by the structure-derived per-write budget above. Between writes a bucket may stand partially drained. Removal walkers skip only paths already removed from expired buckets and delete standing entries normally. The indexOnly delete validation uses the same rule: every surviving entry must match the full row commitment, including entries in expired but standing trees. Missing live entries, missing terminal members in standing trees, and mismatched commitments fail. An indexOnly contract must retain a timestamp-independent proof index, which still has to prove the row's membership after all its TTL entries drain.
The flat-drop path-reuse contract (never re-create a dropped path before
its record drains) holds by construction: bucket paths embed their
window start, and writes never target expired windows. The host side:
drive-abci calls flush_pending_prefix_drops after committing each
block's transaction and once at startup, completing reclamation a crash
may have interrupted.
Fee mechanics
Write operations targeting a TTL'd index's subtrees are classified
ephemeral: the walkers route them into a separate operation batch
(LowLevelDriveOperation::EphemeralGroveOperation), applied after the
standing batch, whose captured cost is consumed on its own terms — added
bytes bill to processing at
FeeStorageVersion::ttl_ephemeral_disk_usage_credit_per_byte
(270 credits/byte, 1% of the storage rate, ~27× a pro-rata week of
retention) and the storage fee contribution is zero. The elements
carry no storage flags, so deletion — the TTL drain or a user delete of
a not-yet-expired document — is basic removal with no refund entries:
there is nothing to refund, which is also where TTL writers collectively
pre-pay the drainage described below. Cost estimation routes through the
same split, so estimated and actual fees stay in the same class.
Drainage itself and the walkers' TTL bookkeeping reads are unbilled:
their costs go to scratch accounting, never to the triggering user. That
is load-bearing for the estimated >= actual fee invariant — the
estimation dry run cannot read state and therefore cannot price
state-dependent drainage, so billing it only on execution would let a
transition pass validation and then overdraw on apply. The unbilled work
is bounded: a capped count of O(1) drop operations plus a handful of
bounded reads per write.
Queries
Unchanged in shape, with one hard rule: on a TTL'd index, expired
windows are not queryable. byStart resolution rejects any start past
the expiry horizon (the same strictly-below predicate the drain uses),
on the server from committed block time and on the verifier from the
quorum-signed response time_ms — so a node cannot serve an expired
window's remnants past a verifying client. The point of the gate is that
a mid-drainage window would otherwise serve a truncated answer that
looks authoritative; rejecting the question is deterministic where
"whatever the drain has left" is not. Because drainage only ever touches
expired buckets, every window the resolver admits is complete, and a
window a past drain fully emptied inside its lifetime never existed —
absence proves normally. The relative selectors (newest / oldest)
can never address an expired window at all (ttl >= range guarantees
it).
Versioning
Everything rides the PV14 grammar, which was unreleased when this landed:
the ttl key joined the meta-schema v3 timeRange map, the two limits
joined SYSTEM_LIMITS_V4 in place, and the ephemeral-bytes rate
joined the shared storage fee table directly — no fee-version fork,
because the rate is dead below PV14 (the grammar does not parse, so no
ephemeral-classified operation can exist to read it). No migration
story exists or is needed.
Index-Only Document Types
Status: implemented and gated at protocol version 14 (meta-schema v3 / parser generation 3). The storage layout is pinned end-to-end against a real grovedb by
index_only_e2e_tests, which runs against the yappr-likes fixture atpackages/rs-drive/tests/supporting_files/contract/yappr-likes/yappr-likes-contract.json. The full ABCI pipeline (transitions, validation, executed-transition proofs) is exercised by theindex_onlytest modules in rs-drive-abci's batch tests.
The problem
A minimal social interaction — a like — is fully expressed by where it sits: which post, which hashtag, which identity. Storing it as an ordinary document costs a serialized body in primary storage (~90–150 bytes plus element flags and tree-node overhead), a primary-tree insertion, and a ~70–90-byte reference per index, for a fact whose entire content is its index position.
An index-only document type (indexOnly: true on the doc-type schema)
stores nothing in primary storage. The index entries ARE the rows:
[DataContractDocuments, contract_id, 1, <doctype>,
<prop 1>, <val 1>, …, <prop K>, <val K>, 0, <terminal value>]
→ Item(<row commitment>, flags)
The terminal is the member key, sitting exactly where a normal
non-unique index keys by document id; the element is an Item instead of
a Reference because there is nothing to point at. It is a per-index
keyword defaulting to $ownerId. It may name any schema property a prefix
position could carry (an identifier with or without a refersTo, a
bounded byte array or string, an integer, a boolean, a date), or an
ordered list of such properties (a composite terminal, whose member
key is their encoded values concatenated). The 0 storage marker,
value-tree types, and the count/sum/ranked tree derivation are
byte-identical to the ordinary non-unique layout, which is what lets the
protocol v14 ranked machinery (see
Document Ranked Trees) serve index-only
types unchanged: "the five most-liked posts in #dash" is an
O(log n + k) read with an O(log n + k) proof, and Items count in
count/ranked trees exactly as References do.
The member key is the terminal value's tree-key encoding, produced by
the same functions the prefix levels use (the walkers and probes through
get_raw_for_document_type, queries and executed proofs through
serialize_value_for_key, synthesis through decode_value_for_tree_keys),
so nothing about it is specific to a 32-byte identifier: a 33-byte public
key, a short string or an integer keys the 0 bucket exactly as it would
key a prefix level, and fee estimation sizes the member key by the
terminal property's declared bound (index_only_terminal_max_key_size)
rather than by a fixed 32. Structural uniqueness spans the terminal value:
one entry per (prefix values, terminal value), so two documents by one
owner that differ only in a scalar terminal are two entries under the
same prefix.
Composite terminals. "terminal": ["kind", "$ownerId"] keys the
member by encode(kind) ‖ owner. Every component but the last must be
fixed width (a byte array with minItems == maxItems, an identifier, an
integer, a boolean, a date), so equality on the leading components is a
clean key range and synthesis can split the key back; a string can only
be the last component; the whole key is capped at 255 bytes. Uniqueness
spans the whole key. Queries bind the components in order: equality
clauses on the leading ones, then at most one range or in clause on the
next (ordered by it), nothing on the rest. The lowering pads the bound
prefix with 0xFF to the key cap for the upper bound of "every key under
this prefix", and addresses the key itself when the bound component is
the last one. After equality-bound components are ignored, orderBy must
start at the first remaining component and follow component order without
gaps, with the same direction for every listed component. A single member-key
walk cannot sort by a later component alone or mix ascending and descending
components.
Flat indexes. An index with no properties at all is flat: its
entries live directly under a level of their own, keyed by a zero byte
followed by each terminal component name preceded by a zero byte
("\0appEphemeralPubKeyHash\0$ownerId"), which no property-name tree can
collide with since property names never contain a zero byte. This level key,
including its separators, must also fit within 255 bytes:
[DataContractDocuments, contract_id, 1, <doctype>, "\0<c1>\0<c2>…", 0, <c1 ‖ c2 ‖ …>]
→ Item(<row commitment> [‖ <entry payload>], flags)
The flat level is registration-time structure, created with the
property-name trees and kept when the last entry goes (the prune stops at
its 0 bucket, as on a preallocated index), so every entry costs the same.
There is no prefix level for an aggregate, a ranking, a time grid, a skip
set or a preallocation to apply to, so a flat index admits none of
those keywords. A clause-free query on a type with a flat index scans the
flat level (every other indexOnly type refuses the by-id shape). Non-proof
responses require this index to cover every property, including optional
ones, just as filtered queries do; otherwise use a proved projection.
The entry payload. entryPayload: ["walletEphemeralPubKey", "encryptedPayload"] on the document type names top-level properties that
live in no index: every entry's item carries them after the 32-byte row
commitment, each length-framed (u16 big-endian), in property-name order:
the type's value slot. Byte arrays store their raw bytes and strings store
UTF-8; other scalars use their tree-key encoding. The length frame preserves
empty byte arrays and strings without null sentinels, and distinguishes an
empty string from a NUL string. A payload
property must be required, scalar and bounded (the sum of the bounds is
capped by the field value limit), and appears in no index as a property
or a terminal component. It is still committed (the commitment hashes
every present property, a payload value through the uncapped payload
encoding), so the delete probes and the executed-transition verifier keep
comparing the item's first 32 bytes only, and synthesis decodes the rest
of the proved element. With more than one index the payload rides in
every entry; fee estimation sizes the item by the commitment plus the
payload bound. Together, a flat composite terminal and an entry payload
make a key-value table:
"indices": [{ "name": "byRequest", "terminal": ["appEphemeralPubKeyHash", "$ownerId"] }],
"entryPayload": ["walletEphemeralPubKey", "encryptedPayload"]
lands at […, "\0appEphemeralPubKeyHash\0$ownerId", 0, hash ‖ owner] → Item(commitment ‖ len ‖ ciphertext ‖ len ‖ wallet key), and a query on
the hash returns every responder's owner id with the payload decoded off
the item, as one proof.
timeRange buckets compose too: a bucketed indexOnly index writes
one commitment entry per containing bucket under the grid-qualified
level, exactly as stored types do — the walkers' bucket fan-out, the
probes' path derivation (entry_keys_for_raw, shared so probe and write
paths cannot drift), and the IN_TIME_RANGE count aggregates are all the
same machinery ("how many likes under #dash this hour"). The source can
only be $createdAt (the prefix rule admits no other timestamp), which
required must carry, so a delete's values reproduce the exact bucket
set. A bucketed index involves $createdAt and therefore never serves as
the proof index; and document synthesis over bucketed entries is refused
with guidance — the bucket level carries bucket-start granularity, not
the document's timestamp, and the raw entries are served by the type's
non-bucketed indexes.
The sum axes compose the same way: a summable: "<prop>" index
stores ItemWithSumItem(<row commitment>, <amount>) terminals — the same
commitment payload, plus the summed property's value — so entries
contribute to ancestor sum trees exactly as stored types'
ReferenceWithSumItem references do ("total tipped to this post", "top
posts by total tipped" via rankedSummable). The doctype-level summable
cross-checks (one canonical summed property, i64-safe integer type,
required membership) apply unchanged, and on delete grovedb reads the
amount off the stored element and propagates the subtraction — the
falsified-amount case dies on the commitment probe first, since the
amount is one of the committed properties.
Governing principle: only what is in the indexes exists and is
recoverable. Prefix property values live in the path, the terminal id in
the member key, $ownerId and $createdAt wherever an index carries them.
There is no document beyond that. One kind of property may sit in no
entry-keeping index: one a summableOffCountIndex index's source fixes
through a reference's where (a like's postAuthor, kept only by
byAuthorPost's counters). Such a property, when no entry-keeping index
holds it, is lacking from a document read back, and its value is the
referenced document's, so a client rebuilding the row for a delete reads it
from there.
The row commitment
Each entry's 32-byte payload is
hash_double(owner ‖ (name ‖ length ‖ raw index bytes)* ‖ [$createdAt])
over the document's PRESENT properties in sorted-name order
(index_only_row_commitment), $createdAt included unless only indexes
whose entries outlive a delete involve it
(index_only_row_commits_created_at, see
below) — every required property must be present,
and an optional property (a skip property of a skipIfAbsent index, the
only optional kind) contributes nothing when absent, not even its name. It binds the
independently stored index projections of one document back into one
logical row: a delete recomputes the commitment from its submitted
values, and every probed entry must carry it. A values tuple spliced from
two different creates — even two creates by the same owner — fails the
comparison on whichever entry belongs to the other row. Entry existence
alone cannot make that distinction; the commitment is what does. The
present-set is part of what is committed: absent (no name emitted) and
present-but-empty (name ‖ 00000000) hash differently, and the variable
present-set stays unambiguous because property names never contain a
zero byte while every length prefix starts with one.
Constraint matrix (parse-time, apply_index_only)
The on-disk layout depends on every one of these, so they run regardless
of full_validation — the same untrusted-boundary rule the doctype
aggregate keywords follow:
| Constraint | Why |
|---|---|
every property in required — except a skip property of a skipIfAbsent index; every ancestor of an indexed dotted path required | the index path is the storage; no null layout exists — the one sanctioned hole removes the whole entry instead |
| every index holding an optional property skips on it (its skip set is all of its optional properties) | an absent value that did not skip would need the null layout this mode has no equivalent of |
| every required property appears in ≥ 1 non-skip index (prefix or terminal); every optional property appears in a skip index whose skip set is that property alone | only indexed values exist, and a skip index carries no value for a document it skips — covered only there, a property would be validated and committed yet written nowhere |
every index embeds $ownerId (prefix or terminal) | entries are self-authorizing: a delete computed with owner = signer can only ever address the signer's own entries |
≥ 1 index is $createdAt-free AND non-skipIfAbsent — the proof index | executed-transition proofs locate entries from the transition's values alone: they can neither reproduce a block timestamp nor anchor on an entry that may not exist |
every terminal component is $ownerId or a schema property passing the indexed-shape limits (no arrays or objects; byte arrays ≤ 255 bytes, strings ≤ 63 characters); every component but the last is fixed width; the whole key ≤ 255 bytes | the member key is the components' tree-key encodings concatenated, derived by the same functions the prefix levels use; a leading component must be splittable back and rangeable; grovedb caps keys at 255 bytes; other system properties are refused because the $createdAt rules walk the prefix properties |
no index declares integerRange | an entry is keyed by the index's values and terminal, and a bucketed level holds window starts: rows differing only in the bucketed integer would claim the same entry in every window they share |
a flat index (no properties) admits no countable / summable / ranked / timeRange / integerRange / skipIfAbsent / preallocated keyword | there is no prefix level for them to apply to |
every entryPayload property is a required, bounded, top-level scalar in no index | the entry value has no representation for an absent property, estimation sizes the item by the bounds, and a property is either a key or a value |
indexed $createdAt requires $createdAt in required | creation only assigns timestamps for required system times |
documentsMutable: false, no transfers/trading/history/transient | no stored row, no revision |
non-unique, non-contested, nullSearchable default | v1 scope |
preallocated requires a fully reference-determined, non-bucketed path | see Preallocated index paths |
| a skip property is an optional, top-level schema property; no ranking sits above the index's deepest skip property | see Conditional participation |
a summableOffCountIndex index sums a source holding every document once, holds every source property, and fixes its other properties through unchanging where values | see Counters (summableOffCountIndex) |
indexOnly and the index set (terminals included, preallocated and
skipIfAbsent flags included) are immutable across contract updates — a
later-added index could never be backfilled, and the walkers derive the
skip from each index's skip set, which therefore cannot drift from
historical entries either. (Respelling skipIfAbsent: true as the array of
the same properties is no change: both parse to the same skip set.)
Conditional participation (skipIfAbsent)
An index may declare skipIfAbsent: a document that omits a property of
the index's skip set writes no entry into that index at all, and a
delete recomputes the same skip from its carried values. true makes the
skip set every optional property of the index (on an indexOnly type it
must be that set anyway, so the array spelling, naming it, parses to the
same index); a skip property may sit at any position, a timeRange window
included. The skip properties are the only properties that may leave
required, and the rules keep three views provably equivalent: the write
walkers write an index's entry only for a document carrying its skip set
(document_takes_part_in_index), the probes derive zero entry paths for
the same documents, and the row commitment pins the exact present-set so
a delete with a different absence pattern fails every probe — a skip
index can neither be force-pruned nor left with an orphan entry.
No stranded trees. The merged index structure shares levels across
indexes and prunes empty trees only upward from an entry, so the walkers
build a level only when an index the document takes part in ends at or
below it (level_reaches_entry). An untagged like under
[$createdAt window, hashtag, postId] still builds the day window shared
with byDayPost, but no hashtag branch under it; under a grid that only
a skip index uses, it builds no window at all. On an indexOnly type every
index holding an optional property skips on it, so a level keyed by an
absent value is never reached.
The semantics are a sparse projection: the index holds exactly the
documents carrying its skip set. Counts, ranked reads and absence proofs
over it answer "among documents with these properties" — a proved empty
position means "no tagged like", not "no like". The query router makes
that opt-in (index_admissible_for_skip_if_absent, in every index picker,
compiled into the verifier too): a skip index is admissible only when the
query binds every skip property (equality, in, range, order-by, or the
ranked property); the generic matcher alone would admit an unbound query
within its difference budget, and an aggregate picker's prefix match could
stop above a deep skip property, silently omitting every skipped row. A
ranking never sits above a skip property (refused at registration: no
query could read it). Structural uniqueness still spans the skip boundary
— a skipped and a taking-part document colliding on any shared non-skip
entry cannot coexist. An absent skip property is distinct from a
present-but-empty value, which indexes normally under its encoded key.
Coverage. A document carrying one optional property but missing another skips every index holding both, so each optional property needs a skip index whose skip set is that property alone: its value is then written whenever the document carries it, and the document stays deletable by values.
The economics are the point: each ranked index costs roughly the same on
every write, so a per-hashtag ranked index on a like doctype used to tax
every like — tagged or not — and forced a '' sentinel onto the
referenced post's hashtag. With byHashtagPost as a skip index, an
untagged like pays only for the indexes it actually appears in, and the
sentinel disappears (see the absence-aware where below).
Lifecycle
- Create reuses
DocumentCreateTransitionV0unchanged. State validation probes every index's entry: ANY existing entry is a duplicate (DuplicateUniqueIndexError), which is also what makes a shorter index a uniqueness constraint over its value projection plus owner — for likes, the[postId]index is the one-like-per-(post, owner) rule.refersTovalidation runs unchanged (it reads transition values, not storage), so a like on a nonexistent post is rejected, and awheredeclaration on the reference ({ "hashtag": "hashtag" }) binds the referenced post's property to the like's own: the referenced document is already fetched for the existence check, so the equality comparison adds no reads, and a like whose hashtag disagrees with its post's is refused. Absence is part of the comparison, strictly: both sides absent agree, one side absent is the same mismatch a differing value would be — a like may omit its hashtag exactly when its post has none (anything laxer would let likes on tagged posts silently deflate per-tag aggregates), which is what lets a compared property double as askipIfAbsentskip property with both sides of the reference optional. The key of an entry, the referenced side, may also name the referenced document's$ownerIdor$creatorId(the referring side must then be an identifier property):{ "$ownerId": "authorId" }binds a like to its post's current owner, so an[authorId, postId]index can be preallocated and ranked per author.$ownerIdfollows the post through transfers and$creatorIdnever changes; either is checked when the like is written, not when the post later moves.$creatorIdis only recorded by transferable or tradeable types of a format-1 contract, which contract registration checks before accepting the declaration. The referring side, an entry's value, may in turn be the like's own$ownerId, the writer:{ "$ownerId": "$ownerId" }lets only the post's current owner create or replace a like on it,{ "$creatorId": "$ownerId" }only its original creator. That is a write gate, checked on create and on every replace of the like, not only when its reference changes, since the post may have been transferred in between; a transfer itself is not re-checked, so on a transferable referring type it governs writing, not holding. A writer gate does not make an owner-prefixed index preallocatable. - Delete is its own transition kind,
DocumentIndexOnlyDeleteTransition { base, data }($action: "indexOnlyDelete"), carrying the full value tuple ($createdAtunder its system key exactly when the row commits to it:index_only_row_commits_created_at). Delete-by-id and delete-by-values are different operations — different payload, authorization model and validation pipeline — so the factory picks the KIND from the doctype's storage mode. Validation and the storage layer both require every entry to exist AND match the row commitment. A by-id delete on an index-only type (and an indexOnlyDelete on a stored type) is rejected by the structure gates; below PV14 the kind is rejected at basic structure, keeping check_tx behavior aligned with pre-4.2 software. - Replace / transfer / purchase / price are structurally impossible.
Entries that outlive a delete (outlivesDelete)
A delete-by-values recomputes every entry of the document from its values,
and a time-window entry is keyed by the bucket starts of $createdAt, which
only the block that included the create assigned. A client that did not keep
that timestamp could not delete the document. outlivesDelete: true on a
timeRange index with a ttl takes the window out of the delete:
- Delete. Neither the state validation probes nor Drive's
row-integrity gate check the index, and the delete walkers do not descend
into it:
level_removes_entryskips a level whose indexes all outlive the delete (IndexLevel::outlives_delete_at_or_belowkeeps every other contract on the old path), and the terminating level of such an index removes nothing. Its entries stay until their window is dropped by the TTL cleanup, which drops whole buckets. - Commitment. When every index involving
$createdAtoutlives deletes, the row commitment leaves the timestamp out, and a delete carries none (structure validation refuses one that does). The executed-transition proof verifier computes the same commitment, so it no longer needs the block time for such a type. - Create. State validation does not probe the index for a duplicate, and the terminal insert writes over an entry already standing at the same key (left by a deleted document with the same key) without reading it: the count does not move, and the entry carries the commitment of the row writing it. Registration makes the index's key hold the key of an index a delete clears that skips nothing, so no two documents in state share an entry, and the within-batch collision tracker leaves these entries to that index.
- Proofs.
index_only_proof_indexnever picks such an index, and the parser requires another one to exist.
Registration admits the keyword only on an indexOnly timeRange index with
a ttl, without a sum and on a type without entryPayload (a kept entry
holds the first document's amount or payload), and requires every schema
property to sit in an index that neither skips nor outlives deletes, and the
index's key to hold the key of such an index. The flag is fixed with the
index (find_first_outlives_delete_change). The index structure caches the
decisions a delete reads (IndexLevel::cleared_on_delete_at_or_below, and at
its root created_at_indexed_only_by_outliving), and Drive's delete refuses a
document carrying a $createdAt its row does not commit to.
The cost is on the aggregates: a deleted document still counts in the windows it wrote until they move past it.
Preallocated index paths
The first entry under a fresh value tuple pays for every tree on its path
— for a like that is the hashtag value tree, the postId property-name
tree, the post's value tree and the 0 member bucket — while the second
entry pays for one item insert. When the index path is a pure function of
a refersTo-referenced document, that lopsidedness is avoidable: an index
may declare preallocated: true iff every index property is either the
referring property itself (its value is the referenced document's $id)
or a referring value of that reference's where (consensus-equal to a
referenced-document property, its $ownerId and $creatorId included),
and the reference is a permanentDocument or moderatedDocument one
targeting a document type of the same contract. A deletableDocument
reference shapes the path the same way but does not qualify: its target can
be deleted without a record, and the trees created alongside it would
outlive it with other owners' entries inside and nothing left to say what
they were keyed by. A moderatedDocument target leaves state only through a
moderator's removal, whose record is never deleted, so its trees outlive it
the way the record does, and a restore (which puts the document back through
the create path) finds them in place. Through such a reference a binding
counts only when the record keeps every key it binds: the referenced
$id or $ownerId, or a property the referenced type lists under
moderatorAbilities.deleteKeepsFields (PreallocationBinding::is_kept_on_removal).
Registration refuses a preallocated index with no such binding
(validate_preallocated_indexes_kept_on_removal, once every document type
of the contract is parsed), and the insert path preallocates only through
one (Index::preallocation_bindings_for_target, given the referenced
type). byHashtagPost ([hashtag, postId]) qualifies:
hashtag through where, postId as the reference;
byLiker ([$ownerId]) cannot, since no referenced document determines
the liker. An [authorId, postId] index whose authorId agrees with the
post's $ownerId qualifies too: the poster is the one owner a referenced
post does determine.
Three things change, all bit-compatible with the fallback layout:
- Insert side (
insert/add_preallocated_index_tree_operations): inserting the referenced document also emits if-not-exists creations of the referring index's dynamic trees, down to the empty0member bucket, derived through the same tree-type helper the entry walkers use — so a preallocated tree is byte-identical to the tree the first entry's create-on-insert path would have made. The poster pays for the structural bytes; storage flags ride only when the contract itself is deletable, because that is a preallocated tree's one deletion path — entry deletes retain it by design, so entry-level flags would be unrefundable dead weight. Shared prefixes (a second post under the same hashtag) deduplicate through the if-not-exists semantics. - Delete side: removing the last member entry stops the empty-tree-pruning climb at the member level, keeping the whole apparatus — the group stays in the ranked secondaries at count 0, and a re-entry is again a plain item insert. (Non-preallocated indexes of the same type keep pruning as before.)
- Nothing else: entry insertion keeps its create-if-missing behavior, so correctness never depends on preallocation. Referenced documents created before a contract update introduced a referring type simply hand the first entry the old price, and their trees — created by the fallback — are retained on delete exactly like preallocated ones.
preallocated composes with skipIfAbsent: every bound key is resolved
before any operation is emitted, and an absent bound value (an untagged
post's hashtag, under the absence-aware agreement) bails without emitting
anything — so a tagged post preallocates the skip index's trees, an
untagged post preallocates only its id-bound indexes, and an untagged
like skips exactly the trees that were never built.
The economics: the referenced document's creator pays for the trees whether or not anyone ever references it (which is why the flag is an explicit opt-in, per index), every entry from the first on costs the same, and "no entries yet" becomes a present-but-empty member bucket — provable as zero results, rankable as a zero-count group — instead of an absent tree.
An entry's proved (path, key) position IS the document, so queries and
proofs synthesize documents through one shared builder
(query/index_only_synthesis.rs, compiled for server and verify): prefix
properties decoded from the path via decode_value_for_tree_keys (the
inverse of the write path's key encoding), the terminal from the member
key. A query through a subset index yields a documented projection. The
synthesized $id is deterministic over the proved position (a
domain-separated, length-framed hash covering every non-owner component,
$createdAt included) — nothing on chain is ever addressed by it.
Executed-transition proofs (waitForStateTransitionResult) prove a create
by the presence of the entry its values produce under the proof index
(the first $ownerId-bearing, non-skipIfAbsent index not involving
$createdAt — contract admission guarantees one exists) and a delete by
its absence, with the
proved entry's payload checked against the transition-derived row
commitment (and, when the proof index is summable, the proved sum
contribution against the created document's amount); prover and verifier
build the same single-entry path query from the transition. The outcome is always AffectedState, never
ExecutionProved: the commitment carries neither id, entropy nor nonce,
so a snapshot cannot bind one specific transition's execution.
Where clauses on the terminal property lower directly onto the entry
level's member keys once every prefix property carries an equality clause:
an equality answers "did I like X" in one query, and a range ordered by
the terminal (terminal > <last seen>, with a limit) walks the entries
page by page — keyset pagination, the indexOnly replacement for
id-shaped startAt cursors, which cannot address a position whose
synthesized id is a one-way hash. Mixed shapes are served through a
prefix pivot: one in clause may sit on the index's last prefix
property instead of the terminal (hashtag == h AND postId IN [p, q] AND $ownerId == me), with everything above it equality-bound, the terminal
clause an equality, and a limit of at least the number of in values.
A range pivot (postId > p in the same query), or an in pivot with
prefix properties below it, is refused, and the error names the index
shape that serves the query: one that lists the equality-bound
properties, the terminal's included, before the ranged property. A
pivot walk opens one branch per pivot value, and grovedb charges a
branch that holds no row one slot of the limit, so a page of such a
query could hold fewer rows than exist, and the response carries no
cursor to say where it stopped. These shapes stay refused until the
storage layer can report where a page stopped. An in pivot on the last
prefix property opens at most one branch per value, so a limit that
covers its values is never used up early. When another index serves the
same query without an incomplete pivot, index selection prefers it over
a pivot index that would win the name-order tie-break.
All shapes prove and verify through the same shared path-query builder.
Not supported on the read surface: by-$id fetches (no primary tree —
rejected with guidance) and startAt cursors (rejected with the keyset
guidance above); ranked / count / range-aggregate queries work unchanged
since they never open value trees.
Counters (summableOffCountIndex)
A summableOffCountIndex index keeps no entry per document. At the value
position of its last property, where another index grows a value tree, a
0 bucket and one entry per document, it keeps one Element::SumItem
holding the number of entries its source index keeps for that group:
before: byAuthorPost → postAuthor → <author> → postId → <post> → 0 → <liker> = Item(commitment)
after: byAuthorPost → postAuthor → <author> → postId → <post> = SumItem(likes)
The tree of the last property is a count-and-sum tree (rangeSummable,
plus rangeCountable for the group count an average divides by), so each
counter counts one group and adds its value to the sum. A level a { "at": ... } ranking
names, and every level between it and the counters, carries those totals
up: its value trees are CountSumTrees from the shallowest average
ranking down and SumTrees above it (a rankedCountable on such an index
is its sum ranking, so no count-only chain arises). The property-name tree
of a level a ranking names is the indexed tree for the axes ranked at it,
ProvableCountProvableSumIndexedTree for [Sum, Avg]; a level between two
ranked levels keeps a plain count-and-sum or sum tree. Grovedb admits a bare
SumItem under that indexed tree from grove version 4.
The write path (add_summable_off_count_counter_operations):
- Create: reads the counter and writes it back one higher, or inserts it at one for the first document of the group. The create is refused before it gets here when the source already holds the entry, so the counter equals the source group's entry count.
- Delete: writes it back one lower once the entries of the indexes that keep them matched the row commitment. A preallocated index keeps the counter at zero; any other removes it with its last document and prunes the trees it leaves empty, up to the document type.
- Once per batch: a counter is written at most once per batch, because
a documents batch carries one transition. A source keyed by more than its
owner (a terminal such as
["$ownerId", "emoji"]) holds several entries of one owner in one group, and each document is converted on its own, so raising that cap needs the counter moves folded across documents first. A second write within one conversion is refused as corrupted code execution. - Storage: a
SumItemis charged a fixed 11 bytes plus flags whatever its value, so a rewrite stores nothing new, and it keeps the flags of the first document that paid for it. - Preallocation: creating the referenced document creates the counter
at zero, in place of the value tree and its empty
0bucket.
The state probes and the duplicate check skip the index (it decides
nothing about a create), it is never the proof index, and document
queries never read it. The count, sum, average and ranked queries do: a
sum query names the source index (sum(byPost)), a count query takes a
counter's sum (its group's documents) where another index's read takes a
count (a ranked or HAVING count walks the Sum secondaries, and its
entries come back as counts), and a point query may stop at a level
carrying the sums, reading that value tree's element. A range count reads
the range sums (DriveDocumentCountQuery::counter_sums_query): every range
count executor and verifier hands the same index and clauses to the sum
surface's counterpart, summing the source index, and reads the sums back as
counts, since grovedb's range count over the counters would count them, one
per group. A range total over an index whose path passes through a ranked
level (its own, or one another index ranks at a shared level) is refused with
a hint to group by the last property, as everywhere: a ranked level's tree is
indexed, and grovedb neither totals a range over an indexed tree nor proves a
range total through one, so the unproven read refuses it too and the two
agree.
What it costs and what it saves
Registration skips the [0] primary-key tree. Each document is exactly
one […values, 0, terminal] → Item(32-byte commitment) per index — no
primary row, no references — cutting storage well past half against a
minimal stored document, with deletion refunds flowing from each entry's
own element flags (the index walkers pass flags for
immutable-yet-deletable index-only types specifically). Estimation pads
the dry-run item above the real payload so estimated fees keep
upper-bounding applied fees across the indexed-tree layers' documented
under-count.
Unit Tests
If you have spent any time reading the Dash Platform codebase, you have probably noticed that test files are everywhere -- and they follow a very specific structure. This chapter walks through the patterns that Platform's unit tests use, why those patterns exist, and how to write your own tests that fit naturally into the codebase.
The Test Module Convention
Nearly every test file in Platform follows the same opening stanza:
#![allow(unused)] fn main() { #[cfg(test)] mod tests { use super::*; // ... additional imports ... } }
This is standard Rust, but the consistency matters. The #[cfg(test)] attribute means
the entire module is compiled only when running cargo test. The use super::*; import
pulls in everything from the parent module, so tests can access the types and functions
they are testing without repeating import paths.
In Platform, tests that validate state transitions live in dedicated tests.rs files:
packages/rs-drive-abci/src/execution/validation/
state_transition/state_transitions/
address_credit_withdrawal/
tests.rs <-- unit tests for withdrawal transitions
address_funds_transfer/
tests.rs <-- unit tests for address-to-address transfers
Each tests.rs file is a self-contained test module for one state transition type. Inside,
tests are further organized into sub-modules by category:
#![allow(unused)] fn main() { #[cfg(test)] mod tests { // ... imports and helpers ... mod structure_validation { use super::*; #[test] fn test_no_inputs_returns_error() { /* ... */ } #[test] fn test_too_many_inputs_returns_error() { /* ... */ } } mod address_state_validation { use super::*; // ... } mod witness_validation { use super::*; // ... } } }
This sub-module approach groups related tests, making cargo test output scannable.
When a test fails, you immediately see tests::structure_validation::test_no_inputs_returns_error
instead of a flat list.
TestPlatformBuilder: Setting Up the World
Most unit tests need a running Platform instance with a database, genesis state, and
configuration. The TestPlatformBuilder provides a fluent API for this:
#![allow(unused)] fn main() { // File: packages/rs-drive-abci/src/test/helpers/setup.rs pub struct TestPlatformBuilder { config: Option<PlatformConfig>, initial_protocol_version: Option<ProtocolVersion>, tempdir: TempDir, } }
The builder creates a TempPlatform -- a Platform instance backed by a temporary
directory that is automatically cleaned up when the test finishes:
#![allow(unused)] fn main() { pub struct TempPlatform<C> { pub platform: Platform<C>, pub tempdir: TempDir, } }
Here is the typical setup pattern:
#![allow(unused)] fn main() { let platform = TestPlatformBuilder::new() .with_config(platform_config) .with_latest_protocol_version() .build_with_mock_rpc() .set_genesis_state(); }
Let's break this down:
new()creates a builder with a freshTempDir.with_config()injects aPlatformConfig(including test-specific overrides).with_latest_protocol_version()pins the platform to the current protocol version.build_with_mock_rpc()constructs thePlatformwith aMockCoreRPCLike-- no real Dash Core node needed.set_genesis_state()writes the initial state tree (system data contracts, etc.) into the database.
Because TempPlatform implements Deref<Target = Platform<C>>, you can call Platform
methods directly on it:
#![allow(unused)] fn main() { impl<C> Deref for TempPlatform<C> { type Target = Platform<C>; fn deref(&self) -> &Self::Target { &self.platform } } }
This means platform.drive, platform.state, and platform.config all work directly.
Helper Functions: Encapsulating Test Patterns
Each test file defines local helper functions that encapsulate repeated setup logic. For example, the withdrawal tests define helpers for creating transitions:
#![allow(unused)] fn main() { fn create_signed_address_credit_withdrawal_transition( signer: &TestAddressSigner, inputs: BTreeMap<PlatformAddress, (AddressNonce, u64)>, output: Option<(PlatformAddress, u64)>, fee_strategy: Vec<AddressFundsFeeStrategyStep>, output_script: CoreScript, ) -> StateTransition { AddressCreditWithdrawalTransitionV0::try_from_inputs_with_signer( inputs, output, AddressFundsFeeStrategy::from(fee_strategy), 1, // core_fee_per_byte Pooling::Never, output_script, signer, 0, // user_fee_increase PlatformVersion::latest(), ) .expect("should create signed transition") } }
And helpers for submitting them to the platform:
#![allow(unused)] fn main() { fn check_tx_is_valid( platform: &TempPlatform<MockCoreRPCLike>, raw_tx: &[u8], platform_version: &PlatformVersion, ) -> bool { let platform_state = platform.state.load(); let platform_ref = PlatformRef { drive: &platform.drive, state: &platform_state, config: &platform.config, core_rpc: &platform.core_rpc, }; let check_result = platform .check_tx(raw_tx, CheckTxLevel::FirstTimeCheck, &platform_ref, platform_version) .expect("expected to check tx"); check_result.is_valid() } }
The key insight is that helpers should be specific to the test file. A withdrawal
test's helper knows about AddressCreditWithdrawalTransition; it does not try to be
a generic state transition factory.
assert_matches! for Error Checking
Platform tests lean heavily on the assert_matches! macro from the assert_matches
crate. This is the idiomatic way to verify error variants in a deeply nested enum
hierarchy:
#![allow(unused)] fn main() { use assert_matches::assert_matches; assert_matches!( processing_result.execution_results().as_slice(), [StateTransitionExecutionResult::UnpaidConsensusError( ConsensusError::BasicError( BasicError::TransitionNoInputsError(_) ) )] ); }
Without assert_matches!, you would need a verbose match block or a chain of
if let statements. The macro makes the expected shape of the error immediately
visible in the test.
For cases where you need to inspect error fields, combine matches! with additional
assertions:
#![allow(unused)] fn main() { let error = result.first_error().unwrap(); assert!( matches!( error, ConsensusError::BasicError( BasicError::TransitionOverMaxInputsError(e) ) if e.actual_inputs() == 17 && e.max_inputs() == 16 ), "Expected TransitionOverMaxInputsError with 17/16, got {:?}", error ); }
The guard clause (if e.actual_inputs() == 17) lets you verify both the variant
and its contents in a single expression.
Processing State Transitions in Tests
The standard way to submit a state transition in unit tests is through
process_raw_state_transitions:
#![allow(unused)] fn main() { let raw_bytes = transition.serialize_to_bytes().unwrap(); let processing_result = platform .platform .process_raw_state_transitions( &vec![raw_bytes], &platform_state, &BlockInfo::default(), &transaction, platform_version, false, // not dry run None, // no extra data ) .expect("expected to process state transition"); }
This is the same code path that runs in production -- your test transition goes through the same validation pipeline that a real block proposer executes.
Deterministic Randomness
Tests that need random data always use a seeded RNG:
#![allow(unused)] fn main() { let mut rng = StdRng::seed_from_u64(567); let output_script = CoreScript::random_p2pkh(&mut rng); }
The seed ensures the test produces identical results every time. If a test fails,
you can reproduce the exact same inputs. Never use thread_rng() or entropy-seeded
RNGs in unit tests.
Feature-Gated Test Compilation
Some tests require features that are expensive or only available in certain contexts.
The testing-config feature gate controls test-specific configuration:
#![allow(unused)] fn main() { #[cfg(feature = "testing-config")] impl PlatformTestConfig { pub fn default_minimal_verifications() -> Self { Self { block_signing: false, store_platform_state: false, block_commit_signature_verification: false, disable_instant_lock_signature_verification: true, disable_checkpoints: true, } } } }
Tests that need the full platform test infrastructure will not compile without
--features testing-config, keeping the main build clean.
OnceLock for Expensive Resources
When a test suite needs an expensive-to-create resource (like a cryptographic key
that takes 30 seconds to build), the OnceLock pattern avoids rebuilding it for
every test:
#![allow(unused)] fn main() { use std::sync::OnceLock; static STATE_TRANSITION_TYPE_COUNTER: OnceLock<Mutex<BTreeMap<String, usize>>> = OnceLock::new(); fn state_transition_counter() -> &'static Mutex<BTreeMap<String, usize>> { STATE_TRANSITION_TYPE_COUNTER.get_or_init(|| Mutex::new(BTreeMap::new())) } }
OnceLock is initialized at most once, the first time any test calls it. Because
test threads share the same process, all tests in the binary reuse the same instance.
This pattern is essential for resources like cryptographic proving keys that are
expensive to construct but immutable once built.
Rules
Do:
- Follow the
#[cfg(test)] mod tests { use super::*; }convention. - Group tests into sub-modules by validation category.
- Use
TestPlatformBuilderfor any test that needs a platform instance. - Use
StdRng::seed_from_u64()for deterministic randomness. - Use
assert_matches!for checking error variants. - Use
OnceLockfor expensive, immutable test resources. - Process transitions through
process_raw_state_transitionsto test the real code path.
Don't:
- Use
thread_rng()or unseeded randomness in tests. - Create ad-hoc Platform instances without
TestPlatformBuilder. - Write generic helper functions that try to handle all state transition types.
- Skip
set_genesis_state()unless you are specifically testing pre-genesis behavior. - Use
unwrap()on validation results -- useassert_matches!to verify error shapes.
Strategy Tests
Unit tests verify that a single state transition behaves correctly. But what about testing an entire chain of blocks with hundreds of identities creating documents, transferring credits, and voting on contested resources -- all at the same time?
That is what strategy tests are for. They are Platform's integration-level simulation framework: you declare what should happen and let the framework simulate it across hundreds of blocks.
The Problem
Consider everything that happens in a real Dash Platform network over 100 blocks:
- Masternodes join, leave, get banned, change IPs
- Quorums rotate and sign blocks
- Identities are created, topped up, and updated
- Documents are inserted, replaced, deleted, and transferred
- Contracts are deployed and updated
- Withdrawals are processed and batched
- Protocol upgrades happen mid-chain
Testing any of these in isolation is straightforward. Testing them together -- where the output of block 47 affects the input of block 48 -- requires something more powerful than a unit test.
Two-Layer Strategy Architecture
Strategy tests use a two-layer design:
Layer 1: Strategy (defined in packages/strategy-tests/src/lib.rs) describes
what operations to perform:
#![allow(unused)] fn main() { pub struct Strategy { /// Identities to create on the first block. pub start_identities: StartIdentities, /// Platform addresses to fund on the first block. pub start_addresses: StartAddresses, /// Contracts to deploy on the second block, /// with optional scheduled updates. pub start_contracts: Vec<( CreatedDataContract, Option<BTreeMap<u64, CreatedDataContract>>, )>, /// Operations to execute each block. pub operations: Vec<Operation>, /// Configuration for ongoing identity creation. pub identity_inserts: IdentityInsertInfo, /// Optional nonce gaps for edge-case testing. pub identity_contract_nonce_gaps: Option<Frequency>, /// Key manager for signing state transitions. pub signer: Option<SimpleSigner>, } }
Layer 2: NetworkStrategy (defined in
packages/rs-drive-abci/tests/strategy_tests/strategy.rs) wraps a Strategy with
network-level configuration:
#![allow(unused)] fn main() { pub struct NetworkStrategy { pub strategy: Strategy, pub total_hpmns: u16, pub extra_normal_mns: u16, pub validator_quorum_count: u16, pub chain_lock_quorum_count: u16, pub instant_lock_quorum_count: u16, pub initial_core_height: u32, pub upgrading_info: Option<UpgradingInfo>, pub core_height_increase: CoreHeightIncrease, pub proposer_strategy: MasternodeListChangesStrategy, pub rotate_quorums: bool, pub failure_testing: Option<FailureStrategy>, pub query_testing: Option<QueryStrategy>, pub verify_state_transition_results: bool, pub max_tx_bytes_per_block: u64, pub independent_process_proposal_verification: bool, pub sign_chain_locks: bool, pub sign_instant_locks: bool, // ... } }
The separation is intentional. Strategy is about application-level behavior
(documents, identities, contracts). NetworkStrategy is about network-level behavior
(masternodes, quorums, block production). By composing them, you can test the same
application strategy under different network conditions.
Operations and Frequency
Each operation in a strategy has a type and a frequency:
#![allow(unused)] fn main() { pub struct Operation { /// The type of operation to perform. pub op_type: OperationType, /// Configuration controlling how often this operation occurs. pub frequency: Frequency, } }
OperationType is an enum covering every kind of platform action:
#![allow(unused)] fn main() { pub enum OperationType { Document(DocumentOp), IdentityTopUp(AmountRange), IdentityUpdate(IdentityUpdateOp), IdentityWithdrawal(AmountRange), ContractCreate(RandomDocumentTypeParameters, DocumentTypeCount), ContractUpdate(DataContractUpdateOp), IdentityTransfer(Option<IdentityTransferInfo>), ResourceVote(ResourceVoteOp), // ... token operations, address operations, etc. } }
Frequency controls when and how many operations occur per block:
#![allow(unused)] fn main() { pub struct Frequency { /// Range for the number of events when a block is selected. pub times_per_block_range: Range<u16>, /// Probability (0.0 to 1.0) that events occur in a given block. pub chance_per_block: Option<f64>, } }
For example, Frequency { times_per_block_range: 1..4, chance_per_block: Some(0.5) }
means: on each block, there is a 50% chance that 1-3 operations of this type will occur.
This probabilistic scheduling creates realistic, varied block content.
Running a Strategy: run_chain_for_strategy
The engine that drives everything is run_chain_for_strategy, defined in
packages/rs-drive-abci/tests/strategy_tests/execution.rs:
#![allow(unused)] fn main() { pub(crate) fn run_chain_for_strategy<'a>( platform: &'a mut Platform<MockCoreRPCLike>, block_count: u64, strategy: NetworkStrategy, config: PlatformConfig, seed: u64, add_voting_keys_to_signer: &mut Option<SimpleSigner>, add_payout_keys_to_signer: &mut Option<SimpleSigner>, ) -> ChainExecutionOutcome<'a> { // ... } }
This function:
- Generates a deterministic RNG from
seed. - Creates the specified number of masternodes and quorums.
- For each block (up to
block_count):- Determines core height increases.
- Generates state transitions based on the strategy's operations and frequencies.
- Simulates ABCI
PrepareProposal/ProcessProposal/FinalizeBlock. - Applies masternode list changes (joins, leaves, bans).
- Rotates quorums if configured.
- Returns a
ChainExecutionOutcomecontaining the final state.
The outcome struct captures everything you need to verify:
#![allow(unused)] fn main() { pub struct ChainExecutionOutcome<'a> { pub abci_app: FullAbciApplication<'a, MockCoreRPCLike>, pub masternode_identity_balances: BTreeMap<[u8; 32], Credits>, pub identities: Vec<Identity>, pub proposers: Vec<MasternodeListItemWithUpdates>, pub validator_quorums: BTreeMap<QuorumHash, TestQuorumInfo>, pub identity_nonce_counter: BTreeMap<Identifier, IdentityNonce>, pub end_epoch_index: u16, pub end_time_ms: u64, pub state_transition_results_per_block: BTreeMap<u64, Vec<(StateTransition, ExecTxResult)>>, // ... } }
Masternode List Changes
The MasternodeListChangesStrategy allows simulating a dynamic validator set:
#![allow(unused)] fn main() { pub struct MasternodeListChangesStrategy { pub new_hpmns: Frequency, pub removed_hpmns: Frequency, pub updated_hpmns: Frequency, pub banned_hpmns: Frequency, pub unbanned_hpmns: Frequency, pub changed_ip_hpmns: Frequency, pub changed_p2p_port_hpmns: Frequency, pub changed_http_port_hpmns: Frequency, pub new_masternodes: Frequency, pub removed_masternodes: Frequency, pub updated_masternodes: Frequency, pub banned_masternodes: Frequency, pub unbanned_masternodes: Frequency, pub changed_ip_masternodes: Frequency, } }
Each field uses Frequency, so you can say "ban 1-2 HPMNs per block with 10%
probability" naturally.
Writing a Strategy Test
Here is a minimal strategy test from the codebase
(packages/rs-drive-abci/tests/strategy_tests/test_cases/basic_tests.rs):
#![allow(unused)] fn main() { #[test] fn run_chain_nothing_happening() { let strategy = NetworkStrategy { strategy: Strategy { start_contracts: vec![], operations: vec![], start_identities: StartIdentities::default(), start_addresses: StartAddresses::default(), identity_inserts: IdentityInsertInfo::default(), identity_contract_nonce_gaps: None, signer: None, }, total_hpmns: 100, extra_normal_mns: 0, validator_quorum_count: 24, chain_lock_quorum_count: 24, upgrading_info: None, proposer_strategy: Default::default(), rotate_quorums: false, failure_testing: None, query_testing: None, verify_state_transition_results: false, ..Default::default() }; let config = PlatformConfig { validator_set: ValidatorSetConfig::default_100_67(), chain_lock: ChainLockConfig::default_100_67(), instant_lock: InstantLockConfig::default_100_67(), execution: ExecutionConfig { verify_sum_trees: true, ..ExecutionConfig::default() }, block_spacing_ms: 3000, testing_configs: PlatformTestConfig::default_minimal_verifications(), ..Default::default() }; let mut platform = TestPlatformBuilder::new() .with_config(config.clone()) .build_with_mock_rpc(); run_chain_for_strategy( &mut platform, 100, strategy, config, 15, &mut None, &mut None, ); } }
This test runs 100 empty blocks with 100 masternodes and verifies that the chain progresses without errors. It is the "smoke test" for the strategy framework itself.
Continuing a Chain
Strategy tests support pausing and resuming with continue_chain_for_strategy:
#![allow(unused)] fn main() { let outcome = run_chain_for_strategy( &mut platform, 50, strategy.clone(), config.clone(), 13, &mut None, &mut None, ); // Later... let continued = continue_chain_for_strategy( outcome, strategy, config, 50, // 50 more blocks &mut None, &mut None, ); }
This is invaluable for testing restart scenarios and verifying that state persists correctly across platform restarts.
How Strategy Tests Differ from Unit Tests
| Aspect | Unit Tests | Strategy Tests |
|---|---|---|
| Scope | One state transition | Hundreds across many blocks |
| Setup | TestPlatformBuilder | run_chain_for_strategy |
| Randomness | Seeded per-test | Seeded once, flows through blocks |
| Masternodes | Not involved | Fully simulated |
| Quorums | Not involved | Rotated and signed |
| Determinism | Yes | Yes (same seed = same outcome) |
| Speed | Fast (seconds) | Slow (minutes for large chains) |
Rules
Do:
- Use strategy tests for multi-block scenarios involving multiple participants.
- Start with
default_minimal_verifications()to speed up test execution. - Use small block counts (10-50) during development, increase for CI.
- Check
state_transition_results_per_blockto verify specific block outcomes. - Use
continue_chain_for_strategyfor restart/persistence testing.
Don't:
- Use strategy tests when a unit test would suffice -- they are much slower.
- Forget to pass a deterministic seed -- non-deterministic strategy tests are useless.
- Set
verify_state_transition_results: trueunless you need it; it adds overhead. - Create strategy tests with more than a few hundred blocks for regular CI runs.
Test Configuration
Platform tests need to run fast. A production node verifies block signatures, checks instant lock proofs, persists platform state to disk, and creates database checkpoints. All of that is essential for security -- and all of it makes tests slow.
This chapter covers the configuration system that lets tests disable expensive checks selectively, the builder that wires everything together, and the mock RPC layer that eliminates the need for a real Dash Core node.
PlatformTestConfig
The heart of test configuration is PlatformTestConfig, defined in
packages/rs-drive-abci/src/config.rs:
#![allow(unused)] fn main() { #[cfg(feature = "testing-config")] pub struct PlatformTestConfig { /// Whether to perform block signing. pub block_signing: bool, /// Whether to store platform state to disk. pub store_platform_state: bool, /// Whether to verify block commit signatures. pub block_commit_signature_verification: bool, /// Whether to disable instant lock signature verification. pub disable_instant_lock_signature_verification: bool, /// Whether to disable checkpoint creation during tests. pub disable_checkpoints: bool, } }
Notice the #[cfg(feature = "testing-config")] gate. This struct does not exist in
production builds. You cannot accidentally ship code that disables signature verification.
Two Default Profiles
PlatformTestConfig provides two defaults, and choosing the right one matters:
Full defaults (Default::default()): Everything enabled. Tests run like a production
node, just with a mock RPC backend:
#![allow(unused)] fn main() { #[cfg(feature = "testing-config")] impl Default for PlatformTestConfig { fn default() -> Self { Self { block_signing: true, store_platform_state: true, block_commit_signature_verification: true, disable_instant_lock_signature_verification: false, disable_checkpoints: true, } } } }
Minimal verifications (default_minimal_verifications()): Disables everything that
is not needed to test application logic:
#![allow(unused)] fn main() { impl PlatformTestConfig { pub fn default_minimal_verifications() -> Self { Self { block_signing: false, store_platform_state: false, block_commit_signature_verification: false, disable_instant_lock_signature_verification: true, disable_checkpoints: true, } } } }
Use default() when testing consensus-critical behavior (block signing, quorum
verification). Use default_minimal_verifications() for everything else -- it is
significantly faster because it skips cryptographic operations.
When to Override Individual Fields
Sometimes you need a custom combination. For example, testing withdrawal transitions requires disabling instant lock verification but keeping everything else:
#![allow(unused)] fn main() { let platform_config = PlatformConfig { testing_configs: PlatformTestConfig { disable_instant_lock_signature_verification: true, ..Default::default() }, ..Default::default() }; }
The ..Default::default() spread syntax fills in the remaining fields with their
defaults. This pattern lets you express "default with one override" clearly.
TestPlatformBuilder
TestPlatformBuilder is the fluent API for constructing a test platform. It lives in
packages/rs-drive-abci/src/test/helpers/setup.rs:
#![allow(unused)] fn main() { pub struct TestPlatformBuilder { config: Option<PlatformConfig>, initial_protocol_version: Option<ProtocolVersion>, tempdir: TempDir, } }
The Builder Chain
The builder supports three configuration methods:
#![allow(unused)] fn main() { impl TestPlatformBuilder { /// Create a new builder with a fresh temporary directory. pub fn new() -> Self { Self::default() } /// Override the platform configuration. pub fn with_config(mut self, config: PlatformConfig) -> Self { self.config = Some(config); self } /// Pin a specific protocol version. pub fn with_initial_protocol_version( mut self, initial_protocol_version: ProtocolVersion, ) -> Self { self.initial_protocol_version = Some(initial_protocol_version); self } /// Use the latest protocol version. pub fn with_latest_protocol_version(mut self) -> Self { self.initial_protocol_version = Some(PlatformVersion::latest().protocol_version); self } } }
Building
The builder has two build methods:
build_with_mock_rpc() creates a TempPlatform<MockCoreRPCLike> -- no real
Dash Core node needed:
#![allow(unused)] fn main() { pub fn build_with_mock_rpc(self) -> TempPlatform<MockCoreRPCLike> { let config = self.config.map(|mut c| { c.db_path = self.tempdir.path().to_path_buf(); c }); let platform = Platform::<MockCoreRPCLike>::open( self.tempdir.path(), config, self.initial_protocol_version .or(Some(PlatformVersion::latest().protocol_version)), ) .expect("should open Platform successfully"); TempPlatform { platform, tempdir: self.tempdir, } } }
Notice how the builder automatically sets db_path to the temp directory -- you
cannot accidentally write to a real database.
build_with_default_rpc() creates a TempPlatform<DefaultCoreRPC> for
integration tests that need a real Dash Core connection.
Initializing State
After building, you choose what initial state to install:
#![allow(unused)] fn main() { // Minimal: just the GroveDB tree structure let platform = TestPlatformBuilder::new() .build_with_mock_rpc() .set_initial_state_structure(); // Full: genesis state with system data contracts let platform = TestPlatformBuilder::new() .with_latest_protocol_version() .build_with_mock_rpc() .set_genesis_state(); // Genesis with specific activation info let platform = TestPlatformBuilder::new() .build_with_mock_rpc() .set_genesis_state_with_activation_info( genesis_time, start_core_block_height, ); }
Most tests want set_genesis_state(). Use set_initial_state_structure() only when
testing the state structure itself.
Loading Test Data Contracts
TempPlatform provides convenience methods for loading test contracts:
#![allow(unused)] fn main() { let (platform, card_game_contract) = TestPlatformBuilder::new() .build_with_mock_rpc() .set_initial_state_structure() .with_crypto_card_game_transfer_only(Transferable::Always); }
This loads a predefined "crypto card game" data contract from
tests/supporting_files/contract/ and applies it to the platform. The returned
DataContract can be used to create documents in subsequent test steps.
Mock RPC: Simulating Dash Core
The MockCoreRPCLike type (from the mockall crate) replaces the real Dash Core
RPC client. It lets tests control exactly what Core "reports" -- which transactions
are confirmed, what the current block height is, which asset locks exist, etc.
In strategy tests, the mock is configured automatically by run_chain_for_strategy.
In unit tests, you typically let the default mock behavior handle things:
#![allow(unused)] fn main() { let platform = TestPlatformBuilder::new() .with_config(platform_config) .build_with_mock_rpc() // <-- MockCoreRPCLike .set_genesis_state(); }
The mock RPC means unit tests require zero external services. They run in CI, on developer laptops, and in sandboxed environments with no network access.
PlatformConfig for Tests vs Production
PlatformConfig is a large struct with many subsections. Here is how tests typically
configure it:
#![allow(unused)] fn main() { let config = PlatformConfig { // Validator set: 100 nodes, 67% threshold validator_set: ValidatorSetConfig::default_100_67(), // Chain lock quorum config chain_lock: ChainLockConfig::default_100_67(), // Instant lock quorum config instant_lock: InstantLockConfig::default_100_67(), // Execution settings execution: ExecutionConfig { verify_sum_trees: true, ..ExecutionConfig::default() }, // Block timing block_spacing_ms: 3000, // Test-specific overrides testing_configs: PlatformTestConfig::default_minimal_verifications(), // Fill the rest with defaults ..Default::default() }; }
The default_100_67() methods create configs for a 100-node network with a 67%
signing threshold -- the standard test network size.
Platform Restart Testing
TempPlatform supports simulating a platform restart by reopening from the same
temporary directory:
#![allow(unused)] fn main() { pub fn open_with_tempdir( tempdir: TempDir, mut config: PlatformConfig, ) -> Self { config.db_path = tempdir.path().to_path_buf(); let platform = Platform::<MockCoreRPCLike>::open( tempdir.path(), Some(config), None, ) .expect("should open Platform successfully"); Self { platform, tempdir } } }
The pattern for restart testing:
#![allow(unused)] fn main() { // Run first phase let outcome = run_chain_for_strategy( &mut platform, 50, strategy, config.clone(), seed, &mut None, &mut None, ); // Extract tempdir (ownership transfer) let tempdir = platform.tempdir; // Reopen -- simulates restart let platform = TempPlatform::open_with_tempdir(tempdir, config); // Verify state survived the restart }
Rules
Do:
- Use
default_minimal_verifications()for tests that do not need signature verification. - Use
Default::default()forPlatformTestConfigwhen testing block signing or quorum logic. - Always build with
build_with_mock_rpc()unless you specifically need Dash Core. - Let the builder manage
db_path-- never set it manually in test configs. - Use
set_genesis_state()for most tests; useset_initial_state_structure()only for low-level storage tests.
Don't:
- Disable verifications in production code --
PlatformTestConfigis#[cfg(feature)]guarded. - Create
Platforminstances directly -- always useTestPlatformBuilder. - Share temporary directories between tests -- each test gets its own
TempDir. - Forget
with_latest_protocol_version()-- without it, the builder still defaults to latest, but being explicit prevents surprises during protocol upgrades. - Use
build_with_default_rpc()in CI -- it requires a running Dash Core node.
Evo SDK Overview
The Evo SDK (@dashevo/evo-sdk) is the primary JavaScript/TypeScript SDK
for building applications on Dash Platform. It provides a high-level,
strongly-typed facade over the WebAssembly-based Rust SDK, working in both
Node.js (≥ 18.18) and modern browsers.
API reference: For detailed per-method documentation with interactive examples, see the Evo SDK Docs.
How it works
┌──────────────────┐
│ Your TypeScript │
│ Application │
└────────┬─────────┘
│ EvoSDK facades (identities, documents, tokens, …)
┌────────▼─────────┐
│ @dashevo/ │
│ evo-sdk │ TypeScript wrapper layer
└────────┬─────────┘
│ calls into compiled WASM module
┌────────▼─────────┐
│ @dashevo/ │
│ wasm-sdk │ Rust SDK compiled to WebAssembly
└────────┬─────────┘
│ gRPC over HTTPS
┌────────▼─────────┐
│ DAPI nodes │ Dash Platform's decentralized API
└──────────────────┘
The Evo SDK does not use JSON-RPC or REST. Every request is a gRPC call to one of the Platform's DAPI nodes. Responses include cryptographic proofs that the SDK verifies against the platform state root, so you do not need to trust any single node.
Facades
The SDK organises its API into domain-specific facades, each accessible as a
property on the EvoSDK instance:
| Facade | Description |
|---|---|
sdk.identities | Fetch, create, update, and top up identities |
sdk.contracts | Fetch, publish, and update data contracts |
sdk.documents | Query, create, replace, delete, and transfer documents |
sdk.tokens | Mint, burn, transfer, freeze tokens and query balances |
sdk.dpns | Register and resolve Dash Platform names |
sdk.addresses | Query balances, transfer credits, withdraw to L1 |
sdk.epoch | Query epoch information and evonode proposed blocks |
sdk.protocol | Protocol version upgrade state and voting |
sdk.stateTransitions | Broadcast and wait for state transitions |
sdk.system | System status, quorum info, and total credits |
sdk.group | Group membership, actions, and contested resources |
sdk.voting | Contested resource vote states and polls |
A standalone wallet namespace is also exported for mnemonic generation, key
derivation, address validation, and message signing — see the
Wallet Utilities chapter.
What it covers
The SDK supports the full set of Platform operations:
- Queries (read-only): fetch identities, contracts, documents, token balances, DPNS names, epoch info, vote states, and more. Every query can return a cryptographic proof.
- State transitions (writes): create identities, deploy contracts, manage documents and tokens, register names, cast votes, and transfer credits.
See the API reference for the complete list of operations with interactive examples.
Getting Started
Installation
npm install @dashevo/evo-sdk
The package is ESM-only ("type": "module") and written in TypeScript with
full type definitions included. In CommonJS projects use a dynamic import():
const { EvoSDK } = await import('@dashevo/evo-sdk');
Requirements: Node.js ≥ 18.18 or any modern browser with WebAssembly support.
Quick start
import { EvoSDK } from '@dashevo/evo-sdk';
// Pick your network: testnetTrusted() for development, mainnetTrusted() for production
const sdk = EvoSDK.testnetTrusted();
// Query the current epoch (connect() is called automatically on first use)
const epoch = await sdk.epoch.current();
console.log('Current epoch:', epoch.index);
// Fetch an existing identity by its base58 ID
const identity = await sdk.identities.fetch('4EfA9Jrvv3nnCFdSf7fad59851iiTRZ6Wcu6YVJ4iSeF');
console.log('Balance:', identity?.getBalance());
Connecting
Calling connect() explicitly is optional. The SDK connects automatically
when you call any facade method for the first time. However, you can call
connect() explicitly if you want to control when the WASM module is
initialized and quorum keys are prefetched:
const sdk = EvoSDK.testnetTrusted();
await sdk.connect(); // optional — triggers WASM init and quorum prefetch now
Calling connect() more than once is a no-op.
Factory helpers
| Helper | Equivalent |
|---|---|
EvoSDK.testnet() | new EvoSDK({ network: 'testnet' }) |
EvoSDK.mainnet() | new EvoSDK({ network: 'mainnet' }) |
EvoSDK.testnetTrusted() | new EvoSDK({ network: 'testnet', trusted: true }) |
EvoSDK.mainnetTrusted() | new EvoSDK({ network: 'mainnet', trusted: true }) |
EvoSDK.local() | new EvoSDK({ network: 'local' }) |
EvoSDK.localTrusted() | new EvoSDK({ network: 'local', trusted: true }) |
Custom addresses
To connect to specific masternodes (useful for testing or private networks):
const sdk = EvoSDK.withAddresses(
['https://52.12.176.90:1443'],
'testnet',
);
await sdk.connect();
Connection options
All factory helpers and the constructor accept an optional ConnectionOptions
object:
const sdk = EvoSDK.testnetTrusted({
settings: {
connectTimeoutMs: 5000,
timeoutMs: 10000,
retries: 3,
banFailedAddress: true,
},
logs: 'info', // 'off' | 'error' | 'warn' | 'info' | 'debug' | 'trace'
proofs: true, // request proofs with every query
version: 8, // pin a specific protocol version
});
Logging
The SDK delegates logging to the underlying Rust/WASM layer. Pass a simple
level string or a full EnvFilter directive:
// Simple level
const sdk = EvoSDK.testnetTrusted({ logs: 'debug' });
// Granular filter
const sdk = EvoSDK.testnetTrusted({ logs: 'wasm_sdk=debug,rs_dapi_client=warn' });
// Change level at runtime (static, affects all instances)
await EvoSDK.setLogLevel('trace');
Trusted Mode and Proof Verification
The problem
Every Platform query response includes a cryptographic proof — a signed hash from the current validator quorum that attests the response matches the platform state tree. To verify these proofs the SDK needs the quorum public keys for the active validator set.
On a full node you can look up quorum keys directly from the Core chain. In a browser or lightweight environment you cannot. Trusted mode solves this.
How trusted mode works
When you create an SDK with trusted: true, the connect() call does an extra
step before returning: it fetches the current quorum public keys from a
well-known HTTPS endpoint and caches them in memory.
const sdk = EvoSDK.testnetTrusted();
await sdk.connect(); // fetches quorum keys, then connects to DAPI
The trust model:
- You trust the HTTPS endpoint (operated by Dash Core Group) to return correct quorum public keys.
- Once the keys are cached, every subsequent query response is verified against them — you do not trust individual DAPI nodes for the data itself.
This is a pragmatic trade-off: you trust one endpoint for the validator set, but verify all actual data cryptographically.
When to use trusted mode
| Scenario | Trusted mode? | Why |
|---|---|---|
| Browser app | Yes | No access to Core chain |
| Node.js script | Yes | Simplest setup |
| Server with Core RPC | Optional | Can fetch quorum keys from your own node |
| Local Docker setup | Yes | Use EvoSDK.localTrusted() |
If you do not use trusted mode, the SDK still works but cannot verify proofs. Queries return data but you are trusting the responding DAPI node.
Proofs in responses
By default, the SDK verifies proofs internally and returns just the data. To
inspect the proof metadata yourself, use the WithProof variants:
// Standard — proof verified internally, returns data only
const identity = await sdk.identities.fetch(id);
// With proof — returns both data and proof metadata
const { data, proof, metadata } = await sdk.identities.fetchWithProof(id);
console.log('Block height:', metadata.height);
console.log('Core chain locked height:', metadata.coreChainLockedHeight);
Some methods also offer an Unproved variant that skips proof verification
entirely, useful when you trust the node or want faster responses:
const identity = await sdk.identities.fetchUnproved(id);
State Transitions
State transitions are the write operations of Dash Platform. Unlike queries (which are free and instant), state transitions modify on-chain state, cost credits, and must be signed with a private key.
API reference: For the full list of state transition methods with parameters and examples, see the State Transitions section of the Evo SDK Docs.
How state transitions work
- Build — The SDK constructs the transition (e.g., "create identity", "register name") from the parameters you provide.
- Sign — You provide a private key (WIF format) and the SDK signs the transition.
- Broadcast — The signed transition is sent to a DAPI node, which propagates it to the Platform chain.
- Wait — The SDK waits for the transition to be included in a block and returns the result.
Identity operations
Create an identity
// Generate keys for the new identity
const keyPair = await wallet.generateKeyPair('testnet');
const identity = await sdk.identities.create({
privateKeyWif: fundingKeyWif, // key with Dash balance for the asset lock
identityPublicKeys: [{
type: 0, // ECDSA_SECP256K1
purpose: 0, // AUTHENTICATION
securityLevel: 0, // MASTER
publicKeyHex: keyPair.publicKeyHex,
}],
});
Top up an identity
await sdk.identities.topUp({
identityId: 'BxPVr5...',
amount: 100000, // credits (1 credit = 1000 duffs)
privateKeyWif: fundingKeyWif,
});
Transfer credits between identities
await sdk.identities.creditTransfer({
identityId: senderIdentityId,
recipientId: recipientIdentityId,
amount: 50000,
privateKeyWif: senderAuthKeyWif,
signingKeyIndex: 0,
nonce: await sdk.identities.nonce(senderIdentityId),
});
Register a key with a budget or an expiry
A key added with totalBudget or expiresAt is registered with those limits (protocol version 14): an application key that can spend at most so many credits, or that stops signing at a block time. Only AUTHENTICATION keys below MASTER may carry them. identityUpdate assigns the key id; the signer holds the identity's MASTER key and the new key's private key, since a new key signs its own registration.
const appKey = new IdentityPublicKeyInCreation({
keyId: 0, // reassigned to the next free id
purpose: 'AUTHENTICATION',
securityLevel: 'CRITICAL',
keyType: 'ECDSA_SECP256K1',
data: appKeyPublicKeyBytes,
totalBudget: 500000000n, // credits this key may take from the identity over its lifetime
expiresAt: 1800000000000n, // optional: block time in milliseconds from which it stops signing
});
await sdk.identities.update({ identity, addPublicKeys: [appKey], signer });
Raise a key's limits
A key registered with a budget or an expiry can be topped up, or have its expiry moved later, without being replaced. The signer holds a MASTER key, or a CRITICAL authentication key without limits and without contract bounds; the budget is added to the total the passed identity's key shows.
const key = await sdk.identities.updateKeyLimits({
identity,
keyId: 5,
addBudget: 100000000n, // credits added to the budget and to what is left of it
expiresAt: 1800000000000n, // optional: a later expiry in milliseconds
signer,
});
Document operations
Create a document
await sdk.documents.create({
contractId: 'GWRSAVFMjXx8HpQFaNJMqBV7MBgMK4br5UESsB4S31Ec',
documentType: 'domain',
document: {
label: 'my-username',
normalizedLabel: 'my-username',
normalizedParentDomainName: 'dash',
records: { identity: identityId },
subdomainRules: { allowSubdomains: false },
},
identityId,
privateKeyWif: authKeyWif,
signingKeyIndex: 0,
nonce: await sdk.identities.contractNonce(identityId, dpnsContractId),
});
Replace, delete, transfer
The sdk.documents facade also provides replace(), delete(),
transfer(), purchase(), and setPrice() methods. See the
API reference for
parameters.
Token operations
// Mint tokens (requires minting authority)
await sdk.tokens.mint({
tokenId: '...',
amount: 1000,
recipientId: '...',
identityId: minterIdentityId,
privateKeyWif: minterKeyWif,
signingKeyIndex: 0,
nonce: await sdk.identities.nonce(minterIdentityId),
});
// Transfer tokens
await sdk.tokens.transfer({
tokenId: '...',
amount: 100,
recipientId: '...',
identityId: senderIdentityId,
privateKeyWif: senderKeyWif,
signingKeyIndex: 0,
nonce: await sdk.identities.nonce(senderIdentityId),
});
DPNS name registration
await sdk.dpns.register({
name: 'alice',
identityId,
privateKeyWif: authKeyWif,
signingKeyIndex: 0,
nonce: await sdk.identities.contractNonce(identityId, dpnsContractId),
});
Waiting for results
The sdk.stateTransitions facade provides low-level control:
// Broadcast a raw state transition and wait for confirmation
const result = await sdk.stateTransitions.waitForResult(stateTransitionHash);
Nonces
Every state transition requires a nonce to prevent replay attacks. There are two types:
- Identity nonce — incremented per identity for identity-level transitions (top-ups, credit transfers, token operations)
- Contract nonce — incremented per identity-contract pair for document and contract transitions
const identityNonce = await sdk.identities.nonce(identityId);
const contractNonce = await sdk.identities.contractNonce(identityId, contractId);
Always fetch the nonce immediately before broadcasting. If another transition lands between your fetch and broadcast, the nonce will be stale and the transition will be rejected.
Wallet Utilities
The Evo SDK exports a standalone wallet namespace with offline cryptographic
utilities. These functions do not require a connected SDK instance — they
initialise the WASM module on first call and work independently.
import { wallet } from '@dashevo/evo-sdk';
Mnemonic management
// Generate a new 12-word mnemonic
const mnemonic = await wallet.generateMnemonic();
// "abandon ability able about above absent ..."
// Validate an existing mnemonic
const valid = await wallet.validateMnemonic(mnemonic);
// Convert to seed bytes (with optional passphrase)
const seed = await wallet.mnemonicToSeed(mnemonic, 'optional-passphrase');
Key derivation
From seed phrase
const keyInfo = await wallet.deriveKeyFromSeedPhrase({
mnemonic,
network: 'testnet',
derivationPath: "m/44'/1'/0'/0/0",
});
// keyInfo.privateKeyWif, keyInfo.publicKeyHex, keyInfo.address
From seed with path
const seed = await wallet.mnemonicToSeed(mnemonic);
const key = await wallet.deriveKeyFromSeedWithPath({
seed,
network: 'testnet',
path: "m/44'/1'/0'/0/0",
});
Standard derivation paths
The SDK provides helpers for Dash-specific derivation paths:
// BIP-44 paths
const bip44 = await wallet.derivationPathBip44Testnet(0, 0, 0);
// "m/44'/1'/0'/0/0"
// DIP-9 Platform paths (identity authentication keys)
const dip9 = await wallet.derivationPathDip9Testnet(0, 0, 0);
// DIP-13 DashPay paths (contact encryption keys)
const dip13 = await wallet.derivationPathDip13Testnet(0);
Extended public key operations
// Convert xprv to xpub
const xpub = await wallet.xprvToXpub(xprv);
// Derive child public key
const childPub = await wallet.deriveChildPublicKey(xpub, 0, false);
Key pair generation
// Generate a random key pair
const keyPair = await wallet.generateKeyPair('testnet');
// keyPair.privateKeyWif, keyPair.publicKeyHex, keyPair.address
// Generate multiple key pairs
const pairs = await wallet.generateKeyPairs('testnet', 5);
// Import from WIF
const imported = await wallet.keyPairFromWif('cPrivateKeyWif...');
// Import from hex
const fromHex = await wallet.keyPairFromHex('abcd1234...', 'testnet');
Address utilities
// Derive address from public key
const address = await wallet.pubkeyToAddress(pubkeyHex, 'testnet');
// Validate an address for a network
const ok = await wallet.validateAddress('yWhatever...', 'testnet');
Message signing
const signature = await wallet.signMessage(
'Hello Dash Platform',
privateKeyWif,
);
DashPay contact keys
For DashPay encrypted messaging, derive contact-specific keys:
const contactKey = await wallet.deriveDashpayContactKey({
mnemonic,
network: 'testnet',
senderIdentityId: '...',
receiverIdentityId: '...',
account: 0,
addressIndex: 0,
});
The returned xpub is the contact payment public key that can be encrypted into
a DashPay contactRequest.encryptedPublicKey field. See
DashPay Contact Requests for the end-to-end
document payload shape.
DashPay Contact Requests
DashPay contact requests are contactRequest documents in the DashPay data
contract. The document links the sender identity to toUserId and carries the
sender's DIP-15 contact payment public key encrypted for the recipient.
The Evo SDK exposes the DIP-15 derivation primitive through
wallet.deriveDashpayContactKey. Applications still need to encrypt the
derived contact xpub before submitting the document.
Document fields
The contactRequest document requires these fields:
type DashpayContactRequestDocument = {
$createdAt: number;
$createdAtCoreBlockHeight: number;
toUserId: Uint8Array;
encryptedPublicKey: Uint8Array;
senderKeyIndex: number;
recipientKeyIndex: number;
accountReference: number;
};
The current DashPay contract schema requires the system field
$createdAtCoreBlockHeight. Older external references may use
coreHeightCreatedAt; do not submit that name to the current contract.
encryptedPublicKey is exactly 96 bytes:
- 16 bytes: AES-CBC initialization vector
- 80 bytes: AES-CBC ciphertext for the sender's 69-byte compact contact xpub
The compact xpub is the parent fingerprint (4 bytes), chain code (32 bytes) and public key (33 bytes), without the version, depth and child number of a full 78-byte BIP32 serialization. Receivers, including the reference mobile wallets and the Rust, Swift and Kotlin SDKs, refuse any other plaintext length. Both 69 and 78 bytes pad to the same 80-byte ciphertext, so the contract cannot catch the mistake: a request that encrypts the full 78 bytes is stored on chain, and the recipient's wallet then drops it.
The sender derives the contact xpub from the sender identity, recipient identity, account, and address index. The sender then encrypts that xpub with an ECDH shared secret from the sender's identity encryption private key and the recipient's identity decryption public key.
Derive the contact payment xpub
import { wallet } from '@dashevo/evo-sdk';
const senderKeyIndex = 0;
const recipientKeyIndex = 0;
const addressIndex = 0;
const contactKey = await wallet.deriveDashpayContactKey({
mnemonic: senderMnemonic,
network: 'testnet',
senderIdentityId,
receiverIdentityId: recipientIdentityId,
account: 0,
addressIndex,
});
// contactKey.xpub is encrypted into contactRequest.encryptedPublicKey.
senderKeyIndex and recipientKeyIndex identify the identity public keys used
for ECDH. addressIndex is the DIP-15 child index used for contact payment key
derivation and is independent from those identity key indexes.
Build the encrypted public key
The following helper shows the byte-level encryption shape. It uses the same
secp256k1 primitives exposed by @dashevo/dashcore-lib and Node.js crypto
for AES-CBC.
import crypto from 'node:crypto';
import dashcore from '@dashevo/dashcore-lib';
function fixed32(value): Buffer {
return Buffer.from(value.toArray('be', 32));
}
function deriveSharedKey({
privateKeyWif,
publicKeyBytes,
}: {
privateKeyWif: string;
publicKeyBytes: Uint8Array;
}): Buffer {
const privateKey = dashcore.PrivateKey.fromWIF(privateKeyWif);
const publicKey = dashcore.PublicKey.fromBuffer(Buffer.from(publicKeyBytes));
const sharedPoint = publicKey.point.mul(privateKey.toBigNumber());
if (sharedPoint.isInfinity()) {
throw new Error('ECDH shared point is invalid');
}
const x = fixed32(sharedPoint.getX());
const y = fixed32(sharedPoint.getY());
const compressedPrefix = Buffer.from([2 | (y[31] & 1)]);
return crypto.createHash('sha256').update(Buffer.concat([compressedPrefix, x])).digest();
}
function compactXpubPayload(xpub: string): Buffer {
const payload = dashcore.encoding.Base58Check.decode(xpub);
if (payload.length !== 78) {
throw new Error(`Invalid DashPay contact xpub length: ${payload.length}`);
}
// BIP32 layout: version(4) depth(1) parentFingerprint(4) childNumber(4)
// chainCode(32) publicKey(33). DIP-15 encrypts only the parent
// fingerprint, chain code and public key: 69 bytes.
return Buffer.concat([payload.subarray(5, 9), payload.subarray(13, 78)]);
}
function encryptContactXpub({
contactXpub,
senderEncryptionPrivateKeyWif,
recipientDecryptionPublicKeyBytes,
}: {
contactXpub: string;
senderEncryptionPrivateKeyWif: string;
recipientDecryptionPublicKeyBytes: Uint8Array;
}): Uint8Array {
const aesKey = deriveSharedKey({
privateKeyWif: senderEncryptionPrivateKeyWif,
publicKeyBytes: recipientDecryptionPublicKeyBytes,
});
const payload = compactXpubPayload(contactXpub);
const iv = crypto.randomBytes(16);
const cipher = crypto.createCipheriv('aes-256-cbc', aesKey, iv);
const encrypted = Buffer.concat([
cipher.update(payload),
cipher.final(),
]);
const encryptedPublicKey = Buffer.concat([iv, encrypted]);
if (encryptedPublicKey.length !== 96) {
throw new Error(`DashPay encryptedPublicKey must be 96 bytes, got ${encryptedPublicKey.length}`);
}
return encryptedPublicKey;
}
Submit the document
const encryptedPublicKey = encryptContactXpub({
contactXpub: contactKey.xpub,
senderEncryptionPrivateKeyWif,
recipientDecryptionPublicKeyBytes,
});
const accountReference = 0;
const document = {
$createdAt: Date.now(),
$createdAtCoreBlockHeight: platformCoreHeight,
toUserId: recipientIdentityIdBytes,
encryptedPublicKey,
senderKeyIndex,
recipientKeyIndex,
accountReference,
};
accountReference above is a placeholder for the current Platform field
accepted by the contactRequest schema. It is not a complete implementation of
any ASK/HMAC-based account-reference obfuscation described in older DIP text.
When querying received requests through the JavaScript SDK, pass identity IDs in
the representation expected by the SDK call being used. The contract stores
toUserId as a 32-byte identifier, while some high-level JavaScript query
helpers accept the base58 identity string and perform the conversion
internally.
Current SDK boundary
wallet.deriveDashpayContactKey handles DIP-15 path derivation. It does not
currently submit DashPay documents or encrypt/decrypt encryptedPublicKey.
Applications need to combine the wallet helper with identity encryption keys
until a higher-level DashPay contact request helper is added to the JavaScript
SDK.
Treat the example as a byte-level reference. A production application should add contract validation, decrypt round-trip tests, and checks that the selected identity keys are active secp256k1 keys bounded for DashPay contact requests.
Networks and Environments
The Evo SDK supports four built-in network configurations plus custom addresses for private or development networks.
Built-in networks
| Network | Factory | DAPI discovery | Use case |
|---|---|---|---|
| Testnet | EvoSDK.testnetTrusted() | Automatic via seed nodes | Development and testing |
| Mainnet | EvoSDK.mainnetTrusted() | Automatic via seed nodes | Production applications |
| Devnet | EvoSDK.devnetTrusted(name) | Automatic via quorums server | Long-lived shared devnets (e.g. 'paloma') |
| Local | EvoSDK.localTrusted() | 127.0.0.1:1443 | Docker-based local development |
For each network, the SDK discovers DAPI endpoints from seed nodes and rotates between them automatically. Failed nodes are temporarily banned so the SDK retries against healthy nodes.
Devnets
Devnets are long-lived shared development networks identified by a short name
(e.g. 'paloma'). The trusted context derives the quorum base URL from the
name as https://quorums.<name>.networks.dash.org:
const sdk = EvoSDK.devnetTrusted('paloma');
await sdk.connect();
If the public quorums DNS for a devnet isn't deployed yet, override the URL:
const sdk = EvoSDK.devnetTrusted('paloma', {
quorumUrl: 'https://quorums.staging.example/',
});
await sdk.connect();
For a devnet without any trusted context (no proof verification), supply explicit DAPI addresses:
const sdk = EvoSDK.devnet('paloma', {
addresses: ['https://10.0.0.5:1443'],
});
await sdk.connect();
Behind the scenes these factories call WasmTrustedContext.prefetchDevnet(name)
or prefetchDevnetWithUrl(url); the same shape is available on
prefetchMainnetWithUrl / prefetchTestnetWithUrl for staging endpoints
(production networks must use https://).
The prefetch issues the current-quorum and previous-quorum requests
concurrently. It also asks the quorum service for masternode addresses, unless
the SDK was given explicit addresses: those take precedence in the builder, so
connect() passes discoverAddresses: false and skips that request.
Local development with Docker
When running a local Platform network via
dashmate,
use the local network:
const sdk = EvoSDK.localTrusted();
await sdk.connect();
This connects to https://127.0.0.1:1443 by default. If your local setup uses
different ports, use custom addresses:
const sdk = EvoSDK.withAddresses(
['https://127.0.0.1:2443'],
'local',
);
await sdk.connect();
Custom masternode addresses
For private devnets, specific nodes, or debugging:
const sdk = EvoSDK.withAddresses(
[
'https://52.12.176.90:1443',
'https://34.217.100.50:1443',
],
'testnet',
);
await sdk.connect();
When custom addresses are provided, the SDK does not perform automatic node discovery — it uses only the addresses you supply.
Browser vs Node.js
The SDK works identically in both environments. The underlying WASM module handles platform differences transparently.
Node.js considerations:
- Requires Node.js ≥ 18.18 (for WebAssembly and
fetchsupport) - ESM-only package — use
import, notrequire - No additional polyfills needed
Browser considerations:
- Works in any browser with WebAssembly support (all modern browsers)
- The WASM module is loaded asynchronously on first
connect()call - Total bundle size includes the compiled Rust SDK (~2-4 MB gzipped)
- gRPC calls use
grpc-webover HTTPS, compatible with standard CORS
Tutorial: Car Sales Management
Build a decentralised car listing and sales application on Dash Platform. By the end you will have a data contract for vehicle listings, the ability to create/query/update listings, and a purchase flow using document transfers.
How this works in practice: Data contracts are deployed once using a Node.js script with a developer identity. After deployment, your browser app uses the published contract ID to create, query, and update documents. Steps 1-2 below are run from Node.js; steps 3 onward can run in either Node.js or the browser.
What you will learn
- Designing a data contract with multiple document types
- Publishing a contract to testnet from a Node.js deployment script
- Creating, querying, and updating documents (Node.js or browser)
- Using document pricing and purchase for a sales flow
Prerequisites
npm install @dashevo/evo-sdk
You need a funded testnet identity. See the Getting Started chapter for setup.
Step 1: Design the data contract
A car sales contract needs two document types: listings (vehicles for sale) and reviews (buyer reviews of sellers).
const carSalesSchema = {
listing: {
type: 'object',
properties: {
make: { type: 'string', maxLength: 63, position: 0 },
model: { type: 'string', maxLength: 63, position: 1 },
year: { type: 'integer', minimum: 1900, maximum: 2100, position: 2 },
mileageKm: { type: 'integer', minimum: 0, position: 3 },
priceUsd: { type: 'integer', minimum: 0, position: 4 },
description: { type: 'string', maxLength: 1024, position: 5 },
imageUrl: { type: 'string', maxLength: 512, format: 'uri', position: 6 },
status: { type: 'string', enum: ['available', 'pending', 'sold'], position: 7 },
},
required: ['make', 'model', 'year', 'priceUsd', 'status'],
additionalProperties: false,
},
review: {
type: 'object',
properties: {
sellerId: { type: 'string', maxLength: 44, position: 0 },
listingId: { type: 'string', maxLength: 44, position: 1 },
rating: { type: 'integer', minimum: 1, maximum: 5, position: 2 },
comment: { type: 'string', maxLength: 512, position: 3 },
},
required: ['sellerId', 'rating'],
additionalProperties: false,
},
};
Step 2: Connect and publish the contract
import { EvoSDK, DataContract, Document, Identifier, IdentitySigner } from '@dashevo/evo-sdk';
const sdk = EvoSDK.testnetTrusted();
await sdk.connect();
// Your identity credentials
const identityId = 'YOUR_IDENTITY_ID';
const privateKeyWif = 'YOUR_PRIVATE_KEY_WIF';
const signingKeyIndex = 0;
// Set up signing
const identity = await sdk.identities.fetch(identityId);
const identityKey = identity.publicKeys[signingKeyIndex];
const signer = new IdentitySigner();
signer.addKeyFromWif(privateKeyWif);
// Publish the data contract
const nonce = await sdk.identities.nonce(identityId);
const dataContract = new DataContract({
ownerId: new Identifier(identityId),
identityNonce: nonce + 1n,
schemas: carSalesSchema,
});
const contract = await sdk.contracts.publish({ dataContract, identityKey, signer });
const contractId = contract.id.toString();
console.log('Contract published:', contractId);
Save the contractId — you will need it for all subsequent operations.
Step 3: Create a listing
const doc = new Document({
documentTypeName: 'listing',
dataContractId: new Identifier(contractId),
ownerId: new Identifier(identityId),
properties: {
make: 'Toyota',
model: 'Camry',
year: 2021,
mileageKm: 45000,
priceUsd: 22500,
description: 'Well-maintained, single owner, full service history.',
status: 'available',
},
});
await sdk.documents.create({ document: doc, identityKey, signer });
console.log('Listing created!');
Step 4: Query listings
// Fetch all available listings
const results = await sdk.documents.query({
dataContractId: contractId,
documentTypeName: 'listing',
where: [['status', '==', 'available']],
orderBy: [['priceUsd', 'asc']],
limit: 20,
});
for (const [id, doc] of results) {
if (!doc) continue;
const data = doc.properties as Record<string, unknown>;
console.log(`${data.year} ${data.make} ${data.model} — $${data.priceUsd}`);
console.log(` ID: ${id}`);
}
Search by make
const toyotas = await sdk.documents.query({
dataContractId: contractId,
documentTypeName: 'listing',
where: [
['make', '==', 'Toyota'],
['status', '==', 'available'],
],
limit: 10,
});
Step 5: Update a listing
Mark a listing as sold:
const listingId = 'THE_LISTING_DOCUMENT_ID';
// Fetch the existing document, modify it, and bump the revision
const existing = await sdk.documents.get(contractId, 'listing', listingId);
existing.properties = { ...existing.properties, status: 'sold' };
existing.revision = (existing.revision ?? 0n) + 1n;
await sdk.documents.replace({ document: existing, identityKey, signer });
console.log('Listing marked as sold');
Step 6: Leave a review
// Set up buyer signing
const buyerIdentity = await sdk.identities.fetch(buyerIdentityId);
const buyerKey = buyerIdentity.publicKeys[0];
const buyerSigner = new IdentitySigner();
buyerSigner.addKeyFromWif(buyerKeyWif);
const reviewDoc = new Document({
documentTypeName: 'review',
dataContractId: new Identifier(contractId),
ownerId: new Identifier(buyerIdentityId),
properties: {
sellerId: 'SELLER_IDENTITY_ID',
listingId: 'THE_LISTING_DOCUMENT_ID',
rating: 5,
comment: 'Great seller, car was exactly as described!',
},
});
await sdk.documents.create({ document: reviewDoc, identityKey: buyerKey, signer: buyerSigner });
Query reviews for a seller
const reviews = await sdk.documents.query({
dataContractId: contractId,
documentTypeName: 'review',
where: [['sellerId', '==', 'SELLER_IDENTITY_ID']],
orderBy: [['rating', 'desc']],
limit: 50,
});
let totalRating = 0;
let count = 0;
for (const [, doc] of reviews) {
if (!doc) continue;
const props = doc.properties as Record<string, unknown>;
totalRating += props.rating as number;
count++;
}
console.log(`Average rating: ${(totalRating / count).toFixed(1)} (${count} reviews)`);
Next steps
- Add indexes to the contract schema for efficient queries on
make,year, andpriceUsd - Add a
locationfield and query by region - Use document pricing (
sdk.documents.setPrice/sdk.documents.purchase) to let buyers pay for premium listing details - Integrate with a frontend framework (React, Vue, etc.) for a full web app
Tutorial: Creating a Basic Token
Environment: This tutorial uses Node.js scripts to deploy a contract and perform token operations. For browser-based applications, deploy the contract from a Node.js script first, then use the contract ID in your frontend code.
Create a fungible token on Dash Platform with minting, transferring, and balance queries. This tutorial walks through the full lifecycle from contract deployment to token operations.
What you will learn
- Defining a data contract with a token configuration
- Minting tokens to an identity
- Transferring tokens between identities
- Querying balances and supply
Prerequisites
npm install @dashevo/evo-sdk
You need a funded testnet identity with enough credits to deploy a contract and perform token operations.
Step 1: Define the token contract
A token is defined as part of a data contract. The contract schema includes a
tokens section alongside the usual document schemas.
import {
EvoSDK, DataContract, Identifier, IdentitySigner,
TokenConfigurationConvention, TokenConfigurationLocalization, TokenConfiguration,
ChangeControlRules, AuthorizedActionTakers, TokenDistributionRules,
TokenKeepsHistoryRules, TokenMarketplaceRules, TokenTradeMode,
} from '@dashevo/evo-sdk';
const sdk = EvoSDK.testnetTrusted();
await sdk.connect();
const identityId = 'YOUR_IDENTITY_ID';
const privateKeyWif = 'YOUR_PRIVATE_KEY_WIF';
const signingKeyIndex = 0;
// Define a contract with a token
const contractSchema = {
// Document types (optional — a token-only contract can have none)
tokenMetadata: {
type: 'object',
properties: {
tokenName: { type: 'string', maxLength: 63, position: 0 },
description: { type: 'string', maxLength: 256, position: 1 },
},
additionalProperties: false,
},
};
// Build the token configuration using SDK classes
const localization = new TokenConfigurationLocalization(true, 'CoffeeCoin', 'CoffeeCoins');
const conventions = new TokenConfigurationConvention({ en: localization }, 2);
const ownerOnly = new ChangeControlRules({
authorizedToMakeChange: AuthorizedActionTakers.ContractOwner(),
adminActionTakers: AuthorizedActionTakers.ContractOwner(),
});
const noOne = new ChangeControlRules({
authorizedToMakeChange: AuthorizedActionTakers.NoOne(),
adminActionTakers: AuthorizedActionTakers.NoOne(),
});
const tokenConfig = new TokenConfiguration({
conventions,
conventionsChangeRules: noOne,
baseSupply: 0n,
maxSupply: 1_000_000_00n, // 1,000,000.00 with 2 decimals
maxSupplyChangeRules: noOne,
keepsHistory: new TokenKeepsHistoryRules({
isKeepingMintingHistory: true,
isKeepingBurningHistory: true,
isKeepingTransferHistory: true,
}),
distributionRules: new TokenDistributionRules({
perpetualDistributionRules: noOne,
newTokensDestinationIdentityRules: noOne,
mintingAllowChoosingDestination: true,
mintingAllowChoosingDestinationRules: noOne,
changeDirectPurchasePricingRules: noOne,
}),
marketplaceRules: new TokenMarketplaceRules(TokenTradeMode.NotTradeable(), noOne),
manualMintingRules: ownerOnly,
manualBurningRules: ownerOnly,
freezeRules: noOne,
unfreezeRules: noOne,
destroyFrozenFundsRules: noOne,
emergencyActionRules: noOne,
mainControlGroupCanBeModified: AuthorizedActionTakers.NoOne(),
});
Optional: let every identity claim a fixed amount once
A once-per-identity distribution turns the token into an open airdrop: any identity can
claim the configured amount exactly once, and the total paid out is bounded only by
maxSupply. The distribution is fixed at registration; there is no change control rule
for it, and no configuration update changes it. Setting it makes the distribution rules
serialize as format version 1, which needs protocol version 14, and adds 0.1 Dash to the
contract's registration fee, like a perpetual or pre-programmed distribution does.
import { TokenOncePerIdentityDistribution } from '@dashevo/evo-sdk';
distributionRules: new TokenDistributionRules({
perpetualDistributionRules: noOne,
newTokensDestinationIdentityRules: noOne,
mintingAllowChoosingDestination: true,
mintingAllowChoosingDestinationRules: noOne,
changeDirectPurchasePricingRules: noOne,
oncePerIdentityDistribution: new TokenOncePerIdentityDistribution(100_00n), // 100.00 tokens
}),
Any identity then claims with a token claim of distribution type oncePerIdentity:
await sdk.tokens.claim({
dataContractId: contract.id,
tokenPosition: 0,
identityId: claimantId,
distributionType: 'oncePerIdentity',
identityKey,
signer,
});
A second claim by the same identity is rejected with TokenOncePerIdentityDistributionAlreadyClaimedError
(code 40722), and a claim that would push the supply past maxSupply is rejected with
TokenMintPastMaxSupplyError.
Step 2: Publish the contract
// Set up signing
const identity = await sdk.identities.fetch(identityId);
const identityKey = identity.publicKeys[signingKeyIndex];
const signer = new IdentitySigner();
signer.addKeyFromWif(privateKeyWif);
const nonce = await sdk.identities.nonce(identityId);
const dataContract = new DataContract({
ownerId: new Identifier(identityId),
identityNonce: nonce + 1n,
schemas: contractSchema,
tokens: { 0: tokenConfig },
});
const contract = await sdk.contracts.publish({ dataContract, identityKey, signer });
const contractId = contract.id.toString();
console.log('Contract published:', contractId);
Step 3: Mint tokens
The contract owner can mint tokens to any identity:
// Token operations require a CRITICAL security level key.
// Fetch a key with the appropriate security level from the identity.
const criticalKey = identity.publicKeys[signingKeyIndex];
const criticalSigner = new IdentitySigner();
criticalSigner.addKeyFromWif(privateKeyWif);
// Mint 10,000.00 CoffeeCoins to yourself
await sdk.tokens.mint({
dataContractId: new Identifier(contractId),
tokenPosition: 0,
amount: 10_000_00n, // 10,000.00 (2 decimal places) — must be bigint
recipientId: new Identifier(identityId),
identityId: new Identifier(identityId),
identityKey: criticalKey,
signer: criticalSigner,
});
console.log('Minted 10,000 CoffeeCoins');
Mint to another identity
await sdk.tokens.mint({
dataContractId: new Identifier(contractId),
tokenPosition: 0,
amount: 500_00n, // 500.00 CoffeeCoins
recipientId: new Identifier('RECIPIENT_IDENTITY_ID'),
identityId: new Identifier(identityId),
identityKey: criticalKey,
signer: criticalSigner,
});
Step 4: Check balances
// Check your own balance
const myBalances = await sdk.tokens.identityBalances(identityId, [contractId]);
let myBalance = 0n;
for (const [id, balance] of myBalances) {
if (id.toString() === contractId) myBalance = balance;
}
console.log('My balance:', Number(myBalance) / 100, 'CoffeeCoins');
// Check multiple identities at once
const balances = await sdk.tokens.balances(
[identityId, 'OTHER_IDENTITY_ID'],
contractId,
);
for (const [id, balance] of balances) {
console.log(`${id}: ${Number(balance) / 100} CoffeeCoins`);
}
Check total supply
const tokenId = await sdk.tokens.calculateId(contractId, 0);
const supply = await sdk.tokens.totalSupply(tokenId);
if (supply) {
console.log('Total supply:', Number(supply.totalSupply) / 100, 'CoffeeCoins');
}
Step 5: Transfer tokens
await sdk.tokens.transfer({
dataContractId: new Identifier(contractId),
tokenPosition: 0,
amount: 25_00n, // 25.00 CoffeeCoins
recipientId: new Identifier('RECIPIENT_IDENTITY_ID'),
senderId: new Identifier(identityId),
identityKey: criticalKey,
signer: criticalSigner,
});
console.log('Transferred 25 CoffeeCoins');
Step 6: Burn tokens
Reduce the supply by burning tokens you own:
await sdk.tokens.burn({
dataContractId: new Identifier(contractId),
tokenPosition: 0,
amount: 100_00n, // 100.00 CoffeeCoins
identityId: new Identifier(identityId),
identityKey: criticalKey,
signer: criticalSigner,
});
console.log('Burned 100 CoffeeCoins');
Full example
Putting it all together as a complete script:
import { EvoSDK, Identifier, IdentitySigner } from '@dashevo/evo-sdk';
async function main() {
const sdk = EvoSDK.testnetTrusted();
await sdk.connect();
const identityId = 'YOUR_IDENTITY_ID';
const privateKeyWif = 'YOUR_PRIVATE_KEY_WIF';
const contractId = 'YOUR_CONTRACT_ID'; // from step 2
// Set up signing (token ops require a CRITICAL security level key)
const identity = await sdk.identities.fetch(identityId);
const identityKey = identity.publicKeys[0];
const signer = new IdentitySigner();
signer.addKeyFromWif(privateKeyWif);
// Check balance
const balances = await sdk.tokens.identityBalances(identityId, [contractId]);
for (const [id, balance] of balances) {
if (id.toString() === contractId) console.log('Balance:', balance);
}
// Transfer
await sdk.tokens.transfer({
dataContractId: new Identifier(contractId),
tokenPosition: 0,
amount: 10_00n,
recipientId: new Identifier('FRIEND_IDENTITY_ID'),
senderId: new Identifier(identityId),
identityKey,
signer,
});
console.log('Transfer complete!');
}
main().catch(console.error);
Next steps
- Add freeze/unfreeze capabilities for compliance scenarios
- Set up a direct purchase price so anyone can buy tokens with credits
- Create a distribution schedule for automatic token rewards
- Use the
tokenMetadatadocument type to store on-chain metadata
Tutorial: Card Game with Tokens
Build a collectible card game on Dash Platform where cards are documents that can be traded, and an in-game currency token is used for purchases. This tutorial combines data contracts, documents, and tokens into a cohesive application.
Environment: Steps 1-2 (contract design and deployment) are run from a Node.js script using a developer/operator identity. Steps 3 onward (minting, trading, querying) can run in either Node.js or a browser app using the published contract ID.
What you will learn
- Designing a contract with both document types and tokens
- Using documents as game items (cards) owned by identities
- Token-based in-game economy (minting rewards, spending on packs)
- Document transfers for card trading between players
- Querying collections and leaderboards
Prerequisites
npm install @dashevo/evo-sdk
You need a funded testnet identity. This tutorial uses two identities to demonstrate trading.
Step 1: Design the game contract
The contract defines three document types and one token:
- card — A collectible card with rarity, power, and element
- deck — A player's active deck configuration
- match — Match result history
- GemToken — In-game currency for buying card packs
import {
TokenConfigurationConvention, TokenConfigurationLocalization, TokenConfiguration,
ChangeControlRules, AuthorizedActionTakers, TokenDistributionRules,
TokenKeepsHistoryRules, TokenMarketplaceRules, TokenTradeMode,
} from '@dashevo/evo-sdk';
const gameSchema = {
card: {
type: 'object',
properties: {
name: { type: 'string', maxLength: 63, position: 0 },
element: { type: 'string', enum: ['fire', 'water', 'earth', 'air', 'shadow'], position: 1 },
rarity: { type: 'string', enum: ['common', 'uncommon', 'rare', 'legendary'], position: 2 },
power: { type: 'integer', minimum: 1, maximum: 100, position: 3 },
defense: { type: 'integer', minimum: 1, maximum: 100, position: 4 },
ability: { type: 'string', maxLength: 128, position: 5 },
edition: { type: 'integer', minimum: 1, position: 6 },
},
required: ['name', 'element', 'rarity', 'power', 'defense', 'edition'],
additionalProperties: false,
},
deck: {
type: 'object',
properties: {
name: { type: 'string', maxLength: 63, position: 0 },
cardIds: {
type: 'array',
items: { type: 'string', maxLength: 44 },
minItems: 5,
maxItems: 10,
position: 1,
},
},
required: ['name', 'cardIds'],
additionalProperties: false,
},
match: {
type: 'object',
properties: {
player1Id: { type: 'string', maxLength: 44, position: 0 },
player2Id: { type: 'string', maxLength: 44, position: 1 },
winnerId: { type: 'string', maxLength: 44, position: 2 },
player1Score: { type: 'integer', minimum: 0, position: 3 },
player2Score: { type: 'integer', minimum: 0, position: 4 },
timestamp: { type: 'integer', position: 5 },
},
required: ['player1Id', 'player2Id', 'winnerId', 'timestamp'],
additionalProperties: false,
},
};
// Build the token configuration using SDK classes
const localization = new TokenConfigurationLocalization(true, 'Gem', 'Gems');
const conventions = new TokenConfigurationConvention({ en: localization }, 0);
const ownerOnly = new ChangeControlRules({
authorizedToMakeChange: AuthorizedActionTakers.ContractOwner(),
adminActionTakers: AuthorizedActionTakers.ContractOwner(),
});
const noOne = new ChangeControlRules({
authorizedToMakeChange: AuthorizedActionTakers.NoOne(),
adminActionTakers: AuthorizedActionTakers.NoOne(),
});
const gemTokenConfig = new TokenConfiguration({
conventions,
conventionsChangeRules: noOne,
baseSupply: 0n,
maxSupply: 10_000_000n, // 10 million Gems total
maxSupplyChangeRules: noOne,
keepsHistory: new TokenKeepsHistoryRules({
isKeepingMintingHistory: true,
isKeepingBurningHistory: true,
isKeepingTransferHistory: true,
}),
distributionRules: new TokenDistributionRules({
perpetualDistributionRules: noOne,
newTokensDestinationIdentityRules: noOne,
mintingAllowChoosingDestination: true,
mintingAllowChoosingDestinationRules: noOne,
changeDirectPurchasePricingRules: noOne,
}),
marketplaceRules: new TokenMarketplaceRules(TokenTradeMode.NotTradeable(), noOne),
manualMintingRules: ownerOnly,
manualBurningRules: ownerOnly,
freezeRules: noOne,
unfreezeRules: noOne,
destroyFrozenFundsRules: noOne,
emergencyActionRules: noOne,
mainControlGroupCanBeModified: AuthorizedActionTakers.NoOne(),
});
Step 2: Deploy the contract
import { EvoSDK, DataContract, Document, Identifier, IdentitySigner } from '@dashevo/evo-sdk';
const sdk = EvoSDK.testnetTrusted();
await sdk.connect();
// Game operator identity
const operatorId = 'OPERATOR_IDENTITY_ID';
const operatorKey = 'OPERATOR_PRIVATE_KEY_WIF';
// Set up signing
const operatorIdentity = await sdk.identities.fetch(operatorId);
const operatorIdentityKey = operatorIdentity.publicKeys[0];
const operatorSigner = new IdentitySigner();
operatorSigner.addKeyFromWif(operatorKey);
const nonce = await sdk.identities.nonce(operatorId);
const dataContract = new DataContract({
ownerId: new Identifier(operatorId),
identityNonce: nonce + 1n,
schemas: gameSchema,
tokens: { 0: gemTokenConfig },
});
const contract = await sdk.contracts.publish({
dataContract,
identityKey: operatorIdentityKey,
signer: operatorSigner,
});
const contractId = contract.id.toString();
console.log('Game contract:', contractId);
Step 3: Mint starter Gems for a new player
When a player joins, give them starter Gems:
async function onboardPlayer(playerId: string) {
// Token operations require a CRITICAL security level key
// Gift 100 Gems to the new player
await sdk.tokens.mint({
dataContractId: new Identifier(contractId),
tokenPosition: 0,
amount: 100n,
recipientId: new Identifier(playerId),
identityId: new Identifier(operatorId),
identityKey: operatorIdentityKey,
signer: operatorSigner,
});
console.log(`Welcomed ${playerId} with 100 Gems`);
}
Step 4: Create a card pack (operator mints cards)
The operator creates cards as documents. Each card is owned by the operator initially, then transferred to players when purchased.
// Define a set of cards for a pack
const starterPack = [
{ name: 'Flame Sprite', element: 'fire', rarity: 'common', power: 15, defense: 10, edition: 1 },
{ name: 'Tidal Guardian', element: 'water', rarity: 'common', power: 10, defense: 20, edition: 1 },
{ name: 'Stone Golem', element: 'earth', rarity: 'uncommon', power: 25, defense: 30, edition: 1 },
{ name: 'Wind Dancer', element: 'air', rarity: 'common', power: 20, defense: 12, edition: 1 },
{ name: 'Shadow Wraith', element: 'shadow', rarity: 'rare', power: 40, defense: 15, edition: 1 },
];
async function createCards(cards: typeof starterPack) {
for (const card of cards) {
const cardDoc = new Document({
documentTypeName: 'card',
dataContractId: new Identifier(contractId),
ownerId: new Identifier(operatorId),
properties: card,
});
await sdk.documents.create({
document: cardDoc,
identityKey: operatorIdentityKey,
signer: operatorSigner,
});
console.log(`Created: ${card.name} (${card.rarity})`);
}
}
await createCards(starterPack);
Step 5: Player buys a card pack
The purchase flow:
- Player spends Gems (transfer to operator)
- Operator transfers card documents to the player
const PACK_PRICE = 50n; // 50 Gems per pack
async function buyPack(playerId: string, playerKey: string) {
// Set up player signing (token ops require CRITICAL security level key)
const playerIdentity = await sdk.identities.fetch(playerId);
const playerIdentityKey = playerIdentity.publicKeys[0];
const playerSigner = new IdentitySigner();
playerSigner.addKeyFromWif(playerKey);
// Player pays Gems to the operator
await sdk.tokens.transfer({
dataContractId: new Identifier(contractId),
tokenPosition: 0,
amount: PACK_PRICE,
recipientId: new Identifier(operatorId),
senderId: new Identifier(playerId),
identityKey: playerIdentityKey,
signer: playerSigner,
});
console.log(`Player paid ${PACK_PRICE} Gems`);
// Operator transfers cards to the player
// (In production, select random cards from available pool)
const availableCards = await sdk.documents.query({
dataContractId: contractId,
documentTypeName: 'card',
where: [['$ownerId', '==', operatorId]],
limit: 5,
});
for (const [cardId, card] of availableCards) {
if (!card) continue;
await sdk.documents.transfer({
document: card,
recipientId: new Identifier(playerId),
identityKey: operatorIdentityKey,
signer: operatorSigner,
});
const props = card.properties as Record<string, unknown>;
console.log(`Transferred ${props.name} to player`);
}
}
Step 6: Query a player's collection
async function getCollection(playerId: string) {
const cards = await sdk.documents.query({
dataContractId: contractId,
documentTypeName: 'card',
where: [['$ownerId', '==', playerId]],
orderBy: [['power', 'desc']],
limit: 100,
});
console.log(`\n${playerId}'s collection:`);
for (const [id, card] of cards) {
if (!card) continue;
const d = card.properties as Record<string, unknown>;
console.log(` [${d.rarity}] ${d.name} — ${d.element} — ATK:${d.power} DEF:${d.defense}`);
}
return cards;
}
Filter by rarity
const legendaries = await sdk.documents.query({
dataContractId: contractId,
documentTypeName: 'card',
where: [
['$ownerId', '==', playerId],
['rarity', '==', 'legendary'],
],
limit: 50,
});
Step 7: Trade cards between players
Player-to-player trading using document transfers:
async function tradeCards(
fromId: string, fromKey: string, fromCardId: string,
toId: string, toKey: string, toCardId: string,
) {
// Set up signers for both players
const fromIdentity = await sdk.identities.fetch(fromId);
const fromIdentityKey = fromIdentity.publicKeys[0];
const fromSigner = new IdentitySigner();
fromSigner.addKeyFromWif(fromKey);
const toIdentity = await sdk.identities.fetch(toId);
const toIdentityKey = toIdentity.publicKeys[0];
const toSigner = new IdentitySigner();
toSigner.addKeyFromWif(toKey);
// Fetch both card documents
const fromCard = await sdk.documents.get(contractId, 'card', fromCardId);
const toCard = await sdk.documents.get(contractId, 'card', toCardId);
// Player A sends their card to Player B
await sdk.documents.transfer({
document: fromCard,
recipientId: new Identifier(toId),
identityKey: fromIdentityKey,
signer: fromSigner,
});
// Player B sends their card to Player A
await sdk.documents.transfer({
document: toCard,
recipientId: new Identifier(fromId),
identityKey: toIdentityKey,
signer: toSigner,
});
console.log('Trade complete!');
}
Step 8: Record a match result
async function recordMatch(
player1Id: string, player2Id: string,
winnerId: string,
p1Score: number, p2Score: number,
) {
const matchDoc = new Document({
documentTypeName: 'match',
dataContractId: new Identifier(contractId),
ownerId: new Identifier(operatorId),
properties: {
player1Id,
player2Id,
winnerId,
player1Score: p1Score,
player2Score: p2Score,
timestamp: Date.now(),
},
});
await sdk.documents.create({
document: matchDoc,
identityKey: operatorIdentityKey,
signer: operatorSigner,
});
// Reward the winner with Gems (token ops require CRITICAL security level key)
await sdk.tokens.mint({
dataContractId: new Identifier(contractId),
tokenPosition: 0,
amount: 10n,
recipientId: new Identifier(winnerId),
identityId: new Identifier(operatorId),
identityKey: operatorIdentityKey,
signer: operatorSigner,
});
console.log(`Match recorded. ${winnerId} wins and earns 10 Gems!`);
}
Step 9: Leaderboard
Query match history to build a win count:
async function getWinCounts() {
const matches = await sdk.documents.query({
dataContractId: contractId,
documentTypeName: 'match',
orderBy: [['timestamp', 'desc']],
limit: 100,
});
const wins = new Map<string, number>();
for (const [, doc] of matches) {
if (!doc) continue;
const props = doc.properties as Record<string, unknown>;
const winner = props.winnerId as string;
wins.set(winner, (wins.get(winner) ?? 0) + 1);
}
// Sort by wins descending
const sorted = [...wins.entries()].sort((a, b) => b[1] - a[1]);
console.log('\nLeaderboard:');
sorted.forEach(([id, count], i) => {
console.log(` ${i + 1}. ${id.slice(0, 8)}... — ${count} wins`);
});
}
Architecture recap
┌──────────────────────────────────────────────────┐
│ Game Contract │
├──────────────────┬───────────────┬───────────────┤
│ card (document) │ deck (doc) │ match (doc) │
│ - name, element │ - cardIds[] │ - players │
│ - rarity, power │ │ - winner │
│ - transferable │ │ - scores │
├──────────────────┴───────────────┴───────────────┤
│ GemToken (token position 0) │
│ - in-game currency │
│ - minted as rewards, spent on packs │
└──────────────────────────────────────────────────┘
Next steps
- Add deck validation — check that a deck only contains cards the player owns
- Implement card pricing with
sdk.documents.setPrice()for a marketplace - Add seasonal editions with different
editionnumbers - Build a real-time game client that listens for match results
- Use groups for guild/clan systems with shared card pools
Tutorial: React Integration
Build a React application that connects to Dash Platform, queries data, and broadcasts state transitions. This tutorial covers SDK initialization in a React context, handling async WASM loading, and patterns for queries and mutations.
What you will learn
- Initializing the Evo SDK in a React app with proper lifecycle management
- Creating a React context/provider for SDK access
- Building hooks for queries and state transitions
- Handling loading, error, and connected states
- Working with the SDK in both development and production builds
Prerequisites
npx create-vite@latest my-dash-app -- --template react-ts
cd my-dash-app
npm install @dashevo/evo-sdk
Vite is recommended because it handles WASM imports natively. Create React App (webpack 4) requires additional configuration for WASM — Vite works out of the box.
Step 1: Create the SDK provider
The SDK must be initialized once and shared across the app. A React context is the natural fit.
src/DashProvider.tsx
import { createContext, useContext, useEffect, useState, type ReactNode } from 'react';
import { EvoSDK } from '@dashevo/evo-sdk';
interface DashContextValue {
sdk: EvoSDK | null;
isConnecting: boolean;
error: string | null;
}
const DashContext = createContext<DashContextValue>({
sdk: null,
isConnecting: true,
error: null,
});
export function useDash() {
return useContext(DashContext);
}
export function useSDK(): EvoSDK {
const { sdk } = useDash();
if (!sdk) throw new Error('SDK not connected. Wrap your app in <DashProvider>.');
return sdk;
}
interface DashProviderProps {
network?: 'testnet' | 'mainnet' | 'local';
children: ReactNode;
}
export function DashProvider({ network = 'testnet', children }: DashProviderProps) {
const [sdk, setSdk] = useState<EvoSDK | null>(null);
const [isConnecting, setIsConnecting] = useState(true);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
let cancelled = false;
async function connect() {
try {
setIsConnecting(true);
setError(null);
const instance = new EvoSDK({ network, trusted: true });
await instance.connect();
if (!cancelled) {
setSdk(instance);
}
} catch (err) {
if (!cancelled) {
setError(err instanceof Error ? err.message : 'Failed to connect');
}
} finally {
if (!cancelled) {
setIsConnecting(false);
}
}
}
connect();
return () => {
cancelled = true;
};
}, [network]);
return (
<DashContext.Provider value={{ sdk, isConnecting, error }}>
{children}
</DashContext.Provider>
);
}
src/main.tsx
import { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import { DashProvider } from './DashProvider';
import App from './App';
createRoot(document.getElementById('root')!).render(
<StrictMode>
<DashProvider network="testnet">
<App />
</DashProvider>
</StrictMode>,
);
Step 2: Build query hooks
Create reusable hooks for common queries. These handle loading and error states automatically.
src/hooks/useDashQuery.ts
import { useEffect, useState } from 'react';
import { useDash } from '../DashProvider';
import type { EvoSDK } from '@dashevo/evo-sdk';
interface QueryState<T> {
data: T | null;
isLoading: boolean;
error: string | null;
refetch: () => void;
}
export function useDashQuery<T>(
queryFn: (sdk: EvoSDK) => Promise<T>,
deps: unknown[] = [],
): QueryState<T> {
const { sdk } = useDash();
const [data, setData] = useState<T | null>(null);
const [isLoading, setIsLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [trigger, setTrigger] = useState(0);
useEffect(() => {
if (!sdk) return;
let cancelled = false;
setIsLoading(true);
queryFn(sdk)
.then((result) => {
if (!cancelled) setData(result);
})
.catch((err) => {
if (!cancelled) setError(err.message);
})
.finally(() => {
if (!cancelled) setIsLoading(false);
});
return () => { cancelled = true; };
}, [sdk, trigger, ...deps]);
return {
data,
isLoading,
error,
refetch: () => setTrigger((n) => n + 1),
};
}
Specific query hooks
src/hooks/useIdentity.ts
import { useDashQuery } from './useDashQuery';
export function useIdentity(identityId: string) {
return useDashQuery(
(sdk) => sdk.identities.fetch(identityId),
[identityId],
);
}
src/hooks/useDocuments.ts
import { useDashQuery } from './useDashQuery';
import type { DocumentsQuery } from '@dashevo/evo-sdk';
export function useDocuments(query: DocumentsQuery) {
return useDashQuery(
(sdk) => sdk.documents.query(query),
[JSON.stringify(query)],
);
}
src/hooks/useTokenBalance.ts
import { useDashQuery } from './useDashQuery';
export function useTokenBalance(identityId: string, tokenId: string) {
return useDashQuery(
async (sdk) => {
const balances = await sdk.tokens.identityBalances(identityId, [tokenId]);
for (const [id, balance] of balances) {
if (id.toString() === tokenId) return balance;
}
return 0n;
},
[identityId, tokenId],
);
}
Step 3: Build a mutation hook
For state transitions (writes), create a hook that manages submission state:
src/hooks/useDashMutation.ts
import { useState, useCallback } from 'react';
import { useSDK } from '../DashProvider';
import type { EvoSDK } from '@dashevo/evo-sdk';
interface MutationState<T> {
execute: () => Promise<T | undefined>;
isSubmitting: boolean;
error: string | null;
reset: () => void;
}
export function useDashMutation<T>(
mutationFn: (sdk: EvoSDK) => Promise<T>,
): MutationState<T> {
const sdk = useSDK();
const [isSubmitting, setIsSubmitting] = useState(false);
const [error, setError] = useState<string | null>(null);
const execute = useCallback(async () => {
try {
setIsSubmitting(true);
setError(null);
const result = await mutationFn(sdk);
return result;
} catch (err) {
setError(err instanceof Error ? err.message : 'Transaction failed');
} finally {
setIsSubmitting(false);
}
}, [sdk, mutationFn]);
return {
execute,
isSubmitting,
error,
reset: () => setError(null),
};
}
Step 4: Build components
Connection status
src/components/ConnectionStatus.tsx
import { useDash } from '../DashProvider';
export function ConnectionStatus() {
const { sdk, isConnecting, error } = useDash();
if (isConnecting) return <span className="status connecting">Connecting...</span>;
if (error) return <span className="status error">Error: {error}</span>;
if (sdk) return <span className="status connected">Connected to testnet</span>;
return null;
}
Identity viewer
src/components/IdentityViewer.tsx
import { useState } from 'react';
import { useIdentity } from '../hooks/useIdentity';
export function IdentityViewer() {
const [identityId, setIdentityId] = useState('');
const [searchId, setSearchId] = useState('');
const { data: identity, isLoading, error } = useIdentity(searchId);
return (
<div>
<h2>Fetch Identity</h2>
<form onSubmit={(e) => { e.preventDefault(); setSearchId(identityId); }}>
<input
value={identityId}
onChange={(e) => setIdentityId(e.target.value)}
placeholder="Enter identity ID"
/>
<button type="submit">Fetch</button>
</form>
{isLoading && searchId && <p>Loading...</p>}
{error && <p className="error">{error}</p>}
{identity && (
<div className="identity-card">
<p><strong>ID:</strong> {identity.id.toString()}</p>
<p><strong>Balance:</strong> {identity.balance.toString()} credits</p>
<p><strong>Public keys:</strong> {identity.publicKeys.length}</p>
</div>
)}
</div>
);
}
Document list (e.g., car listings from the previous tutorial)
src/components/ListingsList.tsx
import { useDocuments } from '../hooks/useDocuments';
const CONTRACT_ID = 'YOUR_CONTRACT_ID';
export function ListingsList() {
const { data: results, isLoading, error, refetch } = useDocuments({
dataContractId: CONTRACT_ID,
documentTypeName: 'listing',
where: [['status', '==', 'available']],
orderBy: [['priceUsd', 'asc']],
limit: 20,
});
if (isLoading) return <p>Loading listings...</p>;
if (error) return <p className="error">{error}</p>;
if (!results || results.size === 0) return <p>No listings found.</p>;
return (
<div>
<h2>Available Cars</h2>
<button onClick={refetch}>Refresh</button>
<ul>
{[...results.entries()].map(([id, doc]) => {
if (!doc) return null;
const d = doc.properties as Record<string, unknown>;
return (
<li key={id}>
<strong>{d.year} {d.make} {d.model}</strong> — ${d.priceUsd}
</li>
);
})}
</ul>
</div>
);
}
Create document form
src/components/CreateListing.tsx
import { useState, useCallback } from 'react';
import { Document, Identifier, IdentitySigner } from '@dashevo/evo-sdk';
import { useDashMutation } from '../hooks/useDashMutation';
const CONTRACT_ID = 'YOUR_CONTRACT_ID';
const IDENTITY_ID = 'YOUR_IDENTITY_ID';
const PRIVATE_KEY = 'YOUR_PRIVATE_KEY_WIF';
const SIGNING_KEY_INDEX = 0;
export function CreateListing() {
const [make, setMake] = useState('');
const [model, setModel] = useState('');
const [year, setYear] = useState(2024);
const [price, setPrice] = useState(0);
const mutation = useDashMutation(
useCallback(
async (sdk) => {
const identity = await sdk.identities.fetch(IDENTITY_ID);
const identityKey = identity.publicKeys[SIGNING_KEY_INDEX];
const signer = new IdentitySigner();
signer.addKeyFromWif(PRIVATE_KEY);
const doc = new Document({
documentTypeName: 'listing',
dataContractId: new Identifier(CONTRACT_ID),
ownerId: new Identifier(IDENTITY_ID),
properties: {
make,
model,
year,
priceUsd: price,
mileageKm: 0,
status: 'available',
},
});
return sdk.documents.create({ document: doc, identityKey, signer });
},
[make, model, year, price],
),
);
return (
<form
onSubmit={async (e) => {
e.preventDefault();
await mutation.execute();
}}
>
<h2>Create Listing</h2>
<input placeholder="Make" value={make} onChange={(e) => setMake(e.target.value)} />
<input placeholder="Model" value={model} onChange={(e) => setModel(e.target.value)} />
<input type="number" placeholder="Year" value={year} onChange={(e) => setYear(+e.target.value)} />
<input type="number" placeholder="Price (USD)" value={price} onChange={(e) => setPrice(+e.target.value)} />
<button type="submit" disabled={mutation.isSubmitting}>
{mutation.isSubmitting ? 'Submitting...' : 'Create Listing'}
</button>
{mutation.error && <p className="error">{mutation.error}</p>}
</form>
);
}
Step 5: Assemble the app
src/App.tsx
import { ConnectionStatus } from './components/ConnectionStatus';
import { IdentityViewer } from './components/IdentityViewer';
import { ListingsList } from './components/ListingsList';
import { CreateListing } from './components/CreateListing';
import { useDash } from './DashProvider';
export default function App() {
const { sdk } = useDash();
return (
<div className="app">
<header>
<h1>Dash Platform App</h1>
<ConnectionStatus />
</header>
{sdk ? (
<main>
<IdentityViewer />
<ListingsList />
<CreateListing />
</main>
) : (
<p>Waiting for SDK connection...</p>
)}
</div>
);
}
Production considerations
Private key management
The examples above hardcode private keys for clarity. In production:
- Never ship private keys in frontend code
- Use a backend service to sign state transitions, or
- Prompt the user for their mnemonic/key at runtime and keep it in memory only
- Consider the
walletnamespace for key derivation from user-provided mnemonics
import { wallet } from '@dashevo/evo-sdk';
async function signWithUserMnemonic(mnemonic: string) {
const keyInfo = await wallet.deriveKeyFromSeedPhrase({
mnemonic,
network: 'testnet',
derivationPath: "m/9'/1'/0'/0/0",
});
return keyInfo.privateKeyWif;
}
Bundle size
The WASM module adds ~2-4 MB (gzipped) to your bundle. To optimise:
- Use code splitting — the SDK module only loads when
connect()is called - Vite handles WASM lazy loading automatically
- Consider loading the SDK only on pages that need it
Error boundaries
Wrap SDK-dependent components in an error boundary to handle WASM initialization failures gracefully:
import { ErrorBoundary } from 'react-error-boundary';
<ErrorBoundary fallback={<p>Failed to load Dash SDK</p>}>
<DashProvider>
<App />
</DashProvider>
</ErrorBoundary>
Network switching
To let users switch networks at runtime, key the provider on the network value:
const [network, setNetwork] = useState<'testnet' | 'mainnet'>('testnet');
<DashProvider network={network} key={network}>
<App />
</DashProvider>
The key prop forces React to unmount and remount the provider, creating a
fresh SDK connection for the new network.
Builder Pattern
The Dash Platform Rust SDK (packages/rs-sdk) is the primary way applications interact
with Dash Platform. Before you can fetch identities, create documents, or broadcast
state transitions, you need an Sdk instance. And to create an Sdk, you use the
builder pattern.
This chapter covers SdkBuilder, the Sdk struct it produces, and the two modes
of operation: normal (real network) and mock (testing).
The Problem
Creating an Sdk requires many pieces of configuration:
- Network addresses (where are the DAPI nodes?)
- Network type (mainnet, testnet, devnet, regtest?)
- Request settings (timeouts, retries, ban policies)
- Context provider (where do cached data contracts and quorum keys come from?)
- Staleness checks (how old can metadata be before we reject it?)
- Platform version (which protocol version should we use?)
- Cancellation token (how do we abort pending requests?)
- TLS certificates (for secure connections)
Most of these have sensible defaults. A constructor with 10 parameters would be unusable. The builder pattern lets you set only what you need.
SdkBuilder
SdkBuilder lives in packages/rs-sdk/src/sdk.rs:
#![allow(unused)] fn main() { pub struct SdkBuilder { addresses: Option<AddressList>, settings: Option<RequestSettings>, network: Network, core_ip: String, core_port: u16, core_user: String, core_password: Zeroizing<String>, proofs: bool, version: &'static PlatformVersion, context_provider: Option<Box<dyn ContextProvider>>, metadata_height_tolerance: Option<u64>, metadata_time_tolerance_ms: Option<u64>, cancel_token: CancellationToken, #[cfg(feature = "mocks")] data_contract_cache_size: NonZeroUsize, #[cfg(feature = "mocks")] token_config_cache_size: NonZeroUsize, #[cfg(feature = "mocks")] quorum_public_keys_cache_size: NonZeroUsize, #[cfg(feature = "mocks")] dump_dir: Option<PathBuf>, #[cfg(not(target_arch = "wasm32"))] ca_certificate: Option<Certificate>, } }
Constructor Methods
The builder offers several constructors for different scenarios:
#![allow(unused)] fn main() { // Normal mode: connect to specified DAPI nodes let sdk = SdkBuilder::new(address_list) .with_network(Network::Testnet) .build()?; // Mock mode: no network, useful for tests let sdk = SdkBuilder::new_mock() .build()?; // Convenience (not yet implemented): let sdk = SdkBuilder::new_testnet().build()?; let sdk = SdkBuilder::new_mainnet().build()?; }
The key distinction: if addresses is Some, you get a real DapiClient that
connects to DAPI nodes over gRPC. If addresses is None (the mock path), you get
a MockDapiClient that responds with pre-programmed data.
Configuration Methods
Every builder method follows the same signature pattern: take mut self, modify a
field, return self:
#![allow(unused)] fn main() { impl SdkBuilder { pub fn with_network(mut self, network: Network) -> Self { self.network = network; self } pub fn with_settings(mut self, settings: RequestSettings) -> Self { self.settings = Some(settings); self } pub fn with_version(mut self, version: &'static PlatformVersion) -> Self { self.version = version; self } pub fn with_context_provider<C: ContextProvider + 'static>( mut self, context_provider: C, ) -> Self { self.context_provider = Some(Box::new(context_provider)); self } pub fn with_cancellation_token( mut self, cancel_token: CancellationToken, ) -> Self { self.cancel_token = cancel_token; self } } }
This lets you chain configuration fluently:
#![allow(unused)] fn main() { let sdk = SdkBuilder::new(addresses) .with_network(Network::Testnet) .with_version(PlatformVersion::latest()) .with_settings(RequestSettings { retries: Some(5), ..Default::default() }) .with_context_provider(my_provider) .with_cancellation_token(token) .build()?; }
Staleness Configuration
The SDK protects against stale responses from out-of-date nodes:
#![allow(unused)] fn main() { // Reject responses whose height is behind by more than 1 block let sdk = SdkBuilder::new(addresses) .with_height_tolerance(Some(1)) // default .with_time_tolerance(Some(360_000)) // 6 minutes .build()?; }
Height tolerance defaults to Some(1) -- if a node returns metadata with a height
more than 1 block behind the last seen height, the SDK considers it stale. Time
tolerance defaults to None (disabled) because it requires synchronized clocks.
Dash Core Integration
For development, the SDK can use Dash Core as a wallet and context provider:
#![allow(unused)] fn main() { let sdk = SdkBuilder::new(addresses) .with_core("127.0.0.1", 19998, "user", "password") .build()?; }
This is a convenience method that internally creates a GrpcContextProvider backed
by Core's RPC interface. For production, you should implement ContextProvider yourself.
Dump Directory
For debugging, the SDK can record all gRPC requests and responses to disk:
#![allow(unused)] fn main() { let sdk = SdkBuilder::new(addresses) .with_dump_dir(Path::new("./sdk-dumps")) .build()?; }
This creates files like msg-*.json, quorum_pubkey-*.json, and
data_contract-*.json that can be replayed in mock mode.
The Sdk Struct
The build() method produces an Sdk:
#![allow(unused)] fn main() { pub struct Sdk { pub network: Network, inner: SdkInstance, proofs: bool, internal_cache: Arc<InternalSdkCache>, context_provider: ArcSwapOption<Box<dyn ContextProvider>>, metadata_last_seen_height: Arc<atomic::AtomicU64>, metadata_height_tolerance: Option<u64>, metadata_time_tolerance_ms: Option<u64>, pub(crate) cancel_token: CancellationToken, pub(crate) dapi_client_settings: RequestSettings, } }
SdkInstance: Normal vs Mock
The inner field is an enum that holds either a real or mock client:
#![allow(unused)] fn main() { enum SdkInstance { Dapi { dapi: DapiClient, version: &'static PlatformVersion, }, #[cfg(feature = "mocks")] Mock { dapi: Arc<Mutex<MockDapiClient>>, mock: Arc<Mutex<MockDashPlatformSdk>>, address_list: AddressList, version: &'static PlatformVersion, }, } }
All public Sdk methods work identically in both modes. Code that uses the SDK
does not know (or care) whether it is talking to a real network or a mock.
Thread Safety
Sdk is Clone and thread-safe. It uses Arc for shared state and ArcSwapOption
for the context provider (which allows lock-free reads). The mock mode uses tokio::Mutex
for the mock client since mock state is modified in async contexts.
Nonce Management
The SDK maintains an internal cache of identity nonces to avoid querying the network on every state transition:
#![allow(unused)] fn main() { pub async fn get_identity_nonce( &self, identity_id: Identifier, bump_first: bool, settings: Option<PutSettings>, ) -> Result<IdentityNonce, Error> { // 1. Check cache // 2. If stale or absent, query Platform // 3. Optionally bump (increment) before returning // 4. Apply IDENTITY_NONCE_VALUE_FILTER mask } }
The cache has a staleness timeout (default: 20 minutes). When bump_first is true,
the nonce is incremented before being returned -- this is used when creating new
state transitions that need the next nonce value.
The Quick Mock Path
For tests that need a mock SDK immediately:
#![allow(unused)] fn main() { let sdk = Sdk::new_mock(); }
This is a shorthand for SdkBuilder::default().build().unwrap(). It creates an
SDK in mock mode with all default settings. You can then configure expectations:
#![allow(unused)] fn main() { let mut sdk = Sdk::new_mock(); sdk.mock().expect_fetch(identity, None); }
Request Settings
The SDK applies a chain of settings to every request:
#![allow(unused)] fn main() { const DEFAULT_REQUEST_SETTINGS: RequestSettings = RequestSettings { retries: Some(3), timeout: None, ban_failed_address: None, connect_timeout: None, max_decoding_message_size: None, }; }
When building, user-provided settings override defaults:
#![allow(unused)] fn main() { let dapi_client_settings = match self.settings { Some(settings) => DEFAULT_REQUEST_SETTINGS.override_by(settings), None => DEFAULT_REQUEST_SETTINGS, }; }
And when making individual requests, per-request settings override global settings:
#![allow(unused)] fn main() { let settings = sdk .dapi_client_settings .override_by(request_specific_settings); }
This three-level cascade (defaults -> builder -> per-request) gives you control without verbosity.
Rules
Do:
- Use
SdkBuilder::new(addresses)for production code with real DAPI connections. - Use
Sdk::new_mock()for quick unit tests. - Set
with_context_provider()in production -- the fallback to Core RPC is for development only. - Use
with_height_tolerance()to detect stale nodes. - Clone the SDK freely -- it is designed for shared ownership via
Arc.
Don't:
- Use
new_testnet()ornew_mainnet()-- they are not implemented yet. - Disable proofs (
with_proofs(false)) in production -- proofs are the security model. - Set
metadata_time_tolerance_mstoo low -- network delays and time skew can cause false positives. - Forget the cancellation token in long-running applications -- without it, you cannot gracefully shut down pending requests.
- Construct
Sdkdirectly -- always use the builder.
Fetch Traits
Reading data from Dash Platform is the most common SDK operation. You need to look up
an identity by its identifier, retrieve a data contract, query documents, check a
balance. The SDK provides a unified abstraction for all of these: the Fetch and
FetchMany traits.
This chapter covers how these traits work, how queries are formed, how proofs are verified, and how different Platform types plug into the system.
The Problem
Platform stores many different types of data: identities, data contracts, documents, balances, epoch info, votes, token configurations, and more. Each requires a different gRPC request, returns a different response, and needs different proof verification logic.
Without an abstraction, every type would need its own fetch function with duplicated
retry logic, proof parsing, metadata validation, and error handling. The Fetch trait
eliminates that duplication.
The Fetch Trait
Fetch is defined in packages/rs-sdk/src/platform/fetch.rs:
#![allow(unused)] fn main() { #[async_trait::async_trait] pub trait Fetch where Self: Sized + Debug + MockResponse + FromProof< <Self as Fetch>::Request, Request = <Self as Fetch>::Request, Response = <<Self as Fetch>::Request as DapiRequest>::Response, >, { /// The gRPC request type used to fetch this object. type Request: TransportRequest + Into<<Self as FromProof<<Self as Fetch>::Request>>::Request>; /// Fetch a single object from Platform. async fn fetch<Q: Query<<Self as Fetch>::Request>>( sdk: &Sdk, query: Q, ) -> Result<Option<Self>, Error> { Self::fetch_with_settings(sdk, query, RequestSettings::default()).await } /// Fetch with metadata (block height, time, etc.) async fn fetch_with_metadata<Q: Query<<Self as Fetch>::Request>>( sdk: &Sdk, query: Q, settings: Option<RequestSettings>, ) -> Result<(Option<Self>, ResponseMetadata), Error> { /* ... */ } /// Fetch with metadata and the raw proof. async fn fetch_with_metadata_and_proof<Q: Query<<Self as Fetch>::Request>>( sdk: &Sdk, query: Q, settings: Option<RequestSettings>, ) -> Result<(Option<Self>, ResponseMetadata, Proof), Error> { /* ... */ } /// Fetch with custom request settings. async fn fetch_with_settings<Q: Query<<Self as Fetch>::Request>>( sdk: &Sdk, query: Q, settings: RequestSettings, ) -> Result<Option<Self>, Error> { /* ... */ } /// Convenience: fetch by identifier. async fn fetch_by_identifier( sdk: &Sdk, id: Identifier, ) -> Result<Option<Self>, Error> where Identifier: Query<<Self as Fetch>::Request>, { Self::fetch(sdk, id).await } } }
The Key Insight: Option Semantics
Notice the return type: Result<Option<Self>, Error>.
Ok(Some(item))-- the object was found and verified.Ok(None)-- the object was proven to not exist. This is not an error; it is a cryptographic proof of absence.Err(error)-- something went wrong (network failure, proof verification failed, etc.)
This design means "not found" is a normal, expected outcome. Code that uses Fetch
does not need to handle "not found" as an error case.
Usage
#![allow(unused)] fn main() { use dash_sdk::platform::{Fetch, Identifier, Identity}; // Fetch an identity let identity = Identity::fetch(&sdk, some_identifier).await?; match identity { Some(id) => println!("Found identity with balance: {}", id.balance()), None => println!("Identity does not exist"), } }
Implementing Fetch for a Type
For most types, implementing Fetch is a one-liner:
#![allow(unused)] fn main() { impl Fetch for Identity { type Request = IdentityRequest; } impl Fetch for dpp::prelude::DataContract { type Request = platform_proto::GetDataContractRequest; } impl Fetch for drive_proof_verifier::types::IdentityBalance { type Request = platform_proto::GetIdentityBalanceRequest; } impl Fetch for drive_proof_verifier::types::IdentityNonceFetcher { type Request = platform_proto::GetIdentityNonceRequest; } impl Fetch for ExtendedEpochInfo { type Request = platform_proto::GetEpochsInfoRequest; } impl Fetch for Vote { type Request = platform_proto::GetContestedResourceIdentityVotesRequest; } impl Fetch for drive_proof_verifier::types::TotalCreditsInPlatform { type Request = platform_proto::GetTotalCreditsInPlatformRequest; } }
The type Request associates each fetchable type with its gRPC request message.
The default method implementations handle everything else -- sending the request,
parsing the proof, verifying metadata. All you need to provide is the request type.
Document: A Custom Override
Documents are special because they depend on a data contract schema for deserialization.
If the cached contract is outdated, deserialization fails. The Document implementation
overrides the default to add retry logic:
#![allow(unused)] fn main() { #[async_trait::async_trait] impl Fetch for Document { type Request = DocumentQuery; async fn fetch_with_metadata_and_proof<Q: Query<<Self as Fetch>::Request>>( sdk: &Sdk, query: Q, settings: Option<RequestSettings>, ) -> Result<(Option<Self>, ResponseMetadata, Proof), Error> { let document_query: DocumentQuery = query.query(sdk.prove())?; // First attempt with current (possibly cached) contract match fetch_request(sdk, &document_query, settings).await { Ok(result) => Ok(result), Err(e) if is_document_deserialization_error(&e) => { // Contract schema might have changed -- refetch it let fresh_query = refetch_contract_for_query(sdk, &document_query).await?; fetch_request(sdk, &fresh_query, settings).await } Err(e) => Err(e), } } } }
If deserialization fails with a CorruptedSerialization error, the SDK refetches the
data contract from the network, updates the cache, and retries. This handles the case
where a contract was updated but the local cache still has the old version.
The Query Trait
Query converts user-friendly search criteria into gRPC request messages:
#![allow(unused)] fn main() { pub trait Query<T: TransportRequest + Mockable>: Send + Debug + Clone { fn query(self, prove: bool) -> Result<T, Error>; } }
The simplest implementation: any TransportRequest is a query for itself:
#![allow(unused)] fn main() { impl<T> Query<T> for T where T: TransportRequest + Sized + Send + Sync + Clone + Debug, { fn query(self, prove: bool) -> Result<T, Error> { if !prove { unimplemented!("queries without proofs are not supported"); } Ok(self) } } }
But you can also implement Query for more ergonomic types. For example, Identifier
implements Query<GetIdentityRequest>, so you can write:
#![allow(unused)] fn main() { let identity = Identity::fetch(&sdk, my_identifier).await?; }
Instead of manually constructing a GetIdentityRequest proto message.
The FromProof Trait
Every Fetch implementation requires that the fetched type implements FromProof.
This trait, defined in packages/rs-drive-proof-verifier/src/proof.rs, verifies the
cryptographic proof returned by the Platform node:
#![allow(unused)] fn main() { pub trait FromProof<Req> { type Request; type Response; /// Parse and verify the proof, returning the requested object. /// /// Returns: /// - Ok(Some(object, metadata)) when found /// - Ok(None) when proven to not exist /// - Err when verification fails fn maybe_from_proof_with_metadata( request: Self::Request, response: Self::Response, network: Network, platform_version: &PlatformVersion, provider: &impl ContextProvider, ) -> Result<(Option<Self>, ResponseMetadata, Proof), Error> where Self: Sized; } }
The chain is: Query produces a request -> DAPI returns a response with a proof ->
FromProof verifies the proof and extracts the object. Every step is type-safe and
generic over the specific Platform type being fetched.
FetchMany: Retrieving Collections
FetchMany extends the pattern to collections:
#![allow(unused)] fn main() { pub trait FetchMany<K: Ord, O: FromIterator<(K, Option<Self>)>> where Self: Sized, O: MockResponse + FromProof<Self::Request, ...> + Send + Default, { type Request: TransportRequest; async fn fetch_many<Q: Query<Self::Request>>( sdk: &Sdk, query: Q, ) -> Result<O, Error> { /* ... */ } // ... with_settings, with_metadata, with_limit variants } }
The O type parameter is the output collection type. It must implement
FromIterator<(K, Option<Self>)> -- a collection of key-value pairs where the value
might be None (proven absent). This handles queries like "fetch documents matching
these criteria" where some requested items might not exist.
The Internal Fetch Pipeline
When you call Identity::fetch(&sdk, id), here is what happens:
-
Query conversion:
id.query(true)produces aGetIdentityRequestwith proofs enabled. -
Request execution with retry: The SDK sends the request to a DAPI node, with automatic retry logic:
#![allow(unused)] fn main() { let fut = |settings: RequestSettings| async move { let response = request.clone().execute(sdk, settings).await?; let (object, metadata, proof) = sdk .parse_proof_with_metadata_and_proof(request, response) .await?; Ok((object, metadata, proof)) }; retry(sdk.address_list(), settings, fut).await } -
Proof verification:
parse_proof_with_metadata_and_proofcallsFromProof::maybe_from_proof_with_metadata, which verifies the GroveDB proof against quorum signatures. -
Metadata validation: The SDK checks that the response metadata (height, time) is fresh enough based on the configured tolerances.
-
Result return: The verified object (or
None) is returned to the caller.
Rules
Do:
- Use
Fetch::fetch()for single-object lookups by identifier. - Use
FetchMany::fetch_many()for queries that return collections. - Handle
Ok(None)as a normal case -- it means the object does not exist, proven cryptographically. - Implement
Fetchfor new types by specifying justtype Request. - Override
fetch_with_metadata_and_proofonly when you need custom logic (like the Document retry pattern).
Don't:
- Treat
Ok(None)as an error -- "not found" is a valid, proven result. - Bypass the
Fetchtrait to make raw gRPC calls -- you would skip proof verification. - Forget to implement
FromProoffor new fetchable types -- without it, proofs cannot be verified. - Disable proofs in production --
query(prove: false)is not supported and will panic. - Implement
Queryconversions that lose information -- the query must fully specify what to fetch.
Put Operations
Reading data from Platform is handled by Fetch. Writing data -- creating documents,
deploying contracts, registering identities -- is handled by the put operation traits.
This chapter covers the write path through the SDK: how state transitions are built,
signed, broadcast, and confirmed.
The Problem
Writing to Platform is fundamentally different from reading. A read is a simple request/response: send a query, get back a proof. A write involves multiple steps:
- Determine the correct nonce for the identity.
- Build a state transition from the data to be written.
- Sign the transition with the identity's private key.
- Broadcast the signed transition to the network.
- Wait for the transition to be included in a block.
- Verify the proof that the write was applied.
Each step can fail independently, and the SDK needs to handle all of them coherently.
The PutDocument Trait
The primary write trait for documents is PutDocument, defined in
packages/rs-sdk/src/platform/transition/put_document.rs:
#![allow(unused)] fn main() { #[async_trait::async_trait] pub trait PutDocument<S: Signer<IdentityPublicKey>>: Waitable { async fn put_to_platform( &self, sdk: &Sdk, document_type: DocumentType, document_state_transition_entropy: Option<[u8; 32]>, identity_public_key: IdentityPublicKey, token_payment_info: Option<TokenPaymentInfo>, signer: &S, settings: Option<PutSettings>, ) -> Result<StateTransition, Error>; async fn put_to_platform_and_wait_for_response( &self, sdk: &Sdk, document_type: DocumentType, document_state_transition_entropy: Option<[u8; 32]>, identity_public_key: IdentityPublicKey, token_payment_info: Option<TokenPaymentInfo>, signer: &S, settings: Option<PutSettings>, ) -> Result<Document, Error>; } }
There are two methods:
put_to_platformbroadcasts the transition and returns immediately. You get back theStateTransitionthat was broadcast but no confirmation that it was applied.put_to_platform_and_wait_for_responsebroadcasts and then waits for the platform to include the transition in a block, returning the confirmedDocument.
The Nonce-Build-Broadcast-Wait Pipeline
Let's walk through put_to_platform step by step:
Step 1: Get the Nonce
#![allow(unused)] fn main() { let new_identity_contract_nonce = sdk .get_identity_contract_nonce( self.owner_id(), document_type.data_contract_id(), true, // bump_first: increment the nonce settings, ) .await?; }
Every identity has a nonce that increments with each state transition targeting a
specific contract. The SDK caches nonces internally and bumps them optimistically.
bump_first: true means "give me the next unused nonce."
The SDK's nonce management is sophisticated:
- Nonces are cached per
(identity_id, contract_id)pair. - If the cache is older than the staleness timeout (default 20 minutes), the SDK re-fetches from Platform.
- If Platform reports a higher nonce than the cache, the cache is updated.
- A filter mask (
IDENTITY_NONCE_VALUE_FILTER) is applied to keep the nonce in the valid range.
Step 2: Build the Transition
The SDK decides whether to create a new document or replace an existing one based on the document's revision:
#![allow(unused)] fn main() { let transition = if self.revision().is_some() && self.revision().unwrap() != INITIAL_REVISION { // This is an update -- create a replacement transition BatchTransition::new_document_replacement_transition_from_document( self.clone(), document_type.as_ref(), &identity_public_key, new_identity_contract_nonce, settings.user_fee_increase.unwrap_or_default(), token_payment_info, signer, sdk.version(), settings.state_transition_creation_options, ) } else { // This is a new document -- generate entropy and create let (document, entropy) = match document_state_transition_entropy { Some(entropy) => (self.clone(), entropy), None => { let mut rng = StdRng::from_entropy(); let mut document = self.clone(); let entropy = rng.gen::<[u8; 32]>(); document.set_id(Document::generate_document_id( &document_type.data_contract_id(), &document.owner_id(), document_type.name(), entropy.as_slice(), new_identity_contract_nonce, sdk.version(), )?); (document, entropy) } }; BatchTransition::new_document_creation_transition_from_document( document, document_type.as_ref(), entropy, &identity_public_key, new_identity_contract_nonce, settings.user_fee_increase.unwrap_or_default(), token_payment_info, signer, sdk.version(), settings.state_transition_creation_options, ) }?; }
For new documents, the SDK generates 32 bytes of entropy (unless you provide your own)
and derives the document ID from it and, from protocol version 14, from the identity
contract nonce it just fetched. sdk.version() tracks the network's protocol version,
so the SDK switches derivation when the network does. Because the ID depends on the
nonce, the ID on the document you pass in is a placeholder: use the ID of the confirmed
document that put_to_platform_and_wait_for_response returns.
The JavaScript SDK follows the same pipeline. sdk.documents.create goes through
put_to_platform_and_wait_for_response and hands the confirmed document back. An app
that builds the transition itself (to sign it separately or cache the signed bytes)
gets the derivation from wasm-dpp2: new DocumentCreateTransition({ document, identityContractNonce }) derives the ID from the document's entropy and the nonce for
the network's protocol version (platformVersion option, latest by default), writes it
onto the transition and back onto document, and the transition is then batched, signed
and broadcast as before. The IDs such a transition carries are final; nothing has to be
hashed on the app side.
Step 3: Validate Structure
Before broadcasting, the SDK validates the transition's basic structure:
#![allow(unused)] fn main() { ensure_valid_state_transition_structure(&transition, sdk.version())?; }
This catches obvious errors (wrong field types, missing required fields) before the transition hits the network, saving a round-trip.
Step 4: Broadcast
#![allow(unused)] fn main() { transition.broadcast(sdk, Some(settings)).await?; }
This sends the serialized transition to a DAPI node.
The BroadcastStateTransition Trait
Broadcasting is implemented as a trait on StateTransition:
#![allow(unused)] fn main() { #[async_trait::async_trait] pub trait BroadcastStateTransition { async fn broadcast( &self, sdk: &Sdk, settings: Option<PutSettings>, ) -> Result<(), Error>; async fn wait_for_response<T: TryFrom<StateTransitionProofResult> + Send>( &self, sdk: &Sdk, settings: Option<PutSettings>, ) -> Result<T, Error>; async fn broadcast_and_wait<T: TryFrom<StateTransitionProofResult> + Send>( &self, sdk: &Sdk, settings: Option<PutSettings>, ) -> Result<T, Error>; } }
Three methods, three use cases:
broadcast: Fire-and-forget. ReturnsOk(())when the node accepts the transition. The response is always empty -- confirmation comes later.wait_for_response: Poll until the transition is included in a block. Returns the proven result.broadcast_and_wait: Combines both -- broadcast, then wait.
The Wait Mechanism
wait_for_response uses the WaitForStateTransitionResult gRPC endpoint. It sends
the transition's hash and blocks until the platform includes it in a block:
#![allow(unused)] fn main() { async fn wait_for_response<T>(&self, sdk: &Sdk, settings: Option<PutSettings>) -> Result<T, Error> { let factory = |request_settings: RequestSettings| async move { let request = self.wait_for_state_transition_result_request()?; let response = request.execute(sdk, request_settings).await?; // Check for broadcast errors if let Some(e) = state_transition_broadcast_error { return Err(Error::from(e)); } // Extract and verify the proof let proof = grpc_response.proof()?; let (_, result) = Drive::verify_state_transition_was_executed_with_proof( self, &block_info, proof.grovedb_proof.as_slice(), &context_provider.as_contract_lookup_fn(sdk.version()), sdk.version(), )?; // Convert to the expected output type T::try_from(result) }; retry(sdk.address_list(), retry_settings, factory).await } }
The wait includes full proof verification: the SDK verifies a GroveDB proof that the state transition was actually applied. This is not just checking a status flag -- it is cryptographic proof of inclusion.
From protocol version 14 the proof of an owned, fee-paying transition (document
and token batches, contract creates and updates, identity updates and key limit
updates, contract moderation) also carries the credit balance of the owner
after it, read from the same state as the result, so a wallet learns what the
write left it with without a second query. The verified
StateTransitionProofOutcome hands it out through owner_balance() (None
for a proof made at an earlier version or a transition without an owner);
wait_for_document_and_owner_balance and
put_to_platform_and_wait_for_response_with_owner_balance return it next to
the document. The balance is a snapshot at the proof's block: it may already
include later transitions of the same identity. A wait that asks for no proof
but sets request_user_balance gets the owner's balance back unverified, as
the response's success_with_owner_balance result, read from a Drive state at
or past the block that executed the write; that works for any transition with
an owner.
Timeout Handling
wait_for_response supports an optional timeout:
#![allow(unused)] fn main() { match wait_timeout { Some(timeout) => { tokio::time::timeout(timeout, future) .await .map_err(|_| Error::TimeoutReached(timeout, details))? } None => future.await, } }
Without a timeout, the wait is unbounded. For production use, always set a timeout
via PutSettings.
The Waitable Trait
Waitable provides type-specific post-processing after a broadcast:
#![allow(unused)] fn main() { #[async_trait::async_trait] pub trait Waitable: Sized { async fn wait_for_response( sdk: &Sdk, state_transition: StateTransition, settings: Option<PutSettings>, ) -> Result<Self, Error>; } }
Each type implements this differently:
DataContract and Vote: straightforward delegation:
#![allow(unused)] fn main() { impl Waitable for DataContract { async fn wait_for_response( sdk: &Sdk, state_transition: StateTransition, settings: Option<PutSettings>, ) -> Result<DataContract, Error> { state_transition.wait_for_response(sdk, settings).await } } }
Document: extracts the single document from the batch transition result:
#![allow(unused)] fn main() { impl Waitable for Document { async fn wait_for_response( sdk: &Sdk, state_transition: StateTransition, settings: Option<PutSettings>, ) -> Result<Self, Error> { // Verify this is a batch transition with exactly one document let doc_id = /* extract from transition */; let mut documents: BTreeMap<Identifier, Option<Document>> = state_transition.wait_for_response(sdk, settings).await?; documents.remove(&doc_id) .ok_or(Error::InvalidProvedResponse(...))? .ok_or(Error::InvalidProvedResponse(...)) } } }
Identity: handles the "already exists" case specially by falling back to a fetch:
#![allow(unused)] fn main() { impl Waitable for Identity { async fn wait_for_response( sdk: &Sdk, state_transition: StateTransition, settings: Option<PutSettings>, ) -> Result<Self, Error> { match state_transition.wait_for_response(sdk, settings).await { Ok(identity) => Ok(identity), Err(Error::AlreadyExists(_)) => { // Identity already exists -- fetch it instead let identity_id = /* extract from transition */; Identity::fetch(sdk, identity_id).await? .ok_or(Error::Generic("proved to not exist but said to exist")) } Err(e) => Err(e), } } } }
Error Handling at the SDK Level
Errors during put operations fall into several categories:
- Nonce errors: The cached nonce was stale. The SDK refreshes nonces on broadcast
failure:
sdk.refresh_identity_nonce(&owner_id).await - Broadcast errors: The network rejected the transition. Returned as
StateTransitionBroadcastError. - Proof errors: The proof verification failed. Returned as
DriveProofErrorwith the raw proof bytes and block info for debugging. - Timeout errors: The transition was not included in time. Returned as
TimeoutReachedwith the timeout duration and a description. - Conversion errors: The proof result could not be converted to the expected
type. Returned as
InvalidProvedResponse.
PutSettings
All put operations accept optional PutSettings:
#![allow(unused)] fn main() { pub struct PutSettings { pub request_settings: RequestSettings, pub identity_nonce_stale_time_s: Option<u64>, pub user_fee_increase: Option<u16>, pub wait_timeout: Option<Duration>, pub state_transition_creation_options: Option<...>, } }
The most important field is wait_timeout. In production, always set it to avoid
hanging indefinitely.
Rules
Do:
- Use
put_to_platform_and_wait_for_responsewhen you need confirmation. - Use
put_to_platformwhen you want fire-and-forget semantics. - Always set
wait_timeoutin production. - Let the SDK manage nonces -- do not manually set them.
- Handle
Error::AlreadyExistsgracefully, especially for identity creation.
Don't:
- Call
broadcastwithout eventually callingwait_for_response-- you will not know if the transition succeeded. - Retry a failed transition without refreshing the nonce -- the old nonce may be consumed.
- Set
user_fee_increaseto zero in congested networks -- your transition may be deprioritized. - Provide custom entropy unless you need deterministic document IDs for testing.
- Ignore
TimeoutReachederrors -- they may indicate network issues that affect subsequent operations.
Identity Keys Deep Dive
Every identity on Dash Platform is controlled by a set of identity public keys. These keys determine what the identity can do: sign state transitions, encrypt messages, transfer credits, vote, or prove masternode ownership. The key system is designed around three axes -- purpose, security level, and key type -- that together define what a key is for, how sensitive it is, and what cryptographic algorithm it uses.
This chapter covers the full key lifecycle: structure, creation, storage, validation, rotation, and the GroveDB tree layout that makes lookups efficient.
Key Structure
An identity public key is represented by IdentityPublicKeyV0:
#![allow(unused)] fn main() { pub struct IdentityPublicKeyV0 { pub id: KeyID, // u32, unique within this identity pub purpose: Purpose, // what the key is used for pub security_level: SecurityLevel, // how sensitive the key is pub key_type: KeyType, // cryptographic algorithm pub read_only: bool, // if true, cannot sign state transitions pub data: BinaryData, // the public key bytes pub disabled_at: Option<TimestampMillis>, // None = active, Some = disabled pub contract_bounds: Option<ContractBounds>, // restrict to specific contract } }
Each field serves a specific role:
-
id(KeyID = u32): Sequential identifier assigned at creation. Key IDs are unique within a single identity but not globally. The ID is used to reference the key in state transitions and storage. -
data(BinaryData): The raw public key bytes. Size depends onkey_type: 33 bytes for ECDSA, 48 bytes for BLS, 20 bytes for hash-based types. -
disabled_at: When set, the key can no longer be used to sign anything. This is a timestamp (milliseconds since epoch), not a boolean, so you know exactly when the key was disabled. -
read_only: A read-only key can verify signatures but cannot be used to sign new state transitions. This is enforced at validation time.
Purpose
The Purpose enum defines what a key is authorized to do:
| Purpose | Value | Description |
|---|---|---|
AUTHENTICATION | 0 | General-purpose signing. Every identity must have at least one MASTER-level authentication key. |
ENCRYPTION | 1 | Encrypt data. Cannot sign documents or state transitions. |
DECRYPTION | 2 | Decrypt data. Cannot sign documents or state transitions. |
TRANSFER | 3 | Sign credit transfers, withdrawals, and token operations. Required at CRITICAL security level. |
SYSTEM | 4 | System operations. Cannot sign documents. |
VOTING | 5 | Cast masternode votes. Cannot sign documents. |
OWNER | 6 | Prove ownership of a masternode or evonode. |
Purposes are grouped by searchability in the storage layer:
- Searchable:
AUTHENTICATION,TRANSFER,VOTING-- these get indexed in the key reference tree so they can be looked up by purpose and security level. - Non-searchable:
ENCRYPTION,DECRYPTION,SYSTEM,OWNER-- stored but not indexed for search.
The practical effect: if a Platform node needs to find "the TRANSFER key for identity X", it can do a direct tree lookup. But finding "the ENCRYPTION key for identity X" requires fetching all keys and filtering client-side.
Security Level
Security levels form a strict hierarchy:
MASTER (0) > CRITICAL (1) > HIGH (2) > MEDIUM (3)
strongest weakest
The numeric value is inverted from what you might expect: lower value = stronger
security. This matters because many operations check
key.security_level().stronger_or_equal_security_than(required_level).
What Security Level Controls
-
What the key can sign. A data contract can require that documents be signed with at least a certain security level. A key at MEDIUM cannot sign a document that requires HIGH or above.
-
What operations the key can perform. Some state transitions require specific security levels:
- Adding/disabling other keys requires MASTER
- Credit transfers require CRITICAL (enforced via the TRANSFER purpose)
- Document operations accept CRITICAL down to the level each document type
requires (
signatureSecurityLevelRequirement, HIGH by default), so MEDIUM only where a type asks for it; a batch holding a token transition needs CRITICAL
-
Which purposes allow which levels. Not all combinations are valid for externally added keys (i.e., keys added via identity create/update transitions):
Purpose Allowed Security Levels AUTHENTICATION MASTER, CRITICAL, HIGH, MEDIUM ENCRYPTION MEDIUM only DECRYPTION MEDIUM only TRANSFER CRITICAL only SYSTEM Not externally addable (platform-managed) VOTING Not externally addable (platform-managed) OWNER Not externally addable (platform-managed) SYSTEM, VOTING, and OWNER keys are created automatically by the platform (e.g., during masternode registration) and cannot be added through state transitions. Attempting to add a key with one of these purposes will fail validation. Similarly, attempting to create a TRANSFER key at HIGH security level will fail because only CRITICAL is allowed for that purpose.
The Master Key Requirement
Every identity must have exactly one MASTER-level AUTHENTICATION key at creation time. This key is the identity's root of trust -- it can add new keys, disable other keys, and perform any operation. Losing access to the master key means losing the ability to manage the identity's key set.
Key Type
The KeyType enum determines the cryptographic algorithm and key size:
| Key Type | Value | Size | Unique | Description |
|---|---|---|---|---|
ECDSA_SECP256K1 | 0 | 33 bytes | Yes | Standard Bitcoin/Dash curve. Default. |
BLS12_381 | 1 | 48 bytes | Yes | BLS signatures, used by masternodes. |
ECDSA_HASH160 | 2 | 20 bytes | No | RIPEMD160(SHA256) of an ECDSA public key. Core address type. |
BIP13_SCRIPT_HASH | 3 | 20 bytes | No | Script hash. Core address type. |
EDDSA_25519_HASH160 | 4 | 20 bytes | No | RIPEMD160(SHA256) of an Ed25519 public key. |
Unique vs Non-Unique Keys
This distinction is critical for understanding how keys are stored and enforced:
-
Unique key types (ECDSA_SECP256K1, BLS12_381): The full public key is stored, and Platform enforces that no two identities can register the same public key. This is checked in both the unique and non-unique hash tables during insertion. If identity A registers an ECDSA key, identity B cannot register the same key bytes.
-
Non-unique key types (ECDSA_HASH160, BIP13_SCRIPT_HASH, EDDSA_25519_HASH160): Only a 20-byte hash is stored. Multiple identities can share the same hash. This makes sense for address-based key types where the same Dash address might legitimately be associated with multiple identities (e.g., through asset lock transactions).
Core Address Key Types
ECDSA_HASH160 and BIP13_SCRIPT_HASH are specifically for linking Platform identities
to Layer 1 (Core) Dash addresses. They store the same 20-byte hash used in Core
addresses, enabling cross-layer identity verification without revealing the full public
key.
Contract Bounds
A key can optionally be restricted to one data contract, one document type of a contract, or every member of a contract group:
#![allow(unused)] fn main() { pub enum ContractBounds { /// Key can only be used within a specific contract SingleContract { id: Identifier }, /// Key can only be used within a specific contract and document type SingleContractDocumentType { id: Identifier, document_type_name: String, }, /// Key can only be used within the members of a contract group (protocol version 14) ContractGroup { id: Identifier }, } }
What the bounds mean depends on the key's purpose:
- ENCRYPTION and DECRYPTION keys: a hint to clients about which contract the key serves.
The bound contract, or document type, must opt in with
requiresIdentityEncryptionBoundedKeyorrequiresIdentityDecryptionBoundedKey, which also fixes the storage rule (unique, multiple, or multiple with a pointer to the latest). These keys cannot be bound to a group. - AUTHENTICATION keys (protocol version 14): a restriction consensus enforces. The key may sign only batch transitions, and every member must be inside the bounds: on the bound contract, of the bound document type, or, for a group bound, a member of the group (the whole contract, the document type, or the token). Any contract or group may be bound, the key cannot be MASTER, and a member outside the bounds is a paid failure.
See docs/protocol/contract-bound-authentication-keys.md for the exact rules and errors.
This enables fine-grained delegation: an identity owner can create a key that is only allowed to interact with one specific dApp, or with one project's set of contracts, limiting exposure if that key is compromised.
Budget and Expiry
Bounds limit where a key may act. From protocol version 14 an AUTHENTICATION key below the
MASTER level can also be limited in how much it may spend and for how long it may sign. The
limits live on the version 1 key, IdentityPublicKeyV1, which is the version 0 key followed by
two optional fields:
#![allow(unused)] fn main() { pub struct IdentityPublicKeyV1 { // ... the IdentityPublicKeyV0 fields, in the same order ... pub total_budget: Option<Credits>, // total credits the key may take from the identity pub expires_at: Option<TimestampMillis>, // block time from which the key no longer signs } }
Version 0 keys are untouched: keys already in state decode as before and a key without limits
is still written as version 0. Read the limits through IdentityPublicKeyGettersV1
(total_budget(), expires_at(), is_expired_at(time_ms), has_limits()), which answers None for
a version 0 key, and turn a key into a limited one with IdentityPublicKey::with_limits.
total_budgetcaps everything state transitions signed with the key take from the identity: fees (net of the storage refunds the same transition returns) and credits moved out, such as a document purchase price. What is left is tracked by Drive next to the key and only goes down. Before a transition runs, everything but the metered processing fee must fit in what is left; that processing fee may take the key over its budget once, after which the key can no longer sign. This is the rule identity balances follow (storage must be covered, processing may leave a debt), applied to the key.expires_atis compared with the time of the block the transition executes in. The key signs atexpires_at - 1and not atexpires_at. A key cannot be registered already expired.
The limits are part of the signable bytes of the transition that registers the key and cannot
be changed afterwards. An identity created from the shielded pool is the exception: it has no
identity signature and its sighash does not cover the limits, so a key that carries a budget or
an expiry is refused there and has to be added with an identity update (a version 1 key without
limits is accepted). Refusals for a spent, exceeded or expired key leave the transition
unpaid, like an identity that cannot afford its fee. Key Budgets and Expiry explains the design: the budget rule, where
each check runs in the validation pipeline, and how Drive keeps the running total. The rules
and error codes as a reference are in docs/protocol/authentication-key-limits.md.
Storage in GroveDB
Identity keys are stored across multiple trees in GroveDB for efficient access patterns.
Identity-Level Trees
Each identity has its own subtree under the root Identities tree:
Identities [RootTree::Identities]
└── {identity_id (32 bytes)}
├── IdentityTreeKeys [128]
│ └── {key_id (varint)} → serialized IdentityPublicKey
│
├── IdentityTreeKeyReferences [160]
│ ├── AUTHENTICATION [0]
│ │ ├── MASTER [0]
│ │ │ └── {key_id} → reference to IdentityTreeKeys
│ │ ├── CRITICAL [1]
│ │ │ └── ...
│ │ ├── HIGH [2]
│ │ │ └── ...
│ │ └── MEDIUM [3] ← pre-created at identity creation
│ │ └── ...
│ ├── TRANSFER [3]
│ │ └── {key_id} → reference
│ └── VOTING [5]
│ └── {key_id} → reference
│
├── IdentityTreeRevision [192]
├── IdentityTreeNonce [64]
├── IdentityContractInfo [32]
└── IdentityTreeKeyBudgets [224] ← created with the first budgeted key
└── {key_id (varint)} → remaining budget (u64, 8 bytes big endian)
IdentityTreeKeys stores the actual serialized key data, keyed by the key ID encoded as a varint.
IdentityTreeKeyReferences provides a searchable index organized by purpose and (for AUTHENTICATION) security level. Each entry is a GroveDB reference pointing back to the actual key in IdentityTreeKeys.
The MEDIUM security level subtree under AUTHENTICATION is pre-created during identity initialization, even if no MEDIUM keys exist yet. Other security level subtrees are created on-demand when a key with that level is first added.
IdentityTreeKeyBudgets holds what is left of the budget of each budgeted key. The key itself is immutable, so the running value lives here. It is fixed width so that spending from a budget replaces the value without changing what is stored, which lets the deduction be applied outside of the fee, like the balance change it accompanies.
Global Key Hash Tables
Two root-level trees provide reverse lookups from key hashes to identity IDs:
UniquePublicKeyHashesToIdentities [24]
└── {key_hash (20 bytes)} → identity_id (32 bytes)
NonUniquePublicKeyKeyHashesToIdentities [8]
└── {key_hash (20 bytes)}
└── {identity_id (32 bytes)} → empty item
The unique table is a flat mapping: one hash to one identity. Insertion fails if the hash already exists in either table.
The non-unique table uses a nested structure: each key hash has a subtree containing identity IDs as keys. This allows multiple identities to share the same key hash.
Key Hash Computation
All key types are hashed to 20 bytes for storage in the hash tables:
- ECDSA_SECP256K1 (33 bytes):
RIPEMD160(SHA256(pubkey)) - BLS12_381 (48 bytes):
RIPEMD160(SHA256(pubkey)) - ECDSA_HASH160 (20 bytes): stored as-is (already a hash)
- BIP13_SCRIPT_HASH (20 bytes): stored as-is
- EDDSA_25519_HASH160 (20 bytes): stored as-is
Key Lifecycle
Creation
Keys are added to an identity either at identity creation or via an IdentityUpdate state transition.
At Identity Creation:
- The state transition includes
IdentityPublicKeyInCreationobjects - Validation enforces exactly one MASTER-level AUTHENTICATION key
- Each key also carries a
signaturefield (proving the creator holds the private key) - After validation, keys are converted to
IdentityPublicKeywithdisabled_at = None
Via IdentityUpdate:
- The
add_public_keysfield carries newIdentityPublicKeyInCreationobjects - Multiple keys can be added in a single transition
- The transition must be signed by a key with sufficient security level
- New key IDs must not collide with existing keys on the identity
The insertion process differs by key type:
- Unique keys: Check both hash tables for conflicts, insert into
UniquePublicKeyHashesToIdentities, insert key data, create references. - Non-unique keys: Create subtree under hash if needed, insert identity ID, insert key data, create references.
Disabling
Keys are disabled (not deleted) via the disable_public_keys field of IdentityUpdate:
#![allow(unused)] fn main() { // IdentityUpdateTransition fields (simplified) pub add_public_keys: Vec<IdentityPublicKeyInCreation>, pub disable_public_keys: Vec<KeyID>, }
When a key is disabled:
- The key is fetched from storage.
disabled_atis set to the current block timestamp (milliseconds).- The serialized key is replaced in
IdentityTreeKeys. - Key references in
IdentityTreeKeyReferencesare refreshed.
A disabled key:
- Cannot be used to sign any state transition (checked during signature verification)
- Remains in storage (can still be read)
- Can be re-enabled in the future
Re-enabling
Keys can be re-enabled by clearing the disabled_at field:
- The key is fetched from storage.
disabled_atis set toNone.- The serialized key is replaced.
- References are refreshed.
Masternode Keys
Masternode identities have a special rule: all their keys are registered as non-unique, regardless of key type. This allows the same BLS key to be used across multiple masternode identities (e.g., during key rotation or when the same operator runs multiple masternodes).
Signing and Verification
When a state transition is signed:
- The transition specifies which key ID it was signed with.
- The key is looked up on the signing identity.
- Validation checks:
- The key exists and is not disabled (
disabled_atmust beNone) - The key's purpose allows this type of state transition
- The key's security level meets the minimum required
- If the key has
contract_bounds, the transition targets the bound contract - If the key is
read_only, it cannot sign - If the key has a
total_budget, some of it is left Two more checks on a limited key need the block time and the fee, so they run with fee validation instead: the key has not expired, and what the transition requires from the budget fits in what is left.
- The key exists and is not disabled (
- The signature is verified using the appropriate algorithm:
- ECDSA_SECP256K1: standard secp256k1 signature verification
- BLS12_381: BLS signature verification
- ECDSA_HASH160: ECDSA verification (key data is a hash, so verification uses the hash comparison path)
Validation Rules Summary
At identity creation:
- Exactly 1 MASTER-level AUTHENTICATION key required
- No duplicate key IDs in the transition
- No duplicate key data for unique key types
- Key count must not exceed
max_public_keys_in_creation(platform config) - Each key's purpose/security level combination must be in the allowed set
At key addition (IdentityUpdate):
- New key IDs must not conflict with existing keys
- Unique key hashes must not exist in either the unique or non-unique global tables
- The signing key must have sufficient security level to add keys
- All purpose/security level constraints apply
- A budget or an expiry is only allowed on an AUTHENTICATION key below MASTER; the budget is not zero and the expiry is after the block time
At signing time:
- Key must not be disabled
- Key purpose must match the operation
- Key security level must be >= the required level
- Contract bounds must match (if set)
- Key must not be read-only
- Key must not be expired, and its budget must cover the transition (if set)
Querying Keys
Find identity by public key hash:
#![allow(unused)] fn main() { // Returns the identity ID that owns this unique public key let identity_id = drive.fetch_identity_id_by_unique_public_key_hash( key_hash, transaction, platform_version )?; }
Find all identities sharing a non-unique key hash:
#![allow(unused)] fn main() { // Returns all identity IDs registered under this non-unique key hash let identity_ids = drive.fetch_identity_ids_by_non_unique_public_key_hash( key_hash, transaction, platform_version )?; }
Fetch all keys for an identity:
#![allow(unused)] fn main() { let keys = identity.public_keys(); // BTreeMap<KeyID, IdentityPublicKey> }
Fetch by purpose and security level:
Uses the IdentityTreeKeyReferences tree to efficiently look up keys without
scanning all keys on the identity.
Design Rationale
Why separate purpose and security level? Purpose defines what a key can do; security level defines how sensitive it is. An AUTHENTICATION key at MEDIUM can sign low-sensitivity documents. An AUTHENTICATION key at MASTER can manage the identity itself. This separation lets identities create keys with exactly the right capabilities -- not too much, not too little.
Why disable instead of delete? Disabled keys remain in storage so that historical signatures can still be verified. If a key were deleted, past state transitions signed by that key would become unverifiable.
Why unique vs non-unique hash tables? Full public keys (ECDSA, BLS) must be globally unique to prevent impersonation. But hash-based keys (ECDSA_HASH160, BIP13_SCRIPT_HASH) represent Dash addresses that may legitimately appear in multiple identities -- for example, when the same address is used in multiple asset lock transactions.
Why pre-create the MEDIUM AUTHENTICATION subtree? This is the most commonly used security level for document signing. Pre-creating its tree at identity creation avoids the cost of creating it on the first document submission.
Why contract bounds? They enable the principle of least privilege. An identity can create a key specifically for interacting with one dApp. If that key is compromised, the damage is limited to that single contract -- the attacker cannot use it to transfer credits or interact with other contracts.
BLAST Sync
Blockchain Layered Address Sync Tree (BLAST) is a privacy-preserving synchronization algorithm used by the Dash Platform SDK. It allows wallets to discover which of their keys exist in a server-side Merkle tree without revealing the specific keys being queried.
BLAST is used for two distinct sync tasks:
- Address balance sync: Discovering which platform addresses have balances and what those balances are.
- Nullifier sync: Checking which nullifiers have been spent in the shielded pool.
Both follow the same trunk/branch tree-scan pattern, extracted into a shared generic algorithm.
Why BLAST Preserves Privacy
The naive approach to syncing — querying each address individually — leaks the wallet's full key set to the server. Even batching all keys into a single request reveals the exact set. The server learns which addresses belong to the same wallet, enabling transaction graph analysis and balance tracking.
BLAST solves this by never querying individual keys. Instead, the wallet queries subtrees of the Merkle tree. Each query returns a chunk of the tree containing the target key alongside many other unrelated keys. The server sees a request for a region of the tree but cannot determine which specific key within that region the wallet cares about.
The privacy guarantee comes from three mechanisms:
-
Trunk query hides intent entirely. The initial trunk query fetches the top levels of the entire tree — the same request regardless of which keys the wallet holds. The server learns nothing from this query.
-
Branch queries request subtrees, not keys. When the wallet needs to resolve a key that traced to a truncated leaf, it requests the subtree rooted at that leaf. The response contains all keys in that subtree, not just the target. The server knows the wallet is interested in some key in the subtree, but not which one.
-
Privacy adjustment enlarges small subtrees. If a leaf's subtree contains fewer than
min_privacy_countelements (default: 32), the wallet queries an ancestor subtree instead — one that contains at least 32 keys. This prevents the server from narrowing down the target when a subtree is too small. For example, if the target key's leaf subtree has only 3 elements, querying it would reveal the target with ~33% probability. By querying an ancestor with 50+ elements, the probability drops to ~2%. -
Incremental catch-up queries block ranges, not addresses. The recent and compacted change queries ask "what changed since block height X?" — they return all address balance changes for all addresses in those blocks, not just the wallet's addresses. The server sees a generic block-range query identical to what any other wallet would send. The wallet filters the results client-side, discarding changes for addresses it doesn't own.
The result: for each sync, the server sees a trunk query (identical for all wallets) plus a small number of branch queries for subtrees containing 32+ keys each, plus block-range queries that reveal no address-specific information. It cannot distinguish which specific keys the wallet owns, how many keys it has, or whether two branch queries target different keys or are privacy-adjusted queries for the same key.
Performance
BLAST sync scales logarithmically with the total number of addresses in the platform state tree. A wallet with 20 addresses performs roughly the same number of queries whether the tree contains 1,000 or 1,000,000,000 addresses.
Estimated Performance for 20 Wallet Addresses
The total query count is Q = 1 (trunk) + 2k × iterations + 1 (recent) where each
key requires 2 branch queries (left + right child) per iteration. See the
Appendix: Query Count Analysis for the full derivation.
xychart-beta
title "Total Queries (full scan, 20 wallet addresses)"
x-axis "Total addresses in tree" ["1K", "10K", "100K", "1M", "10M", "100M", "1B"]
y-axis "Queries" 0 --> 100
bar [42, 42, 42, 62, 72, 82, 82]
xychart-beta
title "Estimated Sync Time (10 concurrent, ~330ms/round)"
x-axis "Total addresses in tree" ["1K", "10K", "100K", "1M", "10M", "100M", "1B"]
y-axis "Seconds" 0 --> 7
line [2.0, 2.0, 2.0, 4.0, 4.6, 5.3, 5.3]
| Total addresses | Tree depth | d_trunk | d_branch | Branch iters | Branch queries | Total queries | Parallel rounds | Est. sync time |
|---|---|---|---|---|---|---|---|---|
| 1,000 | 10 | 9 | 9 | 1 | 40 | 42 | 6 | ~2.0s |
| 10,000 | 14 | 9 | 9 | 1 | 40 | 42 | 6 | ~2.0s |
| 100,000 | 17 | 9 | 9 | 1 | 40 | 42 | 6 | ~2.0s |
| 1,000,000 | 20 | 9 | 9 | 2 | 60 | 62 | 8 | ~4.0s |
| 10,000,000 | 23 | 9 | 9 | 2 | 70 | 72 | 9 | ~4.6s |
| 100,000,000 | 27 | 9 | 9 | 3 | 80 | 82 | 10 | ~5.3s |
| 1,000,000,000 | 30 | 9 | 9 | 3 | 80 | 82 | 10 | ~5.3s |
How to read the table: At 100K addresses (depth 17), the trunk covers 9 levels and one branch iteration covers up to 9 more levels (total 18 > 17). All 20 keys resolve in a single round of 40 branch queries. At 1M (depth 20), trunk + one iteration covers 18 levels — 2 levels short, so ~50% of keys need a second round, adding ~20 queries.
Assumptions: d_trunk = 9, d_branch = 9 (platform version max_depth), P = 32,
2 branch queries per key per iteration (left + right child), 10 concurrent branch
queries, ~330ms per parallel round (measured empirically: ~4s for 1M addresses).
Sync time = rounds × 330ms where rounds = 1 (trunk) + ⌈branch_queries / 10⌉ + 1 (recent).
Note: iteration transitions add overhead, so multi-iteration syncs are slightly slower
than the formula suggests.
Incremental sync (subsequent 15-second re-syncs): 1 query, ~330ms — only the recent changes query, no trunk/branch. If compaction occurred: +1 compacted query.
Key Insight
Going from 1,000 to 1,000,000,000 addresses — a million-fold increase — adds only
~40 queries and ~2.6 seconds. The query count grows as O(k · ⌈log₂(N) / d_branch⌉)
where each step up in tree depth beyond d_trunk + d_branch adds one iteration of
~20 queries (diminishing as keys resolve). With 10-way parallelism, each additional
iteration costs only ~2 extra parallel rounds (~660ms).
Full Sync Flowchart
flowchart TD
Start["<b>sync_address_balances</b><br/>called by wallet"] --> CheckTS{"last_sync_timestamp<br/>provided?"}
CheckTS -->|"No / zero"| FullScan1["<b>FULL SCAN</b><br/>First sync ever"]
CheckTS -->|"Yes"| CheckElapsed{"Elapsed time ><br/>full_rescan_after?<br/><i>default: 7 days</i>"}
CheckElapsed -->|"Yes - stale"| FullScan2["<b>FULL SCAN</b><br/>Data too old"]
CheckElapsed -->|"No - recent"| Incremental["<b>INCREMENTAL ONLY</b><br/>Seed from current_balances<br/>start_height = last_sync_height"]
FullScan1 --> Trunk
FullScan2 --> Trunk
Trunk["<b>Trunk Query</b><br/>Fetch top N levels of Merk tree<br/>Verify proof against quorum-signed root<br/>Sets checkpoint_height from metadata"] --> Classify["<b>Classify Keys</b><br/>BST traversal for each target address<br/><i>Found</i> → record balance<br/><i>Traced to leaf</i> → add to tracker<br/><i>Absent</i> → proven not in tree"]
Classify --> Privacy["<b>Privacy Adjust</b><br/>Leaves with count < 32:<br/>query ancestor subtree instead<br/>Dedup: multiple keys → one query"]
Privacy --> Branch["<b>Branch Queries</b><br/>Parallel: up to 10 concurrent<br/>Iterative: up to 50 rounds<br/>Each round resolves deeper leaves<br/>Gap limit extension after each round"]
Branch -->|"catch_up_from =<br/>checkpoint_height"| Recent
Incremental --> Recent
Recent["<b>Query Recent</b><br/>Single query - no pagination needed<br/>Recent tree max 64 blocks < 100 limit<br/><b>Hold results</b> - do not apply yet<br/>Record observed_tip from metadata"] --> CompCheck{"<b>Compaction Check</b><br/>Has last_known_recent_block?"}
CompCheck -->|"Yes: use RangeAfter<br/>key_exists_as_boundary<br/>on last_known_recent_block"| BoundaryCheck{"Boundary exists<br/>in proof?"}
CompCheck -->|"No: first sync or<br/>empty recent tree"| Compacted
BoundaryCheck -->|"Yes: not compacted"| ApplyRecent
BoundaryCheck -->|"No: compacted away"| Compacted
Compacted["<b>Query Compacted</b><br/>Paginated: 25 ranges per batch<br/>Aggregated block ranges with<br/>SetCredits or AddToCreditsOperations<br/>Apply immediately, advance height"] --> ApplyRecent
ApplyRecent["<b>Apply Held Recent</b><br/>Process per-block entries from recent query<br/>Update balances, call on_address_found<br/>Advance current_height per block"] --> Finalize
Finalize["<b>Finalize</b><br/>new_sync_height = max of current and observed_tip<br/>new_sync_timestamp = latest metadata time<br/>checkpoint_height = trunk query height<br/><i>Caller persists for next sync</i>"]
style FullScan1 fill:#c53030,color:#fff
style FullScan2 fill:#c53030,color:#fff
style Incremental fill:#2b6cb0,color:#fff
style Trunk fill:#1a365d,color:#e2e8f0
style Classify fill:#1a365d,color:#e2e8f0
style Privacy fill:#1a365d,color:#e2e8f0
style Branch fill:#1a365d,color:#e2e8f0
style Recent fill:#1a3a2a,color:#e2e8f0
style CompCheck fill:#744210,color:#fff
style Compacted fill:#c05621,color:#fff
style ApplyRecent fill:#1a3a2a,color:#e2e8f0
style Finalize fill:#2d3748,color:#e2e8f0
The Problem
A wallet holds a set of keys (addresses or nullifiers) and needs to learn which ones exist in a Merkle tree stored by Platform nodes. The naive approach -- querying each key individually -- leaks the wallet's full key set to the server. Even batching the keys into a single request reveals the exact set.
BLAST solves this by querying subtrees of the Merkle tree rather than individual keys. The server returns a chunk of the tree that contains the target key along with many other keys, making it impossible for the server to determine which specific key the wallet cares about.
Algorithm Overview
The sync has two phases: a tree scan for bulk discovery, and incremental catch-up for staying current between scans.
Phase 1: Tree Scan (Trunk/Branch)
The Underlying Data Structure
Platform stores addresses and nullifiers in a Merk tree -- a balanced binary search tree (BST) where each node is keyed and ordered. Every internal node has a left child (keys less than this node) and a right child (keys greater than this node). Each node also carries a Merkle hash of its subtree, making the entire structure cryptographically verifiable.
The trunk query returns a partial view of this BST: the top N levels are fully expanded (you can see the actual keys and values), while deeper subtrees are truncated to hash placeholders. The boundary between "expanded" and "truncated" defines the leaf nodes of the trunk result.
┌──────┐
│ 30 │ ← Root node (key=30)
└──┬───┘
┌──────┴──────┐
▼ ▼
┌──────┐ ┌──────┐
│ 15 │ │ 45 │ ← Internal nodes (expanded)
└──┬───┘ └──┬───┘
┌────┴────┐ ┌────┴────┐
▼ ▼ ▼ ▼
┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐
│ 7 │ │ 22 │ │ 38 │ │ 55 │ ← Leaf nodes
│▓▓▓▓▓│ │▓▓▓▓▓│ │▓▓▓▓▓│ │▓▓▓▓▓│ (children are
└─────┘ └─────┘ └─────┘ └─────┘ hash placeholders)
▓▓▓ = truncated subtree (only hash known, not contents)
The trunk result contains three key pieces of data:
elements: ABTreeMap<Vec<u8>, Element>of key-value pairs at expanded nodes. These are fully resolved -- the wallet can read their values directly.leaf_keys: ABTreeMap<Vec<u8>, LeafInfo>of nodes at the truncation boundary. EachLeafInfohas ahash(for verifying subsequent branch queries) and an optionalcount(number of elements in the truncated subtree).tree: The reconstructed BST structure from the proof, used for key tracing.
Step 1: The Trunk Query
The wallet sends a single trunk query to a Platform node. The request specifies a
max_depth (how many levels of the BST to expand). The server returns the trunk
elements, the leaf boundary information, and a Merkle proof covering the entire
result.
The proof is verified against the quorum-signed root hash, ensuring the server cannot lie about what the tree contains.
#![allow(unused)] fn main() { let (trunk_result, metadata) = PlatformAddressTrunkState::fetch_with_metadata(sdk, (), Some(settings)).await?; }
Step 2: Classifying Target Keys via BST Traversal
After receiving the trunk, the wallet classifies each of its target keys by traversing
the BST structure. The trace_key_to_leaf method performs a standard binary search:
- Start at the root node.
- Compare the target key against the current node's key.
- If equal: the key is found in the trunk elements.
- If less: follow the left child.
- If greater: follow the right child.
- If the current node is a leaf (its children are hash placeholders): the target key is somewhere in this leaf's truncated subtree, but we can't resolve it yet.
- If there is no child to follow: the key is proven absent.
This produces exactly three outcomes for each target key:
| Outcome | What it means | Action |
|---|---|---|
| Found | Key exists in trunk elements | Record the value (balance, spent status) |
| Traced to leaf | Key is in a truncated subtree | Add to KeyLeafTracker for branch querying |
| Absent | No path exists in the BST | Key is cryptographically proven to not exist |
#![allow(unused)] fn main() { for key in target_keys { if trunk_result.elements.contains_key(&key) { // Found directly in trunk -- record it result.found.insert(key); } else if let Some((leaf_key, info)) = trunk_result.trace_key_to_leaf(&key) { // Traces to a leaf subtree -- need a branch query tracker.add_key(key, leaf_key, info); } else { // Proven absent from the tree result.absent.insert(key); } } }
A concrete example: suppose the wallet is looking for key 20 in the tree above.
The BST traversal goes: root 30 (20 < 30, go left) -> node 15 (20 > 15, go right)
-> leaf 22 (children are hash placeholders). Key 20 traces to leaf 22 because
it would be in leaf 22's left subtree. The wallet now knows it needs to query
leaf 22's subtree to determine whether key 20 actually exists.
Note that each target key traces to exactly one leaf -- the BST path is deterministic. Multiple target keys may trace to the same leaf if they are close together in the key space.
Step 3: Privacy Adjustment
Before querying leaf subtrees, the algorithm applies privacy adjustment to prevent the server from learning which specific keys the wallet cares about.
Each leaf in the trunk result has an optional count -- the number of elements in its
truncated subtree. If this count is small (below min_privacy_count, default 32), then
querying that specific leaf reveals too much: the server knows the wallet is interested
in one of only a few keys.
The fix is to query an ancestor higher in the tree that has enough elements to provide cover:
Suppose leaf "22" has count=5 (too small for privacy).
Its parent "15" has count=50 (enough).
Instead of asking: "give me the subtree rooted at 22"
The wallet asks: "give me the subtree rooted at 15"
Now the server sees a query for a 50-element subtree and cannot
tell whether the wallet wants key 20 (in 22's subtree) or
key 10 (in 7's subtree) or any other key under 15.
The get_ancestor method walks up the BST path from the leaf to the root, stopping
at the first ancestor whose count exceeds min_privacy_count. It never returns the
root itself (that would be equivalent to re-fetching the entire trunk). If no ancestor
has enough count, it falls back to the node one level below the root.
The query depth is adjusted when using an ancestor: since the ancestor is higher in the tree, its subtree is deeper, so the depth parameter is reduced by the number of levels climbed.
Deduplication ensures that if multiple target keys expand to the same ancestor, only one branch query is sent.
Step 4: Iterative Branch Queries
For each leaf (or privacy-adjusted ancestor) with unresolved keys, the wallet sends a branch query specifying:
- The leaf's key (identifies which subtree to expand)
- The query depth (how many levels to expand)
- The expected root hash (the leaf's
hashfrom the trunk, used for verification) - The checkpoint height (ensures the branch matches the same tree snapshot as the trunk)
The server returns a GroveBranchQueryResult with the same structure as the trunk:
expanded elements, new leaf keys at the next truncation boundary, and a Merk proof.
The wallet verifies the proof against the expected hash from the parent query.
Each target key in the queried subtree is classified again:
- Found in the branch elements -- resolved.
- Traced to a deeper leaf -- the key is in an even deeper truncated subtree.
The
KeyLeafTrackeris updated to point to the new, deeper leaf. - Absent -- proven to not exist within this subtree.
Iteration 1: Trunk query
┌────────────────────────────────────────────────────┐
│ Key 20 traces to leaf 22 │
│ Key 41 traces to leaf 38 │
└────────────────────────────────────────────────────┘
│
▼
Iteration 2: Branch queries for leaves 22 and 38
┌────────────────────────────────────────────────────┐
│ Key 20: found in leaf 22's subtree → RESOLVED │
│ Key 41: traces to deeper leaf 40 → CONTINUE │
└────────────────────────────────────────────────────┘
│
▼
Iteration 3: Branch query for leaf 40
┌────────────────────────────────────────────────────┐
│ Key 41: proven absent in leaf 40's subtree → DONE │
└────────────────────────────────────────────────────┘
Branch queries run in parallel using FuturesUnordered with configurable concurrency
(max_concurrent_requests, default: 10). The iteration loop continues until all keys
are resolved or max_iterations (default: 50) is reached. In practice, most keys
resolve within 2-3 iterations because each branch query expands several levels of the
tree.
Phase 2: Incremental Catch-Up
After the tree scan produces a snapshot at some checkpoint height, the wallet needs to
catch up to the chain tip. The incremental phase also runs on its own for frequent
re-syncs (when the elapsed time since the last sync is within full_rescan_after_time_s,
default 7 days).
Platform stores per-block address balance changes in a "recent" tree. Every 64 blocks (or 2048 address entries), these per-block entries are compacted into aggregated range entries covering the merged block span. The compacted entries have an expiration TTL and are eventually cleaned up. This means:
- The recent tree never has more than 64 block entries at any time.
- The compacted tree contains historical aggregated ranges for blocks that were compacted away from the recent tree.
checkpoint_height chain_tip
│ │
▼ ▼
──────┬────────────────────────────┬───────────────┤
│ Compacted ranges │ Recent blocks │
│ (aggregated, ≤25/page) │ (per-block, │
│ Only exist after │ always ≤64) │
│ compaction events │ │
└────────────────────────────┴───────────────┘
Step 1: Query Recent First
The wallet always queries recent changes first:
#![allow(unused)] fn main() { GetRecentAddressBalanceChanges { start_height, prove: true } }
Since the recent tree can never exceed 64 entries and the server limit is 100 per request, one query always covers the entire uncompacted range. No pagination is needed for recent changes.
The response contains Vec<BlockAddressBalanceChanges>, where each entry has a
block_height and a map of PlatformAddress → CreditOperation (either SetCredits
or AddToCredits).
Important: The results are held but not applied yet. They may need to be applied after compacted data if compaction occurred.
Step 2: Detect Compaction via Proof
The wallet tracks last_known_recent_block — the highest block height from the
entries returned by the previous recent query. This is a block that was actually
present in the recent tree.
When last_known_recent_block > 0, the recent query uses exclusive start
(RangeAfter) with that height. This causes the height to appear as a boundary
node in the proof. The wallet then uses key_exists_as_boundary to check whether
the block is still in the recent tree:
#![allow(unused)] fn main() { let cursor_exists = Drive::verify_key_exists_as_boundary( proof, &[SavedBlockTransactions, ADDRESS_BALANCES_KEY], &last_known_recent_block.to_be_bytes(), platform_version, )?; }
cursor_exists | Meaning | Action |
|---|---|---|
true | Block still in recent tree — no compaction happened | Apply held recent results directly (Step 4) |
false | Block was compacted away | Query compacted first (Step 3), then apply recent |
When last_known_recent_block == 0: This occurs on the first sync after a tree
scan, or when the recent tree had no entries (no platform address activity on chain).
In this case, the query falls back to inclusive start (RangeFrom) and the compacted
phase always runs. This is the safe default — the compacted query returns empty quickly
if there is no compacted data.
Step 3: Query Compacted (only if needed)
Only runs when compaction is detected or on the first catch-up after a tree scan:
#![allow(unused)] fn main() { GetRecentCompactedAddressBalanceChanges { start_block_height, prove: true } }
Returns Vec<CompactedBlockAddressBalanceChanges>, where each entry has:
start_block_heightandend_block_height(the compacted range)changes: BTreeMap<PlatformAddress, BlockAwareCreditOperation>SetCredits(u64)— absolute balanceAddToCreditsOperations(Vec<(height, credits)>)— deltas by height
The server returns up to 25 ranges per request. If the response contains exactly 25
entries, the client paginates by setting start_block_height = last_end_height + 1
and querying again.
Compacted changes are applied to the result immediately, updating balances and advancing the pagination cursor.
Step 4: Apply Held Recent Results
After compacted data (if any) is applied, the held recent results from Step 1 are processed:
- For each
BlockAddressBalanceChangesentry, for each address change, if the address matches one of the wallet's target addresses, update the balance and callprovider.on_address_found(). - Advance
current_heighttoblock_height + 1for each processed entry.
Finalization
#![allow(unused)] fn main() { result.new_sync_height = max(current_height, observed_tip_height); result.new_sync_timestamp = latest_metadata.time_ms / 1000; }
The caller persists new_sync_height and new_sync_timestamp for the next sync call.
On the next incremental sync, new_sync_height becomes start_height for the recent
query, and new_sync_timestamp determines whether a full tree rescan is needed.
The TrunkBranchSyncOps Trait
The shared algorithm is parameterized by the TrunkBranchSyncOps trait, defined in
packages/rs-sdk/src/platform/trunk_branch_sync/mod.rs. Each sync module implements
this trait to plug in its specific query construction, result processing, and
depth limits.
#![allow(unused)] fn main() { pub trait TrunkBranchSyncOps { /// Module-specific mutable state carried through the scan. type Context<'a>: Send where Self: 'a; /// Immutable config for parallel branch queries (cloned into each task). type BranchQueryConfig: Clone + Send + Sync + 'static; // Trunk async fn execute_trunk_query(sdk, settings, context) -> Result<(GroveTrunkQueryResult, u64, u64), Error>; fn process_trunk_result(trunk_result, context, tracker) -> Result<(), Error>; // Branch fn branch_query_config(context) -> Self::BranchQueryConfig; async fn execute_single_branch_query(sdk, config, key, depth, ...) -> Result<GroveBranchQueryResult, Error>; fn process_branch_result(branch_result, leaf_key, context, tracker) -> Result<(), Error>; // Limits and hooks fn depth_limits(platform_version) -> (u8, u8); fn after_branch_iteration(trunk_result, context, tracker) { } fn on_branch_query(context); fn on_branch_failure(context); fn on_elements_seen(context, count); fn on_iteration(context, iteration); fn set_checkpoint_height(context, height); } }
The two associated types deserve attention:
-
Context<'a>is a GAT (generic associated type) that carries mutable state through the algorithm. For nullifiers, this holds the input keys and result sets. For addresses, it holds the address provider, key-to-index mapping, and result. -
BranchQueryConfigholds immutable parameters needed to construct branch queries that must be sent to async tasks. For nullifiers, this is(pool_type, pool_identifier). For addresses, it is()since no extra parameters are needed.
The after_branch_iteration hook allows the address sync module to implement gap-limit
behavior: after each branch iteration, it checks if the provider has extended its
pending address list and adds newly pending keys to the tracker.
KeyLeafTracker
The KeyLeafTracker (in trunk_branch_sync/tracker.rs) maintains the mapping between
target keys and the leaf subtrees they reside in. It supports:
- Adding keys: When a key traces to a leaf during trunk processing
- Updating keys: When a branch query reveals the key is in a deeper subtree
- Removing keys: When a key is found or proven absent
- Reference counting: Multiple target keys can map to the same leaf; the leaf stays active until all its keys are resolved
#![allow(unused)] fn main() { let mut tracker = KeyLeafTracker::new(); // After trunk query: key traces to leaf subtree tracker.add_key(target_key, leaf_boundary_key, leaf_info); // After branch query: key found in subtree tracker.key_found(&target_key); // After branch query: key in even deeper subtree tracker.update_leaf(&target_key, deeper_leaf_key, deeper_info); // Check what still needs querying let active = tracker.active_leaves(); // leaves with unresolved keys let remaining = tracker.remaining_count(); }
Privacy-Adjusted Leaves (Detail)
The get_privacy_adjusted_leaves function (in trunk_branch_sync/mod.rs) implements
the privacy adjustment described in Phase 1, Step 3. The full logic for each active
leaf:
- Calculate the query depth from the leaf's element count using
calculate_max_tree_depth_from_count(count), clamped to platform-version bounds[min_query_depth, max_query_depth]. - If
count >= min_privacy_count: query this leaf directly at the calculated depth. - If
count < min_privacy_count: calltrunk_result.get_ancestor(&leaf_key, min_privacy_count)to find a higher node. Reduce depth bylevels_up(the number of tree levels climbed) so the total subtree size returned stays reasonable. - If no suitable ancestor exists (rare -- means the entire tree is small): query the leaf anyway, accepting reduced privacy.
- Deduplicate: if two target keys expand to the same ancestor, only one branch query
is emitted (tracked via a
BTreeSet<LeafBoundaryKey>).
Concrete Implementations
Address Balance Sync
The address sync module (platform/address_sync/) implements TrunkBranchSyncOps
as AddressOps<P> where P: AddressProvider.
The AddressProvider trait is implemented by wallets to supply:
- The list of pending addresses to check
- Callbacks when addresses are found or proven absent
- Gap-limit extension (generating new addresses when prior ones are found)
- Current balances for incremental-only mode
#![allow(unused)] fn main() { // First sync -- full tree scan + incremental catch-up let result = sdk.sync_address_balances(&mut wallet, None, None).await?; // Store for next call let height = result.new_sync_height; let timestamp = result.new_sync_timestamp; // Subsequent sync -- incremental only if within 7-day threshold let result = sdk.sync_address_balances(&mut wallet, None, Some(timestamp)).await?; }
Address balance sync uses ItemWithSumItem GroveDB elements where the item value
contains the nonce (4 bytes big-endian) and the sum value contains the credit balance.
Nullifier Sync
The nullifier sync module (platform/nullifier_sync/) implements TrunkBranchSyncOps
as NullifierOps.
Nullifier sync differs from address sync in several ways:
- Target keys are fixed 32-byte arrays (
[u8; 32]) - Branch queries carry extra config:
(pool_type, pool_identifier)to identify the shielded pool - No gap-limit behavior (the
after_branch_iterationhook is not overridden) - Branch query failures are tracked in metrics
#![allow(unused)] fn main() { let nullifiers: Vec<[u8; 32]> = vec![/* ... */]; // First sync -- full tree scan + incremental catch-up let result = sdk.sync_nullifiers(&nullifiers, None, None, None).await?; // Store for next call let height = result.new_sync_height; let timestamp = result.new_sync_timestamp; // Subsequent sync -- incremental only if within 7-day threshold let result = sdk.sync_nullifiers(&nullifiers, None, Some(height), Some(timestamp)).await?; }
Found nullifiers indicate spent notes; absent nullifiers indicate unspent notes.
Sync Mode Decision
Both sync modules use the same logic to decide between full scan and incremental-only:
last_sync_timestamp | Elapsed time | Mode |
|---|---|---|
None | -- | Full tree scan + catch-up |
Some(ts) | < full_rescan_after_time_s | Incremental only |
Some(ts) | >= full_rescan_after_time_s | Full tree scan + catch-up |
The default full_rescan_after_time_s is 604800 (7 days). Setting it to 0 forces a
full tree scan on every call.
Configuration
Both modules expose configuration structs with sensible defaults:
| Parameter | Default | Description |
|---|---|---|
min_privacy_count | 32 | Minimum elements in a queried subtree |
max_concurrent_requests | 10 | Parallel branch queries |
max_iterations | 50 | Safety limit for branch iteration depth |
full_rescan_after_time_s | 604800 | Seconds before forcing a full rescan |
Module Structure
packages/rs-sdk/src/platform/
├── trunk_branch_sync/
│ ├── mod.rs # TrunkBranchSyncOps trait, run_full_tree_scan(),
│ │ # get_privacy_adjusted_leaves(), parallel execution
│ └── tracker.rs # KeyLeafTracker with reference counting
├── address_sync/
│ ├── mod.rs # AddressOps<P> impl, sync_address_balances(),
│ │ # incremental_catch_up()
│ ├── provider.rs # AddressProvider trait
│ └── types.rs # AddressSyncConfig, AddressSyncResult, AddressFunds
└── nullifier_sync/
├── mod.rs # NullifierOps impl, sync_nullifiers(),
│ # incremental_catch_up()
├── provider.rs # NullifierProvider trait
└── types.rs # NullifierSyncConfig, NullifierSyncResult
Rules
Do:
- Use
sdk.sync_address_balances()orsdk.sync_nullifiers()as the entry points. - Persist
new_sync_heightandnew_sync_timestampfrom the result and pass them back on the next sync call. This enables incremental-only mode. - Implement
AddressProviderto integrate with your wallet's key derivation and storage. - Set
min_privacy_counthigh enough that individual key lookups cannot be distinguished. The default of 32 is a reasonable minimum.
Don't:
- Query individual keys directly via the trunk/branch RPCs -- use the sync functions which handle privacy adjustment, iteration, and proof verification.
- Set
max_iterationstoo low -- complex trees may need many rounds. The default of 50 handles trees with millions of entries. - Ignore the
full_rescan_after_time_sthreshold -- without periodic full rescans, the incremental phase could miss changes that occurred before the last known height. - Skip the incremental catch-up phase -- the tree scan snapshot may be slightly stale (the trunk is captured at a specific block height), and the catch-up brings it current.
Appendix: Query Count Analysis
Let N = total addresses in the tree, k = wallet addresses (e.g. 20), d_trunk =
trunk depth (levels expanded), d_branch = levels per branch query, and P = privacy
minimum (default 32).
The Merk tree is a balanced BST with depth D = ⌈log₂(N)⌉.
Trunk query: Always exactly 1 query. Returns the top d_trunk levels,
where d_trunk is between min_depth (6) and max_depth (9) depending on tree
size. Exposes ~2^d_trunk nodes. Cost is independent of k or N.
Key classification: After the trunk, each of the k target keys is classified
by BST traversal (local computation, no queries). Keys found directly in trunk
elements are resolved immediately.
Branch queries per iteration: Unresolved keys trace to trunk leaves. With
privacy adjustment, keys mapping to subtrees smaller than P are promoted to
ancestor subtrees. Multiple keys may share an ancestor, reducing the query count.
Expected unique leaves after deduplication:
L ≈ min(k, 2^d_trunk) · (1 - e^(-k / 2^d_trunk))
Total branch queries: Each target key that traces to a leaf requires 2 branch
queries per iteration — one for the left child subtree and one for the right child —
to determine which side the key falls on. With k = 20 wallet keys, the first
iteration produces 2k = 40 branch queries. If the tree is deeper than
d_trunk + d_branch, some keys trace to deeper leaves and require a second iteration
with fewer keys (typically ~50% resolve per round).
Iterations needed: I = ⌈(D - d_trunk) / d_branch⌉ where D = ⌈log₂(N)⌉.
Both d_trunk and d_branch range from 6 to 9 (configured per platform version).
For trees up to depth ~18 (N ≤ ~260K), one iteration suffices with max depth 9.
Deeper trees need additional iterations.
Incremental catch-up: 1 recent query + 0–1 compacted queries.
Total:
Q_total = 1 (trunk) + 2k × I_eff + Q_incremental
where I_eff accounts for decreasing keys per iteration (not all keys need every
round). In practice, I_eff ≈ I × 0.7 because ~50% of keys resolve per iteration
and later rounds have progressively fewer queries.
Binding Patterns
Dash Platform's core logic is written in Rust. But many developers build applications
in JavaScript -- browser-based wallets, Node.js services, React Native apps. The
wasm-dpp package bridges these worlds by compiling Rust types to WebAssembly and
exposing them as JavaScript classes.
This chapter covers the patterns used to create these bindings: the wrapper struct
pattern, naming conventions, buffer handling at the boundary, getter/setter patterns,
and the Inner trait.
The Problem
Rust and JavaScript have fundamentally different type systems. Rust has ownership,
lifetimes, and zero-cost abstractions. JavaScript has garbage collection, prototype
chains, and dynamic types. wasm-bindgen bridges the gap, but it requires careful
manual work to make the resulting JavaScript API feel natural.
The core challenge: you have a Rust type like Identity with methods like id(),
balance(), and public_keys(). You need to expose it to JavaScript as a class
with methods like getId(), getBalance(), and getPublicKeys() -- following
JavaScript naming conventions while maintaining Rust's safety guarantees.
The Wrapper Struct Pattern
Every Rust type exposed to JavaScript gets a wrapper struct. Here is IdentityWasm
from packages/wasm-dpp/src/identity/identity.rs:
#![allow(unused)] fn main() { #[wasm_bindgen(js_name=Identity)] #[derive(Clone)] pub struct IdentityWasm { inner: Identity, metadata: Option<Metadata>, } }
The pattern has three parts:
#[wasm_bindgen(js_name=Identity)]-- tellswasm_bindgento expose this struct asIdentityin JavaScript, notIdentityWasm.inner: Identity-- the real Rust type, hidden from JavaScript.- Additional fields -- any extra state needed at the WASM boundary (like
metadatahere, which is managed separately in JS).
Why a Wrapper?
You cannot put #[wasm_bindgen] directly on Identity for several reasons:
Identityis defined inrs-dpp, a different crate. You cannot add attributes to types in other crates.Identitymay contain types thatwasm_bindgencannot handle (nested enums, complex generics, trait objects).- The JavaScript API should have camelCase methods (
getId), not Rust-style snake_case (id). - Some conversions (like turning
Identifierinto aBuffer) only make sense at the WASM boundary.
From/Into Conversions
Every wrapper implements bidirectional conversion:
#![allow(unused)] fn main() { impl From<IdentityWasm> for Identity { fn from(identity: IdentityWasm) -> Self { identity.inner } } impl From<Identity> for IdentityWasm { fn from(identity: Identity) -> Self { Self { inner: identity, metadata: None, } } } }
This lets internal Rust code work with the real Identity type while the WASM
boundary works with IdentityWasm. The conversion is zero-cost -- it just moves
the inner value.
JavaScript Method Naming
Methods use #[wasm_bindgen(js_name=...)] to follow JavaScript conventions:
#![allow(unused)] fn main() { #[wasm_bindgen(js_class=Identity)] impl IdentityWasm { #[wasm_bindgen(js_name=getId)] pub fn get_id(&self) -> IdentifierWrapper { self.inner.id().into() } #[wasm_bindgen(js_name=setId)] pub fn set_id(&mut self, id: IdentifierWrapper) { self.inner.set_id(id.into()); } #[wasm_bindgen(js_name=getBalance)] pub fn get_balance(&self) -> u64 { self.inner.balance() } #[wasm_bindgen(js_name=setBalance)] pub fn set_balance(&mut self, balance: u64) { self.inner.set_balance(balance); } } }
Note the #[wasm_bindgen(js_class=Identity)] on the impl block -- this associates
the methods with the Identity JavaScript class (the js_name from the struct).
In JavaScript, these become:
const identity = new Identity(platformVersion);
const id = identity.getId();
identity.setBalance(1000n);
Getter Properties
For simple values, you can use JavaScript getter syntax:
#![allow(unused)] fn main() { #[wasm_bindgen(getter)] pub fn balance(&self) -> u64 { self.inner.balance() } }
In JavaScript, this becomes a property access:
const bal = identity.balance; // no parentheses
Platform uses both patterns -- getBalance() method and balance getter -- for
the same value. This provides flexibility: the getter is concise for reading, the
method is consistent for tools that expect a getter/setter pair.
Constructors
The #[wasm_bindgen(constructor)] attribute creates a JavaScript constructor:
#![allow(unused)] fn main() { #[wasm_bindgen(constructor)] pub fn new(platform_version: u32) -> Result<IdentityWasm, JsValue> { let platform_version = &PlatformVersion::get(platform_version) .map_err(|e| JsValue::from(e.to_string()))?; Identity::default_versioned(platform_version) .map(Into::into) .map_err(from_dpp_err) } }
Notice the error handling: Rust Result becomes a JavaScript throw. The
from_dpp_err function converts Rust ProtocolError into JavaScript error objects.
Buffer Handling at the Boundary
Binary data crosses the WASM boundary as Buffer (a custom type that wraps
Uint8Array):
#![allow(unused)] fn main() { #[wasm_bindgen(js_name=toBuffer)] pub fn to_buffer(&self) -> Result<Buffer, JsValue> { let bytes = PlatformSerializable::serialize_to_bytes( &self.inner.clone() ).with_js_error()?; Ok(Buffer::from_bytes(&bytes)) } #[wasm_bindgen(js_name=fromBuffer)] pub fn from_buffer(buffer: Vec<u8>) -> Result<IdentityWasm, JsValue> { let identity: Identity = PlatformDeserializable::deserialize_from_bytes(buffer.as_slice()) .with_js_error()?; Ok(identity.into()) } }
The toBuffer/fromBuffer pair is the standard serialization interface. Every WASM
type that needs to be stored or transmitted implements this pair.
Handling Complex Types: Arrays and Objects
JavaScript arrays and objects require special handling. For collections of public keys:
#![allow(unused)] fn main() { #[wasm_bindgen(js_name=getPublicKeys)] pub fn get_public_keys(&self) -> Vec<JsValue> { self.inner .public_keys() .values() .cloned() .map(IdentityPublicKeyWasm::from) // Rust -> Wrapper .map(JsValue::from) // Wrapper -> JsValue .collect() } #[wasm_bindgen(js_name=setPublicKeys)] pub fn set_public_keys(&mut self, public_keys: js_sys::Array) -> Result<usize, JsValue> { if public_keys.length() == 0 { return Err("Must use array of PublicKeys".into()); } let public_keys = public_keys .iter() .map(|key| { key.to_wasm::<IdentityPublicKeyWasm>("IdentityPublicKey") .map(|key| { let key = IdentityPublicKey::from(key.to_owned()); (key.id(), key) }) }) .collect::<Result<_, _>>()?; self.inner.set_public_keys(public_keys); Ok(self.inner.public_keys().len()) } }
The to_wasm::<T>("TypeName") helper extracts a Rust wrapper from a JsValue,
validating that it is the correct type.
JSON Serialization
Most WASM types provide toJSON and toObject methods:
#![allow(unused)] fn main() { #[wasm_bindgen(js_name=toJSON)] pub fn to_json(&self) -> Result<JsValue, JsValue> { let mut value = self.inner.to_object().with_js_error()?; // Convert identifiers to Base58 strings for readability value.replace_at_paths( dpp::identity::IDENTIFIER_FIELDS_RAW_OBJECT, ReplacementType::TextBase58, ).map_err(|e| e.to_string())?; // Convert binary key data to Base64 let public_keys = value .get_array_mut_ref(dpp::identity::property_names::PUBLIC_KEYS) .map_err(|e| e.to_string())?; for key in public_keys.iter_mut() { key.replace_at_paths( dpp::identity::identity_public_key::BINARY_DATA_FIELDS, ReplacementType::TextBase64, ).map_err(|e| e.to_string())?; } let json = value.try_into_validating_json() .map_err(|e| e.to_string())? .to_string(); js_sys::JSON::parse(&json) } }
The toJSON method applies human-readable encoding (Base58 for identifiers, Base64
for binary data) before converting to a JavaScript object. This is the format used
in APIs and debugging tools.
The Inner Trait
To standardize wrapper access, Platform defines an Inner trait:
#![allow(unused)] fn main() { impl Inner for IdentityWasm { type InnerItem = Identity; fn into_inner(self) -> Self::InnerItem { self.inner } fn inner(&self) -> &Self::InnerItem { &self.inner } fn inner_mut(&mut self) -> &mut Self::InnerItem { &mut self.inner } } }
This trait provides a consistent way for other Rust code in the WASM layer to access the unwrapped type without knowing the wrapper's internal structure.
Rules
Do:
- Follow the wrapper struct pattern:
TypeWasm { inner: Type }. - Use
js_nameon the struct andjs_classon theimplblock. - Implement
From<Type> for TypeWasmandFrom<TypeWasm> for Type. - Use
Buffer::from_bytesfor binary data crossing the boundary. - Implement
toBuffer/fromBufferfor any type that needs serialization. - Use
from_dpp_errorwith_js_error()for error conversion. - Implement the
Innertrait for consistent wrapper access.
Don't:
- Expose Rust types directly to WASM -- always wrap them.
- Use Rust naming conventions (
get_id) in the JavaScript API -- usejs_name=getId. - Return
Result<T, ProtocolError>from WASM methods -- convert toResult<T, JsValue>. - Forget to handle empty arrays and invalid inputs with clear error messages.
- Expose internal fields that do not make sense in JavaScript (like
ArcorMutex).
Error Macros
Dash Platform defines hundreds of consensus errors. Each one needs a WASM binding so JavaScript code can inspect error codes, read messages, and serialize errors for transport. Writing a wrapper struct for every single error by hand would be tedious, error-prone, and a maintenance burden.
This chapter covers the generic_consensus_error! macro that generates WASM bindings
automatically, the paste! crate that makes it work, the manual binding pattern for
errors that need custom methods, and how the two approaches coexist.
The Problem
Consider Platform's error hierarchy. At the top level:
ConsensusError
BasicError (dozens of variants)
StateError (dozens of variants)
SignatureError (handful of variants)
FeeError (one variant)
Each variant wraps a specific error struct -- TransitionNoInputsError,
DocumentNotFoundError, MasternodeNotFoundError, and so on. There are well over a
hundred of these.
Every one needs a JavaScript class with:
- A
getCode()method returning the numeric error code. - A
messagegetter returning the human-readable error string. - A
serialize()method for wire encoding. - A
From<&RustType>implementation for conversion.
Writing all of this for each error would mean hundreds of nearly identical files.
The generic_consensus_error! Macro
The macro lives in packages/wasm-dpp/src/errors/generic_consensus_error.rs:
#![allow(unused)] fn main() { #[macro_export] macro_rules! generic_consensus_error { ($error_type:ident, $error_instance:expr) => {{ use { dpp::{ consensus::{codes::ErrorWithCode, ConsensusError}, serialization::PlatformSerializableWithPlatformVersion, version::PlatformVersion, }, paste::paste, wasm_bindgen::prelude::wasm_bindgen, $crate::buffer::Buffer, }; paste! { #[derive(Debug)] #[wasm_bindgen(js_name=$error_type)] pub struct [<$error_type Wasm>] { inner: $error_type } impl From<&$error_type> for [<$error_type Wasm>] { fn from(e: &$error_type) -> Self { Self { inner: e.clone() } } } #[wasm_bindgen(js_class=$error_type)] impl [<$error_type Wasm>] { #[wasm_bindgen(js_name=getCode)] pub fn get_code(&self) -> u32 { ConsensusError::from(self.inner.clone()).code() } #[wasm_bindgen(getter)] pub fn message(&self) -> String { self.inner.to_string() } pub fn serialize(&self) -> Result<Buffer, JsError> { let bytes = ConsensusError::from(self.inner.clone()) .serialize_to_bytes_with_platform_version( PlatformVersion::first(), ) .map_err(JsError::from)?; Ok(Buffer::from_bytes(bytes.as_slice())) } } [<$error_type Wasm>]::from($error_instance) } }}; } }
What It Generates
For a call like generic_consensus_error!(MasternodeNotFoundError, e), the macro
generates:
- A wrapper struct:
MasternodeNotFoundErrorWasmwithinner: MasternodeNotFoundError - A From impl:
From<&MasternodeNotFoundError> for MasternodeNotFoundErrorWasm - Three methods:
get_code()-- returns the numeric error codemessage-- returns theDisplaystringserialize()-- encodes the error to bytes
- An instantiation: Creates a
MasternodeNotFoundErrorWasmfrom the error reference
The paste! Crate
The magic of [<$error_type Wasm>] comes from the paste crate. Standard Rust
macros cannot concatenate identifiers -- you cannot write $error_type ## Wasm to
create a new identifier. The paste! macro provides this:
#![allow(unused)] fn main() { paste! { pub struct [<$error_type Wasm>] { ... } // ^^^^^^^^^^^^^^^^^ // pastes to: MasternodeNotFoundErrorWasm } }
Inside paste! { ... }, [<token1 token2>] concatenates tokens into a single
identifier. This is what allows the macro to generate both the JavaScript name
(MasternodeNotFoundError via js_name) and the Rust struct name
(MasternodeNotFoundErrorWasm via paste).
How It Is Used
The macro is used inline within the from_consensus_error_ref function and its
helpers in packages/wasm-dpp/src/errors/consensus/consensus_error.rs. Here is
a representative excerpt:
#![allow(unused)] fn main() { pub fn from_state_error(state_error: &StateError) -> JsValue { match state_error { // Manual wrappers (have custom methods) StateError::DocumentAlreadyPresentError(e) => { DocumentAlreadyPresentErrorWasm::from(e).into() } StateError::DocumentNotFoundError(e) => { DocumentNotFoundErrorWasm::from(e).into() } // Macro-generated wrappers (standard interface only) StateError::MasternodeNotFoundError(e) => { generic_consensus_error!(MasternodeNotFoundError, e).into() } StateError::DocumentContestCurrentlyLockedError(e) => { generic_consensus_error!( DocumentContestCurrentlyLockedError, e ).into() } StateError::TokenIsPausedError(e) => { generic_consensus_error!(TokenIsPausedError, e).into() } // ... dozens more } } }
The pattern is clear: use the macro for errors that need only getCode(), message,
and serialize(). Use manual wrappers for errors that need custom accessors.
Manual Wrappers: When You Need More
Some errors expose domain-specific data that JavaScript code needs to access. For
these, you write a full wrapper by hand. Here is DataContractMaxDepthExceedError
from packages/wasm-dpp/src/errors/consensus/basic/data_contract/:
#![allow(unused)] fn main() { #[wasm_bindgen(js_name=DataContractMaxDepthExceedError)] pub struct DataContractMaxDepthExceedErrorWasm { inner: DataContractMaxDepthExceedError, } impl From<&DataContractMaxDepthExceedError> for DataContractMaxDepthExceedErrorWasm { fn from(e: &DataContractMaxDepthExceedError) -> Self { Self { inner: e.clone() } } } #[wasm_bindgen(js_class=DataContractMaxDepthError)] impl DataContractMaxDepthExceedErrorWasm { #[wasm_bindgen(js_name=getMaxDepth)] pub fn get_max_depth(&self) -> usize { self.inner.max_depth() } #[wasm_bindgen(js_name=getCode)] pub fn get_code(&self) -> u32 { ConsensusError::from(self.inner.clone()).code() } #[wasm_bindgen(getter)] pub fn message(&self) -> String { self.inner.to_string() } } }
The custom get_max_depth() method lets JavaScript inspect the specific limit that
was exceeded. The macro cannot generate these domain-specific accessors -- it only
knows about the three standard methods.
The from_dpp_err Pattern
At the top level, Rust's ProtocolError is converted to a JsValue through
from_dpp_err in packages/wasm-dpp/src/errors/from.rs:
#![allow(unused)] fn main() { pub fn from_dpp_err(pe: ProtocolError) -> JsValue { match pe { ProtocolError::ConsensusError(consensus_error) => { from_consensus_error(*consensus_error) } ProtocolError::DataContractError(e) => { from_data_contract_to_js_error(e) } ProtocolError::Document(e) => { from_document_to_js_error(*e) } ProtocolError::DataContractNotPresentError(err) => { DataContractNotPresentNotConsensusErrorWasm::new( err.data_contract_id() ).into() } ProtocolError::ValueError(value_error) => { PlatformValueErrorWasm::from(value_error).into() } _ => JsValue::from_str( &format!("Error conversion not implemented: {pe:#}") ), } } }
This is the entry point for error conversion. It dispatches to the appropriate conversion function based on the error variant. The fallback case converts unhandled errors to a string -- not ideal, but it ensures no errors are silently swallowed.
The Consensus Error Dispatch
The from_consensus_error_ref function dispatches across the entire consensus
error hierarchy:
#![allow(unused)] fn main() { pub fn from_consensus_error_ref(e: &DPPConsensusError) -> JsValue { match e { DPPConsensusError::FeeError(e) => match e { FeeError::BalanceIsNotEnoughError(e) => BalanceIsNotEnoughErrorWasm::from(e).into(), }, DPPConsensusError::SignatureError(e) => from_signature_error(e), DPPConsensusError::StateError(state_error) => from_state_error(state_error), DPPConsensusError::BasicError(basic_error) => from_basic_error(basic_error), DPPConsensusError::DefaultError => JsError::new("DefaultError").into(), } } }
Each sub-function (from_state_error, from_basic_error, from_signature_error)
handles its category, using either manual wrappers or the macro as appropriate.
Adding a New WASM Error Binding
When a new consensus error is added to rs-dpp, you need to add its WASM binding.
Here is the checklist:
If the error only needs getCode(), message, and serialize():
- Import the error type in
consensus_error.rs. - Add a match arm using the macro:
#![allow(unused)] fn main() { StateError::YourNewError(e) => { generic_consensus_error!(YourNewError, e).into() } }
That is it. The macro handles everything else.
If the error needs custom accessors:
- Create a new file in the appropriate subdirectory under
packages/wasm-dpp/src/errors/consensus/. - Define the wrapper struct,
Fromimpl, and methods following the manual pattern. - Add the wrapper to the
mod.rsfile. - Import it in
consensus_error.rs. - Add a match arm using the manual wrapper:
#![allow(unused)] fn main() { StateError::YourNewError(e) => { YourNewErrorWasm::from(e).into() } }
Rules
Do:
- Use
generic_consensus_error!for errors that need only the standard three methods. - Use manual wrappers when JavaScript needs to access error-specific fields.
- Follow the existing file organization: one file per manual error, grouped by category.
- Always provide
getCode(),message, andserialize()-- these are the standard interface. - Test that new errors convert correctly in both directions.
Don't:
- Write manual wrappers when the macro would suffice -- it just creates maintenance debt.
- Forget to add the match arm in
consensus_error.rs-- unhandled errors will fall through to theDefaultErrorcase or anunreachable_patternsguard. - Use
js_namevalues that differ from the Rust error type name -- JavaScript developers should see the same name they find in documentation. - Skip the
serialize()method -- it is needed for error transport across process boundaries. - Modify
generic_consensus_error!without understanding that every call site will be affected -- it generates code in every match arm that uses it.
API Reference
Auto-generated API documentation for the Dash Platform developer ecosystem.
Each section is built from source and updated by CI when relevant source files change on v*-dev branches.
Rust Crate Docs
Full rustdoc for the Dash Platform workspace — every public type, trait, function, and module across all crates.
Key crates:
| Crate | Description |
|---|---|
dash-sdk | High-level client SDK with builder pattern and fetch traits |
dpp | Dash Platform Protocol — data contracts, documents, identities, state transitions |
drive | Decentralized storage engine built on GroveDB |
drive-abci | ABCI application connecting Tenderdash to Drive |
dapi-grpc | Rust types generated from the gRPC protocol definitions |
rs-dapi-client | Low-level DAPI client with retries and load balancing |
platform-value | Cross-language value representation |
platform-version | Protocol versioning and feature version dispatch |
gRPC API
Protocol Buffer service definitions for the three gRPC endpoints:
- Platform — identity, data contract, document, and token operations
- Core — block, transaction, and masternode queries
- Drive — internal node-to-node replication
Documents every RPC method, request/response message, and field type.
JavaScript / TypeScript API
TypeDoc for the JavaScript and TypeScript client libraries:
- dash (js-dash-sdk) — main SDK for Node.js and browser applications
- wasm-dpp — WebAssembly bindings for Dash Platform Protocol
- wasm-sdk — Browser-facing WASM SDK with TypeScript type definitions