Tenderdash Documentation

Tenderdash is the Byzantine Fault Tolerant consensus engine used by Dash Platform.

The consensus specification covers the primary flow of a height, transitions and decision rules, alternative flows, commit handling, recovery, and known discrepancies. Use the sidebar or search to explore the complete documentation.

Tenderdash

Tenderdash is the Byzantine Fault Tolerant consensus engine used by Dash Platform. It replicates application state across validators and connects to the application through the Application Blockchain Interface (ABCI).

The protocol specification describes the implementation on this branch and links to the functions that define each rule. Historical research and design documents remain available alongside it.

Overview

Quick Start

Get Tenderdash up-and-running quickly with the quick-start guide!

Install

Detailed installation instructions.

What is Tenderdash

Dive into what Tenderdash is and why!

Tenderdash Architectural Overview

A Tenderdash node combines peer networking, consensus, transaction admission, block storage and an ABCI connection to the application. The node package constructs these services and manages their lifecycle.

Communication

  • Peer communication: long-lived P2P connections exchange transactions, blocks, votes, evidence and state-sync messages.
  • Application communication: ABCI requests validate transactions, prepare and process proposals, finalize blocks and query application state. The application can run in the same Go process or through an external ABCI connection.
  • Client communication: the RPC service accepts transactions, answers queries and exposes events and node status. RPC access is separate from P2P networking.
  • Dash Core communication: quorum and ChainLock information connects Platform consensus to Dash Core. Configure the appropriate Core RPC endpoint for your network.
flowchart TB
    Clients[RPC clients] --> RPC[RPC service]
    RPC --> Mempool[Mempool]
    RPC --> Store[Block and state stores]
    Peers[Peer nodes] <--> P2P[P2P networking]
    P2P <--> Consensus[Consensus]
    P2P <--> Mempool
    P2P <--> Sync[Block sync and state sync]
    Consensus --> Store
    Consensus <--> ABCI[ABCI application]
    Mempool <--> ABCI
    Sync <--> ABCI
    Core[Dash Core RPC] --> Consensus

Reactors and P2P services

P2P services handle distinct message types: consensus votes and proposals, transaction gossip, block synchronization, evidence, peer exchange and state synchronization. Multiplexed channels share peer connections while separating these message flows. See the P2P specification for the transport and wire protocol.

Block sync brings a node up to date by downloading and verifying blocks. State sync restores an application snapshot and the consensus state needed to resume processing. When caught up, a node participates in live consensus. See block sync and state sync.

Node roles

A validator holds consensus signing keys and votes when it belongs to the active validator set. A full node verifies and stores blocks without voting. A seed node helps other nodes discover peers. The role is configured by mode in config.toml; see configuration.

For the detailed consensus rules, read the consensus specification.

Install Tenderdash

Dashmate is a part of the Dash Platform and provides a comprehensive solution for installing the entire platform. We highly recommend using Dashmate for a seamless and straightforward installation process.

For detailed instructions on how to set up a node using Dashmate, please refer to the official Dash documentation: Set Up a Node.

From Binary

To download pre-built binaries, see the releases page.

From Source

Install official go into the standard location with standard PATH updates:

curl https://webinstall.dev/go | sh
source ~/.config/envman/PATH.env

Install git, cmake, and build-essential (apt) or build-base (apk) and other libs.

For Debian-based (eg. Ubuntu):

sudo apt update
sudo apt install -y git build-essential cmake libgmp-dev

For Alpine Linux:

apk add --no-cache git build-base cmake gmp-dev gmp-static

Get the source code:

git clone https://github.com/dashpay/tenderdash.git
pushd ./tenderdash/
git submodule init
git submodule update

Check out the release tag required by your network before building. Development branches may contain changes that are not compatible with a deployed network.

Build:

to put the binary in $GOPATH/bin:

make install

to put the binary in ./build:

make build

The latest Tenderdash is now installed. You can verify the installation by running:

tenderdash version

Cross-compilation

To cross-compile for ARM platform, you need to install required compilers. On Ubuntu 20.04+:

sudo apt-get install gcc-10-arm-linux-gnueabi g++-10-arm-linux-gnueabi

To start the build process, execute:

GOOS=linux GOARCH=arm  make

Run

To start a one-node blockchain with a simple in-process application:

tenderdash init validator
tenderdash start --proxy-app=kvstore

Reinstall

If you already have Tenderdash installed, and you make updates, simply

make install

To upgrade, fetch and check out the release tag required by your network, then run make install. Coordinate consensus and application upgrades with the network.

Compile with CLevelDB support

Install LevelDB (minimum version is 1.7).

Install LevelDB with snappy (optionally). Below are commands for Ubuntu:

sudo apt-get update
sudo apt install build-essential

sudo apt-get install libsnappy-dev

wget https://github.com/google/leveldb/archive/v1.20.tar.gz && \
  tar -zxvf v1.20.tar.gz && \
  cd leveldb-1.20/ && \
  make && \
  sudo cp -r out-static/lib* out-shared/lib* /usr/local/lib/ && \
  cd include/ && \
  sudo cp -r leveldb /usr/local/include/ && \
  sudo ldconfig && \
  rm -f v1.20.tar.gz

Set a database backend to cleveldb:

# config/config.toml
db_backend = "cleveldb"

To install Tenderdash, run:

CGO_LDFLAGS="-lsnappy" make install TENDERMINT_BUILD_OPTIONS=cleveldb

or run:

CGO_LDFLAGS="-lsnappy" make build TENDERMINT_BUILD_OPTIONS=cleveldb

which puts the binary in ./build.

Quick Start

Overview

This is a quick start guide. If you have a vague idea about how Tenderdash works and want to get started right away, continue. Make sure you've installed the binary. Check out install if you haven't.

Initialization

Running:

tenderdash init validator

will create the required files for a single, local node.

These files are found in $HOME/.tenderdash:

$ ls $HOME/.tenderdash

config  data

$ ls $HOME/.tenderdash/config/

config.toml  genesis.json  node_key.json  priv_validator.json

For a single, local node, no further configuration is required. Configuring a cluster is covered further below.

Local Node

Start Tenderdash with a simple in-process application:

tenderdash start --proxy-app=kvstore

Note: kvstore is a non persistent app, if you would like to run an application with persistence run --proxy-app=persistent_kvstore

and blocks will start to stream in:

I[01-06|01:45:15.592] Executed block                               module=state height=1 validTxs=0 invalidTxs=0
I[01-06|01:45:15.624] Committed state                              module=state height=1 txs=0 appHash=

Check the status with:

curl -s localhost:26657/status

Sending Transactions

With the KVstore app running, we can send transactions:

curl -s 'localhost:26657/broadcast_tx_commit?tx="abcd"'

and check that it worked with:

curl -s 'localhost:26657/abci_query?data="abcd"'

We can send transactions with a key and value too:

curl -s 'localhost:26657/broadcast_tx_commit?tx="name=satoshi"'

and query the key:

curl -s 'localhost:26657/abci_query?data="name"'

where the value is returned in hex.

Cluster of Nodes

Prepare four hosts with Tenderdash installed. Use the same release and genesis on every node, and restrict administrative RPC access.

Next, use the tenderdash testnet command to create four directories of config files (found in ./mytestnet) and copy each directory to the relevant machine in the cloud, so that each machine has $HOME/mytestnet/node[0-3] directory.

Before you can start the network, you'll need peers identifiers (IPs are not enough and can change). We'll refer to them as ID1, ID2, ID3, ID4.

tenderdash show-node-id --home ./mytestnet/node0
tenderdash show-node-id --home ./mytestnet/node1
tenderdash show-node-id --home ./mytestnet/node2
tenderdash show-node-id --home ./mytestnet/node3

Finally, from each machine, run:

tenderdash start --home ./mytestnet/node0 --proxy-app=kvstore --p2p.persistent-peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656"
tenderdash start --home ./mytestnet/node1 --proxy-app=kvstore --p2p.persistent-peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656"
tenderdash start --home ./mytestnet/node2 --proxy-app=kvstore --p2p.persistent-peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656"
tenderdash start --home ./mytestnet/node3 --proxy-app=kvstore --p2p.persistent-peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656"

Note that after the third node is started, blocks will start to stream in because >2/3 of validators (defined in the genesis.json) have come online. Persistent peers can also be specified in the config.toml. See here for more information about configuration options.

Transactions can then be sent as covered in the single, local node example above.

What is Tenderdash

Tenderdash is the consensus engine used by Dash Platform. It replicates an application across a network of validators, ordering transactions into blocks and agreeing on application state. The application implements the Application BlockChain Interface (ABCI); Tenderdash handles peer communication, transaction gossip and Byzantine fault tolerant consensus.

Consensus and validators

Validators take turns proposing blocks at each height. They vote in two stages, prevote and precommit. To commit a block, precommits for it in the same round must reach the configured voting threshold, and quorum threshold-signature recovery and verification must succeed. The voting threshold can be an explicit override or the Dash quorum-type default; changing it does not lower the quorum's signature-recovery threshold. If a round cannot commit a block, validators move to another round with a new proposer.

The usual BFT safety argument assumes less than one-third of voting power is Byzantine and a voting threshold strictly above two-thirds. Tenderdash accepts some lower thresholds, so that safety bound is not guaranteed by configuration. See the threshold definitions, system-model assumptions and threshold recovery. Safety also depends on correct key management, validation and locking rules. Progress requires enough validators to communicate and sufficiently timely message delivery; a network partition can halt block production.

Tenderdash uses BLS validator keys and quorum information from Dash Core. Dash Core ChainLocks and quorum verification connect Platform consensus to Dash's security model. See the protocol specification, validator guide and consensus specification for details.

Application boundary

An ABCI application defines transaction validity, execution and state storage. Tenderdash treats transaction contents as application-defined bytes.

flowchart LR
    Client[Client] -->|RPC transactions and queries| Node[Tenderdash node]
    Node <-->|P2P blocks, votes and transactions| Peers[Other nodes]
    Node <-->|ABCI requests and responses| App[Deterministic application]
    App --> State[Application state]

Important application calls include:

  • CheckTx checks transactions before admission to the mempool.
  • PrepareProposal lets the proposer application select and prepare transactions.
  • ProcessProposal lets validator applications validate a proposed block.
  • ExtendVote and VerifyVoteExtension support application-defined vote extensions.
  • FinalizeBlock delivers a decided block and persists the resulting application state.
  • Info, InitChain and Query support startup, initialization and queries.
  • Snapshot methods support state sync.

The Go application interface and ABCI specification define the contract. Use the interface and specification from the same release as your node.

Determinism

Every correct validator must compute the same results for the same committed input. Application execution must avoid uncontrolled sources of nondeterminism, including local clocks, unseeded randomness, concurrent access to mutable state, unordered map iteration and platform-dependent floating point behavior.

Application state changes must follow the consensus lifecycle. An application must not let unrelated external writes alter the replicated state.

Start building

Install Tenderdash, follow the quick start, and explore the application development guide. The architecture overview explains the node's components.

Overview

This section will focus on how to operate full nodes, validators and light clients.

Node Types

We will cover the various types of node types within Tenderdash.

Full Node

A full node is a node that participates in the network but will not help secure it. Full nodes can be used to store the entire state of a blockchain. For Tenderdash there are two forms of state. First, blockchain state, this represents the blocks of a blockchain. Secondly, there is Application state, this represents the state that transactions modify. The knowledge of how a transaction can modify state is not held by Tenderdash but rather the application on the other side of the ABCI boundary.

Note: If you have not read about the seperation of consensus and application please take a few minutes to read up on it as it will provide a better understanding to many of the terms we use throughout the documentation. You can find more information on the ABCI here.

As a full node operator you are providing services to the network that helps it come to consensus and others catch up to the current block. Even though a full node only helps the network come to consensus it is important to secure your node from adversarial actors. We recommend using a firewall and a proxy if possible. Running a full node can be easy, but it varies from network to network. Verify your applications documentation prior running a node.

Seed Nodes

A seed node provides a node with a list of peers which a node can connect to. When starting a node you must provide at least one type of node to be able to connect to the desired network. By providing a seed node you will be able to populate your address quickly. A seed node will not be kept as a peer but will disconnect from your node after it has provided a list of peers.

Sentry Node

A sentry node is similar to a full node in almost every way. The difference is a sentry node will have one or more private peers. These peers may be validators or other full nodes in the network. A sentry node is meant to provide a layer of security for your validator, similar to how a firewall works with a computer.

Validators

Validators sign consensus votes when their pro-tx-hash belongs to the active Dash quorum. All validators have voting power 100; application-provided validator-set updates determine membership. See the validator guide and the canonical validator-set rules.

Configuration

Tenderdash Core can be configured via a TOML file in $TMHOME/config/config.toml. Some of these parameters can be overridden by command-line flags. For most users, the options in the ##### main base configuration options ##### are intended to be modified while config options further below are intended for advance power users.

Options

The default configuration file create by tenderdash init has all the parameters set with their default values. It will look something like the file below, however, double check by inspecting the config.toml created with your version of tenderdash installed:

# This is a TOML config file.
# For more information, see https://github.com/toml-lang/toml

# NOTE: Any path below can be absolute (e.g. "/var/myawesomeapp/data") or
# relative to the home directory (e.g. "data"). The home directory is
# "$HOME/.tenderdash" by default, but could be changed via $TMHOME env variable
# or --home cmd flag.

#######################################################################
###                   Main Base Config Options                      ###
#######################################################################

# TCP or UNIX socket address of the ABCI application,
# or the name of an ABCI application compiled in with the Tenderdash binary
proxy-app = "tcp://127.0.0.1:26658"

# A custom human readable name for this node
moniker = "sidewinder"

# Mode of Node: full | validator | seed
# * validator node
#   - all reactors
#   - with priv_validator_key.json, priv_validator_state.json
# * full node
#   - all reactors
#   - No priv_validator_key.json, priv_validator_state.json
# * seed node
#   - only P2P, PEX Reactor
#   - No priv_validator_key.json, priv_validator_state.json
mode = "validator"

# Database backend: goleveldb 
# * goleveldb (github.com/syndtr/goleveldb - most popular implementation)
#   - pure go
#   - stable
db-backend = "goleveldb"

# Database directory
db-dir = "data"

# Output level for logging, including package level options
log-level = "info"

# Output format: 'plain' (colored text) or 'json'
log-format = "plain"

##### additional base config options #####

# Path to the JSON file containing the initial validator set and other meta data
genesis-file = "config/genesis.json"

# Path to the JSON file containing the private key to use for node authentication in the p2p protocol
node-key-file = "config/node_key.json"

# Mechanism to connect to the ABCI application: socket | grpc
abci = "socket"

# If true, query the ABCI app on connecting to a new peer
# so the app can decide if we should keep the connection or not
filter-peers = false

# If set to positive duration, deadlock detection is enabled and set to the given time.
# Use 0 to disable.
# Default: 0
deadlock-detection = "0s"

# If true, block store and state store writes return before they have reached
# the disk. This exists to take the cost of fsync out of a block sync benchmark
# and nothing else. Never enable it on a node whose data matters: a power loss
# can leave the stores behind the application, and the node will refuse to start.
# Default: false
unsafe-no-fsync = false


#######################################################
###       Priv Validator Configuration              ###
#######################################################
[priv-validator]

# Path to the JSON file containing the private key to use as a validator in the consensus protocol
key-file = "config/priv_validator_key.json"

# Path to the JSON file containing the last sign state of a validator
state-file = "data/priv_validator_state.json"

# TCP or UNIX socket address for Tenderdash to listen on for
# connections from an external PrivValidator process
# when the listenAddr is prefixed with grpc instead of tcp it will use the gRPC Client
laddr = ""

# Path to the client certificate generated while creating needed files for secure connection.
# If a remote validator address is provided but no certificate, the connection will be insecure
client-certificate-file = ""

# Client key generated while creating certificates for secure connection
client-key-file = ""

# Path to the Root Certificate Authority used to sign both client and server certificates
root-ca-file = ""


#######################################################################
###                 Advanced Configuration Options                  ###
#######################################################################

#######################################################
###       RPC Server Configuration Options          ###
#######################################################
[rpc]

# TCP or UNIX socket address for the RPC server to listen on
laddr = "tcp://127.0.0.1:26657"

# A list of origins a cross-domain request can be executed from
# Default value '[]' disables cors support
# Use '["*"]' to allow any origin
cors-allowed-origins = []

# This list also permits browser WebSocket origins for `tenderdash light <chainID>`.
# An empty list rejects all requests carrying Origin, including same-host requests.
# Clients without Origin remain supported. Add explicit browser origins as needed;
# wildcard matching follows RPC CORS rules, and ["*"] explicitly allows any origin.

# A list of methods the client is allowed to use with cross-domain requests
cors-allowed-methods = ["HEAD", "GET", "POST", ]

# A list of non simple headers the client is allowed to use with cross-domain requests
cors-allowed-headers = ["Origin", "Accept", "Content-Type", "X-Requested-With", "X-Server-Time", ]

# Activate unsafe RPC commands like /dial-seeds and /unsafe-flush-mempool
unsafe = false

# Maximum number of simultaneous connections (including WebSocket).
# If you want to accept a larger number than the default, make sure
# you increase your OS limits.
# 0 - unlimited.
# Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}
# 1024 - 40 - 10 - 50 = 924 = ~900
max-open-connections = 900

# Maximum number of unique clientIDs that can /subscribe
# If you're using /broadcast_tx_commit, set to the estimated maximum number
# of broadcast_tx_commit calls per block.
max-subscription-clients = 100

# Maximum number of unique queries a given client can /subscribe to
# If you're using a Local RPC client and /broadcast_tx_commit, set this
# to the estimated maximum number of broadcast_tx_commit calls per block.
max-subscriptions-per-client = 5

# If true, disable the websocket interface to the RPC service.  This has
# the effect of disabling the /subscribe, /unsubscribe, and /unsubscribe_all
# methods for event subscription.
#
# Event log settings; check support in your Tenderdash release.
experimental-disable-websocket = false

# The time window size for the event log. All events up to this long before
# the latest (up to EventLogMaxItems) will be available for subscribers to
# fetch via the /events method.  If 0 (the default) the event log and the
# /events RPC method are disabled.
event-log-window-size = "0s"

# The maxiumum number of events that may be retained by the event log.  If
# this value is 0, no upper limit is set. Otherwise, items in excess of
# this number will be discarded from the event log.
#
# Warning: This setting is a safety valve. Setting it too low may cause
# subscribers to miss events.  Try to choose a value higher than the
# maximum worst-case expected event load within the chosen window size in
# ordinary operation.
#
# For example, if the window size is 10 minutes and the node typically
# averages 1000 events per ten minutes, but with occasional known spikes of
# up to 2000, choose a value > 2000.
event-log-max-items = 0

# How long to wait for a tx to be committed during /broadcast_tx_commit.
# WARNING: Using a value larger than 10s will result in increasing the
# global HTTP write timeout, which applies to all connections and endpoints.
# See https://github.com/tendermint/tendermint/issues/3435
timeout-broadcast-tx-commit = "10s"

# Maximum size of request body, in bytes
max-body-bytes = 1000000

# Maximum size of request header, in bytes
max-header-bytes = 1048576

# The path to a file containing certificate that is used to create the HTTPS server.
# Might be either absolute path or path related to Tenderdash's config directory.
# If the certificate is signed by a certificate authority,
# the certFile should be the concatenation of the server's certificate, any intermediates,
# and the CA's certificate.
# NOTE: both tls-cert-file and tls-key-file must be present for Tenderdash to create HTTPS server.
# Otherwise, HTTP server is run.
tls-cert-file = ""

# The path to a file containing matching private key that is used to create the HTTPS server.
# Might be either absolute path or path related to Tenderdash's config directory.
# NOTE: both tls-cert-file and tls-key-file must be present for Tenderdash to create HTTPS server.
# Otherwise, HTTP server is run.
tls-key-file = ""

# pprof listen address (https://golang.org/pkg/net/http/pprof)
pprof-laddr = ""

#######################################################
###           P2P Configuration Options             ###
#######################################################
[p2p]

# Select the p2p internal queue
queue-type = "priority"

# Address to listen for incoming connections
laddr = "tcp://0.0.0.0:26656"

# Address to advertise to peers for them to dial
# If empty, will use the same port as the laddr,
# and will introspect on the listener or use UPnP
# to figure out the address. ip and port are required
# example: 159.89.10.97:26656
external-address = ""

# Comma separated list of seed nodes to connect to
# We only use these if we can’t connect to peers in the addrbook
# NOTE: not used by the new PEX reactor. Please use BootstrapPeers instead.
# TODO: Remove once p2p refactor is complete
# ref: https:#github.com/tendermint/tendermint/issues/5670
seeds = ""

# Comma separated list of peers to be added to the peer store
# on startup. Either BootstrapPeers or PersistentPeers are
# needed for peer discovery
bootstrap-peers = ""

# Comma separated list of nodes to keep persistent connections to
persistent-peers = ""

# UPNP port forwarding
upnp = false

# Maximum number of connections (inbound and outbound).
max-connections = 64

# Rate limits the number of incoming connection attempts per IP address.
max-incoming-connection-attempts = 100

# Set true to enable the peer-exchange reactor
pex = true

# Comma separated list of peer IDs to keep private (will not be gossiped to other peers)
# Warning: IPs will be exposed at /net_info, for more information https://github.com/tendermint/tendermint/issues/3055
private-peer-ids = ""

# Toggle to disable guard against peers connecting from the same ip.
allow-duplicate-ip = false

# Peer connection configuration.
handshake-timeout = "20s"
dial-timeout = "3s"

# Time to wait before flushing messages out on the connection
# TODO: Remove once MConnConnection is removed.
flush-throttle-timeout = "100ms"

# Maximum size of a message packet payload, in bytes; must be greater than 0
# TODO: Remove once MConnConnection is removed.
max-packet-msg-payload-size = 1400

# Rate at which packets can be sent, in bytes/second
# TODO: Remove once MConnConnection is removed.
send-rate = 5120000

# Rate at which packets can be received, in bytes/second
# TODO: Remove once MConnConnection is removed.
recv-rate = 5120000


#######################################################
###          Mempool Configuration Option          ###
#######################################################
[mempool]

recheck = true
broadcast = true

# Maximum number of transactions in the mempool
size = 5000

# Limit the total size of all txs in the mempool.
# This only accounts for raw transactions (e.g. given 1MB transactions and
# max-txs-bytes=5MB, mempool will only accept 5 transactions).
max-txs-bytes = 1073741824

# Size of the cache (used to filter transactions we saw earlier) in transactions
cache-size = 10000

# Do not remove invalid transactions from the cache (default: false)
# Set to true if it's not possible for any invalid transaction to become valid
# again in the future.
# When true, resubmitting a rejected transaction over RPC returns the CheckTx
# result that rejected it, as long as it stays in the cache.
keep-invalid-txs-in-cache = false

# Maximum size of a single transaction.
# NOTE: the max size of a tx transmitted over the network is {max-tx-bytes}.
max-tx-bytes = 1048576

# Maximum size of a batch of transactions to send to a peer
# Including space needed by encoding (one varint per transaction).
# XXX: Unused due to https://github.com/tendermint/tendermint/issues/5796
max-batch-bytes = 0

# ttl-duration, if non-zero, defines the maximum amount of time a transaction
# can exist for in the mempool.
#
# Note, if ttl-num-blocks is also defined, a transaction will be removed if it
# has existed in the mempool at least ttl-num-blocks number of blocks or if it's
# insertion time into the mempool is beyond ttl-duration.
ttl-duration = "0s"

# ttl-num-blocks, if non-zero, defines the maximum number of blocks a transaction
# can exist for in the mempool.
#
# Note, if ttl-duration is also defined, a transaction will be removed if it
# has existed in the mempool at least ttl-num-blocks number of blocks or if
# it's insertion time into the mempool is beyond ttl-duration.
ttl-num-blocks = 0

#######################################################
###         State Sync Configuration Options        ###
#######################################################
[statesync]
# State sync rapidly bootstraps a new node by discovering, fetching, and restoring a state machine
# snapshot from peers instead of fetching and replaying historical blocks. Requires some peers in
# the network to take and serve state machine snapshots. State sync is not attempted if the node
# has any local state (LastBlockHeight > 0). The node will have a truncated block history,
# starting from the height of the snapshot.
enable = false

# State sync uses light client verification to verify state. This can be done either through the
# P2P layer or RPC layer. Set this to true to use the P2P layer. If false (default), RPC layer
# will be used.
use-p2p = false

# If using RPC, at least two addresses need to be provided. They should be compatible with net.Dial,
# for example: "host.example.com:2125"
rpc-servers = ""

# Time to spend discovering snapshots before initiating a restore.
discovery-time = "15s"

# Number of completed snapshot discovery sweeps. When retries are exhausted, the node will
# fall back to the regular block sync. Set to 0 to retry
# indefinitely, never falling back to block sync. Default is 3.
# Note that in pessimistic case, it will take at least (discovery-time * retries) before
# falling back to block sync. Each sweep visits at most 1,024 peers in batches of 16;
# each batch takes one discovery interval. New peer arrivals do not extend a sweep.
retries = 3

# Temporary directory for state sync snapshot chunks, defaults to os.TempDir().
# The synchronizer will create a new, randomly named directory within this directory
# and remove it when the sync is complete.
temp-dir = ""

# The timeout duration before re-requesting a chunk, possibly from a different
# peer (default: 15 seconds).
chunk-request-timeout = "15s"

# The number of concurrent chunk and block fetchers to run (default: 4).
fetchers = "4"

#######################################################
###         Consensus Configuration Options         ###
#######################################################
[consensus]

wal-file = "data/cs.wal/wal"

# Legacy compatibility setting; currently unused by consensus.
# This does not provide double-sign protection. Preserve the signer's state
# and never run two active signers with the same consensus identity.
double-sign-check-height = 0

# EmptyBlocks mode and possible interval between empty blocks
create-empty-blocks = true
create-empty-blocks-interval = "0s"

# Reactor sleep duration parameters. peer-gossip-sleep-duration also sets the
# tick rate a lagging peer's catch-up block-part replay is metered against
# (a fixed 500ms internal throttle): raising this above 500ms disables that
# throttle. peer-query-maj23-sleep-duration also sets how often the node
# re-announces its round state to each peer (recovery latency after a lost
# announcement). Either duration under 1ms is run at 1ms, which only prevents a
# busy loop: well below the defaults, peers drop this node's state updates over
# their per-peer allowance.
peer-gossip-sleep-duration = "100ms"
peer-query-maj23-sleep-duration = "2s"

### Unsafe Timeout Overrides ###

# These fields provide temporary overrides for the Timeout consensus parameters.
# Use of these parameters is strongly discouraged. Using these parameters may have serious
# liveness implications for the validator and for the chain.
#
# Because they make a single node disagree with the rest of the network about
# timing, they are not intended for normal operation.

# This field provides an unsafe override of the Propose timeout consensus parameter.
# This field configures how long the consensus engine will wait for a proposal block before prevoting nil.
# If this field is set to a value greater than 0, it will take effect.
# unsafe-propose-timeout-override = 0s

# This field provides an unsafe override of the ProposeDelta timeout consensus parameter.
# This field configures how much the propose timeout increases with each round.
# If this field is set to a value greater than 0, it will take effect.
# unsafe-propose-timeout-delta-override = 0s

# This field provides an unsafe override of the Vote timeout consensus parameter.
# This field configures how long the consensus engine will wait after
# receiving the configured voting threshold in a round.
# If this field is set to a value greater than 0, it will take effect.
# unsafe-vote-timeout-override = 0s

# This field provides an unsafe override of the VoteDelta timeout consensus parameter.
# This field configures how much the vote timeout increases with each round.
# If this field is set to a value greater than 0, it will take effect.
# unsafe-vote-timeout-delta-override = 0s

#######################################################
###   Transaction Indexer Configuration Options     ###
#######################################################
[tx-index]

# The backend database list to back the indexer.
# An empty list or a list containing "null" disables indexing.
#
# The application will set which txs to index. In some cases a node operator will be able
# to decide which txs to index based on configuration set in the application.
#
# Options:
#   1) "null" (default) - no indexer services.
#   2) "kv" - an indexer backed by key-value storage (see DBBackend).
# When "kv" is chosen "tx.height" and "tx.hash" will always be indexed.
indexer = ["kv"]


#######################################################
###       Instrumentation Configuration Options     ###
#######################################################
[instrumentation]

# When true, Prometheus metrics are served under /metrics on
# PrometheusListenAddr.
# Check out the documentation for the list of available metrics.
prometheus = false

# Address to listen for Prometheus collector(s) connections
prometheus-listen-addr = ":26660"

# Maximum number of simultaneous connections.
# If you want to accept a larger number than the default, make sure
# you increase your OS limits.
# 0 - unlimited.
max-open-connections = 3

# Instrumentation namespace
namespace = "tendermint"

Empty blocks

create-empty-blocks = false makes round 0 wait for transactions above the initial height. A positive create-empty-blocks-interval starts a proposal after that interval even without transactions. Later rounds do not wait for transactions, except on a chain with consensus version 2, where a round entered because the validators skipped a silent proposer's waiting round may wait too, unless the node holds a locked or valid block, the mempool has already reported transactions at this height, a polka exists at the height, or a prevote of that round was recorded; its interval counts from the start of the height. See Proposer liveness and round skipping.

The application can bypass that wait for the next height by setting ResponseFinalizeBlock.propose_next_block_immediately. The hint is local, consumed once and not persisted across restarts. There is no commit timeout before the next height starts.

Tenderdash uses same-block execution: block H's AppHash describes the state after executing H. An application hash change does not require an additional "proof block" at H+1.

See the canonical empty-block rules, execution order and timeout parameters.

P2P settings

This section will cover settings within the p2p section of the config.toml.

  • external-address = is the address that will be advertised for other nodes to use. We recommend setting this field with your public IP and p2p port.
    • We recommend setting an external address. When used in a private network, Tenderdash Core currently doesn't advertise the node's public address. There is active and ongoing work to improve the P2P system, but this is a helpful workaround for now.

  • persistent-peers = is a list of comma separated peers that you will always want to be connected to. If you're already connected to the maximum number of peers, persistent peers will not be added.
  • pex = turns the peer exchange reactor on or off. Validator node will want the pex turned off so it would not begin gossiping to unknown peers on the network. PeX can also be turned off for statically configured networks with fixed network connectivity. For full nodes on open, dynamic networks, it should be turned on.
  • allowlist-only = if true, only peers from persistent-peers and bootstrap-peers are allowed to connect (inbound and outbound).
  • private-peer-ids = is a comma-separated list of node ids that will not be exposed to other peers (i.e., you will not tell other peers about the ids in this list). This can be filled with a validator's node id.

The P2P configuration includes parameters for the router and legacy networking implementations. Check the configuration generated by your installed version.

We will cover the new and deprecated parameters below.

New Parameters

There are three new parameters, which are enabled if use-legacy is set to false.

  • queue-type = sets a type of queue to use in the p2p layer. There are three options available fifo, priority and wdrr. The default is priority
  • bootstrap-peers = is a list of comma seperated peers which will be used to bootstrap the address book.
  • max-connections = is the max amount of allowed inbound and outbound connections.

Deprecated Parameters

These legacy settings differ from the router settings above. Use the settings supported by the networking implementation in your Tenderdash release.

  • max-num-inbound-peers = is the maximum number of peers you will accept inbound connections from at one time (where they dial your address and initiate the connection). This was replaced by max-connections
  • max-num-outbound-peers = is the maximum number of peers you will initiate outbound connects to at one time (where you dial their address and initiate the connection).This was replaced by max-connections
  • unconditional-peer-ids = is similar to persistent-peers except that these peers will be connected to even if you are already connected to the maximum number of peers. This can be a validator node ID on your sentry node. Deprecated
  • seeds = is a list of comma separated seed nodes that you will connect upon a start and ask for peers. A seed node is a node that does not participate in consensus but only helps propagate peers to nodes in the networks Deprecated, replaced by bootstrap peers

Indexing Settings

Operators can configure indexing via the [tx-index] section. The indexer field takes a series of supported indexers. If null is included, indexing will be turned off. All configured indexer names must still be supported.

Supported Indexers

KV

The kv indexer type is an embedded key-value store supported by the main underlying Tenderdash database. Using the kv indexer type allows you to query for block and transaction events directly against Tenderdash's RPC. However, the query syntax is limited to the supported event-query operators.

Removed PostgreSQL indexer

The psql event indexer is no longer supported. Selecting it, including alongside kv or null, returns a configuration error. Choose indexer = ["kv"] for local indexing and RPC searches, or ["null"] to disable indexing. The obsolete psql-conn key is ignored, so existing Dashmate configurations with an empty value remain compatible.

Existing PostgreSQL data is not deleted or migrated, and SQL consumers no longer receive new events. To populate KV indexes, explicitly run tenderdash reindex-event against the retained local blocks and results after selecting kv. Pruned history cannot be reconstructed by this command.

Unsafe Consensus Timeout Overrides

Tenderdash provides a set of unsafe overrides for the consensus timing parameters. They are a safety measure for unusual timing issues, letting an operator override the timings for a single node. Because they make one node disagree with the rest of the network about timing, they are unsafe and are not intended for normal operation.

  • unsafe-propose-timeout-override: How long the Tenderdash consensus engine will wait for a proposal block before prevoting nil.
  • unsafe-propose-timeout-delta-override: How much the propose timeout increases with each round.
  • unsafe-vote-timeout-override: How long the consensus engine will wait after receiving the configured voting threshold in a round.
  • unsafe-vote-timeout-delta-override: How much the vote timeout increases with each round.

Configure a Light Client

Unsupported. The Tenderdash light client is currently unsupported. The implementation details below are provided for reference.

Tenderdash includes a tenderdash light command that runs an RPC proxy backed by a verifying light client. Requests with supported proofs can be checked before results are returned. Other RPC methods do not automatically gain proof verification.

The proxy requires a chain ID, a primary Tenderdash RPC provider and witness providers. The current implementation uses Dash Core verification for quorum signatures, so it also requires access to the appropriate Dash Core RPC service. It does not use upstream trust-height and trust-hash flags.

Inspect the options for your installed release:

tenderdash light --help

Set --primary, --witnesses and the Dash Core connection settings for your network. Use --dir to select an explicit directory for the light client's local database and --laddr to control the proxy listen address.

Keep Core credentials private and restrict RPC network access. Ensure that the application's query proof format is compatible with the proxy before relying on /abci_query verification.

See the light client overview, command implementation and client implementation for the supported configuration and verification behavior.

Logging

Logging adds detail and allows the node operator to better identify what they are looking for. Tenderdash supports log levels on a global and per-module basis. This allows the node operator to see only the information they need and the developer to hone in on specific changes they are working on.

Configuring Log Levels

There are three log levels, info, debug and error. These can be configured either through the command line via tenderdash start --log-level "" or within the config.toml file.

  • info Info represents an informational message. It is used to show that modules have started, stopped and how they are functioning.
  • debug Debug is used to trace various calls or problems. Debug is used widely throughout a codebase and can lead to overly verbose logging.
  • error Error represents something that has gone wrong. An error log can represent a potential problem that can lead to a node halt.

Within the config.toml:

# Output level for logging, including package level options
log-level = "info"

Via the command line:

tenderdash start --log-level "info"

List of modules

Here is the list of modules you may encounter in Tenderdash's log and a little overview what they do.

  • abci-client As mentioned in Application Architecture Guide, Tenderdash acts as an ABCI client with respect to the application and maintains 3 connections: mempool, consensus and query. The code used by Tenderdash Core can be found here.
  • blockchain Provides storage, pool (a group of peers), and reactor for both storing and exchanging blocks between peers.
  • consensus The heart of Tenderdash core, which is the implementation of the consensus algorithm. Includes two "submodules": wal (write-ahead logging) for ensuring data integrity and replay to replay blocks and messages on recovery from a crash. here. You can subscribe to them by calling subscribe RPC method. Refer to RPC docs for additional information.
  • mempool Mempool module handles all incoming transactions, whenever they are coming from peers or the application.
  • p2p Provides an abstraction around peer-to-peer communication. For more details, please check out the README.
  • rpc-server RPC server. For implementation details, please read the doc.go.
  • state Represents the latest state and execution submodule, which executes blocks against the application.
  • statesync Provides a way to quickly sync a node with pruned history.

Walkabout example

We first create three connections (mempool, consensus and query) to the application (running kvstore locally in this case).

I[10-04|13:54:27.364] Starting multiAppConn                        module=proxy impl=multiAppConn
I[10-04|13:54:27.366] Starting localClient                         module=abci-client connection=query impl=localClient
I[10-04|13:54:27.366] Starting localClient                         module=abci-client connection=mempool impl=localClient
I[10-04|13:54:27.367] Starting localClient                         module=abci-client connection=consensus impl=localClient

Then Tenderdash Core and the application perform a handshake.

I[10-04|13:54:27.367] ABCI Handshake                               module=consensus appHeight=90 appHash=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD
I[10-04|13:54:27.368] ABCI Replay Blocks                           module=consensus appHeight=90 storeHeight=90 stateHeight=90
I[10-04|13:54:27.368] Completed ABCI Handshake - Tenderdash and App are synced module=consensus appHeight=90 appHash=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD

After that, we start a few more things like the event switch, reactors, and perform UPNP discover in order to detect the IP address.

I[10-04|13:54:27.374] Starting EventSwitch                         module=types impl=EventSwitch
I[10-04|13:54:27.375] This node is a validator                     module=consensus
I[10-04|13:54:27.379] Starting Node                                module=main impl=Node
I[10-04|13:54:27.381] Local listener                               module=p2p ip=:: port=26656
I[10-04|13:54:27.382] Getting UPNP external address                module=p2p
I[10-04|13:54:30.386] Could not perform UPNP discover              module=p2p err="write udp4 0.0.0.0:38238->239.255.255.250:1900: i/o timeout"
I[10-04|13:54:30.386] Starting DefaultListener                     module=p2p impl=Listener(@10.0.2.15:26656)
I[10-04|13:54:30.387] Starting P2P Switch                          module=p2p impl="P2P Switch"
I[10-04|13:54:30.387] Starting MempoolReactor                      module=mempool impl=MempoolReactor
I[10-04|13:54:30.387] Starting BlockchainReactor                   module=blockchain impl=BlockchainReactor
I[10-04|13:54:30.387] Starting ConsensusReactor                    module=consensus impl=ConsensusReactor
I[10-04|13:54:30.387] ConsensusReactor                             module=consensus fastSync=false
I[10-04|13:54:30.387] Starting ConsensusState                      module=consensus impl=ConsensusState
I[10-04|13:54:30.387] Starting WAL                                 module=consensus wal=/home/vagrant/.tenderdash/data/cs.wal/wal impl=WAL
I[10-04|13:54:30.388] Starting TimeoutTicker                       module=consensus impl=TimeoutTicker

Notice the second row where Tenderdash Core reports that "This node is a validator". It also could be just an observer (regular node).

Next we replay all the messages from the WAL.

I[10-04|13:54:30.390] Catchup by replaying consensus messages      module=consensus height=91
I[10-04|13:54:30.390] Replay: New Step                             module=consensus height=91 round=0 step=RoundStepNewHeight
I[10-04|13:54:30.390] Replay: Done                                 module=consensus

"Started node" message signals that everything is ready for work.

I[10-04|13:54:30.391] Starting RPC HTTP server on tcp socket 0.0.0.0:26657 module=rpc-server
I[10-04|13:54:30.392] Started node                                 module=main nodeInfo="NodeInfo{id: DF22D7C92C91082324A1312F092AA1DA197FA598DBBFB6526E, moniker: anonymous, network: test-chain-3MNw2N [remote , listen 10.0.2.15:26656], version: 0.11.0-10f361fc ([wire_version=0.6.2 p2p_version=0.5.0 consensus_version=v1/0.2.2 rpc_version=0.7.0/3 tx_index=on rpc_addr=tcp://0.0.0.0:26657])}"

Next follows a standard block creation cycle, where we enter a new round, propose a block, receive more than 2/3 of prevotes, then precommits and finally have a chance to commit a block. For details, please refer to Byzantine Consensus Algorithm.

I[10-04|13:54:30.393] enterNewRound(91/0). Current: 91/0/RoundStepNewHeight module=consensus
I[10-04|13:54:30.393] enterPropose(91/0). Current: 91/0/RoundStepNewRound module=consensus
I[10-04|13:54:30.393] enterPropose: Our turn to propose            module=consensus proposer=125B0E3C5512F5C2B0E1109E31885C4511570C42 privValidator="PrivValidator{125B0E3C5512F5C2B0E1109E31885C4511570C42 LH:90, LR:0, LS:3}"
I[10-04|13:54:30.394] Signed proposal                              module=consensus height=91 round=0 proposal="Proposal{91/0 1:21B79872514F (-1,:0:000000000000) {/10EDEDD7C84E.../}}"
I[10-04|13:54:30.397] Received complete proposal block             module=consensus height=91 hash=F671D562C7B9242900A286E1882EE64E5556FE9E
I[10-04|13:54:30.397] enterPrevote(91/0). Current: 91/0/RoundStepPropose module=consensus
I[10-04|13:54:30.397] enterPrevote: ProposalBlock is valid         module=consensus height=91 round=0
I[10-04|13:54:30.398] Signed and pushed vote                       module=consensus height=91 round=0 vote="Vote{0:125B0E3C5512 91/00/1(Prevote) F671D562C7B9 {/89047FFC21D8.../}}" err=null
I[10-04|13:54:30.401] Added to prevote                             module=consensus vote="Vote{0:125B0E3C5512 91/00/1(Prevote) F671D562C7B9 {/89047FFC21D8.../}}" prevotes="VoteSet{H:91 R:0 T:1 +2/3:F671D562C7B9242900A286E1882EE64E5556FE9E:1:21B79872514F BA{1:X} map[]}"
I[10-04|13:54:30.401] enterPrecommit(91/0). Current: 91/0/RoundStepPrevote module=consensus
I[10-04|13:54:30.401] enterPrecommit: +2/3 prevoted proposal block. Locking module=consensus hash=F671D562C7B9242900A286E1882EE64E5556FE9E
I[10-04|13:54:30.402] Signed and pushed vote                       module=consensus height=91 round=0 vote="Vote{0:125B0E3C5512 91/00/2(Precommit) F671D562C7B9 {/80533478E41A.../}}" err=null
I[10-04|13:54:30.404] Added to precommit                           module=consensus vote="Vote{0:125B0E3C5512 91/00/2(Precommit) F671D562C7B9 {/80533478E41A.../}}" precommits="VoteSet{H:91 R:0 T:2 +2/3:F671D562C7B9242900A286E1882EE64E5556FE9E:1:21B79872514F BA{1:X} map[]}"
I[10-04|13:54:30.404] enterCommit(91/0). Current: 91/0/RoundStepPrecommit module=consensus
I[10-04|13:54:30.405] Finalizing commit of block with 0 txs        module=consensus height=91 hash=F671D562C7B9242900A286E1882EE64E5556FE9E root=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD
I[10-04|13:54:30.405] Block{
  Header{
    ChainID:        test-chain-3MNw2N
    Height:         91
    Time:           2017-10-04 13:54:30.393 +0000 UTC
    NumTxs:         0
    LastBlockID:    F15AB8BEF9A6AAB07E457A6E16BC410546AA4DC6:1:D505DA273544
    LastCommit:     56FEF2EFDB8B37E9C6E6D635749DF3169D5F005D
    Data:
    Validators:     CE25FBFF2E10C0D51AA1A07C064A96931BC8B297
    App:            E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD
  }#F671D562C7B9242900A286E1882EE64E5556FE9E
  Data{

  }#
  Commit{
    BlockID:    F15AB8BEF9A6AAB07E457A6E16BC410546AA4DC6:1:D505DA273544
    Precommits: Vote{0:125B0E3C5512 90/00/2(Precommit) F15AB8BEF9A6 {/FE98E2B956F0.../}}
  }#56FEF2EFDB8B37E9C6E6D635749DF3169D5F005D
}#F671D562C7B9242900A286E1882EE64E5556FE9E module=consensus
I[10-04|13:54:30.408] Executed block                               module=state height=91 validTxs=0 invalidTxs=0
I[10-04|13:54:30.410] Committed state                              module=state height=91 txs=0 hash=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD
I[10-04|13:54:30.410] Recheck txs                                  module=mempool numtxs=0 height=91

Metrics

Tenderdash can report and serve the Prometheus metrics, which in their turn can be consumed by Prometheus collector(s).

This functionality is disabled by default.

To enable the Prometheus metrics, set instrumentation.prometheus=true in your config file. Metrics will be served under /metrics on 26660 port by default. Listen address can be changed in the config file (see instrumentation.prometheus\_listen\_addr).

List of available metrics

The following metrics are available:

NameTypeTagsDescription
abci_connection_method_timingHistogrammethod, typeTimings for each of the ABCI methods
consensus_heightGaugeHeight of the chain
consensus_validatorsGaugeNumber of validators
consensus_validators_powerGaugeTotal voting power of all validators
consensus_validator_powerGaugeVoting power of the node if in the validator set
consensus_validator_last_signed_heightGaugeLast height the node signed a block, if the node is a validator
consensus_validator_missed_blocksGaugeTotal amount of blocks missed for the node, if the node is a validator
consensus_missing_validatorsGaugeNumber of validators who did not sign
consensus_missing_validators_powerGaugeTotal voting power of the missing validators
consensus_byzantine_validatorsGaugeNumber of validators who tried to double sign
consensus_byzantine_validators_powerGaugeTotal voting power of the byzantine validators
consensus_block_interval_secondsHistogramTime between this and last block (Block.Header.Time) in seconds
consensus_roundsGaugeNumber of rounds
consensus_num_txsGaugeNumber of transactions
consensus_total_txsGaugeTotal number of transactions committed
consensus_block_partsCounterpeer_idnumber of blockparts transmitted by peer
consensus_latest_block_heightgauge/status sync_info number
consensus_fast_syncinggaugeeither 0 (not fast syncing) or 1 (syncing)
consensus_state_syncinggaugeeither 0 (not state syncing) or 1 (syncing)
consensus_block_size_bytesGaugeBlock size in bytes
consensus_step_durationHistogramstepHistogram of durations for each step in the consensus protocol
consensus_block_gossip_receive_latencyHistogramHistogram of time taken to receive a block in seconds, measure between when a new block is first discovered to when the block is completed
consensus_block_gossip_parts_receivedCountermatches_currentNumber of block parts received by the node
consensus_block_sync_apply_stage_durationHistogramstageTime a block spent in each stage of the block sync applier, in ms: decode, partset, verify_commit, verify_block, save, exec, and wait (idle time between blocks)
consensus_quorum_prevote_delayGaugeInterval in seconds between the proposal timestamp and the timestamp of the earliest prevote that achieved a quorum
consensus_full_prevote_delayGaugeInterval in seconds between the proposal timestamp and the timestamp of the latest prevote in a round where all validators voted
consensus_proposal_timestamp_differenceHistogramDifference between the timestamp in the proposal message and the local time of the validator at the time it received the message
consensus_vote_extension_receive_countCounterstatusNumber of vote extensions received
consensus_proposal_receive_countCounterstatusTotal number of proposals received by the node since process start
consensus_proposal_create_countCounterTotal number of proposals created by the node since process start
consensus_round_voting_power_percentGaugevote_typeA value between 0 and 1.0 representing the percentage of the total voting power per vote type received within a round
consensus_late_votesCountervote_typeNumber of votes received by the node since process start that correspond to earlier heights and rounds than this node is currently in.
consensus_verification_budget_dropsCounterPeer Vote and Commit messages dropped because the signature-verification budget was exhausted
consensus_peer_lane_dropsCounterQueued peer consensus messages dropped by the per-peer scheduler (backlog full or peer disconnected)
consensus_block_part_proof_dropsCounterPeer block parts dropped before proof verification because the sender's invalid-proof allowance was spent
consensus_state_channel_dropsCounterState and VoteSetBits channel messages dropped over the per-peer or node-wide ceiling
consensus_proposal_verify_failuresCounterPeer proposals whose signature failed verification
consensus_commit_verify_failuresCounterreasonCommits (from peers, block sync or this node's own precommits) refused by verification; reason is quorum_hash, extension_count, extensions_rejected, invalid_signature, budget or other
consensus_peer_vote_verify_latency_secondsHistogramSeconds an accepted peer vote spent from being queued by the reactor to being added to the vote set
consensus_verification_budget_saturationGaugeFraction of the node-wide verification budget still available at a peer-message check (1.0 full/idle, 0.0 drained/under pressure)
consensus_peer_lane_active_countGaugeNumber of per-peer lanes currently holding queued messages and taking turns in the scheduler
consensus_peer_lane_max_depthGaugeLargest number of messages queued in any single peer lane
consensus_proposer_heartbeats_sentCounterProposer heartbeats this node signed and broadcast while waiting for transactions as proposer (consensus version 2)
consensus_proposer_heartbeats_receivedCounterNew, verified proposer heartbeats accepted for the round this node waits in
consensus_proposer_silencesCounterRounds in which the proposer went silent while this validator waited for transactions
consensus_round_skipsCounterRounds this node moved past on a round skip certificate from +2/3 of the validators (by default; configuration can change the threshold)
evidence_pool_num_evidenceGaugeNumber of evidence in the evidence pool
evidence_pool_dropped_evidenceCounterreasonInbound gossip evidence refused before verification, labeled by reason (structural, oversized, already_known, height_window, node_budget, peer_budget)
p2p_peersGaugeNumber of peers node's connected to
p2p_peer_receive_bytes_totalCounterpeer_id, chIDnumber of bytes per channel received from a given peer
p2p_peer_send_bytes_totalCounterpeer_id, chIDnumber of bytes per channel sent to a given peer
p2p_peer_pending_send_bytesGaugepeer_idnumber of pending bytes to be sent to a given peer
p2p_router_peer_queue_recvHistogramThe time taken to read off of a peer's queue before sending on the connection
p2p_router_peer_queue_sendHistogramThe time taken to send on a peer's queue which will later be sent on the connection
p2p_router_channel_queue_sendHistogramThe time taken to send on a p2p channel's queue which will later be consumed by the corresponding service
p2p_router_channel_queue_dropped_msgsCounterch_idThe number of messages dropped from a peer's queue for a specific p2p channel
p2p_peer_queue_msg_sizeGaugech_idThe size of messages sent over a peer's queue for a specific p2p channel
mempool_sizeGaugeNumber of uncommitted transactions
mempool_tx_size_bytesHistogramtransaction sizes in bytes
mempool_failed_txsCounternumber of failed transactions
mempool_recheck_timesCounternumber of transactions rechecked in the mempool
state_block_processing_timeHistogramtime between BeginBlock and EndBlock in ms
state_consensus_param_updatesCounternumber of consensus parameter updates returned by the application since process start
state_validator_set_updatesCounternumber of validator set updates returned by the application since process start
state_block_apply_stage_durationHistogramstageTime spent in each stage of applying a committed block, in ms: the ABCI calls, the state and ABCI response writes, the mempool update and the event publish

Useful queries

Percentage of missing + byzantine validators:

((consensus_byzantine_validators_power + consensus_missing_validators_power) / consensus_validators_power) * 100

Rate at which the application is responding to each ABCI method call.

sum(rate(tendermint_abci_connection_method_timing_count[5m])) by (method)

The 95th percentile response time for the application to the deliver_tx ABCI method call.

histogram_quantile(0.95, sum by(le) (rate(tendermint_abci_connection_method_timing_bucket{method="deliver_tx"}[5m])))

Remote signer

Tenderdash provides a remote signer option for validators. A remote signer enables the operator to store the validator key on a different machine minimizing the attack surface if a server were to be compromised.

The remote signer protocol implements a client and server architecture. When Tenderdash requires the public key or signature for a proposal or vote it requests it from the remote signer.

The remote signer protocol does not cover proposer heartbeats or round skip votes. On a chain with consensus version 2, a validator using a remote signer sends neither, so validators skip its rounds that wait for transactions once its heartbeats are missed. See Proposer liveness and round skipping.

To run a secure validator and remote signer system it is recommended to use a VPC (virtual private cloud) or a private connection.

There are two different configurations that can be used: Raw or gRPC.

Raw

While both options use tcp or unix sockets the raw option uses tcp or unix sockets without http. The raw protocol sets up Tenderdash as the server and the remote signer as the client. This aids in not exposing the remote signer to public network.

Warning: Raw will be deprecated in a future major release, we recommend implementing your key management server against the gRPC configuration.

gRPC

gRPC is an RPC framework built with HTTP/2, uses Protocol Buffers to define services and has been standardized within the cloud infrastructure community. gRPC provides a language agnostic way to implement services. This aids developers in the writing key management servers in various different languages.

GRPC utilizes TLS, another widely standardized protocol, to secure connections. There are two forms of TLS to secure a connection, one-way and two-way. One way is when the client identifies the server but the server allows anyone to connect to it. Two-way is when the client identifies the server and the server identifies the client, prohibiting connections from unknown parties.

When using gRPC Tenderdash is setup as the client. Tenderdash will make calls to the remote signer. We recommend not exposing the remote signer to the public network with the use of virtual private cloud.

Securing your remote signers connection is highly recommended, but we provide the option to run it with a insecure connection.

Generating Certificates

To run a secure connection with gRPC we need to generate certificates and keys. We will walkthrough how to self sign certificates for two-way TLS.

There are two ways to generate certificates, openssl and certstarp. Both of these options can be used but we will be covering certstrap because it provides a simpler process then openssl.

  • Install Certstrap:
  go install github.com/square/certstrap@v1.2.0
  • Create certificate authority for self signing.
 # generate self signing ceritificate authority
 certstrap init --common-name "<name_CA>" --expires "20 years"
  • Request a certificate for the server.
    • For generalization purposes we set the ip to 127.0.0.1, but for your node please use the servers IP.
  • Sign the servers certificate with your certificate authority
 # generate server cerificate
 certstrap request-cert -cn server -ip 127.0.0.1
 # self-sign server cerificate with rootCA
 certstrap sign server --CA "<name_CA>" 127.0.0.1
  • Request a certificate for the client.
    • For generalization purposes we set the ip to 127.0.0.1, but for your node please use the clients IP.
  • Sign the clients certificate with your certificate authority
# generate client cerificate
 certstrap request-cert -cn client -ip 127.0.0.1
# self-sign client cerificate with rootCA
 certstrap sign client --CA "<name_CA>" 127.0.0.1

Running in production

If you are building Tenderdash from source for use in production, make sure to check out an appropriate Git tag instead of a branch.

Database

By default, Tenderdash uses the syndtr/goleveldb package for its in-process key-value database. If you want maximal performance, it may be best to install the real C-implementation of LevelDB and compile Tenderdash to use that using make build TENDERMINT_BUILD_OPTIONS=cleveldb. See the install instructions for details.

Tenderdash keeps multiple distinct databases in the $TMROOT/data:

  • blockstore.db: Keeps the entire blockchain - stores blocks, block commits, and block meta data, each indexed by height. Used to sync new peers.
  • evidence.db: Stores all verified evidence of misbehaviour.
  • state.db: Stores the current blockchain state (ie. height, validators, consensus params). Only grows if consensus params or validators change. Also used to temporarily store intermediate results during block processing.
  • tx_index.db: Indexes txs (and their results) by tx hash and by DeliverTx result events.

By default, Tenderdash will only index txs by their hash and height, not by their DeliverTx result events. See indexing transactions for details.

Applications can expose block pruning strategies to the node operator. Please read the documentation of your application to find out more details.

Applications can use state sync to help nodes bootstrap quickly.

Logging

Default logging level (log-level = "info") should suffice for normal operation mode. Read this post for details on how to configure log-level config variable. Some of the modules can be found here. If you're trying to debug Tenderdash or asked to provide logs with debug logging level, you can do so by running Tenderdash with --log-level="debug".

Consensus WAL

Tenderdash uses a write ahead log (WAL) for consensus. The consensus.wal is used to ensure we can recover from a crash at any point in the consensus state machine. It writes all consensus messages (timeouts, proposals, block part, or vote) to a single file, flushing to disk before processing messages from its own validator. Since Tenderdash validators are expected to never sign a conflicting vote, the WAL ensures we can always recover deterministically to the latest state of the consensus without using the network or re-signing any consensus messages. The consensus WAL max size of 1GB and is automatically rotated.

If your consensus.wal is corrupted, see below.

DOS Exposure and Mitigation

Validators are supposed to setup Sentry Node Architecture to prevent Denial-of-service attacks.

P2P

The core of the Tenderdash peer-to-peer system is MConnection. Each connection has MaxPacketMsgPayloadSize, which is the maximum packet size and bounded send & receive queues. One can impose restrictions on send & receive rate per connection (SendRate, RecvRate).

The number of open P2P connections can become quite large, and hit the operating system's open file limit (since TCP connections are considered files on UNIX-based systems). Nodes should be given a sizable open file limit, e.g. 8192, via ulimit -n 8192 or other deployment-specific mechanisms.

RPC

Endpoints returning multiple entries are limited by default to return 30 elements (100 max). See the RPC Documentation for more information.

Rate-limiting and authentication are another key aspects to help protect against DOS attacks. Validators are supposed to use external tools like NGINX or traefik to achieve the same things.

Debugging Tenderdash

If you ever have to debug Tenderdash, the first thing you should probably do is check out the logs. See Logging, where we explain what certain log statements mean.

If, after skimming through the logs, things are not clear still, the next thing to try is querying the /status RPC endpoint. It provides the necessary info: whenever the node is syncing or not, what height it is on, etc.

curl http(s)://{ip}:{rpcPort}/status

/dump_consensus_state will give you a detailed overview of the consensus state (proposer, latest validators, peers states). From it, you should be able to figure out why, for example, the network had halted.

curl http(s)://{ip}:{rpcPort}/dump_consensus_state

There is a reduced version of this endpoint - /consensus_state, which returns just the votes seen at the current height.

If, after consulting with the logs and above endpoints, you still have no idea what's happening, consider using tenderdash debug kill sub-command. This command will scrap all the available info and kill the process. See Debugging for the exact format.

You can inspect the resulting archive yourself or create an issue on GitHub. Before opening an issue however, be sure to check if there's no existing issue already.

Monitoring Tenderdash

Each Tenderdash instance has a standard /health RPC endpoint, which responds with 200 (OK) if everything is fine and 500 (or no response) - if something is wrong.

Other useful endpoints include mentioned earlier /status, /net_info and /validators.

Tenderdash also can report and serve Prometheus metrics. See Metrics.

tenderdash debug dump sub-command can be used to periodically dump useful information into an archive. See Debugging for more information.

What happens when my app dies

You are supposed to run Tenderdash under a process supervisor (like systemd or runit). It will ensure Tenderdash is always running (despite possible errors).

Getting back to the original question, if your application dies, Tenderdash will panic. After a process supervisor restarts your application, Tenderdash should be able to reconnect successfully. The order of restart does not matter for it.

Signal handling

We catch SIGINT and SIGTERM and try to clean up nicely. For other signals we use the default behavior in Go: Default behavior of signals in Go programs.

Corruption

NOTE: Make sure you have a backup of the Tenderdash data directory.

Possible causes

Remember that most corruption is caused by hardware issues:

  • RAID controllers with faulty / worn out battery backup, and an unexpected power loss
  • Hard disk drives with write-back cache enabled, and an unexpected power loss
  • Cheap SSDs with insufficient power-loss protection, and an unexpected power-loss
  • Defective RAM
  • Defective or overheating CPU(s)

Other causes can be:

  • Database systems configured with fsync=off and an OS crash or power loss
  • Filesystems configured to use write barriers plus a storage layer that ignores write barriers. LVM is a particular culprit.
  • Tenderdash bugs
  • Operating system bugs
  • Admin error (e.g., directly modifying Tenderdash data-directory contents)

(Source: https://wiki.postgresql.org/wiki/Corruption)

WAL Corruption

If consensus WAL is corrupted at the latest height and you are trying to start Tenderdash, replay will fail with panic.

Recovering from data corruption can be hard and time-consuming. Here are two approaches you can take:

  1. Delete the WAL file and restart Tenderdash. It will attempt to sync with other peers.
  2. Try to repair the WAL file manually:
  1. Create a backup of the corrupted WAL file:

    cp "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup
    
  2. Use ./scripts/wal2json to create a human-readable version:

    ./scripts/wal2json/wal2json "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal
    
  3. Search for a "CORRUPTED MESSAGE" line.

  4. By looking at the previous message and the message after the corrupted one and looking at the logs, try to rebuild the message. If the consequent messages are marked as corrupted too (this may happen if length header got corrupted or some writes did not make it to the WAL ~ truncation), then remove all the lines starting from the corrupted one and restart Tenderdash.

    $EDITOR /tmp/corrupted_wal
    
  5. After editing, convert this file back into binary form by running:

    ./scripts/json2wal/json2wal /tmp/corrupted_wal  $TMHOME/data/cs.wal/wal
    

Hardware

Processor and Memory

While actual specs vary depending on the load and validators count, minimal requirements are:

  • 1GB RAM
  • 25GB of disk space
  • 1.4 GHz CPU

SSD disks are preferable for applications with high transaction throughput.

Recommended:

  • 2GB RAM
  • 100GB SSD
  • x64 2.0 GHz 2v CPU

Disk usage depends on retained block history and application state. State sync can bootstrap from an application snapshot; configure retention according to your recovery and archival requirements.

Validator signing on 32 bit architectures (or ARM)

Both our ed25519 and secp256k1 implementations require constant time uint64 multiplication. Non-constant time crypto can (and has) leaked private keys on both ed25519 and secp256k1. This doesn't exist in hardware on 32 bit x86 platforms (source), and it depends on the compiler to enforce that it is constant time. It's unclear at this point whenever the Golang compiler does this correctly for all implementations.

We do not support nor recommend running a validator on 32 bit architectures OR the "VIA Nano 2000 Series", and the architectures in the ARM section rated "S-".

Operating Systems

Tenderdash supports Linux and macOS only. Windows is not a supported platform and there are no plans to add support for it.

While we do not favor any particular operating system, more secure and stable Linux server distributions (like CentOS) should be preferred over desktop operating systems (like macOS).

Miscellaneous

Size a production node for your Dash Platform network, application workload and retention policy. Measure disk growth, memory use and consensus latency under representative load before deployment.

Configuration parameters

  • p2p.flush-throttle-timeout
  • p2p.max-packet-msg-payload-size
  • p2p.send-rate
  • p2p.recv-rate

If you are going to use Tenderdash in a private domain and you have a private high-speed network among your peers, it makes sense to lower flush throttle timeout and increase other params.

[p2p]
send-rate=20000000 # 2MB/s
recv-rate=20000000 # 2MB/s
flush-throttle-timeout=10
max-packet-msg-payload-size=10240 # 10KB
  • mempool.broadcast

Setting this to false will stop the mempool from relaying transactions to other peers until they are included in a block. It means only the peer you send the tx to will see it until it is included in a block.

  • consensus.peer-gossip-sleep-duration

You can try to reduce the time your node sleeps before checking if theres something to send its peers.

  • p2p.addr-book-strict

By default, Tenderdash checks whenever a peer's address is routable before saving it to the address book. The address is considered as routable if the IP is valid and within allowed ranges.

This may not be the case for private or local networks, where your IP range is usually strictly limited and private. If that case, you need to set addr-book-strict to false (turn it off).

  • rpc.max-open-connections

By default, the number of simultaneous connections is limited because most OS give you limited number of file descriptors.

If you want to accept greater number of connections, you will need to increase these limits.

Sysctls to tune the system to be able to open more connections

The process file limits must also be increased, e.g. via ulimit -n 8192.

...for N connections, such as 50k:

kern.maxfiles=10000+2*N         # BSD
kern.maxfilesperproc=100+2*N    # BSD
kern.ipc.maxsockets=10000+2*N   # BSD
fs.file-max=10000+2*N           # Linux
net.ipv4.tcp_max_orphans=N      # Linux

# For load-generating clients.
net.ipv4.ip_local_port_range="10000  65535"  # Linux.
net.inet.ip.portrange.first=10000  # BSD/Mac.
net.inet.ip.portrange.last=65535   # (Enough for N < 55535)
net.ipv4.tcp_tw_reuse=1         # Linux
net.inet.tcp.maxtcptw=2*N       # BSD

# If using netfilter on Linux:
net.netfilter.nf_conntrack_max=N
echo $((N/8)) > /sys/module/nf_conntrack/parameters/hashsize

The similar option exists for limiting the number of gRPC connections - rpc.grpc-max-open-connections.

Configure State-Sync

State sync rapidly bootstraps a new node by discovering, fetching, and restoring a state machine snapshot from peers instead of fetching and replaying historical blocks. The node will have a truncated block history, starting from the height of the snapshot.

NOTE: Before trying to use state sync, see if the application you are operating a node for supports it.

State sync is only attempted on first start: if the node already has any local state (LastBlockHeight > 0), it is skipped and the node falls back to block sync.

Unlike upstream Tendermint, Tenderdash does not require trust anchors (trust-height, trust-hash, trust-period). Light blocks are verified by checking the quorum threshold signature against the active validator quorum via Dash Core (quorum verify), so there is no need to obtain a trusted block hash out of band. If you are migrating a configuration that still contains trust-height, trust-hash, or trust-period under [statesync], remove them — they are no longer valid options.

Because light block verification goes through Dash Core, a full node must be configured with a Dash Core RPC connection or it will refuse to start: set core-rpc-host (and core-rpc-username/core-rpc-password as needed) in the [priv-validator] section of config.toml. Validator nodes already have this connection configured.

Dash Core is also what authenticates quorum membership and public key shares before validator state received from peers is installed. If genesis configures VotingPowerThreshold, the threshold received during state sync must match the locally stored genesis value; it cannot be learned safely from a remote state-sync provider.

Under the [statesync] section in config.toml you will find the settings that need to be configured in order for your node to use state sync.

Let's break down the settings:

  • enable: Inform the node that you will be using state sync to bootstrap. This only controls consuming snapshots at first start; full and validator nodes serve snapshots and light blocks to peers regardless of this setting (seed nodes only run peer exchange and serve neither).
  • use-p2p: State sync uses light client verification to verify state. This can be done either through the P2P layer or the RPC layer. Set this to true to use the P2P layer. If false (default), the RPC layer will be used.
  • rpc-servers: Comma-separated list of RPC servers used for light client verification when use-p2p = false. In that mode at least two servers are required (more is always helpful). They should be compatible with net.Dial, for example: host.example.com:2125. Ignored when use-p2p = true.
  • discovery-time: Time to spend discovering snapshots before initiating a restore (default: 15s). Must be 0s or at least 5s. With 0s the node gives up as soon as no suitable snapshot is available and falls back to block sync.
  • retries: Number of completed snapshot discovery sweeps before giving up. When retries are exhausted, the node falls back to regular block sync. Set to 0 to retry indefinitely — the node keeps requesting snapshots forever and never falls back to block sync (default: 3). Each sweep asks every connected peer (at most 1,024) in batches of 16, one batch per discovery-time, so the pessimistic case takes longer than discovery-time * retries; see Snapshot discovery resource limits.
  • temp-dir: Temporary directory for snapshot chunks; defaults to the operating system temporary directory (e.g. /tmp). The synchronizer creates a new, randomly named directory within it and removes it when the sync is complete.
  • chunk-request-timeout: The timeout before re-requesting a chunk, possibly from a different peer (default: 15s). Must be at least 5s.
  • fetchers: The number of concurrent chunk and block fetchers to run (default: 4).

Example configuration for a full node using RPC-based light client verification (the [priv-validator] Dash Core connection is required on full nodes regardless of state sync):

[priv-validator]
core-rpc-host = "127.0.0.1:9998"
core-rpc-username = "dashrpc"
core-rpc-password = "changeme"

[statesync]
enable = true
use-p2p = false
rpc-servers = "seed-1.example.com:26657,seed-2.example.com:26657"

Or, using the P2P layer for verification (no RPC servers needed):

[priv-validator]
core-rpc-host = "127.0.0.1:9998"
core-rpc-username = "dashrpc"
core-rpc-password = "changeme"

[statesync]
enable = true
use-p2p = true

Snapshot discovery resource limits

Snapshot advertisements retain the existing 4,000,000-byte network message limit; there is no smaller limit on application metadata. The node retains at most 64 MiB of unique snapshot hash and metadata payload, including the snapshot being restored, and charges at most 40,000,000 bytes of advertised payload to each peer. Shared snapshots count once globally and once for each supplying peer. Removing an association releases that peer's charge immediately, while an active snapshot remains globally charged until restoration cleanup finishes. If the same snapshot is admitted again while a removed active copy is still in use, both owned copies consume the global budget until the old copy is released. The pool owns copies of the payload and releases its charge when it releases the data. These limits bound retained discovery payload, not process RSS, transport receive queues, decoded messages in flight, chunk data, or application state.

The pool also bounds candidates to 1,024 and peer associations to 10,240. Each rejection history holds at most 1,024 entries; an older rejection can be considered again after it leaves that history. Empty indexes are removed. Under payload pressure, advertisements from less represented peers can replace unselected candidates from peers retaining more data. The active snapshot is never evicted or uncharged before restoration releases it. Admission is an opportunity to try a candidate, not a guarantee against malicious peers controlling many identities.

Discovery sends directed requests to batches of at most 16 peers. Each requested peer may return ten advertisements, counting duplicates and rejected replies. Responses without remaining allowance are ignored without penalizing the peer. Late responses remain eligible while that batch's allowance is open; subsequent batches replace the allowance. Newly connected peers can use unallocated request slots, and otherwise wait for a later discovery sweep.

The retries setting counts completed discovery sweeps. A sweep freezes at most 1,024 connected peers, visits them in batches, and cannot be prolonged by later peer arrivals. Successive sweeps rotate through larger connected-peer lists. Already retained candidates are tried before discovery is declared exhausted. Restoration starts from the best snapshot retained so far, which may come from the first batch only; nodes with more than 16 peers can therefore start from a slightly older snapshot than the newest one available on the network. With no usable responses, a sweep takes up to ceil(min(connected_peers, 1024) / 16) * discovery-time, with at least one discovery interval and a minimum interval of five seconds. Snapshot restoration adds its own processing time. Context cancellation interrupts discovery; retries = 0 continues discovery indefinitely as before.

Validators

Validators are responsible for committing new blocks in the blockchain. These validators participate in the consensus protocol by broadcasting votes which contain cryptographic signatures signed by each validator's private key.

The active validator set is a Dash quorum. Each validator has voting power 100; membership and quorum keys determine participation. See the validator-set terminology and proposer selection.

Becoming a Validator

On Dash Platform, becoming a validator requires operating an evonode with 4,000 DASH collateral and registering it with a ProTx transaction. Follow the Dash evonode setup and registration guide for the current operator requirements and setup procedure.

The initial validator set comes from genesis and ABCI InitChain. On a running chain, the application supplies validator-set updates in PrepareProposal and ProcessProposal; changes agreed in block H take effect at H+1. Joining the P2P network or editing a local genesis file does not add a validator to a live chain.

See validator-set and parameter changes and the startup handshake.

Setting up a Validator

When setting up a validator there are countless ways to configure your setup. This guide is aimed at showing one of them, the sentry node design. This design is mainly for DDOS prevention.

Network Layout

ALT Network Layout

The diagram is based on AWS, other cloud providers will have similar solutions to design a solution. Running nodes is not limited to cloud providers, you can run nodes on bare metal systems as well. The architecture will be the same no matter which setup you decide to go with.

The proposed network diagram is similar to the classical backend/frontend separation of services in a corporate environment. The “backend” in this case is the private network of the validator in the data center. The data center network might involve multiple subnets, firewalls and redundancy devices, which is not detailed on this diagram. The important point is that the data center allows direct connectivity to the chosen cloud environment. Amazon AWS has “Direct Connect”, while Google Cloud has “Partner Interconnect”. This is a dedicated connection to the cloud provider (usually directly to your virtual private cloud instance in one of the regions).

All sentry nodes (the “frontend”) connect to the validator using this private connection. The validator does not have a public IP address to provide its services.

Amazon has multiple availability zones within a region. One can install sentry nodes in other regions too. In this case the second, third and further regions need to have a private connection to the validator node. This can be achieved by VPC Peering (“VPC Network Peering” in Google Cloud). In this case, the second, third and further region sentry nodes will be directed to the first region and through the direct connect to the data center, arriving to the validator.

A more persistent solution (not detailed on the diagram) is to have multiple direct connections to different regions from the data center. This way VPC Peering is not mandatory, although still beneficial for the sentry nodes. This overcomes the risk of depending on one region. It is more costly.

Local Configuration

ALT Local Configuration

The validator will only talk to the sentry that are provided, the sentry nodes will communicate to the validator via a secret connection and the rest of the network through a normal connection. The sentry nodes do have the option of communicating with each other as well.

When initializing nodes there are several parameters in the config.toml that may need to be altered.

  • mode: (full | validator | seed) Mode of node (default: 'full'). If you want to run the node as validator, change it to 'validator'.
  • pex: boolean. This turns the peer exchange reactor on or off for a node. When pex=false, only the persistent-peers list is available for connection.
  • persistent-peers: a comma separated list of nodeID@ip:port values that define a list of peers that are expected to be online at all times. This is necessary at first startup because by setting pex=false the node will not be able to join the network.
  • unconditional-peer-ids: comma separated list of nodeID's. These nodes will be connected to no matter the limits of inbound and outbound peers. This is useful for when sentry nodes have full address books.
  • private-peer-ids: comma separated list of nodeID's. These nodes will not be gossiped to the network. This is an important field as you do not want your validator IP gossiped to the network.
  • addr-book-strict: boolean. By default nodes with a routable address will be considered for connection. If this setting is turned off (false), non-routable IP addresses, like addresses in a private network can be added to the address book.

Validator Node Configuration

Config OptionSetting
modevalidator
pexfalse
persistent-peerslist of sentry nodes
private-peer-idsnone
unconditional-peer-idsoptionally sentry node IDs
addr-book-strictfalse

To run the node as validator ensure mode=validator. The validator node should have pex=false so it does not gossip to the entire network. The persistent peers will be your sentry nodes. Private peers can be left empty as the validator is not trying to hide who it is communicating with. Setting unconditional peers is optional for a validator because they will not have a full address books.

Sentry Node Configuration

Config OptionSetting
modefull
pextrue
persistent-peersvalidator node, optionally other sentry nodes
private-peer-idsvalidator node ID
unconditional-peer-idsvalidator node ID, optionally sentry node IDs
addr-book-strictfalse

The sentry nodes should be able to talk to the entire network hence why pex=true. The persistent peers of a sentry node will be the validator, and optionally other sentry nodes. The sentry nodes should make sure that they do not gossip the validator's ip, to do this you must put the validators nodeID as a private peer. The unconditional peer IDs will be the validator ID and optionally other sentry nodes.

Note: Do not forget to secure your node's firewalls when setting them up.

More Information can be found at these links:

Validator keys

Tenderdash validators use BLS consensus keys and Dash quorum information. Protect both the private key and the signing state that prevents conflicting votes. Never run two active signers with the same consensus identity.

A remote signer must support Tenderdash's BLS keys and quorum-aware signing requests. Check compatibility before choosing a key management server or HSM; support for Ed25519 in another consensus system does not establish compatibility. See remote signers and the signing specification.

Committing a Block

Tenderdash commits contain a recovered BLS threshold block signature and threshold vote-extension signatures, rather than individual validator signatures. The configurable voting threshold and the quorum's signature recovery threshold are distinct; the label "+2/3" in consensus code does not always mean a strict two-thirds count.

Use the canonical threshold definitions, threshold recovery procedure and commit handling rules.

Using ABCI-CLI

abci-cli sends requests directly to an ABCI application for development and debugging. It does not run consensus or construct a verified quorum commit. Use it against an isolated test application, not an application serving a live network.

Install

Build the tool from the same Tenderdash checkout as the application:

make install_abci

Inspect the options for your installed version:

abci-cli --help
abci-cli kvstore --help

Start an example application

Start the in-memory key-value application in one terminal:

abci-cli kvstore --address tcp://127.0.0.1:26658

In another terminal, send basic requests:

abci-cli --address tcp://127.0.0.1:26658 echo hello
abci-cli --address tcp://127.0.0.1:26658 info
abci-cli --address tcp://127.0.0.1:26658 check_tx 0x6b65793d76616c7565

The CLI's default address is tcp://0.0.0.0:26658; these examples explicitly select loopback. The default transport is socket. CheckTx validates a transaction; it does not persist application state.

For gRPC, start a server with --abci grpc and use a bare HOST:PORT client address. The server accepts a tcp:// address, but the gRPC client does not resolve that form correctly and waits for a connection:

abci-cli kvstore --abci grpc --address tcp://127.0.0.1:26659

In another terminal:

abci-cli --abci grpc --address 127.0.0.1:26659 echo hello

Transaction and query arguments accept 0x-prefixed hexadecimal bytes or a string with literal double quotes. For example, these requests encode the same bytes (key=value):

abci-cli --address tcp://127.0.0.1:26658 check_tx 0x6b65793d76616c7565
abci-cli --address tcp://127.0.0.1:26658 check_tx '"key=value"'

Proposal and finalization requests

The implemented request commands include echo, info, query, check_tx, prepare_proposal, process_proposal and finalize_block. Their block-related argument forms are:

prepare_proposal HEIGHT TX [TX ...]
process_proposal HEIGHT [TX ...]
finalize_block HEIGHT APP_HASH_HEX [TX ...]

The height is decimal. Each TX uses the byte encoding described above. APP_HASH_HEX is the application hash as hexadecimal without a 0x prefix; it must match the state prepared and processed for that block. These commands send simplified test requests and do not fill every field required by an arbitrary application's consensus lifecycle. In particular, they cannot set the round, timestamp, proposer, quorum or commit metadata. prepare_proposal fixes max_tx_bytes at 65536, and process_proposal fixes the application protocol version to the example application's version. Consult the command implementation when constructing application-specific tests. The standalone prepare_proposal command requires a transaction; an empty proposal can be sent through batch.

The CLI does not expose request commands for InitChain, ExtendVote, VerifyVoteExtension or the four snapshot methods. There is no flush command; Flush is available through the underlying client interface.

Responses are abbreviated: info prints only data, check_tx prints only code and data, and proposal commands omit transaction results, parameter and validator updates, and other response fields. --verbose does not restore those fields. CLI output therefore cannot validate the full application protocol.

There is no ABCI Commit method or implemented commit command. The application persists state during FinalizeBlock. For a complete working node/application example, use the external Go application guide. The canonical ABCI call sequence defines the lifecycle a real node drives. Some installed help and completion text still mentions commit; that text is stale.

Reuse a connection

console accepts interactive request commands; batch reads them from standard input. Both reuse one connection. For example:

abci-cli --address tcp://127.0.0.1:26658 --verbose batch <<'EOF_COMMANDS'
echo hello
info
check_tx 0x6b65793d76616c7565
EOF_COMMANDS

The console and batch parser splits input on spaces without shell quoting. Encode transaction or query bytes containing spaces as hexadecimal in these modes. Query flags such as --height, --path and --prove apply to standalone query commands; flags inside console or batch input are not applied.

Use an ABCI implementation compatible with your Tenderdash release. See the current application interface and protobuf definitions.

Application Architecture Guide

A Tenderdash deployment separates the consensus node from the application that defines transaction rules and owns application state. A client submits transactions or queries through an RPC or application API.

flowchart LR
    Client[Client application] -->|Transactions and queries| RPC[Tenderdash RPC]
    RPC --> Node[Tenderdash node]
    Node <-->|P2P consensus| Peers[Peer nodes]
    Node <-->|ABCI| App[Application]
    App --> State[Persistent application state]

Tenderdash orders transactions and agrees on blocks. The ABCI application validates transactions with CheckTx, prepares and validates proposals with PrepareProposal and ProcessProposal, and finalizes decided blocks with FinalizeBlock. Queries use Query; the application defines the query format and any proofs it returns.

The application must produce deterministic results for the same consensus input. Keep replicated state changes inside the ABCI lifecycle. External services may provide read APIs, but must not independently mutate that state.

An application written in Go can run in the same process as Tenderdash using a local ABCI client. Other deployments can run the application as a separate process through an ABCI socket or gRPC connection. Protect these connections: ABCI is a trusted interface, not a public client endpoint. Restrict network access and provide authentication and encryption when crossing a network boundary.

Client applications can broadcast transactions using the /broadcast_tx_* RPC methods and query through /abci_query. A successful broadcast does not by itself prove that a transaction was committed successfully. Inspect committed results and verify application proofs according to the application's security model.

See the RPC guide, ABCI specification, Go tutorials and production operations guide.

Getting Started with Application Development

Tenderdash handles P2P communication, transaction gossip and consensus. Your ABCI application defines transaction validation, execution and application state. Start with the application architecture guide and the ABCI overview.

Run an example

Install Tenderdash and initialize a local validator in a new directory:

tenderdash init validator --home ./example-node
tenderdash start --home ./example-node --proxy-app=kvstore

The built-in kvstore is an in-memory demonstration application. Do not use it as persistent production storage.

Submit and query a key-value transaction:

curl -s 'localhost:26657/broadcast_tx_commit?tx="example=rocks"'
curl -s 'localhost:26657/abci_query?data="example"'

Build your application

Applications in other languages must implement the protobuf messages and ABCI lifecycle for the Tenderdash release they target. Check compatibility before using a third-party ABCI library.

Use the application interface, protobuf definitions and ABCI specification as the contract. Test transaction rejection, proposal processing, block finalization, deterministic state hashes and restart recovery before connecting to a live network.

Indexing Transactions

Tenderdash can index committed transaction results and block events for RPC searches and subscriptions. Transaction events come from ExecTxResult.Events in the proposal execution results. Block events come from ResponseProcessProposal.Events. They are published after block finalization; ResponseFinalizeBlock does not contain an events field.

Each event has a type and string-valued attributes. The search key combines the event type and attribute key with a dot. An event of type transfer with an attribute sender=Bob can therefore be queried as transfer.sender='Bob'. Set an attribute's Index flag to true to include it in event indexes.

For the execution lifecycle, see commit and apply. The event publisher shows how committed proposal results become transaction and block events.

Configuration

Operators can configure indexing via the [tx-index] section. The indexer field takes a series of supported indexers. If null is included, indexing will be turned off. All configured indexer names must still be supported.

[tx-index]

# The backend database list to back the indexer.
# If list contains null, meaning no indexer service will be used.
#
# The application will set which txs to index. In some cases a node operator will be able
# to decide which txs to index based on configuration set in the application.
#
# Options:
#   1) "null" (default) - no indexer services.
#   2) "kv" - an indexer backed by key-value storage (see DBBackend).
#     - When "kv" is chosen "tx.height" and "tx.hash" will always be indexed.
# indexer = []

Supported Indexers

KV

The kv indexer type is an embedded key-value store supported by the main underlying Tenderdash database. Using the kv indexer type allows you to query for block and transaction events directly against Tenderdash's RPC. However, the query syntax is limited to the supported event-query operators.

Removed PostgreSQL indexer

The psql event indexer is no longer supported. Selecting it, including alongside kv or null, returns a configuration error. Choose indexer = ["kv"] for local indexing and RPC searches, or ["null"] to disable indexing. The obsolete psql-conn key is ignored, so existing Dashmate configurations with an empty value remain compatible.

Existing PostgreSQL data is not deleted or migrated, and SQL consumers no longer receive new events. To populate KV indexes, explicitly run tenderdash reindex-event against the retained local blocks and results after selecting kv. Pruned history cannot be reconstructed by this command.

Default Indexes

The Tenderdash tx and block event indexer indexes a few select reserved events by default.

Transactions

The following indexes are indexed by default:

  • tx.height
  • tx.hash

Blocks

The following indexes are indexed by default:

  • block.height

Adding Events

Return transaction events in the corresponding ExecTxResult in ResponsePrepareProposal.TxResults / ResponseProcessProposal.TxResults. For example, an individual transaction result can contain:

result := &abci.ExecTxResult{
    Code: 0,
    Events: []abci.Event{
        {
            Type: "transfer",
            Attributes: []abci.EventAttribute{
                {Key: "sender", Value: "Bob", Index: true},
                {Key: "recipient", Value: "Alice", Index: true},
                {Key: "note", Value: "payment", Index: false},
            },
        },
    },
}

Here abci refers to github.com/dashpay/tenderdash/abci/types; append result to the response's transaction results in the same order as the executed transactions. The kv indexer indexes transfer.sender and transfer.recipient; transfer.note is not indexed. An empty event type or attribute key is not indexed. Avoid reserved keys such as tx.hash and tx.height.

Querying Transactions Events

You can query for a paginated set of transaction by their events by calling the /tx_search RPC endpoint:

curl "localhost:26657/tx_search?query=\"transfer.sender='Bob'\"&prove=true"

Check out API docs for more information on query syntax and other options.

Subscribing to Transactions

Clients can subscribe to transactions with the given tags via WebSocket by providing a query to /subscribe RPC endpoint.

{
  "jsonrpc": "2.0",
  "method": "subscribe",
  "id": "0",
  "params": {
    "query": "transfer.sender='Bob'"
  }
}

Check out API docs for more information on query syntax and other options.

Querying Blocks Events

You can query for a paginated set of blocks by their events by calling the /block_search RPC endpoint:

curl "localhost:26657/block_search?query=\"block.height > 10\""

Check out API docs for more information on query syntax and other options.

Application development

Creating a built-in application in Go

An application written in Go can run in the same process as Tenderdash. A local ABCI client calls the application directly, avoiding a network connection between the node and application. The application still follows the same consensus and persistence rules as an external application.

Use the current module and interface

Build against github.com/dashpay/tenderdash at the release used by your network. Tenderdash includes a native BLS dependency; follow the build instructions when preparing your toolchain.

Implement the ABCI application interface. Use the key-value application as a reference for the full lifecycle, including PrepareProposal, ProcessProposal, FinalizeBlock, queries and snapshots.

Construct a local client and node

The relevant imports are:

import (
    abciclient "github.com/dashpay/tenderdash/abci/client"
    "github.com/dashpay/tenderdash/config"
    "github.com/dashpay/tenderdash/node"
)

Given your application app, a configured logger logger, and the application context ctx, the integration is:

client := abciclient.NewLocalClient(logger, app)
service, err := node.New(ctx, conf, logger, client, nil)
if err != nil {
    return err
}
if err := service.Start(ctx); err != nil {
    return err
}
defer func() {
    service.Stop()
    service.Wait()
}()

Stop the service before waiting for it. Keep the process running until cancellation or a shutdown signal, then cancel the context and shut down the node before closing application storage.

The conf value is a validated *config.Config loaded for the intended home directory. The default home is $HOME/.tenderdash; set it explicitly for tests. Initialize the node's configuration and genesis with tenderdash init validator before starting it. Passing nil as the final node.New argument loads genesis from the configured file.

The node constructor and local client are the source of truth for the API signatures in your release.

Concurrency and recovery

The application can receive different classes of ABCI requests while the node is running. Protect shared state, keep queries consistent with committed state, and separate mempool checks and tentative proposal execution from finalization. Persistence must record the height and application hash with the state they identify so that restarting the process can recover consistently.

Test the application independently through the ABCI interface, then exercise the embedded node's startup, shutdown and restart paths. The ABCI specification describes the required behavior.

Creating an external application in Go

An external ABCI application runs separately from the Tenderdash node. The node connects to its socket or gRPC server and sends requests using the ABCI application interface. Use the application interface and protobuf definitions from your Tenderdash release.

Start with the example application

Build Tenderdash and the example ABCI CLI from the same checkout:

make install_abci

Start the supplied key-value application in one terminal:

abci-cli kvstore --address tcp://127.0.0.1:26658

Initialize a local validator and connect it to that application in another terminal. Use a new home directory for this experiment:

tenderdash init validator --home ./example-node
tenderdash start --home ./example-node --proxy-app=tcp://127.0.0.1:26658

Keep the ABCI server bound to loopback for local development. A publicly exposed ABCI endpoint lets callers bypass the node's consensus lifecycle.

Implement your application

Study the key-value application source and its tests. It provides working examples of transaction checking, proposal handling, block finalization, queries and snapshots.

Your Go implementation uses these package paths:

import (
    "github.com/dashpay/tenderdash/abci/server"
    abci "github.com/dashpay/tenderdash/abci/types"
)

The abci.Application interface includes:

  • Info and InitChain for startup and chain initialization.
  • CheckTx for mempool admission; this must not commit application state.
  • PrepareProposal and ProcessProposal for block construction and validation.
  • ExtendVote and VerifyVoteExtension for application vote extensions.
  • FinalizeBlock for applying a decided block and persisting state.
  • Query for application queries, plus the snapshot methods for state sync.

abci.BaseApplication supplies default implementations for scaffolding. Those defaults are not a complete persistent application. Implement the state, height, application hash, validation and recovery behavior your application requires.

Create the server with server.NewServer(logger, address, transport, app), then start the returned service with a context. Stop and wait for the service during shutdown. See the server package and ABCI CLI implementation for lifecycle handling.

Submit and query transactions

With the example key-value application running, submit a transaction:

curl -s 'localhost:26657/broadcast_tx_commit?tx="example=rocks"'

Query the committed value:

curl -s 'localhost:26657/abci_query?data="example"'

Inspect the transaction result as well as transport-level success. Your own application defines the transaction encoding and query format.

Validate application behavior

Test invalid transactions, proposal rejection, repeated requests, crash recovery and snapshot restore. All correct replicas must derive the same application hash from the same committed input. Keep tentative proposal execution isolated from committed state, because proposals can be rejected or superseded.

For the complete lifecycle, read the ABCI specification. For embedding in the same process, follow the built-in application guide.

Tutorials

Overview

This section dives into the internals of Tenderdash.

For full specifications refer to the protocol specification.

Block Structure

The Tenderdash consensus engine records all agreements by a supermajority of nodes into a blockchain, which is replicated among all nodes. This blockchain is accessible via various RPC endpoints, mainly /block?height= to get the full block, as well as /blockchain?minHeight=_&maxHeight=_ to get a list of headers. But what exactly is stored in these blocks?

The specification contains a detailed description of each component - that's the best place to get started.

To dig deeper, check out the types package documentation.

See also:

Block Sync

Formerly known as Fast Sync

In a proof of work blockchain, syncing with the chain is the same process as staying up-to-date with the consensus: download blocks, and look for the one with the most total work. In proof-of-stake, the consensus process is more complex, as it involves rounds of communication between the nodes to determine what block should be committed next. Using this process to sync up with the blockchain from scratch can take a very long time. It's much faster to just download blocks and check the merkle tree of validators than to run the real-time consensus gossip protocol.

Using Block Sync

To support faster syncing, Tenderdash offers a blocksync mode, which is enabled by default, and can be toggled in the config.toml or via --blocksync.enable=false.

In this mode, the Tenderdash daemon will sync hundreds of times faster than if it used the real-time consensus process. Once caught up, the daemon will switch out of Block Sync and into the normal consensus mode. After running for some time, the node is considered caught up if it has at least one peer and it's height is at least as high as the max reported peer height. See the IsCaughtUp method.

Note: There are multiple versions of Block Sync. Please use v0 as the other versions are no longer supported. If you would like to use a different version you can do so by changing the version in the config.toml:

#######################################################
###       Block Sync Configuration Connections       ###
#######################################################
[blocksync]

# If this node is many blocks behind the tip of the chain, BlockSync
# allows them to catchup quickly by downloading blocks in parallel
# and verifying their commits
enable = true

# Block Sync version to use:
#   1) "v0" (default) - the standard Block Sync implementation
#   2) "v2" - DEPRECATED, please use v0
version = "v0"

If we're lagging sufficiently, we should go back to block syncing, but this is an open issue.

The Block Sync event

When the tenderdash blockchain core launches, it might switch to the block-sync mode to catch up the states to the current network best height. the core will emits a fast-sync event to expose the current status and the sync height. Once it catched the network best height, it will switches to the state sync mechanism and then emit another event for exposing the fast-sync complete status and the state height.

The user can query the events by subscribing EventQueryBlockSyncStatus Please check types for the details.

Implementation

To read more on the implamentation please see the reactor doc and the implementation doc

Implementation

Blocksync Reactor

  • coordinates the pool for syncing
  • coordinates the store for persistence
  • coordinates the playing of blocks towards the app using a sm.BlockExecutor
  • handles switching between fastsync and consensus
  • it is a p2p.BaseReactor
  • starts the pool.Start() and its poolRoutine()
  • registers all the concrete types and interfaces for serialisation

poolRoutine

  • listens to these channels:
    • pool requests blocks from a specific peer by posting to requestsCh, block reactor then sends a &bcBlockRequestMessage for a specific height
    • pool signals timeout of a specific peer by posting to timeoutsCh
    • switchToConsensusTicker to periodically try and switch to consensus
    • trySyncTicker to periodically check if we have fallen behind and then catch-up sync
      • if there aren't any new blocks available on the pool it skips syncing
  • tries to sync the app by taking downloaded blocks from the pool, gives them to the app and stores them on disk
  • implements Receive which is called by the switch/peer
    • calls AddBlock on the pool when it receives a new block from a peer

Block Pool

  • responsible for downloading blocks from peers
  • makeRequestersRoutine()
    • removes timeout peers
    • starts new requesters by calling makeNextRequester()
  • requestRoutine():
    • picks a peer and sends the request, then blocks until:
      • pool is stopped by listening to pool.Quit
      • requester is stopped by listening to Quit
      • request is redone
      • we receive a block
        • gotBlockCh is strange

Go Routines in Blocksync Reactor

Go Routines Diagram

Reactor

The Blocksync Reactor's high level responsibility is to enable peers who are far behind the current state of the consensus to quickly catch up by downloading many blocks in parallel, verifying their commits, and executing them against the ABCI application.

Tenderdash full nodes run the Blocksync Reactor as a service to provide blocks to new nodes. New nodes run the Blocksync Reactor in "fast_sync" mode, where they actively make requests for more blocks until they sync up. Once caught up, "fast_sync" mode is disabled and the node switches to using (and turns on) the Consensus Reactor.

Architecture and algorithm

The pseudocode below describes the historical requester-based implementation. The current synchronizer in internal/blocksync uses a worker pool. Five consecutive failed requests remove a peer; a successful response resets that failure count. The client request timeout remains 60 seconds.

Scheduling also accounts for useful service while requests are outstanding:

  • Active waiting begins after the client admits a request and ends at its first correlated response, timeout, or cancellation. Overlapping requests consume time once, not once per request. Worker queues before admission, completed result buffering, block application, and idle intervals do not consume time.
  • A response earns one second plus its encoded bytes divided by 128 KiB/s only after conversion, basic validation, and the requested-height check. Commit verification and application retain their existing later checks. Saved credit is limited to ten seconds, so an old burst cannot hide sustained poor service.
  • More than ten seconds of service debt reduces future assignments. Existing requests drain; after a thirty-second cooldown, one recovery request is allowed. Due recovery requests receive a turn even when other peers are healthy. Two consecutive recovery responses meeting the size-adjusted allowance restore ordinary pipelining. Slow or failed probes restart the cooldown.

These are soft scheduling allowances, not measured wire bandwidth or a guaranteed minimum connection speed. They let prompt tiny blocks earn progress through their count and larger blocks through their bytes. There is no partial-byte telemetry: a large unanswered response can provisionally reduce preference, and its validated bytes can correct that estimate on completion. Local transport and router queues between client admission and resolution remain included in the observation.

A scheduling penalty never disconnects a peer or makes its advertised block range unavailable. A sole source remains eligible for bounded recovery attempts, which prevents a false no-source handoff to consensus. Status refreshes preserve service history; a removed and re-added peer starts fresh. The twenty pending requests per peer, sixty-four outstanding heights, completed-backlog byte budget, validation, and hard failure policy are unchanged. The 60-second request deadline can still reject a large response on a slow connection even if its eventual service rate would meet the scheduling allowance.

Block sync hands over to consensus when one of these holds:

  • the node reaches the highest height any peer reports;
  • no peer reports the next height and no block has been applied for 60 seconds;
  • the node's last applied block has been recent, at most five minutes old by the local clock, for two minutes without a break and without getting more than 30 seconds younger meanwhile;
  • no block has been applied for 10 minutes, and either that wait began on a block less than a day old, or the node is within 10 blocks of every peer that reports the next height.

The heights peers report are not verified, so the last two rules rest on the time of the last applied block instead, which the validators commit to: a node far behind the network applies old blocks. Five minutes covers the three-minute empty-block interval Dash Platform runs with and a margin for extra rounds and clock skew. A last block that keeps getting younger means the node is still closing the gap, so the two minutes start over each time it has gained 30 seconds: a node applying blocks at least a quarter faster than the chain commits them goes on until its last block is within 30 seconds of the chain's. A block time further ahead of the local clock than the synchrony precision does not count as recent. When a node with a recent last block hands over, the consensus catch-up target is the highest height reported within 10 blocks of it. Only a node whose last block was more than a day old when it stopped receiving blocks keeps waiting for as long as a peer reports the next height and a height more than 10 blocks above the node, and resumes as soon as a peer serves again.

Blocks are applied in order, so the next block is the one that matters. When no block has been applied for 60 seconds and the next one has not arrived, it is requested from every peer that reports it, once each, starting a second after the peer first reports it. Sixty seconds is also how long a request may take, so a block that arrives within its request deadline is fetched once, however slow the link.

What operators can expect:

  • A node that has caught up hands over within two to four minutes, whatever heights its peers report, provided a connected peer serves each new block when asked. Peers that delay blocks can stretch this to about 20 minutes, and a single block by about a minute.
  • On a chain that has stopped producing blocks, a restarted node hands over after 10 minutes without a block. If the chain had been stopped for more than a day when the node started waiting, the node keeps waiting while a peer reports the next height and a height more than 10 blocks above the node.
  • A node may hand over behind the network by up to five minutes of blocks: one or two blocks on a chain committing every three minutes, a few hundred on one committing every second. Consensus catch-up closes the rest. On a chain that keeps committing a block every second or two this can take long, and it happens only when the node received blocks less than a quarter faster than the chain produced them.
  • A node less than a day behind the network whose peers all fail to deliver the next block for 10 minutes hands over where it is. Restart it once its peers serve blocks again if consensus catch-up does not close the gap.
  • With only a few peers serving blocks, a node restarting far behind a chain that commits a block every second or two may need much longer to reach the tip. More peers that serve blocks help.
  • A node whose clock runs minutes ahead of the validators', or behind them by more than the synchrony precision, does not see its last block as recent and leaves block sync only by the other rules. Keep the clock synchronised.
  • After a handover below a height a peer reported, the node may hold back its own proposals for up to 10 minutes while consensus catches up. With a recent last block only heights reported within 10 blocks of the node count.
  • A node that stays in block sync while its blocks are recent, or hands over far behind, is worth reporting with its logs: a handover below the highest height a peer reports is logged at warning or error level with its reason.

The historical Blocksync reactor is organised as a set of concurrent tasks:

  • Receive routine of Blocksync Reactor
  • Task for creating Requesters
  • Set of Requesters tasks and - Controller task.

Blocksync Reactor Architecture Diagram

Data structures

These are the core data structures necessarily to provide the Blocksync Reactor logic.

Requester data structure is used to track assignment of request for block at position height to a peer with id equals to peerID.

type Requester {
  mtx          Mutex
  block        Block
  height       int64
  peerID       p2p.ID
  redoChannel  chan p2p.ID //redo may send multi-time; peerId is used to identify repeat
}

Pool is a core data structure that stores last executed block (height), assignment of requests to peers (requesters), current height for each peer and number of pending requests for each peer (peers), maximum peer height, etc.

type Pool {
  mtx                Mutex
  requesters         map[int64]*Requester
  height             int64
  peers              map[p2p.ID]*Peer
  maxPeerHeight      int64
  numPending         int32
  store              BlockStore
  requestsChannel    chan<- BlockRequest
  errorsChannel      chan<- peerError
}

Peer data structure stores for each peer current height and number of pending requests sent to the peer (numPending), etc.

type Peer struct {
  id           p2p.ID
  height       int64
  numPending   int32
  timeout      *time.Timer
  didTimeout   bool
}

BlockRequest is internal data structure used to denote current mapping of request for a block at some height to a peer (PeerID).

type BlockRequest {
  Height int64
  PeerID p2p.ID
}

Receive routine of Blocksync Reactor

It is executed upon message reception on the BlocksyncChannel inside p2p receive routine. There is a separate p2p receive routine (and therefore receive routine of the Blocksync Reactor) executed for each peer. Note that try to send will not block (returns immediately) if outgoing buffer is full.

handleMsg(pool, m):
    upon receiving bcBlockRequestMessage m from peer p:
      block = load block for height m.Height from pool.store
      if block != nil then
        try to send BlockResponseMessage(block) to p
      else
        try to send bcNoBlockResponseMessage(m.Height) to p

    upon receiving bcBlockResponseMessage m from peer p:
      pool.mtx.Lock()
      requester = pool.requesters[m.Height]
      if requester == nil then
        error("peer sent us a block we didn't expect")
        continue

      if requester.block == nil and requester.peerID == p then
        requester.block = m
        pool.numPending -= 1  // atomic decrement
        peer = pool.peers[p]
        if peer != nil then
          peer.numPending--
          if peer.numPending == 0 then
            peer.timeout.Stop()
            // NOTE: we don't send Quit signal to the corresponding requester task!
        else
          trigger peer timeout to expire after peerTimeout
      pool.mtx.Unlock()


    upon receiving bcStatusRequestMessage m from peer p:
      try to send bcStatusResponseMessage(pool.store.Height)

    upon receiving bcStatusResponseMessage m from peer p:
      pool.mtx.Lock()
      peer = pool.peers[p]
      if peer != nil then
        peer.height = m.height
      else
        peer = create new Peer data structure with id = p and height = m.Height
        pool.peers[p] = peer

      if m.Height > pool.maxPeerHeight then
        pool.maxPeerHeight = m.Height
      pool.mtx.Unlock()

onTimeout(p):
  send error message to pool error channel
  peer = pool.peers[p]
  peer.didTimeout = true

Requester tasks

Requester task is responsible for fetching a single block at position height.

fetchBlock(height, pool):
  while true do {
    peerID = nil
    block = nil
    peer = pickAvailablePeer(height)
    peerID = peer.id

    enqueue BlockRequest(height, peerID) to pool.requestsChannel
    redo = false
    while !redo do
      select {
        upon receiving Quit message do
          return
        upon receiving redo message with id on redoChannel do
          if peerID == id {
            mtx.Lock()
            pool.numPending++
            redo = true
            mtx.UnLock()
          }
      }
    }

pickAvailablePeer(height):
  selectedPeer = nil
  while selectedPeer = nil do
    pool.mtx.Lock()
    for each peer in pool.peers do
      if !peer.didTimeout and peer.numPending < maxPendingRequestsPerPeer and peer.height >= height then
        peer.numPending++
        selectedPeer = peer
        break
    pool.mtx.Unlock()

    if selectedPeer = nil then
      sleep requestIntervalMS

  return selectedPeer

sleep for requestIntervalMS

Task for creating Requesters

This task is responsible for continuously creating and starting Requester tasks.

createRequesters(pool):
  while true do
    if !pool.isRunning then break
    if pool.numPending < maxPendingRequests or size(pool.requesters) < maxTotalRequesters then
      pool.mtx.Lock()
      nextHeight = pool.height + size(pool.requesters)
      requester = create new requester for height nextHeight
      pool.requesters[nextHeight] = requester
      pool.numPending += 1 // atomic increment
      start requester task
      pool.mtx.Unlock()
    else
      sleep requestIntervalMS
      pool.mtx.Lock()
      for each peer in pool.peers do
        if !peer.didTimeout && peer.numPending > 0 && peer.curRate < minRecvRate then
          send error on pool error channel
          peer.didTimeout = true
        if peer.didTimeout then
          for each requester in pool.requesters do
            if requester.getPeerID() == peer then
              enqueue msg on requestor's redoChannel
          delete(pool.peers, peerID)
      pool.mtx.Unlock()

Main blocksync reactor controller task

main(pool):
  create trySyncTicker with interval trySyncIntervalMS
  create statusUpdateTicker with interval statusUpdateIntervalSeconds
  create switchToConsensusTicker with interval switchToConsensusIntervalSeconds

  while true do
    select {
   upon receiving BlockRequest(Height, Peer) on pool.requestsChannel:
     try to send bcBlockRequestMessage(Height) to Peer

   upon receiving error(peer) on errorsChannel:
     stop peer for error

   upon receiving message on statusUpdateTickerChannel:
     broadcast bcStatusRequestMessage(bcR.store.Height) // message sent in a separate routine

   upon receiving message on switchToConsensusTickerChannel:
     pool.mtx.Lock()
     receivedBlockOrTimedOut = pool.height > 0 || (time.Now() - pool.startTime) > 5 Seconds
     ourChainIsLongestAmongPeers = pool.maxPeerHeight == 0 || pool.height >= pool.maxPeerHeight
     haveSomePeers = size of pool.peers > 0
     pool.mtx.Unlock()
     if haveSomePeers && receivedBlockOrTimedOut && ourChainIsLongestAmongPeers then
       switch to consensus mode

          upon receiving message on trySyncTickerChannel:
            for i = 0; i < 10; i++ do
              pool.mtx.Lock()
              firstBlock = pool.requesters[pool.height].block
              secondBlock = pool.requesters[pool.height].block
              if firstBlock == nil or secondBlock == nil then continue
              pool.mtx.Unlock()
              verify firstBlock using LastCommit from secondBlock
              if verification failed
                pool.mtx.Lock()
                peerID = pool.requesters[pool.height].peerID
                redoRequestsForPeer(peerId)
                delete(pool.peers, peerID)
                stop peer peerID for error
                pool.mtx.Unlock()
              else
                delete(pool.requesters, pool.height)
                save firstBlock to store
                pool.height++
                execute firstBlock
    }

redoRequestsForPeer(pool, peerId):
  for each requester in pool.requesters do
    if requester.getPeerID() == peerID
     enqueue msg on redoChannel for requester

Channels

Defines maxMsgSize for the maximum size of incoming messages, SendQueueCapacity and RecvBufferCapacity for maximum sending and receiving buffers respectively. These are supposed to prevent amplification attacks by setting up the upper limit on how much data we can receive & send to a peer.

Sending incorrectly encoded data will result in stopping the peer.

Configuration

This file has moved to the nodes section.

Consensus

Tenderdash consensus is a distributed protocol executed by validator processes to agree on the next block to be added to the blockchain. It proceeds in rounds; in each round a proposer suggests a block with a Proposal, and validators vote on it in two phases (Prevote, Precommit). When precommits for a block reach the quorum threshold, their BLS signature shares are recovered into a single threshold signature that forms the block's Commit.

The protocol is specified, from the implementation, in the consensus specification:

The wire format of consensus messages is in Consensus messages; the timestamp rules in force are in Proposal timestamps and time rules (Proposer-Based Timestamps is historical design material). Operational guidance for timestamp problems is in the PBTS runbook.

Evidence

Evidence is used to identify validators who have or are acting malicious. There are multiple types of evidence, to read more on the evidence types please see Evidence Types.

The evidence reactor works similar to the mempool reactor. When evidence is observed, it is sent to all the peers in a repetitive manner. This ensures evidence is sent to as many people as possible to avoid sensoring. After evidence is received by peers and committed in a block it is pruned from the evidence module.

Sending incorrectly encoded data or data exceeding maxMsgSize will result in stopping the peer.

Disabling evidence

Consensus parameter evidence.max_bytes = 0 disables evidence admission, conflict collection and gossip. broadcast_evidence returns an explicit disabled error. Proposals contain no evidence, and the consensus byte limit rejects blocks containing evidence.

Applications can enable or disable evidence on an existing chain through PrepareProposal and ProcessProposal consensus parameter updates. An update in block H takes effect for H+1, only after H commits. A block that disables evidence can still contain evidence under its current positive limit. InitChain may also override the genesis value for the first block.

Nodes keep the evidence database, reactor and advertised P2P channel available throughout their lifetime, so activation works with existing peer connections. While disabled, local admission, conflict collection and periodic gossip skip verification and database access; peer sync timers perform only a cheap mode check. Consensus block validation and recording committed evidence remain active, including evidence in the block that disables admission. Messages whose sends began before disabling may still arrive; disabled receivers discard them.

Pending evidence is retained across disabled periods and restarts, then pruned against the current expiration limits before reactivation. Committed-evidence markers remain available to prevent submitting the same evidence again. Conflicts observed while disabled are not buffered. Restart uses persisted consensus state; state sync installs the restored state before block sync begins, without requiring its evidence mode to match genesis. Existing state-provider verification and trust requirements are unchanged. There is no local configuration override.

How to read logs

This file has moved to the node section.

Light Client

Unsupported. The Tenderdash light client is currently unsupported. The implementation details below are provided for reference.

A light client verifies block headers and commits without running a full application replica. Application state can then be verified through proofs bound to the verified chain state, provided the application supplies compatible proofs.

Tenderdash's light package retrieves light blocks from providers, validates their structure, checks the validator-set binding and verifies commit signatures. Dash Core verification uses quorum information to verify the threshold block signature.

The client can cross-check a primary provider with witnesses. Configure independent providers and the correct Dash Core network; provider availability and correct quorum verification remain part of the security model. A successful RPC response without a verified proof does not establish application-state integrity.

For the exact verification path, see the client implementation. For running the proxy, see Configure a Light Client.

Mempool

The mempool is a in memory pool of potentially valid transactions, both to broadcast to other nodes, as well as to provide to the consensus reactor when it is selected as the block proposer.

There are two sides to the mempool state:

  • External: get, check, and broadcast new transactions
  • Internal: return valid transaction, update list after block commit

External functionality

External functionality is exposed via network interfaces to potentially untrusted actors.

  • CheckTx - triggered via RPC or P2P
  • Broadcast - gossip messages after a successful check

Internal functionality

Internal functionality is exposed via method calls to other code compiled into the tenderdash binary.

  • ReapMaxBytesMaxGas - get txs to propose in the next block. Guarantees that the size of the txs is less than MaxBytes, and gas is less than MaxGas
  • Update - remove tx that were included in last block
  • ABCI.CheckTx - call ABCI app to validate the tx

What does it provide the consensus reactor? What guarantees does it need from the ABCI app? (talk about interleaving processes in concurrency)

Optimizations

The implementation within this library also implements a tx cache. This is so that signatures don't have to be reverified if the tx has already been seen before. However, we only store valid txs in the cache, not invalid ones. This is because invalid txs could become good later. Txs that are included in a block aren't removed from the cache, as they still may be getting received over the p2p network. These txs are stored in the cache by their hash, to mitigate memory concerns.

Applications should implement replay protection, read Replay Protection for more information.

Configuration

The mempool has various configurable paramet

Sending incorrectly encoded data or data exceeding maxMsgSize will result in stopping the peer.

maxMsgSize equals MaxBatchBytes (10MB) + 4 (proto overhead). MaxBatchBytes is a mempool config parameter -> defined locally. The reactor sends transactions to the connected peers in batches. The maximum size of one batch is MaxBatchBytes.

The mempool will not send a tx back to any peer which it received it from.

The reactor assigns an uint16 number for each peer and maintains a map from p2p.ID to uint16. Each mempool transaction carries a list of all the senders ([]uint16). The list is updated every time mempool receives a transaction it is already seen. uint16 assumes that a node will never have over 65535 active peers (0 is reserved for unknown source - e.g. RPC).

Configuration

Here we describe configuration options around mempool. For the purposes of this document, they are described in a toml file, but some of them can also be passed in as environmental variables.

Config:

[mempool]

# Set true to broadcast transactions in the mempool to other nodes
broadcast = true

# Maximum number of transactions in the mempool
size = 5000

# Limit the total size of all txs in the mempool.
# This only accounts for raw transactions (e.g. given 1MB transactions and
# max-txs-bytes=5MB, mempool will only accept 5 transactions).
max-txs-bytes = 1073741824

# Size of the cache (used to filter transactions we saw earlier) in transactions
cache-size = 10000

# Do not remove invalid transactions from the cache (default: false)
# Set to true if it's not possible for any invalid transaction to become valid
# again in the future.
# When true, resubmitting a rejected transaction over RPC returns the CheckTx
# result that rejected it, as long as it stays in the cache.
keep-invalid-txs-in-cache = false

# Maximum size of a single transaction.
# NOTE: the max size of a tx transmitted over the network is {max-tx-bytes}.
max-tx-bytes = 1048576

# Maximum size of a batch of transactions to send to a peer
# Including space needed by encoding (one varint per transaction).
# XXX: Unused due to https://github.com/tendermint/tendermint/issues/5796
max-batch-bytes = 0

Broadcast

Determines whether this node gossips any valid transactions that arrive in mempool. Default is to gossip anything that passes checktx. If this is disabled, transactions are not gossiped, but instead stored locally and added to the next block this node is the proposer.

WalDir

This defines the directory where mempool writes the write-ahead logs. These files can be used to reload unbroadcasted transactions if the node crashes.

If the directory passed in is an absolute path, the wal file is created there. If the directory is a relative path, the path is appended to home directory of the tenderdash process to generate an absolute path to the wal directory (default $HOME/.tenderdash or set via TM_HOME or --home)

Size

Size defines the total amount of transactions stored in the mempool. Default is 5_000 but can be adjusted to any number you would like. The higher the size the more strain on the node.

Max Transactions Bytes

Max transactions bytes defines the total size of all the transactions in the mempool. Default is 1 GB.

Cache size

Cache size determines the size of the cache holding transactions we have already seen. The cache exists to avoid running checktx each time we receive a transaction.

Keep Invalid Transactions In Cache

Keep invalid transactions in cache determines wether a transaction in the cache, which is invalid, should be evicted. An invalid transaction here may mean that the transaction may rely on a different tx that has not been included in a block.

Max Transaction Bytes

Max transaction bytes defines the max size a transaction can be for your node. If you would like your node to only keep track of smaller transactions this field would need to be changed. Default is 1MB.

Max Batch Bytes

Max batch bytes defines the amount of bytes the node will send to a peer. Default is 0.

Note: Unused due to https://github.com/tendermint/tendermint/issues/5796

Metrics

This file has moved to the node section.

Peer Strategy and Exchange

Here we outline the design of the PeerStore and how it used by the Peer Exchange Reactor (PEX) to ensure we are connected to good peers and to gossip peers to others.

Peer Types

Certain peers are special in that they are specified by the user as persistent, which means we auto-redial them if the connection fails, or if we fail to dial them. Some peers can be marked as private, which means we will not put them in the peer store or gossip them to others.

All peers except private peers and peers coming from them are tracked using the peer store.

The rest of our peers are only distinguished by being either inbound (they dialed our public address) or outbound (we dialed them).

Discovery

Peer discovery begins with a list of seeds.

When we don't have enough peers, we

  1. ask existing peers
  2. dial seeds if we're not dialing anyone currently

On startup, we will also immediately dial the given list of persistent_peers, and will attempt to maintain persistent connections with them. If the connections die, or we fail to dial, we will redial every 5s for a few minutes, then switch to an exponential backoff schedule, and after about a day of trying, stop dialing the peer. This behavior is when persistent_peers_max_dial_period is configured to zero.

But If persistent_peers_max_dial_period is set greater than zero, terms between each dial to each persistent peer will not exceed persistent_peers_max_dial_period during exponential backoff. Therefore, dial_period = min(persistent_peers_max_dial_period, exponential_backoff_dial_period) and we keep trying again regardless of maxAttemptsToDial

As long as we have less than MaxNumOutboundPeers, we periodically request additional peers from each of our own and try seeds.

Listening

Peers listen on a configurable ListenAddr that they self-report in their NodeInfo during handshakes with other peers. Peers accept up to MaxNumInboundPeers incoming peers.

Address Book

Peers are tracked via their ID (their PubKey.Address()). Peers are added to the peer store from the PEX when they first connect to us or when we hear about them from other peers.

The peer store is arranged in sets of buckets, and distinguishes between vetted (old) and unvetted (new) peers. It keeps different sets of buckets for vetted and unvetted peers. Buckets provide randomization over peer selection. Peers are put in buckets according to their IP groups.

IP group can be a masked IP (e.g. 1.2.0.0 or 2602:100::) or local for local addresses or unroutable for unroutable addresses. The mask which corresponds to the /16 subnet is used for IPv4, /32 subnet - for IPv6. Each group has a limited number of buckets to prevent DoS attacks coming from that group (e.g. an attacker buying a /16 block of IPs and launching a DoS attack).

highwayhash is used as a hashing function when calculating a bucket.

When placing a peer into a new bucket:

hash(key + sourcegroup + int64(hash(key + group + sourcegroup)) % bucket_per_group) % num_new_buckets

When placing a peer into an old bucket:

hash(key + group + int64(hash(key + addr)) % buckets_per_group) % num_old_buckets

where key - random 24 HEX string, group - IP group of the peer (e.g. 1.2.0.0), sourcegroup - IP group of the sender (peer who sent us this address) (e.g. 174.11.0.0), addr - string representation of the peer's address (e.g. 174.11.10.2:26656).

A vetted peer can only be in one bucket. An unvetted peer can be in multiple buckets, and each instance of the peer can have a different IP:PORT.

If we're trying to add a new peer but there's no space in its bucket, we'll remove the worst peer from that bucket to make room.

Vetting

When a peer is first added, it is unvetted. Marking a peer as vetted is outside the scope of the p2p package. For Tenderdash, a Peer becomes vetted once it has contributed sufficiently at the consensus layer; ie. once it has sent us valid and not-yet-known votes and/or block parts for NumBlocksForVetted blocks. Other users of the p2p package can determine their own conditions for when a peer is marked vetted.

If a peer becomes vetted but there are already too many vetted peers, a randomly selected one of the vetted peers becomes unvetted.

If a peer becomes unvetted (either a new peer, or one that was previously vetted), a randomly selected one of the unvetted peers is removed from the peer store.

More fine-grained tracking of peer behaviour can be done using a trust metric (see below), but it's best to start with something simple.

Select Peers to Dial

When we need more peers, we pick addresses randomly from the addrbook with some configurable bias for unvetted peers. The bias should be lower when we have fewer peers and can increase as we obtain more, ensuring that our first peers are more trustworthy, but always giving us the chance to discover new good peers.

We track the last time we dialed a peer and the number of unsuccessful attempts we've made. If too many attempts are made, we mark the peer as bad.

Connection attempts are made with exponential backoff (plus jitter). Because the selection process happens every ensurePeersPeriod, we might not end up dialing a peer for much longer than the backoff duration.

If we fail to connect to the peer after 16 tries (with exponential backoff), we remove from peer store completely. But for persistent peers, we indefinitely try to dial all persistent peers unless persistent_peers_max_dial_period is configured to zero

Select Peers to Exchange

When we’re asked for peers, we select them as follows:

  • select at most maxGetSelection peers
  • try to select at least minGetSelection peers - if we have less than that, select them all.
  • select a random, unbiased getSelectionPercent of the peers

Send the selected peers. Note we select peers for sending without bias for vetted/unvetted.

Preventing Spam

There are various cases where we decide a peer has misbehaved and we disconnect from them. When this happens, the peer is removed from the peer store and black listed for some amount of time. We call this "Disconnect and Mark". Note that the bad behaviour may be detected outside the PEX reactor itself (for instance, in the mconnection, or another reactor), but it must be communicated to the PEX reactor so it can remove and mark the peer.

In the PEX, if a peer sends us an unsolicited list of peers, or if the peer sends a request too soon after another one, we Disconnect and MarkBad.

Trust Metric

The quality of peers can be tracked in more fine-grained detail using a Proportional-Integral-Derivative (PID) controller that incorporates current, past, and rate-of-change data to inform peer quality.

While a PID trust metric has been implemented, it remains for future work to use it in the PEX.

See the trustmetric and trustmetric useage architecture docs for more details.

RPC

Tenderdash exposes HTTP JSON-RPC methods for node status, blocks, transactions, application queries and events. With RPC enabled, opening the configured RPC listen address in a browser lists the endpoints available on that node.

See the RPC API reference for method parameters and responses. The RPC routes and implementation define the methods supported by this source version.

For clients written in Go, use the RPC client package. For event queries and subscriptions, see event subscriptions.

Restrict RPC exposure according to your deployment. Administrative and debugging methods must not be accessible to untrusted clients. See configuration and running in production.

Running In Production

This file has moved to the nodes section.

State sync

State sync allows new nodes to rapidly bootstrap and join the network by discovering, fetching, and restoring state machine snapshots. For more information, see the state sync ABCI section).

The state sync reactor has two main responsibilities:

  • Serving state machine snapshots taken by the local ABCI application to new nodes joining the network.

  • Discovering existing snapshots and fetching snapshot chunks for an empty local application being bootstrapped.

The state sync process for bootstrapping a new node is described in detail in the section linked above. While technically part of the reactor (see statesync/syncer.go and related components), this document will only cover the P2P reactor component.

For details on the ABCI methods and data types, see the ABCI documentation.

Information on how to configure state sync is located in the nodes section

State Sync P2P Protocol

When a new node begin state syncing, it will ask all peers it encounters if it has any available snapshots:

type snapshotsRequestMessage struct{}

The receiver will query the local ABCI application via ListSnapshots, and send a message containing snapshot metadata (limited to 4 MB) for each of the 10 most recent snapshots:

type snapshotsResponseMessage struct {
 Height   uint64
 Format   uint32
 Chunks   uint32
 Hash     []byte
 Metadata []byte
}

The node running state sync will offer these snapshots to the local ABCI application via OfferSnapshot ABCI calls, and keep track of which peers contain which snapshots. Once a snapshot is accepted, the state syncer will request snapshot chunks from appropriate peers:

type chunkRequestMessage struct {
 Height uint64
 Format uint32
 Index  uint32
}

The receiver will load the requested chunk from its local application via LoadSnapshotChunk, and respond with it (limited to 16 MB):

type chunkResponseMessage struct {
 Height  uint64
 Format  uint32
 Index   uint32
 Chunk   []byte
 Missing bool
}

Here, Missing is used to signify that the chunk was not found on the peer, since an empty chunk is a valid (although unlikely) response.

The returned chunk is given to the ABCI application via ApplySnapshotChunk until the snapshot is restored. If a chunk response is not returned within some time, it will be re-requested, possibly from a different peer.

The ABCI application is able to request peer bans and chunk refetching as part of the ABCI protocol.

If no state sync is in progress (i.e. during normal operation), any unsolicited response messages are discarded.

Subscribing to Events

A Tenderdash node emits events about important state transitions during consensus. These events can be queried by clients via the RPC interface on nodes that enable it. The list of supported event types can be found in the github.com/dashpay/tenderdash/types Go package.

There are two APIs to query events:

  • The legacy streaming API, comprising the subscribe, unsubscribe, and unsubscribe_all RPC methods over websocket.

  • The event log API, comprising the events RPC method.

The event log API supports polling and pagination. Use the API supported by your node and client; the sections below describe both interfaces.

Filter Queries

Event requests take a filter query parameter. A filter query is a string that describes a subset of available event items to return. An empty query matches all events; otherwise a query comprises one or more terms comparing event metadata to target values.

For example, to select new block events, use the term:

tm.event = 'NewBlock'

Multiple terms can be combined with AND (case matters), for example to match the transaction event with a given hash, use the query:

tm.event = 'Tx' AND tx.hash = 'EA7B33F'

Operands may be strings in single quotes ('Tx'), numbers (45), dates, or timestamps.

The comparison operators include =, <, <=, >, >=, and CONTAINS (for substring match). In addition, the EXISTS operator checks for the presence of an attribute regardless of its value.

Attributes

Tenderdash implicitly defines a string-valued tm.event attribute for all event types. Transaction items (type Tx) are also assigned tx.hash (string), giving the hash of the transaction, and and tx.height (number) giving the height of the block containing the transaction. For NewBlock and NewBlockHeader events, Tenderdash defines a block.height attribute giving the height of the block.

Additional attributes can be provided by the application as ABCI Event records in ResponseProcessProposal.Events for block events and ExecTxResult.Events for transaction events. These events are published after finalization; ResponseFinalizeBlock has no events field. The full name of the attribute in the query is formed by combining the type and attribute key with a period.

For example, given the events

[]abci.Event{{
    Type: "reward",
    Attributes: []abci.EventAttribute{
        {Key: "address", Value: "example-address"},
        {Key: "amount", Value: "45.62"},
        {Key: "balance", Value: "100.390001"},
    },
}}

a query may refer to the names reward.address, reward.amount, and reward.balance, as in:

reward.address EXISTS AND reward.balance > 45

Certain application-specific metadata are also indexed for offline queries. See Indexing transactions for more details.

Event Log API

When the rpc.event-log-window-size configuration is enabled, the node maintains maintains a log of all events within this operator-defined time window. This API supersedes the websocket subscription API described below.

Clients can query these events can by long-polling the /events RPC method, which returns the most recent items from the log that match the request parameters. Each item returned includes a cursor that marks its location in the log. Cursors can be passed via the before and after parameters to fetch events earlier in the log.

For example, this request:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "events",
  "params": {
    "filter": {
      "query": "tm.event = 'Tx' AND app.key = 'applesauce'"
    },
    "maxItems": 1,
    "after": ""
  }
}

will return a result similar to the following:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "items": [
      {
        "cursor": "16ee3d5e65be53d8-03d5",
        "event": "Tx",
        "data": {
          "type": "tendermint/event/Tx",
          "value": {
            "height": 70,
            "tx": "YXBwbGVzYXVjZT1zeXJ1cA==",
            "result": {
              "events": [
                {
                  "type": "app",
                  "attributes": [
                    {
                      "key": "creator",
                      "value": "Example User",
                      "index": true
                    },
                    {
                      "key": "key",
                      "value": "applesauce",
                      "index": true
                    },
                    {
                      "key": "index_key",
                      "value": "index is working",
                      "index": true
                    },
                    {
                      "key": "noindex_key",
                      "value": "index is working",
                      "index": false
                    }
                  ]
                }
              ]
            }
          }
        }
      }
    ],
    "more": false,
    "oldest": "16ee3d4c471c3b00-0001",
    "newest": "16ee3d5f2e05a4e0-0400"
  }
}

The "items" array gives the matching items (up to the requested "maxResults") in decreasing time order (i.e., newest to oldest). In this case, there is only one result, but if there are additional results that were not returned, the "more" flag will be true. Calling /events again with the same query and "after" set to the cursor of the newest result (in this example, "16ee3d5e65be53d8-03d5") will fetch newer results.

Go clients can use the eventstream package to simplify the use of this method. The eventstream.Stream automatically handles polling for new events, updating the cursor, and reporting any missed events.

Legacy Streaming API

For a polling interface, use the event log API. See ADR 075 for design context.

To subscribe to events in the streaming API, you must connect to the node RPC service using a websocket. From the command line you can use a tool such as wscat, for example:

wscat ws://127.0.0.1:26657/websocket

To subscribe to events, call the subscribe JSON-RPC method method passing in a filter query for the events you wish to receive:

{
  "jsonrpc": "2.0",
  "method": "subscribe",
  "id": 1,
  "params": {
    "query": "tm.event='NewBlock'"
  }
}

The subscribe method returns an initial response confirming the subscription, then sends additional JSON-RPC response messages containing the matching events as they are published. The subscription continues until either the client explicitly cancels the subscription (by calling unsubscribe or unsubscribe_all) or until the websocket connection is terminated.

Using Tenderdash

Serialized key types such as tendermint/PubKeyBLS12381 are protocol identifiers. Keep those names unchanged when configuring Tenderdash.

This is a guide to using the tenderdash program from the command line. It assumes only that you have the tenderdash binary installed and have some rudimentary idea of what Tenderdash and ABCI are.

You can see the help menu with tenderdash --help, and the version number with tenderdash version.

Directory Root

The default directory for blockchain data is ~/.tenderdash. Override this by setting the TMHOME environment variable.

Initialize

Initialize the root directory by running:

tenderdash init validator

This will create a new private key (priv_validator_key.json), and a genesis file (genesis.json) containing the associated public key, in $TMHOME/config. This is all that's necessary to run a local testnet with one validator.

For more elaborate initialization, see the testnet command:

tenderdash testnet --help

Genesis

genesis.json in the node's config directory defines the initial chain ID, height, time, consensus parameters, application state and validator quorum. Use a unique chain ID of at most 50 bytes, and use the same genesis on every node in the network.

For a local development chain, tenderdash init validator writes a complete genesis and matching signing files. Do not construct a production quorum by copying keys from an example. The current GenesisDoc definition describes the supported fields, including validator pro-tx-hashes, quorum hash, quorum type and threshold public key.

The application may initialize the validator set and consensus parameters via InitChain. See the canonical startup handshake and validator-set rules.

Run

To run a Tenderdash node, use:

tenderdash start

By default, Tenderdash will try to connect to an ABCI application on 127.0.0.1:26658. If you have the kvstore ABCI app installed, run it in another window. If you don't, kill Tenderdash and run an in-process version of the kvstore app:

tenderdash start --proxy-app=kvstore

After a few seconds, you should see blocks start streaming in. Note that blocks are produced regularly, even if there are no transactions. See No Empty Blocks, below, to modify this setting.

Tenderdash supports in-process versions of the counter, kvstore, and noop apps that ship as examples with abci-cli. It's easy to compile your app in-process with Tenderdash if it's written in Go. If your app is not written in Go, run it in another process, and use the --proxy-app flag to specify the address of the socket it is listening on, for instance:

tenderdash start --proxy-app=/var/run/abci.sock

You can find out what flags are supported by running tenderdash start --help.

Transactions

To send a transaction, use curl to make requests to the Tenderdash RPC server, for example:

curl http://localhost:26657/broadcast_tx_commit?tx=\"abcd\"

We can see the chain's status at the /status end-point:

curl http://localhost:26657/status | json_pp

and the latest_app_hash in particular:

curl http://localhost:26657/status | json_pp | grep latest_app_hash

Visit http://localhost:26657 in your browser to see the list of other endpoints. Some take no arguments (like /status), while others specify the argument name and use _ as a placeholder.

TIP: See the RPC API reference.

Formatting

When sending transactions to the RPC interface, the following formatting rules must be followed:

Using GET (with parameters in the URL):

To send a UTF8 string as transaction data, enclose the value of the tx parameter in double quotes:

curl 'http://localhost:26657/broadcast_tx_commit?tx="hello"'

which sends a 5-byte transaction: "h e l l o" [68 65 6c 6c 6f].

Note that the URL in this example is enclosed in single quotes to prevent the shell from interpreting the double quotes. Alternatively, you may escape the double quotes with backslashes:

curl http://localhost:26657/broadcast_tx_commit?tx=\"hello\"

The double-quoted format works with for multibyte characters, as long as they are valid UTF8, for example:

curl 'http://localhost:26657/broadcast_tx_commit?tx="€5"'

sends a 4-byte transaction: "€5" (UTF8) [e2 82 ac 35].

Arbitrary (non-UTF8) transaction data may also be encoded as a string of hexadecimal digits (2 digits per byte). To do this, omit the quotation marks and prefix the hex string with 0x:

curl http://localhost:26657/broadcast_tx_commit?tx=0x68656C6C6F

which sends the 5-byte transaction: [68 65 6c 6c 6f].

Using POST (with parameters in JSON), the transaction data are sent as a JSON string in base64 encoding:

curl http://localhost:26657 -H 'Content-Type: application/json' --data-binary '{
  "jsonrpc": "2.0",
  "id": "anything",
  "method": "broadcast_tx_commit",
  "params": {
    "tx": "aGVsbG8="
  }
}'

which sends the same 5-byte transaction: [68 65 6c 6c 6f].

Note that the hexadecimal encoding of transaction data is not supported in JSON (POST) requests.

Reset

:warning: UNSAFE Only do this in development and only if you can afford to lose all blockchain data!

To reset a blockchain, stop the node and run:

tenderdash unsafe-reset-all

This command will remove the data directory and reset private validator and address book files.

Configuration

Tenderdash uses a config.toml for configuration. For details, see the config specification.

Notable options include the socket address of the application (proxy-app), the listening address of the Tenderdash peer (p2p.laddr), and the listening address of the RPC server (rpc.laddr).

Some fields from the config file can be overwritten with flags.

No Empty Blocks

Configure create-empty-blocks and create-empty-blocks-interval in config.toml to control waiting for transactions at round 0 (and, on consensus version 2, in a round entered after a silent proposer's round was skipped). The application can request an immediate next block through ResponseFinalizeBlock. An app hash change does not itself require an extra block: Tenderdash uses same-block execution.

See the configuration guide and the canonical empty-block rules.

Broadcast API

Earlier, we used the broadcast_tx_commit endpoint to send a transaction. When a transaction is sent to a Tenderdash node, it will run via CheckTx against the application. If it passes CheckTx, it can be admitted to the mempool and gossiped to peers. Admission does not guarantee later inclusion in a block.

Since there are multiple phases to processing a transaction, we offer multiple endpoints to broadcast a transaction:

/broadcast_tx_async
/broadcast_tx_sync
/broadcast_tx_commit

These correspond to no-processing, processing through the mempool, and processing through a block, respectively. That is, broadcast_tx_async, will return right away without waiting to hear if the transaction is even valid, while broadcast_tx_sync will return with the result of running the transaction through CheckTx. Using broadcast_tx_commit will wait until the transaction is committed in a block or until some timeout is reached, but will return right away if the transaction does not pass CheckTx. The return value for broadcast_tx_commit includes check_tx for mempool validation and tx_result for the committed transaction execution result.

The benefit of using broadcast_tx_commit is that the request returns after the transaction is committed (i.e. included in a block). For a quick result, use broadcast_tx_sync, but the transaction will not be committed until later, and by that point its effect on the state may change.

Note the mempool does not provide strong guarantees - just because a tx passed CheckTx (ie. was accepted into the mempool), doesn't mean it will be committed, as nodes with the tx in their mempool may crash before they get to propose. For more information, see the mempool guide.

Tenderdash Networks

All nodes joining a network must use its agreed genesis and compatible software. For a local development chain, initialize files with tenderdash init validator; for a deployed network, obtain its genesis and validator/quorum configuration through the network's deployment process.

priv_validator_key.json contains secret signing material. Protect both keys and signing state; never copy a live consensus identity to a second active signer. See validator keys and recovery.

A validator signs only when it belongs to the active quorum. A non-validator full node follows committed blocks without voting. See the validator guide for membership updates and proposer selection for the versioned, equal-power rotation rules.

Peers

Seed

A seed node is a node who relays the addresses of other peers which they know of. These nodes constantly crawl the network to try to get more peers. The addresses which the seed node relays get saved into a local address book. Once these are in the address book, you will connect to those addresses directly. Basically the seed nodes job is just to relay everyones addresses. You won't connect to seed nodes once you have received enough addresses, so typically you only need them on the first start. The seed node will immediately disconnect from you after sending you some addresses.

Persistent Peer

Persistent peers are people you want to be constantly connected with. If you disconnect you will try to connect directly back to them as opposed to using another address from the address book. On restarts you will always try to connect to these peers regardless of the size of your address book.

All peers relay peers they know of by default. This is called the peer exchange protocol (PeX). With PeX, peers will be gossiping about known peers and forming a network, storing peer addresses in the addrbook. Because of this, you don't have to use a seed node if you have a live persistent peer.

Connecting to Peers

To connect to peers on start-up, specify them in the $TMHOME/config/config.toml or on the command line. Use seeds to specify seed nodes, and persistent-peers to specify peers that your node will maintain persistent connections with.

For example,

tenderdash start --p2p.seeds "f9baeaa15fedf5e1ef7448dd60f46c01f1a9e9c4@1.2.3.4:26656,0491d373a8e0fcf1023aaf18c51d6a1d0d4f31bd@5.6.7.8:26656"

Alternatively, you can use the /dial_seeds endpoint of the RPC to specify seeds for a running node to connect to:

curl 'localhost:26657/dial_seeds?seeds=\["f9baeaa15fedf5e1ef7448dd60f46c01f1a9e9c4@1.2.3.4:26656","0491d373a8e0fcf1023aaf18c51d6a1d0d4f31bd@5.6.7.8:26656"\]'

Note, with PeX enabled, you should not need seeds after the first start.

If you want Tenderdash to connect to specific set of addresses and maintain a persistent connection with each, you can use the --p2p.persistent-peers flag or the corresponding setting in the config.toml or the /dial_peers RPC endpoint to do it without stopping Tenderdash core instance.

tenderdash start --p2p.persistent_peers "429fcf25974313b95673f58d77eacdd434402665@10.11.12.13:26656,96663a3dd0d7b9d17d4c8211b191af259621c693@10.11.12.14:26656"

curl 'localhost:26657/dial_peers?persistent=true&peers=\["429fcf25974313b95673f58d77eacdd434402665@10.11.12.13:26656","96663a3dd0d7b9d17d4c8211b191af259621c693@10.11.12.14:26656"\]'

Adding a Non-Validator

Adding a non-validator is simple. Just copy the original genesis.json to ~/.tenderdash/config on the new machine and start the node, specifying seeds or persistent peers as necessary. If no seeds or persistent peers are specified, the node won't make any blocks, because it's not a validator, and it won't hear about any blocks, because it's not connected to the other peer.

Adding a Validator

Editing a local genesis file does not change the validator set of a running network. Initial validators are established through genesis and InitChain; later changes come from the application and take effect at the next height. Follow the network's quorum deployment process and the canonical validator-set update rules.

Local Network

To run a network locally, say on a single machine, you must change the _laddr fields in the config.toml (or using the flags) so that the listening addresses of the various sockets don't conflict. Additionally, you must set addr_book_strict=false in the config.toml, otherwise Tenderdash's p2p library will deny making connections to peers with the same IP address.

Upgrading

See the UPGRADING.md guide. You may need to reset your chain between major breaking releases. Although, we expect Tenderdash to have fewer breaking releases in the future (especially after 1.0 release).

Validators

This file has moved to the node section.

Overview

Tenderdash has some tools that are associated with it for:

Benchmarking

tm-load-test is an external load testing tool originally developed for upstream networks. Verify its transaction format and RPC compatibility before using it with Tenderdash.

Testnets

This external repository contains upstream testnet configurations. They are not Tenderdash network configurations.

Use Docker Compose to spin up Tenderdash testnets on your local machine.

Use Terraform and Ansible to deploy Tenderdash testnets to the cloud.

See the tenderdash testnet --help command for more help initializing testnets.

Debugging

Tenderdash debug kill

Tenderdash comes with a debug sub-command that allows you to kill a live Tenderdash process while collecting useful information in a compressed archive. The information includes the configuration used, consensus state, network state, the node' status, the WAL, and even the stack trace of the process before exit. These files can be useful to examine when debugging a faulty Tenderdash process.

tenderdash debug kill <pid> </path/to/out.zip> --home=</path/to/app.d>

will write debug info into a compressed archive. The archive will contain the following:

├── config.toml
├── consensus_state.json
├── net_info.json
├── stacktrace.out
├── status.json
└── wal

Under the hood, debug kill fetches info from /status, /net_info, and /dump_consensus_state HTTP endpoints, and kills the process with -6, which catches the go-routine dump.

Tenderdash debug dump

Also, the debug dump sub-command allows you to dump debugging data into compressed archives at a regular interval. These archives contain the goroutine and heap profiles in addition to the consensus state, network info, node status, and even the WAL.

tenderdash debug dump </path/to/out> --home=</path/to/app.d>

will perform similarly to kill except it only polls the node and dumps debugging data every frequency seconds to a compressed archive under a given destination directory. Each archive will contain:

├── consensus_state.json
├── goroutine.out
├── heap.out
├── net_info.json
├── status.json
└── wal

Note: goroutine.out and heap.out will only be written if a profile address is provided and is operational. This command is blocking and will log any error.

Tenderdash Inspect

Tenderdash includes an inspect command for querying Tenderdash's state store and block store over Tenderdash RPC.

When the Tenderdash consensus engine detects inconsistent state, it will crash the entire Tenderdash process. While in this inconsistent state, a node running Tenderdash's consensus engine will not start up. The inspect command runs only a subset of Tenderdash's RPC endpoints for querying the block store and state store. inspect allows operators to query a read-only view of the stage. inspect does not run the consensus engine at all and can therefore be used to debug processes that have crashed due to inconsistent state.

To start the inspect process, run

tenderdash inspect

RPC endpoints

The list of available RPC endpoints can be found by making a request to the RPC port. For an inspect process running on 127.0.0.1:26657, navigate your browser to http://127.0.0.1:26657/ to retrieve the list of enabled RPC endpoints.

Additional information on the Tenderdash RPC endpoints can be found in the rpc documentation.

Debug Like A Pro

Intro

Tenderdash Core is a fairly robust BFT replication engine. Unfortunately, as with other software, failures sometimes do happen. The question is then “what do you do” when the system deviates from the expected behavior.

The first response is usually to take a look at the logs. By default, Tenderdash writes logs to standard output ¹.

I[2020-05-29|03:03:16.145] Committed state                              module=state height=2282 txs=0 appHash=0A27BC6B0477A8A50431704D2FB90DB99CBFCB67A2924B5FBF6D4E78538B67C1I[2020-05-29|03:03:21.690] Executed block                               module=state height=2283 validTxs=0 invalidTxs=0I[2020-05-29|03:03:21.698] Committed state                              module=state height=2283 txs=0 appHash=EB4E409D3AF4095A0757C806BF160B3DE4047AC0416F584BFF78FC0D44C44BF3I[2020-05-29|03:03:27.994] Executed block                               module=state height=2284 validTxs=0 invalidTxs=0I[2020-05-29|03:03:28.003] Committed state                              module=state height=2284 txs=0 appHash=3FC9237718243A2CAEE3A8B03AE05E1FC3CA28AEFE8DF0D3D3DCE00D87462866E[2020-05-29|03:03:32.975] enterPrevote: ProposalBlock is invalid       module=consensus height=2285 round=0 err="wrong signature (#35): C683341000384EA00A345F9DB9608292F65EE83B51752C0A375A9FCFC2BD895E0792A0727925845DC13BA0E208C38B7B12B2218B2FE29B6D9135C53D7F253D05"

If you’re running a validator in production, it might be a good idea to forward the logs for analysis using filebeat or similar tools. Also, you can set up a notification in case of any errors.

The logs should give you the basic idea of what has happened. In the worst-case scenario, the node has stalled and does not produce any logs (or simply panicked).

The next step is to call /status, /net_info, /consensus_state and /dump_consensus_state RPC endpoints.

curl http://<server>:26657/status$ curl http://<server>:26657/net_info$ curl http://<server>:26657/consensus_state$ curl http://<server>:26657/dump_consensus_state

Please note that /consensus_state and /dump_consensus_state may not return a result if the node has stalled (since they try to get a hold of the consensus mutex).

The output of these endpoints contains all the information needed for developers to understand the state of the node. It will give you an idea if the node is lagging behind the network, how many peers it’s connected to, and what the latest consensus state is.

At this point, if the node is stalled and you want to restart it, the best thing you can do is to kill it with -6 signal:

kill -6 <PID>

which will dump the list of the currently running goroutines. The list is super useful when debugging a deadlock.

PID is the Tenderdash’s process ID. You can find it out by running ps -a | grep tenderdash | awk ‘{print $1}’

Tenderdash debug kill

To ease the burden of collecting different pieces of data Tenderdash Core provides the Tenderdash debug kill tool, which will do all of the above steps for you, wrapping everything into a nice archive file.

tenderdash debug kill <pid> </path/to/out.zip> — home=</path/to/app.d>

Here’s the official documentation page — debugging guide

If you’re using a process supervisor, like systemd, it will restart the Tenderdash automatically. We strongly advise you to have one in production. If not, you will need to restart the node by hand.

Another advantage of using Tenderdash debug is that the same archive file can be given to Tenderdash Core developers, in cases where you think there’s a software issue.

Tenderdash debug dump

Okay, but what if the node has not stalled, but its state is degrading over time? Tenderdash debug dump to the rescue!

tenderdash debug dump </path/to/out> — home=</path/to/app.d>

It won’t kill the node, but it will gather all of the above data and package it into an archive file. Plus, it will also make a heap dump, which should help if Tenderdash is leaking memory.

At this point, depending on how severe the degradation is, you may want to restart the process.

Tenderdash Inspect

What if the Tenderdash node will not start up due to inconsistent consensus state?

When a node running the Tenderdash consensus engine detects an inconsistent state it will crash the entire Tenderdash process. The Tenderdash consensus engine cannot be run in this inconsistent state and the so node will fail to start up as a result. The Tenderdash RPC server can provide valuable information for debugging in this situation. The Tenderdash inspect command will run a subset of the Tenderdash RPC server that is useful for debugging inconsistent state.

Running inspect

Start up the inspect tool on the machine where Tenderdash crashed using:

tenderdash inspect --home=</path/to/app.d>

inspect will use the data directory specified in your Tenderdash configuration file. inspect will also run the RPC server at the address specified in your Tenderdash configuration file.

Using inspect

With the inspect server running, you can access RPC endpoints that are critically important for debugging. Calling the /status, /consensus_state and /dump_consensus_state RPC endpoint will return useful information about the Tenderdash consensus state.

Outro

We’re hoping that these Tenderdash tools will become de facto the first response for any accidents.

Let us know what your experience has been so far! Have you had a chance to try tenderdash debug or tenderdash inspect yet?

Report reproducible node issues in the Tenderdash issue tracker.

—

[1]: Of course, you’re free to redirect the Tenderdash’s output to a file or forward it to another server.

Proposer-Based Timestamps Runbook

Tenderdash constrains the timestamps included in each block. These constraints mean that validators may fail to produce valid blocks or may issue nil prevotes for proposed blocks depending on the configuration of the validator's local clock.

What is this document for?

This document provides a set of actionable steps for application developers and node operators to diagnose and fix issues related to clock synchronization and configuration of the Proposer-Based Timestamps SynchronyParams.

Use this runbook if you observe that validators are frequently voting nil for a block that the rest of the network votes for or if validators are frequently producing block proposals that are not voted for by the rest of the network.

Requirements

To use this runbook, you must be running a node that has the Prometheus metrics endpoint enabled and the Tenderdash RPC endpoint enabled and accessible.

It is strongly recommended to also run a Prometheus metrics collector to gather and analyze metrics from the Tenderdash node.

Debugging a Single Node

If you observe that a single validator is frequently failing to produce blocks or voting nil for proposals that other validators vote for and suspect it may be related to clock synchronization, use the following steps to debug and correct the issue.

Check Timely Metric

Tenderdash exposes a histogram metric for the difference between the timestamp in the proposal the and the time read from the node's local clock when the proposal is received.

The histogram exposes multiple metrics on the Prometheus /metrics endpoint called

  • tendermint_consensus_proposal_timestamp_difference_bucket.
  • tendermint_consensus_proposal_timestamp_difference_sum.
  • tendermint_consensus_proposal_timestamp_difference_count.

Each metric is also labeled with the key is_timely, which can have a value of true or false.

From the Prometheus Collector UI

If you are running a Prometheus collector, navigate to the query web interface and select the 'Graph' tab.

Issue a query for the following:

tendermint_consensus_proposal_timestamp_difference_count{is_timely="false"} /
tendermint_consensus_proposal_timestamp_difference_count{is_timely="true"}

This query will graph the ratio of proposals the node considered timely to those it considered untimely. If the ratio is increasing, it means that your node is consistently seeing more proposals that are far from its local clock. If this is the case, you should check to make sure your local clock is properly synchronized to NTP.

From the /metrics url

If you are not running a Prometheus collector, navigate to the /metrics endpoint exposed on the Prometheus metrics port with curl or a browser.

Search for the tendermint_consensus_proposal_timestamp_difference_count metrics. This metric is labeled with is_timely. Investigate the value of tendermint_consensus_proposal_timestamp_difference_count where is_timely="false" and where is_timely="true". Refresh the endpoint and observe if the value of is_timely="false" is growing.

If you observe that is_timely="false" is growing, it means that your node is consistently seeing proposals that are far from its local clock. If this is the case, you should check to make sure your local clock is properly synchronized to NTP.

Checking Clock Sync

NTP configuration and tooling is very specific to the operating system and distribution that your validator node is running. This guide assumes you have timedatectl installed with chrony, a popular tool for interacting with time synchronization on Linux distributions. If you are using an operating system or distribution with a different time synchronization mechanism, please consult the documentation for your operating system to check the status and re-synchronize the daemon.

Check if NTP is Enabled

$ timedatectl

From the output, ensure that NTP service is active. If NTP service is inactive, run:

$ timedatectl set-ntp true

Re-run the timedatectl command and verify that the change has taken effect.

Check if Your NTP Daemon is Synchronized

Check the status of your local chrony NTP daemon using by running the following:

$ chronyc tracking

If the chrony daemon is running, you will see output that indicates its current status. If the chrony daemon is not running, restart it and re-run chronyc tracking.

The System time field of the response should show a value that is much smaller than 100 milliseconds.

If the value is very large, restart the chronyd daemon.

Debugging a Network

If you observe that a network is frequently failing to produce blocks and suspect it may be related to clock synchronization, use the following steps to debug and correct the issue.

Check Prevote Message Delay

Tenderdash exposes metrics that help determine how synchronized the clocks on a network are.

These metrics are visible on the Prometheus /metrics endpoint and are called:

  • tendermint_consensus_quorum_prevote_delay
  • tendermint_consensus_full_prevote_delay

These metrics calculate the difference between the timestamp in the proposal message and the timestamp of a prevote that was issued during consensus.

The tendermint_consensus_quorum_prevote_delay metric is the interval in seconds between the proposal timestamp and the timestamp of the earliest prevote that achieved a quorum during the prevote step.

The tendermint_consensus_full_prevote_delay metric is the interval in seconds between the proposal timestamp and the timestamp of the latest prevote in a round where 100% of the validators voted.

From the Prometheus Collector UI

If you are running a Prometheus collector, navigate to the query web interface and select the 'Graph' tab.

Issue a query for the following:

sum(tendermint_consensus_quorum_prevote_delay) by (proposer_address)

This query will graph the difference in seconds for each proposer on the network.

If the value is much larger for some proposers, then the issue is likely related to the clock synchronization of their nodes. Contact those proposers and ensure that their nodes are properly connected to NTP using the steps for Debugging a Single Node.

If the value is relatively similar for all proposers you should next compare this value to the SynchronyParams values for the network. Continue to the Checking Sychrony steps.

From the /metrics url

If you are not running a Prometheus collector, navigate to the /metrics endpoint exposed on the Prometheus metrics port.

Search for the tendermint_consensus_quorum_prevote_delay metric. There will be one entry of this metric for each proposer_address. If the value of this metric is much larger for some proposers, then the issue is likely related to synchronization of their nodes with NTP. Contact those proposers and ensure that their nodes are properly connected to NTP using the steps for Debugging a Single Node.

If the values are relatively similar for all proposers you should next compare, you'll need to compare this value to the SynchronyParams for the network. Continue to the Checking Sychrony steps.

Checking Synchrony

To determine the currently configured SynchronyParams for your network, issue a request to your node's RPC endpoint. For a node running locally with the RPC server exposed on port 26657, run the following command:

$ curl localhost:26657/consensus_params

The json output will contain a field named synchrony, with the following structure:

{
  "precision": "500000000",
  "message_delay": "3000000000"
}

The precision and message_delay values returned are listed in nanoseconds: In the examples above, the precision is 500ms and the message delay is 3s. Remember, tendermint_consensus_quorum_prevote_delay is listed in seconds. If the tendermint_consensus_quorum_prevote_delay value approaches the sum of precision and message_delay, then the value selected for these parameters is too small. Your application will need to be modified to update the SynchronyParams to have larger values.

Updating SynchronyParams

The SynchronyParams are ConsensusParameters which means they are set and updated by the application running alongside Tenderdash. Coordinate parameter changes through the application and the network upgrade process. Check the application's supported parameter update mechanism.

If the application does not implement a way to update the consensus parameters programatically, then the application itself must be updated to do so. More information on updating the consensus parameters via ABCI can be found in the FinalizeBlock documentation.

Docker Compose

With Docker Compose, you can spin up local testnets with a single command.

Requirements

  1. Install Tenderdash
  2. Install docker
  3. Install docker-compose

Build

Build the tenderdash binary and, optionally, the dashpay/tenderdash:localnode docker image.

Note the binary will be mounted into the container so it can be updated without rebuilding the image.

# Build the linux binary in ./build
make build-linux

# (optionally) Build dashpay/tenderdash:localnode image
make build-docker-localnode

Run a testnet

To start a 4 node testnet run:

make localnet-start

The nodes bind their RPC servers to ports 26657, 26660, 26662, and 26664 on the host.

This file creates a 4-node network using the localnode image.

The nodes of the network expose their P2P and RPC endpoints to the host machine on ports 26656-26657, 26659-26660, 26661-26662, and 26663-26664 respectively.

The first node (node0) exposes two additional ports: 6060 for profiling using pprof, and 9090 - for Prometheus server (if you don't know how to start one check out "First steps | Prometheus").

To update the binary, just rebuild it and restart the nodes:

make build-linux
make localnet-start

Configuration

The make localnet-start creates files for a 4-node testnet in ./build by calling the tenderdash testnet command.

The ./build directory is mounted to the /tendermint mount point to attach the binary and config files to the container.

To change the number of validators / non-validators change the localnet-start Makefile target here:

localnet-start: localnet-stop
  @if ! [ -f build/node0/config/genesis.json ]; then docker run --rm -v $(CURDIR)/build:/tendermint:Z dashpay/tenderdash:localnode testnet --v 5 --n 3 --o . --populate-persistent-peers --starting-ip-address 192.167.10.2 ; fi
  docker compose up

The command now will generate config files for 5 validators and 3 non-validators. Along with generating new config files the docker-compose file needs to be edited. Adding 4 more nodes is required in order to fully utilize the config files that were generated.

  node3: # bump by 1 for every node
    container_name: node3 # bump by 1 for every node
    image: "dashpay/tenderdash:localnode"
    environment:
      - ID=3
      - LOG=${LOG:-tenderdash.log}
    ports:
      - "26663-26664:26656-26657" # Bump 26663-26664 by one for every node
    volumes:
      - ./build:/tendermint:Z
    networks:
      localnet:
        ipv4_address: 192.167.10.5 # bump the final digit by 1 for every node

Before running it, don't forget to cleanup the old files:

# Clear the build folder
rm -rf ./build/node*

Configuring ABCI containers

To use your own ABCI applications with 4-node setup edit the docker-compose.yaml file and add image to your ABCI application.

 abci0:
    container_name: abci0
    image: "abci-image"
    build:
      context: .
      dockerfile: abci.Dockerfile
    command: <insert command to run your abci application>
    networks:
      localnet:
        ipv4_address: 192.167.10.6

  abci1:
    container_name: abci1
    image: "abci-image"
    build:
      context: .
      dockerfile: abci.Dockerfile
    command: <insert command to run your abci application>
    networks:
      localnet:
        ipv4_address: 192.167.10.7

  abci2:
    container_name: abci2
    image: "abci-image"
    build:
      context: .
      dockerfile: abci.Dockerfile
    command: <insert command to run your abci application>
    networks:
      localnet:
        ipv4_address: 192.167.10.8

  abci3:
    container_name: abci3
    image: "abci-image"
    build:
      context: .
      dockerfile: abci.Dockerfile
    command: <insert command to run your abci application>
    networks:
      localnet:
        ipv4_address: 192.167.10.9

Override the command in each node to connect to it's ABCI.

  node0:
    container_name: node0
    image: "dashpay/tenderdash:localnode"
    ports:
      - "26656-26657:26656-26657"
    environment:
      - ID=0
      - LOG=$${LOG:-tenderdash.log}
    volumes:
      - ./build:/tendermint:Z
    command: node --proxy-app=tcp://abci0:26658
    networks:
      localnet:
        ipv4_address: 192.167.10.2

Similarly do for node1, node2 and node3 then run testnet.

Logging

Log is saved under the attached volume, in the tenderdash.log file. If the LOG environment variable is set to stdout at start, the log is not saved, but printed on the screen.

Special binaries

If you have multiple binaries with different names, you can specify which one to run with the BINARY environment variable. The path of the binary is relative to the attached volume.

Terraform & Ansible

Note: These commands/files are not being maintained by the Tenderdash team currently. Please use them carefully.

Automated deployments are done using Terraform to create servers on Digital Ocean then Ansible to create and manage testnets on those servers.

Install

NOTE: see the integration bash script that can be run on a fresh DO droplet and will automatically spin up a 4 node testnet. The script more or less does everything described below.

  • Install Terraform and Ansible on a Linux machine.
  • Create a DigitalOcean API token with read and write capability.
  • Install the python dopy package (pip install dopy)
  • Create SSH keys (ssh-keygen)
  • Set environment variables:
export DO_API_TOKEN="abcdef01234567890abcdef01234567890"
export SSH_KEY_FILE="$HOME/.ssh/id_rsa.pub"

These will be used by both terraform and ansible.

Terraform

This step will create four Digital Ocean droplets. First, go to the correct directory:

cd $GOPATH/src/github.com/dashpay/tenderdash/networks/remote/terraform

then:

terraform init
terraform apply -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE"

and you will get a list of IP addresses that belong to your droplets.

With the droplets created and running, let's setup Ansible.

Ansible

The playbooks in the ansible directory run ansible roles to configure the sentry node architecture. You must switch to this directory to run ansible (cd $GOPATH/src/github.com/dashpay/tenderdash/networks/remote/ansible).

There are several roles that are self-explanatory:

First, we configure our droplets by specifying the paths for tenderdash (BINARY) and the node files (CONFIGDIR). The latter expects any number of directories named node0, node1, ... and so on (equal to the number of droplets created).

To create the node files run:

tenderdash testnet

Then, to configure our droplets run:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet config.yml -e BINARY=$GOPATH/src/github.com/dashpay/tenderdash/build/tenderdash -e CONFIGDIR=$GOPATH/src/github.com/dashpay/tenderdash/networks/remote/ansible/mytestnet

Voila! All your droplets now have the tenderdash binary and required configuration files to run a testnet.

Next, we run the install role:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml

which as you'll see below, executes tenderdash node --proxy-app=kvstore on all droplets. Although we'll soon be modifying this role and running it again, this first execution allows us to get each node_info.id that corresponds to each node_info.listen_addr. (This part will be automated in the future). In your browser (or using curl), for every droplet, go to IP:26657/status and note the two just mentioned node_info fields. Notice that blocks aren't being created (latest_block_height should be zero and not increasing).

Next, open roles/install/templates/systemd.service.j2 and look for the line ExecStart which should look something like:

ExecStart=/usr/bin/tenderdash node --proxy-app=kvstore

and add the --p2p.persistent-peers flag with the relevant information for each node. The resulting file should look something like:

[Unit]
Description={{service}}
Requires=network-online.target
After=network-online.target

[Service]
Restart=on-failure
User={{service}}
Group={{service}}
PermissionsStartOnly=true
ExecStart=/usr/bin/tenderdash node --proxy-app=kvstore --p2p.persistent-peers=167b80242c300bf0ccfb3ced3dec60dc2a81776e@165.227.41.206:26656,3c7a5920811550c04bf7a0b2f1e02ab52317b5e6@165.227.43.146:26656,303a1a4312c30525c99ba66522dd81cca56a361a@159.89.115.32:26656,b686c2a7f4b1b46dca96af3a0f31a6a7beae0be4@159.89.119.125:26656
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

Then, stop the nodes:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet stop.yml

Finally, we run the install role again:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml

to re-run tenderdash node with the new flag, on all droplets. The latest_block_hash should now be changing and latest_block_height increasing. Your testnet is now up and running :)

Peek at the logs with the status role:

ansible-playbook -i inventory/digital_ocean.py -l sentrynet status.yml

Logging

The crudest way is the status role described above. You can also ship logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana) service provider. You can set up your nodes to log there automatically. Create an account and get your API key from the notes on this page, then:

yum install systemd-devel || echo "This will only work on RHEL-based systems."
apt-get install libsystemd-dev || echo "This will only work on Debian-based systems."

go install github.com/mheese/journalbeat@latest
ansible-playbook -i inventory/digital_ocean.py -l sentrynet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345

Cleanup

To remove your droplets, run:

terraform destroy -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE"

Tenderdash Specifications

This directory hosts the canonical Markdown specifications of the Tenderdash Protocol.

It shall be used to describe protocol semantics, namely the BFT consensus engine, leader election, block propagation and light client verification. The specification includes encoding descriptions used in interprocess communication to comply with the protocol. It defines the interface between the application and Tenderdash. The english specifications are often accompanies with a TLA+ specification.

Table of Contents

Contibuting

Contributions are welcome.

Proposals at an early stage can first be drafted as Github issues. To progress, a proposal will often need to be written out and approved as a Request For Comment (RFC).

The standard language for coding blocks is Golang.

If you find discrepancies between the spec and the code that do not have an associated issue or pull request on github, please report them in the Tenderdash issue tracker. Report security vulnerabilities privately through the repository's security policy.

Overview

Tenderdash provides Byzantine Fault Tolerant State Machine Replication using hash-linked batches of transactions. Such transaction batches are called "blocks". Hence, Tenderdash defines a "blockchain".

Each block in Tenderdash has a unique index - its Height. Height's in the blockchain are monotonic. Each block is committed by a known set of weighted Validators. Membership and weighting within this validator set may change over time. Tenderdash guarantees the safety and liveness of the blockchain so long as less than 1/3 of the total weight of the Validator set is malicious or faulty.

A commit in Tenderdash is a set of signed messages from more than 2/3 of the total weight of the current Validator set. Validators take turns proposing blocks and voting on them. Once enough votes are received, the block is considered committed. These votes are included in the next block as proof that the previous block was committed - they cannot be included in the current block, as that block has already been created.

Once a block is committed, it can be executed against an application. The application returns results for each of the transactions in the block. The application can also return changes to be made to the validator set, as well as a cryptographic digest of its latest state.

Tenderdash is designed to enable efficient verification and authentication of the latest state of the blockchain. To achieve this, it embeds cryptographic commitments to certain information in the block "header". This information includes the contents of the block (eg. the transactions), the validator set committing the block, as well as the various results returned by the application. Note, however, that block execution only occurs after a block is committed. Thus, application results can only be included in the next block.

Also note that information like the transaction results and the validator set are never directly included in the block - only their cryptographic digests (Merkle roots) are. Hence, verification of a block requires a separate data structure to store this information. We call this the State. Block verification also requires access to the previous block.

Tenderdash Consensus Protocol

This document and its companions specify the Tenderdash consensus protocol as implemented in internal/consensus and the packages it depends on. Every non-trivial rule cites the function that implements it. Where the implementation diverges from the Tendermint algorithm it descends from, the implementation wins and the divergence is recorded in Known issues and discrepancies.

Audience: protocol engineers and auditors. Readers are expected to know the Tendermint consensus algorithm (consensus paper) and the ABCI interface (ABCI spec).

Documents

DocumentContent
Consensus Protocol (this page)Scope, terminology, system model, round state, step state machine, invariants
Primary flow of a heightThe happy path from NewHeight to the next NewHeight, with the ABCI calls it makes
Transitions and decision rulesFormal transition table: guards, actions and postconditions of every step action; timeout semantics; vote-driven dispatch; prevote, precommit and proposal-validity decision procedures
Alternative flows and edge casesMissing or invalid proposals, locking, valid-block bookkeeping, round skipping, nil rounds, equivocation, vote extensions, timestamps, empty blocks, non-validators, panics, DoS budgets
Proposer liveness and round skippingFrom CONSENSUS_VERSION_2: proposer heartbeats while a round waits for transactions, skip votes, round skip certificates, the round after a skip, their parameters and budgets
Commit handlingCommits built from local precommits and commits received from peers: verification, parking, candidates, application of a commit, gossip
Recovery and synchronisationWAL, crash recovery, ABCI handshake replay, block-sync handover
Known issues and discrepanciesBehaviour that looks like a bug or diverges from the intended algorithm

Related specifications that this one links to rather than restates:

Scope

In scope: the consensus state machine of a single node (validator or full node), the rules that decide what it signs, the interaction with the application over ABCI, and the parts of message exchange that change consensus state. Out of scope: the p2p transport, mempool internals, block sync and state sync (except their handover to consensus), light clients, and the complete DoS-defence design. For operational monitoring, see DoS observability.

Source of truth: the code at the revision this document was written against (branch dev). Citations use path/file.go and a function name; line numbers are omitted because they drift.

Terminology

TermDefinitionSource
Height hIndex of the block being decided. State.LastBlockHeight + 1, or InitialHeight for the first block.StateData.updateToState in internal/consensus/state_data.go
Round rAttempt number within a height, starting at 0.RoundState.Round in internal/consensus/types/round_state.go
StepPosition within a round: one of the RoundStepType values listed in Round steps.internal/consensus/types/round_state.go
Validator setThe members of the active Dash LLMQ quorum: ValidatorSet{Validators, QuorumType, QuorumHash, ThresholdPublicKey, VotingPowerThreshold, HasPublicKeys}. Every validator has voting power DefaultDashVotingPower = 100.types/validator_set.go
Validator / observerA node whose PrivValidator pro-tx-hash is in the validator set is a validator; any other node is an observer. Observers do not sign and ignore votes (see Non-validators).StateData.isValidator, Reactor.handleVoteMessage in internal/consensus/reactor.go
Commit threshold TVoting power at or above which a vote set is treated as "+2/3". T = ValidatorSet.VotingPowerThreshold when that field is set (> 0), otherwise the LLMQ-type threshold QuorumTypeThresholdVotingPower() (threshold member count × 100). A configured T below the strict floor ⌊2·total/3⌋ + 1 is only warned about (BelowStrictThreshold).ValidatorSet.QuorumVotingThresholdPower, ValidatorSet.BelowStrictThreshold in types/validator_set.go; warning logged in StateData.updateToState
Recovery threshold RVoting power whose BLS signature shares suffice to recover the quorum's threshold signature: QuorumTypeThresholdVotingPower(). It ignores the VotingPowerThreshold override.ValidatorSet.QuorumTypeThresholdVotingPower in types/validator_set.go
"+2/3 any"A vote set whose total admitted power ≥ T, regardless of which block ids the votes name.VoteSet.HasTwoThirdsAny in types/vote_set.go
"+2/3 majority" / maj23A vote set whose votes for one block id (possibly nil) reach T. For precommit sets the majority is additionally recognised only once the threshold signature over the block has been recovered and verified (see Threshold recovery).VoteSet.TwoThirdsMajority, VoteSet.addVerifiedVote in types/vote_set.go
PolkaA +2/3 majority of prevotes for one block id (nil included) in one round.HeightVoteSet.POLInfo in internal/consensus/types/height_vote_set.go
POLRoundThe round whose polka a proposal claims as justification; -1 when the proposal claims none.types.Proposal.POLRound
BlockID(Hash, PartSetHeader, StateID). Nil block id = all three empty.types/block.go, types/block_meta.go
StateIDHash of {AppVersion, Height, AppHash, CoreChainLockedHeight, Time} of the block header; the app hash is the state after executing that block (same-block execution).Header.StateID in types/block.go
Commit(Height, Round, BlockID, QuorumHash, ThresholdBlockSignature, ThresholdVoteExtensions): a single aggregate BLS threshold signature plus one threshold signature per threshold-recoverable vote extension. There are no per-validator signatures in a commit.types.Commit in types/block.go
Locked block / roundThe block the node precommitted most recently and the round it did so; -1/nil when none.RoundState.LockedRound/LockedBlock
Valid block / roundThe most recent block (and round) for which the node saw a polka in a round ≥ any previous one.RoundState.ValidRound/ValidBlock
CurrentRoundStateThe application's answer to the latest PrepareProposal/ProcessProposal for the block held in ProposalBlock, tagged with the round it was processed for.RoundState.CurrentRoundState, RoundState.ProcessedBlockID; processedFor in internal/consensus/block_executor.go
Round skip certificateA quorum threshold signature recovered from the skip votes of validators holding R, proving that they stopped hearing from the proposer of (h, r) while waiting for transactions; moves a node to r + 1. CONSENSUS_VERSION_2 only.types.RoundSkipCertificate in types/round_skip.go; Proposer liveness
Parked commitA verified commit for the current height whose block has not yet been assembled (RoundState.Commit != nil).StateData.adoptCommit in internal/consensus/state_data.go
TimelyA proposal whose receive time lies within [ts − Precision, ts + MessageDelay·2^⌊round/10⌋ + Precision] of its timestamp ts.checkTimely in types/validation.go

Throughout, rs denotes the node's RoundState and state the committed sm.State (state as of height h − 1).

System model as enforced by the code

  • Single writer. State.receiveRoutine (internal/consensus/state.go) is the only goroutine that mutates RoundState. It selects over five inputs: transactions becoming available in the mempool (txNotifier.TxsAvailable()), messages (msgInfoQueue.read(): the node's own proposal/votes on an internal queue, peers' messages on per-peer lanes), timer expiries (timeoutTicker.Chan()), proposer-silence reports (proposerLiveness.silent()) and signed round skip votes (roundSkipSigner.signed()). Go's select picks randomly among ready cases, so there is no ordering guarantee between a timeout and a message that are both ready.
  • Actions and guards. Every transition is an ActionHandler registered in the Controller (internal/consensus/state_controller.go) and invoked via Controller.Dispatch. Each Enter* action begins with a staleness guard on (height, round, step); a dispatch that fails its guard is a silent no-op. Dispatches nest: an action may dispatch further actions before returning.
  • Copy-on-read state. Handlers work on a StateData copy taken from StateDataStore and persist it with StateData.Save / StateDataStore.Update, which checks an optimistic version counter (StateDataStore.update). A version mismatch is only logged.
  • Equal voting power. Every validator has power 100; quorum arithmetic is therefore a member count × 100.
  • Threshold signatures. Validators sign with BLS shares of the quorum key. A commit is one recovered threshold signature, verifiable against ValidatorSet.ThresholdPublicKey alone. See Signing.
  • Same-block execution. Header.AppHash is the application state after executing the block's own transactions; consensus validates it against the ProcessProposal result before voting (BlockExecutor.ValidateBlockWithRoundState in internal/state/execution.go).
  • Safety assumption. Safety follows the Tendermint argument only while T ≥ ⌊2·total/3⌋ + 1. The code does not enforce this: a lower VotingPowerThreshold (or an LLMQ type with a 50 % threshold) is accepted with a warning (ValidatorSet.BelowStrictThreshold).
  • Time. The block timestamp is the proposer's local clock (State.makeBlockTime in internal/state/state.go; genesis time for the initial height). Receivers judge timeliness with SynchronyParams (Precision default 505 ms, MessageDelay default 12 s, types/params.go). There is no median/BFT time.

Round state

RoundState (internal/consensus/types/round_state.go) holds, per node:

FieldMeaning
Height, Round, Step, StartTimePosition in the protocol; StartTime is when round 0 of this height may start (= CommitTime of the previous height).
CommitTime, CommitRoundWhen and in which round +2/3 precommits for a block were observed locally; CommitRound = -1 until then.
Validators, LastValidators, ProposerSelectorValidator set for this height, for the previous height, and the proposer-selection strategy.
Proposal, ProposalReceiveTime, ProposalBlock, ProposalBlockPartsThe accepted proposal of the current round, when it was received, and the block being assembled from parts.
LockedRound, LockedBlock, LockedBlockPartsLock.
ValidRound, ValidBlock, ValidBlockParts, ValidBlockRecvTimeValid block. ValidBlockRecvTime is the receive time used when re-proposing.
CommitParked commit (see Commit handling).
VotesHeightVoteSet: prevote and precommit VoteSets for rounds 0..Round+1 plus up to two catch-up rounds per validator.
LastCommitThe commit of height h − 1, included in the next proposed block. There is no LastCommit vote set: late precommits for h − 1 are dropped.
TriggeredTimeoutPrecommitWhether the precommit-wait timeout was already scheduled for this round.
RoundSkipRound skip certificate for the highest skipped round of this height; kept for peers still in that round even after the node moved past it.
TxsReportedWhether the mempool reported transactions at this height (it does so once per height).
ProposeNextBlockImmediatelyHint from ResponseFinalizeBlock.propose_next_block_immediately of the previous height; consumed once, not persisted.
CurrentRoundState, ProcessedBlockIDLatest application result for ProposalBlock (see Terminology).

StateData.updateToState re-initialises Height, Validators, LastValidators, LastCommit, ProposerSelector and StartTime for the new height and resets every proposal, lock, valid-block, commit, round-skip, vote and flag field. CurrentRoundState/ProcessedBlockID are left untouched; they are compared against the new height's block by processedFor. Within a height LockedRound/Block and ValidRound/Block are never reset; Commit is cleared only when a parked commit is rejected (selectParkedCommit, TryAddCommitAction.Execute).

Round steps

RoundStepType is an ordered enum (internal/consensus/types/round_state.go):

ValueStepEntered by
1NewHeightStateData.updateToState (after a block is applied, after block sync, at start)
2NewRoundEnterNewRoundAction
3ProposeEnterProposeAction
4PrevoteEnterPrevoteAction
5PrevoteWaitEnterPrevoteWaitAction
6PrecommitEnterPrecommitAction
7PrecommitWaitnever assigned — EnterPrecommitWaitAction sets TriggeredTimeoutPrecommit but leaves Step at Precommit (see Known issues). The value is still used as the timeout's step and in peer-state comparisons.
8ApplyCommitEnterCommitAction, AddCommitAction
stateDiagram-v2
    direction TB
    [*] --> NewHeight : updateToState (height h, round 0)
    NewHeight --> NewRound : tock NewHeight / EnterNewRound(h,0)
    NewRound --> NewRound : proposer waits for clock > LastBlockTime (NewRound tock rescheduled)
    NewRound --> Propose : EnterPropose (immediately, or after txs available / empty-block interval / proposal or prevotes while waiting)
    Propose --> Prevote : proposal + block (+ POL) complete, or tock Propose
    Prevote --> PrevoteWait : prevotes reach +2/3 any without a usable majority
    Prevote --> Precommit : polka (block held or nil)
    PrevoteWait --> Precommit : polka, or tock PrevoteWait
    Precommit --> Precommit : precommits reach +2/3 any (TriggeredTimeoutPrecommit, PrecommitWait tock scheduled)
    Precommit --> NewRound : tock PrecommitWait / EnterNewRound(h, r+1)
    Precommit --> ApplyCommit : +2/3 precommits for a block (threshold signature recovered)
    state "any step" as Any
    Any --> NewRound : +2/3 any prevotes or precommits at round r' > r (round skip)
    Any --> NewRound : RoundSkipCertificate for round r' ≥ r, before ApplyCommit / EnterNewRound(h, r'+1)
    Any --> ApplyCommit : verified CommitMessage for height h
    ApplyCommit --> NewHeight : block applied (height h+1)
    ApplyCommit --> ApplyCommit : waiting for block parts, or application rejected own commit

The precise guards of every arrow are in Transitions and decision rules.

Invariants

The following hold by construction of the code; each names the function that preserves it.

  1. Lock monotonicity. LockedRound only increases within a height. It is written by StateData.updateLockedBlock (first lock) and by the relock branch of EnterPrecommitAction.Execute (LockedRound = round), both at the current round, and reset to -1 only by StateData.updateToState. There is no unlock: a nil polka at a later round does not clear the lock (TestStateLock_POLDoesNotUnlock in internal/consensus/state_test.go).
  2. Lock implies polka. A node locks on block B at round r only after Prevotes(r).TwoThirdsMajority() == B and only when B is this round's proposal block (EnterPrecommitAction.Execute).
  3. Precommit implies processed. A non-nil precommit for B at round r is signed only after ProcessProposal(B, r) succeeded and ValidateBlockWithRoundState passed for this round (blockExecutor.mustEnsureProcess, blockExecutor.mustValidate called from EnterPrecommitAction.Execute). Consequently ExtendVote is asked for the round the application processed last.
  4. Valid-round monotonicity. ValidRound is set to the current round by StateData.updateValidBlock; callers only invoke it when ValidRound < Round (addVoteUpdateValidBlockMw, ProposalCompletedAction) or from the lock branch at the current round.
  5. One proposal slot per round. Proposaler.Set ignores any proposal while rs.Proposal != nil, so the first proposal that verifies holds the slot. The slot is freed within the round only by dropStaleProposal (StateData.retargetTo), when authenticated votes name another block: a current-round polka (addVoteUpdateValidBlockMw), a polka at precommit time for a block the node does not hold (EnterPrecommitAction.Execute), or a commit (StateData.adoptCommit, proposalUpdater.updateStateData). A proposal for the retargeted block — and only that block, since Set rejects a BlockID that differs from the assembled block's — can then be accepted in the same round. A proposal that fails verification never occupies the slot.
  6. Application result currency. CurrentRoundState describes the application's latest successful ProcessProposal/PrepareProposal for ProposalBlock; a result for another round or block is never reused (blockExecutor.ensureProcess, processedFor, ensureOwnRound).
  7. Commit = recovered threshold signature. VoteSet.MakeCommit panics unless maj23 is set and thresholdBlockSig was recovered; for precommit sets maj23 is set only after recovery and verification succeed (VoteSet.addVerifiedVote, VoteSet.recoverThresholdSignsAndVerify).
  8. Commit matches held block. A commit is applied only when its BlockID (hash, part-set header and state id) equals that of the block held (verifyCommitBlock in internal/consensus/state_try_add_commit.go, called by both prepareCommitForApply and ApplyCommitAction.Execute).
  9. WAL before signature. The WAL is flushed and fsynced before the node signs a vote: voteSigner.signVote returns on a FlushAndSync failure and no vote is signed. On the proposal path the sync is best-effort: EnterProposeAction.Execute logs a FlushAndSync failure and still calls Proposaler.Create, so a proposal may be signed over an unsynced WAL. EndHeightMessage is fsynced after the block is saved and before FinalizeBlock (ApplyCommitAction.Execute).
  10. Height votes only. Votes and commits for any height other than rs.Height are dropped (addVoteValidateVoteMw, StateData.readyToApplyCommit).

Threshold recovery

VoteSet.addVerifiedVote (types/vote_set.go) sets maj23 for a block id when the power behind it reaches T. For precommit sets it then calls recoverThresholdSignsAndVerify:

  1. With a single validator the vote's own signatures are used.
  2. Otherwise canonicalVoteExtensionGroup picks the group of votes that carry an identical ordered vector of threshold-recoverable extensions and whose power is ≥ R (ties broken by lexicographic order of the group identity, empty vector first).
  3. NewSignsRecoverer(...).Recover() recovers the block threshold signature from all votes for the block and the extension threshold signatures from the canonical group's votes.
  4. The recovered signatures are verified against ValidatorSet.ThresholdPublicKey over the canonical vote's sign data.

If any step fails, maj23 and the recovered signatures are rolled back and the attempt is repeated when a later vote makes the block cross T again or makes an extension group cross R. Externally, therefore, a precommit majority is observable only once a verifiable commit can be built from it. Prevote sets set maj23 at T without recovery.

Timeouts and consensus parameters

Timeouts come from ConsensusParams.Timeout of the committed state (changes proposed in block h apply from h + 1, CurrentRoundState in internal/state/current_round_state.go), with zero fields replaced by defaults (TimeoutParams.TimeoutParamsOrDefaults, types/params.go):

ParameterDefaultUsed for
Propose3000 msproposeTimeout(r) = Propose + ProposeDelta·r
ProposeDelta500 ms
Vote1000 msvoteTimeout(r) = Vote + VoteDelta·r (prevote wait, precommit wait, and the signing deadline of a non-nil precommit)
VoteDelta500 ms
Synchrony.Precision505 msTimeliness window
Synchrony.MessageDelay12 sTimeliness window; doubled every 10 rounds
ProposerHeartbeat1 s, derived when only ProposerSilence is setHeartbeat interval of the proposer of a waiting round (CONSENSUS_VERSION_2)
ProposerSilence5 s, derived when only ProposerHeartbeat is setProposerSilenceTimeout(r) = ProposerSilence + ProposerHeartbeat·r; see Parameters

Per-node Unsafe*TimeoutOverride settings in config.ConsensusConfig replace the corresponding parameter locally (StateData.proposeTimeout, StateData.voteTimeout); they desynchronise timing between nodes and are not part of the protocol. There is no commit timeout: StartTime of the next height equals CommitTime, so round 0 starts as soon as the block is applied (StateData.updateToState, roundScheduler.ScheduleRound0). The unsafe-commit-timeout-override / unsafe-bypass-commit-timeout-override settings are deprecated no-ops that only produce a warning (config.ConsensusConfig).

Escalation with the round is linear, not exponential.

Primary flow of a height

This page follows one height h through round 0 when every validator is honest, the network is timely and the application accepts everything. Every step names the function that performs it. Alternative paths branch off at the points marked ⇢ and are specified in Alternative flows and Commit handling.

Sequence

sequenceDiagram
    participant T as timeoutTicker
    participant P as Proposer (validator)
    participant V as Validator (non-proposer)
    participant A as Application (ABCI)
    participant S as Block/State store

    Note over P,V: updateToState: Height=h, Round=0, Step=NewHeight, StartTime=CommitTime
    T->>P: tock NewHeight(h)
    T->>V: tock NewHeight(h)
    P->>P: EnterNewRound(h,0) → EnterPropose(h,0)
    V->>V: EnterNewRound(h,0) → EnterPropose(h,0), schedule Propose timeout
    P->>A: PrepareProposal(txs, LocalLastCommit, misbehavior, round=0, ...)
    A-->>P: TxRecords, AppHash, TxResults, validator/param/chain-lock updates
    P->>P: sign Proposal (fresh block, POLRound=-1), send Proposal + BlockParts to own queue
    P-->>V: ProposalMessage, BlockPartMessages (gossip)
    V->>V: Proposaler.Set (verify proposer signature), AddProposalBlockPart
    V->>V: ProposalCompleted → EnterPrevote(h,0)
    V->>V: ValidateBlock against committed state
    V->>A: ProcessProposal(block, round=0)
    A-->>V: ACCEPT + AppHash/TxResults/...
    V->>V: ValidateBlockWithRoundState, sign PREVOTE(B)
    V-->>P: VoteMessage(prevote)
    Note over P,V: prevotes for B reach T → polka
    V->>V: EnterPrecommit(h,0): lock on B
    V->>A: ExtendVote(B, h, 0)
    A-->>V: vote extensions
    V->>V: sign PRECOMMIT(B, extensions)
    V-->>P: VoteMessage(precommit)
    P->>A: VerifyVoteExtension(peer precommit)
    A-->>P: ACCEPT
    Note over P,V: precommits for B reach T, threshold signature recovered and verified → maj23
    V->>V: EnterCommit(h,0) → Step=ApplyCommit → TryFinalizeCommit
    V->>V: seenCommit = Precommits(0).MakeCommit()
    V->>A: VerifyVoteExtension(canonical commit vote, empty pro_tx_hash)
    A-->>V: ACCEPT
    V->>S: SaveBlock(block, parts, commit), WAL EndHeight(h) fsync
    V->>A: FinalizeBlock(block, commit)
    A-->>V: ResponseFinalizeBlock
    V->>S: save state, mempool update
    V->>V: updateToState: Height=h+1, Round=0, Step=NewHeight, ScheduleRound0
    V->>V: EnterNewRound(h+1,0) dispatched immediately

The proposer runs the same validator steps for its own proposal: its ProposalMessage and BlockPartMessages travel through the internal queue and are handled exactly like a peer's (Proposaler.sendMessages, msgInfoDispatcher).

Step by step

1. New height

Precondition: a block for h − 1 was just applied, or the node started.

StateData.updateToState (internal/consensus/state_data.go) sets Height = h, Round = 0, Step = NewHeight, LastCommit, a fresh HeightVoteSet, a fresh ProposerSelector (from ConsensusParams.Version and the block store), resets proposal/lock/valid/commit fields, and sets StartTime = CommitTime (or now when there is no commit time yet). roundScheduler.ScheduleRound0 arms a NewHeight-step timeout of StartTime − now (≤ 0, so it fires at once).

On the vote-driven path the next height does not even wait for that tock: addVoteDispatchPrecommitMw (internal/consensus/state_add_vote.go) dispatches EnterNewRoundEvent{Height: stateData.Height} right after EnterCommit returned, and because ApplyCommit already incremented the height in place, that is EnterNewRound(h + 1, 0). The later NewHeight tock then fails the EnterNewRound guard and is ignored.

2. New round

State.handleTimeout (internal/consensus/state.go) maps the NewHeight tock to EnterNewRoundEvent{Height}. EnterNewRoundAction.Execute (internal/consensus/state_enter_new_round.go):

  • guard: Height == h, round ≥ Round, and if round == Round then Step == NewHeight;
  • updateRoundStep(0, NewRound) (this also updates the proposer selector);
  • for round 0 the proposal fields are kept as reset by updateToState (a proposal for round 0 may already have arrived);
  • Votes.SetRound(1) so that round 1 votes are tracked for round skipping;
  • TriggeredTimeoutPrecommit = false;
  • decides whether to wait for transactions: waitForTxs = config.WaitForTxs() && (round == 0 || waitsAfterRoundSkip(round)) && Height != InitialHeight, overridden to false by ProposeNextBlockImmediately and, on CONSENSUS_VERSION_2 chains, by a proposal already held for the round (roundSkipping && Proposal != nil). ⇢ Empty blocks. With the default configuration (create-empty-blocks = true, interval 0) it does not wait and dispatches EnterProposeEvent{h, 0}. On CONSENSUS_VERSION_2 chains the proposer of a waiting round sends heartbeats while it waits. ⇢ Proposer liveness.

3. Propose

EnterProposeAction.Execute (internal/consensus/state_enter_propose.go):

  • guard: Height == h, round ≥ Round, and if round == Round then Step < Propose;
  • isProposer = ProposerSelector.GetProposer(Height, Round).ProTxHash == own pro-tx-hash (Proposer selection);
  • if the node is the proposer and state.LastBlockTime > now, it reschedules a NewRound-step timeout for the difference and returns without changing step (block times must increase; proposerWaitTime in internal/consensus/state.go). ⇢ Known issues;
  • schedules the Propose timeout proposeTimeout(round);
  • non-proposers return here and wait for the proposal;
  • the proposer, unless replaying the WAL, applying a parked commit, or suppressed after block sync (catchupTracker.mayPropose), flushes the WAL and calls Proposaler.Create;
  • on return (deferred): updateRoundStep(round, Propose), publish the step, and if isProposalComplete() already holds, dispatch EnterPrevote.

3a. Creating the proposal (proposer only)

Proposaler.Create (internal/consensus/state_proposaler.go):

  1. If checkValidBlock(rs) — ValidBlock != nil and it is timely with respect to ValidBlockRecvTime and ValidRound (or has the genesis time at InitialHeight) — reuse ValidBlock; otherwise build a new block via blockExecutor.create → BlockExecutor.CreateProposalBlock (internal/state/execution.go):
    • commit = LastCommit (an empty commit at InitialHeight);
    • evidence = evpool.PendingEvidence(ConsensusParams.Evidence.MaxBytes);
    • maxDataBytes = types.MaxDataBytes(Block.MaxBytes, commit, evidenceSize) (block overhead, max header, max core chain lock, the last commit's actual size and evidence are subtracted; types/block.go);
    • txs = mempool.ReapMaxBytesMaxGas(maxDataBytes, Block.MaxGas);
    • state.MakeBlock(...) with Time = now (genesis time at InitialHeight), ProposerProTxHash = own, ProposedAppVersion;
    • ABCI PrepareProposal with MaxTxBytes, Txs, LocalLastCommit, Misbehavior, Height, Round, Time, NextValidatorsHash, CoreChainLockedHeight = state.LastCoreChainLockedBlockHeight, ProposerProTxHash, ProposedAppVersion, Version, QuorumHash. An error from the call panics the node (blockExecutor.create); a response that differs for an identical request is only logged (deterministicPrepareProposalHashes);
    • the response is validated; removed transactions are evicted from the mempool; AppHash, ResultsHash, NextValidatorsHash, NextConsensusHash, CoreChainLock, AppVersion and the next validator set/params are written into the header and CurrentRoundState (RoundParamsFromPrepareProposal, CurrentRoundState.UpdateBlock).
  2. makeProposal(h, round, polRound, block, parts), where polRound = ValidRound when ValidBlock is re-proposed and polRound = -1 when a new block was built. A new block has no polka, so it never claims one.
  3. signProposal: PrivValidator.SignProposal(chainID, QuorumType, QuorumHash, proposal) with a context deadline of Timeout.Propose. The proposal signature covers Type, Height, Round, POLRound, BlockID (hash + part-set header), Timestamp, ChainID (CanonicalizeProposal in types/canonical.go); it does not cover CoreChainLockedHeight or BlockID.StateID.
  4. sendMessages: ProposalMessage then one BlockPartMessage per part (BlockPartSizeBytes), all to the internal queue. The reactor gossips them to peers from the resulting round-state events.

4. Receiving the proposal and block

proposalMessageHandler → Proposaler.Set (internal/consensus/state_proposaler.go) accepts the proposal when it is the first for the round, is for the current (Height, Round), has a valid POLRound, a core-chain-locked height not below the committed one, agrees with any block/part set already held, and carries a valid proposer signature (verifyProposal; full rules in Proposal validity). It records ProposalReceiveTime (the queue's enqueue time, chanMsgSender.send in internal/consensus/msg_queue.go) and creates the part set from BlockID.PartSetHeader unless one is under way.

blockPartMessageHandler → AddProposalBlockPartAction.Execute (internal/consensus/state_add_prop_block.go) adds each part (Merkle proof checked by PartSet.AddPart). When the last part arrives the block is decoded, cross-checked against the proposal (hash and part-set header must match, else the proposal is dropped), the proposal's CoreChainLockedHeight and BlockID.StateID are replaced by the values derived from the block, rs.ProposalBlock is set and ProposalCompletedEvent is dispatched.

ProposalCompletedAction.Execute: with Step ≤ Propose and isProposalComplete() (proposal and block held, and for POLRound ≥ 0 the POL round's prevotes already have a majority) → EnterPrevote(h, Round).

5. Prevote

EnterPrevoteAction.Execute (internal/consensus/state_enter_prevote.go) → prevoter.Do (internal/consensus/state_prevoter.go):

  1. StateData.isValidForPrevote: block and proposal held; proposal timestamp equals block time; the proposal is timely (unless the block is already ValidBlock; in replay, only a proposal received too early fails); the core chain lock is consistent (sm.ValidateBlockChainLock); for POLRound == -1 the block's ProposerProTxHash is this round's proposer.
  2. blockExecutor.ensureProcess(rs, Round) → ValidateBlock against the committed state, core chain lock included, then ABCI ProcessProposal (verify = true); a block failing ValidateBlock never reaches the application. The response must be ACCEPT, well-formed, and ValidateBlockWithRoundState must pass — header fields AppHash, ResultsHash, NextValidatorsHash, NextConsensusHash must equal the application's answer, and LastCommit must verify against LastValidators. The result becomes CurrentRoundState tagged with round 0.
  3. blockExecutor.mustValidate (panics on failure, cannot fail after step 2).
  4. prevoter.signVote: with no lock and POLRound == -1, checkProposalBlock is true → voteSigner.signAddVote(PrevoteType, BlockID).

Step Prevote is set on return. A failure in step 1, or ErrBlockRejected (application REJECT) or ErrInvalidBlock (ValidateBlock before the call, ValidateBlockWithRoundState after it) in step 2, yields a nil prevote instead. Any other step-2 error — transport failure, UNKNOWN status or a malformed response — panics the node (prevoter.handleError). ⇢ Prevote rule.

6. Collecting prevotes

Each VoteMessage runs the prevote middleware chain (newAddVoteAction in internal/consensus/state_add_vote.go). Once Prevotes(0) holds T power for B:

  • addVoteUpdateValidBlockMw: ValidRound < 0 == Round and the held block is B → updateValidBlock() (ValidRound = 0, ValidBlock = B, ValidBlockRecvTime = ProposalReceiveTime), retargetTo(B) (a no-op here), publish a ValidBlock event (peers learn the part-set header);
  • addVoteDispatchPrevoteMw: Round == vote.Round, Step ≥ Prevote, majority for B and isProposalComplete() → EnterPrecommit(h, 0).

7. Precommit

EnterPrecommitAction.Execute (internal/consensus/state_enter_precommit.go): polka for B; proposal and block held; timestamps equal; B is this round's proposal block → mustEnsureProcess(rs, 0) (no new ABCI call: round 0 is already processed), mustValidate, updateLockedBlock() (LockedRound = 0, LockedBlock = B), Lock event, signAddVote(PrecommitType, B), then updateValidBlock() (already current).

voteSigner.signVote for a non-nil precommit first calls ABCI ExtendVote (BlockExecutor.ExtendVote; an error or an invalid extension panics), attaches the returned extensions, and signs with deadline voteTimeout(0). The vote carries one BLS share for the block sign item and one per extension (Signing).

8. Collecting precommits

The precommit middleware chain adds addVoteVerifyVoteExtensionMw before the vote set: for a peer's non-nil precommit it verifies the block signature and every extension signature (types.VerifyVoteSignatures), restores the node's own processed round if a commit check changed it (blockExecutor.ensureOwnRound), and asks ABCI VerifyVoteExtension; a rejection drops the vote (it is not stored, so gossip may redeliver it).

When Precommits(0) for B reaches T and threshold recovery succeeds (Threshold recovery), addVoteDispatchPrecommitMw dispatches EnterNewRound(0) (no-op), EnterPrecommit(0) (no-op) and EnterCommitEvent{h, CommitRound: 0}.

9. Commit and apply

EnterCommitAction.Execute (internal/consensus/state_enter_commit.go): guard Height == h, Step < ApplyCommit; proposalUpdater.updateStateData confirms the held block is B; deferred: Step = ApplyCommit (round kept), CommitRound = 0, CommitTime = now, then TryFinalizeCommitEvent.

TryFinalizeCommitAction.Execute → finalizeCommit (internal/consensus/state_try_finalize_commit.go): seenCommit = Precommits(CommitRound).MakeCommit() (height, round, block id, quorum hash, recovered block signature, recovered extension signatures); refuseUnprocessedCommit (cache of rejected vectors, empty here); mustEnsureProcess(rs, seenCommit.Round); verifyCommitExtensions → ABCI VerifyVoteExtension with the commit's canonical vote (empty ValidatorProTxHash marks it as a commit, sm.VerifyCommitExtensions). On acceptance → ApplyCommitEvent{seenCommit}.

ApplyCommitAction.Execute (internal/consensus/state_apply_commit.go):

  1. verifyCommitBlock: part-set header, hash and state id of the commit equal those of the held block.
  2. mustEnsureProcess(rs, commit.Round), mustValidate.
  3. blockStore.SaveBlock(block, parts, commit).
  4. WAL WriteSync(EndHeightMessage{h}) — panics on failure.
  5. blockExecutor.finalize → BlockExecutor.FinalizeBlock (internal/state/execution.go): ValidateBlockWithRoundState again, SaveABCIResponses (the ProcessProposal result, for crash recovery), ABCI FinalizeBlock (block, commit, round, block id), State.Update (Validators ← NextValidators, LastValidators ← Validators, params, chain-lock height, LastAppHash), mempool flush and update (flushMempool, recheck per ABCI.RecheckTx), evidence-pool update, stateStore.Save, pruning per RetainHeight, events. An error panics the node.
  6. updateToState(newState, commit) → height h + 1, step NewHeight.
  7. ProposeNextBlockImmediately is set if the response asks for it.
  8. ScheduleRound0.

Control then returns through the dispatch stack to addVoteDispatchPrecommitMw, which dispatches EnterNewRound(h + 1, 0) immediately (step 1 above).

ABCI calls per height on the happy path

Node roleCalls
ProposerPrepareProposal, ProcessProposal, ExtendVote, VerifyVoteExtension × (n − 1 peer precommits) + 1 (commit), FinalizeBlock
Other validatorProcessProposal, ExtendVote, VerifyVoteExtension × (n − 1) + 1, FinalizeBlock
Observer (not in the validator set)ProcessProposal (once a CommitMessage and the block arrive; peers gossip proposals only to validators), VerifyVoteExtension × 1 (commit), FinalizeBlock — observers neither prevote nor precommit, see Non-validators

There is no ABCI Commit call: the method is removed from the Tenderdash ABCI protocol (proto/tendermint/abci/types.proto reserves its field number). Several ProcessProposal calls per height are possible whenever a later round, a commit for another round, or a block-sync handover requires re-processing (blockExecutor.ensureProcess).

Transitions and decision rules

This page is the formal core of the specification: for every action of the state machine it gives the precondition (guard), the actions and the postcondition, then the decision procedures that choose what to sign. Dispatch(X) means Controller.Dispatch of event X; a dispatch whose guard fails is a no-op.

Notation: h = rs.Height, r = rs.Round, step = rs.Step; Prevotes(x) / Precommits(x) are rs.Votes.Prevotes(x) / rs.Votes.Precommits(x); maj23(S) is S.TwoThirdsMajority(); any23(S) is S.HasTwoThirdsAny(); complete() is StateData.isProposalComplete().

Inputs

InputHandlerResulting dispatch
Tock (H, R, NewHeight)State.handleTimeoutEnterNewRound{H} (round 0)
Tock (H, R, NewRound)State.handleTimeoutEnterPropose{H, R}
Tock (H, R, Propose)State.handleTimeoutpublish TimeoutPropose; EnterPrevote{H, R}
Tock (H, R, PrevoteWait)State.handleTimeoutpublish TimeoutWait; EnterPrecommit{H, R}
Tock (H, R, PrecommitWait)State.handleTimeoutpublish TimeoutWait; EnterPrecommit{H, R}; EnterNewRound{H, R + 1}
TxsAvailableState.handleTxsAvailablesets TxsReported; acts only at r == 0 or in a round that may wait after a round skip (mayWaitAfterRoundSkip): at step NewHeight (and h ≠ InitialHeight) schedule tock (h, 0, NewRound) after StartTime − now + 1 ms; at step NewRound dispatch EnterPropose{h, r}
ProposalMessageproposalMessageHandler → Proposaler.Setsee Proposal acceptance; if accepted while waitingForTxs, EnterPropose{h, r}
BlockPartMessageblockPartMessageHandlerAddProposalBlockPart; on completion ProposalCompleted or AddCommit
VoteMessagevoteMessageHandlerAddVote → vote-driven dispatch
CommitMessagecommitMessageHandlerTryAddCommit (Commit handling)
RoundSkipCertificateMessageroundSkipCertificateMessageHandlerAddRoundSkipCertificate
ProposerHeartbeatMessage, RoundSkipVoteMessageState.handleLivenessMessagenone, or AddRoundSkipCertificate for a certificate the vote completed (Proposer liveness)
Proposer-silence reportState.handleProposerSilencenone; signs a round skip vote off the consensus goroutine
Signed round skip voteState.handleSignedRoundSkipVotenone, or AddRoundSkipCertificate for a certificate the vote completed

Every tock is written to the WAL before handling; handleTimeout drops a tock when ti.Height ≠ h, ti.Round < r, or ti.Round == r && ti.Step < step (internal/consensus/state.go).

Timer semantics

timeoutTicker (internal/consensus/ticker.go) owns one timer. ScheduleTimeout(ti) replaces the pending timer unless ti is older than the last scheduled (Height, Round, Step): a tick is ignored when its height is lower, or its round is lower at the same height, or at the same height and round its step is ≤ the pending step (and the pending step is non-zero). Consequences:

  • a PrevoteWait tick supersedes a still-pending Propose tick for the same round, so the propose timeout never fires once a prevote wait started;
  • a tick for the same step as the pending one at the same (h, r) is dropped even if the pending one already fired, because the ticker remembers the last ti (see Known issues);
  • durations may be zero or negative; such ticks fire immediately.

The receive routine handles the tock only if it still matches (h, r, step) as described above.

Step actions

EnterNewRound(H, R)

EnterNewRoundAction.Execute (internal/consensus/state_enter_new_round.go).

  • Guard: h == H ∧ R ≥ r ∧ (R == r ⇒ step == NewHeight).
  • Actions:
    1. updateRoundStep(R, NewRound) (also ProposerSelector.UpdateHeightRound(h, R)).
    2. If R > 0: Proposal = nil, ProposalReceiveTime = 0; if rs.Commit ≠ nil then retargetTo(Commit.BlockID) (keep collecting the committed block), else ProposalBlock = nil, ProposalBlockParts = nil. LockedBlock, ValidBlock and Commit are kept.
    3. Votes.SetRound(R + 1); TriggeredTimeoutPrecommit = false; save.
    4. Publish NewRound; if a commit is parked also publish the step and a ValidBlock event so peers keep serving the committed block's parts.
    5. waitForTxs = config.WaitForTxs() ∧ (R == 0 ∨ waitsAfterRoundSkip(R)) ∧ h ≠ InitialHeight ∧ ¬ProposeNextBlockImmediately ∧ ¬(roundSkipping ∧ Proposal ≠ nil) where WaitForTxs = ¬CreateEmptyBlocks ∨ CreateEmptyBlocksInterval > 0 (config/config.go); roundSkipping is defined in When it applies and waitsAfterRoundSkip in The round after a skip.
      • waitForTxs ∧ R > 0 ∧ Commit == nil: publish the NewRound step.
      • waitForTxs ∧ CreateEmptyBlocksInterval > 0: schedule tock (h, R, NewRound) after the interval (for R > 0, the interval minus now − StartTime, at least 0); stay in NewRound.
      • waitForTxs ∧ CreateEmptyBlocksInterval == 0: stay in NewRound until TxsAvailable.
      • otherwise (unless the test-only DontAutoPropose): Dispatch(EnterPropose{h, R}).
  • Postcondition: r = R, step = NewRound (or beyond, if EnterPropose ran synchronously).

EnterPropose(H, R)

EnterProposeAction.Execute (internal/consensus/state_enter_propose.go).

  • Guard: h == H ∧ R ≥ r ∧ (R == r ⇒ step < Propose).
  • Actions:
    1. isProposer ← ProposerSelector.GetProposer(h, r) == own pro-tx-hash; applyingCommit ← Commit ≠ nil ∧ holdsProposalBlock(Commit.BlockID).
    2. If isProposer ∧ ¬applyingCommit ∧ state.LastBlockTime > now: schedule tock (H, R, NewRound) after LastBlockTime − now and return without changing step (the guard admits the retry because step is still NewRound).
    3. Schedule tock (H, R, Propose) after proposeTimeout(R).
    4. If ¬isProposer, or applyingCommit, or replay mode, or ¬catchup.mayPropose(h): do nothing more.
    5. Else wal.FlushAndSync() (a failure is only logged); Proposaler.Create(h, R, rs) (happy path §3a). A creation or signing error is logged and the round proceeds as if the proposer were silent.
    6. Deferred: updateRoundStep(R, Propose), publish step; if complete() then Dispatch(EnterPrevote{H, R}).
  • Postcondition: step = Propose, propose timeout armed (except in the clock-wait case of action 2).

Proposal acceptance (Proposaler.Set)

Proposaler.Set (internal/consensus/state_proposaler.go) runs for every ProposalMessage (own or peer's). The message must already pass Proposal.ValidateBasic (types/proposal.go: type, non-negative height and round, POLRound ≥ -1, complete BlockID, CoreChainLockedHeight ≠ MaxUint32, signature present).

flowchart TD
    A[ProposalMessage p] --> B{rs.Proposal != nil?}
    B -- yes --> IGN[ignore]
    B -- no --> C{p.Height == h and p.Round == r?}
    C -- no --> IGN
    C -- yes --> D{parked Commit whose part set is under way and p.BlockID != Commit.BlockID?}
    D -- yes --> IGN
    D -- no --> E{p.POLRound == -1 or 0 <= p.POLRound < p.Round?}
    E -- no --> ERR1[ErrInvalidProposalPOLRound]
    E -- yes --> F{p.CoreChainLockedHeight >= state.LastCoreChainLockedBlockHeight?}
    F -- no --> ERR2[ErrInvalidProposalCoreHeight]
    F -- yes --> G{block already assembled?}
    G -- yes --> H{BlockID and CoreChainLockedHeight equal the block's?}
    H -- no --> ERR3[ErrInvalidProposalBlockID / ErrInvalidProposalCoreHeight]
    H -- yes --> I
    G -- no --> I{roundStateRefusal: commit for same h,r names another block? part set under way has another header?}
    I -- yes --> ERR4[ErrInvalidProposalForCommit / ErrInvalidProposalForPartSet]
    I -- no --> J{proposer public key known?}
    J -- no --> K{parked commit for same h,r with same BlockID?}
    K -- no --> ERR5[ErrUnableToVerifyProposal]
    K -- yes --> OK
    J -- yes --> L{verification budget allows one signature check?}
    L -- no --> ERR6[ErrVerificationBudgetExhausted - local drop]
    L -- yes --> M{proposer.PubKey verifies ProposalBlockSignID?}
    M -- no --> ERR7[ErrInvalidProposalSignature]
    M -- yes --> OK[rs.Proposal = p; ProposalReceiveTime = enqueue time; part set created if none]

Properties:

  • The proposer is ProposerSelector.GetProposer(h, r); the signature is checked against that validator's key for the committed state's QuorumType/QuorumHash (types.ProposalBlockSignID).
  • A proposal that fails is not stored, so a later valid one for the same round can still be accepted; a proposal that fails the signature check is verified again every time it is resent (bounded by the verification budget).
  • Errors are returned to the message dispatcher; the sender is not evicted for any of them.
  • Equivocating proposals: the first valid one is kept; later ones are ignored while it occupies rs.Proposal. The slot is freed within the round only when authenticated votes retarget the round to another block (StateData.retargetTo → dropStaleProposal: current-round polka, precommit-time polka for a block not held, parked or applied commit); a proposal for that block can then be accepted (invariant 5).

AddProposalBlockPart / ProposalCompleted

AddProposalBlockPartAction.Execute, ProposalCompletedAction.Execute (internal/consensus/state_add_prop_block.go).

  • Guard (part): msg.Height == h; ProposalBlockParts ≠ nil; the peer's invalid-proof budget allows it (blockPartProofBudget). The message's round is not checked: parts of the same block from another round are accepted.
  • Actions (part): ProposalBlockParts.AddPart(part) (Merkle proof against the part-set header; ErrPartSetInvalidProof charges the sender's budget); if the accumulated size exceeds ConsensusParams.Block.MaxBytes → error. When complete: decode the block; if rs.Proposal ≠ nil and the block's hash or part-set header differ from the proposal's → rs.Proposal = nil (the block is kept); otherwise the proposal's CoreChainLockedHeight and BlockID.StateID are replaced by the block's; rs.ProposalBlock = block. If a commit is parked → selectParkedCommit (Commit handling).
  • ProposalCompleted:
    1. If maj23(Prevotes(r)) is a non-nil block id B, ValidRound < r, and the held block is B → updateValidBlock().
    2. If step ≤ Propose ∧ complete() → Dispatch(EnterPrevote{h, r}); then if a polka exists → Dispatch(EnterPrecommit{h, r}).
    3. Else if step == ApplyCommit → Dispatch(TryFinalizeCommit{h}).
  • complete() ≡ Proposal ≠ nil ∧ ProposalBlock ≠ nil ∧ (Proposal.POLRound < 0 ∨ Prevotes(POLRound).HasTwoThirdsMajority()) (StateData.isProposalComplete).

EnterPrevote(H, R)

EnterPrevoteAction.Execute (internal/consensus/state_enter_prevote.go).

  • Guard: h == H ∧ R ≥ r ∧ (R == r ⇒ step < Prevote).
  • Actions: run the prevote decision and sign exactly one prevote (block or nil); deferred updateRoundStep(R, Prevote).
  • Postcondition: step = Prevote; one prevote for (h, R) is on the internal queue (if the node is a validator and signing succeeded).

EnterPrevoteWait(H, R)

EnterPrevoteWaitAction.Execute (internal/consensus/state_enter_wait.go).

  • Guard: h == H ∧ R ≥ r ∧ (R == r ⇒ step < PrevoteWait); panics unless any23(Prevotes(R)).
  • Actions: schedule tock (H, R, PrevoteWait) after voteTimeout(R); deferred updateRoundStep(R, PrevoteWait).

EnterPrecommit(H, R)

EnterPrecommitAction.Execute (internal/consensus/state_enter_precommit.go).

  • Guard: h == H ∧ R ≥ r ∧ (R == r ⇒ step < Precommit).
  • Actions: run the precommit decision and sign exactly one precommit; deferred updateRoundStep(R, Precommit).
  • Postcondition: step = Precommit; lock possibly updated (only to R).

EnterPrecommitWait(H, R)

EnterPrecommitWaitAction.Execute (internal/consensus/state_enter_wait.go).

  • Guard: h == H ∧ R ≥ r ∧ (R == r ⇒ ¬TriggeredTimeoutPrecommit); panics unless any23(Precommits(R)).
  • Actions: schedule tock (H, R, PrecommitWait) after voteTimeout(R); deferred TriggeredTimeoutPrecommit = true, publish step.
  • Postcondition: step unchanged (see Known issues).

EnterCommit(H, CR)

EnterCommitAction.Execute (internal/consensus/state_enter_commit.go).

  • Guard: h == H ∧ step < ApplyCommit; panics unless maj23(Precommits(CR)) exists.
  • Actions: proposalUpdater.updateStateData(blockID): if LockedBlock is the committed block, install it as ProposalBlock (replaceProposalBlockOnLockedBlock); if the block is not held, retargetTo(blockID) and publish ValidBlock (start fetching). Deferred: Step = ApplyCommit (round unchanged), CommitRound = CR, CommitTime = now, publish step, Dispatch(TryFinalizeCommit{h}).
  • Postcondition: step = ApplyCommit, CommitRound = CR.

AddRoundSkipCertificate(cert)

AddRoundSkipCertificateAction.Execute (internal/consensus/state_add_round_skip.go).

  • Guard: roundSkipping ∧ cert.Height == h ∧ (RoundSkip == nil ∨ cert.Round > RoundSkip.Round) ∧ step < ApplyCommit (roundSkipCertificateWanted).
  • Actions: verify the certificate unless this node recovered it (budget, quorum hash, threshold signature); write its message to the WAL (not on replay); RoundSkip = cert; broadcast it; if cert.Round ≥ r, Dispatch(EnterNewRound{h, cert.Round + 1}).
  • Postcondition: RoundSkip = cert; r > cert.Round, or r unchanged when the certificate is for an earlier round.

TryFinalizeCommit(H) / ApplyCommit(commit)

See Commit handling.

Vote-driven dispatch

AddVoteAction.Execute (internal/consensus/state_add_vote.go) runs a middleware chain. Order of execution for a prevote: validate → add to vote set → addVoteUpdateValidBlockMw → addVoteDispatchPrevoteMw. For a precommit: validate → addVoteVerifyVoteExtensionMw → add to vote set → addVoteDispatchPrecommitMw. Errors pass through addVoteErrorMw (evidence reporting).

Admission (addVoteValidateVoteMw, HeightVoteSet.addVote, VoteSet.addVote)

A vote is admitted to the set iff all hold:

  1. vote.Height == h; rs.Validators.HasPublicKeys; 0 ≤ vote.ValidatorIndex < |Validators|.
  2. The round's vote set exists, or (HeightVoteSet.checkVoteClaim passes and) the validator has opened fewer than 2 catch-up rounds (peerCatchupRounds); a provisional round is discarded if its first vote is not admitted.
  3. vote.ValidatorProTxHash equals the validator's at that index.
  4. Not a duplicate (same block key and identical block signature → added = false, no error); a different signature for the same block key → ErrVoteNonDeterministicSignature.
  5. The block signature (and, for non-nil precommits, every extension signature) verifies against the validator's share public key for (ChainID, QuorumType, QuorumHash) — Vote.Verify, or evidence from addVoteVerifyVoteExtensionMw that this was done (VoteVerification).
  6. For a non-nil precommit from a peer: ABCI VerifyVoteExtension returned ACCEPT (after blockExecutor.ensureOwnRound).

A vote by a validator who already voted for a different block id in the same round is recorded as conflicting: NewConflictingVoteError is returned and the later vote is kept only if a peer claimed +2/3 for its block id (VoteSet.addVerifiedVote, VoteSet.SetPeerMaj23). addVoteErrorMw reports the pair to the evidence pool (evidencePool.ReportConflictingVotes) unless it is the node's own key.

After a prevote is added

addVoteUpdateValidBlockMw, with P = Prevotes(vote.Round) and B = maj23(P) non-nil:

  • if ValidRound < vote.Round ∧ vote.Round == r: if the held block is B then updateValidBlock(); in all cases retargetTo(B) (drops a proposal for another block, replaces the part set unless its header is B's), save, publish ValidBlock.

addVoteDispatchPrevoteMw:

ConditionDispatch
r < vote.Round ∧ any23(P)EnterNewRound{h, vote.Round} (round skip)
r == vote.Round ∧ step ≥ Prevote ∧ maj23(P) = B ∧ (complete() ∨ B = nil)EnterPrecommit{h, r}
r == vote.Round ∧ step ≥ Prevote ∧ any23(P) (otherwise)EnterPrevoteWait{h, r}
Proposal ≠ nil ∧ Proposal.POLRound == vote.Round ≥ 0 ∧ complete() (and neither of the above)EnterPrevote{h, r} (POL votes completed a proposal with POLRound)
waitingForTxs ∧ (maj23(P) exists ∨ (vote.Round == r ∧ r > 0)) (and none of the above)EnterPropose{h, r} (ends a wait for transactions; CONSENSUS_VERSION_2)

After a precommit is added

addVoteDispatchPrecommitMw, with C = Precommits(vote.Round):

ConditionDispatch
¬maj23(C) ∧ r ≤ vote.Round ∧ any23(C)EnterNewRound{h, vote.Round}; EnterPrecommitWait{h, vote.Round}
maj23(C) = nilEnterNewRound{h, vote.Round}; EnterPrecommit{h, vote.Round}; EnterPrecommitWait{h, vote.Round}
maj23(C) = B ≠ nilEnterNewRound{h, vote.Round}; EnterPrecommit{h, vote.Round}; EnterCommit{h, vote.Round}; then EnterNewRound{Height: rs.Height} (which is h + 1 if the commit was applied synchronously)

A precommit majority at a round below the current one is honoured: the EnterNewRound/EnterPrecommit dispatches are no-ops and EnterCommit runs with CommitRound = vote.Round (TestCommitFromPreviousRound).

Prevote decision (prevoter.Do)

prevoter.Do (internal/consensus/state_prevoter.go) with StateData.isValidForPrevote (internal/consensus/state_data.go).

flowchart TD
    S([EnterPrevote h,r]) --> A{ProposalBlock held?}
    A -- no --> NIL[prevote nil]
    A -- yes --> B{Proposal held?}
    B -- no --> NIL
    B -- yes --> C{Proposal.Timestamp == block.Time?}
    C -- no --> NIL
    C -- yes --> D{block == ValidBlock?}
    D -- yes --> F
    D -- no --> E{timely? InitialHeight: timestamp == genesis time; else receive time within window for round r, in replay only too early fails}
    E -- no --> NIL
    E -- yes --> F{ValidateBlockChainLock ok?}
    F -- no --> NIL
    F -- yes --> G{POLRound == -1 implies block.ProposerProTxHash == proposer h,r?}
    G -- no --> NIL
    G -- yes --> H[ensureProcess for round r unless already processed]
    H --> V{ValidateBlock against committed state ok?}
    V -- no --> NIL
    V -- yes --> I{ABCI ProcessProposal: ACCEPT and header matches app result?}
    I -- REJECT or mismatch --> NIL
    I -- other error --> PANIC[panic]
    I -- yes --> J[mustValidate]
    J --> K{POLRound == -1?}
    K -- yes --> L{LockedRound == -1 or block == LockedBlock?}
    L -- yes --> VOTE[prevote block]
    L -- no --> NIL
    K -- no --> M{maj23 of Prevotes POLRound == block and 0 <= POLRound < r?}
    M -- no --> NIL
    M -- yes --> N{LockedRound <= POLRound or block == LockedBlock?}
    N -- yes --> VOTE
    N -- no --> NIL

Rules, in evaluation order:

  1. P-NoBlock / P-NoProposal. No block or no proposal held → nil (errPrevoteProposalBlockNil, errPrevoteProposalNil). This is the path taken on timeoutPropose. No ABCI call is made.
  2. P-Timestamp. Proposal.Timestamp ≠ ProposalBlock.Time → nil.
  3. P-Timely. Skipped when the block is already ValidBlock. At InitialHeight the timestamp must equal the genesis time; otherwise Proposal.CheckTimely(ProposalReceiveTime, SynchronyParams, r) must be 0 (checkTimely in types/validation.go): ts − Precision ≤ recv ≤ ts + MessageDelay·2^⌊r/10⌋ + Precision (the shift is capped to avoid overflow). In replay mode recv is the time of the proposal's WAL entry, which can lag its receipt, so only recv < ts − Precision (too early) fails.
  4. P-ChainLock. sm.ValidateBlockChainLock(state, block) (internal/state/validation.go): with a CoreChainLock in the block, Header.CoreChainLockedHeight == CoreChainLock.CoreBlockHeight and > state.LastCoreChainLockedBlockHeight; without one, Header.CoreChainLockedHeight == state.LastCoreChainLockedBlockHeight.
  5. P-Proposer. For POLRound == -1 the block header's ProposerProTxHash must be this round's proposer. Re-proposals (POLRound ≥ 0) keep the original header's proposer and are exempt.
  6. P-Process. blockExecutor.ensureProcess(rs, r): ABCI ProcessProposal for round r unless CurrentRoundState already is the result for this block and round. The block is first validated against the committed state (BlockExecutor.ValidateBlock: version, chain ID, height, LastBlockID, core chain lock, LastCommit, proposer, evidence, …); a block that fails is never sent to the application and yields ErrInvalidBlock. ErrBlockRejected (application REJECT) or ErrInvalidBlock (also a well-formed ACCEPT whose contents disagree with the block: header fields differ from the application's answer, LastCommit invalid — ValidateBlockWithRoundState) → nil prevote; any other error → panic (prevoter.handleError). A malformed application response is in the panic class, not the nil-prevote class: UNKNOWN status, an AppHash of the wrong size (ResponseProcessProposal.Validate), or tx_results not matching the transactions (validateExecTxResults) are returned by BlockExecutor.ProcessProposal as plain errors before the ErrInvalidBlock wrapper is reached. Note that ProcessProposal runs before the lock rules, so a locked validator still processes a different proposal and then prevotes nil.
  7. P-Lock-NoPOL (paper line 22–24). POLRound == -1: prevote the block iff LockedRound == -1 ∨ ProposalBlock.BlockID == LockedBlock.BlockID (prevoter.checkProposalBlock).
  8. P-Lock-POL (paper line 28–30). POLRound ≥ 0: prevote the block iff maj23(Prevotes(POLRound)) == ProposalBlock.BlockID ∧ 0 ≤ POLRound < r ∧ (LockedRound ≤ POLRound ∨ ProposalBlock.BlockID == LockedBlock.BlockID) (prevoter.checkPrevoteMaj23). Block ids compare all three fields, including StateID.
  9. Otherwise prevote nil.

EnterPrevote is reached either because the proposal is complete or because timeoutPropose fired. A proposal with POLRound ≥ 0 whose POL prevotes have not arrived is not complete, so the node stays in Propose until those prevotes arrive (addVoteDispatchPrevoteMw, last row) or the timeout fires; in the latter case the block and proposal are held, rules 1–6 can pass, and rule 8 yields nil because maj23(Prevotes(POLRound)) is absent.

Precommit decision (EnterPrecommitAction.Execute)

flowchart TD
    S([EnterPrecommit h,r]) --> A{polka in Prevotes r?}
    A -- no --> NIL[precommit nil, keep lock]
    A -- yes --> B{polka is nil?}
    B -- yes --> NIL
    B -- no --> C{Proposal and ProposalBlock held?}
    C -- no --> NIL
    C -- yes --> D{Proposal.Timestamp == block.Time?}
    D -- no --> NIL
    D -- yes --> E{held block == polka block B?}
    E -- no --> F{LockedBlock == B?}
    F -- yes --> NIL2[precommit nil, keep lock]
    F -- no --> RT[retargetTo B: fetch B; precommit nil]
    E -- yes --> G[mustEnsureProcess r; mustValidate]
    G --> H{LockedBlock == B?}
    H -- yes --> RL[LockedRound = r; relock event; precommit B]
    H -- no --> LK[lock B at r; lock event; precommit B; updateValidBlock]

Rules, in evaluation order:

  1. C-NoPolka. ¬maj23(Prevotes(r)) → precommit nil. Reached from the PrevoteWait / PrecommitWait tocks. The lock is unchanged.
  2. C-POLInfo. HeightVoteSet.POLInfo() returns the highest round with a polka; if it is < r the node panics (cannot happen after rule 1).
  3. C-NilPolka. Polka for nil → precommit nil.
  4. C-NoProposal. Proposal == nil ∨ ProposalBlock == nil → nil. No fetch is started here (the polka middleware already retargeted to B).
  5. C-Timestamp. Proposal.Timestamp ≠ ProposalBlock.Time → nil.
  6. C-LockedNotProposed. Polka block B equals LockedBlock but this round's proposal block is different → nil, lock kept. Rationale in the code: the precommit's extensions are produced by ExtendVote for the block the application processed last at this round; a block locked earlier was processed for an earlier round, and an honest proposer would have re-proposed B. Consequence: a locked validator that misses the re-proposal of its locked block cannot relock in that round (TestStateLock_MissingProposalWhenPOLSeenDoesNotUpdateLock).
  7. C-Lock / C-Relock. B is this round's proposal block: mustEnsureProcess(rs, r) (re-runs ProcessProposal if the block was processed for another round; panics on failure), mustValidate; if already locked on B → LockedRound = r, Relock event, precommit B; else updateLockedBlock() (LockedRound = r, LockedBlock = B), Lock event, precommit B, updateValidBlock() (+ ValidBlock event if it changed).
  8. C-Unknown. Polka for a block the node does not hold and is not locked on → retargetTo(B) (start collecting its parts; a proposal for another block is dropped) and precommit nil.

A non-nil precommit triggers ABCI ExtendVote inside voteSigner.signVote; a nil precommit carries no extensions (Vote.ValidateBasic rejects extensions on prevotes and nil precommits).

Proposal-validity summary

A block becomes prevotable when all of the following hold; the function that checks each is given.

CheckWhere
Proposal signed by proposer(h, r)Proposaler.verifyProposal
POLRound ∈ {-1} ∪ [0, r)Proposaler.Set
Proposal BlockID hash and part-set header equal the assembled block'sAddProposalBlockPartAction.addProposalBlockPart
Parts' Merkle proofs valid; total size ≤ Block.MaxBytesPartSet.AddPart, addProposalBlockPart
Proposal.Timestamp == Header.TimeisValidForPrevote, EnterPrecommitAction.Execute
Timely (PBTS window; in replay, not too early), unless ValidBlockStateData.proposalIsTimely
Core chain lock consistent with statesm.ValidateBlockChainLock, validateBlock
Header.ProposerProTxHash == proposer(h, r) when POLRound == -1isValidForPrevote
Block.ValidateBasic; version, chain id, height, LastBlockID, ConsensusHash, ValidatorsHash, LastCommit threshold signature, proposer is a validator, Time > LastBlockTime (≥ genesis at initial height), evidence sizevalidateBlock in internal/state/validation.go
ABCI ProcessProposal = ACCEPT; AppHash, ResultsHash, NextValidatorsHash, NextConsensusHash equal the application's; app hash sizeBlockExecutor.ProcessProposal, ValidateBlockWithRoundState
Lock rules P-Lock-NoPOL / P-Lock-POLprevoter.shouldVoteBeSent

Alternative flows and edge cases

Each section states the situation, the behaviour the code implements, and the implementing function. Commit-related flows are in Commit handling; crash recovery and sync handover in Recovery and synchronisation.

Proposal missing, late or invalid

Proposer silent or slow

timeoutPropose (proposeTimeout(r)) fires → EnterPrevote. isValidForPrevote fails with errPrevoteProposalBlockNil → nil prevote, no ABCI call (prevoter.Do). When every validator prevotes nil, the +2/3 nil majority dispatches EnterPrecommit directly — no PrevoteWait, no voteTimeout(r) (addVoteDispatchPrevoteMw: TwoThirdsMajority() returns the nil block id, and blockID.IsNil() satisfies the precommit condition) → nil precommit → +2/3 nil precommits → EnterPrecommitWait → voteTimeout(r) → EnterNewRound(r + 1). EnterPrevoteWait is reached only when the prevotes hold +2/3-any without a majority (some validators did receive and prevote the block), in which case voteTimeout(r) precedes EnterPrecommit.

sequenceDiagram
    participant T as timeoutTicker
    participant V as Validator
    participant N as Other validators
    T->>V: tock Propose(h,r)
    V->>V: EnterPrevote: no block → prevote nil
    V-->>N: prevote nil
    N-->>V: prevotes nil
    V->>V: +2/3 nil prevotes → EnterPrecommit: polka nil → precommit nil
    N-->>V: precommits nil
    V->>V: maj23 nil → EnterPrecommitWait (voteTimeout r)
    T->>V: tock PrecommitWait(h,r)
    V->>V: EnterPrecommit (no-op) → EnterNewRound(h,r+1) → EnterPropose(h,r+1)

A proposer that is silent while its round waits for transactions is handled before the propose step on CONSENSUS_VERSION_2 chains: validators that stop hearing its heartbeats skip the round with a round skip certificate (Proposer liveness and round skipping).

Proposal arrives after the node prevoted

Proposaler.Set has no step check, so the proposal is still stored and parts are still collected. ProposalCompletedAction.Execute enters prevote only while step ≤ Propose, so the node does not prevote it. The block can still:

  • become ValidBlock if a polka for it exists or forms (ProposalCompletedAction.Execute, addVoteUpdateValidBlockMw);
  • be precommitted if a polka for it forms while the node is in Prevote (addVoteDispatchPrevoteMw → EnterPrecommit; rule C-Lock);
  • be re-proposed in a later round as ValidBlock.

Proposal for another height or round

Ignored silently (Proposaler.Set). Block parts for another round of the current height are accepted into the current part set when their proofs match its header ("blocks might be reused", addProposalBlockPart); parts for another height are ignored; parts when no part set is expected are ignored. blockPartMessageHandler swallows errors for parts whose round differs from the current one.

Invalid POLRound, bad signature, wrong proposer

ErrInvalidProposalPOLRound, ErrInvalidProposalSignature (counted in ProposalVerifyFailures), see Proposal acceptance. A proposal signed by a validator other than proposer(h, r) fails the signature check because the signature is verified against the expected proposer's key only. The round proceeds as if no proposal had been received.

Proposal contradicts the round's target block

Refused with ErrInvalidProposalForCommit (a verified commit for the same (h, r) names another block), ErrInvalidProposalForPartSet (a part set is already being collected under another header), or ErrInvalidProposalBlockID / ErrInvalidProposalCoreHeight (a block is already assembled and differs) (roundStateRefusal, Proposaler.Set). Rationale: a part-set header is a Merkle root over exactly one block's bytes, so a disagreeing proposal could never be acted on and would make the node reject the parts it is waiting for.

Block id mismatch discovered when the block completes

If the assembled block's hash or part-set header differ from the proposal's, the proposal is dropped (rs.Proposal = nil) but the block is kept (addProposalBlockPart). The node later prevotes nil (errPrevoteProposalNil). The fields not covered by the proposal signature (CoreChainLockedHeight, BlockID.StateID) are instead taken from the authenticated block bytes.

Oversized or malformed block

Accumulated parts above ConsensusParams.Block.MaxBytes → error; an undecodable block → error (addProposalBlockPart). Parts with invalid Merkle proofs are charged to the sender's blockPartProofBudget and dropped silently once it is exhausted (internal/consensus/block_part_budget.go).

Block is invalid, application rejects it, or the header does not match

BlockExecutor.ProcessProposal first validates the block against the committed state (ValidateBlock); a block that fails is never sent to the application. That failure (sm.ErrInvalidBlock), the application returning REJECT (sm.ErrBlockRejected), or a well-formed ACCEPT whose contents disagree with the block — header mismatch / invalid LastCommit (sm.ErrInvalidBlock) — → nil prevote (prevoter.handleError). A failed ProcessProposal clears CurrentRoundState, so no round counts as processed until a later call succeeds (blockExecutor.ensureProcess). Any other error panics the node: transport failure, UNKNOWN status, or a malformed response — an AppHash of the wrong size fails ResponseProcessProposal.Validate and tx_results not matching the transactions fails validateExecTxResults, both returned by BlockExecutor.ProcessProposal as plain errors, not as ErrInvalidBlock. The same block reaching EnterPrecommit via a polka would call mustEnsureProcess, which panics on failure — a node that could not process a block cannot have a polka-matching proposal block for it unless the application is non-deterministic.

During WAL replay a mismatch between a stored block's AppHash and the application's result stops the node (checkAppHashEqualsOneFromBlock in internal/consensus/replayer.go).

Proposal timestamp differs from block time

Nil prevote (errPrevoteTimestampNotEqual) and nil precommit (rule C-Timestamp).

Locking

The lock is defined by LockedRound/LockedBlock and the rules P-Lock-NoPOL, P-Lock-POL and C-Lock, C-Relock, C-LockedNotProposed.

SituationBehaviourEvidence
Locked on B at r; round r + k proposes B with POLRound ≥ LockedRound or without POLPrevote B (lock match)prevoter.checkProposalBlock, checkPrevoteMaj23; TestStateLock_POLRelock
Locked on B; round r + k proposes C ≠ B with POLRound = -1Prevote nil (ProcessProposal still runs for C)TestStateLock_PrevoteNilWhenLockedAndDifferentProposal
Locked on B; proposal C with POLRound = p ≥ LockedRound and a polka for C at pPrevote C; if a polka for C forms this round → lock moves to C (updateLockedBlock)TestStateLock_POLUpdateLock
Locked on B; proposal C with POLRound = p < LockedRoundPrevote nilcheckPrevoteMaj23
Polka for nil in a later roundLock unchanged — no unlockTestStateLock_POLDoesNotUnlock
Polka for B (the locked block) in round r + k but no proposal for B in that roundPrecommit nil, lock unchangedrule C-LockedNotProposed; TestStateLock_MissingProposalWhenPOLSeenDoesNotUpdateLock
Polka for B in round r + k and proposal for BRelock: LockedRound = r + k, precommit B after re-processing for r + kTestStateLock_POLRelock
Proposal for an old block (POLRound pointing at a polka for another block)Not prevoted, not lockedTestStateLock_DoesNotLockOnOldProposal

Liveness consequence of C-LockedNotProposed: when a round's proposer is slow or faulty, validators locked on B do not relock even if the other validators prevote B; the network must wait for a later proposer to re-propose B with POLRound set.

Valid-block bookkeeping

StateData.updateValidBlock sets ValidRound = Round, and when the block differs from the current ValidBlock, ValidBlock = ProposalBlock, ValidBlockParts, and ValidBlockRecvTime = ProposalReceiveTime when the proposal names this block, else now. It is called from:

  1. addVoteUpdateValidBlockMw: a prevote completes a polka for non-nil B in the current round with ValidRound < Round, and the node holds B. Whether or not it holds B, retargetTo(B, polka) then drops a proposal for another block, replaces the part set unless its header is B's, and a ValidBlock event is published so peers send B's parts (NewValidBlockMessage).
  2. ProposalCompletedAction.Execute: the block completes while a polka for it already exists in the current round.
  3. EnterPrecommitAction.Execute: after a first lock.

ValidRound/ValidBlock are never reset within a height and survive EnterNewRound.

Re-proposal. Proposaler.Create reuses ValidBlock iff checkValidBlock: at InitialHeight its time equals the genesis time; otherwise ValidBlock.IsTimely(ValidBlockRecvTime, SynchronyParams, ValidRound). A re-proposal keeps the original header (proposer, time) and carries POLRound = ValidRound. Receivers skip the timeliness check for a block that is their own ValidBlock (isValidForPrevote) and skip the proposer check when POLRound ≥ 0. When ValidBlock is not timely a fresh block is built and proposed with POLRound = -1, so receivers judge it like any new proposal (timeliness, proposer check, rule P-Lock-NoPOL). A validator locked on another block prevotes nil for it; see Known issues.

Round skipping and nil rounds

  • Future-round prevotes. +2/3-any prevotes at vote.Round > r → EnterNewRound(vote.Round) (addVoteDispatchPrevoteMw). The node skips the intermediate rounds without voting in them.
  • Future-round precommits. +2/3-any precommits without a majority at vote.Round ≥ r → EnterNewRound(vote.Round) + EnterPrecommitWait (addVoteDispatchPrecommitMw); the node then waits voteTimeout before r + 1. A precommit majority for a block at any round (including a round below the current one) → EnterCommit with that round (TestCommitFromPreviousRound).
  • Future-round commit. If its block is already held, a verified CommitMessage for commit.Round > r is checked against the block and application before advancing the round; rejection does not move the round. If the block is absent, signature verification can park the commit and advance to its round before the later application check. See Commit handling.
  • Round skip certificate (CONSENSUS_VERSION_2). A verified certificate for round r' ≥ r → EnterNewRound(r' + 1), unless the node is at ApplyCommit; one for r' < r is only kept and gossiped (AddRoundSkipCertificateAction). See Proposer liveness and round skipping.
  • Round bookkeeping. HeightVoteSet keeps vote sets for rounds 0 .. Round + 1 (SetRound(round + 1) in EnterNewRoundAction) plus at most two "catch-up" rounds per validator pro-tx-hash (HeightVoteSet.addVote, ErrGotVoteFromUnwantedRound). Vote sets are never released within a height.
  • All-nil round. Nil polka → nil precommit → nil precommit majority → EnterPrecommitWait — the node still waits voteTimeout(r) even though the round is decided — → EnterNewRound(r + 1).
  • Idempotence. TriggeredTimeoutPrecommit makes EnterPrecommitWait run once per round; the step guards do the same for the other actions.
  • What survives a round change. LockedBlock, ValidBlock, parked Commit, and all vote sets; Proposal and the proposal block/parts are cleared unless a commit is parked (EnterNewRoundAction.Execute).

Equivocation, duplicates and conflicting votes

Implemented in VoteSet.addVote / addVerifiedVote (types/vote_set.go) and addVoteErrorMw (internal/consensus/state_add_vote.go).

  • Identical vote (same validator, block key and block signature): not added, no error.
  • Same validator and block key, different signature: ErrVoteNonDeterministicSignature (BLS signatures are deterministic; logged at error level).
  • Same validator, different block id: the second vote is recorded only in votesByBlock when some peer claimed +2/3 for that block id (peerMaj23, set by VoteSetMaj23Message → VoteSet.SetPeerMaj23); it replaces the first in votes only if it matches maj23. The pair is returned as ErrVoteConflictingVotes and reported to the evidence pool (evidencePool.ReportConflictingVotes) unless the key is the node's own (logged: "did you unsafe_reset a validator?") or no private validator is set (ErrPrivValidatorNotSet). The evidence pool stores it unverified; it reaches the application as Misbehavior only if a proposer includes it (evpool.PendingEvidence in CreateProposalBlock).
  • Conflicting proposals by the same proposer: the first valid one is kept and the second ignored while the first occupies rs.Proposal; a current-round polka or a commit for the second block drops the first (StateData.retargetTo → dropStaleProposal) and the second is then accepted (invariant 5). No evidence is produced.

Vote extensions

  • The node normally asks ABCI ExtendVote when signing a non-nil precommit (voteSigner.signVote). The one exception signs nothing: when VerifyVoteExtension rejects a commit's extensions, no expectation is known for its block and round, and the block is processed for commit.Round, blockExecutor.verifyCommitExtensions calls ExtendVote for that block and round to learn the expected vector. This can happen on any node, validator or not (Parked commit). On both paths the response must parse (NewVoteExtensionsFromABCIExtended); otherwise the node panics.
  • Supported extension types (types/vote_extension.go): THRESHOLD_RECOVER and THRESHOLD_RECOVER_RAW (threshold-recoverable; RAW may carry an application-supplied sign request id that binds no height or round). DEFAULT exists in the enum but is not supported: NewVoteExtensionsFromABCIExtended rejects it in an ExtendVote response (ErrUnsupportedVoteExtensionType), and VoteExtension.Validate (proto/tendermint/types/dash.go) rejects it in a peer's vote (Vote.ValidateBasic). A prevote or nil precommit carrying extensions fails Vote.ValidateBasic.
  • Each peer precommit's extension signatures are verified after the block signature (types.VerifyVoteSignatures), then ABCI VerifyVoteExtension is called with the validator's pro-tx-hash. A rejected vote is not stored (addVoteVerifyVoteExtensionMw). Own votes skip the ABCI call.
  • At most MaxVoteExtensions = 32 extensions are accepted on verification paths (types/quorum_sign_data.go); the signing path is not capped.
  • The commit carries only the threshold-recovered extensions of the canonical group (Threshold recovery). Before a block is saved the application sees them once more as a single VerifyVoteExtension with an empty ValidatorProTxHash (sm.VerifyCommitExtensions).

Proposal timestamps and time rules

  • The proposer sets Header.Time = now (genesis time at InitialHeight, State.makeBlockTime). Before proposing, a proposer whose clock is behind state.LastBlockTime waits for the difference (EnterProposeAction.Execute, proposerWaitTime).
  • Validity: Time > state.LastBlockTime for h > InitialHeight, Time ≥ genesis at InitialHeight (validateBlock).
  • Timeliness is checked only at prevote time, against the proposal's receive time (ProposalReceiveTime = enqueue time of the ProposalMessage, chanMsgSender.send), with the window of checkTimely (rule P-Timely). It is skipped for a block that is the node's ValidBlock, so re-proposals of an already polka'd block are not re-judged. In replay mode the receive time is the time of the proposal's WAL entry, and only a proposal received too early fails. The entry is never earlier than the receipt, so replay refuses only what the node refused when it received the proposal, not always all of it. That time also becomes ValidBlockRecvTime for a block made valid during replay, which decides whether the node re-proposes it after a restart (Proposaler.checkValidBlock), and feeds ProposalTimestampDifference.
  • Timeliness governs re-proposal (Proposaler.checkValidBlock).
  • Metric ProposalTimestampDifference{is_timely} is recorded for proposals with POLRound = -1 (Proposaler.proposalTimestampDifferenceMetric).

Mempool and empty blocks

EnterNewRoundAction.Execute waits for transactions when WaitForTxs() = ¬CreateEmptyBlocks ∨ CreateEmptyBlocksInterval > 0, at round 0 or in a round entered on a round skip certificate that satisfies StateData.waitsAfterRoundSkip (The round after a skip), above InitialHeight, unless ProposeNextBlockImmediately (set from ResponseFinalizeBlock.propose_next_block_immediately of the previous height by ApplyCommitAction.Execute; consumed by the next updateToState; not persisted, so a restart falls back to the configured wait).

State.handleTxsAvailable records the report in TxsReported (the mempool reports once per height) and acts only at round 0 or in a round that may wait after a round skip (StateData.mayWaitAfterRoundSkip): at step NewHeight (and not at InitialHeight) it schedules a NewRound tock for StartTime − now + 1 ms; at step NewRound it dispatches EnterPropose. With CreateEmptyBlocksInterval > 0 the interval tock calls EnterPropose even without transactions (empty block); in a round after a skip the interval counts from StartTime. Other rounds ≥ 1 never wait. On CONSENSUS_VERSION_2 chains a proposal or prevotes can also end the wait (When it applies).

Validator-set and parameter changes

  • Changes come from the application's ValidatorSetUpdate in the PrepareProposal/ProcessProposal response and are reflected in the header's NextValidatorsHash, which every validator checks against its own application's answer (ValidateBlockWithRoundState). With an unchanged QuorumHash the update is applied as a change set; with a new QuorumHash the new set replaces the old one (CurrentRoundState.valsetUpdate in internal/state/current_round_state.go).
  • State.Update makes Validators ← NextValidators, LastValidators ← Validators; the new set votes from height h + 1 (updateToState builds the next HeightVoteSet from it and logs "Updating validators" when the quorum hash or threshold changes).
  • Consensus parameters (timeouts, synchrony, block limits, proposer-selection version) proposed in block h take effect at h + 1 (LastHeightConsensusParamsChanged).
  • Proposer selection version is ConsensusParams.Version.ConsensusVersion (Proposer selection); version 2 also enables proposer liveness.

Core chain locks

  • Header fields: CoreChainLockedHeight; optional CoreChainLock in the block (and CoreChainLockUpdate in the proposal, not signed).
  • ValidateBlockChainLock (prevote) and validateCoreChainLock (validateBlock, before the application call): with a lock, the header height equals the lock's and exceeds state.LastCoreChainLockedBlockHeight; without one, it equals it.
  • Proposaler.Set rejects proposal.CoreChainLockedHeight below the committed height; after the block completes the proposal's value is overwritten from the block.
  • state.LastCoreChainLockedBlockHeight advances only when the block carries a CoreChainLock (CurrentRoundState.UpdateState).

Non-validators and signer problems

  • Observers (node not in Validators): the reactor drops incoming Vote messages (Reactor.handleVoteMessage); the consensus state still runs Enter* actions on timeouts, but voteSigner.signAddVote returns without signing (!Validators.HasProTxHash(own)). Observers progress to the next height only through CommitMessages and block parts (Commit handling). Peers gossip proposals, votes and VoteSetMaj23 only to validator peers, and block parts to non-validators once they announce a commit (gossip_handlers.go).
  • No private validator (privValidator.IsZero()): no votes (ErrPrivValidatorNotSet logged); blockExecutor.create fails with "cannot create proposal block on non-validator", so a proposer without a key is silent and the round times out.
  • Validator set without public keys (HasPublicKeys = false): HeightVoteSet.addRound creates no vote sets and addVote drops every vote; proposals can only be accepted when a verified commit for the same (h, r) names the same block (Proposaler.verifyProposalForNonValidatorSet).
  • Signing failure (PrivValidator.SignVote/SignProposal error or deadline): logged; no vote/proposal and no retry within the step (voteSigner.signAddVote, Proposaler.Create). A failed round skip vote is retried after another silence timeout.
  • Remote signers (socket, gRPC) cannot sign proposer heartbeats or round skip votes (RemoteSignerError 501, types.ErrSigningNotSupported): a validator using one sends neither, so its waiting rounds are skipped (Signers).
  • Observers and liveness messages: a node without the validators' key shares neither verifies nor relays heartbeats and skip votes; it verifies and applies round skip certificates against the quorum public key.
  • Block sync just finished: proposing is suppressed until the node has passed the handover target height or 10 minutes elapsed (catchupTracker.mayPropose, maxCatchupSuppression); voting is not.
  • WAL replay: the node never proposes (EnterProposeAction.Execute, replayMode).

Panics that halt the node

State.receiveRoutine re-panics any panic from a handler except autofile.ErrAutoFileClosed and panics during context cancellation, so the following stop the node (liveness-relevant for operators):

TriggerFunction
WAL WriteSync failure for own messages or EndHeightMessagewalMiddleware, ApplyCommitAction.Execute
PrepareProposal errorblockExecutor.create
ProcessProposal unknown error at prevote; any failure in mustEnsureProcess (precommit, commit, apply)prevoter.handleError, blockExecutor.mustEnsureProcess
Validation failure after a successful ProcessProposalblockExecutor.mustValidate
ExtendVote / VerifyVoteExtension transport error or invalid extensionBlockExecutor.ExtendVote, BlockExecutor.VerifyVoteExtension
FinalizeBlock failureApplyCommitAction.Execute
ABCI client stopped during commit verificationverifyProcessedCommit
Internal invariant violations: EnterPrevoteWait/EnterPrecommitWait without +2/3-any, EnterCommit/finalizeCommit without a majority or without the block, POLInfo round below the current, updateToState height mismatch, non-monotonic HeightVoteSet.SetRound, proposer-selector construction failurerespective Execute functions, StateData.updateToState, HeightVoteSet.SetRound

Errors from message handlers invoked by the reactor are recovered and logged (Reactor.handleMessage), not panicked.

Rate limits and budgets that change protocol timing

These mechanisms never change what a node signs, but they decide whether a message is processed at all (dropped messages are local decisions, not peer faults, unless stated). For operational monitoring, see DoS observability.

  • Reactor admission (internal/consensus/reactor.go, message_cost.go): per-peer token buckets on the vote channel (cost 1 per vote plus one per extension for non-nil precommits; 1 + extensions per commit), the data channel (proposals weigh more than parts) and the state channel; VoteSetMaj23 answers are deduplicated (PeerState.ShouldAnswerVoteSetMaj23). Over budget → silent drop.
  • Peer lanes (internal/consensus/peer_lanes.go): one FIFO per peer (laneCapacity = 512 messages, 64 MiB aggregate), served round-robin by verification cost. The node's own messages bypass lanes via the internal queue (chanMsgSender.send).
  • Verification budget (internal/consensus/verification_budget.go, config.VerificationRateLimit): a global BLS-verification rate; a message waits until its full cost is affordable, and verificationBudgetMiddleware drops what cannot be afforded. It is the outermost middleware so a dropped message is not written to the WAL. ErrVerificationBudgetExhausted never evicts a peer (isPeerFloodableError).
  • Block-part proof budget (block_part_budget.go): per-peer allowance for parts whose Merkle proof fails.
  • Liveness messages (CONSENSUS_VERSION_2): heartbeats, round skip votes and certificates pay extra state-channel tokens only when new, go through the peer lanes and the verification budget, and are sent through a bounded reactor outbox; heartbeats are paced per round (Message admission and budgets).
  • Bounded memory per height: commit candidate slots (one per peer), commitExpectations, MaxVoteExtensions, HeightVoteSet catch-up rounds, maxPeerMaj23s claims per vote set.

Under load a precommit carrying many extensions can wait in a lane for seconds, which is within the propose timeout but may exceed the vote timeout.

Proposer liveness and round skipping

A round that waits for transactions has no timeout running unless create-empty-blocks-interval is set, so a proposer that is offline while the chain is idle is only noticed once a transaction arrives, and that transaction then waits out the propose timeout. From CONSENSUS_VERSION_2 the proposer of a waiting round announces itself with signed heartbeats, and validators that stop hearing them hand the round to the next proposer with a threshold-signed round skip certificate.

Wire formats are in Consensus channel messages; sign bytes and request IDs in Signing.

When it applies

roundSkipping (internal/consensus/proposer_liveness.go) gates everything on this page. It holds when both:

  • ConsensusParams.Version.ConsensusVersion ≥ 2 (VersionParams.ProposerLivenessEnabled, types/params.go), and
  • types.RoundSkipsProvable(Validators): T ≤ R, i.e. a VotingPowerThreshold above the recovery threshold disables round skipping, because a certificate only proves that shares worth R signed (types/round_skip.go).

Otherwise no heartbeat is sent or accepted, no skip vote is signed or collected, every certificate is dropped, and the node behaves as on version 0 or 1.

A node waits for transactions in its round (waitingForTxs) while roundSkipping ∧ step == NewRound ∧ Commit == nil; liveness is tracked while it also holds no proposal for the round (waitingRound). Whether a round waits at all is decided by EnterNewRoundAction.Execute (Mempool and empty blocks). On a version-2 chain a wait also ends, with Dispatch(EnterPropose{h, r}), when:

  • a proposal for the round is accepted (proposalMessageHandler, internal/consensus/msg_handlers.go); a round-0 proposal accepted before the round starts keeps the round from waiting at all;
  • a prevote is added while the node waits and the added prevote's round (this round or an earlier one) has a polka, or the prevote is for the current round and r > 0 (addVoteDispatchPrevoteMw).

A proposal or polka shows that a block is under way; from then on the propose timeout, not the heartbeats, moves the round on.

Heartbeats

State.followLiveness (internal/consensus/state.go) runs once before the receive routine's first input and after every input. It calls proposerLiveness.reconcile, which, while waitingRound holds, either announces (this node is proposer(h, r)) or watches (any other node); otherwise it stops both.

Sending (proposerLiveness.sendHeartbeats, sendHeartbeat):

  • one heartbeat at once and then every ProposerHeartbeat, until the node leaves the round;
  • only while the node could propose (State.proposerReady): the propose step would not suppress it after a block-sync handover (catchupTracker.mayPropose) and the application answers ABCI Echo on every backend that receives PrepareProposal (BlockExecutor.AppReady in internal/state/execution.go, abciclient.EchoRoute) within the interval. At most one probe is in flight (appReadiness, internal/consensus/app_readiness.go);
  • Sequence = max(last + 1, signing time in Unix ms) (heartbeatSequence);
  • one heartbeat is signed at a time, whatever its round; a heartbeat whose round the node left while signing is dropped. A signer error wrapping types.ErrSigningNotSupported stops heartbeats for the rest of the process (signerUnsupported).

Receiving: the reactor screens a heartbeat without a signature check (proposerLiveness.wanted → screen) and queues it in the sender's lane; the consensus goroutine verifies it (proposerLiveness.heard, called from State.handleLivenessMessage). A heartbeat is accepted iff:

  1. its height is the node's and its round is within the relay window [max(r − 4, 0), r + 4] (relayWindow, maxRoundSkipRoundsAhead);
  2. it names proposer(h, round), otherwise errProposerHeartbeatWrongProposer;
  3. it was not seen before, identified by (height, round, sequence, timestamp) (heartbeatID; at most maxSeenHeartbeats = 256 remembered);
  4. its timestamp is within the silence timeout of local time, in either direction;
  5. the node holds the proposer's public key share (validators only);
  6. the round's pace has a token: one per half heartbeat interval, burst 2 (heartbeatPaceBurst);
  7. the verification budget allows one check and ProposerHeartbeat.Verify succeeds (types/proposer_heartbeat.go).

An accepted heartbeat for the round the node watches re-arms its silence timer; one for a nearby round is only relayed. Every accepted heartbeat is relayed once. Heartbeats go only to validator peers at the same height whose round is within the relay window of the heartbeat's round, never back to the sender (Reactor.sendHeartbeat, wantsHeartbeat in internal/consensus/reactor.go). A heartbeat changes no consensus state and is not written to the WAL.

Silence and skip votes

The watch arms a timer of ProposerSilenceTimeout(r) = ProposerSilence + ProposerHeartbeat · r, saturating at the maximum duration (TimeoutParams.ProposerSilenceTimeout, types/params.go). When it fires, State.handleProposerSilence acts only if the report is still current (same height and round, still waiting, no heartbeat re-armed the timer since) and the node is a validator. ProposerSilences counts the first report of a round.

State.voteToSkipRound then:

  • if this validator's vote for the round is already counted, broadcasts it again;
  • otherwise has RoundSkipVote{h, r, own pro-tx-hash, own index} signed off the consensus goroutine (roundSkipSigner, internal/consensus/round_skip_signer.go), one vote at a time, with the silence timeout as the signing deadline. If a vote is still being signed, this one is retried after another silence timeout.

State.handleSignedRoundSkipVote drops the result if the node left the round or a heartbeat arrived meanwhile. ErrSigningNotSupported ends the attempt; any other signing error retries after another silence timeout. A signed vote is counted, broadcast, and broadcast again every silence timeout while the round stays silent.

A skip vote is not a prevote: it locks nothing, and the validator still prevotes and precommits the round's proposal if one arrives. Every validator signs the same bytes for (h, r), so a validator can make only one skip vote per round.

Collecting votes and recovering the certificate

roundSkipCollector (internal/consensus/round_skip_collector.go) holds the skip votes of the current height and quorum for rounds [r, r + 4]. Votes of rounds below r are dropped when the node moves on. A vote from a peer is counted iff its index and pro-tx-hash name the same validator (otherwise errRoundSkipVoteWrongValidator), it is not already counted, the round is not already certified, the node holds the validator's key share, the verification budget allows one check, and RoundSkipVote.Verify succeeds.

A vote is relayed when first counted. A copy of a counted vote that arrives at least half a silence timeout after the last relay has the counted vote relayed again, until the round is certified (roundSkipCollector.relayAgain). Skip votes are not written to the WAL.

Once the counted power reaches RoundSkipThresholdPower = max(T, R) (types/round_skip.go; equal to R whenever roundSkipping holds; +2/3 on mainnet, whose LLMQ_100_67 quorums recover from 67 of 100 shares and set no VotingPowerThreshold), RecoverRoundSkipCertificate recovers the quorum threshold signature over the round's sign bytes from the shares and verifies it against ValidatorSet.ThresholdPublicKey. A failed recovery is retried at once with any votes counted meanwhile, otherwise with the next vote. The recovered certificate is dispatched immediately on the consensus goroutine (State.applyRecoveredCertificate).

Applying a certificate

AddRoundSkipCertificateAction.Execute (internal/consensus/state_add_round_skip.go):

  • Guard (roundSkipCertificateWanted, also checked by the reactor before queuing): roundSkipping ∧ cert.Height == h ∧ (RoundSkip == nil ∨ cert.Round > RoundSkip.Round) ∧ step < ApplyCommit.
  • Actions:
    1. Unless this node recovered it: charge the verification budget and RoundSkipCertificate.Verify (quorum hash must equal the validator set's, else ErrRoundSkipCertificateWrongQuorum; threshold signature against ThresholdPublicKey).
    2. Write the message it came in to the WAL (Write, no fsync); not on replay.
    3. RoundSkip = cert; broadcast it.
    4. If cert.Round < r: stop (the certificate is kept for peers still in that round). Else count RoundSkips and Dispatch(EnterNewRound{h, cert.Round + 1}).

The certificate does not touch locks, valid blocks or vote sets; it only moves the node to a higher round, which the locking rules make safe.

sequenceDiagram
    participant P as Proposer of round r
    participant V as Validators
    participant N as Proposer of round r+1
    P-->>V: ProposerHeartbeat every proposer_heartbeat
    Note over P: goes offline
    V->>V: no heartbeat for ProposerSilenceTimeout(r)
    V-->>V: RoundSkipVote(h, r)
    V->>V: +2/3 skip votes → RoundSkipCertificate(h, r)
    V->>V: EnterNewRound(h, r+1): waits for txs
    N-->>V: ProposerHeartbeat for round r+1

The round after a skip

EnterNewRoundAction.Execute lets round R > 0 wait for transactions, as round 0 does, when StateData.waitsAfterRoundSkip(R) holds (internal/consensus/state_data.go):

  • RoundSkip.Round == R − 1;
  • LockedRound == -1 ∧ ValidRound == -1;
  • the mempool has not reported transactions at this height (TxsReported; the mempool reports once per height, and State.handleTxsAvailable records the report in any round);
  • no polka at the height (HeightVoteSet.POLInfo round -1) and no prevote recorded for round R.

The other wait conditions are unchanged (WaitForTxs(), h ≠ InitialHeight, no ProposeNextBlockImmediately). A waiting round R > 0 publishes its NewRound step, so peers send it the round's proposal. The empty-block tock is scheduled after max(CreateEmptyBlocksInterval − (now − StartTime), 0): the interval counts from the start of the height, not from the skipped round.

Gossip

Two gossip handlers run per peer at the PeerQueryMaj23SleepDuration cadence (internal/consensus/gossip_handlers.go, wired in gossip_peer_worker.go):

  • roundSkipGossipHandler sends the held RoundSkip to every peer at the height that is not committing and has not sent a certificate for that round or a later one (PeerState.HasRoundSkip). Nothing else moves a peer out of a skipped round: it has no votes to follow and the next proposal is sent only to peers already in the next round.
  • roundVoteResendGossipHandler (roundSkipping only, validator peers) sends one random vote per pass: the round's prevotes to a peer waiting in this node's round r > 0; the prevotes and precommits of the peer's round, and of the latest later round with +2/3-any (laterRoundVotes), to a peer in a round this node has left.

Recovery

  • Certificates are WAL msgInfo records; replay applies them again without charging the verification budget. Replay never skips rounds on a roundSkipping chain (below).
  • On a roundSkipping chain WalSkipRoundsToLast is ignored (State.replaySkipsRounds, internal/consensus/replay.go): whether a round after a certificate waits depends on polkas of earlier rounds.
  • The mempool report is not in the WAL. On a roundSkipping chain a replayed Propose step event for the current height enters the recorded round, and enters the propose step if replay left the node waiting (State.readReplayMessage).
  • After a restart into a waiting round the heartbeat or watch starts before the first input (State.followLiveness).

Message admission and budgets

  • State channel. ProposerHeartbeat, RoundSkipVote and RoundSkipCertificate travel on the state channel (p2p.ResolveChannelID). Each pays the state-channel cost of 1 on arrival and livenessTokenCost (100) more only once the checks that need no signature find it new (Reactor.chargeLivenessNews); it then joins the sender's peer lane.
  • Verification budget. Each costs baseMessageCost (1) to verify (peerMessageCost, budgetedMessageCost in internal/consensus/message_cost.go). One that cannot be afforded is dropped; it is sent again.
  • Peer errors. All rejections below are logged at Debug (isPeerFloodableError):
    • a liveness message that does not decode or fails its basic checks (MsgFromProto, errMalformedLivenessMessage) is reported as p2p.ErrInvalidData, on which the router disconnects the sender;
    • a heartbeat from the wrong proposer or a skip vote whose index and pro-tx-hash disagree, found by the reactor's screening, is a non-fatal peer error (PeerStatusBad);
    • a heartbeat, skip vote or certificate whose signature fails against the key the node holds is reported as a fatal peer error, which disconnects the sender (forgedLivenessMiddleware);
    • a certificate for another quorum (ErrRoundSkipCertificateWrongQuorum) is only logged.
  • Outboxes. Liveness messages are sent from a reactor goroutine through a bounded outbox (livenessOutboxSize = 256), each send bounded by voteSetResponseTimeout; a message that finds the outbox full is dropped and sent again on its own schedule. Round-step, valid-block, HasVote and HasCommit broadcasts go through a separate outbox (stateOutboxSize = 1024); after a drop the sender rebroadcasts the current round step (Reactor.sendState). Neither blocks the consensus goroutine.

Signers

FilePV (privval/file.go) and DashCoreSignerClient (privval/dash_core_signer_client.go) sign heartbeats and skip votes (SignProposerHeartbeat, SignRoundSkipVote of the PrivValidator interface, types/priv_validator.go). The socket and gRPC remote signers (privval/signer_client.go, privval/grpc/client.go) return RemoteSignerError code 501, which matches types.ErrSigningNotSupported. A validator behind such a signer sends no heartbeats, so validators skip its waiting rounds after the silence timeout, and it signs no skip votes; it still counts others' votes and applies certificates.

Parameters

TimeoutParams.ProposerHeartbeat (field 7) and ProposerSilence (field 8) come from the committed state's consensus parameters, like the other timeouts. TimeoutParamsOrDefaults derives the values in effect from what is stored:

Stored heartbeatStored silenceHeartbeat in effectSilence in effect
unsetunset1 s5 s
unsetsmin(1 s, s / 5)s
bunsetbmax(5 s, 5·b)
bsbs
  • Validation (ConsensusParams.ValidateConsensusParams): neither negative; the derived heartbeat is positive and shorter than the derived silence.
  • Updates (ConsensusParams.UpdateConsensusParams): an update that names either field, or that raises ConsensusVersion from below 2 to 2 or more, clears both and sets only those it names; the other is derived again.
  • Encoding (ConsensusParams.ToProto): unset fields are omitted.
  • Hashing (ConsensusParams.HashConsensusParams): from version 2, HashedParams.proposer_heartbeat_ns (field 4) and proposer_silence_ns (field 5) carry the values in effect, in nanoseconds; earlier versions leave them unset, so their hash is unchanged.

Metrics

MetricCounts
consensus_proposer_heartbeats_sentHeartbeats this node signed and broadcast
consensus_proposer_heartbeats_receivedNew, verified heartbeats for the round this node waits in
consensus_proposer_silencesRounds in which this validator stopped hearing from the proposer while waiting
consensus_round_skipsRounds this node moved past on a certificate

Defined in internal/consensus/metrics.go.

Commit handling

A commit for height h is (Height, Round, BlockID, QuorumHash, ThresholdBlockSignature, ThresholdVoteExtensions) (types.Commit). A node obtains one in two ways:

  1. locally, from its own precommit vote set once the threshold signature is recovered (VoteSet.MakeCommit);
  2. from a peer or the WAL, as a CommitMessage.

In both cases the block must be held, processed by the application for the commit's round, validated, and its extension vector accepted by the application before it is saved and finalized.

Commit verification

ValidatorSet.verifyCommit (types/validator_set.go) checks:

  1. commit.Height == height and commit.BlockID == blockID (hash, part-set header, state id);
  2. commit.QuorumHash == Validators.QuorumHash (ErrInvalidCommitQuorumHash);
  3. the number of extension signatures matches the number of extensions and is ≤ MaxVoteExtensions (ErrVoteExtensionCountMismatch);
  4. the block threshold signature and every extension threshold signature verify against ThresholdPublicKey over the canonical vote sign items (QuorumSignData.Verify); failure → ErrInvalidCommitSignature.

Only ErrInvalidCommitSignature identifies a dishonest sender (an honest node stores a commit only after verifying it) and leads to eviction (TryAddCommitAction.handleCommitVerifyError); every other failure is only counted in CommitVerifyFailures{reason}. Replayed commits never evict.

Consensus verifies a received commit against the commit's own BlockID (StateData.readyToApplyCommit), not against the proposal it holds: a proposal the network never committed proves nothing about the commit, and checking against it rejected genuine commits (dashpay/tenderdash#1414). Step 1 above is therefore tautological on this path and the block is matched to the commit afterwards by verifyCommitBlock.

Applying a commit built from local precommits

Entered through EnterCommitAction → TryFinalizeCommitAction (internal/consensus/state_try_finalize_commit.go).

  • Precondition: Step == ApplyCommit, maj23(Precommits(CommitRound)) is a non-nil B.
  • If the node does not hold B (!ProposalBlock.HashesTo(B.Hash)) the action returns quietly; EnterCommit has already retargeted the part set to B and published a ValidBlock event, so peers send B's parts; when the last part arrives ProposalCompletedAction (step ApplyCommit) dispatches TryFinalizeCommit again.
  • finalizeCommit: seenCommit = Precommits(CommitRound).MakeCommit(); refuseUnprocessedCommit; mustEnsureProcess(rs, seenCommit.Round); verifyCommitExtensions (ABCI VerifyVoteExtension on the canonical commit vote). If the application rejects its own quorum's extension vector, nothing is persisted, the failure is counted, and the node stays in ApplyCommit waiting for a peer's commit (a panic would restart into the same commit). Otherwise Dispatch(ApplyCommit{seenCommit}).

ApplyCommitAction.Execute is specified in happy path §9. Note that it re-runs ProcessProposal if the block was last processed for a different round than commit.Round (mustEnsureProcess), so the number of ProcessProposal calls per height is not bounded by one.

Commits received from peers

CommitMessage (vote channel) → commitMessageHandler → TryAddCommitAction.Execute (internal/consensus/state_try_add_commit.go).

flowchart TD
    A[CommitMessage c] --> B{rs.Commit parked?}
    B -- yes --> Q{c.Height == h?}
    Q -- yes --> QC[queueCandidate: ValidateBasic, dedupe, verify threshold signature, keep in sender's slot]
    Q -- no --> IGN[ignore]
    B -- no --> C{c.Height == h and c.Round > r?}
    C -- yes --> FR[readyToApplyCommit ignoreProposalBlock=true]
    FR --> FRV{verified?}
    FRV -- no --> ERR[handleCommitVerifyError]
    FRV -- yes --> HELD{block held?}
    HELD -- yes --> VH{verifyHeldCommit ok?}
    VH -- no --> RB[rs.Commit = nil; error]
    VH -- yes --> NR
    HELD -- no --> NR[EnterNewRound h, c.Round]
    NR --> AC{block still held?}
    AC -- yes --> ADD[AddCommit c]
    AC -- no --> PARK[parked; wait for parts]
    C -- no --> P[prepareCommitForApply]
    P --> PV{verified and block held?}
    PV -- error --> ERR
    PV -- not ready --> PK[parked via adoptCommit; publish ValidBlock]
    PV -- yes --> SET[rs.Commit = c] --> ADD

readyToApplyCommit (internal/consensus/state_data.go)

  1. commit.ValidateBasic() (non-nil block id, signature size, known extension types).
  2. commit.Height + 1 == h → ignored ("old commit"); commit.Height ≠ h → ignored. There is no LastCommit vote set to feed.
  3. verifyCommitSignatures(commit.BlockID, commit) — the threshold signature check, charged to the sender's verification budget.
  4. If a complete block is held whose part set has the commit's header but whose hash differs → error ("retained block does not hash to commit hash"): parts determine the block, so waiting cannot resolve it.
  5. With ignoreProposalBlock (future round): adoptCommit and return ready.
  6. If holdsProposalBlock(commit.BlockID) → ready.
  7. Else adoptCommit(commit): retargetTo(commit.BlockID, park) — drops a proposal for another block, replaces the part set unless it already has the commit's header — and rs.Commit = commit; return not ready (parked).

Held block: verifyCommitBlock and verifyProcessedCommit

  • verifyCommitBlock: part-set header, block hash and state id of the commit must equal the held block's. A state-id mismatch is fatal for the commit: applied anyway, it would be recorded with a block id the next height's LastBlockID could never match.
  • verifyProcessedCommit: refuseUnprocessedCommit (expectations cache), ensureProcess(rs, commit.Round) (ProcessProposal for the commit's round if not already processed; ErrClientStopped panics), validate, verifyCommitExtensions (ABCI). A failure returns an error; the commit is not applied.

Commit for the current round (or earlier)

prepareCommitForApply(ignoreProposalBlock = false): readyToApplyCommit, then verifyCommitBlock, then verifyProcessedCommit. On success rs.Commit = commit and Dispatch(AddCommit{commit}). When the block is not held the commit is parked and a ValidBlock event is published so peers serve the block.

Commit for a later round

commit.Round > r: readyToApplyCommit(ignoreProposalBlock = true) verifies the signature and parks the commit. If the block is already held it is verified first (verifyHeldCommit), so that a rejected commit does not move the round (on rejection rs.Commit is reset — the retarget could at most have dropped a proposal whose state id differed, which such a commit could not have matched anyway). Then EnterNewRound(h, commit.Round) — which keeps the committed block as the download target — and, if the block is held, AddCommit.

AddCommitAction.Execute (internal/consensus/state_add_commit.go)

  1. proposalUpdater.updateStateData(commit.BlockID): install the locked block if it is the committed one, else retarget.
  2. If ProposalBlock == nil (not held): return; the completing part will dispatch AddCommit again via selectParkedCommit.
  3. Step = ApplyCommit (round kept), CommitRound = commit.Round, CommitTime = now; Dispatch(ApplyCommit{commit}) (happy path §9); PublishCommitEvent (peers receive HasCommit); Dispatch(EnterNewRound{Height: rs.Height}) for the new height.

Parked commit and replacement candidates

While rs.Commit ≠ nil, further commits for the height are not applied but queued (TryAddCommitAction.queueCandidate, internal/consensus/commit_candidates.go): after ValidateBasic and deduplication against the parked commit and the sender's previous candidate, the threshold signature is verified and the commit is stored in the sender's single slot (overwritten by that sender's later commits; freed on disconnect). Candidates are keyed by (height, round, blockID, hash of the extension vector) because the application's verdict depends on all four.

When the block completes (addProposalBlockPart with rs.Commit ≠ nil), selectParkedCommit tries the parked commit and then the candidates, grouped by round (the round already processed first, then rounds in arrival order), calling verifyHeldCommit for each; the first accepted becomes rs.Commit and is dispatched as AddCommit. If none is accepted, rs.Commit = nil and the completed block continues through the ordinary flow (ProposalCompleted).

commitExpectations (internal/consensus/commit_expectations.go) remembers, per (height, round, block), the extension vector the application's own ExtendVote returns after a rejection, and whether that vector was itself rejected. A later commit for the same (round, block) with a different vector, or with the rejected one, is refused without re-processing the block (blockExecutor.refuseUnprocessedCommit, verifyCommitExtensions). The block and round are authenticated by the threshold signature, the vector is not, so this bounds ProcessProposal re-executions per height to roughly one per foreign round plus returns to the node's own round.

Commit validity depends on the application

A commit's acceptability includes an ABCI verdict (VerifyVoteExtension on the commit vote). Two signature-valid commits for the same block can differ in round or extension vector and receive different verdicts; liveness therefore depends on the application answering deterministically for the round the quorum signed.

Interaction with the step machine

  • Proposaler.Set refuses a proposal that disagrees with a parked commit for the same round and ignores a later-round proposal that would disrupt the download of a parked commit's block.
  • EnterNewRound keeps the part set targeted at a parked commit's block and re-announces it (ValidBlock event).
  • EnterPropose does not propose while a verified commit for a held block is about to be applied (applyingCommit).
  • AddProposalBlockPart dispatches ProposalCompleted only when no commit is parked or the parked one was just rejected.

Commit gossip

(internal/consensus/gossip_handlers.go, gossiper.go, reactor.go)

  • After applying a commit the node broadcasts HasCommit (PublishCommitEvent → reactor broadcast) and every step change broadcasts NewRoundStep.
  • Per peer, the vote/commit gossip loop sends CommitMessage:
    • rs.LastCommit when the peer is exactly one height behind and has not announced the commit (shouldCommitBeGossiped);
    • the stored commit for the peer's height when it is two or more heights behind (shouldCommitBeGossipedForCatchup, blockStore.LoadBlockCommit / LoadSeenCommit).
  • Block parts for a lagging peer come from the block store, one per tick, with a quiet gap of catchupResendInterval between passes (GossipBlockPartsForCatchup); a non-validator peer at the current height receives parts once it announced a commit (shouldBlockPartsBeGossiped).
  • VoteSetMaj23 for the stored commit is sent to validator peers that announced a catch-up commit round (GossipVoteSetMaj23).

Recovery and synchronisation

Write-ahead log

The consensus WAL (internal/consensus/wal.go; file layout in WAL) records, per height:

RecordWritten byDurability
msgInfo — proposal, block part, vote or commit, with the peer id and receive timewalMiddleware (internal/consensus/msg_handlers.go)peer messages Write (buffered); the node's own messages WriteSync (fsync), panic on failure
msgInfo — round skip certificate, once verified and newsAddRoundSkipCertificateAction.ExecuteWrite; a failure is only logged
timeoutInfo — every tockState.receiveRoutineWrite
EventDataRoundState — every step changeEventPublisher.PublishNewRoundStepEventWrite
EndHeightMessage{h}ApplyCommitAction.Execute, after SaveBlock and before FinalizeBlockWriteSync, panic on failure

verificationBudgetMiddleware wraps walMiddleware, so a message dropped for budget reasons leaves no WAL record. Proposer heartbeats and round skip votes are never written (livenessMessage). Before signing a proposal or a vote the WAL is flushed and fsynced (EnterProposeAction.Execute, voteSigner.signVote) so that replay reproduces the same message for the private validator's double-sign protection. A failed sync stops the vote (signVote returns the error, nothing is signed) but not the proposal: EnterProposeAction.Execute logs it and proposes anyway. An empty WAL starts with EndHeightMessage{0} (BaseWAL.OnStart).

Start-up: ABCI handshake

Handshaker.Handshake (internal/consensus/replay.go) calls ABCI Info and BlockReplayer.Replay (internal/consensus/replayer.go) with the application's (LastBlockHeight, LastBlockAppHash):

  1. If the application height is 0: InitChain with the genesis validator set, parameters, InitialCoreHeight; when Tenderdash itself is at height 0 the returned validator set, parameters, app hash and optional genesis time become the initial state (execInitChain).
  2. replayState.validate: the application may not be ahead of the block store; the state may not be ahead of the store; the store may be at most one block ahead of the state; a truncated store must not have pruned the blocks the application needs.
  3. Cases (syncStateIfItIsEqualStore, syncStateIfItIsOneAheadOfStore):
Store vs stateApp vs stateAction
equalbehindreplay stored blocks into the application: per block ProcessProposal(round = commit.Round), VerifyCommitExtensions, FinalizeBlock (replayBlock); the resulting app hash must equal the block's; a rejected commit vector stops replay (commitExtensionsReplayError)
equalequalapp hash must equal state.LastAppHash
store = state + 1behind statereplay all but the last block as above, then apply the last one through BlockExecutor.ApplyBlock with the real application
store = state + 1equal stateapply the last block with the real application (syncStateAt)
store = state + 1equal storethe application already finalized it: apply the last block against a mock application that replays the saved ABCIResponses (newMockProxyApp), so that state and store catch up without re-executing

Start-up: WAL catch-up

State.OnStart (internal/consensus/state.go) rebuilds LastCommit from the stored seen commit (updateStateFromStore), opens the WAL, starts the ticker and, unless doWALCatchup was disabled by block sync, runs State.catchupReplay:

  • the WAL must not contain #ENDHEIGHT h for the current height;
  • it must contain #ENDHEIGHT h − 1 (0 at InitialHeight);
  • every record after it is replayed through readReplayMessage: step events are only logged (except as below), msgInfos are dispatched with fromReplay (no peer budgets, no evictions), tocks go through handleTimeout. With WalSkipRoundsToLast a replayed proposal for a later round advances the round directly; on CONSENSUS_VERSION_2 chains that skip rounds the setting is ignored, and a replayed Propose step event re-enters the recorded round and propose step (State.replaySkipsRounds, State.readReplayMessage; Recovery);
  • replay mode: the node does not propose (EnterProposeAction.Execute) and skips timeliness checks (isValidForPrevote), but it may still attempt to sign votes: replayed messages drive the normal actions, and neither prevoter.Do nor voteSigner.signVote checks replay mode, so a replayed step can also produce a vote that was never signed before the crash. Its earlier votes also come back from the WAL as internal messages. Conflicting block votes are prevented by the signer's persisted state: FilePV reuses the stored block signature for identical sign bytes at the same height/round/step (re-signing only precommit extensions) and refuses regressions and conflicting sign bytes (see WAL). DashCoreSignerClient keeps no such state in Tenderdash;
  • a corrupted WAL is backed up to <wal>.CORRUPTED, repaired by truncating at the first undecodable record (repairWalFile), and replay is retried once; other errors are logged and the node starts anyway.

Then receiveRoutine starts and ScheduleRound0 arms the first tock.

Crash windows

Crash pointRecovery
Before SaveBlockWAL replay re-delivers the height's messages and timeouts; the node re-reaches the same step
After SaveBlock, before FinalizeBlock completed (with or without EndHeightMessage{h})handshake case "store = state + 1, app = state": ApplyBlock with the real application. Without the marker, catchupReplay(h + 1) then fails with "WAL does not contain #ENDHEIGHT for h"; the error is not a corruption error, so State.OnStart logs it and starts without replay
After FinalizeBlock, before stateStore.Savehandshake case "app = store": mock replay from saved ABCIResponses (BlockExecutor.FinalizeBlock saves them before calling the application)

Block-sync handover

Reactor.SwitchToConsensus(state, skipWAL, targetHeight) (internal/consensus/reactor.go):

  1. If targetHeight > state.LastBlockHeight (peers reported more blocks than were applied), arm catchupTracker so that EnterPropose refuses to propose until the node passes targetHeight or maxCatchupSuppression (10 min) elapses; a block built from present-day application state must not be proposed into a height the network committed long ago.
  2. Rebuild LastCommit from the stored seen commit; updateToState; start State; publish the step; mark syncing complete; with skipWAL, disable WAL catch-up.

While the reactor is still in sync mode (WaitSync) it ignores data, vote and vote-set-bits messages.

A node that restarts mid-height and re-joins through consensus (no block sync) relies on peers' commit gossip for the heights it missed (Commit gossip): the proposer-selector reconstruction reads proposers from the block store to resynchronise the round-robin (proposerFromStore).

Known issues and discrepancies

Behaviour found while deriving this specification that looks like a bug, diverges from the Tendermint algorithm, or is otherwise surprising. Each entry describes and cites; none is fixed here. "Inferred" marks conclusions drawn from reading the code without a reproducing test.

An untimely ValidBlock is never re-proposed

Proposaler.Create (internal/consensus/state_proposaler.go) re-proposes ValidBlock only when checkValidBlock finds it timely for ValidRound (ValidBlock.IsTimely(ValidBlockRecvTime, SynchronyParams, ValidRound)); otherwise it builds a fresh block and proposes it with POLRound = -1. The PBTS algorithm checks timeliness only for proposals without a POL round, so a block that already gathered a polka would be re-proposed regardless of its timestamp.

A proposer sees its ValidBlock as untimely when it learned the block late. updateValidBlock dates a block with tmtime.Now() when the node learned it without a matching proposal, which puts the receive time well after block.Time. The proposer logs proposal block is not timely at ERROR level when this happens.

Effect: validators locked on ValidBlock prevote nil for the fresh block, since a lock is cleared only at a new height (see No unlock within a height). If the voting power locked on that block exceeds TotalVotingPower - QuorumVotingThresholdPower and every subsequent proposer sees it as untimely, no block can gather a polka and the height stalls (inferred; no reproducing test).

PrecommitWait step is never entered

RoundStepPrecommitWait exists in RoundStepType (internal/consensus/types/round_state.go) and is used as the step of the precommit-wait timeout and in peer-state comparisons (pickVoteForGossip), but EnterPrecommitWaitAction.Execute (internal/consensus/state_enter_wait.go) only sets TriggeredTimeoutPrecommit and never calls updateRoundStep(round, RoundStepPrecommitWait). A node waiting for the precommit timeout reports step Precommit. Consequences are cosmetic (events, metrics, peer state comparisons), and handleTimeout accepts the PrecommitWait tock because its step is greater than Precommit.

Proposer clock-wait can be swallowed by the ticker

EnterProposeAction.Execute reschedules a (h, r, NewRound) tock when the proposer's clock is behind state.LastBlockTime. The ticker (timeoutTicker.timeoutRoutine) ignores a tick whose step is ≤ the step of the last scheduled tick for the same height and round, and it does not forget the last tick after it fires. When EnterPropose itself was triggered by a (h, r, NewRound) tock — the empty-block interval (EnterNewRoundAction.Execute) or the transactions-available path (State.handleTxsAvailable) — the rescheduled tick is dropped and nothing else moves the node out of NewRound for that round. The node recovers only when the other validators' nil precommits reach +2/3-any (addVoteDispatchPrecommitMw → EnterPrecommitWait → EnterNewRound(r + 1)), costing the network one round. Preconditions: WaitForTxs() true, round 0 (on CONSENSUS_VERSION_2 chains also a round entered on a round skip certificate that waits, whose empty-block interval schedules the same NewRound tock), the node is the proposer, and its clock is behind the previous block time by more than the time since that block was produced. (Inferred; not reproduced.)

Precommit requires this round's proposal even when locked

EnterPrecommitAction.Execute, rule C-LockedNotProposed: a validator locked on B that sees a polka for B in round r' > LockedRound precommits nil unless B is also round r''s proposal block. The Tendermint paper (line 36) also requires a proposal for v in the same round, so this is faithful to the algorithm, but it is stricter than the pre-vote-extension Tendermint implementation, which relocked on the polka alone. The code's rationale is the application contract: ExtendVote is answered for the round the application processed last, and Dash Drive refuses any other. Liveness cost: a locked validator that missed the round's proposal cannot help commit B in that round.

No unlock within a height

LockedRound/LockedBlock are only replaced by a new lock at a later round or reset at the next height (StateData.updateLockedBlock, StateData.updateToState). The paper's unlock-on-nil-polka rule from the original Tendermint implementation is absent by design (TestStateLock_POLDoesNotUnlock), relying on P-Lock-POL to move the lock forward instead.

Commit threshold is configurable below the BFT bound

ValidatorSet.QuorumVotingThresholdPower honours VotingPowerThreshold, which may be lower than ⌊2n/3⌋ + 1 validators; several devnet LLMQ types have a 50 % threshold. The code only warns (ValidatorSet.BelowStrictThreshold, logged by StateData.updateToState). Safety of the protocol is not guaranteed below the strict floor.

Proposer selection v0 skips the first validator after quorum rotation

heightProposerSelector.proposerFromStore (internal/consensus/versioned/selectproposer/height_proposer.go) deliberately sets the proposer to index 1 after a validator-set change ("bug preserved to avoid breaking consensus"); the v1 selector uses index 0. See Proposer selection.

FilePV repeated proposal returns sign bytes as the signature

FilePV.signProposal (privval/file.go) checks for an identical persisted height/round/step and identical canonical proposal bytes. In that branch it assigns proposal.Signature = lss.BlockSignBytes, rather than lss.BlockSignature. The vote path instead reuses BlockSignature. The repeated-proposal path therefore returns serialized sign bytes in a signature field; verification failure is the inferred effect (no reproducing test). This differs from a guarantee of successful signature reuse during recovery. WAL replay itself does not create proposals.

Code/comment drift and dead code

  • BlockExecutor.ProcessProposal (internal/state/execution.go) checks len(resp.AppHash) != appHashSize after NewStateChangeset, but ResponseProcessProposal.Validate already returned for any size other than crypto.DefaultAppHashSize; the later check is unreachable unless SetAppHashSize changes the size, which no non-test caller does.

  • EnterNewRoundAction's doc comment mentions timeoutNewHeight, timeoutCommit and SkipTimeoutCommit, none of which exist; round 0 starts at CommitTime (StateData.updateToState, roundScheduler.ScheduleRound0).

  • EnterPrevoteEvent.AllowOldBlocks is set by ProposalCompletedAction and never read (internal/consensus/state_enter_prevote.go).

  • ErrInvalidProposalNotSet and ErrAddingVote (internal/consensus/state.go) have no non-comment users.

  • ApplyCommitAction.Execute handles commit == nil, but every caller passes a commit.

  • config.ConsensusConfig.DoubleSignCheckHeight is exposed as a CLI flag (cmd/tenderdash/commands/run_node.go) but no consumer was found under internal/consensus (inferred from grep).

  • blockExecutor.create returns (nil, nil) when LastCommit == nil above InitialHeight ("shouldn't happen"); Proposaler.Create would then dereference a nil block in makeProposal. Unreachable in normal operation because updateToState panics on a missing LastCommit.

  • EnterPrecommitAction.Execute logs routine nil precommits (no polka, nil polka) at error level, including the ordinary timeout path.

Non-fatal optimistic-concurrency errors

State.receiveRoutine persists StateData after each handler with a version check (StateDataStore.update); a mismatch is only logged, after the handler's side effects (signed votes, WAL records) already happened. No concurrent writer exists today, so the check is a latent guard rather than a protection.

ABCI

Introduction

ABCI (Application Blockchain Interface) connects Tenderdash's consensus engine to the Application's state machine. The API consists of a set of methods, each with a corresponding Request and Response message type.

The methods are always initiated by Tenderdash. The Application implements its logic for handling all ABCI methods. Thus, Tenderdash always sends the Request* messages and receives the Response* messages in return.

All ABCI messages and methods are defined in protocol buffers. This allows Tenderdash to run with applications written in many programming languages.

This specification is split as follows:

  • Overview and basic concepts - interface's overview and concepts needed to understand other parts of this specification.
  • Methods - complete details on all ABCI methods and message types.
  • Requirements for the Application - formal requirements on the Application's logic to ensure Tenderdash properties such as liveness. These requirements define what Tenderdash expects from the Application; second part on managing ABCI application state and related topics.
  • Tenderdash's expected behavior - specification of how the different ABCI methods may be called by Tenderdash. This explains what the Application is to expect from Tenderdash.
  • Client and Server - for those looking to implement their own ABCI application servers

Application Requirements

This section specifies what Tenderdash expects from the Application. It is structured as a set of formal requirement that can be used for testing and verification of the Application's logic.

Let $p$ and $q$ be two different correct proposers in rounds $r_p$ and $r_q$ respectively, in height $h$. Let $s_{p,h-1}$ be $p$'s Application's state committed for height $h-1$. Let $v_p$ (resp. $v_q$) be the block that $p$'s (resp. $q$'s) Tenderdash passes on to the Application via RequestPrepareProposal as proposer of round $r_p$ (resp $r_q$), height $h$, also known as the raw proposal. Let $v'_p$ (resp. $v'_q$) the possibly modified block $p$'s (resp. $q$'s) Application returns via ResponsePrepareProposal to Tenderdash, also known as the prepared proposal.

Process $p$'s prepared proposal can differ in two different rounds where $p$ is the proposer.

  • Requirement 1 [PrepareProposal, header-changes] When the blockchain is in same-block execution mode, $p$'s Application provides values for the following parameters in ResponsePrepareProposal: AppHash, TxResults, ConsensusParams, ValidatorSetUpdate. Provided values for ConsensusParams and ValidatorSetUpdate MAY be empty to denote that the Application wishes to keep the current values.

Parameters AppHash, TxResults, ConsensusParams, and ValidatorSetUpdate are used by Tenderdash to compute various hashes in the block header that will finally be part of the proposal.

In practical terms, Requirement 1 imply that Tenderdash will panic if the Application is in same-block execution mode and does not provide values for AppHash, TxResults, ConsensusParams, and ValidatorSetUpdate.

  • Requirement 2 [PrepareProposal, timeliness] If $p$'s Application fully executes prepared blocks in PrepareProposal and the network is in a synchronous period while processes $p$ and $q$ are in $r_p$, then the value of TimeoutPropose at $q$ must be such that $q$'s propose timer does not time out (which would result in $q$ prevoting nil in $r_p$).

Full execution of blocks at PrepareProposal time stands on Tenderdash's critical path. Thus, Requirement 2 ensures the Application will set a value for TimeoutPropose such that the time it takes to fully execute blocks in PrepareProposal does not interfere with Tenderdash's propose timer.

  • Requirement 3 [PrepareProposal, tx-size] When $p$'s Application calls ResponsePrepareProposal, the total size in bytes of the transactions returned does not exceed RequestPrepareProposal.max_tx_bytes.

Busy blockchains might seek to maximize the amount of transactions included in each block. Under those conditions, Tenderdash might choose to increase the transactions passed to the Application via RequestPrepareProposal.txs beyond the RequestPrepareProposal.max_tx_bytes limit. The idea is that, if the Application drops some of those transactions, it can still return a transaction list whose byte size is as close to RequestPrepareProposal.max_tx_bytes as possible. Thus, Requirement 3 ensures that the size in bytes of the transaction list returned by the application will never cause the resulting block to go beyond its byte limit.

  • Requirement 4 [PrepareProposal, ProcessProposal, coherence]: For any two correct processes $p$ and $q$, if $q$'s Tenderdash calls RequestProcessProposal on $v'_p$, $q$'s Application returns Accept in ResponseProcessProposal.

Requirement 4 makes sure that blocks proposed by correct processes always pass the correct receiving process's ProcessProposal check. On the other hand, if there is a deterministic bug in PrepareProposal or ProcessProposal (or in both), strictly speaking, this makes all processes that hit the bug byzantine. This is a problem in practice, as very often validators are running the Application from the same codebase, so potentially all would likely hit the bug at the same time. This would result in most (or all) processes prevoting nil, with the serious consequences on Tenderdash's liveness that this entails. Due to its criticality, Requirement 4 is a target for extensive testing and automated verification.

  • Requirement 5 [ProcessProposal, determinism-1]: ProcessProposal is a (deterministic) function of the current state and the block that is about to be applied. In other words, for any correct process $p$, and any arbitrary block $v'$, if $p$'s Tenderdash calls RequestProcessProposal on $v'$ at height $h$, then $p$'s Application's acceptance or rejection exclusively depends on $v'$ and $s_{p,h-1}$.

  • Requirement 6 [ProcessProposal, determinism-2]: For any two correct processes $p$ and $q$, and any arbitrary block $v'$, if $p$'s (resp. $q$'s) Tenderdash calls RequestProcessProposal on $v'$ at height $h$, then $p$'s Application accepts $v'$ if and only if $q$'s Application accepts $v'$. Note that this requirement follows from Requirement 5 and the Agreement property of consensus.

Requirements 5 and 6 ensure that all correct processes will react in the same way to a proposed block, even if the proposer is Byzantine. However, ProcessProposal may contain a bug that renders the acceptance or rejection of the block non-deterministic, and therefore prevents processes hitting the bug from fulfilling Requirements 5 or 6 (effectively making those processes Byzantine). In such a scenario, Tenderdash's liveness cannot be guaranteed. Again, this is a problem in practice if most validators are running the same software, as they are likely to hit the bug at the same point. There is currently no clear solution to help with this situation, so the Application designers/implementors must proceed very carefully with the logic/implementation of ProcessProposal. As a general rule ProcessProposal should always accept the block.

A correct validator broadcasts at most one precommit per height and round. ExtendVote can also run after a rejected commit to discover the expected vector, including on an observer. Applications must support repeated calls and return the same threshold-recoverable subset for the same block, round and committed state. See expected call sequences.

Let $e^r_p$ be the set of vote extensions that the Application of a correct process $p$ returns via ResponseExtendVote in round $r$, height $h$. Let $w^r_p$ be the proposed block that $p$'s Tenderdash passes to the Application via RequestExtendVote in round $r$, height $h$.

  • Requirement 7 [ExtendVote, VerifyVoteExtension, coherence]: For any two correct processes $p$ and $q$, if $q$ receives $e^r_p$ from $p$ in height $h$, $q$'s Application returns Accept in ResponseVerifyVoteExtension.

Requirement 7 constrains the creation and handling of vote extensions in a similar way as Requirement 4 contrains the creation and handling of proposed blocks. Requirement 7 ensures that extensions created by correct processes always pass the VerifyVoteExtension checks performed by correct processes receiving those extensions. However, if there is a (deterministic) bug in ExtendVote or VerifyVoteExtension (or in both), we will face the same liveness issues as described for Requirement 4, as Precommit messages with invalid vote extensions will be discarded.

  • Requirement 8 [VerifyVoteExtension, determinism-1]: VerifyVoteExtension is a (deterministic) function of the current state, the vote extensions received, and the prepared proposal that the extensions refers to. In other words, for any correct process $p$, and any arbitrary vote extension $e$, and any arbitrary block $w$, if $p$'s (resp. $q$'s) Tenderdash calls RequestVerifyVoteExtension on $e$ and $w$ at height $h$, then $p$'s Application's acceptance or rejection exclusively depends on $e$, $w$ and $s_{p,h-1}$.

  • Requirement 9 [VerifyVoteExtension, determinism-2]: For any two correct processes $p$ and $q$, and any arbitrary vote extensions $e$, and any arbitrary block $w$, if $p$'s (resp. $q$'s) Tenderdash calls RequestVerifyVoteExtension on $e$ and $w$ at height $h$, then $p$'s Application accepts $e$ if and only if $q$'s Application accepts $e$. Note that this requirement follows from Requirement 8 and the Agreement property of consensus.

Requirements 8 and 9 ensure that the validation of vote extensions will be deterministic at all correct processes. Requirements 8 and 9 protect against arbitrary vote extensions data from Byzantine processes similarly to Requirements 5 and 6 and proposed blocks. Requirements 8 and 9 can be violated by a bug inducing non-determinism in VerifyVoteExtension. In this case liveness can be compromised. Extra care should be put in the implementation of ExtendVote and VerifyVoteExtension and, as a general rule, VerifyVoteExtension should always accept the vote extensions of a precommit.

Commit verification, a call with an empty validator_pro_tx_hash, is the exception: the Application must reject a commit whose extension vector differs from the threshold-recoverable subset of the extensions it expects for the block, or an altered vector reaches FinalizeBlock. Threshold recovery requires the contributing validators to sign identical extension content, so a valid recovered extension from correct processes matches that expected vector. Rejecting an altered vector therefore costs no liveness; a deterministic bug in the check, however, halts the chain, because no commit for the block can pass it. See VerifyVoteExtension for what Tenderdash does on each path when a commit is rejected.

  • Requirement 10 [all, no-side-effects]: $p$'s calls to RequestPrepareProposal, RequestProcessProposal, RequestExtendVote, and RequestVerifyVoteExtension at height $h$ do not modify $s_{p,h-1}$.

  • Requirement 11 [ExtendVote, FinalizeBlock, non-dependency]: for any correct process $p$, and any vote extensions $e$ that $p$ received at height $h$, the computation of $s_{p,h}$ does not depend on $e$.

The call to correct process $p$'s RequestFinalizeBlock at height $h$, with block $v_{p,h}$ passed as parameter, creates state $s_{p,h}$.

PrepareProposal and ProcessProposal compute the transaction results $T_{p,h}$ for the current block before consensus decides it. FinalizeBlock persists the selected candidate and returns no transaction results.

  • Requirement 12 [FinalizeBlock, determinism-1]: For any correct process $p$, $s_{p,h}$ exclusively depends on $s_{p,h-1}$ and $v_{p,h}$.

  • Requirement 13 [FinalizeBlock, determinism-2]: For any correct process $p$, the contents of $T_{p,h}$ exclusively depend on $s_{p,h-1}$ and $v_{p,h}$.

Note that Requirements 12 and 13, combined with Agreement property of consensus ensure the Application state evolves consistently at all correct processes.

Finally, notice that neither PrepareProposal nor ExtendVote have determinism-related requirements associated. Indeed, PrepareProposal is not required to be deterministic:

  • $v'p$ may depend on $v_p$ and $s{p,h-1}$, but may also depend on other values or operations.
  • $v_p = v_q \nRightarrow v'_p = v'_q$.

Likewise, ExtendVote can also be non-deterministic:

  • $e^r_p$ may depend on $w^r_p$ and $s_{p,h-1}$, but may also depend on other values or operations.
  • $w^r_p = w^r_q \nRightarrow e^r_p = e^r_q$

In Tenderdash this freedom holds for extensions that are not threshold-recoverable, and does not extend to threshold-recoverable ones (THRESHOLD_RECOVER and THRESHOLD_RECOVER_RAW): they are recovered into the commit, so every validator must produce the same ones. ExtendVote must also be free of side effects, and the threshold-recoverable subset of its response must be exactly the vector the Application accepts in a commit-level VerifyVoteExtension for the same block, height and round: after such a VerifyVoteExtension rejects a commit, following ProcessProposal of that block and round, Tenderdash calls ExtendVote on any node to learn that vector (see VerifyVoteExtension).

Connections

Tenderdash uses one application client for consensus, mempool, query and state-sync calls. Its transport determines concurrency: the reference socket server processes one connection sequentially, while gRPC can execute calls concurrently over one connection. Explicitly configured routed clients may use multiple underlying connections. Local clients call the application directly.

Applications MUST synchronize state shared by concurrent calls, including calls on different methods or connections. Waiting for one response does not block other callers. Queries must read committed state. See Client and Server for transport ordering and gRPC concurrency settings.

Outline

  1. Outline
  2. Application execution
  3. Methods overview
    1. Consensus/block execution methods
    2. Mempool methods
    3. Info methods
    4. State-sync methods
    5. Other methods
  4. Same-block execution
    1. Tenderdash timeouts in same-block execution
  5. Determinism
  6. Errors
    1. CheckTx
    2. ExecTxResult
    3. Query
  7. Events
  8. Evidence

Overview and basic concepts

Application execution

↑ Back to Outline

The Application constructs and executes candidate blocks through PrepareProposal and ProcessProposal. It supplies vote extensions through ExtendVote and checks peer votes and commits through VerifyVoteExtension. FinalizeBlock persists the decided candidate after consensus accepts the commit.

Methods overview

↑ Back to Outline

Methods can be classified into four categories: consensus, mempool, info, and state-sync.

Consensus/block execution methods

The first time a new blockchain is started, Tenderdash calls InitChain. From then on, method FinalizeBlock is executed at the end of each block, resulting in an updated Application state. During consensus execution of a block height, before method FinalizeBlock is called, methods PrepareProposal, ProcessProposal, ExtendVote, and VerifyVoteExtension may be called several times. See Tenderdash's expected behavior for details on the possible call sequences of these methods.

  • InitChain: This method initializes the blockchain. Tenderdash calls it once upon genesis.

  • PrepareProposal: Constructs and executes a fresh candidate from mempool transactions, supplying the current block's commitments. A reusable valid block can be proposed without preparing a fresh candidate.

  • ProcessProposal: Executes a candidate deterministically and returns its status and commitments. Processing precedes the lock-based voting decision and can repeat when validating a commit from another round.

  • ExtendVote: Supplies the extensions for a non-nil precommit. It can also run on an observer after a rejected commit, without signing a vote.

  • VerifyVoteExtension: Checks a peer's precommit or a commit's recovered vector. At commit level, the Application must reject a vector that differs from the expected threshold-recoverable subset for that block and round. The methods contract describes the distinct rejection paths.

  • FinalizeBlock: It delivers a decided block to the Application to persist the candidate for the block and round processed last. The current block's application and result hashes were returned by PrepareProposal/ProcessProposal and are already in its header. ResponseFinalizeBlock returns only retention and next-block scheduling hints, not execution results or events. See Commit and apply.

Mempool methods

  • CheckTx: This method allows the Application to validate transactions against its current state, e.g., checking signatures and account balances. If a transaction passes the validation, then Tenderdash adds it to its local mempool, discarding it otherwise. Tenderdash calls it when it receives a new transaction either coming from an external user or another node. Furthermore, Tenderdash can be configured to re-call CheckTx on remaining mempool transactions after FinalizeBlock.

Info methods

  • Info: Used to sync Tenderdash with the Application during a handshake that happens on startup.

  • Query: Clients can use this method to query the Application for information about the application state.

State-sync methods

State sync allows new nodes to rapidly bootstrap by discovering, fetching, and applying state machine snapshots instead of replaying historical blocks. For more details, see the state sync section.

New nodes will discover and request snapshots from other nodes in the P2P network. A Tenderdash node that receives a request for snapshots from a peer will call ListSnapshots on its Application. The Application returns the list of locally avaiable snapshots. Note that the list does not contain the actual snapshot but metadata about it: height at which the snapshot was taken, application-specific verification data and more (see snapshot data type for more details). After receiving a list of available snapshots from a peer, the new node can offer any of the snapshots in the list to its local Application via the OfferSnapshot method. The Application can check at this point the validity of the snapshot metadata.

Snapshots may be quite large and are thus broken into smaller "chunks" that can be assembled into the whole snapshot. Once the Application accepts a snapshot and begins restoring it, Tenderdash will fetch snapshot "chunks" from existing nodes. The node providing "chunks" will fetch them from its local Application using the LoadSnapshotChunk method.

As the new node receives "chunks" it will apply them sequentially to the local application with ApplySnapshotChunk. When all chunks have been applied, the Application's AppHash is retrieved via an Info query. To ensure that the sync proceeded correctly, Tenderdash compares the local Application's AppHash to the AppHash stored on the blockchain (verified via light client verification).

In summary:

  • ListSnapshots: Used by nodes to discover available snapshots on peers.

  • LoadSnapshotChunk: Used by Tenderdash to retrieve snapshot chunks from the application to send to peers.

  • OfferSnapshot: When a node receives a snapshot from a peer, Tenderdash uses this method to offer the snapshot to the Application.

  • ApplySnapshotChunk: Used by Tenderdash to hand snapshot chunks to the Application.

Other methods

Additionally, there is a Flush method that is called on every connection, and an Echo method for debugging, which a waiting proposer also uses as a readiness probe.

More details on managing state across connections can be found in the section on ABCI Applications.

Same-block execution

↑ Back to Outline

Tenderdash uses same-block execution: PrepareProposal and ProcessProposal return the application hash and transaction results for the block being proposed. Those commitments are included in that block's header. Validator-set and parameter changes from height h take effect at h + 1.

The authoritative lifecycle is the primary consensus flow and validator/parameter transition.

Tenderdash timeouts in same-block execution

Proposal preparation and processing are synchronous and consume the round's proposal-time budget. Slow execution can cause other validators to prevote nil. Timeouts are already consensus parameters in ConsensusParams.Timeout; updates from a block apply at the next height. Local unsafe overrides can desynchronize nodes. See Timeouts and consensus parameters for the actual fields, defaults and scheduling rules.

Determinism

↑ Back to Outline

Application processing must produce deterministic consensus commitments for a block, round and last committed state. PrepareProposal constructs a candidate; ProcessProposal verifies its current-block application/results hashes and next-state commitments. Candidate changes must not replace committed application state until FinalizeBlock persists the decided candidate. There is no subsequent ABCI Commit call.

Processing can revisit the same block and round after another execution context has replaced it. Commit verification can also call ExtendVote without signing to discover an expected vector after rejection. These calls must not be treated as once-per-height callbacks; see expected call sequences. Extension generation and verification must not mutate committed application state.

Sources of non-determinism in applications may include:

  • Hardware failures
    • Cosmic rays, overheating, etc.
  • Node-dependent state
    • Random numbers
    • Time
  • Underspecification
    • Library version changes
    • Race conditions
    • Floating point numbers
    • JSON or protobuf serialization
    • Iterating through hash-tables/maps/dictionaries
  • External Sources
    • Filesystem
    • Network calls (eg. some external REST API service)

See #56 for original discussion.

Consensus commits only the consensus-relevant subset of transaction results: Code, Data and GasUsed form ResultsHash. Events and diagnostic fields are not part of that hash. See Header and Events; ResponseFinalizeBlock has no transaction results or events.

Errors

↑ Back to Outline

ResponseQuery, ResponseCheckTx and ExecTxResult entries carry application Code/Codespace values. Code zero means successful application processing; a nonzero transaction code is not the same as rejecting a whole proposal.

ProcessProposal.status controls proposal acceptance, while VerifyVoteExtension.status controls vote/commit-vector acceptance. The nil-vote versus panic distinction and commit rejection paths are part of the consensus contract. ABCI transport errors and malformed responses must not be assumed to act like a nonzero transaction code.

CheckTx

When Tenderdash receives a ResponseCheckTx with a non-zero Code, the associated transaction will not be added to Tenderdash's mempool or it will be removed if it is already included.

ExecTxResult (PrepareProposal and ProcessProposal)

ResponsePrepareProposal.tx_results and ResponseProcessProposal.tx_results contain one result for each included transaction. The consensus-relevant result fields contribute to the current block's ResultsHash, which is checked against the Application's answer before voting. ResponseFinalizeBlock has no tx_results field. See ExecTxResult.

Query

When Tenderdash receives a ResponseQuery with a non-zero Code, this code is returned directly to the client that initiated the query.

Events

↑ Back to Outline

ResponseCheckTx has no events field. For block execution, ResponseProcessProposal.events contains block events and each ExecTxResult can contain transaction events. These results are retained for publication when the block is finalized; ResponseFinalizeBlock has no events field. Applications may respond to these ABCI methods with a set of events. Events allow applications to associate metadata about ABCI method execution with the transactions and blocks this metadata relates to. Events returned via these ABCI methods do not impact Tenderdash consensus in any way and instead exist to power subscriptions and queries of Tenderdash state.

An Event contains a type and a list of EventAttributes, which are key-value string pairs denoting metadata about what happened during the method's (or transaction's) execution. Event values can be used to index transactions and blocks according to what happened during their execution.

Each event has a type which is meant to categorize the event for a particular Response* or Tx. A Response* or Tx may contain multiple events with duplicate type values, where each distinct entry is meant to categorize attributes for a particular event. Every key and value in an event's attributes must be UTF-8 encoded strings along with the event type itself.

message Event {
  string                  type       = 1;
  repeated EventAttribute attributes = 2;
}

The attributes of an Event consist of a key, a value, and an index flag. The index flag notifies the Tenderdash indexer to index the attribute. The value of the index flag is non-deterministic and may vary across different nodes in the network.

message EventAttribute {
  string key = 1;
  string value = 2;
  bool  index = 3;  // nondeterministic
}

Example:

 abci.ExecTxResult{
  // ...
 Events: []abci.Event{
  {
   Type: "validator.provisions",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: true},
    abci.EventAttribute{Key: "amount", Value: "...", Index: true},
    abci.EventAttribute{Key: "balance", Value: "...", Index: true},
   },
  },
  {
   Type: "validator.provisions",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: true},
    abci.EventAttribute{Key: "amount", Value: "...", Index: false},
    abci.EventAttribute{Key: "balance", Value: "...", Index: false},
   },
  },
  {
   Type: "validator.slashed",
   Attributes: []abci.EventAttribute{
    abci.EventAttribute{Key: "address", Value: "...", Index: false},
    abci.EventAttribute{Key: "amount", Value: "...", Index: true},
    abci.EventAttribute{Key: "reason", Value: "...", Index: true},
   },
  },
  // ...
 },
}

Evidence

↑ Back to Outline

Tenderdash's security model relies on the use of "evidence". Evidence is proof of malicious behavior by a network participant. It is the responsibility of Tenderdash to detect such malicious behavior. When malicious behavior is detected, Tenderdash will gossip evidence of the behavior to other nodes and commit the evidence to the chain once it is verified by all validators. This evidence will then be passed on to the Application through ABCI. It is the responsibility of the Application to handle the evidence and exercise punishment.

EvidenceType has the following protobuf format:

enum EvidenceType {
  UNKNOWN               = 0;
  DUPLICATE_VOTE        = 1;
  LIGHT_CLIENT_ATTACK   = 2;
}

There are two forms of evidence: Duplicate Vote and Light Client Attack. More information can be found in either data structures or accountability

Client and Server

This section is for those looking to implement their own ABCI Server, perhaps in a new programming language.

You are expected to have read all previous sections of ABCI specification, namely Basic Concepts, Methods, Application Requirements, and Expected Behavior.

Message Protocol and Synchrony

The message protocol consists of pairs of requests and responses defined in the protobuf file.

Some messages have no fields, while others may include byte-arrays, strings, integers, or custom protobuf types.

For more details on protobuf, see the documentation.

Tenderdash creates one application client and delegates the request classes to it. Ordinary socket and gRPC configurations do not create separate consensus, mempool, query and state-sync connections automatically. A routed client can use multiple underlying clients when explicitly configured.

Each synchronous call waits for its own response. This does not serialize calls from independent callers. Execution order and concurrency depend on the transport and server implementation; see Connections.

Server Implementations

To use ABCI in your programming language of choice, there must be an ABCI server in that language. Tenderdash provides three Go implementations:

  • In-process
  • ABCI-socket
  • gRPC

See the server implementations and ABCI CLI guide. Out-of-process servers must implement Tenderdash's protobuf schema and application requirements.

In Process

The simplest implementation uses function calls in Golang. This means ABCI applications written in Golang can be linked with Tenderdash and run as a single binary. The local client delegates directly to application methods without a shared execution lock, so the application must synchronize concurrent access to its state.

GRPC

If you are not using Golang, but GRPC is available in your language, this is the easiest approach, though it will have significant performance overhead.

Please check GRPC's documentation to know to set up the Application as an ABCI GRPC server.

The client multiplexes calls over one grpc.ClientConn; application methods can execute concurrently. The abci.grpc-concurrency configuration limits streams per method or through the default * rule. Its default is {"*": 0}, which imposes no limit. A limit for one method does not serialize it with methods using a different limit. Protect state shared by concurrent calls.

Socket

Tenderdash's socket-based ABCI interface is an asynchronous, raw socket server which provides ordered message passing over unix or tcp. Messages are serialized using Protobuf3. Each message is prefixed by its byte length encoded as an unsigned protobuf varint. For example, a four-byte message DEADBEEF is sent as 04DEADBEEF. The receiver reads the varint length and then that many message bytes. See the codec.

This framing does not apply to gRPC. The socket client sends requests and matches responses in order on its connection. Tenderdash's reference socket server executes requests sequentially per connection, while separate connections can execute concurrently. A compatible socket server must preserve response order; it must not assume a fixed number of role-specific connections.

Client

There are currently two use-cases for an ABCI client. One is testing tools that allow ABCI requests to be sent to the actual application via command line. An example of this is abci-cli, which accepts CLI commands to send corresponding ABCI requests. The other is a consensus engine, such as Tenderdash, which makes ABCI requests to the application as prescribed by the consensus algorithm used.

Methods

General, mempool and state-sync methods

Echo

  • Request:
    • Message (string): A string to echo back
  • Response:
    • Message (string): The input string
  • Usage:
  • Echo a string to test an abci client/server implementation
  • From consensus version 2, the proposer of a round that waits for transactions probes the backends that receive PrepareProposal with Echo before each heartbeat, once it is not held back after block sync. At most one probe is in flight; an unanswered probe is reused by later heartbeats rather than repeated. A heartbeat is sent only if the backends answer within the heartbeat interval. See Proposer liveness.

Flush

  • Usage:
  • Signals that messages queued on the client should be flushed to the server. It is called periodically by the client implementation to ensure asynchronous requests are actually sent, and is called immediately to make a synchronous request, which returns when the Flush response comes back.

Info

  • Request:

    NameTypeDescriptionField Number
    versionstringThe Tenderdash software semantic version1
    block_versionuint64The Tenderdash Block Protocol version2
    p2p_versionuint64The Tenderdash P2P Protocol version3
    abci_versionstringThe Tenderdash ABCI semantic version4
  • Response:

    NameTypeDescriptionField Number
    datastringSome arbitrary information1
    versionstringThe application software semantic version2
    app_versionuint64The application protocol version3
    last_block_heightint64Latest block finalized and persisted by the application4
    last_block_app_hashbytesApplication hash of that finalized block5
  • Usage:

  • Return information about the application state.

  • Used to sync Tenderdash with the application during a handshake that happens on startup.

  • During the startup handshake, app_version initializes the local state's application version only if no block has been committed yet. It also supplies this node's advertised version, used as Header.ProposedAppVersion when the node proposes a block. Another proposer may advertise a different version.

  • Header.Version.App is the version used for the proposed block's execution. A nonzero ResponsePrepareProposal.app_version overrides its inherited value. Changing Info.app_version on an existing chain does not by itself change the committed execution version.

  • last_block_app_hash and last_block_height describe the state persisted by FinalizeBlock; there is no ABCI Commit method. See ABCI handshake for replay and crash-recovery behavior.

Note: Semantic version is a reference to semantic versioning. Semantic versions in info will be displayed as X.X.x.

InitChain

  • Request:

    NameTypeDescriptionField Number
    timegoogle.protobuf.TimestampGenesis time1
    chain_idstringID of the blockchain.2
    consensus_paramsConsensusParamsInitial consensus-critical parameters.3
    validator_setValidatorSetUpdateInitial genesis validator set, sorted by voting power.4
    app_state_bytesbytesSerialized initial application state. JSON bytes.5
    initial_heightint64Height of the initial block (typically 1).6
    initial_core_heightuint32Height of the initial core chainlock.7
  • Response:

    NameTypeDescriptionField Number
    consensus_paramsConsensusParamsInitial consensus-critical parameters (optional).1
    app_hashbytesInitial application hash, 32 bytes.2
    validator_set_updateValidatorSetUpdateInitial validator set update.3
    next_core_chain_lock_updateCoreChainLockInitial core chain lock update (optional).4
    initial_core_heightuint32Initial core chain lock height.5
    genesis_timegoogle.protobuf.TimestampGenesis time override (optional).6
  • Usage:

  • Called once upon genesis.

  • If ResponseInitChain.validator_set_update.validator_updates is empty, genesis validator membership is retained; quorum-key and quorum-hash updates can still be supplied in validator_set_update.

  • If it is not empty, those validators replace the initial membership supplied through RequestInitChain.validator_set.

  • This allows the app to decide if it wants to accept the initial validator set proposed by Tenderdash (ie. in the genesis file), or if it wants to use a different one (perhaps computed based on some application specific information in the genesis file).

Query

  • Request:

    NameTypeDescriptionField Number
    databytesRaw query bytes. Can be used with or in lieu of Path.1
    pathstringPath field of the request URI. Can be used with or in lieu of data. Apps MUST interpret /store as a query by key on the underlying store. The key SHOULD be specified in the data field. Apps SHOULD allow queries over specific types like /accounts/... or /votes/...2
    heightint64The block height for which you want the query (default=0 returns data for the latest committed block). Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height3
    proveboolReturn Merkle proof with response if possible4
  • Response:

    NameTypeDescriptionField Number
    codeuint32Response code.1
    logstringThe output of the application's logger. May be non-deterministic.2
    infostringAdditional information. May be non-deterministic.3
    indexint64The index of the key in the tree.4
    keybytesThe key of the matching data.5
    valuebytesThe value of the matching data.6
    proof_opsProofOpsSerialized proof for the value data, if requested, to be verified against the app_hash for the given Height.7
    heightint64The block height from which data was derived. Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height8
    codespacestringNamespace for the code.9
  • Usage:

  • Query for data from the application at current or past height.

  • Optionally return Merkle proof.

  • Merkle proof includes self-describing type field to support many types of Merkle trees and encoding formats.

CheckTx

  • Request:

    NameTypeDescriptionField Number
    txbytesThe request transaction bytes1
    typeCheckTxTypeOne of CheckTx_New or CheckTx_Recheck. CheckTx_New is the default and means that a full check of the tranasaction is required. CheckTx_Recheck types are used when the mempool is initiating a normal recheck of a transaction.2
  • Response:

    NameTypeDescriptionField Number
    codeuint32Response code.1
    databytesResult bytes, if any.2
    infostringAdditional information. May be non-deterministic.3
    gas_wantedint64Amount of gas requested for transaction.4
    codespacestringNamespace for the code.5
    senderstringThe transaction's sender (e.g. the signer)6
    priorityint64The transaction's priority (for mempool ordering)7
  • Usage:

  • Technically optional - not involved in processing blocks.

  • Guardian of the mempool: every node runs CheckTx before letting a transaction into its local mempool.

  • The transaction may come from an external user or another node

  • CheckTx validates the transaction against the current state of the application, for example, checking signatures and account balances, but does not apply any of the state changes described in the transaction.

  • Transactions where ResponseCheckTx.Code != 0 will be rejected - they will not be broadcast to other nodes or included in a proposal block.

  • Tenderdash attributes no other value to the response code

ListSnapshots

  • Request:

    NameTypeDescriptionField Number

    Empty request asking the application for a list of snapshots.

  • Response:

    NameTypeDescriptionField Number
    snapshotsrepeated SnapshotList of local state snapshots.1
  • Usage:

  • Used during state sync to discover available snapshots on peers.

  • See Snapshot data type for details.

LoadSnapshotChunk

  • Request:

    NameTypeDescriptionField Number
    heightuint64The height of the snapshot the chunk belongs to.1
    versionuint32The application-specific snapshot version.2
    chunk_idbytesApplication-defined identifier of the requested chunk.3
  • Response:

    NameTypeDescriptionField Number
    chunkbytesThe binary chunk contents, in an arbitray format. Chunk messages cannot be larger than 16 MB including metadata, so 10 MB is a good starting point.1
  • Usage:

  • Used during state sync to retrieve snapshot chunks from peers.

  • Chunk identifiers are bytes, not numeric indexes. The initial identifier is Snapshot.hash; the application supplies further identifiers through ResponseApplySnapshotChunk.next_chunks.

OfferSnapshot

  • Request:

    NameTypeDescriptionField Number
    snapshotSnapshotThe snapshot offered for restoration.1
    app_hashbytesThe light client-verified app hash for this height, from the blockchain.2
  • Response:

    NameTypeDescriptionField Number
    resultResultThe result of the snapshot offer.1

Result

  enum Result {
    UNKNOWN       = 0;  // Unknown result, abort all snapshot restoration
    ACCEPT        = 1;  // Snapshot is accepted, start applying chunks.
    ABORT         = 2;  // Abort snapshot restoration, and don't try any other snapshots.
    REJECT        = 3;  // Reject this specific snapshot, try others.
    REJECT_FORMAT = 4;  // Reject all snapshots with this version, try others.
    REJECT_SENDER = 5;  // Reject all snapshots from all senders of this snapshot, try others.
  }
  • Usage:
  • OfferSnapshot is called when bootstrapping a node using state sync. The application may accept or reject snapshots as appropriate. Upon accepting, Tenderdash will retrieve and apply snapshot chunks via ApplySnapshotChunk. The application may also choose to reject a snapshot in the chunk response, in which case it should be prepared to accept further OfferSnapshot calls.
  • Only AppHash can be trusted, as it has been verified by the light client. Any other data can be spoofed by adversaries, so applications should employ additional verification schemes to avoid denial-of-service attacks. The verified AppHash is automatically checked against the restored application at the end of snapshot restoration.
  • For more information, see the Snapshot data type or the state sync section.

ApplySnapshotChunk

  • Request:

    NameTypeDescriptionField Number
    chunk_idbytesApplication-defined identifier of this chunk.1
    chunkbytesThe binary chunk contents, as returned by LoadSnapshotChunk.2
    senderstringThe P2P ID of the node who sent this chunk.3
  • Response:

    NameTypeDescriptionField Number
    resultResult (see below)The result of applying this chunk.1
    refetch_chunksrepeated bytesIdentifiers of chunks to discard, refetch and reapply, regardless of result.2
    reject_sendersrepeated stringReject the given P2P senders, regardless of Result. Any chunks already applied will not be refetched unless explicitly requested, but queued chunks from these senders will be discarded, and new chunks or other snapshots rejected.3
    next_chunksrepeated bytesChunk identifiers to enqueue after ACCEPT; ignored for other results.4
  enum Result {
    UNKNOWN         = 0;  // Unknown result, abort all snapshot restoration
    ACCEPT          = 1;  // The chunk was accepted.
    ABORT           = 2;  // Abort snapshot restoration, and don't try any other snapshots.
    RETRY           = 3;  // Reapply this chunk, combine with `RefetchChunks` and `RejectSenders` as appropriate.
    RETRY_SNAPSHOT  = 4;  // Restart this snapshot from `OfferSnapshot`, reusing chunks unless instructed otherwise.
    REJECT_SNAPSHOT = 5;  // Reject this snapshot, try a different one.
    COMPLETE_SNAPSHOT = 6;  // Accept this chunk and finish restoring the snapshot.
  }
  • Usage:
  • The application can choose to refetch chunks and/or ban P2P peers as appropriate. Tenderdash will not do this unless instructed by the application.
  • The application may want to verify each chunk, e.g. by attaching chunk hashes in Snapshot.Metadata and/or incrementally verifying contents against AppHash.
  • On ACCEPT, Tenderdash enqueues next_chunks and continues applying chunks. Fetching can run in parallel; identifiers do not impose a numeric application order. The application must discover and validate the required chunks.
  • Return COMPLETE_SNAPSHOT only when restoration is complete. An ACCEPT with no next_chunks, or an empty request queue, does not signal completion.
  • After COMPLETE_SNAPSHOT, Tenderdash calls Info and checks last_block_app_hash against the light-client-verified hash, last_block_height against the snapshot height, and app_version against the version in the reconstructed consensus state. Failed verification prevents successful state sync. The node can then catch up through block sync or enter consensus.
  • If Tenderdash is unable to retrieve the next chunk after some time (e.g. because no suitable peers are available), it will reject the snapshot and try a different one via OfferSnapshot. The application should be prepared to reset and accept it or abort as appropriate.

Consensus methods

PrepareProposal

Parameters and Types

Method NameRequest TypeResponse TypeDescription
PrepareProposalRequestPrepareProposalResponsePrepareProposal

Usage

  • Use the linked request/response schemas for field names and numbers; proposal preparation, processing and finalization have different request shapes.
  • The height and time values match the values from the header of the proposed block.
  • RequestPrepareProposal contains a preliminary set of transactions txs that Tenderdash considers to be a good block proposal, called raw proposal. The Application can modify this set via ResponsePrepareProposal.tx_records (see TxRecord).
    • The Application can reorder, remove or add transactions to the raw proposal. Let tx be a transaction in txs:
      • If the Application considers that tx should not be proposed in this block, e.g., there are other transactions with higher priority, then it should not include it in tx_records. In this case, Tenderdash won't remove tx from the mempool. The Application should be extra-careful, as abusing this feature may cause transactions to stay forever in the mempool.
      • If the Application considers that a tx should not be included in the proposal and removed from the mempool, then the Application should include it in tx_records and mark it as REMOVED. In this case, Tenderdash will remove tx from the mempool.
      • If the Application wants to add a new transaction, then the Application should include it in tx_records and mark it as ADDED. Tenderdash includes it in the proposal without adding it to the mempool.
    • The Application should be aware that removing and adding transactions may compromise traceability.

      Consider the following example: the Application transforms a client-submitted transaction t1 into a second transaction t2, i.e., the Application asks Tenderdash to remove t1 from the mempool and include t2 in the proposal. If a client wants to eventually check what happened to t1, it will discover that t_1 is not in the mempool or in a committed block, getting the wrong idea that t_1 did not make it into a block. Note that t_2 will be in a committed block, but unless the Application tracks this information, no component will be aware of it. Thus, if the Application wants traceability, it is its responsability to support it. For instance, the Application could attach to a transformed transaction a list with the hashes of the transactions it derives from.

  • Tenderdash MAY include a list of transactions in RequestPrepareProposal.txs whose total size in bytes exceeds RequestPrepareProposal.max_tx_bytes. Therefore, if the size of RequestPrepareProposal.txs is greater than RequestPrepareProposal.max_tx_bytes, the Application MUST make sure that the RequestPrepareProposal.max_tx_bytes limit is respected by those transaction records returned in ResponsePrepareProposal.tx_records that are marked as UNMODIFIED or ADDED.
  • Tenderdash uses same-block execution. The Application must provide values for ResponsePrepareProposal.app_hash, ResponsePrepareProposal.tx_results, ResponsePrepareProposal.validator_set_update, ResponsePrepareProposal.core_chain_lock_update and ResponsePrepareProposal.consensus_param_updates, as a result of fully executing the block.
    • The values for ResponsePrepareProposal.validator_set_update, ResponsePrepareProposal.core_chain_lock_update or ResponsePrepareProposal.consensus_param_updates may be empty. In this case, Tenderdash will keep the current values.
    • Validator changes from block H are committed by NextValidatorsHash and become active at H+1. See Validator-set and parameter changes.
    • Consensus parameter changes from PrepareProposal/ProcessProposal for block H take effect at H+1; ResponseFinalizeBlock has no parameter-update field. See Validator-set and parameter changes.
    • It is the responsibility of the Application to set the right value for TimeoutPropose so that the (synchronous) execution of the block does not cause other processes to prevote nil because their propose timeout goes off.
  • Block and transaction events are returned with ProcessProposal, not ResponseFinalizeBlock; see Events.
  • As a sanity check, Tenderdash will check the returned parameters for validity if the Application modified them. In particular, ResponsePrepareProposal.tx_records will be deemed invalid if
    • There is a duplicate transaction in the list.
    • A new or modified transaction is marked as UNMODIFIED or REMOVED.
    • An unmodified transaction is marked as ADDED.
    • A transaction is marked as UNKNOWN.
  • ResponsePrepareProposal.tx_results contains only results of UNMODIFIED and ADDED transactions. REMOVED transactions are omitted. The length of tx_results can be different than the length of tx_records.
  • If Tenderdash fails to validate the ResponsePrepareProposal, Tenderdash will assume the application is faulty and crash.
  • The implementation of PrepareProposal can be non-deterministic.

When does Tenderdash call it?

Tenderdash calls PrepareProposal when creating a fresh proposal, after selecting transactions within the block's byte/gas budget. A valid block is reused only when checkValidBlock permits it; otherwise a fresh block is prepared even if a ValidBlock is recorded. The exact transaction-space calculation, response validation and reuse rules are in Creating the proposal.

ProcessProposal

Parameters and Types

Method NameRequest TypeResponse TypeDescription
ProcessProposalRequestProcessProposalResponseProcessProposal

Usage

  • Contains fields from the proposed block.
    • The Application may fully execute the block as though it was handling RequestFinalizeBlock. However, any resulting state changes must be kept as candidate state, and the Application should be ready to backtrack/discard it in case the decided block is different.
  • The height and timestamp values match the values from the header of the proposed block.
  • If ResponseProcessProposal.status is REJECT, Tenderdash assumes the proposal received is not valid.
  • Tenderdash requires the current block's execution commitments. The Application must provide values for parameters ResponseProcessProposal.app_hash, ResponseProcessProposal.tx_results, ResponseProcessProposal.validator_set_update, and ResponseProcessProposal.consensus_param_updates, so that Tenderdash can then verify the hashes in the block's header are correct. If the hashes mismatch, Tenderdash will reject the block even if ResponseProcessProposal.status was set to ACCEPT.
  • The implementation of ProcessProposal MUST be deterministic. Moreover, the value of ResponseProcessProposal.status MUST exclusively depend on the parameters passed in the call to RequestProcessProposal, and the last committed Application state (see Requirements section).
  • Tenderdash may call ProcessProposal more than once for the same height and round with the same block, for example after the application rejected a commit's vote extensions or when block sync hands over to consensus, and again for a round after processing another round, for example to verify a commit of that round. The Application MUST treat such a call as re-execution that replaces its execution context for the height; FinalizeBlock refers to the round processed last.
  • Moreover, application implementors SHOULD always set ResponseProcessProposal.status to ACCEPT, unless they really know what the potential liveness implications of returning REJECT are.

When does Tenderdash call it?

ensureProcess processes the held block for the round being checked unless the same block/round result is already current. It runs before the prevote lock rules, so a locked validator can process another block and still prevote nil. A block that does not validate against Tenderdash's committed state (for example a wrong height, chain ID or last block ID) is never sent to the application and produces a nil prevote, as do a well-formed application rejection and a block/header mismatch with the application's answer; unknown status, malformed results and transport errors are fatal on this path. See the exact prevote decision.

Processing also occurs for commit application and can repeat after another round changed the execution context; see Commit handling and expected call sequences.

ExtendVote

Parameters and Types

  • Request:

    NameTypeDescriptionField Number
    hashbytesThe header hash of the proposed block that the vote extensions is to refer to.1
    heightint64Height of the proposed block (for sanity check).2
    roundint32Round for the block and its application execution context.3
  • Response:

    NameTypeDescriptionField Number
    vote_extensionsExtendVoteExtensionOptional information signed by Tenderdash.1
  • Usage:

  • ResponseExtendVote.vote_extensions is optional information that, if present, will be signed by Tenderdash and attached to the Precommit message.

  • RequestExtendVote.hash corresponds to the hash of a proposed block that was made available to the application in a previous call to ProcessProposal or PrepareProposal for the current height.

  • ResponseExtendVote.vote_extensions will only be attached to a non-nil Precommit message. If Tenderdash is to precommit nil, it will not call RequestExtendVote.

  • The Application logic that creates extensions that are not threshold-recoverable can be non-deterministic. Threshold-recoverable extensions (THRESHOLD_RECOVER and THRESHOLD_RECOVER_RAW) are recovered into the commit, so they MUST be deterministic.

  • ExtendVote MUST be free of side effects, and the threshold-recoverable subset of its response MUST be exactly the vector the Application accepts in a commit-level VerifyVoteExtension for the same block, height and round.

  • Tenderdash may also call ExtendVote on any node, a non-validator included, after a commit-level VerifyVoteExtension rejection that follows ProcessProposal of that block and round, to learn the vector a commit of that block and round must carry; see VerifyVoteExtension. An ABCI error or an invalid response from this call is fatal, as on the Precommit path.

When does Tenderdash call it?

The normal call is inside signing a non-nil precommit, after the block was processed for that round. The proposal, lock and polka requirements are in the precommit decision. Nil precommits do not call ExtendVote for signing.

After rejecting a commit's extensions, Tenderdash can also call ExtendVote on any node to learn the expected recoverable vector for the processed block and commit round. That response is neither signed nor broadcast; see Parked commits and replacement candidates.

VerifyVoteExtension

Tenderdash calls VerifyVoteExtension for two kinds of request, told apart by validator_pro_tx_hash:

  • Precommit verification (validator_pro_tx_hash set): the vote extensions of one validator's Precommit, exactly as its Application returned them from ExtendVote.
  • Commit verification (validator_pro_tx_hash empty): the extension vector of a commit, sent after ProcessProposal of the committed block with the same hash, height and round (the commit round). The vector holds only the threshold-recoverable extensions (THRESHOLD_RECOVER and THRESHOLD_RECOVER_RAW), in the order ExtendVote returned them; extensions of any other type never appear in a commit. It can be empty.

Parameters and Types

  • Request:

    NameTypeDescriptionField Number
    hashbytesThe header hash of the propsed block that the vote extensions refers to.1
    validator_pro_tx_hashbytesProTxHash of the validator that signed the extensions; empty for commit verification2
    heightint64Height of the block (for sanity check).3
    roundint32Round of the vote; for commit verification, the commit round.4
    vote_extensionsrepeated ExtendVoteExtensionApplication-specific information signed by Tenderdash. Can have 0 length5
  • Response:

    NameTypeDescriptionField Number
    statusVerifyStatusenum signaling if the application accepts the vote extension1
  • Usage:

  • RequestVerifyVoteExtension.vote_extensions can be empty. For a Precommit, the Application's interpretation of it should be that the Application running at the process that sent the vote chose not to extend it. Tenderdash will always call RequestVerifyVoteExtension, even for 0 length vote extensions.

  • For a Precommit, if ResponseVerifyVoteExtension.status is REJECT, Tenderdash will reject the whole received vote. See the Requirements section to understand the potential liveness implications of this.

  • For a commit, the Application MUST compare the vector with the threshold-recoverable subset of the extensions it expects for that block, in order, and return REJECT if they differ: a commit whose block signature is valid can still carry a stripped, duplicated or replayed vector. Tenderdash requires ACCEPT before it saves the block and commit or calls FinalizeBlock. What REJECT does depends on the path:

    • consensus: the commit is discarded without blaming its sender. A commit whose block the node already holds is checked first and never changes the round. A commit for a later round whose block has not arrived moves the node to that round on its threshold signature alone; it is checked when the block arrives, then the commits other peers sent meanwhile. Copies of a commit already waiting are dropped. If the node's own commit is rejected, nothing is persisted and the node waits for a peer's commit. The block is processed again only for a commit of a round other than the one processed last. After a rejection, Tenderdash calls ExtendVote for the block and round, and refuses, without processing the block again or calling the Application, a later commit of that round that would need the block processed again, when its vector differs from the one returned or equals it and was rejected. Consensus WAL replay follows these rules;
    • block sync: the block is not applied; the peer that served it is dropped and the height is requested again;
    • handshake catch-up (application behind the block store): the commit is already in the block store, so the node fails to start. There is no automatic recovery; the operator has to roll back or re-sync the node.
  • Any status other than ACCEPT, including UNKNOWN, counts as REJECT. An error from the ABCI call itself, as opposed to a status, is fatal on every path.

  • The implementation of VerifyVoteExtension MUST be deterministic. Moreover, the value of ResponseVerifyVoteExtension.status MUST exclusively depend on the parameters passed in the call to RequestVerifyVoteExtension, the block processed by ProcessProposal, and the last committed Application state (see Requirements section).

  • For a Precommit, application implementers SHOULD always set ResponseVerifyVoteExtension.status to ACCEPT, unless they really know what the potential liveness implications of returning REJECT are.

When does Tenderdash call it?

For a peer's non-nil precommit, Tenderdash verifies its block and extension signatures, restores its own processed round if needed, then calls the Application with the validator's pro-tx-hash. A rejected vote is not stored and may be redelivered. Own votes skip this Application check; see Vote admission.

For a commit, Tenderdash processes the block for commit.Round and verifies the recovered extension vector with an empty validator pro-tx-hash. Live consensus requires acceptance before saving/finalizing the block. Replacement candidates and stored-block replay have different retry behavior, as described in the usage contract above and Commit handling.

FinalizeBlock

Parameters and Types

Method NameRequest TypeResponse TypeDescription
FinalizeBlockRequestFinalizeBlockResponseFinalizeBlock

Usage

  • Contains the fields of the newly decided block.
  • The height and timestamp values match the values from the header of the proposed block.
  • RequestFinalizeBlock.commit and RequestFinalizeBlock.misbehavior carry commit and misbehavior information; see the linked request schema.
  • The Application persists the state corresponding to RequestFinalizeBlock.block and the round processed last, whose execution commitments were already supplied through PrepareProposal/ProcessProposal. It may reuse that candidate rather than execute the block again.
  • Transaction results and events are returned during ProcessProposal; ResponseFinalizeBlock returns only retain_height and propose_next_block_immediately, with no application hash or events.
  • Application is expected to persist its state at the end of this call, before calling ResponseFinalizeBlock.
  • Later calls to Query can return proofs about the application state anchored in this Merkle root hash.
  • Use ResponseFinalizeBlock.retain_height with caution! If all nodes in the network remove historical blocks then this data is permanently lost, and no new nodes will be able to join the network and bootstrap. Historical blocks may also be required for other purposes, e.g. auditing, replay of non-persisted heights, light client verification, and so on.
  • Set ResponseFinalizeBlock.propose_next_block_immediately when the Application has block-driven work pending that the next block would carry out, for example withdrawal transactions waiting to be signed. The node then does not wait for transactions (create-empty-blocks-interval) before proposing at the next height, in round 0 or, on consensus version 2, in a round entered after a silent proposer's round was skipped. This is a hint to the local node, not part of consensus: it is not compared between validators, it only matters on the node that proposes the next height, and it is consumed once; a node that restarts in between falls back to the configured wait.
  • Just as ProcessProposal, the implementation of FinalizeBlock MUST be deterministic, since it is making the Application's state evolve in the context of state machine replication.
  • Currently, Tenderdash will fill up all fields in RequestFinalizeBlock, even if they were already passed on to the Application via RequestPrepareProposal or RequestProcessProposal. The Application must select the candidate matching the finalized block and the round processed last; a block hash alone does not identify a round's execution context. See Commit validity depends on the application.

When does Tenderdash call it?

Tenderdash finalizes a held block after validating its complete block ID, processing it for the commit round, and accepting the recovered commit's extensions. The commit can come from local precommits or a peer; it does not require receipt of every original vote or proposal. See Commit handling and the exact persistence order in Commit and apply.

Shared data types

Most of the data structures used in ABCI are shared common data structures. In certain cases, ABCI uses different data structures which are documented here:

Validator

See the generated Validator schema.

  • Usage:
  • Validator identified by ProTxHash
  • Used by Misbehavior.validator to identify an offending validator.
  • Public keys are carried separately by validator-set update messages.

ValidatorUpdate

See the generated ValidatorUpdate schema.

  • Usage:
  • Validator identified by ProTxHash
  • Used to tell Tenderdash to update the validator set

ValidatorSetUpdate

See the generated ValidatorSetUpdate schema.

  • Usage:
  • Used to tell Tenderdash to update the validator set update

Misbehavior

See the generated Misbehavior schema.

ConsensusParams

  • Fields:

    NameTypeDescriptionField Number
    blockBlockParamsParameters limiting the size of a block and time between consecutive blocks.1
    evidenceEvidenceParamsParameters limiting the validity of evidence of byzantine behaviour.2
    validatorValidatorParamsParameters limiting the types of public keys validators can use.3
    versionVersionsParamsThe ABCI application version.4

ProofOps

  • Fields:

    NameTypeDescriptionField Number
    opsrepeated ProofOpList of chained Merkle proofs, of possibly different types. The Merkle root of one op is the value being proven in the next op. The Merkle root of the final op should equal the ultimate root hash being verified against..1

ProofOp

  • Fields:

    NameTypeDescriptionField Number
    typestringType of Merkle proof and how it's encoded.1
    keybytesKey in the Merkle tree that this proof is for.2
    databytesEncoded Merkle proof for the key.3

Snapshot

See the generated Snapshot schema.

  • Usage:
  • Used for state sync snapshots, see the state sync section for details.
  • A snapshot is considered identical across nodes only if all fields are equal (including Metadata). Chunks may be retrieved from all nodes that have the same snapshot.
  • When sent across the network, a snapshot message can be at most 4 MB.

Execution and consensus data types

VoteInfo

See the generated VoteInfo schema.

  • Usage:
  • Indicates whether a validator signed the last block, allowing for rewards based on validator availability.
  • This information is typically extracted from a proposed or decided block.

ExtendVoteExtension

See the generated ExtendVoteExtension schema.

  • Usage:
  • Provides a vote extension for signing
  • Each field is mandatory for filling
  • Type can be THRESHOLD_RECOVER, THRESHOLD_RECOVER_RAW or DEFAULT
  • DEFAULT is not supported so far
  • THRESHOLD_RECOVER and THRESHOLD_RECOVER_RAW are deterministic, that means, each validator in a quorum must provide the same vote extension data

VoteExtensionType

enum VoteExtensionType {
  DEFAULT               = 0;
  THRESHOLD_RECOVER     = 1;
  THRESHOLD_RECOVER_RAW = 2;  // signs the raw extension; allows overriding the sign request ID
}

CommitInfo

See the generated CommitInfo schema.

ExecTxResult

See the generated ExecTxResult schema.

TxAction

enum TxAction {
  UNKNOWN    = 0;  // Unknown action
  UNMODIFIED = 1;  // The Application did not modify this transaction.
  ADDED      = 2;  // The Application added this transaction.
  REMOVED    = 3;  // The Application wants this transaction removed from the proposal and the mempool.
  DELAYED    = 4;  // Exclude from this proposal, but keep in the mempool.
}
  • Usage:

  • If Action is UNKNOWN, a problem happened in the Application. Tenderdash will assume the application is faulty and crash.

  • If Action is UNMODIFIED, Tenderdash includes the transaction in the proposal. Nothing to do on the mempool.

  • If Action is ADDED, Tenderdash includes the transaction in the proposal. The transaction is not added to the mempool.

  • If Action is REMOVED, Tenderdash excludes the transaction from the proposal. The transaction is also removed from the mempool if it exists, similar to CheckTx returning false.

  • If Action is DELAYED, Tenderdash excludes the transaction from the proposal and keeps it in the mempool.

TxRecord

See the generated TxRecord schema.

ProposalStatus

enum ProposalStatus {
  UNKNOWN = 0; // Unknown status. Returning this from the application is always an error.
  ACCEPT  = 1; // Status that signals that the application finds the proposal valid.
  REJECT  = 2; // Status that signals that the application finds the proposal invalid.
}
  • Usage:
  • Used within the ProcessProposal response.
    • If Status is UNKNOWN, a problem happened in the Application. Tenderdash will assume the application is faulty and crash.
    • If Status is ACCEPT, the application finds the proposal valid. Tenderdash still validates the returned execution results, which can reject the block. On the prevote path, the lock and POL rules then decide between a prevote for the block and a nil prevote.
    • If Status is REJECT, the application finds the proposal invalid. On the prevote path, Tenderdash prevotes nil; during commit processing or replay, the call fails and the block is not applied. See ProcessProposal for each call path.

VerifyStatus

enum VerifyStatus {
  UNKNOWN = 0; // Unknown status. Returning this from the application is always an error.
  ACCEPT  = 1; // Status that signals that the application finds the vote extension valid.
  REJECT  = 2; // Status that signals that the application finds the vote extension invalid.
}
  • Usage:
  • Used within the VerifyVoteExtension response, for a Precommit or a commit.
    • If Status is ACCEPT, Tenderdash will accept the vote, or the commit, as valid.
    • If Status is REJECT or UNKNOWN, Tenderdash will reject the vote, or the commit, as invalid. Only an error from the ABCI call itself is fatal.

CanonicalVoteExtension

This signing structure belongs to the consensus vote-extension protocol, rather than the ABCI application interface. See vote-extension signing.

  • Fields:

    NameTypeDescriptionField Number
    extensionbytesVote extension provided by the Application.1
    heightsfixed64Height in which the extension was provided.2
    roundsfixed64Round in which the extension was provided.3
    chain_idstringID of the blockchain running consensus.4
    typeVoteExtensionTypeVoteExtensionType5
  • Usage:

  • Tenderdash is to sign the whole data structure and attach it to a Precommit message

  • Upon reception, Tenderdash validates the sender's signature and sanity-checks the values of height, round, and chain_id. Then it sends extension to the Application via RequestVerifyVoteExtension for verification.

Tenderdash's expected behavior

Valid method call sequences

Tenderdash initiates ABCI requests and the Application responds. The consensus protocol specifies the implemented order, including guards, quorum/recovery thresholds and failure paths. Its system model states the safety assumptions; a configurable threshold must not be assumed to satisfy the strict BFT floor in every supported configuration.

The primary flow describes a height that succeeds in round zero. Its ordinary application calls are:

Role or purposeCalls
Proposer creating a fresh blockPrepareProposal supplies transactions, same-block application/results hashes and next-state changes. A reused valid block does not require another PrepareProposal.
Processing before voting or commit applicationProcessProposal for the block and the round being checked, unless that exact result is already current.
Validator signing a non-nil precommitExtendVote after processing for that round. Nil precommits and prevotes carry no extensions.
Peer non-nil precommitVerifyVoteExtension with the validator's pro-tx-hash, after signature verification.
Commit, on any full nodeVerifyVoteExtension with an empty validator pro-tx-hash, after processing for the commit round.
Accepted decided blockFinalizeBlock persists the candidate corresponding to that block and round.

These are not exact per-height callback counts. A node can finish on a commit without receiving every individual precommit. ProcessProposal can run again for the same block/round after a different round has replaced the Application's execution context. Commit checks can process a foreign round and then restore the node's own round before checking an individual precommit. See Commit handling and ABCI calls per height.

After commit-level VerifyVoteExtension rejects a vector, Tenderdash can call ExtendVote for that processed block and commit round to learn the expected vector. This call can occur on an observer and does not sign or broadcast a vote. A replacement commit can require further verification and processing. Rejection of a held-block commit does not itself advance the round; a future-round commit whose block is absent can already advance the round before application validation. The full branch rules are in Commits received from peers and the VerifyVoteExtension contract.

An Application must handle these repeated calls deterministically for the block, round, and last committed state. It must not treat an abbreviated normal-round sequence as an at-most-once guarantee. A successfully completed live height calls FinalizeBlock only after accepting the commit vector. Rejection while replaying an already-stored commit during startup instead terminates replay before FinalizeBlock; see Recovery and synchronisation.

Startup and methods outside block execution

Startup uses Info to reconcile application and Tenderdash state and InitChain when the Application is at height zero; the exact cases, replay calls and crash windows are in ABCI handshake. Snapshot restoration can attempt OfferSnapshot/ApplySnapshotChunk before a successful restore and Info; these are synchronization boundaries rather than ordinary voting rounds.

CheckTx controls mempool admission separately from the block execution sequence. Query, ListSnapshots and LoadSnapshotChunk read application state or existing snapshots. Echo and Flush serve client/debugging purposes; from consensus version 2 a waiting proposer also uses Echo as a readiness probe (Echo). Info can also serve RPC queries independently of startup. These methods are not bounded by the normal height's consensus callback count.

Application execution requirements

Tenderdash requires same-block execution: ResponsePrepareProposal and ResponseProcessProposal supply the current block's application hash and transaction results before it is decided. Returning only a boolean acceptance or waiting until FinalizeBlock to compute those commitments is insufficient. Use the current method schemas and requirements.

Keep speculative execution as candidate state; publish and persist it only when FinalizeBlock selects that block and the round processed last. There is no ABCI Commit method; persistence belongs in FinalizeBlock. ResponseFinalizeBlock has only retain_height and propose_next_block_immediately; it returns neither AppHash, transaction results, nor events. Block/transaction events belong to ProcessProposal results; see Events.

An Application returning no vote extensions must accept an empty expected commit vector and reject a nonempty one at commit-level verification. Other Applications must compare the ordered threshold-recoverable subset they expect for that block and round; independently valid signatures do not establish the vector's identity, count and order. See VerifyVoteExtension.

Protocol Documentation

Table of Contents

Top

tendermint/abci/types.proto

CommitInfo

FieldTypeLabelDescription
roundint32
quorum_hashbytes
block_signaturebytes
threshold_vote_extensionstendermint.types.VoteExtensionrepeated

Event

Event carries application attributes in ResponseProcessProposal and ExecTxResult.

Later, transactions may be queried using these events.

FieldTypeLabelDescription
typestring
attributesEventAttributerepeated

EventAttribute

EventAttribute is a single key-value pair, associated with an event.

FieldTypeLabelDescription
keystring
valuestring
indexboolnondeterministic

ExecTxResult

ExecTxResult contains results of executing one individual transaction.

FieldTypeLabelDescription
codeuint32Response code within codespace; by convention, 0 means success.
databytesResult bytes, if any (arbitrary data, not interpreted by Tenderdash).
logstringThe output of the application's logger. May be non-deterministic.
infostringAdditional information. May be non-deterministic.
gas_usedint64Amount of gas consumed by transaction.
eventsEventrepeatedType & Key-Value events for indexing transactions (e.g. by account).
codespacestringNamespace for the code.

ExtendVoteExtension

Provides a vote extension for signing. type and extension fields are mandatory for filling

FieldTypeLabelDescription
typetendermint.types.VoteExtensionTypeVote extension type can be either DEFAULT, THRESHOLD_RECOVER or THRESHOLD_RECOVER_RAW. The Tenderdash supports only THRESHOLD_RECOVER and THRESHOLD_RECOVER_RAW at this moment.
extensionbytesDeterministic or (Non-Deterministic) extension provided by the sending validator's Application.

For THRESHOLD_RECOVER_RAW, it MUST be 32 bytes.

Sign request ID that will be used to sign the vote extensions. Only applicable for THRESHOLD_RECOVER_RAW vote extension type.

Tenderdash will use SHA256 checksum of sign_request_id when generating quorum signatures of THRESHOLD_RECOVER_RAW vote extensions. It MUST NOT be set for any other vote extension types. | | sign_request_id | bytes | optional | If not set, Tenderdash will generate it based on height and round.

If set, it SHOULD be unique per voting round, and it MUST start with dpevote or \x06plwdtx prefix.

Use with caution - it can have severe security consequences. |

ExtendedVoteInfo

ExtendedVoteInfo

FieldTypeLabelDescription
validatorValidatorThe validator that sent the vote.
signed_last_blockboolIndicates whether the validator signed the last block, allowing for rewards based on validator availability.
vote_extensionbytesNon-deterministic extension provided by the sending validator's application.

Misbehavior

FieldTypeLabelDescription
typeMisbehaviorType
validatorValidatorThe offending validator
heightint64The height when the offense occurred
timegoogle.protobuf.TimestampThe corresponding time where the offense occurred
total_voting_powerint64Total voting power of the validator set in case the ABCI application does not store historical validators. https://github.com/tendermint/tendermint/issues/4581

QuorumHashUpdate

FieldTypeLabelDescription
quorum_hashbytes

Request

Request types

FieldTypeLabelDescription
echoRequestEcho
flushRequestFlush
infoRequestInfo
init_chainRequestInitChain
queryRequestQuery
check_txRequestCheckTx
list_snapshotsRequestListSnapshots
offer_snapshotRequestOfferSnapshot
load_snapshot_chunkRequestLoadSnapshotChunk
apply_snapshot_chunkRequestApplySnapshotChunk
prepare_proposalRequestPrepareProposal
process_proposalRequestProcessProposal
extend_voteRequestExtendVote
verify_vote_extensionRequestVerifyVoteExtension
finalize_blockRequestFinalizeBlock

RequestApplySnapshotChunk

Applies a snapshot chunk.

  • The application can choose to refetch chunks and/or ban P2P peers as appropriate. Tenderdash will not do this unless instructed by the application.
  • The application may want to verify each chunk, e.g. by attaching chunk hashes in Snapshot.Metadata and/or incrementally verifying contents against AppHash.
  • On ACCEPT, Tenderdash enqueues ResponseApplySnapshotChunk.next_chunks and continues applying chunks. Restoration finishes only when the application returns COMPLETE_SNAPSHOT; an ACCEPT without next_chunks does not signal completion.
  • After COMPLETE_SNAPSHOT, Tenderdash makes an ABCI Info call to verify that LastBlockAppHash, LastBlockHeight and AppVersion match the expected values. It then switches to block sync or consensus and joins the network.
  • If Tenderdash is unable to retrieve the next chunk after some time (e.g. because no suitable peers are available), it will reject the snapshot and try a different one via OfferSnapshot. The application should be prepared to reset and accept it or abort as appropriate.
FieldTypeLabelDescription
chunk_idbytesApplication-defined chunk identifier: Snapshot.hash for the first chunk, then identifiers returned in ResponseApplySnapshotChunk.next_chunks.
chunkbytesThe binary chunk contents, as returned by LoadSnapshotChunk.
senderstringThe P2P ID of the node who sent this chunk.

RequestCheckTx

Check if transaction is valid.

  • Technically optional - not involved in processing blocks.
  • Guardian of the mempool: every node runs CheckTx before letting a transaction into its local mempool.
  • The transaction may come from an external user or another node
  • CheckTx validates the transaction against the current state of the application, for example, checking signatures and account balances, but does not apply any of the state changes described in the transaction.
  • Transactions where ResponseCheckTx.Code != 0 will be rejected - they will not be broadcast to other nodes or included in a proposal block.
  • Tenderdash attributes no other value to the response code
FieldTypeLabelDescription
txbytesThe request transaction bytes.
typeCheckTxTypeType or transaction check to execute.

RequestEcho

Echo a string to test an abci client/server implementation

FieldTypeLabelDescription
messagestringA string to echo back

RequestExtendVote

Return the vote_extensions vector for the block and round processed last. Before a validator signs a non-nil precommit, Tenderdash requests the extensions to attach to that vote. It may also call ExtendVote on any node after rejecting a commit vector, to learn the expected vector without signing or broadcasting. The call must be free of side effects. Threshold-recoverable extensions must be deterministic and match the vector accepted at commit-level VerifyVoteExtension. See the ExtendVote contract.

FieldTypeLabelDescription
hashbytesThe header hash of the proposed block that the vote extensions is to refer to.
heightint64Height of the proposed block (for sanity check).
roundint32Round number for the block.

RequestFinalizeBlock

Persist the decided candidate for the block and the round processed last. Tenderdash calls this after processing and accepting the commit extension vector. The application must persist the candidate before returning; execution results and events are provided through proposal processing, not ResponseFinalizeBlock. The response contains retention and next-block scheduling hints only. Use retain_height with care: pruning blocks can prevent replay and bootstrapping. See the FinalizeBlock contract and commit and apply.

FieldTypeLabelDescription
commitCommitInfoInfo about the current commit
misbehaviorMisbehaviorrepeatedList of information about validators that acted incorrectly.
hashbytesThe block header's hash. Present for convenience (can be derived from the block header).
heightint64The height of the finalized block.
roundint32Round number for the block
blocktendermint.types.BlockThe block that was finalized
block_idtendermint.types.BlockIDThe block ID that was finalized

RequestFlush

Signals that messages queued on the client should be flushed to the server. It is called periodically by the client implementation to ensure asynchronous requests are actually sent, and is called immediately to make a synchronous request, which returns when the Flush response comes back.

RequestInfo

Return information about the application's last committed state. Used during startup reconciliation and by RPC queries. Persist the reported height and application hash in FinalizeBlock. See the startup handshake.

FieldTypeLabelDescription
versionstringThe Tenderdash software semantic version.
block_versionuint64The Tenderdash Block Protocol version.
p2p_versionuint64The Tenderdash P2P Protocol version.
abci_versionstringThe Tenderdash ABCI semantic version.

RequestInitChain

Called once upon genesis.

  • If ResponseInitChain.Validators is empty, the initial validator set will be the RequestInitChain.Validators
  • If ResponseInitChain.Validators is not empty, it will be the initial validator set (regardless of what is in RequestInitChain.Validators).
  • This allows the app to decide if it wants to accept the initial validator set proposed by Tenderdash (ie. in the genesis file), or if it wants to use a different one (perhaps computed based on some application specific information in the genesis file).
FieldTypeLabelDescription
timegoogle.protobuf.TimestampGenesis time
chain_idstringID of the blockchain.
consensus_paramstendermint.types.ConsensusParamsInitial consensus-critical parameters.
validator_setValidatorSetUpdateInitial genesis validators, sorted by voting power.
app_state_bytesbytesSerialized initial application state. JSON bytes.
initial_heightint64Height of the initial block (typically 1).
initial_core_heightuint32Initial core chain lock height.

RequestListSnapshots

Lists available snapshots

  • Used during state sync to discover available snapshots on peers.
  • See Snapshot data type for details.

RequestLoadSnapshotChunk

Used during state sync to retrieve snapshot chunks from peers.

FieldTypeLabelDescription
heightuint64The height of the snapshot the chunks belongs to.
versionuint32The application-specific format of the snapshot the chunk belongs to.
chunk_idbytesApplication-defined chunk identifier: Snapshot.hash for the first chunk, then identifiers returned in ResponseApplySnapshotChunk.next_chunks.

RequestOfferSnapshot

Offers a snapshot to the application.

  • OfferSnapshot is called when bootstrapping a node using state sync. The application may accept or reject snapshots as appropriate. Upon accepting, Tenderdash will retrieve and apply snapshot chunks via ApplySnapshotChunk. The application may also choose to reject a snapshot in the chunk response, in which case it should be prepared to accept further OfferSnapshot calls.
  • Only AppHash can be trusted, as it has been verified by the light client. Any other data can be spoofed by adversaries, so applications should employ additional verification schemes to avoid denial-of-service attacks. The verified AppHash is automatically checked against the restored application at the end of snapshot restoration.
  • For more information, see the Snapshot data type or the state sync section.
FieldTypeLabelDescription
snapshotSnapshotThe snapshot offered for restoration.
app_hashbytesThe light client-verified app hash for this height, from the blockchain. 32 bytes.

RequestPrepareProposal

Prepare and execute a new block proposal, potentially altering its transactions. Return same-block application/results hashes and next-state commitments. ADDED transactions enter the proposal without being added to the mempool; REMOVED transactions are removed from both. Changes to validators and consensus parameters from height H take effect at H+1. Preparation may be non-deterministic. See the PrepareProposal contract for transaction constraints, response validation and the exact conditions for preparing a fresh block.

FieldTypeLabelDescription
max_tx_bytesint64Currently configured maximum size in bytes taken by the modified transactions. The modified transactions cannot exceed this size.
txsbytesrepeatedPreliminary list of transactions that have been picked as part of the block to propose. Sent to the app for possible modifications.
local_last_commitCommitInfoInfo about the last commit, obtained locally from Tenderdash's data structures.
misbehaviorMisbehaviorrepeatedList of information about validators that acted incorrectly.
heightint64The height of the block that will be proposed.
timegoogle.protobuf.TimestampTimestamp of the block that that will be proposed.
next_validators_hashbytesMerkle root of the next validator set.
roundint32Round number for the block.
core_chain_locked_heightuint32Core chain lock height to be used when signing this block.
proposer_pro_tx_hashbytesProTxHash of the original proposer of the block.
proposed_app_versionuint64Proposer's latest available app protocol version.
versiontendermint.version.ConsensusApp and block version used to generate the block. App version included in the block can be modified by setting ResponsePrepareProposal.app_version.
quorum_hashbytesquorum_hash contains hash of validator quorum that will sign the block

RequestProcessProposal

Execute a proposed block deterministically against the last committed state. Return its status, same-block application/results hashes and next-state commitments. Keep execution speculative until FinalizeBlock. Processing can repeat for the same block and round after another execution context has replaced it. See the ProcessProposal contract and application requirements.

FieldTypeLabelDescription
txsbytesrepeatedList of transactions that have been picked as part of the proposed
proposed_last_commitCommitInfoInfo about the last commit, obtained from the information in the proposed block.
misbehaviorMisbehaviorrepeatedList of information about validators that acted incorrectly.
hashbytesThe block header's hash of the proposed block. It is computed as a Merkle tree from the header fields.
heightint64The height of the proposed block.
roundint32Round number for the block
timegoogle.protobuf.TimestampTimestamp included in the proposed block.
next_validators_hashbytesMerkle root of the next validator set.
core_chain_locked_heightuint32Core chain lock height to be used when signing this block.
core_chain_lock_updatetendermint.types.CoreChainLockNext core-chain-lock-update for validation in ABCI.
proposer_pro_tx_hashbytesProTxHash of the original proposer of the block.
proposed_app_versionuint64Proposer's latest available app protocol version.
versiontendermint.version.ConsensusApp and block version used to generate the block. App version MUST be verified by the app.
quorum_hashbytesquorum_hash contains hash of validator quorum that will sign the block

RequestQuery

Query for data from the application at current or past height.

  • Optionally return Merkle proof.
  • Merkle proof includes self-describing type field to support many types of Merkle trees and encoding formats.
FieldTypeLabelDescription
databytesRaw query bytes. Can be used with or in lieu of Path.
pathstringPath field of the request URI. Can be used with or in lieu of data. Apps MUST interpret /store as a query by key on the underlying store. The key SHOULD be specified in the data field. Apps SHOULD allow queries over specific types like /accounts/... or /votes/...
heightint64The block height for which you want the query (default=0 returns data for the latest committed block). Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height-1.
proveboolReturn Merkle proof with response if possible.

RequestVerifyVoteExtension

Verify the vote extension

Tenderdash calls it for two kinds of request, told apart by validator_pro_tx_hash:

  • Precommit verification (validator_pro_tx_hash set): the vote extensions of one validator's Precommit, exactly as its Application returned them from ExtendVote.
  • Commit verification (validator_pro_tx_hash empty): the extension vector of a commit, after ProcessProposal of the committed block with the same hash, height and round (the commit round). The vector holds only the threshold-recoverable extensions (THRESHOLD_RECOVER and THRESHOLD_RECOVER_RAW), in the order ExtendVote returned them; extensions of any other type never appear in a commit. It can be empty.

Usage

  • RequestVerifyVoteExtension.vote_extensions can be empty. For a Precommit, the Application's interpretation of it should be that the Application running at the process that sent the vote chose not to extend it. Tenderdash will always call RequestVerifyVoteExtension, even for 0 length vote extensions.
  • For a Precommit, if ResponseVerifyVoteExtension.status is REJECT, Tenderdash will reject the whole received vote. See the Requirements section to understand the potential liveness implications of this.
  • For a commit, the Application MUST compare the vector with the threshold-recoverable subset of the extensions it expects for that block, in order, and return REJECT if they differ: a commit whose block signature is valid can still carry a stripped, duplicated or replayed vector. Tenderdash requires ACCEPT before it saves the block and commit or calls FinalizeBlock. What REJECT does depends on the path:
    • consensus: the commit is discarded without blaming its sender. A commit whose block the node already holds is checked first and never changes the round. A commit for a later round whose block has not arrived moves the node to that round on its threshold signature alone; it is checked when the block arrives, then the commits other peers sent meanwhile. Copies of a commit already waiting are dropped. If the node's own commit is rejected, nothing is persisted and the node waits for a peer's commit. The block is processed again only for a commit of a round other than the one processed last. After a rejection, Tenderdash calls ExtendVote for the block and round, and refuses, without processing the block again or calling the Application, a later commit of that round that would need the block processed again, when its vector differs from the one returned or equals it and was rejected. Consensus WAL replay follows these rules;
    • block sync: the block is not applied; the peer that served it is dropped and the height is requested again;
    • handshake catch-up (application behind the block store): the commit is already in the block store, so the node fails to start; there is no automatic recovery, and the operator has to roll back or re-sync the node.
  • Any status other than ACCEPT, including UNKNOWN, counts as REJECT. An error from the ABCI call itself, as opposed to a status, is fatal on every path.
  • The implementation of VerifyVoteExtension MUST be deterministic. Moreover, the value of ResponseVerifyVoteExtension.status MUST exclusively depend on the parameters passed in the call to RequestVerifyVoteExtension, the block processed by ProcessProposal, and the last committed Application state (see Requirements section).
  • For a Precommit, application implementers SHOULD always set ResponseVerifyVoteExtension.status to ACCEPT, unless they really know what the potential liveness implications of returning REJECT are.

When does Tenderdash call it?

When a validator p is in Tenderdash consensus height h and receives, from another validator q, a Precommit message for a block (not nil) at height h:

  1. If the Precommit message does not contain a vote extensions with a valid signature, Tenderdash discards the message as invalid.
  • a 0-length vote extensions is valid as long as its accompanying signature is also valid.
  1. Else, p's Tenderdash calls RequestVerifyVoteExtension.
  2. The Application returns accept or reject via ResponseVerifyVoteExtension.status.
  3. If the Application returns
  • accept, p's Tenderdash will keep the received vote, together with its corresponding vote extension in its internal data structures. Threshold-recoverable extensions are recovered into CommitInfo when the commit is assembled.
  • reject, p's Tenderdash will deem the Precommit message invalid and discard it.

Whenever a process is about to save a block and its commit, in consensus, block sync or replay, it first calls RequestVerifyVoteExtension for the commit, once ProcessProposal has processed the block. This includes commits the process assembled from its own +2/3 precommits.

FieldTypeLabelDescription
hashbytesThe header hash of the propsed block that the vote extensions refers to.
validator_pro_tx_hashbytesProTxHash of the validator that signed the extensions; empty for commit verification.
heightint64Height of the block (for sanity check).
roundint32Round of the vote; for commit verification, the commit round.
vote_extensionsExtendVoteExtensionrepeatedApplication-specific information signed by Tenderdash. Can have 0 length.

Response

FieldTypeLabelDescription
exceptionResponseException
echoResponseEcho
flushResponseFlush
infoResponseInfo
init_chainResponseInitChain
queryResponseQuery
check_txResponseCheckTx
list_snapshotsResponseListSnapshots
offer_snapshotResponseOfferSnapshot
load_snapshot_chunkResponseLoadSnapshotChunk
apply_snapshot_chunkResponseApplySnapshotChunk
prepare_proposalResponsePrepareProposal
process_proposalResponseProcessProposal
extend_voteResponseExtendVote
verify_vote_extensionResponseVerifyVoteExtension
finalize_blockResponseFinalizeBlock

ResponseApplySnapshotChunk

FieldTypeLabelDescription
resultResponseApplySnapshotChunk.ResultThe result of applying this chunk.
refetch_chunksbytesrepeatedIdentifiers of chunks to discard, refetch and reapply, regardless of result.
reject_sendersstringrepeatedReject the given P2P senders, regardless of Result. Any chunks already applied will not be refetched unless explicitly requested, but queued chunks from these senders will be discarded, and new chunks or other snapshots rejected.
next_chunksbytesrepeatedChunk identifiers to enqueue after ACCEPT; ignored for other results.

ResponseCheckTx

FieldTypeLabelDescription
codeuint32Response code.
databytesResult bytes, if any.
infostringAdditional information. May be non-deterministic.
gas_wantedint64Amount of gas requested for transaction.
codespacestringNamespace for the code.
senderstringThe transaction's sender (e.g. the signer).
priorityint64The transaction's priority (for mempool ordering).

ResponseEcho

FieldTypeLabelDescription
messagestringThe input string.

ResponseException

nondeterministic

FieldTypeLabelDescription
errorstring

ResponseExtendVote

FieldTypeLabelDescription
vote_extensionsExtendVoteExtensionrepeatedOptional information signed by Tenderdash.

ResponseFinalizeBlock

FieldTypeLabelDescription
retain_heightint64Blocks below this height may be removed. Defaults to 0 (retain all).
propose_next_block_immediatelyboolWhen true, this node does not wait for transactions (create-empty-blocks-interval) before proposing at the next height, in round 0 or, on consensus version 2, in a round entered after a silent proposer's round was skipped: the application has block-driven work pending, for example withdrawal transactions to sign. A hint for the local node only, consumed once; a node that restarts in between falls back to the configured wait. Defaults to false.

ResponseFlush

ResponseInfo

FieldTypeLabelDescription
datastringSome arbitrary information.
versionstringThe application software semantic version.
app_versionuint64The application protocol version.
last_block_heightint64Last height persisted by FinalizeBlock.
last_block_app_hashbytesApplication hash at the last persisted height. 32 bytes.

ResponseInitChain

FieldTypeLabelDescription
consensus_paramstendermint.types.ConsensusParamsInitial consensus-critical parameters (optional).
app_hashbytesInitial application hash. 32 bytes.
validator_set_updateValidatorSetUpdateInitial validator set (optional).
next_core_chain_lock_updatetendermint.types.CoreChainLockInitial core chain lock update.
initial_core_heightuint32Initial height of core lock.
genesis_timegoogle.protobuf.TimestampoptionalOverride genesis time with provided time.

ResponseListSnapshots

FieldTypeLabelDescription
snapshotsSnapshotrepeatedList of local state snapshots.

ResponseLoadSnapshotChunk

FieldTypeLabelDescription
chunkbytesThe binary chunk contents, in an arbitray format. Chunk messages cannot be larger than 16 MB including metadata, so 10 MB is a good starting point.

ResponseOfferSnapshot

FieldTypeLabelDescription
resultResponseOfferSnapshot.ResultThe result of the snapshot offer.

ResponsePrepareProposal

FieldTypeLabelDescription
tx_recordsTxRecordrepeatedPossibly modified list of transactions that have been picked as part of the proposed block.
app_hashbytesThe Merkle root hash of the application state. 32 bytes.
tx_resultsExecTxResultrepeatedList of structures containing the data resulting from executing the transactions.
consensus_param_updatestendermint.types.ConsensusParamsChanges to consensus-critical gas, size, and other parameters that will be applied at next height.
core_chain_lock_updatetendermint.types.CoreChainLockCore chain lock that will be used for next block.
validator_set_updateValidatorSetUpdateChanges to validator set that will be applied at next height.
app_versionuint64Application version that was used to create the current proposal.

ResponseProcessProposal

FieldTypeLabelDescription
statusResponseProcessProposal.ProposalStatusenum that signals if the application finds the proposal valid.
app_hashbytesThe Merkle root hash of the application state. 32 bytes.
tx_resultsExecTxResultrepeatedList of structures containing the data resulting from executing the transactions.
consensus_param_updatestendermint.types.ConsensusParamsChanges to consensus-critical gas, size, and other parameters.
validator_set_updateValidatorSetUpdateChanges to validator set (set voting power to 0 to remove).
eventsEventrepeatedType & Key-Value events for indexing

ResponseQuery

FieldTypeLabelDescription
codeuint32Response code.
logstringThe output of the application's logger. May be non-deterministic.
infostringAdditional information. May be non-deterministic.
indexint64The index of the key in the tree.
keybytesThe key of the matching data.
valuebytesThe value of the matching data.
proof_opstendermint.crypto.ProofOpsSerialized proof for the value data, if requested, to be verified against the app_hash for the given Height.
heightint64The block height from which data was derived. Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height-1.
codespacestringNamespace for the code.

ResponseVerifyVoteExtension

FieldTypeLabelDescription
statusResponseVerifyVoteExtension.VerifyStatusenum signaling if the application accepts the vote extension

Snapshot

FieldTypeLabelDescription
heightuint64The height at which the snapshot was taken
versionuint32The application-specific snapshot version
hashbytesArbitrary snapshot hash, equal only if identical
metadatabytesArbitrary application metadata

ThresholdPublicKeyUpdate

FieldTypeLabelDescription
threshold_public_keytendermint.crypto.PublicKey

TxRecord

FieldTypeLabelDescription
actionTxRecord.TxActionWhat should Tenderdash do with this transaction?
txbytesTransaction contents.

TxResult

TxResult contains results of executing the transaction.

One usage is indexing transaction results.

FieldTypeLabelDescription
heightint64
indexuint32
txbytes
resultExecTxResult

Validator

Validator

FieldTypeLabelDescription
powerint64The voting power
pro_tx_hashbytes

ValidatorSetUpdate

ValidatorSetUpdate represents a change in the validator set. It can be used to add, remove, or update a validator.

Validator set update consists of multiple ValidatorUpdate records, each of them can be used to add, remove, or update a validator, according to the following rules:

  1. If a validator with the same public key already exists in the validator set and power is greater than 0, the existing validator will be updated with the new power.
  2. If a validator with the same public key already exists in the validator set and power is 0, the existing validator will be removed from the validator set.
  3. If a validator with the same public key does not exist in the validator set and the power is greater than 0, a new validator will be added to the validator set.
  4. As a special case, if quorum hash has changed, all existing validators will be removed before applying the new validator set update.
FieldTypeLabelDescription
validator_updatesValidatorUpdaterepeated
threshold_public_keytendermint.crypto.PublicKey
quorum_hashbytes

ValidatorUpdate

ValidatorUpdate

FieldTypeLabelDescription
pub_keytendermint.crypto.PublicKey
powerint64
pro_tx_hashbytes
node_addressstringnode_address is an URI containing address of validator (proto://node_id@ip_address:port), for example: tcp://f2dbd9b0a1f541a7c44d34a58674d0262f5feca5@12.34.5.6:1234

VoteInfo

VoteInfo

FieldTypeLabelDescription
validatorValidator
signed_last_blockbool

CheckTxType

Type of transaction check

NameNumberDescription
NEW0NEW is the default and means that a full check of the tranasaction is required.
RECHECK1RECHECK is used when the mempool is initiating a normal recheck of a transaction.

MisbehaviorType

NameNumberDescription
UNKNOWN0
DUPLICATE_VOTE1
LIGHT_CLIENT_ATTACK2

ResponseApplySnapshotChunk.Result

NameNumberDescription
UNKNOWN0Unknown result, abort all snapshot restoration
ACCEPT1Chunk successfully accepted
ABORT2Abort all snapshot restoration
RETRY3Retry chunk (combine with refetch and reject)
RETRY_SNAPSHOT4Retry snapshot (combine with refetch and reject)
REJECT_SNAPSHOT5Reject this snapshot, try others
COMPLETE_SNAPSHOT6Complete this snapshot, no more chunks

ResponseOfferSnapshot.Result

NameNumberDescription
UNKNOWN0Unknown result, abort all snapshot restoration
ACCEPT1Snapshot accepted, apply chunks
ABORT2Abort all snapshot restoration
REJECT3Reject this specific snapshot, try others
REJECT_FORMAT4Reject all snapshots of this format, try others
REJECT_SENDER5Reject all snapshots from the sender(s), try others

ResponseProcessProposal.ProposalStatus

NameNumberDescription
UNKNOWN0Unspecified error occurred
ACCEPT1Proposal accepted
REJECT2Proposal is not valid

ResponseVerifyVoteExtension.VerifyStatus

NameNumberDescription
UNKNOWN0
ACCEPT1
REJECT2

TxRecord.TxAction

TxAction contains App-provided information on what to do with a transaction that is part of a raw proposal

NameNumberDescription
UNKNOWN0Unknown action
UNMODIFIED1The Application did not modify this transaction.
ADDED2The Application added this transaction.
REMOVED3The Application wants this transaction removed from the proposal and the mempool.
DELAYED4The Application wants this transaction removed from the proposal but not the mempool.

ABCIApplication

Scalar Value Types

.proto TypeNotesC++JavaPythonGoC#PHPRuby
doubledoubledoublefloatfloat64doublefloatFloat
floatfloatfloatfloatfloat32floatfloatFloat
int32Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.int32intintint32intintegerBignum or Fixnum (as required)
int64Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.int64longint/longint64longinteger/stringBignum
uint32Uses variable-length encoding.uint32intint/longuint32uintintegerBignum or Fixnum (as required)
uint64Uses variable-length encoding.uint64longint/longuint64ulonginteger/stringBignum or Fixnum (as required)
sint32Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.int32intintint32intintegerBignum or Fixnum (as required)
sint64Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.int64longint/longint64longinteger/stringBignum
fixed32Always four bytes. More efficient than uint32 if values are often greater than 2^28.uint32intintuint32uintintegerBignum or Fixnum (as required)
fixed64Always eight bytes. More efficient than uint64 if values are often greater than 2^56.uint64longint/longuint64ulonginteger/stringBignum
sfixed32Always four bytes.int32intintint32intintegerBignum or Fixnum (as required)
sfixed64Always eight bytes.int64longint/longint64longinteger/stringBignum
boolboolbooleanbooleanboolboolbooleanTrueClass/FalseClass
stringA string must always contain UTF-8 encoded or 7-bit ASCII text.stringStringstr/unicodestringstringstringString (UTF-8)
bytesMay contain any arbitrary sequence of bytes.stringByteStringstr[]byteByteStringstringString (ASCII-8BIT)

Data Structures

Here we describe the data structures in the Tenderdash blockchain and the rules for validating them.

The Tenderdash blockchains consists of a short list of data types:

ProTxHash

ProTxHash is 32 bytes array. Validator provides this parameter via a config.

Block

A block consists of a header, transactions, the commit of the previous block, an optional core chain lock, and a list of evidence of malfeasance (ie. signing conflicting votes).

NameTypeDescriptionValidation
HeaderHeaderHeader corresponding to the block. This field contains information used throughout consensus and other areas of the protocol. To find out what it contains, visit [header] (#header)Must adhere to the validation rules of header
DataDataData contains a list of transactions. The contents of the transaction is unknown to Tenderdash.This field can be empty or populated, but no validation is performed. Applications can perform validation on individual transactions prior to block creation using checkTx.
EvidenceEvidenceListEvidence contains a list of infractions committed by validators.Can be empty, but when populated the validations rules from evidenceList apply
LastCommitCommitThe threshold-signed commit of the previous block.Must not be nil, must carry no threshold block signature at the initial height, and must adhere to the validation rules of commit.
CoreChainLockCoreChainLock (optional)Core chain lock carried by the block.When present, must pass CoreChainLock.ValidateBasic.

Execution

Tenderdash uses same-block execution: PrepareProposal and ProcessProposal return this block's AppHash, transaction results, and pending validator/parameter changes. FinalizeBlock persists the accepted candidate; there is no ABCI Commit call. Changes from block h become active at h + 1.

See the canonical primary flow, proposal-validity summary, and commit application for execution, validation and persistence order. The local committed state is summarized in State; pending next-state values belong to CurrentRoundState.

A block header contains metadata about the block and about the consensus, as well as commitments to the data in the current block, the previous block, and the results returned by the application:

NameTypeDescriptionValidation
VersionVersionVersion defines the application and protocol version being used.Must adhere to the validation rules of Version
ChainIDStringChainID is the ID of the chain. This must be unique to your chain.ChainID must be at most 50 bytes.
Heightint64Height is the height for this header.Must be > 0, >= initialHeight, and == previous Height+1
CoreChainLockedHeightuint32Core chain height used to verify this block.Must match the block's core chain lock, or the state's last locked height without one (ValidateBlockChainLock).
TimeTimeThe timestamp is chosen by the proposer from its local clock (genesis time for the initial block) and accepted by validators only if timely; see Proposal timestamps. The historical median-based rule is described in the BFT-time section.Above the initial height, Time must be greater than the previous block's time. At the initial height, Time must be >= genesis time; a normal prevote additionally requires it to equal genesis time (rule P-Timely).
LastBlockIDBlockIDBlockID of the previous block.Must adhere to the validation rules of blockID. The first block has block.Header.LastBlockID == BlockID{}.
LastCommitHashslice of bytes ([]byte)Checksum of the last commit's threshold block signature (Commit.Hash).Empty or of length 32; must equal LastCommit.Hash()
DataHashslice of bytes ([]byte)MerkleRoot of the hash of transactions. Note: The transactions are hashed before being included in the merkle tree, the leaves of the Merkle tree are the hashes, not the transactions themselves.Empty or of length 32; must equal Data.Hash()
ValidatorsHashslice of bytes ([]byte)Merkle root of the current validator set's threshold public key and quorum hash (ValidatorSet.Hash). The validator list is not hashed, so its order, ascending ProTxHash (ValidatorsByProTxHashes), does not affect this value.Empty or of length 32
NextValidatorsHashslice of bytes ([]byte)Merkle root of the next validator set's threshold public key and quorum hash (ValidatorSet.Hash). The validator list is not hashed, so its order, ascending ProTxHash (ValidatorsByProTxHashes), does not affect this value.Empty or of length 32
ConsensusHashslice of bytes ([]byte)SHA-256 of the protobuf-encoded HashedParams subset of the current consensus parameters (HashConsensusParams).Empty or of length 32
NextConsensusHashslice of bytes ([]byte)The same hash over the consensus parameters of the next block.Empty or of length 32
AppHashslice of bytes ([]byte)Application state hash after executing this block's transactions, returned by the application from PrepareProposal and ProcessProposal.The application's response must be 32 bytes; must equal the application's answer (ValidateBlockWithRoundState).
ResultsHashslice of bytes ([]byte)Merkle root of this block's ExecTxResult values, keeping only Code, Data and GasUsed (TxResultsHash).Empty or of length 32; must equal the application's answer
EvidenceHashslice of bytes ([]byte)MerkleRoot of the evidence of Byzantine behaviour included in this block.Empty or of length 32; must equal Evidence.Hash()
ProposedAppVersionuint64Proposer's latest available application protocol version.
ProposerProTxHashProTxHashProTxHash of the original proposer of the block. Validator must be in the current validatorSet.Must be of length 32

Version

NOTE: that this is more specifically the consensus version and doesn't include information like the P2P Version. (TODO: we should write a comprehensive document about versioning that this can refer to)

NametypeDescriptionValidation
Blockuint64This number represents the version of the block protocol and must be the same throughout an operational networkMust be equal to protocol version being used in a network (block.Version.Block == state.Version.Consensus.Block)
Appuint64Application version used for this block; PrepareProposal can override it.Basic block validation does not require equality with the last committed application version (validateBlock in internal/state/validation.go).

BlockID

The BlockID contains two distinct Merkle roots of the block, the number of parts (ie. len(MakeParts(block))) and a state ID hash.

NameTypeDescriptionValidation
Hashslice of bytes ([]byte)MerkleRoot of all the fields in the header (ie. MerkleRoot(header).Empty (nil vote) or of length 32
PartSetHeaderPartSetHeaderUsed for secure gossiping of the block during consensus, is the MerkleRoot of the complete serialized block cut into parts (ie. MerkleRoot(MakeParts(block))).Must adhere to the validation rules of PartSetHeader
StateIDslice of bytes ([]byte)Hash of the StateID fields a light client needs to verify state; see Signing.Must be of length 32 when Hash is not empty

See MerkleRoot for details.

PartSetHeader

NameTypeDescriptionValidation
Totaluint32Total amount of parts for a blockMust be <= MaxBlockPartsCount; > 0 in a complete BlockID
Hashslice of bytes ([]byte)MerkleRoot of a serialized blockEmpty or of length 32

Part

Part defines a part of a block. In Tenderdash blocks are broken into parts for gossip.

NameTypeDescriptionValidation
indexuint32Index of this partMust equal Proof.Index
bytesbytesContent of this partMust be <= BlockPartSizeBytes long
proofProofMerkle proof of this partMust pass Proof.ValidateBasic

Time

Tenderdash uses the Google.Protobuf.Timestamp format, which uses two integers, one 64 bit integer for Seconds and a 32 bit integer for Nanoseconds.

Data

Data is just a wrapper for a list of transactions, where transactions are arbitrary byte arrays:

NameTypeDescriptionValidation
TxsMatrix of bytes ([][]byte)Slice of transactions.Validation does not occur on this field, this data is unknown to Tenderdash

Commit

Commit proves that a quorum decided a block: it carries the threshold-recovered signatures rather than one signature per validator.

NameTypeDescriptionValidation
Heightint64Height at which this commit was created.Must be >= 0
Roundint32Round that the commit corresponds to.Must be >= 0
BlockIDBlockIDThe blockID of the corresponding block.At height >= 1, must not be nil and must adhere to the validation rules of BlockID.
QuorumHashslice of bytes ([]byte)Hash of the quorum that signed the commit.
ThresholdBlockSignatureSignatureThreshold-recovered block signature.At height >= 1, must be of length 96
ThresholdVoteExtensionsArray of VoteExtensionThreshold-recovered vote extensions.No nil entries; every type must be a defined enum value

BlockIDFlag

BlockIDFlag is defined in types.proto, but Commit has no per-validator signatures and does not use it.

enum BlockIDFlag {
  BLOCK_ID_FLAG_UNKNOWN = 0;
  BLOCK_ID_FLAG_ABSENT  = 1; // signatures for other blocks are also considered absent
  BLOCK_ID_FLAG_COMMIT  = 2;
  BLOCK_ID_FLAG_NIL     = 3;
}

Vote

A vote is a signed message from a validator for a particular block. The vote includes information about the validator signing it. When stored in the blockchain or propagated over the network, votes are encoded in Protobuf. The vote extension is not part of the CanonicalVote.

NameTypeDescriptionValidation
TypeSignedMsgTypeEither prevote or precommit. SignedMsgTypeA Vote is valid if its corresponding fields are included in the enum signedMsgType
Heightint64Height for which this vote was created.Must be >= 0
Roundint32Round that the vote corresponds to.Must be >= 0
BlockIDBlockIDThe blockID of the corresponding block; zero for a nil vote.Must adhere to BlockID validation and be either nil or complete
ValidatorProTxHashProTxHashProTxHash of the validator.Length must be equal to 32
ValidatorIndexint32Index of the validator in the validator set at this height.Must be >= 0
BlockSignatureSignatureValidator's signature of the CanonicalVote sign bytes.Length must be > 0 and <= 96
VoteExtensionsArray of VoteExtensionVote extensions provided by the Application.Must be empty unless this is a non-nil precommit, which must pass VoteExtension validation

VoteExtension

NameTypeDescriptionValidation
TypeVoteExtensionTypeTHRESHOLD_RECOVER or THRESHOLD_RECOVER_RAW. DEFAULT exists in the enum but is not supported.Must not be DEFAULT
Extensionslice of bytes ([]byte)Extension data provided by the Application.Must be of length 32 for THRESHOLD_RECOVER_RAW
SignatureSignatureValidator's signature of the extension.Must be present when Extension is not empty; length must be <= 96
SignRequestIdslice of bytes ([]byte, optional)Application-supplied sign request ID.

CanonicalVote

CanonicalVote is for validator signing. This type will not be present in a block. For signing, votes are represented via CanonicalVote and encoded using protobuf via Vote.SignBytes, which includes the ChainID and omits ValidatorProTxHash and ValidatorIndex. block_id is a checksum of the CanonicalBlockID, and state_id is the BlockID's StateID.

message CanonicalVote {
  SignedMsgType type     = 1;
  sfixed64      height   = 2;
  sfixed64      round    = 3;
  bytes         block_id = 4;
  bytes         state_id = 5;
  string        chain_id = 99;
}

Vote.Verify(chainID, quorumType, quorumHash, pubKey, proTxHash) checks that proTxHash matches ValidatorProTxHash, then verifies the block signature, and any extension signatures, against the 48-byte BLS public key. See Signing.

Proposal

Proposal contains height and round for which this proposal is made, BlockID as a unique identifier of proposed block, timestamp, and POLRound (a so-called Proof-of-Lock (POL) round) that is needed for termination of the consensus. If POLRound >= 0, then BlockID corresponds to the block that is locked in POLRound. The message is signed by the validator private key.

NameTypeDescriptionValidation
TypeSignedMsgTypeRepresents a Proposal SignedMsgTypeMust be ProposalType signedMsgType
Heightint64Height for which this proposal was createdMust be >= 0
Roundint32Round that the proposal corresponds to.Must be >= 0
POLRoundint32Proof of lockMust be >= -1
BlockIDBlockIDThe blockID of the corresponding block.Must be complete and adhere to BlockID validation
TimestampTimeTimestamp chosen by the proposer; equal to the block time.Time
SignatureSignatureSignature by the proposer.Length of signature must be > 0 and <= 96
CoreChainLockedHeightuint32Core chain locked height used to sign and verify the block.Must not be MaxUint32
CoreChainLockUpdateCoreChainLock (optional)Core chain lock used to sign and verify the next block.When present, must pass CoreChainLock.ValidateBasic

SignedMsgType

Signed message type represents a signed messages in consensus.

enum SignedMsgType {

  SIGNED_MSG_TYPE_UNKNOWN = 0;
  // Votes
  SIGNED_MSG_TYPE_PREVOTE   = 1;
  SIGNED_MSG_TYPE_PRECOMMIT = 2;

  // Proposal
  SIGNED_MSG_TYPE_PROPOSAL = 32;

  // Proposer liveness, from CONSENSUS_VERSION_2
  SIGNED_MSG_TYPE_PROPOSER_HEARTBEAT = 48;
  SIGNED_MSG_TYPE_ROUND_SKIP = 49;
}

Signature

Signatures in Tenderdash are raw bytes representing the underlying signature. Consensus signatures are BLS12-381 signatures of 96 bytes (types.SignatureSize).

See the signature spec for more.

EvidenceList

EvidenceList is a simple wrapper for a list of evidence:

NameTypeDescriptionValidation
EvidenceArray of EvidenceList of verified evidenceValidation adheres to individual types of Evidence

Evidence

Evidence in Tenderdash is used to indicate breaches in the consensus by a validator.

More information on how evidence works in Tenderdash can be found here

DuplicateVoteEvidence

DuplicateVoteEvidence represents a validator that has voted for two different blocks in the same round of the same height. Votes are lexicographically sorted on BlockID.

NameTypeDescriptionValidation
VoteAVoteOne of the votes submitted by a validator when they equivocatedVoteA must adhere to Vote validation rules
VoteBVoteThe second vote submitted by a validator when they equivocatedVoteB must adhere to Vote validation rules
TotalVotingPowerint64The total power of the validator set at the height of equivocationMust be equal to nodes own copy of the data
ValidatorPowerint64Power of the equivocating validator at the heightMust be equal to the nodes own copy of the data
TimestampTimeTime of the block where the equivocation occurredMust be equal to the nodes own copy of the data

LightBlock

LightBlock is the core data structure of the light client. It combines two data structures needed for verification (signedHeader & validatorSet).

NameTypeDescriptionValidation
SignedHeaderSignedHeaderThe header and commit, these are used for verification purposes. To find out more visit light client docsMust not be nil and adhere to the validation rules of signedHeader
ValidatorSetValidatorSetThe validatorSet is used to help with verify that the validators in that committed the infraction were truly in the validator set.Must not be nil and adhere to the validation rules of validatorSet

The SignedHeader and ValidatorSet are linked by the hash of the validator set(SignedHeader.ValidatorsHash == ValidatorSet.Hash().

SignedHeader

The SignedhHeader is the header accompanied by the commit to prove it.

NameTypeDescriptionValidation
HeaderHeaderHeaderHeader cannot be nil and must adhere to the Header validation criteria
CommitCommitCommitCommit cannot be nil and must adhere to the Commit criteria

ValidatorSet

NameTypeDescriptionValidation
ValidatorsArray of validatorList of the active validators at a specific height, sorted by ascending ProTxHashThe list of validators can not be empty or nil and must adhere to the validation rules of validator
ProposervalidatorValidator at the set's proposer indexThe proposer index must be in range
ThresholdPublicKeypublic keyThreshold public key of the quorumMust pass ThresholdPublicKeyValid
QuorumTypeint32LLMQ type of the quorum
QuorumHashslice of bytes ([]byte)Hash of the quorumMust pass QuorumHashValid
HasPublicKeysboolWhether validators carry public keys, which holds when the local node is in the setWhen true, each public key must be valid
VotingPowerThresholduint64Minimum voting power required to commit; equals ValidatorParams.voting_power_threshold

Validator

NameTypeDescriptionValidation
ProTxHashProTxHashValidator's identityLength must be of size 32
PubKeyslice of bytes ([]byte)Validator's BLS public key; absent when the local node is not in the setLength must be 48 when validated
VotingPowerint64Validator's voting powercannot be < 0
NodeAddressValidatorAddressURI of the validator's nodeMust be valid when set

Address

Address is a type alias of a slice of bytes, calculated by hashing the public key using sha256 and truncating it to the first 20 bytes (crypto.AddressHash). Validators are identified by ProTxHash, not by address.

ConsensusParams

NameTypeDescriptionField Number
blockBlockParamsParameters limiting the size and gas of a block.1
evidenceEvidenceParamsParameters limiting the validity of evidence of byzantine behaviour.2
validatorValidatorParamsParameters limiting the types of public keys validators can use.3
versionVersionParamsThe ABCI application and consensus versions.4
synchronySynchronyParamsBounds under which a proposal timestamp is considered timely.5
timeoutTimeoutParamsTimeouts of the consensus steps.6
abciABCIParamsApplication Blockchain Interface options.7

BlockParams

NameTypeDescriptionField Number
max_bytesint64Max size of a block, in bytes.1
max_gasint64Max sum of GasWanted in a proposed block. NOTE: blocks that violate this may be committed if there are Byzantine proposers. It's the application's responsibility to handle this when processing a block!2

EvidenceParams

NameTypeDescriptionField Number
max_age_num_blocksint64Max age of evidence, in blocks.1
max_age_durationgoogle.protobuf.DurationMax age of evidence, in time. It should correspond with an app's "unbonding period" or other similar mechanism for handling Nothing-At-Stake attacks.2
max_bytesint64maximum size in bytes of total evidence allowed to be entered into a block3

ValidatorParams

NameTypeDescriptionField Number
pub_key_typesrepeated stringList of accepted public key types. Uses same naming as PubKey.Type.1
voting_power_thresholduint64 (optional)Minimum voting power required to commit a block. See voting_power_threshold.2

voting_power_threshold

A positive voting_power_threshold sets the vote-counting threshold T returned by ValidatorSet.QuorumVotingThresholdPower; zero uses the quorum-type threshold. Every validator contributes power 100. An explicit value must not exceed total power, but a threshold below the strict BFT floor is only warned about.

This override does not lower the separate BLS recovery threshold R returned by QuorumTypeThresholdVotingPower. Prevotes expose a majority at T; precommits also require successful recovery and verification of threshold signatures. A low vote-counting threshold alone does not let one signature share produce a commit. A threshold above R disables round skipping (Proposer liveness). See Threshold recovery for the canonical extension-group and recovery rules, and Commit threshold is configurable below the BFT bound for the documented safety limitation. Changes to this live consensus parameter must not be assumed incapable of changing consensus behavior merely because a parameter is absent from a particular header-hash subset.

VersionParams

NameTypeDescriptionField Number
app_versionuint64The ABCI application version.1
consensus_versionConsensusVersionVersion of the consensus protocol, selecting the proposer strategy. CONSENSUS_VERSION_2 also enables proposer liveness.2

SynchronyParams

NameTypeDescriptionField Number
message_delaygoogle.protobuf.DurationBound for how long a proposal message may take to reach all validators on a newtork and still be considered valid.1
precisiongoogle.protobuf.DurationBound for how skewed a proposer's clock may be from any validator on the network while still producing valid proposals.2

TimeoutParams

NameTypeDescriptionField Number
proposegoogle.protobuf.DurationParameter that, along with propose_delta, configures the timeout for the propose step of the consensus algorithm.1
propose_deltagoogle.protobuf.DurationParameter that, along with propose, configures the timeout for the propose step of the consensus algorithm.2
votegoogle.protobuf.DurationParameter that, along with vote_delta, configures the timeout for the prevote and precommit step of the consensus algorithm.3
vote_deltagoogle.protobuf.DurationParameter that, along with vote, configures the timeout for the prevote and precommit step of the consensus algorithm.4
proposer_heartbeatgoogle.protobuf.DurationHow often the proposer of a round that waits for transactions sends a heartbeat. Used from CONSENSUS_VERSION_2; unset means derived (see Parameters).7
proposer_silencegoogle.protobuf.DurationHow long a validator waits without a heartbeat before it votes to skip the round; grows by proposer_heartbeat per round. Used from CONSENSUS_VERSION_2; unset means derived.8

ABCIParams

NameTypeDescriptionField Number
recheck_txboolIndicates whether to run CheckTx on all remaining transactions after every execution of a block1

Proof

NameTypeDescriptionField Number
totalint64Total number of items.1
indexint64Index item to prove.2
leaf_hashbytesHash of item value.3
auntsrepeated bytesHashes from leaf's sibling to a root's child.4

Encoding

Protocol Buffers

Tenderdash uses Protocol Buffers, specifically proto3, for all data structures.

Please see the Proto3 language guide for more details.

Byte Arrays

The encoding of a byte array is simply the raw-bytes prefixed with the length of the array as a UVarint (what proto calls a Varint).

For details on varints, see the protobuf spec.

For example, the byte-array [0xA, 0xB] would be encoded as 0x020A0B, while a byte-array containing 300 entires beginning with [0xA, 0xB, ...] would be encoded as 0xAC020A0B... where 0xAC02 is the UVarint encoding of 300.

Hashing

Tenderdash uses SHA256 as its hash function. Objects are always serialized before being hashed. So SHA256(obj) is short for SHA256(ProtoEncoding(obj)).

Public Key Cryptography

Tenderdash uses Protobuf Oneof to distinguish between different types public keys, and signatures. Additionally, for each public key, Tenderdash defines an Address function that can be used as a more compact identifier in place of the public key. Here we list the concrete types, their names, and prefix bytes for public keys and signatures, as well as the address schemes for each PubKey. Note for brevity we don't include details of the private keys beyond their type and name.

Key Types

Each type specifies it's own pubkey, address, and signature format.

Ed25519

The address is the first 20-bytes of the SHA256 hash of the raw 32-byte public key:

address = SHA256(pubkey)[:20]

The signature is the raw 64-byte ED25519 signature.

Tenderdash uses zip215 for verification of ed25519 signatures.

Historical upstream note: this change was planned for Tendermint-Go 0.35.

Secp256k1

The address is the first 20-bytes of the SHA256 hash of the raw 32-byte public key:

address = SHA256(pubkey)[:20]

Other Common Types

BitArray

The BitArray is used in some consensus messages to represent votes received from validators, or parts received in a block. It is represented with a struct containing the number of bits (Bits) and the bit-array itself encoded in base64 (Elems).

NameType
bitsint64
elemsslice of int64 ([]int64)

Note BitArray receives a special JSON encoding in the form of x and _ representing 1 and 0. Ie. the BitArray 10110 would be JSON encoded as "x_xx_"

Part

Part is used to break up blocks into pieces that can be gossiped in parallel and securely verified using a Merkle tree of the parts.

Part contains the index of the part (Index), the actual underlying data of the part (Bytes), and a Merkle proof that the part is contained in the set (Proof).

NameType
indexuint32
bytesslice of bytes ([]byte)
proofproof

See details of SimpleProof, below.

MakeParts

Encode an object using Protobuf and slice it into parts. Tenderdash uses a part size of 65536 bytes, and allows a maximum of 1601 parts (see types.MaxBlockPartsCount). This corresponds to the hard-coded block size limit of 100MB.

func MakeParts(block Block) []Part

Merkle Trees

For an overview of Merkle trees, see wikipedia

We use the RFC 6962 specification of a merkle tree, with sha256 as the hash function. Merkle trees are used throughout Tenderdash to compute a cryptographic digest of a data structure. The differences between RFC 6962 and the simplest form a merkle tree are that:

  1. leaf nodes and inner nodes have different hashes. This is for "second pre-image resistance", to prevent the proof to an inner node being valid as the proof of a leaf. The leaf nodes are SHA256(0x00 || leaf_data), and inner nodes are SHA256(0x01 || left_hash || right_hash).

  2. When the number of items isn't a power of two, the left half of the tree is as big as it could be. (The largest power of two less than the number of items) This allows new leaves to be added with less recomputation. For example:

   Simple Tree with 6 items           Simple Tree with 7 items

              *                                  *
             / \                                / \
           /     \                            /     \
         /         \                        /         \
       /             \                    /             \
      *               *                  *               *
     / \             / \                / \             / \
    /   \           /   \              /   \           /   \
   /     \         /     \            /     \         /     \
  *       *       h4     h5          *       *       *       h6
 / \     / \                        / \     / \     / \
h0  h1  h2 h3                      h0  h1  h2  h3  h4  h5

MerkleRoot

The function MerkleRoot is a simple recursive function defined as follows:

// SHA256([]byte{})
func emptyHash() []byte {
    return tmhash.Sum([]byte{})
}

// SHA256(0x00 || leaf)
func leafHash(leaf []byte) []byte {
 return tmhash.Sum(append(0x00, leaf...))
}

// SHA256(0x01 || left || right)
func innerHash(left []byte, right []byte) []byte {
 return tmhash.Sum(append(0x01, append(left, right...)...))
}

// largest power of 2 less than k
func getSplitPoint(k int) { ... }

func MerkleRoot(items [][]byte) []byte{
 switch len(items) {
 case 0:
  return empthHash()
 case 1:
  return leafHash(items[0])
 default:
  k := getSplitPoint(len(items))
  left := MerkleRoot(items[:k])
  right := MerkleRoot(items[k:])
  return innerHash(left, right)
 }
}

Note: MerkleRoot operates on items which are arbitrary byte arrays, not necessarily hashes. For items which need to be hashed first, we introduce the Hashes function:

func Hashes(items [][]byte) [][]byte {
    return SHA256 of each item
}

Note: we will abuse notion and invoke MerkleRoot with arguments of type struct or type []struct. For struct arguments, we compute a [][]byte containing the protobuf encoding of each field in the struct, in the same order the fields appear in the struct. For []struct arguments, we compute a [][]byte by protobuf encoding the individual struct elements.

Merkle Proof

Proof that a leaf is in a Merkle tree is composed as follows:

NameType
totalint64
indexint64
leafHashslice of bytes ([]byte)
auntsMatrix of bytes ([][]byte)

Which is verified as follows:

func (proof Proof) Verify(rootHash []byte, leaf []byte) bool {
 assert(proof.LeafHash, leafHash(leaf)

 computedHash := computeHashFromAunts(proof.Index, proof.Total, proof.LeafHash, proof.Aunts)
    return computedHash == rootHash
}

func computeHashFromAunts(index, total int, leafHash []byte, innerHashes [][]byte) []byte{
 assert(index < total && index >= 0 && total > 0)

 if total == 1{
  assert(len(proof.Aunts) == 0)
  return leafHash
 }

 assert(len(innerHashes) > 0)

 numLeft := getSplitPoint(total) // largest power of 2 less than total
 if index < numLeft {
  leftHash := computeHashFromAunts(index, numLeft, leafHash, innerHashes[:len(innerHashes)-1])
  assert(leftHash != nil)
  return innerHash(leftHash, innerHashes[len(innerHashes)-1])
 }
 rightHash := computeHashFromAunts(index-numLeft, total-numLeft, leafHash, innerHashes[:len(innerHashes)-1])
 assert(rightHash != nil)
 return innerHash(innerHashes[len(innerHashes)-1], rightHash)
}

The number of aunts is limited to 100 (MaxAunts) to protect the node against DOS attacks. This limits the tree size to 2^100 leaves, which should be sufficient for any conceivable purpose.

IAVL+ Tree

Because Tenderdash only uses a Simple Merkle Tree, application developers are expect to use their own Merkle tree in their applications. For example, the IAVL+ Tree - an immutable self-balancing binary tree for persisting application state is used by the Cosmos SDK

JSON

Tenderdash has its own JSON encoding in order to keep backwards compatibility with the previous RPC layer.

Registered types are encoded as:

{
  "type": "<type name>",
  "value": <JSON>
}

For instance, an ED25519 PubKey would look like:

{
  "type": "tendermint/PubKeyEd25519",
  "value": "uZ4h63OFWuQ36ZZ4Bd6NF+/w9fWUwrOncrQsackrsTk="
}

Where the "value" is the base64 encoding of the raw pubkey bytes, and the "type" is the type name for Ed25519 pubkeys.

Signed Messages

Signed messages (eg. votes, proposals) in the consensus are encoded using protobuf.

When signing, the elements of a message are re-ordered so the fixed-length fields are first, making it easy to quickly check the type, height, and round. The ChainID is also appended to the end. We call this encoding the SignBytes. For instance, SignBytes for a vote is the protobuf encoding of the following struct:

message CanonicalVote {
  SignedMsgType             type      = 1;  
  sfixed64                  height    = 2;  // canonicalization requires fixed size encoding here
  sfixed64                  round     = 3;  // canonicalization requires fixed size encoding here
  CanonicalBlockID          block_id  = 4;
  google.protobuf.Timestamp timestamp = 5;
  string                    chain_id  = 6;
}

The field ordering and the fixed sized encoding for the first three fields is optimized to ease parsing of SignBytes in HSMs. It creates fixed offsets for relevant fields that need to be read in this context.

Note: All canonical messages are length prefixed.

For more details, see the signing spec. Also, see the motivating discussion in #1622.

Genesis

The genesis file is the starting point of a chain. An application will populate the app_state field in the genesis with their required fields. Tenderdash is not able to validate this section because it is unaware what application state consists of.

Genesis Fields

  • genesis_time: The time the blockchain started or will start. If nodes are started before this time they will sit idle until the time specified.
  • chain_id: The chain identifier. Every chain should have a unique identifier. When conducting a fork based upgrade, we recommend changing the chainid to avoid network or consensus errors.
  • initial_height: The starting height of the blockchain. When conducting a chain restart to avoid restarting at height 1, the network is able to start at a specified height.
  • consensus_params
    • block
      • max_bytes: The max amount of bytes a block can be.
      • max_gas: The maximum amount of gas that a block can have.
    • evidence
      • max_age_num_blocks: After this preset amount of blocks has passed a single piece of evidence is considered invalid.
      • max_age_duration: After this preset amount of time has passed a single piece of evidence is considered invalid.
      • max_bytes: The max amount of bytes of all evidence included in a block.
    • validator
      • pub_key_types: Lists the validator consensus key types allowed by the chain. Recognized values are ed25519, secp256k1, and bls12381; the default allows only bls12381 (BLS12-381).
    • version
      • app_version: The version of the application. This is set by the application and is used to identify which version of the app a user should be using in order to operate a node.
      • consensus: The consensus protocol version (CONSENSUS_VERSION_*). It selects the proposer strategy; from version 2 it also enables proposer liveness and round skipping.
    • synchrony
      • message_delay: A bound on how long a proposal message may take to reach all validators on a network and still be considered valid.
      • precision: A bound on how skewed the proposer's clock may be from any validator on the network while still producing valid proposals.
    • timeout
      • propose: How long the Tenderdash consensus engine will wait for a proposal block before prevoting nil.
      • propose_delta: How much the propose timeout increase with each round.
      • vote: How long the consensus engine will wait after receiving +2/3 votes in a round.
      • vote_delta: How much the vote timeout increases with each round.
      • proposer_heartbeat: From consensus version 2, how often the proposer of a round that waits for transactions sends a heartbeat. Optional; see Parameters for the defaults.
      • proposer_silence: From consensus version 2, how long a validator waits without a heartbeat before voting to skip the round. Optional; see Parameters for the defaults.
  • validators
    • This is an array of validators. This validator set is used as the starting validator set of the chain. This field can be empty, if the application sets the validator set in InitChain.
  • app_hash: The applications state root hash. This field does not need to be populated at the start of the chain, the application may provide the needed information via Initchain.
  • app_state: This section is filled in by the application and is unknown to Tenderdash.

Core protocol

This section describes the core types and functionality of the Tenderdash protocol implementation.

State

The committed State in internal/state/state.go contains the information needed to validate the next block. It is local state, not a network message or a hash of the whole consensus state. Header commitments and the signed StateID have their own definitions in Data Structures.

FieldsMeaning
ChainID, InitialHeightChain identity and first block height; the initial height is normally 1 and must be positive.
LastBlockHeight, LastBlockRound, LastBlockID, LastBlockTimeLast committed block and its commit round.
LastCoreChainLockedBlockHeightLast committed core chain lock height; unchanged when a block carries no new lock.
Validators, LastValidatorsActive validators for the next block, and the previous set used to verify its LastCommit.
ConsensusParamsParameters used to validate the next block.
LastAppHash, LastResultsHashApplication state and transaction-result commitments of the last committed block.
LastHeightValidatorsChanged, LastHeightConsensusParamsChangedHeights at which changes become active.
VersionConsensus block/application versions and software version.

The ephemeral round state is separate from this committed state. Pending next validators and parameters are held in CurrentRoundState, not a committed State.NextValidators field.

Execution

Tenderdash uses same-block execution: PrepareProposal and ProcessProposal return the application hash and transaction results for the block being proposed. FinalizeBlock persists the accepted candidate state. There is no ABCI Commit call, and ResponseFinalizeBlock does not return an application hash.

The authoritative ordering of validation, block/WAL writes, FinalizeBlock, mempool/evidence updates and state persistence is Commit and apply. Validator-set and parameter changes from block h take effect at h + 1; see Validator-set and parameter changes. For startup reconciliation of application, block-store and state-store heights, see Recovery and synchronisation.

Version

Version distinguishes block protocol version from application version. ConsensusParams.Version.ConsensusVersion additionally selects the proposer-selection strategy and, from version 2, enables proposer liveness.

Block

ConsensusParams.Block.MaxBytes limits block size. Transaction selection uses Block.MaxGas, but applications must validate gas requirements when processing blocks; see BlockParams. The complete block/header/application checks are in the proposal-validity summary.

Evidence

ConsensusParams.Evidence bounds evidence age and bytes. Pending evidence is included during proposal creation. Conflicting-vote handling reports evidence to the pool; reporting is distinct from its inclusion in a block and application-level punishment.

Validator

Validators are identified by ProTxHash, ordered by ascending ProTxHash, and have equal DefaultDashVotingPower = 100. The active quorum includes its type, hash, threshold public key and voting-power threshold; see Terminology and ValidatorSet.

BFT Time

Historical and outdated. Tenderdash does not implement BFT Time. Block timestamps are chosen by the proposer and validated for timeliness as specified in Proposal timestamps and time rules; Proposer-Based Timestamps is historical design material. This page is kept because the PBTS design documents compare against it.

Tendermint provides a deterministic, Byzantine fault-tolerant, source of time. Time in Tendermint is defined with the Time field of the block header.

It satisfies the following properties:

  • Time Monotonicity: Time is monotonically increasing, i.e., given a header H1 for height h1 and a header H2 for height h2 = h1 + 1, H1.Time < H2.Time.
  • Time Validity: Given a set of Commit votes that forms the block.LastCommit field, a range of valid values for the Time field of the block header is defined only by
    Precommit messages (from the LastCommit field) sent by correct processes, i.e., a faulty process cannot arbitrarily increase the Time value.

In the context of Tendermint, time is of type int64 and denotes UNIX time in milliseconds, i.e., corresponds to the number of milliseconds since January 1, 1970. Before defining rules that need to be enforced by the Tendermint consensus protocol, so the properties above holds, we introduce the following definition:

  • median of a Commit is equal to the median of Vote.Time fields of the Vote messages, where the value of Vote.Time is counted number of times proportional to the process voting power. As in Tendermint the voting power is not uniform (one process one vote), a vote message is actually an aggregator of the same votes whose number is equal to the voting power of the process that has casted the corresponding votes message.

Let's consider the following example:

  • we have four processes p1, p2, p3 and p4, with the following voting power distribution (p1, 23), (p2, 27), (p3, 10) and (p4, 10). The total voting power is 70 (N = 3f+1, where N is the total voting power, and f is the maximum voting power of the faulty processes), so we assume that the faulty processes have at most 23 of voting power. Furthermore, we have the following vote messages in some LastCommit field (we ignore all fields except Time field):
    • (p1, 100), (p2, 98), (p3, 1000), (p4, 500). We assume that p3 and p4 are faulty processes. Let's assume that the block.LastCommit message contains votes of processes p2, p3 and p4. Median is then chosen the following way: the value 98 is counted 27 times, the value 1000 is counted 10 times and the value 500 is counted also 10 times. So the median value will be the value 98. No matter what set of messages with at least 2f+1 voting power we choose, the median value will always be between the values sent by correct processes.

We ensure Time Monotonicity and Time Validity properties by the following rules:

  • let rs denotes RoundState (consensus internal state) of some process. Then rs.ProposalBlock.Header.Time == median(rs.LastCommit) && rs.Proposal.Timestamp == rs.ProposalBlock.Header.Time.

  • Furthermore, when creating the vote message, the following rules for determining vote.Time field should hold:

    • if rs.LockedBlock is defined then vote.Time = max(rs.LockedBlock.Timestamp + time.Millisecond, time.Now()), where time.Now() denotes local Unix time in milliseconds

    • else if rs.Proposal is defined then vote.Time = max(rs.Proposal.Timestamp + time.Millisecond,, time.Now()),

    • otherwise, vote.Time = time.Now()). In this case vote is for nil so it is not taken into account for the timestamp of the next block.

Consensus Paper

See the original Tendermint paper, The latest gossip on BFT consensus, by Ethan Buchman, Jae Kwon and Zarko Milosevic.

For the current Tenderdash implementation, use the Tenderdash consensus protocol.

Evidence

Evidence is an important component of Tenderdash's security model. Whilst the core consensus protocol provides correctness guarantees for state machine replication that can tolerate less than 1/3 failures, the evidence system looks to detect and gossip byzantine faults whose combined power is greater than or equal to 1/3. It is worth noting that the evidence system is designed purely to detect possible attacks, gossip them, commit them on chain and inform the application running on top of Tenderdash. Evidence in itself does not punish "bad actors", this is left to the discretion of the application. A common form of punishment is slashing where the validators that were caught violating the protocol have all or a portion of their voting power removed. Evidence, given the assumption that 1/3+ of the network is still byzantine, is susceptible to censorship and should therefore be considered added security on a "best effort" basis.

This document walks through the various forms of evidence, how they are detected, gossiped, verified and committed.

NOTE: Evidence here is internal to Tenderdash and should not be confused with application evidence

Detection

Equivocation

Equivocation is the most fundamental of byzantine faults. Simply put, to prevent replication of state across all nodes, a validator tries to convince some subset of nodes to commit one block whilst convincing another subset to commit a different block. This is achieved by double voting (hence DuplicateVoteEvidence). A successful duplicate vote attack requires greater than 1/3 voting power and a (temporary) network partition between the aforementioned subsets. This is because in consensus, votes are gossiped around. When a node observes two conflicting votes from the same peer, it will use the two votes of evidence and begin gossiping this evidence to other nodes. Verification is addressed further down.

type DuplicateVoteEvidence struct {
    VoteA Vote
    VoteB Vote

    // and abci specific fields
}

Light Client Attacks

Light clients also comply with the 1/3+ security model, however, by using a different, more lightweight verification method they are subject to a different kind of 1/3+ attack whereby the byzantine validators could sign an alternative light block that the light client will think is valid. Detection, explained in greater detail here, involves comparison with multiple other nodes in the hope that at least one is "honest". An "honest" node will return a challenging light block for the light client to validate. If this challenging light block also meets the validation criteria then the light client sends the "forged" light block to the node. Verification is addressed further down.

type LightClientAttackEvidence struct {
    ConflictingBlock LightBlock
    CommonHeight int64

      // and abci specific fields
}

Verification

If a node receives evidence, it will first try to verify it, then persist it. Evidence of byzantine behavior should only be committed once (uniqueness) and should be committed within a certain period from the point that it occurred (timely). Timelines is defined by the EvidenceParams: MaxAgeNumBlocks and MaxAgeDuration. In Proof of Stake chains where validators are bonded, evidence age should be less than the unbonding period so validators still can be punished. Given these two propoerties the following initial checks are made.

  1. Has the evidence expired? This is done by taking the height of the Vote within DuplicateVoteEvidence or CommonHeight within LightClientAttakEvidence. The evidence height is then used to retrieve the header and thus the time of the block that corresponds to the evidence. If CurrentHeight - MaxAgeNumBlocks > EvidenceHeight && CurrentTime - MaxAgeDuration > EvidenceTime, the evidence is considered expired and ignored.

  2. Has the evidence already been committed? The evidence pool tracks the hash of all committed evidence and uses this to determine uniqueness. If a new evidence has the same hash as a committed one, the new evidence will be ignored.

DuplicateVoteEvidence

Valid DuplicateVoteEvidence must adhere to the following rules:

  • Validator Address, Height, Round and Type must be the same for both votes

  • BlockID must be different for both votes (BlockID can be for a nil block)

  • Validator must have been in the validator set at that height

  • Vote signature must be correctly signed. This also uses ChainID so we know that the fault occurred on this chain

LightClientAttackEvidence

Valid Light Client Attack Evidence must adhere to the following rules:

  • If the header of the light block is invalid, thus indicating a lunatic attack, the node must check that they can use verifySkipping from their header at the common height to the conflicting header

  • If the header is valid, then the validator sets are the same and this is either a form of equivocation or amnesia. We therefore check that 2/3 of the validator set also signed the conflicting header.

  • The nodes own header at the same height as the conflicting header must have a different hash to the conflicting header.

  • If the nodes latest header is less in height to the conflicting header, then the node must check that the conflicting block has a time that is less than this latest header (This is a forward lunatic attack).

Gossiping

If a node verifies evidence it then broadcasts it to all peers, continously sending the same evidence once every 10 seconds until the evidence is seen on chain or expires.

Commiting on Chain

Evidence takes strict priority over regular transactions, thus a block is filled with evidence first and transactions take up the remainder of the space. To mitigate the threat of an already punished node from spamming the network with more evidence, the size of the evidence in a block can be capped by EvidenceParams.MaxBytes. Nodes receiving blocks with evidence will validate the evidence before sending Prevote and Precommit votes. The evidence pool will usually cache verifications so that this process is much quicker.

Sending Evidence to the Application

After evidence is committed, the block is then processed by the block executor which delivers the evidence to the application via EndBlock. Evidence is stripped of the actual proof, split up per faulty validator and only the validator, height, time and evidence type is sent.

enum EvidenceType {
  UNKNOWN             = 0;
  DUPLICATE_VOTE      = 1;
  LIGHT_CLIENT_ATTACK = 2;
}

message Evidence {
  EvidenceType type = 1;
  // The offending validator
  Validator validator = 2 [(gogoproto.nullable) = false];
  // The height when the offense occurred
  int64 height = 3;
  // The corresponding time where the offense occurred
  google.protobuf.Timestamp time = 4 [
    (gogoproto.nullable) = false, (gogoproto.stdtime) = true];
  // Total voting power of the validator set in case the ABCI application does
  // not store historical validators.
  // https://github.com/tendermint/tendermint/issues/4581
  int64 total_voting_power = 5;
}

DuplicateVoteEvidence and LightClientAttackEvidence are self-contained in the sense that the evidence can be used to derive the abci.Evidence that is sent to the application. Because of this, extra fields are necessary:

type DuplicateVoteEvidence struct {
  VoteA *Vote
  VoteB *Vote

  // abci specific information
  TotalVotingPower int64
  ValidatorPower   int64
  Timestamp        time.Time
}

type LightClientAttackEvidence struct {
  ConflictingBlock *LightBlock
  CommonHeight     int64

  // abci specific information
  ByzantineValidators []*Validator
  TotalVotingPower    int64       
  Timestamp           time.Time 
}

These ABCI specific fields don't affect validity of the evidence itself but must be consistent amongst nodes and agreed upon on chain. If evidence with the incorrect abci information is sent, a node will create new evidence from it and replace the ABCI fields with the correct information.

Tenderdash Light Client Protocol

Deprecated, please see light-client.

Fork accountability

Deprecated, please see light-client/accountability.

Detection

Deprecated, please see light-client/detection.

Core Verification

Deprecated, please see light-client/accountability.

Proposer-Based Timestamps (PBTS)

For current Tenderdash behavior, see Proposal timestamps and time rules and the proposal validity rules.

Historical Tendermint PBTS research is retained where it has no complete replacement in the current implementation specification:

PBTS: Protocol Specification

Historical and outdated. This Tendermint research draft is retained for its algorithm derivation; it does not specify current Tenderdash behavior. Use Proposal timestamps and time rules and proposal validity for the implemented rules.

Proposal Time

PBTS computes for a proposed value v the proposal time v.time, with bounded difference to the actual real-time the proposed value was generated. The proposal time is read from the clock of the process that proposes a value for the first time, its original proposer.

With PBTS, therefore, we assume that processes have access to synchronized clocks. The proper definition of what it means can be found in the system model, but essentially we assume that two correct processes do not simultaneous read from their clocks time values that differ more than PRECISION, which is a system parameter.

Proposal times are definitive

When a value v is produced by a process, it also assigns the associated proposal time v.time. If the same value v is then re-proposed in a subsequent round of consensus, it retains its original time, assigned by its original proposer.

A value v should re-proposed when it becomes locked by the network, i.e., when it receives 2f + 1 PREVOTES in a round r of consensus. This means that processes with 2f + 1-equivalent voting power accepted, in round r, both v and its associated time v.time. Since the originally proposed value and its associated time were considered valid, there is no reason for reassigning v.time.

In the first version of this specification, proposals were defined as pairs (v, time). In addition, the same value v could be proposed, in different rounds, but would be associated to distinct times each time it was reproposed. Since this possibility does not exist in this second specification, the proposal time became part of the proposed value. With this simplification, several small changes to the arXiv algorithm are no longer required.

Time Monotonicity

Values decided in successive heights of consensus must have increasing times, so:

  • Monotonicity: for any process p and any two decided heights h and h', if h > h' then decision_p[h].time > decision_p[h'].time.

For ensuring time monotonicity, it is enough to ensure that a value v proposed by process p at height h_p has v.time > decision_p[h_p-1].time. So, if process p is the proposer of a round of height h_p and reads from its clock a time now_p <= decision_p[h_p-1], it should postpone the generation of its proposal until now_p > decision_p[h_p-1].

Although it should be considered, this scenario is unlikely during regular operation, as from decision_p[h_p-1].time and the start of height h_p, a complete consensus instance need to terminate.

Notice that monotonicity is not introduced by this proposal, being already ensured by BFTTime. In BFTTime, the Timestamp field of every Precommit message of height h_p sent by a correct process is required to be larger than decision_p[h_p-1].time, as one of such Timestamp fields becomes the time assigned to a value proposed at height h_p.

The time monotonicity of values proposed in heights of consensus is verified by the valid() predicate, to which every proposed value is submitted. A value rejected by the valid() implementation is not accepted by any correct process.

Timely Proposals

PBTS introduces a new requirement for a process to accept a proposal: the proposal must be timely. It is a temporal requirement, associated with the following synchrony (that is, timing) assumptions regarding the behavior of processes and the network:

  • Synchronized clocks: the values simultaneously read from clocks of any two correct processes differ by at most PRECISION;
  • Bounded transmission delays: the real time interval between the sending of a proposal at a correct process, and the reception of the proposal at any correct process is upper bounded by MSGDELAY.

[PBTS-RECEPTION-STEP.1]

Let now_p be the time, read from the clock of process p, at which p receives the proposed value v. The proposal is considered timely by p when:

  1. now_p >= v.time - PRECISION
  2. now_p <= v.time + MSGDELAY + PRECISION

The first condition derives from the fact that the generation and sending of v precedes its reception. The minimum receiving time now_p for v be considered timely by p is derived from the extreme scenario when the clock of p is PRECISION behind of the clock of the proposer of v, and the proposal's transmission delay is 0 (minimum).

The second condition derives from the assumption of an upper bound for the transmission delay of a proposal. The maximum receiving time now_p for v be considered timely by p is derived from the extreme scenario when the clock of p is PRECISION ahead of the clock of the proposer of v, and the proposal's transmission delay is MSGDELAY (maximum).

Updated Consensus Algorithm

The following changes are proposed for the algorithm in the arXiv paper.

New StartRound

There are two additions to the propose round step when executed by the proposer of a round:

  1. to ensure time monotonicity, the proposer does not propose a value until its current local time becomes greater than the previously decided value's time
  2. when the proposer produce a new proposal it sets the proposal's time to its current local time
    • no changes are made to the logic when a proposer has a non-nil validValue, which retains its original proposal time.

[PBTS-ALG-STARTROUND.1]

function StartRound(round) {
 round_p ← round
 step_p ← propose
 if proposer(h_p, round_p) = p {
  wait until now_p > decision_p[h_p-1].time // time monotonicity
  if validValue_p != nil {
   proposal ← validValue_p
  } else {
   proposal ← getValue()
   proposal.time ← now_p // proposal time
  }
   broadcast ⟨PROPOSAL, h_p, round_p, proposal, validRound_p⟩
 } else {
  schedule OnTimeoutPropose(h_p,round_p) to be executed after timeoutPropose(round_p)
 }
}

New Rule Replacing Lines 22 - 27

The rule on line 22 applies to values v proposed for the first time, i.e., for proposals not backed by 2f + 1 PREVOTEs for v in a previous round. The PROPOSAL message, in this case, carry -1 in its validRound field.

The new rule for issuing a PREVOTE for a proposed value v requires the value to be timely. As the timely predicate is evaluated in the moment that the value is received, as part of a PROPOSAL message, we require the PROPOSAL message to be timely.

[PBTS-ALG-UPON-PROP.1]

upon timely(⟨PROPOSAL, h_p, round_p, v, −1⟩) from proposer(h_p, round_p) while step_p = propose do {
  if valid(v) ∧ (lockedRound_p = −1 ∨ lockedValue_p = v) {
    broadcast ⟨PREVOTE, h_p, round_p, id(v)⟩ 
  }
  else {
    broadcast ⟨PREVOTE, h_p, round_p, nil⟩ 
  }
  step_p ← prevote
}

Rules at Lines 28 - 33 remain unchanged

The rule on line 28 applies to values v proposed again in the current round because its proposer received 2f + 1 PREVOTEs for v in a previous round vr. This means that there was a round r <= vr in which 2f + 1 processes accepted v for the first time, and so sent PREVOTEs for v. Which, in turn, means that these processes executed the line 22 of the algorithm, and therefore judged v as a timely proposal.

In other words, we don't need to verify whether v is a timely proposal because at least f + 1 processes judged v as timely in a previous round, and because, since v was re-proposed as a validValue (line 16), v.time has not being updated from its original proposal.

All other rules remains unchanged.

Back to main document.

PBTS: System Model and Properties

Historical and outdated. This Tendermint research draft is retained for its clock model and temporal analysis, which have no complete replacement in the implementation specification. Its guarantees do not describe current Tenderdash. Use the consensus system model and proposal timestamps and time rules for the implemented behavior.

Outline

System Model

[PBTS-CLOCK-NEWTON.0]

There is a reference Newtonian real-time t.

No process has direct access to this reference time, used only for specification purposes. The reference real-time is assumed to be aligned with the Coordinated Universal Time (UTC).

Synchronized clocks

Processes are assumed to be equipped with synchronized clocks, aligned with the Coordinated Universal Time (UTC).

This requires processes to periodically synchronize their local clocks with an external and trusted source of the time (e.g. NTP servers). Each synchronization cycle aligns the process local clock with the external source of time, making it a fairly accurate source of real time. The periodic (re)synchronization aims to correct the drift of local clocks, which tend to pace slightly faster or slower than the real time.

To avoid an excessive level detail in the parameters and guarantees of synchronized clocks, we adopt a single system parameter PRECISION to encapsulate the potential inaccuracy of the synchronization mechanisms, and drifts of local clocks from real time.

[PBTS-CLOCK-PRECISION.0]

There exists a system parameter PRECISION, such that for any two processes p and q, with local clocks C_p and C_q:

  • If p and q are equipped with synchronized clocks, then for any real-time t we have |C_p(t) - C_q(t)| <= PRECISION.

PRECISION thus bounds the difference on the times simultaneously read by processes from their local clocks, so that their clocks can be considered synchronized.

Accuracy

A second relevant clock parameter is accuracy, which binds the values read by processes from their clocks to real time.

[PBTS-CLOCK-ACCURACY.0]

For the sake of completeness, we define a parameter ACCURACY such that:

  • At real time t there is at least one correct process p which clock marks C_p(t) with |C_p(t) - t| <= ACCURACY.

As a consequence, applying the definition of PRECISION, we have:

  • At real time t the synchronized clock of any correct process p marks C_p(t) with |C_p(t) - t| <= ACCURACY + PRECISION.

The reason for not adopting ACCURACY as a system parameter is the assumption that PRECISION >> ACCURACY. This allows us to consider, for practical purposes, that the PRECISION system parameter embodies the ACCURACY model parameter.

Message Delays

The assumption that processes have access to synchronized clocks ensures that proposal times assigned by correct processes have a bounded relation with the real time. It is not enough, however, to identify (and reject) proposal times proposed by Byzantine processes.

To properly evaluate whether the time assigned to a proposal is consistent with the real time, we need some information regarding the time it takes for a message carrying a proposal to reach all its (correct) destinations. More precisely, the maximum delay for delivering a proposal to its destinations allows defining a lower bound, a minimum time that a correct process assigns to proposal. While minimum delay for delivering a proposal to a destination allows defining an upper bound, the maximum time assigned to a proposal.

[PBTS-MSG-DELAY.0]

There exists a system parameter MSGDELAY for end-to-end delays of proposal messages, such for any two correct processes p and q:

  • If p sends a proposal message m at real time t and q receives m at real time t', then t <= t' <= t + MSGDELAY.

Notice that, as a system parameter, MSGDELAY should be observed for any proposal message broadcast by correct processes: it is a worst-case parameter. As message delays depends on the message size, the above requirement implicitly indicates that the size of proposal messages is either fixed or upper bounded.

Problem Statement

In this section we define the properties of Tendermint consensus (cf. the arXiv paper) in this system model.

[PBTS-PROPOSE.0]

A proposer proposes a consensus value v that includes a proposal time v.time.

We then restrict the allowed decisions along the following lines:

[PBTS-INV-AGREEMENT.0]

  • [Agreement] No two correct processes decide on different values v.

This implies that no two correct processes decide on different proposal times v.time.

[PBTS-INV-VALID.0]

  • [Validity] If a correct process decides on value v, then v satisfies a predefined valid predicate.

With respect to PBTS, the valid predicate requires proposal times to be monotonic over heights of consensus:

[PBTS-INV-MONOTONICITY.0]
  • If a correct process decides on value v at the height h of consensus, thus setting decision[h] = v, then v.time > decision[h'].time for all previous heights h' < h.

The monotonicity of proposal times, and external validity in general, implicitly assumes that heights of consensus are executed in order.

[PBTS-INV-TIMELY.0]

  • [Time-Validity] If a correct process decides on value v, then the proposal time v.time was considered timely by at least one correct process.

PBTS introduces a timely predicate that restricts the allowed decisions based on the proposal time v.time associated with a proposed value v. As a synchronous predicate, the time at which it is evaluated impacts on whether a process accepts or reject a proposal time. For this reason, the Time-Validity property refers to the previous evaluation of the timely predicate, detailed in the following section.

Protocol Analysis - Timely proposals

For PBTS, a proposal is a tuple (v, v.time, v.round), where:

  • v is the proposed value;
  • v.time is the associated proposal time;
  • v.round is the round at which v was first proposed.

We include the proposal round v.round in the proposal definition because a value v and its associated proposal time v.time can be proposed in multiple rounds, but the evaluation of the timely predicate is only relevant at round v.round.

Considering the algorithm in the arXiv paper, a new proposal is produced by the getValue() method, invoked by the proposer p of round round_p when starting its proposing round with a nil validValue_p. The first round at which a value v is proposed is then the round at which the proposal for v was produced, and broadcast in a PROPOSAL message with vr = -1.

[PBTS-PROPOSAL-RECEPTION.0]

The timely predicate is evaluated when a process receives a proposal. More precisely, let p be a correct process:

  • proposalReceptionTime(p,r) is the time p reads from its local clock when p is at round r and receives the proposal of round r.

[PBTS-TIMELY.0]

The proposal (v, v.time, v.round) is considered timely by a correct process p if:

  1. proposalReceptionTime(p,v.round) is set, and
  2. proposalReceptionTime(p,v.round) >= v.time - PRECISION, and
  3. proposalReceptionTime(p,v.round) <= v.time + MSGDELAY + PRECISION.

A correct process at round v.round only sends a PREVOTE for v if the associated proposal time v.time is considered timely.

Considering the algorithm in the arXiv paper, the timely predicate is evaluated by a process p when it receives a valid PROPOSAL message from the proposer of the current round round_p with vr = -1.

Timely Proof-of-Locks

A Proof-of-Lock is a set of PREVOTE messages of round of consensus for the same value from processes whose cumulative voting power is at least 2f + 1. We denote as POL(v,r) a proof-of-lock of value v at round r.

For PBTS, we are particularly interested in the POL(v,v.round) produced in the round v.round at which a value v was first proposed. We call it a timely proof-of-lock for v because it can only be observed if at least one correct process considered it timely:

[PBTS-TIMELY-POL.0]

If

  • there is a valid POL(v,r) with r = v.round, and
  • POL(v,v.round) contains a PREVOTE message from at least one correct process,

Then, let p is a such correct process:

  • p received a PROPOSAL message of round v.round, and
  • the PROPOSAL message contained a proposal (v, v.time, v.round), and
  • p was in round v.round and evaluated the proposal time v.time as timely.

The existence of a such correct process p is guaranteed provided that the voting power of Byzantine processes is bounded by 2f.

Derived Proof-of-Locks

The existence of POL(v,r) is a requirement for the decision of v at round r of consensus.

At the same time, the Time-Validity property establishes that if v is decided then a timely proof-of-lock POL(v,v.round) must have been produced.

So, we need to demonstrate here that any valid POL(v,r) is either a timely proof-of-lock or it is derived from a timely proof-of-lock:

[PBTS-DERIVED-POL.0]

If

  • there is a valid POL(v,r), and
  • POL(v,r) contains a PREVOTE message from at least one correct process,

Then

  • there is a valid POL(v,v.round) with v.round <= r which is a timely proof-of-lock.

The above relation is trivially observed when r = v.round, as POL(v,r) must be a timely proof-of-lock. Notice that we cannot have r < v.round, as v.round is defined as the first round at which v was proposed.

For r > v.round we need to demonstrate that if there is a valid POL(v,r), then a timely POL(v,v.round) was previously obtained. We observe that a condition for observing a POL(v,r) is that the proposer of round r has broadcast a PROPOSAL message for v. As r > v.round, we can affirm that v was not produced in round r. Instead, by the protocol operation, v was a valid value for the proposer of round r, which means that if the proposer has observed a POL(v,vr) with vr < r. The above operation considers a correct proposer, but since a POL(v,r) was produced (by hypothesis) we can affirm that at least one correct process (also) observed a POL(v,vr).

Considering the algorithm in the arXiv paper, v was proposed by the proposer p of round round_p because its validValue_p variable was set to v. The PROPOSAL message broadcast by the proposer, in this case, had vr > -1, and it could only be accepted by processes that also observed a POL(v,vr).

Thus, if there is a POL(v,r) with r > v.round, then there is a valid POL(v,vr) with v.round <= vr < r. If vr = v.round then POL(vr,v) is a timely proof-of-lock and we are done. Otherwise, there is another valid POL(v,vr') with v.round <= vr' < vr, and the above reasoning can be recursively applied until we get vr' = v.round and observe a timely proof-of-lock.

Temporal analysis

In this section we present invariants that need be observed for ensuring that PBTS is both safe and live.

In addition to the variables and system parameters already defined, we use beginRound(p,r) as the value of process p's local clock when it starts round r of consensus.

Safety

The safety of PBTS requires that if a value v is decided, then at least one correct process p considered the associated proposal time v.time timely. Following the definition of timely proposals and proof-of-locks, we require this condition to be asserted at a specific round of consensus, defined as v.round:

[PBTS-SAFETY.0]

If

  • there is a valid commit C for a value v
  • C contains a PRECOMMIT message from at least one correct process

then there is a correct process p (not necessarily the same above considered) such that:

  • beginRound(p,v.round) <= proposalReceptionTime(p,v.round) <= beginRound(p,v.round+1) and
  • proposalReceptionTime (p,v.round) - MSGDELAY - PRECISION <= v.time <= proposalReceptionTime(p,v.round) + PRECISION

That is, a correct process p started round v.round and, while still at round v.round, received a PROPOSAL message from round v.round proposing v. Moreover, the reception time of the original proposal for v, according with p's local clock, enabled p to consider the proposal time v.time as timely. This is the requirement established by PBTS for issuing a PREVOTE for the proposal (v, v.time, v.round), so for the eventual decision of v.

Liveness

The liveness of PBTS relies on correct processes accepting proposal times assigned by correct proposers. We thus present a set of conditions for assigning a proposal time v.time so that every correct process should be able to issue a PREVOTE for v.

[PBTS-LIVENESS.0]

If

  • the proposer of a round r of consensus is correct
  • and it proposes a value v for the first time, with associated proposal time v.time

then the proposal (v, v.time, r) is accepted by every correct process provided that:

  • min{p is correct : beginRound(p,r)} <= v.time <= max{p is correct : beginRound(p,r)} and
  • max{p is correct : beginRound(p,r)} <= v.time + MSGDELAY + PRECISION <= min{p is correct : beginRound(p,r+1)}

The first condition establishes a range of safe proposal times v.time for round r. This condition is trivially observed if a correct proposer p sets v.time to the time it reads from its clock when starting round r and proposing v. A PROPOSAL message sent by p at local time v.time should not be received by any correct process before its local clock reads v.time - PRECISION, so that condition 2 of [PBTS-TIMELY.0] is observed.

The second condition establishes that every correct process should start round v.round at a local time that allows v.time to still be considered timely, according to condition 3. of [PBTS-TIMELY.0]. In addition, it requires correct processes to stay long enough in round v.round so that they can receive the PROPOSAL message of round v.round. It assumed here that the proposer of v broadcasts a PROPOSAL message at time v.time, according to its local clock, so that every correct process should receive this message by time v.time + MSGDELAY + PRECISION, according to their local clocks.

Back to main document.

PBTS: Protocol Specification (first draft)

This historical Tendermint draft is superseded by the newer PBTS protocol draft. For current Tenderdash behavior, see Proposal timestamps and time rules and proposal validity.

PBTS: System Model and Properties (first draft)

This historical Tendermint draft is superseded by the newer PBTS system model and properties. For the current Tenderdash specification, see the consensus system model and proposal timestamps and time rules.

Proposer-Based Time (first draft)

Historical and outdated. This Tendermint draft is retained for its original comparison with BFT Time, including aggregate-signature tradeoffs, which the newer protocol draft and newer system model do not reproduce. Its proposed algorithm is superseded by those drafts. For current Tenderdash behavior, use Proposal timestamps and time rules and proposal validity.

Current BFTTime

Description

In Tendermint consensus, the first version of how time is computed and stored in a block works as follows:

  • validators send their current local time as part of precommit messages
  • upon collecting the precommit messages that the proposer uses to build a commit to be put in the next block, the proposer computes the time of the next block as the median (weighted over voting power) of the times in the precommit messages.

Analysis

  1. Fault tolerance. The computed median time is called bfttime as it is indeed fault-tolerant: if less than a third of the validators is faulty (counted in voting power), it is guaranteed that the computed time lies between the minimum and the maximum times sent by correct validators.
  2. Effect of faulty validators. If more than 1/2 of the voting power (which is in fact more than one third and less than two thirds of the voting power) is held by faulty validators, then the time is under total control of the faulty validators. (This is particularly challenging in the context of lightclient security.)
  3. Proposer influence on block time. The proposer of the next block has a degree of freedom in choosing the bfttime, since it computes the median time based on the timestamps from precommit messages sent by 2f + 1 correct validators.
    1. If there are n different timestamps in the precommit messages, the proposer can use any subset of timestamps that add up to 2f + 1 of the voting power in order to compute the median.
    2. If the validators decide in different rounds, the proposer can decide on which round the median computation is based.
  4. Liveness. The liveness of the protocol:
    1. does not depend on clock synchronization,
    2. depends on bounded message delays.
  5. Relation to real time. There is no clock synchronizaton, which implies that there is no relation between the computed block time and real time.
  6. Aggregate signatures. As the precommit messages contain the local times, all these precommit messages typically differ in the time field, which prevents the use of aggregate signatures.

Suggested Proposer-Based Time

Outline

An alternative approach to time has been discussed: Rather than having the validators send the time in the precommit messages, the proposer in the consensus algorithm sends its time in the propose message, and the validators locally check whether the time is OK (by comparing to their local clock).

This proposed solution adds the requirement of having synchronized clocks, and other implicit assumptions.

Comparison of the Suggested Method to the Old One

  1. Fault tolerance. Maintained in the suggested protocol.
  2. Effect of faulty validators. Eliminated in the suggested protocol, that is, the block time can be corrupted only in the extreme case when >2/3 of the validators are faulty.
  3. Proposer influence on block time. The proposer of the next block has less freedom when choosing the block time.
    1. This scenario is eliminated in the suggested protocol, provided that there are <1/3 faulty validators.
    2. This scenario is still there.
  4. Liveness. The liveness of the suggested protocol:
    1. depends on the introduced assumptions on synchronized clocks (see below),
    2. still depends on the message delays (unavoidable).
  5. Relation to real time. We formalize clock synchronization, and obtain a well-defined relation between the block time and real time.
  6. Aggregate signatures. The precommit messages free of time, which allows for aggregate signatures.

Protocol Overview

Proposed Time

We assume that the field proposal in the PROPOSE message is a pair (v, time), of the proposed consensus value v and the proposed time time.

Reception Step

In the reception step at node p at local time now_p, upon receiving a message m:

  • if the message m is of type PROPOSE and satisfies now_p - PRECISION < m.time < now_p + PRECISION + MSGDELAY, then mark the message as timely.
    (PRECISION and MSGDELAY being system parameters, see below)

after the presentation in the dev session, we realized that different semantics for the reception step is closer aligned to the implementation. Instead of dropping propose messages, we keep all of them, and mark timely ones.

Processing Step

  • Start round
arXiv paper Proposer-based time
function StartRound(round) {
 round_p ← round
 step_p ← propose
 if proposer(h_p, round_p) = p {

 
  if validValue_p != nil {

   proposal ← validValue_p
  } else {

   proposal ← getValue()
  }
   broadcast ⟨PROPOSAL, h_p, round_p, proposal, validRound_p⟩
 } else {
  schedule OnTimeoutPropose(h_p,round_p) to 
   be executed after timeoutPropose(round_p)
 }
}
function StartRound(round) {
 round_p ← round
 step_p ← propose
 if proposer(h_p, round_p) = p {
  // new wait condition
  wait until now_p > block time of block h_p - 1
  if validValue_p != nil {
   // add "now_p"
   proposal ← (validValue_p, now_p) 
  } else {
   // add "now_p"
   proposal ← (getValue(), now_p) 
  }
  broadcast ⟨PROPOSAL, h_p, round_p, proposal, validRound_p⟩
 } else {
  schedule OnTimeoutPropose(h_p,round_p) to 
   be executed after timeoutPropose(round_p)
 }
}
  • Rule on lines 28-35
arXiv paper Proposer-based time
upon timely(⟨PROPOSAL, h_p, round_p, v, vr⟩) 
 from proposer(h_p, round_p)
 AND 2f + 1 ⟨PREVOTE, h_p, vr, id(v)⟩ 
while step_p = propose ∧ (vr ≥ 0 ∧ vr < round_p) do {
 if valid(v) ∧ (lockedRound_p ≤ vr ∨ lockedValue_p = v) {
  
  broadcast ⟨PREVOTE, h_p, round_p, id(v)⟩
 } else {
  broadcast ⟨PREVOTE, hp, round_p, nil⟩
 }
}
upon timely(⟨PROPOSAL, h_p, round_p, (v, tprop), vr⟩) 
 from proposer(h_p, round_p) 
 AND 2f + 1 ⟨PREVOTE, h_p, vr, id(v, tvote)⟩ 
 while step_p = propose ∧ (vr ≥ 0 ∧ vr < round_p) do {
  if valid(v) ∧ (lockedRound_p ≤ vr ∨ lockedValue_p = v) {
   // send hash of v and tprop in PREVOTE message
   broadcast ⟨PREVOTE, h_p, round_p, id(v, tprop)⟩
  } else {
   broadcast ⟨PREVOTE, hp, round_p, nil⟩
  }
 }
  • Rule on lines 49-54
arXiv paper Proposer-based time
upon ⟨PROPOSAL, h_p, r, v, ∗⟩ from proposer(h_p, r) 
 AND 2f + 1 ⟨PRECOMMIT, h_p, r, id(v)⟩ 
 while decisionp[h_p] = nil do {
  if valid(v) {

   decision_p [h_p] = v
   h_p ← h_p + 1
   reset lockedRound_p , lockedValue_p, validRound_p and 
    validValue_p to initial values and empty message log 
   StartRound(0)
  }
 }
upon ⟨PROPOSAL, h_p, r, (v,t), ∗⟩ from proposer(h_p, r) 
 AND 2f + 1 ⟨PRECOMMIT, h_p, r, id(v,t)⟩
 while decisionp[h_p] = nil do {
  if valid(v) {
   // decide on time too
   decision_p [h_p] = (v,t) 
   h_p ← h_p + 1
   reset lockedRound_p , lockedValue_p, validRound_p and 
    validValue_p to initial values and empty message log 
   StartRound(0)
  }
 }
  • Other rules are extended in a similar way, or remain unchanged

Property Overview

Safety and Liveness

For safety (Point 1, Point 2, Point 3i) and liveness (Point 4) we need the following assumptions:

  • There exists a system parameter PRECISION such that for any two correct validators V and W, and at any real-time t, their local times C_V(t) and C_W(t) differ by less than PRECISION time units, i.e., |C_V(t) - C_W(t)| < PRECISION
  • The message end-to-end delay between a correct proposer and a correct validator (for PROPOSE messages) is less than MSGDELAY.

Relation to Real-Time

For analyzing real-time safety (Point 5), we use a system parameter ACCURACY, such that for all real-times t and all correct validators V, we have | C_V(t) - t | < ACCURACY.

ACCURACY is not necessarily visible at the code level. We might even view ACCURACY as variable over time. The smaller it is during a consensus instance, the closer the block time will be to real-time.

Note that PRECISION and MSGDELAY show up in the code.

Detailed Specification

This specification describes the changes needed to be done to the Tendermint consensus algorithm as described in the arXiv paper and the simplified specification in TLA+, and makes precise the underlying assumptions and the required properties.

Proposer Selection Procedure

This document specifies how Tenderdash chooses the proposer of a round, as implemented in internal/consensus/versioned/selectproposer. Tenderdash does not use Tendermint's priority-queue (weighted round-robin) algorithm: all validators of a Dash quorum have equal voting power, and the proposer is chosen by rotating an index over the validator set.

Requirements

  • R1 Determinism. For a validator set V, every honest validator computes the same proposer(h, r).
  • R2 Fairness. Within a stable validator set, the rounds of one height rotate through V in order, so no validator proposes twice in a height before every other validator has proposed once. Across heights the guarantee is narrower: in v1 the rotation continues from one height to the next, so any |V| consecutive elections are fair as long as every block is committed in the round it was first proposed; in v0 the same condition makes only the round-0 proposers rotate by one per height, and other elections can repeat a validator early. With V = A,B,C,D, a height lost at round 0 by A and committed at round 1 with a fresh block by B yields the election sequence A, B, B, C, D; committed at round 2 with a fresh block by C, it yields A, B, C, B. A re-proposal also breaks the round-0 rotation: if B re-proposes A's round-0 block and it is committed at round 1, the next height starts with A again.

Both follow from the construction below.

Validator order

ValidatorSet.Validators is sorted by ProTxHash ascending (ValidatorsByProTxHashes, applied in NewValidatorSet and updateWithChangeSet, types/validator_set.go). The set keeps a proposerIndex; ValidatorSet.Proposer() returns Validators[proposerIndex], SetProposer(proTxHash) sets the index to that validator, and IncProposerIndex(n) adds n (possibly negative) modulo |V|.

Two versions

selectproposer.NewProposerSelector (proposer_selector.go) chooses the strategy by ConsensusParams.Version.ConsensusVersion:

ConsensusVersionStrategyFile
0 (default)Height-based: the index advances once per height; the round offset is derived without being persistedheight_proposer.go (heightProposerSelector)
1, 2Height-and-round-based: the index advances once per height and once per round. Version 2 keeps this strategy and adds proposer livenessheight_round_proposer.go (heightRoundProposerSelector)

Changing the version is a consensus-parameter update and takes effect at the next height (Validator-set and parameter changes).

When the index is computed

StateData.updateToState creates a new selector at every height with NewProposerSelector(params, Validators, h, 0, blockStore). Whenever the block store is non-empty and contains h − 1 or h, the constructor derives the index from the store (proposerFromStore, below); it trusts the validator set's own proposerIndex only when there is no usable store (e.g. genesis). During the height, StateData.updateRoundStep calls ProposerSelector.UpdateHeightRound(h, r) on every step change and GetProposer(h, r) answers from that state.

Because every node derives the index from the same stored block at the start of every height, determinism (R1) does not depend on nodes having tracked the same rounds.

proposerFromStore(h, r)

Let V be the validator set for height h, sorted by ProTxHash, and meta(x) the stored block metadata for height x (Header.ProposerProTxHash is the proposer of the round in which the block was first proposed — a re-proposed block keeps its header — and meta.Round is the round in which it was committed).

Casev0 (height_proposer.go)v1 (height_round_proposer.go)
h ∈ {0, 1}V[0]V[0]
meta(h) stored, meta(h).ValidatorsHash == hash(V)idx(meta(h).Proposer) − meta(h).Roundidx(meta(h).Proposer) + r − meta(h).Round
meta(h) stored, hash differserrorerror
only meta(h − 1) stored, meta(h − 1).ValidatorsHash == hash(V)idx(meta(h − 1).Proposer) + 1 − meta(h − 1).Roundidx(meta(h − 1).Proposer) + 1 + r
only meta(h − 1) stored, hash differs (quorum rotation)1 ("index 1 due to a bug in the original code that causes the first validator to never propose; preserved to avoid breaking consensus")0 + r
neither storederrorerror

All indices are modulo |V|. For the height in progress only the meta(h − 1) rows apply.

Within a height

  • v0 (heightProposerSelector.GetProposer): proposer(h, r) = V[(idx₀ + r) mod |V|] where idx₀ is the stored round-0 index; the per-round advance is applied to a copy and never persisted. UpdateHeightRound ignores the round.
  • v1 (heightRoundProposerSelector.updateScores): at the same height IncProposerIndex(r' − r); so proposer(h, r) = V[(idx₀ + r) mod |V|] as well, with the difference that the round offset is persisted in the selector and, in the meta(h) row above, the commit round is subtracted.

Both versions therefore rotate one validator per round. They differ in how the round-0 index of height h relates to height h − 1:

  • v0: idx₀(h) = idx(meta(h − 1).Proposer) − meta(h − 1).Round + 1 — the first proposer of the previous block, rewound by the previous commit round, plus one. When the previous block was first proposed at round p and committed at round c > p (a re-proposal), this is p − c + 1 positions after the previous round-0 proposer rather than one.
  • v1: idx₀(h) = idx(meta(h − 1).Proposer) + 1 — the validator after the first proposer of the previous block, independent of the commit round.

Validator set changes

A new validator set (new QuorumHash) replaces the old one at the next height; meta(h − 1).ValidatorsHash then differs from the new set's hash and the quorum rotation row applies. Within one quorum, additions and removals are applied as a change set and the set is re-sorted by ProTxHash (ValidatorSet.updateWithChangeSet); since the index is recomputed from the stored proposer's position in the new set at the next height, the rotation continues from that validator.

Use by consensus

  • EnterProposeAction.Execute decides whether to propose by comparing the node's pro-tx-hash with GetProposer(h, r).
  • Proposaler.verifyProposal verifies a proposal's signature against GetProposer(h, r)'s key only.
  • StateData.isValidForPrevote requires Header.ProposerProTxHash to equal GetProposer(h, r) for proposals with POLRound = -1.

Consensus

Specification of the Tenderdash consensus protocol, derived from the implementation in internal/consensus.

Contents

The gossip of consensus messages between peers is summarised in Commit handling and Non-validators; the wire format is in Consensus messages.

Validator Signing

Here we specify the rules for validating a proposal and vote before signing. When a validator signs each message, and what it signs, is specified by the consensus protocol. First we include some general notes on validating data structures common to both types. We then provide specific validation rules for each. Finally, we include validation rules to prevent double-signing.

SignedMsgType

The SignedMsgType is a single byte that refers to the type of the message being signed. It is defined in Go as follows:

// SignedMsgType is a type of signed message in the consensus.
type SignedMsgType byte

const (
 // Votes
 PrevoteType   SignedMsgType = 0x01
 PrecommitType SignedMsgType = 0x02

 // Proposals
 ProposalType SignedMsgType = 0x20

 // Proposer liveness, from CONSENSUS_VERSION_2
 ProposerHeartbeatType SignedMsgType = 0x30
 RoundSkipType         SignedMsgType = 0x31
)

All signed messages must correspond to one of these types.

Timestamp

Votes carry no timestamp. A proposal's Timestamp must equal the proposed block's Header.Time, which the proposer takes from its local clock (the genesis time for the initial block). Fresh proposals must be timely with respect to their receive time and the chain's SynchronyParams before a validator prevotes their block. The timeliness check is skipped for a locally recorded valid block and during replay. The rules are specified in Proposal timestamps and time rules and the prevote decision in Transitions and decision rules.

ChainID

ChainID is an unstructured string with a max length of 50-bytes. In the future, the ChainID may become structured, and may take on longer lengths. For now, it is recommended that signers be configured for a particular ChainID, and to only sign votes and proposals corresponding to that ChainID.

BlockID

BlockID contains the block-header hash, PartSetHeader (part count and Merkle root of serialized block parts), and the state-ID hash. A nil vote has all three empty. A complete ID has 32-byte block, part-set and state hashes and a positive part count. Proposals require a complete ID; votes allow nil or complete IDs. See BlockID and the StateID schema below.

Proposals

The structure of a proposal for signing looks like:

type CanonicalProposal struct {
 Type      SignedMsgType // type alias for byte
 Height    int64         `binary:"fixed64"`
 Round     int64         `binary:"fixed64"`
 POLRound  int64         `binary:"fixed64"`
 BlockID   CanonicalBlockID
 Timestamp time.Time
 ChainID   string
}

CanonicalBlockID contains the block hash and canonical part-set header; it excludes BlockID.StateID. The proposal signature also excludes CoreChainLockedHeight. The exact signature-covered fields and replacement of unsigned fields from assembled block bytes are specified in Creating the proposal and Receiving the proposal and block.

The basic structural checks on the proposal before canonicalization include:

p.Type == 0x20
p.Height > 0
p.Round >= 0
p.POLRound >= -1
p.BlockID.IsComplete()

In other words, a proposal is valid for signing if it contains the type of a Proposal (0x20), has a positive, non-zero height, a non-negative round, a POLRound not less than -1, and a complete BlockID.

Votes

Sign bytes for votes are defined as CanonicalVote struct.

The canonical message is encoded using Protobuf. Its checksum is one input to the quorum signing digest, not the complete digest verified by BLS; see Block signature verification on light client. The schema is:

message CanonicalVote {
  SignedMsgType type   = 1;  // type alias for byte
  sfixed64      height = 2;
  sfixed64      round  = 3;
  // block_id is a checksum (sha256) of CanonicalBlockID for the block being voted on
  bytes block_id = 4 [(gogoproto.customname) = "BlockID"];
  // state_id is a checksum  (sha256) of StateID for the block being voted on
  bytes  state_id = 5 [(gogoproto.customname) = "StateID"];
  string chain_id = 99 [(gogoproto.customname) = "ChainID"];
}

block_id is a sha256 checksum of Protobuf-encoded CanonicalBlockID message:

message CanonicalBlockID {
  bytes                  hash            = 1;
  CanonicalPartSetHeader part_set_header = 2 [(gogoproto.nullable) = false];
}

message CanonicalPartSetHeader {
  uint32 total = 1;
  bytes  hash  = 2;
}

state_id is a sha256 checksum of Protobuf-encoded StateID message:

message StateID {
  // AppVersion used when generating the block, equals to Header.Version.App.
  fixed64 app_version = 1 [(gogoproto.customname) = "AppVersion"];
  // Height of block containing this state ID.
  fixed64 height = 2;
  // AppHash used in current block, equal to Header.AppHash. 32 bytes.
  bytes app_hash = 3 [(gogoproto.customname) = "AppHash"];
  // CoreChainLockedHeight for the block, equal to Header.CoreChainLockedHeight.
  fixed32 core_chain_locked_height = 4 [(gogoproto.customname) = "CoreChainLockedHeight"];
  // Time of the block.
  google.protobuf.Timestamp time = 5 [(gogoproto.nullable) = false];
}

A vote is valid if each of the following lines evaluates to true for vote v:

v.Type == 0x1 || v.Type == 0x2
v.Height > 0
v.Round >= 0
v.BlockID.IsNil() || v.BlockID.IsComplete()

In other words, a vote is valid for signing if it contains the type of a Prevote or Precommit (0x1 or 0x2, respectively), has a positive, non-zero height, a non-negative round, and an empty or valid BlockID.

Block signature verification on light client

The block signature in a commit is threshold-recovered from precommit shares and verified with the quorum's ThresholdPublicKey. Its signing context includes ChainID, the commit height and round, full BlockID, QuorumType, and QuorumHash.

Use MakeBlockSignItem or VoteBlockSignID in types/quorum_sign_data.go and types/vote.go to construct the digest:

  1. Build the precommit's canonical vote, including the canonical-block-ID checksum and state-ID hash.
  2. MakeBlockSignItem hashes the canonical vote sign bytes and derives BlockRequestID(height, round) using the dpbvote domain.
  3. SignItem.UpdateSignHash uses the quorum type, quorum hash, request ID and message hash to construct the Dash BLS signing hash, with the helper's byte-order conversions. Verify the signature against that digest and the threshold key.

Proposal signing uses ProposalBlockSignID and its own dpproposal request-ID domain, likewise bound to height, round and quorum context (types/proposal.go). Vote extensions have separate sign items; application-supplied raw-extension request IDs need not bind height and round. See Vote extensions.

This describes digest construction, not a complete light-client trust algorithm. Full-node commit acceptance also verifies every extension threshold signature, checks the held block's complete ID, and requires the application's verdict on the ordered commit extension vector; see Commit verification and Commit validity depends on the application. A valid block signature does not authenticate that vector's count and order.

Proposer heartbeats and round skip votes

From CONSENSUS_VERSION_2, the proposer of a round that waits for transactions signs a heartbeat every proposer_heartbeat, and a validator that hears none for the round's silence timeout signs a vote to skip the round. Both are signed like votes, with the validator's quorum key share over the DIP-7 sign hash of the quorum type, the quorum hash, a request ID, and the SHA256 checksum of the length-delimited, Protobuf-encoded sign bytes below.

A heartbeat's sign bytes are:

message CanonicalProposerHeartbeat {
  SignedMsgType             type      = 1;  // 0x30
  sfixed64                  height    = 2;
  sfixed64                  round     = 3;
  fixed64                   sequence  = 4;
  google.protobuf.Timestamp timestamp = 5;
  string                    chain_id  = 6;
}

Its request ID is the SHA256 checksum of "dphb", then the height as a little-endian uint64, the round as a little-endian uint32, and the sequence as a little-endian uint64. The sequence grows with every heartbeat of a round, so each heartbeat a proposer sends is a signing request of its own. A proposer that restarts with its clock gone back can repeat an ID for a heartbeat with another timestamp; the timestamp is in the sign bytes, so the signature still differs.

A round skip vote's sign bytes are the same for every validator voting to skip a round, so the votes of +2/3 recover into one threshold signature, the round skip certificate:

message CanonicalRoundSkip {
  SignedMsgType type     = 1;  // 0x31
  sfixed64      height   = 2;
  sfixed64      round    = 3;
  string        chain_id = 4;
}

Its request ID is the SHA256 checksum of "dpskip", then the height as a little-endian uint64 and the round as a little-endian uint32.

A heartbeat or skip vote is valid for signing if its height is positive and its round is not negative. Neither changes consensus state or conflicts with another message, so neither is subject to the double-signing rules below.

Invalid Votes and Proposals

Basic structure, signature verification and consensus admissibility are separate checks. Proposal verification failures do not automatically evict their sender; received commits evict only on ErrInvalidCommitSignature, and replay never evicts. Budget exhaustion is a local processing limit, not proof of peer fault. The exact policies are in Proposal acceptance, Vote admission, and Commit verification.

Double Signing

Signers must be careful not to sign conflicting messages, also known as "double signing" or "equivocating". Tenderdash has mechanisms to publish evidence of validators that signed conflicting votes, so they can be punished by the application. Note Tenderdash does not currently handle evidence of conflicting proposals, though it may in the future.

State

FilePV persists height, round, internal signing step, block sign bytes and block signature (FilePVLastSignState in privval/file.go). Its ordered steps are proposal, prevote and precommit; these are not the numeric SignedMsgType values. checkHRS rejects regressions and permits a matching height/round/step only for comparison with the persisted sign bytes.

Proposals

A new proposal signing request must pass the height/round/step checks. Conflicting canonical bytes at the same height/round/proposal step are refused. The current repeated-proposal return path has a documented signature-field discrepancy; it must not be assumed to return a valid reused signature.

Votes

For a new height or round, or a later internal step in the same height/round, FilePV can sign and persist the vote. For identical block sign bytes at an already-signed height/round/step, it reuses the stored block signature; different bytes are refused. Precommit extension signatures are regenerated, since the extensions are not part of the canonical block vote.

Nil and non-nil block IDs are different sign bytes: a signer cannot replace its nil vote with a block vote of the same type at the same height and round. The block-signature reuse rule permits recovery of an identical vote, not a second conflicting vote. See WAL catch-up. DashCoreSignerClient does not maintain FilePV's persisted state inside Tenderdash; signer-side protection must not be inferred from the presence of the client alone.

Other Rules

Signer height/round/step protection does not establish lock eligibility. The consensus state machine decides whether a block may be prevoted or precommitted; see Locking and the precommit decision. A nil polka does not unlock within a height, and relocking a block requires this round's proposal and application processing context.

Historical Tendermint discussions call unjustified lock changes “amnesia”; see the original fork-detection discussion. That history does not introduce an unlock transition or additional signer protection into the current implementation.

WAL

Consensus module writes every message to the WAL (write-ahead log), except proposer heartbeats and round skip votes; round skip certificates are written once verified. Which records are written, with which durability, and how they are replayed is specified in Recovery and synchronisation.

It also issues fsync syscall through File#Sync for messages signed by this node and for the end-of-height marker (to prevent double signing and to delimit heights).

Under the hood, it uses autofile.Group (internal/libs/autofile), which rotates files when those get too big (> 10MB, defaultHeadSizeLimit).

The total maximum size is 1GB (defaultTotalSizeLimit). We only need the latest block and the block before it, but if the former is dragging on across many rounds, we want all those rounds.

Replay

Consensus module will replay all the messages of the last height written to WAL before a crash (if such occurs).

The private validator may try to sign messages during replay because it runs somewhat autonomously and does not know about replay process.

For example, if we got all the way to precommit in the WAL and then crash, after we replay the proposal message, the private validator will try to sign a prevote. But it will fail. That's ok because we’ll see the prevote later in the WAL. Then it will go to precommit, and that time it will work because the private validator contains the LastSignBytes and then we’ll replay the precommit from the WAL.

Make sure to read about WAL corruption and recovery strategies.

P2P Config

Here we describe configuration options around the Peer Exchange. These can be set using flags or via the $TMHOME/config/config.toml file.

Seed Mode

--p2p.seed_mode

The node operates in seed mode. In seed mode, a node continuously crawls the network for peers, and upon incoming connection shares some peers and disconnects.

Seeds

--p2p.seeds “id100000000000000000000000000000000@1.2.3.4:26656,id200000000000000000000000000000000@2.3.4.5:4444”

Dials these seeds when we need more peers. They should return a list of peers and then disconnect. If we already have enough peers in the address book, we may never need to dial them.

Persistent Peers

--p2p.persistent_peers “id100000000000000000000000000000000@1.2.3.4:26656,id200000000000000000000000000000000@2.3.4.5:26656”

Dial these peers and auto-redial them if the connection fails. These are intended to be trusted persistent peers that can help anchor us in the p2p network. The auto-redial uses exponential backoff and will give up after a day of trying to connect.

But If persistent_peers_max_dial_period is set greater than zero, pause between each dial to each persistent peer will not exceed persistent_peers_max_dial_period during exponential backoff and we keep trying again without giving up

Note: If seeds and persistent_peers intersect, the user will be warned that seeds may auto-close connections and that the node may not be able to keep the connection persistent.

Private Peers

--p2p.private_peer_ids “id100000000000000000000000000000000,id200000000000000000000000000000000”

These are IDs of the peers that we do not add to the address book or gossip to other peers. They stay private to us.

Unconditional Peers

--p2p.unconditional_peer_ids “id100000000000000000000000000000000,id200000000000000000000000000000000”

These are IDs of the peers which are allowed to be connected by both inbound or outbound regardless of max_num_inbound_peers or max_num_outbound_peers of user's node reached or not.

P2P Multiplex Connection

MConnection

MConnection is a multiplex connection that supports multiple independent streams with distinct quality of service guarantees atop a single TCP connection. Each stream is known as a Channel and each Channel has a globally unique byte id. Each Channel also has a relative priority that determines the quality of service of the Channel compared to other Channels. The byte id and the relative priorities of each Channel are configured upon initialization of the connection.

The MConnection supports three packet types:

  • Ping
  • Pong
  • Msg

Ping and Pong

The ping and pong messages consist of writing a single byte to the connection; 0x1 and 0x2, respectively.

When we haven't received any messages on an MConnection in time pingTimeout, we send a ping message. When a ping is received on the MConnection, a pong is sent in response only if there are no other messages to send and the peer has not sent us too many pings (TODO).

If a pong or message is not received in sufficient time after a ping, the peer is disconnected from.

Msg

Messages in channels are chopped into smaller msgPackets for multiplexing.

type msgPacket struct {
 ChannelID byte
 EOF       byte // 1 means message ends here.
 Bytes     []byte
}

The msgPacket is serialized using Proto3. The received Bytes of a sequential set of packets are appended together until a packet with EOF=1 is received, then the complete serialized message is returned for processing by the onReceive function of the corresponding channel.

Multiplexing

Messages are sent from a single sendRoutine, which loops over a select statement and results in the sending of a ping, a pong, or a batch of data messages. The batch of data messages may include messages from multiple channels. Message bytes are queued for sending in their respective channel, with each channel holding one unsent message at a time. Messages are chosen for a batch one at a time from the channel with the lowest ratio of recently sent bytes to channel priority.

Sending Messages

There are two methods for sending messages:

func (m MConnection) Send(chID byte, msg interface{}) bool {}
func (m MConnection) TrySend(chID byte, msg interface{}) bool {}

Send(chID, msg) is a blocking call that waits until msg is successfully queued for the channel with the given id byte chID. The message msg is serialized using the tendermint/go-amino submodule's WriteBinary() reflection routine.

TrySend(chID, msg) is a nonblocking call that queues the message msg in the channel with the given id byte chID if the queue is not full; otherwise it returns false immediately.

Send() and TrySend() are also exposed for each Peer.

Peer

Each peer has one MConnection instance, and includes other information such as whether the connection was outbound, whether the connection should be recreated if it closes, various identity information about the node, and other higher level thread-safe data used by the reactors.

Switch/Reactor

The Switch handles peer connections and exposes an API to receive incoming messages on Reactors. Each Reactor is responsible for handling incoming messages of one or more Channels. So while sending outgoing messages is typically performed on the peer, incoming messages are received on the reactor.

// Declare a MyReactor reactor that handles messages on MyChannelID.
type MyReactor struct{}

func (reactor MyReactor) GetChannels() []*ChannelDescriptor {
    return []*ChannelDescriptor{ChannelDescriptor{ID:MyChannelID, Priority: 1}}
}

func (reactor MyReactor) Receive(chID byte, peer *Peer, msgBytes []byte) {
    r, n, err := bytes.NewBuffer(msgBytes), new(int64), new(error)
    msgString := ReadString(r, n, err)
    fmt.Println(msgString)
}

// Other Reactor methods omitted for brevity
...

switch := NewSwitch([]Reactor{MyReactor{}})

...

// Send a random message to all outbound connections
for _, peer := range switch.Peers().List() {
    if peer.IsOutbound() {
        peer.Send(MyChannelID, "Here's a random message")
    }
}

Messages

An implementation of the spec consists of many components. While many parts of these components are implementation specific, the p2p messages are not. In this section we will be covering all the p2p messages of components.

There are two parts to the P2P messages, the message and the channel. The channel is message specific and messages are specific to components of Tenderdash. When a node connect to a peer it will tell the other node which channels are available. This notifies the peer what services the connecting node offers. You can read more on channels in connection.md

Block Sync

Channel

Block sync has one channel.

NameNumber
BlockchainChannel64

Message Types

There are multiple message types for Block Sync

BlockRequest

BlockRequest asks a peer for a block at the height specified.

NameTypeDescriptionField Number
Heightint64Height of requested block1

NoBlockResponse

NoBlockResponse notifies the peer requesting a block that the node does not contain it.

NameTypeDescriptionField Number
Heightint64Height of requested block1

BlockResponse

BlockResponse contains the block requested and its commit.

NameTypeDescriptionField Number
BlockbytesRequested Block, serialized as it was committed: the bytes of its parts, in order. On the wire this is the same as an embedded Block.1
CommitCommitCommit for the requested block2

The receiver cuts Block into parts of 64kB, the last one shorter, and checks the part set header it computes against the one in the commit's block ID. Block is served as stored, because decoding and re-encoding the block can change its bytes.

Block sync rebuilds only this standard cut, which honest proposers use. Consensus does not require it yet: it accepts parts of other sizes, and proofs of a tree larger than the header counts. A block committed that way fails the commit check (wrong block ID) whichever peer serves it, and block sync does not get past its height.

StatusRequest

StatusRequest is an empty message that notifies the peer to respond with the highest and lowest blocks it has stored.

Empty message.

StatusResponse

StatusResponse responds to a peer with the highest and lowest block stored.

NameTypeDescriptionField Number
Heightint64Current Height of a node1
baseint64First known block, if pruning is enabled it will be higher than 11

Message

Message is a oneof protobuf type. The oneof consists of five messages.

NameTypeDescriptionField Number
block_requestBlockRequestRequest a block from a peer1
no_block_responseNoBlockResponseResponse saying it doe snot have the requested block2
block_responseBlockResponseResponse with requested block3
status_requestStatusRequestRequest the highest and lowest block numbers from a peer4
status_responseStatusResponseResponse with the highest and lowest block numbers the store5

Consensus

Wire format of the messages exchanged by the consensus reactor. How a node acts on them is specified in the consensus protocol.

Channel

Consensus has four separate channels. The channel identifiers are listed below.

NameNumber
StateChannel32
DataChannel33
VoteChannel34
VoteSetBitsChannel35

Message Types

Proposal

Proposal is sent when a new block is proposed. It is a suggestion of what the next block in the blockchain should be.

NameTypeDescriptionField Number
proposalProposalProposed Block to come to consensus on1

Vote

Vote is sent to vote for some block (or to inform others that a process does not vote in the current round). Vote contains the validator's information (ProTxHash and index), height and round for which the vote is sent, vote type, blockID if process vote for some block (nil otherwise), the block signature and, on a non-nil precommit, the vote extensions. The message is signed by the validator private key.

NameTypeDescriptionField Number
voteVoteVote for a proposed Block1

BlockPart

BlockPart is sent when gossiping a piece of the proposed block. It contains height, round and the block part.

NameTypeDescriptionField Number
heightint64Height of corresponding block.1
roundint32Round of voting to finalize the block.2
partPartA part of the block.3

NewRoundStep

NewRoundStep is sent for every step transition during the core consensus algorithm execution. It is used in the gossip part of the Tenderdash protocol to inform peers about a current height/round/step a process is in. It is also sent to each peer when the peer connects and then periodically, at the peer-query-maj23-sleep-duration cadence, so that a peer whose copy was lost in transit learns the current step without waiting for the next transition. A receiver ignores a copy that does not advance the height, round or step it already holds for the sender.

NameTypeDescriptionField Number
heightint64Height of corresponding block1
roundint32Round of voting to finalize the block.2
stepuint323
seconds_since_start_timeint644
last_commit_roundint325

NewValidBlock

NewValidBlock advertises a block-part download target: its height, round, part-set header, and the parts the node already holds. is_commit identifies a commit target. Consensus emits this event for a current-round polka or a verified commit even when the proposal or complete block is not held. It is a gossip announcement, not proof that the sender has processed a valid proposal. While the node holds block parts for its round, the periodic per-peer NewRoundStep described above is followed by a NewValidBlock for the same height and round. See valid-block bookkeeping and commit handling.

NameTypeDescriptionField Number
heightint64Height of corresponding block1
roundint32Round of voting to finalize the block.2
block_part_set_headerPartSetHeader3
block_partsBitArray4
is_commitbool5

ProposalPOL

ProposalPOL is sent when a previous block is re-proposed. It is used to inform peers in what round the process learned for this block (ProposalPOLRound), and what prevotes for the re-proposed block the process has.

NameTypeDescriptionField Number
heightint64Height of corresponding block1
proposal_pol_roundint322
proposal_polbitarray3

HasVote

HasVote is sent to indicate that a particular vote has been received. It contains height, round, vote type and the index of the validator that is the originator of the corresponding vote.

NameTypeDescriptionField Number
heightint64Height of corresponding block1
roundint32Round of voting to finalize the block.2
typeSignedMsgType3
indexint324

Commit

Commit is sent on the VoteChannel to deliver a threshold-signed commit for a block.

NameTypeDescriptionField Number
commitCommitCommit for a decided block1

HasCommit

HasCommit is sent to indicate that a process holds the commit for a given height and round.

NameTypeDescriptionField Number
heightint64Height of the commit1
roundint32Round the commit belongs to2

VoteSetMaj23

VoteSetMaj23 is sent to indicate that a process has seen +2/3 votes for some BlockID. It contains height, round, vote type and the BlockID.

NameTypeDescriptionField Number
heightint64Height of corresponding block1
roundint32Round of voting to finalize the block.2
typeSignedMsgType3
block_idBlockID4

VoteSetBits

VoteSetBits is sent to communicate the bit-array of votes a process has seen for a given BlockID. It contains height, round, vote type, BlockID and a bit array of the votes a process has.

NameTypeDescriptionField Number
heightint64Height of corresponding block1
roundint32Round of voting to finalize the block.2
typeSignedMsgType3
block_idBlockID4
votesBitArrayVotes the process has for the BlockID.5

ProposerHeartbeat

ProposerHeartbeat is sent on the StateChannel by the proposer of a round that waits for transactions, and relayed once by validators that verify it (CONSENSUS_VERSION_2; see Proposer liveness).

NameTypeDescriptionField Number
heartbeatProposerHeartbeatThe signed heartbeat.1

ProposerHeartbeat contents

tendermint.types.ProposerHeartbeat:

NameTypeDescriptionField Number
heightint64Height of the round.1
roundint32Round the sender proposes in.2
sequenceuint64Grows with every heartbeat of the round, counting from the signing time in milliseconds.3
proposer_pro_tx_hashbytesProTxHash of the proposer.4
timestampgoogle.protobuf.TimestampSigning time.5
signaturebytesSignature with the proposer's quorum key share (Signing).6

RoundSkipVote

RoundSkipVote is sent on the StateChannel by a validator that heard no heartbeat from the proposer of its waiting round for the silence timeout, and relayed by validators when they first count it and again on a later copy until the round is certified (CONSENSUS_VERSION_2; see Collecting votes).

NameTypeDescriptionField Number
voteRoundSkipVoteThe signed skip vote.1

RoundSkipVote contents

tendermint.types.RoundSkipVote:

NameTypeDescriptionField Number
heightint64Height of the round to skip.1
roundint32Round to skip.2
validator_pro_tx_hashbytesProTxHash of the voting validator.3
validator_indexint32Index of the voting validator in the validator set.4
signaturebytesSignature share over the round's skip sign bytes, the same for every validator.5

RoundSkipCertificate

RoundSkipCertificate is sent on the StateChannel by a node that recovered or received a valid certificate, and resent by gossip to peers at its height that have not sent one for that round or a later one (CONSENSUS_VERSION_2).

NameTypeDescriptionField Number
certificateRoundSkipCertificateThe round skip certificate.1

RoundSkipCertificate contents

tendermint.types.RoundSkipCertificate:

NameTypeDescriptionField Number
heightint64Height of the skipped round.1
roundint32Skipped round.2
quorum_hashbytesQuorum that signed.3
threshold_signaturebytesQuorum threshold signature recovered from the skip votes of +2/3 of the validator set (the mainnet default; the quorum type and VotingPowerThreshold can change it).4

Message

Message is a oneof protobuf type.

NameTypeDescriptionField Number
new_round_stepNewRoundStep1
new_valid_blockNewValidBlock2
proposalProposal3
proposal_polProposalPOL4
block_partBlockPart5
voteVote6
has_voteHasVote7
vote_set_maj23VoteSetMaj238
vote_set_bitsVoteSetBits9
commitCommit10
has_commitHasCommit11
proposer_heartbeatProposerHeartbeat12
round_skip_voteRoundSkipVote13
round_skip_certificateRoundSkipCertificate14

Evidence

Channel

Evidence has one channel. The channel identifier is listed below.

NameNumber
EvidenceChannel56

Message Types

Evidence

Verified evidence that has already been propagated throughout the network. This evidence will appear within the EvidenceList struct of a block as well.

NameTypeDescriptionField Number
evidenceEvidenceValid evidence1

Mempool

Channel

Mempool has one channel. The channel identifier is listed below.

NameNumber
MempoolChannel48

Message Types

There is currently only one message that Mempool broadcasts and receives over the p2p gossip network (via the reactor): TxsMessage

Txs

A list of transactions. These transactions have been checked against the application for validity. This does not mean that the transactions are valid, it is up to the application to check this.

NameTypeDescriptionField Number
txsrepeated bytesList of transactions1

Message

Message is a oneof protobuf type. The one of consists of one message Txs.

NameTypeDescriptionField Number
txsTxsList of transactions1

Peer Exchange

Channels

Pex has one channel. The channel identifier is listed below.

NameNumber
PexChannel0

Message Types

PexRequest

PexRequest is an empty message requesting a list of peers.

EmptyRequest

PexResponse

PexResponse is an list of net addresses provided to a peer to dial.

NameTypeDescriptionField Number
addressesrepeated PexAddressList of peer addresses available to dial1

PexAddress

PexAddress provides needed information for a node to dial a peer. This is in the form of a URL that gets parsed into a NodeAddress. See ParseNodeAddress for more details.

NameTypeDescriptionField Number
urlstringSee golang url1

Message

Message is a oneof protobuf type. The one of consists of two messages.

NameTypeDescriptionField Number
pex_requestPexRequestEmpty request asking for a list of addresses to dial3
pex_responsePexResponseList of addresses to dial4

State Sync

Channels

State sync has four distinct channels. The channel identifiers are listed below.

NameNumber
SnapshotChannel96
ChunkChannel97
LightBlockChannel98
ParamsChannel99

Snapshot and chunk model

Tenderdash uses a content-addressed chunk model that differs from upstream Tendermint. A snapshot is identified by its height, an application-specific version, and a hash; there is no format field and no up-front chunk count. Chunks are identified by an opaque, application-defined chunk_id (in practice the hash of the corresponding node or subtree of the snapshot) rather than by a sequential index.

Chunk discovery is driven by the ABCI application during restoration:

  1. The syncer requests the first chunk using the snapshot hash as the initial chunk_id.
  2. Each applied chunk is handed to the application via ApplySnapshotChunk; the response's next_chunks field lists the chunk IDs to fetch next, and refetch_chunks can ask for chunks to be re-fetched.
  3. Restoration terminates when the application returns the COMPLETE_SNAPSHOT result — not when a predefined number of chunks has been applied.

Message Types

SnapshotsRequest

When a new node begins state syncing, it will ask all peers it encounters if they have any available snapshots. The message has no fields:

NameTypeDescriptionField Number

SnapshotsResponse

The receiver will query the local ABCI application via ListSnapshots, and send a message containing snapshot metadata (limited to 4 MB) for each of the 10 most recent snapshots:

NameTypeDescriptionField Number
heightuint64Height at which the snapshot was taken1
versionuint32Application-specific snapshot version2
hashbytesSnapshot hash; also the ID of the first chunk to request3
metadatabytesArbitrary application data. May be non-deterministic.4

ChunkRequest

The node running state sync will offer these snapshots to the local ABCI application via OfferSnapshot ABCI calls, and keep track of which peers contain which snapshots. Once a snapshot is accepted, the state syncer will request snapshot chunks from appropriate peers:

NameTypeDescriptionField Number
heightuint64Height at which the snapshot was taken1
versionuint32Application-specific snapshot version2
chunk_idbytesContent-addressed ID of the requested chunk3

ChunkResponse

The receiver will load the requested chunk from its local application via LoadSnapshotChunk, and respond with it (limited to 16 MB):

NameTypeDescriptionField Number
heightuint64Height at which the snapshot was taken1
versionuint32Application-specific snapshot version2
chunk_idbytesContent-addressed ID of the chunk3
chunkbytesBinary chunk contents4
missingboolTrue if the chunk was not found on the peer5

Here, missing is used to signify that the chunk was not found on the peer, since an empty chunk is a valid (although unlikely) response.

The returned chunk is given to the ABCI application via ApplySnapshotChunk. The application response determines what happens next: next_chunks lists further chunk IDs to fetch, refetch_chunks requests re-fetching of chunks, reject_senders requests peer bans, and the COMPLETE_SNAPSHOT result ends the restoration. If a chunk response is not returned within some time, it will be re-requested, possibly from a different peer.

LightBlockRequest

To verify state and to provide state relevant information for consensus, the node will ask peers for light blocks at specified heights.

NameTypeDescriptionField Number
heightuint64Height of the light block1

LightBlockResponse

The receiver will retrieve and construct the light block from both the block and state stores. The receiver will verify the data by comparing the hashes and store the header, commit and validator set if necessary. The light block at the height of the snapshot will be used to verify the AppHash.

NameTypeDescriptionField Number
light_blockLightBlockLight block at the height requested1

Unlike upstream Tendermint, light blocks are not verified against a trusted header obtained out of band. Tenderdash's light client verifies the quorum threshold signature on the block commit via Dash Core (quorum verify), so no trust anchors (trusted height, hash, or period) are required.

If no state sync is in progress (i.e. during normal operation), any unsolicited response messages are discarded.

ParamsRequest

In order to build Tenderdash state, the state provider will request the params at the height of the snapshot and use the header to verify it.

NameTypeDescriptionField Number
heightuint64Height of the consensus params1

ParamsResponse

A receiver to the request will use the state store to fetch the consensus params at that height and return it to the sender.

NameTypeDescriptionField Number
heightuint64Height of the consensus params1
consensus_paramsConsensusParamsConsensus params at the height requested2

Peer Discovery

A Tenderdash P2P network has different kinds of nodes with different requirements for connectivity to one another. This document describes what kind of nodes Tenderdash should enable and how they should work.

Seeds

Seeds are the first point of contact for a new node. They return a list of known active peers and then disconnect.

Seeds should operate full nodes with the PEX reactor in a "crawler" mode that continuously explores to validate the availability of peers.

Seeds should only respond with some top percentile of the best peers it knows about.

New Full Node

A new node needs a few things to connect to the network:

  • a list of seeds, which can be provided to Tenderdash via config file or flags, or hardcoded into the software by in-process apps
  • a ChainID, also called Network at the p2p layer
  • a recent block height, H, and hash, HASH for the blockchain.

The values H and HASH must be received and corroborated by means external to Tenderdash, and specific to the user - ie. via the user's trusted social consensus. This requirement to validate H and HASH out-of-band and via social consensus is the essential difference in security models between Proof-of-Work and Proof-of-Stake blockchains.

With the above, the node then queries some seeds for peers for its chain, dials those peers, and runs the Tenderdash protocols with those it successfully connects to.

When the peer catches up to height H, it ensures the block hash matches HASH. If not, Tenderdash will exit, and the user must try again - either they are connected to bad peers or their social consensus is invalid.

Restarted Full Node

A node checks its address book on startup and attempts to connect to peers from there. If it can't connect to any peers after some time, it falls back to the seeds to find more.

Restarted full nodes can run the blockchain or consensus reactor protocols to sync up to the latest state of the blockchain from wherever they were last. In a Proof-of-Stake context, if they are sufficiently far behind (greater than the length of the unbonding period), they will need to validate a recent H and HASH out-of-band again so they know they have synced the correct chain.

Validator Node

A validator node is a node that interfaces with a validator signing key. These nodes require the highest security, and should not accept incoming connections. They should maintain outgoing connections to a controlled set of "Sentry Nodes" that serve as their proxy shield to the rest of the network.

Validators that know and trust each other can accept incoming connections from one another and maintain direct private connectivity via VPN.

Sentry Node

Sentry nodes are guardians of a validator node and provide it access to the rest of the network. They should be well connected to other full nodes on the network. Sentry nodes may be dynamic, but should maintain persistent connections to some evolving random subset of each other. They should always expect to have direct incoming connections from the validator node and its backup(s). They do not report the validator node's address in the PEX and they may be more strict about the quality of peers they keep.

Sentry nodes belonging to validators that trust each other may wish to maintain persistent connections via VPN with one another, but only report each other sparingly in the PEX.

Peers

This document explains how Tenderdash Peers are identified and how they connect to one another.

For details on peer discovery, see the peer exchange (PEX) doc.

Peer Identity

Tenderdash peers are expected to maintain long-term persistent identities in the form of a public key. Each peer has an ID defined as peer.ID == peer.PubKey.Address(), where Address uses the scheme defined in crypto package.

A single peer ID can have multiple IP addresses associated with it, but a node will only ever connect to one at a time.

When attempting to connect to a peer, we use the PeerURL: <ID>@<IP>:<PORT>. We will attempt to connect to the peer at IP:PORT, and verify, via authenticated encryption, that it is in possession of the private key corresponding to <ID>. This prevents man-in-the-middle attacks on the peer layer.

Connections

All p2p connections use TCP. Upon establishing a successful TCP connection with a peer, two handshakes are performed: one for authenticated encryption, and one for Tenderdash versioning. Both handshakes have configurable timeouts (they should complete quickly).

Authenticated Encryption Handshake

Tenderdash implements the Station-to-Station protocol using X25519 keys for Diffie-Helman key-exchange and chacha20poly1305 for encryption.

Upstream Tendermint versions of this protocol (0.32 and below) suffered from malleability attacks whereas an active man in the middle attacker could compromise confidentiality as described in Prime, Order Please! Revisiting Small Subgroup and Invalid Curve Attacks on Protocols using Diffie-Hellman.

We have added dependency on the Merlin a keccak based transcript hashing protocol to ensure non-malleability.

It goes as follows:

  • generate an ephemeral X25519 keypair
  • send the ephemeral public key to the peer
  • wait to receive the peer's ephemeral public key
  • create a new Merlin Transcript with the string "TENDERMINT_SECRET_CONNECTION_TRANSCRIPT_HASH"
  • Sort the ephemeral keys and add the high labeled "EPHEMERAL_UPPER_PUBLIC_KEY" and the low keys labeled "EPHEMERAL_LOWER_PUBLIC_KEY" to the Merlin transcript.
  • compute the Diffie-Hellman shared secret using the peers ephemeral public key and our ephemeral private key
  • add the DH secret to the transcript labeled DH_SECRET.
  • generate two keys to use for encryption (sending and receiving) and a challenge for authentication as follows:
    • create a hkdf-sha256 instance with the key being the diffie hellman shared secret, and info parameter as TENDERMINT_SECRET_CONNECTION_KEY_AND_CHALLENGE_GEN
    • get 64 bytes of output from hkdf-sha256
    • if we had the smaller ephemeral pubkey, use the first 32 bytes for the key for receiving, the second 32 bytes for sending; else the opposite.
  • use a separate nonce for receiving and sending. Both nonces start at 0, and should support the full 96 bit nonce range
  • all communications from now on are encrypted in 1400 byte frames (plus encoding overhead), using the respective secret and nonce. Each nonce is incremented by one after each use.
  • we now have an encrypted channel, but still need to authenticate
  • extract a 32 bytes challenge from merlin transcript with the label "SECRET_CONNECTION_MAC"
  • sign the common challenge obtained from the hkdf with our persistent private key
  • send the amino encoded persistent pubkey and signature to the peer
  • wait to receive the persistent public key and signature from the peer
  • verify the signature on the challenge using the peer's persistent public key

If this is an outgoing connection (we dialed the peer) and we used a peer ID, then finally verify that the peer's persistent public key corresponds to the peer ID we dialed, ie. peer.PubKey.Address() == <ID>.

The connection has now been authenticated. All traffic is encrypted.

Note: only the dialer can authenticate the identity of the peer, but this is what we care about since when we join the network we wish to ensure we have reached the intended peer (and are not being MITMd).

Peer Filter

Before continuing, we check if the new peer has the same ID as ourselves or an existing peer. If so, we disconnect.

We also check the peer's address and public key against an optional whitelist which can be managed through the ABCI app - if the whitelist is enabled and the peer does not qualify, the connection is terminated.

Tenderdash Version Handshake

The Tenderdash Version Handshake allows the peers to exchange their NodeInfo:

type NodeInfo struct {
  Version    p2p.Version
  ID         p2p.ID
  ListenAddr string

  Network    string
  SoftwareVersion    string
  Channels   []int8

  Moniker    string
  Other      NodeInfoOther
}

type Version struct {
 P2P uint64
 Block uint64
 App uint64
}

type NodeInfoOther struct {
 TxIndex          string
 RPCAddress       string
}

The connection is disconnected if:

  • peer.NodeInfo.ID is not equal peerConn.ID
  • peer.NodeInfo.Version.Block does not match ours
  • peer.NodeInfo.Network is not the same as ours
  • peer.Channels does not intersect with our known Channels.
  • peer.NodeInfo.ListenAddr is malformed or is a DNS host that cannot be resolved

At this point, if we have not disconnected, the peer is valid. It is added to the switch and hence all reactors via the AddPeer method. Note that each reactor may handle multiple channels.

Connection Activity

Once a peer is added, incoming messages for a given reactor are handled through that reactor's Receive method, and output messages are sent directly by the Reactors on each peer. A typical reactor maintains per-peer go-routine(s) that handle this.

P2P protocol

RPC spec

This file defines the JSON-RPC spec of Tenderdash. This is meant to be implemented by all clients.

Support

Tenderdash
JSON-RPC 2.0✅
HTTP✅
HTTPS✅
WS✅

Timestamps

Timestamps in the RPC layer of Tenderdash follows RFC3339Nano. The RFC3339Nano format removes trailing zeros from the seconds field.

This means if a block has a timestamp like: 1985-04-12T23:20:50.5200000Z, the value returned in the RPC will be 1985-04-12T23:20:50.52Z.

Info Routes

Health

Node heartbeat

Parameters

None

Request

HTTP
curl http://127.0.0.1:26657/health
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"health\"}"

Response

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {}
}

Status

Get Tenderdash status including node info, pubkey, latest block hash, app hash, block height and time.

Parameters

None

Request

HTTP
curl http://127.0.0.1:26657/status
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"status\"}"

Response

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "node_info": {
      "protocol_version": {
        "p2p": "8",
        "block": "11",
        "app": "0"
      },
      "id": "b93270b358a72a2db30089f3856475bb1f918d6d",
      "listen_addr": "tcp://0.0.0.0:26656",
      "network": "cosmoshub-4",
      "version": "v0.34.8",
      "channels": "40202122233038606100",
      "moniker": "aib-hub-node",
      "other": {
        "tx_index": "on",
        "rpc_address": "tcp://0.0.0.0:26657"
      }
    },
    "sync_info": {
      "latest_block_hash": "50F03C0EAACA8BCA7F9C14189ACE9C05A9A1BBB5268DB63DC6A3C848D1ECFD27",
      "latest_app_hash": "2316CFF7644219F4F15BEE456435F280E2B38955EEA6D4617CCB6D7ABF781C22",
      "latest_block_height": "5622165",
      "latest_block_time": "2021-03-25T14:00:43.356134226Z",
      "earliest_block_hash": "1455A0C15AC49BB506992EC85A3CD4D32367E53A087689815E01A524231C3ADF",
      "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
      "earliest_block_height": "5200791",
      "earliest_block_time": "2019-12-11T16:11:34Z",
      "catching_up": false
    },
    "validator_info": {
      "address": "38FB765D0092470989360ECA1C89CD06C2C1583C",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "Z+8kntVegi1sQiWLYwFSVLNWqdAUGEy7lskL78gxLZI="
      },
      "voting_power": "0"
    }
  }
}

NetInfo

Network information

Parameters

None

Request

HTTP
curl http://127.0.0.1:26657/net_info
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"net_info\"}"

Response

{
  "id": 0,
  "jsonrpc": "2.0",
  "result": {
    "listening": true,
    "listeners": [
      "Listener(@)"
    ],
    "n_peers": "1",
	"peers": [
	  {
		  "node_id": "5576458aef205977e18fd50b274e9b5d9014525a",
		  "url": "tcp://5576458aef205977e18fd50b274e9b5d9014525a@95.179.155.35:26656"
	  }
	]
  }
}

Blockchain

Get block headers. Returned in descending order. May be limited in quantity.

Parameters

  • minHeight (integer): The lowest block to be returned in the response
  • maxHeight (integer): The highest block to be returned in the response

Request

HTTP
curl http://127.0.0.1:26657/blockchain

curl http://127.0.0.1:26657/blockchain?minHeight=1&maxHeight=2
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"blockchain\",\"params\":{\"minHeight\":\"1\", \"maxHeight\":\"2\"}}"

Response

{
  "id": 0,
  "jsonrpc": "2.0",
  "result": {
    "last_height": "1276718",
    "block_metas": [
      {
        "block_id": {
          "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
          "parts": {
            "total": 1,
            "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
          }
        },
        "block_size": 1000000,
        "header": {
          "version": {
            "block": "10",
            "app": "0"
          },
          "chain_id": "cosmoshub-2",
          "height": "12",
          "time": "2019-04-22T17:01:51.701356223Z",
          "last_block_id": {
            "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
            "parts": {
              "total": 1,
              "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
            }
          },
          "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
          "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
          "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
          "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
          "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
          "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
          "last_results_hash": "",
          "evidence_hash": "",
          "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
        },
        "num_txs": "54"
      }
    ]
  }
}

Header

Get a header at a specified height.

Parameters

  • height (integer): height of the requested header. If no height is specified the latest height will be used.

Request

HTTP
curl http://127.0.0.1:26657/header

curl http://127.0.0.1:26657/header?height=1
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"header\",\"params\":{\"height\":\"1\"}}"

Response

{
  "id": 0,
  "jsonrpc": "2.0",
  "result": {
    "header": {
      "version": {
        "block": "10",
        "app": "0"
      },
      "chain_id": "cosmoshub-2",
      "height": "12",
      "time": "2019-04-22T17:01:51.701356223Z",
      "last_block_id": {
        "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
        "parts": {
          "total": 1,
          "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
        }
      },
      "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
      "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
      "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
      "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
      "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
      "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
      "last_results_hash": "",
      "evidence_hash": "",
      "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
    }
  }
}

HeaderByHash

Parameters

  • hash (string): Hash of the header to query for.

Request

HTTP
curl http://127.0.0.1:26657/header_by_hash?hash=0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"header_by_hash\",\"params\":{\"hash\":\"0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED\"}}"

Response

{
  "id": 0,
  "jsonrpc": "2.0",
  "result": {
      "header": {
        "version": {
          "block": "10",
          "app": "0"
        },
        "chain_id": "cosmoshub-2",
        "height": "12",
        "time": "2019-04-22T17:01:51.701356223Z",
        "last_block_id": {
          "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
          "parts": {
            "total": 1,
            "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
          }
        },
        "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
        "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
        "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
        "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
        "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
        "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
        "last_results_hash": "",
        "evidence_hash": "",
        "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
      }
    }
  }
}

Block

Get block at a specified height.

Parameters

  • height (integer): height of the requested block. If no height is specified the latest height will be used.

Request

HTTP
curl http://127.0.0.1:26657/block

curl http://127.0.0.1:26657/block?height=1
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"block\",\"params\":{\"height\":\"1\"}}"

Response

{
  "id": 0,
  "jsonrpc": "2.0",
  "result": {
    "block_id": {
      "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
      "parts": {
        "total": 1,
        "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
      }
    },
    "block": {
      "header": {
        "version": {
          "block": "10",
          "app": "0"
        },
        "chain_id": "cosmoshub-2",
        "height": "12",
        "time": "2019-04-22T17:01:51.701356223Z",
        "last_block_id": {
          "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
          "parts": {
            "total": 1,
            "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
          }
        },
        "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
        "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
        "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
        "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
        "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
        "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
        "last_results_hash": "",
        "evidence_hash": "",
        "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
      },
      "data": [
        "yQHwYl3uCkKoo2GaChRnd+THLQ2RM87nEZrE19910Z28ABIUWW/t8AtIMwcyU0sT32RcMDI9GF0aEAoFdWF0b20SBzEwMDAwMDASEwoNCgV1YXRvbRIEMzEwMRCd8gEaagom61rphyEDoJPxlcjRoNDtZ9xMdvs+lRzFaHe2dl2P5R2yVCWrsHISQKkqX5H1zXAIJuC57yw0Yb03Fwy75VRip0ZBtLiYsUqkOsPUoQZAhDNP+6LY+RUwz/nVzedkF0S29NZ32QXdGv0="
      ],
      "evidence": [
        {
          "type": "string",
          "height": 0,
          "time": 0,
          "total_voting_power": 0,
          "validator": {
            "pub_key": {
              "type": "tendermint/PubKeyEd25519",
              "value": "A6DoBUypNtUAyEHWtQ9bFjfNg8Bo9CrnkUGl6k6OHN4="
            },
            "voting_power": 0,
            "address": "string"
          }
        }
      ],
      "last_commit": {
        "height": 0,
        "round": 0,
        "block_id": {
          "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
          "parts": {
            "total": 1,
            "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
          }
        },
        "signatures": [
          {
            "type": 2,
            "height": "1262085",
            "round": 0,
            "block_id": {
              "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
              "parts": {
                "total": 1,
                "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
              }
            },
            "timestamp": "2019-08-01T11:39:38.867269833Z",
            "validator_address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F",
            "validator_index": 0,
            "signature": "DBchvucTzAUEJnGYpNvMdqLhBAHG4Px8BsOBB3J3mAFCLGeuG7uJqy+nVngKzZdPhPi8RhmE/xcw/M9DOJjEDg=="
          }
        ]
      }
    }
  }
}

BlockByHash

Parameters

  • hash (string): Hash of the block to query for.

Request

HTTP
curl http://127.0.0.1:26657/block_by_hash?hash=0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"block_by_hash\",\"params\":{\"hash\":\"0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED\"}}"

Response

{
  "id": 0,
  "jsonrpc": "2.0",
  "result": {
    "block_id": {
      "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
      "parts": {
        "total": 1,
        "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
      }
    },
    "block": {
      "header": {
        "version": {
          "block": "10",
          "app": "0"
        },
        "chain_id": "cosmoshub-2",
        "height": "12",
        "time": "2019-04-22T17:01:51.701356223Z",
        "last_block_id": {
          "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
          "parts": {
            "total": 1,
            "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
          }
        },
        "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
        "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
        "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
        "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
        "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
        "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
        "last_results_hash": "",
        "evidence_hash": "",
        "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
      },
      "data": [
        "yQHwYl3uCkKoo2GaChRnd+THLQ2RM87nEZrE19910Z28ABIUWW/t8AtIMwcyU0sT32RcMDI9GF0aEAoFdWF0b20SBzEwMDAwMDASEwoNCgV1YXRvbRIEMzEwMRCd8gEaagom61rphyEDoJPxlcjRoNDtZ9xMdvs+lRzFaHe2dl2P5R2yVCWrsHISQKkqX5H1zXAIJuC57yw0Yb03Fwy75VRip0ZBtLiYsUqkOsPUoQZAhDNP+6LY+RUwz/nVzedkF0S29NZ32QXdGv0="
      ],
      "evidence": [
        {
          "type": "string",
          "height": 0,
          "time": 0,
          "total_voting_power": 0,
          "validator": {
            "pub_key": {
              "type": "tendermint/PubKeyEd25519",
              "value": "A6DoBUypNtUAyEHWtQ9bFjfNg8Bo9CrnkUGl6k6OHN4="
            },
            "voting_power": 0,
            "address": "string"
          }
        }
      ],
      "last_commit": {
        "height": 0,
        "round": 0,
        "block_id": {
          "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
          "parts": {
            "total": 1,
            "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
          }
        },
        "signatures": [
          {
            "type": 2,
            "height": "1262085",
            "round": 0,
            "block_id": {
              "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
              "parts": {
                "total": 1,
                "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
              }
            },
            "timestamp": "2019-08-01T11:39:38.867269833Z",
            "validator_address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F",
            "validator_index": 0,
            "signature": "DBchvucTzAUEJnGYpNvMdqLhBAHG4Px8BsOBB3J3mAFCLGeuG7uJqy+nVngKzZdPhPi8RhmE/xcw/M9DOJjEDg=="
          }
        ]
      }
    }
  }
}

BlockResults

Parameters

  • height (integer): Height of the block which contains the results. If no height is specified, the latest block height will be used

Request

HTTP
curl  http://127.0.0.1:26657/block_results


curl  http://127.0.0.1:26657/block_results?height=1
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"block_results\",\"params\":{\"height\":\"1\"}}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "height": "12",
    "total_gas_used": "100",
    "txs_results": [
      {
        "code": "0",
        "data": "",
        "log": "not enough gas",
        "info": "",
        "gas_wanted": "100",
        "gas_used": "100",
        "events": [
          {
            "type": "app",
            "attributes": [
              {
                "key": "YWN0aW9u",
                "value": "c2VuZA==",
                "index": false
              }
            ]
          }
        ],
        "codespace": "ibc"
      }
    ],
    "begin_block_events": [
      {
        "type": "app",
        "attributes": [
          {
            "key": "YWN0aW9u",
            "value": "c2VuZA==",
            "index": false
          }
        ]
      }
    ],
    "end_block": [
      {
        "type": "app",
        "attributes": [
          {
            "key": "YWN0aW9u",
            "value": "c2VuZA==",
            "index": false
          }
        ]
      }
    ],
    "validator_updates": [
      {
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "9tK9IT+FPdf2qm+5c2qaxi10sWP+3erWTKgftn2PaQM="
        },
        "power": "300"
      }
    ],
    "consensus_params_updates": {
      "block": {
        "max_bytes": "22020096",
        "max_gas": "1000",
        "time_iota_ms": "1000"
      },
      "evidence": {
        "max_age": "100000"
      },
      "validator": {
        "pub_key_types": [
          "ed25519"
        ]
      }
    }
  }
}

Commit

Parameters

  • height (integer): Height of the block the requested commit pertains to. If no height is set the latest commit will be returned.

Request

HTTP
curl  http://127.0.0.1:26657/commit


curl  http://127.0.0.1:26657/commit?height=1
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"commit\",\"params\":{\"height\":\"1\"}}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "signed_header": {
      "header": {
        "version": {
          "block": "10",
          "app": "0"
        },
        "chain_id": "cosmoshub-2",
        "height": "12",
        "time": "2019-04-22T17:01:51.701356223Z",
        "last_block_id": {
          "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
          "parts": {
            "total": 1,
            "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
          }
        },
        "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
        "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
        "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
        "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
        "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
        "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
        "last_results_hash": "",
        "evidence_hash": "",
        "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
      },
      "commit": {
        "height": "1311801",
        "round": 0,
        "block_id": {
          "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
          "parts": {
            "total": 1,
            "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
          }
        },
        "signatures": [
          {
            "block_id_flag": 2,
            "validator_address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F",
            "timestamp": "2019-04-22T17:01:58.376629719Z",
            "signature": "14jaTQXYRt8kbLKEhdHq7AXycrFImiLuZx50uOjs2+Zv+2i7RTG/jnObD07Jo2ubZ8xd7bNBJMqkgtkd0oQHAw=="
          }
        ]
      }
    },
    "canonical": true
  }
}

Validators

Parameters

  • height (integer): Block height at which the validators were present on. If no height is set the latest commit will be returned.
  • page (integer):
  • per_page (integer):

Request

HTTP
curl  http://127.0.0.1:26657/validators
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"validators\",\"params\":{\"height\":\"1\", \"page\":\"1\", \"per_page\":\"20\"}}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "block_height": "55",
    "validators": [
      {
        "address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F",
        "pub_key": {
          "type": "tendermint/PubKeyEd25519",
          "value": "9tK9IT+FPdf2qm+5c2qaxi10sWP+3erWTKgftn2PaQM="
        },
        "voting_power": "239727",
        "proposer_priority": "-11896414"
      }
    ],
    "count": "1",
    "total": "25"
  }
}

Genesis

Get Genesis of the chain. If the response is large, this operation will return an error: use genesis_chunked instead.

Request

HTTP
curl  http://127.0.0.1:26657/genesis
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"genesis\"}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "genesis": {
      "genesis_time": "2019-04-22T17:00:00Z",
      "chain_id": "cosmoshub-2",
      "initial_height": "2",
      "consensus_params": {
        "block": {
          "max_bytes": "22020096",
          "max_gas": "1000",
          "time_iota_ms": "1000"
        },
        "evidence": {
          "max_age": "100000"
        },
        "validator": {
          "pub_key_types": [
            "ed25519"
          ]
        }
      },
      "validators": [
        {
          "address": "B00A6323737F321EB0B8D59C6FD497A14B60938A",
          "pub_key": {
            "type": "tendermint/PubKeyEd25519",
            "value": "cOQZvh/h9ZioSeUMZB/1Vy1Xo5x2sjrVjlE/qHnYifM="
          },
          "power": "9328525",
          "name": "Certus One"
        }
      ],
      "app_hash": "",
      "app_state": {}
    }
  }
}

GenesisChunked

Get the genesis document in a chunks to support easily transfering larger documents.

Parameters

  • chunk (integer): the index number of the chunk that you wish to fetch. These IDs are 0 indexed.

Request

HTTP
curl  http://127.0.0.1:26657/genesis_chunked?chunk=0
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"genesis_chunked\",\"params\":{\"chunk\":0}}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
     "chunk": 0,
     "total": 10,
     "data": "dGVuZGVybWludAo="
  }
}

ConsensusParams

Get the consensus parameters.

Parameters

  • height (integer): Block height at which the consensus params would like to be fetched for.

Request

HTTP
curl  http://127.0.0.1:26657/consensus_params
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"consensus_params\"}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "block_height": "1",
    "consensus_params": {
      "block": {
        "max_bytes": "22020096",
        "max_gas": "1000",
        "time_iota_ms": "1000"
      },
      "evidence": {
        "max_age": "100000"
      },
      "validator": {
        "pub_key_types": [
          "ed25519"
        ]
      }
    }
  }
}

UnconfirmedTxs

Get a list of unconfirmed transactions.

Parameters

  • limit (integer) The amount of txs to respond with.

Request

HTTP
curl  http://127.0.0.1:26657/unconfirmed_txs
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"unconfirmed_txs\, \"params\":{\"limit\":\"20\"}}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "n_txs": "82",
    "total": "82",
    "total_bytes": "19974",
    "txs": [
      "gAPwYl3uCjCMTXENChSMnIkb5ZpYHBKIZqecFEV2tuZr7xIUA75/FmYq9WymsOBJ0XSJ8yV8zmQKMIxNcQ0KFIyciRvlmlgcEohmp5wURXa25mvvEhQbrvwbvlNiT+Yjr86G+YQNx7kRVgowjE1xDQoUjJyJG+WaWBwSiGannBRFdrbma+8SFK2m+1oxgILuQLO55n8mWfnbIzyPCjCMTXENChSMnIkb5ZpYHBKIZqecFEV2tuZr7xIUQNGfkmhTNMis4j+dyMDIWXdIPiYKMIxNcQ0KFIyciRvlmlgcEohmp5wURXa25mvvEhS8sL0D0wwgGCItQwVowak5YB38KRIUCg4KBXVhdG9tEgUxMDA1NBDoxRgaagom61rphyECn8x7emhhKdRCB2io7aS/6Cpuq5NbVqbODmqOT3jWw6kSQKUresk+d+Gw0BhjiggTsu8+1voW+VlDCQ1GRYnMaFOHXhyFv7BCLhFWxLxHSAYT8a5XqoMayosZf9mANKdXArA="
    ]
  }
}

NumUnconfirmedTxs

Get data about unconfirmed transactions.

Parameters

None

Request

HTTP
curl  http://127.0.0.1:26657/num_unconfirmed_txs
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"num_unconfirmed_txs\"}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "n_txs": "31",
    "total": "82",
    "total_bytes": "19974"
  }
}

Tx

Parameters

  • hash (string): The hash of the transaction
  • prove (bool): If the response should include proof the transaction was included in a block.

Request

HTTP
curl  http://127.0.0.1:26657/num_unconfirmed_txs
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"num_unconfirmed_txs\"}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "hash": "D70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED",
    "height": "1000",
    "index": 0,
    "tx_result": {
      "log": "[{\"msg_index\":\"0\",\"success\":true,\"log\":\"\"}]",
      "gas_wanted": "200000",
      "gas_used": "28596",
      "tags": [
        {
          "key": "YWN0aW9u",
          "value": "c2VuZA==",
          "index": false
        }
      ]
    },
    "tx": "5wHwYl3uCkaoo2GaChQmSIu8hxpJxLcCuIi8fiHN4TMwrRIU/Af1cEG7Rcs/6LjTl7YjRSymJfYaFAoFdWF0b20SCzE0OTk5OTk1MDAwEhMKDQoFdWF0b20SBDUwMDAQwJoMGmoKJuta6YchAwswBShaB1wkZBctLIhYqBC3JrAI28XGzxP+rVEticGEEkAc+khTkKL9CDE47aDvjEHvUNt+izJfT4KVF2v2JkC+bmlH9K08q3PqHeMI9Z5up+XMusnTqlP985KF+SI5J3ZOIhhNYWRlIGJ5IENpcmNsZSB3aXRoIGxvdmU="
  }
}

Transaction Routes

BroadCastTxSync

Returns with the response from CheckTx. Does not wait for DeliverTx result.

Parameters

  • tx (string): The transaction encoded

Request

HTTP
curl  http://127.0.0.1:26657/broadcast_tx_sync?tx=encoded_tx
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"broadcast_tx_sync\",\"params\":{\"tx\":\"a/encoded_tx/c\"}}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "code": "0",
    "data": "",
    "log": "",
    "codespace": "ibc",
    "hash": "0D33F2F03A5234F38706E43004489E061AC40A2E"
  },
  "error": ""
}

BroadCastTxAsync

Returns right away, with no response. Does not wait for CheckTx nor DeliverTx results.

Parameters

  • tx (string): The transaction encoded

Request

HTTP
curl  http://127.0.0.1:26657/broadcast_tx_async?tx=encoded_tx
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"broadcast_tx_async\",\"params\":{\"tx\":\"a/encoded_tx/c\"}}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "code": "0",
    "data": "",
    "log": "",
    "codespace": "ibc",
    "hash": "0D33F2F03A5234F38706E43004489E061AC40A2E"
  },
  "error": ""
}

CheckTx

Checks the transaction without executing it.

Parameters

  • tx (string): String of the encoded transaction

Request

HTTP
curl  http://127.0.0.1:26657/check_tx?tx=encoded_tx
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"check_tx\",\"params\":{\"tx\":\"a/encoded_tx/c\"}}"

Response

{
  "id": 0,
  "jsonrpc": "2.0",
  "error": "",
  "result": {
    "code": "0",
    "data": "",
    "log": "",
    "info": "",
    "gas_wanted": "1",
    "gas_used": "0",
    "events": [
      {
        "type": "app",
        "attributes": [
          {
            "key": "YWN0aW9u",
            "value": "c2VuZA==",
            "index": false
          }
        ]
      }
    ],
    "codespace": "bank"
  }
}

ABCI Routes

ABCIInfo

Get some info about the application.

Parameters

None

Request

HTTP
curl  http://127.0.0.1:26657/abci_info
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"abci_info\"}"

Response

{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "response": {
      "data": "{\"size\":0}",
      "version": "0.16.1",
      "app_version": "1314126"
    }
  }
}

ABCIQuery

Query the application for some information.

Parameters

  • path (string): Path to the data. This is defined by the application.
  • data (string): The data requested
  • height (integer): Height at which the data is being requested for.
  • prove (bool): Include proofs of the transactions inclusion in the block

Request

HTTP
curl  http://127.0.0.1:26657/abci_query?path="a/b/c"=IHAVENOIDEA&height=1&prove=true
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"abci_query\",\"params\":{\"path\":\"a/b/c\", \"height\":\"1\", \"bool\":\"true\"}}"

Response

{
  "error": "",
  "result": {
    "response": {
      "log": "exists",
      "height": "0",
      "proof": "010114FED0DAD959F36091AD761C922ABA3CBF1D8349990101020103011406AA2262E2F448242DF2C2607C3CDC705313EE3B0001149D16177BC71E445476174622EA559715C293740C",
      "value": "61626364",
      "key": "61626364",
      "index": "-1",
      "code": "0"
    }
  },
  "id": 0,
  "jsonrpc": "2.0"
}

Evidence Routes

BroadcastEvidence

Broadcast evidence of the misbehavior.

Parameters

  • evidence (string):

Request

HTTP
curl http://localhost:26657/broadcast_evidence?evidence=JSON_EVIDENCE_encoded

JSONRPC

curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"broadcast_evidence\",\"params\":{\"evidence\":\"JSON_EVIDENCE_encoded\"}}"

Response

{
  "error": "",
  "result": "",
  "id": 0,
  "jsonrpc": "2.0"
}

Light Client Specification

Historical design material. These Tendermint light-client designs and formal models are retained as reference material.

This directory contains work-in-progress English and TLA+ specifications for the Light Client protocol. Implementations of the light client can be found in Rust and Go.

Light clients are assumed to be initialized once from a trusted source with a trusted header and validator set. The light client protocol allows a client to then securely update its trusted state by requesting and verifying a minimal set of data from a network of full nodes (at least one of which is correct).

The light client is decomposed into two main components:

  • Commit Verification - verify signed headers and associated validator set changes from a single full node, called primary
  • Attack Detection - verify commits across multiple full nodes (called secondaries) and detect conflicts (ie. the existence of a lightclient attack)

In case a lightclient attack is detected, the lightclient submits evidence to a full node which is responsible for "accountability", that is, punishing attackers:

  • Accountability - given evidence for an attack, compute a set of validators that are responsible for it.

Commit Verification

The English specification describes the light client commit verification problem in terms of the temporal properties LCV-DIST-SAFE.1 and LCV-DIST-LIVE.1. Commit verification is assumed to operate within the Tendermint Failure Model, where +2/3 of validators are correct for some time period and validator sets can change arbitrarily at each height.

A light client protocol is also provided, including all checks that need to be performed on headers, commits, and validator sets to satisfy the temporal properties - so a light client can continuously synchronize with a blockchain. Clients can skip possibly many intermediate headers by exploiting overlap in trusted and untrusted validator sets. When there is not enough overlap, a bisection routine can be used to find a minimal set of headers that do provide the required overlap.

The TLA+ specification ver. 001 is a formal description of the commit verification protocol executed by a client, including the safety and termination, which can be model checked with Apalache.

A more detailed TLA+ specification of Light client verification ver. 003 is currently under peer review.

The MC*.tla files contain concrete parameters for the TLA+ specification, in order to do model checking. For instance, MC4_3_faulty.tla contains the following parameters for the nodes, heights, the trusting period, the clock drifts, correctness of the primary node, and the ratio of the faulty processes:

AllNodes == {"n1", "n2", "n3", "n4"}
TRUSTED_HEIGHT == 1
TARGET_HEIGHT == 3
TRUSTING_PERIOD == 1400     \* the trusting period in some time units
CLOCK_DRIFT = 10            \* how much we assume the local clock is drifting
REAL_CLOCK_DRIFT = 3        \* how much the local clock is actually drifting
IS_PRIMARY_CORRECT == FALSE
FAULTY_RATIO == <<1, 3>>    \* < 1 / 3 faulty validators

To run a complete set of experiments, clone apalache and apalache-tests into a directory $DIR and run the following commands:

$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 002bmc-apalache-ok.csv $DIR/apalache . out
./out/run-all.sh

After the experiments have finished, you can collect the logs by executing the following command:

cd ./out
$DIR/apalache-tests/scripts/parse-logs.py --human .

All lines in results.csv should report Deadlock, which means that the algorithm has terminated and no invariant violation was found.

Similar to 002bmc-apalache-ok.csv, file 003bmc-apalache-error.csv specifies the set of experiments that should result in counterexamples:

$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 003bmc-apalache-error.csv $DIR/apalache . out
./out/run-all.sh

All lines in results.csv should report Error.

The following table summarizes the experimental results for Light client verification version 001. The TLA+ properties can be found in the TLA+ specification. The experiments were run in an AWS instance equipped with 32GB RAM and a 4-core Intel® Xeon® CPU E5-2686 v4 @ 2.30GHz CPU. We write “✗=k” when a bug is reported at depth k, and “✓<=k” when no bug is reported up to depth k.

Experimental results

The experimental results for version 003 are to be added.

Attack Detection

The English specification defines light client attacks (and how they differ from blockchain forks), and describes the problem of a light client detecting these attacks by communicating with a network of full nodes, where at least one is correct.

The specification also contains a detection protocol that checks whether the header obtained from the primary via the verification protocol matches corresponding headers provided by the secondaries. If this is not the case, the protocol analyses the verification traces of the involved full nodes and generates evidence of misbehavior that can be submitted to a full node so that the faulty validators can be punished.

The TLA+ specification is a formal description of the detection protocol for two peers, including the safety and termination, which can be model checked with Apalache.

The LCD_MC*.tla files contain concrete parameters for the TLA+ specification, in order to run the model checker. For instance, LCD_MC4_4_faulty.tla contains the following parameters for the nodes, heights, the trusting period, the clock drifts, correctness of the nodes, and the ratio of the faulty processes:

AllNodes == {"n1", "n2", "n3", "n4"}
TRUSTED_HEIGHT == 1
TARGET_HEIGHT == 3
TRUSTING_PERIOD == 1400     \* the trusting period in some time units
CLOCK_DRIFT = 10            \* how much we assume the local clock is drifting
REAL_CLOCK_DRIFT = 3        \* how much the local clock is actually drifting
IS_PRIMARY_CORRECT == FALSE
IS_SECONDARY_CORRECT == FALSE
FAULTY_RATIO == <<1, 3>>    \* < 1 / 3 faulty validators

To run a complete set of experiments, clone apalache and apalache-tests into a directory $DIR and run the following commands:

$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 004bmc-apalache-ok.csv $DIR/apalache . out
./out/run-all.sh

After the experiments have finished, you can collect the logs by executing the following command:

cd ./out
$DIR/apalache-tests/scripts/parse-logs.py --human .

All lines in results.csv should report Deadlock, which means that the algorithm has terminated and no invariant violation was found.

Similar to 004bmc-apalache-ok.csv, file 005bmc-apalache-error.csv specifies the set of experiments that should result in counterexamples:

$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 005bmc-apalache-error.csv $DIR/apalache . out
./out/run-all.sh

All lines in results.csv should report Error.

The detailed experimental results are to be added soon.

Accountability

The English specification defines the protocol that is executed on a full node upon receiving attack evidence from a lightclient. In particular, the protocol handles three types of attacks

  • lunatic
  • equivocation
  • amnesia

We discussed in the last part of the English specification that the non-lunatic cases are defined by having the same validator set in the conflicting blocks. For these cases, computer-aided analysis of Tendermint Consensus in TLA+ shows that equivocation and amnesia capture all non-lunatic attacks.

The TLA+ specification is a formal description of the protocol, including the safety property, which can be model checked with Apalache.

Similar to the other specifications, MC_5_3.tla contains concrete parameters to run the model checker. The specification can be checked within seconds.

tendermint-accountability

Fork accountability

Problem Statement

Tendermint consensus guarantees the following specifications for all heights:

  • agreement -- no two correct full nodes decide differently.
  • validity -- the decided block satisfies the predefined predicate valid().
  • termination -- all correct full nodes eventually decide,

If the faulty validators have less than 1/3 of voting power in the current validator set. In the case where this assumption does not hold, each of the specification may be violated.

The agreement property says that for a given height, any two correct validators that decide on a block for that height decide on the same block. That the block was indeed generated by the blockchain, can be verified starting from a trusted (genesis) block, and checking that all subsequent blocks are properly signed.

However, faulty nodes may forge blocks and try to convince users (light clients) that the blocks had been correctly generated. In addition, Tendermint agreement might be violated in the case where 1/3 or more of the voting power belongs to faulty validators: Two correct validators decide on different blocks. The latter case motivates the term "fork": as Tendermint consensus also agrees on the next validator set, correct validators may have decided on disjoint next validator sets, and the chain branches into two or more partitions (possibly having faulty validators in common) and each branch continues to generate blocks independently of the other.

We say that a fork is a case in which there are two commits for different blocks at the same height of the blockchain. The problem is to ensure that in those cases we are able to detect faulty validators (and not mistakenly accuse correct validators), and incentivize therefore validators to behave according to the protocol specification.

Conceptual Limit. In order to prove misbehavior of a node, we have to show that the behavior deviates from correct behavior with respect to a given algorithm. Thus, an algorithm that detects misbehavior of nodes executing some algorithm A must be defined with respect to algorithm A. In our case, A is Tendermint consensus (+ other protocols in the infrastructure; e.g.,full nodes and the Light Client). If the consensus algorithm is changed/updated/optimized in the future, we have to check whether changes to the accountability algorithm are also required. All the discussions in this document are thus inherently specific to Tendermint consensus and the Light Client specification.

Q: Should we distinguish agreement for validators and full nodes for agreement? The case where all correct validators agree on a block, but a correct full node decides on a different block seems to be slightly less severe that the case where two correct validators decide on different blocks. Still, if a contaminated full node becomes validator that may be problematic later on. Also it is not clear how gossiping is impaired if a contaminated full node is on a different branch.

Remark. In the case 1/3 or more of the voting power belongs to faulty validators, also validity and termination can be broken. Termination can be broken if faulty processes just do not send the messages that are needed to make progress. Due to asynchrony, this is not punishable, because faulty validators can always claim they never received the messages that would have forced them to send messages.

The Misbehavior of Faulty Validators

Forks are the result of faulty validators deviating from the protocol. In principle several such deviations can be detected without a fork actually occurring:

  1. double proposal: A faulty proposer proposes two different values (blocks) for the same height and the same round in Tendermint consensus.

  2. double signing: Tendermint consensus forces correct validators to prevote and precommit for at most one value per round. In case a faulty validator sends multiple prevote and/or precommit messages for different values for the same height/round, this is a misbehavior.

  3. lunatic validator: Tendermint consensus forces correct validators to prevote and precommit only for values v that satisfy valid(v). If faulty validators prevote and precommit for v although valid(v)=false this is misbehavior.

Remark. In isolation, Point 3 is an attack on validity (rather than agreement). However, the prevotes and precommits can then also be used to forge blocks.

  1. amnesia: Tendermint consensus has a locking mechanism. If a validator has some value v locked, then it can only prevote/precommit for v or nil. Sending prevote/precomit message for a different value v' (that is not nil) while holding lock on value v is misbehavior.

  2. spurious messages: In Tendermint consensus most of the message send instructions are guarded by threshold guards, e.g., one needs to receive 2f + 1 prevote messages to send precommit. Faulty validators may send precommit without having received the prevote messages.

Independently of a fork happening, punishing this behavior might be important to prevent forks altogether. This should keep attackers from misbehaving: if less than 1/3 of the voting power is faulty, this misbehavior is detectable but will not lead to a safety violation. Thus, unless they have 1/3 or more (or in some cases more than 2/3) of the voting power attackers have the incentive to not misbehave. If attackers control too much voting power, we have to deal with forks, as discussed in this document.

Two types of forks

  • Fork-Full. Two correct validators decide on different blocks for the same height. Since also the next validator sets are decided upon, the correct validators may be partitioned to participate in two distinct branches of the forked chain.

As in this case we have two different blocks (both having the same right/no right to exist), a central system invariant (one block per height decided by correct validators) is violated. As full nodes are contaminated in this case, the contamination can spread also to light clients. However, even without breaking this system invariant, light clients can be subject to a fork:

  • Fork-Light. All correct validators decide on the same block for height h, but faulty processes (validators or not), forge a different block for that height, in order to fool users (who use the light client).

Attack scenarios

On-chain attacks

Equivocation (one round)

There are several scenarios in which forks might happen. The first is double signing within a round.

  • F1. Equivocation: faulty validators sign multiple vote messages (prevote and/or precommit) for different values during the same round r at a given height h.

Flip-flopping

Tendermint consensus implements a locking mechanism: If a correct validator p receives proposal for value v and 2f + 1 prevotes for a value id(v) in round r, it locks v and remembers r. In this case, p also sends a precommit message for id(v), which later may serve as proof that p locked v. In subsequent rounds, p only sends prevote messages for a value it had previously locked. However, it is possible to change the locked value if in a future round r' > r, if the process receives proposal and 2f + 1 prevotes for a different value v'. In this case, p could send a prevote/precommit for id(v'). This algorithmic feature can be exploited in two ways:

  • F2. Faulty Flip-flopping (Amnesia): faulty validators precommit some value id(v) in round r (value v is locked in round r) and then prevote for different value id(v') in higher round r' > r without previously correctly unlocking value v. In this case faulty processes "forget" that they have locked value v and prevote some other value in the following rounds. Some correct validators might have decided on v in r, and other correct validators decide on v' in r'. Here we can have branching on the main chain (Fork-Full).

  • F3. Correct Flip-flopping (Back to the past): There are some precommit messages signed by (correct) validators for value id(v) in round r. Still, v is not decided upon, and all processes move on to the next round. Then correct validators (correctly) lock and decide a different value v' in some round r' > r. And the correct validators continue; there is no branching on the main chain. However, faulty validators may use the correct precommit messages from round r together with a posteriori generated faulty precommit messages for round r to forge a block for a value that was not decided on the main chain (Fork-Light).

Off-chain attacks

F1-F3 may contaminate the state of full nodes (and even validators). Contaminated (but otherwise correct) full nodes may thus communicate faulty blocks to light clients. Similarly, without actually interfering with the main chain, we can have the following:

  • F4. Phantom validators: faulty validators vote (sign prevote and precommit messages) in heights in which they are not part of the validator sets (at the main chain).

  • F5. Lunatic validator: faulty validator that sign vote messages to support (arbitrary) application state that is different from the application state that resulted from valid state transitions.

Types of victims

We consider three types of potential attack victims:

  • FN: full node
  • LCS: light client with sequential header verification
  • LCB: light client with bisection based header verification

F1 and F2 can be used by faulty validators to actually create multiple branches on the blockchain. That means that correctly operating full nodes decide on different blocks for the same height. Until a fork is detected locally by a full node (by receiving evidence from others or by some other local check that fails), the full node can spread corrupted blocks to light clients.

Remark. If full nodes take a branch different from the one taken by the validators, it may be that the liveness of the gossip protocol may be affected. We should eventually look at this more closely. However, as it does not influence safety it is not a primary concern.

F3 is similar to F1, except that no two correct validators decide on different blocks. It may still be the case that full nodes become affected.

In addition, without creating a fork on the main chain, light clients can be contaminated by more than a third of validators that are faulty and sign a forged header F4 cannot fool correct full nodes as they know the current validator set. Similarly, LCS know who the validators are. Hence, F4 is an attack against LCB that do not necessarily know the complete prefix of headers (Fork-Light), as they trust a header that is signed by at least one correct validator (trusting period method).

The following table gives an overview of how the different attacks may affect different nodes. F1-F3 are on-chain attacks so they can corrupt the state of full nodes. Then if a light client (LCS or LCB) contacts a full node to obtain headers (or blocks), the corrupted state may propagate to the light client.

F4 and F5 are off-chain, that is, these attacks cannot be used to corrupt the state of full nodes (which have sufficient knowledge on the state of the chain to not be fooled).

AttackFNLCSLCB
F1directFNFN
F2directFNFN
F3directFNFN
F4direct
F5direct

Q: Light clients are more vulnerable than full nodes, because the former do only verify headers but do not execute transactions. What kind of certainty is gained by a full node that executes a transaction?

As a full node verifies all transactions, it can only be contaminated by an attack if the blockchain itself violates its invariant (one block per height), that is, in case of a fork that leads to branching.

Detailed Attack Scenarios

Equivocation based attacks

In case of equivocation based attacks, faulty validators sign multiple votes (prevote and/or precommit) in the same round of some height. This attack can be executed on both full nodes and light clients. It requires 1/3 or more of voting power to be executed.

Scenario 1: Equivocation on the main chain

Validators:

  • CA - a set of correct validators with less than 1/3 of the voting power
  • CB - a set of correct validators with less than 1/3 of the voting power
  • CA and CB are disjoint
  • F - a set of faulty validators with 1/3 or more voting power

Observe that this setting violates the Tendermint failure model.

Execution:

  • A faulty proposer proposes block A to CA
  • A faulty proposer proposes block B to CB
  • Validators from the set CA and CB prevote for A and B, respectively.
  • Faulty validators from the set F prevote both for A and B.
  • The faulty prevote messages
    • for A arrive at CA long before the B messages
    • for B arrive at CB long before the A messages
  • Therefore correct validators from set CA and CB will observe more than 2/3 of prevotes for A and B and precommit for A and B, respectively.
  • Faulty validators from the set F precommit both values A and B.
  • Thus, we have more than 2/3 commits for both A and B.

Consequences:

  • Creating evidence of misbehavior is simple in this case as we have multiple messages signed by the same faulty processes for different values in the same round.

  • We have to ensure that these different messages reach a correct process (full node, monitor?), which can submit evidence.

  • This is an attack on the full node level (Fork-Full).

  • It extends also to the light clients,

  • For both we need a detection and recovery mechanism.

Scenario 2: Equivocation to a light client (LCS)

Validators:

  • a set F of faulty validators with more than 2/3 of the voting power.

Execution:

  • for the main chain F behaves nicely
  • F coordinates to sign a block B that is different from the one on the main chain.
  • the light clients obtains B and trusts at as it is signed by more than 2/3 of the voting power.

Consequences:

Once equivocation is used to attack light client it opens space for different kind of attacks as application state can be diverged in any direction. For example, it can modify validator set such that it contains only validators that do not have any stake bonded. Note that after a light client is fooled by a fork, that means that an attacker can change application state and validator set arbitrarily.

In order to detect such (equivocation-based attack), the light client would need to cross check its state with some correct validator (or to obtain a hash of the state from the main chain using out of band channels).

Remark. The light client would be able to create evidence of misbehavior, but this would require to pull potentially a lot of data from correct full nodes. Maybe we need to figure out different architecture where a light client that is attacked will push all its data for the current unbonding period to a correct node that will inspect this data and submit corresponding evidence. There are also architectures that assumes a special role (sometimes called fisherman) whose goal is to collect as much as possible useful data from the network, to do analysis and create evidence transactions. That functionality is outside the scope of this document.

Remark. The difference between LCS and LCB might only be in the amount of voting power needed to convince light client about arbitrary state. In case of LCB where security threshold is at minimum, an attacker can arbitrarily modify application state with 1/3 or more of voting power, while in case of LCS it requires more than 2/3 of the voting power.

Flip-flopping: Amnesia based attacks

In case of amnesia, faulty validators lock some value v in some round r, and then vote for different value v' in higher rounds without correctly unlocking value v. This attack can be used both on full nodes and light clients.

Scenario 3: At most 2/3 of faults

Validators:

  • a set F of faulty validators with 1/3 or more but at most 2/3 of the voting power
  • a set C of correct validators

Execution:

  • Faulty validators commit (without exposing it on the main chain) a block A in round r by collecting more than 2/3 of the voting power (containing correct and faulty validators).
  • All validators (correct and faulty) reach a round r' > r.
  • Some correct validators in C do not lock any value before round r'.
  • The faulty validators in F deviate from Tendermint consensus by ignoring that they locked A in r, and propose a different block B in r'.
  • As the validators in C that have not locked any value find B acceptable, they accept the proposal for B and commit a block B.

Remark. In this case, the more than 1/3 of faulty validators do not need to commit an equivocation (F1) as they only vote once per round in the execution.

Detecting faulty validators in the case of such an attack can be done by the fork accountability mechanism described in the fork accountability report (a Google Doc that is no longer available).

If a light client is attacked using this attack with 1/3 or more of voting power (and less than 2/3), the attacker cannot change the application state arbitrarily. Rather, the attacker is limited to a state a correct validator finds acceptable: In the execution above, correct validators still find the value acceptable, however, the block the light client trusts deviates from the one on the main chain.

Scenario 4: More than 2/3 of faults

In case there is an attack with more than 2/3 of the voting power, an attacker can arbitrarily change application state.

Validators:

  • a set F1 of faulty validators with 1/3 or more of the voting power
  • a set F2 of faulty validators with less than 1/3 of the voting power

Execution

  • Similar to Scenario 3 (however, messages by correct validators are not needed)
  • The faulty validators in F1 lock value A in round r
  • They sign a different value in follow-up rounds
  • F2 does not lock A in round r

Consequences:

  • The validators in F1 will be detectable by the the fork accountability mechanisms.
  • The validators in F2 cannot be detected using this mechanism. Only in case they signed something which conflicts with the application this can be used against them. Otherwise they do not do anything incorrect.
  • This case is not covered by the fork accountability report as it only assumes at most 2/3 of faulty validators.

Q: do we need to define a special kind of attack for the case where a validator sign arbitrarily state? It seems that detecting such attack requires a different mechanism that would require as an evidence a sequence of blocks that led to that state. This might be very tricky to implement.

Back to the past

In this kind of attack, faulty validators take advantage of the fact that they did not sign messages in some of the past rounds. Due to the asynchronous network in which Tendermint operates, we cannot easily differentiate between such an attack and delayed message. This kind of attack can be used at both full nodes and light clients.

Scenario 5

Validators:

  • C1 - a set of correct validators with over 1/3 of the voting power
  • C2 - a set of correct validators with 1/3 of the voting power
  • C1 and C2 are disjoint
  • F - a set of faulty validators with less than 1/3 voting power
  • one additional faulty process q
  • F and q violate the Tendermint failure model.

Execution:

  • in a round r of height h we have C1 precommitting a value A,
  • C2 precommits nil,
  • F does not send any message
  • q precommits nil.
  • In some round r' > r, F and q and C2 commit some other value B different from A.
  • F and fp "go back to the past" and sign precommit message for value A in round r.
  • Together with precomit messages of C1 this is sufficient for a commit for value A.

Consequences:

  • Only a single faulty validator that previously precommited nil did equivocation, while the other 1/3 of faulty validators actually executed an attack that has exactly the same sequence of messages as part of amnesia attack. Detecting this kind of attack boil down to mechanisms for equivocation and amnesia.

Q: should we keep this as a separate kind of attack? It seems that equivocation, amnesia and phantom validators are the only kind of attack we need to support and this gives us security also in other cases. This would not be surprising as equivocation and amnesia are attacks that followed from the protocol and phantom attack is not really an attack to Tendermint but more to the Proof of Stake module.

Phantom validators

In case of phantom validators, processes that are not part of the current validator set but are still bonded (as attack happen during their unbonding period) can be part of the attack by signing vote messages. This attack can be executed against both full nodes and light clients.

Scenario 6

Validators:

  • F -- a set of faulty validators that are not part of the validator set on the main chain at height h + k

Execution:

  • There is a fork, and there exist two different headers for height h + k, with different validator sets:

    • VS2 on the main chain
    • forged header VS2', signed by F (and others)
  • a light client has a trust in a header for height h (and the corresponding validator set VS1).

  • As part of bisection header verification, it verifies the header at height h + k with new validator set VS2'.

Consequences:

  • To detect this, a node needs to see both, the forged header and the canonical header from the chain.
  • If this is the case, detecting these kind of attacks is easy as it just requires verifying if processes are signing messages in heights in which they are not part of the validator set.

Remark. We can have phantom-validator-based attacks as a follow up of equivocation or amnesia based attack where forked state contains validators that are not part of the validator set at the main chain. In this case, they keep signing messages contributed to a forked chain (the wrong branch) although they are not part of the validator set on the main chain. This attack can also be used to attack full node during a period of time it is eclipsed.

Remark. Phantom validator evidence has been removed from implementation as it was deemed, although possibly a plausible form of evidence, not relevant. Any attack on the light client involving a phantom validator will have needed to be initiated by 1/3+ lunatic validators that can forge a new validator set that includes the phantom validator. Only in that case will the light client accept the phantom validators vote. We need only worry about punishing the 1/3+ lunatic cabal, that is the root cause of the attack.

Lunatic validator

Lunatic validator agrees to sign commit messages for arbitrary application state. It is used to attack light clients. Note that detecting this behavior require application knowledge. Detecting this behavior can probably be done by referring to the block before the one in which height happen.

Q: can we say that in this case a validator declines to check if a proposed value is valid before voting for it?

Synopsis

A TLA+ specification of a simplified Tendermint consensus, tuned for fork accountability. The simplifications are as follows:

  • the procotol runs for one height, that is, one-shot consensus

  • this specification focuses on safety, so timeouts are modelled with with non-determinism

  • the proposer function is non-determinstic, no fairness is assumed

  • the messages by the faulty processes are injected right in the initial states

  • every process has the voting power of 1

  • hashes are modelled as identity

Having the above assumptions in mind, the specification follows the pseudo-code of the Tendermint paper: https://arxiv.org/abs/1807.04938

Byzantine processes can demonstrate arbitrary behavior, including no communication. However, we have to show that under the collective evidence collected by the correct processes, at least f+1 Byzantine processes demonstrate one of the following behaviors:

  • Equivocation: a Byzantine process sends two different values in the same round.

  • Amnesia: a Byzantine process locks a value, although it has locked another value in the past.

TLA+ modules

Reasoning about fork scenarios

The theorem statements can be found in TendermintAccInv_004_draft.tla.

First, we would like to show that TypedInv is an inductive invariant. Formally, the statement looks as follows:

THEOREM TypedInvIsInductive ==
    \/ FaultyQuorum
    \//\ Init => TypedInv
      /\ TypedInv /\ [Next]_vars => TypedInv'

When over two-thirds of processes are faulty, TypedInv is not inductive. However, there is no hope to repair the protocol in this case. We run Apalache to prove this theorem only for fixed instances of 4 to 5 validators. Apalache does not parse theorem statements at the moment, so we ran Apalache using a shell script. To find a parameterized argument, one has to use a theorem prover, e.g., TLAPS.

Second, we would like to show that the invariant implies Agreement, that is, no fork, provided that less than one third of processes is faulty. By combining this theorem with the previous theorem, we conclude that the protocol indeed satisfies Agreement under the condition LessThanThirdFaulty.

THEOREM AgreementWhenLessThanThirdFaulty ==
    LessThanThirdFaulty /\ TypedInv => Agreement

Third, in the general case, we either have no fork, or two fork scenarios:

THEOREM AgreementOrFork ==
    ~FaultyQuorum /\ TypedInv => Accountability

Model checking results

Check the report on model checking with Apalache.

To run the model checking experiments, use the script:

./run.sh

This script assumes that the apalache build is available in ~/devl/apalache-unstable.

Results of 001indinv-apalache

1. Awesome plots

1.1. Time (logarithmic scale)

time-log

1.2. Time (linear)

time-log

1.3. Memory (logarithmic scale)

mem-log

1.4. Memory (linear)

mem

1.5. Number of arena cells (linear)

ncells

1.6. Number of SMT clauses (linear)

nclauses

2. Input parameters

nofilenametooltimeoutinitinvnextargs
1MC_n4_f1.tlaapalache10hTypedInvTypedInv--length=1 --cinit=ConstInit
2MC_n4_f2.tlaapalache10hTypedInvTypedInv--length=1 --cinit=ConstInit
3MC_n5_f1.tlaapalache10hTypedInvTypedInv--length=1 --cinit=ConstInit
4MC_n5_f2.tlaapalache10hTypedInvTypedInv--length=1 --cinit=ConstInit
5MC_n4_f1.tlaapalache20hInitTypedInv--length=0 --cinit=ConstInit
6MC_n4_f2.tlaapalache20hInitTypedInv--length=0 --cinit=ConstInit
7MC_n5_f1.tlaapalache20hInitTypedInv--length=0 --cinit=ConstInit
8MC_n5_f2.tlaapalache20hInitTypedInv--length=0 --cinit=ConstInit
9MC_n4_f1.tlaapalache20hTypedInvAgreement--length=0 --cinit=ConstInit
10MC_n4_f2.tlaapalache20hTypedInvAccountability--length=0 --cinit=ConstInit
11MC_n5_f1.tlaapalache20hTypedInvAgreement--length=0 --cinit=ConstInit
12MC_n5_f2.tlaapalache20hTypedInvAccountability--length=0 --cinit=ConstInit

3. Detailed results: 001indinv-apalache-unstable.csv

01:no02:tool03:status04:time_sec05:depth05:mem_kb10:ninit_trans11:ninit_trans12:ncells13:nclauses14:navg_clause_len
1apalacheNoError11m13.0GB00217K1.0M89
2apalacheNoError11m13.0GB00207K1.0M88
3apalacheNoError16m14.0GB00311K2.0M101
4apalacheNoError14m13.0GB00290K1.0M103
5apalacheNoError9s0563MB002.0K14K42
6apalacheNoError10s0657MB002.0K28K43
7apalacheNoError8s0635MB002.0K17K44
8apalacheNoError10s0667MB003.0K32K45
9apalacheNoError5m05s02.0GB00196K889K108
10apalacheNoError8m08s06.0GB002.0M3.0M34
11apalacheNoError9m09s03.0GB00284K1.0M128
12apalacheNoError14m07.0GB004.0M5.0M38

Lightclient Attackers Isolation

Warning: This is the beginning of an unfinished draft. Don't continue reading!

Adversarial nodes may have the incentive to lie to a lightclient about the state of a Tendermint blockchain. An attempt to do so is called attack. Light client verification checks incoming data by checking a so-called "commit", which is a forwarded set of signed messages that is (supposedly) produced during executing Tendermint consensus. Thus, an attack boils down to creating and signing Tendermint consensus messages in deviation from the Tendermint consensus algorithm rules.

As Tendermint consensus and light client verification is safe under the assumption of more than 2/3 of correct voting power per block [TMBC-FM-2THIRDS], this implies that if there was an attack then [TMBC-FM-2THIRDS] was violated, that is, there is a block such that

  • validators deviated from the protocol, and
  • these validators represent more than 1/3 of the voting power in that block.

In the case of an attack, the lightclient attack detection mechanism computes data, so called evidence [LC-DATA-EVIDENCE.1], that can be used

  • to proof that there has been attack [TMBC-LC-EVIDENCE-DATA.1] and
  • as basis to find the actual nodes that deviated from the Tendermint protocol.

This specification considers how a full node in a Tendermint blockchain can isolate a set of attackers that launched the attack. The set should satisfy

  • the set does not contain a correct validator
  • the set contains validators that represent more than 1/3 of the voting power of a block that is still within the unbonding period

Outline

TODO when preparing a version for broader review.

Part I - Basics

For definitions of data structures used here, in particular LightBlocks [LCV-DATA-LIGHTBLOCK.1], cf. Light Client Verification.

Part II - Definition of the Problem

The specification of the detection mechanism describes

  • what is a light client attack,
  • conditions under which the detector will detect a light client attack,
  • and the format of the output data, called evidence, in the case an attack is detected. The format is defined in [LC-DATA-EVIDENCE.1] and looks as follows
type LightClientAttackEvidence struct {
    ConflictingBlock   LightBlock
    CommonHeight       int64
}

The isolator is a function that gets as input evidence ev and a prefix of the blockchain bc at least up to height ev.ConflictingBlock.Header.Height + 1. The output is a set of peerIDs of validators.

We assume that the full node is synchronized with the blockchain and has reached the height ev.ConflictingBlock.Header.Height + 1.

[FN-INV-Output.1]

When an output is generated it satisfies the following properties:

  • If
    • bc[CommonHeight].bfttime is within the unbonding period w.r.t. the time at the full node,
    • ev.ConflictingBlock.Header != bc[ev.ConflictingBlock.Header.Height]
    • Validators in ev.ConflictingBlock.Commit represent more than 1/3 of the voting power in bc[ev.CommonHeight].NextValidators
  • Then: A set of validators in bc[CommonHeight].NextValidators that
    • represent more than 1/3 of the voting power in bc[ev.commonHeight].NextValidators
    • signed Tendermint consensus messages for height ev.ConflictingBlock.Header.Height by violating the Tendermint consensus protocol.
  • Else: the empty set.

Part IV - Protocol

Here we discuss how to solve the problem of isolating misbehaving processes. We describe the function isolateMisbehavingProcesses as well as all the helping functions below. In Part V, we discuss why the solution is complete based on result from analysis with automated tools.

Isolation

Outline

Describe solution (in English), decomposition into functions, where communication to other components happens.

[LCAI-FUNC-MAIN.1]

func isolateMisbehavingProcesses(ev LightClientAttackEvidence, bc Blockchain) []ValidatorAddress {

    reference := bc[ev.conflictingBlock.Header.Height].Header
    ev_header := ev.conflictingBlock.Header

    ref_commit := bc[ev.conflictingBlock.Header.Height + 1].Header.LastCommit // + 1 !!
    ev_commit := ev.conflictingBlock.Commit

    if violatesTMValidity(reference, ev_header) {
        // lunatic light client attack
        signatories := Signers(ev.ConflictingBlock.Commit)
        bonded_vals := Addresses(bc[ev.CommonHeight].NextValidators)
        return intersection(signatories,bonded_vals)

    }
    // If this point is reached the validator sets in reference and ev_header are identical
    else if RoundOf(ref_commit) == RoundOf(ev_commit) {
        // equivocation light client attack
        return intersection(Signers(ref_commit), Signers(ev_commit))
    }
    else {
        // amnesia light client attack
        return IsolateAmnesiaAttacker(ev, bc)
    }
}
  • Implementation comment
    • If the full node has only reached height ev.conflictingBlock.Header.Height then bc[ev.conflictingBlock.Header.Height + 1].Header.LastCommit refers to the locally stored commit for this height. (This commit must be present by the precondition on length(bc).)
    • We check in the precondition that the unbonding period is not expired. However, since time moves on, before handing the validators over Cosmos SDK, the time needs to be checked again to satisfy the contract which requires that only bonded validators are reported. This passing of validators to the SDK is out of scope of this specification.
  • Expected precondition
    • length(bc) >= ev.conflictingBlock.Header.Height
    • ValidAndVerifiedUnbonding(bc[ev.CommonHeight], ev.ConflictingBlock) == SUCCESS
    • ev.ConflictingBlock.Header != bc[ev.ConflictingBlock.Header.Height]
    • TODO: input light blocks pass basic validation
  • Expected postcondition
  • Error condition
    • returns an error if precondition is violated.

Details of the Functions

[LCAI-FUNC-VVU.1]

func ValidAndVerifiedUnbonding(trusted LightBlock, untrusted LightBlock) Result
  • Conditions are identical to [LCV-FUNC-VALID.2] except the precondition "trusted.Header.Time > now - trustingPeriod" is substituted with
    • trusted.Header.Time > now - UnbondingPeriod

[LCAI-FUNC-NONVALID.1]

func violatesTMValidity(ref Header, ev Header) boolean
  • Implementation remarks
    • checks whether the evidence header ev violates the validity property of Tendermint Consensus, by checking agains a reference header
  • Expected precondition
    • ref.Height == ev.Height
  • Expected postcondition
    • returns evaluation of the following disjunction
      [[LCAI-NONVALID-OUTPUT.1]] ==
      ref.ValidatorsHash != ev.ValidatorsHash or
      ref.NextValidatorsHash != ev.NextValidatorsHash or
      ref.ConsensusHash != ev.ConsensusHash or
      ref.AppHash != ev.AppHash or
      ref.LastResultsHash != ev.LastResultsHash
func IsolateAmnesiaAttacker(ev LightClientAttackEvidence, bc Blockchain) []ValidatorAddress
  • Implementation remarks TODO: What should we do here? Refer to the accountability doc?
  • Expected postcondition TODO: What should we do here? Refer to the accountability doc?
func RoundOf(commit Commit) []ValidatorAddress
  • Expected precondition
    • commit is well-formed. In particular all votes are from the same round r.
  • Expected postcondition
    • returns round r that is encoded in all the votes of the commit
func Signers(commit Commit) []ValidatorAddress
  • Expected postcondition
    • returns all validator addresses in commit
func Addresses(vals Validator[]) ValidatorAddress[]
  • Expected postcondition
    • returns all validator addresses in vals

Part V - Completeness

As discussed in the beginning of this document, an attack boils down to creating and signing Tendermint consensus messages in deviation from the Tendermint consensus algorithm rules. The main function isolateMisbehavingProcesses distinguishes three kinds of wrongly signing messages, namely,

  • lunatic: signing invalid blocks
  • equivocation: double-signing valid blocks in the same consensus round
  • amnesia: signing conflicting blocks in different consensus rounds, without having seen a quorum of messages that would have allowed to do so.

The question is whether this captures all attacks. First observe that the first checking in isolateMisbehavingProcesses is violatesTMValidity. It takes care of lunatic attacks. If this check passes, that is, if violatesTMValidity returns FALSE this means that [FN-NONVALID-OUTPUT] evaluates to false, which implies that ref.ValidatorsHash = ev.ValidatorsHash. Hence after violatesTMValidity, all the involved validators are the ones from the blockchain. It is thus sufficient to analyze one instance of Tendermint consensus with a fixed group membership (set of validators). Also it is sufficient to consider two different valid consensus values, that is, binary consensus.

TODO we have analyzed Tendermint consensus with TLA+ and have accompanied Galois in an independent study of the protocol based on Ivy proofs.

References

[supervisor] The specification of the light client supervisor.

[verification] The specification of the light client verification protocol

[detection] The specification of the light client attack detection mechanism.

Lightclient Attackers Isolation

Adversarial nodes may have the incentive to lie to a lightclient about the state of a Tendermint blockchain. An attempt to do so is called attack. Light client verification checks incoming data by checking a so-called "commit", which is a forwarded set of signed messages that is (supposedly) produced during executing Tendermint consensus. Thus, an attack boils down to creating and signing Tendermint consensus messages in deviation from the Tendermint consensus algorithm rules.

As Tendermint consensus and light client verification is safe under the assumption of more than 2/3 of correct voting power per block [TMBC-FM-2THIRDS], this implies that if there was an attack then [TMBC-FM-2THIRDS] was violated, that is, there is a block such that

  • validators deviated from the protocol, and
  • these validators represent more than 1/3 of the voting power in that block.

In the case of an attack, the lightclient attack detection mechanism computes data, so called evidence [LC-DATA-EVIDENCE.1], that can be used

  • to proof that there has been attack [TMBC-LC-EVIDENCE-DATA.1] and
  • as basis to find the actual nodes that deviated from the Tendermint protocol.

This specification considers how a full node in a Tendermint blockchain can isolate a set of attackers that launched the attack. The set should satisfy

  • the set does not contain a correct validator
  • the set contains validators that represent more than 1/3 of the voting power of a block that is still within the unbonding period

Outline

After providing the problem statement, we specify the isolator function and close with the discussion about its correctness which is based on computer-aided analysis of Tendermint Consensus.

Part I - Basics and Definition of the Problem

For definitions of data structures used here, in particular LightBlocks [LCV-DATA-LIGHTBLOCK.1], we refer to the specification of Light Client Verification.

The specification of the detection mechanism describes

  • what is a light client attack,
  • conditions under which the detector will detect a light client attack,
  • and the format of the output data, called evidence, in the case an attack is detected. The format is defined in [LC-DATA-EVIDENCE.1] and looks as follows
type LightClientAttackEvidence struct {
    ConflictingBlock   LightBlock
    CommonHeight       int64
}

The isolator is a function that gets as input evidence ev and a prefix of the blockchain bc at least up to height ev.ConflictingBlock.Header.Height + 1. The output is a set of peerIDs of validators.

We assume that the full node is synchronized with the blockchain and has reached the height ev.ConflictingBlock.Header.Height + 1.

[LCAI-INV-Output.1]

When an output is generated it satisfies the following properties:

  • If
    • bc[CommonHeight].bfttime is within the unbonding period w.r.t. the time at the full node,
    • ev.ConflictingBlock.Header != bc[ev.ConflictingBlock.Header.Height]
    • Validators in ev.ConflictingBlock.Commit represent more than 1/3 of the voting power in bc[ev.CommonHeight].NextValidators
  • Then: The output is a set of validators in bc[CommonHeight].NextValidators that
    • represent more than 1/3 of the voting power in bc[ev.commonHeight].NextValidators
    • signed Tendermint consensus messages for height ev.ConflictingBlock.Header.Height by violating the Tendermint consensus protocol.
  • Else: the empty set.

Part II - Protocol

Here we discuss how to solve the problem of isolating misbehaving processes. We describe the function isolateMisbehavingProcesses as well as all the helping functions below. In Part III, we discuss why the solution is complete based on result from analysis with automated tools.

Isolation

Outline

We first check whether the conflicting block can indeed be verified from the common height. We then first check whether it was a lunatic attack (violating validity). If this is not the case, we check for equivocation. If this also is not the case, we start the on-chain accountability protocol.

[LCAI-FUNC-MAIN.1]

func isolateMisbehavingProcesses(ev LightClientAttackEvidence, bc Blockchain) []ValidatorAddress {

    reference := bc[ev.conflictingBlock.Header.Height].Header
    ev_header := ev.conflictingBlock.Header

    ref_commit := bc[ev.conflictingBlock.Header.Height + 1].Header.LastCommit // + 1 !!
    ev_commit := ev.conflictingBlock.Commit

    if violatesTMValidity(reference, ev_header) {
        // lunatic light client attack
        signatories := Signers(ev.ConflictingBlock.Commit)
        bonded_vals := Addresses(bc[ev.CommonHeight].NextValidators)
        return intersection(signatories,bonded_vals)

    }
    // If this point is reached the validator sets in reference and ev_header are identical
    else if RoundOf(ref_commit) == RoundOf(ev_commit) {
        // equivocation light client attack
        return intersection(Signers(ref_commit), Signers(ev_commit))
    }
    else {
        // amnesia light client attack
        return IsolateAmnesiaAttacker(ev, bc)
    }
}
  • Implementation comment
    • If the full node has only reached height ev.conflictingBlock.Header.Height then bc[ev.conflictingBlock.Header.Height + 1].Header.LastCommit refers to the locally stored commit for this height. (This commit must be present by the precondition on length(bc).)
    • We check in the precondition that the unbonding period is not expired. However, since time moves on, before handing the validators over Cosmos SDK, the time needs to be checked again to satisfy the contract which requires that only bonded validators are reported. This passing of validators to the SDK is out of scope of this specification.
  • Expected precondition
    • length(bc) >= ev.conflictingBlock.Header.Height
    • ValidAndVerifiedUnbonding(bc[ev.CommonHeight], ev.ConflictingBlock) == SUCCESS
    • ev.ConflictingBlock.Header != bc[ev.ConflictingBlock.Header.Height]
    • ev.conflictingBlock satisfies basic validation (in particular all signed messages in the Commit are from the same round)
  • Expected postcondition
  • Error condition
    • returns an error if precondition is violated.

Details of the Functions

[LCAI-FUNC-VVU.1]

func ValidAndVerifiedUnbonding(trusted LightBlock, untrusted LightBlock) Result
  • Conditions are identical to [LCV-FUNC-VALID.2] except the precondition "trusted.Header.Time > now - trustingPeriod" is substituted with
    • trusted.Header.Time > now - UnbondingPeriod

[LCAI-FUNC-NONVALID.1]

func violatesTMValidity(ref Header, ev Header) boolean
  • Implementation remarks
    • checks whether the evidence header ev violates the validity property of Tendermint Consensus, by checking against a reference header
  • Expected precondition
    • ref.Height == ev.Height
  • Expected postcondition
    • returns evaluation of the following disjunction
      [LCAI-NONVALID-OUTPUT.1] ==
      ref.ValidatorsHash != ev.ValidatorsHash or
      ref.NextValidatorsHash != ev.NextValidatorsHash or
      ref.ConsensusHash != ev.ConsensusHash or
      ref.AppHash != ev.AppHash or
      ref.LastResultsHash != ev.LastResultsHash
func IsolateAmnesiaAttacker(ev LightClientAttackEvidence, bc Blockchain) []ValidatorAddress
  • Implementation remarks
  • Expected postcondition
    • returns attackers according to [LCAI-INV-Output.1].
func RoundOf(commit Commit) []ValidatorAddress
  • Expected precondition
    • commit is well-formed. In particular all votes are from the same round r.
  • Expected postcondition
    • returns round r that is encoded in all the votes of the commit
  • Error condition
    • reports error if precondition is violated
func Signers(commit Commit) []ValidatorAddress
  • Expected postcondition
    • returns all validator addresses in commit
func Addresses(vals Validator[]) ValidatorAddress[]
  • Expected postcondition
    • returns all validator addresses in vals

Part III - Completeness

As discussed in the beginning of this document, an attack boils down to creating and signing Tendermint consensus messages in deviation from the Tendermint consensus algorithm rules. The main function isolateMisbehavingProcesses distinguishes three kinds of wrongly signed messages, namely,

  • lunatic: signing invalid blocks
  • equivocation: double-signing valid blocks in the same consensus round
  • amnesia: signing conflicting blocks in different consensus rounds, without having seen a quorum of messages that would have allowed to do so.

The question is whether this captures all attacks. First observe that the first check in isolateMisbehavingProcesses is violatesTMValidity. It takes care of lunatic attacks. If this check passes, that is, if violatesTMValidity returns FALSE this means that [LCAI-NONVALID-OUTPUT.1] evaluates to false, which implies that ref.ValidatorsHash = ev.ValidatorsHash. Hence, after violatesTMValidity, all the involved validators are the ones from the blockchain. It is thus sufficient to analyze one instance of Tendermint consensus with a fixed group membership (set of validators). Also, as we have two different blocks for the same height, it is sufficient to consider two different valid consensus values, that is, binary consensus.

For this fixed group membership, we have analyzed the attacks using the TLA+ specification of Tendermint Consensus in TLA+. We checked that indeed the only possible scenarios that can lead to violation of agreement are equivocation and amnesia. An independent study by Galois of the protocol based on Ivy proofs led to the same conclusion.

References

[supervisor] The specification of the light client supervisor.

[verification] The specification of the light client verification protocol.

[detection] The specification of the light client attack detection mechanism.

[tendermint-accountability]: TLA+ specification to check the types of attacks

Light client attacks

We define a light client attack as detection of conflicting headers for a given height that can be verified starting from the trusted light block. A light client attack is defined in the context of interactions of light client with two peers. One of the peers (called primary) defines a trace of verified light blocks (primary trace) that are being checked against trace of the other peer (called witness) that we call witness trace.

A light client attack is defined by the primary and witness traces that have a common root (the same trusted light block for a common height) but forms conflicting branches (end of traces is for the same height but with different headers). Note that conflicting branches could be arbitrarily big as branches continue to diverge after a bifurcation point. We propose an approach that allows us to define a valid light client attack only with a common light block and a single conflicting light block. We rely on the fact that we assume that the primary is under suspicion (therefore not trusted) and that the witness plays support role to detect and process an attack (therefore trusted). Therefore, once a light client detects an attack, it needs to send to a witness only missing data (common height and conflicting light block) as it has its trace. Keeping light client attack data of constant size saves bandwidth and reduces an attack surface. As we will explain below, although in the context of light client core verification the roles of primary and witness are clearly defined, in case of the attack, we run the same attack detection procedure twice where the roles are swapped. The rationale is that the light client does not know what peer is correct (on a right main branch) so it tries to create and submit an attack evidence to both peers.

Light client attack evidence consists of a conflicting light block and a common height.

type LightClientAttackEvidence struct {
    ConflictingBlock   LightBlock
    CommonHeight       int64
}

Full node can validate a light client attack evidence by executing the following procedure:

func IsValid(lcaEvidence LightClientAttackEvidence, bc Blockchain) boolean {
    commonBlock = GetLightBlock(bc, lcaEvidence.CommonHeight)
    if commonBlock == nil return false

    // Note that trustingPeriod in ValidAndVerified is set to UNBONDING_PERIOD
    verdict = ValidAndVerified(commonBlock, lcaEvidence.ConflictingBlock)
    conflictingHeight = lcaEvidence.ConflictingBlock.Header.Height

    return verdict == OK and bc[conflictingHeight].Header != lcaEvidence.ConflictingBlock.Header
}

Light client attack creation

Given a trusted light block trusted, a light node executes the bisection algorithm to verify header untrusted at some height h. If the bisection algorithm succeeds, then the header untrusted is verified. Headers that are downloaded as part of the bisection algorithm are stored in a store and they are also in the verified state. Therefore, after the bisection algorithm successfully terminates we have a trace of the light blocks ([] LightBlock) we obtained from the primary that we call primary trace.

Primary trace

The following invariant holds for the primary trace:

  • Given a trusted light block, target height h, and primary_trace ([] LightBlock): primary_trace[0] == trusted and primary_trace[len(primary_trace)-1].Height == h and successive light blocks are passing light client verification logic.

Witness with a conflicting header

The verified header at height h is cross-checked with every witness as part of detection. If a witness returns the conflicting header at the height h the following procedure is executed to verify if the conflicting header comes from the valid trace and if that's the case to create an attack evidence:

Helper functions

We assume the following helper functions:

// Returns trace of verified light blocks starting from rootHeight and ending with targetHeight.
Trace(lightStore LightStore, rootHeight int64, targetHeight int64) LightBlock[]

// Returns validator set for the given height
GetValidators(bc Blockchain, height int64) Validator[]

// Returns validator set for the given height
GetValidators(bc Blockchain, height int64) Validator[]

// Return validator addresses for the given validators
GetAddresses(vals Validator[]) ValidatorAddress[]
func DetectLightClientAttacks(primary PeerID,
                              primary_trace []LightBlock,
                              witness PeerID) (LightClientAttackEvidence, LightClientAttackEvidence) {
    primary_lca_evidence, witness_trace = DetectLightClientAttack(primary_trace, witness)

    witness_lca_evidence = nil
    if witness_trace != nil {
        witness_lca_evidence, _ = DetectLightClientAttack(witness_trace, primary)
    }
    return primary_lca_evidence, witness_lca_evidence
}

func DetectLightClientAttack(trace []LightBlock, peer PeerID) (LightClientAttackEvidence, []LightBlock) {

    lightStore = new LightStore().Update(trace[0], StateTrusted)

    for i in 1..len(trace)-1 {
        lightStore, result = VerifyToTarget(peer, lightStore, trace[i].Header.Height)

        if result == ResultFailure then return (nil, nil)

        current = lightStore.Get(trace[i].Header.Height)

        // if obtained header is the same as in the trace we continue with a next height
        if current.Header == trace[i].Header continue

        // we have identified a conflicting header
        commonBlock = trace[i-1]
        conflictingBlock = trace[i]

        return (LightClientAttackEvidence { conflictingBlock, commonBlock.Header.Height },
                Trace(lightStore, trace[i-1].Header.Height, trace[i].Header.Height))
    }
    return (nil, nil)
}

Evidence handling

As part of on chain evidence handling, full nodes identifies misbehaving processes and informs the application, so they can be slashed. Note that only bonded validators should be reported to the application. There are three types of attacks that can be executed against Tendermint light client:

  • lunatic attack
  • equivocation attack and
  • amnesia attack.

We now specify the evidence handling logic.

func detectMisbehavingProcesses(lcAttackEvidence LightClientAttackEvidence, bc Blockchain) []ValidatorAddress {
   assume IsValid(lcaEvidence, bc)

   // lunatic light client attack
   if !isValidBlock(current.Header, conflictingBlock.Header) {
        conflictingCommit = lcAttackEvidence.ConflictingBlock.Commit
        bondedValidators = GetNextValidators(bc, lcAttackEvidence.CommonHeight)

        return getSigners(conflictingCommit) intersection GetAddresses(bondedValidators)

   // equivocation light client attack
   } else if current.Header.Round == conflictingBlock.Header.Round {
        conflictingCommit = lcAttackEvidence.ConflictingBlock.Commit
        trustedCommit = bc[conflictingBlock.Header.Height+1].LastCommit

        return getSigners(trustedCommit) intersection getSigners(conflictingCommit)

   // amnesia light client attack
   } else {
        HandleAmnesiaAttackEvidence(lcAttackEvidence, bc)
   }
}

// Block validity in this context is defined by the trusted header.
func isValidBlock(trusted Header, conflicting Header) boolean {
    return trusted.ValidatorsHash == conflicting.ValidatorsHash and
           trusted.NextValidatorsHash == conflicting.NextValidatorsHash and
           trusted.ConsensusHash == conflicting.ConsensusHash and
           trusted.AppHash == conflicting.AppHash and
           trusted.LastResultsHash == conflicting.LastResultsHash
}

func getSigners(commit Commit) []ValidatorAddress {
    signers = []ValidatorAddress
    for (i, commitSig) in commit.Signatures {
        if commitSig.BlockIDFlag == BlockIDFlagCommit {
            signers.append(commitSig.ValidatorAddress)
        }
    }
    return signers
}

Note that amnesia attack evidence handling involves more complex processing, i.e., cannot be defined simply on amnesia attack evidence. We explain in the following section a protocol for handling amnesia attack evidence.

Amnesia attack evidence handling

Detecting faulty processes in case of the amnesia attack is more complex and cannot be inferred purely based on attack evidence data. In this case, in order to detect misbehaving processes we need access to votes processes sent/received during the conflicting height. Therefore, amnesia handling assumes that validators persist all votes received and sent during multi-round heights (as amnesia attack is only possible in heights that executes over multiple rounds, i.e., commit round > 0).

To simplify description of the algorithm we assume existence of the trusted oracle called monitor that will drive the algorithm and output faulty processes at the end. Monitor can be implemented in a distributed setting as on-chain module. The algorithm works as follows: 1) Monitor sends votesets request to validators of the conflicting height. Validators are expected to send their votesets within predefined timeout. 2) Upon receiving votesets request, validators send their votesets to a monitor.
2) Validators which have not sent its votesets within timeout are considered faulty. 3) The preprocessing of the votesets is done. That means that the received votesets are analyzed and each vote (valid) sent by process p is added to the voteset of the sender p. This phase ensures that votes sent by faulty processes observed by at least one correct validator cannot be excluded from the analysis. 4) Votesets of every validator are analyzed independently to decide whether the validator is correct or faulty. A faulty validators is the one where at least one of those invalid transitions is found: - More than one PREVOTE message is sent in a round - More than one PRECOMMIT message is sent in a round - PRECOMMIT message is sent without receiving +2/3 of voting-power equivalent appropriate PREVOTE messages - PREVOTE message is sent for the value V’ in round r’ and the PRECOMMIT message had been sent for the value V in round r by the same process (r’ > r) and there are no +2/3 of voting-power equivalent PREVOTE(vr, V’) messages (vr ≥ 0 and vr > r and vr < r’) as the justification for sending PREVOTE(r’, V’)

Tendermint fork detection and IBC fork detection

Status

This is a work in progress. This directory captures the ongoing work and discussion on fork detection both in the context of a Tendermint light node and in the context of IBC. It contains the following files

detection.md

a draft of the light node fork detection including "proof of fork" definition, that is, the data structure to submit evidence to full nodes.

discussions.md

A collection of ideas and intuitions from recent discussions

  • the outcome of recent discussion
  • a sketch of the light client supervisor to provide the context in which fork detection happens
  • a discussion about lightstore semantics

req-ibc-detection.md

  • a collection of requirements for fork detection in the IBC context. In particular it contains a section "Required Changes in ICS 007" with necessary updates to ICS 007 to support Tendermint fork detection

draft-functions.md

In order to address the collected requirements, we started to sketch some functions that we will need in the future when we specify in more detail the

  • fork detections
  • proof of fork generation
  • proof of fork verification

on the following components.

  • IBC on-chain components
  • Relayer

TODOs

We decided to merge the files while there are still open points to address to record the current state an move forward. In particular, the following points need to be addressed:

Most likely we will write a specification on the light client supervisor along the outcomes of

that also addresses initialization

This an unfinished draft. Comments are welcome!

TODO: We will need to do small adaptations to the verification spec to reflect the semantics in the LightStore (verified, trusted, untrusted, etc. not needed anymore). In more detail:

  • The state of the Lightstore needs to go. Functions like LatestVerified can keep the name but will ignore state as it will not exist anymore.

  • verification spec should be adapted to the second parameter of VerifyToTarget being a lightblock; new version number of function tag;

  • We should clarify what is the expectation of VerifyToTarget so if it returns TimeoutError it can be assumed faulty. I guess that VerifyToTarget with correct full node should never terminate with TimeoutError.

  • We need to introduce a new version number for the new specification. So we should decide how to handle that.

Light Client Attack Detector

In this specification, we strengthen the light client to be resistant against so-called light client attacks. In a light client attack, all the correct Tendermint full nodes agree on the sequence of generated blocks (no fork), but a set of faulty full nodes attack a light client by generating (signing) a block that deviates from the block of the same height on the blockchain. In order to do so, some of these faulty full nodes must have been validators before and violate [TMBC-FM-2THIRDS], as otherwise, if [TMBC-FM-2THIRDS] would hold, verification would satisfy [LCV-SEQ-SAFE.1].

An attack detector (or detector for short) is a mechanism that is used by the light client supervisor after verification of a new light block with the primary, to cross-check the newly learned light block with other peers (secondaries). It expects as input a light block with some height root (that serves as a root of trust), and a verification trace (a sequence of lightblocks) that the primary provided.

In case the detector observes a light client attack, it computes evidence data that can be used by Tendermint full nodes to isolate a set of faulty full nodes that are still within the unbonding period (more than 1/3 of the voting power of the validator set at some block of the chain), and report them via ABCI to the application of a Tendermint blockchain in order to punish faulty nodes.

Context of this document

The light client verification specification is designed for the Tendermint failure model (1/3 assumption) [TMBC-FM-2THIRDS]. It is safe under this assumption, and live if it can reliably (that is, no message loss, no duplication, and eventually delivered) and timely communicate with a correct full node. If [TMBC-FM-2THIRDS] assumption is violated, the light client can be fooled to trust a light block that was not generated by Tendermint consensus.

This specification, the attack detector, is a "second line of defense", in case the 1/3 assumption is violated. Its goal is to detect a light client attack (conflicting light blocks) and collect evidence. However, it is impractical to probe all full nodes. At this time we consider a simple scheme of maintaining an address book of known full nodes from which a small subset (e.g., 4) are chosen initially to communicate with. More involved book keeping with probabilistic guarantees can be considered at later stages of the project.

The light client maintains a simple address book containing addresses of full nodes that it can pick as primary and secondaries. To obtain a new light block, the light client first does verification with the primary, and then cross-checks the light block (and the trace of light blocks that led to it) with the secondaries using this specification.

Tendermint Consensus and Light Client Attacks

In this section we will give some mathematical definitions of what we mean by light client attacks (that are considered in this specification) and how they differ from main-chain forks. To this end we start by defining some properties of the sequence of blocks that is decided upon by Tendermint consensus in normal operation (if the Tendermint failure model holds [TMBC-FM-2THIRDS]), and then define different deviations that correspond to attack scenarios.

[TMBC-GENESIS.1]

Let Genesis be the agreed-upon initial block (file).

[TMBC-FUNC-SIGN.1]

Let b and c be two light blocks with b.Header.Height + 1 = c.Header.Height. We define the predicate signs(b,c) to hold iff c.Header.LastCommit is in PossibleCommit(b). [TMBC-SOUND-DISTR-POSS-COMMIT.1].

The above encodes sequential verification, that is, intuitively, b.Header.NextValidators = c.Header.Validators and 2/3 of these Validators signed c?

[TMBC-FUNC-SUPPORT.1]

Let b and c be two light blocks. We define the predicate supports(b,c,t) to hold iff

  • t - trustingPeriod < b.Header.Time < t
  • the voting power in b.NextValidators of nodes in c.Commit is more than 1/3 of TotalVotingPower(b.Header.NextValidators)

That is, if the Tendermint failure model holds, then c has been signed by at least one correct full node, cf. [TMBC-VAL-CONTAINS-CORR.1]. The following formalizes that b was properly generated by Tendermint; b can be traced back to genesis

[TMBC-SEQ-ROOTED.1]

Let b be a light block. We define sequ-rooted(b) iff for all i, 1 <= i < h = b.Header.Height, there exist light blocks a(i) s.t.

  • a(1) = Genesis and
  • a(h) = b and
  • signs( a(i) , a(i+1) ).

The following formalizes that c is trusted based on b in skipping verification. Observe that we do not require here (yet) that b was properly generated.

[TMBC-SKIP-TRACE.1]

Let b and c be light blocks. We define skip-trace(b,c,t) if at time t there exists an h and a sequence a(1), ... a(h) s.t.

  • a(1) = b and
  • a(h) = c and
  • supports( a(i), a(i+1), t), for all i, 1 <= i < h.

We call such a sequence a(1), ... a(h) a verification trace.

The following formalizes that two light blocks of the same height should agree on the content of the header. Observe that b and c may disagree on the Commit. This is a special case if the canonical commit has not been decided on, that is, if b.Header.Height is the maximum height of all blocks decided upon by Tendermint at this moment.

[TMBC-SIGN-SKIP-MATCH.1]

Let a, b, c, be light blocks and t a time, we define sign-skip-match(a,b,c,t) = true iff the following implication evaluates to true:

  • sequ-rooted(a) and
  • b.Header.Height = c.Header.Height and
  • skip-trace(a,b,t)
  • skip-trace(a,c,t)

implies b.Header = c.Header.

Observe that sign-skip-match is defined via an implication. If it evaluates to false this means that the left-hand-side of the implication evaluates to true, and the right-hand-side evaluates to false. In particular, there are two different headers b and c that both can be verified from a common block a from the chain. Thus, the following describes an attack.

[TMBC-ATTACK.1]

If there exists three light blocks a, b, and c, with sign-skip-match(a,b,c,t) = false then we have an attack. We say we have an attack at height b.Header.Height and write attack(a,b,c,t).

The lightblock a need not be unique, that is, there may be several blocks that satisfy the above requirement for the same blocks b and c.

[TMBC-ATTACK.1] is a formalization of the violation of the agreement property based on the result of consensus, that is, the generated blocks.

Remark. Violation of agreement is only possible if more than 1/3 of the validators (or next validators) of some previous block deviated from the protocol. The upcoming "accountability" specification will describe how to compute a set of at least 1/3 faulty nodes from two conflicting blocks. []

There are different ways to characterize forks and attack scenarios. This specification uses the "node-based characterization of attacks" which focuses on what kinds of nodes are affected (light nodes vs. full nodes). For future reference and discussion we also provide a "block-based characterization of attacks" below.

Node-based characterization of attacks

[TMBC-MC-FORK.1]

We say there is a (main chain) fork at time t if

  • there are two correct full nodes i and j and
  • i is different from j and
  • i has decided on b and
  • j has decided on c and
  • there exist a such that attack(a,b,c,t).

[TMBC-LC-ATTACK.1]

We say there is a light client attack at time t, if

  • there is no (main chain) fork [TMBC-MC-FORK.1], and
  • there exist nodes that have computed light blocks b and c and
  • there exist a such that attack(a,b,c,t).

We say the attack is at height a.Header.Height.

In this specification we consider detection of light client attacks. Intuitively, the case we consider is that light block b is the one from the blockchain, and some attacker has computed c and tries to wrongly convince the light client that c is the block from the chain.

[TMBC-LC-ATTACK-EVIDENCE.1]

We consider the following case of a light client attack [TMBC-LC-ATTACK.1]:

  • attack(a,b,c,t)
  • there is a peer p1 that has a sequence chain of blocks from a to b
  • skip-trace(a,c,t): by [TMBC-SKIP-TRACE.1] there is a verification trace v of the form a = v(1), ... v(h) = c

Evidence for p1 (that proves an attack) consists for index i of v(i) and v(i+1) such that

  • E1(i). v(i) is equal to the block of chain at height v(i).Height, and
  • E2(i). v(i+1) that is different from the block of chain at height v(i+1).height

Observe p1 can

  • check that v(i+1) differs from its block at that height, and
  • verify v(i+1) in one step from v(i) as v is a verification trace.

Proposition. In the case of attack, evidence exists.
Proof. First observe that

  • (A). (NOT E2(i)) implies E1(i+1)

Now by contradiction assume there is no evidence. Thus

  • for all i, we have NOT E1(i) or NOT E2(i)
  • for i = 1 we have E1(1) and thus NOT E2(1) thus by induction on i, by (A) we have for all i that E1(i)
  • from attack we have E2(h-1), and as there is no evidence for i = h - 1 we get NOT E1(h-1). Contradiction. QED.

[TMBC-LC-EVIDENCE-DATA.1]

To prove the attack to p1, because of Point E1, it is sufficient to submit

  • v(i).Height (rather than v(i)).
  • v(i+1)

This information is evidence for height v(i).Height.

Block-based characterization of attacks

In this section we provide a different characterization of attacks. It is not defined on the nodes that are affected but purely on the content of the blocks. In that sense these definitions are less operational.

They might be relevant for a closer analysis of fork scenarios on the chain, which is out of the scope of this specification.

[TMBC-SIGN-UNIQUE.1]

Let b and c be light blocks, we define the predicate sign-unique(b,c) to evaluate to true iff the following implication evaluates to true:

  • b.Header.Height = c.Header.Height and
  • sequ-rooted(b) and
  • sequ-rooted(c)

implies b = c.

[TMBC-BLOCKS-MCFORK.1]

If there exists two light blocks b and c, with sign-unique(b,c) = false then we have a fork.

The difference of the above definition to [TMBC-MC-FORK.1] is subtle. The latter requires a full node being affected by a bad block while [TMBC-BLOCKS-MCFORK.1] just requires that a bad block exists, possibly in memory of an attacker. The following captures a light client fork. There is no fork up to the height of block b. However, c is of that height, is different, and passes skipping verification. It is a stricter property than [TMBC-LC-ATTACK.1], as [TMBC-LC-ATTACK.1] requires that no correct full node is affected.

[TMBC-BLOCKS-LCFORK.1]

Let a, b, c, be light blocks and t a time. We define light-client-fork(a,b,c,t) iff

  • sign-skip-match(a,b,c,t) = false and
  • sequ-rooted(b) and
  • b is "unique", that is, for all d, sequ-rooted(d) and d.Header.Height = b.Header.Height implies d = b

Finally, let us also define bogus blocks that have no support. Observe that bogus is even defined if there is a fork. Also, for the definition it would be sufficient to restrict a to a.height < b.height (which is implied by the definitions which unfold until supports()).

[TMBC-BOGUS.1]

Let b be a light block and t a time. We define bogus(b,t) iff

  • sequ-rooted(b) = false and
  • for all a, sequ-rooted(a) implies skip-trace(a,b,t) = false

Informal Problem statement

There is no sequential specification: the detector only makes sense in a distributed systems where some nodes misbehave.

We work under the assumption that full nodes and validators are responsible for detecting attacks on the main chain, and the evidence reactor takes care of broadcasting evidence to communicate misbehaving nodes via ABCI to the application, and halt the chain in case of a fork. The point of this specification is to shield a light clients against attacks that cannot be detected by full nodes, and are fully addressed at light clients (and consequently IBC relayers, which use the light client protocols to observe the state of a blockchain). In order to provide full nodes the incentive to follow the protocols when communicating with the light client, this specification also considers the generation of evidence that will also be processed by the Tendermint blockchain.

[LCD-IP-MODEL.1]

The detector is designed under the assumption that

As a result some faulty full nodes may launch an attack on a light client.

The following requirements are operational in that they describe how things should be done, rather than what should be done. However, they do not constitute temporal logic verification conditions. For those, see [LCD-DIST-*] below.

The detector is called in the supervisor as follows

Evidences := AttackDetector(root_of_trust, verifiedLS);`

where

  • root-of-trust is a light block that is trusted (that is, except upon initialization, the primary and the secondaries agreed on in the past), and
  • verifiedLS is a lightstore that contains a verification trace that starts from a lightblock that can be verified with the root-of-trust in one step and ends with a lightblock of the height requested by the user
  • Evidences is a list of evidences for misbehavior

[LCD-IP-STATEMENT.1]

Whenever AttackDetector is called, the detector should for each secondary try to replay the verification trace verifiedLS with the secondary

  • in case replaying leads to detection of a light client attack (one of the lightblocks differ from the one in verifiedLS with the same height), we should return evidence
  • if the secondary cannot provide a verification trace, we have no proof for an attack. Block b may be bogus. In this case the secondary is faulty and it should be replaced.

Assumptions/Incentives/Environment

It is not in the interest of faulty full nodes to talk to the detector as long as the detector is connected to at least one correct full node. This would only increase the likelihood of misbehavior being detected. Also we cannot punish them easily (cheaply). The absence of a response need not be the fault of the full node.

Correct full nodes have the incentive to respond, because the detector may help them to understand whether their header is a good one. We can thus base liveness arguments of the detector on the assumptions that correct full nodes reliably talk to the detector.

Assumptions

[LCD-A-CorrFull.1]

At all times there is at least one correct full node among the primary and the secondaries.

For this version of the detection we take this assumption. It allows us to establish the invariant that the lightblock root-of-trust is always the one from the blockchain, and we can use it as starting point for the evidence computation. Moreover, it allows us to establish the invariant at the supervisor that any lightblock in the (top-level) lightstore is from the blockchain.
In the future we might design a lightclient based on the assumption that at least in regular intervals the lightclient is connected to a correct full node. This will require the detector to reconsider root-of-trust, and remove lightblocks from the top-level lightstore.

[LCD-A-RelComm.1]

Communication between the detector and a correct full node is reliable and bounded in time. Reliable communication means that messages are not lost, not duplicated, and eventually delivered. There is a (known) end-to-end delay Delta, such that if a message is sent at time t then it is received and processed by time t + Delta. This implies that we need a timeout of at least 2 Delta for remote procedure calls to ensure that the response of a correct peer arrives before the timeout expires.

Definitions

Evidence

Following the definition of [TMBC-LC-ATTACK-EVIDENCE.1], by evidence we refer to a variable of the following type

[LC-DATA-EVIDENCE.1]

type LightClientAttackEvidence struct {
    ConflictingBlock   LightBlock
    CommonHeight       int64
}

As the above data is computed for a specific peer, the following data structure wraps the evidence and adds the peerID.

[LC-DATA-EVIDENCE-INT.1]

type InternalEvidence struct {
    Evidence           LightClientAttackEvidence
    Peer               PeerID
}

[LC-SUMBIT-EVIDENCE.1]

func submitEvidence(Evidences []InternalEvidence)
  • Expected postcondition
    • for each ev in Evidences: submit ev.Evidence to ev.Peer

LightStore

Lightblocks and LightStores are defined in the verification specification [LCV-DATA-LIGHTBLOCK.1] and [LCV-DATA-LIGHTSTORE.1]. See the verification specification for details.

(Distributed) Problem statement

As the attack detector is there to reduce the impact of faulty nodes, and faulty nodes imply that there is a distributed system, there is no sequential specification to which this distributed problem statement may refer to.

The detector gets as input a trusted lightblock called root and an auxiliary lightstore called primary_trace with lightblocks that have been verified before, and that were provided by the primary.

[LCD-DIST-INV-ATTACK.1]

If the detector returns evidence for height h [TMBC-LC-EVIDENCE-DATA.1], then there is an attack at height h. [TMBC-LC-ATTACK.1]

[LCD-DIST-INV-STORE.1]

If the detector does not return evidence, then primary_trace contains only blocks from the blockchain.

[LCD-DIST-LIVE.1]

The detector eventually terminates.

[LCD-DIST-TERM-NORMAL.1]

If

  • the primary_trace contains only blocks from the blockchain, and
  • there is no attack, and
  • Secondaries is always non-empty, and
  • the age of root is always less than the trusting period,

then the detector does not return evidence.

[LCD-DIST-TERM-ATTACK.1]

If

  • there is an attack, and
  • a secondary reports a block that conflicts with one of the blocks in primary_trace, and
  • Secondaries is always non-empty, and
  • the age of root is always less than the trusting period,

then the detector returns evidence.

Observe that above we require that "a secondary reports a block that conflicts". If there is an attack, but no secondary tries to launch it against the detector (or the message from the secondary is lost by the network), then there is nothing to detect for us.

[LCD-DIST-SAFE-SECONDARY.1]

No correct secondary is ever replaced.

[LCD-DIST-SAFE-BOGUS.1]

If

  • a secondary reports a bogus lightblock,
  • the age of root is always less than the trusting period,

then the secondary is replaced before the detector terminates.

The above property is quite operational ("reports"), but it captures quite closely the requirement. As the detector only makes sense in a distributed setting, and does not have a sequential specification, less "pure" specification are acceptable.

Protocol

Functions and Data defined in other Specifications

From the supervisor

Replace_Secondary(addr Address, root-of-trust LightBlock)

From the verifier

func VerifyToTarget(primary PeerID, root LightBlock,
                    targetHeight Height) (LightStore, Result)

Note: the above differs from the current version in the second parameter. verification will be revised.

Observe that VerifyToTarget does communication with the secondaries via the function FetchLightBlock.

Shared data of the light client

  • a pool of full nodes FullNodes that have not been contacted before
  • peer set called Secondaries
  • primary

Note that the lightStore is not needed to be shared.

Outline

The problem laid out is solved by calling the function AttackDetector with a lightstore that contains a light block that has just been verified by the verifier.

Then AttackDetector downloads headers from the secondaries. In case a conflicting header is downloaded from a secondary, CreateEvidenceForPeer which computes evidence in the case that indeed an attack is confirmed. It could be that the secondary reports a bogus block, which means that there need not be an attack, and the secondary is replaced.

Details of the functions

[LCD-FUNC-DETECTOR.1]:

func AttackDetector(root LightBlock, primary_trace []LightBlock)
                   ([]InternalEvidence) {

    Evidences := new []InternalEvidence;

    for each secondary in Secondaries {
        // we replay the primary trace with the secondary, in
        // order to generate evidence that we can submit to the
        // secodary. We return the evidence + the trace the
        // secondary told us that spans the evidence at its local store

        EvidenceForSecondary, newroot, secondary_trace, result :=
                CreateEvidenceForPeer(secondary,
                          root,
           primary_trace);
        if result == FaultyPeer {
            Replace_Secondary(root);
        }
        else if result == FoundEvidence {
            // the conflict is not bogus
            Evidences.Add(EvidenceForSecondary);
            // we replay the secondary trace with the primary, ...
            EvidenceForPrimary, _, result :=
        CreateEvidenceForPeer(primary,
                                          newroot,
                                          secondary_trace);
            if result == FoundEvidence {
                Evidences.Add(EvidenceForPrimary);
            }
            // At this point we do not care about the other error
            // codes. We already have generated evidence for an
            // attack and need to stop the lightclient. It does not
            // help to call replace_primary. Also we will use the
            // same primary to check with other secondaries in
            // later iterations of the loop
         }
         // In the case where the secondary reports NoEvidence
         // we do nothing
    }
    return Evidences;
}
  • Expected precondition
    • root and primary trace are a verification trace
  • Expected postcondition
    • solves the problem statement (if attack found, then evidence is reported)
  • Error condition
    • ErrorTrustExpired: fails if root expires (outside trusting period) [LCV-INV-TP.1]
    • ErrorNoPeers: if no peers are left to replace secondaries, and no evidence was found before that happened

func CreateEvidenceForPeer(peer PeerID, root LightBlock, trace LightStore)
                          (Evidence, LightBlock, LightStore, result) {

    common := root;

    for i in 1 .. len(trace) {
        auxLS, result := VerifyToTarget(peer, common, trace[i].Header.Height)
  
        if result != ResultSuccess {
            // something went wrong; peer did not provide a verifyable block
            return (nil, nil, nil, FaultyPeer)
        }
        else {
            if auxLS.LatestVerified().Header != trace[i].Header {
                // the header reported by the peer differs from the
                // reference header in trace but both could be
                // verified from common in one step.
                // we can create evidence for submission to the secondary
                ev := new InternalEvidence;
                ev.Evidence.ConflictingBlock := trace[i];
                ev.Evidence.CommonHeight := common.Height;
                ev.Peer := peer
                return (ev, common, auxLS, FoundEvidence)
            }
            else {
                // the peer agrees with the trace, we move common forward
                // we could delete auxLS as it will be overwritten in
                // the next iteration
                common := trace[i]
            }
        }
    }
    return (nil, nil, nil, NoEvidence)
}
  • Expected precondition
    • root and trace are a verification trace
  • Expected postcondition
    • finds evidence where trace and peer diverge
  • Error condition
    • ErrorTrustExpired: fails if root expires (outside trusting period) [LCV-INV-TP.1]
    • If VerifyToTarget returns error but root is not expired then return FaultyPeer

Correctness arguments

Argument for [LCD-DIST-INV-ATTACK.1]

Under the assumption that root and trace are a verification trace, when in CreateEvidenceForPeer the detector the detector creates evidence, then the lightclient has seen two different headers (one via trace and one via VerifyToTarget for the same height that can both be verified in one step.

Argument for [LCD-DIST-INV-STORE.1]

We assume that there is at least one correct peer, and there is no fork. As a result the correct peer has the correct sequence of blocks. Since the primary_trace is checked block-by-block also against each secondary, and at no point evidence was generated that means at no point there were conflicting blocks.

Argument for [LCD-DIST-LIVE.1]

At the latest when [LCV-INV-TP.1] is violated, AttackDetector terminates.

Argument for [LCD-DIST-TERM-NORMAL.1]

As there are finitely many peers, eventually the main loop terminates. As there is no attack no evidence can be generated.

Argument for [LCD-DIST-TERM-ATTACK.1]

Argument similar to [LCD-DIST-TERM-NORMAL.1]

Argument for [LCD-DIST-SAFE-SECONDARY.1]

Secondaries are only replaced if they time-out or if they report bogus blocks. The former is ruled out by the timing assumption, the latter by correct peers only reporting blocks from the chain.

Argument for [LCD-DIST-SAFE-BOGUS.1]

Once a bogus block is recognized as such the secondary is removed.

References

links to other specifications/ADRs this document refers to

[verification] The specification of the light client verification.

[supervisor] The specification of the light client supervisor.

Light Client Attack Detector

In this specification, we strengthen the light client to be resistant against so-called light client attacks. In a light client attack, all the correct Tendermint full nodes agree on the sequence of generated blocks (no fork), but a set of faulty full nodes attack a light client by generating (signing) a block that deviates from the block of the same height on the blockchain. In order to do so, some of these faulty full nodes must have been validators before and violate the assumption of more than two thirds of "correct voting power" [TMBC-FM-2THIRDS], as otherwise, if [TMBC-FM-2THIRDS] would hold, verification would satisfy [LCV-SEQ-SAFE.1].

An attack detector (or detector for short) is a mechanism that is used by the light client supervisor after verification of a new light block with the primary, to cross-check the newly learned light block with other peers (secondaries). It expects as input a light block with some height root (that serves as a root of trust), and a verification trace (a sequence of lightblocks) that the primary provided.

In case the detector observes a light client attack, it computes evidence data that can be used by Tendermint full nodes to isolate a set of faulty full nodes that are still within the unbonding period (more than 1/3 of the voting power of the validator set at some block of the chain), and report them via ABCI (application/blockchain interface) to the application of a Tendermint blockchain in order to punish faulty nodes.

Context of this document

The light client verification specification is designed for the Tendermint failure model (1/3 assumption) [TMBC-FM-2THIRDS]. It is safe under this assumption, and live if it can reliably (that is, no message loss, no duplication, and eventually delivered) and timely communicate with a correct full node. If [TMBC-FM-2THIRDS] assumption is violated, the light client can be fooled to trust a light block that was not generated by Tendermint consensus.

This specification, the attack detector, is a "second line of defense", in case the 1/3 assumption is violated. Its goal is to detect a light client attack (conflicting light blocks) and collect evidence. However, it is impractical to probe all full nodes. At this time we consider a simple scheme of maintaining an address book of known full nodes from which a small subset (e.g., 4) are chosen initially to communicate with. More involved book keeping with probabilistic guarantees can be considered at later stages of the project.

The light client maintains a simple address book containing addresses of full nodes that it can pick as primary and secondaries. To obtain a new light block, the light client first does verification with the primary, and then cross-checks the light block (and the trace of light blocks that led to it) with the secondaries using this specification.

Outline

Part I - Tendermint Consensus and Light Client Attacks

In this section we will give some mathematical definitions of what we mean by light client attacks (that are considered in this specification) and how they differ from main-chain forks. To this end, we start by defining some properties of the sequence of blocks that is decided upon by Tendermint consensus in normal operation (if the Tendermint failure model holds [TMBC-FM-2THIRDS]), and then define different deviations that correspond to attack scenarios. We consider the notion of light blocks and headers.

[TMBC-GENESIS.1]

Let Genesis be the agreed-upon initial block (file).

[TMBC-FUNC-SIGN.1]

Let b and c be two light blocks with b.Header.Height + 1 = c.Header.Height. We define the predicate signs(b,c) to hold iff c.Header.LastCommit is in PossibleCommit(b). [TMBC-SOUND-DISTR-POSS-COMMIT.1].

The above encodes sequential verification, that is, intuitively, b.Header.NextValidators = c.Header.Validators and 2/3 of these Validators signed c.

[TMBC-FUNC-SUPPORT.1]

Let b and c be two light blocks. We define the predicate supports(b,c,t) to hold iff

  • t - trustingPeriod < b.Header.Time < t
  • the voting power in b.NextValidators of nodes in c.Commit is more than 1/3 of TotalVotingPower(b.Header.NextValidators)

That is, if the Tendermint failure model holds, then c has been signed by at least one correct full node, cf. [TMBC-VAL-CONTAINS-CORR.1]. The following formalizes that b was properly generated by Tendermint; b can be traced back to genesis.

[TMBC-SEQ-ROOTED.1]

Let b be a light block. We define sequ-rooted(b) iff for all i, 1 <= i < h = b.Header.Height, there exist light blocks a(i) s.t.

  • a(1) = Genesis and
  • a(h) = b and
  • signs( a(i) , a(i+1) ).

The following formalizes that c is trusted based on b in skipping verification. Observe that we do not require here (yet) that b was properly generated.

[TMBC-SKIP-TRACE.1]

Let b and c be light blocks. We define skip-trace(b,c,t) if at time t there exists an integer h and a sequence a(1), ... a(h) s.t.

  • a(1) = b and
  • a(h) = c and
  • supports( a(i), a(i+1), t), for all i, 1 <= i < h.

We call such a sequence a(1), ... a(h) a verification trace.

The following formalizes that two light blocks of the same height should agree on the content of the header. Observe that b and c may disagree on the Commit. This is a special case if the canonical commit has not been decided on yet, that is, if b.Header.Height is the maximum height of all blocks decided upon by Tendermint at this moment.

[TMBC-SIGN-SKIP-MATCH.1]

Let a, b, c, be light blocks and t a time, we define sign-skip-match(a,b,c,t) = true iff the following implication evaluates to true:

  • sequ-rooted(a) and
  • b.Header.Height = c.Header.Height and
  • skip-trace(a,b,t)
  • skip-trace(a,c,t)

implies b.Header = c.Header.

Observe that sign-skip-match is defined via an implication. If it evaluates to false this means that the left-hand-side of the implication evaluates to true, and the right-hand-side evaluates to false. In particular, there are two different headers b and c that both can be verified from a common block a from the chain. Thus, the following describes an attack.

[TMBC-ATTACK.1]

If there exists three light blocks a, b, and c, with sign-skip-match(a,b,c,t) = false then we have an attack. We say we have an attack at height b.Header.Height and write attack(a,b,c,t).

The lightblock a need not be unique, that is, there may be several blocks that satisfy the above requirement for the same blocks b and c.

[TMBC-ATTACK.1] is a formalization of the violation of the agreement property based on the result of consensus, that is, the generated blocks.

Remark. Violation of agreement is only possible if more than 1/3 of the validators (or next validators) of some previous block deviated from the protocol. The upcoming "accountability" specification will describe how to compute a set of at least 1/3 faulty nodes from two conflicting blocks. []

There are different ways to characterize forks and attack scenarios. This specification uses the "node-based characterization of attacks" which focuses on what kinds of nodes are affected (light nodes vs. full nodes). For future reference and discussion we also provide a "block-based characterization of attacks" below.

Node-based characterization of attacks

[TMBC-MC-FORK.1]

We say there is a (main chain) fork at time t if

  • there are two correct full nodes i and j and
  • i is different from j and
  • i has decided on b and
  • j has decided on c and
  • there exist a such that attack(a,b,c,t).

[TMBC-LC-ATTACK.1]

We say there is a light client attack at time t, if

  • there is no (main chain) fork [TMBC-MC-FORK.1], and
  • there exist nodes that have computed light blocks b and c and
  • there exist a such that attack(a,b,c,t).

We say the attack is at height a.Header.Height.

In this specification we consider detection of light client attacks. Intuitively, the case we consider is that light block b is the one from the blockchain, and some attacker has computed c and tries to wrongly convince the light client that c is the block from the chain.

[TMBC-LC-ATTACK-EVIDENCE.1]

We consider the following case of a light client attack [TMBC-LC-ATTACK.1]:

  • attack(a,b,c,t)
  • there is a peer p1 that has a sequence chain of blocks from a to b
  • skip-trace(a,c,t): by [TMBC-SKIP-TRACE.1] there is a verification trace v of the form a = v(1), ... v(h) = c

Evidence for p1 (that proves an attack to p1) consists for index i of v(i) and v(i+1) such that

  • E1(i). v(i) is equal to the block of chain at height v(i).Height, and
  • E2(i). v(i+1) that is different from the block of chain at height v(i+1).height

Observe p1 can

  • check that v(i+1) differs from its block at that height, and
  • verify v(i+1) in one step from v(i) as v is a verification trace.

[TMBC-LC-EVIDENCE-DATA.1]

To prove the attack to p1, because of Point E1, it is sufficient to submit

  • v(i).Height (rather than v(i)).
  • v(i+1)

This information is evidence for height v(i).Height.

Block-based characterization of attacks

In this section we provide a different characterization of attacks. It is not defined on the nodes that are affected but purely on the content of the blocks. In that sense these definitions are less operational.

They might be relevant for a closer analysis of fork scenarios on the chain, which is out of the scope of this specification.

[TMBC-SIGN-UNIQUE.1]

Let b and c be light blocks, we define the predicate sign-unique(b,c) to evaluate to true iff the following implication evaluates to true:

  • b.Header.Height = c.Header.Height and
  • sequ-rooted(b) and
  • sequ-rooted(c)

implies b = c.

[TMBC-BLOCKS-MCFORK.1]

If there exists two light blocks b and c, with sign-unique(b,c) = false then we have a fork.

The difference of the above definition to [TMBC-MC-FORK.1] is subtle. The latter requires a full node being affected by a bad block while [TMBC-BLOCKS-MCFORK.1] just requires that a bad block exists, possibly in memory of an attacker. The following captures a light client fork. There is no fork up to the height of block b. However, c is of that height, is different, and passes skipping verification. It is a stricter property than [TMBC-LC-ATTACK.1], as [TMBC-LC-ATTACK.1] requires that no correct full node is affected.

[TMBC-BLOCKS-LCFORK.1]

Let a, b, c, be light blocks and t a time. We define light-client-fork(a,b,c,t) iff

  • sign-skip-match(a,b,c,t) = false and
  • sequ-rooted(b) and
  • b is "unique", that is, for all d, sequ-rooted(d) and d.Header.Height = b.Header.Height implies d = b

Finally, let us also define bogus blocks that have no support. Observe that bogus is even defined if there is a fork. Also, for the definition it would be sufficient to restrict a to a.height < b.height (which is implied by the definitions which unfold until supports()).

[TMBC-BOGUS.1]

Let b be a light block and t a time. We define bogus(b,t) iff

  • sequ-rooted(b) = false and
  • for all a, sequ-rooted(a) implies skip-trace(a,b,t) = false

Part II - Problem Statement

Informal Problem statement

There is no sequential specification: the detector only makes sense in a distributed systems where some nodes misbehave.

We work under the assumption that full nodes and validators are responsible for detecting attacks on the main chain, and the evidence reactor takes care of broadcasting evidence to communicate misbehaving nodes via ABCI to the application, and halt the chain in case of a fork. The point of this specification is to shield a light clients against attacks that cannot be detected by full nodes, and are fully addressed at light clients (and consequently IBC relayers, which use the light client protocols to observe the state of a blockchain). In order to provide full nodes the incentive to follow the protocols when communicating with the light client, this specification also considers the generation of evidence that will also be processed by the Tendermint blockchain.

[LCD-IP-MODEL.1]

The detector is designed under the assumption that

As a result some faulty full nodes may launch an attack on a light client.

The following requirements are operational in that they describe how things should be done, rather than what should be done. However, they do not constitute temporal logic verification conditions. For those, see [LCD-DIST-*] below.

The detector is called in the supervisor as follows

Evidences := AttackDetector(root_of_trust, verifiedLS);`

where

  • root-of-trust is a light block that is trusted (that is, except upon initialization, the primary and the secondaries agreed on in the past), and
  • verifiedLS is a lightstore that contains a verification trace that starts from a lightblock that can be verified with the root-of-trust in one step and ends with a lightblock of the height requested by the user
  • Evidences is a list of evidences for misbehavior

[LCD-IP-STATEMENT.1]

Whenever AttackDetector is called, the detector should for each secondary cross check the largest header in verifiedLS with the corresponding header of the same height provided by the secondary. If there is a deviation, the detector should try to replay the verification trace verifiedLS with the secondary

  • in case replaying leads to detection of a light client attack (one of the lightblocks differ from the one in verifiedLS with the same height), we should return evidence
  • if the secondary cannot provide a verification trace, we have no proof for an attack. Block b may be bogus. In this case the secondary is faulty and it should be replaced.

Assumptions

It is not in the interest of faulty full nodes to talk to the detector as long as the detector is connected to at least one correct full node. This would only increase the likelihood of misbehavior being detected. Also we cannot punish them easily (cheaply). The absence of a response need not be the fault of the full node.

Correct full nodes have the incentive to respond, because the detector may help them to understand whether their header is a good one. We can thus base liveness arguments of the detector on the assumptions that correct full nodes reliably talk to the detector.

[LCD-A-CorrFull.1]

At all times there is at least one correct full node among the primary and the secondaries.

For this version of the detection we take this assumption. It allows us to establish the invariant that the lightblock root-of-trust is always the one from the blockchain, and we can use it as starting point for the evidence computation. Moreover, it allows us to establish the invariant at the supervisor that any lightblock in the (top-level) lightstore is from the blockchain.
In the future we might design a lightclient based on the assumption that at least in regular intervals the lightclient is connected to a correct full node. This will require the detector to reconsider root-of-trust, and remove lightblocks from the top-level lightstore.

[LCD-A-RelComm.1]

Communication between the detector and a correct full node is reliable and bounded in time. Reliable communication means that messages are not lost, not duplicated, and eventually delivered. There is a (known) end-to-end delay Delta, such that if a message is sent at time t then it is received and processed by time t + Delta. This implies that we need a timeout of at least 2 Delta for remote procedure calls to ensure that the response of a correct peer arrives before the timeout expires.

Definitions

Evidence

Following the definition of [TMBC-LC-ATTACK-EVIDENCE.1], by evidence we refer to a variable of the following type

[LC-DATA-EVIDENCE.1]

type LightClientAttackEvidence struct {
    ConflictingBlock   LightBlock
    CommonHeight       int64

    // Evidence also includes application specific data which is not
    // part of verification but is sent to the application once the
    // evidence gets committed on chain.
}

As the above data is computed for a specific peer, the following data structure wraps the evidence and adds the peerID.

[LC-DATA-EVIDENCE-INT.1]

type InternalEvidence struct {
    Evidence           LightClientAttackEvidence
    Peer               PeerID
}

[LC-SUMBIT-EVIDENCE.1]

func submitEvidence(Evidences []InternalEvidence)
  • Expected postcondition
    • for each ev in Evidences: submit ev.Evidence to ev.Peer

LightStore

Lightblocks and LightStores are defined in the verification specification [LCV-DATA-LIGHTBLOCK.1] and [LCV-DATA-LIGHTSTORE.2]. See the verification specification for details.

Distributed Problem statement

As the attack detector is there to reduce the impact of faulty nodes, and faulty nodes imply that there is a distributed system, there is no sequential specification to which this distributed problem statement may refer to.

The detector gets as input a trusted lightblock called root and an auxiliary lightstore called primary_trace with lightblocks that have been verified before, and that were provided by the primary.

[LCD-DIST-INV-ATTACK.1]

If the detector returns evidence for height h [TMBC-LC-EVIDENCE-DATA.1], then there is an attack at height h. [TMBC-LC-ATTACK.1]

[LCD-DIST-INV-STORE.1]

If the detector does not return evidence, then primary_trace contains only blocks from the blockchain.

[LCD-DIST-LIVE.1]

The detector eventually terminates.

[LCD-DIST-TERM-NORMAL.1]

If

  • the primary_trace contains only blocks from the blockchain, and
  • there is no attack, and
  • Secondaries is always non-empty, and
  • the age of root is always less than the trusting period,

then the detector does not return evidence.

[LCD-DIST-TERM-ATTACK.1]

If

  • there is an attack, and
  • a secondary reports a block that conflicts with one of the blocks in primary_trace, and
  • Secondaries is always non-empty, and
  • the age of root is always less than the trusting period,

then the detector returns evidence.

Observe that above we require that "a secondary reports a block that conflicts". If there is an attack, but no secondary tries to launch it against the detector (or the message from the secondary is lost by the network), then there is nothing to detect for us.

[LCD-DIST-SAFE-SECONDARY.1]

No correct secondary is ever replaced.

[LCD-DIST-SAFE-BOGUS.1]

If

  • a secondary reports a bogus lightblock,
  • the age of root is always less than the trusting period,

then the secondary is replaced before the detector terminates.

The above property is quite operational (e.g., the usage of "reports"), but it captures closely the requirement. As the detector only makes sense in a distributed setting, and does not have a sequential specification, a less "pure" specification are acceptable.

Part III - Protocol

Functions and Data defined in other Specifications

From the supervisor

[LC-FUNC-REPLACE-SECONDARY.1]

Replace_Secondary(addr Address, root-of-trust LightBlock)

From the verifier

[LCV-FUNC-MAIN.2]

func VerifyToTarget(primary PeerID, root LightBlock,
                    targetHeight Height) (LightStore, Result)

Observe that VerifyToTarget does communication with the secondaries via the function FetchLightBlock.

Shared data of the light client

  • a pool of full nodes FullNodes that have not been contacted before
  • peer set called Secondaries
  • primary

Note that the lightStore is not needed to be shared.

Outline of solution

The problem laid out is solved by calling the function AttackDetector with a lightstore that contains a light block that has just been verified by the verifier.

Then AttackDetector downloads headers from the secondaries. In case a conflicting header is downloaded from a secondary, it calls CreateEvidenceForPeer which computes evidence in the case that indeed an attack is confirmed. It could be that the secondary reports a bogus block, which means that there need not be an attack, and the secondary is replaced.

Details of the functions

[LCD-FUNC-DETECTOR.2]:

func AttackDetector(root LightBlock, primary_trace []LightBlock)
                   ([]InternalEvidence) {

    Evidences := new []InternalEvidence;

    for each secondary in Secondaries {
        lb, result := FetchLightBlock(secondary,primary_trace.Latest().Header.Height);
        if result != ResultSuccess {
            Replace_Secondary(root);
        }
        else if lb.Header != primary_trace.Latest().Header {
  
            // we replay the primary trace with the secondary, in
            // order to generate evidence that we can submit to the
            // secondary. We return the evidence + the trace the
            // secondary told us that spans the evidence at its local store

            EvidenceForSecondary, newroot, secondary_trace, result :=
                    CreateEvidenceForPeer(secondary,
                                          root,
                                          primary_trace);
            if result == FaultyPeer {
                Replace_Secondary(root);
            }
            else if result == FoundEvidence {
                // the conflict is not bogus
                Evidences.Add(EvidenceForSecondary);
                // we replay the secondary trace with the primary, ...
                EvidenceForPrimary, _, result :=
                        CreateEvidenceForPeer(primary,
                                              newroot,
                                              secondary_trace);
                if result == FoundEvidence {
                    Evidences.Add(EvidenceForPrimary);
                }
                // At this point we do not care about the other error
                // codes. We already have generated evidence for an
                // attack and need to stop the lightclient. It does not
                // help to call replace_primary. Also we will use the
                // same primary to check with other secondaries in
                // later iterations of the loop
            }
            // In the case where the secondary reports NoEvidence
            // after initially it reported a conflicting header.
            // secondary is faulty
            Replace_Secondary(root);
        }
    }
    return Evidences;
}
  • Expected precondition
    • root and primary trace are a verification trace
  • Expected postcondition
    • solves the problem statement (if attack found, then evidence is reported)
  • Error condition
    • ErrorTrustExpired: fails if root expires (outside trusting period) [LCV-INV-TP.1]
    • ErrorNoPeers: if no peers are left to replace secondaries, and no evidence was found before that happened

func CreateEvidenceForPeer(peer PeerID, root LightBlock, trace LightStore)
                          (Evidence, LightBlock, LightStore, result) {

    common := root;

    for i in 1 .. len(trace) {
        auxLS, result := VerifyToTarget(peer, common, trace[i].Header.Height)
  
        if result != ResultSuccess {
            // something went wrong; peer did not provide a verifiable block
            return (nil, nil, nil, FaultyPeer)
        }
        else {
            if auxLS.LatestVerified().Header != trace[i].Header {
                // the header reported by the peer differs from the
                // reference header in trace but both could be
                // verified from common in one step.
                // we can create evidence for submission to the secondary
                ev := new InternalEvidence;
                ev.Evidence.ConflictingBlock := trace[i];
                // CommonHeight is used to indicate the type of attack
                // if the CommonHeight != ConflictingBlock.Height this 
                // is by definition a lunatic attack else it is an
                // equivocation attack
                ev.Evidence.CommonHeight := common.Height;
                ev.Peer := peer
                return (ev, common, auxLS, FoundEvidence)
            }
            else {
                // the peer agrees with the trace, we move common forward.
                // we could delete auxLS as it will be overwritten in
                // the next iteration
                common := trace[i]
            }
        }
    }
    return (nil, nil, nil, NoEvidence)
}
  • Expected precondition
    • root and trace are a verification trace
  • Expected postcondition
    • finds evidence where trace and peer diverge
  • Error condition
    • ErrorTrustExpired: fails if root expires (outside trusting period) [LCV-INV-TP.1]
    • If VerifyToTarget returns error but root is not expired then return FaultyPeer

Correctness arguments

On the existence of evidence

Proposition. In the case of attack, evidence [TMBC-LC-ATTACK-EVIDENCE.1] exists.
Proof. First observe that

  • (A). (NOT E2(i)) implies E1(i+1)

Now by contradiction assume there is no evidence. Thus

  • for all i, we have NOT E1(i) or NOT E2(i)
  • for i = 1 we have E1(1) and thus NOT E2(1) thus by induction on i, by (A) we have for all i that E1(i)
  • from attack we have E2(h-1), and as there is no evidence for i = h - 1 we get NOT E1(h-1). Contradiction. QED.

Argument for [LCD-DIST-INV-ATTACK.1]

Under the assumption that root and trace are a verification trace, when in CreateEvidenceForPeer the detector creates evidence, then the lightclient has seen two different headers (one via trace and one via VerifyToTarget) for the same height that can both be verified in one step.

Argument for [LCD-DIST-INV-STORE.1]

We assume that there is at least one correct peer, and there is no fork. As a result, the correct peer has the correct sequence of blocks. Since the primary_trace is checked block-by-block also against each secondary, and at no point evidence was generated that means at no point there were conflicting blocks.

Argument for [LCD-DIST-LIVE.1]

At the latest when [LCV-INV-TP.1] is violated, AttackDetector terminates.

Argument for [LCD-DIST-TERM-NORMAL.1]

As there are finitely many peers, eventually the main loop terminates. As there is no attack no evidence can be generated.

Argument for [LCD-DIST-TERM-ATTACK.1]

Argument similar to [LCD-DIST-TERM-NORMAL.1]

Argument for [LCD-DIST-SAFE-SECONDARY.1]

Secondaries are only replaced if they time-out or if they report bogus blocks. The former is ruled out by the timing assumption, the latter by correct peers only reporting blocks from the chain.

Argument for [LCD-DIST-SAFE-BOGUS.1]

Once a bogus block is recognized as such the secondary is removed.

References

links to other specifications/ADRs this document refers to

[verification] The specification of the light client verification.

[supervisor] The specification of the light client supervisor.

Results of Discussions and Decisions

  • Generating a minimal proof of fork (as suggested in Issue #5083) is too costly at the light client
    • we do not know all lightblocks from the primary
    • therefore there are many scenarios. we might even need to ask the primary again for additional lightblocks to isolate the branch.

For instance, the light node starts with block at height 1 and the primary provides a block of height 10 that the light node can verify immediately. In cross-checking, a secondary now provides a conflicting header b10 of height 10 that needs another header b5 of height 5 to verify. Now, in order for the light node to convince the primary:

  • The light node cannot just sent b5, as it is not clear whether the fork happened before or after 5
  • The light node cannot just send b10, as the primary would also need b5 for verification
  • In order to minimize the evidence, the light node may try to figure out where the branch happens, e.g., by asking the primary for height 5 (it might be that more queries are required, also to the secondary. However, assuming that in this scenario the primary is faulty it may not respond.

As the main goal is to catch misbehavior of the primary, evidence generation and punishment must not depend on their cooperation. So the moment we have proof of fork (even if it contains several light blocks) we should submit right away.

  • decision: "full" proof of fork consists of two traces that originate in the same lightblock and lead to conflicting headers of the same height.

  • For submission of proof of fork, we may do some optimizations, for instance, we might just submit a trace of lightblocks that verifies a block different from the one the full node knows (we do not send the trace the primary gave us back to the primary)

  • The light client attack is via the primary. Thus we try to catch if the primary installs a bad light block

    • We do not check secondary against secondary
    • For each secondary, we check the primary against one secondary
  • Observe that just two blocks for the same height are not sufficient proof of fork. One of the blocks may be bogus [TMBC-BOGUS.1] which does not constitute slashable behavior.
    Which leads to the question whether the light node should try to do fork detection on its initial block (from subjective initialization). This could be done by doing backwards verification (with the hashes) until a bifurcation block is found. While there are scenarios where a fork could be found, there is also the scenario where a faulty full node feeds the light node with bogus light blocks and forces the light node to check hashes until a bogus chain is out of the trusting period. As a result, the light client should not try to detect a fork for its initial header. The initial header must be trusted as is.

Light Client Sequential Supervisor

TODO: decide where (into which specification) to put the following:

We describe the context on which the fork detector is called by giving a sequential version of the supervisor function. Roughly, it alternates two phases namely:

  • Light Client Verification. As a result, a header of the required height has been downloaded from and verified with the primary.
  • Light Client Fork Detections. As a result the header has been cross-checked with the secondaries. In case there is a fork we submit "proof of fork" and exit.

[LC-FUNC-SUPERVISOR.1]:

func Sequential-Supervisor () (Error) {
    loop {
     // get the next height
        nextHeight := input();
  
  // Verify
        result := NoResult;
        while result != ResultSuccess {
            lightStore,result := VerifyToTarget(primary, lightStore, nextHeight);
            if result == ResultFailure {
    // pick new primary (promote a secondary to primary)
    /// and delete all lightblocks above
             // LastTrusted (they have not been cross-checked)
             Replace_Primary();
   }
        }
  
  // Cross-check
        PoFs := Forkdetector(lightStore, PoFs);
        if PoFs.Empty {
      // no fork detected with secondaries, we trust the new
   // lightblock
            LightStore.Update(testedLB, StateTrusted);
        }
        else {
      // there is a fork, we submit the proofs and exit
            for i, p range PoFs {
                SubmitProofOfFork(p);
            }
            return(ErrorFork);
        }
    }
}

TODO: finish conditions

  • Implementation remark
  • Expected precondition
    • lightStore initialized with trusted header
    • PoFs empty
  • Expected postcondition
    • runs forever, or
    • is terminated by user and satisfies LightStore invariant, or TODO
    • has submitted proof of fork upon detecting a fork
  • Error condition
    • none

Semantics of the LightStore

Currently, a lightblock in the lightstore can be in one of the following states:

  • StateUnverified
  • StateVerified
  • StateFailed
  • StateTrusted

The intuition is that StateVerified captures that the lightblock has been verified with the primary, and StateTrusted is the state after successful cross-checking with the secondaries.

Assuming there is always one correct node among primary and secondaries, and there is no fork on the blockchain, lightblocks that are in StateTrusted can be used by the user with the guarantee of "finality". If a block in StateVerified is used, it might be that detection later finds a fork, and a roll-back might be needed.

Remark: The assumption of one correct node, does not render verification useless. It is true that if the primary and the secondaries return the same block we may trust it. However, if there is a node that provides a different block, the light node still needs verification to understand whether there is a fork, or whether the different block is just bogus (without any support of some previous validator set).

Remark: A light node may choose the full nodes it communicates with (the light node and the full node might even belong to the same stakeholder) so the assumption might be justified in some cases.

In the future, we will do the following changes

  • we assume that only from time to time, the light node is connected to a correct full node
  • this means for some limited time, the light node might have no means to defend against light client attacks
  • as a result we do not have finality
  • once the light node reconnects with a correct full node, it should detect the light client attack and submit evidence.

Under these assumptions, StateTrusted loses its meaning. As a result, it should be removed from the API. We suggest that we replace it with a flag "trusted" that can be used

  • internally for efficiency reasons (to maintain [LCD-INV-TRUSTED-AGREED.1] until a fork is detected)
  • by light client based on the "one correct full node" assumption

Draft of Functions for Fork detection and Proof of Fork Submisstion

This document collects drafts of function for generating and submitting proof of fork in the IBC context

On-chain IBC Component

The following is a suggestions to change the function defined in ICS 007

[TAG-IBC-MISBEHAVIOR.1]

func checkMisbehaviourAndUpdateState(cs: ClientState, PoF: LightNodeProofOfFork)

TODO: finish conditions

  • Implementation remark
  • Expected precondition
    • PoF.TrustedBlock.Header is equal to lightBlock on store with same height
    • both traces end with header of same height
    • headers are different
    • both traces are supported by PoF.TrustedBlock (supports defined in [TMBC-FUNC]), that is, for t = currentTimestamp() (see ICS 024)
      • supports(PoF.TrustedBlock, PoF.PrimaryTrace[1], t)
      • supports(PoF.PrimaryTrace[i], PoF.PrimaryTrace[i+1], t) for 0 < i < length(PoF.PrimaryTrace)
      • supports(PoF.TrustedBlock, PoF.SecondaryTrace[1], t)
      • supports(PoF.SecondaryTrace[i], PoF.SecondaryTrace[i+1], t) for 0 < i < length(PoF.SecondaryTrace)
  • Expected postcondition
    • set cs.FrozenHeight to min(cs.FrozenHeight, PoF.TrustedBlock.Header.Height)
  • Error condition
    • none

The following is a suggestions to add functionality to ICS 002 and 007. I suppose the above is the most efficient way to get the required information. Another option is to subscribe to "header install" events via CosmosSDK

[TAG-IBC-HEIGHTS.1]

func QueryHeightsRange(id, from, to) ([]Height)
  • Expected postcondition
    • returns all heights h, with from <= h <= to for which the IBC component has a consensus state.

This function can be used if the relayer has no information about the IBC component. This allows late-joining relayers to also participate in fork dection and the generation in proof of fork. Alternatively, we may also postulate that relayers are not responsible to detect forks for heights before they started (and subscribed to the transactions reporting fresh headers being installed at the IBC component).

Relayer

Auxiliary Functions to be implemented in the Light Client

[LCV-LS-FUNC-GET-PREV.1]

func (ls LightStore) GetPreviousVerified(height Height) (LightBlock, bool)
  • Expected postcondition
    • returns a verified LightBlock, whose height is maximal among all verified lightblocks with height smaller than height

Relayer Submitting Proof of Fork to the IBC Component

There are two ways the relayer can detect a fork

  • by the fork detector of one of its lightclients
  • be checking the consensus state of the IBC component

The following function ignores how the proof of fork was generated. It takes a proof of fork as input and computes a proof of fork that will be accepted by the IBC component. The problem addressed here is that both, the relayer's light client and the IBC component have incomplete light stores, that might not have all light blocks in common. Hence the relayer has to figure out what the IBC component knows (intuitively, a meeting point between the two lightstores computed in commonRoot) and compute a proof of fork (extendPoF) that the IBC component will accept based on its knowledge.

The auxiliary functions commonRoot and extendPoF are defined below.

[TAG-SUBMIT-POF-IBC.1]

func SubmitIBCProofOfFork(
  lightStore LightStore,
  PoF: LightNodeProofOfFork,
  ibc IBCComponent) (Error) {
    if ibc.queryChainConsensusState(PoF.TrustedBlock.Height) = PoF.TrustedBlock {
  // IBC component has root of PoF on store, we can just submit
        ibc.submitMisbehaviourToClient(ibc.id,PoF)
  return Success
     // note sure about the id parameter
    }
    else {
        // the ibc component does not have the TrustedBlock and might
  // even be on yet a different branch. We have to compute a PoF
  // that the ibc component can verifiy based on its current
        // knowledge
  
        ibcLightBlock, lblock, _, result := commonRoot(lightStore, ibc, PoF.TrustedBlock)

     if result = Success {
   newPoF = extendPoF(ibcLightBlock, lblock, lightStore, PoF)
      ibc.submitMisbehaviourToClient(ibc.id, newPoF)
      return Success
     }
  else{
   return CouldNotGeneratePoF
     }
    }
}

TODO: finish conditions

  • Implementation remark
  • Expected precondition
  • Expected postcondition
  • Error condition
    • none

Auxiliary Functions at the Relayer

If the relayer detects a fork, it has to compute a proof of fork that will convince the IBC component. That is it has to compare the relayer's local lightstore against the lightstore of the IBC component, and find common ancestor lightblocks.

[TAG-COMMON-ROOT.1]

func commonRoot(lightStore LightStore, ibc IBCComponent, lblock
LightBlock) (LightBlock, LightBlock, LightStore, Result) {

    auxLS.Init

       // first we ask for the heights the ibc component is aware of
  ibcHeights = ibc.QueryHeightsRange(
                     ibc.id,
                     lightStore.LowestVerified().Height,
                     lblock.Height - 1);
  // this function does not exist yet. Alternatively, we may
  // request all transactions that installed headers via CosmosSDK
  

        for {
            h, result = max(ibcHeights)
   if result = Empty {
       return (_, _, _, NoRoot)
      }
      ibcLightBlock = ibc.queryChainConsensusState(h)
   auxLS.Update(ibcLightBlock, StateVerified);
      connector, result := Connector(lightStore, ibcLightBlock, lblock.Header.Height)
      if result = success {
       return (ibcLightBlock, connector, auxLS, Success)
   }
   else{
       ibcHeights.remove(h)
      }
  }
}
  • Expected postcondition
    • returns
      • a lightBlock b1 from the IBC component, and
      • a lightBlock b2 from the local lightStore with height less than lblock.Header.Hight, s.t. b1 supports b2, and
      • a lightstore with the blocks downloaded from the ibc component

[TAG-LS-FUNC-CONNECT.1]

func Connector (lightStore LightStore, lb LightBlock, h Height) (LightBlock, bool)
  • Expected postcondition
    • returns a verified LightBlock from lightStore with height less than h that can be verified by lb in one step.

TODO: for the above to work we need an invariant that all verified lightblocks form a chain of trust. Otherwise, we need a lightblock that has a chain of trust to height.

Once the common root is found, a proof of fork that will be accepted by the IBC component needs to be generated. This is done in the following function.

[TAG-EXTEND-POF.1]

func extendPoF (root LightBlock,
                connector LightBlock,
    lightStore LightStore,
    Pof LightNodeProofofFork) (LightNodeProofofFork}
  • Implementation remark
    • PoF is not sufficient to convince an IBC component, so we extend the proof of fork farther in the past
  • Expected postcondition
    • returns a newPOF:
      • newPoF.TrustedBlock = root
      • let prefix = connector + lightStore.Subtrace(connector.Header.Height, PoF.TrustedBlock.Header.Height-1) + PoF.TrustedBlock
        • newPoF.PrimaryTrace = prefix + PoF.PrimaryTrace
        • newPoF.SecondaryTrace = prefix + PoF.SecondaryTrace

Detection a fork at the IBC component

The following functions is assumed to be called regularly to check that latest consensus state of the IBC component. Alternatively, this logic can be executed whenever the relayer is informed (via an event) that a new header has been installed.

[TAG-HANDLER-DETECT-FORK.1]

func DetectIBCFork(ibc IBCComponent, lightStore LightStore) (LightNodeProofOfFork, Error) {
    cs = ibc.queryClientState(ibc);
 lb, found := lightStore.Get(cs.Header.Height)
    if !found {
 **TODO:** need verify to target
        lb, result = LightClient.Main(primary, lightStore, cs.Header.Height)
  // [LCV-FUNC-IBCMAIN.1]
  **TODO** decide what to do following the outcome of Issue #499
  
  // I guess here we have to get into the light client

    }
 if cs != lb {
     // IBC component disagrees with my primary.
  // I fetch the
     ibcLightBlock, lblock, ibcStore, result := commonRoot(lightStore, ibc, lb)
  pof = new LightNodeProofOfFork;
  pof.TrustedBlock := ibcLightBlock
  pof.PrimaryTrace := ibcStore + cs
  pof.SecondaryTrace :=  lightStore.Subtrace(lblock.Header.Height,
                                    lb.Header.Height);
        return(pof, Fork)
 }
 return(nil , NoFork)
}

TODO: finish conditions

  • Implementation remark
    • we ask the handler for the lastest check. Cross-check with the chain. In case they deviate we generate PoF.
    • we assume IBC component is correct. It has verified the consensus state
  • Expected precondition
  • Expected postcondition

Requirements for Fork Detection in the IBC Context

What you need to know about IBC

In the following, I distilled what I considered relevant from

https://github.com/cosmos/ics/tree/master/spec/ics-002-client-semantics

Components and their interface

Tendermint Blockchains

I assume you know what that is.

An IBC/Tendermint correspondence

IBC TermTendermint-RS Spec TermComment
CommitmentRootAppStateapp hash
ConsensusStateLightblocknot all fields are there. NextValidator is definitly needed
ClientStatelatest light block + configuration parameters (e.g., trusting period + frozenHeightNextValidators missing; what is proofSpecs?
frozenHeightheight of forkset when a fork is detected
"would-have-been-fooled"light node fork detectionlight node may submit proof of fork to IBC component to halt it
Height(no epochs)(epoch,height) pair in lexicographical order (compare)
Header~signed headervalidatorSet explicit (no hash); nextValidators missing
Evidencet.b.d.definition unclear "which the light client would have considered valid". Data structure will need to change
verifyValidAndVerifiedsignature does not match perfectly (ClientState vs. LightBlock) + in checkMisbehaviourAndUpdateState it is unclear whether it uses traces or goes to h1 and h2 in one step

Required Changes in ICS 007

  • assert(height > 0) in definition of initialise doesn't match definition of Height as (epoch,height) pair.

  • initialise needs to be updated to new data structures

  • clientState.frozenHeight semantics seem not totally consistent in document. E.g., min needs to be defined over optional value in checkMisbehaviourAndUpdateState. Also, if you are frozen, why do you accept more evidence.

  • checkValidityAndUpdateState

    • verify: it needs to be clarified that checkValidityAndUpdateState does not perform "bisection" (as currently hinted in the text) but performs a single step of "skipping verification", called, ValidAndVerified
    • assert (header.height > clientState.latestHeight): no old headers can be installed. This might be OK, but we need to check interplay with misbehavior
    • clienstState needs to be updated according to complete data structure
  • checkMisbehaviourAndUpdateState: as evidence will contain a trace (or two), the assertion that uses verify will need to change.

  • ICS 002 states w.r.t. queryChainConsensusState that "Note that retrieval of past consensus states by height (as opposed to just the current consensus state) is convenient but not required." For Tendermint fork detection, this seems to be a necessity.

  • Header should become a lightblock

  • Evidence should become LightNodeProofOfFork [LCV-DATA-POF.1]

  • upgradeClientState what is the semantics (in particular what is height doing?).

  • checkMisbehaviourAndUpdateState(cs: ClientState, PoF: LightNodeProofOfFork) needs to be adapted

Handler

A blockchain runs a handler that passively collects information about other blockchains. It can be thought of a state machine that takes input events.

  • the state includes a lightstore (I guess called ConsensusState in IBC)

  • The following function is used to pass a header to a handler

type checkValidityAndUpdateState = (Header) => Void

For Tendermint, it will perform ValidandVerified, that is, it does the trusting period check and the +1/3 check (+2/3 for sequential headers). If it verifies a header, it adds it to its lightstore, if it does not pass verification it drops it. Right now it only accepts a header more recent then the latest header, and drops older ones or ones that could not be verified.

The above paragraph captures what I believe what is the current logic of checkValidityAndUpdateState. It may be subject to change. E.g., maintain a lightstore with state (unverified, verified)

  • The following function is used to pass "evidence" (this we will need to make precise eventually) to a handler
type checkMisbehaviourAndUpdateState = (bytes) => Void

We have to design this, and the data that the handler can use to check that there was some misbehavior (fork) in order react on it, e.g., flagging a situation and stop the protocol.

  • The following function is used to query the light store (ConsensusState)
type queryChainConsensusState = (height: uint64) => ConsensusState

Relayer

  • The active components are called relayer.

  • a relayer contains light clients to two (or more?) blockchains

  • the relayer send headers and data to the handler to invoke checkValidityAndUpdateState and checkMisbehaviourAndUpdateState. It may also query queryChainConsensusState.

  • multiple relayers may talk to one handler. Some relayers might be faulty. We assume existence of at least single correct relayer.

Informal Problem Statement: Fork detection in IBC

Relayer requirement: Evidence for Handler

  • The relayer should provide the handler with "evidence" that there was a fork.

  • The relayer can read the handler's consensus state. Thus the relayer can feed the handler precisely the information the handler needs to detect a fork. What is this information needs to be specified.

  • The information depends on the verification the handler does. It might be necessary to provide a bisection proof (list of lightblocks) so that the handler can verify based on its local lightstore a header h that is conflicting with a header h' in the local lightstore, that is, h != h' and h.Height = h'.Height

Relayer requirement: Fork detection

Let's assume there is a fork at chain A. There are two ways the relayer can figure that out:

  1. as the relayer contains a light client for A, it also includes a fork detector that can detect a fork.

  2. the relayer may also detect a fork by observing that the handler for chain A (on chain B) is on a different branch than the relayer

  • in both detection scenarios, the relayer should submit evidence to full nodes of chain A where there is a fork. As we assume a fullnode has a complete list of blocks, it is sufficient to send "Bucky's evidence" (https://github.com/tendermint/tendermint/issues/5083), that is,

    • two lightblocks from different branches +
    • a lightblock (perhaps just a height) from which both blocks can be verified.
  • in the scenario 2., the relayer must feed the A-handler (on chain B) a proof of a fork on A so that chain B can react accordingly

Handler requirement

  • there are potentially many relayers, some correct some faulty

  • a handler cannot trust the information provided by the relayer, but must verify (Доверя́й, но проверя́й)

  • in case of a fork, we accept that the handler temporarily stores headers (tagged as verified).

  • eventually, a handler should be informed (checkMisbehaviourAndUpdateState) by some relayer that it has verified a header from a fork. Then the handler should do what is required by IBC in this case (stop?)

Challenges in the handler requirement

  • handlers and relayers work on different lightstores. In principle the lightstore need not intersect in any heights a priori

  • if a relayer sees a header h it doesn't know at a handler (queryChainConsensusState), the relayer needs to verify that header. If it cannot do it locally based on downloaded and verified (trusted?) light blocks, it might need to use VerifyToTarget (bisection). To call VerifyToTarget we might keep h in the lightstore. If verification fails, we need to download the "alternative" header of height h.Height to generate evidence for the handler.

  • we have to specify what precisely queryChainConsensusState returns. It cannot be the complete lightstore. Is the last header enough?

  • we would like to assume that every now and then (smaller than the trusting period) a correct relayer checks whether the handler is on a different branch than the relayer. And we would like that this is enough to achieve the Handler requirement.

    • here the correctness argument would be easy if a correct relayer is based on a light client with a trusted state, that is, a light client who never changes its opinion about trusted. Then if such a correct relayer checks-in with a handler, it will detect a fork, and act in time.

    • if the light client does not provide this interface, in the case of a fork, we need some assumption about a correct relayer being on a different branch than the handler, and we need such a relayer to check-in not too late. Also what happens if the relayer's light client is forced to roll-back its lightstore? Does it have to re-check all handlers?

On the interconnectedness of things

In the broader discussion of so-called "fork accountability" there are several subproblems

  • Fork detection

  • Evidence creation and submission

  • Isolating misbehaving nodes (and report them for punishment over abci)

Fork detection

The preliminary specification ./detection.md formalizes the notion of a fork. Roughly, a fork exists if there are two conflicting headers for the same height, where both are supported by bonded full nodes (that have been validators in the near past, that is, within the trusting period). We distinguish between fork on the chain where two conflicting blocks are signed by +2/3 of the validators of that height, and a light client fork where one of the conflicting headers is not signed by +2/3 of the current height, but by +1/3 of the validators of some smaller height.

In principle everyone can detect a fork

  • ./detection talks about the Tendermint light client with a focus on light nodes. A relayer runs such light clients and may detect forks in this way

  • in IBC, a relayer can see that a handler is on a conflicting branch

    • the relayer should feed the handler the necessary information so that it can halt
    • the relayer should report the fork to a full node

Evidence creation and submission

  • the information sent from the relayer to the handler could be called evidence, but this is perhaps a bad idea because the information sent to a full node can also be called evidence. But this evidence might still not be enough as the full node might need to run the "fork accountability" protocol to generate evidence in the form of consensus messages. So perhaps we should introduce different terms for:

    • proof of fork for the handler (basically consisting of lightblocks)
    • proof of fork for a full node (basically consisting of (fewer) lightblocks)
    • proof of misbehavior (consensus messages)

Isolating misbehaving nodes

  • this is the job of a full node.

  • might be subjective in the future: the protocol depends on what the full node believes is the "correct" chain. Right now we postulate that every full node is on the correct chain, that is, there is no fork on the chain.

  • The full node figures out which nodes are

    • lunatic
    • double signing
    • amnesic; using the challenge response protocol
  • We do not punish "phantom" validators

    • currently we understand a phantom validator as a node that
      • signs a block for a height in which it is not in the validator set
      • the node is not part of the +1/3 of previous validators that are used to support the header. Whether we call a validator phantom might be subjective and depend on the header we check against. Their formalization actually seems not so clear.
    • they can only do something if there are +1/3 faulty validators that are either lunatic, double signing, or amnesic.
    • abci requires that we only report bonded validators. So if a node is a "phantom", we would need the check whether the node is bonded, which currently is expensive, as it requires checking blocks from the last three weeks.
    • in the future, with state sync, a correct node might be convinced by faulty nodes that it is in the validator set. Then it might appear to be "phantom" although it behaves correctly

Next steps

The following points are subject to my limited knowledge of the state of the work on IBC. Some/most of it might already exist and we will just need to bring everything together.

  • "proof of fork for a full node" defines a clean interface between fork detection and misbehavior isolation. So it should be produced by protocols (light client, the relayer). So we should fix that first.

  • Given the problems of not having a light client architecture spec, for the relayer we should start with this. E.g.

    • the relayer runs light clients for two chains
    • the relayer regularly queries consensus state of a handler
    • the relayer needs to check the consensus state
      • this involves local checks
      • this involves calling the light client
    • the relayer uses the light client to do IBC business (channels, packets, connections, etc.)
    • the relayer submits proof of fork to handlers and full nodes

the list is definitely not complete. I think part of this (perhaps all) is covered by what Anca presented recently.

We will need to define what we expect from these components

  • for the parts where the relayer talks to the handler, we need to fix the interface, and what the handler does

  • we write specs for these components.

Draft of Light Client Supervisor for discussion

TODOs

This specification in done in parallel with updates on the verification specification. So some hyperlinks have to be placed to the correct files eventually.

Light Client Sequential Supervisor

The light client implements a read operation of a header from the blockchain, by communicating with full nodes, a so-called primary and several so-called witnesses. As some full nodes may be faulty, this functionality must be implemented in a fault-tolerant way.

In the Tendermint blockchain, the validator set may change with every new block. The staking and unbonding mechanism induces a security model: starting at time Time of the header, more than two-thirds of the next validators of a new block are correct for the duration of TrustedPeriod.

Light Client Verification implements the fault-tolerant read operation designed for this security model. That is, it is safe if the model assumptions are satisfied and makes progress if it communicates to a correct primary.

However, if the security model is violated, faulty peers (that have been validators at some point in the past) may launch attacks on the Tendermint network, and on the light client. These attacks as well as an axiomatization of blocks in general are defined in a document that contains the definitions that are currently in detection.md.

If there is a light client attack (but no successful attack on the network), the safety of the verification step may be violated (as we operate outside its basic assumption). The light client also contains a defense mechanism against light clients attacks, called detection.

Light Client Detection implements a cross check of the result of the verification step. If there is a light client attack, and the light client is connected to a correct peer, the light client as a whole is safe, that is, it will not operate on invalid blocks. However, in this case it cannot successfully read, as inconsistent blocks are in the system. However, in this case the detection performs a distributed computation that results in so-called evidence. Evidence can be used to prove to a correct full node that there has been a light client attack.

Light Client Evidence Accountability is a protocol run on a full node to check whether submitted evidence indeed proves the existence of a light client attack. Further, from the evidence and its own knowledge about the blockchain, the full node computes a set of bonded full nodes (that at some point had more than one third of the voting power) that participated in the attack that will be reported via ABCI to the application.

In this document we specify

The details of these two protocols are captured in their own documents, as is the accountability protocol.

Another related line is IBC attack detection and submission at the relayer, as well as attack verification at the IBC handler. This will call for yet another spec.

Status

This document is work in progress. In order to develop the specification step-by-step, it assumes certain details of verification and detection that are not specified in the respective current versions yet. This inconsistencies will be addresses over several upcoming PRs.

Part I - Tendermint Blockchain

See verification spec

Part II - Sequential Problem Definition

[LC-SEQ-INIT-LIVE.1]

Upon initialization, the light client gets as input a header of the blockchain, or the genesis file of the blockchain, and eventually stores a header of the blockchain.

[LC-SEQ-LIVE.1]

The light client gets a sequence of heights as inputs. For each input height targetHeight, it eventually stores the header of height targetHeight.

[LC-SEQ-SAFE.1]

The light client never stores a header which is not in the blockchain.

Part III - Light Client as Distributed System

Computational Model

The light client communicates with remote processes only via the verification and the detection protocols. The respective assumptions are given there.

Distributed Problem Statement

Two Kinds of Liveness

In case of light client attacks, the sequential problem statement cannot always be satisfied. The lightclient cannot decide which block is from the chain and which is not. As a result, the light client just creates evidence, submits it, and terminates. For the liveness property, we thus add the possibility that instead of adding a lightblock, we also might terminate in case there is an attack.

[LC-DIST-TERM.1]

The light client either runs forever or it terminates on attack.

Design choices

[LC-DIST-STORE.1]

The light client has a local data structure called LightStore that contains light blocks (that contain a header).

The light store exposes functions to query and update it. They are specified here.

See light store invariant [LCV-INV-LS-ROOT.2].

[LC-DIST-SAFE.1]

It is always the case that every header in LightStore was generated by an instance of Tendermint consensus.

[LC-DIST-LIVE.1]

Whenever the light client gets a new height h as input,

  • and there is no light client attack up to height h, then the lightclient eventually puts the lightblock of height h in the lightstore and wait for another input.
  • otherwise, that is, if there is a light client attack on height h, then the light client must perform one of the following:
    • it terminates on attack.
    • it eventually puts the lightblock of height h in the lightstore and wait for another input.

Observe that the "existence of a lightclient attack" just means that some node has generated a conflicting block. It does not necessarily mean that a (faulty) peer sends such a block to "our" lightclient. Thus, even if there is an attack somewhere in the system, our lightclient might still continue to operate normally.

Solving the sequential specification

[LC-DIST-SAFE.1] is guaranteed by the detector; in particular it follows from [LCD-DIST-INV-STORE.1] [LCD-DIST-LIVE.1]

Part IV - Light Client Supervisor Protocol

We provide a specification for a sequential Light Client Supervisor. The local code for verification is presented by a sequential function Sequential-Supervisor to highlight the control flow of this functionality. Each lightblock is first verified with a primary, and then cross-checked with secondaries, and if all goes well, the lightblock is added (with the attribute "trusted") to the lightstore. Intermiate lightblocks that were used to verify the target block but were not cross-checked are stored as "verified"

We note that if a different concurrency model is considered for an implementation, the semantics of the lightstore might change: In a concurrent implementation, we might do verification for some height h, add the lightblock to the lightstore, and start concurrent threads that

  • do verification for the next height h' != h
  • do cross-checking for height h. If we find an attack, we remove h from the lightstore.
  • the user might already start to use h

Thus, this concurrency model changes the semantics of the lightstore (not all lightblocks that are read by the user are trusted; they may be removed if we find a problem). Whether this is desirable, and whether the gain in performance is worth it, we keep for future versions/discussion of lightclient protocols.

Definitions

Peers

[LC-DATA-PEERS.1]:

A fixed set of full nodes is provided in the configuration upon initialization. Initially this set is partitioned into

  • one full node that is the primary (singleton set),
  • a set Secondaries (of fixed size, e.g., 3),
  • a set FullNodes; it excludes primary and Secondaries nodes.
  • A set FaultyNodes of nodes that the light client suspects of being faulty; it is initially empty

[LC-INV-NODES.1]:

The detector shall maintain the following invariants:

  • FullNodes \intersect Secondaries = {}
  • FullNodes \intersect FaultyNodes = {}
  • Secondaries \intersect FaultyNodes = {}

and the following transition invariant

  • FullNodes' \union Secondaries' \union FaultyNodes' = FullNodes \union Secondaries \union FaultyNodes

[LC-FUNC-REPLACE-PRIMARY.1]:

Replace_Primary(root-of-trust LightBlock)
  • Implementation remark
    • the primary is replaced by a secondary
    • to maintain a constant size of secondaries, need to
      • pick a new secondary nsec while ensuring [LC-INV-ROOT-AGREED.1]
      • that is, we need to ensure that root-of-trust = FetchLightBlock(nsec, root-of-trust.Header.Height)
  • Expected precondition
    • FullNodes is nonempty
  • Expected postcondition
    • primary is moved to FaultyNodes
    • a secondary s is moved from Secondaries to primary
  • Error condition
    • if precondition is violated

[LC-FUNC-REPLACE-SECONDARY.1]:

Replace_Secondary(addr Address, root-of-trust LightBlock)
  • Implementation remark
    • maintain [LC-INV-ROOT-AGREED.1], that is, ensure root-of-trust = FetchLightBlock(nsec, root-of-trust.Header.Height)
  • Expected precondition
    • FullNodes is nonempty
  • Expected postcondition
    • addr is moved from Secondaries to FaultyNodes
    • an address nsec is moved from FullNodes to Secondaries
  • Error condition
    • if precondition is violated

Data Types

The core data structure of the protocol is the LightBlock.

[LC-DATA-LIGHTBLOCK.1]

type LightBlock struct {
                Header          Header
                Commit          Commit
                Validators      ValidatorSet
                NextValidators  ValidatorSet
                Provider        PeerID
}

[LC-DATA-LIGHTSTORE.1]

LightBlocks are stored in a structure which stores all LightBlock from initialization or received from peers.

type LightStore struct {
        ...
}

We use the functions that the LightStore exposes, which are defined in the verification specification.

Inputs

The lightclient is initialized with LCInitData

[LC-DATA-INIT.1]

type LCInitData struct {
    lightBlock     LightBlock
    genesisDoc     GenesisDoc
}

where only one of the components must be provided. GenesisDoc is defined in the Tenderdash Types.

[LC-DATA-GENESIS.1]

type GenesisDoc struct {
    GenesisTime     time.Time                `json:"genesis_time"`
    ChainID         string                   `json:"chain_id"`
    InitialHeight   int64                    `json:"initial_height"`
    ConsensusParams *tmproto.ConsensusParams `json:"consensus_params,omitempty"`
    Validators      []GenesisValidator       `json:"validators,omitempty"`
    AppHash         tmbytes.HexBytes         `json:"app_hash"`
    AppState        json.RawMessage          `json:"app_state,omitempty"`
}

We use the following function makeblock so that we create a lightblock from the genesis file in order to do verification based on the data from the genesis file using the same verification function we use in normal operation.

[LC-FUNC-MAKEBLOCK.1]

func makeblock (genesisDoc GenesisDoc) (lightBlock LightBlock))
  • Implementation remark
    • none
  • Expected precondition
    • none
  • Expected postcondition
    • lightBlock.Header.Height = genesisDoc.InitialHeight
    • lightBlock.Header.Time = genesisDoc.GenesisTime
    • lightBlock.Header.LastBlockID = nil
    • lightBlock.Header.LastCommit = nil
    • lightBlock.Header.Validators = genesisDoc.Validators
    • lightBlock.Header.NextValidators = genesisDoc.Validators
    • lightBlock.Header.Data = nil
    • lightBlock.Header.AppState = genesisDoc.AppState
    • lightBlock.Header.LastResult = nil
    • lightBlock.Commit = nil
    • lightBlock.Validators = genesisDoc.Validators
    • lightBlock.NextValidators = genesisDoc.Validators
    • lightBlock.Provider = nil
  • Error condition
    • none

Configuration Parameters

[LC-INV-ROOT-AGREED.1]

In the Sequential-Supervisor, it is always the case that the primary and all secondaries agree on lightStore.Latest().

Assumptions

We have to assume that the initialization data (the lightblock or the genesis file) are consistent with the blockchain. This is subjective initialization and it cannot be checked locally.

Invariants

[LC-INV-PEERLIST.1]:

The peer list contains a primary and a secondary.

If the invariant is violated, the light client does not have enough peers to download headers from. As a result, the light client needs to terminate in case this invariant is violated.

Supervisor

Outline

The supervisor implements the functionality of the lightclient. It is initialized with a genesis file or with a lightblock the user trusts. This initialization is subjective, that is, the security of the lightclient is based on the validity of the input. If the genesis file or the lightblock deviate from the actual ones on the blockchain, the lightclient provides no guarantees.

After initialization, the supervisor awaits an input, that is, the height of the next lightblock that should be obtained. Then it downloads, verifies, and cross-checks a lightblock, and if all tests go through, the light block (and possibly other lightblocks) are added to the lightstore, which is returned in an output event to the user.

The following main loop does the interaction with the user (input, output) and calls the following two functions:

  • InitLightClient: it initializes the lightstore either with the provided lightblock or with the lightblock that corresponds to the first block generated by the blockchain (by the validators defined by the genesis file)
  • VerifyAndDetect: takes as input a lightstore and a height and returns the updated lightstore.

[LC-FUNC-SUPERVISOR.1]:

func Sequential-Supervisor (initdata LCInitData) (Error) {

    lightStore,result := InitLightClient(initData);
    if result != OK {
        return result;
    }

    loop {
        // get the next height
        nextHeight := input();
  
        lightStore,result := VerifyAndDetect(lightStore, nextHeight);
  
        if result == OK {
            output(LightStore.Get(targetHeight));
   // we only output a trusted lightblock
        }
        else {
            return result
        }
        // QUESTION: is it OK to generate output event in normal case,
        // and terminate with failure in the (light client) attack case?
    }
}
  • Implementation remark
    • infinite loop unless a light client attack is detected
    • In typical implementations (e.g., the one in Rust), there are mutliple input actions: VerifytoLatest, LatestTrusted, and GetStatus. The information can be easily obtained from the lightstore, so that we do not treat these requests explicitly here but just consider the request for a block of a given height which requires more involved computation and communication.
  • Expected precondition
    • LCInitData contains a genesis file or a lightblock.
  • Expected postcondition
    • if a light client attack is detected: it stops and submits evidence (in InitLightClient or VerifyAndDetect)
    • otherwise: non. It runs forever.
  • Invariant: lightStore contains trusted lightblocks only.
  • Error condition
    • if InitLightClient or VerifyAndDetect fails (if a attack is detected, or if [LCV-INV-TP.1] is violated)

Details of the Functions

Initialization

The light client is based on subjective initialization. It has to trust the initial data given to it by the user. It cannot do any detection of attack. So either upon initialization we obtain a lightblock and just initialize the lightstore with it. Or in case of a genesis file, we download, verify, and cross-check the first block, to initialize the lightstore with this first block. The reason is that we want to maintain [LCV-INV-TP.1] from the beginning.

If the lightclient is initialized with a lightblock, one might think it may increase trust, when one cross-checks the initial light block. However, if a peer provides a conflicting lightblock, the question is to distinguish the case of a bogus block (upon which operation should proceed) from a light client attack (upon which operation should stop). In case of a bogus block, the lightclient might be forced to do backwards verification until the blocks are out of the trusting period, to make sure no previous validator set could have generated the bogus block, which effectively opens up a DoS attack on the lightclient without adding effective robustness.

[LC-FUNC-INIT.1]:

func InitLightClient (initData LCInitData) (LightStore, Error) {

    if LCInitData.LightBlock != nil {
        // we trust the provided initial block.
        newblock := LCInitData.LightBlock
    }
    else {
        genesisBlock := makeblock(initData.genesisDoc);

        result := NoResult;
        while result != ResultSuccess {
            current = FetchLightBlock(PeerList.primary(), genesisBlock.Header.Height + 1)
            // QUESTION: is the height with "+1" OK?

            if CANNOT_VERIFY = ValidAndVerify(genesisBlock, current) {
                Replace_Primary();
            }
            else {
                result = ResultSuccess
            }
        }
  
        // cross-check
  auxLS := new LightStore
  auxLS.Add(current)
        Evidences := AttackDetector(genesisBlock, auxLS)
        if Evidences.Empty {
            newBlock := current
        }
        else {
            // [LC-SUMBIT-EVIDENCE.1]
            submitEvidence(Evidences);
            return(nil, ErrorAttack);
        }
    }

    lightStore := new LightStore;
    lightStore.Add(newBlock);
    return (lightStore, OK);
}

  • Implementation remark
    • none
  • Expected precondition
    • LCInitData contains either a genesis file of a lightblock
    • if genesis it passes ValidateAndComplete() see Tendermint
  • Expected postcondition
    • lightStore initialized with trusted lightblock. It has either been cross-checked (from genesis) or it has initial trust from the user.
  • Error condition
    • if precondition is violated
    • empty peerList

Main verification and detection logic

[LC-FUNC-MAIN-VERIF-DETECT.1]:

func VerifyAndDetect (lightStore LightStore, targetHeight Height)
                     (LightStore, Result) {

    b1, r1 = lightStore.Get(targetHeight)
    if r1 == true {
        if b1.State == StateTrusted {
            // block already there and trusted
            return (lightStore, ResultSuccess)
  }
  else {
            // We have a lightblock in the store, but it has not been 
            // cross-checked by now. We do that now.
            root_of_trust, auxLS := lightstore.TraceTo(b1);
   
            // Cross-check
            Evidences := AttackDetector(root_of_trust, auxLS);
            if Evidences.Empty {
                // no attack detected, we trust the new lightblock
                lightStore.Update(auxLS.Latest(), 
                                  StateTrusted, 
                                  verfiedLS.Latest().verification-root);
                return (lightStore, OK);
            }
            else {
                // there is an attack, we exit
  submitEvidence(Evidences);
                return(lightStore, ErrorAttack);
            }
        }
    }

    // get the lightblock with maximum height smaller than targetHeight
    // would typically be the heighest, if we always move forward
    root_of_trust, r2 = lightStore.LatestPrevious(targetHeight);

    if r2 = false {
        // there is no lightblock from which we can do forward
        // (skipping) verification. Thus we have to go backwards.
        // No cross-check needed. We trust hashes. Therefore, we
        // directly return the result
        return Backwards(primary, lightStore.Lowest(), targetHeight)
    }
    else {
        // Forward verification + detection
        result := NoResult;
        while result != ResultSuccess {
            verifiedLS,result := VerifyToTarget(primary,
                                                root_of_trust,
                                                nextHeight);
            if result == ResultFailure {
                // pick new primary (promote a secondary to primary)
                Replace_Primary(root_of_trust);
            }
            else if result == ResultExpired {
                return (lightStore, result)
            }
        }

        // Cross-check
        Evidences := AttackDetector(root_of_trust, verifiedLS);
        if Evidences.Empty {
            // no attack detected, we trust the new lightblock
            verifiedLS.Update(verfiedLS.Latest(), 
                              StateTrusted, 
                              verfiedLS.Latest().verification-root);
            lightStore.store_chain(verifidLS);
            return (lightStore, OK);
        }
        else {
            // there is an attack, we exit
            return(lightStore, ErrorAttack);
        }
    }
}
  • Implementation remark
    • none
  • Expected precondition
    • none
  • Expected postcondition
    • lightblock of height targetHeight (and possibly additional blocks) added to lightStore
  • Error condition
    • an attack is detected
    • [LC-DATA-PEERLIST-INV.1] is violated

Draft of Light Client Supervisor for discussion

Modification to the initialization

The lightclient is initialized with LCInitData

[LC-DATA-INIT.2]

type LCInitData struct {
    TrustedBlock   LightBlock
    Genesis        GenesisDoc
    TrustedHash    []byte
    TrustedHeight  int64
}

where only one of the components must be provided. GenesisDoc is defined in the Tenderdash Types.

Initialization

The light client is based on subjective initialization. It has to trust the initial data given to it by the user. It cannot perform any detection of an attack yet instead requires an initial point of trust. There are three forms of initial data which are used to obtain the first trusted block:

  • A trusted block from a prior initialization
  • A trusted height and hash
  • A genesis file

The golang light client implementation checks this initial data in that order; first attempting to find a trusted block from the trusted store, then acquiring a light block from the primary at the trusted height and matching the hash, or finally checking for a genesis file to verify the initial header.

The light client doesn't need to check if the trusted block is within the trusted period because it already trusts it, however, if the light block is outside the trust period, there is a higher chance the light client won't be able to verify anything.

Cross-checking this trusted block with providers upon initialization is helpful for ensuring that the node is responsive and correctly configured but does not increase trust since proving a conflicting block is a light client attack and not just a bogus block could result in performing backwards verification beyond the trusted period, thus a fruitless endeavour.

However, with the notion of it's better to fail earlier than later, the golang light client implementation will perform a consistency check on all providers and will error if one returns a different header, allowing the user the opportunity to reinitialize.

[LC-FUNC-INIT.2]:

func InitLightClient(initData LCInitData) (LightStore, Error) {
    var initialBlock LightBlock

    switch {
    case LCInitData.TrustedBlock != nil:
        // we trust the block from a prior initialization
        initialBlock = LCInitData.TrustedBlock

    case LCInitData.TrustedHash != nil:
        untrustedBlock := FetchLightBlock(PeerList.Primary(), LCInitData.TrustedHeight)
        

        // verify that the hashes match
        if untrustedBlock.Hash() != LCInitData.TrustedHash {
            return nil, Error("Primary returned block with different hash")
        }
        // after checking the hash we now trust the block
        initialBlock = untrustedBlock        
    }
    case LCInitData.Genesis != nil:
        untrustedBlock := FetchLightBlock(PeerList.Primary(), LCInitData.Genesis.InitialHeight)
        
        // verify that 2/3+ of the validator set signed the untrustedBlock
        if err := VerifyCommitFull(untrustedBlock.Commit, LCInitData.Genesis.Validators); err != nil {
            return nil, err
        }

        // we can now trust the block
        initialBlock = untrustedBlock
    default:
        return nil, Error("No initial data was provided")

    // This is done in the golang version but is optional and not strictly part of the protocol
    if err := CrossCheck(initialBlock, PeerList.Witnesses()); err != nil {
        return nil, err
    }

    // initialize light store
    lightStore := new LightStore;
    lightStore.Add(newBlock);
    return (lightStore, OK);
}

func CrossCheck(lb LightBlock, witnesses []Provider) error {
    for _, witness := range witnesses {
        witnessBlock := FetchLightBlock(witness, lb.Height)

        if witnessBlock.Hash() != lb.Hash() {
            return Error("Witness has different block")
        }
    }
    return OK
}

  • Implementation remark
    • none
  • Expected precondition
    • LCInitData contains either a genesis file of a lightblock
    • if genesis it passes ValidateAndComplete() see Tendermint
  • Expected postcondition
    • lightStore initialized with trusted lightblock. It has either been cross-checked (from genesis) or it has initial trust from the user.
  • Error condition
    • if precondition is violated
    • empty peerList

Core Verification

Problem statement

We assume that the light client knows a (base) header inithead it trusts (by social consensus or because the light client has decided to trust the header before). The goal is to check whether another header newhead can be trusted based on the data in inithead.

The correctness of the protocol is based on the assumption that inithead was generated by an instance of Tendermint consensus.

Failure Model

For the purpose of the following definitions we assume that there exists a function validators that returns the corresponding validator set for the given hash.

The light client protocol is defined with respect to the following failure model:

Given a known bound TRUSTED_PERIOD, and a block b with header h generated at time Time (i.e. h.Time = Time), a set of validators that hold more than 2/3 of the voting power in validators(b.Header.NextValidatorsHash) is correct until time b.Header.Time + TRUSTED_PERIOD.

Assumption: "correct" is defined w.r.t. realtime (some Newtonian global notion of time, i.e., wall time), while Header.Time corresponds to the BFT time. In this note, we assume that clocks of correct processes are synchronized (for example using NTP), and therefore there is bounded clock drift (CLOCK_DRIFT) between local clocks and BFT time. More precisely, for every correct light client process and every header.Time (i.e. BFT Time, for a header correctly generated by the Tendermint consensus), the following inequality holds: Header.Time < now + CLOCK_DRIFT, where now corresponds to the system clock at the light client process.

Furthermore, we assume that TRUSTED_PERIOD is (several) order of magnitude bigger than CLOCK_DRIFT (TRUSTED_PERIOD >> CLOCK_DRIFT), as CLOCK_DRIFT (using NTP) is in the order of milliseconds and TRUSTED_PERIOD is in the order of weeks.

We expect a light client process defined in this document to be used in the context in which there is some larger period during which misbehaving validators can be detected and punished (we normally refer to it as UNBONDING_PERIOD due to the "bonding" mechanism in modern proof of stake systems). Furthermore, we assume that TRUSTED_PERIOD < UNBONDING_PERIOD and that they are normally of the same order of magnitude, for example TRUSTED_PERIOD = UNBONDING_PERIOD / 2.

The specification in this document considers an implementation of the light client under the Failure Model defined above. Mechanisms like fork accountability and evidence submission are defined in the context of UNBONDING_PERIOD and they incentivize validators to follow the protocol specification defined in this document. If they don't, and we have 1/3 (or more) faulty validators, safety may be violated. Our approach then is to detect these cases (after the fact), and take suitable repair actions (automatic and social). This is discussed in document on Fork accountability.

The term "trusted" above indicates that the correctness of the protocol depends on this assumption. It is in the responsibility of the user that runs the light client to make sure that the risk of trusting a corrupted/forged inithead is negligible.

Remark: This failure model might change to a hybrid version that takes heights into account in the future.

High Level Solution

Upon initialization, the light client is given a header inithead it trusts (by social consensus). When a light clients sees a new signed header snh, it has to decide whether to trust the new header. Trust can be obtained by (possibly) the combination of three methods.

  1. Uninterrupted sequence of headers. Given a trusted header h and an untrusted header h1, the light client trusts a header h1 if it trusts all headers in between h and h1.

  2. Trusted period. Given a trusted header h, an untrusted header h1 > h and TRUSTED_PERIOD during which the failure model holds, we can check whether at least one validator, that has been continuously correct from h.Time until now, has signed h1. If this is the case, we can trust h1.

  3. Bisection. If a check according to 2. (trusted period) fails, the light client can try to obtain a header hp whose height lies between h and h1 in order to check whether h can be used to get trust for hp, and hp can be used to get trust for snh. If this is the case we can trust h1; if not, we continue recursively until either we found set of headers that can build (transitively) trust relation between h and h1, or we failed as two consecutive headers don't verify against each other.

Definitions

Data structures

In the following, only the details of the data structures needed for this specification are given.

  type Header struct {
       Height               int64
       Time                 Time          // the chain time when the header (block) was generated

       LastBlockID          BlockID       // prev block info
       ValidatorsHash       []byte        // hash of the validators for the current block
       NextValidatorsHash   []byte        // hash of the validators for the next block
  }

  type SignedHeader struct {
       Header        Header
       Commit        Commit            // commit for the given header
  }

  type ValidatorSet struct {
       Validators         []Validator
       TotalVotingPower   int64
  }

  type Validator struct {
       Address       Address           // validator address (we assume validator's addresses are unique)
       VotingPower   int64             // validator's voting power
  }

  type TrustedState {
       SignedHeader   SignedHeader
       ValidatorSet   ValidatorSet
  }

Functions

For the purpose of this light client specification, we assume that the Tendermint Full Node exposes the following functions over Tendermint RPC:

    // returns signed header: Header with Commit, for the given height
    func Commit(height int64) (SignedHeader, error)

    // returns validator set for the given height
    func Validators(height int64) (ValidatorSet, error)

Furthermore, we assume the following auxiliary functions:

    // returns true if the commit is for the header, ie. if it contains
    // the correct hash of the header; otherwise false
    func matchingCommit(header Header, commit Commit) bool

    // returns the set of validators from the given validator set that
    // committed the block (that correctly signed the block)
    // it assumes signature verification so it can be computationally expensive
    func signers(commit Commit, validatorSet ValidatorSet) []Validator

    // returns the voting power the validators in v1 have according to their voting power in set v2
    // it does not assume signature verification
    func votingPowerIn(v1 []Validator, v2 ValidatorSet) int64

    // returns hash of the given validator set
    func hash(v2 ValidatorSet) []byte

In the functions below we will be using trustThreshold as a parameter. For simplicity we assume that trustThreshold is a float between 1/3 and 2/3 and we will not be checking it in the pseudo-code.

VerifySingle. The function VerifySingle attempts to validate given untrusted header and the corresponding validator sets based on a given trusted state. It ensures that the trusted state is still within its trusted period, and that the untrusted header is within assumed clockDrift bound of the passed time now. Note that this function is not making external (RPC) calls to the full node; the whole logic is based on the local (given) state. This function is supposed to be used by the IBC handlers.

func VerifySingle(untrustedSh SignedHeader,
                  untrustedVs ValidatorSet,
                  untrustedNextVs ValidatorSet,
                  trustedState TrustedState,
                  trustThreshold float,
                  trustingPeriod Duration,
                  clockDrift Duration,
                  now Time) (TrustedState, error) {

    if untrustedSh.Header.Time > now + clockDrift {
        return (trustedState, ErrInvalidHeaderTime)
    }

    trustedHeader = trustedState.SignedHeader.Header
    if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
        return (state, ErrHeaderNotWithinTrustedPeriod)
    }

    // we assume that time it takes to execute verifySingle function
    // is several order of magnitudes smaller than trustingPeriod
    error = verifySingle(
                trustedState,
                untrustedSh,
                untrustedVs,
                untrustedNextVs,
                trustThreshold)

    if error != nil return (state, error)

    // the untrusted header is now trusted
    newTrustedState = TrustedState(untrustedSh, untrustedNextVs)
    return (newTrustedState, nil)
}

// return true if header is within its light client trusted period; otherwise returns false
func isWithinTrustedPeriod(header Header,
                           trustingPeriod Duration,
                           now Time) bool {

    return header.Time + trustedPeriod > now
}

Note that in case VerifySingle returns without an error (untrusted header is successfully verified) then we have a guarantee that the transition of the trust from trustedState to newTrustedState happened during the trusted period of trustedState.SignedHeader.Header.

TODO: Explain what happens in case VerifySingle returns with an error.

verifySingle. The function verifySingle verifies a single untrusted header against a given trusted state. It includes all validations and signature verification. It is not publicly exposed since it does not check for header expiry (time constraints) and hence it's possible to use it incorrectly.

func verifySingle(trustedState TrustedState,
                  untrustedSh SignedHeader,
                  untrustedVs ValidatorSet,
                  untrustedNextVs ValidatorSet,
                  trustThreshold float) error {

    untrustedHeader = untrustedSh.Header
    untrustedCommit = untrustedSh.Commit

    trustedHeader = trustedState.SignedHeader.Header
    trustedVs = trustedState.ValidatorSet

    if trustedHeader.Height >= untrustedHeader.Height return ErrNonIncreasingHeight
    if trustedHeader.Time >= untrustedHeader.Time return ErrNonIncreasingTime

    // validate the untrusted header against its commit, vals, and next_vals
    error = validateSignedHeaderAndVals(untrustedSh, untrustedVs, untrustedNextVs)
    if error != nil return error

    // check for adjacent headers
    if untrustedHeader.Height == trustedHeader.Height + 1 {
        if trustedHeader.NextValidatorsHash != untrustedHeader.ValidatorsHash {
            return ErrInvalidAdjacentHeaders
        }
    } else {
        error = verifyCommitTrusting(trustedVs, untrustedCommit, untrustedVs, trustThreshold)
        if error != nil return error
    }

    // verify the untrusted commit
    return verifyCommitFull(untrustedVs, untrustedCommit)
}

// returns nil if header and validator sets are consistent; otherwise returns error
func validateSignedHeaderAndVals(signedHeader SignedHeader, vs ValidatorSet, nextVs ValidatorSet) error {
    header = signedHeader.Header
    if hash(vs) != header.ValidatorsHash return ErrInvalidValidatorSet
    if hash(nextVs) != header.NextValidatorsHash return ErrInvalidNextValidatorSet
    if !matchingCommit(header, signedHeader.Commit) return ErrInvalidCommitValue
    return nil
}

// returns nil if at least single correst signer signed the commit; otherwise returns error
func verifyCommitTrusting(trustedVs ValidatorSet,
                          commit Commit,
                          untrustedVs ValidatorSet,
                          trustLevel float) error {

    totalPower := trustedVs.TotalVotingPower
    signedPower := votingPowerIn(signers(commit, untrustedVs), trustedVs)

    // check that the signers account for more than max(1/3, trustLevel) of the voting power
    // this ensures that there is at least single correct validator in the set of signers
    if signedPower < max(1/3, trustLevel) * totalPower return ErrInsufficientVotingPower
    return nil
}

// returns nil if commit is signed by more than 2/3 of voting power of the given validator set
// return error otherwise
func verifyCommitFull(vs ValidatorSet, commit Commit) error {
    totalPower := vs.TotalVotingPower;
    signedPower := votingPowerIn(signers(commit, vs), vs)

    // check the signers account for +2/3 of the voting power
    if signedPower * 3 <= totalPower * 2 return ErrInvalidCommit
    return nil
}

VerifyHeaderAtHeight. The function VerifyHeaderAtHeight captures high level logic, i.e., application call to the light client module to download and verify header for some height.

func VerifyHeaderAtHeight(untrustedHeight int64,
                          trustedState TrustedState,
                          trustThreshold float,
                          trustingPeriod Duration,
                          clockDrift Duration) (TrustedState, error)) {

    trustedHeader := trustedState.SignedHeader.Header

    now := System.Time()
    if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
        return (trustedState, ErrHeaderNotWithinTrustedPeriod)
    }

    newTrustedState, err := VerifyBisection(untrustedHeight,
                                            trustedState,
                                            trustThreshold,
                                            trustingPeriod,
                                            clockDrift,
                                            now)

    if err != nil return (trustedState, err)

    now = System.Time()
    if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
        return (trustedState, ErrHeaderNotWithinTrustedPeriod)
    }

    return (newTrustedState, err)
}

Note that in case VerifyHeaderAtHeight returns without an error (untrusted header is successfully verified) then we have a guarantee that the transition of the trust from trustedState to newTrustedState happened during the trusted period of trustedState.SignedHeader.Header.

In case VerifyHeaderAtHeight returns with an error, then either (i) the full node we are talking to is faulty or (ii) the trusted header has expired (it is outside its trusted period). In case (i) the full node is faulty so light client should disconnect and reinitialise with new peer. In the case (ii) as the trusted header has expired, we need to reinitialise light client with a new trusted header (that is within its trusted period), but we don't necessarily need to disconnect from the full node we are talking to (as we haven't observed full node misbehavior in this case).

VerifyBisection. The function VerifyBisection implements recursive logic for checking if it is possible building trust relationship between trustedState and untrusted header at the given height over finite set of (downloaded and verified) headers.

func VerifyBisection(untrustedHeight int64,
                     trustedState TrustedState,
                     trustThreshold float,
                     trustingPeriod Duration,
                     clockDrift Duration,
                     now Time) (TrustedState, error) {

    untrustedSh, error := Commit(untrustedHeight)
    if error != nil return (trustedState, ErrRequestFailed)

    untrustedHeader = untrustedSh.Header

    // note that we pass now during the recursive calls. This is fine as
    // all other untrusted headers we download during recursion will be
    // for a smaller heights, and therefore should happen before.
    if untrustedHeader.Time > now + clockDrift {
        return (trustedState, ErrInvalidHeaderTime)
    }

    untrustedVs, error := Validators(untrustedHeight)
    if error != nil return (trustedState, ErrRequestFailed)

    untrustedNextVs, error := Validators(untrustedHeight + 1)
    if error != nil return (trustedState, ErrRequestFailed)

    error = verifySingle(
             trustedState,
             untrustedSh,
             untrustedVs,
             untrustedNextVs,
             trustThreshold)

    if fatalError(error) return (trustedState, error)

    if error == nil {
        // the untrusted header is now trusted.
        newTrustedState = TrustedState(untrustedSh, untrustedNextVs)
        return (newTrustedState, nil)
    }

    // at this point in time we need to do bisection
    pivotHeight := ceil((trustedHeader.Height + untrustedHeight) / 2)

    error, newTrustedState = VerifyBisection(pivotHeight,
                                             trustedState,
                                             trustThreshold,
                                             trustingPeriod,
                                             clockDrift,
                                             now)
    if error != nil return (newTrustedState, error)

    return VerifyBisection(untrustedHeight,
                           newTrustedState,
                           trustThreshold,
                           trustingPeriod,
                           clockDrift,
                           now)
}

func fatalError(err) bool {
    return err == ErrHeaderNotWithinTrustedPeriod OR
           err == ErrInvalidAdjacentHeaders OR
           err == ErrNonIncreasingHeight OR
           err == ErrNonIncreasingTime OR
           err == ErrInvalidValidatorSet OR
           err == ErrInvalidNextValidatorSet OR
           err == ErrInvalidCommitValue OR
           err == ErrInvalidCommit
}

The case untrustedHeader.Height < trustedHeader.Height

In the use case where someone tells the light client that application data that is relevant for it can be read in the block of height k and the light client trusts a more recent header, we can use the hashes to verify headers "down the chain." That is, we iterate down the heights and check the hashes in each step.

Remark. For the case were the light client trusts two headers i and j with i < k < j, we should discuss/experiment whether the forward or the backward method is more effective.

func VerifyHeaderBackwards(trustedHeader Header,
                           untrustedHeader Header,
                           trustingPeriod Duration,
                           clockDrift Duration) error {

  if untrustedHeader.Height >= trustedHeader.Height return ErrErrNonDecreasingHeight
  if untrustedHeader.Time >= trustedHeader.Time return ErrNonDecreasingTime

  now := System.Time()
  if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
    return ErrHeaderNotWithinTrustedPeriod
  }

  old := trustedHeader
  for i := trustedHeader.Height - 1; i > untrustedHeader.Height; i-- {
    untrustedSh, error := Commit(i)
    if error != nil return ErrRequestFailed

    if (hash(untrustedSh.Header) != old.LastBlockID.Hash) {
      return ErrInvalidAdjacentHeaders
    }

    old := untrustedSh.Header
  }

  if hash(untrustedHeader) != old.LastBlockID.Hash {
    return ErrInvalidAdjacentHeaders
  }

  now := System.Time()
  if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
    return ErrHeaderNotWithinTrustedPeriod
  }

  return nil
 }

Assumption: In the following, we assume that untrusted_h.Header.height > trusted_h.Header.height. We will quickly discuss the other case in the next section.

We consider the following set-up:

  • the light client communicates with one full node
  • the light client locally stores all the headers that has passed basic verification and that are within light client trust period. In the pseudo code below we write Store.Add(header) for this. If a header failed to verify, then the full node we are talking to is faulty and we should disconnect from it and reinitialise with new peer.
  • If CanTrust returns error, then the light client has seen a forged header or the trusted header has expired (it is outside its trusted period).
    • In case of forged header, the full node is faulty so light client should disconnect and reinitialise with new peer. If the trusted header has expired, we need to reinitialise light client with new trusted header (that is within its trusted period), but we don't necessarily need to disconnect from the full node we are talking to (as we haven't observed full node misbehavior in this case).

Correctness of the Light Client Protocols

Definitions

  • TRUSTED_PERIOD: trusted period
  • for realtime t, the predicate correct(v,t) is true if the validator v follows the protocol until time t (we will see about recovery later).
  • Validator fields. We will write a validator as a tuple (v,p) such that
    • v is the identifier (i.e., validator address; we assume identifiers are unique in each validator set)
    • p is its voting power
  • For each header h, we write trust(h) = true if the light client trusts h.

Failure Model

If a block b with a header h is generated at time Time (i.e. h.Time = Time), then a set of validators that hold more than 2/3 of the voting power in validators(h.NextValidatorsHash) is correct until time h.Time + TRUSTED_PERIOD.

Formally, [ \sum_{(v,p) \in validators(h.NextValidatorsHash) \wedge correct(v,h.Time + TRUSTED_PERIOD)} p > 2/3 \sum_{(v,p) \in validators(h.NextValidatorsHash)} p ]

The light client communicates with a full node and learns new headers. The goal is to locally decide whether to trust a header. Our implementation needs to ensure the following two properties:

  • Light Client Completeness: If a header h was correctly generated by an instance of Tendermint consensus (and its age is less than the trusted period), then the light client should eventually set trust(h) to true.

  • Light Client Accuracy: If a header h was not generated by an instance of Tendermint consensus, then the light client should never set trust(h) to true.

Remark: If in the course of the computation, the light client obtains certainty that some headers were forged by adversaries (that is were not generated by an instance of Tendermint consensus), it may submit (a subset of) the headers it has seen as evidence of misbehavior.

Remark: In Completeness we use "eventually", while in practice trust(h) should be set to true before h.Time + TRUSTED_PERIOD. If not, the header cannot be trusted because it is too old.

Remark: If a header h is marked with trust(h), but it is too old at some point in time we denote with now (h.Time + TRUSTED_PERIOD < now), then the light client should set trust(h) to false again at time now.

Assumption: Initially, the light client has a header inithead that it trusts, that is, inithead was correctly generated by the Tendermint consensus.

To reason about the correctness, we may prove the following invariant.

Verification Condition: light Client Invariant. For each light client l and each header h: if l has set trust(h) = true, then validators that are correct until time h.Time + TRUSTED_PERIOD have more than two thirds of the voting power in validators(h.NextValidatorsHash).

Formally, [ \sum_{(v,p) \in validators(h.NextValidatorsHash) \wedge correct(v,h.Time + TRUSTED_PERIOD)} p > 2/3 \sum_{(v,p) \in validators(h.NextValidatorsHash)} p ]

Remark. To prove the invariant, we will have to prove that the light client only trusts headers that were correctly generated by Tendermint consensus. Then the formula above follows from the failure model.

Details

Observation 1. If h.Time + TRUSTED_PERIOD > now, we trust the validator set validators(h.NextValidatorsHash).

When we say we trust validators(h.NextValidatorsHash) we do not trust that each individual validator in validators(h.NextValidatorsHash) is correct, but we only trust the fact that less than 1/3 of them are faulty (more precisely, the faulty ones have less than 1/3 of the total voting power).

VerifySingle correctness arguments

Light Client Accuracy:

  • Assume by contradiction that untrustedHeader was not generated correctly and the light client sets trust to true because verifySingle returns without error.
  • trustedState is trusted and sufficiently new
  • by the Failure Model, less than 1/3 of the voting power held by faulty validators => at least one correct validator v has signed untrustedHeader.
  • as v is correct up to now, it followed the Tendermint consensus protocol at least up to signing untrustedHeader => untrustedHeader was correctly generated. We arrive at the required contradiction.

Light Client Completeness:

  • The check is successful if sufficiently many validators of trustedState are still validators in the height untrustedHeader.Height and signed untrustedHeader.
  • If untrustedHeader.Height = trustedHeader.Height + 1, and both headers were generated correctly, the test passes.

Verification Condition: We may need a Tendermint invariant stating that if untrustedSignedHeader.Header.Height = trustedHeader.Height + 1 then signers(untrustedSignedHeader.Commit) \subseteq validators(trustedHeader.NextValidatorsHash).

Remark: The variable trustThreshold can be used if the user believes that relying on one correct validator is not sufficient. However, in case of (frequent) changes in the validator set, the higher the trustThreshold is chosen, the more unlikely it becomes that verifySingle returns with an error for non-adjacent headers.

  • VerifyBisection correctness arguments (sketch)*

Light Client Accuracy:

  • Assume by contradiction that the header at untrustedHeight obtained from the full node was not generated correctly and the light client sets trust to true because VerifyBisection returns without an error.
  • VerifyBisection returns without error only if all calls to verifySingle in the recursion return without error (return nil).
  • Thus we have a sequence of headers that all satisfied the verifySingle
  • again a contradiction

light Client Completeness:

This is only ensured if upon Commit(pivot) the light client is always provided with a correctly generated header.

Stalling

With VerifyBisection, a faulty full node could stall a light client by creating a long sequence of headers that are queried one-by-one by the light client and look OK, before the light client eventually detects a problem. There are several ways to address this:

  • Each call to Commit could be issued to a different full node
  • Instead of querying header by header, the light client tells a full node which header it trusts, and the height of the header it needs. The full node responds with the header along with a proof consisting of intermediate headers that the light client can use to verify. Roughly, VerifyBisection would then be executed at the full node.
  • We may set a timeout how long VerifyBisection may take.

Light Client Verification

The light client implements a read operation of a header from the blockchain, by communicating with full nodes. As some full nodes may be faulty, this functionality must be implemented in a fault-tolerant way.

In the Tendermint blockchain, the validator set may change with every new block. The staking and unbonding mechanism induces a security model: starting at time Time of the header, more than two-thirds of the next validators of a new block are correct for the duration of TrustedPeriod. The fault-tolerant read operation is designed for this security model.

The challenge addressed here is that the light client might have a block of height h1 and needs to read the block of height h2 greater than h1. Checking all headers of heights from h1 to h2 might be too costly (e.g., in terms of energy for mobile devices). This specification tries to reduce the number of intermediate blocks that need to be checked, by exploiting the guarantees provided by the security model.

Status

This document is thoroughly reviewed, and the protocol has been formalized in TLA+ and model checked.

Issues that need to be addressed

As it is part of the larger light node, its data structures and functions interact with the fork dectection functionality of the light client. As a result of the work on Pull Request 479 we established the need for an update in the data structures in Issue 499. This will not change the verification logic, but it will record information about verification that can be used in fork detection (in particular in computing more efficiently the proof of fork).

Outline

  • Part I: Introduction of relevant terms of the Tendermint blockchain.

  • Part II: Introduction of the problem addressed by the Lightclient Verification protocol.

    • Verification Informal Problem statement: For the general audience, that is, engineers who want to get an overview over what the component is doing from a bird's eye view.
    • Sequential Problem statement: Provides a mathematical definition of the problem statement in its sequential form, that is, ignoring the distributed aspect of the implementation of the blockchain.
  • Part III: Distributed aspects of the light client, system assumptions and temporal logic specifications.

    • Incentives: how faulty full nodes may benefit from misbehaving and how correct full nodes benefit from cooperating.

    • Computational Model: timing and correctness assumptions.

    • Distributed Problem Statement: temporal properties that formalize safety and liveness properties in the distributed setting.

  • Part IV: Specification of the protocols.

    • Definitions: Describes inputs, outputs, variables used by the protocol, auxiliary functions

    • Core Verification: gives an outline of the solution, and details of the functions used (with preconditions, postconditions, error conditions).

    • Liveness Scenarios: when the light client makes progress depends heavily on the changes in the validator sets of the blockchain. We discuss some typical scenarios.

  • Part V: The above parts focus on a common case where the last verified block has height h1 and the requested height h2 satisfies h2 > h1. For IBC, there are scenarios where this might not be the case. In this part, we provide some preliminaries for supporting this. As not all details of the IBC requirements are clear by now, we do not provide a complete specification at this point. We mark with "Open Question" points that need to be addressed in order to finalize this specification. It should be noted that the technically most challenging case is the one specified in Part IV.

In this document we quite extensively use tags in order to be able to reference assumptions, invariants, etc. in future communication. In these tags we frequently use the following short forms:

  • TMBC: Tendermint blockchain
  • SEQ: for sequential specifications
  • LCV: Lightclient Verification
  • LIVE: liveness
  • SAFE: safety
  • FUNC: function
  • INV: invariant
  • A: assumption

Part I - Tendermint Blockchain

Header Fields necessary for the Light Client

[TMBC-HEADER.1]

A set of blockchain transactions is stored in a data structure called block, which contains a field called header. (The data structure block is defined here). As the header contains hashes to the relevant fields of the block, for the purpose of this specification, we will assume that the blockchain is a list of headers, rather than a list of blocks.

[TMBC-HASH-UNIQUENESS.1]

We assume that every hash in the header identifies the data it hashes. Therefore, in this specification, we do not distinguish between hashes and the data they represent.

[TMBC-HEADER-FIELDS.1]

A header contains the following fields:

  • Height: non-negative integer
  • Time: time (integer)
  • LastBlockID: Hashvalue
  • LastCommit DomainCommit
  • Validators: DomainVal
  • NextValidators: DomainVal
  • Data: DomainTX
  • AppState: DomainApp
  • LastResults: DomainRes

[TMBC-SEQ.1]

The Tendermint blockchain is a list chain of headers.

[TMBC-VALIDATOR-PAIR.1]

Given a full node, a validator pair is a pair (peerID, voting_power), where

  • peerID is the PeerID (public key) of a full node,
  • voting_power is an integer (representing the full node's voting power in a certain consensus instance).

In the Golang implementation the data type for validator pair is called Validator

[TMBC-VALIDATOR-SET.1]

A validator set is a set of validator pairs. For a validator set vs, we write TotalVotingPower(vs) for the sum of the voting powers of its validator pairs.

[TMBC-VOTE.1]

A vote contains a prevote or precommit message sent and signed by a validator node during the execution of consensus. Each message contains the following fields

  • Type: prevote or precommit
  • Height: positive integer
  • Round a positive integer
  • BlockID a Hashvalue of a block (not necessarily a block of the chain)

[TMBC-COMMIT.1]

A commit is a set of precommit message.

Tendermint Failure Model

[TMBC-AUTH-BYZ.1]

We assume the authenticated Byzantine fault model in which no node (faulty or correct) may break digital signatures, but otherwise, no additional assumption is made about the internal behavior of faulty nodes. That is, faulty nodes are only limited in that they cannot forge messages.

[TMBC-TIME-PARAMS.1]

A Tendermint blockchain has the following configuration parameters:

  • unbondingPeriod: a time duration.
  • trustingPeriod: a time duration smaller than unbondingPeriod.

[TMBC-CORRECT.1]

We define a predicate correctUntil(n, t), where n is a node and t is a time point. The predicate correctUntil(n, t) is true if and only if the node n follows all the protocols (at least) until time t.

[TMBC-FM-2THIRDS.1]

If a block h is in the chain, then there exists a subset CorrV of h.NextValidators, such that:

  • TotalVotingPower(CorrV) > 2/3 TotalVotingPower(h.NextValidators); cf. [TMBC-VALIDATOR-SET.1]
  • For every validator pair (n,p) in CorrV, it holds correctUntil(n, h.Time + trustingPeriod); cf. [TMBC-CORRECT.1]

The definition of correct [[TMBC-CORRECT.1]][TMBC-CORRECT-link] refers to realtime, while it is used here with Time and trustingPeriod, which are "hardware times". We do not make a distinction here.

[TMBC-CORR-FULL.1]

Every correct full node locally stores a prefix of the current list of headers from [TMBC-SEQ.1].

What the Light Client Checks

From [TMBC-FM-2THIRDS.1] we directly derive the following observation:

[TMBC-VAL-CONTAINS-CORR.1]

Given a (trusted) block tb of the blockchain, a given set of full nodes N contains a correct node at a real-time t, if

  • t - trustingPeriod < tb.Time < t
  • the voting power in tb.NextValidators of nodes in N is more than 1/3 of TotalVotingPower(tb.NextValidators)

The following describes how a commit for a given block b must look like.

[TMBC-SOUND-DISTR-POSS-COMMIT.1]

For a block b, each element pc of PossibleCommit(b) satisfies:

  • pc contains only votes (cf. [TMBC-VOTE.1]) by validators from b.Validators
  • the sum of the voting powers in pc is greater than 2/3 TotalVotingPower(b.Validators)
  • and there is an r such that each vote v in pc satisfies
    • v.Type = precommit
    • v.Height = b.Height
    • v.Round = r
    • v.blockID = hash(b)

The following property comes from the validity of the consensus: A correct validator node only sends prevote or precommit, if BlockID of the new (to-be-decided) block is equal to the hash of the last block.

[TMBC-VAL-COMMIT.1]

If for a block b, a commit c

  • contains at least one validator pair (v,p) such that v is a correct validator node, and
  • is contained in PossibleCommit(b)

then the block b is on the blockchain.

Context of this document

In this document we specify the light client verification component, called Core Verification. The Core Verification communicates with a full node. As full nodes may be faulty, it cannot trust the received information, but the light client has to check whether the header it receives coincides with the one generated by Tendermint consensus.

The two properties [TMBC-VAL-CONTAINS-CORR.1] and [TMBC-VAL-COMMIT] formalize the checks done by this specification: Given a trusted block tb and an untrusted block ub with a commit cub, one has to check that cub is in PossibleCommit(ub), and that cub contains a correct node using tb.

Part II - Sequential Definition of the Verification Problem

Verification Informal Problem statement

Given a height targetHeight as an input, the Verifier eventually stores a header h of height targetHeight locally. This header h is generated by the Tendermint blockchain. In particular, a header that was not generated by the blockchain should never be stored.

Sequential Problem statement

[LCV-SEQ-LIVE.1]

The Verifier gets as input a height targetHeight, and eventually stores the header of height targetHeight of the blockchain.

[LCV-SEQ-SAFE.1]

The Verifier never stores a header which is not in the blockchain.

Part III - Light Client as Distributed System

Incentives

Faulty full nodes may benefit from lying to the light client, by making the light client accept a block that deviates (e.g., contains additional transactions) from the one generated by Tendermint consensus. Users using the light client might be harmed by accepting a forged header.

The fork detector of the light client may help the correct full nodes to understand whether their header is a good one. Hence, in combination with the light client detector, the correct full nodes have the incentive to respond. We can thus base liveness arguments on the assumption that correct full nodes reliably talk to the light client.

Computational Model

[LCV-A-PEER.1]

The verifier communicates with a full node called primary. No assumption is made about the full node (it may be correct or faulty).

[LCV-A-COMM.1]

Communication between the light client and a correct full node is reliable and bounded in time. Reliable communication means that messages are not lost, not duplicated, and eventually delivered. There is a (known) end-to-end delay Delta, such that if a message is sent at time t then it is received and processes by time t + Delta. This implies that we need a timeout of at least 2 Delta for remote procedure calls to ensure that the response of a correct peer arrives before the timeout expires.

[LCV-A-TFM.1]

The Tendermint blockchain satisfies the Tendermint failure model [TMBC-FM-2THIRDS.1].

[LCV-A-VAL.1]

The system satisfies [TMBC-AUTH-BYZ.1] and [TMBC-FM-2THIRDS.1]. Thus, there is a blockchain that satisfies the soundness requirements (that is, the validation rules in [block]).

Distributed Problem Statement

Two Kinds of Termination

We do not assume that primary is correct. Under this assumption no protocol can guarantee the combination of the sequential properties. Thus, in the (unreliable) distributed setting, we consider two kinds of termination (successful and failure) and we will specify below under what (favorable) conditions Core Verification ensures to terminate successfully, and satisfy the requirements of the sequential problem statement:

[LCV-DIST-TERM.1]

Core Verification either terminates successfully or it terminates with failure.

Design choices

[LCV-DIST-STORE.1]

Core Verification has a local data structure called LightStore that contains light blocks (that contain a header). For each light block we record whether it is verified.

[LCV-DIST-PRIMARY.1]

Core Verification has a local variable primary that contains the PeerID of a full node.

[LCV-DIST-INIT.1]

LightStore is initialized with a header trustedHeader that was correctly generated by the Tendermint consensus. We say trustedHeader is verified.

Temporal Properties

[LCV-DIST-SAFE.1]

It is always the case that every verified header in LightStore was generated by an instance of Tendermint consensus.

[LCV-DIST-LIVE.1]

From time to time, a new instance of Core Verification is called with a height targetHeight greater than the height of any header in LightStore. Each instance must eventually terminate.

  • If
    • the primary is correct (and locally has the block of targetHeight), and
    • LightStore always contains a verified header whose age is less than the trusting period,
      then Core Verification adds a verified header hd with height targetHeight to LightStore and it terminates successfully

These definitions imply that if the primary is faulty, a header may or may not be added to LightStore. In any case, [LCV-DIST-SAFE.1] must hold. The invariant [LCV-DIST-SAFE.1] and the liveness requirement [LCV-DIST-LIVE.1] allow that verified headers are added to LightStore whose height was not passed to the verifier (e.g., intermediate headers used in bisection; see below). Note that for liveness, initially having a trustedHeader within the trustinPeriod is not sufficient. However, as this specification will leave some freedom with respect to the strategy in which order to download intermediate headers, we do not give a more precise liveness specification here. After giving the specification of the protocol, we will discuss some liveness scenarios below.

Solving the sequential specification

This specification provides a partial solution to the sequential specification. The Verifier solves the invariant of the sequential part

[LCV-DIST-SAFE.1] => [LCV-SEQ-SAFE.1]

In the case the primary is correct, and there is a recent header in LightStore, the verifier satisfies the liveness requirements.

⋀ primary is correct
⋀ always ∃ verified header in LightStore. header.Time > now - trustingPeriod
⋀ [LCV-A-Comm.1] ⋀ ( ( [TMBC-CorrFull.1] ⋀ [LCV-DIST-LIVE.1] ) ⟹ [LCV-SEQ-LIVE.1] )

Part IV - Light Client Verification Protocol

We provide a specification for Light Client Verification. The local code for verification is presented by a sequential function VerifyToTarget to highlight the control flow of this functionality. We note that if a different concurrency model is considered for an implementation, the sequential flow of the function may be implemented with mutexes, etc. However, the light client verification is partitioned into three blocks that can be implemented and tested independently:

  • FetchLightBlock is called to download a light block (header) of a given height from a peer.
  • ValidAndVerified is a local code that checks the header.
  • Schedule decides which height to try to verify next. We keep this underspecified as different implementations (currently in Goland and Rust) may implement different optimizations here. We just provide necessary conditions on how the height may evolve.

Definitions

Data Types

The core data structure of the protocol is the LightBlock.

[LCV-DATA-LIGHTBLOCK.1]

type LightBlock struct {
  Header          Header
  Commit          Commit
  Validators      ValidatorSet
}

[LCV-DATA-LIGHTSTORE.1]

LightBlocks are stored in a structure which stores all LightBlock from initialization or received from peers.

type LightStore struct {
 ...
}

Each LightBlock is in one of the following states:

type VerifiedState int

const (
 StateUnverified = iota + 1
 StateVerified
 StateFailed
 StateTrusted
)

Only the detector module sets a lightBlock state to StateTrusted and only if it was StateVerified before.

The LightStore exposes the following functions to query stored LightBlocks.

[LCV-FUNC-GET.1]

func (ls LightStore) Get(height Height) (LightBlock, bool)
  • Expected postcondition
    • returns a LightBlock at a given height or false in the second argument if the LightStore does not contain the specified LightBlock.

[LCV-FUNC-LATEST-VERIF.1]

func (ls LightStore) LatestVerified() LightBlock
  • Expected postcondition
    • returns the highest light block whose state is StateVerified or StateTrusted

[LCV-FUNC-UPDATE.2]

func (ls LightStore) Update(lightBlock LightBlock, 
                            verfiedState VerifiedState
       verifiedBy Height)
  • Expected postcondition
    • The state of the LightBlock is set to verifiedState.
    • verifiedBy of the Lightblock is set to Height

The following function is used only in the detector specification listed here for completeness.

[LCV-FUNC-LATEST-TRUSTED.1]

func (ls LightStore) LatestTrusted() LightBlock
  • Expected postcondition
    • returns the highest light block that has been verified and checked by the detector.

[LCV-FUNC-FILTER.1]

func (ls LightStore) FilterVerified() LightSTore
  • Expected postcondition
    • returns only the LightBlocks with state verified.

Inputs

  • lightStore: stores light blocks that have been downloaded and that passed verification. Initially it contains a light block with trustedHeader.
  • primary: peerID
  • targetHeight: the height of the needed header

Configuration Parameters

  • trustThreshold: a float. Can be used if correctness should not be based on more voting power and 1/3.
  • trustingPeriod: a time duration [TMBC-TIME_PARAMS.1].
  • clockDrift: a time duration. Correction parameter dealing with only approximately synchronized clocks.

Variables

  • nextHeight: initially targetHeight

    nextHeight should be thought of the "height of the next header we need to download and verify"

Assumptions

[LCV-A-INIT.1]

  • trustedHeader is from the blockchain

  • targetHeight > LightStore.LatestVerified.Header.Height

Invariants

[LCV-INV-TP.1]

It is always the case that LightStore.LatestTrusted.Header.Time > now - trustingPeriod.

If the invariant is violated, the light client does not have a header it can trust. A trusted header must be obtained externally, its trust can only be based on social consensus.

Used Remote Functions

We use the functions commit and validators that are provided by the RPC client for Tendermint.

func Commit(height int64) (SignedHeader, error)
  • Implementation remark
    • RPC to full node n
    • JSON sent:
// POST /commit
{
 "jsonrpc": "2.0",
 "id": "ccc84631-dfdb-4adc-b88c-5291ea3c2cfb", // UUID v4, unique per request
 "method": "commit",
 "params": {
  "height": 1234
 }
}
  • Expected precondition
    • header of height exists on blockchain
  • Expected postcondition
    • if n is correct: Returns the signed header of height height from the blockchain if communication is timely (no timeout)
    • if n is faulty: Returns a signed header with arbitrary content
  • Error condition
    • if n is correct: precondition violated or timeout
    • if n is faulty: arbitrary error

func Validators(height int64) (ValidatorSet, error)
  • Implementation remark
    • RPC to full node n
    • JSON sent:
// POST /validators
{
 "jsonrpc": "2.0",
 "id": "ccc84631-dfdb-4adc-b88c-5291ea3c2cfb", // UUID v4, unique per request
 "method": "validators",
 "params": {
  "height": 1234
 }
}
  • Expected precondition
    • header of height exists on blockchain
  • Expected postcondition
    • if n is correct: Returns the validator set of height height from the blockchain if communication is timely (no timeout)
    • if n is faulty: Returns arbitrary validator set
  • Error condition
    • if n is correct: precondition violated or timeout
    • if n is faulty: arbitrary error

Communicating Function

[LCV-FUNC-FETCH.1]

func FetchLightBlock(peer PeerID, height Height) LightBlock
  • Implementation remark
    • RPC to peer at PeerID
    • calls Commit for height and Validators for height and height+1
  • Expected precondition
    • height is less than or equal to height of the peer [LCV-IO-PRE-HEIGHT.1]
  • Expected postcondition:
    • if node is correct:
      • Returns the LightBlock lb of height height that is consistent with the blockchain
      • lb.provider = peer [LCV-IO-POST-PROVIDER.1]
      • lb.Header is a header consistent with the blockchain
      • lb.Validators is the validator set of the blockchain at height nextHeight
      • lb.NextValidators is the validator set of the blockchain at height nextHeight + 1
    • if node is faulty: Returns a LightBlock with arbitrary content [TMBC-AUTH-BYZ.1]
  • Error condition
    • if n is correct: precondition violated
    • if n is faulty: arbitrary error
    • if lb.provider != peer
    • times out after 2 Delta (by assumption n is faulty)

Core Verification

Outline

The VerifyToTarget is the main function and uses the following functions.

  • FetchLightBlock is called to download the next light block. It is the only function that communicates with other nodes
  • ValidAndVerified checks whether header is valid and checks if a new lightBlock should be trusted based on a previously verified lightBlock.
  • Schedule decides which height to try to verify next

In the following description of VerifyToTarget we do not deal with error handling. If any of the above function returns an error, VerifyToTarget just passes the error on.

[LCV-FUNC-MAIN.1]

func VerifyToTarget(primary PeerID, lightStore LightStore,
                    targetHeight Height) (LightStore, Result) {

    nextHeight := targetHeight

    for lightStore.LatestVerified.height < targetHeight {

        // Get next LightBlock for verification
        current, found := lightStore.Get(nextHeight)
        if !found {
            current = FetchLightBlock(primary, nextHeight)
            lightStore.Update(current, StateUnverified)
        }

        // Verify
        verdict = ValidAndVerified(lightStore.LatestVerified, current)

        // Decide whether/how to continue
        if verdict == SUCCESS {
            lightStore.Update(current, StateVerified)
        }
        else if verdict == NOT_ENOUGH_TRUST {
            // do nothing
   // the light block current passed validation, but the validator
            // set is too different to verify it. We keep the state of
   // current at StateUnverified. For a later iteration, Schedule
   // might decide to try verification of that light block again.
        }
        else {
            // verdict is some error code
            lightStore.Update(current, StateFailed)
            // possibly remove all LightBlocks from primary
            return (lightStore, ResultFailure)
        }
        nextHeight = Schedule(lightStore, nextHeight, targetHeight)
    }
    return (lightStore, ResultSuccess)
}
  • Expected precondition
    • lightStore contains a LightBlock within the trustingPeriod [LCV-PRE-TP.1]
    • targetHeight is greater than the height of all the LightBlocks in lightStore
  • Expected postcondition:
    • returns lightStore that contains a LightBlock that corresponds to a block of the blockchain of height targetHeight (that is, the LightBlock has been added to lightStore) [LCV-POST-LS.1]
  • Error conditions
    • if the precondition is violated
    • if ValidAndVerified or FetchLightBlock report an error
    • if [LCV-INV-TP.1] is violated

Details of the Functions

[LCV-FUNC-VALID.1]

func ValidAndVerified(trusted LightBlock, untrusted LightBlock) Result
  • Expected precondition:
    • untrusted is valid, that is, satisfies the soundness checks
    • untrusted is well-formed, that is,
      • untrusted.Header.Time < now + clockDrift
      • untrusted.Validators = hash(untrusted.Header.Validators)
      • untrusted.NextValidators = hash(untrusted.Header.NextValidators)
    • trusted.Header.Time > now - trustingPeriod
    • trusted.Commit is a commit for the header trusted.Header, i.e., it contains the correct hash of the header, and +2/3 of signatures
    • the Height and Time of trusted are smaller than the Height and Time of untrusted, respectively
    • the untrusted.Header is well-formed (passes the tests from [block]), and in particular
      • if the untrusted header unstrusted.Header is the immediate successor of trusted.Header, then it holds that
        • trusted.Header.NextValidators = untrusted.Header.Validators, and moreover,
        • untrusted.Header.Commit
          • contains signatures by more than two-thirds of the validators
          • contains no signature from nodes that are not in trusted.Header.NextValidators
  • Expected postcondition:
    • Returns SUCCESS:
      • if untrusted is the immediate successor of trusted, or otherwise,
      • if the signatures of a set of validators that have more than max(1/3,trustThreshold) of voting power in trusted.Nextvalidators is contained in untrusted.Commit (that is, header passes the tests [TMBC-VAL-CONTAINS-CORR.1] and [TMBC-VAL-COMMIT.1])
    • Returns NOT_ENOUGH_TRUST if:
      • untrusted is not the immediate successor of trusted and the max(1/3,trustThreshold) threshold is not reached (that is, if [TMBC-VAL-CONTAINS-CORR.1] fails and header is does not violate the soundness checks [block]).
  • Error condition:
    • if precondition violated

[LCV-FUNC-SCHEDULE.1]

func Schedule(lightStore, nextHeight, targetHeight) Height
  • Implementation remark: If picks the next height to be verified. We keep the precise choice of the next header under-specified. It is subject to performance optimizations that do not influence the correctness
  • Expected postcondition: [LCV-SCHEDULE-POST.1] Return H s.t.
    1. if lightStore.LatestVerified.Height = nextHeight and lightStore.LatestVerified < targetHeight then
      nextHeight < H <= targetHeight
    2. if lightStore.LatestVerified.Height < nextHeight and lightStore.LatestVerified.Height < targetHeight then
      lightStore.LatestVerified.Height < H < nextHeight
    3. if lightStore.LatestVerified.Height = targetHeight then
      H = targetHeight

Case i. captures the case where the light block at height nextHeight has been verified, and we can choose a height closer to the targetHeight. As we get the lightStore as parameter, the choice of the next height can depend on the lightStore, e.g., we can pick a height for which we have already downloaded a light block. In Case ii. the header of nextHeight could not be verified, and we need to pick a smaller height. In Case iii. is a special case when we have verified the targetHeight.

Solving the distributed specification

trustedStore is implemented by the light blocks in lightStore that have the state StateVerified.

Argument for [LCV-DIST-SAFE.1]

Argument for [LCV-DIST-LIVE.1]

  • If primary is correct,
    • FetchLightBlock will always return a light block consistent with the blockchain
    • ValidAndVerified either verifies the header using the trusting period or falls back to sequential verification
    • If [LCV-INV-TP.1] holds, eventually every header will be verified and core verification terminates successfully.
    • successful termination depends on the age of lightStore.LatestVerified (for instance, initially on the age of trustedHeader) and the changes of the validator sets on the blockchain. We will give some examples below.
  • If primary is faulty,
    • it either provides headers that pass all the tests, and we return with the header
    • it provides one header that fails a test, core verification terminates with failure.
    • it times out and core verification terminates with failure.

Liveness Scenarios

The liveness argument above assumes [LCV-INV-TP.1]

which requires that there is a header that does not expire before the target height is reached. Here we discuss scenarios to ensure this.

Let startHeader be LightStore.LatestVerified when core verification is called (trustedHeader) and startTime be the time core verification is invoked.

In order to ensure liveness, LightStore always needs to contain a verified (or initially trusted) header whose time is within the trusting period. To ensure this, core verification needs to add new headers to LightStore and verify them, before all headers in LightStore expire.

Many changes in validator set

Let's consider Schedule implements bisection, that is, it halves the distance. Assume the case where the validator set changes completely in each block. Then the method in this specification needs to sequentially verify all headers. That is, for

  • W = log_2 (targetHeight - startHeader.Height),

W headers need to be downloaded and checked before the header of height startHeader.Height + 1 is added to LightStore.

  • Let Comp be the local computation time needed to check headers and signatures for one header.

  • Then we need in the worst case Comp + 2 Delta to download and check one header.

  • Then the first time a verified header could be added to LightStore is startTime + W * (Comp + 2 Delta)

  • [TP.1] However, it can only be added if we still have a header in LightStore, which is not expired, that is only the case if

    • startHeader.Time > startTime + WCG * (Comp + 2 Delta) - trustingPeriod,
    • that is, if core verification is started at
      startTime < startHeader.Time + trustingPeriod - WCG * (Comp + 2 Delta)
  • one may then do an inductive argument from this point on, depending on the implementation of Schedule. We may have to account for the headers that are already downloaded, but they are checked against the new LightStore.LatestVerified.

We observe that the worst case time it needs to verify the header of height targetHeight depends mainly on how frequent the validator set on the blockchain changes. That core verification terminates successfully crucially depends on the check [TP.1], that is, that the headers in LightStore do not expire in the time needed to download more headers, which depends on the creation time of the headers in LightStore. That is, termination of core verification is highly depending on the data stored in the blockchain. The current light client core verification protocol exploits that, in practice, changes in the validator set are rare. For instance, consider the following scenario.

No change in validator set

If on the blockchain the validator set of the block at height targetHeight is equal to startHeader.NextValidators:

  • there is one round trip in FetchLightBlock to download the light block of height targetHeight, and Comp to check it.
  • as the validator sets are equal, Verify returns SUCCESS, if startHeader.Time > now - trustingPeriod.
  • that is, if startTime < startHeader.Header.Time + trustingPeriod - 2 Delta - Comp, then core verification terminates successfully

Part V - Supporting the IBC Relayer

The above specification focuses on the most common case, which also constitutes the most challenging task: using the Tendermint security model to verify light blocks without downloading all intermediate blocks. To focus on this challenge, above we have restricted ourselves to the case where targetHeight is greater than the height of any trusted header. This simplified presentation of the algorithm as initially lightStore.LatestVerified() is less than targetHeight, and in the process of verification lightStore.LatestVerified() increases until targetHeight is reached.

For IBC it might be that some "older" header is needed, that is, targetHeight < lightStore.LatestVerified(). In this section we present a preliminary design, and we mark some remaining open questions. If targetHeight < lightStore.LatestVerified() our design separates the following cases:

  • A previous instance of VerifyToTarget has already downloaded the light block of targetHeight. There are two cases
    • the light block has been verified
    • the light block has not been verified yet
  • No light block of targetHeight had been downloaded before. There are two cases:
    • there exists a verified light block of height less than targetHeight
    • otherwise. In this case we need to do "backwards verification" using the hash of the previous block in the LastBlockID field of a header.

Open Question: what are the security assumptions for backward verification. Should we check that the light block we verify from (and/or the checked light block) is within the trusting period?

The design just presents the above case distinction as a function, and defines some auxiliary functions in the same way the protocol was presented in Part IV.

func (ls LightStore) LatestPrevious(height Height) (LightBlock, bool)
  • Expected postcondition
    • returns a light block lb that satisfies:
      • lb is in lightStore
      • lb is verified and not expired
      • lb.Header.Height < height
      • for all b in lightStore s.t. b is verified and not expired it holds lb.Header.Height >= b.Header.Height
    • false in the second argument if the LightStore does not contain such an lb.
func (ls LightStore) MinVerified() (LightBlock, bool)
  • Expected postcondition
    • returns a light block lb that satisfies:
      • lb is in lightStore
      • lb is verified Open Question: replace by trusted?
      • lb.Header.Height is minimal in the lightStore
      • Open Question: according to this, it might be expired (outside the trusting period). This approach appears safe. Are there reasons we should not do that?
    • false in the second argument if the LightStore does not contain such an lb.

If a height that is smaller than the smallest height in the lightstore is required, we check the hashes backwards. This is done with the following function:

[LCV-FUNC-BACKWARDS.1]

func Backwards (primary PeerID, lightStore LightStore, targetHeight Height)
               (LightStore, Result) {
  
    lb,res = lightStore.MinVerified()
    if res = false {
        return (lightStore, ResultFailure)
    }

    latest := lb.Header
    for i := lb.Header.height - 1; i >= targetHeight; i-- {
        // here we download height-by-height. We might first download all
        // headers down to targetHeight and then check them.
        current := FetchLightBlock(primary,i)
        if (hash(current) != latest.Header.LastBlockId) {
            return (lightStore, ResultFailure)
        }
        else {
            lightStore.Update(current, StateVerified)
            // **Open Question:** Do we need a new state type for
            // backwards verified light blocks?
        }
        latest = current
    }
    return (lightStore, ResultSuccess)
}

The following function just decided based on the required height which method should be used.

[LCV-FUNC-IBCMAIN.1]

func Main (primary PeerID, lightStore LightStore, targetHeight Height)
          (LightStore, Result) {

    b1, r1 = lightStore.Get(targetHeight)
    if r1 = true and b1.State = StateVerified {
        // block already there
        return (lightStore, ResultSuccess)
    }

    if targetHeight > lightStore.LatestVerified.height {
     // case of Part IV
        return VerifyToTarget(primary, lightStore, targetHeight)
    }
    else {
        b2, r2 = lightStore.LatestPrevious(targetHeight);
        if r2 = true {
            // make auxiliary lightStore auxLS to call VerifyToTarget.
   // VerifyToTarget uses LatestVerified of the given lightStore
            // For that we need:
            // auxLS.LatestVerified = lightStore.LatestPrevious(targetHeight)
            auxLS.Init;
            auxLS.Update(b2,StateVerified);
            if r1 = true {
                // we need to verify a previously downloaded light block.
                // we add it to the auxiliary store so that VerifyToTarget
                // does not download it again
                auxLS.Update(b1,b1.State);
            }
            auxLS, res2 = VerifyToTarget(primary, auxLS, targetHeight)
            // move all lightblocks from auxLS to lightStore,
            // maintain state
   // we do that whether VerifyToTarget was successful or not
            for i, s range auxLS {
                lighStore.Update(s,s.State)
            }
            return (lightStore, res2)
        }
        else {
            return Backwards(primary, lightStore, targetHeight)
        }
    }
}

References

[block] Specification of the block data structure.

[RPC] RPC client for Tendermint

[fork-detector] The specification of the light client fork detector.

[fullnode] Specification of the full node API

[ibc-rs] Rust implementation of IBC modules and relayer.

[lightclient] The light client ADR [77d2651 on Dec 27, 2019].

Light Client Verification

The light client implements a read operation of a header from the blockchain, by communicating with full nodes. As some full nodes may be faulty, this functionality must be implemented in a fault-tolerant way.

In the Tendermint blockchain, the validator set may change with every new block. The staking and unbonding mechanism induces a security model: starting at time Time of the header, more than two-thirds of the next validators of a new block are correct for the duration of TrustedPeriod. The fault-tolerant read operation is designed for this security model.

The challenge addressed here is that the light client might have a block of height h1 and needs to read the block of height h2 greater than h1. Checking all headers of heights from h1 to h2 might be too costly (e.g., in terms of energy for mobile devices). This specification tries to reduce the number of intermediate blocks that need to be checked, by exploiting the guarantees provided by the security model.

Status

Previous Versions

  • [001_published] is thoroughly reviewed, and the protocol has been formalized in TLA+ and model checked.

Issues that are addressed in this revision

As it is part of the larger light node, its data structures and functions interact with the attack dectection functionality of the light client. As a result of the work on

adaptations to the semantics and functions exposed by the LightStore needed to be made. In contrast to version 001 we specify the following:

  • VerifyToTarget and Backwards are called with a single lightblock as root of trust in contrast to passing the complete lightstore.

  • During verification, we record for each lightblock which other lightblock can be used to verify it in one step. This is needed to generate verification traces that are needed for IBC.

Outline

  • Part I: Introduction of relevant terms of the Tendermint blockchain.

  • Part II: Introduction of the problem addressed by the Lightclient Verification protocol.

    • Verification Informal Problem statement: For the general audience, that is, engineers who want to get an overview over what the component is doing from a bird's eye view.
    • Sequential Problem statement: Provides a mathematical definition of the problem statement in its sequential form, that is, ignoring the distributed aspect of the implementation of the blockchain.
  • Part III: Distributed aspects of the light client, system assumptions and temporal logic specifications.

    • Incentives: how faulty full nodes may benefit from misbehaving and how correct full nodes benefit from cooperating.

    • Computational Model: timing and correctness assumptions.

    • Distributed Problem Statement: temporal properties that formalize safety and liveness properties in the distributed setting.

  • Part IV: Specification of the protocols.

    • Definitions: Describes inputs, outputs, variables used by the protocol, auxiliary functions

    • Core Verification: gives an outline of the solution, and details of the functions used (with preconditions, postconditions, error conditions).

    • Liveness Scenarios: when the light client makes progress depends heavily on the changes in the validator sets of the blockchain. We discuss some typical scenarios.

  • Part V: The above parts focus on a common case where the last verified block has height h1 and the requested height h2 satisfies h2 > h1. For IBC, there are scenarios where this might not be the case. In this part, we provide some preliminaries for supporting this. As not all details of the IBC requirements are clear by now, we do not provide a complete specification at this point. We mark with "Open Question" points that need to be addressed in order to finalize this specification. It should be noted that the technically most challenging case is the one specified in Part IV.

In this document we quite extensively use tags in order to be able to reference assumptions, invariants, etc. in future communication. In these tags we frequently use the following short forms:

  • TMBC: Tendermint blockchain
  • SEQ: for sequential specifications
  • LCV: Lightclient Verification
  • LIVE: liveness
  • SAFE: safety
  • FUNC: function
  • INV: invariant
  • A: assumption

Part I - Tendermint Blockchain

Header Fields necessary for the Light Client

[TMBC-HEADER.1]

A set of blockchain transactions is stored in a data structure called block, which contains a field called header. (The data structure block is defined here). As the header contains hashes to the relevant fields of the block, for the purpose of this specification, we will assume that the blockchain is a list of headers, rather than a list of blocks.

[TMBC-HASH-UNIQUENESS.1]

We assume that every hash in the header identifies the data it hashes. Therefore, in this specification, we do not distinguish between hashes and the data they represent.

[TMBC-HEADER-FIELDS.2]

A header contains the following fields:

  • Height: non-negative integer
  • Time: time (non-negative integer)
  • LastBlockID: Hashvalue
  • LastCommit DomainCommit
  • Validators: DomainVal
  • NextValidators: DomainVal
  • Data: DomainTX
  • AppState: DomainApp
  • LastResults: DomainRes

[TMBC-SEQ.1]

The Tendermint blockchain is a list chain of headers.

[TMBC-VALIDATOR-PAIR.1]

Given a full node, a validator pair is a pair (peerID, voting_power), where

  • peerID is the PeerID (public key) of a full node,
  • voting_power is an integer (representing the full node's voting power in a certain consensus instance).

In the Golang implementation the data type for validator pair is called Validator

[TMBC-VALIDATOR-SET.1]

A validator set is a set of validator pairs. For a validator set vs, we write TotalVotingPower(vs) for the sum of the voting powers of its validator pairs.

[TMBC-VOTE.1]

A vote contains a prevote or precommit message sent and signed by a validator node during the execution of consensus. Each message contains the following fields

  • Type: prevote or precommit
  • Height: positive integer
  • Round a positive integer
  • BlockID a Hashvalue of a block (not necessarily a block of the chain)

[TMBC-COMMIT.1]

A commit is a set of precommit message.

Tendermint Failure Model

[TMBC-AUTH-BYZ.1]

We assume the authenticated Byzantine fault model in which no node (faulty or correct) may break digital signatures, but otherwise, no additional assumption is made about the internal behavior of faulty nodes. That is, faulty nodes are only limited in that they cannot forge messages.

[TMBC-TIME-PARAMS.1]

A Tendermint blockchain has the following configuration parameters:

  • unbondingPeriod: a time duration.
  • trustingPeriod: a time duration smaller than unbondingPeriod.

[TMBC-CORRECT.1]

We define a predicate correctUntil(n, t), where n is a node and t is a time point. The predicate correctUntil(n, t) is true if and only if the node n follows all the protocols (at least) until time t.

[TMBC-FM-2THIRDS.1]

If a block h is in the chain, then there exists a subset CorrV of h.NextValidators, such that:

  • TotalVotingPower(CorrV) > 2/3 TotalVotingPower(h.NextValidators); cf. [TMBC-VALIDATOR-SET.1]
  • For every validator pair (n,p) in CorrV, it holds correctUntil(n, h.Time + trustingPeriod); cf. [TMBC-CORRECT.1]

The definition of correct [[TMBC-CORRECT.1]][TMBC-CORRECT-link] refers to realtime, while it is used here with Time and trustingPeriod, which are "hardware times". We do not make a distinction here.

[TMBC-CORR-FULL.1]

Every correct full node locally stores a prefix of the current list of headers from [TMBC-SEQ.1].

What the Light Client Checks

From [TMBC-FM-2THIRDS.1] we directly derive the following observation:

[TMBC-VAL-CONTAINS-CORR.1]

Given a (trusted) block tb of the blockchain, a given set of full nodes N contains a correct node at a real-time t, if

  • t - trustingPeriod < tb.Time < t
  • the voting power in tb.NextValidators of nodes in N is more than 1/3 of TotalVotingPower(tb.NextValidators)

The following describes how a commit for a given block b must look like.

[TMBC-SOUND-DISTR-POSS-COMMIT.1]

For a block b, each element pc of PossibleCommit(b) satisfies:

  • pc contains only votes (cf. [TMBC-VOTE.1]) by validators from b.Validators
  • the sum of the voting powers in pc is greater than 2/3 TotalVotingPower(b.Validators)
  • and there is an r such that each vote v in pc satisfies
    • v.Type = precommit
    • v.Height = b.Height
    • v.Round = r
    • v.blockID = hash(b)

The following property comes from the validity of the consensus: A correct validator node only sends prevote or precommit, if BlockID of the new (to-be-decided) block is equal to the hash of the last block.

[TMBC-VAL-COMMIT.1]

If for a block b, a commit c

  • contains at least one validator pair (v,p) such that v is a correct validator node, and
  • is contained in PossibleCommit(b)

then the block b is on the blockchain.

Context of this document

In this document we specify the light client verification component, called Core Verification. The Core Verification communicates with a full node. As full nodes may be faulty, it cannot trust the received information, but the light client has to check whether the header it receives coincides with the one generated by Tendermint consensus.

The two properties [TMBC-VAL-CONTAINS-CORR.1] and [TMBC-VAL-COMMIT] formalize the checks done by this specification: Given a trusted block tb and an untrusted block ub with a commit cub, one has to check that cub is in PossibleCommit(ub), and that cub contains a correct node using tb.

Part II - Sequential Definition of the Verification Problem

Verification Informal Problem statement

Given a height targetHeight as an input, the Verifier eventually stores a header h of height targetHeight locally. This header h is generated by the Tendermint blockchain. In particular, a header that was not generated by the blockchain should never be stored.

Sequential Problem statement

[LCV-SEQ-LIVE.1]

The Verifier gets as input a height targetHeight, and eventually stores the header of height targetHeight of the blockchain.

[LCV-SEQ-SAFE.1]

The Verifier never stores a header which is not in the blockchain.

Part III - Light Client as Distributed System

Incentives

Faulty full nodes may benefit from lying to the light client, by making the light client accept a block that deviates (e.g., contains additional transactions) from the one generated by Tendermint consensus. Users using the light client might be harmed by accepting a forged header.

The attack detector of the light client may help the correct full nodes to understand whether their header is a good one. Hence, in combination with the light client detector, the correct full nodes have the incentive to respond. We can thus base liveness arguments on the assumption that correct full nodes reliably talk to the light client.

Computational Model

[LCV-A-PEER.1]

The verifier communicates with a full node called primary. No assumption is made about the full node (it may be correct or faulty).

[LCV-A-COMM.1]

Communication between the light client and a correct full node is reliable and bounded in time. Reliable communication means that messages are not lost, not duplicated, and eventually delivered. There is a (known) end-to-end delay Delta, such that if a message is sent at time t then it is received and processes by time t + Delta. This implies that we need a timeout of at least 2 Delta for remote procedure calls to ensure that the response of a correct peer arrives before the timeout expires.

[LCV-A-TFM.1]

The Tendermint blockchain satisfies the Tendermint failure model [TMBC-FM-2THIRDS.1].

[LCV-A-VAL.1]

The system satisfies [TMBC-AUTH-BYZ.1] and [TMBC-FM-2THIRDS.1]. Thus, there is a blockchain that satisfies the soundness requirements (that is, the validation rules in [block]).

Distributed Problem Statement

Two Kinds of Termination

We do not assume that primary is correct. Under this assumption no protocol can guarantee the combination of the sequential properties. Thus, in the (unreliable) distributed setting, we consider two kinds of termination (successful and failure) and we will specify below under what (favorable) conditions Core Verification ensures to terminate successfully, and satisfy the requirements of the sequential problem statement:

[LCV-DIST-TERM.1]

Core Verification either terminates successfully or it terminates with failure.

Design choices

[LCV-DIST-STORE.2]

Core Verification returns a data structure called LightStore that contains light blocks (that contain a header).

[LCV-DIST-INIT.2]

Core Verification is called with

  • primary: the PeerID of a full node (with verification communicates)
  • root: a light block (the root of trust)
  • targetHeight: a height (the height of a header that should be obtained)

Temporal Properties

[LCV-DIST-SAFE.2]

It is always the case that every header in LightStore was generated by an instance of Tendermint consensus.

[LCV-DIST-LIVE.2]

If a new instance of Core Verification is called with a height targetHeight greater than root.Header.Height it must must eventually terminate.

  • If
    • the primary is correct (and locally has the block of targetHeight), and
    • the age of root is always less than the trusting period,
      then Core Verification adds a verified header hd with height targetHeight to LightStore and it terminates successfully

These definitions imply that if the primary is faulty, a header may or may not be added to LightStore. In any case, [LCV-DIST-SAFE.2] must hold. The invariant [LCV-DIST-SAFE.2] and the liveness requirement [LCV-DIST-LIVE.2] allow that verified headers are added to LightStore whose height was not passed to the verifier (e.g., intermediate headers used in bisection; see below). Note that for liveness, initially having a root within the trustinPeriod is not sufficient. However, as this specification will leave some freedom with respect to the strategy in which order to download intermediate headers, we do not give a more precise liveness specification here. After giving the specification of the protocol, we will discuss some liveness scenarios below.

Solving the sequential specification

This specification provides a partial solution to the sequential specification. The Verifier solves the invariant of the sequential part

[LCV-DIST-SAFE.2] => [LCV-SEQ-SAFE.1]

In the case the primary is correct, and root is a recent header in LightStore, the verifier satisfies the liveness requirements.

⋀ primary is correct
⋀ root.header.Time > now - trustingPeriod
⋀ [LCV-A-Comm.1] ⋀ ( ( [TMBC-CorrFull.1] ⋀ [LCV-DIST-LIVE.2] ) ⟹ [LCV-SEQ-LIVE.1] )

Part IV - Light Client Verification Protocol

We provide a specification for Light Client Verification. The local code for verification is presented by a sequential function VerifyToTarget to highlight the control flow of this functionality. We note that if a different concurrency model is considered for an implementation, the sequential flow of the function may be implemented with mutexes, etc. However, the light client verification is partitioned into three blocks that can be implemented and tested independently:

  • FetchLightBlock is called to download a light block (header) of a given height from a peer.
  • ValidAndVerified is a local code that checks the header.
  • Schedule decides which height to try to verify next. We keep this underspecified as different implementations (currently in Goland and Rust) may implement different optimizations here. We just provide necessary conditions on how the height may evolve.

Definitions

Data Types

The core data structure of the protocol is the LightBlock.

[LCV-DATA-LIGHTBLOCK.1]

type LightBlock struct {
  Header          Header
  Commit          Commit
  Validators      ValidatorSet
}

[LCV-DATA-LIGHTSTORE.2]

LightBlocks are stored in a structure which stores all LightBlock from initialization or received from peers.

type LightStore struct {
 ...
}

[LCV-DATA-LS-ROOT.2]

For each lightblock in a lightstore we record in a field verification-root of type Height.

verification-root records the height of a lightblock that can be used to verify the lightblock in one step

[LCV-INV-LS-ROOT.2]

At all times, if a lightblock b in a lightstore has b.verification-root = h, then

  • the lightstore contains a lightblock with height h, or
  • b has the minimal height of all lightblocks in lightstore, then b.verification-root should be nil.

The LightStore exposes the following functions to query stored LightBlocks.

[LCV-DATA-LS-STATE.1]

Each LightBlock is in one of the following states:

type VerifiedState int

const (
 StateUnverified = iota + 1
 StateVerified
 StateFailed
 StateTrusted
)

[LCV-FUNC-GET.1]

func (ls LightStore) Get(height Height) (LightBlock, bool)
  • Expected postcondition
    • returns a LightBlock at a given height or false in the second argument if the LightStore does not contain the specified LightBlock.

[LCV-FUNC-LATEST.1]

func (ls LightStore) Latest() LightBlock
  • Expected postcondition
    • returns the highest light block

[LCV-FUNC-ADD.1]

func (ls LightStore) Add(newBlock)
  • Expected precondition
    • the lightstore is empty
  • Expected postcondition
    • adds newBlock into light store

[LCV-FUNC-STORE.1]

func (ls LightStore) store_chain(newLS LightStore)
  • Expected postcondition
    • adds newLS to the lightStore.

[LCV-FUNC-LATEST-VERIF.2]

func (ls LightStore) LatestVerified() LightBlock
  • Expected postcondition
    • returns the highest light block whose state is StateVerified

[LCV-FUNC-FILTER.1]

func (ls LightStore) FilterVerified() LightStore
  • Expected postcondition
    • returns all the lightblocks of the lightstore with state StateVerified

[LCV-FUNC-UPDATE.2]

func (ls LightStore) Update(lightBlock LightBlock, verfiedState
VerifiedState, root-height Height)
  • Expected postcondition
    • the lightblock is part of the lightstore
    • The state of the LightBlock is set to verifiedState.
    • The verification-root of the LightBlock is set to root-height
func (ls LightStore) TraceTo(lightBlock LightBlock) (LightBlock, LightStore)
  • Expected postcondition
    • returns a trusted lightblock root from the lightstore with a height less than lightBlock
    • returns a lightstore that contains lightblocks that constitute a verification trace from root to lightBlock (including lightBlock)

Inputs

  • root: A light block that is trusted
  • primary: peerID
  • targetHeight: the height of the needed header

Configuration Parameters

  • trustThreshold: a float. Can be used if correctness should not be based on more voting power and 1/3.
  • trustingPeriod: a time duration [TMBC-TIME_PARAMS.1].
  • clockDrift: a time duration. Correction parameter dealing with only approximately synchronized clocks.

Variables

  • nextHeight: initially targetHeight

    nextHeight should be thought of the "height of the next header we need to download and verify"

Assumptions

[LCV-A-INIT.2]

  • root is from the blockchain

  • targetHeight > root.Header.Height

Invariants

[LCV-INV-TP.1]

It is always the case that LightStore.LatestTrusted.Header.Time > now - trustingPeriod.

If the invariant is violated, the light client does not have a header it can trust. A trusted header must be obtained externally, its trust can only be based on social consensus.
We use the convention that root is assumed to be verified.

Used Remote Functions

We use the functions commit and validators that are provided by the RPC client for Tendermint.

func Commit(height int64) (SignedHeader, error)
  • Implementation remark
    • RPC to full node n
    • JSON sent:
// POST /commit
{
 "jsonrpc": "2.0",
 "id": "ccc84631-dfdb-4adc-b88c-5291ea3c2cfb", // UUID v4, unique per request
 "method": "commit",
 "params": {
  "height": 1234
 }
}
  • Expected precondition
    • header of height exists on blockchain
  • Expected postcondition
    • if n is correct: Returns the signed header of height height from the blockchain if communication is timely (no timeout)
    • if n is faulty: Returns a signed header with arbitrary content
  • Error condition
    • if n is correct: precondition violated or timeout
    • if n is faulty: arbitrary error

----;

func Validators(height int64) (ValidatorSet, error)
  • Implementation remark
    • RPC to full node n
    • JSON sent:
// POST /validators
{
 "jsonrpc": "2.0",
 "id": "ccc84631-dfdb-4adc-b88c-5291ea3c2cfb", // UUID v4, unique per request
 "method": "validators",
 "params": {
  "height": 1234
 }
}
  • Expected precondition
    • header of height exists on blockchain
  • Expected postcondition
    • if n is correct: Returns the validator set of height height from the blockchain if communication is timely (no timeout)
    • if n is faulty: Returns arbitrary validator set
  • Error condition
    • if n is correct: precondition violated or timeout
    • if n is faulty: arbitrary error

----;

Communicating Function

[LCV-FUNC-FETCH.1]

func FetchLightBlock(peer PeerID, height Height) LightBlock
  • Implementation remark
    • RPC to peer at PeerID
    • calls Commit for height and Validators for height and height+1
  • Expected precondition
    • height is less than or equal to height of the peer [LCV-IO-PRE-HEIGHT.1]
  • Expected postcondition:
    • if node is correct:
      • Returns the LightBlock lb of height height that is consistent with the blockchain
      • lb.provider = peer [LCV-IO-POST-PROVIDER.1]
      • lb.Header is a header consistent with the blockchain
      • lb.Validators is the validator set of the blockchain at height nextHeight
      • lb.NextValidators is the validator set of the blockchain at height nextHeight + 1
    • if node is faulty: Returns a LightBlock with arbitrary content [TMBC-AUTH-BYZ.1]
  • Error condition
    • if n is correct: precondition violated
    • if n is faulty: arbitrary error
    • if lb.provider != peer
    • times out after 2 Delta (by assumption n is faulty)

----;

Core Verification

Outline

The VerifyToTarget is the main function and uses the following functions.

  • FetchLightBlock is called to download the next light block. It is the only function that communicates with other nodes
  • ValidAndVerified checks whether header is valid and checks if a new lightBlock should be trusted based on a previously verified lightBlock.
  • Schedule decides which height to try to verify next

In the following description of VerifyToTarget we do not deal with error handling. If any of the above function returns an error, VerifyToTarget just passes the error on.

[LCV-FUNC-MAIN.2]

func VerifyToTarget(primary PeerID, root LightBlock,
                    targetHeight Height) (LightStore, Result) {

    lightStore = new LightStore;
    lightStore.Update(root, StateVerified, root.verifiedBy);
    nextHeight := targetHeight;

    for lightStore.LatestVerified.height < targetHeight {

        // Get next LightBlock for verification
        current, found := lightStore.Get(nextHeight)
        if !found {
            current = FetchLightBlock(primary, nextHeight)
            lightStore.Update(current, StateUnverified, nil)
        }

        // Verify
        verdict = ValidAndVerified(lightStore.LatestVerified, current)

        // Decide whether/how to continue
        if verdict == SUCCESS {
            lightStore.Update(current, StateVerified, lightStore.LatestVerified.Height)
        }
        else if verdict == NOT_ENOUGH_TRUST {
            // do nothing
            // the light block current passed validation, but the validator
            // set is too different to verify it. We keep the state of
            // current at StateUnverified. For a later iteration, Schedule
            // might decide to try verification of that light block again.
        }
        else {
            // verdict is some error code
            lightStore.Update(current, StateFailed, nil)
            return (nil, ResultFailure)
        }
        nextHeight = Schedule(lightStore, nextHeight, targetHeight)
    }
    return (lightStore.FilterVerified, ResultSuccess)
}
  • Expected precondition
    • root is within the trustingPeriod [LCV-PRE-TP.1]
    • targetHeight is greater than the height of root
  • Expected postcondition:
    • returns lightStore that contains a LightBlock that corresponds to a block of the blockchain of height targetHeight (that is, the LightBlock has been added to lightStore) [LCV-POST-LS.1]
  • Error conditions
    • if the precondition is violated
    • if ValidAndVerified or FetchLightBlock report an error
    • if [LCV-INV-TP.1] is violated

Details of the Functions

[LCV-FUNC-VALID.2]

func ValidAndVerified(trusted LightBlock, untrusted LightBlock) Result
  • Expected precondition:
    • untrusted is valid, that is, satisfies the soundness checks
    • untrusted is well-formed, that is,
      • untrusted.Header.Time < now + clockDrift
      • untrusted.Validators = hash(untrusted.Header.Validators)
      • untrusted.NextValidators = hash(untrusted.Header.NextValidators)
    • trusted.Header.Time > now - trustingPeriod
    • the Height and Time of trusted are smaller than the Height and Time of untrusted, respectively
    • the untrusted.Header is well-formed (passes the tests from [block]), and in particular
      • if the untrusted header unstrusted.Header is the immediate successor of trusted.Header, then it holds that
        • trusted.Header.NextValidators = untrusted.Header.Validators, and moreover,
        • untrusted.Header.Commit
          • contains signatures by more than two-thirds of the validators
          • contains no signature from nodes that are not in trusted.Header.NextValidators
  • Expected postcondition:
    • Returns SUCCESS:
      • if untrusted is the immediate successor of trusted, or otherwise,
      • if the signatures of a set of validators that have more than max(1/3,trustThreshold) of voting power in trusted.Nextvalidators is contained in untrusted.Commit (that is, header passes the tests [TMBC-VAL-CONTAINS-CORR.1] and [TMBC-VAL-COMMIT.1])
    • Returns NOT_ENOUGH_TRUST if:
      • untrusted is not the immediate successor of trusted and the max(1/3,trustThreshold) threshold is not reached (that is, if [TMBC-VAL-CONTAINS-CORR.1] fails and header is does not violate the soundness checks [block]).
  • Error condition:
    • if precondition violated

----;

[LCV-FUNC-SCHEDULE.1]

func Schedule(lightStore, nextHeight, targetHeight) Height
  • Implementation remark: If picks the next height to be verified. We keep the precise choice of the next header under-specified. It is subject to performance optimizations that do not influence the correctness
  • Expected postcondition: [LCV-SCHEDULE-POST.1] Return H s.t.
    1. if lightStore.LatestVerified.Height = nextHeight and lightStore.LatestVerified < targetHeight then
      nextHeight < H <= targetHeight
    2. if lightStore.LatestVerified.Height < nextHeight and lightStore.LatestVerified.Height < targetHeight then
      lightStore.LatestVerified.Height < H < nextHeight
    3. if lightStore.LatestVerified.Height = targetHeight then
      H = targetHeight

Case i. captures the case where the light block at height nextHeight has been verified, and we can choose a height closer to the targetHeight. As we get the lightStore as parameter, the choice of the next height can depend on the lightStore, e.g., we can pick a height for which we have already downloaded a light block. In Case ii. the header of nextHeight could not be verified, and we need to pick a smaller height. In Case iii. is a special case when we have verified the targetHeight.

Solving the distributed specification

Analogous to [001_published]

Liveness Scenarios

Analogous to [001_published]

Part V - Supporting the IBC Relayer

The above specification focuses on the most common case, which also constitutes the most challenging task: using the Tendermint security model to verify light blocks without downloading all intermediate blocks. To focus on this challenge, above we have restricted ourselves to the case where targetHeight is greater than the height of any trusted header. This simplified presentation of the algorithm as initially lightStore.LatestVerified() is less than targetHeight, and in the process of verification lightStore.LatestVerified() increases until targetHeight is reached.

For IBC there are two additional challenges:

  1. it might be that some "older" header is needed, that is, targetHeight < lightStore.LatestVerified(). The supervisor checks whether it is in this case by calling LatestPrevious and MinVerified and if so it calls Backwards. All these functions are specified below.

  2. In order to submit proof of a light client attack, a relayer may need to submit a verification trace. This it is important to compute such a trace efficiently. That it can be done is based on the invariant [LCV-INV-LS-ROOT.2] that needs to be maintained by the light client. In particular VerifyToTarget and Backwards need to take care of setting verification-root.

[LCV-FUNC-LATEST-PREV.2]

func (ls LightStore) LatestPrevious(height Height) (LightBlock, bool)
  • Expected postcondition
    • returns a light block lb that satisfies:
      • lb is in lightStore
      • lb is in StateTrusted
      • lb is not expired
      • lb.Header.Height < height
      • for all b in lightStore s.t. b is trusted and not expired it holds lb.Header.Height >= b.Header.Height
    • false in the second argument if the LightStore does not contain such an lb.

----;

[LCV-FUNC-LOWEST.2]

func (ls LightStore) Lowest() (LightBlock)
  • Expected postcondition
    • returns the lowest trusted light block within trusting period

----;

[LCV-FUNC-MIN.2]

func (ls LightStore) MinVerified() (LightBlock, bool)
  • Expected postcondition
    • returns a light block lb that satisfies:
      • lb is in lightStore
      • lb.Header.Height is minimal in the lightStore
    • false in the second argument if the LightStore does not contain such an lb.

If a height that is smaller than the smallest height in the lightstore is required, we check the hashes backwards. This is done with the following function:

[LCV-FUNC-BACKWARDS.2]

func Backwards (primary PeerID, root LightBlock, targetHeight Height)
               (LightStore, Result) {
  
    lb := root;
    lightStore := new LightStore;
    lightStore.Update(lb, StateTrusted, lb.verifiedBy)

    latest := lb.Header
    for i := lb.Header.height - 1; i >= targetHeight; i-- {
        // here we download height-by-height. We might first download all
        // headers down to targetHeight and then check them.
        current := FetchLightBlock(primary,i)
        if (hash(current) != latest.Header.LastBlockId) {
            return (nil, ResultFailure)
        }
        else {
            // latest and current are linked together by LastBlockId
            // therefore it is not relevant which we verified first
            // for consistency, we store latest was veried using
            // current so that the verifiedBy is always pointing down
            // the chain
            lightStore.Update(current, StateTrusted, nil)
            lightStore.Update(latest, StateTrusted, current.Header.Height)
        }
        latest = current
    }
    return (lightStore, ResultSuccess)
}

References

[block] Specification of the block data structure.

[RPC] RPC client for Tendermint

[attack-detector] The specification of the light client attack detector.

[fullnode] Specification of the full node API

[ibc-rs] Rust implementation of IBC modules and relayer.

[lightclient] The light client ADR [77d2651 on Dec 27, 2019].

Light Client Verificaiton

[LCV-FUNC-VERIFYCOMMITLIGHT.1]

VerifyCommitLight verifies that 2/3+ of the signatures for a validator set were for a given blockID. The function will finish early and thus may not check all signatures.

func VerifyCommitLight(chainID string, vals *ValidatorSet, blockID BlockID,
height int64, commit *Commit) error {
  // run a basic validation of the arguments
  if err := verifyBasicValsAndCommit(vals, commit, height, blockID); err != nil {
    return err
  }

  // calculate voting power needed
  votingPowerNeeded := vals.TotalVotingPower() * 2 / 3

  var (
    val                *Validator
    valIdx             int32
    seenVals                 = make(map[int32]int, len(commit.Signatures))
    talliedVotingPower int64 = 0
    voteSignBytes      []byte
  )
  for idx, commitSig := range commit.Signatures {
    // ignore all commit signatures that are not for the block
    if !commitSig.ForBlock() {
      continue
    }

    // If the vals and commit have a 1-to-1 correspondance we can retrieve
    // them by index else we need to retrieve them by address
    if lookUpByIndex {
      val = vals.Validators[idx]
    } else {
      valIdx, val = vals.GetByAddress(commitSig.ValidatorAddress)  

      // if the signature doesn't belong to anyone in the validator set
      // then we just skip over it
      if val == nil {
        continue
      }

      // because we are getting validators by address we need to make sure
      // that the same validator doesn't commit twice
      if firstIndex, ok := seenVals[valIdx]; ok {
        secondIndex := idx
        return fmt.Errorf("double vote from %v (%d and %d)", val, firstIndex, secondIndex)
      }
      seenVals[valIdx] = idx
    }

    voteSignBytes = commit.VoteSignBytes(chainID, int32(idx))

    if !val.PubKey.VerifySignature(voteSignBytes, commitSig.Signature) {
      return fmt.Errorf("wrong signature (#%d): %X", idx, commitSig.Signature)
    }

    // Add the voting power of the validator
    // to the tally
    talliedVotingPower += val.VotingPower

    // check if we have enough signatures and can thus exit early
    if talliedVotingPower > votingPowerNeeded {
      return nil
    }
  }

  if got, needed := talliedVotingPower, votingPowerNeeded; got <= needed {
    return ErrNotEnoughVotingPowerSigned{Got: got, Needed: needed}
  }

  return nil
}

Ivy Proofs

Copyright (c) 2020 Galois, Inc.
SPDX-License-Identifier: Apache-2.0

Contents

This folder contains:

  • tendermint.ivy, a specification of Tendermint algorithm as described in The latest gossip on BFT consensus by E. Buchman, J. Kwon, Z. Milosevic.
  • abstract_tendermint.ivy, a more abstract specification of Tendermint that is more verification-friendly.
  • classic_safety.ivy, a proof that Tendermint satisfies the classic safety property of BFT consensus: if every two quorums have a well-behaved node in common, then no two well-behaved nodes ever disagree.
  • accountable_safety_1.ivy, a proof that, assuming every quorum contains at least one well-behaved node, if two well-behaved nodes disagree, then there is evidence demonstrating at least f+1 nodes misbehaved.
  • accountable_safety_2.ivy, a proof that, regardless of any assumption about quorums, well-behaved nodes cannot be framed by malicious nodes. In other words, malicious nodes can never construct evidence that incriminates a well-behaved node.
  • network_shim.ivy, the network model and a convenience shim object to interface with the Tendermint specification.
  • domain_model.ivy, a specification of the domain model underlying the Tendermint specification, i.e. rounds, value, quorums, etc.

All specifications and proofs are written in Ivy.

The license above applies to all files in this folder.

Building and running

The easiest way to check the proofs is to use Docker.

  1. Install Docker and Docker Compose.
  2. Build a Docker image: docker compose build
  3. Run the proofs inside the Docker container: docker compose run tendermint-proof. This will check all the proofs with the ivy_check command and write the output of ivy_check to a subdirectory of `./output/'

Security

Report vulnerabilities privately using the security policy.

Vulnerability names

This inherited list records candidate names for vulnerabilities. Follow the security policy to report a vulnerability; names in this list do not indicate that a vulnerability exists or has been disclosed.

Here is a list of pre-allocated vulnerability names, to be crossed-out as they are used. Hopefully we don't make it all the way through the list!

Alderfly Aphid Ant Bedbug Beetle Bookworm Bumblebee Butterfly Caterpillar Cicada Centipede Cockroach Cranefly Cricket Damselfly Dragonfly Earthworm Earwig Flea Firefly Gnat Grasshopper Hornet Horsefly Jumpingbean Katydid Lacewing Ladybug Leafroller Locust Louse Maggot Mantis Mealybug Midge Mite Mosquito Moth Palmerworm Parasite Sawfly Scarab Silkworm Silverfish Skeletonizer Slug Sphinx Stinkbug Termite Thrip Viceroy Walkingstick Wasp Weevil Whitefly Yellowjacket

ADR 073: Adopt LibP2P

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 2021-11-02: Initial Draft (@tychoish)

Status

Proposed.

Context

As part of the upstream 0.35 development cycle, the original team completed the first phase of the work described in ADRs 61 and 62, which included a large scale refactoring of the reactors and the p2p message routing. This replaced the switch and many of the other legacy components without breaking protocol or network-level interoperability and left the legacy connection/socket handling code.

Following the release, the team has reexamined the state of the code and the design, as well as Tenderdash's requirements. The notes from that process are available in the P2P Roadmap RFC.

This ADR supersedes the decisions made in ADRs 60 and 61, but builds on the completed portions of this work. Previously, the boundaries of peer management, message handling, and the higher level business logic (e.g., "the reactors") were intermingled, and core elements of the p2p system were responsible for the orchestration of higher-level business logic. Refactoring the legacy components made it more obvious that this entanglement of responsibilities had outsized influence on the entire implementation, making it difficult to iterate within the current abstractions. It would not be viable to maintain interoperability with legacy systems while also achieving many of our broader objectives.

LibP2P is a thoroughly-specified implementation of a peer-to-peer networking stack, designed specifically for systems such as ours. Adopting LibP2P as the basis of Tenderdash will allow the Tenderdash team to focus more of their time on other differentiating aspects of the system, and make it possible for the ecosystem as a whole to take advantage of tooling and efforts of the LibP2P platform.

Alternative Approaches

As discussed in the P2P Roadmap RFC, the primary alternative would be to continue development of Tenderdash's home-grown peer-to-peer layer. While that would give the Tenderdash team maximal control over the peer system, the current design is unexceptional on its own merits, and the prospective maintenance burden for this system exceeds our tolerances for the medium term.

Tenderdash can and should differentiate itself not on the basis of its networking implementation or peer management tools, but providing a consistent operator experience, a battle-tested consensus algorithm, and an ergonomic user experience.

Decision

The upstream proposal targeted libp2p adoption during its 0.37 development cycle, replacing the bespoke Tenderdash P2P stack. This will remove the Endpoint, Transport, Connection, and PeerManager abstractions and leave the reactors, p2p.Router and p2p.Channel abstractions.

LibP2P may obviate the need for a dedicated peer exchange (PEX) reactor, which would also in turn obviate the need for a dedicated seed mode. If this is the case, then all of this functionality would be removed.

If it turns out (based on the advice of Protocol Labs) that it makes sense to maintain separate pubsub or gossipsub topics per-message-type, then the Router abstraction could also be entirely subsumed.

Detailed Design

Implementation Changes

The seams in the P2P implementation between the higher level constructs (reactors), the routing layer (Router) and the lower level connection and peer management code make this operation relatively straightforward to implement. A key goal in this design is to minimize the impact on the reactors (potentially entirely,) and completely remove the lower level components (e.g., Transport, Connection and PeerManager) using the separation afforded by the Router layer. The current state of the code makes these changes relatively surgical, and limited to a small number of methods:

  • p2p.Router.OpenChannel will still return a Channel structure which will continue to serve as a pipe between the reactors and the Router. The implementation will no longer need the queue implementation, and will instead start goroutines that are responsible for routing the messages from the channel to libp2p fundamentals, replacing the current p2p.Router.routeChannel.

  • The current p2p.Router.dialPeers and p2p.Router.acceptPeers, are responsible for establishing outbound and inbound connections, respectively. These methods will be removed, along with p2p.Router.openConnection, and the libp2p connection manager will be responsible for maintaining network connectivity.

  • The p2p.Channel interface will change to replace Go channels with a more functional interface for sending messages. New methods on this object will take contexts to support safe cancellation, and return errors, and will block rather than running asynchronously. The Out channel through which reactors send messages to Peers, will be replaced by a Send method, and the Error channel will be replaced by an Error method.

  • Reactors will be passed an interface that will allow them to access Peer information from libp2p. This will supplant the p2p.PeerUpdates subscription.

  • Add some kind of heartbeat message at the application level (e.g. with a reactor,) potentially connected to libp2p's DHT to be used by reactors for service discovery, message targeting, or other features.

  • Replace the existing/legacy handshake protocol with Noise.

This project will initially use the TCP-based transport protocols within libp2p. QUIC is also available as an option that we may implement later. We will not support mixed networks in the initial release, but will revisit that possibility later if there is a demonstrated need.

Upgrade and Compatibility

Because the routers and all current P2P libraries are internal packages and not part of the public API, the only changes to the public API surface area of Tenderdash will be different configuration file options, replacing the current P2P options with options relevant to libp2p.

However, it will not be possible to run a network with both networking stacks active at once, so the upgrade to the version of Tenderdash will need to be coordinated between all nodes of the network. This is consistent with the expectations around upgrades for Tenderdash moving forward, and will help manage both the complexity of the project and the implementation timeline.

Open Questions

  • What is the role of Protocol Labs in the implementation of libp2p in Tenderdash, both during the initial implementation and on an ongoing basis thereafter?

  • Should all P2P traffic for a given node be pushed to a single topic, so that a topic maps to a specific ChainID, or should each reactor (or type of message) have its own topic? How many topics can a libp2p network support? Is there testing that validates the capabilities?

  • Tenderdash presently provides a very coarse QoS-like functionality using priorities based on message-type. This intuitively/theoretically ensures that evidence and consensus messages don't get starved by blocksync/statesync messages. It's unclear if we can or should attempt to replicate this with libp2p.

  • What kind of QoS functionality does libp2p provide and what kind of metrics does libp2p provide about it's QoS functionality?

  • Is it possible to store additional (and potentially arbitrary) information into the DHT as part of the heartbeats between nodes, such as the latest height, and then access that in the reactors. How frequently can the DHT be updated?

  • Does it make sense to have reactors continue to consume inbound messages from a Channel (In) or is there another interface or pattern that we should consider?

    • We should avoid exposing Go channels when possible, and likely some kind of alternate iterator likely makes sense for processing messages within the reactors.
  • What are the security and protocol implications of tracking information from peer heartbeats and exposing that to reactors?

  • How much (or how little) configuration can Tenderdash provide for libp2p, particularly on the first release?

    • In general, we should not support byo-functionality for libp2p components within Tenderdash, and reduce the configuration surface area, as much as possible.
  • What are the best ways to provide request/response semantics for reactors on top of libp2p? Will it be possible to add request/response semantics in a future release or is there anticipatory work that needs to be done as part of the initial release?

Consequences

Positive

  • Reduce the maintenance burden for the Tenderdash Core team by removing a large swath of legacy code that has proven to be difficult to modify safely.

  • Remove the responsibility for maintaining and developing the entire peer management system (p2p) and stack.

  • Provide users with a more stable peer and networking system, Tenderdash can improve operator experience and network stability.

Negative

  • By deferring to library implementations for peer management and networking, Tenderdash loses some flexibility for innovating at the peer and networking level. However, Tenderdash should be innovating primarily at the consensus layer, and libp2p does not preclude optimization or development in the peer layer.

  • Libp2p is a large dependency and Tenderdash would become dependent upon Protocol Labs' release cycle and prioritization for bug fixes. If this proves onerous, it's possible to maintain a vendor fork of relevant components as needed.

Neutral

  • N/A

References

ADR 74: Migrate Timeout Parameters to Consensus Parameters

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 03-Jan-2022: Initial draft (@williambanfield)
  • 13-Jan-2022: Updated to indicate work on upgrade path needed (@williambanfield)

Status

Proposed

Context

Background

Tenderdash's consensus timeout parameters are currently configured locally by each validator in the validator's config.toml. This means that the validators on a Tenderdash network may have different timeouts from each other. There is no reason for validators on the same network to configure different timeout values. Proper functioning of the Tenderdash consensus algorithm relies on these parameters being uniform across validators.

The configurable values are as follows:

  • TimeoutPropose
    • How long the consensus algorithm waits for a proposal block before issuing a prevote.
    • If no prevote arrives by TimeoutPropose, then the consensus algorithm will issue a nil prevote.
  • TimeoutProposeDelta
    • How much the TimeoutPropose grows each round.
  • TimeoutPrevote
    • How long the consensus algorithm waits after receiving +2/3 prevotes with no quorum for a value before issuing a precommit for nil. (See the arXiv paper, Algorithm 1, Line 34)
  • TimeoutPrevoteDelta
    • How much the TimeoutPrevote increases with each round.
  • TimeoutPrecommit
    • How long the consensus algorithm waits after receiving +2/3 precommits that do not have a quorum for a value before entering the next round. (See the arXiv paper, Algorithm 1, Line 47)
  • TimeoutPrecommitDelta
    • How much the TimeoutPrecommit increases with each round.
  • TimeoutCommit
    • How long the consensus algorithm waits after committing a block but before starting the new height.
    • This gives a validator a chance to receive slow precommits.
  • SkipTimeoutCommit
    • Make progress as soon as the node has 100% of the precommits.

Overview of Change

We will consolidate the timeout parameters and migrate them from the node-local config.toml file into the network-global consensus parameters.

The 8 timeout parameters will be consolidated down to 6. These will be as follows:

  • TimeoutPropose
    • Same as current TimeoutPropose.
  • TimeoutProposeDelta
    • Same as current TimeoutProposeDelta.
  • TimeoutVote
    • How long validators wait for votes in both the prevote and precommit phase of the consensus algorithm. This parameter subsumes the current TimeoutPrevote and TimeoutPrecommit parameters.
  • TimeoutVoteDelta
    • How much the TimeoutVote will grow each successive round. This parameter subsumes the current TimeoutPrevoteDelta and TimeoutPrecommitDelta parameters.
  • TimeoutCommit
    • Same as current TimeoutCommit.
  • BypassCommitTimeout
    • Same as current SkipTimeoutCommit, renamed for clarity.

A safe default will be provided by Tenderdash for each of these parameters and networks will be able to update the parameters as they see fit. Local updates to these parameters will no longer be possible; instead, the application will control updating the parameters. Applications using the Cosmos SDK will be automatically be able to change the values of these consensus parameters via a governance proposal.

The original proposal considered this change low-risk because many upstream networks used the default timeout values. This observation does not replace validation of the settings required by a Tenderdash network.

Why this parameter consolidation?

Reducing the number of parameters is good for UX. Fewer superfluous parameters makes running and operating a Tenderdash network less confusing.

The Prevote and Precommit messages are both similar sizes, require similar amounts of processing so there is no strong need for them to be configured separately.

The TimeoutPropose parameter governs how long Tenderdash will wait for the proposed block to be gossiped. Blocks are much larger than votes and therefore tend to be gossiped much more slowly. It therefore makes sense to keep TimeoutPropose and the TimeoutProposeDelta as parameters separate from the vote timeouts.

TimeoutCommit is used by chains to ensure that the network waits for the votes from slower validators before proceeding to the next height. Without this timeout, the votes from slower validators would consistently not be included in blocks and those validators would not be counted as 'up' from the chain's perspective. Being down damages a validator's reputation and causes potential stakers to think twice before delegating to that validator.

TimeoutCommit also prevents the network from producing the next height as soon as validators on the fastest hardware with a summed voting power of +2/3 of the network's total have completed execution of the block. Allowing the network to proceed as soon as the fastest +2/3 completed execution would have a cumulative effect over heights, eventually leaving slower validators unable to participate in consensus at all. TimeoutCommit therefore allows networks to have greater variability in hardware. Additional discussion of this can be found in upstream issue 5911 and spec issue 359.

Alternative Approaches

Hardcode the parameters

Many Tenderdash networks run on similar cloud-hosted infrastructure. Therefore, they have similar bandwidth and machine resources. The timings for propagating votes and blocks are likely to be reasonably similar across networks. As a result, the timeout parameters are good candidates for being hardcoded. Hardcoding the timeouts in Tenderdash would mean entirely removing these parameters from any configuration that could be altered by either an application or a node operator. Instead, Tenderdash would ship with a set of timeouts and all applications using Tenderdash would use this exact same set of values.

While Tenderdash nodes often run with similar bandwidth and on similar cloud-hosted machines, there are enough points of variability to make configuring consensus timeouts meaningful. Namely, Tenderdash network topologies are likely to be very different from chain to chain. Additionally, applications may vary greatly in how long the Commit phase may take. Applications that perform more work during Commit require a longer TimeoutCommit to allow the application to complete its work and be prepared for the next height.

Decision

The decision has been made to implement this work, with the caveat that the specific mechanism for introducing the new parameters to chains is still ongoing.

Detailed Design

New Consensus Parameters

A new TimeoutParams message will be added to the [params.proto file][consensus-params-proto]. This message will have the following form:

message TimeoutParams {
 google.protobuf.Duration propose = 1;
 google.protobuf.Duration propose_delta = 2;
 google.protobuf.Duration vote = 3;
 google.protobuf.Duration vote_delta = 4;
 google.protobuf.Duration commit = 5;
 bool bypass_commit_timeout = 6;
}

This new message will be added as a field into the [ConsensusParams message][consensus-params-proto]. The same default values that are currently set for these parameters in the local configuration file will be used as the defaults for these new consensus parameters in the consensus parameter defaults.

The new consensus parameters will be subject to the same validity rules as the current configuration values, namely, each value must be non-negative.

Migration

The new ConsensusParameters will be added during an upcoming release. In this release, the old config.toml parameters will cease to control the timeouts and an error will be logged on nodes that continue to specify these values. The specific mechanism by which these parameters will added to a chain is being discussed in RFC-009 and will be decided ahead of the next release.

The specific mechanism for adding these parameters depends on work related to soft upgrades, which is still ongoing.

Consequences

Positive

  • Timeout parameters will be equal across all of the validators in a Tenderdash network.
  • Remove superfluous timeout parameters.

Negative

Neutral

  • Timeout parameters require consensus to change.

References

ADR 075: RPC Event Subscription Interface

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 01-Mar-2022: Update long-polling interface (@creachadair).
  • 10-Feb-2022: Updates to reflect implementation.
  • 26-Jan-2022: Marked accepted.
  • 22-Jan-2022: Updated and expanded (@creachadair).
  • 20-Nov-2021: Initial draft (@creachadair).

Status

Accepted


Background & Context

For context, see RFC 006: Event Subscription.

The Tenderdash RPC service permits clients to subscribe to the event stream generated by a consensus node. This allows clients to observe the state of the consensus network, including details of the consensus algorithm state machine, proposals, transaction delivery, and block completion. The application may also attach custom key-value attributes to events to expose application-specific details to clients.

The event subscription API in the RPC service currently comprises three methods:

  1. subscribe: A request to subscribe to the events matching a specific query expression. Events can be filtered by their key-value attributes, including custom attributes provided by the application.

  2. unsubscribe: A request to cancel an existing subscription based on its query expression.

  3. unsubscribe_all: A request to cancel all existing subscriptions belonging to the client.

There are some important technical and UX issues with the current RPC event subscription API. The rest of this ADR outlines these problems in detail, and proposes a new API scheme intended to address them.

Issue 1: Persistent connections

To subscribe to a node's event stream, a client needs a persistent connection to the node. Unlike the other methods of the service, for which each call is serviced by a short-lived HTTP round trip, subscription delivers a continuous stream of events to the client by hijacking the HTTP channel for a websocket. The stream (and hence the HTTP request) persists until either the subscription is explicitly cancelled, or the connection is closed.

There are several problems with this API:

  1. Expensive per-connection state: The server must maintain a substantial amount of state per subscriber client:

    • The current implementation uses a WebSocket for each active subscriber. The connection must be maintained even if there are no matching events for a given client.

      The server can drop idle connections to save resources, but doing so terminates all subscriptions on those connections and forces those clients to re-connect, adding additional resource churn for the server.

    • In addition, the server maintains a separate buffer of undelivered events for each client. This is to reduce the dual risks that a client will miss events, and that a slow client could "push back" on the publisher, impeding the progress of consensus.

      Because event traffic is quite bursty, queues can potentially take up a lot of memory. Moreover, each subscriber may have a different filter query, so the server winds up having to duplicate the same events among multiple subscriber queues. Not only does this add memory pressure, but it does so most at the worst possible time, i.e., when the server is already under load from high event traffic.

  2. Operational access control is difficult: The server's websocket interface exposes all the RPC service endpoints, not only the subscription methods. This includes methods that allow callers to inject arbitrary transactions (broadcast_tx_*) and evidence (broadcast_evidence) into the network, remove transactions (remove_tx), and request arbitrary amounts of chain state.

    Filtering requests to the GET endpoint is straightforward: A reverse proxy like nginx can easily filter methods by URL path. Filtering POST requests takes a bit more work, but can be managed with a filter program that speaks FastCGI and parses JSON-RPC request bodies.

    Filtering the websocket interface requires a dedicated proxy implementation. Although nginx can reverse-proxy websockets, it does not support filtering websocket traffic via FastCGI. The operator would need to either implement a custom nginx extension module or build and run a standalone proxy that implements websocket and filters each session. Apart from the work, this also makes the system even more resource intensive, as well as introducing yet another connection that could potentially time out or stall on full buffers.

    Even for the simple case of restricting access to only event subscription, there is no easy solution currently: Once a caller has access to the websocket endpoint, it has complete access to the RPC service.

Issue 2: Inconvenient client API

The subscription interface has some inconvenient features for the client as well as the server. These include:

  1. Non-standard protocol: The RPC service is mostly JSON-RPC 2.0, but the subscription interface diverges from the standard.

    In a standard JSON-RPC 2.0 call, the client initiates a request to the server with a unique ID, and the server concludes the call by sending a reply for that ID. The subscribe implementation, however, sends multiple responses to the client's request:

    • The client sends subscribe with some ID x and the desired query

    • The server responds with ID x and an empty confirmation response.

    • The server then (repeatedly) sends event result responses with ID x, one for each item with a matching event.

    Standard JSON-RPC clients will reject the subsequent replies, as they announce a request ID (x) that is already complete. This means a caller has to implement Tenderdash-specific handling for these responses.

    Moreover, the result format is different between the initial confirmation and the subsequent responses. This means a caller has to implement special logic for decoding the first response versus the subsequent ones.

  2. No way to detect data loss: The subscriber connection can be terminated for many reasons. Even ignoring ordinary network issues (e.g., packet loss):

    • The server will drop messages and/or close the websocket if its write buffer fills, or if the queue of undelivered matching events is not drained fast enough. The client has no way to discover that messages were dropped even if the connection remains open.

    • Either the client or the server may close the websocket if the websocket PING and PONG exchanges are not handled correctly, or frequently enough. Even if correctly implemented, this may fail if the system is under high load and cannot service those control messages in a timely manner.

    When the connection is terminated, the server drops all the subscriptions for that client (as if it had called unsubscribe_all). Even if the client reconnects, any events that were published during the period between the disconnect and re-connect and re-subscription will be silently lost, and the client has no way to discover that it missed some relevant messages.

  3. No way to replay old events: Even if a client knew it had missed some events (due to a disconnection, for example), the API provides no way for the client to "play back" events it may have missed.

  4. Large response sizes: Some event data can be quite large, and there can be substantial duplication across items. The API allows the client to select which events are reported, but has no way to control which parts of a matching event it wishes to receive.

    This can be costly on the server (which has to marshal those data into JSON), the network, and the client (which has to unmarshal the result and then pick through for the components that are relevant to it).

    Besides being inefficient, this also contributes to some of the persistent connection issues mentioned above, e.g., filling up the websocket write buffer and forcing the server to queue potentially several copies of a large value in memory.

  5. Client identity is tied to network address: The Tenderdash event API identifies each subscriber by a (Client ID, Query) pair. In the RPC service, the query is provided by the client, but the client ID is set to the TCP address of the client (typically "host:port" or "ip:port").

    This means that even if the server did not drop subscriptions immediately when the websocket connection is closed, a client may not be able to reattach to its existing subscription. Dialing a new connection is likely to result in a different port (and, depending on their own proxy setup, possibly a different public IP).

    In isolation, this problem would be easy to work around with a new subscription parameter, but it would require several other changes to the handling of event subscriptions for that workaround to become useful.


Decision

To address the described problems, we will:

  1. Introduce a new API for event subscription to the Tenderdash RPC service. The proposed API is described in Detailed Design below.

  2. The upstream proposal targeted the v0.36 release, during which the current ("streaming") API will remain available as-is, but deprecated.

  3. The streaming API will be entirely removed in release v0.37, which will require all users of event subscription to switch to the new API.

Point for discussion: Given that ABCI++ and PBTS are the main priorities for v0.36, it would be fine to slip the first phase of this work to v0.37. Unless there is a time problem, however, the proposed design does not disrupt the work on ABCI++ or PBTS, and will not increase the scope of breaking changes. Therefore the plan is to begin in v0.36 and slip only if necessary.


Detailed Design

Design Goals

Specific goals of this design include:

  1. Remove the need for a persistent connection to each subscription client. Subscribers should use the same HTTP request flow for event subscription requests as for other RPC calls.

  2. The server retains minimal state (possibly none) per-subscriber. In particular:

    • The server does not buffer unconsumed writes nor queue undelivered events on a per-client basis.
    • A client that stalls or goes idle does not cost the server any resources.
    • Any event data that is buffered or stored is shared among all subscribers, and is not duplicated per client.
  3. Slow clients have no impact (or minimal impact) on the rate of progress of the consensus algorithm, beyond the ambient overhead of servicing individual RPC requests.

  4. Clients can tell when they have missed events matching their subscription, within some reasonable (configurable) window of time, and can "replay" events within that window to catch up.

  5. Nice to have: It should be easy to use the event subscription API from existing standard tools and libraries, including command-line use for testing and experimentation.

Definitions

  • The event stream of a node is a single, time-ordered, heterogeneous stream of event items.

  • Each event item comprises an event datum (for example, block header metadata for a new-block event), and zero or more optional events.

  • An event means the ABCI Event data type, which comprises a string type and zero or more string key-value event attributes.

    The use of the new terms "event item" and "event datum" is to avert confusion between the values that are published to the event bus (what we call here "event items") and the ABCI Event data type.

  • The node assigns each event item a unique identifier string called a cursor. A cursor must be unique among all events published by a single node, but it is not required to be unique globally across nodes.

    Cursors are time-ordered so that given event items A and B, if A was published before B, then cursor(A) < cursor(B) in lexicographic order.

    A minimum viable cursor implementation is a tuple consisting of a timestamp and a sequence number (e.g., 16CCC798FB5F4670-0123). However, it may also be useful to append basic type information to a cursor, to allow efficient filtering (e.g., 16CCC87E91869050-0091:BeginBlock).

    The initial implementation will use the minimum viable format.

Discussion

The node maintains an event log, a shared ordered record of the events published to its event bus within an operator-configurable time window. The initial implementation will store the event log in-memory, and the operator will be given two per-node configuration settings. Note, these names are provisional:

  • [rpc] event-log-window-size: A duration before the latest published event, during which the node will retain event items published. Setting this value to zero disables event subscription.

  • [rpc] event-log-max-items: A maximum number of event items that the node will retain within the time window. If the number of items exceeds this value, the node discardes the oldest items in the window. Setting this value to zero means that no limit is imposed on the number of items.

The node will retain all events within the time window, provided they do not exceed the maximum number. These config parameters allow the operator to loosely regulate how much memory and storage the node allocates to the event log. The client can use the server reply to tell whether the events it wants are still available from the event log.

The event log is shared among all subscribers to the node.

Discussion point: Should events persist across node restarts?

The current event API does not persist events across restarts, so this new design does not either. Note, however, that we may "spill" older event data to disk as a way of controlling memory use. Such usage is ephemeral, however, and does not need to be tracked as node data (e.g., it could be temp files).

Query API

To retrieve event data, the client will call the (new) RPC method events. The parameters of this method will correspond to the following Go types:

type EventParams struct {
    // Optional filter spec. If nil or empty, all items are eligible.
    Filter *Filter `json:"filter"`

    // The maximum number of eligible results to return.
    // If zero or negative, the server will report a default number.
    MaxResults int `json:"max_results"`

    // Return only items after this cursor. If empty, the limit is just
    // before the the beginning of the event log.
    After string `json:"after"`

    // Return only items before this cursor.  If empty, the limit is just
    // after the head of the event log.
    Before string `json:"before"`

    // Wait for up to this long for events to be available.
    WaitTime time.Duration `json:"wait_time"`
}

type Filter struct {
    Query string `json:"query"`
}

Discussion point: The initial implementation will not cache filter queries for the client. If this turns out to be a performance issue in production, the service can keep a small shared cache of compiled queries. Given the improvements from #7319 et seq., this should not be necessary.

Discussion point: For the initial implementation, the new API will use the existing query language as-is. Future work may extend the Filter message with a more structured and/or expressive query surface, but that is beyond the scope of this design.

The semantics of the request are as follows: An item in the event log is eligible for a query if:

  • It is newer than the after cursor (if set).
  • It is older than the before cursor (if set).
  • It matches the filter (if set).

Among the eligible items in the log, the server returns up to max_results of the newest items, in reverse order of cursor. If max_results is unset the server chooses a number to return, and will cap max_results at a sensible limit.

The wait_time parameter is used to effect polling. If before is empty and no items are available, the server will wait for up to wait_time for matching items to arrive at the head of the log. If wait_time is zero or negative, the server will wait for a default (positive) interval.

If before non-empty, wait_time is ignored: new results are only added to the head of the log, so there is no need to wait. This allows the client to poll for new data, and "page" backward through matching event items. This is discussed in more detail below.

The server will set a sensible cap on the maximum wait_time, overriding client-requested intervals longer than that.

A successful reply from the events request corresponds to the following Go types:

type EventReply struct {
    // The items matching the request parameters, from newest
    // to oldest, if any were available within the timeout.
    Items []*EventItem `json:"items"`

    // This is true if there is at least one older matching item
    // available in the log that was not returned.
    More bool `json:"more"`

    // The cursor of the oldest item in the log at the time of this reply,
    // or "" if the log is empty.
    Oldest string `json:"oldest"`

    // The cursor of the newest item in the log at the time of this reply,
    // or "" if the log is empty.
    Newest string `json:"newest"`
}

type EventItem struct {
    // The cursor of this item.
    Cursor string `json:"cursor"`

    // The encoded event data for this item.
    // The type identifies the structure of the value.
    Data struct {
        Type  string          `json:"type"`
        Value json.RawMessage `json:"value"`
    } `json:"data"`
}

The oldest and newest fields of the reply report the cursors of the oldest and newest items (of any kind) recorded in the event log at the time of the reply, or are "" if the log is empty.

The data field contains the type-specific event datum. The datum carries any ABCI events that may have been defined.

Discussion point: Based on issue #7273, I did not include a separate field in the response for the ABCI events, since it duplicates data already stored elsewhere in the event data.

The semantics of the reply are as follows:

  • If items is non-empty:

    • Items are ordered from newest to oldest.

    • If more is true, there is at least one additional, older item in the event log that was not returned (in excess of max_results).

      In this case the client can fetch the next page by setting before in a new request, to the cursor of the oldest item fetched (i.e., the last one in items).

    • Otherwise (if more is false), all the matching results have been reported (pagination is complete).

    • The first element of items identifies the newest item considered. Subsequent poll requests can set after to this cursor to skip items that were already retrieved.

  • If items is empty:

    • If the before was set in the request, there are no further eligible items for this query in the log (pagination is complete).

      This is just a safety case; the client can detect this without issuing another call by consulting the more field of the previous reply.

    • If the before was empty in the request, no eligible items were available before the wait_time expired. The client may poll again to wait for more event items.

A client can store cursor values to detect data loss and to recover from crashes and connectivity issues:

  • After a crash, the client requests events after the newest cursor it has seen. If the reply indicates that cursor is no longer in range, the client may (conservatively) conclude some event data may have been lost.

  • On the other hand, if it is in range, the client can then page back through the results that it missed, and then resume polling. As long as its recovery cursor does not age out before it finishes, the client can be sure it has all the relevant results.

Other Notes

  • The new API supports two general "modes" of operation:

    1. In ordinary operation, clients will long-poll the head of the event log for new events matching their criteria (by setting a wait_time and no before).

    2. If there are more events than the client requested, or if the client needs to to read older events to recover from a stall or crash, clients will page backward through the event log (by setting before and after).

  • While the new API requires explicit polling by the client, it makes better use of the node's existing HTTP infrastructure (e.g., connection pools). Moreover, the direct implementation is easier to use from standard tools and client libraries for HTTP and JSON-RPC.

    Explicit polling does shift the burden of timeliness to the client. That is arguably preferable, however, given that the RPC service is ancillary to the node's primary goal, viz., consensus. The details of polling can be easily hidden from client applications with simple libraries.

  • The format of a cursor is considered opaque to the client. Clients must not parse cursor values, but they may rely on their ordering properties.

  • To maintain the event log, the server must prune items outside the time window and in excess of the item limit.

    The initial implementation will do this by checking the tail of the event log after each new item is published. If the number of items in the log exceeds the item limit, it will delete oldest items until the log is under the limit; then discard any older than the time window before the latest.

    To minimize coordination interference between the publisher (the event bus) and the subcribers (the events service handlers), the event log will be stored as a persistent linear queue with shared structure (a cons list). A single reader-writer mutex will guard the "head" of the queue where new items are published:

    • To publish a new item, the publisher acquires the write lock, conses a new item to the front of the existing queue, and replaces the head pointer with the new item.

    • To scan the queue, a reader acquires the read lock, captures the head pointer, and then releases the lock. The rest of its request can be served without holding a lock, since the queue structure will not change.

      When a reader wants to wait, it will yield the lock and wait on a condition that is signaled when the publisher swings the pointer.

    • To prune the queue, the publisher (who is the sole writer) will track the queue length and the age of the oldest item separately. When the length and or age exceed the configured bounds, it will construct a new queue spine on the same items, discarding out-of-band values.

      Pruning can be done while the publisher already holds the write lock, or could be done outside the lock entirely: Once the new queue is constructed, the lock can be re-acquired to swing the pointer. This costs some extra allocations for the cons cells, but avoids duplicating any event items. The pruning step is a simple linear scan down the first (up to) max-items elements of the queue, to find the breakpoint of age and length.

      Moreover, the publisher can amortize the cost of pruning by item count, if necessary, by pruning length "more aggressively" than the configuration requires (e.g., reducing to 3/4 of the maximum rather than 1/1).

      Before publication, readers see the existing queue head. After the publisher adds an item and prunes the queue, new readers see the new head; readers already scanning the old queue continue using their captured head.

Migration Plan

This design requires that clients eventually migrate to the new event subscription API, but provides a full release cycle with both APIs in place to make this burden more tractable. The migration strategy is broadly:

Phase 1: Release v0.36.

  • Implement the new events endpoint, keeping the existing methods as they are.
  • Update the Go clients to support the new events endpoint, and handle polling.
  • Update the old endpoints to log annoyingly about their own deprecation.
  • Write tutorials about how to migrate client usage.

At or shortly after release, we should proactively update the Cosmos SDK to use the new API, to remove a disincentive to upgrading.

Phase 2: Release v0.37

  • During development, we should actively seek out any existing users of the streaming event subscription API and help them migrate.
  • Possibly also: Spend some time writing clients for JS, Rust, et al.
  • Release: Delete the old implementation and all the websocket support code.

Discussion point: Even though the plan is to keep the existing service, we might take the opportunity to restrict the websocket endpoint to only the event streaming service, removing the other endpoints. To minimize the disruption for users in the v0.36 cycle, I have decided not to do this for the first phase.

If we wind up pushing this design into v0.37, however, we should re-evaulate this partial turn-down of the websocket.

Future Work

  • This design does not immediately address the problem of allowing the client to control which data are reported back for event items. That concern is deferred to future work. However, it would be straightforward to extend the filter and/or the request parameters to allow more control.

  • The node currently stores a subset of event data (specifically the block and transaction events) for use in reindexing. While these data are redundant with the event log described in this document, they are not sufficient to cover event subscription, as they omit other event types.

    In the future we should investigate consolidating or removing event data from the state store entirely. For now this issue is out of scope for purposes of updating the RPC API. We may be able to piggyback on the database unification plans (see RFC 001) to store the event log separately, so its pruning policy does not need to be tied to the block and state stores.

  • This design reuses the existing filter query language from the old API. In the future we may want to use a more structured and/or expressive query. The Filter object can be extended with more fields as needed to support this.

  • Some users have trouble communicating with the RPC service because of configuration problems like improperly-set CORS policies. While this design does not address those issues directly, we might want to revisit how we set policies in the RPC service to make it less susceptible to confusing errors caused by misconfiguration.


Consequences

  • ✅ Reduces the number of transport options for RPC. Supports RFC 002.
  • ️✅ Removes the primary non-standard use of JSON-RPC.
  • ⛔️ Forces clients to migrate to a different API (eventually).
  • ↕️ API requires clients to poll, but this reduces client state on the server.
  • ↕️ We have to maintain both implementations for a whole release, but this gives clients time to migrate.

Alternative Approaches

The following alternative approaches were considered:

  1. Leave it alone. Since existing tools mostly already work with the API as it stands today, we could leave it alone and do our best to improve its performance and reliability.

    Based on many issues reported by users and node operators (e.g., #3380, #6439, #6729, #7247), the problems described here affect even the existing use that works. Investing further incremental effort in the existing API is unlikely to address these issues.

  2. Design a better streaming API. Instead of polling, we might try to design a better "streaming" API for event subscription.

    A significant advantage of switching away from streaming is to remove the need for persistent connections between the node and subscribers. A new streaming protocol design would lose that advantage, and would still need a way to let clients recover and replay.

    This approach might look better if we decided to use a different protocol for event subscription, say gRPC instead of JSON-RPC. That choice, however, would be just as breaking for existing clients, for marginal benefit. Moreover, this option increases both the complexity and the resource cost on the node implementation.

    Given that resource consumption and complexity are important considerations, this option was not chosen.

  3. Defer to an external event broker. We might remove the entire event subscription infrastructure from the node, and define an optional interface to allow the node to publish all its events to an external event broker, such as Apache Kafka.

    This has the advantage of greatly simplifying the node, but at a great cost to the node operator: To enable event subscription in this design, the operator has to stand up and maintain a separate process in communion with the node, and configuration changes would have to be coordinated across both.

    Moreover, this approach would be highly disruptive to existing client use, and migration would probably require switching to third-party libraries. Despite the potential benefits for the node itself, the costs to operators and clients seems too large for this to be the best option.

    Publishing to an external event broker might be a worthwhile future project, if there is any demand for it. That decision is out of scope for this design, as it interacts with the design of the indexer as well.


References

ADR 076: Combine Spec and Tendermint Repositories

Historical upstream design record. Project and release names below identify the original implementation; they do not describe Tenderdash releases. See the current protocol specification for Tenderdash.

Changelog

  • 2022-02-04: Initial Draft. (@tychoish)

Status

Accepted.

Context

While the specification for Tendermint was originally in the same repository as the Go implementation, at some point the specification was split from the core repository and maintained separately from the implementation. While this makes sense in promoting a conceptual separation of specification and implementation, in practice this separation was a premature optimization, apparently aimed at supporting alternate implementations of Tendermint.

The operational and documentary burden of maintaining a separate spec repo has not returned value to justify its cost. There are no active projects to develop alternate implementations of Tendermint based on the common specification, and having separate repositories creates an ongoing burden to coordinate versions, documentation, and releases.

Decision

The specification repository will be merged back into the Tendermint core repository.

Stakeholders including representatives from the maintainers of the spec, the Go implementation, and the Tendermint Rust library, agreed to merge the repositories in the Tendermint core dev meeting on 27 January 2022, including @williambanfield @cmwaters @creachadair and @thanethomson.

Alternative Approaches

The main alternative we considered was to keep separate repositories, and to introduce a coordinated versioning scheme between the two, so that users could figure out which spec versions go with which versions of the core implementation.

We decided against this on the grounds that it would further complicate the release process for both repositories, without mitigating any of the other existing issues.

Detailed Design

Clone and merge the master branch of the tendermint/spec repository as a branch of the tendermint/tendermint, to ensure the commit history of both repositories remains intact.

Implementation Instructions

  1. Within the tendermint repository, execute the following commands to add a new branch with the history of the master branch of spec:

    git remote add spec git@github.com:tendermint/spec.git
    git fetch spec
    git checkout -b spec-master spec/master
    mkdir spec
     git ls-tree -z --name-only HEAD | xargs -0 -I {} git mv {} subdir/
     git commit -m "spec: organize specification prior to merge"
     git checkout -b spec-merge-mainline origin/master
     git merge --allow-unrelated-histories spec-master
    

    This merges the spec into the tendermint/tendermint repository as a normal branch. This commit can also be backported to the 0.35 branch, if needed.

  2. Migrate outstanding issues from tendermint/spec to the tendermint/tendermint repository.

  3. In the specification repository, add redirect to the README and mark the repository as archived.

Consequences

Positive

Easier maintenance for the specification will obviate a number of complicated and annoying versioning problems, and will help prevent the possibility of the specification and the implementation drifting apart.

Additionally, co-locating the specification will help encourage cross-pollination and collaboration, between engineers focusing on the specification and the protocol and engineers focusing on the implementation.

Negative

Co-locating the spec and Go implementation has the potential effect of prioritizing the Go implementation with regards to the spec, and making it difficult to think about alternate implementations of the Tendermint algorithm. Although we may want to foster additional Tendermint implementations in the future, this isn't an active goal in our current roadmap, and not merging these repos doesn't change the fact that the Go implementation of Tendermint is already the primary implementation.

Neutral

N/A

References

  • https://github.com/tendermint/spec
  • https://github.com/tendermint/tendermint

ADR 077: Configurable Block Retention

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 2020-03-23: Initial draft (@erikgrinaker)
  • 2020-03-25: Use local config for snapshot interval (@erikgrinaker)
  • 2020-03-31: Use ABCI commit response for block retention hint
  • 2020-04-02: Resolved open questions
  • 2021-02-11: Migrate to tendermint repo (Originally RFC 001)

Author(s)

  • Erik Grinaker (@erikgrinaker)

Context

Currently, all Tenderdash nodes contain the complete sequence of blocks from genesis up to some height (typically the latest chain height). This will no longer be true when the following features are released:

  • Block pruning: removes historical blocks and associated data (e.g. validator sets) up to some height, keeping only the most recent blocks.

  • State sync: bootstraps a new node by syncing state machine snapshots at a given height, but not historical blocks and associated data.

To maintain the integrity of the chain, the use of these features must be coordinated such that necessary historical blocks will not become unavailable or lost forever. In particular:

  • Some nodes should have complete block histories, for auditability, querying, and bootstrapping.

  • The majority of nodes should retain blocks longer than the Cosmos SDK unbonding period, for light client verification.

  • Some nodes must take and serve state sync snapshots with snapshot intervals less than the block retention periods, to allow new nodes to state sync and then replay blocks to catch up.

  • Applications may not persist their state on commit, and require block replay on restart.

  • Only a minority of nodes can be state synced within the unbonding period, for light client verification and to serve block histories for catch-up.

However, it is unclear if and how we should enforce this. It may not be possible to technically enforce all of these without knowing the state of the entire network, but it may also be unrealistic to expect this to be enforced entirely through social coordination. This is especially unfortunate since the consequences of misconfiguration can be permanent chain-wide data loss.

Proposal

Add a new field retain_height to the ABCI ResponseCommit message:

service ABCIApplication {
  rpc Commit(RequestCommit) returns (ResponseCommit);
}

message RequestCommit {}

message ResponseCommit {
  // reserve 1
  bytes  data          = 2; // the Merkle root hash
  uint64 retain_height = 3; // the oldest block height to retain
}

Upon ABCI Commit, which finalizes execution of a block in the state machine, Tenderdash removes all data for heights lower than retain_height. This allows the state machine to control block retention, which is preferable since only it can determine the significance of historical blocks. By default (i.e. with retain_height=0) all historical blocks are retained.

Removed data includes not only blocks, but also headers, commit info, consensus params, validator sets, and so on. In the first iteration this will be done synchronously, since the number of heights removed for each run is assumed to be small (often 1) in the typical case. It can be made asynchronous at a later time if this is shown to be necessary.

Since retain_height is dynamic, it is possible for it to refer to a height which has already been removed. For example, commit at height 100 may return retain_height=90 while commit at height 101 may return retain_height=80. This is allowed, and will be ignored - it is the application's responsibility to return appropriate values.

State sync will eventually support backfilling heights, via e.g. a snapshot metadata field backfill_height, but in the initial version it will have a fully truncated block history.

Cosmos SDK Example

As an example, we'll consider how the Cosmos SDK might make use of this. The specific details should be discussed in a separate SDK proposal.

The returned retain_height would be the lowest height that satisfies:

  • Unbonding time: the time interval in which validators can be economically punished for misbehavior. Blocks in this interval must be auditable e.g. by the light client.

  • IAVL snapshot interval: the block interval at which the underlying IAVL database is persisted to disk, e.g. every 10000 heights. Blocks since the last IAVL snapshot must be available for replay on application restart.

  • State sync snapshots: blocks since the oldest available snapshot must be available for state sync nodes to catch up (oldest because a node may be restoring an old snapshot while a new snapshot was taken).

  • Local config: archive nodes may want to retain more or all blocks, e.g. via a local config option min-retain-blocks. There may also be a need to vary rentention for other nodes, e.g. sentry nodes which do not need historical blocks.

The retention height is the earliest height required by any of these constraints.

Status

Accepted

Consequences

Positive

  • Application-specified block retention allows the application to take all relevant factors into account and prevent necessary blocks from being accidentally removed.

  • Node operators can independently decide whether they want to provide complete block histories (if local configuration for this is provided) and snapshots.

Negative

  • Social coordination is required to run archival nodes, failure to do so may lead to permanent loss of historical blocks.

  • Social coordination is required to run snapshot nodes, failure to do so may lead to inability to run state sync, and inability to bootstrap new nodes at all if no archival nodes are online.

Neutral

  • Reduced block retention requires application changes, and cannot be controlled directly in Tenderdash.

  • Application-specified block retention may set a lower bound on disk space requirements for all nodes.

References

ADR 078: Non-Zero Genesis

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 2020-07-26: Initial draft (@erikgrinaker)
  • 2020-07-28: Use weak chain linking, i.e. predecessor field (@erikgrinaker)
  • 2020-07-31: Drop chain linking (@erikgrinaker)
  • 2020-08-03: Add State.InitialHeight (@erikgrinaker)
  • 2021-02-11: Migrate to tendermint repo (Originally RFC 002)

Author(s)

  • Erik Grinaker (@erikgrinaker)

Context

The recommended upgrade path for block protocol-breaking upgrades is currently to hard fork the chain (see e.g. cosmoshub-3 upgrade. This is done by halting all validators at a predetermined height, exporting the application state via application-specific tooling, and creating an entirely new chain using the exported application state.

As far as Tenderdash is concerned, the upgraded chain is a completely separate chain, with e.g. a new chain ID and genesis file. Notably, the new chain starts at height 1, and has none of the old chain's block history. This causes problems for integrators, e.g. coin exchanges and wallets, that assume a monotonically increasing height for a given blockchain. Users also find it confusing that a given height can now refer to distinct states depending on the chain version.

An ideal solution would be to always retain block backwards compatibility in such a way that chain history is never lost on upgrades. However, this may require a significant amount of engineering work that is not viable for the planned Stargate release (Tendermint 0.34), and may prove too restrictive for future development.

As a first step, allowing the new chain to start from an initial height specified in the genesis file would at least provide monotonically increasing heights. There was a proposal to include the last block header of the previous chain as well, but since the genesis file is not verified and hashed (only specific fields are) this would not be trustworthy.

External tooling will be required to map historical heights onto e.g. archive nodes that contain blocks from previous chain version. Tenderdash will not include any such functionality.

Proposal

Tenderdash will allow chains to start from an arbitrary initial height:

  • A new field initial_height is added to the genesis file, defaulting to 1. It can be set to any non-negative integer, and 0 is considered equivalent to 1.

  • A new field InitialHeight is added to the ABCI RequestInitChain message, with the same value and semantics as the genesis field.

  • A new field InitialHeight is added to the state.State struct, where 0 is considered invalid. Including the field here simplifies implementation, since the genesis value does not have to be propagated throughout the code base separately, but it is not strictly necessary.

ABCI applications may have to be updated to handle arbitrary initial heights, otherwise the initial block may fail.

Status

Accepted

Consequences

Positive

  • Heights can be unique throughout the history of a "logical" chain, across hard fork upgrades.

Negative

  • Upgrades still cause loss of block history.

  • Integrators will have to map height ranges to specific archive nodes/networks to query history.

Neutral

  • There is no explicit link to the last block of the previous chain.

References

ADR 079: Ed25519 Verification

Historical upstream design record. Project and release names below identify the original implementation; they do not describe Tenderdash releases. See the current protocol specification for Tenderdash.

Changelog

  • 2020-08-21: Initial RFC
  • 2021-02-11: Migrate RFC to tendermint repo (Originally RFC 003)

Author(s)

  • Marko (@marbar3778)

Context

Ed25519 keys are the only supported key types for Tendermint validators currently. Tendermint-Go wraps the ed25519 key implementation from the go standard library. As more clients are implemented to communicate with the canonical Tendermint implementation (Tendermint-Go) different implementations of ed25519 will be used. Due to RFC 8032 not guaranteeing implementation compatibility, Tendermint clients must to come to an agreement of how to guarantee implementation compatibility. Zcash has multiple implementations of their client and have identified this as a problem as well. The team at Zcash has made a proposal to address this issue, Zcash improvement proposal 215.

Proposal

  • Tendermint-Go would adopt hdevalence/ed25519consensus.

    • This library is implements ed25519.Verify() in accordance to zip-215. Tendermint-go will continue to use crypto/ed25519 for signing and key generation.
  • Tendermint-rs would adopt ed25519-zebra

Signature verification is one of the major bottlenecks of Tendermint-go, batch verification can not be used unless it has the same consensus rules, ZIP 215 makes verification safe in consensus critical areas.

This change constitutes a breaking changes, therefore must be done in a major release. No changes to validator keys or operations will be needed for this change to be enabled.

This change has no impact on signature aggregation. To enable this signature aggregation Tendermint will have to use different signature schema (Schnorr, BLS, ...). Secondly, this change will enable safe batch verification for the Tendermint-Go client. Batch verification for the rust client is already supported in the library being used.

As part of the acceptance of this proposal it would be best to contract or discuss with a third party the process of conducting a security review of the go library.

Status

Proposed

Consequences

Positive

  • Consistent signature verification across implementations
  • Enable safe batch verification

Negative

Tendermint-Go

  • Third_party dependency
    • library has not gone through a security review.
    • unclear maintenance schedule
  • Fragmentation of the ed25519 key for the go implementation, verification is done using a third party library while the rest uses the go standard library

Neutral

References

It’s 255:19AM. Do you know what your validation criteria are?

ADR 080: ReverseSync - fetching historical data

Changelog

  • 2021-02-11: Migrate to tendermint repo (Originally RFC 005)
  • 2021-04-19: Use P2P to gossip necessary data for reverse sync.
  • 2021-03-03: Simplify proposal to the state sync case.
  • 2021-02-17: Add notes on asynchronicity of processes.
  • 2020-12-10: Rename backfill blocks to reverse sync.
  • 2020-11-25: Initial draft.

Author(s)

  • Callum Waters (@cmwaters)

Context

Two new features: Block pruning and State sync meant nodes no longer needed a complete history of the blockchain. This introduced some challenges of its own which were covered and subsequently tackled with RFC-001. The RFC allowed applications to set a block retention height; an upper bound on what blocks would be pruned. However nodes who state sync past this upper bound (which is necessary as snapshots must be saved within the trusting period for the assisting light client to verify) have no means of backfilling the blocks to meet the retention limit. This could be a problem as nodes who state sync and then eventually switch to consensus (or fast sync) may not have the block and validator history to verify evidence causing them to panic if they see 2/3 commit on what the node believes to be an invalid block.

Thus, this RFC sets out to instil a minimum block history invariant amongst honest nodes.

Proposal

A backfill mechanism can simply be defined as an algorithm for fetching, verifying and storing, headers and validator sets of a height prior to the current base of the node's blockchain. In matching the terminology used for other data retrieving protocols (i.e. fast sync and state sync), we call this method ReverseSync.

We will define the mechanism in four sections:

  • Usage
  • Design
  • Verification
  • Termination

Usage

For now, we focus purely on the case of a state syncing node, whom after syncing to a height will need to verify historical data in order to be capable of processing new blocks. We can denote the earliest height that the node will need to verify and store in order to be able to verify any evidence that might arise as the max_historical_height/time. Both height and time are necessary as this maps to the BFT time used for evidence expiration. After acquiring State, we calculate these parameters as:

max_historical_height = max(state.InitialHeight, state.LastBlockHeight - state.ConsensusParams.EvidenceAgeHeight)
max_historical_time = max(GenesisTime, state.LastBlockTime.Sub(state.ConsensusParams.EvidenceAgeTime))

Before starting either fast sync or consensus, we then run the following synchronous process:

func ReverseSync(max_historical_height int64, max_historical_time time.Time) error

Where we fetch and verify blocks until a block A where A.Height <= max_historical_height and A.Time <= max_historical_time.

Upon successfully reverse syncing, a node can now safely continue. As this feature is only used as part of state sync, one can think of this as merely an extension to it.

In the future we may want to extend this functionality to allow nodes to fetch historical blocks for reasons of accountability or data accessibility.

Design

This section will provide a high level overview of some of the more important characteristics of the design, saving the more tedious details as an ADR.

P2P

Implementation of this RFC will require the addition of a new channel and two new messages.

message LightBlockRequest {
  uint64 height = 1;
}
message LightBlockResponse {
  Header header = 1;
  Commit commit = 2;
  ValidatorSet validator_set = 3;
}

The P2P path may also enable P2P networked light clients and a state sync that also doesn't need to rely on RPC.

Verification

ReverseSync is used to fetch the following data structures:

  • Header
  • Commit
  • ValidatorSet

Nodes will also need to be able to verify these. This can be achieved by first retrieving the header at the base height from the block store. From this trusted header, the node hashes each of the three data structures and checks that they are correct.

  1. The trusted header's last block ID matches the hash of the new header

    header[height].LastBlockID == hash(header[height-1])
    
  2. The trusted header's last commit hash matches the hash of the new commit

    header[height].LastCommitHash == hash(commit[height-1])
    
  3. Given that the node now trusts the new header, check that the header's validator set hash matches the hash of the validator set

    header[height-1].ValidatorsHash == hash(validatorSet[height-1])
    

Termination

ReverseSync draws a lot of parallels with fast sync. An important consideration for fast sync that also extends to ReverseSync is termination. ReverseSync will finish it's task when one of the following conditions have been met:

  1. It reaches a block A where A.Height <= max_historical_height and A.Time <= max_historical_time.
  2. None of it's peers reports to have the block at the height below the processes current block.
  3. A global timeout.

This implies that we can't guarantee adequate history and thus the term "invariant" can't be used in the strictest sense. In the case that the first condition isn't met, the node will log an error and optimistically attempt to continue with either fast sync or consensus.

Alternative Solutions

The need for a minimum block history invariant stems purely from the need to validate evidence (although there may be some application relevant needs as well). Because of this, an alternative, could be to simply trust whatever the 2/3+ majority has agreed upon and in the case where a node is at the head of the blockchain, you simply abstain from voting.

As it stands, if 2/3+ vote on evidence you can't verify, in the same manner if 2/3+ vote on a header that a node sees as invalid (perhaps due to a different app hash), the node will halt.

Another alternative is the method with which the relevant data is retrieved. Instead of introducing new messages to the P2P layer, RPC could have been used instead.

The aforementioned data is already available via the following RPC endpoints: /commit for Header's' and /validators for ValidatorSet's'. It was decided predominantly due to the instability of the current RPC infrastructure that P2P be used instead.

Status

Proposed

Consequences

Positive

  • Ensures a minimum block history invariant for honest nodes. This will allow nodes to verify evidence.

Negative

  • Statesync will be slower as more processing is required.

Neutral

  • By having validator sets served through p2p, this would make it easier to extend p2p support to light clients and state sync.
  • In the future, it may also be possible to extend this feature to allow for nodes to freely fetch and verify prior blocks

References

ADR 081: Protocol Buffers Management

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 2022-02-28: First draft

Status

Accepted

Tracking issue

Context

At present, we manage the Protocol Buffers schema files ("protos") that define our wire-level data formats within the Tenderdash repository itself (see the proto directory). Recently, we have been making use of Buf, both locally and in CI, in order to generate Go stubs, and lint and check .proto files for breaking changes.

The version of Buf used at the time of this decision was v1beta1, and it was discussed in #7975 and in weekly calls as to whether we should upgrade to v1 and harmonize our approach with that used by the Cosmos SDK. The team managing the Cosmos SDK was primarily interested in having our protos versioned and easily accessible from the Buf registry.

The three main sets of stakeholders for the .proto files and their needs, as currently understood, are as follows.

  1. Tenderdash needs Go code generated from .proto files.
  2. Consumers of Tenderdash's .proto files, specifically projects that want to interoperate with Tenderdash and need to generate code for their own programming language, want to be able to access these files in a reliable and efficient way.
  3. The Tenderdash Core team wants to provide stable interfaces that are as easy as possible to maintain, on which consumers can depend, and to be able to notify those consumers promptly when those interfaces change. To this end, we want to:
    1. Prevent any breaking changes from being introduced in minor/patch releases of Tenderdash. Only major version updates should be able to contain breaking interface changes.
    2. Prevent generated code from diverging from the Protobuf schema files.

There was also discussion surrounding the notion of automated documentation generation and hosting, but it is not clear at this time whether this would be that valuable to any of our stakeholders. What will, of course, be valuable at minimum would be better documentation (in comments) of the .proto files themselves.

Alternative Approaches

Meeting stakeholders' needs

  1. Go stub generation from protos. We could use:
    1. Buf. This approach has been rather cumbersome up to this point, and it is not clear what Buf really provides beyond that which protoc provides to justify the additional complexity in configuring Buf for stub generation.
    2. protoc - the Protocol Buffers compiler.
  2. Notification of breaking changes:
    1. Buf in CI for all pull requests to release branches only (and not on master).
    2. Buf in CI on every pull request to every branch (this was the case at the time of this decision, and the team decided that the signal-to-noise ratio for this approach was too low to be of value).
  3. .proto linting:
    1. Buf in CI on every pull request
  4. .proto formatting:
    1. clang-format locally and a clang-format GitHub Action in CI to check that files are formatted properly on every pull request.
  5. Sharing of .proto files in a versioned, reliable manner:
    1. Consumers could simply clone the Tenderdash repository, check out a specific commit, tag or branch and manually copy out all of the .proto files they need. This requires no effort from the Tenderdash Core team and will continue to be an option for consumers. The drawback of this approach is that it requires manual coding/scripting to implement and is brittle in the face of bigger changes.
    2. Uploading our .proto files to Buf's registry on every release. This is by far the most seamless for consumers of our .proto files, but requires the dependency on Buf. This has the additional benefit that the Buf registry will automatically generate and host documentation for these protos.
    3. We could create a process that, upon release, creates a .zip file containing our .proto files.

Prototool was not considered as it appears deprecated, and the ecosystem seems to be converging on Buf at this time.

Tooling complexity

The more tools we have in our build/CI processes, the more complex and fragile repository/CI management becomes, and the longer it takes to onboard new team members. Maintainability is a core concern here.

Buf sustainability and costs

One of the primary considerations regarding the usage of Buf is whether, for example, access to its registry will eventually become a paid-for/subscription-based service and whether this is valuable enough for us and the ecosystem to pay for such a service. At this time, it appears as though Buf will never charge for hosting open source projects' protos.

Another consideration was Buf's sustainability as a project - what happens when their resources run out? Will there be a strong and broad enough open source community to continue maintaining it?

Local Buf usage options

Local usage of Buf (i.e. not in CI) can be accomplished in two ways:

  1. Installing the relevant tools individually.
  2. By way of its Docker image.

Local installation of Buf requires developers to manually keep their toolchains up-to-date. The Docker option comes with a number of complexities, including how the file system permissions of code generated by a Docker container differ between platforms (e.g. on Linux, Buf-generated code ends up being owned by root).

The trouble with the Docker-based approach is that we make use of the gogoprotobuf plugin for protoc. Continuing to use the Docker-based approach to using Buf will mean that we will have to continue building our own custom Docker image with embedded gogoprotobuf.

Along these lines, we could eventually consider coming up with a Nix- or redo-based approach to developer tooling to ensure tooling consistency across the team and for anyone who wants to be able to contribute to Tenderdash.

Decision

  1. We will adopt Buf for now for proto generation, linting, breakage checking and its registry (mainly in CI, with optional usage locally).
  2. Failing CI when checking for breaking changes in .proto files will only happen when performing minor/patch releases.
  3. Local tooling will be favored over Docker-based tooling.

Detailed Design

We currently aim to:

  1. Update to Buf v1 to facilitate linting, breakage checking and uploading to the Buf registry.
  2. Configure CI appropriately for proto management:
    1. Uploading protos to the Buf registry on every release (e.g. the approach used by the Cosmos SDK).
    2. Linting on every pull request (e.g. the approach used by the Cosmos SDK). The linter passing should be considered a requirement for accepting PRs.
    3. Checking for breaking changes in minor/patch version releases and failing CI accordingly - see #8003.
    4. Add clang-format GitHub Action to check .proto file formatting. Format checking should be considered a requirement for accepting PRs.
  3. Update the Tenderdash Makefile to primarily facilitate local Protobuf stub generation, linting, formatting and breaking change checking. More specifically:
    1. This includes removing the dependency on Docker and introducing the dependency on local toolchain installation. CI-based equivalents, where relevant, will rely on specific GitHub Actions instead of the Makefile.
    2. Go code generation will rely on protoc directly.

Consequences

Positive

  • We will still offer Go stub generation, proto linting and breakage checking.
  • Breakage checking will only happen on minor/patch releases to increase the signal-to-noise ratio in CI.
  • Versioned protos will be made available via Buf's registry upon every release.

Negative

  • Developers/contributors will need to install the relevant Protocol Buffers-related tooling (Buf, gogoprotobuf, clang-format) locally in order to build, lint, format and check .proto files for breaking changes.

Neutral

References

ADR D001: Inter validator set messaging

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 2021-09-29: Initial version of the document
  • 2021-10-05: Clarification of the document based on peer review
  • 2021-10-22: Updated document based on implementation
  • 2022-01-05: Review after implementation is done

Table of contents

  1. Context
    1. Definitions
    2. Problem statement
  2. Alternative Approaches
    1. Scenario 1: Trigger rotation from ABCI app passing addresses of all quorum members
    2. Scenario 2: Trigger rotation from ABCI app passing quorum id to rotate
    3. Scenario 3: Rotate on Tenderdash
  3. Decision
  4. Detailed Design
    1. What are the user requirements?
    2. What systems will be affected?
    3. What new data structures are needed, what data structures need changes?
    4. What new APIs will be needed, what APIs will change?
      1. ABCI Protocol
      2. P2P Handshake
    5. What are the efficiency considerations (time/space)?
    6. What are the expected access patterns (load/throughput)?
    7. Are there any logging, monitoring, or observability needs?
    8. Are there any security considerations?
    9. Are there any privacy considerations?
    10. How will the changes be tested?
    11. How will the changes be broken up for ease of review?
    12. Will these changes require a breaking (major) release?
    13. Does this change require coordination with the SDK or any other software?
    14. Future work
  5. Status
  6. Consequences
    1. Positive
    2. Negative
    3. Neutral
  7. References

Context

Definitions

For the needs of this document, we define the following types of nodes:

  • Validator nodes, which participate in the consensus protocol, verify and sign blocks,
  • Full nodes, which verify and execute blocks. Full nodes verify the recovered threshold signature of the block commit message against the current validator set's quorum threshold public key.

Note: there are also light nodes, but they are not relevant for this discussion.

A Validator Set is a group of Validator nodes responsible for driving consensus at any given time. For each block, one of the validator nodes is a proposer. Each other validator votes on the validity of the block.

Only one Validator Set can be active at the given time. After a predefined number of blocks, the ABCI app initiates the Validator Set rotation to make another Validator Set active.

Problem statement

The consensus process requires direct communication between Validators, as Full nodes do not know the public keys of Validators and therefore can neither verify nor propagate any consensus protocol message individually signed by these Validators. It means that only Validators can propagate votes. However, Full nodes can propagate the final commit message, which instead contains the recovered threshold signature of the pre-commit voting round. Full nodes can perform this verification because they know the Validator Set's quorum threshold public key.

Each node (regardless of its type) randomly selects peer nodes to connect. There is no differentiation between Full and Validator nodes, and there is no method to ensure that each Validator is (directly or through other Validators) connected to some other Validators. In an extreme case, a Validator node can be connected only to Full nodes, effectively blocking all inbound and outbound block proposals and voting, hence excluding it from participation in the consensus. It is also possible that Validator nodes form "islands" where a few Validators are connected between themselves but then are separated from other Validator nodes by full nodes.

Alternative Approaches

Scenario 1: Trigger rotation from ABCI app passing addresses of all quorum members

During Validator Set rotation, each Validator node shall connect to a predefined number of other Validators that belong to the same Validator Set.

In the current implementation, the ABCI application manages the rotation of Validator Sets. When rotation is needed, the application sends information about the new Validator Set in response to the EndBlock message.

EndBlock message shall be extended by adding a network address of each Validator node. Based on that, each member of the active Validator Set will connect with a predefined set of Peer Validators belonging to the same Validator Set. A list of Peer Validators will be determined using the algorithm described in DIP 0006.

This solution introduces a risk of delayed propagation of new block due to limited connection slots between Full and Validator nodes. To mitigate this risk, we can implement the number of Full node connection slots independently from Validator node connection slots.

Scenario 2: Trigger rotation from ABCI app passing quorum id to rotate

This scenario is very similar to Scenario 1. However, instead of passing addresses of all Validators that are members of a Validator Set, only quorum ID is passed.

This solution requires Tenderdash to determine Validators addresses, effectively re-implementing logic already done in the ABCI App.

Scenario 3: Rotate on Tenderdash

ABCI application can send the whole master node list and quorum list to Tenderdash and potentially rotate on its side. However, it would split the responsibility for Validator Set rotation between multiple components. It would also require much more additional implementation. It would be better to keep the current Tenderdash design and concept being a logic-less consensus library and keep all business logic in the ABCI application.

Decision

After discussion, we will implement the solution described in Scenario 1. It will allow us to:

  • reuse logic already implemented as part of ABCI App,
  • keep Tenderdash a logic-less consensus library,
  • keep single responsibility principle.

Detailed Design

What are the user requirements?

Having a network with much more Full nodes than Validators ensure each Validator Node can connect (directly or through another Validator) to any other Validator which is a member of the same active Validator Set.

What systems will be affected?

  • Tenderdash
  • ABCI App (Drive)

What new data structures are needed, what data structures need changes?

  1. New package dash to group Tenderdash-specific code
  2. dashtypes.ValidatorAddress representing an address of a validator received from the ABCI app
  3. Introduced several internal helper data structures (sortableValidator, sortedValidatorList, validatorMap)

What new APIs will be needed, what APIs will change?

ABCI Protocol

In the ABCI protocol, we add a network address of each member of the active Validator Set to the ResponseEndBlock.ValidatorSetUpdate.ValidatorUpdate structure. The network address should be a URI:

tcp://<node_id>@<address>:<port>

where:

  • <node_id> - node ID (can be generated based on node P2P public key, see p2p.PubKeyToID)
  • <address> - IP address of the validator node
  • <port> - p2p port number of the validator node

The ValidatorUpdate structure looks as follows:

message ValidatorUpdate {
  tendermint.crypto.PublicKey pub_key      = 1 [(gogoproto.nullable) = true];
  int64                       power        = 2;
  bytes                       pro_tx_hash  = 3;
  string                      node_address = 4 [(gogoproto.nullable) = true];  // address of the Validator
}

If the ABCI app cannot determine <node_id>, it can leave it empty. In this case, Tenderdash will try to retrieve a public key from its address book or remote host and derive node ID from it, as described below.

P2P Handshake

To retrieve node ID from a remote node, local node can establish a short-lived connection to the remote node to download its public key. This feature does not require any changes to the P2P protocol; it just uses existing capabilities.

A public key request starts with a standard flow Diffie-Hellman protocol, as implemented in conn.MakeSecretConnection(). Once ephemeral keys are exchanged between nodes, local node proceeds to shareAuthSignature(). However, instead of sending its own AuthSigMessage, it waits for remote node's auth signature to arrive. As the communication is asynchronous, the remote node sends its AuthSigMessage. That is when the local node closes the connection.

This algorithm introduces a potential node ID spoofing risk, discussed later.

What are the efficiency considerations (time/space)?

  1. This change will increase the amount of data sent by the ABCI application to Tenderdash. The connection between them is local. Therefore the change should not have any noticeable performance or bandwidth impact.
  2. Improved consensus performance thanks to a shorter and more predictable distance between each Validator in a Validator Set.

In the future, additional optimizations can further improve the performance, for example:

  1. use Unix sockets for communication between Tenderdash and ABCI App,
  2. use shared memory to speed up inter-process communication; for instance, ABCI should keep (signed) information about active validator-set in shared memory and send only a trigger to take them from there.

What are the expected access patterns (load/throughput)?

  1. Each rotation of the active Validator Set will cause the ABCI application to send additional data to Tenderdash. The number of blocks between rotation events is a configuration parameter of the ABCI application.

  2. Each Validator in the active validator set will establish a few more tcp connections.

  3. Retrieving public keys for validators can be resource and time-intensive, as it establishes new connections to validators and performs a DH handshake. This issue is mitigated by: a. ensuring that this feature is used only when needed b. refactoring and simplification of the P2P protocol and source code used to retrieve public keys

    Note that for the needs of Inter-Validator Set Messaging, retrieving public keys is executed only for log2(n-1) Validators, where n is size of active Validator Set.

Are there any logging, monitoring, or observability needs?

  1. Operator can determine a list of node peers, together with their type (Full, inactive Validator, active Validator) and connection status. This information can be part of debug logs.

Are there any security considerations?

  1. This change makes connectivity between Validators more predictable. It can make it a bit easier for a malicious user to block communication for a given Validator node to make it unable to participate in the Validator Set. However, this risk is already present at the ABCI application level, so the change does not introduce new risks.

  2. Missing node ID can be determined based on a public key, retrieved without proper validation. It cannot be guaranteed that it will not be spoofed.

    That can allow an attacker who controls the network connectivity to perform a man-in-the-middle attack. As the attacker does not have the Validator's private keys, he/she cannot perform unauthorized operations. However, it is possible to intercept communication and isolate the attacked Validator from the quorum. In the future, additional validation of the Validator's private key based on proTxHash can help mitigate this risk.

    If the attacker controls the network connectivity, he can still drop that traffic on the network level.

Are there any privacy considerations?

This change does not impact user privacy.

How will the changes be tested?

Testing strategy for this change involves:

  1. Reproduction of the problem statement.
  2. Implementation of unit tests for the new code
  3. Starting a long-living end-to-end cluster (at least a few days of operation) and monitoring of the outcome
  4. Deployment to the test net

How will the changes be broken up for ease of review?

This change consists of the following tasks:

  1. Enhancement of the ABCI protocol:
    1. Addition of new field on Tenderdash
    2. Update of end-to-end tests to reflect ABCI protocol changes
    3. Implementation of ABCI protocol changes inside the ABCI application
  2. Tenderdash reactions on Validator Set rotation events:
    1. Storing and updating Validator Set members network addresses
    2. Establishing connections within Validator Sets
    3. Unit tests that emulate and validate Inter Validator Set implementation

Will these changes require a breaking (major) release?

As the change introduces a new required field into the ABCI app, it requires implementation on the ABCI App side. However, this feature shall be backward-compatible.

Does this change require coordination with the SDK or any other software?

The change requires coordination with the ABCI application.

Future work

  1. Implement additional validation of the Validator's private key based on proTxHash.
  2. Extend service data in Deterministic Masternode Lists (DIP-3) to keep node ID, p2p port, and other platform information. It will allow us to remove the handshake process described in "Security considerations" above.

Status

{Deprecated|Proposed|Accepted|Declined}

Accepted

Consequences

Positive

  1. Improved stability of active Validator Set operations.
  2. Quicker Voting rounds.

Negative

  1. Additional complexity of ABCI protocol
  2. Additional changes compared to the upstream Tendermint project make backporting harder.

Neutral

References

Service lifecycle core

BaseService.Start gives OnStart an attempt-scoped child context. Parent cancellation and manual Stop both cancel it. Go(ctx, fn) registers workers using that context or a derived context; registration is synchronized with shutdown and rejected for a different attempt or after shutdown begins.

Shutdown closes admission, marks IsRunning false, cancels the context and calls OnStop. It then joins registered workers, calls optional OnDrain, and releases Wait. Stopping() signals cancellation separately from completion. Stop invokes OnStop synchronously but does not join registered workers. Concurrent or recursive calls to Stop return immediately. During OnStart, Stop cancels immediately and defers the shutdown hook until successful startup returns; Wait still joins the attempt.

Hooks run outside the lifecycle mutex. OnStop must unblock workers without waiting for its own registered work. Workers may call Stop; neither hooks nor workers may call their own service's Wait. Release resources shared with workers in OnDrain, or after Stop followed by Wait.

IsRunning is false during startup and shutdown. Wait joins the current attempt, including startup and cleanup; it returns immediately before startup. Failed OnStart cancels and joins its workers before retry becomes possible. The implementation must roll back acquired resources: failed attempts do not invoke OnStop or OnDrain. Successfully stopped services cannot restart. Composite retries also require recreating children already successfully stopped.

First migration stage

Consensus State registers its receive loop and joins its queue fan-in and WAL before returning. The timeout ticker, WAL and autofile group use shared worker tracking. AutoFile explicitly joins its timer/signal worker. WAL ownership lasts until the receive loop's final write. Consensus handoff admission is also joined, so a late blocksync handoff cannot start State after reactor shutdown. The caller may cancel its wait after admission, but the reactor still owns and joins the handoff. Startup and the resulting State use the reactor's lifetime.

Node waits for reactors before closing event sinks and stores. The blocked receive-loop regression and original node teardown tests verify this path without using leaktest as an extra production-shutdown barrier.

This is a partial migration. Plain goroutines are not tracked automatically. RPC/WebSocket handlers, mempool rechecks, reactor gossip and other service workers retain their existing ownership mechanisms. This change does not claim that Node.Wait joins every helper or introduce the broader durable-finalization and dependency-lifetime changes from #1515.

Three compatibility bridges preserve parent-context lifetimes: blocksync application during handoff, connection error callbacks after I/O shutdown, and an admitted consensus commit during direct State.Stop. Parent cancellation still aborts these operations. In particular, reactor/node shutdown can interrupt finalization and require WAL recovery. Their existing ownership mechanisms remain in place; this does not detach application work from node shutdown.

Cancellation compatibility audit

Cancellation precedes OnStop for every BaseService implementation, not only the managed-worker migrations. The in-tree hook/callback audit covers these groups:

ImplementationsCancellation and cleanup policy
Consensus State, reactor, ticker, WAL; autofile GroupManaged work is joined; commit uses the State parent context; file ownership ends with explicit Close.
Blocksync reactor and SynchronizerHandoff preserves application work across Synchronizer.Stop; stopping its parent reactor may cancel it. Reactor OnStop stops and waits for the Synchronizer, so reactor Wait covers in-flight application.
P2P Router and MConnectionConnection callbacks retain the parent. Router removes peer bookkeeping even with a canceled context; final PeerStatusDown broadcasts are best effort.
PEX, mempool, statesync and evidence reactorsProcessing uses the work context; stop hooks close resources/dispatchers or require no extra signal. Existing plain-worker ownership is unchanged.
EventBus, pubsub Server and indexer ServiceEvent processing is cancellation-driven; indexer closes sinks in its existing hook. No guarantee of a live cleanup callback context.
ABCI local, socket, gRPC and routed clients; proxy; socket/gRPC serversStop hooks close transports, drain requests or stop children without requiring a live work context.
SignerServer, signer listener/dialer endpointsHooks close endpoints/listeners; dial/retry work observes cancellation.
ValidatorConnExecutor and light RPC ClientUnsubscribe uses a separate bounded cleanup context; client hooks invoke owned closers.
Node and seed nodeParent cancellation stops children; hooks wait for existing service completion before closing owned stores.

This is a cancellation compatibility audit, not a claim that unmigrated plain workers are all joined. Subscribers must stop with their owner, without relying on a final router broadcast. Avoid detaching broadcasts from cancellation: unresponsive subscribers could otherwise block shutdown.

Group rotation reuses its AutoFile and closes the current descriptor before renaming; it does not create a worker or signal registration per rotation. The owner must Close the group, including when it never started, to join the head's single timer/signal worker. Context cancellation alone does not release the head.

CLI shutdown deadline

The start command cancels the node on SIGINT or SIGTERM and waits for Node.Wait before returning. --shutdown-timeout controls this wait (default 30 seconds; must be positive). Set the deployment supervisor's termination grace period longer than this timeout to allow Tenderdash to report incomplete cleanup. If the deadline expires, the command returns an error and the Tenderdash binary exits nonzero. This does not skip reactor joins or close stores while workers remain active. An embedded command caller must also terminate on this error; cleanup may still be running. The deadline bounds shutdown after startup returns, not a blocked startup hook.

RFC 006: Event Subscription

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 30-Oct-2021: Initial draft (@creachadair)

Abstract

The Tenderdash consensus node allows clients to subscribe to its event stream via methods on its RPC service. The ability to view the event stream is valuable for clients, but the current implementation has some deficiencies that make it difficult for some clients to use effectively. This RFC documents these issues and discusses possible approaches to solving them.

Background

A running Tenderdash consensus node exports a JSON-RPC service that provides a large set of methods for inspecting and interacting with the node. One important cluster of these methods are the subscribe, unsubscribe, and unsubscribe_all methods, which permit clients to subscribe to a filtered stream of the events generated by the node as it runs.

Unlike the other methods of the service, the methods in the "event subscription" cluster are not accessible via ordinary HTTP GET or POST requests, but require upgrading the HTTP connection to a websocket. This is necessary because the subscribe request needs a persistent channel to deliver results back to the client, and an ordinary HTTP connection does not reliably persist across multiple requests. Since these methods do not work properly without a persistent channel, they are only exported via a websocket connection, and are not routed for plain HTTP.

Discussion

There are some operational problems with the current implementation of event subscription in the RPC service:

  • Event delivery is not valid JSON-RPC. When a client issues a subscribe request, the server replies (correctly) with an initial empty acknowledgement ({}). After that, each matching event is delivered "unsolicited" (without another request from the client), as a separate response object with the same ID as the initial request.

    This matters because it means a standard JSON-RPC client library can't interact correctly with the event subscription mechanism.

    Even for clients that can handle unsolicited values pushed by the server, these responses are invalid: They have an ID, so they cannot be treated as notifications; but the ID corresponds to a request that was already completed. In practice, this means that general-purpose JSON-RPC libraries cannot use this method correctly -- it requires a custom client.

    The Go RPC client from the Tenderdash core can support this case, but clients in other languages have no easy solution.

    This is the cause of issue #2949.

  • Subscriptions are terminated by disconnection. When the connection to the client is interrupted, the subscription is silently dropped.

    This is a reasonable behavior, but it matters because a client whose subscription is dropped gets no useful error feedback, just a closed connection. Should they try again? Is the node overloaded? Was the client too slow? Did the caller forget to respond to pings? Debugging these kinds of failures is unnecessarily painful.

    Websockets compound this, because websocket connections time out if no traffic is seen for a while, and keeping them alive requires active cooperation between the client and server. With a plain TCP socket, liveness is handled transparently by the keepalive mechanism. On a websocket, however, one side has to occasionally send a PING (if the connection is otherwise idle). The other side must return a matching PONG in time, or the connection is dropped. Apart from being tedious, this is highly susceptible to CPU load.

    The Tenderdash Go implementation automatically sends and responds to pings. Clients in other languages (or not wanting to use the Tenderdash libraries) need to handle it explicitly. This burdens the client for no practical benefit: A subscriber has no information about when matching events may be available, so it shouldn't have to participate in keeping the connection alive.

  • Mismatched load profiles. Most of the RPC service is mainly important for low-volume local use, either by the application the node serves (e.g., the ABCI methods) or by the node operator (e.g., the info methods). Event subscription is important for remote clients, and may represent a much higher volume of traffic.

    This matters because both are using the same JSON-RPC mechanism. For low-volume local use, the ergonomics of JSON-RPC are a good fit: It's easy to issue queries from the command line (e.g., using curl) or to write scripts that call the RPC methods to monitor the running node.

    For high-volume remote use, JSON-RPC is not such a good fit: Even leaving aside the non-standard delivery protocol mentioned above, the time and memory cost of encoding event data matters for the stability of the node when there can be potentially hundreds of subscribers. Moreover, a subscription is long-lived compared to most RPC methods, in that it may persist as long the node is active.

  • Mismatched security profiles. The RPC service exports several methods that should not be open to arbitrary remote callers, both for correctness reasons (e.g., remove_tx and broadcast_tx_*) and for operational stability reasons (e.g., tx_search). A node may still need to expose events, however, to support UI tools.

    This matters, because all the methods share the same network endpoint. While it is possible to block the top-level GET and POST handlers with a proxy, exposing the /websocket handler exposes not only the event subscription methods, but the rest of the service as well.

Possible Improvements

There are several things we could do to improve the experience of developers who need to subscribe to events from the consensus node. These are not all mutually exclusive.

  1. Split event subscription into a separate service. Instead of exposing event subscription on the same endpoint as the rest of the RPC service, dedicate a separate endpoint on the node for only event subscription. The rest of the RPC services (sans events) would remain as-is.

    This would make it easy to disable or firewall outside access to sensitive RPC methods, without blocking access to event subscription (and vice versa). This is probably worth doing, even if we don't take any of the other steps described here.

  2. Use a different protocol for event subscription. There are various ways we could approach this, depending how much we're willing to shake up the current API. Here are sketches of a few options:

    • Keep the websocket, but rework the API to be more JSON-RPC compliant, perhaps by converting event delivery into notifications. This is less up-front change for existing clients, but retains all of the existing implementation complexity, and doesn't contribute much toward more serious performance and UX improvements later.

    • Switch from websocket to plain HTTP, and rework the subscription API to use a more conventional request/response pattern instead of streaming. This is a little more up-front work for existing clients, but leverages better library support for clients not written in Go.

      The protocol would become more chatty, but we could mitigate that with batching, and in return we would get more control over what to do about slow clients: Instead of simply silently dropping them, as we do now, we could drop messages and signal the client that they missed some data ("M dropped messages since your last poll").

      This option is probably the best balance between work, API change, and benefit, and has a nice incidental effect that it would be easier to debug subscriptions from the command-line, like the other RPC methods.

    • Switch to gRPC: Preserves a persistent connection and gives us a more efficient binary wire format (protobuf), at the cost of much more work for clients and harder debugging. This may be the best option if performance and server load are our top concerns.

      Given that we are currently using JSON-RPC, however, I'm not convinced the costs of encoding and sending messages on the event subscription channel are the limiting factor on subscription efficiency, however.

  3. Delegate event subscriptions to a proxy. Give responsibility for managing event subscription to a proxy that runs separately from the node, and switch the node to push events to the proxy (like a webhook) instead of serving subscribers directly. This is more work for the operator (another process to configure and run) but may scale better for big networks.

    I mention this option for completeness, but making this change would be a fairly substantial project. If we want to consider shifting responsibility for event subscription outside the node anyway, we should probably be more systematic about it. For a more principled approach, see point (4) below.

  4. Move event subscription downstream of indexing. We are already planning to give applications more control over event indexing. By extension, we might allow the application to also control how events are filtered, queried, and subscribed. Having the application control these concerns, rather than the node, might make life easier for developers building UI and tools for that application.

    This is a much larger change, so I don't think it is likely to be practical in the near-term, but it's worth considering as a broader option. Some of the existing code for filtering and selection could be made more reusable, so applications would not need to reinvent everything.

References

RFC 007 : Deterministic Proto Byte Serialization

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 09-Dec-2021: Initial draft (@williambanfield).

Abstract

This document discusses the issue of stable byte-representation of serialized messages within Tenderdash and describes a few possible routes that could be taken to address it.

Background

We use the byte representations of wire-format proto messages to produce and verify hashes of data within the Tenderdash codebase as well as for producing and verifying cryptographic signatures over these signed bytes.

The protocol buffer encoding spec does not guarantee that the byte representation of a protocol buffer message will be the same between two calls to an encoder. While there is a mode to force the encoder to produce the same byte representation of messages within a single binary, these guarantees are not good enough for our use case in Tenderdash. We require multiple different versions of a binary running Tenderdash to be able to inter-operate. Additionally, we require that multiple different systems written in different languages be able to participate in different aspects of the protocols of Tenderdash and be able to verify the integrity of the messages they each produce.

While this has not yet created a problem that we know of in a running network, we should make sure to provide stronger guarantees around the serialized representation of the messages used within the Tenderdash consensus algorithm to prevent any issue from occurring.

Discussion

Proto has the following points of variability that can produce non-deterministic byte representation:

  1. Encoding order of fields within a message.

Proto allows fields to be encoded in any order and even be repeated.

  1. Encoding order of elements of a repeated field.

repeated fields in a proto message can be serialized in any order.

  1. Presence or absence of default values.

Types in proto have defined default values similar to Go's zero values. Writing or omitting a default value are both legal ways of encoding a wire message.

  1. Serialization of 'unknown' fields.

Unknown fields can be present when a message is created by a binary with a newer version of the proto that contains fields that the deserializer in a different binary does not yet know about. Deserializers in binaries that do not know about the field will maintain the bytes of the unknown field but not place them into the deserialized structure.

We have a few options to consider when producing this stable representation.

Options for deterministic byte representation

Use only compliant serializers and constrain field usage

According to Cosmos-SDK ADR-27, when message types obey a simple set of rules, gogoproto produces a consistent byte representation of serialized messages. This seems promising, although more research is needed to guarantee gogoproto always produces a consistent set of bytes on serialized messages. This would solve the problem within Tenderdash as written in Go, but would require ensuring that there are similar serializers written in other languages that produce the same output as gogoproto.

Reorder serialized bytes to ensure determinism.

The serialized form of a proto message can be transformed into a canonical representation by applying simple rules to the serialized bytes. Re-ordering the serialized bytes would allow Tenderdash to produce a canonical byte representation without having to simultaneously maintain a custom proto marshaller.

This could be implemented as a function in many languages that performed the following producing bytes to sign or hashing:

  1. Does not add any of the data from unknown fields into the type to hash.

Tenderdash should not run into a case where it needs to verify the integrity of data with unknown fields for the following reasons:

The purpose of checking hash equality within Tenderdash is to ensure that its local copy of data matches the data that the network agreed on. There should therefore not be a case where a process is checking hash equality using data that it did not expect to receive. What the data represent may be opaque to the process, such as when checking the transactions in a block, but the process will still have expected to receive this data, despite not understanding what their internal structure is. It's not clear what it would mean to verify that a block contains data that a process does not know about.

The same reasoning applies for signature verification within Tenderdash. Processes verify that a digital signature signed over a set of bytes by locally reconstructing the data structure that the digital signature signed using the process's local data.

  1. Reordered all message fields to be in tag-sorted order.

Tag-sorting top-level fields will place all fields of the same tag in a adjacent to eachother within the serialized representation.

  1. Reordered the contents of all repeated fields to be in lexicographically sorted order.

repeated fields will appear in a message as having the same tag but will contain different contents. Therefore, lexicographical sorting will produce a stable ordering of fields with the same tag.

  1. Deleted all default values from the byte representation.

Encoders can include default values or omit them. Most encoders appear to omit them but we may wish to delete them just to be safe.

  1. Recursively performed these operations on any length-delimited subfields.

Length delimited fields may contain messages, strings, or just bytes. However, it's not possible to know what data is being represented by such a field. A 'string' may happen to have the same structure as an embedded message and we cannot disambiguate. For this reason, we must apply these same rules to all subfields that may contain messages. Because we cannot know if we have totally mangled the interior 'string' or not, this data should never be deserialized or used for anything beyond hashing.

A prototype implementation by @creachadair of this can be found in the wirepb repo. This could be implemented in multiple languages more simply than ensuring that there are canonical proto serializers that match in each language.

Future work

We should add clear documentation to the Tenderdash codebase every time we compare hashes of proto messages or use proto serialized bytes to produces a digital signatures that we have been careful to ensure that the hashes are performed properly.

References

RFC 008: Don't Panic

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 2021-12-17: initial draft (@tychoish)

Abstract

Today, the Tenderdash core codebase has panics in a number of cases as a response to exceptional situations. These panics complicate testing, and might make tenderdash components difficult to use as a library in some circumstances. This document outlines a project of converting panics to errors and describes the situations where its safe to panic.

Background

Panics in Go are a great mechanism for aborting the current execution for truly exceptional situations (e.g. memory errors, data corruption, processes initialization); however, because they resemble exceptions in other languages, it can be easy to over use them in the implementation of software architectures. This certainly happened in the history of Tenderdash, and as we embark on the project of stabilizing the package, we find ourselves in the right moment to reexamine our use of panics, and largely where panics happen in the code base.

There are still some situations where panics are acceptable and desireable, but it's important that Tenderdash, as a project, comes to consensus--perhaps in the text of this document--on the situations where it is acceptable to panic.

References

Discussion

Acceptable Panics

Initialization

It is unambiguously safe (and desireable) to panic in init() functions in response to any kind of error. These errors are caught by tests, and occur early enough in process initialization that they won't cause unexpected runtime crashes.

Other code that is called early in process initialization MAY panic, in some situations if it's not possible to return an error or cause the process to abort early, although these situations should be vanishingly slim.

Data Corruption

If Tenderdash code encounters an inconsistency that could be attributed to data corruption or a logical impossibility it is safer to panic and crash the process than continue to attempt to make progress in these situations.

Examples including reading data out of the storage engine that is invalid or corrupt, or encountering an ambiguous situation where the process should halt. Generally these forms of corruption are detected after interacting with a trusted but external data source, and reflect situations where the author thinks its safer to terminate the process immediately rather than allow execution to continue.

Unrecoverable Consensus Failure

In general, a panic should be used in the case of unrecoverable consensus failures. If a process detects that the network is behaving in an incoherent way and it does not have a clearly defined and mechanism for recovering, the process should panic.

Static Validity

It is acceptable to panic for invariant violations, within a library or package, in situations that should be statically impossible, because there is no way to make these kinds of assertions at compile time.

For example, type-asserting interface{} values returned by container/list and container/heap (and similar), is acceptable, because package authors should have exclusive control of the inputs to these containers. Packages should not expose the ability to add arbitrary values to these data structures.

Controlled Panics Within Libraries

In some algorithms with highly recursive structures or very nested call patterns, using a panic, in combination with conditional recovery handlers results in more manageable code. Ultimately this is a limited application, and implementations that use panics internally should only recover conditionally, filtering out panics rather than ignoring or handling all panics.

Request Handling

Code that handles responses to incoming/external requests (e.g. http.Handler) should avoid panics, but practice this isn't totally possible, and it makes sense that request handlers have some kind of default recovery mechanism that will prevent one request from terminating a service.

Unacceptable Panics

In no other situation is it acceptable for the code to panic:

  • there should be no controlled panics that callers are required to handle across library/package boundaries.
  • callers of library functions should not expect panics.
  • ensuring that arbitrary go routines can't panic.
  • ensuring that there are no arbitrary panics in core production code, espically code that can run at any time during the lifetime of a process.
  • all test code and fixture should report normal test assertions with a mechanism like testify's require assertion rather than calling panic directly.

The goal of this increased "panic rigor" is to ensure that any escaped panic is reflects a fixable bug in Tenderdash.

Removing Panics

The process for removing panics involve a few steps, and will be part of an ongoing process of code modernization:

  • converting existing explicit panics to errors in cases where it's possible to return an error, the errors can and should be handled, and returning an error would not lead to data corruption or cover up data corruption.

  • increase rigor around operations that can cause runtime errors, like type assertions, nil pointer errors, array bounds access issues, and either avoid these situations or return errors where possible.

  • remove generic panic handlers which could cover and hide known panics.

RFC 009 : Consensus Parameter Upgrade Considerations

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 06-Jan-2011: Initial draft (@williambanfield).

Abstract

This document discusses the challenges of adding additional consensus parameters to Tenderdash and proposes a few solutions that can enable addition of consensus parameters in a backwards-compatible way.

Background

This section provides an overview of the issues of adding consensus parameters to Tenderdash.

Hash Compatibility

Tenderdash produces a hash of a subset of the consensus parameters. The values that are hashed currently are the BlockMaxGas and the BlockMaxSize. These are currently in the HashedParams struct. This hash is included in the block and validators use it to validate that their local view of the consensus parameters matches what the rest of the network is configured with.

Any new consensus parameters added to Tenderdash should be included in this hash. This presents a challenge for verification of historical blocks when consensus parameters are added. If a network produced blocks with a version of Tenderdash that did not yet have the new consensus parameters, the parameter hash it produced will not reference the new parameters. Any nodes joining the network with the newer version of Tenderdash will have the new consensus parameters. Tenderdash will need to handle this case so that new versions of Tenderdash with new consensus parameters can still validate old blocks correctly without having to do anything overly complex or hacky.

Allowing Developer-Defined Values and the EndBlock Problem

When new consensus parameters are added, application developers may wish to set values for them so that the developer-defined values may be used as soon as the software upgrades. We do not currently have a clean mechanism for handling this.

Consensus parameter updates are communicated from the application to Tenderdash within EndBlock of some height H and take effect at the next height, H+1. This means that for updates that add a consensus parameter, there is a single height where the new parameters cannot take effect. The parameters did not exist in the version of the software that emitted the EndBlock response for height H-1, so they cannot take effect at height H. The first height that the updated params can take effect is height H+1. As of now, height H must run with the defaults.

Discussion

Hash Compatibility

This section discusses possible solutions to the problem of maintaining backwards-compatibility of hashed parameters while adding new parameters.

Never Hash Defaults

One solution to the problem of backwards-compatibility is to never include parameters in the hash if the are using the default value. This means that blocks produced before the parameters existed will have implicitly been created with the defaults. This works because any software with newer versions of Tenderdash must be using the defaults for new parameters when validating old blocks since the defaults can not have been updated until a height at which the parameters existed.

Only Update HashedParams on Hash-Breaking Releases

An alternate solution to never hashing defaults is to not update the hashed parameters on non-hash-breaking releases. This means that when new consensus parameters are added to Tenderdash, there may be a release that makes use of the parameters but does not verify that they are the same across all validators by referencing them in the hash. This seems reasonably safe given the fact that only a very far subset of the consensus parameters are currently verified at all.

Version The Consensus Parameter Hash Scheme

The upcoming work on soft upgrades proposes applying different hashing rules depending on the active block version. The consensus parameter hash could be versioned in the same way. When different block versions are used, a different set of consensus parameters will be included in the hash.

Developer Defined Values

This section discusses possible solutions to the problem of allowing application developers to define values for the new parameters during the upgrade that adds the parameters.

Using InitChain for New Values

One solution to the problem of allowing application developers to define values for new consensus parameters is to call the InitChain ABCI method on application startup and fetch the value for any new consensus parameters. The response object contains a field for ConsensusParameter updates so this may serve as a natural place to put this logic.

This poses a few difficulties. Nodes replaying old blocks while running new software do not ever call InitChain after the initial time. They will therefore not have a way to determine that the parameters changed at some height by using a call to InitChain. The EndBlock response is how parameter changes at a height are currently communicated to Tenderdash and conflating these cases seems risky.

Force Defaults For Single Height

An alternate option is to not use InitChain and instead require chains to use the default values of the new parameters for a single height.

As documented in the upcoming ADR-74, popular chains often simply use the default values. Additionally, great care is being taken to ensure that logic governed by upcoming consensus parameters is not liveness-breaking. This means that, at worst-case, chains will experience a single slow height while waiting for the new values to by applied.

Add a new UpgradeChain method

An additional method for allowing chains to update the consensus parameters that do not yet exist is to add a new UpgradeChain method to ABCI. The upgrade chain method would be called when the chain detects that the version of block that it is about to produce does not match the previous block. This method would be called after EndBlock and would return the set of consensus parameters to use at the next height. It would therefore give an application the chance to set the new consensus parameters before running a height with these new parameter.

References

RFC 011: Remove Gas From Tenderdash

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 03-Feb-2022: Initial draft (@williambanfield).
  • 10-Feb-2022: Update in response to feedback (@williambanfield).
  • 11-Feb-2022: Add reflection on MaxGas during consensus (@williambanfield).

Abstract

In the v0.25.0 release, Tenderdash added a mechanism for tracking 'Gas' in the mempool. At a high level, Gas allows applications to specify how much it will cost the network, often in compute resources, to execute a given transaction. While such a mechanism is common in blockchain applications, it is not generalizable enough to be a maintained as a part of Tenderdash. This RFC explores the possibility of removing the concept of Gas from Tenderdash while still allowing applications the power to control the contents of blocks to achieve similar goals.

Background

The notion of Gas was included in the original Ethereum whitepaper and exists as an important feature of the Ethereum blockchain.

The whitepaper describes Gas as an Anti-DoS mechanism. The Ethereum Virtual Machine provides a Turing complete execution platform. Without any limitations, malicious actors could waste computation resources by directing the EVM to perform large or even infinite computations. Gas serves as a metering mechanism to prevent this.

Gas appears to have been added to Tenderdash multiple times, initially as part of a now defunct /vm package, and in its most recent iteration as part of v0.25.0 as a mechanism to limit the transactions that will be included in the block by an additional parameter.

Gas has gained adoption within the Cosmos ecosystem as part of the Cosmos SDK. The SDK provides facilities for tracking how much 'Gas' a transaction is expected to take and a mechanism for tracking how much gas a transaction has already taken.

Non-SDK applications also make use of the concept of Gas. Anoma appears to implement a gas system to meter the transactions it executes.

While the notion of gas is present in projects that make use of Tenderdash, it is not a concern of Tenderdash's. Tenderdash's value and goal is producing blocks via a distributed consensus algorithm. Tenderdash relies on the application specific code to decide how to handle the transactions Tenderdash has produced (or if the application wants to consider them at all). Gas is an application concern.

Our implementation of Gas is not currently enforced by consensus. Our current validation check that occurs during block propagation does not verify that the block is under the configured MaxGas. Ensuring that the transactions in a proposed block do not exceed MaxGas would require input from the application during propagation. The ProcessProposal method introduced as part of ABCI++ would enable such input but would further entwine Tenderdash and the application. The issue of checking MaxGas during block propagation is important because it demonstrates that the feature as it currently exists is not implemented as fully as it perhaps should be.

Our implementation of Gas is causing issues for node operators and relayers. At the moment, transactions that overflow the configured 'MaxGas' can be silently rejected from the mempool. Overflowing MaxGas is the only way that a transaction can be considered invalid that is not directly a result of failing the CheckTx. Operators, and the application, do not know that a transaction was removed from the mempool for this reason. A stateless check of this nature is exactly what CheckTx exists for and there is no reason for the mempool to keep track of this data separately. A special MempoolError field was added in v0.35 to communicate to clients that a transaction failed after CheckTx. While this should alleviate the pain for operators wishing to understand if their transaction was included in the mempool, it highlights that the abstraction of what is included in the mempool is not currently well defined.

Removing Gas from Tenderdash and the mempool would allow for the mempool to be a better abstraction: any transaction that arrived at CheckTx and passed the check will either be a candidate for a later block or evicted after a TTL is reached or to make room for other, higher priority transactions. All other transactions are completely invalid and can be discarded forever.

Removing gas will not be completely straightforward. It will mean ensuring that equivalent functionality can be implemented outside of the mempool using the mempool's API.

Discussion

This section catalogs the functionality that will need to exist within the Tenderdash mempool to allow Gas to be removed and replaced by application-side bookkeeping.

Requirement: Provide Mempool Tx Sorting Mechanism

Gas produces a market for inclusion in a block. On many networks, a gas fee is included in pending transactions. This fee indicates how much a user is willing to pay per unit of execution and the fees are distributed to validators.

Validators wishing to extract higher gas fees are incentivized to include transactions with the highest listed gas fees into each block. This produces a natural ordering of the pending transactions. Applications wishing to implement a gas mechanism need to be able to order the transactions in the mempool. This can trivially be accomplished by sorting transactions using the priority field available to applications as part of v0.35's ResponseCheckTx message.

Requirement: Allow Application-Defined Block Resizing

When creating a block proposal, Tenderdash pulls a set of possible transactions out of the mempool to include in the next block. Tenderdash uses MaxGas to limit the set of transactions it pulls out of the mempool fetching a set of transactions whose sum is less than MaxGas.

By removing gas tracking from Tenderdash's mempool, Tenderdash will need to provide a way for applications to determine an acceptable set of transactions to include in the block.

This is what the new ABCI++ PrepareProposal method is useful for. Applications that wish to limit the contents of a block by an application-defined limit may do so by removing transactions from the proposal it is passed during PrepareProposal. Applications wishing to reach parity with the current Gas implementation may do so by creating an application-side limit: filtering out transactions from PrepareProposal the cause the proposal the exceed the maximum gas. Additionally, applications can currently opt to have all transactions in the mempool delivered during PrepareProposal by passing -1 for MaxGas and MaxBytes into ReapMaxBytesMaxGas.

Requirement: Handle Transaction Metadata

Moving the gas mechanism into applications adds an additional piece of complexity to applications. The application must now track how much gas it expects a transaction to consume. The mempool currently handles this bookkeeping responsibility and uses the estimated gas to determine the set of transactions to include in the block. In order to task the application with keeping track of this metadata, we should make it easier for the application to do so. In general, we'll want to keep only one copy of this type of metadata in the program at a time, either in the application or in Tenderdash.

The following sections are possible solutions to the problem of storing transaction metadata without duplication.

Metadata Handling: EvictTx Callback

A possible approach to handling transaction metadata is by adding a new EvictTx ABCI method. Whenever the mempool is removing a transaction, either because it has reached its TTL or because it failed RecheckTx, EvictTx would be called with the transaction hash. This would indicate to the application that it could free any metadata it was storing about the transaction such as the computed gas fee.

Eviction callbacks are pretty common in caching systems, so this would be very well-worn territory.

Metadata Handling: Application-Specific Metadata Field(s)

An alternative approach to handling transaction metadata would be would be the addition of a new application-metadata field in the ResponseCheckTx. This field would be a protocol buffer message whose contents were entirely opaque to Tenderdash. The application would be responsible for marshalling and unmarshalling whatever data it stored in this field. During PrepareProposal, the application would be passed this metadata along with the transaction, allowing the application to use it to perform any necessary filtering.

If either of these proposed metadata handling techniques are selected, it's likely useful to enable applications to gossip metadata along with the transaction it is gossiping. This could easily take the form of an opaque proto message that is gossiped along with the transaction.

References

RFC 012: Event Indexing Revisited

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 11-Feb-2022: Add terminological notes.
  • 10-Feb-2022: Updated from review feedback.
  • 07-Feb-2022: Initial draft (@creachadair)

Abstract

A Tenderdash node allows ABCI events associated with block and transaction processing to be "indexed" into persistent storage. The original Tenderdash implementation provided a fixed, built-in proprietary indexer for such events.

In response to user requests to customize indexing, ADR 065 introduced an "event sink" interface that allows developers (at least in theory) to plug in alternative index storage.

Although ADR-065 was a good first step toward customization, its implementation model does not satisfy all the user requirements. Moreover, this approach leaves some existing technical issues with indexing unsolved.

This RFC documents these concerns, and discusses some potential approaches to solving them. This RFC does not propose a specific technical decision. It is meant to unify and focus some of the disparate discussions of the topic.

Background

We begin with some important terminological context. The term "event" in Tenderdash can be confusing, as the same word is used for multiple related but distinct concepts:

  1. ABCI Events refer to the key-value metadata attached to blocks and transactions by the application. These values are represented by the ABCI Event protobuf message type.

  2. Consensus Events refer to the data published by the Tenderdash node to its pubsub bus in response to various consensus state transitions and other important activities, such as round updates, votes, transaction delivery, and block completion.

This confusion is compounded because some "consensus event" values also have "ABCI event" metadata attached to them. Notably, block and transaction items typically have ABCI metadata assigned by the application.

Indexers and RPC clients subscribed to the pubsub bus receive consensus events, but they identify which ones to care about using query expressions that match against the ABCI events associated with them.

In the discussion that follows, we will use the term event item to refer to a datum published to or received from the pubsub bus, and ABCI event or event metadata to refer to the key/value annotations.

Indexing in this context means recording the association between certain ABCI metadata and the blocks or transactions they're attached to. The ABCI metadata typically carry application-specific details like sender and recipient addresses, catgory tags, and so forth, that are not part of consensus but are used by UI tools to find and display transactions of interest.

The consensus node records the blocks and transactions as part of its block store, but does not persist the application metadata. Metadata persistence is the task of the indexer, which can be (optionally) enabled by the node operator.

History

The original indexer built in to Tenderdash stored index data in an embedded tm-db database with a proprietary key layout. In ADR 065, we noted that this implementation has both performance and scaling problems under load. Moreover, the only practical way to query the index data is via the query filter language used for event subscription. Issue #1161 appears to be a motivational context for that ADR.

To mitigate both of these concerns, we introduced the EventSink interface, combining the original transaction and block indexer interfaces along with some service plumbing. Using this interface, a developer can plug in an indexer that uses a more efficient storage engine, and provides a more expressive query language. As a proof-of-concept, we built a PostgreSQL event sink that exports data to a PostgreSQL database.

Although this approach addressed some of the immediate concerns, there are several issues for custom indexing that have not been fully addressed. Here we will discuss them in more detail.

For further context, including links to user reports and related work, see also the Pluggable custom event indexing tracking issue issue.

Issue 1: Tight Coupling

The EventSink interface supports multiple implementations, but plugging in implementations still requires tight integration with the node. In particular:

  • Any custom indexer must either be written in Go and compiled in to the Tenderdash binary, or the developer must write a Go shim to communicate with the implementation and build that into the Tenderdash binary.

  • This means to support a custom indexer, it either has to be integrated into the Tenderdash core repository, or every installation that uses that indexer must fetch or build a patched version of Tenderdash.

The problem with integrating indexers into Tenderdash Core is that every user of Tenderdash Core takes a dependency on all supported indexers, including those they never use. Even if the unused code is disabled with build tags, users have to remember to do this or potentially be exposed to security issues that may arise in any of the custom indexers. This is a risk for Tenderdash, which is a trust-critical component of all applications built on it.

The problem with not integrating indexers into Tenderdash Core is that any developer who wants to use a particular indexer must now fetch or build a patched version of the core code that includes the custom indexer. Besides being inconvenient, this makes it harder for users to upgrade their node, since they need to either re-apply their patches directly or wait for an intermediary to do it for them.

Even for developers who have written their applications in Go and link with the consensus node directly (e.g., using the Cosmos SDK), these issues add a potentially significant complication to the build process.

Issue 2: Legacy Compatibility

The EventSink interface retains several limitations of the original proprietary indexer. These include:

  • The indexer has no control over which event items are reported. Only the exact block and transaction events that were reported to the original indexer are reported to a custom indexer.

  • The interface requires the implementation to define methods for the legacy search and query API. This requirement comes from the integation with the event subscription RPC API, but actually supporting these methods is not trivial.

At present, only the original KV indexer implements the query methods. Even the proof-of-concept PostgreSQL implementation simply reports errors for all calls to these methods.

Even for a plugin written in Go, implementing these methods "correctly" would require parsing and translating the custom query language over whatever storage platform the indexer uses.

For a plugin not written in Go, even beyond the cost of integration the developer would have to re-implement the entire query language.

Issue 3: Indexing Delays Consensus

Within the node, indexing hooks in to the same internal pubsub dispatcher that is used to export event items to the event subscription RPC API. In contrast with RPC subscribers, however, indexing is a "privileged" subscriber: If an RPC subscriber is "too slow", the node may terminate the subscription and disconnect the client. That means that RPC subscribers may lose (miss) event items. The indexer, however, is "unbuffered", and the publisher will never drop or disconnect from it. If the indexer is slow, the publisher will block until it returns, to ensure that no event items are lost.

In practice, this means that the performance of the indexer has a direct effect on the performance of the consensus node: If the indexer is slow or stalls, it will slow or halt the progress of consensus. Users have already reported this problem even with the built-in indexer (see, for example, #7247). Extending this concern to arbitrary user-defined custom indexers gives that risk a much larger surface area.

Discussion

It is not possible to simultaneously guarantee that publishing event items will not delay consensus, and also that all event items of interest are always completely indexed.

Therefore, our choice is between eliminating delay (and minimizing loss) or eliminating loss (and minimizing delay). Currently, we take the second approach, which has led to user complaints about consensus delays due to indexing and subscription overhead.

  • If we agree that consensus performance supersedes index completeness, our design choices are to constrain the likelihood and frequency of missing event items.

  • If we decide that consensus performance is more important than index completeness, our option is to minimize overhead on the event delivery path and document that indexer plugins constrain the rate of consensus.

Since we have user reports requesting both properties, we have to choose one or the other. Since the primary job of the consensus engine is to correctly, robustly, reliablly, and efficiently replicate application state across the network, I believe the correct choice is to favor consensus performance.

An important consideration for this decision is that a node does not index application metadata separately: If indexing is disabled, there is no built-in mechanism to go back and replay or reconstruct the data that an indexer would have stored. The node does store the blockchain itself (i.e., the blocks and their transactions), so potentially some use cases currently handled by the indexer could be handled by the node. For example, allowing clients to ask whether a given transaction ID has been committed to a block could in principle be done without an indexer, since it does not depend on application metadata.

Inevitably, a question will arise whether we could implement both strategies and toggle between them with a flag. That would be a worst-case scenario, requiring us to maintain the complexity of two very-different operational concerns. If our goal is that Tenderdash should be as simple, efficient, and trustworthy as posible, there is not a strong case for making these options configurable: We should pick a side and commit to it.

Design Principles

Although there is no unique "best" solution to the issues described above, there are some specific principles that a solution should include:

  1. A custom indexer should not require integration into Tenderdash core. A developer or node operator can create, build, deploy, and use a custom indexer with a stock build of the Tenderdash consensus node.

  2. Custom indexers cannot stall consensus. An indexer that is slow or stalls cannot slow down or prevent core consensus from making progress.

    The plugin interface must give node operators control over the tolerances for acceptable indexer performance, and the means to detect when indexers are falling outside those tolerances, but indexer failures should "fail safe" with respect to consensus (even if that means the indexer may miss some data, in sufficiently-extreme circumstances).

  3. Custom indexers control which event items they index. A custom indexer is not limited to only the current transaction and block events, but can observe any event item published by the node.

  4. Custom indexing is forward-compatible. Adding new event item types or metadata to the consensus node should not require existing custom indexers to be rebuilt or modified, unless they want to take advantage of the new data.

  5. Indexers are responsible for answering queries. An indexer plugin is not required to support the legacy query filter language, nor to be compatible with the legacy RPC endpoints for accessing them. Any APIs for clients to query a custom index are the responsibility of the indexer, not the node.

Open Questions

Given the constraints outlined above, there are important design questions we must answer to guide any specific changes:

  1. What is an acceptable probability that, given sufficiently extreme operational issues, an indexer might miss some number of events?

    There are two parts to this question: One is what constitutes an extreme operational problem, the other is how likely we are to miss some number of events items.

    • If the consensus is that no event item must ever be missed, no matter how bad the operational circumstances, then we must accept that indexing can slow or halt consensus arbitrarily. It is impossible to guarantee complete index coverage without potentially unbounded delays.

    • Otherwise, how much data can we afford to lose and how often? For example, if we can ensure no event item will be lost unless the indexer halts for at least five minutes, is that acceptable? What probabilities and time ranges are reasonable for real production environments?

  2. What level of operational overhead is acceptable to impose on node operators to support indexing?

    Are node operators willing to configure and run custom indexers as sidecar type processes alongside a node? How much indexer setup above and beyond the work of setting up the underlying node in isolation is tractable in production networks?

    The answer to this question also informs the question of whether we should keep an "in-process" indexing option, and to what extent that option needs to satisfy the suggested design principles.

    Relatedly, to what extent do we need to be concerned about the cost of encoding and sending event items to an external process (e.g., as JSON blobs or protobuf wire messages)? Given that the node already encodes event items as JSON for subscription purposes, the overhead would be negligible for the node itself, but the indexer would have to decode to process the results.

  3. What (if any) query APIs does the consensus node need to export, independent of the indexer implementation?

    One typical example is whether the node should be able to answer queries like "is this transaction ID in a block?" Currently, a node cannot answer this query unless it runs the built-in KV indexer. Does the node need to continue to support that query even for nodes that disable the KV indexer, or which use a custom indexer?

Informal Design Intent

The design principles described above implicate several components of the Tenderdash node, beyond just the indexer. In the context of ADR 075, we are re-working the RPC event subscription API to improve some of the UX issues discussed above for RPC clients. It is our expectation that a solution for pluggable custom indexing will take advantage of some of the same work.

On that basis, the design approach I am considering for custom indexing looks something like this (subject to refinement):

  1. A custom indexer runs as a separate process from the node.

  2. The indexer subscribes to event items via the ADR 075 events API.

    This means indexers would receive event payloads as JSON rather than protobuf, but since we already have to support JSON encoding for the RPC interface anyway, that should not increase complexity for the node.

  3. The existing PostgreSQL indexer gets reworked to have this form, and no longer built as part of the Tenderdash core binary.

    We can retain the code in the core repository as a proof-of-concept, or perhaps create a separate repository with contributed indexers and move it there.

  4. (Possibly) Deprecate and remove the legacy KV indexer, or disable it by default. If we decide to remove it, we can also remove the legacy RPC endpoints for querying the KV indexer.

    If we plan to do this, we should also investigate providing a way for clients to query whether a given transaction ID has landed in a block. That serves a common need, and currently only works if the KV indexer is enabled, but could be addressed more simply using the other data a node already has stored, without having to answer more general queries.

References

RFC 013: ABCI++

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 2020-01-11: initialized
  • 2022-02-11: Migrate RFC to tendermint repo (Originally RFC 004)

Author(s)

  • Dev (@valardragon)
  • Sunny (@sunnya97)

Context

ABCI is the interface between the consensus engine and the application. It defines when the application can talk to consensus during the execution of a blockchain. At the moment, the application can only act at one phase in consensus, immediately after a block has been finalized.

This restriction on the application prohibits numerous features for the application, including many scalability improvements that are now better understood than when ABCI was first written. For example, many of the scalability proposals can be boiled down to "Make the miner / block proposers / validators do work, so the network does not have to". This includes optimizations such as tx-level signature aggregation, state transition proofs, etc. Furthermore, many new security properties cannot be achieved in the current paradigm, as the application cannot enforce validators do more than just finalize txs. This includes features such as threshold cryptography, and guaranteed IBC connection attempts. We propose introducing three new phases to ABCI to enable these new features, and renaming the existing methods for block execution.

Prepare Proposal phase

This phase aims to allow the block proposer to perform more computation, to reduce load on all other full nodes, and light clients in the network. It is intended to enable features such as batch optimizations on the transaction data (e.g. signature aggregation, zk rollup style validity proofs, etc.), enabling stateless blockchains with validator provided authentication paths, etc.

This new phase will only be executed by the block proposer. The application will take in the block header and raw transaction data output by the consensus engine's mempool. It will then return block data that is prepared for gossip on the network, and additional fields to include into the block header.

Process Proposal Phase

This phase aims to allow applications to determine validity of a new block proposal, and execute computation on the block data, prior to the blocks finalization. It is intended to enable applications to reject block proposals with invalid data, and to enable alternate pipelined execution models. (Such as Ethereum-style immediate execution)

This phase will be executed by all full nodes upon receiving a block, though on the application side it can do more work in the even that the current node is a validator.

Vote Extension Phase

This phase aims to allow applications to require their validators do more than just validate blocks. Example usecases of this include validator determined price oracles, validator guaranteed IBC connection attempts, and validator based threshold crypto.

This adds an app-determined data field that every validator must include with their vote, and these will thus appear in the header.

Rename {BeginBlock, [DeliverTx], EndBlock} to FinalizeBlock

The prior phases gives the application more flexibility in their execution model for a block, and they obsolete the current methods for how the consensus engine relates the block data to the state machine. Thus we refactor the existing methods to better reflect what is happening in the new ABCI model.

This rename doesn't on its own enable anything new, but instead improves naming to clarify the expectations from the application in this new communication model. The existing ABCI methods BeginBlock, [DeliverTx], EndBlock are renamed to a single method called FinalizeBlock.

Summary

We include a more detailed list of features / scaling improvements that are blocked, and which new phases resolve them at the end of this document.

On the top is the existing definition of ABCI, and on the bottom is the proposed ABCI++.

Proposal

Below we suggest an API to add these three new phases. In this document, sometimes the final round of voting is referred to as precommit for clarity in how it acts in the Tenderdash case.

Prepare Proposal

Note, APIs in this section will change after Vote Extensions, we list the adjusted APIs further in the proposal.

The Prepare Proposal phase allows the block proposer to perform application-dependent work in a block, to lower the amount of work the rest of the network must do. This enables batch optimizations to a block, which has been empirically demonstrated to be a key component for scaling. This phase introduces the following ABCI method

#![allow(unused)]
fn main() {
fn PrepareProposal(Block) -> BlockData
}

where BlockData is a type alias for however data is internally stored within the consensus engine. In Tenderdash Core today, this is []Tx.

The application may read the entire block proposal, and mutate the block data fields. Mutated transactions will still get removed from the mempool later on, as the mempool rechecks all transactions after a block is executed.

The PrepareProposal API will be modified in the vote extensions section, for allowing the application to modify the header.

Process Proposal

The Process Proposal phase sends the block data to the state machine, prior to running the last round of votes on the state machine. This enables features such as allowing validators to reject a block according to whether state machine deems it valid, and changing block execution pipeline.

We introduce three new methods,

#![allow(unused)]
fn main() {
fn VerifyHeader(header: Header, isValidator: bool) -> ResponseVerifyHeader {...}
fn ProcessProposal(block: Block) -> ResponseProcessProposal {...}
fn RevertProposal(height: usize, round: usize) {...}
}

where

#![allow(unused)]
fn main() {
struct ResponseVerifyHeader {
    accept_header: bool,
    evidence: Vec<Evidence>
}
struct ResponseProcessProposal {
    accept_block: bool,
    evidence: Vec<Evidence>
}
}

Upon receiving a block header, every validator runs VerifyHeader(header, isValidator). The reason for why VerifyHeader is split from ProcessProposal is due to the later sections for Preprocess Proposal and Vote Extensions, where there may be application dependent data in the header that must be verified before accepting the header. If the returned ResponseVerifyHeader.accept_header is false, then the validator must precommit nil on this block, and reject all other precommits on this block. ResponseVerifyHeader.evidence is appended to the validators local EvidencePool.

Upon receiving an entire block proposal (in the current implementation, all "block parts"), every validator runs ProcessProposal(block). If the returned ResponseProcessProposal.accept_block is false, then the validator must precommit nil on this block, and reject all other precommits on this block. ResponseProcessProposal.evidence is appended to the validators local EvidencePool.

Once a validator knows that consensus has failed to be achieved for a given block, it must run RevertProposal(block.height, block.round), in order to signal to the application to revert any potentially mutative state changes it may have made. In Tenderdash, this occurs when incrementing rounds.

RFC: How do we handle the scenario where honest node A finalized on round x, and honest node B finalized on round x + 1? (e.g. when 2f precommits are publicly known, and a validator precommits themself but doesn't broadcast, but they increment rounds) Is this a real concern? The state root derived could change if everyone finalizes on round x+1, not round x, as the state machine can depend non-uniformly on timestamp.

The application is expected to cache the block data for later execution.

The isValidator flag is set according to whether the current node is a validator or a full node. This is intended to allow for beginning validator-dependent computation that will be included later in vote extensions. (An example of this is threshold decryptions of ciphertexts.)

DeliverTx rename to FinalizeBlock

After implementing ProcessProposal, txs no longer need to be delivered during the block execution phase. Instead, they are already in the state machine. Thus BeginBlock, DeliverTx, EndBlock can all be replaced with a single ABCI method for ExecuteBlock. Internally the application may still structure its method for executing the block as BeginBlock, DeliverTx, EndBlock. However, it is overly restrictive to enforce that the block be executed after it is finalized. There are multiple other, very reasonable pipelined execution models one can go for. So instead we suggest calling this succession of methods FinalizeBlock. We propose the following API

Replace the BeginBlock, DeliverTx, EndBlock ABCI methods with the following method

#![allow(unused)]
fn main() {
fn FinalizeBlock() -> ResponseFinalizeBlock
}

where ResponseFinalizeBlock has the following API, in terms of what already exists

#![allow(unused)]
fn main() {
struct ResponseFinalizeBlock {
    updates: ResponseEndBlock,
    tx_results: Vec<ResponseDeliverTx>
}
}

ResponseEndBlock should then be renamed to ConsensusUpdates and ResponseDeliverTx should be renamed to ResponseTx.

Vote Extensions

The Vote Extensions phase allow applications to force their validators to do more than just validate within consensus. This is done by allowing the application to add more data to their votes, in the final round of voting. (Namely the precommit) This additional application data will then appear in the block header.

First we discuss the API changes to the vote struct directly

#![allow(unused)]
fn main() {
fn ExtendVote(height: u64, round: u64) -> (UnsignedAppVoteData, SelfAuthenticatingAppData)
fn VerifyVoteExtension(signed_app_vote_data: Vec<u8>, self_authenticating_app_vote_data: Vec<u8>) -> bool
}

There are two types of data that the application can enforce validators to include with their vote. There is data that the app needs the validator to sign over in their vote, and there can be self-authenticating vote data. Self-authenticating here means that the application upon seeing these bytes, knows its valid, came from the validator and is non-malleable. We give an example of each type of vote data here, to make their roles clearer.

  • Unsigned app vote data: A use case of this is if you wanted validator backed oracles, where each validator independently signs some oracle data in their vote, and the median of these values is used on chain. Thus we leverage consensus' signing process for convenience, and use that same key to sign the oracle data.
  • Self-authenticating vote data: A use case of this is in threshold random beacons. Every validator produces a threshold beacon share. This threshold beacon share can be verified by any node in the network, given the share and the validators public key (which is not the same as its consensus public key). However, this decryption share will not make it into the subsequent block's header. They will be aggregated by the subsequent block proposer to get a single random beacon value that will appear in the subsequent block's header. Everyone can then verify that this aggregated value came from the requisite threshold of the validator set, without increasing the bandwidth for full nodes or light clients. To achieve this goal, the self-authenticating vote data cannot be signed over by the consensus key along with the rest of the vote, as that would require all full nodes & light clients to know this data in order to verify the vote.

The CanonicalVote struct will acommodate the UnsignedAppVoteData field by adding another string to its encoding, after the chain-id. This should not interfere with existing hardware signing integrations, as it does not affect the constant offset for the height and round, and the vote size does not have an explicit upper bound. (So adding this unsigned app vote data field is equivalent from the HSM's perspective as having a superlong chain-ID)

RFC: Please comment if you think it will be fine to have elongate the message the HSM signs, or if we need to explore pre-hashing the app vote data.

The flow of these methods is that when a validator has to precommit, Tenderdash will first produce a precommit canonical vote without the application vote data. It will then pass it to the application, which will return unsigned application vote data, and self authenticating application vote data. It will bundle the unsigned_application_vote_data into the canonical vote, and pass it to the HSM to sign. Finally it will package the self-authenticating app vote data, and the signed_vote_data together, into one final Vote struct to be passed around the network.

Changes to Prepare Proposal Phase

There are many use cases where the additional data from vote extensions can be batch optimized. This is mainly of interest when the votes include self-authenticating app vote data that be batched together, or the unsigned app vote data is the same across all votes. To allow for this, we change the PrepareProposal API to the following

#![allow(unused)]
fn main() {
fn PrepareProposal(Block, UnbatchedHeader) -> (BlockData, Header)
}

where UnbatchedHeader essentially contains a "RawCommit", the Header contains a batch-optimized commit and an additional "Application Data" field in its root. This will involve a number of changes to core data structures, which will be gone over in the ADR. The Unbatched header and rawcommit will never be broadcasted, they will be completely internal to consensus.

Inter-process communication (IPC) effects

For brevity in exposition above, we did not discuss the trade-offs that may occur in interprocess communication delays that these changs will introduce. These new ABCI methods add more locations where the application must communicate with the consensus engine. In most configurations, we expect that the consensus engine and the application will be either statically or dynamically linked, so all communication is a matter of at most adjusting the memory model the data is layed out within. This memory model conversion is typically considered negligible, as delay here is measured on the order of microseconds at most, whereas we face milisecond delays due to cryptography and network overheads. Thus we ignore the overhead in the case of linked libraries.

In the case where the consensus engine and the application are ran in separate processes, and thus communicate with a form of Inter-process communication (IPC), the delays can easily become on the order of miliseconds based upon the data sent. Thus its important to consider whats happening here. We go through this phase by phase.

Prepare proposal IPC overhead

This requires a round of IPC communication, where both directions are quite large. Namely the proposer communicating an entire block to the application. However, this can be mitigated by splitting up PrepareProposal into two distinct, async methods, one for the block IPC communication, and one for the Header IPC communication.

Then for chains where the block data does not depend on the header data, the block data IPC communication can proceed in parallel to the prior block's voting phase. (As a node can know whether or not its the leader in the next round)

Furthermore, this IPC communication is expected to be quite low relative to the amount of p2p gossip time it takes to send the block data around the network, so this is perhaps a premature concern until more sophisticated block gossip protocols are implemented.

Process Proposal IPC overhead

This phase changes the amount of time available for the consensus engine to deliver a block's data to the state machine. Before, the block data for block N would be delivered to the state machine upon receiving a commit for block N and then be executed. The state machine would respond after executing the txs and before prevoting. The time for block delivery from the consensus engine to the state machine after this change is the time of receiving block proposal N to the to time precommit on proposal N. It is expected that this difference is unimportant in practice, as this time is in parallel to one round of p2p communication for prevoting, which is expected to be significantly less than the time for the consensus engine to deliver a block to the state machine.

Vote Extension IPC overhead

This has a small amount of data, but does incur an IPC round trip delay. This IPC round trip delay is pretty negligible as compared the variance in vote gossip time. (the IPC delay is typically on the order of 10 microseconds)

Status

Proposed

Consequences

Positive

  • Enables a large number of new features for applications
  • Supports both immediate and delayed execution models
  • Allows application specific data from each validator
  • Allows for batch optimizations across txs, and votes

Negative

  • This is a breaking change to all existing ABCI clients, however the application should be able to have a thin wrapper to replicate existing ABCI behavior.

    • PrepareProposal - can be a no-op
    • Process Proposal - has to cache the block, but can otherwise be a no-op
    • Vote Extensions - can be a no-op
    • Finalize Block - Can black-box call BeginBlock, DeliverTx, EndBlock given the cached block data
  • Vote Extensions adds more complexity to core Tenderdash Data Structures

  • Allowing alternate alternate execution models will lead to a proliferation of new ways for applications to violate expected guarantees.

Neutral

  • IPC overhead considerations change, but mostly for the better

References

Reference for IPC delay constants: https://web.archive.org/web/2020/http://pages.cs.wisc.edu/~adityav/Evaluation_of_Inter_Process_Communication_Mechanisms.pdf

Short list of blocked features / scaling improvements with required ABCI++ Phases

FeaturePrepareProposalProcessProposalVote Extensions
Tx based signature aggregationX
SNARK proof of valid state transitionX
Validator provided authentication paths in stateless blockchainsX
Immediate ExecutionX
Simple soft forksX
Validator guaranteed IBC connection attemptsX
Validator based price oraclesX
Immediate Execution with increased time for block executionXXX
Threshold Encrypted txsXXX

RFC 014: Semantic Versioning

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 2021-11-19: Initial Draft
  • 2021-02-11: Migrate RFC to tendermint repo (Originally RFC 006)

Author(s)

  • Callum Waters @cmwaters

Context

We use versioning as an instrument to hold a set of promises to users and signal when such a set changes and how. In the conventional sense of a Go library, major versions signal that the public Go API’s have changed in a breaking way and thus require the users of such libraries to change their usage accordingly. Tenderdash is a bit different in that there are multiple users: application developers (both in-process and out-of-process), node operators, and external clients. More importantly, both how these users interact with Tenderdash and what's important to these users differs from how users interact and what they find important in a more conventional library.

This document attempts to encapsulate the discussions around versioning in Tenderdash and draws upon them to propose a guide to how Tenderdash uses versioning to make promises to its users.

For a versioning policy to make sense, we must also address the intended frequency of breaking changes. The strictest guarantees in the world will not help users if we plan to break them with every release.

Finally I would like to remark that this RFC only addresses the "what", as in what are the rules for versioning. The "how" of Tenderdash implementing the versioning rules we choose, will be addressed in a later RFC on Soft Upgrades.

Discussion

We first begin with a round up of the various users and a set of assumptions on what these users expect from Tenderdash in regards to versioning:

  1. Application Developers, those that use the ABCI to build applications on top of Tenderdash, are chiefly concerned with that API. Breaking changes will force developers to modify large portions of their codebase to accommodate for the changes. Some ABCI changes such as introducing priority for the mempool don't require any effort and can be lazily adopted whilst changes like ABCI++ may force applications to redesign their entire execution system. It's also worth considering that the API's for go developers differ to developers of other languages. The former here can use the entire Tenderdash library, most notably the local RPC methods, and so the team must be wary of all public Go API's.

  2. Node Operators, those running node infrastructure, are predominantly concerned with downtime, complexity and frequency of upgrading, and avoiding data loss. They may be also concerned about changes that may break the scripts and tooling they use to supervise their nodes.

  3. External Clients are those that perform any of the following:

    • consume the RPC endpoints of nodes like /block
    • subscribe to the event stream
    • make queries to the indexer

    This set are concerned with chain upgrades which will impact their ability to query state and block data as well as broadcast transactions. Examples include wallets and block explorers.

  4. IBC module and relayers. The developers of IBC and consumers of their software are concerned about changes that may affect a chain's ability to send arbitrary messages to another chain. Specifically, these users are affected by any breaking changes to the light client verification algorithm.

Although we present them here as having different concerns, in a broader sense these user groups share a concern for the end users of applications. A crucial principle guiding this RFC is that the ability for chains to provide continual service is more important than the actual upgrade burden put on the developers of these chains. This means some extra burden for application developers is tolerable if it minimizes or substantially reduces downtime for the end user.

Modes of Interprocess Communication

Tenderdash has two primary mechanisms to communicate with other processes: RPC and P2P. The division marks the boundary between the internal and external components of the network:

  • The P2P layer is used in all cases that nodes (of any type) need to communicate with one another.
  • The RPC interface is for any outside process that wants to communicate with a node.

The design principle here is that communication via RPC is to a trusted source and thus the RPC service prioritizes inspection rather than verification. The P2P interface is the primary medium for verification.

As an example, an in-browser light client would verify headers (and perhaps application state) via the p2p layer, and then pass along information on to the client via RPC (or potentially directly via a separate API).

The main exceptions to this are the IBC module and relayers, which are external to the node but also require verifiable data. Breaking changes to the light client verification path mean that all neighbouring chains that are connected will no longer be able to verify state transitions and thus pass messages back and forward.

Proposal

Tenderdash version labels will follow the syntax of Semantic Versions 2.0.0 with a major, minor and patch version. The version components will be interpreted according to these rules:

For the entire cycle of a major version in Tenderdash:

  • All blocks and state data in a blockchain can be queried. All headers can be verified even across minor version changes. Nodes can both block sync and state sync from genesis to the head of the chain.
  • Nodes in a network are able to communicate and perform BFT state machine replication so long as the agreed network version is the lowest of all nodes in a network. For example, nodes using version 1.5.x and 1.2.x can operate together so long as the network version is 1.2 or lower (but still within the 1.x range). This rule essentially captures the concept of network backwards compatibility.
  • Node RPC endpoints will remain compatible with existing external clients:
    • New endpoints may be added, but old endpoints may not be removed.
    • Old endpoints may be extended to add new request and response fields, but requests not using those fields must function as before the change.
  • Migrations should be automatic. Upgrading of one node can happen asynchronously with respect to other nodes (although agreement of a network-wide upgrade must still occur synchronously via consensus).

For the entire cycle of a minor version in Tenderdash:

  • Public Go API's, for example in node or abci packages will not change in a way that requires any consumer (not just application developers) to modify their code.
  • No breaking changes to the block protocol. This means that all block related data structures should not change in a way that breaks any of the hashes, the consensus engine or light client verification.
  • Upgrades between minor versions may not result in any downtime (i.e., no migrations are required), nor require any changes to the config files to continue with the existing behavior. A minor version upgrade will require only stopping the existing process, swapping the binary, and starting the new process.

A new patch version of Tenderdash will only contain bug fixes and updates that impact the security and stability of Tenderdash.

These guarantees will come into effect at release 1.0.

Status

Proposed

Consequences

Positive

  • Clearer communication of what versioning means to us and the effect they have on our users.

Negative

  • Can potentially incur greater engineering effort to uphold and follow these guarantees.

Neutral

References

RFC 015: ABCI++ TX Mutation

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 23-Feb-2022: Initial draft (@williambanfield).
  • 28-Feb-2022: Revised draft (@williambanfield).

Abstract

A previous version of the ABCI++ specification detailed a mechanism for proposers to replace transactions in the proposed block. This scheme required the proposer to construct new transactions and mark these new transactions as replacing other removed transactions. The specification was ambiguous as to how the replacement may be communicated to peer nodes. This RFC discusses issues with this mechanism and possible solutions.

Background

What is the proposed change?

A previous version of the ABCI++ specification proposed mechanisms for adding, removing, and replacing transactions in a proposed block. To replace a transaction, the application running ProcessProposal could mark a transaction as replaced by other application-supplied transactions by returning a new transaction marked with the ADDED flag setting the new_hashes field of the removed transaction to contain the list of transaction hashes that replace it. In that previous specification for ABCI++, the full use of the new_hashes field is left somewhat ambiguous. At present, these hashes are not gossiped and are not eventually included in the block to signal replacement to other nodes. The specification did indicate that the transactions specified in the new_hashes field will be removed from the mempool but it's not clear how peer nodes will learn about them.

What systems would be affected by adding transaction replacement?

The 'transaction' is a central building block of a Tenderdash blockchain, so adding a mechanism for transaction replacement would require changes to many aspects of Tenderdash.

The following is a rough list of the functionality that this mechanism would affect:

Transaction indexing

Tenderdash's indexer stores transactions and transaction results using the hash of the executed transaction as the key and the ABCI results and transaction bytes as the value.

To allow transaction replacement, the replaced transactions would need to stored as well in the indexer, likely as a mapping of original transaction to list of transaction hashes that replaced the original transaction.

Transaction inclusion proofs

The result of a transaction query includes a Merkle proof of the existence of the transaction in the block chain. This proof is built as a merkle tree of the hashes of all of the transactions in the block where the queried transaction was executed.

To allow transaction replacement, these proofs would need to be updated to prove that a replaced transaction was included by replacement in the block.

RPC-based transaction query parameters and results

Tenderdash's RPC allows clients to retrieve information about transactions via the /tx_search and /tx RPC endpoints.

RPC query results containing replaced transactions would need to be updated to include information on replaced transactions, either by returning results for all of the replaced transactions, or by including a response with just the hashes of the replaced transactions which clients could proceed to query individually.

Mempool transaction removal

Additional logic would need to be added to the Tenderdash mempool to clear out replaced transactions after each block is executed. Tenderdash currently removes executed transactions from the mempool, so this would be a pretty straightforward change.

Discussion

What value may be added to Tenderdash by introducing transaction replacement?

Transaction replacement would would enable applications to aggregate or disaggregate transactions.

For aggregation, a set of transactions that all related work, such as transferring tokens between the same two accounts, could be replaced with a single transaction, i.e. one that transfers a single sum from one account to the other. Applications that make frequent use of aggregation may be able to achieve a higher throughput. Aggregation would decrease the space occupied by a single client-submitted transaction in the block, allowing more client-submitted transactions to be executed per block.

For disaggregation, a very complex transaction could be split into multiple smaller transactions. This may be useful if an application wishes to perform more fine-grained indexing on intermediate parts of a multi-part transaction.

Drawbacks to transaction replacement

Transaction replacement would require updating and shimming many of the places that Tenderdash records and exposes information about executed transactions. While systems within Tenderdash could be updated to account for transaction replacement, such a system would leave new issues and rough edges.

No way of guaranteeing correct replacement

If a user issues a transaction to the network and the transaction is replaced, the user has no guarantee that the replacement was correct. For example, suppose a set of users issue transactions A, B, and C and they are all aggregated into a new transaction, D. There is nothing guaranteeing that D was constructed correctly from the inputs. The only way for users to ensure D is correct would be if D contained all of the information of its constituent transactions, in which case, nothing is really gained by the replacement.

Replacement transactions not signed by submitter

Abstractly, Tenderdash simply views transactions as a ball of bytes and therefore should be fine with replacing one for another. However, many applications require that transactions submitted to the chain be signed by some private key to authenticate and authorize the transaction. Replaced transactions could not be signed by the submitter, only by the application node. Therefore, any use of transaction replacement could not contain authorization from the submitter and would either need to grant application-submitted transactions power to perform application logic on behalf of a user without their consent.

Granting this power to application-submitted transactions would be very dangerous and therefore might not be of much value to application developers. Transaction replacement might only be really safe in the case of application-submitted transactions or for transactions that require no authorization. For such transactions, it's quite not quite clear what the utility of replacement is: the application can already generate any transactions that it wants. The fact that such a transaction was a replacement is not particularly relevant to participants in the chain since the application is merely replacing its own transactions.

New vector for censorship

Depending on the implementation, transaction replacement may allow a node signal to the rest of the chain that some transaction should no longer be considered for execution. Honest nodes will use the replacement mechanism to signal that a transaction has been aggregated. Malicious nodes will be granted a new vector for censoring transactions. There is no guarantee that a replaced transactions is actually executed at all. A malicious node could censor a transaction by simply listing it as replaced. Honest nodes seeing the replacement would flush the transaction from their mempool and not execute or propose it it in later blocks.

Transaction tracking implementations

This section discusses possible ways to flesh out the implementation of transaction replacement. Specifically, this section proposes a few alternative ways that Tenderdash blockchains could track and store transaction replacements.

Include transaction replacements in the block

One option to track transaction replacement is to include information on the transaction replacement within the block. An additional structure may be added the block of the following form:

message Block {
...
  repeated Replacement replacements = 5;
}

message Replacement {
  bytes          included_tx_key   = 1;
  repeated bytes replaced_txs_keys = 2;
}

Applications executing PrepareProposal would return the list of replacements and Tenderdash would include an encoding of these replacements in the block that is gossiped and committed.

Tenderdash's transaction indexing would include a new mapping for each replaced transaction key to the committed transaction. Transaction inclusion proofs would be updated to include these additional new transaction keys in the Merkle tree and queries for transaction hashes that were replaced would return information indicating that the transaction was replaced along with the hash of the transaction that replaced it.

Block validation of gossiped blocks would be updated to check that each of the included_txs_key matches the hash of some transaction in the proposed block.

Implementing the changes described in this section would allow Tenderdash to gossip and index transaction replacements as part of block propagation. These changes would still require the application to certify that the replacements were valid. This validation may be performed in one of two ways:

  1. Applications optimistically trust that the proposer performed a legitimate replacement.

In this validation scheme, applications would not verify that the substitution is valid during consensus and instead simply trust that the proposer is correct. This would have the drawback of allowing a malicious proposer to remove transactions it did not want executed.

  1. Applications completely validate transaction replacement.

In this validation scheme, applications that allow replacement would check that each listed replaced transaction was correctly reflected in the replacement transaction. In order to perform such validation, the node would need to have the replaced transactions locally. This could be accomplished one of a few ways: by querying the mempool, by adding an additional p2p gossip channel for transaction replacements, or by including the replaced transactions in the block. Replacement validation via mempool querying would require the node to have received all of the replaced transactions in the mempool which is far from guaranteed. Adding an additional gossip channel would make gossiping replaced transactions a requirement for consensus to proceed, since all nodes would need to receive all replacement messages before considering a block valid. Finally, including replaced transactions in the block seems to obviate any benefit gained from performing a transaction replacement since the replaced transaction and the original transactions would now both appear in the block.

Application defined transaction replacement

An additional option for allowing transaction replacement is to leave it entirely as a responsibility of the application. The PrepareProposal ABCI++ call allows for applications to add new transactions to a proposed block. Applications that wished to implement a transaction replacement mechanism would be free to do so without the newly defined new_hashes field. Applications wishing to implement transaction replacement would add the aggregated transactions in the PrepareProposal response, and include one additional bookkeeping transaction that listed all of the replacements, with a similar scheme to the new_hashes field described in ABCI++. This new bookkeeping transaction could be used by the application to determine which transactions to clear from the mempool in future calls to CheckTx.

The meaning of any transaction in the block is completely opaque to Tenderdash, so applications performing this style of replacement would not be able to have the replacement reflected in any most of Tenderdash's transaction tracking mechanisms, such as transaction indexing and the /tx endpoint.

Application defined Tx Keys

Tenderdash currently uses cryptographic hashes, SHA256, as a key for each transaction. As noted in the section on systems that would require changing, this key is used to identify the transaction in the mempool, in the indexer, and within the RPC system.

An alternative approach to allowing ProcessProposal to specify a set of transaction replacements would be instead to allow the application to specify an additional key or set of keys for each transaction during ProcessProposal. This new secondary_keys set would be included in the block and therefore gossiped during block propagation. Additional RPC endpoints could be exposed to query by the application-defined keys.

Applications wishing to implement replacement would leverage this new field by providing the replaced transaction hashes as the secondary_keys and checking their validity during ProcessProposal. During RecheckTx the application would then be responsible for clearing out transactions that matched the secondary_keys.

It is worth noting that something like this would be possible without secondary_keys. An application wishing to implement a system like this one could define a replacement transaction, as discussed in the section on application-defined transaction replacement, and use a custom ABCI event type to communicate that the replacement should be indexed within Tenderdash's ABCI event indexing.

Complexity to value-add tradeoff

It is worth remarking that adding a system like this may introduce a decent amount of new complexity into Tenderdash. An approach that leaves much of the replacement logic to Tenderdash would require altering the core transaction indexing and querying data. In many of the cases listed, a system for transaction replacement is possible without explicitly defining it as part of PrepareProposal. Since applications can now add transactions during PrepareProposal they can and should leverage this functionality to include additional bookkeeping transactions in the block. It may be worth encouraging applications to discover new and interesting ways to leverage this power instead of immediately solving the problem for them.

References

RFC 016: Node Architecture

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • April 8, 2022: Initial draft (@cmwaters)
  • April 15, 2022: Incorporation of feedback

Abstract

The node package is the entry point into the Tenderdash codebase, used both by the command line and programatically to create the nodes that make up a network. The package has suffered the most from the evolution of the codebase, becoming bloated as developers clipped on their bits of code here and there to get whatever feature they wanted working.

The decisions made at the node level have the biggest impact to simplifying the protocols within them, unlocking better internal designs and making Tenderdash more intuitive to use and easier to understand from the outside. Work, in minor increments, has already begun on this section of the codebase. This document exists to spark forth the necessary discourse in a few related areas that will help the team to converge on the long term makeup of the node.

Discussion

The following is a list of points of discussion around the architecture of the node:

Dependency Tree

The node object is currently stuffed with every component that possibly exists within Tenderdash. In the constructor, all objects are built and interlaid with one another in some awkward dance. My guiding principle is that the node should only be made up of the components that it wants to have direct control of throughout its life. The node is a service which currently has the purpose of starting other services up in a particular order and stopping them all when commanded to do so. However, there are many services which are not direct dependents i.e. the mempool and evidence services should only be working when the consensus service is running. I propose to form more of a hierarchical structure of dependents which forces us to be clear about the relations that one component has to the other. More concretely, I propose the following dependency tree:

node dependency tree

Many of the further discussion topics circle back to this representation of the node.

It's also important to distinguish two dimensions which may require different characteristics of the architecture. There is the starting and stopping of services and their general lifecycle management. What is the correct order of operations to starting a node for example. Then there is the question of the needs of the service during actual operation. Then there is the question of what resources each service needs access to during its operation. Some need to publish events, others need access to data stores, and so forth.

An alternative model and one that perhaps better suits the latter of these dimensions is the notion of an internal message passing system. Either the events bus or p2p layer could serve as a viable transport. This would essentially allow all services to communicate with any other service and could perhaps provide a solution to the coordination problem (presented below) without a centralized coordinator. The other main advantage is that such a system would be more robust to disruptions and changes to the code which may make a hierarchical structure quickly outdated and suboptimal. The addition of message routing is an added complexity to implement, will increase the degree of asynchronicity in the system and may make it harder to debug problems that are across multiple services.

Coordination of State Advancing Mechanisms

Advancement of state in Tenderdash is simply defined in heights: If the node is at height n, how does it get to height n + 1 and so on. Based on this definition we have three components that help a node to advance in height: consensus, statesync and blocksync. The way these components behave currently is very tightly coupled to one another with references passed back and forth. My guiding principle is that each of these should be able to operate completely independently of each other, e.g. a node should be able to run solely blocksync indefinitely. There have been several ideas suggested towards improving this flow. I've been leaning strongly towards a centralized system, whereby an orchestrator (in this case the node) decides what services to start and stop. In a decentralized message passing system, individual services make their decision based upon a "global" shared state i.e. if my height is less that 10 below the average peer height, I as consensus, should stop (knowing that blocksync has the same condition for starting). As the example illustrates, each mechanism will still need to be aware of the presence of other mechanisms.

Both centralized and decentralized systems rely on the communication of the nodes current height and a judgement on the height of the head of the chain. The latter, working out the head of the chain, is quite a difficult challenge as their is nothing preventing the node from acting maliciously and providing a different height. Currently both blocksync, consensus (and to a certain degree statesync), have parallel systems where peers communicate their height. This could be streamlined with the consensus (or even the p2p layer), broadcasting peer heights and either the node or the other state advancing mechanisms acting accordingly.

Currently, when a node starts, it turns on every service that it is attached to. This means that while a node is syncing up by requesting blocks, it is also receiving transactions and votes, as well as snapshot and block requests. This is a needless use of bandwidth. An implementation of an orchestrator, regardless of whether the system is heirachical or not, should look to be able to open and close channels dynamically and effectively broadcast which services it is running. Integrating this with service discovery may also lead to a better serivce to peers.

The orchestrator allows for some deal of variablity in how a node is constructed. Does it just run blocksync, shadowing the head of the chain and be highly available for querying. Does it rely on state sync at all? An important question that arises from this dynamicism is we ideally want to encourage nodes to provide as much of their resources as possible so that their is a healthy amount of providers to consumers. Do we make all services compulsory or allow for them to be disabled? Arguably it's possible that a user forks the codebase and rips out the blocksync code because they want to reduce bandwidth so this is more a question of how easy do we want to make this for users.

Block Executor

The block executor is an important component that is currently used by both consensus and blocksync to execute transactions and update application state. Principally, I think it should be the only component that can write (and possibly even read) the block and state stores, and we should clean up other direct dependencies on the storage engine if we can. This would mean:

  • The reactors Consensus, BlockSync and StateSync should all import the executor for advancing state ie. ApplyBlock and BootstrapState.
  • Pruning should also be a concern of the block executor as well as FinalizeBlock and Commit. This can simplify consensus to focus just on the consensus part.

The Interprocess communication systems: RPC, P2P, ABCI, and Events

The schematic supplied above shows the relations between the different services, the node, the block executor, and the storage layer. Represented as colored dots are the components responsible for different roles of interprocess communication (IPC). These components permeate throughout the code base, seeping into most services. What can provide powerful functionality on one hand can also become a twisted vine, creating messy corner cases and convoluting the protocols themselves. A lot of the thinking around how we want our IPC systens to function has been summarised in this RFC. In this section, I'd like to focus the reader on the relation between the IPC and the node structure. An issue that has frequently risen is that the RPC has control of the components where it strikes me as being more logical for the component to dictate the information that is emitted/available and the knobs it wishes to expose. The RPC is also inextricably tied to the node instance and has situations where it is passed pointers directly to the storage engine and other components.

I am currently convinced of the approach that the p2p layer takes and would like to see other IPC components follow suit. This would mean that the RPC and events system would be constructed in the node yet would pass the adequate methods to register endpoints and topics to the sub components. For example,

// Methods from the RPC and event bus that would be passed into the constructor of components like "consensus"
// NOTE: This is a hypothetical construction to convey the idea. An actual implementation may differ.
func RegisterRoute(path string, handler func(http.ResponseWriter, *http.Request))

func RegisterTopic(name string) EventPublisher

type EventPublisher func (context.Context, types.EventData, []abci.Event)

This would give the components control to the information they want to expose and keep all relevant logic within that package. It accomodates more to a dynamic system where services can switch on and off. Each component would also receive access to the logger and metrics system for introspection and debuggability.

IPC Rubric

I'd like to aim to reach a state where we as a team have either an implicit or explicit rubric which can determine, in the event of some new need to communicate information, what tool it should use for doing this. In the case of inter node communication, this is obviously the p2p stack (with perhaps the exception of the light client). Metrics and logging also have clear usage patterns. RPC and the events system are less clear. The RPC is used for debugging data and fine tuned operator control as it is for general public querying and transaction submission. The RPC is also known to have been plumbed back into the application for historical queries. The events system, similarly, is used for consuming transaction events as it is for the testing of consensus state transitions.

Principally, I think we should look to change our language away from what the actual transport is and more towards what it's being used for and to whom. We call it a peer to peer layer and not the underlying tcp connection. In the same way, we should look to split RPC into an operator interface (RPC Internal), a public interface (RPC External) and a bidirectional ABCI.

Seperation of consumers and suppliers

When a service such as blocksync is turned on, it automatically begins requesting blocks to verify and apply them as it also tries to serve them to other peers catching up. We should look to distinguish these two aspects: supplying of information and consuming of information in many of these components. More concretely, I'd suggest:

  • The blocksync and statesync service, i.e. supplying information for those trying to catch up should only start running once a node has caught up i.e. after running the blocksync and/or state sync processes
  • The blocksync and state sync processes have defined termination clauses that inform the orchestrator when they are done and where they finished.
    • One way of achieving this would be that every process both passes and returns the State object
    • In some cases, a node may specify that it wants to run blocksync indefinitely.
  • The mempool should also indicate whether it wants to receive transactions or to send them only (one-directional mempool)
  • Similarly, the light client itself only requests information whereas the light client service (currently part of state sync) can do both.
  • This distinction needs to be communicated in the p2p layer handshake itself but should also be changeable over the lifespan of the connection.

RFC 017: ABCI++ Vote Extension Propagation

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 11-Apr-2022: Initial draft (@sergio-mena).
  • 15-Apr-2022: Addressed initial comments. First complete version (@sergio-mena).
  • 09-May-2022: Addressed all outstanding comments.

Abstract

According to the ABCI++ specification (as of 11-Apr-2022), a validator MUST provide a signed vote extension for each non-nil precommit vote of height h that it uses to propose a block in height h+1. When a validator is up to date, this is easy to do, but when a validator needs to catch up this is far from trivial as this data cannot be retrieved from the blockchain.

This RFC presents and compares the different options to address this problem, which have been proposed in discussions by the original upstream team.

Document Structure

The RFC is structured as follows. In the Background section, subsections Problem Description and Cases to Address explain the problem at hand from a high level perspective, i.e., abstracting away from the current Tenderdash implementation. In contrast, subsection Current Catch-up Mechanisms delves into the details of the current Tenderdash code.

In the Discussion section, subsection Solutions Proposed is also worded abstracting away from implementation details, whilst subsections Feasibility of the Proposed Solutions and Current Limitations and Possible Implementations analize the viability of one of the proposed solutions in the context of Tenderdash's architecture based on reactors. Finally, Formalization Work briefly discusses the work still needed demonstrate the correctness of the chosen solution.

The high level subsections are aimed at readers who are familiar with consensus algorithms, in particular with the one described in the Tendermint white paper, but who are not necessarily acquainted with the details of the Tenderdash codebase. The other subsections, which go into implementation details, are best understood by engineers with deep knowledge of the implementation of Tenderdash's blocksync and consensus reactors.

Background

Basic Definitions

This document assumes that all validators have equal voting power for the sake of simplicity. This is done without loss of generality.

There are two types of votes in Tenderdash: prevotes and precommits. Votes can be nil or refer to a proposed block. This RFC focuses on precommits, also known as precommit votes. In this document we sometimes call them simply votes.

Validators send precommit votes to their peer nodes in precommit messages. According to the ABCI++ specification, a precommit message MUST also contain a vote extension. This mandatory vote extension can be empty, but MUST be signed with the same key as the precommit vote (i.e., the sending validator's). Nevertheless, the vote extension is signed independently from the vote, so a vote can be separated from its extension. The reason for vote extensions to be mandatory in precommit messages is that, otherwise, a (malicious) node can omit a vote extension while still providing/forwarding/sending the corresponding precommit vote.

The validator set at height h is denoted valseth. A commit for height h consists of more than 2nh/3 precommit votes voting for a block b, where nh denotes the size of valseth. A commit does not contain nil precommit votes, and all votes in it refer to the same block. An extended commit is a commit where every precommit vote has its respective vote extension attached.

Problem Description

In the version of ABCI present up to Tendermint v0.35, for any height h, a validator v MUST have the decided block b and a commit for height h in order to decide at height h. Then, v just needs a commit for height h to propose at height h+1, in the rounds of h+1 where v is a proposer.

In ABCI++, the information that a validator v MUST have to be able to decide in h does not change with respect to pre-existing ABCI: the decided block b and a commit for h. In contrast, for proposing in h+1, a commit for h is not enough: v MUST now have an extended commit.

When a validator takes an active part in consensus at height h, it has all the data it needs in memory, in its consensus state, to decide on h and propose in h+1. Things are not so easy in the cases when v cannot take part in consensus because it is late (e.g., it falls behind, it crashes and recovers, or it just starts after the others). If v does not take part, it cannot actively gather precommit messages (which include vote extensions) in order to decide. Before ABCI++, this was not a problem: full nodes are supposed to persist past blocks in the block store, so other nodes would realise that v is late and send it the missing decided block at height h and the corresponding commit (kept in block h+1) so that v can catch up. However, we cannot apply this catch-up technique for ABCI++, as the vote extensions, which are part of the needed extended commit are not part of the blockchain.

Cases to Address

Before we tackle the description of the possible cases we need to address, let us describe the following incremental improvement to the ABCI++ logic. Upon decision, a full node persists (e.g., in the block store) the extended commit that allowed the node to decide. For the moment, let us assume the node only needs to keep its most recent extended commit, and MAY remove any older extended commits from persistent storage. This improvement is so obvious that all solutions described in the Discussion section use it as a building block. Moreover, it completely addresses by itself some of the cases described in this subsection.

We now describe the cases (i.e. possible runs of the system) that have been raised in different discussions and need to be addressed. They are (roughly) ordered from easiest to hardest to deal with.

  • (a) Happy path: all validators advance together, no crash.

    This case is included for completeness. All validators have taken part in height h. Even if some of them did not manage to send a precommit message for the decided block, they all receive enough precommit messages to be able to decide. As vote extensions are mandatory in precommit messages, every validator v trivially has all the information, namely the decided block and the extended commit, needed to propose in height h+1 for the rounds in which v is the proposer.

    No problem to solve here.

  • (b) All validators advance together, then all crash at the same height.

    This case has been raised in some discussions, the main concern being whether the vote extensions for the previous height would be lost across the network. With the improvement described above, namely persisting the latest extended commit at decision time, this case is solved. When a crashed validator recovers, it recovers the last extended commit from persistent storage and handshakes with the Application. If need be, it also reconstructs messages for the unfinished height (including all precommits received) from the WAL. Then, the validator can resume where it was at the time of the crash. Thus, as extensions are persisted, either in the WAL (in the form of received precommit messages), or in the latest extended commit, the only way that vote extensions needed to start the next height could be lost forever would be if all validators crashed and never recovered (e.g. disk corruption). Since a correct node MUST eventually recover, this violates Tenderdash's assumption of more than 2nh/3 correct validators for every height h.

    No problem to solve here.

  • (c) Lagging majority.

    Let us assume the validator set does not change between h and h+1. It is not possible by the nature of the Tendermint algorithm, which requires more than 2nh/3 precommit votes for some round of height h in order to make progress. So, only up to nh/3 validators can lag behind.

    On the other hand, for the case where there are changes to the validator set between h and h+1 please see case (d) below, where the extreme case is discussed.

  • (d) Validator set changes completely between h and h+1.

    If sets valseth and valseth+1 are disjoint, more than 2nh/3 of validators in height h should have actively participated in conensus in h. So, as of height h, only a minority of validators in h can be lagging behind, although they could all lag behind from h+1 on, as they are no longer validators, only full nodes. This situation falls under the assumptions of case (h) below.

    As for validators in valseth+1, as they were not validators as of height h, they could all be lagging behind by that time. However, by the time h finishes and h+1 begins, the chain will halt until more than 2nh+1/3 of them have caught up and started consensus at height h+1. If set valseth+1 does not change in h+2 and subsequent heights, only up to nh+1/3 validators will be able to lag behind. Thus, we have converted this case into case (h) below.

  • (e) Enough validators crash to block the rest.

    In this case, blockchain progress halts, i.e. surviving full nodes keep increasing rounds indefinitely, until some of the crashed validators are able to recover. Those validators that recover first will handshake with the Application and recover at the height they crashed, which is still the same the nodes that did not crash are stuck in, so they don't need to catch up. Further, they had persisted the extended commit for the previous height. Nothing to solve.

    For those validators recovering later, we are in case (h) below.

  • (f) Some validators crash, but not enough to block progress.

    When the correct processes that crashed recover, they handshake with the Application and resume at the height they were at when they crashed. As the blockchain did not stop making progress, the recovered processes are likely to have fallen behind with respect to the progressing majority.

    At this point, the recovered processes are in case (h) below.

  • (g) A new full node starts.

    The reasoning here also applies to the case when more than one full node are starting. When the full node starts from scratch, it has no state (its current height is 0). Ignoring statesync for the time being, the node just needs to catch up by applying past blocks one by one (after verifying them).

    Thus, the node is in case (h) below.

  • (h) Advancing majority, lagging minority

    In this case, some nodes are late. More precisely, at the present time, a set of full nodes, denoted Lhp, are falling behind (e.g., temporary disconnection or network partition, memory thrashing, crashes, new nodes) an arbitrary number of heights: between hs and hp, where hs < hp, and hp is the highest height any correct full node has reached so far.

    The correct full nodes that reached hp were able to decide for hp-1. Therefore, less than nhp-1/3 validators of hp-1 can be part of Lhp, since enough up-to-date validators needed to actively participate in consensus for hp-1.

    Since, at the present time, no node in Lhp took part in any consensus between hs and hp-1, the reasoning above can be extended to validator set changes between hs and hp-1. This results in the following restriction on the full nodes that can be part of Lhp.

    • ∀ h, where hs ≤ h < hp, | valseth ∩ Lhp | < nh/3

    If this property does not hold for a particular height h, where hs ≤ h < hp, Tenderdash could not have progressed beyond h and therefore no full node could have reached hp (a contradiction).

    These lagging nodes in Lhp need to catch up. They have to obtain the information needed to make progress from other nodes. For each height h between hs and hp-2, this includes the decided block for h, and the precommit votes also for deciding h (which can be extracted from the block at height h+1).

    At a given height hc (where possibly hc << hp), a full node in Lhp will consider itself caught up, based on the (maybe out of date) information it is getting from its peers. Then, the node needs to be ready to propose at height hc+1, which requires having received the vote extensions for hc. As the vote extensions are not stored in the blocks, and it is difficult to have strong guarantees on when a late node considers itself caught up, providing the late node with the right vote extensions for the right height poses a problem.

At this point, we have described and compared all cases raised in discussions leading up to this RFC. The list above aims at being exhaustive. The analysis of each case included above makes all of them converge into case (h).

Current Catch-up Mechanisms

We now briefly describe the current catch-up mechanisms in the reactors concerned in Tenderdash.

Statesync

Full nodes optionally run statesync just after starting, when they start from scratch. If statesync succeeds, an Application snapshot is installed, and Tenderdash jumps from height 0 directly to the height the Application snapshop represents, without applying the block of any previous height. Some light blocks are received and stored in the block store for running light-client verification of all the skipped blocks. Light blocks are incomplete blocks, typically containing the header and the canonical commit but, e.g., no transactions. They are stored in the block store as "signed headers".

The statesync reactor is not really relevant for solving the problem discussed in this RFC. We will nevertheless mention it when needed; in particular, to understand some corner cases.

Blocksync

The blocksync reactor kicks in after start up or recovery (and, optionally, after statesync is done) and sends the following messages to its peers:

  • StatusRequest to query the height its peers are currently at, and
  • BlockRequest, asking for blocks of heights the local node is missing.

Using BlockResponse messages received from peers, the blocksync reactor validates each received block using the block of the following height, saves the block in the block store, and sends the block to the Application for execution.

If blocksync has validated and applied the block for the height previous to the highest seen in a StatusResponse message, or if no progress has been made after a timeout, the node considers itself as caught up and switches to the consensus reactor.

Consensus Reactor

The consensus reactor runs the full Tendermint algorithm. For a validator this means it has to propose blocks, and send/receive prevote/precommit messages, as mandated by Tenderdash, before it can decide and move on to the next height.

If a full node that is running the consensus reactor falls behind at height h, when a peer node realises this it will retrieve the canonical commit of h+1 from the block store, and convert it into a set of precommit votes and will send those to the late node.

Discussion

Solutions Proposed

These are the solutions proposed in discussions leading up to this RFC.

  • Solution 0. Vote extensions are made best effort in the specification.

    This is the simplest solution, considered as a way to provide vote extensions in a simple enough way so that it can be part of v0.36. It consists in changing the specification so as to not require that precommit votes used upon PrepareProposal contain their corresponding vote extensions. In other words, we render vote extensions optional. There are strong implications stemming from such a relaxation of the original specification.

    • As a vote extension is signed separately from the vote it is extending, an intermediate node can now remove (i.e., censor) vote extensions from precommit messages at will.
    • Further, there is no point anymore in the spec requiring the Application to accept a vote extension passed via VerifyVoteExtension to consider a precommit message valid in its entirety. Remember this behavior of VerifyVoteExtension is adding a constraint to Tenderdash's conditions for liveness. In this situation, it is better and simpler to just drop the vote extension rejected by the Application via VerifyVoteExtension, but still consider the precommit vote itself valid as long as its signature verifies.
  • Solution 1. Include vote extensions in the blockchain.

    Another obvious solution, which has somehow been considered in the past, is to include the vote extensions and their signatures in the blockchain. The blockchain would thus include the extended commit, rather than a regular commit, as the structure to be canonicalized in the next block. With this solution, the current mechanisms implemented both in the blocksync and consensus reactors would still be correct, as all the information a node needs to catch up, and to start proposing when it considers itself as caught-up, can now be recovered from past blocks saved in the block store.

    This solution has two main drawbacks.

    • As the block format must change, upgrading a chain requires a hard fork. Furthermore, all existing light client implementations will stop working until they are upgraded to deal with the new format (e.g., how certain hashes calculated and/or how certain signatures are checked). For instance, let us consider IBC, which relies on light clients. An IBC connection between two chains will be broken if only one chain upgrades.
    • The extra information (i.e., the vote extensions) that is now kept in the blockchain is not really needed at every height for a late node to catch up.
      • This information is only needed to be able to propose at the height the validator considers itself as caught-up. If a validator is indeed late for height h, it is useless (although correct) for it to call PrepareProposal, or ExtendVote, since the block is already decided.
      • Moreover, some use cases require pretty sizeable vote extensions, which would result in an important waste of space in the blockchain.
  • Solution 2. Skip propose step in Tendermint algorithm.

    This solution consists in modifying the Tendermint algorithm to skip the send proposal step in heights where the node does not have the required vote extensions to populate the call to PrepareProposal. The main idea behind this is that it should only happen when the validator is late and, therefore, up-to-date validators have already proposed (and decided) for that height. A small variation of this solution is, rather than skipping the send proposal step, the validator sends a special empty or bottom (⊥) proposal to signal other nodes that it is not ready to propose at (any round of) the current height.

    The appeal of this solution is its simplicity. A possible implementation does not need to extend the data structures, or change the current catch-up mechanisms implemented in the blocksync or in the consensus reactor. When we lack the needed information (vote extensions), we simply rely on another correct validator to propose a valid block in other rounds of the current height.

    However, this solution can be attacked by a byzantine node in the network in the following way. Let us consider the following scenario:

    • all validators in valseth send out precommit messages, with vote extensions, for height h, round 0, roughly at the same time,
    • all those precommit messages contain non-nil precommit votes, which vote for block b
    • all those precommit messages sent in height h, round 0, and all messages sent in height h, round r > 0 get delayed indefinitely, so,
    • all validators in valseth keep waiting for enough precommit messages for height h, round 0, needed for deciding in height h
    • an intermediate (malicious) full node m manages to receive block b, and gather more than 2nh/3 precommit messages for height h, round 0,
    • one way or another, the solution should have either (a) a mechanism for a full node to tell another full node it is late, or (b) a mechanism for a full node to conclude it is late based on other full nodes' messages; any of these mechanisms should, at the very least, require the late node receiving the decided block and a commit (not necessarily an extended commit) for h,
    • node m uses the gathered precommit messages to build a commit for height h, round 0,
    • in order to convince full nodes that they are late, node m either (a) tells them they are late, or (b) shows them it (i.e. m) is ahead, by sending them block b, along with the commit for height h, round 0,
    • all full nodes conclude they are late from m's behavior, and use block b and the commit for height h, round 0, to decide on height h, and proceed to height h+1.

    At this point, all full nodes, including all validators in valseth+1, have advanced to height h+1 believing they are late, and so, expecting the hypothetical leading majority of validators in valseth+1 to propose for h+1. As a result, the blockhain grinds to a halt. A (rather complex) ad-hoc mechanism would need to be carried out by node operators to roll back all validators to the precommit step of height h, round r, so that they can regenerate vote extensions (remember vote extensions are non-deterministic) and continue execution.

  • Solution 3. Require extended commits to be available at switching time.

    This one is more involved than all previous solutions, and builds on an idea present in Solution 2: vote extensions are actually not needed for Tenderdash to make progress as long as the validator is certain it is late.

    We define two modes. The first is denoted catch-up mode, and Tenderdash only calls FinalizeBlock for each height when in this mode. The second is denoted consensus mode, in which the validator considers itself up to date and fully participates in consensus and calls PrepareProposal/ProcessProposal, ExtendVote, and VerifyVoteExtension, before calling FinalizeBlock.

    The catch-up mode does not need vote extension information to make progress, as all it needs is the decided block at each height to call FinalizeBlock and keep the state-machine replication making progress. The consensus mode, on the other hand, does need vote extension information when starting every height.

    Validators are in consensus mode by default. When a validator in consensus mode falls behind for whatever reason, e.g. cases (b), (d), (e), (f), (g), or (h) above, we introduce the following key safety property:

    • for every height hp, a full node f in hp refuses to switch to catch-up mode until there exists a height h' such that:
      • p has received and (light-client) verified the blocks of all heights h, where hp ≤ h ≤ h'
      • it has received an extended commit for h' and has verified:
        • the precommit vote signatures in the extended commit
        • the vote extension signatures in the extended commit: each is signed with the same key as the precommit vote it extends

    If the condition above holds for hp, namely receiving a valid sequence of blocks in the f's future, and an extended commit corresponding to the last block in the sequence, then node f:

    • switches to catch-up mode,
    • applies all blocks between hp and h' (calling FinalizeBlock only), and
    • switches back to consensus mode using the extended commit for h' to propose in the rounds of h' + 1 where it is the proposer.

    This mechanism, together with the invariant it uses, ensures that the node cannot be attacked by being fed a block without extensions to make it believe it is late, in a similar way as explained for Solution 2.

Feasibility of the Proposed Solutions

Solution 0, besides the drawbacks described in the previous section, provides guarantees that are weaker than the rest. The Application does not have the assurance that more than 2nh/3 vote extensions will always be available when calling PrepareProposal at height h+1. This level of guarantees is probably not strong enough for vote extensions to be useful for some important use cases that motivated them in the first place, e.g., encrypted mempool transactions.

Solution 1, while being simple in that the changes needed in the current Tenderdash codebase would be rather small, is changing the block format, and would therefore require all blockchains using Tendermint v0.35 or earlier to hard-fork when upgrading to v0.36.

Since Solution 2 can be attacked, one might prefer Solution 3, even if it is more involved to implement. Further, we must elaborate on how we can turn Solution 3, described in abstract terms in the previous section, into a concrete implementation compatible with the current Tenderdash codebase.

Current Limitations and Possible Implementations

The main limitations affecting the current version of Tenderdash are the following.

  • The current version of the blocksync reactor does not use the full light client verification algorithm to validate blocks coming from other peers.
  • The code being structured into the blocksync and consensus reactors, only switching from the blocksync reactor to the consensus reactor is supported; switching in the opposite direction is not supported. Alternatively, the consensus reactor could have a mechanism allowing a late node to catch up by skipping calls to PrepareProposal/ProcessProposal, and ExtendVote/VerifyVoteExtension and only calling FinalizeBlock for each height. Such a mechanism does not exist at the time of writing this RFC.

The blocksync reactor featuring light client verification is being actively worked on (tentatively for v0.37). So it is best if this RFC does not try to delve into that problem, but just makes sure its outcomes are compatible with that effort.

In subsection Cases to Address, we concluded that we can focus on solving case (h) in theoretical terms. However, as the current Tenderdash version does not yet support switching back to blocksync once a node has switched to consensus, we need to split case (h) into two cases. When a full node needs to catch up...

  • (h.1) ... it has not switched yet from the blocksync reactor to the consensus reactor, or

  • (h.2) ... it has already switched to the consensus reactor.

This is important in order to discuss the different possible implementations.

Base Implementation: Persist and Propagate Extended Commit History

In order to circumvent the fact that we cannot switch from the consensus reactor back to blocksync, rather than just keeping the few most recent extended commits, nodes will need to keep and gossip a backlog of extended commits so that the consensus reactor can still propose and decide in out-of-date heights (even if those proposals will be useless).

The base implementation - for which an experimental patch exists - consists in the conservative approach of persisting in the block store all extended commits for which we have also stored the full block. Currently, when statesync is run at startup, it saves light blocks. This base implementation does not seek to receive or persist extended commits for those light blocks as they would not be of any use.

Then, we modify the blocksync reactor so that peers always send requested full blocks together with the corresponding extended commit in the BlockResponse messages. This guarantees that the block store being reconstructed by blocksync has the same information as that of peers that are up to date (at least starting from the latest snapshot applied by statesync before starting blocksync). Thus, blocksync has all the data it requires to switch to the consensus reactor, as long as one of the following exit conditions are met:

  • The node is still at height 0 (where no commit or extended commit is needed)
  • The node has processed at least 1 block in blocksync

The second condition is needed in case the node has installed an Application snapshot during statesync. If that is the case, at the time blocksync starts, the block store only has the data statesync has saved: light blocks, and no extended commits. Hence we need to blocksync at least one block from another node, which will be sent with its corresponding extended commit, before we can switch to consensus.

As a side note, a chain might be started at a height hi > 0, all other heights h < hi being non-existent. In this case, the chain is still considered to be at height 0 before block hi is applied, so the first condition above allows the node to switch to consensus even if blocksync has not processed any block (which is always the case if all nodes are starting from scratch).

When a validator falls behind while having already switched to the consensus reactor, a peer node can simply retrieve the extended commit for the required height from the block store and reconstruct a set of precommit votes together with their extensions and send them in the form of precommit messages to the validator falling behind, regardless of whether the peer node holds the extended commit because it actually participated in that consensus and thus received the precommit messages, or it received the extended commit via a BlockResponse message while running blocksync.

This solution requires a few changes to the consensus reactor:

  • upon saving the block for a given height in the block store at decision time, save the corresponding extended commit as well
  • in the catch-up mechanism, when a node realizes that another peer is more than 2 heights behind, it uses the extended commit (rather than the canoncial commit as done previously) to reconstruct the precommit votes with their corresponding extensions

The changes to the blocksync reactor are more substantial:

  • the BlockResponse message is extended to include the extended commit of the same height as the block included in the response (just as they are stored in the block store)
  • structure bpRequester is likewise extended to hold the received extended commits coming in BlockResponse messages
  • method PeekTwoBlocks is modified to also return the extended commit corresponding to the first block
  • when successfully verifying a received block, the reactor saves its corresponding extended commit in the block store

The two main drawbacks of this base implementation are:

  • the increased size taken by the block store, in particular with big extensions
  • the increased bandwith taken by the new format of BlockResponse

Possible Optimization: Pruning the Extended Commit History

If we cannot switch from the consensus reactor back to the blocksync reactor we cannot prune the extended commit backlog in the block store without sacrificing the implementation's correctness. The asynchronous nature of our distributed system model allows a process to fall behing an arbitrary number of heights, and thus all extended commits need to be kept just in case a node that late had previously switched to the consensus reactor.

However, there is a possibility to optimize the base implementation. Every time we enter a new height, we could prune from the block store all extended commits that are more than d heights in the past. Then, we need to handle two new situations, roughly equivalent to cases (h.1) and (h.2) described above.

  • (h.1) A node starts from scratch or recovers after a crash. In thisy case, we need to modify the blocksync reactor's base implementation.
    • when receiving a BlockResponse message, it MUST accept that the extended commit set to nil,
    • when sending a BlockResponse message, if the block store contains the extended commit for that height, it MUST set it in the message, otherwise it sets it to nil,
    • the exit conditions used for the base implementation are no longer valid; the only reliable exit condition now consists in making sure that the last block processed by blocksync was received with the corresponding commit, and not nil; this extended commit will allow the node to switch from the blocksync reactor to the consensus reactor and immediately act as a proposer if required.
  • (h.2) A node already running the consensus reactor falls behind beyond d heights. In principle, the node will be stuck forever as no other node can provide the vote extensions it needs to make progress (they all have pruned the corresponding extended commit). However we can manually have the node crash and recover as a workaround. This effectively converts this case into (h.1).

Formalization Work

A formalization work to show or prove the correctness of the different use cases and solutions presented here (and any other that may be found) needs to be carried out. A question that needs a precise answer is how many extended commits (one?, two?) a node needs to keep in persistent memory when implementing Solution 3 described above without Tenderdash's current limitations. Another important invariant we need to prove formally is that the set of vote extensions required to make progress will always be held somewhere in the network.

References

RFC 018: BLS Signature Aggregation Exploration

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 01-April-2022: Initial draft (@williambanfield).
  • 15-April-2022: Draft complete (@williambanfield).

Abstract

Background

Glossary

The terms that are attached to these types of cryptographic signing systems become confusing quickly. Different sources appear to use slightly different meanings of each term and this can certainly add to the confusion. Below is a brief glossary that may be helpful in understanding the discussion that follows.

  • Short Signature: A signature that does not vary in length with the number of signers.
  • Multi-Signature: A signature generated over a single message where, given the message and signature, a verifier is able to determine that all parties signed the message. May be short or may vary with the number of signers.
  • Aggregated Signature: A short signature generated over messages with possibly different content where, given the messages and signature, a verifier should be able to determine that all the parties signed the designated messages.
  • Threshold Signature: A short signature generated from multiple signers where, given a message and the signature, a verifier is able to determine that a large enough share of the parties signed the message. The identities of the parties that contributed to the signature are not revealed.
  • BLS Signature: An elliptic-curve pairing-based signature system that has some nice properties for short multi-signatures. May stand for Boneh-Lynn-Schacham or Barreto-Lynn-Scott depending on the context. A BLS signature is type of signature scheme that is distinct from other forms of elliptic-curve signatures such as ECDSA and EdDSA.
  • Interactive: Cryptographic scheme where parties need to perform one or more request-response cycles to produce the cryptographic material. For example, an interactive signature scheme may require the signer and the verifier to cooperate to create and/or verify the signature, rather than a signature being created ahead of time.
  • Non-interactive: Cryptographic scheme where parties do not need to perform any request-response cycles to produce the cryptographic material.

Brief notes on pairing-based elliptic-curve cryptography

Pairing-based elliptic-curve cryptography is quite complex and relies on several types of high-level math. Cryptography, in general, relies on being able to find problems with an asymmetry between the difficulty of calculating the solution and verifying that a given solution is correct.

Pairing-based cryptography works by operating on mathematical functions that satisfy the property of bilinear mapping. This property is satisfied for functions e with values P, Q, R and S where e(P, Q + R) = e(P, Q) * e(P, R) and e(P + S, Q) = e(P, Q) * e(S, Q). The most familiar example of this is exponentiation. Written in common notation, g^P*(Q+R) = g^(P*Q) * g^(P*R) for some value g.

Pairing-based elliptic-curve cryptography creates a bilinear mapping using elliptic curves over a finite field. With some original curve, you can define two groups, G1 and G2 which are points of the original curve modulo different values. Finally, you define a third group Gt, where points from G1 and G2 satisfy the property of bilinearity with Gt. In this scheme, the function e takes as inputs points in G1 and G2 and outputs values in Gt. Succintly, given some point P in G1 and some point Q in G1, e(P, Q) = C where C is in Gt. You can efficiently compute the mapping of points in G1 and G2 into Gt, but you cannot efficiently determine what points were summed and paired to produce the value in Gt.

Functions are then defined to map digital signatures, messages, and keys into and out of points of G1 or G2 and signature verification is the process of calculating if a set of values representing a message, public key, and digital signature produce the same value in Gt through e.

Signatures can be created as either points in G1 with public keys being created as points in G2 or vice versa. For the case of BLS12-381, the popular curve used, points in G1 are represented with 48 bytes and points in G2 are represented with 96 bytes. It is up to the implementer of the cryptosystem to decide which should be larger, the public keys or the signatures.

BLS signatures rely on pairing-based elliptic-curve cryptography to produce various types of signatures. For a more in-depth but still high level discussion pairing-based elliptic-curve cryptography, see Vitalik Buterin's post on Exploring Elliptic Curve Pairings. For much more in depth discussion, see the specific paper on BLS12-381, Short signatures from the Weil Pairing and Compact Multi-Signatures for Smaller Blockchains.

Adoption

BLS signatures have already gained traction within several popular projects.

  • Algorand is working on an implementation.
  • Zcash has adopted BLS12-381 into the protocol.
  • Ethereum 2.0 has adopted BLS12-381 into the protocol.
  • Chia Network has adopted BLS for signing blocks.
  • Ostracon, a fork of Tendermint has adopted BLS for signing blocks.

What systems may be affected by adding aggregated signatures?

Gossip

Gossip could be updated to aggregate vote signatures during a consensus round. This appears to be of frankly little utility. Creating an aggregated signature incurs overhead, so frequently re-aggregating may incur a significant overhead. How costly this is is still subject to further investigation and performance testing.

Even if vote signatures were aggregated before gossip, each validator would still need to receive and verify vote extension data from each (individual) peer validator in order for consensus to proceed. That displaces any advantage gained by aggregating signatures across the vote message in the presence of vote extensions.

Block Creation

When creating a block, the proposer may create a small set of short multi-signatures and attach these to the block instead of including one signature per validator.

Block Verification

Currently, we verify each validator signature using the public key associated with that validator. With signature aggregation, verification of blocks would not verify many signatures individually, but would instead check the (single) multi-signature using the public keys stored by the validator. This would also require a mechanism for indicating which validators are included in the aggregated signature.

IBC Relaying

IBC would no longer need to transmit a large set of signatures when updating state. These state updates do not happen for every IBC packet, only when changing an IBC light client's view of the counterparty chain's state. General [IBC packets][ibc-packet] only contain enough information to correctly route the data to the counterparty chain.

IBC does persist commit signatures to the chain in these MsgUpdateClient message when updating state. This message would no longer need the full set of unique signatures and would instead only need one signature for all of the data in the header.

Adding BLS signatures would create a new signature type that must be understood by the IBC module and by the relayers. For some operations, such as state updates, the set of data written into the chain and received by the IBC module could be slightly smaller.

Discussion

What are the proposed benefits to aggregated signatures?

Reduce Block Size

At the moment, a commit contains a 64-byte (512-bit) signature for each validator that voted for the block. For the Cosmos Hub, which has 175 validators in the active set, this amounts to about 11 KiB per block. That gives an upper bound of around 113 GiB over the lifetime of the chain's 10.12M blocks. (Note, the Hub has increased the number of validators in the active set over time so the total signature size over the history of the chain is likely somewhat less than that).

Signature aggregation would only produce two signatures for the entire block. One for the yeas and one for the nays. Each BLS aggregated signature is 48 bytes, per the IETF standard of BLS signatures. Over the lifetime of the same Cosmos Hub chain, that would amount to about 1 GB, a savings of 112 GB. While that is a large factor of reduction it's worth bearing in mind that, at GCP's cost of $.026 USD per GB, that is a total savings of around $2.50 per month.

Reduce Signature Creation and Verification Time

From the IETF draft standard on BLS Signatures, BLS signatures can be created in 370 microseconds and verified in 2700 microseconds. Our current Ed25519 implementation was benchmarked locally to take 13.9 microseconds to produce a signature and 2.03 milliseconds to batch verify 128 signatures, which is slightly fewer than the 175 in the Hub. blst, a popular implementation of BLS signature aggregation was benchmarked to perform verification on 100 signatures in 1.5 milliseconds when run locally on an 8 thread machine and pre-aggregated public keys. It is worth noting that the ed25519 library verification time grew steadily with the number of signatures, whereas the bls library verification time remains constant. This is because the number of operations used to verify a signature does not grow at all with the number of signatures included in the aggregate signature (as long as the signers signed over the same message data as is the case in Tenderdash).

It is worth noting that this would also represent a degredation in signature verification time for chains with small validator sets. When batch verifying only 32 signatures, our ed25519 library takes .57 milliseconds, whereas BLS would still require the same 1.5 milliseconds.

For massive validator sets, blst dominates, taking the same 1.5 milliseconds to check an aggregated signature from 1024 validators versus our ed25519 library's 13.066 milliseconds to batch verify a set of that size.

Reduce Light-Client Verification Time

The light client aims to be a faster and lighter-weight way to verify that a block was voted on by a Tenderdash network. The light client fetches Tenderdash block headers and commit signatures, performing public key verification to ensure that the associated validator set signed the block. Reducing the size of the commit signature would allow the light client to fetch block data more quickly.

Additionally, the faster signature verification times of BLS signatures mean that light client verification would proceed more quickly.

However, verification of an aggregated signature is all-or-nothing. The verifier cannot check that some singular signer had a signature included in the block. Instead, the verifier must use all public keys to check if some signature was included. This does mean that any light client implementation must always be able to fetch all public keys for any height instead of potentially being able to check if some singular validator's key signed the block.

Reduce Gossip Bandwidth

Vote Gossip

It is possible to aggregate subsets of signatures during voting, so that the network need not gossip all n validator signatures to all n validators. Theoretically, subsets of the signatures could be aggregated during consensus and vote messages could carry those aggregated signatures. Implementing this would certainly increase the complexity of the gossip layer but could possibly reduce the total number of signatures required to be verified by each validator.

Block Gossip

A reduction in the block size as a result of signature aggregation would naturally lead to a reduction in the bandwidth required to gossip a block. Each validator would only send and receive the smaller aggregated signatures instead of the full list of multi-signatures as we have them now.

What are the drawbacks to aggregated signatures?

Heterogeneous key types cannot be aggregated

Aggregation requires a specific signature algorithm, and our legacy signing schemes cannot be aggregated. In practice, this means that aggregated signatures could be created for a subset of validators using BLS signatures, and validators with other key types (such as Ed25519) would still have to be be separately propagated in blocks and votes.

Many HSMs do not support aggregated signatures

Hardware Signing Modules (HSM) are a popular way to manage private keys. They provide additional security for key management and should be used when possible for storing highly sensitive private key material.

Below is a list of popular HSMs along with their support for BLS signatures.

I cannot find support listed for Google Cloud, although perhaps it exists.

Feasibility of implementation

This section outlines the various hurdles that would exist to implementing BLS signature aggregation into Tenderdash. It aims to demonstrate that we could implement BLS signatures but that it would incur risk and require breaking changes for a reasonably unclear benefit.

Can aggregated signatures be added as soft-upgrades?

In my estimation, yes. With the implementation of proposer-based timestamps, all validators now produce signatures on only one of two messages:

  1. A CanonicalVote where the BlockID is the hash of the block or
  2. A CanonicalVote where the BlockID is nil.

The block structure can be updated to perform hashing and validation in a new way as a soft upgrade. This would look like adding a new section to the Block.Commit structure alongside the current Commit.Signatures field. This new field, tentatively named AggregatedSignature would contain the following structure:

message AggregatedSignature {
  // yeas is a BitArray representing which validators in the active validator
  // set issued a 'yea' vote for the block.
  tendermint.libs.bits.BitArray yeas = 1;

  // absent is a BitArray representing which validators in the active
  // validator set did not issue votes for the block.
  tendermint.libs.bits.BitArray absent = 2;

  // yea_signature is an aggregated signature produced from all of the vote
  // signatures for the block.
  repeated bytes yea_signature = 3;

  // nay_signature is an aggregated signature produced from all of the vote
  // signatures from votes for 'nil' for this block.
  // nay_signature should be made from all of the validators that were both not
  // in the 'yeas' BitArray and not in the 'absent' BitArray.
  repeated bytes nay_signature = 4;
}

Adding this new field as a soft upgrade would mean hashing this data structure into the blockID along with the old Commit.Signatures when both are present as well as ensuring that the voting power represented in the new AggregatedSignature and Signatures field was enough to commit the block during block validation. One can certainly imagine other possible schemes for implementing this but the above should serve as a simple enough proof of concept.

Implementing vote-time and commit-time signature aggregation separately

Implementing aggregated BLS signatures as part of the block structure can easily be achieved without implementing any 'vote-time' signature aggregation. The block proposer would gather all of the votes, complete with signatures, as it does now, and produce a set of aggregate signatures from all of the individual vote signatures.

Implementing 'vote-time' signature aggregation cannot be achieved without also implementing commit-time signature aggregation. This is because such signatures cannot be dis-aggregated into their constituent pieces. Therefore, in order to implement 'vote-time' signature aggregation, we would need to either first implement 'commit-time' signature aggregation, or implement both 'vote-time' signature aggregation while also updating the block creation and verification protocols to allow for aggregated signatures.

Updating IBC clients

In order for IBC clients to function, they must be able to perform light-client verification of blocks on counterparty chains. Because BLS signatures are not currently part of light-clients, chains that transmit messages over IBC cannot update to using BLS signatures without their counterparties first being upgraded to parse and verify BLS. If chains upgrade without their counterparties first updating, they will lose the ability to interoperate with non-updated chains.

New attack surfaces

BLS signatures and signature aggregation comes with a new set of attack surfaces. Additionally, it's not clear that all possible major attacks are currently known on the BLS aggregation schemes since new ones have been discovered since the ietf draft standard was written. The known attacks are manageable and are listed below. Our implementation would need to prevent against these but this does not appear to present a significant hurdle to implementation.

Rogue key attack prevention

Generating an aggregated signature requires guarding against what is called a rogue key attack. A rogue key attack is one in which a malicious actor can craft an aggregate key that can produce signatures that appear to include a signature from a private key that the malicious actor does not actually know. In Tenderdash terms, this would look like a Validator producing a vote signed by both itself and some other validator where the other validator did not actually produce the vote itself.

The main mechanisms for preventing this require that each entity prove that it can can sign data with just their private key. The options involve either ensuring that each entity sign a different message when producing every signature or producing a proof of possession (PoP) when announcing their key to the network.

A PoP is a message that demonstrates ownership of a private key. A simple scheme for PoP is one where the entity announcing its new public key to the network includes a digital signature over the bytes of the public key generated using the associated private key. Everyone receiving the public key and associated proof-of-possession can easily verify the signature and be sure the entity owns the private key.

This PoP scheme suits the Tenderdash use case quite well since validator keys change infrequently so the associated PoPs would not be onerous to produce, verify, and store. Using this scheme allows signature verification to proceed more quickly, since all signatures are over identical data and can therefore be checked using an aggregated public key instead of one at a time, public key by public key.

Summing Zero Attacks

Summing zero attacks are attacks that rely on using the '0' point of an elliptic curve. For BLS signatures, if the point 0 is chosen as the private key, then the 0 point will also always be the public key and all signatures produced by the key will also be the 0 point. This is easy enough to detect when verifying each signature individually.

However, because BLS signature aggregation creates an aggregated signature and an aggregated public key, a set of colluding signers can create a pair or set of signatures that are non-zero but which aggregate ("sum") to 0. The signatures that sum zero along with the summed public key of the colluding signers will verify any message. This would allow the colluding signers to sign any block or message with the same signature. This would be reasonably easy to detect and create evidence for because, in all other cases, the same signature should not verify more than message. It's not exactly clear how such an attack would advantage the colluding validators because the normal mechanisms of evidence gathering would still detect the double signing, regardless of the signatures on both blocks being identical.

Backwards Compatibility

Backwards compatibility is an important consideration for signature verification. Specifically, it is important to consider whether chains using current versions of IBC would be able to interact with chains adopting BLS.

Because the Block shared by IBC and Tenderdash is produced and parsed using protobuf, new structures can be added to the Block without breaking the ability of legacy users to parse the new structure. Breaking changes between current users of IBC and new Tenderdash blocks only occur if data that is relied upon by the current users is no longer included in the current fields.

For the case of BLS aggregated signatures, a new AggregatedSignature field can therefore be added to the Commit field without breaking current users. Current users will be broken when counterparty chains upgrade to the new version and begin using BLS signatures. Once counterparty chains begin using BLS signatures, the BlockID hashes will include hashes of the AggregatedSignature data structure that the legacy users will not be able to compute. Additionally, the legacy software will not be able to parse and verify the signatures to ensure that a supermajority of validators from the counterparty chain signed the block.

Library Support

Libraries for BLS signature creation are limited in number, although active development appears to be ongoing. Cryptographic algorithms are difficult to implement correctly and correctness issues are extremely serious and dangerous. No further exploration of BLS should be undertaken without strong assurance of a well-tested library with continuing support for creating and verifying BLS signatures.

At the moment, there is one candidate, blst, that appears to be the most mature and well vetted. While this library is undergoing continuing auditing and is supported by funds from the Ethereum foundation, adopting a new cryptographic library presents some serious risks. Namely, if the support for the library were to be discontinued, Tenderdash may become saddled with the requirement of supporting a very complex piece of software or force a massive ecosystem-wide migration away from BLS signatures.

This is one of the more serious reasons to avoid adopting BLS signatures at this time. There is no gold standard library. Some projects look promising, but no project has been formally verified with a long term promise of being supported well into the future.

Go Standard Library

The Go Standard library has no implementation of BLS signatures.

BLST

blst, or 'blast' is an implementation of BLS signatures written in C that provides bindings into Go as part of the repository. This library is actively undergoing formal verification by Galois and previously received an initial audit by NCC group, a firm I'd never heard of.

blst is targeted for use in prysm, the golang implementation of Ethereum 2.0.

Gnark-Crypto

Gnark-Crypto is a Go-native implementation of elliptic-curve pairing-based cryptography. It is not audited and is documented as 'as-is', although development appears to be active so formal verification may be forthcoming.

CIRCL

CIRCL is a go-native implementation of several cryptographic primitives, bls12-381 among them. The library is written and maintained by Cloudflare and appears to receive frequent contributions. However, it lists itself as experimental and urges users to take caution before using it in production.

Added complexity to light client verification

Implementing BLS signature aggregation in Tenderdash would pose issues for the light client. The light client currently validates a subset of the signatures on a block when performing the verification algorithm. This is no longer possible with an aggregated signature. Aggregated signature verification is all-or-nothing. The light client could no longer check that a subset of validators from some set of validators is represented in the signature. Instead, it would need to create a new aggregated key with all the stated signers for each height it verified where the validator set changed.

This means that the speed advantages gained by using BLS cannot be fully realized by the light client since the client needs to perform the expensive operation of re-aggregating the public key. Aggregation is not constant time in the number of keys and instead grows linearly. When benchmarked locally, blst public key aggregation of 128 keys took 2.43 milliseconds. This, along with the 1.5 milliseconds to verify a signature would raise light client signature verification time to 3.9 milliseconds, a time above the previously mentioned batch verification time using our ed25519 library of 2.0 milliseconds.

Schemes to cache aggregated subsets of keys could certainly cut this time down at the cost of adding complexity to the light client.

Added complexity to evidence handling

Implementing BLS signature aggregation in Tenderdash would add complexity to the evidence handling within Tenderdash. Currently, the light client can submit evidence of a fork attempt to the chain. This evidence consists of the set of validators that double-signed, including their public keys, with the conflicting block.

We can quickly check that the listed validators double signed by verifying that each of their signatures are in the submitted conflicting block. A BLS signature scheme would change this by requiring the light client to submit the public keys of all of the validators that signed the conflicting block so that the aggregated signature may be checked against the full signature set. Again, aggregated signature verification is all-or-nothing, so without all of the public keys, we cannot verify the signature at all. These keys would be retrievable. Any party that wanted to create a fork would want to convince a network that its fork is legitimate, so it would need to gossip the public keys. This does not hamper the feasibility of implementing BLS signature aggregation into Tenderdash, but does represent yet another piece of added complexity to the associated protocols.

Open Questions

  • Q: Can you aggregate Ed25519 signatures in Tenderdash?
    • There is a suggested scheme in github issue 7892, but additional rigor would be required to fully verify its correctness.

Current Consideration

Adopting a signature aggregation scheme presents some serious risks and costs to the Tenderdash project. It requires multiple backwards-incompatible changes to the code, namely a change in the structure of the block and a new backwards-incompatible signature and key type. It risks adding a new signature type for which new attack types are still being discovered and for which no industry standard, battle-tested library yet exists.

The gains boasted by this new signing scheme are modest: Verification time is marginally faster and block sizes shrink by a few kilobytes. These are relatively minor gains in exchange for the complexity of the change and the listed risks of the technology. We should take a wait-and-see approach to BLS signature aggregation, monitoring the up-and-coming projects and consider implementing it as the libraries and standards develop.

References

RFC 019: Configuration File Versioning

Historical upstream design record. Project and release names below identify the original implementation; they do not describe Tenderdash releases. See the current protocol specification for Tenderdash.

Changelog

  • 19-Apr-2022: Initial draft (@creachadair)
  • 20-Apr-2022: Updates from review feedback (@creachadair)

Abstract

Updating configuration settings is an essential part of upgrading an existing node to a new version of the Tendermint software. Unfortunately, it is also currently a very manual process. This document discusses some of the history of changes to the config format, actions we've taken to improve the tooling for configuration upgrades, and additional steps we may want to consider.

Background

A Tendermint node reads configuration settings at startup from a TOML formatted text file, typically named config.toml. The contents of this file are defined by the github.com/tendermint/tendermint/config.

Although many settings in this file remain valid from one version of Tendermint to the next, new versions of Tendermint often add, update, and remove settings. These changes often require manual intervention by operators who are upgrading their nodes.

I propose we should provide better tools and documentation to help operators make configuration changes correctly during version upgrades. Ideally, as much as possible of any configuration file update should be automated, and where that is not possible or practical, we should provide clear, explicit directions for what steps need to be taken manually. Moreover, when the node discovers incorrect or invalid configuration, we should improve the diagnostics it emits so that the operator can quickly and easily find the relevant documentation, without having to grep through source code.

Discussion

By convention, we are supposed to document required changes to the config file in the UPGRADING.md file for the release that introduces them. Although we have mostly done this, the level of detail in the upgrading instructions is often insufficient for an operator to correctly update their file.

The updates vary widely in complexity: Operators may need to add new required settings, update obsolete values for existing settings, move or rename existing settings within the file, or remove obsolete settings (which are thus invalid). Here are a few examples of each of these cases:

  • New required settings: Tendermint v0.35 added a new top-level mode setting that determines whether a node runs as a validator, a full node, or a seed node. The default value is "full", which means the operator of a validator must manually add mode = "validator" (or set the --mode flag on the command line) for their node to come up in the correct mode.

  • Updated obsolete values: Tendermint v0.35 removed support for versions "v1" and "v2" of the blocksync (formerly "fastsync") protocol, requiring any node using either of those values to update to "v0".

  • Moved/renamed settings: Version v0.34 moved the top-level pprof_laddr setting under the [rpc] section.

    Version v0.35 renamed every setting in the file from snake_case to kebab-case, moved the top-level fast_sync setting into the [blocksync] section as (itself renamed from [fastsync]), and moved all the top-level priv-validator-* settings under a new [priv-validator] section with their prefix trimmed off.

  • Removed obsolete settings: Version v0.34 removed the index_all_keys and index_keys settings from the [tx_index] section; version v0.35 removed the wal-dir setting from the [mempool] section, and version v0.36 removed the [blocksync] section entirely.

While many of these changes are mentioned in the config section of the upgrade instructions, some are not mentioned at all, or are hidden in other parts of the doc. For instance, the v0.34 pprof_laddr change was documented only as an RPC flag change. (A savvy reader might realize that the flag --rpc.pprof_laddr implies a corresponding config section, but it omits the related detail that there was a top-level setting that's been renamed). The lesson here is not that the docs are bad, but to point out that prose is not the most efficient format to convey detailed changes like this. The upgrading instructions are still valuable for the human reader to understand what to expect.

Concrete Steps

As part of the v0.36 development cycle, we spent some time reverse-engineering the configuration changes since the v0.34 release and built an experimental command-line tool called confix, whose job it is to automatically update the settings in a config.toml file to the latest version. We also backported a version of this tool into the v0.35.x branch at release v0.35.4.

This tool should work fine for configuration files created by Tendermint v0.34 and later, but does not (yet) know how to handle changes from prior versions of Tendermint. Part of the difficulty for older versions is simply logistical: To figure out which changes to apply, we need to understand something about the version that made the file, as well as the version we're converting it to.

Discussion point: In the future we might want to consider incorporating this into the node CLI directly, but we're keeping it separate for now until we can get some feedback from operators.

For the experiment, we handled this by carefully searching the history of config format changes for shibboleths to bound the version: For example, the [fastsync] section was added in Tendermint v0.32 and renamed [blocksync] in Tendermint v0.35. So if we see a [fastsync] section, we have some confidence that the file was created by v0.32, v0.33, or v0.34.

But such signals are delicate: The [blocksync] section was removed in v0.36, so if we do not find [fastsync], we cannot conclude from that alone that the file is from v0.31 or earlier -- we have to look for corroborating details. While such "sniffing" tactics are fine for an experiment, they aren't as robust as we might like.

This is especially relevant for configuration files that may have already been manually upgraded across several versions by the time we are asked to update them again. Another related concern is that we'd like to make sure conversion is idempotent, so that it would be safe to rerun the tool over an already-converted file without breaking anything.

Config Versioning

One obvious tactic we could use for future releases is add a version marker to the config file. This would give tools like confix (and the node itself) a way to calibrate their expectations. Rather than being a version for the file itself, however, this version marker would indicate which version of Tendermint is needed to read the file.

Provisionally, this might look something like:

# THe minimum version of Tendermint compatible with the contents of
# this configuration file.
config-version = 'v0.35'

When initializing a new node, Tendermint would populate this field with its own version (e.g., v0.36). When conducting an upgrade, tools like confix can then use this to decide which conversions are valid, and then update the value accordingly. After converting a file marked 'v0.35' to'v0.37', the conversion tool sets the file's config-version to reflect its compatibility.

Discussion point: This example presumes we would keep config files compatible within a given release cycle, e.g., all of v0.36.x. We could also use patch numbers here, if we think there's some reason to permit changes that would require config file edits at that granularity. I don't think we should, but that's a design question to consider.

Upon seeing an up-to-date version marker, the conversion tool can simply exit with a diagnostic like "this file is already up-to-date", rather than sniffing the keyspace and potentially introducing errors. In addition, this would let a tool detect config files that are newer than the one it understands, and issue a safe diagnostic rather than doing something wrong. Plus, besides avoiding potentially unsafe conversions, this would also serve as human-readable documentation that the file is up-to-date for a given version.

Adding a config version would not address the problem of how to convert files created by older versions of Tendermint, but it would at least help us build more robust config tooling going forward.

Stability and Change

In light of the discussion so far, it is natural to examine why we make so many changes to the configuration file from one version to the next, and whether we could reduce friction by being more conservative about what we make configurable, what config changes we make over time, and how we roll them out.

Some changes, like renaming everything from snake case to kebab case, are entirely gratuitous. We could safely agree not to make those kinds of changes. Apart from that obvious case, however, many other configuration settings provide value to node operators in cases where there is no simple, universal setting that matches every application.

Taking a high-level view, there are several broad reasons why we might want to make changes to configuration settings:

  • Lessons learned: Configuration settings are a good way to try things out in production, before making more invasive changes to the consensus protocol.

    For example, up until Tendermint v0.35, consensus timeouts were specified as per-node configuration settings (e.g., timeout-precommit et al.). This allowed operators to tune these values for the needs of their network, but had the downside that individually-misconfigured nodes could stall consensus.

    Based on that experience, these timeouts have been deprecated in Tendermint v0.36 and converted to consensus parameters, to be consistent across all nodes in the network.

  • Migration & experimentation: Introducing new features and updating old features can complicate migration for existing users of the software. Temporary or "experimental" configuration settings can be a valuable way to mitigate that friction.

    For example, Tendermint v0.36 introduces a new RPC event subscription endpoint (see ADR 075) that will eventually replace the existing webwocket-based interface. To give users time to migrate, v0.36 adds an experimental-disable-websocket setting, defaulted to false, that allows operators to selectively disable the websocket API for testing purposes during the conversion. This setting is designed to be removed in v0.37, when the old interface is no longer supported.

  • Ongoing maintenance: Sometimes configuration settings become obsolete, and the cost of removing them trades off against the potential risks of leaving a non-functional or deprecated knob hooked up indefinitely.

    For example, Tendermint v0.35 deprecated two alternate implementations of the blocksync protocol, one of which was deleted entirely (v1) and one of which was scheduled for removal (v2). The blocksync.version setting, which had been added as a migration aid, became obsolete and needed to be updated.

    Despite our best intentions, sometimes engineering designs do not work out. It's just as important to leave room to back out of changes we have since reconsidered, as it is to support migrations forward onto new and improved code.

  • Clarity and legibility: Besides configuring the software, another important purpose of a config file is to document intent for the humans who operate and maintain the software. Operators need adjust settings to keep the node running, and developers need to know what options were in use when something goes wrong so they can diagnose and fix bugs. The legibility of a config file as a human artifact is also thus important.

    For example, Tendermint v0.35 moved settings related to validator private keys from the top-level section of the configuration file to their own designated [priv-validator] section. Although this change did not make any difference to the meaning of those settings, it made the organization of the file easier to understand, and allowed the names of the individual settings to be simplified (e.g., priv-validator-key-file became simply key-file in the new section).

    Although such changes are "gratuitous" with respect to the software, there is often value in making things more legible for the humans. While there is no simple rule to define the line, the Potter Stewart principle can be used with due care.

Keeping these examples in mind, we can and should take reasonable steps to avoid churn in the configuration file across versions where we can. However, we must also accept that part of the reason for having a config file is to allow us flexibility elsewhere in the design. On that basis, we should not attempt to be too dogmatic about config changes either. Unlike changes in the block protocol, for example, which affect every user of every network that adopts them, config changes are relatively self-contained.

There are few guiding principles I think we can use to strike a sensible balance:

  1. No gratuitous changes. Aesthetic changes that do not enhance legibility, avert confusion, or clarity documentation, should be entirely avoided.

  2. Prefer mechanical changes. Whenever it is practical, change settings in a way that can be updated by a tool without operator judgement. This implies finding safe, universal defaults for new settings, and not changing the default values of existing settings.

    Even if that means we have to make multiple changes (e.g., add a new setting in the current version, deprecate the old one, and remove the old one in the next version) it's preferable if we can mechanize each step.

  3. Clearly signal intent. When adding temporary or experimental settings, they should be clearly named and documented as such. Use long names and suggestive prefixes (e.g., experimental-*) so that they stand out when read in the config file or printed in logs.

    Relatedly, using temporary or experimental settings should cause the software to emit diagnostic logs at runtime. These log messages should be easy to grep for, and should contain pointers to more complete documentation (say, issue numbers or URLs) that the operator can read, as well as a hint about when the setting is expected to become invalid. For example:

    WARNING: Websocket RPC access is deprecated and will be removed in
    Tendermint v0.37. See https://tinyurl.com/adr075 for more information.
    
  4. Consider both directions. When adding a configuration setting, take some time during the implementation process to think about how the setting could be removed, as well as how it will be rolled out. This applies even for settings we imagine should be permanent. Experience may cause is to rethink our original design intent more broadly than we expected.

    This does not mean we have to spend a long time picking nits over the design of every setting; merely that we should convince ourselves we could undo it without making too big a mess later. Even a little extra effort up front can sometimes save a lot.

References

Appendix: Research Notes

Discovering when various configuration settings were added, updated, and removed turns out to be surprisingly tedious. To solve this puzzle, we had to answer the following questions:

  1. What changes were made between v0.x and v0.y? This is further complicated by cases where we have backported config changes into the middle of an earlier release cycle (e.g., psql-conn from v0.35.x into v0.34.13).

  2. When during the development cycle were those changes made? This allows us to recognize features that were backported into a previous release.

  3. What were the default values of the changed settings, and did they change at all during or across the release boundary?

Each step of the configuration update plan is commented with a link to one or more PRs where that change was made. The sections below discuss how we found these references.

Tracking Changes Across Releases

To figure out what changed between two releases, we built a tool called condiff, which performs a "keyspace" diff of two TOML documents. This diff respects the structure of the TOML file, but ignores comments, blank lines, and configuration values, so that we can see what was added and removed.

To use it, run:

go run ./scripts/confix/condiff old.toml new.toml

This tool works on any TOML documents, but for our purposes we needed Tendermint config.toml files. The easiest way to get these is to build the node binary for your version of interest, run tendermint init on a clean home directory, and copy the generated config file out. The testdata directory for the confix tool has configs generated from the heads of each release branch from v0.31 through v0.35.

If you want to reproduce this yourself, it looks something like this:

# Example for Tendermint v0.32.
git checkout --track origin/v0.32.x
go get golang.org/x/sys/unix
go mod tidy
make build
rm -fr -- tmhome
./build/tendermint --home=tmhome init
cp tmhome/config/config.toml config-v32.toml

Be advised that the further back you go, the more idiosyncrasies you will encounter. For example, Tendermint v0.31 and earlier predate Go modules (v0.31 used dep), and lack backport branches. And you may need to do some editing of Makefile rules once you get back into the 20s.

Note that when diffing config files across the v0.34/v0.35 gap, the swap from snake_case to kebab-case makes it look like everything changed. The condiff tool has a -desnake flag that normalizes all the keys to kebab case in both inputs before comparison.

Locating Additions and Deletions

To figure out when a configuration setting was added or removed, your tool of choice is git bisect. The only tricky part is finding the endpoints for the search. If the transition happened within a release, you can use that release's backport branch as the endpoint (if it has one, e.g., v0.35.x).

However, the start point can be more problematic. The backport branches are not ancestors of master or of each other, which means you need to find some point in history prior to the change but still attached to the mainline. For recent releases there is a dev root (e.g., v0.35.0-dev, v0.34.0-dev1, etc.). These are not named consistently, but you can usually grep the output of git tag to find them.

In the worst case you could try starting from the root commit of the repo, but that turns out not to work in all cases. We've done some branching shenanigans over the years that mean the root is not a direct ancestor of all our release branches. When you find this you will probably swear a lot. I did.

Once you have a start and end point (say, v0.35.0-dev and master), you can bisect in the usual way. I use git grep on the config directory to check whether the case I am looking for is present. For example, to find when the [fastsync] section was removed:

# Setup:
git checkout master
git bisect start
git bisect bad                 # it's not present on tip of master.
git bisect good v0.34.0-dev1   # it was present at the start of v0.34.
# Now repeat this until it gives you a specific commit:
if git grep -q '\[fastsync\]' config ; then git bisect good ; else git bisect bad ; fi

The above example finds where a config was removed: To find where a setting was added, do the same thing except reverse the sense of the test (if ! git grep -q ...).

RFC 021: The Future of the Socket Protocol

Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.

Changelog

  • 19-May-2022: Initial draft (@creachadair)
  • 19-Jul-2022: Converted from ADR to RFC (@creachadair)

Abstract

This RFC captures some technical discussion about the ABCI socket protocol that was originally documented to solicit an architectural decision. This topic was not high-enough priority as of this writing to justify making a final decision.

For that reason, the text of this RFC has the general structure of an ADR, but should be viewed primarily as a record of the issue for future reference.

Background

The Application Blockchain Interface (ABCI) is a client-server protocol used by the Tenderdash consensus engine to communicate with the application on whose behalf it performs state replication. There are currently three transport options available for ABCI applications:

  1. In-process: Applications written in Go can be linked directly into the same binary as the consensus node. Such applications use a "local" ABCI connection, which exposes application methods to the node as direct function calls.

  2. Socket protocol: Out-of-process applications may export the ABCI service via a custom socket protocol that sends requests and responses over a Unix-domain or TCP socket connection as length-prefixed protocol buffers. In Tenderdash, this is handled by the socket client.

  3. gRPC: Out-of-process applications may export the ABCI service via gRPC. In Tenderdash, this is handled by the gRPC client.

Both the out-of-process options (2) and (3) have a long history in Tenderdash. The beginnings of the gRPC client were added in May 2016 when ABCI was still hosted in a separate repository, and the socket client (formerly called the "remote client") was part of ABCI from its inception in November 2015.

At that time when ABCI was first being developed, the gRPC project was very new (it launched Q4 2015) and it was not an obvious choice for use in Tenderdash. It took a while before the language coverage and quality of gRPC reached a point where it could be a viable solution for out-of-process applications. For that reason, it made sense for the initial design of ABCI to focus on a custom protocol for out-of-process applications.

Problem Statement

For practical reasons, ABCI needs an interprocess communication option to support applications not written in Go. The two practical options are RPC and FFI, and for operational reasons an RPC mechanism makes more sense.

The socket protocol has not changed all that substantially since its original design, and has the advantage of being simple to implement in almost any reasonable language. However, its simplicity includes some limitations that have had a negative impact on the stability and performance of out-of-process applications using it. In particular:

  • The protocol lacks request identifiers, so the client and server must return responses in strict FIFO order. Even if the client issues requests that have no dependency on each other, the protocol has no way except order of issue to map responses to requests.

    This reduces (in some cases substantially) the concurrency an application can exploit, since the parallelism of requests in flight is gated by the slowest active request at any moment. There have been complaints from some network operators on that basis.

  • The protocol lacks method identifiers, so the only way for the client and server to understand which operation is requested is to dispatch on the type of the request and response payloads. For responses, this means that any error condition is terminal not only to the request, but to the entire ABCI client.

    The historical intent of terminating for any error seems to have been that all ABCI errors are unrecoverable and hence protocol fatal (see Note 1). In practice, however, this greatly complicates debugging a faulty node, since the only way to respond to errors is to panic the node which loses valuable context that could have been logged.

  • There are subtle concurrency management dependencies between the client and the server that are not clearly documented anywhere, and it is very easy for small changes in both the client and the server to lead to tricky deadlocks, panics, race conditions, and slowdowns. As a recent example of this, see https://github.com/tendermint/tendermint/pull/8581.

These limitations are fixable, but one important question is whether it is worthwhile to fix them. We can add request and method identifiers, for example, but doing so would be a breaking change to the protocol requiring every application using it to update. If applications have to migrate anyway, the stability and language coverage of gRPC have improved a lot, and today it is probably simpler to set up and maintain an application using gRPC transport than to reimplement the Tenderdash socket protocol.

Moreover, gRPC addresses all the above issues out-of-the-box, and requires (much) less custom code for both the server (i.e., the application) and the client. The project is well-funded and widely-used, which makes it a safe bet for a dependency.

Decision

There is a set of related alternatives to consider:

  • Question 1: Designate a single IPC standard for out-of-process applications?

    Claim: We should converge on one (and only one) IPC option for out-of-process applications. We should choose an option that, after a suitable period of deprecation for alternatives, will address most or all the highest-impact uses of Tenderdash. Maintaining multiple options increases the surface area for bugs and vulnerabilities, and we should not have multiple options for basic interfaces without a clear and well-documented reason.

  • Question 2a: Choose gRPC and deprecate/remove the socket protocol?

    Claim: Maintaining and improving a custom RPC protocol is a substantial project and not directly relevant to the requirements of consensus. We would be better served by depending on a well-maintained open-source library like gRPC.

  • Question 2b: Improve the socket protocol and deprecate/remove gRPC?

    Claim: If we find meaningful advantages to maintaining our own custom RPC protocol in Tenderdash, we should treat it as a first-class project within the core and invest in making it good enough that we do not require other options.

One important consideration when discussing these questions is that any outcome which includes keeping the socket protocol will have eventual migration impacts for out-of-process applications regardless. To fix the limitations of the socket protocol as it is currently designed will require making breaking changes to the protocol. So, while we may put off a migration cost for out-of-process applications by retaining the socket protocol in the short term, we will eventually have to pay those costs to fix the problems in its current design.

Detailed Design

  1. If we choose to standardize on gRPC, the main work in Tenderdash core will be removing and cleaning up the code for the socket client and server.

    Besides the code cleanup, we will also need to clearly document a deprecation schedule, and invest time in making the migration easier for applications currently using the socket protocol.

    Point for discussion: Migrating from the socket protocol to gRPC should mostly be a plumbing change, as long as we do it during a release in which we are not making other breaking changes to ABCI. However, the effort may be more or less depending on how gRPC integration works in the application's implementation language, and would have to be sure networks have plenty of time not only to make the change but to verify that it preserves the function of the network.

    What questions should we be asking node operators and application developers to understand the migration costs better?

  2. If we choose to keep only the socket protocol, we will need to follow up with a more detailed design for extending and upgrading the protocol to fix the existing performance and operational issues with the protocol.

    Moreover, since the gRPC interface has been around for a long time we will also need a deprecation plan for it.

  3. If we choose to keep both options, we will still need to do all the work of (2), but the gRPC implementation should not require any immediate changes.

Alternatives Considered

  • FFI. Another approach we could take is to use a C-based FFI interface so that applications written in other languages are linked directly with the consensus node, an option currently only available for Go applications.

    An FFI interface is possible for a lot of languages, but FFI support varies widely in coverage and quality across languages and the points of friction can be tricky to work around. Moreover, it's much harder to add FFI support to a language where it's missing after-the-fact for an application developer.

    Although a basic FFI interface is not too difficult on the Go side, the C shims for an FFI can get complicated if there's a lot of variability in the runtime environment on the other end.

    If we want to have one answer for non-Go applications, we are better off picking an IPC-based solution (whether that's gRPC or an extension of our custom socket protocol or something else).

Consequences

  • Standardize on gRPC

    • ✅ Addresses existing performance and operational issues.
    • ✅ Replaces custom code with a well-maintained widely-used library.
    • ✅ Aligns with Cosmos SDK, which already uses gRPC extensively.
    • ✅ Aligns with priv validator interface, for which the socket protocol is already deprecated for gRPC.
    • ❓ Applications will be hard to implement in a language without gRPC support.
    • ⛔ All users of the socket protocol have to migrate to gRPC, and we believe most current out-of-process applications use the socket protocol.
  • Standardize on socket protocol

    • ✅ Less immediate impact for existing users (but see below).
    • ✅ Simplifies ABCI API surface by removing gRPC.
    • ❓ Users of the socket protocol will have a (smaller) migration.
    • ❓ Potentially easier to implement for languages that do not have support.
    • ⛔ Need to do all the work to fix the socket protocol (which will require existing users to update anyway later).
    • ⛔ Ongoing maintenance burden for per-language server implementations.
  • Keep both options

    • ✅ Less immediate impact for existing users (but see below).
    • ❓ Users of the socket protocol will have a (smaller) migration.
    • ⛔ Still need to do all the work to fix the socket protocol (which will require existing users to update anyway later).
    • ⛔ Requires ongoing maintenance and support of both gRPC and socket protocol integrations.

References

Notes

  • Note 1: The choice to make all ABCI errors protocol-fatal was intended to avoid the risk that recovering an application error could cause application state to diverge. Divergence can break consensus, so it's essential to avoid it.

    This is a sound principle, but conflates protocol errors with "mechanical" errors such as timeouts, resoures exhaustion, failed connections, and so on. Because the protocol has no way to distinguish these conditions, the only way for an application to report an error is to panic or crash.

    Whether a node is running in the same process as the application or as a separate process, application errors should not be suppressed or hidden. However, it's important to ensure that errors are handled at a consistent and well-defined point in the protocol: Having the application panic or crash rather than reporting an error means the node sees different results depending on whether the application runs in-process or out-of-process, even if the application logic is otherwise identical.

Appendix: Known Implementations of ABCI Socket Protocol

This is a list of known implementations of the Tendermint custom socket protocol. Note that in most cases I have not checked how complete or correct these implementations are; these are based on search results and a cursory visual inspection.

Consensus DoS — observability

A Grafana dashboard and Prometheus alert rules for watching the consensus DoS throttle, built to answer one question at a glance:

Is the node shedding the attack, or shedding its honest peers?

Files

  • grafana-dashboard.json — three-band dashboard (chain health / throttle firing / honest service). Import into Grafana or drop into a provisioning directory.
  • prometheus-alerts.yml — alert rules that page on honest symptoms (chain struggling, honest-vote latency high), not on drop counters. Load alongside the deploy repo's existing ChainHalt / RoundTooHigh rules.

Reading it

The dashboard has three rows:

  1. Is the chain healthy? — block production rate, rounds, prevote delays.
  2. Is the throttle firing, and where? — drops by mechanism, evidence by reason. Rising here under attack is expected and good.
  3. Is honest service still landing? — accepted-vote latency (p50/p99), budget saturation, lane depth.

The judgement:

middle band (drops)bottom band (honest)verdict
highlatency low, saturation off the floor, rounds flatshedding the attack — healthy
highlatency climbing toward the vote/propose timeout, saturation pinned at 0, rounds climbingstarving honest work

The single most important panel is accepted peer-vote latency p99. Threshold lines are drawn at the vote timeout (1 s) and propose timeout (3 s).

Deployment facts (dash-network-deploy)

  • Metrics carry the namespace prefix drive_tenderdash (set in dashmate's config.toml.dot). If your deployment uses a different [instrumentation] namespace, adjust the metric prefixes in both files.
  • Prometheus scrapes HP masternodes at :36660 — seed nodes are not scraped (they expose the endpoint but no job targets them). The dashboard therefore reflects validator nodes.
  • The metrics endpoint is enabled by the deploy tool's dashmate.json.j2 override, not by vanilla dashmate (whose default is off / localhost). A first devnet needs no config change to observe these — the endpoint and scrape already exist.
  • Provisioning: the deploy repo's roles/metrics/ ships Prometheus + Grafana but no dashboards in git. Add grafana-dashboard.json to a Grafana provisioning path (or import via the UI), and append prometheus-alerts.yml to the Prometheus rules the metrics role loads.

Metrics used

New with the DoS work — throttle firing: consensus_verification_budget_drops, consensus_peer_lane_drops, consensus_block_part_proof_drops, consensus_state_channel_drops, consensus_proposal_verify_failures, evidence_pool_dropped_evidence{reason}.

New — honest service: consensus_peer_vote_verify_latency_seconds (histogram), consensus_verification_budget_saturation (gauge, 1 = idle → 0 = drained), consensus_peer_lane_active_count, consensus_peer_lane_max_depth.

Stock, for chain health: consensus_latest_block_height, consensus_rounds, consensus_quorum_prevote_delay, consensus_full_prevote_delay.

(All shown without the drive_tenderdash_ namespace prefix that the deployment adds.)

Building the documentation

Tenderdash uses mdBook, like the Dash Platform book. The site includes the guides in docs/, all Markdown specifications in spec/, and the RPC reference from rpc/openapi/.

Local build

Install Python 3.10 or later and mdBook 0.4.40 (the version used by Platform):

cargo install mdbook --version 0.4.40 --locked
make build-docs

Run make serve-docs and open http://localhost:3000/. To serve an existing build, run python3 -m http.server 8000 --directory book/book and open http://localhost:8000/. Use HTTP so the RPC reference can fetch its schema. MDBOOK=/path/to/mdbook make build-docs selects a particular binary. For guides and specifications, edit their canonical files in docs/ or spec/, then run the build again. When using the server, rerun make prepare-docs after editing canonical files; mdBook watches the staged copies.

book/SUMMARY.md defines the navigation. Add every new Markdown page there; the build fails if a source page is missing from the summary or a summary entry has no source file. The HTML build also rejects missing local pages or assets. The consensus protocol and all its companion chapters are listed explicitly under Protocol specifications.

The preparation script copies sources and assets into ignored book/src/, strips VuePress metadata, resolves directory links, and points links to repository source files at the build's Git revision. It also stages the RPC reference at api/rpc/. It does not switch branches or modify documentation sources. Both book/src/ and book/book/ are disposable build output. Set DOCS_SOURCE_REF to override the Git revision used for source links.

Use fenced mermaid blocks for diagrams. The book uses the same vendored Mermaid 11.6.0 distribution as Platform; its MIT license is in book/mermaid-LICENSE. Diagrams render locally without a CDN. mdBook provides local full-text search without an external search service.

Preview and publishing

.github/workflows/docs-deployment.yml builds pull requests and uploads a documentation-preview artifact. Download it and extract it, then run python3 -m http.server 8000 from the extracted directory and open http://localhost:8000/. The RPC reference requires HTTP serving. Pull requests have read-only repository permissions and cannot deploy.

Pushes to the repository's default branch publish the same output using GitHub Pages artifacts and the github-pages environment, like Platform. This follows default-branch changes without editing the workflow.

To publish another branch manually, open the workflow in Actions, choose Run workflow, and select the branch. The workflow must exist on the default branch for GitHub to show that control. In Settings → Pages, select GitHub Actions as the build source. The github-pages environment's deployment rules must permit the default branch and any branches selected for manual publication. The workflow reports the deployed URL. Ordinary pushes to other branches do not publish; pull requests remain previews.

Naming and attribution

Use Tenderdash for this product, its CLI, and current source links. Preserve copyright notices, original research titles and citations, external project names, and identifiers required by the implementation, such as tendermint.abci, JSON public-key type names, metric names, and TLA+/Ivy module names. Do not rename wire types to change their display name.

Generate the ABCI API page from its protobuf source with make proto-doc; spec/abci/api.md is generated and must not be edited by hand.