Tenderdash Documentation
Tenderdash is the Byzantine Fault Tolerant consensus engine used by Dash Platform.
- Guides and introduction
- Operating nodes
- Application development
- Protocol specifications
- Consensus protocol
- RPC API
- Build and contribute to this documentation
The consensus specification covers the primary flow of a height, transitions and decision rules, alternative flows, commit handling, recovery, and known discrepancies. Use the sidebar or search to explore the complete documentation.
Tenderdash
Tenderdash is the Byzantine Fault Tolerant consensus engine used by Dash Platform. It replicates application state across validators and connects to the application through the Application Blockchain Interface (ABCI).
- Introduction and installation
- Operating a node
- Application development
- Core components
- Protocol specifications
- Consensus protocol: primary flow, transitions, alternative flows, commit handling, recovery, and known discrepancies
- RPC API (available in the generated site)
- Building and contributing to these docs
The protocol specification describes the implementation on this branch and links to the functions that define each rule. Historical research and design documents remain available alongside it.
Overview
Quick Start
Get Tenderdash up-and-running quickly with the quick-start guide!
Install
Detailed installation instructions.
What is Tenderdash
Dive into what Tenderdash is and why!
Tenderdash Architectural Overview
A Tenderdash node combines peer networking, consensus, transaction admission,
block storage and an ABCI connection to the application. The
node package
constructs these services and manages their lifecycle.
Communication
- Peer communication: long-lived P2P connections exchange transactions, blocks, votes, evidence and state-sync messages.
- Application communication: ABCI requests validate transactions, prepare and process proposals, finalize blocks and query application state. The application can run in the same Go process or through an external ABCI connection.
- Client communication: the RPC service accepts transactions, answers queries and exposes events and node status. RPC access is separate from P2P networking.
- Dash Core communication: quorum and ChainLock information connects Platform consensus to Dash Core. Configure the appropriate Core RPC endpoint for your network.
flowchart TB
Clients[RPC clients] --> RPC[RPC service]
RPC --> Mempool[Mempool]
RPC --> Store[Block and state stores]
Peers[Peer nodes] <--> P2P[P2P networking]
P2P <--> Consensus[Consensus]
P2P <--> Mempool
P2P <--> Sync[Block sync and state sync]
Consensus --> Store
Consensus <--> ABCI[ABCI application]
Mempool <--> ABCI
Sync <--> ABCI
Core[Dash Core RPC] --> Consensus
Reactors and P2P services
P2P services handle distinct message types: consensus votes and proposals, transaction gossip, block synchronization, evidence, peer exchange and state synchronization. Multiplexed channels share peer connections while separating these message flows. See the P2P specification for the transport and wire protocol.
Block sync brings a node up to date by downloading and verifying blocks. State sync restores an application snapshot and the consensus state needed to resume processing. When caught up, a node participates in live consensus. See block sync and state sync.
Node roles
A validator holds consensus signing keys and votes when it belongs to the
active validator set. A full node verifies and stores blocks without voting.
A seed node helps other nodes discover peers. The role is configured by
mode in config.toml; see configuration.
For the detailed consensus rules, read the consensus specification.
Install Tenderdash
Using Dashmate (Recommended Method)
Dashmate is a part of the Dash Platform and provides a comprehensive solution for installing the entire platform. We highly recommend using Dashmate for a seamless and straightforward installation process.
For detailed instructions on how to set up a node using Dashmate, please refer to the official Dash documentation: Set Up a Node.
From Binary
To download pre-built binaries, see the releases page.
From Source
Install official go into the standard location with standard PATH updates:
curl https://webinstall.dev/go | sh
source ~/.config/envman/PATH.env
Install git, cmake, and build-essential (apt) or build-base (apk) and other libs.
For Debian-based (eg. Ubuntu):
sudo apt update
sudo apt install -y git build-essential cmake libgmp-dev
For Alpine Linux:
apk add --no-cache git build-base cmake gmp-dev gmp-static
Get the source code:
git clone https://github.com/dashpay/tenderdash.git
pushd ./tenderdash/
git submodule init
git submodule update
Check out the release tag required by your network before building. Development branches may contain changes that are not compatible with a deployed network.
Build:
to put the binary in $GOPATH/bin:
make install
to put the binary in ./build:
make build
The latest Tenderdash is now installed. You can verify the installation by running:
tenderdash version
Cross-compilation
To cross-compile for ARM platform, you need to install required compilers. On Ubuntu 20.04+:
sudo apt-get install gcc-10-arm-linux-gnueabi g++-10-arm-linux-gnueabi
To start the build process, execute:
GOOS=linux GOARCH=arm make
Run
To start a one-node blockchain with a simple in-process application:
tenderdash init validator
tenderdash start --proxy-app=kvstore
Reinstall
If you already have Tenderdash installed, and you make updates, simply
make install
To upgrade, fetch and check out the release tag required by your network, then
run make install. Coordinate consensus and application upgrades with the network.
Compile with CLevelDB support
Install LevelDB (minimum version is 1.7).
Install LevelDB with snappy (optionally). Below are commands for Ubuntu:
sudo apt-get update
sudo apt install build-essential
sudo apt-get install libsnappy-dev
wget https://github.com/google/leveldb/archive/v1.20.tar.gz && \
tar -zxvf v1.20.tar.gz && \
cd leveldb-1.20/ && \
make && \
sudo cp -r out-static/lib* out-shared/lib* /usr/local/lib/ && \
cd include/ && \
sudo cp -r leveldb /usr/local/include/ && \
sudo ldconfig && \
rm -f v1.20.tar.gz
Set a database backend to cleveldb:
# config/config.toml
db_backend = "cleveldb"
To install Tenderdash, run:
CGO_LDFLAGS="-lsnappy" make install TENDERMINT_BUILD_OPTIONS=cleveldb
or run:
CGO_LDFLAGS="-lsnappy" make build TENDERMINT_BUILD_OPTIONS=cleveldb
which puts the binary in ./build.
Quick Start
Overview
This is a quick start guide. If you have a vague idea about how Tenderdash works and want to get started right away, continue. Make sure you've installed the binary. Check out install if you haven't.
Initialization
Running:
tenderdash init validator
will create the required files for a single, local node.
These files are found in $HOME/.tenderdash:
$ ls $HOME/.tenderdash
config data
$ ls $HOME/.tenderdash/config/
config.toml genesis.json node_key.json priv_validator.json
For a single, local node, no further configuration is required. Configuring a cluster is covered further below.
Local Node
Start Tenderdash with a simple in-process application:
tenderdash start --proxy-app=kvstore
Note:
kvstoreis a non persistent app, if you would like to run an application with persistence run--proxy-app=persistent_kvstore
and blocks will start to stream in:
I[01-06|01:45:15.592] Executed block module=state height=1 validTxs=0 invalidTxs=0
I[01-06|01:45:15.624] Committed state module=state height=1 txs=0 appHash=
Check the status with:
curl -s localhost:26657/status
Sending Transactions
With the KVstore app running, we can send transactions:
curl -s 'localhost:26657/broadcast_tx_commit?tx="abcd"'
and check that it worked with:
curl -s 'localhost:26657/abci_query?data="abcd"'
We can send transactions with a key and value too:
curl -s 'localhost:26657/broadcast_tx_commit?tx="name=satoshi"'
and query the key:
curl -s 'localhost:26657/abci_query?data="name"'
where the value is returned in hex.
Cluster of Nodes
Prepare four hosts with Tenderdash installed. Use the same release and genesis on every node, and restrict administrative RPC access.
Next, use the tenderdash testnet command to create four directories of config files (found in ./mytestnet) and copy each directory to the relevant machine in the cloud, so that each machine has $HOME/mytestnet/node[0-3] directory.
Before you can start the network, you'll need peers identifiers (IPs are not enough and can change). We'll refer to them as ID1, ID2, ID3, ID4.
tenderdash show-node-id --home ./mytestnet/node0
tenderdash show-node-id --home ./mytestnet/node1
tenderdash show-node-id --home ./mytestnet/node2
tenderdash show-node-id --home ./mytestnet/node3
Finally, from each machine, run:
tenderdash start --home ./mytestnet/node0 --proxy-app=kvstore --p2p.persistent-peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656"
tenderdash start --home ./mytestnet/node1 --proxy-app=kvstore --p2p.persistent-peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656"
tenderdash start --home ./mytestnet/node2 --proxy-app=kvstore --p2p.persistent-peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656"
tenderdash start --home ./mytestnet/node3 --proxy-app=kvstore --p2p.persistent-peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656"
Note that after the third node is started, blocks will start to stream in
because >2/3 of validators (defined in the genesis.json) have come online.
Persistent peers can also be specified in the config.toml. See here for more information about configuration options.
Transactions can then be sent as covered in the single, local node example above.
What is Tenderdash
Tenderdash is the consensus engine used by Dash Platform. It replicates an application across a network of validators, ordering transactions into blocks and agreeing on application state. The application implements the Application BlockChain Interface (ABCI); Tenderdash handles peer communication, transaction gossip and Byzantine fault tolerant consensus.
Consensus and validators
Validators take turns proposing blocks at each height. They vote in two stages, prevote and precommit. To commit a block, precommits for it in the same round must reach the configured voting threshold, and quorum threshold-signature recovery and verification must succeed. The voting threshold can be an explicit override or the Dash quorum-type default; changing it does not lower the quorum's signature-recovery threshold. If a round cannot commit a block, validators move to another round with a new proposer.
The usual BFT safety argument assumes less than one-third of voting power is Byzantine and a voting threshold strictly above two-thirds. Tenderdash accepts some lower thresholds, so that safety bound is not guaranteed by configuration. See the threshold definitions, system-model assumptions and threshold recovery. Safety also depends on correct key management, validation and locking rules. Progress requires enough validators to communicate and sufficiently timely message delivery; a network partition can halt block production.
Tenderdash uses BLS validator keys and quorum information from Dash Core. Dash Core ChainLocks and quorum verification connect Platform consensus to Dash's security model. See the protocol specification, validator guide and consensus specification for details.
Application boundary
An ABCI application defines transaction validity, execution and state storage. Tenderdash treats transaction contents as application-defined bytes.
flowchart LR
Client[Client] -->|RPC transactions and queries| Node[Tenderdash node]
Node <-->|P2P blocks, votes and transactions| Peers[Other nodes]
Node <-->|ABCI requests and responses| App[Deterministic application]
App --> State[Application state]
Important application calls include:
CheckTxchecks transactions before admission to the mempool.PrepareProposallets the proposer application select and prepare transactions.ProcessProposallets validator applications validate a proposed block.ExtendVoteandVerifyVoteExtensionsupport application-defined vote extensions.FinalizeBlockdelivers a decided block and persists the resulting application state.Info,InitChainandQuerysupport startup, initialization and queries.- Snapshot methods support state sync.
The Go application interface and ABCI specification define the contract. Use the interface and specification from the same release as your node.
Determinism
Every correct validator must compute the same results for the same committed input. Application execution must avoid uncontrolled sources of nondeterminism, including local clocks, unseeded randomness, concurrent access to mutable state, unordered map iteration and platform-dependent floating point behavior.
Application state changes must follow the consensus lifecycle. An application must not let unrelated external writes alter the replicated state.
Start building
Install Tenderdash, follow the quick start, and explore the application development guide. The architecture overview explains the node's components.
Overview
This section will focus on how to operate full nodes, validators and light clients.
Node Types
We will cover the various types of node types within Tenderdash.
Full Node
A full node is a node that participates in the network but will not help secure it. Full nodes can be used to store the entire state of a blockchain. For Tenderdash there are two forms of state. First, blockchain state, this represents the blocks of a blockchain. Secondly, there is Application state, this represents the state that transactions modify. The knowledge of how a transaction can modify state is not held by Tenderdash but rather the application on the other side of the ABCI boundary.
Note: If you have not read about the seperation of consensus and application please take a few minutes to read up on it as it will provide a better understanding to many of the terms we use throughout the documentation. You can find more information on the ABCI here.
As a full node operator you are providing services to the network that helps it come to consensus and others catch up to the current block. Even though a full node only helps the network come to consensus it is important to secure your node from adversarial actors. We recommend using a firewall and a proxy if possible. Running a full node can be easy, but it varies from network to network. Verify your applications documentation prior running a node.
Seed Nodes
A seed node provides a node with a list of peers which a node can connect to. When starting a node you must provide at least one type of node to be able to connect to the desired network. By providing a seed node you will be able to populate your address quickly. A seed node will not be kept as a peer but will disconnect from your node after it has provided a list of peers.
Sentry Node
A sentry node is similar to a full node in almost every way. The difference is a sentry node will have one or more private peers. These peers may be validators or other full nodes in the network. A sentry node is meant to provide a layer of security for your validator, similar to how a firewall works with a computer.
Validators
Validators sign consensus votes when their pro-tx-hash belongs to the active
Dash quorum. All validators have voting power 100; application-provided
validator-set updates determine membership. See the validator guide
and the canonical validator-set rules.
Configuration
Tenderdash Core can be configured via a TOML file in
$TMHOME/config/config.toml. Some of these parameters can be overridden by
command-line flags. For most users, the options in the ##### main base configuration options ##### are intended to be modified while config options
further below are intended for advance power users.
Options
The default configuration file create by tenderdash init has all
the parameters set with their default values. It will look something
like the file below, however, double check by inspecting the
config.toml created with your version of tenderdash installed:
# This is a TOML config file.
# For more information, see https://github.com/toml-lang/toml
# NOTE: Any path below can be absolute (e.g. "/var/myawesomeapp/data") or
# relative to the home directory (e.g. "data"). The home directory is
# "$HOME/.tenderdash" by default, but could be changed via $TMHOME env variable
# or --home cmd flag.
#######################################################################
### Main Base Config Options ###
#######################################################################
# TCP or UNIX socket address of the ABCI application,
# or the name of an ABCI application compiled in with the Tenderdash binary
proxy-app = "tcp://127.0.0.1:26658"
# A custom human readable name for this node
moniker = "sidewinder"
# Mode of Node: full | validator | seed
# * validator node
# - all reactors
# - with priv_validator_key.json, priv_validator_state.json
# * full node
# - all reactors
# - No priv_validator_key.json, priv_validator_state.json
# * seed node
# - only P2P, PEX Reactor
# - No priv_validator_key.json, priv_validator_state.json
mode = "validator"
# Database backend: goleveldb
# * goleveldb (github.com/syndtr/goleveldb - most popular implementation)
# - pure go
# - stable
db-backend = "goleveldb"
# Database directory
db-dir = "data"
# Output level for logging, including package level options
log-level = "info"
# Output format: 'plain' (colored text) or 'json'
log-format = "plain"
##### additional base config options #####
# Path to the JSON file containing the initial validator set and other meta data
genesis-file = "config/genesis.json"
# Path to the JSON file containing the private key to use for node authentication in the p2p protocol
node-key-file = "config/node_key.json"
# Mechanism to connect to the ABCI application: socket | grpc
abci = "socket"
# If true, query the ABCI app on connecting to a new peer
# so the app can decide if we should keep the connection or not
filter-peers = false
# If set to positive duration, deadlock detection is enabled and set to the given time.
# Use 0 to disable.
# Default: 0
deadlock-detection = "0s"
# If true, block store and state store writes return before they have reached
# the disk. This exists to take the cost of fsync out of a block sync benchmark
# and nothing else. Never enable it on a node whose data matters: a power loss
# can leave the stores behind the application, and the node will refuse to start.
# Default: false
unsafe-no-fsync = false
#######################################################
### Priv Validator Configuration ###
#######################################################
[priv-validator]
# Path to the JSON file containing the private key to use as a validator in the consensus protocol
key-file = "config/priv_validator_key.json"
# Path to the JSON file containing the last sign state of a validator
state-file = "data/priv_validator_state.json"
# TCP or UNIX socket address for Tenderdash to listen on for
# connections from an external PrivValidator process
# when the listenAddr is prefixed with grpc instead of tcp it will use the gRPC Client
laddr = ""
# Path to the client certificate generated while creating needed files for secure connection.
# If a remote validator address is provided but no certificate, the connection will be insecure
client-certificate-file = ""
# Client key generated while creating certificates for secure connection
client-key-file = ""
# Path to the Root Certificate Authority used to sign both client and server certificates
root-ca-file = ""
#######################################################################
### Advanced Configuration Options ###
#######################################################################
#######################################################
### RPC Server Configuration Options ###
#######################################################
[rpc]
# TCP or UNIX socket address for the RPC server to listen on
laddr = "tcp://127.0.0.1:26657"
# A list of origins a cross-domain request can be executed from
# Default value '[]' disables cors support
# Use '["*"]' to allow any origin
cors-allowed-origins = []
# This list also permits browser WebSocket origins for `tenderdash light <chainID>`.
# An empty list rejects all requests carrying Origin, including same-host requests.
# Clients without Origin remain supported. Add explicit browser origins as needed;
# wildcard matching follows RPC CORS rules, and ["*"] explicitly allows any origin.
# A list of methods the client is allowed to use with cross-domain requests
cors-allowed-methods = ["HEAD", "GET", "POST", ]
# A list of non simple headers the client is allowed to use with cross-domain requests
cors-allowed-headers = ["Origin", "Accept", "Content-Type", "X-Requested-With", "X-Server-Time", ]
# Activate unsafe RPC commands like /dial-seeds and /unsafe-flush-mempool
unsafe = false
# Maximum number of simultaneous connections (including WebSocket).
# If you want to accept a larger number than the default, make sure
# you increase your OS limits.
# 0 - unlimited.
# Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}
# 1024 - 40 - 10 - 50 = 924 = ~900
max-open-connections = 900
# Maximum number of unique clientIDs that can /subscribe
# If you're using /broadcast_tx_commit, set to the estimated maximum number
# of broadcast_tx_commit calls per block.
max-subscription-clients = 100
# Maximum number of unique queries a given client can /subscribe to
# If you're using a Local RPC client and /broadcast_tx_commit, set this
# to the estimated maximum number of broadcast_tx_commit calls per block.
max-subscriptions-per-client = 5
# If true, disable the websocket interface to the RPC service. This has
# the effect of disabling the /subscribe, /unsubscribe, and /unsubscribe_all
# methods for event subscription.
#
# Event log settings; check support in your Tenderdash release.
experimental-disable-websocket = false
# The time window size for the event log. All events up to this long before
# the latest (up to EventLogMaxItems) will be available for subscribers to
# fetch via the /events method. If 0 (the default) the event log and the
# /events RPC method are disabled.
event-log-window-size = "0s"
# The maxiumum number of events that may be retained by the event log. If
# this value is 0, no upper limit is set. Otherwise, items in excess of
# this number will be discarded from the event log.
#
# Warning: This setting is a safety valve. Setting it too low may cause
# subscribers to miss events. Try to choose a value higher than the
# maximum worst-case expected event load within the chosen window size in
# ordinary operation.
#
# For example, if the window size is 10 minutes and the node typically
# averages 1000 events per ten minutes, but with occasional known spikes of
# up to 2000, choose a value > 2000.
event-log-max-items = 0
# How long to wait for a tx to be committed during /broadcast_tx_commit.
# WARNING: Using a value larger than 10s will result in increasing the
# global HTTP write timeout, which applies to all connections and endpoints.
# See https://github.com/tendermint/tendermint/issues/3435
timeout-broadcast-tx-commit = "10s"
# Maximum size of request body, in bytes
max-body-bytes = 1000000
# Maximum size of request header, in bytes
max-header-bytes = 1048576
# The path to a file containing certificate that is used to create the HTTPS server.
# Might be either absolute path or path related to Tenderdash's config directory.
# If the certificate is signed by a certificate authority,
# the certFile should be the concatenation of the server's certificate, any intermediates,
# and the CA's certificate.
# NOTE: both tls-cert-file and tls-key-file must be present for Tenderdash to create HTTPS server.
# Otherwise, HTTP server is run.
tls-cert-file = ""
# The path to a file containing matching private key that is used to create the HTTPS server.
# Might be either absolute path or path related to Tenderdash's config directory.
# NOTE: both tls-cert-file and tls-key-file must be present for Tenderdash to create HTTPS server.
# Otherwise, HTTP server is run.
tls-key-file = ""
# pprof listen address (https://golang.org/pkg/net/http/pprof)
pprof-laddr = ""
#######################################################
### P2P Configuration Options ###
#######################################################
[p2p]
# Select the p2p internal queue
queue-type = "priority"
# Address to listen for incoming connections
laddr = "tcp://0.0.0.0:26656"
# Address to advertise to peers for them to dial
# If empty, will use the same port as the laddr,
# and will introspect on the listener or use UPnP
# to figure out the address. ip and port are required
# example: 159.89.10.97:26656
external-address = ""
# Comma separated list of seed nodes to connect to
# We only use these if we can’t connect to peers in the addrbook
# NOTE: not used by the new PEX reactor. Please use BootstrapPeers instead.
# TODO: Remove once p2p refactor is complete
# ref: https:#github.com/tendermint/tendermint/issues/5670
seeds = ""
# Comma separated list of peers to be added to the peer store
# on startup. Either BootstrapPeers or PersistentPeers are
# needed for peer discovery
bootstrap-peers = ""
# Comma separated list of nodes to keep persistent connections to
persistent-peers = ""
# UPNP port forwarding
upnp = false
# Maximum number of connections (inbound and outbound).
max-connections = 64
# Rate limits the number of incoming connection attempts per IP address.
max-incoming-connection-attempts = 100
# Set true to enable the peer-exchange reactor
pex = true
# Comma separated list of peer IDs to keep private (will not be gossiped to other peers)
# Warning: IPs will be exposed at /net_info, for more information https://github.com/tendermint/tendermint/issues/3055
private-peer-ids = ""
# Toggle to disable guard against peers connecting from the same ip.
allow-duplicate-ip = false
# Peer connection configuration.
handshake-timeout = "20s"
dial-timeout = "3s"
# Time to wait before flushing messages out on the connection
# TODO: Remove once MConnConnection is removed.
flush-throttle-timeout = "100ms"
# Maximum size of a message packet payload, in bytes; must be greater than 0
# TODO: Remove once MConnConnection is removed.
max-packet-msg-payload-size = 1400
# Rate at which packets can be sent, in bytes/second
# TODO: Remove once MConnConnection is removed.
send-rate = 5120000
# Rate at which packets can be received, in bytes/second
# TODO: Remove once MConnConnection is removed.
recv-rate = 5120000
#######################################################
### Mempool Configuration Option ###
#######################################################
[mempool]
recheck = true
broadcast = true
# Maximum number of transactions in the mempool
size = 5000
# Limit the total size of all txs in the mempool.
# This only accounts for raw transactions (e.g. given 1MB transactions and
# max-txs-bytes=5MB, mempool will only accept 5 transactions).
max-txs-bytes = 1073741824
# Size of the cache (used to filter transactions we saw earlier) in transactions
cache-size = 10000
# Do not remove invalid transactions from the cache (default: false)
# Set to true if it's not possible for any invalid transaction to become valid
# again in the future.
# When true, resubmitting a rejected transaction over RPC returns the CheckTx
# result that rejected it, as long as it stays in the cache.
keep-invalid-txs-in-cache = false
# Maximum size of a single transaction.
# NOTE: the max size of a tx transmitted over the network is {max-tx-bytes}.
max-tx-bytes = 1048576
# Maximum size of a batch of transactions to send to a peer
# Including space needed by encoding (one varint per transaction).
# XXX: Unused due to https://github.com/tendermint/tendermint/issues/5796
max-batch-bytes = 0
# ttl-duration, if non-zero, defines the maximum amount of time a transaction
# can exist for in the mempool.
#
# Note, if ttl-num-blocks is also defined, a transaction will be removed if it
# has existed in the mempool at least ttl-num-blocks number of blocks or if it's
# insertion time into the mempool is beyond ttl-duration.
ttl-duration = "0s"
# ttl-num-blocks, if non-zero, defines the maximum number of blocks a transaction
# can exist for in the mempool.
#
# Note, if ttl-duration is also defined, a transaction will be removed if it
# has existed in the mempool at least ttl-num-blocks number of blocks or if
# it's insertion time into the mempool is beyond ttl-duration.
ttl-num-blocks = 0
#######################################################
### State Sync Configuration Options ###
#######################################################
[statesync]
# State sync rapidly bootstraps a new node by discovering, fetching, and restoring a state machine
# snapshot from peers instead of fetching and replaying historical blocks. Requires some peers in
# the network to take and serve state machine snapshots. State sync is not attempted if the node
# has any local state (LastBlockHeight > 0). The node will have a truncated block history,
# starting from the height of the snapshot.
enable = false
# State sync uses light client verification to verify state. This can be done either through the
# P2P layer or RPC layer. Set this to true to use the P2P layer. If false (default), RPC layer
# will be used.
use-p2p = false
# If using RPC, at least two addresses need to be provided. They should be compatible with net.Dial,
# for example: "host.example.com:2125"
rpc-servers = ""
# Time to spend discovering snapshots before initiating a restore.
discovery-time = "15s"
# Number of completed snapshot discovery sweeps. When retries are exhausted, the node will
# fall back to the regular block sync. Set to 0 to retry
# indefinitely, never falling back to block sync. Default is 3.
# Note that in pessimistic case, it will take at least (discovery-time * retries) before
# falling back to block sync. Each sweep visits at most 1,024 peers in batches of 16;
# each batch takes one discovery interval. New peer arrivals do not extend a sweep.
retries = 3
# Temporary directory for state sync snapshot chunks, defaults to os.TempDir().
# The synchronizer will create a new, randomly named directory within this directory
# and remove it when the sync is complete.
temp-dir = ""
# The timeout duration before re-requesting a chunk, possibly from a different
# peer (default: 15 seconds).
chunk-request-timeout = "15s"
# The number of concurrent chunk and block fetchers to run (default: 4).
fetchers = "4"
#######################################################
### Consensus Configuration Options ###
#######################################################
[consensus]
wal-file = "data/cs.wal/wal"
# Legacy compatibility setting; currently unused by consensus.
# This does not provide double-sign protection. Preserve the signer's state
# and never run two active signers with the same consensus identity.
double-sign-check-height = 0
# EmptyBlocks mode and possible interval between empty blocks
create-empty-blocks = true
create-empty-blocks-interval = "0s"
# Reactor sleep duration parameters. peer-gossip-sleep-duration also sets the
# tick rate a lagging peer's catch-up block-part replay is metered against
# (a fixed 500ms internal throttle): raising this above 500ms disables that
# throttle. peer-query-maj23-sleep-duration also sets how often the node
# re-announces its round state to each peer (recovery latency after a lost
# announcement). Either duration under 1ms is run at 1ms, which only prevents a
# busy loop: well below the defaults, peers drop this node's state updates over
# their per-peer allowance.
peer-gossip-sleep-duration = "100ms"
peer-query-maj23-sleep-duration = "2s"
### Unsafe Timeout Overrides ###
# These fields provide temporary overrides for the Timeout consensus parameters.
# Use of these parameters is strongly discouraged. Using these parameters may have serious
# liveness implications for the validator and for the chain.
#
# Because they make a single node disagree with the rest of the network about
# timing, they are not intended for normal operation.
# This field provides an unsafe override of the Propose timeout consensus parameter.
# This field configures how long the consensus engine will wait for a proposal block before prevoting nil.
# If this field is set to a value greater than 0, it will take effect.
# unsafe-propose-timeout-override = 0s
# This field provides an unsafe override of the ProposeDelta timeout consensus parameter.
# This field configures how much the propose timeout increases with each round.
# If this field is set to a value greater than 0, it will take effect.
# unsafe-propose-timeout-delta-override = 0s
# This field provides an unsafe override of the Vote timeout consensus parameter.
# This field configures how long the consensus engine will wait after
# receiving the configured voting threshold in a round.
# If this field is set to a value greater than 0, it will take effect.
# unsafe-vote-timeout-override = 0s
# This field provides an unsafe override of the VoteDelta timeout consensus parameter.
# This field configures how much the vote timeout increases with each round.
# If this field is set to a value greater than 0, it will take effect.
# unsafe-vote-timeout-delta-override = 0s
#######################################################
### Transaction Indexer Configuration Options ###
#######################################################
[tx-index]
# The backend database list to back the indexer.
# An empty list or a list containing "null" disables indexing.
#
# The application will set which txs to index. In some cases a node operator will be able
# to decide which txs to index based on configuration set in the application.
#
# Options:
# 1) "null" (default) - no indexer services.
# 2) "kv" - an indexer backed by key-value storage (see DBBackend).
# When "kv" is chosen "tx.height" and "tx.hash" will always be indexed.
indexer = ["kv"]
#######################################################
### Instrumentation Configuration Options ###
#######################################################
[instrumentation]
# When true, Prometheus metrics are served under /metrics on
# PrometheusListenAddr.
# Check out the documentation for the list of available metrics.
prometheus = false
# Address to listen for Prometheus collector(s) connections
prometheus-listen-addr = ":26660"
# Maximum number of simultaneous connections.
# If you want to accept a larger number than the default, make sure
# you increase your OS limits.
# 0 - unlimited.
max-open-connections = 3
# Instrumentation namespace
namespace = "tendermint"
Empty blocks
create-empty-blocks = false makes round 0 wait for transactions above the
initial height. A positive create-empty-blocks-interval starts a proposal after
that interval even without transactions. Later rounds do not wait for
transactions, except on a chain with consensus version 2, where a round entered
because the validators skipped a silent proposer's waiting round may wait too,
unless the node holds a locked or valid block, the mempool has already reported
transactions at this height, a polka exists at the height, or a prevote of that
round was recorded; its interval counts from the start of the height. See
Proposer liveness and round skipping.
The application can bypass that wait for the next height by setting
ResponseFinalizeBlock.propose_next_block_immediately. The hint is local,
consumed once and not persisted across restarts. There is no commit timeout
before the next height starts.
Tenderdash uses same-block execution: block H's AppHash describes the state
after executing H. An application hash change does not require an additional
"proof block" at H+1.
See the canonical empty-block rules, execution order and timeout parameters.
P2P settings
This section will cover settings within the p2p section of the config.toml.
external-address= is the address that will be advertised for other nodes to use. We recommend setting this field with your public IP and p2p port.-
We recommend setting an external address. When used in a private network, Tenderdash Core currently doesn't advertise the node's public address. There is active and ongoing work to improve the P2P system, but this is a helpful workaround for now.
-
persistent-peers= is a list of comma separated peers that you will always want to be connected to. If you're already connected to the maximum number of peers, persistent peers will not be added.pex= turns the peer exchange reactor on or off. Validator node will want thepexturned off so it would not begin gossiping to unknown peers on the network. PeX can also be turned off for statically configured networks with fixed network connectivity. For full nodes on open, dynamic networks, it should be turned on.allowlist-only= if true, only peers frompersistent-peersandbootstrap-peersare allowed to connect (inbound and outbound).private-peer-ids= is a comma-separated list of node ids that will not be exposed to other peers (i.e., you will not tell other peers about the ids in this list). This can be filled with a validator's node id.
The P2P configuration includes parameters for the router and legacy networking implementations. Check the configuration generated by your installed version.
We will cover the new and deprecated parameters below.
New Parameters
There are three new parameters, which are enabled if use-legacy is set to false.
queue-type= sets a type of queue to use in the p2p layer. There are three options availablefifo,priorityandwdrr. The default is prioritybootstrap-peers= is a list of comma seperated peers which will be used to bootstrap the address book.max-connections= is the max amount of allowed inbound and outbound connections.
Deprecated Parameters
These legacy settings differ from the router settings above. Use the settings supported by the networking implementation in your Tenderdash release.
max-num-inbound-peers= is the maximum number of peers you will accept inbound connections from at one time (where they dial your address and initiate the connection). This was replaced bymax-connectionsmax-num-outbound-peers= is the maximum number of peers you will initiate outbound connects to at one time (where you dial their address and initiate the connection).This was replaced bymax-connectionsunconditional-peer-ids= is similar topersistent-peersexcept that these peers will be connected to even if you are already connected to the maximum number of peers. This can be a validator node ID on your sentry node. Deprecatedseeds= is a list of comma separated seed nodes that you will connect upon a start and ask for peers. A seed node is a node that does not participate in consensus but only helps propagate peers to nodes in the networks Deprecated, replaced by bootstrap peers
Indexing Settings
Operators can configure indexing via the [tx-index] section. The indexer
field takes a series of supported indexers. If null is included, indexing will
be turned off. All configured indexer names must still be supported.
Supported Indexers
KV
The kv indexer type is an embedded key-value store supported by the main
underlying Tenderdash database. Using the kv indexer type allows you to query
for block and transaction events directly against Tenderdash's RPC. However, the
query syntax is limited to the supported event-query operators.
Removed PostgreSQL indexer
The psql event indexer is no longer supported. Selecting it, including alongside
kv or null, returns a configuration error. Choose indexer = ["kv"] for local
indexing and RPC searches, or ["null"] to disable indexing. The obsolete
psql-conn key is ignored, so existing Dashmate configurations with an empty
value remain compatible.
Existing PostgreSQL data is not deleted or migrated, and SQL consumers no longer
receive new events. To populate KV indexes, explicitly run tenderdash reindex-event against the retained local blocks and results after selecting kv.
Pruned history cannot be reconstructed by this command.
Unsafe Consensus Timeout Overrides
Tenderdash provides a set of unsafe overrides for the consensus timing parameters. They are a safety measure for unusual timing issues, letting an operator override the timings for a single node. Because they make one node disagree with the rest of the network about timing, they are unsafe and are not intended for normal operation.
unsafe-propose-timeout-override: How long the Tenderdash consensus engine will wait for a proposal block before prevoting nil.unsafe-propose-timeout-delta-override: How much the propose timeout increases with each round.unsafe-vote-timeout-override: How long the consensus engine will wait after receiving the configured voting threshold in a round.unsafe-vote-timeout-delta-override: How much the vote timeout increases with each round.
Configure a Light Client
Unsupported. The Tenderdash light client is currently unsupported. The implementation details below are provided for reference.
Tenderdash includes a tenderdash light command that runs an RPC proxy backed by
a verifying light client. Requests with supported proofs can be checked before
results are returned. Other RPC methods do not automatically gain proof verification.
The proxy requires a chain ID, a primary Tenderdash RPC provider and witness providers. The current implementation uses Dash Core verification for quorum signatures, so it also requires access to the appropriate Dash Core RPC service. It does not use upstream trust-height and trust-hash flags.
Inspect the options for your installed release:
tenderdash light --help
Set --primary, --witnesses and the Dash Core connection settings for your
network. Use --dir to select an explicit directory for the light client's
local database and --laddr to control the proxy listen address.
Keep Core credentials private and restrict RPC network access. Ensure that the
application's query proof format is compatible with the proxy before relying
on /abci_query verification.
See the light client overview, command implementation and client implementation for the supported configuration and verification behavior.
Logging
Logging adds detail and allows the node operator to better identify what they are looking for. Tenderdash supports log levels on a global and per-module basis. This allows the node operator to see only the information they need and the developer to hone in on specific changes they are working on.
Configuring Log Levels
There are three log levels, info, debug and error. These can be configured either through the command line via tenderdash start --log-level "" or within the config.toml file.
infoInfo represents an informational message. It is used to show that modules have started, stopped and how they are functioning.debugDebug is used to trace various calls or problems. Debug is used widely throughout a codebase and can lead to overly verbose logging.errorError represents something that has gone wrong. An error log can represent a potential problem that can lead to a node halt.
Within the config.toml:
# Output level for logging, including package level options
log-level = "info"
Via the command line:
tenderdash start --log-level "info"
List of modules
Here is the list of modules you may encounter in Tenderdash's log and a little overview what they do.
abci-clientAs mentioned in Application Architecture Guide, Tenderdash acts as an ABCI client with respect to the application and maintains 3 connections: mempool, consensus and query. The code used by Tenderdash Core can be found here.blockchainProvides storage, pool (a group of peers), and reactor for both storing and exchanging blocks between peers.consensusThe heart of Tenderdash core, which is the implementation of the consensus algorithm. Includes two "submodules":wal(write-ahead logging) for ensuring data integrity andreplayto replay blocks and messages on recovery from a crash. here. You can subscribe to them by callingsubscribeRPC method. Refer to RPC docs for additional information.mempoolMempool module handles all incoming transactions, whenever they are coming from peers or the application.p2pProvides an abstraction around peer-to-peer communication. For more details, please check out the README.rpc-serverRPC server. For implementation details, please read the doc.go.stateRepresents the latest state and execution submodule, which executes blocks against the application.statesyncProvides a way to quickly sync a node with pruned history.
Walkabout example
We first create three connections (mempool, consensus and query) to the
application (running kvstore locally in this case).
I[10-04|13:54:27.364] Starting multiAppConn module=proxy impl=multiAppConn
I[10-04|13:54:27.366] Starting localClient module=abci-client connection=query impl=localClient
I[10-04|13:54:27.366] Starting localClient module=abci-client connection=mempool impl=localClient
I[10-04|13:54:27.367] Starting localClient module=abci-client connection=consensus impl=localClient
Then Tenderdash Core and the application perform a handshake.
I[10-04|13:54:27.367] ABCI Handshake module=consensus appHeight=90 appHash=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD
I[10-04|13:54:27.368] ABCI Replay Blocks module=consensus appHeight=90 storeHeight=90 stateHeight=90
I[10-04|13:54:27.368] Completed ABCI Handshake - Tenderdash and App are synced module=consensus appHeight=90 appHash=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD
After that, we start a few more things like the event switch, reactors, and perform UPNP discover in order to detect the IP address.
I[10-04|13:54:27.374] Starting EventSwitch module=types impl=EventSwitch
I[10-04|13:54:27.375] This node is a validator module=consensus
I[10-04|13:54:27.379] Starting Node module=main impl=Node
I[10-04|13:54:27.381] Local listener module=p2p ip=:: port=26656
I[10-04|13:54:27.382] Getting UPNP external address module=p2p
I[10-04|13:54:30.386] Could not perform UPNP discover module=p2p err="write udp4 0.0.0.0:38238->239.255.255.250:1900: i/o timeout"
I[10-04|13:54:30.386] Starting DefaultListener module=p2p impl=Listener(@10.0.2.15:26656)
I[10-04|13:54:30.387] Starting P2P Switch module=p2p impl="P2P Switch"
I[10-04|13:54:30.387] Starting MempoolReactor module=mempool impl=MempoolReactor
I[10-04|13:54:30.387] Starting BlockchainReactor module=blockchain impl=BlockchainReactor
I[10-04|13:54:30.387] Starting ConsensusReactor module=consensus impl=ConsensusReactor
I[10-04|13:54:30.387] ConsensusReactor module=consensus fastSync=false
I[10-04|13:54:30.387] Starting ConsensusState module=consensus impl=ConsensusState
I[10-04|13:54:30.387] Starting WAL module=consensus wal=/home/vagrant/.tenderdash/data/cs.wal/wal impl=WAL
I[10-04|13:54:30.388] Starting TimeoutTicker module=consensus impl=TimeoutTicker
Notice the second row where Tenderdash Core reports that "This node is a validator". It also could be just an observer (regular node).
Next we replay all the messages from the WAL.
I[10-04|13:54:30.390] Catchup by replaying consensus messages module=consensus height=91
I[10-04|13:54:30.390] Replay: New Step module=consensus height=91 round=0 step=RoundStepNewHeight
I[10-04|13:54:30.390] Replay: Done module=consensus
"Started node" message signals that everything is ready for work.
I[10-04|13:54:30.391] Starting RPC HTTP server on tcp socket 0.0.0.0:26657 module=rpc-server
I[10-04|13:54:30.392] Started node module=main nodeInfo="NodeInfo{id: DF22D7C92C91082324A1312F092AA1DA197FA598DBBFB6526E, moniker: anonymous, network: test-chain-3MNw2N [remote , listen 10.0.2.15:26656], version: 0.11.0-10f361fc ([wire_version=0.6.2 p2p_version=0.5.0 consensus_version=v1/0.2.2 rpc_version=0.7.0/3 tx_index=on rpc_addr=tcp://0.0.0.0:26657])}"
Next follows a standard block creation cycle, where we enter a new round, propose a block, receive more than 2/3 of prevotes, then precommits and finally have a chance to commit a block. For details, please refer to Byzantine Consensus Algorithm.
I[10-04|13:54:30.393] enterNewRound(91/0). Current: 91/0/RoundStepNewHeight module=consensus
I[10-04|13:54:30.393] enterPropose(91/0). Current: 91/0/RoundStepNewRound module=consensus
I[10-04|13:54:30.393] enterPropose: Our turn to propose module=consensus proposer=125B0E3C5512F5C2B0E1109E31885C4511570C42 privValidator="PrivValidator{125B0E3C5512F5C2B0E1109E31885C4511570C42 LH:90, LR:0, LS:3}"
I[10-04|13:54:30.394] Signed proposal module=consensus height=91 round=0 proposal="Proposal{91/0 1:21B79872514F (-1,:0:000000000000) {/10EDEDD7C84E.../}}"
I[10-04|13:54:30.397] Received complete proposal block module=consensus height=91 hash=F671D562C7B9242900A286E1882EE64E5556FE9E
I[10-04|13:54:30.397] enterPrevote(91/0). Current: 91/0/RoundStepPropose module=consensus
I[10-04|13:54:30.397] enterPrevote: ProposalBlock is valid module=consensus height=91 round=0
I[10-04|13:54:30.398] Signed and pushed vote module=consensus height=91 round=0 vote="Vote{0:125B0E3C5512 91/00/1(Prevote) F671D562C7B9 {/89047FFC21D8.../}}" err=null
I[10-04|13:54:30.401] Added to prevote module=consensus vote="Vote{0:125B0E3C5512 91/00/1(Prevote) F671D562C7B9 {/89047FFC21D8.../}}" prevotes="VoteSet{H:91 R:0 T:1 +2/3:F671D562C7B9242900A286E1882EE64E5556FE9E:1:21B79872514F BA{1:X} map[]}"
I[10-04|13:54:30.401] enterPrecommit(91/0). Current: 91/0/RoundStepPrevote module=consensus
I[10-04|13:54:30.401] enterPrecommit: +2/3 prevoted proposal block. Locking module=consensus hash=F671D562C7B9242900A286E1882EE64E5556FE9E
I[10-04|13:54:30.402] Signed and pushed vote module=consensus height=91 round=0 vote="Vote{0:125B0E3C5512 91/00/2(Precommit) F671D562C7B9 {/80533478E41A.../}}" err=null
I[10-04|13:54:30.404] Added to precommit module=consensus vote="Vote{0:125B0E3C5512 91/00/2(Precommit) F671D562C7B9 {/80533478E41A.../}}" precommits="VoteSet{H:91 R:0 T:2 +2/3:F671D562C7B9242900A286E1882EE64E5556FE9E:1:21B79872514F BA{1:X} map[]}"
I[10-04|13:54:30.404] enterCommit(91/0). Current: 91/0/RoundStepPrecommit module=consensus
I[10-04|13:54:30.405] Finalizing commit of block with 0 txs module=consensus height=91 hash=F671D562C7B9242900A286E1882EE64E5556FE9E root=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD
I[10-04|13:54:30.405] Block{
Header{
ChainID: test-chain-3MNw2N
Height: 91
Time: 2017-10-04 13:54:30.393 +0000 UTC
NumTxs: 0
LastBlockID: F15AB8BEF9A6AAB07E457A6E16BC410546AA4DC6:1:D505DA273544
LastCommit: 56FEF2EFDB8B37E9C6E6D635749DF3169D5F005D
Data:
Validators: CE25FBFF2E10C0D51AA1A07C064A96931BC8B297
App: E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD
}#F671D562C7B9242900A286E1882EE64E5556FE9E
Data{
}#
Commit{
BlockID: F15AB8BEF9A6AAB07E457A6E16BC410546AA4DC6:1:D505DA273544
Precommits: Vote{0:125B0E3C5512 90/00/2(Precommit) F15AB8BEF9A6 {/FE98E2B956F0.../}}
}#56FEF2EFDB8B37E9C6E6D635749DF3169D5F005D
}#F671D562C7B9242900A286E1882EE64E5556FE9E module=consensus
I[10-04|13:54:30.408] Executed block module=state height=91 validTxs=0 invalidTxs=0
I[10-04|13:54:30.410] Committed state module=state height=91 txs=0 hash=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD
I[10-04|13:54:30.410] Recheck txs module=mempool numtxs=0 height=91
Metrics
Tenderdash can report and serve the Prometheus metrics, which in their turn can be consumed by Prometheus collector(s).
This functionality is disabled by default.
To enable the Prometheus metrics, set instrumentation.prometheus=true in your
config file. Metrics will be served under /metrics on 26660 port by default.
Listen address can be changed in the config file (see
instrumentation.prometheus\_listen\_addr).
List of available metrics
The following metrics are available:
| Name | Type | Tags | Description |
|---|---|---|---|
| abci_connection_method_timing | Histogram | method, type | Timings for each of the ABCI methods |
| consensus_height | Gauge | Height of the chain | |
| consensus_validators | Gauge | Number of validators | |
| consensus_validators_power | Gauge | Total voting power of all validators | |
| consensus_validator_power | Gauge | Voting power of the node if in the validator set | |
| consensus_validator_last_signed_height | Gauge | Last height the node signed a block, if the node is a validator | |
| consensus_validator_missed_blocks | Gauge | Total amount of blocks missed for the node, if the node is a validator | |
| consensus_missing_validators | Gauge | Number of validators who did not sign | |
| consensus_missing_validators_power | Gauge | Total voting power of the missing validators | |
| consensus_byzantine_validators | Gauge | Number of validators who tried to double sign | |
| consensus_byzantine_validators_power | Gauge | Total voting power of the byzantine validators | |
| consensus_block_interval_seconds | Histogram | Time between this and last block (Block.Header.Time) in seconds | |
| consensus_rounds | Gauge | Number of rounds | |
| consensus_num_txs | Gauge | Number of transactions | |
| consensus_total_txs | Gauge | Total number of transactions committed | |
| consensus_block_parts | Counter | peer_id | number of blockparts transmitted by peer |
| consensus_latest_block_height | gauge | /status sync_info number | |
| consensus_fast_syncing | gauge | either 0 (not fast syncing) or 1 (syncing) | |
| consensus_state_syncing | gauge | either 0 (not state syncing) or 1 (syncing) | |
| consensus_block_size_bytes | Gauge | Block size in bytes | |
| consensus_step_duration | Histogram | step | Histogram of durations for each step in the consensus protocol |
| consensus_block_gossip_receive_latency | Histogram | Histogram of time taken to receive a block in seconds, measure between when a new block is first discovered to when the block is completed | |
| consensus_block_gossip_parts_received | Counter | matches_current | Number of block parts received by the node |
| consensus_block_sync_apply_stage_duration | Histogram | stage | Time a block spent in each stage of the block sync applier, in ms: decode, partset, verify_commit, verify_block, save, exec, and wait (idle time between blocks) |
| consensus_quorum_prevote_delay | Gauge | Interval in seconds between the proposal timestamp and the timestamp of the earliest prevote that achieved a quorum | |
| consensus_full_prevote_delay | Gauge | Interval in seconds between the proposal timestamp and the timestamp of the latest prevote in a round where all validators voted | |
| consensus_proposal_timestamp_difference | Histogram | Difference between the timestamp in the proposal message and the local time of the validator at the time it received the message | |
| consensus_vote_extension_receive_count | Counter | status | Number of vote extensions received |
| consensus_proposal_receive_count | Counter | status | Total number of proposals received by the node since process start |
| consensus_proposal_create_count | Counter | Total number of proposals created by the node since process start | |
| consensus_round_voting_power_percent | Gauge | vote_type | A value between 0 and 1.0 representing the percentage of the total voting power per vote type received within a round |
| consensus_late_votes | Counter | vote_type | Number of votes received by the node since process start that correspond to earlier heights and rounds than this node is currently in. |
| consensus_verification_budget_drops | Counter | Peer Vote and Commit messages dropped because the signature-verification budget was exhausted | |
| consensus_peer_lane_drops | Counter | Queued peer consensus messages dropped by the per-peer scheduler (backlog full or peer disconnected) | |
| consensus_block_part_proof_drops | Counter | Peer block parts dropped before proof verification because the sender's invalid-proof allowance was spent | |
| consensus_state_channel_drops | Counter | State and VoteSetBits channel messages dropped over the per-peer or node-wide ceiling | |
| consensus_proposal_verify_failures | Counter | Peer proposals whose signature failed verification | |
| consensus_commit_verify_failures | Counter | reason | Commits (from peers, block sync or this node's own precommits) refused by verification; reason is quorum_hash, extension_count, extensions_rejected, invalid_signature, budget or other |
| consensus_peer_vote_verify_latency_seconds | Histogram | Seconds an accepted peer vote spent from being queued by the reactor to being added to the vote set | |
| consensus_verification_budget_saturation | Gauge | Fraction of the node-wide verification budget still available at a peer-message check (1.0 full/idle, 0.0 drained/under pressure) | |
| consensus_peer_lane_active_count | Gauge | Number of per-peer lanes currently holding queued messages and taking turns in the scheduler | |
| consensus_peer_lane_max_depth | Gauge | Largest number of messages queued in any single peer lane | |
| consensus_proposer_heartbeats_sent | Counter | Proposer heartbeats this node signed and broadcast while waiting for transactions as proposer (consensus version 2) | |
| consensus_proposer_heartbeats_received | Counter | New, verified proposer heartbeats accepted for the round this node waits in | |
| consensus_proposer_silences | Counter | Rounds in which the proposer went silent while this validator waited for transactions | |
| consensus_round_skips | Counter | Rounds this node moved past on a round skip certificate from +2/3 of the validators (by default; configuration can change the threshold) | |
| evidence_pool_num_evidence | Gauge | Number of evidence in the evidence pool | |
| evidence_pool_dropped_evidence | Counter | reason | Inbound gossip evidence refused before verification, labeled by reason (structural, oversized, already_known, height_window, node_budget, peer_budget) |
| p2p_peers | Gauge | Number of peers node's connected to | |
| p2p_peer_receive_bytes_total | Counter | peer_id, chID | number of bytes per channel received from a given peer |
| p2p_peer_send_bytes_total | Counter | peer_id, chID | number of bytes per channel sent to a given peer |
| p2p_peer_pending_send_bytes | Gauge | peer_id | number of pending bytes to be sent to a given peer |
| p2p_router_peer_queue_recv | Histogram | The time taken to read off of a peer's queue before sending on the connection | |
| p2p_router_peer_queue_send | Histogram | The time taken to send on a peer's queue which will later be sent on the connection | |
| p2p_router_channel_queue_send | Histogram | The time taken to send on a p2p channel's queue which will later be consumed by the corresponding service | |
| p2p_router_channel_queue_dropped_msgs | Counter | ch_id | The number of messages dropped from a peer's queue for a specific p2p channel |
| p2p_peer_queue_msg_size | Gauge | ch_id | The size of messages sent over a peer's queue for a specific p2p channel |
| mempool_size | Gauge | Number of uncommitted transactions | |
| mempool_tx_size_bytes | Histogram | transaction sizes in bytes | |
| mempool_failed_txs | Counter | number of failed transactions | |
| mempool_recheck_times | Counter | number of transactions rechecked in the mempool | |
| state_block_processing_time | Histogram | time between BeginBlock and EndBlock in ms | |
| state_consensus_param_updates | Counter | number of consensus parameter updates returned by the application since process start | |
| state_validator_set_updates | Counter | number of validator set updates returned by the application since process start | |
| state_block_apply_stage_duration | Histogram | stage | Time spent in each stage of applying a committed block, in ms: the ABCI calls, the state and ABCI response writes, the mempool update and the event publish |
Useful queries
Percentage of missing + byzantine validators:
((consensus_byzantine_validators_power + consensus_missing_validators_power) / consensus_validators_power) * 100
Rate at which the application is responding to each ABCI method call.
sum(rate(tendermint_abci_connection_method_timing_count[5m])) by (method)
The 95th percentile response time for the application to the deliver_tx ABCI method call.
histogram_quantile(0.95, sum by(le) (rate(tendermint_abci_connection_method_timing_bucket{method="deliver_tx"}[5m])))
Remote signer
Tenderdash provides a remote signer option for validators. A remote signer enables the operator to store the validator key on a different machine minimizing the attack surface if a server were to be compromised.
The remote signer protocol implements a client and server architecture. When Tenderdash requires the public key or signature for a proposal or vote it requests it from the remote signer.
The remote signer protocol does not cover proposer heartbeats or round skip votes. On a chain with consensus version 2, a validator using a remote signer sends neither, so validators skip its rounds that wait for transactions once its heartbeats are missed. See Proposer liveness and round skipping.
To run a secure validator and remote signer system it is recommended to use a VPC (virtual private cloud) or a private connection.
There are two different configurations that can be used: Raw or gRPC.
Raw
While both options use tcp or unix sockets the raw option uses tcp or unix sockets without http. The raw protocol sets up Tenderdash as the server and the remote signer as the client. This aids in not exposing the remote signer to public network.
Warning: Raw will be deprecated in a future major release, we recommend implementing your key management server against the gRPC configuration.
gRPC
gRPC is an RPC framework built with HTTP/2, uses Protocol Buffers to define services and has been standardized within the cloud infrastructure community. gRPC provides a language agnostic way to implement services. This aids developers in the writing key management servers in various different languages.
GRPC utilizes TLS, another widely standardized protocol, to secure connections. There are two forms of TLS to secure a connection, one-way and two-way. One way is when the client identifies the server but the server allows anyone to connect to it. Two-way is when the client identifies the server and the server identifies the client, prohibiting connections from unknown parties.
When using gRPC Tenderdash is setup as the client. Tenderdash will make calls to the remote signer. We recommend not exposing the remote signer to the public network with the use of virtual private cloud.
Securing your remote signers connection is highly recommended, but we provide the option to run it with a insecure connection.
Generating Certificates
To run a secure connection with gRPC we need to generate certificates and keys. We will walkthrough how to self sign certificates for two-way TLS.
There are two ways to generate certificates, openssl and certstarp. Both of these options can be used but we will be covering certstrap because it provides a simpler process then openssl.
- Install
Certstrap:
go install github.com/square/certstrap@v1.2.0
- Create certificate authority for self signing.
# generate self signing ceritificate authority
certstrap init --common-name "<name_CA>" --expires "20 years"
- Request a certificate for the server.
- For generalization purposes we set the ip to
127.0.0.1, but for your node please use the servers IP.
- For generalization purposes we set the ip to
- Sign the servers certificate with your certificate authority
# generate server cerificate
certstrap request-cert -cn server -ip 127.0.0.1
# self-sign server cerificate with rootCA
certstrap sign server --CA "<name_CA>" 127.0.0.1
- Request a certificate for the client.
- For generalization purposes we set the ip to
127.0.0.1, but for your node please use the clients IP.
- For generalization purposes we set the ip to
- Sign the clients certificate with your certificate authority
# generate client cerificate
certstrap request-cert -cn client -ip 127.0.0.1
# self-sign client cerificate with rootCA
certstrap sign client --CA "<name_CA>" 127.0.0.1
Running in production
If you are building Tenderdash from source for use in production, make sure to check out an appropriate Git tag instead of a branch.
Database
By default, Tenderdash uses the syndtr/goleveldb package for its in-process
key-value database. If you want maximal performance, it may be best to install
the real C-implementation of LevelDB and compile Tenderdash to use that using
make build TENDERMINT_BUILD_OPTIONS=cleveldb. See the install
instructions for details.
Tenderdash keeps multiple distinct databases in the $TMROOT/data:
blockstore.db: Keeps the entire blockchain - stores blocks, block commits, and block meta data, each indexed by height. Used to sync new peers.evidence.db: Stores all verified evidence of misbehaviour.state.db: Stores the current blockchain state (ie. height, validators, consensus params). Only grows if consensus params or validators change. Also used to temporarily store intermediate results during block processing.tx_index.db: Indexes txs (and their results) by tx hash and by DeliverTx result events.
By default, Tenderdash will only index txs by their hash and height, not by their DeliverTx result events. See indexing transactions for details.
Applications can expose block pruning strategies to the node operator. Please read the documentation of your application to find out more details.
Applications can use state sync to help nodes bootstrap quickly.
Logging
Default logging level (log-level = "info") should suffice for
normal operation mode. Read this
post
for details on how to configure log-level config variable. Some of the
modules can be found here. If
you're trying to debug Tenderdash or asked to provide logs with debug
logging level, you can do so by running Tenderdash with
--log-level="debug".
Consensus WAL
Tenderdash uses a write ahead log (WAL) for consensus. The consensus.wal is used to ensure we can recover from a crash at any point
in the consensus state machine. It writes all consensus messages (timeouts, proposals, block part, or vote)
to a single file, flushing to disk before processing messages from its own
validator. Since Tenderdash validators are expected to never sign a conflicting vote, the
WAL ensures we can always recover deterministically to the latest state of the consensus without
using the network or re-signing any consensus messages. The consensus WAL max size of 1GB and is automatically rotated.
If your consensus.wal is corrupted, see below.
DOS Exposure and Mitigation
Validators are supposed to setup Sentry Node Architecture to prevent Denial-of-service attacks.
P2P
The core of the Tenderdash peer-to-peer system is MConnection. Each
connection has MaxPacketMsgPayloadSize, which is the maximum packet
size and bounded send & receive queues. One can impose restrictions on
send & receive rate per connection (SendRate, RecvRate).
The number of open P2P connections can become quite large, and hit the operating system's open
file limit (since TCP connections are considered files on UNIX-based systems). Nodes should be
given a sizable open file limit, e.g. 8192, via ulimit -n 8192 or other deployment-specific
mechanisms.
RPC
Endpoints returning multiple entries are limited by default to return 30 elements (100 max). See the RPC Documentation for more information.
Rate-limiting and authentication are another key aspects to help protect against DOS attacks. Validators are supposed to use external tools like NGINX or traefik to achieve the same things.
Debugging Tenderdash
If you ever have to debug Tenderdash, the first thing you should probably do is check out the logs. See Logging, where we explain what certain log statements mean.
If, after skimming through the logs, things are not clear still, the next thing
to try is querying the /status RPC endpoint. It provides the necessary info:
whenever the node is syncing or not, what height it is on, etc.
curl http(s)://{ip}:{rpcPort}/status
/dump_consensus_state will give you a detailed overview of the consensus
state (proposer, latest validators, peers states). From it, you should be able
to figure out why, for example, the network had halted.
curl http(s)://{ip}:{rpcPort}/dump_consensus_state
There is a reduced version of this endpoint - /consensus_state, which returns
just the votes seen at the current height.
If, after consulting with the logs and above endpoints, you still have no idea
what's happening, consider using tenderdash debug kill sub-command. This
command will scrap all the available info and kill the process. See
Debugging for the exact format.
You can inspect the resulting archive yourself or create an issue on GitHub. Before opening an issue however, be sure to check if there's no existing issue already.
Monitoring Tenderdash
Each Tenderdash instance has a standard /health RPC endpoint, which responds
with 200 (OK) if everything is fine and 500 (or no response) - if something is
wrong.
Other useful endpoints include mentioned earlier /status, /net_info and
/validators.
Tenderdash also can report and serve Prometheus metrics. See Metrics.
tenderdash debug dump sub-command can be used to periodically dump useful
information into an archive. See Debugging for more
information.
What happens when my app dies
You are supposed to run Tenderdash under a process supervisor (like systemd or runit). It will ensure Tenderdash is always running (despite possible errors).
Getting back to the original question, if your application dies, Tenderdash will panic. After a process supervisor restarts your application, Tenderdash should be able to reconnect successfully. The order of restart does not matter for it.
Signal handling
We catch SIGINT and SIGTERM and try to clean up nicely. For other signals we use the default behavior in Go: Default behavior of signals in Go programs.
Corruption
NOTE: Make sure you have a backup of the Tenderdash data directory.
Possible causes
Remember that most corruption is caused by hardware issues:
- RAID controllers with faulty / worn out battery backup, and an unexpected power loss
- Hard disk drives with write-back cache enabled, and an unexpected power loss
- Cheap SSDs with insufficient power-loss protection, and an unexpected power-loss
- Defective RAM
- Defective or overheating CPU(s)
Other causes can be:
- Database systems configured with fsync=off and an OS crash or power loss
- Filesystems configured to use write barriers plus a storage layer that ignores write barriers. LVM is a particular culprit.
- Tenderdash bugs
- Operating system bugs
- Admin error (e.g., directly modifying Tenderdash data-directory contents)
(Source: https://wiki.postgresql.org/wiki/Corruption)
WAL Corruption
If consensus WAL is corrupted at the latest height and you are trying to start Tenderdash, replay will fail with panic.
Recovering from data corruption can be hard and time-consuming. Here are two approaches you can take:
- Delete the WAL file and restart Tenderdash. It will attempt to sync with other peers.
- Try to repair the WAL file manually:
-
Create a backup of the corrupted WAL file:
cp "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup -
Use
./scripts/wal2jsonto create a human-readable version:./scripts/wal2json/wal2json "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal -
Search for a "CORRUPTED MESSAGE" line.
-
By looking at the previous message and the message after the corrupted one and looking at the logs, try to rebuild the message. If the consequent messages are marked as corrupted too (this may happen if length header got corrupted or some writes did not make it to the WAL ~ truncation), then remove all the lines starting from the corrupted one and restart Tenderdash.
$EDITOR /tmp/corrupted_wal -
After editing, convert this file back into binary form by running:
./scripts/json2wal/json2wal /tmp/corrupted_wal $TMHOME/data/cs.wal/wal
Hardware
Processor and Memory
While actual specs vary depending on the load and validators count, minimal requirements are:
- 1GB RAM
- 25GB of disk space
- 1.4 GHz CPU
SSD disks are preferable for applications with high transaction throughput.
Recommended:
- 2GB RAM
- 100GB SSD
- x64 2.0 GHz 2v CPU
Disk usage depends on retained block history and application state. State sync can bootstrap from an application snapshot; configure retention according to your recovery and archival requirements.
Validator signing on 32 bit architectures (or ARM)
Both our ed25519 and secp256k1 implementations require constant time
uint64 multiplication. Non-constant time crypto can (and has) leaked
private keys on both ed25519 and secp256k1. This doesn't exist in hardware
on 32 bit x86 platforms (source), and it
depends on the compiler to enforce that it is constant time. It's unclear at
this point whenever the Golang compiler does this correctly for all
implementations.
We do not support nor recommend running a validator on 32 bit architectures OR the "VIA Nano 2000 Series", and the architectures in the ARM section rated "S-".
Operating Systems
Tenderdash supports Linux and macOS only. Windows is not a supported platform and there are no plans to add support for it.
While we do not favor any particular operating system, more secure and stable Linux server distributions (like CentOS) should be preferred over desktop operating systems (like macOS).
Miscellaneous
Size a production node for your Dash Platform network, application workload and retention policy. Measure disk growth, memory use and consensus latency under representative load before deployment.
Configuration parameters
p2p.flush-throttle-timeoutp2p.max-packet-msg-payload-sizep2p.send-ratep2p.recv-rate
If you are going to use Tenderdash in a private domain and you have a private high-speed network among your peers, it makes sense to lower flush throttle timeout and increase other params.
[p2p]
send-rate=20000000 # 2MB/s
recv-rate=20000000 # 2MB/s
flush-throttle-timeout=10
max-packet-msg-payload-size=10240 # 10KB
mempool.broadcast
Setting this to false will stop the mempool from relaying transactions to other peers until they are included in a block. It means only the peer you send the tx to will see it until it is included in a block.
consensus.peer-gossip-sleep-duration
You can try to reduce the time your node sleeps before checking if theres something to send its peers.
p2p.addr-book-strict
By default, Tenderdash checks whenever a peer's address is routable before saving it to the address book. The address is considered as routable if the IP is valid and within allowed ranges.
This may not be the case for private or local networks, where your IP range is usually
strictly limited and private. If that case, you need to set addr-book-strict
to false (turn it off).
rpc.max-open-connections
By default, the number of simultaneous connections is limited because most OS give you limited number of file descriptors.
If you want to accept greater number of connections, you will need to increase these limits.
Sysctls to tune the system to be able to open more connections
The process file limits must also be increased, e.g. via ulimit -n 8192.
...for N connections, such as 50k:
kern.maxfiles=10000+2*N # BSD
kern.maxfilesperproc=100+2*N # BSD
kern.ipc.maxsockets=10000+2*N # BSD
fs.file-max=10000+2*N # Linux
net.ipv4.tcp_max_orphans=N # Linux
# For load-generating clients.
net.ipv4.ip_local_port_range="10000 65535" # Linux.
net.inet.ip.portrange.first=10000 # BSD/Mac.
net.inet.ip.portrange.last=65535 # (Enough for N < 55535)
net.ipv4.tcp_tw_reuse=1 # Linux
net.inet.tcp.maxtcptw=2*N # BSD
# If using netfilter on Linux:
net.netfilter.nf_conntrack_max=N
echo $((N/8)) > /sys/module/nf_conntrack/parameters/hashsize
The similar option exists for limiting the number of gRPC connections -
rpc.grpc-max-open-connections.
Configure State-Sync
State sync rapidly bootstraps a new node by discovering, fetching, and restoring a state machine snapshot from peers instead of fetching and replaying historical blocks. The node will have a truncated block history, starting from the height of the snapshot.
NOTE: Before trying to use state sync, see if the application you are operating a node for supports it.
State sync is only attempted on first start: if the node already has any local
state (LastBlockHeight > 0), it is skipped and the node falls back to block
sync.
Unlike upstream Tendermint, Tenderdash does not require trust anchors
(trust-height, trust-hash, trust-period). Light blocks are verified by
checking the quorum threshold signature against the active validator quorum via
Dash Core (quorum verify), so there is no need to obtain a trusted block hash
out of band. If you are migrating a configuration that still contains
trust-height, trust-hash, or trust-period under [statesync], remove
them — they are no longer valid options.
Because light block verification goes through Dash Core, a full node must
be configured with a Dash Core RPC connection or it will refuse to start: set
core-rpc-host (and core-rpc-username/core-rpc-password as needed) in the
[priv-validator] section of config.toml. Validator nodes already have this
connection configured.
Dash Core is also what authenticates quorum membership and public key shares
before validator state received from peers is installed. If genesis configures
VotingPowerThreshold, the threshold received during state sync must match the
locally stored genesis value; it cannot be learned safely from a remote
state-sync provider.
Under the [statesync] section in config.toml you will find the settings
that need to be configured in order for your node to use state sync.
Let's break down the settings:
enable: Inform the node that you will be using state sync to bootstrap. This only controls consuming snapshots at first start; full and validator nodes serve snapshots and light blocks to peers regardless of this setting (seed nodes only run peer exchange and serve neither).use-p2p: State sync uses light client verification to verify state. This can be done either through the P2P layer or the RPC layer. Set this totrueto use the P2P layer. Iffalse(default), the RPC layer will be used.rpc-servers: Comma-separated list of RPC servers used for light client verification whenuse-p2p = false. In that mode at least two servers are required (more is always helpful). They should be compatible withnet.Dial, for example:host.example.com:2125. Ignored whenuse-p2p = true.discovery-time: Time to spend discovering snapshots before initiating a restore (default:15s). Must be0sor at least5s. With0sthe node gives up as soon as no suitable snapshot is available and falls back to block sync.retries: Number of completed snapshot discovery sweeps before giving up. When retries are exhausted, the node falls back to regular block sync. Set to0to retry indefinitely — the node keeps requesting snapshots forever and never falls back to block sync (default:3). Each sweep asks every connected peer (at most 1,024) in batches of 16, one batch perdiscovery-time, so the pessimistic case takes longer thandiscovery-time * retries; see Snapshot discovery resource limits.temp-dir: Temporary directory for snapshot chunks; defaults to the operating system temporary directory (e.g./tmp). The synchronizer creates a new, randomly named directory within it and removes it when the sync is complete.chunk-request-timeout: The timeout before re-requesting a chunk, possibly from a different peer (default:15s). Must be at least5s.fetchers: The number of concurrent chunk and block fetchers to run (default:4).
Example configuration for a full node using RPC-based light client
verification (the [priv-validator] Dash Core connection is required on full
nodes regardless of state sync):
[priv-validator]
core-rpc-host = "127.0.0.1:9998"
core-rpc-username = "dashrpc"
core-rpc-password = "changeme"
[statesync]
enable = true
use-p2p = false
rpc-servers = "seed-1.example.com:26657,seed-2.example.com:26657"
Or, using the P2P layer for verification (no RPC servers needed):
[priv-validator]
core-rpc-host = "127.0.0.1:9998"
core-rpc-username = "dashrpc"
core-rpc-password = "changeme"
[statesync]
enable = true
use-p2p = true
Snapshot discovery resource limits
Snapshot advertisements retain the existing 4,000,000-byte network message limit; there is no smaller limit on application metadata. The node retains at most 64 MiB of unique snapshot hash and metadata payload, including the snapshot being restored, and charges at most 40,000,000 bytes of advertised payload to each peer. Shared snapshots count once globally and once for each supplying peer. Removing an association releases that peer's charge immediately, while an active snapshot remains globally charged until restoration cleanup finishes. If the same snapshot is admitted again while a removed active copy is still in use, both owned copies consume the global budget until the old copy is released. The pool owns copies of the payload and releases its charge when it releases the data. These limits bound retained discovery payload, not process RSS, transport receive queues, decoded messages in flight, chunk data, or application state.
The pool also bounds candidates to 1,024 and peer associations to 10,240. Each rejection history holds at most 1,024 entries; an older rejection can be considered again after it leaves that history. Empty indexes are removed. Under payload pressure, advertisements from less represented peers can replace unselected candidates from peers retaining more data. The active snapshot is never evicted or uncharged before restoration releases it. Admission is an opportunity to try a candidate, not a guarantee against malicious peers controlling many identities.
Discovery sends directed requests to batches of at most 16 peers. Each requested peer may return ten advertisements, counting duplicates and rejected replies. Responses without remaining allowance are ignored without penalizing the peer. Late responses remain eligible while that batch's allowance is open; subsequent batches replace the allowance. Newly connected peers can use unallocated request slots, and otherwise wait for a later discovery sweep.
The retries setting counts completed discovery sweeps. A sweep freezes at most
1,024 connected peers, visits them in batches, and cannot be prolonged by later
peer arrivals. Successive sweeps rotate through larger connected-peer lists.
Already retained candidates are tried before discovery is declared exhausted.
Restoration starts from the best snapshot retained so far, which may come from
the first batch only; nodes with more than 16 peers can therefore start from a
slightly older snapshot than the newest one available on the network.
With no usable responses, a sweep takes up to
ceil(min(connected_peers, 1024) / 16) * discovery-time, with at least one discovery
interval and a minimum interval of five seconds. Snapshot restoration adds its
own processing time. Context cancellation interrupts discovery; retries = 0
continues discovery indefinitely as before.
Validators
Validators are responsible for committing new blocks in the blockchain. These validators participate in the consensus protocol by broadcasting votes which contain cryptographic signatures signed by each validator's private key.
The active validator set is a Dash quorum. Each validator has voting power
100; membership and quorum keys determine participation. See the
validator-set terminology
and proposer selection.
Becoming a Validator
On Dash Platform, becoming a validator requires operating an evonode with 4,000 DASH collateral and registering it with a ProTx transaction. Follow the Dash evonode setup and registration guide for the current operator requirements and setup procedure.
The initial validator set comes from genesis and ABCI InitChain. On a running
chain, the application supplies validator-set updates in PrepareProposal and
ProcessProposal; changes agreed in block H take effect at H+1. Joining the P2P
network or editing a local genesis file does not add a validator to a live chain.
See validator-set and parameter changes and the startup handshake.
Setting up a Validator
When setting up a validator there are countless ways to configure your setup. This guide is aimed at showing one of them, the sentry node design. This design is mainly for DDOS prevention.
Network Layout

The diagram is based on AWS, other cloud providers will have similar solutions to design a solution. Running nodes is not limited to cloud providers, you can run nodes on bare metal systems as well. The architecture will be the same no matter which setup you decide to go with.
The proposed network diagram is similar to the classical backend/frontend separation of services in a corporate environment. The “backend” in this case is the private network of the validator in the data center. The data center network might involve multiple subnets, firewalls and redundancy devices, which is not detailed on this diagram. The important point is that the data center allows direct connectivity to the chosen cloud environment. Amazon AWS has “Direct Connect”, while Google Cloud has “Partner Interconnect”. This is a dedicated connection to the cloud provider (usually directly to your virtual private cloud instance in one of the regions).
All sentry nodes (the “frontend”) connect to the validator using this private connection. The validator does not have a public IP address to provide its services.
Amazon has multiple availability zones within a region. One can install sentry nodes in other regions too. In this case the second, third and further regions need to have a private connection to the validator node. This can be achieved by VPC Peering (“VPC Network Peering” in Google Cloud). In this case, the second, third and further region sentry nodes will be directed to the first region and through the direct connect to the data center, arriving to the validator.
A more persistent solution (not detailed on the diagram) is to have multiple direct connections to different regions from the data center. This way VPC Peering is not mandatory, although still beneficial for the sentry nodes. This overcomes the risk of depending on one region. It is more costly.
Local Configuration

The validator will only talk to the sentry that are provided, the sentry nodes will communicate to the validator via a secret connection and the rest of the network through a normal connection. The sentry nodes do have the option of communicating with each other as well.
When initializing nodes there are several parameters in the config.toml that may need to be altered.
mode:(full | validator | seed) Mode of node (default: 'full'). If you want to run the node as validator, change it to 'validator'.pex:boolean. This turns the peer exchange reactor on or off for a node. Whenpex=false, only thepersistent-peerslist is available for connection.persistent-peers:a comma separated list ofnodeID@ip:portvalues that define a list of peers that are expected to be online at all times. This is necessary at first startup because by settingpex=falsethe node will not be able to join the network.unconditional-peer-ids:comma separated list of nodeID's. These nodes will be connected to no matter the limits of inbound and outbound peers. This is useful for when sentry nodes have full address books.private-peer-ids:comma separated list of nodeID's. These nodes will not be gossiped to the network. This is an important field as you do not want your validator IP gossiped to the network.addr-book-strict:boolean. By default nodes with a routable address will be considered for connection. If this setting is turned off (false), non-routable IP addresses, like addresses in a private network can be added to the address book.
Validator Node Configuration
| Config Option | Setting |
|---|---|
| mode | validator |
| pex | false |
| persistent-peers | list of sentry nodes |
| private-peer-ids | none |
| unconditional-peer-ids | optionally sentry node IDs |
| addr-book-strict | false |
To run the node as validator ensure mode=validator. The validator node should have pex=false so it does not gossip to the entire network. The persistent peers will be your sentry nodes. Private peers can be left empty as the validator is not trying to hide who it is communicating with. Setting unconditional peers is optional for a validator because they will not have a full address books.
Sentry Node Configuration
| Config Option | Setting |
|---|---|
| mode | full |
| pex | true |
| persistent-peers | validator node, optionally other sentry nodes |
| private-peer-ids | validator node ID |
| unconditional-peer-ids | validator node ID, optionally sentry node IDs |
| addr-book-strict | false |
The sentry nodes should be able to talk to the entire network hence why pex=true. The persistent peers of a sentry node will be the validator, and optionally other sentry nodes. The sentry nodes should make sure that they do not gossip the validator's ip, to do this you must put the validators nodeID as a private peer. The unconditional peer IDs will be the validator ID and optionally other sentry nodes.
Note: Do not forget to secure your node's firewalls when setting them up.
More Information can be found at these links:
- https://web.archive.org/web/2021/https://kb.certus.one/
- https://forum.cosmos.network/t/sentry-node-architecture-overview/454
Validator keys
Tenderdash validators use BLS consensus keys and Dash quorum information. Protect both the private key and the signing state that prevents conflicting votes. Never run two active signers with the same consensus identity.
A remote signer must support Tenderdash's BLS keys and quorum-aware signing requests. Check compatibility before choosing a key management server or HSM; support for Ed25519 in another consensus system does not establish compatibility. See remote signers and the signing specification.
Committing a Block
Tenderdash commits contain a recovered BLS threshold block signature and threshold vote-extension signatures, rather than individual validator signatures. The configurable voting threshold and the quorum's signature recovery threshold are distinct; the label "+2/3" in consensus code does not always mean a strict two-thirds count.
Use the canonical threshold definitions, threshold recovery procedure and commit handling rules.
Using ABCI-CLI
abci-cli sends requests directly to an ABCI application for development and
debugging. It does not run consensus or construct a verified quorum commit.
Use it against an isolated test application, not an application serving a live
network.
Install
Build the tool from the same Tenderdash checkout as the application:
make install_abci
Inspect the options for your installed version:
abci-cli --help
abci-cli kvstore --help
Start an example application
Start the in-memory key-value application in one terminal:
abci-cli kvstore --address tcp://127.0.0.1:26658
In another terminal, send basic requests:
abci-cli --address tcp://127.0.0.1:26658 echo hello
abci-cli --address tcp://127.0.0.1:26658 info
abci-cli --address tcp://127.0.0.1:26658 check_tx 0x6b65793d76616c7565
The CLI's default address is tcp://0.0.0.0:26658; these examples explicitly
select loopback. The default transport is socket.
CheckTx validates a transaction; it does not persist application state.
For gRPC, start a server with --abci grpc and use a bare HOST:PORT client
address. The server accepts a tcp:// address, but the gRPC client does not
resolve that form correctly and waits for a connection:
abci-cli kvstore --abci grpc --address tcp://127.0.0.1:26659
In another terminal:
abci-cli --abci grpc --address 127.0.0.1:26659 echo hello
Transaction and query arguments accept 0x-prefixed hexadecimal bytes or a
string with literal double quotes. For example, these requests encode the
same bytes (key=value):
abci-cli --address tcp://127.0.0.1:26658 check_tx 0x6b65793d76616c7565
abci-cli --address tcp://127.0.0.1:26658 check_tx '"key=value"'
Proposal and finalization requests
The implemented request commands include echo, info, query, check_tx,
prepare_proposal, process_proposal and finalize_block. Their block-related
argument forms are:
prepare_proposal HEIGHT TX [TX ...]
process_proposal HEIGHT [TX ...]
finalize_block HEIGHT APP_HASH_HEX [TX ...]
The height is decimal. Each TX uses the byte encoding described above.
APP_HASH_HEX is the application hash as hexadecimal without a 0x prefix;
it must match the state prepared and processed for that block. These commands
send simplified test requests and do not fill every field required by an
arbitrary application's consensus lifecycle. In particular, they cannot set the
round, timestamp, proposer, quorum or commit metadata. prepare_proposal fixes
max_tx_bytes at 65536, and process_proposal fixes the application protocol
version to the example application's version. Consult the
command implementation
when constructing application-specific tests. The standalone prepare_proposal
command requires a transaction; an empty proposal can be sent through batch.
The CLI does not expose request commands for InitChain, ExtendVote,
VerifyVoteExtension or the four snapshot methods. There is no flush command;
Flush is available through the underlying client interface.
Responses are abbreviated: info prints only data, check_tx prints only
code and data, and proposal commands omit transaction results, parameter and
validator updates, and other response fields. --verbose does not restore those
fields. CLI output therefore cannot validate the full application protocol.
There is no ABCI Commit method or implemented commit command. The application
persists state during FinalizeBlock. For a complete working node/application
example, use the external Go application guide.
The canonical ABCI call sequence
defines the lifecycle a real node drives.
Some installed help and completion text still mentions commit; that text is
stale.
Reuse a connection
console accepts interactive request commands; batch reads them from standard
input. Both reuse one connection. For example:
abci-cli --address tcp://127.0.0.1:26658 --verbose batch <<'EOF_COMMANDS'
echo hello
info
check_tx 0x6b65793d76616c7565
EOF_COMMANDS
The console and batch parser splits input on spaces without shell quoting.
Encode transaction or query bytes containing spaces as hexadecimal in these
modes. Query flags such as --height, --path and --prove apply to standalone
query commands; flags inside console or batch input are not applied.
Use an ABCI implementation compatible with your Tenderdash release. See the current application interface and protobuf definitions.
Application Architecture Guide
A Tenderdash deployment separates the consensus node from the application that defines transaction rules and owns application state. A client submits transactions or queries through an RPC or application API.
flowchart LR
Client[Client application] -->|Transactions and queries| RPC[Tenderdash RPC]
RPC --> Node[Tenderdash node]
Node <-->|P2P consensus| Peers[Peer nodes]
Node <-->|ABCI| App[Application]
App --> State[Persistent application state]
Tenderdash orders transactions and agrees on blocks. The ABCI application
validates transactions with CheckTx, prepares and validates proposals with
PrepareProposal and ProcessProposal, and finalizes decided blocks with
FinalizeBlock. Queries use Query; the application defines the query format
and any proofs it returns.
The application must produce deterministic results for the same consensus input. Keep replicated state changes inside the ABCI lifecycle. External services may provide read APIs, but must not independently mutate that state.
An application written in Go can run in the same process as Tenderdash using a local ABCI client. Other deployments can run the application as a separate process through an ABCI socket or gRPC connection. Protect these connections: ABCI is a trusted interface, not a public client endpoint. Restrict network access and provide authentication and encryption when crossing a network boundary.
Client applications can broadcast transactions using the /broadcast_tx_* RPC
methods and query through /abci_query. A successful broadcast does not by itself
prove that a transaction was committed successfully. Inspect committed results
and verify application proofs according to the application's security model.
See the RPC guide, ABCI specification, Go tutorials and production operations guide.
Getting Started with Application Development
Tenderdash handles P2P communication, transaction gossip and consensus. Your ABCI application defines transaction validation, execution and application state. Start with the application architecture guide and the ABCI overview.
Run an example
Install Tenderdash and initialize a local validator in a new directory:
tenderdash init validator --home ./example-node
tenderdash start --home ./example-node --proxy-app=kvstore
The built-in kvstore is an in-memory demonstration application. Do not use it
as persistent production storage.
Submit and query a key-value transaction:
curl -s 'localhost:26657/broadcast_tx_commit?tx="example=rocks"'
curl -s 'localhost:26657/abci_query?data="example"'
Build your application
- External Go application: run an ABCI server separately from the node.
- Built-in Go application: use a local ABCI client in the same process.
- ABCI CLI: send requests directly to an application during development and debugging.
Applications in other languages must implement the protobuf messages and ABCI lifecycle for the Tenderdash release they target. Check compatibility before using a third-party ABCI library.
Use the application interface, protobuf definitions and ABCI specification as the contract. Test transaction rejection, proposal processing, block finalization, deterministic state hashes and restart recovery before connecting to a live network.
Indexing Transactions
Tenderdash can index committed transaction results and block events for RPC
searches and subscriptions. Transaction events come from ExecTxResult.Events
in the proposal execution results. Block events come from
ResponseProcessProposal.Events. They are published after block finalization;
ResponseFinalizeBlock does not contain an events field.
Each event has a type and string-valued attributes. The search key combines the
event type and attribute key with a dot. An event of type transfer with an
attribute sender=Bob can therefore be queried as transfer.sender='Bob'.
Set an attribute's Index flag to true to include it in event indexes.
For the execution lifecycle, see commit and apply. The event publisher shows how committed proposal results become transaction and block events.
Configuration
Operators can configure indexing via the [tx-index] section. The indexer
field takes a series of supported indexers. If null is included, indexing will
be turned off. All configured indexer names must still be supported.
[tx-index]
# The backend database list to back the indexer.
# If list contains null, meaning no indexer service will be used.
#
# The application will set which txs to index. In some cases a node operator will be able
# to decide which txs to index based on configuration set in the application.
#
# Options:
# 1) "null" (default) - no indexer services.
# 2) "kv" - an indexer backed by key-value storage (see DBBackend).
# - When "kv" is chosen "tx.height" and "tx.hash" will always be indexed.
# indexer = []
Supported Indexers
KV
The kv indexer type is an embedded key-value store supported by the main
underlying Tenderdash database. Using the kv indexer type allows you to query
for block and transaction events directly against Tenderdash's RPC. However, the
query syntax is limited to the supported event-query operators.
Removed PostgreSQL indexer
The psql event indexer is no longer supported. Selecting it, including alongside
kv or null, returns a configuration error. Choose indexer = ["kv"] for local
indexing and RPC searches, or ["null"] to disable indexing. The obsolete
psql-conn key is ignored, so existing Dashmate configurations with an empty
value remain compatible.
Existing PostgreSQL data is not deleted or migrated, and SQL consumers no longer
receive new events. To populate KV indexes, explicitly run tenderdash reindex-event against the retained local blocks and results after selecting kv.
Pruned history cannot be reconstructed by this command.
Default Indexes
The Tenderdash tx and block event indexer indexes a few select reserved events by default.
Transactions
The following indexes are indexed by default:
tx.heighttx.hash
Blocks
The following indexes are indexed by default:
block.height
Adding Events
Return transaction events in the corresponding ExecTxResult in
ResponsePrepareProposal.TxResults / ResponseProcessProposal.TxResults.
For example, an individual transaction result can contain:
result := &abci.ExecTxResult{
Code: 0,
Events: []abci.Event{
{
Type: "transfer",
Attributes: []abci.EventAttribute{
{Key: "sender", Value: "Bob", Index: true},
{Key: "recipient", Value: "Alice", Index: true},
{Key: "note", Value: "payment", Index: false},
},
},
},
}
Here abci refers to github.com/dashpay/tenderdash/abci/types; append result
to the response's transaction results in the same order as the executed
transactions. The kv indexer indexes transfer.sender and
transfer.recipient; transfer.note is not indexed. An empty event type or
attribute key is not indexed. Avoid reserved keys such as tx.hash and
tx.height.
Querying Transactions Events
You can query for a paginated set of transaction by their events by calling the
/tx_search RPC endpoint:
curl "localhost:26657/tx_search?query=\"transfer.sender='Bob'\"&prove=true"
Check out API docs for more information on query syntax and other options.
Subscribing to Transactions
Clients can subscribe to transactions with the given tags via WebSocket by providing
a query to /subscribe RPC endpoint.
{
"jsonrpc": "2.0",
"method": "subscribe",
"id": "0",
"params": {
"query": "transfer.sender='Bob'"
}
}
Check out API docs for more information on query syntax and other options.
Querying Blocks Events
You can query for a paginated set of blocks by their events by calling the
/block_search RPC endpoint:
curl "localhost:26657/block_search?query=\"block.height > 10\""
Check out API docs for more information on query syntax and other options.
Application development
Creating a built-in application in Go
An application written in Go can run in the same process as Tenderdash. A local ABCI client calls the application directly, avoiding a network connection between the node and application. The application still follows the same consensus and persistence rules as an external application.
Use the current module and interface
Build against github.com/dashpay/tenderdash at the release used by your network.
Tenderdash includes a native BLS dependency; follow the
build instructions when preparing your toolchain.
Implement the ABCI application interface.
Use the key-value application
as a reference for the full lifecycle, including PrepareProposal,
ProcessProposal, FinalizeBlock, queries and snapshots.
Construct a local client and node
The relevant imports are:
import (
abciclient "github.com/dashpay/tenderdash/abci/client"
"github.com/dashpay/tenderdash/config"
"github.com/dashpay/tenderdash/node"
)
Given your application app, a configured logger logger, and the application
context ctx, the integration is:
client := abciclient.NewLocalClient(logger, app)
service, err := node.New(ctx, conf, logger, client, nil)
if err != nil {
return err
}
if err := service.Start(ctx); err != nil {
return err
}
defer func() {
service.Stop()
service.Wait()
}()
Stop the service before waiting for it. Keep the process running until cancellation or a shutdown signal, then cancel the context and shut down the node before closing application storage.
The conf value is a validated *config.Config loaded for the intended home
directory. The default home is $HOME/.tenderdash; set it explicitly for tests.
Initialize the node's configuration and genesis with tenderdash init validator
before starting it. Passing nil as the final node.New argument loads genesis
from the configured file.
The node constructor and local client are the source of truth for the API signatures in your release.
Concurrency and recovery
The application can receive different classes of ABCI requests while the node is running. Protect shared state, keep queries consistent with committed state, and separate mempool checks and tentative proposal execution from finalization. Persistence must record the height and application hash with the state they identify so that restarting the process can recover consistently.
Test the application independently through the ABCI interface, then exercise the embedded node's startup, shutdown and restart paths. The ABCI specification describes the required behavior.
Creating an external application in Go
An external ABCI application runs separately from the Tenderdash node. The node connects to its socket or gRPC server and sends requests using the ABCI application interface. Use the application interface and protobuf definitions from your Tenderdash release.
Start with the example application
Build Tenderdash and the example ABCI CLI from the same checkout:
make install_abci
Start the supplied key-value application in one terminal:
abci-cli kvstore --address tcp://127.0.0.1:26658
Initialize a local validator and connect it to that application in another terminal. Use a new home directory for this experiment:
tenderdash init validator --home ./example-node
tenderdash start --home ./example-node --proxy-app=tcp://127.0.0.1:26658
Keep the ABCI server bound to loopback for local development. A publicly exposed ABCI endpoint lets callers bypass the node's consensus lifecycle.
Implement your application
Study the key-value application source and its tests. It provides working examples of transaction checking, proposal handling, block finalization, queries and snapshots.
Your Go implementation uses these package paths:
import (
"github.com/dashpay/tenderdash/abci/server"
abci "github.com/dashpay/tenderdash/abci/types"
)
The abci.Application interface includes:
InfoandInitChainfor startup and chain initialization.CheckTxfor mempool admission; this must not commit application state.PrepareProposalandProcessProposalfor block construction and validation.ExtendVoteandVerifyVoteExtensionfor application vote extensions.FinalizeBlockfor applying a decided block and persisting state.Queryfor application queries, plus the snapshot methods for state sync.
abci.BaseApplication supplies default implementations for scaffolding. Those
defaults are not a complete persistent application. Implement the state, height,
application hash, validation and recovery behavior your application requires.
Create the server with
server.NewServer(logger, address, transport, app), then start the returned
service with a context. Stop and wait for the service during shutdown. See the
server package
and ABCI CLI implementation
for lifecycle handling.
Submit and query transactions
With the example key-value application running, submit a transaction:
curl -s 'localhost:26657/broadcast_tx_commit?tx="example=rocks"'
Query the committed value:
curl -s 'localhost:26657/abci_query?data="example"'
Inspect the transaction result as well as transport-level success. Your own application defines the transaction encoding and query format.
Validate application behavior
Test invalid transactions, proposal rejection, repeated requests, crash recovery and snapshot restore. All correct replicas must derive the same application hash from the same committed input. Keep tentative proposal execution isolated from committed state, because proposals can be rejected or superseded.
For the complete lifecycle, read the ABCI specification. For embedding in the same process, follow the built-in application guide.
Tutorials
Overview
This section dives into the internals of Tenderdash.
- Using Tenderdash
- Subscribing to events
- Block Structure
- RPC
- Block Sync
- State Sync
- Mempool
- Light Client
- Consensus
- Peer Exchange (PEX)
- Evidence
For full specifications refer to the protocol specification.
Block Structure
The Tenderdash consensus engine records all agreements by a
supermajority of nodes into a blockchain, which is replicated among all
nodes. This blockchain is accessible via various RPC endpoints, mainly
/block?height= to get the full block, as well as
/blockchain?minHeight=_&maxHeight=_ to get a list of headers. But what
exactly is stored in these blocks?
The specification contains a detailed description of each component - that's the best place to get started.
To dig deeper, check out the types package documentation.
See also:
Block Sync
Formerly known as Fast Sync
In a proof of work blockchain, syncing with the chain is the same process as staying up-to-date with the consensus: download blocks, and look for the one with the most total work. In proof-of-stake, the consensus process is more complex, as it involves rounds of communication between the nodes to determine what block should be committed next. Using this process to sync up with the blockchain from scratch can take a very long time. It's much faster to just download blocks and check the merkle tree of validators than to run the real-time consensus gossip protocol.
Using Block Sync
To support faster syncing, Tenderdash offers a blocksync mode, which
is enabled by default, and can be toggled in the config.toml or via
--blocksync.enable=false.
In this mode, the Tenderdash daemon will sync hundreds of times faster
than if it used the real-time consensus process. Once caught up, the
daemon will switch out of Block Sync and into the normal consensus mode.
After running for some time, the node is considered caught up if it
has at least one peer and it's height is at least as high as the max
reported peer height. See the IsCaughtUp
method.
Note: There are multiple versions of Block Sync. Please use v0 as the other versions are no longer supported.
If you would like to use a different version you can do so by changing the version in the config.toml:
#######################################################
### Block Sync Configuration Connections ###
#######################################################
[blocksync]
# If this node is many blocks behind the tip of the chain, BlockSync
# allows them to catchup quickly by downloading blocks in parallel
# and verifying their commits
enable = true
# Block Sync version to use:
# 1) "v0" (default) - the standard Block Sync implementation
# 2) "v2" - DEPRECATED, please use v0
version = "v0"
If we're lagging sufficiently, we should go back to block syncing, but this is an open issue.
The Block Sync event
When the tenderdash blockchain core launches, it might switch to the block-sync
mode to catch up the states to the current network best height. the core will emits
a fast-sync event to expose the current status and the sync height. Once it catched
the network best height, it will switches to the state sync mechanism and then emit
another event for exposing the fast-sync complete status and the state height.
The user can query the events by subscribing EventQueryBlockSyncStatus
Please check types for the details.
Implementation
To read more on the implamentation please see the reactor doc and the implementation doc
Implementation
Blocksync Reactor
- coordinates the pool for syncing
- coordinates the store for persistence
- coordinates the playing of blocks towards the app using a sm.BlockExecutor
- handles switching between fastsync and consensus
- it is a p2p.BaseReactor
- starts the pool.Start() and its poolRoutine()
- registers all the concrete types and interfaces for serialisation
poolRoutine
- listens to these channels:
- pool requests blocks from a specific peer by posting to requestsCh, block reactor then sends a &bcBlockRequestMessage for a specific height
- pool signals timeout of a specific peer by posting to timeoutsCh
- switchToConsensusTicker to periodically try and switch to consensus
- trySyncTicker to periodically check if we have fallen behind and then catch-up sync
- if there aren't any new blocks available on the pool it skips syncing
- tries to sync the app by taking downloaded blocks from the pool, gives them to the app and stores them on disk
- implements Receive which is called by the switch/peer
- calls AddBlock on the pool when it receives a new block from a peer
Block Pool
- responsible for downloading blocks from peers
- makeRequestersRoutine()
- removes timeout peers
- starts new requesters by calling makeNextRequester()
- requestRoutine():
- picks a peer and sends the request, then blocks until:
- pool is stopped by listening to pool.Quit
- requester is stopped by listening to Quit
- request is redone
- we receive a block
- gotBlockCh is strange
- picks a peer and sends the request, then blocks until:
Go Routines in Blocksync Reactor

Reactor
The Blocksync Reactor's high level responsibility is to enable peers who are far behind the current state of the consensus to quickly catch up by downloading many blocks in parallel, verifying their commits, and executing them against the ABCI application.
Tenderdash full nodes run the Blocksync Reactor as a service to provide blocks to new nodes. New nodes run the Blocksync Reactor in "fast_sync" mode, where they actively make requests for more blocks until they sync up. Once caught up, "fast_sync" mode is disabled and the node switches to using (and turns on) the Consensus Reactor.
Architecture and algorithm
The pseudocode below describes the historical requester-based implementation.
The current synchronizer in internal/blocksync uses a worker pool. Five
consecutive failed requests remove a peer; a successful response resets that
failure count. The client request timeout remains 60 seconds.
Scheduling also accounts for useful service while requests are outstanding:
- Active waiting begins after the client admits a request and ends at its first correlated response, timeout, or cancellation. Overlapping requests consume time once, not once per request. Worker queues before admission, completed result buffering, block application, and idle intervals do not consume time.
- A response earns one second plus its encoded bytes divided by 128 KiB/s only after conversion, basic validation, and the requested-height check. Commit verification and application retain their existing later checks. Saved credit is limited to ten seconds, so an old burst cannot hide sustained poor service.
- More than ten seconds of service debt reduces future assignments. Existing requests drain; after a thirty-second cooldown, one recovery request is allowed. Due recovery requests receive a turn even when other peers are healthy. Two consecutive recovery responses meeting the size-adjusted allowance restore ordinary pipelining. Slow or failed probes restart the cooldown.
These are soft scheduling allowances, not measured wire bandwidth or a guaranteed minimum connection speed. They let prompt tiny blocks earn progress through their count and larger blocks through their bytes. There is no partial-byte telemetry: a large unanswered response can provisionally reduce preference, and its validated bytes can correct that estimate on completion. Local transport and router queues between client admission and resolution remain included in the observation.
A scheduling penalty never disconnects a peer or makes its advertised block range unavailable. A sole source remains eligible for bounded recovery attempts, which prevents a false no-source handoff to consensus. Status refreshes preserve service history; a removed and re-added peer starts fresh. The twenty pending requests per peer, sixty-four outstanding heights, completed-backlog byte budget, validation, and hard failure policy are unchanged. The 60-second request deadline can still reject a large response on a slow connection even if its eventual service rate would meet the scheduling allowance.
Block sync hands over to consensus when one of these holds:
- the node reaches the highest height any peer reports;
- no peer reports the next height and no block has been applied for 60 seconds;
- the node's last applied block has been recent, at most five minutes old by the local clock, for two minutes without a break and without getting more than 30 seconds younger meanwhile;
- no block has been applied for 10 minutes, and either that wait began on a block less than a day old, or the node is within 10 blocks of every peer that reports the next height.
The heights peers report are not verified, so the last two rules rest on the time of the last applied block instead, which the validators commit to: a node far behind the network applies old blocks. Five minutes covers the three-minute empty-block interval Dash Platform runs with and a margin for extra rounds and clock skew. A last block that keeps getting younger means the node is still closing the gap, so the two minutes start over each time it has gained 30 seconds: a node applying blocks at least a quarter faster than the chain commits them goes on until its last block is within 30 seconds of the chain's. A block time further ahead of the local clock than the synchrony precision does not count as recent. When a node with a recent last block hands over, the consensus catch-up target is the highest height reported within 10 blocks of it. Only a node whose last block was more than a day old when it stopped receiving blocks keeps waiting for as long as a peer reports the next height and a height more than 10 blocks above the node, and resumes as soon as a peer serves again.
Blocks are applied in order, so the next block is the one that matters. When no block has been applied for 60 seconds and the next one has not arrived, it is requested from every peer that reports it, once each, starting a second after the peer first reports it. Sixty seconds is also how long a request may take, so a block that arrives within its request deadline is fetched once, however slow the link.
What operators can expect:
- A node that has caught up hands over within two to four minutes, whatever heights its peers report, provided a connected peer serves each new block when asked. Peers that delay blocks can stretch this to about 20 minutes, and a single block by about a minute.
- On a chain that has stopped producing blocks, a restarted node hands over after 10 minutes without a block. If the chain had been stopped for more than a day when the node started waiting, the node keeps waiting while a peer reports the next height and a height more than 10 blocks above the node.
- A node may hand over behind the network by up to five minutes of blocks: one or two blocks on a chain committing every three minutes, a few hundred on one committing every second. Consensus catch-up closes the rest. On a chain that keeps committing a block every second or two this can take long, and it happens only when the node received blocks less than a quarter faster than the chain produced them.
- A node less than a day behind the network whose peers all fail to deliver the next block for 10 minutes hands over where it is. Restart it once its peers serve blocks again if consensus catch-up does not close the gap.
- With only a few peers serving blocks, a node restarting far behind a chain that commits a block every second or two may need much longer to reach the tip. More peers that serve blocks help.
- A node whose clock runs minutes ahead of the validators', or behind them by more than the synchrony precision, does not see its last block as recent and leaves block sync only by the other rules. Keep the clock synchronised.
- After a handover below a height a peer reported, the node may hold back its own proposals for up to 10 minutes while consensus catches up. With a recent last block only heights reported within 10 blocks of the node count.
- A node that stays in block sync while its blocks are recent, or hands over far behind, is worth reporting with its logs: a handover below the highest height a peer reports is logged at warning or error level with its reason.
The historical Blocksync reactor is organised as a set of concurrent tasks:
- Receive routine of Blocksync Reactor
- Task for creating Requesters
- Set of Requesters tasks and - Controller task.

Data structures
These are the core data structures necessarily to provide the Blocksync Reactor logic.
Requester data structure is used to track assignment of request for block at position height to a peer with id equals to peerID.
type Requester {
mtx Mutex
block Block
height int64
peerID p2p.ID
redoChannel chan p2p.ID //redo may send multi-time; peerId is used to identify repeat
}
Pool is a core data structure that stores last executed block (height), assignment of requests to peers (requesters), current height for each peer and number of pending requests for each peer (peers), maximum peer height, etc.
type Pool {
mtx Mutex
requesters map[int64]*Requester
height int64
peers map[p2p.ID]*Peer
maxPeerHeight int64
numPending int32
store BlockStore
requestsChannel chan<- BlockRequest
errorsChannel chan<- peerError
}
Peer data structure stores for each peer current height and number of pending requests sent to the peer (numPending), etc.
type Peer struct {
id p2p.ID
height int64
numPending int32
timeout *time.Timer
didTimeout bool
}
BlockRequest is internal data structure used to denote current mapping of request for a block at some height to a peer (PeerID).
type BlockRequest {
Height int64
PeerID p2p.ID
}
Receive routine of Blocksync Reactor
It is executed upon message reception on the BlocksyncChannel inside p2p receive routine. There is a separate p2p receive routine (and therefore receive routine of the Blocksync Reactor) executed for each peer. Note that try to send will not block (returns immediately) if outgoing buffer is full.
handleMsg(pool, m):
upon receiving bcBlockRequestMessage m from peer p:
block = load block for height m.Height from pool.store
if block != nil then
try to send BlockResponseMessage(block) to p
else
try to send bcNoBlockResponseMessage(m.Height) to p
upon receiving bcBlockResponseMessage m from peer p:
pool.mtx.Lock()
requester = pool.requesters[m.Height]
if requester == nil then
error("peer sent us a block we didn't expect")
continue
if requester.block == nil and requester.peerID == p then
requester.block = m
pool.numPending -= 1 // atomic decrement
peer = pool.peers[p]
if peer != nil then
peer.numPending--
if peer.numPending == 0 then
peer.timeout.Stop()
// NOTE: we don't send Quit signal to the corresponding requester task!
else
trigger peer timeout to expire after peerTimeout
pool.mtx.Unlock()
upon receiving bcStatusRequestMessage m from peer p:
try to send bcStatusResponseMessage(pool.store.Height)
upon receiving bcStatusResponseMessage m from peer p:
pool.mtx.Lock()
peer = pool.peers[p]
if peer != nil then
peer.height = m.height
else
peer = create new Peer data structure with id = p and height = m.Height
pool.peers[p] = peer
if m.Height > pool.maxPeerHeight then
pool.maxPeerHeight = m.Height
pool.mtx.Unlock()
onTimeout(p):
send error message to pool error channel
peer = pool.peers[p]
peer.didTimeout = true
Requester tasks
Requester task is responsible for fetching a single block at position height.
fetchBlock(height, pool):
while true do {
peerID = nil
block = nil
peer = pickAvailablePeer(height)
peerID = peer.id
enqueue BlockRequest(height, peerID) to pool.requestsChannel
redo = false
while !redo do
select {
upon receiving Quit message do
return
upon receiving redo message with id on redoChannel do
if peerID == id {
mtx.Lock()
pool.numPending++
redo = true
mtx.UnLock()
}
}
}
pickAvailablePeer(height):
selectedPeer = nil
while selectedPeer = nil do
pool.mtx.Lock()
for each peer in pool.peers do
if !peer.didTimeout and peer.numPending < maxPendingRequestsPerPeer and peer.height >= height then
peer.numPending++
selectedPeer = peer
break
pool.mtx.Unlock()
if selectedPeer = nil then
sleep requestIntervalMS
return selectedPeer
sleep for requestIntervalMS
Task for creating Requesters
This task is responsible for continuously creating and starting Requester tasks.
createRequesters(pool):
while true do
if !pool.isRunning then break
if pool.numPending < maxPendingRequests or size(pool.requesters) < maxTotalRequesters then
pool.mtx.Lock()
nextHeight = pool.height + size(pool.requesters)
requester = create new requester for height nextHeight
pool.requesters[nextHeight] = requester
pool.numPending += 1 // atomic increment
start requester task
pool.mtx.Unlock()
else
sleep requestIntervalMS
pool.mtx.Lock()
for each peer in pool.peers do
if !peer.didTimeout && peer.numPending > 0 && peer.curRate < minRecvRate then
send error on pool error channel
peer.didTimeout = true
if peer.didTimeout then
for each requester in pool.requesters do
if requester.getPeerID() == peer then
enqueue msg on requestor's redoChannel
delete(pool.peers, peerID)
pool.mtx.Unlock()
Main blocksync reactor controller task
main(pool):
create trySyncTicker with interval trySyncIntervalMS
create statusUpdateTicker with interval statusUpdateIntervalSeconds
create switchToConsensusTicker with interval switchToConsensusIntervalSeconds
while true do
select {
upon receiving BlockRequest(Height, Peer) on pool.requestsChannel:
try to send bcBlockRequestMessage(Height) to Peer
upon receiving error(peer) on errorsChannel:
stop peer for error
upon receiving message on statusUpdateTickerChannel:
broadcast bcStatusRequestMessage(bcR.store.Height) // message sent in a separate routine
upon receiving message on switchToConsensusTickerChannel:
pool.mtx.Lock()
receivedBlockOrTimedOut = pool.height > 0 || (time.Now() - pool.startTime) > 5 Seconds
ourChainIsLongestAmongPeers = pool.maxPeerHeight == 0 || pool.height >= pool.maxPeerHeight
haveSomePeers = size of pool.peers > 0
pool.mtx.Unlock()
if haveSomePeers && receivedBlockOrTimedOut && ourChainIsLongestAmongPeers then
switch to consensus mode
upon receiving message on trySyncTickerChannel:
for i = 0; i < 10; i++ do
pool.mtx.Lock()
firstBlock = pool.requesters[pool.height].block
secondBlock = pool.requesters[pool.height].block
if firstBlock == nil or secondBlock == nil then continue
pool.mtx.Unlock()
verify firstBlock using LastCommit from secondBlock
if verification failed
pool.mtx.Lock()
peerID = pool.requesters[pool.height].peerID
redoRequestsForPeer(peerId)
delete(pool.peers, peerID)
stop peer peerID for error
pool.mtx.Unlock()
else
delete(pool.requesters, pool.height)
save firstBlock to store
pool.height++
execute firstBlock
}
redoRequestsForPeer(pool, peerId):
for each requester in pool.requesters do
if requester.getPeerID() == peerID
enqueue msg on redoChannel for requester
Channels
Defines maxMsgSize for the maximum size of incoming messages,
SendQueueCapacity and RecvBufferCapacity for maximum sending and
receiving buffers respectively. These are supposed to prevent amplification
attacks by setting up the upper limit on how much data we can receive & send to
a peer.
Sending incorrectly encoded data will result in stopping the peer.
Configuration
This file has moved to the nodes section.
Consensus
Tenderdash consensus is a distributed protocol executed by validator processes
to agree on the next block to be added to the blockchain. It proceeds in
rounds; in each round a proposer suggests a block with a Proposal, and
validators vote on it in two phases (Prevote, Precommit). When precommits
for a block reach the quorum threshold, their BLS signature shares are
recovered into a single threshold signature that forms the block's Commit.
The protocol is specified, from the implementation, in the consensus specification:
- Primary flow of a height
- Transitions and decision rules
- Alternative flows and edge cases (including proposal timestamps and message gossip to non-validators)
- Commit handling and gossip
- Recovery and synchronisation
- Known issues and discrepancies
The wire format of consensus messages is in Consensus messages; the timestamp rules in force are in Proposal timestamps and time rules (Proposer-Based Timestamps is historical design material). Operational guidance for timestamp problems is in the PBTS runbook.
Evidence
Evidence is used to identify validators who have or are acting malicious. There are multiple types of evidence, to read more on the evidence types please see Evidence Types.
The evidence reactor works similar to the mempool reactor. When evidence is observed, it is sent to all the peers in a repetitive manner. This ensures evidence is sent to as many people as possible to avoid sensoring. After evidence is received by peers and committed in a block it is pruned from the evidence module.
Sending incorrectly encoded data or data exceeding maxMsgSize will result
in stopping the peer.
Disabling evidence
Consensus parameter evidence.max_bytes = 0 disables evidence admission,
conflict collection and gossip. broadcast_evidence returns an explicit disabled
error. Proposals contain no evidence, and the consensus byte limit rejects blocks
containing evidence.
Applications can enable or disable evidence on an existing chain through
PrepareProposal and ProcessProposal consensus parameter updates. An update in
block H takes effect for H+1, only after H commits. A block that disables evidence
can still contain evidence under its current positive limit. InitChain may
also override the genesis value for the first block.
Nodes keep the evidence database, reactor and advertised P2P channel available throughout their lifetime, so activation works with existing peer connections. While disabled, local admission, conflict collection and periodic gossip skip verification and database access; peer sync timers perform only a cheap mode check. Consensus block validation and recording committed evidence remain active, including evidence in the block that disables admission. Messages whose sends began before disabling may still arrive; disabled receivers discard them.
Pending evidence is retained across disabled periods and restarts, then pruned against the current expiration limits before reactivation. Committed-evidence markers remain available to prevent submitting the same evidence again. Conflicts observed while disabled are not buffered. Restart uses persisted consensus state; state sync installs the restored state before block sync begins, without requiring its evidence mode to match genesis. Existing state-provider verification and trust requirements are unchanged. There is no local configuration override.
How to read logs
This file has moved to the node section.
Light Client
Unsupported. The Tenderdash light client is currently unsupported. The implementation details below are provided for reference.
A light client verifies block headers and commits without running a full application replica. Application state can then be verified through proofs bound to the verified chain state, provided the application supplies compatible proofs.
Tenderdash's light package retrieves light blocks from providers, validates their structure, checks the validator-set binding and verifies commit signatures. Dash Core verification uses quorum information to verify the threshold block signature.
The client can cross-check a primary provider with witnesses. Configure independent providers and the correct Dash Core network; provider availability and correct quorum verification remain part of the security model. A successful RPC response without a verified proof does not establish application-state integrity.
For the exact verification path, see the client implementation. For running the proxy, see Configure a Light Client.
Mempool
The mempool is a in memory pool of potentially valid transactions, both to broadcast to other nodes, as well as to provide to the consensus reactor when it is selected as the block proposer.
There are two sides to the mempool state:
- External: get, check, and broadcast new transactions
- Internal: return valid transaction, update list after block commit
External functionality
External functionality is exposed via network interfaces to potentially untrusted actors.
- CheckTx - triggered via RPC or P2P
- Broadcast - gossip messages after a successful check
Internal functionality
Internal functionality is exposed via method calls to other code compiled into the tenderdash binary.
- ReapMaxBytesMaxGas - get txs to propose in the next block. Guarantees that the size of the txs is less than MaxBytes, and gas is less than MaxGas
- Update - remove tx that were included in last block
- ABCI.CheckTx - call ABCI app to validate the tx
What does it provide the consensus reactor? What guarantees does it need from the ABCI app? (talk about interleaving processes in concurrency)
Optimizations
The implementation within this library also implements a tx cache. This is so that signatures don't have to be reverified if the tx has already been seen before. However, we only store valid txs in the cache, not invalid ones. This is because invalid txs could become good later. Txs that are included in a block aren't removed from the cache, as they still may be getting received over the p2p network. These txs are stored in the cache by their hash, to mitigate memory concerns.
Applications should implement replay protection, read Replay Protection for more information.
Configuration
The mempool has various configurable paramet
Sending incorrectly encoded data or data exceeding maxMsgSize will result
in stopping the peer.
maxMsgSize equals MaxBatchBytes (10MB) + 4 (proto overhead).
MaxBatchBytes is a mempool config parameter -> defined locally. The reactor
sends transactions to the connected peers in batches. The maximum size of one
batch is MaxBatchBytes.
The mempool will not send a tx back to any peer which it received it from.
The reactor assigns an uint16 number for each peer and maintains a map from
p2p.ID to uint16. Each mempool transaction carries a list of all the senders
([]uint16). The list is updated every time mempool receives a transaction it
is already seen. uint16 assumes that a node will never have over 65535 active
peers (0 is reserved for unknown source - e.g. RPC).
Configuration
Here we describe configuration options around mempool. For the purposes of this document, they are described in a toml file, but some of them can also be passed in as environmental variables.
Config:
[mempool]
# Set true to broadcast transactions in the mempool to other nodes
broadcast = true
# Maximum number of transactions in the mempool
size = 5000
# Limit the total size of all txs in the mempool.
# This only accounts for raw transactions (e.g. given 1MB transactions and
# max-txs-bytes=5MB, mempool will only accept 5 transactions).
max-txs-bytes = 1073741824
# Size of the cache (used to filter transactions we saw earlier) in transactions
cache-size = 10000
# Do not remove invalid transactions from the cache (default: false)
# Set to true if it's not possible for any invalid transaction to become valid
# again in the future.
# When true, resubmitting a rejected transaction over RPC returns the CheckTx
# result that rejected it, as long as it stays in the cache.
keep-invalid-txs-in-cache = false
# Maximum size of a single transaction.
# NOTE: the max size of a tx transmitted over the network is {max-tx-bytes}.
max-tx-bytes = 1048576
# Maximum size of a batch of transactions to send to a peer
# Including space needed by encoding (one varint per transaction).
# XXX: Unused due to https://github.com/tendermint/tendermint/issues/5796
max-batch-bytes = 0
Broadcast
Determines whether this node gossips any valid transactions that arrive in mempool. Default is to gossip anything that passes checktx. If this is disabled, transactions are not gossiped, but instead stored locally and added to the next block this node is the proposer.
WalDir
This defines the directory where mempool writes the write-ahead logs. These files can be used to reload unbroadcasted transactions if the node crashes.
If the directory passed in is an absolute path, the wal file is
created there. If the directory is a relative path, the path is
appended to home directory of the tenderdash process to
generate an absolute path to the wal directory
(default $HOME/.tenderdash or set via TM_HOME or --home)
Size
Size defines the total amount of transactions stored in the mempool. Default is 5_000 but can be adjusted to any number you would like. The higher the size the more strain on the node.
Max Transactions Bytes
Max transactions bytes defines the total size of all the transactions in the mempool. Default is 1 GB.
Cache size
Cache size determines the size of the cache holding transactions we have already seen. The cache exists to avoid running checktx each time we receive a transaction.
Keep Invalid Transactions In Cache
Keep invalid transactions in cache determines wether a transaction in the cache, which is invalid, should be evicted. An invalid transaction here may mean that the transaction may rely on a different tx that has not been included in a block.
Max Transaction Bytes
Max transaction bytes defines the max size a transaction can be for your node. If you would like your node to only keep track of smaller transactions this field would need to be changed. Default is 1MB.
Max Batch Bytes
Max batch bytes defines the amount of bytes the node will send to a peer. Default is 0.
Note: Unused due to https://github.com/tendermint/tendermint/issues/5796
Metrics
This file has moved to the node section.
Peer Strategy and Exchange
Here we outline the design of the PeerStore and how it used by the Peer Exchange Reactor (PEX) to ensure we are connected to good peers and to gossip peers to others.
Peer Types
Certain peers are special in that they are specified by the user as persistent,
which means we auto-redial them if the connection fails, or if we fail to dial
them.
Some peers can be marked as private, which means
we will not put them in the peer store or gossip them to others.
All peers except private peers and peers coming from them are tracked using the peer store.
The rest of our peers are only distinguished by being either inbound (they dialed our public address) or outbound (we dialed them).
Discovery
Peer discovery begins with a list of seeds.
When we don't have enough peers, we
- ask existing peers
- dial seeds if we're not dialing anyone currently
On startup, we will also immediately dial the given list of persistent_peers,
and will attempt to maintain persistent connections with them. If the
connections die, or we fail to dial, we will redial every 5s for a few minutes,
then switch to an exponential backoff schedule, and after about a day of
trying, stop dialing the peer. This behavior is when persistent_peers_max_dial_period is configured to zero.
But If persistent_peers_max_dial_period is set greater than zero, terms between each dial to each persistent peer
will not exceed persistent_peers_max_dial_period during exponential backoff.
Therefore, dial_period = min(persistent_peers_max_dial_period, exponential_backoff_dial_period)
and we keep trying again regardless of maxAttemptsToDial
As long as we have less than MaxNumOutboundPeers, we periodically request
additional peers from each of our own and try seeds.
Listening
Peers listen on a configurable ListenAddr that they self-report in their
NodeInfo during handshakes with other peers. Peers accept up to
MaxNumInboundPeers incoming peers.
Address Book
Peers are tracked via their ID (their PubKey.Address()). Peers are added to the peer store from the PEX when they first connect to us or when we hear about them from other peers.
The peer store is arranged in sets of buckets, and distinguishes between vetted (old) and unvetted (new) peers. It keeps different sets of buckets for vetted and unvetted peers. Buckets provide randomization over peer selection. Peers are put in buckets according to their IP groups.
IP group can be a masked IP (e.g. 1.2.0.0 or 2602:100::) or local for
local addresses or unroutable for unroutable addresses. The mask which
corresponds to the /16 subnet is used for IPv4, /32 subnet - for IPv6.
Each group has a limited number of buckets to prevent DoS attacks coming from
that group (e.g. an attacker buying a /16 block of IPs and launching a DoS
attack).
highwayhash is used as a hashing function when calculating a bucket.
When placing a peer into a new bucket:
hash(key + sourcegroup + int64(hash(key + group + sourcegroup)) % bucket_per_group) % num_new_buckets
When placing a peer into an old bucket:
hash(key + group + int64(hash(key + addr)) % buckets_per_group) % num_old_buckets
where key - random 24 HEX string, group - IP group of the peer (e.g. 1.2.0.0),
sourcegroup - IP group of the sender (peer who sent us this address) (e.g. 174.11.0.0),
addr - string representation of the peer's address (e.g. 174.11.10.2:26656).
A vetted peer can only be in one bucket. An unvetted peer can be in multiple buckets, and each instance of the peer can have a different IP:PORT.
If we're trying to add a new peer but there's no space in its bucket, we'll remove the worst peer from that bucket to make room.
Vetting
When a peer is first added, it is unvetted.
Marking a peer as vetted is outside the scope of the p2p package.
For Tenderdash, a Peer becomes vetted once it has contributed sufficiently
at the consensus layer; ie. once it has sent us valid and not-yet-known
votes and/or block parts for NumBlocksForVetted blocks.
Other users of the p2p package can determine their own conditions for when a peer is marked vetted.
If a peer becomes vetted but there are already too many vetted peers, a randomly selected one of the vetted peers becomes unvetted.
If a peer becomes unvetted (either a new peer, or one that was previously vetted), a randomly selected one of the unvetted peers is removed from the peer store.
More fine-grained tracking of peer behaviour can be done using a trust metric (see below), but it's best to start with something simple.
Select Peers to Dial
When we need more peers, we pick addresses randomly from the addrbook with some configurable bias for unvetted peers. The bias should be lower when we have fewer peers and can increase as we obtain more, ensuring that our first peers are more trustworthy, but always giving us the chance to discover new good peers.
We track the last time we dialed a peer and the number of unsuccessful attempts we've made. If too many attempts are made, we mark the peer as bad.
Connection attempts are made with exponential backoff (plus jitter). Because
the selection process happens every ensurePeersPeriod, we might not end up
dialing a peer for much longer than the backoff duration.
If we fail to connect to the peer after 16 tries (with exponential backoff), we
remove from peer store completely. But for persistent peers, we indefinitely try to
dial all persistent peers unless persistent_peers_max_dial_period is configured to zero
Select Peers to Exchange
When we’re asked for peers, we select them as follows:
- select at most
maxGetSelectionpeers - try to select at least
minGetSelectionpeers - if we have less than that, select them all. - select a random, unbiased
getSelectionPercentof the peers
Send the selected peers. Note we select peers for sending without bias for vetted/unvetted.
Preventing Spam
There are various cases where we decide a peer has misbehaved and we disconnect from them. When this happens, the peer is removed from the peer store and black listed for some amount of time. We call this "Disconnect and Mark". Note that the bad behaviour may be detected outside the PEX reactor itself (for instance, in the mconnection, or another reactor), but it must be communicated to the PEX reactor so it can remove and mark the peer.
In the PEX, if a peer sends us an unsolicited list of peers, or if the peer sends a request too soon after another one, we Disconnect and MarkBad.
Trust Metric
The quality of peers can be tracked in more fine-grained detail using a Proportional-Integral-Derivative (PID) controller that incorporates current, past, and rate-of-change data to inform peer quality.
While a PID trust metric has been implemented, it remains for future work to use it in the PEX.
See the trustmetric and trustmetric useage architecture docs for more details.
RPC
Tenderdash exposes HTTP JSON-RPC methods for node status, blocks, transactions, application queries and events. With RPC enabled, opening the configured RPC listen address in a browser lists the endpoints available on that node.
See the RPC API reference for method parameters and responses. The RPC routes and implementation define the methods supported by this source version.
For clients written in Go, use the RPC client package. For event queries and subscriptions, see event subscriptions.
Restrict RPC exposure according to your deployment. Administrative and debugging methods must not be accessible to untrusted clients. See configuration and running in production.
Running In Production
This file has moved to the nodes section.
State sync
State sync allows new nodes to rapidly bootstrap and join the network by discovering, fetching, and restoring state machine snapshots. For more information, see the state sync ABCI section).
The state sync reactor has two main responsibilities:
-
Serving state machine snapshots taken by the local ABCI application to new nodes joining the network.
-
Discovering existing snapshots and fetching snapshot chunks for an empty local application being bootstrapped.
The state sync process for bootstrapping a new node is described in detail in the section linked
above. While technically part of the reactor (see statesync/syncer.go and related components),
this document will only cover the P2P reactor component.
For details on the ABCI methods and data types, see the ABCI documentation.
Information on how to configure state sync is located in the nodes section
State Sync P2P Protocol
When a new node begin state syncing, it will ask all peers it encounters if it has any available snapshots:
type snapshotsRequestMessage struct{}
The receiver will query the local ABCI application via ListSnapshots, and send a message
containing snapshot metadata (limited to 4 MB) for each of the 10 most recent snapshots:
type snapshotsResponseMessage struct {
Height uint64
Format uint32
Chunks uint32
Hash []byte
Metadata []byte
}
The node running state sync will offer these snapshots to the local ABCI application via
OfferSnapshot ABCI calls, and keep track of which peers contain which snapshots. Once a snapshot
is accepted, the state syncer will request snapshot chunks from appropriate peers:
type chunkRequestMessage struct {
Height uint64
Format uint32
Index uint32
}
The receiver will load the requested chunk from its local application via LoadSnapshotChunk,
and respond with it (limited to 16 MB):
type chunkResponseMessage struct {
Height uint64
Format uint32
Index uint32
Chunk []byte
Missing bool
}
Here, Missing is used to signify that the chunk was not found on the peer, since an empty
chunk is a valid (although unlikely) response.
The returned chunk is given to the ABCI application via ApplySnapshotChunk until the snapshot
is restored. If a chunk response is not returned within some time, it will be re-requested,
possibly from a different peer.
The ABCI application is able to request peer bans and chunk refetching as part of the ABCI protocol.
If no state sync is in progress (i.e. during normal operation), any unsolicited response messages are discarded.
Subscribing to Events
A Tenderdash node emits events about important state transitions during
consensus. These events can be queried by clients via the RPC interface
on nodes that enable it. The list of supported event types can
be found in the github.com/dashpay/tenderdash/types Go package.
There are two APIs to query events:
-
The legacy streaming API, comprising the
subscribe,unsubscribe, andunsubscribe_allRPC methods over websocket. -
The event log API, comprising the
eventsRPC method.
The event log API supports polling and pagination. Use the API supported by your node and client; the sections below describe both interfaces.
Filter Queries
Event requests take a filter query parameter. A filter query is a string that describes a subset of available event items to return. An empty query matches all events; otherwise a query comprises one or more terms comparing event metadata to target values.
For example, to select new block events, use the term:
tm.event = 'NewBlock'
Multiple terms can be combined with AND (case matters), for example to match
the transaction event with a given hash, use the query:
tm.event = 'Tx' AND tx.hash = 'EA7B33F'
Operands may be strings in single quotes ('Tx'), numbers (45), dates, or
timestamps.
The comparison operators include =, <, <=, >, >=, and CONTAINS (for
substring match). In addition, the EXISTS operator checks for the presence
of an attribute regardless of its value.
Attributes
Tenderdash implicitly defines a string-valued tm.event attribute for all
event types. Transaction items (type Tx) are also assigned tx.hash
(string), giving the hash of the transaction, and and tx.height (number)
giving the height of the block containing the transaction. For NewBlock and
NewBlockHeader events, Tenderdash defines a block.height attribute giving
the height of the block.
Additional attributes can be provided by the application as ABCI Event
records in ResponseProcessProposal.Events for block events and
ExecTxResult.Events for transaction events. These events are published after
finalization; ResponseFinalizeBlock has no events field. The full name
of the attribute in the query is formed by combining the type and attribute
key with a period.
For example, given the events
[]abci.Event{{
Type: "reward",
Attributes: []abci.EventAttribute{
{Key: "address", Value: "example-address"},
{Key: "amount", Value: "45.62"},
{Key: "balance", Value: "100.390001"},
},
}}
a query may refer to the names reward.address, reward.amount, and reward.balance, as in:
reward.address EXISTS AND reward.balance > 45
Certain application-specific metadata are also indexed for offline queries. See Indexing transactions for more details.
Event Log API
When the rpc.event-log-window-size
configuration is enabled, the node maintains maintains a log of all events
within this operator-defined time window. This API supersedes the websocket
subscription API described below.
Clients can query these events can by long-polling the /events RPC method,
which returns the most recent items from the log that match the request
parameters. Each item returned includes a cursor that marks its
location in the log. Cursors can be passed via the before and after
parameters to fetch events earlier in the log.
For example, this request:
{
"jsonrpc": "2.0",
"id": 1,
"method": "events",
"params": {
"filter": {
"query": "tm.event = 'Tx' AND app.key = 'applesauce'"
},
"maxItems": 1,
"after": ""
}
}
will return a result similar to the following:
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"items": [
{
"cursor": "16ee3d5e65be53d8-03d5",
"event": "Tx",
"data": {
"type": "tendermint/event/Tx",
"value": {
"height": 70,
"tx": "YXBwbGVzYXVjZT1zeXJ1cA==",
"result": {
"events": [
{
"type": "app",
"attributes": [
{
"key": "creator",
"value": "Example User",
"index": true
},
{
"key": "key",
"value": "applesauce",
"index": true
},
{
"key": "index_key",
"value": "index is working",
"index": true
},
{
"key": "noindex_key",
"value": "index is working",
"index": false
}
]
}
]
}
}
}
}
],
"more": false,
"oldest": "16ee3d4c471c3b00-0001",
"newest": "16ee3d5f2e05a4e0-0400"
}
}
The "items" array gives the matching items (up to the requested
"maxResults") in decreasing time order (i.e., newest to oldest). In this
case, there is only one result, but if there are additional results that were
not returned, the "more" flag will be true. Calling /events again with the
same query and "after" set to the cursor of the newest result (in this
example, "16ee3d5e65be53d8-03d5") will fetch newer results.
Go clients can use the eventstream package to simplify the use
of this method. The eventstream.Stream automatically handles polling for new
events, updating the cursor, and reporting any missed events.
Legacy Streaming API
For a polling interface, use the event log API. See ADR 075 for design context.
To subscribe to events in the streaming API, you must connect to the node RPC service using a websocket. From the command line you can use a tool such as wscat, for example:
wscat ws://127.0.0.1:26657/websocket
To subscribe to events, call the subscribe JSON-RPC method method passing in
a filter query for the events you wish to receive:
{
"jsonrpc": "2.0",
"method": "subscribe",
"id": 1,
"params": {
"query": "tm.event='NewBlock'"
}
}
The subscribe method returns an initial response confirming the subscription,
then sends additional JSON-RPC response messages containing the matching events
as they are published. The subscription continues until either the client
explicitly cancels the subscription (by calling unsubscribe or
unsubscribe_all) or until the websocket connection is terminated.
Using Tenderdash
Serialized key types such as
tendermint/PubKeyBLS12381are protocol identifiers. Keep those names unchanged when configuring Tenderdash.
This is a guide to using the tenderdash program from the command line.
It assumes only that you have the tenderdash binary installed and have
some rudimentary idea of what Tenderdash and ABCI are.
You can see the help menu with tenderdash --help, and the version
number with tenderdash version.
Directory Root
The default directory for blockchain data is ~/.tenderdash. Override
this by setting the TMHOME environment variable.
Initialize
Initialize the root directory by running:
tenderdash init validator
This will create a new private key (priv_validator_key.json), and a
genesis file (genesis.json) containing the associated public key, in
$TMHOME/config. This is all that's necessary to run a local testnet
with one validator.
For more elaborate initialization, see the testnet command:
tenderdash testnet --help
Genesis
genesis.json in the node's config directory defines the initial chain ID,
height, time, consensus parameters, application state and validator quorum.
Use a unique chain ID of at most 50 bytes, and use the same genesis on every
node in the network.
For a local development chain, tenderdash init validator writes a complete
genesis and matching signing files. Do not construct a production quorum by
copying keys from an example. The current
GenesisDoc definition
describes the supported fields, including validator pro-tx-hashes, quorum hash,
quorum type and threshold public key.
The application may initialize the validator set and consensus parameters via
InitChain. See the canonical startup handshake
and validator-set rules.
Run
To run a Tenderdash node, use:
tenderdash start
By default, Tenderdash will try to connect to an ABCI application on
127.0.0.1:26658. If you have the kvstore ABCI app installed, run it in
another window. If you don't, kill Tenderdash and run an in-process version of
the kvstore app:
tenderdash start --proxy-app=kvstore
After a few seconds, you should see blocks start streaming in. Note that blocks are produced regularly, even if there are no transactions. See No Empty Blocks, below, to modify this setting.
Tenderdash supports in-process versions of the counter, kvstore, and noop
apps that ship as examples with abci-cli. It's easy to compile your app
in-process with Tenderdash if it's written in Go. If your app is not written in
Go, run it in another process, and use the --proxy-app flag to specify the
address of the socket it is listening on, for instance:
tenderdash start --proxy-app=/var/run/abci.sock
You can find out what flags are supported by running tenderdash start --help.
Transactions
To send a transaction, use curl to make requests to the Tenderdash RPC
server, for example:
curl http://localhost:26657/broadcast_tx_commit?tx=\"abcd\"
We can see the chain's status at the /status end-point:
curl http://localhost:26657/status | json_pp
and the latest_app_hash in particular:
curl http://localhost:26657/status | json_pp | grep latest_app_hash
Visit http://localhost:26657 in your browser to see the list of other
endpoints. Some take no arguments (like /status), while others specify
the argument name and use _ as a placeholder.
TIP: See the RPC API reference.
Formatting
When sending transactions to the RPC interface, the following formatting rules must be followed:
Using GET (with parameters in the URL):
To send a UTF8 string as transaction data, enclose the value of the tx
parameter in double quotes:
curl 'http://localhost:26657/broadcast_tx_commit?tx="hello"'
which sends a 5-byte transaction: "h e l l o" [68 65 6c 6c 6f].
Note that the URL in this example is enclosed in single quotes to prevent the shell from interpreting the double quotes. Alternatively, you may escape the double quotes with backslashes:
curl http://localhost:26657/broadcast_tx_commit?tx=\"hello\"
The double-quoted format works with for multibyte characters, as long as they are valid UTF8, for example:
curl 'http://localhost:26657/broadcast_tx_commit?tx="€5"'
sends a 4-byte transaction: "€5" (UTF8) [e2 82 ac 35].
Arbitrary (non-UTF8) transaction data may also be encoded as a string of
hexadecimal digits (2 digits per byte). To do this, omit the quotation marks
and prefix the hex string with 0x:
curl http://localhost:26657/broadcast_tx_commit?tx=0x68656C6C6F
which sends the 5-byte transaction: [68 65 6c 6c 6f].
Using POST (with parameters in JSON), the transaction data are sent as a JSON
string in base64 encoding:
curl http://localhost:26657 -H 'Content-Type: application/json' --data-binary '{
"jsonrpc": "2.0",
"id": "anything",
"method": "broadcast_tx_commit",
"params": {
"tx": "aGVsbG8="
}
}'
which sends the same 5-byte transaction: [68 65 6c 6c 6f].
Note that the hexadecimal encoding of transaction data is not supported in
JSON (POST) requests.
Reset
:warning: UNSAFE Only do this in development and only if you can afford to lose all blockchain data!
To reset a blockchain, stop the node and run:
tenderdash unsafe-reset-all
This command will remove the data directory and reset private validator and address book files.
Configuration
Tenderdash uses a config.toml for configuration. For details, see the
config specification.
Notable options include the socket address of the application
(proxy-app), the listening address of the Tenderdash peer
(p2p.laddr), and the listening address of the RPC server
(rpc.laddr).
Some fields from the config file can be overwritten with flags.
No Empty Blocks
Configure create-empty-blocks and create-empty-blocks-interval in
config.toml to control waiting for transactions at round 0 (and, on consensus
version 2, in a round entered after a silent proposer's round was skipped). The application
can request an immediate next block through ResponseFinalizeBlock.
An app hash change does not itself require an extra block: Tenderdash uses
same-block execution.
See the configuration guide and the canonical empty-block rules.
Broadcast API
Earlier, we used the broadcast_tx_commit endpoint to send a
transaction. When a transaction is sent to a Tenderdash node, it will
run via CheckTx against the application. If it passes CheckTx, it
can be admitted to the mempool and gossiped to peers. Admission does not
guarantee later inclusion in a block.
Since there are multiple phases to processing a transaction, we offer multiple endpoints to broadcast a transaction:
/broadcast_tx_async
/broadcast_tx_sync
/broadcast_tx_commit
These correspond to no-processing, processing through the mempool, and
processing through a block, respectively. That is, broadcast_tx_async,
will return right away without waiting to hear if the transaction is
even valid, while broadcast_tx_sync will return with the result of
running the transaction through CheckTx. Using broadcast_tx_commit
will wait until the transaction is committed in a block or until some
timeout is reached, but will return right away if the transaction does
not pass CheckTx. The return value for broadcast_tx_commit includes
check_tx for mempool validation and tx_result for the committed
transaction execution result.
The benefit of using broadcast_tx_commit is that the request returns
after the transaction is committed (i.e. included in a block). For a quick result, use
broadcast_tx_sync, but the transaction will not be committed until
later, and by that point its effect on the state may change.
Note the mempool does not provide strong guarantees - just because a tx passed CheckTx (ie. was accepted into the mempool), doesn't mean it will be committed, as nodes with the tx in their mempool may crash before they get to propose. For more information, see the mempool guide.
Tenderdash Networks
All nodes joining a network must use its agreed genesis and compatible software.
For a local development chain, initialize files with tenderdash init validator;
for a deployed network, obtain its genesis and validator/quorum configuration
through the network's deployment process.
priv_validator_key.json contains secret signing material. Protect both keys and
signing state; never copy a live consensus identity to a second active signer.
See validator keys and
recovery.
A validator signs only when it belongs to the active quorum. A non-validator full node follows committed blocks without voting. See the validator guide for membership updates and proposer selection for the versioned, equal-power rotation rules.
Peers
Seed
A seed node is a node who relays the addresses of other peers which they know of. These nodes constantly crawl the network to try to get more peers. The addresses which the seed node relays get saved into a local address book. Once these are in the address book, you will connect to those addresses directly. Basically the seed nodes job is just to relay everyones addresses. You won't connect to seed nodes once you have received enough addresses, so typically you only need them on the first start. The seed node will immediately disconnect from you after sending you some addresses.
Persistent Peer
Persistent peers are people you want to be constantly connected with. If you disconnect you will try to connect directly back to them as opposed to using another address from the address book. On restarts you will always try to connect to these peers regardless of the size of your address book.
All peers relay peers they know of by default. This is called the peer exchange protocol (PeX). With PeX, peers will be gossiping about known peers and forming a network, storing peer addresses in the addrbook. Because of this, you don't have to use a seed node if you have a live persistent peer.
Connecting to Peers
To connect to peers on start-up, specify them in the
$TMHOME/config/config.toml or on the command line. Use seeds to
specify seed nodes, and
persistent-peers to specify peers that your node will maintain
persistent connections with.
For example,
tenderdash start --p2p.seeds "f9baeaa15fedf5e1ef7448dd60f46c01f1a9e9c4@1.2.3.4:26656,0491d373a8e0fcf1023aaf18c51d6a1d0d4f31bd@5.6.7.8:26656"
Alternatively, you can use the /dial_seeds endpoint of the RPC to
specify seeds for a running node to connect to:
curl 'localhost:26657/dial_seeds?seeds=\["f9baeaa15fedf5e1ef7448dd60f46c01f1a9e9c4@1.2.3.4:26656","0491d373a8e0fcf1023aaf18c51d6a1d0d4f31bd@5.6.7.8:26656"\]'
Note, with PeX enabled, you should not need seeds after the first start.
If you want Tenderdash to connect to specific set of addresses and
maintain a persistent connection with each, you can use the
--p2p.persistent-peers flag or the corresponding setting in the
config.toml or the /dial_peers RPC endpoint to do it without
stopping Tenderdash core instance.
tenderdash start --p2p.persistent_peers "429fcf25974313b95673f58d77eacdd434402665@10.11.12.13:26656,96663a3dd0d7b9d17d4c8211b191af259621c693@10.11.12.14:26656"
curl 'localhost:26657/dial_peers?persistent=true&peers=\["429fcf25974313b95673f58d77eacdd434402665@10.11.12.13:26656","96663a3dd0d7b9d17d4c8211b191af259621c693@10.11.12.14:26656"\]'
Adding a Non-Validator
Adding a non-validator is simple. Just copy the original genesis.json
to ~/.tenderdash/config on the new machine and start the node,
specifying seeds or persistent peers as necessary. If no seeds or
persistent peers are specified, the node won't make any blocks, because
it's not a validator, and it won't hear about any blocks, because it's
not connected to the other peer.
Adding a Validator
Editing a local genesis file does not change the validator set of a running
network. Initial validators are established through genesis and InitChain;
later changes come from the application and take effect at the next height.
Follow the network's quorum deployment process and the canonical
validator-set update rules.
Local Network
To run a network locally, say on a single machine, you must change the _laddr
fields in the config.toml (or using the flags) so that the listening
addresses of the various sockets don't conflict. Additionally, you must set
addr_book_strict=false in the config.toml, otherwise Tenderdash's p2p
library will deny making connections to peers with the same IP address.
Upgrading
See the UPGRADING.md guide. You may need to reset your chain between major breaking releases. Although, we expect Tenderdash to have fewer breaking releases in the future (especially after 1.0 release).
Validators
This file has moved to the node section.
Overview
Tenderdash has some tools that are associated with it for:
Benchmarking
tm-load-test is an external load testing tool originally developed for upstream
networks. Verify its transaction format and RPC compatibility before using it
with Tenderdash.
Testnets
This external repository contains upstream testnet configurations. They are not Tenderdash network configurations.
Use Docker Compose to spin up Tenderdash testnets on your local machine.
Use Terraform and Ansible to deploy Tenderdash testnets to the cloud.
See the tenderdash testnet --help command for more help initializing testnets.
Debugging
Tenderdash debug kill
Tenderdash comes with a debug sub-command that allows you to kill a live
Tenderdash process while collecting useful information in a compressed archive.
The information includes the configuration used, consensus state, network
state, the node' status, the WAL, and even the stack trace of the process
before exit. These files can be useful to examine when debugging a faulty
Tenderdash process.
tenderdash debug kill <pid> </path/to/out.zip> --home=</path/to/app.d>
will write debug info into a compressed archive. The archive will contain the following:
├── config.toml
├── consensus_state.json
├── net_info.json
├── stacktrace.out
├── status.json
└── wal
Under the hood, debug kill fetches info from /status, /net_info, and
/dump_consensus_state HTTP endpoints, and kills the process with -6, which
catches the go-routine dump.
Tenderdash debug dump
Also, the debug dump sub-command allows you to dump debugging data into
compressed archives at a regular interval. These archives contain the goroutine
and heap profiles in addition to the consensus state, network info, node
status, and even the WAL.
tenderdash debug dump </path/to/out> --home=</path/to/app.d>
will perform similarly to kill except it only polls the node and
dumps debugging data every frequency seconds to a compressed archive under a
given destination directory. Each archive will contain:
├── consensus_state.json
├── goroutine.out
├── heap.out
├── net_info.json
├── status.json
└── wal
Note: goroutine.out and heap.out will only be written if a profile address is provided and is operational. This command is blocking and will log any error.
Tenderdash Inspect
Tenderdash includes an inspect command for querying Tenderdash's state store and block
store over Tenderdash RPC.
When the Tenderdash consensus engine detects inconsistent state, it will crash the
entire Tenderdash process.
While in this inconsistent state, a node running Tenderdash's consensus engine will not start up.
The inspect command runs only a subset of Tenderdash's RPC endpoints for querying the block store
and state store.
inspect allows operators to query a read-only view of the stage.
inspect does not run the consensus engine at all and can therefore be used to debug
processes that have crashed due to inconsistent state.
To start the inspect process, run
tenderdash inspect
RPC endpoints
The list of available RPC endpoints can be found by making a request to the RPC port.
For an inspect process running on 127.0.0.1:26657, navigate your browser to
http://127.0.0.1:26657/ to retrieve the list of enabled RPC endpoints.
Additional information on the Tenderdash RPC endpoints can be found in the rpc documentation.
Debug Like A Pro
Intro
Tenderdash Core is a fairly robust BFT replication engine. Unfortunately, as with other software, failures sometimes do happen. The question is then “what do you do” when the system deviates from the expected behavior.
The first response is usually to take a look at the logs. By default, Tenderdash writes logs to standard output ¹.
I[2020-05-29|03:03:16.145] Committed state module=state height=2282 txs=0 appHash=0A27BC6B0477A8A50431704D2FB90DB99CBFCB67A2924B5FBF6D4E78538B67C1I[2020-05-29|03:03:21.690] Executed block module=state height=2283 validTxs=0 invalidTxs=0I[2020-05-29|03:03:21.698] Committed state module=state height=2283 txs=0 appHash=EB4E409D3AF4095A0757C806BF160B3DE4047AC0416F584BFF78FC0D44C44BF3I[2020-05-29|03:03:27.994] Executed block module=state height=2284 validTxs=0 invalidTxs=0I[2020-05-29|03:03:28.003] Committed state module=state height=2284 txs=0 appHash=3FC9237718243A2CAEE3A8B03AE05E1FC3CA28AEFE8DF0D3D3DCE00D87462866E[2020-05-29|03:03:32.975] enterPrevote: ProposalBlock is invalid module=consensus height=2285 round=0 err="wrong signature (#35): C683341000384EA00A345F9DB9608292F65EE83B51752C0A375A9FCFC2BD895E0792A0727925845DC13BA0E208C38B7B12B2218B2FE29B6D9135C53D7F253D05"
If you’re running a validator in production, it might be a good idea to forward the logs for analysis using filebeat or similar tools. Also, you can set up a notification in case of any errors.
The logs should give you the basic idea of what has happened. In the worst-case scenario, the node has stalled and does not produce any logs (or simply panicked).
The next step is to call /status, /net_info, /consensus_state and /dump_consensus_state RPC endpoints.
curl http://<server>:26657/status$ curl http://<server>:26657/net_info$ curl http://<server>:26657/consensus_state$ curl http://<server>:26657/dump_consensus_state
Please note that /consensus_state and /dump_consensus_state may not return a result if the node has stalled (since they try to get a hold of the consensus mutex).
The output of these endpoints contains all the information needed for developers to understand the state of the node. It will give you an idea if the node is lagging behind the network, how many peers it’s connected to, and what the latest consensus state is.
At this point, if the node is stalled and you want to restart it, the best thing you can do is to kill it with -6 signal:
kill -6 <PID>
which will dump the list of the currently running goroutines. The list is super useful when debugging a deadlock.
PID is the Tenderdash’s process ID. You can find it out by running ps -a | grep tenderdash | awk ‘{print $1}’
Tenderdash debug kill
To ease the burden of collecting different pieces of data Tenderdash Core provides the Tenderdash debug kill tool, which will do all of the above steps for you, wrapping everything into a nice archive file.
tenderdash debug kill <pid> </path/to/out.zip> — home=</path/to/app.d>
Here’s the official documentation page — debugging guide
If you’re using a process supervisor, like systemd, it will restart the Tenderdash automatically. We strongly advise you to have one in production. If not, you will need to restart the node by hand.
Another advantage of using Tenderdash debug is that the same archive file can be given to Tenderdash Core developers, in cases where you think there’s a software issue.
Tenderdash debug dump
Okay, but what if the node has not stalled, but its state is degrading over time? Tenderdash debug dump to the rescue!
tenderdash debug dump </path/to/out> — home=</path/to/app.d>
It won’t kill the node, but it will gather all of the above data and package it into an archive file. Plus, it will also make a heap dump, which should help if Tenderdash is leaking memory.
At this point, depending on how severe the degradation is, you may want to restart the process.
Tenderdash Inspect
What if the Tenderdash node will not start up due to inconsistent consensus state?
When a node running the Tenderdash consensus engine detects an inconsistent state
it will crash the entire Tenderdash process.
The Tenderdash consensus engine cannot be run in this inconsistent state and the so node
will fail to start up as a result.
The Tenderdash RPC server can provide valuable information for debugging in this situation.
The Tenderdash inspect command will run a subset of the Tenderdash RPC server
that is useful for debugging inconsistent state.
Running inspect
Start up the inspect tool on the machine where Tenderdash crashed using:
tenderdash inspect --home=</path/to/app.d>
inspect will use the data directory specified in your Tenderdash configuration file.
inspect will also run the RPC server at the address specified in your Tenderdash configuration file.
Using inspect
With the inspect server running, you can access RPC endpoints that are critically important
for debugging.
Calling the /status, /consensus_state and /dump_consensus_state RPC endpoint
will return useful information about the Tenderdash consensus state.
Outro
We’re hoping that these Tenderdash tools will become de facto the first response for any accidents.
Let us know what your experience has been so far! Have you had a chance to try tenderdash debug or tenderdash inspect yet?
Report reproducible node issues in the Tenderdash issue tracker.
—
[1]: Of course, you’re free to redirect the Tenderdash’s output to a file or forward it to another server.
Proposer-Based Timestamps Runbook
Tenderdash constrains the timestamps included in each block. These
constraints mean that validators may
fail to produce valid blocks or may issue nil prevotes for proposed blocks
depending on the configuration of the validator's local clock.
What is this document for?
This document provides a set of actionable steps for application developers and node operators to diagnose and fix issues related to clock synchronization and configuration of the Proposer-Based Timestamps SynchronyParams.
Use this runbook if you observe that validators are frequently voting nil for a block that the rest
of the network votes for or if validators are frequently producing block proposals
that are not voted for by the rest of the network.
Requirements
To use this runbook, you must be running a node that has the Prometheus metrics endpoint enabled and the Tenderdash RPC endpoint enabled and accessible.
It is strongly recommended to also run a Prometheus metrics collector to gather and analyze metrics from the Tenderdash node.
Debugging a Single Node
If you observe that a single validator is frequently failing to produce blocks or voting nil for proposals that other validators vote for and suspect it may be related to clock synchronization, use the following steps to debug and correct the issue.
Check Timely Metric
Tenderdash exposes a histogram metric for the difference between the timestamp in the proposal the and the time read from the node's local clock when the proposal is received.
The histogram exposes multiple metrics on the Prometheus /metrics endpoint called
tendermint_consensus_proposal_timestamp_difference_bucket.tendermint_consensus_proposal_timestamp_difference_sum.tendermint_consensus_proposal_timestamp_difference_count.
Each metric is also labeled with the key is_timely, which can have a value of
true or false.
From the Prometheus Collector UI
If you are running a Prometheus collector, navigate to the query web interface and select the 'Graph' tab.
Issue a query for the following:
tendermint_consensus_proposal_timestamp_difference_count{is_timely="false"} /
tendermint_consensus_proposal_timestamp_difference_count{is_timely="true"}
This query will graph the ratio of proposals the node considered timely to those it considered untimely. If the ratio is increasing, it means that your node is consistently seeing more proposals that are far from its local clock. If this is the case, you should check to make sure your local clock is properly synchronized to NTP.
From the /metrics url
If you are not running a Prometheus collector, navigate to the /metrics endpoint
exposed on the Prometheus metrics port with curl or a browser.
Search for the tendermint_consensus_proposal_timestamp_difference_count metrics.
This metric is labeled with is_timely. Investigate the value of
tendermint_consensus_proposal_timestamp_difference_count where is_timely="false"
and where is_timely="true". Refresh the endpoint and observe if the value of is_timely="false"
is growing.
If you observe that is_timely="false" is growing, it means that your node is consistently
seeing proposals that are far from its local clock. If this is the case, you should check
to make sure your local clock is properly synchronized to NTP.
Checking Clock Sync
NTP configuration and tooling is very specific to the operating system and distribution
that your validator node is running. This guide assumes you have timedatectl installed with
chrony, a popular tool for interacting with time
synchronization on Linux distributions. If you are using an operating system or
distribution with a different time synchronization mechanism, please consult the
documentation for your operating system to check the status and re-synchronize the daemon.
Check if NTP is Enabled
$ timedatectl
From the output, ensure that NTP service is active. If NTP service is inactive, run:
$ timedatectl set-ntp true
Re-run the timedatectl command and verify that the change has taken effect.
Check if Your NTP Daemon is Synchronized
Check the status of your local chrony NTP daemon using by running the following:
$ chronyc tracking
If the chrony daemon is running, you will see output that indicates its current status.
If the chrony daemon is not running, restart it and re-run chronyc tracking.
The System time field of the response should show a value that is much smaller than 100
milliseconds.
If the value is very large, restart the chronyd daemon.
Debugging a Network
If you observe that a network is frequently failing to produce blocks and suspect it may be related to clock synchronization, use the following steps to debug and correct the issue.
Check Prevote Message Delay
Tenderdash exposes metrics that help determine how synchronized the clocks on a network are.
These metrics are visible on the Prometheus /metrics endpoint and are called:
tendermint_consensus_quorum_prevote_delaytendermint_consensus_full_prevote_delay
These metrics calculate the difference between the timestamp in the proposal message and the timestamp of a prevote that was issued during consensus.
The tendermint_consensus_quorum_prevote_delay metric is the interval in seconds
between the proposal timestamp and the timestamp of the earliest prevote that
achieved a quorum during the prevote step.
The tendermint_consensus_full_prevote_delay metric is the interval in seconds
between the proposal timestamp and the timestamp of the latest prevote in a round
where 100% of the validators voted.
From the Prometheus Collector UI
If you are running a Prometheus collector, navigate to the query web interface and select the 'Graph' tab.
Issue a query for the following:
sum(tendermint_consensus_quorum_prevote_delay) by (proposer_address)
This query will graph the difference in seconds for each proposer on the network.
If the value is much larger for some proposers, then the issue is likely related to the clock synchronization of their nodes. Contact those proposers and ensure that their nodes are properly connected to NTP using the steps for Debugging a Single Node.
If the value is relatively similar for all proposers you should next compare this
value to the SynchronyParams values for the network. Continue to the Checking
Sychrony steps.
From the /metrics url
If you are not running a Prometheus collector, navigate to the /metrics endpoint
exposed on the Prometheus metrics port.
Search for the tendermint_consensus_quorum_prevote_delay metric. There will be one
entry of this metric for each proposer_address. If the value of this metric is
much larger for some proposers, then the issue is likely related to synchronization of their
nodes with NTP. Contact those proposers and ensure that their nodes are properly connected
to NTP using the steps for Debugging a Single Node.
If the values are relatively similar for all proposers you should next compare,
you'll need to compare this value to the SynchronyParams for the network. Continue
to the Checking Sychrony steps.
Checking Synchrony
To determine the currently configured SynchronyParams for your network, issue a
request to your node's RPC endpoint. For a node running locally with the RPC server
exposed on port 26657, run the following command:
$ curl localhost:26657/consensus_params
The json output will contain a field named synchrony, with the following structure:
{
"precision": "500000000",
"message_delay": "3000000000"
}
The precision and message_delay values returned are listed in nanoseconds:
In the examples above, the precision is 500ms and the message delay is 3s.
Remember, tendermint_consensus_quorum_prevote_delay is listed in seconds.
If the tendermint_consensus_quorum_prevote_delay value approaches the sum of precision and message_delay,
then the value selected for these parameters is too small. Your application will
need to be modified to update the SynchronyParams to have larger values.
Updating SynchronyParams
The SynchronyParams are ConsensusParameters which means they are set and updated
by the application running alongside Tenderdash. Coordinate parameter changes through the application and the network
upgrade process. Check the application's supported parameter update mechanism.
If the application does not implement a way to update the consensus parameters programatically, then the application itself must be updated to do so. More information on updating the consensus parameters via ABCI can be found in the FinalizeBlock documentation.
Docker Compose
With Docker Compose, you can spin up local testnets with a single command.
Requirements
Build
Build the tenderdash binary and, optionally, the dashpay/tenderdash:localnode
docker image.
Note the binary will be mounted into the container so it can be updated without rebuilding the image.
# Build the linux binary in ./build
make build-linux
# (optionally) Build dashpay/tenderdash:localnode image
make build-docker-localnode
Run a testnet
To start a 4 node testnet run:
make localnet-start
The nodes bind their RPC servers to ports 26657, 26660, 26662, and 26664 on the host.
This file creates a 4-node network using the localnode image.
The nodes of the network expose their P2P and RPC endpoints to the host machine on ports 26656-26657, 26659-26660, 26661-26662, and 26663-26664 respectively.
The first node (node0) exposes two additional ports: 6060 for profiling using
pprof, and 9090 - for Prometheus
server (if you don't know how to start one check out "First steps |
Prometheus").
To update the binary, just rebuild it and restart the nodes:
make build-linux
make localnet-start
Configuration
The make localnet-start creates files for a 4-node testnet in ./build by
calling the tenderdash testnet command.
The ./build directory is mounted to the /tendermint mount point to attach
the binary and config files to the container.
To change the number of validators / non-validators change the localnet-start Makefile target here:
localnet-start: localnet-stop
@if ! [ -f build/node0/config/genesis.json ]; then docker run --rm -v $(CURDIR)/build:/tendermint:Z dashpay/tenderdash:localnode testnet --v 5 --n 3 --o . --populate-persistent-peers --starting-ip-address 192.167.10.2 ; fi
docker compose up
The command now will generate config files for 5 validators and 3 non-validators. Along with generating new config files the docker-compose file needs to be edited. Adding 4 more nodes is required in order to fully utilize the config files that were generated.
node3: # bump by 1 for every node
container_name: node3 # bump by 1 for every node
image: "dashpay/tenderdash:localnode"
environment:
- ID=3
- LOG=${LOG:-tenderdash.log}
ports:
- "26663-26664:26656-26657" # Bump 26663-26664 by one for every node
volumes:
- ./build:/tendermint:Z
networks:
localnet:
ipv4_address: 192.167.10.5 # bump the final digit by 1 for every node
Before running it, don't forget to cleanup the old files:
# Clear the build folder
rm -rf ./build/node*
Configuring ABCI containers
To use your own ABCI applications with 4-node setup edit the docker-compose.yaml file and add image to your ABCI application.
abci0:
container_name: abci0
image: "abci-image"
build:
context: .
dockerfile: abci.Dockerfile
command: <insert command to run your abci application>
networks:
localnet:
ipv4_address: 192.167.10.6
abci1:
container_name: abci1
image: "abci-image"
build:
context: .
dockerfile: abci.Dockerfile
command: <insert command to run your abci application>
networks:
localnet:
ipv4_address: 192.167.10.7
abci2:
container_name: abci2
image: "abci-image"
build:
context: .
dockerfile: abci.Dockerfile
command: <insert command to run your abci application>
networks:
localnet:
ipv4_address: 192.167.10.8
abci3:
container_name: abci3
image: "abci-image"
build:
context: .
dockerfile: abci.Dockerfile
command: <insert command to run your abci application>
networks:
localnet:
ipv4_address: 192.167.10.9
Override the command in each node to connect to it's ABCI.
node0:
container_name: node0
image: "dashpay/tenderdash:localnode"
ports:
- "26656-26657:26656-26657"
environment:
- ID=0
- LOG=$${LOG:-tenderdash.log}
volumes:
- ./build:/tendermint:Z
command: node --proxy-app=tcp://abci0:26658
networks:
localnet:
ipv4_address: 192.167.10.2
Similarly do for node1, node2 and node3 then run testnet.
Logging
Log is saved under the attached volume, in the tenderdash.log file. If the
LOG environment variable is set to stdout at start, the log is not saved,
but printed on the screen.
Special binaries
If you have multiple binaries with different names, you can specify which one
to run with the BINARY environment variable. The path of the binary is relative
to the attached volume.
Terraform & Ansible
Note: These commands/files are not being maintained by the Tenderdash team currently. Please use them carefully.
Automated deployments are done using Terraform to create servers on Digital Ocean then Ansible to create and manage testnets on those servers.
Install
NOTE: see the integration bash script that can be run on a fresh DO droplet and will automatically spin up a 4 node testnet. The script more or less does everything described below.
- Install Terraform and Ansible on a Linux machine.
- Create a DigitalOcean API token with read and write capability.
- Install the python dopy package (
pip install dopy) - Create SSH keys (
ssh-keygen) - Set environment variables:
export DO_API_TOKEN="abcdef01234567890abcdef01234567890"
export SSH_KEY_FILE="$HOME/.ssh/id_rsa.pub"
These will be used by both terraform and ansible.
Terraform
This step will create four Digital Ocean droplets. First, go to the correct directory:
cd $GOPATH/src/github.com/dashpay/tenderdash/networks/remote/terraform
then:
terraform init
terraform apply -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE"
and you will get a list of IP addresses that belong to your droplets.
With the droplets created and running, let's setup Ansible.
Ansible
The playbooks in the ansible
directory
run ansible roles to configure the sentry node architecture. You must
switch to this directory to run ansible
(cd $GOPATH/src/github.com/dashpay/tenderdash/networks/remote/ansible).
There are several roles that are self-explanatory:
First, we configure our droplets by specifying the paths for tenderdash
(BINARY) and the node files (CONFIGDIR). The latter expects any
number of directories named node0, node1, ... and so on (equal to the
number of droplets created).
To create the node files run:
tenderdash testnet
Then, to configure our droplets run:
ansible-playbook -i inventory/digital_ocean.py -l sentrynet config.yml -e BINARY=$GOPATH/src/github.com/dashpay/tenderdash/build/tenderdash -e CONFIGDIR=$GOPATH/src/github.com/dashpay/tenderdash/networks/remote/ansible/mytestnet
Voila! All your droplets now have the tenderdash binary and required
configuration files to run a testnet.
Next, we run the install role:
ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml
which as you'll see below, executes
tenderdash node --proxy-app=kvstore on all droplets. Although we'll
soon be modifying this role and running it again, this first execution
allows us to get each node_info.id that corresponds to each
node_info.listen_addr. (This part will be automated in the future). In
your browser (or using curl), for every droplet, go to IP:26657/status
and note the two just mentioned node_info fields. Notice that blocks
aren't being created (latest_block_height should be zero and not
increasing).
Next, open roles/install/templates/systemd.service.j2 and look for the
line ExecStart which should look something like:
ExecStart=/usr/bin/tenderdash node --proxy-app=kvstore
and add the --p2p.persistent-peers flag with the relevant information
for each node. The resulting file should look something like:
[Unit]
Description={{service}}
Requires=network-online.target
After=network-online.target
[Service]
Restart=on-failure
User={{service}}
Group={{service}}
PermissionsStartOnly=true
ExecStart=/usr/bin/tenderdash node --proxy-app=kvstore --p2p.persistent-peers=167b80242c300bf0ccfb3ced3dec60dc2a81776e@165.227.41.206:26656,3c7a5920811550c04bf7a0b2f1e02ab52317b5e6@165.227.43.146:26656,303a1a4312c30525c99ba66522dd81cca56a361a@159.89.115.32:26656,b686c2a7f4b1b46dca96af3a0f31a6a7beae0be4@159.89.119.125:26656
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM
[Install]
WantedBy=multi-user.target
Then, stop the nodes:
ansible-playbook -i inventory/digital_ocean.py -l sentrynet stop.yml
Finally, we run the install role again:
ansible-playbook -i inventory/digital_ocean.py -l sentrynet install.yml
to re-run tenderdash node with the new flag, on all droplets. The
latest_block_hash should now be changing and latest_block_height
increasing. Your testnet is now up and running :)
Peek at the logs with the status role:
ansible-playbook -i inventory/digital_ocean.py -l sentrynet status.yml
Logging
The crudest way is the status role described above. You can also ship logs to Logz.io, an Elastic stack (Elastic search, Logstash and Kibana) service provider. You can set up your nodes to log there automatically. Create an account and get your API key from the notes on this page, then:
yum install systemd-devel || echo "This will only work on RHEL-based systems."
apt-get install libsystemd-dev || echo "This will only work on Debian-based systems."
go install github.com/mheese/journalbeat@latest
ansible-playbook -i inventory/digital_ocean.py -l sentrynet logzio.yml -e LOGZIO_TOKEN=ABCDEFGHIJKLMNOPQRSTUVWXYZ012345
Cleanup
To remove your droplets, run:
terraform destroy -var DO_API_TOKEN="$DO_API_TOKEN" -var SSH_KEY_FILE="$SSH_KEY_FILE"
Tenderdash Specifications
This directory hosts the canonical Markdown specifications of the Tenderdash Protocol.
It shall be used to describe protocol semantics, namely the BFT consensus engine, leader election, block propagation and light client verification. The specification includes encoding descriptions used in interprocess communication to comply with the protocol. It defines the interface between the application and Tenderdash. The english specifications are often accompanies with a TLA+ specification.
Table of Contents
- Overview
- Application Blockchain Interface
- Encoding and Digests
- Core Data Structures
- State
- Consensus Protocol (source-derived; see the consensus index)
- Proposer Selection
- Time (historical design: Proposer-Based Timestamps)
- Light Client
- The Base P2P Layer
- Peer Exchange (PEX)
- Remote Procedure Calls
- Write-Ahead Log
- Ivy Proofs
Contibuting
Contributions are welcome.
Proposals at an early stage can first be drafted as Github issues. To progress, a proposal will often need to be written out and approved as a Request For Comment (RFC).
The standard language for coding blocks is Golang.
If you find discrepancies between the spec and the code that do not have an associated issue or pull request on github, please report them in the Tenderdash issue tracker. Report security vulnerabilities privately through the repository's security policy.
Overview
Tenderdash provides Byzantine Fault Tolerant State Machine Replication using hash-linked batches of transactions. Such transaction batches are called "blocks". Hence, Tenderdash defines a "blockchain".
Each block in Tenderdash has a unique index - its Height. Height's in the blockchain are monotonic. Each block is committed by a known set of weighted Validators. Membership and weighting within this validator set may change over time. Tenderdash guarantees the safety and liveness of the blockchain so long as less than 1/3 of the total weight of the Validator set is malicious or faulty.
A commit in Tenderdash is a set of signed messages from more than 2/3 of the total weight of the current Validator set. Validators take turns proposing blocks and voting on them. Once enough votes are received, the block is considered committed. These votes are included in the next block as proof that the previous block was committed - they cannot be included in the current block, as that block has already been created.
Once a block is committed, it can be executed against an application. The application returns results for each of the transactions in the block. The application can also return changes to be made to the validator set, as well as a cryptographic digest of its latest state.
Tenderdash is designed to enable efficient verification and authentication of the latest state of the blockchain. To achieve this, it embeds cryptographic commitments to certain information in the block "header". This information includes the contents of the block (eg. the transactions), the validator set committing the block, as well as the various results returned by the application. Note, however, that block execution only occurs after a block is committed. Thus, application results can only be included in the next block.
Also note that information like the transaction results and the validator set are never
directly included in the block - only their cryptographic digests (Merkle roots) are.
Hence, verification of a block requires a separate data structure to store this information.
We call this the State. Block verification also requires access to the previous block.
Tenderdash Consensus Protocol
This document and its companions specify the Tenderdash consensus protocol
as implemented in internal/consensus and the packages it depends on.
Every non-trivial rule cites the function that implements it. Where the
implementation diverges from the Tendermint algorithm it descends from, the
implementation wins and the divergence is recorded in
Known issues and discrepancies.
Audience: protocol engineers and auditors. Readers are expected to know the Tendermint consensus algorithm (consensus paper) and the ABCI interface (ABCI spec).
Documents
| Document | Content |
|---|---|
| Consensus Protocol (this page) | Scope, terminology, system model, round state, step state machine, invariants |
| Primary flow of a height | The happy path from NewHeight to the next NewHeight, with the ABCI calls it makes |
| Transitions and decision rules | Formal transition table: guards, actions and postconditions of every step action; timeout semantics; vote-driven dispatch; prevote, precommit and proposal-validity decision procedures |
| Alternative flows and edge cases | Missing or invalid proposals, locking, valid-block bookkeeping, round skipping, nil rounds, equivocation, vote extensions, timestamps, empty blocks, non-validators, panics, DoS budgets |
| Proposer liveness and round skipping | From CONSENSUS_VERSION_2: proposer heartbeats while a round waits for transactions, skip votes, round skip certificates, the round after a skip, their parameters and budgets |
| Commit handling | Commits built from local precommits and commits received from peers: verification, parking, candidates, application of a commit, gossip |
| Recovery and synchronisation | WAL, crash recovery, ABCI handshake replay, block-sync handover |
| Known issues and discrepancies | Behaviour that looks like a bug or diverges from the intended algorithm |
Related specifications that this one links to rather than restates:
- Proposer selection — which validator proposes at
(height, round). - Signing — sign bytes, request IDs and double-signing rules for proposals and votes.
- Write-ahead log — WAL file layout and replay.
- Evidence — evidence model for misbehaviour.
- Consensus channel messages — wire format of the messages exchanged.
- ABCI expected behaviour — the ABCI call sequence from the application's point of view.
- Proposer-based timestamps — pointers to current rules and retained historical research; the rules in force are in Proposal timestamps and time rules.
Scope
In scope: the consensus state machine of a single node (validator or full node), the rules that decide what it signs, the interaction with the application over ABCI, and the parts of message exchange that change consensus state. Out of scope: the p2p transport, mempool internals, block sync and state sync (except their handover to consensus), light clients, and the complete DoS-defence design. For operational monitoring, see DoS observability.
Source of truth: the code at the revision this document was written against
(branch dev). Citations use path/file.go and a function name; line
numbers are omitted because they drift.
Terminology
| Term | Definition | Source |
|---|---|---|
Height h | Index of the block being decided. State.LastBlockHeight + 1, or InitialHeight for the first block. | StateData.updateToState in internal/consensus/state_data.go |
Round r | Attempt number within a height, starting at 0. | RoundState.Round in internal/consensus/types/round_state.go |
| Step | Position within a round: one of the RoundStepType values listed in Round steps. | internal/consensus/types/round_state.go |
| Validator set | The members of the active Dash LLMQ quorum: ValidatorSet{Validators, QuorumType, QuorumHash, ThresholdPublicKey, VotingPowerThreshold, HasPublicKeys}. Every validator has voting power DefaultDashVotingPower = 100. | types/validator_set.go |
| Validator / observer | A node whose PrivValidator pro-tx-hash is in the validator set is a validator; any other node is an observer. Observers do not sign and ignore votes (see Non-validators). | StateData.isValidator, Reactor.handleVoteMessage in internal/consensus/reactor.go |
Commit threshold T | Voting power at or above which a vote set is treated as "+2/3". T = ValidatorSet.VotingPowerThreshold when that field is set (> 0), otherwise the LLMQ-type threshold QuorumTypeThresholdVotingPower() (threshold member count × 100). A configured T below the strict floor ⌊2·total/3⌋ + 1 is only warned about (BelowStrictThreshold). | ValidatorSet.QuorumVotingThresholdPower, ValidatorSet.BelowStrictThreshold in types/validator_set.go; warning logged in StateData.updateToState |
Recovery threshold R | Voting power whose BLS signature shares suffice to recover the quorum's threshold signature: QuorumTypeThresholdVotingPower(). It ignores the VotingPowerThreshold override. | ValidatorSet.QuorumTypeThresholdVotingPower in types/validator_set.go |
| "+2/3 any" | A vote set whose total admitted power ≥ T, regardless of which block ids the votes name. | VoteSet.HasTwoThirdsAny in types/vote_set.go |
| "+2/3 majority" / maj23 | A vote set whose votes for one block id (possibly nil) reach T. For precommit sets the majority is additionally recognised only once the threshold signature over the block has been recovered and verified (see Threshold recovery). | VoteSet.TwoThirdsMajority, VoteSet.addVerifiedVote in types/vote_set.go |
| Polka | A +2/3 majority of prevotes for one block id (nil included) in one round. | HeightVoteSet.POLInfo in internal/consensus/types/height_vote_set.go |
| POLRound | The round whose polka a proposal claims as justification; -1 when the proposal claims none. | types.Proposal.POLRound |
| BlockID | (Hash, PartSetHeader, StateID). Nil block id = all three empty. | types/block.go, types/block_meta.go |
| StateID | Hash of {AppVersion, Height, AppHash, CoreChainLockedHeight, Time} of the block header; the app hash is the state after executing that block (same-block execution). | Header.StateID in types/block.go |
| Commit | (Height, Round, BlockID, QuorumHash, ThresholdBlockSignature, ThresholdVoteExtensions): a single aggregate BLS threshold signature plus one threshold signature per threshold-recoverable vote extension. There are no per-validator signatures in a commit. | types.Commit in types/block.go |
| Locked block / round | The block the node precommitted most recently and the round it did so; -1/nil when none. | RoundState.LockedRound/LockedBlock |
| Valid block / round | The most recent block (and round) for which the node saw a polka in a round ≥ any previous one. | RoundState.ValidRound/ValidBlock |
| CurrentRoundState | The application's answer to the latest PrepareProposal/ProcessProposal for the block held in ProposalBlock, tagged with the round it was processed for. | RoundState.CurrentRoundState, RoundState.ProcessedBlockID; processedFor in internal/consensus/block_executor.go |
| Round skip certificate | A quorum threshold signature recovered from the skip votes of validators holding R, proving that they stopped hearing from the proposer of (h, r) while waiting for transactions; moves a node to r + 1. CONSENSUS_VERSION_2 only. | types.RoundSkipCertificate in types/round_skip.go; Proposer liveness |
| Parked commit | A verified commit for the current height whose block has not yet been assembled (RoundState.Commit != nil). | StateData.adoptCommit in internal/consensus/state_data.go |
| Timely | A proposal whose receive time lies within [ts − Precision, ts + MessageDelay·2^⌊round/10⌋ + Precision] of its timestamp ts. | checkTimely in types/validation.go |
Throughout, rs denotes the node's RoundState and state the committed
sm.State (state as of height h − 1).
System model as enforced by the code
- Single writer.
State.receiveRoutine(internal/consensus/state.go) is the only goroutine that mutatesRoundState. Itselects over five inputs: transactions becoming available in the mempool (txNotifier.TxsAvailable()), messages (msgInfoQueue.read(): the node's own proposal/votes on an internal queue, peers' messages on per-peer lanes), timer expiries (timeoutTicker.Chan()), proposer-silence reports (proposerLiveness.silent()) and signed round skip votes (roundSkipSigner.signed()). Go'sselectpicks randomly among ready cases, so there is no ordering guarantee between a timeout and a message that are both ready. - Actions and guards. Every transition is an
ActionHandlerregistered in theController(internal/consensus/state_controller.go) and invoked viaController.Dispatch. EachEnter*action begins with a staleness guard on(height, round, step); a dispatch that fails its guard is a silent no-op. Dispatches nest: an action may dispatch further actions before returning. - Copy-on-read state. Handlers work on a
StateDatacopy taken fromStateDataStoreand persist it withStateData.Save/StateDataStore.Update, which checks an optimistic version counter (StateDataStore.update). A version mismatch is only logged. - Equal voting power. Every validator has power 100; quorum arithmetic is therefore a member count × 100.
- Threshold signatures. Validators sign with BLS shares of the quorum key.
A commit is one recovered threshold signature, verifiable against
ValidatorSet.ThresholdPublicKeyalone. See Signing. - Same-block execution.
Header.AppHashis the application state after executing the block's own transactions; consensus validates it against theProcessProposalresult before voting (BlockExecutor.ValidateBlockWithRoundStateininternal/state/execution.go). - Safety assumption. Safety follows the Tendermint argument only while
T ≥ ⌊2·total/3⌋ + 1. The code does not enforce this: a lowerVotingPowerThreshold(or an LLMQ type with a 50 % threshold) is accepted with a warning (ValidatorSet.BelowStrictThreshold). - Time. The block timestamp is the proposer's local clock
(
State.makeBlockTimeininternal/state/state.go; genesis time for the initial height). Receivers judge timeliness withSynchronyParams(Precisiondefault 505 ms,MessageDelaydefault 12 s,types/params.go). There is no median/BFT time.
Round state
RoundState (internal/consensus/types/round_state.go) holds, per node:
| Field | Meaning |
|---|---|
Height, Round, Step, StartTime | Position in the protocol; StartTime is when round 0 of this height may start (= CommitTime of the previous height). |
CommitTime, CommitRound | When and in which round +2/3 precommits for a block were observed locally; CommitRound = -1 until then. |
Validators, LastValidators, ProposerSelector | Validator set for this height, for the previous height, and the proposer-selection strategy. |
Proposal, ProposalReceiveTime, ProposalBlock, ProposalBlockParts | The accepted proposal of the current round, when it was received, and the block being assembled from parts. |
LockedRound, LockedBlock, LockedBlockParts | Lock. |
ValidRound, ValidBlock, ValidBlockParts, ValidBlockRecvTime | Valid block. ValidBlockRecvTime is the receive time used when re-proposing. |
Commit | Parked commit (see Commit handling). |
Votes | HeightVoteSet: prevote and precommit VoteSets for rounds 0..Round+1 plus up to two catch-up rounds per validator. |
LastCommit | The commit of height h − 1, included in the next proposed block. There is no LastCommit vote set: late precommits for h − 1 are dropped. |
TriggeredTimeoutPrecommit | Whether the precommit-wait timeout was already scheduled for this round. |
RoundSkip | Round skip certificate for the highest skipped round of this height; kept for peers still in that round even after the node moved past it. |
TxsReported | Whether the mempool reported transactions at this height (it does so once per height). |
ProposeNextBlockImmediately | Hint from ResponseFinalizeBlock.propose_next_block_immediately of the previous height; consumed once, not persisted. |
CurrentRoundState, ProcessedBlockID | Latest application result for ProposalBlock (see Terminology). |
StateData.updateToState re-initialises Height, Validators,
LastValidators, LastCommit, ProposerSelector and StartTime for the
new height and resets every proposal, lock, valid-block, commit,
round-skip, vote and flag field. CurrentRoundState/ProcessedBlockID are left untouched; they
are compared against the new height's block by processedFor. Within a
height LockedRound/Block and ValidRound/Block are never reset; Commit
is cleared only when a parked commit is rejected (selectParkedCommit,
TryAddCommitAction.Execute).
Round steps
RoundStepType is an ordered enum (internal/consensus/types/round_state.go):
| Value | Step | Entered by |
|---|---|---|
| 1 | NewHeight | StateData.updateToState (after a block is applied, after block sync, at start) |
| 2 | NewRound | EnterNewRoundAction |
| 3 | Propose | EnterProposeAction |
| 4 | Prevote | EnterPrevoteAction |
| 5 | PrevoteWait | EnterPrevoteWaitAction |
| 6 | Precommit | EnterPrecommitAction |
| 7 | PrecommitWait | never assigned — EnterPrecommitWaitAction sets TriggeredTimeoutPrecommit but leaves Step at Precommit (see Known issues). The value is still used as the timeout's step and in peer-state comparisons. |
| 8 | ApplyCommit | EnterCommitAction, AddCommitAction |
stateDiagram-v2
direction TB
[*] --> NewHeight : updateToState (height h, round 0)
NewHeight --> NewRound : tock NewHeight / EnterNewRound(h,0)
NewRound --> NewRound : proposer waits for clock > LastBlockTime (NewRound tock rescheduled)
NewRound --> Propose : EnterPropose (immediately, or after txs available / empty-block interval / proposal or prevotes while waiting)
Propose --> Prevote : proposal + block (+ POL) complete, or tock Propose
Prevote --> PrevoteWait : prevotes reach +2/3 any without a usable majority
Prevote --> Precommit : polka (block held or nil)
PrevoteWait --> Precommit : polka, or tock PrevoteWait
Precommit --> Precommit : precommits reach +2/3 any (TriggeredTimeoutPrecommit, PrecommitWait tock scheduled)
Precommit --> NewRound : tock PrecommitWait / EnterNewRound(h, r+1)
Precommit --> ApplyCommit : +2/3 precommits for a block (threshold signature recovered)
state "any step" as Any
Any --> NewRound : +2/3 any prevotes or precommits at round r' > r (round skip)
Any --> NewRound : RoundSkipCertificate for round r' ≥ r, before ApplyCommit / EnterNewRound(h, r'+1)
Any --> ApplyCommit : verified CommitMessage for height h
ApplyCommit --> NewHeight : block applied (height h+1)
ApplyCommit --> ApplyCommit : waiting for block parts, or application rejected own commit
The precise guards of every arrow are in Transitions and decision rules.
Invariants
The following hold by construction of the code; each names the function that preserves it.
- Lock monotonicity.
LockedRoundonly increases within a height. It is written byStateData.updateLockedBlock(first lock) and by the relock branch ofEnterPrecommitAction.Execute(LockedRound = round), both at the current round, and reset to-1only byStateData.updateToState. There is no unlock: a nil polka at a later round does not clear the lock (TestStateLock_POLDoesNotUnlockininternal/consensus/state_test.go). - Lock implies polka. A node locks on block
Bat roundronly afterPrevotes(r).TwoThirdsMajority() == Band only whenBis this round's proposal block (EnterPrecommitAction.Execute). - Precommit implies processed. A non-nil precommit for
Bat roundris signed only afterProcessProposal(B, r)succeeded andValidateBlockWithRoundStatepassed for this round (blockExecutor.mustEnsureProcess,blockExecutor.mustValidatecalled fromEnterPrecommitAction.Execute). ConsequentlyExtendVoteis asked for the round the application processed last. - Valid-round monotonicity.
ValidRoundis set to the current round byStateData.updateValidBlock; callers only invoke it whenValidRound < Round(addVoteUpdateValidBlockMw,ProposalCompletedAction) or from the lock branch at the current round. - One proposal slot per round.
Proposaler.Setignores any proposal whilers.Proposal != nil, so the first proposal that verifies holds the slot. The slot is freed within the round only bydropStaleProposal(StateData.retargetTo), when authenticated votes name another block: a current-round polka (addVoteUpdateValidBlockMw), a polka at precommit time for a block the node does not hold (EnterPrecommitAction.Execute), or a commit (StateData.adoptCommit,proposalUpdater.updateStateData). A proposal for the retargeted block — and only that block, sinceSetrejects aBlockIDthat differs from the assembled block's — can then be accepted in the same round. A proposal that fails verification never occupies the slot. - Application result currency.
CurrentRoundStatedescribes the application's latest successfulProcessProposal/PrepareProposalforProposalBlock; a result for another round or block is never reused (blockExecutor.ensureProcess,processedFor,ensureOwnRound). - Commit = recovered threshold signature.
VoteSet.MakeCommitpanics unlessmaj23is set andthresholdBlockSigwas recovered; for precommit setsmaj23is set only after recovery and verification succeed (VoteSet.addVerifiedVote,VoteSet.recoverThresholdSignsAndVerify). - Commit matches held block. A commit is applied only when its
BlockID(hash, part-set header and state id) equals that of the block held (verifyCommitBlockininternal/consensus/state_try_add_commit.go, called by bothprepareCommitForApplyandApplyCommitAction.Execute). - WAL before signature. The WAL is flushed and fsynced before the node
signs a vote:
voteSigner.signVotereturns on aFlushAndSyncfailure and no vote is signed. On the proposal path the sync is best-effort:EnterProposeAction.Executelogs aFlushAndSyncfailure and still callsProposaler.Create, so a proposal may be signed over an unsynced WAL.EndHeightMessageis fsynced after the block is saved and beforeFinalizeBlock(ApplyCommitAction.Execute). - Height votes only. Votes and commits for any height other than
rs.Heightare dropped (addVoteValidateVoteMw,StateData.readyToApplyCommit).
Threshold recovery
VoteSet.addVerifiedVote (types/vote_set.go) sets maj23 for a block id
when the power behind it reaches T. For precommit sets it then calls
recoverThresholdSignsAndVerify:
- With a single validator the vote's own signatures are used.
- Otherwise
canonicalVoteExtensionGrouppicks the group of votes that carry an identical ordered vector of threshold-recoverable extensions and whose power is≥ R(ties broken by lexicographic order of the group identity, empty vector first). NewSignsRecoverer(...).Recover()recovers the block threshold signature from all votes for the block and the extension threshold signatures from the canonical group's votes.- The recovered signatures are verified against
ValidatorSet.ThresholdPublicKeyover the canonical vote's sign data.
If any step fails, maj23 and the recovered signatures are rolled back and
the attempt is repeated when a later vote makes the block cross T again or
makes an extension group cross R. Externally, therefore, a precommit
majority is observable only once a verifiable commit can be built from it.
Prevote sets set maj23 at T without recovery.
Timeouts and consensus parameters
Timeouts come from ConsensusParams.Timeout of the committed state (changes
proposed in block h apply from h + 1, CurrentRoundState in
internal/state/current_round_state.go), with zero fields replaced by defaults
(TimeoutParams.TimeoutParamsOrDefaults, types/params.go):
| Parameter | Default | Used for |
|---|---|---|
Propose | 3000 ms | proposeTimeout(r) = Propose + ProposeDelta·r |
ProposeDelta | 500 ms | |
Vote | 1000 ms | voteTimeout(r) = Vote + VoteDelta·r (prevote wait, precommit wait, and the signing deadline of a non-nil precommit) |
VoteDelta | 500 ms | |
Synchrony.Precision | 505 ms | Timeliness window |
Synchrony.MessageDelay | 12 s | Timeliness window; doubled every 10 rounds |
ProposerHeartbeat | 1 s, derived when only ProposerSilence is set | Heartbeat interval of the proposer of a waiting round (CONSENSUS_VERSION_2) |
ProposerSilence | 5 s, derived when only ProposerHeartbeat is set | ProposerSilenceTimeout(r) = ProposerSilence + ProposerHeartbeat·r; see Parameters |
Per-node Unsafe*TimeoutOverride settings in config.ConsensusConfig replace
the corresponding parameter locally (StateData.proposeTimeout,
StateData.voteTimeout); they desynchronise timing between nodes and are not
part of the protocol. There is no commit timeout: StartTime of the next
height equals CommitTime, so round 0 starts as soon as the block is applied
(StateData.updateToState, roundScheduler.ScheduleRound0). The
unsafe-commit-timeout-override / unsafe-bypass-commit-timeout-override
settings are deprecated no-ops that only produce a warning
(config.ConsensusConfig).
Escalation with the round is linear, not exponential.
Primary flow of a height
This page follows one height h through round 0 when every validator is
honest, the network is timely and the application accepts everything. Every
step names the function that performs it. Alternative paths branch off at the
points marked ⇢ and are specified in
Alternative flows and
Commit handling.
Sequence
sequenceDiagram
participant T as timeoutTicker
participant P as Proposer (validator)
participant V as Validator (non-proposer)
participant A as Application (ABCI)
participant S as Block/State store
Note over P,V: updateToState: Height=h, Round=0, Step=NewHeight, StartTime=CommitTime
T->>P: tock NewHeight(h)
T->>V: tock NewHeight(h)
P->>P: EnterNewRound(h,0) → EnterPropose(h,0)
V->>V: EnterNewRound(h,0) → EnterPropose(h,0), schedule Propose timeout
P->>A: PrepareProposal(txs, LocalLastCommit, misbehavior, round=0, ...)
A-->>P: TxRecords, AppHash, TxResults, validator/param/chain-lock updates
P->>P: sign Proposal (fresh block, POLRound=-1), send Proposal + BlockParts to own queue
P-->>V: ProposalMessage, BlockPartMessages (gossip)
V->>V: Proposaler.Set (verify proposer signature), AddProposalBlockPart
V->>V: ProposalCompleted → EnterPrevote(h,0)
V->>V: ValidateBlock against committed state
V->>A: ProcessProposal(block, round=0)
A-->>V: ACCEPT + AppHash/TxResults/...
V->>V: ValidateBlockWithRoundState, sign PREVOTE(B)
V-->>P: VoteMessage(prevote)
Note over P,V: prevotes for B reach T → polka
V->>V: EnterPrecommit(h,0): lock on B
V->>A: ExtendVote(B, h, 0)
A-->>V: vote extensions
V->>V: sign PRECOMMIT(B, extensions)
V-->>P: VoteMessage(precommit)
P->>A: VerifyVoteExtension(peer precommit)
A-->>P: ACCEPT
Note over P,V: precommits for B reach T, threshold signature recovered and verified → maj23
V->>V: EnterCommit(h,0) → Step=ApplyCommit → TryFinalizeCommit
V->>V: seenCommit = Precommits(0).MakeCommit()
V->>A: VerifyVoteExtension(canonical commit vote, empty pro_tx_hash)
A-->>V: ACCEPT
V->>S: SaveBlock(block, parts, commit), WAL EndHeight(h) fsync
V->>A: FinalizeBlock(block, commit)
A-->>V: ResponseFinalizeBlock
V->>S: save state, mempool update
V->>V: updateToState: Height=h+1, Round=0, Step=NewHeight, ScheduleRound0
V->>V: EnterNewRound(h+1,0) dispatched immediately
The proposer runs the same validator steps for its own proposal: its
ProposalMessage and BlockPartMessages travel through the internal queue
and are handled exactly like a peer's (Proposaler.sendMessages,
msgInfoDispatcher).
Step by step
1. New height
Precondition: a block for h − 1 was just applied, or the node started.
StateData.updateToState (internal/consensus/state_data.go) sets
Height = h, Round = 0, Step = NewHeight, LastCommit, a fresh
HeightVoteSet, a fresh ProposerSelector (from ConsensusParams.Version
and the block store), resets proposal/lock/valid/commit fields, and sets
StartTime = CommitTime (or now when there is no commit time yet).
roundScheduler.ScheduleRound0 arms a NewHeight-step timeout of
StartTime − now (≤ 0, so it fires at once).
On the vote-driven path the next height does not even wait for that tock:
addVoteDispatchPrecommitMw (internal/consensus/state_add_vote.go)
dispatches EnterNewRoundEvent{Height: stateData.Height} right after
EnterCommit returned, and because ApplyCommit already incremented the
height in place, that is EnterNewRound(h + 1, 0). The later NewHeight
tock then fails the EnterNewRound guard and is ignored.
2. New round
State.handleTimeout (internal/consensus/state.go) maps the NewHeight
tock to EnterNewRoundEvent{Height}. EnterNewRoundAction.Execute
(internal/consensus/state_enter_new_round.go):
- guard:
Height == h,round ≥ Round, and ifround == RoundthenStep == NewHeight; updateRoundStep(0, NewRound)(this also updates the proposer selector);- for round 0 the proposal fields are kept as reset by
updateToState(a proposal for round 0 may already have arrived); Votes.SetRound(1)so that round 1 votes are tracked for round skipping;TriggeredTimeoutPrecommit = false;- decides whether to wait for transactions:
waitForTxs = config.WaitForTxs() && (round == 0 || waitsAfterRoundSkip(round)) && Height != InitialHeight, overridden tofalsebyProposeNextBlockImmediatelyand, onCONSENSUS_VERSION_2chains, by a proposal already held for the round (roundSkipping && Proposal != nil). ⇢ Empty blocks. With the default configuration (create-empty-blocks = true, interval 0) it does not wait and dispatchesEnterProposeEvent{h, 0}. OnCONSENSUS_VERSION_2chains the proposer of a waiting round sends heartbeats while it waits. ⇢ Proposer liveness.
3. Propose
EnterProposeAction.Execute (internal/consensus/state_enter_propose.go):
- guard:
Height == h,round ≥ Round, and ifround == RoundthenStep < Propose; isProposer = ProposerSelector.GetProposer(Height, Round).ProTxHash == own pro-tx-hash(Proposer selection);- if the node is the proposer and
state.LastBlockTime > now, it reschedules aNewRound-step timeout for the difference and returns without changing step (block times must increase;proposerWaitTimeininternal/consensus/state.go). ⇢ Known issues; - schedules the
ProposetimeoutproposeTimeout(round); - non-proposers return here and wait for the proposal;
- the proposer, unless replaying the WAL, applying a parked commit, or
suppressed after block sync (
catchupTracker.mayPropose), flushes the WAL and callsProposaler.Create; - on return (deferred):
updateRoundStep(round, Propose), publish the step, and ifisProposalComplete()already holds, dispatchEnterPrevote.
3a. Creating the proposal (proposer only)
Proposaler.Create (internal/consensus/state_proposaler.go):
- If
checkValidBlock(rs)—ValidBlock != niland it is timely with respect toValidBlockRecvTimeandValidRound(or has the genesis time atInitialHeight) — reuseValidBlock; otherwise build a new block viablockExecutor.create→BlockExecutor.CreateProposalBlock(internal/state/execution.go):commit = LastCommit(an empty commit atInitialHeight);evidence = evpool.PendingEvidence(ConsensusParams.Evidence.MaxBytes);maxDataBytes = types.MaxDataBytes(Block.MaxBytes, commit, evidenceSize)(block overhead, max header, max core chain lock, the last commit's actual size and evidence are subtracted;types/block.go);txs = mempool.ReapMaxBytesMaxGas(maxDataBytes, Block.MaxGas);state.MakeBlock(...)withTime = now(genesis time atInitialHeight),ProposerProTxHash = own,ProposedAppVersion;- ABCI
PrepareProposalwithMaxTxBytes,Txs,LocalLastCommit,Misbehavior,Height,Round,Time,NextValidatorsHash,CoreChainLockedHeight = state.LastCoreChainLockedBlockHeight,ProposerProTxHash,ProposedAppVersion,Version,QuorumHash. An error from the call panics the node (blockExecutor.create); a response that differs for an identical request is only logged (deterministicPrepareProposalHashes); - the response is validated; removed transactions are evicted from the
mempool;
AppHash,ResultsHash,NextValidatorsHash,NextConsensusHash,CoreChainLock,AppVersionand the next validator set/params are written into the header andCurrentRoundState(RoundParamsFromPrepareProposal,CurrentRoundState.UpdateBlock).
makeProposal(h, round, polRound, block, parts), wherepolRound = ValidRoundwhenValidBlockis re-proposed andpolRound = -1when a new block was built. A new block has no polka, so it never claims one.signProposal:PrivValidator.SignProposal(chainID, QuorumType, QuorumHash, proposal)with a context deadline ofTimeout.Propose. The proposal signature coversType, Height, Round, POLRound, BlockID (hash + part-set header), Timestamp, ChainID(CanonicalizeProposalintypes/canonical.go); it does not coverCoreChainLockedHeightorBlockID.StateID.sendMessages:ProposalMessagethen oneBlockPartMessageper part (BlockPartSizeBytes), all to the internal queue. The reactor gossips them to peers from the resulting round-state events.
4. Receiving the proposal and block
proposalMessageHandler → Proposaler.Set (internal/consensus/state_proposaler.go)
accepts the proposal when it is the first for the round, is for the current
(Height, Round), has a valid POLRound, a core-chain-locked height not
below the committed one, agrees with any block/part set already held, and
carries a valid proposer signature (verifyProposal; full rules in
Proposal validity).
It records ProposalReceiveTime (the queue's enqueue time,
chanMsgSender.send in internal/consensus/msg_queue.go) and creates the
part set from BlockID.PartSetHeader unless one is under way.
blockPartMessageHandler → AddProposalBlockPartAction.Execute
(internal/consensus/state_add_prop_block.go) adds each part (Merkle proof
checked by PartSet.AddPart). When the last part arrives the block is
decoded, cross-checked against the proposal (hash and part-set header must
match, else the proposal is dropped), the proposal's CoreChainLockedHeight
and BlockID.StateID are replaced by the values derived from the block,
rs.ProposalBlock is set and ProposalCompletedEvent is dispatched.
ProposalCompletedAction.Execute: with Step ≤ Propose and
isProposalComplete() (proposal and block held, and for POLRound ≥ 0 the
POL round's prevotes already have a majority) → EnterPrevote(h, Round).
5. Prevote
EnterPrevoteAction.Execute (internal/consensus/state_enter_prevote.go)
→ prevoter.Do (internal/consensus/state_prevoter.go):
StateData.isValidForPrevote: block and proposal held; proposal timestamp equals block time; the proposal is timely (unless the block is alreadyValidBlock; in replay, only a proposal received too early fails); the core chain lock is consistent (sm.ValidateBlockChainLock); forPOLRound == -1the block'sProposerProTxHashis this round's proposer.blockExecutor.ensureProcess(rs, Round)→ValidateBlockagainst the committed state, core chain lock included, then ABCIProcessProposal(verify = true); a block failingValidateBlocknever reaches the application. The response must beACCEPT, well-formed, andValidateBlockWithRoundStatemust pass — header fieldsAppHash,ResultsHash,NextValidatorsHash,NextConsensusHashmust equal the application's answer, andLastCommitmust verify againstLastValidators. The result becomesCurrentRoundStatetagged with round 0.blockExecutor.mustValidate(panics on failure, cannot fail after step 2).prevoter.signVote: with no lock andPOLRound == -1,checkProposalBlockis true →voteSigner.signAddVote(PrevoteType, BlockID).
Step Prevote is set on return. A failure in step 1, or ErrBlockRejected
(application REJECT) or ErrInvalidBlock (ValidateBlock before the call,
ValidateBlockWithRoundState after it) in step 2, yields a nil prevote
instead. Any other step-2 error — transport failure, UNKNOWN status or a
malformed response — panics the node (prevoter.handleError). ⇢ Prevote rule.
6. Collecting prevotes
Each VoteMessage runs the prevote middleware chain
(newAddVoteAction in internal/consensus/state_add_vote.go). Once
Prevotes(0) holds T power for B:
addVoteUpdateValidBlockMw:ValidRound < 0 == Roundand the held block isB→updateValidBlock()(ValidRound = 0,ValidBlock = B,ValidBlockRecvTime = ProposalReceiveTime),retargetTo(B)(a no-op here), publish aValidBlockevent (peers learn the part-set header);addVoteDispatchPrevoteMw:Round == vote.Round,Step ≥ Prevote, majority forBandisProposalComplete()→EnterPrecommit(h, 0).
7. Precommit
EnterPrecommitAction.Execute (internal/consensus/state_enter_precommit.go):
polka for B; proposal and block held; timestamps equal; B is this
round's proposal block → mustEnsureProcess(rs, 0) (no new ABCI call: round 0
is already processed), mustValidate, updateLockedBlock()
(LockedRound = 0, LockedBlock = B), Lock event,
signAddVote(PrecommitType, B), then updateValidBlock() (already current).
voteSigner.signVote for a non-nil precommit first calls ABCI ExtendVote
(BlockExecutor.ExtendVote; an error or an invalid extension panics),
attaches the returned extensions, and signs with deadline voteTimeout(0).
The vote carries one BLS share for the block sign item and one per extension
(Signing).
8. Collecting precommits
The precommit middleware chain adds addVoteVerifyVoteExtensionMw before the
vote set: for a peer's non-nil precommit it verifies the block signature and
every extension signature (types.VerifyVoteSignatures), restores the node's
own processed round if a commit check changed it (blockExecutor.ensureOwnRound),
and asks ABCI VerifyVoteExtension; a rejection drops the vote (it is not
stored, so gossip may redeliver it).
When Precommits(0) for B reaches T and threshold recovery succeeds
(Threshold recovery),
addVoteDispatchPrecommitMw dispatches EnterNewRound(0) (no-op),
EnterPrecommit(0) (no-op) and EnterCommitEvent{h, CommitRound: 0}.
9. Commit and apply
EnterCommitAction.Execute (internal/consensus/state_enter_commit.go):
guard Height == h, Step < ApplyCommit; proposalUpdater.updateStateData
confirms the held block is B; deferred: Step = ApplyCommit (round kept),
CommitRound = 0, CommitTime = now, then TryFinalizeCommitEvent.
TryFinalizeCommitAction.Execute → finalizeCommit
(internal/consensus/state_try_finalize_commit.go): seenCommit = Precommits(CommitRound).MakeCommit() (height, round, block id, quorum hash,
recovered block signature, recovered extension signatures);
refuseUnprocessedCommit (cache of rejected vectors, empty here);
mustEnsureProcess(rs, seenCommit.Round); verifyCommitExtensions → ABCI
VerifyVoteExtension with the commit's canonical vote (empty
ValidatorProTxHash marks it as a commit, sm.VerifyCommitExtensions). On
acceptance → ApplyCommitEvent{seenCommit}.
ApplyCommitAction.Execute (internal/consensus/state_apply_commit.go):
verifyCommitBlock: part-set header, hash and state id of the commit equal those of the held block.mustEnsureProcess(rs, commit.Round),mustValidate.blockStore.SaveBlock(block, parts, commit).- WAL
WriteSync(EndHeightMessage{h})— panics on failure. blockExecutor.finalize→BlockExecutor.FinalizeBlock(internal/state/execution.go):ValidateBlockWithRoundStateagain,SaveABCIResponses(theProcessProposalresult, for crash recovery), ABCIFinalizeBlock(block, commit, round, block id),State.Update(Validators ← NextValidators,LastValidators ← Validators, params, chain-lock height,LastAppHash), mempool flush and update (flushMempool, recheck perABCI.RecheckTx), evidence-pool update,stateStore.Save, pruning perRetainHeight, events. An error panics the node.updateToState(newState, commit)→ heighth + 1, stepNewHeight.ProposeNextBlockImmediatelyis set if the response asks for it.ScheduleRound0.
Control then returns through the dispatch stack to
addVoteDispatchPrecommitMw, which dispatches EnterNewRound(h + 1, 0)
immediately (step 1 above).
ABCI calls per height on the happy path
| Node role | Calls |
|---|---|
| Proposer | PrepareProposal, ProcessProposal, ExtendVote, VerifyVoteExtension × (n − 1 peer precommits) + 1 (commit), FinalizeBlock |
| Other validator | ProcessProposal, ExtendVote, VerifyVoteExtension × (n − 1) + 1, FinalizeBlock |
| Observer (not in the validator set) | ProcessProposal (once a CommitMessage and the block arrive; peers gossip proposals only to validators), VerifyVoteExtension × 1 (commit), FinalizeBlock — observers neither prevote nor precommit, see Non-validators |
There is no ABCI Commit call: the method is removed from the Tenderdash
ABCI protocol (proto/tendermint/abci/types.proto reserves its field
number). Several ProcessProposal calls per height are possible whenever a
later round, a commit for another round, or a block-sync handover requires
re-processing (blockExecutor.ensureProcess).
Transitions and decision rules
This page is the formal core of the specification: for every action of the
state machine it gives the precondition (guard), the actions and the
postcondition, then the decision procedures that choose what to sign.
Dispatch(X) means Controller.Dispatch of event X; a dispatch whose guard
fails is a no-op.
Notation: h = rs.Height, r = rs.Round, step = rs.Step;
Prevotes(x) / Precommits(x) are rs.Votes.Prevotes(x) /
rs.Votes.Precommits(x); maj23(S) is S.TwoThirdsMajority();
any23(S) is S.HasTwoThirdsAny(); complete() is
StateData.isProposalComplete().
Inputs
| Input | Handler | Resulting dispatch |
|---|---|---|
Tock (H, R, NewHeight) | State.handleTimeout | EnterNewRound{H} (round 0) |
Tock (H, R, NewRound) | State.handleTimeout | EnterPropose{H, R} |
Tock (H, R, Propose) | State.handleTimeout | publish TimeoutPropose; EnterPrevote{H, R} |
Tock (H, R, PrevoteWait) | State.handleTimeout | publish TimeoutWait; EnterPrecommit{H, R} |
Tock (H, R, PrecommitWait) | State.handleTimeout | publish TimeoutWait; EnterPrecommit{H, R}; EnterNewRound{H, R + 1} |
TxsAvailable | State.handleTxsAvailable | sets TxsReported; acts only at r == 0 or in a round that may wait after a round skip (mayWaitAfterRoundSkip): at step NewHeight (and h ≠ InitialHeight) schedule tock (h, 0, NewRound) after StartTime − now + 1 ms; at step NewRound dispatch EnterPropose{h, r} |
ProposalMessage | proposalMessageHandler → Proposaler.Set | see Proposal acceptance; if accepted while waitingForTxs, EnterPropose{h, r} |
BlockPartMessage | blockPartMessageHandler | AddProposalBlockPart; on completion ProposalCompleted or AddCommit |
VoteMessage | voteMessageHandler | AddVote → vote-driven dispatch |
CommitMessage | commitMessageHandler | TryAddCommit (Commit handling) |
RoundSkipCertificateMessage | roundSkipCertificateMessageHandler | AddRoundSkipCertificate |
ProposerHeartbeatMessage, RoundSkipVoteMessage | State.handleLivenessMessage | none, or AddRoundSkipCertificate for a certificate the vote completed (Proposer liveness) |
| Proposer-silence report | State.handleProposerSilence | none; signs a round skip vote off the consensus goroutine |
| Signed round skip vote | State.handleSignedRoundSkipVote | none, or AddRoundSkipCertificate for a certificate the vote completed |
Every tock is written to the WAL before handling; handleTimeout drops a
tock when ti.Height ≠ h, ti.Round < r, or ti.Round == r && ti.Step < step
(internal/consensus/state.go).
Timer semantics
timeoutTicker (internal/consensus/ticker.go) owns one timer.
ScheduleTimeout(ti) replaces the pending timer unless ti is older than the
last scheduled (Height, Round, Step): a tick is ignored when its height is
lower, or its round is lower at the same height, or at the same height and
round its step is ≤ the pending step (and the pending step is non-zero).
Consequences:
- a
PrevoteWaittick supersedes a still-pendingProposetick for the same round, so the propose timeout never fires once a prevote wait started; - a tick for the same step as the pending one at the same
(h, r)is dropped even if the pending one already fired, because the ticker remembers the lastti(see Known issues); - durations may be zero or negative; such ticks fire immediately.
The receive routine handles the tock only if it still matches
(h, r, step) as described above.
Step actions
EnterNewRound(H, R)
EnterNewRoundAction.Execute (internal/consensus/state_enter_new_round.go).
- Guard:
h == H ∧ R ≥ r ∧ (R == r ⇒ step == NewHeight). - Actions:
updateRoundStep(R, NewRound)(alsoProposerSelector.UpdateHeightRound(h, R)).- If
R > 0:Proposal = nil,ProposalReceiveTime = 0; ifrs.Commit ≠ nilthenretargetTo(Commit.BlockID)(keep collecting the committed block), elseProposalBlock = nil,ProposalBlockParts = nil.LockedBlock,ValidBlockandCommitare kept. Votes.SetRound(R + 1);TriggeredTimeoutPrecommit = false; save.- Publish
NewRound; if a commit is parked also publish the step and aValidBlockevent so peers keep serving the committed block's parts. waitForTxs = config.WaitForTxs() ∧ (R == 0 ∨ waitsAfterRoundSkip(R)) ∧ h ≠ InitialHeight ∧ ¬ProposeNextBlockImmediately ∧ ¬(roundSkipping ∧ Proposal ≠ nil)whereWaitForTxs = ¬CreateEmptyBlocks ∨ CreateEmptyBlocksInterval > 0(config/config.go);roundSkippingis defined in When it applies andwaitsAfterRoundSkipin The round after a skip.waitForTxs ∧ R > 0 ∧ Commit == nil: publish theNewRoundstep.waitForTxs ∧ CreateEmptyBlocksInterval > 0: schedule tock(h, R, NewRound)after the interval (forR > 0, the interval minusnow − StartTime, at least 0); stay inNewRound.waitForTxs ∧ CreateEmptyBlocksInterval == 0: stay inNewRounduntilTxsAvailable.- otherwise (unless the test-only
DontAutoPropose):Dispatch(EnterPropose{h, R}).
- Postcondition:
r = R,step = NewRound(or beyond, ifEnterProposeran synchronously).
EnterPropose(H, R)
EnterProposeAction.Execute (internal/consensus/state_enter_propose.go).
- Guard:
h == H ∧ R ≥ r ∧ (R == r ⇒ step < Propose). - Actions:
isProposer ← ProposerSelector.GetProposer(h, r) == own pro-tx-hash;applyingCommit ← Commit ≠ nil ∧ holdsProposalBlock(Commit.BlockID).- If
isProposer ∧ ¬applyingCommit ∧ state.LastBlockTime > now: schedule tock(H, R, NewRound)afterLastBlockTime − nowand return without changing step (the guard admits the retry becausestepis stillNewRound). - Schedule tock
(H, R, Propose)afterproposeTimeout(R). - If
¬isProposer, orapplyingCommit, or replay mode, or¬catchup.mayPropose(h): do nothing more. - Else
wal.FlushAndSync()(a failure is only logged);Proposaler.Create(h, R, rs)(happy path §3a). A creation or signing error is logged and the round proceeds as if the proposer were silent. - Deferred:
updateRoundStep(R, Propose), publish step; ifcomplete()thenDispatch(EnterPrevote{H, R}).
- Postcondition:
step = Propose, propose timeout armed (except in the clock-wait case of action 2).
Proposal acceptance (Proposaler.Set)
Proposaler.Set (internal/consensus/state_proposaler.go) runs for every
ProposalMessage (own or peer's). The message must already pass
Proposal.ValidateBasic (types/proposal.go: type, non-negative height and
round, POLRound ≥ -1, complete BlockID, CoreChainLockedHeight ≠ MaxUint32,
signature present).
flowchart TD
A[ProposalMessage p] --> B{rs.Proposal != nil?}
B -- yes --> IGN[ignore]
B -- no --> C{p.Height == h and p.Round == r?}
C -- no --> IGN
C -- yes --> D{parked Commit whose part set is under way and p.BlockID != Commit.BlockID?}
D -- yes --> IGN
D -- no --> E{p.POLRound == -1 or 0 <= p.POLRound < p.Round?}
E -- no --> ERR1[ErrInvalidProposalPOLRound]
E -- yes --> F{p.CoreChainLockedHeight >= state.LastCoreChainLockedBlockHeight?}
F -- no --> ERR2[ErrInvalidProposalCoreHeight]
F -- yes --> G{block already assembled?}
G -- yes --> H{BlockID and CoreChainLockedHeight equal the block's?}
H -- no --> ERR3[ErrInvalidProposalBlockID / ErrInvalidProposalCoreHeight]
H -- yes --> I
G -- no --> I{roundStateRefusal: commit for same h,r names another block? part set under way has another header?}
I -- yes --> ERR4[ErrInvalidProposalForCommit / ErrInvalidProposalForPartSet]
I -- no --> J{proposer public key known?}
J -- no --> K{parked commit for same h,r with same BlockID?}
K -- no --> ERR5[ErrUnableToVerifyProposal]
K -- yes --> OK
J -- yes --> L{verification budget allows one signature check?}
L -- no --> ERR6[ErrVerificationBudgetExhausted - local drop]
L -- yes --> M{proposer.PubKey verifies ProposalBlockSignID?}
M -- no --> ERR7[ErrInvalidProposalSignature]
M -- yes --> OK[rs.Proposal = p; ProposalReceiveTime = enqueue time; part set created if none]
Properties:
- The proposer is
ProposerSelector.GetProposer(h, r); the signature is checked against that validator's key for the committed state'sQuorumType/QuorumHash(types.ProposalBlockSignID). - A proposal that fails is not stored, so a later valid one for the same round can still be accepted; a proposal that fails the signature check is verified again every time it is resent (bounded by the verification budget).
- Errors are returned to the message dispatcher; the sender is not evicted for any of them.
- Equivocating proposals: the first valid one is kept; later ones are ignored
while it occupies
rs.Proposal. The slot is freed within the round only when authenticated votes retarget the round to another block (StateData.retargetTo→dropStaleProposal: current-round polka, precommit-time polka for a block not held, parked or applied commit); a proposal for that block can then be accepted (invariant 5).
AddProposalBlockPart / ProposalCompleted
AddProposalBlockPartAction.Execute, ProposalCompletedAction.Execute
(internal/consensus/state_add_prop_block.go).
- Guard (part):
msg.Height == h;ProposalBlockParts ≠ nil; the peer's invalid-proof budget allows it (blockPartProofBudget). The message's round is not checked: parts of the same block from another round are accepted. - Actions (part):
ProposalBlockParts.AddPart(part)(Merkle proof against the part-set header;ErrPartSetInvalidProofcharges the sender's budget); if the accumulated size exceedsConsensusParams.Block.MaxBytes→ error. When complete: decode the block; ifrs.Proposal ≠ niland the block's hash or part-set header differ from the proposal's →rs.Proposal = nil(the block is kept); otherwise the proposal'sCoreChainLockedHeightandBlockID.StateIDare replaced by the block's;rs.ProposalBlock = block. If a commit is parked →selectParkedCommit(Commit handling). - ProposalCompleted:
- If
maj23(Prevotes(r))is a non-nil block idB,ValidRound < r, and the held block isB→updateValidBlock(). - If
step ≤ Propose ∧ complete()→Dispatch(EnterPrevote{h, r}); then if a polka exists →Dispatch(EnterPrecommit{h, r}). - Else if
step == ApplyCommit→Dispatch(TryFinalizeCommit{h}).
- If
complete() ≡ Proposal ≠ nil ∧ ProposalBlock ≠ nil ∧ (Proposal.POLRound < 0 ∨ Prevotes(POLRound).HasTwoThirdsMajority())(StateData.isProposalComplete).
EnterPrevote(H, R)
EnterPrevoteAction.Execute (internal/consensus/state_enter_prevote.go).
- Guard:
h == H ∧ R ≥ r ∧ (R == r ⇒ step < Prevote). - Actions: run the prevote decision and
sign exactly one prevote (block or nil); deferred
updateRoundStep(R, Prevote). - Postcondition:
step = Prevote; one prevote for(h, R)is on the internal queue (if the node is a validator and signing succeeded).
EnterPrevoteWait(H, R)
EnterPrevoteWaitAction.Execute (internal/consensus/state_enter_wait.go).
- Guard:
h == H ∧ R ≥ r ∧ (R == r ⇒ step < PrevoteWait); panics unlessany23(Prevotes(R)). - Actions: schedule tock
(H, R, PrevoteWait)aftervoteTimeout(R); deferredupdateRoundStep(R, PrevoteWait).
EnterPrecommit(H, R)
EnterPrecommitAction.Execute (internal/consensus/state_enter_precommit.go).
- Guard:
h == H ∧ R ≥ r ∧ (R == r ⇒ step < Precommit). - Actions: run the precommit decision
and sign exactly one precommit; deferred
updateRoundStep(R, Precommit). - Postcondition:
step = Precommit; lock possibly updated (only toR).
EnterPrecommitWait(H, R)
EnterPrecommitWaitAction.Execute (internal/consensus/state_enter_wait.go).
- Guard:
h == H ∧ R ≥ r ∧ (R == r ⇒ ¬TriggeredTimeoutPrecommit); panics unlessany23(Precommits(R)). - Actions: schedule tock
(H, R, PrecommitWait)aftervoteTimeout(R); deferredTriggeredTimeoutPrecommit = true, publish step. - Postcondition:
stepunchanged (see Known issues).
EnterCommit(H, CR)
EnterCommitAction.Execute (internal/consensus/state_enter_commit.go).
- Guard:
h == H ∧ step < ApplyCommit; panics unlessmaj23(Precommits(CR))exists. - Actions:
proposalUpdater.updateStateData(blockID): ifLockedBlockis the committed block, install it asProposalBlock(replaceProposalBlockOnLockedBlock); if the block is not held,retargetTo(blockID)and publishValidBlock(start fetching). Deferred:Step = ApplyCommit(round unchanged),CommitRound = CR,CommitTime = now, publish step,Dispatch(TryFinalizeCommit{h}). - Postcondition:
step = ApplyCommit,CommitRound = CR.
AddRoundSkipCertificate(cert)
AddRoundSkipCertificateAction.Execute (internal/consensus/state_add_round_skip.go).
- Guard:
roundSkipping ∧ cert.Height == h ∧ (RoundSkip == nil ∨ cert.Round > RoundSkip.Round) ∧ step < ApplyCommit(roundSkipCertificateWanted). - Actions: verify the certificate unless this node recovered it (budget,
quorum hash, threshold signature); write its message to the WAL (not on replay);
RoundSkip = cert; broadcast it; ifcert.Round ≥ r,Dispatch(EnterNewRound{h, cert.Round + 1}). - Postcondition:
RoundSkip = cert;r > cert.Round, orrunchanged when the certificate is for an earlier round.
TryFinalizeCommit(H) / ApplyCommit(commit)
See Commit handling.
Vote-driven dispatch
AddVoteAction.Execute (internal/consensus/state_add_vote.go) runs a
middleware chain. Order of execution for a prevote: validate → add to
vote set → addVoteUpdateValidBlockMw → addVoteDispatchPrevoteMw. For a
precommit: validate → addVoteVerifyVoteExtensionMw → add to vote set →
addVoteDispatchPrecommitMw. Errors pass through addVoteErrorMw
(evidence reporting).
Admission (addVoteValidateVoteMw, HeightVoteSet.addVote, VoteSet.addVote)
A vote is admitted to the set iff all hold:
vote.Height == h;rs.Validators.HasPublicKeys;0 ≤ vote.ValidatorIndex < |Validators|.- The round's vote set exists, or (
HeightVoteSet.checkVoteClaimpasses and) the validator has opened fewer than 2 catch-up rounds (peerCatchupRounds); a provisional round is discarded if its first vote is not admitted. vote.ValidatorProTxHashequals the validator's at that index.- Not a duplicate (same block key and identical block signature →
added = false, no error); a different signature for the same block key →ErrVoteNonDeterministicSignature. - The block signature (and, for non-nil precommits, every extension
signature) verifies against the validator's share public key for
(ChainID, QuorumType, QuorumHash)—Vote.Verify, or evidence fromaddVoteVerifyVoteExtensionMwthat this was done (VoteVerification). - For a non-nil precommit from a peer: ABCI
VerifyVoteExtensionreturnedACCEPT(afterblockExecutor.ensureOwnRound).
A vote by a validator who already voted for a different block id in the
same round is recorded as conflicting: NewConflictingVoteError is returned
and the later vote is kept only if a peer claimed +2/3 for its block id
(VoteSet.addVerifiedVote, VoteSet.SetPeerMaj23). addVoteErrorMw
reports the pair to the evidence pool (evidencePool.ReportConflictingVotes)
unless it is the node's own key.
After a prevote is added
addVoteUpdateValidBlockMw, with P = Prevotes(vote.Round) and
B = maj23(P) non-nil:
- if
ValidRound < vote.Round ∧ vote.Round == r: if the held block isBthenupdateValidBlock(); in all casesretargetTo(B)(drops a proposal for another block, replaces the part set unless its header isB's), save, publishValidBlock.
addVoteDispatchPrevoteMw:
| Condition | Dispatch |
|---|---|
r < vote.Round ∧ any23(P) | EnterNewRound{h, vote.Round} (round skip) |
r == vote.Round ∧ step ≥ Prevote ∧ maj23(P) = B ∧ (complete() ∨ B = nil) | EnterPrecommit{h, r} |
r == vote.Round ∧ step ≥ Prevote ∧ any23(P) (otherwise) | EnterPrevoteWait{h, r} |
Proposal ≠ nil ∧ Proposal.POLRound == vote.Round ≥ 0 ∧ complete() (and neither of the above) | EnterPrevote{h, r} (POL votes completed a proposal with POLRound) |
waitingForTxs ∧ (maj23(P) exists ∨ (vote.Round == r ∧ r > 0)) (and none of the above) | EnterPropose{h, r} (ends a wait for transactions; CONSENSUS_VERSION_2) |
After a precommit is added
addVoteDispatchPrecommitMw, with C = Precommits(vote.Round):
| Condition | Dispatch |
|---|---|
¬maj23(C) ∧ r ≤ vote.Round ∧ any23(C) | EnterNewRound{h, vote.Round}; EnterPrecommitWait{h, vote.Round} |
maj23(C) = nil | EnterNewRound{h, vote.Round}; EnterPrecommit{h, vote.Round}; EnterPrecommitWait{h, vote.Round} |
maj23(C) = B ≠ nil | EnterNewRound{h, vote.Round}; EnterPrecommit{h, vote.Round}; EnterCommit{h, vote.Round}; then EnterNewRound{Height: rs.Height} (which is h + 1 if the commit was applied synchronously) |
A precommit majority at a round below the current one is honoured: the
EnterNewRound/EnterPrecommit dispatches are no-ops and EnterCommit runs
with CommitRound = vote.Round (TestCommitFromPreviousRound).
Prevote decision (prevoter.Do)
prevoter.Do (internal/consensus/state_prevoter.go) with
StateData.isValidForPrevote (internal/consensus/state_data.go).
flowchart TD
S([EnterPrevote h,r]) --> A{ProposalBlock held?}
A -- no --> NIL[prevote nil]
A -- yes --> B{Proposal held?}
B -- no --> NIL
B -- yes --> C{Proposal.Timestamp == block.Time?}
C -- no --> NIL
C -- yes --> D{block == ValidBlock?}
D -- yes --> F
D -- no --> E{timely? InitialHeight: timestamp == genesis time; else receive time within window for round r, in replay only too early fails}
E -- no --> NIL
E -- yes --> F{ValidateBlockChainLock ok?}
F -- no --> NIL
F -- yes --> G{POLRound == -1 implies block.ProposerProTxHash == proposer h,r?}
G -- no --> NIL
G -- yes --> H[ensureProcess for round r unless already processed]
H --> V{ValidateBlock against committed state ok?}
V -- no --> NIL
V -- yes --> I{ABCI ProcessProposal: ACCEPT and header matches app result?}
I -- REJECT or mismatch --> NIL
I -- other error --> PANIC[panic]
I -- yes --> J[mustValidate]
J --> K{POLRound == -1?}
K -- yes --> L{LockedRound == -1 or block == LockedBlock?}
L -- yes --> VOTE[prevote block]
L -- no --> NIL
K -- no --> M{maj23 of Prevotes POLRound == block and 0 <= POLRound < r?}
M -- no --> NIL
M -- yes --> N{LockedRound <= POLRound or block == LockedBlock?}
N -- yes --> VOTE
N -- no --> NIL
Rules, in evaluation order:
- P-NoBlock / P-NoProposal. No block or no proposal held → nil
(
errPrevoteProposalBlockNil,errPrevoteProposalNil). This is the path taken ontimeoutPropose. No ABCI call is made. - P-Timestamp.
Proposal.Timestamp ≠ ProposalBlock.Time→ nil. - P-Timely. Skipped when the block is already
ValidBlock. AtInitialHeightthe timestamp must equal the genesis time; otherwiseProposal.CheckTimely(ProposalReceiveTime, SynchronyParams, r)must be 0 (checkTimelyintypes/validation.go):ts − Precision ≤ recv ≤ ts + MessageDelay·2^⌊r/10⌋ + Precision(the shift is capped to avoid overflow). In replay moderecvis the time of the proposal's WAL entry, which can lag its receipt, so onlyrecv < ts − Precision(too early) fails. - P-ChainLock.
sm.ValidateBlockChainLock(state, block)(internal/state/validation.go): with aCoreChainLockin the block,Header.CoreChainLockedHeight == CoreChainLock.CoreBlockHeightand> state.LastCoreChainLockedBlockHeight; without one,Header.CoreChainLockedHeight == state.LastCoreChainLockedBlockHeight. - P-Proposer. For
POLRound == -1the block header'sProposerProTxHashmust be this round's proposer. Re-proposals (POLRound ≥ 0) keep the original header's proposer and are exempt. - P-Process.
blockExecutor.ensureProcess(rs, r): ABCIProcessProposalfor roundrunlessCurrentRoundStatealready is the result for this block and round. The block is first validated against the committed state (BlockExecutor.ValidateBlock: version, chain ID, height,LastBlockID, core chain lock,LastCommit, proposer, evidence, …); a block that fails is never sent to the application and yieldsErrInvalidBlock.ErrBlockRejected(applicationREJECT) orErrInvalidBlock(also a well-formedACCEPTwhose contents disagree with the block: header fields differ from the application's answer,LastCommitinvalid —ValidateBlockWithRoundState) → nil prevote; any other error → panic (prevoter.handleError). A malformed application response is in the panic class, not the nil-prevote class:UNKNOWNstatus, anAppHashof the wrong size (ResponseProcessProposal.Validate), ortx_resultsnot matching the transactions (validateExecTxResults) are returned byBlockExecutor.ProcessProposalas plain errors before theErrInvalidBlockwrapper is reached. Note thatProcessProposalruns before the lock rules, so a locked validator still processes a different proposal and then prevotes nil. - P-Lock-NoPOL (paper line 22–24).
POLRound == -1: prevote the block iffLockedRound == -1 ∨ ProposalBlock.BlockID == LockedBlock.BlockID(prevoter.checkProposalBlock). - P-Lock-POL (paper line 28–30).
POLRound ≥ 0: prevote the block iffmaj23(Prevotes(POLRound)) == ProposalBlock.BlockID ∧ 0 ≤ POLRound < r ∧ (LockedRound ≤ POLRound ∨ ProposalBlock.BlockID == LockedBlock.BlockID)(prevoter.checkPrevoteMaj23). Block ids compare all three fields, includingStateID. - Otherwise prevote nil.
EnterPrevote is reached either because the proposal is complete or because
timeoutPropose fired. A proposal with POLRound ≥ 0 whose POL prevotes have
not arrived is not complete, so the node stays in Propose until those
prevotes arrive (addVoteDispatchPrevoteMw, last row) or the timeout fires;
in the latter case the block and proposal are held, rules 1–6 can pass, and
rule 8 yields nil because maj23(Prevotes(POLRound)) is absent.
Precommit decision (EnterPrecommitAction.Execute)
flowchart TD
S([EnterPrecommit h,r]) --> A{polka in Prevotes r?}
A -- no --> NIL[precommit nil, keep lock]
A -- yes --> B{polka is nil?}
B -- yes --> NIL
B -- no --> C{Proposal and ProposalBlock held?}
C -- no --> NIL
C -- yes --> D{Proposal.Timestamp == block.Time?}
D -- no --> NIL
D -- yes --> E{held block == polka block B?}
E -- no --> F{LockedBlock == B?}
F -- yes --> NIL2[precommit nil, keep lock]
F -- no --> RT[retargetTo B: fetch B; precommit nil]
E -- yes --> G[mustEnsureProcess r; mustValidate]
G --> H{LockedBlock == B?}
H -- yes --> RL[LockedRound = r; relock event; precommit B]
H -- no --> LK[lock B at r; lock event; precommit B; updateValidBlock]
Rules, in evaluation order:
- C-NoPolka.
¬maj23(Prevotes(r))→ precommit nil. Reached from thePrevoteWait/PrecommitWaittocks. The lock is unchanged. - C-POLInfo.
HeightVoteSet.POLInfo()returns the highest round with a polka; if it is< rthe node panics (cannot happen after rule 1). - C-NilPolka. Polka for nil → precommit nil.
- C-NoProposal.
Proposal == nil ∨ ProposalBlock == nil→ nil. No fetch is started here (the polka middleware already retargeted toB). - C-Timestamp.
Proposal.Timestamp ≠ ProposalBlock.Time→ nil. - C-LockedNotProposed. Polka block
BequalsLockedBlockbut this round's proposal block is different → nil, lock kept. Rationale in the code: the precommit's extensions are produced byExtendVotefor the block the application processed last at this round; a block locked earlier was processed for an earlier round, and an honest proposer would have re-proposedB. Consequence: a locked validator that misses the re-proposal of its locked block cannot relock in that round (TestStateLock_MissingProposalWhenPOLSeenDoesNotUpdateLock). - C-Lock / C-Relock.
Bis this round's proposal block:mustEnsureProcess(rs, r)(re-runsProcessProposalif the block was processed for another round; panics on failure),mustValidate; if already locked onB→LockedRound = r,Relockevent, precommitB; elseupdateLockedBlock()(LockedRound = r,LockedBlock = B),Lockevent, precommitB,updateValidBlock()(+ValidBlockevent if it changed). - C-Unknown. Polka for a block the node does not hold and is not locked
on →
retargetTo(B)(start collecting its parts; a proposal for another block is dropped) and precommit nil.
A non-nil precommit triggers ABCI ExtendVote inside voteSigner.signVote;
a nil precommit carries no extensions (Vote.ValidateBasic rejects
extensions on prevotes and nil precommits).
Proposal-validity summary
A block becomes prevotable when all of the following hold; the function that checks each is given.
| Check | Where |
|---|---|
Proposal signed by proposer(h, r) | Proposaler.verifyProposal |
POLRound ∈ {-1} ∪ [0, r) | Proposaler.Set |
Proposal BlockID hash and part-set header equal the assembled block's | AddProposalBlockPartAction.addProposalBlockPart |
Parts' Merkle proofs valid; total size ≤ Block.MaxBytes | PartSet.AddPart, addProposalBlockPart |
Proposal.Timestamp == Header.Time | isValidForPrevote, EnterPrecommitAction.Execute |
Timely (PBTS window; in replay, not too early), unless ValidBlock | StateData.proposalIsTimely |
| Core chain lock consistent with state | sm.ValidateBlockChainLock, validateBlock |
Header.ProposerProTxHash == proposer(h, r) when POLRound == -1 | isValidForPrevote |
Block.ValidateBasic; version, chain id, height, LastBlockID, ConsensusHash, ValidatorsHash, LastCommit threshold signature, proposer is a validator, Time > LastBlockTime (≥ genesis at initial height), evidence size | validateBlock in internal/state/validation.go |
ABCI ProcessProposal = ACCEPT; AppHash, ResultsHash, NextValidatorsHash, NextConsensusHash equal the application's; app hash size | BlockExecutor.ProcessProposal, ValidateBlockWithRoundState |
| Lock rules P-Lock-NoPOL / P-Lock-POL | prevoter.shouldVoteBeSent |
Alternative flows and edge cases
Each section states the situation, the behaviour the code implements, and the implementing function. Commit-related flows are in Commit handling; crash recovery and sync handover in Recovery and synchronisation.
Proposal missing, late or invalid
Proposer silent or slow
timeoutPropose (proposeTimeout(r)) fires → EnterPrevote.
isValidForPrevote fails with errPrevoteProposalBlockNil → nil prevote,
no ABCI call (prevoter.Do). When every validator prevotes nil, the +2/3
nil majority dispatches EnterPrecommit directly — no PrevoteWait, no
voteTimeout(r) (addVoteDispatchPrevoteMw: TwoThirdsMajority() returns
the nil block id, and blockID.IsNil() satisfies the precommit condition) →
nil precommit → +2/3 nil precommits → EnterPrecommitWait → voteTimeout(r)
→ EnterNewRound(r + 1). EnterPrevoteWait is reached only when the
prevotes hold +2/3-any without a majority (some validators did receive
and prevote the block), in which case voteTimeout(r) precedes
EnterPrecommit.
sequenceDiagram
participant T as timeoutTicker
participant V as Validator
participant N as Other validators
T->>V: tock Propose(h,r)
V->>V: EnterPrevote: no block → prevote nil
V-->>N: prevote nil
N-->>V: prevotes nil
V->>V: +2/3 nil prevotes → EnterPrecommit: polka nil → precommit nil
N-->>V: precommits nil
V->>V: maj23 nil → EnterPrecommitWait (voteTimeout r)
T->>V: tock PrecommitWait(h,r)
V->>V: EnterPrecommit (no-op) → EnterNewRound(h,r+1) → EnterPropose(h,r+1)
A proposer that is silent while its round waits for transactions is handled
before the propose step on CONSENSUS_VERSION_2 chains: validators that stop
hearing its heartbeats skip the round with a round skip certificate
(Proposer liveness and round skipping).
Proposal arrives after the node prevoted
Proposaler.Set has no step check, so the proposal is still stored and parts
are still collected. ProposalCompletedAction.Execute enters prevote only
while step ≤ Propose, so the node does not prevote it. The block can still:
- become
ValidBlockif a polka for it exists or forms (ProposalCompletedAction.Execute,addVoteUpdateValidBlockMw); - be precommitted if a polka for it forms while the node is in
Prevote(addVoteDispatchPrevoteMw→EnterPrecommit; rule C-Lock); - be re-proposed in a later round as
ValidBlock.
Proposal for another height or round
Ignored silently (Proposaler.Set). Block parts for another round of the
current height are accepted into the current part set when their proofs match
its header ("blocks might be reused", addProposalBlockPart); parts for
another height are ignored; parts when no part set is expected are ignored.
blockPartMessageHandler swallows errors for parts whose round differs from
the current one.
Invalid POLRound, bad signature, wrong proposer
ErrInvalidProposalPOLRound, ErrInvalidProposalSignature (counted in
ProposalVerifyFailures), see
Proposal acceptance.
A proposal signed by a validator other than proposer(h, r) fails the
signature check because the signature is verified against the expected
proposer's key only. The round proceeds as if no proposal had been received.
Proposal contradicts the round's target block
Refused with ErrInvalidProposalForCommit (a verified commit for the same
(h, r) names another block), ErrInvalidProposalForPartSet (a part set is
already being collected under another header), or ErrInvalidProposalBlockID
/ ErrInvalidProposalCoreHeight (a block is already assembled and differs)
(roundStateRefusal, Proposaler.Set). Rationale: a part-set header is a
Merkle root over exactly one block's bytes, so a disagreeing proposal could
never be acted on and would make the node reject the parts it is waiting for.
Block id mismatch discovered when the block completes
If the assembled block's hash or part-set header differ from the proposal's,
the proposal is dropped (rs.Proposal = nil) but the block is kept
(addProposalBlockPart). The node later prevotes nil
(errPrevoteProposalNil). The fields not covered by the proposal signature
(CoreChainLockedHeight, BlockID.StateID) are instead taken from the
authenticated block bytes.
Oversized or malformed block
Accumulated parts above ConsensusParams.Block.MaxBytes → error; an
undecodable block → error (addProposalBlockPart). Parts with invalid Merkle
proofs are charged to the sender's blockPartProofBudget and dropped
silently once it is exhausted (internal/consensus/block_part_budget.go).
Block is invalid, application rejects it, or the header does not match
BlockExecutor.ProcessProposal first validates the block against the
committed state (ValidateBlock); a block that fails is never sent to the
application. That failure (sm.ErrInvalidBlock), the application returning
REJECT (sm.ErrBlockRejected), or a well-formed ACCEPT whose contents
disagree with the block — header mismatch / invalid LastCommit
(sm.ErrInvalidBlock) — → nil prevote (prevoter.handleError). A failed ProcessProposal clears
CurrentRoundState, so no round counts as processed until a later call
succeeds (blockExecutor.ensureProcess). Any other error panics
the node: transport failure, UNKNOWN status, or a malformed response — an
AppHash of the wrong size fails ResponseProcessProposal.Validate and
tx_results not matching the transactions fails validateExecTxResults,
both returned by BlockExecutor.ProcessProposal as plain errors, not as
ErrInvalidBlock. The same block
reaching EnterPrecommit via a polka would call mustEnsureProcess, which
panics on failure — a node that could not process a block cannot have a
polka-matching proposal block for it unless the application is
non-deterministic.
During WAL replay a mismatch between a stored block's AppHash and the
application's result stops the node (checkAppHashEqualsOneFromBlock in
internal/consensus/replayer.go).
Proposal timestamp differs from block time
Nil prevote (errPrevoteTimestampNotEqual) and nil precommit
(rule C-Timestamp).
Locking
The lock is defined by LockedRound/LockedBlock and the rules
P-Lock-NoPOL, P-Lock-POL
and C-Lock, C-Relock, C-LockedNotProposed.
| Situation | Behaviour | Evidence |
|---|---|---|
Locked on B at r; round r + k proposes B with POLRound ≥ LockedRound or without POL | Prevote B (lock match) | prevoter.checkProposalBlock, checkPrevoteMaj23; TestStateLock_POLRelock |
Locked on B; round r + k proposes C ≠ B with POLRound = -1 | Prevote nil (ProcessProposal still runs for C) | TestStateLock_PrevoteNilWhenLockedAndDifferentProposal |
Locked on B; proposal C with POLRound = p ≥ LockedRound and a polka for C at p | Prevote C; if a polka for C forms this round → lock moves to C (updateLockedBlock) | TestStateLock_POLUpdateLock |
Locked on B; proposal C with POLRound = p < LockedRound | Prevote nil | checkPrevoteMaj23 |
| Polka for nil in a later round | Lock unchanged — no unlock | TestStateLock_POLDoesNotUnlock |
Polka for B (the locked block) in round r + k but no proposal for B in that round | Precommit nil, lock unchanged | rule C-LockedNotProposed; TestStateLock_MissingProposalWhenPOLSeenDoesNotUpdateLock |
Polka for B in round r + k and proposal for B | Relock: LockedRound = r + k, precommit B after re-processing for r + k | TestStateLock_POLRelock |
Proposal for an old block (POLRound pointing at a polka for another block) | Not prevoted, not locked | TestStateLock_DoesNotLockOnOldProposal |
Liveness consequence of C-LockedNotProposed: when a round's proposer is slow
or faulty, validators locked on B do not relock even if the other
validators prevote B; the network must wait for a later proposer to
re-propose B with POLRound set.
Valid-block bookkeeping
StateData.updateValidBlock sets ValidRound = Round, and when the block
differs from the current ValidBlock, ValidBlock = ProposalBlock,
ValidBlockParts, and ValidBlockRecvTime = ProposalReceiveTime when the
proposal names this block, else now. It is called from:
addVoteUpdateValidBlockMw: a prevote completes a polka for non-nilBin the current round withValidRound < Round, and the node holdsB. Whether or not it holdsB,retargetTo(B, polka)then drops a proposal for another block, replaces the part set unless its header isB's, and aValidBlockevent is published so peers sendB's parts (NewValidBlockMessage).ProposalCompletedAction.Execute: the block completes while a polka for it already exists in the current round.EnterPrecommitAction.Execute: after a first lock.
ValidRound/ValidBlock are never reset within a height and survive
EnterNewRound.
Re-proposal. Proposaler.Create reuses ValidBlock iff
checkValidBlock: at InitialHeight its time equals the genesis time;
otherwise ValidBlock.IsTimely(ValidBlockRecvTime, SynchronyParams, ValidRound).
A re-proposal keeps the original header (proposer, time) and carries
POLRound = ValidRound. Receivers skip the timeliness check for a block that
is their own ValidBlock (isValidForPrevote) and skip the proposer check
when POLRound ≥ 0. When ValidBlock is not timely a fresh block is
built and proposed with POLRound = -1, so receivers judge it like any new
proposal (timeliness, proposer check, rule P-Lock-NoPOL). A validator locked
on another block prevotes nil for it; see
Known issues.
Round skipping and nil rounds
- Future-round prevotes.
+2/3-any prevotes atvote.Round > r→EnterNewRound(vote.Round)(addVoteDispatchPrevoteMw). The node skips the intermediate rounds without voting in them. - Future-round precommits.
+2/3-any precommits without a majority atvote.Round ≥ r→EnterNewRound(vote.Round)+EnterPrecommitWait(addVoteDispatchPrecommitMw); the node then waitsvoteTimeoutbeforer + 1. A precommit majority for a block at any round (including a round below the current one) →EnterCommitwith that round (TestCommitFromPreviousRound). - Future-round commit. If its block is already held, a verified
CommitMessageforcommit.Round > ris checked against the block and application before advancing the round; rejection does not move the round. If the block is absent, signature verification can park the commit and advance to its round before the later application check. See Commit handling. - Round skip certificate (
CONSENSUS_VERSION_2). A verified certificate for roundr' ≥ r→EnterNewRound(r' + 1), unless the node is atApplyCommit; one forr' < ris only kept and gossiped (AddRoundSkipCertificateAction). See Proposer liveness and round skipping. - Round bookkeeping.
HeightVoteSetkeeps vote sets for rounds0 .. Round + 1(SetRound(round + 1)inEnterNewRoundAction) plus at most two "catch-up" rounds per validator pro-tx-hash (HeightVoteSet.addVote,ErrGotVoteFromUnwantedRound). Vote sets are never released within a height. - All-nil round. Nil polka → nil precommit → nil precommit majority →
EnterPrecommitWait— the node still waitsvoteTimeout(r)even though the round is decided — →EnterNewRound(r + 1). - Idempotence.
TriggeredTimeoutPrecommitmakesEnterPrecommitWaitrun once per round; the step guards do the same for the other actions. - What survives a round change.
LockedBlock,ValidBlock, parkedCommit, and all vote sets;Proposaland the proposal block/parts are cleared unless a commit is parked (EnterNewRoundAction.Execute).
Equivocation, duplicates and conflicting votes
Implemented in VoteSet.addVote / addVerifiedVote (types/vote_set.go)
and addVoteErrorMw (internal/consensus/state_add_vote.go).
- Identical vote (same validator, block key and block signature): not added, no error.
- Same validator and block key, different signature:
ErrVoteNonDeterministicSignature(BLS signatures are deterministic; logged at error level). - Same validator, different block id: the second vote is recorded only in
votesByBlockwhen some peer claimed +2/3 for that block id (peerMaj23, set byVoteSetMaj23Message→VoteSet.SetPeerMaj23); it replaces the first invotesonly if it matchesmaj23. The pair is returned asErrVoteConflictingVotesand reported to the evidence pool (evidencePool.ReportConflictingVotes) unless the key is the node's own (logged: "did you unsafe_reset a validator?") or no private validator is set (ErrPrivValidatorNotSet). The evidence pool stores it unverified; it reaches the application asMisbehavioronly if a proposer includes it (evpool.PendingEvidenceinCreateProposalBlock). - Conflicting proposals by the same proposer: the first valid one is
kept and the second ignored while the first occupies
rs.Proposal; a current-round polka or a commit for the second block drops the first (StateData.retargetTo→dropStaleProposal) and the second is then accepted (invariant 5). No evidence is produced.
Vote extensions
- The node normally asks ABCI
ExtendVotewhen signing a non-nil precommit (voteSigner.signVote). The one exception signs nothing: whenVerifyVoteExtensionrejects a commit's extensions, no expectation is known for its block and round, and the block is processed forcommit.Round,blockExecutor.verifyCommitExtensionscallsExtendVotefor that block and round to learn the expected vector. This can happen on any node, validator or not (Parked commit). On both paths the response must parse (NewVoteExtensionsFromABCIExtended); otherwise the node panics. - Supported extension types (
types/vote_extension.go):THRESHOLD_RECOVERandTHRESHOLD_RECOVER_RAW(threshold-recoverable;RAWmay carry an application-supplied sign request id that binds no height or round).DEFAULTexists in the enum but is not supported:NewVoteExtensionsFromABCIExtendedrejects it in anExtendVoteresponse (ErrUnsupportedVoteExtensionType), andVoteExtension.Validate(proto/tendermint/types/dash.go) rejects it in a peer's vote (Vote.ValidateBasic). A prevote or nil precommit carrying extensions failsVote.ValidateBasic. - Each peer precommit's extension signatures are verified after the block
signature (
types.VerifyVoteSignatures), then ABCIVerifyVoteExtensionis called with the validator's pro-tx-hash. A rejected vote is not stored (addVoteVerifyVoteExtensionMw). Own votes skip the ABCI call. - At most
MaxVoteExtensions = 32extensions are accepted on verification paths (types/quorum_sign_data.go); the signing path is not capped. - The commit carries only the threshold-recovered extensions of the
canonical group (Threshold recovery).
Before a block is saved the application sees them once more as a single
VerifyVoteExtensionwith an emptyValidatorProTxHash(sm.VerifyCommitExtensions).
Proposal timestamps and time rules
- The proposer sets
Header.Time = now(genesis time atInitialHeight,State.makeBlockTime). Before proposing, a proposer whose clock is behindstate.LastBlockTimewaits for the difference (EnterProposeAction.Execute,proposerWaitTime). - Validity:
Time > state.LastBlockTimeforh > InitialHeight,Time ≥ genesisatInitialHeight(validateBlock). - Timeliness is checked only at prevote time, against the proposal's
receive time (
ProposalReceiveTime= enqueue time of theProposalMessage,chanMsgSender.send), with the window ofcheckTimely(rule P-Timely). It is skipped for a block that is the node'sValidBlock, so re-proposals of an already polka'd block are not re-judged. In replay mode the receive time is the time of the proposal's WAL entry, and only a proposal received too early fails. The entry is never earlier than the receipt, so replay refuses only what the node refused when it received the proposal, not always all of it. That time also becomesValidBlockRecvTimefor a block made valid during replay, which decides whether the node re-proposes it after a restart (Proposaler.checkValidBlock), and feedsProposalTimestampDifference. - Timeliness governs re-proposal (
Proposaler.checkValidBlock). - Metric
ProposalTimestampDifference{is_timely}is recorded for proposals withPOLRound = -1(Proposaler.proposalTimestampDifferenceMetric).
Mempool and empty blocks
EnterNewRoundAction.Execute waits for transactions when
WaitForTxs() = ¬CreateEmptyBlocks ∨ CreateEmptyBlocksInterval > 0, at
round 0 or in a round entered on a round skip certificate that satisfies
StateData.waitsAfterRoundSkip
(The round after a skip),
above InitialHeight, unless ProposeNextBlockImmediately
(set from ResponseFinalizeBlock.propose_next_block_immediately of the
previous height by ApplyCommitAction.Execute; consumed by the next
updateToState; not persisted, so a restart falls back to the configured
wait).
State.handleTxsAvailable records the report in TxsReported (the mempool
reports once per height) and acts only at round 0 or in a round that may wait
after a round skip (StateData.mayWaitAfterRoundSkip): at step NewHeight
(and not at InitialHeight) it schedules a NewRound tock for
StartTime − now + 1 ms; at step NewRound it dispatches EnterPropose.
With CreateEmptyBlocksInterval > 0 the interval tock calls EnterPropose
even without transactions (empty block); in a round after a skip the interval
counts from StartTime. Other rounds ≥ 1 never wait. On
CONSENSUS_VERSION_2 chains a proposal or prevotes can also end the wait
(When it applies).
Validator-set and parameter changes
- Changes come from the application's
ValidatorSetUpdatein thePrepareProposal/ProcessProposalresponse and are reflected in the header'sNextValidatorsHash, which every validator checks against its own application's answer (ValidateBlockWithRoundState). With an unchangedQuorumHashthe update is applied as a change set; with a newQuorumHashthe new set replaces the old one (CurrentRoundState.valsetUpdateininternal/state/current_round_state.go). State.UpdatemakesValidators ← NextValidators,LastValidators ← Validators; the new set votes from heighth + 1(updateToStatebuilds the nextHeightVoteSetfrom it and logs "Updating validators" when the quorum hash or threshold changes).- Consensus parameters (timeouts, synchrony, block limits, proposer-selection
version) proposed in block
htake effect ath + 1(LastHeightConsensusParamsChanged). - Proposer selection version is
ConsensusParams.Version.ConsensusVersion(Proposer selection); version 2 also enables proposer liveness.
Core chain locks
- Header fields:
CoreChainLockedHeight; optionalCoreChainLockin the block (andCoreChainLockUpdatein the proposal, not signed). ValidateBlockChainLock(prevote) andvalidateCoreChainLock(validateBlock, before the application call): with a lock, the header height equals the lock's and exceedsstate.LastCoreChainLockedBlockHeight; without one, it equals it.Proposaler.Setrejectsproposal.CoreChainLockedHeightbelow the committed height; after the block completes the proposal's value is overwritten from the block.state.LastCoreChainLockedBlockHeightadvances only when the block carries aCoreChainLock(CurrentRoundState.UpdateState).
Non-validators and signer problems
- Observers (node not in
Validators): the reactor drops incomingVotemessages (Reactor.handleVoteMessage); the consensus state still runsEnter*actions on timeouts, butvoteSigner.signAddVotereturns without signing (!Validators.HasProTxHash(own)). Observers progress to the next height only throughCommitMessages and block parts (Commit handling). Peers gossip proposals, votes andVoteSetMaj23only to validator peers, and block parts to non-validators once they announce a commit (gossip_handlers.go). - No private validator (
privValidator.IsZero()): no votes (ErrPrivValidatorNotSetlogged);blockExecutor.createfails with "cannot create proposal block on non-validator", so a proposer without a key is silent and the round times out. - Validator set without public keys (
HasPublicKeys = false):HeightVoteSet.addRoundcreates no vote sets andaddVotedrops every vote; proposals can only be accepted when a verified commit for the same(h, r)names the same block (Proposaler.verifyProposalForNonValidatorSet). - Signing failure (
PrivValidator.SignVote/SignProposalerror or deadline): logged; no vote/proposal and no retry within the step (voteSigner.signAddVote,Proposaler.Create). A failed round skip vote is retried after another silence timeout. - Remote signers (socket, gRPC) cannot sign proposer heartbeats or round
skip votes (
RemoteSignerError501,types.ErrSigningNotSupported): a validator using one sends neither, so its waiting rounds are skipped (Signers). - Observers and liveness messages: a node without the validators' key shares neither verifies nor relays heartbeats and skip votes; it verifies and applies round skip certificates against the quorum public key.
- Block sync just finished: proposing is suppressed until the node has
passed the handover target height or 10 minutes elapsed
(
catchupTracker.mayPropose,maxCatchupSuppression); voting is not. - WAL replay: the node never proposes (
EnterProposeAction.Execute,replayMode).
Panics that halt the node
State.receiveRoutine re-panics any panic from a handler except
autofile.ErrAutoFileClosed and panics during context cancellation, so the
following stop the node (liveness-relevant for operators):
| Trigger | Function |
|---|---|
WAL WriteSync failure for own messages or EndHeightMessage | walMiddleware, ApplyCommitAction.Execute |
PrepareProposal error | blockExecutor.create |
ProcessProposal unknown error at prevote; any failure in mustEnsureProcess (precommit, commit, apply) | prevoter.handleError, blockExecutor.mustEnsureProcess |
Validation failure after a successful ProcessProposal | blockExecutor.mustValidate |
ExtendVote / VerifyVoteExtension transport error or invalid extension | BlockExecutor.ExtendVote, BlockExecutor.VerifyVoteExtension |
FinalizeBlock failure | ApplyCommitAction.Execute |
| ABCI client stopped during commit verification | verifyProcessedCommit |
Internal invariant violations: EnterPrevoteWait/EnterPrecommitWait without +2/3-any, EnterCommit/finalizeCommit without a majority or without the block, POLInfo round below the current, updateToState height mismatch, non-monotonic HeightVoteSet.SetRound, proposer-selector construction failure | respective Execute functions, StateData.updateToState, HeightVoteSet.SetRound |
Errors from message handlers invoked by the reactor are recovered and
logged (Reactor.handleMessage), not panicked.
Rate limits and budgets that change protocol timing
These mechanisms never change what a node signs, but they decide whether a message is processed at all (dropped messages are local decisions, not peer faults, unless stated). For operational monitoring, see DoS observability.
- Reactor admission (
internal/consensus/reactor.go,message_cost.go): per-peer token buckets on the vote channel (cost 1 per vote plus one per extension for non-nil precommits; 1 + extensions per commit), the data channel (proposals weigh more than parts) and the state channel;VoteSetMaj23answers are deduplicated (PeerState.ShouldAnswerVoteSetMaj23). Over budget → silent drop. - Peer lanes (
internal/consensus/peer_lanes.go): one FIFO per peer (laneCapacity = 512messages, 64 MiB aggregate), served round-robin by verification cost. The node's own messages bypass lanes via the internal queue (chanMsgSender.send). - Verification budget (
internal/consensus/verification_budget.go,config.VerificationRateLimit): a global BLS-verification rate; a message waits until its full cost is affordable, andverificationBudgetMiddlewaredrops what cannot be afforded. It is the outermost middleware so a dropped message is not written to the WAL.ErrVerificationBudgetExhaustednever evicts a peer (isPeerFloodableError). - Block-part proof budget (
block_part_budget.go): per-peer allowance for parts whose Merkle proof fails. - Liveness messages (
CONSENSUS_VERSION_2): heartbeats, round skip votes and certificates pay extra state-channel tokens only when new, go through the peer lanes and the verification budget, and are sent through a bounded reactor outbox; heartbeats are paced per round (Message admission and budgets). - Bounded memory per height: commit candidate slots (one per peer),
commitExpectations,MaxVoteExtensions,HeightVoteSetcatch-up rounds,maxPeerMaj23sclaims per vote set.
Under load a precommit carrying many extensions can wait in a lane for seconds, which is within the propose timeout but may exceed the vote timeout.
Proposer liveness and round skipping
A round that waits for transactions has no timeout running unless
create-empty-blocks-interval is set, so a proposer that is offline while the
chain is idle is only noticed once a transaction arrives, and that transaction
then waits out the propose timeout. From CONSENSUS_VERSION_2 the proposer of
a waiting round announces itself with signed heartbeats, and validators that
stop hearing them hand the round to the next proposer with a threshold-signed
round skip certificate.
Wire formats are in Consensus channel messages; sign bytes and request IDs in Signing.
When it applies
roundSkipping (internal/consensus/proposer_liveness.go) gates everything on
this page. It holds when both:
ConsensusParams.Version.ConsensusVersion ≥ 2(VersionParams.ProposerLivenessEnabled,types/params.go), andtypes.RoundSkipsProvable(Validators):T ≤ R, i.e. aVotingPowerThresholdabove the recovery threshold disables round skipping, because a certificate only proves that shares worthRsigned (types/round_skip.go).
Otherwise no heartbeat is sent or accepted, no skip vote is signed or collected, every certificate is dropped, and the node behaves as on version 0 or 1.
A node waits for transactions in its round (waitingForTxs) while
roundSkipping ∧ step == NewRound ∧ Commit == nil; liveness is tracked while
it also holds no proposal for the round (waitingRound). Whether a round waits
at all is decided by EnterNewRoundAction.Execute
(Mempool and empty blocks).
On a version-2 chain a wait also ends, with Dispatch(EnterPropose{h, r}),
when:
- a proposal for the round is accepted (
proposalMessageHandler,internal/consensus/msg_handlers.go); a round-0 proposal accepted before the round starts keeps the round from waiting at all; - a prevote is added while the node waits and the added prevote's round (this
round or an earlier one) has a polka, or the prevote is for the current round
and
r > 0(addVoteDispatchPrevoteMw).
A proposal or polka shows that a block is under way; from then on the propose timeout, not the heartbeats, moves the round on.
Heartbeats
State.followLiveness (internal/consensus/state.go) runs once before the
receive routine's first input and after every input. It calls
proposerLiveness.reconcile, which, while waitingRound holds, either
announces (this node is proposer(h, r)) or watches (any other node);
otherwise it stops both.
Sending (proposerLiveness.sendHeartbeats, sendHeartbeat):
- one heartbeat at once and then every
ProposerHeartbeat, until the node leaves the round; - only while the node could propose (
State.proposerReady): the propose step would not suppress it after a block-sync handover (catchupTracker.mayPropose) and the application answers ABCIEchoon every backend that receivesPrepareProposal(BlockExecutor.AppReadyininternal/state/execution.go,abciclient.EchoRoute) within the interval. At most one probe is in flight (appReadiness,internal/consensus/app_readiness.go); Sequence = max(last + 1, signing time in Unix ms)(heartbeatSequence);- one heartbeat is signed at a time, whatever its round; a heartbeat whose
round the node left while signing is dropped. A signer error wrapping
types.ErrSigningNotSupportedstops heartbeats for the rest of the process (signerUnsupported).
Receiving: the reactor screens a heartbeat without a signature check
(proposerLiveness.wanted → screen) and queues it in the sender's lane; the
consensus goroutine verifies it (proposerLiveness.heard, called from
State.handleLivenessMessage). A heartbeat is accepted iff:
- its height is the node's and its round is within the relay window
[max(r − 4, 0), r + 4](relayWindow,maxRoundSkipRoundsAhead); - it names
proposer(h, round), otherwiseerrProposerHeartbeatWrongProposer; - it was not seen before, identified by
(height, round, sequence, timestamp)(heartbeatID; at mostmaxSeenHeartbeats = 256remembered); - its timestamp is within the silence timeout of local time, in either direction;
- the node holds the proposer's public key share (validators only);
- the round's pace has a token: one per half heartbeat interval, burst 2
(
heartbeatPaceBurst); - the verification budget allows one check and
ProposerHeartbeat.Verifysucceeds (types/proposer_heartbeat.go).
An accepted heartbeat for the round the node watches re-arms its silence timer;
one for a nearby round is only relayed. Every accepted heartbeat is relayed
once. Heartbeats go only to validator peers at the same height whose round is
within the relay window of the heartbeat's round, never back to the sender
(Reactor.sendHeartbeat, wantsHeartbeat in internal/consensus/reactor.go).
A heartbeat changes no consensus state and is not written to the WAL.
Silence and skip votes
The watch arms a timer of ProposerSilenceTimeout(r) = ProposerSilence + ProposerHeartbeat · r, saturating at the maximum duration
(TimeoutParams.ProposerSilenceTimeout, types/params.go). When it fires,
State.handleProposerSilence acts only if the report is still current (same
height and round, still waiting, no heartbeat re-armed the timer since) and
the node is a validator. ProposerSilences counts the first report of a round.
State.voteToSkipRound then:
- if this validator's vote for the round is already counted, broadcasts it again;
- otherwise has
RoundSkipVote{h, r, own pro-tx-hash, own index}signed off the consensus goroutine (roundSkipSigner,internal/consensus/round_skip_signer.go), one vote at a time, with the silence timeout as the signing deadline. If a vote is still being signed, this one is retried after another silence timeout.
State.handleSignedRoundSkipVote drops the result if the node left the round
or a heartbeat arrived meanwhile. ErrSigningNotSupported ends the attempt;
any other signing error retries after another silence timeout. A signed vote
is counted, broadcast, and broadcast again every silence timeout while the
round stays silent.
A skip vote is not a prevote: it locks nothing, and the validator still
prevotes and precommits the round's proposal if one arrives. Every validator
signs the same bytes for (h, r), so a validator can make only one skip vote
per round.
Collecting votes and recovering the certificate
roundSkipCollector (internal/consensus/round_skip_collector.go) holds the
skip votes of the current height and quorum for rounds [r, r + 4]. Votes of
rounds below r are dropped when the node moves on. A vote from a peer is
counted iff its index and pro-tx-hash name the same validator (otherwise
errRoundSkipVoteWrongValidator), it is not already counted, the round is not
already certified, the node holds the validator's key share, the verification
budget allows one check, and RoundSkipVote.Verify succeeds.
A vote is relayed when first counted. A copy of a counted vote that arrives at
least half a silence timeout after the last relay has the counted vote relayed
again, until the round is certified (roundSkipCollector.relayAgain). Skip
votes are not written to the WAL.
Once the counted power reaches
RoundSkipThresholdPower = max(T, R) (types/round_skip.go; equal to R
whenever roundSkipping holds; +2/3 on mainnet, whose LLMQ_100_67 quorums
recover from 67 of 100 shares and set no VotingPowerThreshold),
RecoverRoundSkipCertificate recovers the
quorum threshold signature over the round's sign bytes from the shares and
verifies it against ValidatorSet.ThresholdPublicKey. A failed recovery is
retried at once with any votes counted meanwhile, otherwise with the next vote.
The recovered certificate is dispatched immediately on the consensus goroutine
(State.applyRecoveredCertificate).
Applying a certificate
AddRoundSkipCertificateAction.Execute
(internal/consensus/state_add_round_skip.go):
- Guard (
roundSkipCertificateWanted, also checked by the reactor before queuing):roundSkipping ∧ cert.Height == h ∧ (RoundSkip == nil ∨ cert.Round > RoundSkip.Round) ∧ step < ApplyCommit. - Actions:
- Unless this node recovered it: charge the verification budget and
RoundSkipCertificate.Verify(quorum hash must equal the validator set's, elseErrRoundSkipCertificateWrongQuorum; threshold signature againstThresholdPublicKey). - Write the message it came in to the WAL (
Write, no fsync); not on replay. RoundSkip = cert; broadcast it.- If
cert.Round < r: stop (the certificate is kept for peers still in that round). Else countRoundSkipsandDispatch(EnterNewRound{h, cert.Round + 1}).
- Unless this node recovered it: charge the verification budget and
The certificate does not touch locks, valid blocks or vote sets; it only moves the node to a higher round, which the locking rules make safe.
sequenceDiagram
participant P as Proposer of round r
participant V as Validators
participant N as Proposer of round r+1
P-->>V: ProposerHeartbeat every proposer_heartbeat
Note over P: goes offline
V->>V: no heartbeat for ProposerSilenceTimeout(r)
V-->>V: RoundSkipVote(h, r)
V->>V: +2/3 skip votes → RoundSkipCertificate(h, r)
V->>V: EnterNewRound(h, r+1): waits for txs
N-->>V: ProposerHeartbeat for round r+1
The round after a skip
EnterNewRoundAction.Execute lets round R > 0 wait for transactions, as
round 0 does, when StateData.waitsAfterRoundSkip(R) holds
(internal/consensus/state_data.go):
RoundSkip.Round == R − 1;LockedRound == -1 ∧ ValidRound == -1;- the mempool has not reported transactions at this height (
TxsReported; the mempool reports once per height, andState.handleTxsAvailablerecords the report in any round); - no polka at the height (
HeightVoteSet.POLInforound-1) and no prevote recorded for roundR.
The other wait conditions are unchanged (WaitForTxs(), h ≠ InitialHeight,
no ProposeNextBlockImmediately). A waiting round R > 0 publishes its
NewRound step, so peers send it the round's proposal. The empty-block tock is
scheduled after max(CreateEmptyBlocksInterval − (now − StartTime), 0): the
interval counts from the start of the height, not from the skipped round.
Gossip
Two gossip handlers run per peer at the PeerQueryMaj23SleepDuration cadence
(internal/consensus/gossip_handlers.go, wired in gossip_peer_worker.go):
roundSkipGossipHandlersends the heldRoundSkipto every peer at the height that is not committing and has not sent a certificate for that round or a later one (PeerState.HasRoundSkip). Nothing else moves a peer out of a skipped round: it has no votes to follow and the next proposal is sent only to peers already in the next round.roundVoteResendGossipHandler(roundSkippingonly, validator peers) sends one random vote per pass: the round's prevotes to a peer waiting in this node's roundr > 0; the prevotes and precommits of the peer's round, and of the latest later round with +2/3-any (laterRoundVotes), to a peer in a round this node has left.
Recovery
- Certificates are WAL
msgInforecords; replay applies them again without charging the verification budget. Replay never skips rounds on aroundSkippingchain (below). - On a
roundSkippingchainWalSkipRoundsToLastis ignored (State.replaySkipsRounds,internal/consensus/replay.go): whether a round after a certificate waits depends on polkas of earlier rounds. - The mempool report is not in the WAL. On a
roundSkippingchain a replayedProposestep event for the current height enters the recorded round, and enters the propose step if replay left the node waiting (State.readReplayMessage). - After a restart into a waiting round the heartbeat or watch starts before the
first input (
State.followLiveness).
Message admission and budgets
- State channel.
ProposerHeartbeat,RoundSkipVoteandRoundSkipCertificatetravel on the state channel (p2p.ResolveChannelID). Each pays the state-channel cost of 1 on arrival andlivenessTokenCost(100) more only once the checks that need no signature find it new (Reactor.chargeLivenessNews); it then joins the sender's peer lane. - Verification budget. Each costs
baseMessageCost(1) to verify (peerMessageCost,budgetedMessageCostininternal/consensus/message_cost.go). One that cannot be afforded is dropped; it is sent again. - Peer errors. All rejections below are logged at Debug
(
isPeerFloodableError):- a liveness message that does not decode or fails its basic checks
(
MsgFromProto,errMalformedLivenessMessage) is reported asp2p.ErrInvalidData, on which the router disconnects the sender; - a heartbeat from the wrong proposer or a skip vote whose index and
pro-tx-hash disagree, found by the reactor's screening, is a non-fatal
peer error (
PeerStatusBad); - a heartbeat, skip vote or certificate whose signature fails against the
key the node holds is reported as a fatal peer error, which disconnects
the sender (
forgedLivenessMiddleware); - a certificate for another quorum (
ErrRoundSkipCertificateWrongQuorum) is only logged.
- a liveness message that does not decode or fails its basic checks
(
- Outboxes. Liveness messages are sent from a reactor goroutine through a
bounded outbox (
livenessOutboxSize = 256), each send bounded byvoteSetResponseTimeout; a message that finds the outbox full is dropped and sent again on its own schedule. Round-step, valid-block,HasVoteandHasCommitbroadcasts go through a separate outbox (stateOutboxSize = 1024); after a drop the sender rebroadcasts the current round step (Reactor.sendState). Neither blocks the consensus goroutine.
Signers
FilePV (privval/file.go) and DashCoreSignerClient
(privval/dash_core_signer_client.go) sign heartbeats and skip votes
(SignProposerHeartbeat, SignRoundSkipVote of the PrivValidator
interface, types/priv_validator.go).
The socket and gRPC remote signers (privval/signer_client.go,
privval/grpc/client.go) return RemoteSignerError code 501, which matches
types.ErrSigningNotSupported. A validator behind such a signer sends no
heartbeats, so validators skip its waiting rounds after the silence timeout,
and it signs no skip votes; it still counts others' votes and applies
certificates.
Parameters
TimeoutParams.ProposerHeartbeat (field 7) and ProposerSilence (field 8)
come from the committed state's consensus parameters, like the other timeouts.
TimeoutParamsOrDefaults derives the values in effect from what is stored:
| Stored heartbeat | Stored silence | Heartbeat in effect | Silence in effect |
|---|---|---|---|
| unset | unset | 1 s | 5 s |
| unset | s | min(1 s, s / 5) | s |
b | unset | b | max(5 s, 5·b) |
b | s | b | s |
- Validation (
ConsensusParams.ValidateConsensusParams): neither negative; the derived heartbeat is positive and shorter than the derived silence. - Updates (
ConsensusParams.UpdateConsensusParams): an update that names either field, or that raisesConsensusVersionfrom below 2 to 2 or more, clears both and sets only those it names; the other is derived again. - Encoding (
ConsensusParams.ToProto): unset fields are omitted. - Hashing (
ConsensusParams.HashConsensusParams): from version 2,HashedParams.proposer_heartbeat_ns(field 4) andproposer_silence_ns(field 5) carry the values in effect, in nanoseconds; earlier versions leave them unset, so their hash is unchanged.
Metrics
| Metric | Counts |
|---|---|
consensus_proposer_heartbeats_sent | Heartbeats this node signed and broadcast |
consensus_proposer_heartbeats_received | New, verified heartbeats for the round this node waits in |
consensus_proposer_silences | Rounds in which this validator stopped hearing from the proposer while waiting |
consensus_round_skips | Rounds this node moved past on a certificate |
Defined in internal/consensus/metrics.go.
Commit handling
A commit for height h is (Height, Round, BlockID, QuorumHash, ThresholdBlockSignature, ThresholdVoteExtensions) (types.Commit). A node
obtains one in two ways:
- locally, from its own precommit vote set once the threshold signature
is recovered (
VoteSet.MakeCommit); - from a peer or the WAL, as a
CommitMessage.
In both cases the block must be held, processed by the application for the commit's round, validated, and its extension vector accepted by the application before it is saved and finalized.
Commit verification
ValidatorSet.verifyCommit (types/validator_set.go) checks:
commit.Height == heightandcommit.BlockID == blockID(hash, part-set header, state id);commit.QuorumHash == Validators.QuorumHash(ErrInvalidCommitQuorumHash);- the number of extension signatures matches the number of extensions and is
≤ MaxVoteExtensions(ErrVoteExtensionCountMismatch); - the block threshold signature and every extension threshold signature
verify against
ThresholdPublicKeyover the canonical vote sign items (QuorumSignData.Verify); failure →ErrInvalidCommitSignature.
Only ErrInvalidCommitSignature identifies a dishonest sender (an honest
node stores a commit only after verifying it) and leads to eviction
(TryAddCommitAction.handleCommitVerifyError); every other failure is only
counted in CommitVerifyFailures{reason}. Replayed commits never evict.
Consensus verifies a received commit against the commit's own BlockID
(StateData.readyToApplyCommit), not against the proposal it holds: a
proposal the network never committed proves nothing about the commit, and
checking against it rejected genuine commits (dashpay/tenderdash#1414). Step 1
above is therefore tautological on this path and the block is matched to the
commit afterwards by verifyCommitBlock.
Applying a commit built from local precommits
Entered through EnterCommitAction → TryFinalizeCommitAction
(internal/consensus/state_try_finalize_commit.go).
- Precondition:
Step == ApplyCommit,maj23(Precommits(CommitRound))is a non-nilB. - If the node does not hold
B(!ProposalBlock.HashesTo(B.Hash)) the action returns quietly;EnterCommithas already retargeted the part set toBand published aValidBlockevent, so peers sendB's parts; when the last part arrivesProposalCompletedAction(stepApplyCommit) dispatchesTryFinalizeCommitagain. finalizeCommit:seenCommit = Precommits(CommitRound).MakeCommit();refuseUnprocessedCommit;mustEnsureProcess(rs, seenCommit.Round);verifyCommitExtensions(ABCIVerifyVoteExtensionon the canonical commit vote). If the application rejects its own quorum's extension vector, nothing is persisted, the failure is counted, and the node stays inApplyCommitwaiting for a peer's commit (a panic would restart into the same commit). OtherwiseDispatch(ApplyCommit{seenCommit}).
ApplyCommitAction.Execute is specified in
happy path §9. Note that it
re-runs ProcessProposal if the block was last processed for a different
round than commit.Round (mustEnsureProcess), so the number of
ProcessProposal calls per height is not bounded by one.
Commits received from peers
CommitMessage (vote channel) → commitMessageHandler →
TryAddCommitAction.Execute (internal/consensus/state_try_add_commit.go).
flowchart TD
A[CommitMessage c] --> B{rs.Commit parked?}
B -- yes --> Q{c.Height == h?}
Q -- yes --> QC[queueCandidate: ValidateBasic, dedupe, verify threshold signature, keep in sender's slot]
Q -- no --> IGN[ignore]
B -- no --> C{c.Height == h and c.Round > r?}
C -- yes --> FR[readyToApplyCommit ignoreProposalBlock=true]
FR --> FRV{verified?}
FRV -- no --> ERR[handleCommitVerifyError]
FRV -- yes --> HELD{block held?}
HELD -- yes --> VH{verifyHeldCommit ok?}
VH -- no --> RB[rs.Commit = nil; error]
VH -- yes --> NR
HELD -- no --> NR[EnterNewRound h, c.Round]
NR --> AC{block still held?}
AC -- yes --> ADD[AddCommit c]
AC -- no --> PARK[parked; wait for parts]
C -- no --> P[prepareCommitForApply]
P --> PV{verified and block held?}
PV -- error --> ERR
PV -- not ready --> PK[parked via adoptCommit; publish ValidBlock]
PV -- yes --> SET[rs.Commit = c] --> ADD
readyToApplyCommit (internal/consensus/state_data.go)
commit.ValidateBasic()(non-nil block id, signature size, known extension types).commit.Height + 1 == h→ ignored ("old commit");commit.Height ≠ h→ ignored. There is noLastCommitvote set to feed.verifyCommitSignatures(commit.BlockID, commit)— the threshold signature check, charged to the sender's verification budget.- If a complete block is held whose part set has the commit's header but whose hash differs → error ("retained block does not hash to commit hash"): parts determine the block, so waiting cannot resolve it.
- With
ignoreProposalBlock(future round):adoptCommitand return ready. - If
holdsProposalBlock(commit.BlockID)→ ready. - Else
adoptCommit(commit):retargetTo(commit.BlockID, park)— drops a proposal for another block, replaces the part set unless it already has the commit's header — andrs.Commit = commit; return not ready (parked).
Held block: verifyCommitBlock and verifyProcessedCommit
verifyCommitBlock: part-set header, block hash and state id of the commit must equal the held block's. A state-id mismatch is fatal for the commit: applied anyway, it would be recorded with a block id the next height'sLastBlockIDcould never match.verifyProcessedCommit:refuseUnprocessedCommit(expectations cache),ensureProcess(rs, commit.Round)(ProcessProposalfor the commit's round if not already processed;ErrClientStoppedpanics),validate,verifyCommitExtensions(ABCI). A failure returns an error; the commit is not applied.
Commit for the current round (or earlier)
prepareCommitForApply(ignoreProposalBlock = false): readyToApplyCommit,
then verifyCommitBlock, then verifyProcessedCommit. On success
rs.Commit = commit and Dispatch(AddCommit{commit}). When the block is not
held the commit is parked and a ValidBlock event is published so peers
serve the block.
Commit for a later round
commit.Round > r: readyToApplyCommit(ignoreProposalBlock = true) verifies
the signature and parks the commit. If the block is already held it is
verified first (verifyHeldCommit), so that a rejected commit does not move
the round (on rejection rs.Commit is reset — the retarget could at most
have dropped a proposal whose state id differed, which such a commit could
not have matched anyway). Then EnterNewRound(h, commit.Round) — which keeps
the committed block as the download target — and, if the block is held,
AddCommit.
AddCommitAction.Execute (internal/consensus/state_add_commit.go)
proposalUpdater.updateStateData(commit.BlockID): install the locked block if it is the committed one, else retarget.- If
ProposalBlock == nil(not held): return; the completing part will dispatchAddCommitagain viaselectParkedCommit. Step = ApplyCommit(round kept),CommitRound = commit.Round,CommitTime = now;Dispatch(ApplyCommit{commit})(happy path §9);PublishCommitEvent(peers receiveHasCommit);Dispatch(EnterNewRound{Height: rs.Height})for the new height.
Parked commit and replacement candidates
While rs.Commit ≠ nil, further commits for the height are not applied but
queued (TryAddCommitAction.queueCandidate,
internal/consensus/commit_candidates.go): after ValidateBasic and
deduplication against the parked commit and the sender's previous candidate,
the threshold signature is verified and the commit is stored in the sender's
single slot (overwritten by that sender's later commits; freed on
disconnect). Candidates are keyed by (height, round, blockID, hash of the extension vector) because the application's verdict depends on all four.
When the block completes (addProposalBlockPart with rs.Commit ≠ nil),
selectParkedCommit tries the parked commit and then the candidates, grouped
by round (the round already processed first, then rounds in arrival order),
calling verifyHeldCommit for each; the first accepted becomes rs.Commit
and is dispatched as AddCommit. If none is accepted, rs.Commit = nil and
the completed block continues through the ordinary flow
(ProposalCompleted).
commitExpectations (internal/consensus/commit_expectations.go) remembers,
per (height, round, block), the extension vector the application's own
ExtendVote returns after a rejection, and whether that vector was itself
rejected. A later commit for the same (round, block) with a different
vector, or with the rejected one, is refused without re-processing the block
(blockExecutor.refuseUnprocessedCommit, verifyCommitExtensions). The
block and round are authenticated by the threshold signature, the vector is
not, so this bounds ProcessProposal re-executions per height to roughly one
per foreign round plus returns to the node's own round.
Commit validity depends on the application
A commit's acceptability includes an ABCI verdict (VerifyVoteExtension on
the commit vote). Two signature-valid commits for the same block can differ
in round or extension vector and receive different verdicts; liveness
therefore depends on the application answering deterministically for the
round the quorum signed.
Interaction with the step machine
Proposaler.Setrefuses a proposal that disagrees with a parked commit for the same round and ignores a later-round proposal that would disrupt the download of a parked commit's block.EnterNewRoundkeeps the part set targeted at a parked commit's block and re-announces it (ValidBlockevent).EnterProposedoes not propose while a verified commit for a held block is about to be applied (applyingCommit).AddProposalBlockPartdispatchesProposalCompletedonly when no commit is parked or the parked one was just rejected.
Commit gossip
(internal/consensus/gossip_handlers.go, gossiper.go, reactor.go)
- After applying a commit the node broadcasts
HasCommit(PublishCommitEvent→ reactor broadcast) and every step change broadcastsNewRoundStep. - Per peer, the vote/commit gossip loop sends
CommitMessage:rs.LastCommitwhen the peer is exactly one height behind and has not announced the commit (shouldCommitBeGossiped);- the stored commit for the peer's height when it is two or more heights
behind (
shouldCommitBeGossipedForCatchup,blockStore.LoadBlockCommit/LoadSeenCommit).
- Block parts for a lagging peer come from the block store, one per tick,
with a quiet gap of
catchupResendIntervalbetween passes (GossipBlockPartsForCatchup); a non-validator peer at the current height receives parts once it announced a commit (shouldBlockPartsBeGossiped). VoteSetMaj23for the stored commit is sent to validator peers that announced a catch-up commit round (GossipVoteSetMaj23).
Recovery and synchronisation
Write-ahead log
The consensus WAL (internal/consensus/wal.go; file layout in
WAL) records, per height:
| Record | Written by | Durability |
|---|---|---|
msgInfo — proposal, block part, vote or commit, with the peer id and receive time | walMiddleware (internal/consensus/msg_handlers.go) | peer messages Write (buffered); the node's own messages WriteSync (fsync), panic on failure |
msgInfo — round skip certificate, once verified and news | AddRoundSkipCertificateAction.Execute | Write; a failure is only logged |
timeoutInfo — every tock | State.receiveRoutine | Write |
EventDataRoundState — every step change | EventPublisher.PublishNewRoundStepEvent | Write |
EndHeightMessage{h} | ApplyCommitAction.Execute, after SaveBlock and before FinalizeBlock | WriteSync, panic on failure |
verificationBudgetMiddleware wraps walMiddleware, so a message dropped
for budget reasons leaves no WAL record. Proposer heartbeats and round skip
votes are never written (livenessMessage). Before signing a proposal or a vote
the WAL is flushed and fsynced (EnterProposeAction.Execute,
voteSigner.signVote) so that replay reproduces the same message for the
private validator's double-sign protection. A failed sync stops the vote
(signVote returns the error, nothing is signed) but not the proposal:
EnterProposeAction.Execute logs it and proposes anyway. An empty WAL starts with
EndHeightMessage{0} (BaseWAL.OnStart).
Start-up: ABCI handshake
Handshaker.Handshake (internal/consensus/replay.go) calls ABCI Info and
BlockReplayer.Replay (internal/consensus/replayer.go) with the
application's (LastBlockHeight, LastBlockAppHash):
- If the application height is 0:
InitChainwith the genesis validator set, parameters,InitialCoreHeight; when Tenderdash itself is at height 0 the returned validator set, parameters, app hash and optional genesis time become the initial state (execInitChain). replayState.validate: the application may not be ahead of the block store; the state may not be ahead of the store; the store may be at most one block ahead of the state; a truncated store must not have pruned the blocks the application needs.- Cases (
syncStateIfItIsEqualStore,syncStateIfItIsOneAheadOfStore):
| Store vs state | App vs state | Action |
|---|---|---|
| equal | behind | replay stored blocks into the application: per block ProcessProposal(round = commit.Round), VerifyCommitExtensions, FinalizeBlock (replayBlock); the resulting app hash must equal the block's; a rejected commit vector stops replay (commitExtensionsReplayError) |
| equal | equal | app hash must equal state.LastAppHash |
| store = state + 1 | behind state | replay all but the last block as above, then apply the last one through BlockExecutor.ApplyBlock with the real application |
| store = state + 1 | equal state | apply the last block with the real application (syncStateAt) |
| store = state + 1 | equal store | the application already finalized it: apply the last block against a mock application that replays the saved ABCIResponses (newMockProxyApp), so that state and store catch up without re-executing |
Start-up: WAL catch-up
State.OnStart (internal/consensus/state.go) rebuilds LastCommit from
the stored seen commit (updateStateFromStore), opens the WAL, starts the
ticker and, unless doWALCatchup was disabled by block sync, runs
State.catchupReplay:
- the WAL must not contain
#ENDHEIGHT hfor the current height; - it must contain
#ENDHEIGHT h − 1(0atInitialHeight); - every record after it is replayed through
readReplayMessage: step events are only logged (except as below),msgInfos are dispatched withfromReplay(no peer budgets, no evictions), tocks go throughhandleTimeout. WithWalSkipRoundsToLasta replayed proposal for a later round advances the round directly; onCONSENSUS_VERSION_2chains that skip rounds the setting is ignored, and a replayedProposestep event re-enters the recorded round and propose step (State.replaySkipsRounds,State.readReplayMessage; Recovery); - replay mode: the node does not propose (
EnterProposeAction.Execute) and skips timeliness checks (isValidForPrevote), but it may still attempt to sign votes: replayed messages drive the normal actions, and neitherprevoter.DonorvoteSigner.signVotechecks replay mode, so a replayed step can also produce a vote that was never signed before the crash. Its earlier votes also come back from the WAL as internal messages. Conflicting block votes are prevented by the signer's persisted state:FilePVreuses the stored block signature for identical sign bytes at the same height/round/step (re-signing only precommit extensions) and refuses regressions and conflicting sign bytes (see WAL).DashCoreSignerClientkeeps no such state in Tenderdash; - a corrupted WAL is backed up to
<wal>.CORRUPTED, repaired by truncating at the first undecodable record (repairWalFile), and replay is retried once; other errors are logged and the node starts anyway.
Then receiveRoutine starts and ScheduleRound0 arms the first tock.
Crash windows
| Crash point | Recovery |
|---|---|
Before SaveBlock | WAL replay re-delivers the height's messages and timeouts; the node re-reaches the same step |
After SaveBlock, before FinalizeBlock completed (with or without EndHeightMessage{h}) | handshake case "store = state + 1, app = state": ApplyBlock with the real application. Without the marker, catchupReplay(h + 1) then fails with "WAL does not contain #ENDHEIGHT for h"; the error is not a corruption error, so State.OnStart logs it and starts without replay |
After FinalizeBlock, before stateStore.Save | handshake case "app = store": mock replay from saved ABCIResponses (BlockExecutor.FinalizeBlock saves them before calling the application) |
Block-sync handover
Reactor.SwitchToConsensus(state, skipWAL, targetHeight)
(internal/consensus/reactor.go):
- If
targetHeight > state.LastBlockHeight(peers reported more blocks than were applied), armcatchupTrackerso thatEnterProposerefuses to propose until the node passestargetHeightormaxCatchupSuppression(10 min) elapses; a block built from present-day application state must not be proposed into a height the network committed long ago. - Rebuild
LastCommitfrom the stored seen commit;updateToState; startState; publish the step; mark syncing complete; withskipWAL, disable WAL catch-up.
While the reactor is still in sync mode (WaitSync) it ignores data, vote
and vote-set-bits messages.
A node that restarts mid-height and re-joins through consensus (no block
sync) relies on peers' commit gossip for the heights it missed
(Commit gossip): the proposer-selector
reconstruction reads proposers from the block store to resynchronise the
round-robin (proposerFromStore).
Known issues and discrepancies
Behaviour found while deriving this specification that looks like a bug, diverges from the Tendermint algorithm, or is otherwise surprising. Each entry describes and cites; none is fixed here. "Inferred" marks conclusions drawn from reading the code without a reproducing test.
An untimely ValidBlock is never re-proposed
Proposaler.Create (internal/consensus/state_proposaler.go) re-proposes
ValidBlock only when checkValidBlock finds it timely for ValidRound
(ValidBlock.IsTimely(ValidBlockRecvTime, SynchronyParams, ValidRound));
otherwise it builds a fresh block and proposes it with POLRound = -1. The
PBTS algorithm checks timeliness only for proposals without a POL round, so a
block that already gathered a polka would be re-proposed regardless of its
timestamp.
A proposer sees its ValidBlock as untimely when it learned the block late.
updateValidBlock dates a block with tmtime.Now() when the node learned it
without a matching proposal, which puts the receive time well after
block.Time. The proposer logs proposal block is not timely at ERROR level
when this happens.
Effect: validators locked on ValidBlock prevote nil for the fresh block,
since a lock is cleared only at a new height (see
No unlock within a height). If the voting power
locked on that block exceeds TotalVotingPower - QuorumVotingThresholdPower
and every subsequent proposer sees it as untimely, no block can gather a
polka and the height stalls (inferred; no reproducing test).
PrecommitWait step is never entered
RoundStepPrecommitWait exists in RoundStepType
(internal/consensus/types/round_state.go) and is used as the step of the
precommit-wait timeout and in peer-state comparisons
(pickVoteForGossip), but EnterPrecommitWaitAction.Execute
(internal/consensus/state_enter_wait.go) only sets
TriggeredTimeoutPrecommit and never calls updateRoundStep(round, RoundStepPrecommitWait). A node waiting for the precommit timeout reports
step Precommit. Consequences are cosmetic (events, metrics, peer state
comparisons), and handleTimeout accepts the PrecommitWait tock because
its step is greater than Precommit.
Proposer clock-wait can be swallowed by the ticker
EnterProposeAction.Execute reschedules a (h, r, NewRound) tock when the
proposer's clock is behind state.LastBlockTime. The ticker
(timeoutTicker.timeoutRoutine) ignores a tick whose step is ≤ the step of
the last scheduled tick for the same height and round, and it does not forget
the last tick after it fires. When EnterPropose itself was triggered by a
(h, r, NewRound) tock — the empty-block interval
(EnterNewRoundAction.Execute) or the transactions-available path
(State.handleTxsAvailable) — the rescheduled tick is dropped and nothing
else moves the node out of NewRound for that round. The node recovers only
when the other validators' nil precommits reach +2/3-any
(addVoteDispatchPrecommitMw → EnterPrecommitWait → EnterNewRound(r + 1)),
costing the network one round. Preconditions: WaitForTxs() true, round 0
(on CONSENSUS_VERSION_2 chains also a round entered on a round skip
certificate that waits, whose empty-block interval schedules the same
NewRound tock), the node is the proposer, and its clock is behind the previous block time by
more than the time since that block was produced. (Inferred; not reproduced.)
Precommit requires this round's proposal even when locked
EnterPrecommitAction.Execute, rule C-LockedNotProposed: a validator locked
on B that sees a polka for B in round r' > LockedRound precommits nil
unless B is also round r''s proposal block. The Tendermint paper (line
36) also requires a proposal for v in the same round, so this is faithful
to the algorithm, but it is stricter than the pre-vote-extension Tendermint
implementation, which relocked on the polka alone. The code's rationale is the
application contract: ExtendVote is answered for the round the application
processed last, and Dash Drive refuses any other. Liveness cost: a locked
validator that missed the round's proposal cannot help commit B in that
round.
No unlock within a height
LockedRound/LockedBlock are only replaced by a new lock at a later round
or reset at the next height (StateData.updateLockedBlock,
StateData.updateToState). The paper's unlock-on-nil-polka rule from the
original Tendermint implementation is absent by design
(TestStateLock_POLDoesNotUnlock), relying on P-Lock-POL to move the lock
forward instead.
Commit threshold is configurable below the BFT bound
ValidatorSet.QuorumVotingThresholdPower honours VotingPowerThreshold,
which may be lower than ⌊2n/3⌋ + 1 validators; several devnet LLMQ types
have a 50 % threshold. The code only warns
(ValidatorSet.BelowStrictThreshold, logged by StateData.updateToState).
Safety of the protocol is not guaranteed below the strict floor.
Proposer selection v0 skips the first validator after quorum rotation
heightProposerSelector.proposerFromStore
(internal/consensus/versioned/selectproposer/height_proposer.go) deliberately
sets the proposer to index 1 after a validator-set change ("bug preserved
to avoid breaking consensus"); the v1 selector uses index 0. See
Proposer selection.
FilePV repeated proposal returns sign bytes as the signature
FilePV.signProposal (privval/file.go) checks for an identical persisted
height/round/step and identical canonical proposal bytes. In that branch it assigns
proposal.Signature = lss.BlockSignBytes, rather than lss.BlockSignature.
The vote path instead reuses BlockSignature. The repeated-proposal path therefore
returns serialized sign bytes in a signature field; verification failure is the
inferred effect (no reproducing test). This differs from a guarantee of successful
signature reuse during recovery. WAL replay itself does not create proposals.
Code/comment drift and dead code
-
BlockExecutor.ProcessProposal(internal/state/execution.go) checkslen(resp.AppHash) != appHashSizeafterNewStateChangeset, butResponseProcessProposal.Validatealready returned for any size other thancrypto.DefaultAppHashSize; the later check is unreachable unlessSetAppHashSizechanges the size, which no non-test caller does. -
EnterNewRoundAction's doc comment mentionstimeoutNewHeight,timeoutCommitandSkipTimeoutCommit, none of which exist; round 0 starts atCommitTime(StateData.updateToState,roundScheduler.ScheduleRound0). -
EnterPrevoteEvent.AllowOldBlocksis set byProposalCompletedActionand never read (internal/consensus/state_enter_prevote.go). -
ErrInvalidProposalNotSetandErrAddingVote(internal/consensus/state.go) have no non-comment users. -
ApplyCommitAction.Executehandlescommit == nil, but every caller passes a commit. -
config.ConsensusConfig.DoubleSignCheckHeightis exposed as a CLI flag (cmd/tenderdash/commands/run_node.go) but no consumer was found underinternal/consensus(inferred from grep). -
blockExecutor.createreturns(nil, nil)whenLastCommit == nilaboveInitialHeight("shouldn't happen");Proposaler.Createwould then dereference a nil block inmakeProposal. Unreachable in normal operation becauseupdateToStatepanics on a missingLastCommit. -
EnterPrecommitAction.Executelogs routine nil precommits (no polka, nil polka) at error level, including the ordinary timeout path.
Non-fatal optimistic-concurrency errors
State.receiveRoutine persists StateData after each handler with a version
check (StateDataStore.update); a mismatch is only logged, after the
handler's side effects (signed votes, WAL records) already happened. No
concurrent writer exists today, so the check is a latent guard rather than a
protection.
ABCI
Introduction
ABCI (Application Blockchain Interface) connects Tenderdash's
consensus engine to the Application's state machine.
The API consists of a set of methods, each with a corresponding Request and Response
message type.
The methods are always initiated by Tenderdash. The Application implements its logic
for handling all ABCI methods.
Thus, Tenderdash always sends the Request* messages and receives the Response* messages
in return.
All ABCI messages and methods are defined in protocol buffers. This allows Tenderdash to run with applications written in many programming languages.
This specification is split as follows:
- Overview and basic concepts - interface's overview and concepts needed to understand other parts of this specification.
- Methods - complete details on all ABCI methods and message types.
- Requirements for the Application - formal requirements on the Application's logic to ensure Tenderdash properties such as liveness. These requirements define what Tenderdash expects from the Application; second part on managing ABCI application state and related topics.
- Tenderdash's expected behavior - specification of how the different ABCI methods may be called by Tenderdash. This explains what the Application is to expect from Tenderdash.
- Client and Server - for those looking to implement their own ABCI application servers
Application Requirements
This section specifies what Tenderdash expects from the Application. It is structured as a set of formal requirement that can be used for testing and verification of the Application's logic.
Let $p$ and $q$ be two different correct proposers in rounds $r_p$ and $r_q$ respectively, in height $h$.
Let $s_{p,h-1}$ be $p$'s Application's state committed for height $h-1$.
Let $v_p$ (resp. $v_q$) be the block that $p$'s (resp. $q$'s) Tenderdash passes on to the Application
via RequestPrepareProposal as proposer of round $r_p$ (resp $r_q$), height $h$, also known as the
raw proposal.
Let $v'_p$ (resp. $v'_q$) the possibly modified block $p$'s (resp. $q$'s) Application returns via
ResponsePrepareProposal to Tenderdash, also known as the prepared proposal.
Process $p$'s prepared proposal can differ in two different rounds where $p$ is the proposer.
- Requirement 1 [
PrepareProposal, header-changes] When the blockchain is in same-block execution mode, $p$'s Application provides values for the following parameters inResponsePrepareProposal: AppHash, TxResults, ConsensusParams, ValidatorSetUpdate. Provided values for ConsensusParams and ValidatorSetUpdate MAY be empty to denote that the Application wishes to keep the current values.
Parameters AppHash, TxResults, ConsensusParams, and ValidatorSetUpdate are used by Tenderdash to compute various hashes in the block header that will finally be part of the proposal.
In practical terms, Requirement 1 imply that Tenderdash will panic if the Application is in same-block execution mode and does not provide values for AppHash, TxResults, ConsensusParams, and ValidatorSetUpdate.
- Requirement 2 [
PrepareProposal, timeliness] If $p$'s Application fully executes prepared blocks inPrepareProposaland the network is in a synchronous period while processes $p$ and $q$ are in $r_p$, then the value of TimeoutPropose at $q$ must be such that $q$'s propose timer does not time out (which would result in $q$ prevoting nil in $r_p$).
Full execution of blocks at PrepareProposal time stands on Tenderdash's critical path. Thus,
Requirement 2 ensures the Application will set a value for TimeoutPropose such that the time it takes
to fully execute blocks in PrepareProposal does not interfere with Tenderdash's propose timer.
- Requirement 3 [
PrepareProposal, tx-size] When $p$'s Application callsResponsePrepareProposal, the total size in bytes of the transactions returned does not exceedRequestPrepareProposal.max_tx_bytes.
Busy blockchains might seek to maximize the amount of transactions included in each block. Under those conditions,
Tenderdash might choose to increase the transactions passed to the Application via RequestPrepareProposal.txs
beyond the RequestPrepareProposal.max_tx_bytes limit. The idea is that, if the Application drops some of
those transactions, it can still return a transaction list whose byte size is as close to
RequestPrepareProposal.max_tx_bytes as possible. Thus, Requirement 3 ensures that the size in bytes of the
transaction list returned by the application will never cause the resulting block to go beyond its byte limit.
- Requirement 4 [
PrepareProposal,ProcessProposal, coherence]: For any two correct processes $p$ and $q$, if $q$'s Tenderdash callsRequestProcessProposalon $v'_p$, $q$'s Application returns Accept inResponseProcessProposal.
Requirement 4 makes sure that blocks proposed by correct processes always pass the correct receiving process's
ProcessProposal check.
On the other hand, if there is a deterministic bug in PrepareProposal or ProcessProposal (or in both),
strictly speaking, this makes all processes that hit the bug byzantine. This is a problem in practice,
as very often validators are running the Application from the same codebase, so potentially all would
likely hit the bug at the same time. This would result in most (or all) processes prevoting nil, with the
serious consequences on Tenderdash's liveness that this entails. Due to its criticality, Requirement 4 is a
target for extensive testing and automated verification.
-
Requirement 5 [
ProcessProposal, determinism-1]:ProcessProposalis a (deterministic) function of the current state and the block that is about to be applied. In other words, for any correct process $p$, and any arbitrary block $v'$, if $p$'s Tenderdash callsRequestProcessProposalon $v'$ at height $h$, then $p$'s Application's acceptance or rejection exclusively depends on $v'$ and $s_{p,h-1}$. -
Requirement 6 [
ProcessProposal, determinism-2]: For any two correct processes $p$ and $q$, and any arbitrary block $v'$, if $p$'s (resp. $q$'s) Tenderdash callsRequestProcessProposalon $v'$ at height $h$, then $p$'s Application accepts $v'$ if and only if $q$'s Application accepts $v'$. Note that this requirement follows from Requirement 5 and the Agreement property of consensus.
Requirements 5 and 6 ensure that all correct processes will react in the same way to a proposed block, even
if the proposer is Byzantine. However, ProcessProposal may contain a bug that renders the
acceptance or rejection of the block non-deterministic, and therefore prevents processes hitting
the bug from fulfilling Requirements 5 or 6 (effectively making those processes Byzantine).
In such a scenario, Tenderdash's liveness cannot be guaranteed.
Again, this is a problem in practice if most validators are running the same software, as they are likely
to hit the bug at the same point. There is currently no clear solution to help with this situation, so
the Application designers/implementors must proceed very carefully with the logic/implementation
of ProcessProposal. As a general rule ProcessProposal should always accept the block.
A correct validator broadcasts at most one precommit per height and round.
ExtendVote can also run after a rejected commit to discover the expected vector,
including on an observer. Applications must support repeated calls and return the
same threshold-recoverable subset for the same block, round and committed state.
See expected call sequences.
Let $e^r_p$ be the set of vote extensions that the Application of a correct process $p$ returns via ResponseExtendVote in round $r$, height $h$.
Let $w^r_p$ be the proposed block that $p$'s Tenderdash passes to the Application via RequestExtendVote in round $r$, height $h$.
- Requirement 7 [
ExtendVote,VerifyVoteExtension, coherence]: For any two correct processes $p$ and $q$, if $q$ receives $e^r_p$ from $p$ in height $h$, $q$'s Application returns Accept inResponseVerifyVoteExtension.
Requirement 7 constrains the creation and handling of vote extensions in a similar way as Requirement 4
contrains the creation and handling of proposed blocks.
Requirement 7 ensures that extensions created by correct processes always pass the VerifyVoteExtension
checks performed by correct processes receiving those extensions.
However, if there is a (deterministic) bug in ExtendVote or VerifyVoteExtension (or in both),
we will face the same liveness issues as described for Requirement 4, as Precommit messages with invalid vote
extensions will be discarded.
-
Requirement 8 [
VerifyVoteExtension, determinism-1]:VerifyVoteExtensionis a (deterministic) function of the current state, the vote extensions received, and the prepared proposal that the extensions refers to. In other words, for any correct process $p$, and any arbitrary vote extension $e$, and any arbitrary block $w$, if $p$'s (resp. $q$'s) Tenderdash callsRequestVerifyVoteExtensionon $e$ and $w$ at height $h$, then $p$'s Application's acceptance or rejection exclusively depends on $e$, $w$ and $s_{p,h-1}$. -
Requirement 9 [
VerifyVoteExtension, determinism-2]: For any two correct processes $p$ and $q$, and any arbitrary vote extensions $e$, and any arbitrary block $w$, if $p$'s (resp. $q$'s) Tenderdash callsRequestVerifyVoteExtensionon $e$ and $w$ at height $h$, then $p$'s Application accepts $e$ if and only if $q$'s Application accepts $e$. Note that this requirement follows from Requirement 8 and the Agreement property of consensus.
Requirements 8 and 9 ensure that the validation of vote extensions will be deterministic at all
correct processes.
Requirements 8 and 9 protect against arbitrary vote extensions data from Byzantine processes
similarly to Requirements 5 and 6 and proposed blocks.
Requirements 8 and 9 can be violated by a bug inducing non-determinism in
VerifyVoteExtension. In this case liveness can be compromised.
Extra care should be put in the implementation of ExtendVote and VerifyVoteExtension and,
as a general rule, VerifyVoteExtension should always accept the vote extensions of a precommit.
Commit verification, a call with an empty validator_pro_tx_hash, is the exception: the Application
must reject a commit whose extension vector differs from the threshold-recoverable subset of the
extensions it expects for the block, or an altered vector reaches FinalizeBlock. Threshold recovery
requires the contributing validators to sign identical extension content, so a valid recovered extension
from correct processes matches that expected vector. Rejecting an altered vector therefore costs no
liveness; a deterministic bug in the check, however, halts the chain, because no commit
for the block can pass it. See VerifyVoteExtension for what
Tenderdash does on each path when a commit is rejected.
-
Requirement 10 [all, no-side-effects]: $p$'s calls to
RequestPrepareProposal,RequestProcessProposal,RequestExtendVote, andRequestVerifyVoteExtensionat height $h$ do not modify $s_{p,h-1}$. -
Requirement 11 [
ExtendVote,FinalizeBlock, non-dependency]: for any correct process $p$, and any vote extensions $e$ that $p$ received at height $h$, the computation of $s_{p,h}$ does not depend on $e$.
The call to correct process $p$'s RequestFinalizeBlock at height $h$, with block $v_{p,h}$
passed as parameter, creates state $s_{p,h}$.
PrepareProposal and ProcessProposal compute the transaction results $T_{p,h}$
for the current block before consensus decides it. FinalizeBlock persists the
selected candidate and returns no transaction results.
-
Requirement 12 [
FinalizeBlock, determinism-1]: For any correct process $p$, $s_{p,h}$ exclusively depends on $s_{p,h-1}$ and $v_{p,h}$. -
Requirement 13 [
FinalizeBlock, determinism-2]: For any correct process $p$, the contents of $T_{p,h}$ exclusively depend on $s_{p,h-1}$ and $v_{p,h}$.
Note that Requirements 12 and 13, combined with Agreement property of consensus ensure the Application state evolves consistently at all correct processes.
Finally, notice that neither PrepareProposal nor ExtendVote have determinism-related
requirements associated.
Indeed, PrepareProposal is not required to be deterministic:
- $v'p$ may depend on $v_p$ and $s{p,h-1}$, but may also depend on other values or operations.
- $v_p = v_q \nRightarrow v'_p = v'_q$.
Likewise, ExtendVote can also be non-deterministic:
- $e^r_p$ may depend on $w^r_p$ and $s_{p,h-1}$, but may also depend on other values or operations.
- $w^r_p = w^r_q \nRightarrow e^r_p = e^r_q$
In Tenderdash this freedom holds for extensions that are not threshold-recoverable, and does not
extend to threshold-recoverable ones (THRESHOLD_RECOVER and THRESHOLD_RECOVER_RAW): they are
recovered into the commit, so every validator must produce the same ones. ExtendVote must also be
free of side effects, and the threshold-recoverable subset of its response must be exactly the vector
the Application accepts in a commit-level VerifyVoteExtension for the same block, height and round:
after such a VerifyVoteExtension rejects a commit, following ProcessProposal of that block and
round, Tenderdash calls ExtendVote on any node to learn that vector (see
VerifyVoteExtension).
Connections
Tenderdash uses one application client for consensus, mempool, query and state-sync calls. Its transport determines concurrency: the reference socket server processes one connection sequentially, while gRPC can execute calls concurrently over one connection. Explicitly configured routed clients may use multiple underlying connections. Local clients call the application directly.
Applications MUST synchronize state shared by concurrent calls, including calls on different methods or connections. Waiting for one response does not block other callers. Queries must read committed state. See Client and Server for transport ordering and gRPC concurrency settings.
Outline
- Outline
- Application execution
- Methods overview
- Same-block execution
- Determinism
- Errors
- Events
- Evidence
Overview and basic concepts
Application execution
The Application constructs and executes candidate blocks through PrepareProposal
and ProcessProposal. It supplies vote extensions through ExtendVote and
checks peer votes and commits through VerifyVoteExtension. FinalizeBlock
persists the decided candidate after consensus accepts the commit.
Methods overview
Methods can be classified into four categories: consensus, mempool, info, and state-sync.
Consensus/block execution methods
The first time a new blockchain is started, Tenderdash calls
InitChain. From then on, method FinalizeBlock is executed at the end of each
block, resulting in an updated Application state.
During consensus execution of a block height, before method FinalizeBlock is
called, methods PrepareProposal, ProcessProposal, ExtendVote, and
VerifyVoteExtension may be called several times.
See Tenderdash's expected behavior
for details on the possible call sequences of these methods.
-
InitChain: This method initializes the blockchain. Tenderdash calls it once upon genesis.
-
PrepareProposal: Constructs and executes a fresh candidate from mempool transactions, supplying the current block's commitments. A reusable valid block can be proposed without preparing a fresh candidate.
-
ProcessProposal: Executes a candidate deterministically and returns its status and commitments. Processing precedes the lock-based voting decision and can repeat when validating a commit from another round.
-
ExtendVote: Supplies the extensions for a non-nil precommit. It can also run on an observer after a rejected commit, without signing a vote.
-
VerifyVoteExtension: Checks a peer's precommit or a commit's recovered vector. At commit level, the Application must reject a vector that differs from the expected threshold-recoverable subset for that block and round. The methods contract describes the distinct rejection paths.
-
FinalizeBlock: It delivers a decided block to the Application to persist the candidate for the block and round processed last. The current block's application and result hashes were returned by
PrepareProposal/ProcessProposaland are already in its header.ResponseFinalizeBlockreturns only retention and next-block scheduling hints, not execution results or events. See Commit and apply.
Mempool methods
- CheckTx: This method allows the Application to validate transactions against its current state, e.g., checking signatures and account balances. If a transaction passes the validation, then Tenderdash adds it to its local mempool, discarding it otherwise. Tenderdash calls it when it receives a new transaction either coming from an external user or another node. Furthermore, Tenderdash can be configured to re-call
CheckTxon remaining mempool transactions afterFinalizeBlock.
Info methods
-
Info: Used to sync Tenderdash with the Application during a handshake that happens on startup.
-
Query: Clients can use this method to query the Application for information about the application state.
State-sync methods
State sync allows new nodes to rapidly bootstrap by discovering, fetching, and applying state machine snapshots instead of replaying historical blocks. For more details, see the state sync section.
New nodes will discover and request snapshots from other nodes in the P2P network.
A Tenderdash node that receives a request for snapshots from a peer will call
ListSnapshots on its Application. The Application returns the list of locally avaiable snapshots.
Note that the list does not contain the actual snapshot but metadata about it: height at which the snapshot was taken, application-specific verification data and more (see snapshot data type for more details). After receiving a list of available snapshots from a peer, the new node can offer any of the snapshots in the list to its local Application via the OfferSnapshot method. The Application can check at this point the validity of the snapshot metadata.
Snapshots may be quite large and are thus broken into smaller "chunks" that can be
assembled into the whole snapshot. Once the Application accepts a snapshot and
begins restoring it, Tenderdash will fetch snapshot "chunks" from existing nodes.
The node providing "chunks" will fetch them from its local Application using
the LoadSnapshotChunk method.
As the new node receives "chunks" it will apply them sequentially to the local
application with ApplySnapshotChunk. When all chunks have been applied, the
Application's AppHash is retrieved via an Info query.
To ensure that the sync proceeded correctly, Tenderdash compares the local Application's AppHash to the AppHash stored on the blockchain (verified via
light client verification).
In summary:
-
ListSnapshots: Used by nodes to discover available snapshots on peers.
-
LoadSnapshotChunk: Used by Tenderdash to retrieve snapshot chunks from the application to send to peers.
-
OfferSnapshot: When a node receives a snapshot from a peer, Tenderdash uses this method to offer the snapshot to the Application.
-
ApplySnapshotChunk: Used by Tenderdash to hand snapshot chunks to the Application.
Other methods
Additionally, there is a Flush method that is called on every connection, and an Echo method for debugging, which a waiting proposer also uses as a readiness probe.
More details on managing state across connections can be found in the section on ABCI Applications.
Same-block execution
Tenderdash uses same-block execution: PrepareProposal and ProcessProposal
return the application hash and transaction results for the block being proposed.
Those commitments are included in that block's header. Validator-set and
parameter changes from height h take effect at h + 1.
The authoritative lifecycle is the primary consensus flow and validator/parameter transition.
Tenderdash timeouts in same-block execution
Proposal preparation and processing are synchronous and consume the round's
proposal-time budget. Slow execution can cause other validators to prevote nil.
Timeouts are already consensus parameters in ConsensusParams.Timeout; updates
from a block apply at the next height. Local unsafe overrides can desynchronize
nodes. See Timeouts and consensus parameters
for the actual fields, defaults and scheduling rules.
Determinism
Application processing must produce deterministic consensus commitments for a
block, round and last committed state. PrepareProposal constructs a candidate;
ProcessProposal verifies its current-block application/results hashes and
next-state commitments. Candidate changes must not replace committed application
state until FinalizeBlock persists the decided candidate. There is no subsequent
ABCI Commit call.
Processing can revisit the same block and round after another execution context
has replaced it. Commit verification can also call ExtendVote without signing
to discover an expected vector after rejection. These calls must not be treated as
once-per-height callbacks; see
expected call sequences.
Extension generation and verification must not mutate committed application state.
Sources of non-determinism in applications may include:
- Hardware failures
- Cosmic rays, overheating, etc.
- Node-dependent state
- Random numbers
- Time
- Underspecification
- Library version changes
- Race conditions
- Floating point numbers
- JSON or protobuf serialization
- Iterating through hash-tables/maps/dictionaries
- External Sources
- Filesystem
- Network calls (eg. some external REST API service)
See #56 for original discussion.
Consensus commits only the consensus-relevant subset of transaction results:
Code, Data and GasUsed form ResultsHash. Events and diagnostic fields are
not part of that hash. See Header and
Events; ResponseFinalizeBlock has no transaction results or events.
Errors
ResponseQuery, ResponseCheckTx and ExecTxResult entries carry application
Code/Codespace values. Code zero means successful application processing;
a nonzero transaction code is not the same as rejecting a whole proposal.
ProcessProposal.status controls proposal acceptance, while
VerifyVoteExtension.status controls vote/commit-vector acceptance. The
nil-vote versus panic distinction
and commit rejection paths are part of
the consensus contract. ABCI transport errors and malformed responses must not be
assumed to act like a nonzero transaction code.
CheckTx
When Tenderdash receives a ResponseCheckTx with a non-zero Code, the associated
transaction will not be added to Tenderdash's mempool or it will be removed if
it is already included.
ExecTxResult (PrepareProposal and ProcessProposal)
ResponsePrepareProposal.tx_results and ResponseProcessProposal.tx_results
contain one result for each included transaction. The consensus-relevant result
fields contribute to the current block's ResultsHash, which is checked against
the Application's answer before voting. ResponseFinalizeBlock has no
tx_results field. See ExecTxResult.
Query
When Tenderdash receives a ResponseQuery with a non-zero Code, this code is
returned directly to the client that initiated the query.
Events
ResponseCheckTx has no events field. For block execution,
ResponseProcessProposal.events contains block events and each ExecTxResult
can contain transaction events. These results are retained for publication when
the block is finalized; ResponseFinalizeBlock has no events field.
Applications may respond to these ABCI methods with a set of events.
Events allow applications to associate metadata about ABCI method execution with the
transactions and blocks this metadata relates to.
Events returned via these ABCI methods do not impact Tenderdash consensus in any way
and instead exist to power subscriptions and queries of Tenderdash state.
An Event contains a type and a list of EventAttributes, which are key-value
string pairs denoting metadata about what happened during the method's (or transaction's)
execution. Event values can be used to index transactions and blocks according to what
happened during their execution.
Each event has a type which is meant to categorize the event for a particular
Response* or Tx. A Response* or Tx may contain multiple events with duplicate
type values, where each distinct entry is meant to categorize attributes for a
particular event. Every key and value in an event's attributes must be UTF-8
encoded strings along with the event type itself.
message Event {
string type = 1;
repeated EventAttribute attributes = 2;
}
The attributes of an Event consist of a key, a value, and an index flag. The
index flag notifies the Tenderdash indexer to index the attribute. The value of
the index flag is non-deterministic and may vary across different nodes in the network.
message EventAttribute {
string key = 1;
string value = 2;
bool index = 3; // nondeterministic
}
Example:
abci.ExecTxResult{
// ...
Events: []abci.Event{
{
Type: "validator.provisions",
Attributes: []abci.EventAttribute{
abci.EventAttribute{Key: "address", Value: "...", Index: true},
abci.EventAttribute{Key: "amount", Value: "...", Index: true},
abci.EventAttribute{Key: "balance", Value: "...", Index: true},
},
},
{
Type: "validator.provisions",
Attributes: []abci.EventAttribute{
abci.EventAttribute{Key: "address", Value: "...", Index: true},
abci.EventAttribute{Key: "amount", Value: "...", Index: false},
abci.EventAttribute{Key: "balance", Value: "...", Index: false},
},
},
{
Type: "validator.slashed",
Attributes: []abci.EventAttribute{
abci.EventAttribute{Key: "address", Value: "...", Index: false},
abci.EventAttribute{Key: "amount", Value: "...", Index: true},
abci.EventAttribute{Key: "reason", Value: "...", Index: true},
},
},
// ...
},
}
Evidence
Tenderdash's security model relies on the use of "evidence". Evidence is proof of malicious behavior by a network participant. It is the responsibility of Tenderdash to detect such malicious behavior. When malicious behavior is detected, Tenderdash will gossip evidence of the behavior to other nodes and commit the evidence to the chain once it is verified by all validators. This evidence will then be passed on to the Application through ABCI. It is the responsibility of the Application to handle the evidence and exercise punishment.
EvidenceType has the following protobuf format:
enum EvidenceType {
UNKNOWN = 0;
DUPLICATE_VOTE = 1;
LIGHT_CLIENT_ATTACK = 2;
}
There are two forms of evidence: Duplicate Vote and Light Client Attack. More information can be found in either data structures or accountability
Client and Server
This section is for those looking to implement their own ABCI Server, perhaps in a new programming language.
You are expected to have read all previous sections of ABCI specification, namely Basic Concepts, Methods, Application Requirements, and Expected Behavior.
Message Protocol and Synchrony
The message protocol consists of pairs of requests and responses defined in the protobuf file.
Some messages have no fields, while others may include byte-arrays, strings, integers, or custom protobuf types.
For more details on protobuf, see the documentation.
Tenderdash creates one application client and delegates the request classes to
it. Ordinary socket and gRPC configurations do not create separate consensus,
mempool, query and state-sync connections automatically. A routed client can
use multiple underlying clients when explicitly configured.
Each synchronous call waits for its own response. This does not serialize calls from independent callers. Execution order and concurrency depend on the transport and server implementation; see Connections.
Server Implementations
To use ABCI in your programming language of choice, there must be an ABCI server in that language. Tenderdash provides three Go implementations:
- In-process
- ABCI-socket
- gRPC
See the server implementations and ABCI CLI guide. Out-of-process servers must implement Tenderdash's protobuf schema and application requirements.
In Process
The simplest implementation uses function calls in Golang. This means ABCI applications written in Golang can be linked with Tenderdash and run as a single binary. The local client delegates directly to application methods without a shared execution lock, so the application must synchronize concurrent access to its state.
GRPC
If you are not using Golang, but GRPC is available in your language, this is the easiest approach, though it will have significant performance overhead.
Please check GRPC's documentation to know to set up the Application as an ABCI GRPC server.
The client multiplexes calls over one grpc.ClientConn; application methods can
execute concurrently. The abci.grpc-concurrency configuration limits streams
per method or through the default * rule. Its default is {"*": 0}, which
imposes no limit. A limit for one method does not serialize it with methods
using a different limit. Protect state shared by concurrent calls.
Socket
Tenderdash's socket-based ABCI interface is an asynchronous,
raw socket server which provides ordered message passing over unix or tcp.
Messages are serialized using Protobuf3. Each message is prefixed by its byte
length encoded as an unsigned protobuf varint. For example, a four-byte message
DEADBEEF is sent as 04DEADBEEF. The receiver reads the varint length and then
that many message bytes. See the codec.
This framing does not apply to gRPC. The socket client sends requests and matches responses in order on its connection. Tenderdash's reference socket server executes requests sequentially per connection, while separate connections can execute concurrently. A compatible socket server must preserve response order; it must not assume a fixed number of role-specific connections.
Client
There are currently two use-cases for an ABCI client. One is testing
tools that allow ABCI requests to be sent to the actual application via
command line. An example of this is abci-cli, which accepts CLI commands
to send corresponding ABCI requests.
The other is a consensus engine, such as Tenderdash,
which makes ABCI requests to the application as prescribed by the consensus
algorithm used.
Methods
General, mempool and state-sync methods
Echo
- Request:
Message (string): A string to echo back
- Response:
Message (string): The input string
- Usage:
- Echo a string to test an abci client/server implementation
- From consensus version 2, the proposer of a round that waits for transactions probes
the backends that receive
PrepareProposalwithEchobefore each heartbeat, once it is not held back after block sync. At most one probe is in flight; an unanswered probe is reused by later heartbeats rather than repeated. A heartbeat is sent only if the backends answer within the heartbeat interval. See Proposer liveness.
Flush
- Usage:
- Signals that messages queued on the client should be flushed to the server. It is called periodically by the client implementation to ensure asynchronous requests are actually sent, and is called immediately to make a synchronous request, which returns when the Flush response comes back.
Info
-
Request:
Name Type Description Field Number version string The Tenderdash software semantic version 1 block_version uint64 The Tenderdash Block Protocol version 2 p2p_version uint64 The Tenderdash P2P Protocol version 3 abci_version string The Tenderdash ABCI semantic version 4 -
Response:
Name Type Description Field Number data string Some arbitrary information 1 version string The application software semantic version 2 app_version uint64 The application protocol version 3 last_block_height int64 Latest block finalized and persisted by the application 4 last_block_app_hash bytes Application hash of that finalized block 5 -
Usage:
-
Return information about the application state.
-
Used to sync Tenderdash with the application during a handshake that happens on startup.
-
During the startup handshake,
app_versioninitializes the local state's application version only if no block has been committed yet. It also supplies this node's advertised version, used asHeader.ProposedAppVersionwhen the node proposes a block. Another proposer may advertise a different version. -
Header.Version.Appis the version used for the proposed block's execution. A nonzeroResponsePrepareProposal.app_versionoverrides its inherited value. ChangingInfo.app_versionon an existing chain does not by itself change the committed execution version. -
last_block_app_hashandlast_block_heightdescribe the state persisted byFinalizeBlock; there is no ABCICommitmethod. See ABCI handshake for replay and crash-recovery behavior.
Note: Semantic version is a reference to semantic versioning. Semantic versions in info will be displayed as X.X.x.
InitChain
-
Request:
Name Type Description Field Number time google.protobuf.Timestamp Genesis time 1 chain_id string ID of the blockchain. 2 consensus_params ConsensusParams Initial consensus-critical parameters. 3 validator_set ValidatorSetUpdate Initial genesis validator set, sorted by voting power. 4 app_state_bytes bytes Serialized initial application state. JSON bytes. 5 initial_height int64 Height of the initial block (typically 1).6 initial_core_height uint32 Height of the initial core chainlock. 7 -
Response:
Name Type Description Field Number consensus_params ConsensusParams Initial consensus-critical parameters (optional). 1 app_hash bytes Initial application hash, 32 bytes. 2 validator_set_update ValidatorSetUpdate Initial validator set update. 3 next_core_chain_lock_update CoreChainLock Initial core chain lock update (optional). 4 initial_core_height uint32 Initial core chain lock height. 5 genesis_time google.protobuf.Timestamp Genesis time override (optional). 6 -
Usage:
-
Called once upon genesis.
-
If
ResponseInitChain.validator_set_update.validator_updatesis empty, genesis validator membership is retained; quorum-key and quorum-hash updates can still be supplied invalidator_set_update. -
If it is not empty, those validators replace the initial membership supplied through
RequestInitChain.validator_set. -
This allows the app to decide if it wants to accept the initial validator set proposed by Tenderdash (ie. in the genesis file), or if it wants to use a different one (perhaps computed based on some application specific information in the genesis file).
Query
-
Request:
Name Type Description Field Number data bytes Raw query bytes. Can be used with or in lieu of Path. 1 path string Path field of the request URI. Can be used with or in lieu of data. Apps MUST interpret/storeas a query by key on the underlying store. The key SHOULD be specified in thedatafield. Apps SHOULD allow queries over specific types like/accounts/...or/votes/...2 height int64 The block height for which you want the query (default=0 returns data for the latest committed block). Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height 3 prove bool Return Merkle proof with response if possible 4 -
Response:
Name Type Description Field Number code uint32 Response code. 1 log string The output of the application's logger. May be non-deterministic. 2 info string Additional information. May be non-deterministic. 3 index int64 The index of the key in the tree. 4 key bytes The key of the matching data. 5 value bytes The value of the matching data. 6 proof_ops ProofOps Serialized proof for the value data, if requested, to be verified against the app_hashfor the given Height.7 height int64 The block height from which data was derived. Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height 8 codespace string Namespace for the code.9 -
Usage:
-
Query for data from the application at current or past height.
-
Optionally return Merkle proof.
-
Merkle proof includes self-describing
typefield to support many types of Merkle trees and encoding formats.
CheckTx
-
Request:
Name Type Description Field Number tx bytes The request transaction bytes 1 type CheckTxType One of CheckTx_NeworCheckTx_Recheck.CheckTx_Newis the default and means that a full check of the tranasaction is required.CheckTx_Rechecktypes are used when the mempool is initiating a normal recheck of a transaction.2 -
Response:
Name Type Description Field Number code uint32 Response code. 1 data bytes Result bytes, if any. 2 info string Additional information. May be non-deterministic. 3 gas_wanted int64 Amount of gas requested for transaction. 4 codespace string Namespace for the code.5 sender string The transaction's sender (e.g. the signer) 6 priority int64 The transaction's priority (for mempool ordering) 7 -
Usage:
-
Technically optional - not involved in processing blocks.
-
Guardian of the mempool: every node runs
CheckTxbefore letting a transaction into its local mempool. -
The transaction may come from an external user or another node
-
CheckTxvalidates the transaction against the current state of the application, for example, checking signatures and account balances, but does not apply any of the state changes described in the transaction. -
Transactions where
ResponseCheckTx.Code != 0will be rejected - they will not be broadcast to other nodes or included in a proposal block. -
Tenderdash attributes no other value to the response code
ListSnapshots
-
Request:
Name Type Description Field Number Empty request asking the application for a list of snapshots.
-
Response:
Name Type Description Field Number snapshots repeated Snapshot List of local state snapshots. 1 -
Usage:
-
Used during state sync to discover available snapshots on peers.
-
See
Snapshotdata type for details.
LoadSnapshotChunk
-
Request:
Name Type Description Field Number height uint64 The height of the snapshot the chunk belongs to. 1 version uint32 The application-specific snapshot version. 2 chunk_id bytes Application-defined identifier of the requested chunk. 3 -
Response:
Name Type Description Field Number chunk bytes The binary chunk contents, in an arbitray format. Chunk messages cannot be larger than 16 MB including metadata, so 10 MB is a good starting point. 1 -
Usage:
-
Used during state sync to retrieve snapshot chunks from peers.
-
Chunk identifiers are bytes, not numeric indexes. The initial identifier is
Snapshot.hash; the application supplies further identifiers throughResponseApplySnapshotChunk.next_chunks.
OfferSnapshot
-
Request:
Name Type Description Field Number snapshot Snapshot The snapshot offered for restoration. 1 app_hash bytes The light client-verified app hash for this height, from the blockchain. 2 -
Response:
Name Type Description Field Number result Result The result of the snapshot offer. 1
Result
enum Result {
UNKNOWN = 0; // Unknown result, abort all snapshot restoration
ACCEPT = 1; // Snapshot is accepted, start applying chunks.
ABORT = 2; // Abort snapshot restoration, and don't try any other snapshots.
REJECT = 3; // Reject this specific snapshot, try others.
REJECT_FORMAT = 4; // Reject all snapshots with this version, try others.
REJECT_SENDER = 5; // Reject all snapshots from all senders of this snapshot, try others.
}
- Usage:
OfferSnapshotis called when bootstrapping a node using state sync. The application may accept or reject snapshots as appropriate. Upon accepting, Tenderdash will retrieve and apply snapshot chunks viaApplySnapshotChunk. The application may also choose to reject a snapshot in the chunk response, in which case it should be prepared to accept furtherOfferSnapshotcalls.- Only
AppHashcan be trusted, as it has been verified by the light client. Any other data can be spoofed by adversaries, so applications should employ additional verification schemes to avoid denial-of-service attacks. The verifiedAppHashis automatically checked against the restored application at the end of snapshot restoration. - For more information, see the
Snapshotdata type or the state sync section.
ApplySnapshotChunk
-
Request:
Name Type Description Field Number chunk_id bytes Application-defined identifier of this chunk. 1 chunk bytes The binary chunk contents, as returned by LoadSnapshotChunk.2 sender string The P2P ID of the node who sent this chunk. 3 -
Response:
Name Type Description Field Number result Result (see below) The result of applying this chunk. 1 refetch_chunks repeated bytes Identifiers of chunks to discard, refetch and reapply, regardless of result.2 reject_senders repeated string Reject the given P2P senders, regardless of Result. Any chunks already applied will not be refetched unless explicitly requested, but queued chunks from these senders will be discarded, and new chunks or other snapshots rejected.3 next_chunks repeated bytes Chunk identifiers to enqueue after ACCEPT; ignored for other results.4
enum Result {
UNKNOWN = 0; // Unknown result, abort all snapshot restoration
ACCEPT = 1; // The chunk was accepted.
ABORT = 2; // Abort snapshot restoration, and don't try any other snapshots.
RETRY = 3; // Reapply this chunk, combine with `RefetchChunks` and `RejectSenders` as appropriate.
RETRY_SNAPSHOT = 4; // Restart this snapshot from `OfferSnapshot`, reusing chunks unless instructed otherwise.
REJECT_SNAPSHOT = 5; // Reject this snapshot, try a different one.
COMPLETE_SNAPSHOT = 6; // Accept this chunk and finish restoring the snapshot.
}
- Usage:
- The application can choose to refetch chunks and/or ban P2P peers as appropriate. Tenderdash will not do this unless instructed by the application.
- The application may want to verify each chunk, e.g. by attaching chunk hashes in
Snapshot.Metadataand/or incrementally verifying contents againstAppHash. - On
ACCEPT, Tenderdash enqueuesnext_chunksand continues applying chunks. Fetching can run in parallel; identifiers do not impose a numeric application order. The application must discover and validate the required chunks. - Return
COMPLETE_SNAPSHOTonly when restoration is complete. AnACCEPTwith nonext_chunks, or an empty request queue, does not signal completion. - After
COMPLETE_SNAPSHOT, Tenderdash callsInfoand checkslast_block_app_hashagainst the light-client-verified hash,last_block_heightagainst the snapshot height, andapp_versionagainst the version in the reconstructed consensus state. Failed verification prevents successful state sync. The node can then catch up through block sync or enter consensus. - If Tenderdash is unable to retrieve the next chunk after some time (e.g. because no suitable
peers are available), it will reject the snapshot and try a different one via
OfferSnapshot. The application should be prepared to reset and accept it or abort as appropriate.
Consensus methods
PrepareProposal
Parameters and Types
| Method Name | Request Type | Response Type | Description |
|---|---|---|---|
| PrepareProposal | RequestPrepareProposal | ResponsePrepareProposal |
Usage
- Use the linked request/response schemas for field names and numbers; proposal preparation, processing and finalization have different request shapes.
- The height and time values match the values from the header of the proposed block.
RequestPrepareProposalcontains a preliminary set of transactionstxsthat Tenderdash considers to be a good block proposal, called raw proposal. The Application can modify this set viaResponsePrepareProposal.tx_records(see TxRecord).- The Application can reorder, remove or add transactions to the raw proposal. Let
txbe a transaction intxs:- If the Application considers that
txshould not be proposed in this block, e.g., there are other transactions with higher priority, then it should not include it intx_records. In this case, Tenderdash won't removetxfrom the mempool. The Application should be extra-careful, as abusing this feature may cause transactions to stay forever in the mempool. - If the Application considers that a
txshould not be included in the proposal and removed from the mempool, then the Application should include it intx_recordsand mark it asREMOVED. In this case, Tenderdash will removetxfrom the mempool. - If the Application wants to add a new transaction, then the Application should include it in
tx_recordsand mark it asADDED. Tenderdash includes it in the proposal without adding it to the mempool.
- If the Application considers that
- The Application should be aware that removing and adding transactions may compromise traceability.
Consider the following example: the Application transforms a client-submitted transaction
t1into a second transactiont2, i.e., the Application asks Tenderdash to removet1from the mempool and includet2in the proposal. If a client wants to eventually check what happened tot1, it will discover thatt_1is not in the mempool or in a committed block, getting the wrong idea thatt_1did not make it into a block. Note thatt_2will be in a committed block, but unless the Application tracks this information, no component will be aware of it. Thus, if the Application wants traceability, it is its responsability to support it. For instance, the Application could attach to a transformed transaction a list with the hashes of the transactions it derives from.
- The Application can reorder, remove or add transactions to the raw proposal. Let
- Tenderdash MAY include a list of transactions in
RequestPrepareProposal.txswhose total size in bytes exceedsRequestPrepareProposal.max_tx_bytes. Therefore, if the size ofRequestPrepareProposal.txsis greater thanRequestPrepareProposal.max_tx_bytes, the Application MUST make sure that theRequestPrepareProposal.max_tx_byteslimit is respected by those transaction records returned inResponsePrepareProposal.tx_recordsthat are marked asUNMODIFIEDorADDED. - Tenderdash uses same-block execution. The Application must provide values for
ResponsePrepareProposal.app_hash,ResponsePrepareProposal.tx_results,ResponsePrepareProposal.validator_set_update,ResponsePrepareProposal.core_chain_lock_updateandResponsePrepareProposal.consensus_param_updates, as a result of fully executing the block.- The values for
ResponsePrepareProposal.validator_set_update,ResponsePrepareProposal.core_chain_lock_updateorResponsePrepareProposal.consensus_param_updatesmay be empty. In this case, Tenderdash will keep the current values. - Validator changes from block
Hare committed byNextValidatorsHashand become active atH+1. See Validator-set and parameter changes. - Consensus parameter changes from
PrepareProposal/ProcessProposalfor blockHtake effect atH+1;ResponseFinalizeBlockhas no parameter-update field. See Validator-set and parameter changes. - It is the responsibility of the Application to set the right value for TimeoutPropose so that
the (synchronous) execution of the block does not cause other processes to prevote
nilbecause their propose timeout goes off.
- The values for
- Block and transaction events are returned with
ProcessProposal, notResponseFinalizeBlock; see Events. - As a sanity check, Tenderdash will check the returned parameters for validity if the Application modified them.
In particular,
ResponsePrepareProposal.tx_recordswill be deemed invalid if- There is a duplicate transaction in the list.
- A new or modified transaction is marked as
UNMODIFIEDorREMOVED. - An unmodified transaction is marked as
ADDED. - A transaction is marked as
UNKNOWN.
ResponsePrepareProposal.tx_resultscontains only results ofUNMODIFIEDandADDEDtransactions.REMOVEDtransactions are omitted. The length oftx_resultscan be different than the length oftx_records.- If Tenderdash fails to validate the
ResponsePrepareProposal, Tenderdash will assume the application is faulty and crash. - The implementation of
PrepareProposalcan be non-deterministic.
When does Tenderdash call it?
Tenderdash calls PrepareProposal when creating a fresh proposal, after selecting
transactions within the block's byte/gas budget. A valid block is reused only when
checkValidBlock permits it; otherwise a fresh block is prepared even if a
ValidBlock is recorded. The exact transaction-space calculation, response
validation and reuse rules are in
Creating the proposal.
ProcessProposal
Parameters and Types
| Method Name | Request Type | Response Type | Description |
|---|---|---|---|
| ProcessProposal | RequestProcessProposal | ResponseProcessProposal |
Usage
- Contains fields from the proposed block.
- The Application may fully execute the block as though it was handling
RequestFinalizeBlock. However, any resulting state changes must be kept as candidate state, and the Application should be ready to backtrack/discard it in case the decided block is different.
- The Application may fully execute the block as though it was handling
- The height and timestamp values match the values from the header of the proposed block.
- If
ResponseProcessProposal.statusisREJECT, Tenderdash assumes the proposal received is not valid. - Tenderdash requires the current block's execution commitments. The Application must provide values
for parameters
ResponseProcessProposal.app_hash,ResponseProcessProposal.tx_results,ResponseProcessProposal.validator_set_update, andResponseProcessProposal.consensus_param_updates, so that Tenderdash can then verify the hashes in the block's header are correct. If the hashes mismatch, Tenderdash will reject the block even ifResponseProcessProposal.statuswas set toACCEPT. - The implementation of
ProcessProposalMUST be deterministic. Moreover, the value ofResponseProcessProposal.statusMUST exclusively depend on the parameters passed in the call toRequestProcessProposal, and the last committed Application state (see Requirements section). - Tenderdash may call
ProcessProposalmore than once for the same height and round with the same block, for example after the application rejected a commit's vote extensions or when block sync hands over to consensus, and again for a round after processing another round, for example to verify a commit of that round. The Application MUST treat such a call as re-execution that replaces its execution context for the height;FinalizeBlockrefers to the round processed last. - Moreover, application implementors SHOULD always set
ResponseProcessProposal.statustoACCEPT, unless they really know what the potential liveness implications of returningREJECTare.
When does Tenderdash call it?
ensureProcess processes the held block for the round being checked unless the
same block/round result is already current. It runs before the prevote lock rules,
so a locked validator can process another block and still prevote nil. A block
that does not validate against Tenderdash's committed state (for example a wrong
height, chain ID or last block ID) is never sent to the application and produces
a nil prevote, as do a well-formed application rejection and a block/header
mismatch with the application's answer;
unknown status, malformed results and transport errors are fatal on this path.
See the exact prevote decision.
Processing also occurs for commit application and can repeat after another round changed the execution context; see Commit handling and expected call sequences.
ExtendVote
Parameters and Types
-
Request:
Name Type Description Field Number hash bytes The header hash of the proposed block that the vote extensions is to refer to. 1 height int64 Height of the proposed block (for sanity check). 2 round int32 Round for the block and its application execution context. 3 -
Response:
Name Type Description Field Number vote_extensions ExtendVoteExtension Optional information signed by Tenderdash. 1 -
Usage:
-
ResponseExtendVote.vote_extensionsis optional information that, if present, will be signed by Tenderdash and attached to the Precommit message. -
RequestExtendVote.hashcorresponds to the hash of a proposed block that was made available to the application in a previous call toProcessProposalorPrepareProposalfor the current height. -
ResponseExtendVote.vote_extensionswill only be attached to a non-nilPrecommit message. If Tenderdash is to precommitnil, it will not callRequestExtendVote. -
The Application logic that creates extensions that are not threshold-recoverable can be non-deterministic. Threshold-recoverable extensions (
THRESHOLD_RECOVERandTHRESHOLD_RECOVER_RAW) are recovered into the commit, so they MUST be deterministic. -
ExtendVoteMUST be free of side effects, and the threshold-recoverable subset of its response MUST be exactly the vector the Application accepts in a commit-levelVerifyVoteExtensionfor the same block, height and round. -
Tenderdash may also call
ExtendVoteon any node, a non-validator included, after a commit-levelVerifyVoteExtensionrejection that followsProcessProposalof that block and round, to learn the vector a commit of that block and round must carry; see VerifyVoteExtension. An ABCI error or an invalid response from this call is fatal, as on the Precommit path.
When does Tenderdash call it?
The normal call is inside signing a non-nil precommit, after the block was
processed for that round. The proposal, lock and polka requirements are in the
precommit decision.
Nil precommits do not call ExtendVote for signing.
After rejecting a commit's extensions, Tenderdash can also call ExtendVote on
any node to learn the expected recoverable vector for the processed block and
commit round. That response is neither signed nor broadcast; see
Parked commits and replacement candidates.
VerifyVoteExtension
Tenderdash calls VerifyVoteExtension for two kinds of request, told apart by
validator_pro_tx_hash:
- Precommit verification (
validator_pro_tx_hashset): the vote extensions of one validator's Precommit, exactly as its Application returned them fromExtendVote. - Commit verification (
validator_pro_tx_hashempty): the extension vector of a commit, sent afterProcessProposalof the committed block with the samehash,heightandround(the commit round). The vector holds only the threshold-recoverable extensions (THRESHOLD_RECOVERandTHRESHOLD_RECOVER_RAW), in the orderExtendVotereturned them; extensions of any other type never appear in a commit. It can be empty.
Parameters and Types
-
Request:
Name Type Description Field Number hash bytes The header hash of the propsed block that the vote extensions refers to. 1 validator_pro_tx_hash bytes ProTxHash of the validator that signed the extensions; empty for commit verification 2 height int64 Height of the block (for sanity check). 3 round int32 Round of the vote; for commit verification, the commit round. 4 vote_extensions repeated ExtendVoteExtension Application-specific information signed by Tenderdash. Can have 0 length 5 -
Response:
Name Type Description Field Number status VerifyStatus enumsignaling if the application accepts the vote extension1 -
Usage:
-
RequestVerifyVoteExtension.vote_extensionscan be empty. For a Precommit, the Application's interpretation of it should be that the Application running at the process that sent the vote chose not to extend it. Tenderdash will always callRequestVerifyVoteExtension, even for 0 length vote extensions. -
For a Precommit, if
ResponseVerifyVoteExtension.statusisREJECT, Tenderdash will reject the whole received vote. See the Requirements section to understand the potential liveness implications of this. -
For a commit, the Application MUST compare the vector with the threshold-recoverable subset of the extensions it expects for that block, in order, and return
REJECTif they differ: a commit whose block signature is valid can still carry a stripped, duplicated or replayed vector. Tenderdash requiresACCEPTbefore it saves the block and commit or callsFinalizeBlock. WhatREJECTdoes depends on the path:- consensus: the commit is discarded without blaming its sender. A commit whose block the node
already holds is checked first and never changes the round. A commit for a later round whose
block has not arrived moves the node to that round on its threshold signature alone; it is
checked when the block arrives, then the commits other peers sent meanwhile. Copies of a
commit already waiting are dropped. If the node's own commit is rejected, nothing is
persisted and the node waits for a peer's commit. The block is processed again only for a
commit of a round other than the one processed last. After a rejection, Tenderdash calls
ExtendVotefor the block and round, and refuses, without processing the block again or calling the Application, a later commit of that round that would need the block processed again, when its vector differs from the one returned or equals it and was rejected. Consensus WAL replay follows these rules; - block sync: the block is not applied; the peer that served it is dropped and the height is requested again;
- handshake catch-up (application behind the block store): the commit is already in the block store, so the node fails to start. There is no automatic recovery; the operator has to roll back or re-sync the node.
- consensus: the commit is discarded without blaming its sender. A commit whose block the node
already holds is checked first and never changes the round. A commit for a later round whose
block has not arrived moves the node to that round on its threshold signature alone; it is
checked when the block arrives, then the commits other peers sent meanwhile. Copies of a
commit already waiting are dropped. If the node's own commit is rejected, nothing is
persisted and the node waits for a peer's commit. The block is processed again only for a
commit of a round other than the one processed last. After a rejection, Tenderdash calls
-
Any status other than
ACCEPT, includingUNKNOWN, counts asREJECT. An error from the ABCI call itself, as opposed to a status, is fatal on every path. -
The implementation of
VerifyVoteExtensionMUST be deterministic. Moreover, the value ofResponseVerifyVoteExtension.statusMUST exclusively depend on the parameters passed in the call toRequestVerifyVoteExtension, the block processed byProcessProposal, and the last committed Application state (see Requirements section). -
For a Precommit, application implementers SHOULD always set
ResponseVerifyVoteExtension.statustoACCEPT, unless they really know what the potential liveness implications of returningREJECTare.
When does Tenderdash call it?
For a peer's non-nil precommit, Tenderdash verifies its block and extension signatures, restores its own processed round if needed, then calls the Application with the validator's pro-tx-hash. A rejected vote is not stored and may be redelivered. Own votes skip this Application check; see Vote admission.
For a commit, Tenderdash processes the block for commit.Round and verifies the
recovered extension vector with an empty validator pro-tx-hash. Live consensus
requires acceptance before saving/finalizing the block. Replacement candidates and
stored-block replay have different retry behavior, as described in the usage
contract above and Commit handling.
FinalizeBlock
Parameters and Types
| Method Name | Request Type | Response Type | Description |
|---|---|---|---|
| FinalizeBlock | RequestFinalizeBlock | ResponseFinalizeBlock |
Usage
- Contains the fields of the newly decided block.
- The height and timestamp values match the values from the header of the proposed block.
RequestFinalizeBlock.commitandRequestFinalizeBlock.misbehaviorcarry commit and misbehavior information; see the linked request schema.- The Application persists the state corresponding to
RequestFinalizeBlock.blockand the round processed last, whose execution commitments were already supplied throughPrepareProposal/ProcessProposal. It may reuse that candidate rather than execute the block again. - Transaction results and events are returned during
ProcessProposal;ResponseFinalizeBlockreturns onlyretain_heightandpropose_next_block_immediately, with no application hash or events. - Application is expected to persist its state at the end of this call, before calling
ResponseFinalizeBlock. - Later calls to
Querycan return proofs about the application state anchored in this Merkle root hash. - Use
ResponseFinalizeBlock.retain_heightwith caution! If all nodes in the network remove historical blocks then this data is permanently lost, and no new nodes will be able to join the network and bootstrap. Historical blocks may also be required for other purposes, e.g. auditing, replay of non-persisted heights, light client verification, and so on. - Set
ResponseFinalizeBlock.propose_next_block_immediatelywhen the Application has block-driven work pending that the next block would carry out, for example withdrawal transactions waiting to be signed. The node then does not wait for transactions (create-empty-blocks-interval) before proposing at the next height, in round 0 or, on consensus version 2, in a round entered after a silent proposer's round was skipped. This is a hint to the local node, not part of consensus: it is not compared between validators, it only matters on the node that proposes the next height, and it is consumed once; a node that restarts in between falls back to the configured wait. - Just as
ProcessProposal, the implementation ofFinalizeBlockMUST be deterministic, since it is making the Application's state evolve in the context of state machine replication. - Currently, Tenderdash will fill up all fields in
RequestFinalizeBlock, even if they were already passed on to the Application viaRequestPrepareProposalorRequestProcessProposal. The Application must select the candidate matching the finalized block and the round processed last; a block hash alone does not identify a round's execution context. See Commit validity depends on the application.
When does Tenderdash call it?
Tenderdash finalizes a held block after validating its complete block ID, processing it for the commit round, and accepting the recovered commit's extensions. The commit can come from local precommits or a peer; it does not require receipt of every original vote or proposal. See Commit handling and the exact persistence order in Commit and apply.
Shared data types
Most of the data structures used in ABCI are shared common data structures. In certain cases, ABCI uses different data structures which are documented here:
Validator
See the generated Validator schema.
- Usage:
- Validator identified by ProTxHash
- Used by
Misbehavior.validatorto identify an offending validator. - Public keys are carried separately by validator-set update messages.
ValidatorUpdate
See the generated ValidatorUpdate schema.
- Usage:
- Validator identified by ProTxHash
- Used to tell Tenderdash to update the validator set
ValidatorSetUpdate
See the generated ValidatorSetUpdate schema.
- Usage:
- Used to tell Tenderdash to update the validator set update
Misbehavior
See the generated Misbehavior schema.
ConsensusParams
-
Fields:
Name Type Description Field Number block BlockParams Parameters limiting the size of a block and time between consecutive blocks. 1 evidence EvidenceParams Parameters limiting the validity of evidence of byzantine behaviour. 2 validator ValidatorParams Parameters limiting the types of public keys validators can use. 3 version VersionsParams The ABCI application version. 4
ProofOps
-
Fields:
Name Type Description Field Number ops repeated ProofOp List of chained Merkle proofs, of possibly different types. The Merkle root of one op is the value being proven in the next op. The Merkle root of the final op should equal the ultimate root hash being verified against.. 1
ProofOp
-
Fields:
Name Type Description Field Number type string Type of Merkle proof and how it's encoded. 1 key bytes Key in the Merkle tree that this proof is for. 2 data bytes Encoded Merkle proof for the key. 3
Snapshot
See the generated Snapshot schema.
- Usage:
- Used for state sync snapshots, see the state sync section for details.
- A snapshot is considered identical across nodes only if all fields are equal (including
Metadata). Chunks may be retrieved from all nodes that have the same snapshot. - When sent across the network, a snapshot message can be at most 4 MB.
Execution and consensus data types
VoteInfo
See the generated VoteInfo schema.
- Usage:
- Indicates whether a validator signed the last block, allowing for rewards based on validator availability.
- This information is typically extracted from a proposed or decided block.
ExtendVoteExtension
See the generated ExtendVoteExtension schema.
- Usage:
- Provides a vote extension for signing
- Each field is mandatory for filling
- Type can be
THRESHOLD_RECOVER,THRESHOLD_RECOVER_RAWorDEFAULT DEFAULTis not supported so farTHRESHOLD_RECOVERandTHRESHOLD_RECOVER_RAWare deterministic, that means, each validator in a quorum must provide the same vote extension data
VoteExtensionType
enum VoteExtensionType {
DEFAULT = 0;
THRESHOLD_RECOVER = 1;
THRESHOLD_RECOVER_RAW = 2; // signs the raw extension; allows overriding the sign request ID
}
CommitInfo
See the generated CommitInfo schema.
ExecTxResult
See the generated ExecTxResult schema.
TxAction
enum TxAction {
UNKNOWN = 0; // Unknown action
UNMODIFIED = 1; // The Application did not modify this transaction.
ADDED = 2; // The Application added this transaction.
REMOVED = 3; // The Application wants this transaction removed from the proposal and the mempool.
DELAYED = 4; // Exclude from this proposal, but keep in the mempool.
}
-
Usage:
-
If
ActionisUNKNOWN, a problem happened in the Application. Tenderdash will assume the application is faulty and crash. -
If
ActionisUNMODIFIED, Tenderdash includes the transaction in the proposal. Nothing to do on the mempool. -
If
ActionisADDED, Tenderdash includes the transaction in the proposal. The transaction is not added to the mempool. -
If
ActionisREMOVED, Tenderdash excludes the transaction from the proposal. The transaction is also removed from the mempool if it exists, similar toCheckTxreturning false. -
If
ActionisDELAYED, Tenderdash excludes the transaction from the proposal and keeps it in the mempool.
TxRecord
See the generated TxRecord schema.
ProposalStatus
enum ProposalStatus {
UNKNOWN = 0; // Unknown status. Returning this from the application is always an error.
ACCEPT = 1; // Status that signals that the application finds the proposal valid.
REJECT = 2; // Status that signals that the application finds the proposal invalid.
}
- Usage:
- Used within the ProcessProposal response.
- If
StatusisUNKNOWN, a problem happened in the Application. Tenderdash will assume the application is faulty and crash. - If
StatusisACCEPT, the application finds the proposal valid. Tenderdash still validates the returned execution results, which can reject the block. On the prevote path, the lock and POL rules then decide between a prevote for the block and a nil prevote. - If
StatusisREJECT, the application finds the proposal invalid. On the prevote path, Tenderdash prevotes nil; during commit processing or replay, the call fails and the block is not applied. See ProcessProposal for each call path.
- If
VerifyStatus
enum VerifyStatus {
UNKNOWN = 0; // Unknown status. Returning this from the application is always an error.
ACCEPT = 1; // Status that signals that the application finds the vote extension valid.
REJECT = 2; // Status that signals that the application finds the vote extension invalid.
}
- Usage:
- Used within the VerifyVoteExtension response, for a Precommit or a commit.
- If
StatusisACCEPT, Tenderdash will accept the vote, or the commit, as valid. - If
StatusisREJECTorUNKNOWN, Tenderdash will reject the vote, or the commit, as invalid. Only an error from the ABCI call itself is fatal.
- If
CanonicalVoteExtension
This signing structure belongs to the consensus vote-extension protocol, rather than the ABCI application interface. See vote-extension signing.
-
Fields:
Name Type Description Field Number extension bytes Vote extension provided by the Application. 1 height sfixed64 Height in which the extension was provided. 2 round sfixed64 Round in which the extension was provided. 3 chain_id string ID of the blockchain running consensus. 4 type VoteExtensionType VoteExtensionType 5 -
Usage:
-
Tenderdash is to sign the whole data structure and attach it to a Precommit message
-
Upon reception, Tenderdash validates the sender's signature and sanity-checks the values of
height,round, andchain_id. Then it sendsextensionto the Application viaRequestVerifyVoteExtensionfor verification.
Tenderdash's expected behavior
Valid method call sequences
Tenderdash initiates ABCI requests and the Application responds. The consensus protocol specifies the implemented order, including guards, quorum/recovery thresholds and failure paths. Its system model states the safety assumptions; a configurable threshold must not be assumed to satisfy the strict BFT floor in every supported configuration.
The primary flow describes a height that succeeds in round zero. Its ordinary application calls are:
| Role or purpose | Calls |
|---|---|
| Proposer creating a fresh block | PrepareProposal supplies transactions, same-block application/results hashes and next-state changes. A reused valid block does not require another PrepareProposal. |
| Processing before voting or commit application | ProcessProposal for the block and the round being checked, unless that exact result is already current. |
| Validator signing a non-nil precommit | ExtendVote after processing for that round. Nil precommits and prevotes carry no extensions. |
| Peer non-nil precommit | VerifyVoteExtension with the validator's pro-tx-hash, after signature verification. |
| Commit, on any full node | VerifyVoteExtension with an empty validator pro-tx-hash, after processing for the commit round. |
| Accepted decided block | FinalizeBlock persists the candidate corresponding to that block and round. |
These are not exact per-height callback counts. A node can finish on a commit
without receiving every individual precommit. ProcessProposal can run again for
the same block/round after a different round has replaced the Application's
execution context. Commit checks can process a foreign round and then restore the
node's own round before checking an individual precommit. See
Commit handling and
ABCI calls per height.
After commit-level VerifyVoteExtension rejects a vector, Tenderdash can call
ExtendVote for that processed block and commit round to learn the expected
vector. This call can occur on an observer and does not sign or broadcast a vote.
A replacement commit can require further verification and processing. Rejection of
a held-block commit does not itself advance the round; a future-round commit whose
block is absent can already advance the round before application validation.
The full branch rules are in
Commits received from peers
and the VerifyVoteExtension contract.
An Application must handle these repeated calls deterministically for the block,
round, and last committed state. It must not treat an abbreviated normal-round
sequence as an at-most-once guarantee. A successfully completed live height calls
FinalizeBlock only after accepting the commit vector. Rejection while replaying
an already-stored commit during startup instead terminates replay before
FinalizeBlock; see Recovery and synchronisation.
Startup and methods outside block execution
Startup uses Info to reconcile application and Tenderdash state and InitChain
when the Application is at height zero; the exact cases, replay calls and crash
windows are in ABCI handshake.
Snapshot restoration can attempt OfferSnapshot/ApplySnapshotChunk before a
successful restore and Info; these are synchronization boundaries rather than
ordinary voting rounds.
CheckTx controls mempool admission separately from the block execution sequence.
Query, ListSnapshots and LoadSnapshotChunk read application state or existing
snapshots. Echo and Flush serve client/debugging purposes; from consensus version 2 a
waiting proposer also uses Echo as a readiness probe
(Echo). Info can also serve
RPC queries independently of startup. These methods are not bounded by the normal
height's consensus callback count.
Application execution requirements
Tenderdash requires same-block execution: ResponsePrepareProposal and
ResponseProcessProposal supply the current block's application hash and
transaction results before it is decided. Returning only a boolean acceptance or
waiting until FinalizeBlock to compute those commitments is insufficient.
Use the current method schemas and requirements.
Keep speculative execution as candidate state; publish and persist it only when
FinalizeBlock selects that block and the round processed last. There is no ABCI
Commit method; persistence belongs in FinalizeBlock.
ResponseFinalizeBlock has only retain_height and
propose_next_block_immediately; it returns neither AppHash, transaction results,
nor events. Block/transaction events belong to ProcessProposal results; see
Events.
An Application returning no vote extensions must accept an empty expected commit vector and reject a nonempty one at commit-level verification. Other Applications must compare the ordered threshold-recoverable subset they expect for that block and round; independently valid signatures do not establish the vector's identity, count and order. See VerifyVoteExtension.
Protocol Documentation
Table of Contents
tendermint/abci/types.proto
CommitInfo
| Field | Type | Label | Description |
|---|---|---|---|
| round | int32 | ||
| quorum_hash | bytes | ||
| block_signature | bytes | ||
| threshold_vote_extensions | tendermint.types.VoteExtension | repeated |
Event
Event carries application attributes in ResponseProcessProposal and ExecTxResult.
Later, transactions may be queried using these events.
| Field | Type | Label | Description |
|---|---|---|---|
| type | string | ||
| attributes | EventAttribute | repeated |
EventAttribute
EventAttribute is a single key-value pair, associated with an event.
ExecTxResult
ExecTxResult contains results of executing one individual transaction.
| Field | Type | Label | Description |
|---|---|---|---|
| code | uint32 | Response code within codespace; by convention, 0 means success. | |
| data | bytes | Result bytes, if any (arbitrary data, not interpreted by Tenderdash). | |
| log | string | The output of the application's logger. May be non-deterministic. | |
| info | string | Additional information. May be non-deterministic. | |
| gas_used | int64 | Amount of gas consumed by transaction. | |
| events | Event | repeated | Type & Key-Value events for indexing transactions (e.g. by account). |
| codespace | string | Namespace for the code. |
ExtendVoteExtension
Provides a vote extension for signing. type and extension fields are mandatory for filling
| Field | Type | Label | Description |
|---|---|---|---|
| type | tendermint.types.VoteExtensionType | Vote extension type can be either DEFAULT, THRESHOLD_RECOVER or THRESHOLD_RECOVER_RAW. The Tenderdash supports only THRESHOLD_RECOVER and THRESHOLD_RECOVER_RAW at this moment. | |
| extension | bytes | Deterministic or (Non-Deterministic) extension provided by the sending validator's Application. |
For THRESHOLD_RECOVER_RAW, it MUST be 32 bytes.
Sign request ID that will be used to sign the vote extensions. Only applicable for THRESHOLD_RECOVER_RAW vote extension type.
Tenderdash will use SHA256 checksum of sign_request_id when generating quorum signatures of THRESHOLD_RECOVER_RAW vote extensions. It MUST NOT be set for any other vote extension types. |
| sign_request_id | bytes | optional | If not set, Tenderdash will generate it based on height and round.
If set, it SHOULD be unique per voting round, and it MUST start with dpevote or \x06plwdtx prefix.
Use with caution - it can have severe security consequences. |
ExtendedVoteInfo
ExtendedVoteInfo
| Field | Type | Label | Description |
|---|---|---|---|
| validator | Validator | The validator that sent the vote. | |
| signed_last_block | bool | Indicates whether the validator signed the last block, allowing for rewards based on validator availability. | |
| vote_extension | bytes | Non-deterministic extension provided by the sending validator's application. |
Misbehavior
| Field | Type | Label | Description |
|---|---|---|---|
| type | MisbehaviorType | ||
| validator | Validator | The offending validator | |
| height | int64 | The height when the offense occurred | |
| time | google.protobuf.Timestamp | The corresponding time where the offense occurred | |
| total_voting_power | int64 | Total voting power of the validator set in case the ABCI application does not store historical validators. https://github.com/tendermint/tendermint/issues/4581 |
QuorumHashUpdate
| Field | Type | Label | Description |
|---|---|---|---|
| quorum_hash | bytes |
Request
Request types
| Field | Type | Label | Description |
|---|---|---|---|
| echo | RequestEcho | ||
| flush | RequestFlush | ||
| info | RequestInfo | ||
| init_chain | RequestInitChain | ||
| query | RequestQuery | ||
| check_tx | RequestCheckTx | ||
| list_snapshots | RequestListSnapshots | ||
| offer_snapshot | RequestOfferSnapshot | ||
| load_snapshot_chunk | RequestLoadSnapshotChunk | ||
| apply_snapshot_chunk | RequestApplySnapshotChunk | ||
| prepare_proposal | RequestPrepareProposal | ||
| process_proposal | RequestProcessProposal | ||
| extend_vote | RequestExtendVote | ||
| verify_vote_extension | RequestVerifyVoteExtension | ||
| finalize_block | RequestFinalizeBlock |
RequestApplySnapshotChunk
Applies a snapshot chunk.
- The application can choose to refetch chunks and/or ban P2P peers as appropriate. Tenderdash will not do this unless instructed by the application.
- The application may want to verify each chunk, e.g. by attaching chunk hashes in
Snapshot.Metadataand/or incrementally verifying contents against AppHash. - On ACCEPT, Tenderdash enqueues
ResponseApplySnapshotChunk.next_chunksand continues applying chunks. Restoration finishes only when the application returns COMPLETE_SNAPSHOT; an ACCEPT withoutnext_chunksdoes not signal completion. - After COMPLETE_SNAPSHOT, Tenderdash makes an ABCI Info call to verify that LastBlockAppHash, LastBlockHeight and AppVersion match the expected values. It then switches to block sync or consensus and joins the network.
- If Tenderdash is unable to retrieve the next chunk after some time (e.g. because no suitable peers are available), it will reject the snapshot and try a different one via OfferSnapshot. The application should be prepared to reset and accept it or abort as appropriate.
| Field | Type | Label | Description |
|---|---|---|---|
| chunk_id | bytes | Application-defined chunk identifier: Snapshot.hash for the first chunk, then identifiers returned in ResponseApplySnapshotChunk.next_chunks. | |
| chunk | bytes | The binary chunk contents, as returned by LoadSnapshotChunk. | |
| sender | string | The P2P ID of the node who sent this chunk. |
RequestCheckTx
Check if transaction is valid.
- Technically optional - not involved in processing blocks.
- Guardian of the mempool: every node runs CheckTx before letting a transaction into its local mempool.
- The transaction may come from an external user or another node
- CheckTx validates the transaction against the current state of the application, for example, checking signatures and account balances, but does not apply any of the state changes described in the transaction.
- Transactions where ResponseCheckTx.Code != 0 will be rejected - they will not be broadcast to other nodes or included in a proposal block.
- Tenderdash attributes no other value to the response code
| Field | Type | Label | Description |
|---|---|---|---|
| tx | bytes | The request transaction bytes. | |
| type | CheckTxType | Type or transaction check to execute. |
RequestEcho
Echo a string to test an abci client/server implementation
| Field | Type | Label | Description |
|---|---|---|---|
| message | string | A string to echo back |
RequestExtendVote
Return the vote_extensions vector for the block and round processed last. Before a validator signs a non-nil precommit, Tenderdash requests the extensions to attach to that vote. It may also call ExtendVote on any node after rejecting a commit vector, to learn the expected vector without signing or broadcasting. The call must be free of side effects. Threshold-recoverable extensions must be deterministic and match the vector accepted at commit-level VerifyVoteExtension. See the ExtendVote contract.
| Field | Type | Label | Description |
|---|---|---|---|
| hash | bytes | The header hash of the proposed block that the vote extensions is to refer to. | |
| height | int64 | Height of the proposed block (for sanity check). | |
| round | int32 | Round number for the block. |
RequestFinalizeBlock
Persist the decided candidate for the block and the round processed last. Tenderdash calls this after processing and accepting the commit extension vector. The application must persist the candidate before returning; execution results and events are provided through proposal processing, not ResponseFinalizeBlock. The response contains retention and next-block scheduling hints only. Use retain_height with care: pruning blocks can prevent replay and bootstrapping. See the FinalizeBlock contract and commit and apply.
| Field | Type | Label | Description |
|---|---|---|---|
| commit | CommitInfo | Info about the current commit | |
| misbehavior | Misbehavior | repeated | List of information about validators that acted incorrectly. |
| hash | bytes | The block header's hash. Present for convenience (can be derived from the block header). | |
| height | int64 | The height of the finalized block. | |
| round | int32 | Round number for the block | |
| block | tendermint.types.Block | The block that was finalized | |
| block_id | tendermint.types.BlockID | The block ID that was finalized |
RequestFlush
Signals that messages queued on the client should be flushed to the server. It is called periodically by the client implementation to ensure asynchronous requests are actually sent, and is called immediately to make a synchronous request, which returns when the Flush response comes back.
RequestInfo
Return information about the application's last committed state. Used during startup reconciliation and by RPC queries. Persist the reported height and application hash in FinalizeBlock. See the startup handshake.
| Field | Type | Label | Description |
|---|---|---|---|
| version | string | The Tenderdash software semantic version. | |
| block_version | uint64 | The Tenderdash Block Protocol version. | |
| p2p_version | uint64 | The Tenderdash P2P Protocol version. | |
| abci_version | string | The Tenderdash ABCI semantic version. |
RequestInitChain
Called once upon genesis.
- If ResponseInitChain.Validators is empty, the initial validator set will be the RequestInitChain.Validators
- If ResponseInitChain.Validators is not empty, it will be the initial validator set (regardless of what is in RequestInitChain.Validators).
- This allows the app to decide if it wants to accept the initial validator set proposed by Tenderdash (ie. in the genesis file), or if it wants to use a different one (perhaps computed based on some application specific information in the genesis file).
| Field | Type | Label | Description |
|---|---|---|---|
| time | google.protobuf.Timestamp | Genesis time | |
| chain_id | string | ID of the blockchain. | |
| consensus_params | tendermint.types.ConsensusParams | Initial consensus-critical parameters. | |
| validator_set | ValidatorSetUpdate | Initial genesis validators, sorted by voting power. | |
| app_state_bytes | bytes | Serialized initial application state. JSON bytes. | |
| initial_height | int64 | Height of the initial block (typically 1). | |
| initial_core_height | uint32 | Initial core chain lock height. |
RequestListSnapshots
Lists available snapshots
- Used during state sync to discover available snapshots on peers.
- See Snapshot data type for details.
RequestLoadSnapshotChunk
Used during state sync to retrieve snapshot chunks from peers.
| Field | Type | Label | Description |
|---|---|---|---|
| height | uint64 | The height of the snapshot the chunks belongs to. | |
| version | uint32 | The application-specific format of the snapshot the chunk belongs to. | |
| chunk_id | bytes | Application-defined chunk identifier: Snapshot.hash for the first chunk, then identifiers returned in ResponseApplySnapshotChunk.next_chunks. |
RequestOfferSnapshot
Offers a snapshot to the application.
- OfferSnapshot is called when bootstrapping a node using state sync. The application may accept or reject snapshots as appropriate. Upon accepting, Tenderdash will retrieve and apply snapshot chunks via ApplySnapshotChunk. The application may also choose to reject a snapshot in the chunk response, in which case it should be prepared to accept further OfferSnapshot calls.
- Only AppHash can be trusted, as it has been verified by the light client. Any other data can be spoofed by adversaries, so applications should employ additional verification schemes to avoid denial-of-service attacks. The verified AppHash is automatically checked against the restored application at the end of snapshot restoration.
- For more information, see the Snapshot data type or the state sync section.
| Field | Type | Label | Description |
|---|---|---|---|
| snapshot | Snapshot | The snapshot offered for restoration. | |
| app_hash | bytes | The light client-verified app hash for this height, from the blockchain. 32 bytes. |
RequestPrepareProposal
Prepare and execute a new block proposal, potentially altering its transactions. Return same-block application/results hashes and next-state commitments. ADDED transactions enter the proposal without being added to the mempool; REMOVED transactions are removed from both. Changes to validators and consensus parameters from height H take effect at H+1. Preparation may be non-deterministic. See the PrepareProposal contract for transaction constraints, response validation and the exact conditions for preparing a fresh block.
| Field | Type | Label | Description |
|---|---|---|---|
| max_tx_bytes | int64 | Currently configured maximum size in bytes taken by the modified transactions. The modified transactions cannot exceed this size. | |
| txs | bytes | repeated | Preliminary list of transactions that have been picked as part of the block to propose. Sent to the app for possible modifications. |
| local_last_commit | CommitInfo | Info about the last commit, obtained locally from Tenderdash's data structures. | |
| misbehavior | Misbehavior | repeated | List of information about validators that acted incorrectly. |
| height | int64 | The height of the block that will be proposed. | |
| time | google.protobuf.Timestamp | Timestamp of the block that that will be proposed. | |
| next_validators_hash | bytes | Merkle root of the next validator set. | |
| round | int32 | Round number for the block. | |
| core_chain_locked_height | uint32 | Core chain lock height to be used when signing this block. | |
| proposer_pro_tx_hash | bytes | ProTxHash of the original proposer of the block. | |
| proposed_app_version | uint64 | Proposer's latest available app protocol version. | |
| version | tendermint.version.Consensus | App and block version used to generate the block. App version included in the block can be modified by setting ResponsePrepareProposal.app_version. | |
| quorum_hash | bytes | quorum_hash contains hash of validator quorum that will sign the block |
RequestProcessProposal
Execute a proposed block deterministically against the last committed state. Return its status, same-block application/results hashes and next-state commitments. Keep execution speculative until FinalizeBlock. Processing can repeat for the same block and round after another execution context has replaced it. See the ProcessProposal contract and application requirements.
| Field | Type | Label | Description |
|---|---|---|---|
| txs | bytes | repeated | List of transactions that have been picked as part of the proposed |
| proposed_last_commit | CommitInfo | Info about the last commit, obtained from the information in the proposed block. | |
| misbehavior | Misbehavior | repeated | List of information about validators that acted incorrectly. |
| hash | bytes | The block header's hash of the proposed block. It is computed as a Merkle tree from the header fields. | |
| height | int64 | The height of the proposed block. | |
| round | int32 | Round number for the block | |
| time | google.protobuf.Timestamp | Timestamp included in the proposed block. | |
| next_validators_hash | bytes | Merkle root of the next validator set. | |
| core_chain_locked_height | uint32 | Core chain lock height to be used when signing this block. | |
| core_chain_lock_update | tendermint.types.CoreChainLock | Next core-chain-lock-update for validation in ABCI. | |
| proposer_pro_tx_hash | bytes | ProTxHash of the original proposer of the block. | |
| proposed_app_version | uint64 | Proposer's latest available app protocol version. | |
| version | tendermint.version.Consensus | App and block version used to generate the block. App version MUST be verified by the app. | |
| quorum_hash | bytes | quorum_hash contains hash of validator quorum that will sign the block |
RequestQuery
Query for data from the application at current or past height.
- Optionally return Merkle proof.
- Merkle proof includes self-describing type field to support many types of Merkle trees and encoding formats.
| Field | Type | Label | Description |
|---|---|---|---|
| data | bytes | Raw query bytes. Can be used with or in lieu of Path. | |
| path | string | Path field of the request URI. Can be used with or in lieu of data. Apps MUST interpret /store as a query by key on the underlying store. The key SHOULD be specified in the data field. Apps SHOULD allow queries over specific types like /accounts/... or /votes/... | |
| height | int64 | The block height for which you want the query (default=0 returns data for the latest committed block). Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height-1. | |
| prove | bool | Return Merkle proof with response if possible. |
RequestVerifyVoteExtension
Verify the vote extension
Tenderdash calls it for two kinds of request, told apart by validator_pro_tx_hash:
- Precommit verification (
validator_pro_tx_hashset): the vote extensions of one validator's Precommit, exactly as its Application returned them fromExtendVote. - Commit verification (
validator_pro_tx_hashempty): the extension vector of a commit, afterProcessProposalof the committed block with the samehash,heightandround(the commit round). The vector holds only the threshold-recoverable extensions (THRESHOLD_RECOVERandTHRESHOLD_RECOVER_RAW), in the orderExtendVotereturned them; extensions of any other type never appear in a commit. It can be empty.
Usage
RequestVerifyVoteExtension.vote_extensionscan be empty. For a Precommit, the Application's interpretation of it should be that the Application running at the process that sent the vote chose not to extend it. Tenderdash will always callRequestVerifyVoteExtension, even for 0 length vote extensions.- For a Precommit, if
ResponseVerifyVoteExtension.statusisREJECT, Tenderdash will reject the whole received vote. See the Requirements section to understand the potential liveness implications of this. - For a commit, the Application MUST compare the vector with the threshold-recoverable subset of the
extensions it expects for that block, in order, and return
REJECTif they differ: a commit whose block signature is valid can still carry a stripped, duplicated or replayed vector. Tenderdash requiresACCEPTbefore it saves the block and commit or callsFinalizeBlock. WhatREJECTdoes depends on the path:- consensus: the commit is discarded without blaming its sender. A commit whose block the node
already holds is checked first and never changes the round. A commit for a later round whose
block has not arrived moves the node to that round on its threshold signature alone; it is
checked when the block arrives, then the commits other peers sent meanwhile. Copies of a commit
already waiting are dropped. If the node's own commit is rejected, nothing is persisted and the
node waits for a peer's commit. The block is processed again only for a commit of a round other
than the one processed last. After a rejection, Tenderdash calls
ExtendVotefor the block and round, and refuses, without processing the block again or calling the Application, a later commit of that round that would need the block processed again, when its vector differs from the one returned or equals it and was rejected. Consensus WAL replay follows these rules; - block sync: the block is not applied; the peer that served it is dropped and the height is requested again;
- handshake catch-up (application behind the block store): the commit is already in the block store, so the node fails to start; there is no automatic recovery, and the operator has to roll back or re-sync the node.
- consensus: the commit is discarded without blaming its sender. A commit whose block the node
already holds is checked first and never changes the round. A commit for a later round whose
block has not arrived moves the node to that round on its threshold signature alone; it is
checked when the block arrives, then the commits other peers sent meanwhile. Copies of a commit
already waiting are dropped. If the node's own commit is rejected, nothing is persisted and the
node waits for a peer's commit. The block is processed again only for a commit of a round other
than the one processed last. After a rejection, Tenderdash calls
- Any status other than
ACCEPT, includingUNKNOWN, counts asREJECT. An error from the ABCI call itself, as opposed to a status, is fatal on every path. - The implementation of
VerifyVoteExtensionMUST be deterministic. Moreover, the value ofResponseVerifyVoteExtension.statusMUST exclusively depend on the parameters passed in the call toRequestVerifyVoteExtension, the block processed byProcessProposal, and the last committed Application state (see Requirements section). - For a Precommit, application implementers SHOULD always set
ResponseVerifyVoteExtension.statustoACCEPT, unless they really know what the potential liveness implications of returningREJECTare.
When does Tenderdash call it?
When a validator p is in Tenderdash consensus height h and receives, from another validator q,
a Precommit message for a block (not nil) at height h:
- If the Precommit message does not contain a vote extensions with a valid signature, Tenderdash discards the message as invalid.
- a 0-length vote extensions is valid as long as its accompanying signature is also valid.
- Else, p's Tenderdash calls
RequestVerifyVoteExtension. - The Application returns accept or reject via
ResponseVerifyVoteExtension.status. - If the Application returns
- accept, p's Tenderdash will keep the received vote, together with its corresponding vote extension in its internal data structures. Threshold-recoverable extensions are recovered into CommitInfo when the commit is assembled.
- reject, p's Tenderdash will deem the Precommit message invalid and discard it.
Whenever a process is about to save a block and its commit, in consensus, block sync or replay, it
first calls RequestVerifyVoteExtension for the commit, once ProcessProposal has processed the
block. This includes commits the process assembled from its own +2/3 precommits.
| Field | Type | Label | Description |
|---|---|---|---|
| hash | bytes | The header hash of the propsed block that the vote extensions refers to. | |
| validator_pro_tx_hash | bytes | ProTxHash of the validator that signed the extensions; empty for commit verification. | |
| height | int64 | Height of the block (for sanity check). | |
| round | int32 | Round of the vote; for commit verification, the commit round. | |
| vote_extensions | ExtendVoteExtension | repeated | Application-specific information signed by Tenderdash. Can have 0 length. |
Response
| Field | Type | Label | Description |
|---|---|---|---|
| exception | ResponseException | ||
| echo | ResponseEcho | ||
| flush | ResponseFlush | ||
| info | ResponseInfo | ||
| init_chain | ResponseInitChain | ||
| query | ResponseQuery | ||
| check_tx | ResponseCheckTx | ||
| list_snapshots | ResponseListSnapshots | ||
| offer_snapshot | ResponseOfferSnapshot | ||
| load_snapshot_chunk | ResponseLoadSnapshotChunk | ||
| apply_snapshot_chunk | ResponseApplySnapshotChunk | ||
| prepare_proposal | ResponsePrepareProposal | ||
| process_proposal | ResponseProcessProposal | ||
| extend_vote | ResponseExtendVote | ||
| verify_vote_extension | ResponseVerifyVoteExtension | ||
| finalize_block | ResponseFinalizeBlock |
ResponseApplySnapshotChunk
| Field | Type | Label | Description |
|---|---|---|---|
| result | ResponseApplySnapshotChunk.Result | The result of applying this chunk. | |
| refetch_chunks | bytes | repeated | Identifiers of chunks to discard, refetch and reapply, regardless of result. |
| reject_senders | string | repeated | Reject the given P2P senders, regardless of Result. Any chunks already applied will not be refetched unless explicitly requested, but queued chunks from these senders will be discarded, and new chunks or other snapshots rejected. |
| next_chunks | bytes | repeated | Chunk identifiers to enqueue after ACCEPT; ignored for other results. |
ResponseCheckTx
| Field | Type | Label | Description |
|---|---|---|---|
| code | uint32 | Response code. | |
| data | bytes | Result bytes, if any. | |
| info | string | Additional information. May be non-deterministic. | |
| gas_wanted | int64 | Amount of gas requested for transaction. | |
| codespace | string | Namespace for the code. | |
| sender | string | The transaction's sender (e.g. the signer). | |
| priority | int64 | The transaction's priority (for mempool ordering). |
ResponseEcho
| Field | Type | Label | Description |
|---|---|---|---|
| message | string | The input string. |
ResponseException
nondeterministic
| Field | Type | Label | Description |
|---|---|---|---|
| error | string |
ResponseExtendVote
| Field | Type | Label | Description |
|---|---|---|---|
| vote_extensions | ExtendVoteExtension | repeated | Optional information signed by Tenderdash. |
ResponseFinalizeBlock
| Field | Type | Label | Description |
|---|---|---|---|
| retain_height | int64 | Blocks below this height may be removed. Defaults to 0 (retain all). | |
| propose_next_block_immediately | bool | When true, this node does not wait for transactions (create-empty-blocks-interval) before proposing at the next height, in round 0 or, on consensus version 2, in a round entered after a silent proposer's round was skipped: the application has block-driven work pending, for example withdrawal transactions to sign. A hint for the local node only, consumed once; a node that restarts in between falls back to the configured wait. Defaults to false. |
ResponseFlush
ResponseInfo
| Field | Type | Label | Description |
|---|---|---|---|
| data | string | Some arbitrary information. | |
| version | string | The application software semantic version. | |
| app_version | uint64 | The application protocol version. | |
| last_block_height | int64 | Last height persisted by FinalizeBlock. | |
| last_block_app_hash | bytes | Application hash at the last persisted height. 32 bytes. |
ResponseInitChain
| Field | Type | Label | Description |
|---|---|---|---|
| consensus_params | tendermint.types.ConsensusParams | Initial consensus-critical parameters (optional). | |
| app_hash | bytes | Initial application hash. 32 bytes. | |
| validator_set_update | ValidatorSetUpdate | Initial validator set (optional). | |
| next_core_chain_lock_update | tendermint.types.CoreChainLock | Initial core chain lock update. | |
| initial_core_height | uint32 | Initial height of core lock. | |
| genesis_time | google.protobuf.Timestamp | optional | Override genesis time with provided time. |
ResponseListSnapshots
| Field | Type | Label | Description |
|---|---|---|---|
| snapshots | Snapshot | repeated | List of local state snapshots. |
ResponseLoadSnapshotChunk
| Field | Type | Label | Description |
|---|---|---|---|
| chunk | bytes | The binary chunk contents, in an arbitray format. Chunk messages cannot be larger than 16 MB including metadata, so 10 MB is a good starting point. |
ResponseOfferSnapshot
| Field | Type | Label | Description |
|---|---|---|---|
| result | ResponseOfferSnapshot.Result | The result of the snapshot offer. |
ResponsePrepareProposal
| Field | Type | Label | Description |
|---|---|---|---|
| tx_records | TxRecord | repeated | Possibly modified list of transactions that have been picked as part of the proposed block. |
| app_hash | bytes | The Merkle root hash of the application state. 32 bytes. | |
| tx_results | ExecTxResult | repeated | List of structures containing the data resulting from executing the transactions. |
| consensus_param_updates | tendermint.types.ConsensusParams | Changes to consensus-critical gas, size, and other parameters that will be applied at next height. | |
| core_chain_lock_update | tendermint.types.CoreChainLock | Core chain lock that will be used for next block. | |
| validator_set_update | ValidatorSetUpdate | Changes to validator set that will be applied at next height. | |
| app_version | uint64 | Application version that was used to create the current proposal. |
ResponseProcessProposal
| Field | Type | Label | Description |
|---|---|---|---|
| status | ResponseProcessProposal.ProposalStatus | enum that signals if the application finds the proposal valid. | |
| app_hash | bytes | The Merkle root hash of the application state. 32 bytes. | |
| tx_results | ExecTxResult | repeated | List of structures containing the data resulting from executing the transactions. |
| consensus_param_updates | tendermint.types.ConsensusParams | Changes to consensus-critical gas, size, and other parameters. | |
| validator_set_update | ValidatorSetUpdate | Changes to validator set (set voting power to 0 to remove). | |
| events | Event | repeated | Type & Key-Value events for indexing |
ResponseQuery
| Field | Type | Label | Description |
|---|---|---|---|
| code | uint32 | Response code. | |
| log | string | The output of the application's logger. May be non-deterministic. | |
| info | string | Additional information. May be non-deterministic. | |
| index | int64 | The index of the key in the tree. | |
| key | bytes | The key of the matching data. | |
| value | bytes | The value of the matching data. | |
| proof_ops | tendermint.crypto.ProofOps | Serialized proof for the value data, if requested, to be verified against the app_hash for the given Height. | |
| height | int64 | The block height from which data was derived. Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height-1. | |
| codespace | string | Namespace for the code. |
ResponseVerifyVoteExtension
| Field | Type | Label | Description |
|---|---|---|---|
| status | ResponseVerifyVoteExtension.VerifyStatus | enum signaling if the application accepts the vote extension |
Snapshot
| Field | Type | Label | Description |
|---|---|---|---|
| height | uint64 | The height at which the snapshot was taken | |
| version | uint32 | The application-specific snapshot version | |
| hash | bytes | Arbitrary snapshot hash, equal only if identical | |
| metadata | bytes | Arbitrary application metadata |
ThresholdPublicKeyUpdate
| Field | Type | Label | Description |
|---|---|---|---|
| threshold_public_key | tendermint.crypto.PublicKey |
TxRecord
| Field | Type | Label | Description |
|---|---|---|---|
| action | TxRecord.TxAction | What should Tenderdash do with this transaction? | |
| tx | bytes | Transaction contents. |
TxResult
TxResult contains results of executing the transaction.
One usage is indexing transaction results.
| Field | Type | Label | Description |
|---|---|---|---|
| height | int64 | ||
| index | uint32 | ||
| tx | bytes | ||
| result | ExecTxResult |
Validator
Validator
ValidatorSetUpdate
ValidatorSetUpdate represents a change in the validator set. It can be used to add, remove, or update a validator.
Validator set update consists of multiple ValidatorUpdate records, each of them can be used to add, remove, or update a validator, according to the following rules:
- If a validator with the same public key already exists in the validator set and power is greater than 0, the existing validator will be updated with the new power.
- If a validator with the same public key already exists in the validator set and power is 0, the existing validator will be removed from the validator set.
- If a validator with the same public key does not exist in the validator set and the power is greater than 0, a new validator will be added to the validator set.
- As a special case, if quorum hash has changed, all existing validators will be removed before applying the new validator set update.
| Field | Type | Label | Description |
|---|---|---|---|
| validator_updates | ValidatorUpdate | repeated | |
| threshold_public_key | tendermint.crypto.PublicKey | ||
| quorum_hash | bytes |
ValidatorUpdate
ValidatorUpdate
| Field | Type | Label | Description |
|---|---|---|---|
| pub_key | tendermint.crypto.PublicKey | ||
| power | int64 | ||
| pro_tx_hash | bytes | ||
| node_address | string | node_address is an URI containing address of validator (proto://node_id@ip_address:port), for example: tcp://f2dbd9b0a1f541a7c44d34a58674d0262f5feca5@12.34.5.6:1234 |
VoteInfo
VoteInfo
CheckTxType
Type of transaction check
| Name | Number | Description |
|---|---|---|
| NEW | 0 | NEW is the default and means that a full check of the tranasaction is required. |
| RECHECK | 1 | RECHECK is used when the mempool is initiating a normal recheck of a transaction. |
MisbehaviorType
| Name | Number | Description |
|---|---|---|
| UNKNOWN | 0 | |
| DUPLICATE_VOTE | 1 | |
| LIGHT_CLIENT_ATTACK | 2 |
ResponseApplySnapshotChunk.Result
| Name | Number | Description |
|---|---|---|
| UNKNOWN | 0 | Unknown result, abort all snapshot restoration |
| ACCEPT | 1 | Chunk successfully accepted |
| ABORT | 2 | Abort all snapshot restoration |
| RETRY | 3 | Retry chunk (combine with refetch and reject) |
| RETRY_SNAPSHOT | 4 | Retry snapshot (combine with refetch and reject) |
| REJECT_SNAPSHOT | 5 | Reject this snapshot, try others |
| COMPLETE_SNAPSHOT | 6 | Complete this snapshot, no more chunks |
ResponseOfferSnapshot.Result
| Name | Number | Description |
|---|---|---|
| UNKNOWN | 0 | Unknown result, abort all snapshot restoration |
| ACCEPT | 1 | Snapshot accepted, apply chunks |
| ABORT | 2 | Abort all snapshot restoration |
| REJECT | 3 | Reject this specific snapshot, try others |
| REJECT_FORMAT | 4 | Reject all snapshots of this format, try others |
| REJECT_SENDER | 5 | Reject all snapshots from the sender(s), try others |
ResponseProcessProposal.ProposalStatus
| Name | Number | Description |
|---|---|---|
| UNKNOWN | 0 | Unspecified error occurred |
| ACCEPT | 1 | Proposal accepted |
| REJECT | 2 | Proposal is not valid |
ResponseVerifyVoteExtension.VerifyStatus
| Name | Number | Description |
|---|---|---|
| UNKNOWN | 0 | |
| ACCEPT | 1 | |
| REJECT | 2 |
TxRecord.TxAction
TxAction contains App-provided information on what to do with a transaction that is part of a raw proposal
| Name | Number | Description |
|---|---|---|
| UNKNOWN | 0 | Unknown action |
| UNMODIFIED | 1 | The Application did not modify this transaction. |
| ADDED | 2 | The Application added this transaction. |
| REMOVED | 3 | The Application wants this transaction removed from the proposal and the mempool. |
| DELAYED | 4 | The Application wants this transaction removed from the proposal but not the mempool. |
ABCIApplication
| Method Name | Request Type | Response Type | Description |
|---|---|---|---|
| Echo | RequestEcho | ResponseEcho | Echo a string to test an abci client/server implementation |
| Flush | RequestFlush | ResponseFlush | |
| Info | RequestInfo | ResponseInfo | |
| CheckTx | RequestCheckTx | ResponseCheckTx | |
| Query | RequestQuery | ResponseQuery | |
| InitChain | RequestInitChain | ResponseInitChain | |
| ListSnapshots | RequestListSnapshots | ResponseListSnapshots | |
| OfferSnapshot | RequestOfferSnapshot | ResponseOfferSnapshot | |
| LoadSnapshotChunk | RequestLoadSnapshotChunk | ResponseLoadSnapshotChunk | |
| ApplySnapshotChunk | RequestApplySnapshotChunk | ResponseApplySnapshotChunk | |
| PrepareProposal | RequestPrepareProposal | ResponsePrepareProposal | |
| ProcessProposal | RequestProcessProposal | ResponseProcessProposal | |
| ExtendVote | RequestExtendVote | ResponseExtendVote | |
| VerifyVoteExtension | RequestVerifyVoteExtension | ResponseVerifyVoteExtension | |
| FinalizeBlock | RequestFinalizeBlock | ResponseFinalizeBlock |
Scalar Value Types
Data Structures
Here we describe the data structures in the Tenderdash blockchain and the rules for validating them.
The Tenderdash blockchains consists of a short list of data types:
- Data Structures
ProTxHash
ProTxHash is 32 bytes array. Validator provides this parameter via a config.
Block
A block consists of a header, transactions, the commit of the previous block, an optional core chain lock, and a list of evidence of malfeasance (ie. signing conflicting votes).
| Name | Type | Description | Validation |
|---|---|---|---|
| Header | Header | Header corresponding to the block. This field contains information used throughout consensus and other areas of the protocol. To find out what it contains, visit [header] (#header) | Must adhere to the validation rules of header |
| Data | Data | Data contains a list of transactions. The contents of the transaction is unknown to Tenderdash. | This field can be empty or populated, but no validation is performed. Applications can perform validation on individual transactions prior to block creation using checkTx. |
| Evidence | EvidenceList | Evidence contains a list of infractions committed by validators. | Can be empty, but when populated the validations rules from evidenceList apply |
| LastCommit | Commit | The threshold-signed commit of the previous block. | Must not be nil, must carry no threshold block signature at the initial height, and must adhere to the validation rules of commit. |
| CoreChainLock | CoreChainLock (optional) | Core chain lock carried by the block. | When present, must pass CoreChainLock.ValidateBasic. |
Execution
Tenderdash uses same-block execution: PrepareProposal and ProcessProposal
return this block's AppHash, transaction results, and pending validator/parameter
changes. FinalizeBlock persists the accepted candidate; there is no ABCI
Commit call. Changes from block h become active at h + 1.
See the canonical primary flow,
proposal-validity summary,
and commit application
for execution, validation and persistence order. The local committed state is
summarized in State; pending next-state values belong to
CurrentRoundState.
Header
A block header contains metadata about the block and about the consensus, as well as commitments to the data in the current block, the previous block, and the results returned by the application:
| Name | Type | Description | Validation |
|---|---|---|---|
| Version | Version | Version defines the application and protocol version being used. | Must adhere to the validation rules of Version |
| ChainID | String | ChainID is the ID of the chain. This must be unique to your chain. | ChainID must be at most 50 bytes. |
| Height | int64 | Height is the height for this header. | Must be > 0, >= initialHeight, and == previous Height+1 |
| CoreChainLockedHeight | uint32 | Core chain height used to verify this block. | Must match the block's core chain lock, or the state's last locked height without one (ValidateBlockChainLock). |
| Time | Time | The timestamp is chosen by the proposer from its local clock (genesis time for the initial block) and accepted by validators only if timely; see Proposal timestamps. The historical median-based rule is described in the BFT-time section. | Above the initial height, Time must be greater than the previous block's time. At the initial height, Time must be >= genesis time; a normal prevote additionally requires it to equal genesis time (rule P-Timely). |
| LastBlockID | BlockID | BlockID of the previous block. | Must adhere to the validation rules of blockID. The first block has block.Header.LastBlockID == BlockID{}. |
| LastCommitHash | slice of bytes ([]byte) | Checksum of the last commit's threshold block signature (Commit.Hash). | Empty or of length 32; must equal LastCommit.Hash() |
| DataHash | slice of bytes ([]byte) | MerkleRoot of the hash of transactions. Note: The transactions are hashed before being included in the merkle tree, the leaves of the Merkle tree are the hashes, not the transactions themselves. | Empty or of length 32; must equal Data.Hash() |
| ValidatorsHash | slice of bytes ([]byte) | Merkle root of the current validator set's threshold public key and quorum hash (ValidatorSet.Hash). The validator list is not hashed, so its order, ascending ProTxHash (ValidatorsByProTxHashes), does not affect this value. | Empty or of length 32 |
| NextValidatorsHash | slice of bytes ([]byte) | Merkle root of the next validator set's threshold public key and quorum hash (ValidatorSet.Hash). The validator list is not hashed, so its order, ascending ProTxHash (ValidatorsByProTxHashes), does not affect this value. | Empty or of length 32 |
| ConsensusHash | slice of bytes ([]byte) | SHA-256 of the protobuf-encoded HashedParams subset of the current consensus parameters (HashConsensusParams). | Empty or of length 32 |
| NextConsensusHash | slice of bytes ([]byte) | The same hash over the consensus parameters of the next block. | Empty or of length 32 |
| AppHash | slice of bytes ([]byte) | Application state hash after executing this block's transactions, returned by the application from PrepareProposal and ProcessProposal. | The application's response must be 32 bytes; must equal the application's answer (ValidateBlockWithRoundState). |
| ResultsHash | slice of bytes ([]byte) | Merkle root of this block's ExecTxResult values, keeping only Code, Data and GasUsed (TxResultsHash). | Empty or of length 32; must equal the application's answer |
| EvidenceHash | slice of bytes ([]byte) | MerkleRoot of the evidence of Byzantine behaviour included in this block. | Empty or of length 32; must equal Evidence.Hash() |
| ProposedAppVersion | uint64 | Proposer's latest available application protocol version. | |
| ProposerProTxHash | ProTxHash | ProTxHash of the original proposer of the block. Validator must be in the current validatorSet. | Must be of length 32 |
Version
NOTE: that this is more specifically the consensus version and doesn't include information like the P2P Version. (TODO: we should write a comprehensive document about versioning that this can refer to)
| Name | type | Description | Validation |
|---|---|---|---|
| Block | uint64 | This number represents the version of the block protocol and must be the same throughout an operational network | Must be equal to protocol version being used in a network (block.Version.Block == state.Version.Consensus.Block) |
| App | uint64 | Application version used for this block; PrepareProposal can override it. | Basic block validation does not require equality with the last committed application version (validateBlock in internal/state/validation.go). |
BlockID
The BlockID contains two distinct Merkle roots of the block, the number of parts (ie. len(MakeParts(block))) and a state ID hash.
| Name | Type | Description | Validation |
|---|---|---|---|
| Hash | slice of bytes ([]byte) | MerkleRoot of all the fields in the header (ie. MerkleRoot(header). | Empty (nil vote) or of length 32 |
| PartSetHeader | PartSetHeader | Used for secure gossiping of the block during consensus, is the MerkleRoot of the complete serialized block cut into parts (ie. MerkleRoot(MakeParts(block))). | Must adhere to the validation rules of PartSetHeader |
| StateID | slice of bytes ([]byte) | Hash of the StateID fields a light client needs to verify state; see Signing. | Must be of length 32 when Hash is not empty |
See MerkleRoot for details.
PartSetHeader
| Name | Type | Description | Validation |
|---|---|---|---|
| Total | uint32 | Total amount of parts for a block | Must be <= MaxBlockPartsCount; > 0 in a complete BlockID |
| Hash | slice of bytes ([]byte) | MerkleRoot of a serialized block | Empty or of length 32 |
Part
Part defines a part of a block. In Tenderdash blocks are broken into parts for gossip.
| Name | Type | Description | Validation |
|---|---|---|---|
| index | uint32 | Index of this part | Must equal Proof.Index |
| bytes | bytes | Content of this part | Must be <= BlockPartSizeBytes long |
| proof | Proof | Merkle proof of this part | Must pass Proof.ValidateBasic |
Time
Tenderdash uses the Google.Protobuf.Timestamp format, which uses two integers, one 64 bit integer for Seconds and a 32 bit integer for Nanoseconds.
Data
Data is just a wrapper for a list of transactions, where transactions are arbitrary byte arrays:
| Name | Type | Description | Validation |
|---|---|---|---|
| Txs | Matrix of bytes ([][]byte) | Slice of transactions. | Validation does not occur on this field, this data is unknown to Tenderdash |
Commit
Commit proves that a quorum decided a block: it carries the threshold-recovered signatures rather than one signature per validator.
| Name | Type | Description | Validation |
|---|---|---|---|
| Height | int64 | Height at which this commit was created. | Must be >= 0 |
| Round | int32 | Round that the commit corresponds to. | Must be >= 0 |
| BlockID | BlockID | The blockID of the corresponding block. | At height >= 1, must not be nil and must adhere to the validation rules of BlockID. |
| QuorumHash | slice of bytes ([]byte) | Hash of the quorum that signed the commit. | |
| ThresholdBlockSignature | Signature | Threshold-recovered block signature. | At height >= 1, must be of length 96 |
| ThresholdVoteExtensions | Array of VoteExtension | Threshold-recovered vote extensions. | No nil entries; every type must be a defined enum value |
BlockIDFlag
BlockIDFlag is defined in types.proto, but Commit has no per-validator signatures and does not use it.
enum BlockIDFlag {
BLOCK_ID_FLAG_UNKNOWN = 0;
BLOCK_ID_FLAG_ABSENT = 1; // signatures for other blocks are also considered absent
BLOCK_ID_FLAG_COMMIT = 2;
BLOCK_ID_FLAG_NIL = 3;
}
Vote
A vote is a signed message from a validator for a particular block.
The vote includes information about the validator signing it. When stored in the blockchain or propagated over the network, votes are encoded in Protobuf.
The vote extension is not part of the CanonicalVote.
| Name | Type | Description | Validation |
|---|---|---|---|
| Type | SignedMsgType | Either prevote or precommit. SignedMsgType | A Vote is valid if its corresponding fields are included in the enum signedMsgType |
| Height | int64 | Height for which this vote was created. | Must be >= 0 |
| Round | int32 | Round that the vote corresponds to. | Must be >= 0 |
| BlockID | BlockID | The blockID of the corresponding block; zero for a nil vote. | Must adhere to BlockID validation and be either nil or complete |
| ValidatorProTxHash | ProTxHash | ProTxHash of the validator. | Length must be equal to 32 |
| ValidatorIndex | int32 | Index of the validator in the validator set at this height. | Must be >= 0 |
| BlockSignature | Signature | Validator's signature of the CanonicalVote sign bytes. | Length must be > 0 and <= 96 |
| VoteExtensions | Array of VoteExtension | Vote extensions provided by the Application. | Must be empty unless this is a non-nil precommit, which must pass VoteExtension validation |
VoteExtension
| Name | Type | Description | Validation |
|---|---|---|---|
| Type | VoteExtensionType | THRESHOLD_RECOVER or THRESHOLD_RECOVER_RAW. DEFAULT exists in the enum but is not supported. | Must not be DEFAULT |
| Extension | slice of bytes ([]byte) | Extension data provided by the Application. | Must be of length 32 for THRESHOLD_RECOVER_RAW |
| Signature | Signature | Validator's signature of the extension. | Must be present when Extension is not empty; length must be <= 96 |
| SignRequestId | slice of bytes ([]byte, optional) | Application-supplied sign request ID. |
CanonicalVote
CanonicalVote is for validator signing. This type will not be present in a block. For signing, votes are represented via CanonicalVote and encoded using protobuf via Vote.SignBytes, which includes the ChainID and omits ValidatorProTxHash and ValidatorIndex. block_id is a checksum of the CanonicalBlockID, and state_id is the BlockID's StateID.
message CanonicalVote {
SignedMsgType type = 1;
sfixed64 height = 2;
sfixed64 round = 3;
bytes block_id = 4;
bytes state_id = 5;
string chain_id = 99;
}
Vote.Verify(chainID, quorumType, quorumHash, pubKey, proTxHash) checks that proTxHash matches ValidatorProTxHash, then verifies the block signature, and any extension signatures, against the 48-byte BLS public key. See Signing.
Proposal
Proposal contains height and round for which this proposal is made, BlockID as a unique identifier of proposed block, timestamp, and POLRound (a so-called Proof-of-Lock (POL) round) that is needed for termination of the consensus. If POLRound >= 0, then BlockID corresponds to the block that is locked in POLRound. The message is signed by the validator private key.
| Name | Type | Description | Validation |
|---|---|---|---|
| Type | SignedMsgType | Represents a Proposal SignedMsgType | Must be ProposalType signedMsgType |
| Height | int64 | Height for which this proposal was created | Must be >= 0 |
| Round | int32 | Round that the proposal corresponds to. | Must be >= 0 |
| POLRound | int32 | Proof of lock | Must be >= -1 |
| BlockID | BlockID | The blockID of the corresponding block. | Must be complete and adhere to BlockID validation |
| Timestamp | Time | Timestamp chosen by the proposer; equal to the block time. | Time |
| Signature | Signature | Signature by the proposer. | Length of signature must be > 0 and <= 96 |
| CoreChainLockedHeight | uint32 | Core chain locked height used to sign and verify the block. | Must not be MaxUint32 |
| CoreChainLockUpdate | CoreChainLock (optional) | Core chain lock used to sign and verify the next block. | When present, must pass CoreChainLock.ValidateBasic |
SignedMsgType
Signed message type represents a signed messages in consensus.
enum SignedMsgType {
SIGNED_MSG_TYPE_UNKNOWN = 0;
// Votes
SIGNED_MSG_TYPE_PREVOTE = 1;
SIGNED_MSG_TYPE_PRECOMMIT = 2;
// Proposal
SIGNED_MSG_TYPE_PROPOSAL = 32;
// Proposer liveness, from CONSENSUS_VERSION_2
SIGNED_MSG_TYPE_PROPOSER_HEARTBEAT = 48;
SIGNED_MSG_TYPE_ROUND_SKIP = 49;
}
Signature
Signatures in Tenderdash are raw bytes representing the underlying signature. Consensus signatures are BLS12-381 signatures of 96 bytes (types.SignatureSize).
See the signature spec for more.
EvidenceList
EvidenceList is a simple wrapper for a list of evidence:
| Name | Type | Description | Validation |
|---|---|---|---|
| Evidence | Array of Evidence | List of verified evidence | Validation adheres to individual types of Evidence |
Evidence
Evidence in Tenderdash is used to indicate breaches in the consensus by a validator.
More information on how evidence works in Tenderdash can be found here
DuplicateVoteEvidence
DuplicateVoteEvidence represents a validator that has voted for two different blocks
in the same round of the same height. Votes are lexicographically sorted on BlockID.
| Name | Type | Description | Validation |
|---|---|---|---|
| VoteA | Vote | One of the votes submitted by a validator when they equivocated | VoteA must adhere to Vote validation rules |
| VoteB | Vote | The second vote submitted by a validator when they equivocated | VoteB must adhere to Vote validation rules |
| TotalVotingPower | int64 | The total power of the validator set at the height of equivocation | Must be equal to nodes own copy of the data |
| ValidatorPower | int64 | Power of the equivocating validator at the height | Must be equal to the nodes own copy of the data |
| Timestamp | Time | Time of the block where the equivocation occurred | Must be equal to the nodes own copy of the data |
LightBlock
LightBlock is the core data structure of the light client. It combines two data structures needed for verification (signedHeader & validatorSet).
| Name | Type | Description | Validation |
|---|---|---|---|
| SignedHeader | SignedHeader | The header and commit, these are used for verification purposes. To find out more visit light client docs | Must not be nil and adhere to the validation rules of signedHeader |
| ValidatorSet | ValidatorSet | The validatorSet is used to help with verify that the validators in that committed the infraction were truly in the validator set. | Must not be nil and adhere to the validation rules of validatorSet |
The SignedHeader and ValidatorSet are linked by the hash of the validator set(SignedHeader.ValidatorsHash == ValidatorSet.Hash().
SignedHeader
The SignedhHeader is the header accompanied by the commit to prove it.
| Name | Type | Description | Validation |
|---|---|---|---|
| Header | Header | Header | Header cannot be nil and must adhere to the Header validation criteria |
| Commit | Commit | Commit | Commit cannot be nil and must adhere to the Commit criteria |
ValidatorSet
| Name | Type | Description | Validation |
|---|---|---|---|
| Validators | Array of validator | List of the active validators at a specific height, sorted by ascending ProTxHash | The list of validators can not be empty or nil and must adhere to the validation rules of validator |
| Proposer | validator | Validator at the set's proposer index | The proposer index must be in range |
| ThresholdPublicKey | public key | Threshold public key of the quorum | Must pass ThresholdPublicKeyValid |
| QuorumType | int32 | LLMQ type of the quorum | |
| QuorumHash | slice of bytes ([]byte) | Hash of the quorum | Must pass QuorumHashValid |
| HasPublicKeys | bool | Whether validators carry public keys, which holds when the local node is in the set | When true, each public key must be valid |
| VotingPowerThreshold | uint64 | Minimum voting power required to commit; equals ValidatorParams.voting_power_threshold |
Validator
| Name | Type | Description | Validation |
|---|---|---|---|
| ProTxHash | ProTxHash | Validator's identity | Length must be of size 32 |
| PubKey | slice of bytes ([]byte) | Validator's BLS public key; absent when the local node is not in the set | Length must be 48 when validated |
| VotingPower | int64 | Validator's voting power | cannot be < 0 |
| NodeAddress | ValidatorAddress | URI of the validator's node | Must be valid when set |
Address
Address is a type alias of a slice of bytes, calculated by hashing the public key using sha256 and truncating it to the first 20 bytes (crypto.AddressHash). Validators are identified by ProTxHash, not by address.
ConsensusParams
| Name | Type | Description | Field Number |
|---|---|---|---|
| block | BlockParams | Parameters limiting the size and gas of a block. | 1 |
| evidence | EvidenceParams | Parameters limiting the validity of evidence of byzantine behaviour. | 2 |
| validator | ValidatorParams | Parameters limiting the types of public keys validators can use. | 3 |
| version | VersionParams | The ABCI application and consensus versions. | 4 |
| synchrony | SynchronyParams | Bounds under which a proposal timestamp is considered timely. | 5 |
| timeout | TimeoutParams | Timeouts of the consensus steps. | 6 |
| abci | ABCIParams | Application Blockchain Interface options. | 7 |
BlockParams
| Name | Type | Description | Field Number |
|---|---|---|---|
| max_bytes | int64 | Max size of a block, in bytes. | 1 |
| max_gas | int64 | Max sum of GasWanted in a proposed block. NOTE: blocks that violate this may be committed if there are Byzantine proposers. It's the application's responsibility to handle this when processing a block! | 2 |
EvidenceParams
| Name | Type | Description | Field Number |
|---|---|---|---|
| max_age_num_blocks | int64 | Max age of evidence, in blocks. | 1 |
| max_age_duration | google.protobuf.Duration | Max age of evidence, in time. It should correspond with an app's "unbonding period" or other similar mechanism for handling Nothing-At-Stake attacks. | 2 |
| max_bytes | int64 | maximum size in bytes of total evidence allowed to be entered into a block | 3 |
ValidatorParams
| Name | Type | Description | Field Number |
|---|---|---|---|
| pub_key_types | repeated string | List of accepted public key types. Uses same naming as PubKey.Type. | 1 |
| voting_power_threshold | uint64 (optional) | Minimum voting power required to commit a block. See voting_power_threshold. | 2 |
voting_power_threshold
A positive voting_power_threshold sets the vote-counting threshold T returned
by ValidatorSet.QuorumVotingThresholdPower; zero uses the quorum-type threshold.
Every validator contributes power 100. An explicit value must not exceed total
power, but a threshold below the strict BFT floor is only warned about.
This override does not lower the separate BLS recovery threshold R returned by
QuorumTypeThresholdVotingPower. Prevotes expose a majority at T; precommits
also require successful recovery and verification of threshold signatures. A low
vote-counting threshold alone does not let one signature share produce a commit.
A threshold above R disables round skipping
(Proposer liveness).
See Threshold recovery for the
canonical extension-group and recovery rules, and
Commit threshold is configurable below the BFT bound
for the documented safety limitation. Changes to this live consensus parameter
must not be assumed incapable of changing consensus behavior merely because a
parameter is absent from a particular header-hash subset.
VersionParams
| Name | Type | Description | Field Number |
|---|---|---|---|
| app_version | uint64 | The ABCI application version. | 1 |
| consensus_version | ConsensusVersion | Version of the consensus protocol, selecting the proposer strategy. CONSENSUS_VERSION_2 also enables proposer liveness. | 2 |
SynchronyParams
| Name | Type | Description | Field Number |
|---|---|---|---|
| message_delay | google.protobuf.Duration | Bound for how long a proposal message may take to reach all validators on a newtork and still be considered valid. | 1 |
| precision | google.protobuf.Duration | Bound for how skewed a proposer's clock may be from any validator on the network while still producing valid proposals. | 2 |
TimeoutParams
| Name | Type | Description | Field Number |
|---|---|---|---|
| propose | google.protobuf.Duration | Parameter that, along with propose_delta, configures the timeout for the propose step of the consensus algorithm. | 1 |
| propose_delta | google.protobuf.Duration | Parameter that, along with propose, configures the timeout for the propose step of the consensus algorithm. | 2 |
| vote | google.protobuf.Duration | Parameter that, along with vote_delta, configures the timeout for the prevote and precommit step of the consensus algorithm. | 3 |
| vote_delta | google.protobuf.Duration | Parameter that, along with vote, configures the timeout for the prevote and precommit step of the consensus algorithm. | 4 |
| proposer_heartbeat | google.protobuf.Duration | How often the proposer of a round that waits for transactions sends a heartbeat. Used from CONSENSUS_VERSION_2; unset means derived (see Parameters). | 7 |
| proposer_silence | google.protobuf.Duration | How long a validator waits without a heartbeat before it votes to skip the round; grows by proposer_heartbeat per round. Used from CONSENSUS_VERSION_2; unset means derived. | 8 |
ABCIParams
| Name | Type | Description | Field Number |
|---|---|---|---|
| recheck_tx | bool | Indicates whether to run CheckTx on all remaining transactions after every execution of a block | 1 |
Proof
| Name | Type | Description | Field Number |
|---|---|---|---|
| total | int64 | Total number of items. | 1 |
| index | int64 | Index item to prove. | 2 |
| leaf_hash | bytes | Hash of item value. | 3 |
| aunts | repeated bytes | Hashes from leaf's sibling to a root's child. | 4 |
Encoding
Protocol Buffers
Tenderdash uses Protocol Buffers, specifically proto3, for all data structures.
Please see the Proto3 language guide for more details.
Byte Arrays
The encoding of a byte array is simply the raw-bytes prefixed with the length of
the array as a UVarint (what proto calls a Varint).
For details on varints, see the protobuf spec.
For example, the byte-array [0xA, 0xB] would be encoded as 0x020A0B,
while a byte-array containing 300 entires beginning with [0xA, 0xB, ...] would
be encoded as 0xAC020A0B... where 0xAC02 is the UVarint encoding of 300.
Hashing
Tenderdash uses SHA256 as its hash function.
Objects are always serialized before being hashed.
So SHA256(obj) is short for SHA256(ProtoEncoding(obj)).
Public Key Cryptography
Tenderdash uses Protobuf Oneof to distinguish between different types public keys, and signatures. Additionally, for each public key, Tenderdash defines an Address function that can be used as a more compact identifier in place of the public key. Here we list the concrete types, their names, and prefix bytes for public keys and signatures, as well as the address schemes for each PubKey. Note for brevity we don't include details of the private keys beyond their type and name.
Key Types
Each type specifies it's own pubkey, address, and signature format.
Ed25519
The address is the first 20-bytes of the SHA256 hash of the raw 32-byte public key:
address = SHA256(pubkey)[:20]
The signature is the raw 64-byte ED25519 signature.
Tenderdash uses zip215 for verification of ed25519 signatures.
Historical upstream note: this change was planned for Tendermint-Go 0.35.
Secp256k1
The address is the first 20-bytes of the SHA256 hash of the raw 32-byte public key:
address = SHA256(pubkey)[:20]
Other Common Types
BitArray
The BitArray is used in some consensus messages to represent votes received from
validators, or parts received in a block. It is represented
with a struct containing the number of bits (Bits) and the bit-array itself
encoded in base64 (Elems).
| Name | Type |
|---|---|
| bits | int64 |
| elems | slice of int64 ([]int64) |
Note BitArray receives a special JSON encoding in the form of x and _
representing 1 and 0. Ie. the BitArray 10110 would be JSON encoded as
"x_xx_"
Part
Part is used to break up blocks into pieces that can be gossiped in parallel and securely verified using a Merkle tree of the parts.
Part contains the index of the part (Index), the actual
underlying data of the part (Bytes), and a Merkle proof that the part is contained in
the set (Proof).
| Name | Type |
|---|---|
| index | uint32 |
| bytes | slice of bytes ([]byte) |
| proof | proof |
See details of SimpleProof, below.
MakeParts
Encode an object using Protobuf and slice it into parts.
Tenderdash uses a part size of 65536 bytes, and allows a maximum of 1601 parts
(see types.MaxBlockPartsCount). This corresponds to the hard-coded block size
limit of 100MB.
func MakeParts(block Block) []Part
Merkle Trees
For an overview of Merkle trees, see wikipedia
We use the RFC 6962 specification of a merkle tree, with sha256 as the hash function. Merkle trees are used throughout Tenderdash to compute a cryptographic digest of a data structure. The differences between RFC 6962 and the simplest form a merkle tree are that:
-
leaf nodes and inner nodes have different hashes. This is for "second pre-image resistance", to prevent the proof to an inner node being valid as the proof of a leaf. The leaf nodes are
SHA256(0x00 || leaf_data), and inner nodes areSHA256(0x01 || left_hash || right_hash). -
When the number of items isn't a power of two, the left half of the tree is as big as it could be. (The largest power of two less than the number of items) This allows new leaves to be added with less recomputation. For example:
Simple Tree with 6 items Simple Tree with 7 items
* *
/ \ / \
/ \ / \
/ \ / \
/ \ / \
* * * *
/ \ / \ / \ / \
/ \ / \ / \ / \
/ \ / \ / \ / \
* * h4 h5 * * * h6
/ \ / \ / \ / \ / \
h0 h1 h2 h3 h0 h1 h2 h3 h4 h5
MerkleRoot
The function MerkleRoot is a simple recursive function defined as follows:
// SHA256([]byte{})
func emptyHash() []byte {
return tmhash.Sum([]byte{})
}
// SHA256(0x00 || leaf)
func leafHash(leaf []byte) []byte {
return tmhash.Sum(append(0x00, leaf...))
}
// SHA256(0x01 || left || right)
func innerHash(left []byte, right []byte) []byte {
return tmhash.Sum(append(0x01, append(left, right...)...))
}
// largest power of 2 less than k
func getSplitPoint(k int) { ... }
func MerkleRoot(items [][]byte) []byte{
switch len(items) {
case 0:
return empthHash()
case 1:
return leafHash(items[0])
default:
k := getSplitPoint(len(items))
left := MerkleRoot(items[:k])
right := MerkleRoot(items[k:])
return innerHash(left, right)
}
}
Note: MerkleRoot operates on items which are arbitrary byte arrays, not
necessarily hashes. For items which need to be hashed first, we introduce the
Hashes function:
func Hashes(items [][]byte) [][]byte {
return SHA256 of each item
}
Note: we will abuse notion and invoke MerkleRoot with arguments of type struct or type []struct.
For struct arguments, we compute a [][]byte containing the protobuf encoding of each
field in the struct, in the same order the fields appear in the struct.
For []struct arguments, we compute a [][]byte by protobuf encoding the individual struct elements.
Merkle Proof
Proof that a leaf is in a Merkle tree is composed as follows:
| Name | Type |
|---|---|
| total | int64 |
| index | int64 |
| leafHash | slice of bytes ([]byte) |
| aunts | Matrix of bytes ([][]byte) |
Which is verified as follows:
func (proof Proof) Verify(rootHash []byte, leaf []byte) bool {
assert(proof.LeafHash, leafHash(leaf)
computedHash := computeHashFromAunts(proof.Index, proof.Total, proof.LeafHash, proof.Aunts)
return computedHash == rootHash
}
func computeHashFromAunts(index, total int, leafHash []byte, innerHashes [][]byte) []byte{
assert(index < total && index >= 0 && total > 0)
if total == 1{
assert(len(proof.Aunts) == 0)
return leafHash
}
assert(len(innerHashes) > 0)
numLeft := getSplitPoint(total) // largest power of 2 less than total
if index < numLeft {
leftHash := computeHashFromAunts(index, numLeft, leafHash, innerHashes[:len(innerHashes)-1])
assert(leftHash != nil)
return innerHash(leftHash, innerHashes[len(innerHashes)-1])
}
rightHash := computeHashFromAunts(index-numLeft, total-numLeft, leafHash, innerHashes[:len(innerHashes)-1])
assert(rightHash != nil)
return innerHash(innerHashes[len(innerHashes)-1], rightHash)
}
The number of aunts is limited to 100 (MaxAunts) to protect the node against DOS attacks.
This limits the tree size to 2^100 leaves, which should be sufficient for any
conceivable purpose.
IAVL+ Tree
Because Tenderdash only uses a Simple Merkle Tree, application developers are expect to use their own Merkle tree in their applications. For example, the IAVL+ Tree - an immutable self-balancing binary tree for persisting application state is used by the Cosmos SDK
JSON
Tenderdash has its own JSON encoding in order to keep backwards compatibility with the previous RPC layer.
Registered types are encoded as:
{
"type": "<type name>",
"value": <JSON>
}
For instance, an ED25519 PubKey would look like:
{
"type": "tendermint/PubKeyEd25519",
"value": "uZ4h63OFWuQ36ZZ4Bd6NF+/w9fWUwrOncrQsackrsTk="
}
Where the "value" is the base64 encoding of the raw pubkey bytes, and the
"type" is the type name for Ed25519 pubkeys.
Signed Messages
Signed messages (eg. votes, proposals) in the consensus are encoded using protobuf.
When signing, the elements of a message are re-ordered so the fixed-length fields
are first, making it easy to quickly check the type, height, and round.
The ChainID is also appended to the end.
We call this encoding the SignBytes. For instance, SignBytes for a vote is the protobuf encoding of the following struct:
message CanonicalVote {
SignedMsgType type = 1;
sfixed64 height = 2; // canonicalization requires fixed size encoding here
sfixed64 round = 3; // canonicalization requires fixed size encoding here
CanonicalBlockID block_id = 4;
google.protobuf.Timestamp timestamp = 5;
string chain_id = 6;
}
The field ordering and the fixed sized encoding for the first three fields is optimized to ease parsing of SignBytes in HSMs. It creates fixed offsets for relevant fields that need to be read in this context.
Note: All canonical messages are length prefixed.
For more details, see the signing spec. Also, see the motivating discussion in #1622.
Genesis
The genesis file is the starting point of a chain. An application will populate the app_state field in the genesis with their required fields. Tenderdash is not able to validate this section because it is unaware what application state consists of.
Genesis Fields
genesis_time: The time the blockchain started or will start. If nodes are started before this time they will sit idle until the time specified.chain_id: The chain identifier. Every chain should have a unique identifier. When conducting a fork based upgrade, we recommend changing the chainid to avoid network or consensus errors.initial_height: The starting height of the blockchain. When conducting a chain restart to avoid restarting at height 1, the network is able to start at a specified height.consensus_paramsblockmax_bytes: The max amount of bytes a block can be.max_gas: The maximum amount of gas that a block can have.
evidencemax_age_num_blocks: After this preset amount of blocks has passed a single piece of evidence is considered invalid.max_age_duration: After this preset amount of time has passed a single piece of evidence is considered invalid.max_bytes: The max amount of bytes of all evidence included in a block.
validatorpub_key_types: Lists the validator consensus key types allowed by the chain. Recognized values areed25519,secp256k1, andbls12381; the default allows onlybls12381(BLS12-381).
versionapp_version: The version of the application. This is set by the application and is used to identify which version of the app a user should be using in order to operate a node.consensus: The consensus protocol version (CONSENSUS_VERSION_*). It selects the proposer strategy; from version 2 it also enables proposer liveness and round skipping.
synchronymessage_delay: A bound on how long a proposal message may take to reach all validators on a network and still be considered valid.precision: A bound on how skewed the proposer's clock may be from any validator on the network while still producing valid proposals.
timeoutpropose: How long the Tenderdash consensus engine will wait for a proposal block before prevoting nil.propose_delta: How much the propose timeout increase with each round.vote: How long the consensus engine will wait after receiving +2/3 votes in a round.vote_delta: How much the vote timeout increases with each round.proposer_heartbeat: From consensus version 2, how often the proposer of a round that waits for transactions sends a heartbeat. Optional; see Parameters for the defaults.proposer_silence: From consensus version 2, how long a validator waits without a heartbeat before voting to skip the round. Optional; see Parameters for the defaults.
validators- This is an array of validators. This validator set is used as the starting validator set of the chain. This field can be empty, if the application sets the validator set in
InitChain.
- This is an array of validators. This validator set is used as the starting validator set of the chain. This field can be empty, if the application sets the validator set in
app_hash: The applications state root hash. This field does not need to be populated at the start of the chain, the application may provide the needed information viaInitchain.app_state: This section is filled in by the application and is unknown to Tenderdash.
Core protocol
This section describes the core types and functionality of the Tenderdash protocol implementation.
State
The committed State in internal/state/state.go
contains the information needed to validate the next block. It is local state,
not a network message or a hash of the whole consensus state. Header commitments
and the signed StateID have their own definitions in
Data Structures.
| Fields | Meaning |
|---|---|
ChainID, InitialHeight | Chain identity and first block height; the initial height is normally 1 and must be positive. |
LastBlockHeight, LastBlockRound, LastBlockID, LastBlockTime | Last committed block and its commit round. |
LastCoreChainLockedBlockHeight | Last committed core chain lock height; unchanged when a block carries no new lock. |
Validators, LastValidators | Active validators for the next block, and the previous set used to verify its LastCommit. |
ConsensusParams | Parameters used to validate the next block. |
LastAppHash, LastResultsHash | Application state and transaction-result commitments of the last committed block. |
LastHeightValidatorsChanged, LastHeightConsensusParamsChanged | Heights at which changes become active. |
Version | Consensus block/application versions and software version. |
The ephemeral round state is separate
from this committed state. Pending next validators and parameters are held in
CurrentRoundState, not a committed State.NextValidators field.
Execution
Tenderdash uses same-block execution: PrepareProposal and ProcessProposal
return the application hash and transaction results for the block being proposed.
FinalizeBlock persists the accepted candidate state. There is no ABCI Commit
call, and ResponseFinalizeBlock does not return an application hash.
The authoritative ordering of validation, block/WAL writes, FinalizeBlock,
mempool/evidence updates and state persistence is
Commit and apply.
Validator-set and parameter changes from block h take effect at h + 1; see
Validator-set and parameter changes.
For startup reconciliation of application, block-store and state-store heights,
see Recovery and synchronisation.
Version
Version distinguishes block protocol version from
application version. ConsensusParams.Version.ConsensusVersion additionally
selects the proposer-selection strategy
and, from version 2, enables
proposer liveness.
Block
ConsensusParams.Block.MaxBytes limits block size. Transaction selection uses
Block.MaxGas, but applications must validate gas requirements when processing
blocks; see BlockParams.
The complete block/header/application checks are in the
proposal-validity summary.
Evidence
ConsensusParams.Evidence bounds evidence age and bytes. Pending evidence is
included during proposal creation.
Conflicting-vote handling
reports evidence to the pool; reporting is distinct from its inclusion in a block
and application-level punishment.
Validator
Validators are identified by ProTxHash, ordered by ascending ProTxHash, and
have equal DefaultDashVotingPower = 100. The active quorum includes its type,
hash, threshold public key and voting-power threshold; see
Terminology and
ValidatorSet.
BFT Time
Historical and outdated. Tenderdash does not implement BFT Time. Block timestamps are chosen by the proposer and validated for timeliness as specified in Proposal timestamps and time rules; Proposer-Based Timestamps is historical design material. This page is kept because the PBTS design documents compare against it.
Tendermint provides a deterministic, Byzantine fault-tolerant, source of time. Time in Tendermint is defined with the Time field of the block header.
It satisfies the following properties:
- Time Monotonicity: Time is monotonically increasing, i.e., given
a header H1 for height h1 and a header H2 for height
h2 = h1 + 1,H1.Time < H2.Time. - Time Validity: Given a set of Commit votes that forms the
block.LastCommitfield, a range of valid values for the Time field of the block header is defined only by
Precommit messages (from the LastCommit field) sent by correct processes, i.e., a faulty process cannot arbitrarily increase the Time value.
In the context of Tendermint, time is of type int64 and denotes UNIX time in milliseconds, i.e., corresponds to the number of milliseconds since January 1, 1970. Before defining rules that need to be enforced by the Tendermint consensus protocol, so the properties above holds, we introduce the following definition:
- median of a Commit is equal to the median of
Vote.Timefields of theVotemessages, where the value ofVote.Timeis counted number of times proportional to the process voting power. As in Tendermint the voting power is not uniform (one process one vote), a vote message is actually an aggregator of the same votes whose number is equal to the voting power of the process that has casted the corresponding votes message.
Let's consider the following example:
- we have four processes p1, p2, p3 and p4, with the following voting power distribution (p1, 23), (p2, 27), (p3, 10)
and (p4, 10). The total voting power is 70 (
N = 3f+1, whereNis the total voting power, andfis the maximum voting power of the faulty processes), so we assume that the faulty processes have at most 23 of voting power. Furthermore, we have the following vote messages in some LastCommit field (we ignore all fields except Time field):- (p1, 100), (p2, 98), (p3, 1000), (p4, 500). We assume that p3 and p4 are faulty processes. Let's assume that the
block.LastCommitmessage contains votes of processes p2, p3 and p4. Median is then chosen the following way: the value 98 is counted 27 times, the value 1000 is counted 10 times and the value 500 is counted also 10 times. So the median value will be the value 98. No matter what set of messages with at least2f+1voting power we choose, the median value will always be between the values sent by correct processes.
- (p1, 100), (p2, 98), (p3, 1000), (p4, 500). We assume that p3 and p4 are faulty processes. Let's assume that the
We ensure Time Monotonicity and Time Validity properties by the following rules:
-
let rs denotes
RoundState(consensus internal state) of some process. Thenrs.ProposalBlock.Header.Time == median(rs.LastCommit) && rs.Proposal.Timestamp == rs.ProposalBlock.Header.Time. -
Furthermore, when creating the
votemessage, the following rules for determiningvote.Timefield should hold:-
if
rs.LockedBlockis defined thenvote.Time = max(rs.LockedBlock.Timestamp + time.Millisecond, time.Now()), wheretime.Now()denotes local Unix time in milliseconds -
else if
rs.Proposalis defined thenvote.Time = max(rs.Proposal.Timestamp + time.Millisecond,, time.Now()), -
otherwise,
vote.Time = time.Now()). In this case vote is fornilso it is not taken into account for the timestamp of the next block.
-
Consensus Paper
See the original Tendermint paper, The latest gossip on BFT consensus, by Ethan Buchman, Jae Kwon and Zarko Milosevic.
For the current Tenderdash implementation, use the Tenderdash consensus protocol.
Evidence
Evidence is an important component of Tenderdash's security model. Whilst the core consensus protocol provides correctness guarantees for state machine replication that can tolerate less than 1/3 failures, the evidence system looks to detect and gossip byzantine faults whose combined power is greater than or equal to 1/3. It is worth noting that the evidence system is designed purely to detect possible attacks, gossip them, commit them on chain and inform the application running on top of Tenderdash. Evidence in itself does not punish "bad actors", this is left to the discretion of the application. A common form of punishment is slashing where the validators that were caught violating the protocol have all or a portion of their voting power removed. Evidence, given the assumption that 1/3+ of the network is still byzantine, is susceptible to censorship and should therefore be considered added security on a "best effort" basis.
This document walks through the various forms of evidence, how they are detected, gossiped, verified and committed.
NOTE: Evidence here is internal to Tenderdash and should not be confused with application evidence
Detection
Equivocation
Equivocation is the most fundamental of byzantine faults. Simply put, to prevent
replication of state across all nodes, a validator tries to convince some subset
of nodes to commit one block whilst convincing another subset to commit a
different block. This is achieved by double voting (hence
DuplicateVoteEvidence). A successful duplicate vote attack requires greater
than 1/3 voting power and a (temporary) network partition between the aforementioned
subsets. This is because in consensus, votes are gossiped around. When a node
observes two conflicting votes from the same peer, it will use the two votes of
evidence and begin gossiping this evidence to other nodes. Verification is addressed further down.
type DuplicateVoteEvidence struct {
VoteA Vote
VoteB Vote
// and abci specific fields
}
Light Client Attacks
Light clients also comply with the 1/3+ security model, however, by using a different, more lightweight verification method they are subject to a different kind of 1/3+ attack whereby the byzantine validators could sign an alternative light block that the light client will think is valid. Detection, explained in greater detail here, involves comparison with multiple other nodes in the hope that at least one is "honest". An "honest" node will return a challenging light block for the light client to validate. If this challenging light block also meets the validation criteria then the light client sends the "forged" light block to the node. Verification is addressed further down.
type LightClientAttackEvidence struct {
ConflictingBlock LightBlock
CommonHeight int64
// and abci specific fields
}
Verification
If a node receives evidence, it will first try to verify it, then persist it.
Evidence of byzantine behavior should only be committed once (uniqueness) and
should be committed within a certain period from the point that it occurred
(timely). Timelines is defined by the EvidenceParams: MaxAgeNumBlocks and
MaxAgeDuration. In Proof of Stake chains where validators are bonded, evidence
age should be less than the unbonding period so validators still can be
punished. Given these two propoerties the following initial checks are made.
-
Has the evidence expired? This is done by taking the height of the
VotewithinDuplicateVoteEvidenceorCommonHeightwithinLightClientAttakEvidence. The evidence height is then used to retrieve the header and thus the time of the block that corresponds to the evidence. IfCurrentHeight - MaxAgeNumBlocks > EvidenceHeight&&CurrentTime - MaxAgeDuration > EvidenceTime, the evidence is considered expired and ignored. -
Has the evidence already been committed? The evidence pool tracks the hash of all committed evidence and uses this to determine uniqueness. If a new evidence has the same hash as a committed one, the new evidence will be ignored.
DuplicateVoteEvidence
Valid DuplicateVoteEvidence must adhere to the following rules:
-
Validator Address, Height, Round and Type must be the same for both votes
-
BlockID must be different for both votes (BlockID can be for a nil block)
-
Validator must have been in the validator set at that height
-
Vote signature must be correctly signed. This also uses
ChainIDso we know that the fault occurred on this chain
LightClientAttackEvidence
Valid Light Client Attack Evidence must adhere to the following rules:
-
If the header of the light block is invalid, thus indicating a lunatic attack, the node must check that they can use
verifySkippingfrom their header at the common height to the conflicting header -
If the header is valid, then the validator sets are the same and this is either a form of equivocation or amnesia. We therefore check that 2/3 of the validator set also signed the conflicting header.
-
The nodes own header at the same height as the conflicting header must have a different hash to the conflicting header.
-
If the nodes latest header is less in height to the conflicting header, then the node must check that the conflicting block has a time that is less than this latest header (This is a forward lunatic attack).
Gossiping
If a node verifies evidence it then broadcasts it to all peers, continously sending the same evidence once every 10 seconds until the evidence is seen on chain or expires.
Commiting on Chain
Evidence takes strict priority over regular transactions, thus a block is filled
with evidence first and transactions take up the remainder of the space. To
mitigate the threat of an already punished node from spamming the network with
more evidence, the size of the evidence in a block can be capped by
EvidenceParams.MaxBytes. Nodes receiving blocks with evidence will validate
the evidence before sending Prevote and Precommit votes. The evidence pool
will usually cache verifications so that this process is much quicker.
Sending Evidence to the Application
After evidence is committed, the block is then processed by the block executor
which delivers the evidence to the application via EndBlock. Evidence is
stripped of the actual proof, split up per faulty validator and only the
validator, height, time and evidence type is sent.
enum EvidenceType {
UNKNOWN = 0;
DUPLICATE_VOTE = 1;
LIGHT_CLIENT_ATTACK = 2;
}
message Evidence {
EvidenceType type = 1;
// The offending validator
Validator validator = 2 [(gogoproto.nullable) = false];
// The height when the offense occurred
int64 height = 3;
// The corresponding time where the offense occurred
google.protobuf.Timestamp time = 4 [
(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
// Total voting power of the validator set in case the ABCI application does
// not store historical validators.
// https://github.com/tendermint/tendermint/issues/4581
int64 total_voting_power = 5;
}
DuplicateVoteEvidence and LightClientAttackEvidence are self-contained in
the sense that the evidence can be used to derive the abci.Evidence that is
sent to the application. Because of this, extra fields are necessary:
type DuplicateVoteEvidence struct {
VoteA *Vote
VoteB *Vote
// abci specific information
TotalVotingPower int64
ValidatorPower int64
Timestamp time.Time
}
type LightClientAttackEvidence struct {
ConflictingBlock *LightBlock
CommonHeight int64
// abci specific information
ByzantineValidators []*Validator
TotalVotingPower int64
Timestamp time.Time
}
These ABCI specific fields don't affect validity of the evidence itself but must be consistent amongst nodes and agreed upon on chain. If evidence with the incorrect abci information is sent, a node will create new evidence from it and replace the ABCI fields with the correct information.
Tenderdash Light Client Protocol
Deprecated, please see light-client.
Fork accountability
Deprecated, please see light-client/accountability.
Detection
Deprecated, please see light-client/detection.
Core Verification
Deprecated, please see light-client/accountability.
Proposer-Based Timestamps (PBTS)
For current Tenderdash behavior, see Proposal timestamps and time rules and the proposal validity rules.
Historical Tendermint PBTS research is retained where it has no complete replacement in the current implementation specification:
- Protocol specification
- System model and properties
- Original BFT-Time comparison
- TLA+ model (first draft)
PBTS: Protocol Specification
Historical and outdated. This Tendermint research draft is retained for its algorithm derivation; it does not specify current Tenderdash behavior. Use Proposal timestamps and time rules and proposal validity for the implemented rules.
Proposal Time
PBTS computes for a proposed value v the proposal time v.time, with bounded difference to the actual real-time the proposed value was generated.
The proposal time is read from the clock of the process that proposes a value for the first time, its original proposer.
With PBTS, therefore, we assume that processes have access to synchronized clocks.
The proper definition of what it means can be found in the system model,
but essentially we assume that two correct processes do not simultaneous read from their clocks
time values that differ more than PRECISION, which is a system parameter.
Proposal times are definitive
When a value v is produced by a process, it also assigns the associated proposal time v.time.
If the same value v is then re-proposed in a subsequent round of consensus,
it retains its original time, assigned by its original proposer.
A value v should re-proposed when it becomes locked by the network, i.e., when it receives 2f + 1 PREVOTES in a round r of consensus.
This means that processes with 2f + 1-equivalent voting power accepted, in round r, both v and its associated time v.time.
Since the originally proposed value and its associated time were considered valid, there is no reason for reassigning v.time.
In the first version of this specification, proposals were defined as pairs (v, time).
In addition, the same value v could be proposed, in different rounds, but would be associated to distinct times each time it was reproposed.
Since this possibility does not exist in this second specification, the proposal time became part of the proposed value.
With this simplification, several small changes to the arXiv algorithm are no longer required.
Time Monotonicity
Values decided in successive heights of consensus must have increasing times, so:
- Monotonicity: for any process
pand any two decided heightshandh', ifh > h'thendecision_p[h].time > decision_p[h'].time.
For ensuring time monotonicity, it is enough to ensure that a value v proposed by process p at height h_p has v.time > decision_p[h_p-1].time.
So, if process p is the proposer of a round of height h_p and reads from its clock a time now_p <= decision_p[h_p-1],
it should postpone the generation of its proposal until now_p > decision_p[h_p-1].
Although it should be considered, this scenario is unlikely during regular operation, as from
decision_p[h_p-1].timeand the start of heighth_p, a complete consensus instance need to terminate.
Notice that monotonicity is not introduced by this proposal, being already ensured by BFTTime.
In BFTTime, the Timestamp field of every Precommit message of height h_p sent by a correct process is required to be larger than decision_p[h_p-1].time, as one of such Timestamp fields becomes the time assigned to a value proposed at height h_p.
The time monotonicity of values proposed in heights of consensus is verified by the valid() predicate, to which every proposed value is submitted.
A value rejected by the valid() implementation is not accepted by any correct process.
Timely Proposals
PBTS introduces a new requirement for a process to accept a proposal: the proposal must be timely.
It is a temporal requirement, associated with the following synchrony (that is, timing)
assumptions regarding the behavior of processes and the network:
- Synchronized clocks: the values simultaneously read from clocks of any two correct processes differ by at most
PRECISION; - Bounded transmission delays: the real time interval between the sending of a proposal at a correct process, and the reception of the proposal at any correct process is upper bounded by
MSGDELAY.
[PBTS-RECEPTION-STEP.1]
Let now_p be the time, read from the clock of process p, at which p receives the proposed value v.
The proposal is considered timely by p when:
now_p >= v.time - PRECISIONnow_p <= v.time + MSGDELAY + PRECISION
The first condition derives from the fact that the generation and sending of v precedes its reception.
The minimum receiving time now_p for v be considered timely by p is derived from the extreme scenario when
the clock of p is PRECISION behind of the clock of the proposer of v, and the proposal's transmission delay is 0 (minimum).
The second condition derives from the assumption of an upper bound for the transmission delay of a proposal.
The maximum receiving time now_p for v be considered timely by p is derived from the extreme scenario when
the clock of p is PRECISION ahead of the clock of the proposer of v, and the proposal's transmission delay is MSGDELAY (maximum).
Updated Consensus Algorithm
The following changes are proposed for the algorithm in the arXiv paper.
New StartRound
There are two additions to the propose round step when executed by the proposer of a round:
- to ensure time monotonicity, the proposer does not propose a value until its current local time becomes greater than the previously decided value's time
- when the proposer produce a new proposal it sets the proposal's time to its current local time
- no changes are made to the logic when a proposer has a non-nil
validValue, which retains its original proposal time.
- no changes are made to the logic when a proposer has a non-nil
[PBTS-ALG-STARTROUND.1]
function StartRound(round) {
round_p ← round
step_p ← propose
if proposer(h_p, round_p) = p {
wait until now_p > decision_p[h_p-1].time // time monotonicity
if validValue_p != nil {
proposal ← validValue_p
} else {
proposal ← getValue()
proposal.time ← now_p // proposal time
}
broadcast ⟨PROPOSAL, h_p, round_p, proposal, validRound_p⟩
} else {
schedule OnTimeoutPropose(h_p,round_p) to be executed after timeoutPropose(round_p)
}
}
New Rule Replacing Lines 22 - 27
The rule on line 22 applies to values v proposed for the first time, i.e., for proposals not backed by 2f + 1 PREVOTEs for v in a previous round.
The PROPOSAL message, in this case, carry -1 in its validRound field.
The new rule for issuing a PREVOTE for a proposed value v requires the value to be timely.
As the timely predicate is evaluated in the moment that the value is received,
as part of a PROPOSAL message, we require the PROPOSAL message to be timely.
[PBTS-ALG-UPON-PROP.1]
upon timely(⟨PROPOSAL, h_p, round_p, v, −1⟩) from proposer(h_p, round_p) while step_p = propose do {
if valid(v) ∧ (lockedRound_p = −1 ∨ lockedValue_p = v) {
broadcast ⟨PREVOTE, h_p, round_p, id(v)⟩
}
else {
broadcast ⟨PREVOTE, h_p, round_p, nil⟩
}
step_p ← prevote
}
Rules at Lines 28 - 33 remain unchanged
The rule on line 28 applies to values v proposed again in the current round because its proposer received 2f + 1 PREVOTEs for v in a previous round vr.
This means that there was a round r <= vr in which 2f + 1 processes accepted v for the first time, and so sent PREVOTEs for v.
Which, in turn, means that these processes executed the line 22 of the algorithm, and therefore judged v as a timely proposal.
In other words, we don't need to verify whether v is a timely proposal because at least f + 1 processes judged v as timely in a previous round,
and because, since v was re-proposed as a validValue (line 16), v.time has not being updated from its original proposal.
All other rules remains unchanged.
Back to main document.
PBTS: System Model and Properties
Historical and outdated. This Tendermint research draft is retained for its clock model and temporal analysis, which have no complete replacement in the implementation specification. Its guarantees do not describe current Tenderdash. Use the consensus system model and proposal timestamps and time rules for the implemented behavior.
Outline
System Model
[PBTS-CLOCK-NEWTON.0]
There is a reference Newtonian real-time t.
No process has direct access to this reference time, used only for specification purposes. The reference real-time is assumed to be aligned with the Coordinated Universal Time (UTC).
Synchronized clocks
Processes are assumed to be equipped with synchronized clocks, aligned with the Coordinated Universal Time (UTC).
This requires processes to periodically synchronize their local clocks with an external and trusted source of the time (e.g. NTP servers). Each synchronization cycle aligns the process local clock with the external source of time, making it a fairly accurate source of real time. The periodic (re)synchronization aims to correct the drift of local clocks, which tend to pace slightly faster or slower than the real time.
To avoid an excessive level detail in the parameters and guarantees of
synchronized clocks, we adopt a single system parameter PRECISION to
encapsulate the potential inaccuracy of the synchronization mechanisms,
and drifts of local clocks from real time.
[PBTS-CLOCK-PRECISION.0]
There exists a system parameter PRECISION, such that
for any two processes p and q, with local clocks C_p and C_q:
- If
pandqare equipped with synchronized clocks, then for any real-timetwe have|C_p(t) - C_q(t)| <= PRECISION.
PRECISION thus bounds the difference on the times simultaneously read by processes
from their local clocks, so that their clocks can be considered synchronized.
Accuracy
A second relevant clock parameter is accuracy, which binds the values read by processes from their clocks to real time.
[PBTS-CLOCK-ACCURACY.0]
For the sake of completeness, we define a parameter ACCURACY such that:
- At real time
tthere is at least one correct processpwhich clock marksC_p(t)with|C_p(t) - t| <= ACCURACY.
As a consequence, applying the definition of PRECISION, we have:
- At real time
tthe synchronized clock of any correct processpmarksC_p(t)with|C_p(t) - t| <= ACCURACY + PRECISION.
The reason for not adopting ACCURACY as a system parameter is the assumption
that PRECISION >> ACCURACY.
This allows us to consider, for practical purposes, that the PRECISION system
parameter embodies the ACCURACY model parameter.
Message Delays
The assumption that processes have access to synchronized clocks ensures that proposal times assigned by correct processes have a bounded relation with the real time. It is not enough, however, to identify (and reject) proposal times proposed by Byzantine processes.
To properly evaluate whether the time assigned to a proposal is consistent with the real time, we need some information regarding the time it takes for a message carrying a proposal to reach all its (correct) destinations. More precisely, the maximum delay for delivering a proposal to its destinations allows defining a lower bound, a minimum time that a correct process assigns to proposal. While minimum delay for delivering a proposal to a destination allows defining an upper bound, the maximum time assigned to a proposal.
[PBTS-MSG-DELAY.0]
There exists a system parameter MSGDELAY for end-to-end delays of proposal messages,
such for any two correct processes p and q:
- If
psends a proposal messagemat real timetandqreceivesmat real timet', thent <= t' <= t + MSGDELAY.
Notice that, as a system parameter, MSGDELAY should be observed for any
proposal message broadcast by correct processes: it is a worst-case parameter.
As message delays depends on the message size, the above requirement implicitly
indicates that the size of proposal messages is either fixed or upper bounded.
Problem Statement
In this section we define the properties of Tendermint consensus (cf. the arXiv paper) in this system model.
[PBTS-PROPOSE.0]
A proposer proposes a consensus value v that includes a proposal time
v.time.
We then restrict the allowed decisions along the following lines:
[PBTS-INV-AGREEMENT.0]
- [Agreement] No two correct processes decide on different values
v.
This implies that no two correct processes decide on different proposal times
v.time.
[PBTS-INV-VALID.0]
- [Validity] If a correct process decides on value
v, thenvsatisfies a predefinedvalidpredicate.
With respect to PBTS, the valid predicate requires proposal times to be
monotonic over heights of
consensus:
[PBTS-INV-MONOTONICITY.0]
- If a correct process decides on value
vat the heighthof consensus, thus settingdecision[h] = v, thenv.time > decision[h'].timefor all previous heightsh' < h.
The monotonicity of proposal times, and external validity in general, implicitly assumes that heights of consensus are executed in order.
[PBTS-INV-TIMELY.0]
- [Time-Validity] If a correct process decides on value
v, then the proposal timev.timewas consideredtimelyby at least one correct process.
PBTS introduces a timely predicate that restricts the allowed decisions based
on the proposal time v.time associated with a proposed value v.
As a synchronous predicate, the time at which it is evaluated impacts on
whether a process accepts or reject a proposal time.
For this reason, the Time-Validity property refers to the previous evaluation
of the timely predicate, detailed in the following section.
Protocol Analysis - Timely proposals
For PBTS, a proposal is a tuple (v, v.time, v.round), where:
vis the proposed value;v.timeis the associated proposal time;v.roundis the round at whichvwas first proposed.
We include the proposal round v.round in the proposal definition because a
value v and its associated proposal time v.time can be proposed in multiple
rounds, but the evaluation of the timely predicate is only relevant at round
v.round.
Considering the algorithm in the arXiv paper, a new proposal is produced by the
getValue()method, invoked by the proposerpof roundround_pwhen starting its proposing round with a nilvalidValue_p. The first round at which a valuevis proposed is then the round at which the proposal forvwas produced, and broadcast in aPROPOSALmessage withvr = -1.
[PBTS-PROPOSAL-RECEPTION.0]
The timely predicate is evaluated when a process receives a proposal.
More precisely, let p be a correct process:
proposalReceptionTime(p,r)is the timepreads from its local clock whenpis at roundrand receives the proposal of roundr.
[PBTS-TIMELY.0]
The proposal (v, v.time, v.round) is considered timely by a correct process
p if:
proposalReceptionTime(p,v.round)is set, andproposalReceptionTime(p,v.round) >= v.time - PRECISION, andproposalReceptionTime(p,v.round) <= v.time + MSGDELAY + PRECISION.
A correct process at round v.round only sends a PREVOTE for v if the
associated proposal time v.time is considered timely.
Considering the algorithm in the arXiv paper, the
timelypredicate is evaluated by a processpwhen it receives a validPROPOSALmessage from the proposer of the current roundround_pwithvr = -1.
Timely Proof-of-Locks
A Proof-of-Lock is a set of PREVOTE messages of round of consensus for the
same value from processes whose cumulative voting power is at least 2f + 1.
We denote as POL(v,r) a proof-of-lock of value v at round r.
For PBTS, we are particularly interested in the POL(v,v.round) produced in
the round v.round at which a value v was first proposed.
We call it a timely proof-of-lock for v because it can only be observed
if at least one correct process considered it timely:
[PBTS-TIMELY-POL.0]
If
- there is a valid
POL(v,r)withr = v.round, and POL(v,v.round)contains aPREVOTEmessage from at least one correct process,
Then, let p is a such correct process:
preceived aPROPOSALmessage of roundv.round, and- the
PROPOSALmessage contained a proposal(v, v.time, v.round), and pwas in roundv.roundand evaluated the proposal timev.timeastimely.
The existence of a such correct process p is guaranteed provided that the
voting power of Byzantine processes is bounded by 2f.
Derived Proof-of-Locks
The existence of POL(v,r) is a requirement for the decision of v at round
r of consensus.
At the same time, the Time-Validity property establishes that if v is decided
then a timely proof-of-lock POL(v,v.round) must have been produced.
So, we need to demonstrate here that any valid POL(v,r) is either a timely
proof-of-lock or it is derived from a timely proof-of-lock:
[PBTS-DERIVED-POL.0]
If
- there is a valid
POL(v,r), and POL(v,r)contains aPREVOTEmessage from at least one correct process,
Then
- there is a valid
POL(v,v.round)withv.round <= rwhich is a timely proof-of-lock.
The above relation is trivially observed when r = v.round, as POL(v,r) must
be a timely proof-of-lock.
Notice that we cannot have r < v.round, as v.round is defined as the first
round at which v was proposed.
For r > v.round we need to demonstrate that if there is a valid POL(v,r),
then a timely POL(v,v.round) was previously obtained.
We observe that a condition for observing a POL(v,r) is that the proposer of
round r has broadcast a PROPOSAL message for v.
As r > v.round, we can affirm that v was not produced in round r.
Instead, by the protocol operation, v was a valid value for the proposer of
round r, which means that if the proposer has observed a POL(v,vr) with vr < r.
The above operation considers a correct proposer, but since a POL(v,r) was
produced (by hypothesis) we can affirm that at least one correct process (also)
observed a POL(v,vr).
Considering the algorithm in the arXiv paper,
vwas proposed by the proposerpof roundround_pbecause itsvalidValue_pvariable was set tov. ThePROPOSALmessage broadcast by the proposer, in this case, hadvr > -1, and it could only be accepted by processes that also observed aPOL(v,vr).
Thus, if there is a POL(v,r) with r > v.round, then there is a valid
POL(v,vr) with v.round <= vr < r.
If vr = v.round then POL(vr,v) is a timely proof-of-lock and we are done.
Otherwise, there is another valid POL(v,vr') with v.round <= vr' < vr,
and the above reasoning can be recursively applied until we get vr' = v.round
and observe a timely proof-of-lock.
Temporal analysis
In this section we present invariants that need be observed for ensuring that PBTS is both safe and live.
In addition to the variables and system parameters already defined, we use
beginRound(p,r) as the value of process p's local clock
when it starts round r of consensus.
Safety
The safety of PBTS requires that if a value v is decided, then at least one
correct process p considered the associated proposal time v.time timely.
Following the definition of timely proposals and
proof-of-locks, we require this condition to be asserted at a specific round of
consensus, defined as v.round:
[PBTS-SAFETY.0]
If
- there is a valid commit
Cfor a valuev Ccontains aPRECOMMITmessage from at least one correct process
then there is a correct process p (not necessarily the same above considered) such that:
beginRound(p,v.round) <= proposalReceptionTime(p,v.round) <= beginRound(p,v.round+1)andproposalReceptionTime (p,v.round) - MSGDELAY - PRECISION <= v.time <= proposalReceptionTime(p,v.round) + PRECISION
That is, a correct process p started round v.round and, while still at
round v.round, received a PROPOSAL message from round v.round proposing
v.
Moreover, the reception time of the original proposal for v, according with
p's local clock, enabled p to consider the proposal time v.time as
timely.
This is the requirement established by PBTS for issuing a PREVOTE for the
proposal (v, v.time, v.round), so for the eventual decision of v.
Liveness
The liveness of PBTS relies on correct processes accepting proposal times
assigned by correct proposers.
We thus present a set of conditions for assigning a proposal time v.time so
that every correct process should be able to issue a PREVOTE for v.
[PBTS-LIVENESS.0]
If
- the proposer of a round
rof consensus is correct - and it proposes a value
vfor the first time, with associated proposal timev.time
then the proposal (v, v.time, r) is accepted by every correct process provided that:
min{p is correct : beginRound(p,r)} <= v.time <= max{p is correct : beginRound(p,r)}andmax{p is correct : beginRound(p,r)} <= v.time + MSGDELAY + PRECISION <= min{p is correct : beginRound(p,r+1)}
The first condition establishes a range of safe proposal times v.time for round r.
This condition is trivially observed if a correct proposer p sets v.time to the time it
reads from its clock when starting round r and proposing v.
A PROPOSAL message sent by p at local time v.time should not be received
by any correct process before its local clock reads v.time - PRECISION, so
that condition 2 of [PBTS-TIMELY.0] is observed.
The second condition establishes that every correct process should start round
v.round at a local time that allows v.time to still be considered timely,
according to condition 3. of [PBTS-TIMELY.0].
In addition, it requires correct processes to stay long enough in round
v.round so that they can receive the PROPOSAL message of round v.round.
It assumed here that the proposer of v broadcasts a PROPOSAL message at
time v.time, according to its local clock, so that every correct process
should receive this message by time v.time + MSGDELAY + PRECISION, according
to their local clocks.
Back to main document.
PBTS: Protocol Specification (first draft)
This historical Tendermint draft is superseded by the newer PBTS protocol draft. For current Tenderdash behavior, see Proposal timestamps and time rules and proposal validity.
PBTS: System Model and Properties (first draft)
This historical Tendermint draft is superseded by the newer PBTS system model and properties. For the current Tenderdash specification, see the consensus system model and proposal timestamps and time rules.
Proposer-Based Time (first draft)
Historical and outdated. This Tendermint draft is retained for its original comparison with BFT Time, including aggregate-signature tradeoffs, which the newer protocol draft and newer system model do not reproduce. Its proposed algorithm is superseded by those drafts. For current Tenderdash behavior, use Proposal timestamps and time rules and proposal validity.
Current BFTTime
Description
In Tendermint consensus, the first version of how time is computed and stored in a block works as follows:
- validators send their current local time as part of
precommitmessages - upon collecting the
precommitmessages that the proposer uses to build a commit to be put in the next block, the proposer computes thetimeof the next block as the median (weighted over voting power) of the times in theprecommitmessages.
Analysis
- Fault tolerance. The computed median time is called
bfttimeas it is indeed fault-tolerant: if less than a third of the validators is faulty (counted in voting power), it is guaranteed that the computed time lies between the minimum and the maximum times sent by correct validators. - Effect of faulty validators. If more than
1/2of the voting power (which is in fact more than one third and less than two thirds of the voting power) is held by faulty validators, then the time is under total control of the faulty validators. (This is particularly challenging in the context of lightclient security.) - Proposer influence on block time. The proposer of the next block has a degree of freedom in choosing the
bfttime, since it computes the median time based on the timestamps fromprecommitmessages sent by2f + 1correct validators.- If there are
ndifferent timestamps in theprecommitmessages, the proposer can use any subset of timestamps that add up to2f + 1of the voting power in order to compute the median. - If the validators decide in different rounds, the proposer can decide on which round the median computation is based.
- If there are
- Liveness. The liveness of the protocol:
- does not depend on clock synchronization,
- depends on bounded message delays.
- Relation to real time. There is no clock synchronizaton, which implies that there is no relation between the computed block
timeand real time. - Aggregate signatures. As the
precommitmessages contain the local times, all theseprecommitmessages typically differ in the time field, which prevents the use of aggregate signatures.
Suggested Proposer-Based Time
Outline
An alternative approach to time has been discussed: Rather than having the validators send the time in the precommit messages, the proposer in the consensus algorithm sends its time in the propose message, and the validators locally check whether the time is OK (by comparing to their local clock).
This proposed solution adds the requirement of having synchronized clocks, and other implicit assumptions.
Comparison of the Suggested Method to the Old One
- Fault tolerance. Maintained in the suggested protocol.
- Effect of faulty validators. Eliminated in the suggested protocol,
that is, the block
timecan be corrupted only in the extreme case when>2/3of the validators are faulty. - Proposer influence on block time. The proposer of the next block
has less freedom when choosing the block time.
- This scenario is eliminated in the suggested protocol, provided that there are
<1/3faulty validators. - This scenario is still there.
- This scenario is eliminated in the suggested protocol, provided that there are
- Liveness. The liveness of the suggested protocol:
- depends on the introduced assumptions on synchronized clocks (see below),
- still depends on the message delays (unavoidable).
- Relation to real time. We formalize clock synchronization, and obtain a well-defined relation between the block
timeand real time. - Aggregate signatures. The
precommitmessages free of time, which allows for aggregate signatures.
Protocol Overview
Proposed Time
We assume that the field proposal in the PROPOSE message is a pair (v, time), of the proposed consensus value v and the proposed time time.
Reception Step
In the reception step at node p at local time now_p, upon receiving a message m:
- if the message
mis of typePROPOSEand satisfiesnow_p - PRECISION < m.time < now_p + PRECISION + MSGDELAY, then mark the message astimely.
(PRECISIONandMSGDELAYbeing system parameters, see below)
after the presentation in the dev session, we realized that different semantics for the reception step is closer aligned to the implementation. Instead of dropping propose messages, we keep all of them, and mark timely ones.
Processing Step
- Start round
| arXiv paper | Proposer-based time |
|---|---|
|
|
- Rule on lines 28-35
| arXiv paper | Proposer-based time |
|---|---|
|
|
- Rule on lines 49-54
| arXiv paper | Proposer-based time |
|---|---|
|
|
- Other rules are extended in a similar way, or remain unchanged
Property Overview
Safety and Liveness
For safety (Point 1, Point 2, Point 3i) and liveness (Point 4) we need the following assumptions:
- There exists a system parameter
PRECISIONsuch that for any two correct validatorsVandW, and at any real-timet, their local timesC_V(t)andC_W(t)differ by less thanPRECISIONtime units, i.e.,|C_V(t) - C_W(t)| < PRECISION - The message end-to-end delay between a correct proposer and a correct validator (for
PROPOSEmessages) is less thanMSGDELAY.
Relation to Real-Time
For analyzing real-time safety (Point 5), we use a system parameter ACCURACY, such that for all real-times t and all correct validators V, we have | C_V(t) - t | < ACCURACY.
ACCURACYis not necessarily visible at the code level. We might even viewACCURACYas variable over time. The smaller it is during a consensus instance, the closer the block time will be to real-time.Note that
PRECISIONandMSGDELAYshow up in the code.
Detailed Specification
This specification describes the changes needed to be done to the Tendermint consensus algorithm as described in the arXiv paper and the simplified specification in TLA+, and makes precise the underlying assumptions and the required properties.
Proposer Selection Procedure
This document specifies how Tenderdash chooses the proposer of a round, as
implemented in internal/consensus/versioned/selectproposer. Tenderdash does
not use Tendermint's priority-queue (weighted round-robin) algorithm: all
validators of a Dash quorum have equal voting power, and the proposer is chosen
by rotating an index over the validator set.
Requirements
- R1 Determinism. For a validator set
V, every honest validator computes the sameproposer(h, r). - R2 Fairness. Within a stable validator set, the rounds of one height
rotate through
Vin order, so no validator proposes twice in a height before every other validator has proposed once. Across heights the guarantee is narrower: inv1the rotation continues from one height to the next, so any|V|consecutive elections are fair as long as every block is committed in the round it was first proposed; inv0the same condition makes only the round-0 proposers rotate by one per height, and other elections can repeat a validator early. WithV = A,B,C,D, a height lost at round 0 byAand committed at round 1 with a fresh block byByields the election sequenceA, B, B, C, D; committed at round 2 with a fresh block byC, it yieldsA, B, C, B. A re-proposal also breaks the round-0 rotation: ifBre-proposesA's round-0 block and it is committed at round 1, the next height starts withAagain.
Both follow from the construction below.
Validator order
ValidatorSet.Validators is sorted by ProTxHash ascending
(ValidatorsByProTxHashes, applied in NewValidatorSet and
updateWithChangeSet, types/validator_set.go). The set keeps a
proposerIndex; ValidatorSet.Proposer() returns Validators[proposerIndex],
SetProposer(proTxHash) sets the index to that validator, and
IncProposerIndex(n) adds n (possibly negative) modulo |V|.
Two versions
selectproposer.NewProposerSelector (proposer_selector.go) chooses the
strategy by ConsensusParams.Version.ConsensusVersion:
ConsensusVersion | Strategy | File |
|---|---|---|
| 0 (default) | Height-based: the index advances once per height; the round offset is derived without being persisted | height_proposer.go (heightProposerSelector) |
| 1, 2 | Height-and-round-based: the index advances once per height and once per round. Version 2 keeps this strategy and adds proposer liveness | height_round_proposer.go (heightRoundProposerSelector) |
Changing the version is a consensus-parameter update and takes effect at the next height (Validator-set and parameter changes).
When the index is computed
StateData.updateToState creates a new selector at every height with
NewProposerSelector(params, Validators, h, 0, blockStore). Whenever the
block store is non-empty and contains h − 1 or h, the constructor derives
the index from the store (proposerFromStore, below); it trusts the
validator set's own proposerIndex only when there is no usable store (e.g.
genesis). During the height, StateData.updateRoundStep calls
ProposerSelector.UpdateHeightRound(h, r) on every step change and
GetProposer(h, r) answers from that state.
Because every node derives the index from the same stored block at the start of every height, determinism (R1) does not depend on nodes having tracked the same rounds.
proposerFromStore(h, r)
Let V be the validator set for height h, sorted by ProTxHash, and
meta(x) the stored block metadata for height x (Header.ProposerProTxHash
is the proposer of the round in which the block was first proposed — a
re-proposed block keeps its header — and meta.Round is the round in which
it was committed).
| Case | v0 (height_proposer.go) | v1 (height_round_proposer.go) |
|---|---|---|
h ∈ {0, 1} | V[0] | V[0] |
meta(h) stored, meta(h).ValidatorsHash == hash(V) | idx(meta(h).Proposer) − meta(h).Round | idx(meta(h).Proposer) + r − meta(h).Round |
meta(h) stored, hash differs | error | error |
only meta(h − 1) stored, meta(h − 1).ValidatorsHash == hash(V) | idx(meta(h − 1).Proposer) + 1 − meta(h − 1).Round | idx(meta(h − 1).Proposer) + 1 + r |
only meta(h − 1) stored, hash differs (quorum rotation) | 1 ("index 1 due to a bug in the original code that causes the first validator to never propose; preserved to avoid breaking consensus") | 0 + r |
| neither stored | error | error |
All indices are modulo |V|. For the height in progress only the
meta(h − 1) rows apply.
Within a height
- v0 (
heightProposerSelector.GetProposer):proposer(h, r) = V[(idx₀ + r) mod |V|]whereidx₀is the stored round-0 index; the per-round advance is applied to a copy and never persisted.UpdateHeightRoundignores the round. - v1 (
heightRoundProposerSelector.updateScores): at the same heightIncProposerIndex(r' − r); soproposer(h, r) = V[(idx₀ + r) mod |V|]as well, with the difference that the round offset is persisted in the selector and, in themeta(h)row above, the commit round is subtracted.
Both versions therefore rotate one validator per round. They differ in how
the round-0 index of height h relates to height h − 1:
- v0:
idx₀(h) = idx(meta(h − 1).Proposer) − meta(h − 1).Round + 1— the first proposer of the previous block, rewound by the previous commit round, plus one. When the previous block was first proposed at roundpand committed at roundc > p(a re-proposal), this isp − c + 1positions after the previous round-0 proposer rather than one. - v1:
idx₀(h) = idx(meta(h − 1).Proposer) + 1— the validator after the first proposer of the previous block, independent of the commit round.
Validator set changes
A new validator set (new QuorumHash) replaces the old one at the next
height; meta(h − 1).ValidatorsHash then differs from the new set's hash and
the quorum rotation row applies. Within one quorum, additions and
removals are applied as a change set and the set is re-sorted by ProTxHash
(ValidatorSet.updateWithChangeSet); since the index is recomputed from the
stored proposer's position in the new set at the next height, the rotation
continues from that validator.
Use by consensus
EnterProposeAction.Executedecides whether to propose by comparing the node's pro-tx-hash withGetProposer(h, r).Proposaler.verifyProposalverifies a proposal's signature againstGetProposer(h, r)'s key only.StateData.isValidForPrevoterequiresHeader.ProposerProTxHashto equalGetProposer(h, r)for proposals withPOLRound = -1.
Consensus
Specification of the Tenderdash consensus protocol, derived from the
implementation in internal/consensus.
Contents
- Consensus Protocol — overview, terminology, system model, round state, step state machine and invariants; the entry point.
- Proposer Selection — which validator proposes at a given height and round.
- Signing — sign bytes, request IDs and double-signing rules for proposals and votes.
- Write Ahead Log — the log the consensus state machine uses to recover from crashes.
- Evidence — detection and handling of misbehaviour.
- Proposer-Based Timestamps — pointers to current rules and retained historical research; the rules in force are in Proposal timestamps and time rules.
- BFT Time — historical: the median-based timestamp rule Tenderdash no longer uses.
- Consensus Paper — the Tendermint consensus paper (arxiv) with proofs of safety and termination for the algorithm the implementation descends from.
- Light Client Protocol — deprecated pointers to the light-client specification.
The gossip of consensus messages between peers is summarised in Commit handling and Non-validators; the wire format is in Consensus messages.
Validator Signing
Here we specify the rules for validating a proposal and vote before signing. When a validator signs each message, and what it signs, is specified by the consensus protocol. First we include some general notes on validating data structures common to both types. We then provide specific validation rules for each. Finally, we include validation rules to prevent double-signing.
SignedMsgType
The SignedMsgType is a single byte that refers to the type of the message
being signed. It is defined in Go as follows:
// SignedMsgType is a type of signed message in the consensus.
type SignedMsgType byte
const (
// Votes
PrevoteType SignedMsgType = 0x01
PrecommitType SignedMsgType = 0x02
// Proposals
ProposalType SignedMsgType = 0x20
// Proposer liveness, from CONSENSUS_VERSION_2
ProposerHeartbeatType SignedMsgType = 0x30
RoundSkipType SignedMsgType = 0x31
)
All signed messages must correspond to one of these types.
Timestamp
Votes carry no timestamp. A proposal's Timestamp must equal the proposed
block's Header.Time, which the proposer takes from its local clock (the
genesis time for the initial block). Fresh proposals must be timely with respect
to their receive time and the chain's SynchronyParams before a validator
prevotes their block. The timeliness check is skipped for a locally recorded
valid block and during replay. The rules are specified in
Proposal timestamps and time rules
and the prevote decision in
Transitions and decision rules.
ChainID
ChainID is an unstructured string with a max length of 50-bytes. In the future, the ChainID may become structured, and may take on longer lengths. For now, it is recommended that signers be configured for a particular ChainID, and to only sign votes and proposals corresponding to that ChainID.
BlockID
BlockID contains the block-header hash, PartSetHeader (part count and Merkle
root of serialized block parts), and the state-ID hash. A nil vote has all three
empty. A complete ID has 32-byte block, part-set and state hashes and a positive
part count. Proposals require a complete ID; votes allow nil or complete IDs.
See BlockID and the StateID schema below.
Proposals
The structure of a proposal for signing looks like:
type CanonicalProposal struct {
Type SignedMsgType // type alias for byte
Height int64 `binary:"fixed64"`
Round int64 `binary:"fixed64"`
POLRound int64 `binary:"fixed64"`
BlockID CanonicalBlockID
Timestamp time.Time
ChainID string
}
CanonicalBlockID contains the block hash and canonical part-set header; it
excludes BlockID.StateID. The proposal signature also excludes
CoreChainLockedHeight. The exact signature-covered fields and replacement of
unsigned fields from assembled block bytes are specified in
Creating the proposal
and Receiving the proposal and block.
The basic structural checks on the proposal before canonicalization include:
p.Type == 0x20
p.Height > 0
p.Round >= 0
p.POLRound >= -1
p.BlockID.IsComplete()
In other words, a proposal is valid for signing if it contains the type of a Proposal (0x20), has a positive, non-zero height, a non-negative round, a POLRound not less than -1, and a complete BlockID.
Votes
Sign bytes for votes are defined as CanonicalVote struct.
The canonical message is encoded using Protobuf. Its checksum is one input to the quorum signing digest, not the complete digest verified by BLS; see Block signature verification on light client. The schema is:
message CanonicalVote {
SignedMsgType type = 1; // type alias for byte
sfixed64 height = 2;
sfixed64 round = 3;
// block_id is a checksum (sha256) of CanonicalBlockID for the block being voted on
bytes block_id = 4 [(gogoproto.customname) = "BlockID"];
// state_id is a checksum (sha256) of StateID for the block being voted on
bytes state_id = 5 [(gogoproto.customname) = "StateID"];
string chain_id = 99 [(gogoproto.customname) = "ChainID"];
}
block_id is a sha256 checksum of Protobuf-encoded CanonicalBlockID message:
message CanonicalBlockID {
bytes hash = 1;
CanonicalPartSetHeader part_set_header = 2 [(gogoproto.nullable) = false];
}
message CanonicalPartSetHeader {
uint32 total = 1;
bytes hash = 2;
}
state_id is a sha256 checksum of Protobuf-encoded StateID message:
message StateID {
// AppVersion used when generating the block, equals to Header.Version.App.
fixed64 app_version = 1 [(gogoproto.customname) = "AppVersion"];
// Height of block containing this state ID.
fixed64 height = 2;
// AppHash used in current block, equal to Header.AppHash. 32 bytes.
bytes app_hash = 3 [(gogoproto.customname) = "AppHash"];
// CoreChainLockedHeight for the block, equal to Header.CoreChainLockedHeight.
fixed32 core_chain_locked_height = 4 [(gogoproto.customname) = "CoreChainLockedHeight"];
// Time of the block.
google.protobuf.Timestamp time = 5 [(gogoproto.nullable) = false];
}
A vote is valid if each of the following lines evaluates to true for vote v:
v.Type == 0x1 || v.Type == 0x2
v.Height > 0
v.Round >= 0
v.BlockID.IsNil() || v.BlockID.IsComplete()
In other words, a vote is valid for signing if it contains the type of a Prevote or Precommit (0x1 or 0x2, respectively), has a positive, non-zero height, a non-negative round, and an empty or valid BlockID.
Block signature verification on light client
The block signature in a commit is threshold-recovered from precommit shares
and verified with the quorum's ThresholdPublicKey. Its signing context includes
ChainID, the commit height and round, full BlockID, QuorumType, and
QuorumHash.
Use MakeBlockSignItem or VoteBlockSignID in
types/quorum_sign_data.go and
types/vote.go to construct the digest:
- Build the precommit's canonical vote, including the canonical-block-ID checksum and state-ID hash.
MakeBlockSignItemhashes the canonical vote sign bytes and derivesBlockRequestID(height, round)using thedpbvotedomain.SignItem.UpdateSignHashuses the quorum type, quorum hash, request ID and message hash to construct the Dash BLS signing hash, with the helper's byte-order conversions. Verify the signature against that digest and the threshold key.
Proposal signing uses ProposalBlockSignID and its own dpproposal request-ID
domain, likewise bound to height, round and quorum context
(types/proposal.go). Vote extensions have separate
sign items; application-supplied raw-extension request IDs need not bind height
and round. See Vote extensions.
This describes digest construction, not a complete light-client trust algorithm. Full-node commit acceptance also verifies every extension threshold signature, checks the held block's complete ID, and requires the application's verdict on the ordered commit extension vector; see Commit verification and Commit validity depends on the application. A valid block signature does not authenticate that vector's count and order.
Proposer heartbeats and round skip votes
From CONSENSUS_VERSION_2, the proposer of a round that waits for transactions
signs a heartbeat every proposer_heartbeat, and a validator that hears none
for the round's silence timeout signs a vote to skip the round. Both are signed
like votes, with the validator's quorum key share over the DIP-7 sign hash of
the quorum type, the quorum hash, a request ID, and the SHA256 checksum of the
length-delimited, Protobuf-encoded sign bytes below.
A heartbeat's sign bytes are:
message CanonicalProposerHeartbeat {
SignedMsgType type = 1; // 0x30
sfixed64 height = 2;
sfixed64 round = 3;
fixed64 sequence = 4;
google.protobuf.Timestamp timestamp = 5;
string chain_id = 6;
}
Its request ID is the SHA256 checksum of "dphb", then the height as a
little-endian uint64, the round as a little-endian uint32, and the sequence as
a little-endian uint64. The sequence grows with every heartbeat of a round, so
each heartbeat a proposer sends is a signing request of its own. A proposer
that restarts with its clock gone back can repeat an ID for a heartbeat with
another timestamp; the timestamp is in the sign bytes, so the signature still
differs.
A round skip vote's sign bytes are the same for every validator voting to skip a round, so the votes of +2/3 recover into one threshold signature, the round skip certificate:
message CanonicalRoundSkip {
SignedMsgType type = 1; // 0x31
sfixed64 height = 2;
sfixed64 round = 3;
string chain_id = 4;
}
Its request ID is the SHA256 checksum of "dpskip", then the height as a
little-endian uint64 and the round as a little-endian uint32.
A heartbeat or skip vote is valid for signing if its height is positive and its round is not negative. Neither changes consensus state or conflicts with another message, so neither is subject to the double-signing rules below.
Invalid Votes and Proposals
Basic structure, signature verification and consensus admissibility are separate
checks. Proposal verification failures do not automatically evict their sender;
received commits evict only on ErrInvalidCommitSignature, and replay never
evicts. Budget exhaustion is a local processing limit, not proof of peer fault.
The exact policies are in Proposal acceptance,
Vote admission,
and Commit verification.
Double Signing
Signers must be careful not to sign conflicting messages, also known as "double signing" or "equivocating". Tenderdash has mechanisms to publish evidence of validators that signed conflicting votes, so they can be punished by the application. Note Tenderdash does not currently handle evidence of conflicting proposals, though it may in the future.
State
FilePV persists height, round, internal signing step, block sign bytes and
block signature (FilePVLastSignState in privval/file.go).
Its ordered steps are proposal, prevote and precommit; these are not the numeric
SignedMsgType values. checkHRS rejects regressions and permits a matching
height/round/step only for comparison with the persisted sign bytes.
Proposals
A new proposal signing request must pass the height/round/step checks. Conflicting canonical bytes at the same height/round/proposal step are refused. The current repeated-proposal return path has a documented signature-field discrepancy; it must not be assumed to return a valid reused signature.
Votes
For a new height or round, or a later internal step in the same height/round,
FilePV can sign and persist the vote. For identical block sign bytes at an
already-signed height/round/step, it reuses the stored block signature; different
bytes are refused. Precommit extension signatures are regenerated, since the
extensions are not part of the canonical block vote.
Nil and non-nil block IDs are different sign bytes: a signer cannot replace its
nil vote with a block vote of the same type at the same height and round.
The block-signature reuse rule permits recovery of an identical vote, not a second
conflicting vote. See WAL catch-up.
DashCoreSignerClient does not maintain FilePV's persisted state inside
Tenderdash; signer-side protection must not be inferred from the presence of the
client alone.
Other Rules
Signer height/round/step protection does not establish lock eligibility. The consensus state machine decides whether a block may be prevoted or precommitted; see Locking and the precommit decision. A nil polka does not unlock within a height, and relocking a block requires this round's proposal and application processing context.
Historical Tendermint discussions call unjustified lock changes “amnesia”; see the original fork-detection discussion. That history does not introduce an unlock transition or additional signer protection into the current implementation.
WAL
Consensus module writes every message to the WAL (write-ahead log), except proposer heartbeats and round skip votes; round skip certificates are written once verified. Which records are written, with which durability, and how they are replayed is specified in Recovery and synchronisation.
It also issues fsync syscall through File#Sync for messages signed by this node and for the end-of-height marker (to prevent double signing and to delimit heights).
Under the hood, it uses autofile.Group (internal/libs/autofile),
which rotates files when those get too big (> 10MB, defaultHeadSizeLimit).
The total maximum size is 1GB (defaultTotalSizeLimit). We only need the
latest block and the block before it, but if the former is dragging on across
many rounds, we want all those rounds.
Replay
Consensus module will replay all the messages of the last height written to WAL before a crash (if such occurs).
The private validator may try to sign messages during replay because it runs somewhat autonomously and does not know about replay process.
For example, if we got all the way to precommit in the WAL and then crash,
after we replay the proposal message, the private validator will try to sign a
prevote. But it will fail. That's ok because we’ll see the prevote later in the
WAL. Then it will go to precommit, and that time it will work because the
private validator contains the LastSignBytes and then we’ll replay the
precommit from the WAL.
Make sure to read about WAL corruption and recovery strategies.
P2P Config
Here we describe configuration options around the Peer Exchange.
These can be set using flags or via the $TMHOME/config/config.toml file.
Seed Mode
--p2p.seed_mode
The node operates in seed mode. In seed mode, a node continuously crawls the network for peers, and upon incoming connection shares some peers and disconnects.
Seeds
--p2p.seeds “id100000000000000000000000000000000@1.2.3.4:26656,id200000000000000000000000000000000@2.3.4.5:4444”
Dials these seeds when we need more peers. They should return a list of peers and then disconnect. If we already have enough peers in the address book, we may never need to dial them.
Persistent Peers
--p2p.persistent_peers “id100000000000000000000000000000000@1.2.3.4:26656,id200000000000000000000000000000000@2.3.4.5:26656”
Dial these peers and auto-redial them if the connection fails. These are intended to be trusted persistent peers that can help anchor us in the p2p network. The auto-redial uses exponential backoff and will give up after a day of trying to connect.
But If persistent_peers_max_dial_period is set greater than zero,
pause between each dial to each persistent peer will not exceed persistent_peers_max_dial_period
during exponential backoff and we keep trying again without giving up
Note: If seeds and persistent_peers intersect,
the user will be warned that seeds may auto-close connections
and that the node may not be able to keep the connection persistent.
Private Peers
--p2p.private_peer_ids “id100000000000000000000000000000000,id200000000000000000000000000000000”
These are IDs of the peers that we do not add to the address book or gossip to other peers. They stay private to us.
Unconditional Peers
--p2p.unconditional_peer_ids “id100000000000000000000000000000000,id200000000000000000000000000000000”
These are IDs of the peers which are allowed to be connected by both inbound or outbound regardless of
max_num_inbound_peers or max_num_outbound_peers of user's node reached or not.
P2P Multiplex Connection
MConnection
MConnection is a multiplex connection that supports multiple independent streams
with distinct quality of service guarantees atop a single TCP connection.
Each stream is known as a Channel and each Channel has a globally unique byte id.
Each Channel also has a relative priority that determines the quality of service
of the Channel compared to other Channels.
The byte id and the relative priorities of each Channel are configured upon
initialization of the connection.
The MConnection supports three packet types:
- Ping
- Pong
- Msg
Ping and Pong
The ping and pong messages consist of writing a single byte to the connection; 0x1 and 0x2, respectively.
When we haven't received any messages on an MConnection in time pingTimeout, we send a ping message.
When a ping is received on the MConnection, a pong is sent in response only if there are no other messages
to send and the peer has not sent us too many pings (TODO).
If a pong or message is not received in sufficient time after a ping, the peer is disconnected from.
Msg
Messages in channels are chopped into smaller msgPackets for multiplexing.
type msgPacket struct {
ChannelID byte
EOF byte // 1 means message ends here.
Bytes []byte
}
The msgPacket is serialized using Proto3.
The received Bytes of a sequential set of packets are appended together
until a packet with EOF=1 is received, then the complete serialized message
is returned for processing by the onReceive function of the corresponding channel.
Multiplexing
Messages are sent from a single sendRoutine, which loops over a select statement and results in the sending
of a ping, a pong, or a batch of data messages. The batch of data messages may include messages from multiple channels.
Message bytes are queued for sending in their respective channel, with each channel holding one unsent message at a time.
Messages are chosen for a batch one at a time from the channel with the lowest ratio of recently sent bytes to channel priority.
Sending Messages
There are two methods for sending messages:
func (m MConnection) Send(chID byte, msg interface{}) bool {}
func (m MConnection) TrySend(chID byte, msg interface{}) bool {}
Send(chID, msg) is a blocking call that waits until msg is successfully queued
for the channel with the given id byte chID. The message msg is serialized
using the tendermint/go-amino submodule's WriteBinary() reflection routine.
TrySend(chID, msg) is a nonblocking call that queues the message msg in the channel
with the given id byte chID if the queue is not full; otherwise it returns false immediately.
Send() and TrySend() are also exposed for each Peer.
Peer
Each peer has one MConnection instance, and includes other information such as whether the connection
was outbound, whether the connection should be recreated if it closes, various identity information about the node,
and other higher level thread-safe data used by the reactors.
Switch/Reactor
The Switch handles peer connections and exposes an API to receive incoming messages
on Reactors. Each Reactor is responsible for handling incoming messages of one
or more Channels. So while sending outgoing messages is typically performed on the peer,
incoming messages are received on the reactor.
// Declare a MyReactor reactor that handles messages on MyChannelID.
type MyReactor struct{}
func (reactor MyReactor) GetChannels() []*ChannelDescriptor {
return []*ChannelDescriptor{ChannelDescriptor{ID:MyChannelID, Priority: 1}}
}
func (reactor MyReactor) Receive(chID byte, peer *Peer, msgBytes []byte) {
r, n, err := bytes.NewBuffer(msgBytes), new(int64), new(error)
msgString := ReadString(r, n, err)
fmt.Println(msgString)
}
// Other Reactor methods omitted for brevity
...
switch := NewSwitch([]Reactor{MyReactor{}})
...
// Send a random message to all outbound connections
for _, peer := range switch.Peers().List() {
if peer.IsOutbound() {
peer.Send(MyChannelID, "Here's a random message")
}
}
Messages
An implementation of the spec consists of many components. While many parts of these components are implementation specific, the p2p messages are not. In this section we will be covering all the p2p messages of components.
There are two parts to the P2P messages, the message and the channel. The channel is message specific and messages are specific to components of Tenderdash. When a node connect to a peer it will tell the other node which channels are available. This notifies the peer what services the connecting node offers. You can read more on channels in connection.md
Block Sync
Channel
Block sync has one channel.
| Name | Number |
|---|---|
| BlockchainChannel | 64 |
Message Types
There are multiple message types for Block Sync
BlockRequest
BlockRequest asks a peer for a block at the height specified.
| Name | Type | Description | Field Number |
|---|---|---|---|
| Height | int64 | Height of requested block | 1 |
NoBlockResponse
NoBlockResponse notifies the peer requesting a block that the node does not contain it.
| Name | Type | Description | Field Number |
|---|---|---|---|
| Height | int64 | Height of requested block | 1 |
BlockResponse
BlockResponse contains the block requested and its commit.
| Name | Type | Description | Field Number |
|---|---|---|---|
| Block | bytes | Requested Block, serialized as it was committed: the bytes of its parts, in order. On the wire this is the same as an embedded Block. | 1 |
| Commit | Commit | Commit for the requested block | 2 |
The receiver cuts Block into parts of 64kB, the last one shorter, and checks
the part set header it computes against the one in the commit's block ID.
Block is served as stored, because decoding and re-encoding the block can
change its bytes.
Block sync rebuilds only this standard cut, which honest proposers use.
Consensus does not require it yet: it accepts parts of other sizes, and proofs
of a tree larger than the header counts. A block committed that way fails the
commit check (wrong block ID) whichever peer serves it, and block sync does
not get past its height.
StatusRequest
StatusRequest is an empty message that notifies the peer to respond with the highest and lowest blocks it has stored.
Empty message.
StatusResponse
StatusResponse responds to a peer with the highest and lowest block stored.
| Name | Type | Description | Field Number |
|---|---|---|---|
| Height | int64 | Current Height of a node | 1 |
| base | int64 | First known block, if pruning is enabled it will be higher than 1 | 1 |
Message
Message is a oneof protobuf type. The oneof consists of five messages.
| Name | Type | Description | Field Number |
|---|---|---|---|
| block_request | BlockRequest | Request a block from a peer | 1 |
| no_block_response | NoBlockResponse | Response saying it doe snot have the requested block | 2 |
| block_response | BlockResponse | Response with requested block | 3 |
| status_request | StatusRequest | Request the highest and lowest block numbers from a peer | 4 |
| status_response | StatusResponse | Response with the highest and lowest block numbers the store | 5 |
Consensus
Wire format of the messages exchanged by the consensus reactor. How a node acts on them is specified in the consensus protocol.
Channel
Consensus has four separate channels. The channel identifiers are listed below.
| Name | Number |
|---|---|
| StateChannel | 32 |
| DataChannel | 33 |
| VoteChannel | 34 |
| VoteSetBitsChannel | 35 |
Message Types
Proposal
Proposal is sent when a new block is proposed. It is a suggestion of what the next block in the blockchain should be.
| Name | Type | Description | Field Number |
|---|---|---|---|
| proposal | Proposal | Proposed Block to come to consensus on | 1 |
Vote
Vote is sent to vote for some block (or to inform others that a process does not vote in the
current round). Vote contains the validator's
information (ProTxHash and index), height and round for which the vote is sent, vote type,
blockID if process vote for some block (nil otherwise), the block signature and, on a non-nil
precommit, the vote extensions. The message is signed by the validator private key.
| Name | Type | Description | Field Number |
|---|---|---|---|
| vote | Vote | Vote for a proposed Block | 1 |
BlockPart
BlockPart is sent when gossiping a piece of the proposed block. It contains height, round and the block part.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of corresponding block. | 1 |
| round | int32 | Round of voting to finalize the block. | 2 |
| part | Part | A part of the block. | 3 |
NewRoundStep
NewRoundStep is sent for every step transition during the core consensus algorithm execution.
It is used in the gossip part of the Tenderdash protocol to inform peers about a current
height/round/step a process is in. It is also sent to each peer when the peer connects and
then periodically, at the peer-query-maj23-sleep-duration cadence, so that a peer whose
copy was lost in transit learns the current step without waiting for the next transition.
A receiver ignores a copy that does not advance the height, round or step it already holds
for the sender.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of corresponding block | 1 |
| round | int32 | Round of voting to finalize the block. | 2 |
| step | uint32 | 3 | |
| seconds_since_start_time | int64 | 4 | |
| last_commit_round | int32 | 5 |
NewValidBlock
NewValidBlock advertises a block-part download target: its height, round,
part-set header, and the parts the node already holds. is_commit identifies a
commit target. Consensus emits this event for a current-round polka or a verified
commit even when the proposal or complete block is not held. It is a gossip
announcement, not proof that the sender has processed a valid proposal. While the
node holds block parts for its round, the periodic per-peer NewRoundStep described
above is followed by a NewValidBlock for the same height and round.
See valid-block bookkeeping
and commit handling.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of corresponding block | 1 |
| round | int32 | Round of voting to finalize the block. | 2 |
| block_part_set_header | PartSetHeader | 3 | |
| block_parts | BitArray | 4 | |
| is_commit | bool | 5 |
ProposalPOL
ProposalPOL is sent when a previous block is re-proposed. It is used to inform peers in what round the process learned for this block (ProposalPOLRound), and what prevotes for the re-proposed block the process has.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of corresponding block | 1 |
| proposal_pol_round | int32 | 2 | |
| proposal_pol | bitarray | 3 |
HasVote
HasVote is sent to indicate that a particular vote has been received. It contains height, round, vote type and the index of the validator that is the originator of the corresponding vote.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of corresponding block | 1 |
| round | int32 | Round of voting to finalize the block. | 2 |
| type | SignedMsgType | 3 | |
| index | int32 | 4 |
Commit
Commit is sent on the VoteChannel to deliver a threshold-signed commit for a block.
| Name | Type | Description | Field Number |
|---|---|---|---|
| commit | Commit | Commit for a decided block | 1 |
HasCommit
HasCommit is sent to indicate that a process holds the commit for a given height and round.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of the commit | 1 |
| round | int32 | Round the commit belongs to | 2 |
VoteSetMaj23
VoteSetMaj23 is sent to indicate that a process has seen +2/3 votes for some BlockID. It contains height, round, vote type and the BlockID.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of corresponding block | 1 |
| round | int32 | Round of voting to finalize the block. | 2 |
| type | SignedMsgType | 3 | |
| block_id | BlockID | 4 |
VoteSetBits
VoteSetBits is sent to communicate the bit-array of votes a process has seen for a given BlockID. It contains height, round, vote type, BlockID and a bit array of the votes a process has.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of corresponding block | 1 |
| round | int32 | Round of voting to finalize the block. | 2 |
| type | SignedMsgType | 3 | |
| block_id | BlockID | 4 | |
| votes | BitArray | Votes the process has for the BlockID. | 5 |
ProposerHeartbeat
ProposerHeartbeat is sent on the StateChannel by the proposer of a round that
waits for transactions, and relayed once by validators that verify it
(CONSENSUS_VERSION_2; see
Proposer liveness).
| Name | Type | Description | Field Number |
|---|---|---|---|
| heartbeat | ProposerHeartbeat | The signed heartbeat. | 1 |
ProposerHeartbeat contents
tendermint.types.ProposerHeartbeat:
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of the round. | 1 |
| round | int32 | Round the sender proposes in. | 2 |
| sequence | uint64 | Grows with every heartbeat of the round, counting from the signing time in milliseconds. | 3 |
| proposer_pro_tx_hash | bytes | ProTxHash of the proposer. | 4 |
| timestamp | google.protobuf.Timestamp | Signing time. | 5 |
| signature | bytes | Signature with the proposer's quorum key share (Signing). | 6 |
RoundSkipVote
RoundSkipVote is sent on the StateChannel by a validator that heard no
heartbeat from the proposer of its waiting round for the silence timeout, and
relayed by validators when they first count it and again on a later copy until
the round is certified (CONSENSUS_VERSION_2; see
Collecting votes).
| Name | Type | Description | Field Number |
|---|---|---|---|
| vote | RoundSkipVote | The signed skip vote. | 1 |
RoundSkipVote contents
tendermint.types.RoundSkipVote:
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of the round to skip. | 1 |
| round | int32 | Round to skip. | 2 |
| validator_pro_tx_hash | bytes | ProTxHash of the voting validator. | 3 |
| validator_index | int32 | Index of the voting validator in the validator set. | 4 |
| signature | bytes | Signature share over the round's skip sign bytes, the same for every validator. | 5 |
RoundSkipCertificate
RoundSkipCertificate is sent on the StateChannel by a node that recovered or
received a valid certificate, and resent by gossip to peers at its height that
have not sent one for that round or a later one (CONSENSUS_VERSION_2).
| Name | Type | Description | Field Number |
|---|---|---|---|
| certificate | RoundSkipCertificate | The round skip certificate. | 1 |
RoundSkipCertificate contents
tendermint.types.RoundSkipCertificate:
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | int64 | Height of the skipped round. | 1 |
| round | int32 | Skipped round. | 2 |
| quorum_hash | bytes | Quorum that signed. | 3 |
| threshold_signature | bytes | Quorum threshold signature recovered from the skip votes of +2/3 of the validator set (the mainnet default; the quorum type and VotingPowerThreshold can change it). | 4 |
Message
Message is a oneof protobuf type.
| Name | Type | Description | Field Number |
|---|---|---|---|
| new_round_step | NewRoundStep | 1 | |
| new_valid_block | NewValidBlock | 2 | |
| proposal | Proposal | 3 | |
| proposal_pol | ProposalPOL | 4 | |
| block_part | BlockPart | 5 | |
| vote | Vote | 6 | |
| has_vote | HasVote | 7 | |
| vote_set_maj23 | VoteSetMaj23 | 8 | |
| vote_set_bits | VoteSetBits | 9 | |
| commit | Commit | 10 | |
| has_commit | HasCommit | 11 | |
| proposer_heartbeat | ProposerHeartbeat | 12 | |
| round_skip_vote | RoundSkipVote | 13 | |
| round_skip_certificate | RoundSkipCertificate | 14 |
Evidence
Channel
Evidence has one channel. The channel identifier is listed below.
| Name | Number |
|---|---|
| EvidenceChannel | 56 |
Message Types
Evidence
Verified evidence that has already been propagated throughout the network. This evidence will appear within the EvidenceList struct of a block as well.
| Name | Type | Description | Field Number |
|---|---|---|---|
| evidence | Evidence | Valid evidence | 1 |
Mempool
Channel
Mempool has one channel. The channel identifier is listed below.
| Name | Number |
|---|---|
| MempoolChannel | 48 |
Message Types
There is currently only one message that Mempool broadcasts and receives over
the p2p gossip network (via the reactor): TxsMessage
Txs
A list of transactions. These transactions have been checked against the application for validity. This does not mean that the transactions are valid, it is up to the application to check this.
| Name | Type | Description | Field Number |
|---|---|---|---|
| txs | repeated bytes | List of transactions | 1 |
Message
Message is a oneof protobuf type. The one of consists of one message Txs.
| Name | Type | Description | Field Number |
|---|---|---|---|
| txs | Txs | List of transactions | 1 |
Peer Exchange
Channels
Pex has one channel. The channel identifier is listed below.
| Name | Number |
|---|---|
| PexChannel | 0 |
Message Types
PexRequest
PexRequest is an empty message requesting a list of peers.
EmptyRequest
PexResponse
PexResponse is an list of net addresses provided to a peer to dial.
| Name | Type | Description | Field Number |
|---|---|---|---|
| addresses | repeated PexAddress | List of peer addresses available to dial | 1 |
PexAddress
PexAddress provides needed information for a node to dial a peer. This is in the form of a URL that gets parsed
into a NodeAddress. See ParseNodeAddress for more details.
| Name | Type | Description | Field Number |
|---|---|---|---|
| url | string | See golang url | 1 |
Message
Message is a oneof protobuf type. The one of consists of two messages.
| Name | Type | Description | Field Number |
|---|---|---|---|
| pex_request | PexRequest | Empty request asking for a list of addresses to dial | 3 |
| pex_response | PexResponse | List of addresses to dial | 4 |
State Sync
Channels
State sync has four distinct channels. The channel identifiers are listed below.
| Name | Number |
|---|---|
| SnapshotChannel | 96 |
| ChunkChannel | 97 |
| LightBlockChannel | 98 |
| ParamsChannel | 99 |
Snapshot and chunk model
Tenderdash uses a content-addressed chunk model that differs from upstream
Tendermint. A snapshot is identified by its height, an application-specific
version, and a hash; there is no format field and no up-front chunk
count. Chunks are identified by an opaque, application-defined chunk_id
(in practice the hash of the corresponding node or subtree of the snapshot)
rather than by a sequential index.
Chunk discovery is driven by the ABCI application during restoration:
- The syncer requests the first chunk using the snapshot
hashas the initialchunk_id. - Each applied chunk is handed to the application via
ApplySnapshotChunk; the response'snext_chunksfield lists the chunk IDs to fetch next, andrefetch_chunkscan ask for chunks to be re-fetched. - Restoration terminates when the application returns the
COMPLETE_SNAPSHOTresult — not when a predefined number of chunks has been applied.
Message Types
SnapshotsRequest
When a new node begins state syncing, it will ask all peers it encounters if they have any available snapshots. The message has no fields:
| Name | Type | Description | Field Number |
|---|
SnapshotsResponse
The receiver will query the local ABCI application via ListSnapshots, and
send a message containing snapshot metadata (limited to 4 MB) for each of the
10 most recent snapshots:
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | uint64 | Height at which the snapshot was taken | 1 |
| version | uint32 | Application-specific snapshot version | 2 |
| hash | bytes | Snapshot hash; also the ID of the first chunk to request | 3 |
| metadata | bytes | Arbitrary application data. May be non-deterministic. | 4 |
ChunkRequest
The node running state sync will offer these snapshots to the local ABCI
application via OfferSnapshot ABCI calls, and keep track of which peers
contain which snapshots. Once a snapshot is accepted, the state syncer will
request snapshot chunks from appropriate peers:
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | uint64 | Height at which the snapshot was taken | 1 |
| version | uint32 | Application-specific snapshot version | 2 |
| chunk_id | bytes | Content-addressed ID of the requested chunk | 3 |
ChunkResponse
The receiver will load the requested chunk from its local application via
LoadSnapshotChunk, and respond with it (limited to 16 MB):
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | uint64 | Height at which the snapshot was taken | 1 |
| version | uint32 | Application-specific snapshot version | 2 |
| chunk_id | bytes | Content-addressed ID of the chunk | 3 |
| chunk | bytes | Binary chunk contents | 4 |
| missing | bool | True if the chunk was not found on the peer | 5 |
Here, missing is used to signify that the chunk was not found on the peer,
since an empty chunk is a valid (although unlikely) response.
The returned chunk is given to the ABCI application via ApplySnapshotChunk.
The application response determines what happens next: next_chunks lists
further chunk IDs to fetch, refetch_chunks requests re-fetching of chunks,
reject_senders requests peer bans, and the COMPLETE_SNAPSHOT result ends
the restoration. If a chunk response is not returned within some time, it will
be re-requested, possibly from a different peer.
LightBlockRequest
To verify state and to provide state relevant information for consensus, the node will ask peers for light blocks at specified heights.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | uint64 | Height of the light block | 1 |
LightBlockResponse
The receiver will retrieve and construct the light block from both the block
and state stores. The receiver will verify the data by comparing the hashes
and store the header, commit and validator set if necessary. The light block
at the height of the snapshot will be used to verify the AppHash.
| Name | Type | Description | Field Number |
|---|---|---|---|
| light_block | LightBlock | Light block at the height requested | 1 |
Unlike upstream Tendermint, light blocks are not verified against a trusted
header obtained out of band. Tenderdash's light client verifies the quorum
threshold signature on the block commit via Dash Core (quorum verify), so no
trust anchors (trusted height, hash, or period) are required.
If no state sync is in progress (i.e. during normal operation), any unsolicited response messages are discarded.
ParamsRequest
In order to build Tenderdash state, the state provider will request the params at the height of the snapshot and use the header to verify it.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | uint64 | Height of the consensus params | 1 |
ParamsResponse
A receiver to the request will use the state store to fetch the consensus params at that height and return it to the sender.
| Name | Type | Description | Field Number |
|---|---|---|---|
| height | uint64 | Height of the consensus params | 1 |
| consensus_params | ConsensusParams | Consensus params at the height requested | 2 |
Peer Discovery
A Tenderdash P2P network has different kinds of nodes with different requirements for connectivity to one another. This document describes what kind of nodes Tenderdash should enable and how they should work.
Seeds
Seeds are the first point of contact for a new node. They return a list of known active peers and then disconnect.
Seeds should operate full nodes with the PEX reactor in a "crawler" mode that continuously explores to validate the availability of peers.
Seeds should only respond with some top percentile of the best peers it knows about.
New Full Node
A new node needs a few things to connect to the network:
- a list of seeds, which can be provided to Tenderdash via config file or flags, or hardcoded into the software by in-process apps
- a
ChainID, also calledNetworkat the p2p layer - a recent block height, H, and hash, HASH for the blockchain.
The values H and HASH must be received and corroborated by means external to Tenderdash, and specific to the user - ie. via the user's trusted social consensus.
This requirement to validate H and HASH out-of-band and via social consensus
is the essential difference in security models between Proof-of-Work and Proof-of-Stake blockchains.
With the above, the node then queries some seeds for peers for its chain, dials those peers, and runs the Tenderdash protocols with those it successfully connects to.
When the peer catches up to height H, it ensures the block hash matches HASH. If not, Tenderdash will exit, and the user must try again - either they are connected to bad peers or their social consensus is invalid.
Restarted Full Node
A node checks its address book on startup and attempts to connect to peers from there. If it can't connect to any peers after some time, it falls back to the seeds to find more.
Restarted full nodes can run the blockchain or consensus reactor protocols to sync up
to the latest state of the blockchain from wherever they were last.
In a Proof-of-Stake context, if they are sufficiently far behind (greater than the length
of the unbonding period), they will need to validate a recent H and HASH out-of-band again
so they know they have synced the correct chain.
Validator Node
A validator node is a node that interfaces with a validator signing key. These nodes require the highest security, and should not accept incoming connections. They should maintain outgoing connections to a controlled set of "Sentry Nodes" that serve as their proxy shield to the rest of the network.
Validators that know and trust each other can accept incoming connections from one another and maintain direct private connectivity via VPN.
Sentry Node
Sentry nodes are guardians of a validator node and provide it access to the rest of the network. They should be well connected to other full nodes on the network. Sentry nodes may be dynamic, but should maintain persistent connections to some evolving random subset of each other. They should always expect to have direct incoming connections from the validator node and its backup(s). They do not report the validator node's address in the PEX and they may be more strict about the quality of peers they keep.
Sentry nodes belonging to validators that trust each other may wish to maintain persistent connections via VPN with one another, but only report each other sparingly in the PEX.
Peers
This document explains how Tenderdash Peers are identified and how they connect to one another.
For details on peer discovery, see the peer exchange (PEX) doc.
Peer Identity
Tenderdash peers are expected to maintain long-term persistent identities in the form of a public key.
Each peer has an ID defined as peer.ID == peer.PubKey.Address(), where Address uses the scheme defined in crypto package.
A single peer ID can have multiple IP addresses associated with it, but a node will only ever connect to one at a time.
When attempting to connect to a peer, we use the PeerURL: <ID>@<IP>:<PORT>.
We will attempt to connect to the peer at IP:PORT, and verify,
via authenticated encryption, that it is in possession of the private key
corresponding to <ID>. This prevents man-in-the-middle attacks on the peer layer.
Connections
All p2p connections use TCP. Upon establishing a successful TCP connection with a peer, two handshakes are performed: one for authenticated encryption, and one for Tenderdash versioning. Both handshakes have configurable timeouts (they should complete quickly).
Authenticated Encryption Handshake
Tenderdash implements the Station-to-Station protocol using X25519 keys for Diffie-Helman key-exchange and chacha20poly1305 for encryption.
Upstream Tendermint versions of this protocol (0.32 and below) suffered from malleability attacks whereas an active man in the middle attacker could compromise confidentiality as described in Prime, Order Please! Revisiting Small Subgroup and Invalid Curve Attacks on Protocols using Diffie-Hellman.
We have added dependency on the Merlin a keccak based transcript hashing protocol to ensure non-malleability.
It goes as follows:
- generate an ephemeral X25519 keypair
- send the ephemeral public key to the peer
- wait to receive the peer's ephemeral public key
- create a new Merlin Transcript with the string "TENDERMINT_SECRET_CONNECTION_TRANSCRIPT_HASH"
- Sort the ephemeral keys and add the high labeled "EPHEMERAL_UPPER_PUBLIC_KEY" and the low keys labeled "EPHEMERAL_LOWER_PUBLIC_KEY" to the Merlin transcript.
- compute the Diffie-Hellman shared secret using the peers ephemeral public key and our ephemeral private key
- add the DH secret to the transcript labeled DH_SECRET.
- generate two keys to use for encryption (sending and receiving) and a challenge for authentication as follows:
- create a hkdf-sha256 instance with the key being the diffie hellman shared secret, and info parameter as
TENDERMINT_SECRET_CONNECTION_KEY_AND_CHALLENGE_GEN - get 64 bytes of output from hkdf-sha256
- if we had the smaller ephemeral pubkey, use the first 32 bytes for the key for receiving, the second 32 bytes for sending; else the opposite.
- create a hkdf-sha256 instance with the key being the diffie hellman shared secret, and info parameter as
- use a separate nonce for receiving and sending. Both nonces start at 0, and should support the full 96 bit nonce range
- all communications from now on are encrypted in 1400 byte frames (plus encoding overhead), using the respective secret and nonce. Each nonce is incremented by one after each use.
- we now have an encrypted channel, but still need to authenticate
- extract a 32 bytes challenge from merlin transcript with the label "SECRET_CONNECTION_MAC"
- sign the common challenge obtained from the hkdf with our persistent private key
- send the amino encoded persistent pubkey and signature to the peer
- wait to receive the persistent public key and signature from the peer
- verify the signature on the challenge using the peer's persistent public key
If this is an outgoing connection (we dialed the peer) and we used a peer ID,
then finally verify that the peer's persistent public key corresponds to the peer ID we dialed,
ie. peer.PubKey.Address() == <ID>.
The connection has now been authenticated. All traffic is encrypted.
Note: only the dialer can authenticate the identity of the peer, but this is what we care about since when we join the network we wish to ensure we have reached the intended peer (and are not being MITMd).
Peer Filter
Before continuing, we check if the new peer has the same ID as ourselves or an existing peer. If so, we disconnect.
We also check the peer's address and public key against an optional whitelist which can be managed through the ABCI app - if the whitelist is enabled and the peer does not qualify, the connection is terminated.
Tenderdash Version Handshake
The Tenderdash Version Handshake allows the peers to exchange their NodeInfo:
type NodeInfo struct {
Version p2p.Version
ID p2p.ID
ListenAddr string
Network string
SoftwareVersion string
Channels []int8
Moniker string
Other NodeInfoOther
}
type Version struct {
P2P uint64
Block uint64
App uint64
}
type NodeInfoOther struct {
TxIndex string
RPCAddress string
}
The connection is disconnected if:
peer.NodeInfo.IDis not equalpeerConn.IDpeer.NodeInfo.Version.Blockdoes not match ourspeer.NodeInfo.Networkis not the same as ourspeer.Channelsdoes not intersect with our known Channels.peer.NodeInfo.ListenAddris malformed or is a DNS host that cannot be resolved
At this point, if we have not disconnected, the peer is valid.
It is added to the switch and hence all reactors via the AddPeer method.
Note that each reactor may handle multiple channels.
Connection Activity
Once a peer is added, incoming messages for a given reactor are handled through
that reactor's Receive method, and output messages are sent directly by the Reactors
on each peer. A typical reactor maintains per-peer go-routine(s) that handle this.
P2P protocol
RPC spec
This file defines the JSON-RPC spec of Tenderdash. This is meant to be implemented by all clients.
Support
| Tenderdash | |
|---|---|
| JSON-RPC 2.0 | ✅ |
| HTTP | ✅ |
| HTTPS | ✅ |
| WS | ✅ |
Timestamps
Timestamps in the RPC layer of Tenderdash follows RFC3339Nano. The RFC3339Nano format removes trailing zeros from the seconds field.
This means if a block has a timestamp like: 1985-04-12T23:20:50.5200000Z, the value returned in the RPC will be 1985-04-12T23:20:50.52Z.
Info Routes
Health
Node heartbeat
Parameters
None
Request
HTTP
curl http://127.0.0.1:26657/health
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"health\"}"
Response
{
"jsonrpc": "2.0",
"id": -1,
"result": {}
}
Status
Get Tenderdash status including node info, pubkey, latest block hash, app hash, block height and time.
Parameters
None
Request
HTTP
curl http://127.0.0.1:26657/status
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"status\"}"
Response
{
"jsonrpc": "2.0",
"id": -1,
"result": {
"node_info": {
"protocol_version": {
"p2p": "8",
"block": "11",
"app": "0"
},
"id": "b93270b358a72a2db30089f3856475bb1f918d6d",
"listen_addr": "tcp://0.0.0.0:26656",
"network": "cosmoshub-4",
"version": "v0.34.8",
"channels": "40202122233038606100",
"moniker": "aib-hub-node",
"other": {
"tx_index": "on",
"rpc_address": "tcp://0.0.0.0:26657"
}
},
"sync_info": {
"latest_block_hash": "50F03C0EAACA8BCA7F9C14189ACE9C05A9A1BBB5268DB63DC6A3C848D1ECFD27",
"latest_app_hash": "2316CFF7644219F4F15BEE456435F280E2B38955EEA6D4617CCB6D7ABF781C22",
"latest_block_height": "5622165",
"latest_block_time": "2021-03-25T14:00:43.356134226Z",
"earliest_block_hash": "1455A0C15AC49BB506992EC85A3CD4D32367E53A087689815E01A524231C3ADF",
"earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
"earliest_block_height": "5200791",
"earliest_block_time": "2019-12-11T16:11:34Z",
"catching_up": false
},
"validator_info": {
"address": "38FB765D0092470989360ECA1C89CD06C2C1583C",
"pub_key": {
"type": "tendermint/PubKeyEd25519",
"value": "Z+8kntVegi1sQiWLYwFSVLNWqdAUGEy7lskL78gxLZI="
},
"voting_power": "0"
}
}
}
NetInfo
Network information
Parameters
None
Request
HTTP
curl http://127.0.0.1:26657/net_info
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"net_info\"}"
Response
{
"id": 0,
"jsonrpc": "2.0",
"result": {
"listening": true,
"listeners": [
"Listener(@)"
],
"n_peers": "1",
"peers": [
{
"node_id": "5576458aef205977e18fd50b274e9b5d9014525a",
"url": "tcp://5576458aef205977e18fd50b274e9b5d9014525a@95.179.155.35:26656"
}
]
}
}
Blockchain
Get block headers. Returned in descending order. May be limited in quantity.
Parameters
minHeight (integer): The lowest block to be returned in the responsemaxHeight (integer): The highest block to be returned in the response
Request
HTTP
curl http://127.0.0.1:26657/blockchain
curl http://127.0.0.1:26657/blockchain?minHeight=1&maxHeight=2
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"blockchain\",\"params\":{\"minHeight\":\"1\", \"maxHeight\":\"2\"}}"
Response
{
"id": 0,
"jsonrpc": "2.0",
"result": {
"last_height": "1276718",
"block_metas": [
{
"block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"block_size": 1000000,
"header": {
"version": {
"block": "10",
"app": "0"
},
"chain_id": "cosmoshub-2",
"height": "12",
"time": "2019-04-22T17:01:51.701356223Z",
"last_block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
"data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
"validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
"app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
"last_results_hash": "",
"evidence_hash": "",
"proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
},
"num_txs": "54"
}
]
}
}
Header
Get a header at a specified height.
Parameters
height (integer): height of the requested header. If no height is specified the latest height will be used.
Request
HTTP
curl http://127.0.0.1:26657/header
curl http://127.0.0.1:26657/header?height=1
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"header\",\"params\":{\"height\":\"1\"}}"
Response
{
"id": 0,
"jsonrpc": "2.0",
"result": {
"header": {
"version": {
"block": "10",
"app": "0"
},
"chain_id": "cosmoshub-2",
"height": "12",
"time": "2019-04-22T17:01:51.701356223Z",
"last_block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
"data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
"validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
"app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
"last_results_hash": "",
"evidence_hash": "",
"proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
}
}
}
HeaderByHash
Parameters
hash (string): Hash of the header to query for.
Request
HTTP
curl http://127.0.0.1:26657/header_by_hash?hash=0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"header_by_hash\",\"params\":{\"hash\":\"0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED\"}}"
Response
{
"id": 0,
"jsonrpc": "2.0",
"result": {
"header": {
"version": {
"block": "10",
"app": "0"
},
"chain_id": "cosmoshub-2",
"height": "12",
"time": "2019-04-22T17:01:51.701356223Z",
"last_block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
"data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
"validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
"app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
"last_results_hash": "",
"evidence_hash": "",
"proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
}
}
}
}
Block
Get block at a specified height.
Parameters
height (integer): height of the requested block. If no height is specified the latest height will be used.
Request
HTTP
curl http://127.0.0.1:26657/block
curl http://127.0.0.1:26657/block?height=1
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"block\",\"params\":{\"height\":\"1\"}}"
Response
{
"id": 0,
"jsonrpc": "2.0",
"result": {
"block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"block": {
"header": {
"version": {
"block": "10",
"app": "0"
},
"chain_id": "cosmoshub-2",
"height": "12",
"time": "2019-04-22T17:01:51.701356223Z",
"last_block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
"data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
"validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
"app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
"last_results_hash": "",
"evidence_hash": "",
"proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
},
"data": [
"yQHwYl3uCkKoo2GaChRnd+THLQ2RM87nEZrE19910Z28ABIUWW/t8AtIMwcyU0sT32RcMDI9GF0aEAoFdWF0b20SBzEwMDAwMDASEwoNCgV1YXRvbRIEMzEwMRCd8gEaagom61rphyEDoJPxlcjRoNDtZ9xMdvs+lRzFaHe2dl2P5R2yVCWrsHISQKkqX5H1zXAIJuC57yw0Yb03Fwy75VRip0ZBtLiYsUqkOsPUoQZAhDNP+6LY+RUwz/nVzedkF0S29NZ32QXdGv0="
],
"evidence": [
{
"type": "string",
"height": 0,
"time": 0,
"total_voting_power": 0,
"validator": {
"pub_key": {
"type": "tendermint/PubKeyEd25519",
"value": "A6DoBUypNtUAyEHWtQ9bFjfNg8Bo9CrnkUGl6k6OHN4="
},
"voting_power": 0,
"address": "string"
}
}
],
"last_commit": {
"height": 0,
"round": 0,
"block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"signatures": [
{
"type": 2,
"height": "1262085",
"round": 0,
"block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"timestamp": "2019-08-01T11:39:38.867269833Z",
"validator_address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F",
"validator_index": 0,
"signature": "DBchvucTzAUEJnGYpNvMdqLhBAHG4Px8BsOBB3J3mAFCLGeuG7uJqy+nVngKzZdPhPi8RhmE/xcw/M9DOJjEDg=="
}
]
}
}
}
}
BlockByHash
Parameters
hash (string): Hash of the block to query for.
Request
HTTP
curl http://127.0.0.1:26657/block_by_hash?hash=0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"block_by_hash\",\"params\":{\"hash\":\"0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED\"}}"
Response
{
"id": 0,
"jsonrpc": "2.0",
"result": {
"block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"block": {
"header": {
"version": {
"block": "10",
"app": "0"
},
"chain_id": "cosmoshub-2",
"height": "12",
"time": "2019-04-22T17:01:51.701356223Z",
"last_block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
"data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
"validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
"app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
"last_results_hash": "",
"evidence_hash": "",
"proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
},
"data": [
"yQHwYl3uCkKoo2GaChRnd+THLQ2RM87nEZrE19910Z28ABIUWW/t8AtIMwcyU0sT32RcMDI9GF0aEAoFdWF0b20SBzEwMDAwMDASEwoNCgV1YXRvbRIEMzEwMRCd8gEaagom61rphyEDoJPxlcjRoNDtZ9xMdvs+lRzFaHe2dl2P5R2yVCWrsHISQKkqX5H1zXAIJuC57yw0Yb03Fwy75VRip0ZBtLiYsUqkOsPUoQZAhDNP+6LY+RUwz/nVzedkF0S29NZ32QXdGv0="
],
"evidence": [
{
"type": "string",
"height": 0,
"time": 0,
"total_voting_power": 0,
"validator": {
"pub_key": {
"type": "tendermint/PubKeyEd25519",
"value": "A6DoBUypNtUAyEHWtQ9bFjfNg8Bo9CrnkUGl6k6OHN4="
},
"voting_power": 0,
"address": "string"
}
}
],
"last_commit": {
"height": 0,
"round": 0,
"block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"signatures": [
{
"type": 2,
"height": "1262085",
"round": 0,
"block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"timestamp": "2019-08-01T11:39:38.867269833Z",
"validator_address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F",
"validator_index": 0,
"signature": "DBchvucTzAUEJnGYpNvMdqLhBAHG4Px8BsOBB3J3mAFCLGeuG7uJqy+nVngKzZdPhPi8RhmE/xcw/M9DOJjEDg=="
}
]
}
}
}
}
BlockResults
Parameters
height (integer): Height of the block which contains the results. If no height is specified, the latest block height will be used
Request
HTTP
curl http://127.0.0.1:26657/block_results
curl http://127.0.0.1:26657/block_results?height=1
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"block_results\",\"params\":{\"height\":\"1\"}}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"height": "12",
"total_gas_used": "100",
"txs_results": [
{
"code": "0",
"data": "",
"log": "not enough gas",
"info": "",
"gas_wanted": "100",
"gas_used": "100",
"events": [
{
"type": "app",
"attributes": [
{
"key": "YWN0aW9u",
"value": "c2VuZA==",
"index": false
}
]
}
],
"codespace": "ibc"
}
],
"begin_block_events": [
{
"type": "app",
"attributes": [
{
"key": "YWN0aW9u",
"value": "c2VuZA==",
"index": false
}
]
}
],
"end_block": [
{
"type": "app",
"attributes": [
{
"key": "YWN0aW9u",
"value": "c2VuZA==",
"index": false
}
]
}
],
"validator_updates": [
{
"pub_key": {
"type": "tendermint/PubKeyEd25519",
"value": "9tK9IT+FPdf2qm+5c2qaxi10sWP+3erWTKgftn2PaQM="
},
"power": "300"
}
],
"consensus_params_updates": {
"block": {
"max_bytes": "22020096",
"max_gas": "1000",
"time_iota_ms": "1000"
},
"evidence": {
"max_age": "100000"
},
"validator": {
"pub_key_types": [
"ed25519"
]
}
}
}
}
Commit
Parameters
height (integer): Height of the block the requested commit pertains to. If no height is set the latest commit will be returned.
Request
HTTP
curl http://127.0.0.1:26657/commit
curl http://127.0.0.1:26657/commit?height=1
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"commit\",\"params\":{\"height\":\"1\"}}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"signed_header": {
"header": {
"version": {
"block": "10",
"app": "0"
},
"chain_id": "cosmoshub-2",
"height": "12",
"time": "2019-04-22T17:01:51.701356223Z",
"last_block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812",
"data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73",
"validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0",
"consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8",
"app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C",
"last_results_hash": "",
"evidence_hash": "",
"proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E"
},
"commit": {
"height": "1311801",
"round": 0,
"block_id": {
"hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7",
"parts": {
"total": 1,
"hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD"
}
},
"signatures": [
{
"block_id_flag": 2,
"validator_address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F",
"timestamp": "2019-04-22T17:01:58.376629719Z",
"signature": "14jaTQXYRt8kbLKEhdHq7AXycrFImiLuZx50uOjs2+Zv+2i7RTG/jnObD07Jo2ubZ8xd7bNBJMqkgtkd0oQHAw=="
}
]
}
},
"canonical": true
}
}
Validators
Parameters
height (integer): Block height at which the validators were present on. If no height is set the latest commit will be returned.page (integer):per_page (integer):
Request
HTTP
curl http://127.0.0.1:26657/validators
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"validators\",\"params\":{\"height\":\"1\", \"page\":\"1\", \"per_page\":\"20\"}}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"block_height": "55",
"validators": [
{
"address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F",
"pub_key": {
"type": "tendermint/PubKeyEd25519",
"value": "9tK9IT+FPdf2qm+5c2qaxi10sWP+3erWTKgftn2PaQM="
},
"voting_power": "239727",
"proposer_priority": "-11896414"
}
],
"count": "1",
"total": "25"
}
}
Genesis
Get Genesis of the chain. If the response is large, this operation
will return an error: use genesis_chunked instead.
Request
HTTP
curl http://127.0.0.1:26657/genesis
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"genesis\"}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"genesis": {
"genesis_time": "2019-04-22T17:00:00Z",
"chain_id": "cosmoshub-2",
"initial_height": "2",
"consensus_params": {
"block": {
"max_bytes": "22020096",
"max_gas": "1000",
"time_iota_ms": "1000"
},
"evidence": {
"max_age": "100000"
},
"validator": {
"pub_key_types": [
"ed25519"
]
}
},
"validators": [
{
"address": "B00A6323737F321EB0B8D59C6FD497A14B60938A",
"pub_key": {
"type": "tendermint/PubKeyEd25519",
"value": "cOQZvh/h9ZioSeUMZB/1Vy1Xo5x2sjrVjlE/qHnYifM="
},
"power": "9328525",
"name": "Certus One"
}
],
"app_hash": "",
"app_state": {}
}
}
}
GenesisChunked
Get the genesis document in a chunks to support easily transfering larger documents.
Parameters
chunk(integer): the index number of the chunk that you wish to fetch. These IDs are 0 indexed.
Request
HTTP
curl http://127.0.0.1:26657/genesis_chunked?chunk=0
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"genesis_chunked\",\"params\":{\"chunk\":0}}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"chunk": 0,
"total": 10,
"data": "dGVuZGVybWludAo="
}
}
ConsensusParams
Get the consensus parameters.
Parameters
height (integer): Block height at which the consensus params would like to be fetched for.
Request
HTTP
curl http://127.0.0.1:26657/consensus_params
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"consensus_params\"}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"block_height": "1",
"consensus_params": {
"block": {
"max_bytes": "22020096",
"max_gas": "1000",
"time_iota_ms": "1000"
},
"evidence": {
"max_age": "100000"
},
"validator": {
"pub_key_types": [
"ed25519"
]
}
}
}
}
UnconfirmedTxs
Get a list of unconfirmed transactions.
Parameters
limit (integer)The amount of txs to respond with.
Request
HTTP
curl http://127.0.0.1:26657/unconfirmed_txs
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"unconfirmed_txs\, \"params\":{\"limit\":\"20\"}}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"n_txs": "82",
"total": "82",
"total_bytes": "19974",
"txs": [
"gAPwYl3uCjCMTXENChSMnIkb5ZpYHBKIZqecFEV2tuZr7xIUA75/FmYq9WymsOBJ0XSJ8yV8zmQKMIxNcQ0KFIyciRvlmlgcEohmp5wURXa25mvvEhQbrvwbvlNiT+Yjr86G+YQNx7kRVgowjE1xDQoUjJyJG+WaWBwSiGannBRFdrbma+8SFK2m+1oxgILuQLO55n8mWfnbIzyPCjCMTXENChSMnIkb5ZpYHBKIZqecFEV2tuZr7xIUQNGfkmhTNMis4j+dyMDIWXdIPiYKMIxNcQ0KFIyciRvlmlgcEohmp5wURXa25mvvEhS8sL0D0wwgGCItQwVowak5YB38KRIUCg4KBXVhdG9tEgUxMDA1NBDoxRgaagom61rphyECn8x7emhhKdRCB2io7aS/6Cpuq5NbVqbODmqOT3jWw6kSQKUresk+d+Gw0BhjiggTsu8+1voW+VlDCQ1GRYnMaFOHXhyFv7BCLhFWxLxHSAYT8a5XqoMayosZf9mANKdXArA="
]
}
}
NumUnconfirmedTxs
Get data about unconfirmed transactions.
Parameters
None
Request
HTTP
curl http://127.0.0.1:26657/num_unconfirmed_txs
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"num_unconfirmed_txs\"}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"n_txs": "31",
"total": "82",
"total_bytes": "19974"
}
}
Tx
Parameters
hash (string): The hash of the transactionprove (bool): If the response should include proof the transaction was included in a block.
Request
HTTP
curl http://127.0.0.1:26657/num_unconfirmed_txs
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"num_unconfirmed_txs\"}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"hash": "D70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED",
"height": "1000",
"index": 0,
"tx_result": {
"log": "[{\"msg_index\":\"0\",\"success\":true,\"log\":\"\"}]",
"gas_wanted": "200000",
"gas_used": "28596",
"tags": [
{
"key": "YWN0aW9u",
"value": "c2VuZA==",
"index": false
}
]
},
"tx": "5wHwYl3uCkaoo2GaChQmSIu8hxpJxLcCuIi8fiHN4TMwrRIU/Af1cEG7Rcs/6LjTl7YjRSymJfYaFAoFdWF0b20SCzE0OTk5OTk1MDAwEhMKDQoFdWF0b20SBDUwMDAQwJoMGmoKJuta6YchAwswBShaB1wkZBctLIhYqBC3JrAI28XGzxP+rVEticGEEkAc+khTkKL9CDE47aDvjEHvUNt+izJfT4KVF2v2JkC+bmlH9K08q3PqHeMI9Z5up+XMusnTqlP985KF+SI5J3ZOIhhNYWRlIGJ5IENpcmNsZSB3aXRoIGxvdmU="
}
}
Transaction Routes
BroadCastTxSync
Returns with the response from CheckTx. Does not wait for DeliverTx result.
Parameters
tx (string): The transaction encoded
Request
HTTP
curl http://127.0.0.1:26657/broadcast_tx_sync?tx=encoded_tx
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"broadcast_tx_sync\",\"params\":{\"tx\":\"a/encoded_tx/c\"}}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"code": "0",
"data": "",
"log": "",
"codespace": "ibc",
"hash": "0D33F2F03A5234F38706E43004489E061AC40A2E"
},
"error": ""
}
BroadCastTxAsync
Returns right away, with no response. Does not wait for CheckTx nor DeliverTx results.
Parameters
tx (string): The transaction encoded
Request
HTTP
curl http://127.0.0.1:26657/broadcast_tx_async?tx=encoded_tx
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"broadcast_tx_async\",\"params\":{\"tx\":\"a/encoded_tx/c\"}}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"code": "0",
"data": "",
"log": "",
"codespace": "ibc",
"hash": "0D33F2F03A5234F38706E43004489E061AC40A2E"
},
"error": ""
}
CheckTx
Checks the transaction without executing it.
Parameters
tx (string): String of the encoded transaction
Request
HTTP
curl http://127.0.0.1:26657/check_tx?tx=encoded_tx
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"check_tx\",\"params\":{\"tx\":\"a/encoded_tx/c\"}}"
Response
{
"id": 0,
"jsonrpc": "2.0",
"error": "",
"result": {
"code": "0",
"data": "",
"log": "",
"info": "",
"gas_wanted": "1",
"gas_used": "0",
"events": [
{
"type": "app",
"attributes": [
{
"key": "YWN0aW9u",
"value": "c2VuZA==",
"index": false
}
]
}
],
"codespace": "bank"
}
}
ABCI Routes
ABCIInfo
Get some info about the application.
Parameters
None
Request
HTTP
curl http://127.0.0.1:26657/abci_info
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"abci_info\"}"
Response
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"response": {
"data": "{\"size\":0}",
"version": "0.16.1",
"app_version": "1314126"
}
}
}
ABCIQuery
Query the application for some information.
Parameters
path (string): Path to the data. This is defined by the application.data (string): The data requestedheight (integer): Height at which the data is being requested for.prove (bool): Include proofs of the transactions inclusion in the block
Request
HTTP
curl http://127.0.0.1:26657/abci_query?path="a/b/c"=IHAVENOIDEA&height=1&prove=true
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"abci_query\",\"params\":{\"path\":\"a/b/c\", \"height\":\"1\", \"bool\":\"true\"}}"
Response
{
"error": "",
"result": {
"response": {
"log": "exists",
"height": "0",
"proof": "010114FED0DAD959F36091AD761C922ABA3CBF1D8349990101020103011406AA2262E2F448242DF2C2607C3CDC705313EE3B0001149D16177BC71E445476174622EA559715C293740C",
"value": "61626364",
"key": "61626364",
"index": "-1",
"code": "0"
}
},
"id": 0,
"jsonrpc": "2.0"
}
Evidence Routes
BroadcastEvidence
Broadcast evidence of the misbehavior.
Parameters
evidence (string):
Request
HTTP
curl http://localhost:26657/broadcast_evidence?evidence=JSON_EVIDENCE_encoded
JSONRPC
curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"broadcast_evidence\",\"params\":{\"evidence\":\"JSON_EVIDENCE_encoded\"}}"
Response
{
"error": "",
"result": "",
"id": 0,
"jsonrpc": "2.0"
}
Light Client Specification
Historical design material. These Tendermint light-client designs and formal models are retained as reference material.
This directory contains work-in-progress English and TLA+ specifications for the Light Client protocol. Implementations of the light client can be found in Rust and Go.
Light clients are assumed to be initialized once from a trusted source with a trusted header and validator set. The light client protocol allows a client to then securely update its trusted state by requesting and verifying a minimal set of data from a network of full nodes (at least one of which is correct).
The light client is decomposed into two main components:
- Commit Verification - verify signed headers and associated validator set changes from a single full node, called primary
- Attack Detection - verify commits across multiple full nodes (called secondaries) and detect conflicts (ie. the existence of a lightclient attack)
In case a lightclient attack is detected, the lightclient submits evidence to a full node which is responsible for "accountability", that is, punishing attackers:
- Accountability - given evidence for an attack, compute a set of validators that are responsible for it.
Commit Verification
The English specification describes the light client commit verification problem in terms of the temporal properties LCV-DIST-SAFE.1 and LCV-DIST-LIVE.1. Commit verification is assumed to operate within the Tendermint Failure Model, where +2/3 of validators are correct for some time period and validator sets can change arbitrarily at each height.
A light client protocol is also provided, including all checks that need to be performed on headers, commits, and validator sets to satisfy the temporal properties - so a light client can continuously synchronize with a blockchain. Clients can skip possibly many intermediate headers by exploiting overlap in trusted and untrusted validator sets. When there is not enough overlap, a bisection routine can be used to find a minimal set of headers that do provide the required overlap.
The TLA+ specification ver. 001 is a formal description of the commit verification protocol executed by a client, including the safety and termination, which can be model checked with Apalache.
A more detailed TLA+ specification of Light client verification ver. 003 is currently under peer review.
The MC*.tla files contain concrete parameters for the
TLA+ specification, in order to do model checking.
For instance, MC4_3_faulty.tla contains the following parameters
for the nodes, heights, the trusting period, the clock drifts,
correctness of the primary node, and the ratio of the faulty processes:
AllNodes == {"n1", "n2", "n3", "n4"}
TRUSTED_HEIGHT == 1
TARGET_HEIGHT == 3
TRUSTING_PERIOD == 1400 \* the trusting period in some time units
CLOCK_DRIFT = 10 \* how much we assume the local clock is drifting
REAL_CLOCK_DRIFT = 3 \* how much the local clock is actually drifting
IS_PRIMARY_CORRECT == FALSE
FAULTY_RATIO == <<1, 3>> \* < 1 / 3 faulty validators
To run a complete set of experiments, clone apalache and apalache-tests into a directory $DIR and run the following commands:
$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 002bmc-apalache-ok.csv $DIR/apalache . out
./out/run-all.sh
After the experiments have finished, you can collect the logs by executing the following command:
cd ./out
$DIR/apalache-tests/scripts/parse-logs.py --human .
All lines in results.csv should report Deadlock, which means that the algorithm
has terminated and no invariant violation was found.
Similar to 002bmc-apalache-ok.csv, file 003bmc-apalache-error.csv specifies the set of experiments that should result in counterexamples:
$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 003bmc-apalache-error.csv $DIR/apalache . out
./out/run-all.sh
All lines in results.csv should report Error.
The following table summarizes the experimental results for Light client verification version 001. The TLA+ properties can be found in the TLA+ specification. The experiments were run in an AWS instance equipped with 32GB RAM and a 4-core Intel® Xeon® CPU E5-2686 v4 @ 2.30GHz CPU. We write “✗=k” when a bug is reported at depth k, and “✓<=k” when no bug is reported up to depth k.

The experimental results for version 003 are to be added.
Attack Detection
The English specification defines light client attacks (and how they differ from blockchain forks), and describes the problem of a light client detecting these attacks by communicating with a network of full nodes, where at least one is correct.
The specification also contains a detection protocol that checks whether the header obtained from the primary via the verification protocol matches corresponding headers provided by the secondaries. If this is not the case, the protocol analyses the verification traces of the involved full nodes and generates evidence of misbehavior that can be submitted to a full node so that the faulty validators can be punished.
The TLA+ specification is a formal description of the detection protocol for two peers, including the safety and termination, which can be model checked with Apalache.
The LCD_MC*.tla files contain concrete parameters for the
TLA+ specification,
in order to run the model checker.
For instance, LCD_MC4_4_faulty.tla
contains the following parameters
for the nodes, heights, the trusting period, the clock drifts,
correctness of the nodes, and the ratio of the faulty processes:
AllNodes == {"n1", "n2", "n3", "n4"}
TRUSTED_HEIGHT == 1
TARGET_HEIGHT == 3
TRUSTING_PERIOD == 1400 \* the trusting period in some time units
CLOCK_DRIFT = 10 \* how much we assume the local clock is drifting
REAL_CLOCK_DRIFT = 3 \* how much the local clock is actually drifting
IS_PRIMARY_CORRECT == FALSE
IS_SECONDARY_CORRECT == FALSE
FAULTY_RATIO == <<1, 3>> \* < 1 / 3 faulty validators
To run a complete set of experiments, clone apalache and apalache-tests into a directory $DIR and run the following commands:
$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 004bmc-apalache-ok.csv $DIR/apalache . out
./out/run-all.sh
After the experiments have finished, you can collect the logs by executing the following command:
cd ./out
$DIR/apalache-tests/scripts/parse-logs.py --human .
All lines in results.csv should report Deadlock, which means that the algorithm
has terminated and no invariant violation was found.
Similar to 004bmc-apalache-ok.csv, file 005bmc-apalache-error.csv specifies the set of experiments that should result in counterexamples:
$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 005bmc-apalache-error.csv $DIR/apalache . out
./out/run-all.sh
All lines in results.csv should report Error.
The detailed experimental results are to be added soon.
Accountability
The English specification defines the protocol that is executed on a full node upon receiving attack evidence from a lightclient. In particular, the protocol handles three types of attacks
- lunatic
- equivocation
- amnesia
We discussed in the last part of the English specification that the non-lunatic cases are defined by having the same validator set in the conflicting blocks. For these cases, computer-aided analysis of Tendermint Consensus in TLA+ shows that equivocation and amnesia capture all non-lunatic attacks.
The TLA+ specification is a formal description of the protocol, including the safety property, which can be model checked with Apalache.
Similar to the other specifications, MC_5_3.tla contains concrete parameters to run the model checker. The specification can be checked within seconds.
Fork accountability
Problem Statement
Tendermint consensus guarantees the following specifications for all heights:
- agreement -- no two correct full nodes decide differently.
- validity -- the decided block satisfies the predefined predicate valid().
- termination -- all correct full nodes eventually decide,
If the faulty validators have less than 1/3 of voting power in the current validator set. In the case where this assumption does not hold, each of the specification may be violated.
The agreement property says that for a given height, any two correct validators that decide on a block for that height decide on the same block. That the block was indeed generated by the blockchain, can be verified starting from a trusted (genesis) block, and checking that all subsequent blocks are properly signed.
However, faulty nodes may forge blocks and try to convince users (light clients) that the blocks had been correctly generated. In addition, Tendermint agreement might be violated in the case where 1/3 or more of the voting power belongs to faulty validators: Two correct validators decide on different blocks. The latter case motivates the term "fork": as Tendermint consensus also agrees on the next validator set, correct validators may have decided on disjoint next validator sets, and the chain branches into two or more partitions (possibly having faulty validators in common) and each branch continues to generate blocks independently of the other.
We say that a fork is a case in which there are two commits for different blocks at the same height of the blockchain. The problem is to ensure that in those cases we are able to detect faulty validators (and not mistakenly accuse correct validators), and incentivize therefore validators to behave according to the protocol specification.
Conceptual Limit. In order to prove misbehavior of a node, we have to show that the behavior deviates from correct behavior with respect to a given algorithm. Thus, an algorithm that detects misbehavior of nodes executing some algorithm A must be defined with respect to algorithm A. In our case, A is Tendermint consensus (+ other protocols in the infrastructure; e.g.,full nodes and the Light Client). If the consensus algorithm is changed/updated/optimized in the future, we have to check whether changes to the accountability algorithm are also required. All the discussions in this document are thus inherently specific to Tendermint consensus and the Light Client specification.
Q: Should we distinguish agreement for validators and full nodes for agreement? The case where all correct validators agree on a block, but a correct full node decides on a different block seems to be slightly less severe that the case where two correct validators decide on different blocks. Still, if a contaminated full node becomes validator that may be problematic later on. Also it is not clear how gossiping is impaired if a contaminated full node is on a different branch.
Remark. In the case 1/3 or more of the voting power belongs to faulty validators, also validity and termination can be broken. Termination can be broken if faulty processes just do not send the messages that are needed to make progress. Due to asynchrony, this is not punishable, because faulty validators can always claim they never received the messages that would have forced them to send messages.
The Misbehavior of Faulty Validators
Forks are the result of faulty validators deviating from the protocol. In principle several such deviations can be detected without a fork actually occurring:
-
double proposal: A faulty proposer proposes two different values (blocks) for the same height and the same round in Tendermint consensus.
-
double signing: Tendermint consensus forces correct validators to prevote and precommit for at most one value per round. In case a faulty validator sends multiple prevote and/or precommit messages for different values for the same height/round, this is a misbehavior.
-
lunatic validator: Tendermint consensus forces correct validators to prevote and precommit only for values v that satisfy valid(v). If faulty validators prevote and precommit for v although valid(v)=false this is misbehavior.
Remark. In isolation, Point 3 is an attack on validity (rather than agreement). However, the prevotes and precommits can then also be used to forge blocks.
-
amnesia: Tendermint consensus has a locking mechanism. If a validator has some value v locked, then it can only prevote/precommit for v or nil. Sending prevote/precomit message for a different value v' (that is not nil) while holding lock on value v is misbehavior.
-
spurious messages: In Tendermint consensus most of the message send instructions are guarded by threshold guards, e.g., one needs to receive 2f + 1 prevote messages to send precommit. Faulty validators may send precommit without having received the prevote messages.
Independently of a fork happening, punishing this behavior might be important to prevent forks altogether. This should keep attackers from misbehaving: if less than 1/3 of the voting power is faulty, this misbehavior is detectable but will not lead to a safety violation. Thus, unless they have 1/3 or more (or in some cases more than 2/3) of the voting power attackers have the incentive to not misbehave. If attackers control too much voting power, we have to deal with forks, as discussed in this document.
Two types of forks
- Fork-Full. Two correct validators decide on different blocks for the same height. Since also the next validator sets are decided upon, the correct validators may be partitioned to participate in two distinct branches of the forked chain.
As in this case we have two different blocks (both having the same right/no right to exist), a central system invariant (one block per height decided by correct validators) is violated. As full nodes are contaminated in this case, the contamination can spread also to light clients. However, even without breaking this system invariant, light clients can be subject to a fork:
- Fork-Light. All correct validators decide on the same block for height h, but faulty processes (validators or not), forge a different block for that height, in order to fool users (who use the light client).
Attack scenarios
On-chain attacks
Equivocation (one round)
There are several scenarios in which forks might happen. The first is double signing within a round.
- F1. Equivocation: faulty validators sign multiple vote messages (prevote and/or precommit) for different values during the same round r at a given height h.
Flip-flopping
Tendermint consensus implements a locking mechanism: If a correct validator p receives proposal for value v and 2f + 1 prevotes for a value id(v) in round r, it locks v and remembers r. In this case, p also sends a precommit message for id(v), which later may serve as proof that p locked v. In subsequent rounds, p only sends prevote messages for a value it had previously locked. However, it is possible to change the locked value if in a future round r' > r, if the process receives proposal and 2f + 1 prevotes for a different value v'. In this case, p could send a prevote/precommit for id(v'). This algorithmic feature can be exploited in two ways:
-
F2. Faulty Flip-flopping (Amnesia): faulty validators precommit some value id(v) in round r (value v is locked in round r) and then prevote for different value id(v') in higher round r' > r without previously correctly unlocking value v. In this case faulty processes "forget" that they have locked value v and prevote some other value in the following rounds. Some correct validators might have decided on v in r, and other correct validators decide on v' in r'. Here we can have branching on the main chain (Fork-Full).
-
F3. Correct Flip-flopping (Back to the past): There are some precommit messages signed by (correct) validators for value id(v) in round r. Still, v is not decided upon, and all processes move on to the next round. Then correct validators (correctly) lock and decide a different value v' in some round r' > r. And the correct validators continue; there is no branching on the main chain. However, faulty validators may use the correct precommit messages from round r together with a posteriori generated faulty precommit messages for round r to forge a block for a value that was not decided on the main chain (Fork-Light).
Off-chain attacks
F1-F3 may contaminate the state of full nodes (and even validators). Contaminated (but otherwise correct) full nodes may thus communicate faulty blocks to light clients. Similarly, without actually interfering with the main chain, we can have the following:
-
F4. Phantom validators: faulty validators vote (sign prevote and precommit messages) in heights in which they are not part of the validator sets (at the main chain).
-
F5. Lunatic validator: faulty validator that sign vote messages to support (arbitrary) application state that is different from the application state that resulted from valid state transitions.
Types of victims
We consider three types of potential attack victims:
- FN: full node
- LCS: light client with sequential header verification
- LCB: light client with bisection based header verification
F1 and F2 can be used by faulty validators to actually create multiple branches on the blockchain. That means that correctly operating full nodes decide on different blocks for the same height. Until a fork is detected locally by a full node (by receiving evidence from others or by some other local check that fails), the full node can spread corrupted blocks to light clients.
Remark. If full nodes take a branch different from the one taken by the validators, it may be that the liveness of the gossip protocol may be affected. We should eventually look at this more closely. However, as it does not influence safety it is not a primary concern.
F3 is similar to F1, except that no two correct validators decide on different blocks. It may still be the case that full nodes become affected.
In addition, without creating a fork on the main chain, light clients can be contaminated by more than a third of validators that are faulty and sign a forged header F4 cannot fool correct full nodes as they know the current validator set. Similarly, LCS know who the validators are. Hence, F4 is an attack against LCB that do not necessarily know the complete prefix of headers (Fork-Light), as they trust a header that is signed by at least one correct validator (trusting period method).
The following table gives an overview of how the different attacks may affect different nodes. F1-F3 are on-chain attacks so they can corrupt the state of full nodes. Then if a light client (LCS or LCB) contacts a full node to obtain headers (or blocks), the corrupted state may propagate to the light client.
F4 and F5 are off-chain, that is, these attacks cannot be used to corrupt the state of full nodes (which have sufficient knowledge on the state of the chain to not be fooled).
| Attack | FN | LCS | LCB |
|---|---|---|---|
| F1 | direct | FN | FN |
| F2 | direct | FN | FN |
| F3 | direct | FN | FN |
| F4 | direct | ||
| F5 | direct |
Q: Light clients are more vulnerable than full nodes, because the former do only verify headers but do not execute transactions. What kind of certainty is gained by a full node that executes a transaction?
As a full node verifies all transactions, it can only be contaminated by an attack if the blockchain itself violates its invariant (one block per height), that is, in case of a fork that leads to branching.
Detailed Attack Scenarios
Equivocation based attacks
In case of equivocation based attacks, faulty validators sign multiple votes (prevote and/or precommit) in the same round of some height. This attack can be executed on both full nodes and light clients. It requires 1/3 or more of voting power to be executed.
Scenario 1: Equivocation on the main chain
Validators:
- CA - a set of correct validators with less than 1/3 of the voting power
- CB - a set of correct validators with less than 1/3 of the voting power
- CA and CB are disjoint
- F - a set of faulty validators with 1/3 or more voting power
Observe that this setting violates the Tendermint failure model.
Execution:
- A faulty proposer proposes block A to CA
- A faulty proposer proposes block B to CB
- Validators from the set CA and CB prevote for A and B, respectively.
- Faulty validators from the set F prevote both for A and B.
- The faulty prevote messages
- for A arrive at CA long before the B messages
- for B arrive at CB long before the A messages
- Therefore correct validators from set CA and CB will observe more than 2/3 of prevotes for A and B and precommit for A and B, respectively.
- Faulty validators from the set F precommit both values A and B.
- Thus, we have more than 2/3 commits for both A and B.
Consequences:
-
Creating evidence of misbehavior is simple in this case as we have multiple messages signed by the same faulty processes for different values in the same round.
-
We have to ensure that these different messages reach a correct process (full node, monitor?), which can submit evidence.
-
This is an attack on the full node level (Fork-Full).
-
It extends also to the light clients,
-
For both we need a detection and recovery mechanism.
Scenario 2: Equivocation to a light client (LCS)
Validators:
- a set F of faulty validators with more than 2/3 of the voting power.
Execution:
- for the main chain F behaves nicely
- F coordinates to sign a block B that is different from the one on the main chain.
- the light clients obtains B and trusts at as it is signed by more than 2/3 of the voting power.
Consequences:
Once equivocation is used to attack light client it opens space for different kind of attacks as application state can be diverged in any direction. For example, it can modify validator set such that it contains only validators that do not have any stake bonded. Note that after a light client is fooled by a fork, that means that an attacker can change application state and validator set arbitrarily.
In order to detect such (equivocation-based attack), the light client would need to cross check its state with some correct validator (or to obtain a hash of the state from the main chain using out of band channels).
Remark. The light client would be able to create evidence of misbehavior, but this would require to pull potentially a lot of data from correct full nodes. Maybe we need to figure out different architecture where a light client that is attacked will push all its data for the current unbonding period to a correct node that will inspect this data and submit corresponding evidence. There are also architectures that assumes a special role (sometimes called fisherman) whose goal is to collect as much as possible useful data from the network, to do analysis and create evidence transactions. That functionality is outside the scope of this document.
Remark. The difference between LCS and LCB might only be in the amount of voting power needed to convince light client about arbitrary state. In case of LCB where security threshold is at minimum, an attacker can arbitrarily modify application state with 1/3 or more of voting power, while in case of LCS it requires more than 2/3 of the voting power.
Flip-flopping: Amnesia based attacks
In case of amnesia, faulty validators lock some value v in some round r, and then vote for different value v' in higher rounds without correctly unlocking value v. This attack can be used both on full nodes and light clients.
Scenario 3: At most 2/3 of faults
Validators:
- a set F of faulty validators with 1/3 or more but at most 2/3 of the voting power
- a set C of correct validators
Execution:
- Faulty validators commit (without exposing it on the main chain) a block A in round r by collecting more than 2/3 of the voting power (containing correct and faulty validators).
- All validators (correct and faulty) reach a round r' > r.
- Some correct validators in C do not lock any value before round r'.
- The faulty validators in F deviate from Tendermint consensus by ignoring that they locked A in r, and propose a different block B in r'.
- As the validators in C that have not locked any value find B acceptable, they accept the proposal for B and commit a block B.
Remark. In this case, the more than 1/3 of faulty validators do not need to commit an equivocation (F1) as they only vote once per round in the execution.
Detecting faulty validators in the case of such an attack can be done by the fork accountability mechanism described in the fork accountability report (a Google Doc that is no longer available).
If a light client is attacked using this attack with 1/3 or more of voting power (and less than 2/3), the attacker cannot change the application state arbitrarily. Rather, the attacker is limited to a state a correct validator finds acceptable: In the execution above, correct validators still find the value acceptable, however, the block the light client trusts deviates from the one on the main chain.
Scenario 4: More than 2/3 of faults
In case there is an attack with more than 2/3 of the voting power, an attacker can arbitrarily change application state.
Validators:
- a set F1 of faulty validators with 1/3 or more of the voting power
- a set F2 of faulty validators with less than 1/3 of the voting power
Execution
- Similar to Scenario 3 (however, messages by correct validators are not needed)
- The faulty validators in F1 lock value A in round r
- They sign a different value in follow-up rounds
- F2 does not lock A in round r
Consequences:
- The validators in F1 will be detectable by the the fork accountability mechanisms.
- The validators in F2 cannot be detected using this mechanism. Only in case they signed something which conflicts with the application this can be used against them. Otherwise they do not do anything incorrect.
- This case is not covered by the fork accountability report as it only assumes at most 2/3 of faulty validators.
Q: do we need to define a special kind of attack for the case where a validator sign arbitrarily state? It seems that detecting such attack requires a different mechanism that would require as an evidence a sequence of blocks that led to that state. This might be very tricky to implement.
Back to the past
In this kind of attack, faulty validators take advantage of the fact that they did not sign messages in some of the past rounds. Due to the asynchronous network in which Tendermint operates, we cannot easily differentiate between such an attack and delayed message. This kind of attack can be used at both full nodes and light clients.
Scenario 5
Validators:
- C1 - a set of correct validators with over 1/3 of the voting power
- C2 - a set of correct validators with 1/3 of the voting power
- C1 and C2 are disjoint
- F - a set of faulty validators with less than 1/3 voting power
- one additional faulty process q
- F and q violate the Tendermint failure model.
Execution:
- in a round r of height h we have C1 precommitting a value A,
- C2 precommits nil,
- F does not send any message
- q precommits nil.
- In some round r' > r, F and q and C2 commit some other value B different from A.
- F and fp "go back to the past" and sign precommit message for value A in round r.
- Together with precomit messages of C1 this is sufficient for a commit for value A.
Consequences:
- Only a single faulty validator that previously precommited nil did equivocation, while the other 1/3 of faulty validators actually executed an attack that has exactly the same sequence of messages as part of amnesia attack. Detecting this kind of attack boil down to mechanisms for equivocation and amnesia.
Q: should we keep this as a separate kind of attack? It seems that equivocation, amnesia and phantom validators are the only kind of attack we need to support and this gives us security also in other cases. This would not be surprising as equivocation and amnesia are attacks that followed from the protocol and phantom attack is not really an attack to Tendermint but more to the Proof of Stake module.
Phantom validators
In case of phantom validators, processes that are not part of the current validator set but are still bonded (as attack happen during their unbonding period) can be part of the attack by signing vote messages. This attack can be executed against both full nodes and light clients.
Scenario 6
Validators:
- F -- a set of faulty validators that are not part of the validator set on the main chain at height h + k
Execution:
-
There is a fork, and there exist two different headers for height h + k, with different validator sets:
- VS2 on the main chain
- forged header VS2', signed by F (and others)
-
a light client has a trust in a header for height h (and the corresponding validator set VS1).
-
As part of bisection header verification, it verifies the header at height h + k with new validator set VS2'.
Consequences:
- To detect this, a node needs to see both, the forged header and the canonical header from the chain.
- If this is the case, detecting these kind of attacks is easy as it just requires verifying if processes are signing messages in heights in which they are not part of the validator set.
Remark. We can have phantom-validator-based attacks as a follow up of equivocation or amnesia based attack where forked state contains validators that are not part of the validator set at the main chain. In this case, they keep signing messages contributed to a forked chain (the wrong branch) although they are not part of the validator set on the main chain. This attack can also be used to attack full node during a period of time it is eclipsed.
Remark. Phantom validator evidence has been removed from implementation as it was deemed, although possibly a plausible form of evidence, not relevant. Any attack on the light client involving a phantom validator will have needed to be initiated by 1/3+ lunatic validators that can forge a new validator set that includes the phantom validator. Only in that case will the light client accept the phantom validators vote. We need only worry about punishing the 1/3+ lunatic cabal, that is the root cause of the attack.
Lunatic validator
Lunatic validator agrees to sign commit messages for arbitrary application state. It is used to attack light clients. Note that detecting this behavior require application knowledge. Detecting this behavior can probably be done by referring to the block before the one in which height happen.
Q: can we say that in this case a validator declines to check if a proposed value is valid before voting for it?
Synopsis
A TLA+ specification of a simplified Tendermint consensus, tuned for fork accountability. The simplifications are as follows:
-
the procotol runs for one height, that is, one-shot consensus
-
this specification focuses on safety, so timeouts are modelled with with non-determinism
-
the proposer function is non-determinstic, no fairness is assumed
-
the messages by the faulty processes are injected right in the initial states
-
every process has the voting power of 1
-
hashes are modelled as identity
Having the above assumptions in mind, the specification follows the pseudo-code of the Tendermint paper: https://arxiv.org/abs/1807.04938
Byzantine processes can demonstrate arbitrary behavior, including
no communication. However, we have to show that under the collective evidence
collected by the correct processes, at least f+1 Byzantine processes demonstrate
one of the following behaviors:
-
Equivocation: a Byzantine process sends two different values in the same round.
-
Amnesia: a Byzantine process locks a value, although it has locked another value in the past.
TLA+ modules
-
TendermintAcc_004_draft is the protocol specification,
-
TendermintAccInv_004_draft contains an inductive invariant for establishing the protocol safety as well as the forking cases,
-
MC_n<n>_f<f>, e.g., MC_n4_f1, contains fixed constants for model checking with the Apalache model checker, -
TendermintAccTrace_004_draft shows how to restrict the execution space to a fixed sequence of actions (e.g., to instantiate a counterexample),
-
TendermintAccDebug_004_draft contains the useful definitions for debugging the protocol specification with TLC and Apalache.
Reasoning about fork scenarios
The theorem statements can be found in TendermintAccInv_004_draft.tla.
First, we would like to show that TypedInv is an inductive invariant.
Formally, the statement looks as follows:
THEOREM TypedInvIsInductive ==
\/ FaultyQuorum
\//\ Init => TypedInv
/\ TypedInv /\ [Next]_vars => TypedInv'
When over two-thirds of processes are faulty, TypedInv is not inductive.
However, there is no hope to repair the protocol in this case. We run
Apalache to prove this theorem
only for fixed instances of 4 to 5 validators. Apalache does not parse theorem
statements at the moment, so we ran Apalache using a shell script. To find a
parameterized argument, one has to use a theorem prover, e.g., TLAPS.
Second, we would like to show that the invariant implies Agreement, that is,
no fork, provided that less than one third of processes is faulty. By combining
this theorem with the previous theorem, we conclude that the protocol indeed
satisfies Agreement under the condition LessThanThirdFaulty.
THEOREM AgreementWhenLessThanThirdFaulty ==
LessThanThirdFaulty /\ TypedInv => Agreement
Third, in the general case, we either have no fork, or two fork scenarios:
THEOREM AgreementOrFork ==
~FaultyQuorum /\ TypedInv => Accountability
Model checking results
Check the report on model checking with Apalache.
To run the model checking experiments, use the script:
./run.sh
This script assumes that the apalache build is available in
~/devl/apalache-unstable.
Results of 001indinv-apalache
1. Awesome plots
1.1. Time (logarithmic scale)
1.2. Time (linear)
1.3. Memory (logarithmic scale)
1.4. Memory (linear)
1.5. Number of arena cells (linear)
1.6. Number of SMT clauses (linear)
2. Input parameters
| no | filename | tool | timeout | init | inv | next | args |
|---|---|---|---|---|---|---|---|
| 1 | MC_n4_f1.tla | apalache | 10h | TypedInv | TypedInv | --length=1 --cinit=ConstInit | |
| 2 | MC_n4_f2.tla | apalache | 10h | TypedInv | TypedInv | --length=1 --cinit=ConstInit | |
| 3 | MC_n5_f1.tla | apalache | 10h | TypedInv | TypedInv | --length=1 --cinit=ConstInit | |
| 4 | MC_n5_f2.tla | apalache | 10h | TypedInv | TypedInv | --length=1 --cinit=ConstInit | |
| 5 | MC_n4_f1.tla | apalache | 20h | Init | TypedInv | --length=0 --cinit=ConstInit | |
| 6 | MC_n4_f2.tla | apalache | 20h | Init | TypedInv | --length=0 --cinit=ConstInit | |
| 7 | MC_n5_f1.tla | apalache | 20h | Init | TypedInv | --length=0 --cinit=ConstInit | |
| 8 | MC_n5_f2.tla | apalache | 20h | Init | TypedInv | --length=0 --cinit=ConstInit | |
| 9 | MC_n4_f1.tla | apalache | 20h | TypedInv | Agreement | --length=0 --cinit=ConstInit | |
| 10 | MC_n4_f2.tla | apalache | 20h | TypedInv | Accountability | --length=0 --cinit=ConstInit | |
| 11 | MC_n5_f1.tla | apalache | 20h | TypedInv | Agreement | --length=0 --cinit=ConstInit | |
| 12 | MC_n5_f2.tla | apalache | 20h | TypedInv | Accountability | --length=0 --cinit=ConstInit |
3. Detailed results: 001indinv-apalache-unstable.csv
| 01:no | 02:tool | 03:status | 04:time_sec | 05:depth | 05:mem_kb | 10:ninit_trans | 11:ninit_trans | 12:ncells | 13:nclauses | 14:navg_clause_len |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | apalache | NoError | 11m | 1 | 3.0GB | 0 | 0 | 217K | 1.0M | 89 |
| 2 | apalache | NoError | 11m | 1 | 3.0GB | 0 | 0 | 207K | 1.0M | 88 |
| 3 | apalache | NoError | 16m | 1 | 4.0GB | 0 | 0 | 311K | 2.0M | 101 |
| 4 | apalache | NoError | 14m | 1 | 3.0GB | 0 | 0 | 290K | 1.0M | 103 |
| 5 | apalache | NoError | 9s | 0 | 563MB | 0 | 0 | 2.0K | 14K | 42 |
| 6 | apalache | NoError | 10s | 0 | 657MB | 0 | 0 | 2.0K | 28K | 43 |
| 7 | apalache | NoError | 8s | 0 | 635MB | 0 | 0 | 2.0K | 17K | 44 |
| 8 | apalache | NoError | 10s | 0 | 667MB | 0 | 0 | 3.0K | 32K | 45 |
| 9 | apalache | NoError | 5m05s | 0 | 2.0GB | 0 | 0 | 196K | 889K | 108 |
| 10 | apalache | NoError | 8m08s | 0 | 6.0GB | 0 | 0 | 2.0M | 3.0M | 34 |
| 11 | apalache | NoError | 9m09s | 0 | 3.0GB | 0 | 0 | 284K | 1.0M | 128 |
| 12 | apalache | NoError | 14m | 0 | 7.0GB | 0 | 0 | 4.0M | 5.0M | 38 |
Lightclient Attackers Isolation
Warning: This is the beginning of an unfinished draft. Don't continue reading!
Adversarial nodes may have the incentive to lie to a lightclient about the state of a Tendermint blockchain. An attempt to do so is called attack. Light client verification checks incoming data by checking a so-called "commit", which is a forwarded set of signed messages that is (supposedly) produced during executing Tendermint consensus. Thus, an attack boils down to creating and signing Tendermint consensus messages in deviation from the Tendermint consensus algorithm rules.
As Tendermint consensus and light client verification is safe under the assumption of more than 2/3 of correct voting power per block [TMBC-FM-2THIRDS], this implies that if there was an attack then [TMBC-FM-2THIRDS] was violated, that is, there is a block such that
- validators deviated from the protocol, and
- these validators represent more than 1/3 of the voting power in that block.
In the case of an attack, the lightclient attack detection mechanism computes data, so called evidence [LC-DATA-EVIDENCE.1], that can be used
- to proof that there has been attack [TMBC-LC-EVIDENCE-DATA.1] and
- as basis to find the actual nodes that deviated from the Tendermint protocol.
This specification considers how a full node in a Tendermint blockchain can isolate a set of attackers that launched the attack. The set should satisfy
- the set does not contain a correct validator
- the set contains validators that represent more than 1/3 of the voting power of a block that is still within the unbonding period
Outline
TODO when preparing a version for broader review.
Part I - Basics
For definitions of data structures used here, in particular LightBlocks [LCV-DATA-LIGHTBLOCK.1], cf. Light Client Verification.
Part II - Definition of the Problem
The specification of the detection mechanism describes
- what is a light client attack,
- conditions under which the detector will detect a light client attack,
- and the format of the output data, called evidence, in the case an attack is detected. The format is defined in [LC-DATA-EVIDENCE.1] and looks as follows
type LightClientAttackEvidence struct {
ConflictingBlock LightBlock
CommonHeight int64
}
The isolator is a function that gets as input evidence ev
and a prefix of the blockchain bc at least up to height ev.ConflictingBlock.Header.Height + 1. The output is a set of peerIDs of validators.
We assume that the full node is synchronized with the blockchain and has reached the height ev.ConflictingBlock.Header.Height + 1.
[FN-INV-Output.1]
When an output is generated it satisfies the following properties:
- If
bc[CommonHeight].bfttimeis within the unbonding period w.r.t. the time at the full node,ev.ConflictingBlock.Header != bc[ev.ConflictingBlock.Header.Height]- Validators in
ev.ConflictingBlock.Commitrepresent more than 1/3 of the voting power inbc[ev.CommonHeight].NextValidators
- Then: A set of validators in
bc[CommonHeight].NextValidatorsthat- represent more than 1/3 of the voting power in
bc[ev.commonHeight].NextValidators - signed Tendermint consensus messages for height
ev.ConflictingBlock.Header.Heightby violating the Tendermint consensus protocol.
- represent more than 1/3 of the voting power in
- Else: the empty set.
Part IV - Protocol
Here we discuss how to solve the problem of isolating misbehaving processes. We describe the function isolateMisbehavingProcesses as well as all the helping functions below. In Part V, we discuss why the solution is complete based on result from analysis with automated tools.
Isolation
Outline
Describe solution (in English), decomposition into functions, where communication to other components happens.
[LCAI-FUNC-MAIN.1]
func isolateMisbehavingProcesses(ev LightClientAttackEvidence, bc Blockchain) []ValidatorAddress {
reference := bc[ev.conflictingBlock.Header.Height].Header
ev_header := ev.conflictingBlock.Header
ref_commit := bc[ev.conflictingBlock.Header.Height + 1].Header.LastCommit // + 1 !!
ev_commit := ev.conflictingBlock.Commit
if violatesTMValidity(reference, ev_header) {
// lunatic light client attack
signatories := Signers(ev.ConflictingBlock.Commit)
bonded_vals := Addresses(bc[ev.CommonHeight].NextValidators)
return intersection(signatories,bonded_vals)
}
// If this point is reached the validator sets in reference and ev_header are identical
else if RoundOf(ref_commit) == RoundOf(ev_commit) {
// equivocation light client attack
return intersection(Signers(ref_commit), Signers(ev_commit))
}
else {
// amnesia light client attack
return IsolateAmnesiaAttacker(ev, bc)
}
}
- Implementation comment
- If the full node has only reached height
ev.conflictingBlock.Header.Heightthenbc[ev.conflictingBlock.Header.Height + 1].Header.LastCommitrefers to the locally stored commit for this height. (This commit must be present by the precondition onlength(bc).) - We check in the precondition that the unbonding period is not expired. However, since time moves on, before handing the validators over Cosmos SDK, the time needs to be checked again to satisfy the contract which requires that only bonded validators are reported. This passing of validators to the SDK is out of scope of this specification.
- If the full node has only reached height
- Expected precondition
length(bc) >= ev.conflictingBlock.Header.HeightValidAndVerifiedUnbonding(bc[ev.CommonHeight], ev.ConflictingBlock) == SUCCESSev.ConflictingBlock.Header != bc[ev.ConflictingBlock.Header.Height]- TODO: input light blocks pass basic validation
- Expected postcondition
- [FN-INV-Output.1] holds
- Error condition
- returns an error if precondition is violated.
Details of the Functions
[LCAI-FUNC-VVU.1]
func ValidAndVerifiedUnbonding(trusted LightBlock, untrusted LightBlock) Result
- Conditions are identical to [LCV-FUNC-VALID.2] except the precondition "trusted.Header.Time > now - trustingPeriod" is substituted with
trusted.Header.Time > now - UnbondingPeriod
[LCAI-FUNC-NONVALID.1]
func violatesTMValidity(ref Header, ev Header) boolean
- Implementation remarks
- checks whether the evidence header
evviolates the validity property of Tendermint Consensus, by checking agains a reference header
- checks whether the evidence header
- Expected precondition
ref.Height == ev.Height
- Expected postcondition
- returns evaluation of the following disjunction
[[LCAI-NONVALID-OUTPUT.1]] ==
ref.ValidatorsHash != ev.ValidatorsHashor
ref.NextValidatorsHash != ev.NextValidatorsHashor
ref.ConsensusHash != ev.ConsensusHashor
ref.AppHash != ev.AppHashor
ref.LastResultsHash != ev.LastResultsHash
- returns evaluation of the following disjunction
func IsolateAmnesiaAttacker(ev LightClientAttackEvidence, bc Blockchain) []ValidatorAddress
- Implementation remarks TODO: What should we do here? Refer to the accountability doc?
- Expected postcondition TODO: What should we do here? Refer to the accountability doc?
func RoundOf(commit Commit) []ValidatorAddress
- Expected precondition
commitis well-formed. In particular all votes are from the same roundr.
- Expected postcondition
- returns round
rthat is encoded in all the votes of the commit
- returns round
func Signers(commit Commit) []ValidatorAddress
- Expected postcondition
- returns all validator addresses in
commit
- returns all validator addresses in
func Addresses(vals Validator[]) ValidatorAddress[]
- Expected postcondition
- returns all validator addresses in
vals
- returns all validator addresses in
Part V - Completeness
As discussed in the beginning of this document, an attack boils down to creating and signing Tendermint consensus messages in deviation from the Tendermint consensus algorithm rules.
The main function isolateMisbehavingProcesses distinguishes three kinds of wrongly signing messages, namely,
- lunatic: signing invalid blocks
- equivocation: double-signing valid blocks in the same consensus round
- amnesia: signing conflicting blocks in different consensus rounds, without having seen a quorum of messages that would have allowed to do so.
The question is whether this captures all attacks.
First observe that the first checking in isolateMisbehavingProcesses is violatesTMValidity. It takes care of lunatic attacks. If this check passes, that is, if violatesTMValidity returns FALSE this means that [FN-NONVALID-OUTPUT] evaluates to false, which implies that ref.ValidatorsHash = ev.ValidatorsHash. Hence after violatesTMValidity, all the involved validators are the ones from the blockchain. It is thus sufficient to analyze one instance of Tendermint consensus with a fixed group membership (set of validators). Also it is sufficient to consider two different valid consensus values, that is, binary consensus.
TODO we have analyzed Tendermint consensus with TLA+ and have accompanied Galois in an independent study of the protocol based on Ivy proofs.
References
[supervisor] The specification of the light client supervisor.
[verification] The specification of the light client verification protocol
[detection] The specification of the light client attack detection mechanism.
Lightclient Attackers Isolation
Adversarial nodes may have the incentive to lie to a lightclient about the state of a Tendermint blockchain. An attempt to do so is called attack. Light client verification checks incoming data by checking a so-called "commit", which is a forwarded set of signed messages that is (supposedly) produced during executing Tendermint consensus. Thus, an attack boils down to creating and signing Tendermint consensus messages in deviation from the Tendermint consensus algorithm rules.
As Tendermint consensus and light client verification is safe under the assumption of more than 2/3 of correct voting power per block [TMBC-FM-2THIRDS], this implies that if there was an attack then [TMBC-FM-2THIRDS] was violated, that is, there is a block such that
- validators deviated from the protocol, and
- these validators represent more than 1/3 of the voting power in that block.
In the case of an attack, the lightclient attack detection mechanism computes data, so called evidence [LC-DATA-EVIDENCE.1], that can be used
- to proof that there has been attack [TMBC-LC-EVIDENCE-DATA.1] and
- as basis to find the actual nodes that deviated from the Tendermint protocol.
This specification considers how a full node in a Tendermint blockchain can isolate a set of attackers that launched the attack. The set should satisfy
- the set does not contain a correct validator
- the set contains validators that represent more than 1/3 of the voting power of a block that is still within the unbonding period
Outline
After providing the problem statement, we specify the isolator function and close with the discussion about its correctness which is based on computer-aided analysis of Tendermint Consensus.
Part I - Basics and Definition of the Problem
For definitions of data structures used here, in particular LightBlocks [LCV-DATA-LIGHTBLOCK.1], we refer to the specification of Light Client Verification.
The specification of the detection mechanism describes
- what is a light client attack,
- conditions under which the detector will detect a light client attack,
- and the format of the output data, called evidence, in the case an attack is detected. The format is defined in [LC-DATA-EVIDENCE.1] and looks as follows
type LightClientAttackEvidence struct {
ConflictingBlock LightBlock
CommonHeight int64
}
The isolator is a function that gets as input evidence ev
and a prefix of the blockchain bc at least up to height ev.ConflictingBlock.Header.Height + 1. The output is a set of peerIDs of validators.
We assume that the full node is synchronized with the blockchain and has reached the height ev.ConflictingBlock.Header.Height + 1.
[LCAI-INV-Output.1]
When an output is generated it satisfies the following properties:
- If
bc[CommonHeight].bfttimeis within the unbonding period w.r.t. the time at the full node,ev.ConflictingBlock.Header != bc[ev.ConflictingBlock.Header.Height]- Validators in
ev.ConflictingBlock.Commitrepresent more than 1/3 of the voting power inbc[ev.CommonHeight].NextValidators
- Then: The output is a set of validators in
bc[CommonHeight].NextValidatorsthat- represent more than 1/3 of the voting power in
bc[ev.commonHeight].NextValidators - signed Tendermint consensus messages for height
ev.ConflictingBlock.Header.Heightby violating the Tendermint consensus protocol.
- represent more than 1/3 of the voting power in
- Else: the empty set.
Part II - Protocol
Here we discuss how to solve the problem of isolating misbehaving processes. We describe the function isolateMisbehavingProcesses as well as all the helping functions below. In Part III, we discuss why the solution is complete based on result from analysis with automated tools.
Isolation
Outline
We first check whether the conflicting block can indeed be verified from the common height. We then first check whether it was a lunatic attack (violating validity). If this is not the case, we check for equivocation. If this also is not the case, we start the on-chain accountability protocol.
[LCAI-FUNC-MAIN.1]
func isolateMisbehavingProcesses(ev LightClientAttackEvidence, bc Blockchain) []ValidatorAddress {
reference := bc[ev.conflictingBlock.Header.Height].Header
ev_header := ev.conflictingBlock.Header
ref_commit := bc[ev.conflictingBlock.Header.Height + 1].Header.LastCommit // + 1 !!
ev_commit := ev.conflictingBlock.Commit
if violatesTMValidity(reference, ev_header) {
// lunatic light client attack
signatories := Signers(ev.ConflictingBlock.Commit)
bonded_vals := Addresses(bc[ev.CommonHeight].NextValidators)
return intersection(signatories,bonded_vals)
}
// If this point is reached the validator sets in reference and ev_header are identical
else if RoundOf(ref_commit) == RoundOf(ev_commit) {
// equivocation light client attack
return intersection(Signers(ref_commit), Signers(ev_commit))
}
else {
// amnesia light client attack
return IsolateAmnesiaAttacker(ev, bc)
}
}
- Implementation comment
- If the full node has only reached height
ev.conflictingBlock.Header.Heightthenbc[ev.conflictingBlock.Header.Height + 1].Header.LastCommitrefers to the locally stored commit for this height. (This commit must be present by the precondition onlength(bc).) - We check in the precondition that the unbonding period is not expired. However, since time moves on, before handing the validators over Cosmos SDK, the time needs to be checked again to satisfy the contract which requires that only bonded validators are reported. This passing of validators to the SDK is out of scope of this specification.
- If the full node has only reached height
- Expected precondition
length(bc) >= ev.conflictingBlock.Header.HeightValidAndVerifiedUnbonding(bc[ev.CommonHeight], ev.ConflictingBlock) == SUCCESSev.ConflictingBlock.Header != bc[ev.ConflictingBlock.Header.Height]ev.conflictingBlocksatisfies basic validation (in particular all signed messages in the Commit are from the same round)
- Expected postcondition
- [FN-INV-Output.1] holds
- Error condition
- returns an error if precondition is violated.
Details of the Functions
[LCAI-FUNC-VVU.1]
func ValidAndVerifiedUnbonding(trusted LightBlock, untrusted LightBlock) Result
- Conditions are identical to [LCV-FUNC-VALID.2] except the precondition "trusted.Header.Time > now - trustingPeriod" is substituted with
trusted.Header.Time > now - UnbondingPeriod
[LCAI-FUNC-NONVALID.1]
func violatesTMValidity(ref Header, ev Header) boolean
- Implementation remarks
- checks whether the evidence header
evviolates the validity property of Tendermint Consensus, by checking against a reference header
- checks whether the evidence header
- Expected precondition
ref.Height == ev.Height
- Expected postcondition
- returns evaluation of the following disjunction
[LCAI-NONVALID-OUTPUT.1] ==
ref.ValidatorsHash != ev.ValidatorsHashor
ref.NextValidatorsHash != ev.NextValidatorsHashor
ref.ConsensusHash != ev.ConsensusHashor
ref.AppHash != ev.AppHashor
ref.LastResultsHash != ev.LastResultsHash
- returns evaluation of the following disjunction
func IsolateAmnesiaAttacker(ev LightClientAttackEvidence, bc Blockchain) []ValidatorAddress
- Implementation remarks
- This triggers the query/response protocol.
- Expected postcondition
- returns attackers according to [LCAI-INV-Output.1].
func RoundOf(commit Commit) []ValidatorAddress
- Expected precondition
commitis well-formed. In particular all votes are from the same roundr.
- Expected postcondition
- returns round
rthat is encoded in all the votes of the commit
- returns round
- Error condition
- reports error if precondition is violated
func Signers(commit Commit) []ValidatorAddress
- Expected postcondition
- returns all validator addresses in
commit
- returns all validator addresses in
func Addresses(vals Validator[]) ValidatorAddress[]
- Expected postcondition
- returns all validator addresses in
vals
- returns all validator addresses in
Part III - Completeness
As discussed in the beginning of this document, an attack boils down to creating and signing Tendermint consensus messages in deviation from the Tendermint consensus algorithm rules.
The main function isolateMisbehavingProcesses distinguishes three kinds of wrongly signed messages, namely,
- lunatic: signing invalid blocks
- equivocation: double-signing valid blocks in the same consensus round
- amnesia: signing conflicting blocks in different consensus rounds, without having seen a quorum of messages that would have allowed to do so.
The question is whether this captures all attacks.
First observe that the first check in isolateMisbehavingProcesses is violatesTMValidity. It takes care of lunatic attacks. If this check passes, that is, if violatesTMValidity returns FALSE this means that [LCAI-NONVALID-OUTPUT.1] evaluates to false, which implies that ref.ValidatorsHash = ev.ValidatorsHash. Hence, after violatesTMValidity, all the involved validators are the ones from the blockchain. It is thus sufficient to analyze one instance of Tendermint consensus with a fixed group membership (set of validators). Also, as we have two different blocks for the same height, it is sufficient to consider two different valid consensus values, that is, binary consensus.
For this fixed group membership, we have analyzed the attacks using the TLA+ specification of Tendermint Consensus in TLA+. We checked that indeed the only possible scenarios that can lead to violation of agreement are equivocation and amnesia. An independent study by Galois of the protocol based on Ivy proofs led to the same conclusion.
References
[supervisor] The specification of the light client supervisor.
[verification] The specification of the light client verification protocol.
[detection] The specification of the light client attack detection mechanism.
[tendermint-accountability]: TLA+ specification to check the types of attacks
Light client attacks
We define a light client attack as detection of conflicting headers for a given height that can be verified starting from the trusted light block. A light client attack is defined in the context of interactions of light client with two peers. One of the peers (called primary) defines a trace of verified light blocks (primary trace) that are being checked against trace of the other peer (called witness) that we call witness trace.
A light client attack is defined by the primary and witness traces that have a common root (the same trusted light block for a common height) but forms conflicting branches (end of traces is for the same height but with different headers). Note that conflicting branches could be arbitrarily big as branches continue to diverge after a bifurcation point. We propose an approach that allows us to define a valid light client attack only with a common light block and a single conflicting light block. We rely on the fact that we assume that the primary is under suspicion (therefore not trusted) and that the witness plays support role to detect and process an attack (therefore trusted). Therefore, once a light client detects an attack, it needs to send to a witness only missing data (common height and conflicting light block) as it has its trace. Keeping light client attack data of constant size saves bandwidth and reduces an attack surface. As we will explain below, although in the context of light client core verification the roles of primary and witness are clearly defined, in case of the attack, we run the same attack detection procedure twice where the roles are swapped. The rationale is that the light client does not know what peer is correct (on a right main branch) so it tries to create and submit an attack evidence to both peers.
Light client attack evidence consists of a conflicting light block and a common height.
type LightClientAttackEvidence struct {
ConflictingBlock LightBlock
CommonHeight int64
}
Full node can validate a light client attack evidence by executing the following procedure:
func IsValid(lcaEvidence LightClientAttackEvidence, bc Blockchain) boolean {
commonBlock = GetLightBlock(bc, lcaEvidence.CommonHeight)
if commonBlock == nil return false
// Note that trustingPeriod in ValidAndVerified is set to UNBONDING_PERIOD
verdict = ValidAndVerified(commonBlock, lcaEvidence.ConflictingBlock)
conflictingHeight = lcaEvidence.ConflictingBlock.Header.Height
return verdict == OK and bc[conflictingHeight].Header != lcaEvidence.ConflictingBlock.Header
}
Light client attack creation
Given a trusted light block trusted, a light node executes the bisection algorithm to verify header
untrusted at some height h. If the bisection algorithm succeeds, then the header untrusted is verified.
Headers that are downloaded as part of the bisection algorithm are stored in a store and they are also in
the verified state. Therefore, after the bisection algorithm successfully terminates we have a trace of
the light blocks ([] LightBlock) we obtained from the primary that we call primary trace.
Primary trace
The following invariant holds for the primary trace:
- Given a
trustedlight block, target heighth, andprimary_trace([] LightBlock): primary_trace[0] == trusted and primary_trace[len(primary_trace)-1].Height == h and successive light blocks are passing light client verification logic.
Witness with a conflicting header
The verified header at height h is cross-checked with every witness as part of
detection.
If a witness returns the conflicting header at the height h the following procedure is executed to verify
if the conflicting header comes from the valid trace and if that's the case to create an attack evidence:
Helper functions
We assume the following helper functions:
// Returns trace of verified light blocks starting from rootHeight and ending with targetHeight.
Trace(lightStore LightStore, rootHeight int64, targetHeight int64) LightBlock[]
// Returns validator set for the given height
GetValidators(bc Blockchain, height int64) Validator[]
// Returns validator set for the given height
GetValidators(bc Blockchain, height int64) Validator[]
// Return validator addresses for the given validators
GetAddresses(vals Validator[]) ValidatorAddress[]
func DetectLightClientAttacks(primary PeerID,
primary_trace []LightBlock,
witness PeerID) (LightClientAttackEvidence, LightClientAttackEvidence) {
primary_lca_evidence, witness_trace = DetectLightClientAttack(primary_trace, witness)
witness_lca_evidence = nil
if witness_trace != nil {
witness_lca_evidence, _ = DetectLightClientAttack(witness_trace, primary)
}
return primary_lca_evidence, witness_lca_evidence
}
func DetectLightClientAttack(trace []LightBlock, peer PeerID) (LightClientAttackEvidence, []LightBlock) {
lightStore = new LightStore().Update(trace[0], StateTrusted)
for i in 1..len(trace)-1 {
lightStore, result = VerifyToTarget(peer, lightStore, trace[i].Header.Height)
if result == ResultFailure then return (nil, nil)
current = lightStore.Get(trace[i].Header.Height)
// if obtained header is the same as in the trace we continue with a next height
if current.Header == trace[i].Header continue
// we have identified a conflicting header
commonBlock = trace[i-1]
conflictingBlock = trace[i]
return (LightClientAttackEvidence { conflictingBlock, commonBlock.Header.Height },
Trace(lightStore, trace[i-1].Header.Height, trace[i].Header.Height))
}
return (nil, nil)
}
Evidence handling
As part of on chain evidence handling, full nodes identifies misbehaving processes and informs the application, so they can be slashed. Note that only bonded validators should be reported to the application. There are three types of attacks that can be executed against Tendermint light client:
- lunatic attack
- equivocation attack and
- amnesia attack.
We now specify the evidence handling logic.
func detectMisbehavingProcesses(lcAttackEvidence LightClientAttackEvidence, bc Blockchain) []ValidatorAddress {
assume IsValid(lcaEvidence, bc)
// lunatic light client attack
if !isValidBlock(current.Header, conflictingBlock.Header) {
conflictingCommit = lcAttackEvidence.ConflictingBlock.Commit
bondedValidators = GetNextValidators(bc, lcAttackEvidence.CommonHeight)
return getSigners(conflictingCommit) intersection GetAddresses(bondedValidators)
// equivocation light client attack
} else if current.Header.Round == conflictingBlock.Header.Round {
conflictingCommit = lcAttackEvidence.ConflictingBlock.Commit
trustedCommit = bc[conflictingBlock.Header.Height+1].LastCommit
return getSigners(trustedCommit) intersection getSigners(conflictingCommit)
// amnesia light client attack
} else {
HandleAmnesiaAttackEvidence(lcAttackEvidence, bc)
}
}
// Block validity in this context is defined by the trusted header.
func isValidBlock(trusted Header, conflicting Header) boolean {
return trusted.ValidatorsHash == conflicting.ValidatorsHash and
trusted.NextValidatorsHash == conflicting.NextValidatorsHash and
trusted.ConsensusHash == conflicting.ConsensusHash and
trusted.AppHash == conflicting.AppHash and
trusted.LastResultsHash == conflicting.LastResultsHash
}
func getSigners(commit Commit) []ValidatorAddress {
signers = []ValidatorAddress
for (i, commitSig) in commit.Signatures {
if commitSig.BlockIDFlag == BlockIDFlagCommit {
signers.append(commitSig.ValidatorAddress)
}
}
return signers
}
Note that amnesia attack evidence handling involves more complex processing, i.e., cannot be defined simply on amnesia attack evidence. We explain in the following section a protocol for handling amnesia attack evidence.
Amnesia attack evidence handling
Detecting faulty processes in case of the amnesia attack is more complex and cannot be inferred purely based on attack evidence data. In this case, in order to detect misbehaving processes we need access to votes processes sent/received during the conflicting height. Therefore, amnesia handling assumes that validators persist all votes received and sent during multi-round heights (as amnesia attack is only possible in heights that executes over multiple rounds, i.e., commit round > 0).
To simplify description of the algorithm we assume existence of the trusted oracle called monitor that will
drive the algorithm and output faulty processes at the end. Monitor can be implemented in a
distributed setting as on-chain module. The algorithm works as follows:
1) Monitor sends votesets request to validators of the conflicting height. Validators
are expected to send their votesets within predefined timeout.
2) Upon receiving votesets request, validators send their votesets to a monitor.
2) Validators which have not sent its votesets within timeout are considered faulty.
3) The preprocessing of the votesets is done. That means that the received votesets are analyzed
and each vote (valid) sent by process p is added to the voteset of the sender p. This phase ensures that
votes sent by faulty processes observed by at least one correct validator cannot be excluded from the analysis.
4) Votesets of every validator are analyzed independently to decide whether the validator is correct or faulty.
A faulty validators is the one where at least one of those invalid transitions is found:
- More than one PREVOTE message is sent in a round
- More than one PRECOMMIT message is sent in a round
- PRECOMMIT message is sent without receiving +2/3 of voting-power equivalent
appropriate PREVOTE messages
- PREVOTE message is sent for the value V’ in round r’ and the PRECOMMIT message had
been sent for the value V in round r by the same process (r’ > r) and there are no
+2/3 of voting-power equivalent PREVOTE(vr, V’) messages (vr ≥ 0 and vr > r and vr < r’)
as the justification for sending PREVOTE(r’, V’)
Tendermint fork detection and IBC fork detection
Status
This is a work in progress. This directory captures the ongoing work and discussion on fork detection both in the context of a Tendermint light node and in the context of IBC. It contains the following files
detection.md
a draft of the light node fork detection including "proof of fork" definition, that is, the data structure to submit evidence to full nodes.
discussions.md
A collection of ideas and intuitions from recent discussions
- the outcome of recent discussion
- a sketch of the light client supervisor to provide the context in which fork detection happens
- a discussion about lightstore semantics
req-ibc-detection.md
- a collection of requirements for fork detection in the IBC context. In particular it contains a section "Required Changes in ICS 007" with necessary updates to ICS 007 to support Tendermint fork detection
draft-functions.md
In order to address the collected requirements, we started to sketch some functions that we will need in the future when we specify in more detail the
- fork detections
- proof of fork generation
- proof of fork verification
on the following components.
- IBC on-chain components
- Relayer
TODOs
We decided to merge the files while there are still open points to address to record the current state an move forward. In particular, the following points need to be addressed:
-
https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466504876
-
https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466493900
-
https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466489045
-
https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466491471
Most likely we will write a specification on the light client supervisor along the outcomes of
that also addresses initialization
This an unfinished draft. Comments are welcome!
TODO: We will need to do small adaptations to the verification spec to reflect the semantics in the LightStore (verified, trusted, untrusted, etc. not needed anymore). In more detail:
-
The state of the Lightstore needs to go. Functions like
LatestVerifiedcan keep the name but will ignore state as it will not exist anymore. -
verification spec should be adapted to the second parameter of
VerifyToTargetbeing a lightblock; new version number of function tag; -
We should clarify what is the expectation of VerifyToTarget so if it returns TimeoutError it can be assumed faulty. I guess that VerifyToTarget with correct full node should never terminate with TimeoutError.
-
We need to introduce a new version number for the new specification. So we should decide how to handle that.
Light Client Attack Detector
In this specification, we strengthen the light client to be resistant against so-called light client attacks. In a light client attack, all the correct Tendermint full nodes agree on the sequence of generated blocks (no fork), but a set of faulty full nodes attack a light client by generating (signing) a block that deviates from the block of the same height on the blockchain. In order to do so, some of these faulty full nodes must have been validators before and violate [TMBC-FM-2THIRDS], as otherwise, if [TMBC-FM-2THIRDS] would hold, verification would satisfy [LCV-SEQ-SAFE.1].
An attack detector (or detector for short) is a mechanism that is used by the light client supervisor after verification of a new light block with the primary, to cross-check the newly learned light block with other peers (secondaries). It expects as input a light block with some height root (that serves as a root of trust), and a verification trace (a sequence of lightblocks) that the primary provided.
In case the detector observes a light client attack, it computes evidence data that can be used by Tendermint full nodes to isolate a set of faulty full nodes that are still within the unbonding period (more than 1/3 of the voting power of the validator set at some block of the chain), and report them via ABCI to the application of a Tendermint blockchain in order to punish faulty nodes.
Context of this document
The light client verification specification is designed for the Tendermint failure model (1/3 assumption) [TMBC-FM-2THIRDS]. It is safe under this assumption, and live if it can reliably (that is, no message loss, no duplication, and eventually delivered) and timely communicate with a correct full node. If [TMBC-FM-2THIRDS] assumption is violated, the light client can be fooled to trust a light block that was not generated by Tendermint consensus.
This specification, the attack detector, is a "second line of defense", in case the 1/3 assumption is violated. Its goal is to detect a light client attack (conflicting light blocks) and collect evidence. However, it is impractical to probe all full nodes. At this time we consider a simple scheme of maintaining an address book of known full nodes from which a small subset (e.g., 4) are chosen initially to communicate with. More involved book keeping with probabilistic guarantees can be considered at later stages of the project.
The light client maintains a simple address book containing addresses of full nodes that it can pick as primary and secondaries. To obtain a new light block, the light client first does verification with the primary, and then cross-checks the light block (and the trace of light blocks that led to it) with the secondaries using this specification.
Tendermint Consensus and Light Client Attacks
In this section we will give some mathematical definitions of what we mean by light client attacks (that are considered in this specification) and how they differ from main-chain forks. To this end we start by defining some properties of the sequence of blocks that is decided upon by Tendermint consensus in normal operation (if the Tendermint failure model holds [TMBC-FM-2THIRDS]), and then define different deviations that correspond to attack scenarios.
[TMBC-GENESIS.1]
Let Genesis be the agreed-upon initial block (file).
[TMBC-FUNC-SIGN.1]
Let b and c be two light blocks with b.Header.Height + 1 = c.Header.Height. We define the predicate signs(b,c) to hold iff c.Header.LastCommit is in PossibleCommit(b). [TMBC-SOUND-DISTR-POSS-COMMIT.1].
The above encodes sequential verification, that is, intuitively, b.Header.NextValidators = c.Header.Validators and 2/3 of these Validators signed c?
[TMBC-FUNC-SUPPORT.1]
Let b and c be two light blocks. We define the predicate supports(b,c,t) to hold iff
- t - trustingPeriod < b.Header.Time < t
- the voting power in b.NextValidators of nodes in c.Commit is more than 1/3 of TotalVotingPower(b.Header.NextValidators)
That is, if the Tendermint failure model holds, then c has been signed by at least one correct full node, cf. [TMBC-VAL-CONTAINS-CORR.1]. The following formalizes that b was properly generated by Tendermint; b can be traced back to genesis
[TMBC-SEQ-ROOTED.1]
Let b be a light block. We define sequ-rooted(b) iff for all i, 1 <= i < h = b.Header.Height, there exist light blocks a(i) s.t.
- a(1) = Genesis and
- a(h) = b and
- signs( a(i) , a(i+1) ).
The following formalizes that c is trusted based on b in skipping verification. Observe that we do not require here (yet) that b was properly generated.
[TMBC-SKIP-TRACE.1]
Let b and c be light blocks. We define skip-trace(b,c,t) if at time t there exists an h and a sequence a(1), ... a(h) s.t.
- a(1) = b and
- a(h) = c and
- supports( a(i), a(i+1), t), for all i, 1 <= i < h.
We call such a sequence a(1), ... a(h) a verification trace.
The following formalizes that two light blocks of the same height should agree on the content of the header. Observe that b and c may disagree on the Commit. This is a special case if the canonical commit has not been decided on, that is, if b.Header.Height is the maximum height of all blocks decided upon by Tendermint at this moment.
[TMBC-SIGN-SKIP-MATCH.1]
Let a, b, c, be light blocks and t a time, we define sign-skip-match(a,b,c,t) = true iff the following implication evaluates to true:
- sequ-rooted(a) and
- b.Header.Height = c.Header.Height and
- skip-trace(a,b,t)
- skip-trace(a,c,t)
implies b.Header = c.Header.
Observe that sign-skip-match is defined via an implication. If it evaluates to false this means that the left-hand-side of the implication evaluates to true, and the right-hand-side evaluates to false. In particular, there are two different headers b and c that both can be verified from a common block a from the chain. Thus, the following describes an attack.
[TMBC-ATTACK.1]
If there exists three light blocks a, b, and c, with sign-skip-match(a,b,c,t) = false then we have an attack. We say we have an attack at height b.Header.Height and write attack(a,b,c,t).
The lightblock a need not be unique, that is, there may be several blocks that satisfy the above requirement for the same blocks b and c.
[TMBC-ATTACK.1] is a formalization of the violation of the agreement property based on the result of consensus, that is, the generated blocks.
Remark. Violation of agreement is only possible if more than 1/3 of the validators (or next validators) of some previous block deviated from the protocol. The upcoming "accountability" specification will describe how to compute a set of at least 1/3 faulty nodes from two conflicting blocks. []
There are different ways to characterize forks and attack scenarios. This specification uses the "node-based characterization of attacks" which focuses on what kinds of nodes are affected (light nodes vs. full nodes). For future reference and discussion we also provide a "block-based characterization of attacks" below.
Node-based characterization of attacks
[TMBC-MC-FORK.1]
We say there is a (main chain) fork at time t if
- there are two correct full nodes i and j and
- i is different from j and
- i has decided on b and
- j has decided on c and
- there exist a such that attack(a,b,c,t).
[TMBC-LC-ATTACK.1]
We say there is a light client attack at time t, if
- there is no (main chain) fork [TMBC-MC-FORK.1], and
- there exist nodes that have computed light blocks b and c and
- there exist a such that attack(a,b,c,t).
We say the attack is at height a.Header.Height.
In this specification we consider detection of light client attacks. Intuitively, the case we consider is that light block b is the one from the blockchain, and some attacker has computed c and tries to wrongly convince the light client that c is the block from the chain.
[TMBC-LC-ATTACK-EVIDENCE.1]
We consider the following case of a light client attack [TMBC-LC-ATTACK.1]:
- attack(a,b,c,t)
- there is a peer p1 that has a sequence chain of blocks from a to b
- skip-trace(a,c,t): by [TMBC-SKIP-TRACE.1] there is a verification trace v of the form a = v(1), ... v(h) = c
Evidence for p1 (that proves an attack) consists for index i of v(i) and v(i+1) such that
- E1(i). v(i) is equal to the block of chain at height v(i).Height, and
- E2(i). v(i+1) that is different from the block of chain at height v(i+1).height
Observe p1 can
- check that v(i+1) differs from its block at that height, and
- verify v(i+1) in one step from v(i) as v is a verification trace.
Proposition. In the case of attack, evidence exists.
Proof. First observe that
- (A). (NOT E2(i)) implies E1(i+1)
Now by contradiction assume there is no evidence. Thus
- for all i, we have NOT E1(i) or NOT E2(i)
- for i = 1 we have E1(1) and thus NOT E2(1) thus by induction on i, by (A) we have for all i that E1(i)
- from attack we have E2(h-1), and as there is no evidence for i = h - 1 we get NOT E1(h-1). Contradiction. QED.
[TMBC-LC-EVIDENCE-DATA.1]
To prove the attack to p1, because of Point E1, it is sufficient to submit
- v(i).Height (rather than v(i)).
- v(i+1)
This information is evidence for height v(i).Height.
Block-based characterization of attacks
In this section we provide a different characterization of attacks. It is not defined on the nodes that are affected but purely on the content of the blocks. In that sense these definitions are less operational.
They might be relevant for a closer analysis of fork scenarios on the chain, which is out of the scope of this specification.
[TMBC-SIGN-UNIQUE.1]
Let b and c be light blocks, we define the predicate sign-unique(b,c) to evaluate to true iff the following implication evaluates to true:
- b.Header.Height = c.Header.Height and
- sequ-rooted(b) and
- sequ-rooted(c)
implies b = c.
[TMBC-BLOCKS-MCFORK.1]
If there exists two light blocks b and c, with sign-unique(b,c) = false then we have a fork.
The difference of the above definition to [TMBC-MC-FORK.1] is subtle. The latter requires a full node being affected by a bad block while [TMBC-BLOCKS-MCFORK.1] just requires that a bad block exists, possibly in memory of an attacker. The following captures a light client fork. There is no fork up to the height of block b. However, c is of that height, is different, and passes skipping verification. It is a stricter property than [TMBC-LC-ATTACK.1], as [TMBC-LC-ATTACK.1] requires that no correct full node is affected.
[TMBC-BLOCKS-LCFORK.1]
Let a, b, c, be light blocks and t a time. We define light-client-fork(a,b,c,t) iff
- sign-skip-match(a,b,c,t) = false and
- sequ-rooted(b) and
- b is "unique", that is, for all d, sequ-rooted(d) and d.Header.Height = b.Header.Height implies d = b
Finally, let us also define bogus blocks that have no support. Observe that bogus is even defined if there is a fork. Also, for the definition it would be sufficient to restrict a to a.height < b.height (which is implied by the definitions which unfold until supports()).
[TMBC-BOGUS.1]
Let b be a light block and t a time. We define bogus(b,t) iff
- sequ-rooted(b) = false and
- for all a, sequ-rooted(a) implies skip-trace(a,b,t) = false
Informal Problem statement
There is no sequential specification: the detector only makes sense in a distributed systems where some nodes misbehave.
We work under the assumption that full nodes and validators are responsible for detecting attacks on the main chain, and the evidence reactor takes care of broadcasting evidence to communicate misbehaving nodes via ABCI to the application, and halt the chain in case of a fork. The point of this specification is to shield a light clients against attacks that cannot be detected by full nodes, and are fully addressed at light clients (and consequently IBC relayers, which use the light client protocols to observe the state of a blockchain). In order to provide full nodes the incentive to follow the protocols when communicating with the light client, this specification also considers the generation of evidence that will also be processed by the Tendermint blockchain.
[LCD-IP-MODEL.1]
The detector is designed under the assumption that
- [TMBC-FM-2THIRDS] may be violated
- there is no fork on the main chain.
As a result some faulty full nodes may launch an attack on a light client.
The following requirements are operational in that they describe how things should be done, rather than what should be done. However, they do not constitute temporal logic verification conditions. For those, see [LCD-DIST-*] below.
The detector is called in the supervisor as follows
Evidences := AttackDetector(root_of_trust, verifiedLS);`
where
root-of-trustis a light block that is trusted (that is, except upon initialization, the primary and the secondaries agreed on in the past), andverifiedLSis a lightstore that contains a verification trace that starts from a lightblock that can be verified with theroot-of-trustin one step and ends with a lightblock of the height requested by the userEvidencesis a list of evidences for misbehavior
[LCD-IP-STATEMENT.1]
Whenever AttackDetector is called, the detector should for each
secondary try to replay the verification trace verifiedLS with the
secondary
- in case replaying leads to detection of a light client attack (one of the lightblocks differ from the one in verifiedLS with the same height), we should return evidence
- if the secondary cannot provide a verification trace, we have no proof for an attack. Block b may be bogus. In this case the secondary is faulty and it should be replaced.
Assumptions/Incentives/Environment
It is not in the interest of faulty full nodes to talk to the detector as long as the detector is connected to at least one correct full node. This would only increase the likelihood of misbehavior being detected. Also we cannot punish them easily (cheaply). The absence of a response need not be the fault of the full node.
Correct full nodes have the incentive to respond, because the detector may help them to understand whether their header is a good one. We can thus base liveness arguments of the detector on the assumptions that correct full nodes reliably talk to the detector.
Assumptions
[LCD-A-CorrFull.1]
At all times there is at least one correct full node among the primary and the secondaries.
For this version of the detection we take this assumption. It allows us to establish the invariant that the lightblock
root-of-trustis always the one from the blockchain, and we can use it as starting point for the evidence computation. Moreover, it allows us to establish the invariant at the supervisor that any lightblock in the (top-level) lightstore is from the blockchain.
In the future we might design a lightclient based on the assumption that at least in regular intervals the lightclient is connected to a correct full node. This will require the detector to reconsiderroot-of-trust, and remove lightblocks from the top-level lightstore.
[LCD-A-RelComm.1]
Communication between the detector and a correct full node is reliable and bounded in time. Reliable communication means that messages are not lost, not duplicated, and eventually delivered. There is a (known) end-to-end delay Delta, such that if a message is sent at time t then it is received and processed by time t + Delta. This implies that we need a timeout of at least 2 Delta for remote procedure calls to ensure that the response of a correct peer arrives before the timeout expires.
Definitions
Evidence
Following the definition of [TMBC-LC-ATTACK-EVIDENCE.1], by evidence we refer to a variable of the following type
[LC-DATA-EVIDENCE.1]
type LightClientAttackEvidence struct {
ConflictingBlock LightBlock
CommonHeight int64
}
As the above data is computed for a specific peer, the following data structure wraps the evidence and adds the peerID.
[LC-DATA-EVIDENCE-INT.1]
type InternalEvidence struct {
Evidence LightClientAttackEvidence
Peer PeerID
}
[LC-SUMBIT-EVIDENCE.1]
func submitEvidence(Evidences []InternalEvidence)
- Expected postcondition
- for each
evinEvidences: submitev.Evidencetoev.Peer
- for each
LightStore
Lightblocks and LightStores are defined in the verification specification [LCV-DATA-LIGHTBLOCK.1] and [LCV-DATA-LIGHTSTORE.1]. See the verification specification for details.
(Distributed) Problem statement
As the attack detector is there to reduce the impact of faulty nodes, and faulty nodes imply that there is a distributed system, there is no sequential specification to which this distributed problem statement may refer to.
The detector gets as input a trusted lightblock called root and an auxiliary lightstore called primary_trace with lightblocks that have been verified before, and that were provided by the primary.
[LCD-DIST-INV-ATTACK.1]
If the detector returns evidence for height h [TMBC-LC-EVIDENCE-DATA.1], then there is an attack at height h. [TMBC-LC-ATTACK.1]
[LCD-DIST-INV-STORE.1]
If the detector does not return evidence, then primary_trace contains only blocks from the blockchain.
[LCD-DIST-LIVE.1]
The detector eventually terminates.
[LCD-DIST-TERM-NORMAL.1]
If
- the primary_trace contains only blocks from the blockchain, and
- there is no attack, and
- Secondaries is always non-empty, and
- the age of root is always less than the trusting period,
then the detector does not return evidence.
[LCD-DIST-TERM-ATTACK.1]
If
- there is an attack, and
- a secondary reports a block that conflicts with one of the blocks in primary_trace, and
- Secondaries is always non-empty, and
- the age of root is always less than the trusting period,
then the detector returns evidence.
Observe that above we require that "a secondary reports a block that conflicts". If there is an attack, but no secondary tries to launch it against the detector (or the message from the secondary is lost by the network), then there is nothing to detect for us.
[LCD-DIST-SAFE-SECONDARY.1]
No correct secondary is ever replaced.
[LCD-DIST-SAFE-BOGUS.1]
If
- a secondary reports a bogus lightblock,
- the age of root is always less than the trusting period,
then the secondary is replaced before the detector terminates.
The above property is quite operational ("reports"), but it captures quite closely the requirement. As the detector only makes sense in a distributed setting, and does not have a sequential specification, less "pure" specification are acceptable.
Protocol
Functions and Data defined in other Specifications
From the supervisor
Replace_Secondary(addr Address, root-of-trust LightBlock)
From the verifier
func VerifyToTarget(primary PeerID, root LightBlock,
targetHeight Height) (LightStore, Result)
Note: the above differs from the current version in the second parameter. verification will be revised.
Observe that VerifyToTarget does communication with the secondaries
via the function FetchLightBlock.
Shared data of the light client
- a pool of full nodes FullNodes that have not been contacted before
- peer set called Secondaries
- primary
Note that the lightStore is not needed to be shared.
Outline
The problem laid out is solved by calling the function AttackDetector
with a lightstore that contains a light block that has just been
verified by the verifier.
Then AttackDetector downloads headers from the secondaries. In case
a conflicting header is downloaded from a secondary,
CreateEvidenceForPeer which computes evidence in the case that
indeed an attack is confirmed. It could be that the secondary reports
a bogus block, which means that there need not be an attack, and the
secondary is replaced.
Details of the functions
[LCD-FUNC-DETECTOR.1]:
func AttackDetector(root LightBlock, primary_trace []LightBlock)
([]InternalEvidence) {
Evidences := new []InternalEvidence;
for each secondary in Secondaries {
// we replay the primary trace with the secondary, in
// order to generate evidence that we can submit to the
// secodary. We return the evidence + the trace the
// secondary told us that spans the evidence at its local store
EvidenceForSecondary, newroot, secondary_trace, result :=
CreateEvidenceForPeer(secondary,
root,
primary_trace);
if result == FaultyPeer {
Replace_Secondary(root);
}
else if result == FoundEvidence {
// the conflict is not bogus
Evidences.Add(EvidenceForSecondary);
// we replay the secondary trace with the primary, ...
EvidenceForPrimary, _, result :=
CreateEvidenceForPeer(primary,
newroot,
secondary_trace);
if result == FoundEvidence {
Evidences.Add(EvidenceForPrimary);
}
// At this point we do not care about the other error
// codes. We already have generated evidence for an
// attack and need to stop the lightclient. It does not
// help to call replace_primary. Also we will use the
// same primary to check with other secondaries in
// later iterations of the loop
}
// In the case where the secondary reports NoEvidence
// we do nothing
}
return Evidences;
}
- Expected precondition
- root and primary trace are a verification trace
- Expected postcondition
- solves the problem statement (if attack found, then evidence is reported)
- Error condition
ErrorTrustExpired: fails if root expires (outside trusting period) [LCV-INV-TP.1]ErrorNoPeers: if no peers are left to replace secondaries, and no evidence was found before that happened
func CreateEvidenceForPeer(peer PeerID, root LightBlock, trace LightStore)
(Evidence, LightBlock, LightStore, result) {
common := root;
for i in 1 .. len(trace) {
auxLS, result := VerifyToTarget(peer, common, trace[i].Header.Height)
if result != ResultSuccess {
// something went wrong; peer did not provide a verifyable block
return (nil, nil, nil, FaultyPeer)
}
else {
if auxLS.LatestVerified().Header != trace[i].Header {
// the header reported by the peer differs from the
// reference header in trace but both could be
// verified from common in one step.
// we can create evidence for submission to the secondary
ev := new InternalEvidence;
ev.Evidence.ConflictingBlock := trace[i];
ev.Evidence.CommonHeight := common.Height;
ev.Peer := peer
return (ev, common, auxLS, FoundEvidence)
}
else {
// the peer agrees with the trace, we move common forward
// we could delete auxLS as it will be overwritten in
// the next iteration
common := trace[i]
}
}
}
return (nil, nil, nil, NoEvidence)
}
- Expected precondition
- root and trace are a verification trace
- Expected postcondition
- finds evidence where trace and peer diverge
- Error condition
ErrorTrustExpired: fails if root expires (outside trusting period) [LCV-INV-TP.1]- If
VerifyToTargetreturns error but root is not expired then returnFaultyPeer
Correctness arguments
Argument for [LCD-DIST-INV-ATTACK.1]
Under the assumption that root and trace are a verification trace,
when in CreateEvidenceForPeer the detector the detector creates
evidence, then the lightclient has seen two different headers (one via
trace and one via VerifyToTarget for the same height that can both
be verified in one step.
Argument for [LCD-DIST-INV-STORE.1]
We assume that there is at least one correct peer, and there is no fork. As a result the correct peer has the correct sequence of blocks. Since the primary_trace is checked block-by-block also against each secondary, and at no point evidence was generated that means at no point there were conflicting blocks.
Argument for [LCD-DIST-LIVE.1]
At the latest when [LCV-INV-TP.1] is violated,
AttackDetector terminates.
Argument for [LCD-DIST-TERM-NORMAL.1]
As there are finitely many peers, eventually the main loop terminates. As there is no attack no evidence can be generated.
Argument for [LCD-DIST-TERM-ATTACK.1]
Argument similar to [LCD-DIST-TERM-NORMAL.1]
Argument for [LCD-DIST-SAFE-SECONDARY.1]
Secondaries are only replaced if they time-out or if they report bogus blocks. The former is ruled out by the timing assumption, the latter by correct peers only reporting blocks from the chain.
Argument for [LCD-DIST-SAFE-BOGUS.1]
Once a bogus block is recognized as such the secondary is removed.
References
links to other specifications/ADRs this document refers to
[verification] The specification of the light client verification.
[supervisor] The specification of the light client supervisor.
Light Client Attack Detector
In this specification, we strengthen the light client to be resistant against so-called light client attacks. In a light client attack, all the correct Tendermint full nodes agree on the sequence of generated blocks (no fork), but a set of faulty full nodes attack a light client by generating (signing) a block that deviates from the block of the same height on the blockchain. In order to do so, some of these faulty full nodes must have been validators before and violate the assumption of more than two thirds of "correct voting power" [TMBC-FM-2THIRDS], as otherwise, if [TMBC-FM-2THIRDS] would hold, verification would satisfy [LCV-SEQ-SAFE.1].
An attack detector (or detector for short) is a mechanism that is used by the light client supervisor after verification of a new light block with the primary, to cross-check the newly learned light block with other peers (secondaries). It expects as input a light block with some height root (that serves as a root of trust), and a verification trace (a sequence of lightblocks) that the primary provided.
In case the detector observes a light client attack, it computes evidence data that can be used by Tendermint full nodes to isolate a set of faulty full nodes that are still within the unbonding period (more than 1/3 of the voting power of the validator set at some block of the chain), and report them via ABCI (application/blockchain interface) to the application of a Tendermint blockchain in order to punish faulty nodes.
Context of this document
The light client verification specification is designed for the Tendermint failure model (1/3 assumption) [TMBC-FM-2THIRDS]. It is safe under this assumption, and live if it can reliably (that is, no message loss, no duplication, and eventually delivered) and timely communicate with a correct full node. If [TMBC-FM-2THIRDS] assumption is violated, the light client can be fooled to trust a light block that was not generated by Tendermint consensus.
This specification, the attack detector, is a "second line of defense", in case the 1/3 assumption is violated. Its goal is to detect a light client attack (conflicting light blocks) and collect evidence. However, it is impractical to probe all full nodes. At this time we consider a simple scheme of maintaining an address book of known full nodes from which a small subset (e.g., 4) are chosen initially to communicate with. More involved book keeping with probabilistic guarantees can be considered at later stages of the project.
The light client maintains a simple address book containing addresses of full nodes that it can pick as primary and secondaries. To obtain a new light block, the light client first does verification with the primary, and then cross-checks the light block (and the trace of light blocks that led to it) with the secondaries using this specification.
Outline
-
Part I: Formal definitions of lightclient attacks, based on basic properties of Tendermint consensus.
-
Node-based characterization of attacks. The definition of attacks used in the problem statement of this specification.
-
Block-based characterization of attacks. Alternative definitions provided for future reference.
-
-
Part II: Problem statement of lightclient attack detection
-
Part III: The protocol
Part I - Tendermint Consensus and Light Client Attacks
In this section we will give some mathematical definitions of what we mean by light client attacks (that are considered in this specification) and how they differ from main-chain forks. To this end, we start by defining some properties of the sequence of blocks that is decided upon by Tendermint consensus in normal operation (if the Tendermint failure model holds [TMBC-FM-2THIRDS]), and then define different deviations that correspond to attack scenarios. We consider the notion of light blocks and headers.
[TMBC-GENESIS.1]
Let Genesis be the agreed-upon initial block (file).
[TMBC-FUNC-SIGN.1]
Let b and c be two light blocks with b.Header.Height + 1 = c.Header.Height. We define the predicate signs(b,c) to hold iff c.Header.LastCommit is in PossibleCommit(b). [TMBC-SOUND-DISTR-POSS-COMMIT.1].
The above encodes sequential verification, that is, intuitively, b.Header.NextValidators = c.Header.Validators and 2/3 of these Validators signed c.
[TMBC-FUNC-SUPPORT.1]
Let b and c be two light blocks. We define the predicate supports(b,c,t) to hold iff
- t - trustingPeriod < b.Header.Time < t
- the voting power in b.NextValidators of nodes in c.Commit is more than 1/3 of TotalVotingPower(b.Header.NextValidators)
That is, if the Tendermint failure model holds, then c has been signed by at least one correct full node, cf. [TMBC-VAL-CONTAINS-CORR.1]. The following formalizes that b was properly generated by Tendermint; b can be traced back to genesis.
[TMBC-SEQ-ROOTED.1]
Let b be a light block. We define sequ-rooted(b) iff for all i, 1 <= i < h = b.Header.Height, there exist light blocks a(i) s.t.
- a(1) = Genesis and
- a(h) = b and
- signs( a(i) , a(i+1) ).
The following formalizes that c is trusted based on b in skipping verification. Observe that we do not require here (yet) that b was properly generated.
[TMBC-SKIP-TRACE.1]
Let b and c be light blocks. We define skip-trace(b,c,t) if at time t there exists an integer h and a sequence a(1), ... a(h) s.t.
- a(1) = b and
- a(h) = c and
- supports( a(i), a(i+1), t), for all i, 1 <= i < h.
We call such a sequence a(1), ... a(h) a verification trace.
The following formalizes that two light blocks of the same height should agree on the content of the header. Observe that b and c may disagree on the Commit. This is a special case if the canonical commit has not been decided on yet, that is, if b.Header.Height is the maximum height of all blocks decided upon by Tendermint at this moment.
[TMBC-SIGN-SKIP-MATCH.1]
Let a, b, c, be light blocks and t a time, we define sign-skip-match(a,b,c,t) = true iff the following implication evaluates to true:
- sequ-rooted(a) and
- b.Header.Height = c.Header.Height and
- skip-trace(a,b,t)
- skip-trace(a,c,t)
implies b.Header = c.Header.
Observe that sign-skip-match is defined via an implication. If it evaluates to false this means that the left-hand-side of the implication evaluates to true, and the right-hand-side evaluates to false. In particular, there are two different headers b and c that both can be verified from a common block a from the chain. Thus, the following describes an attack.
[TMBC-ATTACK.1]
If there exists three light blocks a, b, and c, with sign-skip-match(a,b,c,t) = false then we have an attack. We say we have an attack at height b.Header.Height and write attack(a,b,c,t).
The lightblock a need not be unique, that is, there may be several blocks that satisfy the above requirement for the same blocks b and c.
[TMBC-ATTACK.1] is a formalization of the violation of the agreement property based on the result of consensus, that is, the generated blocks.
Remark. Violation of agreement is only possible if more than 1/3 of the validators (or next validators) of some previous block deviated from the protocol. The upcoming "accountability" specification will describe how to compute a set of at least 1/3 faulty nodes from two conflicting blocks. []
There are different ways to characterize forks and attack scenarios. This specification uses the "node-based characterization of attacks" which focuses on what kinds of nodes are affected (light nodes vs. full nodes). For future reference and discussion we also provide a "block-based characterization of attacks" below.
Node-based characterization of attacks
[TMBC-MC-FORK.1]
We say there is a (main chain) fork at time t if
- there are two correct full nodes i and j and
- i is different from j and
- i has decided on b and
- j has decided on c and
- there exist a such that attack(a,b,c,t).
[TMBC-LC-ATTACK.1]
We say there is a light client attack at time t, if
- there is no (main chain) fork [TMBC-MC-FORK.1], and
- there exist nodes that have computed light blocks b and c and
- there exist a such that attack(a,b,c,t).
We say the attack is at height a.Header.Height.
In this specification we consider detection of light client attacks. Intuitively, the case we consider is that light block b is the one from the blockchain, and some attacker has computed c and tries to wrongly convince the light client that c is the block from the chain.
[TMBC-LC-ATTACK-EVIDENCE.1]
We consider the following case of a light client attack [TMBC-LC-ATTACK.1]:
- attack(a,b,c,t)
- there is a peer p1 that has a sequence chain of blocks from a to b
- skip-trace(a,c,t): by [TMBC-SKIP-TRACE.1] there is a verification trace v of the form a = v(1), ... v(h) = c
Evidence for p1 (that proves an attack to p1) consists for index i of v(i) and v(i+1) such that
- E1(i). v(i) is equal to the block of chain at height v(i).Height, and
- E2(i). v(i+1) that is different from the block of chain at height v(i+1).height
Observe p1 can
- check that v(i+1) differs from its block at that height, and
- verify v(i+1) in one step from v(i) as v is a verification trace.
[TMBC-LC-EVIDENCE-DATA.1]
To prove the attack to p1, because of Point E1, it is sufficient to submit
- v(i).Height (rather than v(i)).
- v(i+1)
This information is evidence for height v(i).Height.
Block-based characterization of attacks
In this section we provide a different characterization of attacks. It is not defined on the nodes that are affected but purely on the content of the blocks. In that sense these definitions are less operational.
They might be relevant for a closer analysis of fork scenarios on the chain, which is out of the scope of this specification.
[TMBC-SIGN-UNIQUE.1]
Let b and c be light blocks, we define the predicate sign-unique(b,c) to evaluate to true iff the following implication evaluates to true:
- b.Header.Height = c.Header.Height and
- sequ-rooted(b) and
- sequ-rooted(c)
implies b = c.
[TMBC-BLOCKS-MCFORK.1]
If there exists two light blocks b and c, with sign-unique(b,c) = false then we have a fork.
The difference of the above definition to [TMBC-MC-FORK.1] is subtle. The latter requires a full node being affected by a bad block while [TMBC-BLOCKS-MCFORK.1] just requires that a bad block exists, possibly in memory of an attacker. The following captures a light client fork. There is no fork up to the height of block b. However, c is of that height, is different, and passes skipping verification. It is a stricter property than [TMBC-LC-ATTACK.1], as [TMBC-LC-ATTACK.1] requires that no correct full node is affected.
[TMBC-BLOCKS-LCFORK.1]
Let a, b, c, be light blocks and t a time. We define light-client-fork(a,b,c,t) iff
- sign-skip-match(a,b,c,t) = false and
- sequ-rooted(b) and
- b is "unique", that is, for all d, sequ-rooted(d) and d.Header.Height = b.Header.Height implies d = b
Finally, let us also define bogus blocks that have no support. Observe that bogus is even defined if there is a fork. Also, for the definition it would be sufficient to restrict a to a.height < b.height (which is implied by the definitions which unfold until supports()).
[TMBC-BOGUS.1]
Let b be a light block and t a time. We define bogus(b,t) iff
- sequ-rooted(b) = false and
- for all a, sequ-rooted(a) implies skip-trace(a,b,t) = false
Part II - Problem Statement
Informal Problem statement
There is no sequential specification: the detector only makes sense in a distributed systems where some nodes misbehave.
We work under the assumption that full nodes and validators are responsible for detecting attacks on the main chain, and the evidence reactor takes care of broadcasting evidence to communicate misbehaving nodes via ABCI to the application, and halt the chain in case of a fork. The point of this specification is to shield a light clients against attacks that cannot be detected by full nodes, and are fully addressed at light clients (and consequently IBC relayers, which use the light client protocols to observe the state of a blockchain). In order to provide full nodes the incentive to follow the protocols when communicating with the light client, this specification also considers the generation of evidence that will also be processed by the Tendermint blockchain.
[LCD-IP-MODEL.1]
The detector is designed under the assumption that
- [TMBC-FM-2THIRDS] may be violated
- there is no fork on the main chain.
As a result some faulty full nodes may launch an attack on a light client.
The following requirements are operational in that they describe how things should be done, rather than what should be done. However, they do not constitute temporal logic verification conditions. For those, see [LCD-DIST-*] below.
The detector is called in the supervisor as follows
Evidences := AttackDetector(root_of_trust, verifiedLS);`
where
root-of-trustis a light block that is trusted (that is, except upon initialization, the primary and the secondaries agreed on in the past), andverifiedLSis a lightstore that contains a verification trace that starts from a lightblock that can be verified with theroot-of-trustin one step and ends with a lightblock of the height requested by the userEvidencesis a list of evidences for misbehavior
[LCD-IP-STATEMENT.1]
Whenever AttackDetector is called, the detector should for each
secondary cross check the largest header in verifiedLS with the
corresponding header of the same height provided by the secondary. If
there is a deviation, the detector should
try to replay the verification trace verifiedLS with the
secondary
- in case replaying leads to detection of a light client attack (one of the lightblocks differ from the one in verifiedLS with the same height), we should return evidence
- if the secondary cannot provide a verification trace, we have no proof for an attack. Block b may be bogus. In this case the secondary is faulty and it should be replaced.
Assumptions
It is not in the interest of faulty full nodes to talk to the detector as long as the detector is connected to at least one correct full node. This would only increase the likelihood of misbehavior being detected. Also we cannot punish them easily (cheaply). The absence of a response need not be the fault of the full node.
Correct full nodes have the incentive to respond, because the detector may help them to understand whether their header is a good one. We can thus base liveness arguments of the detector on the assumptions that correct full nodes reliably talk to the detector.
[LCD-A-CorrFull.1]
At all times there is at least one correct full node among the primary and the secondaries.
For this version of the detection we take this assumption. It allows us to establish the invariant that the lightblock
root-of-trustis always the one from the blockchain, and we can use it as starting point for the evidence computation. Moreover, it allows us to establish the invariant at the supervisor that any lightblock in the (top-level) lightstore is from the blockchain.
In the future we might design a lightclient based on the assumption that at least in regular intervals the lightclient is connected to a correct full node. This will require the detector to reconsiderroot-of-trust, and remove lightblocks from the top-level lightstore.
[LCD-A-RelComm.1]
Communication between the detector and a correct full node is reliable and bounded in time. Reliable communication means that messages are not lost, not duplicated, and eventually delivered. There is a (known) end-to-end delay Delta, such that if a message is sent at time t then it is received and processed by time t + Delta. This implies that we need a timeout of at least 2 Delta for remote procedure calls to ensure that the response of a correct peer arrives before the timeout expires.
Definitions
Evidence
Following the definition of [TMBC-LC-ATTACK-EVIDENCE.1], by evidence we refer to a variable of the following type
[LC-DATA-EVIDENCE.1]
type LightClientAttackEvidence struct {
ConflictingBlock LightBlock
CommonHeight int64
// Evidence also includes application specific data which is not
// part of verification but is sent to the application once the
// evidence gets committed on chain.
}
As the above data is computed for a specific peer, the following data structure wraps the evidence and adds the peerID.
[LC-DATA-EVIDENCE-INT.1]
type InternalEvidence struct {
Evidence LightClientAttackEvidence
Peer PeerID
}
[LC-SUMBIT-EVIDENCE.1]
func submitEvidence(Evidences []InternalEvidence)
- Expected postcondition
- for each
evinEvidences: submitev.Evidencetoev.Peer
- for each
LightStore
Lightblocks and LightStores are defined in the verification specification [LCV-DATA-LIGHTBLOCK.1] and [LCV-DATA-LIGHTSTORE.2]. See the verification specification for details.
Distributed Problem statement
As the attack detector is there to reduce the impact of faulty nodes, and faulty nodes imply that there is a distributed system, there is no sequential specification to which this distributed problem statement may refer to.
The detector gets as input a trusted lightblock called root and an auxiliary lightstore called primary_trace with lightblocks that have been verified before, and that were provided by the primary.
[LCD-DIST-INV-ATTACK.1]
If the detector returns evidence for height h [TMBC-LC-EVIDENCE-DATA.1], then there is an attack at height h. [TMBC-LC-ATTACK.1]
[LCD-DIST-INV-STORE.1]
If the detector does not return evidence, then primary_trace contains only blocks from the blockchain.
[LCD-DIST-LIVE.1]
The detector eventually terminates.
[LCD-DIST-TERM-NORMAL.1]
If
- the primary_trace contains only blocks from the blockchain, and
- there is no attack, and
- Secondaries is always non-empty, and
- the age of root is always less than the trusting period,
then the detector does not return evidence.
[LCD-DIST-TERM-ATTACK.1]
If
- there is an attack, and
- a secondary reports a block that conflicts with one of the blocks in primary_trace, and
- Secondaries is always non-empty, and
- the age of root is always less than the trusting period,
then the detector returns evidence.
Observe that above we require that "a secondary reports a block that conflicts". If there is an attack, but no secondary tries to launch it against the detector (or the message from the secondary is lost by the network), then there is nothing to detect for us.
[LCD-DIST-SAFE-SECONDARY.1]
No correct secondary is ever replaced.
[LCD-DIST-SAFE-BOGUS.1]
If
- a secondary reports a bogus lightblock,
- the age of root is always less than the trusting period,
then the secondary is replaced before the detector terminates.
The above property is quite operational (e.g., the usage of "reports"), but it captures closely the requirement. As the detector only makes sense in a distributed setting, and does not have a sequential specification, a less "pure" specification are acceptable.
Part III - Protocol
Functions and Data defined in other Specifications
From the supervisor
Replace_Secondary(addr Address, root-of-trust LightBlock)
From the verifier
func VerifyToTarget(primary PeerID, root LightBlock,
targetHeight Height) (LightStore, Result)
Observe that VerifyToTarget does communication with the secondaries
via the function FetchLightBlock.
Shared data of the light client
- a pool of full nodes FullNodes that have not been contacted before
- peer set called Secondaries
- primary
Note that the lightStore is not needed to be shared.
Outline of solution
The problem laid out is solved by calling the function AttackDetector
with a lightstore that contains a light block that has just been
verified by the verifier.
Then AttackDetector downloads headers from the secondaries. In case
a conflicting header is downloaded from a secondary, it calls
CreateEvidenceForPeer which computes evidence in the case that
indeed an attack is confirmed. It could be that the secondary reports
a bogus block, which means that there need not be an attack, and the
secondary is replaced.
Details of the functions
[LCD-FUNC-DETECTOR.2]:
func AttackDetector(root LightBlock, primary_trace []LightBlock)
([]InternalEvidence) {
Evidences := new []InternalEvidence;
for each secondary in Secondaries {
lb, result := FetchLightBlock(secondary,primary_trace.Latest().Header.Height);
if result != ResultSuccess {
Replace_Secondary(root);
}
else if lb.Header != primary_trace.Latest().Header {
// we replay the primary trace with the secondary, in
// order to generate evidence that we can submit to the
// secondary. We return the evidence + the trace the
// secondary told us that spans the evidence at its local store
EvidenceForSecondary, newroot, secondary_trace, result :=
CreateEvidenceForPeer(secondary,
root,
primary_trace);
if result == FaultyPeer {
Replace_Secondary(root);
}
else if result == FoundEvidence {
// the conflict is not bogus
Evidences.Add(EvidenceForSecondary);
// we replay the secondary trace with the primary, ...
EvidenceForPrimary, _, result :=
CreateEvidenceForPeer(primary,
newroot,
secondary_trace);
if result == FoundEvidence {
Evidences.Add(EvidenceForPrimary);
}
// At this point we do not care about the other error
// codes. We already have generated evidence for an
// attack and need to stop the lightclient. It does not
// help to call replace_primary. Also we will use the
// same primary to check with other secondaries in
// later iterations of the loop
}
// In the case where the secondary reports NoEvidence
// after initially it reported a conflicting header.
// secondary is faulty
Replace_Secondary(root);
}
}
return Evidences;
}
- Expected precondition
- root and primary trace are a verification trace
- Expected postcondition
- solves the problem statement (if attack found, then evidence is reported)
- Error condition
ErrorTrustExpired: fails if root expires (outside trusting period) [LCV-INV-TP.1]ErrorNoPeers: if no peers are left to replace secondaries, and no evidence was found before that happened
func CreateEvidenceForPeer(peer PeerID, root LightBlock, trace LightStore)
(Evidence, LightBlock, LightStore, result) {
common := root;
for i in 1 .. len(trace) {
auxLS, result := VerifyToTarget(peer, common, trace[i].Header.Height)
if result != ResultSuccess {
// something went wrong; peer did not provide a verifiable block
return (nil, nil, nil, FaultyPeer)
}
else {
if auxLS.LatestVerified().Header != trace[i].Header {
// the header reported by the peer differs from the
// reference header in trace but both could be
// verified from common in one step.
// we can create evidence for submission to the secondary
ev := new InternalEvidence;
ev.Evidence.ConflictingBlock := trace[i];
// CommonHeight is used to indicate the type of attack
// if the CommonHeight != ConflictingBlock.Height this
// is by definition a lunatic attack else it is an
// equivocation attack
ev.Evidence.CommonHeight := common.Height;
ev.Peer := peer
return (ev, common, auxLS, FoundEvidence)
}
else {
// the peer agrees with the trace, we move common forward.
// we could delete auxLS as it will be overwritten in
// the next iteration
common := trace[i]
}
}
}
return (nil, nil, nil, NoEvidence)
}
- Expected precondition
- root and trace are a verification trace
- Expected postcondition
- finds evidence where trace and peer diverge
- Error condition
ErrorTrustExpired: fails if root expires (outside trusting period) [LCV-INV-TP.1]- If
VerifyToTargetreturns error but root is not expired then returnFaultyPeer
Correctness arguments
On the existence of evidence
Proposition. In the case of attack,
evidence [TMBC-LC-ATTACK-EVIDENCE.1]
exists.
Proof. First observe that
- (A). (NOT E2(i)) implies E1(i+1)
Now by contradiction assume there is no evidence. Thus
- for all i, we have NOT E1(i) or NOT E2(i)
- for i = 1 we have E1(1) and thus NOT E2(1) thus by induction on i, by (A) we have for all i that E1(i)
- from attack we have E2(h-1), and as there is no evidence for i = h - 1 we get NOT E1(h-1). Contradiction. QED.
Argument for [LCD-DIST-INV-ATTACK.1]
Under the assumption that root and trace are a verification trace,
when in CreateEvidenceForPeer the detector creates
evidence, then the lightclient has seen two different headers (one via
trace and one via VerifyToTarget) for the same height that can both
be verified in one step.
Argument for [LCD-DIST-INV-STORE.1]
We assume that there is at least one correct peer, and there is no fork. As a result, the correct peer has the correct sequence of blocks. Since the primary_trace is checked block-by-block also against each secondary, and at no point evidence was generated that means at no point there were conflicting blocks.
Argument for [LCD-DIST-LIVE.1]
At the latest when [LCV-INV-TP.1] is violated,
AttackDetector terminates.
Argument for [LCD-DIST-TERM-NORMAL.1]
As there are finitely many peers, eventually the main loop terminates. As there is no attack no evidence can be generated.
Argument for [LCD-DIST-TERM-ATTACK.1]
Argument similar to [LCD-DIST-TERM-NORMAL.1]
Argument for [LCD-DIST-SAFE-SECONDARY.1]
Secondaries are only replaced if they time-out or if they report bogus blocks. The former is ruled out by the timing assumption, the latter by correct peers only reporting blocks from the chain.
Argument for [LCD-DIST-SAFE-BOGUS.1]
Once a bogus block is recognized as such the secondary is removed.
References
links to other specifications/ADRs this document refers to
[verification] The specification of the light client verification.
[supervisor] The specification of the light client supervisor.
Results of Discussions and Decisions
- Generating a minimal proof of fork (as suggested in Issue #5083) is too costly at the light client
- we do not know all lightblocks from the primary
- therefore there are many scenarios. we might even need to ask the primary again for additional lightblocks to isolate the branch.
For instance, the light node starts with block at height 1 and the primary provides a block of height 10 that the light node can verify immediately. In cross-checking, a secondary now provides a conflicting header b10 of height 10 that needs another header b5 of height 5 to verify. Now, in order for the light node to convince the primary:
- The light node cannot just sent b5, as it is not clear whether the fork happened before or after 5
- The light node cannot just send b10, as the primary would also need b5 for verification
- In order to minimize the evidence, the light node may try to figure out where the branch happens, e.g., by asking the primary for height 5 (it might be that more queries are required, also to the secondary. However, assuming that in this scenario the primary is faulty it may not respond.
As the main goal is to catch misbehavior of the primary, evidence generation and punishment must not depend on their cooperation. So the moment we have proof of fork (even if it contains several light blocks) we should submit right away.
-
decision: "full" proof of fork consists of two traces that originate in the same lightblock and lead to conflicting headers of the same height.
-
For submission of proof of fork, we may do some optimizations, for instance, we might just submit a trace of lightblocks that verifies a block different from the one the full node knows (we do not send the trace the primary gave us back to the primary)
-
The light client attack is via the primary. Thus we try to catch if the primary installs a bad light block
- We do not check secondary against secondary
- For each secondary, we check the primary against one secondary
-
Observe that just two blocks for the same height are not sufficient proof of fork. One of the blocks may be bogus [TMBC-BOGUS.1] which does not constitute slashable behavior.
Which leads to the question whether the light node should try to do fork detection on its initial block (from subjective initialization). This could be done by doing backwards verification (with the hashes) until a bifurcation block is found. While there are scenarios where a fork could be found, there is also the scenario where a faulty full node feeds the light node with bogus light blocks and forces the light node to check hashes until a bogus chain is out of the trusting period. As a result, the light client should not try to detect a fork for its initial header. The initial header must be trusted as is.
Light Client Sequential Supervisor
TODO: decide where (into which specification) to put the following:
We describe the context on which the fork detector is called by giving a sequential version of the supervisor function. Roughly, it alternates two phases namely:
- Light Client Verification. As a result, a header of the required height has been downloaded from and verified with the primary.
- Light Client Fork Detections. As a result the header has been cross-checked with the secondaries. In case there is a fork we submit "proof of fork" and exit.
[LC-FUNC-SUPERVISOR.1]:
func Sequential-Supervisor () (Error) {
loop {
// get the next height
nextHeight := input();
// Verify
result := NoResult;
while result != ResultSuccess {
lightStore,result := VerifyToTarget(primary, lightStore, nextHeight);
if result == ResultFailure {
// pick new primary (promote a secondary to primary)
/// and delete all lightblocks above
// LastTrusted (they have not been cross-checked)
Replace_Primary();
}
}
// Cross-check
PoFs := Forkdetector(lightStore, PoFs);
if PoFs.Empty {
// no fork detected with secondaries, we trust the new
// lightblock
LightStore.Update(testedLB, StateTrusted);
}
else {
// there is a fork, we submit the proofs and exit
for i, p range PoFs {
SubmitProofOfFork(p);
}
return(ErrorFork);
}
}
}
TODO: finish conditions
- Implementation remark
- Expected precondition
- lightStore initialized with trusted header
- PoFs empty
- Expected postcondition
- runs forever, or
- is terminated by user and satisfies LightStore invariant, or TODO
- has submitted proof of fork upon detecting a fork
- Error condition
- none
Semantics of the LightStore
Currently, a lightblock in the lightstore can be in one of the following states:
- StateUnverified
- StateVerified
- StateFailed
- StateTrusted
The intuition is that StateVerified captures that the lightblock has
been verified with the primary, and StateTrusted is the state after
successful cross-checking with the secondaries.
Assuming there is always one correct node among primary and
secondaries, and there is no fork on the blockchain, lightblocks that
are in StateTrusted can be used by the user with the guarantee of
"finality". If a block in StateVerified is used, it might be that
detection later finds a fork, and a roll-back might be needed.
Remark: The assumption of one correct node, does not render verification useless. It is true that if the primary and the secondaries return the same block we may trust it. However, if there is a node that provides a different block, the light node still needs verification to understand whether there is a fork, or whether the different block is just bogus (without any support of some previous validator set).
Remark: A light node may choose the full nodes it communicates with (the light node and the full node might even belong to the same stakeholder) so the assumption might be justified in some cases.
In the future, we will do the following changes
- we assume that only from time to time, the light node is connected to a correct full node
- this means for some limited time, the light node might have no means to defend against light client attacks
- as a result we do not have finality
- once the light node reconnects with a correct full node, it should detect the light client attack and submit evidence.
Under these assumptions, StateTrusted loses its meaning. As a
result, it should be removed from the API. We suggest that we replace
it with a flag "trusted" that can be used
- internally for efficiency reasons (to maintain [LCD-INV-TRUSTED-AGREED.1] until a fork is detected)
- by light client based on the "one correct full node" assumption
Draft of Functions for Fork detection and Proof of Fork Submisstion
This document collects drafts of function for generating and submitting proof of fork in the IBC context
On-chain IBC Component
The following is a suggestions to change the function defined in ICS 007
[TAG-IBC-MISBEHAVIOR.1]
func checkMisbehaviourAndUpdateState(cs: ClientState, PoF: LightNodeProofOfFork)
TODO: finish conditions
- Implementation remark
- Expected precondition
- PoF.TrustedBlock.Header is equal to lightBlock on store with same height
- both traces end with header of same height
- headers are different
- both traces are supported by PoF.TrustedBlock (
supportsdefined in [TMBC-FUNC]), that is, fort = currentTimestamp()(see ICS 024)- supports(PoF.TrustedBlock, PoF.PrimaryTrace[1], t)
- supports(PoF.PrimaryTrace[i], PoF.PrimaryTrace[i+1], t) for 0 < i < length(PoF.PrimaryTrace)
- supports(PoF.TrustedBlock, PoF.SecondaryTrace[1], t)
- supports(PoF.SecondaryTrace[i], PoF.SecondaryTrace[i+1], t) for 0 < i < length(PoF.SecondaryTrace)
- Expected postcondition
- set cs.FrozenHeight to min(cs.FrozenHeight, PoF.TrustedBlock.Header.Height)
- Error condition
- none
The following is a suggestions to add functionality to ICS 002 and 007. I suppose the above is the most efficient way to get the required information. Another option is to subscribe to "header install" events via CosmosSDK
[TAG-IBC-HEIGHTS.1]
func QueryHeightsRange(id, from, to) ([]Height)
- Expected postcondition
- returns all heights h, with from <= h <= to for which the IBC component has a consensus state.
This function can be used if the relayer has no information about the IBC component. This allows late-joining relayers to also participate in fork dection and the generation in proof of fork. Alternatively, we may also postulate that relayers are not responsible to detect forks for heights before they started (and subscribed to the transactions reporting fresh headers being installed at the IBC component).
Relayer
Auxiliary Functions to be implemented in the Light Client
[LCV-LS-FUNC-GET-PREV.1]
func (ls LightStore) GetPreviousVerified(height Height) (LightBlock, bool)
- Expected postcondition
- returns a verified LightBlock, whose height is maximal among all
verified lightblocks with height smaller than
height
- returns a verified LightBlock, whose height is maximal among all
verified lightblocks with height smaller than
Relayer Submitting Proof of Fork to the IBC Component
There are two ways the relayer can detect a fork
- by the fork detector of one of its lightclients
- be checking the consensus state of the IBC component
The following function ignores how the proof of fork was generated.
It takes a proof of fork as input and computes a proof of fork that
will be accepted by the IBC component.
The problem addressed here is that both, the relayer's light client
and the IBC component have incomplete light stores, that might
not have all light blocks in common.
Hence the relayer has to figure out what the IBC component knows
(intuitively, a meeting point between the two lightstores
computed in commonRoot) and compute a proof of fork
(extendPoF) that the IBC component will accept based on its
knowledge.
The auxiliary functions commonRoot and extendPoF are
defined below.
[TAG-SUBMIT-POF-IBC.1]
func SubmitIBCProofOfFork(
lightStore LightStore,
PoF: LightNodeProofOfFork,
ibc IBCComponent) (Error) {
if ibc.queryChainConsensusState(PoF.TrustedBlock.Height) = PoF.TrustedBlock {
// IBC component has root of PoF on store, we can just submit
ibc.submitMisbehaviourToClient(ibc.id,PoF)
return Success
// note sure about the id parameter
}
else {
// the ibc component does not have the TrustedBlock and might
// even be on yet a different branch. We have to compute a PoF
// that the ibc component can verifiy based on its current
// knowledge
ibcLightBlock, lblock, _, result := commonRoot(lightStore, ibc, PoF.TrustedBlock)
if result = Success {
newPoF = extendPoF(ibcLightBlock, lblock, lightStore, PoF)
ibc.submitMisbehaviourToClient(ibc.id, newPoF)
return Success
}
else{
return CouldNotGeneratePoF
}
}
}
TODO: finish conditions
- Implementation remark
- Expected precondition
- Expected postcondition
- Error condition
- none
Auxiliary Functions at the Relayer
If the relayer detects a fork, it has to compute a proof of fork that will convince the IBC component. That is it has to compare the relayer's local lightstore against the lightstore of the IBC component, and find common ancestor lightblocks.
[TAG-COMMON-ROOT.1]
func commonRoot(lightStore LightStore, ibc IBCComponent, lblock
LightBlock) (LightBlock, LightBlock, LightStore, Result) {
auxLS.Init
// first we ask for the heights the ibc component is aware of
ibcHeights = ibc.QueryHeightsRange(
ibc.id,
lightStore.LowestVerified().Height,
lblock.Height - 1);
// this function does not exist yet. Alternatively, we may
// request all transactions that installed headers via CosmosSDK
for {
h, result = max(ibcHeights)
if result = Empty {
return (_, _, _, NoRoot)
}
ibcLightBlock = ibc.queryChainConsensusState(h)
auxLS.Update(ibcLightBlock, StateVerified);
connector, result := Connector(lightStore, ibcLightBlock, lblock.Header.Height)
if result = success {
return (ibcLightBlock, connector, auxLS, Success)
}
else{
ibcHeights.remove(h)
}
}
}
- Expected postcondition
- returns
- a lightBlock b1 from the IBC component, and
- a lightBlock b2 from the local lightStore with height less than lblock.Header.Hight, s.t. b1 supports b2, and
- a lightstore with the blocks downloaded from the ibc component
- returns
[TAG-LS-FUNC-CONNECT.1]
func Connector (lightStore LightStore, lb LightBlock, h Height) (LightBlock, bool)
- Expected postcondition
- returns a verified LightBlock from lightStore with height less than h that can be verified by lb in one step.
TODO: for the above to work we need an invariant that all verified lightblocks form a chain of trust. Otherwise, we need a lightblock that has a chain of trust to height.
Once the common root is found, a proof of fork that will be accepted by the IBC component needs to be generated. This is done in the following function.
[TAG-EXTEND-POF.1]
func extendPoF (root LightBlock,
connector LightBlock,
lightStore LightStore,
Pof LightNodeProofofFork) (LightNodeProofofFork}
- Implementation remark
- PoF is not sufficient to convince an IBC component, so we extend the proof of fork farther in the past
- Expected postcondition
- returns a newPOF:
- newPoF.TrustedBlock = root
- let prefix =
connector +
lightStore.Subtrace(connector.Header.Height, PoF.TrustedBlock.Header.Height-1) +
PoF.TrustedBlock
- newPoF.PrimaryTrace = prefix + PoF.PrimaryTrace
- newPoF.SecondaryTrace = prefix + PoF.SecondaryTrace
- returns a newPOF:
Detection a fork at the IBC component
The following functions is assumed to be called regularly to check that latest consensus state of the IBC component. Alternatively, this logic can be executed whenever the relayer is informed (via an event) that a new header has been installed.
[TAG-HANDLER-DETECT-FORK.1]
func DetectIBCFork(ibc IBCComponent, lightStore LightStore) (LightNodeProofOfFork, Error) {
cs = ibc.queryClientState(ibc);
lb, found := lightStore.Get(cs.Header.Height)
if !found {
**TODO:** need verify to target
lb, result = LightClient.Main(primary, lightStore, cs.Header.Height)
// [LCV-FUNC-IBCMAIN.1]
**TODO** decide what to do following the outcome of Issue #499
// I guess here we have to get into the light client
}
if cs != lb {
// IBC component disagrees with my primary.
// I fetch the
ibcLightBlock, lblock, ibcStore, result := commonRoot(lightStore, ibc, lb)
pof = new LightNodeProofOfFork;
pof.TrustedBlock := ibcLightBlock
pof.PrimaryTrace := ibcStore + cs
pof.SecondaryTrace := lightStore.Subtrace(lblock.Header.Height,
lb.Header.Height);
return(pof, Fork)
}
return(nil , NoFork)
}
TODO: finish conditions
- Implementation remark
- we ask the handler for the lastest check. Cross-check with the chain. In case they deviate we generate PoF.
- we assume IBC component is correct. It has verified the consensus state
- Expected precondition
- Expected postcondition
Requirements for Fork Detection in the IBC Context
What you need to know about IBC
In the following, I distilled what I considered relevant from
https://github.com/cosmos/ics/tree/master/spec/ics-002-client-semantics
Components and their interface
Tendermint Blockchains
I assume you know what that is.
An IBC/Tendermint correspondence
| IBC Term | Tendermint-RS Spec Term | Comment |
|---|---|---|
CommitmentRoot | AppState | app hash |
ConsensusState | Lightblock | not all fields are there. NextValidator is definitly needed |
ClientState | latest light block + configuration parameters (e.g., trusting period + frozenHeight | NextValidators missing; what is proofSpecs? |
frozenHeight | height of fork | set when a fork is detected |
| "would-have-been-fooled" | light node fork detection | light node may submit proof of fork to IBC component to halt it |
Height | (no epochs) | (epoch,height) pair in lexicographical order (compare) |
Header | ~signed header | validatorSet explicit (no hash); nextValidators missing |
Evidence | t.b.d. | definition unclear "which the light client would have considered valid". Data structure will need to change |
verify | ValidAndVerified | signature does not match perfectly (ClientState vs. LightBlock) + in checkMisbehaviourAndUpdateState it is unclear whether it uses traces or goes to h1 and h2 in one step |
Some IBC links
Required Changes in ICS 007
-
assert(height > 0)in definition ofinitialisedoesn't match definition ofHeightas (epoch,height) pair. -
initialiseneeds to be updated to new data structures -
clientState.frozenHeightsemantics seem not totally consistent in document. E.g.,minneeds to be defined over optional value incheckMisbehaviourAndUpdateState. Also, if you are frozen, why do you accept more evidence. -
checkValidityAndUpdateStateverify: it needs to be clarified that checkValidityAndUpdateState does not perform "bisection" (as currently hinted in the text) but performs a single step of "skipping verification", called,ValidAndVerifiedassert (header.height > clientState.latestHeight): no old headers can be installed. This might be OK, but we need to check interplay with misbehavior- clienstState needs to be updated according to complete data structure
-
checkMisbehaviourAndUpdateState: as evidence will contain a trace (or two), the assertion that uses verify will need to change. -
ICS 002 states w.r.t.
queryChainConsensusStatethat "Note that retrieval of past consensus states by height (as opposed to just the current consensus state) is convenient but not required." For Tendermint fork detection, this seems to be a necessity. -
Headershould become a lightblock -
Evidenceshould becomeLightNodeProofOfFork[LCV-DATA-POF.1] -
upgradeClientStatewhat is the semantics (in particular what isheightdoing?). -
checkMisbehaviourAndUpdateState(cs: ClientState, PoF: LightNodeProofOfFork)needs to be adapted
Handler
A blockchain runs a handler that passively collects information about other blockchains. It can be thought of a state machine that takes input events.
-
the state includes a lightstore (I guess called
ConsensusStatein IBC) -
The following function is used to pass a header to a handler
type checkValidityAndUpdateState = (Header) => Void
For Tendermint, it will perform
ValidandVerified, that is, it does the trusting period check and the
+1/3 check (+2/3 for sequential headers).
If it verifies a header, it adds it to its lightstore,
if it does not pass verification it drops it.
Right now it only accepts a header more recent then the latest
header,
and drops older
ones or ones that could not be verified.
The above paragraph captures what I believe what is the current logic of
checkValidityAndUpdateState. It may be subject to change. E.g., maintain a lightstore with state (unverified, verified)
- The following function is used to pass "evidence" (this we will need to make precise eventually) to a handler
type checkMisbehaviourAndUpdateState = (bytes) => Void
We have to design this, and the data that the handler can use to check that there was some misbehavior (fork) in order react on it, e.g., flagging a situation and stop the protocol.
- The following function is used to query the light store (
ConsensusState)
type queryChainConsensusState = (height: uint64) => ConsensusState
Relayer
-
The active components are called relayer.
-
a relayer contains light clients to two (or more?) blockchains
-
the relayer send headers and data to the handler to invoke
checkValidityAndUpdateStateandcheckMisbehaviourAndUpdateState. It may also queryqueryChainConsensusState. -
multiple relayers may talk to one handler. Some relayers might be faulty. We assume existence of at least single correct relayer.
Informal Problem Statement: Fork detection in IBC
Relayer requirement: Evidence for Handler
-
The relayer should provide the handler with "evidence" that there was a fork.
-
The relayer can read the handler's consensus state. Thus the relayer can feed the handler precisely the information the handler needs to detect a fork. What is this information needs to be specified.
-
The information depends on the verification the handler does. It might be necessary to provide a bisection proof (list of lightblocks) so that the handler can verify based on its local lightstore a header h that is conflicting with a header h' in the local lightstore, that is, h != h' and h.Height = h'.Height
Relayer requirement: Fork detection
Let's assume there is a fork at chain A. There are two ways the relayer can figure that out:
-
as the relayer contains a light client for A, it also includes a fork detector that can detect a fork.
-
the relayer may also detect a fork by observing that the handler for chain A (on chain B) is on a different branch than the relayer
-
in both detection scenarios, the relayer should submit evidence to full nodes of chain A where there is a fork. As we assume a fullnode has a complete list of blocks, it is sufficient to send "Bucky's evidence" (https://github.com/tendermint/tendermint/issues/5083), that is,
- two lightblocks from different branches +
- a lightblock (perhaps just a height) from which both blocks can be verified.
-
in the scenario 2., the relayer must feed the A-handler (on chain B) a proof of a fork on A so that chain B can react accordingly
Handler requirement
-
there are potentially many relayers, some correct some faulty
-
a handler cannot trust the information provided by the relayer, but must verify (Доверя́й, но проверя́й)
-
in case of a fork, we accept that the handler temporarily stores headers (tagged as verified).
-
eventually, a handler should be informed (
checkMisbehaviourAndUpdateState) by some relayer that it has verified a header from a fork. Then the handler should do what is required by IBC in this case (stop?)
Challenges in the handler requirement
-
handlers and relayers work on different lightstores. In principle the lightstore need not intersect in any heights a priori
-
if a relayer sees a header h it doesn't know at a handler (
queryChainConsensusState), the relayer needs to verify that header. If it cannot do it locally based on downloaded and verified (trusted?) light blocks, it might need to useVerifyToTarget(bisection). To callVerifyToTargetwe might keep h in the lightstore. If verification fails, we need to download the "alternative" header of height h.Height to generate evidence for the handler. -
we have to specify what precisely
queryChainConsensusStatereturns. It cannot be the complete lightstore. Is the last header enough? -
we would like to assume that every now and then (smaller than the trusting period) a correct relayer checks whether the handler is on a different branch than the relayer. And we would like that this is enough to achieve the Handler requirement.
-
here the correctness argument would be easy if a correct relayer is based on a light client with a trusted state, that is, a light client who never changes its opinion about trusted. Then if such a correct relayer checks-in with a handler, it will detect a fork, and act in time.
-
if the light client does not provide this interface, in the case of a fork, we need some assumption about a correct relayer being on a different branch than the handler, and we need such a relayer to check-in not too late. Also what happens if the relayer's light client is forced to roll-back its lightstore? Does it have to re-check all handlers?
-
On the interconnectedness of things
In the broader discussion of so-called "fork accountability" there are several subproblems
-
Fork detection
-
Evidence creation and submission
-
Isolating misbehaving nodes (and report them for punishment over abci)
Fork detection
The preliminary specification ./detection.md formalizes the notion of a fork. Roughly, a fork exists if there are two conflicting headers for the same height, where both are supported by bonded full nodes (that have been validators in the near past, that is, within the trusting period). We distinguish between fork on the chain where two conflicting blocks are signed by +2/3 of the validators of that height, and a light client fork where one of the conflicting headers is not signed by +2/3 of the current height, but by +1/3 of the validators of some smaller height.
In principle everyone can detect a fork
-
./detection talks about the Tendermint light client with a focus on light nodes. A relayer runs such light clients and may detect forks in this way
-
in IBC, a relayer can see that a handler is on a conflicting branch
- the relayer should feed the handler the necessary information so that it can halt
- the relayer should report the fork to a full node
Evidence creation and submission
-
the information sent from the relayer to the handler could be called evidence, but this is perhaps a bad idea because the information sent to a full node can also be called evidence. But this evidence might still not be enough as the full node might need to run the "fork accountability" protocol to generate evidence in the form of consensus messages. So perhaps we should introduce different terms for:
- proof of fork for the handler (basically consisting of lightblocks)
- proof of fork for a full node (basically consisting of (fewer) lightblocks)
- proof of misbehavior (consensus messages)
Isolating misbehaving nodes
-
this is the job of a full node.
-
might be subjective in the future: the protocol depends on what the full node believes is the "correct" chain. Right now we postulate that every full node is on the correct chain, that is, there is no fork on the chain.
-
The full node figures out which nodes are
- lunatic
- double signing
- amnesic; using the challenge response protocol
-
We do not punish "phantom" validators
- currently we understand a phantom validator as a node that
- signs a block for a height in which it is not in the validator set
- the node is not part of the +1/3 of previous validators that are used to support the header. Whether we call a validator phantom might be subjective and depend on the header we check against. Their formalization actually seems not so clear.
- they can only do something if there are +1/3 faulty validators that are either lunatic, double signing, or amnesic.
- abci requires that we only report bonded validators. So if a node is a "phantom", we would need the check whether the node is bonded, which currently is expensive, as it requires checking blocks from the last three weeks.
- in the future, with state sync, a correct node might be convinced by faulty nodes that it is in the validator set. Then it might appear to be "phantom" although it behaves correctly
- currently we understand a phantom validator as a node that
Next steps
The following points are subject to my limited knowledge of the state of the work on IBC. Some/most of it might already exist and we will just need to bring everything together.
-
"proof of fork for a full node" defines a clean interface between fork detection and misbehavior isolation. So it should be produced by protocols (light client, the relayer). So we should fix that first.
-
Given the problems of not having a light client architecture spec, for the relayer we should start with this. E.g.
- the relayer runs light clients for two chains
- the relayer regularly queries consensus state of a handler
- the relayer needs to check the consensus state
- this involves local checks
- this involves calling the light client
- the relayer uses the light client to do IBC business (channels, packets, connections, etc.)
- the relayer submits proof of fork to handlers and full nodes
the list is definitely not complete. I think part of this (perhaps all) is covered by what Anca presented recently.
We will need to define what we expect from these components
-
for the parts where the relayer talks to the handler, we need to fix the interface, and what the handler does
-
we write specs for these components.
Draft of Light Client Supervisor for discussion
TODOs
This specification in done in parallel with updates on the verification specification. So some hyperlinks have to be placed to the correct files eventually.
Light Client Sequential Supervisor
The light client implements a read operation of a header from the blockchain, by communicating with full nodes, a so-called primary and several so-called witnesses. As some full nodes may be faulty, this functionality must be implemented in a fault-tolerant way.
In the Tendermint blockchain, the validator set may change with every new block. The staking and unbonding mechanism induces a security model: starting at time Time of the header, more than two-thirds of the next validators of a new block are correct for the duration of TrustedPeriod.
Light Client Verification implements the fault-tolerant read operation designed for this security model. That is, it is safe if the model assumptions are satisfied and makes progress if it communicates to a correct primary.
However, if the security model is violated, faulty peers (that have been validators at some point in the past) may launch attacks on the Tendermint network, and on the light client. These attacks as well as an axiomatization of blocks in general are defined in a document that contains the definitions that are currently in detection.md.
If there is a light client attack (but no successful attack on the network), the safety of the verification step may be violated (as we operate outside its basic assumption). The light client also contains a defense mechanism against light clients attacks, called detection.
Light Client Detection implements a cross check of the result of the verification step. If there is a light client attack, and the light client is connected to a correct peer, the light client as a whole is safe, that is, it will not operate on invalid blocks. However, in this case it cannot successfully read, as inconsistent blocks are in the system. However, in this case the detection performs a distributed computation that results in so-called evidence. Evidence can be used to prove to a correct full node that there has been a light client attack.
Light Client Evidence Accountability is a protocol run on a full node to check whether submitted evidence indeed proves the existence of a light client attack. Further, from the evidence and its own knowledge about the blockchain, the full node computes a set of bonded full nodes (that at some point had more than one third of the voting power) that participated in the attack that will be reported via ABCI to the application.
In this document we specify
- Initialization of the Light Client
- The interaction of verification and detection
The details of these two protocols are captured in their own documents, as is the accountability protocol.
Another related line is IBC attack detection and submission at the relayer, as well as attack verification at the IBC handler. This will call for yet another spec.
Status
This document is work in progress. In order to develop the specification step-by-step, it assumes certain details of verification and detection that are not specified in the respective current versions yet. This inconsistencies will be addresses over several upcoming PRs.
Part I - Tendermint Blockchain
Part II - Sequential Problem Definition
[LC-SEQ-INIT-LIVE.1]
Upon initialization, the light client gets as input a header of the blockchain, or the genesis file of the blockchain, and eventually stores a header of the blockchain.
[LC-SEQ-LIVE.1]
The light client gets a sequence of heights as inputs. For each input height targetHeight, it eventually stores the header of height targetHeight.
[LC-SEQ-SAFE.1]
The light client never stores a header which is not in the blockchain.
Part III - Light Client as Distributed System
Computational Model
The light client communicates with remote processes only via the verification and the detection protocols. The respective assumptions are given there.
Distributed Problem Statement
Two Kinds of Liveness
In case of light client attacks, the sequential problem statement cannot always be satisfied. The lightclient cannot decide which block is from the chain and which is not. As a result, the light client just creates evidence, submits it, and terminates. For the liveness property, we thus add the possibility that instead of adding a lightblock, we also might terminate in case there is an attack.
[LC-DIST-TERM.1]
The light client either runs forever or it terminates on attack.
Design choices
[LC-DIST-STORE.1]
The light client has a local data structure called LightStore that contains light blocks (that contain a header).
The light store exposes functions to query and update it. They are specified here.
See light store invariant [LCV-INV-LS-ROOT.2].
[LC-DIST-SAFE.1]
It is always the case that every header in LightStore was generated by an instance of Tendermint consensus.
[LC-DIST-LIVE.1]
Whenever the light client gets a new height h as input,
- and there is no light client attack up to height h, then the lightclient eventually puts the lightblock of height h in the lightstore and wait for another input.
- otherwise, that is, if there
is a light client attack on height h, then the light client
must perform one of the following:
- it terminates on attack.
- it eventually puts the lightblock of height h in the lightstore and wait for another input.
Observe that the "existence of a lightclient attack" just means that some node has generated a conflicting block. It does not necessarily mean that a (faulty) peer sends such a block to "our" lightclient. Thus, even if there is an attack somewhere in the system, our lightclient might still continue to operate normally.
Solving the sequential specification
[LC-DIST-SAFE.1] is guaranteed by the detector; in particular it follows from [LCD-DIST-INV-STORE.1] [LCD-DIST-LIVE.1]
Part IV - Light Client Supervisor Protocol
We provide a specification for a sequential Light Client Supervisor.
The local code for verification is presented by a sequential function
Sequential-Supervisor to highlight the control flow of this
functionality. Each lightblock is first verified with a primary, and then
cross-checked with secondaries, and if all goes well, the lightblock
is
added (with the attribute "trusted") to the
lightstore. Intermiate lightblocks that were used to verify the target
block but were not cross-checked are stored as "verified"
We note that if a different concurrency model is considered for an implementation, the semantics of the lightstore might change: In a concurrent implementation, we might do verification for some height h, add the lightblock to the lightstore, and start concurrent threads that
- do verification for the next height h' != h
- do cross-checking for height h. If we find an attack, we remove h from the lightstore.
- the user might already start to use h
Thus, this concurrency model changes the semantics of the lightstore (not all lightblocks that are read by the user are trusted; they may be removed if we find a problem). Whether this is desirable, and whether the gain in performance is worth it, we keep for future versions/discussion of lightclient protocols.
Definitions
Peers
[LC-DATA-PEERS.1]:
A fixed set of full nodes is provided in the configuration upon initialization. Initially this set is partitioned into
- one full node that is the primary (singleton set),
- a set Secondaries (of fixed size, e.g., 3),
- a set FullNodes; it excludes primary and Secondaries nodes.
- A set FaultyNodes of nodes that the light client suspects of being faulty; it is initially empty
[LC-INV-NODES.1]:
The detector shall maintain the following invariants:
- FullNodes \intersect Secondaries = {}
- FullNodes \intersect FaultyNodes = {}
- Secondaries \intersect FaultyNodes = {}
and the following transition invariant
- FullNodes' \union Secondaries' \union FaultyNodes' = FullNodes \union Secondaries \union FaultyNodes
[LC-FUNC-REPLACE-PRIMARY.1]:
Replace_Primary(root-of-trust LightBlock)
- Implementation remark
- the primary is replaced by a secondary
- to maintain a constant size of secondaries, need to
- pick a new secondary nsec while ensuring [LC-INV-ROOT-AGREED.1]
- that is, we need to ensure that root-of-trust = FetchLightBlock(nsec, root-of-trust.Header.Height)
- Expected precondition
- FullNodes is nonempty
- Expected postcondition
- primary is moved to FaultyNodes
- a secondary s is moved from Secondaries to primary
- Error condition
- if precondition is violated
[LC-FUNC-REPLACE-SECONDARY.1]:
Replace_Secondary(addr Address, root-of-trust LightBlock)
- Implementation remark
- maintain [LC-INV-ROOT-AGREED.1], that is, ensure root-of-trust = FetchLightBlock(nsec, root-of-trust.Header.Height)
- Expected precondition
- FullNodes is nonempty
- Expected postcondition
- addr is moved from Secondaries to FaultyNodes
- an address nsec is moved from FullNodes to Secondaries
- Error condition
- if precondition is violated
Data Types
The core data structure of the protocol is the LightBlock.
[LC-DATA-LIGHTBLOCK.1]
type LightBlock struct {
Header Header
Commit Commit
Validators ValidatorSet
NextValidators ValidatorSet
Provider PeerID
}
[LC-DATA-LIGHTSTORE.1]
LightBlocks are stored in a structure which stores all LightBlock from initialization or received from peers.
type LightStore struct {
...
}
We use the functions that the LightStore exposes, which are defined in the verification specification.
Inputs
The lightclient is initialized with LCInitData
[LC-DATA-INIT.1]
type LCInitData struct {
lightBlock LightBlock
genesisDoc GenesisDoc
}
where only one of the components must be provided. GenesisDoc is
defined in the Tenderdash
Types.
[LC-DATA-GENESIS.1]
type GenesisDoc struct {
GenesisTime time.Time `json:"genesis_time"`
ChainID string `json:"chain_id"`
InitialHeight int64 `json:"initial_height"`
ConsensusParams *tmproto.ConsensusParams `json:"consensus_params,omitempty"`
Validators []GenesisValidator `json:"validators,omitempty"`
AppHash tmbytes.HexBytes `json:"app_hash"`
AppState json.RawMessage `json:"app_state,omitempty"`
}
We use the following function
makeblock so that we create a lightblock from the genesis
file in order to do verification based on the data from the genesis
file using the same verification function we use in normal operation.
[LC-FUNC-MAKEBLOCK.1]
func makeblock (genesisDoc GenesisDoc) (lightBlock LightBlock))
- Implementation remark
- none
- Expected precondition
- none
- Expected postcondition
- lightBlock.Header.Height = genesisDoc.InitialHeight
- lightBlock.Header.Time = genesisDoc.GenesisTime
- lightBlock.Header.LastBlockID = nil
- lightBlock.Header.LastCommit = nil
- lightBlock.Header.Validators = genesisDoc.Validators
- lightBlock.Header.NextValidators = genesisDoc.Validators
- lightBlock.Header.Data = nil
- lightBlock.Header.AppState = genesisDoc.AppState
- lightBlock.Header.LastResult = nil
- lightBlock.Commit = nil
- lightBlock.Validators = genesisDoc.Validators
- lightBlock.NextValidators = genesisDoc.Validators
- lightBlock.Provider = nil
- Error condition
- none
Configuration Parameters
[LC-INV-ROOT-AGREED.1]
In the Sequential-Supervisor, it is always the case that the primary and all secondaries agree on lightStore.Latest().
Assumptions
We have to assume that the initialization data (the lightblock or the genesis file) are consistent with the blockchain. This is subjective initialization and it cannot be checked locally.
Invariants
[LC-INV-PEERLIST.1]:
The peer list contains a primary and a secondary.
If the invariant is violated, the light client does not have enough peers to download headers from. As a result, the light client needs to terminate in case this invariant is violated.
Supervisor
Outline
The supervisor implements the functionality of the lightclient. It is initialized with a genesis file or with a lightblock the user trusts. This initialization is subjective, that is, the security of the lightclient is based on the validity of the input. If the genesis file or the lightblock deviate from the actual ones on the blockchain, the lightclient provides no guarantees.
After initialization, the supervisor awaits an input, that is, the height of the next lightblock that should be obtained. Then it downloads, verifies, and cross-checks a lightblock, and if all tests go through, the light block (and possibly other lightblocks) are added to the lightstore, which is returned in an output event to the user.
The following main loop does the interaction with the user (input, output) and calls the following two functions:
InitLightClient: it initializes the lightstore either with the provided lightblock or with the lightblock that corresponds to the first block generated by the blockchain (by the validators defined by the genesis file)VerifyAndDetect: takes as input a lightstore and a height and returns the updated lightstore.
[LC-FUNC-SUPERVISOR.1]:
func Sequential-Supervisor (initdata LCInitData) (Error) {
lightStore,result := InitLightClient(initData);
if result != OK {
return result;
}
loop {
// get the next height
nextHeight := input();
lightStore,result := VerifyAndDetect(lightStore, nextHeight);
if result == OK {
output(LightStore.Get(targetHeight));
// we only output a trusted lightblock
}
else {
return result
}
// QUESTION: is it OK to generate output event in normal case,
// and terminate with failure in the (light client) attack case?
}
}
- Implementation remark
- infinite loop unless a light client attack is detected
- In typical implementations (e.g., the one in Rust),
there are mutliple input actions:
VerifytoLatest,LatestTrusted, andGetStatus. The information can be easily obtained from the lightstore, so that we do not treat these requests explicitly here but just consider the request for a block of a given height which requires more involved computation and communication.
- Expected precondition
- LCInitData contains a genesis file or a lightblock.
- Expected postcondition
- if a light client attack is detected: it stops and submits
evidence (in
InitLightClientorVerifyAndDetect) - otherwise: non. It runs forever.
- if a light client attack is detected: it stops and submits
evidence (in
- Invariant: lightStore contains trusted lightblocks only.
- Error condition
- if
InitLightClientorVerifyAndDetectfails (if a attack is detected, or if [LCV-INV-TP.1] is violated)
- if
Details of the Functions
Initialization
The light client is based on subjective initialization. It has to trust the initial data given to it by the user. It cannot do any detection of attack. So either upon initialization we obtain a lightblock and just initialize the lightstore with it. Or in case of a genesis file, we download, verify, and cross-check the first block, to initialize the lightstore with this first block. The reason is that we want to maintain [LCV-INV-TP.1] from the beginning.
If the lightclient is initialized with a lightblock, one might think it may increase trust, when one cross-checks the initial light block. However, if a peer provides a conflicting lightblock, the question is to distinguish the case of a bogus block (upon which operation should proceed) from a light client attack (upon which operation should stop). In case of a bogus block, the lightclient might be forced to do backwards verification until the blocks are out of the trusting period, to make sure no previous validator set could have generated the bogus block, which effectively opens up a DoS attack on the lightclient without adding effective robustness.
[LC-FUNC-INIT.1]:
func InitLightClient (initData LCInitData) (LightStore, Error) {
if LCInitData.LightBlock != nil {
// we trust the provided initial block.
newblock := LCInitData.LightBlock
}
else {
genesisBlock := makeblock(initData.genesisDoc);
result := NoResult;
while result != ResultSuccess {
current = FetchLightBlock(PeerList.primary(), genesisBlock.Header.Height + 1)
// QUESTION: is the height with "+1" OK?
if CANNOT_VERIFY = ValidAndVerify(genesisBlock, current) {
Replace_Primary();
}
else {
result = ResultSuccess
}
}
// cross-check
auxLS := new LightStore
auxLS.Add(current)
Evidences := AttackDetector(genesisBlock, auxLS)
if Evidences.Empty {
newBlock := current
}
else {
// [LC-SUMBIT-EVIDENCE.1]
submitEvidence(Evidences);
return(nil, ErrorAttack);
}
}
lightStore := new LightStore;
lightStore.Add(newBlock);
return (lightStore, OK);
}
- Implementation remark
- none
- Expected precondition
- LCInitData contains either a genesis file of a lightblock
- if genesis it passes
ValidateAndComplete()see Tendermint
- Expected postcondition
- lightStore initialized with trusted lightblock. It has either been cross-checked (from genesis) or it has initial trust from the user.
- Error condition
- if precondition is violated
- empty peerList
Main verification and detection logic
[LC-FUNC-MAIN-VERIF-DETECT.1]:
func VerifyAndDetect (lightStore LightStore, targetHeight Height)
(LightStore, Result) {
b1, r1 = lightStore.Get(targetHeight)
if r1 == true {
if b1.State == StateTrusted {
// block already there and trusted
return (lightStore, ResultSuccess)
}
else {
// We have a lightblock in the store, but it has not been
// cross-checked by now. We do that now.
root_of_trust, auxLS := lightstore.TraceTo(b1);
// Cross-check
Evidences := AttackDetector(root_of_trust, auxLS);
if Evidences.Empty {
// no attack detected, we trust the new lightblock
lightStore.Update(auxLS.Latest(),
StateTrusted,
verfiedLS.Latest().verification-root);
return (lightStore, OK);
}
else {
// there is an attack, we exit
submitEvidence(Evidences);
return(lightStore, ErrorAttack);
}
}
}
// get the lightblock with maximum height smaller than targetHeight
// would typically be the heighest, if we always move forward
root_of_trust, r2 = lightStore.LatestPrevious(targetHeight);
if r2 = false {
// there is no lightblock from which we can do forward
// (skipping) verification. Thus we have to go backwards.
// No cross-check needed. We trust hashes. Therefore, we
// directly return the result
return Backwards(primary, lightStore.Lowest(), targetHeight)
}
else {
// Forward verification + detection
result := NoResult;
while result != ResultSuccess {
verifiedLS,result := VerifyToTarget(primary,
root_of_trust,
nextHeight);
if result == ResultFailure {
// pick new primary (promote a secondary to primary)
Replace_Primary(root_of_trust);
}
else if result == ResultExpired {
return (lightStore, result)
}
}
// Cross-check
Evidences := AttackDetector(root_of_trust, verifiedLS);
if Evidences.Empty {
// no attack detected, we trust the new lightblock
verifiedLS.Update(verfiedLS.Latest(),
StateTrusted,
verfiedLS.Latest().verification-root);
lightStore.store_chain(verifidLS);
return (lightStore, OK);
}
else {
// there is an attack, we exit
return(lightStore, ErrorAttack);
}
}
}
- Implementation remark
- none
- Expected precondition
- none
- Expected postcondition
- lightblock of height targetHeight (and possibly additional blocks) added to lightStore
- Error condition
- an attack is detected
- [LC-DATA-PEERLIST-INV.1] is violated
Draft of Light Client Supervisor for discussion
Modification to the initialization
The lightclient is initialized with LCInitData
[LC-DATA-INIT.2]
type LCInitData struct {
TrustedBlock LightBlock
Genesis GenesisDoc
TrustedHash []byte
TrustedHeight int64
}
where only one of the components must be provided. GenesisDoc is
defined in the Tenderdash
Types.
Initialization
The light client is based on subjective initialization. It has to trust the initial data given to it by the user. It cannot perform any detection of an attack yet instead requires an initial point of trust. There are three forms of initial data which are used to obtain the first trusted block:
- A trusted block from a prior initialization
- A trusted height and hash
- A genesis file
The golang light client implementation checks this initial data in that order; first attempting to find a trusted block from the trusted store, then acquiring a light block from the primary at the trusted height and matching the hash, or finally checking for a genesis file to verify the initial header.
The light client doesn't need to check if the trusted block is within the trusted period because it already trusts it, however, if the light block is outside the trust period, there is a higher chance the light client won't be able to verify anything.
Cross-checking this trusted block with providers upon initialization is helpful for ensuring that the node is responsive and correctly configured but does not increase trust since proving a conflicting block is a light client attack and not just a bogus block could result in performing backwards verification beyond the trusted period, thus a fruitless endeavour.
However, with the notion of it's better to fail earlier than later, the golang light client implementation will perform a consistency check on all providers and will error if one returns a different header, allowing the user the opportunity to reinitialize.
[LC-FUNC-INIT.2]:
func InitLightClient(initData LCInitData) (LightStore, Error) {
var initialBlock LightBlock
switch {
case LCInitData.TrustedBlock != nil:
// we trust the block from a prior initialization
initialBlock = LCInitData.TrustedBlock
case LCInitData.TrustedHash != nil:
untrustedBlock := FetchLightBlock(PeerList.Primary(), LCInitData.TrustedHeight)
// verify that the hashes match
if untrustedBlock.Hash() != LCInitData.TrustedHash {
return nil, Error("Primary returned block with different hash")
}
// after checking the hash we now trust the block
initialBlock = untrustedBlock
}
case LCInitData.Genesis != nil:
untrustedBlock := FetchLightBlock(PeerList.Primary(), LCInitData.Genesis.InitialHeight)
// verify that 2/3+ of the validator set signed the untrustedBlock
if err := VerifyCommitFull(untrustedBlock.Commit, LCInitData.Genesis.Validators); err != nil {
return nil, err
}
// we can now trust the block
initialBlock = untrustedBlock
default:
return nil, Error("No initial data was provided")
// This is done in the golang version but is optional and not strictly part of the protocol
if err := CrossCheck(initialBlock, PeerList.Witnesses()); err != nil {
return nil, err
}
// initialize light store
lightStore := new LightStore;
lightStore.Add(newBlock);
return (lightStore, OK);
}
func CrossCheck(lb LightBlock, witnesses []Provider) error {
for _, witness := range witnesses {
witnessBlock := FetchLightBlock(witness, lb.Height)
if witnessBlock.Hash() != lb.Hash() {
return Error("Witness has different block")
}
}
return OK
}
- Implementation remark
- none
- Expected precondition
- LCInitData contains either a genesis file of a lightblock
- if genesis it passes
ValidateAndComplete()see Tendermint
- Expected postcondition
- lightStore initialized with trusted lightblock. It has either been cross-checked (from genesis) or it has initial trust from the user.
- Error condition
- if precondition is violated
- empty peerList
Core Verification
Problem statement
We assume that the light client knows a (base) header inithead it trusts (by social consensus or because
the light client has decided to trust the header before). The goal is to check whether another header
newhead can be trusted based on the data in inithead.
The correctness of the protocol is based on the assumption that inithead was generated by an instance of
Tendermint consensus.
Failure Model
For the purpose of the following definitions we assume that there exists a function
validators that returns the corresponding validator set for the given hash.
The light client protocol is defined with respect to the following failure model:
Given a known bound TRUSTED_PERIOD, and a block b with header h generated at time Time
(i.e. h.Time = Time), a set of validators that hold more than 2/3 of the voting power
in validators(b.Header.NextValidatorsHash) is correct until time b.Header.Time + TRUSTED_PERIOD.
Assumption: "correct" is defined w.r.t. realtime (some Newtonian global notion of time, i.e., wall time),
while Header.Time corresponds to the BFT time. In this note, we assume that clocks of correct processes
are synchronized (for example using NTP), and therefore there is bounded clock drift (CLOCK_DRIFT) between local clocks and
BFT time. More precisely, for every correct light client process and every header.Time (i.e. BFT Time, for a header correctly
generated by the Tendermint consensus), the following inequality holds: Header.Time < now + CLOCK_DRIFT,
where now corresponds to the system clock at the light client process.
Furthermore, we assume that TRUSTED_PERIOD is (several) order of magnitude bigger than CLOCK_DRIFT (TRUSTED_PERIOD >> CLOCK_DRIFT),
as CLOCK_DRIFT (using NTP) is in the order of milliseconds and TRUSTED_PERIOD is in the order of weeks.
We expect a light client process defined in this document to be used in the context in which there is some
larger period during which misbehaving validators can be detected and punished (we normally refer to it as UNBONDING_PERIOD
due to the "bonding" mechanism in modern proof of stake systems). Furthermore, we assume that
TRUSTED_PERIOD < UNBONDING_PERIOD and that they are normally of the same order of magnitude, for example
TRUSTED_PERIOD = UNBONDING_PERIOD / 2.
The specification in this document considers an implementation of the light client under the Failure Model defined above.
Mechanisms like fork accountability and evidence submission are defined in the context of UNBONDING_PERIOD and
they incentivize validators to follow the protocol specification defined in this document. If they don't,
and we have 1/3 (or more) faulty validators, safety may be violated. Our approach then is
to detect these cases (after the fact), and take suitable repair actions (automatic and social).
This is discussed in document on Fork accountability.
The term "trusted" above indicates that the correctness of the protocol depends on
this assumption. It is in the responsibility of the user that runs the light client to make sure that the risk
of trusting a corrupted/forged inithead is negligible.
Remark: This failure model might change to a hybrid version that takes heights into account in the future.
High Level Solution
Upon initialization, the light client is given a header inithead it trusts (by
social consensus). When a light clients sees a new signed header snh, it has to decide whether to trust the new
header. Trust can be obtained by (possibly) the combination of three methods.
-
Uninterrupted sequence of headers. Given a trusted header
hand an untrusted headerh1, the light client trusts a headerh1if it trusts all headers in betweenhandh1. -
Trusted period. Given a trusted header
h, an untrusted headerh1 > handTRUSTED_PERIODduring which the failure model holds, we can check whether at least one validator, that has been continuously correct fromh.Timeuntil now, has signedh1. If this is the case, we can trusth1. -
Bisection. If a check according to 2. (trusted period) fails, the light client can try to obtain a header
hpwhose height lies betweenhandh1in order to check whetherhcan be used to get trust forhp, andhpcan be used to get trust forsnh. If this is the case we can trusth1; if not, we continue recursively until either we found set of headers that can build (transitively) trust relation betweenhandh1, or we failed as two consecutive headers don't verify against each other.
Definitions
Data structures
In the following, only the details of the data structures needed for this specification are given.
type Header struct {
Height int64
Time Time // the chain time when the header (block) was generated
LastBlockID BlockID // prev block info
ValidatorsHash []byte // hash of the validators for the current block
NextValidatorsHash []byte // hash of the validators for the next block
}
type SignedHeader struct {
Header Header
Commit Commit // commit for the given header
}
type ValidatorSet struct {
Validators []Validator
TotalVotingPower int64
}
type Validator struct {
Address Address // validator address (we assume validator's addresses are unique)
VotingPower int64 // validator's voting power
}
type TrustedState {
SignedHeader SignedHeader
ValidatorSet ValidatorSet
}
Functions
For the purpose of this light client specification, we assume that the Tendermint Full Node exposes the following functions over Tendermint RPC:
// returns signed header: Header with Commit, for the given height
func Commit(height int64) (SignedHeader, error)
// returns validator set for the given height
func Validators(height int64) (ValidatorSet, error)
Furthermore, we assume the following auxiliary functions:
// returns true if the commit is for the header, ie. if it contains
// the correct hash of the header; otherwise false
func matchingCommit(header Header, commit Commit) bool
// returns the set of validators from the given validator set that
// committed the block (that correctly signed the block)
// it assumes signature verification so it can be computationally expensive
func signers(commit Commit, validatorSet ValidatorSet) []Validator
// returns the voting power the validators in v1 have according to their voting power in set v2
// it does not assume signature verification
func votingPowerIn(v1 []Validator, v2 ValidatorSet) int64
// returns hash of the given validator set
func hash(v2 ValidatorSet) []byte
In the functions below we will be using trustThreshold as a parameter. For simplicity
we assume that trustThreshold is a float between 1/3 and 2/3 and we will not be checking it
in the pseudo-code.
VerifySingle. The function VerifySingle attempts to validate given untrusted header and the corresponding validator sets
based on a given trusted state. It ensures that the trusted state is still within its trusted period,
and that the untrusted header is within assumed clockDrift bound of the passed time now.
Note that this function is not making external (RPC) calls to the full node; the whole logic is
based on the local (given) state. This function is supposed to be used by the IBC handlers.
func VerifySingle(untrustedSh SignedHeader,
untrustedVs ValidatorSet,
untrustedNextVs ValidatorSet,
trustedState TrustedState,
trustThreshold float,
trustingPeriod Duration,
clockDrift Duration,
now Time) (TrustedState, error) {
if untrustedSh.Header.Time > now + clockDrift {
return (trustedState, ErrInvalidHeaderTime)
}
trustedHeader = trustedState.SignedHeader.Header
if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
return (state, ErrHeaderNotWithinTrustedPeriod)
}
// we assume that time it takes to execute verifySingle function
// is several order of magnitudes smaller than trustingPeriod
error = verifySingle(
trustedState,
untrustedSh,
untrustedVs,
untrustedNextVs,
trustThreshold)
if error != nil return (state, error)
// the untrusted header is now trusted
newTrustedState = TrustedState(untrustedSh, untrustedNextVs)
return (newTrustedState, nil)
}
// return true if header is within its light client trusted period; otherwise returns false
func isWithinTrustedPeriod(header Header,
trustingPeriod Duration,
now Time) bool {
return header.Time + trustedPeriod > now
}
Note that in case VerifySingle returns without an error (untrusted header
is successfully verified) then we have a guarantee that the transition of the trust
from trustedState to newTrustedState happened during the trusted period of
trustedState.SignedHeader.Header.
TODO: Explain what happens in case VerifySingle returns with an error.
verifySingle. The function verifySingle verifies a single untrusted header
against a given trusted state. It includes all validations and signature verification.
It is not publicly exposed since it does not check for header expiry (time constraints)
and hence it's possible to use it incorrectly.
func verifySingle(trustedState TrustedState,
untrustedSh SignedHeader,
untrustedVs ValidatorSet,
untrustedNextVs ValidatorSet,
trustThreshold float) error {
untrustedHeader = untrustedSh.Header
untrustedCommit = untrustedSh.Commit
trustedHeader = trustedState.SignedHeader.Header
trustedVs = trustedState.ValidatorSet
if trustedHeader.Height >= untrustedHeader.Height return ErrNonIncreasingHeight
if trustedHeader.Time >= untrustedHeader.Time return ErrNonIncreasingTime
// validate the untrusted header against its commit, vals, and next_vals
error = validateSignedHeaderAndVals(untrustedSh, untrustedVs, untrustedNextVs)
if error != nil return error
// check for adjacent headers
if untrustedHeader.Height == trustedHeader.Height + 1 {
if trustedHeader.NextValidatorsHash != untrustedHeader.ValidatorsHash {
return ErrInvalidAdjacentHeaders
}
} else {
error = verifyCommitTrusting(trustedVs, untrustedCommit, untrustedVs, trustThreshold)
if error != nil return error
}
// verify the untrusted commit
return verifyCommitFull(untrustedVs, untrustedCommit)
}
// returns nil if header and validator sets are consistent; otherwise returns error
func validateSignedHeaderAndVals(signedHeader SignedHeader, vs ValidatorSet, nextVs ValidatorSet) error {
header = signedHeader.Header
if hash(vs) != header.ValidatorsHash return ErrInvalidValidatorSet
if hash(nextVs) != header.NextValidatorsHash return ErrInvalidNextValidatorSet
if !matchingCommit(header, signedHeader.Commit) return ErrInvalidCommitValue
return nil
}
// returns nil if at least single correst signer signed the commit; otherwise returns error
func verifyCommitTrusting(trustedVs ValidatorSet,
commit Commit,
untrustedVs ValidatorSet,
trustLevel float) error {
totalPower := trustedVs.TotalVotingPower
signedPower := votingPowerIn(signers(commit, untrustedVs), trustedVs)
// check that the signers account for more than max(1/3, trustLevel) of the voting power
// this ensures that there is at least single correct validator in the set of signers
if signedPower < max(1/3, trustLevel) * totalPower return ErrInsufficientVotingPower
return nil
}
// returns nil if commit is signed by more than 2/3 of voting power of the given validator set
// return error otherwise
func verifyCommitFull(vs ValidatorSet, commit Commit) error {
totalPower := vs.TotalVotingPower;
signedPower := votingPowerIn(signers(commit, vs), vs)
// check the signers account for +2/3 of the voting power
if signedPower * 3 <= totalPower * 2 return ErrInvalidCommit
return nil
}
VerifyHeaderAtHeight. The function VerifyHeaderAtHeight captures high level
logic, i.e., application call to the light client module to download and verify header
for some height.
func VerifyHeaderAtHeight(untrustedHeight int64,
trustedState TrustedState,
trustThreshold float,
trustingPeriod Duration,
clockDrift Duration) (TrustedState, error)) {
trustedHeader := trustedState.SignedHeader.Header
now := System.Time()
if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
return (trustedState, ErrHeaderNotWithinTrustedPeriod)
}
newTrustedState, err := VerifyBisection(untrustedHeight,
trustedState,
trustThreshold,
trustingPeriod,
clockDrift,
now)
if err != nil return (trustedState, err)
now = System.Time()
if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
return (trustedState, ErrHeaderNotWithinTrustedPeriod)
}
return (newTrustedState, err)
}
Note that in case VerifyHeaderAtHeight returns without an error (untrusted header
is successfully verified) then we have a guarantee that the transition of the trust
from trustedState to newTrustedState happened during the trusted period of
trustedState.SignedHeader.Header.
In case VerifyHeaderAtHeight returns with an error, then either (i) the full node we are talking to is faulty
or (ii) the trusted header has expired (it is outside its trusted period). In case (i) the full node is faulty so
light client should disconnect and reinitialise with new peer. In the case (ii) as the trusted header has expired,
we need to reinitialise light client with a new trusted header (that is within its trusted period),
but we don't necessarily need to disconnect from the full node we are talking to (as we haven't observed full node misbehavior in this case).
VerifyBisection. The function VerifyBisection implements
recursive logic for checking if it is possible building trust
relationship between trustedState and untrusted header at the given height over
finite set of (downloaded and verified) headers.
func VerifyBisection(untrustedHeight int64,
trustedState TrustedState,
trustThreshold float,
trustingPeriod Duration,
clockDrift Duration,
now Time) (TrustedState, error) {
untrustedSh, error := Commit(untrustedHeight)
if error != nil return (trustedState, ErrRequestFailed)
untrustedHeader = untrustedSh.Header
// note that we pass now during the recursive calls. This is fine as
// all other untrusted headers we download during recursion will be
// for a smaller heights, and therefore should happen before.
if untrustedHeader.Time > now + clockDrift {
return (trustedState, ErrInvalidHeaderTime)
}
untrustedVs, error := Validators(untrustedHeight)
if error != nil return (trustedState, ErrRequestFailed)
untrustedNextVs, error := Validators(untrustedHeight + 1)
if error != nil return (trustedState, ErrRequestFailed)
error = verifySingle(
trustedState,
untrustedSh,
untrustedVs,
untrustedNextVs,
trustThreshold)
if fatalError(error) return (trustedState, error)
if error == nil {
// the untrusted header is now trusted.
newTrustedState = TrustedState(untrustedSh, untrustedNextVs)
return (newTrustedState, nil)
}
// at this point in time we need to do bisection
pivotHeight := ceil((trustedHeader.Height + untrustedHeight) / 2)
error, newTrustedState = VerifyBisection(pivotHeight,
trustedState,
trustThreshold,
trustingPeriod,
clockDrift,
now)
if error != nil return (newTrustedState, error)
return VerifyBisection(untrustedHeight,
newTrustedState,
trustThreshold,
trustingPeriod,
clockDrift,
now)
}
func fatalError(err) bool {
return err == ErrHeaderNotWithinTrustedPeriod OR
err == ErrInvalidAdjacentHeaders OR
err == ErrNonIncreasingHeight OR
err == ErrNonIncreasingTime OR
err == ErrInvalidValidatorSet OR
err == ErrInvalidNextValidatorSet OR
err == ErrInvalidCommitValue OR
err == ErrInvalidCommit
}
The case untrustedHeader.Height < trustedHeader.Height
In the use case where someone tells the light client that application data that is relevant for it
can be read in the block of height k and the light client trusts a more recent header, we can use the
hashes to verify headers "down the chain." That is, we iterate down the heights and check the hashes in each step.
Remark. For the case were the light client trusts two headers i and j with i < k < j, we should
discuss/experiment whether the forward or the backward method is more effective.
func VerifyHeaderBackwards(trustedHeader Header,
untrustedHeader Header,
trustingPeriod Duration,
clockDrift Duration) error {
if untrustedHeader.Height >= trustedHeader.Height return ErrErrNonDecreasingHeight
if untrustedHeader.Time >= trustedHeader.Time return ErrNonDecreasingTime
now := System.Time()
if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
return ErrHeaderNotWithinTrustedPeriod
}
old := trustedHeader
for i := trustedHeader.Height - 1; i > untrustedHeader.Height; i-- {
untrustedSh, error := Commit(i)
if error != nil return ErrRequestFailed
if (hash(untrustedSh.Header) != old.LastBlockID.Hash) {
return ErrInvalidAdjacentHeaders
}
old := untrustedSh.Header
}
if hash(untrustedHeader) != old.LastBlockID.Hash {
return ErrInvalidAdjacentHeaders
}
now := System.Time()
if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) {
return ErrHeaderNotWithinTrustedPeriod
}
return nil
}
Assumption: In the following, we assume that untrusted_h.Header.height > trusted_h.Header.height. We will quickly discuss the other case in the next section.
We consider the following set-up:
- the light client communicates with one full node
- the light client locally stores all the headers that has passed basic verification and that are within light client trust period. In the pseudo code below we write Store.Add(header) for this. If a header failed to verify, then the full node we are talking to is faulty and we should disconnect from it and reinitialise with new peer.
- If
CanTrustreturns error, then the light client has seen a forged header or the trusted header has expired (it is outside its trusted period).- In case of forged header, the full node is faulty so light client should disconnect and reinitialise with new peer. If the trusted header has expired, we need to reinitialise light client with new trusted header (that is within its trusted period), but we don't necessarily need to disconnect from the full node we are talking to (as we haven't observed full node misbehavior in this case).
Correctness of the Light Client Protocols
Definitions
TRUSTED_PERIOD: trusted period- for realtime
t, the predicatecorrect(v,t)is true if the validatorvfollows the protocol until timet(we will see about recovery later). - Validator fields. We will write a validator as a tuple
(v,p)such thatvis the identifier (i.e., validator address; we assume identifiers are unique in each validator set)pis its voting power
- For each header
h, we writetrust(h) = trueif the light client trustsh.
Failure Model
If a block b with a header h is generated at time Time (i.e. h.Time = Time), then a set of validators that
hold more than 2/3 of the voting power in validators(h.NextValidatorsHash) is correct until time
h.Time + TRUSTED_PERIOD.
Formally, [ \sum_{(v,p) \in validators(h.NextValidatorsHash) \wedge correct(v,h.Time + TRUSTED_PERIOD)} p > 2/3 \sum_{(v,p) \in validators(h.NextValidatorsHash)} p ]
The light client communicates with a full node and learns new headers. The goal is to locally decide whether to trust a header. Our implementation needs to ensure the following two properties:
-
Light Client Completeness: If a header
hwas correctly generated by an instance of Tendermint consensus (and its age is less than the trusted period), then the light client should eventually settrust(h)totrue. -
Light Client Accuracy: If a header
hwas not generated by an instance of Tendermint consensus, then the light client should never settrust(h)to true.
Remark: If in the course of the computation, the light client obtains certainty that some headers were forged by adversaries (that is were not generated by an instance of Tendermint consensus), it may submit (a subset of) the headers it has seen as evidence of misbehavior.
Remark: In Completeness we use "eventually", while in practice trust(h) should be set to true before h.Time + TRUSTED_PERIOD. If not, the header
cannot be trusted because it is too old.
Remark: If a header h is marked with trust(h), but it is too old at some point in time we denote with now (h.Time + TRUSTED_PERIOD < now),
then the light client should set trust(h) to false again at time now.
Assumption: Initially, the light client has a header inithead that it trusts, that is, inithead was correctly generated by the Tendermint consensus.
To reason about the correctness, we may prove the following invariant.
Verification Condition: light Client Invariant.
For each light client l and each header h:
if l has set trust(h) = true,
then validators that are correct until time h.Time + TRUSTED_PERIOD have more than two thirds of the voting power in validators(h.NextValidatorsHash).
Formally, [ \sum_{(v,p) \in validators(h.NextValidatorsHash) \wedge correct(v,h.Time + TRUSTED_PERIOD)} p > 2/3 \sum_{(v,p) \in validators(h.NextValidatorsHash)} p ]
Remark. To prove the invariant, we will have to prove that the light client only trusts headers that were correctly generated by Tendermint consensus. Then the formula above follows from the failure model.
Details
Observation 1. If h.Time + TRUSTED_PERIOD > now, we trust the validator set validators(h.NextValidatorsHash).
When we say we trust validators(h.NextValidatorsHash) we do not trust that each individual validator in validators(h.NextValidatorsHash)
is correct, but we only trust the fact that less than 1/3 of them are faulty (more precisely, the faulty ones have less than 1/3 of the total voting power).
VerifySingle correctness arguments
Light Client Accuracy:
- Assume by contradiction that
untrustedHeaderwas not generated correctly and the light client sets trust to true becauseverifySinglereturns without error. trustedStateis trusted and sufficiently new- by the Failure Model, less than
1/3of the voting power held by faulty validators => at least one correct validatorvhas signeduntrustedHeader. - as
vis correct up to now, it followed the Tendermint consensus protocol at least up to signinguntrustedHeader=>untrustedHeaderwas correctly generated. We arrive at the required contradiction.
Light Client Completeness:
- The check is successful if sufficiently many validators of
trustedStateare still validators in the heightuntrustedHeader.Heightand signeduntrustedHeader. - If
untrustedHeader.Height = trustedHeader.Height + 1, and both headers were generated correctly, the test passes.
Verification Condition: We may need a Tendermint invariant stating that if untrustedSignedHeader.Header.Height = trustedHeader.Height + 1 then
signers(untrustedSignedHeader.Commit) \subseteq validators(trustedHeader.NextValidatorsHash).
Remark: The variable trustThreshold can be used if the user believes that relying on one correct validator is not sufficient.
However, in case of (frequent) changes in the validator set, the higher the trustThreshold is chosen, the more unlikely it becomes that
verifySingle returns with an error for non-adjacent headers.
VerifyBisectioncorrectness arguments (sketch)*
Light Client Accuracy:
- Assume by contradiction that the header at
untrustedHeightobtained from the full node was not generated correctly and the light client sets trust to true becauseVerifyBisectionreturns without an error. VerifyBisectionreturns without error only if all calls toverifySinglein the recursion return without error (returnnil).- Thus we have a sequence of headers that all satisfied the
verifySingle - again a contradiction
light Client Completeness:
This is only ensured if upon Commit(pivot) the light client is always provided with a correctly generated header.
Stalling
With VerifyBisection, a faulty full node could stall a light client by creating a long sequence of headers that are queried one-by-one by the light client and look OK,
before the light client eventually detects a problem. There are several ways to address this:
- Each call to
Commitcould be issued to a different full node - Instead of querying header by header, the light client tells a full node which header it trusts, and the height of the header it needs. The full node responds with
the header along with a proof consisting of intermediate headers that the light client can use to verify. Roughly,
VerifyBisectionwould then be executed at the full node. - We may set a timeout how long
VerifyBisectionmay take.
Light Client Verification
The light client implements a read operation of a header from the blockchain, by communicating with full nodes. As some full nodes may be faulty, this functionality must be implemented in a fault-tolerant way.
In the Tendermint blockchain, the validator set may change with every new block. The staking and unbonding mechanism induces a security model: starting at time Time of the header, more than two-thirds of the next validators of a new block are correct for the duration of TrustedPeriod. The fault-tolerant read operation is designed for this security model.
The challenge addressed here is that the light client might have a block of height h1 and needs to read the block of height h2 greater than h1. Checking all headers of heights from h1 to h2 might be too costly (e.g., in terms of energy for mobile devices). This specification tries to reduce the number of intermediate blocks that need to be checked, by exploiting the guarantees provided by the security model.
Status
This document is thoroughly reviewed, and the protocol has been formalized in TLA+ and model checked.
Issues that need to be addressed
As it is part of the larger light node, its data structures and functions interact with the fork dectection functionality of the light client. As a result of the work on Pull Request 479 we established the need for an update in the data structures in Issue 499. This will not change the verification logic, but it will record information about verification that can be used in fork detection (in particular in computing more efficiently the proof of fork).
Outline
-
Part I: Introduction of relevant terms of the Tendermint blockchain.
-
Part II: Introduction of the problem addressed by the Lightclient Verification protocol.
- Verification Informal Problem statement: For the general audience, that is, engineers who want to get an overview over what the component is doing from a bird's eye view.
- Sequential Problem statement: Provides a mathematical definition of the problem statement in its sequential form, that is, ignoring the distributed aspect of the implementation of the blockchain.
-
Part III: Distributed aspects of the light client, system assumptions and temporal logic specifications.
-
Incentives: how faulty full nodes may benefit from misbehaving and how correct full nodes benefit from cooperating.
-
Computational Model: timing and correctness assumptions.
-
Distributed Problem Statement: temporal properties that formalize safety and liveness properties in the distributed setting.
-
-
Part IV: Specification of the protocols.
-
Definitions: Describes inputs, outputs, variables used by the protocol, auxiliary functions
-
Core Verification: gives an outline of the solution, and details of the functions used (with preconditions, postconditions, error conditions).
-
Liveness Scenarios: when the light client makes progress depends heavily on the changes in the validator sets of the blockchain. We discuss some typical scenarios.
-
-
Part V: The above parts focus on a common case where the last verified block has height h1 and the requested height h2 satisfies h2 > h1. For IBC, there are scenarios where this might not be the case. In this part, we provide some preliminaries for supporting this. As not all details of the IBC requirements are clear by now, we do not provide a complete specification at this point. We mark with "Open Question" points that need to be addressed in order to finalize this specification. It should be noted that the technically most challenging case is the one specified in Part IV.
In this document we quite extensively use tags in order to be able to reference assumptions, invariants, etc. in future communication. In these tags we frequently use the following short forms:
- TMBC: Tendermint blockchain
- SEQ: for sequential specifications
- LCV: Lightclient Verification
- LIVE: liveness
- SAFE: safety
- FUNC: function
- INV: invariant
- A: assumption
Part I - Tendermint Blockchain
Header Fields necessary for the Light Client
[TMBC-HEADER.1]
A set of blockchain transactions is stored in a data structure called block, which contains a field called header. (The data structure block is defined here). As the header contains hashes to the relevant fields of the block, for the purpose of this specification, we will assume that the blockchain is a list of headers, rather than a list of blocks.
[TMBC-HASH-UNIQUENESS.1]
We assume that every hash in the header identifies the data it hashes. Therefore, in this specification, we do not distinguish between hashes and the data they represent.
[TMBC-HEADER-FIELDS.1]
A header contains the following fields:
Height: non-negative integerTime: time (integer)LastBlockID: HashvalueLastCommitDomainCommitValidators: DomainValNextValidators: DomainValData: DomainTXAppState: DomainAppLastResults: DomainRes
[TMBC-SEQ.1]
The Tendermint blockchain is a list chain of headers.
[TMBC-VALIDATOR-PAIR.1]
Given a full node, a validator pair is a pair (peerID, voting_power), where
- peerID is the PeerID (public key) of a full node,
- voting_power is an integer (representing the full node's voting power in a certain consensus instance).
In the Golang implementation the data type for validator pair is called
Validator
[TMBC-VALIDATOR-SET.1]
A validator set is a set of validator pairs. For a validator set vs, we write TotalVotingPower(vs) for the sum of the voting powers of its validator pairs.
[TMBC-VOTE.1]
A vote contains a prevote or precommit message sent and signed by
a validator node during the execution of consensus. Each
message contains the following fields
Type: prevote or precommitHeight: positive integerRounda positive integerBlockIDa Hashvalue of a block (not necessarily a block of the chain)
[TMBC-COMMIT.1]
A commit is a set of precommit message.
Tendermint Failure Model
[TMBC-AUTH-BYZ.1]
We assume the authenticated Byzantine fault model in which no node (faulty or correct) may break digital signatures, but otherwise, no additional assumption is made about the internal behavior of faulty nodes. That is, faulty nodes are only limited in that they cannot forge messages.
[TMBC-TIME-PARAMS.1]
A Tendermint blockchain has the following configuration parameters:
- unbondingPeriod: a time duration.
- trustingPeriod: a time duration smaller than unbondingPeriod.
[TMBC-CORRECT.1]
We define a predicate correctUntil(n, t), where n is a node and t is a time point. The predicate correctUntil(n, t) is true if and only if the node n follows all the protocols (at least) until time t.
[TMBC-FM-2THIRDS.1]
If a block h is in the chain, then there exists a subset CorrV of h.NextValidators, such that:
- TotalVotingPower(CorrV) > 2/3 TotalVotingPower(h.NextValidators); cf. [TMBC-VALIDATOR-SET.1]
- For every validator pair (n,p) in CorrV, it holds correctUntil(n, h.Time + trustingPeriod); cf. [TMBC-CORRECT.1]
The definition of correct [[TMBC-CORRECT.1]][TMBC-CORRECT-link] refers to realtime, while it is used here with Time and trustingPeriod, which are "hardware times". We do not make a distinction here.
[TMBC-CORR-FULL.1]
Every correct full node locally stores a prefix of the current list of headers from [TMBC-SEQ.1].
What the Light Client Checks
From [TMBC-FM-2THIRDS.1] we directly derive the following observation:
[TMBC-VAL-CONTAINS-CORR.1]
Given a (trusted) block tb of the blockchain, a given set of full nodes N contains a correct node at a real-time t, if
- t - trustingPeriod < tb.Time < t
- the voting power in tb.NextValidators of nodes in N is more than 1/3 of TotalVotingPower(tb.NextValidators)
The following describes how a commit for a given block b must look like.
[TMBC-SOUND-DISTR-POSS-COMMIT.1]
For a block b, each element pc of PossibleCommit(b) satisfies:
- pc contains only votes (cf. [TMBC-VOTE.1]) by validators from b.Validators
- the sum of the voting powers in pc is greater than 2/3 TotalVotingPower(b.Validators)
- and there is an r such that each vote v in pc satisfies
- v.Type = precommit
- v.Height = b.Height
- v.Round = r
- v.blockID = hash(b)
The following property comes from the validity of the consensus: A correct validator node only sends
prevoteorprecommit, ifBlockIDof the new (to-be-decided) block is equal to the hash of the last block.
[TMBC-VAL-COMMIT.1]
If for a block b, a commit c
- contains at least one validator pair (v,p) such that v is a correct validator node, and
- is contained in PossibleCommit(b)
then the block b is on the blockchain.
Context of this document
In this document we specify the light client verification component, called Core Verification. The Core Verification communicates with a full node. As full nodes may be faulty, it cannot trust the received information, but the light client has to check whether the header it receives coincides with the one generated by Tendermint consensus.
The two properties [TMBC-VAL-CONTAINS-CORR.1] and [TMBC-VAL-COMMIT] formalize the checks done by this specification: Given a trusted block tb and an untrusted block ub with a commit cub, one has to check that cub is in PossibleCommit(ub), and that cub contains a correct node using tb.
Part II - Sequential Definition of the Verification Problem
Verification Informal Problem statement
Given a height targetHeight as an input, the Verifier eventually stores a header h of height targetHeight locally. This header h is generated by the Tendermint blockchain. In particular, a header that was not generated by the blockchain should never be stored.
Sequential Problem statement
[LCV-SEQ-LIVE.1]
The Verifier gets as input a height targetHeight, and eventually stores the header of height targetHeight of the blockchain.
[LCV-SEQ-SAFE.1]
The Verifier never stores a header which is not in the blockchain.
Part III - Light Client as Distributed System
Incentives
Faulty full nodes may benefit from lying to the light client, by making the light client accept a block that deviates (e.g., contains additional transactions) from the one generated by Tendermint consensus. Users using the light client might be harmed by accepting a forged header.
The fork detector of the light client may help the correct full nodes to understand whether their header is a good one. Hence, in combination with the light client detector, the correct full nodes have the incentive to respond. We can thus base liveness arguments on the assumption that correct full nodes reliably talk to the light client.
Computational Model
[LCV-A-PEER.1]
The verifier communicates with a full node called primary. No assumption is made about the full node (it may be correct or faulty).
[LCV-A-COMM.1]
Communication between the light client and a correct full node is reliable and bounded in time. Reliable communication means that messages are not lost, not duplicated, and eventually delivered. There is a (known) end-to-end delay Delta, such that if a message is sent at time t then it is received and processes by time t + Delta. This implies that we need a timeout of at least 2 Delta for remote procedure calls to ensure that the response of a correct peer arrives before the timeout expires.
[LCV-A-TFM.1]
The Tendermint blockchain satisfies the Tendermint failure model [TMBC-FM-2THIRDS.1].
[LCV-A-VAL.1]
The system satisfies [TMBC-AUTH-BYZ.1] and [TMBC-FM-2THIRDS.1]. Thus, there is a blockchain that satisfies the soundness requirements (that is, the validation rules in [block]).
Distributed Problem Statement
Two Kinds of Termination
We do not assume that primary is correct. Under this assumption no protocol can guarantee the combination of the sequential properties. Thus, in the (unreliable) distributed setting, we consider two kinds of termination (successful and failure) and we will specify below under what (favorable) conditions Core Verification ensures to terminate successfully, and satisfy the requirements of the sequential problem statement:
[LCV-DIST-TERM.1]
Core Verification either terminates successfully or it terminates with failure.
Design choices
[LCV-DIST-STORE.1]
Core Verification has a local data structure called LightStore that contains light blocks (that contain a header). For each light block we record whether it is verified.
[LCV-DIST-PRIMARY.1]
Core Verification has a local variable primary that contains the PeerID of a full node.
[LCV-DIST-INIT.1]
LightStore is initialized with a header trustedHeader that was correctly generated by the Tendermint consensus. We say trustedHeader is verified.
Temporal Properties
[LCV-DIST-SAFE.1]
It is always the case that every verified header in LightStore was generated by an instance of Tendermint consensus.
[LCV-DIST-LIVE.1]
From time to time, a new instance of Core Verification is called with a height targetHeight greater than the height of any header in LightStore. Each instance must eventually terminate.
- If
- the primary is correct (and locally has the block of targetHeight), and
- LightStore always contains a verified header whose age is less than the
trusting period,
then Core Verification adds a verified header hd with height targetHeight to LightStore and it terminates successfully
These definitions imply that if the primary is faulty, a header may or may not be added to LightStore. In any case, [LCV-DIST-SAFE.1] must hold. The invariant [LCV-DIST-SAFE.1] and the liveness requirement [LCV-DIST-LIVE.1] allow that verified headers are added to LightStore whose height was not passed to the verifier (e.g., intermediate headers used in bisection; see below). Note that for liveness, initially having a trustedHeader within the trustinPeriod is not sufficient. However, as this specification will leave some freedom with respect to the strategy in which order to download intermediate headers, we do not give a more precise liveness specification here. After giving the specification of the protocol, we will discuss some liveness scenarios below.
Solving the sequential specification
This specification provides a partial solution to the sequential specification. The Verifier solves the invariant of the sequential part
[LCV-DIST-SAFE.1] => [LCV-SEQ-SAFE.1]
In the case the primary is correct, and there is a recent header in LightStore, the verifier satisfies the liveness requirements.
⋀ primary is correct
⋀ always ∃ verified header in LightStore. header.Time > now - trustingPeriod
⋀ [LCV-A-Comm.1] ⋀ (
( [TMBC-CorrFull.1] ⋀
[LCV-DIST-LIVE.1] )
⟹ [LCV-SEQ-LIVE.1]
)
Part IV - Light Client Verification Protocol
We provide a specification for Light Client Verification. The local
code for verification is presented by a sequential function
VerifyToTarget to highlight the control flow of this functionality.
We note that if a different concurrency model is considered for
an implementation, the sequential flow of the function may be
implemented with mutexes, etc. However, the light client verification
is partitioned into three blocks that can be implemented and tested
independently:
FetchLightBlockis called to download a light block (header) of a given height from a peer.ValidAndVerifiedis a local code that checks the header.Scheduledecides which height to try to verify next. We keep this underspecified as different implementations (currently in Goland and Rust) may implement different optimizations here. We just provide necessary conditions on how the height may evolve.
Definitions
Data Types
The core data structure of the protocol is the LightBlock.
[LCV-DATA-LIGHTBLOCK.1]
type LightBlock struct {
Header Header
Commit Commit
Validators ValidatorSet
}
[LCV-DATA-LIGHTSTORE.1]
LightBlocks are stored in a structure which stores all LightBlock from initialization or received from peers.
type LightStore struct {
...
}
Each LightBlock is in one of the following states:
type VerifiedState int
const (
StateUnverified = iota + 1
StateVerified
StateFailed
StateTrusted
)
Only the detector module sets a lightBlock state to
StateTrustedand only if it wasStateVerifiedbefore.
The LightStore exposes the following functions to query stored LightBlocks.
[LCV-FUNC-GET.1]
func (ls LightStore) Get(height Height) (LightBlock, bool)
- Expected postcondition
- returns a LightBlock at a given height or false in the second argument if the LightStore does not contain the specified LightBlock.
[LCV-FUNC-LATEST-VERIF.1]
func (ls LightStore) LatestVerified() LightBlock
- Expected postcondition
- returns the highest light block whose state is
StateVerifiedorStateTrusted
- returns the highest light block whose state is
[LCV-FUNC-UPDATE.2]
func (ls LightStore) Update(lightBlock LightBlock,
verfiedState VerifiedState
verifiedBy Height)
- Expected postcondition
- The state of the LightBlock is set to verifiedState.
- verifiedBy of the Lightblock is set to Height
The following function is used only in the detector specification listed here for completeness.
[LCV-FUNC-LATEST-TRUSTED.1]
func (ls LightStore) LatestTrusted() LightBlock
- Expected postcondition
- returns the highest light block that has been verified and checked by the detector.
[LCV-FUNC-FILTER.1]
func (ls LightStore) FilterVerified() LightSTore
- Expected postcondition
- returns only the LightBlocks with state verified.
Inputs
- lightStore: stores light blocks that have been downloaded and that passed verification. Initially it contains a light block with trustedHeader.
- primary: peerID
- targetHeight: the height of the needed header
Configuration Parameters
- trustThreshold: a float. Can be used if correctness should not be based on more voting power and 1/3.
- trustingPeriod: a time duration [TMBC-TIME_PARAMS.1].
- clockDrift: a time duration. Correction parameter dealing with only approximately synchronized clocks.
Variables
- nextHeight: initially targetHeight
nextHeight should be thought of the "height of the next header we need to download and verify"
Assumptions
[LCV-A-INIT.1]
-
trustedHeader is from the blockchain
-
targetHeight > LightStore.LatestVerified.Header.Height
Invariants
[LCV-INV-TP.1]
It is always the case that LightStore.LatestTrusted.Header.Time > now - trustingPeriod.
If the invariant is violated, the light client does not have a header it can trust. A trusted header must be obtained externally, its trust can only be based on social consensus.
Used Remote Functions
We use the functions commit and validators that are provided
by the RPC client for Tendermint.
func Commit(height int64) (SignedHeader, error)
- Implementation remark
- RPC to full node n
- JSON sent:
// POST /commit
{
"jsonrpc": "2.0",
"id": "ccc84631-dfdb-4adc-b88c-5291ea3c2cfb", // UUID v4, unique per request
"method": "commit",
"params": {
"height": 1234
}
}
- Expected precondition
- header of
heightexists on blockchain
- header of
- Expected postcondition
- if n is correct: Returns the signed header of height
heightfrom the blockchain if communication is timely (no timeout) - if n is faulty: Returns a signed header with arbitrary content
- if n is correct: Returns the signed header of height
- Error condition
- if n is correct: precondition violated or timeout
- if n is faulty: arbitrary error
func Validators(height int64) (ValidatorSet, error)
- Implementation remark
- RPC to full node n
- JSON sent:
// POST /validators
{
"jsonrpc": "2.0",
"id": "ccc84631-dfdb-4adc-b88c-5291ea3c2cfb", // UUID v4, unique per request
"method": "validators",
"params": {
"height": 1234
}
}
- Expected precondition
- header of
heightexists on blockchain
- header of
- Expected postcondition
- if n is correct: Returns the validator set of height
heightfrom the blockchain if communication is timely (no timeout) - if n is faulty: Returns arbitrary validator set
- if n is correct: Returns the validator set of height
- Error condition
- if n is correct: precondition violated or timeout
- if n is faulty: arbitrary error
Communicating Function
[LCV-FUNC-FETCH.1]
func FetchLightBlock(peer PeerID, height Height) LightBlock
- Implementation remark
- RPC to peer at PeerID
- calls
Commitfor height andValidatorsfor height and height+1
- Expected precondition
heightis less than or equal to height of the peer [LCV-IO-PRE-HEIGHT.1]
- Expected postcondition:
- if node is correct:
- Returns the LightBlock lb of height
heightthat is consistent with the blockchain - lb.provider = peer [LCV-IO-POST-PROVIDER.1]
- lb.Header is a header consistent with the blockchain
- lb.Validators is the validator set of the blockchain at height nextHeight
- lb.NextValidators is the validator set of the blockchain at height nextHeight + 1
- Returns the LightBlock lb of height
- if node is faulty: Returns a LightBlock with arbitrary content [TMBC-AUTH-BYZ.1]
- if node is correct:
- Error condition
- if n is correct: precondition violated
- if n is faulty: arbitrary error
- if lb.provider != peer
- times out after 2 Delta (by assumption n is faulty)
Core Verification
Outline
The VerifyToTarget is the main function and uses the following functions.
FetchLightBlockis called to download the next light block. It is the only function that communicates with other nodesValidAndVerifiedchecks whether header is valid and checks if a new lightBlock should be trusted based on a previously verified lightBlock.Scheduledecides which height to try to verify next
In the following description of VerifyToTarget we do not deal with error
handling. If any of the above function returns an error, VerifyToTarget just
passes the error on.
[LCV-FUNC-MAIN.1]
func VerifyToTarget(primary PeerID, lightStore LightStore,
targetHeight Height) (LightStore, Result) {
nextHeight := targetHeight
for lightStore.LatestVerified.height < targetHeight {
// Get next LightBlock for verification
current, found := lightStore.Get(nextHeight)
if !found {
current = FetchLightBlock(primary, nextHeight)
lightStore.Update(current, StateUnverified)
}
// Verify
verdict = ValidAndVerified(lightStore.LatestVerified, current)
// Decide whether/how to continue
if verdict == SUCCESS {
lightStore.Update(current, StateVerified)
}
else if verdict == NOT_ENOUGH_TRUST {
// do nothing
// the light block current passed validation, but the validator
// set is too different to verify it. We keep the state of
// current at StateUnverified. For a later iteration, Schedule
// might decide to try verification of that light block again.
}
else {
// verdict is some error code
lightStore.Update(current, StateFailed)
// possibly remove all LightBlocks from primary
return (lightStore, ResultFailure)
}
nextHeight = Schedule(lightStore, nextHeight, targetHeight)
}
return (lightStore, ResultSuccess)
}
- Expected precondition
- lightStore contains a LightBlock within the trustingPeriod [LCV-PRE-TP.1]
- targetHeight is greater than the height of all the LightBlocks in lightStore
- Expected postcondition:
- returns lightStore that contains a LightBlock that corresponds to a block of the blockchain of height targetHeight (that is, the LightBlock has been added to lightStore) [LCV-POST-LS.1]
- Error conditions
- if the precondition is violated
- if
ValidAndVerifiedorFetchLightBlockreport an error - if [LCV-INV-TP.1] is violated
Details of the Functions
[LCV-FUNC-VALID.1]
func ValidAndVerified(trusted LightBlock, untrusted LightBlock) Result
- Expected precondition:
- untrusted is valid, that is, satisfies the soundness checks
- untrusted is well-formed, that is,
- untrusted.Header.Time < now + clockDrift
- untrusted.Validators = hash(untrusted.Header.Validators)
- untrusted.NextValidators = hash(untrusted.Header.NextValidators)
- trusted.Header.Time > now - trustingPeriod
- trusted.Commit is a commit for the header trusted.Header, i.e., it contains the correct hash of the header, and +2/3 of signatures
- the
HeightandTimeoftrustedare smaller than the Height andTimeofuntrusted, respectively - the untrusted.Header is well-formed (passes the tests from
[block]), and in particular
- if the untrusted header
unstrusted.Headeris the immediate successor oftrusted.Header, then it holds that- trusted.Header.NextValidators = untrusted.Header.Validators, and moreover,
- untrusted.Header.Commit
- contains signatures by more than two-thirds of the validators
- contains no signature from nodes that are not in trusted.Header.NextValidators
- if the untrusted header
- Expected postcondition:
- Returns
SUCCESS:- if untrusted is the immediate successor of trusted, or otherwise,
- if the signatures of a set of validators that have more than max(1/3,trustThreshold) of voting power in trusted.Nextvalidators is contained in untrusted.Commit (that is, header passes the tests [TMBC-VAL-CONTAINS-CORR.1] and [TMBC-VAL-COMMIT.1])
- Returns
NOT_ENOUGH_TRUSTif:- untrusted is not the immediate successor of trusted and the max(1/3,trustThreshold) threshold is not reached (that is, if [TMBC-VAL-CONTAINS-CORR.1] fails and header is does not violate the soundness checks [block]).
- Returns
- Error condition:
- if precondition violated
[LCV-FUNC-SCHEDULE.1]
func Schedule(lightStore, nextHeight, targetHeight) Height
- Implementation remark: If picks the next height to be verified. We keep the precise choice of the next header under-specified. It is subject to performance optimizations that do not influence the correctness
- Expected postcondition: [LCV-SCHEDULE-POST.1]
Return H s.t.
- if lightStore.LatestVerified.Height = nextHeight and
lightStore.LatestVerified < targetHeight then
nextHeight < H <= targetHeight - if lightStore.LatestVerified.Height < nextHeight and
lightStore.LatestVerified.Height < targetHeight then
lightStore.LatestVerified.Height < H < nextHeight - if lightStore.LatestVerified.Height = targetHeight then
H = targetHeight
- if lightStore.LatestVerified.Height = nextHeight and
lightStore.LatestVerified < targetHeight then
Case i. captures the case where the light block at height nextHeight has been verified, and we can choose a height closer to the targetHeight. As we get the lightStore as parameter, the choice of the next height can depend on the lightStore, e.g., we can pick a height for which we have already downloaded a light block. In Case ii. the header of nextHeight could not be verified, and we need to pick a smaller height. In Case iii. is a special case when we have verified the targetHeight.
Solving the distributed specification
trustedStore is implemented by the light blocks in lightStore that have the state StateVerified.
Argument for [LCV-DIST-SAFE.1]
ValidAndVerifiedimplements the soundness checks and the checks [TMBC-VAL-CONTAINS-CORR.1] and [TMBC-VAL-COMMIT.1] under the assumption [TMBC-FM-2THIRDS.1]- Only if
ValidAndVerifiedreturns withSUCCESS, the state of a light block is set to StateVerified.
Argument for [LCV-DIST-LIVE.1]
- If primary is correct,
FetchLightBlockwill always return a light block consistent with the blockchainValidAndVerifiedeither verifies the header using the trusting period or falls back to sequential verification- If [LCV-INV-TP.1] holds, eventually every header will be verified and core verification terminates successfully.
- successful termination depends on the age of lightStore.LatestVerified (for instance, initially on the age of trustedHeader) and the changes of the validator sets on the blockchain. We will give some examples below.
- If primary is faulty,
- it either provides headers that pass all the tests, and we return with the header
- it provides one header that fails a test, core verification terminates with failure.
- it times out and core verification terminates with failure.
Liveness Scenarios
The liveness argument above assumes [LCV-INV-TP.1]
which requires that there is a header that does not expire before the target height is reached. Here we discuss scenarios to ensure this.
Let startHeader be LightStore.LatestVerified when core verification is called (trustedHeader) and startTime be the time core verification is invoked.
In order to ensure liveness, LightStore always needs to contain a verified (or initially trusted) header whose time is within the trusting period. To ensure this, core verification needs to add new headers to LightStore and verify them, before all headers in LightStore expire.
Many changes in validator set
Let's consider Schedule implements
bisection, that is, it halves the distance.
Assume the case where the validator set changes completely in each
block. Then the
method in this specification needs to
sequentially verify all headers. That is, for
- W = log_2 (targetHeight - startHeader.Height),
W headers need to be downloaded and checked before the header of height startHeader.Height + 1 is added to LightStore.
-
Let Comp be the local computation time needed to check headers and signatures for one header.
-
Then we need in the worst case Comp + 2 Delta to download and check one header.
-
Then the first time a verified header could be added to LightStore is startTime + W * (Comp + 2 Delta)
-
[TP.1] However, it can only be added if we still have a header in LightStore, which is not expired, that is only the case if
- startHeader.Time > startTime + WCG * (Comp + 2 Delta) - trustingPeriod,
- that is, if core verification is started at
startTime < startHeader.Time + trustingPeriod - WCG * (Comp + 2 Delta)
-
one may then do an inductive argument from this point on, depending on the implementation of
Schedule. We may have to account for the headers that are already downloaded, but they are checked against the new LightStore.LatestVerified.
We observe that the worst case time it needs to verify the header of height targetHeight depends mainly on how frequent the validator set on the blockchain changes. That core verification terminates successfully crucially depends on the check [TP.1], that is, that the headers in LightStore do not expire in the time needed to download more headers, which depends on the creation time of the headers in LightStore. That is, termination of core verification is highly depending on the data stored in the blockchain. The current light client core verification protocol exploits that, in practice, changes in the validator set are rare. For instance, consider the following scenario.
No change in validator set
If on the blockchain the validator set of the block at height targetHeight is equal to startHeader.NextValidators:
- there is one round trip in
FetchLightBlockto download the light block of height targetHeight, and Comp to check it. - as the validator sets are equal,
VerifyreturnsSUCCESS, if startHeader.Time > now - trustingPeriod. - that is, if startTime < startHeader.Header.Time + trustingPeriod - 2 Delta - Comp, then core verification terminates successfully
Part V - Supporting the IBC Relayer
The above specification focuses on the most common case, which also
constitutes the most challenging task: using the Tendermint security
model to verify light blocks without
downloading all intermediate blocks. To focus on this challenge, above
we have restricted ourselves to the case where targetHeight is
greater than the height of any trusted header. This simplified
presentation of the algorithm as initially
lightStore.LatestVerified() is less than targetHeight, and in the
process of verification lightStore.LatestVerified() increases until
targetHeight is reached.
For IBC it might be that some "older" header is needed, that is, targetHeight < lightStore.LatestVerified(). In this section we present a preliminary design, and we mark some remaining open questions. If targetHeight < lightStore.LatestVerified() our design separates the following cases:
- A previous instance of
VerifyToTargethas already downloaded the light block of targetHeight. There are two cases- the light block has been verified
- the light block has not been verified yet
- No light block of targetHeight had been downloaded before. There
are two cases:
- there exists a verified light block of height less than targetHeight
- otherwise. In this case we need to do "backwards verification"
using the hash of the previous block in the
LastBlockIDfield of a header.
Open Question: what are the security assumptions for backward verification. Should we check that the light block we verify from (and/or the checked light block) is within the trusting period?
The design just presents the above case distinction as a function, and defines some auxiliary functions in the same way the protocol was presented in Part IV.
func (ls LightStore) LatestPrevious(height Height) (LightBlock, bool)
- Expected postcondition
- returns a light block lb that satisfies:
- lb is in lightStore
- lb is verified and not expired
- lb.Header.Height < height
- for all b in lightStore s.t. b is verified and not expired it holds lb.Header.Height >= b.Header.Height
- false in the second argument if the LightStore does not contain such an lb.
- returns a light block lb that satisfies:
func (ls LightStore) MinVerified() (LightBlock, bool)
- Expected postcondition
- returns a light block lb that satisfies:
- lb is in lightStore
- lb is verified Open Question: replace by trusted?
- lb.Header.Height is minimal in the lightStore
- Open Question: according to this, it might be expired (outside the trusting period). This approach appears safe. Are there reasons we should not do that?
- false in the second argument if the LightStore does not contain such an lb.
- returns a light block lb that satisfies:
If a height that is smaller than the smallest height in the lightstore is required, we check the hashes backwards. This is done with the following function:
[LCV-FUNC-BACKWARDS.1]
func Backwards (primary PeerID, lightStore LightStore, targetHeight Height)
(LightStore, Result) {
lb,res = lightStore.MinVerified()
if res = false {
return (lightStore, ResultFailure)
}
latest := lb.Header
for i := lb.Header.height - 1; i >= targetHeight; i-- {
// here we download height-by-height. We might first download all
// headers down to targetHeight and then check them.
current := FetchLightBlock(primary,i)
if (hash(current) != latest.Header.LastBlockId) {
return (lightStore, ResultFailure)
}
else {
lightStore.Update(current, StateVerified)
// **Open Question:** Do we need a new state type for
// backwards verified light blocks?
}
latest = current
}
return (lightStore, ResultSuccess)
}
The following function just decided based on the required height which method should be used.
[LCV-FUNC-IBCMAIN.1]
func Main (primary PeerID, lightStore LightStore, targetHeight Height)
(LightStore, Result) {
b1, r1 = lightStore.Get(targetHeight)
if r1 = true and b1.State = StateVerified {
// block already there
return (lightStore, ResultSuccess)
}
if targetHeight > lightStore.LatestVerified.height {
// case of Part IV
return VerifyToTarget(primary, lightStore, targetHeight)
}
else {
b2, r2 = lightStore.LatestPrevious(targetHeight);
if r2 = true {
// make auxiliary lightStore auxLS to call VerifyToTarget.
// VerifyToTarget uses LatestVerified of the given lightStore
// For that we need:
// auxLS.LatestVerified = lightStore.LatestPrevious(targetHeight)
auxLS.Init;
auxLS.Update(b2,StateVerified);
if r1 = true {
// we need to verify a previously downloaded light block.
// we add it to the auxiliary store so that VerifyToTarget
// does not download it again
auxLS.Update(b1,b1.State);
}
auxLS, res2 = VerifyToTarget(primary, auxLS, targetHeight)
// move all lightblocks from auxLS to lightStore,
// maintain state
// we do that whether VerifyToTarget was successful or not
for i, s range auxLS {
lighStore.Update(s,s.State)
}
return (lightStore, res2)
}
else {
return Backwards(primary, lightStore, targetHeight)
}
}
}
References
[block] Specification of the block data structure.
[RPC] RPC client for Tendermint
[fork-detector] The specification of the light client fork detector.
[fullnode] Specification of the full node API
[ibc-rs] Rust implementation of IBC modules and relayer.
[lightclient] The light client ADR [77d2651 on Dec 27, 2019].
Light Client Verification
The light client implements a read operation of a header from the blockchain, by communicating with full nodes. As some full nodes may be faulty, this functionality must be implemented in a fault-tolerant way.
In the Tendermint blockchain, the validator set may change with every new block. The staking and unbonding mechanism induces a security model: starting at time Time of the header, more than two-thirds of the next validators of a new block are correct for the duration of TrustedPeriod. The fault-tolerant read operation is designed for this security model.
The challenge addressed here is that the light client might have a block of height h1 and needs to read the block of height h2 greater than h1. Checking all headers of heights from h1 to h2 might be too costly (e.g., in terms of energy for mobile devices). This specification tries to reduce the number of intermediate blocks that need to be checked, by exploiting the guarantees provided by the security model.
Status
Previous Versions
- [001_published] is thoroughly reviewed, and the protocol has been formalized in TLA+ and model checked.
Issues that are addressed in this revision
As it is part of the larger light node, its data structures and functions interact with the attack dectection functionality of the light client. As a result of the work on
-
attack detection for light nodes
-
attack detection for IBC and relayer requirements
-
light client supervisor (also in Rust proposal)
adaptations to the semantics and functions exposed by the LightStore needed to be made. In contrast to version 001 we specify the following:
-
VerifyToTargetandBackwardsare called with a single lightblock as root of trust in contrast to passing the complete lightstore. -
During verification, we record for each lightblock which other lightblock can be used to verify it in one step. This is needed to generate verification traces that are needed for IBC.
Outline
-
Part I: Introduction of relevant terms of the Tendermint blockchain.
-
Part II: Introduction of the problem addressed by the Lightclient Verification protocol.
- Verification Informal Problem statement: For the general audience, that is, engineers who want to get an overview over what the component is doing from a bird's eye view.
- Sequential Problem statement: Provides a mathematical definition of the problem statement in its sequential form, that is, ignoring the distributed aspect of the implementation of the blockchain.
-
Part III: Distributed aspects of the light client, system assumptions and temporal logic specifications.
-
Incentives: how faulty full nodes may benefit from misbehaving and how correct full nodes benefit from cooperating.
-
Computational Model: timing and correctness assumptions.
-
Distributed Problem Statement: temporal properties that formalize safety and liveness properties in the distributed setting.
-
-
Part IV: Specification of the protocols.
-
Definitions: Describes inputs, outputs, variables used by the protocol, auxiliary functions
-
Core Verification: gives an outline of the solution, and details of the functions used (with preconditions, postconditions, error conditions).
-
Liveness Scenarios: when the light client makes progress depends heavily on the changes in the validator sets of the blockchain. We discuss some typical scenarios.
-
-
Part V: The above parts focus on a common case where the last verified block has height h1 and the requested height h2 satisfies h2 > h1. For IBC, there are scenarios where this might not be the case. In this part, we provide some preliminaries for supporting this. As not all details of the IBC requirements are clear by now, we do not provide a complete specification at this point. We mark with "Open Question" points that need to be addressed in order to finalize this specification. It should be noted that the technically most challenging case is the one specified in Part IV.
In this document we quite extensively use tags in order to be able to reference assumptions, invariants, etc. in future communication. In these tags we frequently use the following short forms:
- TMBC: Tendermint blockchain
- SEQ: for sequential specifications
- LCV: Lightclient Verification
- LIVE: liveness
- SAFE: safety
- FUNC: function
- INV: invariant
- A: assumption
Part I - Tendermint Blockchain
Header Fields necessary for the Light Client
[TMBC-HEADER.1]
A set of blockchain transactions is stored in a data structure called block, which contains a field called header. (The data structure block is defined here). As the header contains hashes to the relevant fields of the block, for the purpose of this specification, we will assume that the blockchain is a list of headers, rather than a list of blocks.
[TMBC-HASH-UNIQUENESS.1]
We assume that every hash in the header identifies the data it hashes. Therefore, in this specification, we do not distinguish between hashes and the data they represent.
[TMBC-HEADER-FIELDS.2]
A header contains the following fields:
Height: non-negative integerTime: time (non-negative integer)LastBlockID: HashvalueLastCommitDomainCommitValidators: DomainValNextValidators: DomainValData: DomainTXAppState: DomainAppLastResults: DomainRes
[TMBC-SEQ.1]
The Tendermint blockchain is a list chain of headers.
[TMBC-VALIDATOR-PAIR.1]
Given a full node, a validator pair is a pair (peerID, voting_power), where
- peerID is the PeerID (public key) of a full node,
- voting_power is an integer (representing the full node's voting power in a certain consensus instance).
In the Golang implementation the data type for validator pair is called
Validator
[TMBC-VALIDATOR-SET.1]
A validator set is a set of validator pairs. For a validator set vs, we write TotalVotingPower(vs) for the sum of the voting powers of its validator pairs.
[TMBC-VOTE.1]
A vote contains a prevote or precommit message sent and signed by
a validator node during the execution of consensus. Each
message contains the following fields
Type: prevote or precommitHeight: positive integerRounda positive integerBlockIDa Hashvalue of a block (not necessarily a block of the chain)
[TMBC-COMMIT.1]
A commit is a set of precommit message.
Tendermint Failure Model
[TMBC-AUTH-BYZ.1]
We assume the authenticated Byzantine fault model in which no node (faulty or correct) may break digital signatures, but otherwise, no additional assumption is made about the internal behavior of faulty nodes. That is, faulty nodes are only limited in that they cannot forge messages.
[TMBC-TIME-PARAMS.1]
A Tendermint blockchain has the following configuration parameters:
- unbondingPeriod: a time duration.
- trustingPeriod: a time duration smaller than unbondingPeriod.
[TMBC-CORRECT.1]
We define a predicate correctUntil(n, t), where n is a node and t is a time point. The predicate correctUntil(n, t) is true if and only if the node n follows all the protocols (at least) until time t.
[TMBC-FM-2THIRDS.1]
If a block h is in the chain, then there exists a subset CorrV of h.NextValidators, such that:
- TotalVotingPower(CorrV) > 2/3 TotalVotingPower(h.NextValidators); cf. [TMBC-VALIDATOR-SET.1]
- For every validator pair (n,p) in CorrV, it holds correctUntil(n, h.Time + trustingPeriod); cf. [TMBC-CORRECT.1]
The definition of correct [[TMBC-CORRECT.1]][TMBC-CORRECT-link] refers to realtime, while it is used here with Time and trustingPeriod, which are "hardware times". We do not make a distinction here.
[TMBC-CORR-FULL.1]
Every correct full node locally stores a prefix of the current list of headers from [TMBC-SEQ.1].
What the Light Client Checks
From [TMBC-FM-2THIRDS.1] we directly derive the following observation:
[TMBC-VAL-CONTAINS-CORR.1]
Given a (trusted) block tb of the blockchain, a given set of full nodes N contains a correct node at a real-time t, if
- t - trustingPeriod < tb.Time < t
- the voting power in tb.NextValidators of nodes in N is more than 1/3 of TotalVotingPower(tb.NextValidators)
The following describes how a commit for a given block b must look like.
[TMBC-SOUND-DISTR-POSS-COMMIT.1]
For a block b, each element pc of PossibleCommit(b) satisfies:
- pc contains only votes (cf. [TMBC-VOTE.1]) by validators from b.Validators
- the sum of the voting powers in pc is greater than 2/3 TotalVotingPower(b.Validators)
- and there is an r such that each vote v in pc satisfies
- v.Type = precommit
- v.Height = b.Height
- v.Round = r
- v.blockID = hash(b)
The following property comes from the validity of the consensus: A correct validator node only sends
prevoteorprecommit, ifBlockIDof the new (to-be-decided) block is equal to the hash of the last block.
[TMBC-VAL-COMMIT.1]
If for a block b, a commit c
- contains at least one validator pair (v,p) such that v is a correct validator node, and
- is contained in PossibleCommit(b)
then the block b is on the blockchain.
Context of this document
In this document we specify the light client verification component, called Core Verification. The Core Verification communicates with a full node. As full nodes may be faulty, it cannot trust the received information, but the light client has to check whether the header it receives coincides with the one generated by Tendermint consensus.
The two properties [TMBC-VAL-CONTAINS-CORR.1] and [TMBC-VAL-COMMIT] formalize the checks done by this specification: Given a trusted block tb and an untrusted block ub with a commit cub, one has to check that cub is in PossibleCommit(ub), and that cub contains a correct node using tb.
Part II - Sequential Definition of the Verification Problem
Verification Informal Problem statement
Given a height targetHeight as an input, the Verifier eventually stores a header h of height targetHeight locally. This header h is generated by the Tendermint blockchain. In particular, a header that was not generated by the blockchain should never be stored.
Sequential Problem statement
[LCV-SEQ-LIVE.1]
The Verifier gets as input a height targetHeight, and eventually stores the header of height targetHeight of the blockchain.
[LCV-SEQ-SAFE.1]
The Verifier never stores a header which is not in the blockchain.
Part III - Light Client as Distributed System
Incentives
Faulty full nodes may benefit from lying to the light client, by making the light client accept a block that deviates (e.g., contains additional transactions) from the one generated by Tendermint consensus. Users using the light client might be harmed by accepting a forged header.
The attack detector of the light client may help the correct full nodes to understand whether their header is a good one. Hence, in combination with the light client detector, the correct full nodes have the incentive to respond. We can thus base liveness arguments on the assumption that correct full nodes reliably talk to the light client.
Computational Model
[LCV-A-PEER.1]
The verifier communicates with a full node called primary. No assumption is made about the full node (it may be correct or faulty).
[LCV-A-COMM.1]
Communication between the light client and a correct full node is reliable and bounded in time. Reliable communication means that messages are not lost, not duplicated, and eventually delivered. There is a (known) end-to-end delay Delta, such that if a message is sent at time t then it is received and processes by time t + Delta. This implies that we need a timeout of at least 2 Delta for remote procedure calls to ensure that the response of a correct peer arrives before the timeout expires.
[LCV-A-TFM.1]
The Tendermint blockchain satisfies the Tendermint failure model [TMBC-FM-2THIRDS.1].
[LCV-A-VAL.1]
The system satisfies [TMBC-AUTH-BYZ.1] and [TMBC-FM-2THIRDS.1]. Thus, there is a blockchain that satisfies the soundness requirements (that is, the validation rules in [block]).
Distributed Problem Statement
Two Kinds of Termination
We do not assume that primary is correct. Under this assumption no protocol can guarantee the combination of the sequential properties. Thus, in the (unreliable) distributed setting, we consider two kinds of termination (successful and failure) and we will specify below under what (favorable) conditions Core Verification ensures to terminate successfully, and satisfy the requirements of the sequential problem statement:
[LCV-DIST-TERM.1]
Core Verification either terminates successfully or it terminates with failure.
Design choices
[LCV-DIST-STORE.2]
Core Verification returns a data structure called LightStore that contains light blocks (that contain a header).
[LCV-DIST-INIT.2]
Core Verification is called with
- primary: the PeerID of a full node (with verification communicates)
- root: a light block (the root of trust)
- targetHeight: a height (the height of a header that should be obtained)
Temporal Properties
[LCV-DIST-SAFE.2]
It is always the case that every header in LightStore was generated by an instance of Tendermint consensus.
[LCV-DIST-LIVE.2]
If a new instance of Core Verification is called with a height targetHeight greater than root.Header.Height it must must eventually terminate.
- If
- the primary is correct (and locally has the block of targetHeight), and
- the age of root is always less than the trusting period,
then Core Verification adds a verified header hd with height targetHeight to LightStore and it terminates successfully
These definitions imply that if the primary is faulty, a header may or may not be added to LightStore. In any case, [LCV-DIST-SAFE.2] must hold. The invariant [LCV-DIST-SAFE.2] and the liveness requirement [LCV-DIST-LIVE.2] allow that verified headers are added to LightStore whose height was not passed to the verifier (e.g., intermediate headers used in bisection; see below). Note that for liveness, initially having a root within the trustinPeriod is not sufficient. However, as this specification will leave some freedom with respect to the strategy in which order to download intermediate headers, we do not give a more precise liveness specification here. After giving the specification of the protocol, we will discuss some liveness scenarios below.
Solving the sequential specification
This specification provides a partial solution to the sequential specification. The Verifier solves the invariant of the sequential part
[LCV-DIST-SAFE.2] => [LCV-SEQ-SAFE.1]
In the case the primary is correct, and root is a recent header in LightStore, the verifier satisfies the liveness requirements.
⋀ primary is correct
⋀ root.header.Time > now - trustingPeriod
⋀ [LCV-A-Comm.1] ⋀ (
( [TMBC-CorrFull.1] ⋀
[LCV-DIST-LIVE.2] )
⟹ [LCV-SEQ-LIVE.1]
)
Part IV - Light Client Verification Protocol
We provide a specification for Light Client Verification. The local
code for verification is presented by a sequential function
VerifyToTarget to highlight the control flow of this functionality.
We note that if a different concurrency model is considered for
an implementation, the sequential flow of the function may be
implemented with mutexes, etc. However, the light client verification
is partitioned into three blocks that can be implemented and tested
independently:
FetchLightBlockis called to download a light block (header) of a given height from a peer.ValidAndVerifiedis a local code that checks the header.Scheduledecides which height to try to verify next. We keep this underspecified as different implementations (currently in Goland and Rust) may implement different optimizations here. We just provide necessary conditions on how the height may evolve.
Definitions
Data Types
The core data structure of the protocol is the LightBlock.
[LCV-DATA-LIGHTBLOCK.1]
type LightBlock struct {
Header Header
Commit Commit
Validators ValidatorSet
}
[LCV-DATA-LIGHTSTORE.2]
LightBlocks are stored in a structure which stores all LightBlock from initialization or received from peers.
type LightStore struct {
...
}
[LCV-DATA-LS-ROOT.2]
For each lightblock in a lightstore we record in a field verification-root of
type Height.
verification-rootrecords the height of a lightblock that can be used to verify the lightblock in one step
[LCV-INV-LS-ROOT.2]
At all times, if a lightblock b in a lightstore has b.verification-root = h, then
- the lightstore contains a lightblock with height h, or
- b has the minimal height of all lightblocks in lightstore, then b.verification-root should be nil.
The LightStore exposes the following functions to query stored LightBlocks.
[LCV-DATA-LS-STATE.1]
Each LightBlock is in one of the following states:
type VerifiedState int
const (
StateUnverified = iota + 1
StateVerified
StateFailed
StateTrusted
)
[LCV-FUNC-GET.1]
func (ls LightStore) Get(height Height) (LightBlock, bool)
- Expected postcondition
- returns a LightBlock at a given height or false in the second argument if the LightStore does not contain the specified LightBlock.
[LCV-FUNC-LATEST.1]
func (ls LightStore) Latest() LightBlock
- Expected postcondition
- returns the highest light block
[LCV-FUNC-ADD.1]
func (ls LightStore) Add(newBlock)
- Expected precondition
- the lightstore is empty
- Expected postcondition
- adds newBlock into light store
[LCV-FUNC-STORE.1]
func (ls LightStore) store_chain(newLS LightStore)
- Expected postcondition
- adds
newLSto the lightStore.
- adds
[LCV-FUNC-LATEST-VERIF.2]
func (ls LightStore) LatestVerified() LightBlock
- Expected postcondition
- returns the highest light block whose state is
StateVerified
- returns the highest light block whose state is
[LCV-FUNC-FILTER.1]
func (ls LightStore) FilterVerified() LightStore
- Expected postcondition
- returns all the lightblocks of the lightstore with state
StateVerified
- returns all the lightblocks of the lightstore with state
[LCV-FUNC-UPDATE.2]
func (ls LightStore) Update(lightBlock LightBlock, verfiedState
VerifiedState, root-height Height)
- Expected postcondition
- the lightblock is part of the lightstore
- The state of the LightBlock is set to verifiedState.
- The verification-root of the LightBlock is set to root-height
func (ls LightStore) TraceTo(lightBlock LightBlock) (LightBlock, LightStore)
- Expected postcondition
- returns a trusted lightblock
rootfrom the lightstore with a height less thanlightBlock - returns a lightstore that contains lightblocks that constitute a
verification trace from
roottolightBlock(includinglightBlock)
- returns a trusted lightblock
Inputs
- root: A light block that is trusted
- primary: peerID
- targetHeight: the height of the needed header
Configuration Parameters
- trustThreshold: a float. Can be used if correctness should not be based on more voting power and 1/3.
- trustingPeriod: a time duration [TMBC-TIME_PARAMS.1].
- clockDrift: a time duration. Correction parameter dealing with only approximately synchronized clocks.
Variables
- nextHeight: initially targetHeight
nextHeight should be thought of the "height of the next header we need to download and verify"
Assumptions
[LCV-A-INIT.2]
-
root is from the blockchain
-
targetHeight > root.Header.Height
Invariants
[LCV-INV-TP.1]
It is always the case that LightStore.LatestTrusted.Header.Time > now - trustingPeriod.
If the invariant is violated, the light client does not have a header it can trust. A trusted header must be obtained externally, its trust can only be based on social consensus.
We use the convention that root is assumed to be verified.
Used Remote Functions
We use the functions commit and validators that are provided
by the RPC client for Tendermint.
func Commit(height int64) (SignedHeader, error)
- Implementation remark
- RPC to full node n
- JSON sent:
// POST /commit
{
"jsonrpc": "2.0",
"id": "ccc84631-dfdb-4adc-b88c-5291ea3c2cfb", // UUID v4, unique per request
"method": "commit",
"params": {
"height": 1234
}
}
- Expected precondition
- header of
heightexists on blockchain
- header of
- Expected postcondition
- if n is correct: Returns the signed header of height
heightfrom the blockchain if communication is timely (no timeout) - if n is faulty: Returns a signed header with arbitrary content
- if n is correct: Returns the signed header of height
- Error condition
- if n is correct: precondition violated or timeout
- if n is faulty: arbitrary error
----;
func Validators(height int64) (ValidatorSet, error)
- Implementation remark
- RPC to full node n
- JSON sent:
// POST /validators
{
"jsonrpc": "2.0",
"id": "ccc84631-dfdb-4adc-b88c-5291ea3c2cfb", // UUID v4, unique per request
"method": "validators",
"params": {
"height": 1234
}
}
- Expected precondition
- header of
heightexists on blockchain
- header of
- Expected postcondition
- if n is correct: Returns the validator set of height
heightfrom the blockchain if communication is timely (no timeout) - if n is faulty: Returns arbitrary validator set
- if n is correct: Returns the validator set of height
- Error condition
- if n is correct: precondition violated or timeout
- if n is faulty: arbitrary error
----;
Communicating Function
[LCV-FUNC-FETCH.1]
func FetchLightBlock(peer PeerID, height Height) LightBlock
- Implementation remark
- RPC to peer at PeerID
- calls
Commitfor height andValidatorsfor height and height+1
- Expected precondition
heightis less than or equal to height of the peer [LCV-IO-PRE-HEIGHT.1]
- Expected postcondition:
- if node is correct:
- Returns the LightBlock lb of height
heightthat is consistent with the blockchain - lb.provider = peer [LCV-IO-POST-PROVIDER.1]
- lb.Header is a header consistent with the blockchain
- lb.Validators is the validator set of the blockchain at height nextHeight
- lb.NextValidators is the validator set of the blockchain at height nextHeight + 1
- Returns the LightBlock lb of height
- if node is faulty: Returns a LightBlock with arbitrary content [TMBC-AUTH-BYZ.1]
- if node is correct:
- Error condition
- if n is correct: precondition violated
- if n is faulty: arbitrary error
- if lb.provider != peer
- times out after 2 Delta (by assumption n is faulty)
----;
Core Verification
Outline
The VerifyToTarget is the main function and uses the following functions.
FetchLightBlockis called to download the next light block. It is the only function that communicates with other nodesValidAndVerifiedchecks whether header is valid and checks if a new lightBlock should be trusted based on a previously verified lightBlock.Scheduledecides which height to try to verify next
In the following description of VerifyToTarget we do not deal with error
handling. If any of the above function returns an error, VerifyToTarget just
passes the error on.
[LCV-FUNC-MAIN.2]
func VerifyToTarget(primary PeerID, root LightBlock,
targetHeight Height) (LightStore, Result) {
lightStore = new LightStore;
lightStore.Update(root, StateVerified, root.verifiedBy);
nextHeight := targetHeight;
for lightStore.LatestVerified.height < targetHeight {
// Get next LightBlock for verification
current, found := lightStore.Get(nextHeight)
if !found {
current = FetchLightBlock(primary, nextHeight)
lightStore.Update(current, StateUnverified, nil)
}
// Verify
verdict = ValidAndVerified(lightStore.LatestVerified, current)
// Decide whether/how to continue
if verdict == SUCCESS {
lightStore.Update(current, StateVerified, lightStore.LatestVerified.Height)
}
else if verdict == NOT_ENOUGH_TRUST {
// do nothing
// the light block current passed validation, but the validator
// set is too different to verify it. We keep the state of
// current at StateUnverified. For a later iteration, Schedule
// might decide to try verification of that light block again.
}
else {
// verdict is some error code
lightStore.Update(current, StateFailed, nil)
return (nil, ResultFailure)
}
nextHeight = Schedule(lightStore, nextHeight, targetHeight)
}
return (lightStore.FilterVerified, ResultSuccess)
}
- Expected precondition
- root is within the trustingPeriod [LCV-PRE-TP.1]
- targetHeight is greater than the height of root
- Expected postcondition:
- returns lightStore that contains a LightBlock that corresponds to a block of the blockchain of height targetHeight (that is, the LightBlock has been added to lightStore) [LCV-POST-LS.1]
- Error conditions
- if the precondition is violated
- if
ValidAndVerifiedorFetchLightBlockreport an error - if [LCV-INV-TP.1] is violated
Details of the Functions
[LCV-FUNC-VALID.2]
func ValidAndVerified(trusted LightBlock, untrusted LightBlock) Result
- Expected precondition:
- untrusted is valid, that is, satisfies the soundness checks
- untrusted is well-formed, that is,
- untrusted.Header.Time < now + clockDrift
- untrusted.Validators = hash(untrusted.Header.Validators)
- untrusted.NextValidators = hash(untrusted.Header.NextValidators)
- trusted.Header.Time > now - trustingPeriod
- the
HeightandTimeoftrustedare smaller than the Height andTimeofuntrusted, respectively - the untrusted.Header is well-formed (passes the tests from
[block]), and in particular
- if the untrusted header
unstrusted.Headeris the immediate successor oftrusted.Header, then it holds that- trusted.Header.NextValidators = untrusted.Header.Validators, and moreover,
- untrusted.Header.Commit
- contains signatures by more than two-thirds of the validators
- contains no signature from nodes that are not in trusted.Header.NextValidators
- if the untrusted header
- Expected postcondition:
- Returns
SUCCESS:- if untrusted is the immediate successor of trusted, or otherwise,
- if the signatures of a set of validators that have more than max(1/3,trustThreshold) of voting power in trusted.Nextvalidators is contained in untrusted.Commit (that is, header passes the tests [TMBC-VAL-CONTAINS-CORR.1] and [TMBC-VAL-COMMIT.1])
- Returns
NOT_ENOUGH_TRUSTif:- untrusted is not the immediate successor of trusted and the max(1/3,trustThreshold) threshold is not reached (that is, if [TMBC-VAL-CONTAINS-CORR.1] fails and header is does not violate the soundness checks [block]).
- Returns
- Error condition:
- if precondition violated
----;
[LCV-FUNC-SCHEDULE.1]
func Schedule(lightStore, nextHeight, targetHeight) Height
- Implementation remark: If picks the next height to be verified. We keep the precise choice of the next header under-specified. It is subject to performance optimizations that do not influence the correctness
- Expected postcondition: [LCV-SCHEDULE-POST.1]
Return H s.t.
- if lightStore.LatestVerified.Height = nextHeight and
lightStore.LatestVerified < targetHeight then
nextHeight < H <= targetHeight - if lightStore.LatestVerified.Height < nextHeight and
lightStore.LatestVerified.Height < targetHeight then
lightStore.LatestVerified.Height < H < nextHeight - if lightStore.LatestVerified.Height = targetHeight then
H = targetHeight
- if lightStore.LatestVerified.Height = nextHeight and
lightStore.LatestVerified < targetHeight then
Case i. captures the case where the light block at height nextHeight has been verified, and we can choose a height closer to the targetHeight. As we get the lightStore as parameter, the choice of the next height can depend on the lightStore, e.g., we can pick a height for which we have already downloaded a light block. In Case ii. the header of nextHeight could not be verified, and we need to pick a smaller height. In Case iii. is a special case when we have verified the targetHeight.
Solving the distributed specification
Analogous to [001_published]
Liveness Scenarios
Analogous to [001_published]
Part V - Supporting the IBC Relayer
The above specification focuses on the most common case, which also
constitutes the most challenging task: using the Tendermint security
model to verify light blocks without
downloading all intermediate blocks. To focus on this challenge, above
we have restricted ourselves to the case where targetHeight is
greater than the height of any trusted header. This simplified
presentation of the algorithm as initially
lightStore.LatestVerified() is less than targetHeight, and in the
process of verification lightStore.LatestVerified() increases until
targetHeight is reached.
For IBC there are two additional challenges:
-
it might be that some "older" header is needed, that is, targetHeight < lightStore.LatestVerified(). The supervisor checks whether it is in this case by calling
LatestPreviousandMinVerifiedand if so it callsBackwards. All these functions are specified below. -
In order to submit proof of a light client attack, a relayer may need to submit a verification trace. This it is important to compute such a trace efficiently. That it can be done is based on the invariant [LCV-INV-LS-ROOT.2] that needs to be maintained by the light client. In particular
VerifyToTargetandBackwardsneed to take care of settingverification-root.
[LCV-FUNC-LATEST-PREV.2]
func (ls LightStore) LatestPrevious(height Height) (LightBlock, bool)
- Expected postcondition
- returns a light block lb that satisfies:
- lb is in lightStore
- lb is in StateTrusted
- lb is not expired
- lb.Header.Height < height
- for all b in lightStore s.t. b is trusted and not expired it holds lb.Header.Height >= b.Header.Height
- false in the second argument if the LightStore does not contain such an lb.
- returns a light block lb that satisfies:
----;
[LCV-FUNC-LOWEST.2]
func (ls LightStore) Lowest() (LightBlock)
- Expected postcondition
- returns the lowest trusted light block within trusting period
----;
[LCV-FUNC-MIN.2]
func (ls LightStore) MinVerified() (LightBlock, bool)
- Expected postcondition
- returns a light block lb that satisfies:
- lb is in lightStore
- lb.Header.Height is minimal in the lightStore
- false in the second argument if the LightStore does not contain such an lb.
- returns a light block lb that satisfies:
If a height that is smaller than the smallest height in the lightstore is required, we check the hashes backwards. This is done with the following function:
[LCV-FUNC-BACKWARDS.2]
func Backwards (primary PeerID, root LightBlock, targetHeight Height)
(LightStore, Result) {
lb := root;
lightStore := new LightStore;
lightStore.Update(lb, StateTrusted, lb.verifiedBy)
latest := lb.Header
for i := lb.Header.height - 1; i >= targetHeight; i-- {
// here we download height-by-height. We might first download all
// headers down to targetHeight and then check them.
current := FetchLightBlock(primary,i)
if (hash(current) != latest.Header.LastBlockId) {
return (nil, ResultFailure)
}
else {
// latest and current are linked together by LastBlockId
// therefore it is not relevant which we verified first
// for consistency, we store latest was veried using
// current so that the verifiedBy is always pointing down
// the chain
lightStore.Update(current, StateTrusted, nil)
lightStore.Update(latest, StateTrusted, current.Header.Height)
}
latest = current
}
return (lightStore, ResultSuccess)
}
References
[block] Specification of the block data structure.
[RPC] RPC client for Tendermint
[attack-detector] The specification of the light client attack detector.
[fullnode] Specification of the full node API
[ibc-rs] Rust implementation of IBC modules and relayer.
[lightclient] The light client ADR [77d2651 on Dec 27, 2019].
Light Client Verificaiton
[LCV-FUNC-VERIFYCOMMITLIGHT.1]
VerifyCommitLight verifies that 2/3+ of the signatures for a validator set were for a given blockID. The function will finish early and thus may not check all signatures.
func VerifyCommitLight(chainID string, vals *ValidatorSet, blockID BlockID,
height int64, commit *Commit) error {
// run a basic validation of the arguments
if err := verifyBasicValsAndCommit(vals, commit, height, blockID); err != nil {
return err
}
// calculate voting power needed
votingPowerNeeded := vals.TotalVotingPower() * 2 / 3
var (
val *Validator
valIdx int32
seenVals = make(map[int32]int, len(commit.Signatures))
talliedVotingPower int64 = 0
voteSignBytes []byte
)
for idx, commitSig := range commit.Signatures {
// ignore all commit signatures that are not for the block
if !commitSig.ForBlock() {
continue
}
// If the vals and commit have a 1-to-1 correspondance we can retrieve
// them by index else we need to retrieve them by address
if lookUpByIndex {
val = vals.Validators[idx]
} else {
valIdx, val = vals.GetByAddress(commitSig.ValidatorAddress)
// if the signature doesn't belong to anyone in the validator set
// then we just skip over it
if val == nil {
continue
}
// because we are getting validators by address we need to make sure
// that the same validator doesn't commit twice
if firstIndex, ok := seenVals[valIdx]; ok {
secondIndex := idx
return fmt.Errorf("double vote from %v (%d and %d)", val, firstIndex, secondIndex)
}
seenVals[valIdx] = idx
}
voteSignBytes = commit.VoteSignBytes(chainID, int32(idx))
if !val.PubKey.VerifySignature(voteSignBytes, commitSig.Signature) {
return fmt.Errorf("wrong signature (#%d): %X", idx, commitSig.Signature)
}
// Add the voting power of the validator
// to the tally
talliedVotingPower += val.VotingPower
// check if we have enough signatures and can thus exit early
if talliedVotingPower > votingPowerNeeded {
return nil
}
}
if got, needed := talliedVotingPower, votingPowerNeeded; got <= needed {
return ErrNotEnoughVotingPowerSigned{Got: got, Needed: needed}
}
return nil
}
Ivy Proofs
Copyright (c) 2020 Galois, Inc.
SPDX-License-Identifier: Apache-2.0
Contents
This folder contains:
tendermint.ivy, a specification of Tendermint algorithm as described in The latest gossip on BFT consensus by E. Buchman, J. Kwon, Z. Milosevic.abstract_tendermint.ivy, a more abstract specification of Tendermint that is more verification-friendly.classic_safety.ivy, a proof that Tendermint satisfies the classic safety property of BFT consensus: if every two quorums have a well-behaved node in common, then no two well-behaved nodes ever disagree.accountable_safety_1.ivy, a proof that, assuming every quorum contains at least one well-behaved node, if two well-behaved nodes disagree, then there is evidence demonstrating at least f+1 nodes misbehaved.accountable_safety_2.ivy, a proof that, regardless of any assumption about quorums, well-behaved nodes cannot be framed by malicious nodes. In other words, malicious nodes can never construct evidence that incriminates a well-behaved node.network_shim.ivy, the network model and a convenienceshimobject to interface with the Tendermint specification.domain_model.ivy, a specification of the domain model underlying the Tendermint specification, i.e. rounds, value, quorums, etc.
All specifications and proofs are written in Ivy.
The license above applies to all files in this folder.
Building and running
The easiest way to check the proofs is to use Docker.
- Install Docker and Docker Compose.
- Build a Docker image:
docker compose build - Run the proofs inside the Docker container:
docker compose run tendermint-proof. This will check all the proofs with theivy_checkcommand and write the output ofivy_checkto a subdirectory of `./output/'
Security
Report vulnerabilities privately using the security policy.
Vulnerability names
This inherited list records candidate names for vulnerabilities. Follow the security policy to report a vulnerability; names in this list do not indicate that a vulnerability exists or has been disclosed.
Here is a list of pre-allocated vulnerability names, to be crossed-out as they are used. Hopefully we don't make it all the way through the list!
Alderfly Aphid Ant Bedbug Beetle Bookworm Bumblebee Butterfly Caterpillar Cicada Centipede Cockroach Cranefly Cricket Damselfly Dragonfly Earthworm Earwig Flea Firefly Gnat Grasshopper Hornet Horsefly Jumpingbean Katydid Lacewing Ladybug Leafroller Locust Louse Maggot Mantis Mealybug Midge Mite Mosquito Moth Palmerworm Parasite Sawfly Scarab Silkworm Silverfish Skeletonizer Slug Sphinx Stinkbug Termite Thrip Viceroy Walkingstick Wasp Weevil Whitefly Yellowjacket
ADR 073: Adopt LibP2P
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 2021-11-02: Initial Draft (@tychoish)
Status
Proposed.
Context
As part of the upstream 0.35 development cycle, the original team completed the first phase of the work described in ADRs 61 and 62, which included a large scale refactoring of the reactors and the p2p message routing. This replaced the switch and many of the other legacy components without breaking protocol or network-level interoperability and left the legacy connection/socket handling code.
Following the release, the team has reexamined the state of the code and the design, as well as Tenderdash's requirements. The notes from that process are available in the P2P Roadmap RFC.
This ADR supersedes the decisions made in ADRs 60 and 61, but builds on the completed portions of this work. Previously, the boundaries of peer management, message handling, and the higher level business logic (e.g., "the reactors") were intermingled, and core elements of the p2p system were responsible for the orchestration of higher-level business logic. Refactoring the legacy components made it more obvious that this entanglement of responsibilities had outsized influence on the entire implementation, making it difficult to iterate within the current abstractions. It would not be viable to maintain interoperability with legacy systems while also achieving many of our broader objectives.
LibP2P is a thoroughly-specified implementation of a peer-to-peer networking stack, designed specifically for systems such as ours. Adopting LibP2P as the basis of Tenderdash will allow the Tenderdash team to focus more of their time on other differentiating aspects of the system, and make it possible for the ecosystem as a whole to take advantage of tooling and efforts of the LibP2P platform.
Alternative Approaches
As discussed in the P2P Roadmap RFC, the primary alternative would be to continue development of Tenderdash's home-grown peer-to-peer layer. While that would give the Tenderdash team maximal control over the peer system, the current design is unexceptional on its own merits, and the prospective maintenance burden for this system exceeds our tolerances for the medium term.
Tenderdash can and should differentiate itself not on the basis of its networking implementation or peer management tools, but providing a consistent operator experience, a battle-tested consensus algorithm, and an ergonomic user experience.
Decision
The upstream proposal targeted libp2p adoption during its 0.37 development cycle,
replacing the bespoke Tenderdash P2P stack. This will remove the
Endpoint, Transport, Connection, and PeerManager abstractions
and leave the reactors, p2p.Router and p2p.Channel
abstractions.
LibP2P may obviate the need for a dedicated peer exchange (PEX) reactor, which would also in turn obviate the need for a dedicated seed mode. If this is the case, then all of this functionality would be removed.
If it turns out (based on the advice of Protocol Labs) that it makes
sense to maintain separate pubsub or gossipsub topics
per-message-type, then the Router abstraction could also
be entirely subsumed.
Detailed Design
Implementation Changes
The seams in the P2P implementation between the higher level
constructs (reactors), the routing layer (Router) and the lower
level connection and peer management code make this operation
relatively straightforward to implement. A key
goal in this design is to minimize the impact on the reactors
(potentially entirely,) and completely remove the lower level
components (e.g., Transport, Connection and PeerManager) using the
separation afforded by the Router layer. The current state of the
code makes these changes relatively surgical, and limited to a small
number of methods:
-
p2p.Router.OpenChannelwill still return aChannelstructure which will continue to serve as a pipe between the reactors and theRouter. The implementation will no longer need the queue implementation, and will instead start goroutines that are responsible for routing the messages from the channel to libp2p fundamentals, replacing the currentp2p.Router.routeChannel. -
The current
p2p.Router.dialPeersandp2p.Router.acceptPeers, are responsible for establishing outbound and inbound connections, respectively. These methods will be removed, along withp2p.Router.openConnection, and the libp2p connection manager will be responsible for maintaining network connectivity. -
The
p2p.Channelinterface will change to replace Go channels with a more functional interface for sending messages. New methods on this object will take contexts to support safe cancellation, and return errors, and will block rather than running asynchronously. TheOutchannel through which reactors send messages to Peers, will be replaced by aSendmethod, and the Error channel will be replaced by anErrormethod. -
Reactors will be passed an interface that will allow them to access Peer information from libp2p. This will supplant the
p2p.PeerUpdatessubscription. -
Add some kind of heartbeat message at the application level (e.g. with a reactor,) potentially connected to libp2p's DHT to be used by reactors for service discovery, message targeting, or other features.
-
Replace the existing/legacy handshake protocol with Noise.
This project will initially use the TCP-based transport protocols within libp2p. QUIC is also available as an option that we may implement later. We will not support mixed networks in the initial release, but will revisit that possibility later if there is a demonstrated need.
Upgrade and Compatibility
Because the routers and all current P2P libraries are internal
packages and not part of the public API, the only changes to the public
API surface area of Tenderdash will be different configuration
file options, replacing the current P2P options with options relevant
to libp2p.
However, it will not be possible to run a network with both networking stacks active at once, so the upgrade to the version of Tenderdash will need to be coordinated between all nodes of the network. This is consistent with the expectations around upgrades for Tenderdash moving forward, and will help manage both the complexity of the project and the implementation timeline.
Open Questions
-
What is the role of Protocol Labs in the implementation of libp2p in Tenderdash, both during the initial implementation and on an ongoing basis thereafter?
-
Should all P2P traffic for a given node be pushed to a single topic, so that a topic maps to a specific ChainID, or should each reactor (or type of message) have its own topic? How many topics can a libp2p network support? Is there testing that validates the capabilities?
-
Tenderdash presently provides a very coarse QoS-like functionality using priorities based on message-type. This intuitively/theoretically ensures that evidence and consensus messages don't get starved by blocksync/statesync messages. It's unclear if we can or should attempt to replicate this with libp2p.
-
What kind of QoS functionality does libp2p provide and what kind of metrics does libp2p provide about it's QoS functionality?
-
Is it possible to store additional (and potentially arbitrary) information into the DHT as part of the heartbeats between nodes, such as the latest height, and then access that in the reactors. How frequently can the DHT be updated?
-
Does it make sense to have reactors continue to consume inbound messages from a Channel (
In) or is there another interface or pattern that we should consider?- We should avoid exposing Go channels when possible, and likely some kind of alternate iterator likely makes sense for processing messages within the reactors.
-
What are the security and protocol implications of tracking information from peer heartbeats and exposing that to reactors?
-
How much (or how little) configuration can Tenderdash provide for libp2p, particularly on the first release?
- In general, we should not support byo-functionality for libp2p components within Tenderdash, and reduce the configuration surface area, as much as possible.
-
What are the best ways to provide request/response semantics for reactors on top of libp2p? Will it be possible to add request/response semantics in a future release or is there anticipatory work that needs to be done as part of the initial release?
Consequences
Positive
-
Reduce the maintenance burden for the Tenderdash Core team by removing a large swath of legacy code that has proven to be difficult to modify safely.
-
Remove the responsibility for maintaining and developing the entire peer management system (p2p) and stack.
-
Provide users with a more stable peer and networking system, Tenderdash can improve operator experience and network stability.
Negative
-
By deferring to library implementations for peer management and networking, Tenderdash loses some flexibility for innovating at the peer and networking level. However, Tenderdash should be innovating primarily at the consensus layer, and libp2p does not preclude optimization or development in the peer layer.
-
Libp2p is a large dependency and Tenderdash would become dependent upon Protocol Labs' release cycle and prioritization for bug fixes. If this proves onerous, it's possible to maintain a vendor fork of relevant components as needed.
Neutral
- N/A
References
ADR 74: Migrate Timeout Parameters to Consensus Parameters
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 03-Jan-2022: Initial draft (@williambanfield)
- 13-Jan-2022: Updated to indicate work on upgrade path needed (@williambanfield)
Status
Proposed
Context
Background
Tenderdash's consensus timeout parameters are currently configured locally by each validator in the validator's config.toml. This means that the validators on a Tenderdash network may have different timeouts from each other. There is no reason for validators on the same network to configure different timeout values. Proper functioning of the Tenderdash consensus algorithm relies on these parameters being uniform across validators.
The configurable values are as follows:
TimeoutPropose- How long the consensus algorithm waits for a proposal block before issuing a prevote.
- If no prevote arrives by
TimeoutPropose, then the consensus algorithm will issue a nil prevote.
TimeoutProposeDelta- How much the
TimeoutProposegrows each round.
- How much the
TimeoutPrevote- How long the consensus algorithm waits after receiving +2/3 prevotes with no quorum for a value before issuing a precommit for nil. (See the arXiv paper, Algorithm 1, Line 34)
TimeoutPrevoteDelta- How much the
TimeoutPrevoteincreases with each round.
- How much the
TimeoutPrecommit- How long the consensus algorithm waits after receiving +2/3 precommits that do not have a quorum for a value before entering the next round. (See the arXiv paper, Algorithm 1, Line 47)
TimeoutPrecommitDelta- How much the
TimeoutPrecommitincreases with each round.
- How much the
TimeoutCommit- How long the consensus algorithm waits after committing a block but before starting the new height.
- This gives a validator a chance to receive slow precommits.
SkipTimeoutCommit- Make progress as soon as the node has 100% of the precommits.
Overview of Change
We will consolidate the timeout parameters and migrate them from the node-local
config.toml file into the network-global consensus parameters.
The 8 timeout parameters will be consolidated down to 6. These will be as follows:
TimeoutPropose- Same as current
TimeoutPropose.
- Same as current
TimeoutProposeDelta- Same as current
TimeoutProposeDelta.
- Same as current
TimeoutVote- How long validators wait for votes in both the prevote
and precommit phase of the consensus algorithm. This parameter subsumes
the current
TimeoutPrevoteandTimeoutPrecommitparameters.
- How long validators wait for votes in both the prevote
and precommit phase of the consensus algorithm. This parameter subsumes
the current
TimeoutVoteDelta- How much the
TimeoutVotewill grow each successive round. This parameter subsumes the currentTimeoutPrevoteDeltaandTimeoutPrecommitDeltaparameters.
- How much the
TimeoutCommit- Same as current
TimeoutCommit.
- Same as current
BypassCommitTimeout- Same as current
SkipTimeoutCommit, renamed for clarity.
- Same as current
A safe default will be provided by Tenderdash for each of these parameters and networks will be able to update the parameters as they see fit. Local updates to these parameters will no longer be possible; instead, the application will control updating the parameters. Applications using the Cosmos SDK will be automatically be able to change the values of these consensus parameters via a governance proposal.
The original proposal considered this change low-risk because many upstream networks used the default timeout values. This observation does not replace validation of the settings required by a Tenderdash network.
Why this parameter consolidation?
Reducing the number of parameters is good for UX. Fewer superfluous parameters makes running and operating a Tenderdash network less confusing.
The Prevote and Precommit messages are both similar sizes, require similar amounts of processing so there is no strong need for them to be configured separately.
The TimeoutPropose parameter governs how long Tenderdash will wait for the proposed
block to be gossiped. Blocks are much larger than votes and therefore tend to be
gossiped much more slowly. It therefore makes sense to keep TimeoutPropose and
the TimeoutProposeDelta as parameters separate from the vote timeouts.
TimeoutCommit is used by chains to ensure that the network waits for the votes from
slower validators before proceeding to the next height. Without this timeout, the votes
from slower validators would consistently not be included in blocks and those validators
would not be counted as 'up' from the chain's perspective. Being down damages a validator's
reputation and causes potential stakers to think twice before delegating to that validator.
TimeoutCommit also prevents the network from producing the next height as soon as validators
on the fastest hardware with a summed voting power of +2/3 of the network's total have
completed execution of the block. Allowing the network to proceed as soon as the fastest
+2/3 completed execution would have a cumulative effect over heights, eventually
leaving slower validators unable to participate in consensus at all. TimeoutCommit
therefore allows networks to have greater variability in hardware. Additional
discussion of this can be found in upstream issue 5911
and spec issue 359.
Alternative Approaches
Hardcode the parameters
Many Tenderdash networks run on similar cloud-hosted infrastructure. Therefore, they have similar bandwidth and machine resources. The timings for propagating votes and blocks are likely to be reasonably similar across networks. As a result, the timeout parameters are good candidates for being hardcoded. Hardcoding the timeouts in Tenderdash would mean entirely removing these parameters from any configuration that could be altered by either an application or a node operator. Instead, Tenderdash would ship with a set of timeouts and all applications using Tenderdash would use this exact same set of values.
While Tenderdash nodes often run with similar bandwidth and on similar cloud-hosted
machines, there are enough points of variability to make configuring
consensus timeouts meaningful. Namely, Tenderdash network topologies are likely to be
very different from chain to chain. Additionally, applications may vary greatly in
how long the Commit phase may take. Applications that perform more work during Commit
require a longer TimeoutCommit to allow the application to complete its work
and be prepared for the next height.
Decision
The decision has been made to implement this work, with the caveat that the specific mechanism for introducing the new parameters to chains is still ongoing.
Detailed Design
New Consensus Parameters
A new TimeoutParams message will be added to the [params.proto file][consensus-params-proto].
This message will have the following form:
message TimeoutParams {
google.protobuf.Duration propose = 1;
google.protobuf.Duration propose_delta = 2;
google.protobuf.Duration vote = 3;
google.protobuf.Duration vote_delta = 4;
google.protobuf.Duration commit = 5;
bool bypass_commit_timeout = 6;
}
This new message will be added as a field into the [ConsensusParams
message][consensus-params-proto]. The same default values that are currently
set for these parameters in the local configuration
file will be used as the defaults for these new consensus parameters in the
consensus parameter defaults.
The new consensus parameters will be subject to the same validity rules as the current configuration values, namely, each value must be non-negative.
Migration
The new ConsensusParameters will be added during an upcoming release. In this
release, the old config.toml parameters will cease to control the timeouts and
an error will be logged on nodes that continue to specify these values. The specific
mechanism by which these parameters will added to a chain is being discussed in
RFC-009 and will be decided ahead of the next release.
The specific mechanism for adding these parameters depends on work related to soft upgrades, which is still ongoing.
Consequences
Positive
- Timeout parameters will be equal across all of the validators in a Tenderdash network.
- Remove superfluous timeout parameters.
Negative
Neutral
- Timeout parameters require consensus to change.
References
ADR 075: RPC Event Subscription Interface
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 01-Mar-2022: Update long-polling interface (@creachadair).
- 10-Feb-2022: Updates to reflect implementation.
- 26-Jan-2022: Marked accepted.
- 22-Jan-2022: Updated and expanded (@creachadair).
- 20-Nov-2021: Initial draft (@creachadair).
Status
Accepted
Background & Context
For context, see RFC 006: Event Subscription.
The Tenderdash RPC service permits clients to subscribe to the event stream generated by a consensus node. This allows clients to observe the state of the consensus network, including details of the consensus algorithm state machine, proposals, transaction delivery, and block completion. The application may also attach custom key-value attributes to events to expose application-specific details to clients.
The event subscription API in the RPC service currently comprises three methods:
-
subscribe: A request to subscribe to the events matching a specific query expression. Events can be filtered by their key-value attributes, including custom attributes provided by the application. -
unsubscribe: A request to cancel an existing subscription based on its query expression. -
unsubscribe_all: A request to cancel all existing subscriptions belonging to the client.
There are some important technical and UX issues with the current RPC event subscription API. The rest of this ADR outlines these problems in detail, and proposes a new API scheme intended to address them.
Issue 1: Persistent connections
To subscribe to a node's event stream, a client needs a persistent connection to the node. Unlike the other methods of the service, for which each call is serviced by a short-lived HTTP round trip, subscription delivers a continuous stream of events to the client by hijacking the HTTP channel for a websocket. The stream (and hence the HTTP request) persists until either the subscription is explicitly cancelled, or the connection is closed.
There are several problems with this API:
-
Expensive per-connection state: The server must maintain a substantial amount of state per subscriber client:
-
The current implementation uses a WebSocket for each active subscriber. The connection must be maintained even if there are no matching events for a given client.
The server can drop idle connections to save resources, but doing so terminates all subscriptions on those connections and forces those clients to re-connect, adding additional resource churn for the server.
-
In addition, the server maintains a separate buffer of undelivered events for each client. This is to reduce the dual risks that a client will miss events, and that a slow client could "push back" on the publisher, impeding the progress of consensus.
Because event traffic is quite bursty, queues can potentially take up a lot of memory. Moreover, each subscriber may have a different filter query, so the server winds up having to duplicate the same events among multiple subscriber queues. Not only does this add memory pressure, but it does so most at the worst possible time, i.e., when the server is already under load from high event traffic.
-
-
Operational access control is difficult: The server's websocket interface exposes all the RPC service endpoints, not only the subscription methods. This includes methods that allow callers to inject arbitrary transactions (
broadcast_tx_*) and evidence (broadcast_evidence) into the network, remove transactions (remove_tx), and request arbitrary amounts of chain state.Filtering requests to the GET endpoint is straightforward: A reverse proxy like nginx can easily filter methods by URL path. Filtering POST requests takes a bit more work, but can be managed with a filter program that speaks FastCGI and parses JSON-RPC request bodies.
Filtering the websocket interface requires a dedicated proxy implementation. Although nginx can reverse-proxy websockets, it does not support filtering websocket traffic via FastCGI. The operator would need to either implement a custom nginx extension module or build and run a standalone proxy that implements websocket and filters each session. Apart from the work, this also makes the system even more resource intensive, as well as introducing yet another connection that could potentially time out or stall on full buffers.
Even for the simple case of restricting access to only event subscription, there is no easy solution currently: Once a caller has access to the websocket endpoint, it has complete access to the RPC service.
Issue 2: Inconvenient client API
The subscription interface has some inconvenient features for the client as well as the server. These include:
-
Non-standard protocol: The RPC service is mostly JSON-RPC 2.0, but the subscription interface diverges from the standard.
In a standard JSON-RPC 2.0 call, the client initiates a request to the server with a unique ID, and the server concludes the call by sending a reply for that ID. The
subscribeimplementation, however, sends multiple responses to the client's request:-
The client sends
subscribewith some IDxand the desired query -
The server responds with ID
xand an empty confirmation response. -
The server then (repeatedly) sends event result responses with ID
x, one for each item with a matching event.
Standard JSON-RPC clients will reject the subsequent replies, as they announce a request ID (
x) that is already complete. This means a caller has to implement Tenderdash-specific handling for these responses.Moreover, the result format is different between the initial confirmation and the subsequent responses. This means a caller has to implement special logic for decoding the first response versus the subsequent ones.
-
-
No way to detect data loss: The subscriber connection can be terminated for many reasons. Even ignoring ordinary network issues (e.g., packet loss):
-
The server will drop messages and/or close the websocket if its write buffer fills, or if the queue of undelivered matching events is not drained fast enough. The client has no way to discover that messages were dropped even if the connection remains open.
-
Either the client or the server may close the websocket if the websocket PING and PONG exchanges are not handled correctly, or frequently enough. Even if correctly implemented, this may fail if the system is under high load and cannot service those control messages in a timely manner.
When the connection is terminated, the server drops all the subscriptions for that client (as if it had called
unsubscribe_all). Even if the client reconnects, any events that were published during the period between the disconnect and re-connect and re-subscription will be silently lost, and the client has no way to discover that it missed some relevant messages. -
-
No way to replay old events: Even if a client knew it had missed some events (due to a disconnection, for example), the API provides no way for the client to "play back" events it may have missed.
-
Large response sizes: Some event data can be quite large, and there can be substantial duplication across items. The API allows the client to select which events are reported, but has no way to control which parts of a matching event it wishes to receive.
This can be costly on the server (which has to marshal those data into JSON), the network, and the client (which has to unmarshal the result and then pick through for the components that are relevant to it).
Besides being inefficient, this also contributes to some of the persistent connection issues mentioned above, e.g., filling up the websocket write buffer and forcing the server to queue potentially several copies of a large value in memory.
-
Client identity is tied to network address: The Tenderdash event API identifies each subscriber by a (Client ID, Query) pair. In the RPC service, the query is provided by the client, but the client ID is set to the TCP address of the client (typically "host:port" or "ip:port").
This means that even if the server did not drop subscriptions immediately when the websocket connection is closed, a client may not be able to reattach to its existing subscription. Dialing a new connection is likely to result in a different port (and, depending on their own proxy setup, possibly a different public IP).
In isolation, this problem would be easy to work around with a new subscription parameter, but it would require several other changes to the handling of event subscriptions for that workaround to become useful.
Decision
To address the described problems, we will:
-
Introduce a new API for event subscription to the Tenderdash RPC service. The proposed API is described in Detailed Design below.
-
The upstream proposal targeted the v0.36 release, during which the current ("streaming") API will remain available as-is, but deprecated.
-
The streaming API will be entirely removed in release v0.37, which will require all users of event subscription to switch to the new API.
Point for discussion: Given that ABCI++ and PBTS are the main priorities for v0.36, it would be fine to slip the first phase of this work to v0.37. Unless there is a time problem, however, the proposed design does not disrupt the work on ABCI++ or PBTS, and will not increase the scope of breaking changes. Therefore the plan is to begin in v0.36 and slip only if necessary.
Detailed Design
Design Goals
Specific goals of this design include:
-
Remove the need for a persistent connection to each subscription client. Subscribers should use the same HTTP request flow for event subscription requests as for other RPC calls.
-
The server retains minimal state (possibly none) per-subscriber. In particular:
- The server does not buffer unconsumed writes nor queue undelivered events on a per-client basis.
- A client that stalls or goes idle does not cost the server any resources.
- Any event data that is buffered or stored is shared among all subscribers, and is not duplicated per client.
-
Slow clients have no impact (or minimal impact) on the rate of progress of the consensus algorithm, beyond the ambient overhead of servicing individual RPC requests.
-
Clients can tell when they have missed events matching their subscription, within some reasonable (configurable) window of time, and can "replay" events within that window to catch up.
-
Nice to have: It should be easy to use the event subscription API from existing standard tools and libraries, including command-line use for testing and experimentation.
Definitions
-
The event stream of a node is a single, time-ordered, heterogeneous stream of event items.
-
Each event item comprises an event datum (for example, block header metadata for a new-block event), and zero or more optional events.
-
An event means the ABCI
Eventdata type, which comprises a string type and zero or more string key-value event attributes.The use of the new terms "event item" and "event datum" is to avert confusion between the values that are published to the event bus (what we call here "event items") and the ABCI
Eventdata type. -
The node assigns each event item a unique identifier string called a cursor. A cursor must be unique among all events published by a single node, but it is not required to be unique globally across nodes.
Cursors are time-ordered so that given event items A and B, if A was published before B, then cursor(A) < cursor(B) in lexicographic order.
A minimum viable cursor implementation is a tuple consisting of a timestamp and a sequence number (e.g.,
16CCC798FB5F4670-0123). However, it may also be useful to append basic type information to a cursor, to allow efficient filtering (e.g.,16CCC87E91869050-0091:BeginBlock).The initial implementation will use the minimum viable format.
Discussion
The node maintains an event log, a shared ordered record of the events published to its event bus within an operator-configurable time window. The initial implementation will store the event log in-memory, and the operator will be given two per-node configuration settings. Note, these names are provisional:
-
[rpc] event-log-window-size: A duration before the latest published event, during which the node will retain event items published. Setting this value to zero disables event subscription. -
[rpc] event-log-max-items: A maximum number of event items that the node will retain within the time window. If the number of items exceeds this value, the node discardes the oldest items in the window. Setting this value to zero means that no limit is imposed on the number of items.
The node will retain all events within the time window, provided they do not exceed the maximum number. These config parameters allow the operator to loosely regulate how much memory and storage the node allocates to the event log. The client can use the server reply to tell whether the events it wants are still available from the event log.
The event log is shared among all subscribers to the node.
Discussion point: Should events persist across node restarts?
The current event API does not persist events across restarts, so this new design does not either. Note, however, that we may "spill" older event data to disk as a way of controlling memory use. Such usage is ephemeral, however, and does not need to be tracked as node data (e.g., it could be temp files).
Query API
To retrieve event data, the client will call the (new) RPC method events.
The parameters of this method will correspond to the following Go types:
type EventParams struct {
// Optional filter spec. If nil or empty, all items are eligible.
Filter *Filter `json:"filter"`
// The maximum number of eligible results to return.
// If zero or negative, the server will report a default number.
MaxResults int `json:"max_results"`
// Return only items after this cursor. If empty, the limit is just
// before the the beginning of the event log.
After string `json:"after"`
// Return only items before this cursor. If empty, the limit is just
// after the head of the event log.
Before string `json:"before"`
// Wait for up to this long for events to be available.
WaitTime time.Duration `json:"wait_time"`
}
type Filter struct {
Query string `json:"query"`
}
Discussion point: The initial implementation will not cache filter queries for the client. If this turns out to be a performance issue in production, the service can keep a small shared cache of compiled queries. Given the improvements from #7319 et seq., this should not be necessary.
Discussion point: For the initial implementation, the new API will use the existing query language as-is. Future work may extend the Filter message with a more structured and/or expressive query surface, but that is beyond the scope of this design.
The semantics of the request are as follows: An item in the event log is eligible for a query if:
- It is newer than the
aftercursor (if set). - It is older than the
beforecursor (if set). - It matches the filter (if set).
Among the eligible items in the log, the server returns up to max_results of
the newest items, in reverse order of cursor. If max_results is unset the
server chooses a number to return, and will cap max_results at a sensible
limit.
The wait_time parameter is used to effect polling. If before is empty and
no items are available, the server will wait for up to wait_time for matching
items to arrive at the head of the log. If wait_time is zero or negative, the
server will wait for a default (positive) interval.
If before non-empty, wait_time is ignored: new results are only added to
the head of the log, so there is no need to wait. This allows the client to
poll for new data, and "page" backward through matching event items. This is
discussed in more detail below.
The server will set a sensible cap on the maximum wait_time, overriding
client-requested intervals longer than that.
A successful reply from the events request corresponds to the following Go
types:
type EventReply struct {
// The items matching the request parameters, from newest
// to oldest, if any were available within the timeout.
Items []*EventItem `json:"items"`
// This is true if there is at least one older matching item
// available in the log that was not returned.
More bool `json:"more"`
// The cursor of the oldest item in the log at the time of this reply,
// or "" if the log is empty.
Oldest string `json:"oldest"`
// The cursor of the newest item in the log at the time of this reply,
// or "" if the log is empty.
Newest string `json:"newest"`
}
type EventItem struct {
// The cursor of this item.
Cursor string `json:"cursor"`
// The encoded event data for this item.
// The type identifies the structure of the value.
Data struct {
Type string `json:"type"`
Value json.RawMessage `json:"value"`
} `json:"data"`
}
The oldest and newest fields of the reply report the cursors of the oldest
and newest items (of any kind) recorded in the event log at the time of the
reply, or are "" if the log is empty.
The data field contains the type-specific event datum. The datum carries any
ABCI events that may have been defined.
Discussion point: Based on issue #7273, I did not include a separate field in the response for the ABCI events, since it duplicates data already stored elsewhere in the event data.
The semantics of the reply are as follows:
-
If
itemsis non-empty:-
Items are ordered from newest to oldest.
-
If
moreis true, there is at least one additional, older item in the event log that was not returned (in excess ofmax_results).In this case the client can fetch the next page by setting
beforein a new request, to the cursor of the oldest item fetched (i.e., the last one initems). -
Otherwise (if
moreis false), all the matching results have been reported (pagination is complete). -
The first element of
itemsidentifies the newest item considered. Subsequent poll requests can setafterto this cursor to skip items that were already retrieved.
-
-
If
itemsis empty:-
If the
beforewas set in the request, there are no further eligible items for this query in the log (pagination is complete).This is just a safety case; the client can detect this without issuing another call by consulting the
morefield of the previous reply. -
If the
beforewas empty in the request, no eligible items were available before thewait_timeexpired. The client may poll again to wait for more event items.
-
A client can store cursor values to detect data loss and to recover from crashes and connectivity issues:
-
After a crash, the client requests events after the newest cursor it has seen. If the reply indicates that cursor is no longer in range, the client may (conservatively) conclude some event data may have been lost.
-
On the other hand, if it is in range, the client can then page back through the results that it missed, and then resume polling. As long as its recovery cursor does not age out before it finishes, the client can be sure it has all the relevant results.
Other Notes
-
The new API supports two general "modes" of operation:
-
In ordinary operation, clients will long-poll the head of the event log for new events matching their criteria (by setting a
wait_timeand nobefore). -
If there are more events than the client requested, or if the client needs to to read older events to recover from a stall or crash, clients will page backward through the event log (by setting
beforeandafter).
-
-
While the new API requires explicit polling by the client, it makes better use of the node's existing HTTP infrastructure (e.g., connection pools). Moreover, the direct implementation is easier to use from standard tools and client libraries for HTTP and JSON-RPC.
Explicit polling does shift the burden of timeliness to the client. That is arguably preferable, however, given that the RPC service is ancillary to the node's primary goal, viz., consensus. The details of polling can be easily hidden from client applications with simple libraries.
-
The format of a cursor is considered opaque to the client. Clients must not parse cursor values, but they may rely on their ordering properties.
-
To maintain the event log, the server must prune items outside the time window and in excess of the item limit.
The initial implementation will do this by checking the tail of the event log after each new item is published. If the number of items in the log exceeds the item limit, it will delete oldest items until the log is under the limit; then discard any older than the time window before the latest.
To minimize coordination interference between the publisher (the event bus) and the subcribers (the
eventsservice handlers), the event log will be stored as a persistent linear queue with shared structure (a cons list). A single reader-writer mutex will guard the "head" of the queue where new items are published:-
To publish a new item, the publisher acquires the write lock, conses a new item to the front of the existing queue, and replaces the head pointer with the new item.
-
To scan the queue, a reader acquires the read lock, captures the head pointer, and then releases the lock. The rest of its request can be served without holding a lock, since the queue structure will not change.
When a reader wants to wait, it will yield the lock and wait on a condition that is signaled when the publisher swings the pointer.
-
To prune the queue, the publisher (who is the sole writer) will track the queue length and the age of the oldest item separately. When the length and or age exceed the configured bounds, it will construct a new queue spine on the same items, discarding out-of-band values.
Pruning can be done while the publisher already holds the write lock, or could be done outside the lock entirely: Once the new queue is constructed, the lock can be re-acquired to swing the pointer. This costs some extra allocations for the cons cells, but avoids duplicating any event items. The pruning step is a simple linear scan down the first (up to) max-items elements of the queue, to find the breakpoint of age and length.
Moreover, the publisher can amortize the cost of pruning by item count, if necessary, by pruning length "more aggressively" than the configuration requires (e.g., reducing to 3/4 of the maximum rather than 1/1).
Before publication, readers see the existing queue head. After the publisher adds an item and prunes the queue, new readers see the new head; readers already scanning the old queue continue using their captured head.
-
Migration Plan
This design requires that clients eventually migrate to the new event subscription API, but provides a full release cycle with both APIs in place to make this burden more tractable. The migration strategy is broadly:
Phase 1: Release v0.36.
- Implement the new
eventsendpoint, keeping the existing methods as they are. - Update the Go clients to support the new
eventsendpoint, and handle polling. - Update the old endpoints to log annoyingly about their own deprecation.
- Write tutorials about how to migrate client usage.
At or shortly after release, we should proactively update the Cosmos SDK to use the new API, to remove a disincentive to upgrading.
Phase 2: Release v0.37
- During development, we should actively seek out any existing users of the streaming event subscription API and help them migrate.
- Possibly also: Spend some time writing clients for JS, Rust, et al.
- Release: Delete the old implementation and all the websocket support code.
Discussion point: Even though the plan is to keep the existing service, we might take the opportunity to restrict the websocket endpoint to only the event streaming service, removing the other endpoints. To minimize the disruption for users in the v0.36 cycle, I have decided not to do this for the first phase.
If we wind up pushing this design into v0.37, however, we should re-evaulate this partial turn-down of the websocket.
Future Work
-
This design does not immediately address the problem of allowing the client to control which data are reported back for event items. That concern is deferred to future work. However, it would be straightforward to extend the filter and/or the request parameters to allow more control.
-
The node currently stores a subset of event data (specifically the block and transaction events) for use in reindexing. While these data are redundant with the event log described in this document, they are not sufficient to cover event subscription, as they omit other event types.
In the future we should investigate consolidating or removing event data from the state store entirely. For now this issue is out of scope for purposes of updating the RPC API. We may be able to piggyback on the database unification plans (see RFC 001) to store the event log separately, so its pruning policy does not need to be tied to the block and state stores.
-
This design reuses the existing filter query language from the old API. In the future we may want to use a more structured and/or expressive query. The Filter object can be extended with more fields as needed to support this.
-
Some users have trouble communicating with the RPC service because of configuration problems like improperly-set CORS policies. While this design does not address those issues directly, we might want to revisit how we set policies in the RPC service to make it less susceptible to confusing errors caused by misconfiguration.
Consequences
- ✅ Reduces the number of transport options for RPC. Supports RFC 002.
- ️✅ Removes the primary non-standard use of JSON-RPC.
- ⛔️ Forces clients to migrate to a different API (eventually).
- ↕️ API requires clients to poll, but this reduces client state on the server.
- ↕️ We have to maintain both implementations for a whole release, but this gives clients time to migrate.
Alternative Approaches
The following alternative approaches were considered:
-
Leave it alone. Since existing tools mostly already work with the API as it stands today, we could leave it alone and do our best to improve its performance and reliability.
Based on many issues reported by users and node operators (e.g., #3380, #6439, #6729, #7247), the problems described here affect even the existing use that works. Investing further incremental effort in the existing API is unlikely to address these issues.
-
Design a better streaming API. Instead of polling, we might try to design a better "streaming" API for event subscription.
A significant advantage of switching away from streaming is to remove the need for persistent connections between the node and subscribers. A new streaming protocol design would lose that advantage, and would still need a way to let clients recover and replay.
This approach might look better if we decided to use a different protocol for event subscription, say gRPC instead of JSON-RPC. That choice, however, would be just as breaking for existing clients, for marginal benefit. Moreover, this option increases both the complexity and the resource cost on the node implementation.
Given that resource consumption and complexity are important considerations, this option was not chosen.
-
Defer to an external event broker. We might remove the entire event subscription infrastructure from the node, and define an optional interface to allow the node to publish all its events to an external event broker, such as Apache Kafka.
This has the advantage of greatly simplifying the node, but at a great cost to the node operator: To enable event subscription in this design, the operator has to stand up and maintain a separate process in communion with the node, and configuration changes would have to be coordinated across both.
Moreover, this approach would be highly disruptive to existing client use, and migration would probably require switching to third-party libraries. Despite the potential benefits for the node itself, the costs to operators and clients seems too large for this to be the best option.
Publishing to an external event broker might be a worthwhile future project, if there is any demand for it. That decision is out of scope for this design, as it interacts with the design of the indexer as well.
References
- RFC 006: Event Subscription
- Tenderdash RPC service
- Event query grammar
- RFC 6455: The WebSocket protocol
- JSON-RPC 2.0 Specification
- Nginx proxy server
- FastCGI
- RFC 001: Storage Engines & Database Layer
- RFC 002: Interprocess Communication in Tendermint
- Issues:
- rpc/client: test that client resubscribes upon disconnect (#3380)
- Too high memory usage when creating many events subscriptions (#6439)
- Tendermint emits events faster than clients can pull them (#6729)
- indexer: unbuffered event subscription slow down the consensus (#7247)
- rpc: remove duplication of events when querying (#7273)
ADR 076: Combine Spec and Tendermint Repositories
Historical upstream design record. Project and release names below identify the original implementation; they do not describe Tenderdash releases. See the current protocol specification for Tenderdash.
Changelog
- 2022-02-04: Initial Draft. (@tychoish)
Status
Accepted.
Context
While the specification for Tendermint was originally in the same repository as the Go implementation, at some point the specification was split from the core repository and maintained separately from the implementation. While this makes sense in promoting a conceptual separation of specification and implementation, in practice this separation was a premature optimization, apparently aimed at supporting alternate implementations of Tendermint.
The operational and documentary burden of maintaining a separate spec repo has not returned value to justify its cost. There are no active projects to develop alternate implementations of Tendermint based on the common specification, and having separate repositories creates an ongoing burden to coordinate versions, documentation, and releases.
Decision
The specification repository will be merged back into the Tendermint core repository.
Stakeholders including representatives from the maintainers of the spec, the Go implementation, and the Tendermint Rust library, agreed to merge the repositories in the Tendermint core dev meeting on 27 January 2022, including @williambanfield @cmwaters @creachadair and @thanethomson.
Alternative Approaches
The main alternative we considered was to keep separate repositories, and to introduce a coordinated versioning scheme between the two, so that users could figure out which spec versions go with which versions of the core implementation.
We decided against this on the grounds that it would further complicate the release process for both repositories, without mitigating any of the other existing issues.
Detailed Design
Clone and merge the master branch of the tendermint/spec repository
as a branch of the tendermint/tendermint, to ensure the commit history
of both repositories remains intact.
Implementation Instructions
-
Within the
tendermintrepository, execute the following commands to add a new branch with the history of the master branch ofspec:git remote add spec git@github.com:tendermint/spec.git git fetch spec git checkout -b spec-master spec/master mkdir spec git ls-tree -z --name-only HEAD | xargs -0 -I {} git mv {} subdir/ git commit -m "spec: organize specification prior to merge" git checkout -b spec-merge-mainline origin/master git merge --allow-unrelated-histories spec-masterThis merges the spec into the
tendermint/tendermintrepository as a normal branch. This commit can also be backported to the 0.35 branch, if needed. -
Migrate outstanding issues from
tendermint/specto thetendermint/tendermintrepository. -
In the specification repository, add redirect to the README and mark the repository as archived.
Consequences
Positive
Easier maintenance for the specification will obviate a number of complicated and annoying versioning problems, and will help prevent the possibility of the specification and the implementation drifting apart.
Additionally, co-locating the specification will help encourage cross-pollination and collaboration, between engineers focusing on the specification and the protocol and engineers focusing on the implementation.
Negative
Co-locating the spec and Go implementation has the potential effect of prioritizing the Go implementation with regards to the spec, and making it difficult to think about alternate implementations of the Tendermint algorithm. Although we may want to foster additional Tendermint implementations in the future, this isn't an active goal in our current roadmap, and not merging these repos doesn't change the fact that the Go implementation of Tendermint is already the primary implementation.
Neutral
N/A
References
- https://github.com/tendermint/spec
- https://github.com/tendermint/tendermint
ADR 077: Configurable Block Retention
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 2020-03-23: Initial draft (@erikgrinaker)
- 2020-03-25: Use local config for snapshot interval (@erikgrinaker)
- 2020-03-31: Use ABCI commit response for block retention hint
- 2020-04-02: Resolved open questions
- 2021-02-11: Migrate to tendermint repo (Originally RFC 001)
Author(s)
- Erik Grinaker (@erikgrinaker)
Context
Currently, all Tenderdash nodes contain the complete sequence of blocks from genesis up to some height (typically the latest chain height). This will no longer be true when the following features are released:
-
Block pruning: removes historical blocks and associated data (e.g. validator sets) up to some height, keeping only the most recent blocks.
-
State sync: bootstraps a new node by syncing state machine snapshots at a given height, but not historical blocks and associated data.
To maintain the integrity of the chain, the use of these features must be coordinated such that necessary historical blocks will not become unavailable or lost forever. In particular:
-
Some nodes should have complete block histories, for auditability, querying, and bootstrapping.
-
The majority of nodes should retain blocks longer than the Cosmos SDK unbonding period, for light client verification.
-
Some nodes must take and serve state sync snapshots with snapshot intervals less than the block retention periods, to allow new nodes to state sync and then replay blocks to catch up.
-
Applications may not persist their state on commit, and require block replay on restart.
-
Only a minority of nodes can be state synced within the unbonding period, for light client verification and to serve block histories for catch-up.
However, it is unclear if and how we should enforce this. It may not be possible to technically enforce all of these without knowing the state of the entire network, but it may also be unrealistic to expect this to be enforced entirely through social coordination. This is especially unfortunate since the consequences of misconfiguration can be permanent chain-wide data loss.
Proposal
Add a new field retain_height to the ABCI ResponseCommit message:
service ABCIApplication {
rpc Commit(RequestCommit) returns (ResponseCommit);
}
message RequestCommit {}
message ResponseCommit {
// reserve 1
bytes data = 2; // the Merkle root hash
uint64 retain_height = 3; // the oldest block height to retain
}
Upon ABCI Commit, which finalizes execution of a block in the state machine, Tenderdash removes all data for heights lower than retain_height. This allows the state machine to control block retention, which is preferable since only it can determine the significance of historical blocks. By default (i.e. with retain_height=0) all historical blocks are retained.
Removed data includes not only blocks, but also headers, commit info, consensus params, validator sets, and so on. In the first iteration this will be done synchronously, since the number of heights removed for each run is assumed to be small (often 1) in the typical case. It can be made asynchronous at a later time if this is shown to be necessary.
Since retain_height is dynamic, it is possible for it to refer to a height which has already been removed. For example, commit at height 100 may return retain_height=90 while commit at height 101 may return retain_height=80. This is allowed, and will be ignored - it is the application's responsibility to return appropriate values.
State sync will eventually support backfilling heights, via e.g. a snapshot metadata field backfill_height, but in the initial version it will have a fully truncated block history.
Cosmos SDK Example
As an example, we'll consider how the Cosmos SDK might make use of this. The specific details should be discussed in a separate SDK proposal.
The returned retain_height would be the lowest height that satisfies:
-
Unbonding time: the time interval in which validators can be economically punished for misbehavior. Blocks in this interval must be auditable e.g. by the light client.
-
IAVL snapshot interval: the block interval at which the underlying IAVL database is persisted to disk, e.g. every 10000 heights. Blocks since the last IAVL snapshot must be available for replay on application restart.
-
State sync snapshots: blocks since the oldest available snapshot must be available for state sync nodes to catch up (oldest because a node may be restoring an old snapshot while a new snapshot was taken).
-
Local config: archive nodes may want to retain more or all blocks, e.g. via a local config option
min-retain-blocks. There may also be a need to vary rentention for other nodes, e.g. sentry nodes which do not need historical blocks.
The retention height is the earliest height required by any of these constraints.
Status
Accepted
Consequences
Positive
-
Application-specified block retention allows the application to take all relevant factors into account and prevent necessary blocks from being accidentally removed.
-
Node operators can independently decide whether they want to provide complete block histories (if local configuration for this is provided) and snapshots.
Negative
-
Social coordination is required to run archival nodes, failure to do so may lead to permanent loss of historical blocks.
-
Social coordination is required to run snapshot nodes, failure to do so may lead to inability to run state sync, and inability to bootstrap new nodes at all if no archival nodes are online.
Neutral
-
Reduced block retention requires application changes, and cannot be controlled directly in Tenderdash.
-
Application-specified block retention may set a lower bound on disk space requirements for all nodes.
References
-
State sync ADR: https://github.com/tendermint/tendermint/blob/master/docs/architecture/adr-053-state-sync-prototype.md
-
State sync issue: https://github.com/tendermint/tendermint/issues/828
-
Block pruning issue: https://github.com/tendermint/tendermint/issues/3652
ADR 078: Non-Zero Genesis
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 2020-07-26: Initial draft (@erikgrinaker)
- 2020-07-28: Use weak chain linking, i.e.
predecessorfield (@erikgrinaker) - 2020-07-31: Drop chain linking (@erikgrinaker)
- 2020-08-03: Add
State.InitialHeight(@erikgrinaker) - 2021-02-11: Migrate to tendermint repo (Originally RFC 002)
Author(s)
- Erik Grinaker (@erikgrinaker)
Context
The recommended upgrade path for block protocol-breaking upgrades is currently to hard fork the
chain (see e.g. cosmoshub-3 upgrade.
This is done by halting all validators at a predetermined height, exporting the application
state via application-specific tooling, and creating an entirely new chain using the exported
application state.
As far as Tenderdash is concerned, the upgraded chain is a completely separate chain, with e.g. a new chain ID and genesis file. Notably, the new chain starts at height 1, and has none of the old chain's block history. This causes problems for integrators, e.g. coin exchanges and wallets, that assume a monotonically increasing height for a given blockchain. Users also find it confusing that a given height can now refer to distinct states depending on the chain version.
An ideal solution would be to always retain block backwards compatibility in such a way that chain history is never lost on upgrades. However, this may require a significant amount of engineering work that is not viable for the planned Stargate release (Tendermint 0.34), and may prove too restrictive for future development.
As a first step, allowing the new chain to start from an initial height specified in the genesis file would at least provide monotonically increasing heights. There was a proposal to include the last block header of the previous chain as well, but since the genesis file is not verified and hashed (only specific fields are) this would not be trustworthy.
External tooling will be required to map historical heights onto e.g. archive nodes that contain blocks from previous chain version. Tenderdash will not include any such functionality.
Proposal
Tenderdash will allow chains to start from an arbitrary initial height:
-
A new field
initial_heightis added to the genesis file, defaulting to1. It can be set to any non-negative integer, and0is considered equivalent to1. -
A new field
InitialHeightis added to the ABCIRequestInitChainmessage, with the same value and semantics as the genesis field. -
A new field
InitialHeightis added to thestate.Statestruct, where0is considered invalid. Including the field here simplifies implementation, since the genesis value does not have to be propagated throughout the code base separately, but it is not strictly necessary.
ABCI applications may have to be updated to handle arbitrary initial heights, otherwise the initial block may fail.
Status
Accepted
Consequences
Positive
- Heights can be unique throughout the history of a "logical" chain, across hard fork upgrades.
Negative
-
Upgrades still cause loss of block history.
-
Integrators will have to map height ranges to specific archive nodes/networks to query history.
Neutral
- There is no explicit link to the last block of the previous chain.
References
ADR 079: Ed25519 Verification
Historical upstream design record. Project and release names below identify the original implementation; they do not describe Tenderdash releases. See the current protocol specification for Tenderdash.
Changelog
- 2020-08-21: Initial RFC
- 2021-02-11: Migrate RFC to tendermint repo (Originally RFC 003)
Author(s)
- Marko (@marbar3778)
Context
Ed25519 keys are the only supported key types for Tendermint validators currently. Tendermint-Go wraps the ed25519 key implementation from the go standard library. As more clients are implemented to communicate with the canonical Tendermint implementation (Tendermint-Go) different implementations of ed25519 will be used. Due to RFC 8032 not guaranteeing implementation compatibility, Tendermint clients must to come to an agreement of how to guarantee implementation compatibility. Zcash has multiple implementations of their client and have identified this as a problem as well. The team at Zcash has made a proposal to address this issue, Zcash improvement proposal 215.
Proposal
-
Tendermint-Go would adopt hdevalence/ed25519consensus.
- This library is implements
ed25519.Verify()in accordance to zip-215. Tendermint-go will continue to usecrypto/ed25519for signing and key generation.
- This library is implements
-
Tendermint-rs would adopt ed25519-zebra
- related issue
Signature verification is one of the major bottlenecks of Tendermint-go, batch verification can not be used unless it has the same consensus rules, ZIP 215 makes verification safe in consensus critical areas.
This change constitutes a breaking changes, therefore must be done in a major release. No changes to validator keys or operations will be needed for this change to be enabled.
This change has no impact on signature aggregation. To enable this signature aggregation Tendermint will have to use different signature schema (Schnorr, BLS, ...). Secondly, this change will enable safe batch verification for the Tendermint-Go client. Batch verification for the rust client is already supported in the library being used.
As part of the acceptance of this proposal it would be best to contract or discuss with a third party the process of conducting a security review of the go library.
Status
Proposed
Consequences
Positive
- Consistent signature verification across implementations
- Enable safe batch verification
Negative
Tendermint-Go
- Third_party dependency
- library has not gone through a security review.
- unclear maintenance schedule
- Fragmentation of the ed25519 key for the go implementation, verification is done using a third party library while the rest uses the go standard library
Neutral
References
It’s 255:19AM. Do you know what your validation criteria are?
ADR 080: ReverseSync - fetching historical data
Changelog
- 2021-02-11: Migrate to tendermint repo (Originally RFC 005)
- 2021-04-19: Use P2P to gossip necessary data for reverse sync.
- 2021-03-03: Simplify proposal to the state sync case.
- 2021-02-17: Add notes on asynchronicity of processes.
- 2020-12-10: Rename backfill blocks to reverse sync.
- 2020-11-25: Initial draft.
Author(s)
- Callum Waters (@cmwaters)
Context
Two new features: Block pruning and State sync meant nodes no longer needed a complete history of the blockchain. This introduced some challenges of its own which were covered and subsequently tackled with RFC-001. The RFC allowed applications to set a block retention height; an upper bound on what blocks would be pruned. However nodes who state sync past this upper bound (which is necessary as snapshots must be saved within the trusting period for the assisting light client to verify) have no means of backfilling the blocks to meet the retention limit. This could be a problem as nodes who state sync and then eventually switch to consensus (or fast sync) may not have the block and validator history to verify evidence causing them to panic if they see 2/3 commit on what the node believes to be an invalid block.
Thus, this RFC sets out to instil a minimum block history invariant amongst honest nodes.
Proposal
A backfill mechanism can simply be defined as an algorithm for fetching, verifying and storing, headers and validator sets of a height prior to the current base of the node's blockchain. In matching the terminology used for other data retrieving protocols (i.e. fast sync and state sync), we call this method ReverseSync.
We will define the mechanism in four sections:
- Usage
- Design
- Verification
- Termination
Usage
For now, we focus purely on the case of a state syncing node, whom after
syncing to a height will need to verify historical data in order to be capable
of processing new blocks. We can denote the earliest height that the node will
need to verify and store in order to be able to verify any evidence that might
arise as the max_historical_height/time. Both height and time are necessary
as this maps to the BFT time used for evidence expiration. After acquiring
State, we calculate these parameters as:
max_historical_height = max(state.InitialHeight, state.LastBlockHeight - state.ConsensusParams.EvidenceAgeHeight)
max_historical_time = max(GenesisTime, state.LastBlockTime.Sub(state.ConsensusParams.EvidenceAgeTime))
Before starting either fast sync or consensus, we then run the following synchronous process:
func ReverseSync(max_historical_height int64, max_historical_time time.Time) error
Where we fetch and verify blocks until a block A where
A.Height <= max_historical_height and A.Time <= max_historical_time.
Upon successfully reverse syncing, a node can now safely continue. As this feature is only used as part of state sync, one can think of this as merely an extension to it.
In the future we may want to extend this functionality to allow nodes to fetch historical blocks for reasons of accountability or data accessibility.
Design
This section will provide a high level overview of some of the more important characteristics of the design, saving the more tedious details as an ADR.
P2P
Implementation of this RFC will require the addition of a new channel and two new messages.
message LightBlockRequest {
uint64 height = 1;
}
message LightBlockResponse {
Header header = 1;
Commit commit = 2;
ValidatorSet validator_set = 3;
}
The P2P path may also enable P2P networked light clients and a state sync that also doesn't need to rely on RPC.
Verification
ReverseSync is used to fetch the following data structures:
HeaderCommitValidatorSet
Nodes will also need to be able to verify these. This can be achieved by first retrieving the header at the base height from the block store. From this trusted header, the node hashes each of the three data structures and checks that they are correct.
-
The trusted header's last block ID matches the hash of the new header
header[height].LastBlockID == hash(header[height-1]) -
The trusted header's last commit hash matches the hash of the new commit
header[height].LastCommitHash == hash(commit[height-1]) -
Given that the node now trusts the new header, check that the header's validator set hash matches the hash of the validator set
header[height-1].ValidatorsHash == hash(validatorSet[height-1])
Termination
ReverseSync draws a lot of parallels with fast sync. An important consideration for fast sync that also extends to ReverseSync is termination. ReverseSync will finish it's task when one of the following conditions have been met:
- It reaches a block
AwhereA.Height <= max_historical_heightandA.Time <= max_historical_time. - None of it's peers reports to have the block at the height below the processes current block.
- A global timeout.
This implies that we can't guarantee adequate history and thus the term "invariant" can't be used in the strictest sense. In the case that the first condition isn't met, the node will log an error and optimistically attempt to continue with either fast sync or consensus.
Alternative Solutions
The need for a minimum block history invariant stems purely from the need to validate evidence (although there may be some application relevant needs as well). Because of this, an alternative, could be to simply trust whatever the 2/3+ majority has agreed upon and in the case where a node is at the head of the blockchain, you simply abstain from voting.
As it stands, if 2/3+ vote on evidence you can't verify, in the same manner if 2/3+ vote on a header that a node sees as invalid (perhaps due to a different app hash), the node will halt.
Another alternative is the method with which the relevant data is retrieved. Instead of introducing new messages to the P2P layer, RPC could have been used instead.
The aforementioned data is already available via the following RPC endpoints:
/commit for Header's' and /validators for ValidatorSet's'. It was
decided predominantly due to the instability of the current RPC infrastructure
that P2P be used instead.
Status
Proposed
Consequences
Positive
- Ensures a minimum block history invariant for honest nodes. This will allow nodes to verify evidence.
Negative
- Statesync will be slower as more processing is required.
Neutral
- By having validator sets served through p2p, this would make it easier to extend p2p support to light clients and state sync.
- In the future, it may also be possible to extend this feature to allow for nodes to freely fetch and verify prior blocks
References
ADR 081: Protocol Buffers Management
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 2022-02-28: First draft
Status
Accepted
Context
At present, we manage the Protocol Buffers schema files ("protos") that define
our wire-level data formats within the Tenderdash repository itself (see the
proto directory). Recently, we have been making use of Buf,
both locally and in CI, in order to generate Go stubs, and lint and check
.proto files for breaking changes.
The version of Buf used at the time of this decision was v1beta1, and it was
discussed in #7975 and in weekly calls as to whether we should upgrade to
v1 and harmonize our approach with that used by the Cosmos SDK. The team
managing the Cosmos SDK was primarily interested in having our protos versioned
and easily accessible from the Buf registry.
The three main sets of stakeholders for the .proto files and their needs, as
currently understood, are as follows.
- Tenderdash needs Go code generated from
.protofiles. - Consumers of Tenderdash's
.protofiles, specifically projects that want to interoperate with Tenderdash and need to generate code for their own programming language, want to be able to access these files in a reliable and efficient way. - The Tenderdash Core team wants to provide stable interfaces that are as easy
as possible to maintain, on which consumers can depend, and to be able to
notify those consumers promptly when those interfaces change. To this end, we
want to:
- Prevent any breaking changes from being introduced in minor/patch releases of Tenderdash. Only major version updates should be able to contain breaking interface changes.
- Prevent generated code from diverging from the Protobuf schema files.
There was also discussion surrounding the notion of automated documentation
generation and hosting, but it is not clear at this time whether this would be
that valuable to any of our stakeholders. What will, of course, be valuable at
minimum would be better documentation (in comments) of the .proto files
themselves.
Alternative Approaches
Meeting stakeholders' needs
- Go stub generation from protos. We could use:
- Notification of breaking changes:
- Buf in CI for all pull requests to release branches only (and not on
master). - Buf in CI on every pull request to every branch (this was the case at the time of this decision, and the team decided that the signal-to-noise ratio for this approach was too low to be of value).
- Buf in CI for all pull requests to release branches only (and not on
.protolinting:- Buf in CI on every pull request
.protoformatting:- clang-format locally and a clang-format GitHub Action in CI to check that files are formatted properly on every pull request.
- Sharing of
.protofiles in a versioned, reliable manner:- Consumers could simply clone the Tenderdash repository, check out a
specific commit, tag or branch and manually copy out all of the
.protofiles they need. This requires no effort from the Tenderdash Core team and will continue to be an option for consumers. The drawback of this approach is that it requires manual coding/scripting to implement and is brittle in the face of bigger changes. - Uploading our
.protofiles to Buf's registry on every release. This is by far the most seamless for consumers of our.protofiles, but requires the dependency on Buf. This has the additional benefit that the Buf registry will automatically generate and host documentation for these protos. - We could create a process that, upon release, creates a
.zipfile containing our.protofiles.
- Consumers could simply clone the Tenderdash repository, check out a
specific commit, tag or branch and manually copy out all of the
Popular alternatives to Buf
Prototool was not considered as it appears deprecated, and the ecosystem seems to be converging on Buf at this time.
Tooling complexity
The more tools we have in our build/CI processes, the more complex and fragile repository/CI management becomes, and the longer it takes to onboard new team members. Maintainability is a core concern here.
Buf sustainability and costs
One of the primary considerations regarding the usage of Buf is whether, for example, access to its registry will eventually become a paid-for/subscription-based service and whether this is valuable enough for us and the ecosystem to pay for such a service. At this time, it appears as though Buf will never charge for hosting open source projects' protos.
Another consideration was Buf's sustainability as a project - what happens when their resources run out? Will there be a strong and broad enough open source community to continue maintaining it?
Local Buf usage options
Local usage of Buf (i.e. not in CI) can be accomplished in two ways:
- Installing the relevant tools individually.
- By way of its Docker image.
Local installation of Buf requires developers to manually keep their toolchains
up-to-date. The Docker option comes with a number of complexities, including
how the file system permissions of code generated by a Docker container differ
between platforms (e.g. on Linux, Buf-generated code ends up being owned by
root).
The trouble with the Docker-based approach is that we make use of the
gogoprotobuf plugin for protoc. Continuing to use the Docker-based approach
to using Buf will mean that we will have to continue building our own custom
Docker image with embedded gogoprotobuf.
Along these lines, we could eventually consider coming up with a Nix- or redo-based approach to developer tooling to ensure tooling consistency across the team and for anyone who wants to be able to contribute to Tenderdash.
Decision
- We will adopt Buf for now for proto generation, linting, breakage checking and its registry (mainly in CI, with optional usage locally).
- Failing CI when checking for breaking changes in
.protofiles will only happen when performing minor/patch releases. - Local tooling will be favored over Docker-based tooling.
Detailed Design
We currently aim to:
- Update to Buf
v1to facilitate linting, breakage checking and uploading to the Buf registry. - Configure CI appropriately for proto management:
- Uploading protos to the Buf registry on every release (e.g. the approach used by the Cosmos SDK).
- Linting on every pull request (e.g. the approach used by the Cosmos SDK). The linter passing should be considered a requirement for accepting PRs.
- Checking for breaking changes in minor/patch version releases and failing CI accordingly - see #8003.
- Add clang-format GitHub Action to check
.protofile formatting. Format checking should be considered a requirement for accepting PRs.
- Update the Tenderdash
Makefileto primarily facilitate local Protobuf stub generation, linting, formatting and breaking change checking. More specifically:- This includes removing the dependency on Docker and introducing the dependency on local toolchain installation. CI-based equivalents, where relevant, will rely on specific GitHub Actions instead of the Makefile.
- Go code generation will rely on
protocdirectly.
Consequences
Positive
- We will still offer Go stub generation, proto linting and breakage checking.
- Breakage checking will only happen on minor/patch releases to increase the signal-to-noise ratio in CI.
- Versioned protos will be made available via Buf's registry upon every release.
Negative
- Developers/contributors will need to install the relevant Protocol
Buffers-related tooling (Buf, gogoprotobuf, clang-format) locally in order to
build, lint, format and check
.protofiles for breaking changes.
Neutral
References
- Protocol Buffers
- Buf
- #7975
- protoc - The Protocol Buffers compiler
ADR D001: Inter validator set messaging
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 2021-09-29: Initial version of the document
- 2021-10-05: Clarification of the document based on peer review
- 2021-10-22: Updated document based on implementation
- 2022-01-05: Review after implementation is done
Table of contents
- Context
- Alternative Approaches
- Decision
- Detailed Design
- What are the user requirements?
- What systems will be affected?
- What new data structures are needed, what data structures need changes?
- What new APIs will be needed, what APIs will change?
- What are the efficiency considerations (time/space)?
- What are the expected access patterns (load/throughput)?
- Are there any logging, monitoring, or observability needs?
- Are there any security considerations?
- Are there any privacy considerations?
- How will the changes be tested?
- How will the changes be broken up for ease of review?
- Will these changes require a breaking (major) release?
- Does this change require coordination with the SDK or any other software?
- Future work
- Status
- Consequences
- References
Context
Definitions
For the needs of this document, we define the following types of nodes:
- Validator nodes, which participate in the consensus protocol, verify and sign blocks,
- Full nodes, which verify and execute blocks. Full nodes verify the recovered threshold signature of the block commit message against the current validator set's quorum threshold public key.
Note: there are also light nodes, but they are not relevant for this discussion.
A Validator Set is a group of Validator nodes responsible for driving consensus at any given time. For each block, one of the validator nodes is a proposer. Each other validator votes on the validity of the block.
Only one Validator Set can be active at the given time. After a predefined number of blocks, the ABCI app initiates the Validator Set rotation to make another Validator Set active.
Problem statement
The consensus process requires direct communication between Validators, as Full nodes do not know the public keys of Validators and therefore can neither verify nor propagate any consensus protocol message individually signed by these Validators. It means that only Validators can propagate votes. However, Full nodes can propagate the final commit message, which instead contains the recovered threshold signature of the pre-commit voting round. Full nodes can perform this verification because they know the Validator Set's quorum threshold public key.
Each node (regardless of its type) randomly selects peer nodes to connect. There is no differentiation between Full and Validator nodes, and there is no method to ensure that each Validator is (directly or through other Validators) connected to some other Validators. In an extreme case, a Validator node can be connected only to Full nodes, effectively blocking all inbound and outbound block proposals and voting, hence excluding it from participation in the consensus. It is also possible that Validator nodes form "islands" where a few Validators are connected between themselves but then are separated from other Validator nodes by full nodes.
Alternative Approaches
Scenario 1: Trigger rotation from ABCI app passing addresses of all quorum members
During Validator Set rotation, each Validator node shall connect to a predefined number of other Validators that belong to the same Validator Set.
In the current implementation, the ABCI application manages the rotation of Validator Sets. When rotation is needed,
the application sends information about the new Validator Set in response to the EndBlock message.
EndBlock message shall be extended by adding a network address of each Validator node. Based on that, each
member of the active Validator Set will connect with a predefined set of Peer Validators belonging to the same
Validator Set. A list of Peer Validators will be determined using the algorithm described in
DIP 0006.
This solution introduces a risk of delayed propagation of new block due to limited connection slots between Full and Validator nodes. To mitigate this risk, we can implement the number of Full node connection slots independently from Validator node connection slots.
Scenario 2: Trigger rotation from ABCI app passing quorum id to rotate
This scenario is very similar to Scenario 1. However, instead of passing addresses of all Validators that are members of a Validator Set, only quorum ID is passed.
This solution requires Tenderdash to determine Validators addresses, effectively re-implementing logic already done in the ABCI App.
Scenario 3: Rotate on Tenderdash
ABCI application can send the whole master node list and quorum list to Tenderdash and potentially rotate on its side. However, it would split the responsibility for Validator Set rotation between multiple components. It would also require much more additional implementation. It would be better to keep the current Tenderdash design and concept being a logic-less consensus library and keep all business logic in the ABCI application.
Decision
After discussion, we will implement the solution described in Scenario 1. It will allow us to:
- reuse logic already implemented as part of ABCI App,
- keep Tenderdash a logic-less consensus library,
- keep single responsibility principle.
Detailed Design
What are the user requirements?
Having a network with much more Full nodes than Validators ensure each Validator Node can connect (directly or through another Validator) to any other Validator which is a member of the same active Validator Set.
What systems will be affected?
- Tenderdash
- ABCI App (Drive)
What new data structures are needed, what data structures need changes?
- New package
dashto group Tenderdash-specific code dashtypes.ValidatorAddressrepresenting an address of a validator received from the ABCI app- Introduced several internal helper data structures (
sortableValidator,sortedValidatorList,validatorMap)
What new APIs will be needed, what APIs will change?
ABCI Protocol
In the ABCI protocol, we add a network address of each member of the active Validator Set to the
ResponseEndBlock.ValidatorSetUpdate.ValidatorUpdate structure. The network address should be a URI:
tcp://<node_id>@<address>:<port>
where:
<node_id>- node ID (can be generated based on node P2P public key, see p2p.PubKeyToID)<address>- IP address of the validator node<port>- p2p port number of the validator node
The ValidatorUpdate structure looks as follows:
message ValidatorUpdate {
tendermint.crypto.PublicKey pub_key = 1 [(gogoproto.nullable) = true];
int64 power = 2;
bytes pro_tx_hash = 3;
string node_address = 4 [(gogoproto.nullable) = true]; // address of the Validator
}
If the ABCI app cannot determine <node_id>, it can leave it empty. In this case, Tenderdash will try to retrieve a public
key from its address book or remote host and derive node ID from it, as described below.
P2P Handshake
To retrieve node ID from a remote node, local node can establish a short-lived connection to the remote node to download its public key. This feature does not require any changes to the P2P protocol; it just uses existing capabilities.
A public key request starts with a standard flow Diffie-Hellman protocol, as implemented in conn.MakeSecretConnection(). Once ephemeral keys are exchanged between nodes, local node proceeds to shareAuthSignature(). However, instead of sending its own AuthSigMessage, it waits for remote node's auth signature to arrive. As the communication is asynchronous, the remote node sends its AuthSigMessage. That is when the local node closes the connection.
This algorithm introduces a potential node ID spoofing risk, discussed later.
What are the efficiency considerations (time/space)?
- This change will increase the amount of data sent by the ABCI application to Tenderdash. The connection between them is local. Therefore the change should not have any noticeable performance or bandwidth impact.
- Improved consensus performance thanks to a shorter and more predictable distance between each Validator in a Validator Set.
In the future, additional optimizations can further improve the performance, for example:
- use Unix sockets for communication between Tenderdash and ABCI App,
- use shared memory to speed up inter-process communication; for instance, ABCI should keep (signed) information about active validator-set in shared memory and send only a trigger to take them from there.
What are the expected access patterns (load/throughput)?
-
Each rotation of the active Validator Set will cause the ABCI application to send additional data to Tenderdash. The number of blocks between rotation events is a configuration parameter of the ABCI application.
-
Each Validator in the active validator set will establish a few more tcp connections.
-
Retrieving public keys for validators can be resource and time-intensive, as it establishes new connections to validators and performs a DH handshake. This issue is mitigated by: a. ensuring that this feature is used only when needed b. refactoring and simplification of the P2P protocol and source code used to retrieve public keys
Note that for the needs of Inter-Validator Set Messaging, retrieving public keys is executed only for
log2(n-1)Validators, wherenis size of active Validator Set.
Are there any logging, monitoring, or observability needs?
- Operator can determine a list of node peers, together with their type (Full, inactive Validator, active Validator) and connection status. This information can be part of debug logs.
Are there any security considerations?
-
This change makes connectivity between Validators more predictable. It can make it a bit easier for a malicious user to block communication for a given Validator node to make it unable to participate in the Validator Set. However, this risk is already present at the ABCI application level, so the change does not introduce new risks.
-
Missing node ID can be determined based on a public key, retrieved without proper validation. It cannot be guaranteed that it will not be spoofed.
That can allow an attacker who controls the network connectivity to perform a man-in-the-middle attack. As the attacker does not have the Validator's private keys, he/she cannot perform unauthorized operations. However, it is possible to intercept communication and isolate the attacked Validator from the quorum. In the future, additional validation of the Validator's private key based on proTxHash can help mitigate this risk.
If the attacker controls the network connectivity, he can still drop that traffic on the network level.
Are there any privacy considerations?
This change does not impact user privacy.
How will the changes be tested?
Testing strategy for this change involves:
- Reproduction of the problem statement.
- Implementation of unit tests for the new code
- Starting a long-living end-to-end cluster (at least a few days of operation) and monitoring of the outcome
- Deployment to the test net
How will the changes be broken up for ease of review?
This change consists of the following tasks:
- Enhancement of the ABCI protocol:
- Addition of new field on Tenderdash
- Update of end-to-end tests to reflect ABCI protocol changes
- Implementation of ABCI protocol changes inside the ABCI application
- Tenderdash reactions on Validator Set rotation events:
- Storing and updating Validator Set members network addresses
- Establishing connections within Validator Sets
- Unit tests that emulate and validate Inter Validator Set implementation
Will these changes require a breaking (major) release?
As the change introduces a new required field into the ABCI app, it requires implementation on the ABCI App side. However, this feature shall be backward-compatible.
Does this change require coordination with the SDK or any other software?
The change requires coordination with the ABCI application.
Future work
- Implement additional validation of the Validator's private key based on proTxHash.
- Extend service data in Deterministic Masternode Lists (DIP-3) to keep node ID, p2p port, and other platform information. It will allow us to remove the handshake process described in "Security considerations" above.
Status
{Deprecated|Proposed|Accepted|Declined}
Accepted
Consequences
Positive
- Improved stability of active Validator Set operations.
- Quicker Voting rounds.
Negative
- Additional complexity of ABCI protocol
- Additional changes compared to the upstream Tendermint project make backporting harder.
Neutral
References
- Dash Core Group Release Announcement: Dash Platform v0.20 on Testnet
- DIP 0006: Long Living Masternode Quorums (LLMQ)
- DIP 0003: Deterministic Masternode Lists (DML)
Service lifecycle core
BaseService.Start gives OnStart an attempt-scoped child context. Parent
cancellation and manual Stop both cancel it. Go(ctx, fn) registers workers
using that context or a derived context; registration is synchronized with
shutdown and rejected for a different attempt or after shutdown begins.
Shutdown closes admission, marks IsRunning false, cancels the context and
calls OnStop. It then joins registered workers, calls optional OnDrain, and
releases Wait. Stopping() signals cancellation separately from completion.
Stop invokes OnStop synchronously but does not join registered workers.
Concurrent or recursive calls to Stop return immediately. During OnStart,
Stop cancels immediately and defers the shutdown hook until successful startup
returns; Wait still joins the attempt.
Hooks run outside the lifecycle mutex. OnStop must unblock workers without
waiting for its own registered work. Workers may call Stop; neither hooks nor
workers may call their own service's Wait. Release resources shared with
workers in OnDrain, or after Stop followed by Wait.
IsRunning is false during startup and shutdown. Wait joins the current
attempt, including startup and cleanup; it returns immediately before startup.
Failed OnStart cancels and joins its workers before retry becomes possible.
The implementation must roll back acquired resources: failed attempts do not
invoke OnStop or OnDrain. Successfully stopped services cannot restart.
Composite retries also require recreating children already successfully stopped.
First migration stage
Consensus State registers its receive loop and joins its queue fan-in and WAL before returning. The timeout ticker, WAL and autofile group use shared worker tracking. AutoFile explicitly joins its timer/signal worker. WAL ownership lasts until the receive loop's final write. Consensus handoff admission is also joined, so a late blocksync handoff cannot start State after reactor shutdown. The caller may cancel its wait after admission, but the reactor still owns and joins the handoff. Startup and the resulting State use the reactor's lifetime.
Node waits for reactors before closing event sinks and stores. The blocked receive-loop regression and original node teardown tests verify this path without using leaktest as an extra production-shutdown barrier.
This is a partial migration. Plain goroutines are not tracked automatically. RPC/WebSocket handlers, mempool rechecks, reactor gossip and other service workers retain their existing ownership mechanisms. This change does not claim that Node.Wait joins every helper or introduce the broader durable-finalization and dependency-lifetime changes from #1515.
Three compatibility bridges preserve parent-context lifetimes: blocksync application during handoff, connection error callbacks after I/O shutdown, and an admitted consensus commit during direct State.Stop. Parent cancellation still aborts these operations. In particular, reactor/node shutdown can interrupt finalization and require WAL recovery. Their existing ownership mechanisms remain in place; this does not detach application work from node shutdown.
Cancellation compatibility audit
Cancellation precedes OnStop for every BaseService implementation, not only the managed-worker migrations. The in-tree hook/callback audit covers these groups:
| Implementations | Cancellation and cleanup policy |
|---|---|
| Consensus State, reactor, ticker, WAL; autofile Group | Managed work is joined; commit uses the State parent context; file ownership ends with explicit Close. |
| Blocksync reactor and Synchronizer | Handoff preserves application work across Synchronizer.Stop; stopping its parent reactor may cancel it. Reactor OnStop stops and waits for the Synchronizer, so reactor Wait covers in-flight application. |
| P2P Router and MConnection | Connection callbacks retain the parent. Router removes peer bookkeeping even with a canceled context; final PeerStatusDown broadcasts are best effort. |
| PEX, mempool, statesync and evidence reactors | Processing uses the work context; stop hooks close resources/dispatchers or require no extra signal. Existing plain-worker ownership is unchanged. |
| EventBus, pubsub Server and indexer Service | Event processing is cancellation-driven; indexer closes sinks in its existing hook. No guarantee of a live cleanup callback context. |
| ABCI local, socket, gRPC and routed clients; proxy; socket/gRPC servers | Stop hooks close transports, drain requests or stop children without requiring a live work context. |
| SignerServer, signer listener/dialer endpoints | Hooks close endpoints/listeners; dial/retry work observes cancellation. |
| ValidatorConnExecutor and light RPC Client | Unsubscribe uses a separate bounded cleanup context; client hooks invoke owned closers. |
| Node and seed node | Parent cancellation stops children; hooks wait for existing service completion before closing owned stores. |
This is a cancellation compatibility audit, not a claim that unmigrated plain workers are all joined. Subscribers must stop with their owner, without relying on a final router broadcast. Avoid detaching broadcasts from cancellation: unresponsive subscribers could otherwise block shutdown.
Group rotation reuses its AutoFile and closes the current descriptor before renaming; it does not create a worker or signal registration per rotation. The owner must Close the group, including when it never started, to join the head's single timer/signal worker. Context cancellation alone does not release the head.
CLI shutdown deadline
The start command cancels the node on SIGINT or SIGTERM and waits for
Node.Wait before returning. --shutdown-timeout controls this wait (default
30 seconds; must be positive). Set the deployment supervisor's termination grace
period longer than this timeout to allow Tenderdash to report incomplete cleanup.
If the deadline expires, the command returns an error and the Tenderdash binary
exits nonzero. This does not skip reactor joins or close stores while workers
remain active. An embedded command caller must also terminate on this error;
cleanup may still be running. The deadline bounds shutdown after startup returns,
not a blocked startup hook.
RFC 006: Event Subscription
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 30-Oct-2021: Initial draft (@creachadair)
Abstract
The Tenderdash consensus node allows clients to subscribe to its event stream via methods on its RPC service. The ability to view the event stream is valuable for clients, but the current implementation has some deficiencies that make it difficult for some clients to use effectively. This RFC documents these issues and discusses possible approaches to solving them.
Background
A running Tenderdash consensus node exports a JSON-RPC service
that provides a large set of methods for inspecting and
interacting with the node. One important cluster of these methods are the
subscribe, unsubscribe, and unsubscribe_all methods, which permit clients
to subscribe to a filtered stream of the events generated by the node
as it runs.
Unlike the other methods of the service, the methods in the "event
subscription" cluster are not accessible via ordinary HTTP GET or POST
requests, but require upgrading the HTTP connection to a
websocket. This is necessary because the subscribe request needs a
persistent channel to deliver results back to the client, and an ordinary HTTP
connection does not reliably persist across multiple requests. Since these
methods do not work properly without a persistent channel, they are only
exported via a websocket connection, and are not routed for plain HTTP.
Discussion
There are some operational problems with the current implementation of event subscription in the RPC service:
-
Event delivery is not valid JSON-RPC. When a client issues a
subscriberequest, the server replies (correctly) with an initial empty acknowledgement ({}). After that, each matching event is delivered "unsolicited" (without another request from the client), as a separate response object with the same ID as the initial request.This matters because it means a standard JSON-RPC client library can't interact correctly with the event subscription mechanism.
Even for clients that can handle unsolicited values pushed by the server, these responses are invalid: They have an ID, so they cannot be treated as notifications; but the ID corresponds to a request that was already completed. In practice, this means that general-purpose JSON-RPC libraries cannot use this method correctly -- it requires a custom client.
The Go RPC client from the Tenderdash core can support this case, but clients in other languages have no easy solution.
This is the cause of issue #2949.
-
Subscriptions are terminated by disconnection. When the connection to the client is interrupted, the subscription is silently dropped.
This is a reasonable behavior, but it matters because a client whose subscription is dropped gets no useful error feedback, just a closed connection. Should they try again? Is the node overloaded? Was the client too slow? Did the caller forget to respond to pings? Debugging these kinds of failures is unnecessarily painful.
Websockets compound this, because websocket connections time out if no traffic is seen for a while, and keeping them alive requires active cooperation between the client and server. With a plain TCP socket, liveness is handled transparently by the keepalive mechanism. On a websocket, however, one side has to occasionally send a PING (if the connection is otherwise idle). The other side must return a matching PONG in time, or the connection is dropped. Apart from being tedious, this is highly susceptible to CPU load.
The Tenderdash Go implementation automatically sends and responds to pings. Clients in other languages (or not wanting to use the Tenderdash libraries) need to handle it explicitly. This burdens the client for no practical benefit: A subscriber has no information about when matching events may be available, so it shouldn't have to participate in keeping the connection alive.
-
Mismatched load profiles. Most of the RPC service is mainly important for low-volume local use, either by the application the node serves (e.g., the ABCI methods) or by the node operator (e.g., the info methods). Event subscription is important for remote clients, and may represent a much higher volume of traffic.
This matters because both are using the same JSON-RPC mechanism. For low-volume local use, the ergonomics of JSON-RPC are a good fit: It's easy to issue queries from the command line (e.g., using
curl) or to write scripts that call the RPC methods to monitor the running node.For high-volume remote use, JSON-RPC is not such a good fit: Even leaving aside the non-standard delivery protocol mentioned above, the time and memory cost of encoding event data matters for the stability of the node when there can be potentially hundreds of subscribers. Moreover, a subscription is long-lived compared to most RPC methods, in that it may persist as long the node is active.
-
Mismatched security profiles. The RPC service exports several methods that should not be open to arbitrary remote callers, both for correctness reasons (e.g.,
remove_txandbroadcast_tx_*) and for operational stability reasons (e.g.,tx_search). A node may still need to expose events, however, to support UI tools.This matters, because all the methods share the same network endpoint. While it is possible to block the top-level GET and POST handlers with a proxy, exposing the
/websockethandler exposes not only the event subscription methods, but the rest of the service as well.
Possible Improvements
There are several things we could do to improve the experience of developers who need to subscribe to events from the consensus node. These are not all mutually exclusive.
-
Split event subscription into a separate service. Instead of exposing event subscription on the same endpoint as the rest of the RPC service, dedicate a separate endpoint on the node for only event subscription. The rest of the RPC services (sans events) would remain as-is.
This would make it easy to disable or firewall outside access to sensitive RPC methods, without blocking access to event subscription (and vice versa). This is probably worth doing, even if we don't take any of the other steps described here.
-
Use a different protocol for event subscription. There are various ways we could approach this, depending how much we're willing to shake up the current API. Here are sketches of a few options:
-
Keep the websocket, but rework the API to be more JSON-RPC compliant, perhaps by converting event delivery into notifications. This is less up-front change for existing clients, but retains all of the existing implementation complexity, and doesn't contribute much toward more serious performance and UX improvements later.
-
Switch from websocket to plain HTTP, and rework the subscription API to use a more conventional request/response pattern instead of streaming. This is a little more up-front work for existing clients, but leverages better library support for clients not written in Go.
The protocol would become more chatty, but we could mitigate that with batching, and in return we would get more control over what to do about slow clients: Instead of simply silently dropping them, as we do now, we could drop messages and signal the client that they missed some data ("M dropped messages since your last poll").
This option is probably the best balance between work, API change, and benefit, and has a nice incidental effect that it would be easier to debug subscriptions from the command-line, like the other RPC methods.
-
Switch to gRPC: Preserves a persistent connection and gives us a more efficient binary wire format (protobuf), at the cost of much more work for clients and harder debugging. This may be the best option if performance and server load are our top concerns.
Given that we are currently using JSON-RPC, however, I'm not convinced the costs of encoding and sending messages on the event subscription channel are the limiting factor on subscription efficiency, however.
-
-
Delegate event subscriptions to a proxy. Give responsibility for managing event subscription to a proxy that runs separately from the node, and switch the node to push events to the proxy (like a webhook) instead of serving subscribers directly. This is more work for the operator (another process to configure and run) but may scale better for big networks.
I mention this option for completeness, but making this change would be a fairly substantial project. If we want to consider shifting responsibility for event subscription outside the node anyway, we should probably be more systematic about it. For a more principled approach, see point (4) below.
-
Move event subscription downstream of indexing. We are already planning to give applications more control over event indexing. By extension, we might allow the application to also control how events are filtered, queried, and subscribed. Having the application control these concerns, rather than the node, might make life easier for developers building UI and tools for that application.
This is a much larger change, so I don't think it is likely to be practical in the near-term, but it's worth considering as a broader option. Some of the existing code for filtering and selection could be made more reusable, so applications would not need to reinvent everything.
References
- Tenderdash RPC service
- Tenderdash RPC routes
- Discussion of the event system
- Discussion about RPC transport options (from RFC 002)
- RFC 6455: The websocket protocol
- JSON-RPC 2.0 Specification
RFC 007 : Deterministic Proto Byte Serialization
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 09-Dec-2021: Initial draft (@williambanfield).
Abstract
This document discusses the issue of stable byte-representation of serialized messages within Tenderdash and describes a few possible routes that could be taken to address it.
Background
We use the byte representations of wire-format proto messages to produce and verify hashes of data within the Tenderdash codebase as well as for producing and verifying cryptographic signatures over these signed bytes.
The protocol buffer encoding spec does not guarantee that the byte representation of a protocol buffer message will be the same between two calls to an encoder. While there is a mode to force the encoder to produce the same byte representation of messages within a single binary, these guarantees are not good enough for our use case in Tenderdash. We require multiple different versions of a binary running Tenderdash to be able to inter-operate. Additionally, we require that multiple different systems written in different languages be able to participate in different aspects of the protocols of Tenderdash and be able to verify the integrity of the messages they each produce.
While this has not yet created a problem that we know of in a running network, we should make sure to provide stronger guarantees around the serialized representation of the messages used within the Tenderdash consensus algorithm to prevent any issue from occurring.
Discussion
Proto has the following points of variability that can produce non-deterministic byte representation:
- Encoding order of fields within a message.
Proto allows fields to be encoded in any order and even be repeated.
- Encoding order of elements of a repeated field.
repeated fields in a proto message can be serialized in any order.
- Presence or absence of default values.
Types in proto have defined default values similar to Go's zero values. Writing or omitting a default value are both legal ways of encoding a wire message.
- Serialization of 'unknown' fields.
Unknown fields can be present when a message is created by a binary with a newer version of the proto that contains fields that the deserializer in a different binary does not yet know about. Deserializers in binaries that do not know about the field will maintain the bytes of the unknown field but not place them into the deserialized structure.
We have a few options to consider when producing this stable representation.
Options for deterministic byte representation
Use only compliant serializers and constrain field usage
According to Cosmos-SDK ADR-27, when message types obey a simple set of rules, gogoproto produces a consistent byte representation of serialized messages. This seems promising, although more research is needed to guarantee gogoproto always produces a consistent set of bytes on serialized messages. This would solve the problem within Tenderdash as written in Go, but would require ensuring that there are similar serializers written in other languages that produce the same output as gogoproto.
Reorder serialized bytes to ensure determinism.
The serialized form of a proto message can be transformed into a canonical representation by applying simple rules to the serialized bytes. Re-ordering the serialized bytes would allow Tenderdash to produce a canonical byte representation without having to simultaneously maintain a custom proto marshaller.
This could be implemented as a function in many languages that performed the following producing bytes to sign or hashing:
- Does not add any of the data from unknown fields into the type to hash.
Tenderdash should not run into a case where it needs to verify the integrity of data with unknown fields for the following reasons:
The purpose of checking hash equality within Tenderdash is to ensure that its local copy of data matches the data that the network agreed on. There should therefore not be a case where a process is checking hash equality using data that it did not expect to receive. What the data represent may be opaque to the process, such as when checking the transactions in a block, but the process will still have expected to receive this data, despite not understanding what their internal structure is. It's not clear what it would mean to verify that a block contains data that a process does not know about.
The same reasoning applies for signature verification within Tenderdash. Processes verify that a digital signature signed over a set of bytes by locally reconstructing the data structure that the digital signature signed using the process's local data.
- Reordered all message fields to be in tag-sorted order.
Tag-sorting top-level fields will place all fields of the same tag in a adjacent to eachother within the serialized representation.
- Reordered the contents of all
repeatedfields to be in lexicographically sorted order.
repeated fields will appear in a message as having the same tag but will contain different
contents. Therefore, lexicographical sorting will produce a stable ordering of
fields with the same tag.
- Deleted all default values from the byte representation.
Encoders can include default values or omit them. Most encoders appear to omit them but we may wish to delete them just to be safe.
- Recursively performed these operations on any length-delimited subfields.
Length delimited fields may contain messages, strings, or just bytes. However, it's not possible to know what data is being represented by such a field. A 'string' may happen to have the same structure as an embedded message and we cannot disambiguate. For this reason, we must apply these same rules to all subfields that may contain messages. Because we cannot know if we have totally mangled the interior 'string' or not, this data should never be deserialized or used for anything beyond hashing.
A prototype implementation by @creachadair of this can be found in the wirepb repo. This could be implemented in multiple languages more simply than ensuring that there are canonical proto serializers that match in each language.
Future work
We should add clear documentation to the Tenderdash codebase every time we compare hashes of proto messages or use proto serialized bytes to produces a digital signatures that we have been careful to ensure that the hashes are performed properly.
References
RFC 008: Don't Panic
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 2021-12-17: initial draft (@tychoish)
Abstract
Today, the Tenderdash core codebase has panics in a number of cases as a response to exceptional situations. These panics complicate testing, and might make tenderdash components difficult to use as a library in some circumstances. This document outlines a project of converting panics to errors and describes the situations where its safe to panic.
Background
Panics in Go are a great mechanism for aborting the current execution for truly exceptional situations (e.g. memory errors, data corruption, processes initialization); however, because they resemble exceptions in other languages, it can be easy to over use them in the implementation of software architectures. This certainly happened in the history of Tenderdash, and as we embark on the project of stabilizing the package, we find ourselves in the right moment to reexamine our use of panics, and largely where panics happen in the code base.
There are still some situations where panics are acceptable and desireable, but it's important that Tenderdash, as a project, comes to consensus--perhaps in the text of this document--on the situations where it is acceptable to panic.
References
Discussion
Acceptable Panics
Initialization
It is unambiguously safe (and desireable) to panic in init()
functions in response to any kind of error. These errors are caught by
tests, and occur early enough in process initialization that they
won't cause unexpected runtime crashes.
Other code that is called early in process initialization MAY panic, in some situations if it's not possible to return an error or cause the process to abort early, although these situations should be vanishingly slim.
Data Corruption
If Tenderdash code encounters an inconsistency that could be attributed to data corruption or a logical impossibility it is safer to panic and crash the process than continue to attempt to make progress in these situations.
Examples including reading data out of the storage engine that is invalid or corrupt, or encountering an ambiguous situation where the process should halt. Generally these forms of corruption are detected after interacting with a trusted but external data source, and reflect situations where the author thinks its safer to terminate the process immediately rather than allow execution to continue.
Unrecoverable Consensus Failure
In general, a panic should be used in the case of unrecoverable consensus failures. If a process detects that the network is behaving in an incoherent way and it does not have a clearly defined and mechanism for recovering, the process should panic.
Static Validity
It is acceptable to panic for invariant violations, within a library or package, in situations that should be statically impossible, because there is no way to make these kinds of assertions at compile time.
For example, type-asserting interface{} values returned by
container/list and container/heap (and similar), is acceptable,
because package authors should have exclusive control of the inputs to
these containers. Packages should not expose the ability to add
arbitrary values to these data structures.
Controlled Panics Within Libraries
In some algorithms with highly recursive structures or very nested call patterns, using a panic, in combination with conditional recovery handlers results in more manageable code. Ultimately this is a limited application, and implementations that use panics internally should only recover conditionally, filtering out panics rather than ignoring or handling all panics.
Request Handling
Code that handles responses to incoming/external requests
(e.g. http.Handler) should avoid panics, but practice this isn't
totally possible, and it makes sense that request handlers have some
kind of default recovery mechanism that will prevent one request from
terminating a service.
Unacceptable Panics
In no other situation is it acceptable for the code to panic:
- there should be no controlled panics that callers are required to handle across library/package boundaries.
- callers of library functions should not expect panics.
- ensuring that arbitrary go routines can't panic.
- ensuring that there are no arbitrary panics in core production code, espically code that can run at any time during the lifetime of a process.
- all test code and fixture should report normal test assertions with
a mechanism like testify's
requireassertion rather than calling panic directly.
The goal of this increased "panic rigor" is to ensure that any escaped panic is reflects a fixable bug in Tenderdash.
Removing Panics
The process for removing panics involve a few steps, and will be part of an ongoing process of code modernization:
-
converting existing explicit panics to errors in cases where it's possible to return an error, the errors can and should be handled, and returning an error would not lead to data corruption or cover up data corruption.
-
increase rigor around operations that can cause runtime errors, like type assertions, nil pointer errors, array bounds access issues, and either avoid these situations or return errors where possible.
-
remove generic panic handlers which could cover and hide known panics.
RFC 009 : Consensus Parameter Upgrade Considerations
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 06-Jan-2011: Initial draft (@williambanfield).
Abstract
This document discusses the challenges of adding additional consensus parameters to Tenderdash and proposes a few solutions that can enable addition of consensus parameters in a backwards-compatible way.
Background
This section provides an overview of the issues of adding consensus parameters to Tenderdash.
Hash Compatibility
Tenderdash produces a hash of a subset of the consensus parameters. The values
that are hashed currently are the BlockMaxGas and the BlockMaxSize. These
are currently in the HashedParams struct. This hash is included
in the block and validators use it to validate that their local view of the consensus
parameters matches what the rest of the network is configured with.
Any new consensus parameters added to Tenderdash should be included in this hash. This presents a challenge for verification of historical blocks when consensus parameters are added. If a network produced blocks with a version of Tenderdash that did not yet have the new consensus parameters, the parameter hash it produced will not reference the new parameters. Any nodes joining the network with the newer version of Tenderdash will have the new consensus parameters. Tenderdash will need to handle this case so that new versions of Tenderdash with new consensus parameters can still validate old blocks correctly without having to do anything overly complex or hacky.
Allowing Developer-Defined Values and the EndBlock Problem
When new consensus parameters are added, application developers may wish to set values for them so that the developer-defined values may be used as soon as the software upgrades. We do not currently have a clean mechanism for handling this.
Consensus parameter updates are communicated from the application to Tenderdash
within EndBlock of some height H and take effect at the next height, H+1.
This means that for updates that add a consensus parameter, there is a single
height where the new parameters cannot take effect. The parameters did not exist
in the version of the software that emitted the EndBlock response for height H-1,
so they cannot take effect at height H. The first height that the updated params
can take effect is height H+1. As of now, height H must run with the defaults.
Discussion
Hash Compatibility
This section discusses possible solutions to the problem of maintaining backwards-compatibility of hashed parameters while adding new parameters.
Never Hash Defaults
One solution to the problem of backwards-compatibility is to never include parameters in the hash if the are using the default value. This means that blocks produced before the parameters existed will have implicitly been created with the defaults. This works because any software with newer versions of Tenderdash must be using the defaults for new parameters when validating old blocks since the defaults can not have been updated until a height at which the parameters existed.
Only Update HashedParams on Hash-Breaking Releases
An alternate solution to never hashing defaults is to not update the hashed parameters on non-hash-breaking releases. This means that when new consensus parameters are added to Tenderdash, there may be a release that makes use of the parameters but does not verify that they are the same across all validators by referencing them in the hash. This seems reasonably safe given the fact that only a very far subset of the consensus parameters are currently verified at all.
Version The Consensus Parameter Hash Scheme
The upcoming work on soft upgrades proposes applying different hashing rules depending on the active block version. The consensus parameter hash could be versioned in the same way. When different block versions are used, a different set of consensus parameters will be included in the hash.
Developer Defined Values
This section discusses possible solutions to the problem of allowing application developers to define values for the new parameters during the upgrade that adds the parameters.
Using InitChain for New Values
One solution to the problem of allowing application developers to define values
for new consensus parameters is to call the InitChain ABCI method on application
startup and fetch the value for any new consensus parameters. The response object
contains a field for ConsensusParameter updates so this may serve as a natural place
to put this logic.
This poses a few difficulties. Nodes replaying old blocks while running new
software do not ever call InitChain after the initial time. They will therefore
not have a way to determine that the parameters changed at some height by using a
call to InitChain. The EndBlock response is how parameter changes at a height
are currently communicated to Tenderdash and conflating these cases seems risky.
Force Defaults For Single Height
An alternate option is to not use InitChain and instead require chains to use the
default values of the new parameters for a single height.
As documented in the upcoming ADR-74, popular chains often simply use the default values. Additionally, great care is being taken to ensure that logic governed by upcoming consensus parameters is not liveness-breaking. This means that, at worst-case, chains will experience a single slow height while waiting for the new values to by applied.
Add a new UpgradeChain method
An additional method for allowing chains to update the consensus parameters that
do not yet exist is to add a new UpgradeChain method to ABCI. The upgrade chain
method would be called when the chain detects that the version of block that it
is about to produce does not match the previous block. This method would be called
after EndBlock and would return the set of consensus parameters to use at the
next height. It would therefore give an application the chance to set the new
consensus parameters before running a height with these new parameter.
References
RFC 011: Remove Gas From Tenderdash
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 03-Feb-2022: Initial draft (@williambanfield).
- 10-Feb-2022: Update in response to feedback (@williambanfield).
- 11-Feb-2022: Add reflection on MaxGas during consensus (@williambanfield).
Abstract
In the v0.25.0 release, Tenderdash added a mechanism for tracking 'Gas' in the mempool. At a high level, Gas allows applications to specify how much it will cost the network, often in compute resources, to execute a given transaction. While such a mechanism is common in blockchain applications, it is not generalizable enough to be a maintained as a part of Tenderdash. This RFC explores the possibility of removing the concept of Gas from Tenderdash while still allowing applications the power to control the contents of blocks to achieve similar goals.
Background
The notion of Gas was included in the original Ethereum whitepaper and exists as an important feature of the Ethereum blockchain.
The whitepaper describes Gas as an Anti-DoS mechanism. The Ethereum Virtual Machine provides a Turing complete execution platform. Without any limitations, malicious actors could waste computation resources by directing the EVM to perform large or even infinite computations. Gas serves as a metering mechanism to prevent this.
Gas appears to have been added to Tenderdash multiple times, initially as part of
a now defunct /vm package, and in its most recent iteration as part of v0.25.0
as a mechanism to limit the transactions that will be included in the block by an additional
parameter.
Gas has gained adoption within the Cosmos ecosystem as part of the Cosmos SDK. The SDK provides facilities for tracking how much 'Gas' a transaction is expected to take and a mechanism for tracking how much gas a transaction has already taken.
Non-SDK applications also make use of the concept of Gas. Anoma appears to implement a gas system to meter the transactions it executes.
While the notion of gas is present in projects that make use of Tenderdash, it is not a concern of Tenderdash's. Tenderdash's value and goal is producing blocks via a distributed consensus algorithm. Tenderdash relies on the application specific code to decide how to handle the transactions Tenderdash has produced (or if the application wants to consider them at all). Gas is an application concern.
Our implementation of Gas is not currently enforced by consensus. Our current validation check that
occurs during block propagation does not verify that the block is under the configured MaxGas.
Ensuring that the transactions in a proposed block do not exceed MaxGas would require
input from the application during propagation. The ProcessProposal method introduced
as part of ABCI++ would enable such input but would further entwine Tenderdash and
the application. The issue of checking MaxGas during block propagation is important
because it demonstrates that the feature as it currently exists is not implemented
as fully as it perhaps should be.
Our implementation of Gas is causing issues for node operators and relayers. At
the moment, transactions that overflow the configured 'MaxGas' can be silently rejected
from the mempool. Overflowing MaxGas is the only way that a transaction can be considered
invalid that is not directly a result of failing the CheckTx. Operators, and the application,
do not know that a transaction was removed from the mempool for this reason. A stateless check
of this nature is exactly what CheckTx exists for and there is no reason for the mempool
to keep track of this data separately. A special MempoolError field
was added in v0.35 to communicate to clients that a transaction failed after CheckTx.
While this should alleviate the pain for operators wishing to understand if their
transaction was included in the mempool, it highlights that the abstraction of
what is included in the mempool is not currently well defined.
Removing Gas from Tenderdash and the mempool would allow for the mempool to be a better
abstraction: any transaction that arrived at CheckTx and passed the check will either be
a candidate for a later block or evicted after a TTL is reached or to make room for
other, higher priority transactions. All other transactions are completely invalid and can be discarded forever.
Removing gas will not be completely straightforward. It will mean ensuring that equivalent functionality can be implemented outside of the mempool using the mempool's API.
Discussion
This section catalogs the functionality that will need to exist within the Tenderdash mempool to allow Gas to be removed and replaced by application-side bookkeeping.
Requirement: Provide Mempool Tx Sorting Mechanism
Gas produces a market for inclusion in a block. On many networks, a gas fee is included in pending transactions. This fee indicates how much a user is willing to pay per unit of execution and the fees are distributed to validators.
Validators wishing to extract higher gas fees are incentivized to include transactions
with the highest listed gas fees into each block. This produces a natural ordering
of the pending transactions. Applications wishing to implement a gas mechanism need
to be able to order the transactions in the mempool. This can trivially be accomplished
by sorting transactions using the priority field available to applications as part of
v0.35's ResponseCheckTx message.
Requirement: Allow Application-Defined Block Resizing
When creating a block proposal, Tenderdash pulls a set of possible transactions out of the mempool to include in the next block. Tenderdash uses MaxGas to limit the set of transactions it pulls out of the mempool fetching a set of transactions whose sum is less than MaxGas.
By removing gas tracking from Tenderdash's mempool, Tenderdash will need to provide a way for applications to determine an acceptable set of transactions to include in the block.
This is what the new ABCI++ PrepareProposal method is useful for. Applications
that wish to limit the contents of a block by an application-defined limit may
do so by removing transactions from the proposal it is passed during PrepareProposal.
Applications wishing to reach parity with the current Gas implementation may do
so by creating an application-side limit: filtering out transactions from
PrepareProposal the cause the proposal the exceed the maximum gas. Additionally,
applications can currently opt to have all transactions in the mempool delivered
during PrepareProposal by passing -1 for MaxGas and MaxBytes into
ReapMaxBytesMaxGas.
Requirement: Handle Transaction Metadata
Moving the gas mechanism into applications adds an additional piece of complexity to applications. The application must now track how much gas it expects a transaction to consume. The mempool currently handles this bookkeeping responsibility and uses the estimated gas to determine the set of transactions to include in the block. In order to task the application with keeping track of this metadata, we should make it easier for the application to do so. In general, we'll want to keep only one copy of this type of metadata in the program at a time, either in the application or in Tenderdash.
The following sections are possible solutions to the problem of storing transaction metadata without duplication.
Metadata Handling: EvictTx Callback
A possible approach to handling transaction metadata is by adding a new EvictTx
ABCI method. Whenever the mempool is removing a transaction, either because it has
reached its TTL or because it failed RecheckTx, EvictTx would be called with
the transaction hash. This would indicate to the application that it could free any
metadata it was storing about the transaction such as the computed gas fee.
Eviction callbacks are pretty common in caching systems, so this would be very well-worn territory.
Metadata Handling: Application-Specific Metadata Field(s)
An alternative approach to handling transaction metadata would be would be the
addition of a new application-metadata field in the ResponseCheckTx. This field
would be a protocol buffer message whose contents were entirely opaque to Tenderdash.
The application would be responsible for marshalling and unmarshalling whatever data
it stored in this field. During PrepareProposal, the application would be passed
this metadata along with the transaction, allowing the application to use it to perform
any necessary filtering.
If either of these proposed metadata handling techniques are selected, it's likely useful to enable applications to gossip metadata along with the transaction it is gossiping. This could easily take the form of an opaque proto message that is gossiped along with the transaction.
References
RFC 012: Event Indexing Revisited
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 11-Feb-2022: Add terminological notes.
- 10-Feb-2022: Updated from review feedback.
- 07-Feb-2022: Initial draft (@creachadair)
Abstract
A Tenderdash node allows ABCI events associated with block and transaction processing to be "indexed" into persistent storage. The original Tenderdash implementation provided a fixed, built-in proprietary indexer for such events.
In response to user requests to customize indexing, ADR 065 introduced an "event sink" interface that allows developers (at least in theory) to plug in alternative index storage.
Although ADR-065 was a good first step toward customization, its implementation model does not satisfy all the user requirements. Moreover, this approach leaves some existing technical issues with indexing unsolved.
This RFC documents these concerns, and discusses some potential approaches to solving them. This RFC does not propose a specific technical decision. It is meant to unify and focus some of the disparate discussions of the topic.
Background
We begin with some important terminological context. The term "event" in Tenderdash can be confusing, as the same word is used for multiple related but distinct concepts:
-
ABCI Events refer to the key-value metadata attached to blocks and transactions by the application. These values are represented by the ABCI
Eventprotobuf message type. -
Consensus Events refer to the data published by the Tenderdash node to its pubsub bus in response to various consensus state transitions and other important activities, such as round updates, votes, transaction delivery, and block completion.
This confusion is compounded because some "consensus event" values also have "ABCI event" metadata attached to them. Notably, block and transaction items typically have ABCI metadata assigned by the application.
Indexers and RPC clients subscribed to the pubsub bus receive consensus events, but they identify which ones to care about using query expressions that match against the ABCI events associated with them.
In the discussion that follows, we will use the term event item to refer to a datum published to or received from the pubsub bus, and ABCI event or event metadata to refer to the key/value annotations.
Indexing in this context means recording the association between certain ABCI metadata and the blocks or transactions they're attached to. The ABCI metadata typically carry application-specific details like sender and recipient addresses, catgory tags, and so forth, that are not part of consensus but are used by UI tools to find and display transactions of interest.
The consensus node records the blocks and transactions as part of its block store, but does not persist the application metadata. Metadata persistence is the task of the indexer, which can be (optionally) enabled by the node operator.
History
The original indexer built in to Tenderdash stored index data in an
embedded tm-db database with a proprietary key layout.
In ADR 065, we noted that this implementation has both performance
and scaling problems under load. Moreover, the only practical way to query the
index data is via the query filter language used for event
subscription. Issue #1161 appears to be a motivational context for that ADR.
To mitigate both of these concerns, we introduced the EventSink
interface, combining the original transaction and block indexer interfaces
along with some service plumbing. Using this interface, a developer can plug
in an indexer that uses a more efficient storage engine, and provides a more
expressive query language. As a proof-of-concept, we built a PostgreSQL event
sink that exports data to a PostgreSQL database.
Although this approach addressed some of the immediate concerns, there are several issues for custom indexing that have not been fully addressed. Here we will discuss them in more detail.
For further context, including links to user reports and related work, see also the Pluggable custom event indexing tracking issue issue.
Issue 1: Tight Coupling
The EventSink interface supports multiple implementations, but plugging in
implementations still requires tight integration with the node. In particular:
-
Any custom indexer must either be written in Go and compiled in to the Tenderdash binary, or the developer must write a Go shim to communicate with the implementation and build that into the Tenderdash binary.
-
This means to support a custom indexer, it either has to be integrated into the Tenderdash core repository, or every installation that uses that indexer must fetch or build a patched version of Tenderdash.
The problem with integrating indexers into Tenderdash Core is that every user of Tenderdash Core takes a dependency on all supported indexers, including those they never use. Even if the unused code is disabled with build tags, users have to remember to do this or potentially be exposed to security issues that may arise in any of the custom indexers. This is a risk for Tenderdash, which is a trust-critical component of all applications built on it.
The problem with not integrating indexers into Tenderdash Core is that any developer who wants to use a particular indexer must now fetch or build a patched version of the core code that includes the custom indexer. Besides being inconvenient, this makes it harder for users to upgrade their node, since they need to either re-apply their patches directly or wait for an intermediary to do it for them.
Even for developers who have written their applications in Go and link with the consensus node directly (e.g., using the Cosmos SDK), these issues add a potentially significant complication to the build process.
Issue 2: Legacy Compatibility
The EventSink interface retains several limitations of the original
proprietary indexer. These include:
-
The indexer has no control over which event items are reported. Only the exact block and transaction events that were reported to the original indexer are reported to a custom indexer.
-
The interface requires the implementation to define methods for the legacy search and query API. This requirement comes from the integation with the event subscription RPC API, but actually supporting these methods is not trivial.
At present, only the original KV indexer implements the query methods. Even the proof-of-concept PostgreSQL implementation simply reports errors for all calls to these methods.
Even for a plugin written in Go, implementing these methods "correctly" would require parsing and translating the custom query language over whatever storage platform the indexer uses.
For a plugin not written in Go, even beyond the cost of integration the developer would have to re-implement the entire query language.
Issue 3: Indexing Delays Consensus
Within the node, indexing hooks in to the same internal pubsub dispatcher that is used to export event items to the event subscription RPC API. In contrast with RPC subscribers, however, indexing is a "privileged" subscriber: If an RPC subscriber is "too slow", the node may terminate the subscription and disconnect the client. That means that RPC subscribers may lose (miss) event items. The indexer, however, is "unbuffered", and the publisher will never drop or disconnect from it. If the indexer is slow, the publisher will block until it returns, to ensure that no event items are lost.
In practice, this means that the performance of the indexer has a direct effect on the performance of the consensus node: If the indexer is slow or stalls, it will slow or halt the progress of consensus. Users have already reported this problem even with the built-in indexer (see, for example, #7247). Extending this concern to arbitrary user-defined custom indexers gives that risk a much larger surface area.
Discussion
It is not possible to simultaneously guarantee that publishing event items will not delay consensus, and also that all event items of interest are always completely indexed.
Therefore, our choice is between eliminating delay (and minimizing loss) or eliminating loss (and minimizing delay). Currently, we take the second approach, which has led to user complaints about consensus delays due to indexing and subscription overhead.
-
If we agree that consensus performance supersedes index completeness, our design choices are to constrain the likelihood and frequency of missing event items.
-
If we decide that consensus performance is more important than index completeness, our option is to minimize overhead on the event delivery path and document that indexer plugins constrain the rate of consensus.
Since we have user reports requesting both properties, we have to choose one or the other. Since the primary job of the consensus engine is to correctly, robustly, reliablly, and efficiently replicate application state across the network, I believe the correct choice is to favor consensus performance.
An important consideration for this decision is that a node does not index application metadata separately: If indexing is disabled, there is no built-in mechanism to go back and replay or reconstruct the data that an indexer would have stored. The node does store the blockchain itself (i.e., the blocks and their transactions), so potentially some use cases currently handled by the indexer could be handled by the node. For example, allowing clients to ask whether a given transaction ID has been committed to a block could in principle be done without an indexer, since it does not depend on application metadata.
Inevitably, a question will arise whether we could implement both strategies and toggle between them with a flag. That would be a worst-case scenario, requiring us to maintain the complexity of two very-different operational concerns. If our goal is that Tenderdash should be as simple, efficient, and trustworthy as posible, there is not a strong case for making these options configurable: We should pick a side and commit to it.
Design Principles
Although there is no unique "best" solution to the issues described above, there are some specific principles that a solution should include:
-
A custom indexer should not require integration into Tenderdash core. A developer or node operator can create, build, deploy, and use a custom indexer with a stock build of the Tenderdash consensus node.
-
Custom indexers cannot stall consensus. An indexer that is slow or stalls cannot slow down or prevent core consensus from making progress.
The plugin interface must give node operators control over the tolerances for acceptable indexer performance, and the means to detect when indexers are falling outside those tolerances, but indexer failures should "fail safe" with respect to consensus (even if that means the indexer may miss some data, in sufficiently-extreme circumstances).
-
Custom indexers control which event items they index. A custom indexer is not limited to only the current transaction and block events, but can observe any event item published by the node.
-
Custom indexing is forward-compatible. Adding new event item types or metadata to the consensus node should not require existing custom indexers to be rebuilt or modified, unless they want to take advantage of the new data.
-
Indexers are responsible for answering queries. An indexer plugin is not required to support the legacy query filter language, nor to be compatible with the legacy RPC endpoints for accessing them. Any APIs for clients to query a custom index are the responsibility of the indexer, not the node.
Open Questions
Given the constraints outlined above, there are important design questions we must answer to guide any specific changes:
-
What is an acceptable probability that, given sufficiently extreme operational issues, an indexer might miss some number of events?
There are two parts to this question: One is what constitutes an extreme operational problem, the other is how likely we are to miss some number of events items.
-
If the consensus is that no event item must ever be missed, no matter how bad the operational circumstances, then we must accept that indexing can slow or halt consensus arbitrarily. It is impossible to guarantee complete index coverage without potentially unbounded delays.
-
Otherwise, how much data can we afford to lose and how often? For example, if we can ensure no event item will be lost unless the indexer halts for at least five minutes, is that acceptable? What probabilities and time ranges are reasonable for real production environments?
-
-
What level of operational overhead is acceptable to impose on node operators to support indexing?
Are node operators willing to configure and run custom indexers as sidecar type processes alongside a node? How much indexer setup above and beyond the work of setting up the underlying node in isolation is tractable in production networks?
The answer to this question also informs the question of whether we should keep an "in-process" indexing option, and to what extent that option needs to satisfy the suggested design principles.
Relatedly, to what extent do we need to be concerned about the cost of encoding and sending event items to an external process (e.g., as JSON blobs or protobuf wire messages)? Given that the node already encodes event items as JSON for subscription purposes, the overhead would be negligible for the node itself, but the indexer would have to decode to process the results.
-
What (if any) query APIs does the consensus node need to export, independent of the indexer implementation?
One typical example is whether the node should be able to answer queries like "is this transaction ID in a block?" Currently, a node cannot answer this query unless it runs the built-in KV indexer. Does the node need to continue to support that query even for nodes that disable the KV indexer, or which use a custom indexer?
Informal Design Intent
The design principles described above implicate several components of the Tenderdash node, beyond just the indexer. In the context of ADR 075, we are re-working the RPC event subscription API to improve some of the UX issues discussed above for RPC clients. It is our expectation that a solution for pluggable custom indexing will take advantage of some of the same work.
On that basis, the design approach I am considering for custom indexing looks something like this (subject to refinement):
-
A custom indexer runs as a separate process from the node.
-
The indexer subscribes to event items via the ADR 075 events API.
This means indexers would receive event payloads as JSON rather than protobuf, but since we already have to support JSON encoding for the RPC interface anyway, that should not increase complexity for the node.
-
The existing PostgreSQL indexer gets reworked to have this form, and no longer built as part of the Tenderdash core binary.
We can retain the code in the core repository as a proof-of-concept, or perhaps create a separate repository with contributed indexers and move it there.
-
(Possibly) Deprecate and remove the legacy KV indexer, or disable it by default. If we decide to remove it, we can also remove the legacy RPC endpoints for querying the KV indexer.
If we plan to do this, we should also investigate providing a way for clients to query whether a given transaction ID has landed in a block. That serves a common need, and currently only works if the KV indexer is enabled, but could be addressed more simply using the other data a node already has stored, without having to answer more general queries.
References
- ADR 065: Custom Event Indexing
- ADR 075: RPC Event Subscription Interface
- Cosmos SDK
- Event subscription RPC
- KV transaction indexer
- Pluggable custom event indexing (#7135)
- PostgreSQL event sink
- Query filter language
- Stream events to postgres for indexing (#1161)
- Unbuffered event subscription slow down the consensus (#7247)
EventSinkinterfacetm-dblibrary
RFC 013: ABCI++
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 2020-01-11: initialized
- 2022-02-11: Migrate RFC to tendermint repo (Originally RFC 004)
Author(s)
- Dev (@valardragon)
- Sunny (@sunnya97)
Context
ABCI is the interface between the consensus engine and the application. It defines when the application can talk to consensus during the execution of a blockchain. At the moment, the application can only act at one phase in consensus, immediately after a block has been finalized.
This restriction on the application prohibits numerous features for the application, including many scalability improvements that are now better understood than when ABCI was first written. For example, many of the scalability proposals can be boiled down to "Make the miner / block proposers / validators do work, so the network does not have to". This includes optimizations such as tx-level signature aggregation, state transition proofs, etc. Furthermore, many new security properties cannot be achieved in the current paradigm, as the application cannot enforce validators do more than just finalize txs. This includes features such as threshold cryptography, and guaranteed IBC connection attempts. We propose introducing three new phases to ABCI to enable these new features, and renaming the existing methods for block execution.
Prepare Proposal phase
This phase aims to allow the block proposer to perform more computation, to reduce load on all other full nodes, and light clients in the network. It is intended to enable features such as batch optimizations on the transaction data (e.g. signature aggregation, zk rollup style validity proofs, etc.), enabling stateless blockchains with validator provided authentication paths, etc.
This new phase will only be executed by the block proposer. The application will take in the block header and raw transaction data output by the consensus engine's mempool. It will then return block data that is prepared for gossip on the network, and additional fields to include into the block header.
Process Proposal Phase
This phase aims to allow applications to determine validity of a new block proposal, and execute computation on the block data, prior to the blocks finalization. It is intended to enable applications to reject block proposals with invalid data, and to enable alternate pipelined execution models. (Such as Ethereum-style immediate execution)
This phase will be executed by all full nodes upon receiving a block, though on the application side it can do more work in the even that the current node is a validator.
Vote Extension Phase
This phase aims to allow applications to require their validators do more than just validate blocks. Example usecases of this include validator determined price oracles, validator guaranteed IBC connection attempts, and validator based threshold crypto.
This adds an app-determined data field that every validator must include with their vote, and these will thus appear in the header.
Rename {BeginBlock, [DeliverTx], EndBlock} to FinalizeBlock
The prior phases gives the application more flexibility in their execution model for a block, and they obsolete the current methods for how the consensus engine relates the block data to the state machine. Thus we refactor the existing methods to better reflect what is happening in the new ABCI model.
This rename doesn't on its own enable anything new, but instead improves naming to clarify the expectations from the application in this new communication model. The existing ABCI methods BeginBlock, [DeliverTx], EndBlock are renamed to a single method called FinalizeBlock.
Summary
We include a more detailed list of features / scaling improvements that are blocked, and which new phases resolve them at the end of this document.
Proposal
Below we suggest an API to add these three new phases. In this document, sometimes the final round of voting is referred to as precommit for clarity in how it acts in the Tenderdash case.
Prepare Proposal
Note, APIs in this section will change after Vote Extensions, we list the adjusted APIs further in the proposal.
The Prepare Proposal phase allows the block proposer to perform application-dependent work in a block, to lower the amount of work the rest of the network must do. This enables batch optimizations to a block, which has been empirically demonstrated to be a key component for scaling. This phase introduces the following ABCI method
#![allow(unused)] fn main() { fn PrepareProposal(Block) -> BlockData }
where BlockData is a type alias for however data is internally stored within the consensus engine. In Tenderdash Core today, this is []Tx.
The application may read the entire block proposal, and mutate the block data fields. Mutated transactions will still get removed from the mempool later on, as the mempool rechecks all transactions after a block is executed.
The PrepareProposal API will be modified in the vote extensions section, for allowing the application to modify the header.
Process Proposal
The Process Proposal phase sends the block data to the state machine, prior to running the last round of votes on the state machine. This enables features such as allowing validators to reject a block according to whether state machine deems it valid, and changing block execution pipeline.
We introduce three new methods,
#![allow(unused)] fn main() { fn VerifyHeader(header: Header, isValidator: bool) -> ResponseVerifyHeader {...} fn ProcessProposal(block: Block) -> ResponseProcessProposal {...} fn RevertProposal(height: usize, round: usize) {...} }
where
#![allow(unused)] fn main() { struct ResponseVerifyHeader { accept_header: bool, evidence: Vec<Evidence> } struct ResponseProcessProposal { accept_block: bool, evidence: Vec<Evidence> } }
Upon receiving a block header, every validator runs VerifyHeader(header, isValidator). The reason for why VerifyHeader is split from ProcessProposal is due to the later sections for Preprocess Proposal and Vote Extensions, where there may be application dependent data in the header that must be verified before accepting the header.
If the returned ResponseVerifyHeader.accept_header is false, then the validator must precommit nil on this block, and reject all other precommits on this block. ResponseVerifyHeader.evidence is appended to the validators local EvidencePool.
Upon receiving an entire block proposal (in the current implementation, all "block parts"), every validator runs ProcessProposal(block). If the returned ResponseProcessProposal.accept_block is false, then the validator must precommit nil on this block, and reject all other precommits on this block. ResponseProcessProposal.evidence is appended to the validators local EvidencePool.
Once a validator knows that consensus has failed to be achieved for a given block, it must run RevertProposal(block.height, block.round), in order to signal to the application to revert any potentially mutative state changes it may have made. In Tenderdash, this occurs when incrementing rounds.
RFC: How do we handle the scenario where honest node A finalized on round x, and honest node B finalized on round x + 1? (e.g. when 2f precommits are publicly known, and a validator precommits themself but doesn't broadcast, but they increment rounds) Is this a real concern? The state root derived could change if everyone finalizes on round x+1, not round x, as the state machine can depend non-uniformly on timestamp.
The application is expected to cache the block data for later execution.
The isValidator flag is set according to whether the current node is a validator or a full node. This is intended to allow for beginning validator-dependent computation that will be included later in vote extensions. (An example of this is threshold decryptions of ciphertexts.)
DeliverTx rename to FinalizeBlock
After implementing ProcessProposal, txs no longer need to be delivered during the block execution phase. Instead, they are already in the state machine. Thus BeginBlock, DeliverTx, EndBlock can all be replaced with a single ABCI method for ExecuteBlock. Internally the application may still structure its method for executing the block as BeginBlock, DeliverTx, EndBlock. However, it is overly restrictive to enforce that the block be executed after it is finalized. There are multiple other, very reasonable pipelined execution models one can go for. So instead we suggest calling this succession of methods FinalizeBlock. We propose the following API
Replace the BeginBlock, DeliverTx, EndBlock ABCI methods with the following method
#![allow(unused)] fn main() { fn FinalizeBlock() -> ResponseFinalizeBlock }
where ResponseFinalizeBlock has the following API, in terms of what already exists
#![allow(unused)] fn main() { struct ResponseFinalizeBlock { updates: ResponseEndBlock, tx_results: Vec<ResponseDeliverTx> } }
ResponseEndBlock should then be renamed to ConsensusUpdates and ResponseDeliverTx should be renamed to ResponseTx.
Vote Extensions
The Vote Extensions phase allow applications to force their validators to do more than just validate within consensus. This is done by allowing the application to add more data to their votes, in the final round of voting. (Namely the precommit) This additional application data will then appear in the block header.
First we discuss the API changes to the vote struct directly
#![allow(unused)] fn main() { fn ExtendVote(height: u64, round: u64) -> (UnsignedAppVoteData, SelfAuthenticatingAppData) fn VerifyVoteExtension(signed_app_vote_data: Vec<u8>, self_authenticating_app_vote_data: Vec<u8>) -> bool }
There are two types of data that the application can enforce validators to include with their vote. There is data that the app needs the validator to sign over in their vote, and there can be self-authenticating vote data. Self-authenticating here means that the application upon seeing these bytes, knows its valid, came from the validator and is non-malleable. We give an example of each type of vote data here, to make their roles clearer.
- Unsigned app vote data: A use case of this is if you wanted validator backed oracles, where each validator independently signs some oracle data in their vote, and the median of these values is used on chain. Thus we leverage consensus' signing process for convenience, and use that same key to sign the oracle data.
- Self-authenticating vote data: A use case of this is in threshold random beacons. Every validator produces a threshold beacon share. This threshold beacon share can be verified by any node in the network, given the share and the validators public key (which is not the same as its consensus public key). However, this decryption share will not make it into the subsequent block's header. They will be aggregated by the subsequent block proposer to get a single random beacon value that will appear in the subsequent block's header. Everyone can then verify that this aggregated value came from the requisite threshold of the validator set, without increasing the bandwidth for full nodes or light clients. To achieve this goal, the self-authenticating vote data cannot be signed over by the consensus key along with the rest of the vote, as that would require all full nodes & light clients to know this data in order to verify the vote.
The CanonicalVote struct will acommodate the UnsignedAppVoteData field by adding another string to its encoding, after the chain-id. This should not interfere with existing hardware signing integrations, as it does not affect the constant offset for the height and round, and the vote size does not have an explicit upper bound. (So adding this unsigned app vote data field is equivalent from the HSM's perspective as having a superlong chain-ID)
RFC: Please comment if you think it will be fine to have elongate the message the HSM signs, or if we need to explore pre-hashing the app vote data.
The flow of these methods is that when a validator has to precommit, Tenderdash will first produce a precommit canonical vote without the application vote data. It will then pass it to the application, which will return unsigned application vote data, and self authenticating application vote data. It will bundle the unsigned_application_vote_data into the canonical vote, and pass it to the HSM to sign. Finally it will package the self-authenticating app vote data, and the signed_vote_data together, into one final Vote struct to be passed around the network.
Changes to Prepare Proposal Phase
There are many use cases where the additional data from vote extensions can be batch optimized. This is mainly of interest when the votes include self-authenticating app vote data that be batched together, or the unsigned app vote data is the same across all votes. To allow for this, we change the PrepareProposal API to the following
#![allow(unused)] fn main() { fn PrepareProposal(Block, UnbatchedHeader) -> (BlockData, Header) }
where UnbatchedHeader essentially contains a "RawCommit", the Header contains a batch-optimized commit and an additional "Application Data" field in its root. This will involve a number of changes to core data structures, which will be gone over in the ADR.
The Unbatched header and rawcommit will never be broadcasted, they will be completely internal to consensus.
Inter-process communication (IPC) effects
For brevity in exposition above, we did not discuss the trade-offs that may occur in interprocess communication delays that these changs will introduce. These new ABCI methods add more locations where the application must communicate with the consensus engine. In most configurations, we expect that the consensus engine and the application will be either statically or dynamically linked, so all communication is a matter of at most adjusting the memory model the data is layed out within. This memory model conversion is typically considered negligible, as delay here is measured on the order of microseconds at most, whereas we face milisecond delays due to cryptography and network overheads. Thus we ignore the overhead in the case of linked libraries.
In the case where the consensus engine and the application are ran in separate processes, and thus communicate with a form of Inter-process communication (IPC), the delays can easily become on the order of miliseconds based upon the data sent. Thus its important to consider whats happening here. We go through this phase by phase.
Prepare proposal IPC overhead
This requires a round of IPC communication, where both directions are quite large. Namely the proposer communicating an entire block to the application.
However, this can be mitigated by splitting up PrepareProposal into two distinct, async methods, one for the block IPC communication, and one for the Header IPC communication.
Then for chains where the block data does not depend on the header data, the block data IPC communication can proceed in parallel to the prior block's voting phase. (As a node can know whether or not its the leader in the next round)
Furthermore, this IPC communication is expected to be quite low relative to the amount of p2p gossip time it takes to send the block data around the network, so this is perhaps a premature concern until more sophisticated block gossip protocols are implemented.
Process Proposal IPC overhead
This phase changes the amount of time available for the consensus engine to deliver a block's data to the state machine. Before, the block data for block N would be delivered to the state machine upon receiving a commit for block N and then be executed. The state machine would respond after executing the txs and before prevoting. The time for block delivery from the consensus engine to the state machine after this change is the time of receiving block proposal N to the to time precommit on proposal N. It is expected that this difference is unimportant in practice, as this time is in parallel to one round of p2p communication for prevoting, which is expected to be significantly less than the time for the consensus engine to deliver a block to the state machine.
Vote Extension IPC overhead
This has a small amount of data, but does incur an IPC round trip delay. This IPC round trip delay is pretty negligible as compared the variance in vote gossip time. (the IPC delay is typically on the order of 10 microseconds)
Status
Proposed
Consequences
Positive
- Enables a large number of new features for applications
- Supports both immediate and delayed execution models
- Allows application specific data from each validator
- Allows for batch optimizations across txs, and votes
Negative
-
This is a breaking change to all existing ABCI clients, however the application should be able to have a thin wrapper to replicate existing ABCI behavior.
- PrepareProposal - can be a no-op
- Process Proposal - has to cache the block, but can otherwise be a no-op
- Vote Extensions - can be a no-op
- Finalize Block - Can black-box call BeginBlock, DeliverTx, EndBlock given the cached block data
-
Vote Extensions adds more complexity to core Tenderdash Data Structures
-
Allowing alternate alternate execution models will lead to a proliferation of new ways for applications to violate expected guarantees.
Neutral
- IPC overhead considerations change, but mostly for the better
References
Reference for IPC delay constants: https://web.archive.org/web/2020/http://pages.cs.wisc.edu/~adityav/Evaluation_of_Inter_Process_Communication_Mechanisms.pdf
Short list of blocked features / scaling improvements with required ABCI++ Phases
| Feature | PrepareProposal | ProcessProposal | Vote Extensions |
|---|---|---|---|
| Tx based signature aggregation | X | ||
| SNARK proof of valid state transition | X | ||
| Validator provided authentication paths in stateless blockchains | X | ||
| Immediate Execution | X | ||
| Simple soft forks | X | ||
| Validator guaranteed IBC connection attempts | X | ||
| Validator based price oracles | X | ||
| Immediate Execution with increased time for block execution | X | X | X |
| Threshold Encrypted txs | X | X | X |
RFC 014: Semantic Versioning
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 2021-11-19: Initial Draft
- 2021-02-11: Migrate RFC to tendermint repo (Originally RFC 006)
Author(s)
- Callum Waters @cmwaters
Context
We use versioning as an instrument to hold a set of promises to users and signal when such a set changes and how. In the conventional sense of a Go library, major versions signal that the public Go API’s have changed in a breaking way and thus require the users of such libraries to change their usage accordingly. Tenderdash is a bit different in that there are multiple users: application developers (both in-process and out-of-process), node operators, and external clients. More importantly, both how these users interact with Tenderdash and what's important to these users differs from how users interact and what they find important in a more conventional library.
This document attempts to encapsulate the discussions around versioning in Tenderdash and draws upon them to propose a guide to how Tenderdash uses versioning to make promises to its users.
For a versioning policy to make sense, we must also address the intended frequency of breaking changes. The strictest guarantees in the world will not help users if we plan to break them with every release.
Finally I would like to remark that this RFC only addresses the "what", as in what are the rules for versioning. The "how" of Tenderdash implementing the versioning rules we choose, will be addressed in a later RFC on Soft Upgrades.
Discussion
We first begin with a round up of the various users and a set of assumptions on what these users expect from Tenderdash in regards to versioning:
-
Application Developers, those that use the ABCI to build applications on top of Tenderdash, are chiefly concerned with that API. Breaking changes will force developers to modify large portions of their codebase to accommodate for the changes. Some ABCI changes such as introducing priority for the mempool don't require any effort and can be lazily adopted whilst changes like ABCI++ may force applications to redesign their entire execution system. It's also worth considering that the API's for go developers differ to developers of other languages. The former here can use the entire Tenderdash library, most notably the local RPC methods, and so the team must be wary of all public Go API's.
-
Node Operators, those running node infrastructure, are predominantly concerned with downtime, complexity and frequency of upgrading, and avoiding data loss. They may be also concerned about changes that may break the scripts and tooling they use to supervise their nodes.
-
External Clients are those that perform any of the following:
- consume the RPC endpoints of nodes like
/block - subscribe to the event stream
- make queries to the indexer
This set are concerned with chain upgrades which will impact their ability to query state and block data as well as broadcast transactions. Examples include wallets and block explorers.
- consume the RPC endpoints of nodes like
-
IBC module and relayers. The developers of IBC and consumers of their software are concerned about changes that may affect a chain's ability to send arbitrary messages to another chain. Specifically, these users are affected by any breaking changes to the light client verification algorithm.
Although we present them here as having different concerns, in a broader sense these user groups share a concern for the end users of applications. A crucial principle guiding this RFC is that the ability for chains to provide continual service is more important than the actual upgrade burden put on the developers of these chains. This means some extra burden for application developers is tolerable if it minimizes or substantially reduces downtime for the end user.
Modes of Interprocess Communication
Tenderdash has two primary mechanisms to communicate with other processes: RPC and P2P. The division marks the boundary between the internal and external components of the network:
- The P2P layer is used in all cases that nodes (of any type) need to communicate with one another.
- The RPC interface is for any outside process that wants to communicate with a node.
The design principle here is that communication via RPC is to a trusted source and thus the RPC service prioritizes inspection rather than verification. The P2P interface is the primary medium for verification.
As an example, an in-browser light client would verify headers (and perhaps application state) via the p2p layer, and then pass along information on to the client via RPC (or potentially directly via a separate API).
The main exceptions to this are the IBC module and relayers, which are external to the node but also require verifiable data. Breaking changes to the light client verification path mean that all neighbouring chains that are connected will no longer be able to verify state transitions and thus pass messages back and forward.
Proposal
Tenderdash version labels will follow the syntax of Semantic Versions 2.0.0 with a major, minor and patch version. The version components will be interpreted according to these rules:
For the entire cycle of a major version in Tenderdash:
- All blocks and state data in a blockchain can be queried. All headers can be verified even across minor version changes. Nodes can both block sync and state sync from genesis to the head of the chain.
- Nodes in a network are able to communicate and perform BFT state machine replication so long as the agreed network version is the lowest of all nodes in a network. For example, nodes using version 1.5.x and 1.2.x can operate together so long as the network version is 1.2 or lower (but still within the 1.x range). This rule essentially captures the concept of network backwards compatibility.
- Node RPC endpoints will remain compatible with existing external clients:
- New endpoints may be added, but old endpoints may not be removed.
- Old endpoints may be extended to add new request and response fields, but requests not using those fields must function as before the change.
- Migrations should be automatic. Upgrading of one node can happen asynchronously with respect to other nodes (although agreement of a network-wide upgrade must still occur synchronously via consensus).
For the entire cycle of a minor version in Tenderdash:
- Public Go API's, for example in
nodeorabcipackages will not change in a way that requires any consumer (not just application developers) to modify their code. - No breaking changes to the block protocol. This means that all block related data structures should not change in a way that breaks any of the hashes, the consensus engine or light client verification.
- Upgrades between minor versions may not result in any downtime (i.e., no migrations are required), nor require any changes to the config files to continue with the existing behavior. A minor version upgrade will require only stopping the existing process, swapping the binary, and starting the new process.
A new patch version of Tenderdash will only contain bug fixes and updates that impact the security and stability of Tenderdash.
These guarantees will come into effect at release 1.0.
Status
Proposed
Consequences
Positive
- Clearer communication of what versioning means to us and the effect they have on our users.
Negative
- Can potentially incur greater engineering effort to uphold and follow these guarantees.
Neutral
References
RFC 015: ABCI++ TX Mutation
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 23-Feb-2022: Initial draft (@williambanfield).
- 28-Feb-2022: Revised draft (@williambanfield).
Abstract
A previous version of the ABCI++ specification detailed a mechanism for proposers to replace transactions in the proposed block. This scheme required the proposer to construct new transactions and mark these new transactions as replacing other removed transactions. The specification was ambiguous as to how the replacement may be communicated to peer nodes. This RFC discusses issues with this mechanism and possible solutions.
Background
What is the proposed change?
A previous version of the ABCI++ specification proposed mechanisms for adding, removing, and replacing
transactions in a proposed block. To replace a transaction, the application running
ProcessProposal could mark a transaction as replaced by other application-supplied
transactions by returning a new transaction marked with the ADDED flag setting
the new_hashes field of the removed transaction to contain the list of transaction hashes
that replace it. In that previous specification for ABCI++, the full use of the
new_hashes field is left somewhat ambiguous. At present, these hashes are not
gossiped and are not eventually included in the block to signal replacement to
other nodes. The specification did indicate that the transactions specified in
the new_hashes field will be removed from the mempool but it's not clear how
peer nodes will learn about them.
What systems would be affected by adding transaction replacement?
The 'transaction' is a central building block of a Tenderdash blockchain, so adding a mechanism for transaction replacement would require changes to many aspects of Tenderdash.
The following is a rough list of the functionality that this mechanism would affect:
Transaction indexing
Tenderdash's indexer stores transactions and transaction results using the hash of the executed transaction as the key and the ABCI results and transaction bytes as the value.
To allow transaction replacement, the replaced transactions would need to stored as well in the indexer, likely as a mapping of original transaction to list of transaction hashes that replaced the original transaction.
Transaction inclusion proofs
The result of a transaction query includes a Merkle proof of the existence of the transaction in the block chain. This proof is built as a merkle tree of the hashes of all of the transactions in the block where the queried transaction was executed.
To allow transaction replacement, these proofs would need to be updated to prove that a replaced transaction was included by replacement in the block.
RPC-based transaction query parameters and results
Tenderdash's RPC allows clients to retrieve information about transactions via the
/tx_search and /tx RPC endpoints.
RPC query results containing replaced transactions would need to be updated to include information on replaced transactions, either by returning results for all of the replaced transactions, or by including a response with just the hashes of the replaced transactions which clients could proceed to query individually.
Mempool transaction removal
Additional logic would need to be added to the Tenderdash mempool to clear out replaced transactions after each block is executed. Tenderdash currently removes executed transactions from the mempool, so this would be a pretty straightforward change.
Discussion
What value may be added to Tenderdash by introducing transaction replacement?
Transaction replacement would would enable applications to aggregate or disaggregate transactions.
For aggregation, a set of transactions that all related work, such as transferring tokens between the same two accounts, could be replaced with a single transaction, i.e. one that transfers a single sum from one account to the other. Applications that make frequent use of aggregation may be able to achieve a higher throughput. Aggregation would decrease the space occupied by a single client-submitted transaction in the block, allowing more client-submitted transactions to be executed per block.
For disaggregation, a very complex transaction could be split into multiple smaller transactions. This may be useful if an application wishes to perform more fine-grained indexing on intermediate parts of a multi-part transaction.
Drawbacks to transaction replacement
Transaction replacement would require updating and shimming many of the places that Tenderdash records and exposes information about executed transactions. While systems within Tenderdash could be updated to account for transaction replacement, such a system would leave new issues and rough edges.
No way of guaranteeing correct replacement
If a user issues a transaction to the network and the transaction is replaced, the user has no guarantee that the replacement was correct. For example, suppose a set of users issue transactions A, B, and C and they are all aggregated into a new transaction, D. There is nothing guaranteeing that D was constructed correctly from the inputs. The only way for users to ensure D is correct would be if D contained all of the information of its constituent transactions, in which case, nothing is really gained by the replacement.
Replacement transactions not signed by submitter
Abstractly, Tenderdash simply views transactions as a ball of bytes and therefore should be fine with replacing one for another. However, many applications require that transactions submitted to the chain be signed by some private key to authenticate and authorize the transaction. Replaced transactions could not be signed by the submitter, only by the application node. Therefore, any use of transaction replacement could not contain authorization from the submitter and would either need to grant application-submitted transactions power to perform application logic on behalf of a user without their consent.
Granting this power to application-submitted transactions would be very dangerous and therefore might not be of much value to application developers. Transaction replacement might only be really safe in the case of application-submitted transactions or for transactions that require no authorization. For such transactions, it's quite not quite clear what the utility of replacement is: the application can already generate any transactions that it wants. The fact that such a transaction was a replacement is not particularly relevant to participants in the chain since the application is merely replacing its own transactions.
New vector for censorship
Depending on the implementation, transaction replacement may allow a node signal to the rest of the chain that some transaction should no longer be considered for execution. Honest nodes will use the replacement mechanism to signal that a transaction has been aggregated. Malicious nodes will be granted a new vector for censoring transactions. There is no guarantee that a replaced transactions is actually executed at all. A malicious node could censor a transaction by simply listing it as replaced. Honest nodes seeing the replacement would flush the transaction from their mempool and not execute or propose it it in later blocks.
Transaction tracking implementations
This section discusses possible ways to flesh out the implementation of transaction replacement. Specifically, this section proposes a few alternative ways that Tenderdash blockchains could track and store transaction replacements.
Include transaction replacements in the block
One option to track transaction replacement is to include information on the transaction replacement within the block. An additional structure may be added the block of the following form:
message Block {
...
repeated Replacement replacements = 5;
}
message Replacement {
bytes included_tx_key = 1;
repeated bytes replaced_txs_keys = 2;
}
Applications executing PrepareProposal would return the list of replacements and
Tenderdash would include an encoding of these replacements in the block that is gossiped
and committed.
Tenderdash's transaction indexing would include a new mapping for each replaced transaction key to the committed transaction. Transaction inclusion proofs would be updated to include these additional new transaction keys in the Merkle tree and queries for transaction hashes that were replaced would return information indicating that the transaction was replaced along with the hash of the transaction that replaced it.
Block validation of gossiped blocks would be updated to check that each of the
included_txs_key matches the hash of some transaction in the proposed block.
Implementing the changes described in this section would allow Tenderdash to gossip and index transaction replacements as part of block propagation. These changes would still require the application to certify that the replacements were valid. This validation may be performed in one of two ways:
- Applications optimistically trust that the proposer performed a legitimate replacement.
In this validation scheme, applications would not verify that the substitution is valid during consensus and instead simply trust that the proposer is correct. This would have the drawback of allowing a malicious proposer to remove transactions it did not want executed.
- Applications completely validate transaction replacement.
In this validation scheme, applications that allow replacement would check that each listed replaced transaction was correctly reflected in the replacement transaction. In order to perform such validation, the node would need to have the replaced transactions locally. This could be accomplished one of a few ways: by querying the mempool, by adding an additional p2p gossip channel for transaction replacements, or by including the replaced transactions in the block. Replacement validation via mempool querying would require the node to have received all of the replaced transactions in the mempool which is far from guaranteed. Adding an additional gossip channel would make gossiping replaced transactions a requirement for consensus to proceed, since all nodes would need to receive all replacement messages before considering a block valid. Finally, including replaced transactions in the block seems to obviate any benefit gained from performing a transaction replacement since the replaced transaction and the original transactions would now both appear in the block.
Application defined transaction replacement
An additional option for allowing transaction replacement is to leave it entirely as a responsibility
of the application. The PrepareProposal ABCI++ call allows for applications to add
new transactions to a proposed block. Applications that wished to implement a transaction
replacement mechanism would be free to do so without the newly defined new_hashes field.
Applications wishing to implement transaction replacement would add the aggregated
transactions in the PrepareProposal response, and include one additional bookkeeping
transaction that listed all of the replacements, with a similar scheme to the new_hashes
field described in ABCI++. This new bookkeeping transaction could be used by the
application to determine which transactions to clear from the mempool in future calls
to CheckTx.
The meaning of any transaction in the block is completely opaque to Tenderdash,
so applications performing this style of replacement would not be able to have the replacement
reflected in any most of Tenderdash's transaction tracking mechanisms, such as transaction indexing
and the /tx endpoint.
Application defined Tx Keys
Tenderdash currently uses cryptographic hashes, SHA256, as a key for each transaction. As noted in the section on systems that would require changing, this key is used to identify the transaction in the mempool, in the indexer, and within the RPC system.
An alternative approach to allowing ProcessProposal to specify a set of transaction
replacements would be instead to allow the application to specify an additional key or set
of keys for each transaction during ProcessProposal. This new secondary_keys set
would be included in the block and therefore gossiped during block propagation.
Additional RPC endpoints could be exposed to query by the application-defined keys.
Applications wishing to implement replacement would leverage this new field by providing the
replaced transaction hashes as the secondary_keys and checking their validity during
ProcessProposal. During RecheckTx the application would then be responsible for
clearing out transactions that matched the secondary_keys.
It is worth noting that something like this would be possible without secondary_keys.
An application wishing to implement a system like this one could define a replacement
transaction, as discussed in the section on application-defined transaction replacement,
and use a custom ABCI event type to communicate that the replacement should
be indexed within Tenderdash's ABCI event indexing.
Complexity to value-add tradeoff
It is worth remarking that adding a system like this may introduce a decent amount
of new complexity into Tenderdash. An approach that leaves much of the replacement
logic to Tenderdash would require altering the core transaction indexing and querying
data. In many of the cases listed, a system for transaction replacement is possible
without explicitly defining it as part of PrepareProposal. Since applications
can now add transactions during PrepareProposal they can and should leverage this
functionality to include additional bookkeeping transactions in the block. It may
be worth encouraging applications to discover new and interesting ways to leverage this
power instead of immediately solving the problem for them.
References
RFC 016: Node Architecture
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- April 8, 2022: Initial draft (@cmwaters)
- April 15, 2022: Incorporation of feedback
Abstract
The node package is the entry point into the Tenderdash codebase, used both by the command line and programatically to create the nodes that make up a network. The package has suffered the most from the evolution of the codebase, becoming bloated as developers clipped on their bits of code here and there to get whatever feature they wanted working.
The decisions made at the node level have the biggest impact to simplifying the protocols within them, unlocking better internal designs and making Tenderdash more intuitive to use and easier to understand from the outside. Work, in minor increments, has already begun on this section of the codebase. This document exists to spark forth the necessary discourse in a few related areas that will help the team to converge on the long term makeup of the node.
Discussion
The following is a list of points of discussion around the architecture of the node:
Dependency Tree
The node object is currently stuffed with every component that possibly exists within Tenderdash. In the constructor, all objects are built and interlaid with one another in some awkward dance. My guiding principle is that the node should only be made up of the components that it wants to have direct control of throughout its life. The node is a service which currently has the purpose of starting other services up in a particular order and stopping them all when commanded to do so. However, there are many services which are not direct dependents i.e. the mempool and evidence services should only be working when the consensus service is running. I propose to form more of a hierarchical structure of dependents which forces us to be clear about the relations that one component has to the other. More concretely, I propose the following dependency tree:
Many of the further discussion topics circle back to this representation of the node.
It's also important to distinguish two dimensions which may require different characteristics of the architecture. There is the starting and stopping of services and their general lifecycle management. What is the correct order of operations to starting a node for example. Then there is the question of the needs of the service during actual operation. Then there is the question of what resources each service needs access to during its operation. Some need to publish events, others need access to data stores, and so forth.
An alternative model and one that perhaps better suits the latter of these dimensions is the notion of an internal message passing system. Either the events bus or p2p layer could serve as a viable transport. This would essentially allow all services to communicate with any other service and could perhaps provide a solution to the coordination problem (presented below) without a centralized coordinator. The other main advantage is that such a system would be more robust to disruptions and changes to the code which may make a hierarchical structure quickly outdated and suboptimal. The addition of message routing is an added complexity to implement, will increase the degree of asynchronicity in the system and may make it harder to debug problems that are across multiple services.
Coordination of State Advancing Mechanisms
Advancement of state in Tenderdash is simply defined in heights: If the node is at height n, how does it get to height n + 1 and so on. Based on this definition we have three components that help a node to advance in height: consensus, statesync and blocksync. The way these components behave currently is very tightly coupled to one another with references passed back and forth. My guiding principle is that each of these should be able to operate completely independently of each other, e.g. a node should be able to run solely blocksync indefinitely. There have been several ideas suggested towards improving this flow. I've been leaning strongly towards a centralized system, whereby an orchestrator (in this case the node) decides what services to start and stop. In a decentralized message passing system, individual services make their decision based upon a "global" shared state i.e. if my height is less that 10 below the average peer height, I as consensus, should stop (knowing that blocksync has the same condition for starting). As the example illustrates, each mechanism will still need to be aware of the presence of other mechanisms.
Both centralized and decentralized systems rely on the communication of the nodes current height and a judgement on the height of the head of the chain. The latter, working out the head of the chain, is quite a difficult challenge as their is nothing preventing the node from acting maliciously and providing a different height. Currently both blocksync, consensus (and to a certain degree statesync), have parallel systems where peers communicate their height. This could be streamlined with the consensus (or even the p2p layer), broadcasting peer heights and either the node or the other state advancing mechanisms acting accordingly.
Currently, when a node starts, it turns on every service that it is attached to. This means that while a node is syncing up by requesting blocks, it is also receiving transactions and votes, as well as snapshot and block requests. This is a needless use of bandwidth. An implementation of an orchestrator, regardless of whether the system is heirachical or not, should look to be able to open and close channels dynamically and effectively broadcast which services it is running. Integrating this with service discovery may also lead to a better serivce to peers.
The orchestrator allows for some deal of variablity in how a node is constructed. Does it just run blocksync, shadowing the head of the chain and be highly available for querying. Does it rely on state sync at all? An important question that arises from this dynamicism is we ideally want to encourage nodes to provide as much of their resources as possible so that their is a healthy amount of providers to consumers. Do we make all services compulsory or allow for them to be disabled? Arguably it's possible that a user forks the codebase and rips out the blocksync code because they want to reduce bandwidth so this is more a question of how easy do we want to make this for users.
Block Executor
The block executor is an important component that is currently used by both consensus and blocksync to execute transactions and update application state. Principally, I think it should be the only component that can write (and possibly even read) the block and state stores, and we should clean up other direct dependencies on the storage engine if we can. This would mean:
- The reactors Consensus, BlockSync and StateSync should all import the executor for advancing state ie.
ApplyBlockandBootstrapState. - Pruning should also be a concern of the block executor as well as
FinalizeBlockandCommit. This can simplify consensus to focus just on the consensus part.
The Interprocess communication systems: RPC, P2P, ABCI, and Events
The schematic supplied above shows the relations between the different services, the node, the block executor, and the storage layer. Represented as colored dots are the components responsible for different roles of interprocess communication (IPC). These components permeate throughout the code base, seeping into most services. What can provide powerful functionality on one hand can also become a twisted vine, creating messy corner cases and convoluting the protocols themselves. A lot of the thinking around how we want our IPC systens to function has been summarised in this RFC. In this section, I'd like to focus the reader on the relation between the IPC and the node structure. An issue that has frequently risen is that the RPC has control of the components where it strikes me as being more logical for the component to dictate the information that is emitted/available and the knobs it wishes to expose. The RPC is also inextricably tied to the node instance and has situations where it is passed pointers directly to the storage engine and other components.
I am currently convinced of the approach that the p2p layer takes and would like to see other IPC components follow suit. This would mean that the RPC and events system would be constructed in the node yet would pass the adequate methods to register endpoints and topics to the sub components. For example,
// Methods from the RPC and event bus that would be passed into the constructor of components like "consensus"
// NOTE: This is a hypothetical construction to convey the idea. An actual implementation may differ.
func RegisterRoute(path string, handler func(http.ResponseWriter, *http.Request))
func RegisterTopic(name string) EventPublisher
type EventPublisher func (context.Context, types.EventData, []abci.Event)
This would give the components control to the information they want to expose and keep all relevant logic within that package. It accomodates more to a dynamic system where services can switch on and off. Each component would also receive access to the logger and metrics system for introspection and debuggability.
IPC Rubric
I'd like to aim to reach a state where we as a team have either an implicit or explicit rubric which can determine, in the event of some new need to communicate information, what tool it should use for doing this. In the case of inter node communication, this is obviously the p2p stack (with perhaps the exception of the light client). Metrics and logging also have clear usage patterns. RPC and the events system are less clear. The RPC is used for debugging data and fine tuned operator control as it is for general public querying and transaction submission. The RPC is also known to have been plumbed back into the application for historical queries. The events system, similarly, is used for consuming transaction events as it is for the testing of consensus state transitions.
Principally, I think we should look to change our language away from what the actual transport is and more towards what it's being used for and to whom. We call it a peer to peer layer and not the underlying tcp connection. In the same way, we should look to split RPC into an operator interface (RPC Internal), a public interface (RPC External) and a bidirectional ABCI.
Seperation of consumers and suppliers
When a service such as blocksync is turned on, it automatically begins requesting blocks to verify and apply them as it also tries to serve them to other peers catching up. We should look to distinguish these two aspects: supplying of information and consuming of information in many of these components. More concretely, I'd suggest:
- The blocksync and statesync service, i.e. supplying information for those trying to catch up should only start running once a node has caught up i.e. after running the blocksync and/or state sync processes
- The blocksync and state sync processes have defined termination clauses that inform the orchestrator when they are done and where they finished.
- One way of achieving this would be that every process both passes and returns the
Stateobject - In some cases, a node may specify that it wants to run blocksync indefinitely.
- One way of achieving this would be that every process both passes and returns the
- The mempool should also indicate whether it wants to receive transactions or to send them only (one-directional mempool)
- Similarly, the light client itself only requests information whereas the light client service (currently part of state sync) can do both.
- This distinction needs to be communicated in the p2p layer handshake itself but should also be changeable over the lifespan of the connection.
RFC 017: ABCI++ Vote Extension Propagation
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 11-Apr-2022: Initial draft (@sergio-mena).
- 15-Apr-2022: Addressed initial comments. First complete version (@sergio-mena).
- 09-May-2022: Addressed all outstanding comments.
Abstract
According to the
ABCI++ specification
(as of 11-Apr-2022), a validator MUST provide a signed vote extension for each non-nil precommit vote
of height h that it uses to propose a block in height h+1. When a validator is up to
date, this is easy to do, but when a validator needs to catch up this is far from trivial as this data
cannot be retrieved from the blockchain.
This RFC presents and compares the different options to address this problem, which have been proposed in discussions by the original upstream team.
Document Structure
The RFC is structured as follows. In the Background section, subsections Problem Description and Cases to Address explain the problem at hand from a high level perspective, i.e., abstracting away from the current Tenderdash implementation. In contrast, subsection Current Catch-up Mechanisms delves into the details of the current Tenderdash code.
In the Discussion section, subsection Solutions Proposed is also worded abstracting away from implementation details, whilst subsections Feasibility of the Proposed Solutions and Current Limitations and Possible Implementations analize the viability of one of the proposed solutions in the context of Tenderdash's architecture based on reactors. Finally, Formalization Work briefly discusses the work still needed demonstrate the correctness of the chosen solution.
The high level subsections are aimed at readers who are familiar with consensus algorithms, in particular with the one described in the Tendermint white paper, but who are not necessarily acquainted with the details of the Tenderdash codebase. The other subsections, which go into implementation details, are best understood by engineers with deep knowledge of the implementation of Tenderdash's blocksync and consensus reactors.
Background
Basic Definitions
This document assumes that all validators have equal voting power for the sake of simplicity. This is done without loss of generality.
There are two types of votes in Tenderdash: prevotes and precommits. Votes can be nil or refer to
a proposed block. This RFC focuses on precommits,
also known as precommit votes. In this document we sometimes call them simply votes.
Validators send precommit votes to their peer nodes in precommit messages. According to the ABCI++ specification, a precommit message MUST also contain a vote extension. This mandatory vote extension can be empty, but MUST be signed with the same key as the precommit vote (i.e., the sending validator's). Nevertheless, the vote extension is signed independently from the vote, so a vote can be separated from its extension. The reason for vote extensions to be mandatory in precommit messages is that, otherwise, a (malicious) node can omit a vote extension while still providing/forwarding/sending the corresponding precommit vote.
The validator set at height h is denoted valseth. A commit for height h consists of more
than 2nh/3 precommit votes voting for a block b, where nh denotes the size of
valseth. A commit does not contain nil precommit votes, and all votes in it refer to the
same block. An extended commit is a commit where every precommit vote has its respective vote extension
attached.
Problem Description
In the version of ABCI present up to Tendermint v0.35, for any height h, a validator v MUST have the decided block b and a commit for height h in order to decide at height h. Then, v just needs a commit for height h to propose at height h+1, in the rounds of h+1 where v is a proposer.
In ABCI++, the information that a validator v MUST have to be able to decide in h does not change with respect to pre-existing ABCI: the decided block b and a commit for h. In contrast, for proposing in h+1, a commit for h is not enough: v MUST now have an extended commit.
When a validator takes an active part in consensus at height h, it has all the data it needs in memory, in its consensus state, to decide on h and propose in h+1. Things are not so easy in the cases when v cannot take part in consensus because it is late (e.g., it falls behind, it crashes and recovers, or it just starts after the others). If v does not take part, it cannot actively gather precommit messages (which include vote extensions) in order to decide. Before ABCI++, this was not a problem: full nodes are supposed to persist past blocks in the block store, so other nodes would realise that v is late and send it the missing decided block at height h and the corresponding commit (kept in block h+1) so that v can catch up. However, we cannot apply this catch-up technique for ABCI++, as the vote extensions, which are part of the needed extended commit are not part of the blockchain.
Cases to Address
Before we tackle the description of the possible cases we need to address, let us describe the following incremental improvement to the ABCI++ logic. Upon decision, a full node persists (e.g., in the block store) the extended commit that allowed the node to decide. For the moment, let us assume the node only needs to keep its most recent extended commit, and MAY remove any older extended commits from persistent storage. This improvement is so obvious that all solutions described in the Discussion section use it as a building block. Moreover, it completely addresses by itself some of the cases described in this subsection.
We now describe the cases (i.e. possible runs of the system) that have been raised in different discussions and need to be addressed. They are (roughly) ordered from easiest to hardest to deal with.
-
(a) Happy path: all validators advance together, no crash.
This case is included for completeness. All validators have taken part in height h. Even if some of them did not manage to send a precommit message for the decided block, they all receive enough precommit messages to be able to decide. As vote extensions are mandatory in precommit messages, every validator v trivially has all the information, namely the decided block and the extended commit, needed to propose in height h+1 for the rounds in which v is the proposer.
No problem to solve here.
-
(b) All validators advance together, then all crash at the same height.
This case has been raised in some discussions, the main concern being whether the vote extensions for the previous height would be lost across the network. With the improvement described above, namely persisting the latest extended commit at decision time, this case is solved. When a crashed validator recovers, it recovers the last extended commit from persistent storage and handshakes with the Application. If need be, it also reconstructs messages for the unfinished height (including all precommits received) from the WAL. Then, the validator can resume where it was at the time of the crash. Thus, as extensions are persisted, either in the WAL (in the form of received precommit messages), or in the latest extended commit, the only way that vote extensions needed to start the next height could be lost forever would be if all validators crashed and never recovered (e.g. disk corruption). Since a correct node MUST eventually recover, this violates Tenderdash's assumption of more than 2nh/3 correct validators for every height h.
No problem to solve here.
-
(c) Lagging majority.
Let us assume the validator set does not change between h and h+1. It is not possible by the nature of the Tendermint algorithm, which requires more than 2nh/3 precommit votes for some round of height h in order to make progress. So, only up to nh/3 validators can lag behind.
On the other hand, for the case where there are changes to the validator set between h and h+1 please see case (d) below, where the extreme case is discussed.
-
(d) Validator set changes completely between h and h+1.
If sets valseth and valseth+1 are disjoint, more than 2nh/3 of validators in height h should have actively participated in conensus in h. So, as of height h, only a minority of validators in h can be lagging behind, although they could all lag behind from h+1 on, as they are no longer validators, only full nodes. This situation falls under the assumptions of case (h) below.
As for validators in valseth+1, as they were not validators as of height h, they could all be lagging behind by that time. However, by the time h finishes and h+1 begins, the chain will halt until more than 2nh+1/3 of them have caught up and started consensus at height h+1. If set valseth+1 does not change in h+2 and subsequent heights, only up to nh+1/3 validators will be able to lag behind. Thus, we have converted this case into case (h) below.
-
(e) Enough validators crash to block the rest.
In this case, blockchain progress halts, i.e. surviving full nodes keep increasing rounds indefinitely, until some of the crashed validators are able to recover. Those validators that recover first will handshake with the Application and recover at the height they crashed, which is still the same the nodes that did not crash are stuck in, so they don't need to catch up. Further, they had persisted the extended commit for the previous height. Nothing to solve.
For those validators recovering later, we are in case (h) below.
-
(f) Some validators crash, but not enough to block progress.
When the correct processes that crashed recover, they handshake with the Application and resume at the height they were at when they crashed. As the blockchain did not stop making progress, the recovered processes are likely to have fallen behind with respect to the progressing majority.
At this point, the recovered processes are in case (h) below.
-
(g) A new full node starts.
The reasoning here also applies to the case when more than one full node are starting. When the full node starts from scratch, it has no state (its current height is 0). Ignoring statesync for the time being, the node just needs to catch up by applying past blocks one by one (after verifying them).
Thus, the node is in case (h) below.
-
(h) Advancing majority, lagging minority
In this case, some nodes are late. More precisely, at the present time, a set of full nodes, denoted Lhp, are falling behind (e.g., temporary disconnection or network partition, memory thrashing, crashes, new nodes) an arbitrary number of heights: between hs and hp, where hs < hp, and hp is the highest height any correct full node has reached so far.
The correct full nodes that reached hp were able to decide for hp-1. Therefore, less than nhp-1/3 validators of hp-1 can be part of Lhp, since enough up-to-date validators needed to actively participate in consensus for hp-1.
Since, at the present time, no node in Lhp took part in any consensus between hs and hp-1, the reasoning above can be extended to validator set changes between hs and hp-1. This results in the following restriction on the full nodes that can be part of Lhp.
- ∀ h, where hs ≤ h < hp, | valseth ∩ Lhp | < nh/3
If this property does not hold for a particular height h, where hs ≤ h < hp, Tenderdash could not have progressed beyond h and therefore no full node could have reached hp (a contradiction).
These lagging nodes in Lhp need to catch up. They have to obtain the information needed to make progress from other nodes. For each height h between hs and hp-2, this includes the decided block for h, and the precommit votes also for deciding h (which can be extracted from the block at height h+1).
At a given height hc (where possibly hc << hp), a full node in Lhp will consider itself caught up, based on the (maybe out of date) information it is getting from its peers. Then, the node needs to be ready to propose at height hc+1, which requires having received the vote extensions for hc. As the vote extensions are not stored in the blocks, and it is difficult to have strong guarantees on when a late node considers itself caught up, providing the late node with the right vote extensions for the right height poses a problem.
At this point, we have described and compared all cases raised in discussions leading up to this RFC. The list above aims at being exhaustive. The analysis of each case included above makes all of them converge into case (h).
Current Catch-up Mechanisms
We now briefly describe the current catch-up mechanisms in the reactors concerned in Tenderdash.
Statesync
Full nodes optionally run statesync just after starting, when they start from scratch. If statesync succeeds, an Application snapshot is installed, and Tenderdash jumps from height 0 directly to the height the Application snapshop represents, without applying the block of any previous height. Some light blocks are received and stored in the block store for running light-client verification of all the skipped blocks. Light blocks are incomplete blocks, typically containing the header and the canonical commit but, e.g., no transactions. They are stored in the block store as "signed headers".
The statesync reactor is not really relevant for solving the problem discussed in this RFC. We will nevertheless mention it when needed; in particular, to understand some corner cases.
Blocksync
The blocksync reactor kicks in after start up or recovery (and, optionally, after statesync is done) and sends the following messages to its peers:
StatusRequestto query the height its peers are currently at, andBlockRequest, asking for blocks of heights the local node is missing.
Using BlockResponse messages received from peers, the blocksync reactor validates each received
block using the block of the following height, saves the block in the block store, and sends the
block to the Application for execution.
If blocksync has validated and applied the block for the height previous to the highest seen in
a StatusResponse message, or if no progress has been made after a timeout, the node considers
itself as caught up and switches to the consensus reactor.
Consensus Reactor
The consensus reactor runs the full Tendermint algorithm. For a validator this means it has to propose blocks, and send/receive prevote/precommit messages, as mandated by Tenderdash, before it can decide and move on to the next height.
If a full node that is running the consensus reactor falls behind at height h, when a peer node realises this it will retrieve the canonical commit of h+1 from the block store, and convert it into a set of precommit votes and will send those to the late node.
Discussion
Solutions Proposed
These are the solutions proposed in discussions leading up to this RFC.
-
Solution 0. Vote extensions are made best effort in the specification.
This is the simplest solution, considered as a way to provide vote extensions in a simple enough way so that it can be part of v0.36. It consists in changing the specification so as to not require that precommit votes used upon
PrepareProposalcontain their corresponding vote extensions. In other words, we render vote extensions optional. There are strong implications stemming from such a relaxation of the original specification.- As a vote extension is signed separately from the vote it is extending, an intermediate node can now remove (i.e., censor) vote extensions from precommit messages at will.
- Further, there is no point anymore in the spec requiring the Application to accept a vote extension
passed via
VerifyVoteExtensionto consider a precommit message valid in its entirety. Remember this behavior ofVerifyVoteExtensionis adding a constraint to Tenderdash's conditions for liveness. In this situation, it is better and simpler to just drop the vote extension rejected by the Application viaVerifyVoteExtension, but still consider the precommit vote itself valid as long as its signature verifies.
-
Solution 1. Include vote extensions in the blockchain.
Another obvious solution, which has somehow been considered in the past, is to include the vote extensions and their signatures in the blockchain. The blockchain would thus include the extended commit, rather than a regular commit, as the structure to be canonicalized in the next block. With this solution, the current mechanisms implemented both in the blocksync and consensus reactors would still be correct, as all the information a node needs to catch up, and to start proposing when it considers itself as caught-up, can now be recovered from past blocks saved in the block store.
This solution has two main drawbacks.
- As the block format must change, upgrading a chain requires a hard fork. Furthermore, all existing light client implementations will stop working until they are upgraded to deal with the new format (e.g., how certain hashes calculated and/or how certain signatures are checked). For instance, let us consider IBC, which relies on light clients. An IBC connection between two chains will be broken if only one chain upgrades.
- The extra information (i.e., the vote extensions) that is now kept in the blockchain is not really
needed at every height for a late node to catch up.
- This information is only needed to be able to propose at the height the validator considers
itself as caught-up. If a validator is indeed late for height h, it is useless (although
correct) for it to call
PrepareProposal, orExtendVote, since the block is already decided. - Moreover, some use cases require pretty sizeable vote extensions, which would result in an important waste of space in the blockchain.
- This information is only needed to be able to propose at the height the validator considers
itself as caught-up. If a validator is indeed late for height h, it is useless (although
correct) for it to call
-
Solution 2. Skip propose step in Tendermint algorithm.
This solution consists in modifying the Tendermint algorithm to skip the send proposal step in heights where the node does not have the required vote extensions to populate the call to
PrepareProposal. The main idea behind this is that it should only happen when the validator is late and, therefore, up-to-date validators have already proposed (and decided) for that height. A small variation of this solution is, rather than skipping the send proposal step, the validator sends a special empty or bottom (⊥) proposal to signal other nodes that it is not ready to propose at (any round of) the current height.The appeal of this solution is its simplicity. A possible implementation does not need to extend the data structures, or change the current catch-up mechanisms implemented in the blocksync or in the consensus reactor. When we lack the needed information (vote extensions), we simply rely on another correct validator to propose a valid block in other rounds of the current height.
However, this solution can be attacked by a byzantine node in the network in the following way. Let us consider the following scenario:
- all validators in valseth send out precommit messages, with vote extensions, for height h, round 0, roughly at the same time,
- all those precommit messages contain non-
nilprecommit votes, which vote for block b - all those precommit messages sent in height h, round 0, and all messages sent in height h, round r > 0 get delayed indefinitely, so,
- all validators in valseth keep waiting for enough precommit messages for height h, round 0, needed for deciding in height h
- an intermediate (malicious) full node m manages to receive block b, and gather more than 2nh/3 precommit messages for height h, round 0,
- one way or another, the solution should have either (a) a mechanism for a full node to tell another full node it is late, or (b) a mechanism for a full node to conclude it is late based on other full nodes' messages; any of these mechanisms should, at the very least, require the late node receiving the decided block and a commit (not necessarily an extended commit) for h,
- node m uses the gathered precommit messages to build a commit for height h, round 0,
- in order to convince full nodes that they are late, node m either (a) tells them they are late, or (b) shows them it (i.e. m) is ahead, by sending them block b, along with the commit for height h, round 0,
- all full nodes conclude they are late from m's behavior, and use block b and the commit for height h, round 0, to decide on height h, and proceed to height h+1.
At this point, all full nodes, including all validators in valseth+1, have advanced to height h+1 believing they are late, and so, expecting the hypothetical leading majority of validators in valseth+1 to propose for h+1. As a result, the blockhain grinds to a halt. A (rather complex) ad-hoc mechanism would need to be carried out by node operators to roll back all validators to the precommit step of height h, round r, so that they can regenerate vote extensions (remember vote extensions are non-deterministic) and continue execution.
-
Solution 3. Require extended commits to be available at switching time.
This one is more involved than all previous solutions, and builds on an idea present in Solution 2: vote extensions are actually not needed for Tenderdash to make progress as long as the validator is certain it is late.
We define two modes. The first is denoted catch-up mode, and Tenderdash only calls
FinalizeBlockfor each height when in this mode. The second is denoted consensus mode, in which the validator considers itself up to date and fully participates in consensus and callsPrepareProposal/ProcessProposal,ExtendVote, andVerifyVoteExtension, before callingFinalizeBlock.The catch-up mode does not need vote extension information to make progress, as all it needs is the decided block at each height to call
FinalizeBlockand keep the state-machine replication making progress. The consensus mode, on the other hand, does need vote extension information when starting every height.Validators are in consensus mode by default. When a validator in consensus mode falls behind for whatever reason, e.g. cases (b), (d), (e), (f), (g), or (h) above, we introduce the following key safety property:
- for every height hp, a full node f in hp refuses to switch to catch-up
mode until there exists a height h' such that:
- p has received and (light-client) verified the blocks of all heights h, where hp ≤ h ≤ h'
- it has received an extended commit for h' and has verified:
- the precommit vote signatures in the extended commit
- the vote extension signatures in the extended commit: each is signed with the same key as the precommit vote it extends
If the condition above holds for hp, namely receiving a valid sequence of blocks in the f's future, and an extended commit corresponding to the last block in the sequence, then node f:
- switches to catch-up mode,
- applies all blocks between hp and h' (calling
FinalizeBlockonly), and - switches back to consensus mode using the extended commit for h' to propose in the rounds of h' + 1 where it is the proposer.
This mechanism, together with the invariant it uses, ensures that the node cannot be attacked by being fed a block without extensions to make it believe it is late, in a similar way as explained for Solution 2.
- for every height hp, a full node f in hp refuses to switch to catch-up
mode until there exists a height h' such that:
Feasibility of the Proposed Solutions
Solution 0, besides the drawbacks described in the previous section, provides guarantees that are
weaker than the rest. The Application does not have the assurance that more than 2nh/3 vote
extensions will always be available when calling PrepareProposal at height h+1.
This level of guarantees is probably not strong enough for vote extensions to be useful for some
important use cases that motivated them in the first place, e.g., encrypted mempool transactions.
Solution 1, while being simple in that the changes needed in the current Tenderdash codebase would be rather small, is changing the block format, and would therefore require all blockchains using Tendermint v0.35 or earlier to hard-fork when upgrading to v0.36.
Since Solution 2 can be attacked, one might prefer Solution 3, even if it is more involved to implement. Further, we must elaborate on how we can turn Solution 3, described in abstract terms in the previous section, into a concrete implementation compatible with the current Tenderdash codebase.
Current Limitations and Possible Implementations
The main limitations affecting the current version of Tenderdash are the following.
- The current version of the blocksync reactor does not use the full light client verification algorithm to validate blocks coming from other peers.
- The code being structured into the blocksync and consensus reactors, only switching from the
blocksync reactor to the consensus reactor is supported; switching in the opposite direction is
not supported. Alternatively, the consensus reactor could have a mechanism allowing a late node
to catch up by skipping calls to
PrepareProposal/ProcessProposal, andExtendVote/VerifyVoteExtensionand only callingFinalizeBlockfor each height. Such a mechanism does not exist at the time of writing this RFC.
The blocksync reactor featuring light client verification is being actively worked on (tentatively for v0.37). So it is best if this RFC does not try to delve into that problem, but just makes sure its outcomes are compatible with that effort.
In subsection Cases to Address, we concluded that we can focus on solving case (h) in theoretical terms. However, as the current Tenderdash version does not yet support switching back to blocksync once a node has switched to consensus, we need to split case (h) into two cases. When a full node needs to catch up...
-
(h.1) ... it has not switched yet from the blocksync reactor to the consensus reactor, or
-
(h.2) ... it has already switched to the consensus reactor.
This is important in order to discuss the different possible implementations.
Base Implementation: Persist and Propagate Extended Commit History
In order to circumvent the fact that we cannot switch from the consensus reactor back to blocksync, rather than just keeping the few most recent extended commits, nodes will need to keep and gossip a backlog of extended commits so that the consensus reactor can still propose and decide in out-of-date heights (even if those proposals will be useless).
The base implementation - for which an experimental patch exists - consists in the conservative approach of persisting in the block store all extended commits for which we have also stored the full block. Currently, when statesync is run at startup, it saves light blocks. This base implementation does not seek to receive or persist extended commits for those light blocks as they would not be of any use.
Then, we modify the blocksync reactor so that peers always send requested full blocks together
with the corresponding extended commit in the BlockResponse messages. This guarantees that the
block store being reconstructed by blocksync has the same information as that of peers that are
up to date (at least starting from the latest snapshot applied by statesync before starting blocksync).
Thus, blocksync has all the data it requires to switch to the consensus reactor, as long as one of
the following exit conditions are met:
- The node is still at height 0 (where no commit or extended commit is needed)
- The node has processed at least 1 block in blocksync
The second condition is needed in case the node has installed an Application snapshot during statesync. If that is the case, at the time blocksync starts, the block store only has the data statesync has saved: light blocks, and no extended commits. Hence we need to blocksync at least one block from another node, which will be sent with its corresponding extended commit, before we can switch to consensus.
As a side note, a chain might be started at a height hi > 0, all other heights h < hi being non-existent. In this case, the chain is still considered to be at height 0 before block hi is applied, so the first condition above allows the node to switch to consensus even if blocksync has not processed any block (which is always the case if all nodes are starting from scratch).
When a validator falls behind while having already switched to the consensus reactor, a peer node can
simply retrieve the extended commit for the required height from the block store and reconstruct a set of
precommit votes together with their extensions and send them in the form of precommit messages to the
validator falling behind, regardless of whether the peer node holds the extended commit because it
actually participated in that consensus and thus received the precommit messages, or it received the extended commit via a BlockResponse message while running blocksync.
This solution requires a few changes to the consensus reactor:
- upon saving the block for a given height in the block store at decision time, save the corresponding extended commit as well
- in the catch-up mechanism, when a node realizes that another peer is more than 2 heights behind, it uses the extended commit (rather than the canoncial commit as done previously) to reconstruct the precommit votes with their corresponding extensions
The changes to the blocksync reactor are more substantial:
- the
BlockResponsemessage is extended to include the extended commit of the same height as the block included in the response (just as they are stored in the block store) - structure
bpRequesteris likewise extended to hold the received extended commits coming inBlockResponsemessages - method
PeekTwoBlocksis modified to also return the extended commit corresponding to the first block - when successfully verifying a received block, the reactor saves its corresponding extended commit in the block store
The two main drawbacks of this base implementation are:
- the increased size taken by the block store, in particular with big extensions
- the increased bandwith taken by the new format of
BlockResponse
Possible Optimization: Pruning the Extended Commit History
If we cannot switch from the consensus reactor back to the blocksync reactor we cannot prune the extended commit backlog in the block store without sacrificing the implementation's correctness. The asynchronous nature of our distributed system model allows a process to fall behing an arbitrary number of heights, and thus all extended commits need to be kept just in case a node that late had previously switched to the consensus reactor.
However, there is a possibility to optimize the base implementation. Every time we enter a new height, we could prune from the block store all extended commits that are more than d heights in the past. Then, we need to handle two new situations, roughly equivalent to cases (h.1) and (h.2) described above.
- (h.1) A node starts from scratch or recovers after a crash. In thisy case, we need to modify the
blocksync reactor's base implementation.
- when receiving a
BlockResponsemessage, it MUST accept that the extended commit set tonil, - when sending a
BlockResponsemessage, if the block store contains the extended commit for that height, it MUST set it in the message, otherwise it sets it tonil, - the exit conditions used for the base implementation are no longer valid; the only reliable exit
condition now consists in making sure that the last block processed by blocksync was received with
the corresponding commit, and not
nil; this extended commit will allow the node to switch from the blocksync reactor to the consensus reactor and immediately act as a proposer if required.
- when receiving a
- (h.2) A node already running the consensus reactor falls behind beyond d heights. In principle, the node will be stuck forever as no other node can provide the vote extensions it needs to make progress (they all have pruned the corresponding extended commit). However we can manually have the node crash and recover as a workaround. This effectively converts this case into (h.1).
Formalization Work
A formalization work to show or prove the correctness of the different use cases and solutions presented here (and any other that may be found) needs to be carried out. A question that needs a precise answer is how many extended commits (one?, two?) a node needs to keep in persistent memory when implementing Solution 3 described above without Tenderdash's current limitations. Another important invariant we need to prove formally is that the set of vote extensions required to make progress will always be held somewhere in the network.
References
RFC 018: BLS Signature Aggregation Exploration
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 01-April-2022: Initial draft (@williambanfield).
- 15-April-2022: Draft complete (@williambanfield).
Abstract
Background
Glossary
The terms that are attached to these types of cryptographic signing systems become confusing quickly. Different sources appear to use slightly different meanings of each term and this can certainly add to the confusion. Below is a brief glossary that may be helpful in understanding the discussion that follows.
- Short Signature: A signature that does not vary in length with the number of signers.
- Multi-Signature: A signature generated over a single message where, given the message and signature, a verifier is able to determine that all parties signed the message. May be short or may vary with the number of signers.
- Aggregated Signature: A short signature generated over messages with possibly different content where, given the messages and signature, a verifier should be able to determine that all the parties signed the designated messages.
- Threshold Signature: A short signature generated from multiple signers where, given a message and the signature, a verifier is able to determine that a large enough share of the parties signed the message. The identities of the parties that contributed to the signature are not revealed.
- BLS Signature: An elliptic-curve pairing-based signature system that has some nice properties for short multi-signatures. May stand for Boneh-Lynn-Schacham or Barreto-Lynn-Scott depending on the context. A BLS signature is type of signature scheme that is distinct from other forms of elliptic-curve signatures such as ECDSA and EdDSA.
- Interactive: Cryptographic scheme where parties need to perform one or more request-response cycles to produce the cryptographic material. For example, an interactive signature scheme may require the signer and the verifier to cooperate to create and/or verify the signature, rather than a signature being created ahead of time.
- Non-interactive: Cryptographic scheme where parties do not need to perform any request-response cycles to produce the cryptographic material.
Brief notes on pairing-based elliptic-curve cryptography
Pairing-based elliptic-curve cryptography is quite complex and relies on several types of high-level math. Cryptography, in general, relies on being able to find problems with an asymmetry between the difficulty of calculating the solution and verifying that a given solution is correct.
Pairing-based cryptography works by operating on mathematical functions that
satisfy the property of bilinear mapping. This property is satisfied for
functions e with values P, Q, R and S where e(P, Q + R) = e(P, Q) * e(P, R)
and e(P + S, Q) = e(P, Q) * e(S, Q). The most familiar example of this is
exponentiation. Written in common notation, g^P*(Q+R) = g^(P*Q) * g^(P*R) for
some value g.
Pairing-based elliptic-curve cryptography creates a bilinear mapping using
elliptic curves over a finite field. With some original curve, you can define two groups,
G1 and G2 which are points of the original curve modulo different values.
Finally, you define a third group Gt, where points from G1 and G2 satisfy
the property of bilinearity with Gt. In this scheme, the function e takes
as inputs points in G1 and G2 and outputs values in Gt. Succintly, given
some point P in G1 and some point Q in G1, e(P, Q) = C where C is in Gt.
You can efficiently compute the mapping of points in G1 and G2 into Gt,
but you cannot efficiently determine what points were summed and paired to
produce the value in Gt.
Functions are then defined to map digital signatures, messages, and keys into
and out of points of G1 or G2 and signature verification is the process
of calculating if a set of values representing a message, public key, and digital
signature produce the same value in Gt through e.
Signatures can be created as either points in G1 with public keys being
created as points in G2 or vice versa. For the case of BLS12-381, the popular
curve used, points in G1 are represented with 48 bytes and points in G2 are
represented with 96 bytes. It is up to the implementer of the cryptosystem to
decide which should be larger, the public keys or the signatures.
BLS signatures rely on pairing-based elliptic-curve cryptography to produce various types of signatures. For a more in-depth but still high level discussion pairing-based elliptic-curve cryptography, see Vitalik Buterin's post on Exploring Elliptic Curve Pairings. For much more in depth discussion, see the specific paper on BLS12-381, Short signatures from the Weil Pairing and Compact Multi-Signatures for Smaller Blockchains.
Adoption
BLS signatures have already gained traction within several popular projects.
- Algorand is working on an implementation.
- Zcash has adopted BLS12-381 into the protocol.
- Ethereum 2.0 has adopted BLS12-381 into the protocol.
- Chia Network has adopted BLS for signing blocks.
- Ostracon, a fork of Tendermint has adopted BLS for signing blocks.
What systems may be affected by adding aggregated signatures?
Gossip
Gossip could be updated to aggregate vote signatures during a consensus round. This appears to be of frankly little utility. Creating an aggregated signature incurs overhead, so frequently re-aggregating may incur a significant overhead. How costly this is is still subject to further investigation and performance testing.
Even if vote signatures were aggregated before gossip, each validator would still need to receive and verify vote extension data from each (individual) peer validator in order for consensus to proceed. That displaces any advantage gained by aggregating signatures across the vote message in the presence of vote extensions.
Block Creation
When creating a block, the proposer may create a small set of short multi-signatures and attach these to the block instead of including one signature per validator.
Block Verification
Currently, we verify each validator signature using the public key associated with that validator. With signature aggregation, verification of blocks would not verify many signatures individually, but would instead check the (single) multi-signature using the public keys stored by the validator. This would also require a mechanism for indicating which validators are included in the aggregated signature.
IBC Relaying
IBC would no longer need to transmit a large set of signatures when updating state. These state updates do not happen for every IBC packet, only when changing an IBC light client's view of the counterparty chain's state. General [IBC packets][ibc-packet] only contain enough information to correctly route the data to the counterparty chain.
IBC does persist commit signatures to the chain in these MsgUpdateClient
message when updating state. This message would no longer need the full set
of unique signatures and would instead only need one signature for all of the
data in the header.
Adding BLS signatures would create a new signature type that must be understood by the IBC module and by the relayers. For some operations, such as state updates, the set of data written into the chain and received by the IBC module could be slightly smaller.
Discussion
What are the proposed benefits to aggregated signatures?
Reduce Block Size
At the moment, a commit contains a 64-byte (512-bit) signature for each validator that voted for the block. For the Cosmos Hub, which has 175 validators in the active set, this amounts to about 11 KiB per block. That gives an upper bound of around 113 GiB over the lifetime of the chain's 10.12M blocks. (Note, the Hub has increased the number of validators in the active set over time so the total signature size over the history of the chain is likely somewhat less than that).
Signature aggregation would only produce two signatures for the entire block. One for the yeas and one for the nays. Each BLS aggregated signature is 48 bytes, per the IETF standard of BLS signatures. Over the lifetime of the same Cosmos Hub chain, that would amount to about 1 GB, a savings of 112 GB. While that is a large factor of reduction it's worth bearing in mind that, at GCP's cost of $.026 USD per GB, that is a total savings of around $2.50 per month.
Reduce Signature Creation and Verification Time
From the IETF draft standard on BLS Signatures, BLS signatures can be
created in 370 microseconds and verified in 2700 microseconds. Our current
Ed25519 implementation was benchmarked locally to take
13.9 microseconds to produce a signature and 2.03 milliseconds to batch verify
128 signatures, which is slightly fewer than the 175 in the Hub. blst, a popular
implementation of BLS signature aggregation was benchmarked to perform verification
on 100 signatures in 1.5 milliseconds when run locally
on an 8 thread machine and pre-aggregated public keys. It is worth noting that
the ed25519 library verification time grew steadily with the number of signatures,
whereas the bls library verification time remains constant. This is because the
number of operations used to verify a signature does not grow at all with the
number of signatures included in the aggregate signature (as long as the signers
signed over the same message data as is the case in Tenderdash).
It is worth noting that this would also represent a degredation in signature verification time for chains with small validator sets. When batch verifying only 32 signatures, our ed25519 library takes .57 milliseconds, whereas BLS would still require the same 1.5 milliseconds.
For massive validator sets, blst dominates, taking the same 1.5 milliseconds to check an aggregated signature from 1024 validators versus our ed25519 library's 13.066 milliseconds to batch verify a set of that size.
Reduce Light-Client Verification Time
The light client aims to be a faster and lighter-weight way to verify that a block was voted on by a Tenderdash network. The light client fetches Tenderdash block headers and commit signatures, performing public key verification to ensure that the associated validator set signed the block. Reducing the size of the commit signature would allow the light client to fetch block data more quickly.
Additionally, the faster signature verification times of BLS signatures mean that light client verification would proceed more quickly.
However, verification of an aggregated signature is all-or-nothing. The verifier cannot check that some singular signer had a signature included in the block. Instead, the verifier must use all public keys to check if some signature was included. This does mean that any light client implementation must always be able to fetch all public keys for any height instead of potentially being able to check if some singular validator's key signed the block.
Reduce Gossip Bandwidth
Vote Gossip
It is possible to aggregate subsets of signatures during voting, so that the network need not gossip all n validator signatures to all n validators. Theoretically, subsets of the signatures could be aggregated during consensus and vote messages could carry those aggregated signatures. Implementing this would certainly increase the complexity of the gossip layer but could possibly reduce the total number of signatures required to be verified by each validator.
Block Gossip
A reduction in the block size as a result of signature aggregation would naturally lead to a reduction in the bandwidth required to gossip a block. Each validator would only send and receive the smaller aggregated signatures instead of the full list of multi-signatures as we have them now.
What are the drawbacks to aggregated signatures?
Heterogeneous key types cannot be aggregated
Aggregation requires a specific signature algorithm, and our legacy signing schemes cannot be aggregated. In practice, this means that aggregated signatures could be created for a subset of validators using BLS signatures, and validators with other key types (such as Ed25519) would still have to be be separately propagated in blocks and votes.
Many HSMs do not support aggregated signatures
Hardware Signing Modules (HSM) are a popular way to manage private keys. They provide additional security for key management and should be used when possible for storing highly sensitive private key material.
Below is a list of popular HSMs along with their support for BLS signatures.
- YubiKey
- Amazon Cloud HSM
- Ledger
I cannot find support listed for Google Cloud, although perhaps it exists.
Feasibility of implementation
This section outlines the various hurdles that would exist to implementing BLS signature aggregation into Tenderdash. It aims to demonstrate that we could implement BLS signatures but that it would incur risk and require breaking changes for a reasonably unclear benefit.
Can aggregated signatures be added as soft-upgrades?
In my estimation, yes. With the implementation of proposer-based timestamps, all validators now produce signatures on only one of two messages:
- A CanonicalVote where the BlockID is the hash of the block or
- A
CanonicalVotewhere theBlockIDis nil.
The block structure can be updated to perform hashing and validation in a new
way as a soft upgrade. This would look like adding a new section to the Block.Commit structure
alongside the current Commit.Signatures field. This new field, tentatively named
AggregatedSignature would contain the following structure:
message AggregatedSignature {
// yeas is a BitArray representing which validators in the active validator
// set issued a 'yea' vote for the block.
tendermint.libs.bits.BitArray yeas = 1;
// absent is a BitArray representing which validators in the active
// validator set did not issue votes for the block.
tendermint.libs.bits.BitArray absent = 2;
// yea_signature is an aggregated signature produced from all of the vote
// signatures for the block.
repeated bytes yea_signature = 3;
// nay_signature is an aggregated signature produced from all of the vote
// signatures from votes for 'nil' for this block.
// nay_signature should be made from all of the validators that were both not
// in the 'yeas' BitArray and not in the 'absent' BitArray.
repeated bytes nay_signature = 4;
}
Adding this new field as a soft upgrade would mean hashing this data structure
into the blockID along with the old Commit.Signatures when both are present
as well as ensuring that the voting power represented in the new
AggregatedSignature and Signatures field was enough to commit the block
during block validation. One can certainly imagine other possible schemes for
implementing this but the above should serve as a simple enough proof of concept.
Implementing vote-time and commit-time signature aggregation separately
Implementing aggregated BLS signatures as part of the block structure can easily be achieved without implementing any 'vote-time' signature aggregation. The block proposer would gather all of the votes, complete with signatures, as it does now, and produce a set of aggregate signatures from all of the individual vote signatures.
Implementing 'vote-time' signature aggregation cannot be achieved without also implementing commit-time signature aggregation. This is because such signatures cannot be dis-aggregated into their constituent pieces. Therefore, in order to implement 'vote-time' signature aggregation, we would need to either first implement 'commit-time' signature aggregation, or implement both 'vote-time' signature aggregation while also updating the block creation and verification protocols to allow for aggregated signatures.
Updating IBC clients
In order for IBC clients to function, they must be able to perform light-client verification of blocks on counterparty chains. Because BLS signatures are not currently part of light-clients, chains that transmit messages over IBC cannot update to using BLS signatures without their counterparties first being upgraded to parse and verify BLS. If chains upgrade without their counterparties first updating, they will lose the ability to interoperate with non-updated chains.
New attack surfaces
BLS signatures and signature aggregation comes with a new set of attack surfaces. Additionally, it's not clear that all possible major attacks are currently known on the BLS aggregation schemes since new ones have been discovered since the ietf draft standard was written. The known attacks are manageable and are listed below. Our implementation would need to prevent against these but this does not appear to present a significant hurdle to implementation.
Rogue key attack prevention
Generating an aggregated signature requires guarding against what is called a rogue key attack. A rogue key attack is one in which a malicious actor can craft an aggregate key that can produce signatures that appear to include a signature from a private key that the malicious actor does not actually know. In Tenderdash terms, this would look like a Validator producing a vote signed by both itself and some other validator where the other validator did not actually produce the vote itself.
The main mechanisms for preventing this require that each entity prove that it can can sign data with just their private key. The options involve either ensuring that each entity sign a different message when producing every signature or producing a proof of possession (PoP) when announcing their key to the network.
A PoP is a message that demonstrates ownership of a private key. A simple scheme for PoP is one where the entity announcing its new public key to the network includes a digital signature over the bytes of the public key generated using the associated private key. Everyone receiving the public key and associated proof-of-possession can easily verify the signature and be sure the entity owns the private key.
This PoP scheme suits the Tenderdash use case quite well since validator keys change infrequently so the associated PoPs would not be onerous to produce, verify, and store. Using this scheme allows signature verification to proceed more quickly, since all signatures are over identical data and can therefore be checked using an aggregated public key instead of one at a time, public key by public key.
Summing Zero Attacks
Summing zero attacks are attacks that rely on using the '0' point of an elliptic curve. For BLS signatures, if the point 0 is chosen as the private key, then the 0 point will also always be the public key and all signatures produced by the key will also be the 0 point. This is easy enough to detect when verifying each signature individually.
However, because BLS signature aggregation creates an aggregated signature and an aggregated public key, a set of colluding signers can create a pair or set of signatures that are non-zero but which aggregate ("sum") to 0. The signatures that sum zero along with the summed public key of the colluding signers will verify any message. This would allow the colluding signers to sign any block or message with the same signature. This would be reasonably easy to detect and create evidence for because, in all other cases, the same signature should not verify more than message. It's not exactly clear how such an attack would advantage the colluding validators because the normal mechanisms of evidence gathering would still detect the double signing, regardless of the signatures on both blocks being identical.
Backwards Compatibility
Backwards compatibility is an important consideration for signature verification. Specifically, it is important to consider whether chains using current versions of IBC would be able to interact with chains adopting BLS.
Because the Block shared by IBC and Tenderdash is produced and parsed using
protobuf, new structures can be added to the Block without breaking the
ability of legacy users to parse the new structure. Breaking changes between
current users of IBC and new Tenderdash blocks only occur if data that is
relied upon by the current users is no longer included in the current fields.
For the case of BLS aggregated signatures, a new AggregatedSignature field
can therefore be added to the Commit field without breaking current users.
Current users will be broken when counterparty chains upgrade to the new version
and begin using BLS signatures. Once counterparty chains begin using BLS
signatures, the BlockID hashes will include hashes of the AggregatedSignature
data structure that the legacy users will not be able to compute. Additionally,
the legacy software will not be able to parse and verify the signatures to
ensure that a supermajority of validators from the counterparty chain signed
the block.
Library Support
Libraries for BLS signature creation are limited in number, although active development appears to be ongoing. Cryptographic algorithms are difficult to implement correctly and correctness issues are extremely serious and dangerous. No further exploration of BLS should be undertaken without strong assurance of a well-tested library with continuing support for creating and verifying BLS signatures.
At the moment, there is one candidate, blst, that appears to be the most
mature and well vetted. While this library is undergoing continuing auditing
and is supported by funds from the Ethereum foundation, adopting a new cryptographic
library presents some serious risks. Namely, if the support for the library were
to be discontinued, Tenderdash may become saddled with the requirement of supporting
a very complex piece of software or force a massive ecosystem-wide migration away
from BLS signatures.
This is one of the more serious reasons to avoid adopting BLS signatures at this time. There is no gold standard library. Some projects look promising, but no project has been formally verified with a long term promise of being supported well into the future.
Go Standard Library
The Go Standard library has no implementation of BLS signatures.
BLST
blst, or 'blast' is an implementation of BLS signatures written in C that provides bindings into Go as part of the repository. This library is actively undergoing formal verification by Galois and previously received an initial audit by NCC group, a firm I'd never heard of.
blst is targeted for use in prysm, the golang implementation of Ethereum 2.0.
Gnark-Crypto
Gnark-Crypto is a Go-native implementation of elliptic-curve pairing-based cryptography. It is not audited and is documented as 'as-is', although development appears to be active so formal verification may be forthcoming.
CIRCL
CIRCL is a go-native implementation of several cryptographic primitives, bls12-381 among them. The library is written and maintained by Cloudflare and appears to receive frequent contributions. However, it lists itself as experimental and urges users to take caution before using it in production.
Added complexity to light client verification
Implementing BLS signature aggregation in Tenderdash would pose issues for the light client. The light client currently validates a subset of the signatures on a block when performing the verification algorithm. This is no longer possible with an aggregated signature. Aggregated signature verification is all-or-nothing. The light client could no longer check that a subset of validators from some set of validators is represented in the signature. Instead, it would need to create a new aggregated key with all the stated signers for each height it verified where the validator set changed.
This means that the speed advantages gained by using BLS cannot be fully realized by the light client since the client needs to perform the expensive operation of re-aggregating the public key. Aggregation is not constant time in the number of keys and instead grows linearly. When benchmarked locally, blst public key aggregation of 128 keys took 2.43 milliseconds. This, along with the 1.5 milliseconds to verify a signature would raise light client signature verification time to 3.9 milliseconds, a time above the previously mentioned batch verification time using our ed25519 library of 2.0 milliseconds.
Schemes to cache aggregated subsets of keys could certainly cut this time down at the cost of adding complexity to the light client.
Added complexity to evidence handling
Implementing BLS signature aggregation in Tenderdash would add complexity to the evidence handling within Tenderdash. Currently, the light client can submit evidence of a fork attempt to the chain. This evidence consists of the set of validators that double-signed, including their public keys, with the conflicting block.
We can quickly check that the listed validators double signed by verifying that each of their signatures are in the submitted conflicting block. A BLS signature scheme would change this by requiring the light client to submit the public keys of all of the validators that signed the conflicting block so that the aggregated signature may be checked against the full signature set. Again, aggregated signature verification is all-or-nothing, so without all of the public keys, we cannot verify the signature at all. These keys would be retrievable. Any party that wanted to create a fork would want to convince a network that its fork is legitimate, so it would need to gossip the public keys. This does not hamper the feasibility of implementing BLS signature aggregation into Tenderdash, but does represent yet another piece of added complexity to the associated protocols.
Open Questions
- Q: Can you aggregate Ed25519 signatures in Tenderdash?
- There is a suggested scheme in github issue 7892, but additional rigor would be required to fully verify its correctness.
Current Consideration
Adopting a signature aggregation scheme presents some serious risks and costs to the Tenderdash project. It requires multiple backwards-incompatible changes to the code, namely a change in the structure of the block and a new backwards-incompatible signature and key type. It risks adding a new signature type for which new attack types are still being discovered and for which no industry standard, battle-tested library yet exists.
The gains boasted by this new signing scheme are modest: Verification time is marginally faster and block sizes shrink by a few kilobytes. These are relatively minor gains in exchange for the complexity of the change and the listed risks of the technology. We should take a wait-and-see approach to BLS signature aggregation, monitoring the up-and-coming projects and consider implementing it as the libraries and standards develop.
References
RFC 019: Configuration File Versioning
Historical upstream design record. Project and release names below identify the original implementation; they do not describe Tenderdash releases. See the current protocol specification for Tenderdash.
Changelog
- 19-Apr-2022: Initial draft (@creachadair)
- 20-Apr-2022: Updates from review feedback (@creachadair)
Abstract
Updating configuration settings is an essential part of upgrading an existing node to a new version of the Tendermint software. Unfortunately, it is also currently a very manual process. This document discusses some of the history of changes to the config format, actions we've taken to improve the tooling for configuration upgrades, and additional steps we may want to consider.
Background
A Tendermint node reads configuration settings at startup from a TOML formatted
text file, typically named config.toml. The contents of this file are defined
by the github.com/tendermint/tendermint/config.
Although many settings in this file remain valid from one version of Tendermint to the next, new versions of Tendermint often add, update, and remove settings. These changes often require manual intervention by operators who are upgrading their nodes.
I propose we should provide better tools and documentation to help operators make configuration changes correctly during version upgrades. Ideally, as much as possible of any configuration file update should be automated, and where that is not possible or practical, we should provide clear, explicit directions for what steps need to be taken manually. Moreover, when the node discovers incorrect or invalid configuration, we should improve the diagnostics it emits so that the operator can quickly and easily find the relevant documentation, without having to grep through source code.
Discussion
By convention, we are supposed to document required changes to the config file
in the UPGRADING.md file for the release that introduces them. Although we
have mostly done this, the level of detail in the upgrading instructions is
often insufficient for an operator to correctly update their file.
The updates vary widely in complexity: Operators may need to add new required settings, update obsolete values for existing settings, move or rename existing settings within the file, or remove obsolete settings (which are thus invalid). Here are a few examples of each of these cases:
-
New required settings: Tendermint v0.35 added a new top-level
modesetting that determines whether a node runs as a validator, a full node, or a seed node. The default value is"full", which means the operator of a validator must manually addmode = "validator"(or set the--modeflag on the command line) for their node to come up in the correct mode. -
Updated obsolete values: Tendermint v0.35 removed support for versions
"v1"and"v2"of the blocksync (formerly "fastsync") protocol, requiring any node using either of those values to update to"v0". -
Moved/renamed settings: Version v0.34 moved the top-level
pprof_laddrsetting under the[rpc]section.Version v0.35 renamed every setting in the file from
snake_casetokebab-case, moved the top-levelfast_syncsetting into the[blocksync]section as (itself renamed from[fastsync]), and moved all the top-levelpriv-validator-*settings under a new[priv-validator]section with their prefix trimmed off. -
Removed obsolete settings: Version v0.34 removed the
index_all_keysandindex_keyssettings from the[tx_index]section; version v0.35 removed thewal-dirsetting from the[mempool]section, and version v0.36 removed the[blocksync]section entirely.
While many of these changes are mentioned in the config section of the upgrade
instructions, some are not mentioned at all, or are hidden in other parts of
the doc. For instance, the v0.34 pprof_laddr change was documented only as an
RPC flag change. (A savvy reader might realize that the flag --rpc.pprof_laddr
implies a corresponding config section, but it omits the related detail that
there was a top-level setting that's been renamed). The lesson here is not
that the docs are bad, but to point out that prose is not the most efficient
format to convey detailed changes like this. The upgrading instructions are
still valuable for the human reader to understand what to expect.
Concrete Steps
As part of the v0.36 development cycle, we spent some time reverse-engineering
the configuration changes since the v0.34 release and built an experimental
command-line tool called confix, whose job it is to automatically
update the settings in a config.toml file to the latest version. We also
backported a version of this tool into the v0.35.x branch at release v0.35.4.
This tool should work fine for configuration files created by Tendermint v0.34 and later, but does not (yet) know how to handle changes from prior versions of Tendermint. Part of the difficulty for older versions is simply logistical: To figure out which changes to apply, we need to understand something about the version that made the file, as well as the version we're converting it to.
Discussion point: In the future we might want to consider incorporating this into the node CLI directly, but we're keeping it separate for now until we can get some feedback from operators.
For the experiment, we handled this by carefully searching the history of
config format changes for shibboleths to bound the version: For example, the
[fastsync] section was added in Tendermint v0.32 and renamed [blocksync] in
Tendermint v0.35. So if we see a [fastsync] section, we have some confidence
that the file was created by v0.32, v0.33, or v0.34.
But such signals are delicate: The [blocksync] section was removed in v0.36,
so if we do not find [fastsync], we cannot conclude from that alone that the
file is from v0.31 or earlier -- we have to look for corroborating details.
While such "sniffing" tactics are fine for an experiment, they aren't as robust
as we might like.
This is especially relevant for configuration files that may have already been manually upgraded across several versions by the time we are asked to update them again. Another related concern is that we'd like to make sure conversion is idempotent, so that it would be safe to rerun the tool over an already-converted file without breaking anything.
Config Versioning
One obvious tactic we could use for future releases is add a version marker to
the config file. This would give tools like confix (and the node itself) a
way to calibrate their expectations. Rather than being a version for the file
itself, however, this version marker would indicate which version of Tendermint
is needed to read the file.
Provisionally, this might look something like:
# THe minimum version of Tendermint compatible with the contents of
# this configuration file.
config-version = 'v0.35'
When initializing a new node, Tendermint would populate this field with its own
version (e.g., v0.36). When conducting an upgrade, tools like confix can
then use this to decide which conversions are valid, and then update the value
accordingly. After converting a file marked 'v0.35' to'v0.37', the
conversion tool sets the file's config-version to reflect its compatibility.
Discussion point: This example presumes we would keep config files compatible within a given release cycle, e.g., all of v0.36.x. We could also use patch numbers here, if we think there's some reason to permit changes that would require config file edits at that granularity. I don't think we should, but that's a design question to consider.
Upon seeing an up-to-date version marker, the conversion tool can simply exit with a diagnostic like "this file is already up-to-date", rather than sniffing the keyspace and potentially introducing errors. In addition, this would let a tool detect config files that are newer than the one it understands, and issue a safe diagnostic rather than doing something wrong. Plus, besides avoiding potentially unsafe conversions, this would also serve as human-readable documentation that the file is up-to-date for a given version.
Adding a config version would not address the problem of how to convert files created by older versions of Tendermint, but it would at least help us build more robust config tooling going forward.
Stability and Change
In light of the discussion so far, it is natural to examine why we make so many changes to the configuration file from one version to the next, and whether we could reduce friction by being more conservative about what we make configurable, what config changes we make over time, and how we roll them out.
Some changes, like renaming everything from snake case to kebab case, are entirely gratuitous. We could safely agree not to make those kinds of changes. Apart from that obvious case, however, many other configuration settings provide value to node operators in cases where there is no simple, universal setting that matches every application.
Taking a high-level view, there are several broad reasons why we might want to make changes to configuration settings:
-
Lessons learned: Configuration settings are a good way to try things out in production, before making more invasive changes to the consensus protocol.
For example, up until Tendermint v0.35, consensus timeouts were specified as per-node configuration settings (e.g.,
timeout-precommitet al.). This allowed operators to tune these values for the needs of their network, but had the downside that individually-misconfigured nodes could stall consensus.Based on that experience, these timeouts have been deprecated in Tendermint v0.36 and converted to consensus parameters, to be consistent across all nodes in the network.
-
Migration & experimentation: Introducing new features and updating old features can complicate migration for existing users of the software. Temporary or "experimental" configuration settings can be a valuable way to mitigate that friction.
For example, Tendermint v0.36 introduces a new RPC event subscription endpoint (see ADR 075) that will eventually replace the existing webwocket-based interface. To give users time to migrate, v0.36 adds an
experimental-disable-websocketsetting, defaulted tofalse, that allows operators to selectively disable the websocket API for testing purposes during the conversion. This setting is designed to be removed in v0.37, when the old interface is no longer supported. -
Ongoing maintenance: Sometimes configuration settings become obsolete, and the cost of removing them trades off against the potential risks of leaving a non-functional or deprecated knob hooked up indefinitely.
For example, Tendermint v0.35 deprecated two alternate implementations of the blocksync protocol, one of which was deleted entirely (
v1) and one of which was scheduled for removal (v2). Theblocksync.versionsetting, which had been added as a migration aid, became obsolete and needed to be updated.Despite our best intentions, sometimes engineering designs do not work out. It's just as important to leave room to back out of changes we have since reconsidered, as it is to support migrations forward onto new and improved code.
-
Clarity and legibility: Besides configuring the software, another important purpose of a config file is to document intent for the humans who operate and maintain the software. Operators need adjust settings to keep the node running, and developers need to know what options were in use when something goes wrong so they can diagnose and fix bugs. The legibility of a config file as a human artifact is also thus important.
For example, Tendermint v0.35 moved settings related to validator private keys from the top-level section of the configuration file to their own designated
[priv-validator]section. Although this change did not make any difference to the meaning of those settings, it made the organization of the file easier to understand, and allowed the names of the individual settings to be simplified (e.g.,priv-validator-key-filebecame simplykey-filein the new section).Although such changes are "gratuitous" with respect to the software, there is often value in making things more legible for the humans. While there is no simple rule to define the line, the Potter Stewart principle can be used with due care.
Keeping these examples in mind, we can and should take reasonable steps to avoid churn in the configuration file across versions where we can. However, we must also accept that part of the reason for having a config file is to allow us flexibility elsewhere in the design. On that basis, we should not attempt to be too dogmatic about config changes either. Unlike changes in the block protocol, for example, which affect every user of every network that adopts them, config changes are relatively self-contained.
There are few guiding principles I think we can use to strike a sensible balance:
-
No gratuitous changes. Aesthetic changes that do not enhance legibility, avert confusion, or clarity documentation, should be entirely avoided.
-
Prefer mechanical changes. Whenever it is practical, change settings in a way that can be updated by a tool without operator judgement. This implies finding safe, universal defaults for new settings, and not changing the default values of existing settings.
Even if that means we have to make multiple changes (e.g., add a new setting in the current version, deprecate the old one, and remove the old one in the next version) it's preferable if we can mechanize each step.
-
Clearly signal intent. When adding temporary or experimental settings, they should be clearly named and documented as such. Use long names and suggestive prefixes (e.g.,
experimental-*) so that they stand out when read in the config file or printed in logs.Relatedly, using temporary or experimental settings should cause the software to emit diagnostic logs at runtime. These log messages should be easy to grep for, and should contain pointers to more complete documentation (say, issue numbers or URLs) that the operator can read, as well as a hint about when the setting is expected to become invalid. For example:
WARNING: Websocket RPC access is deprecated and will be removed in Tendermint v0.37. See https://tinyurl.com/adr075 for more information. -
Consider both directions. When adding a configuration setting, take some time during the implementation process to think about how the setting could be removed, as well as how it will be rolled out. This applies even for settings we imagine should be permanent. Experience may cause is to rethink our original design intent more broadly than we expected.
This does not mean we have to spend a long time picking nits over the design of every setting; merely that we should convince ourselves we could undo it without making too big a mess later. Even a little extra effort up front can sometimes save a lot.
References
- Tendermint
configpackage confixcommand-line toolcondiffcommand-line tool- Configuration update plan
- ADR 075: RPC Event Subscription Interface
Appendix: Research Notes
Discovering when various configuration settings were added, updated, and removed turns out to be surprisingly tedious. To solve this puzzle, we had to answer the following questions:
-
What changes were made between v0.x and v0.y? This is further complicated by cases where we have backported config changes into the middle of an earlier release cycle (e.g.,
psql-connfrom v0.35.x into v0.34.13). -
When during the development cycle were those changes made? This allows us to recognize features that were backported into a previous release.
-
What were the default values of the changed settings, and did they change at all during or across the release boundary?
Each step of the configuration update plan is commented with a link to one or more PRs where that change was made. The sections below discuss how we found these references.
Tracking Changes Across Releases
To figure out what changed between two releases, we built a tool called
condiff, which performs a "keyspace" diff of two TOML documents.
This diff respects the structure of the TOML file, but ignores comments, blank
lines, and configuration values, so that we can see what was added and removed.
To use it, run:
go run ./scripts/confix/condiff old.toml new.toml
This tool works on any TOML documents, but for our purposes we needed
Tendermint config.toml files. The easiest way to get these is to build the
node binary for your version of interest, run tendermint init on a clean home
directory, and copy the generated config file out. The testdata
directory for the confix tool has configs generated from the heads of each
release branch from v0.31 through v0.35.
If you want to reproduce this yourself, it looks something like this:
# Example for Tendermint v0.32.
git checkout --track origin/v0.32.x
go get golang.org/x/sys/unix
go mod tidy
make build
rm -fr -- tmhome
./build/tendermint --home=tmhome init
cp tmhome/config/config.toml config-v32.toml
Be advised that the further back you go, the more idiosyncrasies you will encounter. For example, Tendermint v0.31 and earlier predate Go modules (v0.31 used dep), and lack backport branches. And you may need to do some editing of Makefile rules once you get back into the 20s.
Note that when diffing config files across the v0.34/v0.35 gap, the swap from
snake_case to kebab-case makes it look like everything changed. The
condiff tool has a -desnake flag that normalizes all the keys to kebab case
in both inputs before comparison.
Locating Additions and Deletions
To figure out when a configuration setting was added or removed, your tool of
choice is git bisect. The only tricky part is finding the endpoints for the
search. If the transition happened within a release, you can use that
release's backport branch as the endpoint (if it has one, e.g., v0.35.x).
However, the start point can be more problematic. The backport branches are not
ancestors of master or of each other, which means you need to find some point
in history prior to the change but still attached to the mainline. For recent
releases there is a dev root (e.g., v0.35.0-dev, v0.34.0-dev1, etc.). These
are not named consistently, but you can usually grep the output of git tag to
find them.
In the worst case you could try starting from the root commit of the repo, but that turns out not to work in all cases. We've done some branching shenanigans over the years that mean the root is not a direct ancestor of all our release branches. When you find this you will probably swear a lot. I did.
Once you have a start and end point (say, v0.35.0-dev and master), you can
bisect in the usual way. I use git grep on the config directory to check
whether the case I am looking for is present. For example, to find when the
[fastsync] section was removed:
# Setup:
git checkout master
git bisect start
git bisect bad # it's not present on tip of master.
git bisect good v0.34.0-dev1 # it was present at the start of v0.34.
# Now repeat this until it gives you a specific commit:
if git grep -q '\[fastsync\]' config ; then git bisect good ; else git bisect bad ; fi
The above example finds where a config was removed: To find where a setting was
added, do the same thing except reverse the sense of the test (if ! git grep -q ...).
RFC 021: The Future of the Socket Protocol
Inherited design record. This proposal provides design context, not a guarantee of current behavior. See the current protocol specification.
Changelog
- 19-May-2022: Initial draft (@creachadair)
- 19-Jul-2022: Converted from ADR to RFC (@creachadair)
Abstract
This RFC captures some technical discussion about the ABCI socket protocol that was originally documented to solicit an architectural decision. This topic was not high-enough priority as of this writing to justify making a final decision.
For that reason, the text of this RFC has the general structure of an ADR, but should be viewed primarily as a record of the issue for future reference.
Background
The Application Blockchain Interface (ABCI) is a client-server protocol used by the Tenderdash consensus engine to communicate with the application on whose behalf it performs state replication. There are currently three transport options available for ABCI applications:
-
In-process: Applications written in Go can be linked directly into the same binary as the consensus node. Such applications use a "local" ABCI connection, which exposes application methods to the node as direct function calls.
-
Socket protocol: Out-of-process applications may export the ABCI service via a custom socket protocol that sends requests and responses over a Unix-domain or TCP socket connection as length-prefixed protocol buffers. In Tenderdash, this is handled by the socket client.
-
gRPC: Out-of-process applications may export the ABCI service via gRPC. In Tenderdash, this is handled by the gRPC client.
Both the out-of-process options (2) and (3) have a long history in Tenderdash. The beginnings of the gRPC client were added in May 2016 when ABCI was still hosted in a separate repository, and the socket client (formerly called the "remote client") was part of ABCI from its inception in November 2015.
At that time when ABCI was first being developed, the gRPC project was very new (it launched Q4 2015) and it was not an obvious choice for use in Tenderdash. It took a while before the language coverage and quality of gRPC reached a point where it could be a viable solution for out-of-process applications. For that reason, it made sense for the initial design of ABCI to focus on a custom protocol for out-of-process applications.
Problem Statement
For practical reasons, ABCI needs an interprocess communication option to support applications not written in Go. The two practical options are RPC and FFI, and for operational reasons an RPC mechanism makes more sense.
The socket protocol has not changed all that substantially since its original design, and has the advantage of being simple to implement in almost any reasonable language. However, its simplicity includes some limitations that have had a negative impact on the stability and performance of out-of-process applications using it. In particular:
-
The protocol lacks request identifiers, so the client and server must return responses in strict FIFO order. Even if the client issues requests that have no dependency on each other, the protocol has no way except order of issue to map responses to requests.
This reduces (in some cases substantially) the concurrency an application can exploit, since the parallelism of requests in flight is gated by the slowest active request at any moment. There have been complaints from some network operators on that basis.
-
The protocol lacks method identifiers, so the only way for the client and server to understand which operation is requested is to dispatch on the type of the request and response payloads. For responses, this means that any error condition is terminal not only to the request, but to the entire ABCI client.
The historical intent of terminating for any error seems to have been that all ABCI errors are unrecoverable and hence protocol fatal (see Note 1). In practice, however, this greatly complicates debugging a faulty node, since the only way to respond to errors is to panic the node which loses valuable context that could have been logged.
-
There are subtle concurrency management dependencies between the client and the server that are not clearly documented anywhere, and it is very easy for small changes in both the client and the server to lead to tricky deadlocks, panics, race conditions, and slowdowns. As a recent example of this, see https://github.com/tendermint/tendermint/pull/8581.
These limitations are fixable, but one important question is whether it is worthwhile to fix them. We can add request and method identifiers, for example, but doing so would be a breaking change to the protocol requiring every application using it to update. If applications have to migrate anyway, the stability and language coverage of gRPC have improved a lot, and today it is probably simpler to set up and maintain an application using gRPC transport than to reimplement the Tenderdash socket protocol.
Moreover, gRPC addresses all the above issues out-of-the-box, and requires (much) less custom code for both the server (i.e., the application) and the client. The project is well-funded and widely-used, which makes it a safe bet for a dependency.
Decision
There is a set of related alternatives to consider:
-
Question 1: Designate a single IPC standard for out-of-process applications?
Claim: We should converge on one (and only one) IPC option for out-of-process applications. We should choose an option that, after a suitable period of deprecation for alternatives, will address most or all the highest-impact uses of Tenderdash. Maintaining multiple options increases the surface area for bugs and vulnerabilities, and we should not have multiple options for basic interfaces without a clear and well-documented reason.
-
Question 2a: Choose gRPC and deprecate/remove the socket protocol?
Claim: Maintaining and improving a custom RPC protocol is a substantial project and not directly relevant to the requirements of consensus. We would be better served by depending on a well-maintained open-source library like gRPC.
-
Question 2b: Improve the socket protocol and deprecate/remove gRPC?
Claim: If we find meaningful advantages to maintaining our own custom RPC protocol in Tenderdash, we should treat it as a first-class project within the core and invest in making it good enough that we do not require other options.
One important consideration when discussing these questions is that any outcome which includes keeping the socket protocol will have eventual migration impacts for out-of-process applications regardless. To fix the limitations of the socket protocol as it is currently designed will require making breaking changes to the protocol. So, while we may put off a migration cost for out-of-process applications by retaining the socket protocol in the short term, we will eventually have to pay those costs to fix the problems in its current design.
Detailed Design
-
If we choose to standardize on gRPC, the main work in Tenderdash core will be removing and cleaning up the code for the socket client and server.
Besides the code cleanup, we will also need to clearly document a deprecation schedule, and invest time in making the migration easier for applications currently using the socket protocol.
Point for discussion: Migrating from the socket protocol to gRPC should mostly be a plumbing change, as long as we do it during a release in which we are not making other breaking changes to ABCI. However, the effort may be more or less depending on how gRPC integration works in the application's implementation language, and would have to be sure networks have plenty of time not only to make the change but to verify that it preserves the function of the network.
What questions should we be asking node operators and application developers to understand the migration costs better?
-
If we choose to keep only the socket protocol, we will need to follow up with a more detailed design for extending and upgrading the protocol to fix the existing performance and operational issues with the protocol.
Moreover, since the gRPC interface has been around for a long time we will also need a deprecation plan for it.
-
If we choose to keep both options, we will still need to do all the work of (2), but the gRPC implementation should not require any immediate changes.
Alternatives Considered
-
FFI. Another approach we could take is to use a C-based FFI interface so that applications written in other languages are linked directly with the consensus node, an option currently only available for Go applications.
An FFI interface is possible for a lot of languages, but FFI support varies widely in coverage and quality across languages and the points of friction can be tricky to work around. Moreover, it's much harder to add FFI support to a language where it's missing after-the-fact for an application developer.
Although a basic FFI interface is not too difficult on the Go side, the C shims for an FFI can get complicated if there's a lot of variability in the runtime environment on the other end.
If we want to have one answer for non-Go applications, we are better off picking an IPC-based solution (whether that's gRPC or an extension of our custom socket protocol or something else).
Consequences
-
Standardize on gRPC
- ✅ Addresses existing performance and operational issues.
- ✅ Replaces custom code with a well-maintained widely-used library.
- ✅ Aligns with Cosmos SDK, which already uses gRPC extensively.
- ✅ Aligns with priv validator interface, for which the socket protocol is already deprecated for gRPC.
- ❓ Applications will be hard to implement in a language without gRPC support.
- ⛔ All users of the socket protocol have to migrate to gRPC, and we believe most current out-of-process applications use the socket protocol.
-
Standardize on socket protocol
- ✅ Less immediate impact for existing users (but see below).
- ✅ Simplifies ABCI API surface by removing gRPC.
- ❓ Users of the socket protocol will have a (smaller) migration.
- ❓ Potentially easier to implement for languages that do not have support.
- ⛔ Need to do all the work to fix the socket protocol (which will require existing users to update anyway later).
- ⛔ Ongoing maintenance burden for per-language server implementations.
-
Keep both options
- ✅ Less immediate impact for existing users (but see below).
- ❓ Users of the socket protocol will have a (smaller) migration.
- ⛔ Still need to do all the work to fix the socket protocol (which will require existing users to update anyway later).
- ⛔ Requires ongoing maintenance and support of both gRPC and socket protocol integrations.
References
- Application Blockchain Interface (ABCI)
- Tenderdash ABCI socket client
- Tenderdash ABCI gRPC client
- Initial commit of gRPC client
Notes
-
Note 1: The choice to make all ABCI errors protocol-fatal was intended to avoid the risk that recovering an application error could cause application state to diverge. Divergence can break consensus, so it's essential to avoid it.
This is a sound principle, but conflates protocol errors with "mechanical" errors such as timeouts, resoures exhaustion, failed connections, and so on. Because the protocol has no way to distinguish these conditions, the only way for an application to report an error is to panic or crash.
Whether a node is running in the same process as the application or as a separate process, application errors should not be suppressed or hidden. However, it's important to ensure that errors are handled at a consistent and well-defined point in the protocol: Having the application panic or crash rather than reporting an error means the node sees different results depending on whether the application runs in-process or out-of-process, even if the application logic is otherwise identical.
Appendix: Known Implementations of ABCI Socket Protocol
This is a list of known implementations of the Tendermint custom socket protocol. Note that in most cases I have not checked how complete or correct these implementations are; these are based on search results and a cursory visual inspection.
- Tenderdash Core (Go): client, server
- Informal Systems tendermint-rs (Rust): client, server
- Tendermint js-abci (JS): server
- Hotmoka ABCI (Java): server
- Tower ABCI (Rust): server
- abci-host (Clojure): server
- abci_server (Erlang): server
- py-abci (Python): server
- scala-tendermint-server (Scala; repository no longer available, archived copy)
- kepler (Rust): server
Consensus DoS — observability
A Grafana dashboard and Prometheus alert rules for watching the consensus DoS throttle, built to answer one question at a glance:
Is the node shedding the attack, or shedding its honest peers?
Files
grafana-dashboard.json— three-band dashboard (chain health / throttle firing / honest service). Import into Grafana or drop into a provisioning directory.prometheus-alerts.yml— alert rules that page on honest symptoms (chain struggling, honest-vote latency high), not on drop counters. Load alongside the deploy repo's existingChainHalt/RoundTooHighrules.
Reading it
The dashboard has three rows:
- Is the chain healthy? — block production rate, rounds, prevote delays.
- Is the throttle firing, and where? — drops by mechanism, evidence by reason. Rising here under attack is expected and good.
- Is honest service still landing? — accepted-vote latency (p50/p99), budget saturation, lane depth.
The judgement:
| middle band (drops) | bottom band (honest) | verdict |
|---|---|---|
| high | latency low, saturation off the floor, rounds flat | shedding the attack — healthy |
| high | latency climbing toward the vote/propose timeout, saturation pinned at 0, rounds climbing | starving honest work |
The single most important panel is accepted peer-vote latency p99. Threshold lines are drawn at the vote timeout (1 s) and propose timeout (3 s).
Deployment facts (dash-network-deploy)
- Metrics carry the namespace prefix
drive_tenderdash(set in dashmate'sconfig.toml.dot). If your deployment uses a different[instrumentation] namespace, adjust the metric prefixes in both files. - Prometheus scrapes HP masternodes at
:36660— seed nodes are not scraped (they expose the endpoint but no job targets them). The dashboard therefore reflects validator nodes. - The metrics endpoint is enabled by the deploy tool's
dashmate.json.j2override, not by vanilla dashmate (whose default is off / localhost). A first devnet needs no config change to observe these — the endpoint and scrape already exist. - Provisioning: the deploy repo's
roles/metrics/ships Prometheus + Grafana but no dashboards in git. Addgrafana-dashboard.jsonto a Grafana provisioning path (or import via the UI), and appendprometheus-alerts.ymlto the Prometheus rules the metrics role loads.
Metrics used
New with the DoS work — throttle firing:
consensus_verification_budget_drops, consensus_peer_lane_drops,
consensus_block_part_proof_drops, consensus_state_channel_drops,
consensus_proposal_verify_failures, evidence_pool_dropped_evidence{reason}.
New — honest service:
consensus_peer_vote_verify_latency_seconds (histogram),
consensus_verification_budget_saturation (gauge, 1 = idle → 0 = drained),
consensus_peer_lane_active_count, consensus_peer_lane_max_depth.
Stock, for chain health:
consensus_latest_block_height, consensus_rounds,
consensus_quorum_prevote_delay, consensus_full_prevote_delay.
(All shown without the drive_tenderdash_ namespace prefix that the deployment adds.)
Building the documentation
Tenderdash uses mdBook, like the Dash
Platform book. The site includes the guides in docs/, all Markdown specifications
in spec/, and the RPC reference from rpc/openapi/.
Local build
Install Python 3.10 or later and mdBook 0.4.40 (the version used by Platform):
cargo install mdbook --version 0.4.40 --locked
make build-docs
Run make serve-docs and open http://localhost:3000/. To serve an existing
build, run python3 -m http.server 8000 --directory book/book and open
http://localhost:8000/. Use HTTP so the RPC reference can fetch its schema.
MDBOOK=/path/to/mdbook make build-docs selects a particular binary.
For guides and specifications, edit their canonical files in docs/ or spec/,
then run the build again. When using the server, rerun make prepare-docs after
editing canonical files; mdBook watches the staged copies.
book/SUMMARY.md defines the navigation. Add every new Markdown page there;
the build fails if a source page is missing from the summary or a summary entry
has no source file. The HTML build also rejects missing local pages or assets.
The consensus protocol and all its companion chapters are
listed explicitly under Protocol specifications.
The preparation script copies sources and assets into ignored book/src/, strips
VuePress metadata, resolves directory links, and points links to repository source
files at the build's Git revision. It also stages the RPC reference at api/rpc/.
It does not switch branches or modify documentation sources. Both book/src/ and
book/book/ are disposable build output. Set DOCS_SOURCE_REF to override the
Git revision used for source links.
Diagrams and search
Use fenced mermaid blocks for diagrams. The book uses the same vendored
Mermaid 11.6.0 distribution as Platform; its MIT license is in
book/mermaid-LICENSE. Diagrams render locally without a CDN. mdBook provides
local full-text search without an external search service.
Preview and publishing
.github/workflows/docs-deployment.yml builds pull requests and uploads a
documentation-preview artifact. Download it and extract it, then run
python3 -m http.server 8000 from the extracted directory and open
http://localhost:8000/. The RPC reference requires HTTP serving.
Pull requests have read-only repository permissions and cannot deploy.
Pushes to the repository's default branch publish the same output using GitHub
Pages artifacts and the github-pages environment, like Platform. This follows
default-branch changes without editing the workflow.
To publish another branch manually, open the workflow in Actions, choose
Run workflow, and select the branch. The workflow must exist on the default
branch for GitHub to show that control. In Settings → Pages, select GitHub
Actions as the build source. The github-pages environment's deployment rules
must permit the default branch and any branches selected for manual publication.
The workflow reports the deployed URL. Ordinary pushes to other branches do not
publish; pull requests remain previews.
Naming and attribution
Use Tenderdash for this product, its CLI, and current source links. Preserve
copyright notices, original research titles and citations, external project names,
and identifiers required by the implementation, such as tendermint.abci, JSON
public-key type names, metric names, and TLA+/Ivy module names. Do not rename wire
types to change their display name.
Generate the ABCI API page from its protobuf source with make proto-doc;
spec/abci/api.md is generated and must not be edited by hand.