Metrics

Tenderdash can report and serve the Prometheus metrics, which in their turn can be consumed by Prometheus collector(s).

This functionality is disabled by default.

To enable the Prometheus metrics, set instrumentation.prometheus=true in your config file. Metrics will be served under /metrics on 26660 port by default. Listen address can be changed in the config file (see instrumentation.prometheus\_listen\_addr).

List of available metrics

The following metrics are available:

NameTypeTagsDescription
abci_connection_method_timingHistogrammethod, typeTimings for each of the ABCI methods
consensus_heightGaugeHeight of the chain
consensus_validatorsGaugeNumber of validators
consensus_validators_powerGaugeTotal voting power of all validators
consensus_validator_powerGaugeVoting power of the node if in the validator set
consensus_validator_last_signed_heightGaugeLast height the node signed a block, if the node is a validator
consensus_validator_missed_blocksGaugeTotal amount of blocks missed for the node, if the node is a validator
consensus_missing_validatorsGaugeNumber of validators who did not sign
consensus_missing_validators_powerGaugeTotal voting power of the missing validators
consensus_byzantine_validatorsGaugeNumber of validators who tried to double sign
consensus_byzantine_validators_powerGaugeTotal voting power of the byzantine validators
consensus_block_interval_secondsHistogramTime between this and last block (Block.Header.Time) in seconds
consensus_roundsGaugeNumber of rounds
consensus_num_txsGaugeNumber of transactions
consensus_total_txsGaugeTotal number of transactions committed
consensus_block_partsCounterpeer_idnumber of blockparts transmitted by peer
consensus_latest_block_heightgauge/status sync_info number
consensus_fast_syncinggaugeeither 0 (not fast syncing) or 1 (syncing)
consensus_state_syncinggaugeeither 0 (not state syncing) or 1 (syncing)
consensus_block_size_bytesGaugeBlock size in bytes
consensus_step_durationHistogramstepHistogram of durations for each step in the consensus protocol
consensus_block_gossip_receive_latencyHistogramHistogram of time taken to receive a block in seconds, measure between when a new block is first discovered to when the block is completed
consensus_block_gossip_parts_receivedCountermatches_currentNumber of block parts received by the node
consensus_block_sync_apply_stage_durationHistogramstageTime a block spent in each stage of the block sync applier, in ms: decode, partset, verify_commit, verify_block, save, exec, and wait (idle time between blocks)
consensus_quorum_prevote_delayGaugeInterval in seconds between the proposal timestamp and the timestamp of the earliest prevote that achieved a quorum
consensus_full_prevote_delayGaugeInterval in seconds between the proposal timestamp and the timestamp of the latest prevote in a round where all validators voted
consensus_proposal_timestamp_differenceHistogramDifference between the timestamp in the proposal message and the local time of the validator at the time it received the message
consensus_vote_extension_receive_countCounterstatusNumber of vote extensions received
consensus_proposal_receive_countCounterstatusTotal number of proposals received by the node since process start
consensus_proposal_create_countCounterTotal number of proposals created by the node since process start
consensus_round_voting_power_percentGaugevote_typeA value between 0 and 1.0 representing the percentage of the total voting power per vote type received within a round
consensus_late_votesCountervote_typeNumber of votes received by the node since process start that correspond to earlier heights and rounds than this node is currently in.
consensus_verification_budget_dropsCounterPeer Vote and Commit messages dropped because the signature-verification budget was exhausted
consensus_peer_lane_dropsCounterQueued peer consensus messages dropped by the per-peer scheduler (backlog full or peer disconnected)
consensus_block_part_proof_dropsCounterPeer block parts dropped before proof verification because the sender's invalid-proof allowance was spent
consensus_state_channel_dropsCounterState and VoteSetBits channel messages dropped over the per-peer or node-wide ceiling
consensus_proposal_verify_failuresCounterPeer proposals whose signature failed verification
consensus_commit_verify_failuresCounterreasonCommits (from peers, block sync or this node's own precommits) refused by verification; reason is quorum_hash, extension_count, extensions_rejected, invalid_signature, budget or other
consensus_peer_vote_verify_latency_secondsHistogramSeconds an accepted peer vote spent from being queued by the reactor to being added to the vote set
consensus_verification_budget_saturationGaugeFraction of the node-wide verification budget still available at a peer-message check (1.0 full/idle, 0.0 drained/under pressure)
consensus_peer_lane_active_countGaugeNumber of per-peer lanes currently holding queued messages and taking turns in the scheduler
consensus_peer_lane_max_depthGaugeLargest number of messages queued in any single peer lane
consensus_proposer_heartbeats_sentCounterProposer heartbeats this node signed and broadcast while waiting for transactions as proposer (consensus version 2)
consensus_proposer_heartbeats_receivedCounterNew, verified proposer heartbeats accepted for the round this node waits in
consensus_proposer_silencesCounterRounds in which the proposer went silent while this validator waited for transactions
consensus_round_skipsCounterRounds this node moved past on a round skip certificate from +2/3 of the validators (by default; configuration can change the threshold)
evidence_pool_num_evidenceGaugeNumber of evidence in the evidence pool
evidence_pool_dropped_evidenceCounterreasonInbound gossip evidence refused before verification, labeled by reason (structural, oversized, already_known, height_window, node_budget, peer_budget)
p2p_peersGaugeNumber of peers node's connected to
p2p_peer_receive_bytes_totalCounterpeer_id, chIDnumber of bytes per channel received from a given peer
p2p_peer_send_bytes_totalCounterpeer_id, chIDnumber of bytes per channel sent to a given peer
p2p_peer_pending_send_bytesGaugepeer_idnumber of pending bytes to be sent to a given peer
p2p_router_peer_queue_recvHistogramThe time taken to read off of a peer's queue before sending on the connection
p2p_router_peer_queue_sendHistogramThe time taken to send on a peer's queue which will later be sent on the connection
p2p_router_channel_queue_sendHistogramThe time taken to send on a p2p channel's queue which will later be consumed by the corresponding service
p2p_router_channel_queue_dropped_msgsCounterch_idThe number of messages dropped from a peer's queue for a specific p2p channel
p2p_peer_queue_msg_sizeGaugech_idThe size of messages sent over a peer's queue for a specific p2p channel
mempool_sizeGaugeNumber of uncommitted transactions
mempool_tx_size_bytesHistogramtransaction sizes in bytes
mempool_failed_txsCounternumber of failed transactions
mempool_recheck_timesCounternumber of transactions rechecked in the mempool
state_block_processing_timeHistogramtime between BeginBlock and EndBlock in ms
state_consensus_param_updatesCounternumber of consensus parameter updates returned by the application since process start
state_validator_set_updatesCounternumber of validator set updates returned by the application since process start
state_block_apply_stage_durationHistogramstageTime spent in each stage of applying a committed block, in ms: the ABCI calls, the state and ABCI response writes, the mempool update and the event publish

Useful queries

Percentage of missing + byzantine validators:

((consensus_byzantine_validators_power + consensus_missing_validators_power) / consensus_validators_power) * 100

Rate at which the application is responding to each ABCI method call.

sum(rate(tendermint_abci_connection_method_timing_count[5m])) by (method)

The 95th percentile response time for the application to the deliver_tx ABCI method call.

histogram_quantile(0.95, sum by(le) (rate(tendermint_abci_connection_method_timing_bucket{method="deliver_tx"}[5m])))