Metrics
Tenderdash can report and serve the Prometheus metrics, which in their turn can be consumed by Prometheus collector(s).
This functionality is disabled by default.
To enable the Prometheus metrics, set instrumentation.prometheus=true in your
config file. Metrics will be served under /metrics on 26660 port by default.
Listen address can be changed in the config file (see
instrumentation.prometheus\_listen\_addr).
List of available metrics
The following metrics are available:
| Name | Type | Tags | Description |
|---|---|---|---|
| abci_connection_method_timing | Histogram | method, type | Timings for each of the ABCI methods |
| consensus_height | Gauge | Height of the chain | |
| consensus_validators | Gauge | Number of validators | |
| consensus_validators_power | Gauge | Total voting power of all validators | |
| consensus_validator_power | Gauge | Voting power of the node if in the validator set | |
| consensus_validator_last_signed_height | Gauge | Last height the node signed a block, if the node is a validator | |
| consensus_validator_missed_blocks | Gauge | Total amount of blocks missed for the node, if the node is a validator | |
| consensus_missing_validators | Gauge | Number of validators who did not sign | |
| consensus_missing_validators_power | Gauge | Total voting power of the missing validators | |
| consensus_byzantine_validators | Gauge | Number of validators who tried to double sign | |
| consensus_byzantine_validators_power | Gauge | Total voting power of the byzantine validators | |
| consensus_block_interval_seconds | Histogram | Time between this and last block (Block.Header.Time) in seconds | |
| consensus_rounds | Gauge | Number of rounds | |
| consensus_num_txs | Gauge | Number of transactions | |
| consensus_total_txs | Gauge | Total number of transactions committed | |
| consensus_block_parts | Counter | peer_id | number of blockparts transmitted by peer |
| consensus_latest_block_height | gauge | /status sync_info number | |
| consensus_fast_syncing | gauge | either 0 (not fast syncing) or 1 (syncing) | |
| consensus_state_syncing | gauge | either 0 (not state syncing) or 1 (syncing) | |
| consensus_block_size_bytes | Gauge | Block size in bytes | |
| consensus_step_duration | Histogram | step | Histogram of durations for each step in the consensus protocol |
| consensus_block_gossip_receive_latency | Histogram | Histogram of time taken to receive a block in seconds, measure between when a new block is first discovered to when the block is completed | |
| consensus_block_gossip_parts_received | Counter | matches_current | Number of block parts received by the node |
| consensus_block_sync_apply_stage_duration | Histogram | stage | Time a block spent in each stage of the block sync applier, in ms: decode, partset, verify_commit, verify_block, save, exec, and wait (idle time between blocks) |
| consensus_quorum_prevote_delay | Gauge | Interval in seconds between the proposal timestamp and the timestamp of the earliest prevote that achieved a quorum | |
| consensus_full_prevote_delay | Gauge | Interval in seconds between the proposal timestamp and the timestamp of the latest prevote in a round where all validators voted | |
| consensus_proposal_timestamp_difference | Histogram | Difference between the timestamp in the proposal message and the local time of the validator at the time it received the message | |
| consensus_vote_extension_receive_count | Counter | status | Number of vote extensions received |
| consensus_proposal_receive_count | Counter | status | Total number of proposals received by the node since process start |
| consensus_proposal_create_count | Counter | Total number of proposals created by the node since process start | |
| consensus_round_voting_power_percent | Gauge | vote_type | A value between 0 and 1.0 representing the percentage of the total voting power per vote type received within a round |
| consensus_late_votes | Counter | vote_type | Number of votes received by the node since process start that correspond to earlier heights and rounds than this node is currently in. |
| consensus_verification_budget_drops | Counter | Peer Vote and Commit messages dropped because the signature-verification budget was exhausted | |
| consensus_peer_lane_drops | Counter | Queued peer consensus messages dropped by the per-peer scheduler (backlog full or peer disconnected) | |
| consensus_block_part_proof_drops | Counter | Peer block parts dropped before proof verification because the sender's invalid-proof allowance was spent | |
| consensus_state_channel_drops | Counter | State and VoteSetBits channel messages dropped over the per-peer or node-wide ceiling | |
| consensus_proposal_verify_failures | Counter | Peer proposals whose signature failed verification | |
| consensus_commit_verify_failures | Counter | reason | Commits (from peers, block sync or this node's own precommits) refused by verification; reason is quorum_hash, extension_count, extensions_rejected, invalid_signature, budget or other |
| consensus_peer_vote_verify_latency_seconds | Histogram | Seconds an accepted peer vote spent from being queued by the reactor to being added to the vote set | |
| consensus_verification_budget_saturation | Gauge | Fraction of the node-wide verification budget still available at a peer-message check (1.0 full/idle, 0.0 drained/under pressure) | |
| consensus_peer_lane_active_count | Gauge | Number of per-peer lanes currently holding queued messages and taking turns in the scheduler | |
| consensus_peer_lane_max_depth | Gauge | Largest number of messages queued in any single peer lane | |
| consensus_proposer_heartbeats_sent | Counter | Proposer heartbeats this node signed and broadcast while waiting for transactions as proposer (consensus version 2) | |
| consensus_proposer_heartbeats_received | Counter | New, verified proposer heartbeats accepted for the round this node waits in | |
| consensus_proposer_silences | Counter | Rounds in which the proposer went silent while this validator waited for transactions | |
| consensus_round_skips | Counter | Rounds this node moved past on a round skip certificate from +2/3 of the validators (by default; configuration can change the threshold) | |
| evidence_pool_num_evidence | Gauge | Number of evidence in the evidence pool | |
| evidence_pool_dropped_evidence | Counter | reason | Inbound gossip evidence refused before verification, labeled by reason (structural, oversized, already_known, height_window, node_budget, peer_budget) |
| p2p_peers | Gauge | Number of peers node's connected to | |
| p2p_peer_receive_bytes_total | Counter | peer_id, chID | number of bytes per channel received from a given peer |
| p2p_peer_send_bytes_total | Counter | peer_id, chID | number of bytes per channel sent to a given peer |
| p2p_peer_pending_send_bytes | Gauge | peer_id | number of pending bytes to be sent to a given peer |
| p2p_router_peer_queue_recv | Histogram | The time taken to read off of a peer's queue before sending on the connection | |
| p2p_router_peer_queue_send | Histogram | The time taken to send on a peer's queue which will later be sent on the connection | |
| p2p_router_channel_queue_send | Histogram | The time taken to send on a p2p channel's queue which will later be consumed by the corresponding service | |
| p2p_router_channel_queue_dropped_msgs | Counter | ch_id | The number of messages dropped from a peer's queue for a specific p2p channel |
| p2p_peer_queue_msg_size | Gauge | ch_id | The size of messages sent over a peer's queue for a specific p2p channel |
| mempool_size | Gauge | Number of uncommitted transactions | |
| mempool_tx_size_bytes | Histogram | transaction sizes in bytes | |
| mempool_failed_txs | Counter | number of failed transactions | |
| mempool_recheck_times | Counter | number of transactions rechecked in the mempool | |
| state_block_processing_time | Histogram | time between BeginBlock and EndBlock in ms | |
| state_consensus_param_updates | Counter | number of consensus parameter updates returned by the application since process start | |
| state_validator_set_updates | Counter | number of validator set updates returned by the application since process start | |
| state_block_apply_stage_duration | Histogram | stage | Time spent in each stage of applying a committed block, in ms: the ABCI calls, the state and ABCI response writes, the mempool update and the event publish |
Useful queries
Percentage of missing + byzantine validators:
((consensus_byzantine_validators_power + consensus_missing_validators_power) / consensus_validators_power) * 100
Rate at which the application is responding to each ABCI method call.
sum(rate(tendermint_abci_connection_method_timing_count[5m])) by (method)
The 95th percentile response time for the application to the deliver_tx ABCI method call.
histogram_quantile(0.95, sum by(le) (rate(tendermint_abci_connection_method_timing_bucket{method="deliver_tx"}[5m])))