WAL

Consensus module writes every message to the WAL (write-ahead log), except proposer heartbeats and round skip votes; round skip certificates are written once verified. Which records are written, with which durability, and how they are replayed is specified in Recovery and synchronisation.

It also issues fsync syscall through File#Sync for messages signed by this node and for the end-of-height marker (to prevent double signing and to delimit heights).

Under the hood, it uses autofile.Group (internal/libs/autofile), which rotates files when those get too big (> 10MB, defaultHeadSizeLimit).

The total maximum size is 1GB (defaultTotalSizeLimit). We only need the latest block and the block before it, but if the former is dragging on across many rounds, we want all those rounds.

Replay

Consensus module will replay all the messages of the last height written to WAL before a crash (if such occurs).

The private validator may try to sign messages during replay because it runs somewhat autonomously and does not know about replay process.

For example, if we got all the way to precommit in the WAL and then crash, after we replay the proposal message, the private validator will try to sign a prevote. But it will fail. That's ok because we’ll see the prevote later in the WAL. Then it will go to precommit, and that time it will work because the private validator contains the LastSignBytes and then we’ll replay the precommit from the WAL.

Make sure to read about WAL corruption and recovery strategies.