Recovery and synchronisation

Write-ahead log

The consensus WAL (internal/consensus/wal.go; file layout in WAL) records, per height:

RecordWritten byDurability
msgInfo — proposal, block part, vote or commit, with the peer id and receive timewalMiddleware (internal/consensus/msg_handlers.go)peer messages Write (buffered); the node's own messages WriteSync (fsync), panic on failure
msgInfo — round skip certificate, once verified and newsAddRoundSkipCertificateAction.ExecuteWrite; a failure is only logged
timeoutInfo — every tockState.receiveRoutineWrite
EventDataRoundState — every step changeEventPublisher.PublishNewRoundStepEventWrite
EndHeightMessage{h}ApplyCommitAction.Execute, after SaveBlock and before FinalizeBlockWriteSync, panic on failure

verificationBudgetMiddleware wraps walMiddleware, so a message dropped for budget reasons leaves no WAL record. Proposer heartbeats and round skip votes are never written (livenessMessage). Before signing a proposal or a vote the WAL is flushed and fsynced (EnterProposeAction.Execute, voteSigner.signVote) so that replay reproduces the same message for the private validator's double-sign protection. A failed sync stops the vote (signVote returns the error, nothing is signed) but not the proposal: EnterProposeAction.Execute logs it and proposes anyway. An empty WAL starts with EndHeightMessage{0} (BaseWAL.OnStart).

Start-up: ABCI handshake

Handshaker.Handshake (internal/consensus/replay.go) calls ABCI Info and BlockReplayer.Replay (internal/consensus/replayer.go) with the application's (LastBlockHeight, LastBlockAppHash):

  1. If the application height is 0: InitChain with the genesis validator set, parameters, InitialCoreHeight; when Tenderdash itself is at height 0 the returned validator set, parameters, app hash and optional genesis time become the initial state (execInitChain).
  2. replayState.validate: the application may not be ahead of the block store; the state may not be ahead of the store; the store may be at most one block ahead of the state; a truncated store must not have pruned the blocks the application needs.
  3. Cases (syncStateIfItIsEqualStore, syncStateIfItIsOneAheadOfStore):
Store vs stateApp vs stateAction
equalbehindreplay stored blocks into the application: per block ProcessProposal(round = commit.Round), VerifyCommitExtensions, FinalizeBlock (replayBlock); the resulting app hash must equal the block's; a rejected commit vector stops replay (commitExtensionsReplayError)
equalequalapp hash must equal state.LastAppHash
store = state + 1behind statereplay all but the last block as above, then apply the last one through BlockExecutor.ApplyBlock with the real application
store = state + 1equal stateapply the last block with the real application (syncStateAt)
store = state + 1equal storethe application already finalized it: apply the last block against a mock application that replays the saved ABCIResponses (newMockProxyApp), so that state and store catch up without re-executing

Start-up: WAL catch-up

State.OnStart (internal/consensus/state.go) rebuilds LastCommit from the stored seen commit (updateStateFromStore), opens the WAL, starts the ticker and, unless doWALCatchup was disabled by block sync, runs State.catchupReplay:

  • the WAL must not contain #ENDHEIGHT h for the current height;
  • it must contain #ENDHEIGHT h − 1 (0 at InitialHeight);
  • every record after it is replayed through readReplayMessage: step events are only logged (except as below), msgInfos are dispatched with fromReplay (no peer budgets, no evictions), tocks go through handleTimeout. With WalSkipRoundsToLast a replayed proposal for a later round advances the round directly; on CONSENSUS_VERSION_2 chains that skip rounds the setting is ignored, and a replayed Propose step event re-enters the recorded round and propose step (State.replaySkipsRounds, State.readReplayMessage; Recovery);
  • replay mode: the node does not propose (EnterProposeAction.Execute) and skips timeliness checks (isValidForPrevote), but it may still attempt to sign votes: replayed messages drive the normal actions, and neither prevoter.Do nor voteSigner.signVote checks replay mode, so a replayed step can also produce a vote that was never signed before the crash. Its earlier votes also come back from the WAL as internal messages. Conflicting block votes are prevented by the signer's persisted state: FilePV reuses the stored block signature for identical sign bytes at the same height/round/step (re-signing only precommit extensions) and refuses regressions and conflicting sign bytes (see WAL). DashCoreSignerClient keeps no such state in Tenderdash;
  • a corrupted WAL is backed up to <wal>.CORRUPTED, repaired by truncating at the first undecodable record (repairWalFile), and replay is retried once; other errors are logged and the node starts anyway.

Then receiveRoutine starts and ScheduleRound0 arms the first tock.

Crash windows

Crash pointRecovery
Before SaveBlockWAL replay re-delivers the height's messages and timeouts; the node re-reaches the same step
After SaveBlock, before FinalizeBlock completed (with or without EndHeightMessage{h})handshake case "store = state + 1, app = state": ApplyBlock with the real application. Without the marker, catchupReplay(h + 1) then fails with "WAL does not contain #ENDHEIGHT for h"; the error is not a corruption error, so State.OnStart logs it and starts without replay
After FinalizeBlock, before stateStore.Savehandshake case "app = store": mock replay from saved ABCIResponses (BlockExecutor.FinalizeBlock saves them before calling the application)

Block-sync handover

Reactor.SwitchToConsensus(state, skipWAL, targetHeight) (internal/consensus/reactor.go):

  1. If targetHeight > state.LastBlockHeight (peers reported more blocks than were applied), arm catchupTracker so that EnterPropose refuses to propose until the node passes targetHeight or maxCatchupSuppression (10 min) elapses; a block built from present-day application state must not be proposed into a height the network committed long ago.
  2. Rebuild LastCommit from the stored seen commit; updateToState; start State; publish the step; mark syncing complete; with skipWAL, disable WAL catch-up.

While the reactor is still in sync mode (WaitSync) it ignores data, vote and vote-set-bits messages.

A node that restarts mid-height and re-joins through consensus (no block sync) relies on peers' commit gossip for the heights it missed (Commit gossip): the proposer-selector reconstruction reads proposers from the block store to resynchronise the round-robin (proposerFromStore).