Respond to a sequencer outage

Use this page when your Arbitrum chain stops producing blocks or stops posting batches to the parent chain. Work through the steps in order. Each step links to the page that holds the fix. For background on what happens during an outage, see Sequencer outages.

1. Find out which outage you haveCopy Anchor Link

Check two things: whether blocks are still being produced, and whether batches are still landing on the parent chain.

For the metrics and alert thresholds, see the Sequencer and Batch poster sections of the monitoring guide.

Write down the time the outage started. You need it in step 4.

2. Restore block productionCopy Anchor Link

1. Work through the diagnostic order. Sequencer troubleshooting lists the checks and the fix for each error message.
2. If you run the sequencer coordinator, confirm that a sequencer holds the lockout. Read coordinator.chosen in Redis. If you set chosen-healthcheck-addr, you can query that endpoint instead. It returns 200 on the chosen sequencer and 503 on the others. If no sequencer is chosen, see No sequencer is chosen.
3. If the sequencer logs cannot sequence, check the parent-chain connection first, then the host clock. See The sequencer refuses to build a block.

3. Restore batch postingCopy Anchor Link

1. Check the batch poster balance. If arb/batchposter/wallet/eth is below your funding threshold, fund the account.
2. Check for a reverted batch. If the batch poster logs batch was reverted, not posting any more batches, find the cause of the revert, then restart the batch poster. See Recovery mechanism 2.
3. If the batch poster key is lost or compromised, rotate it. See Batch poster key rotation.
4. If you run a standby batch poster, confirm that it took the lock. The lock works only when you set node.batch-poster.redis-url. It expires 1 minute after the active batch poster stops. See Redis failover.
5. If someone force-included a message during the outage, the first batch can fail. A force inclusion does not update the message count the batch poster expects. If the first batch fails, set dangerous.allow-posting-first-batch-when-sequencer-message-count-mismatch for the first batch. See Recovery mechanism 2.

For other causes, see Batch poster troubleshooting.

4. Tell your usersCopy Anchor Link

Tell your users what is down and how they can still reach the chain:

• Which job stopped. Say whether the chain has stopped producing blocks or has stopped posting batches.
• How to send a transaction anyway. Point users to Forced transactions. They can now send a transaction through the Delayed Inbox.
• When force inclusion opens. A message sent to the Delayed Inbox can be force-included one force-inclusion window after the parent-chain block it landed in. To get the exact block, call forceInclusionDeadline(blockNumber) on the Sequencer Inbox with that block number. The result accounts for the Censorship Timeout. Give your users the window in hours, and the deadline block if you have it. See Settings you control.
• How to withdraw ETH. Point users to Withdraw ETH while the sequencer is down.
• During a batch-posting outage, recent transactions can change; a force inclusion can drop soft-confirmed transactions. Tell users which transactions might need to be resent.

5. After recoveryCopy Anchor Link

1. Confirm that blocks and batches are flowing. Watch arb/batchposter/estimated_batch_backlog and arb/feed/backlog/messages drain back to normal.
2. Check for a reorg. If someone force-included a message while blocks were unposted, nodes that follow the feed dropped those blocks. The sequencer re-sequences up to 1,024 dropped messages, and only while it is active. See When a force inclusion reorders the chain.
3. Check the Censorship Timeout buffer if you enabled it. A long outage drains the buffer, so the force-inclusion window stays shorter until it refills. The buffer refills about 1 day for every 20 days without long delays. See How the Censorship Timeout shortens the window.
4. Review your setup. Decide whether you need a standby sequencer, a standby batch poster, new alerts, or a different force-inclusion window. See Sequencer outages.