There are three main components running on the validator node: the Chain, the Indexer and the Admin-backend. Of these, the Chain is responsible for running the blockchain. If it is working properly, the blockchain is in sync, block production and reward production are continuous.
The Indexer and Admin components are user experience features, they are responsible for easier readability and management of the blockchain. If a crash occurs in these components, it does not affect the operation of the blockchain, only the display of data and ease of management.
Where can I check if block production is working properly if there is an Indexer crash on my node?
– When the “Last finalized block” on the “Blockchain status” card at the bottom right of the validator dashboard in Mosaic Explorer is constantly changing (every 6-30 seconds), and the elapsed time is no more than 1 minute, the chain running on the node is in sync with the entire blockchain. If your validator is in the “active” state, block production is also uninterrupted.
– In the node selector in the lower left corner of Mosaic Explorer, select a node that does not have a red indexer or other red text next to its name, and switch to this node to check your validator wallet address! Connect your validator wallet, and click the “View account” button in the menu that opens by clicking the wallet selector button! On the Account Details page that opens, under the “Rewards” tab, you can see whether the rewards for block production are arriving in your wallet. (Due to the random validator selection, the rewards are not evenly distributed over time, so there may be a one-day break between two rewards.)
– Subscribe to Sentinel notifications, and until you receive a notification that your validator status has changed from “normal” to “faulted” or “chilled”, your validator is working properly.
Mosaic Sentinel (https://mainnet-sentinel.mosaicchain.io/) is an application that monitors the status changes of the validator associated with the specified wallet address on the blockchain and sends a notification of these changes to the specified email address. Sentinel does not monitor a validator node, but the data on the public blockchain. A validator node in an active (normal) state must send a signal (“heartbeat”) once in each session that it is available. If the validator node has been selected for block production in a session, it must be available throughout that session and must also produce blocks according to the chain conditions. If any of the above is not met by the validator node, the slashing and status change will occur at the end of the given session. You can receive a notification about this status change via Sentinel.
What can I do about the Indexer crash?
There are two known cases of Indexer crashes on the mainnet:
1. There are nodes that have not run the latest automatic update. This may be indicated by the fact that fresh data may still appear on the page when loading, but shortly after the circle indicating the lack of indexer connection appears in the lower right corner and the data provided by the indexer is no longer updated. If you do not see “Up to date” in the Installed packages card of the Validator Dashboard in the mosaic-pro-indexer row (or any other), press the “Update” button!
2. Some users may experience a problem that results from the node not properly synchronizing with the Polkadot relay chain and the last finalized block is too old. In such a case, the indexer also lags behind in synchronization. This is manifested by the fact that the latest block number on the block page does not change and the time displayed next to it may not be a few seconds, but even several days. This phenomenon sometimes resolves itself, but restarting the validator machine can also help find the right relay chain peers and continue synchronization.
Our developers are currently working on investigating this and finding a long-term solution.
If the latest version of Indexer is running on the validator node, the machine has also been restarted if necessary, and the indexer synchronization still does not start, send an email to hello@mosaicchain.io with the name of the node, a precise description of the error, and screenshots!