Validator Migration

This guide covers how to safely move a running validator to a new server without risking double-signing or extended downtime.

Overview

Validator migration is necessary when:

• Upgrading to better hardware
• Switching hosting providers
• Responding to a compromised or failing server
• Relocating for geographic diversity or latency improvements

This is the most dangerous routine operation a validator can perform. If both the old and new servers sign blocks simultaneously, even for a single block, your validator is permanently tombstoned. There is no appeal process and no recovery.

Risks

Double-signing is permanent and irreversible

If both servers produce a signature at the same block height, the network treats this as a double-sign. There is no grace period, no warning, and no undo. Plan carefully and follow every step.

Scenario	Result
Both servers sign at the same height	Tombstoned — permanent ban
Slash penalty	5% of all staked tokens (yours and delegators')
Recovery	None — must create a new validator with new keys and new stake
Delegator impact	All delegators lose 5% of their delegation

Pre-Migration Checklist

Before starting the migration, verify every item:

• New server is provisioned and synced — monod status | jq '.sync_info.catching_up' returns false
• New server does NOT have the validator key — confirm ~/.mono/config/priv_validator_key.json does not exist on the new server
• Key backup verified — you have a secure, encrypted backup of your validator key accessible from a location other than the old server
• Old server is currently signing blocks — confirm via monod query slashing signing-info or Monoscan
• Migration window planned — choose a low-traffic period to minimize missed blocks
• Communication sent — notify delegators of planned brief downtime if applicable
• Both servers accessible — confirm SSH access to old and new servers before starting

Step-by-Step Migration

Step 1: Prepare the New Server

Install monod (or use Monarch CLI) and sync the new node fully. The new server must be a working full node before migration.

# On the new server — install and initialize
monarch install
monod init <moniker> --chain-id <chain-id>

# Copy the canonical config
curl -sL https://raw.githubusercontent.com/monolythium/networks/prod/testnet/app.toml \
  -o ~/.mono/config/app.toml

# Start syncing (as a full node, NOT a validator)
monarch start

Wait until sync is complete:

# Must return false
monod status | jq '.sync_info.catching_up'

info

The new server runs as a regular full node during this phase. It does not have a validator key and cannot sign blocks. This is safe and expected.

Step 2: Stop the Old Validator

Stop the validator process on the old server completely.

# Using Monarch CLI
monarch stop

# Or using systemd directly
sudo systemctl stop monod

Verify the process is fully stopped:

# Must return nothing
pgrep monod

# Double-check
systemctl is-active monod
# Should return "inactive"

Step 3: Confirm the Old Node Is Stopped

Wait for at least one block before proceeding

After stopping the old node, wait for at least one new block to be produced on the network. This ensures any in-flight signatures from the old node have been committed or expired. Rushing this step is how double-signs happen during migrations.

# On any synced node, check the latest block height
monod status | jq '.sync_info.latest_block_height'

# Wait 10-15 seconds, check again — height should have advanced
monod status | jq '.sync_info.latest_block_height'

Only proceed when you are certain the old node is no longer producing signatures.

Step 4: Export the Validator Key

Export the validator key from the old server using Monarch CLI, which creates an encrypted file:

# On the old server
monarch keys validator export
# Creates an encrypted .enc file at the path shown in output

If not using Monarch, manually copy the key file:

# Location of the raw key
cat ~/.mono/config/priv_validator_key.json

warning

The priv_validator_key.json file is the most sensitive file in your validator setup. Anyone with this file can sign blocks as your validator.

Step 5: Transfer the Key to the New Server

Transfer the encrypted key file to the new server. Never transfer an unencrypted key over the network.

# Transfer the encrypted export (preferred)
scp ~/.mono/validator-key-export.enc user@new-server:~/

# If transferring the raw key, encrypt it first
gpg -c ~/.mono/config/priv_validator_key.json
scp ~/.mono/config/priv_validator_key.json.gpg user@new-server:~/

Use an encrypted channel

Always use scp or rsync over SSH. Never use unencrypted transfer methods (FTP, HTTP, email, paste in chat). If the key is intercepted, your validator can be permanently compromised.

Step 6: Import the Key on the New Server

Import the validator key on the new server:

# Using Monarch CLI (from encrypted export)
monarch keys validator import ~/validator-key-export.enc

# Or manually place the raw key
cp ~/priv_validator_key.json ~/.mono/config/priv_validator_key.json
chmod 600 ~/.mono/config/priv_validator_key.json

Verify the key was imported correctly:

# Should show your validator's consensus public key
monod comet show-validator

Compare the output with what the old server reported. The consensus public key must match exactly.

Step 7: Start the New Validator

Start the validator on the new server:

# Using Monarch CLI
monarch start

# Or using systemd
sudo systemctl start monod

Step 8: Verify the Migration

Confirm the new server is signing blocks:

# Check signing info — missed_blocks_counter should stop increasing
monod query slashing signing-info $(monod comet show-validator)

# Check validator status
monod query staking validator <operator-address>

# Using Monarch
monarch status

Also verify on Monoscan that your validator is active and signing.

Check	Expected
catching_up	false
missed_blocks_counter	Stable or decreasing
jailed_until	In the past or 0001-01-01
tombstoned	false
Monoscan status	Active, signing

Clean up the old server

After confirming the migration is successful, securely delete the validator key from the old server:

# On the old server
shred -u ~/.mono/config/priv_validator_key.json

Do not leave validator keys on decommissioned servers.

Emergency Rollback

If the new server fails to start signing or encounters errors after importing the key:

1. Stop the new server immediately:

   # On the new server
   monarch stop
   # or
   sudo systemctl stop monod

2. Verify the new server is fully stopped:

   pgrep monod
   # Must return nothing

3. Start the old server again:

   # On the old server
   monarch start
   # or
   sudo systemctl start monod

warning

Rollback is only safe if the new server never successfully signed a block. If both servers have signed blocks at any point, rollback will not prevent tombstoning — the evidence is already on-chain.

Using KMS Instead

The entire migration risk exists because the validator key lives on the same machine as the node. Remote signing solutions eliminate this problem entirely.

With TMKMS or Horcrux, the signing key lives on a separate, dedicated machine (or a threshold-signing cluster). When you migrate the validator node:

1. Spin up the new node
2. Point it at the same KMS
3. Stop the old node
4. The KMS enforces single-signature per height regardless of which node is connected

This means no key transfer, no risk of duplicate signing, and near-zero downtime.

See KMS for detailed setup instructions.

FAQ

What if both servers signed a block?

You are tombstoned. This is permanent and cannot be reversed by any mechanism. You will need to create an entirely new validator with a new key, stake 200,000 LYTH again (100,000 burn + 100,000 self-delegation), and rebuild your delegation from scratch.

Can I test the migration first?

Yes, and you should. Sync the new server fully and verify everything works as a full node before performing the actual key migration. Confirm:

• The node syncs and stays at chain tip
• Peers connect normally
• RPC and API endpoints respond
• Disk, CPU, and memory are within expected ranges

The only step you cannot safely test in advance is actual block signing with the validator key.

How long will my validator be offline?

Typically 2 to 5 minutes if both servers are prepared and you have practiced the steps. The downtime window is:

1. Stop old server (~10 seconds)
2. Wait for block confirmation (~10 seconds)
3. Transfer and import key (~30 seconds to 2 minutes depending on method)
4. Start new server and first signed block (~30 seconds)

This is well within the 100-block signing window, so you will not be jailed for a smooth migration.

Should I unjail before migrating?

If your validator is currently jailed, fix the issue first, unjail, and confirm you are signing blocks before attempting a migration. Migrating a jailed validator adds unnecessary complexity.

Can I migrate to a different hosting provider?

Yes. The process is identical regardless of whether the new server is on the same provider, a different provider, or in a different geographic region. The only requirement is that the new server can sync and connect to the Monolythium network.

Related

• Security Guide — Server hardening and access control
• Slashing — Slashing parameters and penalties
• KMS — Remote signing setup with TMKMS/Horcrux
• Best Practices — Operational guidelines for validators