Troubleshooting

This page covers common issues encountered when operating a Monolythium node and how to resolve them.

Start with Doctor

Before diving into specific issues, run monarch doctor for a comprehensive health audit. It checks 13 categories and provides actionable recommendations for most problems.

---

Node Not Syncing

Symptoms: Block height is not increasing. catching_up remains true indefinitely. Node appears stuck.

Check Peer Connectivity

# Using Monarch
monarch status

# Using monod
curl -s localhost:26657/net_info | jq '.result.n_peers'

If the peer count is 0 or very low, update your peers:

monarch peers update
monarch restart

Try Snapshot Sync

If syncing from genesis is stalled (common around height 140K due to pruned peers), use snapshot sync:

monarch stop
monarch wipe --preserve-keys
monarch join --network Testnet --sync-mode snapshot --moniker my-node

Check Firewall

Ensure port 26656 (P2P) is open:

sudo ufw status
# Port 26656/tcp should be ALLOW

If not:

sudo ufw allow 26656/tcp

---

Validator Jailed

Symptoms: monarch validator status shows jailed status. The validator is not producing blocks.

1. Check Missed Blocks

monarch validator status

# Or query signing info
monod query slashing signing-info $(monod comet show-validator)

2. Identify the Cause

Common reasons:

• Node was offline -- the process crashed or the server rebooted without auto-start
• Node lost peers -- isolated from the network
• Disk full -- node could not write new blocks
• Configuration drift -- incorrect chain ID or peers after an update

3. Fix the Underlying Issue

# Run the health audit
monarch doctor

# Fix detected issues
monarch repair

# Update peers if needed
monarch peers update

# Restart the node
monarch restart

4. Wait for the Jail Period

Check the remaining jail time in the signing info output. You cannot unjail until the period expires.

5. Unjail

monarch validator unjail --from validator

---

Monarch Connect Cannot Connect

Symptoms: Monarch Connect shows "Connection Failed" or "Timeout" for a server.

Verify Server Reachability

# From your local machine
ping <server-ip>
ssh <username>@<server-ip>

Check SSH Key Path

In Monarch Connect, verify the SSH key path points to a valid private key file. The key must be readable by your user account.

Verify SSH Port

The default SSH port is 22. If your server uses a different port, update the server profile in Monarch Connect.

Try Password Authentication

If key-based auth fails, temporarily switch to password auth in the server profile to verify the connection works. Then troubleshoot the key issue.

Check Server SSH Configuration

On the server:

# Verify SSH is running
sudo systemctl status sshd

# Check for IP restrictions
grep -i "allowusers\|allowgroups\|listenaddress" /etc/ssh/sshd_config

---

Interactive Command Hangs

Symptoms: A command started through Monarch Connect stops responding, showing no output and not completing.

Use PTY Terminal Mode

Some commands require a pseudo-terminal (PTY). In Monarch Connect, use the Terminal tab for interactive commands rather than the GUI buttons.

Check Process State

# On the server via SSH
ps aux | grep monod

If the process is stuck, kill it and restart:

monarch stop
monarch start

---

Metrics Not Collecting

Symptoms: monarch metrics status shows metrics are not being collected. No data in Monarch Connect charts.

Verify Cron Installation

monarch metrics status

If disabled, enable it:

monarch metrics enable

Check User Permissions

The cron job runs as your user. Ensure the metrics directory is writable:

ls -la ~/.mono/metrics/

If the directory does not exist or has wrong permissions:

mkdir -p ~/.mono/metrics
chmod 700 ~/.mono/metrics

Check Cron Service

sudo systemctl status cron
# or
sudo systemctl status crond

Verify Node Is Running

Metrics collection queries the local RPC endpoint. If the node is stopped, metrics collection will fail silently.

monarch status

---

Config Drift

Symptoms: monarch doctor reports config drift. Node behaves unexpectedly after an update.

Identify Drift

monarch doctor

Look for warnings about mismatched chain IDs, incorrect seeds, or outdated peers in config.toml or app.toml.

Fix Drift

# Preview changes
monarch repair --dry-run

# Apply fixes
monarch repair

The repair command updates:

• chain-id in client.toml
• evm-chain-id in app.toml
• seeds in config.toml
• persistent_peers in config.toml

It preserves node_key.json, priv_validator_key.json, and all other settings.

---

Height Stall Around 140K

Symptoms: Node syncs up to around block 140,000 and then stalls or progresses very slowly.

Cause

This is typically caused by a peer with pruned state at that height range. The syncing node requests blocks that the peer has already pruned.

Fix

Use snapshot sync to bypass the problematic range:

monarch stop
monarch wipe --preserve-keys
monarch join --network Testnet --sync-mode snapshot --moniker my-node

Or use state sync:

monarch stop
monarch wipe --preserve-keys
monarch join --network Testnet --sync-mode state-sync --moniker my-node

---

Permission Denied

Symptoms: permission denied errors when running monod or accessing ~/.mono.

Fix Ownership

# Fix ownership of the data directory
sudo chown -R $(whoami):$(whoami) ~/.mono

# Fix permissions on sensitive key files
chmod 600 ~/.mono/config/priv_validator_key.json
chmod 600 ~/.mono/config/node_key.json
chmod 700 ~/.mono

Check Systemd User Directive

If running via systemd, ensure the User directive matches the owner of ~/.mono:

# Check current user
grep "^User=" /etc/systemd/system/monod.service

# Verify ownership matches
ls -la ~/.mono/

Non-Root Execution

Do not run monod as root. If you previously ran it as root, fix the ownership:

sudo chown -R monod:monod /home/monod/.mono

---

Port Already in Use

Symptoms: bind: address already in use error when starting the node.

Identify the Conflicting Process

# Check what is using port 26656
ss -tlnp | grep 26656

# Or use lsof
sudo lsof -i :26656

Resolve

If another monod process is running:

monarch stop
# Wait a moment, then start again
monarch start

If a different process is using the port:

# Kill the conflicting process (if safe)
sudo kill <PID>

# Or change the port in config.toml
# Find: laddr = "tcp://0.0.0.0:26656"
# Change to a different port

---

AppHash Mismatch

Symptoms: Node crashes with AppHash mismatch or wrong Block.Header.AppHash error.

Cause

This is usually caused by an incorrect EVM chain ID or by running a mismatched binary version against existing chain data.

Fix

# Run repair to fix configuration
monarch repair

# If that does not resolve it, wipe and re-sync
monarch stop
monarch wipe --preserve-keys
monarch join --network Testnet --sync-mode snapshot --moniker my-node

Data Loss

Wiping chain data requires re-syncing from the network. This does not affect your keys if you use --preserve-keys.

---

Out of Disk Space

Symptoms: no space left on device errors. Node crashes or refuses to start.

Check Disk Usage

df -h
du -sh ~/.mono/data/

Prune Data

If running a full node (not archive), you can prune old state:

# Check current pruning settings in app.toml
grep "pruning" ~/.mono/config/app.toml

Increase Disk

For validators, the recommended long-term solution is to increase disk capacity. NVMe storage is recommended for consistent performance.

Emergency Cleanup

# Clear old logs
monarch logs --clear

# Check for old snapshot files
ls -la ~/.mono/data/snapshots/

---

Node Crashes on Start

Symptoms: Node starts and immediately crashes, or crashes within a few seconds.

Check Logs

monarch logs --level error

# Or check the last 50 lines
journalctl -u monod -n 50 --no-pager

Common Causes

Error Message	Cause	Fix
wrong chain ID	Mismatched chain-id in config	monarch repair
genesis doc mismatch	Wrong genesis file	Re-download with monarch join
failed to read key	Corrupt or missing key file	Restore from backup
panic: runtime error	Binary version mismatch	monarch upgrade --version <correct>
too many open files	File descriptor limit too low	Increase LimitNOFILE in systemd or ulimit

---

Diagnostic Commands Reference

What to Check	Command
Overall health	monarch doctor
Node status	monarch status
Validator status	monarch validator status
Peer connectivity	monarch peers validate
Metrics collection	monarch metrics status
Recent events	monarch metrics events --since $(date -d '1 hour ago' +%s)
Error logs	monarch logs --level error
Process status	monarch monit
Resource usage	monarch top
Disk usage	df -h && du -sh ~/.mono/data/

---

Getting Help

If these troubleshooting steps do not resolve your issue:

1. Run monarch doctor --json and save the output
2. Collect recent error logs: monarch logs --level error
3. Note your monod version: monarch version
4. Visit the Monolythium community channels with this information

---

Related

• Monarch CLI -- Full CLI reference
• Security Guide -- Key management and hardening
• Validator Operations -- Validator lifecycle guide
• Node Metrics -- Metrics and event detection
• Monitoring -- Monitoring setup