Node Metrics
Monarch CLI collects time-series metrics from your node every 60 seconds when enabled. This data powers local diagnostics, event detection, and the Monarch Connect dashboard charts.
Enabling Metrics Collection
monarch metrics enable
This installs a cron job that runs every 60 seconds, collecting a snapshot of node and system state. To disable:
monarch metrics disable
Check current collection status:
monarch metrics status
Collected Fields
Each snapshot records 24 fields:
| Field | Type | Description |
|---|---|---|
timestamp | int | Unix timestamp of the snapshot |
height | int | Current block height |
sync_status | string | synced, catching_up, or stalled |
catching_up | bool | Whether the node is still syncing |
peers | int | Number of connected peers |
block_time | float | Time since last block (seconds) |
latest_block_time | string | ISO 8601 timestamp of the latest block |
validator_rank | int | Validator position in the active set (0 if not a validator) |
tokens | string | Total bonded tokens (in alyth) |
delegator_count | int | Number of unique delegators |
commission_rate | float | Current commission rate |
missed_blocks | int | Missed blocks in the current signing window |
jailed | bool | Whether the validator is jailed |
voting_power | int | Validator voting power |
cpu_percent | float | System CPU usage percentage |
memory_bytes | int | Memory used by the node process |
memory_total | int | Total system memory |
disk_used | int | Disk space used (bytes) |
disk_total | int | Total disk capacity (bytes) |
uptime | int | Node process uptime (seconds) |
process_status | string | running, stopped, or crashed |
monod_version | string | Installed monod version |
monarch_version | string | Installed monarch version |
chain_id | string | Current chain ID |
Event Detection
Monarch analyzes collected metrics and detects 14 event types:
| Event | Trigger | Severity |
|---|---|---|
jailing | Validator becomes jailed | Critical |
unjailing | Validator is unjailed | Info |
rank_change | Validator rank shifts by 3+ positions | Warning |
delegator_change | Delegator count changes significantly | Info |
missed_block_spike | Missed blocks increase sharply within a window | Warning |
disk_warning | Disk usage exceeds 80% | Warning |
stale_block | No new block for 30+ seconds | Warning |
sync_started | Node begins catching up | Info |
sync_completed | Node finishes syncing | Info |
height_stall | Block height unchanged for 5+ minutes | Critical |
peer_drop | Peer count drops below 3 | Warning |
memory_warning | Memory usage exceeds 90% of system total | Warning |
process_restart | Node process restarted unexpectedly | Warning |
upgrade_available | New monod version detected | Info |
Events are stored alongside metrics and can be queried independently.
Storage
Metrics are stored as JSONL (JSON Lines) files in the node home directory:
~/.mono/metrics/
├── metrics-2026-03-30.jsonl
├── metrics-2026-03-29.jsonl
├── events-2026-03-30.jsonl
└── events-2026-03-29.jsonl
Storage Characteristics
| Property | Value |
|---|---|
| Format | JSONL (one JSON object per line) |
| Location | ~/.mono/metrics/ |
| Daily size | ~42 KB (metrics) |
| Auto-rotation | Files rotated at 5 MB |
| Retention | 7 days (older files auto-deleted) |
At ~42 KB per day, a full week of metrics uses under 300 KB of disk space.
Querying Metrics
CLI Queries
# Query metrics from the last 24 hours
monarch metrics query --since $(date -d '24 hours ago' +%s) --json
# Query with a row limit
monarch metrics query --since 1711756800 --limit 500 --json
# Query detected events
monarch metrics events --since 1711756800 --limit 100 --json
# Query events from the last hour
monarch metrics events --since $(date -d '1 hour ago' +%s) --json
Output is JSON, suitable for piping to jq or other tools:
# Get the latest block height
monarch metrics query --limit 1 --json | jq '.[0].height'
# Find all jailing events
monarch metrics events --json | jq '.[] | select(.type == "jailing")'
Monarch Connect Integration
When Monarch Connect is connected to a server with metrics enabled, the Metrics tab displays interactive charts:
- Block height -- trend line showing chain progress
- Peer count -- connection health over time
- CPU and memory -- resource consumption patterns
- Disk usage -- capacity planning with threshold markers
- Validator rank -- rank movement with event markers
- Token balance -- bonded tokens over time
- Missed blocks -- signing performance
Time Range Filters
Select from preset ranges: 1 hour, 6 hours, 24 hours, 7 days, or 30 days.
Event Timeline
Events appear as annotated markers on charts and in a chronological timeline view below the charts. Critical events (jailing, height stall) are highlighted in red.
Export Options
Static File Export
Export metrics as a JSON file for use with external analysis tools:
monarch metrics query --since 0 --json > metrics-export.json
If you plan to share exported metrics publicly, be aware that fields like cpu_percent, memory_bytes, and disk_used reveal server resource information. Consider filtering these fields before sharing.
Prometheus Endpoint
Monarch can expose a Prometheus-compatible metrics endpoint for scraping:
# Exposed on localhost only, no authentication
# Configure in ~/.mono/config/monarch.toml
[metrics]
prometheus = true
prometheus_addr = "127.0.0.1:9100"
The Prometheus endpoint should only be bound to 127.0.0.1. Do not expose it to the public internet. Use an SSH tunnel or reverse proxy with authentication if you need remote access.
Security Considerations
- Metrics contain no secrets -- no private keys, passwords, or mnemonics
- Metrics do contain server resource info (CPU, memory, disk) which could reveal infrastructure details
- The Prometheus endpoint should remain localhost-only
- Static exports intended for public sharing should have resource fields stripped
- JSONL files in
~/.mono/metrics/have standard file permissions (readable only by the node user)
Related
- Monarch CLI -- CLI reference for metrics commands
- Monarch Connect -- Desktop GUI with metrics charts
- Monitoring -- General monitoring setup
- Validator Operations -- Validator monitoring best practices