Remote Signing (KMS)
Remote signing moves your validator's consensus key off the validator server and onto a dedicated signing machine. This prevents key exposure if the validator is compromised and enforces single-signer semantics to protect against accidental double-signing.
Why Remote Signing
Your validator key (priv_validator_key.json) signs every block your validator proposes or votes on. By default, this key lives on the validator server itself. If that server is compromised, an attacker gains the ability to double-sign, which results in permanent slashing (tombstoning) with no recovery path.
Remote signing solves this by placing the key on a separate, hardened machine that:
- Enforces that only one signature is produced per height/round/step
- Maintains a persistent state file to prevent signing regressions
- Can use hardware security modules (HSMs) for tamper-resistant key storage
- Limits the attack surface to a minimal, single-purpose server
Options Overview
| Solution | Signing Model | Key Storage | Complexity | Best For |
|---|---|---|---|---|
| TMKMS | Single signer | Softsign (file) or YubiHSM 2 | Moderate | Most validators |
| Horcrux | Threshold (N-of-M) | Split key shares across servers | High | Large operations, no single point of failure |
| Raw HSM | Single signer | Hardware device (PKCS#11) | Very high | Maximum security, custom integrations |
TMKMS Setup
TMKMS (Tendermint Key Management System) is the most widely used remote signer in the Cosmos ecosystem. It supports both software-based signing (softsign) and hardware-backed signing via YubiHSM 2.
Requirements
| Component | Specification |
|---|---|
| Dedicated server | 1 CPU, 1 GB RAM minimum (signing is lightweight) |
| OS | Linux (Ubuntu 22.04+ recommended) |
| Rust toolchain | For building from source |
| Network | Private connection between KMS and validator (VLAN, VPN, or firewalled) |
The connection between TMKMS and your validator carries signing requests in cleartext over a secret-connection-encrypted channel. Always use a private network or VPN between the two machines. Never expose port 26658 to the public internet.
Installation
Install TMKMS with software signing support:
cargo install tmkms --features=softsign
For YubiHSM 2 hardware support:
cargo install tmkms --features=yubihsm
Verify the installation:
tmkms version
Initialize Configuration
tmkms init /opt/tmkms
This creates the directory structure with a default tmkms.toml and generates a secret connection key.
Import Your Validator Key
Copy priv_validator_key.json from your validator server to the KMS machine, then import it:
tmkms softsign import /tmp/priv_validator_key.json \
--output /opt/tmkms/secrets/signing.key
After importing and verifying that remote signing works, remove priv_validator_key.json from the validator server. Having the key in two places defeats the purpose of remote signing and creates double-sign risk.
Configuration
Edit /opt/tmkms/tmkms.toml:
[[chain]]
id = "mono_6940-1"
key_format = { type = "bech32", account_key_prefix = "monopub", consensus_key_prefix = "monovalconspub" }
state_file = "/opt/tmkms/state/priv_validator_state.json"
[[validator]]
chain_id = "mono_6940-1"
addr = "tcp://VALIDATOR_IP:26658"
secret_key = "/opt/tmkms/secrets/secret_connection.key"
protocol_version = "v0.34"
reconnect = true
Replace VALIDATOR_IP with the private IP address of your validator server.
| Field | Description |
|---|---|
id | Cosmos chain ID of the network |
key_format | Bech32 prefixes for Monolythium addresses |
state_file | Tracks last signed height/round/step to prevent double-signing |
addr | Validator's private signing listener address |
secret_key | Key used to establish an encrypted connection to the validator |
protocol_version | Must match your CometBFT/LythiumBFT version |
reconnect | Automatically reconnect if the validator restarts |
Start TMKMS
tmkms start -c /opt/tmkms/tmkms.toml
For production, create a systemd service:
[Unit]
Description=TMKMS Signing Service
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=tmkms
ExecStart=/usr/local/bin/tmkms start -c /opt/tmkms/tmkms.toml
Restart=on-failure
RestartSec=3
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
sudo systemctl enable tmkms
sudo systemctl start tmkms
Validator config.toml Changes
On the validator server, edit ~/.mono/config/config.toml to accept remote signing connections:
# Listen for remote signer connections
priv_validator_laddr = "tcp://0.0.0.0:26658"
Using monod directly:
# Edit config.toml
sed -i 's|priv_validator_laddr = ""|priv_validator_laddr = "tcp://0.0.0.0:26658"|' ~/.mono/config/config.toml
Using Monarch CLI:
monarch config set priv_validator_laddr "tcp://0.0.0.0:26658"
Firewall Rules
Only allow the KMS server to connect on port 26658:
sudo ufw allow from KMS_SERVER_IP to any port 26658 proto tcp
sudo ufw deny 26658
Replace KMS_SERVER_IP with the private IP of your TMKMS machine.
Restart the Validator
# Using Monarch CLI
monarch restart
# Using monod directly
sudo systemctl restart monod
Horcrux Overview
Horcrux implements threshold signing using Shamir's Secret Sharing. Instead of storing the full validator key on a single machine, the key is split into shares distributed across multiple cosigner nodes. A configurable threshold (e.g., 2-of-3 or 3-of-5) must agree before any block is signed.
How It Works
- The validator key is split into N shares during setup
- Each cosigner holds one share and runs a Horcrux process
- When the validator needs to sign, it requests signatures from all cosigners
- At least M cosigners must respond with valid partial signatures
- The partial signatures are combined into a full signature
Advantages
- No single machine holds the complete key
- Tolerates up to N-M cosigner failures
- Eliminates single point of failure for signing
Trade-offs
- Significantly more complex to set up and maintain
- Requires 3-5 dedicated servers for cosigner nodes
- Higher latency for signing (network round trips between cosigners)
- Debugging signing failures is more involved
For Horcrux setup instructions, refer to the Horcrux documentation.
Monitoring
Verify Remote Signing
After starting TMKMS and restarting the validator, confirm signing is working:
# Check TMKMS logs for successful connections
journalctl -u tmkms -f
# Look for: "connected to tcp://VALIDATOR_IP:26658"
# Look for: "signed PreVote" or "signed PreCommit"
# On the validator, verify blocks are being signed
curl -s localhost:26657/status | jq '.result.validator_info'
Using Monarch CLI:
monarch status
# Verify "signing" field shows active
What to Monitor
| Metric | What It Means | Alert On |
|---|---|---|
| TMKMS connection status | Whether KMS is connected to the validator | Disconnected for > 30 seconds |
| Signed blocks | Blocks your validator has signed | Missed blocks increasing |
| State file updates | Last signed height advancing | Height not advancing |
| TMKMS process status | Service running | Process crash or restart |
Log Messages to Watch
| Log Pattern | Meaning |
|---|---|
connected to tcp://... | Successful connection to validator |
signed PreVote / signed PreCommit | Normal signing operations |
attempted double sign | TMKMS blocked a double-sign attempt (this is protection working correctly) |
connection refused | Validator is not listening on 26658 |
state file height regression | State file may be corrupted or stale |
FAQ
Does TMKMS prevent all double-signing?
TMKMS prevents accidental double-signing caused by common operational errors like running two validators with the same key during migration. It maintains a state file that tracks the last signed height, round, and step. However, if an attacker duplicates both the signing key and the state file to a separate machine, TMKMS on the attacker's machine would not know about signatures produced by the legitimate instance. Remote signing is defense in depth, not an absolute guarantee.
Can I use TMKMS on testnet?
Yes, and it is strongly recommended. Testnet is the ideal environment to practice TMKMS setup, test failover procedures, and build operational confidence before deploying to mainnet.
What happens if the TMKMS server goes down?
Your validator stops signing blocks because it has no local key to fall back on. The validator will begin missing blocks and eventually be jailed for downtime. This results in a 0.01% slash and a minimum 10-minute jail period. Critically, downtime does not result in tombstoning -- you can unjail once the signing service is restored.
Can I run TMKMS on the same server as the validator?
You can, but it defeats most of the security benefits. The primary value of remote signing is key isolation on a separate machine. Running both on the same server still provides the state-file double-sign protection but not the key isolation.
What about YubiHSM 2 vs softsign?
Softsign stores the key in an encrypted file on disk. YubiHSM 2 stores the key in tamper-resistant hardware where it cannot be extracted. For most validators, softsign on a dedicated server provides a strong security improvement over local key storage. YubiHSM 2 is recommended for validators with significant stake or high-security requirements.
Related
- Slashing - Slashing penalties and recovery
- Best Practices - Validator operational best practices
- Security Guide - Key management and server hardening
- Peer Roles - Sentry architecture for validator protection