Bootstrapping crypto infrastructure with shell and config management

Running infrastructure for a cryptocurrency project is ordinary systems administration with the stakes turned up. The daemons are just long-running processes; the servers are just Linux boxes. But every one of those processes can move money, every server holds or touches keys, and a mistake is not a rolled-back transaction — it is funds gone for good. That difference changes how you build everything. Here is the war story of bootstrapping that infrastructure reproducibly, and the lessons that only a money-handling system teaches you.

The problem: many daemons, zero room for snowflakes

The shape of the system was several coin daemons (Bitcoin and a handful of altcoins), each a long-running process exposing a JSON-RPC interface, plus the wallet, monitoring, and backup machinery around them. Every daemon needed the same disciplined treatment: a hardened host, a synced chain, a locked-down RPC, monitored liveness, and backed-up wallet data.

The temptation, under deadline, is to SSH in and set each one up by hand. We had already learned where that road ends — a fleet of snowflake servers nobody can reproduce, each subtly different, each a mystery six months later. With money on the line, “I think this one is configured correctly” is not good enough. Everything had to be code: reproducible, reviewable, and rebuildable from scratch.

Shell for bootstrap, Ansible for convergence

We split the work along a line that has served us well since. Shell scripts handle bare bootstrap — the handful of imperative steps to take a fresh box from nothing to “Ansible can manage it”: set the hostname, create the deploy user, install Python and an SSH key. Short, dumb, run once.

#!/usr/bin/env bash
set -euo pipefail            # fail loudly — never half-configure a money box
apt-get update
apt-get install -y python sudo
useradd -m -s /bin/bash deploy
install -d -m 700 /home/deploy/.ssh
echo "$DEPLOY_KEY" > /home/deploy/.ssh/authorized_keys
chmod 600 /home/deploy/.ssh/authorized_keys

The set -euo pipefail at the top is not boilerplate — it is a safety rule. On a machine that will hold keys, a script that plows on after an error is how you end up with a half-configured box you think is hardened. Fail loud, fail early.

Everything after bootstrap is Ansible — idempotent roles that converge each host to a declared state: the hardened sshd, the firewall, the coin daemon, its config, its monitoring. The division of labour: shell gets you to a managed state once; Ansible keeps you there forever, and proves it on every run.

Daemons as managed services, not hand-started processes

A coin daemon that someone started by hand in a screen session is a liability — it will not survive a reboot, nobody knows the exact flags it was launched with, and its restart behaviour is whatever the shell happened to do. Every daemon went under a proper init/supervisor with a declared config, so the service definition is the documentation:

# rendered from an Ansible template, one per coin
[program:dogecoind]
command=/usr/local/bin/dogecoind -conf=/etc/coins/dogecoin.conf -printtoconsole
user=coin
autostart=true
autorestart=true
stdout_logfile=/var/log/coins/dogecoin.log

Now a reboot brings every daemon back exactly as configured, a crash restarts it, and the launch flags live in version control rather than someone’s memory. The chain data, of course, lives on a dedicated, backed-up volume — never on the ephemeral root disk — for exactly the reasons running these nodes in containers had already taught us: the process is disposable, the data is not.

Key handling: the part that keeps you up at night

This is where crypto infrastructure stops being normal sysadmin. The keys that control funds cannot be treated like ordinary config. The hard-won rules:

• Hot and cold are different worlds. Only the minimum needed for day-to-day operation (a hot wallet with limited funds) lives on an internet-connected machine. The bulk sits in cold storage — keys generated and kept on a machine that has never touched the network. No amount of server hardening substitutes for simply not having the keys reachable.
• Secrets never live in the repo. The infrastructure is code and public to the team; the secrets are not. RPC passwords, wallet passphrases, and API keys are injected at deploy time from an encrypted store (Ansible Vault), never committed in plain text, never echoed into logs or the process list.
• RPC is reachable from nothing it does not need to be. Same lesson as running a single node, now non-negotiable across the fleet: the P2P port is public, the RPC port is bound to localhost or a private interface, the firewall denies by default and allows by exception. A money-moving interface exposed to the internet is not a risk, it is an eventual certainty of loss.
• Backups are encrypted, automated, and tested. A wallet backup is a bearer instrument — anyone who has it has the funds — so backups are encrypted at rest, and the restore is tested, because an untested backup is a guess.

Monitoring liveness and divergence

For ordinary services you monitor uptime and latency. For coin daemons you also monitor things specific to the domain, because “the process is running” is not the same as “the node is healthy”. The checks that earned their place:

• Is it running and responsive to RPC? A daemon can be up but wedged.
• Is the block height keeping pace with the network? A node that has silently stalled or forked is worse than a dead one — it gives confidently wrong answers.
• Is disk filling? Chains only grow; a full disk takes a daemon down hard.
• Did the RPC port ever become externally reachable? Alert on it as a security regression, not just an outage.

These ran on a schedule and paged on failure, because the cost of finding out late that a wallet host had drifted was not a slow page — it was potential loss.

What the project taught us

Stripped of the cryptocurrency specifics, this was an intense course in infrastructure discipline, and the lessons generalise to anything where mistakes are expensive:

• Reproducibility is a security property, not just a convenience. A box you can rebuild from code is a box you can reason about; a snowflake is an unknown, and unknowns near money are unacceptable.
• Separate what is disposable from what is precious — process from data, hot from cold, code from secrets — and design the lifecycle of each deliberately.
• Fail loud. set -euo pipefail, deny-by-default firewalls, alerts on security regressions: when the downside is permanent, a system that stops at the first sign of trouble beats one that helpfully carries on.

We have carried this posture into every project since, crypto or not. The cryptocurrency context simply removed the safety net that makes sloppiness survivable elsewhere — and working without that net, for a while, made us considerably better at building infrastructure that deserves to be trusted with something that matters.