Skip to content
← All posts
5 min read Dawid Skłodowski

Commanding an EC2 fleet from your console

Clicking through the AWS console does not scale past a handful of servers. How we drive an EC2 fleet from Ruby — declaring instances as data, reconciling desired against actual, and tagging your way to sanity.

The AWS web console is a fine place to launch your first server and a terrible place to manage your tenth. Every instance you create by clicking is a snowflake: you cannot diff it, you cannot review it, and six months later nobody remembers which security group it is in or why. The moment you have more than a couple of machines, you want your fleet described as code — something you can read, version, and re-run.

This is the era just before Terraform, and CloudFormation’s JSON is its own kind of pain, so we have been building this in Ruby with the aws-sdk gem. It turns out a surprisingly small amount of code gives you a console tool that launches, inspects, and tears down an entire fleet on command. Here is the shape of it.

Declare instances as data

The key idea — the one that survives whatever tool you eventually use — is to stop performing infrastructure changes and start declaring the desired state. Write down what the fleet should look like, in plain data, and let the tool make reality match:

FLEET = {
  "web-1" => { type: "m3.medium", role: "web",    az: "eu-west-1a" },
  "web-2" => { type: "m3.medium", role: "web",    az: "eu-west-1b" },
  "worker-1" => { type: "c3.large", role: "worker", az: "eu-west-1a" },
  "db-1"  => { type: "m3.large",  role: "db",     az: "eu-west-1a" },
}

This is config, not commands. It lives in your repo, it diffs cleanly in a pull request, and it is the single source of truth for what should exist. Everything the tool does is in service of making AWS agree with this hash.

Talk to EC2 from Ruby

The aws-sdk gem wraps the EC2 API. Launching an instance is a method call:

require "aws-sdk"

ec2 = AWS::EC2.new(region: "eu-west-1")

def launch(name, spec, ec2)
  instance = ec2.instances.create(
    image_id:          ami_for(spec[:role]),
    instance_type:     spec[:type],
    availability_zone: spec[:az],
    key_name:          "deploy",
    security_groups:   [spec[:role]],
  )
  instance.tags["Name"] = name
  instance.tags["Role"] = spec[:role]
  instance.tags["ManagedBy"] = "fleet-tool"
  instance
end

Two things in there matter more than the launch itself. The AMI per role (ami_for) means each kind of machine boots from a known, pre-baked image rather than being configured by hand after launch — a web box and a worker box start life already different and already correct. And the tags are not decoration; they are how you will find this instance again.

Tags are your primary key

This is the lesson that took us longest to internalise: in a cloud where instances come and go and their IDs are opaque (i-0a1b2c3d), tags are how you address your fleet. Tag every instance with at least a Name, a Role, and a marker that your tool created it (ManagedBy). Now you can ask AWS questions in your own vocabulary:

# everything the tool manages
managed = ec2.instances.filter("tag:ManagedBy", "fleet-tool")

# all the web servers, running
web = ec2.instances
         .filter("tag:Role", "web")
         .filter("instance-state-name", "running")

Without disciplined tagging you are reduced to staring at a list of cryptic IDs and IP addresses. With it, “restart all the workers” or “show me what is running in eu-west-1b” becomes a one-liner. The ManagedBy tag in particular is what keeps your tool from ever touching a hand-launched instance it should leave alone.

Reconcile desired against actual

With desired state as data and actual state queryable by tag, the core of the tool writes itself: compare the two and act on the difference.

def reconcile(fleet, ec2)
  running = ec2.instances
               .filter("tag:ManagedBy", "fleet-tool")
               .each_with_object({}) { |i, h| h[i.tags["Name"]] = i if i.status == :running }

  # launch anything declared but not running
  (fleet.keys - running.keys).each do |name|
    puts "launching #{name}"
    launch(name, fleet[name], ec2)
  end

  # flag anything running but no longer declared
  (running.keys - fleet.keys).each do |name|
    puts "ORPHAN: #{name} is running but not in the fleet — terminate? "
  end
end

This desired − actual pattern is the heart of every infrastructure-as-code tool that came after, Terraform included. Declared-but-missing instances get launched; running-but-undeclared instances are surfaced as orphans for you to decide on (deliberately not auto-terminated — destroying machines should never be a silent side effect). Run it once and it converges the fleet; run it again and it does nothing, because reality already matches. That idempotence is the property that makes the tool safe to run on a schedule or in a panic.

From a fleet to a console

Wrap the pieces in a few subcommands and you have an operations console for your infrastructure:

case ARGV.first
when "status"     then print_status(FLEET, ec2)
when "reconcile"  then reconcile(FLEET, ec2)
when "ssh"        then exec_ssh(ARGV[1], ec2)   # ssh by Name tag, not IP
when "run"        then run_everywhere(ARGV[1..], ec2)  # a command on every web box
end

The ssh subcommand is a small thing that pays for the whole tool: you type ssh web-1 and it resolves the Name tag to the current public IP and connects — no more copy-pasting addresses out of the console. run fans a shell command across every instance of a role, which is how you check a version or tail a log across the fleet without logging into each box by hand. (This kind of name-addressable, fleet-wide command runner is exactly what a tool like guignol exists to be.)

Know the limits

This approach is deliberately modest, and it is worth being honest about where it stops. It manages instances; it does not manage the inside of them — that is the job of your baked AMIs plus a configuration tool. It has no real state file, so it infers actual state from tags every run, which is robust but coarse. And it is not a substitute for the dedicated tools arriving now: CloudFormation can express a whole stack declaratively, and the just-announced Terraform promises to do this across providers with a proper plan/apply cycle and a state model. When your infrastructure outgrows a single hash, move to those.

But do not skip building the small version first. Writing a few hundred lines of Ruby against the EC2 API teaches you, concretely, the ideas the big tools formalise: infrastructure as data, tags as identity, and reconciliation of desired against actual. Once those are in your hands, every infrastructure tool you meet afterwards is just a more powerful expression of the same three ideas — and you will use them better for having built them yourself.

#ruby #aws #devops