Seven Machines, One Command

A fleet you cannot touch

This weekend I was away from my homelab for two days. All I left behind was one Windows machine that could reach the lab's management network, which gave me the Proxmox web GUI and a console. No physical access, no keyboard on any server.

That sounded like the perfect weekend to finally learn Ansible. My reasoning: if I can manage the whole fleet from a borrowed screen two towns away, then the automation is real. If I can only manage it while standing next to it, I have not automated anything.

By the end of the evening, seven machines answered one command at once. This post is how I got there, including the walls I hit, because the walls taught more than the wins.

What Ansible actually is

Ansible needs nothing installed on the machines it manages. No agents, no daemons. It is a program on ONE machine (the control node) that connects to every other machine over plain SSH and runs small Python snippets there.

You describe the machines in an inventory file. Groups in the inventory work like contact groups in a phone: k3s for my three Kubernetes VMs, monitoring for the observability container, lab as the group of groups that means everyone.

[k3s]
k3s-0 ansible_host=10.0.20.50 ansible_user=rocky
k3s-1 ansible_host=10.0.20.51 ansible_user=rocky
k3s-2 ansible_host=10.0.20.52 ansible_user=rocky
 
[lab:children]
cicd
monitoring
k3s
app_hosts

One detail that took a moment to click: ansible.cfg is read from the current directory. Your project folder IS the configuration scope. Stand in the wrong folder and Ansible quietly uses different settings.

Building the control seat from a command line

The control node itself did not exist yet. I could have clicked through the Proxmox GUI wizard, but I chose the CLI on purpose: I wanted to know what the wizard actually does. One command creates a container:

pct create 144 local:vztmpl/rockylinux-9-default_20240828_amd64.tar.xz \
  --hostname ansible-01 --memory 1024 --cores 1 \
  --net0 name=eth0,bridge=vmbr0,tag=20,ip=10.0.20.30/24,gw=10.0.20.254 \
  --rootfs local-lvm:8 --unprivileged 1

My first attempt failed: the template did not exist on that node. Two lessons in one error. First, local storage is per-node, not shared: a template downloaded on one Proxmox node is invisible to its brothers, so pveam download has to run on the node that will host the container. Second, pct create cleans up after itself: it removed the half-made disk volume on failure and left no debris. A failed create is free.

Three doors to seven machines

Ansible speaks SSH, so the control node's public key has to reach every machine. This turned into the most educational hour of the weekend, because the fleet has three kinds of doors:

Door 1: containers. For LXC containers you do not need SSH at all to plant a key. From the Proxmox node hosting them, pct exec runs commands directly inside:

pct exec 141 -- mkdir -p /root/.ssh
pct exec 141 -- sh -c 'grep -qxF "ssh-ed25519 AAAA... root@ansible-01" /root/.ssh/authorized_keys || echo "ssh-ed25519 AAAA... root@ansible-01" >> /root/.ssh/authorized_keys'

The grep guard before the append means I can run it twice without duplicating the key. That is idempotence, hand-written. Remember that word.

Door 2: the machine with no door. One container refused SSH entirely. The minimal Rocky Linux template ships WITHOUT an SSH server. dnf install -y openssh-server && systemctl enable --now sshd first, then the key.

Door 3: the VMs, where root is not the door. ssh-copy-id as root failed on every VM, and that is healthy. My Kubernetes VMs came from a cloud image whose default user is rocky. My app host is Debian, which disables root password SSH by default, so its door is my personal user. On a real fleet, root is almost never the door, and the inventory records the truth per machine with ansible_user=.

One more hard-won rule: never batch-paste interactive commands. Each ssh-copy-id stops to ask for a password, and the password prompt EATS the next pasted line. One command, one paste, read the output.

Seven pongs

ansible lab -m ping

Seven machines, seven green pong replies, and I was not in the building. That moment is why this post exists.

The first role: encode what your hands know

Monitoring in my lab covers six machines, but the three Kubernetes VMs were blind spots. I had installed Prometheus node_exporter by hand before, once, on one machine. An Ansible role is how you turn that hand-knowledge into fleet knowledge: a folder with conventional names that any playbook can apply with one line.

roles/node_exporter/
├── tasks/main.yml        # what to do
├── templates/            # files with variables
├── handlers/main.yml     # reactions that fire only on change
└── files/                # artifacts to push

Handlers deserve a sentence: a normal task runs every time, but a handler runs ONLY when a task that notifies it reports "changed", once, at the end. Edit the systemd unit file next month and the service restarts automatically. Run the playbook when nothing changed and nothing restarts. Shell scripts cannot make that distinction.

The wall: a handshake cut short

The role's download task failed on all three VMs with the same strange error:

Failure downloading https://github.com/prometheus/node_exporter/.../node_exporter-1.8.2.linux-amd64.tar.gz,
Connection failure: [ASN1: NOT_ENOUGH_DATA] not enough data (_ssl.c:4192)

Earlier that evening, ansible-galaxy on the control node had failed with the IDENTICAL error reaching a completely different site. Two different source machines, two different destinations, one identical TLS error. Something was mangling handshakes.

Instead of guessing, I ran one discriminating test on the control node:

curl -fSLo /tmp/node_exporter.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz

It downloaded 10 MB at full speed. That single command split the world in two: the lab's internet path was fine, and the problem lived specifically in Python's downloader (both Ansible's unarchive and ansible-galaxy download with Python). I did not fully root-cause the Python TLS issue that night, and that is okay, because the workaround turned out to be the better design anyway.

Push, don't pull

I downloaded the tarball once on the control node, parked it in the role's files/ directory, and changed one task: remove the URL and remote_src, and unarchive now copies the artifact over SSH and unpacks it on the target.

- name: unpack from control node
  ansible.builtin.unarchive:
    src: node_exporter-1.8.2.linux-amd64.tar.gz
    dest: /tmp
    creates: /tmp/node_exporter-1.8.2.linux-amd64

This is how real fleets work. Production servers usually cannot reach the open internet at all, on purpose; my own database VLAN has no internet by design. Artifacts flow from one controlled distribution point. The TLS flake forced me into the pattern I should have started with.

The proof is changed=0

Ansible's promise over shell scripts is idempotence: you describe the state you want, and re-applying is free. I watched it work live, mid-failure. On the first run the "create node_exporter user" task said changed. On the second run, with the download still broken further down, that same task said ok. The user already existed, so Ansible touched nothing.

The graduation test for any playbook: run it twice. The second run must end with changed=0 on every host. Not because nothing executed, but because every task checked its condition and found it already satisfied.

My second run, from the real console, just after half past midnight:

PLAY RECAP *********************************************************************
k3s-0    : ok=5    changed=0    unreachable=0    failed=0    skipped=1
k3s-1    : ok=5    changed=0    unreachable=0    failed=0    skipped=1
k3s-2    : ok=5    changed=0    unreachable=0    failed=0    skipped=1

That skipped=1 is not a problem: it is the creates: guard on the unpack task declining to redo work that is already done. A skip on round two IS the guard working.

And the proof the service actually lives:

curl -s http://10.0.20.50:9100/metrics | head -3
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0

I ran the curl twice and the garbage-collection value changed between the two calls. That small difference matters: it means a real process is breathing on the other end, not a cached page.

What I actually learned

• The prompt is the tell. Twice I pasted commands into the wrong context: once into the Proxmox node instead of the container, once into the wrong directory. Both times, the shell prompt had been announcing it the whole time. Read the hostname AND the path before every paste.
• Root is not the door. Real fleets have different users per machine, and the inventory is where that truth lives.
• One discriminating test beats ten guesses. The curl test did not fix anything; it told me exactly WHERE the problem was not.
• Push beats pull for getting artifacts onto a fleet. One controlled download, then SSH does the distribution.
• Idempotence is the whole point. If you cannot run it twice safely, you wrote a script, not automation.

Pictures for this post

• Terminal: ansible lab -m ping with the seven pong replies
• Terminal: second playbook run ending in changed=0 on all three hosts
• Terminal: curl -s http://10.0.20.50:9100/metrics | head -3 showing live metrics