← all postsamirmuz.com →
2026-07-15prometheusgrafananode-exportertelegramalertingrocky-linuxpalo-altointermediate

Giving the Lab a Pager

In the previous post, a server died and stayed dead for five weeks before I noticed. The autopsy taught me the lesson every ops book repeats and every homelab ignores: a system nobody watches fails in silence.

12 min readseries: Homelab Infrastructure Series5 views

Why I built this the same day a server autopsy ended

In the previous post, a server died and stayed dead for five weeks before I noticed. The autopsy taught me the lesson every ops book repeats and every homelab ignores: a system nobody watches fails in silence.

So the same day, I gave my lab a pager. This post is the honest build log: Prometheus, node_exporter on six machines across two Linux families, Grafana, and an alert that buzzes my phone through Telegram. Including every mistake, because the mistakes taught me more than the happy path.

The goal was one sentence: if a machine goes down, my phone buzzes, without me looking at anything.

The architecture, and one deliberate placement decision

diagram

One decision I want to defend: monitoring does NOT live on the management VLAN. My management VLAN is the out-of-band plane, for hardware and hypervisor access only, the same way an enterprise keeps its OOB network clean. Monitoring is a workload. Workloads live in application VLANs, and the firewall gets one precise rule: the monitoring host, and only it, may reach the fleet on the node_exporter port. One source IP, one port, nothing else. Segmentation is only real if you keep honoring it when it is inconvenient.

Two Linux families, two installs

My fleet is mixed on purpose (I want the enterprise muscle memory), and installing the same agent on both families is a small lesson in itself:

Debian-family boxes get the packaged exporter, one line:

apt install prometheus-node-exporter

Rocky Linux boxes get the raw upstream binary plus a hand-written systemd unit, because the EPEL story for the exporter is not as clean. Download, unpack, create a dedicated user, write the unit, enable.

And here came gotcha number one: minimal Rocky images do not even ship tar. I spent real time suspecting my proxy and my firewall rules before discovering the box simply could not unpack the archive I had successfully downloaded. Install tar first. Check the boring thing first.

The gotchas that cost me real time

I am listing these because every one of them is invisible in tutorials:

  1. Validate the Prometheus config BEFORE restarting. promtool check config exists for a reason. Restart first and a syntax error takes your monitoring down, which is a special kind of irony. Related trap: in a non-login shell, /usr/local/bin may not be on PATH, so the tool "does not exist" until you call it by full path.
  2. A duplicate job_name puts Prometheus in a crash-loop. Two scrape jobs with the same name is not a warning, it is a refusal to start.
  3. Changing a target's labels creates a NEW time series. The old series lingers as a ghost until it ages out, so your dashboard briefly shows both. Do not panic, give it fifteen minutes.
  4. Alert threshold semantics matter. The rule that catches a dead machine is up is below 1. I first wrote "is below 0", which can never fire, because up is never negative. An alert that cannot fire is worse than no alert, because you trust it.
  5. The Telegram message template lives on the CONTACT POINT, not on the alert rule. I kept editing the rule's annotation and wondering why my formatting never appeared. The message box on the contact point, with parse mode HTML, is the one that renders.
  6. grafana_state_reason: Updated is housekeeping, not an outage. When you edit a firing rule, Grafana resolves and re-evaluates it. The "resolved" message that arrives right after you edit is the edit, not a recovery.

The drill: prove the pager works

A pager you have never tested is a rumor. So I ran the drill:

systemctl stop node_exporter

Two minutes later my phone buzzed: an alert named the machine, its IP, and its site. I started the exporter again. The resolved message followed on its own.

That buzz was the whole project. Not the dashboards, not the graphs. The buzz that arrives when I am not looking. (The real alert message is in the screenshots below, name, IP and site included, exactly as it landed on my phone.)

What this cost and what it bought

Total spend: zero ringgit. One small VM, packaged software, a free Telegram bot.

What it bought: the previous post can never happen to me the same way again. A disk that screams for two days will page me on day one. The lab now tells me when it is sick instead of waiting for me to visit.

If you run anything you care about, build the pager before you need the autopsy.

Pictures to add later

  1. Grafana dashboard with the 6 targets green (node exporter full dashboard)
  2. The Telegram DOWN alert on your phone (the money shot)
  3. The Telegram resolved message
  4. Optional: the firewall rule showing the single-source :9100 allow
← back to blog