The Server That Died in Silence

The node was grey for five weeks and nobody noticed

Here is an uncomfortable question for anyone running a homelab: if one of your servers died right now, how long would it take you to find out?

For me, the honest answer turned out to be five weeks.

One evening I opened the Proxmox web UI and saw it: pve-main, the Dell R630 that hosts my main services, was grey. Not red, not alarmed. Grey. The node itself was unreachable, its management plane dead. And yet, strangely, the services running on it were still up. My website was still serving. The virtual machines still answered.

A server that is dead and alive at the same time. This is the story of how I performed an autopsy on it without ever rebooting it, what the machine's own black box revealed, and the lesson that changed how I run everything.

---

A zombie node

First, why were the services still alive?

The answer is Ceph. My cluster stores VM disks on a replicated Ceph pool that spans multiple nodes. When pve-main's own system disk died, the Linux kernel and the running guests kept going from memory, and every disk read and write they made went over the network to the surviving Ceph replicas. The node was a zombie: kernel alive in RAM, brain gone, hands still working.

That is also why this failure was so quiet. Nothing user-facing broke. No page went down. The failure hid inside the redundancy that was supposed to protect me, and it succeeded for five weeks.

Do not trust your first theory

My first theory was wrong. So was my second.

With no SSH into the node, I was reading the situation from the outside: cluster status, screenshots, half-remembered hardware layout. I convinced myself twice that I knew which disk had failed, and both times the conviction was built on absence of evidence, not evidence. Each time I had to stop and ask myself the question every debugger should tattoo somewhere visible: how am I actually sure of this? Both times the honest answer was "I am not, it just fits the story I already have."

What finally settled it was not cleverness. It was the machine's own flight recorder.

The iDRAC Lifecycle Log is a black box, use it

Dell servers carry iDRAC, an out-of-band management controller that keeps working even when the operating system is gone. And iDRAC keeps a Lifecycle Log: a timestamped record of every hardware event the machine has ever seen.

Reading it was like reading an autopsy report:

• June 7: Disk 0 in bay 0 reports unrecoverable media errors. Six of them.
• June 9, 14:45:16: the disk fails permanently.
• June 9, 14:45:24: eight seconds later, the PERC RAID controller deletes the virtual disk that lived on it.

The disk had been screaming for two full days before it died, and nobody was listening.

The deleted virtual disk was the OS boot volume, and here is the design mistake that made it fatal: it was a single-disk RAID-0. No mirror. One disk, one point of failure, carrying the operating system of my most important node. The five other disks in the machine were healthy the whole time. The failure was exactly as big as my laziest provisioning decision, no bigger.

The second lie: frozen dashboards

There was one more trap inside this incident, and it is the one that scares me most in hindsight.

My Ceph status had been reporting healthy the entire time. The reason: the Ceph manager daemon that publishes those statistics had died with the node, and the numbers on the dashboard were simply the last ones it ever published. Frozen. When I recreated the manager on a surviving node, the real state appeared: the pool was degraded, running on two out of three copies, one more disk loss away from freezing all I/O.

A dead messenger does not report its own death. If your monitoring can fail in a way that looks identical to good news, you do not have monitoring.

What I did, and deliberately did not do

The most important decision of the whole incident: do not reboot the zombie. The node only worked because the kernel was still in RAM. A reboot with no boot disk would have turned a quiet degradation into a hard outage. Instead:

1. Evacuate what matters. I migrated data and services off the dying environment.
2. Buy the right disk, not the cheapest. The replacement is an enterprise SSD with power-loss protection, because the boot disk of a hypervisor deserves better than whatever is lying around.
3. Write the rebuild as a runbook BEFORE touching hardware, including backing up the node's cluster configuration before removing it, because Proxmox deletes a node's config directory when you remove it from the cluster. A rebuild improvised at the keyboard is where the second disaster comes from.

The lesson that actually changed something

A system nobody watches will fail in silence. Redundancy does not tell you when it is being consumed. Dashboards can freeze while smiling at you.

The same day I finished this autopsy, I built a monitoring stack for my lab with alerting straight to my phone, and I tested it by killing a service and watching the alert arrive. That story is its own post, because the two belong together: this post is the disease, that one is the cure.

The disk screamed for two days. Next time, something will be listening.

---

Pictures to add later

1. iDRAC Physical Disks view (the 5 healthy disks, bay 0 empty/failed)
2. iDRAC Virtual Disks view (the surviving VDs, VD4 gone)
3. The Lifecycle Log entries from Jun 07 and Jun 09 (the money shot)
4. Proxmox UI with pve-main grey (if screenshot exists)