The Ghost Was a Cable

A brand-new server that cannot find its own gateway

I built a backup vault for my lab: a small Rocky Linux container on the most restricted VLAN I have, the database zone. Fresh container, correct IP, correct gateway. First test:

ping 10.0.30.254        # the gateway
# 100% packet loss

ip neigh
# 10.0.30.254 dev eth0 FAILED

FAILED on the ARP line is the important part. ARP is the hello of Layer 2: before any IP packet moves, the machine shouts "who has 10.0.30.254?" and the gateway is supposed to answer. Nothing answered. My container was shouting into a void.

This post is about the two days I spent chasing that void through firewall zones and interface tables, and the fix, which turned out to be a screwdriver.

---

Theory one: the dead port

My lab has an unusual shape: the Palo Alto PA-220 is not just the firewall, it is also the switch. The three Proxmox nodes plug directly into its ports, and each port carries tagged sub-interfaces for my VLANs.

The firewall's interface table showed one port with a red link-down icon. I knew (or believed I knew) which node was plugged into which port, and by my mental map, the node hosting my new container hung off exactly that dead port. Case closed: dead uplink, no Layer 2, ARP can never resolve. I was one step from migrating the container to a different node to route around the problem.

Two things stopped me. First, the same node's untagged management traffic worked perfectly: it pulled OS templates from the internet daily. A truly dead port does not half-work. Second, a rule I had learned painfully two days earlier on this same firewall: never assert topology, prove it.

The MAC table does not have theories

Every switch keeps a table mapping MAC addresses to the physical port it learned them on. It is not a diagram somebody drew, not a memory of how things were cabled, not an assumption. It is what the hardware actually observed. On the PA-220:

show mac all

Two facts fell out, and they killed both my theories at once:

1. The hypervisor's MAC address was learned on ethernet1/5, a bare port that was never configured with any VLAN sub-interfaces. Not the "dead" port. Not the port from my mental map either. The cabling in my head and the cabling in the rack were two different labs.
2. The container's MAC appeared nowhere in the table. Its tagged VLAN frames were arriving at a port that had no matching sub-interface, so the firewall dropped them before learning anything. Shouting into a void, literally.

That also explained the half-working state perfectly: untagged management frames matched the port's basic config and passed, tagged VLAN frames found no sub-interface and died. "Management works but tagged VLANs die" is the signature of an untrunked port, and it is a different disease from a dead cable or a firewall policy drop.

The fix was a screwdriver

No config change fixed this, because no config was wrong. The ports I had prepared (with sub-interfaces, zones, VLAN objects) were correct and waiting. The cables were just in the wrong jacks. I re-cabled all three nodes to the layout I had intended all along, one node per prepared port, and labeled them this time.

ping 10.0.30.254
# 64 bytes from 10.0.30.254: icmp_seq=1 ttl=64 time=0.89 ms

ip neigh
# 10.0.30.254 dev eth0 lladdr 08:30:xx:xx:xx:xx REACHABLE

FAILED to REACHABLE on the first ping after the move. The MAC table now showed the container's address exactly where it belonged.

One step remained: the vault pulls backups from an app server in a different zone, and my inter-zone default is deny. One explicit security rule (backup vault to app host, SSH only, logged), with a before/after proof:

timeout 3 bash -c '</dev/tcp/10.0.20.10/22' && echo SSH-OPEN || echo SSH-BLOCKED
# before the rule: SSH-BLOCKED
# after the rule:  SSH-OPEN

What I keep from this

1. A red link-down icon usually means an empty jack, not a broken port. Ports do not die nearly as often as cables go missing.
2. Never trust the port numbering in your head. After any physical work, the patch panel and your memory drift apart. show mac address-table (or show mac all on PAN-OS) is the only map that cannot lie, because the hardware wrote it, not you.
3. Learn the signatures. ARP FAILED = Layer 2 problem, stop reading firewall policies. Management passes but tagged dies = missing trunk/sub-interface on that specific port. Each signature cuts your search space in half.
4. When two theories in a row die on contact with evidence, stop theorizing and go read the physical layer. It is cheaper than being clever.

The vault runs fine now. The ghost was never in the machine. It was in the space between what I remembered doing and what I had done.

---

Pictures to add later

1. PA-220 Network > Interfaces table (the link state column with red/green icons)
2. Terminal: ip neigh showing FAILED, then REACHABLE after the re-cable
3. Optional: the re-cabled ports, labeled (a photo makes this post feel real)