← all postsamirmuz.com →
2026-07-17post-mortempalo-altolayer2troubleshootingpan-osvlanzonestcpdumpadvanced

The Switch Was the Firewall

My CI runner could not SSH to my production host. Same VLAN, both up, both healthy, and every packet between them disappeared. No firewall log. No error. No RST. Just silence.

11 min readseries: Homelab Infrastructure Series11 views

Two machines, same VLAN, and packets that vanish without a trace

My CI runner could not SSH to my production host. Same VLAN, both up, both healthy, and every packet between them disappeared. No firewall log. No error. No RST. Just silence.

It took two sessions, a diagnostic ladder through every layer, one wrong theory, and one config screen to find the truth: the box I thought was only my firewall was also my switch, and switched traffic was dying somewhere no log can see.

This is the story, told in the order it unfolded, because the order is the lesson.

The ladder: prove each layer before blaming the next

Rung 1: is Layer 2 alive? After a failed ping, check the ARP table:

ip neigh
# 10.0.20.10 dev ens18 lladdr bc:24:xx:xx:xx:xx REACHABLE

ARP resolved with a real MAC. The two machines share a broadcast domain and can exchange ARP. This matters enormously: it rules out cabling, VLAN tags, and dead links in one command. (I initially hand-waved "maybe they are on different VLANs" and got called on it. Never assert topology, prove it.)

Rung 2: are the hosts themselves blocking? ufw inactive, iptables -S all-ACCEPT on both ends. One honest note for your own ladder: native nftables rules hide from iptables -S, so nft list ruleset is the real rule-out.

Rung 3: is the hypervisor firewall involved? pve-firewall status disabled, and a second container on the same node was equally blocked, so it was not CT-specific.

Rung 4: the verdict machine, tcpdump on the destination.

tcpdump -ni ens18 host 10.0.20.21
# ARP request... ARP reply... and then, for ICMP and TCP: nothing. ever.

That result is a signature worth memorizing: ARP passes, IP dies. ARP is being bridged between the machines, IP packets are being eaten in transit. Something INLINE between the two hosts forwards Layer 2 chatter but filters Layer 3, which is exactly how an inline firewall behaves.

The wrong theory, and the config screen that ended it

My first inline suspect was a virtual wire (a transparent bump-in-the-wire firewall mode). Wrong: the Virtual Wires tab was empty. An empty vwire tab does not acquit the firewall though, and this is where the real discovery happened.

Opening Network > Interfaces changed everything I thought I knew about my own lab: there is no separate switch. The PA-220 IS the switch. The Proxmox nodes plug directly into its ethernet ports. Each port carries tagged Layer 2 sub-interfaces (.10, .20, .30), bridged per VLAN by VLAN objects, with L3 SVIs (vlan.10, vlan.20, vlan.30) acting as each VLAN's gateway. Traffic between two machines on the same VLAN never gets routed; it gets switched through those L2 sub-interfaces.

And in the interface table, one column told the whole story. The tagged L2 sub-interfaces had Security Zone: none.

diagram

Why it was invisible: the drop happens before policy exists

On PAN-OS, a packet must belong to a security zone before security policy can even be evaluated. A Layer 2 sub-interface with no zone drops switched traffic pre-policy: before any rule matches, before any rule can log. That produced the three facts that made this hunt so disorienting:

  • Nothing in the traffic log, even after I enabled logging on the default rules. You cannot log a packet that never reached policy.
  • ARP still worked, because ARP is not IP; it rides the bridge.
  • Routed traffic worked perfectly. Internet access, cross-VLAN flows, my VPN path to the host: all fine, because routed flows classify through the SVI's L3 zone. Only switched, same-VLAN, node-to-node traffic died.

A related trap that cost me an evening earlier in this saga: PAN-OS default deny rules do not log out of the box. An empty traffic log filter does not mean traffic is not being blocked. Silence IS the deny. Override interzone-default and intrazone-default to log at session end; you want the firewall to narrate its refusals.

The fix: give the L2 interfaces a home

Zones have types on PAN-OS. An L2 sub-interface can only join a Layer2-type zone. So: create l2-web, l2-app, l2-db, one per VLAN, and put each VLAN's sub-interfaces inside its zone. Switched same-VLAN traffic then classifies as intrazone (same zone in and out), and the intrazone-default rule allows it.

Commit. Retry. The runner reached the host on the first attempt, and the deploy pipeline that had been fail-safing at its first gate went green end to end.

Takeaways

  • "ARP passes, IP dies, and nothing logs anywhere" on Palo Alto: think zoneless L2 sub-interface first, virtual wire second. Check the interface TYPE column and the Security Zone column before any more captures.
  • Zoneless-interface drops are invisible to all logging. The only smell is "Security Zone: none" in the interfaces table. Know where your platform drops packets before policy, because no amount of log-staring finds those.
  • Climb the ladder in order: ARP table, host firewalls, hypervisor firewall, tcpdump on the NIC. When both endpoints audit clean, stop capturing and read the interconnect's CONFIG. The answer was in a table the whole time.
  • Know what your boxes actually are. I ran this firewall for weeks without internalizing that it was also my switch, and that switched and routed traffic take completely different paths through it, with completely different failure modes.

Pictures to add later

  1. PA-220 Network > Interfaces: the L2 subinterfaces with the Security Zone column (the "none" smell)
  2. The Zones page showing the Layer2-type zones after the fix
  3. Optional: tcpdump output showing ARP reply followed by silence
← back to blog