k3s Is Ubuntu, kubeadm Is Linux From Scratch

The question that stalls every homelab Kubernetes plan

"Should I run real Kubernetes, or is k3s cheating?"

I circled that question for days while planning my three-node lab, and no explanation stuck until one analogy did: k3s is Ubuntu, kubeadm is Linux From Scratch. Both are Linux.

Ubuntu is not fake Linux because the installer made choices for you. And k3s is not fake Kubernetes: it is a CNCF-certified conformant distribution. Same API, same kubectl, same YAML, same Deployments, Services, Ingress and RBAC. Skills transfer one to one. What k3s hides is the bootstrap pain: PKI, etcd assembly, control-plane wiring. Valuable pain, the LFS kind, worth feeling once later, but not the place to start when the goal is learning to OPERATE a cluster.

So: k3s, three nodes, one control plane, two agents. This post covers the one architecture ruling that shaped everything, the honest RAM math, the install (with the kernel panic), and a security finding that made my week.

The one ruling everything hangs on: the cluster stays stateless

Before installing anything, I locked one decision: Kubernetes gets no state. All state lives outside the cluster, in the database VLAN, on plain VMs.

PostgreSQL and object storage (MinIO) run as ordinary VMs in the most restricted zone of my lab. The cluster runs only stateless workloads.

Why this is the right call for a small lab, and not a cop-out:

Stateful Kubernetes is the hardest, most RAM-hungry part. StatefulSets, PVCs, storage classes, CSI drivers, operators, or a distributed storage layer like Longhorn or Ceph. On 24 GB nodes, that budget is better spent on workloads.
Cattle nodes. With zero persistent data on the cluster, I can rebuild any node, or the whole cluster, on a whim, without touching a byte of data. The freedom to tear down and rebuild is exactly what makes a lab a lab.
It is a legitimate production pattern, not a shortcut: compute on Kubernetes, data on dedicated hosts or a managed database service. Plenty of real companies run precisely this shape.

diagram

The honest RAM budget (because RAM is the wall, not CPU)

Three nodes, 24 GB each, about 2 GB reserved for the hypervisor. Everything has to fit, so I budgeted before building instead of discovering the wall later:

Node	Runs	Budget
node 0	k3s server 6 + Postgres primary 6 + dev DB 2 + headroom	22 GB
node 1	k3s agent 6 + Postgres standby 6 + GitLab CE 6 + runner 2	22 GB
node 2	k3s agent 6 + Prometheus 4 + Grafana 2 + MinIO 2 + vault 1	22 GB

Three findings from doing this on paper first:

GitLab CE is the single biggest consumer (6 GB, wants 8). It stays, because a working CI/CD pipeline is worth more than the RAM it costs, but it is the first thing I would trade for Forgejo (~0.5 GB) under pressure.
High availability doubles database RAM. The standby is not free. Nobody mentions this in "add a replica" tutorials.
CPU with overcommit is fine. RAM is the wall. On small hardware, budget RAM like money.

The install, including the kernel panic

The actual k3s install is famously short. Server first, then agents join with a token:

# on the server node
curl -sfL https://get.k3s.io | sh -

# on each agent
curl -sfL https://get.k3s.io | K3S_URL=https://<server-ip>:6443 K3S_TOKEN=<token> sh -

Copy the kubeconfig to your workstation, point KUBECONFIG at it, and the cluster answers from anywhere with a route and the credential:

kubectl get nodes
# k3s-0   Ready    control-plane,etcd,master
# k3s-1   Ready    <none>
# k3s-2   Ready    <none>

(A cluster is an API server; kubectl is just an authenticated HTTPS client. No SSH involved. The kubeconfig is the keycard.)

But before the happy screenshot, the VMs would not even boot. Rocky Linux 9 kernel-panicked instantly on first start, and the console said why:

Fatal glibc error: CPU does not support x86-64-v2

RHEL 9 family distros require the x86-64-v2 CPU baseline, and a hypervisor's default virtual CPU model (kvm64) predates it. The VM presents an ancient CPU to the guest and glibc refuses to run. One flag fixes it, presenting the real host CPU to the guest:

qm set <vmid> --cpu host

All three nodes booted. If you run RHEL-family guests on Proxmox or QEMU, --cpu host (or any v2-capable model) is simply required.

Policies before workloads

The first object applied to the cluster was not an app. It was a default-deny NetworkPolicy in the namespace, before any workload existed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: amirmuz
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

Every future pod starts mute and isolated, and each permission gets added explicitly, on purpose. The reverse order (apps first, lockdown "later") is how "later" becomes "never".

And here is the finding that genuinely pleased me: the cluster needed ZERO new firewall rules. My VLAN zone rules already covered app to db, management to app, and app to internet. Kubernetes did not change the security architecture, it moved in as a tenant of it. Two independent enforcement layers now exist: the firewall between VLANs, NetworkPolicy between pods. Segmentation done early pays dividends you did not plan for.

Takeaways

Pick the distribution that matches what you are trying to LEARN. Operating a cluster: k3s. Feeling the bootstrap pain: kubeadm, later, deliberately.
Decide where state lives before the first node boots. Stateless-cluster is a real pattern that buys rebuild freedom and skips the hardest k8s topic.
Budget RAM on paper first. The wall is real and it is not CPU.
--cpu host for RHEL 9 family guests. Learn it from a blog post instead of a kernel panic.
Default-deny before the first workload. Policies-then-apps is a one-way door that only stays open in this direction.

Pictures to add later

kubectl get nodes showing three Ready nodes (the money shot)
The kernel panic console with the x86-64-v2 error (if screenshot exists)
kubectl describe networkpolicy default-deny-all output