Skip to content

IT Help Blog

Plain English tech help for small business owners. No jargon, just solutions.

Need hands-on help? Get in touch →

CKA Road Trip: Kubernetes Health Endpoints

Every major Kubernetes component exposes HTTP endpoints you can curl to check if it's alive. Useful when kubectl isn't working and you need to verify what's actually running.


The Endpoints

# apiserver
curl -k https://localhost:6443/healthz
curl -k https://localhost:6443/livez
curl -k https://localhost:6443/readyz
curl -k https://localhost:6443/readyz?verbose   # shows each check by name

# kubelet
curl -k https://localhost:10250/healthz

# scheduler
curl -k https://localhost:10259/healthz

# controller-manager
curl -k https://localhost:10257/healthz

# etcd — needs certs
curl -k https://localhost:2379/health \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  --cacert /etc/kubernetes/pki/etcd/ca.crt

All return ok when healthy.

/readyz?verbose is the most useful — shows each individual check:

[+] ping ok
[+] etcd ok
[+] poststarthook/start-informers ok
[-] some-check failed   ← tells you exactly what's wrong

Where to Run These From

This is the part that trips people up. localhost means different things depending on where you are.

From the controlplane node (SSH'd in)

You are on the Linux host. localhost here is the node itself.

ssh controlplane

curl -k https://localhost:6443/healthz      # reaches apiserver ✓
curl -k https://localhost:10250/healthz     # reaches kubelet ✓
curl -k https://localhost:10259/healthz     # reaches scheduler ✓
curl -k https://localhost:10257/healthz     # reaches controller-manager ✓
curl -k https://localhost:2379/health ...   # reaches etcd ✓

All components run on the controlplane node, so localhost works for all of them.

From a worker node (SSH'd in)

You are on a different Linux host. The apiserver, etcd, scheduler, controller-manager are NOT here.

ssh node01

curl -k https://localhost:10250/healthz     # reaches THIS node's kubelet ✓
curl -k https://localhost:6443/healthz      # FAILS — apiserver not on this node ✗
curl -k https://172.30.1.2:6443/healthz    # works — using controlplane IP ✓

From inside a pod (kubectl exec)

This is the most confusing one. When you kubectl exec into a pod, you are inside a container. That container has its own network namespace — its own localhost, its own loopback. It is completely separate from the node's network.

kubectl exec -it some-pod -- /bin/sh

# inside the container:
curl localhost:6443       # FAILS — localhost here is the container, not the node
curl localhost:10250      # FAILS — same reason

# to reach the apiserver from inside a container:
curl -k https://kubernetes.default.svc.cluster.local/healthz   # ✓
curl -k https://10.96.0.1/healthz                               # ✓ (kubernetes service ClusterIP)

# scheduler and controller-manager — NOT reachable from pods at all
# they only bind to localhost on the controlplane node, intentionally

Why scheduler and controller-manager are localhost-only

They don't need to accept connections from anything except the apiserver, and the apiserver talks to them on the same node. Binding to an external interface would expose them unnecessarily. So they listen on 127.0.0.1 only — unreachable from pods or other nodes.


The Mental Model

controlplane node
  127.0.0.1:6443    ← apiserver    (also on node IP — reachable from anywhere)
  127.0.0.1:10250   ← kubelet      (also on node IP)
  127.0.0.1:10259   ← scheduler    (localhost ONLY)
  127.0.0.1:10257   ← controller-manager (localhost ONLY)
  127.0.0.1:2379    ← etcd         (localhost ONLY)

worker node
  127.0.0.1:10250   ← kubelet (its own kubelet)

pod/container
  127.0.0.1         ← the container itself, nothing else
  10.96.0.1         ← kubernetes service → routes to apiserver

The key distinction: localhost inside a container is the container's own loopback. It has nothing to do with the node it's running on.

697

CKA Road Trip: Kubernetes Networking — From the Ground Up

Kubernetes networking is confusing because there are multiple layers of "network" stacked on top of each other. Once you understand each layer and what it owns, it stops being magic.


Layer 0 — The Linux Host Network

Before Kubernetes exists, you have a Linux machine with a network interface:

controlplane node
  eth0: 172.30.1.2     ← the real IP of this machine
  lo:   127.0.0.1      ← loopback, local to this machine only

This is the node network. Machines talk to each other here. 172.30.1.2 is reachable from node01 at 172.30.2.2. Normal networking.


Layer 1 — Linux Network Namespaces

This is where containers come in. When a container starts, the kernel creates a network namespace for it. Think of a network namespace as a completely separate, isolated copy of the networking stack.

Inside a network namespace: - its own network interfaces - its own IP address - its own routing table - its own loopback (127.0.0.1)

The container has no idea the host network exists. Its localhost is its own loopback, not the node's.

host network namespace          container network namespace
  eth0: 172.30.1.2               eth0: 192.168.1.5  ← pod IP
  lo:   127.0.0.1                lo:   127.0.0.1     ← container's OWN loopback

These are two completely separate localhostes. This is the source of most networking confusion.


Layer 2 — The veth Pair (The Wire)

A container in its own namespace can't talk to anything. It needs a wire connecting it to the outside world.

That wire is a veth pair — two virtual network interfaces connected like a cable. What goes in one end comes out the other.

host side                    container side
  veth_abc123  ←──────────→  eth0 (inside container)
  (on the bridge)              (pod IP: 192.168.1.5)

The host end plugs into a bridge (think: a virtual network switch). The container end is the pod's eth0. Every pod gets one veth pair.


Layer 3 — The Bridge (The Switch)

The bridge connects all the veth pairs on a node. Pods on the same node talk through the bridge.

         bridge (cni0: 192.168.1.1)
         /            \
   veth_pod_A      veth_pod_B
       |                |
   pod A             pod B
192.168.1.2      192.168.1.3

Pod A pings Pod B: 1. Pod A sends packet to 192.168.1.3 2. Goes through veth_pod_A to the bridge 3. Bridge forwards to veth_pod_B 4. Pod B receives it

No iptables, no routing — pure L2 switching on the same node.


Layer 4 — Pod IPs (Cross-Node)

Pods on different nodes need to reach each other too. The CNI plugin (Flannel, Cilium, Calico) handles this.

Each node gets a block of pod IPs:

controlplane: pods get 192.168.0.0/24
node01:       pods get 192.168.1.0/24

Cross-node traffic goes through the CNI plugin — either encapsulated in a tunnel (Flannel VXLAN) or routed directly (Calico BGP). The pod doesn't know or care. It just sends to the destination pod IP and the CNI handles getting it there.

Key point: every pod in the cluster gets a unique IP. Any pod can reach any other pod directly by IP — no NAT, no port mapping needed. This is the Kubernetes networking model.


Layer 5 — Services and ClusterIP

Pod IPs are ephemeral. A pod dies, its IP is gone. A new pod gets a new IP. You can't hardcode pod IPs.

A Service gives you a stable IP that never changes. It's called a ClusterIP.

nginx-service   ClusterIP: 10.96.45.123:80
              load balances to:
                pod-A: 192.168.1.2:80
                pod-B: 192.168.1.3:80

But here's the thing — the ClusterIP is not assigned to any interface. You can't ping it. It exists only as an iptables rule written by kube-proxy on every node.

When a pod sends traffic to 10.96.45.123:80: 1. Packet hits the iptables KUBE-SERVICES chain 2. iptables randomly picks a pod IP (load balancing) 3. DNAT rewrites the destination to the pod IP 4. Packet routes normally to the pod

The ClusterIP is just a hook in iptables. That's it.


Layer 6 — DNS

Nobody remembers 10.96.45.123. DNS maps service names to ClusterIPs.

Every pod has /etc/resolv.conf pointing at CoreDNS:

nameserver 10.96.0.10      ← CoreDNS ClusterIP
search default.svc.cluster.local svc.cluster.local cluster.local

When a pod does curl nginx-service: 1. DNS lookup: nginx-service → resolves to nginx-service.default.svc.cluster.local 2. CoreDNS returns 10.96.45.123 3. Pod sends to 10.96.45.123 4. iptables DNAT → pod IP 5. Packet reaches the pod


The localhost confusion — fully explained

Now that you understand network namespaces:

controlplane node (host namespace)
  127.0.0.1:6443  ← apiserver listening here

pod on that node (its own namespace)
  127.0.0.1       ← the pod's OWN loopback
                    completely separate from the node's loopback

When you kubectl exec into a pod and run curl localhost:6443 — you are inside the pod's network namespace. Its localhost is its own loopback. The apiserver is on the node's loopback, which is a completely different network namespace. The packet never leaves the pod's namespace, never reaches the node, never reaches the apiserver.

To reach the apiserver from inside a pod:

# use the kubernetes service — this IS reachable from any pod
curl -k https://kubernetes.default.svc.cluster.local/healthz
# or
curl -k https://10.96.0.1/healthz   # kubernetes service ClusterIP

This works because 10.96.0.1 is a ClusterIP — iptables on the node rewrites it to the apiserver's actual IP and port.


The Full Picture

                    ┌─────────────────────────────────┐
                    │         controlplane node         │
                    │  eth0: 172.30.1.2  (node IP)     │
                    │  lo:   127.0.0.1   (node loopback)│
                    │    apiserver: 127.0.0.1:6443      │
                    │    etcd:      127.0.0.1:2379      │
                    │                                   │
                    │  ┌─────────────┐                  │
                    │  │   pod A     │ ← own namespace  │
                    │  │ 192.168.0.2 │                  │
                    │  │ lo:127.0.0.1│ ← pod's loopback │
                    │  └──────┬──────┘                  │
                    │      veth pair                    │
                    │      bridge cni0                  │
                    └─────────────────────────────────┘
                              │ cross-node via CNI
                    ┌─────────────────────────────────┐
                    │           node01                  │
                    │  eth0: 172.30.2.2                 │
                    │  ┌─────────────┐                  │
                    │  │   pod B     │                  │
                    │  │ 192.168.1.3 │                  │
                    │  └─────────────┘                  │
                    └─────────────────────────────────┘

services (ClusterIP) — exist only as iptables rules, on every node
DNS (CoreDNS)        — a pod, reachable via its own ClusterIP 10.96.0.10

The Rules Worth Memorising

Pod to pod (same node): direct via bridge — no NAT, no routing.

Pod to pod (different node): CNI handles it — pod just sends to the pod IP.

Pod to service: iptables DNAT on the node — ClusterIP gets rewritten to a pod IP.

Pod to apiserver: use kubernetes.default.svc.cluster.local — never localhost.

Node to component: localhost works — you're on the same host.

localhost inside a container: the container's own loopback only — nothing else.

697

CKA Road Trip: Every Path That Matters in Kubernetes

Kubernetes isn't one thing in one place. It's a set of components, each with their own config files, certs, and data directories spread across the filesystem. When something breaks, knowing where to look is half the fix.


/etc/kubernetes/

The main Kubernetes config directory. Lives on the controlplane node.

/etc/kubernetes/
  manifests/                  # static pod manifests — control plane lives here
    kube-apiserver.yaml
    kube-controller-manager.yaml
    kube-scheduler.yaml
    etcd.yaml
  pki/                        # all TLS certs and keys
    ca.crt / ca.key           # cluster CA
    apiserver.crt / apiserver.key
    apiserver-etcd-client.crt / .key
    apiserver-kubelet-client.crt / .key
    etcd/
      ca.crt
      server.crt / server.key
  kubelet.conf                # kubelet's kubeconfig
  controller-manager.conf
  scheduler.conf
  admin.conf                  # admin kubeconfig — source of ~/.kube/config

manifests/ — the kubelet watches this directory directly. No API server involved. Drop a yaml in, the pod starts. Edit it, the pod restarts. This is how the control plane bootstraps itself and why you fix broken control plane components by editing files here, not with kubectl.

pki/ — every TLS cert the cluster uses. apiserver cert, etcd client certs, kubelet client certs. When you see x509: certificate errors, the answer is in here.


~/.kube/config

kubectl's kubeconfig. Where kubectl gets the server address, port, and credentials.

clusters:
- cluster:
    server: https://172.30.1.2:6443   # ← port typo here = kubectl dead
    certificate-authority-data: ...
  name: kubernetes
users:
- name: kubernetes-admin
  user:
    client-certificate-data: ...
    client-key-data: ...

If kubectl can't connect, check this file first. The error message will tell you the URL it's trying — if the port looks wrong, it came from here.

cat ~/.kube/config | grep server

/var/lib/kubelet/

Kubelet runtime data. Lives on every node.

/var/lib/kubelet/
  config.yaml          # kubelet configuration — cgroup driver, eviction thresholds
  kubeconfig           # kubelet's auth to the apiserver
  pki/
    kubelet.crt / kubelet.key
    kubelet-client-current.pem

config.yaml — if the kubelet won't start, this is usually why. Malformed config, wrong cgroup driver, missing fields.

cat /var/lib/kubelet/config.yaml
journalctl -u kubelet -n 50 --no-pager

/var/lib/etcd/

etcd's data directory. The actual cluster database.

/var/lib/etcd/
  member/
    snap/      # snapshots
    wal/       # write-ahead log

You don't edit files here directly. You interact with etcd via etcdctl. But this is where the data lives — and this is what you're backing up when you run etcdctl snapshot save.

If this directory is corrupted or missing, the cluster loses all state.


/etc/cni/net.d/

CNI plugin configuration. Tells the container runtime which CNI plugin to use and how.

/etc/cni/net.d/
  10-flannel.conflist     # if using Flannel
  10-calico.conflist      # if using Calico
  05-cilium.conflist      # if using Cilium

If pods are stuck in ContainerCreating with network errors, check here. The CNI config might be missing or malformed.


/opt/cni/bin/

CNI plugin binaries. The actual executables that set up pod networking.

ls /opt/cni/bin/
# flannel  bridge  host-local  loopback  portmap  ...

If the CNI binary is missing, pods can't get IPs. The config in /etc/cni/net.d/ points at a binary that doesn't exist.


/var/log/pods/

Container logs on disk. Organised by namespace, pod name, pod UID, container name.

/var/log/pods/
  <namespace>_<pod-name>_<pod-uid>/
    <container-name>/
      0.log    # current log file
      1.log    # rotated

kubectl logs reads from here and strips the JSON wrapper. When kubectl isn't available — node issues, apiserver down — you can read logs directly:

cat /var/log/pods/kube-system_kube-apiserver-controlplane_*/kube-apiserver/0.log

/var/log/containers/

Symlinks to /var/log/pods/. Older tooling uses this path. Same data, different entrypoint.

ls /var/log/containers/
# kube-apiserver-controlplane_kube-system_kube-apiserver-abc123.log -> /var/log/pods/...

/run/containerd/

containerd's runtime socket. How kubectl exec, kubectl logs, and the kubelet talk to containerd.

/run/containerd/
  containerd.sock    # the Unix socket

If containerd is dead, this socket won't exist or won't respond. crictl and the kubelet both talk through here.

systemctl status containerd
crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps

/var/lib/containerd/

containerd's data directory. Images and container layers live here.

/var/lib/containerd/
  io.containerd.content.v1.content/
    blobs/sha256/          # raw image layer blobs
  io.containerd.snapshots.v1.overlayfs/
    snapshots/             # unpacked OverlayFS layers
  io.containerd.metadata.v1.bolt/
    meta.db                # metadata database

If a node is running out of disk space, this directory is usually why. Image layers accumulate.

du -sh /var/lib/containerd/
crictl images    # see what's cached
crictl rmi --prune   # remove unused images

The Troubleshooting Map

kubectl can't connect
  → ~/.kube/config (wrong server, port typo)

control plane component broken
  → /etc/kubernetes/manifests/ (fix the static pod yaml)

TLS / cert errors
  → /etc/kubernetes/pki/

kubelet won't start
  → /var/lib/kubelet/config.yaml
  → journalctl -u kubelet

pod stuck in ContainerCreating
  → /etc/cni/net.d/ (CNI config)
  → /opt/cni/bin/ (CNI binary missing)

container logs when kubectl isn't working
  → /var/log/pods/

node disk pressure
  → /var/lib/containerd/ (image layer bloat)

etcd backup / restore
  → /var/lib/etcd/ (data lives here)
  → /etc/kubernetes/pki/etcd/ (certs for etcdctl)

697

Linux Networking From Zero — The 4 Things You Need to Understand Kubernetes Networking

Before Kubernetes. Before containers. Just Linux.

If you understand these 4 things, Kubernetes networking stops being magic and becomes obvious. If you don't, no amount of Kubernetes articles will help.


1. The Network Interface

A network interface is how a machine sends and receives data on a network. Think of it as a socket in the wall — the physical plug point between your machine and the outside world.

On a Linux machine:

ip a
# 1: lo: <LOOPBACK>
#    inet 127.0.0.1/8
# 2: eth0: <BROADCAST,MULTICAST,UP>
#    inet 192.168.1.10/24

Two interfaces here:

eth0 — the real network interface. Has IP 192.168.1.10. This is how this machine talks to other machines. Data going out leaves through eth0. Data coming in arrives through eth0.

lo — the loopback interface. Has IP 127.0.0.1. This is special — it never leaves the machine. It's a self-addressed envelope. When you curl localhost, the packet goes into lo and comes straight back out to the same machine. No network cable involved. Nothing leaves.

This is critical. 127.0.0.1 and localhost are not "the machine" in an abstract sense. They are specifically the loopback interface lo. Traffic sent to 127.0.0.1 goes to lo and stays on that machine. It cannot reach any other machine. It cannot be seen by any other machine.


2. The Routing Table

When your machine wants to send a packet, it needs to know where to send it. The routing table is the map it uses to make that decision.

ip route
# default via 192.168.1.1 dev eth0
# 192.168.1.0/24 dev eth0 proto kernel scope link

Two rules here:

192.168.1.0/24 dev eth0 — any packet going to an IP in the range 192.168.1.0 to 192.168.1.255 — send it out through eth0 directly. These machines are on the same network. No middleman needed.

default via 192.168.1.1 dev eth0 — any packet going anywhere else — send it to 192.168.1.1 (the router/gateway) through eth0. The router knows where to forward it from there.

The machine checks the routing table top to bottom, picks the first rule that matches the destination IP, and sends the packet out through the specified interface.

If no rule matches and there's no default — the packet is dropped. The machine has no idea where to send it.

The key point: the routing table is per-machine. Every machine has its own. Every container has its own. This is why networking breaks when routing tables are wrong or missing.


3. The Network Namespace

Here is where containers start making sense.

A network namespace is a completely isolated copy of the entire Linux networking stack. Not a different machine — the same kernel — but a completely separate set of:

  • network interfaces
  • routing tables
  • iptables rules
  • port bindings

When you create a new network namespace, it starts with nothing. No interfaces except a DOWN loopback. No routes. Empty iptables. No way to reach anything.

# create a new network namespace called "myns"
ip netns add myns

# run a command inside it
ip netns exec myns ip a
# 1: lo: <LOOPBACK>  ← only loopback, and it's DOWN
#    inet 127.0.0.1/8

ip netns exec myns ip route
# (empty — no routes at all)

From inside myns, you cannot reach the internet. You cannot reach the host. You cannot reach anything. It is completely empty.

From the host, myns doesn't exist on the network at all. The host's eth0 has no idea myns is there.

This is what a container is. When Docker or containerd creates a container, it creates a new network namespace. The container's process runs inside that namespace. It gets its own interfaces, its own routing table, its own 127.0.0.1. The host's network is completely invisible to it.

This is why localhost inside a container is the container's own loopback — not the host's. The container is in a different network namespace. It has its own lo. The host's lo is in a different namespace entirely.


4. The veth Pair

A network namespace starts isolated. To make it useful, you need to connect it to something. That connection is a veth pair.

A veth pair is two virtual network interfaces linked together like a pipe. Whatever you send into one end comes out the other end. They always come in pairs — you cannot have just one.

# create a veth pair: veth-host and veth-container
ip link add veth-host type veth peer name veth-container

# currently both ends are on the host
ip a | grep veth
# veth-host
# veth-container

# move veth-container into the namespace
ip link set veth-container netns myns

# now:
# veth-host      → on the host
# veth-container → inside myns namespace

# configure the host end
ip addr add 10.0.0.1/24 dev veth-host
ip link set veth-host up

# configure the container end
ip netns exec myns ip addr add 10.0.0.2/24 dev veth-container
ip netns exec myns ip link set veth-container up
ip netns exec myns ip link set lo up

# test
ip netns exec myns ping 10.0.0.1   # namespace pings host end ✓
ping 10.0.0.2                       # host pings namespace ✓

The namespace now has connectivity — but only to the host end of the veth pair. Not the internet. Not other namespaces. Just the one wire you gave it.

This is exactly what Docker does for every container. One veth pair per container. One end on the host, one end inside the container's network namespace. The container calls its end eth0.


How These 4 Things Connect

Start from nothing and build up:

Step 1 — bare machine:

eth0: 192.168.1.10   ← real interface, talks to the world
lo:   127.0.0.1      ← loopback, stays on this machine
routing table tells packets which interface to use

Step 2 — create a network namespace:

host namespace         new namespace (myns)
  eth0: 192.168.1.10     lo: 127.0.0.1 (DOWN)
  lo:   127.0.0.1        (nothing else)

myns is completely isolated — no way in or out

Step 3 — add a veth pair:

host namespace              myns namespace
  eth0: 192.168.1.10          eth0: 10.0.0.2  ← veth-container renamed
  lo:   127.0.0.1             lo:   127.0.0.1
  veth-host: 10.0.0.1 ←──→ veth-container: 10.0.0.2

myns can now talk to the host via the veth pair
myns still cannot reach the internet

Step 4 — add routing + NAT for internet access:

host enables IP forwarding
host adds NAT rule: traffic from 10.0.0.0/24 → masquerade as 192.168.1.10

myns adds default route: all traffic → via 10.0.0.1 (the host end)

now myns can reach the internet through the host

This is a Docker container with bridge networking. Every container is a network namespace connected to the host via a veth pair, with the host doing NAT to give it internet access.


The 5 Things to Remember

A network interface is how a machine connects to a network. eth0 is real. lo is loopback — never leaves the machine.

The routing table decides where each packet goes based on the destination IP. No route = packet dropped.

A network namespace is a completely isolated networking stack. Its own interfaces, routes, iptables, and its own 127.0.0.1. What happens in a namespace stays in that namespace.

A veth pair is the wire connecting two namespaces. Always two ends. Move one end into a namespace, the other stays on the host.

localhost inside a container is the container's own loopback in its own namespace. It is not the host's loopback. They share a kernel but not a network namespace.

697

CKA Road Trip: Why Would I Schedule a Pod on the Control Plane?

Short answer: you usually wouldn't. But here's why the option exists.


The Default Taint

The control plane node has a taint on it by default:

node-role.kubernetes.io/control-plane:NoSchedule

That means: don't schedule pods here unless they explicitly tolerate it. Without a toleration, the scheduler sees the taint and skips the node silently. No error. Just one less node available.


The Three Legitimate Reasons

Node-level agents that need to run everywhere. A security scanner, log collector, or monitoring agent that must observe every node — including the control plane. If it doesn't run there, you have a blind spot. That's the real production use case for the toleration.

Single or minimal node clusters. In a lab with only a controlplane and one worker, not tolerating the taint means half your cluster is off-limits for scheduling. Fine in prod, painful in a two-node lab.

The exam. KillerKoda and similar environments use minimal setups. You'll hit exercises where pods need to land on the control plane just to satisfy the task.


The Toleration

tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

Add this under spec in your pod or DaemonSet template. Without it the control plane node is invisible to the scheduler.


Why the Taint Exists

The control plane runs etcd, the API server, the scheduler, the controller manager. It's the most critical node in the cluster. Resource contention here means the whole cluster degrades. The taint enforces separation by default — workloads go on workers, infrastructure stays on the control plane.

The control plane components themselves are static pods. The kubelet places them directly, bypassing the scheduler entirely, so the taint doesn't affect them.


The Rule

In production with real worker nodes: leave the taint alone, never schedule workloads on the control plane.

In a lab or for genuine node-level agents: add the toleration and be deliberate about it.

CKA Road Trip: What Is a DaemonSet


What It Is

A DaemonSet ensures exactly one copy of a pod runs on every node in the cluster. New node joins → pod automatically created on it. Node removed → pod goes with it.


Real Uses

  • Log collectors (Fluentd, Filebeat)
  • Monitoring agents (Prometheus node-exporter)
  • Network plugins (Cilium, Flannel)
  • kube-proxy itself is a DaemonSet

Anything that needs to run on every node, once per node.


vs Deployment

Deployment says: run N copies, put them wherever the scheduler decides.

DaemonSet says: run exactly one copy on every node, no exceptions.


vs Static Pod

DaemonSet Static Pod
Managed by controller manager kubelet (file on disk)
Defined in etcd via API server /etc/kubernetes/manifests/
kubectl works yes read-only mirror only
Survives control plane outage no yes
Use case node-level agents control plane bootstrap

DaemonSet is a proper Kubernetes resource — updatable, rollbackable, kubectl works on it normally. Tradeoff: if the control plane goes down, the DaemonSet controller can't manage it.

Static pod has zero dependency on the control plane. The kubelet manages it from a file on disk directly.


The Decision Rule

Do you need this to survive a control plane outage?

  • No → DaemonSet
  • Yes → Static pod

In practice, almost nothing needs to survive a control plane outage except the control plane components themselves — which is exactly why they're static pods.

697

CKA Road Trip: K8s Components — What Each One Does

K8s is not a monolith. It's separate processes, each owning one job, talking to each other over HTTP.


The Components

etcd — the database. Stores every object in the cluster as key-value pairs. Every other component is stateless — they read/write etcd and that's where reality lives. If etcd dies, the cluster loses its mind.

kube-apiserver — the only door into etcd. Nobody talks to etcd directly except the API server. Everything — kubectl, kubelet, controller manager, scheduler — talks through here. It handles auth, validation, then reads/writes etcd.

kube-controller-manager — the reconciliation engine. Watches the API server in a loop: desired state vs actual state, gap found → fix it. ReplicaSet wants 3 pods, 1 exists → create 2 more. Does not actually run containers.

kube-scheduler — decides which node a pod runs on. Sees an unassigned pod, picks a node based on resources/taints/affinity, writes that assignment to the API server. That's it. Doesn't create the pod either.

kubelet — the agent on every node. Watches the API server for pods assigned to its node. When it sees one, it tells the container runtime to run it. The only component that touches real Linux processes. Also manages static pods from /etc/kubernetes/manifests/ with zero dependency on the API server.

kubectl — not a cluster component. A CLI on your machine that sends HTTP requests to the API server. k get pods = GET request to the API server. Nothing more.


The Flow: kubectl create deployment

kubectl → API server → etcd (deployment stored)
    controller manager sees new deployment
    creates ReplicaSet → creates pod objects (no node assigned yet)
    scheduler sees unassigned pods
    picks a node → writes assignment to API server → etcd
    kubelet on that node sees pod assigned to it
    tells containerd → container starts running

Every arrow is an HTTP call to the API server. Nobody talks to anyone else directly.


The One Thing That Makes It Click

API server + etcd = single source of truth. Every other component watches that source and reacts to it. They're all independently running processes that agree on one shared database. Restart the controller manager — no state lost, because state lives in etcd, not in the process.

697

CKA Road Trip: Node NotReady + etcd Backup

Two tasks, one exercise.


Part 1 — Node NotReady

k get nodes
# controlplane   NotReady
k describe node controlplane
# Conditions:
#   Ready   Unknown   NodeStatusUnknown   Kubelet stopped posting node status.

The condition message is the signal. Kubelet stopped posting node status means one thing — the kubelet process is dead.

ssh controlplane
systemctl status kubelet
# Active: inactive (dead)

systemctl start kubelet
systemctl status kubelet
# Active: active (running)

exit
k get nodes
# controlplane   Ready

The kubelet was stopped. Start it, node recovers.


Part 2 — etcd Backup

Verify etcd is running first:

k get pods -n kube-system | grep etcd
# etcd-controlplane   1/1   Running

Take the snapshot:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  snapshot save /opt/cluster_backup.db > backup.txt 2>&1

The three certs are always required — etcd won't talk without mTLS. Find them at:

/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-etcd-client.key

> backup.txt 2>&1 redirects both stdout and stderr to the file. Without the > before backup.txt etcdctl sees it as a second argument and throws snapshot save expects one argument.


The Diagnostic Chain

node NotReady
k describe node → "Kubelet stopped posting node status"
ssh into node
systemctl status kubelet → inactive
systemctl start kubelet
node Ready

Kubelet stopped posting node status is unambiguous. Go straight to the kubelet, don't waste time elsewhere.

CKA Road Trip: SSH Into a Node — Troubleshooting Commands

Once a node is NotReady, kubectl becomes limited. You SSH in and it's just Linux from there.


The Commands, In Order

# is the kubelet alive?
systemctl status kubelet

# is the container runtime alive?
systemctl status containerd

# kubelet logs — what's it complaining about?
journalctl -u kubelet -n 50 --no-pager

# disk space — is the node full?
df -h

# memory — is it under pressure?
free -m

# are containers actually running at the OS level?
crictl ps

# what does the kubelet config look like?
cat /var/lib/kubelet/config.yaml

# static pod manifests — anything broken here?
ls /etc/kubernetes/manifests/

What Each One Tells You

systemctl status kubelet — is the kubelet process running or dead. First thing to check, every time.

systemctl status containerd — is the container runtime up. If containerd is dead, no containers can start even if kubelet is fine.

journalctl -u kubelet -n 50 --no-pager — the last 50 kubelet log lines. This is where the actual error is. Typo in a binary name, missing config file, cert error — it'll be here.

df -h — disk pressure. A full disk kills the kubelet. Nodes with full disks go NotReady silently from kubectl's perspective.

free -m — memory pressure. Same idea — resource exhaustion shows as NotReady.

crictl ps — shows containers running at the containerd level, bypassing Kubernetes entirely. Useful when kubectl shows nothing but containers might still be running. Think of it as docker ps for the CRI layer.

cat /var/lib/kubelet/config.yaml — the kubelet's own config file. If it's malformed or missing, the kubelet won't start.

ls /etc/kubernetes/manifests/ — static pod manifests for control plane components. A broken yaml here means apiserver, etcd, scheduler, or controller-manager won't start.


The Pattern

The first two commands tell you if the key processes are up. journalctl tells you why if they're not. df and free rule out resource pressure. Everything else is digging deeper once you know the direction.

Fixing comes after. Troubleshoot first, understand what's broken, then fix it.

CKA Road Trip: ConfigMap vs Secret


The Spec

spec:
  containers:
  - env:
    - name: POSTGRES_DB
      valueFrom:
        configMapKeyRef:
          key: POSTGRES_DB
          name: postgres-db-config
    - name: POSTGRES_USER
      valueFrom:
        configMapKeyRef:
          key: POSTGRES_USER
          name: postgres-db-config
    - name: POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          key: POSTGRES_PASSWORD
          name: postgres-db-secret
    image: postgres:latest
    name: postgres-container
    ports:
    - containerPort: 5432

Is configMapKeyRef where the config is coming from? A file?

No. The config is not coming from a file.

configMapKeyRef means the value is pulled from a ConfigMap object that already exists in the cluster. Someone created it beforehand with kubectl create configmap or applied a YAML manifest for it.

When the pod starts, Kubernetes goes:

find ConfigMap named "postgres-db-config" in this namespace
  → get the value at key "POSTGRES_DB"
  → inject it as env var POSTGRES_DB into the container

Same for the Secret — secretKeyRef does the same thing but pulls from a Secret object instead.

Neither of these exist in the pod spec itself. They're separate objects that must already exist in the cluster before the pod starts. If they don't, the pod won't start — CreateContainerConfigError.

k get configmap postgres-db-config -o yaml
k get secret postgres-db-secret -o yaml

So — take the KEY POSTGRES_DB from the ConfigMap postgres-db-config and create an env var named POSTGRES_DB with that value?

Exactly.

name: postgres-db-config   ← which ConfigMap to look in
key:  POSTGRES_DB          ← which key inside that ConfigMap
name: POSTGRES_DB          ← what to call the env var in the container

Go to ConfigMap postgres-db-config, grab the value stored under key POSTGRES_DB, expose it inside the container as environment variable POSTGRES_DB.

If the ConfigMap looks like this:

data:
  POSTGRES_DB: mydb
  POSTGRES_USER: admin

Then inside the container: echo $POSTGRES_DBmydb.


Why use a ConfigMap?

Because you don't want to hardcode config values inside the image or the pod spec.

Without ConfigMap:

env:
  - name: POSTGRES_DB
    value: mydb

That value is baked into the manifest. Every environment — dev, staging, prod — needs a different manifest, or you're running prod with dev values.

With a ConfigMap, the same pod spec works everywhere. The ConfigMap changes per environment. The person managing app config doesn't need to touch the deployment manifest.

Why not a Secret for everything then? ConfigMaps are for non-sensitive config — database name, port, feature flags. Secrets are for passwords, tokens, keys. That's the split in this spec — DB name and user in ConfigMap, password in Secret.


But Secrets are only encoded, not encrypted

Correct. Base64 encoding, not encryption. Anyone who can read the Secret object can decode it in seconds:

echo "cGFzc3dvcmQ=" | base64 -d
# password

Secrets are only as safe as your cluster access controls. If someone can run kubectl get secret postgres-db-secret -o yaml they have the password.

The real protection comes from RBAC — restricting who can read Secret objects, etcd encryption at rest — encrypting the etcd database itself (not on by default), and external secret managers — Vault, AWS Secrets Manager — where the actual value never lives in etcd at all.


So why use Secrets over ConfigMap if they're not even encrypted?

Fair point. The difference isn't security — it's intent and tooling.

ConfigMap — plaintext, visible everywhere, no special handling. For stuff that's genuinely not sensitive: DB name, port number, feature flags.

Secret — base64, but the cluster treats it differently:

  • Not printed in logs by default
  • Not shown in kubectl describe pod output
  • Can be encrypted at rest if you configure it
  • RBAC policies can restrict Secret access separately from ConfigMap access
  • External secret managers (Vault etc.) integrate with the Secret API, not ConfigMap

It's not that Secrets are secure out of the box. It's that they're the hook point for security. You build protection around them. ConfigMaps have no such hook — they're never meant to hold sensitive data so no tooling exists to protect them.

Put a password in a ConfigMap and it'll show up in kubectl describe pod, in logs, in dashboards, everywhere. Put it in a Secret and at least the ecosystem knows to handle it carefully.

697