Kubernetes Cluster Upgrade — Full Reference¶

Reference: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

The Big Picture First¶

A Kubernetes cluster upgrade isn't one thing — it's upgrading three separate packages that happen to work together:

Package	What it is
`kubeadm`	The tool that bootstraps and upgrades cluster components (API server, scheduler, controller-manager, etc.)
`kubelet`	The agent that runs on every node. Manages containers. Upgrading this restarts it, which restarts all containers on that node.
`kubectl`	The CLI client. Not strictly required to upgrade at the same time, but should match the cluster version.

They are separate apt packages. Installing one does not touch the others. You must do them in order: 1. Upgrade kubeadm first 2. Use kubeadm to upgrade the control plane components 3. Upgrade kubelet and kubectl 4. Restart kubelet

And you do that separately on every node — controlplane first, then each worker.

The One-Minor-Version Rule¶

You can only upgrade one minor version at a time. v1.29 → v1.30 → v1.31. Never v1.29 → v1.31 directly.

This exists because the API server guarantees it can talk to kubelets within one minor version of each other. Skip two versions and you break that guarantee — nodes may fail, pods may not schedule, weird breakage.

Patch versions (v1.29.3 → v1.29.4) are fine to jump between freely.

Exam interpretation: "Upgrade to the latest version" means the next available version up — not the absolute latest. If you're on v1.34.3 and v1.34.4 and v1.34.5 are available, install v1.34.4. Using apt upgrade without pinning gives you v1.34.5 and is wrong.

apt-mark — Package Pinning¶

Why it exists¶

You never want an automated apt upgrade or unattended-upgrades job to silently bump your kubelet version at 3am. A kubelet upgrade restarts kubelet, which takes down every container on that node. So all Kubernetes packages are held — pinned to their current version.

apt-mark hold kubeadm kubelet kubectl    # pin — excluded from all upgrades
apt-mark unhold kubeadm                  # unpin — allow version change
apt-mark showhold                        # list what's currently pinned

The pattern you always follow¶

Every package upgrade looks like this, no exceptions:

apt-mark unhold <package>                     # 1. remove pin
apt-get update                                # 2. refresh package index
apt-get install -y <package>='<version>'      # 3. install exact version
apt-mark hold <package>                       # 4. re-pin

Why re-pin? If you forget step 4, the next apt upgrade picks it up and installs the latest. That's exactly what you were protecting against.

Why apt-get update every time? apt caches the package index locally. If a new version was published since the last update, apt doesn't know it exists yet. Without apt-get update, apt-cache madison won't show the new version.

Finding available versions — apt-cache madison¶

apt-get update
apt-cache madison kubeadm

Output looks like:

   kubeadm | 1.34.5-1.1 | https://pkgs.k8s.io/...
   kubeadm | 1.34.4-1.1 | https://pkgs.k8s.io/...
   kubeadm | 1.34.3-1.1 | https://pkgs.k8s.io/...

madison = shows all available versions from all configured repos, newest first.

The version format: what is `-1.1`?¶

1.34.4-1.1 = Kubernetes version 1.34.4, package revision 1 for this distro. It's the Debian packaging build number, not part of Kubernetes itself. Always include it when installing:

apt-get install -y kubeadm='1.34.4-1.1'    # correct
apt-get install -y kubeadm=1.34.4           # may fail or pick wrong package

Chained command with &&¶

apt-mark unhold kubeadm && \
apt-get update && \
apt-get install -y kubeadm='1.34.4-1.1' && \
apt-mark hold kubeadm

&& = only continue if the previous command succeeded. If apt-get update fails (network down, repo broken), nothing installs. You want the chain to stop on failure, not silently install with a stale index.

Compare with ; which runs the next command regardless — that's wrong here.

kubectl drain¶

kubectl drain prepares a node for maintenance. It does two things:

Marks the node as unschedulable (same as kubectl cordon) — no new pods will be scheduled to it
Evicts all running pods from the node — they get rescheduled on other nodes

kubectl drain <node-name> --ignore-daemonsets

Why --ignore-daemonsets is always required¶

DaemonSet pods are supposed to run on every node by design — that's what a DaemonSet is. If you drain without --ignore-daemonsets, kubectl will refuse and print an error because it can't evict DaemonSet pods (they'd just come back immediately). This flag tells drain to skip them rather than fail.

What drain does not do¶

It doesn't delete DaemonSet pods. They stay running during maintenance. Kube-proxy, Calico/Flannel network agents, log collectors — all DaemonSet pods — keep running on the node throughout the upgrade.

What gets evicted¶

Everything else — Deployments, ReplicaSets, StatefulSets, standalone Pods. Their pods are deleted from this node and rescheduled elsewhere (unless there's nowhere else to schedule them — more on that below).

Drain can fail if:¶

Pods with no controller (bare pods not managed by a Deployment/etc) — drain refuses because these can't be rescheduled. Fix: add --force to delete them anyway (data loss risk if they're not ephemeral)
PodDisruptionBudgets blocking eviction — a PDB can say "never have fewer than 2 replicas running." If draining would violate that, drain waits or fails. Fix: --disable-eviction to force past PDB (last resort)
Only one node in the cluster — pods have nowhere to go. You're stuck. This is a test environment limitation.

kubectl drain node01 --ignore-daemonsets             # standard
kubectl drain node01 --ignore-daemonsets --force     # force delete bare pods too

kubectl cordon / uncordon¶

cordon¶

kubectl cordon <node-name>

Marks the node SchedulingDisabled. The scheduler will not place new pods here. Existing pods keep running — nothing is evicted. Think of it as "closed for new business."

kubectl drain calls cordon internally as its first step — you don't need to cordon separately before draining.

uncordon¶

kubectl uncordon <node-name>

Removes the SchedulingDisabled taint. The node is now schedulable again. Existing pods that got evicted during drain don't automatically move back — they stay where they landed. But new pods can now land here again.

Critical: Always uncordon after the upgrade is done. If you forget, the node stays unschedulable permanently and you wonder why workloads aren't spreading across nodes.

Check node status¶

kubectl get nodes

A drained/cordoned node shows Ready,SchedulingDisabled in the STATUS column. After uncordon it shows Ready.

Controlplane Upgrade — Full Sequence¶

On the controlplane node:

# Step 1 — Find the next version
kubectl get nodes                                    # see current version
apt-get update && apt-cache madison kubeadm         # list available versions

# Step 2 — Upgrade kubeadm
apt-mark unhold kubeadm && \
apt-get update && \
apt-get install -y kubeadm='1.34.4-1.1' && \
apt-mark hold kubeadm

# Step 3 — Verify kubeadm
kubeadm version                                     # confirm new version installed

# Step 4 — Check upgrade plan
kubeadm upgrade plan                                # dry run — shows what will be upgraded and to what

# Step 5 — Apply the upgrade
kubeadm upgrade apply v1.34.4                      # upgrades: apiserver, scheduler, controller-manager, kube-proxy, CoreDNS, etcd

# Step 6 — Drain controlplane (evict pods, mark unschedulable)
kubectl drain controlplane --ignore-daemonsets

# Step 7 — Upgrade kubelet and kubectl
apt-mark unhold kubelet kubectl && \
apt-get update && \
apt-get install -y kubelet='1.34.4-1.1' kubectl='1.34.4-1.1' && \
apt-mark hold kubelet kubectl

# Step 8 — Reload and restart kubelet
systemctl daemon-reload
systemctl restart kubelet

# Step 9 — Uncordon controlplane
kubectl uncordon controlplane

# Step 10 — Verify
kubectl get nodes                                   # controlplane should show new version and Ready

kubeadm upgrade plan¶

Shows you: - Current component versions - What version they'd be upgraded to - Whether any manual steps are required - Certificate expiry warnings

Always run this before apply. It's a read-only check — nothing gets changed.

kubeadm upgrade apply v1.34.4¶

This is the command that actually upgrades the control plane components. It: - Pulls the new container images for apiserver, scheduler, controller-manager - Updates their static pod manifests (in /etc/kubernetes/manifests/) - Upgrades kube-proxy DaemonSet - Upgrades CoreDNS - Optionally upgrades etcd (if not external) - Updates kubeadm's internal config

This only upgrades the control plane components running as pods. It does not touch kubelet. Kubelet is a systemd service on the host — it gets upgraded separately via apt.

systemctl daemon-reload && systemctl restart kubelet¶

systemctl daemon-reload       # reload systemd's view of service unit files
systemctl restart kubelet     # restart the kubelet service

Why daemon-reload? When kubelet is upgraded via apt, the new package may ship an updated service unit file (/lib/systemd/system/kubelet.service.d/...). systemd doesn't automatically pick up changes to unit files — daemon-reload tells it to re-read them. Without this, systemctl restart kubelet starts the new binary but with the old service configuration.

Why restart? kubelet is a long-running daemon. Installing a new binary doesn't restart it — the old binary keeps running until you tell systemd to restart the service.

Worker Node Upgrade — Full Sequence¶

Workers use kubeadm upgrade node (not apply). The distinction matters.

From the controlplane first:

kubectl drain node01 --ignore-daemonsets     # evict pods from node01, mark unschedulable

Then SSH into the worker:

ssh node01

# Step 1 — Upgrade kubeadm
apt-mark unhold kubeadm && \
apt-get update && \
apt-get install -y kubeadm='1.34.4-1.1' && \
apt-mark hold kubeadm

# Step 2 — Upgrade the node config
kubeadm upgrade node                          # NOT apply — this is the worker-specific command

# Step 3 — Upgrade kubelet and kubectl
apt-mark unhold kubelet kubectl && \
apt-get update && \
apt-get install -y kubelet='1.34.4-1.1' kubectl='1.34.4-1.1' && \
apt-mark hold kubelet kubectl

# Step 4 — Reload and restart kubelet
systemctl daemon-reload
systemctl restart kubelet

exit

Back on controlplane:

kubectl uncordon node01
kubectl get nodes                             # verify node01 shows new version and Ready

kubeadm upgrade apply vs kubeadm upgrade node¶

Command	Used on	What it does
`kubeadm upgrade apply v1.X.X`	Controlplane only	Upgrades API server, scheduler, controller-manager, CoreDNS, kube-proxy, etcd
`kubeadm upgrade node`	Worker nodes only	Pulls updated kubelet config from the API server, updates the local node config

Workers don't run control plane components, so they don't need apply. They use upgrade node to sync their kubelet configuration with what the upgraded control plane expects.

If you run kubeadm upgrade apply on a worker node, it will fail — the worker doesn't have the control plane config it needs to run that command.

Common Mistakes¶

Mistake	What goes wrong
`apt upgrade` without pinning version	Installs the absolute latest, not the next version — jumps multiple minor versions
Forgetting `apt-get update` before install	apt doesn't know new versions exist → "package not found"
`kubeadm upgrade node` on controlplane	Wrong command — use `upgrade apply` on controlplane
`kubeadm upgrade apply` on worker	Fails — worker doesn't have control plane config
Forgetting to drain before kubelet upgrade	Pods on that node get disrupted mid-upgrade
Forgetting to uncordon	Node stays SchedulingDisabled permanently
Forgetting `systemctl daemon-reload`	New kubelet binary starts with old service unit config
Upgrading kubelet before `kubeadm upgrade apply`	API server and kubelet version mismatch during the window
Skipping `kubeadm version` check	Silent install failure means you run upgrade apply with the old binary

Full Exam Cheatsheet¶

# ── CHECK CURRENT CLUSTER VERSION ─────────────────────────────────
kubectl get nodes

# ── FIND NEXT VERSION ─────────────────────────────────────────────
apt-get update && apt-cache madison kubeadm

# ── CONTROLPLANE ──────────────────────────────────────────────────
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm='1.34.4-1.1' && apt-mark hold kubeadm
kubeadm version
kubeadm upgrade plan
kubeadm upgrade apply v1.34.4
kubectl drain controlplane --ignore-daemonsets
apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet='1.34.4-1.1' kubectl='1.34.4-1.1' && apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon controlplane
kubectl get nodes

# ── WORKER (drain from controlplane first) ────────────────────────
kubectl drain node01 --ignore-daemonsets
ssh node01
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm='1.34.4-1.1' && apt-mark hold kubeadm
kubeadm upgrade node
apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet='1.34.4-1.1' kubectl='1.34.4-1.1' && apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
exit
kubectl uncordon node01
kubectl get nodes