infrapuzzle/bootstrap/README.md

Bootstrap

This directory contains the automation, configurations, and playbooks used to provision and upgrade the single-node Kubernetes cluster haumdaucher (136.243.23.215) using Kubespray.

---

Architecture & Single-Node Cluster Constraints

haumdaucher is a single-node cluster (control plane, etcd, and worker all run on the same machine). This topology requires specific configuration choices during cluster maintenance:

Important

Crucial Single-Node Invariants

1. Never Cordon or Drain the Node: Draining a single-node cluster evicts all workloads with nowhere to schedule them, causing pods to stay in Pending state and breaking critical cluster services (such as DNS, CNI, ingress, and local storage). In inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml, the following settings are enforced:

   drain_nodes: false
   upgrade_node_always_cordon: false
   upgrade_node_fail_if_drain_fails: false
   

2. In-Place Upgrades via cluster.yml: Instead of upgrade-cluster.yml (which defaults to draining nodes), single-node upgrades should use the in-place upgrade mechanism:

   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
   

3. Version Number Formatting: Kubespray 2.28 interpolates v{{ kube_version }} across many internal tasks. Therefore, kube_version in k8s-cluster.yml MUST NOT include a leading v:

   kube_version: 1.32.13   # Correct
   # kube_version: v1.32.13 # INCORRECT (leads to v1.32.13 -> vv1.32.13 errors)
   

4. Host OS Lifecycle Constraint (Ubuntu 20.04 LTS): Kubespray release-2.28 is the final release line supporting Ubuntu 20.04. Kubespray release-2.29+ drops Ubuntu 20.04 support and requires Ubuntu 22.04 or 24.04. Do not bump VERSION in init.sh beyond release-2.28 until the host OS is upgraded.

---

Quickstart

1. Environment Initialization

Source init.sh from this directory. It clones Kubespray, applies patches from patches/, configures the Python virtualenv, and syncs ./inventory/ into ./kubespray/inventory/:

source ./init.sh

(Note: init.sh also exports the sync-inventory alias to quickly sync local ./inventory/ modifications into ./kubespray/inventory/ without re-cloning).

---

Step-by-Step Upgrade Runbook

Upgrades are conducted in two decoupled phases:

1. Phase 1: Upgrade the Kubespray tooling baseline while keeping the Kubernetes version unchanged \rightarrow Apply \rightarrow Verify.
2. Phase 2: Upgrade Kubernetes components (API server, controller manager, scheduler, kubelet, kube-proxy) \rightarrow Apply \rightarrow Verify.

Step 0: Pre-Flight Checks & Backups

1. Verify SSH and sudo access:

   ssh moritz@136.243.23.215 "sudo kubectl get nodes -o wide"
   

2. Check Pod Health:

   ssh moritz@136.243.23.215 "sudo kubectl get pods -A | grep -v -E 'Running|Completed'"
   

3. Take a Manual etcd Snapshot:

   ssh moritz@136.243.23.215 "sudo etcdctl \
     --cacert=/etc/ssl/etcd/ssl/ca.pem \
     --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem \
     --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem \
     snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db"
   

---

Step 1: Reconcile Kubespray Baseline (Phase 1)

1. Verify VERSION in init.sh matches the target Kubespray branch (e.g. release-2.28).
2. Run source ./init.sh to initialize the environment.
3. Apply the baseline configuration:

   cd kubespray
   ansible-playbook -i inventory/prod/inventory.ini cluster.yml
   

4. Verify cluster health:

   kubectl get nodes -o wide
   kubectl get pods -n kube-system
   

---

Step 2: Upgrade Kubernetes Version (Phase 2)

1. In bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml, update kube_version:

   kube_version: 1.32.13
   

2. Sync the configuration into the kubespray/ workspace:

   sync-inventory
   

3. Run the in-place upgrade playbook:

   cd kubespray
   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
   

---

Step 3: Post-Upgrade Verification

1. Check component versions:

   kubectl version
   

   Ensure both Client Version and Server Version report v1.32.13.

2. Verify Node status:

   kubectl get nodes -o wide
   

   Ensure status is Ready and version matches v1.32.13.

3. Verify System Pods:

   kubectl get pods -n kube-system
   

   All control plane components (kube-apiserver, kube-controller-manager, kube-scheduler, coredns, kube-flannel, kube-proxy, traefik) should be 1/1 Running.

4. Verify Workloads:

   kubectl get pods -A | grep -v -E "Running|Completed"
   

---

Troubleshooting & Edge Cases

Kubeadm Preflight CreateJob Failure

Kubeadm 1.31+ preflight executes an ephemeral upgrade-health-check-... batch job with a 15-second timeout. On single-node clusters or under load, this may exceed 15s. To prevent the upgrade from halting:

• CreateJob is included in kubeadm_ignore_preflight_errors in k8s-cluster.yml.
• The patch patches/kubeadm-upgrade-ignore-preflight.patch ensures Kubespray's v1beta4 kubeadm configuration propagates this flag during upgrade apply.

Kubelet Housekeeping Stalls / Static Pod Timeout

If kubeadm upgrade apply times out waiting for static pod hash changes:

• Check journalctl -u kubelet -n 100 for Housekeeping took longer than expected or orphaned pod ... failed to rmdir().
• If an old container unmount left a stale bind-mount busy, lazily detach it:

  sudo umount -l /var/lib/kubelet/pods/<orphaned-pod-uid>/volumes/...
  sudo systemctl restart kubelet
  

---

Directory Layout

• init.sh: Environment bootstrapping script and upgrade instructions.
• inventory/: Source-of-truth inventory and variable files.
• patches/: Git patches applied to upstream Kubespray on checkout.
• kubespray/: (Ephemeral) Cloned Kubespray repository.