chore(bootstrap): upgrade kubespray to release-2.28 and kubernetes to v1.32.13

- Bump Kubespray baseline to release-2.28 (last release supporting Ubuntu 20.04)
- Upgrade Kubernetes control plane and node to v1.32.13
- Configure single-node cluster settings (drain_nodes: false)
- Add patch for Kubespray v1beta4 kubeadm preflight error handling
- Update init.sh, AGENTS.md, and README.md with two-phase upgrade instructions

bootstrap/.gitignore

@@ -1,2 +1,4 @@
 kubespray
 venv
+credentials/
+inventory/**/credentials/

bootstrap/AGENTS.md

@@ -15,7 +15,7 @@ The core workflow is encapsulated in `init.sh`.
 **Purpose**: Prepares the environment and Kubespray for deployment.
 **Actions**:
 1.  **Clean Slate**: Deletes existing `kubespray/` directory.
-2.  **Clone**: Clones Kubespray (version defined in variable `VERSION`, e.g., `release-2.27`).
+2.  **Clone**: Clones Kubespray (version defined in variable `VERSION`, e.g., `release-2.28`).
 3.  **Environment**: Sets up Python virtualenv via `pyenv` and installs `requirements.txt`.
 4.  **Sync**: Copies local `./inventory/` configurations into `./kubespray/inventory/`.
 
@@ -26,12 +26,13 @@ The core workflow is encapsulated in `init.sh`.
     ```
 2.  **Deploy / Upgrade**:
     After sourcing, go to the `kubespray` directory and run the Ansible playbooks as instructed by the script output.
-    *   **Standard Run**:
+    *   **Phase 1: Reconcile Kubespray baseline (stay on current k8s version)**:
         ```bash
         cd kubespray
         ansible-playbook -i inventory/prod/inventory.ini cluster.yml
         ```
-    *   **Forced Upgrade**:
+    *   **Phase 2: Upgrade Kubernetes version (e.g. to v1.32.13)**:
+        Update `kube_version` in `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, re-sync, and run:
         ```bash
         cd kubespray
         ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml

bootstrap/README.md

@@ -103,12 +103,60 @@ This runs everything and is kind of idempotent:
 ansible-playbook -i inventory/prod/inventory.ini cluster.yml
 ```
 
-## Upgrade to 2.31.3
+## Upgrade to Kubernetes 1.31 (Previous)
 
 Required to execute: 
 
-```
+```sh
 ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e drain_nodes=false upgrade-cluster.yml
 ```
 
 + set a feature flag: https://github.com/kubernetes-sigs/kubespray/issues/11887
+
+## Upgrade to Kubespray release-2.28 & Kubernetes v1.32.13
+
+> [!NOTE]
+> Upgrade philosophy: decoupled two-phase upgrade.
+> 1. Upgrade Kubespray tooling baseline (keeping k8s version unchanged).
+> 2. Upgrade Kubernetes control plane and worker components.
+>
+> **Important**:
+> - `haumdaucher` is a **single-node cluster**. Always use `-e drain_nodes=false` when running `upgrade-cluster.yml` to prevent the upgrade from hanging on draining.
+> - **Ubuntu 20.04 LTS constraint**: Kubespray `release-2.28` is the **last** release officially supporting Ubuntu 20.04. Upstream Kubespray `release-2.29`+ requires Ubuntu 22.04 or 24.04.
+
+### Phase 1: Kubespray Baseline Upgrade (Retaining k8s v1.31.9)
+1. Ensure `init.sh` has `VERSION="release-2.28"`.
+2. Initialize environment:
+   ```sh
+   . ./init.sh
+   ```
+3. Run cluster playbook to reconcile Kubespray baseline:
+   ```sh
+   ansible-playbook -i inventory/prod/inventory.ini cluster.yml
+   ```
+4. Verify cluster health:
+   ```sh
+   kubectl get nodes -o wide
+   kubectl get pods -A | grep -v -E "Running|Completed"
+   ```
+
+### Phase 2: Kubernetes Upgrade to v1.32.13
+1. In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update:
+   ```yaml
+   kube_version: 1.32.13
+   ```
+2. Re-sync inventory to ephemeral `kubespray/`:
+   ```sh
+   rsync -a ./inventory/ ./kubespray/inventory/
+   ```
+3. Run upgrade playbook:
+   ```sh
+   cd kubespray
+   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
+   ```
+4. Verify upgraded cluster:
+   ```sh
+   kubectl version
+   kubectl get nodes -o wide
+   kubectl get pods -A | grep -v -E "Running|Completed"
+   ```

bootstrap/init.sh

@@ -6,14 +6,19 @@
 echo "######################################################################################"
 echo "##  Reinit repository"
 rm -rf kubespray
-VERSION="release-2.27"
+VERSION="release-2.28"
 git clone --branch $VERSION https://github.com/kubernetes-sigs/kubespray.git
+if [ -d patches ]; then
+  for p in patches/*.patch; do
+    [ -f "$p" ] && (echo "## Applying patch $p" && git -C kubespray apply "../$p")
+  done
+fi
 
 echo "######################################################################################"
 echo "##  Activating pyenv venv"
 eval "$(pyenv init -)"
 eval "$(pyenv virtualenv-init -)"
-pyenv virtualenv 3.10.8 infrapuzzle-bootstrap
+pyenv virtualenvs --bare | grep -q "^infrapuzzle-bootstrap$" || pyenv virtualenv 3.10.8 infrapuzzle-bootstrap
 pyenv activate infrapuzzle-bootstrap
 python -m pip install -r kubespray/requirements.txt
 
@@ -46,9 +51,9 @@ echo "## * syncing config in kubespray dir"
 rsync -a ./inventory/ ./kubespray/inventory/
 echo "## * changing into kubespray dir"
 cd kubespray
-echo "## * execute the following command to force new settings"
+echo "## * Step 1: Reconcile Kubespray baseline (keep current k8s version):"
 echo "ansible-playbook -i inventory/prod/inventory.ini cluster.yml"
-echo "## * execute the following command to force new settings with forced cluster upgrade"
+echo "## * Step 2: Once verified, bump kube_version in inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml and run:"
 echo "ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml"
 
 # cp -r kubespray/inventory/sample kubespray/inventory/prod

bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml

@@ -27,7 +27,17 @@ kube_kubeadm_apiserver_extra_args:
 kube_api_anonymous_auth: true
 
 ## Change this to use another Kubernetes version, e.g. a current beta release
-kube_version: v1.31.9
+kube_version: 1.32.13
+
+# Single-node cluster settings: prevent draining or cordoning the only node
+drain_nodes: false
+upgrade_node_always_cordon: false
+upgrade_node_fail_if_drain_fails: false
+kubeadm_ignore_preflight_errors:
+  - ImagePull
+  - SystemVerification
+  - FileContent--proc-sys-net-bridge-bridge-nf-call-iptables
+  - CreateJob
 
 # kubernetes image repo define
 #kube_image_repo: "k8s.gcr.io"

bootstrap/patches/kubeadm-upgrade-ignore-preflight.patch

@@ -0,0 +1,20 @@
+diff --git a/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml b/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
+index 523956d1d..75723a6e0 100644
+--- a/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
++++ b/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
+@@ -17,6 +17,7 @@
+     --force
+     {%- else %}
+     --config={{ kube_config_dir }}/kubeadm-config.yaml
++    --ignore-preflight-errors={{ kubeadm_ignore_preflight_errors | join(',') }}
+     {%- endif %}
+     {%- if kube_version is version('1.32.0', '>=') %}
+     --skip-phases={{ kubeadm_init_phases_skip | join(',') }}
+@@ -38,6 +39,7 @@
+     {% if kubeadm_patches | length > 0 %}--patches={{ kubeadm_patches_dir }}{% endif %}
+     {%- else %}
+     --config={{ kube_config_dir }}/kubeadm-config.yaml
++    --ignore-preflight-errors={{ kubeadm_ignore_preflight_errors | join(',') }}
+     {%- endif %}
+     --skip-phases={{ kubeadm_upgrade_node_phases_skip | join(',') }}
+   register: kubeadm_upgrade