docs(bootstrap): document single-node cluster upgrade procedures and modernize init.sh

- Update init.sh with smart checkout, patch application, sync-inventory alias, and runbook output - Rewrite bootstrap/README.md with step-by-step upgrade guide, architectural invariants, and troubleshooting - Update bootstrap/AGENTS.md with single-node operational rules and OS constraints
2026-09-19 22:58:22 +02:00
parent 87649e6e22
commit 80a056f720
3 changed files with 264 additions and 215 deletions
@@ -1,50 +1,83 @@
 # AGENTS.md

 > [!NOTE]
-> This directory handles the **bootstrapping and provisioning** of the Haumdaucher Kubernetes cluster using **Kubespray**.
+> This directory handles the **bootstrapping, provisioning, and in-place upgrades** of the Haumdaucher Kubernetes cluster using **Kubespray**.

 ## Project Overview
 *   **Tool**: [Kubespray](https://github.com/kubernetes-sigs/kubespray) (Ansible-based).
-*   **Method**: The local `inventory/` is the source of truth, which is synced into a checked-out Kubespray repository.
-*   **Idempotency**: The process is designed to be repeatable. The `kubespray` folder is treated as ephemeral and is re-created by `init.sh`.
+*   **Topology**: **Single-node cluster** (`haumdaucher`, `136.243.23.215`). The single node acts simultaneously as control plane, etcd member, and worker.
+*   **Host OS**: Ubuntu 20.04.6 LTS.
+*   **Method**: The local `inventory/` is the source of truth, synced into the checked-out Kubespray repository.
+*   **Idempotency & Patching**: The `kubespray/` directory is ephemeral and managed by `init.sh`. Any essential Kubespray fixes are persisted in `patches/` and applied automatically.
+
+---
+
+## Architectural Invariants for Single-Node Upgrades
+
+> [!IMPORTANT]
+> The single-node topology imposes strict operational constraints that must be observed by all agents and operators:
+
+1. **NO DRAIN / NO CORDON**:
+   - Draining or cordoning a single-node cluster evicts all workloads with nowhere to schedule them, causing cascading failures and pending pods.
+   - `drain_nodes: false`, `upgrade_node_always_cordon: false`, and `upgrade_node_fail_if_drain_fails: false` MUST remain configured in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
+   - Use the in-place upgrade mechanism: `ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml`. This invokes `kubeadm-upgrade.yml` without cordon/drain overhead.
+2. **VERSION STRING FORMAT**:
+   - In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, `kube_version` must be specified **without** the leading `v` (e.g., `1.32.13`, NOT `v1.32.13`), as Kubespray 2.28 interpolates `v{{ kube_version }}` in download URLs and package names.
+3. **UBUNTU 20.04 OS CONSTRAINT**:
+   - Kubespray `release-2.28` is the **final release line** supporting Ubuntu 20.04 LTS. Upstream `release-2.29`+ drops Ubuntu 20.04 support. Do NOT bump `VERSION` in `init.sh` beyond `release-2.28` until the host operating system has been upgraded to Ubuntu 22.04 or 24.04.
+4. **ETCD BACKUPS**:
+   - Always take a manual etcd snapshot before initiating any cluster upgrade.
+
+---

 ## Workflow & Scripts
-The core workflow is encapsulated in `init.sh`.

-### `init.sh`
-**Purpose**: Prepares the environment and Kubespray for deployment.
-**Actions**:
-1.  **Clean Slate**: Deletes existing `kubespray/` directory.
-2.  **Clone**: Clones Kubespray (version defined in variable `VERSION`, e.g., `release-2.28`).
-3.  **Environment**: Sets up Python virtualenv via `pyenv` and installs `requirements.txt`.
-4.  **Sync**: Copies local `./inventory/` configurations into `./kubespray/inventory/`.
+### 1. Source `init.sh`
+```bash
+source init.sh
+```
+**Actions performed**:
+- Clones / checks out the Kubespray branch defined by `VERSION` (`release-2.28`).
+- Applies monorepo patches in `patches/*.patch` (e.g., propagating `--ignore-preflight-errors` for `v1beta4` kubeadm configs).
+- Activates the `infrapuzzle-bootstrap` pyenv virtualenv and verifies Python requirements.
+- Syncs `./inventory/` to `./kubespray/inventory/`.
+- Exports the `sync-inventory` alias.
+- Changes working directory to `./kubespray`.

-### Usage
-1.  **Source the script**:
-    ```bash
-    source init.sh
-    ```
-2.  **Deploy / Upgrade**:
-    After sourcing, go to the `kubespray` directory and run the Ansible playbooks as instructed by the script output.
-    *   **Phase 1: Reconcile Kubespray baseline (stay on current k8s version)**:
-        ```bash
-        cd kubespray
-        ansible-playbook -i inventory/prod/inventory.ini cluster.yml
-        ```
-    *   **Phase 2: Upgrade Kubernetes version (e.g. to v1.32.13)**:
-        Update `kube_version` in `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, re-sync, and run:
-        ```bash
-        cd kubespray
-        ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
-        ```
+### 2. Upgrading the Single-Node Cluster (Two-Phase Decoupled Workflow)
+
+#### Phase 1: Reconcile Kubespray Tooling Baseline (Keep existing k8s version)
+```bash
+cd kubespray
+ansible-playbook -i inventory/prod/inventory.ini cluster.yml
+```
+*Verification*: Check node readiness and that no application pods crashed.
+
+#### Phase 2: Upgrade Kubernetes Version (e.g. to 1.32.13)
+1. Edit `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`:
+   ```yaml
+   kube_version: 1.32.13
+   ```
+2. Re-sync inventory from `bootstrap/` or `bootstrap/kubespray/`:
+   ```bash
+   sync-inventory
+   ```
+3. Run the in-place upgrade:
+   ```bash
+   cd kubespray
+   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
+   ```
+4. Verify the cluster:
+   ```bash
+   kubectl version
+   kubectl get nodes -o wide
+   kubectl get pods -A | grep -v -E "Running|Completed"
+   ```
+
+---

 ## Directory Structure
-*   `init.sh`: The entry point script. **Source of truth for Kubespray version.**
-*   `inventory/`: Contains cluster inventory configurations (hosts, variables). **Edit this, not the one in `kubespray/`**.
-*   `kubespray/`: (Ignored/Ephemeral) The checked-out Kubespray repository. **Do not edit files here directly**; they will be overwritten.
-
-## Configuration Updates
-To upgrade Kubespray or change cluster config:
-1.  **Version Upgrade**: Update `VERSION` in `init.sh` (e.g., to `release-2.28`).
-2.  **Config Changes**: Modify files in `./inventory/`.
-3.  **Apply**: Run `source init.sh` then execute the Ansible playbook.
+*   `init.sh`: The entrypoint script (sources environment, applies patches, sets up inventory).
+*   `patches/`: Monorepo patches applied automatically to ephemeral `kubespray/`.
+*   `inventory/`: Source of truth for cluster inventory and variables.
+*   `kubespray/`: (Ephemeral / Gitignored) Cloned Kubespray repository.