# Bootstrap

This directory contains the automation, configurations, and playbooks used to provision and upgrade the single-node Kubernetes cluster **`haumdaucher`** (`136.243.23.215`) using **[Kubespray](https://github.com/kubernetes-sigs/kubespray)**.

---

## Architecture & Single-Node Cluster Constraints

`haumdaucher` is a **single-node cluster** (control plane, etcd, and worker all run on the same machine). This topology requires specific configuration choices during cluster maintenance:

> [!IMPORTANT]
> ### Crucial Single-Node Invariants
> 1. **Never Cordon or Drain the Node**:
>    Draining a single-node cluster evicts all workloads with nowhere to schedule them, causing pods to stay in `Pending` state and breaking critical cluster services (such as DNS, CNI, ingress, and local storage).
>    In [inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml), the following settings are enforced:
>    ```yaml
>    drain_nodes: false
>    upgrade_node_always_cordon: false
>    upgrade_node_fail_if_drain_fails: false
>    ```
> 2. **In-Place Upgrades via `cluster.yml`**:
>    Instead of `upgrade-cluster.yml` (which defaults to draining nodes), single-node upgrades should use the in-place upgrade mechanism:
>    ```sh
>    ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
>    ```
> 3. **Version Number Formatting**:
>    Kubespray 2.28 interpolates `v{{ kube_version }}` across many internal tasks. Therefore, `kube_version` in `k8s-cluster.yml` MUST NOT include a leading `v`:
>    ```yaml
>    kube_version: 1.32.13   # Correct
>    # kube_version: v1.32.13 # INCORRECT (leads to v1.32.13 -> vv1.32.13 errors)
>    ```
> 4. **Host OS Lifecycle Constraint (Ubuntu 20.04 LTS)**:
>    Kubespray `release-2.28` is the **final release line supporting Ubuntu 20.04**. Kubespray `release-2.29`+ drops Ubuntu 20.04 support and requires Ubuntu 22.04 or 24.04. Do not bump `VERSION` in `init.sh` beyond `release-2.28` until the host OS is upgraded.

---

## Quickstart

### 1. Environment Initialization
Source `init.sh` from this directory. It clones Kubespray, applies patches from `patches/`, configures the Python virtualenv, and syncs `./inventory/` into `./kubespray/inventory/`:

```sh
source ./init.sh
```

*(Note: `init.sh` also exports the `sync-inventory` alias to quickly sync local `./inventory/` modifications into `./kubespray/inventory/` without re-cloning).*

---

## Step-by-Step Upgrade Runbook

Upgrades are conducted in two decoupled phases:
1. **Phase 1**: Upgrade the Kubespray tooling baseline while keeping the Kubernetes version unchanged $\rightarrow$ Apply $\rightarrow$ Verify.
2. **Phase 2**: Upgrade Kubernetes components (API server, controller manager, scheduler, kubelet, kube-proxy) $\rightarrow$ Apply $\rightarrow$ Verify.

### Step 0: Pre-Flight Checks & Backups

1. **Verify SSH and sudo access**:
   ```sh
   ssh moritz@136.243.23.215 "sudo kubectl get nodes -o wide"
   ```

2. **Check Pod Health**:
   ```sh
   ssh moritz@136.243.23.215 "sudo kubectl get pods -A | grep -v -E 'Running|Completed'"
   ```

3. **Take a Manual etcd Snapshot**:
   ```sh
   ssh moritz@136.243.23.215 "sudo etcdctl \
     --cacert=/etc/ssl/etcd/ssl/ca.pem \
     --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem \
     --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem \
     snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db"
   ```

---

### Step 1: Reconcile Kubespray Baseline (Phase 1)

1. Verify `VERSION` in `init.sh` matches the target Kubespray branch (e.g. `release-2.28`).
2. Run `source ./init.sh` to initialize the environment.
3. Apply the baseline configuration:
   ```sh
   cd kubespray
   ansible-playbook -i inventory/prod/inventory.ini cluster.yml
   ```
4. Verify cluster health:
   ```sh
   kubectl get nodes -o wide
   kubectl get pods -n kube-system
   ```

---

### Step 2: Upgrade Kubernetes Version (Phase 2)

1. In `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update `kube_version`:
   ```yaml
   kube_version: 1.32.13
   ```

2. Sync the configuration into the `kubespray/` workspace:
   ```sh
   sync-inventory
   ```

3. Run the in-place upgrade playbook:
   ```sh
   cd kubespray
   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
   ```

---

### Step 3: Post-Upgrade Verification

1. **Check component versions**:
   ```sh
   kubectl version
   ```
   Ensure both `Client Version` and `Server Version` report `v1.32.13`.

2. **Verify Node status**:
   ```sh
   kubectl get nodes -o wide
   ```
   Ensure status is `Ready` and version matches `v1.32.13`.

3. **Verify System Pods**:
   ```sh
   kubectl get pods -n kube-system
   ```
   All control plane components (`kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `coredns`, `kube-flannel`, `kube-proxy`, `traefik`) should be `1/1 Running`.

4. **Verify Workloads**:
   ```sh
   kubectl get pods -A | grep -v -E "Running|Completed"
   ```

---

## Troubleshooting & Edge Cases

### Kubeadm Preflight `CreateJob` Failure
Kubeadm 1.31+ preflight executes an ephemeral `upgrade-health-check-...` batch job with a 15-second timeout. On single-node clusters or under load, this may exceed 15s. To prevent the upgrade from halting:
- `CreateJob` is included in `kubeadm_ignore_preflight_errors` in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
- The patch `patches/kubeadm-upgrade-ignore-preflight.patch` ensures Kubespray's `v1beta4` kubeadm configuration propagates this flag during `upgrade apply`.

### Kubelet Housekeeping Stalls / Static Pod Timeout
If `kubeadm upgrade apply` times out waiting for static pod hash changes:
- Check `journalctl -u kubelet -n 100` for `Housekeeping took longer than expected` or `orphaned pod ... failed to rmdir()`.
- If an old container unmount left a stale bind-mount busy, lazily detach it:
  ```sh
  sudo umount -l /var/lib/kubelet/pods/<orphaned-pod-uid>/volumes/...
  sudo systemctl restart kubelet
  ```

---

## Directory Layout
* `init.sh`: Environment bootstrapping script and upgrade instructions.
* `inventory/`: Source-of-truth inventory and variable files.
* `patches/`: Git patches applied to upstream Kubespray on checkout.
* `kubespray/`: (Ephemeral) Cloned Kubespray repository.