# AGENTS.md

> [!NOTE]
> This directory handles the **bootstrapping, provisioning, and in-place upgrades** of the Haumdaucher Kubernetes cluster using **Kubespray**.

## Project Overview
*   **Tool**: [Kubespray](https://github.com/kubernetes-sigs/kubespray) (Ansible-based).
*   **Topology**: **Single-node cluster** (`haumdaucher`, `136.243.23.215`). The single node acts simultaneously as control plane, etcd member, and worker.
*   **Host OS**: Ubuntu 20.04.6 LTS.
*   **Method**: The local `inventory/` is the source of truth, synced into the checked-out Kubespray repository.
*   **Idempotency & Patching**: The `kubespray/` directory is ephemeral and managed by `init.sh`. Any essential Kubespray fixes are persisted in `patches/` and applied automatically.

---

## Architectural Invariants for Single-Node Upgrades

> [!IMPORTANT]
> The single-node topology imposes strict operational constraints that must be observed by all agents and operators:

1. **NO DRAIN / NO CORDON**:
   - Draining or cordoning a single-node cluster evicts all workloads with nowhere to schedule them, causing cascading failures and pending pods.
   - `drain_nodes: false`, `upgrade_node_always_cordon: false`, and `upgrade_node_fail_if_drain_fails: false` MUST remain configured in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
   - Use the in-place upgrade mechanism: `ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml`. This invokes `kubeadm-upgrade.yml` without cordon/drain overhead.
2. **VERSION STRING FORMAT**:
   - In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, `kube_version` must be specified **without** the leading `v` (e.g., `1.32.13`, NOT `v1.32.13`), as Kubespray 2.28 interpolates `v{{ kube_version }}` in download URLs and package names.
3. **UBUNTU 20.04 OS CONSTRAINT**:
   - Kubespray `release-2.28` is the **final release line** supporting Ubuntu 20.04 LTS. Upstream `release-2.29`+ drops Ubuntu 20.04 support. Do NOT bump `VERSION` in `init.sh` beyond `release-2.28` until the host operating system has been upgraded to Ubuntu 22.04 or 24.04.
4. **ETCD BACKUPS**:
   - Always take a manual etcd snapshot before initiating any cluster upgrade.

---

## Workflow & Scripts

### 1. Source `init.sh`
```bash
source init.sh
```
**Actions performed**:
- Clones / checks out the Kubespray branch defined by `VERSION` (`release-2.28`).
- Applies monorepo patches in `patches/*.patch` (e.g., propagating `--ignore-preflight-errors` for `v1beta4` kubeadm configs).
- Activates the `infrapuzzle-bootstrap` pyenv virtualenv and verifies Python requirements.
- Syncs `./inventory/` to `./kubespray/inventory/`.
- Exports the `sync-inventory` alias.
- Changes working directory to `./kubespray`.

### 2. Upgrading the Single-Node Cluster (Two-Phase Decoupled Workflow)

#### Phase 1: Reconcile Kubespray Tooling Baseline (Keep existing k8s version)
```bash
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
*Verification*: Check node readiness and that no application pods crashed.

#### Phase 2: Upgrade Kubernetes Version (e.g. to 1.32.13)
1. Edit `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`:
   ```yaml
   kube_version: 1.32.13
   ```
2. Re-sync inventory from `bootstrap/` or `bootstrap/kubespray/`:
   ```bash
   sync-inventory
   ```
3. Run the in-place upgrade:
   ```bash
   cd kubespray
   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
   ```
4. Verify the cluster:
   ```bash
   kubectl version
   kubectl get nodes -o wide
   kubectl get pods -A | grep -v -E "Running|Completed"
   ```

---

## Directory Structure
*   `init.sh`: The entrypoint script (sources environment, applies patches, sets up inventory).
*   `patches/`: Monorepo patches applied automatically to ephemeral `kubespray/`.
*   `inventory/`: Source of truth for cluster inventory and variables.
*   `kubespray/`: (Ephemeral / Gitignored) Cloned Kubespray repository.