infrapuzzle/bootstrap/AGENTS.md

AGENTS.md

Note

This directory handles the bootstrapping, provisioning, and in-place upgrades of the Haumdaucher Kubernetes cluster using Kubespray.

Project Overview

• Tool: Kubespray (Ansible-based).
• Topology: Single-node cluster (haumdaucher, 136.243.23.215). The single node acts simultaneously as control plane, etcd member, and worker.
• Host OS: Ubuntu 20.04.6 LTS.
• Method: The local inventory/ is the source of truth, synced into the checked-out Kubespray repository.
• Idempotency & Patching: The kubespray/ directory is ephemeral and managed by init.sh. Any essential Kubespray fixes are persisted in patches/ and applied automatically.

---

Architectural Invariants for Single-Node Upgrades

Important

The single-node topology imposes strict operational constraints that must be observed by all agents and operators:

1. NO DRAIN / NO CORDON:
   • Draining or cordoning a single-node cluster evicts all workloads with nowhere to schedule them, causing cascading failures and pending pods.
   • drain_nodes: false, upgrade_node_always_cordon: false, and upgrade_node_fail_if_drain_fails: false MUST remain configured in k8s-cluster.yml.
   • Use the in-place upgrade mechanism: ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml. This invokes kubeadm-upgrade.yml without cordon/drain overhead.
2. VERSION STRING FORMAT:
   • In inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml, kube_version must be specified without the leading v (e.g., 1.32.13, NOT v1.32.13), as Kubespray 2.28 interpolates v{{ kube_version }} in download URLs and package names.
3. UBUNTU 20.04 OS CONSTRAINT:
   • Kubespray release-2.28 is the final release line supporting Ubuntu 20.04 LTS. Upstream release-2.29+ drops Ubuntu 20.04 support. Do NOT bump VERSION in init.sh beyond release-2.28 until the host operating system has been upgraded to Ubuntu 22.04 or 24.04.
4. ETCD BACKUPS:
   • Always take a manual etcd snapshot before initiating any cluster upgrade.

---

Workflow & Scripts

1. Source init.sh

source init.sh

Actions performed:

• Clones / checks out the Kubespray branch defined by VERSION (release-2.28).
• Applies monorepo patches in patches/*.patch (e.g., propagating --ignore-preflight-errors for v1beta4 kubeadm configs).
• Activates the infrapuzzle-bootstrap pyenv virtualenv and verifies Python requirements.
• Syncs ./inventory/ to ./kubespray/inventory/.
• Exports the sync-inventory alias.
• Changes working directory to ./kubespray.

2. Upgrading the Single-Node Cluster (Two-Phase Decoupled Workflow)

Phase 1: Reconcile Kubespray Tooling Baseline (Keep existing k8s version)

cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml

Verification: Check node readiness and that no application pods crashed.

Phase 2: Upgrade Kubernetes Version (e.g. to 1.32.13)

1. Edit bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml:

   kube_version: 1.32.13
   

2. Re-sync inventory from bootstrap/ or bootstrap/kubespray/:

   sync-inventory
   

3. Run the in-place upgrade:

   cd kubespray
   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
   

4. Verify the cluster:

   kubectl version
   kubectl get nodes -o wide
   kubectl get pods -A | grep -v -E "Running|Completed"
   

---

Directory Structure

• init.sh: The entrypoint script (sources environment, applies patches, sets up inventory).
• patches/: Monorepo patches applied automatically to ephemeral kubespray/.
• inventory/: Source of truth for cluster inventory and variables.
• kubespray/: (Ephemeral / Gitignored) Cloned Kubespray repository.