Moritz Graf
80a056f720
- Update init.sh with smart checkout, patch application, sync-inventory alias, and runbook output - Rewrite bootstrap/README.md with step-by-step upgrade guide, architectural invariants, and troubleshooting - Update bootstrap/AGENTS.md with single-node operational rules and OS constraints
83 lines
4.0 KiB
Markdown
83 lines
4.0 KiB
Markdown
# AGENTS.md
|
|
|
|
> [!NOTE]
|
|
> This directory handles the **bootstrapping, provisioning, and in-place upgrades** of the Haumdaucher Kubernetes cluster using **Kubespray**.
|
|
|
|
## Project Overview
|
|
* **Tool**: [Kubespray](https://github.com/kubernetes-sigs/kubespray) (Ansible-based).
|
|
* **Topology**: **Single-node cluster** (`haumdaucher`, `136.243.23.215`). The single node acts simultaneously as control plane, etcd member, and worker.
|
|
* **Host OS**: Ubuntu 20.04.6 LTS.
|
|
* **Method**: The local `inventory/` is the source of truth, synced into the checked-out Kubespray repository.
|
|
* **Idempotency & Patching**: The `kubespray/` directory is ephemeral and managed by `init.sh`. Any essential Kubespray fixes are persisted in `patches/` and applied automatically.
|
|
|
|
---
|
|
|
|
## Architectural Invariants for Single-Node Upgrades
|
|
|
|
> [!IMPORTANT]
|
|
> The single-node topology imposes strict operational constraints that must be observed by all agents and operators:
|
|
|
|
1. **NO DRAIN / NO CORDON**:
|
|
- Draining or cordoning a single-node cluster evicts all workloads with nowhere to schedule them, causing cascading failures and pending pods.
|
|
- `drain_nodes: false`, `upgrade_node_always_cordon: false`, and `upgrade_node_fail_if_drain_fails: false` MUST remain configured in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
|
|
- Use the in-place upgrade mechanism: `ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml`. This invokes `kubeadm-upgrade.yml` without cordon/drain overhead.
|
|
2. **VERSION STRING FORMAT**:
|
|
- In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, `kube_version` must be specified **without** the leading `v` (e.g., `1.32.13`, NOT `v1.32.13`), as Kubespray 2.28 interpolates `v{{ kube_version }}` in download URLs and package names.
|
|
3. **UBUNTU 20.04 OS CONSTRAINT**:
|
|
- Kubespray `release-2.28` is the **final release line** supporting Ubuntu 20.04 LTS. Upstream `release-2.29`+ drops Ubuntu 20.04 support. Do NOT bump `VERSION` in `init.sh` beyond `release-2.28` until the host operating system has been upgraded to Ubuntu 22.04 or 24.04.
|
|
4. **ETCD BACKUPS**:
|
|
- Always take a manual etcd snapshot before initiating any cluster upgrade.
|
|
|
|
---
|
|
|
|
## Workflow & Scripts
|
|
|
|
### 1. Source `init.sh`
|
|
```bash
|
|
source init.sh
|
|
```
|
|
**Actions performed**:
|
|
- Clones / checks out the Kubespray branch defined by `VERSION` (`release-2.28`).
|
|
- Applies monorepo patches in `patches/*.patch` (e.g., propagating `--ignore-preflight-errors` for `v1beta4` kubeadm configs).
|
|
- Activates the `infrapuzzle-bootstrap` pyenv virtualenv and verifies Python requirements.
|
|
- Syncs `./inventory/` to `./kubespray/inventory/`.
|
|
- Exports the `sync-inventory` alias.
|
|
- Changes working directory to `./kubespray`.
|
|
|
|
### 2. Upgrading the Single-Node Cluster (Two-Phase Decoupled Workflow)
|
|
|
|
#### Phase 1: Reconcile Kubespray Tooling Baseline (Keep existing k8s version)
|
|
```bash
|
|
cd kubespray
|
|
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
|
|
```
|
|
*Verification*: Check node readiness and that no application pods crashed.
|
|
|
|
#### Phase 2: Upgrade Kubernetes Version (e.g. to 1.32.13)
|
|
1. Edit `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`:
|
|
```yaml
|
|
kube_version: 1.32.13
|
|
```
|
|
2. Re-sync inventory from `bootstrap/` or `bootstrap/kubespray/`:
|
|
```bash
|
|
sync-inventory
|
|
```
|
|
3. Run the in-place upgrade:
|
|
```bash
|
|
cd kubespray
|
|
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
|
|
```
|
|
4. Verify the cluster:
|
|
```bash
|
|
kubectl version
|
|
kubectl get nodes -o wide
|
|
kubectl get pods -A | grep -v -E "Running|Completed"
|
|
```
|
|
|
|
---
|
|
|
|
## Directory Structure
|
|
* `init.sh`: The entrypoint script (sources environment, applies patches, sets up inventory).
|
|
* `patches/`: Monorepo patches applied automatically to ephemeral `kubespray/`.
|
|
* `inventory/`: Source of truth for cluster inventory and variables.
|
|
* `kubespray/`: (Ephemeral / Gitignored) Cloned Kubespray repository. |