Moritz Graf
80a056f720
- Update init.sh with smart checkout, patch application, sync-inventory alias, and runbook output - Rewrite bootstrap/README.md with step-by-step upgrade guide, architectural invariants, and troubleshooting - Update bootstrap/AGENTS.md with single-node operational rules and OS constraints
166 lines
6.3 KiB
Markdown
166 lines
6.3 KiB
Markdown
# Bootstrap
|
|
|
|
This directory contains the automation, configurations, and playbooks used to provision and upgrade the single-node Kubernetes cluster **`haumdaucher`** (`136.243.23.215`) using **[Kubespray](https://github.com/kubernetes-sigs/kubespray)**.
|
|
|
|
---
|
|
|
|
## Architecture & Single-Node Cluster Constraints
|
|
|
|
`haumdaucher` is a **single-node cluster** (control plane, etcd, and worker all run on the same machine). This topology requires specific configuration choices during cluster maintenance:
|
|
|
|
> [!IMPORTANT]
|
|
> ### Crucial Single-Node Invariants
|
|
> 1. **Never Cordon or Drain the Node**:
|
|
> Draining a single-node cluster evicts all workloads with nowhere to schedule them, causing pods to stay in `Pending` state and breaking critical cluster services (such as DNS, CNI, ingress, and local storage).
|
|
> In [inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml), the following settings are enforced:
|
|
> ```yaml
|
|
> drain_nodes: false
|
|
> upgrade_node_always_cordon: false
|
|
> upgrade_node_fail_if_drain_fails: false
|
|
> ```
|
|
> 2. **In-Place Upgrades via `cluster.yml`**:
|
|
> Instead of `upgrade-cluster.yml` (which defaults to draining nodes), single-node upgrades should use the in-place upgrade mechanism:
|
|
> ```sh
|
|
> ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
|
|
> ```
|
|
> 3. **Version Number Formatting**:
|
|
> Kubespray 2.28 interpolates `v{{ kube_version }}` across many internal tasks. Therefore, `kube_version` in `k8s-cluster.yml` MUST NOT include a leading `v`:
|
|
> ```yaml
|
|
> kube_version: 1.32.13 # Correct
|
|
> # kube_version: v1.32.13 # INCORRECT (leads to v1.32.13 -> vv1.32.13 errors)
|
|
> ```
|
|
> 4. **Host OS Lifecycle Constraint (Ubuntu 20.04 LTS)**:
|
|
> Kubespray `release-2.28` is the **final release line supporting Ubuntu 20.04**. Kubespray `release-2.29`+ drops Ubuntu 20.04 support and requires Ubuntu 22.04 or 24.04. Do not bump `VERSION` in `init.sh` beyond `release-2.28` until the host OS is upgraded.
|
|
|
|
---
|
|
|
|
## Quickstart
|
|
|
|
### 1. Environment Initialization
|
|
Source `init.sh` from this directory. It clones Kubespray, applies patches from `patches/`, configures the Python virtualenv, and syncs `./inventory/` into `./kubespray/inventory/`:
|
|
|
|
```sh
|
|
source ./init.sh
|
|
```
|
|
|
|
*(Note: `init.sh` also exports the `sync-inventory` alias to quickly sync local `./inventory/` modifications into `./kubespray/inventory/` without re-cloning).*
|
|
|
|
---
|
|
|
|
## Step-by-Step Upgrade Runbook
|
|
|
|
Upgrades are conducted in two decoupled phases:
|
|
1. **Phase 1**: Upgrade the Kubespray tooling baseline while keeping the Kubernetes version unchanged $\rightarrow$ Apply $\rightarrow$ Verify.
|
|
2. **Phase 2**: Upgrade Kubernetes components (API server, controller manager, scheduler, kubelet, kube-proxy) $\rightarrow$ Apply $\rightarrow$ Verify.
|
|
|
|
### Step 0: Pre-Flight Checks & Backups
|
|
|
|
1. **Verify SSH and sudo access**:
|
|
```sh
|
|
ssh moritz@136.243.23.215 "sudo kubectl get nodes -o wide"
|
|
```
|
|
|
|
2. **Check Pod Health**:
|
|
```sh
|
|
ssh moritz@136.243.23.215 "sudo kubectl get pods -A | grep -v -E 'Running|Completed'"
|
|
```
|
|
|
|
3. **Take a Manual etcd Snapshot**:
|
|
```sh
|
|
ssh moritz@136.243.23.215 "sudo etcdctl \
|
|
--cacert=/etc/ssl/etcd/ssl/ca.pem \
|
|
--cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem \
|
|
--key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem \
|
|
snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db"
|
|
```
|
|
|
|
---
|
|
|
|
### Step 1: Reconcile Kubespray Baseline (Phase 1)
|
|
|
|
1. Verify `VERSION` in `init.sh` matches the target Kubespray branch (e.g. `release-2.28`).
|
|
2. Run `source ./init.sh` to initialize the environment.
|
|
3. Apply the baseline configuration:
|
|
```sh
|
|
cd kubespray
|
|
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
|
|
```
|
|
4. Verify cluster health:
|
|
```sh
|
|
kubectl get nodes -o wide
|
|
kubectl get pods -n kube-system
|
|
```
|
|
|
|
---
|
|
|
|
### Step 2: Upgrade Kubernetes Version (Phase 2)
|
|
|
|
1. In `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update `kube_version`:
|
|
```yaml
|
|
kube_version: 1.32.13
|
|
```
|
|
|
|
2. Sync the configuration into the `kubespray/` workspace:
|
|
```sh
|
|
sync-inventory
|
|
```
|
|
|
|
3. Run the in-place upgrade playbook:
|
|
```sh
|
|
cd kubespray
|
|
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
|
|
```
|
|
|
|
---
|
|
|
|
### Step 3: Post-Upgrade Verification
|
|
|
|
1. **Check component versions**:
|
|
```sh
|
|
kubectl version
|
|
```
|
|
Ensure both `Client Version` and `Server Version` report `v1.32.13`.
|
|
|
|
2. **Verify Node status**:
|
|
```sh
|
|
kubectl get nodes -o wide
|
|
```
|
|
Ensure status is `Ready` and version matches `v1.32.13`.
|
|
|
|
3. **Verify System Pods**:
|
|
```sh
|
|
kubectl get pods -n kube-system
|
|
```
|
|
All control plane components (`kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `coredns`, `kube-flannel`, `kube-proxy`, `traefik`) should be `1/1 Running`.
|
|
|
|
4. **Verify Workloads**:
|
|
```sh
|
|
kubectl get pods -A | grep -v -E "Running|Completed"
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting & Edge Cases
|
|
|
|
### Kubeadm Preflight `CreateJob` Failure
|
|
Kubeadm 1.31+ preflight executes an ephemeral `upgrade-health-check-...` batch job with a 15-second timeout. On single-node clusters or under load, this may exceed 15s. To prevent the upgrade from halting:
|
|
- `CreateJob` is included in `kubeadm_ignore_preflight_errors` in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
|
|
- The patch `patches/kubeadm-upgrade-ignore-preflight.patch` ensures Kubespray's `v1beta4` kubeadm configuration propagates this flag during `upgrade apply`.
|
|
|
|
### Kubelet Housekeeping Stalls / Static Pod Timeout
|
|
If `kubeadm upgrade apply` times out waiting for static pod hash changes:
|
|
- Check `journalctl -u kubelet -n 100` for `Housekeeping took longer than expected` or `orphaned pod ... failed to rmdir()`.
|
|
- If an old container unmount left a stale bind-mount busy, lazily detach it:
|
|
```sh
|
|
sudo umount -l /var/lib/kubelet/pods/<orphaned-pod-uid>/volumes/...
|
|
sudo systemctl restart kubelet
|
|
```
|
|
|
|
---
|
|
|
|
## Directory Layout
|
|
* `init.sh`: Environment bootstrapping script and upgrade instructions.
|
|
* `inventory/`: Source-of-truth inventory and variable files.
|
|
* `patches/`: Git patches applied to upstream Kubespray on checkout.
|
|
* `kubespray/`: (Ephemeral) Cloned Kubespray repository.
|