moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(bootstrap): document single-node cluster upgrade procedures and modernize init.sh

Browse Source 
- Update init.sh with smart checkout, patch application, sync-inventory alias, and runbook output
- Rewrite bootstrap/README.md with step-by-step upgrade guide, architectural invariants, and troubleshooting
- Update bootstrap/AGENTS.md with single-node operational rules and OS constraints
This commit is contained in:
Moritz Graf
2026-09-19 22:58:22 +02:00
parent 87649e6e22
commit 80a056f720
3 changed files with 264 additions and 215 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bootstrap/README.md
+129 -126
View File
@@ -1,162 +1,165 @@
# Bootstrap
The following lines document how to initalize a fresh cluster. On a real cluster, or using Vagrant. It therefore assumes to clone kubespray to this folder. It will be excluded in *.gitignore* and all files are in this folder.
This directory contains the automation, configurations, and playbooks used to provision and upgrade the single-node Kubernetes cluster **`haumdaucher`** (`136.243.23.215`) using **[Kubespray](https://github.com/kubernetes-sigs/kubespray)**.
The version to be initialized in configured in the `init.sh` script.
---
## Architecture & Single-Node Cluster Constraints
`haumdaucher` is a **single-node cluster** (control plane, etcd, and worker all run on the same machine). This topology requires specific configuration choices during cluster maintenance:
> [!IMPORTANT]
> ### Crucial Single-Node Invariants
> 1. **Never Cordon or Drain the Node**:
> Draining a single-node cluster evicts all workloads with nowhere to schedule them, causing pods to stay in `Pending` state and breaking critical cluster services (such as DNS, CNI, ingress, and local storage).
> In [inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml), the following settings are enforced:
> ```yaml
> drain_nodes: false
> upgrade_node_always_cordon: false
> upgrade_node_fail_if_drain_fails: false
> ```
> 2. **In-Place Upgrades via `cluster.yml`**:
> Instead of `upgrade-cluster.yml` (which defaults to draining nodes), single-node upgrades should use the in-place upgrade mechanism:
> ```sh
> ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
> ```
> 3. **Version Number Formatting**:
> Kubespray 2.28 interpolates `v{{ kube_version }}` across many internal tasks. Therefore, `kube_version` in `k8s-cluster.yml` MUST NOT include a leading `v`:
> ```yaml
> kube_version: 1.32.13 # Correct
> # kube_version: v1.32.13 # INCORRECT (leads to v1.32.13 -> vv1.32.13 errors)
> ```
> 4. **Host OS Lifecycle Constraint (Ubuntu 20.04 LTS)**:
> Kubespray `release-2.28` is the **final release line supporting Ubuntu 20.04**. Kubespray `release-2.29`+ drops Ubuntu 20.04 support and requires Ubuntu 22.04 or 24.04. Do not bump `VERSION` in `init.sh` beyond `release-2.28` until the host OS is upgraded.
---
## Quickstart
### 1. Environment Initialization
Source `init.sh` from this directory. It clones Kubespray, applies patches from `patches/`, configures the Python virtualenv, and syncs `./inventory/` into `./kubespray/inventory/`:
```sh
. ./init.sh
source ./init.sh
```
See [kubespray.io](https://kubespray.io/) on detailed information about kubespray. Though it seems to be a littlebit outdated.
*(Note: `init.sh` also exports the `sync-inventory` alias to quickly sync local `./inventory/` modifications into `./kubespray/inventory/` without re-cloning).*
## Vagrant
---
```sh
cd kubespray
vagrant up
# up and abkle to ssh
vagrant ssh k8s-1
```
## Step-by-Step Upgrade Runbook
## Prod
Upgrades are conducted in two decoupled phases:
1. **Phase 1**: Upgrade the Kubespray tooling baseline while keeping the Kubernetes version unchanged $\rightarrow$ Apply $\rightarrow$ Verify.
2. **Phase 2**: Upgrade Kubernetes components (API server, controller manager, scheduler, kubelet, kube-proxy) $\rightarrow$ Apply $\rightarrow$ Verify.
Prepare server:
* deactivate swap!
* `moritz username ALL=(ALL) NOPASSWD:ALL`
* `
### Step 0: Pre-Flight Checks & Backups
```sh
ssh centos@<ip>
# auth via pw
sudo su - root
adduser moritz
visudo # add as sudo user
su - moritz
sudo yum -y install vim python3
ssh-keygen
vim .ssh/authorized_users # paste key
chmod 644 .ssh/authorized_keys
# check whether login works with ssh key
sudo vim /etc/ssh/sshd_config # remove pw auth & root login
sudo yum upgrade -y && sudo reboot
```
Install Kubernetes:
```sh
. ./init.sh
# follow instructions from output, sth like:
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
And get credentials:
```sh
ssh <ip>
sudo su - root
cd
cp -r .kube /home/moritz/
chown -R moritz. /home/moritz/.kube
#ctrl + d
kubectl get ns # test connection
#ctrl + d
scp haumdaucher.de:/home/moritz/.kube/config .kube/config
```
Foreward in k8s-directory.
## Upgrade cluster
Check the current default value of `kube_version` in cloned repository.
**FOR K8S VERSION - Edit /Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml**
```sh
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e kube_version=v1.22.10 -e upgrade_cluster_setup=true cluster.yml
# or just the newest version
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
# upgrade to specific calico version (did not trigger/ failed)
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e calico_version=v3.15.2 cluster.yml --tags=network
```
History:
* 2020-04-18 kube_version=v1.16.8 kubespray_branch=release-2.12
## Add node
See [documentation](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/nodes.md).
Note: This was more or less a trial and error approach. Running different playbooks over and over again got it right at some point.
```sh
ansible-playbook -i inventory/prod/inventory.ini --limit=ns3088070.ip-37-59-40.eu,ns3100058.ip-37-59-61.eu scale.yml
ansible-playbook -i inventory/prod/inventory.ini --limit=etcd,kube-master -e ignore_assert_errors=yes cluster.yml
```
This runs everything and is kind of idempotent:
```sh
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
## Upgrade to Kubernetes 1.31 (Previous)
Required to execute:
```sh
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e drain_nodes=false upgrade-cluster.yml
```
+ set a feature flag: https://github.com/kubernetes-sigs/kubespray/issues/11887
## Upgrade to Kubespray release-2.28 & Kubernetes v1.32.13
> [!NOTE]
> Upgrade philosophy: decoupled two-phase upgrade.
> 1. Upgrade Kubespray tooling baseline (keeping k8s version unchanged).
> 2. Upgrade Kubernetes control plane and worker components.
>
> **Important**:
> - `haumdaucher` is a **single-node cluster**. Always use `-e drain_nodes=false` when running `upgrade-cluster.yml` to prevent the upgrade from hanging on draining.
> - **Ubuntu 20.04 LTS constraint**: Kubespray `release-2.28` is the **last** release officially supporting Ubuntu 20.04. Upstream Kubespray `release-2.29`+ requires Ubuntu 22.04 or 24.04.
### Phase 1: Kubespray Baseline Upgrade (Retaining k8s v1.31.9)
1. Ensure `init.sh` has `VERSION="release-2.28"`.
2. Initialize environment:
1. **Verify SSH and sudo access**:
```sh
. ./init.sh
ssh moritz@136.243.23.215 "sudo kubectl get nodes -o wide"
```
3. Run cluster playbook to reconcile Kubespray baseline:
2. **Check Pod Health**:
```sh
ssh moritz@136.243.23.215 "sudo kubectl get pods -A | grep -v -E 'Running|Completed'"
```
3. **Take a Manual etcd Snapshot**:
```sh
ssh moritz@136.243.23.215 "sudo etcdctl \
--cacert=/etc/ssl/etcd/ssl/ca.pem \
--cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem \
--key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem \
snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db"
```
---
### Step 1: Reconcile Kubespray Baseline (Phase 1)
1. Verify `VERSION` in `init.sh` matches the target Kubespray branch (e.g. `release-2.28`).
2. Run `source ./init.sh` to initialize the environment.
3. Apply the baseline configuration:
```sh
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
4. Verify cluster health:
```sh
kubectl get nodes -o wide
kubectl get pods -A | grep -v -E "Running|Completed"
kubectl get pods -n kube-system
```
### Phase 2: Kubernetes Upgrade to v1.32.13
1. In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update:
---
### Step 2: Upgrade Kubernetes Version (Phase 2)
1. In `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update `kube_version`:
```yaml
kube_version: 1.32.13
```
2. Re-sync inventory to ephemeral `kubespray/`:
2. Sync the configuration into the `kubespray/` workspace:
```sh
rsync -a ./inventory/ ./kubespray/inventory/
sync-inventory
```
3. Run upgrade playbook:
3. Run the in-place upgrade playbook:
```sh
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
```
4. Verify upgraded cluster:
---
### Step 3: Post-Upgrade Verification
1. **Check component versions**:
```sh
kubectl version
```
Ensure both `Client Version` and `Server Version` report `v1.32.13`.
2. **Verify Node status**:
```sh
kubectl get nodes -o wide
```
Ensure status is `Ready` and version matches `v1.32.13`.
3. **Verify System Pods**:
```sh
kubectl get pods -n kube-system
```
All control plane components (`kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `coredns`, `kube-flannel`, `kube-proxy`, `traefik`) should be `1/1 Running`.
4. **Verify Workloads**:
```sh
kubectl get pods -A | grep -v -E "Running|Completed"
```
---
## Troubleshooting & Edge Cases
### Kubeadm Preflight `CreateJob` Failure
Kubeadm 1.31+ preflight executes an ephemeral `upgrade-health-check-...` batch job with a 15-second timeout. On single-node clusters or under load, this may exceed 15s. To prevent the upgrade from halting:
- `CreateJob` is included in `kubeadm_ignore_preflight_errors` in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
- The patch `patches/kubeadm-upgrade-ignore-preflight.patch` ensures Kubespray's `v1beta4` kubeadm configuration propagates this flag during `upgrade apply`.
### Kubelet Housekeeping Stalls / Static Pod Timeout
If `kubeadm upgrade apply` times out waiting for static pod hash changes:
- Check `journalctl -u kubelet -n 100` for `Housekeeping took longer than expected` or `orphaned pod ... failed to rmdir()`.
- If an old container unmount left a stale bind-mount busy, lazily detach it:
```sh
sudo umount -l /var/lib/kubelet/pods/<orphaned-pod-uid>/volumes/...
sudo systemctl restart kubelet
```
---
## Directory Layout
* `init.sh`: Environment bootstrapping script and upgrade instructions.
* `inventory/`: Source-of-truth inventory and variable files.
* `patches/`: Git patches applied to upstream Kubespray on checkout.
* `kubespray/`: (Ephemeral) Cloned Kubespray repository.