moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(bootstrap): document single-node cluster upgrade procedures and modernize init.sh

Browse Source 
- Update init.sh with smart checkout, patch application, sync-inventory alias, and runbook output
- Rewrite bootstrap/README.md with step-by-step upgrade guide, architectural invariants, and troubleshooting
- Update bootstrap/AGENTS.md with single-node operational rules and OS constraints
This commit is contained in:
Moritz Graf
2026-09-19 22:58:22 +02:00
parent 87649e6e22
commit 80a056f720
3 changed files with 264 additions and 215 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bootstrap/AGENTS.md
+71 -38
View File
@@ -1,50 +1,83 @@
# AGENTS.md
> [!NOTE]
> This directory handles the **bootstrapping and provisioning** of the Haumdaucher Kubernetes cluster using **Kubespray**.
> This directory handles the **bootstrapping, provisioning, and in-place upgrades** of the Haumdaucher Kubernetes cluster using **Kubespray**.
## Project Overview
* **Tool**: [Kubespray](https://github.com/kubernetes-sigs/kubespray) (Ansible-based).
* **Method**: The local `inventory/` is the source of truth, which is synced into a checked-out Kubespray repository.
* **Idempotency**: The process is designed to be repeatable. The `kubespray` folder is treated as ephemeral and is re-created by `init.sh`.
* **Topology**: **Single-node cluster** (`haumdaucher`, `136.243.23.215`). The single node acts simultaneously as control plane, etcd member, and worker.
* **Host OS**: Ubuntu 20.04.6 LTS.
* **Method**: The local `inventory/` is the source of truth, synced into the checked-out Kubespray repository.
* **Idempotency & Patching**: The `kubespray/` directory is ephemeral and managed by `init.sh`. Any essential Kubespray fixes are persisted in `patches/` and applied automatically.
---
## Architectural Invariants for Single-Node Upgrades
> [!IMPORTANT]
> The single-node topology imposes strict operational constraints that must be observed by all agents and operators:
1. **NO DRAIN / NO CORDON**:
- Draining or cordoning a single-node cluster evicts all workloads with nowhere to schedule them, causing cascading failures and pending pods.
- `drain_nodes: false`, `upgrade_node_always_cordon: false`, and `upgrade_node_fail_if_drain_fails: false` MUST remain configured in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
- Use the in-place upgrade mechanism: `ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml`. This invokes `kubeadm-upgrade.yml` without cordon/drain overhead.
2. **VERSION STRING FORMAT**:
- In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, `kube_version` must be specified **without** the leading `v` (e.g., `1.32.13`, NOT `v1.32.13`), as Kubespray 2.28 interpolates `v{{ kube_version }}` in download URLs and package names.
3. **UBUNTU 20.04 OS CONSTRAINT**:
- Kubespray `release-2.28` is the **final release line** supporting Ubuntu 20.04 LTS. Upstream `release-2.29`+ drops Ubuntu 20.04 support. Do NOT bump `VERSION` in `init.sh` beyond `release-2.28` until the host operating system has been upgraded to Ubuntu 22.04 or 24.04.
4. **ETCD BACKUPS**:
- Always take a manual etcd snapshot before initiating any cluster upgrade.
---
## Workflow & Scripts
The core workflow is encapsulated in `init.sh`.
### `init.sh`
**Purpose**: Prepares the environment and Kubespray for deployment.
**Actions**:
1. **Clean Slate**: Deletes existing `kubespray/` directory.
2. **Clone**: Clones Kubespray (version defined in variable `VERSION`, e.g., `release-2.28`).
3. **Environment**: Sets up Python virtualenv via `pyenv` and installs `requirements.txt`.
4. **Sync**: Copies local `./inventory/` configurations into `./kubespray/inventory/`.
### 1. Source `init.sh`
```bash
source init.sh
```
**Actions performed**:
- Clones / checks out the Kubespray branch defined by `VERSION` (`release-2.28`).
- Applies monorepo patches in `patches/*.patch` (e.g., propagating `--ignore-preflight-errors` for `v1beta4` kubeadm configs).
- Activates the `infrapuzzle-bootstrap` pyenv virtualenv and verifies Python requirements.
- Syncs `./inventory/` to `./kubespray/inventory/`.
- Exports the `sync-inventory` alias.
- Changes working directory to `./kubespray`.
### Usage
1. **Source the script**:
```bash
source init.sh
```
2. **Deploy / Upgrade**:
After sourcing, go to the `kubespray` directory and run the Ansible playbooks as instructed by the script output.
* **Phase 1: Reconcile Kubespray baseline (stay on current k8s version)**:
```bash
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
* **Phase 2: Upgrade Kubernetes version (e.g. to v1.32.13)**:
Update `kube_version` in `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, re-sync, and run:
```bash
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
```
### 2. Upgrading the Single-Node Cluster (Two-Phase Decoupled Workflow)
#### Phase 1: Reconcile Kubespray Tooling Baseline (Keep existing k8s version)
```bash
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
*Verification*: Check node readiness and that no application pods crashed.
#### Phase 2: Upgrade Kubernetes Version (e.g. to 1.32.13)
1. Edit `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`:
```yaml
kube_version: 1.32.13
```
2. Re-sync inventory from `bootstrap/` or `bootstrap/kubespray/`:
```bash
sync-inventory
```
3. Run the in-place upgrade:
```bash
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
```
4. Verify the cluster:
```bash
kubectl version
kubectl get nodes -o wide
kubectl get pods -A | grep -v -E "Running|Completed"
```
---
## Directory Structure
* `init.sh`: The entry point script. **Source of truth for Kubespray version.**
* `inventory/`: Contains cluster inventory configurations (hosts, variables). **Edit this, not the one in `kubespray/`**.
* `kubespray/`: (Ignored/Ephemeral) The checked-out Kubespray repository. **Do not edit files here directly**; they will be overwritten.
## Configuration Updates
To upgrade Kubespray or change cluster config:
1. **Version Upgrade**: Update `VERSION` in `init.sh` (e.g., to `release-2.28`).
2. **Config Changes**: Modify files in `./inventory/`.
3. **Apply**: Run `source init.sh` then execute the Ansible playbook.
* `init.sh`: The entrypoint script (sources environment, applies patches, sets up inventory).
* `patches/`: Monorepo patches applied automatically to ephemeral `kubespray/`.
* `inventory/`: Source of truth for cluster inventory and variables.
* `kubespray/`: (Ephemeral / Gitignored) Cloned Kubespray repository.
bootstrap/README.md
+129 -126
View File
@@ -1,162 +1,165 @@
# Bootstrap
The following lines document how to initalize a fresh cluster. On a real cluster, or using Vagrant. It therefore assumes to clone kubespray to this folder. It will be excluded in *.gitignore* and all files are in this folder.
This directory contains the automation, configurations, and playbooks used to provision and upgrade the single-node Kubernetes cluster **`haumdaucher`** (`136.243.23.215`) using **[Kubespray](https://github.com/kubernetes-sigs/kubespray)**.
The version to be initialized in configured in the `init.sh` script.
---
## Architecture & Single-Node Cluster Constraints
`haumdaucher` is a **single-node cluster** (control plane, etcd, and worker all run on the same machine). This topology requires specific configuration choices during cluster maintenance:
> [!IMPORTANT]
> ### Crucial Single-Node Invariants
> 1. **Never Cordon or Drain the Node**:
> Draining a single-node cluster evicts all workloads with nowhere to schedule them, causing pods to stay in `Pending` state and breaking critical cluster services (such as DNS, CNI, ingress, and local storage).
> In [inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml), the following settings are enforced:
> ```yaml
> drain_nodes: false
> upgrade_node_always_cordon: false
> upgrade_node_fail_if_drain_fails: false
> ```
> 2. **In-Place Upgrades via `cluster.yml`**:
> Instead of `upgrade-cluster.yml` (which defaults to draining nodes), single-node upgrades should use the in-place upgrade mechanism:
> ```sh
> ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
> ```
> 3. **Version Number Formatting**:
> Kubespray 2.28 interpolates `v{{ kube_version }}` across many internal tasks. Therefore, `kube_version` in `k8s-cluster.yml` MUST NOT include a leading `v`:
> ```yaml
> kube_version: 1.32.13 # Correct
> # kube_version: v1.32.13 # INCORRECT (leads to v1.32.13 -> vv1.32.13 errors)
> ```
> 4. **Host OS Lifecycle Constraint (Ubuntu 20.04 LTS)**:
> Kubespray `release-2.28` is the **final release line supporting Ubuntu 20.04**. Kubespray `release-2.29`+ drops Ubuntu 20.04 support and requires Ubuntu 22.04 or 24.04. Do not bump `VERSION` in `init.sh` beyond `release-2.28` until the host OS is upgraded.
---
## Quickstart
### 1. Environment Initialization
Source `init.sh` from this directory. It clones Kubespray, applies patches from `patches/`, configures the Python virtualenv, and syncs `./inventory/` into `./kubespray/inventory/`:
```sh
. ./init.sh
source ./init.sh
```
See [kubespray.io](https://kubespray.io/) on detailed information about kubespray. Though it seems to be a littlebit outdated.
*(Note: `init.sh` also exports the `sync-inventory` alias to quickly sync local `./inventory/` modifications into `./kubespray/inventory/` without re-cloning).*
## Vagrant
---
```sh
cd kubespray
vagrant up
# up and abkle to ssh
vagrant ssh k8s-1
```
## Step-by-Step Upgrade Runbook
## Prod
Upgrades are conducted in two decoupled phases:
1. **Phase 1**: Upgrade the Kubespray tooling baseline while keeping the Kubernetes version unchanged $\rightarrow$ Apply $\rightarrow$ Verify.
2. **Phase 2**: Upgrade Kubernetes components (API server, controller manager, scheduler, kubelet, kube-proxy) $\rightarrow$ Apply $\rightarrow$ Verify.
Prepare server:
* deactivate swap!
* `moritz username ALL=(ALL) NOPASSWD:ALL`
* `
### Step 0: Pre-Flight Checks & Backups
```sh
ssh centos@<ip>
# auth via pw
sudo su - root
adduser moritz
visudo # add as sudo user
su - moritz
sudo yum -y install vim python3
ssh-keygen
vim .ssh/authorized_users # paste key
chmod 644 .ssh/authorized_keys
# check whether login works with ssh key
sudo vim /etc/ssh/sshd_config # remove pw auth & root login
sudo yum upgrade -y && sudo reboot
```
Install Kubernetes:
```sh
. ./init.sh
# follow instructions from output, sth like:
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
And get credentials:
```sh
ssh <ip>
sudo su - root
cd
cp -r .kube /home/moritz/
chown -R moritz. /home/moritz/.kube
#ctrl + d
kubectl get ns # test connection
#ctrl + d
scp haumdaucher.de:/home/moritz/.kube/config .kube/config
```
Foreward in k8s-directory.
## Upgrade cluster
Check the current default value of `kube_version` in cloned repository.
**FOR K8S VERSION - Edit /Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml**
```sh
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e kube_version=v1.22.10 -e upgrade_cluster_setup=true cluster.yml
# or just the newest version
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
# upgrade to specific calico version (did not trigger/ failed)
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e calico_version=v3.15.2 cluster.yml --tags=network
```
History:
* 2020-04-18 kube_version=v1.16.8 kubespray_branch=release-2.12
## Add node
See [documentation](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/nodes.md).
Note: This was more or less a trial and error approach. Running different playbooks over and over again got it right at some point.
```sh
ansible-playbook -i inventory/prod/inventory.ini --limit=ns3088070.ip-37-59-40.eu,ns3100058.ip-37-59-61.eu scale.yml
ansible-playbook -i inventory/prod/inventory.ini --limit=etcd,kube-master -e ignore_assert_errors=yes cluster.yml
```
This runs everything and is kind of idempotent:
```sh
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
## Upgrade to Kubernetes 1.31 (Previous)
Required to execute:
```sh
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e drain_nodes=false upgrade-cluster.yml
```
+ set a feature flag: https://github.com/kubernetes-sigs/kubespray/issues/11887
## Upgrade to Kubespray release-2.28 & Kubernetes v1.32.13
> [!NOTE]
> Upgrade philosophy: decoupled two-phase upgrade.
> 1. Upgrade Kubespray tooling baseline (keeping k8s version unchanged).
> 2. Upgrade Kubernetes control plane and worker components.
>
> **Important**:
> - `haumdaucher` is a **single-node cluster**. Always use `-e drain_nodes=false` when running `upgrade-cluster.yml` to prevent the upgrade from hanging on draining.
> - **Ubuntu 20.04 LTS constraint**: Kubespray `release-2.28` is the **last** release officially supporting Ubuntu 20.04. Upstream Kubespray `release-2.29`+ requires Ubuntu 22.04 or 24.04.
### Phase 1: Kubespray Baseline Upgrade (Retaining k8s v1.31.9)
1. Ensure `init.sh` has `VERSION="release-2.28"`.
2. Initialize environment:
1. **Verify SSH and sudo access**:
```sh
. ./init.sh
ssh moritz@136.243.23.215 "sudo kubectl get nodes -o wide"
```
3. Run cluster playbook to reconcile Kubespray baseline:
2. **Check Pod Health**:
```sh
ssh moritz@136.243.23.215 "sudo kubectl get pods -A | grep -v -E 'Running|Completed'"
```
3. **Take a Manual etcd Snapshot**:
```sh
ssh moritz@136.243.23.215 "sudo etcdctl \
--cacert=/etc/ssl/etcd/ssl/ca.pem \
--cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem \
--key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem \
snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db"
```
---
### Step 1: Reconcile Kubespray Baseline (Phase 1)
1. Verify `VERSION` in `init.sh` matches the target Kubespray branch (e.g. `release-2.28`).
2. Run `source ./init.sh` to initialize the environment.
3. Apply the baseline configuration:
```sh
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
4. Verify cluster health:
```sh
kubectl get nodes -o wide
kubectl get pods -A | grep -v -E "Running|Completed"
kubectl get pods -n kube-system
```
### Phase 2: Kubernetes Upgrade to v1.32.13
1. In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update:
---
### Step 2: Upgrade Kubernetes Version (Phase 2)
1. In `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update `kube_version`:
```yaml
kube_version: 1.32.13
```
2. Re-sync inventory to ephemeral `kubespray/`:
2. Sync the configuration into the `kubespray/` workspace:
```sh
rsync -a ./inventory/ ./kubespray/inventory/
sync-inventory
```
3. Run upgrade playbook:
3. Run the in-place upgrade playbook:
```sh
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
```
4. Verify upgraded cluster:
---
### Step 3: Post-Upgrade Verification
1. **Check component versions**:
```sh
kubectl version
```
Ensure both `Client Version` and `Server Version` report `v1.32.13`.
2. **Verify Node status**:
```sh
kubectl get nodes -o wide
```
Ensure status is `Ready` and version matches `v1.32.13`.
3. **Verify System Pods**:
```sh
kubectl get pods -n kube-system
```
All control plane components (`kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `coredns`, `kube-flannel`, `kube-proxy`, `traefik`) should be `1/1 Running`.
4. **Verify Workloads**:
```sh
kubectl get pods -A | grep -v -E "Running|Completed"
```
---
## Troubleshooting & Edge Cases
### Kubeadm Preflight `CreateJob` Failure
Kubeadm 1.31+ preflight executes an ephemeral `upgrade-health-check-...` batch job with a 15-second timeout. On single-node clusters or under load, this may exceed 15s. To prevent the upgrade from halting:
- `CreateJob` is included in `kubeadm_ignore_preflight_errors` in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
- The patch `patches/kubeadm-upgrade-ignore-preflight.patch` ensures Kubespray's `v1beta4` kubeadm configuration propagates this flag during `upgrade apply`.
### Kubelet Housekeeping Stalls / Static Pod Timeout
If `kubeadm upgrade apply` times out waiting for static pod hash changes:
- Check `journalctl -u kubelet -n 100` for `Housekeeping took longer than expected` or `orphaned pod ... failed to rmdir()`.
- If an old container unmount left a stale bind-mount busy, lazily detach it:
```sh
sudo umount -l /var/lib/kubelet/pods/<orphaned-pod-uid>/volumes/...
sudo systemctl restart kubelet
```
---
## Directory Layout
* `init.sh`: Environment bootstrapping script and upgrade instructions.
* `inventory/`: Source-of-truth inventory and variable files.
* `patches/`: Git patches applied to upstream Kubespray on checkout.
* `kubespray/`: (Ephemeral) Cloned Kubespray repository.
bootstrap/init.sh
+64 -51
View File
@@ -1,65 +1,78 @@
#!/bin/sh
# check whether script is sourced
[[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "script is executed, but needs to be sourced" && exit 1
# Ensure script is sourced, not executed directly
[[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "Error: init.sh must be sourced: source ./init.sh (or . ./init.sh)" && exit 1
echo "######################################################################################"
echo "## Reinit repository"
rm -rf kubespray
VERSION="release-2.28"
git clone --branch $VERSION https://github.com/kubernetes-sigs/kubespray.git
if [ -d patches ]; then
for p in patches/*.patch; do
[ -f "$p" ] && (echo "## Applying patch $p" && git -C kubespray apply "../$p")
BOOTSTRAP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VERSION="release-2.28" # Note: release-2.28 is the final release line supporting Ubuntu 20.04 LTS
echo "======================================================================================"
echo "## [1/4] Checking / Cloning Kubespray ($VERSION)"
echo "======================================================================================"
if [ ! -d "$BOOTSTRAP_DIR/kubespray/.git" ]; then
rm -rf "$BOOTSTRAP_DIR/kubespray"
git clone --branch "$VERSION" --depth 1 https://github.com/kubernetes-sigs/kubespray.git "$BOOTSTRAP_DIR/kubespray"
else
echo "Existing kubespray checkout detected. Ensuring correct branch..."
git -C "$BOOTSTRAP_DIR/kubespray" fetch origin "$VERSION" --depth 1
git -C "$BOOTSTRAP_DIR/kubespray" checkout "$VERSION"
fi
if [ -d "$BOOTSTRAP_DIR/patches" ]; then
for p in "$BOOTSTRAP_DIR"/patches/*.patch; do
if [ -f "$p" ]; then
echo "## Applying patch $(basename "$p")..."
git -C "$BOOTSTRAP_DIR/kubespray" apply --check "$p" 2>/dev/null && git -C "$BOOTSTRAP_DIR/kubespray" apply "$p" || echo "Patch $(basename "$p") already applied or skipped."
fi
done
fi
echo "######################################################################################"
echo "## Activating pyenv venv"
echo "======================================================================================"
echo "## [2/4] Activating Python Virtualenv"
echo "======================================================================================"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
pyenv virtualenvs --bare | grep -q "^infrapuzzle-bootstrap$" || pyenv virtualenv 3.10.8 infrapuzzle-bootstrap
pyenv activate infrapuzzle-bootstrap
python -m pip install -r kubespray/requirements.txt
python -m pip install -q -r "$BOOTSTRAP_DIR/kubespray/requirements.txt"
echo "======================================================================================"
echo "## [3/4] Syncing Inventory Configuration"
echo "======================================================================================"
rsync -a "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/"
echo "######################################################################################"
echo "## Customizing vagrant dev env"
mkdir -p kubespray/vagrant
cat << EOF > kubespray/vagrant/config.rb
\$instance_name_prefix = "k8s"
\$vm_cpus = 4
\$num_instances = 1
\$os = "centos"
\$subnet = "10.0.20"
\$network_plugin = "calico"
\$shared_folders = { 'temp/docker_rpms' => "/var/cache/yum/x86_64/7/docker-ce/packages" }
\$kube_node_instances_with_disks_number = 0
EOF
# Helper function for re-syncing changes made in ./inventory to ephemeral ./kubespray/inventory
sync-inventory() {
rsync -av "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/"
}
export -f sync-inventory 2>/dev/null || true
alias sync-inventory="rsync -av '$BOOTSTRAP_DIR/inventory/' '$BOOTSTRAP_DIR/kubespray/inventory/'"
# make the rpm cache
mkdir -p kubespray/temp/docker_rpms
echo "###############"
echo "Now cd to kubespray and 'vagrant up'"
echo "======================================================================================"
echo "## [4/4] Environment Ready! Single-Node Cluster Upgrade Workflow"
echo "======================================================================================"
echo ""
echo "export KUBECONFIG=\"$( pwd )/kubespray/inventory/sample/artifacts/admin.conf\""
echo "######################################################################################"
echo "## * syncing config in kubespray dir"
rsync -a ./inventory/ ./kubespray/inventory/
echo "## * changing into kubespray dir"
cd kubespray
echo "## * Step 1: Reconcile Kubespray baseline (keep current k8s version):"
echo "ansible-playbook -i inventory/prod/inventory.ini cluster.yml"
echo "## * Step 2: Once verified, bump kube_version in inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml and run:"
echo "ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml"
# cp -r kubespray/inventory/sample kubespray/inventory/prod
# rm kubespray/inventory/prod/inventory.ini
# cp ./prod.ini kubespray/inventory/prod/inventory.ini
#gsed -i "s/kube_network_plugin: .*/kube_network_plugin: flannel/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml
# gsed -i "s/# calico_iptables_backend: \"Legacy\"/calico_iptables_backend: \"NFT\"/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-calico.yml
#echo 'calico_iptables_backend: "NFT"' >> ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-cluster.yml
#gsed -i "s/metrics_server_enabled: .*/metrics_server_enabled: true/" ./kubespray/inventory/prod/group_vars/k8s-cluster/addons.yml
echo "Current directory changed to: $BOOTSTRAP_DIR/kubespray"
cd "$BOOTSTRAP_DIR/kubespray"
echo ""
echo "=== PRE-UPGRADE CHECKLIST ==="
echo "1. Verify SSH connection and sudo access:"
echo " ssh moritz@136.243.23.215 'sudo kubectl get nodes -o wide'"
echo "2. Take manual etcd snapshot before upgrading:"
echo " ssh moritz@136.243.23.215 'sudo etcdctl --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db'"
echo ""
echo "=== TWO-PHASE UPGRADE WORKFLOW ==="
echo "Phase 1: Reconcile Kubespray baseline (keeps current k8s version):"
echo " ansible-playbook -i inventory/prod/inventory.ini cluster.yml"
echo ""
echo "Phase 2: Upgrade Kubernetes version (e.g. to 1.32.13):"
echo " a) Update 'kube_version: <target>' in ../inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml"
echo " b) Sync inventory: sync-inventory"
echo " c) Run in-place single-node upgrade:"
echo " ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml"
echo ""
echo "=== POST-UPGRADE VERIFICATION ==="
echo " kubectl version"
echo " kubectl get nodes -o wide"
echo " kubectl get pods -A | grep -v -E 'Running|Completed'"
echo "======================================================================================"