moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(bootstrap): document single-node cluster upgrade procedures and modernize init.sh

Browse Source 
- Update init.sh with smart checkout, patch application, sync-inventory alias, and runbook output
- Rewrite bootstrap/README.md with step-by-step upgrade guide, architectural invariants, and troubleshooting
- Update bootstrap/AGENTS.md with single-node operational rules and OS constraints
This commit is contained in:
Moritz Graf
2026-09-19 22:58:22 +02:00
parent 87649e6e22
commit 80a056f720
3 changed files with 264 additions and 215 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bootstrap/AGENTS.md
+71 -38
View File
@@ -1,50 +1,83 @@
# AGENTS.md # AGENTS.md
> [!NOTE] > [!NOTE]
> This directory handles the **bootstrapping and provisioning** of the Haumdaucher Kubernetes cluster using **Kubespray**. > This directory handles the **bootstrapping, provisioning, and in-place upgrades** of the Haumdaucher Kubernetes cluster using **Kubespray**.
## Project Overview ## Project Overview
* **Tool**: [Kubespray](https://github.com/kubernetes-sigs/kubespray) (Ansible-based). * **Tool**: [Kubespray](https://github.com/kubernetes-sigs/kubespray) (Ansible-based).
* **Method**: The local `inventory/` is the source of truth, which is synced into a checked-out Kubespray repository. * **Topology**: **Single-node cluster** (`haumdaucher`, `136.243.23.215`). The single node acts simultaneously as control plane, etcd member, and worker.
* **Idempotency**: The process is designed to be repeatable. The `kubespray` folder is treated as ephemeral and is re-created by `init.sh`. * **Host OS**: Ubuntu 20.04.6 LTS.
* **Method**: The local `inventory/` is the source of truth, synced into the checked-out Kubespray repository.
* **Idempotency & Patching**: The `kubespray/` directory is ephemeral and managed by `init.sh`. Any essential Kubespray fixes are persisted in `patches/` and applied automatically.
---
## Architectural Invariants for Single-Node Upgrades
> [!IMPORTANT]
> The single-node topology imposes strict operational constraints that must be observed by all agents and operators:
1. **NO DRAIN / NO CORDON**:
- Draining or cordoning a single-node cluster evicts all workloads with nowhere to schedule them, causing cascading failures and pending pods.
- `drain_nodes: false`, `upgrade_node_always_cordon: false`, and `upgrade_node_fail_if_drain_fails: false` MUST remain configured in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
- Use the in-place upgrade mechanism: `ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml`. This invokes `kubeadm-upgrade.yml` without cordon/drain overhead.
2. **VERSION STRING FORMAT**:
- In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, `kube_version` must be specified **without** the leading `v` (e.g., `1.32.13`, NOT `v1.32.13`), as Kubespray 2.28 interpolates `v{{ kube_version }}` in download URLs and package names.
3. **UBUNTU 20.04 OS CONSTRAINT**:
- Kubespray `release-2.28` is the **final release line** supporting Ubuntu 20.04 LTS. Upstream `release-2.29`+ drops Ubuntu 20.04 support. Do NOT bump `VERSION` in `init.sh` beyond `release-2.28` until the host operating system has been upgraded to Ubuntu 22.04 or 24.04.
4. **ETCD BACKUPS**:
- Always take a manual etcd snapshot before initiating any cluster upgrade.
---
## Workflow & Scripts ## Workflow & Scripts
The core workflow is encapsulated in `init.sh`.
### `init.sh` ### 1. Source `init.sh`
**Purpose**: Prepares the environment and Kubespray for deployment. ```bash
**Actions**: source init.sh
1. **Clean Slate**: Deletes existing `kubespray/` directory. ```
2. **Clone**: Clones Kubespray (version defined in variable `VERSION`, e.g., `release-2.28`). **Actions performed**:
3. **Environment**: Sets up Python virtualenv via `pyenv` and installs `requirements.txt`. - Clones / checks out the Kubespray branch defined by `VERSION` (`release-2.28`).
4. **Sync**: Copies local `./inventory/` configurations into `./kubespray/inventory/`. - Applies monorepo patches in `patches/*.patch` (e.g., propagating `--ignore-preflight-errors` for `v1beta4` kubeadm configs).
- Activates the `infrapuzzle-bootstrap` pyenv virtualenv and verifies Python requirements.
- Syncs `./inventory/` to `./kubespray/inventory/`.
- Exports the `sync-inventory` alias.
- Changes working directory to `./kubespray`.
### Usage ### 2. Upgrading the Single-Node Cluster (Two-Phase Decoupled Workflow)
1. **Source the script**:
```bash #### Phase 1: Reconcile Kubespray Tooling Baseline (Keep existing k8s version)
source init.sh ```bash
``` cd kubespray
2. **Deploy / Upgrade**: ansible-playbook -i inventory/prod/inventory.ini cluster.yml
After sourcing, go to the `kubespray` directory and run the Ansible playbooks as instructed by the script output. ```
* **Phase 1: Reconcile Kubespray baseline (stay on current k8s version)**: *Verification*: Check node readiness and that no application pods crashed.
```bash
cd kubespray #### Phase 2: Upgrade Kubernetes Version (e.g. to 1.32.13)
ansible-playbook -i inventory/prod/inventory.ini cluster.yml 1. Edit `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`:
``` ```yaml
* **Phase 2: Upgrade Kubernetes version (e.g. to v1.32.13)**: kube_version: 1.32.13
Update `kube_version` in `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, re-sync, and run: ```
```bash 2. Re-sync inventory from `bootstrap/` or `bootstrap/kubespray/`:
cd kubespray ```bash
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml sync-inventory
``` ```
3. Run the in-place upgrade:
```bash
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
```
4. Verify the cluster:
```bash
kubectl version
kubectl get nodes -o wide
kubectl get pods -A | grep -v -E "Running|Completed"
```
---
## Directory Structure ## Directory Structure
* `init.sh`: The entry point script. **Source of truth for Kubespray version.** * `init.sh`: The entrypoint script (sources environment, applies patches, sets up inventory).
* `inventory/`: Contains cluster inventory configurations (hosts, variables). **Edit this, not the one in `kubespray/`**. * `patches/`: Monorepo patches applied automatically to ephemeral `kubespray/`.
* `kubespray/`: (Ignored/Ephemeral) The checked-out Kubespray repository. **Do not edit files here directly**; they will be overwritten. * `inventory/`: Source of truth for cluster inventory and variables.
* `kubespray/`: (Ephemeral / Gitignored) Cloned Kubespray repository.
## Configuration Updates
To upgrade Kubespray or change cluster config:
1. **Version Upgrade**: Update `VERSION` in `init.sh` (e.g., to `release-2.28`).
2. **Config Changes**: Modify files in `./inventory/`.
3. **Apply**: Run `source init.sh` then execute the Ansible playbook.
bootstrap/README.md
+129 -126
View File
@@ -1,162 +1,165 @@
# Bootstrap # Bootstrap
The following lines document how to initalize a fresh cluster. On a real cluster, or using Vagrant. It therefore assumes to clone kubespray to this folder. It will be excluded in *.gitignore* and all files are in this folder. This directory contains the automation, configurations, and playbooks used to provision and upgrade the single-node Kubernetes cluster **`haumdaucher`** (`136.243.23.215`) using **[Kubespray](https://github.com/kubernetes-sigs/kubespray)**.
The version to be initialized in configured in the `init.sh` script. ---
## Architecture & Single-Node Cluster Constraints
`haumdaucher` is a **single-node cluster** (control plane, etcd, and worker all run on the same machine). This topology requires specific configuration choices during cluster maintenance:
> [!IMPORTANT]
> ### Crucial Single-Node Invariants
> 1. **Never Cordon or Drain the Node**:
> Draining a single-node cluster evicts all workloads with nowhere to schedule them, causing pods to stay in `Pending` state and breaking critical cluster services (such as DNS, CNI, ingress, and local storage).
> In [inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml), the following settings are enforced:
> ```yaml
> drain_nodes: false
> upgrade_node_always_cordon: false
> upgrade_node_fail_if_drain_fails: false
> ```
> 2. **In-Place Upgrades via `cluster.yml`**:
> Instead of `upgrade-cluster.yml` (which defaults to draining nodes), single-node upgrades should use the in-place upgrade mechanism:
> ```sh
> ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
> ```
> 3. **Version Number Formatting**:
> Kubespray 2.28 interpolates `v{{ kube_version }}` across many internal tasks. Therefore, `kube_version` in `k8s-cluster.yml` MUST NOT include a leading `v`:
> ```yaml
> kube_version: 1.32.13 # Correct
> # kube_version: v1.32.13 # INCORRECT (leads to v1.32.13 -> vv1.32.13 errors)
> ```
> 4. **Host OS Lifecycle Constraint (Ubuntu 20.04 LTS)**:
> Kubespray `release-2.28` is the **final release line supporting Ubuntu 20.04**. Kubespray `release-2.29`+ drops Ubuntu 20.04 support and requires Ubuntu 22.04 or 24.04. Do not bump `VERSION` in `init.sh` beyond `release-2.28` until the host OS is upgraded.
---
## Quickstart
### 1. Environment Initialization
Source `init.sh` from this directory. It clones Kubespray, applies patches from `patches/`, configures the Python virtualenv, and syncs `./inventory/` into `./kubespray/inventory/`:
```sh ```sh
. ./init.sh source ./init.sh
``` ```
See [kubespray.io](https://kubespray.io/) on detailed information about kubespray. Though it seems to be a littlebit outdated. *(Note: `init.sh` also exports the `sync-inventory` alias to quickly sync local `./inventory/` modifications into `./kubespray/inventory/` without re-cloning).*
## Vagrant ---
```sh ## Step-by-Step Upgrade Runbook
cd kubespray
vagrant up
# up and abkle to ssh
vagrant ssh k8s-1
```
## Prod Upgrades are conducted in two decoupled phases:
1. **Phase 1**: Upgrade the Kubespray tooling baseline while keeping the Kubernetes version unchanged $\rightarrow$ Apply $\rightarrow$ Verify.
2. **Phase 2**: Upgrade Kubernetes components (API server, controller manager, scheduler, kubelet, kube-proxy) $\rightarrow$ Apply $\rightarrow$ Verify.
Prepare server: ### Step 0: Pre-Flight Checks & Backups
* deactivate swap!
* `moritz username ALL=(ALL) NOPASSWD:ALL`
* `
```sh 1. **Verify SSH and sudo access**:
ssh centos@<ip>
# auth via pw
sudo su - root
adduser moritz
visudo # add as sudo user
su - moritz
sudo yum -y install vim python3
ssh-keygen
vim .ssh/authorized_users # paste key
chmod 644 .ssh/authorized_keys
# check whether login works with ssh key
sudo vim /etc/ssh/sshd_config # remove pw auth & root login
sudo yum upgrade -y && sudo reboot
```
Install Kubernetes:
```sh
. ./init.sh
# follow instructions from output, sth like:
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
And get credentials:
```sh
ssh <ip>
sudo su - root
cd
cp -r .kube /home/moritz/
chown -R moritz. /home/moritz/.kube
#ctrl + d
kubectl get ns # test connection
#ctrl + d
scp haumdaucher.de:/home/moritz/.kube/config .kube/config
```
Foreward in k8s-directory.
## Upgrade cluster
Check the current default value of `kube_version` in cloned repository.
**FOR K8S VERSION - Edit /Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml**
```sh
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e kube_version=v1.22.10 -e upgrade_cluster_setup=true cluster.yml
# or just the newest version
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
# upgrade to specific calico version (did not trigger/ failed)
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e calico_version=v3.15.2 cluster.yml --tags=network
```
History:
* 2020-04-18 kube_version=v1.16.8 kubespray_branch=release-2.12
## Add node
See [documentation](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/nodes.md).
Note: This was more or less a trial and error approach. Running different playbooks over and over again got it right at some point.
```sh
ansible-playbook -i inventory/prod/inventory.ini --limit=ns3088070.ip-37-59-40.eu,ns3100058.ip-37-59-61.eu scale.yml
ansible-playbook -i inventory/prod/inventory.ini --limit=etcd,kube-master -e ignore_assert_errors=yes cluster.yml
```
This runs everything and is kind of idempotent:
```sh
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```
## Upgrade to Kubernetes 1.31 (Previous)
Required to execute:
```sh
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e drain_nodes=false upgrade-cluster.yml
```
+ set a feature flag: https://github.com/kubernetes-sigs/kubespray/issues/11887
## Upgrade to Kubespray release-2.28 & Kubernetes v1.32.13
> [!NOTE]
> Upgrade philosophy: decoupled two-phase upgrade.
> 1. Upgrade Kubespray tooling baseline (keeping k8s version unchanged).
> 2. Upgrade Kubernetes control plane and worker components.
>
> **Important**:
> - `haumdaucher` is a **single-node cluster**. Always use `-e drain_nodes=false` when running `upgrade-cluster.yml` to prevent the upgrade from hanging on draining.
> - **Ubuntu 20.04 LTS constraint**: Kubespray `release-2.28` is the **last** release officially supporting Ubuntu 20.04. Upstream Kubespray `release-2.29`+ requires Ubuntu 22.04 or 24.04.
### Phase 1: Kubespray Baseline Upgrade (Retaining k8s v1.31.9)
1. Ensure `init.sh` has `VERSION="release-2.28"`.
2. Initialize environment:
```sh ```sh
. ./init.sh ssh moritz@136.243.23.215 "sudo kubectl get nodes -o wide"
``` ```
3. Run cluster playbook to reconcile Kubespray baseline:
2. **Check Pod Health**:
```sh ```sh
ssh moritz@136.243.23.215 "sudo kubectl get pods -A | grep -v -E 'Running|Completed'"
```
3. **Take a Manual etcd Snapshot**:
```sh
ssh moritz@136.243.23.215 "sudo etcdctl \
--cacert=/etc/ssl/etcd/ssl/ca.pem \
--cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem \
--key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem \
snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db"
```
---
### Step 1: Reconcile Kubespray Baseline (Phase 1)
1. Verify `VERSION` in `init.sh` matches the target Kubespray branch (e.g. `release-2.28`).
2. Run `source ./init.sh` to initialize the environment.
3. Apply the baseline configuration:
```sh
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml ansible-playbook -i inventory/prod/inventory.ini cluster.yml
``` ```
4. Verify cluster health: 4. Verify cluster health:
```sh ```sh
kubectl get nodes -o wide kubectl get nodes -o wide
kubectl get pods -A | grep -v -E "Running|Completed" kubectl get pods -n kube-system
``` ```
### Phase 2: Kubernetes Upgrade to v1.32.13 ---
1. In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update:
### Step 2: Upgrade Kubernetes Version (Phase 2)
1. In `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update `kube_version`:
```yaml ```yaml
kube_version: 1.32.13 kube_version: 1.32.13
``` ```
2. Re-sync inventory to ephemeral `kubespray/`:
2. Sync the configuration into the `kubespray/` workspace:
```sh ```sh
rsync -a ./inventory/ ./kubespray/inventory/ sync-inventory
``` ```
3. Run upgrade playbook:
3. Run the in-place upgrade playbook:
```sh ```sh
cd kubespray cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
``` ```
4. Verify upgraded cluster:
---
### Step 3: Post-Upgrade Verification
1. **Check component versions**:
```sh ```sh
kubectl version kubectl version
```
Ensure both `Client Version` and `Server Version` report `v1.32.13`.
2. **Verify Node status**:
```sh
kubectl get nodes -o wide kubectl get nodes -o wide
```
Ensure status is `Ready` and version matches `v1.32.13`.
3. **Verify System Pods**:
```sh
kubectl get pods -n kube-system
```
All control plane components (`kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `coredns`, `kube-flannel`, `kube-proxy`, `traefik`) should be `1/1 Running`.
4. **Verify Workloads**:
```sh
kubectl get pods -A | grep -v -E "Running|Completed" kubectl get pods -A | grep -v -E "Running|Completed"
``` ```
---
## Troubleshooting & Edge Cases
### Kubeadm Preflight `CreateJob` Failure
Kubeadm 1.31+ preflight executes an ephemeral `upgrade-health-check-...` batch job with a 15-second timeout. On single-node clusters or under load, this may exceed 15s. To prevent the upgrade from halting:
- `CreateJob` is included in `kubeadm_ignore_preflight_errors` in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
- The patch `patches/kubeadm-upgrade-ignore-preflight.patch` ensures Kubespray's `v1beta4` kubeadm configuration propagates this flag during `upgrade apply`.
### Kubelet Housekeeping Stalls / Static Pod Timeout
If `kubeadm upgrade apply` times out waiting for static pod hash changes:
- Check `journalctl -u kubelet -n 100` for `Housekeeping took longer than expected` or `orphaned pod ... failed to rmdir()`.
- If an old container unmount left a stale bind-mount busy, lazily detach it:
```sh
sudo umount -l /var/lib/kubelet/pods/<orphaned-pod-uid>/volumes/...
sudo systemctl restart kubelet
```
---
## Directory Layout
* `init.sh`: Environment bootstrapping script and upgrade instructions.
* `inventory/`: Source-of-truth inventory and variable files.
* `patches/`: Git patches applied to upstream Kubespray on checkout.
* `kubespray/`: (Ephemeral) Cloned Kubespray repository.
bootstrap/init.sh
+64 -51
View File
@@ -1,65 +1,78 @@
#!/bin/sh #!/bin/sh
# check whether script is sourced # Ensure script is sourced, not executed directly
[[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "script is executed, but needs to be sourced" && exit 1 [[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "Error: init.sh must be sourced: source ./init.sh (or . ./init.sh)" && exit 1
echo "######################################################################################" BOOTSTRAP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
echo "## Reinit repository" VERSION="release-2.28" # Note: release-2.28 is the final release line supporting Ubuntu 20.04 LTS
rm -rf kubespray
VERSION="release-2.28" echo "======================================================================================"
git clone --branch $VERSION https://github.com/kubernetes-sigs/kubespray.git echo "## [1/4] Checking / Cloning Kubespray ($VERSION)"
if [ -d patches ]; then echo "======================================================================================"
for p in patches/*.patch; do if [ ! -d "$BOOTSTRAP_DIR/kubespray/.git" ]; then
[ -f "$p" ] && (echo "## Applying patch $p" && git -C kubespray apply "../$p") rm -rf "$BOOTSTRAP_DIR/kubespray"
git clone --branch "$VERSION" --depth 1 https://github.com/kubernetes-sigs/kubespray.git "$BOOTSTRAP_DIR/kubespray"
else
echo "Existing kubespray checkout detected. Ensuring correct branch..."
git -C "$BOOTSTRAP_DIR/kubespray" fetch origin "$VERSION" --depth 1
git -C "$BOOTSTRAP_DIR/kubespray" checkout "$VERSION"
fi
if [ -d "$BOOTSTRAP_DIR/patches" ]; then
for p in "$BOOTSTRAP_DIR"/patches/*.patch; do
if [ -f "$p" ]; then
echo "## Applying patch $(basename "$p")..."
git -C "$BOOTSTRAP_DIR/kubespray" apply --check "$p" 2>/dev/null && git -C "$BOOTSTRAP_DIR/kubespray" apply "$p" || echo "Patch $(basename "$p") already applied or skipped."
fi
done done
fi fi
echo "######################################################################################" echo "======================================================================================"
echo "## Activating pyenv venv" echo "## [2/4] Activating Python Virtualenv"
echo "======================================================================================"
eval "$(pyenv init -)" eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)" eval "$(pyenv virtualenv-init -)"
pyenv virtualenvs --bare | grep -q "^infrapuzzle-bootstrap$" || pyenv virtualenv 3.10.8 infrapuzzle-bootstrap pyenv virtualenvs --bare | grep -q "^infrapuzzle-bootstrap$" || pyenv virtualenv 3.10.8 infrapuzzle-bootstrap
pyenv activate infrapuzzle-bootstrap pyenv activate infrapuzzle-bootstrap
python -m pip install -r kubespray/requirements.txt python -m pip install -q -r "$BOOTSTRAP_DIR/kubespray/requirements.txt"
echo "======================================================================================"
echo "## [3/4] Syncing Inventory Configuration"
echo "======================================================================================"
rsync -a "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/"
echo "######################################################################################" # Helper function for re-syncing changes made in ./inventory to ephemeral ./kubespray/inventory
echo "## Customizing vagrant dev env" sync-inventory() {
mkdir -p kubespray/vagrant rsync -av "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/"
cat << EOF > kubespray/vagrant/config.rb }
\$instance_name_prefix = "k8s" export -f sync-inventory 2>/dev/null || true
\$vm_cpus = 4 alias sync-inventory="rsync -av '$BOOTSTRAP_DIR/inventory/' '$BOOTSTRAP_DIR/kubespray/inventory/'"
\$num_instances = 1
\$os = "centos"
\$subnet = "10.0.20"
\$network_plugin = "calico"
\$shared_folders = { 'temp/docker_rpms' => "/var/cache/yum/x86_64/7/docker-ce/packages" }
\$kube_node_instances_with_disks_number = 0
EOF
# make the rpm cache echo "======================================================================================"
mkdir -p kubespray/temp/docker_rpms echo "## [4/4] Environment Ready! Single-Node Cluster Upgrade Workflow"
echo "======================================================================================"
echo "###############"
echo "Now cd to kubespray and 'vagrant up'"
echo "" echo ""
echo "export KUBECONFIG=\"$( pwd )/kubespray/inventory/sample/artifacts/admin.conf\"" echo "Current directory changed to: $BOOTSTRAP_DIR/kubespray"
cd "$BOOTSTRAP_DIR/kubespray"
echo ""
echo "######################################################################################" echo "=== PRE-UPGRADE CHECKLIST ==="
echo "## * syncing config in kubespray dir" echo "1. Verify SSH connection and sudo access:"
rsync -a ./inventory/ ./kubespray/inventory/ echo " ssh moritz@136.243.23.215 'sudo kubectl get nodes -o wide'"
echo "## * changing into kubespray dir" echo "2. Take manual etcd snapshot before upgrading:"
cd kubespray echo " ssh moritz@136.243.23.215 'sudo etcdctl --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db'"
echo "## * Step 1: Reconcile Kubespray baseline (keep current k8s version):" echo ""
echo "ansible-playbook -i inventory/prod/inventory.ini cluster.yml" echo "=== TWO-PHASE UPGRADE WORKFLOW ==="
echo "## * Step 2: Once verified, bump kube_version in inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml and run:" echo "Phase 1: Reconcile Kubespray baseline (keeps current k8s version):"
echo "ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml" echo " ansible-playbook -i inventory/prod/inventory.ini cluster.yml"
echo ""
# cp -r kubespray/inventory/sample kubespray/inventory/prod echo "Phase 2: Upgrade Kubernetes version (e.g. to 1.32.13):"
# rm kubespray/inventory/prod/inventory.ini echo " a) Update 'kube_version: <target>' in ../inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml"
# cp ./prod.ini kubespray/inventory/prod/inventory.ini echo " b) Sync inventory: sync-inventory"
#gsed -i "s/kube_network_plugin: .*/kube_network_plugin: flannel/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml echo " c) Run in-place single-node upgrade:"
# gsed -i "s/# calico_iptables_backend: \"Legacy\"/calico_iptables_backend: \"NFT\"/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-calico.yml echo " ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml"
#echo 'calico_iptables_backend: "NFT"' >> ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-cluster.yml echo ""
#gsed -i "s/metrics_server_enabled: .*/metrics_server_enabled: true/" ./kubespray/inventory/prod/group_vars/k8s-cluster/addons.yml echo "=== POST-UPGRADE VERIFICATION ==="
echo " kubectl version"
echo " kubectl get nodes -o wide"
echo " kubectl get pods -A | grep -v -E 'Running|Completed'"
echo "======================================================================================"