diff --git a/bootstrap/AGENTS.md b/bootstrap/AGENTS.md index de2d7a3..d76dada 100644 --- a/bootstrap/AGENTS.md +++ b/bootstrap/AGENTS.md @@ -1,50 +1,83 @@ # AGENTS.md > [!NOTE] -> This directory handles the **bootstrapping and provisioning** of the Haumdaucher Kubernetes cluster using **Kubespray**. +> This directory handles the **bootstrapping, provisioning, and in-place upgrades** of the Haumdaucher Kubernetes cluster using **Kubespray**. ## Project Overview * **Tool**: [Kubespray](https://github.com/kubernetes-sigs/kubespray) (Ansible-based). -* **Method**: The local `inventory/` is the source of truth, which is synced into a checked-out Kubespray repository. -* **Idempotency**: The process is designed to be repeatable. The `kubespray` folder is treated as ephemeral and is re-created by `init.sh`. +* **Topology**: **Single-node cluster** (`haumdaucher`, `136.243.23.215`). The single node acts simultaneously as control plane, etcd member, and worker. +* **Host OS**: Ubuntu 20.04.6 LTS. +* **Method**: The local `inventory/` is the source of truth, synced into the checked-out Kubespray repository. +* **Idempotency & Patching**: The `kubespray/` directory is ephemeral and managed by `init.sh`. Any essential Kubespray fixes are persisted in `patches/` and applied automatically. + +--- + +## Architectural Invariants for Single-Node Upgrades + +> [!IMPORTANT] +> The single-node topology imposes strict operational constraints that must be observed by all agents and operators: + +1. **NO DRAIN / NO CORDON**: + - Draining or cordoning a single-node cluster evicts all workloads with nowhere to schedule them, causing cascading failures and pending pods. + - `drain_nodes: false`, `upgrade_node_always_cordon: false`, and `upgrade_node_fail_if_drain_fails: false` MUST remain configured in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml). + - Use the in-place upgrade mechanism: `ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml`. This invokes `kubeadm-upgrade.yml` without cordon/drain overhead. +2. **VERSION STRING FORMAT**: + - In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, `kube_version` must be specified **without** the leading `v` (e.g., `1.32.13`, NOT `v1.32.13`), as Kubespray 2.28 interpolates `v{{ kube_version }}` in download URLs and package names. +3. **UBUNTU 20.04 OS CONSTRAINT**: + - Kubespray `release-2.28` is the **final release line** supporting Ubuntu 20.04 LTS. Upstream `release-2.29`+ drops Ubuntu 20.04 support. Do NOT bump `VERSION` in `init.sh` beyond `release-2.28` until the host operating system has been upgraded to Ubuntu 22.04 or 24.04. +4. **ETCD BACKUPS**: + - Always take a manual etcd snapshot before initiating any cluster upgrade. + +--- ## Workflow & Scripts -The core workflow is encapsulated in `init.sh`. -### `init.sh` -**Purpose**: Prepares the environment and Kubespray for deployment. -**Actions**: -1. **Clean Slate**: Deletes existing `kubespray/` directory. -2. **Clone**: Clones Kubespray (version defined in variable `VERSION`, e.g., `release-2.28`). -3. **Environment**: Sets up Python virtualenv via `pyenv` and installs `requirements.txt`. -4. **Sync**: Copies local `./inventory/` configurations into `./kubespray/inventory/`. +### 1. Source `init.sh` +```bash +source init.sh +``` +**Actions performed**: +- Clones / checks out the Kubespray branch defined by `VERSION` (`release-2.28`). +- Applies monorepo patches in `patches/*.patch` (e.g., propagating `--ignore-preflight-errors` for `v1beta4` kubeadm configs). +- Activates the `infrapuzzle-bootstrap` pyenv virtualenv and verifies Python requirements. +- Syncs `./inventory/` to `./kubespray/inventory/`. +- Exports the `sync-inventory` alias. +- Changes working directory to `./kubespray`. -### Usage -1. **Source the script**: - ```bash - source init.sh - ``` -2. **Deploy / Upgrade**: - After sourcing, go to the `kubespray` directory and run the Ansible playbooks as instructed by the script output. - * **Phase 1: Reconcile Kubespray baseline (stay on current k8s version)**: - ```bash - cd kubespray - ansible-playbook -i inventory/prod/inventory.ini cluster.yml - ``` - * **Phase 2: Upgrade Kubernetes version (e.g. to v1.32.13)**: - Update `kube_version` in `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, re-sync, and run: - ```bash - cd kubespray - ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml - ``` +### 2. Upgrading the Single-Node Cluster (Two-Phase Decoupled Workflow) + +#### Phase 1: Reconcile Kubespray Tooling Baseline (Keep existing k8s version) +```bash +cd kubespray +ansible-playbook -i inventory/prod/inventory.ini cluster.yml +``` +*Verification*: Check node readiness and that no application pods crashed. + +#### Phase 2: Upgrade Kubernetes Version (e.g. to 1.32.13) +1. Edit `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`: + ```yaml + kube_version: 1.32.13 + ``` +2. Re-sync inventory from `bootstrap/` or `bootstrap/kubespray/`: + ```bash + sync-inventory + ``` +3. Run the in-place upgrade: + ```bash + cd kubespray + ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml + ``` +4. Verify the cluster: + ```bash + kubectl version + kubectl get nodes -o wide + kubectl get pods -A | grep -v -E "Running|Completed" + ``` + +--- ## Directory Structure -* `init.sh`: The entry point script. **Source of truth for Kubespray version.** -* `inventory/`: Contains cluster inventory configurations (hosts, variables). **Edit this, not the one in `kubespray/`**. -* `kubespray/`: (Ignored/Ephemeral) The checked-out Kubespray repository. **Do not edit files here directly**; they will be overwritten. - -## Configuration Updates -To upgrade Kubespray or change cluster config: -1. **Version Upgrade**: Update `VERSION` in `init.sh` (e.g., to `release-2.28`). -2. **Config Changes**: Modify files in `./inventory/`. -3. **Apply**: Run `source init.sh` then execute the Ansible playbook. \ No newline at end of file +* `init.sh`: The entrypoint script (sources environment, applies patches, sets up inventory). +* `patches/`: Monorepo patches applied automatically to ephemeral `kubespray/`. +* `inventory/`: Source of truth for cluster inventory and variables. +* `kubespray/`: (Ephemeral / Gitignored) Cloned Kubespray repository. \ No newline at end of file diff --git a/bootstrap/README.md b/bootstrap/README.md index b99059d..0872dfb 100644 --- a/bootstrap/README.md +++ b/bootstrap/README.md @@ -1,162 +1,165 @@ # Bootstrap -The following lines document how to initalize a fresh cluster. On a real cluster, or using Vagrant. It therefore assumes to clone kubespray to this folder. It will be excluded in *.gitignore* and all files are in this folder. +This directory contains the automation, configurations, and playbooks used to provision and upgrade the single-node Kubernetes cluster **`haumdaucher`** (`136.243.23.215`) using **[Kubespray](https://github.com/kubernetes-sigs/kubespray)**. -The version to be initialized in configured in the `init.sh` script. +--- + +## Architecture & Single-Node Cluster Constraints + +`haumdaucher` is a **single-node cluster** (control plane, etcd, and worker all run on the same machine). This topology requires specific configuration choices during cluster maintenance: + +> [!IMPORTANT] +> ### Crucial Single-Node Invariants +> 1. **Never Cordon or Drain the Node**: +> Draining a single-node cluster evicts all workloads with nowhere to schedule them, causing pods to stay in `Pending` state and breaking critical cluster services (such as DNS, CNI, ingress, and local storage). +> In [inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml), the following settings are enforced: +> ```yaml +> drain_nodes: false +> upgrade_node_always_cordon: false +> upgrade_node_fail_if_drain_fails: false +> ``` +> 2. **In-Place Upgrades via `cluster.yml`**: +> Instead of `upgrade-cluster.yml` (which defaults to draining nodes), single-node upgrades should use the in-place upgrade mechanism: +> ```sh +> ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml +> ``` +> 3. **Version Number Formatting**: +> Kubespray 2.28 interpolates `v{{ kube_version }}` across many internal tasks. Therefore, `kube_version` in `k8s-cluster.yml` MUST NOT include a leading `v`: +> ```yaml +> kube_version: 1.32.13 # Correct +> # kube_version: v1.32.13 # INCORRECT (leads to v1.32.13 -> vv1.32.13 errors) +> ``` +> 4. **Host OS Lifecycle Constraint (Ubuntu 20.04 LTS)**: +> Kubespray `release-2.28` is the **final release line supporting Ubuntu 20.04**. Kubespray `release-2.29`+ drops Ubuntu 20.04 support and requires Ubuntu 22.04 or 24.04. Do not bump `VERSION` in `init.sh` beyond `release-2.28` until the host OS is upgraded. + +--- + +## Quickstart + +### 1. Environment Initialization +Source `init.sh` from this directory. It clones Kubespray, applies patches from `patches/`, configures the Python virtualenv, and syncs `./inventory/` into `./kubespray/inventory/`: ```sh -. ./init.sh +source ./init.sh ``` -See [kubespray.io](https://kubespray.io/) on detailed information about kubespray. Though it seems to be a littlebit outdated. +*(Note: `init.sh` also exports the `sync-inventory` alias to quickly sync local `./inventory/` modifications into `./kubespray/inventory/` without re-cloning).* -## Vagrant +--- -```sh -cd kubespray -vagrant up -# up and abkle to ssh -vagrant ssh k8s-1 -``` +## Step-by-Step Upgrade Runbook -## Prod +Upgrades are conducted in two decoupled phases: +1. **Phase 1**: Upgrade the Kubespray tooling baseline while keeping the Kubernetes version unchanged $\rightarrow$ Apply $\rightarrow$ Verify. +2. **Phase 2**: Upgrade Kubernetes components (API server, controller manager, scheduler, kubelet, kube-proxy) $\rightarrow$ Apply $\rightarrow$ Verify. -Prepare server: -* deactivate swap! -* `moritz username ALL=(ALL) NOPASSWD:ALL` -* ` +### Step 0: Pre-Flight Checks & Backups -```sh -ssh centos@ -# auth via pw -sudo su - root -adduser moritz -visudo # add as sudo user -su - moritz -sudo yum -y install vim python3 -ssh-keygen -vim .ssh/authorized_users # paste key -chmod 644 .ssh/authorized_keys -# check whether login works with ssh key -sudo vim /etc/ssh/sshd_config # remove pw auth & root login -sudo yum upgrade -y && sudo reboot -``` - -Install Kubernetes: - -```sh -. ./init.sh -# follow instructions from output, sth like: -cd kubespray -ansible-playbook -i inventory/prod/inventory.ini cluster.yml -``` - -And get credentials: - -```sh -ssh -sudo su - root -cd -cp -r .kube /home/moritz/ -chown -R moritz. /home/moritz/.kube -#ctrl + d -kubectl get ns # test connection -#ctrl + d -scp haumdaucher.de:/home/moritz/.kube/config .kube/config -``` - -Foreward in k8s-directory. - -## Upgrade cluster - -Check the current default value of `kube_version` in cloned repository. - -**FOR K8S VERSION - Edit /Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml** - -```sh -cd kubespray -ansible-playbook -i inventory/prod/inventory.ini -e kube_version=v1.22.10 -e upgrade_cluster_setup=true cluster.yml -# or just the newest version -ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml -# upgrade to specific calico version (did not trigger/ failed) -ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e calico_version=v3.15.2 cluster.yml --tags=network -``` - -History: - -* 2020-04-18 kube_version=v1.16.8 kubespray_branch=release-2.12 - -## Add node - -See [documentation](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/nodes.md). - -Note: This was more or less a trial and error approach. Running different playbooks over and over again got it right at some point. - -```sh -ansible-playbook -i inventory/prod/inventory.ini --limit=ns3088070.ip-37-59-40.eu,ns3100058.ip-37-59-61.eu scale.yml -ansible-playbook -i inventory/prod/inventory.ini --limit=etcd,kube-master -e ignore_assert_errors=yes cluster.yml -``` - -This runs everything and is kind of idempotent: - -```sh -ansible-playbook -i inventory/prod/inventory.ini cluster.yml -``` - -## Upgrade to Kubernetes 1.31 (Previous) - -Required to execute: - -```sh -ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e drain_nodes=false upgrade-cluster.yml -``` - -+ set a feature flag: https://github.com/kubernetes-sigs/kubespray/issues/11887 - -## Upgrade to Kubespray release-2.28 & Kubernetes v1.32.13 - -> [!NOTE] -> Upgrade philosophy: decoupled two-phase upgrade. -> 1. Upgrade Kubespray tooling baseline (keeping k8s version unchanged). -> 2. Upgrade Kubernetes control plane and worker components. -> -> **Important**: -> - `haumdaucher` is a **single-node cluster**. Always use `-e drain_nodes=false` when running `upgrade-cluster.yml` to prevent the upgrade from hanging on draining. -> - **Ubuntu 20.04 LTS constraint**: Kubespray `release-2.28` is the **last** release officially supporting Ubuntu 20.04. Upstream Kubespray `release-2.29`+ requires Ubuntu 22.04 or 24.04. - -### Phase 1: Kubespray Baseline Upgrade (Retaining k8s v1.31.9) -1. Ensure `init.sh` has `VERSION="release-2.28"`. -2. Initialize environment: +1. **Verify SSH and sudo access**: ```sh - . ./init.sh + ssh moritz@136.243.23.215 "sudo kubectl get nodes -o wide" ``` -3. Run cluster playbook to reconcile Kubespray baseline: + +2. **Check Pod Health**: ```sh + ssh moritz@136.243.23.215 "sudo kubectl get pods -A | grep -v -E 'Running|Completed'" + ``` + +3. **Take a Manual etcd Snapshot**: + ```sh + ssh moritz@136.243.23.215 "sudo etcdctl \ + --cacert=/etc/ssl/etcd/ssl/ca.pem \ + --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem \ + --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem \ + snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db" + ``` + +--- + +### Step 1: Reconcile Kubespray Baseline (Phase 1) + +1. Verify `VERSION` in `init.sh` matches the target Kubespray branch (e.g. `release-2.28`). +2. Run `source ./init.sh` to initialize the environment. +3. Apply the baseline configuration: + ```sh + cd kubespray ansible-playbook -i inventory/prod/inventory.ini cluster.yml ``` 4. Verify cluster health: ```sh kubectl get nodes -o wide - kubectl get pods -A | grep -v -E "Running|Completed" + kubectl get pods -n kube-system ``` -### Phase 2: Kubernetes Upgrade to v1.32.13 -1. In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update: +--- + +### Step 2: Upgrade Kubernetes Version (Phase 2) + +1. In `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update `kube_version`: ```yaml kube_version: 1.32.13 ``` -2. Re-sync inventory to ephemeral `kubespray/`: + +2. Sync the configuration into the `kubespray/` workspace: ```sh - rsync -a ./inventory/ ./kubespray/inventory/ + sync-inventory ``` -3. Run upgrade playbook: + +3. Run the in-place upgrade playbook: ```sh cd kubespray ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml ``` -4. Verify upgraded cluster: + +--- + +### Step 3: Post-Upgrade Verification + +1. **Check component versions**: ```sh kubectl version + ``` + Ensure both `Client Version` and `Server Version` report `v1.32.13`. + +2. **Verify Node status**: + ```sh kubectl get nodes -o wide + ``` + Ensure status is `Ready` and version matches `v1.32.13`. + +3. **Verify System Pods**: + ```sh + kubectl get pods -n kube-system + ``` + All control plane components (`kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `coredns`, `kube-flannel`, `kube-proxy`, `traefik`) should be `1/1 Running`. + +4. **Verify Workloads**: + ```sh kubectl get pods -A | grep -v -E "Running|Completed" ``` + +--- + +## Troubleshooting & Edge Cases + +### Kubeadm Preflight `CreateJob` Failure +Kubeadm 1.31+ preflight executes an ephemeral `upgrade-health-check-...` batch job with a 15-second timeout. On single-node clusters or under load, this may exceed 15s. To prevent the upgrade from halting: +- `CreateJob` is included in `kubeadm_ignore_preflight_errors` in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml). +- The patch `patches/kubeadm-upgrade-ignore-preflight.patch` ensures Kubespray's `v1beta4` kubeadm configuration propagates this flag during `upgrade apply`. + +### Kubelet Housekeeping Stalls / Static Pod Timeout +If `kubeadm upgrade apply` times out waiting for static pod hash changes: +- Check `journalctl -u kubelet -n 100` for `Housekeeping took longer than expected` or `orphaned pod ... failed to rmdir()`. +- If an old container unmount left a stale bind-mount busy, lazily detach it: + ```sh + sudo umount -l /var/lib/kubelet/pods//volumes/... + sudo systemctl restart kubelet + ``` + +--- + +## Directory Layout +* `init.sh`: Environment bootstrapping script and upgrade instructions. +* `inventory/`: Source-of-truth inventory and variable files. +* `patches/`: Git patches applied to upstream Kubespray on checkout. +* `kubespray/`: (Ephemeral) Cloned Kubespray repository. diff --git a/bootstrap/init.sh b/bootstrap/init.sh index 287ad0c..95f584e 100755 --- a/bootstrap/init.sh +++ b/bootstrap/init.sh @@ -1,65 +1,78 @@ #!/bin/sh -# check whether script is sourced -[[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "script is executed, but needs to be sourced" && exit 1 +# Ensure script is sourced, not executed directly +[[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "Error: init.sh must be sourced: source ./init.sh (or . ./init.sh)" && exit 1 -echo "######################################################################################" -echo "## Reinit repository" -rm -rf kubespray -VERSION="release-2.28" -git clone --branch $VERSION https://github.com/kubernetes-sigs/kubespray.git -if [ -d patches ]; then - for p in patches/*.patch; do - [ -f "$p" ] && (echo "## Applying patch $p" && git -C kubespray apply "../$p") +BOOTSTRAP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +VERSION="release-2.28" # Note: release-2.28 is the final release line supporting Ubuntu 20.04 LTS + +echo "======================================================================================" +echo "## [1/4] Checking / Cloning Kubespray ($VERSION)" +echo "======================================================================================" +if [ ! -d "$BOOTSTRAP_DIR/kubespray/.git" ]; then + rm -rf "$BOOTSTRAP_DIR/kubespray" + git clone --branch "$VERSION" --depth 1 https://github.com/kubernetes-sigs/kubespray.git "$BOOTSTRAP_DIR/kubespray" +else + echo "Existing kubespray checkout detected. Ensuring correct branch..." + git -C "$BOOTSTRAP_DIR/kubespray" fetch origin "$VERSION" --depth 1 + git -C "$BOOTSTRAP_DIR/kubespray" checkout "$VERSION" +fi + +if [ -d "$BOOTSTRAP_DIR/patches" ]; then + for p in "$BOOTSTRAP_DIR"/patches/*.patch; do + if [ -f "$p" ]; then + echo "## Applying patch $(basename "$p")..." + git -C "$BOOTSTRAP_DIR/kubespray" apply --check "$p" 2>/dev/null && git -C "$BOOTSTRAP_DIR/kubespray" apply "$p" || echo "Patch $(basename "$p") already applied or skipped." + fi done fi -echo "######################################################################################" -echo "## Activating pyenv venv" +echo "======================================================================================" +echo "## [2/4] Activating Python Virtualenv" +echo "======================================================================================" eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)" pyenv virtualenvs --bare | grep -q "^infrapuzzle-bootstrap$" || pyenv virtualenv 3.10.8 infrapuzzle-bootstrap pyenv activate infrapuzzle-bootstrap -python -m pip install -r kubespray/requirements.txt +python -m pip install -q -r "$BOOTSTRAP_DIR/kubespray/requirements.txt" +echo "======================================================================================" +echo "## [3/4] Syncing Inventory Configuration" +echo "======================================================================================" +rsync -a "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/" -echo "######################################################################################" -echo "## Customizing vagrant dev env" -mkdir -p kubespray/vagrant -cat << EOF > kubespray/vagrant/config.rb -\$instance_name_prefix = "k8s" -\$vm_cpus = 4 -\$num_instances = 1 -\$os = "centos" -\$subnet = "10.0.20" -\$network_plugin = "calico" -\$shared_folders = { 'temp/docker_rpms' => "/var/cache/yum/x86_64/7/docker-ce/packages" } -\$kube_node_instances_with_disks_number = 0 -EOF +# Helper function for re-syncing changes made in ./inventory to ephemeral ./kubespray/inventory +sync-inventory() { + rsync -av "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/" +} +export -f sync-inventory 2>/dev/null || true +alias sync-inventory="rsync -av '$BOOTSTRAP_DIR/inventory/' '$BOOTSTRAP_DIR/kubespray/inventory/'" -# make the rpm cache -mkdir -p kubespray/temp/docker_rpms - -echo "###############" -echo "Now cd to kubespray and 'vagrant up'" +echo "======================================================================================" +echo "## [4/4] Environment Ready! Single-Node Cluster Upgrade Workflow" +echo "======================================================================================" echo "" -echo "export KUBECONFIG=\"$( pwd )/kubespray/inventory/sample/artifacts/admin.conf\"" - - -echo "######################################################################################" -echo "## * syncing config in kubespray dir" -rsync -a ./inventory/ ./kubespray/inventory/ -echo "## * changing into kubespray dir" -cd kubespray -echo "## * Step 1: Reconcile Kubespray baseline (keep current k8s version):" -echo "ansible-playbook -i inventory/prod/inventory.ini cluster.yml" -echo "## * Step 2: Once verified, bump kube_version in inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml and run:" -echo "ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml" - -# cp -r kubespray/inventory/sample kubespray/inventory/prod -# rm kubespray/inventory/prod/inventory.ini -# cp ./prod.ini kubespray/inventory/prod/inventory.ini -#gsed -i "s/kube_network_plugin: .*/kube_network_plugin: flannel/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml -# gsed -i "s/# calico_iptables_backend: \"Legacy\"/calico_iptables_backend: \"NFT\"/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-calico.yml -#echo 'calico_iptables_backend: "NFT"' >> ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-cluster.yml -#gsed -i "s/metrics_server_enabled: .*/metrics_server_enabled: true/" ./kubespray/inventory/prod/group_vars/k8s-cluster/addons.yml +echo "Current directory changed to: $BOOTSTRAP_DIR/kubespray" +cd "$BOOTSTRAP_DIR/kubespray" +echo "" +echo "=== PRE-UPGRADE CHECKLIST ===" +echo "1. Verify SSH connection and sudo access:" +echo " ssh moritz@136.243.23.215 'sudo kubectl get nodes -o wide'" +echo "2. Take manual etcd snapshot before upgrading:" +echo " ssh moritz@136.243.23.215 'sudo etcdctl --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db'" +echo "" +echo "=== TWO-PHASE UPGRADE WORKFLOW ===" +echo "Phase 1: Reconcile Kubespray baseline (keeps current k8s version):" +echo " ansible-playbook -i inventory/prod/inventory.ini cluster.yml" +echo "" +echo "Phase 2: Upgrade Kubernetes version (e.g. to 1.32.13):" +echo " a) Update 'kube_version: ' in ../inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml" +echo " b) Sync inventory: sync-inventory" +echo " c) Run in-place single-node upgrade:" +echo " ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml" +echo "" +echo "=== POST-UPGRADE VERIFICATION ===" +echo " kubectl version" +echo " kubectl get nodes -o wide" +echo " kubectl get pods -A | grep -v -E 'Running|Completed'" +echo "======================================================================================"