diff --git a/bootstrap/.gitignore b/bootstrap/.gitignore index d08070b..f56018c 100644 --- a/bootstrap/.gitignore +++ b/bootstrap/.gitignore @@ -1,2 +1,4 @@ kubespray venv +credentials/ +inventory/**/credentials/ diff --git a/bootstrap/AGENTS.md b/bootstrap/AGENTS.md index 0e9d988..de2d7a3 100644 --- a/bootstrap/AGENTS.md +++ b/bootstrap/AGENTS.md @@ -15,7 +15,7 @@ The core workflow is encapsulated in `init.sh`. **Purpose**: Prepares the environment and Kubespray for deployment. **Actions**: 1. **Clean Slate**: Deletes existing `kubespray/` directory. -2. **Clone**: Clones Kubespray (version defined in variable `VERSION`, e.g., `release-2.27`). +2. **Clone**: Clones Kubespray (version defined in variable `VERSION`, e.g., `release-2.28`). 3. **Environment**: Sets up Python virtualenv via `pyenv` and installs `requirements.txt`. 4. **Sync**: Copies local `./inventory/` configurations into `./kubespray/inventory/`. @@ -26,12 +26,13 @@ The core workflow is encapsulated in `init.sh`. ``` 2. **Deploy / Upgrade**: After sourcing, go to the `kubespray` directory and run the Ansible playbooks as instructed by the script output. - * **Standard Run**: + * **Phase 1: Reconcile Kubespray baseline (stay on current k8s version)**: ```bash cd kubespray ansible-playbook -i inventory/prod/inventory.ini cluster.yml ``` - * **Forced Upgrade**: + * **Phase 2: Upgrade Kubernetes version (e.g. to v1.32.13)**: + Update `kube_version` in `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, re-sync, and run: ```bash cd kubespray ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml diff --git a/bootstrap/README.md b/bootstrap/README.md index 667a964..b99059d 100644 --- a/bootstrap/README.md +++ b/bootstrap/README.md @@ -103,12 +103,60 @@ This runs everything and is kind of idempotent: ansible-playbook -i inventory/prod/inventory.ini cluster.yml ``` -## Upgrade to 2.31.3 +## Upgrade to Kubernetes 1.31 (Previous) Required to execute: -``` +```sh ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e drain_nodes=false upgrade-cluster.yml ``` + set a feature flag: https://github.com/kubernetes-sigs/kubespray/issues/11887 + +## Upgrade to Kubespray release-2.28 & Kubernetes v1.32.13 + +> [!NOTE] +> Upgrade philosophy: decoupled two-phase upgrade. +> 1. Upgrade Kubespray tooling baseline (keeping k8s version unchanged). +> 2. Upgrade Kubernetes control plane and worker components. +> +> **Important**: +> - `haumdaucher` is a **single-node cluster**. Always use `-e drain_nodes=false` when running `upgrade-cluster.yml` to prevent the upgrade from hanging on draining. +> - **Ubuntu 20.04 LTS constraint**: Kubespray `release-2.28` is the **last** release officially supporting Ubuntu 20.04. Upstream Kubespray `release-2.29`+ requires Ubuntu 22.04 or 24.04. + +### Phase 1: Kubespray Baseline Upgrade (Retaining k8s v1.31.9) +1. Ensure `init.sh` has `VERSION="release-2.28"`. +2. Initialize environment: + ```sh + . ./init.sh + ``` +3. Run cluster playbook to reconcile Kubespray baseline: + ```sh + ansible-playbook -i inventory/prod/inventory.ini cluster.yml + ``` +4. Verify cluster health: + ```sh + kubectl get nodes -o wide + kubectl get pods -A | grep -v -E "Running|Completed" + ``` + +### Phase 2: Kubernetes Upgrade to v1.32.13 +1. In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update: + ```yaml + kube_version: 1.32.13 + ``` +2. Re-sync inventory to ephemeral `kubespray/`: + ```sh + rsync -a ./inventory/ ./kubespray/inventory/ + ``` +3. Run upgrade playbook: + ```sh + cd kubespray + ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml + ``` +4. Verify upgraded cluster: + ```sh + kubectl version + kubectl get nodes -o wide + kubectl get pods -A | grep -v -E "Running|Completed" + ``` diff --git a/bootstrap/init.sh b/bootstrap/init.sh index 9469fe4..287ad0c 100755 --- a/bootstrap/init.sh +++ b/bootstrap/init.sh @@ -6,14 +6,19 @@ echo "######################################################################################" echo "## Reinit repository" rm -rf kubespray -VERSION="release-2.27" +VERSION="release-2.28" git clone --branch $VERSION https://github.com/kubernetes-sigs/kubespray.git +if [ -d patches ]; then + for p in patches/*.patch; do + [ -f "$p" ] && (echo "## Applying patch $p" && git -C kubespray apply "../$p") + done +fi echo "######################################################################################" echo "## Activating pyenv venv" eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)" -pyenv virtualenv 3.10.8 infrapuzzle-bootstrap +pyenv virtualenvs --bare | grep -q "^infrapuzzle-bootstrap$" || pyenv virtualenv 3.10.8 infrapuzzle-bootstrap pyenv activate infrapuzzle-bootstrap python -m pip install -r kubespray/requirements.txt @@ -46,9 +51,9 @@ echo "## * syncing config in kubespray dir" rsync -a ./inventory/ ./kubespray/inventory/ echo "## * changing into kubespray dir" cd kubespray -echo "## * execute the following command to force new settings" +echo "## * Step 1: Reconcile Kubespray baseline (keep current k8s version):" echo "ansible-playbook -i inventory/prod/inventory.ini cluster.yml" -echo "## * execute the following command to force new settings with forced cluster upgrade" +echo "## * Step 2: Once verified, bump kube_version in inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml and run:" echo "ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml" # cp -r kubespray/inventory/sample kubespray/inventory/prod diff --git a/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml b/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml index 90d5bfc..9ff2615 100644 --- a/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml +++ b/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml @@ -27,7 +27,17 @@ kube_kubeadm_apiserver_extra_args: kube_api_anonymous_auth: true ## Change this to use another Kubernetes version, e.g. a current beta release -kube_version: v1.31.9 +kube_version: 1.32.13 + +# Single-node cluster settings: prevent draining or cordoning the only node +drain_nodes: false +upgrade_node_always_cordon: false +upgrade_node_fail_if_drain_fails: false +kubeadm_ignore_preflight_errors: + - ImagePull + - SystemVerification + - FileContent--proc-sys-net-bridge-bridge-nf-call-iptables + - CreateJob # kubernetes image repo define #kube_image_repo: "k8s.gcr.io" diff --git a/bootstrap/patches/kubeadm-upgrade-ignore-preflight.patch b/bootstrap/patches/kubeadm-upgrade-ignore-preflight.patch new file mode 100644 index 0000000..2bd878e --- /dev/null +++ b/bootstrap/patches/kubeadm-upgrade-ignore-preflight.patch @@ -0,0 +1,20 @@ +diff --git a/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml b/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml +index 523956d1d..75723a6e0 100644 +--- a/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml ++++ b/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml +@@ -17,6 +17,7 @@ + --force + {%- else %} + --config={{ kube_config_dir }}/kubeadm-config.yaml ++ --ignore-preflight-errors={{ kubeadm_ignore_preflight_errors | join(',') }} + {%- endif %} + {%- if kube_version is version('1.32.0', '>=') %} + --skip-phases={{ kubeadm_init_phases_skip | join(',') }} +@@ -38,6 +39,7 @@ + {% if kubeadm_patches | length > 0 %}--patches={{ kubeadm_patches_dir }}{% endif %} + {%- else %} + --config={{ kube_config_dir }}/kubeadm-config.yaml ++ --ignore-preflight-errors={{ kubeadm_ignore_preflight_errors | join(',') }} + {%- endif %} + --skip-phases={{ kubeadm_upgrade_node_phases_skip | join(',') }} + register: kubeadm_upgrade