infrapuzzle/bootstrap/README.md

Bootstrap

The following lines document how to initalize a fresh cluster. On a real cluster, or using Vagrant. It therefore assumes to clone kubespray to this folder. It will be excluded in .gitignore and all files are in this folder.

The version to be initialized in configured in the init.sh script.

. ./init.sh

See kubespray.io on detailed information about kubespray. Though it seems to be a littlebit outdated.

Vagrant

cd kubespray
vagrant up
# up and abkle to ssh
vagrant ssh k8s-1

Prod

Prepare server:

• deactivate swap!
• moritz username ALL=(ALL) NOPASSWD:ALL
• `

ssh centos@<ip>
# auth via pw
sudo su - root
adduser moritz
visudo # add as sudo user
su - moritz
sudo yum -y install vim python3
ssh-keygen
vim .ssh/authorized_users # paste key
chmod 644 .ssh/authorized_keys
# check whether login works with ssh key
sudo vim /etc/ssh/sshd_config # remove pw auth & root login
sudo yum upgrade -y && sudo reboot

Install Kubernetes:

. ./init.sh
# follow instructions from output, sth like:
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml

And get credentials:

ssh <ip>
sudo su - root
cd
cp -r .kube /home/moritz/
chown -R moritz. /home/moritz/.kube
#ctrl + d
kubectl get ns # test connection
#ctrl + d
scp haumdaucher.de:/home/moritz/.kube/config .kube/config

Foreward in k8s-directory.

Upgrade cluster

Check the current default value of kube_version in cloned repository.

FOR K8S VERSION - Edit /Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml

cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e kube_version=v1.22.10 -e upgrade_cluster_setup=true cluster.yml
# or just the newest version
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
# upgrade to specific calico version (did not trigger/ failed)
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e calico_version=v3.15.2 cluster.yml --tags=network

History:

• 2020-04-18 kube_version=v1.16.8 kubespray_branch=release-2.12

Add node

See documentation.

Note: This was more or less a trial and error approach. Running different playbooks over and over again got it right at some point.

ansible-playbook -i inventory/prod/inventory.ini --limit=ns3088070.ip-37-59-40.eu,ns3100058.ip-37-59-61.eu scale.yml
ansible-playbook -i inventory/prod/inventory.ini --limit=etcd,kube-master -e ignore_assert_errors=yes cluster.yml

This runs everything and is kind of idempotent:

ansible-playbook -i inventory/prod/inventory.ini cluster.yml

Upgrade to Kubernetes 1.31 (Previous)

Required to execute:

ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e drain_nodes=false upgrade-cluster.yml

• set a feature flag: https://github.com/kubernetes-sigs/kubespray/issues/11887

Upgrade to Kubespray release-2.28 & Kubernetes v1.32.13

Note

Upgrade philosophy: decoupled two-phase upgrade.

1. Upgrade Kubespray tooling baseline (keeping k8s version unchanged).
2. Upgrade Kubernetes control plane and worker components.

Important:

• haumdaucher is a single-node cluster. Always use -e drain_nodes=false when running upgrade-cluster.yml to prevent the upgrade from hanging on draining.
• Ubuntu 20.04 LTS constraint: Kubespray release-2.28 is the last release officially supporting Ubuntu 20.04. Upstream Kubespray release-2.29+ requires Ubuntu 22.04 or 24.04.

Phase 1: Kubespray Baseline Upgrade (Retaining k8s v1.31.9)

1. Ensure init.sh has VERSION="release-2.28".
2. Initialize environment:

   . ./init.sh
   

3. Run cluster playbook to reconcile Kubespray baseline:

   ansible-playbook -i inventory/prod/inventory.ini cluster.yml
   

4. Verify cluster health:

   kubectl get nodes -o wide
   kubectl get pods -A | grep -v -E "Running|Completed"
   

Phase 2: Kubernetes Upgrade to v1.32.13

1. In inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml, update:

   kube_version: 1.32.13
   

2. Re-sync inventory to ephemeral kubespray/:

   rsync -a ./inventory/ ./kubespray/inventory/
   

3. Run upgrade playbook:

   cd kubespray
   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
   

4. Verify upgraded cluster:

   kubectl version
   kubectl get nodes -o wide
   kubectl get pods -A | grep -v -E "Running|Completed"