infrapuzzle/bootstrap/README.md

# Bootstrap

The following lines document how to initalize a fresh cluster. On a real cluster, or using Vagrant. It therefore assumes to clone kubespray to this folder. It will be excluded in *.gitignore* and all files are in this folder.

The version to be initialized in configured in the `init.sh` script.

```sh
. ./init.sh
```

See [kubespray.io](https://kubespray.io/) on detailed information about kubespray. Though it seems to be a littlebit outdated.

## Vagrant

```sh
cd kubespray
vagrant up
# up and abkle to ssh
vagrant ssh k8s-1
```

## Prod

Prepare server:
* deactivate swap!
* `moritz username     ALL=(ALL) NOPASSWD:ALL`
* `

```sh
ssh centos@<ip>
# auth via pw
sudo su - root
adduser moritz
visudo # add as sudo user
su - moritz
sudo yum -y install vim python3
ssh-keygen
vim .ssh/authorized_users # paste key
chmod 644 .ssh/authorized_keys
# check whether login works with ssh key
sudo vim /etc/ssh/sshd_config # remove pw auth & root login
sudo yum upgrade -y && sudo reboot
```

Install Kubernetes:

```sh
. ./init.sh
# follow instructions from output, sth like:
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```

And get credentials:

```sh
ssh <ip>
sudo su - root
cd
cp -r .kube /home/moritz/
chown -R moritz. /home/moritz/.kube
#ctrl + d
kubectl get ns # test connection
#ctrl + d
scp haumdaucher.de:/home/moritz/.kube/config .kube/config
```

Foreward in k8s-directory.

## Upgrade cluster

Check the current default value of `kube_version` in cloned repository.

**FOR K8S VERSION - Edit /Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml**

```sh
cd kubespray
ansible-playbook -i inventory/prod/inventory.ini -e kube_version=v1.22.10 -e upgrade_cluster_setup=true cluster.yml
# or just the newest version
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
# upgrade to specific calico version (did not trigger/ failed)
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e calico_version=v3.15.2 cluster.yml --tags=network
```

History:

* 2020-04-18 kube_version=v1.16.8 kubespray_branch=release-2.12

## Add node

See [documentation](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/nodes.md).

Note: This was more or less a trial and error approach. Running different playbooks over and over again got it right at some point.

```sh
ansible-playbook -i inventory/prod/inventory.ini --limit=ns3088070.ip-37-59-40.eu,ns3100058.ip-37-59-61.eu scale.yml
ansible-playbook -i inventory/prod/inventory.ini --limit=etcd,kube-master -e ignore_assert_errors=yes cluster.yml
```

This runs everything and is kind of idempotent:

```sh
ansible-playbook -i inventory/prod/inventory.ini cluster.yml
```

## Upgrade to Kubernetes 1.31 (Previous)

Required to execute: 

```sh
ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e drain_nodes=false upgrade-cluster.yml
```

+ set a feature flag: https://github.com/kubernetes-sigs/kubespray/issues/11887

## Upgrade to Kubespray release-2.28 & Kubernetes v1.32.13

> [!NOTE]
> Upgrade philosophy: decoupled two-phase upgrade.
> 1. Upgrade Kubespray tooling baseline (keeping k8s version unchanged).
> 2. Upgrade Kubernetes control plane and worker components.
>
> **Important**:
> - `haumdaucher` is a **single-node cluster**. Always use `-e drain_nodes=false` when running `upgrade-cluster.yml` to prevent the upgrade from hanging on draining.
> - **Ubuntu 20.04 LTS constraint**: Kubespray `release-2.28` is the **last** release officially supporting Ubuntu 20.04. Upstream Kubespray `release-2.29`+ requires Ubuntu 22.04 or 24.04.

### Phase 1: Kubespray Baseline Upgrade (Retaining k8s v1.31.9)
1. Ensure `init.sh` has `VERSION="release-2.28"`.
2. Initialize environment:
   ```sh
   . ./init.sh
   ```
3. Run cluster playbook to reconcile Kubespray baseline:
   ```sh
   ansible-playbook -i inventory/prod/inventory.ini cluster.yml
   ```
4. Verify cluster health:
   ```sh
   kubectl get nodes -o wide
   kubectl get pods -A | grep -v -E "Running|Completed"
   ```

### Phase 2: Kubernetes Upgrade to v1.32.13
1. In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update:
   ```yaml
   kube_version: 1.32.13
   ```
2. Re-sync inventory to ephemeral `kubespray/`:
   ```sh
   rsync -a ./inventory/ ./kubespray/inventory/
   ```
3. Run upgrade playbook:
   ```sh
   cd kubespray
   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
   ```
4. Verify upgraded cluster:
   ```sh
   kubectl version
   kubectl get nodes -o wide
   kubectl get pods -A | grep -v -E "Running|Completed"
   ```