infrapuzzle/k8s

k8s

This folder holds all the services required for my private infrastructure. Following contraints apply:

• Order of implementation is top down.
• Every namespace has a subfolder within this subdirectory.
• helm3

Operations

Cleanup Error pods.

kubectl get pods | grep Error | cut -d' ' -f 1 | xargs kubectl delete pod

Redeploy a deployment:

DEPLOYMENT="rstudio"
NAMESPACE="datalab"
kubectl patch deployment $DEPLOYMENT -n $NAMESPACE -p "{\"spec\": {\"template\": {\"metadata\": { \"labels\": {  \"redeploy\": \"$( date +%s )\"}}}}}"

helm modify release data

# see https://gist.github.com/DzeryCZ/c4adf39d4a1a99ae6e594a183628eaee
kubectl get secret sh.helm.release.v1.mailu.v8 -n mailu -o json | jq .data.release | tr -d '"' | base64 -d | base64 -d | gzip -d > tmp_v8_mailu
# edit as you like
kubcetl edit -n mailu secret sh.helm.release.v1.mailu.v8
# delete the "release" stuff and copy and paste :)

Deployment (non persistent stuff)

Traefik Ingress Controller (Replaces retired ingress-nginx)

Apply Dual IngressClass configuration (supporting both legacy nginx and new traefik names on the same controller):

kubectl apply -f traefik/ingress-classes.yaml

Apply with Helm:

helm repo add traefik https://traefik.github.io/charts
helm repo update
helm upgrade --install traefik traefik/traefik -n kube-system -f traefik/traefik-values.yaml

Legacy ingress-nginx (RETIRED)

The community ingress-nginx is retired (EOL March 2026). During hot-swap, it was scaled down to 0 replicas. Keep it scaled down for a few days before deleting:

# To scale down (executed during migration)
kubectl scale daemonset ingress-nginx-controller -n ingress-nginx --replicas=0

# Permanent cleanup (execute after 3-7 days safety buffer)
# helm uninstall ingress-nginx -n ingress-nginx
# kubectl delete ns ingress-nginx

cert-manager

Apply with helm. See chart.:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install --create-namespace cert-manager jetstack/cert-manager -n cert-manager -f cert-manager/cert-manager.yaml
# apply the two issuer classes
kubectl apply -f cert-manager/staging-issuer.yaml
kubectl apply -f cert-manager/production-issuer.yaml

To test all this you may use the kuaard demo project:

$ kubectl apply -f kuard
# checkout: https://kuard.haumdaucher.de
$ kubectl delete -f kuard

openebs

Update with the follwoing command. Chart can be found here.

Pitfall:

• On fresh installation: activate ndmOperator, so that CRDs are correctly installed. It may be deactivated afterwards.

helm repo add openebs https://openebs.github.io/charts
helm repo update
helm upgrade --install --create-namespace -f openebs/openebs.yml openebs --namespace openebs openebs/openebs
k apply -f openebs/storageclass.yml

minio (bitnami)

Switching to Bitnami chart as "normal" chart just too big.

Links:

• minio-console.haumdaucher.de
• minio.haumdaucher.de

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
# 11.02.2024: Removed and reinstalled due to upgrade problem
helm upgrade --install -f minio/minio.secret.yaml --namespace minio --create-namespace minio bitnami/minio --version 14.8.1

velero

Backup tool. See chart README.

helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm repo update
# Apply CRDs first (required on upgrades across major versions)
helm show crds vmware-tanzu/velero --version 12.1.0 | kubectl apply -f -
helm upgrade --install --create-namespace --namespace velero -f ./velero/velero.secret.yaml velero vmware-tanzu/velero --version 12.1.0
# Secrets for rclone (minio + gdrive)
kubectl create secret generic rclone-config --from-file=./velero/rclone.secret -n velero
kubectl create secret generic gdrive-sa-key --from-file=gdrive_sa.json.secret=./velero/gdrive_sa.json.secret -n velero
# Deploy GDrive sync cronjob
kubectl apply -f velero/gdrive_sync.yml
# Teardown
helm delete velero -n velero
kubectl delete ns velero

A manual backup may be created executing the following command. Note: Keep backed-up namespaces in sync with config from helm chart!!!

DATE=$( date +%Y%m%d )
velero backup create $DATE --include-namespaces coder,home-assistant,n8n,influxdb --wait

Backup Retention Policy

Three schedules run automatically, providing a tiered retention policy. Kopia (used for fs-backup) deduplicates at the block level, so each backup only stores changed data — incrementals come for free.

Schedule	Cron	TTL	Keeps
velero-daily-backup	0 2 * * *	8 days	Last 7 daily backups
velero-weekly-backup	0 3 * * 0	43 days	Last 6 weekly backups (Sundays)
velero-monthly-backup	0 4 1 * *	183 days	Last 6 monthly backups (1st of month)

# Check all schedules and their last run
velero schedule get

# Check current backups and their expiry
velero backup get

coder

Coder is a code server see https://coder.com/docs/v2/latest/install/kubernetes.

kubectl create namespace coder
# Install PostgreSQL
helm repo add bitnami https://charts.bitnami.com/bitnami
# create secret for postgresdb
kubectl apply -f coder/postgres_users.secret.yaml
helm upgrade --install coder-db bitnami/postgresql \
    -n coder \
    -f coder/bitnami_postgresql.secret.yaml \
    --version 15.2.5 
# db url postgres://coder:<password>@coder-db-postgresql.coder.svc.cluster.local:5432/coder?sslmode=disable
# Uses Bitnami PostgreSQL example. If you have another database,
# change to the proper URL.
USERPASS=$( kubectl get secret -n coder coder-db-postgresql -o=jsonpath='{.data.password}' | base64 -D )
kubectl create secret generic coder-db-url -n coder \
   --from-literal=url="postgres://coder:${USERPASS}@coder-db-postgresql.coder.svc.cluster.local:5432/coder?sslmode=disable"
helm repo add coder-v2 https://helm.coder.com/v2
#
helm upgrade --install coder coder-v2/coder \
    --namespace coder \
    --values coder/coder.secret.yaml \
    --version 2.10.0

llm

See llm

Add private docker registry

TODO: chart no longer exists. Check how to replace this someday.

# create secret base64 encoded and put it in htpasswd helm chart
USER='moritz'
PASSWORD='xxx'
docker run --entrypoint htpasswd --rm registry:2 -Bbn $USER $PASSWORD
# #
# 1. Add the modern repo
helm repo add twuni https://twuni.github.io/docker-registry.helm
helm repo update

# 2. Install the new one
helm upgrade --install docker-registry twuni/docker-registry \
  --namespace development \
  --create-namespace \
  -f development/registry.secret.yaml

### 3. Verification
Once deployed, verify you can login from your local machine:
```bash
docker login registry.haumdaucher.de -u moritz

creating docker-pull-secret

Create credentials secret according to docu:

namespaces="datalab moritz web haumdaucher"
# the following is ONLY required to update the secret file!!
for i in $( echo $namespaces ) ; do
  kubectl create secret docker-registry registry-haumdaucher-de \
    -n $i \
    --docker-server=registry.haumdaucher.de \
    --docker-username=moritz \
    --docker-password='xxxxxxx' \
    --docker-email=moritz@moritzgraf.de \
    --dry-run -o yaml > ./${i}/docker-pull.yaml.secret
done
# apply (may be executed as needed)
for i in $( echo $namespaces ) ; do
  kubectl apply -f datalab/docker-pull.yaml.secret -n $i
done

For kubeflow:

cat << EOF >> config.json
{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "$( echo -n 'moritz:password' | base64 )"
        }
    }
}
EOF
kubectl create -n kubeflow configmap docker-config --from-file=config.json
rm config.json

networking with calico

Install calicoctl in cluster

kubectl apply -n kube-system -f https://docs.projectcalico.org/manifests/calicoctl.yaml

Then you may send commands like:

kubectl exec -ti -n kube-system calicoctl -- /calicoctl get workloadendpoints -n mailu

Or on the server directly:

sudo -E /usr/local/bin/calicoctl node checksystem

metrics

See this documentation.

kubectl exec -ti -n kube-system calicoctl -- /calicoctl patch felixConfiguration default  --patch '{"spec":{"prometheusMetricsEnabled": true}}'
kubectl apply -n kube-system -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: felix-metrics-svc
  namespace: kube-system
spec:
  selector:
    k8s-app: calico-node
  ports:
  - port: 9091
    targetPort: 9091
EOF

metrics-server

Getting resources (already done):

cd kube-system
curl -L -o metrics-server.yml https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
# add parameters to deployment:
#           - --kubelet-preferred-address-types=InternalIP
#           - --v=2
#           - --kubelet-insecure-tls

Implement metrics-server:

kubectl apply -n kube-system -f kube-system/metrics-server.yml

ameliegraf

Note: Not yet finished. Switched back to portfolio adresses.

The website redirection for ameliegraf.de.

k create ns ameliegraf
k apply -f ameliegraf/ameliegraf.yml

Deployment (persistent stuff)

From here everything should be covered by the backup. Implenting those objects should already be performed by the velero backup.

rstudio

*DISABLED IN FAVOR OF KUBEFLOW

Currently only for one user:

kubectl apply -f datalab/rstudio.yaml
kubectl delete -f datalab/rstudio.yaml

tt-rss

Includes persistent data from mariadb table tt-rss.

helm upgrade --install tt-rss-mariadb bitnami/mariadb -n tt-rss -f tt-rss/tt-rss-mariadb.secret.yml
helm upgrade --install tt-rss-phpmyadmin bitnami/phpmyadmin -n tt-rss -f tt-rss/tt-rss-phpmyadmin.yml

helm upgrade --install tt-rss k8s-at-home/tt-rss -n tt-rss -f tt-rss/tt-rss.helm.secret.yml

monitoring

The prometheus-operator, now called kube-prometheus-stack is used.

kubectl create ns monitoring
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install --create-namespace prometheus-operator prometheus-community/kube-prometheus-stack -n monitoring -f monitoring/prometheus-operator.secret.yml --version 56.6.1
# alert configuration
NAMESPACES_TO_ALERT=( kube-system monitoring cert-manager datalab ingress-nginx mailu minio velero web openebs )
for i in "${NAMESPACES_TO_ALERT[@]}"; do
  kubectl apply -f monitoring/alertmanagerconfig.secret.yaml -n $i
done

influxdb

Used to store hass data long term.

helm repo add influxdata https://helm.influxdata.com/
helm upgrade --install influxdb -n influxdb influxdata/influxdb2 -f influxdb/influxdb2.secret.yml --create-namespace

home-assistant hass

How to generate token (not really required): https://github.com/hahn-th/homematicip-rest-api Using this helm chart: https://github.com/pajikos/home-assistant-helm-chart

Install chart:

# secret for auth in hass-code
k apply -f home-assistant/hass-code-auth.secret.yml
#
helm repo add pajikos http://pajikos.github.io/home-assistant-helm-chart/
helm repo update
#helm show values pajikos/home-assistant > ./home-assistant/home-assistant.yaml
k create ns home-assistant
helm upgrade --install home-assistant pajikos/home-assistant -n home-assistant -f ./home-assistant/home-assistant.yaml

robusta

k create ns robusta
helm repo add robusta https://robusta-charts.storage.googleapis.com
helm repo update
helm upgrade --install robusta robusta/robusta -n robusta -f ./robusta/robusta.yaml

gitea (old, no longer existant, do not use)

In case my PRs have been accepted this is no longer necessary:

git clone git@github.com:iptizer/gitea-chart.git

# from chart repo
helm upgrade --install gitea k8s-land/gitea -n development -f development/gitea.secret.yml
# from local folder
helm upgrade --install gitea ./gitea-chart -n development -f development/gitea.secret.yml

# phpmyadmin
helm upgrade --install gitea-phpmyadmin bitnami/phpmyadmin -n development -f development/gitea-phpmyadmin.yml

gitea (new set up)

Chart used: https://gitea.com/gitea/helm-gitea

History: Manually deleted the old git server and reapplied a new one.

# 1. Add/Update Repo
helm repo add gitea-charts https://dl.gitea.com/charts/
helm repo update

# 2. Install
helm upgrade --install gitea gitea-charts/gitea \
  --namespace development \
  -f development/gitea.secret.yml

backup & restore

See the backup cronjob in the /backup/ folder.

For backup & restore see gitea documentation.

Download the gitea-dump locally and proceed with the following commands:

❯ mkdir gitea_restore
❯ mv gitea-dump-1587901016.zip gitea_restore
❯ cd gitea_restore
❯ unzip gitea-dump-1587901016.zip
Archive:  gitea-dump-1587901016.zip
  inflating: gitea-repo.zip          
   creating: custom/
[...]

Import of sql may be done via phpmyadmin.

Copy to remote pod:

kubectl cp ./gitea-repo.zip gitea-gitea-69cd9bc59b-q2b2f:/data/git/

And finally unzip inside shell on pod:

cd /data/git/
unzip gitea-repo.zip
mv repositories/ gitea-repositories/
chown -R git. ./gitea-repositories/

Then login to git.moritzgraf.de and proceed with default values, or adjust them.

octobot

Deployment instructions for Octobot. Dex is used for authenticating.

kubectl create ns octobot
helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm repo update
helm upgrade --install -n octobot oauth2-octobot oauth2-proxy/oauth2-proxy -f ./octobot/oauth2.secret.values
kubectl apply $(ls octobot/*.yaml | awk ' { print " -f " $1 } ')

octobot-fabi

Deployment instructions for Octobot. Dex is used for authenticating.

kubectl create ns octobot-fabi
helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm repo update
helm upgrade --install -n octobot-fabi oauth2-octobot oauth2-proxy/oauth2-proxy -f ./octobot-fabi/oauth2.secret.values
kubectl apply $(ls octobot-fabi/*.yaml | awk ' { print " -f " $1 } ')

nextcloud

Chart GitHub

Configuring Nextcloud

helm repo add nextcloud https://nextcloud.github.io/helm/
helm repo update
helm upgrade --install nextcloud nextcloud/nextcloud -n nextcloud --version 4.6.2 -f nextcloud/nextcloud.secret.yml
helm upgrade --install nextcloud-phpmyadmin bitnami/phpmyadmin -n nextcloud -f nextcloud/nextcloud-phpmyadmin.yml

Execute occ in container:

runuser --user www-data -- /usr/local/bin/php /var/www/html/occ

Download nextcloud server and extract occ:

curl -L -o nextcloud-v27.1.6.tar.gz https://github.com/nextcloud/server/archive/refs/tags/v27.1.6.tar.gz

Delete stuff for upgrade (sometime necessary):

kubectl delete sts/nextcloud-mariadb -n nextcloud
kubectl delete sts/nextcloud-redis-master -n nextcloud
kubectl delete deployment nextcloud -n nextcloud

SyncURL for DavX5 => https://cloud.haumdaucher.de/remote.php/dav/principals/users/moritz/

Unknown why normal url is not working. See https://help.nextcloud.com/t/davx5-couldnt-find-caldav-or-carddav-service/68669

backup & restore

#TODO with Velero

influxdb & mosquitto

datalab

helm repo add influxdata https://helm.influxdata.com/
# influx stuff
helm upgrade --install influxdb2 influxdata/influxdb2 --namespace datalab --values datalab/influxdb2.yml
helm upgrade --install telegraf-operator influxdata/telegraf-operator --namespace datalab --values datalab/telegraf-operator.yml
#
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install postgres bitnami/postgresql --namespace datalab --values datalab/postgres.yml.secret
#mqtt
helm repo add t3n https://storage.googleapis.com/t3n-helm-charts
helm repo update
helm upgrade --install mosquitto t3n/mosquitto --namespace datalab --values datalab/mosquitto.secret.yml

work with it

# retrieve admin password
echo $(kubectl get secret influxdb2-auth -o "jsonpath={.data['admin-password']}" --namespace datalab | base64 --decode)

Tear down:

helm uninstall influxdb --namespace datalab

timescaledb @datalab

git clone git@github.com:timescale/timescaledb-kubernetes.git ../../timescaledb-kubernetes
../../timescaledb-kubernetes/charts/timescaledb-single/generate_kustomization.sh timescaledb
cp -r "../../timescaledb-kubernetes/charts/timescaledb-single/kustomize/timescaledb" ./datalab/timescaledb.secret
kubectl apply -n datalab -k ./datalab/timescaledb.secret
helm repo add timescaledb 'https://raw.githubusercontent.com/timescale/timescaledb-kubernetes/master/charts/repo/'
helm install timescaledb timescaledb/timescaledb-single --namespace datalab --values datalab/timescaledb.yml

mailu

Using the mailu helm chart.

helm repo add mailu https://mailu.github.io/helm-charts/
helm repo update
kubectl apply -f mailu/mailu.secret-key.secret.yml
helm upgrade --install mailu mailu/mailu -n mailu -f mailu/mailu.secret.yml --version 0.3.5
#helm upgrade --install mailu ../../mailu-helm-charts/mailu/ -n mailu -f mailu/mailu.secret.yml
helm template mailu mailu/mailu -n mailu -f mailu/mailu.secret.yml
#helm template mailu ../../mailu-helm-charts/mailu/ -n mailu -f mailu/mailu.secret.yml

SECRET_KEY

There was an ERROR as follows:

ERROR:root:Can't read SECRET_KEY from file: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/config.py", line 8, in <module>
    system.set_env()
  File "/app/venv/lib/python3.10/site-packages/socrate/system.py", line 35, in set_env
    raise exc
  File "/app/venv/lib/python3.10/site-packages/socrate/system.py", line 32, in set_env
    secret_key = open(os.environ.get("SECRET_KEY_FILE"), "r").read().strip()
TypeError: expected str, bytes or os.PathLike object, not NoneTypew

The fix was to add the env variable as follows to the failing deployments:

          - name: SECRET_KEY
            value: "fa5faeD9aegietaesahbiequ5Pe9au"

troubleshooting

Test imap from console:

openssl s_client -crlf -connect moritzgraf.de:993

migrate to GoogleWorkspace

Namespace migrateis used.

kubectl create ns migrate

old migrate (from before GoogleWorkspace)

# backup on moritzgraf.de
ssh moritzgraf.de "sudo su - docker -c 'cd /home/docker/mailu && docker-compose stop' && sudo su - root -c 'cd /home/docker && tar czvf mailu.tar.gz ./mailu && mv mailu.tar.gz /home/moritz/ && chown moritz. /home/moritz/mailu.tar.gz' && scp mailu.tar.gz haumdaucher.de:/home/moritz/"
# terraform change
cd ../terraform && terraform apply
# helm apply
cd ../k8s
helm upgrade --create-namespace --install mailu mailu/mailu -n mailu -f mailu/mailu.secret.yml
# apply mailu and scale all to 0
kc mailu
k scale --replicas=0 --all=true deploy
# apply restore pod
k apply -f mailu/restore.yaml
# copy tar gz
ssh one
k cp mailu.tar.gz restore:/data/
# exec to pod and arrange persistence
k exec -it restore -- bash
cd /data
tar xzvf mailu.tar.gz
mv mailu/data/* ./admin/
mv mailu/dkim/* ./dkim/
mv mailu/mail/* ./dovecotmail/
chown -R mail:man ./dovecotmail
# scale up
k scale --replicas=1 --all=true deploy

Checks:

• browser mail.moritzgraf.de & login
• browser mail.moritzgraf.de/admin

mopbot

Mopbot deployment has been moved to the mopbot repository itself.

zebrium

zebrium.io hat "AI powered" troubleshooting. This is a test use-case for my cluster. It was installed using the following command:

kubectl create namespace zebrium
helm install zlog-collector zlog-collector --namespace zebrium --repo https://raw.githubusercontent.com/zebrium/ze-kubernetes-collector/master/charts --set zebrium.collectorUrl=https://zapi03.zebrium.com,zebrium.authToken=4CFDFC1B806869D78185776AD7D940B0B16254AC,zebrium.deployment=iptizer-zebrium-deploy,zebrium.timezone=Europe/Berlin
helm install zstats-collector zstats --namespace zebrium --repo https://raw.githubusercontent.com/zebrium/ze-stats/master/charts --set zebrium.collectorUrl=https://zapi03.zebrium.com/stats/api/v1/zstats,zebrium.authToken=4CFDFC1B806869D78185776AD7D940B0B16254AC,zebrium.deployment=iptizer-zebrium-deploy

Benchmark with dbench

Should be usually uncommented, but dbbench may be executed to check speed of local filesystem:

k create ns dbench
k apply -f dbench/
k delete -f dbench

Web

kubectl create ns web
kubectl apply -n web ./re
kubectl apply -f web/

Kubeflow

The whole Kubeflow deployment is documented in a seperate repository:

• https://git.moritzgraf.de/moritz/datalab-kubeflow

Archive

Deployments previously used.

Jupyter

DEPRECATED: Using Kubeflow instead. Moved to _archive.

Using the project zero-to-jupyterhub. Helm chart can be found here.

helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm upgrade --cleanup-on-fail --install jupyter jupyterhub/jupyterhub --namespace datalab --values datalab/jupyter-values.yaml
helm delete jupyter --namespace datalab

Tekton

DEPRECATED: Using Argo from Kubeflow instead.

Implementation as described in the docs.

kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl apply --filename https://github.com/tektoncd/dashboard/releases/latest/download/tekton-dashboard-release.yaml
#basic-auth, see https://kubernetes.github.io/ingress-nginx/examples/auth/basic/
htpasswd -c ./tekton-pipelines/auth tekton
kubectl delete secret -n tekton-pipelines basic-auth
kubectl create secret -n tekton-pipelines generic basic-auth --from-file=tekton-pipelines/auth
kubectl apply -f tekton-pipelines/tekton-ingress.yml
rm tekton-pipelines/auth

k delete ns tekton-pipelines

Install client side tools:

brew tap tektoncd/tools
brew install tektoncd/tools/tektoncd-cli

cnpg - Cloud Native Postgres

cnpg is a postgres operator that is recommended by n8n. So I installed it.

Instructions here

Executed:

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.26/releases/cnpg-1.26.1.yaml

n8n

Using helm chart from https://github.com/8gears/n8n-helm-chart.

kubectl create ns n8n
helm upgrade --cleanup-on-fail --install mop-n8n \
oci://8gears.container-registry.com/library/n8n \
--namespace n8n --values n8n/n8n.secret.yml --version 2.0.1

To verify installation was correcet, use the following command:

helm get manifest mop-n8n -n n8n | less

n8n-fabi

Important

Notice: This deployment is currently not deployed.

# kubectl create ns n8n-fabi
# helm upgrade --cleanup-on-fail --install fabi-n8n \
# oci://8gears.container-registry.com/library/n8n \
# --namespace n8n-fabi --values n8n-fabi/n8n-fabi.secret.yml --version 2.0.1

LLM (Ollama)

Hosts a local LLM (Llama 3 8B) for inference.

• Deploy: kubectl apply -f llm/ollama.yaml
• Verification: Check pods in llm namespace.

OpenClaw

An autonomous AI agent platform.

1. Create Namespace: kubectl apply -f openclaw/namespace.yaml
2. Configure Secrets:
   • Edit openclaw/openclaw.secret.yaml.
   • Gemini: Replace change-me with your Gemini API Key.
   • Telegram: Replace telegram-bot-token with your Bot Token.
   • Gateway: Token is pre-filled (randomly generated). Change if desired.
   • Auth: Generate auth string using htpasswd -nb <user> <password> and replace the placeholder.
   • Encrypt: Ensure the file is encrypted with git crypt before committing!
3. Deploy: kubectl apply -f openclaw/openclaw.secret.yaml
4. Access: https://openclaw.haumdaucher.de (Authentication required)