moritz
/
infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
infrapuzzle/k8s/AGENTS.md

8.7 KiB
Raw Blame History

AGENTS.md

[!NOTE] This file describes the constraints and conventions for the k8s directory, which contains deployments for the haumdaucher.de Kubernetes cluster.

Project Overview

This directory contains the Kubernetes manifests and Helm charts for a single-node Kubernetes cluster (Haumdaucher).

  • Domain: *.haumdaucher.de
  • Orchestration: Self-managed Kubernetes (single node).
  • Ingress: ingress-nginx
  • SSL: cert-manager (LetsEncrypt)

Directory Structure

  • Top-level folders: Each folder corresponds to a Kubernetes namespace.
    • Example: mailu/ contains resources for the mailu namespace.
  • Documentation: README.md is the authoritative source for deployment commands. Always check it before running commands.

Code Style & Conventions

  • Helm Version: Helm 3 (helm) is used.
  • Implementation Order: Top-down.
  • Naming: Namespaces matches folder names.
  • Formatting: Standard YAML conventions.

Security & Secrets

[!IMPORTANT] Git-Crypt is enforced. Do not touch encrypted files unless you have the key and know how to unlock them.

Encrypted File Patterns:

  • *.secret
  • *.secret.yaml
  • *.secret.values
  • *.secret.sh

Remote Access

It is possible to execute commands on the remote Linux node for information retrieval or troubleshooting.

  • Host: haumdaucher.de
  • User: moritz (local user)
  • Privileges: Use sudo to gain root privileges.

[!CAUTION] SSH Identity Required: The agent cannot enter an SSH passphrase. If SSH commands fail with authentication errors, request the user to run ssh-add locally to load their identity.

Command Execution

You can execute commands remotely via SSH. This is useful for checking node-level resources (memory, disk, etc.) that kubectl might not expose directly.

Example: Check Memory Usage

ssh moritz@haumdaucher.de "free -h"

Example: Check Disk Usage (with sudo)

ssh -t moritz@haumdaucher.de "sudo df -h"

Note: The -t flag forces pseudo-terminal allocation, which is often required for sudo prompts.

Deployment Instructions

Always consult README.md first. Deployments vary between Helm charts and raw manifests.

Common Patterns

  • Helm: 
    helm upgrade --install <release> <chart> -n <namespace> -f <values-file>
  • Kubectl: 
    kubectl apply -f <folder>/<file>.yaml

Post-Implementation Verification

[!IMPORTANT] Verification Workflow: After a new implementation or configuration change, always:

  1. Run kubectl apply -f <file>.yaml.
  2. Run kubectl rollout restart deployment <deployment-name> -n <namespace> if applying a ConfigMap/Secret that a deployment depends on.
  3. Wait for 30 seconds to allow pods to roll over.
  4. Check logs using kubectl logs -n <namespace> -l <label> --tail=100.

The agent must always ask the user for permission to execute this verification workflow after making changes.

Operational Tasks

  • Cleanup Error Pods: 
    kubectl get pods | grep Error | cut -d' ' -f 1 | xargs kubectl delete pod

Ingress Configuration

Ingress resources must follow these strict conventions to work with the cluster's ingress controller (traefik) and certificate manager (cert-manager).

Annotations

All Ingress resources must include:

annotations:
  kubernetes.io/ingress.class: "traefik"
  cert-manager.io/cluster-issuer: "letsencrypt-prod"
  kubernetes.io/tls-acme: "true"
  # Standard nginx tweaks (if using dual class) or traefik configurations
  nginx.ingress.kubernetes.io/proxy-body-size: "0"
  nginx.ingress.kubernetes.io/ssl-redirect: "true"
  nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

Hostnames & TLS

  • Domain: Use a subdomain of haumdaucher.de or moritzgraf.de.
  • TLS Secret Name: Must use hyphens instead of dots.
  • Pattern: <subdomain>-<domain>-<tld>
  • Example: n8n.moritzgraf.de -> n8n-moritzgraf-de

Example

spec:
  ingressClassName: traefik
  tls:
    - hosts:
        - n8n.moritzgraf.de
      secretName: n8n-moritzgraf-de
  rules:
    - host: n8n.moritzgraf.de
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: n8n
                port:
                  number: 5678

Storage / Persistence

The cluster uses OpenEBS for dynamic local storage provisioning.

PersistentVolumeClaims (PVC)

  • Provisioner: openebs.io/local (or similar, managed via openebs-hostpath).
  • StorageClass: openebs-hostpath.
  • AccessMode: Typically ReadWriteOnce (RWO) as it's local storage.

To request storage, simply create a PVC or configure Helm charts to use the default storage class (or explicitly openebs-hostpath).

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-data
spec:
  storageClassName: openebs-hostpath
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Deployment Constraints

  • Resources: Always define requests and limits for CPU and Memory to ensure fair scheduling on the single node.
  • Namespaces: Every application gets its own namespace.
  • Secrets: Encrypt all secrets using git-crypt.

Network Setup (VPN & Routing)

The Kubernetes cluster (Haumdaucher) is connected to the home network via a dedicated WireGuard VPN tunnel.

graph TD
    subgraph "Kubernetes Cluster (haumdaucher.de - 136.243.23.215)"
        subgraph "monitoring namespace"
            Exporter[fritzbox-exporter Pod]
        end
        HostNet[Host Network Namespace]
        WgPod[wireguard Pod hostNetwork: true]
    end

    subgraph "Home Network (192.168.10.0/24)"
        FB[FRITZ!Box Gateway - 192.168.10.1]
        Taupi[Taupi Fan Shelly - 192.168.10.168]
    end

    Exporter -- "Queries 192.168.10.1" --> HostNet
    HostNet -- "Route: 192.168.10.0/24 via wg0" --> WgPod
    WgPod -- "WireGuard VPN Tunnel (51820/UDP)" --> FB
    FB -- "Accesses local subnet" --> Taupi

Components & Routing

  1. Home Network: 192.168.10.0/24. Contains the home devices (e.g., Shelly plug at 192.168.10.168) and the gateway FRITZ!Box at 192.168.10.1.
  2. Kubernetes Cluster Network: Pod subnets (10.233.64.0/24 etc.) running on the remote public server (136.243.23.215).
  3. WireGuard VPN Link (wg0):
    • The wireguard pod in the wireguard namespace is configured with hostNetwork: true. This exposes the wg0 interface directly in the host's root network namespace.
    • The cluster host has a static tunnel IP of 192.168.11.1/24 on wg0.
    • The FRITZ!Box acts as the active peer client, initiating the connection to the host node at 136.243.23.215:51820.
    • The host routing table directs home network traffic over the tunnel: 
      192.168.10.0/24 dev wg0 scope link
    • Pods inside the cluster (like fritzbox-exporter) query 192.168.10.1. The traffic is forwarded by the host's default CNI routing to the host's network namespace, which matching the 192.168.10.0/24 subnet route and sends it over wg0 to the FRITZ!Box.

FritzBox Exporter Egress Reject Workaround (Option B)

The Problem: When the fritzbox-exporter pod queries the FRITZ!Box TR-064 API at 192.168.10.1 from outside the home LAN subnet (using the transit WireGuard subnet IP 192.168.11.1), the FRITZ!Box's internal security policy redirects the client to its public WAN IP (e.g. 212.42.244.122) for portal login/authentication (login_sid.lua). When the dynamic WAN IP of the FRITZ!Box changes or the WAN ports are closed, these queries to the public IP timeout. Because Go's default HTTP client doesn't enforce a timeout, the exporter hangs for Go/Linux's default TCP connection handshake timeout of 2 minutes (120 seconds). This blocks all other metric scrapes and causes Prometheus target scrape timeouts.

The Solution: Rather than disabling CPU/RAM/temperature metrics (-nolua) or modifying the remote FRITZ!Box WireGuard subnet configuration, we block the exporter pod from reaching the public WAN IP.

  • We add an initContainer in fritzbox-exporter.yaml running with NET_ADMIN privileges.
  • The initContainer installs iptables and adds rules directly to the pod's shared network namespace.
  • It permits traffic to local private subnets (RFC1918 ranges) but rejects all TCP egress to public WAN IPs on port 80/443 with a TCP RST (Reset) immediately.
  • This causes redirected authentication requests to fail in milliseconds instead of hanging for 2 minutes. The exporter immediately falls back to scrape TR-064 metrics (WAN sync speed, bytes) successfully, avoiding Prometheus scrapes timeouts.