213 lines
8.7 KiB
Markdown
213 lines
8.7 KiB
Markdown
# AGENTS.md
|
|
|
|
> [!NOTE]
|
|
> This file describes the constraints and conventions for the `k8s` directory, which contains deployments for the **haumdaucher.de** Kubernetes cluster.
|
|
|
|
## Project Overview
|
|
This directory contains the Kubernetes manifests and Helm charts for a single-node Kubernetes cluster (Haumdaucher).
|
|
* **Domain**: `*.haumdaucher.de`
|
|
* **Orchestration**: Self-managed Kubernetes (single node).
|
|
* **Ingress**: `ingress-nginx`
|
|
* **SSL**: `cert-manager` (LetsEncrypt)
|
|
|
|
## Directory Structure
|
|
* **Top-level folders**: Each folder corresponds to a Kubernetes **namespace**.
|
|
* Example: `mailu/` contains resources for the `mailu` namespace.
|
|
* **Documentation**: `README.md` is the **authoritative source** for deployment commands. Always check it before running commands.
|
|
|
|
## Code Style & Conventions
|
|
* **Helm Version**: Helm 3 (`helm`) is used.
|
|
* **Implementation Order**: Top-down.
|
|
* **Naming**: Namespaces matches folder names.
|
|
* **Formatting**: Standard YAML conventions.
|
|
|
|
## Security & Secrets
|
|
> [!IMPORTANT]
|
|
> **Git-Crypt is enforced.**
|
|
> Do not touch encrypted files unless you have the key and know how to unlock them.
|
|
|
|
**Encrypted File Patterns**:
|
|
* `*.secret`
|
|
* `*.secret.yaml`
|
|
* `*.secret.values`
|
|
* `*.secret.sh`
|
|
|
|
## Remote Access
|
|
It is possible to execute commands on the remote Linux node for information retrieval or troubleshooting.
|
|
|
|
* **Host**: `haumdaucher.de`
|
|
* **User**: `moritz` (local user)
|
|
* **Privileges**: Use `sudo` to gain root privileges.
|
|
|
|
> [!CAUTION]
|
|
> **SSH Identity Required**: The agent cannot enter an SSH passphrase.
|
|
> If SSH commands fail with authentication errors, request the user to run `ssh-add` locally to load their identity.
|
|
|
|
### Command Execution
|
|
You can execute commands remotely via SSH. This is useful for checking node-level resources (memory, disk, etc.) that `kubectl` might not expose directly.
|
|
|
|
**Example: Check Memory Usage**
|
|
```bash
|
|
ssh moritz@haumdaucher.de "free -h"
|
|
```
|
|
|
|
**Example: Check Disk Usage (with sudo)**
|
|
```bash
|
|
ssh -t moritz@haumdaucher.de "sudo df -h"
|
|
```
|
|
*Note: The `-t` flag forces pseudo-terminal allocation, which is often required for `sudo` prompts.*
|
|
|
|
|
|
## Deployment Instructions
|
|
**Always consult `README.md` first.** Deployments vary between Helm charts and raw manifests.
|
|
|
|
### Common Patterns
|
|
* **Helm**:
|
|
```bash
|
|
helm upgrade --install <release> <chart> -n <namespace> -f <values-file>
|
|
```
|
|
* **Kubectl**:
|
|
```bash
|
|
kubectl apply -f <folder>/<file>.yaml
|
|
```
|
|
|
|
### Post-Implementation Verification
|
|
> [!IMPORTANT]
|
|
> **Verification Workflow**:
|
|
> After a new implementation or configuration change, always:
|
|
> 1. Run `kubectl apply -f <file>.yaml`.
|
|
> 2. Run `kubectl rollout restart deployment <deployment-name> -n <namespace>` if applying a ConfigMap/Secret that a deployment depends on.
|
|
> 3. Wait for 30 seconds to allow pods to roll over.
|
|
> 4. Check logs using `kubectl logs -n <namespace> -l <label> --tail=100`.
|
|
>
|
|
> The agent **must always ask the user** for permission to execute this verification workflow after making changes.
|
|
|
|
### Operational Tasks
|
|
* **Cleanup Error Pods**:
|
|
```bash
|
|
kubectl get pods | grep Error | cut -d' ' -f 1 | xargs kubectl delete pod
|
|
```
|
|
|
|
## Ingress Configuration
|
|
Ingress resources **must** follow these strict conventions to work with the cluster's ingress controller (`traefik`) and certificate manager (`cert-manager`).
|
|
|
|
### Annotations
|
|
All Ingress resources must include:
|
|
```yaml
|
|
annotations:
|
|
kubernetes.io/ingress.class: "traefik"
|
|
cert-manager.io/cluster-issuer: "letsencrypt-prod"
|
|
kubernetes.io/tls-acme: "true"
|
|
# Standard nginx tweaks (if using dual class) or traefik configurations
|
|
nginx.ingress.kubernetes.io/proxy-body-size: "0"
|
|
nginx.ingress.kubernetes.io/ssl-redirect: "true"
|
|
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
|
|
```
|
|
|
|
### Hostnames & TLS
|
|
* **Domain**: Use a subdomain of `haumdaucher.de` or `moritzgraf.de`.
|
|
* **TLS Secret Name**: Must use **hyphens** instead of dots.
|
|
* **Pattern**: `<subdomain>-<domain>-<tld>`
|
|
* **Example**: `n8n.moritzgraf.de` -> `n8n-moritzgraf-de`
|
|
|
|
### Example
|
|
```yaml
|
|
spec:
|
|
ingressClassName: traefik
|
|
tls:
|
|
- hosts:
|
|
- n8n.moritzgraf.de
|
|
secretName: n8n-moritzgraf-de
|
|
rules:
|
|
- host: n8n.moritzgraf.de
|
|
http:
|
|
paths:
|
|
- path: /
|
|
pathType: Prefix
|
|
backend:
|
|
service:
|
|
name: n8n
|
|
port:
|
|
number: 5678
|
|
```
|
|
|
|
## Storage / Persistence
|
|
The cluster uses **OpenEBS** for dynamic local storage provisioning.
|
|
|
|
### PersistentVolumeClaims (PVC)
|
|
* **Provisioner**: `openebs.io/local` (or similar, managed via `openebs-hostpath`).
|
|
* **StorageClass**: `openebs-hostpath`.
|
|
* **AccessMode**: Typically `ReadWriteOnce` (RWO) as it's local storage.
|
|
|
|
To request storage, simply create a PVC or configure Helm charts to use the default storage class (or explicitly `openebs-hostpath`).
|
|
|
|
```yaml
|
|
kind: PersistentVolumeClaim
|
|
apiVersion: v1
|
|
metadata:
|
|
name: my-data
|
|
spec:
|
|
storageClassName: openebs-hostpath
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
```
|
|
|
|
## Deployment Constraints
|
|
* **Resources**: Always define `requests` and `limits` for CPU and Memory to ensure fair scheduling on the single node.
|
|
* **Namespaces**: Every application gets its own namespace.
|
|
* **Secrets**: Encrypt all secrets using `git-crypt`.
|
|
|
|
## Network Setup (VPN & Routing)
|
|
|
|
The Kubernetes cluster (Haumdaucher) is connected to the home network via a dedicated WireGuard VPN tunnel.
|
|
|
|
```mermaid
|
|
graph TD
|
|
subgraph "Kubernetes Cluster (haumdaucher.de - 136.243.23.215)"
|
|
subgraph "monitoring namespace"
|
|
Exporter[fritzbox-exporter Pod]
|
|
end
|
|
HostNet[Host Network Namespace]
|
|
WgPod[wireguard Pod hostNetwork: true]
|
|
end
|
|
|
|
subgraph "Home Network (192.168.10.0/24)"
|
|
FB[FRITZ!Box Gateway - 192.168.10.1]
|
|
Taupi[Taupi Fan Shelly - 192.168.10.168]
|
|
end
|
|
|
|
Exporter -- "Queries 192.168.10.1" --> HostNet
|
|
HostNet -- "Route: 192.168.10.0/24 via wg0" --> WgPod
|
|
WgPod -- "WireGuard VPN Tunnel (51820/UDP)" --> FB
|
|
FB -- "Accesses local subnet" --> Taupi
|
|
```
|
|
|
|
### Components & Routing
|
|
1. **Home Network**: `192.168.10.0/24`. Contains the home devices (e.g., Shelly plug at `192.168.10.168`) and the gateway FRITZ!Box at `192.168.10.1`.
|
|
2. **Kubernetes Cluster Network**: Pod subnets (`10.233.64.0/24` etc.) running on the remote public server (`136.243.23.215`).
|
|
3. **WireGuard VPN Link (`wg0`)**:
|
|
- The `wireguard` pod in the `wireguard` namespace is configured with `hostNetwork: true`. This exposes the `wg0` interface directly in the host's root network namespace.
|
|
- The cluster host has a static tunnel IP of `192.168.11.1/24` on `wg0`.
|
|
- The FRITZ!Box acts as the active peer client, initiating the connection to the host node at `136.243.23.215:51820`.
|
|
- The host routing table directs home network traffic over the tunnel:
|
|
```bash
|
|
192.168.10.0/24 dev wg0 scope link
|
|
```
|
|
- Pods inside the cluster (like `fritzbox-exporter`) query `192.168.10.1`. The traffic is forwarded by the host's default CNI routing to the host's network namespace, which matching the `192.168.10.0/24` subnet route and sends it over `wg0` to the FRITZ!Box.
|
|
|
|
### FritzBox Exporter Egress Reject Workaround (Option B)
|
|
|
|
**The Problem**:
|
|
When the `fritzbox-exporter` pod queries the FRITZ!Box TR-064 API at `192.168.10.1` from outside the home LAN subnet (using the transit WireGuard subnet IP `192.168.11.1`), the FRITZ!Box's internal security policy redirects the client to its public WAN IP (e.g. `212.42.244.122`) for portal login/authentication (`login_sid.lua`).
|
|
When the dynamic WAN IP of the FRITZ!Box changes or the WAN ports are closed, these queries to the public IP timeout. Because Go's default HTTP client doesn't enforce a timeout, the exporter hangs for Go/Linux's default TCP connection handshake timeout of **2 minutes (120 seconds)**. This blocks all other metric scrapes and causes Prometheus target scrape timeouts.
|
|
|
|
**The Solution**:
|
|
Rather than disabling CPU/RAM/temperature metrics (`-nolua`) or modifying the remote FRITZ!Box WireGuard subnet configuration, we block the exporter pod from reaching the public WAN IP.
|
|
- We add an `initContainer` in `fritzbox-exporter.yaml` running with `NET_ADMIN` privileges.
|
|
- The `initContainer` installs `iptables` and adds rules directly to the pod's shared network namespace.
|
|
- It permits traffic to local private subnets (RFC1918 ranges) but rejects all TCP egress to public WAN IPs on port 80/443 with a `TCP RST` (Reset) immediately.
|
|
- This causes redirected authentication requests to fail in milliseconds instead of hanging for 2 minutes. The exporter immediately falls back to scrape TR-064 metrics (WAN sync speed, bytes) successfully, avoiding Prometheus scrapes timeouts.
|