diff --git a/k8s/AGENTS.md b/k8s/AGENTS.md index 390bf73..7917116 100644 --- a/k8s/AGENTS.md +++ b/k8s/AGENTS.md @@ -160,3 +160,53 @@ spec: * **Namespaces**: Every application gets its own namespace. * **Secrets**: Encrypt all secrets using `git-crypt`. +## Network Setup (VPN & Routing) + +The Kubernetes cluster (Haumdaucher) is connected to the home network via a dedicated WireGuard VPN tunnel. + +```mermaid +graph TD + subgraph "Kubernetes Cluster (haumdaucher.de - 136.243.23.215)" + subgraph "monitoring namespace" + Exporter[fritzbox-exporter Pod] + end + HostNet[Host Network Namespace] + WgPod[wireguard Pod hostNetwork: true] + end + + subgraph "Home Network (192.168.10.0/24)" + FB[FRITZ!Box Gateway - 192.168.10.1] + Taupi[Taupi Fan Shelly - 192.168.10.168] + end + + Exporter -- "Queries 192.168.10.1" --> HostNet + HostNet -- "Route: 192.168.10.0/24 via wg0" --> WgPod + WgPod -- "WireGuard VPN Tunnel (51820/UDP)" --> FB + FB -- "Accesses local subnet" --> Taupi +``` + +### Components & Routing +1. **Home Network**: `192.168.10.0/24`. Contains the home devices (e.g., Shelly plug at `192.168.10.168`) and the gateway FRITZ!Box at `192.168.10.1`. +2. **Kubernetes Cluster Network**: Pod subnets (`10.233.64.0/24` etc.) running on the remote public server (`136.243.23.215`). +3. **WireGuard VPN Link (`wg0`)**: + - The `wireguard` pod in the `wireguard` namespace is configured with `hostNetwork: true`. This exposes the `wg0` interface directly in the host's root network namespace. + - The cluster host has a static tunnel IP of `192.168.11.1/24` on `wg0`. + - The FRITZ!Box acts as the active peer client, initiating the connection to the host node at `136.243.23.215:51820`. + - The host routing table directs home network traffic over the tunnel: + ```bash + 192.168.10.0/24 dev wg0 scope link + ``` + - Pods inside the cluster (like `fritzbox-exporter`) query `192.168.10.1`. The traffic is forwarded by the host's default CNI routing to the host's network namespace, which matching the `192.168.10.0/24` subnet route and sends it over `wg0` to the FRITZ!Box. + +### FritzBox Exporter Egress Reject Workaround (Option B) + +**The Problem**: +When the `fritzbox-exporter` pod queries the FRITZ!Box TR-064 API at `192.168.10.1` from outside the home LAN subnet (using the transit WireGuard subnet IP `192.168.11.1`), the FRITZ!Box's internal security policy redirects the client to its public WAN IP (e.g. `212.42.244.122`) for portal login/authentication (`login_sid.lua`). +When the dynamic WAN IP of the FRITZ!Box changes or the WAN ports are closed, these queries to the public IP timeout. Because Go's default HTTP client doesn't enforce a timeout, the exporter hangs for Go/Linux's default TCP connection handshake timeout of **2 minutes (120 seconds)**. This blocks all other metric scrapes and causes Prometheus target scrape timeouts. + +**The Solution**: +Rather than disabling CPU/RAM/temperature metrics (`-nolua`) or modifying the remote FRITZ!Box WireGuard subnet configuration, we block the exporter pod from reaching the public WAN IP. +- We add an `initContainer` in `fritzbox-exporter.yaml` running with `NET_ADMIN` privileges. +- The `initContainer` installs `iptables` and adds rules directly to the pod's shared network namespace. +- It permits traffic to local private subnets (RFC1918 ranges) but rejects all TCP egress to public WAN IPs on port 80/443 with a `TCP RST` (Reset) immediately. +- This causes redirected authentication requests to fail in milliseconds instead of hanging for 2 minutes. The exporter immediately falls back to scrape TR-064 metrics (WAN sync speed, bytes) successfully, avoiding Prometheus scrapes timeouts. diff --git a/k8s/README.md b/k8s/README.md index 3cbd8f9..1bb3069 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -371,6 +371,15 @@ for i in "${NAMESPACES_TO_ALERT[@]}"; do done ``` +### Checking alerts history + +A Python script is available to retrieve and analyze the Prometheus alert history (for the last 14 days) by querying the cluster's Prometheus API through the Grafana pod: + +```bash +python3 k8s/monitoring/scripts/analyze_alerts.py +``` + + ### influxdb Used to store hass data long term. diff --git a/k8s/home-assistant/prometheusrules.secret.yaml b/k8s/home-assistant/prometheusrules.secret.yaml index 752e2d4..bcea717 100644 Binary files a/k8s/home-assistant/prometheusrules.secret.yaml and b/k8s/home-assistant/prometheusrules.secret.yaml differ diff --git a/k8s/influxdb/monitoring.secret.yaml b/k8s/influxdb/monitoring.secret.yaml index a975a8f..9a2dc5d 100644 Binary files a/k8s/influxdb/monitoring.secret.yaml and b/k8s/influxdb/monitoring.secret.yaml differ diff --git a/k8s/monitoring/alertmanagerconfig.secret.yaml b/k8s/monitoring/alertmanagerconfig.secret.yaml index df713f4..0f54b64 100644 Binary files a/k8s/monitoring/alertmanagerconfig.secret.yaml and b/k8s/monitoring/alertmanagerconfig.secret.yaml differ diff --git a/k8s/monitoring/fritzbox-exporter.yaml b/k8s/monitoring/fritzbox-exporter.yaml index 32d760c..13b7394 100644 --- a/k8s/monitoring/fritzbox-exporter.yaml +++ b/k8s/monitoring/fritzbox-exporter.yaml @@ -15,6 +15,34 @@ spec: labels: app: fritzbox-exporter spec: + initContainers: + # This initContainer implements the Egress Reject Workaround (Option B). + # The FRITZ!Box TR-064 API redirects login_sid.lua calls to its public WAN IP because + # we query it from a different subnet (192.168.11.1). When the WAN IP is dynamic or port + # 80/443 are closed, the HTTP client hangs for the default 2-minute Linux TCP timeout, + # failing the entire Prometheus scrape. + # This container injects iptables rules into the pod's network namespace to instantly + # reject (TCP RST) egress to any public WAN IPs on port 80/443. The exporter will immediately + # fail the login step (in ms) and continue scraping TR-064 metrics on port 49000 (which is allowed). + - name: init-iptables + image: alpine:latest + securityContext: + capabilities: + add: + - NET_ADMIN + command: ["/bin/sh", "-c"] + args: + - | + apk add --no-cache iptables + # Allow all local RFC1918 subnets and loopback + iptables -A OUTPUT -d 127.0.0.1/32 -j ACCEPT + iptables -A OUTPUT -d 10.0.0.0/8 -j ACCEPT + iptables -A OUTPUT -d 172.16.0.0/12 -j ACCEPT + iptables -A OUTPUT -d 192.168.0.0/16 -j ACCEPT + # Reject port 80 and 443 egress to any public IP (non-RFC1918) + iptables -A OUTPUT -p tcp --dport 80 -j REJECT --reject-with tcp-reset + iptables -A OUTPUT -p tcp --dport 443 -j REJECT --reject-with tcp-reset + echo "Egress iptables REJECT rules injected successfully" containers: - name: fritzbox-exporter image: ghcr.io/sberk42/fritzbox_exporter/fritzbox_exporter:latest @@ -95,7 +123,7 @@ spec: rules: - alert: FritzBoxExporterOffline expr: up{job="fritzbox-exporter"} == 0 - for: 5m + for: 1h labels: severity: warning annotations: diff --git a/k8s/monitoring/scripts/analyze_alerts.py b/k8s/monitoring/scripts/analyze_alerts.py new file mode 100755 index 0000000..bb7ab15 --- /dev/null +++ b/k8s/monitoring/scripts/analyze_alerts.py @@ -0,0 +1,168 @@ +#!/usr/bin/env python3 +import subprocess +import json +import urllib.parse +import datetime +import os +import sys + +def get_grafana_pod(): + cmd = ["kubectl", "get", "pods", "-n", "monitoring", "-l", "app.kubernetes.io/name=grafana", "-o", "jsonpath={.items[0].metadata.name}"] + res = subprocess.run(cmd, capture_output=True, text=True) + if res.returncode == 0 and res.stdout.strip(): + return res.stdout.strip() + return "prometheus-operator-grafana-7f66b47fc6-dtdtg" + +GRAFANA_POD = get_grafana_pod() + +def query_range_kubectl(query, start_time, end_time, step): + params = { + "query": query, + "start": start_time.isoformat(), + "end": end_time.isoformat(), + "step": step + } + query_str = urllib.parse.urlencode(params) + url = f"http://prometheus-operated:9090/api/v1/query_range?{query_str}" + + cmd = [ + "kubectl", "-n", "monitoring", "exec", + GRAFANA_POD, + "-c", "grafana", "--", + "curl", "-s", url + ] + try: + res = subprocess.run(cmd, capture_output=True, text=True, check=True) + data = json.loads(res.stdout) + if data.get("status") == "success": + return data.get("data", {}).get("result", []) + else: + print(f"Prometheus API error: {data}", file=sys.stderr) + except Exception as e: + print(f"Failed to query via kubectl: {e}", file=sys.stderr) + return [] + +def format_duration(seconds): + days = int(seconds // (24 * 3600)) + seconds %= (24 * 3600) + hours = int(seconds // 3600) + seconds %= 3600 + minutes = int(seconds // 60) + seconds = int(seconds % 60) + + parts = [] + if days > 0: parts.append(f"{days}d") + if hours > 0: parts.append(f"{hours}h") + if minutes > 0: parts.append(f"{minutes}m") + if seconds > 0 or not parts: parts.append(f"{seconds}s") + return " ".join(parts) + +def parse_periods(values, step_seconds): + if not values: + return [] + + sorted_values = sorted([(float(ts), int(float(val))) for ts, val in values], key=lambda x: x[0]) + + periods = [] + current_start = None + last_ts = None + + for ts, val in sorted_values: + if val != 1: + if current_start is not None: + periods.append({ + "start": current_start, + "end": last_ts, + "duration": last_ts - current_start + step_seconds, + "ongoing": False + }) + current_start = None + continue + + if current_start is None: + current_start = ts + else: + if ts - last_ts > step_seconds * 3: + periods.append({ + "start": current_start, + "end": last_ts, + "duration": last_ts - current_start + step_seconds, + "ongoing": False + }) + current_start = ts + last_ts = ts + + if current_start is not None: + periods.append({ + "start": current_start, + "end": last_ts, + "duration": last_ts - current_start + step_seconds, + "ongoing": True + }) + + return periods + +def main(): + print(f"Querying Prometheus from pod {GRAFANA_POD}...") + + now = datetime.datetime.now(datetime.timezone.utc) + start_time = now - datetime.timedelta(days=14) + step = "2m" + step_seconds = 120 + + print(f"Analyzing alert history from {start_time.strftime('%Y-%m-%d %H:%M:%S')} to {now.strftime('%Y-%m-%d %H:%M:%S')} UTC...") + + # Query ALERTS metric + results = query_range_kubectl('ALERTS', start_time, now, step) + + if not results: + print("No alerts found in the specified range.") + return + + all_events = [] + + for result in results: + metric = result.get("metric", {}) + values = result.get("values", []) + + alert_name = metric.get("alertname", "Unknown") + alert_state = metric.get("alertstate", "Unknown") + + # Extract target labels that distinguish this alert instance + ignored_keys = {"__name__", "alertname", "alertstate"} + labels = {k: v for k, v in metric.items() if k not in ignored_keys} + + periods = parse_periods(values, step_seconds) + for p in periods: + all_events.append({ + "alertname": alert_name, + "alertstate": alert_state, + "labels": labels, + "start": p["start"], + "end": p["end"], + "duration": p["duration"], + "ongoing": p.get("ongoing", False) + }) + + # Sort events by start time descending + all_events.sort(key=lambda x: x["start"], reverse=True) + + print(f"\nFound {len(all_events)} alert events in the last 14 days:") + print("-" * 100) + + # Print summary table + for i, ev in enumerate(all_events): + start_dt = datetime.datetime.fromtimestamp(ev["start"], datetime.timezone.utc) + end_dt = datetime.datetime.fromtimestamp(ev["end"], datetime.timezone.utc) + ongoing_str = " (Ongoing)" if ev["ongoing"] else "" + + labels_str = ", ".join([f"{k}={v}" for k, v in ev["labels"].items()]) + + print(f"[{i+1}] {ev['alertname']} ({ev['alertstate']})") + print(f" Duration: {format_duration(ev['duration'])}{ongoing_str}") + print(f" Timeline: {start_dt.strftime('%Y-%m-%d %H:%M:%S')} -> {end_dt.strftime('%Y-%m-%d %H:%M:%S')} UTC") + print(f" Labels: {labels_str}") + print("-" * 100) + +if __name__ == "__main__": + main() diff --git a/k8s/monitoring/taupi-fan.yml b/k8s/monitoring/taupi-fan.yml index 00f15a1..bb50001 100644 --- a/k8s/monitoring/taupi-fan.yml +++ b/k8s/monitoring/taupi-fan.yml @@ -59,7 +59,7 @@ spec: rules: - alert: TaupiFanOffline expr: up{job="taupi-fan"} == 0 - for: 8h + for: 1h labels: severity: critical annotations: @@ -67,7 +67,7 @@ spec: description: "The Shelly plug at 192.168.10.168 is unreachable by Prometheus. Check Wi-Fi connection or power status." - alert: TaupiFanSensorsStale expr: taupi_lost_connection_seconds_innen > 600 or taupi_lost_connection_seconds_aussen > 600 - for: 5m + for: 1h labels: severity: warning annotations: @@ -75,7 +75,7 @@ spec: description: "The BLE sensors for temperature and humidity inside or outside have not sent new data for over 10 minutes." - alert: TaupiFanMoldDanger expr: taupi_is_critical == 1 - for: 30m + for: 1h labels: severity: warning annotations: diff --git a/k8s/n8n/n8n.secret.yml b/k8s/n8n/n8n.secret.yml index 5eb77dd..307ab52 100644 --- a/k8s/n8n/n8n.secret.yml +++ b/k8s/n8n/n8n.secret.yml @@ -146,15 +146,15 @@ extraManifests: rules: - alert: n8nInstanceDown expr: up{job="mop-n8n"} == 0 - for: 5m + for: 1h labels: severity: critical annotations: summary: "n8n instance is down or unresponsive" - description: "n8n scraper has failed for the last 5 minutes. The application might be frozen or crashed." + description: "n8n scraper has failed for the last 1 hour. The application might be frozen or crashed." - alert: n8nPodRestarts expr: rate(kube_pod_container_status_restarts_total{container="n8n"}[15m]) * 900 > 1 - for: 5m + for: 1h labels: severity: warning annotations: @@ -162,17 +162,17 @@ extraManifests: description: "n8n has restarted in the last 15 minutes. This might indicate liveness probe failures due to database issues or memory limit exhaustion." - alert: n8nNodeEventLoopLag expr: n8n_nodejs_eventloop_lag_seconds{job="mop-n8n"} > 1 - for: 5m + for: 1h labels: severity: warning annotations: summary: "n8n event loop lag is high" - description: "n8n Node.js event loop lag has exceeded 1 second for the last 5 minutes. This can make the UI unresponsive and logins fail." + description: "n8n Node.js event loop lag has exceeded 1 second for the last 1 hour. This can make the UI unresponsive and logins fail." - alert: n8nPodNotReady expr: kube_pod_status_ready{condition="true", pod=~"mop-n8n-.*"} == 0 - for: 5m + for: 1h labels: severity: critical annotations: summary: "n8n pod is not ready" - description: "n8n pod has been in non-ready state for more than 5 minutes. This is likely due to failing readiness probes (/healthz check failing, possibly database connection issues)." \ No newline at end of file + description: "n8n pod has been in non-ready state for more than 1 hour. This is likely due to failing readiness probes (/healthz check failing, possibly database connection issues)." \ No newline at end of file diff --git a/k8s/wireguard/monitoring.yaml b/k8s/wireguard/monitoring.yaml index 4dcbd7b..4e10ac5 100644 --- a/k8s/wireguard/monitoring.yaml +++ b/k8s/wireguard/monitoring.yaml @@ -47,7 +47,7 @@ spec: rules: - alert: WireguardExporterDown expr: up{job="wireguard-exporter"} == 0 - for: 5m + for: 1h labels: severity: critical annotations: @@ -55,7 +55,7 @@ spec: description: "The WireGuard Prometheus exporter is unreachable. The pod may have crashed or is unresponsive." - alert: WireguardInterfaceDown expr: absent(wireguard_sent_bytes_total{interface="wg0"}) == 1 - for: 5m + for: 1h labels: severity: critical annotations: @@ -63,7 +63,7 @@ spec: description: "The WireGuard interface wg0 is not reporting any statistics. The VPN tunnel might be down or inactive." - alert: WireguardPeerOffline expr: (time() - wireguard_latest_handshake_seconds{interface="wg0"}) > 300 - for: 5m + for: 1h labels: severity: critical annotations: