monitoring: optimize alerts and mitigate fritzbox-exporter timeouts

k8s/AGENTS.md

@@ -160,3 +160,53 @@ spec:
 *   **Namespaces**: Every application gets its own namespace.
 *   **Secrets**: Encrypt all secrets using `git-crypt`.
 
+## Network Setup (VPN & Routing)
+
+The Kubernetes cluster (Haumdaucher) is connected to the home network via a dedicated WireGuard VPN tunnel.
+
+```mermaid
+graph TD
+    subgraph "Kubernetes Cluster (haumdaucher.de - 136.243.23.215)"
+        subgraph "monitoring namespace"
+            Exporter[fritzbox-exporter Pod]
+        end
+        HostNet[Host Network Namespace]
+        WgPod[wireguard Pod hostNetwork: true]
+    end
+
+    subgraph "Home Network (192.168.10.0/24)"
+        FB[FRITZ!Box Gateway - 192.168.10.1]
+        Taupi[Taupi Fan Shelly - 192.168.10.168]
+    end
+
+    Exporter -- "Queries 192.168.10.1" --> HostNet
+    HostNet -- "Route: 192.168.10.0/24 via wg0" --> WgPod
+    WgPod -- "WireGuard VPN Tunnel (51820/UDP)" --> FB
+    FB -- "Accesses local subnet" --> Taupi
+```
+
+### Components & Routing
+1. **Home Network**: `192.168.10.0/24`. Contains the home devices (e.g., Shelly plug at `192.168.10.168`) and the gateway FRITZ!Box at `192.168.10.1`.
+2. **Kubernetes Cluster Network**: Pod subnets (`10.233.64.0/24` etc.) running on the remote public server (`136.243.23.215`).
+3. **WireGuard VPN Link (`wg0`)**:
+   - The `wireguard` pod in the `wireguard` namespace is configured with `hostNetwork: true`. This exposes the `wg0` interface directly in the host's root network namespace.
+   - The cluster host has a static tunnel IP of `192.168.11.1/24` on `wg0`.
+   - The FRITZ!Box acts as the active peer client, initiating the connection to the host node at `136.243.23.215:51820`.
+   - The host routing table directs home network traffic over the tunnel:
+     ```bash
+     192.168.10.0/24 dev wg0 scope link
+     ```
+   - Pods inside the cluster (like `fritzbox-exporter`) query `192.168.10.1`. The traffic is forwarded by the host's default CNI routing to the host's network namespace, which matching the `192.168.10.0/24` subnet route and sends it over `wg0` to the FRITZ!Box.
+
+### FritzBox Exporter Egress Reject Workaround (Option B)
+
+**The Problem**:
+When the `fritzbox-exporter` pod queries the FRITZ!Box TR-064 API at `192.168.10.1` from outside the home LAN subnet (using the transit WireGuard subnet IP `192.168.11.1`), the FRITZ!Box's internal security policy redirects the client to its public WAN IP (e.g. `212.42.244.122`) for portal login/authentication (`login_sid.lua`). 
+When the dynamic WAN IP of the FRITZ!Box changes or the WAN ports are closed, these queries to the public IP timeout. Because Go's default HTTP client doesn't enforce a timeout, the exporter hangs for Go/Linux's default TCP connection handshake timeout of **2 minutes (120 seconds)**. This blocks all other metric scrapes and causes Prometheus target scrape timeouts.
+
+**The Solution**:
+Rather than disabling CPU/RAM/temperature metrics (`-nolua`) or modifying the remote FRITZ!Box WireGuard subnet configuration, we block the exporter pod from reaching the public WAN IP.
+- We add an `initContainer` in `fritzbox-exporter.yaml` running with `NET_ADMIN` privileges.
+- The `initContainer` installs `iptables` and adds rules directly to the pod's shared network namespace.
+- It permits traffic to local private subnets (RFC1918 ranges) but rejects all TCP egress to public WAN IPs on port 80/443 with a `TCP RST` (Reset) immediately.
+- This causes redirected authentication requests to fail in milliseconds instead of hanging for 2 minutes. The exporter immediately falls back to scrape TR-064 metrics (WAN sync speed, bytes) successfully, avoiding Prometheus scrapes timeouts.

k8s/README.md

@@ -371,6 +371,15 @@ for i in "${NAMESPACES_TO_ALERT[@]}"; do
 done
 ```
 
+### Checking alerts history
+
+A Python script is available to retrieve and analyze the Prometheus alert history (for the last 14 days) by querying the cluster's Prometheus API through the Grafana pod:
+
+```bash
+python3 k8s/monitoring/scripts/analyze_alerts.py
+```
+
+
 ### influxdb
 
 Used to store hass data long term.

k8s/monitoring/fritzbox-exporter.yaml

@@ -15,6 +15,34 @@ spec:
       labels:
         app: fritzbox-exporter
     spec:
+      initContainers:
+      # This initContainer implements the Egress Reject Workaround (Option B).
+      # The FRITZ!Box TR-064 API redirects login_sid.lua calls to its public WAN IP because
+      # we query it from a different subnet (192.168.11.1). When the WAN IP is dynamic or port
+      # 80/443 are closed, the HTTP client hangs for the default 2-minute Linux TCP timeout,
+      # failing the entire Prometheus scrape.
+      # This container injects iptables rules into the pod's network namespace to instantly
+      # reject (TCP RST) egress to any public WAN IPs on port 80/443. The exporter will immediately
+      # fail the login step (in ms) and continue scraping TR-064 metrics on port 49000 (which is allowed).
+      - name: init-iptables
+        image: alpine:latest
+        securityContext:
+          capabilities:
+            add:
+              - NET_ADMIN
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            apk add --no-cache iptables
+            # Allow all local RFC1918 subnets and loopback
+            iptables -A OUTPUT -d 127.0.0.1/32 -j ACCEPT
+            iptables -A OUTPUT -d 10.0.0.0/8 -j ACCEPT
+            iptables -A OUTPUT -d 172.16.0.0/12 -j ACCEPT
+            iptables -A OUTPUT -d 192.168.0.0/16 -j ACCEPT
+            # Reject port 80 and 443 egress to any public IP (non-RFC1918)
+            iptables -A OUTPUT -p tcp --dport 80 -j REJECT --reject-with tcp-reset
+            iptables -A OUTPUT -p tcp --dport 443 -j REJECT --reject-with tcp-reset
+            echo "Egress iptables REJECT rules injected successfully"
       containers:
       - name: fritzbox-exporter
         image: ghcr.io/sberk42/fritzbox_exporter/fritzbox_exporter:latest
@@ -95,7 +123,7 @@ spec:
     rules:
     - alert: FritzBoxExporterOffline
       expr: up{job="fritzbox-exporter"} == 0
-      for: 5m
+      for: 1h
       labels:
         severity: warning
       annotations:

k8s/monitoring/scripts/analyze_alerts.py

@@ -0,0 +1,168 @@
+#!/usr/bin/env python3
+import subprocess
+import json
+import urllib.parse
+import datetime
+import os
+import sys
+
+def get_grafana_pod():
+    cmd = ["kubectl", "get", "pods", "-n", "monitoring", "-l", "app.kubernetes.io/name=grafana", "-o", "jsonpath={.items[0].metadata.name}"]
+    res = subprocess.run(cmd, capture_output=True, text=True)
+    if res.returncode == 0 and res.stdout.strip():
+        return res.stdout.strip()
+    return "prometheus-operator-grafana-7f66b47fc6-dtdtg"
+
+GRAFANA_POD = get_grafana_pod()
+
+def query_range_kubectl(query, start_time, end_time, step):
+    params = {
+        "query": query,
+        "start": start_time.isoformat(),
+        "end": end_time.isoformat(),
+        "step": step
+    }
+    query_str = urllib.parse.urlencode(params)
+    url = f"http://prometheus-operated:9090/api/v1/query_range?{query_str}"
+    
+    cmd = [
+        "kubectl", "-n", "monitoring", "exec",
+        GRAFANA_POD,
+        "-c", "grafana", "--",
+        "curl", "-s", url
+    ]
+    try:
+        res = subprocess.run(cmd, capture_output=True, text=True, check=True)
+        data = json.loads(res.stdout)
+        if data.get("status") == "success":
+            return data.get("data", {}).get("result", [])
+        else:
+            print(f"Prometheus API error: {data}", file=sys.stderr)
+    except Exception as e:
+        print(f"Failed to query via kubectl: {e}", file=sys.stderr)
+    return []
+
+def format_duration(seconds):
+    days = int(seconds // (24 * 3600))
+    seconds %= (24 * 3600)
+    hours = int(seconds // 3600)
+    seconds %= 3600
+    minutes = int(seconds // 60)
+    seconds = int(seconds % 60)
+    
+    parts = []
+    if days > 0: parts.append(f"{days}d")
+    if hours > 0: parts.append(f"{hours}h")
+    if minutes > 0: parts.append(f"{minutes}m")
+    if seconds > 0 or not parts: parts.append(f"{seconds}s")
+    return " ".join(parts)
+
+def parse_periods(values, step_seconds):
+    if not values:
+        return []
+    
+    sorted_values = sorted([(float(ts), int(float(val))) for ts, val in values], key=lambda x: x[0])
+    
+    periods = []
+    current_start = None
+    last_ts = None
+    
+    for ts, val in sorted_values:
+        if val != 1:
+            if current_start is not None:
+                periods.append({
+                    "start": current_start,
+                    "end": last_ts,
+                    "duration": last_ts - current_start + step_seconds,
+                    "ongoing": False
+                })
+                current_start = None
+            continue
+            
+        if current_start is None:
+            current_start = ts
+        else:
+            if ts - last_ts > step_seconds * 3:
+                periods.append({
+                    "start": current_start,
+                    "end": last_ts,
+                    "duration": last_ts - current_start + step_seconds,
+                    "ongoing": False
+                })
+                current_start = ts
+        last_ts = ts
+        
+    if current_start is not None:
+        periods.append({
+            "start": current_start,
+            "end": last_ts,
+            "duration": last_ts - current_start + step_seconds,
+            "ongoing": True
+        })
+        
+    return periods
+
+def main():
+    print(f"Querying Prometheus from pod {GRAFANA_POD}...")
+    
+    now = datetime.datetime.now(datetime.timezone.utc)
+    start_time = now - datetime.timedelta(days=14)
+    step = "2m"
+    step_seconds = 120
+    
+    print(f"Analyzing alert history from {start_time.strftime('%Y-%m-%d %H:%M:%S')} to {now.strftime('%Y-%m-%d %H:%M:%S')} UTC...")
+    
+    # Query ALERTS metric
+    results = query_range_kubectl('ALERTS', start_time, now, step)
+    
+    if not results:
+        print("No alerts found in the specified range.")
+        return
+        
+    all_events = []
+    
+    for result in results:
+        metric = result.get("metric", {})
+        values = result.get("values", [])
+        
+        alert_name = metric.get("alertname", "Unknown")
+        alert_state = metric.get("alertstate", "Unknown")
+        
+        # Extract target labels that distinguish this alert instance
+        ignored_keys = {"__name__", "alertname", "alertstate"}
+        labels = {k: v for k, v in metric.items() if k not in ignored_keys}
+        
+        periods = parse_periods(values, step_seconds)
+        for p in periods:
+            all_events.append({
+                "alertname": alert_name,
+                "alertstate": alert_state,
+                "labels": labels,
+                "start": p["start"],
+                "end": p["end"],
+                "duration": p["duration"],
+                "ongoing": p.get("ongoing", False)
+            })
+            
+    # Sort events by start time descending
+    all_events.sort(key=lambda x: x["start"], reverse=True)
+    
+    print(f"\nFound {len(all_events)} alert events in the last 14 days:")
+    print("-" * 100)
+    
+    # Print summary table
+    for i, ev in enumerate(all_events):
+        start_dt = datetime.datetime.fromtimestamp(ev["start"], datetime.timezone.utc)
+        end_dt = datetime.datetime.fromtimestamp(ev["end"], datetime.timezone.utc)
+        ongoing_str = " (Ongoing)" if ev["ongoing"] else ""
+        
+        labels_str = ", ".join([f"{k}={v}" for k, v in ev["labels"].items()])
+        
+        print(f"[{i+1}] {ev['alertname']} ({ev['alertstate']})")
+        print(f"    Duration: {format_duration(ev['duration'])}{ongoing_str}")
+        print(f"    Timeline: {start_dt.strftime('%Y-%m-%d %H:%M:%S')} -> {end_dt.strftime('%Y-%m-%d %H:%M:%S')} UTC")
+        print(f"    Labels:   {labels_str}")
+        print("-" * 100)
+
+if __name__ == "__main__":
+    main()

k8s/monitoring/taupi-fan.yml

@@ -59,7 +59,7 @@ spec:
       rules:
         - alert: TaupiFanOffline
           expr: up{job="taupi-fan"} == 0
-          for: 8h
+          for: 1h
           labels:
             severity: critical
           annotations:
@@ -67,7 +67,7 @@ spec:
             description: "The Shelly plug at 192.168.10.168 is unreachable by Prometheus. Check Wi-Fi connection or power status."
         - alert: TaupiFanSensorsStale
           expr: taupi_lost_connection_seconds_innen > 600 or taupi_lost_connection_seconds_aussen > 600
-          for: 5m
+          for: 1h
           labels:
             severity: warning
           annotations:
@@ -75,7 +75,7 @@ spec:
             description: "The BLE sensors for temperature and humidity inside or outside have not sent new data for over 10 minutes."
         - alert: TaupiFanMoldDanger
           expr: taupi_is_critical == 1
-          for: 30m
+          for: 1h
           labels:
             severity: warning
           annotations:

k8s/n8n/n8n.secret.yml

@@ -146,15 +146,15 @@ extraManifests:
           rules:
             - alert: n8nInstanceDown
               expr: up{job="mop-n8n"} == 0
-              for: 5m
+              for: 1h
               labels:
                 severity: critical
               annotations:
                 summary: "n8n instance is down or unresponsive"
-                description: "n8n scraper has failed for the last 5 minutes. The application might be frozen or crashed."
+                description: "n8n scraper has failed for the last 1 hour. The application might be frozen or crashed."
             - alert: n8nPodRestarts
               expr: rate(kube_pod_container_status_restarts_total{container="n8n"}[15m]) * 900 > 1
-              for: 5m
+              for: 1h
               labels:
                 severity: warning
               annotations:
@@ -162,17 +162,17 @@ extraManifests:
                 description: "n8n has restarted in the last 15 minutes. This might indicate liveness probe failures due to database issues or memory limit exhaustion."
             - alert: n8nNodeEventLoopLag
               expr: n8n_nodejs_eventloop_lag_seconds{job="mop-n8n"} > 1
-              for: 5m
+              for: 1h
               labels:
                 severity: warning
               annotations:
                 summary: "n8n event loop lag is high"
-                description: "n8n Node.js event loop lag has exceeded 1 second for the last 5 minutes. This can make the UI unresponsive and logins fail."
+                description: "n8n Node.js event loop lag has exceeded 1 second for the last 1 hour. This can make the UI unresponsive and logins fail."
             - alert: n8nPodNotReady
               expr: kube_pod_status_ready{condition="true", pod=~"mop-n8n-.*"} == 0
-              for: 5m
+              for: 1h
               labels:
                 severity: critical
               annotations:
                 summary: "n8n pod is not ready"
-                description: "n8n pod has been in non-ready state for more than 5 minutes. This is likely due to failing readiness probes (/healthz check failing, possibly database connection issues)."
+                description: "n8n pod has been in non-ready state for more than 1 hour. This is likely due to failing readiness probes (/healthz check failing, possibly database connection issues)."

k8s/wireguard/monitoring.yaml

@@ -47,7 +47,7 @@ spec:
       rules:
         - alert: WireguardExporterDown
           expr: up{job="wireguard-exporter"} == 0
-          for: 5m
+          for: 1h
           labels:
             severity: critical
           annotations:
@@ -55,7 +55,7 @@ spec:
             description: "The WireGuard Prometheus exporter is unreachable. The pod may have crashed or is unresponsive."
         - alert: WireguardInterfaceDown
           expr: absent(wireguard_sent_bytes_total{interface="wg0"}) == 1
-          for: 5m
+          for: 1h
           labels:
             severity: critical
           annotations:
@@ -63,7 +63,7 @@ spec:
             description: "The WireGuard interface wg0 is not reporting any statistics. The VPN tunnel might be down or inactive."
         - alert: WireguardPeerOffline
           expr: (time() - wireguard_latest_handshake_seconds{interface="wg0"}) > 300
-          for: 5m
+          for: 1h
           labels:
             severity: critical
           annotations: