monitoring: optimize alerts and mitigate fritzbox-exporter timeouts
This commit is contained in:
Moritz Graf
parent
c5458821a7
commit
bdabc56afe
|
|
@ -160,3 +160,53 @@ spec:
|
|||
* **Namespaces**: Every application gets its own namespace.
|
||||
* **Secrets**: Encrypt all secrets using `git-crypt`.
|
||||
|
||||
## Network Setup (VPN & Routing)
|
||||
|
||||
The Kubernetes cluster (Haumdaucher) is connected to the home network via a dedicated WireGuard VPN tunnel.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Kubernetes Cluster (haumdaucher.de - 136.243.23.215)"
|
||||
subgraph "monitoring namespace"
|
||||
Exporter[fritzbox-exporter Pod]
|
||||
end
|
||||
HostNet[Host Network Namespace]
|
||||
WgPod[wireguard Pod hostNetwork: true]
|
||||
end
|
||||
|
||||
subgraph "Home Network (192.168.10.0/24)"
|
||||
FB[FRITZ!Box Gateway - 192.168.10.1]
|
||||
Taupi[Taupi Fan Shelly - 192.168.10.168]
|
||||
end
|
||||
|
||||
Exporter -- "Queries 192.168.10.1" --> HostNet
|
||||
HostNet -- "Route: 192.168.10.0/24 via wg0" --> WgPod
|
||||
WgPod -- "WireGuard VPN Tunnel (51820/UDP)" --> FB
|
||||
FB -- "Accesses local subnet" --> Taupi
|
||||
```
|
||||
|
||||
### Components & Routing
|
||||
1. **Home Network**: `192.168.10.0/24`. Contains the home devices (e.g., Shelly plug at `192.168.10.168`) and the gateway FRITZ!Box at `192.168.10.1`.
|
||||
2. **Kubernetes Cluster Network**: Pod subnets (`10.233.64.0/24` etc.) running on the remote public server (`136.243.23.215`).
|
||||
3. **WireGuard VPN Link (`wg0`)**:
|
||||
- The `wireguard` pod in the `wireguard` namespace is configured with `hostNetwork: true`. This exposes the `wg0` interface directly in the host's root network namespace.
|
||||
- The cluster host has a static tunnel IP of `192.168.11.1/24` on `wg0`.
|
||||
- The FRITZ!Box acts as the active peer client, initiating the connection to the host node at `136.243.23.215:51820`.
|
||||
- The host routing table directs home network traffic over the tunnel:
|
||||
```bash
|
||||
192.168.10.0/24 dev wg0 scope link
|
||||
```
|
||||
- Pods inside the cluster (like `fritzbox-exporter`) query `192.168.10.1`. The traffic is forwarded by the host's default CNI routing to the host's network namespace, which matching the `192.168.10.0/24` subnet route and sends it over `wg0` to the FRITZ!Box.
|
||||
|
||||
### FritzBox Exporter Egress Reject Workaround (Option B)
|
||||
|
||||
**The Problem**:
|
||||
When the `fritzbox-exporter` pod queries the FRITZ!Box TR-064 API at `192.168.10.1` from outside the home LAN subnet (using the transit WireGuard subnet IP `192.168.11.1`), the FRITZ!Box's internal security policy redirects the client to its public WAN IP (e.g. `212.42.244.122`) for portal login/authentication (`login_sid.lua`).
|
||||
When the dynamic WAN IP of the FRITZ!Box changes or the WAN ports are closed, these queries to the public IP timeout. Because Go's default HTTP client doesn't enforce a timeout, the exporter hangs for Go/Linux's default TCP connection handshake timeout of **2 minutes (120 seconds)**. This blocks all other metric scrapes and causes Prometheus target scrape timeouts.
|
||||
|
||||
**The Solution**:
|
||||
Rather than disabling CPU/RAM/temperature metrics (`-nolua`) or modifying the remote FRITZ!Box WireGuard subnet configuration, we block the exporter pod from reaching the public WAN IP.
|
||||
- We add an `initContainer` in `fritzbox-exporter.yaml` running with `NET_ADMIN` privileges.
|
||||
- The `initContainer` installs `iptables` and adds rules directly to the pod's shared network namespace.
|
||||
- It permits traffic to local private subnets (RFC1918 ranges) but rejects all TCP egress to public WAN IPs on port 80/443 with a `TCP RST` (Reset) immediately.
|
||||
- This causes redirected authentication requests to fail in milliseconds instead of hanging for 2 minutes. The exporter immediately falls back to scrape TR-064 metrics (WAN sync speed, bytes) successfully, avoiding Prometheus scrapes timeouts.
|
||||
|
|
|
|||
|
|
@ -371,6 +371,15 @@ for i in "${NAMESPACES_TO_ALERT[@]}"; do
|
|||
done
|
||||
```
|
||||
|
||||
### Checking alerts history
|
||||
|
||||
A Python script is available to retrieve and analyze the Prometheus alert history (for the last 14 days) by querying the cluster's Prometheus API through the Grafana pod:
|
||||
|
||||
```bash
|
||||
python3 k8s/monitoring/scripts/analyze_alerts.py
|
||||
```
|
||||
|
||||
|
||||
### influxdb
|
||||
|
||||
Used to store hass data long term.
|
||||
|
|
|
|||
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -15,6 +15,34 @@ spec:
|
|||
labels:
|
||||
app: fritzbox-exporter
|
||||
spec:
|
||||
initContainers:
|
||||
# This initContainer implements the Egress Reject Workaround (Option B).
|
||||
# The FRITZ!Box TR-064 API redirects login_sid.lua calls to its public WAN IP because
|
||||
# we query it from a different subnet (192.168.11.1). When the WAN IP is dynamic or port
|
||||
# 80/443 are closed, the HTTP client hangs for the default 2-minute Linux TCP timeout,
|
||||
# failing the entire Prometheus scrape.
|
||||
# This container injects iptables rules into the pod's network namespace to instantly
|
||||
# reject (TCP RST) egress to any public WAN IPs on port 80/443. The exporter will immediately
|
||||
# fail the login step (in ms) and continue scraping TR-064 metrics on port 49000 (which is allowed).
|
||||
- name: init-iptables
|
||||
image: alpine:latest
|
||||
securityContext:
|
||||
capabilities:
|
||||
add:
|
||||
- NET_ADMIN
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
apk add --no-cache iptables
|
||||
# Allow all local RFC1918 subnets and loopback
|
||||
iptables -A OUTPUT -d 127.0.0.1/32 -j ACCEPT
|
||||
iptables -A OUTPUT -d 10.0.0.0/8 -j ACCEPT
|
||||
iptables -A OUTPUT -d 172.16.0.0/12 -j ACCEPT
|
||||
iptables -A OUTPUT -d 192.168.0.0/16 -j ACCEPT
|
||||
# Reject port 80 and 443 egress to any public IP (non-RFC1918)
|
||||
iptables -A OUTPUT -p tcp --dport 80 -j REJECT --reject-with tcp-reset
|
||||
iptables -A OUTPUT -p tcp --dport 443 -j REJECT --reject-with tcp-reset
|
||||
echo "Egress iptables REJECT rules injected successfully"
|
||||
containers:
|
||||
- name: fritzbox-exporter
|
||||
image: ghcr.io/sberk42/fritzbox_exporter/fritzbox_exporter:latest
|
||||
|
|
@ -95,7 +123,7 @@ spec:
|
|||
rules:
|
||||
- alert: FritzBoxExporterOffline
|
||||
expr: up{job="fritzbox-exporter"} == 0
|
||||
for: 5m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,168 @@
|
|||
#!/usr/bin/env python3
|
||||
import subprocess
|
||||
import json
|
||||
import urllib.parse
|
||||
import datetime
|
||||
import os
|
||||
import sys
|
||||
|
||||
def get_grafana_pod():
|
||||
cmd = ["kubectl", "get", "pods", "-n", "monitoring", "-l", "app.kubernetes.io/name=grafana", "-o", "jsonpath={.items[0].metadata.name}"]
|
||||
res = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if res.returncode == 0 and res.stdout.strip():
|
||||
return res.stdout.strip()
|
||||
return "prometheus-operator-grafana-7f66b47fc6-dtdtg"
|
||||
|
||||
GRAFANA_POD = get_grafana_pod()
|
||||
|
||||
def query_range_kubectl(query, start_time, end_time, step):
|
||||
params = {
|
||||
"query": query,
|
||||
"start": start_time.isoformat(),
|
||||
"end": end_time.isoformat(),
|
||||
"step": step
|
||||
}
|
||||
query_str = urllib.parse.urlencode(params)
|
||||
url = f"http://prometheus-operated:9090/api/v1/query_range?{query_str}"
|
||||
|
||||
cmd = [
|
||||
"kubectl", "-n", "monitoring", "exec",
|
||||
GRAFANA_POD,
|
||||
"-c", "grafana", "--",
|
||||
"curl", "-s", url
|
||||
]
|
||||
try:
|
||||
res = subprocess.run(cmd, capture_output=True, text=True, check=True)
|
||||
data = json.loads(res.stdout)
|
||||
if data.get("status") == "success":
|
||||
return data.get("data", {}).get("result", [])
|
||||
else:
|
||||
print(f"Prometheus API error: {data}", file=sys.stderr)
|
||||
except Exception as e:
|
||||
print(f"Failed to query via kubectl: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
def format_duration(seconds):
|
||||
days = int(seconds // (24 * 3600))
|
||||
seconds %= (24 * 3600)
|
||||
hours = int(seconds // 3600)
|
||||
seconds %= 3600
|
||||
minutes = int(seconds // 60)
|
||||
seconds = int(seconds % 60)
|
||||
|
||||
parts = []
|
||||
if days > 0: parts.append(f"{days}d")
|
||||
if hours > 0: parts.append(f"{hours}h")
|
||||
if minutes > 0: parts.append(f"{minutes}m")
|
||||
if seconds > 0 or not parts: parts.append(f"{seconds}s")
|
||||
return " ".join(parts)
|
||||
|
||||
def parse_periods(values, step_seconds):
|
||||
if not values:
|
||||
return []
|
||||
|
||||
sorted_values = sorted([(float(ts), int(float(val))) for ts, val in values], key=lambda x: x[0])
|
||||
|
||||
periods = []
|
||||
current_start = None
|
||||
last_ts = None
|
||||
|
||||
for ts, val in sorted_values:
|
||||
if val != 1:
|
||||
if current_start is not None:
|
||||
periods.append({
|
||||
"start": current_start,
|
||||
"end": last_ts,
|
||||
"duration": last_ts - current_start + step_seconds,
|
||||
"ongoing": False
|
||||
})
|
||||
current_start = None
|
||||
continue
|
||||
|
||||
if current_start is None:
|
||||
current_start = ts
|
||||
else:
|
||||
if ts - last_ts > step_seconds * 3:
|
||||
periods.append({
|
||||
"start": current_start,
|
||||
"end": last_ts,
|
||||
"duration": last_ts - current_start + step_seconds,
|
||||
"ongoing": False
|
||||
})
|
||||
current_start = ts
|
||||
last_ts = ts
|
||||
|
||||
if current_start is not None:
|
||||
periods.append({
|
||||
"start": current_start,
|
||||
"end": last_ts,
|
||||
"duration": last_ts - current_start + step_seconds,
|
||||
"ongoing": True
|
||||
})
|
||||
|
||||
return periods
|
||||
|
||||
def main():
|
||||
print(f"Querying Prometheus from pod {GRAFANA_POD}...")
|
||||
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
start_time = now - datetime.timedelta(days=14)
|
||||
step = "2m"
|
||||
step_seconds = 120
|
||||
|
||||
print(f"Analyzing alert history from {start_time.strftime('%Y-%m-%d %H:%M:%S')} to {now.strftime('%Y-%m-%d %H:%M:%S')} UTC...")
|
||||
|
||||
# Query ALERTS metric
|
||||
results = query_range_kubectl('ALERTS', start_time, now, step)
|
||||
|
||||
if not results:
|
||||
print("No alerts found in the specified range.")
|
||||
return
|
||||
|
||||
all_events = []
|
||||
|
||||
for result in results:
|
||||
metric = result.get("metric", {})
|
||||
values = result.get("values", [])
|
||||
|
||||
alert_name = metric.get("alertname", "Unknown")
|
||||
alert_state = metric.get("alertstate", "Unknown")
|
||||
|
||||
# Extract target labels that distinguish this alert instance
|
||||
ignored_keys = {"__name__", "alertname", "alertstate"}
|
||||
labels = {k: v for k, v in metric.items() if k not in ignored_keys}
|
||||
|
||||
periods = parse_periods(values, step_seconds)
|
||||
for p in periods:
|
||||
all_events.append({
|
||||
"alertname": alert_name,
|
||||
"alertstate": alert_state,
|
||||
"labels": labels,
|
||||
"start": p["start"],
|
||||
"end": p["end"],
|
||||
"duration": p["duration"],
|
||||
"ongoing": p.get("ongoing", False)
|
||||
})
|
||||
|
||||
# Sort events by start time descending
|
||||
all_events.sort(key=lambda x: x["start"], reverse=True)
|
||||
|
||||
print(f"\nFound {len(all_events)} alert events in the last 14 days:")
|
||||
print("-" * 100)
|
||||
|
||||
# Print summary table
|
||||
for i, ev in enumerate(all_events):
|
||||
start_dt = datetime.datetime.fromtimestamp(ev["start"], datetime.timezone.utc)
|
||||
end_dt = datetime.datetime.fromtimestamp(ev["end"], datetime.timezone.utc)
|
||||
ongoing_str = " (Ongoing)" if ev["ongoing"] else ""
|
||||
|
||||
labels_str = ", ".join([f"{k}={v}" for k, v in ev["labels"].items()])
|
||||
|
||||
print(f"[{i+1}] {ev['alertname']} ({ev['alertstate']})")
|
||||
print(f" Duration: {format_duration(ev['duration'])}{ongoing_str}")
|
||||
print(f" Timeline: {start_dt.strftime('%Y-%m-%d %H:%M:%S')} -> {end_dt.strftime('%Y-%m-%d %H:%M:%S')} UTC")
|
||||
print(f" Labels: {labels_str}")
|
||||
print("-" * 100)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -59,7 +59,7 @@ spec:
|
|||
rules:
|
||||
- alert: TaupiFanOffline
|
||||
expr: up{job="taupi-fan"} == 0
|
||||
for: 8h
|
||||
for: 1h
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
|
|
@ -67,7 +67,7 @@ spec:
|
|||
description: "The Shelly plug at 192.168.10.168 is unreachable by Prometheus. Check Wi-Fi connection or power status."
|
||||
- alert: TaupiFanSensorsStale
|
||||
expr: taupi_lost_connection_seconds_innen > 600 or taupi_lost_connection_seconds_aussen > 600
|
||||
for: 5m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
|
|
@ -75,7 +75,7 @@ spec:
|
|||
description: "The BLE sensors for temperature and humidity inside or outside have not sent new data for over 10 minutes."
|
||||
- alert: TaupiFanMoldDanger
|
||||
expr: taupi_is_critical == 1
|
||||
for: 30m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
|
|
|
|||
|
|
@ -146,15 +146,15 @@ extraManifests:
|
|||
rules:
|
||||
- alert: n8nInstanceDown
|
||||
expr: up{job="mop-n8n"} == 0
|
||||
for: 5m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "n8n instance is down or unresponsive"
|
||||
description: "n8n scraper has failed for the last 5 minutes. The application might be frozen or crashed."
|
||||
description: "n8n scraper has failed for the last 1 hour. The application might be frozen or crashed."
|
||||
- alert: n8nPodRestarts
|
||||
expr: rate(kube_pod_container_status_restarts_total{container="n8n"}[15m]) * 900 > 1
|
||||
for: 5m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
|
|
@ -162,17 +162,17 @@ extraManifests:
|
|||
description: "n8n has restarted in the last 15 minutes. This might indicate liveness probe failures due to database issues or memory limit exhaustion."
|
||||
- alert: n8nNodeEventLoopLag
|
||||
expr: n8n_nodejs_eventloop_lag_seconds{job="mop-n8n"} > 1
|
||||
for: 5m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "n8n event loop lag is high"
|
||||
description: "n8n Node.js event loop lag has exceeded 1 second for the last 5 minutes. This can make the UI unresponsive and logins fail."
|
||||
description: "n8n Node.js event loop lag has exceeded 1 second for the last 1 hour. This can make the UI unresponsive and logins fail."
|
||||
- alert: n8nPodNotReady
|
||||
expr: kube_pod_status_ready{condition="true", pod=~"mop-n8n-.*"} == 0
|
||||
for: 5m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "n8n pod is not ready"
|
||||
description: "n8n pod has been in non-ready state for more than 5 minutes. This is likely due to failing readiness probes (/healthz check failing, possibly database connection issues)."
|
||||
description: "n8n pod has been in non-ready state for more than 1 hour. This is likely due to failing readiness probes (/healthz check failing, possibly database connection issues)."
|
||||
|
|
@ -47,7 +47,7 @@ spec:
|
|||
rules:
|
||||
- alert: WireguardExporterDown
|
||||
expr: up{job="wireguard-exporter"} == 0
|
||||
for: 5m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
|
|
@ -55,7 +55,7 @@ spec:
|
|||
description: "The WireGuard Prometheus exporter is unreachable. The pod may have crashed or is unresponsive."
|
||||
- alert: WireguardInterfaceDown
|
||||
expr: absent(wireguard_sent_bytes_total{interface="wg0"}) == 1
|
||||
for: 5m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
|
|
@ -63,7 +63,7 @@ spec:
|
|||
description: "The WireGuard interface wg0 is not reporting any statistics. The VPN tunnel might be down or inactive."
|
||||
- alert: WireguardPeerOffline
|
||||
expr: (time() - wireguard_latest_handshake_seconds{interface="wg0"}) > 300
|
||||
for: 5m
|
||||
for: 1h
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
|
|
|
|||
Loading…
Reference in New Issue