From bdabc56afec41c0d93adf92164e6ab002543a875 Mon Sep 17 00:00:00 2001 From: Moritz Graf Date: Sat, 11 Jul 2026 07:40:06 +0200 Subject: [PATCH] monitoring: optimize alerts and mitigate fritzbox-exporter timeouts --- k8s/AGENTS.md | 50 ++++++ k8s/README.md | 9 + .../prometheusrules.secret.yaml | Bin 1557 -> 1553 bytes k8s/influxdb/monitoring.secret.yaml | Bin 1428 -> 1424 bytes k8s/monitoring/alertmanagerconfig.secret.yaml | Bin 1779 -> 2313 bytes k8s/monitoring/fritzbox-exporter.yaml | 30 +++- k8s/monitoring/scripts/analyze_alerts.py | 168 ++++++++++++++++++ k8s/monitoring/taupi-fan.yml | 6 +- k8s/n8n/n8n.secret.yml | 14 +- k8s/wireguard/monitoring.yaml | 6 +- 10 files changed, 269 insertions(+), 14 deletions(-) create mode 100755 k8s/monitoring/scripts/analyze_alerts.py diff --git a/k8s/AGENTS.md b/k8s/AGENTS.md index 390bf73..7917116 100644 --- a/k8s/AGENTS.md +++ b/k8s/AGENTS.md @@ -160,3 +160,53 @@ spec: * **Namespaces**: Every application gets its own namespace. * **Secrets**: Encrypt all secrets using `git-crypt`. +## Network Setup (VPN & Routing) + +The Kubernetes cluster (Haumdaucher) is connected to the home network via a dedicated WireGuard VPN tunnel. + +```mermaid +graph TD + subgraph "Kubernetes Cluster (haumdaucher.de - 136.243.23.215)" + subgraph "monitoring namespace" + Exporter[fritzbox-exporter Pod] + end + HostNet[Host Network Namespace] + WgPod[wireguard Pod hostNetwork: true] + end + + subgraph "Home Network (192.168.10.0/24)" + FB[FRITZ!Box Gateway - 192.168.10.1] + Taupi[Taupi Fan Shelly - 192.168.10.168] + end + + Exporter -- "Queries 192.168.10.1" --> HostNet + HostNet -- "Route: 192.168.10.0/24 via wg0" --> WgPod + WgPod -- "WireGuard VPN Tunnel (51820/UDP)" --> FB + FB -- "Accesses local subnet" --> Taupi +``` + +### Components & Routing +1. **Home Network**: `192.168.10.0/24`. Contains the home devices (e.g., Shelly plug at `192.168.10.168`) and the gateway FRITZ!Box at `192.168.10.1`. +2. **Kubernetes Cluster Network**: Pod subnets (`10.233.64.0/24` etc.) running on the remote public server (`136.243.23.215`). +3. **WireGuard VPN Link (`wg0`)**: + - The `wireguard` pod in the `wireguard` namespace is configured with `hostNetwork: true`. This exposes the `wg0` interface directly in the host's root network namespace. + - The cluster host has a static tunnel IP of `192.168.11.1/24` on `wg0`. + - The FRITZ!Box acts as the active peer client, initiating the connection to the host node at `136.243.23.215:51820`. + - The host routing table directs home network traffic over the tunnel: + ```bash + 192.168.10.0/24 dev wg0 scope link + ``` + - Pods inside the cluster (like `fritzbox-exporter`) query `192.168.10.1`. The traffic is forwarded by the host's default CNI routing to the host's network namespace, which matching the `192.168.10.0/24` subnet route and sends it over `wg0` to the FRITZ!Box. + +### FritzBox Exporter Egress Reject Workaround (Option B) + +**The Problem**: +When the `fritzbox-exporter` pod queries the FRITZ!Box TR-064 API at `192.168.10.1` from outside the home LAN subnet (using the transit WireGuard subnet IP `192.168.11.1`), the FRITZ!Box's internal security policy redirects the client to its public WAN IP (e.g. `212.42.244.122`) for portal login/authentication (`login_sid.lua`). +When the dynamic WAN IP of the FRITZ!Box changes or the WAN ports are closed, these queries to the public IP timeout. Because Go's default HTTP client doesn't enforce a timeout, the exporter hangs for Go/Linux's default TCP connection handshake timeout of **2 minutes (120 seconds)**. This blocks all other metric scrapes and causes Prometheus target scrape timeouts. + +**The Solution**: +Rather than disabling CPU/RAM/temperature metrics (`-nolua`) or modifying the remote FRITZ!Box WireGuard subnet configuration, we block the exporter pod from reaching the public WAN IP. +- We add an `initContainer` in `fritzbox-exporter.yaml` running with `NET_ADMIN` privileges. +- The `initContainer` installs `iptables` and adds rules directly to the pod's shared network namespace. +- It permits traffic to local private subnets (RFC1918 ranges) but rejects all TCP egress to public WAN IPs on port 80/443 with a `TCP RST` (Reset) immediately. +- This causes redirected authentication requests to fail in milliseconds instead of hanging for 2 minutes. The exporter immediately falls back to scrape TR-064 metrics (WAN sync speed, bytes) successfully, avoiding Prometheus scrapes timeouts. diff --git a/k8s/README.md b/k8s/README.md index 3cbd8f9..1bb3069 100644 --- a/k8s/README.md +++ b/k8s/README.md @@ -371,6 +371,15 @@ for i in "${NAMESPACES_TO_ALERT[@]}"; do done ``` +### Checking alerts history + +A Python script is available to retrieve and analyze the Prometheus alert history (for the last 14 days) by querying the cluster's Prometheus API through the Grafana pod: + +```bash +python3 k8s/monitoring/scripts/analyze_alerts.py +``` + + ### influxdb Used to store hass data long term. diff --git a/k8s/home-assistant/prometheusrules.secret.yaml b/k8s/home-assistant/prometheusrules.secret.yaml index 752e2d4d477b72529852a988571401157fa00812..bcea7175383aef820dba9ce837d1fc2d205fcb0e 100644 GIT binary patch literal 1553 zcmV+s2JZO)M@dveQdv+`0AwG2R7x98AqwPJ3yg?hdGorl0Kwa3m)vT29ukH<48$_G z=!iIO3mr@2Ycyjh6bJEKowt$a&HmfNH-;zT6Zv@<4NHUy&Yu);^cfxQBIEBLvg;3; zW*!{^F;YQEu?{wA*2iWO>1F-^)~mez$hJ|veO&}yTn*ZzMDNsbBtQ~8Sy~T#Do()a zD^UpYb~i+z3`cn$CpXrQvnZxhgkT5^^RmVcx&`t<{U(DyrK~C>Idfcq&g=5x?XPeY zBS^bedB6DgacV?46R)q{V4p+zrOjcW*oV+#L6Z$~X$11FH_AscPCM=@wm`K$QHia4 z%*H>92i~&6DDmDA?kC*G^=Ae(*uZrR)eX_&CLM$-;E5yYUM6+iTd-(sysjl_Vo{Gh zTRM*lFiEE5QPK1OnoV1=H|Aos}iqX3jv(TJ0UKWnTb4}43@b#uy#E3~3WRrUHP?vhL`(Z^$>1pRISt047qlc)e9xfT8k#M|v|V^5kWmt+R=wb^SnIVB0Z|4L8VqCL2(Hy4zxABfv)Cd>;Is%F90YMygmCj(b! zO*_Lb@T$uTjrF$+tJC48=OkA-QMvMY6lPNePYF(l=^0H=7oo0H-g45?bG$q*rft9F z0jt^7_b~&Uuqiac!au4{IOzBM}PAGP+y>Bz!rqvk_=ZmAABe&;yIKVz_=hKx#( zlfykWk-2HKcDo9=pZ>I88Zs;2DHLN01Mp}RDe2{EJT(MZ?nL+Yh;x#B7T!N}v27%W zj9esUP>awNs3*a#SC3x3FNCw2K= zyT=w+{>kS9Ff0YGMLjUWU-W!{+D5LxOH%oRUfNAm3nRaif!SjO-gjw=OV0PB`-%2# zE)X_@lZcJIA@7SuWxuM~e8Uq!4delh2(H?|`rmS2sx#p$siG+V=Coot?7|vMgcVw* znryY&ueQ{dKyQLL{$1!$`TfgX6eKmYk9{CMN`lnMK{e0BGnr!Kmzgo9rtU-LQrxjz z;p!s=Qj;_!uZDf7Wx}oDaeG3W-v+rgy(Z>rFaHr)P-mis@aUp|sK%UH)E6-F)+L1{ DNzwqD literal 1557 zcmV+w2I~0$M@dveQdv+`07^?MCti$`T1`^KE9j+C0&OcgOmZ*3M4uL~g}{=TQ97oy z5Bd@nvfGs}!&JoZ7M7uH_ z`7Am7t+4jei0P!Umhd)-@@0o!zBmzr#l05Twc7_r9wn|QD~6rWi^nuY_x8wkIcy}< zqVy#)nn)R-)D1U901{|!AVJ8!hu1c8b{fFBFBKjfbAs1f?497dn^cjQb_|{~czwih zdJlb1?EYojzX2S^Ba`ux)JQR*&G)>@CMsHH$QALUh3y8ap!U&cd6;4Qs3}kCJTKn8 z5hLvWiUAv$M|iD3&@QBN4*L=oX|mm8&#mj%o@Qa zLd$a>)?(47xxG+3b01yzyG}xcH4;PUqYVK+ga19QH86Ez%*`brT$JUEJ@S1gh3XRi z$!YoyYpp}G3-J6dKovXQ^s&jl+MQj09Sp%8Ff`mGn z2!dfBF*j(^2k=Xrx=75pI=r~%+ep(Vsym4dIYCm&uJO}n`4RbWiU2TZ$8Fv(kEAzI zh+0_M9wdC+?R8RF=S^fnr@B2|8eptL-EYqqKCI}j*qa9slH^)z4F40mR%Oh=8!HZK z5?m58ErOGr0A!Xpj?;DLBl>H>>gQ8Yr^yCV3e?S6QIMMpG4hmfwjem#Cpx{dVBu~1 znol+WgW|TJ3U;!UT>BYcvK3!(@MUM0+ePttvHVYvsFZKx z9e|7%qpY;!B6F#l-GEls`uy;g{j7m(IhP+`;j`2=`>We8%YAB{@#iAfXY~}+hUU?+ zQc!{ZBA6i!VJ4G;=Xe4S@z~Je45#-b3a*Lw37NUvLYDk(+tIH>YS939DW3h`lQZs0 z0^>=gnPR*G!Msh_N=d{36u?+ELnW2fznwC$-bD4MErvyEIfWW>cYR~>;XUEi!Jj74#}(MSfC@p z781mfG!CE%G#W;g)Megvs|n83^D~nAS$9kzL#;L^^Fvv+^;-?kE06pZek%2Pd)B3XiT|g0}$E0f6~#vyJW%u0P96nl7AA}xKA~aU=u-XZhcg9)OOvh zoy3M@AWwBA!-=R!Ht$Dno&=dl?^55YHAC-!i-tR}#F+wJTz+X7_Q%Yqb$$+BaiFe? zd{56uE7DEo7aKrRuvd4bcmT=X#$HY0wi%T=J$1g3Z%-x8^9bn$H3j-2HA!~~ zLV}rEu_r2zS@19}Bkvh)-$JOQX!M>ht`*i*EA0Pvc1NJ+HR0+g-j*mLqKWcV?Kp{N z{Q+GT-Ea2xglBs9B^vYABWQt^D&BiRK6rNSvIg(g`x;&Bo5my5EciUWp)g27Jh7lsmIY6Z}D@cHiJ@xjG%1h^wB}DeNmKrN6_SOVm zw4%vBw@X;fCOha~VDxl0=hd21)ApqpT=>~0fyU!kObchE zuGp_v%~%5PP4xX22g<)nT$ns{(!R0Ky&RK}ISmL)I5?Z%O>{kr)dT1QH|08io!9sZ Ha0_?1I`j(n diff --git a/k8s/influxdb/monitoring.secret.yaml b/k8s/influxdb/monitoring.secret.yaml index a975a8f6d64d82bc5f3be7c1c99b07bd06803beb..9a2dc5d7c377f1211001ac47f954cacec9acc400 100644 GIT binary patch literal 1424 zcmV;B1#kKQM@dveQdv+`01zIqDf>vi*;Z-4VoqnFpq*+FMiFZ-0Tby#A?G!FGO~R{ z8$5hjkn+LN(?bnHWs@MA^6PL0uo^Z|T`5$7 zpiA=^5+gD7BPmPC;#;EK>e7Z#rf?VU$y-)!jR=2u7Q2cWpYIbh6Lx)m-tzs z=CE24EMXxz*}&+PRW>2?xHf>-C?&>lpt)Z2Y_vZFA>@rd>r+)|I3w(|Xkhje0VsB} z{om3l;+y|2c6~>kbkfq7XNVQkUX3{6CQG=oVX!dO&J&~lY%qx)AlF~z_=^y@FkkKG zj9CXMaoL#-PdKrFSW#b+z@g-oLi2fAz|CuzQ3X^{s{@|{$i~>k>+D4II&&d|hsoAP z+rZm#9MXGmDasqHmeVkzHBl^*9NISlX$%vVeXGsQBKB)%XbW?`C8qU5OgkH;4X|w} zOO_OV>xY@PB?EB()j&)3i8&E+AqH7xHUsu%<>A^aoVUkczd|sY#V*|_PP`6fT<+KA zQweoxWB#Q3uJ*U?eL6R#fyB^ihW@6b({&d%VqEu*o>fWs6f?(# z22YUV9B-}T4AHEfcw{>^6{Z`|Afsyj;f{(a1m%PqP|6u>uPR4q^!vmbRiMJMFpDTG zQsMpU@l7SGraI&vlNLZf`L2*lGhq~53h89GgGAx)MPJ40RR=D zlay$}TK1t*Z?{(zT!o6v-^3$7?qDM*ZZB;MmweDLjGF5Rr@O?kkW(VEi{w9YY#8EV znnR}n6a&DnLBCO`cE6VX{=zb5sLJ0#P_0spp9!0`bFL74EQD^rVy$95rH5x)c4WL4 zyQ*Vlv5$MI%ARNU>YmRdWUST(CiQ{j)9SYS2qPrD#stjcG>{;)uaL_%>R;1Qf^s7{ z2%U&~B%W^&j@V}{^2)GyO;qBKQJ5RKz%CQV2utb!tgBg0W-YGWmPQqNrzsA1Mn<7s zFk7^IC)NofmgrYZndNL zF*hm0<}Jo>WaTivdrY#22DX5mlXhq>R{Gv^t_S-UpNz_zk#KTXA5rBu* zHZvdiDnA7N1Ky4@X)MO!lNNr>QSi``Sad?(M&0g?T>j#l-a4QfyOgJXJMH04UAazH zrm=FfNv4;W)l61N>)w(YDluybJcaT+RbB0QCuEdR&WJ-srw);Tlw{>{I+d8Xr4z z^ikE^WNON&OMnQdIQ%+QaKKTHx!KyYPk^o#JCW6?VxH@5ZJv#|A2DDN*MEv1Xiq|} z#FO~x`thH3|IC;|n9?3&Dr&Qm%qXxZ>)2<#n3+v6VB@wsizEU94P zv~!5}-Ic0GY+Co}tUfDI@u_GR2ac1xNA4|2{=tU6-&`y83ZVBpx_~~2v`hOXCdgq) zEK>Y1`6akb2n4|Pu2;IthV^H{x8btpgdcqNegeG7rj=sJVUs literal 1428 zcmV;F1#9{MM@dveQdv+`0AE0U#CdvgpaD%_b6O#?La_h;-*(|HkV?qTV=>r%<(U}$np)%3YybLt=(FE%H{ zD&F*|w*%Kp*sW~I^c%h3wrVZ)HV-8`VW}$9Qz7+J(d@>LXHP(O)3808E>J!x$*<#X zoxmbS7Hb769FRlOj&XY3j%(}uoD_VT^mI-_RnT~%izGV>Umg0;3X=uc@;Tx$$t%ix zDrEL^Wq_~R!{ZQobz646^JfQfXi9W70q{HZA+#|cmk?5Pt+AE)oB5^YU%@wMSpe5A zLyY|wYhWaY6X-cuO~F*f3jrklX=iPqLFFI&)%Zv_$`k z7%r^NUC{URZKT)^FgBH)XtE_M154d2!`5h$>Pm}h6*hUbxc`5RjrhS0BTZUWq8tYo zE!9R>)BseXijQ`?Uh?FJ?_>iRe8O5yC#pz0IA!hHDb>_9R6?{Rxdg&pAnEC*fULdw z7o%*jB#u?DPwV2K;e!QWf~t0B_7d6?>~=2GPvLPsc-TfSv440UfD0j-cUJ*v??^Q^{N*l`A>p$RtP} zd4p)1XAc`J%C}@^DfS5l{pdXcX{K0s~(eE6E`KJX|DUH^&Vn7hpqH8 zG3lZ)0Cz%S%ng}!0-M)L?9&`Y65br${Y|CqxJy*%bt*<9>>iC-t;A{fE>xN=|627q zQM8U{L#tkX9EyJvYD}-SPA9xeMZt9wO*nUtu@ePkTEqo1`$`Q=1XNX$P4Wj)IdUu8 zp{I(@-cRcHl?M_~)Uh@Du!u;Fb1AB_Zw&)_bMC-08xai&0YTNY4bs)&825!QC+Oz< zcr~zRdc++$({>=o7NA15oa?ywkZXNY<0%Z~fr1<^H$}pTo}ZE_WIGh3>CU5_3T08~ zkn)*J1%p41+_%nNL8LCJgsnF;NQ8Em!0ng${N%apa@Qsr#+u@w@~OSSGyrw{`oMOZrbqOw?&&tQ=zficpli^Lu;pGCyx9JU}RG1*ecH`~uEp%F-xe z=b~~z0yY&s4|hpRlZ2+tf!jNE>*a{nQEZ=myqfTeDV4+uxQyD=TO*Q%egO{g|GDrl z>Ys@6Nrl_y1-jnZYmqoYlya*j@kNVqT^iUy^CvjqA~n+gOU0xmzO&&XhYdSDpY3a1 zgCR0aultq0l#27vgMaFD6;87<#faM3a|oEeOIf!h8__;HW^=x=8PR3ffSZ;P{1mwv zI2;HC)sPK7$ln5E$(D@aTy|-?d5IY51@vl(+!+qbM`P)=|>kJQ{DeyUI^O0~f`3U6dzYJ#b?yEC@iz0!rq iwW-nhbbfp@-x>@{-Z z2n3cH*BWl2TFu|FRTc>O8Cc7Gd$7z}R21u4gG~dq=?NLOOT!j)D)CUqF`W29W^&PR z!Sg9q6zeJD7xCmuiT3W&1*z)>Z_m6G*&?I!0Z>T5hNyqu^tQZdbmq8nviHt{Z;GH^ ziITBb4I1pIwSn9-=z-- zC;YII-?nu?z?cZX+L81k&Jon@;GynpN~mIh@-p3L~Eu`z=D#~=Q&5sRa1L+HvWKw1v zbZ06u#@}-2HRMuZ zq~gLFwpnTyTZ0fy%hXQTq?E=K({TBSt!4R$%L)E=M>XpEgo6w>l>Oc@2S;G_iH;a55l2 z-=jlj_-8vPR|$YZ3AS&A_3%gJvFNkU3MWCrM{aa;7Dbq3$C?2t zG3sQLV%c;Z%#lhv>qhGi^@dIC@k@7YM3*b?I4~d@#~tWto=^5M*hr+tpq-XQ6aeTE z9%+?%s36$MFbVv-$h=tymT1C$PXxM4hoFi>BW^?dMa-%W76JJ2 zr#I2exn2u_SR<#WQ`PdbHQfpOq6@iu(J*7I2o;2I?s<@*PoLn8$O17AKEL%#25g=c zY_p7Orjv=*Thw=dcDiDCu&Jq}NGih*IVLgUfUKJ-#KTA=g*Hrps+$ zU4kQ@c11Ozl`ePAN}ktf#@|kl$nJ|bF4&nk0k{T1`TDLfw#vNsUYHFH2j1e^7kPfy zY@@wAI+F6br?iPi-1ei3W=N}6B{XE~(M#4Du;VJ*fI!h6g$hQ-+IGLOp^Dv_TX&y9~9`soNyeq;;U(0_UuL7_IyTf6s%4B@5>965@ksl>`)b7 z!L*&a{&1QUVgjmSF(q+5YS>{;_-(ge0Qbs(i_UZ6V8>BO{`~~Gi2%(*0^51PdQ#;r z7xpL1KDIY1ziGEw^j^fc{#a6&u#QL--XSDBCIB ze;=B+HHB9&1a_M>!9f)BxRm$cO+@8#^tuQN!Jx%8{L4r4eba|%|M!wypg&3VSg6Z; zB zA(VE@B@3?#B3YwHyCrcQ(=!ym$yntCO*?o(C%38-_1rG?vAbX_BpaYP@p#eYM=Pe zAT++v`Llj|_gQ4^6_%uUd8(N8Qq?{v0*%>$m=9w}xZ~ZzCTiR1g1iV~wf};jquZU@ zXDmdHd%d1XNt?U)xC}o{Bvh9=1+g^*YuZ44LkU$Zc=~#-xGe89*F%3e&Nwm#RyXe8 zCUp%lyBn*HF3_DwUr|gGl$fdV;EsQFAn5#6V-EWpMna|ox_+A&s6DWki1lC5%>k;B z*l3DgI}dSnm$>!QO~g`Nx}#^lv!Z$2J@K@>IdUD(tND@6RjE%+4ZtqEvtfJ?$Bf2| z7e+L^NIWvAL4Jq_MluqB#}diyP7_J%x#x|8@Z+S5fCFdCZO)ZKC{d;+V8QXx;O>@_ z6|~2{90i;qL;T(k8>zjo@)6%Mw)jH5+6ju4>mzhwU<~118DqAvb0aND0|DG?)fm^)i&J^ZFWu!deB(8E1gr literal 1779 zcmVRgv4z#MjEGQX_n|#uPTR=2$rb3?9? zI;mDN8*DC!c^eTiG9!gCx!$=++b5A3T*LfmY}pMW z9c0B$c9LJ-#h%99M%KHGaq6L}bJ?$>bW!F*L6Sa-E)iF9wl+Pz0E&H;e|rUK`N4`8 zER`l{8jVE)L^^8Y!c+(a1Lix7nBYh*A{+M#Q8RY9-B@gVfoiFP1e@DEBq&~L~C9zIF{>ax3hpqJ+^2EMb zP;WjmJXCR~jwHR!TI|!OoKhXw0Cu7%a$FNZZxQTFAR=U<7GAL6Ura%nD70Vtj4-KI z33Rhu05Ho1g&WXyya$vthAt1Dbk_qimd!A!x+k}HqL{x;c{#;@KR7N4`oLBvBS-KR zNVdr>({}qunir>v3gn~8bixizLgssnR2D9^xL0S7ta;=8?uo0rGTVVc~3ygZR4EHxPMf$xA@?lhgkEBHuelHtFGZk^i1n*^j3jFbt zB18FS`{c`?J>hJ*bL&0c*plzpjJadn`o)sce0o)BCcMI&Eh~2}t4JYfBHs)mIg6iLynt z?gx-4hv{aM_@B1>glyXz1Z+E4&p756DIIML%W^GCyUG-aV*Hd^$@#s*M*V!rMJOST zn=RB%7v)=Yqu=UZ{gPDdRDU{Op~qE{OWSI052B3+{M4eHDX0ku`4o=@U=SuZeBO8b zJzyJqsfYeBIS4)G(fDyt3$>qvSw`S@Q<<;{FLYttp@aBr6IC-*?Hj5Pn#9 zE*}KBL{0 0: parts.append(f"{days}d") + if hours > 0: parts.append(f"{hours}h") + if minutes > 0: parts.append(f"{minutes}m") + if seconds > 0 or not parts: parts.append(f"{seconds}s") + return " ".join(parts) + +def parse_periods(values, step_seconds): + if not values: + return [] + + sorted_values = sorted([(float(ts), int(float(val))) for ts, val in values], key=lambda x: x[0]) + + periods = [] + current_start = None + last_ts = None + + for ts, val in sorted_values: + if val != 1: + if current_start is not None: + periods.append({ + "start": current_start, + "end": last_ts, + "duration": last_ts - current_start + step_seconds, + "ongoing": False + }) + current_start = None + continue + + if current_start is None: + current_start = ts + else: + if ts - last_ts > step_seconds * 3: + periods.append({ + "start": current_start, + "end": last_ts, + "duration": last_ts - current_start + step_seconds, + "ongoing": False + }) + current_start = ts + last_ts = ts + + if current_start is not None: + periods.append({ + "start": current_start, + "end": last_ts, + "duration": last_ts - current_start + step_seconds, + "ongoing": True + }) + + return periods + +def main(): + print(f"Querying Prometheus from pod {GRAFANA_POD}...") + + now = datetime.datetime.now(datetime.timezone.utc) + start_time = now - datetime.timedelta(days=14) + step = "2m" + step_seconds = 120 + + print(f"Analyzing alert history from {start_time.strftime('%Y-%m-%d %H:%M:%S')} to {now.strftime('%Y-%m-%d %H:%M:%S')} UTC...") + + # Query ALERTS metric + results = query_range_kubectl('ALERTS', start_time, now, step) + + if not results: + print("No alerts found in the specified range.") + return + + all_events = [] + + for result in results: + metric = result.get("metric", {}) + values = result.get("values", []) + + alert_name = metric.get("alertname", "Unknown") + alert_state = metric.get("alertstate", "Unknown") + + # Extract target labels that distinguish this alert instance + ignored_keys = {"__name__", "alertname", "alertstate"} + labels = {k: v for k, v in metric.items() if k not in ignored_keys} + + periods = parse_periods(values, step_seconds) + for p in periods: + all_events.append({ + "alertname": alert_name, + "alertstate": alert_state, + "labels": labels, + "start": p["start"], + "end": p["end"], + "duration": p["duration"], + "ongoing": p.get("ongoing", False) + }) + + # Sort events by start time descending + all_events.sort(key=lambda x: x["start"], reverse=True) + + print(f"\nFound {len(all_events)} alert events in the last 14 days:") + print("-" * 100) + + # Print summary table + for i, ev in enumerate(all_events): + start_dt = datetime.datetime.fromtimestamp(ev["start"], datetime.timezone.utc) + end_dt = datetime.datetime.fromtimestamp(ev["end"], datetime.timezone.utc) + ongoing_str = " (Ongoing)" if ev["ongoing"] else "" + + labels_str = ", ".join([f"{k}={v}" for k, v in ev["labels"].items()]) + + print(f"[{i+1}] {ev['alertname']} ({ev['alertstate']})") + print(f" Duration: {format_duration(ev['duration'])}{ongoing_str}") + print(f" Timeline: {start_dt.strftime('%Y-%m-%d %H:%M:%S')} -> {end_dt.strftime('%Y-%m-%d %H:%M:%S')} UTC") + print(f" Labels: {labels_str}") + print("-" * 100) + +if __name__ == "__main__": + main() diff --git a/k8s/monitoring/taupi-fan.yml b/k8s/monitoring/taupi-fan.yml index 00f15a1..bb50001 100644 --- a/k8s/monitoring/taupi-fan.yml +++ b/k8s/monitoring/taupi-fan.yml @@ -59,7 +59,7 @@ spec: rules: - alert: TaupiFanOffline expr: up{job="taupi-fan"} == 0 - for: 8h + for: 1h labels: severity: critical annotations: @@ -67,7 +67,7 @@ spec: description: "The Shelly plug at 192.168.10.168 is unreachable by Prometheus. Check Wi-Fi connection or power status." - alert: TaupiFanSensorsStale expr: taupi_lost_connection_seconds_innen > 600 or taupi_lost_connection_seconds_aussen > 600 - for: 5m + for: 1h labels: severity: warning annotations: @@ -75,7 +75,7 @@ spec: description: "The BLE sensors for temperature and humidity inside or outside have not sent new data for over 10 minutes." - alert: TaupiFanMoldDanger expr: taupi_is_critical == 1 - for: 30m + for: 1h labels: severity: warning annotations: diff --git a/k8s/n8n/n8n.secret.yml b/k8s/n8n/n8n.secret.yml index 5eb77dd..307ab52 100644 --- a/k8s/n8n/n8n.secret.yml +++ b/k8s/n8n/n8n.secret.yml @@ -146,15 +146,15 @@ extraManifests: rules: - alert: n8nInstanceDown expr: up{job="mop-n8n"} == 0 - for: 5m + for: 1h labels: severity: critical annotations: summary: "n8n instance is down or unresponsive" - description: "n8n scraper has failed for the last 5 minutes. The application might be frozen or crashed." + description: "n8n scraper has failed for the last 1 hour. The application might be frozen or crashed." - alert: n8nPodRestarts expr: rate(kube_pod_container_status_restarts_total{container="n8n"}[15m]) * 900 > 1 - for: 5m + for: 1h labels: severity: warning annotations: @@ -162,17 +162,17 @@ extraManifests: description: "n8n has restarted in the last 15 minutes. This might indicate liveness probe failures due to database issues or memory limit exhaustion." - alert: n8nNodeEventLoopLag expr: n8n_nodejs_eventloop_lag_seconds{job="mop-n8n"} > 1 - for: 5m + for: 1h labels: severity: warning annotations: summary: "n8n event loop lag is high" - description: "n8n Node.js event loop lag has exceeded 1 second for the last 5 minutes. This can make the UI unresponsive and logins fail." + description: "n8n Node.js event loop lag has exceeded 1 second for the last 1 hour. This can make the UI unresponsive and logins fail." - alert: n8nPodNotReady expr: kube_pod_status_ready{condition="true", pod=~"mop-n8n-.*"} == 0 - for: 5m + for: 1h labels: severity: critical annotations: summary: "n8n pod is not ready" - description: "n8n pod has been in non-ready state for more than 5 minutes. This is likely due to failing readiness probes (/healthz check failing, possibly database connection issues)." \ No newline at end of file + description: "n8n pod has been in non-ready state for more than 1 hour. This is likely due to failing readiness probes (/healthz check failing, possibly database connection issues)." \ No newline at end of file diff --git a/k8s/wireguard/monitoring.yaml b/k8s/wireguard/monitoring.yaml index 4dcbd7b..4e10ac5 100644 --- a/k8s/wireguard/monitoring.yaml +++ b/k8s/wireguard/monitoring.yaml @@ -47,7 +47,7 @@ spec: rules: - alert: WireguardExporterDown expr: up{job="wireguard-exporter"} == 0 - for: 5m + for: 1h labels: severity: critical annotations: @@ -55,7 +55,7 @@ spec: description: "The WireGuard Prometheus exporter is unreachable. The pod may have crashed or is unresponsive." - alert: WireguardInterfaceDown expr: absent(wireguard_sent_bytes_total{interface="wg0"}) == 1 - for: 5m + for: 1h labels: severity: critical annotations: @@ -63,7 +63,7 @@ spec: description: "The WireGuard interface wg0 is not reporting any statistics. The VPN tunnel might be down or inactive." - alert: WireguardPeerOffline expr: (time() - wireguard_latest_handshake_seconds{interface="wg0"}) > 300 - for: 5m + for: 1h labels: severity: critical annotations: