moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

refactor(structure): reorganize monorepo with projects, terraform hierarchy, and strict k8s namespace rules

Browse Source
This commit is contained in:
Moritz Graf
2026-10-12 13:49:41 +02:00
parent 4831147087
commit 8f01b75bd6
4766 changed files with 551 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/AGENTS.md
+3 -2
View File
@@ -11,8 +11,9 @@ This directory contains the Kubernetes manifests and Helm charts for a single-no
* **SSL**: `cert-manager` (LetsEncrypt)
## Directory Structure
* **Top-level folders**: Each folder corresponds to a Kubernetes **namespace**.
* Example: `mailu/` contains resources for the `mailu` namespace.
* **Strict Namespace Mapping**: **Every top-level subdirectory in `k8s/` MUST correspond strictly to a Kubernetes namespace.**
* Example: `mailu/` contains resources for the `mailu` namespace, `grabengasse/` for `grabengasse`, `haumdaucher/` for `haumdaucher`, etc.
* **NO non-namespace folders**: Loose utility scripts belong in the root `scripts/` directory, application source code belongs in `projects/`, and general documentation belongs in namespace READMEs or `docs/`.
* **Documentation**: `README.md` is the **authoritative source** for deployment commands. Always check it before running commands.
## Code Style & Conventions
k8s/README.md
+7 -7
View File
@@ -678,7 +678,7 @@ kubectl create ns migrate
# backup on moritzgraf.de
ssh moritzgraf.de "sudo su - docker -c 'cd /home/docker/mailu && docker-compose stop' && sudo su - root -c 'cd /home/docker && tar czvf mailu.tar.gz ./mailu && mv mailu.tar.gz /home/moritz/ && chown moritz. /home/moritz/mailu.tar.gz' && scp mailu.tar.gz haumdaucher.de:/home/moritz/"
# terraform change
cd ../terraform && terraform apply
cd ../terraform/infrastructure && terraform apply
# helm apply
cd ../k8s
helm upgrade --create-namespace --install mailu mailu/mailu -n mailu -f mailu/mailu.secret.yml
@@ -873,13 +873,13 @@ Grabengasse 7 home website and backend service deployed in the `grabengasse` nam
# Login to local registry (if not already logged in)
docker login registry.haumdaucher.de -u moritz -p "PPf10musSh!!"
# Build and push backend image
cd website/grabengasse_backend
make all
# Build and push images via root projects/grabengasse Makefile
cd projects/grabengasse
make build push
# Build and push frontend image
cd ../grabengasse_frontend
make all
# Or build individual components:
cd projects/grabengasse/frontend && make all
cd ../backend && make all
```
### Deploy to Kubernetes
k8s/auth/README.md
+2 -2
View File
@@ -170,10 +170,10 @@ The Authentik login screen is customized with a **Haumdaucher** glassmorphism th
* **Branded Footer**: "HAUMDAUCHER Wurst und Spezialitäten n.e.V." subtle watermark.
### Managing Theme via Terraform
Branding is provisioned via the `authentik_brand` resource in `terraform/authentik.tf` using the stylesheet `terraform/haumdaucher-theme.css`:
Branding is provisioned via the `authentik_brand` resource in `terraform/infrastructure/authentik.tf` using the stylesheet `terraform/infrastructure/haumdaucher-theme.css`:
```bash
cd terraform
cd terraform/infrastructure
terraform apply -target=authentik_brand.haumdaucher
```
k8s/auth/USER_MANAGEMENT.md
+5 -5
View File
@@ -22,7 +22,7 @@ This guide covers everything you need to know about managing users in Authentik
| Question | Answer & Key Mechanism |
| :--- | :--- |
| **How to add new users?** | **4 Ways**: (1) Admin Web UI (`Directory > Users`), (2) **Invitations flow** (`Directory > Invitations` - recommended for self-service), (3) **Terraform** (`authentik_user` resource), or (4) **REST API / Python script** for bulk imports. |
| **Can this be done in Terraform?** | **Yes.** Use `resource "authentik_user"` in `terraform/authentik.tf`. Supports custom attributes via `jsonencode()`, group assignments, and active status. |
| **Can this be done in Terraform?** | **Yes.** Use `resource "authentik_user"` in `terraform/infrastructure/authentik.tf`. Supports custom attributes via `jsonencode()`, group assignments, and active status. |
| **Can I change the email sent to the user?** | **Yes.** Configured in **Flows & Stages > Stages > Email Stage**. You can customize the subject line, token expiration, and HTML template (written in Django template syntax with access to user fields and custom attributes). |
| **Can I limit what a user has access to?** | **Yes.** Through **Policy Bindings** attached to Applications: Group Membership Policies (e.g., `infra-admins` vs `users`), IP/Geo Policies, or Python Expression Policies checking custom attributes (e.g., only active club members). |
| **How to handle Excel tables with custom attributes?** | Authentik stores arbitrary key-value pairs in `user.attributes` (JSON). You can import an Excel/CSV file with a Python script or Terraform `csvdecode()`, map fields like `haumdaucher_mitgliedsnummer` or `second_email`, and expose them to downstream apps (Nextcloud, Grafana, Gitea) via OIDC Scope Mappings. |
@@ -75,7 +75,7 @@ If you encounter this screen when opening an invite link:
### Solution: The Haumdaucher Custom Invitation Flow (`haumdaucher-invitation-flow`)
We provision a dedicated, streamlined flow in `terraform/authentik.tf`:
We provision a dedicated, streamlined flow in `terraform/infrastructure/authentik.tf`:
```mermaid
graph LR
@@ -122,7 +122,7 @@ graph LR
## 🏗️ Method 2: User Provisioning via Terraform
You can manage users, groups, and permissions directly in `terraform/authentik.tf`.
You can manage users, groups, and permissions directly in `terraform/infrastructure/authentik.tf`.
### 1. Single User Definition
```hcl
@@ -161,7 +161,7 @@ max.mustermann,Max Mustermann,max@example.com,max.privat@gmail.com,HM-2024-042,f
anna.schmidt,Anna Schmidt,anna@example.com,anna.work@company.de,HM-2024-043,true
```
In `terraform/authentik.tf`:
In `terraform/infrastructure/authentik.tf`:
```hcl
locals {
csv_users = csvdecode(file("${path.module}/users.csv"))
@@ -479,7 +479,7 @@ graph TD
| **haumdaucher.de** | `haumdaucher.de` | ForwardAuth / OIDC | **`Admin`** ✅ | **`Member Access`** ✅ | ❌ *Denied* |
### Policy Bindings in Terraform
All policy bindings are enforced directly in `terraform/authentik.tf`:
All policy bindings are enforced directly in `terraform/infrastructure/authentik.tf`:
```hcl
# 1. Admin-Only Tools (Restricted strictly to infra-admins)
k8s/htpasswd.py
-33
View File
@@ -1,33 +0,0 @@
import bcrypt
import sys
def generate_htpasswd(username, password):
# Generate a salt
salt = bcrypt.gensalt()
# Hash the password with the generated salt
hashed_password = bcrypt.hashpw(password.encode('utf-8'), salt)
# Format the string for the .htpasswd
htpasswd_string = f"{username}:{hashed_password.decode('utf-8')}"
return htpasswd_string
def main():
# Check if the correct number of arguments are provided
if len(sys.argv) != 3:
print("Usage: python script.py <username> <password>")
sys.exit(1)
# Get the username and password from the command-line arguments
username = sys.argv[1]
password = sys.argv[2]
# Generate the htpasswd string
htpasswd_string = generate_htpasswd(username, password)
# Print the htpasswd string to stdout
print(htpasswd_string)
if __name__ == "__main__":
main()
k8s/llm_hosting.md → k8s/llm/README.md
View File
k8s/monitoring/README_Z2M.md
+1 -1
View File
@@ -57,7 +57,7 @@ The dashboard is exposed on your custom domain with TLS and basic authentication
* **Username:** `admin`
* **Password:** `moritz-z2m-2026`
*Note: You can change the basic auth password by running the `k8s/htpasswd.py` script on your machine to generate a new bcrypt hash, then updating the `users` value in `k8s/monitoring/zigbee2mqtt.secret.yaml`.*
*Note: You can change the basic auth password by running the `scripts/htpasswd.py` script on your machine to generate a new bcrypt hash, then updating the `users` value in `k8s/monitoring/zigbee2mqtt.secret.yaml`.*
---
k8s/openclaw/scripts/compact_sessions.py
+275
View File
@@ -0,0 +1,275 @@
#!/usr/bin/env python3
"""
Nightly multi-agent session maintenance and compaction script for OpenClaw.
1. Pre-warms OS page cache for SQLite databases on spinning disks.
2. Recovers any wedged or locked sessions via recover_sessions.py.
3. Prunes abandoned isolated cron sessions (>48h old) to prevent token/state bloat.
4. Tier A Hard-Reset: Resets koch, hebamme, and shopping topics to 0 tokens (clean archive).
5. Tier B Smart-Compaction: Compacts active sessions exceeding token thresholds (5s pacing).
6. Checkpoints SQLite WAL journals across all agent databases to disk.
7. Optimizes SQLite query planner without blocking full VACUUM.
"""
import glob
import json
import os
import sqlite3
import subprocess
import sys
import time
THRESHOLD_TOKENS = 20000
THRESHOLD_TRANSCRIPT_BYTES = 300000
THRESHOLD_EVENT_COUNT = 40
CRON_MAX_AGE_HOURS = 48
RPC_TIMEOUT_MS = 180000
# Tier A: Event-logging agents whose state is 100% externalized (DuckDB / JSONL / Groceries DB).
# Nightly reset to 0 prevents cross-day hallucination and eliminates LLM compaction token costs.
TIER_A_RESET_TARGETS = [
("koch", "agent:koch:telegram:group:-1004326921510:topic:21"),
("hebamme", "agent:hebamme:telegram:group:-1004326921510:topic:22"),
("shopping", "agent:shopping:telegram:group:-1004326921510:topic:115"),
]
def prewarm_page_cache():
print("=== Pre-warming SQLite databases into OS page cache ===")
t0 = time.time()
try:
subprocess.run(
"cat /home/node/.openclaw/state/openclaw.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true",
shell=True,
timeout=90.0
)
print(f"Page cache pre-warmed in {time.time() - t0:.2f}s")
except Exception as e:
print(f"Page cache pre-warming warning: {e}")
def prune_stale_cron_sessions():
print(f"=== Pruning isolated cron sessions older than {CRON_MAX_AGE_HOURS}h ===")
cutoff_ms = int(time.time() * 1000) - (CRON_MAX_AGE_HOURS * 3600 * 1000)
total_pruned = 0
total_tokens_reclaimed = 0
for db in sorted(glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite")):
agent = db.split("/")[5]
try:
conn = sqlite3.connect(db, timeout=30.0)
conn.execute("PRAGMA foreign_keys = ON;")
cursor = conn.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type=\"table\"")
all_tables = set(r[0] for r in cursor.fetchall())
cursor.execute("SELECT session_key, current_session_id, updated_at, entry_json FROM session_nodes WHERE session_key LIKE \"%:cron:%\"")
stale = []
for k, sid, up, ej in cursor.fetchall():
if up and up < cutoff_ms:
tok = 0
if ej:
try:
tok = json.loads(ej).get("totalTokens", 0) or 0
except Exception:
pass
stale.append((k, sid, tok))
if stale:
tokens_reclaimed = sum(t[2] for t in stale)
for k, sid, _ in stale:
for t in ["trajectory_runtime_events", "transcript_events", "transcript_event_identities", "session_transcript_active_events", "transcript_rewrite_watermarks", "session_transcript_index_state", "session_transcript_fts", "session_conversations"]:
if t in all_tables and sid:
cursor.execute(f"DELETE FROM {t} WHERE session_id = ?", (sid,))
for t in ["session_participants", "heartbeat_outcomes", "session_progress_cards", "board_widgets", "board_tabs"]:
if t in all_tables:
cursor.execute(f"DELETE FROM {t} WHERE session_key = ?", (k,))
if "memory_entry_origins" in all_tables:
cursor.execute("DELETE FROM memory_entry_origins WHERE session_key = ? OR session_id = ?", (k, sid or ""))
if "session_windows" in all_tables:
cursor.execute("DELETE FROM session_windows WHERE session_key = ? OR session_id = ?", (k, sid or ""))
cursor.execute("DELETE FROM session_nodes WHERE session_key = ?", (k,))
conn.commit()
total_pruned += len(stale)
total_tokens_reclaimed += tokens_reclaimed
print(f" [{agent}] Pruned {len(stale)} dead cron sessions ({tokens_reclaimed} tokens reclaimed)")
# Unconditionally resolve any dangling foreign key references across all tables
while True:
cursor.execute("PRAGMA foreign_key_check;")
violations = cursor.fetchall()
if not violations:
break
for table, rowid, parent, fkid in violations:
cursor.execute(f"DELETE FROM {table} WHERE rowid = ?", (rowid,))
conn.commit()
conn.close()
except Exception as e:
print(f" [{agent}] Pruning error: {e}")
# Clear resolved quarantine entries if all databases are healthy
try:
q_path = "/home/node/.openclaw/state/openclaw-quarantine.sqlite"
if os.path.exists(q_path):
q_conn = sqlite3.connect(q_path)
q_cur = q_conn.cursor()
q_cur.execute("DELETE FROM quarantined_databases")
q_conn.commit()
q_conn.close()
except Exception as e:
print(f"Quarantine cleanup error: {e}")
print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.")
def reset_tier_a_sessions():
print("=== Tier A: Daily Hard-Reset to 0 (koch, hebamme, shopping) ===")
for agent, key in TIER_A_RESET_TARGETS:
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
if not os.path.exists(db_path):
continue
try:
con = sqlite3.connect(db_path, timeout=10.0)
row = con.execute("SELECT current_session_id FROM session_nodes WHERE session_key=?", (key,)).fetchone()
con.close()
if not row or not row[0]:
print(f" [Tier A] {agent} ({key}) already at 0 (no active session).")
continue
print(f" [Tier A] Resetting {agent} ({key})...")
r = subprocess.run(
["openclaw", "sessions", "delete", key, "--agent", agent, "--yes"],
capture_output=True, text=True, timeout=60.0
)
if r.returncode == 0:
print(f" ✓ Reset success: {r.stdout.strip()}")
else:
print(f" ✗ Reset error: {r.stderr.strip()}")
except Exception as e:
print(f" ✗ Exception resetting {agent}: {e}")
time.sleep(1)
def get_session_metrics(agent, key, sid=None):
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
if not os.path.exists(db_path):
return 0, 0
try:
con = sqlite3.connect(db_path, timeout=5.0)
cur = con.cursor()
if not sid:
row = cur.execute("SELECT current_session_id FROM session_nodes WHERE session_key=?", (key,)).fetchone()
sid = row[0] if row else None
if not sid:
con.close()
return 0, 0
ev_cnt, ev_bytes = cur.execute(
"SELECT count(*), coalesce(sum(length(event_json)), 0) FROM transcript_events WHERE session_id=?",
(sid,)
).fetchone()
con.close()
return ev_cnt, ev_bytes
except Exception:
return 0, 0
def compact_active_sessions():
tier_a_keys = set(t[1] for t in TIER_A_RESET_TARGETS)
print(f"=== Tier B: Running Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens / {THRESHOLD_TRANSCRIPT_BYTES//1024}KB, LLM summarization) ===")
res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True)
if res.returncode != 0:
print(f"Failed to list sessions: {res.stderr}")
return
sessions = json.loads(res.stdout).get("sessions", [])
compacted = 0
skipped = 0
for s in sessions:
key = s.get("key", "")
agent = s.get("agentId") or "main"
tokens = s.get("totalTokens") or 0
sid = s.get("sessionId")
# Skip Tier A agents as they are handled via hard-reset
if key in tier_a_keys:
continue
# Target main sessions and all telegram sessions (topics and direct)
is_target = (
key == "agent:main:main" or
":telegram:" in key or
(key.startswith("agent:") and key.endswith(":main"))
)
if is_target and not key.endswith(":heartbeat"):
ev_cnt, ev_bytes = get_session_metrics(agent, key, sid)
needs_compaction = (
tokens >= THRESHOLD_TOKENS or
ev_bytes >= THRESHOLD_TRANSCRIPT_BYTES or
(tokens == 0 and ev_cnt >= THRESHOLD_EVENT_COUNT)
)
if needs_compaction:
stat_str = f"{tokens} tokens" if tokens else f"{ev_bytes//1024}KB / {ev_cnt} events"
print(f"Compacting {agent} ({stat_str}) -> {key}...")
c_res = subprocess.run(
["openclaw", "sessions", "compact", key, "--agent", agent, "--timeout", str(RPC_TIMEOUT_MS)],
capture_output=True, text=True
)
if c_res.returncode == 0:
print(f" ✓ {c_res.stdout.strip()}")
compacted += 1
elif "Already compacted" in c_res.stderr or "Already compacted" in c_res.stdout:
print(f" ✓ Already compacted.")
skipped += 1
else:
print(f" ✗ {c_res.stderr.strip()}; retrying with --max-lines 80 fallback...")
fb_res = subprocess.run(
["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", "80", "--timeout", str(RPC_TIMEOUT_MS)],
capture_output=True, text=True
)
if fb_res.returncode == 0:
print(f" ✓ (fallback) {fb_res.stdout.strip()}")
compacted += 1
else:
print(f" ✗ Fallback failed: {fb_res.stderr.strip()}")
time.sleep(5) # 5s pause to avoid rotational HDD I/O saturation
else:
skipped += 1
print(f"Compaction complete: {compacted} compacted, {skipped} skipped.")
def checkpoint_wal_journals():
print("=== Checkpointing SQLite WAL journals across all agent databases ===")
dbs = ["/home/node/.openclaw/state/openclaw.sqlite"] + glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite")
wal_freed = 0
for db in dbs:
wal = db + "-wal"
if os.path.exists(wal):
sz = os.path.getsize(wal)
try:
con = sqlite3.connect(db, timeout=30.0)
con.execute("PRAGMA wal_checkpoint(TRUNCATE);")
con.close()
wal_freed += sz
except Exception as e:
print(f" ✗ Error checkpointing {db}: {e}")
print(f"WAL Checkpoint complete: {wal_freed / (1024*1024):.2f} MB merged to disk.")
def vacuum_and_optimize():
print("=== Running SQLite OPTIMIZE across all databases ===")
dbs = ["/home/node/.openclaw/state/openclaw.sqlite"] + glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite")
for db in dbs:
try:
con = sqlite3.connect(db, timeout=30.0)
con.execute("PRAGMA optimize;")
con.close()
except Exception as e:
print(f" ✗ Error optimizing {db}: {e}")
print("SQLite optimization complete.")
def main():
prewarm_page_cache()
try:
subprocess.run(["python3", "/home/node/.openclaw/scripts/recover_sessions.py"], check=False)
except Exception:
pass
prune_stale_cron_sessions()
reset_tier_a_sessions()
compact_active_sessions()
checkpoint_wal_journals()
vacuum_and_optimize()
if __name__ == "__main__":
main()