diff --git a/docs/adr/0004-structured-personal-data-datastore.md b/docs/adr/0004-structured-personal-data-datastore.md new file mode 100644 index 0000000..43b72f9 --- /dev/null +++ b/docs/adr/0004-structured-personal-data-datastore.md @@ -0,0 +1,141 @@ +--- +status: "accepted" +date: 2026-10-06 +decision-makers: ["Moritz Graf", "Antigravity"] +consulted: [] +informed: [] +--- + +# ADR 0004: Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki) + +## Context and Problem Statement + +OpenClaw manages personal logging and family assistance for the Graf family across several daily domains: +1. **Personal Nutrition:** Daily food intake, macro tracking (protein, carbs, sugar, fat, fiber, salt, calories) in `entities/nutrition-log/moritz/*.md`. +2. **Baby Sleep & Routines:** Sleep logs, naps, night sleep, bottle feeds, and developmental milestones for Mila in `entities/people/mila-schlafprotokoll.md`. +3. **Sports & Physiological Balance:** Activities, daily calorie expenditure (RMR + active), and net caloric surplus/deficit synced with Garmin Connect. +4. **Culinary Knowledge:** Recipes with ingredient proportions, cooking steps, and hardware settings. +5. **Points of Interest (POIs):** Local stores, farm stands (Hofläden), beer gardens, and playgrounds in Regensburg. + +Historically, this data was logged directly into Markdown tables and bulleted lists in the Knowledge Wiki (`knowledge-wiki/`). An analysis of existing log files revealed severe systemic failure modes: +- **LLM Arithmetic Drift:** In `2026-09.md` and `2026-10.md`, the LLM consistently failed at daily macro sums (errors up to 75 kcal and 7.5 g protein per day). The derived "Vortags-Bilanz" (caloric deficit/surplus against Garmin) was calculated from incorrect totals. +- **Structural and Chronological Degradation:** In `mila-schlafprotokoll.md`, dates drifted out of order (`29.09 → Sep Summary → 30.09 → 01.10 → 03.10 → 02.10 → 28.09`), and every night sleep was duplicated (once in the evening, once the next morning). +- **Inability to Perform Spatial & Aggregate Queries:** Questions like *"What was Mila's average night sleep duration over the last 14 days?"* or *"Which beer gardens are within 500m of Grabengasse 7?"* required either multi-turn LLM hallucination or reading hundreds of lines of raw text. + +How do we establish a structured, reliable, and queryable data layer for OpenClaw that guarantees mathematical precision while upholding our Everything-as-Code (ADR 0003) and Git-as-Source-of-Truth architecture? + +--- + +## Decision Drivers + +* **Mathematical Precision:** Zero LLM arithmetic. All summations, caloric balances, sleep durations, and nutrient rollups must be computed by a deterministic SQL engine. +* **Git as the Single Source of Truth (ADR 0003):** No external database silos (PostgreSQL, standalone SQLite PVCs) that can drift from the git-backed wiki. All state must remain git-versioned, diffable, and merge-friendly. +* **Separation of Concerns (Math vs. Narrative):** Delegate structured math, joins, and coordinates to DuckDB. Delegate natural-language understanding, nuance, packaging photo interpretation, and cooking advice to the LLM. +* **Flexible Composability:** Foods can become ingredients in recipes, which can yield composite foods (e.g. wheat → baking mix → bread; tomatoes → sauce → pizza). +* **Multi-Channel Presentation:** Auto-updated Markdown files for SilverBullet (`wiki.moritzgraf.de`), Telegram inline text with `chart-to-chat` PNGs, and mobile-friendly HTML dashboards on `hosting.haumdaucher.de/family/`. +* **Zero-Tax Execution:** In-memory query execution in Python via `uv` without adding heavy runtime containers or pod memory overhead. + +--- + +## Considered Options + +1. **Option 1: Stricter Markdown Prompts + Python Lint Rules** + * Keep Markdown tables; add linter checks in `okf_lint.py` to flag wrong sums. + * *Drawback*: Reactive detection rather than proactive prevention. Cannot do spatial searches or aggregate SQL queries. + +2. **Option 2: External Relational Database (PostgreSQL / PVC SQLite)** + * Deploy PostgreSQL or write to a binary SQLite database on the Kubernetes PVC. + * *Drawback*: Violates ADR 0003. Creates a separate data silo outside git, introduces backup synchronization risks, and binary files cannot be cleanly merged or diffed in Git. + +3. **Option 3: Pure Event Sourcing for All Data** + * Force every piece of data (including stores, recipes, and food labels) into an immutable event log. + * *Drawback*: Over-engineers static catalogs. Catalog entities (like places or recipes) become awkward to manage and query when forced into point-in-time event streams. + +4. **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git (Selected)** + * Store structured records in line-delimited JSON (`.jsonl`) inside the Knowledge Wiki repository (`data/`), using in-memory DuckDB as the calculation and query engine. + * Divide data into three natural archetypes: **Events**, **Entities**, and **Documents**. + +--- + +## Decision Outcome + +Chosen Option: **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git**. + +### 1. The Three Data Archetypes + +``` +┌────────────────────────────────────────────────────────────────────────┐ +│ Three Data Archetypes │ +├─────────────────────────┬─────────────────────────┬────────────────────┤ +│ 1. EVENTS (Time-Series) │ 2. ENTITIES (Catalogs) │ 3. DOCUMENTS (Prose│ +├─────────────────────────┼─────────────────────────┼────────────────────┤ +│ • data/events/nutrition/│ • data/entities/foods/ │ • entities/recipes/│ +│ • data/events/sleep/ │ • data/entities/recipes/│ • syntheses/places/│ +│ • data/events/feeding/ │ • data/entities/places/ │ • sources/*.pdf │ +│ • data/events/body/ │ • data/entities/people/ │ │ +│ [Point & Interval Logs] │ [Durable Slug Identifiers] [Markdown / Media]│ +└────────────┬────────────┴────────────┬────────────┴──────────┬─────────┘ + │ │ │ + ▼ ▼ │ + ┌──────────────────────────────────────────────┐ │ + │ In-Memory DuckDB SQL Engine │ │ + │ • Schema Validation (STRUCT & CHECK) │ │ + │ • SQL Aggregations & Daily Deficit Joins │ │ + │ • Spatial Queries (ST_DWithin / Haversine) │ │ + └──────────────────────┬───────────────────────┘ │ + │ │ + ┌────────────┴────────────┐ │ + ▼ ▼ │ + ┌──────────────────┐ ┌──────────────────┐ │ + │ Telegram Output │ │ Generated Views │ │ + │ • Text Summaries │ │ • Markdown Views ◄────────────┘ + │ • chart-to-chat │ │ • HTML Dashboards│ (Frontmatter: + └──────────────────┘ │ (/family/...) │ datastore_ref) + └──────────────────┘ +``` + +1. **Events (Time-Series Logs):** + - Stored in `data/events//.jsonl`, sorted by `(start, id)`. + - Unified envelope: `id` (ULID), `kind`, `subject`, `actor`, `start`, `end` (supports open intervals where `end=NULL`), `estimated`, `meal`, `note`, `payload` (JSON cast to typed STRUCT), `source_ref`. +2. **Entities (Structured Catalogs):** + - Stored in `data/entities/.jsonl`, sorted by slug `id`. + - `foods`: EU LMIV per-100g nutritional profile, ingredient lists, barcodes, density. + - `recipes`: Ingredients list (linking `foods` with exact grams or raw text), servings, calculated macros. + - `places`: Points of interest with `lat`, `lon`, category, address, tags, and linked documents. + - `people`: Family profile registry. +3. **Documents (Prose & Media):** + - Stored as standard Markdown in `knowledge-wiki/`. + - Deterministic two-way link: Document YAML frontmatter contains `datastore_ref: ":"`, while the entity JSONL record lists `documents: ["path/to/doc.md"]`. + +### 2. Dedicated Raw Garmin Mirror (`data/garmin/`) +- Nightly zero-LLM Python sync script fetches raw Garmin API responses directly into: + - `data/garmin/activities/.jsonl` + - `data/garmin/daily/.jsonl` + - `data/garmin/sleep/.jsonl` +- DuckDB queries raw JSON directly with `read_json_auto()` and computes the daily caloric deficit/surplus by joining nutrition intake with Garmin total daily energy expenditure. + +### 3. Spatial & Geospatial Radius Lookups +- DuckDB attempts to initialize the official `spatial` extension (`LOAD spatial;`). +- If offline or binary installation fails, it transparently falls back to an internal pure-SQL Haversine macro `distance_m(lat1, lon1, lat2, lon2)`, ensuring zero-failure execution across any environment. + +### 4. Privacy & Hosting Routes (`hosting.haumdaucher.de`) +- **Public (`/recipes/*`, `/places/*`, `/public/*`):** Publicly accessible recipe cards and shareable dossiers. +- **Family (`/family/*`):** Protected by Traefik ForwardAuth linked to Authentik group `family` (nutrition dashboards, baby sleep charts). +- **Private (`/private/*`):** Protected by Authentik for personal vital logs. + +--- + +## Consequences + +### Positive +- **Guaranteed Calculation Accuracy:** All calories, macronutrient totals, sleep statistics, and energy balances are computed in SQL. Zero LLM math errors. +- **Git-Native & Mergable:** Text-based JSONL files partition data cleanly and produce clean 1-line git diffs. +- **Offline & Low-Latency:** In-memory DuckDB reads months of JSONL in single-digit milliseconds. +- **Preserved Human Readability:** SilverBullet (`wiki.moritzgraf.de`) and Obsidian continue to display updated Markdown tables generated directly from the database. + +### Negative & Mitigations +- **Generated Markdown Views Must Not Be Hand-Edited:** + - *Mitigation*: Generated files include a prominent `` warning comment. + - *Mitigation*: Git pre-commit hook and the weekly IT Guy audit verify that generated views match the underlying JSONL data. +- **Dual-Step Agent Retrieval:** Agents execute `datastore query` for structured data, then `wiki_get` on linked documents if full narrative text is needed. + - *Mitigation*: Clear instructions and query helpers in the `datastore` skill prompt. diff --git a/docs/adr/README.md b/docs/adr/README.md index f523fec..ef91b90 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -13,6 +13,7 @@ We follow the [Markdown Architectural Decision Records (MADR)](https://adr.githu | [**0001**](0001-hybrid-mcp-tool-search-and-subagents.md) | **Hybrid MCP Tool Search and Subagent Delegation for AI Agents** | `accepted` | 2026-09-12 | OpenClaw / AI / MCP | | [**0002**](0002-openclaw-model-selection-tiering-and-economics.md) | **OpenClaw Multi-Tier Model Selection, Latency Optimization, and Inference Economics** | `proposed` | 2026-09-13 | OpenClaw / AI / Models | | [**0003**](0003-everything-as-code-and-reproducible-infrastructure.md) | **Everything as Code (EaC) and Reproducible Declarative Infrastructure** | `accepted` | 2026-09-17 | Architecture / Monorepo / EaC | +| [**0004**](0004-structured-personal-data-datastore.md) | **Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)** | `accepted` | 2026-10-06 | OpenClaw / Data / Storage | --- diff --git a/k8s/minio/hosting-ingress.yaml b/k8s/minio/hosting-ingress.yaml index 9369e15..170aa8d 100644 --- a/k8s/minio/hosting-ingress.yaml +++ b/k8s/minio/hosting-ingress.yaml @@ -7,6 +7,7 @@ spec: addPrefix: prefix: /hosting --- +# Public Ingress for hosting.haumdaucher.de (Recipes, Public Assets) apiVersion: networking.k8s.io/v1 kind: Ingress metadata: @@ -16,6 +17,7 @@ metadata: kubernetes.io/ingress.class: "traefik" cert-manager.io/cluster-issuer: "letsencrypt-prod" kubernetes.io/tls-acme: "true" + traefik.ingress.kubernetes.io/router.priority: "10" traefik.ingress.kubernetes.io/router.middlewares: minio-hosting-add-prefix@kubernetescrd spec: ingressClassName: traefik @@ -34,3 +36,34 @@ spec: - hosts: - hosting.haumdaucher.de secretName: hosting-haumdaucher-de-tls +--- +# Protected Ingress for /family (Family Dashboards, Nutrition & Sleep Analytics) +# Protected by Authentik ForwardAuth +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: minio-hosting-family + namespace: minio + annotations: + kubernetes.io/ingress.class: "traefik" + cert-manager.io/cluster-issuer: "letsencrypt-prod" + kubernetes.io/tls-acme: "true" + traefik.ingress.kubernetes.io/router.priority: "100" + traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd,minio-hosting-add-prefix@kubernetescrd +spec: + ingressClassName: traefik + rules: + - host: hosting.haumdaucher.de + http: + paths: + - path: /family + pathType: Prefix + backend: + service: + name: minio + port: + number: 9000 + tls: + - hosts: + - hosting.haumdaucher.de + secretName: hosting-haumdaucher-de-tls diff --git a/k8s/openclaw/openclaw-instance.secret.yaml b/k8s/openclaw/openclaw-instance.secret.yaml index a2536d8..e2882cf 100644 Binary files a/k8s/openclaw/openclaw-instance.secret.yaml and b/k8s/openclaw/openclaw-instance.secret.yaml differ diff --git a/k8s/openclaw/setup-garmin-cron.sh b/k8s/openclaw/setup-garmin-cron.sh new file mode 100755 index 0000000..9437cdb --- /dev/null +++ b/k8s/openclaw/setup-garmin-cron.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# ============================================================================== +# setup-garmin-cron.sh +# Declaratively registers the Garmin Connect Nightly Sync in OpenClaw. +# Schedule: Daily at 03:30 Europe/Berlin (cron: 30 3 * * *) +# ============================================================================== + +set -euo pipefail + +NAMESPACE="${NAMESPACE:-openclaw}" +POD_NAME="${POD_NAME:-openclaw-0}" +CONTAINER_NAME="${CONTAINER_NAME:-openclaw}" + +JOB_NAME="garmin-nightly-mirror" +CRON_SCHEDULE="30 3 * * *" +TIMEZONE="Europe/Berlin" +TELEGRAM_GROUP="-1004326921510" +TELEGRAM_TOPIC="20" + +echo "=== Registering Garmin Connect Nightly Sync in OpenClaw ($POD_NAME) ===" + +# 1. Check if job already exists +EXISTING_ID=$(kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list 2>/dev/null | grep "$JOB_NAME" | awk '{print $1}' || true) + +if [ -n "$EXISTING_ID" ]; then + echo "Found existing job with ID $EXISTING_ID. Removing for idempotent recreation..." + kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron rm "$EXISTING_ID" || true +fi + +# 2. Add the job +echo "Creating automation $JOB_NAME ($CRON_SCHEDULE @ $TIMEZONE)..." + +MESSAGE=$(cat << 'EOF' +Führe den nächtlichen Garmin Connect Sync durch. +Vorgehen: +1. Führe den Garmin-Sync-Befehl aus: + uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/garmin_sync.py --days 2 --data-dir /home/node/.openclaw/wiki/main/data +2. Aktualisiere die gerenderten Markdown-Ansichten: + PYTHONPATH=/home/node/mop-skills/mop-skills/skills/datastore/scripts uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/render_views.py --data-dir /home/node/.openclaw/wiki/main/data --wiki-dir /home/node/.openclaw/wiki/main +3. Git Commit & Push im Knowledge Wiki: + cd /home/node/.openclaw/wiki/main && git add data/garmin/ entities/nutrition-log/ && git diff --staged --quiet || (git commit -m "chore(garmin): automated nightly mirror [skip ci]" && git push origin main) +4. Finale Antwort: + - Bei Erfolg: Gib exakt 'NO_REPLY' aus (intentional silence, kein Telegram-Spam in der Nacht). + - Nur bei Authentifizierungsfehlern: Informiere kurz via Telegram, dass ein Garmin-Re-Auth nötig ist. +EOF +) + +kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \ + --name "$JOB_NAME" \ + --description "Nächtlicher deterministischer Spiegel der Garmin Connect Daten (Aktivitäten, Daily Summary, Schlaf) in Git JSONL" \ + --cron "$CRON_SCHEDULE" \ + --tz "$TIMEZONE" \ + --agent "fitness" \ + --model "openrouter/deepseek/deepseek-v4.1-flash" \ + --session "isolated" \ + --channel "telegram" \ + --to "$TELEGRAM_GROUP" \ + --thread-id "$TELEGRAM_TOPIC" \ + --message "$MESSAGE" + +echo "=== Automation successfully registered! ===" +kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list | grep "$JOB_NAME" || true diff --git a/knowledge-wiki b/knowledge-wiki index 43d07da..1397837 160000 --- a/knowledge-wiki +++ b/knowledge-wiki @@ -1 +1 @@ -Subproject commit 43d07dafa013aa6757c971df31bc779335996af5 +Subproject commit 139783762361e3c755009606d998f103690bea9e