feat(datastore): implement structured personal data layer (ADR 0004, ForwardAuth /family, and OpenClaw protocol)

2026-10-06 18:18:27 +02:00
parent 9d890a290f
commit f717a9a2f7
6 changed files with 238 additions and 1 deletions
@@ -0,0 +1,141 @@
+---
+status: "accepted"
+date: 2026-10-06
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0004: Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)
+
+## Context and Problem Statement
+
+OpenClaw manages personal logging and family assistance for the Graf family across several daily domains:
+1. **Personal Nutrition:** Daily food intake, macro tracking (protein, carbs, sugar, fat, fiber, salt, calories) in `entities/nutrition-log/moritz/*.md`.
+2. **Baby Sleep & Routines:** Sleep logs, naps, night sleep, bottle feeds, and developmental milestones for Mila in `entities/people/mila-schlafprotokoll.md`.
+3. **Sports & Physiological Balance:** Activities, daily calorie expenditure (RMR + active), and net caloric surplus/deficit synced with Garmin Connect.
+4. **Culinary Knowledge:** Recipes with ingredient proportions, cooking steps, and hardware settings.
+5. **Points of Interest (POIs):** Local stores, farm stands (Hofläden), beer gardens, and playgrounds in Regensburg.
+
+Historically, this data was logged directly into Markdown tables and bulleted lists in the Knowledge Wiki (`knowledge-wiki/`). An analysis of existing log files revealed severe systemic failure modes:
+- **LLM Arithmetic Drift:** In `2026-09.md` and `2026-10.md`, the LLM consistently failed at daily macro sums (errors up to 75 kcal and 7.5 g protein per day). The derived "Vortags-Bilanz" (caloric deficit/surplus against Garmin) was calculated from incorrect totals.
+- **Structural and Chronological Degradation:** In `mila-schlafprotokoll.md`, dates drifted out of order (`29.09 → Sep Summary → 30.09 → 01.10 → 03.10 → 02.10 → 28.09`), and every night sleep was duplicated (once in the evening, once the next morning).
+- **Inability to Perform Spatial & Aggregate Queries:** Questions like *"What was Mila's average night sleep duration over the last 14 days?"* or *"Which beer gardens are within 500m of Grabengasse 7?"* required either multi-turn LLM hallucination or reading hundreds of lines of raw text.
+
+How do we establish a structured, reliable, and queryable data layer for OpenClaw that guarantees mathematical precision while upholding our Everything-as-Code (ADR 0003) and Git-as-Source-of-Truth architecture?
+
+---
+
+## Decision Drivers
+
+* **Mathematical Precision:** Zero LLM arithmetic. All summations, caloric balances, sleep durations, and nutrient rollups must be computed by a deterministic SQL engine.
+* **Git as the Single Source of Truth (ADR 0003):** No external database silos (PostgreSQL, standalone SQLite PVCs) that can drift from the git-backed wiki. All state must remain git-versioned, diffable, and merge-friendly.
+* **Separation of Concerns (Math vs. Narrative):** Delegate structured math, joins, and coordinates to DuckDB. Delegate natural-language understanding, nuance, packaging photo interpretation, and cooking advice to the LLM.
+* **Flexible Composability:** Foods can become ingredients in recipes, which can yield composite foods (e.g. wheat → baking mix → bread; tomatoes → sauce → pizza).
+* **Multi-Channel Presentation:** Auto-updated Markdown files for SilverBullet (`wiki.moritzgraf.de`), Telegram inline text with `chart-to-chat` PNGs, and mobile-friendly HTML dashboards on `hosting.haumdaucher.de/family/`.
+* **Zero-Tax Execution:** In-memory query execution in Python via `uv` without adding heavy runtime containers or pod memory overhead.
+
+---
+
+## Considered Options
+
+1. **Option 1: Stricter Markdown Prompts + Python Lint Rules**
+   * Keep Markdown tables; add linter checks in `okf_lint.py` to flag wrong sums.
+   * *Drawback*: Reactive detection rather than proactive prevention. Cannot do spatial searches or aggregate SQL queries.
+
+2. **Option 2: External Relational Database (PostgreSQL / PVC SQLite)**
+   * Deploy PostgreSQL or write to a binary SQLite database on the Kubernetes PVC.
+   * *Drawback*: Violates ADR 0003. Creates a separate data silo outside git, introduces backup synchronization risks, and binary files cannot be cleanly merged or diffed in Git.
+
+3. **Option 3: Pure Event Sourcing for All Data**
+   * Force every piece of data (including stores, recipes, and food labels) into an immutable event log.
+   * *Drawback*: Over-engineers static catalogs. Catalog entities (like places or recipes) become awkward to manage and query when forced into point-in-time event streams.
+
+4. **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git (Selected)**
+   * Store structured records in line-delimited JSON (`.jsonl`) inside the Knowledge Wiki repository (`data/`), using in-memory DuckDB as the calculation and query engine.
+   * Divide data into three natural archetypes: **Events**, **Entities**, and **Documents**.
+
+---
+
+## Decision Outcome
+
+Chosen Option: **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git**.
+
+### 1. The Three Data Archetypes
+
+```
+┌────────────────────────────────────────────────────────────────────────┐
+│                        Three Data Archetypes                           │
+├─────────────────────────┬─────────────────────────┬────────────────────┤
+│ 1. EVENTS (Time-Series) │ 2. ENTITIES (Catalogs)  │ 3. DOCUMENTS (Prose│
+├─────────────────────────┼─────────────────────────┼────────────────────┤
+│ • data/events/nutrition/│ • data/entities/foods/  │ • entities/recipes/│
+│ • data/events/sleep/    │ • data/entities/recipes/│ • syntheses/places/│
+│ • data/events/feeding/  │ • data/entities/places/ │ • sources/*.pdf    │
+│ • data/events/body/     │ • data/entities/people/ │                    │
+│ [Point & Interval Logs] │ [Durable Slug Identifiers] [Markdown / Media]│
+└────────────┬────────────┴────────────┬────────────┴──────────┬─────────┘
+             │                         │                       │
+             ▼                         ▼                       │
+    ┌──────────────────────────────────────────────┐           │
+    │         In-Memory DuckDB SQL Engine          │           │
+    │  • Schema Validation (STRUCT & CHECK)        │           │
+    │  • SQL Aggregations & Daily Deficit Joins    │           │
+    │  • Spatial Queries (ST_DWithin / Haversine)  │           │
+    └──────────────────────┬───────────────────────┘           │
+                           │                                   │
+              ┌────────────┴────────────┐                      │
+              ▼                         ▼                      │
+     ┌──────────────────┐      ┌──────────────────┐            │
+     │ Telegram Output  │      │ Generated Views  │            │
+     │ • Text Summaries │      │ • Markdown Views ◄────────────┘
+     │ • chart-to-chat  │      │ • HTML Dashboards│ (Frontmatter:
+     └──────────────────┘      │   (/family/...)  │  datastore_ref)
+                               └──────────────────┘
+```
+
+1. **Events (Time-Series Logs):**
+   - Stored in `data/events/<domain>/<YYYY-MM>.jsonl`, sorted by `(start, id)`.
+   - Unified envelope: `id` (ULID), `kind`, `subject`, `actor`, `start`, `end` (supports open intervals where `end=NULL`), `estimated`, `meal`, `note`, `payload` (JSON cast to typed STRUCT), `source_ref`.
+2. **Entities (Structured Catalogs):**
+   - Stored in `data/entities/<collection>.jsonl`, sorted by slug `id`.
+   - `foods`: EU LMIV per-100g nutritional profile, ingredient lists, barcodes, density.
+   - `recipes`: Ingredients list (linking `foods` with exact grams or raw text), servings, calculated macros.
+   - `places`: Points of interest with `lat`, `lon`, category, address, tags, and linked documents.
+   - `people`: Family profile registry.
+3. **Documents (Prose & Media):**
+   - Stored as standard Markdown in `knowledge-wiki/`.
+   - Deterministic two-way link: Document YAML frontmatter contains `datastore_ref: "<collection>:<id>"`, while the entity JSONL record lists `documents: ["path/to/doc.md"]`.
+
+### 2. Dedicated Raw Garmin Mirror (`data/garmin/`)
+- Nightly zero-LLM Python sync script fetches raw Garmin API responses directly into:
+  - `data/garmin/activities/<YYYY-MM>.jsonl`
+  - `data/garmin/daily/<YYYY-MM>.jsonl`
+  - `data/garmin/sleep/<YYYY-MM>.jsonl`
+- DuckDB queries raw JSON directly with `read_json_auto()` and computes the daily caloric deficit/surplus by joining nutrition intake with Garmin total daily energy expenditure.
+
+### 3. Spatial & Geospatial Radius Lookups
+- DuckDB attempts to initialize the official `spatial` extension (`LOAD spatial;`).
+- If offline or binary installation fails, it transparently falls back to an internal pure-SQL Haversine macro `distance_m(lat1, lon1, lat2, lon2)`, ensuring zero-failure execution across any environment.
+
+### 4. Privacy & Hosting Routes (`hosting.haumdaucher.de`)
+- **Public (`/recipes/*`, `/places/*`, `/public/*`):** Publicly accessible recipe cards and shareable dossiers.
+- **Family (`/family/*`):** Protected by Traefik ForwardAuth linked to Authentik group `family` (nutrition dashboards, baby sleep charts).
+- **Private (`/private/*`):** Protected by Authentik for personal vital logs.
+
+---
+
+## Consequences
+
+### Positive
+- **Guaranteed Calculation Accuracy:** All calories, macronutrient totals, sleep statistics, and energy balances are computed in SQL. Zero LLM math errors.
+- **Git-Native & Mergable:** Text-based JSONL files partition data cleanly and produce clean 1-line git diffs.
+- **Offline & Low-Latency:** In-memory DuckDB reads months of JSONL in single-digit milliseconds.
+- **Preserved Human Readability:** SilverBullet (`wiki.moritzgraf.de`) and Obsidian continue to display updated Markdown tables generated directly from the database.
+
+### Negative & Mitigations
+- **Generated Markdown Views Must Not Be Hand-Edited:**
+  - *Mitigation*: Generated files include a prominent `<!-- DO NOT EDIT: GENERATED BY DATASTORE -->` warning comment.
+  - *Mitigation*: Git pre-commit hook and the weekly IT Guy audit verify that generated views match the underlying JSONL data.
+- **Dual-Step Agent Retrieval:** Agents execute `datastore query` for structured data, then `wiki_get` on linked documents if full narrative text is needed.
+  - *Mitigation*: Clear instructions and query helpers in the `datastore` skill prompt.
@@ -13,6 +13,7 @@ We follow the [Markdown Architectural Decision Records (MADR)](https://adr.githu
 | [**0001**](0001-hybrid-mcp-tool-search-and-subagents.md) | **Hybrid MCP Tool Search and Subagent Delegation for AI Agents** | `accepted` | 2026-09-12 | OpenClaw / AI / MCP |
 | [**0002**](0002-openclaw-model-selection-tiering-and-economics.md) | **OpenClaw Multi-Tier Model Selection, Latency Optimization, and Inference Economics** | `proposed` | 2026-09-13 | OpenClaw / AI / Models |
 | [**0003**](0003-everything-as-code-and-reproducible-infrastructure.md) | **Everything as Code (EaC) and Reproducible Declarative Infrastructure** | `accepted` | 2026-09-17 | Architecture / Monorepo / EaC |
+| [**0004**](0004-structured-personal-data-datastore.md) | **Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)** | `accepted` | 2026-10-06 | OpenClaw / Data / Storage |

 ---

@@ -7,6 +7,7 @@ spec:
  addPrefix:
    prefix: /hosting
 ---
+# Public Ingress for hosting.haumdaucher.de (Recipes, Public Assets)
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
@@ -16,6 +17,7 @@ metadata:
    kubernetes.io/ingress.class: "traefik"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    kubernetes.io/tls-acme: "true"
+    traefik.ingress.kubernetes.io/router.priority: "10"
    traefik.ingress.kubernetes.io/router.middlewares: minio-hosting-add-prefix@kubernetescrd
 spec:
  ingressClassName: traefik
@@ -34,3 +36,34 @@ spec:
  - hosts:
    - hosting.haumdaucher.de
    secretName: hosting-haumdaucher-de-tls
+---
+# Protected Ingress for /family (Family Dashboards, Nutrition & Sleep Analytics)
+# Protected by Authentik ForwardAuth
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: minio-hosting-family
+  namespace: minio
+  annotations:
+    kubernetes.io/ingress.class: "traefik"
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    kubernetes.io/tls-acme: "true"
+    traefik.ingress.kubernetes.io/router.priority: "100"
+    traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd,minio-hosting-add-prefix@kubernetescrd
+spec:
+  ingressClassName: traefik
+  rules:
+  - host: hosting.haumdaucher.de
+    http:
+      paths:
+      - path: /family
+        pathType: Prefix
+        backend:
+          service:
+            name: minio
+            port:
+              number: 9000
+  tls:
+  - hosts:
+    - hosting.haumdaucher.de
+    secretName: hosting-haumdaucher-de-tls
@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+# ==============================================================================
+# setup-garmin-cron.sh
+# Declaratively registers the Garmin Connect Nightly Sync in OpenClaw.
+# Schedule: Daily at 03:30 Europe/Berlin (cron: 30 3 * * *)
+# ==============================================================================
+
+set -euo pipefail
+
+NAMESPACE="${NAMESPACE:-openclaw}"
+POD_NAME="${POD_NAME:-openclaw-0}"
+CONTAINER_NAME="${CONTAINER_NAME:-openclaw}"
+
+JOB_NAME="garmin-nightly-mirror"
+CRON_SCHEDULE="30 3 * * *"
+TIMEZONE="Europe/Berlin"
+TELEGRAM_GROUP="-1004326921510"
+TELEGRAM_TOPIC="20"
+
+echo "=== Registering Garmin Connect Nightly Sync in OpenClaw ($POD_NAME) ==="
+
+# 1. Check if job already exists
+EXISTING_ID=$(kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list 2>/dev/null | grep "$JOB_NAME" | awk '{print $1}' || true)
+
+if [ -n "$EXISTING_ID" ]; then
+  echo "Found existing job with ID $EXISTING_ID. Removing for idempotent recreation..."
+  kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron rm "$EXISTING_ID" || true
+fi
+
+# 2. Add the job
+echo "Creating automation $JOB_NAME ($CRON_SCHEDULE @ $TIMEZONE)..."
+
+MESSAGE=$(cat << 'EOF'
+Führe den nächtlichen Garmin Connect Sync durch.
+Vorgehen:
+1. Führe den Garmin-Sync-Befehl aus:
+   uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/garmin_sync.py --days 2 --data-dir /home/node/.openclaw/wiki/main/data
+2. Aktualisiere die gerenderten Markdown-Ansichten:
+   PYTHONPATH=/home/node/mop-skills/mop-skills/skills/datastore/scripts uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/render_views.py --data-dir /home/node/.openclaw/wiki/main/data --wiki-dir /home/node/.openclaw/wiki/main
+3. Git Commit & Push im Knowledge Wiki:
+   cd /home/node/.openclaw/wiki/main && git add data/garmin/ entities/nutrition-log/ && git diff --staged --quiet || (git commit -m "chore(garmin): automated nightly mirror [skip ci]" && git push origin main)
+4. Finale Antwort:
+   - Bei Erfolg: Gib exakt 'NO_REPLY' aus (intentional silence, kein Telegram-Spam in der Nacht).
+   - Nur bei Authentifizierungsfehlern: Informiere kurz via Telegram, dass ein Garmin-Re-Auth nötig ist.
+EOF
+)
+
+kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \
+  --name "$JOB_NAME" \
+  --description "Nächtlicher deterministischer Spiegel der Garmin Connect Daten (Aktivitäten, Daily Summary, Schlaf) in Git JSONL" \
+  --cron "$CRON_SCHEDULE" \
+  --tz "$TIMEZONE" \
+  --agent "fitness" \
+  --model "openrouter/deepseek/deepseek-v4.1-flash" \
+  --session "isolated" \
+  --channel "telegram" \
+  --to "$TELEGRAM_GROUP" \
+  --thread-id "$TELEGRAM_TOPIC" \
+  --message "$MESSAGE"
+
+echo "=== Automation successfully registered! ==="
+kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list | grep "$JOB_NAME" || true