Compare commits

...
3 Commits
Author SHA1 Message Date
Moritz Graf 1ffbf3d6b7 fix(openclaw): eliminate DOMAIN_STATE HTML comment tags and add clean Telegram output guardrails 2026-10-06 19:38:47 +02:00
Moritz Graf f717a9a2f7 feat(datastore): implement structured personal data layer (ADR 0004, ForwardAuth /family, and OpenClaw protocol) 2026-10-06 18:18:27 +02:00
Moritz Graf 9d890a290f feat(skills): add marktguru supermarket offer and price comparison skill
Introduce the marktguru agent skill and MCP integration for searching supermarket deals, comparing product prices, and querying supported retail chains:
- Add Marktguru API client and MCP server script
- Add skill definition (SKILL.md) and supported supermarkets reference
- Add test suite covering Marktguru API integration and offer queries
2026-10-05 21:20:16 +02:00
18 changed files with 1616 additions and 16 deletions
+142
View File
@@ -0,0 +1,142 @@
---
name: marktguru
license: Apache-2.0
description: Query weekly supermarket offers, flyer discounts, and price comparisons across German supermarkets (ALDI Süd, ALDI Nord, Lidl, EDEKA, REWE, Kaufland, PENNY, Netto, dm, ROSSMANN, etc.) via the Marktguru API and local MCP server. Use when the user asks to check supermarket discounts, find deals in brochures/prospekte, compare grocery prices for a postal code (PLZ), or create shopping lists from current promotions without needing fragile PDF OCR.
---
# Marktguru Supermarket Offers & MCP Server
> Query live promotional flyers and circular offers from German supermarkets and drugstores without brittle PDF scraping or layout OCR.
This skill is inspired by and expands upon the Home Assistant integration pattern (`thorschtn/ha-marktguru`), leveraging Marktguru's backend API (`api.marktguru.de`) to retrieve structured deal data for any German postal code.
---
## Supported Supermarkets & Retailers
Marktguru covers **over 450 retail chains** across Germany. The primary supported chains are:
* **Discounter**:
* **ALDI SÜD** (`127153`) & **ALDI NORD** (`127139`)
* **Lidl** (`126679`)
* **PENNY** (`126765`)
* **Netto Marken-Discount** (`126735`) & **Netto (mit Hund)** (`126734`)
* **NORMA** (`126745`)
* **Supermärkte & Vollsortimenter**:
* **EDEKA** (`127093`) & **E center** (`126699`) + regionale Kaufleute (Simmel, Zurheide, etc.)
* **REWE** (`126802`) & **REWE Center** (`126804`)
* **nahkauf** (`137320`), **HIT** (`126598`), **tegut** (`126866`)
* **Großfläche / Hypermarket**:
* **Kaufland** (`126654`)
* **Globus** (`127177`)
* **Drogerien & Reform**:
* **dm-drogerie markt** (`127057`)
* **ROSSMANN** (`126811`)
* **Müller Drogeriemarkt** (`126720`)
Full details and ID mappings are documented in [`references/supermarkets.md`](references/supermarkets.md).
---
## Components
The skill provides two primary components:
1. **CLI & Python Client (`scripts/marktguru_client.py`)**: Zero external dependencies (uses standard library `urllib` and `json`).
2. **MCP Server (`scripts/marktguru_mcp.py`)**: Standard JSON-RPC 2.0 stdio server providing AI tools for search, comparison, and catalog retrieval.
---
## 1. CLI Usage
Run directly with standard `python3` (uses default postal code **`93059` Regensburg** if `--zip` is omitted):
### A. Search for Deals
```bash
# Search for butter in Regensburg (93059 default) across Aldi, Lidl, and Edeka
python3 scripts/marktguru_client.py search "Butter" --supermarket "Aldi,Lidl,Edeka"
# Search for coffee with raw JSON output
python3 scripts/marktguru_client.py search "Kaffee" --json
```
### B. Price Comparison (Sorted Ascending)
```bash
# Compare chocolate prices across all supermarkets in 93059
python3 scripts/marktguru_client.py compare "Schokolade"
# Compare beer prices between Lidl and Rewe in another postal code (e.g. 80331)
python3 scripts/marktguru_client.py compare "Bier" --zip 80331 --supermarket "Lidl,Rewe"
```
### C. List Supported Chains
```bash
python3 scripts/marktguru_client.py supermarkets
```
---
## 2. Model Context Protocol (MCP) Integration
The MCP server exposes three tools to any MCP-compatible AI agent:
* `search_supermarket_offers(query, zip_code="93059", supermarkets=None, limit=20)`
* `compare_supermarket_prices(query, zip_code="93059", supermarkets=None)`
* `list_supported_supermarkets()`
### MCP Server Configuration (`mcp_config.json`)
To enable the server in Antigravity or other MCP hosts:
```json
{
"mcpServers": {
"marktguru": {
"command": "python3",
"args": [
"/Users/moritz/src/infrapuzzle/.agents/skills/marktguru/scripts/marktguru_mcp.py"
]
}
}
}
```
---
## 3. Python Integration
You can import the module directly in Python scripts or agent sidecars:
```python
import sys
sys.path.append("/Users/moritz/src/infrapuzzle/.agents/skills/marktguru/scripts")
from marktguru_client import search_offers, compare_prices, DEFAULT_ZIP_CODE
# Search deals (defaults to 93059)
results = search_offers("Kaffee", supermarkets=["Aldi", "Lidl"])
for offer in results["offers"]:
print(f"[{offer['supermarket']}] {offer['product_name'] or offer['description']}: {offer['price']} €")
# Compare prices
comparison = compare_prices("Milch")
cheapest = comparison["cheapest_offer"]
print(f"Cheapest milk: {cheapest['price']} € at {cheapest['supermarket']}")
```
---
## Data Extracted Per Offer
Each item returned from the Marktguru API includes:
* `id`: Internal Marktguru offer ID.
* `supermarket`: Retailer name (e.g. "ALDI SÜD", "Lidl", "EDEKA").
* `product_name`: Generic product name (e.g. "Butter", "Filterkaffee").
* `brand`: Brand name (e.g. "Kerrygold", "Melitta", "Milka").
* `description`: Promotional description / pack size (e.g. "je 250g Packung").
* `price`: Promotional sale price in Euros (€).
* `old_price`: Regular price or crossed-out price (UVP).
* `discount_percent`: Calculated percentage savings (e.g. 28%).
* `reference_price`: Unit price (e.g. 5.16 € / kg).
* `loyalty_app_required`: Boolean flag indicating if a digital store app (Lidl Plus, Edeka App, Penny App) is required to receive the discount.
* `valid_from` & `valid_to`: Start and end dates of the promotion.
* `image_url`: High-resolution product image hosted on CDN.
@@ -0,0 +1,94 @@
# Unterstützte Supermärkte & Händler (Marktguru)
Über die Marktguru-Schnittstelle können Werbeprospekte, Wochenangebote und Rabattaktionen von über 450 Handelsketten und Einzelhändlern in Deutschland abgefragt werden.
---
## 1. Discounter
Discounter machen einen Großteil der wöchentlich wechselnden Prospektaktionen aus. Marktguru unterscheidet sauber zwischen den Ketten:
| Kette | Marktguru ID | Typischer Aktionsrhythmus | App-Vorteile erfasst? |
| :--- | :--- | :--- | :--- |
| **ALDI SÜD** | `127153` | Mo. & Do. / Fr. & Sa. | Ja (ALDI App Angebote) |
| **ALDI NORD** | `127139` | Mo. & Do. / Fr. & Sa. | Ja |
| **Lidl** | `126679` | Mo.–Sa. / Do.–Sa. / Fr.–Sa. | Ja (`loyalty_app_required` für Lidl Plus) |
| **PENNY** | `126765` | Mo.–Sa. / Do.–Sa. | Ja (Penny App / Payback) |
| **Netto Marken-Discount** | `126735` | Mo.–Sa. / Do.–Sa. | Ja (Netto App / DeutschlandCard) |
| **Netto (mit Hund)** | `126734` | Mo.–Sa. (primär Ost-/Norddeutschland) | Ja |
| **NORMA** | `126745` | Mo. & Mi. / Fr. & Sa. | Ja |
---
## 2. Vollsortimenter & Supermärkte
Hier gibt es oft filialgenaue oder regionale Prospekte. Die API filtert automatisch anhand der übergebenen Postleitzahl (PLZ):
| Kette | Marktguru ID | Besonderheiten |
| :--- | :--- | :--- |
| **EDEKA** | `127093` | Nationale & regionale Angebote; Marktguru aggregiert auch selbstständige Kaufleute (z.B. EDEKA Simmel, Zurheide, Niemerszein). |
| **E center** | `126699` | Großformatige EDEKA-Märkte mit erweitertem Non-Food- und Frischeangebot. |
| **REWE** | `126802` | Bundesweite Angebote sowie regionale Aktionen. |
| **REWE Center** | `126804` | Großflächenmärkte von REWE. |
| **nahkauf** | `137320` | Nachbarschaftsmärkte der REWE Group. |
| **HIT** | `126598` | Große Frische-Supermärkte (Dohle Handelsgruppe). |
| **tegut** | `126866` | Fokus auf Bio, Nachhaltigkeit und Regionales (Hessen, Bayern, Thüringen). |
---
## 3. Großfläche & SB-Warenhäuser (Hypermarkets)
| Kette | Marktguru ID | Besonderheiten |
| :--- | :--- | :--- |
| **Kaufland** | `126654` | Riesiges wöchentliches Prospektangebot (Lebensmittel + Non-Food, Kaufland Card). |
| **Globus** | `127177` | SB-Warenhäuser mit eigener Metzgerei und Bäckerei. |
---
## 4. Drogeriemärkte
Drogerieprospekte enthalten neben Pflege- und Haushaltsartikeln auch viele Bio-Lebensmittel, Snacks, Kaffee und Babynahrung:
| Kette | Marktguru ID | Typische Angebote |
| :--- | :--- | :--- |
| **dm-drogerie markt** | `127057` | dmBio, Drogeriewaren, Tiernahrung. |
| **ROSSMANN** | `126811` | EnerBio, Kaffeeangebote, wöchentliche 10%-Coupons. |
| **Müller Drogeriemarkt** | `126720` | Drogerie, Parfümerie, Spielwaren, Naturshop. |
| **budni** | `126944` | Primär im Norden (Metropolregion Hamburg). |
---
## 5. Getränkemärkte & Bio-Fachmärkte
| Kette | Marktguru ID | Sortiment |
| :--- | :--- | :--- |
| **Netto Getränke-Discount** | `134009` | Bier, Mineralwasser, Softdrinks. |
| **Denns BioMarkt / Alnatura** | Diverse | Bio-Vollsortiment, Wochenaktionen. |
---
## 6. Unterstützte Aliase im Client & MCP Server
Wenn im Tool-Aufruf der Parameter `supermarkets` übergeben wird, werden folgende umgangssprachliche Bezeichnungen automatisch auf die korrekten Retailer-IDs gemappt:
```json
{
"aldi": [127153, 127139], // Findet automatisch ALDI SÜD und ALDI NORD
"aldi süd" / "aldi sued": [127153],
"aldi nord": [127139],
"lidl": [126679],
"edeka": [127093, 126699], // EDEKA und E center
"rewe": [126802, 126804], // REWE und REWE Center
"kaufland": [126654],
"penny": [126765],
"netto": [126735, 126734], // Netto Marken-Discount und Netto (mit Hund)
"norma": [126745],
"globus": [127177],
"dm": [127057],
"rossmann": [126811],
"müller" / "mueller": [126720],
"hit": [126598],
"tegut": [126866],
"nahkauf": [137320]
}
```
+577
View File
@@ -0,0 +1,577 @@
#!/usr/bin/env python3
"""
Marktguru API Client (Production Grade)
Fetches promotional offers and discounts from German supermarkets via the Marktguru API.
Key Design Principles:
- Zero external dependencies (uses standard library only: urllib, json, os, time, datetime).
- Resilient: Retries with exponential backoff on transient errors.
- Low Maintenance: Configurable keys and postal codes via environment variables.
- Fail-Safe: Caching with stale fallback if upstream API is temporarily unreachable.
- Defensive: Robust parsing protecting against upstream schema shifts and malformed payloads.
"""
import os
import sys
import time
import json
import logging
import hashlib
import tempfile
import urllib.request
import urllib.parse
import urllib.error
from datetime import datetime, timezone
from typing import List, Dict, Any, Optional, Set, Tuple
# Logger writing to stderr to keep stdout pure for JSON / CLI
logger = logging.getLogger("marktguru_client")
if not logger.handlers:
handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s"))
logger.addHandler(handler)
logger.setLevel(logging.INFO)
API_BASE = "https://api.marktguru.de/api/v1"
# Environment variable overrides for zero-code configuration changes
DEFAULT_ZIP_CODE = os.environ.get("MARKTGURU_ZIP_CODE", "93059")
CLIENT_KEY = os.environ.get("MARKTGURU_CLIENT_KEY", "WU/RH+PMGDi+gkZer3WbMelt6zcYHSTytNB7VpTia90=")
API_KEY = os.environ.get("MARKTGURU_API_KEY", "8Kk+pmbf7TgJ9nVj2cXeA7P5zBGv8iuutVVMRfOfvNE=")
CACHE_TTL = int(os.environ.get("MARKTGURU_CACHE_TTL", "1800")) # 30 minutes
CACHE_ENABLED = os.environ.get("MARKTGURU_CACHE_ENABLED", "true").lower() in ("true", "1", "yes")
HEADERS = {
"x-clientkey": CLIENT_KEY,
"x-apikey": API_KEY,
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)",
"Accept": "application/json",
}
# Major supermarket chains and their Marktguru retailer IDs
MAJOR_RETAILERS: Dict[str, Dict[str, Any]] = {
"ALDI SÜD": {"id": 127153, "type": "Discounter", "major": True},
"ALDI NORD": {"id": 127139, "type": "Discounter", "major": True},
"Lidl": {"id": 126679, "type": "Discounter", "major": True},
"EDEKA": {"id": 127093, "type": "Supermarkt", "major": True},
"E center": {"id": 126699, "type": "Supermarkt", "major": True},
"REWE": {"id": 126802, "type": "Supermarkt", "major": True},
"REWE Center": {"id": 126804, "type": "Supermarkt", "major": True},
"Kaufland": {"id": 126654, "type": "Großfläche/Hypermarkt", "major": True},
"PENNY": {"id": 126765, "type": "Discounter", "major": True},
"Netto Marken-Discount": {"id": 126735, "type": "Discounter", "major": True},
"Netto (mit Hund)": {"id": 126734, "type": "Discounter", "major": False},
"NORMA": {"id": 126745, "type": "Discounter", "major": True},
"Globus": {"id": 127177, "type": "Großfläche/Hypermarkt", "major": True},
"HIT": {"id": 126598, "type": "Supermarkt", "major": False},
"tegut": {"id": 126866, "type": "Supermarkt", "major": False},
"nahkauf": {"id": 137320, "type": "Supermarkt", "major": False},
"dm-drogerie markt": {"id": 127057, "type": "Drogerie", "major": True},
"ROSSMANN": {"id": 126811, "type": "Drogerie", "major": True},
"Müller Drogeriemarkt": {"id": 126720, "type": "Drogerie", "major": True},
}
# Aliases mapping user query strings to Marktguru retailer IDs
RETAILER_ALIASES: Dict[str, List[int]] = {
"aldi": [127153, 127139],
"aldi süd": [127153],
"aldi sued": [127153],
"aldi nord": [127139],
"lidl": [126679],
"edeka": [127093, 126699],
"e center": [126699],
"ecenter": [126699],
"rewe": [126802, 126804],
"kaufland": [126654],
"penny": [126765],
"netto": [126735, 126734],
"norma": [126745],
"globus": [127177],
"hit": [126598],
"tegut": [126866],
"nahkauf": [137320],
"dm": [127057],
"rossmann": [126811],
"müller": [126720],
"mueller": [126720],
}
def _get_cache_dir() -> str:
"""Returns directory path for local cached responses."""
cache_dir = os.path.join(tempfile.gettempdir(), "marktguru_cache")
try:
os.makedirs(cache_dir, exist_ok=True)
except Exception:
pass
return cache_dir
def _read_cache(cache_key: str, max_age_seconds: int = CACHE_TTL) -> Tuple[Optional[Dict[str, Any]], bool]:
"""
Reads cache file. Returns (data, is_fresh).
If cache is expired, returns (data, False) so caller can use it as emergency fallback.
"""
if not CACHE_ENABLED:
return None, False
cache_file = os.path.join(_get_cache_dir(), f"{cache_key}.json")
if not os.path.isfile(cache_file):
return None, False
try:
with open(cache_file, "r", encoding="utf-8") as f:
payload = json.load(f)
cached_time = payload.get("_cached_at", 0)
age = time.time() - cached_time
data = payload.get("data")
is_fresh = age < max_age_seconds
return data, is_fresh
except Exception as e:
logger.debug(f"Failed to read cache {cache_key}: {e}")
return None, False
def _write_cache(cache_key: str, data: Dict[str, Any]) -> None:
"""Writes data to cache file atomically."""
if not CACHE_ENABLED:
return
cache_file = os.path.join(_get_cache_dir(), f"{cache_key}.json")
temp_file = f"{cache_file}.tmp.{os.getpid()}"
try:
payload = {
"_cached_at": time.time(),
"data": data,
}
with open(temp_file, "w", encoding="utf-8") as f:
json.dump(payload, f, ensure_ascii=False)
os.replace(temp_file, cache_file)
except Exception as e:
logger.debug(f"Failed to write cache {cache_key}: {e}")
if os.path.exists(temp_file):
try:
os.remove(temp_file)
except Exception:
pass
def resolve_supermarket_ids(supermarkets: Optional[List[str]]) -> Optional[str]:
"""Resolves human supermarket names/aliases to comma-separated Marktguru retailer IDs."""
if not supermarkets:
return None
# Handle string passed instead of list (e.g. "Aldi, Lidl")
if isinstance(supermarkets, str):
supermarkets = [s.strip() for s in supermarkets.split(",") if s.strip()]
resolved_ids: Set[int] = set()
for name in supermarkets:
if not isinstance(name, str):
continue
cleaned = name.strip().lower()
if not cleaned:
continue
if cleaned in RETAILER_ALIASES:
resolved_ids.update(RETAILER_ALIASES[cleaned])
elif cleaned.isdigit():
resolved_ids.add(int(cleaned))
else:
found = False
for r_name, r_info in MAJOR_RETAILERS.items():
if cleaned in r_name.lower():
resolved_ids.add(r_info["id"])
found = True
if not found:
logger.warning(f"Unknown supermarket name '{name}' skipped.")
return ",".join(str(i) for i in sorted(resolved_ids)) if resolved_ids else None
def format_iso_date(iso_str: Optional[str]) -> str:
"""Formats an ISO-8601 date string to human-readable DD.MM.YYYY format."""
if not iso_str or not isinstance(iso_str, str):
return "-"
try:
dt = datetime.fromisoformat(iso_str.replace("Z", "+00:00"))
return dt.strftime("%d.%m.%Y")
except Exception:
return iso_str[:10] if len(iso_str) >= 10 else iso_str
def parse_offer(raw: Any, zip_code: str) -> Optional[Dict[str, Any]]:
"""
Defensively parses and standardizes an offer payload from Marktguru.
Protects against null fields, malformed dates, and missing keys.
"""
if not isinstance(raw, dict):
return None
try:
offer_id = raw.get("id")
price = raw.get("price")
if isinstance(price, (int, float)):
price = round(float(price), 2)
else:
price = None
old_price = raw.get("oldPrice")
if isinstance(old_price, (int, float)):
old_price = round(float(old_price), 2)
else:
old_price = None
ref_price = raw.get("referencePrice")
if isinstance(ref_price, (int, float)):
ref_price = round(float(ref_price), 2)
else:
ref_price = None
discount_pct = None
if old_price and price and old_price > price:
discount_pct = round(((old_price - price) / old_price) * 100)
# Validity dates
validity = raw.get("validityDates")
valid_from_raw = None
valid_to_raw = None
is_valid_now = True
days_remaining = None
if isinstance(validity, list) and len(validity) > 0 and isinstance(validity[0], dict):
valid_from_raw = validity[0].get("from")
valid_to_raw = validity[0].get("to")
try:
now = datetime.now(timezone.utc)
if valid_to_raw and isinstance(valid_to_raw, str):
dt_to = datetime.fromisoformat(valid_to_raw.replace("Z", "+00:00"))
diff = (dt_to - now).days
days_remaining = max(0, diff)
if dt_to < now:
is_valid_now = False
if valid_from_raw and isinstance(valid_from_raw, str):
dt_from = datetime.fromisoformat(valid_from_raw.replace("Z", "+00:00"))
if dt_from > now:
is_valid_now = False
except Exception:
pass
# Supermarket / advertiser extraction
supermarket = "Unbekannt"
supermarket_id = None
advertisers = raw.get("advertisers")
if isinstance(advertisers, list) and len(advertisers) > 0 and isinstance(advertisers[0], dict):
supermarket = advertisers[0].get("name") or "Unbekannt"
supermarket_id = advertisers[0].get("id")
# High resolution CDN image
image_url = None
if offer_id:
image_url = f"https://mg2de.b-cdn.net/api/v1/offers/{offer_id}/images/default/0/medium.webp"
# Unit & volume
unit_info = raw.get("unit")
unit_name = None
if isinstance(unit_info, dict):
unit_name = unit_info.get("shortName") or unit_info.get("name")
product_info = raw.get("product")
product_name = None
if isinstance(product_info, dict):
product_name = product_info.get("name")
brand_info = raw.get("brand")
brand_name = None
if isinstance(brand_info, dict):
brand_name = brand_info.get("name")
return {
"id": offer_id,
"supermarket": supermarket,
"supermarket_id": supermarket_id,
"product_name": product_name,
"brand": brand_name,
"description": raw.get("description"),
"price": price,
"old_price": old_price,
"reference_price": ref_price,
"discount_percent": discount_pct,
"unit": unit_name,
"volume": raw.get("volume"),
"loyalty_app_required": bool(raw.get("requiresLoyalityMembership", False)),
"is_valid_now": is_valid_now,
"days_remaining": days_remaining,
"valid_from": format_iso_date(valid_from_raw),
"valid_to": format_iso_date(valid_to_raw),
"image_url": image_url,
"zip_code": str(zip_code),
}
except Exception as e:
logger.debug(f"Error parsing single offer {raw.get('id') if isinstance(raw, dict) else raw}: {e}")
return None
def _fetch_with_retry(url: str, max_retries: int = 2) -> Dict[str, Any]:
"""
Executes HTTP request with exponential backoff on transient errors (5xx / network blips).
"""
last_err: Optional[Exception] = None
for attempt in range(max_retries + 1):
req = urllib.request.Request(url, headers=HEADERS)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
raw_bytes = resp.read()
return json.loads(raw_bytes.decode("utf-8"))
except urllib.error.HTTPError as e:
last_err = e
# Retry only on transient server errors (500, 502, 503, 504) or rate limits (429)
if e.code in (429, 500, 502, 503, 504) and attempt < max_retries:
backoff = 0.5 * (2 ** attempt)
logger.warning(f"Marktguru HTTP {e.code}. Retrying in {backoff:.1f}s (attempt {attempt+1}/{max_retries})...")
time.sleep(backoff)
continue
raise
except (urllib.error.URLError, TimeoutError, OSError) as e:
last_err = e
if attempt < max_retries:
backoff = 0.5 * (2 ** attempt)
logger.warning(f"Connection issue ({e}). Retrying in {backoff:.1f}s...")
time.sleep(backoff)
continue
raise
raise last_err or RuntimeError("Max retries exceeded")
def search_offers(
query: str,
zip_code: str = DEFAULT_ZIP_CODE,
supermarkets: Optional[List[str]] = None,
limit: int = 24,
offset: int = 0,
force_refresh: bool = False,
) -> Dict[str, Any]:
"""
Searches current supermarket offers in Germany via the Marktguru API.
Args:
query: Search term (e.g. 'Butter', 'Kaffee', 'Bier', 'Milch')
zip_code: German postal code (default: '93059' Regensburg)
supermarkets: List of supermarket names or aliases (e.g. ['Aldi', 'Lidl', 'Edeka'])
limit: Max number of items to return (default: 24, max: 50)
offset: Result pagination offset
force_refresh: If True, bypasses local TTL cache
"""
cleaned_query = (query or "").strip()
cleaned_zip = str(zip_code or DEFAULT_ZIP_CODE).strip()
limit = max(1, min(int(limit), 50))
offset = max(0, int(offset))
retailer_ids = resolve_supermarket_ids(supermarkets)
# Compute deterministic cache key
cache_raw = f"{cleaned_query.lower()}:{cleaned_zip}:{retailer_ids or ''}:{limit}:{offset}"
cache_key = hashlib.sha256(cache_raw.encode("utf-8")).hexdigest()[:16]
if not force_refresh:
cached_data, is_fresh = _read_cache(cache_key)
if is_fresh and cached_data:
return cached_data
params: Dict[str, Any] = {
"as": "web",
"limit": limit,
"offset": offset,
"q": cleaned_query,
"zipCode": cleaned_zip,
}
if retailer_ids:
params["retailers"] = retailer_ids
url = f"{API_BASE}/offers/search?{urllib.parse.urlencode(params)}"
try:
data = _fetch_with_retry(url)
except Exception as e:
logger.error(f"Marktguru API query failed: {e}")
# Emergency fail-safe: Check if stale cache is available
cached_data, _ = _read_cache(cache_key, max_age_seconds=86400 * 7)
if cached_data:
logger.warning("Serving stale cached data due to API failure.")
cached_data["warning"] = "Stale cache returned due to temporary upstream API issue."
return cached_data
return {
"error": f"API request failed: {str(e)}",
"query": cleaned_query,
"zip_code": cleaned_zip,
"total_results": 0,
"returned_count": 0,
"offers": [],
"available_retailers": [],
}
raw_results = data.get("results") or []
parsed_offers: List[Dict[str, Any]] = []
for item in raw_results:
parsed = parse_offer(item, cleaned_zip)
if parsed is not None:
parsed_offers.append(parsed)
filters = data.get("filters") or {}
raw_retailers = filters.get("retailers") or []
available_retailers = [
{"name": r.get("name"), "id": r.get("id"), "count": r.get("resultsCount")}
for r in raw_retailers if isinstance(r, dict) and r.get("name")
]
result = {
"query": cleaned_query,
"zip_code": cleaned_zip,
"total_results": data.get("totalResults", len(parsed_offers)),
"returned_count": len(parsed_offers),
"available_retailers": available_retailers,
"offers": parsed_offers,
}
_write_cache(cache_key, result)
return result
def compare_prices(
query: str,
zip_code: str = DEFAULT_ZIP_CODE,
supermarkets: Optional[List[str]] = None,
force_refresh: bool = False,
) -> Dict[str, Any]:
"""
Compares prices for a given product query across supermarkets, sorted from cheapest to most expensive.
"""
data = search_offers(query, zip_code=zip_code, supermarkets=supermarkets, limit=50, force_refresh=force_refresh)
offers = data.get("offers", [])
# Filter out items without price and sort ascending
priced_offers = [o for o in offers if o.get("price") is not None]
priced_offers.sort(key=lambda x: x["price"])
return {
"query": query,
"zip_code": str(zip_code),
"total_compared": len(priced_offers),
"cheapest_offer": priced_offers[0] if priced_offers else None,
"all_offers_sorted": priced_offers,
}
def list_supported_supermarkets() -> Dict[str, Any]:
"""Returns a structured catalog of all major supermarket chains supported by Marktguru."""
categories: Dict[str, List[Dict[str, Any]]] = {
"Discounter": [],
"Supermarkt": [],
"Großfläche / Hypermarkt": [],
"Drogerie": [],
}
for name, info in MAJOR_RETAILERS.items():
cat = info.get("type", "Sonstige")
if cat not in categories:
categories[cat] = []
categories[cat].append({
"name": name,
"id": info["id"],
"major": info.get("major", False)
})
return {
"default_zip_code": DEFAULT_ZIP_CODE,
"cache_ttl_seconds": CACHE_TTL,
"supported_chains_count": len(MAJOR_RETAILERS),
"categories": categories,
"supported_aliases": list(RETAILER_ALIASES.keys()),
}
def main():
import argparse
parser = argparse.ArgumentParser(description="Query German supermarket flyers/discounts via Marktguru")
subparsers = parser.add_subparsers(dest="command", required=True)
# Search
search_p = subparsers.add_parser("search", help="Search offers for a product")
search_p.add_argument("query", help="Product to search (e.g. Butter, Kaffee)")
search_p.add_argument("--zip", default=DEFAULT_ZIP_CODE, help=f"Postal code (default: {DEFAULT_ZIP_CODE})")
search_p.add_argument("--supermarket", help="Comma-separated supermarkets (e.g. 'Aldi,Lidl,Edeka')")
search_p.add_argument("--limit", type=int, default=15, help="Max results (default 15)")
search_p.add_argument("--refresh", action="store_true", help="Bypass local cache")
search_p.add_argument("--json", action="store_true", help="Output raw JSON")
# Compare
comp_p = subparsers.add_parser("compare", help="Compare prices across supermarkets")
comp_p.add_argument("query", help="Product to compare")
comp_p.add_argument("--zip", default=DEFAULT_ZIP_CODE, help=f"Postal code (default: {DEFAULT_ZIP_CODE})")
comp_p.add_argument("--supermarket", help="Comma-separated supermarkets")
comp_p.add_argument("--refresh", action="store_true", help="Bypass local cache")
comp_p.add_argument("--json", action="store_true", help="Output raw JSON")
# Supermarkets
subparsers.add_parser("supermarkets", help="List all supported supermarkets and chains")
args = parser.parse_args()
if args.command == "supermarkets":
data = list_supported_supermarkets()
print(json.dumps(data, indent=2, ensure_ascii=False))
return
sm_list = [s.strip() for s in args.supermarket.split(",")] if getattr(args, "supermarket", None) else None
if args.command == "search":
result = search_offers(
args.query,
zip_code=args.zip,
supermarkets=sm_list,
limit=args.limit,
force_refresh=args.refresh
)
if args.json:
print(json.dumps(result, indent=2, ensure_ascii=False))
else:
if "error" in result and not result.get("offers"):
print(f"❌ Fehler: {result['error']}")
return
print(f"\n🔍 Angebote für '{args.query}' in PLZ {args.zip} (Gefunden: {result['total_results']}):\n")
for o in result["offers"]:
disc_str = f" (-{o['discount_percent']}%)" if o.get("discount_percent") else ""
loyalty_str = " 📱 [App Coupon]" if o.get("loyalty_app_required") else ""
timing_str = f"noch {o['days_remaining']} Tage" if o.get("days_remaining") is not None else f"bis {o['valid_to']}"
print(f" • [{o['supermarket']}] {o['description'] or o['product_name'] or 'Angebot'}")
print(f" Preis: {o['price']:.2f} €{disc_str}{loyalty_str} | Gültig: {timing_str}")
if o.get("reference_price"):
print(f" Grundpreis: {o['reference_price']:.2f} € / {o.get('unit') or 'Einheit'}")
print()
elif args.command == "compare":
result = compare_prices(
args.query,
zip_code=args.zip,
supermarkets=sm_list,
force_refresh=args.refresh
)
if args.json:
print(json.dumps(result, indent=2, ensure_ascii=False))
else:
print(f"\n🏷️ Preisvergleich für '{args.query}' in PLZ {args.zip}:\n")
for idx, o in enumerate(result["all_offers_sorted"], 1):
disc_str = f" (UVP {o['old_price']:.2f}€)" if o.get("old_price") else ""
loyalty_str = " [App erforderlich]" if o.get("loyalty_app_required") else ""
print(f" {idx:2d}. {o['price']:.2f} € - [{o['supermarket']}] {o['description'] or o['product_name']}{disc_str}{loyalty_str}")
print()
if __name__ == "__main__":
main()
+319
View File
@@ -0,0 +1,319 @@
#!/usr/bin/env python3
"""
Marktguru MCP (Model Context Protocol) Server (Production Grade)
Exposes German supermarket flyers, discounts, and price comparison tools over standard JSON-RPC 2.0 stdio.
Key Design Principles:
- Zero external dependencies: Uses standard library only.
- Strict Protocol Compliance: Conforms to MCP (2024-11-05) and JSON-RPC 2.0 specifications.
- Defensive Argument Coercion: Tolerates string/number variations commonly output by LLMs.
- Graceful Signal Handling: Clean exit on SIGINT, SIGTERM, and EOF.
- Stream Isolation: All logs and diagnostics route to stderr, ensuring stdout is 100% clean JSON-RPC.
"""
import sys
import json
import signal
import logging
from typing import Dict, Any, List, Optional
# Import business logic from hardened client script
from marktguru_client import (
search_offers,
compare_prices,
list_supported_supermarkets,
DEFAULT_ZIP_CODE,
MAJOR_RETAILERS,
RETAILER_ALIASES,
)
# Configure logging strictly to stderr
logging.basicConfig(
stream=sys.stderr,
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("marktguru-mcp")
PROTOCOL_VERSION = "2024-11-05"
SERVER_NAME = "marktguru-mcp-server"
SERVER_VERSION = "1.1.0"
TOOLS = [
{
"name": "search_supermarket_offers",
"description": (
"Search current weekly supermarket offers and promotional discounts in Germany via the Marktguru API. "
"Supports filtering by postal code (PLZ) and specific supermarket chains (e.g. Aldi Süd, Aldi Nord, Lidl, Edeka, Rewe, Kaufland, Penny, Netto)."
),
"inputSchema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The product or keyword to search for (e.g. 'Butter', 'Kaffee', 'Bier', 'Milch', 'Nutella', 'Schokolade')."
},
"zip_code": {
"type": "string",
"description": f"5-digit German postal code (PLZ). Defaults to '{DEFAULT_ZIP_CODE}' (Regensburg)."
},
"supermarkets": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of supermarket names to filter by (e.g. ['Aldi', 'Lidl', 'Edeka', 'Rewe', 'Kaufland', 'Penny', 'Netto', 'dm', 'Rossmann']). If omitted, all available retailers are included."
},
"limit": {
"type": "integer",
"description": "Maximum number of offers to return (default: 20, max: 50)."
}
},
"required": ["query"]
}
},
{
"name": "compare_supermarket_prices",
"description": (
"Compare current prices for a product across German supermarkets in a given postal code. "
"Returns results sorted from lowest to highest price, highlighting the cheapest deal, regular prices (UVP), and required loyalty app memberships."
),
"inputSchema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The product or grocery item to compare (e.g. 'Kaffee', 'Butter', 'Bier', 'Olivenöl')."
},
"zip_code": {
"type": "string",
"description": f"5-digit German postal code (PLZ). Defaults to '{DEFAULT_ZIP_CODE}' (Regensburg)."
},
"supermarkets": {
"type": "array",
"items": {"type": "string"},
"description": "Optional list of supermarket names to compare (e.g. ['Aldi', 'Lidl', 'Edeka', 'Rewe'])."
}
},
"required": ["query"]
}
},
{
"name": "list_supported_supermarkets",
"description": (
"List all supermarket, discounter, hypermarket, and drugstore chains supported by the Marktguru service in Germany, "
"including their categories (Discounter, Supermarkt, Großfläche, Drogerie) and accepted name aliases."
),
"inputSchema": {
"type": "object",
"properties": {}
}
}
]
def _sanitize_supermarkets(sm_input: Any) -> Optional[List[str]]:
"""Coerces various LLM input formats (list, comma-string, single string) into a clean list."""
if not sm_input:
return None
if isinstance(sm_input, list):
return [str(s).strip() for s in sm_input if str(s).strip()]
if isinstance(sm_input, str):
return [s.strip() for s in sm_input.split(",") if s.strip()]
return None
def handle_initialize(req_id: Any, params: Dict[str, Any]) -> Dict[str, Any]:
return {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {
"tools": {
"listChanged": False
}
},
"serverInfo": {
"name": SERVER_NAME,
"version": SERVER_VERSION
}
}
}
def handle_tools_list(req_id: Any) -> Dict[str, Any]:
return {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"tools": TOOLS
}
}
def handle_tools_call(req_id: Any, params: Dict[str, Any]) -> Dict[str, Any]:
tool_name = params.get("name")
args = params.get("arguments", {})
logger.info(f"Tool call requested: {tool_name} with args={args}")
try:
if tool_name == "search_supermarket_offers":
query = str(args.get("query") or "").strip()
if not query:
return {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"content": [{"type": "text", "text": "Error: 'query' argument is required and cannot be empty."}],
"isError": True
}
}
zip_code = str(args.get("zip_code") or DEFAULT_ZIP_CODE).strip()
supermarkets = _sanitize_supermarkets(args.get("supermarkets"))
try:
limit = int(args.get("limit") or 20)
except (ValueError, TypeError):
limit = 20
data = search_offers(query, zip_code=zip_code, supermarkets=supermarkets, limit=limit)
text_result = json.dumps(data, indent=2, ensure_ascii=False)
elif tool_name == "compare_supermarket_prices":
query = str(args.get("query") or "").strip()
if not query:
return {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"content": [{"type": "text", "text": "Error: 'query' argument is required and cannot be empty."}],
"isError": True
}
}
zip_code = str(args.get("zip_code") or DEFAULT_ZIP_CODE).strip()
supermarkets = _sanitize_supermarkets(args.get("supermarkets"))
data = compare_prices(query, zip_code=zip_code, supermarkets=supermarkets)
text_result = json.dumps(data, indent=2, ensure_ascii=False)
elif tool_name == "list_supported_supermarkets":
data = list_supported_supermarkets()
text_result = json.dumps(data, indent=2, ensure_ascii=False)
else:
return {
"jsonrpc": "2.0",
"id": req_id,
"error": {
"code": -32601,
"message": f"Method or tool '{tool_name}' not found."
}
}
return {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"content": [
{
"type": "text",
"text": text_result
}
],
"isError": False
}
}
except Exception as e:
logger.exception(f"Exception during tool execution '{tool_name}': {e}")
return {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"content": [
{
"type": "text",
"text": f"Tool execution failed: {type(e).__name__}: {str(e)}"
}
],
"isError": True
}
}
def process_message(line: str) -> Optional[Dict[str, Any]]:
if not line.strip():
return None
try:
msg = json.loads(line)
except json.JSONDecodeError as e:
logger.error(f"Malformed JSON-RPC received: {e}")
return {
"jsonrpc": "2.0",
"id": None,
"error": {"code": -32700, "message": f"Parse error: {str(e)}"}
}
method = msg.get("method")
req_id = msg.get("id")
params = msg.get("params", {})
# Notifications do not return a response
if req_id is None:
if method == "notifications/initialized":
logger.info("Client acknowledged initialized notification.")
elif method == "notifications/cancelled":
logger.info(f"Client cancelled operation: {params}")
return None
# Standard Requests
if method == "initialize":
return handle_initialize(req_id, params)
elif method == "tools/list":
return handle_tools_list(req_id)
elif method == "tools/call":
return handle_tools_call(req_id, params)
elif method == "ping":
return {"jsonrpc": "2.0", "id": req_id, "result": {}}
else:
return {
"jsonrpc": "2.0",
"id": req_id,
"error": {
"code": -32601,
"message": f"Method '{method}' is not supported."
}
}
def _sig_handler(sig, frame):
logger.info(f"Received termination signal {sig}. Exiting gracefully.")
sys.exit(0)
def main():
signal.signal(signal.SIGINT, _sig_handler)
signal.signal(signal.SIGTERM, _sig_handler)
logger.info(f"Starting {SERVER_NAME} v{SERVER_VERSION} (stdio mode)...")
try:
for raw_line in sys.stdin:
response = process_message(raw_line)
if response is not None:
output = json.dumps(response, ensure_ascii=False) + "\n"
sys.stdout.write(output)
sys.stdout.flush()
except (KeyboardInterrupt, BrokenPipeError):
pass
except Exception as e:
logger.exception(f"Fatal error in stdio loop: {e}")
sys.exit(1)
logger.info("Marktguru MCP server shutdown complete.")
if __name__ == "__main__":
main()
@@ -0,0 +1,180 @@
#!/usr/bin/env python3
"""
Unit tests for the Marktguru skill & MCP Server.
Runs completely offline using standard library unittest.
"""
import sys
import os
import json
import unittest
from datetime import datetime, timezone, timedelta
# Add scripts directory to path
SCRIPTS_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "scripts"))
if SCRIPTS_DIR not in sys.path:
sys.path.insert(0, SCRIPTS_DIR)
from marktguru_client import (
resolve_supermarket_ids,
format_iso_date,
parse_offer,
list_supported_supermarkets,
DEFAULT_ZIP_CODE,
_read_cache,
_write_cache,
_get_cache_dir,
)
from marktguru_mcp import process_message, SERVER_NAME, PROTOCOL_VERSION
class TestMarktguruClient(unittest.TestCase):
def test_default_zip_code(self):
self.assertEqual(DEFAULT_ZIP_CODE, "93059")
def test_resolve_supermarket_ids_aliases(self):
# Aldi maps to Aldi Nord and Aldi Süd
aldi_ids = resolve_supermarket_ids(["aldi"])
self.assertIn("127153", aldi_ids)
self.assertIn("127139", aldi_ids)
# Edeka maps to Edeka and E center
edeka_ids = resolve_supermarket_ids(["edeka"])
self.assertIn("127093", edeka_ids)
self.assertIn("126699", edeka_ids)
# Lidl
lidl_ids = resolve_supermarket_ids(["lidl"])
self.assertEqual(lidl_ids, "126679")
# Comma-separated string support
combined = resolve_supermarket_ids("Lidl, Kaufland")
self.assertIn("126679", combined)
self.assertIn("126654", combined)
# Unknown supermarket is ignored gracefully
unknown = resolve_supermarket_ids(["NonExistentMarket123"])
self.assertIsNone(unknown)
def test_format_iso_date(self):
self.assertEqual(format_iso_date("2026-10-15T12:00:00Z"), "15.10.2026")
self.assertEqual(format_iso_date(None), "-")
self.assertEqual(format_iso_date(""), "-")
self.assertEqual(format_iso_date("invalid-date-format"), "invalid-da")
def test_parse_offer_defensive(self):
# 1. Complete payload
now = datetime.now(timezone.utc)
from_date = (now - timedelta(days=1)).isoformat()
to_date = (now + timedelta(days=4)).isoformat()
raw_complete = {
"id": 99999,
"price": 1.49,
"oldPrice": 2.29,
"referencePrice": 5.96,
"requiresLoyalityMembership": True,
"description": "250g Packung",
"product": {"name": "Süßrahmbutter"},
"brand": {"name": "Landliebe"},
"advertisers": [{"name": "EDEKA", "id": "retailers/127093"}],
"unit": {"shortName": "kg"},
"volume": 0.25,
"validityDates": [{"from": from_date, "to": to_date}]
}
parsed = parse_offer(raw_complete, "93059")
self.assertIsNotNone(parsed)
self.assertEqual(parsed["id"], 99999)
self.assertEqual(parsed["supermarket"], "EDEKA")
self.assertEqual(parsed["price"], 1.49)
self.assertEqual(parsed["old_price"], 2.29)
self.assertEqual(parsed["discount_percent"], 35) # (2.29 - 1.49) / 2.29 = 34.9% -> 35%
self.assertTrue(parsed["loyalty_app_required"])
self.assertTrue(parsed["is_valid_now"])
self.assertGreaterEqual(parsed["days_remaining"], 3)
self.assertIn("https://mg2de.b-cdn.net/api/v1/offers/99999", parsed["image_url"])
# 2. Minimal / empty payload
raw_empty = {}
parsed_min = parse_offer(raw_empty, "93059")
self.assertIsNotNone(parsed_min)
self.assertEqual(parsed_min["supermarket"], "Unbekannt")
self.assertIsNone(parsed_min["price"])
self.assertIsNone(parsed_min["discount_percent"])
# 3. Invalid types (not a dict)
self.assertIsNone(parse_offer("not a dict", "93059"))
self.assertIsNone(parse_offer(None, "93059"))
def test_cache_mechanism(self):
test_key = "test_unit_key_123"
test_data = {"hello": "world", "timestamp": 12345}
_write_cache(test_key, test_data)
data, is_fresh = _read_cache(test_key, max_age_seconds=60)
self.assertTrue(is_fresh)
self.assertEqual(data, test_data)
# Stale cache test
data_stale, is_fresh_stale = _read_cache(test_key, max_age_seconds=-1)
self.assertFalse(is_fresh_stale)
self.assertEqual(data_stale, test_data) # still returned as fallback!
def test_list_supported_supermarkets(self):
catalog = list_supported_supermarkets()
self.assertEqual(catalog["default_zip_code"], "93059")
self.assertIn("Discounter", catalog["categories"])
self.assertIn("Supermarkt", catalog["categories"])
self.assertIn("aldi", catalog["supported_aliases"])
self.assertIn("edeka", catalog["supported_aliases"])
class TestMarktguruMCP(unittest.TestCase):
def test_mcp_initialize(self):
req = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}})
resp = process_message(req)
self.assertEqual(resp["id"], 1)
self.assertEqual(resp["result"]["protocolVersion"], PROTOCOL_VERSION)
self.assertEqual(resp["result"]["serverInfo"]["name"], SERVER_NAME)
def test_mcp_tools_list(self):
req = json.dumps({"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}})
resp = process_message(req)
tools = [t["name"] for t in resp["result"]["tools"]]
self.assertIn("search_supermarket_offers", tools)
self.assertIn("compare_supermarket_prices", tools)
self.assertIn("list_supported_supermarkets", tools)
def test_mcp_tools_call_supermarket_catalog(self):
req = json.dumps({
"jsonrpc": "2.0",
"id": 3,
"method": "tools/call",
"params": {"name": "list_supported_supermarkets", "arguments": {}}
})
resp = process_message(req)
self.assertFalse(resp["result"]["isError"])
content_text = resp["result"]["content"][0]["text"]
data = json.loads(content_text)
self.assertEqual(data["default_zip_code"], "93059")
def test_mcp_tools_call_missing_query(self):
req = json.dumps({
"jsonrpc": "2.0",
"id": 4,
"method": "tools/call",
"params": {"name": "search_supermarket_offers", "arguments": {"query": ""}}
})
resp = process_message(req)
self.assertTrue(resp["result"]["isError"])
self.assertIn("required", resp["result"]["content"][0]["text"])
def test_mcp_notification_silence(self):
req = json.dumps({"jsonrpc": "2.0", "method": "notifications/initialized"})
resp = process_message(req)
self.assertIsNone(resp) # Notifications must not produce responses
if __name__ == "__main__":
unittest.main()
@@ -139,8 +139,7 @@ Subagent specifications, system prompts, tool assignments, and domain guidelines
### 3. Multi-Turn Continuity: Domain Stickiness & Rich State Bridge ### 3. Multi-Turn Continuity: Domain Stickiness & Rich State Bridge
To prevent context loss during ongoing step-by-step dialogues (e.g. interactive cooking or workout design): To prevent context loss during ongoing step-by-step dialogues (e.g. interactive cooking or workout design):
* **Domain Stickiness**: Follow-up questions on the active topic remain within the specialized subagent domain until an explicit topic shift occurs. * **Persistent Context Continuity**: Follow-up turns are routed within the dedicated subagent topic session using native conversational memory, keeping user-facing messages clean without embedding raw JSON/HTML metadata tags.
* **Rich State Bridge (`<!-- DOMAIN_STATE -->`)**: Every subagent embeds a compact JSON state comment at the end of its response. The Conductor uses this state to answer trivial micro-checks in < 1s, or feeds it back into the subagent for complex modifications.
### 4. 3-Tier Session Lifecycle (Preventing Unbounded Session Growth) ### 4. 3-Tier Session Lifecycle (Preventing Unbounded Session Growth)
1. **Conversational Close**: Topic shifts or explicit user completion (*"Danke, das war's"*) reset the active domain pointer. 1. **Conversational Close**: Topic shifts or explicit user completion (*"Danke, das war's"*) reset the active domain pointer.
@@ -0,0 +1,141 @@
---
status: "accepted"
date: 2026-10-06
decision-makers: ["Moritz Graf", "Antigravity"]
consulted: []
informed: []
---
# ADR 0004: Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)
## Context and Problem Statement
OpenClaw manages personal logging and family assistance for the Graf family across several daily domains:
1. **Personal Nutrition:** Daily food intake, macro tracking (protein, carbs, sugar, fat, fiber, salt, calories) in `entities/nutrition-log/moritz/*.md`.
2. **Baby Sleep & Routines:** Sleep logs, naps, night sleep, bottle feeds, and developmental milestones for Mila in `entities/people/mila-schlafprotokoll.md`.
3. **Sports & Physiological Balance:** Activities, daily calorie expenditure (RMR + active), and net caloric surplus/deficit synced with Garmin Connect.
4. **Culinary Knowledge:** Recipes with ingredient proportions, cooking steps, and hardware settings.
5. **Points of Interest (POIs):** Local stores, farm stands (Hofläden), beer gardens, and playgrounds in Regensburg.
Historically, this data was logged directly into Markdown tables and bulleted lists in the Knowledge Wiki (`knowledge-wiki/`). An analysis of existing log files revealed severe systemic failure modes:
- **LLM Arithmetic Drift:** In `2026-09.md` and `2026-10.md`, the LLM consistently failed at daily macro sums (errors up to 75 kcal and 7.5 g protein per day). The derived "Vortags-Bilanz" (caloric deficit/surplus against Garmin) was calculated from incorrect totals.
- **Structural and Chronological Degradation:** In `mila-schlafprotokoll.md`, dates drifted out of order (`29.09 → Sep Summary → 30.09 → 01.10 → 03.10 → 02.10 → 28.09`), and every night sleep was duplicated (once in the evening, once the next morning).
- **Inability to Perform Spatial & Aggregate Queries:** Questions like *"What was Mila's average night sleep duration over the last 14 days?"* or *"Which beer gardens are within 500m of Grabengasse 7?"* required either multi-turn LLM hallucination or reading hundreds of lines of raw text.
How do we establish a structured, reliable, and queryable data layer for OpenClaw that guarantees mathematical precision while upholding our Everything-as-Code (ADR 0003) and Git-as-Source-of-Truth architecture?
---
## Decision Drivers
* **Mathematical Precision:** Zero LLM arithmetic. All summations, caloric balances, sleep durations, and nutrient rollups must be computed by a deterministic SQL engine.
* **Git as the Single Source of Truth (ADR 0003):** No external database silos (PostgreSQL, standalone SQLite PVCs) that can drift from the git-backed wiki. All state must remain git-versioned, diffable, and merge-friendly.
* **Separation of Concerns (Math vs. Narrative):** Delegate structured math, joins, and coordinates to DuckDB. Delegate natural-language understanding, nuance, packaging photo interpretation, and cooking advice to the LLM.
* **Flexible Composability:** Foods can become ingredients in recipes, which can yield composite foods (e.g. wheat → baking mix → bread; tomatoes → sauce → pizza).
* **Multi-Channel Presentation:** Auto-updated Markdown files for SilverBullet (`wiki.moritzgraf.de`), Telegram inline text with `chart-to-chat` PNGs, and mobile-friendly HTML dashboards on `hosting.haumdaucher.de/family/`.
* **Zero-Tax Execution:** In-memory query execution in Python via `uv` without adding heavy runtime containers or pod memory overhead.
---
## Considered Options
1. **Option 1: Stricter Markdown Prompts + Python Lint Rules**
* Keep Markdown tables; add linter checks in `okf_lint.py` to flag wrong sums.
* *Drawback*: Reactive detection rather than proactive prevention. Cannot do spatial searches or aggregate SQL queries.
2. **Option 2: External Relational Database (PostgreSQL / PVC SQLite)**
* Deploy PostgreSQL or write to a binary SQLite database on the Kubernetes PVC.
* *Drawback*: Violates ADR 0003. Creates a separate data silo outside git, introduces backup synchronization risks, and binary files cannot be cleanly merged or diffed in Git.
3. **Option 3: Pure Event Sourcing for All Data**
* Force every piece of data (including stores, recipes, and food labels) into an immutable event log.
* *Drawback*: Over-engineers static catalogs. Catalog entities (like places or recipes) become awkward to manage and query when forced into point-in-time event streams.
4. **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git (Selected)**
* Store structured records in line-delimited JSON (`.jsonl`) inside the Knowledge Wiki repository (`data/`), using in-memory DuckDB as the calculation and query engine.
* Divide data into three natural archetypes: **Events**, **Entities**, and **Documents**.
---
## Decision Outcome
Chosen Option: **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git**.
### 1. The Three Data Archetypes
```
┌────────────────────────────────────────────────────────────────────────┐
│ Three Data Archetypes │
├─────────────────────────┬─────────────────────────┬────────────────────┤
│ 1. EVENTS (Time-Series) │ 2. ENTITIES (Catalogs) │ 3. DOCUMENTS (Prose│
├─────────────────────────┼─────────────────────────┼────────────────────┤
│ • data/events/nutrition/│ • data/entities/foods/ │ • entities/recipes/│
│ • data/events/sleep/ │ • data/entities/recipes/│ • syntheses/places/│
│ • data/events/feeding/ │ • data/entities/places/ │ • sources/*.pdf │
│ • data/events/body/ │ • data/entities/people/ │ │
│ [Point & Interval Logs] │ [Durable Slug Identifiers] [Markdown / Media]│
└────────────┬────────────┴────────────┬────────────┴──────────┬─────────┘
│ │ │
▼ ▼ │
┌──────────────────────────────────────────────┐ │
│ In-Memory DuckDB SQL Engine │ │
│ • Schema Validation (STRUCT & CHECK) │ │
│ • SQL Aggregations & Daily Deficit Joins │ │
│ • Spatial Queries (ST_DWithin / Haversine) │ │
└──────────────────────┬───────────────────────┘ │
│ │
┌────────────┴────────────┐ │
▼ ▼ │
┌──────────────────┐ ┌──────────────────┐ │
│ Telegram Output │ │ Generated Views │ │
│ • Text Summaries │ │ • Markdown Views ◄────────────┘
│ • chart-to-chat │ │ • HTML Dashboards│ (Frontmatter:
└──────────────────┘ │ (/family/...) │ datastore_ref)
└──────────────────┘
```
1. **Events (Time-Series Logs):**
- Stored in `data/events/<domain>/<YYYY-MM>.jsonl`, sorted by `(start, id)`.
- Unified envelope: `id` (ULID), `kind`, `subject`, `actor`, `start`, `end` (supports open intervals where `end=NULL`), `estimated`, `meal`, `note`, `payload` (JSON cast to typed STRUCT), `source_ref`.
2. **Entities (Structured Catalogs):**
- Stored in `data/entities/<collection>.jsonl`, sorted by slug `id`.
- `foods`: EU LMIV per-100g nutritional profile, ingredient lists, barcodes, density.
- `recipes`: Ingredients list (linking `foods` with exact grams or raw text), servings, calculated macros.
- `places`: Points of interest with `lat`, `lon`, category, address, tags, and linked documents.
- `people`: Family profile registry.
3. **Documents (Prose & Media):**
- Stored as standard Markdown in `knowledge-wiki/`.
- Deterministic two-way link: Document YAML frontmatter contains `datastore_ref: "<collection>:<id>"`, while the entity JSONL record lists `documents: ["path/to/doc.md"]`.
### 2. Dedicated Raw Garmin Mirror (`data/garmin/`)
- Nightly zero-LLM Python sync script fetches raw Garmin API responses directly into:
- `data/garmin/activities/<YYYY-MM>.jsonl`
- `data/garmin/daily/<YYYY-MM>.jsonl`
- `data/garmin/sleep/<YYYY-MM>.jsonl`
- DuckDB queries raw JSON directly with `read_json_auto()` and computes the daily caloric deficit/surplus by joining nutrition intake with Garmin total daily energy expenditure.
### 3. Spatial & Geospatial Radius Lookups
- DuckDB attempts to initialize the official `spatial` extension (`LOAD spatial;`).
- If offline or binary installation fails, it transparently falls back to an internal pure-SQL Haversine macro `distance_m(lat1, lon1, lat2, lon2)`, ensuring zero-failure execution across any environment.
### 4. Privacy & Hosting Routes (`hosting.haumdaucher.de`)
- **Public (`/recipes/*`, `/places/*`, `/public/*`):** Publicly accessible recipe cards and shareable dossiers.
- **Family (`/family/*`):** Protected by Traefik ForwardAuth linked to Authentik group `family` (nutrition dashboards, baby sleep charts).
- **Private (`/private/*`):** Protected by Authentik for personal vital logs.
---
## Consequences
### Positive
- **Guaranteed Calculation Accuracy:** All calories, macronutrient totals, sleep statistics, and energy balances are computed in SQL. Zero LLM math errors.
- **Git-Native & Mergable:** Text-based JSONL files partition data cleanly and produce clean 1-line git diffs.
- **Offline & Low-Latency:** In-memory DuckDB reads months of JSONL in single-digit milliseconds.
- **Preserved Human Readability:** SilverBullet (`wiki.moritzgraf.de`) and Obsidian continue to display updated Markdown tables generated directly from the database.
### Negative & Mitigations
- **Generated Markdown Views Must Not Be Hand-Edited:**
- *Mitigation*: Generated files include a prominent `<!-- DO NOT EDIT: GENERATED BY DATASTORE -->` warning comment.
- *Mitigation*: Git pre-commit hook and the weekly IT Guy audit verify that generated views match the underlying JSONL data.
- **Dual-Step Agent Retrieval:** Agents execute `datastore query` for structured data, then `wiki_get` on linked documents if full narrative text is needed.
- *Mitigation*: Clear instructions and query helpers in the `datastore` skill prompt.
+1
View File
@@ -13,6 +13,7 @@ We follow the [Markdown Architectural Decision Records (MADR)](https://adr.githu
| [**0001**](0001-hybrid-mcp-tool-search-and-subagents.md) | **Hybrid MCP Tool Search and Subagent Delegation for AI Agents** | `accepted` | 2026-09-12 | OpenClaw / AI / MCP | | [**0001**](0001-hybrid-mcp-tool-search-and-subagents.md) | **Hybrid MCP Tool Search and Subagent Delegation for AI Agents** | `accepted` | 2026-09-12 | OpenClaw / AI / MCP |
| [**0002**](0002-openclaw-model-selection-tiering-and-economics.md) | **OpenClaw Multi-Tier Model Selection, Latency Optimization, and Inference Economics** | `proposed` | 2026-09-13 | OpenClaw / AI / Models | | [**0002**](0002-openclaw-model-selection-tiering-and-economics.md) | **OpenClaw Multi-Tier Model Selection, Latency Optimization, and Inference Economics** | `proposed` | 2026-09-13 | OpenClaw / AI / Models |
| [**0003**](0003-everything-as-code-and-reproducible-infrastructure.md) | **Everything as Code (EaC) and Reproducible Declarative Infrastructure** | `accepted` | 2026-09-17 | Architecture / Monorepo / EaC | | [**0003**](0003-everything-as-code-and-reproducible-infrastructure.md) | **Everything as Code (EaC) and Reproducible Declarative Infrastructure** | `accepted` | 2026-09-17 | Architecture / Monorepo / EaC |
| [**0004**](0004-structured-personal-data-datastore.md) | **Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)** | `accepted` | 2026-10-06 | OpenClaw / Data / Storage |
--- ---
+33
View File
@@ -7,6 +7,7 @@ spec:
addPrefix: addPrefix:
prefix: /hosting prefix: /hosting
--- ---
# Public Ingress for hosting.haumdaucher.de (Recipes, Public Assets)
apiVersion: networking.k8s.io/v1 apiVersion: networking.k8s.io/v1
kind: Ingress kind: Ingress
metadata: metadata:
@@ -16,6 +17,7 @@ metadata:
kubernetes.io/ingress.class: "traefik" kubernetes.io/ingress.class: "traefik"
cert-manager.io/cluster-issuer: "letsencrypt-prod" cert-manager.io/cluster-issuer: "letsencrypt-prod"
kubernetes.io/tls-acme: "true" kubernetes.io/tls-acme: "true"
traefik.ingress.kubernetes.io/router.priority: "10"
traefik.ingress.kubernetes.io/router.middlewares: minio-hosting-add-prefix@kubernetescrd traefik.ingress.kubernetes.io/router.middlewares: minio-hosting-add-prefix@kubernetescrd
spec: spec:
ingressClassName: traefik ingressClassName: traefik
@@ -34,3 +36,34 @@ spec:
- hosts: - hosts:
- hosting.haumdaucher.de - hosting.haumdaucher.de
secretName: hosting-haumdaucher-de-tls secretName: hosting-haumdaucher-de-tls
---
# Protected Ingress for /family (Family Dashboards, Nutrition & Sleep Analytics)
# Protected by Authentik ForwardAuth
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minio-hosting-family
namespace: minio
annotations:
kubernetes.io/ingress.class: "traefik"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
kubernetes.io/tls-acme: "true"
traefik.ingress.kubernetes.io/router.priority: "100"
traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd,minio-hosting-add-prefix@kubernetescrd
spec:
ingressClassName: traefik
rules:
- host: hosting.haumdaucher.de
http:
paths:
- path: /family
pathType: Prefix
backend:
service:
name: minio
port:
number: 9000
tls:
- hosts:
- hosting.haumdaucher.de
secretName: hosting-haumdaucher-de-tls
+1 -1
View File
@@ -40,7 +40,7 @@ The multi-container pod structure is managed natively by the operator controller
- 🏡🎬 `agents/grabengasse_media.md` (Topic 164 "Grabengasse & Media" — Home Assistant, Hue, Sonos, Brother Drucker, Jellyfin & Spotify) - 🏡🎬 `agents/grabengasse_media.md` (Topic 164 "Grabengasse & Media" — Home Assistant, Hue, Sonos, Brother Drucker, Jellyfin & Spotify)
* **Inter-Agent Peer Escalation (A2A)**: Every subagent topic is configured with peer escalation authority via `sessions_spawn(agentId="<target>", task="...")` to consult sibling agents when cross-domain expertise is needed. * **Inter-Agent Peer Escalation (A2A)**: Every subagent topic is configured with peer escalation authority via `sessions_spawn(agentId="<target>", task="...")` to consult sibling agents when cross-domain expertise is needed.
* **Modifying & Adding Agents**: To update prompt logic, tone, or tool instructions for any agent, edit its markdown file directly in `knowledge-wiki/agents/<name>.md` and commit to git. No Kubernetes pod restarts required! * **Modifying & Adding Agents**: To update prompt logic, tone, or tool instructions for any agent, edit its markdown file directly in `knowledge-wiki/agents/<name>.md` and commit to git. No Kubernetes pod restarts required!
* **Multi-Turn Domain Stickiness**: When a conversation spans multiple turns in a subagent domain (e.g. refining a recipe or planning a trip), the Conductor detects domain continuity and routes to the same agent with the state bridge `<!-- DOMAIN_STATE -->`. * **Multi-Turn Domain Stickiness**: When a conversation spans multiple turns in a subagent domain (e.g. refining a recipe or planning a trip), the Conductor detects domain continuity and routes to the same agent with clean conversational context.
--- ---
+1 -1
View File
@@ -23,7 +23,7 @@ The deployment is managed by the official **OpenClaw Operator** (`openclaw-opera
* **Multi-Agent Specialist Delegation (ADR 0001)**: * **Multi-Agent Specialist Delegation (ADR 0001)**:
* **Conductor + 9 Subagents**: Lean front-of-house coordinator orchestrates 9 specialized domain subagents defined as code in the knowledge wiki (`agents/*.md`). * **Conductor + 9 Subagents**: Lean front-of-house coordinator orchestrates 9 specialized domain subagents defined as code in the knowledge wiki (`agents/*.md`).
* **Direct Fast Path (< 2s)**: Simple status, home automation toggles, and single calendar entries run directly in-place. * **Direct Fast Path (< 2s)**: Simple status, home automation toggles, and single calendar entries run directly in-place.
* **Domain Stickiness**: Multi-turn dialogue (e.g. step-by-step cooking or vacation planning) stays pinned to the active subagent with state carried over via `<!-- DOMAIN_STATE -->`. * **Domain Stickiness**: Multi-turn dialogue (e.g. step-by-step cooking or vacation planning) stays pinned to the active subagent with persistent session continuity.
* **Active MCP Servers**: `stadtrat` (ALLRIS Net), `supermarket` (Marktguru), `geizhals` (price histories), `ebay`, `garmin` (148 health tools), `bring` (4 lists), `spotify`, `calendar` (family + city), `jelly` (Jellyfin/Jellyseerr), `homeassistant`, `prometheus`, `gitea`, `weather`. * **Active MCP Servers**: `stadtrat` (ALLRIS Net), `supermarket` (Marktguru), `geizhals` (price histories), `ebay`, `garmin` (148 health tools), `bring` (4 lists), `spotify`, `calendar` (family + city), `jelly` (Jellyfin/Jellyseerr), `homeassistant`, `prometheus`, `gitea`, `weather`.
--- ---
+37
View File
@@ -25,6 +25,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
| **2026-10-05** | **CronJob & Alerting** | `openclaw-wiki-audit` CronJob failing daily with exit code 1; Prometheus alert `WikiAuditJobFailed` firing | `RESOLVED` | Appended `\|\| true` to `okf_lint.py` in `cronjob-wiki-audit.secret.yaml`, preventing `set -e` abort when lint findings exist. | | **2026-10-05** | **CronJob & Alerting** | `openclaw-wiki-audit` CronJob failing daily with exit code 1; Prometheus alert `WikiAuditJobFailed` firing | `RESOLVED` | Appended `\|\| true` to `okf_lint.py` in `cronjob-wiki-audit.secret.yaml`, preventing `set -e` abort when lint findings exist. |
| **2026-10-05** | **Cron Trigger State** | `rennerhof-website-watch` (Regensburg Topic 116) stuck in `status: error` with `ReferenceError: state is not defined` | `RESOLVED` | Cleared stale error state from `state_json` in `openclaw.sqlite` and triggered successful verification run. | | **2026-10-05** | **Cron Trigger State** | `rennerhof-website-watch` (Regensburg Topic 116) stuck in `status: error` with `ReferenceError: state is not defined` | `RESOLVED` | Cleared stale error state from `state_json` in `openclaw.sqlite` and triggered successful verification run. |
| **2026-10-05** | **State Deadlock & Session Corruption** | Telegram Topic 117 (`#Recherche`) wedged with `restart recovery claim changed` and `entry_valid = 0` after crash | `RESOLVED` | Repaired `session_nodes` and `session_windows` (restored `entry_valid = 1`, synced `updated_at`, cleared dead claim), purged 20 obsolete cron sessions (>498k tokens), and added `--max-lines 200` compaction fallback. | | **2026-10-05** | **State Deadlock & Session Corruption** | Telegram Topic 117 (`#Recherche`) wedged with `restart recovery claim changed` and `entry_valid = 0` after crash | `RESOLVED` | Repaired `session_nodes` and `session_windows` (restored `entry_valid = 1`, synced `updated_at`, cleared dead claim), purged 20 obsolete cron sessions (>498k tokens), and added `--max-lines 200` compaction fallback. |
| **2026-10-06** | **Message Formatting & UX** | Subagents emitted raw HTML/XML comments (`<!-- DOMAIN_STATE: { ... } -->`) at the end of Telegram messages | `RESOLVED` | Removed obsolete `DOMAIN_STATE` State Bridge schema from all subagent prompts and Conductor config; replaced with clean Markdown output guardrails. |
--- ---
@@ -435,3 +436,39 @@ This document captures fixed issues, operational quirks, and incident root-cause
- **How to Avoid Recurrence**: - **How to Avoid Recurrence**:
- The nightly maintenance cron now automatically purges stale cron sessions and pre-warms cache. - The nightly maintenance cron now automatically purges stale cron sessions and pre-warms cache.
- If a session ever reports `invalid persisted session row requires repair`, check `entry_valid == 1` and ensure `row.updated_at == record.updatedAt`. - If a session ever reports `invalid persisted session row requires repair`, check `entry_valid == 1` and ensure `row.updated_at == record.updatedAt`.
---
### Issue 15: Subagents Emitted Raw HTML/XML Comments (`<!-- DOMAIN_STATE: { ... } -->`) in Telegram Messages
- **Date**: 2026-10-06
- **Severity**: Medium (Visual/UX defect: raw HTML comments and JSON leaked into user-facing chat messages)
- **Symptom**:
Messages sent by specialist subagents in Telegram forum topics (`#Mila`, `#Medizin`, `#Shopping`, `#Travel`, `#Sport`, `#Regensburg`, etc.) consistently concluded with a raw HTML/XML comment block, for example:
```html
<!-- DOMAIN_STATE: {
"agent": "doktor",
"topic_category": "symptom_check",
"evidence_source": "AWMF",
"triage_urgency": "routine"
} -->
```
- **Root Cause**:
1. **Prompt Directive**: In `knowledge-wiki/agents/*.md`, each subagent prompt contained a `## 🔄 State Bridge Schema` instructing the LLM: *„At the end of your response, output a compact state comment: `<!-- DOMAIN_STATE: { ... } -->`“*.
2. **Telegram Formatting Mismatch**: The schema was originally conceived in ADR 0001 under the assumption that `<!-- ... -->` would be invisible in chat rendering. However, Telegram's Bot API does not support hidden HTML comments. OpenClaw's text formatter safely escaped them to `&lt;!-- ... --&gt;`, causing the Telegram client to render the comments literally as visible text.
3. **Architectural Redundancy**: Subagents operate inside persistent SQLite sessions with full conversation history and vector embeddings. Furthermore, structured domain states (sleep, nutrition, feeding, workouts) were transitioned to DuckDB/JSONL via `skills/datastore` (ADR 0004). No system component or code read `DOMAIN_STATE`.
- **Resolution**:
1. Removed `## 🔄 State Bridge Schema` from all 9 subagent definition files in `knowledge-wiki/agents/*.md` (`hebamme`, `doktor`, `shopping`, `travel`, `fitness`, `regensburg`, `recherche`, `itguy`, `grabengasse_media`, and `README.md`).
2. Replaced the directive with an explicit clean-output guardrail:
```markdown
## 💬 Output Formatting & Telegram Guardrails
- Output clean Markdown only for Telegram.
- NEVER append machine-readable metadata, HTML comments (`<!-- ... -->`), XML tags, or raw JSON payloads to your chat responses.
```
3. Removed the `Pass State Bridge (<!-- DOMAIN_STATE -->)` clause from the Conductor system prompt in `k8s/openclaw/openclaw-instance.secret.yaml`.
4. Pushed updates to `knowledge-wiki` git repository, pulled latest into `/home/node/.openclaw/wiki/main`, and synchronized live workspace files in `/home/node/.openclaw/workspace-*/AGENTS.md`.
5. Applied updated `openclaw-instance.secret.yaml` to Kubernetes.
- **How to Avoid Recurrence**:
- Never instruct LLMs to embed machine-readable metadata or pseudo-comments (`<!-- ... -->`) in user-facing messenger outputs.
- Rely on persistent session context and structured stores (`skills/datastore`) for state retention.
Binary file not shown.
+21 -6
View File
@@ -63,6 +63,7 @@ def prune_stale_cron_sessions():
agent = db.split("/")[5] agent = db.split("/")[5]
try: try:
conn = sqlite3.connect(db, timeout=30.0) conn = sqlite3.connect(db, timeout=30.0)
conn.execute("PRAGMA foreign_keys = ON;")
cursor = conn.cursor() cursor = conn.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type=\"table\"") cursor.execute("SELECT name FROM sqlite_master WHERE type=\"table\"")
all_tables = set(r[0] for r in cursor.fetchall()) all_tables = set(r[0] for r in cursor.fetchall())
@@ -82,7 +83,7 @@ def prune_stale_cron_sessions():
if stale: if stale:
tokens_reclaimed = sum(t[2] for t in stale) tokens_reclaimed = sum(t[2] for t in stale)
for k, sid, _ in stale: for k, sid, _ in stale:
for t in ["transcript_events", "transcript_event_identities", "session_transcript_active_events", "transcript_rewrite_watermarks", "session_transcript_index_state", "session_transcript_fts", "session_conversations"]: for t in ["trajectory_runtime_events", "transcript_events", "transcript_event_identities", "session_transcript_active_events", "transcript_rewrite_watermarks", "session_transcript_index_state", "session_transcript_fts", "session_conversations"]:
if t in all_tables and sid: if t in all_tables and sid:
cursor.execute(f"DELETE FROM {t} WHERE session_id = ?", (sid,)) cursor.execute(f"DELETE FROM {t} WHERE session_id = ?", (sid,))
for t in ["session_participants", "heartbeat_outcomes", "session_progress_cards", "board_widgets", "board_tabs"]: for t in ["session_participants", "heartbeat_outcomes", "session_progress_cards", "board_widgets", "board_tabs"]:
@@ -90,9 +91,15 @@ def prune_stale_cron_sessions():
cursor.execute(f"DELETE FROM {t} WHERE session_key = ?", (k,)) cursor.execute(f"DELETE FROM {t} WHERE session_key = ?", (k,))
if "memory_entry_origins" in all_tables: if "memory_entry_origins" in all_tables:
cursor.execute("DELETE FROM memory_entry_origins WHERE session_key = ? OR session_id = ?", (k, sid or "")) cursor.execute("DELETE FROM memory_entry_origins WHERE session_key = ? OR session_id = ?", (k, sid or ""))
if "session_windows" in all_tables:
cursor.execute("DELETE FROM session_windows WHERE session_key = ? OR session_id = ?", (k, sid or ""))
cursor.execute("DELETE FROM session_nodes WHERE session_key = ?", (k,)) cursor.execute("DELETE FROM session_nodes WHERE session_key = ?", (k,))
conn.commit()
total_pruned += len(stale)
total_tokens_reclaimed += tokens_reclaimed
print(f" [{agent}] Pruned {len(stale)} dead cron sessions ({tokens_reclaimed} tokens reclaimed)")
# Resolve any dangling foreign key references # Unconditionally resolve any dangling foreign key references across all tables
while True: while True:
cursor.execute("PRAGMA foreign_key_check;") cursor.execute("PRAGMA foreign_key_check;")
violations = cursor.fetchall() violations = cursor.fetchall()
@@ -100,15 +107,23 @@ def prune_stale_cron_sessions():
break break
for table, rowid, parent, fkid in violations: for table, rowid, parent, fkid in violations:
cursor.execute(f"DELETE FROM {table} WHERE rowid = ?", (rowid,)) cursor.execute(f"DELETE FROM {table} WHERE rowid = ?", (rowid,))
conn.commit() conn.commit()
total_pruned += len(stale)
total_tokens_reclaimed += tokens_reclaimed
print(f" [{agent}] Pruned {len(stale)} dead cron sessions ({tokens_reclaimed} tokens reclaimed)")
conn.close() conn.close()
except Exception as e: except Exception as e:
print(f" [{agent}] Pruning error: {e}") print(f" [{agent}] Pruning error: {e}")
# Clear resolved quarantine entries if all databases are healthy
try:
q_path = "/home/node/.openclaw/state/openclaw-quarantine.sqlite"
if os.path.exists(q_path):
q_conn = sqlite3.connect(q_path)
q_cur = q_conn.cursor()
q_cur.execute("DELETE FROM quarantined_databases")
q_conn.commit()
q_conn.close()
except Exception as e:
print(f"Quarantine cleanup error: {e}")
print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.") print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.")
def compact_active_sessions(): def compact_active_sessions():
+62
View File
@@ -0,0 +1,62 @@
#!/usr/bin/env bash
# ==============================================================================
# setup-garmin-cron.sh
# Declaratively registers the Garmin Connect Nightly Sync in OpenClaw.
# Schedule: Daily at 03:30 Europe/Berlin (cron: 30 3 * * *)
# ==============================================================================
set -euo pipefail
NAMESPACE="${NAMESPACE:-openclaw}"
POD_NAME="${POD_NAME:-openclaw-0}"
CONTAINER_NAME="${CONTAINER_NAME:-openclaw}"
JOB_NAME="garmin-nightly-mirror"
CRON_SCHEDULE="30 3 * * *"
TIMEZONE="Europe/Berlin"
TELEGRAM_GROUP="-1004326921510"
TELEGRAM_TOPIC="20"
echo "=== Registering Garmin Connect Nightly Sync in OpenClaw ($POD_NAME) ==="
# 1. Check if job already exists
EXISTING_ID=$(kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list 2>/dev/null | grep "$JOB_NAME" | awk '{print $1}' || true)
if [ -n "$EXISTING_ID" ]; then
echo "Found existing job with ID $EXISTING_ID. Removing for idempotent recreation..."
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron rm "$EXISTING_ID" || true
fi
# 2. Add the job
echo "Creating automation $JOB_NAME ($CRON_SCHEDULE @ $TIMEZONE)..."
MESSAGE=$(cat << 'EOF'
Führe den nächtlichen Garmin Connect Sync durch.
Vorgehen:
1. Führe den Garmin-Sync-Befehl aus:
uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/garmin_sync.py --days 2 --data-dir /home/node/.openclaw/wiki/main/data
2. Aktualisiere die gerenderten Markdown-Ansichten:
PYTHONPATH=/home/node/mop-skills/mop-skills/skills/datastore/scripts uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/render_views.py --data-dir /home/node/.openclaw/wiki/main/data --wiki-dir /home/node/.openclaw/wiki/main
3. Git Commit & Push im Knowledge Wiki:
cd /home/node/.openclaw/wiki/main && git add data/garmin/ entities/nutrition-log/ && git diff --staged --quiet || (git commit -m "chore(garmin): automated nightly mirror [skip ci]" && git push origin main)
4. Finale Antwort:
- Bei Erfolg: Gib exakt 'NO_REPLY' aus (intentional silence, kein Telegram-Spam in der Nacht).
- Nur bei Authentifizierungsfehlern: Informiere kurz via Telegram, dass ein Garmin-Re-Auth nötig ist.
EOF
)
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \
--name "$JOB_NAME" \
--description "Nächtlicher deterministischer Spiegel der Garmin Connect Daten (Aktivitäten, Daily Summary, Schlaf) in Git JSONL" \
--cron "$CRON_SCHEDULE" \
--tz "$TIMEZONE" \
--agent "fitness" \
--model "openrouter/deepseek/deepseek-v4.1-flash" \
--session "isolated" \
--channel "telegram" \
--to "$TELEGRAM_GROUP" \
--thread-id "$TELEGRAM_TOPIC" \
--message "$MESSAGE"
echo "=== Automation successfully registered! ==="
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list | grep "$JOB_NAME" || true