Compare commits
3
Commits
c3c7a4c8cf
...
1ffbf3d6b7
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1ffbf3d6b7 | ||
|
|
f717a9a2f7 | ||
|
|
9d890a290f |
@@ -0,0 +1,142 @@
|
||||
---
|
||||
name: marktguru
|
||||
license: Apache-2.0
|
||||
description: Query weekly supermarket offers, flyer discounts, and price comparisons across German supermarkets (ALDI Süd, ALDI Nord, Lidl, EDEKA, REWE, Kaufland, PENNY, Netto, dm, ROSSMANN, etc.) via the Marktguru API and local MCP server. Use when the user asks to check supermarket discounts, find deals in brochures/prospekte, compare grocery prices for a postal code (PLZ), or create shopping lists from current promotions without needing fragile PDF OCR.
|
||||
---
|
||||
|
||||
# Marktguru Supermarket Offers & MCP Server
|
||||
|
||||
> Query live promotional flyers and circular offers from German supermarkets and drugstores without brittle PDF scraping or layout OCR.
|
||||
|
||||
This skill is inspired by and expands upon the Home Assistant integration pattern (`thorschtn/ha-marktguru`), leveraging Marktguru's backend API (`api.marktguru.de`) to retrieve structured deal data for any German postal code.
|
||||
|
||||
---
|
||||
|
||||
## Supported Supermarkets & Retailers
|
||||
|
||||
Marktguru covers **over 450 retail chains** across Germany. The primary supported chains are:
|
||||
|
||||
* **Discounter**:
|
||||
* **ALDI SÜD** (`127153`) & **ALDI NORD** (`127139`)
|
||||
* **Lidl** (`126679`)
|
||||
* **PENNY** (`126765`)
|
||||
* **Netto Marken-Discount** (`126735`) & **Netto (mit Hund)** (`126734`)
|
||||
* **NORMA** (`126745`)
|
||||
* **Supermärkte & Vollsortimenter**:
|
||||
* **EDEKA** (`127093`) & **E center** (`126699`) + regionale Kaufleute (Simmel, Zurheide, etc.)
|
||||
* **REWE** (`126802`) & **REWE Center** (`126804`)
|
||||
* **nahkauf** (`137320`), **HIT** (`126598`), **tegut** (`126866`)
|
||||
* **Großfläche / Hypermarket**:
|
||||
* **Kaufland** (`126654`)
|
||||
* **Globus** (`127177`)
|
||||
* **Drogerien & Reform**:
|
||||
* **dm-drogerie markt** (`127057`)
|
||||
* **ROSSMANN** (`126811`)
|
||||
* **Müller Drogeriemarkt** (`126720`)
|
||||
|
||||
Full details and ID mappings are documented in [`references/supermarkets.md`](references/supermarkets.md).
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
The skill provides two primary components:
|
||||
|
||||
1. **CLI & Python Client (`scripts/marktguru_client.py`)**: Zero external dependencies (uses standard library `urllib` and `json`).
|
||||
2. **MCP Server (`scripts/marktguru_mcp.py`)**: Standard JSON-RPC 2.0 stdio server providing AI tools for search, comparison, and catalog retrieval.
|
||||
|
||||
---
|
||||
|
||||
## 1. CLI Usage
|
||||
|
||||
Run directly with standard `python3` (uses default postal code **`93059` Regensburg** if `--zip` is omitted):
|
||||
|
||||
### A. Search for Deals
|
||||
```bash
|
||||
# Search for butter in Regensburg (93059 default) across Aldi, Lidl, and Edeka
|
||||
python3 scripts/marktguru_client.py search "Butter" --supermarket "Aldi,Lidl,Edeka"
|
||||
|
||||
# Search for coffee with raw JSON output
|
||||
python3 scripts/marktguru_client.py search "Kaffee" --json
|
||||
```
|
||||
|
||||
### B. Price Comparison (Sorted Ascending)
|
||||
```bash
|
||||
# Compare chocolate prices across all supermarkets in 93059
|
||||
python3 scripts/marktguru_client.py compare "Schokolade"
|
||||
|
||||
# Compare beer prices between Lidl and Rewe in another postal code (e.g. 80331)
|
||||
python3 scripts/marktguru_client.py compare "Bier" --zip 80331 --supermarket "Lidl,Rewe"
|
||||
```
|
||||
|
||||
### C. List Supported Chains
|
||||
```bash
|
||||
python3 scripts/marktguru_client.py supermarkets
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Model Context Protocol (MCP) Integration
|
||||
|
||||
The MCP server exposes three tools to any MCP-compatible AI agent:
|
||||
|
||||
* `search_supermarket_offers(query, zip_code="93059", supermarkets=None, limit=20)`
|
||||
* `compare_supermarket_prices(query, zip_code="93059", supermarkets=None)`
|
||||
* `list_supported_supermarkets()`
|
||||
|
||||
### MCP Server Configuration (`mcp_config.json`)
|
||||
|
||||
To enable the server in Antigravity or other MCP hosts:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"marktguru": {
|
||||
"command": "python3",
|
||||
"args": [
|
||||
"/Users/moritz/src/infrapuzzle/.agents/skills/marktguru/scripts/marktguru_mcp.py"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Python Integration
|
||||
|
||||
You can import the module directly in Python scripts or agent sidecars:
|
||||
|
||||
```python
|
||||
import sys
|
||||
sys.path.append("/Users/moritz/src/infrapuzzle/.agents/skills/marktguru/scripts")
|
||||
from marktguru_client import search_offers, compare_prices, DEFAULT_ZIP_CODE
|
||||
|
||||
# Search deals (defaults to 93059)
|
||||
results = search_offers("Kaffee", supermarkets=["Aldi", "Lidl"])
|
||||
for offer in results["offers"]:
|
||||
print(f"[{offer['supermarket']}] {offer['product_name'] or offer['description']}: {offer['price']} €")
|
||||
|
||||
# Compare prices
|
||||
comparison = compare_prices("Milch")
|
||||
cheapest = comparison["cheapest_offer"]
|
||||
print(f"Cheapest milk: {cheapest['price']} € at {cheapest['supermarket']}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Extracted Per Offer
|
||||
|
||||
Each item returned from the Marktguru API includes:
|
||||
* `id`: Internal Marktguru offer ID.
|
||||
* `supermarket`: Retailer name (e.g. "ALDI SÜD", "Lidl", "EDEKA").
|
||||
* `product_name`: Generic product name (e.g. "Butter", "Filterkaffee").
|
||||
* `brand`: Brand name (e.g. "Kerrygold", "Melitta", "Milka").
|
||||
* `description`: Promotional description / pack size (e.g. "je 250g Packung").
|
||||
* `price`: Promotional sale price in Euros (€).
|
||||
* `old_price`: Regular price or crossed-out price (UVP).
|
||||
* `discount_percent`: Calculated percentage savings (e.g. 28%).
|
||||
* `reference_price`: Unit price (e.g. 5.16 € / kg).
|
||||
* `loyalty_app_required`: Boolean flag indicating if a digital store app (Lidl Plus, Edeka App, Penny App) is required to receive the discount.
|
||||
* `valid_from` & `valid_to`: Start and end dates of the promotion.
|
||||
* `image_url`: High-resolution product image hosted on CDN.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Unterstützte Supermärkte & Händler (Marktguru)
|
||||
|
||||
Über die Marktguru-Schnittstelle können Werbeprospekte, Wochenangebote und Rabattaktionen von über 450 Handelsketten und Einzelhändlern in Deutschland abgefragt werden.
|
||||
|
||||
---
|
||||
|
||||
## 1. Discounter
|
||||
|
||||
Discounter machen einen Großteil der wöchentlich wechselnden Prospektaktionen aus. Marktguru unterscheidet sauber zwischen den Ketten:
|
||||
|
||||
| Kette | Marktguru ID | Typischer Aktionsrhythmus | App-Vorteile erfasst? |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **ALDI SÜD** | `127153` | Mo. & Do. / Fr. & Sa. | Ja (ALDI App Angebote) |
|
||||
| **ALDI NORD** | `127139` | Mo. & Do. / Fr. & Sa. | Ja |
|
||||
| **Lidl** | `126679` | Mo.–Sa. / Do.–Sa. / Fr.–Sa. | Ja (`loyalty_app_required` für Lidl Plus) |
|
||||
| **PENNY** | `126765` | Mo.–Sa. / Do.–Sa. | Ja (Penny App / Payback) |
|
||||
| **Netto Marken-Discount** | `126735` | Mo.–Sa. / Do.–Sa. | Ja (Netto App / DeutschlandCard) |
|
||||
| **Netto (mit Hund)** | `126734` | Mo.–Sa. (primär Ost-/Norddeutschland) | Ja |
|
||||
| **NORMA** | `126745` | Mo. & Mi. / Fr. & Sa. | Ja |
|
||||
|
||||
---
|
||||
|
||||
## 2. Vollsortimenter & Supermärkte
|
||||
|
||||
Hier gibt es oft filialgenaue oder regionale Prospekte. Die API filtert automatisch anhand der übergebenen Postleitzahl (PLZ):
|
||||
|
||||
| Kette | Marktguru ID | Besonderheiten |
|
||||
| :--- | :--- | :--- |
|
||||
| **EDEKA** | `127093` | Nationale & regionale Angebote; Marktguru aggregiert auch selbstständige Kaufleute (z.B. EDEKA Simmel, Zurheide, Niemerszein). |
|
||||
| **E center** | `126699` | Großformatige EDEKA-Märkte mit erweitertem Non-Food- und Frischeangebot. |
|
||||
| **REWE** | `126802` | Bundesweite Angebote sowie regionale Aktionen. |
|
||||
| **REWE Center** | `126804` | Großflächenmärkte von REWE. |
|
||||
| **nahkauf** | `137320` | Nachbarschaftsmärkte der REWE Group. |
|
||||
| **HIT** | `126598` | Große Frische-Supermärkte (Dohle Handelsgruppe). |
|
||||
| **tegut** | `126866` | Fokus auf Bio, Nachhaltigkeit und Regionales (Hessen, Bayern, Thüringen). |
|
||||
|
||||
---
|
||||
|
||||
## 3. Großfläche & SB-Warenhäuser (Hypermarkets)
|
||||
|
||||
| Kette | Marktguru ID | Besonderheiten |
|
||||
| :--- | :--- | :--- |
|
||||
| **Kaufland** | `126654` | Riesiges wöchentliches Prospektangebot (Lebensmittel + Non-Food, Kaufland Card). |
|
||||
| **Globus** | `127177` | SB-Warenhäuser mit eigener Metzgerei und Bäckerei. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Drogeriemärkte
|
||||
|
||||
Drogerieprospekte enthalten neben Pflege- und Haushaltsartikeln auch viele Bio-Lebensmittel, Snacks, Kaffee und Babynahrung:
|
||||
|
||||
| Kette | Marktguru ID | Typische Angebote |
|
||||
| :--- | :--- | :--- |
|
||||
| **dm-drogerie markt** | `127057` | dmBio, Drogeriewaren, Tiernahrung. |
|
||||
| **ROSSMANN** | `126811` | EnerBio, Kaffeeangebote, wöchentliche 10%-Coupons. |
|
||||
| **Müller Drogeriemarkt** | `126720` | Drogerie, Parfümerie, Spielwaren, Naturshop. |
|
||||
| **budni** | `126944` | Primär im Norden (Metropolregion Hamburg). |
|
||||
|
||||
---
|
||||
|
||||
## 5. Getränkemärkte & Bio-Fachmärkte
|
||||
|
||||
| Kette | Marktguru ID | Sortiment |
|
||||
| :--- | :--- | :--- |
|
||||
| **Netto Getränke-Discount** | `134009` | Bier, Mineralwasser, Softdrinks. |
|
||||
| **Denns BioMarkt / Alnatura** | Diverse | Bio-Vollsortiment, Wochenaktionen. |
|
||||
|
||||
---
|
||||
|
||||
## 6. Unterstützte Aliase im Client & MCP Server
|
||||
|
||||
Wenn im Tool-Aufruf der Parameter `supermarkets` übergeben wird, werden folgende umgangssprachliche Bezeichnungen automatisch auf die korrekten Retailer-IDs gemappt:
|
||||
|
||||
```json
|
||||
{
|
||||
"aldi": [127153, 127139], // Findet automatisch ALDI SÜD und ALDI NORD
|
||||
"aldi süd" / "aldi sued": [127153],
|
||||
"aldi nord": [127139],
|
||||
"lidl": [126679],
|
||||
"edeka": [127093, 126699], // EDEKA und E center
|
||||
"rewe": [126802, 126804], // REWE und REWE Center
|
||||
"kaufland": [126654],
|
||||
"penny": [126765],
|
||||
"netto": [126735, 126734], // Netto Marken-Discount und Netto (mit Hund)
|
||||
"norma": [126745],
|
||||
"globus": [127177],
|
||||
"dm": [127057],
|
||||
"rossmann": [126811],
|
||||
"müller" / "mueller": [126720],
|
||||
"hit": [126598],
|
||||
"tegut": [126866],
|
||||
"nahkauf": [137320]
|
||||
}
|
||||
```
|
||||
Binary file not shown.
Binary file not shown.
+577
@@ -0,0 +1,577 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Marktguru API Client (Production Grade)
|
||||
Fetches promotional offers and discounts from German supermarkets via the Marktguru API.
|
||||
|
||||
Key Design Principles:
|
||||
- Zero external dependencies (uses standard library only: urllib, json, os, time, datetime).
|
||||
- Resilient: Retries with exponential backoff on transient errors.
|
||||
- Low Maintenance: Configurable keys and postal codes via environment variables.
|
||||
- Fail-Safe: Caching with stale fallback if upstream API is temporarily unreachable.
|
||||
- Defensive: Robust parsing protecting against upstream schema shifts and malformed payloads.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import json
|
||||
import logging
|
||||
import hashlib
|
||||
import tempfile
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
import urllib.error
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Dict, Any, Optional, Set, Tuple
|
||||
|
||||
# Logger writing to stderr to keep stdout pure for JSON / CLI
|
||||
logger = logging.getLogger("marktguru_client")
|
||||
if not logger.handlers:
|
||||
handler = logging.StreamHandler(sys.stderr)
|
||||
handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s"))
|
||||
logger.addHandler(handler)
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
API_BASE = "https://api.marktguru.de/api/v1"
|
||||
|
||||
# Environment variable overrides for zero-code configuration changes
|
||||
DEFAULT_ZIP_CODE = os.environ.get("MARKTGURU_ZIP_CODE", "93059")
|
||||
CLIENT_KEY = os.environ.get("MARKTGURU_CLIENT_KEY", "WU/RH+PMGDi+gkZer3WbMelt6zcYHSTytNB7VpTia90=")
|
||||
API_KEY = os.environ.get("MARKTGURU_API_KEY", "8Kk+pmbf7TgJ9nVj2cXeA7P5zBGv8iuutVVMRfOfvNE=")
|
||||
CACHE_TTL = int(os.environ.get("MARKTGURU_CACHE_TTL", "1800")) # 30 minutes
|
||||
CACHE_ENABLED = os.environ.get("MARKTGURU_CACHE_ENABLED", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
HEADERS = {
|
||||
"x-clientkey": CLIENT_KEY,
|
||||
"x-apikey": API_KEY,
|
||||
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
# Major supermarket chains and their Marktguru retailer IDs
|
||||
MAJOR_RETAILERS: Dict[str, Dict[str, Any]] = {
|
||||
"ALDI SÜD": {"id": 127153, "type": "Discounter", "major": True},
|
||||
"ALDI NORD": {"id": 127139, "type": "Discounter", "major": True},
|
||||
"Lidl": {"id": 126679, "type": "Discounter", "major": True},
|
||||
"EDEKA": {"id": 127093, "type": "Supermarkt", "major": True},
|
||||
"E center": {"id": 126699, "type": "Supermarkt", "major": True},
|
||||
"REWE": {"id": 126802, "type": "Supermarkt", "major": True},
|
||||
"REWE Center": {"id": 126804, "type": "Supermarkt", "major": True},
|
||||
"Kaufland": {"id": 126654, "type": "Großfläche/Hypermarkt", "major": True},
|
||||
"PENNY": {"id": 126765, "type": "Discounter", "major": True},
|
||||
"Netto Marken-Discount": {"id": 126735, "type": "Discounter", "major": True},
|
||||
"Netto (mit Hund)": {"id": 126734, "type": "Discounter", "major": False},
|
||||
"NORMA": {"id": 126745, "type": "Discounter", "major": True},
|
||||
"Globus": {"id": 127177, "type": "Großfläche/Hypermarkt", "major": True},
|
||||
"HIT": {"id": 126598, "type": "Supermarkt", "major": False},
|
||||
"tegut": {"id": 126866, "type": "Supermarkt", "major": False},
|
||||
"nahkauf": {"id": 137320, "type": "Supermarkt", "major": False},
|
||||
"dm-drogerie markt": {"id": 127057, "type": "Drogerie", "major": True},
|
||||
"ROSSMANN": {"id": 126811, "type": "Drogerie", "major": True},
|
||||
"Müller Drogeriemarkt": {"id": 126720, "type": "Drogerie", "major": True},
|
||||
}
|
||||
|
||||
# Aliases mapping user query strings to Marktguru retailer IDs
|
||||
RETAILER_ALIASES: Dict[str, List[int]] = {
|
||||
"aldi": [127153, 127139],
|
||||
"aldi süd": [127153],
|
||||
"aldi sued": [127153],
|
||||
"aldi nord": [127139],
|
||||
"lidl": [126679],
|
||||
"edeka": [127093, 126699],
|
||||
"e center": [126699],
|
||||
"ecenter": [126699],
|
||||
"rewe": [126802, 126804],
|
||||
"kaufland": [126654],
|
||||
"penny": [126765],
|
||||
"netto": [126735, 126734],
|
||||
"norma": [126745],
|
||||
"globus": [127177],
|
||||
"hit": [126598],
|
||||
"tegut": [126866],
|
||||
"nahkauf": [137320],
|
||||
"dm": [127057],
|
||||
"rossmann": [126811],
|
||||
"müller": [126720],
|
||||
"mueller": [126720],
|
||||
}
|
||||
|
||||
|
||||
def _get_cache_dir() -> str:
|
||||
"""Returns directory path for local cached responses."""
|
||||
cache_dir = os.path.join(tempfile.gettempdir(), "marktguru_cache")
|
||||
try:
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
except Exception:
|
||||
pass
|
||||
return cache_dir
|
||||
|
||||
|
||||
def _read_cache(cache_key: str, max_age_seconds: int = CACHE_TTL) -> Tuple[Optional[Dict[str, Any]], bool]:
|
||||
"""
|
||||
Reads cache file. Returns (data, is_fresh).
|
||||
If cache is expired, returns (data, False) so caller can use it as emergency fallback.
|
||||
"""
|
||||
if not CACHE_ENABLED:
|
||||
return None, False
|
||||
|
||||
cache_file = os.path.join(_get_cache_dir(), f"{cache_key}.json")
|
||||
if not os.path.isfile(cache_file):
|
||||
return None, False
|
||||
|
||||
try:
|
||||
with open(cache_file, "r", encoding="utf-8") as f:
|
||||
payload = json.load(f)
|
||||
cached_time = payload.get("_cached_at", 0)
|
||||
age = time.time() - cached_time
|
||||
data = payload.get("data")
|
||||
is_fresh = age < max_age_seconds
|
||||
return data, is_fresh
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to read cache {cache_key}: {e}")
|
||||
return None, False
|
||||
|
||||
|
||||
def _write_cache(cache_key: str, data: Dict[str, Any]) -> None:
|
||||
"""Writes data to cache file atomically."""
|
||||
if not CACHE_ENABLED:
|
||||
return
|
||||
|
||||
cache_file = os.path.join(_get_cache_dir(), f"{cache_key}.json")
|
||||
temp_file = f"{cache_file}.tmp.{os.getpid()}"
|
||||
try:
|
||||
payload = {
|
||||
"_cached_at": time.time(),
|
||||
"data": data,
|
||||
}
|
||||
with open(temp_file, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False)
|
||||
os.replace(temp_file, cache_file)
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to write cache {cache_key}: {e}")
|
||||
if os.path.exists(temp_file):
|
||||
try:
|
||||
os.remove(temp_file)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def resolve_supermarket_ids(supermarkets: Optional[List[str]]) -> Optional[str]:
|
||||
"""Resolves human supermarket names/aliases to comma-separated Marktguru retailer IDs."""
|
||||
if not supermarkets:
|
||||
return None
|
||||
|
||||
# Handle string passed instead of list (e.g. "Aldi, Lidl")
|
||||
if isinstance(supermarkets, str):
|
||||
supermarkets = [s.strip() for s in supermarkets.split(",") if s.strip()]
|
||||
|
||||
resolved_ids: Set[int] = set()
|
||||
for name in supermarkets:
|
||||
if not isinstance(name, str):
|
||||
continue
|
||||
cleaned = name.strip().lower()
|
||||
if not cleaned:
|
||||
continue
|
||||
|
||||
if cleaned in RETAILER_ALIASES:
|
||||
resolved_ids.update(RETAILER_ALIASES[cleaned])
|
||||
elif cleaned.isdigit():
|
||||
resolved_ids.add(int(cleaned))
|
||||
else:
|
||||
found = False
|
||||
for r_name, r_info in MAJOR_RETAILERS.items():
|
||||
if cleaned in r_name.lower():
|
||||
resolved_ids.add(r_info["id"])
|
||||
found = True
|
||||
if not found:
|
||||
logger.warning(f"Unknown supermarket name '{name}' skipped.")
|
||||
|
||||
return ",".join(str(i) for i in sorted(resolved_ids)) if resolved_ids else None
|
||||
|
||||
|
||||
def format_iso_date(iso_str: Optional[str]) -> str:
|
||||
"""Formats an ISO-8601 date string to human-readable DD.MM.YYYY format."""
|
||||
if not iso_str or not isinstance(iso_str, str):
|
||||
return "-"
|
||||
try:
|
||||
dt = datetime.fromisoformat(iso_str.replace("Z", "+00:00"))
|
||||
return dt.strftime("%d.%m.%Y")
|
||||
except Exception:
|
||||
return iso_str[:10] if len(iso_str) >= 10 else iso_str
|
||||
|
||||
|
||||
def parse_offer(raw: Any, zip_code: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Defensively parses and standardizes an offer payload from Marktguru.
|
||||
Protects against null fields, malformed dates, and missing keys.
|
||||
"""
|
||||
if not isinstance(raw, dict):
|
||||
return None
|
||||
|
||||
try:
|
||||
offer_id = raw.get("id")
|
||||
price = raw.get("price")
|
||||
if isinstance(price, (int, float)):
|
||||
price = round(float(price), 2)
|
||||
else:
|
||||
price = None
|
||||
|
||||
old_price = raw.get("oldPrice")
|
||||
if isinstance(old_price, (int, float)):
|
||||
old_price = round(float(old_price), 2)
|
||||
else:
|
||||
old_price = None
|
||||
|
||||
ref_price = raw.get("referencePrice")
|
||||
if isinstance(ref_price, (int, float)):
|
||||
ref_price = round(float(ref_price), 2)
|
||||
else:
|
||||
ref_price = None
|
||||
|
||||
discount_pct = None
|
||||
if old_price and price and old_price > price:
|
||||
discount_pct = round(((old_price - price) / old_price) * 100)
|
||||
|
||||
# Validity dates
|
||||
validity = raw.get("validityDates")
|
||||
valid_from_raw = None
|
||||
valid_to_raw = None
|
||||
is_valid_now = True
|
||||
days_remaining = None
|
||||
|
||||
if isinstance(validity, list) and len(validity) > 0 and isinstance(validity[0], dict):
|
||||
valid_from_raw = validity[0].get("from")
|
||||
valid_to_raw = validity[0].get("to")
|
||||
|
||||
try:
|
||||
now = datetime.now(timezone.utc)
|
||||
if valid_to_raw and isinstance(valid_to_raw, str):
|
||||
dt_to = datetime.fromisoformat(valid_to_raw.replace("Z", "+00:00"))
|
||||
diff = (dt_to - now).days
|
||||
days_remaining = max(0, diff)
|
||||
if dt_to < now:
|
||||
is_valid_now = False
|
||||
|
||||
if valid_from_raw and isinstance(valid_from_raw, str):
|
||||
dt_from = datetime.fromisoformat(valid_from_raw.replace("Z", "+00:00"))
|
||||
if dt_from > now:
|
||||
is_valid_now = False
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Supermarket / advertiser extraction
|
||||
supermarket = "Unbekannt"
|
||||
supermarket_id = None
|
||||
advertisers = raw.get("advertisers")
|
||||
if isinstance(advertisers, list) and len(advertisers) > 0 and isinstance(advertisers[0], dict):
|
||||
supermarket = advertisers[0].get("name") or "Unbekannt"
|
||||
supermarket_id = advertisers[0].get("id")
|
||||
|
||||
# High resolution CDN image
|
||||
image_url = None
|
||||
if offer_id:
|
||||
image_url = f"https://mg2de.b-cdn.net/api/v1/offers/{offer_id}/images/default/0/medium.webp"
|
||||
|
||||
# Unit & volume
|
||||
unit_info = raw.get("unit")
|
||||
unit_name = None
|
||||
if isinstance(unit_info, dict):
|
||||
unit_name = unit_info.get("shortName") or unit_info.get("name")
|
||||
|
||||
product_info = raw.get("product")
|
||||
product_name = None
|
||||
if isinstance(product_info, dict):
|
||||
product_name = product_info.get("name")
|
||||
|
||||
brand_info = raw.get("brand")
|
||||
brand_name = None
|
||||
if isinstance(brand_info, dict):
|
||||
brand_name = brand_info.get("name")
|
||||
|
||||
return {
|
||||
"id": offer_id,
|
||||
"supermarket": supermarket,
|
||||
"supermarket_id": supermarket_id,
|
||||
"product_name": product_name,
|
||||
"brand": brand_name,
|
||||
"description": raw.get("description"),
|
||||
"price": price,
|
||||
"old_price": old_price,
|
||||
"reference_price": ref_price,
|
||||
"discount_percent": discount_pct,
|
||||
"unit": unit_name,
|
||||
"volume": raw.get("volume"),
|
||||
"loyalty_app_required": bool(raw.get("requiresLoyalityMembership", False)),
|
||||
"is_valid_now": is_valid_now,
|
||||
"days_remaining": days_remaining,
|
||||
"valid_from": format_iso_date(valid_from_raw),
|
||||
"valid_to": format_iso_date(valid_to_raw),
|
||||
"image_url": image_url,
|
||||
"zip_code": str(zip_code),
|
||||
}
|
||||
except Exception as e:
|
||||
logger.debug(f"Error parsing single offer {raw.get('id') if isinstance(raw, dict) else raw}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def _fetch_with_retry(url: str, max_retries: int = 2) -> Dict[str, Any]:
|
||||
"""
|
||||
Executes HTTP request with exponential backoff on transient errors (5xx / network blips).
|
||||
"""
|
||||
last_err: Optional[Exception] = None
|
||||
|
||||
for attempt in range(max_retries + 1):
|
||||
req = urllib.request.Request(url, headers=HEADERS)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as resp:
|
||||
raw_bytes = resp.read()
|
||||
return json.loads(raw_bytes.decode("utf-8"))
|
||||
except urllib.error.HTTPError as e:
|
||||
last_err = e
|
||||
# Retry only on transient server errors (500, 502, 503, 504) or rate limits (429)
|
||||
if e.code in (429, 500, 502, 503, 504) and attempt < max_retries:
|
||||
backoff = 0.5 * (2 ** attempt)
|
||||
logger.warning(f"Marktguru HTTP {e.code}. Retrying in {backoff:.1f}s (attempt {attempt+1}/{max_retries})...")
|
||||
time.sleep(backoff)
|
||||
continue
|
||||
raise
|
||||
except (urllib.error.URLError, TimeoutError, OSError) as e:
|
||||
last_err = e
|
||||
if attempt < max_retries:
|
||||
backoff = 0.5 * (2 ** attempt)
|
||||
logger.warning(f"Connection issue ({e}). Retrying in {backoff:.1f}s...")
|
||||
time.sleep(backoff)
|
||||
continue
|
||||
raise
|
||||
|
||||
raise last_err or RuntimeError("Max retries exceeded")
|
||||
|
||||
|
||||
def search_offers(
|
||||
query: str,
|
||||
zip_code: str = DEFAULT_ZIP_CODE,
|
||||
supermarkets: Optional[List[str]] = None,
|
||||
limit: int = 24,
|
||||
offset: int = 0,
|
||||
force_refresh: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Searches current supermarket offers in Germany via the Marktguru API.
|
||||
|
||||
Args:
|
||||
query: Search term (e.g. 'Butter', 'Kaffee', 'Bier', 'Milch')
|
||||
zip_code: German postal code (default: '93059' Regensburg)
|
||||
supermarkets: List of supermarket names or aliases (e.g. ['Aldi', 'Lidl', 'Edeka'])
|
||||
limit: Max number of items to return (default: 24, max: 50)
|
||||
offset: Result pagination offset
|
||||
force_refresh: If True, bypasses local TTL cache
|
||||
"""
|
||||
cleaned_query = (query or "").strip()
|
||||
cleaned_zip = str(zip_code or DEFAULT_ZIP_CODE).strip()
|
||||
limit = max(1, min(int(limit), 50))
|
||||
offset = max(0, int(offset))
|
||||
|
||||
retailer_ids = resolve_supermarket_ids(supermarkets)
|
||||
|
||||
# Compute deterministic cache key
|
||||
cache_raw = f"{cleaned_query.lower()}:{cleaned_zip}:{retailer_ids or ''}:{limit}:{offset}"
|
||||
cache_key = hashlib.sha256(cache_raw.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
if not force_refresh:
|
||||
cached_data, is_fresh = _read_cache(cache_key)
|
||||
if is_fresh and cached_data:
|
||||
return cached_data
|
||||
|
||||
params: Dict[str, Any] = {
|
||||
"as": "web",
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
"q": cleaned_query,
|
||||
"zipCode": cleaned_zip,
|
||||
}
|
||||
if retailer_ids:
|
||||
params["retailers"] = retailer_ids
|
||||
|
||||
url = f"{API_BASE}/offers/search?{urllib.parse.urlencode(params)}"
|
||||
|
||||
try:
|
||||
data = _fetch_with_retry(url)
|
||||
except Exception as e:
|
||||
logger.error(f"Marktguru API query failed: {e}")
|
||||
# Emergency fail-safe: Check if stale cache is available
|
||||
cached_data, _ = _read_cache(cache_key, max_age_seconds=86400 * 7)
|
||||
if cached_data:
|
||||
logger.warning("Serving stale cached data due to API failure.")
|
||||
cached_data["warning"] = "Stale cache returned due to temporary upstream API issue."
|
||||
return cached_data
|
||||
|
||||
return {
|
||||
"error": f"API request failed: {str(e)}",
|
||||
"query": cleaned_query,
|
||||
"zip_code": cleaned_zip,
|
||||
"total_results": 0,
|
||||
"returned_count": 0,
|
||||
"offers": [],
|
||||
"available_retailers": [],
|
||||
}
|
||||
|
||||
raw_results = data.get("results") or []
|
||||
parsed_offers: List[Dict[str, Any]] = []
|
||||
for item in raw_results:
|
||||
parsed = parse_offer(item, cleaned_zip)
|
||||
if parsed is not None:
|
||||
parsed_offers.append(parsed)
|
||||
|
||||
filters = data.get("filters") or {}
|
||||
raw_retailers = filters.get("retailers") or []
|
||||
available_retailers = [
|
||||
{"name": r.get("name"), "id": r.get("id"), "count": r.get("resultsCount")}
|
||||
for r in raw_retailers if isinstance(r, dict) and r.get("name")
|
||||
]
|
||||
|
||||
result = {
|
||||
"query": cleaned_query,
|
||||
"zip_code": cleaned_zip,
|
||||
"total_results": data.get("totalResults", len(parsed_offers)),
|
||||
"returned_count": len(parsed_offers),
|
||||
"available_retailers": available_retailers,
|
||||
"offers": parsed_offers,
|
||||
}
|
||||
|
||||
_write_cache(cache_key, result)
|
||||
return result
|
||||
|
||||
|
||||
def compare_prices(
|
||||
query: str,
|
||||
zip_code: str = DEFAULT_ZIP_CODE,
|
||||
supermarkets: Optional[List[str]] = None,
|
||||
force_refresh: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Compares prices for a given product query across supermarkets, sorted from cheapest to most expensive.
|
||||
"""
|
||||
data = search_offers(query, zip_code=zip_code, supermarkets=supermarkets, limit=50, force_refresh=force_refresh)
|
||||
offers = data.get("offers", [])
|
||||
|
||||
# Filter out items without price and sort ascending
|
||||
priced_offers = [o for o in offers if o.get("price") is not None]
|
||||
priced_offers.sort(key=lambda x: x["price"])
|
||||
|
||||
return {
|
||||
"query": query,
|
||||
"zip_code": str(zip_code),
|
||||
"total_compared": len(priced_offers),
|
||||
"cheapest_offer": priced_offers[0] if priced_offers else None,
|
||||
"all_offers_sorted": priced_offers,
|
||||
}
|
||||
|
||||
|
||||
def list_supported_supermarkets() -> Dict[str, Any]:
|
||||
"""Returns a structured catalog of all major supermarket chains supported by Marktguru."""
|
||||
categories: Dict[str, List[Dict[str, Any]]] = {
|
||||
"Discounter": [],
|
||||
"Supermarkt": [],
|
||||
"Großfläche / Hypermarkt": [],
|
||||
"Drogerie": [],
|
||||
}
|
||||
|
||||
for name, info in MAJOR_RETAILERS.items():
|
||||
cat = info.get("type", "Sonstige")
|
||||
if cat not in categories:
|
||||
categories[cat] = []
|
||||
categories[cat].append({
|
||||
"name": name,
|
||||
"id": info["id"],
|
||||
"major": info.get("major", False)
|
||||
})
|
||||
|
||||
return {
|
||||
"default_zip_code": DEFAULT_ZIP_CODE,
|
||||
"cache_ttl_seconds": CACHE_TTL,
|
||||
"supported_chains_count": len(MAJOR_RETAILERS),
|
||||
"categories": categories,
|
||||
"supported_aliases": list(RETAILER_ALIASES.keys()),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description="Query German supermarket flyers/discounts via Marktguru")
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
# Search
|
||||
search_p = subparsers.add_parser("search", help="Search offers for a product")
|
||||
search_p.add_argument("query", help="Product to search (e.g. Butter, Kaffee)")
|
||||
search_p.add_argument("--zip", default=DEFAULT_ZIP_CODE, help=f"Postal code (default: {DEFAULT_ZIP_CODE})")
|
||||
search_p.add_argument("--supermarket", help="Comma-separated supermarkets (e.g. 'Aldi,Lidl,Edeka')")
|
||||
search_p.add_argument("--limit", type=int, default=15, help="Max results (default 15)")
|
||||
search_p.add_argument("--refresh", action="store_true", help="Bypass local cache")
|
||||
search_p.add_argument("--json", action="store_true", help="Output raw JSON")
|
||||
|
||||
# Compare
|
||||
comp_p = subparsers.add_parser("compare", help="Compare prices across supermarkets")
|
||||
comp_p.add_argument("query", help="Product to compare")
|
||||
comp_p.add_argument("--zip", default=DEFAULT_ZIP_CODE, help=f"Postal code (default: {DEFAULT_ZIP_CODE})")
|
||||
comp_p.add_argument("--supermarket", help="Comma-separated supermarkets")
|
||||
comp_p.add_argument("--refresh", action="store_true", help="Bypass local cache")
|
||||
comp_p.add_argument("--json", action="store_true", help="Output raw JSON")
|
||||
|
||||
# Supermarkets
|
||||
subparsers.add_parser("supermarkets", help="List all supported supermarkets and chains")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "supermarkets":
|
||||
data = list_supported_supermarkets()
|
||||
print(json.dumps(data, indent=2, ensure_ascii=False))
|
||||
return
|
||||
|
||||
sm_list = [s.strip() for s in args.supermarket.split(",")] if getattr(args, "supermarket", None) else None
|
||||
|
||||
if args.command == "search":
|
||||
result = search_offers(
|
||||
args.query,
|
||||
zip_code=args.zip,
|
||||
supermarkets=sm_list,
|
||||
limit=args.limit,
|
||||
force_refresh=args.refresh
|
||||
)
|
||||
if args.json:
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
if "error" in result and not result.get("offers"):
|
||||
print(f"❌ Fehler: {result['error']}")
|
||||
return
|
||||
|
||||
print(f"\n🔍 Angebote für '{args.query}' in PLZ {args.zip} (Gefunden: {result['total_results']}):\n")
|
||||
for o in result["offers"]:
|
||||
disc_str = f" (-{o['discount_percent']}%)" if o.get("discount_percent") else ""
|
||||
loyalty_str = " 📱 [App Coupon]" if o.get("loyalty_app_required") else ""
|
||||
timing_str = f"noch {o['days_remaining']} Tage" if o.get("days_remaining") is not None else f"bis {o['valid_to']}"
|
||||
print(f" • [{o['supermarket']}] {o['description'] or o['product_name'] or 'Angebot'}")
|
||||
print(f" Preis: {o['price']:.2f} €{disc_str}{loyalty_str} | Gültig: {timing_str}")
|
||||
if o.get("reference_price"):
|
||||
print(f" Grundpreis: {o['reference_price']:.2f} € / {o.get('unit') or 'Einheit'}")
|
||||
print()
|
||||
|
||||
elif args.command == "compare":
|
||||
result = compare_prices(
|
||||
args.query,
|
||||
zip_code=args.zip,
|
||||
supermarkets=sm_list,
|
||||
force_refresh=args.refresh
|
||||
)
|
||||
if args.json:
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
print(f"\n🏷️ Preisvergleich für '{args.query}' in PLZ {args.zip}:\n")
|
||||
for idx, o in enumerate(result["all_offers_sorted"], 1):
|
||||
disc_str = f" (UVP {o['old_price']:.2f}€)" if o.get("old_price") else ""
|
||||
loyalty_str = " [App erforderlich]" if o.get("loyalty_app_required") else ""
|
||||
print(f" {idx:2d}. {o['price']:.2f} € - [{o['supermarket']}] {o['description'] or o['product_name']}{disc_str}{loyalty_str}")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+319
@@ -0,0 +1,319 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Marktguru MCP (Model Context Protocol) Server (Production Grade)
|
||||
Exposes German supermarket flyers, discounts, and price comparison tools over standard JSON-RPC 2.0 stdio.
|
||||
|
||||
Key Design Principles:
|
||||
- Zero external dependencies: Uses standard library only.
|
||||
- Strict Protocol Compliance: Conforms to MCP (2024-11-05) and JSON-RPC 2.0 specifications.
|
||||
- Defensive Argument Coercion: Tolerates string/number variations commonly output by LLMs.
|
||||
- Graceful Signal Handling: Clean exit on SIGINT, SIGTERM, and EOF.
|
||||
- Stream Isolation: All logs and diagnostics route to stderr, ensuring stdout is 100% clean JSON-RPC.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
import signal
|
||||
import logging
|
||||
from typing import Dict, Any, List, Optional
|
||||
|
||||
# Import business logic from hardened client script
|
||||
from marktguru_client import (
|
||||
search_offers,
|
||||
compare_prices,
|
||||
list_supported_supermarkets,
|
||||
DEFAULT_ZIP_CODE,
|
||||
MAJOR_RETAILERS,
|
||||
RETAILER_ALIASES,
|
||||
)
|
||||
|
||||
# Configure logging strictly to stderr
|
||||
logging.basicConfig(
|
||||
stream=sys.stderr,
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
|
||||
)
|
||||
logger = logging.getLogger("marktguru-mcp")
|
||||
|
||||
PROTOCOL_VERSION = "2024-11-05"
|
||||
SERVER_NAME = "marktguru-mcp-server"
|
||||
SERVER_VERSION = "1.1.0"
|
||||
|
||||
TOOLS = [
|
||||
{
|
||||
"name": "search_supermarket_offers",
|
||||
"description": (
|
||||
"Search current weekly supermarket offers and promotional discounts in Germany via the Marktguru API. "
|
||||
"Supports filtering by postal code (PLZ) and specific supermarket chains (e.g. Aldi Süd, Aldi Nord, Lidl, Edeka, Rewe, Kaufland, Penny, Netto)."
|
||||
),
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "The product or keyword to search for (e.g. 'Butter', 'Kaffee', 'Bier', 'Milch', 'Nutella', 'Schokolade')."
|
||||
},
|
||||
"zip_code": {
|
||||
"type": "string",
|
||||
"description": f"5-digit German postal code (PLZ). Defaults to '{DEFAULT_ZIP_CODE}' (Regensburg)."
|
||||
},
|
||||
"supermarkets": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Optional list of supermarket names to filter by (e.g. ['Aldi', 'Lidl', 'Edeka', 'Rewe', 'Kaufland', 'Penny', 'Netto', 'dm', 'Rossmann']). If omitted, all available retailers are included."
|
||||
},
|
||||
"limit": {
|
||||
"type": "integer",
|
||||
"description": "Maximum number of offers to return (default: 20, max: 50)."
|
||||
}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "compare_supermarket_prices",
|
||||
"description": (
|
||||
"Compare current prices for a product across German supermarkets in a given postal code. "
|
||||
"Returns results sorted from lowest to highest price, highlighting the cheapest deal, regular prices (UVP), and required loyalty app memberships."
|
||||
),
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "The product or grocery item to compare (e.g. 'Kaffee', 'Butter', 'Bier', 'Olivenöl')."
|
||||
},
|
||||
"zip_code": {
|
||||
"type": "string",
|
||||
"description": f"5-digit German postal code (PLZ). Defaults to '{DEFAULT_ZIP_CODE}' (Regensburg)."
|
||||
},
|
||||
"supermarkets": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"},
|
||||
"description": "Optional list of supermarket names to compare (e.g. ['Aldi', 'Lidl', 'Edeka', 'Rewe'])."
|
||||
}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "list_supported_supermarkets",
|
||||
"description": (
|
||||
"List all supermarket, discounter, hypermarket, and drugstore chains supported by the Marktguru service in Germany, "
|
||||
"including their categories (Discounter, Supermarkt, Großfläche, Drogerie) and accepted name aliases."
|
||||
),
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def _sanitize_supermarkets(sm_input: Any) -> Optional[List[str]]:
|
||||
"""Coerces various LLM input formats (list, comma-string, single string) into a clean list."""
|
||||
if not sm_input:
|
||||
return None
|
||||
if isinstance(sm_input, list):
|
||||
return [str(s).strip() for s in sm_input if str(s).strip()]
|
||||
if isinstance(sm_input, str):
|
||||
return [s.strip() for s in sm_input.split(",") if s.strip()]
|
||||
return None
|
||||
|
||||
|
||||
def handle_initialize(req_id: Any, params: Dict[str, Any]) -> Dict[str, Any]:
|
||||
return {
|
||||
"jsonrpc": "2.0",
|
||||
"id": req_id,
|
||||
"result": {
|
||||
"protocolVersion": PROTOCOL_VERSION,
|
||||
"capabilities": {
|
||||
"tools": {
|
||||
"listChanged": False
|
||||
}
|
||||
},
|
||||
"serverInfo": {
|
||||
"name": SERVER_NAME,
|
||||
"version": SERVER_VERSION
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def handle_tools_list(req_id: Any) -> Dict[str, Any]:
|
||||
return {
|
||||
"jsonrpc": "2.0",
|
||||
"id": req_id,
|
||||
"result": {
|
||||
"tools": TOOLS
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def handle_tools_call(req_id: Any, params: Dict[str, Any]) -> Dict[str, Any]:
|
||||
tool_name = params.get("name")
|
||||
args = params.get("arguments", {})
|
||||
|
||||
logger.info(f"Tool call requested: {tool_name} with args={args}")
|
||||
|
||||
try:
|
||||
if tool_name == "search_supermarket_offers":
|
||||
query = str(args.get("query") or "").strip()
|
||||
if not query:
|
||||
return {
|
||||
"jsonrpc": "2.0",
|
||||
"id": req_id,
|
||||
"result": {
|
||||
"content": [{"type": "text", "text": "Error: 'query' argument is required and cannot be empty."}],
|
||||
"isError": True
|
||||
}
|
||||
}
|
||||
|
||||
zip_code = str(args.get("zip_code") or DEFAULT_ZIP_CODE).strip()
|
||||
supermarkets = _sanitize_supermarkets(args.get("supermarkets"))
|
||||
try:
|
||||
limit = int(args.get("limit") or 20)
|
||||
except (ValueError, TypeError):
|
||||
limit = 20
|
||||
|
||||
data = search_offers(query, zip_code=zip_code, supermarkets=supermarkets, limit=limit)
|
||||
text_result = json.dumps(data, indent=2, ensure_ascii=False)
|
||||
|
||||
elif tool_name == "compare_supermarket_prices":
|
||||
query = str(args.get("query") or "").strip()
|
||||
if not query:
|
||||
return {
|
||||
"jsonrpc": "2.0",
|
||||
"id": req_id,
|
||||
"result": {
|
||||
"content": [{"type": "text", "text": "Error: 'query' argument is required and cannot be empty."}],
|
||||
"isError": True
|
||||
}
|
||||
}
|
||||
|
||||
zip_code = str(args.get("zip_code") or DEFAULT_ZIP_CODE).strip()
|
||||
supermarkets = _sanitize_supermarkets(args.get("supermarkets"))
|
||||
|
||||
data = compare_prices(query, zip_code=zip_code, supermarkets=supermarkets)
|
||||
text_result = json.dumps(data, indent=2, ensure_ascii=False)
|
||||
|
||||
elif tool_name == "list_supported_supermarkets":
|
||||
data = list_supported_supermarkets()
|
||||
text_result = json.dumps(data, indent=2, ensure_ascii=False)
|
||||
|
||||
else:
|
||||
return {
|
||||
"jsonrpc": "2.0",
|
||||
"id": req_id,
|
||||
"error": {
|
||||
"code": -32601,
|
||||
"message": f"Method or tool '{tool_name}' not found."
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
"jsonrpc": "2.0",
|
||||
"id": req_id,
|
||||
"result": {
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": text_result
|
||||
}
|
||||
],
|
||||
"isError": False
|
||||
}
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Exception during tool execution '{tool_name}': {e}")
|
||||
return {
|
||||
"jsonrpc": "2.0",
|
||||
"id": req_id,
|
||||
"result": {
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": f"Tool execution failed: {type(e).__name__}: {str(e)}"
|
||||
}
|
||||
],
|
||||
"isError": True
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def process_message(line: str) -> Optional[Dict[str, Any]]:
|
||||
if not line.strip():
|
||||
return None
|
||||
|
||||
try:
|
||||
msg = json.loads(line)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Malformed JSON-RPC received: {e}")
|
||||
return {
|
||||
"jsonrpc": "2.0",
|
||||
"id": None,
|
||||
"error": {"code": -32700, "message": f"Parse error: {str(e)}"}
|
||||
}
|
||||
|
||||
method = msg.get("method")
|
||||
req_id = msg.get("id")
|
||||
params = msg.get("params", {})
|
||||
|
||||
# Notifications do not return a response
|
||||
if req_id is None:
|
||||
if method == "notifications/initialized":
|
||||
logger.info("Client acknowledged initialized notification.")
|
||||
elif method == "notifications/cancelled":
|
||||
logger.info(f"Client cancelled operation: {params}")
|
||||
return None
|
||||
|
||||
# Standard Requests
|
||||
if method == "initialize":
|
||||
return handle_initialize(req_id, params)
|
||||
elif method == "tools/list":
|
||||
return handle_tools_list(req_id)
|
||||
elif method == "tools/call":
|
||||
return handle_tools_call(req_id, params)
|
||||
elif method == "ping":
|
||||
return {"jsonrpc": "2.0", "id": req_id, "result": {}}
|
||||
else:
|
||||
return {
|
||||
"jsonrpc": "2.0",
|
||||
"id": req_id,
|
||||
"error": {
|
||||
"code": -32601,
|
||||
"message": f"Method '{method}' is not supported."
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def _sig_handler(sig, frame):
|
||||
logger.info(f"Received termination signal {sig}. Exiting gracefully.")
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
def main():
|
||||
signal.signal(signal.SIGINT, _sig_handler)
|
||||
signal.signal(signal.SIGTERM, _sig_handler)
|
||||
|
||||
logger.info(f"Starting {SERVER_NAME} v{SERVER_VERSION} (stdio mode)...")
|
||||
|
||||
try:
|
||||
for raw_line in sys.stdin:
|
||||
response = process_message(raw_line)
|
||||
if response is not None:
|
||||
output = json.dumps(response, ensure_ascii=False) + "\n"
|
||||
sys.stdout.write(output)
|
||||
sys.stdout.flush()
|
||||
except (KeyboardInterrupt, BrokenPipeError):
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.exception(f"Fatal error in stdio loop: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
logger.info("Marktguru MCP server shutdown complete.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,180 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Unit tests for the Marktguru skill & MCP Server.
|
||||
Runs completely offline using standard library unittest.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import unittest
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
# Add scripts directory to path
|
||||
SCRIPTS_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "scripts"))
|
||||
if SCRIPTS_DIR not in sys.path:
|
||||
sys.path.insert(0, SCRIPTS_DIR)
|
||||
|
||||
from marktguru_client import (
|
||||
resolve_supermarket_ids,
|
||||
format_iso_date,
|
||||
parse_offer,
|
||||
list_supported_supermarkets,
|
||||
DEFAULT_ZIP_CODE,
|
||||
_read_cache,
|
||||
_write_cache,
|
||||
_get_cache_dir,
|
||||
)
|
||||
from marktguru_mcp import process_message, SERVER_NAME, PROTOCOL_VERSION
|
||||
|
||||
|
||||
class TestMarktguruClient(unittest.TestCase):
|
||||
def test_default_zip_code(self):
|
||||
self.assertEqual(DEFAULT_ZIP_CODE, "93059")
|
||||
|
||||
def test_resolve_supermarket_ids_aliases(self):
|
||||
# Aldi maps to Aldi Nord and Aldi Süd
|
||||
aldi_ids = resolve_supermarket_ids(["aldi"])
|
||||
self.assertIn("127153", aldi_ids)
|
||||
self.assertIn("127139", aldi_ids)
|
||||
|
||||
# Edeka maps to Edeka and E center
|
||||
edeka_ids = resolve_supermarket_ids(["edeka"])
|
||||
self.assertIn("127093", edeka_ids)
|
||||
self.assertIn("126699", edeka_ids)
|
||||
|
||||
# Lidl
|
||||
lidl_ids = resolve_supermarket_ids(["lidl"])
|
||||
self.assertEqual(lidl_ids, "126679")
|
||||
|
||||
# Comma-separated string support
|
||||
combined = resolve_supermarket_ids("Lidl, Kaufland")
|
||||
self.assertIn("126679", combined)
|
||||
self.assertIn("126654", combined)
|
||||
|
||||
# Unknown supermarket is ignored gracefully
|
||||
unknown = resolve_supermarket_ids(["NonExistentMarket123"])
|
||||
self.assertIsNone(unknown)
|
||||
|
||||
def test_format_iso_date(self):
|
||||
self.assertEqual(format_iso_date("2026-10-15T12:00:00Z"), "15.10.2026")
|
||||
self.assertEqual(format_iso_date(None), "-")
|
||||
self.assertEqual(format_iso_date(""), "-")
|
||||
self.assertEqual(format_iso_date("invalid-date-format"), "invalid-da")
|
||||
|
||||
def test_parse_offer_defensive(self):
|
||||
# 1. Complete payload
|
||||
now = datetime.now(timezone.utc)
|
||||
from_date = (now - timedelta(days=1)).isoformat()
|
||||
to_date = (now + timedelta(days=4)).isoformat()
|
||||
|
||||
raw_complete = {
|
||||
"id": 99999,
|
||||
"price": 1.49,
|
||||
"oldPrice": 2.29,
|
||||
"referencePrice": 5.96,
|
||||
"requiresLoyalityMembership": True,
|
||||
"description": "250g Packung",
|
||||
"product": {"name": "Süßrahmbutter"},
|
||||
"brand": {"name": "Landliebe"},
|
||||
"advertisers": [{"name": "EDEKA", "id": "retailers/127093"}],
|
||||
"unit": {"shortName": "kg"},
|
||||
"volume": 0.25,
|
||||
"validityDates": [{"from": from_date, "to": to_date}]
|
||||
}
|
||||
|
||||
parsed = parse_offer(raw_complete, "93059")
|
||||
self.assertIsNotNone(parsed)
|
||||
self.assertEqual(parsed["id"], 99999)
|
||||
self.assertEqual(parsed["supermarket"], "EDEKA")
|
||||
self.assertEqual(parsed["price"], 1.49)
|
||||
self.assertEqual(parsed["old_price"], 2.29)
|
||||
self.assertEqual(parsed["discount_percent"], 35) # (2.29 - 1.49) / 2.29 = 34.9% -> 35%
|
||||
self.assertTrue(parsed["loyalty_app_required"])
|
||||
self.assertTrue(parsed["is_valid_now"])
|
||||
self.assertGreaterEqual(parsed["days_remaining"], 3)
|
||||
self.assertIn("https://mg2de.b-cdn.net/api/v1/offers/99999", parsed["image_url"])
|
||||
|
||||
# 2. Minimal / empty payload
|
||||
raw_empty = {}
|
||||
parsed_min = parse_offer(raw_empty, "93059")
|
||||
self.assertIsNotNone(parsed_min)
|
||||
self.assertEqual(parsed_min["supermarket"], "Unbekannt")
|
||||
self.assertIsNone(parsed_min["price"])
|
||||
self.assertIsNone(parsed_min["discount_percent"])
|
||||
|
||||
# 3. Invalid types (not a dict)
|
||||
self.assertIsNone(parse_offer("not a dict", "93059"))
|
||||
self.assertIsNone(parse_offer(None, "93059"))
|
||||
|
||||
def test_cache_mechanism(self):
|
||||
test_key = "test_unit_key_123"
|
||||
test_data = {"hello": "world", "timestamp": 12345}
|
||||
|
||||
_write_cache(test_key, test_data)
|
||||
data, is_fresh = _read_cache(test_key, max_age_seconds=60)
|
||||
self.assertTrue(is_fresh)
|
||||
self.assertEqual(data, test_data)
|
||||
|
||||
# Stale cache test
|
||||
data_stale, is_fresh_stale = _read_cache(test_key, max_age_seconds=-1)
|
||||
self.assertFalse(is_fresh_stale)
|
||||
self.assertEqual(data_stale, test_data) # still returned as fallback!
|
||||
|
||||
def test_list_supported_supermarkets(self):
|
||||
catalog = list_supported_supermarkets()
|
||||
self.assertEqual(catalog["default_zip_code"], "93059")
|
||||
self.assertIn("Discounter", catalog["categories"])
|
||||
self.assertIn("Supermarkt", catalog["categories"])
|
||||
self.assertIn("aldi", catalog["supported_aliases"])
|
||||
self.assertIn("edeka", catalog["supported_aliases"])
|
||||
|
||||
|
||||
class TestMarktguruMCP(unittest.TestCase):
|
||||
def test_mcp_initialize(self):
|
||||
req = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}})
|
||||
resp = process_message(req)
|
||||
self.assertEqual(resp["id"], 1)
|
||||
self.assertEqual(resp["result"]["protocolVersion"], PROTOCOL_VERSION)
|
||||
self.assertEqual(resp["result"]["serverInfo"]["name"], SERVER_NAME)
|
||||
|
||||
def test_mcp_tools_list(self):
|
||||
req = json.dumps({"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}})
|
||||
resp = process_message(req)
|
||||
tools = [t["name"] for t in resp["result"]["tools"]]
|
||||
self.assertIn("search_supermarket_offers", tools)
|
||||
self.assertIn("compare_supermarket_prices", tools)
|
||||
self.assertIn("list_supported_supermarkets", tools)
|
||||
|
||||
def test_mcp_tools_call_supermarket_catalog(self):
|
||||
req = json.dumps({
|
||||
"jsonrpc": "2.0",
|
||||
"id": 3,
|
||||
"method": "tools/call",
|
||||
"params": {"name": "list_supported_supermarkets", "arguments": {}}
|
||||
})
|
||||
resp = process_message(req)
|
||||
self.assertFalse(resp["result"]["isError"])
|
||||
content_text = resp["result"]["content"][0]["text"]
|
||||
data = json.loads(content_text)
|
||||
self.assertEqual(data["default_zip_code"], "93059")
|
||||
|
||||
def test_mcp_tools_call_missing_query(self):
|
||||
req = json.dumps({
|
||||
"jsonrpc": "2.0",
|
||||
"id": 4,
|
||||
"method": "tools/call",
|
||||
"params": {"name": "search_supermarket_offers", "arguments": {"query": ""}}
|
||||
})
|
||||
resp = process_message(req)
|
||||
self.assertTrue(resp["result"]["isError"])
|
||||
self.assertIn("required", resp["result"]["content"][0]["text"])
|
||||
|
||||
def test_mcp_notification_silence(self):
|
||||
req = json.dumps({"jsonrpc": "2.0", "method": "notifications/initialized"})
|
||||
resp = process_message(req)
|
||||
self.assertIsNone(resp) # Notifications must not produce responses
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -139,8 +139,7 @@ Subagent specifications, system prompts, tool assignments, and domain guidelines
|
||||
|
||||
### 3. Multi-Turn Continuity: Domain Stickiness & Rich State Bridge
|
||||
To prevent context loss during ongoing step-by-step dialogues (e.g. interactive cooking or workout design):
|
||||
* **Domain Stickiness**: Follow-up questions on the active topic remain within the specialized subagent domain until an explicit topic shift occurs.
|
||||
* **Rich State Bridge (`<!-- DOMAIN_STATE -->`)**: Every subagent embeds a compact JSON state comment at the end of its response. The Conductor uses this state to answer trivial micro-checks in < 1s, or feeds it back into the subagent for complex modifications.
|
||||
* **Persistent Context Continuity**: Follow-up turns are routed within the dedicated subagent topic session using native conversational memory, keeping user-facing messages clean without embedding raw JSON/HTML metadata tags.
|
||||
|
||||
### 4. 3-Tier Session Lifecycle (Preventing Unbounded Session Growth)
|
||||
1. **Conversational Close**: Topic shifts or explicit user completion (*"Danke, das war's"*) reset the active domain pointer.
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
---
|
||||
status: "accepted"
|
||||
date: 2026-10-06
|
||||
decision-makers: ["Moritz Graf", "Antigravity"]
|
||||
consulted: []
|
||||
informed: []
|
||||
---
|
||||
|
||||
# ADR 0004: Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)
|
||||
|
||||
## Context and Problem Statement
|
||||
|
||||
OpenClaw manages personal logging and family assistance for the Graf family across several daily domains:
|
||||
1. **Personal Nutrition:** Daily food intake, macro tracking (protein, carbs, sugar, fat, fiber, salt, calories) in `entities/nutrition-log/moritz/*.md`.
|
||||
2. **Baby Sleep & Routines:** Sleep logs, naps, night sleep, bottle feeds, and developmental milestones for Mila in `entities/people/mila-schlafprotokoll.md`.
|
||||
3. **Sports & Physiological Balance:** Activities, daily calorie expenditure (RMR + active), and net caloric surplus/deficit synced with Garmin Connect.
|
||||
4. **Culinary Knowledge:** Recipes with ingredient proportions, cooking steps, and hardware settings.
|
||||
5. **Points of Interest (POIs):** Local stores, farm stands (Hofläden), beer gardens, and playgrounds in Regensburg.
|
||||
|
||||
Historically, this data was logged directly into Markdown tables and bulleted lists in the Knowledge Wiki (`knowledge-wiki/`). An analysis of existing log files revealed severe systemic failure modes:
|
||||
- **LLM Arithmetic Drift:** In `2026-09.md` and `2026-10.md`, the LLM consistently failed at daily macro sums (errors up to 75 kcal and 7.5 g protein per day). The derived "Vortags-Bilanz" (caloric deficit/surplus against Garmin) was calculated from incorrect totals.
|
||||
- **Structural and Chronological Degradation:** In `mila-schlafprotokoll.md`, dates drifted out of order (`29.09 → Sep Summary → 30.09 → 01.10 → 03.10 → 02.10 → 28.09`), and every night sleep was duplicated (once in the evening, once the next morning).
|
||||
- **Inability to Perform Spatial & Aggregate Queries:** Questions like *"What was Mila's average night sleep duration over the last 14 days?"* or *"Which beer gardens are within 500m of Grabengasse 7?"* required either multi-turn LLM hallucination or reading hundreds of lines of raw text.
|
||||
|
||||
How do we establish a structured, reliable, and queryable data layer for OpenClaw that guarantees mathematical precision while upholding our Everything-as-Code (ADR 0003) and Git-as-Source-of-Truth architecture?
|
||||
|
||||
---
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
* **Mathematical Precision:** Zero LLM arithmetic. All summations, caloric balances, sleep durations, and nutrient rollups must be computed by a deterministic SQL engine.
|
||||
* **Git as the Single Source of Truth (ADR 0003):** No external database silos (PostgreSQL, standalone SQLite PVCs) that can drift from the git-backed wiki. All state must remain git-versioned, diffable, and merge-friendly.
|
||||
* **Separation of Concerns (Math vs. Narrative):** Delegate structured math, joins, and coordinates to DuckDB. Delegate natural-language understanding, nuance, packaging photo interpretation, and cooking advice to the LLM.
|
||||
* **Flexible Composability:** Foods can become ingredients in recipes, which can yield composite foods (e.g. wheat → baking mix → bread; tomatoes → sauce → pizza).
|
||||
* **Multi-Channel Presentation:** Auto-updated Markdown files for SilverBullet (`wiki.moritzgraf.de`), Telegram inline text with `chart-to-chat` PNGs, and mobile-friendly HTML dashboards on `hosting.haumdaucher.de/family/`.
|
||||
* **Zero-Tax Execution:** In-memory query execution in Python via `uv` without adding heavy runtime containers or pod memory overhead.
|
||||
|
||||
---
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Option 1: Stricter Markdown Prompts + Python Lint Rules**
|
||||
* Keep Markdown tables; add linter checks in `okf_lint.py` to flag wrong sums.
|
||||
* *Drawback*: Reactive detection rather than proactive prevention. Cannot do spatial searches or aggregate SQL queries.
|
||||
|
||||
2. **Option 2: External Relational Database (PostgreSQL / PVC SQLite)**
|
||||
* Deploy PostgreSQL or write to a binary SQLite database on the Kubernetes PVC.
|
||||
* *Drawback*: Violates ADR 0003. Creates a separate data silo outside git, introduces backup synchronization risks, and binary files cannot be cleanly merged or diffed in Git.
|
||||
|
||||
3. **Option 3: Pure Event Sourcing for All Data**
|
||||
* Force every piece of data (including stores, recipes, and food labels) into an immutable event log.
|
||||
* *Drawback*: Over-engineers static catalogs. Catalog entities (like places or recipes) become awkward to manage and query when forced into point-in-time event streams.
|
||||
|
||||
4. **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git (Selected)**
|
||||
* Store structured records in line-delimited JSON (`.jsonl`) inside the Knowledge Wiki repository (`data/`), using in-memory DuckDB as the calculation and query engine.
|
||||
* Divide data into three natural archetypes: **Events**, **Entities**, and **Documents**.
|
||||
|
||||
---
|
||||
|
||||
## Decision Outcome
|
||||
|
||||
Chosen Option: **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git**.
|
||||
|
||||
### 1. The Three Data Archetypes
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────────┐
|
||||
│ Three Data Archetypes │
|
||||
├─────────────────────────┬─────────────────────────┬────────────────────┤
|
||||
│ 1. EVENTS (Time-Series) │ 2. ENTITIES (Catalogs) │ 3. DOCUMENTS (Prose│
|
||||
├─────────────────────────┼─────────────────────────┼────────────────────┤
|
||||
│ • data/events/nutrition/│ • data/entities/foods/ │ • entities/recipes/│
|
||||
│ • data/events/sleep/ │ • data/entities/recipes/│ • syntheses/places/│
|
||||
│ • data/events/feeding/ │ • data/entities/places/ │ • sources/*.pdf │
|
||||
│ • data/events/body/ │ • data/entities/people/ │ │
|
||||
│ [Point & Interval Logs] │ [Durable Slug Identifiers] [Markdown / Media]│
|
||||
└────────────┬────────────┴────────────┬────────────┴──────────┬─────────┘
|
||||
│ │ │
|
||||
▼ ▼ │
|
||||
┌──────────────────────────────────────────────┐ │
|
||||
│ In-Memory DuckDB SQL Engine │ │
|
||||
│ • Schema Validation (STRUCT & CHECK) │ │
|
||||
│ • SQL Aggregations & Daily Deficit Joins │ │
|
||||
│ • Spatial Queries (ST_DWithin / Haversine) │ │
|
||||
└──────────────────────┬───────────────────────┘ │
|
||||
│ │
|
||||
┌────────────┴────────────┐ │
|
||||
▼ ▼ │
|
||||
┌──────────────────┐ ┌──────────────────┐ │
|
||||
│ Telegram Output │ │ Generated Views │ │
|
||||
│ • Text Summaries │ │ • Markdown Views ◄────────────┘
|
||||
│ • chart-to-chat │ │ • HTML Dashboards│ (Frontmatter:
|
||||
└──────────────────┘ │ (/family/...) │ datastore_ref)
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
1. **Events (Time-Series Logs):**
|
||||
- Stored in `data/events/<domain>/<YYYY-MM>.jsonl`, sorted by `(start, id)`.
|
||||
- Unified envelope: `id` (ULID), `kind`, `subject`, `actor`, `start`, `end` (supports open intervals where `end=NULL`), `estimated`, `meal`, `note`, `payload` (JSON cast to typed STRUCT), `source_ref`.
|
||||
2. **Entities (Structured Catalogs):**
|
||||
- Stored in `data/entities/<collection>.jsonl`, sorted by slug `id`.
|
||||
- `foods`: EU LMIV per-100g nutritional profile, ingredient lists, barcodes, density.
|
||||
- `recipes`: Ingredients list (linking `foods` with exact grams or raw text), servings, calculated macros.
|
||||
- `places`: Points of interest with `lat`, `lon`, category, address, tags, and linked documents.
|
||||
- `people`: Family profile registry.
|
||||
3. **Documents (Prose & Media):**
|
||||
- Stored as standard Markdown in `knowledge-wiki/`.
|
||||
- Deterministic two-way link: Document YAML frontmatter contains `datastore_ref: "<collection>:<id>"`, while the entity JSONL record lists `documents: ["path/to/doc.md"]`.
|
||||
|
||||
### 2. Dedicated Raw Garmin Mirror (`data/garmin/`)
|
||||
- Nightly zero-LLM Python sync script fetches raw Garmin API responses directly into:
|
||||
- `data/garmin/activities/<YYYY-MM>.jsonl`
|
||||
- `data/garmin/daily/<YYYY-MM>.jsonl`
|
||||
- `data/garmin/sleep/<YYYY-MM>.jsonl`
|
||||
- DuckDB queries raw JSON directly with `read_json_auto()` and computes the daily caloric deficit/surplus by joining nutrition intake with Garmin total daily energy expenditure.
|
||||
|
||||
### 3. Spatial & Geospatial Radius Lookups
|
||||
- DuckDB attempts to initialize the official `spatial` extension (`LOAD spatial;`).
|
||||
- If offline or binary installation fails, it transparently falls back to an internal pure-SQL Haversine macro `distance_m(lat1, lon1, lat2, lon2)`, ensuring zero-failure execution across any environment.
|
||||
|
||||
### 4. Privacy & Hosting Routes (`hosting.haumdaucher.de`)
|
||||
- **Public (`/recipes/*`, `/places/*`, `/public/*`):** Publicly accessible recipe cards and shareable dossiers.
|
||||
- **Family (`/family/*`):** Protected by Traefik ForwardAuth linked to Authentik group `family` (nutrition dashboards, baby sleep charts).
|
||||
- **Private (`/private/*`):** Protected by Authentik for personal vital logs.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- **Guaranteed Calculation Accuracy:** All calories, macronutrient totals, sleep statistics, and energy balances are computed in SQL. Zero LLM math errors.
|
||||
- **Git-Native & Mergable:** Text-based JSONL files partition data cleanly and produce clean 1-line git diffs.
|
||||
- **Offline & Low-Latency:** In-memory DuckDB reads months of JSONL in single-digit milliseconds.
|
||||
- **Preserved Human Readability:** SilverBullet (`wiki.moritzgraf.de`) and Obsidian continue to display updated Markdown tables generated directly from the database.
|
||||
|
||||
### Negative & Mitigations
|
||||
- **Generated Markdown Views Must Not Be Hand-Edited:**
|
||||
- *Mitigation*: Generated files include a prominent `<!-- DO NOT EDIT: GENERATED BY DATASTORE -->` warning comment.
|
||||
- *Mitigation*: Git pre-commit hook and the weekly IT Guy audit verify that generated views match the underlying JSONL data.
|
||||
- **Dual-Step Agent Retrieval:** Agents execute `datastore query` for structured data, then `wiki_get` on linked documents if full narrative text is needed.
|
||||
- *Mitigation*: Clear instructions and query helpers in the `datastore` skill prompt.
|
||||
@@ -13,6 +13,7 @@ We follow the [Markdown Architectural Decision Records (MADR)](https://adr.githu
|
||||
| [**0001**](0001-hybrid-mcp-tool-search-and-subagents.md) | **Hybrid MCP Tool Search and Subagent Delegation for AI Agents** | `accepted` | 2026-09-12 | OpenClaw / AI / MCP |
|
||||
| [**0002**](0002-openclaw-model-selection-tiering-and-economics.md) | **OpenClaw Multi-Tier Model Selection, Latency Optimization, and Inference Economics** | `proposed` | 2026-09-13 | OpenClaw / AI / Models |
|
||||
| [**0003**](0003-everything-as-code-and-reproducible-infrastructure.md) | **Everything as Code (EaC) and Reproducible Declarative Infrastructure** | `accepted` | 2026-09-17 | Architecture / Monorepo / EaC |
|
||||
| [**0004**](0004-structured-personal-data-datastore.md) | **Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)** | `accepted` | 2026-10-06 | OpenClaw / Data / Storage |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ spec:
|
||||
addPrefix:
|
||||
prefix: /hosting
|
||||
---
|
||||
# Public Ingress for hosting.haumdaucher.de (Recipes, Public Assets)
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
@@ -16,6 +17,7 @@ metadata:
|
||||
kubernetes.io/ingress.class: "traefik"
|
||||
cert-manager.io/cluster-issuer: "letsencrypt-prod"
|
||||
kubernetes.io/tls-acme: "true"
|
||||
traefik.ingress.kubernetes.io/router.priority: "10"
|
||||
traefik.ingress.kubernetes.io/router.middlewares: minio-hosting-add-prefix@kubernetescrd
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
@@ -34,3 +36,34 @@ spec:
|
||||
- hosts:
|
||||
- hosting.haumdaucher.de
|
||||
secretName: hosting-haumdaucher-de-tls
|
||||
---
|
||||
# Protected Ingress for /family (Family Dashboards, Nutrition & Sleep Analytics)
|
||||
# Protected by Authentik ForwardAuth
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: minio-hosting-family
|
||||
namespace: minio
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: "traefik"
|
||||
cert-manager.io/cluster-issuer: "letsencrypt-prod"
|
||||
kubernetes.io/tls-acme: "true"
|
||||
traefik.ingress.kubernetes.io/router.priority: "100"
|
||||
traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd,minio-hosting-add-prefix@kubernetescrd
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: hosting.haumdaucher.de
|
||||
http:
|
||||
paths:
|
||||
- path: /family
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: minio
|
||||
port:
|
||||
number: 9000
|
||||
tls:
|
||||
- hosts:
|
||||
- hosting.haumdaucher.de
|
||||
secretName: hosting-haumdaucher-de-tls
|
||||
|
||||
@@ -40,7 +40,7 @@ The multi-container pod structure is managed natively by the operator controller
|
||||
- 🏡🎬 `agents/grabengasse_media.md` (Topic 164 "Grabengasse & Media" — Home Assistant, Hue, Sonos, Brother Drucker, Jellyfin & Spotify)
|
||||
* **Inter-Agent Peer Escalation (A2A)**: Every subagent topic is configured with peer escalation authority via `sessions_spawn(agentId="<target>", task="...")` to consult sibling agents when cross-domain expertise is needed.
|
||||
* **Modifying & Adding Agents**: To update prompt logic, tone, or tool instructions for any agent, edit its markdown file directly in `knowledge-wiki/agents/<name>.md` and commit to git. No Kubernetes pod restarts required!
|
||||
* **Multi-Turn Domain Stickiness**: When a conversation spans multiple turns in a subagent domain (e.g. refining a recipe or planning a trip), the Conductor detects domain continuity and routes to the same agent with the state bridge `<!-- DOMAIN_STATE -->`.
|
||||
* **Multi-Turn Domain Stickiness**: When a conversation spans multiple turns in a subagent domain (e.g. refining a recipe or planning a trip), the Conductor detects domain continuity and routes to the same agent with clean conversational context.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ The deployment is managed by the official **OpenClaw Operator** (`openclaw-opera
|
||||
* **Multi-Agent Specialist Delegation (ADR 0001)**:
|
||||
* **Conductor + 9 Subagents**: Lean front-of-house coordinator orchestrates 9 specialized domain subagents defined as code in the knowledge wiki (`agents/*.md`).
|
||||
* **Direct Fast Path (< 2s)**: Simple status, home automation toggles, and single calendar entries run directly in-place.
|
||||
* **Domain Stickiness**: Multi-turn dialogue (e.g. step-by-step cooking or vacation planning) stays pinned to the active subagent with state carried over via `<!-- DOMAIN_STATE -->`.
|
||||
* **Domain Stickiness**: Multi-turn dialogue (e.g. step-by-step cooking or vacation planning) stays pinned to the active subagent with persistent session continuity.
|
||||
* **Active MCP Servers**: `stadtrat` (ALLRIS Net), `supermarket` (Marktguru), `geizhals` (price histories), `ebay`, `garmin` (148 health tools), `bring` (4 lists), `spotify`, `calendar` (family + city), `jelly` (Jellyfin/Jellyseerr), `homeassistant`, `prometheus`, `gitea`, `weather`.
|
||||
|
||||
---
|
||||
|
||||
@@ -25,6 +25,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
| **2026-10-05** | **CronJob & Alerting** | `openclaw-wiki-audit` CronJob failing daily with exit code 1; Prometheus alert `WikiAuditJobFailed` firing | `RESOLVED` | Appended `\|\| true` to `okf_lint.py` in `cronjob-wiki-audit.secret.yaml`, preventing `set -e` abort when lint findings exist. |
|
||||
| **2026-10-05** | **Cron Trigger State** | `rennerhof-website-watch` (Regensburg Topic 116) stuck in `status: error` with `ReferenceError: state is not defined` | `RESOLVED` | Cleared stale error state from `state_json` in `openclaw.sqlite` and triggered successful verification run. |
|
||||
| **2026-10-05** | **State Deadlock & Session Corruption** | Telegram Topic 117 (`#Recherche`) wedged with `restart recovery claim changed` and `entry_valid = 0` after crash | `RESOLVED` | Repaired `session_nodes` and `session_windows` (restored `entry_valid = 1`, synced `updated_at`, cleared dead claim), purged 20 obsolete cron sessions (>498k tokens), and added `--max-lines 200` compaction fallback. |
|
||||
| **2026-10-06** | **Message Formatting & UX** | Subagents emitted raw HTML/XML comments (`<!-- DOMAIN_STATE: { ... } -->`) at the end of Telegram messages | `RESOLVED` | Removed obsolete `DOMAIN_STATE` State Bridge schema from all subagent prompts and Conductor config; replaced with clean Markdown output guardrails. |
|
||||
|
||||
---
|
||||
|
||||
@@ -435,3 +436,39 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
- **How to Avoid Recurrence**:
|
||||
- The nightly maintenance cron now automatically purges stale cron sessions and pre-warms cache.
|
||||
- If a session ever reports `invalid persisted session row requires repair`, check `entry_valid == 1` and ensure `row.updated_at == record.updatedAt`.
|
||||
|
||||
---
|
||||
|
||||
### Issue 15: Subagents Emitted Raw HTML/XML Comments (`<!-- DOMAIN_STATE: { ... } -->`) in Telegram Messages
|
||||
|
||||
- **Date**: 2026-10-06
|
||||
- **Severity**: Medium (Visual/UX defect: raw HTML comments and JSON leaked into user-facing chat messages)
|
||||
- **Symptom**:
|
||||
Messages sent by specialist subagents in Telegram forum topics (`#Mila`, `#Medizin`, `#Shopping`, `#Travel`, `#Sport`, `#Regensburg`, etc.) consistently concluded with a raw HTML/XML comment block, for example:
|
||||
```html
|
||||
<!-- DOMAIN_STATE: {
|
||||
"agent": "doktor",
|
||||
"topic_category": "symptom_check",
|
||||
"evidence_source": "AWMF",
|
||||
"triage_urgency": "routine"
|
||||
} -->
|
||||
```
|
||||
- **Root Cause**:
|
||||
1. **Prompt Directive**: In `knowledge-wiki/agents/*.md`, each subagent prompt contained a `## 🔄 State Bridge Schema` instructing the LLM: *„At the end of your response, output a compact state comment: `<!-- DOMAIN_STATE: { ... } -->`“*.
|
||||
2. **Telegram Formatting Mismatch**: The schema was originally conceived in ADR 0001 under the assumption that `<!-- ... -->` would be invisible in chat rendering. However, Telegram's Bot API does not support hidden HTML comments. OpenClaw's text formatter safely escaped them to `<!-- ... -->`, causing the Telegram client to render the comments literally as visible text.
|
||||
3. **Architectural Redundancy**: Subagents operate inside persistent SQLite sessions with full conversation history and vector embeddings. Furthermore, structured domain states (sleep, nutrition, feeding, workouts) were transitioned to DuckDB/JSONL via `skills/datastore` (ADR 0004). No system component or code read `DOMAIN_STATE`.
|
||||
- **Resolution**:
|
||||
1. Removed `## 🔄 State Bridge Schema` from all 9 subagent definition files in `knowledge-wiki/agents/*.md` (`hebamme`, `doktor`, `shopping`, `travel`, `fitness`, `regensburg`, `recherche`, `itguy`, `grabengasse_media`, and `README.md`).
|
||||
2. Replaced the directive with an explicit clean-output guardrail:
|
||||
```markdown
|
||||
## 💬 Output Formatting & Telegram Guardrails
|
||||
- Output clean Markdown only for Telegram.
|
||||
- NEVER append machine-readable metadata, HTML comments (`<!-- ... -->`), XML tags, or raw JSON payloads to your chat responses.
|
||||
```
|
||||
3. Removed the `Pass State Bridge (<!-- DOMAIN_STATE -->)` clause from the Conductor system prompt in `k8s/openclaw/openclaw-instance.secret.yaml`.
|
||||
4. Pushed updates to `knowledge-wiki` git repository, pulled latest into `/home/node/.openclaw/wiki/main`, and synchronized live workspace files in `/home/node/.openclaw/workspace-*/AGENTS.md`.
|
||||
5. Applied updated `openclaw-instance.secret.yaml` to Kubernetes.
|
||||
- **How to Avoid Recurrence**:
|
||||
- Never instruct LLMs to embed machine-readable metadata or pseudo-comments (`<!-- ... -->`) in user-facing messenger outputs.
|
||||
- Rely on persistent session context and structured stores (`skills/datastore`) for state retention.
|
||||
|
||||
|
||||
Binary file not shown.
@@ -63,6 +63,7 @@ def prune_stale_cron_sessions():
|
||||
agent = db.split("/")[5]
|
||||
try:
|
||||
conn = sqlite3.connect(db, timeout=30.0)
|
||||
conn.execute("PRAGMA foreign_keys = ON;")
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type=\"table\"")
|
||||
all_tables = set(r[0] for r in cursor.fetchall())
|
||||
@@ -82,7 +83,7 @@ def prune_stale_cron_sessions():
|
||||
if stale:
|
||||
tokens_reclaimed = sum(t[2] for t in stale)
|
||||
for k, sid, _ in stale:
|
||||
for t in ["transcript_events", "transcript_event_identities", "session_transcript_active_events", "transcript_rewrite_watermarks", "session_transcript_index_state", "session_transcript_fts", "session_conversations"]:
|
||||
for t in ["trajectory_runtime_events", "transcript_events", "transcript_event_identities", "session_transcript_active_events", "transcript_rewrite_watermarks", "session_transcript_index_state", "session_transcript_fts", "session_conversations"]:
|
||||
if t in all_tables and sid:
|
||||
cursor.execute(f"DELETE FROM {t} WHERE session_id = ?", (sid,))
|
||||
for t in ["session_participants", "heartbeat_outcomes", "session_progress_cards", "board_widgets", "board_tabs"]:
|
||||
@@ -90,25 +91,39 @@ def prune_stale_cron_sessions():
|
||||
cursor.execute(f"DELETE FROM {t} WHERE session_key = ?", (k,))
|
||||
if "memory_entry_origins" in all_tables:
|
||||
cursor.execute("DELETE FROM memory_entry_origins WHERE session_key = ? OR session_id = ?", (k, sid or ""))
|
||||
if "session_windows" in all_tables:
|
||||
cursor.execute("DELETE FROM session_windows WHERE session_key = ? OR session_id = ?", (k, sid or ""))
|
||||
cursor.execute("DELETE FROM session_nodes WHERE session_key = ?", (k,))
|
||||
|
||||
# Resolve any dangling foreign key references
|
||||
while True:
|
||||
cursor.execute("PRAGMA foreign_key_check;")
|
||||
violations = cursor.fetchall()
|
||||
if not violations:
|
||||
break
|
||||
for table, rowid, parent, fkid in violations:
|
||||
cursor.execute(f"DELETE FROM {table} WHERE rowid = ?", (rowid,))
|
||||
|
||||
conn.commit()
|
||||
total_pruned += len(stale)
|
||||
total_tokens_reclaimed += tokens_reclaimed
|
||||
print(f" [{agent}] Pruned {len(stale)} dead cron sessions ({tokens_reclaimed} tokens reclaimed)")
|
||||
|
||||
# Unconditionally resolve any dangling foreign key references across all tables
|
||||
while True:
|
||||
cursor.execute("PRAGMA foreign_key_check;")
|
||||
violations = cursor.fetchall()
|
||||
if not violations:
|
||||
break
|
||||
for table, rowid, parent, fkid in violations:
|
||||
cursor.execute(f"DELETE FROM {table} WHERE rowid = ?", (rowid,))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
except Exception as e:
|
||||
print(f" [{agent}] Pruning error: {e}")
|
||||
|
||||
# Clear resolved quarantine entries if all databases are healthy
|
||||
try:
|
||||
q_path = "/home/node/.openclaw/state/openclaw-quarantine.sqlite"
|
||||
if os.path.exists(q_path):
|
||||
q_conn = sqlite3.connect(q_path)
|
||||
q_cur = q_conn.cursor()
|
||||
q_cur.execute("DELETE FROM quarantined_databases")
|
||||
q_conn.commit()
|
||||
q_conn.close()
|
||||
except Exception as e:
|
||||
print(f"Quarantine cleanup error: {e}")
|
||||
|
||||
print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.")
|
||||
|
||||
def compact_active_sessions():
|
||||
|
||||
Executable
+62
@@ -0,0 +1,62 @@
|
||||
#!/usr/bin/env bash
|
||||
# ==============================================================================
|
||||
# setup-garmin-cron.sh
|
||||
# Declaratively registers the Garmin Connect Nightly Sync in OpenClaw.
|
||||
# Schedule: Daily at 03:30 Europe/Berlin (cron: 30 3 * * *)
|
||||
# ==============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
NAMESPACE="${NAMESPACE:-openclaw}"
|
||||
POD_NAME="${POD_NAME:-openclaw-0}"
|
||||
CONTAINER_NAME="${CONTAINER_NAME:-openclaw}"
|
||||
|
||||
JOB_NAME="garmin-nightly-mirror"
|
||||
CRON_SCHEDULE="30 3 * * *"
|
||||
TIMEZONE="Europe/Berlin"
|
||||
TELEGRAM_GROUP="-1004326921510"
|
||||
TELEGRAM_TOPIC="20"
|
||||
|
||||
echo "=== Registering Garmin Connect Nightly Sync in OpenClaw ($POD_NAME) ==="
|
||||
|
||||
# 1. Check if job already exists
|
||||
EXISTING_ID=$(kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list 2>/dev/null | grep "$JOB_NAME" | awk '{print $1}' || true)
|
||||
|
||||
if [ -n "$EXISTING_ID" ]; then
|
||||
echo "Found existing job with ID $EXISTING_ID. Removing for idempotent recreation..."
|
||||
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron rm "$EXISTING_ID" || true
|
||||
fi
|
||||
|
||||
# 2. Add the job
|
||||
echo "Creating automation $JOB_NAME ($CRON_SCHEDULE @ $TIMEZONE)..."
|
||||
|
||||
MESSAGE=$(cat << 'EOF'
|
||||
Führe den nächtlichen Garmin Connect Sync durch.
|
||||
Vorgehen:
|
||||
1. Führe den Garmin-Sync-Befehl aus:
|
||||
uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/garmin_sync.py --days 2 --data-dir /home/node/.openclaw/wiki/main/data
|
||||
2. Aktualisiere die gerenderten Markdown-Ansichten:
|
||||
PYTHONPATH=/home/node/mop-skills/mop-skills/skills/datastore/scripts uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/render_views.py --data-dir /home/node/.openclaw/wiki/main/data --wiki-dir /home/node/.openclaw/wiki/main
|
||||
3. Git Commit & Push im Knowledge Wiki:
|
||||
cd /home/node/.openclaw/wiki/main && git add data/garmin/ entities/nutrition-log/ && git diff --staged --quiet || (git commit -m "chore(garmin): automated nightly mirror [skip ci]" && git push origin main)
|
||||
4. Finale Antwort:
|
||||
- Bei Erfolg: Gib exakt 'NO_REPLY' aus (intentional silence, kein Telegram-Spam in der Nacht).
|
||||
- Nur bei Authentifizierungsfehlern: Informiere kurz via Telegram, dass ein Garmin-Re-Auth nötig ist.
|
||||
EOF
|
||||
)
|
||||
|
||||
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \
|
||||
--name "$JOB_NAME" \
|
||||
--description "Nächtlicher deterministischer Spiegel der Garmin Connect Daten (Aktivitäten, Daily Summary, Schlaf) in Git JSONL" \
|
||||
--cron "$CRON_SCHEDULE" \
|
||||
--tz "$TIMEZONE" \
|
||||
--agent "fitness" \
|
||||
--model "openrouter/deepseek/deepseek-v4.1-flash" \
|
||||
--session "isolated" \
|
||||
--channel "telegram" \
|
||||
--to "$TELEGRAM_GROUP" \
|
||||
--thread-id "$TELEGRAM_TOPIC" \
|
||||
--message "$MESSAGE"
|
||||
|
||||
echo "=== Automation successfully registered! ==="
|
||||
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list | grep "$JOB_NAME" || true
|
||||
+1
-1
Submodule knowledge-wiki updated: 43d07dafa0...55c809dd79
Reference in New Issue
Block a user