diff --git a/k8s/openclaw/AGENTS.md b/k8s/openclaw/AGENTS.md index 0947ea4..183b034 100644 --- a/k8s/openclaw/AGENTS.md +++ b/k8s/openclaw/AGENTS.md @@ -57,8 +57,9 @@ To perform modifications or rotate api keys: ``` 4. **Register / Sync Automations**: Run the automation setup scripts to register OpenClaw Cron tasks: ```bash + ./k8s/openclaw/setup-garmin-cron.sh # Nightly Garmin Connect sync (Daily 03:30) + ./k8s/openclaw/setup-compaction-cron.sh # Nightly multi-agent session compaction (Daily 04:45) ./k8s/openclaw/setup-itguy-cron.sh # Weekly health audit (Sun 03:00) - ./k8s/openclaw/setup-compaction-cron.sh # Nightly multi-agent session compaction (Daily 03:30) ``` --- diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index f8b4982..49a95de 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -31,6 +31,7 @@ This document captures fixed issues, operational quirks, and incident root-cause | **2026-10-07** | **Compaction Latency & Live Stall** | Telegram `#Sport` (Topic 20) appeared unresponsive for >3 minutes after Moritz's message | `RESOLVED` | Identified 54s inline safeguard compaction + 35s prompt prep + 85s tool execution (181s total) caused by stale token metadata (`totalTokens: null`) slipping past nightly cron. Hardened `compact_sessions.py` with SQLite transcript inspection, lowered compaction threshold to 20k / 300KB, enabled `agents.defaults.compaction.notifyUser: true` for visible compaction notices on Telegram, and proactively compacted bloated sessions. | | **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. | | **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis, decoupled `recherche.md` into Modus A/B, raised memory limit to 6Gi and probe timeout to 25s, scoped extraPaths to main agent, purged 1.8GB duplicate vector DBs (94% drop), and built automated turn-replay daemon with Telegram notice. | +| **2026-10-11** | **Automation & Garmin Sync** | Garmin Connect sync stopped updating overnight; cron job missing from OpenClaw; untracked activities folder | `RESOLVED` | Identified `garmin-nightly-mirror` was never registered in OpenClaw; created missing `activities` directory, hardened `setup-garmin-cron.sh` with JSON declaration lookup and `--announce`, registered automation (Daily 03:30 Berlin), and verified full sync through 2026-10-11. | --- @@ -770,3 +771,31 @@ This document captures fixed issues, operational quirks, and incident root-cause - **How to Avoid Recurrence**: - Always declare PEP 723 script metadata on all Python utilities invoked via `uv run`. - Provide explicit tool invocations (`exec(command=...)`) in agent prompts so reasoning models never stall in thinking-only loops. + +### Issue 24: Missing Garmin Sync Cron Automation & Missing Activities Directory + +- **Date**: 2026-10-11 +- **Severity**: Medium (Automated nightly mirror of Garmin Connect metrics did not run; user suspected Garmin sync was broken) +- **Symptom**: + Moritz noticed that Garmin Connect metrics (daily total expenditure, active calories, sleep scores, and activities) were not appearing in the newly deployed health dashboard (`moritz.html`) and nutrition log for recent days, suspecting the sync cron job was broken. +- **Root Cause Isolated**: + 1. **Unregistered Cron Job**: The setup script `k8s/openclaw/setup-garmin-cron.sh` had been authored during datastore implementation but was never registered in OpenClaw's cron scheduler (`openclaw cron list` showed 22 jobs, none for Garmin). + 2. **Grep Truncation Failure**: In `setup-garmin-cron.sh`, the check `openclaw cron list | grep "$JOB_NAME"` was susceptible to CLI table truncation (e.g. `garmin-nightly-m...`), preventing reliable idempotent management. + 3. **Missing Announce Flag**: The job lacked `--announce`, preventing fallback delivery to Telegram Topic 20 if authentication errors occurred. + 4. **Missing Directory on PVC**: `garmin_sync.py` failed when saving activities because `/home/node/.openclaw/wiki/main/data/garmin/activities` was missing on the PVC. + 5. **Outdated Dashboard Pipeline Step**: `setup-garmin-cron.sh` only called `render_views.py` and omitted `update_moritz_website.py` and staging `public/`, leaving the web dashboard unrefreshed after nightly runs. +- **Resolution**: + 1. **Hardened `setup-garmin-cron.sh`**: + - Switched idempotency lookup to `openclaw cron list --json` with `--declaration-key`. + - Added `--announce` for delivery of 2FA re-auth notices to Telegram Topic 20 (`fitness`). + - Added `update_moritz_website.py` and `git add public/` to the nightly automation pipeline. + 2. **Created Activities Store**: + - Created `data/garmin/activities/` on the PVC and verified activity ingestion. + 3. **Executed Setup Script**: + - Successfully registered `garmin-nightly-mirror` (`id: ea1f35b0-9b04-4433-bd1f-7266e60292e3`, cron: `30 3 * * * @ Europe/Berlin`). + 4. **Backfilled Metrics**: + - Ran full sync for 2026-10-09, 2026-10-10, and 2026-10-11. All daily summaries, sleep logs, and activities were written to JSONL, views were re-rendered, and commits pushed to `knowledge-wiki`. +- **How to Avoid Recurrence**: + - Always register cron jobs with `--declaration-key` and check existing jobs via JSON rather than table text grep. + - Document all declarative setup scripts in `AGENTS.md` under `Register / Sync Automations`. + - Ensure all data directories (`daily`, `sleep`, `activities`) are safely created in `GarminMirrorWriter`. diff --git a/k8s/openclaw/setup-garmin-cron.sh b/k8s/openclaw/setup-garmin-cron.sh index 9437cdb..9ee2644 100755 --- a/k8s/openclaw/setup-garmin-cron.sh +++ b/k8s/openclaw/setup-garmin-cron.sh @@ -20,7 +20,7 @@ TELEGRAM_TOPIC="20" echo "=== Registering Garmin Connect Nightly Sync in OpenClaw ($POD_NAME) ===" # 1. Check if job already exists -EXISTING_ID=$(kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list 2>/dev/null | grep "$JOB_NAME" | awk '{print $1}' || true) +EXISTING_ID=$(kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list --json 2>/dev/null | python3 -c 'import sys, json; data=json.load(sys.stdin); print(next((j["id"] for j in data.get("jobs", []) if j.get("declarationKey") == "'"$JOB_NAME"'" or j.get("name") == "'"$JOB_NAME"'"), ""))' 2>/dev/null || true) if [ -n "$EXISTING_ID" ]; then echo "Found existing job with ID $EXISTING_ID. Removing for idempotent recreation..." @@ -37,16 +37,19 @@ Vorgehen: uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/garmin_sync.py --days 2 --data-dir /home/node/.openclaw/wiki/main/data 2. Aktualisiere die gerenderten Markdown-Ansichten: PYTHONPATH=/home/node/mop-skills/mop-skills/skills/datastore/scripts uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/render_views.py --data-dir /home/node/.openclaw/wiki/main/data --wiki-dir /home/node/.openclaw/wiki/main -3. Git Commit & Push im Knowledge Wiki: - cd /home/node/.openclaw/wiki/main && git add data/garmin/ entities/nutrition-log/ && git diff --staged --quiet || (git commit -m "chore(garmin): automated nightly mirror [skip ci]" && git push origin main) -4. Finale Antwort: +3. Aktualisiere das persönliche Health & Nutrition Dashboard (moritz.html): + uv run /home/node/mop-skills/mop-skills/skills/datastore/scripts/update_moritz_website.py --data-dir /home/node/.openclaw/wiki/main/data +4. Git Commit & Push im Knowledge Wiki: + cd /home/node/.openclaw/wiki/main && git add data/garmin/ entities/nutrition-log/ public/ && git diff --staged --quiet || (git commit -m "chore(garmin): automated nightly mirror [skip ci]" && git push origin main) +5. Finale Antwort: - Bei Erfolg: Gib exakt 'NO_REPLY' aus (intentional silence, kein Telegram-Spam in der Nacht). - - Nur bei Authentifizierungsfehlern: Informiere kurz via Telegram, dass ein Garmin-Re-Auth nötig ist. + - Nur bei Authentifizierungsfehlern: Informiere kurz via Telegram (Thema #Sport), dass ein Garmin-Re-Auth nötig ist, oder starte den Re-Auth-Flow. EOF ) kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \ --name "$JOB_NAME" \ + --declaration-key "$JOB_NAME" \ --description "Nächtlicher deterministischer Spiegel der Garmin Connect Daten (Aktivitäten, Daily Summary, Schlaf) in Git JSONL" \ --cron "$CRON_SCHEDULE" \ --tz "$TIMEZONE" \ @@ -56,6 +59,7 @@ kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron a --channel "telegram" \ --to "$TELEGRAM_GROUP" \ --thread-id "$TELEGRAM_TOPIC" \ + --announce \ --message "$MESSAGE" echo "=== Automation successfully registered! ===" diff --git a/knowledge-wiki b/knowledge-wiki index c7bd682..577afee 160000 --- a/knowledge-wiki +++ b/knowledge-wiki @@ -1 +1 @@ -Subproject commit c7bd682206da94f2dda044b09acfd566b70eccf4 +Subproject commit 577afeecb81c42829551c3808c9a44331ea89e64