diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index 0817303..efbf8bd 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -495,8 +495,18 @@ This document captures fixed issues, operational quirks, and incident root-cause 2. **Session Normalization**: Repaired `grabengasse_media` (Topic 164) and `koch` (Topic 21), resetting `status: "done"`, clearing recovery run IDs, and restoring `entry_valid = 1`. 3. **InitContainer Self-Healing**: Injected `export PATH="/home/node/.openclaw/bin:/home/node/.openclaw/.local/bin:$PATH"` and added automated pre-flight self-healing to `initContainers[0]` in `openclaw-instance.secret.yaml`. On every container boot, the script now automatically cascades foreign key repairs, unlocks hung `running` sessions, and clears quarantine entries. 4. **Maintenance Script Hardening**: Updated `k8s/openclaw/setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` with `PRAGMA foreign_keys = ON;`, explicit deletion across child tables, and unconditional FK verification across all 12 agent databases. + 5. **Model Fallback Chain for Subagents**: Replaced bare string model configurations (`model: "openrouter/z-ai/glm-5.3-flash"`) across all 11 subagents in `k8s/openclaw/openclaw-instance.secret.yaml` with structured objects specifying explicit fallbacks (`primary: "openrouter/z-ai/glm-5.3-flash"`, `fallbacks: ["openrouter/deepseek/deepseek-v4.1-flash", "openrouter/qwen/qwen3.8-flash"]`). This prevents reasoning dropouts on GLM-5.3-Flash from halting turns and eliminates the OpenClaw Doctor configuration warning. + 6. **Koch Agent Grounding & Standard Foods Catalog**: + - Added EDEKA Bio Herzstücke Körniger Frischkäse (150 g) and Philadelphia Der Körnige (200 g) to `knowledge-wiki/entities/standard-foods.md` and `knowledge-wiki/data/entities/foods.jsonl`. + - Hardened `knowledge-wiki/agents/koch.md` with explicit distinction: capturing packaging photos / ingredients is NOT intake, never touches `entities/nutrition-log/`, and must only update `foods.jsonl` / `standard-foods.md`. Intake logging is strictly prohibited from using `edit` and must use `datastore.py add --kind nutrition.intake`. + 7. **Observability & Alerting**: Created `k8s/openclaw/monitoring.yaml` deploying a `PrometheusRule` (`openclaw-alerts`) covering: + - `OpenClawPodNotReady`: Alerts (critical) if `openclaw-0` is unready for >5 minutes. + - `OpenClawCrashLooping`: Alerts (warning) if container restarts exceed 2 within 15 minutes. + - `OpenClawDataVolumeLow`: Alerts (warning) if PVC `openclaw-data` has <15% free disk space. - **How to Avoid Recurrence**: - Always ensure `PRAGMA foreign_keys = ON;` is set on every SQLite connection performing deletes. - Rely on the initContainer self-healing hook to automatically heal state before the gateway daemon starts. - Avoid legacy Markdown editing for nutrition logs; enforce structured datastore CLI tooling (`skills/datastore`). + - Configure multi-tier model fallbacks per subagent so provider stalls or reasoning drops fail over seamlessly. + - Rely on Prometheus alerting rules for instant notification of pod readiness and restart anomalies.