diff --git a/docs/adr/0002-openclaw-model-selection-tiering-and-economics.md b/docs/adr/0002-openclaw-model-selection-tiering-and-economics.md index 15f255d..18da5cb 100644 --- a/docs/adr/0002-openclaw-model-selection-tiering-and-economics.md +++ b/docs/adr/0002-openclaw-model-selection-tiering-and-economics.md @@ -76,8 +76,8 @@ How do we architect the full-spectrum model selection across **all modalities** │ │ │ │ │ │ │ 2. gemini-3-pro-│ │ flash-tts │ │ Vision Fallback:│ │ │ │ │ │ image │ │ (FB: MiniMax 2.8│ │ google/ │ │ │ │ │ │ │ │ Speech HD) │ -│ gemini-3.1- │ │ │ │ │ │ │ │ │ -│ flash-image │ │ │ │ │ │ │ │ │ +│ gemini-3.8- │ │ │ │ │ │ │ │ │ +│ flash │ │ │ │ │ │ │ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘ ``` @@ -110,7 +110,7 @@ How do we architect the full-spectrum model selection across **all modalities** ### 4. Vision Input vs. Text-to-Image Generation (Tier 4) * **Crucial Distinction**: - - **Vision Input (`imageModel`)**: Reading photos sent to Telegram (e.g. food ingredients, plants, receipts). Handled natively by `z-ai/glm-5.3-flash`, with `google/gemini-3.1-flash-image` as fallback. + - **Vision Input (`imageModel`)**: Reading photos sent to Telegram (e.g. food ingredients, plants, receipts). Handled by `google/gemini-3.8-flash` or `z-ai/glm-5.3-flash`, with `google/gemini-2.5-flash` as fallback. - **Text-to-Image Synthesis (`mediaModels.image`)**: Creating new pictures upon user request (e.g. *"Zeichne mir ein Profilbild von Octobot"*). Chat LLMs return 404 here. * **Image Synthesis Engine**: - **Primary**: `openrouter/google/gemini-3.1-flash-image` (Nano Banana 2). diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index efbf8bd..afda9a3 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -510,3 +510,30 @@ This document captures fixed issues, operational quirks, and incident root-cause - Configure multi-tier model fallbacks per subagent so provider stalls or reasoning drops fail over seamlessly. - Rely on Prometheus alerting rules for instant notification of pod readiness and restart anomalies. +--- + +### Issue 17: Vision Multimodal Dropout in Kochen Agent & Image Model Modality Conflation + +- **Date**: 2026-10-06 +- **Severity**: High (Dropped meal intake turn; user received: *„The tool run finished, but no final summary was produced...“*) +- **Symptom**: + Moritz sent a text meal report in Telegram Topic 21 (`#Kochen`): + *„Ich habe gegessen einen Tomatensalat mit 300g Tomaten, 150g körniger frischkäse, 30g balsamico Essig, 5g öl“*. + The bot executed multiple tool runs and finally answered: + *„The tool run finished, but no final summary was produced. I did not repeat any completed actions.“* + The intake event was never written to DuckDB or rendered in the nutrition log. +- **Root Cause**: + 1. **Unresolved In-Context Turn**: In the active session transcript, the earlier message containing 4 packaging photos of *körniger Frischkäse* had ended without an assistant completion. When the user sent the Tomatensalat text, the agent attempted to fulfill both requests, invoking `view_image` to inspect the 4 cached images. + 2. **Image Model Modality Conflation**: In `agents.defaults.imageModel`, the fallback was declared as `openrouter/google/gemini-3.1-flash-image`. While named `flash-image`, this is Google's text-to-image raster generation model (`modality: text+image->text+image`), not an image understanding/vision model (`text+image->text`). + 3. **Empty Text Completion in GLM-5.3-Flash**: When 4 high-resolution images were loaded into context, `z-ai/glm-5.3-flash` produced reasoning tokens but emitted empty whitespace text (`" "`). OpenClaw treated this as an incomplete turn and attempted failover, but the configured image fallback model could not process chat completions. +- **Resolution**: + 1. **Logged Intake Event**: Manually recorded the Tomatensalat into DuckDB via `datastore.py add` (485 g, 274.0 kcal, 19.3 g Eiweiß, 17.3 g KH, 12.4 g Fett) with timestamp `2026-10-06T20:43:00+02:00`. + 2. **Re-rendered Views**: Ran `render_views.py` to update `entities/nutrition-log/moritz/2026-10.md` and HTML dashboards, then committed and pushed to `knowledge-wiki`. + 3. **Telegram Confirmation**: Dispatched the exact confirmation message with meal macros and updated daily totals directly to Telegram Topic 21 using `openclaw message send --channel telegram --target "-1004326921510" --thread-id 21`. + 4. **Configured Dedicated Multimodal Vision Models**: Corrected `agents.defaults.imageModel` in `k8s/openclaw/openclaw-instance.secret.yaml` and ADR 0002: + - Primary: `openrouter/google/gemini-3.8-flash` + - Fallbacks: `openrouter/google/gemini-2.5-flash`, `openrouter/z-ai/glm-5.3-flash`. +- **How to Avoid Recurrence**: + - Never configure diffusion or raster generation models (`*-image` without chat/text output) as `imageModel` for visual input understanding. + - Employ dedicated multimodal frontier models (Gemini Flash) with proven OCR and label comprehension for vision tasks. + diff --git a/k8s/openclaw/openclaw-instance.secret.yaml b/k8s/openclaw/openclaw-instance.secret.yaml index 8ca820f..40d71bb 100644 Binary files a/k8s/openclaw/openclaw-instance.secret.yaml and b/k8s/openclaw/openclaw-instance.secret.yaml differ diff --git a/knowledge-wiki b/knowledge-wiki index 77a051e..2642060 160000 --- a/knowledge-wiki +++ b/knowledge-wiki @@ -1 +1 @@ -Subproject commit 77a051e797b397131f6f15cb46ac854753422301 +Subproject commit 26420601349d25b2e64880c53ab94c92c8afc788