fix(openclaw): configure gemini-3.8-flash vision model, document Issue 17, and update wiki submodule

2026-10-06 21:13:35 +02:00
parent bfa8ee19b7
commit 4e2a7120d6
4 changed files with 31 additions and 4 deletions
@@ -76,8 +76,8 @@ How do we architect the full-spectrum model selection across **all modalities**
 │                 │ │                 │   │                 │   │ 2. gemini-3-pro-│ │ flash-tts       │
 │ Vision Fallback:│ │                 │   │                 │   │    image        │ │ (FB: MiniMax 2.8│
 │ google/         │ │                 │   │                 │   │                 │ │  Speech HD)     │
-│ gemini-3.1-     │ │                 │   │                 │   │                 │ │                 │
-│ flash-image     │ │                 │   │                 │   │                 │ │                 │
+│ gemini-3.8-     │ │                 │   │                 │   │                 │ │                 │
+│ flash           │ │                 │   │                 │   │                 │ │                 │
 └─────────────────┘ └─────────────────┘   └─────────────────┘   └─────────────────┘ └─────────────────┘
 ```

@@ -110,7 +110,7 @@ How do we architect the full-spectrum model selection across **all modalities**

 ### 4. Vision Input vs. Text-to-Image Generation (Tier 4)
 * **Crucial Distinction**:
-  - **Vision Input (`imageModel`)**: Reading photos sent to Telegram (e.g. food ingredients, plants, receipts). Handled natively by `z-ai/glm-5.3-flash`, with `google/gemini-3.1-flash-image` as fallback.
+  - **Vision Input (`imageModel`)**: Reading photos sent to Telegram (e.g. food ingredients, plants, receipts). Handled by `google/gemini-3.8-flash` or `z-ai/glm-5.3-flash`, with `google/gemini-2.5-flash` as fallback.
  - **Text-to-Image Synthesis (`mediaModels.image`)**: Creating new pictures upon user request (e.g. *"Zeichne mir ein Profilbild von Octobot"*). Chat LLMs return 404 here.
 * **Image Synthesis Engine**:
  - **Primary**: `openrouter/google/gemini-3.1-flash-image` (Nano Banana 2).
@@ -510,3 +510,30 @@ This document captures fixed issues, operational quirks, and incident root-cause
  - Configure multi-tier model fallbacks per subagent so provider stalls or reasoning drops fail over seamlessly.
  - Rely on Prometheus alerting rules for instant notification of pod readiness and restart anomalies.

+---
+
+### Issue 17: Vision Multimodal Dropout in Kochen Agent & Image Model Modality Conflation
+
+- **Date**: 2026-10-06
+- **Severity**: High (Dropped meal intake turn; user received: *„The tool run finished, but no final summary was produced...“*)
+- **Symptom**:
+  Moritz sent a text meal report in Telegram Topic 21 (`#Kochen`):
+  *„Ich habe gegessen einen Tomatensalat mit 300g Tomaten, 150g körniger frischkäse, 30g balsamico Essig, 5g öl“*.
+  The bot executed multiple tool runs and finally answered:
+  *„The tool run finished, but no final summary was produced. I did not repeat any completed actions.“*
+  The intake event was never written to DuckDB or rendered in the nutrition log.
+- **Root Cause**:
+  1. **Unresolved In-Context Turn**: In the active session transcript, the earlier message containing 4 packaging photos of *körniger Frischkäse* had ended without an assistant completion. When the user sent the Tomatensalat text, the agent attempted to fulfill both requests, invoking `view_image` to inspect the 4 cached images.
+  2. **Image Model Modality Conflation**: In `agents.defaults.imageModel`, the fallback was declared as `openrouter/google/gemini-3.1-flash-image`. While named `flash-image`, this is Google's text-to-image raster generation model (`modality: text+image->text+image`), not an image understanding/vision model (`text+image->text`).
+  3. **Empty Text Completion in GLM-5.3-Flash**: When 4 high-resolution images were loaded into context, `z-ai/glm-5.3-flash` produced reasoning tokens but emitted empty whitespace text (`" "`). OpenClaw treated this as an incomplete turn and attempted failover, but the configured image fallback model could not process chat completions.
+- **Resolution**:
+  1. **Logged Intake Event**: Manually recorded the Tomatensalat into DuckDB via `datastore.py add` (485 g, 274.0 kcal, 19.3 g Eiweiß, 17.3 g KH, 12.4 g Fett) with timestamp `2026-10-06T20:43:00+02:00`.
+  2. **Re-rendered Views**: Ran `render_views.py` to update `entities/nutrition-log/moritz/2026-10.md` and HTML dashboards, then committed and pushed to `knowledge-wiki`.
+  3. **Telegram Confirmation**: Dispatched the exact confirmation message with meal macros and updated daily totals directly to Telegram Topic 21 using `openclaw message send --channel telegram --target "-1004326921510" --thread-id 21`.
+  4. **Configured Dedicated Multimodal Vision Models**: Corrected `agents.defaults.imageModel` in `k8s/openclaw/openclaw-instance.secret.yaml` and ADR 0002:
+     - Primary: `openrouter/google/gemini-3.8-flash`
+     - Fallbacks: `openrouter/google/gemini-2.5-flash`, `openrouter/z-ai/glm-5.3-flash`.
+- **How to Avoid Recurrence**:
+  - Never configure diffusion or raster generation models (`*-image` without chat/text output) as `imageModel` for visual input understanding.
+  - Employ dedicated multimodal frontier models (Gemini Flash) with proven OCR and label comprehension for vision tasks.
+