fix(openclaw): allocate Gemini 3.8 Flash to koch agent, harden vision and tool protocols

This commit is contained in:
Moritz Graf
2026-10-12 19:52:11 +02:00
parent 393d928b19
commit a12b596053
4 changed files with 36 additions and 2 deletions
+34
View File
@@ -870,4 +870,38 @@ This document captures fixed issues, operational quirks, and incident root-cause
- Keep `keepRecentTokens: 3000` and `maxActiveTranscriptBytes: 180kb` in declarative manifests.
- Rely on deep-compaction with fallback rather than hard-resets across all topic agents.
---
### Issue 27: OpenClaw Vision Routing Bypass, GLM-5.3-Flash Image OCR Failure, and Tool-Calling Protocol Hardening
- **Date**: 2026-10-12
- **Severity**: High (Photos in Topic 21 #Kochen rejected as "blurry" despite 1080p high quality, turns halted with "The tool run finished, but no final summary was produced", tool errors in recherche/grabengasse)
- **Symptom**:
1. The user posted high-resolution photos of food packaging and nutrition labels (e.g. IronMaxx Clear Whey Isolate Zero Kirsche, 1080x1064 px).
2. Koch agent repeatedly responded that the photo was blurry and illegible.
3. Internal chain-of-thought hallucinated that the label was a small blurry thumbnail with ~4px text reading "MIRKIC" instead of "KIRSCHE".
4. The turn completed with `stopReason: "stop"` without emitting any visible text, causing OpenClaw to emit `The tool run finished, but no final summary was produced. I did not repeat any completed actions.`
5. Ad-hoc Python image manipulation failed due to missing `PIL` (Pillow).
6. Sporadic tool call errors across topics: `tool_search` wrapped in `tool_call`, `sessions_history` negative offsets, `ask_user` flat parameters, missing `googleapiclient` without `uv run`.
- **Root Cause**:
1. **OpenClaw `imageModel` Fallback Behavior**:
OpenClaw's configuration schema defines `imageModel` as: *"Optional image model used when the primary model lacks image input."*
Because `z-ai/glm-5.3-flash` advertises `input_modalities: ["text", "image", "video"]` on OpenRouter, OpenClaw considered it image-capable and never routed images to `imageModel: "openrouter/google/gemini-3.8-flash"`.
GLM-5.3-Flash's actual vision perception on dense German labels is poor, leading it to misread text, conclude it cannot read it, and finish its thinking block without generating any output text.
2. **Empty Turn Finalization Fallback**:
When a model finishes with only thinking tokens and zero output text or tool calls, OpenClaw's delivery mirror triggers `settled-finalization-fallback`.
3. **Python Runtime Deficiencies**:
Core libraries (`pillow`, `google-api-python-client`, `pypdf`) were not installed in the persistent Python interpreter at `/home/openclaw/.openclaw/.local/python/...`.
- **Resolution**:
1. **Dedicated Vision Model for Koch**:
Configured `koch`'s primary model to `openrouter/google/gemini-3.8-flash` (fallbacks: `glm-5.3-flash`, `deepseek-v4.1-flash`) both live via `openclaw config set` and in `openclaw-instance.secret.yaml`. Gemini 3.8 Flash provides SOTA OCR and dense-table extraction at $0.075 / 1M cached prompt tokens.
2. **Prompt-Hardening in `knowledge-wiki`**:
- Updated `knowledge-wiki/agents/koch.md` with strict rules: mandatory visible text output on every turn (no silent thinking stops), Vision-first protocol (transcribe all tables without claiming blurriness).
- Updated `knowledge-wiki/agents/README.md` with cross-agent tool-calling standards (`tool_search` top-level only, `sessions_history` non-negative offset + `sessionKey`, `ask_user` questions array, `uv run` for Python scripts).
3. **Pre-installed Python Libraries**:
Installed `pillow`, `google-api-python-client`, and `pypdf` into the persistent Python environment using `uv pip install --break-system-packages`, and added the command to `openclaw-instance.secret.yaml` for permanent self-healing.
- **How to Avoid Recurrence**:
- For agents that routinely process user photos or document scans, assign dedicated vision models (`google/gemini-3.8-flash`) as the agent's primary model rather than relying on `agents.defaults.imageModel`.
- Maintain the response guarantee in agent prompts to prevent empty turns when reasoning models encounter uncertainty.
Binary file not shown.