fix(openclaw): upgrade fitness agent model to gemini-3.8-flash and document issue 28

2026-10-12 22:04:22 +02:00
parent 6acceec701
commit d84d7b0bec
2 changed files with 30 additions and 0 deletions
@@ -906,4 +906,34 @@ This document captures fixed issues, operational quirks, and incident root-cause
  - For agents that routinely process user photos or document scans, assign dedicated vision models (`google/gemini-3.8-flash`) as the agent's primary model rather than relying on `agents.defaults.imageModel`.
  - Maintain the response guarantee in agent prompts to prevent empty turns when reasoning models encounter uncertainty.

+---
+
+### Issue 28: Topic 20 #Sport Transcript Blindness: GLM-5.3-Flash Reasoning Hallucination on Audio Inbound Attachments
+
+- **Date**: 2026-10-12
+- **Severity**: Medium (User sent 24s voice note requesting Cable Workout A evaluation from Garmin; transcript was echoed to Telegram but agent claimed no transcript arrived)
+- **Symptom**:
+  1. The user recorded a 24s voice note asking for a Garmin evaluation of "Cable Workout A" (history, frequency, progression).
+  2. The gateway transcribed the audio correctly via Whisper and echoed it back to Telegram (`messageId: 2739`).
+  3. The prompt delivered to the agent contained the full transcript: `[Audio]\nTranscript:\nIch habe gerade ein Workout gemacht, das Cable Workout A...`.
+  4. Yet after 78 tool steps and 19.5 minutes of looping, the agent (`fitness`) replied:
+     *"Moin mop 👋 Dein Sprachnachricht ist bei mir ohne Transkript angekommen – ich kann nur raten, was drin war. 🎙️"*.
+- **Root Cause**:
+  1. **GLM-5.3-Flash Multimodal Confusion**:
+     The message context included both the text transcript and the staged `.ogg` attachment in `message.media`.
+     In its internal reasoning (`thinking`), GLM-5.3-Flash hallucinated:
+     *"Voice message from Moritz. Need transcript — it's not provided this time? The context shows a voice message but no transcript. Hmm, the inbound is voice; transcript usually would be in context. Not visible. Maybe I need to transcribe."*
+     The model fixated on the audio file attachment, concluded it had to transcribe the file itself, and completely ignored the actual transcript string right below `[Audio]\nTranscript:`.
+  2. **19.5-Minute Looping Failure**:
+     The agent attempted to locate local audio tools (`which whisper ffmpeg`), searched memory files, found an unrelated October 7 note about protein powder in the wiki, and spent 78 turns cycling between file reads until giving up with a false negative.
+- **Resolution**:
+  1. **Model Upgraded to Gemini 3.8 Flash**:
+     Replaced `primary: "openrouter/z-ai/glm-5.3-flash"` with `openrouter/google/gemini-3.8-flash` (fallbacks: `glm-5.3-flash`, `deepseek-v4.1-flash`, `qwen3.8-flash`) for agent `fitness` both live via `openclaw config set` and declaratively in `k8s/openclaw/openclaw-instance.secret.yaml`. Gemini 3.8 Flash natively handles multimodal metadata without overlooking inline text transcripts.
+  2. **Garmin Workout Data Synced & Evaluated**:
+     Executed `garmin_sync.py` to mirror the latest workouts into `data/garmin/activities/2026-10.jsonl`.
+     Retrieved all 3 sessions of Moritz's `SuperSetACable-55mins` (2026-09-28, 2026-10-06, 2026-10-12) and extracted the workout progression data.
+- **How to Avoid Recurrence**:
+  - Keep interactive topic agents receiving media or voice notes on Gemini 3.8 Flash or DeepSeek V4.1 Flash rather than GLM-5.3-Flash.
+  - In agent system prompts, reinforce that `[Audio]\nTranscript:\n<text>` already contains the authoritative, finalized speech-to-text result and requires no local audio file re-processing.
+