Skip to main content

​
Chat Behavior Troubleshooting Playbook

This playbook is for investigating prompt failures, wrong operations, and regressions in the site editor assistant. For background on the editing pipeline, see How It Works. For the full telemetry event reference, see Chat Telemetry Events.

​
What telemetry is captured

The orchestrator now emits structured chat telemetry events for each request:
  • received
  • forced_plan
  • plan_attempt_failed
  • plan_generated
  • plan_apply_failed
  • repair_attempt
  • repair_generated
  • result
Each event includes:
  • request id
  • session
  • requested/effective slug
  • model key/model used
  • planner source (openai, anthropic, gemini, or demo)
  • prompt hash (stable fingerprint)
  • prompt excerpt (short preview)
  • prompt length
  • intent/op types/op count (when available)
  • outcome + error category (when available)

​
Persistence

Telemetry is persisted to NDJSON so it survives restarts.
  • Default file: .data/chat-telemetry.ndjson
  • Env override: CHAT_TELEMETRY_FILE
  • Disable persistence: CHAT_TELEMETRY_PERSIST=0
  • In-memory buffer size: CHAT_TELEMETRY_LIMIT (default 500)
On startup, orchestrator loads recent telemetry entries from disk.

​
APIs for debugging

​
1) Raw telemetry stream (filtered)

GET /telemetry/chat Query params:
  • limit (default 100, max 1000)
  • session
  • phase
  • outcome
Example:

​
2) Review summary for manual test runs

GET /telemetry/chat/review Query params:
  • limit (default 300, max 2000)
  • session
Returns:
  • analyzed count
  • applied/failed counts
  • failure rate
  • failure breakdown by outcome
  • failure breakdown by reason category
  • top failed prompts (grouped by prompt hash)
  • automatic recommendations
Example:

​
UI debug mode (for screenshots)

Enable debug metadata directly in assistant response cards:
  1. Open the editor settings (the … more-options button in the chat header → Settings).
  2. Enable Developer mode, then the Debug mode toggle it reveals.
  3. Run the prompt and capture screenshot.
Each assistant card includes a Debug panel with:
  • traceId
  • promptHash
  • outcome
  • reason category
  • intent
  • opCount and ops
  • prompt excerpt
Use this panel when sharing failures so engineering can reproduce and convert them into regression tests quickly.

​
Standard workflow after manual UI testing

  1. Use a dedicated session for a test batch (for example: manual-2026-03-01-a).
  2. Run your manual prompts in UI.
  3. Pull review summary:
  1. Inspect top failures:
    • high schema_violation: normalization/repair gaps
    • high not_found: slug/block resolution gaps
    • high ambiguity: clarification prompts too weak
  2. Drill into raw events for one problematic prompt hash:
  1. Convert top failed prompts into regression tests in:
    • apps/orchestrator/src/nlp-ops.test.ts
  • Always run manual tests with an explicit session id.
  • Keep a fixed prompt suite for weekly regression checks.
  • Add a test for every new recurring failed prompt family.
  • Track failure rate trend (/telemetry/chat/review) before/after changes.

​
Quick checklist when behavior is wrong

  • Did the model fail to produce valid schema?
  • Did normalization repair aliases correctly?
  • Was deterministic fallback expected but not triggered?
  • Was the apply step blocked by not-found or no-effective-change?
  • Did clarification context leak across unrelated intents?

​
”Chat won’t edit one prop on my custom block”

The shape of this report is specific and worth recognising, because every check you would run to diagnose it passes. Symptoms, all at once:
  • POST /chat answers 200 with status: "info", an empty changes array, and a summary that names your block but describes the failure as a styling limitation.
  • POST /ops with the identical update_props patch applies instantly.
  • /blocks/manifest is correct, the block is selected, and the schema is right.
  • One prop fails and its siblings succeed. subheading works; heading does not.
That last asymmetry is the tell: a model that could not read the schema would fail on every prop, not one. Something deterministic is rewriting a name. Check the server log for the strip. The line is emitted at warn:
propName is what the planner asked for and allowedProps is what your block declares. If propName is a name from Avocado’s own catalogue (title, q, a, quote) and your block uses a different one, the planner’s output was rewritten on the way through — see below. If propName is something visual (backgroundColor, animation), the model genuinely promised a capability the block does not have, which is the case that message was written for. Two things to check on your side:
  1. Is the prop declared? The strip compares against the block’s registered schema. A prop that only exists behind .catchall() is not declared, and is stripped even though POST /ops would accept it. Declare every prop you want the chat to reach.
  2. Set CHAT_ADAPTIVE_SCHEMA_CONTEXT=1. Without it, a single-field edit is sent to the model with no block contracts at all, so it has to guess your prop names from its priors — and its priors are the built-in catalogue. See Block schema contracts.
Before 0.3.2 the normalizer also renamed heading → title on every block type not literally called Hero, and rewrote question/answer/testimonial/ review inside every list prop of every block, regardless of type. Both now ask the registry first and only rename when the block cannot take the key the planner used and can take the one it would be rewritten to. If you are pinned below 0.3.2 and hit this, upgrading is the fix.