Skip to content

Fix verify hint listing LLM-hidden assertion tools - #261

Open
pengdev wants to merge 1 commit into
block:mainfrom
pengdev:fix/verify-hint-hidden-tools
Open

pengdev wants to merge 1 commit into
block:mainfrom
pengdev:fix/verify-hint-hidden-tools

Conversation

@pengdev

@pengdev pengdev commented Oct 9, 2026

Copy link
Copy Markdown

Summary

trailblaze verify's LLM hint told the model it MUST call one of assertVisibleBySelector, assertVisibleWithText, or assertWithAI to confirm an assertion — but the first two are registered @TrailblazeToolClass(surfaceToLlm = false, ...) (one's own source comment reads "DO NOT USE GIVE THIS TOOL TO THE LLM"; the other is @Deprecated). The model can never actually see or call either one.

Result: verify reliably returned exit 1 (FAILED) regardless of what was actually on screen — reproduced on a real Android emulator against claude-sonnet-5, where the failure reasoning showed the model hunting for a tool it could never find, even while correctly describing a genuinely healthy render in the same response.

This swaps the two unreachable names for the two tools actually surfaced to the LLM today — assertVisible and assertWithAi (also fixes a casing typo, "assertWithAI" → "assertWithAi") — and drops the now-stale Oxford comma from the original 3-item list.

Verification

  • Confirmed via @TrailblazeToolClass annotations in trailblaze-common that assertVisibleBySelector/assertVisibleWithText are hidden (surfaceToLlm = false) and assertVisible/assertWithAi are not.
  • Confirmed assertVisible is genuinely usable in verify mode and still produces a recordable trail (delegates internally to the recordable assertVisibleBySelector), via existing tests StepToolSetVerifyModeTest and StepToolSetDirectToolsTest.
  • Swept the repo for any other LLM-facing hint/prompt referencing a hidden tool — found none; this was the only one.
  • Built from source (scripts/install-trailblaze-source.sh) and re-ran the same verify assertion against a live Android emulator before and after the fix:
    • Before: exit 1 on both a healthy map render and a genuine negative (Home screen, no map) — same failure both times, reasoning stuck on the nonexistent tool.
    • After: exit 0 on the healthy render ("not blank" correctly confirmed), exit 1 on the negative — this time for the correct reason ("home launcher, no map"), no tool-hallucination in the reasoning.

Test plan

  • Read-only source verification of tool visibility annotations
  • Existing unit tests (StepToolSetVerifyModeTest, StepToolSetDirectToolsTest) confirm assertVisible behaves correctly in verify mode
  • Live end-to-end re-test on an Android emulator with a real Anthropic key, both before and after the fix, both a healthy and a broken render

@pengdev
pengdev force-pushed the fix/verify-hint-hidden-tools branch from 25cb0ad to a40bcfc Compare October 9, 2026 14:49
The verify step's hint told the model it MUST call one of
assertVisibleBySelector, assertVisibleWithText, or assertWithAI, but the
first two are registered with surfaceToLlm = false (one's source comment
reads "DO NOT USE GIVE THIS TOOL TO THE LLM"; the other is deprecated).
The model can't see or call either one, so it got stuck looking for a
nonexistent tool and reported FAILED regardless of what was actually on
screen, including on a genuinely correct render.

Swap the hint to name the two tools actually surfaced to the LLM today:
assertVisible and assertWithAi (also fixes the casing on the latter).

Verified against a real Android emulator with claude-sonnet-5 both before
and after: before the fix, verify returned exit 1 on both a healthy
render and a real negative (Home screen) with a hallucinated-tool
confusion in its own reasoning; after the fix, it returns exit 0 on the
healthy render and exit 1 on the negative, for the correct reason.

Signed-off-by: Peng Liu <liupengking1@gmail.com>
@pengdev
pengdev force-pushed the fix/verify-hint-hidden-tools branch from a40bcfc to 3c9d148 Compare October 9, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant