Skip to content

LLM taps by ref instead of x/y, OCR-checked iOS labels, failed-trail report buttons - #260

Merged
handstandsam merged 1 commit into
mainfrom
upstream-2026.10.07
Oct 7, 2026
Merged

handstandsam merged 1 commit into
mainfrom
upstream-2026.10.07

Conversation

@handstandsam

Copy link
Copy Markdown
Collaborator

Summary

  • The LLM taps by ref, not x/y. tapOnPoint leaves the default core_interaction toolset; offered beside tap, models picked coordinates and missed. Recorded tapOnPoint steps still decode and replay, and a target opts back in with tools: [tapOnPoint].
  • iOS visible strings are checked against the screenshot. On macOS, report export runs Vision OCR over each iOS AXe capture, so an icon-only button's label is filed as accessibility text rather than drawn text.
  • Report: failed trails get trail and replay buttons, and a configurable "Fix this category" launcher action on analysis pages.
  • Subprocess tools keep their full JSON Schema, so nested arguments (e.g. overrides[].value) coerce to the declared type on dispatch.
  • Event diffs summarize list-shaped events by their items instead of [object Object].
  • Host-only tool check accepts a requires-host-tools.txt allowlist from repos that vendor the framework.
  • Devlogs: Trailblaze vs Maestro (TypeScript tools, subflows, Maestro MCP), and why the UI moved from Compose to a TypeScript web app; older Compose entries marked superseded.

馃 Generated with Claude Code

@handstandsam
handstandsam merged commit ec49f40 into main Oct 7, 2026
12 checks passed
@handstandsam
handstandsam deleted the upstream-2026.10.07 branch October 7, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant