[FEATURE] macOS Accessibility API integration for computer use tools

[inference-gateway-maintainer]

Summary

The computer use tools (MouseClick, MouseMove, Type, Screenshot) currently interact with the UI by actually moving the mouse and typing. On macOS, the Accessibility API can read UI element positions, labels, and states without physically moving the cursor - enabling text-only models (like DeepSeek) to perform GUI tasks by reading the accessibility tree instead of relying on vision.

Acceptance Criteria

• Add a use_accessibility_api parameter (or similar) to computer use tools that, when enabled, reads the macOS accessibility tree instead of moving the mouse or typing with the keyboard.
• The accessibility API mode should return element positions, labels, and actionable states as structured text, so a non-vision model can navigate the UI by coordinates.
• Background tasks should be able to use the accessibility API for non-invasive UI inspection.
• The feature should gracefully degrade on non-macOS platforms (no-op or clear error).

Context

This is a follow-up from PR #990. The vision annotation pipeline already lets text-only models understand screen content via the annotator, but they still can't interact with the UI without vision. The macOS Accessibility API is a natural fit: it provides the same element-level information the annotator extracts from screenshots, but natively and without needing a vision model side-call.

For text-only models like DeepSeek, this would enable:

• Reading button labels and menu items
• Getting element positions for targeted clicks
• Understanding UI state without screenshot annotation
• Performing multi-step GUI workflows via background tasks

Related

• PR feat: add named frame sources and vision annotation pipeline #990 - Vision annotation pipeline
• Issue [FEATURE] Vision annotation pipeline: let cheaper/text-only models understand screen and camera frames #895 - Original feature request
• macOS Accessibility API documentation