Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    desktop vision controller

    1

    Automate Windows UI interactions using visual capture, OCR, and coordinate-based clicks with safety confirmations.

    Free

    3 usesSecurity scanned
    desktop vision controller

    desktop vision controller

    Example session with this skill installed

    Capture the window titled 'Calculator', find the number '8' using OCR ( no incluyed ) , and click it. I confirm this action.

    • Read your context and instructions
    • Compiled the desktop vision controller

    I have successfully captured the Calculator window, identified the coordinates for the '8' button, and performed the click as requested.

    Desktop Action Log

    StepActionTargetResultCoordinates/Path
    1capturewindow:CalculatorSuccesscaptures/calc_snap_17158200.png
    2ocrcaptures/calc_snap_17158200.pngText Found"8" detected at 120, 450
    3click_at120,450SuccessUser Confirmed: True

    Next steps

    • Verify if the Calculator display now shows the number '8'.
    • Specify the next digit or operator you want to click.
    • Use type_text if you need to input a full equation string instead of individual clicks.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Automating desktop applications without accessible APIs or DOM elements is slow and error-prone. Developers struggle to bridge the gap between AI reasoning and physical UI interaction on Windows systems.

    What it does

    • Captures high-resolution screenshots of the full desktop, specific windows, or defined coordinate regions.
    • Performs OCR on captured images to identify text labels and UI elements for interaction.
    • Executes precise mouse clicks and text input based on pixel coordinates or text labels.
    • Provides a list of active window titles to target specific applications for automation.

    Frameworks & tools

    Windows Desktop Environment, MCP (Model Context Protocol), desktop_vision_controller MCP server.

    Why this beats prompting it yourself

    Generic prompts often hallucinate UI coordinates or fail to handle the safety implications of OS-level actions. This skill enforces a structured schema, mandatory user confirmation for clicks/typing, and standardizes error handling for offline desktop automation.

    Use cases

    • Automating data entry into legacy Windows software that lacks an API.
    • Performing visual regression testing or UI audits on desktop applications.
    • Building AI-driven tutorials that interact with real software in real-time.
    • Extracting text from images or non-selectable UI components via OCR.

    Known limitations

    Requires the desktop_vision_controller MCP server to be installed and running. Text input must not be used for sensitive passwords.

    Requires the desktop_vision_controller MCP server. Offline execution only. UI actions require explicit user confirmation via a boolean flag. Not suitable for password handling.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    3 uses

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 2 months ago

    • Free with an Agensi account

    Permissions

    No special permissions declared or detected

    Listed2 months ago

    What's inside

    Frequently Asked Questions

    This skill enables AI agents to interact with Windows applications by taking screenshots, reading on-screen text via OCR, and executing precise mouse clicks or keystrokes, effectively bridging the gap between LLM reasoning and physical UI actions.

    This skill requires the desktop_vision_controller MCP (Model Context Protocol) server to be installed on your local Windows machine to bridge the communication between your AI agent and the operating system.

    Yes, as a safety measure, this skill enforces a structured schema that requires manual user confirmation before the agent executes any clicks or text input, ensuring you maintain control over OS-level actions.

    While it can read text from any application using OCR, it is currently optimized for Windows environments and requires the host system to be running the dedicated MCP server for screen capture and input simulation.

    You will receive the skill definition and configuration files needed to integrate the Desktop Vision Controller into your MCP-compatible AI client, along with documentation for the required local server setup.