Computer Use

Automating desktop tasks often fails because agents click blindly or keystrokes land in the wrong window. This skill provides a hardware-level interface for local machine control with a strict focus-validation loop to ensure actions only occur where intended.

What it does

• Visual perception captures full screenshots or high-speed frames for real-time visual verification.
• Text recognition extracts strings from the screen or specific regions using OCR.
• Precise input executes mouse clicks, scrolls, and keyboard sequences including hotkeys.
• Window management tracks active windows by PID and handle, allowing for explicit focus and safe closing.
• Guarded execution prevents input leakage by validating the active window handle before typing or clicking.

How it works

1. Perceive the current state by requesting a frame or OCR data to identify UI elements.
2. Target a specific window by listing active processes to acquire the correct window handle (HWND).
3. Act by sending a command with the target handle to ensure the focus has not shifted.
4. Verify the result by taking a follow-up screenshot to confirm the UI changed as expected.

Frameworks & tools

The skill interfaces with a local Python-based REST server or MCP server. It supports Windows and Linux (X11) environments, utilizing native windowing APIs for window tracking and input injection.

Why this beats prompting it yourself

Writing raw scripts for UI automation lacks a feedback loop, leading to broken flows when a window pop-up appears or focus changes. This skill implements a mandatory perception-action-verification cycle and hardware-level guards that prevent the agent from accidentally typing sensitive data into the wrong application.

Use cases

• Automating data entry between a legacy desktop application and a web browser.
• Performing end-to-end UI testing for desktop software without writing brittle scripts.
• Orchestrating cross-app workflows like opening a file, editing it, and uploading it.

Known limitations

Does not support macOS or Linux Wayland sessions. OCR performance on Linux is slower than Windows, and certain system-level hotkeys like Alt+F4 or Ctrl+Alt+Del are blocked for safety.