Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Llava Vision Assistant

    1

    Use a LLaVA vision-language model for visual question answering, image captioning, OCR and region-level grounding. Covers local inference via transformers/llama

    Free

    3 usesSecurity scanned
    Llava Vision Assistant

    Llava Vision Assistant

    Example session with this skill installed

    Show me how to use llama.cpp to get the bounding box of a 'Submit' button in screenshot.png using LLaVA.

    • Read your context and instructions
    • Compiled the llava vision assistant

    Use the llama-llava-cli with the GGUF model and mmproj file. Prompt: "Output the bounding box of the Submit button as x1,y1,x2,y2 normalized to 0-1000." Result: [450, 800, 550, 850]. Use Pillow to draw this box on your source image to verify the hit.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Integrating visual intelligence into apps often forces a choice between expensive proprietary APIs or complex, undocumented local setups. Extracting text from screenshots or identifying UI elements programmatically remains brittle without a multimodal model.

    What it does

    • Implements visual question answering (VQA) for screenshots, photos, and charts.
    • Performs OCR to transcribe text verbatim from images into raw text lines.
    • Executes region-level grounding to identify object coordinates via normalized bounding boxes.
    • Configures local inference via transformers or llama.cpp for fully offline processing.
    • Handles batch image processing for dataset pre-labeling and validation.

    Frameworks & tools

    Hugging Face Transformers, llama.cpp, GGUF, Python, Pillow, and LLaVA 1.5 checkpoints.

    Why this beats prompting it yourself

    Prompting a vision model is only half the battle. This skill provides the exact templates for local GPU/CPU deployment and the post-processing scripts needed to turn raw model output into structured data like bounding boxes and JSON lines.

    Use cases

    • Describe screenshots automatically to populate bug reports or documentation.
    • Extract structured data from legacy dashboard photos where no API exists.
    • Automate UI testing by locating buttons and elements via spatial grounding.
    • Pre-label large image datasets before manual human review.

    Known limitations

    Not suitable for pixel-level editing or high-stakes medical diagnosis. Dense document parsing is better handled by dedicated OCR engines.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse All Claude Skills.

    Not for pixel-level editing or medical-grade certainty. Dense document parsing may require dedicated OCR engines. Local 7B models require ~5-6GB RAM.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    3 uses

    Times used to date

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 3 months ago

    • Free with an Agensi account

    Permissions

    Terminal / Shell
    Listed3 months ago

    What's inside

    Frequently Asked Questions

    This skill provides the specialized code and configuration needed to run LLaVA (Large Language-and-Vision Assistant) for tasks like image captioning, spatial grounding, and OCR without relying on expensive external APIs.

    The skill is designed for local inference and is compatible with Python environments using Hugging Face Transformers or the llama.cpp framework for GGUF-based deployment.

    You will receive the implementation scripts for VQA and grounding, local deployment templates for GPU/CPU, and post-processing tools to convert model outputs into structured JSON and bounding box coordinates.

    While this skill excels at identifying UI elements and general image descriptions, it is not recommended for high-stakes medical imaging or dense, multi-page document parsing which require specialized OCR engines.

    Yes, it includes specialized templates for llama.cpp and GGUF checkpoints, allowing you to run powerful vision-language tasks fully offline on your own hardware.

    Free