Video Analyzer

video-analyzer (Claude Skill)

Converts video files into structured Markdown and JSON reports using local preprocessing and Claude Vision.

This Claude Skill acts as a bridge for LLM environments (such as Claude Projects) that do not natively support direct video uploads. It extracts audio transcripts, identifies key visual transitions, groups frames by color composition, and structures this data into AI-friendly and human-readable formats.

Architectural Overview

The skill operates through a decoupled multi-stage pipeline designed to minimize API token usage while capturing critical structural and textual details:

1. Input Video Processing: The source file is ingested by the preprocessor script to decouple audio and visual data.
2. Audio Analysis: The extracted track is converted and transcribed locally using a high-efficiency Whisper engine.
3. Frame Selection: Visual frames are extracted at a scaled resolution and clustered based on color metrics to remove redundancy.
4. Multimodal Synthesis: Claude Vision analyzes the optimized frame selection to track character data and visual state.
5. Report Generation: A final compilation stage bundles metadata, timestamps, transcriptions, and visual observations into unified JSON and Markdown outputs.

Technical Specifications & Pipeline Stages

Stage 1: Extraction & Preprocessing (process_video.py)

The pipeline begins by isolating the audio track and extracting downscaled video frames.

• Frame Extraction: Frames are extracted at a configurable rate (default: 0.5 fps, or one frame every 2 seconds) and capped at a maximum count (default: 30 frames) to keep processing efficient.
• Resolution Scaling: Frames are scaled so that their longest edge is at most 1568px while preserving the original aspect ratio. This keeps the visual token budget predictable (approximately 1,400 tokens per frame).
• Audio Extraction & Transcription: The audio track is resampled to mono 16kHz. It is then transcribed locally using OpenAI Whisper (defaulting to the highly efficient large-v3-turbo model). The engine supports both