Optimization-Loop

Auto-Optimize is an autonomous optimization loop for text-based resources such as agent skills, prompts, and campaign copy. Inspired by Andrej Karpathy's autoresearch methodology, it applies the same principle that works for training models — modify, measure, keep or discard, repeat — to the domain of text optimization.

The skill operates on files stored in a dedicated targets directory. Each optimization cycle makes exactly one targeted change, evaluates the result using either live metrics or simulated LLM-judge scoring, and logs the outcome to a results file. If the change improves the score, it's kept and becomes the new baseline. If it performs the same or worse, it's discarded via a hard git reset. This ensures the optimization path is always traceable, reversible, and never accumulates unverified changes.

The evaluation method adapts to the target type: skills are scored on task success rate through agent testing, prompts through simulated conversation scoring across multiple criteria, and campaigns through real click and conversion data. When live metrics are unavailable, the skill uses delegate_task with an evaluator LLM to simulate and score conversations based on per-target evaluation criteria.

Key constraints keep the process grounded: only one change per experiment, a simplicity criterion where removing code that performs equally is always kept, no bundling of multiple ideas, and changes must be large enough to matter but small enough to review. The loop runs continuously until interrupted by a human, and it never asks for permission to continue — it just optimizes.