AI Research Engineering Playbooks

AI research often stalls on engineering overhead rather than scientific insight. This skill library provides disciplined workflows to transform general coding agents into practical research engineers. It replaces vague prompting with structured playbooks for data preparation, experiment execution, and rigorous evaluation.

What it does

• Standardizes experiments by providing reusable playbooks for data cleaning, training configurations, and benchmarking.
• Ensures reproducibility through mandatory dataset fingerprinting, config snapshots, and environment logging.
• Optimizes training with specific patterns for LoRA fine-tuning, parameter-efficient adapters, and checkpoint management.
• Automates evaluation via benchmark protocols that calculate exact match, edit distance, and statistical significance.
• Diagnoses failures using mechanistic diagnostic tools to hook into activations and identify layers causing hallucinations or refusals.
• Generates reports by turning raw metrics and logs into paper-style summaries with ablation studies and error analysis.

How it works

1. Identify the specific research question, such as fine-tuning or inference tuning.
2. Select the corresponding skill module from categories like data curation, training, or safety.
3. Execute a dry run on a small data subset to verify the configuration and environment.
4. Run the full experiment using provided Bash and YAML templates for consistent artifact logging.

Frameworks & tools

Built for Python-based ML workflows using Hugging Face PEFT, safetensors, vLLM for serving, and TransformerLens for mechanistic interpretability.

Why this beats prompting it yourself

Standard prompting often ignores reproducibility and data hygiene, leading to "hallucinated" results. This skill enforces a rigorous engineering loop that requires metric verification and failure analysis before declaring success.

Use cases

• Domain adaptation via LoRA fine-tuning for specific vertical tasks like finance or legal QA.
• Performance profiling to identify bottlenecks in LLM serving latency and throughput.
• Safety red-teaming to detect jailbreaks and calibrate refusal behavior.
• Data auditing to detect contamination and leakage across training and evaluation splits.

Known limitations

Does not automate the creation of novel scientific theory. Requires an environment with pre-installed GPU drivers and ML libraries like PyTorch or JAX to execute playbooks.