local llm selection

Running a local LLM is rarely just about saving money on API credits. It introduces operational overhead, hardware constraints, and potential quality trade-offs that can stall a project. This skill provides a technical framework for evaluating whether to stick with a hosted API or migrate to a local stack like Ollama, llama.cpp, or vLLM.

What it does

• Local vs hosted evaluation compares privacy, volume, and latency requirements to determine the best deployment strategy.
• VRAM sizing calculates model and quantization requirements against hardware limits to prevent out-of-memory errors.
• Quantization selection weighs the trade-offs between FP16, Q8, Q5, and Q4 formats to balance speed and quality.
• Fine-tuning triage determines if a project requires QLoRA or if RAG and advanced prompting are sufficient.
• Architecture planning defines how to implement env-gated provider paths for swappable local/hosted backends.

How it works

1. Define the workload requirements including data privacy needs, expected volume, and hardware availability.
2. Calculate VRAM overhead by multiplying parameters by bytes-per-weight and adding headroom for KV cache.
3. Compare model specialized variants like choosing a code-tuned model over a general chat model for dev tasks.
4. Draft a provider-agnostic implementation using a clean architecture pattern to gate local models behind environment flags.

Frameworks & tools

The skill focuses on local inference engines like Ollama, llama.cpp, and vLLM. It handles quantization formats including GGUF and EXL2, and covers fine-tuning workflows involving QLoRA and PEFT.

Why this beats prompting it yourself

Generic LLMs often provide overly optimistic advice about local hosting or fail to account for the specific VRAM footprint of KV caches. This skill enforces a "hosted-first" production bias and requires rigorous cost-complexity math before recommending hardware-bound solutions.

Use cases

• Deciding if a high-volume data pipeline should migrate from OpenAI to local vLLM nodes.
• Selecting the largest possible coding model that fits on a developer's 12GB or 16GB GPU.
• Evaluating whether inconsistent JSON output justifies a QLoRA fine-tune vs improved RAG.
• Setting up a local development environment that works offline without breaking production API paths.

Known limitations

Does not provide benchmark scores for specific model versions. Hardware calculations assume standard transformer architectures and may not account for specialized NPU or Apple Silicon optimizations.