Hardware Model Sizing

The problem

Developers often struggle to predict if a local LLM will actually run on their hardware without hitting Out-of-Memory (OOM) errors. Marketing nicknames and raw file sizes on Hugging Face are misleading because they ignore KV cache overhead, operating system requirements, and the physics of discrete vs. unified memory pools.

What it does

• Inventories local hardware to identify discrete VRAM pools and unified memory availability without relying on marketing stickers.
• Calculates real-world usable budget by subtracting OS and runtime overhead from raw hardware specs.
• Queries Hugging Face GGUF file trees to sum multi-file shards and verify actual weight sizes on disk.
• Factors in KV cache memory requirements based on your specific target context length and data type.
• Categorizes models into "fits", "offload", or "don't bother" buckets for your specific machine.

Why this beats prompting it yourself

General-purpose LLMs frequently hallucinate that you can sum the VRAM of two different GPUs or forget that MoE models require the full weight set in memory. This skill uses a deterministic calculation process and live API lookups to ensure you don't waste hours downloading a 40GB model that will OOM at 8k context.

Use cases

• Determining the largest model family and quantization level that fits a specific developer workstation.
• Validating vendor claims about running massive models on entry-level hardware before purchasing.
• Sizing a machine's model runway to see if it can handle 128k context tasks versus 8k chat.
• Auditing a multi-GPU setup to decide between tensor splitting or running separate instances.

Known limitations

Does not support cloud API model routing, image/video diffusion models, or sizing for training and LoRA fine-tuning. Assumes GGUF/llama.cpp runtime.