rag architect

The problem

Basic RAG implementations often fail in production due to low retrieval quality, high latency, or unpredictable costs. Developers struggle to move beyond simple tutorials to a system that handles hallucinations and scales efficiently.

What it does

• Provides a decision framework for selecting chunking strategies, embedding models, and vector databases based on scale and cost.
• Includes runnable diagnostic scripts to audit retrieval quality, profile latency by component, and project monthly costs.
• Deploy an evaluation harness for synthetic test generation and LLM-as-judge scoring to detect regressions in CI.
• Implements production patterns like query rewriting (HyDE), contextual compression, and multi-tenant namespace isolation.
• Contains debugging runbooks for fixing common issues like low recall, hallucinations, and high context costs.

Frameworks & tools

Python 3.10+, OpenAI/Anthropic APIs or local vLLM/Ollama, Redis, and vector databases including pgvector, Pinecone, Weaviate, Qdrant, and Chroma.

Why this beats prompting it yourself

General LLM prompts lack the architectural depth required for production infrastructure engineering. This skill provides a structured codebase of diagnostic scripts and evaluation gates that are faster to deploy than building a custom testing suite from scratch.

Use cases

• Optimize an existing RAG pipeline that is returning irrelevant context or hallucinating.
• Calculate the cost and latency impact of switching from OpenAI to local embedding models.
• Establish a CI/CD gate that prevents merging RAG changes if faithfulness scores drop.
• Scale a single-user RAG system to a multi-tenant architecture with per-tenant isolation.

Known limitations

Requires Python 3.10 or higher. Synthetic test generation and LLM-as-judge evaluation will incur API costs during execution.