- Home
- Skills
- Agents & Orchestration
- Qdrant Search Quality: Relevance Debugging
More screenshots
Works with the AI Agents you already use
Qdrant Search Quality: Relevance Debugging
Fix Qdrant relevance the data-first way: a diagnosis ladder (chunking, model, query, config last), a golden set with recall@k, and significance tests so wins are proven.
$5
Qdrant Search Quality: Relevance Debugging
Example session with this skill installed
Since we re-indexed our support docs with smaller chunks last month, search feels worse — users say the right article exists but never shows up. Our first instinct was to tune HNSW. Diagnose it properly.
- Read your context and instructions
- Compiled the qdrant search quality
DIAGNOSIS RUN — produced by applying the skill; HNSW explicitly not touched yet.
Step 1, failing set: 14 real queries pulled from support tickets, each with the article that should have ranked. Failure type on 11 of 14: the right article is in the index but absent from the top 10 ("missing"), not buried at rank 6–10.
Step 2, exact-match gate: searched a distinctive phrase from a known article ("resetting a stuck provisioning token"). Found instantly. Ingest is intact — the ladder proceeds; this is a ranking/chunking problem, not a missing-data problem.
Step 3, rung 1 — chunking: read 20 random chunks from the re-indexed collection. Finding: the new splitter cut at 180 tokens with no overlap and no regard for sentence boundaries; 13 of 20 sampled chunks begin mid-sentence, and several procedure articles are split so the question ("how do I reset…") sits in one chunk and the steps in the next. Neither chunk alone matches a how-to query well. This matches the timeline exactly: quality dropped when chunking changed, not when anything about the model or Qdrant did.
Measurement before fixing: golden set built from the failing set plus 60 curated query→article pairs (versioned v1). Baseline recall@10 with current chunking: 0.57. Single change tested: re-chunk at 350 tokens, sentence-aware splitter, 15% overlap, titles and section headers added as payload. Recall@10 on the same golden set: 0.79 (paired permutation test, 95% CI [0.74, 0.84], p < 0.01). Precision@5 also rose, so the win isn't recall bought with junk.
Verdict: ship the re-chunking; the embedding model rung never needed testing, and the HNSW tuning the team wanted to start with would have multiplied a broken pipeline. Next candidate after this lands: hybrid fusion for the exact-term queries (error codes) still failing in the set.
Connects securely to your tools. The creator never sees your data.
About this skill
Most "vector search is bad" complaints are not vector search problems — and the standard response, a week of HNSW tuning, fixes none of them. This skill walks a diagnosis ladder in the order problems actually occur: chunking first, then the embedding model, then the query side, and Qdrant configuration last, as the multiplier it is rather than the repair it isn't. It starts with the standing rule — reproduce before you tune — and a failing set of 10–20 real queries with the results that should have come back, because if you can't name the expected results you have an undefined problem, not a quality problem. An exact-match sanity check gates the whole ladder: if a known phrase can't be found, ingest is broken and tuning stops. Each rung names its specific checks — mid-sentence chunk splits that can cost 30–40% of retrieval quality, chunk size against question size, metadata poverty, duplicates; domain mismatch, quantization damage, mixed-model indexes after a swap, multilingual gaps; vague queries, filters-first, query embedding care — and only then ef, m, and quantization, tuned by sweeping one parameter against recall and p95 latency on a frozen snapshot. Evaluation gets its own discipline: a human-curated, versioned golden set of 50–200 query-to-document pairs, recall@k as the primary metric with precision/nDCG and MRR for their failure shapes, k chosen from what the product actually consumes, and a paired significance test with a confidence interval before any improvement is declared — a 3-point gain on 50 queries is noise, and the skill says so. Strategy upgrades come last and in effort order: hybrid keyword+dense with RRF fusion, cross-encoder reranking, MMR for diversity, relevance feedback. A symptom-to-rung decision table, a failing-set record format, golden-set rules, and a fix-report template are included, plus a worked example where a model swap dropped recall@10 from 0.74 to 0.58 and the fix that shipped (model B + hybrid, 0.83, p < 0.01) never touched the database config. Edge cases cover tiny corpora, multi-tenancy, drifting collections, and contaminated feedback loops.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Agents & Orchestration.
- The diagnosis needs expected results: someone who knows the corpus must help build the failing and golden sets. Without that, the skill stops at "undefined problem" — by design.
- Worked numbers come from document-search workloads; code search and product catalogs have different chunking and metric profiles, though the ladder order holds.
- Significance testing needs a golden set of reasonable size; with under ~30 pairs the skill will report directional results and say the evidence is thin.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Permissions
No special permissions declared or detected