Popper Probe
Convert fuzzy research claims into sharp, falsifiable hypothesis files through an adversarial Popperian dialogue.
Free
Works with the AI Agents you already use
Claude Code
Cursor
Codex CLI
Muse
Dots+22 moreClaude CodeCursorCodex CLIMuseDots
OpenClaw
Manus
Grok
Grok Bot+18 agentspopper-probe
Example session with this skill installed
I think LLMs perform better at reasoning tasks when the prompt is written in the style of a Socratic dialogue. I want to turn this into a testable hypothesis.
- Read your context and instructions
- Compiled the popper-probe
I'll help you sharpen that claim into a falsifiable hypothesis.
Process Summary
We've completed the Popperian intake dialogue. I have operationalized the "Socratic style" as a specific prompt template, defined the "reasoning tasks" as the Big-Bench Hard (BBH) suite, and identified a 5% accuracy delta as the falsifier.
hypothesis.md
---
slug: llm-socratic-reasoning-delta
created: 2024-05-22
status: active
falsifiability_gate: passed
literature_pass: none
---
# GPT-4o accuracy on BBH tasks increases by >=5% when using Socratic-style prompting compared to zero-shot chain-of-thought.
> ## Original framing
> "I think LLMs perform better at reasoning tasks when the prompt is written in the style of a Socratic dialogue."
## Operational restatement
GPT-4o (version 2024-05-13) will achieve at least a 5 percentage point higher aggregate score on the Big-Bench Hard (BBH) reasoning benchmark when using the 'Socratic-v1' template compared to a standard 'Zero-shot CoT' baseline.
## Falsifier(s)
- The Socratic-v1 template results in an aggregate score within ±1% of the Zero-shot CoT baseline.
- The Socratic-v1 template produces a statistically significant decrease in accuracy across >50% of individual sub-tasks.
## Test design
- **Methods:** Automated evaluation using the BBH evaluation harness.
- **Design:** Randomized trial of 250 prompts per sub-task. Temperature set to 0.0 for reproducibility.
- **Instruments:** OpenAI API (GPT-4o), Python evaluation script.
## Auxiliary assumptions
- The Socratic-v1 template does not exceed the model's context window limits.
- The BBH tasks are not already contaminated within the model's training data for these specific prompt styles.
## Distinctiveness
- **Prediction:** This hypothesis predicts that the structural framing of dialogue improves latent reasoning, whereas competing 'token-limit' theories suggest performance is purely a function of response length.
## Intake log
- 2024-05-22: Initial intake completed. User refined 'better' to '>=5% accuracy'.
Next steps
- Run
python3 scripts/validate_hypothesis.py popper-corpus/llm-socratic-reasoning-delta/hypothesis.mdto confirm schema. - Place any related research papers in
popper-corpus/llm-socratic-reasoning-delta/refs/. - Begin the experimental run using the identified BBH harness.
Connects securely to your tools. The creator never sees your data.
About this skill
Research often stalls because hypotheses are too fuzzy to test. This skill acts as a technical sparring partner, forcing you to convert empirical conjectures into sharp, falsifiable markdown files within your local corpus. It uses a structured Popperian dialogue to strip away ambiguity and define exactly what observation would prove a claim wrong.
What it does
- Operational restatement translates vague ideas into measurable, third-party verifiable claims with specific units and magnitudes.
- Falsifier identification requires you to name the exact experimental result that would force you to abandon the hypothesis.
- Test design sketching documents methods, instruments, controls, and sample size intuition for the proposed experiment.
- Auxiliary assumption logging captures the hidden requirements that must be true for your test to remain valid.
- Automated corpus management writes approved hypotheses to a local markdown directory with YAML frontmatter for future tracking.
How it works
- Preamble offers an optional state-of-the-art orientation based on your supplied papers or its own internal knowledge.
- Sharpening runs through four specific probes: restatement, falsifier, test design, and distinctiveness.
- Drafting generates a hypothesis file based on your dialogue and presents it for your review.
- Validation writes the final
.mdfile to your popper-corpus directory once you approve the draft.
Frameworks & tools
This skill produces standard Markdown with YAML frontmatter. It includes a schema validation step using a provided Python script (validate_hypothesis.py) and handles local file I/O for repository management.
Why this beats prompting it yourself
Generic LLMs tend to be overly agreeable, validating weak claims instead of challenging them. This skill is hardcoded to be adversarial, blocking the "termination path" until a concrete falsifier is named, which prevents researchers from committing to unfalsifiable work.
Use cases
- Converting a whitepaper intuition into a rigorous experimental design.
- Documenting the assumptions behind a new machine learning architecture.
- Auditing a fuzzy business claim before spending resources on A/B testing.
- Building a structured research corpus for a doctoral thesis or lab project.
Known limitations
Does not perform automated web searches unless explicitly requested in Probe 0. It cannot audit your existing corpus for contradictions or log new observations against old files.
How to use
Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill passes our trust checks below.
Security scanned
Passed our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scanned
Verified clean 7 days ago
- Free to download with an account