Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    trading agent evaluation suite

    1

    A deterministic evaluation suite for scoring trading agent logic, math, and policy adherence against reviewed baselines.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    trading agent evaluation suite

    trading agent evaluation suite

    Example session with this skill installed

    Analyze the results in evaluation_logs.json for the 'aggressive-growth' agent version 2.1. Compare it against the gold-standard fixture for position sizing and news-drift scenarios.

    • Read your context and instructions
    • Compiled the trading agent evaluation

    Analysis complete. I have processed 50 scenarios for agent 'aggressive-growth' v2.1. The evaluation detected a 12% regression in numeric precision for arithmetic-heavy tasks compared to v2.0, while policy adherence for unauthorized tools remained at 100%.

    Trading Agent Evaluation Scorecard

    Metric CategoryPass RateMean Absolute ErrorCritical FailuresNotes
    Position Sizing82%0.045 BTC4Errors in Kelly Criterion calc
    News Drift90%N/A22 instances of 'misleading news' bias
    Tool Policy100%N/A0No unauthorized tool calls detected
    Arithmetic76%12.40 USD7Regressed vs v2.0 baseline
    Latency (Input)N/A450ms0As reported in source logs

    Failure Taxonomy

    • Precision Error (7): Agent rounded down prematurely on high-frequency sizing prompts.
    • Sentiment Bias (2): Agent ignored 'held-out' bearish data in favor of 'misleading bullish' news injection.
    • Logic Regress (4): Version 2.1 failed cases that Version 2.0 passed in the 'synthetic-math-fixture'.

    Next steps

    1. Review the precision_error_logs to adjust the absolute tolerance settings in analyze.py.
    2. Re-run the 'News Drift' scenarios using the updated adversarial_v3 test suite.
    3. Investigate the arithmetic regression before promoting v2.1 to the paper-trading environment.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Developers lack a structured way to verify if their trading agents make sound financial decisions or follow strict policy constraints before risking capital. Manual review of execution logs is slow and prone to overlooking subtle numeric errors or unauthorized tool usage.

    What it does

    • Evaluates trading agent outputs against human-reviewed exact or numeric expectations.
    • Identifies failures in arithmetic, position sizing, and response to misleading market news.
    • Categorizes failures into a specific taxonomy for regression testing and debugging.
    • Flags unauthorized tool calls that bypass isolation boundaries.
    • Produces a comparative scorecard between different model versions or configurations.

    Why this beats prompting it yourself

    General LLM prompting often suffers from 'fluency bias,' where an agent explains its reasoning well despite getting the math or policy wrong. This skill enforces strict, deterministic scoring of structured data, ensuring that citations and numeric tolerances are validated rather than just sounding plausible.

    Use cases

    • Regression testing agent performance after updating underlying LLM versions.
    • Benchmarking position sizing logic against a set of verified arithmetic fixtures.
    • Detecting policy violations in agents exposed to adversarial 'misleading news' prompts.
    • Generating a failure taxonomy for audits before moving from paper trading to production.

    Known limitations

    Does not include a model runner, broker adapter, or semantic grading engine. Requires manual mapping of external logs to the specific input contract.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Trading & Investing.

    Requires logs to follow a specific input contract. Does not perform semantic grading, latency measurement, or live execution. Restricted to offline analysis only.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Software
    Python

    Permissions

    Terminal / Shell
    Listed1 month ago

    What's inside

    Frequently Asked Questions

    It scores recorded agent answers and tool calls against supplied exact or numeric expectations. It can flag answer mismatches and unauthorized tools, but it does not prove that an agent is safe, compliant, profitable, or ready for live capital.

    The scorer is model- and framework-agnostic when an external runner can export cases as the documented JSON, including expected and actual answers, model version, reviewed metadata, and recorded tool calls. It does not run the agent or connect to an LLM provider.

    The package includes the skill instructions, standalone Python scoring helper, normalized JSON input contract, synthetic valid and adverse fixtures, generated example report, runtime guide, source notes, smoke-test script, and logo. It does not include a model runner, semantic grader, broker adapter, or automatic scorecard service.

    This suite does not execute trades or interface directly with brokers; it functions as a validation layer for your agent's logic. You will need to map your agent's logs to the suite's input contract to perform the evaluation.

    Yes, if you run both versions externally on the same recorded cases and provide their answers, model versions, tool calls, and latency values in the input contract. The scorer compares those supplied results; it does not execute either model or measure performance itself.

    New in Trading & Investing