Agent Tool Call Resilience Engineer

The Problem

Your agent calls an API that returns 429 (rate limited). It retries immediately. 429 again. Retries again. Ten rapid-fire retries, each burning tokens, each getting the same response. Or worse: the API never responds, the agent waits 30 seconds, retries, waits 30 seconds, repeats for 5 minutes. 200K+ tokens wasted on a dead endpoint. The agent didn't crash — it "worked" while silently burning your budget on operations that could never succeed.

What You Get

• Error classification engine — automatically categorizes HTTP responses as transient (429, 5xx, 503, connection refused) or permanent (401, 403, 400, 404) and applies the correct strategy without retrying permanent errors
• Exponential backoff with jitter — implements retry delays that grow exponentially (1s, 2s, 4s) with random jitter to prevent thundering herd effects when multiple agents retry simultaneously
• Circuit breaker pattern — after N consecutive failures, trips the circuit and skips all calls to that service for a cooldown period, then probes with a single request to check recovery
• Rate-limit awareness — parses Retry-After headers from 429 responses, pre-applies delays when approaching known rate limits, and tracks calls-per-minute per service
• Timeout management — enforces maximum wait times on every network call with timeout escalation on first retry, preventing indefinite hangs on unresponsive services
• Fallback chain orchestration — defines ordered fallbacks (primary → backup → cached data → degraded mode) so critical operations complete even when primary services are down

Why This Beats Prompting It Yourself

"Add retry logic" produces a naive retry loop that retries every error, has no backoff, and retries forever. This skill implements the production resilience patterns that SREs use — error classification, backoff with jitter, circuit breakers, and fallback chains — as enforced behaviors, not suggestions. Without it, your agent's retry strategy is "try again immediately," which is the most expensive failure mode in production agent deployments.

Example Session

You: "My agent keeps burning tokens retrying a down API"

Agent: "Resilience assessment for your API integration:

Problem identified: No error classification — agent retries 401 (permanent) the same way as 503 (transient). No backoff — retries fire at 0ms intervals. No circuit breaker — 50+ retries against a dead endpoint before giving up.

Applying resilience patterns:

1. Error Classification:

   • 4xx (except 429) → PERMANENT → do NOT retry, escalate to human
   • 429 → TRANSIENT → respect Retry-After header
   • 5xx/503/504 → TRANSIENT → retry with backoff
   • Connection refused → TRANSIENT → retry with backoff + jitter

2. Backoff Strategy:

   • Retry 1: 1.0s + up to 0.5s jitter
   • Retry 2: 2.0s + up to 1.0s jitter
   • Retry 3: 4.0s + up to 2.0s jitter (max retries: 3)

3. Circuit Breaker:

   • Trip after 5 consecutive failures
   • Cooldown: 30 seconds
   • Half-open: single probe call after cooldown
   • Fallback: cached data (max_age: 24h) while circuit is open

Expected impact: 200K+ tokens wasted on retries → <5K tokens for structured recovery."

Use Cases

• Production agents calling external APIs where rate limits and downtime are routine
• Multi-service orchestration where one service's failure shouldn't block the entire workflow
• Batch processing pipelines that make hundreds of sequential API calls
• Web scraping agents that encounter rate limits, CAPTCHAs, and temporary blocks
• CI/CD integration agents that call deployment APIs, container registries, and cloud services

Known Limitations

This skill teaches resilience patterns but does not implement them in code — it produces the behavioral specifications that the agent should follow. Circuit breaker state is session-scoped and does not persist across agent restarts. For persistent circuit breaker state, implement as a shared service (Redis-backed) rather than in-agent state. Not every tool call needs full resilience — reserve circuit breakers for critical operations; simple data fetches need only backoff.