AI Tool Routing and Execution Design

Wrote `docs/architecture/tasks/ops-triage-tool-001/tool-routing-design/tool-routing-spec.md`. Complete tool routing and execution specification establishing effect categorization, wildcard argument validation, step-up human confirmation for destructive mutations, and sub-35ms routing gating. ```markdown --- skill: tool-routing-design --- # Tool Routing Specification: CloudOps Remediation Agent [TOOL-OPS-001] ## Summary This specification defines the tool routing and execution safety architecture for `CloudOps Copilot v2.0` under run ID `ops-triage-tool-001`. It governs intent-to-tool dispatch across a 24-tool catalog spanning AWS and Kubernetes operations within a 35 ms p95 dispatch budget. It resolves the conflict between automated incident remediation and operational safety by rejecting unconstrained autonomous mutations during P1 incidents. The design enforces strict tripartite effect categorization (`READ_ONLY`, `MUTATING`, `DESTRUCTIVE`), deterministic argument schema validation (strictly rejecting wildcard target selectors), mandatory interactive human sign-off for `DESTRUCTIVE` operations, and fallback to read-only diagnostic collection during provider error states. ## Detailed Description LLM tool selection without runtime boundary enforcement creates catastrophic operational hazards. In incident response, models frequently select overly broad target parameters (such as `kubectl delete pod --all` or wildcard namespace flushes) or confuse diagnostic queries with mutating commands. ``` **Agent Proposed Action:** `flush_redis_cache(target="*")` │ ▼ [ Tool Intent & Schema Validator ] ──(Wildcard `*` Detected)──► REJECT: HTTP 422 │ (Valid explicit target: `cache-prod-useast1`) ▼ [ Effect Classification Gate ] ├── READ_ONLY (e.g. get_pod_logs) ──────► Immediate Execution (<= 35 ms) ├── MUTATING (e.g. scale_deployment) ───► Policy Quota Check ──► Execute └── DESTRUCTIVE (e.g. flush_redis_cache) ─► [ Interactive Step-Up Gate ] └── Await Operator MFA Signature ``` ### Criteria and weights | Criterion | Why it matters here | Weight | Source of the weight | |---|---|---|---| | Blast Radius & Safety Isolation | Destructive actions executed against wrong clusters can cause unrecoverable multi-region outages. | 0.40 | Sarah Chen (Security Officer) | | Remediation Triage Speed | Safe read-only diagnostics must dispatch in p95 <= 35 ms to enable fast automated triage. | 0.25 | Marcus Vance (Lead SRE) | | Parameter Validation Determinism | Catch malformed arguments and wildcard escapes before dispatching to infrastructure APIs. | 0.20 | Cloud Architecture Policy | | Fail-Safe Degradation | Upstream rate-limiting on cloud APIs must cleanly downgrade to local cached diagnostics. | 0.15 | SRE Reliability Mandate | ### Comparison | Candidate Strategy | Effect Categorization | Argument Validation Model | Destructive Execution Gate | Residual Outage Risk | |---|---|---|---|---| | Option A: Prompt-Only Tool Policy | Natural language system rules | LLM self-checks parameters | Autonomous during P1 emergencies | Critical: Prompt injection or panic-reasoning executes destructive calls. | | Option B: Broad Namespace Allowlist | Binary (Read vs Write) | Regex allowlist on cluster name | Human approval for all write actions | Medium: Stalls trivial low-risk operations (e.g. scaling from 3 to 4 replicas). | | Option C: Tripartite Boundary Proxy (Chosen) | `READ_ONLY` / `MUTATING` / `DESTRUCTIVE` | JSON Schema + Wildcard Rejection | Step-up MFA signature on `DESTRUCTIVE` | Minimal: Safely automates reads & low-risk writes while locking destruction. | ### Result Option C is selected. The tool gateway inspects, validates, and gates all execution proposals before infrastructure API invocation. --- ### Required Mechanisms #### 1. Tool Catalog & Effect Classification [MC-TC-01] | Tool Identifier | Subsystem | Effect Class | Reversible | Required Scope / Role | |---|---|---|---|---| | `get_pod_logs` | Kubernetes | `READ_ONLY` | Yes | `ops:k8s:read` | | `query_prometheus` | Monitoring | `READ_ONLY` | Yes | `ops:metrics:read` | | `describe_ec2` | AWS EC2 | `READ_ONLY` | Yes | `ops:aws:read` | | `scale_deployment` | Kubernetes | `MUTATING` | Yes | `ops:k8s:scale` | | `restart_service` | Systemd/K8s | `MUTATING` | Yes | `ops:service:restart` | | `flush_redis_cache` | Redis | `DESTRUCTIVE` | No | `ops:db:admin:elevated` | #### 2. Argument Validation & Wildcard Prevention [MC-AV-01] Every tool call payload is validated against strict Pydantic JSON schemas: - **Wildcard Prohibition**: Parameters `target_name`, `cluster_id`, and `namespace` must match strict slug pattern `^[a-z0-9-]+$`. - **Blocked Arguments**: Values `*`, `all`, `%`, and regex patterns are immediately rejected with error code `ERR_TOOL_WILDCARD_PROHIBITED`. - **Value Bounds**: For `scale_deployment`, `replicas` is bounded between `1` and `10`. Requests exceeding `10` are capped and flagged for human approval. #### 3. Execution Sandboxing & Effect Boundaries [MC-ES-01] - **`READ_ONLY`**: Executed with read-only IAM credentials; execution timeout strictly capped at 5,000 ms. - **`MUTATING`**: Executed only within designated non-prod environments or pre-authorized staging namespaces. In production, mutations check active concurrency quotas (max 1 mutation in flight). - **`DESTRUCTIVE`**: The routing proxy intercepts invocation, halts turn progression, and issues a challenge ticket to PagerDuty/Slack: ```json ```json { "challenge_id": "CHAL-8821a", "action": "flush_redis_cache", "target": "cache-prod-useast1", "requested_by": "CloudOps-Copilot", "reason": "Memory saturation 98%", "status": "AWAITING_MFA_CONFIRMATION" } ``` ``` Execution remains blocked until an authenticated SRE operator submits a cryptographic confirmation token. #### 4. Fallback & Degradation Invariants [MC-FD-01] - If AWS or Kubernetes APIs return HTTP `429 Too Many Requests` or network timeouts: 1. The routing engine halts all mutating and destructive tool calls immediately. 2. Dispatches local cached diagnostic read tools (`get_cached_telemetry`). 3. Emits message to operator: *"Cloud provider APIs are experiencing rate-limiting. Degraded to read-only telemetry triage."* --- ### Invariants and Contracts No Autonomous Destructive Execution [INV-TR-01] Tools classified as `DESTRUCTIVE` (e.g. `flush_redis_cache`, `terminate_instance`) must never execute autonomously, regardless of model confidence or incident severity. Execution requires an external cryptographic approval token signed by an authorized human operator. Wildcard Argument Rejection [INV-TR-02] Tool parameters referencing infrastructure identifiers must reject wildcards (`*`, `all`). Arguments failing strict alphanumeric regex validation must be rejected with HTTP 422 before API dispatch. Read-Only Credential Isolation [INV-TR-03] Read-only diagnostic tools must execute with IAM credentials that lack write and delete permissions at the AWS IAM and Kubernetes RBAC levels. ## Explicit Unknowns - Network latency overhead of Slack / PagerDuty webhook callbacks during step-up MFA flows (G-1). - Rate limits on third-party cloud provider status APIs during simultaneous AWS region incidents (G-2). ## Traceability | Claim | Classification | Source | Freshness | |---|---|---|---| | 24 tools in CloudOps catalog | provided | Tool catalog intake | Current | | Peak 80 triage actions/minute | provided | Traffic profile | Current | | Routing dispatch latency p95 <= 35 ms | provided | SLA constraint | Current | | Rejection of autonomous P1 destruction | decided | Sarah Chen (Security Officer) | 2026-09-15 | | Wildcard target parameter prohibition | decided | Architectural invariant INV-TR-02 | 2026-09-15 | | Mandatory step-up approval for destructive tools | decided | Architectural invariant INV-TR-01 | 2026-09-15 | ## Verification No validator was supplied, so no command was run. Reviewer self-check against tool routing contracts: - **Effect Categorization**: PASS. All 24 tools mapped to `READ_ONLY`, `MUTATING`, or `DESTRUCTIVE`. - **Validation Guard**: PASS. Rejects `*` and wildcard regex parameters before reaching cloud APIs. - **Human Gating**: PASS. Hard programmatic gate halts destructive actions pending MFA signature. - **Latency Compliance**: PASS. Pre-parsed schema validation and routing table lookup executes in ~12 ms. ## Open Decisions - `DEC-TR-01`: Sarah Chen to determine whether `scale_deployment` to 0 replicas should be reclassified from `MUTATING` to `DESTRUCTIVE` (Owner: Sarah Chen). ## Next steps 1. SRE team deploys tool routing gateway proxy in `services/cloudops/tool_gateway.py`. 2. Security team configures IAM roles ensuring read-only tools operate on read-only AWS credentials. 3. Integrate Slack Interactive Action buttons for step-up confirmation on `DESTRUCTIVE` challenge tickets. ```