=== LEGACY AUTOMATION REFACTOR PLAN === Current system: A single Python process synchronizes CRM leads with an email marketing platform and accounting system. Architecture assessment: The workflow is currently at Automation Maturity Level 1: Fragile Script. Primary risks: - business rules and integration code are tightly coupled - external API failure stops the full workflow - no standardized connector interfaces - no idempotency protection - no distinction between transient and permanent failures - no structured logging - partial completion is not recorded - no controlled replay - manual deployment creates operational risk - institutional knowledge is concentrated in one developer Target architecture: 1. CRM Source Adapter Retrieves or receives CRM lead events through a stable internal interface. 2. Lead Validation Service Validates required fields, normalizes data, and rejects incomplete records before external calls. 3. Workflow Orchestrator Coordinates each workflow step, records state, applies routing rules, and manages partial completion. 4. Email Platform Adapter Implements the standardized connector interface for the email marketing provider. 5. Accounting Platform Adapter Implements the same connector contract for the accounting provider. 6. Idempotency and Workflow State Store Records processed event IDs, completed steps, connector references, and retry state. 7. Queue and Worker Layer Controls concurrency, supports backpressure, and protects external providers from request bursts. 8. Dead-Letter and Manual Review Queue Stores permanently failed or retry-exhausted items for controlled investigation and replay. 9. Observability Layer Provides structured logs, metrics, alerts, and end-to-end correlation IDs. Standard connector contract: Input: - workflow_id - correlation_id - operation - entity_type - entity_id - normalized_payload - idempotency_key - attempt_number Output: - status - external_reference - retryable - error_code - safe_message - processed_at - connector_version Error strategy: Validation errors: - retryable: no - route: manual correction queue - severity: warning Authentication errors: - retryable: limited - route: operations escalation - severity: critical after repeated occurrence Rate-limit errors: - retryable: yes - respect Retry-After - use exponential backoff with jitter - reduce connector concurrency dynamically Temporary provider errors: - retryable: yes - maximum attempts: configurable - route to dead-letter queue after exhaustion Permanent business rejection: - retryable: no - record provider response safely - route to manual review Idempotency strategy: - derive a deterministic idempotency key from source system, lead ID, workflow version, and operation - store successful connector operations - skip previously completed steps - never repeat a successful accounting or email operation after partial failure - allow controlled replay of failed steps only Structured logging fields: - timestamp - severity - service - module - workflow_id - correlation_id - lead_id - connector - operation - status - duration_ms - retry_count - error_code - environment - component_version Required metrics: - total workflows - success rate - partial failure rate - retry rate - rate-limit events - connector latency - dead-letter count - queue depth - duplicate suppression count - manual intervention count Alert conditions: - no successful synchronization during expected interval - sustained failure rate above threshold - dead-letter queue growth - repeated authentication errors - queue backlog - provider rate-limit saturation - connector latency SLA breach Testing strategy: - unit tests for normalization and business rules - connector contract tests - integration tests with provider sandboxes or mocks - timeout tests - rate-limit tests - duplicate event tests - partial failure tests - retry exhaustion tests - dead-letter creation tests - controlled replay tests - deployment rollback test - end-to-end staging test Deployment strategy: - containerized worker and orchestrator - environment-specific configuration - secrets stored outside source code - CI quality gates - staged deployment - limited-volume pilot - progressive rollout - rollback to previous image version Migration roadmap: Phase 1: Document the current workflow and external API behavior. Phase 2: Extract provider adapters without changing business behavior. Phase 3: Introduce workflow state and idempotency. Phase 4: Add queue-based execution and bounded retries. Phase 5: Add structured logs, metrics, and alerts. Phase 6: Create dead-letter handling and controlled replay. Phase 7: Run parallel validation against the existing process. Phase 8: Deploy gradually and retire the legacy script. Acceptance criteria: - no silent failures - every workflow has a traceable state - successful steps are not repeated after partial failure - duplicate source events do not create duplicate external records - rate-limit responses do not crash the system - retries are bounded and observable - failed items can be reviewed and replayed safely - provider-specific code remains isolated in adapters - a new developer can understand the workflow from the documentation - rollback can be completed through the approved release process ROI considerations: Measure: - manual recovery hours eliminated - duplicate correction work reduced - failed synchronization rate reduced - support tickets reduced - onboarding time for new developers reduced - incident investigation time reduced - maintenance cost compared with the legacy script