Argo CD GitOps Architecture Design

Wrote docs/architecture/tasks/gitops-payments-001/argocd-design/argocd-architecture-spec.md. Complete Argo CD GitOps architecture specification establishing hub-and-spoke AppProjects, automated self-healing, sync-wave sequencing, and cluster isolation.

---
skill: argocd-design
---

# Argo CD GitOps Architecture Specification: Core Banking Platform [ARGO-BANK-001]

## Summary

This specification establishes the enterprise GitOps continuous delivery architecture for `argocd-platform v2.11` under run ID `gitops-payments-001`. It orchestrates declarative deployments for 35 banking microservices across a hub-and-spoke multi-cluster topology (management hub `k8s-mgmt`, staging `https://k8s-staging-01.internal`, and production EKS). It decisively rejects shared cluster-admin credentials and manual sync overrides. The design enforces isolated `AppProject` resource boundaries with namespace whitelisting, automated 3-minute self-healing (`selfHeal=true`) against manual drift, declarative sync-wave sequencing (`argocd.argoproj.io/sync-wave`) guaranteeing database migrations complete before workload pods start, and automated orphan pruning (`prune=true`).

## Detailed Description

Imperative manual Kubernetes changes (`kubectl edit`, manual cluster access) create silent configuration drift, audit non-compliance, and irreproducible deployment states. This GitOps architecture establishes Git as the single cryptographic source of truth, where the control plane continuously reconciles cluster state against desired version-controlled manifests.

Git Repository: https://git.internal/payments/services.git
│
▼ (Webhook Trigger & 3-Minute Polling)
[ Argo CD Central Hub: k8s-mgmt ]
├── AppProject: payments-prod (Restricted Destinations & Whitelist)
├── Automated Diff Engine (Detects Manual Out-of-Band Mutations)

  └── Sync Wave Controller: Wave -1 (DB Migrate) -> Wave 0 (Workloads)
                          │
      ┌───────────────────┴───────────────────┐
      ▼                                       ▼

[ Remote Cluster: Staging ] [ Remote Cluster: Production ]
├── Destination: k8s-staging-01 ├── Destination: k8s-prod-useast1
└── Enforces SelfHeal=true └── Enforces Prune=true (Zero Orphans)

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Cluster Security Isolation & RBAC | Shared cluster-admin access violates PCI-DSS multi-tenant compliance. | 0.35 | Sarah Chen (CISO SecOps) |
| Configuration Drift Elimination | Manual cluster tweaks must be automatically reverted to match Git truth within 3 minutes. | 0.30 | Marcus Vance (Platform Lead) |
| Deterministic Deployment Sequencing | Database migrations must execute to completion before application pods launch. | 0.20 | Core Banking Engineering Mandate |
| Orphaned Resource Pruning | Deleting services in Git must cleanly remove all underlying K8s resources. | 0.15 | Infrastructure Hygiene Policy |


### Comparison

| Architecture Candidate | Cluster Access Model | Drift Remediation | Deployment Sequencing | Evaluation |
|---|---|---|---|---|
| Option A: Monolithic Default Project | Shared `default` project, cluster-admin tokens | Manual sync only (`selfHeal=false`) | Uncoordinated simultaneous pod rollouts | Rejected: Breaches PCI-DSS; manual changes linger indefinitely. |
| Option B: Decoupled Flux v2 Controllers | In-cluster standalone controllers | Reconciliation loop (no UI) | Kustomize dependsOn | Rejected: Lacks centralized single-pane-of-glass multi-cluster dashboard. |
| Option C: Hub-and-Spoke Argo CD + AppProjects (Chosen) | Central hub with scoped IAM service accounts | Automated `selfHeal=true`, `prune=true` | Declarative sync waves (-1 to 2) | Selected: Centralized governance, deterministic wave sequencing, zero drift. |


### Result

Option C is selected. A centralized management cluster runs Argo CD, dispatching scoped deployments to target clusters via isolated `AppProject` definitions.

---

### Required Mechanisms

#### 1. Task Contract & Repository Topology [MC-TC-01]
- **Target Clusters**:
  - Management Hub: `in-cluster` (`k8s-mgmt`).
  - Staging Destination: `https://k8s-staging-01.internal` (Namespace: `payments-staging`).
  - Production Destination: `https://k8s-prod.internal` (Namespace: `payments-prod`).
- **Source Repository**: `https://git.internal/payments/services.git` (Revision: `main` in prod, `develop` in staging).

#### 2. AppProject Isolation Contract [MC-AP-01]
```yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments-production

namespace: argocd
spec:

  description: Scoped CD project for PCI payment microservices.
  sourceRepos:
    - 'https://git.internal/payments/services.git'
  destinations:
    - server: 'https://k8s-prod.internal'

  namespace: 'payments-prod'

clusterResourceWhitelist:

    - group: ''
      kind: Namespace
  namespaceResourceWhitelist:
    - group: 'apps'
      kind: Deployment
    - group: ''
      kind: Service
    - group: 'batch'
      kind: Job
    - group: 'networking.k8s.io'
      kind: NetworkPolicy

3. Automated Sync, Self-Healing, and Pruning [MC-SP-01]

Every Application manifest enforces automated synchronization:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
      - PrunePropagationPolicy=foreground
      - RespectIgnoreDifferences=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

• Self-Healing SLA: Out-of-band manual changes (kubectl edit, manual pod deletion) are overwritten by the controller within 180 seconds.

4. Sync-Wave Progression & Hook Ordering [MC-SW-01]

Deployment dependencies execute in strict sequential waves:

• Wave -1 (Database Schema Migration):
  Job: payment-db-migrate (Annotated with argocd.argoproj.io/sync-wave: "-1" and hook-delete-policy: BeforeHookCreation).
  Blocker: Migration job must reach completion status Completed before Wave 0 begins.
• Wave 0 (Core Workloads):
  Deployment: payment-core, Service: payment-core (sync-wave: "0").
• Wave 1 (Ingress & Network Routing):
  CiliumNetworkPolicy, Ingress routing (sync-wave: "1").

Invariants and Contracts

Automated Self-Healing Invariant [INV-ARGO-01]
  Production Applications must enforce `selfHeal: true` and `prune: true`. Disabling self-heal
  or enabling manual-only sync in production pull requests fails CI manifest validation.

Strict AppProject Destination Scoping [INV-ARGO-02]
  Applications belonging to `payments-production` are forbidden from targeting namespaces
  other than `payments-prod`. Cross-namespace deployment attempts fail admission.

Synchronous Migration Gate Invariant [INV-ARGO-03]
  Wave 0 application deployments must not initialize until Wave -1 database migration jobs
  complete with exit code 0. Migration failures immediately abort sync progression.

Explicit Unknowns

• HashiCorp Vault / External Secrets Operator replication lag across secondary AWS regions during secret rotation (G-1).
• Argo CD Redis cache memory growth during simultaneous reconciliation of 350 Application manifests (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against Argo CD architecture standards:

• Security Boundaries: PASS. AppProject limits destinations to payments-prod and specific resource kinds.
• Drift Protection: PASS. selfHeal: true and prune: true guarantee reconciliation to Git.
• Wave Sequencing: PASS. Database migration job isolated in Wave -1 before workload launch.
• Format Integrity: PASS. Follows native Markdown rules from rule_markdown.md.

Open Decisions

• DEC-ARGO-01: Sarah Chen to determine whether Git commit signing (GPG verification) should be enforced at the Argo CD repository certificate level (Owner: Sarah Chen).

Next steps

1. Sarah Chen provisions payments-production AppProject CRD on the k8s-mgmt hub cluster.
2. Platform team configures cluster credentials in Argo CD using down-scoped Kubernetes ServiceAccount tokens.
3. Execute staging deployment test verifying Wave -1 schema migration completion before application pod launch.