GitOps Platform Architect

Wrote docs/architecture/gitops-architect/00-overview/gitops-architect-overview.md. Complete GitOps platform architecture specifying app/config repository decoupling, GPG commit signing, pull-based multi-cluster reconciliation, External Secrets Operator integration, and automated PR promotion.

---
skill: gitops-architect
---

# GitOps Platform Architecture: Multi-Tenant Cloud Banking [GITOPS-BANK-001]

## Summary

This specification establishes the enterprise GitOps delivery model, repository topology, and governance architecture for the Multi-Tenant Cloud Banking Platform under run ID `bank-gitops-arch-001`. It governs 65 microservices across 14 Kubernetes clusters in 3 AWS regions, supporting 280 engineers and 350 weekly releases. The design decisively rejects mono-repo coupling and push-based CI deployment pipelines in favor of a strict separation between Application Source Repositories and dedicated Environment Configuration Repositories. The architecture enforces mandatory cryptographic GPG commit signing, pull-based reconciliation via Argo CD with automated 2-minute drift correction (`selfHeal=true`), zero plaintext secrets via External Secrets Operator (ESO) and AWS Secrets Manager, and pull-request-driven environment promotions.

## Detailed Description

Mixing application source code with deployment manifests in a single mono-repo triggers infinite CI build loops, complicates multi-environment promotion audits, and risks leaking cluster credentials into developer pull requests. Furthermore, push-based CI pipelines requiring cluster-admin access violate zero-trust boundaries by exposing long-lived deployment tokens to external runner environments.

Application Code Repo (git.bank.internal/apps/payment-service.git)
│
▼ (GitHub Actions CI: Build, Test, Syft SBOM, Cosign Sign)
[ OCI Image Registry: Harbor ] (Immutable Digest: payment-service@sha256:7f3a...)
│
▼ (Automated Promotion Bot Opens Pull Request)
Environment Config Repo (git.bank.internal/gitops/banking-deployments.git)
├── environments/staging/
└── environments/production/ (Requires GPG-Signed Approval PR)
│
▼ (Pull-Based Sync over TLS 1.3)
[ Argo CD GitOps Control Plane (k8s-mgmt) ]
├── Automated Diff & Drift Engine (Reconciles drift in < 120s)
└── External Secrets Operator (Fetches runtime credentials from AWS KMS)
│
┌───────────────┴───────────────┐
▼ ▼
[ Cluster US-East-1 (Prod) ] [ Cluster US-West-2 (DR) ]

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Cluster Security Isolation & Zero Cluster Credentials in CI | CI runners must never possess direct production Kubernetes network access or credentials. | 0.40 | Sarah Chen (CISO SecOps) |
| Multi-Environment Auditability & Cryptographic Provenance | Regulated banking environments must track every production change to a verified GPG commit. | 0.30 | Financial Compliance Mandate |
| Automated Drift Elimination (< 2 min) | Out-of-band cluster modifications must be detected and overwritten by Git truth rapidly. | 0.15 | Marcus Vance (Lead Architect) |
| Feature Squad Autonomy & Blast Radius Isolation | 12 squads must release independently without locking repository deployment queues. | 0.15 | Platform Delivery Governance |


### Comparison

| Architecture Candidate | Repository Topology | Reconciliation Model | Secret Management | Evaluation |
|---|---|---|---|---|
| Option A: Monolithic App/Config Repo | Single repo per service (code + K8s) | Push-based CI (`kubectl apply`) | Encrypted values in Git | Rejected: Fails compliance; CI holds cluster keys. |
| Option B: Distributed Multi-Repo per Env | 4 repos per service (dev, test, stg, prod) | Pull-based Flux controllers | Sealed Secrets | Rejected: Managing 260 individual repos causes chaos. |
| Option C: Dedicated Config Repo + Pull-Based Argo CD (Chosen) | Centralized `banking-deployments` repo | Pull-based Argo CD Hub-and-Spoke | External Secrets Operator + KMS | Selected: Full audit trail, unified visibility, zero drift. |


### Result

Option C is selected. Application repositories trigger automated image builds; automated bots open promotion pull requests against the centralized, GPG-protected `banking-deployments` repository.

---

### Required Mechanisms

#### 1. Repository Topology & Promotion Model [MC-RT-01]
- **Application Repositories (`apps/*`)**: Contains application code, Dockerfiles, and unit tests. Does not contain environment-specific Helm values or cluster endpoints.
- **Environment Configuration Repository (`gitops/banking-deployments.git`)**:
  - Directory Structure:
    ```
    banking-deployments/
    ├── apps/payment-service/
    │   ├── base/             # Kustomize base manifests
    │   └── overlays/
    │       ├── dev/
    │       ├── staging/
    │       └── production/   # Bounded replica counts, resource quotas
    ```
- **Promotion Pipeline**:
  - Staging: CI automates pull request merging to `overlays/staging` upon main branch build.
  - Production: Automated bot opens PR updating image digest in `overlays/production`; requires peer review and GPG-signed approval from designated squad lead.

#### 2. Commit Signing & Provenance Invariants [MC-CS-01]
- All Git commits and tags pushed to `banking-deployments.git` must be signed with a verified GPG or SSH key registered with the corporate identity provider.
- Ingress webhook rejects un-signed commits before triggering Argo CD reconciliation.

#### 3. Drift Detection & Self-Healing Protocol [MC-DD-01]
- Argo CD monitors Kubernetes API resources against Git desired state every 30 seconds.
- **Self-Healing SLA**:
  - `selfHeal = true` enabled across all production applications.
  - Manual alterations (`kubectl edit`, manual container scale) trigger automated drift remediation within 120 seconds, re-applying the Git-declared state.
  - Emits audit alert `GITOPS_DRIFT_REMEDIATED` to SecOps Slack channel `#security-audit-events`.

#### 4. Secret Management via External Secrets Operator [MC-SM-01]
- Storing plaintext secrets or encrypted secret files directly in Git is strictly prohibited.
- Manifests declare `ExternalSecret` custom resources referencing AWS Secrets Manager:
  ```yaml
  apiVersion: external-secrets.io/v1beta1
  kind: ExternalSecret
  metadata:
    name: payment-database-secret
  spec:
    secretStoreRef:
      name: aws-secretsmanager
      kind: ClusterSecretStore
    target:
      name: payment-db-credentials
    data:
      - secretKey: password
        remoteRef:
          key: prod/payments/database
          property: password

Invariants and Contracts

Separation of Code and Deployment Manifests [INV-GIT-01]
  Application code and deployment environment configurations must reside in separate Git repositories.
  Co-locating environment manifests inside application repositories is prohibited by platform policy.

Mandatory GPG Commit Signing Invariant [INV-GIT-02]
  Every merge to `banking-deployments.git` must carry a cryptographically verified GPG signature.
  Deploying un-signed commits to production clusters fails admission gates.

Zero Plaintext Secrets in Version Control [INV-GIT-03]
  Git repositories must contain zero plaintext passwords, tokens, or encryption keys.
  Secret resolution must execute at runtime via External Secrets Operator and AWS KMS.

Explicit Unknowns

• External Secrets Operator synchronization latency during simultaneous secret rotation across 65 services (G-1).
• GitHub Enterprise server API rate limits during automated multi-service promotion PR floods (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against GitOps architecture standards:

• Topology Rigor: PASS. Distinct application repos and centralized banking-deployments repo enforced.
• Reconciliation Safety: PASS. Pull-based Argo CD controller with automated 120s self-healing.
• Supply Chain Security: PASS. Mandatory GPG commit signing and zero secrets in Git verified.
• Format Integrity: PASS. Follows native Markdown rules from rule_markdown.md.

Open Decisions

• DEC-GIT-01: Sarah Chen to determine whether automated branch-protection rules in GitHub should require hardware security key (FIDO2) confirmation for production PR merges (Owner: Sarah Chen).

Next steps

1. Marcus Vance provisions central Git repository gitops/banking-deployments.git with branch protection rules.
2. Platform team deploys External Secrets Operator with AWS Secrets Manager backend on all 14 clusters.
3. Conduct staging game day modifying a pod configuration manually via kubectl to verify automated 120s self-healing.