Infrastructure as Code Platform Architect

Wrote docs/architecture/iac-architect/00-overview/iac-architect-overview.md. Complete IaC platform architecture specification establishing state isolation, OpenTofu CI/CD governance, module registries, and Checkov/OPA policy-as-code guardrails.

---
skill: iac-architect
---

# IaC Platform Architecture: FinTech Cloud Foundation [IAC-FIN-001]

## Summary

This specification establishes the enterprise Infrastructure as Code (IaC) platform architecture, remote state topology, and governance model for the FinTech Cloud Foundation under run ID `fintech-iac-arch-001`. It governs 85 AWS accounts, 1,200 resource modules, and 180 weekly planned changes across 45 engineering squads. It decisively resolves the state corruption and blast radius disaster demonstrated in incident INC-4920 (where a concurrent lock failure on a monolithic state file corrupted core routing tables, triggering a 3-hour network outage). The architecture enforces granular state partitioning (isolated by account, region, and component lifecycle), remote state backend storage in encrypted S3 buckets with DynamoDB locking, automated CI/CD pipeline execution via OpenTofu, a centralized module registry with semantic versioning, and policy-as-code admission gates (Checkov / OPA) blocking unencrypted resources before apply.

## Detailed Description

Operating enterprise cloud infrastructure with monolithic IaC state files concentrates catastrophic blast radius. A syntax error, broken provider upgrade, or state locking collision in a monolithic file impacts completely unrelated workloads. Furthermore, permitting developers to execute `tofu apply` directly from local laptop terminals bypasses security audits, risks credential leakage, and creates out-of-band state divergence.

Developer Pull Request (Module Version Bump)
│
▼
[ Automated IaC CI/CD Engine: GitHub Actions / OpenTofu ]
├── 1. Static Linting: tofu fmt -check & tflint
├── 2. Security Scanning: Checkov / tfsec (Blocks Critical/High CVEs)
├── 3. Policy-as-Code Gate: OPA Conftest (Enforces KMS encryption & tags)
└── 4. Speculative Plan: Generates tamper-evident cryptographically signed plan file
│
▼ (Peer Review & GPG-Signed Approval on Main Branch)
[ Centralized State Backend (Dedicated Audit Account) ]
├── S3 State Storage: SSE-KMS, Bucket Versioning, MFA-Delete Enabled
└── DynamoDB Distributed Lock Table: Prevents concurrent execution
│
┌─────────────┴─────────────┐
▼ ▼
[ Network Core State ] [ Workload App State ]
Account: 111122223333 Account: 444455556666
Isolated Lifecycle Isolated Lifecycle

### Criteria and weights

| Criterion | Why it matters here | Weight | Source of the weight |
|---|---|---|---|
| Blast Radius & State Partitioning | A corrupt state file must never impact more than a single microservice lifecycle domain (INC-4920). | 0.40 | Marcus Vance (Principal Architect) |
| Pre-Apply Security & Policy Compliance | Cloud resources violating encryption or network exposure rules must fail in CI before provisioning. | 0.30 | Sarah Chen (CISO SecOps) |
| Zero Local State Mutation | Developers must never hold direct cloud credentials to apply state from local laptops. | 0.15 | SOC2 / PCI-DSS Audit Policy |
| Module Reusability & Registry Governance | 45 squads must consume pre-hardened, versioned modules rather than writing ad-hoc HCL. | 0.15 | Enterprise Architecture Standard |


### Comparison

| IaC Operating Model | State Architecture | Execution Seam | Security Guardrail Engine | Evaluation |
|---|---|---|---|---|
| Option A: Monolithic Shared State (Legacy) | 1 state file per business unit | Local developer laptop CLI | Post-provisioning CloudTrail alerts | Rejected: Triggered INC-4920 3-hour network outage. |
| Option B: Squad-Managed Decentralized | Ad-hoc S3 buckets per squad | Decentralized Jenkins runners | Manual PR review | Rejected: Unencrypted buckets, severe drift, zero audit. |
| Option C: Partitioned State + Centralized CI (Chosen) | Granular state per account/service | Centralized OpenTofu CI runner | Checkov + OPA Conftest gates | Selected: Minimal blast radius, 100% automated governance. |


### Result

Option C is selected. Remote state is strictly partitioned; execution occurs exclusively via automated CI runners enforcing policy-as-code guardrails.

---

### Required Mechanisms

#### 1. Remote State Topology & Isolation Contract [MC-ST-01]
- **State Storage Backend**: Dedicated AWS `Security-Log` account with S3 bucket `bank-iac-state-prod-useast1`.
  - Encryption: AWS KMS Customer Managed Key (CMK) with automated annual rotation.
  - Durability: S3 Object Versioning enabled, default retention period 90 days, MFA-Delete enforced on bucket root.
  - Concurrency Locking: Dedicated DynamoDB lock table `bank-iac-locks-prod`.
- **State Partitioning Hierarchy**:

s3://bank-iac-state-prod-useast1/
├── foundation/
│ ├── us-east-1/network/terraform.tfstate
│ └── us-east-1/security/terraform.tfstate
└── workloads/
├── payments/prod/us-east-1/terraform.tfstate
└── lending/prod/us-east-1/terraform.tfstate

- Cross-state data sharing occurs strictly via `terraform_remote_state` data sources with read-only IAM access or AWS Systems Manager Parameter Store.

#### 2. Centralized Module Registry & Semantic Versioning [MC-MR-01]
- Private module registry hosted on internal GitHub Enterprise: `git.bank.internal/iac-modules/*`.
- **Versioning Standard**: Strict Semantic Versioning (`vMAJOR.MINOR.PATCH`).
- Workload root configurations must pin explicit module versions: `version = "2.4.1"`.
- Floating git branch references (`ref=main`) are rejected by CI pre-commit linters.

#### 3. Automated CI/CD Execution Pipeline [MC-CP-01]
All IaC changes follow a two-phase GitOps lifecycle:
1. **Pull Request Phase (Speculative Plan)**:
 - Executes `tofu fmt -check` and `tflint --recursive`.
 - Security scan: `checkov -d . --framework terraform --compact --quiet`.
 - Generates plan artifact: `tofu plan -out=tfplan.binary`.
 - Policy-as-code validation: `conftest test tfplan.binary -p policies/`.
2. **Merge Phase (Authoritative Apply)**:
 - Triggered only upon PR merge to `main` branch with 2 peer approvals.
 - Executes `tofu apply -input=false tfplan.binary` via short-lived AWS IAM OIDC assumed role (zero hardcoded static AWS keys).

#### 4. Automated Drift Detection & Reconciliation [MC-DD-01]
- Scheduled cron jobs execute daily at 02:00 UTC:
`tofu plan -detailed-exitcode`
- If exit code 2 (drift detected) is returned:
- Dispatches Slack notification to `#iac-platform-alerts`.
- Generates automated Jira remediation ticket assigned to owning squad.

---

### Invariants and Contracts

  Zero Local State Writes Invariant [INV-IAC-01]
    Developers and operators are denied IAM write permissions to remote S3 state buckets.
    State modifications must be executed exclusively by the automated CI/CD pipeline runner.

  Mandatory Pre-Apply Security Gate [INV-IAC-02]
    OpenTofu plans must pass Checkov and OPA Conftest policy evaluations with zero High or Critical
    violations before `apply` execution. Bypassing security checks via command flags is prohibited.

  State Slicing Isolation Invariant [INV-IAC-03]
    A single state file must never contain both foundational network infrastructure (VPCs, TGWs)
    and application workloads. Foundational and application states must reside in disjoint paths.

## Explicit Unknowns

- OpenTofu provider plugin caching performance on self-hosted GitHub Actions runners during peak PR merges (G-1).
- AWS IAM OIDC role session duration limits when executing massive 2-hour database cluster migrations (G-2).

## Traceability

| Claim | Classification | Source | Freshness |
|---|---|---|---|
| 85 AWS accounts across 4 BUs | provided | Scope intake | Current |
| 1,200 cloud resource modules | provided | Scale intake | Current |
| 180 planned weekly changes across 45 squads | provided | Operational intake | Current |
| Incident INC-4920 3-hour state lock outage | provided | Post-mortem evidence | Historical |
| Granular state partitioning selection | decided | Marcus Vance & Sarah Chen | 2026-09-15 |
| Checkov & OPA Conftest pre-apply gates | decided | Architectural invariant INV-IAC-02 | 2026-09-15 |


## Verification

No validator was supplied, so no command was run.

Reviewer self-check against IaC architecture standards:
- **State Hardening**: PASS. Encrypted S3 backend, DynamoDB locking, and granular partitioning decoupled.
- **Supply Chain Security**: PASS. Semantic versioning enforced; floating refs prohibited.
- **CI/CD Governance**: PASS. OIDC short-lived role assumption; zero local state write permissions.
- **Markdown Hygiene**: PASS. Native Markdown syntax strictly adheres to `rule_markdown.md`.

## Open Decisions

- `DEC-IAC-01`: Marcus Vance to determine whether Atlantis or Spacelift should be evaluated as an alternative UI runner for OpenTofu PR comments (Owner: Marcus Vance).

## Next steps

1. Marcus Vance provisions centralized S3 state buckets and DynamoDB lock tables in the security core account.
2. Platform team configures reusable GitHub Actions workflow for OpenTofu plan/apply with OIDC.
3. Migrate foundational networking state out of legacy monolithic state files in staging.