Data Mesh Architecture Style Evaluation

Wrote docs/architecture/styles/mesh-style-evaluation.md. Complete Data Mesh architecture style evaluation establishing decentralized domain data products, self-serve infrastructure platforms, federated computational governance, and regulatory reporting compliance.

skill: mesh-style

Architecture Style Evaluation: Data Mesh Architecture [STYLE-MESH-001]

Summary

This specification establishes the architectural style evaluation of

Data Mesh Architecture for commercial-risk-analytics-platform v3.0 under run ID lending-data-mesh-001. It evaluates architectural candidates for handling analytical reporting and regulatory data products across 35 cross-divisional consumers (Commercial Lending, Credit Risk, Treasury, and Basel III Regulatory Compliance) ingesting 14 TB of daily financial transaction data. The evaluation resolves the chronic organizational bottleneck demonstrated in incident INC-4931 (where a centralized data engineering team became an unscalable pipeline bottleneck, delaying Basel III compliance reporting for 4 months and triggering a $2.2M regulatory penalty). The evaluation compares three primary architecture styles: Monolithic Central Data Lakehouse (Snowflake), Hub-and-Spoke Data Warehouse, and

Data Mesh (Decentralized Domain Data Products). It selects Data Mesh as the optimal style, specifying domain-owned data products, automated self-serve platform infrastructure, OPA-driven federated computational governance, and strict data quantum output port contracts.

Detailed Description

Centralized data architectures (data lakes and warehouses) fail to scale organizationally as an enterprise grows. Domain knowledge resides with business domain squads, yet data ingestion and pipeline maintenance are thrown over the wall to a centralized data engineering team that lacks domain context. This creates brittle ETL pipelines, finger-pointing when source schemas mutate, and multi-month backlogs for analytical queries. Data Mesh (Zhamak Dehghani) treats data as a first-class product owned directly by domain squads. The central team shifts from writing pipelines to building a self-serve platform that enables domain teams to independently publish, govern, and observe their own data products.

Enterprise Business Domains (Decentralized Ownership)
                         │
        ┌────────────────┼────────────────┐
        ▼                ▼                ▼
[ Commercial Lending ]  [ Credit Risk ]  [ Treasury Clearing ]
  ├── Domain Team Owns:   ├── Domain Team Owns:   ├── Domain Team Owns:
  │   `LoanOrigination`   │   `CovenantMetrics`   │   `LiquiditySettlement`
  └── Data Product Port:  └── Data Product Port:  └── Data Product Port:
      S3 Parquet + Iceberg    Delta Lake Tables       Kafka Event Stream
                         │
                         ▼ (Consumes via Self-Serve Data Infrastructure)
[ Self-Serve Data Platform (Storage, Orchestration, Lineage, Catalog) ]
                         │
                         ▼ (Governed by Automated Policies)
[ Federated Computational Governance: Open Policy Agent (OPA) ]
  ├── Automatic PII Masking (GDPR / CCPA)
  └── Standardized Schema Verification & Lineage Tracking

Criteria and weights

Comparison

Result

Option C is selected. Data Mesh decentralizes analytical data ownership to business domain squads; central engineering builds the self-serve platform; automated OPA policies enforce compliance governance.

Required Mechanisms

1. Domain Ownership Decentralization [MC-DO-01]

• The Commercial Lending and Credit Risk squads assume full end-to-end responsibility for both their operational microservices and their analytical data products:
  • commercial-lending squad owns and publishes the loan_origination_mart data product.
  • Squads allocate 20% of engineering bandwidth to analytical data modeling and data quality maintenance.

2. Data as a Product & Quantum Contracts [MC-DP-01]

• Data Product Definition (commercial_loan_product_v3):
  • Code: Transformation dbt models and Apache Iceberg table definitions.
  • Data: Apache Parquet files stored on AWS S3 with Iceberg ACID metadata.
  • Metadata: Sourced schema definitions, ownership metadata, and OpenLineage tracking.
  • Output Ports: Read-only SQL queries via AWS Athena and Trino connectors.
• Service Level Objectives (SLOs):
  • Data Freshness: Updated every 60 minutes (well within the 4-hour budget).
  • Schema Stability: 90-day deprecation notice period for analytical breaking changes.

3. Self-Serve Data Infrastructure Platform [MC-SP-01]

• Central Platform Engineering provides automated Terraform and Kubernetes primitives:
  • One-click provisioning of Iceberg storage buckets, dbt workflow runners, and Trino query catalogs.
  • Domain squads deploy data products without managing raw cloud storage permissions or cluster infrastructure.

4. Federated Computational Governance [MC-FG-01]

• Automated governance enforced as code via Open Policy Agent (OPA):
  • Ingestion Gate: Every data product output port must pass automated schema linting.
  • Data Privacy Policy: Automated hashing or masking of customer PII fields (borrower_ssn, tax_id) enforced at query time based on consumer IAM roles.

Invariants and Contracts

Decentralized Domain Ownership Invariant [INV-MESH-01]
  Domain squads must own the lifecycle, modeling, and quality of their published data products.
  Transferring raw un-modeled operational database dumps to a central data team is strictly prohibited.

Explicit Data Product Output Port Contract [INV-MESH-02]
  Data products must expose clean, versioned output ports (e.g. Iceberg/Trino SQL or Kafka streams).
  Analytical consumers querying operational microservice transactional databases directly is barred.

Automated Computational Governance Enforcement [INV-MESH-03]
  All published data products must register schemas in the central catalog and comply with automated
  OPA privacy and access policies. Deploying un-governed data products is blocked by CI/CD gates.

Explicit Unknowns

• Operational cost overhead of running distributed Iceberg metadata catalogs across 8 disparate AWS accounts (G-1).
• Training timeline required to upskill 80 full-stack software engineers in analytical dbt and SQL modeling (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against Data Mesh standards:

• Decentralization: PASS. Domain squads own analytical data products directly; central bottleneck removed.
• Product Thinking: PASS. Data products package code, data, metadata, and output ports with explicit SLOs.
• Governance Automation: PASS. OPA computational policies automate PII masking and schema compliance.
• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-MESH-01: David O'Reilly to determine whether Apache Iceberg or Delta Lake should be standardized as the single universal storage format across all AWS accounts (Owner: David O'Reilly).

Next steps

1. Platform team deploys central AWS Glue / Trino self-serve catalog infrastructure.
2. Commercial Lending squad authors the pilot loan_origination_mart data product using dbt and Iceberg.
3. Conduct regulatory compliance review validating automated PII masking in Basel III analytics queries.