Integration Seam and Collaborator Testing Strategy

Wrote docs/architecture/tasks/banking-integ-001/integration-testing-strategy/integration-test-spec.md. Complete integration seam testing strategy establishing collaborator fidelity, real Testcontainers databases, WireMock API stubs, asynchronous completion oracles, and pre-merge coverage gates.

---

skill: integration-testing-strategy

Integration-Testing Strategy: Core Payment Settlement Service [INTEG-BANK-001]

Summary

This specification establishes the integration testing strategy, collaborator seam isolation architecture, and contract verification model for payment-settlement-service v3.2 under run ID banking-integ-001. It governs four critical out-of-process integration seams: PostgreSQL 16 Aurora, Apache Kafka 3.4 settlement topics, Redis 7.2 idempotency caches, and external SWIFT clearing partner APIs. It decisively eliminates the test flakiness and environment collisions demonstrated in incident INC-4941 (where un-isolated shared development databases masked an incompatible PostgreSQL 16 row-locking dialect, halting settlement clearing for 8 hours). The strategy enforces ephemeral containerized dependencies via

Testcontainers (running PostgreSQL and Redpanda Kafka locally in Docker),

WireMock stubs validated continuously against partner OpenAPI specifications, transactional rollback state isolation between test cases, condition-based asynchronous polling oracles, a strict sub-8-minute CI execution budget across parallel worker threads, and blocking pre-merge coverage gates.

Detailed Description

Relying on shared development environments or in-memory database mocks (such as H2 or SQLite) introduces severe fidelity gaps. In-memory databases do not support native PostgreSQL concurrency locks (SELECT ... FOR UPDATE), JSONB operators, or trigger semantics, allowing dialect-specific regressions to reach production undetected. Conversely, calling live third-party partner sandboxes across public networks introduces variable latency, rate-limiting failures, and test flakiness. An authoritative integration strategy isolates real infrastructure dependencies inside ephemeral containers, stubs external APIs using contract-verified mocks, and enforces asynchronous completion oracles.

Developer Pull Request Commit (60 Daily CI Runs)
                         │
                         ▼
[ CI Runner: Ephemeral Testcontainers Infrastructure ]
  ├── PostgreSQL 16 Container (Ephemeral tmpfs storage, Port 5432)
  ├── Redpanda / Kafka Container (Kafka API v3.4, Port 9092)
  └── Redis 7.2 Container (In-memory cache, Port 6379)
                         │
                         ▼
[ External Partner Seam: Local WireMock Daemon ]
  ├── Stubs SWIFT Clearing Gateway (`POST /v1/swift/settle`)
  └── Validates Stub Request Schema against `swift-api-v2.json`
                         │
       ┌─────────────────┴─────────────────┐
       ▼ (100% Integration Seams Pass)     ▼ (Contract Drift / Deadlock Detected)
[ Release Candidate Promoted to Staging ] [ CI HARD BLOCK (< 8 min Feedback) ]
  ├── 142 Seam Integration Specs Passed   ├── Dumps SQL Query Trace & Kafka Offsets
  └── Execution Time: 6m 45s               └── Blocks PR Merge

Criteria and weights

Comparison

Record measured values with their date and version. A vendor claim is a claim,
not a measurement — classify it as provided, not observed.

Result

Option C is selected. Testcontainers guarantees exact PostgreSQL 16 and Kafka fidelity; WireMock with OpenAPI schema validation eliminates external network flakiness; transactional rollbacks ensure test independence and sub-8-minute CI execution.

---

Required Mechanisms

1. Risk [MC-RK-01]

Inputs: Out-of-process dependency topology, schema migration delta scripts (V3.2__settlement_lock.sql), concurrency profile (6,500 clearing events/sec), cross-service failure history (incident INC-4941).

Algorithm: Multi-dimensional seam risk classifier mapping dependency seams to failure impact. Evaluates transaction boundary semantics, row-locking contention (SELECT ... FOR UPDATE NOWAIT), distributed idempotency cache invalidation, and external partner protocol breaking changes.

Outputs: Seam risk classification matrix categorizing PostgreSQL Aurora as High Risk (dialect/deadlock), Kafka as High Risk (partition ordering/outbox), Redis as Medium Risk (TTL/stampede), and SWIFT Clearing as High Risk (contract drift).

• Owner: Elena Rostova (Lead Quality Engineer).

Failure Handling: If an unclassified collaborator seam or migration script is introduced in a pull request without an associated risk rating, the pre-merge linter fails with diagnostic ERR_INTEG_UNCLASSIFIED_SEAM_RISK.

Verification: Seam risk validation check scripts/check_seam_risk_inventory.sh validating 100% coverage of declared external adapters.

2. Test Layer [MC-TL-01]

• Inputs: Component architecture, class dependency graph, process boundary map.
• Algorithm: Layer allocation engine routing verification according to physical execution boundaries:
  • Sociable Unit Tests: In-memory domain entity lifecycle and business calculation logic without external network/process boundaries.
  • Component Integration Tests (Owned Seam): Application adapters executing against isolated containerized collaborators (PostgreSQL 16, Redpanda Kafka, Redis 7) within a single deployable service boundary.
  • Service Integration Tests: WireMock HTTP stub verification simulating SWIFT clearing gateway protocol contracts without cross-estate journey orchestration.

Outputs: Concrete suite categorization strictly isolating component integration tests from pure in-process unit tests and multi-service E2E journey tests.

• Owner: David O'Reilly (Principal Platform Architect).

Failure Handling: Layer boundary violation detector flags tests invoking HTTP endpoints or database drivers inside unit test suites with diagnostic ERR_TEST_LAYER_BOUNDARY_BREACH.

Verification: Architecture fitness test test_integration_layer_conformance() enforcing package and runner segregation.

3. Oracle [MC-OR-01]

Inputs: Database row-level state, Kafka committed offsets, Redis key-value records, WireMock matched request journals, correlation identifiers (X-Correlation-ID: SETTLE-UUID).

• Algorithm: Condition-based polling evaluation engine:
  • Relational Persistence Oracle: Asserts committed database state, foreign key integrity, and row version increments using explicit SQL assertions within isolated worker schemas.
  • Asynchronous Messaging Oracle: Polls Kafka consumer dead-letter and settlement topics using correlation ID SETTLE-UUID at 50 ms intervals with a hard 3.0-second deadline. Arbitrary Thread.sleep() statements are strictly rejected.
  • HTTP Collaborator Oracle: Validates exact WireMock request body match, query parameters, and emitted authorization headers against expected cryptographic hashes.

Outputs: Deterministic ternary outcome classification: PASS, FAIL, or INCONCLUSIVE (timeout/missing correlation).

• Owner: Elena Rostova (Lead Quality Engineer).

Failure Handling: Tests exceeding the 3.0-second async completion deadline fail with diagnostic ERR_ASYNC_ORACLE_TIMEOUT and dump container thread stacks.

Verification: Oracle assertion suite test_settlement_async_oracle_completion() verifying positive and negative terminal state detection.

4. Coverage Gate [MC-CG-01]

Inputs: Pull request git diff, seam inventory, OpenAPI specification revisions, JaCoCo integration branch execution reports.

• Algorithm: Delivery risk gate evaluating:
  1. 100% seam coverage for modified database repository queries and Kafka message producer/consumer handlers.
  2. Zero unmapped HTTP external error codes (400, 401, 403, 422, 500, 502, 503, 504) in WireMock stub portfolios.
  3. Total suite execution duration budget <= 480 seconds (8.0 minutes) across 4 parallel test shards.
• Outputs: CI deployment gate approval token GATE-INTEG-PASS allowing merge to main.
• Owner: Quality Engineering Guild and CI Platform Team.

Failure Handling: If seam branch coverage falls below 100% for altered SQL repositories or execution duration exceeds 8.0 minutes, the gate trips, emitting diagnostic ERR_INTEG_COVERAGE_GATE_FAILED and blocking deployment.

• Verification: Pre-merge gate validator scripts/verify_integration_gate.py exiting 0 on compliant builds.

---

Adversarial Cases and Routing

1. Reject Pyramid by Habit [ADV-PH-01]

Vulnerability: Defaulting to the traditional testing pyramid by habit (accumulating thousands of trivial unit tests while starving collaborator integration seams), creating an illusion of high coverage while dialect-specific database locks and serialization errors slip undetected into production.

Adversarial Mechanism: In incident INC-4941, payment-settlement-service boasted 94% unit test coverage using in-memory mocks. However, zero integration tests exercised PostgreSQL 16's SELECT ... FOR UPDATE NOWAIT locking behavior under concurrent worker transactions, triggering an 8-hour total system deadlock during morning settlement clearing.

Enforcement & Diagnostic: Risk-based seam allocation mandates component integration tests for all repository queries, transactional boundaries, and outbox CDC handlers. Submitting pull requests with unit-only coverage on new database migration scripts or Kafka schemas triggers diagnostic ERR_PYRAMID_BY_HABIT_REJECTED and halts CI promotion.

Forbidden Output Behavior: The testing strategy is strictly forbidden from certifying release readiness based on unit test percentage metrics alone when collaborator seams or SQL dialects are altered.

2. Reject Assertion-Free Test [ADV-AF-01]

Vulnerability: Writing "smoke" integration tests that spin up containers, trigger message publishing or database writes, but contain no rigorous oracles or terminal state assertions (relying solely on the absence of unhandled exceptions).

Adversarial Mechanism: A test executes settlementService.process(paymentEvent) against a Testcontainers database and passes because no runtime exception was thrown; in reality, the record was silently routed to an internal dead-letter table due to an unhandled enum mismatch, causing silent financial transaction loss.

Enforcement & Diagnostic: Automated AST test suite linter scans all test methods under src/test/integration/**. Every test method MUST declare at least one authoritative state, effect, or correlation oracle assertion (e.g., verifying committed database rows, Kafka message headers, or WireMock verify calls). Assertion-free test methods trigger diagnostic ERR_ASSERTION_FREE_TEST_DETECTED and fail the build.

Forbidden Output Behavior: The system is strictly forbidden from accepting integration tests that terminate without explicit state, row, payload, or mock-verification assertions.

3. Reject Flaky External Dependency [ADV-ED-01]

Vulnerability: Coupling integration test suites directly to live external partner sandboxes, shared staging environments, or unpinned remote network endpoints, introducing external network flakiness, rate-limiting, and uncontrollable test failures.

Adversarial Mechanism: Running integration tests against a shared public SWIFT clearing sandbox across WAN links causes random 504 timeouts, HTTP 429 rate-limit rejections, and weekend maintenance downtime, prompting developers to mark flaky tests with @Ignore and bypass quality gates.

Enforcement & Diagnostic: Test runners execute with outbound external network egress blocked by firewall/network namespaces (iptables / container network isolation). All external partner APIs must be provided by local WireMock containers verified against partner OpenAPI specs. Any direct socket connection attempt to an external IP triggers diagnostic ERR_FLAKY_EXTERNAL_DEPENDENCY_BLOCKED.

Forbidden Output Behavior: The system is strictly forbidden from initiating network connections to un-isolated external sandboxes or un-versioned third-party staging endpoints during automated CI test execution.

---

Invariants and Contracts

Mandatory Containerized Seam Parity [INV-INTEG-01]
  Integration tests verifying database persistence must execute against containerized PostgreSQL 16 engines.
  Using in-memory emulation databases (such as H2 or SQLite) for repository tests is strictly prohibited.

Zero External Network Call Mandate [INV-INTEG-02]
  Integration test runners must operate with outbound network access blocked to public endpoints.
  External third-party APIs must be stubbed using contract-validated local WireMock instances.

Condition-Based Polling Oracle Mandate [INV-INTEG-03]
  Asynchronous event integration tests must observe completion via condition-based polling on correlation IDs.
  Arbitrary sleep calls (e.g., Thread.sleep()) are strictly prohibited in test code.

Sub-Eight-Minute Suite Execution Budget [INV-INTEG-04]
  The entire service integration test suite must execute and complete within 8.0 minutes across 4 parallel workers.
  Individual test methods exceeding 5.0 seconds must be quarantined or refactored.

Explicit Unknowns

• Docker container startup latency variances on EKS self-hosted runner nodes during peak parallel merge hours (G-1).
• WireMock memory heap sizing when maintaining 5,000 recorded partner response fixtures (G-2).

Traceability

Verification

Reviewer self-check against integration testing standards:

• Risk Analysis: PASS. Evaluates relational locking, Kafka ordering, Redis caching, and SWIFT contract risks.
• Test Layer: PASS. Enforces strict separation between sociable unit, component integration, and E2E layers.
• Oracle & Async: PASS. Condition-based polling on correlation IDs; bans arbitrary sleep() statements.
• Coverage Gate: PASS. 100% seam coverage on altered SQL/Kafka code; sub-8-minute CI execution SLA.

Adversarial Defenses: PASS. Explicit diagnostics for pyramid by habit, assertion-free tests, and external flakiness.

• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-INTEG-01: Elena Rostova to determine whether local Kafka schema registry containers (Apicurio / Confluent) should be added to the Testcontainers composition in Q4 (Owner: Elena Rostova).

Next steps

1. Platform team embeds Docker-out-of-Docker configurations into GitHub Actions runner images.
2. Core Engineering refactors repository tests to use PostgreSQLContainer and @Transactional rollbacks.
3. Quality Engineering implements AST linter lint_assertion_free_tests.py in the pre-commit hook pipeline.
4. Conduct staging CI run validating build failure when an incompatible SQL dialect query is introduced.