End-to-End Journey Testing Strategy

Wrote docs/architecture/tasks/banking-e2e-001/e2e-testing-strategy/e2e-testing-strategy.md. Complete end-to-end journey testing strategy establishing critical path taxonomy, ephemeral API test data factories, deterministic wait patterns, and automated flake quarantine rules.

---

skill: e2e-testing-strategy

End-to-End Testing Strategy: Retail Onboarding Journey [E2E-BANK-001]

Summary

This specification establishes the end-to-end (E2E) customer journey testing strategy, automated test data provisioning architecture, and flakiness elimination framework for retail-banking-onboarding v3.2 under run ID banking-e2e-001. It qualifies 18 critical user onboarding flows across Web (Playwright) and Mobile (Appium/XCUITest/UiAutomator) spanning 14 microservices. It decisively eliminates the catastrophic test flakiness and release bypass risks demonstrated in incident INC-4940 (where a 42% flake rate led engineers to bypass CI gates, allowing a broken KYC verification release that blocked 12,000 new banking applicants). The strategy enforces a disciplined critical path taxonomy (focusing on the top 18 core revenue-generating paths), ephemeral isolated test accounts generated via dedicated backend API factories, deterministic web/mobile event-driven polling (strictly banning arbitrary sleep() statements), a hard 18-minute CI execution budget across parallel sharded workers, and an automated quarantine policy isolating tests with > 1.0% failure rates.

Detailed Description

Relying on hundreds of brittle, UI-driven end-to-end tests connected to shared staging databases and third-party credit bureau endpoints guarantees high false-positive failure rates. When test pipelines fail intermittently due to network jitter or test data collisions, engineering teams develop alert fatigue and routinely bypass release gates. An authoritative E2E strategy restricts full-stack UI journeys strictly to business-critical user funnels, decouples tests from shared state through ephemeral test data seeding, and enforces deterministic synchronization.

Pull Request Commit: `retail-banking-onboarding`
                           │
                           ▼
[ CI E2E Orchestrator: GitHub Actions Matrix (8 Shards) ]
  ├── 1. Generates Ephemeral Test Persona via API Seeding Factory
  │      └── Mints Unique Account: `test_user_01J8N6B...` (Isolated Ledger State)
  ├── 2. Executes 18 Critical Journey Specs in Parallel (Playwright / Appium)
  │      ├── Journey 1: Customer Self-Registration & SMS Verification
  │      ├── Journey 2: Identity Document KYC Upload & Facial Match
  │      └── Journey 3: Initial Debit Card Deposit & Account Activation
                           │
       ┌───────────────────┴───────────────────┐
       ▼ (Passes 100% Deterministic Assertions) ▼ (Flake / Network Glitch Detected)
[ Green Release Promotion Gate ]       [ Flake Analysis & Quarantine Daemon ]
  ├── Suite Completed in <= 14 mins      ├── Retries Failed Test Once in Clean Sandbox
  └── Ready for Production Canary        └── If Flaky > 1.0%: Quarantines to #quarantine-tests

Criteria and weights

Comparison

Result

Option C is selected. Focusing on 18 critical paths reduces test runtime by 70%; API data factories eliminate collisions; deterministic polling eliminates flakiness.

---

Required Mechanisms

1. Critical Journey Taxonomy (The Critical 18) [MC-CJ-01]

Tier 1: Core Acquisition Funnel (6 Journeys)

1. E2E-ONB-01: Mobile new user phone verification with synthetic SMS OTP.
2. E2E-ONB-02: Identity document upload (Passport / Driver License) with mock KYC provider.
3. E2E-ONB-03: Biometric facial liveness verification and identity matching.
4. E2E-ONB-04: Core ledger account generation and routing number assignment.
5. E2E-ONB-05: Initial debit card funding ($50 transfer) via mock payment rail.
6. E2E-ONB-06: Password / FIDO2 credential enrollment and first biometric login.

Tier 2: Account Servicing & Transfers (6 Journeys)

7. E2E-TRF-01 through E2E-TRF-06: Inter-account transfers, statement downloads, and push alerts.

Tier 3: Edge Cases & Error Recovery (6 Journeys)

8. E2E-ERR-01 through E2E-ERR-06: Expired document retry, network disconnect recovery, KYC review escalation.

2. Test Data Seeding & API Factory Isolation [MC-DF-01]

• Prohibition: Tests must not rely on pre-existing hardcoded user accounts (testuser1@bank.internal).
• API Test Factory (/api/test-harness/users):
  • Before test execution, the Playwright / Appium worker calls the backend test factory via authenticated HTTP POST.
  • The factory provisions a unique, ephemeral customer record with UUID suffix in < 800 ms.
  • Database records are tagged with ephemeral_test_run_id: "run-01J8N...".
  • Post-test hook dispatches an automated cleanup command purging test account state from Aurora and Redis.

3. Deterministic Wait Patterns & Flakiness Elimination [MC-FL-01]

Strict Prohibition: time.sleep(), page.waitForTimeout(), and arbitrary millisecond delays are

strictly prohibited in test code.

• Deterministic Assertion Standards:
  • Web (Playwright): Uses web-first auto-waiting assertions (expect(page.getByRole("button")).toBeVisible()).
  • Asynchronous Events (KYC Processing): Tests poll the backend status API using bounded exponential backoff with a hard timeout of 15 seconds:

    await expect.poll(async () => {
      const res = await fetchAccountStatus(user.id);
      return res.kyc_status;
    }, { timeout: 15000, intervals: [500, 1000, 2000] }).toBe("APPROVED");
    

4. Automated Flake Quarantine & Gate Enforcement [MC-QG-01]

• If a test fails in a pull request pipeline:
  1. The test runner retries the failed spec exactly once in a freshly provisioned container sandbox.
  2. If the test passes on the second attempt, the build is flagged as

FLAKY and metric test_flake_count is incremented.
3. If a test exhibits $> \mathbf{1.0%}$ flake rate across 100 consecutive runs, a GitHub Actions bot automatically applies the @quarantined annotation.
4. Quarantined tests run in a non-blocking advisory pipeline until a dedicated reliability engineer resolves the root cause.

---

Invariants and Contracts

Zero Arbitrary Sleep Delay Invariant [INV-E2E-01]
  Test scripts must not contain arbitrary sleep or fixed-duration thread pause statements.
  All state transitions must be verified using deterministic event listeners or auto-waiting assertions.

Mandatory Test Account Isolation [INV-E2E-02]
  Every end-to-end test execution must operate against a dedicated, dynamically seeded customer persona.
  Reusing static shared user accounts across concurrent test shards is strictly prohibited.

Eighteen-Minute Execution Budget Floor [INV-E2E-03]
  The complete critical-18 E2E test suite must execute and complete within 18 minutes.
  Pull request pipelines exceeding 18 minutes trigger automated test shard rebalancing.

Explicit Unknowns

• Appium iOS Simulator rendering latency variances when executing 8 concurrent mobile simulator instances per Mac bare-metal runner (G-1).
• Third-party mock biometric vendor SDK initialization overhead during high-frequency parallel test worker startup (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against E2E testing strategy standards:

• Scope Discipline: PASS. Strictly limited to 18 critical acquisition and transaction funnels.
• Data Safety: PASS. Ephemeral API factories generate isolated user state; zero shared data collisions.
• Flakiness Defense: PASS. Auto-waiting assertions replace arbitrary sleeps; automated quarantine for flakes > 1%.
• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-E2E-01: Elena Rostova to determine whether visual regression testing (Percy / Applitools) should be included in the pre-merge gate or decoupled to nightly canary runs (Owner: Elena Rostova).

Next steps

1. Marcus Vance provisions 8 parallel Mac and Linux GitHub Actions runners for sharded execution.
2. Platform team implements ephemeral customer seeding endpoints in the test harness API.
3. Quality Engineering refactors the 18 critical onboarding specs to replace legacy sleep() calls with auto-waiting assertions.