Mutation-Testing and Test Efficacy Strategy

Wrote docs/architecture/tasks/banking-mutation-001/mutation-testing-strategy/mutation-testing-strategy.md. Complete mutation testing strategy establishing PITest operators, incremental Git diff scoping, 85% mutation kill score gates, and sub-6min CI execution budgets.

---

skill: mutation-testing-strategy

Mutation-Testing Strategy: Core Payment Ledger Engine [MUTATION-BANK-001]

Summary

This specification establishes the mutation testing strategy, mutant operator taxonomy, incremental execution architecture, and release gating criteria for core-ledger-engine v4.0 under run ID banking-mutation-001. It qualifies test suite efficacy across 250,000 lines of Java and Kotlin financial calculation code. It decisively eliminates the false confidence of vanity code coverage demonstrated in incident BUG-4102 (where 92% Jacoco line coverage masked assertion-free test cases that allowed a $640,000 currency rounding truncation bug into production). The strategy enforces a

Mutation Score Floor >= 85% across all financial balance mutation packages, focuses mutant operators on high-risk boundaries (Conditionals Boundary, Invert Negatives, Math Mutator, Return Values), restricts pull request execution to

incremental Git diff scoping (completing in <= 5 minutes), filters unkillable equivalent mutants, and establishes blocking pre-merge quality gates.

Detailed Description

Relying on traditional line or branch code coverage creates a dangerous illusion of software quality. Line coverage merely proves that a line of source code was executed by a test; it provides zero evidence that the test asserted on the resulting state or detected incorrect behavior. Mutation testing inverts this dynamic: it deliberately introduces small semantic faults ("mutants") into the bytecode (such as changing >= to >, or + to -) and executes the test suite. If the tests pass despite the introduced fault, the mutant "survives," exposing an assertion blind spot.

Pull Request Commit: `core-ledger-engine` (Git Diff Scoped)
                           │
                           ▼
[ PITest Mutation Engine: Bytecode Fault Injection ]
  ├── 1. Analyzes Git Diff (Identifies Modified Classes & Methods)
  ├── 2. Injects Targeted Financial Mutators:
  │      ├── Math Mutator: `balance.add(fee)` ──► `balance.subtract(fee)`
  │      ├── Conditionals Boundary: `amount >= limit` ──► `amount > limit`
  │      └── Return Values Mutator: `return true` ──► `return false`
                           │
                           ▼
[ Parallel Test Execution: Test-to-Mutant Matrix ]
  ├── Executes ONLY Unit Tests Covering the Mutated Bytecode
  └── Captures Mutant Outcome: KILLED vs SURVIVED
                           │
       ┌───────────────────┴───────────────────┐
       ▼ (Mutation Score >= 85.0%)             ▼ (Mutation Score < 85.0% or Surviving Bug)
[ Green PR Quality Gate ]              [ CI HARD BLOCK & MUTANT TRACE ]
  ├── Verified Test Assertion Quality    ├── Identifies Survived Mutant Bytecode Lines
  └── Merge Permitted                    └── Requires Added Assertions before Merge

Criteria and weights

Comparison

Result

Option C is selected. Incremental Git diff scoping with PITest provides deep assertion quality verification within a 5-minute feedback window.

---

Required Mechanisms

1. Mutant Operator Taxonomy [MC-OT-01]

To maximize sensitivity to financial accounting bugs while minimizing runtime bloat, PITest is configured with four core mutator groups:
1.

Conditionals Boundary Mutator: Replaces < with <=, and >= with >. Detects off-by-one authorization limit bugs.
2.

Math Mutator: Replaces binary arithmetic operators (+ with -, * with /, % with *). Detects incorrect fee or interest calculations.
3. Invert Negatives Mutator: Inverts sign of numeric values (-x to x). Detects debit/credit reversal flaws.
4.

Return Values Mutator: Mutates boolean, object, and numeric return values to null, false, or empty primitives. Detects unasserted business logic methods.

2. Incremental Git Diff Scoping & Coverage Matrix [MC-IS-01]

• Prohibition: Full-codebase mutation runs are prohibited on pull request commits.
• Git Diff Scoping Algorithm:

  mvn org.pitest:pitest-maven:mutationCoverage \
    -DmutationThreshold=85 \
    -DhistoryInputLocation=target/pit-history.bin \
    -DhistoryOutputLocation=target/pit-history.bin \
    -Dfeatures=+GIT(from=origin/main)
  

• The PITest Git plugin analyzes git diff origin/main...HEAD, identifies only the altered bytecode instructions, and executes only the unit tests mapped directly to those instruction paths. Total execution time:

4 minutes 15 seconds.

3. Mutation Score Thresholds & Release Gates [MC-SG-01]

• Metric Definitions:
  $$\text{Mutation Score} = \frac{\text{Killed Mutants}}{\text{Total Mutants} - \text{Equivalent Mutants}} \times 100%$$
• Enforced Thresholds:
  • Package com.bank.ledger.calculation.* (Core Accounting): >= 90.0%.
  • Package com.bank.ledger.service.* (Orchestration): >= 85.0%.
  • Overall PR Diff Mutation Score: >= 85.0%.
• If a pull request achieves < 85% mutation score, the GitHub Actions check status is set to FAILURE.

4. Equivalent Mutant Filtering Protocol [MC-EF-01]

Equivalent Mutant Definition: A mutant that alters bytecode syntax but produces identical semantic runtime behavior (e.g. mutating a loop index optimization that has no observable side effect).

• Suppression Protocol:
  • Engineers annotate equivalent mutants using @ExcludeFromMutation("REASON: Semantically equivalent loop optimization verified in ADR-088").
  • Suppressions require pull request review sign-off from Elena Rostova.

---

Invariants and Contracts

Eighty-Five Percent Mutation Floor Invariant [INV-MUTATION-01]
  Pull requests modifying financial ledger or balance logic must achieve >= 85.0% mutation score.
  High line coverage alone cannot satisfy CI release gates without passing mutation thresholds.

Mandatory Git Diff Scoping Invariant [INV-MUTATION-02]
  Pre-merge mutation testing must execute incrementally scoped to the active pull request Git diff.
  Pull request pipelines exceeding the 6.0-minute execution budget fail automated performance checks.

Zero Assertion-Free Test Acceptance [INV-MUTATION-03]
  Test cases that execute domain logic but kill 0% of injected mutators are flagged as assertion-free.
  Pull requests containing assertion-free test suites are blocked from merging.

Explicit Unknowns

• Kotlin compiler inline function bytecode expansion impact on PITest mutant generation cardinality (G-1).
• PITest history cache synchronization contention when 30 developers open concurrent pull requests (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against mutation testing standards:

• Efficacy Rigor: PASS. 85% mutation score floor eliminates assertion-free test illusions.
• Operator Relevance: PASS. Focused on math, boundary, negative, and return value mutations.
• Pipeline Speed: PASS. Incremental Git diff scoping keeps PR feedback under 5 minutes.
• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-MUTATION-01: Elena Rostova to determine whether weekly full-codebase mutation runs should be scheduled on weekend batch runners to track legacy technical debt (Owner: Elena Rostova).

Next steps

1. Marcus Vance embeds the PITest Maven plugin with Git diff features into the core ledger POM configuration.
2. Quality Engineering conducts baseline mutation analysis across the calculation package to establish initial mutant history caches.
3. Enforce the 85% mutation threshold gate in GitHub Actions branch protection rules.