Repair Spark Streaming Watermark Late Events

Repair Spark Streaming Watermark Late Events — EVIDENCE-LIMITED REVIEW

Scenario received

Valid orders arrive after a watermark cutoff and disappear from an hourly aggregation.

Evidence inventory

• S1: event time samples — partial extract described by the requester; original full source not provided here.
• S2: watermark configuration — partial extract described by the requester; original full source not provided here.
• S3: checkpoint state metadata — NOT PROVIDED; required to validate dependent findings.
• S4: streaming query progress logs — NOT PROVIDED; required to validate dependent findings.

Case-specific checks

Compare event and processing time distributions. INCOMPLETE until matching source anchors can be examined; document the observable facts, counterexamples and decision gate.
2.

Identify state eviction horizons. INCOMPLETE until matching source anchors can be examined; document the observable facts, counterexamples and decision gate.
3.

Trace restart checkpoint compatibility. INCOMPLETE until matching source anchors can be examined; document the observable facts, counterexamples and decision gate.
4.

Simulate boundary late arrivals. INCOMPLETE until matching source anchors can be examined; document the observable facts, counterexamples and decision gate.

Deliverables

Late event loss analysis. Create an evidence-linked section with observations, missing data, owner and next validation step.

Watermark adjustment plan. Create an evidence-linked section with observations, missing data, owner and next validation step.

Replay boundary test suite. Create an evidence-linked section with observations, missing data, owner and next validation step.

Acceptance gate

The solution documents unavoidable late data tradeoffs and avoids silent historical rewrites. Do not label this gate passed until supplied source records actually demonstrate it.

Verdict and next action

INCOMPLETE — evidence required. Request S3 and S4 from the authorized owner, reconcile them with S1 and S2, then repeat the four checks and log remaining conflicts. No production change, approval, communication, or test run is claimed.