Repair Multi-Region Failover Races

I have analyzed the failover race condition between us-east-1 and us-west-2. The issue stems from a lack of generation identity in your worker heartbeats, allowing the 'degraded' region to continue processing the task queue without verifying its current authority status.

Regional Write Fencing Implementation

To resolve this, we must implement a monotonic generation counter (fence token) stored in a global consensus layer (like DynamoDB or Consul). Every write must include this token, and the storage layer must reject writes with a lower generation than the current global maximum.

// Proposed Fencing Logic for Task Workers
interface TaskMetadata {
  id: string;
  generation: number; // The fencing token
  region: string;
}

async function processTask(task: TaskMetadata) {
  const currentAuthority = await authorityService.getGlobalGeneration();
  
  // 1. Pre-flight check
  if (task.generation < currentAuthority.generation) {
    console.error(`Fencing violation: Task gen ${task.generation} < Current ${currentAuthority.generation}`);
    return; // Drop task, it belongs to a stale region
  }

  // 2. Atomic Write with Condition Expression
  try {
    await db.update({
      TableName: 'RegionalTasks',
      Key: { id: task.id },
      UpdateExpression: 'SET val = :val, gen = :gen',
      // Ensure the write only happens if no newer generation has taken over
      ConditionExpression: 'attribute_not_exists(gen) OR gen <= :gen',
      ExpressionAttributeValues: {
        ':val': task.data,
        ':gen': task.generation
      }
    });
  } catch (err) {
    if (err.code === 'ConditionalCheckFailedException') {
      // Logic for handling a write that arrived after a newer region took over
      await handleStaleWrite(task);
    }
  }
}

Deterministic Test Plan

1. Initialize Authority: Set global generation to 10.
2. Simulate Lag: Start a worker in us-east-1 with generation: 10.
3. Trigger Failover: Update global generation to 11 (simulating us-west-2 takeover).
4. Execute Stale Write: Attempt the us-east-1 write.
5. Verify Invariant: Assert the database rejects the write and the us-east-1 worker enters a 'dormant' state.

Next steps

• Audit the authorityService to ensure it uses a strongly consistent read for generation checks.
• Update the CI/CD pipeline to increment the global generation counter during regional traffic shifts.
• Implement a 'drain' signal for workers that detects generation mismatches via heartbeats.