Repair DNS, Discovery & Endpoint Failures

I have analyzed the connection logic and identified a stale endpoint caching issue within the application-level connection pool, which is ignoring the DNS TTL during pod reshuffles.

Repair for Stale Endpoint Resolution

The following fix introduces a TTL-aware resolver and forces the connection pool to refresh its internal cache when a connection is dropped or a threshold is met, ensuring it doesn't rely on OS-level caching that might be stale.

const dns = require('dns').promises;
const { Pool } = require('pg'); // Example using pg-pool

/**
 * Enhanced resolver that respects TTL and handles address-family selection.
 */
class DiscoveryResolver {
  constructor(hostname, ttl = 30000) {
    this.hostname = hostname;
    this.ttl = ttl;
    this.cache = null;
    this.lastLookup = 0;
  }

  async resolve() {
    const now = Date.now();
    if (this.cache && (now - this.lastLookup) < this.ttl) {
      return this.cache;
    }

    // Resolve using address family 4 to avoid IPv6 stalls if not supported
    const addresses = await dns.resolve4(this.hostname);
    this.cache = addresses[Math.floor(Math.random() * addresses.length)];
    this.lastLookup = now;
    return this.cache;
  }
}

// Invariant: Connection pool must refresh address if connection fails
const resolver = new DiscoveryResolver('db.internal.svc.cluster.local');

const pool = new Pool({
  host: async () => await resolver.resolve(),
  max: 20,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000
});

pool.on('error', (err, client) => {
  console.error('Connection failure, invalidating cache');
  resolver.lastLookup = 0; // Force refresh on next attempt
});

Deterministic Test Plan

Next steps

1. Verify the NDOTS setting in your resolv.conf to ensure internal lookups aren't suffering from excessive search suffix traversals.
2. Check if the database client library supports dynamic host resolution; some libraries bind the IP at instantiation.
3. Apply the DiscoveryResolver pattern to your other internal microservices.