Repair Health Check Failure Cascades

The service is suffering from a startup latency mismatch. The liveness probe is active before the application has completed its initialization phase, leading to a SIGTERM during the model load.

Diagnostic & Repair Plan

The current configuration lacks a startupProbe, forcing the livenessProbe to handle both initial boot and ongoing health. I have defined a new startup grace period that allows the model to load while keeping liveness checks tight for post-boot failures.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-inference-api
spec:
  template:
    spec:
      containers:
      - name: api
        image: ml-api:v2.1.0
        # Startup probe gives the model 90s to load (30 * 3s)
        startupProbe:
          httpGet:
            path: /healthz/startup
            port: 8080
          failureThreshold: 30
          periodSeconds: 3
        # Liveness probe only takes over after startup succeeds
        livenessProbe:
          httpGet:
            path: /healthz/liveness
            port: 8080
          failureThreshold: 3
          periodSeconds: 10
        # Readiness probe ensures traffic only hits when model is warm
        readinessProbe:
          httpGet:
            path: /healthz/readiness
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Assumptions

• The Go service exposes a /healthz/startup endpoint that returns 200 OK only after the model is fully resident in memory.
• The 2GB model load time is consistently under 90 seconds in the target environment.

Next steps

1. Update the /healthz/startup endpoint in your Go source to verify model checksums.
2. Apply the modified manifest to your staging namespace.
3. Monitor kubectl get pods -w to verify the transition from startup to readiness.