Graceful Kubernetes Shutdowns: Draining Traffic Without Dropping Work

A deployment can have perfect health checks and still return errors during every rollout.

The reason is that Kubernetes termination is not a single atomic event. Pod deletion, endpoint removal, load-balancer updates, signal delivery, connection closure, and application shutdown happen across different components and on different timelines. A process that exits immediately on SIGTERM can disappear before traffic stops arriving. A process that waits too long can be killed before its requests or jobs finish.

A reliable shutdown therefore needs an explicit protocol: stop advertising capacity, allow routing changes to propagate, stop accepting new work, finish or safely abandon existing work, and exit before Kubernetes enforces the deadline.

What happens when Kubernetes terminates a pod

A pod may terminate because of a rollout, scale-down, eviction, manual deletion, or node shutdown. For ordinary graceful termination, the sequence is approximately:

1. The pod receives a deletion timestamp and enters the terminating state.
2. Kubernetes begins updating endpoint information so routing components stop selecting it.
3. The kubelet starts the pod's termination grace-period countdown.
4. The kubelet runs any configured preStop hooks.
5. After a container's hook finishes, the runtime asks its main process to terminate—normally with SIGTERM unless a different stop signal applies.
6. If containers remain after the grace period expires, Kubernetes forcibly stops them, typically with SIGKILL.

These steps overlap with asynchronous control-plane and network updates:

sequenceDiagram
    participant C as Client
    participant LB as Load balancer
    participant K as Kubernetes
    participant P as Pod

    K->>P: Pod termination starts
    K-->>LB: Endpoint becomes terminating
    K->>P: Run preStop hook
    C->>LB: New request
    LB->>P: Route using stale endpoint state
    P-->>C: Complete request
    K->>P: Send SIGTERM
    P->>P: Stop intake and drain work
    P-->>K: Process exits

The important detail is that endpoint removal is not synchronized with signal handling. EndpointSlice updates must reach kube-proxy, an ingress controller, service mesh, cloud load balancer, or another routing layer. Some components also maintain their own health-check intervals and caches.

Consequently, traffic may continue reaching a terminating pod for a short period after deletion starts. This is expected distributed-system behavior, not necessarily a Kubernetes defect.

Readiness is routing intent, not an instant traffic barrier

Readiness probes answer whether a pod should receive new traffic. When readiness fails, Kubernetes updates the pod's endpoint state, but that does not instantly recall every copy of the old routing decision.

Several delays can exist:

• The next readiness probe may not run immediately.
• The API server and EndpointSlice controllers must publish state changes.
• Node-level proxies, ingress controllers, and service meshes must consume them.
• External load balancers may use separate health checks or deregistration delays.
• Clients may reuse TCP, HTTP keep-alive, or HTTP/2 connections that already target the pod.

During pod deletion, modern Kubernetes endpoint data represents terminating endpoints so consumers can avoid using them for new traffic. However, the full path still depends on each routing component observing and honoring that state.

This explains a common rollout failure:

1. Readiness is healthy immediately before deletion.
2. Kubernetes starts removing the endpoint.
3. The application receives SIGTERM and exits immediately.
4. A proxy with stale endpoint state sends another request.
5. The client receives a reset, refusal, or gateway error.

The readiness probe was accurate when sampled. It simply did not provide atomic coordination between routing and process exit.

Separate liveness from readiness

Liveness and readiness have different purposes:

• Liveness asks whether Kubernetes should restart the container.
• Readiness asks whether the pod should receive traffic.

A draining process is usually alive but not ready. Do not make liveness fail during normal draining; a restart adds noise and can interrupt the shutdown protocol.

Readiness should return failure as soon as the application enters its draining state. During normal operation, it can also check whether the process can serve requests, but avoid coupling it to every transient downstream dependency. Overly sensitive readiness checks can remove too much capacity at once.

Build shutdown as a state machine

A useful application model has three states:

stateDiagram-v2
    [*] --> Ready
    Ready --> Draining: termination requested
    Draining --> Stopped: work completed or deadline reached
    Stopped --> [*]

The transition to Draining should be one-way and idempotent. Repeated signals or hook calls must not restart intake or launch multiple shutdown routines.

A typical HTTP shutdown sequence is:

1. Mark the application unready.
2. Wait briefly for endpoint changes to propagate.
3. Stop accepting new connections or requests.
4. Allow active requests to complete within a bounded timeout.
5. Cancel remaining work and close resources.
6. Exit cleanly.

The propagation delay is environment-specific. It should be measured across the actual ingress, service mesh, and load balancer path rather than copied blindly from an example.

Handling signals and in-flight HTTP requests

The application must receive termination signals correctly. In a container, the application often runs as PID 1. Shell wrapper scripts that do not use exec may absorb signals or fail to forward them.

Prefer:

ENTRYPOINT ["/app/server"]

If a wrapper is necessary, replace the shell process:

#!/bin/sh
set -eu
exec /app/server "$@"

The following simplified Go server demonstrates the key mechanics:

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "sync/atomic"
    "syscall"
    "time"
)

func main() {
    var draining atomic.Bool

    mux := http.NewServeMux()
    mux.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
        if draining.Load() {
            http.Error(w, "draining", http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    })
    mux.HandleFunc("/work", handleWork)

    server := &http.Server{
        Addr:              ":8080",
        Handler:           mux,
        ReadHeaderTimeout: 5 * time.Second,
    }

    errCh := make(chan error, 1)
    go func() {
        errCh <- server.ListenAndServe()
    }()

    signalCtx, stop := signal.NotifyContext(
        context.Background(),
        syscall.SIGTERM,
        syscall.SIGINT,
    )
    defer stop()

    select {
    case err := <-errCh:
        if !errors.Is(err, http.ErrServerClosed) {
            log.Fatal(err)
        }
        return
    case <-signalCtx.Done():
    }

    draining.Store(true)

    // Leave time for readiness and routing changes to propagate.
    time.Sleep(5 * time.Second)

    shutdownCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
    defer cancel()

    if err := server.Shutdown(shutdownCtx); err != nil {
        log.Printf("graceful shutdown expired: %v", err)
        _ = server.Close()
    }
}

func handleWork(w http.ResponseWriter, r *http.Request) {
    // Perform request work using r.Context() where cancellation is appropriate.
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ok"))
}

Server.Shutdown stops accepting new connections, closes idle connections, and waits for active handlers to return. The timeout bounds that wait. The final Close prevents the process from hanging indefinitely after the budget expires.

This is a starting point, not a complete policy. Applications must decide which operations should honor request cancellation. Canceling a database query after the client disconnects may save capacity, while canceling midway through an externally visible side effect may leave ambiguous results. Transactions, idempotency keys, and retry-safe APIs are still necessary.

Also verify the behavior of the actual server stack. HTTP/2 multiplexes many streams over one connection, gRPC has long-lived streams, and WebSockets may remain open indefinitely. Graceful shutdown may send protocol-level drain signals, reject new streams, or simply wait for connections depending on the framework and proxy. Put explicit maximum lifetimes or shutdown policies around long-lived sessions.

Using preStop without consuming the whole deadline

A preStop hook can create a propagation buffer before SIGTERM reaches the application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  template:
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          image: example/orders-api:1.2.3
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 2
            failureThreshold: 1
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]

This works because pod termination and endpoint updates begin while the process remains available to handle requests routed using stale state. After the hook finishes, the application receives SIGTERM and begins its own drain sequence.

There are important tradeoffs:

• The termination grace-period countdown includes preStop execution.
• A long sleep reduces the time available for real work to finish.
• Minimal or distroless images may not contain /bin/sh or sleep.
• A fixed delay should cover observed propagation, not hide an unbounded shutdown.

An alternative is a preStop HTTP endpoint that atomically marks the application as draining. That can begin draining earlier, but it adds an internal administrative API that must be idempotent, secured from ordinary traffic, and designed so the hook does not block forever.

Avoid accidentally double-counting delays. In the example above, the five-second hook plus the application's five-second post-signal wait consume roughly ten seconds before active-request draining. With a 45-second grace period and a 25-second server timeout, only about ten seconds remain for startup overhead, hook variance, cleanup, and runtime behavior. Every phase must fit within one shared budget.

Background workers need different shutdown semantics

HTTP servers stop accepting requests and wait for handlers. Queue consumers must stop claiming jobs, then decide what to do with jobs already claimed.

A safe worker sequence is:

1. Mark the worker unready if readiness controls job delivery or operational capacity.
2. Pause polling, subscription intake, or message prefetch.
3. Finish jobs that can complete before the deadline.
4. Acknowledge a job only after its durable side effects succeed.
5. Release, negatively acknowledge, or allow the lease to expire for unfinished jobs.
6. Close clients and exit.

The exact mechanism depends on the queue, but the invariants are portable. Work must either complete and be acknowledged or remain eligible for retry. Acknowledging at receipt time creates data-loss risk; acknowledging too late creates duplicate processing.

Because forced termination, process crashes, and node failures can occur without a complete graceful sequence, handlers should be idempotent. Useful techniques include unique operation keys, database constraints, transactional outbox patterns, and compare-and-set state transitions.

Visibility or lease timeouts also need alignment with shutdown behavior. If a worker may take 30 seconds to finish, a 10-second lease without renewal can cause another worker to process the same job concurrently. Conversely, a very long lease delays recovery when a pod dies abruptly.

For a process that serves HTTP and consumes jobs, coordinate both subsystems under one deadline: stop job intake first, remove HTTP readiness, drain requests and current jobs, then close shared databases and clients. Closing shared dependencies before handlers stop is a frequent source of shutdown-time errors.

Race conditions that survive healthy probes

Several deployment failures recur even in mature systems:

Exiting immediately on SIGTERM

Endpoint updates are still propagating, so stale routes reach a closed socket. Add a measured propagation window and graceful server shutdown.

Remaining ready while draining

The application stops accepting work but /ready still returns success. Routing components continue selecting a pod that is intentionally rejecting requests. Make draining part of readiness state.

Closing dependencies before work finishes

A shutdown handler closes the database pool, then active requests fail while committing. Stop intake, wait for work, and close shared resources last.

Letting one request consume the full grace period

If application and Kubernetes deadlines expire together, Kubernetes may send SIGKILL before cleanup completes. Set the application's shutdown timeout shorter than terminationGracePeriodSeconds and reserve margin.

Ignoring persistent connections

Removing an endpoint prevents new selections, but existing keep-alive, HTTP/2, gRPC, or WebSocket connections may continue sending work. Verify protocol-specific draining across the application and proxy.

Assuming all termination is graceful

preStop and SIGTERM cannot protect against every kernel crash, out-of-memory kill, hardware failure, or network partition. Graceful shutdown reduces routine rollout failures; durable queues, retries, idempotency, and replication handle abrupt loss.

Relying on a PodDisruptionBudget

A PodDisruptionBudget limits certain voluntary disruptions so enough replicas remain available. It does not make a terminating pod drain correctly, prevent all concurrent failures, or control application shutdown ordering.

Define and test a shutdown budget

Treat terminationGracePeriodSeconds as a budget rather than a generous constant. A 45-second budget might be divided into:

• 5 seconds for preStop and initial endpoint propagation
• 5 seconds for application-level readiness propagation
• 25 seconds for requests or jobs to finish
• 5 seconds for resource cleanup
• 5 seconds of safety margin

Instrument each phase. Useful observations include:

• Time from deletion timestamp to readiness failure
• Time until the pod receives no new requests
• Number and age of active requests at termination
• Jobs completed, retried, or abandoned during shutdown
• Graceful shutdown timeouts and forced kills
• Deployment-time rates of resets, gateway errors, and canceled operations

Test the real path by repeatedly deleting pods under load. Include slow requests, keep-alive reuse, HTTP/2 or gRPC if applicable, queue jobs near the maximum duration, and multiple simultaneous terminations. A unit test of the signal handler cannot reveal propagation delays in an ingress controller or cloud load balancer.

Rollout configuration matters as well. maxUnavailable, maxSurge, replica count, and readiness behavior determine whether enough capacity remains while old pods drain and new pods warm up. Correct per-pod shutdown cannot compensate for a rollout that removes capacity faster than replacements become ready.

Conclusion

Graceful Kubernetes shutdown is a coordination problem across the application, kubelet, control plane, proxies, load balancers, clients, and job systems. Readiness probes express routing intent, but they do not create an instantaneous traffic barrier.

A production-ready design enters an explicit draining state, allows endpoint changes to propagate, stops accepting new work, completes or safely retries in-flight operations, and exits with margin before Kubernetes enforces its deadline. Once this behavior is bounded, observable, and tested under real traffic, routine deployments stop behaving like small outages.