A Kubernetes container can be running while the application is still warming a cache, unable to reach a critical dependency, or deadlocked. Treating every health question as the same boolean creates two expensive failure modes: traffic reaches Pods too early, or the kubelet restarts a process that only needed temporary relief from traffic.

Kubernetes provides three probes with separate jobs. A startup probe protects slow initialization. A readiness probe controls whether a Pod receives Service traffic. A liveness probe decides when a container should restart. This tutorial designs lightweight HTTP endpoints, configures all three probes in a Deployment, tunes their timing from measurements, and shows how to diagnose probe failures in production.

Give each probe one responsibility

  • Startup: Has initialization completed? Until it succeeds, Kubernetes does not run readiness or liveness probes.
  • Readiness: Can this instance accept new traffic now? Failure removes the Pod from matching Service EndpointSlices without restarting it.
  • Liveness: Is the process irrecoverably stuck? Repeated failure causes the kubelet to restart the container according to its restart policy.

A dependency outage should usually make an instance unready, not dead. If every Pod restarts because a shared database is unavailable, startup load and reconnect storms can make the incident worse.

Design cheap and honest health endpoints

For an HTTP service, expose small endpoints that do not require authentication and return quickly only from the Pod network. Their responses should not include secrets, stack traces, versions, or dependency credentials.

// Express example
let startupComplete = false


app.get('/health/startup', (req, res) => {
  res.sendStatus(startupComplete ? 204 : 503)
})


app.get('/health/live', (req, res) => {
  res.sendStatus(eventLoopIsMakingProgress() ? 204 : 503)
})


app.get('/health/ready', async (req, res) => {
  const acceptingTraffic =
    startupComplete &&
    !isShuttingDown &&
    await databasePool.canBorrowConnection({ timeoutMs: 200 })


  res.sendStatus(acceptingTraffic ? 204 : 503)
})

The liveness endpoint checks whether this process can make progress. It deliberately avoids a full database query, third-party API call, or deep business transaction. Readiness may include a small set of dependencies required to serve requests, but it still needs strict timeouts and bounded work.

Configure startup, readiness, and liveness together

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: api
          image: ghcr.io/example/checkout-api:2.4.1
          ports:
            - name: http
              containerPort: 3000


      startupProbe:
        httpGet:
          path: /health/startup
          port: http
        periodSeconds: 2
        timeoutSeconds: 1
        failureThreshold: 30

      readinessProbe:
        httpGet:
          path: /health/ready
          port: http
        periodSeconds: 5
        timeoutSeconds: 1
        failureThreshold: 2
        successThreshold: 1

      livenessProbe:
        httpGet:
          path: /health/live
          port: http
        periodSeconds: 10
        timeoutSeconds: 1
        failureThreshold: 3</code></pre>



The startup budget is failureThreshold × periodSeconds: this example allows roughly 60 seconds before startup failures trigger a restart. Once startup succeeds, readiness and liveness begin. Named ports improve readability for HTTP and TCP probes; built-in gRPC probes require a numeric port.



Tune timing from observed behavior


Probe values are operational policy, not magic defaults. Measure cold startup under realistic CPU and memory limits, including image initialization, migrations handled elsewhere, JIT compilation, and cache warmup. Set the startup budget above a high percentile plus a reasonable margin.


For readiness, choose a period and failure threshold that remove an unhealthy instance quickly without reacting to a single scheduling pause. A five-second period and two failures means removal generally begins after about ten seconds, plus processing delay. Liveness should be more conservative because a false positive kills useful work.


Keep timeoutSeconds below the probe period. If a readiness handler calls dependencies, its internal timeouts should be shorter than the probe timeout so it can return a controlled failure rather than being cut off.



Make shutdown cooperate with readiness


During termination, stop accepting new application work before the process exits. Handle SIGTERM, mark readiness false, drain active requests, and close resources within terminationGracePeriodSeconds.


let isShuttingDown = false


process.on('SIGTERM', async () => {
  isShuttingDown = true
  server.close(async () => {
    await databasePool.end()
    process.exit(0)
  })
})


Readiness transitions and Service routing are distributed operations, so existing connections and a small amount of in-flight traffic can remain. Applications still need graceful request handling, idempotency, and timeouts.



Choose the right probe mechanism


httpGet is usually the clearest option for HTTP applications because it can test an application-level path. A tcpSocket probe proves only that a port accepts a connection. An exec probe runs a command inside the container and can add process overhead; avoid shell-heavy scripts. A built-in grpc probe works with the gRPC health checking protocol and has been stable since Kubernetes 1.27.


Each probe must define exactly one mechanism. Probe endpoints should listen on the Pod interface and port the kubelet can reach; testing only through an external ingress adds unrelated network components and changes the question being asked.



Validate probes before rollout


kubectl apply --server-side --dry-run=server -f deployment.yaml
kubectl apply -f deployment.yaml
kubectl rollout status deployment/checkout-api --timeout=5m


kubectl get pods -l app=checkout-api -w
kubectl get endpointslices -l kubernetes.io/service-name=checkout-api
kubectl describe pod <pod-name>


kubectl describe pod shows probe-related events such as timeouts, refused connections, and HTTP failure codes. EndpointSlices reveal whether ready Pods are available to the Service. Also graph container restarts, ready replica count, probe duration, and application-level errors during the rollout.



Troubleshooting common failures


Pods enter CrashLoopBackOff during startup


Check events and previous-container logs with kubectl logs <pod> --previous. If the application eventually starts but liveness kills it first, add or enlarge a startup probe. Also confirm CPU limits are not stretching initialization far beyond your test measurements.


Ready Pods receive errors immediately


The readiness endpoint may report success before routes, caches, connection pools, or configuration are usable. Make the application set its startup-complete state only after required initialization, and test the real request path during a staged rollout.


A database incident restarts every Pod


Remove the database check from liveness. Use readiness to pause new traffic if the API cannot function without the database. Keep liveness focused on local process progress.


Probes time out only under load


Health handlers may be competing with ordinary requests on a saturated event loop or worker pool. Reserve capacity, keep handlers constant-time, examine CPU throttling and garbage collection, and set resource requests that reflect actual demand. Raising timeouts alone can hide saturation.


HTTPS or gRPC probes fail unexpectedly


Verify the mechanism against the Kubernetes version and feature gates used by the cluster. Current Kubernetes 1.37 documentation includes alpha options for h2c HTTP probes and TLS mode for gRPC probes; alpha features are disabled by default and should not be assumed portable across clusters.



Production checklist


    
  
  • Use separate endpoints and semantics for startup, readiness, and liveness.
    • 
  
  • Keep liveness local, cheap, and conservative.
    • 
  
  • Give readiness dependency checks strict, short timeouts.
    • 
  
  • Derive startup budgets from cold-start measurements under real limits.
    • 
  
  • Mark readiness false and drain requests during graceful shutdown.
    • 
  
  • Validate manifests server-side and observe EndpointSlices during rollout.
    • 
  
  • Alert on restart rate, unavailable replicas, and sustained readiness failures.
    • 
  
  • Test dependency outages, CPU pressure, slow startup, and termination before production.
    • 




Official sources