Linux / shell

Graceful shutdown ignored because SIGTERM reaches the wrong process

Written and reviewed by Sahil Srivastav

ContainersPID 1Graceful shutdown
Exit Code:    137

What this error actually means

The container runtime signals the container’s main process during ordinary termination, normally with SIGTERM unless the stop signal is configured differently. If that process is a shell wrapper and the application is its child, the signal may never reach the application. When the grace period expires, forced termination removes any opportunity to finish work cleanly.

An exec-form Dockerfile instruction avoids an implicit shell, but it is not a guarantee that the application is the main process. ENTRYPOINT can launch a script, and that script can still start the server as a child. The last step of a simple single-process wrapper should replace the shell with the server using exec.

Signal delivery and graceful behaviour are separate requirements. The correct process must receive the signal, install an appropriate handler, stop accepting new work and complete or safely abandon existing work within a deadline. Exit 137 is consistent with the final forced kill, but also occurs for OOM kills; tie it to the shutdown timeline before diagnosing a signal-routing bug.

Causes, most common first

  1. 1Shell-form startup leaves a wrapper as the main process. CMD app or ENTRYPOINT app can invoke a shell, which may not forward the stop signal to its child. Inspect the actual process tree because some shells optimise simple commands. The Dockerfile’s visual shape alone does not establish the running PID relationship.
  2. 2An entrypoint script starts the application without exec. The image uses JSON-form ENTRYPOINT but the script ends with java, node or a backgrounded server. The wrapper remains responsible for signals and waiting. Adding another wrapper for environment setup can reintroduce the bug after an otherwise correct Dockerfile change.
  3. 3The application receives the signal but cannot finish. A handler waits on an outbound call with no deadline, a worker pool cannot drain, or shutdown closes dependencies before workers release them. This is a shutdown-ordering bug, not a PID-routing bug; increasing the grace period only helps if progress is actually being made.
  4. 4Lifecycle hooks consume the available shutdown time. In Kubernetes, preStop work is part of the termination budget. A long sleep or stalled hook can leave little time for the application after it receives TERM. Endpoint and load-balancer updates also propagate asynchronously, so draining needs more than assuming traffic stops instantly.

When you see it

  • Every rollout takes almost the entire termination grace period
  • Application shutdown-hook logs never appear even though pod termination starts
  • The process handles SIGTERM in a local terminal but not through docker stop
  • In-flight requests reset or queue jobs repeat during deployments despite healthy steady-state traffic

How to diagnose it

Step 1

Read the process tree inside the container

Replace demo, app-pod and app with the real workload. These commands require ps and cat in the image; for a minimal image use an approved runtime inspection method. In a shared process namespace, PID numbering differs, so identify the runtime-designated main process rather than assuming the application must literally be PID 1.

kubectl -n demo exec app-pod -c app -- ps -eo pid,ppid,args
kubectl -n demo exec app-pod -c app -- cat /proc/1/cmdline

Step 2

Inspect the whole launch chain

Compare image Entrypoint and Cmd with pod command and args overrides, then read every wrapper script. Search for a final foreground command without exec, an ampersand, or a supervisor whose forwarding policy is unspecified.

docker image inspect registry.example.com/team/app:release-42 --format '{{json .Config.Entrypoint}} {{json .Config.Cmd}}'
kubectl -n demo get pod app-pod -o jsonpath='{.spec.containers[*].command}{"\n"}{.spec.containers[*].args}{"\n"}'

Step 3

Record a controlled shutdown with work in flight

In staging, start a bounded long request or job, terminate the container through the ordinary platform path, and record handler entry, readiness change, last completed operation and process exit. Signalling the child directly can prove the handler works while bypassing the launch-chain defect you need to test.

Step 4

Compare hooks with the total grace budget

Read preStop and terminationGracePeriodSeconds together. If the handler starts but cannot finish, capture thread or task state during that interval. If it never starts, verify the configured stop signal and process ownership before extending timeouts.

kubectl -n demo get pod app-pod -o jsonpath='{.spec.terminationGracePeriodSeconds}{"\n"}{range .spec.containers[*]}{.name}{"\t"}{.lifecycle}{"\n"}{end}'

The fix

Use exec-form ENTRYPOINT for a single foreground application. If a wrapper must prepare configuration, let it finish setup and then exec the command with preserved argument boundaries. The exec replaces the shell instead of introducing another child, allowing the runtime’s signal to reach the application directly.

When the container intentionally manages several child processes, use a suitable init or supervisor with explicit signal forwarding, child reaping and exit-status behaviour. A minimal init helps with those responsibilities but cannot implement the application’s transactional drain or decide when a job is safe to acknowledge.

In the application handler, stop new admissions, begin draining and impose a deadline on outstanding operations. Keep dependencies alive until the workers that use them finish, then flush bounded buffers and close resources. For jobs that cannot finish, leave ownership or acknowledgement in a state that permits safe retry.

Budget propagation delay, preStop work and application draining within the platform’s grace interval. Increase that interval only when measurements justify the required work. Validate request completion and durable outcomes during the test; a clean process exit alone cannot prove that no accepted operation was lost.

# Dockerfile: launch the wrapper directly.
COPY --chmod=755 entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["node", "server.js"]

# entrypoint.sh: replace the shell after any bounded setup.
#!/bin/sh
set -eu
exec "$@"

# The application must still install its SIGTERM handler and drain work.

How to stop it coming back

  • Test the built image through the normal stop path instead of killing only the child process
  • Assert accepted work completes or remains safely retryable across a rollout
  • Log shutdown phases with timestamps so missing delivery differs from slow draining
  • Review new entrypoint wrappers for argument preservation, exec and child-process ownership

Practise production debugging in a real repository

Reading about a failure and reproducing one are different skills. Gronex ships broken backend repositories with failing test suites that encode the real invariant, so you debug from evidence instead of memorising symptoms.

FAQ

Does JSON-form ENTRYPOINT solve every signal problem?

No. It removes an implicit shell, but a script or supervisor named by that entrypoint can still fail to forward signals. Inspect the resulting process tree and test the actual stop path.

Can a SIGKILL handler flush the final writes?

No. SIGKILL cannot be caught or handled. Flush during the graceful interval, use bounded operations, and design durable work so an abrupt interruption can be recovered safely.

Why does adding a long preStop sleep not fix lost requests?

It may give routing changes time to propagate, but it also consumes the shutdown budget. It does not implement signal forwarding, stop new admissions or guarantee completion of in-flight work. Measure the entire termination sequence.

Related

Other causes of exit code 137Readiness and traffic removalExecutor shutdown hangs

Other errors engineers hit next to this one

Full error and symptom index →