Python

daemonic processes are not allowed to have children

Written and reviewed by Sahil Srivastav

PythonmultiprocessingProcess topology
AssertionError: daemonic processes are not allowed to have children
  File "multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon')

What this error actually means

Python marks pool workers daemon-like so they cannot outlive the parent and leave orphaned grandchildren. The assertion fires before the child starts; it is an intentional topology rule, not an OS permission problem.

Nested process pools are usually a sign that ownership of parallelism is unclear. If an outer pool already has one process per CPU, each worker creating another pool oversubscribes cores and multiplies memory. Threads inside a process may be appropriate for I/O, while CPU work should have one bounded process layer.

The fix is to choose one component as the process supervisor and pass work to it, rather than trying to defeat the daemon flag.

Causes, most common first

  1. 1A pool worker creates another Process. The worker inherits the daemon setting from the pool.
  2. 2A task library hides a nested pool. Data or ML libraries may create workers inside an already managed worker.
  3. 3Process topology is configured at multiple layers. Web server, task runner, and application each add concurrency.

When you see it

  • A task succeeds serially but fails inside Pool.map
  • The traceback points to Process.start assertion
  • Nested libraries such as joblib or BLAS create unexpected workers
  • CPU usage and memory spike when nesting is attempted

How to diagnose it

Step 1

Print process and daemon state

Confirm which layer owns the failing call.

python - <<'PY'
import multiprocessing as mp
print(mp.current_process().pid, mp.current_process().daemon)
PY

Step 2

List worker creation sites

Search direct and indirect executors; inspect library settings for worker counts.

rg -n 'Pool\(|ProcessPoolExecutor|n_jobs|workers|parallel' .

Step 3

Measure oversubscription

Compare process count and CPU quota while the task runs.

ps -eLf | rg 'python|celery' | wc -l
nproc

The fix

Flatten the topology: let the outer scheduler own process parallelism and make inner code serial.

Use a thread pool for blocking I/O inside a process when the library is thread-safe.

Move nested CPU work to a separate queue or service with an explicit resource budget.

If a third-party library spawns workers, configure its worker count to one inside the process pool.

Do not monkey-patch daemon flags; cleanup and failure semantics become your responsibility.

# outer pool owns CPU parallelism
with multiprocessing.Pool(processes=4) as pool:
    results = pool.map(process_one, items)

def process_one(item):
    # no nested ProcessPool here; use bounded synchronous work
    return transform(item)

How to stop it coming back

  • Document one concurrency owner per service
  • Set BLAS and library worker counts explicitly
  • Test under the real container CPU quota
  • Track child process counts and RSS
  • Prefer queue-based fan-out over recursive process creation

Practise production debugging in a real repository

Reading about a failure and reproducing one are different skills. Gronex ships broken backend repositories with failing test suites that encode the real invariant, so you debug from evidence instead of memorising symptoms.

FAQ

Can I subclass Pool to make workers non-daemonic?

Possible hacks exist, but they bypass the lifecycle guarantee and are fragile across Python versions. Flatten the topology or move the work to a supervisor.

Should I use threads instead?

For I/O-bound inner work, often yes. For pure Python CPU work, use one process layer because the GIL still applies to threads.

Why does this happen only in production?

Production enables the outer worker pool or task runner, exposing a nested pool that local serial execution never reaches.

Related

Celery WorkerLostError

Other errors engineers hit next to this one

Full error and symptom index →