Blocking calls inside async def starving the event loop

What this error actually means

An asyncio application runs all your coroutines on one thread. Concurrency comes entirely from coroutines yielding at `await` points; between two `await`s, your code owns the thread exclusively. A synchronous call that takes 200 milliseconds therefore does not cost 200 milliseconds of latency to one request — it adds 200 milliseconds to every request currently in flight, plus everything queued behind them, plus the health check, plus the heartbeat that keeps your Kafka or database connection registered.

That multiplication is why a blocked loop presents as a whole-service problem with no single slow endpoint. Percentiles rise together, timeouts appear in unrelated code, and the traces show time spent waiting rather than working. The endpoint doing the blocking often looks fine in its own metrics.

This is also the most easily introduced async bug, because the blocking call is usually correct code used in the wrong context: `requests.get` instead of an async client, `psycopg2` instead of `asyncpg`, `time.sleep` instead of `asyncio.sleep`, a `bcrypt` hash, a 30 MB `json.loads`, a Pillow resize, a `pandas` group-by. None of them raise. None of them look like a concurrency mistake. The only signal is lag.

asyncio will tell you, if you ask. In debug mode the loop times every callback and logs `Executing <Handle ...> took N seconds` for anything exceeding `slow_callback_duration` (0.1s by default). That log line names the source location — it is the cheapest possible diagnostic for this entire class of problem.

How to diagnose it

PYTHONASYNCIODEBUG=1 python -X dev app.py
# or, in code:
loop = asyncio.get_running_loop()
loop.set_debug(True)
loop.slow_callback_duration = 0.05

async def lag_monitor(interval=0.25):
    while True:
        start = time.perf_counter()
        await asyncio.sleep(interval)
        LAG.observe(time.perf_counter() - start - interval)

py-spy dump --pid <pid>
py-spy top --pid <pid>

hey -n 200 -c 1 http://localhost:8000/report
hey -n 200 -c 50 http://localhost:8000/report

The fix

For I/O, use a client that yields. `httpx.AsyncClient` or `aiohttp` instead of `requests`, `asyncpg` or SQLAlchemy’s async engine instead of `psycopg2`, `redis.asyncio` instead of the sync client, `aioboto3` where the AWS SDK is on a hot path. This is the real fix: the loop stays responsive because the call actually suspends.

For work that cannot yield, move it off the loop thread with `await asyncio.to_thread(fn, *args)` — a thread pool suits blocking I/O and C extensions that release the GIL. For genuinely CPU-bound Python, a thread is not enough because of the GIL; use `loop.run_in_executor(ProcessPoolExecutor(), ...)` and accept the serialisation cost, or hand the work to a task queue.

Bound the offload. An unbounded thread pool converts a loop-blocking bug into a thread-explosion bug, and `asyncio.to_thread` uses the default executor whose size is limited but shared with everything else. Give heavy work its own executor with an explicit maximum and a queue, so overload produces backpressure rather than resource exhaustion.

Use the framework’s escape hatch instead of fighting it. In FastAPI, a handler declared `def` rather than `async def` runs in a threadpool automatically — so a legacy endpoint full of sync calls is safer as a plain `def`. Declaring it `async def` and leaving the sync calls inside is the single most common way this bug enters a FastAPI codebase.

Set timeouts everywhere. On a blocked loop, a missing timeout on one dependency becomes a service-wide stall rather than one slow request, so timeouts are load-bearing for isolation and not merely for tidiness.

# Freezes every concurrent request for the duration of the call
@app.get("/report")
async def report(user_id: int):
    resp = requests.get(f"{BILLING}/usage/{user_id}", timeout=10)   # blocks the loop
    digest = bcrypt.hashpw(resp.content, bcrypt.gensalt())          # blocks the loop
    return {"digest": digest.decode()}

# Async client for I/O; CPU work moved to a bounded executor
CPU = ProcessPoolExecutor(max_workers=2)

@app.get("/report")
async def report(user_id: int):
    resp = await http.get(f"{BILLING}/usage/{user_id}", timeout=10.0)
    loop = asyncio.get_running_loop()
    digest = await loop.run_in_executor(CPU, hash_bytes, resp.content)
    return {"digest": digest}

How to stop it coming back

• Export event-loop lag as a first-class metric and alarm on it; it detects this class of regression within one deploy
• Run tests and local development with `-X dev` and `PYTHONASYNCIODEBUG=1` so slow callbacks are visible before review
• Forbid sync clients in async modules with a lint rule — `requests`, `time.sleep`, `psycopg2` imported inside async code should fail CI
• Declare handlers `def` when they contain sync work and `async def` only when they are genuinely non-blocking; do not mix inside one function
• Load-test at realistic concurrency, not one request at a time; serial latency looks perfect on a blocked loop

FAQ

Related

Other errors engineers hit next to this one

• 429 Too Many Requests and Retry-After
• nginx 499 client closed request
• upstream prematurely closed connection
• Intermittent 502 after an idle keep-alive connection
• SSL certificate problem: unable to get local issuer certificate
• ERR_INCOMPLETE_CHUNKED_ENCODING
• Request timeouts cascading into pool exhaustion
• Connection reset by peer on a long-polling endpoint