Python

psycopg2.InterfaceError: connection already closed

Written and reviewed by Sahil Srivastav

psycopg2PostgreSQLConnection lifecycle
psycopg2.InterfaceError: connection already closed
  File "/app/repo/orders.py", line 48, in fetch
    cur.execute(sql, (customer_id,))
  File "/usr/lib/python3/dist-packages/psycopg2/extensions.py", line 129, in cursor
    return self._cnn.cursor()

What this error actually means

A `psycopg2` connection is a Python object wrapping a TCP socket. This error means the object still exists and your code still holds a reference to it, but the underlying socket is no longer usable — either because something closed it, or because the driver marked it dead after a previous failure. The driver refuses the operation locally without going near the network, which is why the traceback is short and contains no PostgreSQL error code at all.

The mechanism that trips most teams: `psycopg2` closes the connection *itself* when a network-level error occurs. If a query dies with `server closed the connection unexpectedly`, the connection is marked closed from that moment on. Every later use of the same object raises `InterfaceError` instead of the original cause. So the exception you are looking at is frequently the second, third and hundredth symptom of one earlier event whose real message was logged elsewhere — or swallowed.

That distinction decides the fix. `OperationalError: server closed the connection unexpectedly` names why the socket died. `InterfaceError: connection already closed` only tells you that something already did. Chasing the second one without finding the first one is the usual reason this bug survives three attempted fixes.

Causes, most common first

  1. 1A connection reused after the server or a proxy dropped it. The dominant cause. PgBouncer `server_idle_timeout`, a cloud load balancer idle timeout (often 350 seconds), a `tcp_keepalive` gap, or a PostgreSQL failover all close the socket silently. Your pool hands the dead connection to the next request, which discovers the closure the hard way.
  2. 2The connection was closed by an earlier error on the same connection. Any network-level failure makes psycopg2 close the connection. If your code catches broadly, logs at debug level and carries on with the same connection object, the real cause disappears and you get a stream of `InterfaceError` instead.
  3. 3A connection inherited across fork(). A connection opened before `fork()` and used in the child shares one socket between two processes. The two sides interleave protocol bytes, one of them closes, and both see corruption or closure. This is the classic Gunicorn preload bug, and the equivalent in Celery with the prefork pool.
  4. 4Your own code closing the connection too early. A context manager whose `__exit__` closes the connection rather than the transaction, a `finally: conn.close()` that runs while a generator is still streaming rows, or returning a connection to a pool and continuing to use the reference you kept.
  5. 5The backend was terminated on purpose. `pg_terminate_backend`, an `idle_in_transaction_session_timeout`, a `statement_timeout` escalation, or the OOM killer taking the backend process. Here PostgreSQL logs the reason, so the server log — not the application log — is where the answer is.

When you see it

  • A burst of identical failures from one process while other processes stay healthy
  • The first failure in the log is a different error — `OperationalError`, `server closed the connection unexpectedly`, or an SSL error — and everything after it is `InterfaceError`
  • It appears after an idle period: the first request of the morning fails, the retry succeeds
  • A forked worker fails immediately on its first query while the parent was fine
  • It starts after putting PgBouncer or a load balancer between the app and PostgreSQL

How to diagnose it

Step 1

Find the first failure, not this one

Search the log window for the earliest error on that worker before the `InterfaceError` flood began. That message carries the real cause and usually a reason string PostgreSQL wrote.

grep -n -m5 -E 'OperationalError|server closed the connection|SSL (SYSCALL|connection has been closed)' app.log

Step 2

Ask PostgreSQL why the backend went away

The server log records terminations with a reason. `FATAL: terminating connection due to administrator command` means someone or something killed it; `idle-in-transaction timeout` means your code held a transaction open.

grep -E 'terminating connection|disconnection: session time' /var/log/postgresql/postgresql-*.log | tail -20

Step 3

Check whether the connection is closed before you use it

`connection.closed` is `0` for open, non-zero once closed. Logging it at borrow time on a sampled basis proves whether your pool is handing out dead connections, which is the difference between a pool configuration fix and a code fix.

python -c "import psycopg2; c=psycopg2.connect(dsn); c.close(); print(c.closed)"

Step 4

Rule fork in or out

Log the process id alongside `id(connection)` at every query. The same connection identity under two different pids is the fork bug and nothing else.

logger.info('pid=%s conn=%s', os.getpid(), id(conn))

The fix

Make dead connections detectable before use rather than during use. With SQLAlchemy that is `create_engine(dsn, pool_pre_ping=True, pool_recycle=<seconds>)`: pre-ping issues a trivial round trip on checkout and transparently replaces a connection that fails it, and `pool_recycle` discards connections older than a bound. Set the recycle value *below* the shortest idle timeout in the path — PgBouncer’s `server_idle_timeout`, the load balancer’s idle timeout, whichever is smaller. If you manage raw `psycopg2` connections yourself, the equivalent is checking `conn.closed` and reconnecting on checkout.

Stop reusing a connection after any exception that could be network-level. On error, discard the connection instead of returning it to the pool. With a pool this is an invalidation, not a close; with raw connections it is `conn.close()` and open a new one. A connection that has seen a socket error is never safe to reuse, regardless of what the retry logic believes.

For the fork case, never inherit a connection. Open connections lazily after the worker starts, or dispose the pool in the post-fork hook — SQLAlchemy’s `engine.dispose()` in Gunicorn’s `post_fork` and Celery’s `worker_process_init` signal. If you preload the application, the pool must be created per worker.

Make retries idempotent and bounded before you add them. Retrying a read after a stale-connection failure is safe; retrying a write whose transaction outcome you do not know is how duplicate rows get created. Retry at the transaction boundary with a fresh connection, not by re-issuing a statement on the dead one.

Set TCP keepalives on the connection so a silently dropped socket is detected in seconds rather than at next use: `keepalives=1 keepalives_idle=30 keepalives_interval=10 keepalives_count=5` in the DSN. This converts a mysterious hang into a prompt, catchable error.

# Dead connection handed out, and reused after failing
conn = pool.getconn()
try:
    with conn.cursor() as cur:
        cur.execute(sql)
finally:
    pool.putconn(conn)          # returns a possibly-dead connection to the pool

# Validate on checkout, discard on any network-level failure
engine = create_engine(
    dsn,
    pool_pre_ping=True,         # cheap round trip before handing the connection over
    pool_recycle=240,           # below PgBouncer server_idle_timeout / LB idle timeout
    connect_args={
        "keepalives": 1,
        "keepalives_idle": 30,
        "keepalives_interval": 10,
        "keepalives_count": 5,
    },
)

# Per-worker pool, never inherited across fork()
def post_fork(server, worker):
    engine.dispose(close=False)

How to stop it coming back

  • Keep `pool_recycle` strictly below every idle timeout in the path, and record those timeouts next to the setting so a proxy change does not silently break it
  • Log the first exception on a connection at error level and invalidate the connection there; a broad `except Exception: pass` around database work turns one diagnosable failure into a thousand undiagnosable ones
  • Dispose pools in `post_fork` / `worker_process_init` as a matter of course, even when you do not preload today
  • Treat a failover drill as a test: kill the primary in staging and assert the service recovers without operator action
  • Alarm on the rate of connection establishment, not just on errors — a pool that silently recreates connections constantly is telling you something is closing them

Practise production debugging in a real repository

Reading about a failure and reproducing one are different skills. Gronex ships broken backend repositories with failing test suites that encode the real invariant, so you debug from evidence instead of memorising symptoms.

FAQ

Is this the same as "server closed the connection unexpectedly"?

They are consecutive, not identical. The server-closed message is the original event, raised as `OperationalError`. `InterfaceError: connection already closed` is what every subsequent use of that same connection object raises. Find the first one; the second one has no diagnostic content.

Will pool_pre_ping slow everything down?

It adds one trivial round trip per checkout — microseconds on a local network, and only on checkout rather than per query. Compared with a user-visible failure and a retry it is essentially free. If even that matters, `pool_recycle` alone removes most of the exposure without any extra round trip.

Why does it only happen with PgBouncer?

Because PgBouncer closes idle server connections on its own schedule, so your pool holds a handle to something that no longer exists. Also check `pool_mode`: in transaction mode, server-side prepared statements and session state do not survive between transactions, which produces different but equally confusing failures.

Should I just wrap every query in a retry?

Not as the primary fix, and never without discarding the connection first. A retry on the same dead connection raises the identical error, and a blind retry of a write can duplicate it if the original transaction actually committed before the socket died. Fix detection on checkout, then retry whole idempotent transactions.

Related

Other errors engineers hit next to this one

Full error and symptom index →